Elasticsearch 6 and memory
Summary
Trying to run elasticsearch 6 in the gitlab runner as a service.
my sevices i nthe gitlab-ci.yml that generate issue:
services:
- postgres:latest
- docker.elastic.co/elasticsearch/elasticsearch:6.2.2
- redis:latestIt seems to not be able to run the service and close it down.
I do notice max virtual memory areas vm.max_map_count and I do run my own worker. how do I set this when it scales up and down?
Steps to reproduce
Add docker.elastic.co/elasticsearch/elasticsearch:6.2.2 to the services in the gitlab-ci.yml and run
Actual behavior
it will give a log when running with stopping messages and then it will fail when the tests try to connect to the elasticsearch.
Expected behavior
elasticsearch 6.0 runs smoothly and work in the runner.
Relevant logs and/or screenshots
Waiting for services to be up and running...
*** WARNING: Service runner-88879534-project-3237414-concurrent-0-docker.elastic.co__elasticsearch__elasticsearch-1 probably didn't start properly.
service runner-88879534-project-3237414-concurrent-0-docker.elastic.co__elasticsearch__elasticsearch-1-wait-for-service did timeout
2018-03-08T12:11:16.961652213Z [2018-03-08T12:11:16,958][INFO ][o.e.n.Node ] [] initializing ...
2018-03-08T12:11:17.113189783Z [2018-03-08T12:11:17,112][INFO ][o.e.e.NodeEnvironment ] [6842vYc] using [1] data paths, mounts [[/ (overlay)]], net usable_space [52.3gb], net total_space [55.8gb], types [overlay]
2018-03-08T12:11:17.116212281Z [2018-03-08T12:11:17,115][INFO ][o.e.e.NodeEnvironment ] [6842vYc] heap size [1007.3mb], compressed ordinary object pointers [true]
2018-03-08T12:11:17.119132076Z [2018-03-08T12:11:17,118][INFO ][o.e.n.Node ] node name [6842vYc] derived from node ID [6842vYcbS2GS1hMNfnkpdg]; set [node.name] to override
2018-03-08T12:11:17.119768031Z [2018-03-08T12:11:17,119][INFO ][o.e.n.Node ] version[6.2.2], pid[1], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.14.19-coreos/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
2018-03-08T12:11:17.120191121Z [2018-03-08T12:11:17,119][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.zfxGVxTB, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
2018-03-08T12:11:21.284918505Z [2018-03-08T12:11:21,284][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [aggs-matrix-stats]
2018-03-08T12:11:21.285273016Z [2018-03-08T12:11:21,285][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [analysis-common]
2018-03-08T12:11:21.285441635Z [2018-03-08T12:11:21,285][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [ingest-common]
2018-03-08T12:11:21.285676288Z [2018-03-08T12:11:21,285][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [lang-expression]
2018-03-08T12:11:21.285842575Z [2018-03-08T12:11:21,285][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [lang-mustache]
2018-03-08T12:11:21.286058782Z [2018-03-08T12:11:21,285][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [lang-painless]
2018-03-08T12:11:21.286235056Z [2018-03-08T12:11:21,286][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [mapper-extras]
2018-03-08T12:11:21.286412097Z [2018-03-08T12:11:21,286][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [parent-join]
2018-03-08T12:11:21.286581135Z [2018-03-08T12:11:21,286][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [percolator]
2018-03-08T12:11:21.286806677Z [2018-03-08T12:11:21,286][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [rank-eval]
2018-03-08T12:11:21.286972775Z [2018-03-08T12:11:21,286][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [reindex]
2018-03-08T12:11:21.287147675Z [2018-03-08T12:11:21,286][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [repository-url]
2018-03-08T12:11:21.287323780Z [2018-03-08T12:11:21,287][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [transport-netty4]
2018-03-08T12:11:21.287571848Z [2018-03-08T12:11:21,287][INFO ][o.e.p.PluginsService ] [6842vYc] loaded module [tribe]
2018-03-08T12:11:21.288550585Z [2018-03-08T12:11:21,288][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [ingest-geoip]
2018-03-08T12:11:21.289353624Z [2018-03-08T12:11:21,288][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [ingest-user-agent]
2018-03-08T12:11:21.289522118Z [2018-03-08T12:11:21,289][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [x-pack-core]
2018-03-08T12:11:21.290387416Z [2018-03-08T12:11:21,289][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [x-pack-deprecation]
2018-03-08T12:11:21.290561913Z [2018-03-08T12:11:21,290][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [x-pack-graph]
2018-03-08T12:11:21.291243336Z [2018-03-08T12:11:21,291][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [x-pack-logstash]
2018-03-08T12:11:21.291413666Z [2018-03-08T12:11:21,291][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [x-pack-ml]
2018-03-08T12:11:21.291574616Z [2018-03-08T12:11:21,291][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [x-pack-monitoring]
2018-03-08T12:11:21.291723641Z [2018-03-08T12:11:21,291][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [x-pack-security]
2018-03-08T12:11:21.291890388Z [2018-03-08T12:11:21,291][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [x-pack-upgrade]
2018-03-08T12:11:21.292058019Z [2018-03-08T12:11:21,291][INFO ][o.e.p.PluginsService ] [6842vYc] loaded plugin [x-pack-watcher]
2018-03-08T12:11:29.396064710Z [2018-03-08T12:11:29,395][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/104] [Main.cc@128] controller (64 bit): Version 6.2.2 (Build 61c43548e5e0f2) Copyright (c) 2018 Elasticsearch BV
2018-03-08T12:11:30.627284591Z [2018-03-08T12:11:30,626][INFO ][o.e.d.DiscoveryModule ] [6842vYc] using discovery type [zen]
2018-03-08T12:11:32.133996763Z [2018-03-08T12:11:32,133][INFO ][o.e.n.Node ] initialized
2018-03-08T12:11:32.134730785Z [2018-03-08T12:11:32,133][INFO ][o.e.n.Node ] [6842vYc] starting ...
2018-03-08T12:11:32.549018050Z [2018-03-08T12:11:32,548][INFO ][o.e.t.TransportService ] [6842vYc] publish_address {172.17.0.3:9300}, bound_addresses {0.0.0.0:9300}
2018-03-08T12:11:32.573461887Z [2018-03-08T12:11:32,573][INFO ][o.e.b.BootstrapChecks ] [6842vYc] bound or publishing to a non-loopback address, enforcing bootstrap checks
2018-03-08T12:11:32.584550011Z ERROR: [1] bootstrap checks failed
2018-03-08T12:11:32.584591305Z [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
2018-03-08T12:11:32.587551982Z [2018-03-08T12:11:32,587][INFO ][o.e.n.Node ] [6842vYc] stopping ...
2018-03-08T12:11:32.609481974Z [2018-03-08T12:11:32,609][INFO ][o.e.n.Node ] [6842vYc] stopped
2018-03-08T12:11:32.609716366Z [2018-03-08T12:11:32,609][INFO ][o.e.n.Node ] [6842vYc] closing ...
2018-03-08T12:11:32.626986407Z [2018-03-08T12:11:32,626][INFO ][o.e.n.Node ] [6842vYc] closed
2018-03-08T12:11:32.630063240Z [2018-03-08T12:11:32,629][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
*********Environment description
Running our own runner that scales up and down.
Used GitLab Runner version
Running with gitlab-runner 10.0.2 (a9a76a50)
on autoscaler-runner (88879534)
Using Docker executor with image ruby:2.4.3 ...