Skip to content

Sync Full & Partial Reconcilation From Rails & Apply

Zhaochen Li requested to merge zl/570-add-settings-and-apply-reconclilation-interval into master

Issue - Agent New Settings Config With Default API valu... (#570 - closed)

Sync Full & Partial Reconcilation From Rails & Apply

Tested locally with the code on Rails changes and also additional logging locally.

From the logging below, we could see, when start the service, we first directly triggered a full reconciliation, and the default interval is 0 for full and 10 for partial.

Then with this reconciliation, we apply intervals fetched back from Rails, I entered 100 for full, and 40 for partial, on the Rails side.

Then we could see the next partial reconciliation happened after 40 seconds.

Then i turned down the Rails app, so the up-coming reconciliation loop fails, but it still takes the previous setted 100 for full, and 40 for partial. This action is minic the behaviour of network issue or Rails server down, so the api call failed.

Then i start the rails app again with new config, 200 for full and 10 for partial.

And we could see, there is immediate partial reconciliation after 10s.

~/Projects/gitlab-development-kit/gitlab-k8s-agent   zl/570-add-settings-and-apply-reconclilation-interval ±✚  bazel run //cmd/agentk -- --kas-address=grpc://gdk.test:8150 --token-file=$HOME/.gitlab-agent-token.txt
INFO: Analyzed target //cmd/agentk:agentk (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //cmd/agentk:agentk up-to-date:
  /private/var/tmp/_bazel_zhaochen.li/1a01dd6ed557b352576661e9e26c613f/execroot/gitlab_k8s_agent/bazel-out/darwin_arm64-fastbuild/bin/cmd/agentk/agentk_/agentk
INFO: Elapsed time: 0.182s, Critical Path: 0.05s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: /private/var/tmp/_bazel_zhaochen.li/1a01dd6ed557b352576661e9e26c613f/execroot/gitlab_k8s_agent/bazel-out/darwin_arm64-fastbuild/bin/cmd/agentk/agentk_/agentk '--kas-address=grpc://gdk.test:8150' '--token-file=/Users/zhaochen.li/.gitlab-agent-token.txt'
{"level":"info","time":"2024-05-20T13:44:25.767+1000","msg":"Module is disabled. Set the 'OCS_ENABLED' environment variable to 'true` to enable it","mod_name":"starboard_vulnerability"}
{"level":"info","time":"2024-05-20T13:44:25.775+1000","msg":"Flux is not installed, skipping module. A restart is required for this to be checked again","mod_name":"flux"}
{"level":"info","time":"2024-05-20T13:44:25.775+1000","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"[::]:8080"}
{"level":"info","time":"2024-05-20T13:44:26.453+1000","msg":"attempting to acquire leader lease default/agent-2-lock...","agent_id":2}
{"level":"info","time":"2024-05-20T13:44:26.465+1000","msg":"successfully acquired lease default/agent-2-lock","agent_id":2}
{"level":"info","time":"2024-05-20T13:44:26.465+1000","msg":"Event occurred","object":{"name":"agent-2-lock","namespace":"default"},"fieldPath":"","kind":"Lease","apiVersion":"coordination.k8s.io/v1","type":"Normal","reason":"LeaderElection","message":"remotedev became leader","agent_id":2}
{"level":"debug","time":"2024-05-20T13:44:26.465+1000","msg":"Remote Development - starting reconciler run","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2024-05-20T13:44:26.465+1000","msg":"zl-test starting reconciliation loop","mod_name":"remote_development","full interval":0,"partial interval":10,"agent_id":2}
{"level":"info","time":"2024-05-20T13:44:26.465+1000","msg":"starting full reconciliation","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:44:26.566+1000","msg":"Running reconciliation loop","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:44:26.566+1000","msg":"Making GitLab request","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:44:26.720+1000","msg":"Made request to the Rails API","mod_name":"remote_development","status_code":201,"request_id":"01HYA1KFYBDE0BWQAXN7PYNA24","duration_in_ms":154,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:44:26.722+1000","msg":"Read body from the Rails API","mod_name":"remote_development","payload_size_in_bytes":129,"workspace_data_count":0,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:44:26.722+1000","msg":"Reconciliation loop ended","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2024-05-20T13:44:26.722+1000","msg":"zl-test finished full reconciliation loop","mod_name":"remote_development","full interval":100,"partial interval":40,"agent_id":2}
{"level":"info","time":"2024-05-20T13:45:06.723+1000","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:45:06.723+1000","msg":"Running reconciliation loop","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:45:06.723+1000","msg":"Making GitLab request","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:45:06.833+1000","msg":"Made request to the Rails API","mod_name":"remote_development","status_code":201,"request_id":"01HYA1MQ573AGYKGH7PDQJNPDK","duration_in_ms":110,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:45:06.834+1000","msg":"Read body from the Rails API","mod_name":"remote_development","payload_size_in_bytes":129,"workspace_data_count":0,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:45:06.834+1000","msg":"Reconciliation loop ended","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2024-05-20T13:45:06.834+1000","msg":"zl-test finished partial reconciliation loop","mod_name":"remote_development","full interval":100,"partial interval":40,"agent_id":2}
{"level":"warn","time":"2024-05-20T13:45:34.673+1000","msg":"[core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: \"172.16.123.1:8150\", ServerName: \"gdk.test:8150\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"debug","time":"2024-05-20T13:45:35.674+1000","msg":"Handled a connection successfully","mod_name":"agentk2kas_tunnel","agent_id":2}
{"level":"debug","time":"2024-05-20T13:45:35.675+1000","msg":"Handled a connection successfully","mod_name":"agentk2kas_tunnel","agent_id":2}
{"level":"warn","time":"2024-05-20T13:45:35.675+1000","msg":"[core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: \"172.16.123.1:8150\", ServerName: \"gdk.test:8150\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"warn","time":"2024-05-20T13:45:37.294+1000","msg":"[core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: \"172.16.123.1:8150\", ServerName: \"gdk.test:8150\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"warn","time":"2024-05-20T13:45:40.232+1000","msg":"[core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: \"172.16.123.1:8150\", ServerName: \"gdk.test:8150\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"warn","time":"2024-05-20T13:45:44.170+1000","msg":"[core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: \"172.16.123.1:8150\", ServerName: \"gdk.test:8150\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"info","time":"2024-05-20T13:45:46.834+1000","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:45:46.835+1000","msg":"Running reconciliation loop","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:45:46.835+1000","msg":"Making GitLab request","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:45:46.835+1000","msg":"Reconciliation loop ended","mod_name":"remote_development","agent_id":2}
{"level":"error","time":"2024-05-20T13:45:46.835+1000","msg":"Remote Dev - partial reconciliation cycle ended with error","mod_name":"remote_development","error":"error making api request: rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"info","time":"2024-05-20T13:45:46.835+1000","msg":"zl-test finished partial reconciliation loop","mod_name":"remote_development","full interval":100,"partial interval":40,"agent_id":2}
{"level":"warn","time":"2024-05-20T13:45:50.623+1000","msg":"[core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: \"172.16.123.1:8150\", ServerName: \"gdk.test:8150\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"warn","time":"2024-05-20T13:46:00.509+1000","msg":"[core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: \"172.16.123.1:8150\", ServerName: \"gdk.test:8150\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"info","time":"2024-05-20T13:46:06.722+1000","msg":"starting full reconciliation","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2024-05-20T13:46:06.722+1000","msg":"informer stopped","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:46:06.823+1000","msg":"Running reconciliation loop","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:46:06.823+1000","msg":"Making GitLab request","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:46:06.823+1000","msg":"Reconciliation loop ended","mod_name":"remote_development","agent_id":2}
{"level":"error","time":"2024-05-20T13:46:06.823+1000","msg":"Remote Dev - full reconciliation cycle ended with error","mod_name":"remote_development","error":"error making api request: rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"info","time":"2024-05-20T13:46:06.823+1000","msg":"zl-test finished full reconciliation loop","mod_name":"remote_development","full interval":100,"partial interval":40,"agent_id":2}
{"level":"warn","time":"2024-05-20T13:46:16.455+1000","msg":"[core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: \"172.16.123.1:8150\", ServerName: \"gdk.test:8150\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"info","time":"2024-05-20T13:46:26.836+1000","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:46:26.836+1000","msg":"Running reconciliation loop","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:46:26.839+1000","msg":"Making GitLab request","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:46:26.840+1000","msg":"Reconciliation loop ended","mod_name":"remote_development","agent_id":2}
{"level":"error","time":"2024-05-20T13:46:26.840+1000","msg":"Remote Dev - partial reconciliation cycle ended with error","mod_name":"remote_development","error":"error making api request: rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 172.16.123.1:8150: connect: connection refused\"","agent_id":2}
{"level":"info","time":"2024-05-20T13:46:26.840+1000","msg":"zl-test finished partial reconciliation loop","mod_name":"remote_development","full interval":100,"partial interval":40,"agent_id":2}
{"level":"info","time":"2024-05-20T13:47:06.840+1000","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:06.840+1000","msg":"Running reconciliation loop","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:06.840+1000","msg":"Making GitLab request","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:06.922+1000","msg":"Made request to the Rails API","mod_name":"remote_development","status_code":201,"request_id":"01HYA1RCEZ7S9H5CGCCRSPSPYJ","duration_in_ms":82,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:06.924+1000","msg":"Read body from the Rails API","mod_name":"remote_development","payload_size_in_bytes":129,"workspace_data_count":0,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:06.924+1000","msg":"Reconciliation loop ended","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2024-05-20T13:47:06.924+1000","msg":"zl-test finished partial reconciliation loop","mod_name":"remote_development","full interval":200,"partial interval":10,"agent_id":2}
{"level":"info","time":"2024-05-20T13:47:16.925+1000","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:16.925+1000","msg":"Running reconciliation loop","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:16.925+1000","msg":"Making GitLab request","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:17.027+1000","msg":"Made request to the Rails API","mod_name":"remote_development","status_code":201,"request_id":"01HYA1RPA1Q49XAV63DY4MR8BR","duration_in_ms":102,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:17.029+1000","msg":"Read body from the Rails API","mod_name":"remote_development","payload_size_in_bytes":129,"workspace_data_count":0,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:17.029+1000","msg":"Reconciliation loop ended","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2024-05-20T13:47:17.029+1000","msg":"zl-test finished partial reconciliation loop","mod_name":"remote_development","full interval":200,"partial interval":10,"agent_id":2}
{"level":"info","time":"2024-05-20T13:47:27.030+1000","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:27.030+1000","msg":"Running reconciliation loop","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:27.030+1000","msg":"Making GitLab request","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:27.747+1000","msg":"Made request to the Rails API","mod_name":"remote_development","status_code":201,"request_id":"01HYA1S05TZZ717E01284F04TT","duration_in_ms":717,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:27.749+1000","msg":"Read body from the Rails API","mod_name":"remote_development","payload_size_in_bytes":129,"workspace_data_count":0,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:27.749+1000","msg":"Reconciliation loop ended","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2024-05-20T13:47:27.749+1000","msg":"zl-test finished partial reconciliation loop","mod_name":"remote_development","full interval":200,"partial interval":10,"agent_id":2}
{"level":"info","time":"2024-05-20T13:47:37.749+1000","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:37.750+1000","msg":"Running reconciliation loop","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:37.750+1000","msg":"Making GitLab request","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:37.775+1000","msg":"Made request to the Rails API","mod_name":"remote_development","status_code":201,"request_id":"01HYA1SAMRRSW65BVNR83GAHZ0","duration_in_ms":25,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:37.777+1000","msg":"Read body from the Rails API","mod_name":"remote_development","payload_size_in_bytes":129,"workspace_data_count":0,"agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:37.777+1000","msg":"Reconciliation loop ended","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2024-05-20T13:47:37.777+1000","msg":"zl-test finished partial reconciliation loop","mod_name":"remote_development","full interval":200,"partial interval":10,"agent_id":2}
^C{"level":"debug","time":"2024-05-20T13:47:41.596+1000","msg":"Canceled connection","mod_name":"agentk2kas_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:41.596+1000","msg":"Connection done","mod_name":"agentk2kas_tunnel","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:41.596+1000","msg":"Canceled connection","mod_name":"agentk2kas_tunnel","error":"rpc error: code = Canceled desc = context canceled","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:41.596+1000","msg":"Connection done","mod_name":"agentk2kas_tunnel","agent_id":2}
{"level":"info","time":"2024-05-20T13:47:41.597+1000","msg":"informer stopped","mod_name":"remote_development","agent_id":2}
{"level":"debug","time":"2024-05-20T13:47:41.597+1000","msg":"Remote Development - reconciler run ended","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2024-05-20T13:47:41.603+1000","msg":"Event occurred","object":{"name":"agent-2-lock","namespace":"default"},"fieldPath":"","kind":"Lease","apiVersion":"coordination.k8s.io/v1","type":"Normal","reason":"LeaderElection","message":"remotedev stopped leading","agent_id":2}
Edited by Zhaochen Li

Merge request reports