Agent doesn't log any errors when it can't connect to Rails

MR: Pending

Description

Currently, when there is Rails outage, the agentk doesn't log any issues.

kas logs some connection issues but only as a debug level logs.

Acceptance Criteria

• Investigate if it is possible for agentk to identify Rails outage
• Capture such errors if possible/available and log them within remote_dev module for easier debugging

Technical Requirements

• Same as acceptance criteria

Design Requirements

• NA

Impact Assessment

• NA

Original Description

Currently, when there is Rails outage, the agentk doesn't log any issues.

kas logs some connection issues but only as a debug level logs.

---

Reproduce with

1. turn of rails with gdk stop rails-web
2. kas starts logging debug messages like

{"level":"debug","time":"2023-04-25T14:46:27.944+0200","msg":"gRPC -> HTTP: SendMsg(HttpResponse_Data) failed","grpc_service":"gitlab.agent.gitlab_access.rpc.GitlabAccess","grpc_method":"MakeRequest","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"debug","time":"2023-04-25T14:46:37.948+0200","msg":"Handling POST /","grpc_service":"gitlab.agent.gitlab_access.rpc.GitlabAccess","grpc_method":"MakeRequest"}

1. agentk keeps logging that it starts a partial update

{"level":"info","time":"2023-04-25T14:46:26.304+0200","msg":"starting partial update","mod_name":"remote_development","agent_id":2}

added to epic &10461 (closed)

changed milestone to %16.0

added Category:Remote Development backend devopscreate featureaddition groupide sectiondev typefeature workflowready for development labels

changed title from Log errors in agent if it can't connect to Rails to Agent doesn't log any errors when it can't connect to Rails

changed the description

mentioned in issue #407723 (closed)

mentioned in epic &10461 (closed)

mentioned in issue gitlab-org/quality/triage-reports#12229 (closed)

added quad-planningcomplete-no-action label

changed milestone to %16.1

added missed:16.0 label

added rd-workflowunprioritized label

changed milestone to %16.2

added missed:16.1 label

changed iteration to Workspaces Jul 17, 2023 - Jul 23, 2023

removed iteration Workspaces Jul 17, 2023 - Jul 23, 2023

changed milestone to %Next 1-3 releases

@hkhanna2

Can you refine this issue using the Remote Development issue template.

• https://about.gitlab.com/handbook/engineering/development/dev/create/ide/#2-pre-iteration-planning-meeting

assigned to @hkhanna2

changed the description

added maintenanceusability refined typemaintenance labels and removed typefeature label

removed featureaddition label

removed refined label

added 1 deleted label

All SUS-impacting issues need to have a proper severity label set. Please add a severity label, remove the automation:ux-missing-labels label, and then reply to this comment briefly explaining your reasoning for providing this severity.

If you are not the DRI for this area and would like help determining the best severity, please @ the appropriate person for assistance.

This message was generated automatically. You're welcome to improve it.

added automation:ux-missing-labels label

Update 2023-08-03

I'm unable to reproduce this issue and it seems like agent prints logs when it is unable to connect to rails.

Steps carried out

1. Start gitlab locally using gdk
2. Start KAS & agentk locally
3. Verify the smooth functioning of remote dev module in agentk indicated by the presence of reconciliation logs

{"level":"info","time":"2023-08-03T11:55:14.857+0800","msg":"starting full sync","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2023-08-03T11:55:24.858+0800","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2023-08-03T11:55:34.944+0800","msg":"starting partial update","mod_name":"remote_development","agent_id":2}

4. Stop rails using gdk stop rails-web
5. There were agent logs indicating error when connecting to Rails

{"level":"info","time":"2023-08-03T11:57:19.486+0800","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"error","time":"2023-08-03T11:57:21.092+0800","msg":"Remote Dev - partial sync cycle ended with error","mod_name":"remote_development","error":"unexpected status code: 502","agent_id":2}
{"level":"info","time":"2023-08-03T11:57:31.094+0800","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"error","time":"2023-08-03T11:57:32.702+0800","msg":"Remote Dev - partial sync cycle ended with error","mod_name":"remote_development","error":"unexpected status code: 502","agent_id":2}

6. Start rails again using gdk start rails-web
7. The agent recovers and is able to resume its normal functioning as indicated by missing error logs

{"level":"info","time":"2023-08-03T12:03:38.860+0800","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2023-08-03T12:03:48.917+0800","msg":"starting partial update","mod_name":"remote_development","agent_id":2}
{"level":"info","time":"2023-08-03T12:03:59.034+0800","msg":"starting partial update","mod_name":"remote_development","agent_id":2}

The way I see it, maybe this issue was addressed in a later fix but it doesn't exist anymore.

@oregand is it ok to close this issue? Tomas is on leave for a while so we're unlikely to continue any discussion on this for a while. Maybe it can be re-opened when he's back / he has any concerns?

is it ok to close this issue? Tomas is on leave for a while so we're unlikely to continue any discussion on this for a while. Maybe it can be re-opened when he's back / he has any concerns?

Lets close this out for now and circle back later if the need comes up

closed

@viktomas this issue had the Category:Remote Development label and was closed. The rd-workflowdone label has been applied.

added rd-workflowdone label and removed rd-workflowunprioritized label

changed milestone to %16.4