Issue page bug - `realtime_changes` contains overwrites new description with old one

Description

Noticed on .com

Screencast_2022-03-17_11_42_46

For some reason the realtime_changes endpoint was considered NOT_CHANGED and continued returning the old description. This caused the new description to be replaced seconds after loading the page.

After some time, it looks like realtime_changes fixed itself:

#355922 (closed)

added bugavailability devopsplan frontend severity2 typebug labels

@johnhope or @donaldcook is this bug known? Is there someone on the team who could take a look at this?

I don't see changes in the backend code that could've caused this. Can't rule out a saturation issue with a downstream service, such as Redis, but as far as I can see the /realtime_changes endpoint loads the issuable each time and returns the current state as in the database.

This is strange though because it implies the update hadn't been persisted to the database yet 🤔

@donaldcook are you aware of any changes on the FE?

are you aware of any changes on the FE?

@johnhope None that I am aware of. We did some work on the description component around tasks, but that was not merged in until the 18th (!81294 (merged)), so don't think that is it. The underlying Polling service hasn't been touched in over a year.

added sectiondev label

added priority2 label

severity2 with ~availability can only have priority1

Please see the Availability prioritization guidelines for more detail.

mentioned in issue #356222 (closed)

added backend label

severity2 with ~availability can only have priority1

Please see the Availability prioritization guidelines for more detail.

For some reason the realtime_changes endpoint was considered NOT_CHANGED and continued returning the old description.

This is just how our ETag-based polling works. When the backend returns a 304, it actually returns an empty response body. But with 304s, browsers treat that as "use the previous response".

I think it was @mikegreiling that reported this happening before.

I think there are many edge cases where the response to a poll would be the old description but the latest ETag. An example would be DB replication lag. It could also happen if a description happen and it fails on the ETag update part.

Unfortunately, this does not fix itself on the next poll and actually breaks things even after refresh as described in this issue. The workaround for this would be to clear the browser cache or click "Disable cache" in the inspector.

If we can move these polling endpoints to use subscriptions, we won't have these types of problems.

Given there was a site-wide degradation incident at the time related to DB pressure I'm going to chalk this up to that and close it out.

@jlear as DRI for groupproject management and @pslaughter as the author please reopen if you disagree.

We're likely to continue to have problems during periods of DB pressure but I think the time spent building mitigations to that would be better spent on #300208.

closed