Errors do not always show up

While playing around with https://gitlab.com/gitlab-org/opstrace/opstrace/-/error_tracking , I've noticed sometimes errors pop out and sometimes they don’t. I'd say around 30% of the times, no errors are shown. Not sure if this is specific to opstrace, or apply to any project.

I did not test it locally or staging.

added to epic &64

moved from opstrace-ui#253 (moved)

changed the description

changed milestone to %16.0

@drosse - please see the following guidance and update this issue.

	1 Error
	Please add typebug typefeature, typemaintenance and a subtype label to this issue.

If you do not feel the purpose of this issue matches one of the types, you may apply the typeignore label to exclude it from type tracking metrics and future prompts.

This message was generated automatically. You're welcome to improve it.

added devopsmonitor groupobservability labels

added sectionops label

changed the description

It happens on production but also canary (see https://next.gitlab.com/).

I wasn't able to reproduce this behavior on staging in https://staging.gitlab.com/gitlab-org/monitor/monitor-sandbox/-/error_tracking.

mentioned in issue gitlab-org/monitor/respond#230 (closed)

added priority1 label and removed devopsmonitor groupobservability sectionops labels

added groupobservability label

added devopsmonitor sectionops labels

added typebug label

assigned to @joe-shaw

added workflowin dev label

@drosse is there a consistent way to reproduce this?

I've hard refreshed the page a bunch and I always see the error showing.

I also no longer can reproduce it

Hmm... Let me check the MRs since it was reported and see if anything potentially fixed it.

Note that I was only able to reproduce it on production but not on staging.

See #2146 (comment 1349753406)

I feel something was off with the redis workers?

A few things have changed in production since the release before this issue was reported (April 6th) and the subsequent releases (last on April 11th).

• ClickHouse got bigger nodes and more resources assigned.
• Some deployment manifests stuff not really related to error tracking.

So, probably none of that stuff, and there was no release when this issue was reported.

You then also see the issue again on the 12th. So I think we can rule out GOB changes fixing this.

Redis workers, the ones in GitLab presumably? Are they used for error tracking?

Redis workers, the ones in GitLab presumably? Are they used for error tracking?

Yes, GitLab is using reactive caching in order to fetch data from external APIs. See:

• https://gitlab.com/gitlab-org/gitlab/-/blob/76148ca2e7defa457d6ec873ba3230a6bedc6c72/app/models/error_tracking/project_error_tracking_setting.rb#L117-121
• https://gitlab.com/gitlab-org/gitlab/-/blob/76148ca2e7defa457d6ec873ba3230a6bedc6c72/app/models/error_tracking/project_error_tracking_setting.rb#L155-156

Ah yes, in that case it sounds like a classic caching problem, given the apparent non-determinism.

I'll keep an eye on it for a day or so and close it.

Yeah I can also not repro anymore!

I noticed this morning that they were missing again, but then appeared to have been "Resolved" (i.e. I can see the error when switching to the resolved view).

It appears I cannot "Unresolve" the error now @drosse any idea if that ever worked?

I don't think I've played around with the status updates yet. Wanna give it a go and create an issue if something is broken?

Yeah, I was just checking to see if we knew it was already an issue.

I'll open an issue for it.

#2159 (closed)

mentioned in issue #2158

mentioned in issue gitlab-org/monitor/respond#232 (closed)

added Deliverable label

Right, I'm closing this one as we have other particular errors and I cannot reproduce this one specifically.

closed

changed epic to &71 (closed)