Regression: HTTP 500 on first daily git fetch (NilClass in FetchStatisticsIncrementService) - Started in 18.6.1

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

After upgrading to GitLab 18.6.1 (and persisting in 18.7.0-ee), we are experiencing intermittent HTTP 500 errors when performing git clone or git pull operations.

The error strictly occurs on the first access of the day (UTC) for a given repository. Retrying the exact same command immediately afterwards succeeds.

Steps to reproduce

  1. Upgrade GitLab from 18.5.x to 18.6.1 (or 18.7.0).
  2. Wait for the date to change (UTC).
  3. Perform a git clone or git fetch on a repository that has not been accessed yet that day.
  4. The client receives an HTTP 500 error.
  5. Performing the command again succeeds.

Example Project

Any project in the instance.

```json

What is the current bug behavior?

The first git operation fails with: error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500

$ git pull 
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 fatal: expected flush after ref listing

Production logs show a NoMethodError: undefined method 'increment_fetch_count' for nil:NilClass in FetchStatisticsIncrementService.

What is the expected correct behavior?

Git operations should succeed on the first attempt without internal server errors, regardless of daily statistics initialization.

Relevant logs and/or screenshots

Upgrade Path & History: The issue started appearing immediately after the upgrade to 18.6.1.

  • Path: 18.4.2 -> 18.5.3 -> 18.6.1 (Issue started here) -> 18.7.0 (Issue persists)

Error Log (production_json.log):

{
  "method": "POST",
  "path": "/root/web.git/git-upload-pack",
  "controller": "Repositories::GitHttpController",
  "action": "git_upload_pack",
  "status": 500,
  "exception.class": "NoMethodError",
  "exception.message": "undefined method 'increment_fetch_count' for nil:NilClass",
  "exception.backtrace": [
    "app/services/projects/fetch_statistics_increment_service.rb:14:in `execute'",
    "app/controllers/repositories/git_http_controller.rb:110:in `update_fetch_statistics'"
  ]
}

Environment:

  • GitLab Version: 18.7.0-ee
  • OS: RHEL/CentOS
  • PostgreSQL: 16.10
  • Installation Type: Omnibus

Validated Workaround

After testing, we found that our previous script failed in EE environments due to different association names (project_state vs repository_state).

We have updated the script to be compatible with both editions and fixed variable scope issues. This script successfully pre-warms the daily statistics, bypassing the 500 Internal Server Error caused by the buggy increment_fetch_count logic.

The Fix Script (Universal)

Script: /opt/gitlab/scripts/init_daily_stats.rb

#!/opt/gitlab/embedded/bin/ruby

# Load Rails environment
require '/opt/gitlab/embedded/service/gitlab-rails/config/environment'

today = Date.current
puts "=== Pre-warming Daily Statistics for #{today} ==="

@success_count = 0
@failed_count = 0

# Iterate through ALL projects to ensure 100% coverage
Project.find_each(batch_size: 100) do |project|
  begin
    # 1. Ensure state record exists (Handles both EE and CE associations)
    if project.respond_to?(:project_state) # EE Environment
      project.create_project_state! unless project.project_state
    elsif project.respond_to?(:repository_state) # CE Environment
      project.create_repository_state! unless project.repository_state
    end

    # 2. Pre-create the daily statistic record
    # CRITICAL: Existence of this record forces the application to skip the 
    # broken 'increment_fetch_count' code path in FetchStatisticsIncrementService.
    ProjectDailyStatistic.find_or_create_by!(
      project_id: project.id,
      date: today
    )
    
    @success_count += 1
  rescue => e
    @failed_count += 1
    puts "FAIL: Project #{project.full_path} (ID: #{project.id}) - #{e.message}"
  end
end

puts "=== Finished ==="
puts "Success: #{@success_count}"
puts "Failed: #{@failed_count}"

Cron Job Configuration:

# Run at 01:00 AM daily 0 1 * * * /opt/gitlab/bin/gitlab-rails runner /opt/gitlab/scripts/init_daily_stats.rb >> /var/log/gitlab/init_daily_stats.log 2>&1

\

Edited by 🤖 GitLab Bot 🤖