Skip to content

Add password authentication logging to improve visibility

Nick Malcolmrequested to merge
log-password-authentication-events into master

What does this MR do and why?

Logs successful password authentications in find_with_user_password, including which authenticator was used (database, LDAP, OAuth).

This provides visibility into password usage across Git HTTP, JWT, and OAuth ROPC endpoints without adding noise from subsequent authn/z checks. As we only log when password is present, which only occurs during authentication flows, we keep logging as minimal as possible.

  • Log successful authentications with user, authenticator, and request context (IP, user agent, path) when available
  • Pass request parameter through user_with_password_for_git to enable request context logging
  • Extract password_authenticators_for_user method to reduce complexity of find_with_user_password
  • Use a request-based gitlab_com_derisk Feature Flag to enable a representative sample to be collected without performance impact. The derisk type also indicates the temporary nature of this logging.

Resolves https://gitlab.com/gitlab-org/gitlab/-/issues/570179+s

References

Implementation Notes

Solution Design

  • Passwords are checked at times other than authentication - e.g. MRs that require password entry to approve and admins visiting the admin page.
    • Solution 1 We could pass request down deeper in to the stack. It is already in Gitlab::Auth, just not as deep as we need.
    • This will let us log what path was requested - we can focus on just the endpoints we want
  • In controllers, we're too far away from find_with_user_password to know which Authenticator it was that succeeded. We just get :gitlab_or_ldap, for example - not database vs ldap?...
    • Solution 2 We could modify find_with_user_password to return [user, authenticator] but that's a not-insignificant change to a core part of the auth code
  • Solution 3 We could modify Auth::Result so it captures which authenticator was used
    • Currently it has type :gitlab_or_ldap
    • This would be a large amount of work if we want to capture the authenticator for all types of Auth results. And if we don't do that, future folks might be confused why the value is only present after password authn and not other types

Decision: Solution 1. It's the simplest to implement, at the small cost of passing a variable around.

Log format

I couldn't spot a standard for logging, so copied some existing code

  • User attributes
  • The authenticator used (usually Database - the one we're interested in filtering on - but it can be LDAP or other)
  • Request details including:
    • remote IP
    • user agent (could help determine what clients are calling us - docker, git, etc)
Gitlab::AuthLogger.info(
    message: message,
    user_id: user.id,
    username: user.username,
    authenticator: authenticator_class_name,
    remote_ip: request&.remote_ip,
    request_method: request&.request_method,
    path: request&.filtered_path,
    ua: request&.user_agent
)

Copied from:

# app/controllers/repositories/git_http_controller.rb
    Gitlab::AuthLogger.error(
      message: 'Rack_Attack',
      env: :blocklist,
      remote_ip: request.ip,
      request_method: request.request_method,
      path: request.filtered_path
    )
# app/channels/application_cable/logging.rb
    def notification_payload(_)
      super.merge!(
        Labkit::Correlation::CorrelationId::LOG_KEY => request.request_id,
        user_id: current_user&.id,
        username: current_user&.username,
        remote_ip: request.remote_ip,
        ua: request.env['HTTP_USER_AGENT']
      )
    end

How to set up and validate locally

Set the Feature Flag:

  • Feature.enable(:log_find_with_user_password) or
  • Feature.enable_percentage_of_actors(:log_find_with_user_password, 50) to test logging only happens ~50% of the time
  • Feature.disable(:log_find_with_user_password) (default) to test that logging does not happen

Tail the auth logs:

% tail -f log/auth_json.log

Git

Pre-req: Create a repository for the user.

% git clone https://testuser:redacted@gdk.test:3443/testuser/test-repo.git
{"severity":"INFO","time":"2025-09-18T01:15:43.612Z","correlation_id":"01K5D68FJQNHTD89Y12FM9M0TC","meta.caller_id":"Repositories::GitHttpController#info_refs","meta.feature_category":"source_code_management","message":"Gitlab::Auth find_with_user_password succeeded","user_id":94,"username":"testuser","authenticator":"Gitlab::Auth::Database::Authentication","remote_ip":"172.16.123.1","request_method":"GET","path":"/testuser/test-repo.git/info/refs?service=git-upload-pack","ua":"git/2.49.0"}
{"severity":"INFO","time":"2025-09-18T01:15:44.445Z","correlation_id":"01K5D68GD1RX4QTWEZBY5VGQH1","meta.caller_id":"Repositories::GitHttpController#git_upload_pack","meta.feature_category":"source_code_management","message":"Gitlab::Auth find_with_user_password succeeded","user_id":94,"username":"testuser","authenticator":"Gitlab::Auth::Database::Authentication","remote_ip":"172.16.123.1","request_method":"POST","path":"/testuser/test-repo.git/git-upload-pack","ua":"git/2.49.0"}
{"severity":"INFO","time":"2025-09-18T01:15:46.074Z","correlation_id":"01K5D68HZE60064H71JE4APA7G","meta.caller_id":"Repositories::GitHttpController#git_upload_pack","meta.feature_category":"source_code_management","message":"Gitlab::Auth find_with_user_password succeeded","user_id":94,"username":"testuser","authenticator":"Gitlab::Auth::Database::Authentication","remote_ip":"172.16.123.1","request_method":"POST","path":"/testuser/test-repo.git/git-upload-pack","ua":"git/2.49.0"}

JWT authentication (Container Registry)

Pre-req: Follow https://gitlab-org.gitlab.io/gitlab-development-kit/howto/registry/ if needed, including configuring an insecure registry.

% docker login -u testuser -p redacted http://registry.test:5100
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
==> log/auth_json.log <==
{"severity":"INFO","time":"2025-09-18T01:17:05.647Z","correlation_id":"01K5D6AZJ1XK18WF91ABCFAMH9","meta.caller_id":"JwtController#auth","meta.feature_category":"container_registry","message":"Gitlab::Auth find_with_user_password succeeded","user_id":94,"username":"testuser","authenticator":"Gitlab::Auth::Database::Authentication","remote_ip":"172.16.123.1","request_method":"GET","path":"/jwt/auth?account=testuser\u0026client_id=docker\u0026offline_token=[FILTERED]\u0026service=container_registry","ua":"docker/27.3.1 go/go1.23.9 git-commit/41ca978a0a5400cc24b274137efa9f25517fcc0b kernel/6.6.96-0-virt os/linux arch/arm64 UpstreamClient(Docker-Client/28.3.3-rd \\(darwin\\))"}

OAuth password grant (ROPC)

% curl -X POST -d "grant_type=password&username=testuser&password=redacted" \
  https://gdk.test:3443/oauth/token
{"access_token":"...","token_type":"Bearer","expires_in":7200,"refresh_token":"...","scope":"api","created_at":1758158257}
{"severity":"INFO","time":"2025-09-18T01:17:37.599Z","correlation_id":"01K5D6BYBXPEPEB85QNWGEZP45","message":"Gitlab::Auth find_with_user_password succeeded","user_id":94,"username":"testuser","authenticator":"Gitlab::Auth::Database::Authentication","remote_ip":"172.16.123.1","request_method":"POST","path":"/oauth/token","ua":"curl/8.7.1"}

Regular sign in

Note: they're out of scope. But for completeness' sake:

  1. Sign in
  2. Observe no log events in auth_log.json - these logs exist already in audit_log.json: 
    {"severity":"INFO","time":"2025-09-17T03:47:22.310Z","correlation_id":"01K5AWHEDWMNCMVHTFR8RVTN3M","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","id":1110,"author_id":94,"entity_id":94,"entity_type":"User","details":{"with":"standard","event_name":"authenticated_with_password","author_name":"Test User","author_class":"User","target_id":94,"target_type":"User","target_details":"Test User","custom_message":"Signed in with standard authentication","ip_address":"172.16.123.1","entity_path":"testuser"},"ip_address":"172.16.123.1","author_name":"Test User","entity_path":"testuser","target_details":"Test User","created_at":"2025-09-17T03:47:22.271Z","target_type":"User","target_id":94,"with":"standard","event_name":"authenticated_with_password","author_class":"User","custom_message":"Signed in with standard authentication"}

How much will this increase logging?

Since this references the volume of production requests, please see the confidential issue: https://gitlab.com/gitlab-org/gitlab/-/issues/570179#note_2761857659

In 0e86f4 we introduced a Feature Flag so that we can collect a representative sample of logs instead of all logs.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Nick Malcolm

Merge request reports