Geo status API endpoint fails in schema migration
I was doing an upgrade from 12.1 to 12.2 on the primary node, and while doing that, I was running:
watch -d -c -n1 curl --silent --header 'PRIVATE-TOKEN:$(cat token)' 'https://toon-two.gogitlab.ml/api/v4/geo_nodes/2/status' \| jq '.' -C
This was nicely giving my JSON, until the schema migrations were replicated to the secondary.
Log file
Started GET "/api/v4/geo_nodes/2/status" for 81.83.195.177 at 2019-09-19 10:40:38 +0000
ActiveModel::UnknownAttributeError (unknown attribute 'container_repositories_count' for GeoNodeStatus.):
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-5.2.3/lib/active_model/attribute_assignment.rb:53:in `_assign_attribute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-5.2.3/lib/active_model/attribute_assignment.rb:44:in `block in _assign_attributes'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-5.2.3/lib/active_model/attribute_assignment.rb:43:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-5.2.3/lib/active_model/attribute_assignment.rb:43:in `_assign_attributes'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/attribute_assignment.rb:23:in `_assign_attributes'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-5.2.3/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:28:in `perform_attribute_assignment'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/core.rb:315:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/inheritance.rb:66:in `new'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/inheritance.rb:66:in `new'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/geo_node_status.rb:107:in `fast_current_node_status'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/api/geo_nodes.rb:79:in `block in geo_node_status'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/api/geo_nodes.rb:78:in `geo_node_status'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/api/geo_nodes.rb:111:in `block (3 levels) in <class:GeoNodes>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `block (2 levels) in generate_api_method'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in `block in generate_api_method'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in `block in build_stack'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource.rb:20:in `_call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/abstract/handler.rb:17:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource.rb:20:in `_call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/abstract/handler.rb:17:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in `block in call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:121:in `process_route'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:74:in `block in identity'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:93:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:72:in `identity'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:57:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:137:in `with_optimization'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:56:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:119:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:45:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:40:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:57:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:17:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:103:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:16:in `call'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/jira/middleware.rb:17:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:13:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.3.0/lib/labkit/correlation/correlation_id.rb:18:in `use_id'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/batch-loader-1.4.0/lib/batch_loader/middleware.rb:11:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/apollo_upload_server-2.0.0.beta.3/lib/apollo_upload_server/middleware.rb:20:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:42:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:28:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:563:in `maintain_worker_count'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:293:in `join'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
/opt/gitlab/embedded/bin/unicorn:23:in `load'
/opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
So I'm not sure yet why, but the schema changes broke the grape endpoint. I wonder if other endpoints would be broken too, or if it depends on how we implemented the Geo status endpoint.