Fix avg_cycle_analytics giving an uncaught error

Fixes for projects = Project.sorted_by_activity.limit(10)

• Handles uncaught error
• Handles lack of index for

Issue #208788 (closed), Diagnosis https://gitlab.com/gitlab-org/telemetry/issues/299#note_297908450, Parent issue https://gitlab.com/gitlab-org/telemetry/issues/308

1. Catch errors

--- Gitlab::UsageData.to_json(force_refresh: true)
--- 'ea8bf810-1d6f-4a6a-b4fd-93e8cbd8b57f'
--SubmitUsagePingService.new.execute

[ gprd ] production> Time.now
=> 2020-02-28 14:07:55 +0000
[ gprd ] production> SubmitUsagePingService.new.execute

Time.now
unknown OID 28: failed to recognize type of 'relfrozenxid'. It will be treated as String.
unknown OID 1034: failed to recognize type of 'relacl'. It will be treated as String.
Traceback (most recent call last):
       16: from app/services/submit_usage_ping_service.rb:22:in `execute'
       15: from lib/gitlab/usage_data.rb:24:in `to_json'
       14: from lib/gitlab/usage_data.rb:10:in `data'
       13: from lib/gitlab/usage_data.rb:11:in `block in data'
       12: from ee/lib/ee/gitlab/usage_data.rb:26:in `uncached_data'
       11: from lib/gitlab/usage_data.rb:20:in `uncached_data'
       10: from lib/gitlab/usage_data.rb:124:in `cycle_analytics_usage_data'
        9: from lib/gitlab/cycle_analytics/usage_data.rb:19:in `to_json'
        8: from lib/gitlab/cycle_analytics/usage_data.rb:34:in `medians_per_stage'
        7: from lib/gitlab/cycle_analytics/usage_data.rb:34:in `each_with_object'
        6: from ee/lib/gitlab/database/load_balancing/connection_proxy.rb:41:in `select_all'
        5: from ee/lib/gitlab/database/load_balancing/connection_proxy.rb:62:in `read_using_load_balancer'
        4: from ee/lib/gitlab/database/load_balancing/load_balancer.rb:83:in `read_write'
        3: from ee/lib/gitlab/database/load_balancing/load_balancer.rb:134:in `retry_with_backoff'
        2: from ee/lib/gitlab/database/load_balancing/load_balancer.rb:84:in `block in read_write'
        1: from ee/lib/gitlab/database/load_balancing/connection_proxy.rb:63:in `block in read_using_load_balancer'
ActiveRecord::QueryCanceled (PG::QueryCanceled: ERROR:  canceling statement due to statement timeout)
[ gprd ] production>
[ gprd ] production> Time.now
=> 2020-02-28 22:26:16 +0000

2. Query Optimization

After

Discussed with the "manage analytics" stage group:

• We need to conform to original scope in this iteration scope :sorted_by_activity, -> { reorder(Arel.sql("GREATEST(COALESCE(last_activity_at, '1970-01-01'), COALESCE(last_repository_updated_at, '1970-01-01')) DESC")) }
• We can't add indexes due to immutable functions GREATEST
• can't use from_union due to order(10) not being respected inside the union
• nulls must be last and needed in some self-hosted instances

SELECT "projects".* FROM "projects" WHERE "projects"."last_activity_at" IS NOT NULL ORDER BY "projects"."last_activity_at" DESC LIMIT 10
https://explain.depesz.com/s/3pMv   => < 20ms cold 

SELECT "projects".* FROM "projects" WHERE "projects"."last_repository_updated_at" IS NOT NULL ORDER BY "projects"."last_repository_updated_at" DESC LIMIT 10
https://explain.depesz.com/s/3pMv  => < 20ms cold 

SELECT "projects".* FROM "projects" WHERE "projects"."last_activity_at" IS NULL AND "projects"."last_repository_updated_at" IS NULL LIMIT 10
https://explain.depesz.com/s/3pMv => < 20ms cold

Before

SELECT "projects".*
FROM "projects"
ORDER BY GREATEST(COALESCE(last_activity_at, '1970-01-01'), COALESCE(last_repository_updated_at, '1970-01-01')) DESC
LIMIT 10

Query plan: https://explain.depesz.com/s/1z14

Time: 1.525 min
  - planning: 0.700 ms
  - execution: 1.525 min
    - I/O read: 57.163 s
    - I/O write: 1.765 s

Shared buffers:
  - hits: 3 (~24.00 KiB) from the buffer pool
  - reads: 443629 (~3.40 GiB) from the OS file cache, including disk I/O
  - dirtied: 36442 (~284.70 MiB)
  - writes: 32347 (~252.70 MiB)