Incubation:APM Test environment issues sessions expiring

Seems like sessions start to expire after some time in the test environment, with some series POST requests returning a 403.

A particular host is also sending only a minimal number of series data whilst others remain on par with each other and send the expected amount.

Seems likely this is something to do with the intake request and the hostname value being changed over time.

Count of metrics per host in a 5 minute window:

┌─host──────────────────────────────────────────────────────────────┬─count(host)─┐
│ gke-apm-test-clickhouse-75f1cc4a-v5v7.c.jshaw-8349b024.internal   │       17440 │
│ gke-apm-test-clickhouse-75f1cc4a-dmjs.c.jshaw-8349b024.internal   │       17917 │
│ gke-apm-test-default-pool-b81efa52-kqbd.c.jshaw-8349b024.internal │          38 │
│ gke-apm-test-default-pool-b81efa52-v4sz.c.jshaw-8349b024.internal │       20350 │
└───────────────────────────────────────────────────────────────────┴─────────────┘

Various "get session" errors are visible in the gateway logs for certain requests. It would appear invalid session hashes are being generated for certain series intake requests:

{"error":"session 3ae6f5eecf94de51124b692a134675e317c82969313ec13cb27774848219a40c project ID is nil","level":"error","msg":"get session","time":"2021-12-07T14:38:44Z"}                                                                                                               │
│ {"error":"session 3ae6f5eecf94de51124b692a134675e317c82969313ec13cb27774848219a40c project ID is nil","level":"error","msg":"get session","time":"2021-12-07T14:38:59Z"}                                                                                                               │
│ {"error":"session db303736231d27ba1c3821cc1fc04786a42c5b0f742935adc44e425513756282 project ID is nil","level":"error","msg":"get session","time":"2021-12-07T14:39:14Z"}                                                                                                               │
│ {"error":"session ab413c78a1b74392b24ca959ad8fd71c2f74c159426eb44346854daa5a14e1a4 project ID is nil","level":"error","msg":"get session","time":"2021-12-07T14:39:29Z"}                                                                                                               │
│ {"error":"session ab413c78a1b74392b24ca959ad8fd71c2f74c159426eb44346854daa5a14e1a4 project ID is nil","level":"error","msg":"get session","time":"2021-12-07T14:39:44Z"}                                                                                                               │
│ {"error":"session db303736231d27ba1c3821cc1fc04786a42c5b0f742935adc44e425513756282 project ID is nil","level":"error","msg":"get session","time":"2021-12-07T14:39:59Z"}                                                                                                               │
│ {"error":"session db303736231d27ba1c3821cc1fc04786a42c5b0f742935adc44e425513756282 project ID is nil","level":"error","msg":"get session","time":"2021-12-07T14:40:14Z"}                                                                                                               │
│ {"error":"session d4e15c09fab9ddbf193aa1df8b39cce8cc28b1d61220c0380d58303bb2eceb28 project ID is nil","level":"error","msg":"get session","time":"2021-12-07T14:40:29Z"}       

Investigation shows this is directly related to collecting kubernetes_state metrics, e.g.:

{Host:gke-apm-test-default-pool-b81efa52-kqbd-apm-test Interval:0 Metric:kubernetes_state.pod.scheduled Points:[{Time:1638912946 Value:1}] Tags:[condition:true kube_namespace:gateway namespace:gateway node:gke-apm-test-default-pool-b81efa52-kqbd phase:running pod:gitlab-apm-test-kube-state-metrics-6fcb84c476-kd2vz pod_name:gitlab-apm-test-kube-state-metrics-6fcb84c476-kd2vz pod_phase:running] Type:gauge}

I'm not sure what these host names relate to, but all kubernetes_state metrics use this host name format instead of the expected format (in this case gke-apm-test-default-pool-b81efa52-kqbd.c.jshaw-8349b024.internal).