Bug with snowplow structured events
Problem
The se_va
field is supposed to be a float and is being set as a bool.
We're seeing events come in that fail to process with the following error:
"errors": [
{
"level": "error",
"message": "Field [se_va]: cannot convert [false] to Double-like String"
}
There's some additional data with the event and when it's base64 decoded I find data in it like this:
{
"e": "se",
"se_ca": "projects:new",
"se_ac": "activate_form_input",
"se_la": "blank_project",
"se_pr": "init_with_readme",
"se_va": "false"
...
}
The full error event is here in the collapsed "Details" section
{
"line": "CwBkAAAADzE2Mi4xNTYuMTQ0LjEzNgoAyAAAAWrq5OThCwDSAAAABVVURi04CwDcAAAAEnNzYy0wLjEyLjAta2luZXNpcwsBLAAAAHlNb3ppbGxhLzUuMCAoTWFjaW50b3NoOyBJbnRlbCBNYWMgT1MgWCAxMF8xNF81KSBBcHBsZVdlYktpdC81MzcuMzYgKEtIVE1MLCBsaWtlIEdlY2tvKSBDaHJvbWUvNzQuMC4zNzI5LjE1NyBTYWZhcmkvNTM3LjM2CwE2AAAAE2h0dHBzOi8vZ2l0bGFiLmNvbS8LAUAAAAAjL2NvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy90cDILAVQAAAQVeyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9wYXlsb2FkX2RhdGEvanNvbnNjaGVtYS8xLTAtNCIsImRhdGEiOlt7ImUiOiJzZSIsInNlX2NhIjoicHJvamVjdHM6bmV3Iiwic2VfYWMiOiJhY3RpdmF0ZV9mb3JtX2lucHV0Iiwic2VfbGEiOiJibGFua19wcm9qZWN0Iiwic2VfcHIiOiJpbml0X3dpdGhfcmVhZG1lIiwic2VfdmEiOiJmYWxzZSIsInR2IjoianMtMi4xMC4wIiwidG5hIjoiY2YiLCJhaWQiOiJnaXRsYWIiLCJwIjoid2ViIiwidHoiOiJBbWVyaWNhL0xvc19BbmdlbGVzIiwibGFuZyI6ImVuLVVTIiwiY3MiOiJVVEYtOCIsImZfcGRmIjoiMSIsImZfcXQiOiIwIiwiZl9yZWFscCI6IjAiLCJmX3dtYSI6IjAiLCJmX2RpciI6IjAiLCJmX2ZsYSI6IjAiLCJmX2phdmEiOiIwIiwiZl9nZWFycyI6IjAiLCJmX2FnIjoiMCIsInJlcyI6IjE0NDB4OTAwIiwiY2QiOiIyNCIsImVpZCI6IjIyYzNmY2Y0LTEwYmItNGQwMi1iZGRjLTExMTM0YTBmYWVlOSIsImR0bSI6IjE1NTg3MTkwMzA0NTciLCJjeCI6ImV5SnpZMmhsYldFaU9pSnBaMngxT21OdmJTNXpibTkzY0d4dmQyRnVZV3g1ZEdsamN5NXpibTkzY0d4dmR5OWpiMjUwWlhoMGN5OXFjMjl1YzJOb1pXMWhMekV0TUMwd0lpd2laR0YwWVNJNlczc2ljMk5vWlcxaElqb2lhV2RzZFRwamIyMHVjMjV2ZDNCc2IzZGhibUZzZVhScFkzTXVjMjV2ZDNCc2IzY3ZkMlZpWDNCaFoyVXZhbk52Ym5OamFHVnRZUzh4TFRBdE1DSXNJbVJoZEdFaU9uc2lhV1FpT2lJNFl6aGhaVFppT1MwMlpETTVMVFEwT0dJdE9UZ3pPUzAzWVRVek5EVmxNakkyTmpJaWZYMWRmUSIsInZwIjoiMTI1Nng2NDQiLCJkcyI6IjEyNTZ4ODQ4IiwidmlkIjoiMSIsInNpZCI6IjFlYmQ2MjczLTE1MzUtNGE2ZS1iZDc1LTMxZTZlZmE5NTU5OSIsImR1aWQiOiJhOWQ3NzZhOC0xYTdkLTRjODItYjYxOC1mYWQwNTYxMjBlYmYiLCJyZWZyIjoiaHR0cHM6Ly9naXRsYWIuY29tLyIsInVybCI6Imh0dHBzOi8vZ2l0bGFiLmNvbS9wcm9qZWN0cy9uZXciLCJzdG0iOiIxNTU4NzE5MDMwNDU4In1dfQ8BXgsAAAANAAAAHUhvc3Q6IHNub3dwbG93LnRyeC5naXRsYWIubmV0AAAAC0FjY2VwdDogKi8qAAAAIkFjY2VwdC1FbmNvZGluZzogZ3ppcCwgZGVmbGF0ZSwgYnIAAAAgQWNjZXB0LUxhbmd1YWdlOiBlbi1VUywgZW47cT0wLjkAAAAaT3JpZ2luOiBodHRwczovL2dpdGxhYi5jb20AAAAcUmVmZXJlcjogaHR0cHM6Ly9naXRsYWIuY29tLwAAAIVVc2VyLUFnZW50OiBNb3ppbGxhLzUuMCAoTWFjaW50b3NoOyBJbnRlbCBNYWMgT1MgWCAxMF8xNF81KSBBcHBsZVdlYktpdC81MzcuMzYgKEtIVE1MLCBsaWtlIEdlY2tvKSBDaHJvbWUvNzQuMC4zNzI5LjE1NyBTYWZhcmkvNTM3LjM2AAAAIFgtRm9yd2FyZGVkLUZvcjogMTYyLjE1Ni4xNDQuMTM2AAAAFVgtRm9yd2FyZGVkLVBvcnQ6IDQ0MwAAABhYLUZvcndhcmRlZC1Qcm90bzogaHR0cHMAAAAWQ29ubmVjdGlvbjoga2VlcC1hbGl2ZQAAABtUaW1lb3V0LUFjY2VzczogPGZ1bmN0aW9uMT4AAAAQYXBwbGljYXRpb24vanNvbgsBaAAAABBhcHBsaWNhdGlvbi9qc29uCwGQAAAAF3Nub3dwbG93LnRyeC5naXRsYWIubmV0CwGaAAAAJDgxMmZkOTkwLTY5YTktNDZhYy05MjA5LTE2ZTI5NzY4MzNmYQt6aQAAAEFpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9Db2xsZWN0b3JQYXlsb2FkL3RocmlmdC8xLTAtMAA=",
"errors": [
{
"level": "error",
"message": "Field [se_va]: cannot convert [false] to Double-like String"
}
],
"failure_tstamp": "2019-05-24T17:30:31.494Z"
}
If you base64 decode (https://www.base64decode.org/) the data in "Line" you'll see:
�d���162.156.144.136
���j����UTF-8����ssc-0.12.0-kinesis,���yMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.366���https://gitlab.com/@���#/com.snowplowanalytics.snowplow/tp2T��{"schema":"iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4","data":[{"e":"se","se_ca":"projects:new","se_ac":"activate_form_input","se_la":"blank_project","se_pr":"init_with_readme","se_va":"false","tv":"js-2.10.0","tna":"cf","aid":"gitlab","p":"web","tz":"America/Los_Angeles","lang":"en-US","cs":"UTF-8","f_pdf":"1","f_qt":"0","f_realp":"0","f_wma":"0","f_dir":"0","f_fla":"0","f_java":"0","f_gears":"0","f_ag":"0","res":"1440x900","cd":"24","eid":"22c3fcf4-10bb-4d02-bddc-11134a0faee9","dtm":"1558719030457","cx":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jb250ZXh0cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvd2ViX3BhZ2UvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiaWQiOiI4YzhhZTZiOS02ZDM5LTQ0OGItOTgzOS03YTUzNDVlMjI2NjIifX1dfQ","vp":"1256x644","ds":"1256x848","vid":"1","sid":"1ebd6273-1535-4a6e-bd75-31e6efa95599","duid":"a9d776a8-1a7d-4c82-b618-fad056120ebf","refr":"https://gitlab.com/","url":"https://gitlab.com/projects/new","stm":"1558719030458"}]}^���
���Host: snowplow.trx.gitlab.net���Accept: */*���"Accept-Encoding: gzip, deflate, br��� Accept-Language: en-US, en;q=0.9���Origin: https://gitlab.com���Referer: https://gitlab.com/���User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36��� X-Forwarded-For: 162.156.144.136���X-Forwarded-Port: 443���X-Forwarded-Proto: https���Connection: keep-alive���Timeout-Access: ���application/jsonh���application/json���snowplow.trx.gitlab.net���$812fd990-69a9-46ac-9209-16e2976833fazi���Aiglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0�
Which shows that at least two schemas apply - the CollectorPayload one which is Thrift encoded https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0 and the JSON Schema one https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4#L223-L224 which seems to indicate that se_va
can be a string.
I'm able to recreate this event by creating a new project via https://gitlab.com/projects/new?nav_source=navbar with the "Initialize repository with a README" flag set.
The event model here https://github.com/snowplow/snowplow/wiki/Canonical-event-model#235-custom-structured-events shows that this field must be a decimal, not a bool or a string.
Proposal
The structured event value should just be a decimal number. I'm guessing the error is https://gitlab.com/gitlab-org/gitlab-ee/blob/24af74b7635929c62ec19ac75a2f0d93d21081e9/app/views/projects/project_templates/_built_in_templates.html.haml#L18 where template.name
isn't guaranteed to be a decimal?
Result
Next steps (if any)
@donaldcook for review and next steps
FYI @tipyn