Secure - Metrics - SaaS - Create self describing JSON schema for Snowplow
Problem to solve
In order to send vulnerability tracking events for #229617, we need to create a self describing JSON schema to provide details of the given tracking event.
Proposal
-
Create a new self describing JSON schema in the iglu repository. This JSON schema should be generic so that it can be used for all secure products: { "data_sources": 3, "pipeline_id": 123, "merge_request_id": 29, "project_id": 465, "user_id": null, "scanner": { "type": "container_scanning", "name": "Clair", "version": "2.1.4" }, "branch_type": "feature", "vulnerabilities": { "diff": { "added": 100, "fixed": 50, "existing": 10 }, "severities": { "critical": 4, "high": 40, "medium": 23, "low": 0, "info": 0, "unknown": 0, "undefined": 0 } } }
-
Involve both secure (Category:DAST ~"Category:Dependency Scanning" ~"Category:License Compliance" Category:SAST "Category:Secret Detection") and telemetry ("Category:Metrics") team members to discuss and agree upon the schema design -
Manually test the schema using Gitlab::Tracking.self_describing_event Tested locally:
-
Good event:
SECURE_CONTEXT_SCHEMA_URL = 'iglu:com.gitlab/secure_context/jsonschema/1-0-0' data = { data_sources: 3, pipeline_id: 123, merge_request_id: 29, project_id: 465, user_id: nil, scanner: { type: "container_scanning", name: "Clair", version: "2.1.4" }, branch_type: "feature", vulnerabilities: { diff: { added: 100, fixed: 50, existing: 10 }, severities: { critical: 4, high: 40, medium: 23, low: 0, info: 0, unknown: 0, undefined: 0 } } }.compact include Gitlab::Tracking::ControllerConcern track_self_describing_event(SECURE_CONTEXT_SCHEMA_URL, data) [19] pry(main)> track_self_describing_event(SECURE_CONTEXT_SCHEMA_URL, data) ApplicationSetting Load (1.4ms) SELECT "application_settings".* FROM "application_settings" ORDER BY "application_settings"."id" DESC LIMIT $1 [["LIMIT", 1]] => #<SnowplowTracker::Tracker:0x00007fe486cc3630 @config={"encode_base64"=>true}, @emitters= [#<SnowplowTracker::AsyncEmitter:0x00007fe4a4ea6510 @all_processed_condition= #<MonitorMixin::ConditionVariable:0x00007fe4a4ea59d0 @cond=#<Thread::ConditionVariable:0x00007fe4a4ea59a8>, @monitor=#<Thread::Queue:0x00007fe4a4ea5ae8 @mon_count=0, @mon_mutex=#<Thread::Mutex:0x00007fe4a4ea5a20>, @mon_mutex_owner_object_id=70309998046580, @mon_owner=nil>>, @buffer=[], @buffer_size=1, @collector_uri="http://localhost:9090/i", @lock=#<Monitor:0x00007fe4a4ea5520 @mon_count=0, @mon_mutex=#<Thread::Mutex:0x00007fe4a4ea5480>, @mon_mutex_owner_object_id=70309998045840, @mon_owner=nil>, @method="get", @on_failure=nil, @on_success=nil, @queue=#<Thread::Queue:0x00007fe4a4ea5ae8 @mon_count=0, @mon_mutex=#<Thread::Mutex:0x00007fe4a4ea5a20>, @mon_mutex_owner_object_id=70309998046580, @mon_owner=nil>, @results_unprocessed=1>], @standard_nv_pairs={"tna"=>"gl", "tv"=>"rb-0.6.1", "aid"=>"gitlab"}, @subject=#<SnowplowTracker::Subject:0x00007fe4a4ea5200 @standard_nv_pairs={"p"=>"srv"}>>
http://localhost:9090/micro/all
{ total: 1, good: 1, bad: 0 }
http://localhost:9090/micro/good
[ { "event": { "api": { "vendor": "com.snowplowanalytics.snowplow", "version": "tp1" }, "parameters": { "e": "ue", "eid": "1996ef40-f8cb-4c35-a253-237cd38ee2b7", "aid": "gitlab", "tna": "gl", "stm": "1600048453835", "tv": "rb-0.6.1", "p": "srv", "dtm": "1600048453790", "ue_px": "eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5naXRsYWIvc2VjdXJlX2NvbnRleHQvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiZGF0YV9zb3VyY2VzIjozLCJwaXBlbGluZV9pZCI6MTIzLCJtZXJnZV9yZXF1ZXN0X2lkIjoyOSwicHJvamVjdF9pZCI6NDY1LCJzY2FubmVyIjp7InR5cGUiOiJjb250YWluZXJfc2Nhbm5pbmciLCJuYW1lIjoiQ2xhaXIiLCJ2ZXJzaW9uIjoiMi4xLjQifSwiYnJhbmNoX3R5cGUiOiJmZWF0dXJlIiwidnVsbmVyYWJpbGl0aWVzIjp7ImRpZmYiOnsiYWRkZWQiOjEwMCwiZml4ZWQiOjUwLCJleGlzdGluZyI6MTB9LCJzZXZlcml0aWVzIjp7ImNyaXRpY2FsIjo0LCJoaWdoIjo0MCwibWVkaXVtIjoyMywibG93IjowLCJpbmZvIjowLCJ1bmtub3duIjowLCJ1bmRlZmluZWQiOjB9fX19fQ==" }, "contentType": null, "source": { "name": "ssc-0.15.0-stdout$", "encoding": "UTF-8", "hostname": "localhost" }, "context": { "timestamp": "2020-09-14T01:54:13.856Z", "ipAddress": "172.17.0.1", "useragent": "Ruby", "refererUri": null, "headers": [ "Accept-Encoding: gzip, deflate;q=0.6, identity;q=0.3", "Accept: */*", "User-Agent: Ruby", "Connection: close", "Host: localhost:9090", "Timeout-Access: <function1>" ], "userId": "1c7a8ba1-d475-46d9-b49f-cc25dd02e926" } }, "eventType": "ue", "schema": "iglu:com.gitlab/secure_context/jsonschema/1-0-0", "contexts": null } ]
Decoded data:
[22] pry(main)> Base64::decode64("eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5naXRsYWIvc2VjdXJlX2NvbnRleHQvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiZGF0YV9zb3VyY2VzIjozLCJwaXBlbGluZV9pZCI6MTIzLCJtZXJnZV9yZXF1ZXN0X2lkIjoyOSwicHJvamVjdF9pZCI6NDY1LCJzY2FubmVyIjp7InR5cGUiOiJjb250YWluZXJfc2Nhbm5pbmciLCJuYW1lIjoiQ2xhaXIiLCJ2ZXJzaW9uIjoiMi4xLjQifSwiYnJhbmNoX3R5cGUiOiJmZWF0dXJlIiwidnVsbmVyYWJpbGl0aWVzIjp7ImRpZmYiOnsiYWRkZWQiOjEwMCwiZml4ZWQiOjUwLCJleGlzdGluZyI6MTB9LCJzZXZlcml0aWVzIjp7ImNyaXRpY2FsIjo0LCJoaWdoIjo0MCwibWVkaXVtIjoyMywibG93IjowLCJpbmZvIjowLCJ1bmtub3duIjowLCJ1bmRlZmluZWQiOjB9fX19fQ==") { "data": { "data": { "vulnerabilities": { "severities": { "undefined": 0, "unknown": 0, "info": 0, "low": 0, "medium": 23, "high": 40, "critical": 4 }, "diff": { "existing": 10, "fixed": 50, "added": 100 } }, "branch_type": "feature", "scanner": { "version": "2.1.4", "name": "Clair", "type": "container_scanning" }, "project_id": 465, "merge_request_id": 29, "pipeline_id": 123, "data_sources": 3 }, "schema": "iglu:com.gitlab/secure_context/jsonschema/1-0-0" }, "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0" }
-
Bad event:
SECURE_CONTEXT_SCHEMA_URL = 'iglu:com.gitlab/secure_context/jsonschema/1-0-0' data = { data_sources: 3, pipeline_id: 123, merge_request_id: 29, project_id: 465, user_id: nil, scanner: { type: "container_scanning", name: "Clair", version: "2.1.4" }, branch_type: "feature", vulnerabilities: { diff: { added: "100", fixed: 50, existing: 10 }, severities: { critical: 4, high: 40, medium: 23, low: 0, info: 0, unknown: 0, undefined: 0 } } }.compact include Gitlab::Tracking::ControllerConcern track_self_describing_event(SECURE_CONTEXT_SCHEMA_URL, data) [24] pry(main)> track_self_describing_event(SECURE_CONTEXT_SCHEMA_URL, data) ApplicationSetting Load (3.6ms) SELECT "application_settings".* FROM "application_settings" ORDER BY "application_settings"."id" DESC LIMIT $1 [["LIMIT", 1]] => #<SnowplowTracker::Tracker:0x00007fe486cc3630 @config={"encode_base64"=>true}, @emitters= [#<SnowplowTracker::AsyncEmitter:0x00007fe4a4ea6510 @all_processed_condition= #<MonitorMixin::ConditionVariable:0x00007fe4a4ea59d0 @cond=#<Thread::ConditionVariable:0x00007fe4a4ea59a8>, @monitor=#<Thread::Queue:0x00007fe4a4ea5ae8 @mon_count=0, @mon_mutex=#<Thread::Mutex:0x00007fe4a4ea5a20>, @mon_mutex_owner_object_id=70309998046580, @mon_owner=nil>>, @buffer=[], @buffer_size=1, @collector_uri="http://localhost:9090/i", @lock=#<Monitor:0x00007fe4a4ea5520 @mon_count=0, @mon_mutex=#<Thread::Mutex:0x00007fe4a4ea5480>, @mon_mutex_owner_object_id=70309998045840, @mon_owner=nil>, @method="get", @on_failure=nil, @on_success=nil, @queue=#<Thread::Queue:0x00007fe4a4ea5ae8 @mon_count=0, @mon_mutex=#<Thread::Mutex:0x00007fe4a4ea5a20>, @mon_mutex_owner_object_id=70309998046580, @mon_owner=nil>, @results_unprocessed=1>], @standard_nv_pairs={"tna"=>"gl", "tv"=>"rb-0.6.1", "aid"=>"gitlab"}, @subject=#<SnowplowTracker::Subject:0x00007fe4a4ea5200 @standard_nv_pairs={"p"=>"srv"}>>
http://localhost:9090/micro/all
{ total: 2, good: 1, bad: 1 }
http://localhost:9090/micro/bad
[ { "collectorPayload": { "api": { "vendor": "com.snowplowanalytics.snowplow", "version": "tp1" }, "querystring": [ "e=ue", "ue_px=eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5naXRsYWIvc2VjdXJlX2NvbnRleHQvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiZGF0YV9zb3VyY2VzIjozLCJwaXBlbGluZV9pZCI6MTIzLCJtZXJnZV9yZXF1ZXN0X2lkIjoyOSwicHJvamVjdF9pZCI6NDY1LCJzY2FubmVyIjp7InR5cGUiOiJjb250YWluZXJfc2Nhbm5pbmciLCJuYW1lIjoiQ2xhaXIiLCJ2ZXJzaW9uIjoiMi4xLjQifSwiYnJhbmNoX3R5cGUiOiJmZWF0dXJlIiwidnVsbmVyYWJpbGl0aWVzIjp7ImRpZmYiOnsiYWRkZWQiOiIxMDAiLCJmaXhlZCI6NTAsImV4aXN0aW5nIjoxMH0sInNldmVyaXRpZXMiOnsiY3JpdGljYWwiOjQsImhpZ2giOjQwLCJtZWRpdW0iOjIzLCJsb3ciOjAsImluZm8iOjAsInVua25vd24iOjAsInVuZGVmaW5lZCI6MH19fX19", "dtm=1600049215902", "p=srv", "tna=gl", "tv=rb-0.6.1", "aid=gitlab", "eid=d7c3ca32-22ce-49f0-abf5-94c77be95a60", "stm=1600049215948" ], "contentType": null, "body": null, "source": { "name": "ssc-0.15.0-stdout$", "encoding": "UTF-8", "hostname": "localhost" }, "context": { "timestamp": "2020-09-14T02:06:55.991Z", "ipAddress": "172.17.0.1", "useragent": "Ruby", "refererUri": null, "headers": [ "Accept-Encoding: gzip, deflate;q=0.6, identity;q=0.3", "Accept: */*", "User-Agent: Ruby", "Connection: close", "Host: localhost:9090", "Timeout-Access: <function1>" ], "userId": "8ade72a1-52c9-4479-958d-f5b42960bf35" } }, "errors": [ "An error occured while extracting the info about the event.", "Error while extracting the schema of the event: instance type (string) does not match any allowed primitive type (allowed: [\"integer\"])" ] } ]
-
Documentation
User-facing documentation not necessary
Availability & Testing
Add unit testing as required by the iglu project
What does success look like, and how can we measure that?
A secure tracking event can be sent and recorded in Snowplow
What is the type of buyer?
Is this a cross-stage feature?
Yes, this will affect Category:DAST ~"Category:Dependency Scanning" ~"Category:License Compliance" Category:SAST Category:Secret Detection