Make including values and metadata opt-out
With these changes, the Spotter CLI will include values and metadata by default. An opt-out mechanism is introduced with --exclude-values and exclude-metadata.
The motivation for the changes is the desire to give users the most of Spotter right away when they start using it. Values from playbooks (e.g., parameter values from Ansible modules, variables from Ansible plays, etc.) and metadata (i.e., file names, line, and column numbers, YAML markers) are needed for Spotter to produce additional tips for improving Ansible content and for an enriched user experience in the Spotter App. The second reason is that there is no real need for opt-in behavior. Since the CLI already discovers most of the secrets (e.g., passwords, SSH keys, cloud credentials, etc.) there are no worries not to send these values to the backend (still, for enterprises that want to self-control data flow we support the on-prem version of Spotter). Users that are not using the --include- flags mostly do this because they don't know about them.
Since we and the users want to get most of Spotter just with spotter scan ., we made including values and metadata opt-out with the new CLI flags that would signal the CLI to omit collecting and uploading any values or metadata. We also deprecated --include-values and --include-metadata flags and remove their usage in integration tests. We added a new integration test for excluding values and metadata.
Click to expand for a demonstration of the changes
Let's have this playbook:
- name: OpenSSL Example
hosts: localhost
tasks:
- name: Ensure that the server certificate belongs to the specified private key
openssl_certificate:
path: "{{ config_path }}/certificates/server.crt"
privatekey_path: "{{ config_path }}/certificates/server.key"
provider: assertonlyAnd now see that scanning with --exclude-values gives us one check result less.
(.venv) user@ubuntu:~/spotter-cli$ spotter scan playbook.yml
playbook.yml:4:7: ERROR: [E903] Use a fully-qualified name, such as community.crypto.x509_certificate instead of openssl_certificate. View docs at https://docs.steampunk.si/plugins/community/crypto/latest/module/x509_certificate.html.
playbook.yml:4:7: ERROR: [E601] Value of provider in module openssl_certificate is set to assertonly which does not match any of the possible choices listed in [acme, entrust, ownca, selfsigned]. View docs at https://docs.steampunk.si/plugins/community/crypto/2.14.1/module/x509_certificate.html.
------------------------------------------------------------------------
Spotter took 0.355 s to scan your input.
It resulted in 2 error(s), 0 warning(s) and 0 hint(s).
Overall status: ERROR
(.venv) user@ubuntu:~/spotter-cli$ spotter scan --exclude-values playbook.yml
playbook.yml:4:7: ERROR: [E903] Use a fully-qualified name, such as community.crypto.x509_certificate instead of openssl_certificate. View docs at https://docs.steampunk.si/plugins/community/crypto/latest/module/x509_certificate.html.If we export the scan payload with spotter scan playbook.yml -e payload.json we get the payload with all the playbook values and the metadata.
{
"environment": {
"python_version": "3.10.12",
"ansible_version": {
"ansible_core": "2.15.2",
"ansible_base": null,
"ansible": "[core 2.15.2]"
},
"installed_collections": [],
"ansible_config": {
"ANSIBLE_FORCE_COLOR(env: ANSIBLE_FORCE_COLOR)": "False",
"CONFIG_FILE()": "None"
},
"galaxy_yml": {},
"collection_requirements": {
"collections": [
{
"name": "community.crypto",
"version": "2.14.1"
}
]
},
"cli_scan_args": {
"parse_values": true,
"include_values": true,
"include_metadata": true,
"rewrite": false,
"display_level": "hint",
"profile": "default",
"skip_checks": [],
"enforce_checks": []
}
},
"tasks": [
{
"task_id": "1483b62b-21d6-4e52-b9e4-dc41234588bb",
"task_args": {
"name": "Ensure that the server certificate belongs to the specified private key",
"openssl_certificate": {
"path": "{{ config_path }}/certificates/server.crt",
"privatekey_path": "{{ config_path }}/certificates/server.key",
"provider": "assertonly"
}
},
"spotter_metadata": {
"file": "/home/anzeluzar/Desktop/spotter-cli/playbook.yml",
"line": 4,
"column": 7,
"start_mark_index": 58,
"end_mark_index": 319
},
"spotter_obfuscated": [],
"spotter_noqa": []
}
],
"playbooks": [
{
"playbook_id": "6aefcb9c-9d69-450c-aceb-1ecd6fb78a47",
"plays": [
{
"play_id": "6afab5b4-8b10-4ed3-8a9d-1769b03952e5",
"play_args": {
"name": "OpenSSL Example",
"hosts": "localhost"
},
"spotter_metadata": {
"file": "/home/anzeluzar/Desktop/spotter-cli/playbook.yml",
"line": 1,
"column": 3,
"start_mark_index": 2,
"end_mark_index": 319
},
"spotter_obfuscated": []
}
]
}
]
}And if we export with spotter scan --exclude-values --exclude-metadata playbook.yml -e payload.json we get the JSON payload without values and metadata.
{
"environment": {
"python_version": "3.10.12",
"ansible_version": {
"ansible_core": "2.15.2",
"ansible_base": null,
"ansible": "[core 2.15.2]"
},
"installed_collections": [],
"ansible_config": {
"ANSIBLE_FORCE_COLOR(env: ANSIBLE_FORCE_COLOR)": "False",
"CONFIG_FILE()": "None"
},
"galaxy_yml": {},
"collection_requirements": {
"collections": [
{
"name": "community.crypto",
"version": "2.14.1"
}
]
},
"cli_scan_args": {
"parse_values": false,
"include_values": false,
"include_metadata": false,
"rewrite": false,
"display_level": "hint",
"profile": "default",
"skip_checks": [],
"enforce_checks": []
}
},
"tasks": [
{
"task_id": "dbeacf0e-7a66-4ec1-85cd-8e5469a0a875",
"task_args": {
"name": null,
"openssl_certificate": {
"path": null,
"privatekey_path": null,
"provider": null
}
},
"spotter_noqa": []
}
],
"playbooks": [
{
"playbook_id": "f0999724-ef7a-4e6c-9b8f-ecd66a8ba4da",
"plays": [
{
"play_id": "a4071296-c63e-4912-bd45-d426aa1bcb89",
"play_args": {
"name": null,
"hosts": null
}
}
]
}
]
}Implements #306 (closed) and #307 (closed).