### Problem Postman GUI allows users to export a given environment to a JSON file. The Postman environment file contains a list of well-known variables defined by the user. On the other hand, a user of API security (DAST API or API Fuzzing) can provide values for the variables by using a JSON document, which is not compatible with the Postman environment file. The problem is that the customer has to translate the Postman environment file to a JSON document compatible with API Security (DAST API or API Fuzzing), or manually provide the JSON file format when with a few clicks they can get an updated file from the Postman GUI. ### Possible solution Allow API Fuzzing or DAST API to consume the Postman environment file to provide variable values. ### Remarks - Postman GUI allows the user to export different files, all of them may contain variables. - [`Postman data`](https://learning.postman.com/docs/getting-started/importing-and-exporting-data/#exporting-postman-data) file, - [`Postman environment`](https://learning.postman.com/docs/getting-started/importing-and-exporting-data/#exporting-environments) file, this file can be created by selecting `Environments`, in the left tab, then selecting a previously created environment. Last, in the right side use the dots menu (`...`) and select `Export`. Notice that there is also a special environment called: `Global` is a fixed environment, which [can be exported](https://learning.postman.com/docs/sending-requests/variables/#downloading-global-environments) in a separate file. - [`Postman data dumps`](https://learning.postman.com/docs/getting-started/importing-and-exporting-data/#exporting-data-dumps) - Exported Postman file may not contain values for all variables: - For example, [Dynamic variables](https://learning.postman.com/docs/writing-scripts/script-references/variables-list/) are not provided in exported environments. - Another case is when a [variable is set programmatically](https://learning.postman.com/docs/sending-requests/variables/#defining-variables-in-scripts). The GUI or runners (e.g. Newman) creates the variable during collection execution. - [Variables have different scopes](https://learning.postman.com/docs/sending-requests/variables/#variable-scopes), and some of the variables are set as global (exported only in workspace) while other are exported in the environment. - Data variables are not included in the `Postman environment` file. - There is not much documentation about formats other than [`Postman data Schema`](https://schema.postman.com/) ### Proposal 1. [x] ~~Enable FeatureFlag~~ 1. [x] ~~Add FeatureFlag service~~ 1. [x] ~~Add FlagName to this feature~~ 1. [X] **Add support to new file format `postman environmental file`.** 1. [X] Detect and load postman environmental file. - a postman environmental file, where the JSON document is an object featuring properties: - `_postman_variable_scope` set to `environment` or `global`. (Required) - `values` are set to an array. Each entry of the array is a JSON object, with at least the following properties: `key`, `value`, `type`, and `enabled`. 1. [x] Update/Add tests 1. [x] Add unit test using postman environmental file. 1. [x] **Support multiple files in `_POSTMAN_COLLECTION_VARIABLES`** 1. [x] Allow accepting multiple files in `_POSTMAN_COLLECTION_VARIABLES`, file paths are comma-separated. Each file in `_POSTMAN_COLLECTION_VARIABLES` can be: - a postman environmental file - a flat JSON object, using key-value pairs. (This is the current implementation) 1. [x] Update runner options 1. [x] to accept a list of files. 1. [x] to validate file is accessible 1. [x] to validate file format is correct (file can be either postman environmental file or flat JSON object) 1. [x] Update postman reader to detect and load postman environmental files, otherwise treat file a flat JSON object. 1. [x] Update documentation in API Fuzzing and DAST API to: 1. [x] mention that `_POSTMAN_COLLECTION_VARIABLES` supports multiple files. 1. [x] mention that a given entry in `_POSTMAN_COLLECTION_VARIABLES` can be either a postman environmental file or a JSON flat object. 1. [x] mention that environmental files may or may not provide all variables, and environmental variable `_POSTMAN_COLLECTION_VARIABLES_GENERATION` may produce values for non-user-defined variables. 1. [x] ~~Remove the warning about using only flat JSON objects.~~ Add note that after 15.0 there is support for Postman Environment format. 1. [x] Add tests 1. [x] Add/Update Unit test providing none, a single, and multiple files. Mixing up two formats. 1. [x] Add unit test for new runner option validations: files exist, the file does not exist, the file has valid content, file has invalid content (empty, non-valid JSON, incomplete JSON). 1. [x] **Add support to scope variables** 1. [x] Load and unload [variables from each scope](https://learning.postman.com/docs/sending-requests/variables/#variable-scopes) 1. [x] `Global`: Values can be set thru GUI in environment tab, and they cab be **Loaded** from files provided in `_POSTMAN_COLLECTION_VARIABLES`. The file must be a JSON document with a JSON object with a property `_postman_variable_scope` set to `globals` 1. [x] `Collection`: Values can be set thru GUI in Collection details, and they cab be **Loaded** from the file provided in `_POSTMAN_COLLECTION`. Variables are taken from array node at `$.variable`. 1. [x] `Environment`: Values can be set thru GUI in environment tab, and they cab be **Loaded** from files provided in `_POSTMAN_COLLECTION_VARIABLES`. The file must be a JSON document with a JSON object with a property `_postman_variable_scope` set to `environment` 1. [x] `Json`: Values loaded from KeyValur Pairs from JSON document. **NOTE** This is not a postman scope, but it is defined to support the old variable file format. 1. [-] ~~`Data`: This will not be implemented.~~ 1. [-] ~~`Local`: Value seem to be set only in scripts in GUI. [Postman schema](https://schema.postman.com/collection/json/v2.1.0/draft-07/docs/index.html) defines variables for folders and items (requests). It seems UI does not provide a mechanism to edit them, and documentation does not specify their scope either. **TDB** Since they are in a hierarchy structure, they can be treat as `Local` and loaded and unloaded based on the hierarchy.~~ 1. [-] ~~**Load** from the file provided in `_POSTMAN_COLLECTION`. Variables are taken from array node at `$.item[*].variable`.~~ 1. [-] ~~From folder(s)~~ 1. [-] ~~From item(s)~~ 1. [-] ~~**Unload** after the request has been created.~~ 1. [x] Provide a mechanism to resolve variable by name. Resolution should respect hierarchy: `Global`<`Collection`<`Environment`<`Json`<`Data`<`Local`. For example, if a given variable is defined in `Global` and `Local`, `Local` value should be used. 1. [x] Resolution should unfold variables until no matches are found. `globalBaseUrl={{globalScheme}}://{{globalHost}}{{globalBasePath}}` thus in order to get value for `globalBaseUrl`, the value for `globalScheme`, `globalHost` and `globalBasePath` should also be resolved. 1. [x] Prevent circular reference when unfolding variables 1. [x] Add tests 1. [x] Check hierarchy resolution 1. [x] Check variable unfolding 1. [x] Check circular reference prevention, e.g.: `var1="{{var2}}"` and `var2={{var1}}` in `http://foo?q={{var1}}` 1. [x] Check indirect variable reference, e.g.: `var1="var2"` and `var2={{var1}}` in `http://foo?q={{{{var1}}}}` 1. [x] **Introduce a new environmental variable `_POSTMAN_COLLECTION_VARIABLES_GENERATION`.** Instruct the request generator how to deal with undefined variables. There are three possible behaviors, represented by an enum: - Enum value - `NONE`: Do not remove text of the unknown variable. Keep it as it is. - `DYNAMIC`: Replace only [dynamic variables](https://learning.postman.com/docs/writing-scripts/script-references/variables-list/), other unknown variables will not be replaced. - `ALL`: Replace any unknown variable with a text value. - Default to: `DYNAMIC` (in order to mimic postman runner behavior) - A unknown variable may fall into one of two categories: - `Dynamic variable`: Any variable prefixed with `$`. For these variables, a sample value can be formatted based on the [list of postman dynamic variables](https://learning.postman.com/docs/writing-scripts/script-references/variables-list/). For instance `$guid` can return `611c2e81-2ccb-42d8-9ddc-2d0bfa65c1b4` instead of a generic fixed text. - `Any other unknown variables`: Since there is no hint about their expected type or format, we can set a fixed text such as: `unknown-value` or a formatted text such as: `unset-var-<varname>`. - The scanner may produce a warning log when variables for all unknown variables. This is because unknown variables can be part of any part of the request, and could lead to producing invalid content for the request generation. 1. [x] Update `worker-entry` to accept new environmental variable 1. [x] Add validation to accept only supported values 1. [x] Update runner options 1. [x] Update postman reader to check `_POSTMAN_COLLECTION_VARIABLES_GENERATION` and generate values for: 1. [x] Dynamic variables (on `DYNAMIC` or `ALL`) 1. [x] Any other unknown variables (only when `ALL`) 1. In the scanner, log a list of the auto generated variables: 1. [x] Collect name of the unknown variables while generating requests 1. [x] After generating request log the unknown variables . 1. [x] Add test 1. [x] Add unit test with and without variable set. 1. [x] Add test to check generated values for `dynamic variables` 1. [x] Add documentation for `_POSTMAN_COLLECTION_VARIABLES_GENERATION` in API Fuzzing and DAST API 1. [x] Explain when it is expected to be used. 1. [x] Point out that generated value may not fit the expected value, and suggest providing environmental files. 1. [x] ~~Remove feature flag~~