*Created by: svoop* :warning: See related issue ofmx#1 as well. --- This draft describes how contributors exchange data by use of OFMX documents as discussed with @philhug. ## Terms * **importer** – External source of OFMX (e.g. the "aip2aixm" tool) * **editor** – The OFM Editor * **database** – The OFM database * **consumer** – External consumer of OFMX * **core feature** – See [OFMX wiki](/openflightmaps/ofmx/wikis/Features). Core features in this draft include all of their auxiliary features as well. ## Import An external data source is translated to OFMX by an importer and then uploaded to the editor. The editor aggregates and validates the data which is then written to the database. Workflow: 1. OFMX is imported to the editor and translated into a DOM. 2. All `@mid` attributes are *removed* from the DOM. 3. `@mid` attributes are generated on every `*Uid` element. 4. The editor diffs `@mid` lists to detect additions and deletions. 5. A core feature is... &emsp;– *not* flagged as protected: The editor removes all deletions from the database. &emsp;– flagged as protected: The editor logs all skipped deletions. 6. The editor generates a payload digest of every core feature and builds an ordered map. 7. The editor diffs the payload hash digest maps to detect modifications. 8. A core feature is... &emsp;– *not* flagged as protected: The editor updates all modifications on the database. &emsp;– flagged as protected: The editor logs all skipped modifications. 9. The editor displays the log to the importer. ## Export The editor reads the data from the database, assembles it into an OFMX file, validates it and delivers it to the consumer. Workflow: 1. The editor reads the data from the database and assembles it to OFMX. 2. The `@mid` remain in the markup (to facilitate consuming the data). It is important to inform the consumers that `@mid` may only be used to match features contained in *this* OFMX document since `@mid` may change across different exports as the data and therefore `*Uid` evolve. 3. The OFMX is validated and on... &emsp;– pass: The OFMX is delivered. &emsp;– fail: An error is logged, displayed and an admin is notified. ## Moving Parts ### Namespaces Every OFMX document contains a UUID in `OFMX-Snapshot@namespace`. The database stores the data in such a fashion as to guarantee, imports to one namespace cannot alter the data from another namespace in any way. The first upload with a new UUID creates the corresponding namespace. All subsequent imports have access to the data stored within this namespace only. ### `*Uid` and `@mid` OFMX identifies and associates features by use of `*Uid` tags. To facilitate the matching of `*Uid`, the editor generates `@mid` when the OFMX is imported: The editor reads the OFMX into a DOM. In a first pass, it removes all `@mid` attributes already present anywhere in the DOM. In a second pass, it generates `@mid` attributes for every `*Uid` element by hashing the `*Uid` payload into a UUID. From this point forward, the editor can associate features easily with `*Uid@mid`. The `@mid` remain in the OFMX document. :warning: `*Uid` elements are usually six characters long (e.g. `AhpUid`), however, there is one exception: `AseUidSameExtent` found as part of [<span>`Adg`</span>](/openflightmaps/ofmx/wikis/Airspace#adg-derived-geometry). To find all `*Uid` element nodes, the DOM therefore has to be scanned e.g. with `/^[A-Z][a-z]{2}Uid.*$/`. ### Payload Digest The editor builds a digest over the *entire* payload of a core feature (including its auxiliary features) by use of a hash function such as MD5. Payload digests are written to the database, but are not part of the OFMX document! :warning: Payload digests may be an overkill if it's possible to otherwise reliably diff the complete payload of a core feature. ### Detection of Additions and Deletions Additions and deletions are detected by diffing the `*Uid@mid` of all core features present in the database versus present in the import. Build two ordered lists of all core feature `@mid` e.g. as "feature#mid": ``` Ase#19027994-d89d-475b-989d-f137be2ed6f5 Ase#54ca8ef9-0dbe-4cf1-9a09-a9f28c2e6a13 Org#8f570e02-4052-43e9-9123-8a64c3a7c141 (...) ``` Diffing the two lists disclose additions and deletions. ### Detection of Modifications Detection of modifications are detected by diffing the payload digests of all core features present in the database versus present in the import. Build two ordered maps of all payload digests to core features e.g. as "digest -> feature#mid": ``` 814df75121ec307a61dd05994016352e -> Ase#19027994-d89d-475b-989d-f137be2ed6f5 86c19cac55ef2e2a4d953504d45d1d4c -> Org#8f570e02-4052-43e9-9123-8a64c3a7c141 e152476ea80f6fe49dcdb8da9c6b582c -> Ase#54ca8ef9-0dbe-4cf1-9a09-a9f28c2e6a13 (...) ``` Diffing the two lists of map keys disclose modifications and point to the modified core feature. ### Protection of Manual Edits Whenever a core feature is manually edited with the editor (e.g. fixing typos in the remarks, fixing geometry etc), a protection flag is set on the database. When this flag is set, the core feature must not be overwritten by subsequent imports anymore. A message should be logged instead instructing the uploader that this particular core feature has to be modified manually. However, protected core features may be deleted if they no longer appear in a subsequent import. ## Edge Cases Some core features have to be dealt with manually at all times. Say, a TMA airspace of an airport in region A extends into another region B. This TMA may present in only one or both relevant AIP. Or it's split in two parts present on the two relevant AIP. OFM defines rules as to which region takes ownership over such edge cases which is why they usually have to be entered manually by use of the editor. The importer must assure not to include edge cases such as a partial TMA in the above example.