ydb_env_set and ydb_env_unset handle an expanded set of "out of the box" use cases

Final Release Note

Under the directory specified by $ydb_dir (defaulting to $HOME/.yottadb), the ydb_env_set file when sourced:

• Ensures a standard environment for YottaDB, creating one should an environment not exist.
• Creates new database files for any regions that do not have database files, under the assumption that missing database files correspond to temporary regions with scratch globals.
• If the database was not shut down cleanly (e.g., when a system is rebooted after a crash), recovers the database using MUPIP JOURNAL RECOVER BACKWARD if the database does not have replication turned on and MUPIP JOURNAL ROLLBACK BACKWARD if replication is turned on. All database regions to be recovered must have before image journaling enabled and on at the time of the crash.
• Sets reasonable values for ydb_* environment variables and their gtm* counterparts (the latter so that application code and scripts that query the latter continue to work correctly).
• Defines environment variablesydb_xc_* / GTMXC_* for $ydb_dist/plugin/*.xc files with M to C call-out tables.
• Adds $ydb_dist to $PATH, as well as $ydb_dist/plugin/bin if it exists.
• Sourcing ydb_env_set saves environment variables that it sets so that sourcing ydb_env_unset subsequently restores those variables to their prior values.

Previously:

• Sourcing ydb_env_set created an environment if one did not exist, and provided values to environment variables that were not set, letting the ydb script perform any needed recovery. However, that only served those using YottaDB using the M language, since those using YottaDB using other Supported languages would not use the ydb script.
• Environment variables had to be explicitly added for M-to-C call-out tables in $ydb_dist/plugin/*.xc files.
• Sourcing ydb_env_set created aliases for mupip and mumps (gde continues to be an alias).

While the behavior of the combination of ydb_env_set and ydb_env_unset is not upward compatible (for example, it sets ydb_routines and gtmroutines to the environment under $ydb_dir where previously it made consistent existing values of $ydb_routines and $gtmroutines), they are intended to be upward compatible for common "out of the box" usage; with the expectation that more complex scenarios would use application-specific scripting. In addition to improved support for non-M users, they handle a larger number of situations that previously would have required application-specific scripting.

[#429 (closed)]

Description

In the event a system is crashed and recovered, sourcing the ydb_env_set file should rollback the database if replicated, recover regions with before image journaling, and give error messages about an inability to recover for other regions.

Current behavior

• If neither ydb_dir nor gtmgbldir is defined

  • If $ydb_dir/$ydb_rel/g/yottadb.gld exists, write shell export commands to set both ydb_gbdldir and gtmgbldir to point to it.
  • Otherwise create a default (using $ydb_dist/defaults) global directory at $ydb_dir/$ydb_rel/g/yottadb.gld and a database file with before-image journaling, and write shell export commands to set both ydb_gbldir and gtmgbldir.

• Otherwise, write shell export commands to set ydb_gbldir and gtmgbldir to the current value of $ydb_gbldir.

Note that the above has a minor bug. The last otherwise part should set both ydb_gbldir and gtmgbldir to $ydb_gbldir if it is defined, and to $gtmgbldir if it is not.

Proposed new behavior

(Robustify means recover / rollback as needed, and if possible to ensure the database is usable; generating an error otherwise.)

• If neither ydb_gbldir nor gtmgbldir is defined

  • If $ydb_dir/$ydb_rel/g/yottadb.gld does not exist, create a default global directory and database file with before-image journaling.
  • If $ydb_dir/$ydb_rel/g/yottadb.gld exists, robustify database as described below.

• If either or both ydb_gbldir or gtmgbldir is defined, the global directory is as specified, with $ydb_gbldir taking precedence over $gtmgbldir.

  • If the global directory file does not exist, create a default global directory and the database file with before-image journaling. Regardless of the location of the global directory, the database and journal files will reside in $ydb_dir/$ydb_rel/g/.
  • If the global directory exists, robustify the database as described below.

In all cases, write shell export commands to set ydb_gbldir and gtmgbldir to point to the global directory, with $ydb_gbldir taking precedence if both are defined.

An unrelated improvement is to send error output to stderr, rather than stdout.

Robustify Database

For each region in the global directory, use mupip dumpfhead to capture record("sgmnt_data.machine_name"), record("sgmnt_data.semid"), record("sgmnt_data.shmid"), record("sgmnt_data.repl_state"), record("sgmnt_data.jnl_before_image"), record("sgmnt_data.jnl_file_name"), record("sgmnt_data.jnl_state")

Cases:

Every region falls into one of the following:

• record("sgmnt_data.machine_name") is empty (database file is closed cleanly); or
• record("sgmnt_data.machine_name") is not empty (database file header shows it open), and
  • record("sgmnt_data.semid") corresponds to an existing semaphore id, and
  • record("sgmnt_data.shmid") corresponds to an existing shared memory segment

⇒ This is the normal operating state of database files, and there is nothing for ydb_env_set to do.

• record("sgmnt_data.machine_name") for one or more regions is not empty (database file header shows it open), and
  • record("sgmnt_data.semid") does not correspond to an existing semaphore id, or
  • record("sgmnt_data.shmid") does not correspond to an existing shared memory segment

⇒ Rollback or recovery needed.

Rollback or Recovery

• If any region with non-empty record("sgmnt_data.machine_name") does not have record("sgmnt_data.jnl_state")=2 and record("sgmnt_data.jnl_before_image")=1 ⇒ Error (backward recovery / rollback not possible)

• If all regions with non-empty record("sgmnt_data.machine_name") also have record("sgmnt_data.repl_state")=1 ⇒ Perform mupip journal -rollback -backward "*"

• If no region with non-empty record("sgmnt_data.machine_name") has record("sgmnt_data.repl_state")=1 ⇒ Perform mupip journal -recover -backward <regionlist> where <regionlist> is the list of regions with non-empty record("sgmnt_data.machine_name").

• If some but not all regions with non-empty record("sgmnt_data.machine_name") have record("sgmnt_data.repl_state")=1 ⇒ Error (inconsistent database state)

Draft Release Note