Overview

Records files contain information about every record in a segment. They also contain radar and processing parameters and metadata such as trajectory and attitude information.

The standard location is:

fullfile(gRadar.support_path,'records',radar_name,season_name)

where: radar_name is 'accum', 'kuband', 'rds', or 'snow' season_name follows the YYYY_LOCATION_PLATFORM format

The standard naming convention for records files is:

records_YYYYMMDD_SS.mat

where:
YYYYMMDD_SS = zero padded segment number (e.g. 20101230_02)

Record File Fields (file_version '1')

Nx is the number of records in this segment. Nb is the number of boards (or separate ADC channel files) for this segment. Nf(board) is the number of raw files associated with a particular board from this segment. The fields in the records file are:

|param_records| struct containing parameters passed in during creation of the records file| |pitch| 1 by Nx double vector containing the pitch of each record in radians. Positive nose up. Zero is level flight.| |phase_correction| Optional field (assumed to be zero if not assigned). Cell array for each waveform. Each waveform entry contains Nc by Nx single vector containing a phase correction for each record. Units of radians. This is the amount of phase to remove (i.e. a value of 30 deg means that the wf-adc pair is leading by 30 deg and the data_load.m will remove or lag 30 deg from this wf-adc pair when channel equalization is to be applied.)| |radar_name| string containing radar name| |raw |structure is radar dependent, contains the original raw header fields related to time synchronization for debugging| |relative_filename| Nb by 1 cell vector of Nf(board) by 1 cell vector of raw filenames, only the filename is included (not the path)| |relative_rec_num| Nb by 1 cell vector of Nf(board) by 1 uint32 specifying the record number of the first record of each file| |roll| 1 by Nx double vector containing the roll of each record in radians. Positive means right wing tip down. Zero is level flight.| |settings| structure is radar dependent,| |Tadc_correction| Optional field (assumed to be zero if not assigned). Cell array for each waveform. Each waveform entry contains Nc by Nx single vector containing a time correction for each record. Units of seconds. This is the amount of time to remove (i.e. a value of 1e-6 means that the wf-adc pair is 1e-6 seconds later than it should be and the data_load.m will remove this time during loading when channel equalization is to be applied.)|

Trajectory

Generally the trajectory information does not have the lever arm correction applied.

GPS Time

We use GPS time as the official time reference (rather than UTC time). We have a conversion function utc_leap_seconds.m.

Surface

Since the radar data itself is generally used for this ranging calculation, the units are propagation time in seconds.

Record File Versions

Version 1: original versions and did not contain "ver" or "radar_name" field Version 2: first version to contain "ver" or "radar_name" field Version 3: version cleans up mistakes and inefficiencies in the records file

raw and notes

Contents depend on the radar_name.

ACORDS

• raw.seconds contains the seconds count (UTC seconds from epoch)
• raw.fraction contains an all zeros placeholder for code unity (fraction' is a usecond clock that is added to 'seconds' in basic_load_acrods.m)
• raw.wfs contains the raw file waveforms in their original structure

ACCUM2

• raw.epri contains the free-running 64 bit 250 MHz counter

MCORDS2

• raw.epri contains the EPRI count (one count per transmit pulse)
• raw.seconds contains the seconds count (UTC seconds of day from NMEA string or other source)
• raw.fractions internal counter operated at fs that is reset each time the raw.seconds is incremented

MCRDS

• raw.radar_time contains the free-running 64 bit 10 MHz counter.
• raw.comp_time contains the host computer time (may not be set properly or be very accurate, but is used to help in synchronizing the GPS data to the radar data)
• raw.wfs contains the raw file waveforms in their original structure (a new wfs structure is included only when it differs from the last file)
• raw.wfs_file contains the file number that each entry of raw.wfs corresponds to

Snow and Kuband

• raw.epri contains the EPRI count (one count per transmit pulse)
• raw.seconds contains the seconds count (UTC seconds of day from NMEA string or other source)
• raw.fraction internal counter operated at fs that is reset each time the raw.seconds is incremented

Snow2 and Kuband2

• raw.epri contains the EPRI count (one count per transmit pulse)
• raw.seconds contains the seconds count (UTC seconds of day from NMEA string or other source)
• raw.fraction internal counter operated at fs that is reset each time the raw.seconds is incremented

Settings

Settings is a struct that contains fields that are custom to each radar_name.

Accum

Assume N different settings were used during the collection of a data segment.
records.settings.wfs_record(1:N): vector which contains the first record for each unique setting
records.settings.wfs(1:N): struct array of settings (one struct array per unique setting). Each element of the struct array contains the following fields:
 .wfs(:): Struct array containing waveform parameters with these fields:
   .presums: 8

Accum2

Assume N different settings were used during the collection of a data segment.
records.settings.wfs_record(1:N): vector which contains the first record for each unique setting
records.settings.wfs(1:N): struct array of settings (one struct array per unique setting). Each element of the struct array contains the following fields:
 .wfs(:): Struct array containing waveform parameters with these fields:
  .num_sam: 10000
  .which_bits: 0
  .t0: 2.3600e-07
  .presums: 128
  .bit_shifts: 7

SUPPORT WILL BE REMOVED FROM THESE FIELDS AND BUILT DIRECTLY INTO load_accum2_data.m AT SOME POINT:
records.settings.range_gate_start: [1:num_records] double vector holding the range gate start in fs clocked samples
records.settings.range_gate_duration: [1:num_records] double vector holding the range gate duration in fs clocked samples
records.settings.trigger_delay: [1:num_records] double vector holding the trigger delay in fs/4 clocks (trigger_delay and range_gate_start add together)
records.settings.num_coh_ave: [1:num_records] double vector representing number of coherent averages (i.e. presums)

ACORDS

Assume N different settings were used during the collection of a data segment.
records.settings.wfs_record(1:N): vector which contains the first record for each unique setting
records.settings.wfs(1:N): struct array of settings (one struct array per unique setting). Each element of the struct array contains the following fields:
 .wfs(:): Struct array containing waveform parameters with these fields:
  .presums: 128
  .bit_shifts: 0
  .f0: 140000000
  .f1: 160000000
  .Tpd: 3.00909e-06
  .wf_gen_clk: 110000000
  .daq_clk: 55000000
  .prf: 2500
  .tx_win: 0
  .num_samp: 3000
  .t0: 0
  .adc_gains: 100
  .blank: 4.0000e-06

records.settings{1}.wfs(1).wfs{:}
           wfs: (cell vector of structures)

wfs{:}
    wfs: [1x2 struct]

wfs(:).wfs
       presums: 128
    bit_shifts: 0
            f0: 140000000
            f1: 160000000
           tpd: 3.00909e-06
    wf_gen_clk: 110000000
       daq_clk: 55000000
           prf: 2500
        tx_win: 0
      num_samp: 3000
            t0: 0
     adc_gains: 100
         blank: 4.0000e-06

Arena

Data files stored in directories:

 board0, board1, board2, etc.

Each data file in a directory contains data streams

 stream0, stream1, stream2, etc.

Streams are either identified by profile processor (0 to 63) or by mode/subchannel when the profile processor is not present.

When a profile processor is not used, a lookup table is used to convert the mode/subchannel to a virtual profile processor. The is passed in through param.records.profiles in an N by 3 matrix representing N profiles. The first number is the mode, the second number is the subchannel, and the third number is the virtual profile mode to use.

Only the offsets for one profile are stored in the records.offset field. This profile is specified by param.records.epri_profile. records.settings fields:

The profile files, named records_YYYYMMDD_SS_PP.mat (PP for profile) contain the following fields:

MCoRDS2

Assume N different settings were used during the collection of a data segment.
records.settings.wfs_record(1:N): vector which contains the first record for each unique setting
records.settings.wfs(1:N): struct array of settings (one struct array per unique setting). Each element of the struct array contains the following fields:
 .wfs(:): Struct array containing waveform parameters with these fields:
  .wf_idx: 0
  .num_wfs: 2
  .presums: 6
  .bit_shifts: 3
  .start_idx: 1200
  .t0: 0
  .stop_idx: 3389
  .num_sam: 2189
  .offset: 4

ERROR IN CURRENT FORMAT: Because N was always 1 for this radar, the "wfs(1:N)" is currently missing. This means:
records.settings.wfs(1:N).wfs(:).FIELD
  is referenced as:
records.settings.wfs(:).FIELD

MCoRDS

Assume N different settings were used during the collection of a data segment.
records.settings.wfs_record(1:N): vector which contains the first record for each unique setting
records.settings.wfs(1:N): struct array of settings (one struct array per unique setting). Each element of the struct array contains the following fields:
 .wfs(:): Struct array containing waveform parameters with these fields:
  .presums: 8
  .bit_shifts: 0
  .t0: -1.0000e-05
  .start_idx: 100
  .num_sam: 31400
  .adc_gains: [54 54 54 54 54 54 54 54]

ERROR IN CURRENT FORMAT: Because N was always 1 for this radar, the "wfs(1:N)" is currently missing. This means:
records.settings.wfs(1:N).wfs(:).FIELD
  is referenced as:
records.settings.wfs(:).FIELD

MCRDS

Assume N different settings were used during the collection of a data segment.
records.settings.wfs_record(1:N): vector which contains the first record for each unique setting
records.settings.wfs(1:N): struct array of settings (one struct array per unique setting). Each element of the struct array contains the following fields:
 .wfs(:): Struct array containing waveform parameters with these fields:
  .presums: 64
  .shifts: 2
  .fs: 120000000
  .f0: 140000000
  .f1: 160000000
  .Tpd: 3.0000e-06
  .Tadc: 1.5000e-06
  .start_idx: 180
  .num_sam: 4672
  .adc_gains: 28.1838
  .adc_en: [1 1 1 1 1 0 0 0]

Snow and Kuband

Assume N different settings were used during the collection of a data segment.
records.settings.wfs_record(1:N): vector which contains the first record for each unique setting
records.settings.wfs(1:N): struct array of settings (one struct array per unique setting). Each element of the struct array contains the following fields:
 .nyquist_zone: (uint8)
 .loopback: (uint8)
 .wfs(:): Struct array containing waveform parameters with these fields:
  .presums: 8
  .bit_shifts: 0
  .t0: -1.0000e-05
  .start_idx: 100
  .num_sam: 31400
  .adc_gains: 20

Snow2 and Kuband2

Assume N different settings were used during the collection of a data segment.
records.settings.wfs_record(1:N): vector which contains the first record for each unique setting
records.settings.wfs(1:N): struct array of settings (one struct array per unique setting). Each element of the struct array contains the following fields:
 .nyquist_zone: (uint8)
 .loopback: (uint8)
 .wfs(:): Struct array containing waveform parameters with these fields:
   .presums: 8
   .bit_shifts: 0
   .t0: -1.0000e-05
   .start_idx: 100
   .num_sam: 31400
   .adc_gains: 20

Records Binary Auxiliary File (.nc version 3)

The NetCDF file is for quick loading of subarrays from the records file. The current standard uses netcdf_from_mat.m to do the conversion. To read data out, use the Matlab NetCDF tools (e.g. ncread.m) or netcdf_to_mat.m. The latter is useful for loading cell arrays and structs since it puts them back together again properly for you, ncread will only read N-D arrays properly.

Records File Creation

All records creation is done with records_create.m after preprocess.m is run.

Once the temporary file headers created during preprocess.m are loaded in, the workspace is saved.

Then a function called records_create_sync is called which corrects header data, syncs the GPS data, and writes the records file.