MWMAKE (Microwindow Selection) - User's Guide

PO-MA-OXF-GS-0016

ESA Contract 11886/96/NL/GS CCN5 Task 3

Page updated: 22-JAN-04 AD

Applies to MWMAKE v01MAR02 and later (previous v12APR01 user's guide)

Contents

  1. Introduction
  2. Compiling MWMAKE
  3. Input Spectra
  4. Terminal Inputs
  5. Example Driver File
  6. Output Files
  7. Figure of Merit (FOM)
  8. Growth Algorithms
  9. Termination Criteria
  10. Microwindow Database Files
  11. Appendix: Full list of Input Spectra
  12. Additional software
  13. References

1. Introduction

MWMAKE is the Oxford Microwindow/Occupation Matrix selection program developed for the MIPAS instrument (
Dudhia et al, 2002). It is a stand-alone FORTRAN-77 code, requiring a number of pre-computed input spectra to model both the retrieval and the various error sources. MWMAKE itself does not perform any radiative transfer calculations.

The program can either be used to select from a pre-existing set of microwindows, or to select new microwindows, or both. Selection is based on maximising a Figure of Merit (FOM), a scalar quantity which quantifies both the random and systematic error components of the retrieval.

An existing set of microwindows (in a `microwindow database' file) can be incorporated in three ways. In order of increasing complexity, these are

  1. `Excluded' Database: the spectral range of each existing microwindow in the file is flagged so that no new microwindows can overlap these.
  2. `Included' Database (unsorted): all microwindows will be read and added to the retrieval sequentially as if they had been selected in that sequence.
  3. `Included' Database (sorted): all microwindows are tested and the one giving the greatest improvement selected first, the process is repeated to select the next best microwindow, and so on while some improvement continues to be obtained.

    Searching for new microwindows, either from scratch or following on after incorporating an existing microwindow database, is a two-stage process

    1. Survey: in which each measurement (wavenumber,altitude) is tested individually for the improvement it would introduce to the retrieval (effectively treated as single-point microwindows). This creates a (wavenumber,altitude) map of `usefulness' of individual points.
    2. Growth: starting with a measurement in a region of high density of individually useful points, microwindows are grown `outwards' by adding adjacent measurements until no further improvement is obtained. This is repeated for a number of trial microwindows different starting points, and the best grown microwindow is retained.
    Although the new microwindow search procedure should produce a good approximation to a set of microwindows ordered by merit, if such an ordering is required it is recommended that MWMAKE is rerun supplying the list of microwindows as an `included' database which is then sorted

    A related use of the program is to generate different `occupation matrices', optimal subsets of microwindows selected for a particular retrieval ranges and/or situations when measurements are missing in particular bands or tangent altitudes.

    Back to Contents


    2. Compiling MWMAKE

    The source code consists of one program module mwmake.for plus a number of subroutines *.for and include modules *.inc. These should all be placed in a new directory.

    Then the module namfil.for should be edited so that the string constant PTBDIR (initially set to '/home/crun/eodg/mipas/spectra/') points to the top level directory that will contain the input spectra for the various atmospheres. The length of the character constant (C*30 initially) also has to be adjusted according to the new directory name (max of C*54).

    NB: The subroutine dattim.for calls a FORTRAN 90 function DATE_AND_TIME in order to return the current date and time, written to some output files. This is not implemented for Sun systems, so a replacement subroutine date_and_time.for is supplied ( see Additional Software) and should be included in the directory before compilation.

    The code can then simply be compiled as an executable mwmake using

    
              f77 *.for -o mwmake
    

    Back to Contents


    3. Input Spectra

    MWMAKE operates by analysing sets of precomputed spectra representing
    Jacobians for each retrieved species (including continuum) and various Error Sources represented as 1-sigma perturbation spectra. These are stored as binary files, one file containing the Jacobian or Error spectrum for one MIPAS band for one tangent altitude for one atmosphere.

    Jacobian spectra have names including "jac", e.g.

    
         ptb_d06000.bin_day_jac_h2o
    
    where d is the band, 06000 is the tangent height (6km), day is the atmosphere, h2o is the retrieved species.

    Error spectra have names excluding "jac", e.g

    
         ptb_d06000.bin_day_var_h2o
    
    where var_h2o is the error source (variability of H2O, in this case),

    Spectra exist for all 17 altitudes 6-68km represented in the nominal MIPAS scan pattern, but not all bands (eg there are no CH4 Jacobian or error spectra for the A band since there are no CH4 spectral features in the A band). The full list of spectra is given in the Appendix.

    All these spectra should be placed in a subdirectories of the top level directory (as defined by PTBDIR, see section on Compiling MWMAKE).
    ./day/jac day atmosphere, Jacobian spectra (anything with "jac" in the filename)
    ./day/err day atmosphere, error spectra (anything without "jac" in the filename)
    similarly for the other atmospheres ngt, sum, win and equ

    The spectra are generated as ASCII files (using the RFM), but, in order to save space and increase access speeds, stored as binary files (see binasc.f conversion program).

    If it is necessary to access the spectra as ASCII files, just edit the filenames in namfil.for. The subroutine which actually opens the files (getspc.for) already checks for a string ".bin" in the filename and treats the file as binary/ASCII appropriately.

    Back to Contents

    4. Terminal Inputs

    On execution, MWMAKE will prompt the user for information defining the task to be performed, most of which have reasonable defaults (ie the user can type a carriage return, indicated as <CR>). Input can either be via a terminal, or using a driver file. The driver file option can be invoked either by specifying the file name in response to the first prompt, or directly using the Unix construction, e.g.

    mwmake < mwmake.drv

    The terminal dialogue/prompts are described in sequence below, together with possible user responses and defaults.

    R-MWMAKE: Running MWMAKE v.01MAR02 (or later version)
    Info only: identifies the version of the code being executed. This version ID is also written to all output files.

    Driver file [<CR>=terminal input]:
    Either the name of a driver file which contains all subsequent responses, or <CR> signifying inputs via terminal.
    Example Responses
    mwmake_ch4.drv
    Name of a driver file containing subsequent user-responses
    <CR> (Default - set in mwmake.for)
    No driver file - all subsequent inputs via terminal

    Target species:
    Parameter(s) to be retrieved, eg:
    ch4, h2o, hno3, n2o, no2, o3 and/or pt (upper/lower case not significant, at least one space or comma between species to be retrieved simultaneously).
    pt retrieves both pressure and temperature profiles jointly, assuming assuming a priori pointing knowledge (random 1.5% pressure between adjacent sweeps). ctm (continuum) can also be entered as a target parameter: a continuum profile is always retrieved for each microwindow but if specified as a target species then this is also incorporated in the Figure of Merit calculation.
    Example Responses
    CH4 N2O
    Methane and nitrous oxide joint-retrieval microwindows
    NB: There is no default option

    [prompt only if joint retrieval specified in previous response, or retrieved molecule > 4 characters in length]
    Retrieval code (up to C*4):
    Microwindows have labels of the form RRRRnnnn where nnnn is a number (padded with leading zeros). For single target species with molecule names 4 characters or less (including 'PT'), RRRR is set automatically to be the the retrieved species padded with trailing underscores. For other cases the user has to supply the 4-character code (case sensitive).
    Example Responses
    pTwv
    Pressure, temperature and Water Vapour joint retrieval (ie generate MWs with labels like pTwv0001)
    NB: There is no default option

    Atmosphere: [day<CR>,ngt,sum,win,equ,glo,glw]:
    The three-letter code selecting atmosphere(s) to be used for evaluating the Figure of Merit. First five are as in input spectra, additional options are
    glo Global average of 5 atmospheres
    glw Global average plus extra weight for polar winter
    Example Responses
    glw
    Use Polar winter-weighted global set
    <CR> (Default - set in mwmreq.for)
    Use mid-latitude day-time only

    VMR Accuracy requirement? [<CR>=25.0%]: (if molecule selected as the (first) target species),
    or
    Temp Accuracy requirement? [<CR>=3.0K]: (if pT selected as the (first) target species.)
    This sets up a constant value for the 'target accuracy profile', although how it is used is defined by other parameters. In the case of joint retrievals, it is defined for the first species specified in the Target species section.
    Example Responses
    15
    Target profile set to 15% (assuming VMR) or 15K (assuming pT)
    <CR> (Default - set in mwmreq.for)
    Target profile set to 25% for VMR retrievals, 3K for temperature retrievals.

    Waveno.range or bands [<CR>=all]:
    The lower, upper wavenumber limits [cm-1] to be used for microwindow selection, or, alternatively, a list of MIPAS bands:
    • A 685-970 cm-1
    • AB 1020-1170 cm-1
    • B 1215-1500 cm-1
    • C 1570-1750 cm-1
    • D 1820-2410 cm-1
    (v14MAY02 onwards) if a wavenumber range is specified, it is assumed that this only applies to the search range for new microwindows. However, if bands are specified it is assumed that this applies to new microwindows and to included microwindows (in earlier versions a specified wavenumber range also applied to both)
    Note: No points are selected within 0.2 cm-1 of the MIPAS band edges (to allow for AILS convolution)
    Example Responses
    1200 1300
    Search range 1200-1300 cm-1, although this will actually only only cover 1215-1300cm-1 since 1200 lies between the AB and B bands, and the B band nominally covers 1215-1500cm-1.
    a d b (case-insensitive)
    Search MIPAS bands A, B and D only and, if existing microwindows are to be included, include only MWs within these bands.
    <CR> (Default - set in reqbnd.for)
    Search entire MIPAS range (685-2410 cm-1, excluding gaps between filters, which are not covered by the input spectra)

    I-REQBND: Searching Wavenumber range 685.200 2409.800 (for example)
    Info only: verification of the lower and upper wavenumber limits to be searched.
    Note: the above response would also be obtained if the user had typed a d b in response to the previous prompt - this message only lists the extreme lower and upper wavenumbers and not the details of bands to be skipped in between these limits.

    Merit(0:7), CPU Cost(0:9) and Mask(0:4) Fns? [<CR>= 0 0 0]:
    Integer parameters defining Merit function, CPU Cost function and MW growth algorithm to be used.
    These 3 digits are concatenated to form the `identifier' (ijk) for this particular run of MWMAKE (although this will be overridden if missing bands/altitudes are selected)
    Example Responses
    4 1 2
    Use Merit Fn#4, CPU Cost Fn#1 and MW Growth Algorithm#2 Output filenames will contain the string '412'.
    <CR> (Default - set in mwmreq.for)
    Use Merit Fn#0, CPU Cost Fn#0 and MW Growth Algorithm#0. Output filenames will contain the string '000'.

    List of retrieval altitudes [km] [<CR>=6,9,..68]:
    Tangent altitudes for both measurements and retrieval (arbitrary order, but separated by at least 0.5km)
    Example Responses
    5 6.5 8 9.5 11 12.5 14 15.5 17 18.5 20 25 30 40
    MIPAS Special Observation Mode "S2"
    <CR> (Default - set in reqalt.for)
    Use nominal scan pattern (6-68 km in 17 steps)

    List of missing altitudes [km] [<CR>=none]
    Any tangent altitudes missing from the measurements (but not the retrieval) - used for constructing Occupation Matrices for missing-data scenarios. These altitudes have to be a subset of the altitude list supplied previously, or an error message will result. This overrides the usual identifier used in output filenames.
    Example Responses:
    9 11
    missing measurements at 9 and 12 km
    <CR> (Default - set in reqalt.for)
    retrieve at 8, 11, ... 53km (16 tangent heights), with spectra available at all altitudes.

    [prompt only if missing altitude(s) have been specified in previous response]
    Missing Band (A,AB,B,C or D)? [<CR>=all]:
    For the missing altitude(s), the spectra may either be missing for all bands or for just one band.
    Example Responses
    ab (case insensitive)
    At previously specified missing altitudes, only AB band missing (A, B, C and D band spectra all available).
    <CR> (Default set in reqalt.for)
    All bands missing at missing altitude(s).

    Termination criteria Npt,NMW,DMrt,ReqF? [<CR>=5000 10 0.0 0.0]:
    The microwindow selection will finish when any of these criteria are met.
    See section on Termination Criteria for explanations.
    Example Responses
    99999 10 0.1 1.0
    Terminate when either 10 microwindows have been found, or FOM no longer increases by at least 0.1, or when the requirement profile has been met at all altitudes
    <CR> (Default - set in reqend.for)
    Terminate when either 5000 points or 10 microwindows have been used, or when FOM no longer increases, and ignore requirement profile as a termination criterion.

    Minimum ctm info/level [<CR>=0 bits]:
    The minimum continuum information content required for each altitude in each microwindow. 1 `bit' corresponds to a factor 2 reduction from the a priori uncertainty (set at +/- 0.1/km extinction).
    Example Responses
    5
    5 bits, as currently used for MIPAS microwindows, equivalent to extinction retrieval to accuracy of +/-0.003/km
    <CR> (Default - set in mwmreq.for)
    No continuum requirement (allows opaque tangent paths)

    I-MWMREQ: Min Ctm = A Pr.Cov* 9.7656250E-04 (for example)
    Info only: verification of the fraction of the a priori covariance that will be used as a threshold value for the continuum retrieval. Since covariance is the square of the uncertainty, this reduces by a factor 4 for every `bit' selected, so for 5 bits, 9.8E-04 = 1/1024.

    Min,Max Trials for each new MW? [<CR>= 30 99]:
    The minimum and maximum number of trial MWs that will be grown before a new MW selected (see Growth). After the minimum number of trials, the best of any acceptable microwindow found so far will be selected. If no acceptable MWs have been found, selection continues until one is found, or the maximum number of trials is reached at which point the selection ceases with a message
       I-MWMAKE: No more MWs found
    
    Increasing the minimum number of trials increases the likelihood of a better MW being found at each stage, but also increases the program time. 99 is the maximum that fits the 2-digit representation of trial# in the diagnostics file.
    Example Responses
    10 10
    Fix 10 trials for each new microwindow.
    <CR> (Default - set in mwmake.for)
    min 30 max 99 trials for each new microwindow

    Exclude database
    Name of a microwindow database file (IMK-format) containing a list of microwindows whose spectral ranges will be excluded to prevent new microwindows having spectral overlaps.
    Example Responses
    MW_CH4_103.DAT
    Exclude spectral grid points covered by any microwindow within database MW_CH4_103.DAT
    <CR> (Default - set in mwmprv.for)
    No excluded spectral points

    Include database
    Name of an microwindow database file (IMK-format) containing a list of microwindows to be included in the retrieval prior to the addition of new microwindows. Microwindows from this file will be skipped if any of the following apply:
    1. MW not completely within selected wavenumber range; or
    2. Any wavenumber grid pt within the MW already flagged to be `excluded' (eg if already included in another MW); or
    3. MW altitude range does not overlap the required range; or
    4. MW altitude range only overlaps at 'missing' altitudes; or
    5. No 'T' masks in any included MW altitudes
    Example Responses
    MW_CH4_103.DAT
    Include microwindows within database MW_CH4_103.DAT
    <CR> (Default - set in mwmprv.for)
    No previous microwindows used.

    [Prompt only if an `included database' is specified in the previous response. There may a slight pause at this point as the entire file is scanned to determine the highest microwindow# currently used]
    Sort database (Y/N)? [<CR>=N]:
    An 'included' database may be 'sorted', in which case microwindows from that database will be selected in sequence according to improvement in the figure of merit. If the database is not sorted, the microwindows are included in the sequence in which they occur in the file.
    Note: The previously specified termination criteria generally apply during the inclusion of predefined MWs as well as when creating new MWs. However, if predefined MWs are to be included without sorting, the 'DMrt' criterion is temporarily suspended since the MWs in the file are assume to occur in a random sequence (and therefore some may temporarily decrease the figure of merit depending on the order in which they are incorporated).
    Example Responses
    y (case insensitive)
    Sort microwindows in `included database'
    <CR> (Default - set in mwmprv.for)
    Use microwindows in `included database' in the order in which they occur

    New MW#i from i=? (0=none) [<CR>=0001]:
    ['0001' in the prompt message will be replaced by N+1 if an `included database' containing microwindows up to number #N has been specified]
    Microwindows are identified by labels such as 'CH4_0015' and the user can specify the starting number for the sequence of new microwindows to be created.
    Note: new microwindows will not be created if the termination criteria are met while the `include database' is being read.
    Example Responses
    0
    Create no new microwindows (i.e. simply evaluate and/or sort microwindows supplied via an `include database')
    21
    Create new microwindows starting from #0021
    <CR> (Default - set in mwmprv.for)
    Create new microwindows starting from number given in prompt message

    After this record, no further user input is required.

    Back to Contents


    5. Example Driver File

    Below is an example of a driver file for MWMAKE. Record starting with '!' signify a comment. The first record *must* be a comment, but after that they are optional and may be inserted anywhere (here, they are used to clarify the sequence of prompts requiring each response). Note that responses given to `List of missing altitudes' and `Include Database' determine what the next prompt will be.
    ! MWMAKE Driver File
    ! Target species: 
    ch4
    !  Atmosphere: [day,ngt,sum,win,equ,glo,glw]:
    glw
    ! Temp Accuracy requirement? [=3.0K]:
    
    ! Waveno.range or bands [=all]:
    ab c d
    ! Merit(0:7), CPU Cost(0:9) and Mask(0:3) Fns? [= 0 0 0]: 
    0 0 3
    ! List of retrieval altitudes [km] [=6,9,..68]:
    
    ! List of missing altitudes [km] [=none]: 
    
    ! NB: If any missing altitude specified, next entry should be response to
    !     prompt for missing bands
    ! Termination criteria Npt,NMW,DMrt,ReqF? [= 5000 10 0.0 0.0]:
    
    ! Minimum ctm info/level [=0 bits]:
    5
    ! Min,Max Trials for each new MW? [= 30  99]:
    
    ! Exclude database:
    
    ! Include database:
    MW_CH4_103.DAT
    ! sort (only applies if an Include Database has been specified previously)
    N
    ! New MW#i from i=? (0=none) [=0001]:
    26
    ! End
    

    Back to Contents


    6. Output Files

    A run of MWMAKE will produce the following output files. rtv is the retrieved species (lower case). If no missing altitudes are selected, then ijk are the selected Merit/CPU/Growth parameters. If missing altitudes are selected, then i indicates the missing band (i=0 all bands, i=1 A band, i=2 AB band etc) and jk becomes the (highest) missing altitude [km] (e.g. jk=42 if both 39 and 42 km specified as missing).
    rtv_ijk.dia
    Diagnostics file. Opened at start of program. Regularly updated.
    rtv_ijk.err
    Error analysis. Opened at start of program. Updated after each microwindow.
    rtv_ijk.lst
    Microwindow listing. Simplified list of microwindows and absorbers for each. Opened at start of program. Updated after each microwindow.
    rtv_ijk.tmp
    Temporary file for creating microwindow file. Opened at start of program. Rewritten prior to production of each microwindow file, and can be deleted upon completion.
    RRRRnnnn.imk
    Microwindow file (for new microwindows only), RRRR is the upper-case version of rtv padded to 4-characters with underscores nnnn is the microwindow number. Created as each new microwindow is found.
    OM_RRRRijk.DAT
    Occupation matrix. Opened at start of program. Data written at end of program.

    Diagnostics File

    Diagnostics of the MW selection/creation process. Types of record:
    Diagnostics Header
    2 records at start of each file giving general info on this particular run, e.g.
    
    ! Microwindow diag. 1020.200:2409.800 MRT=1 CPU=0 MSK=3 glw MWMAKE v.01MAR02
    ! Sweep range= 6.0:68.0  Missing band=none Missing alt=none
    
    Description
    1020.200:2409.800
    Lower and Upper Wavenumber limits (cm-1) of search range (i.e., ab, b, c and d bands here)
    MRT=1 CPU=0 MSK=3
    Selected Merit Fn#1, CPU Cost Fn#0, MW Growth Algorithm#3.
    glw
    atmosphere(s) used for selection
    MWMAKE v.01MAR02
    Version of MWMAKE used
    Sweep range= 6.0:68.0
    Spectra cover tangent altitudes from 6-68 km
    Missing band=none
    No missing bands
    Missing alt=none
    No missing altitudes i.e., all 17 spectra from 6 to 68 km are available.

    Microwindow Diagnostics
    6 records for each microwindow included or created, e.g.
    
    CH4_0022 1607.750 1610.750 15 60 1573  918  2308  1225  19.802   4.405  12.036
     34.4 38.2 41.0 35.5 32.3 32.0 36.0 42.7 41.9 40.6 45.1 49.6 48.3 59.8 86.7100.0
     44.6 41.9 43.1 26.4 19.4 16.1 14.7 15.8 12.3 14.4 20.8 14.9 12.0 20.2 80.4  0.0
     56.3 56.7 59.5 44.2 37.7 35.8 38.9 45.6 43.7 43.1 49.6 51.7 49.8 63.2118.2100.0
    
    Description
    CH4_0022
    Microwindow label (RRRRnnnn format).
    1607.750 1610.750
    Lower, Upper wavenumber boundaries for microwindow (3cm-1 is the maximum width, =121 spectral grid points @ 0.025cm-1 spacing)
    15 60
    Lower, Upper altitude boundaries for microwindow (=13 altitudes)
    1573 918
    1573 points in this microwindow (=121 spectral grid * 13 altitude grid) of which 918 points are masked `FALSE'
    2308 1225
    Cumulative totals for number of points and number of F masks so far for this and any preceding microwindows.
    19.802
    Cumulative Shannon information content for the retrieval (i.e. evaluated using Merit Fn#4, irrespective of merit function selected for this run.)
    4.405
    Figure of Merit increment for this microwindow, evaluated according to the selected Merit Fn (#1 in this case)
    12.036
    Cumulative Figure of Merit for the current selection.
    34.4 38.2 41.0 ...
    Random Error component of current retrieval for lowest 16 altitudes (i.e. 34.4% uncertainty at 6km, 38.2% at 9km, etc)
    44.6 41.9 43.1 ...
    Systematic Error component at same altitudes
    56.3 56.7 59.5 ...
    Total Error component at same altitudes (= Sqrt (Random^2 + Systematic^2) )

    Trial Diagnostics
    One record generated for each trial microwindow, e.g.
    
    Trial#  5  Wno=1305.500[1305.750]1306.150  Alt=52[60]68  Npt=  81  Mer=  2.544*
    
    Description
    Trial# 5
    5th trial microwindow to be grown
    Wno=1305.500[1305.750]1306.150
    Lower boundary, [Starting point], Upper boundary in the wavenumber domain.
    Alt=52[60]68
    Lower boundary, [Starting point], Upper boundary in the tangent height domain
    Npt=81
    Number of points in the microwindow.
    Mer= 2.544
    Increment in the Figure of Merit if this microwindow is used.
    An additional character may be present at the end of the record:
    * = best microwindow so far (as in example)
    - = MW excluded since it fails to meet the minimum spectral width requirement of 2 grid points
    c = MW excluded since it fails to meet the minimum continuum information requirement

    Mask Optimisation Diagnostics
    2 Records summarising start and end of mask optimisation (only if MW Growth Algoritms #2 or #3 selected), e.g.
    
     I-OPTMSK: Iter#           0 Merit:   1.642324     NMask:         500
     I-OPTMSK: Iter#           5 Merit:   1.655388     NMask:         495
    
    Description
    Iter# 0
    Iteration number, 0 being the state prior to any toggling. The logical status of one mask is changed each iteration, so in this case there were 5 mask changes. It is possible for the status of a mask is changed from F to T in one iteration and then back to F in a subsequent iteration, although not the next iteration).
    Merit: 1.642324
    Value of the Figure of Merit increment upon completion of iteration.
    NMask: 500
    Number of False masks in microwindow. In this example, the toggling set 5 points from F to T. Since there were 5 iterations it follows that no points were set from T to F.

    Termination Diagnostics
    Record indicating the reason why the selection terminated, e.g.
    
    I-MWMWRT: Terminated due to No.MWs = limit (  5)
    

    Error Analysis File

    The retrieval error and its components after each MW is incorporated. Types of record:
    Error Analysis Header
    2 records at start of file, containing same information as Diagnostics Header, e.g.
    
    ! MWMAKE error analysis 1020.200:2409.800 MRT=1 CPU=0 MSK=3 glw MWMAKE v.01MAR02
    ! Sweep range= 6.0:68.0  Missing band=none Missing alt=none
    

    Array size information
    1 record containing the number of retrieved target parameters and number of profile levels, e.g.
    
             3        17
    
    Description
    3
    Number of target parameters (pT counts as two)
    17
    Number of profile levels

    List of target parameters
    1 record containing the list of retrieved species, e.g.
    
    tem       pre       h2o 
    

    List of retrieval altitudes
    1 or more records containing list of retrieval altitudes, 16 altitudes per record, e.g.
    
      6.0  9.0 12.0 15.0 18.0 21.0 24.0 27.0 30.0 33.0 36.0 39.0 42.0 47.0 52.0 60.0
     68.0
    

    Error Profiles
    2 or more records per error profile. The first 3 sets of records are the Random, Systematic and Total Error profiles, as in the Microwindow Diagnostics but with the first record giving the title of each and subsequent record(s) containing error values for all retrieval altitudes (not just the lowest 16). For joint retrievals the first row(s) are for the first listed retrieval parameter, followed by the next parameters starting on a new row etc. Then sets of records for each significant systematic error component, e.g.
    
     hitran
      6.5 13.4  5.7  4.7  3.2  4.6  3.4  1.8  1.2  1.2  1.4  1.2  1.2  2.2  2.5  0.0
      0.0
    
    Description
    hitran
    Error source (see Spectra for full list).
    6.5 13.4 5.7 ...
    HITRAN database uncertainties contribute 6.5% error at the lowest profile level, 13.4% at the next level etc. Errors in a temperature retrieval are written to 2 decimal places and expressed in K.

    Microwindow Listing

    List of microwindows in selection/creation sequence.
    Microwindow Listing Header
    2 records at the start of file, containing same information as the Diagnostics Header, e.g.
    
    ! Microwindow list 1020.200:2409.800 MRT=1 CPU=0 MSK=3 glw MWMAKE v.01MAR02
    ! Sweep range= 6.0:68.0  Missing band=none Missing alt=none
    

    Microwindow Information
    One record for each microwindow, fully defining microwindow apart from mask information, e.g.
    
      3 CH4_0022 1607.750 1610.750 15 60  6  1  3  4 10  8 11
    
    Description
    3
    Number of this microwindow in selection/creation sequence
    CH4_0022
    Label, in RRRRnnnn format.
    1607.750 1610.750
    Lower and upper wavenumber [cm-1] boundaries
    15 60
    Lower and upper tangent altitude [km] boundaries
    6 1 3 4 10 8 11
    List of HITRAN/RFM codes for each absorber. Target species first (6=CH4, with 2 = CO2 for temperature microwindows), but order of remainder is not significant.

    Occupation Matrix

    This file contains a summary of the microwindows and altitudes used for a particular selection. The format of occupation matrices is defined in
    [2], and the following Field# and Descriptions are taken from that document. Note that '#' indicates an optional comment record, but those generated by MWMAKE are listed below as they sometimes contain additional information not specified in the file format. A typical file is constructed as follows
    Field#0 Common File Header
    General header for all ESA files, containing the generation time and a comment describing the version. For files generated by MWMAKE the comment contains the same information as the first record of the Diagnostics Header), e.g.
    
    27-MAR-2001 16:06:13.046000
    # Oxf Occupation matrix 1020.200:2409.800 MRT=1 CPU=0 MSK=3 glw MWMAKE
    v.01MAR02  
    
    Field#1 Label of Occupation Matrix
    OM Labels are of the form OM_RRRRijk (where OM_RRRRijk.DAT is the file name)
    
    # OM label:
    OM_CH4_023
    
    Notes
    1. RRRR is the retrieved species, upper case, padded to 4 characters with trailing underscores (`_').
    2. ijk will either be the selection criteria, or will represent the specified missing bands and altitudes. E.g, '023' could either indicate Merit Fn#0, CPU Fn#2 and Growth Algorithm#3, or that this OM was selected for all bands missing ('0') at 23 km ('23') altitude.

    [Comment] Figure of Merit
    Figure of Merit for this particular selection, used to compare different OMs. Note that this is inserted here as a comment as it is not part of the defined file format, e.g.
    
    # Figure of Merit=  56.00363
    
    Field#2 Number of Altitudes (Rows) in this Matrix
    Number of tangent altitudes for spectra, e.g,
    
    # No of altitudes:
     16
    
    Field#3 Altitude Bands within which this Occupation Matrix is Valid
    Lower and Upper altitudes for each spectrum, assuming 1.5 km tolerance. For spectra nominally at 8 - 53km, this gives 16 records (in descending altitude)
    
    # Altitude bands:
       51.50     54.50
       48.50     51.50
        ....
        6.50      9.50
    
    Field#4 Number of Microwindows (columns) in this Occupation Matrix
    Number of MWs selected, e.g.,
    
    # Number of MWs:
     10
    
    Field#5 Labels of Microwindows
    List of MW Labels, 7 per record, sorted by increasing wavenumber e.g.,
    
      CH4_0006  CH4_0004  CH4_0007  CH4_0001  CH4_0008  CH4_0009  CH4_0005
      CH4_0002  CH4_0003  CH4_0010
    
    Field#6 Occupation Matrix
    The occupation matrix itself, one column per microwindow in sequence shown by preceding list of labels, one row per tangent altitude (descending order) as listed in the altitude bands. A figure '1' indicates MW/Altitude is used, '0' indicates that it is not, e.g.
    
    # Occupation matrix:
    #B B B B B B B B B B
     1 1 0 1 0 1 0 1 1 0
     1 1 0 1 0 1 1 1 1 0
     ...
     1 1 1 1 1 1 1 1 0 1
    
    Notes
    1. The second comment line (#B B ...) indicates the MIPAS band containing each microwindow (all B-band microwindows in this case).
    2. There should be at least one '1' in each column, indicating that each microwindow uses data from at least one tangent height
    3. If any sweeps (and bands) are designated as missing, there will be '0' for all entries on the corresponding row (and band)
    4. If any rows of the OM are entirely zero, the retrieval level will be set to zero for that level as well.

    Field#7 Logical Retrieval Vector
    Set of flags indicating whether a retrieval is attempted ('1') or not ('0') at each sweep altitude, e.g., if the retrieval is not attempted at the highest altitude,
    
    # Retrieval vector:
     0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
    
    Notes
    1. If the retrieval does not meet the requirement accuracy at a particular altitude, the retrieval vector is set to 0
    2. If a retrieval vector is set to 0 at either end of the profile, all corresponding tangent altitude row elements of the occupation matrix also set to 0. For pT retrievals this also applies if an internal retrieval vector element is set to 0.

    Field#8 Number of Sweeps for which Parameters will be Fitted
    No of 1's in the retrieval vector, e.g.,
    
    # No. sweeps for which profile fitted
     16
    
    Other records of the Occupation Matrix depend only on the species retrieved and not the microwindow selection.

    Back to Contents


    7. Figure of Merit

    The Figure of Merit H, measured in `bits', represents some form of the information content of a set of measurements. It is defined as:
    H = -0.5 log2 ( (Mrtv/Mapr) * (Crtv/Capr) )
    where M is the
    Merit Function and and C the CPU Cost Function evaluated before (apr) and after (rtv) the retrieval.

    This value is determined within the function fmerit.for

    Merit Functions

    MWMAKE uses a merit function with the general form
    M = D ( Ft*St + Fs*Ss + Fr*Sr )
    where Ft,Fs,Fr are weighting factors for the total, systematic, requirement error covariances St,Ss,Sr, and D is some type of determinant operator. The weighting factors and determinant operator are set by the
    user-selected value of merit function# (0-7).

    Weighting Factors
    0 or 4: Ft=1 Fs=0 Fr=0 (select according to total error only)
    1 or 5: Ft=1/2 Fs=1 Fr=0 (bias towards low sys errors)
    2 or 6: Ft=1 Fs=0 Fr=1 (bias towards improvements at levels where requirement not met)
    3 or 7: Ft=1/3 Fs=2/3 Fr=1 (bias towards low sys errors & where req.not met)
    Determinant Operator
    0-3 D based only on product of diagonals
    4-7 D based on determinant of full covariance matrix
    Note: for pt retrieval the single state vector element corresponding to pressure is given an extra weight n corresponding to the number of temperature elements in the profile (so that, overall, pressure and temperature have equal weights).

    CPU Cost Function

    This represents a modification to the Figure of Merit expressing the penalty associated with CPU cost, i.e., how much improvement in the retrieval can be traded for faster processing time.

    The CPU time T required to perform a retrieval is (somewhat arbitrarily) represented by

    T = 1000 + Npt + 100*Nmw
    where Npt is the total number of points (i.e. measurements) and Nmw is the total number of microwindows (representing some overhead associated with adding each new microwindow as well as the additional points included).

    The user-selected value of the CPU cost-function j (an integer in the range 0:9) determines how this time T is converted to CPU Cost C according to

    C = Tnj/10
    Where n is the number of profile levels to be retrieved. Thus a value j=0 corresponds to no CPU cost at all (C= 1 = constant), and a maximum value j=9 corresponds to a weighting close to the C = (Npt)n limit that can be sustained only by the Merit function decreasing proportionally to (Npt)n, which generally corresponds to the limit of random noise reduction (M*C = constant), which is too large a CPU cost for practical purposes (typically M reduces with Npt raised to a smaller exponent than n).

    For existing microwindows which have associated SVD-decomposed look-up tables (so that the total number of singular values, Nsv, is written in the database) there are additional CPU-cost functions which may be used to sort a database. These are used for values of j = A, B, C ... F (i.e. 10-15 in hexadecimal notation). In these cases, the CPU cost is modified so that Npt is now scaled by Nsv/100:

    T = 1000 + Npt*Nsv/100 + 100*Nmw
    C = Tn(j-9)/10

    Back to Contents


    8. Growth Algorithms

    A microwindow size is defined by its dimensions w*a (w = number of spectral grid points, a = number of altitude grid points), with w = a = 1 initially. It is also possible to have individual measurements excluded from within the microwindow by setting a `False' mask but the CPU cost estimates are based only on N = w*a (this is because it is not practical to exclude masked points from the forward model calculations, which is the time-consuming part, they are only removed in the matrix operations associated with updating the retrieval estimate).

    There are two methods for growing microwindows

    1. Edgewise: increase dimension w or a by one so as to include all points along whichever boundary gives the most improvement. This scheme also tests increments of 2 (i.e., adding two rows or two columns at a time) to allow for more expansion possibilites.
    2. Pointwise: add single points around perimeter according to which gives the most improvement, and once no further points are found increment w and/or a to represent the new boundaries and repeat.

      Apart from lack of improvement, growth can also be truncated by specified altitude/wavenumber boundaries, or by a maximum width requirement of 3cm-1 (w=121 spectral points @ 0.025cm-1 resolution).

      Note that the Edgewise algorithm will only produce contiguous microwindows (all points set `True', meaning `use all points withing boundaries'), while the Pointwise algorithm will generally produce perforated algorithms by choosing to omit certain points (set `False').

      For microwindows grown pointwise, there are two further steps which can be applied to optimise the mask (applied to the finally selected microwindow only:

      Mask optimisation
      `Toggle' each point within the microwindow (i.e. set `True' to `False' or vice-versa), save the new status of whichever point gives the most improvement and continue until toggling each point reduces the quality of the retrieval.
      Mask uniformity
      An additional penalty can be imposed to discourage fragmented mask patterns, i.e., encourage solitary `True' points surrounded by `False' points to be set `False' and vice-versa. A larger penalty is applied for non-uniformity in the altitude domain than in the spectral domain.

      (v14MAY02 onwards) Remask: for included microwindows, there is a further option which is to reapply mask optimisation plus uniformity. This creates new microwindows but constrained to have the same altitude/wavenumber boundaries as the old microwindow. Any subsequent microwindows which are grown are then generated pointwise, also applying mask optimisation plus mask uniformity.

      The particular growth algorithm used is determined by a user-supplied Mask Function, a parameter in the range 0-3.

      0: Edgewise (i.e. contiguous microwindows only)
      1: Pointwise
      2: Pointwise plus mask optimisation
      3: Pointwise plus mask optimisation plus mask uniformity
      4: Remask included MWs, then as #3 for any additional microwindows

      Back to Contents


      9. Termination Criteria

      Microwindow selection, whether by growing new microwindows or using an existing database, will terminate when any one of four criteria are met.
      Total Number of Points Npt
      The total number of points selected (ie including those which are "mask ed") exceeds the specified number.
      Disabled by setting a large number, eg 999999.
      Total Number of Microwindows NMW
      The total number of microwindows found/used.
      Disabled by setting a large number, eg 9999.
      Minimum increments in FOM DMrt No microwindow can be found that increases the FOM by at least this amount
      Note: this test is switched off while an existing set of microwindows is `included' without being sorted since there is no guarantee that the microwindows are stored in the database in a sequence that will give a monotonically increasing FOM, nor that the FOM used to create the database is the same as currently being used.
      Disabled by setting a large negative number, eg -1.0E6 (although you will almost certainly want to retain a minimum value of 0.0)
      Requirement accuracy factor ReqF
      Factor * Requirement accuracy which has to be met at all altitudes. Eg, if the requirement profile is 25%, and a figure 0.6 is supplied for this parameter, the program will terminate when the total retrieval error is 15% or less at all levels.
      Note that this factor allows different values of the requirement accuracy to be specified for the Merit Function and to terminate the selection.
      Disabled by setting a value 0 (since zero error cannot be achieved).

      Back to Contents


      10. Microwindow Database File

      The ESA format for a microwindow database is defined in
      [1] and [2]. When MWMAKE creates new microwindows it writes each one as its own database file with a name RRRRnnnn.imk where RRRRnnnn is the Microwindow Label, eg O3__0002. This is the only file in which mask information is stored. For microwindows without masks (i.e. generated using Mask Fn#0), the microwindow is fully specified, and probably more conveniently read, from the Microwindow Listing.

      When sets of microwindows are to be `excluded' or `included', these have to be contained in a single database file (use imkmrg.for).

      Back to Contents


      11. Appendix: Full List of Input Spectra

      This is the list of sample filenames of spectra avaialable for use with MWMAKE. Spectra exist for all 5 atmospheres, for all 5 bands except where otherwise stated, and for all 17 altitudes in the nominal scan pattern:
      • 6km, 9km, ... 42 km @ 3km intervals, 47, 52, 60, and 68km.
      None of the spectra is mandatory so, to exclude a particular error source, simply remove the files from the PTBDIR sub-directory. However, warning messages are issued if Jacobian spectra are missing for a required band, or any of the error spectra which normally exist for all bands.

      Jacobians

      Jacobian spectra represent the difference between the nominal radiance and the radiance if the target quantity is perturbed at the tangent level. Perturbation sizes are 1K for temperature, 1% for VMR or pressure, 1e-4/km extinction for continuum.
      • ptb_b06000.bin_jac_day_ch4 (AB, B, C, D)
      • ptb_a06000.bin_jac_day_h2o
      • ptb_a06000.bin_jac_day_hno3 (A, AB, B, C)
      • ptb_a06000.bin_jac_day_n2o
      • ptb_a06000.bin_jac_day_no2 (A, B, C)
      • ptb_a06000.bin_jac_day_o3
      • ptb_a06000.bin_jac_day_pre
      • ptb_a06000.bin_jac_day_tem
      • ptb_a06000.bin_jac_day_ctm

      Composition Uncertainties

      These represent the difference between spectra calculated for the nominal atmosphere and an atmosphere where the constitutent profile is perturbed by its estimated climatological variability (+1 sigma).
      • ptb_a06000.bin_day_var_c2h2 (A)
      • ptb_a06000.bin_day_var_c2h6 (A)
      • ptb_a06000.bin_day_var_ccl4 (A)
      • ptb_b06000.bin_day_var_ch4 (AB, B, C, D)
      • ptb_a06000.bin_day_var_clo (A)
      • ptb_a06000.bin_day_var_clono2 (A, B, C)
      • ptb_a06000.bin_day_var_co (D)
      • ptb_a06000.bin_day_var_co2 (A, AB, B, D)
      • ptb_a06000.bin_day_var_cof2 (A, B, D)
      • ptb_a06000.bin_day_var_f11 (A, AB)
      • ptb_a06000.bin_day_var_f12 (A, AB)
      • ptb_b06000.bin_day_var_f14 (B)
      • ptb_a06000.bin_day_var_f22 (A, AB, B)
      • ptb_a06000.bin_day_var_h2o
      • ptb_a06000.bin_day_var_hcn (A, B)
      • ptb_a06000.bin_day_var_hno3 (A, AB, B, C)
      • ptb_a06000.bin_day_var_hno4 (A)
      • ptb_b06000.bin_day_var_hocl (B)
      • ptb_a06000.bin_day_var_n2o
      • ptb_a06000.bin_day_var_n2o5 (A, B, C)
      • ptb_a06000.bin_day_var_nh3 (A, AB, C, D)
      • ptb_c06000.bin_day_var_no (C, D)
      • ptb_a06000.bin_day_var_no2 (A, B, C)
      • ptb_a06000.bin_day_var_o3
      • ptb_a06000.bin_day_var_ocs (A, D)
      • ptb_a06000.bin_day_var_sf6 (A)
      • ptb_a06000.bin_day_var_so2 (AB, B)
      MWMAKE also uses a Jacobian for the retrieval of the radiometric offset, but this is simply a constant 1 [nW/(cm2.ster.cm-1)] which is set internally.

      Other Error Sources

      • ptb_a06000.bin_day_co2mix (A, D) Ignoring CO2 line-mixing
      • ptb_a06000.bin_day_ctmerr Uncertainty in CO2, H2O, O2, N2 continuum absorption (25%)
      • ptb_a06000.bin_day_gain Uncertainty in radiometric gain calibration (1%, i.e. these are 1% of the nominal radiance spectrum)
      • ptb_a06000.bin_day_gra Ignoring horizontal gradients (temperature gradient of 1K/100km)
      • ptb_a06000.bin_day_hitran Uncertainties in HITRAN database (not including uncertainties in absolute line strength)
      • ptb_a06000.bin_day_ils Uncertainties in ILS
      • ptb_a06000.bin_day_nonlte Ignoring non-LTE effects
      • ptb_a06000.bin_day_shift Uncertainty in spectral calibration (0.001cm-1)
      In addition, MWMAKE uses the temperature and pressure Jacobian spectra to parameterise the propagation of a 1K, 150 m (equivalent to 2% pressure) temperature, pointing retrieval error into the constituent retrievals.

      Errors due to Gain, ILS and HITRAN uncertainties are assumed uncorrelated between microwindows. Errors due to temperature and pointing are assumed uncorrelated between tangent altitudes.

      Back to Contents


      12. Additional Software

      rfmrd.pro
      IDL procedure to read binary/ASCII spectra
      binasc.f
      Fortran program to convert spectra binary-ASCII or ASCII-binary
      imkmrg.f
      Fortran program to merge RRRRnnnn.imk IMK-format database files produced for each MW into a single database file suitable for input as excluded or included.
      date_and_time.for
      Fortran subroutine for Sun systems replacing FORTRAN 90 function.
      Back to Contents

      13. References

      1. The Revised Microwindow Database Format
        Tech Note PO-TN-IMK-GS-0002 (April, 1999)
      2. ASCII Input Data Interface Control Document (DD53) PO-IF-DOG-GS-0002 (April, 1999)
      3. Dudhia, A., V. L. Jay and C. D. Rodgers Microwindow Selection for High-Spectral-Resolution Sounders App. Optics, 41, 3665, 2002

      Atmospheric, Oceanic and Planetary Physics Home Page