NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Data > FlaggingAndBlanking

Changes | Index | Contents | Search | Statistics | Jump to Topic

Previous Version of Document

• Data Editing for GBT Standard Observing Modes
  • 1. Introduction
    • 1.1 Definitions
    • 1.2 Types of Data Editing
    • 1.3 Data Objects and Parameterization
      • 1.3.1 Basic Continuum (Pointing, Focus, Tipping)
      • 1.3.2 Spectral Line
      • 1.3.3 Imaging
      • 1.3.4 Advanced Continuum (Imaging, Penn Array)
  • 2. Workflow through GBT System
  • 3. Storing Data Quality Information
    • 3.1 Quality information stored with unprocessed raw data (engineering FITS files)
    • 3.2 Quality information stored with pre-processed raw data (exported, unprocessed SDFITS files)
    • 3.3 Quality information stored with post-processed data (processed SDFITS files)
  • 4. Impacts to Analysis Procedures
    • 4.1 Handling of Blanked Data
    • 4.2 Handling of Flags When Averaging Spectra
  • 5. Other Schemes
  • 6. Scenarios
    • 6.1 Handling of bad lags from the spectrometer
    • 6.2 Averaging multiple spectra where flagged areas differ between them
    • 6.3 Need another test case! maybe for correlation, not strict selection
  • 7. Implications for Systems Development
• Design of Flagging and Blanking for GBTIDL
  • • 1. Blanking in GBTIDL
    • 2. Flagging
      • 2.1 Setting Flags Based on Strict Selection
      • 2.2 Setting Flags Based on Correlation with Ancillary Data or Applied Algorithm
      • 2.3 Undoing Flags
    • 3. Storage of Flagging Information
      • 3.1 Flag File Structure
      • 3.2 Commenting out Flags
      • 3.3 Querying Flags
      • 3.4 Selectively Using Flags from Retrieval and Averaging Procedures
    • 4. Examples
    • 5. Issues not Addressed by the GBTIDL Scheme

Data Editing for GBT Standard Observing Modes

1. Introduction

It is often necessary to be selective when analyzing a data set, that is, to account for missing, bad, or irrelevant data prior to applying algorithms. Data quality issues can be caused by collecting data when it should not be collected (e.g. the antenna is off source, or a switching signal is transitioning), by radio frequency interference (RFI), or other faults that produce intermittent time or frequency-dependent bad data.

Policy and requirements for both automatic and manual data editing are critical not only for today's scientists and data analysis, but also in support of a pipeline heuristics system that will guide the production of high-quality data from a future GBT postprocessing pipeline. To obtain science data products which are of the highest quality, pipeline production requires that invalid data is handled appropriately and automatically, and that good data is not thrown out. This process can be complicated because data quality objectives will vary depending upon frequency and observing mode, observer's intent, and other factors. As a result, the scientist's constraints on a Scheduling Block, pipeline heuristics, and the flagging/blanking strategy are closely intertwined.

The purpose of this document is to define the requirements, and outline design issues and constraints for the implementation of data editing. Once complete, this document will be formally reviewed. Critical success factors for this document are as follows:

• To define the global view of flagging and blanking for the GBT, and identify how an implementation of flagging within GBTIDL supports long-term goals.
• To identify how real operational issues like the spectrometer bad lags problem will be addressed, and how similar issues will be handled as they arise in the future.
• To outline the relationship and intended uses for manual flagging versus autoflagging.
• To establish how flags should be handled in the analysis software for common post-processing operations.

In this document we give a general description of the concepts associated with data editing as they pertain to the GBT. The attached document, "Design of Flagging and Blanking for GBTIDL," provides the model to be employed in GBTIDL for the final stage of the process where the researcher manually generates flags to use with the processed data.

Automatic determination of the flagging criteria, depending on the data quality and project's goals, is an important long term goal but beyond the scope of this analysis. Similarly, this analysis does not currently address data editing for VLBI, radar, or pulsar data.

1.1 Definitions

The following definitions are used throughout this document, in order of their initial appearance in the dataflow:

Control System Blanking: The process of selectively capturing or irretrievably discarding information during raw data production, performed in the control system hardware (e.g. automatic RFI extraction machines) or software.

Pre-Processing: Any restructuring of the raw data prior to being written to an export data format. (For example, detection of bad lags in the spectrometer raw data would be considered a preprocessing task, because it is done prior to SDFITS generation.)

Data Capture: The process of combining raw data into other representations of raw data without transforming that data in any way. (This is currently handled by "filler" processes such as the online SDFITS filler and gbtmsfiller. In the future, data streams from the control system will be combined, according to the science data model, to form export data suitable for archiving.)

Post-Processing: Any manipulation of the data that is done after the raw data is collected, such as calibration, averaging, etc. Post-processing changes the nature of the data, whereas pre-processing and data capture do not.

Post-Processing Blanking (also called Masking): Discarding data during the offline analysis process. In the latter case, blanked values cannot be undone easily because they are replacement values, stored as data. The only way to undo a blanked value is to return to the original source of data, and in the case of a spectral line dataset, reload the (possibly uncalibrated) spectra.

Processed Data: A data file that has been transformed from the original raw state. For example, a calibrated representation of a data set would be considered a processed file. Reading and rewriting a raw data file row-for-row is not considered a processed file. The reason to distinguish processed data files is that the user may wish to take a first pass at calibrating the raw SDFITS data, and write out the result of that first step as a new data file -- the "processed data". Then the processed data could be inspected and used for flagging and additional averaging.

Record: Describe this without defining SDrecord; in other words, talk about its content and not its firm structure.

Flagging: Flagging is the process of marking, but not replacing, various data objects without actually modifying the entries in the exported/archived data. Flags are stored separately from the data, and can be undone easily. The primary purpose of flagging is to identify data that needs to be excluded from operations within the data analysis software or automated pipeline, such as calibration and averaging. As such, flags are attached primarily to raw data and data that have not yet been averaged. For spectral line data, flagging can be performed with respect to various parameters, such as spectral channels, integrations, or entire scans.

It is possible to carry flagging information along through the entire data analysis process, even after averaging (or other operations) have occured. It is not possible to "reverse" later stages, but the flag data can indicate that at least some data which contributed to this data-point was flagged. This could be useful - e.g. to indicate errors in the averaging algorithm. If all the "funny" processed data is associated with data for which some input was flagged, you might get suspicious. We should consider if we want to do this.

When data requires flagging, an iterative approach to reduction is often required. Here is a typical scenario:

1. Some raw data is automatically blanked or flagged by the data capture process.
2. Calibrate the (remaining) raw data.
3. Examine the calibrated data and determine whether any flagging is required to improve the calibration.
4. If necessary, flag the offending data and return to step 2.
5. Store the calibrated data.
6. Examine the calibrated data (e.g. compate successive data sets) and determine further flagging requirements.
7. Perform the final stages of data reduction, for example averaging of calibrated data.
8. Again examine the reduced data, and if necessary return to step 2 or step 6 to modify the flagging commands as necessary.
9. Proceed with the analysis of the final, reduced data.

Most flagging is anticipated to take place on uncalibrated data prior to raw data being combined to form processed data. However, it is not forbidden to flag calibrated, averaged data. The user should use caution in such cases because the header parameters used in the parametrization of flags can be changed during averaging operations, and so may not apply correctly to averaged data sets. The header parameters should not become invalid as a result of flagging, however; the software should take care of this. For this reason, when flagging averaged data it is generally best to flag by record number rather than using the alternate parametrization. In the procedure outlined earlier in this section, flagging in Step 3 should be parametrized by scan, polarization, etc. while flagging in step 6 should be parametrized by record number.

1.2 Types of Data Editing

There are many different terms for the selective reduction of the total solution space of data that is possible from an observation. These can be classified by location in the process that generates the raw data, by the mechanism that produced the flag, and by the technique used to generate the flag.

By location in the data flow:
1. Control System
   • Blanking Only.
   • Completely Irreversible.
   • Operations are performed in the hardware/firmware or in the control system software. When performed in the hardware, "clean" data are written to the raw data files (e.g. data produced by spectrometers with automatic RFI detection and cleansing algorithms). When performed in the software, data that are or could potentially be recorded are simply not stored (e.g. not recording data that is outside the boundaries of an observation/scan).
   • In either case, the incoming data is irretrievably lost, thus the process is irreversible.
2. Pre-processing
   • Flagging Only (since intent is to fully automate quality checks at this level, process must be reversible).
   • Reversible by ignoring flagged information in the Data Capture/Pre-processing stage.
   • Would include the checks required for Ka-band as specified in ModificationRequest12C705.
   • Flags produced must be useful, in near real time, in the quick look pipeline.
   • An example of flags produced at this stage would be antenna off-source flags.
   • Storage of these flags should be within the collection of engineering FITS files, and could be in its own separate group of FITS files.
3. Data Capture
   • Flagging Only (since intent is to fully automate quality checks at this level, process must be reversible).
   • Reversible through regeneration of the raw export data (SDFITS file).
   • Should handle all cases which require looking at data in a form different than what is required by the Science Data Model (e.g. examining spectra in lag space instead of frequency space).
   • Storage of these flags should be within the export data (SDFITS file).
4. Data Post-Processing
   • Blanking or Flagging.
   • Fully reversible by the end user.

By the mechanism that produced the flag, and where in the raw data generation process it was produced:

1. Pre-processing flags. Automated data checks are run at the time of data production during the data capture process, such as for detection of non-physical results (e.g. bad lags in the spectrometer), to note when the LO was locked, etc. As a result, this information can be stored as part of the raw data that is created as an observation executes.
2. Post-processing flags. As a result, this information is stored separately from the dataset (for example, in a flags file) at which point the observer or data analysis must be responsible for its storage.
   • Flags generated by algorithms/autoflagging procedures accessible from the data analysis software.
   • Flags generated manually by the user.

By technique:

• Strict selection (e.g. flag all data for a particular IF, within a channel range, for a particular IF and within a channel range, and so forth).
• Correlation with ancillary data (e.g. flag all entries that correspond to times when the system temperature was too high).
• Application of an algorithm (e.g. flag all entities for which the rms noise exceeds a certain threshold).

Regardless of where in the raw data generation process the flag was raised, the mechanism by which the criterion is tested and the flag is written should be the same in the software.

1.3 Data Objects and Parameterization

1.3.1 Basic Continuum (Pointing, Focus, Tipping)

Note from Richard: Pointing scans can suffer from RFI just as badly as anything else. On the other hand, the data are currently median filtered, and the data analysis process (fitting functional forms with only a few free parameters) is fairly robust to some bad data, so this isn't a high priority. But it shouldn't be considered any differently from any other type of data.

1.3.2 Spectral Line

To follow the scheme used throughout the single dish data model, flags will be parameterized according to:

• scan number
• integration number
• polarization number
• IF number
• beam number
• channel number (spectral resolution element)

A second parameterization will be allowed as well: Data can be flagged according to record number, and channel number. Both parameterizations should be possible in any anlaysis package. Averaging, analysis, and display procedures in an analysis package must then respect blank values and flags and take the appropriate action when blanked or flagged data are encountered.

What is the relationship between "record number" and these other quantities. Remember we are not in the GBTIDL section yet...

1.3.3 Imaging

Flagging by data entity or pixel or both? Or do we just defer to whatever method is chosen by new AIPS++? Is there a difference in how the flagging is done between the two types of mapping? What do we want to do about users wanting to flag by clicking on an image?

1.3.4 Advanced Continuum (Imaging, Penn Array)

Continuum Imaging, Penn Array, or else explicitly note that the flagging solution doesn't cover these cases. That would be a bit of a shame but might be pragmatically necessary...

2. Workflow through GBT System

The workflow for flagging and blanking through the GBT will be very similar to that for the EVLA and ALMA, with one exception: the GBT currently uses a file-based control system, as opposed to operating a full data capture process. In the steps below, "data" refers to one of the data objects referred to in Section 1.3.

1. Control System Blanking: The M&C system determines that data is unusable for all scientific intentions, and should not be stored as raw data.
2. Pre-Processing Flagging: The M&C system determines that an error is present, but the data should still be stored as raw data. These flags are used by the quick look system.
3. Data Capture/Pipeline-Driven Flagging: The data capture process determines that an error is present, but the data should still be stored.
4. User Flagging/Blanking: The researcher determines that there is an issue with the data, and attaches appropriate flagging information to the exported data, or alternatively eliminates sections from the export data which are unusable for his or her scientific intent.

INSERT SCHEMATIC HERE

3. Storing Data Quality Information

3.1 Quality information stored with unprocessed raw data (engineering FITS files)

Checks done at the control system level must be recorded in the raw data.

I think this should be done by producing QualityLog.fits files that contain quality information which can be used by GFM or data capture wherever necessary. NMR

Requirements for storage?

3.2 Quality information stored with pre-processed raw data (exported, unprocessed SDFITS files)

At the data capture stage, raw data (now stored as engineering FITS files, but to be data streams in the future) are combined according to the science data model and srored as export data. Currently, GBT data is exported in the SDFITS format. This SDFITS file must contain a representation of all the data quality issues that were uncovered during the production process for two purposes: a) to guide the quick look pipeline, and b) to provide guidance to the researcher. A good example of the type of problem that should be recorded in the SDFITS file are indications of bad lags coming from the GBT spectrometer.

I think this is the QUALITY column, or related, in the SDFITS file. NMR

Requirements for storage?

3.3 Quality information stored with post-processed data (processed SDFITS files)

What are we going to do when the observer has QUALITY info in the SDFITS file, and extra flags that he/she has added? What if they want to store multiple flags files?

Requirements for storage?

4. Impacts to Analysis Procedures

One of the main drivers for masking and flagging is to be able to correctly average data together. For example, when flagging data on the basis of whether the antenna is on source, the integration time must be adjusted appropriately. Then, when different integrations are averaged together, this must be done with the correct weights (proportional to Tsys*sqrt(integ time)) applied. This section describes how ancilliary data (effective integration times, weight arrays, etc.) will be modified when data editing is performed, and how data processing operations will correctly take this into account.

Who can put in additional info?

4.1 Handling of Blanked Data

The following analysis procedures will be impacted by blanked data in any offline data reduction package:

What types of analysis commands will blanking affect, and how will the blanked data be handled? Instead of listing the GBTIDL commands here, describe the GBTIDL command and then describe how that command will behave when it encounters blanked data. Jim's original text, which I'd like to see turned into a complete table below: Certain analysis procedures will require special cnsideration in dealing with blanked data. For example, the add procedure will need to consider how to add a blanked and a non-blanked data value. Either the result is itself a blank, or the result is the non-blanked value. Each analysis procedure that requires such consideration will need a parameter to give the user flexibility in handling blank values. We will need to consider each analysis procedure on an individual basis to determine the default behavior as well as the options.

YES! This is a great section. This is the meat of the document, and needs to be spelt out in detail. It might be tedious, but I think it is required, and well worth it. RMP

Description of Analysis Procedure	Default Handling of Blanked Data	Options
Add	When a blanked and non-blanked value are added, the result will be ???	The user should be able to specify an optional parameter for ??? which changes the default behavior to ???

4.2 Handling of Flags When Averaging Spectra

bla

5. Other Schemes

Describe how other telescopes/packages handle flagging and blanking, pros & cons, & why we need/don't need

AO/Parkes, JCMT, AIPS++/DISH, AIPS, ALMA SDM, EVLA, VLBA

6. Scenarios

6.1 Handling of bad lags from the spectrometer

The goal of this scenario is to automatically detect when a) an entire chip in the spectrometer (1024 lags) has gone bad, and b) when chips are operational but individual lags are bad.

there are two different algorithms that need to be in place:

(1) this looks for when an entire chip (1024 lags) or chips has gone bad. In this case the data should be flagged and not used. To find these cases, one needs to put a 1024 lag "filter" on the data, and compare every 1024 lags with the median of all other lags. The medians should always be the same, at least if the first ~100 lags are ignored (and maybe even if they are included). If the median is not the same then those data should be flagged. (Note that we should check this - i may be that you need them to be the same w/i 1 sigma, or something like that)

(2) the 2nd algorithm would look for the case where only a couple lages have gone bad. this is the case where those lags can just be replaced by the mean of the lags surrounding it. In this case the algorithm should just comparee very lag with data 10 or so lags away (or perhaps a median filter?), anything that is off by more than a few sigma should be replaced. A couple thoughts on this: (1) we'll have to play to see exactly how many sigma "a few" is; (2) this data should still receive some sort of flag so that the observer knows this has happend; (3) the first 100 or so (TBD) lags should not be considered in this algorithm

One last thought - the obsrver shold, of course, have the option of turning at least algorithm (2) off, so that flaggin still happens but the data isn't changed.

6.2 Averaging multiple spectra where flagged areas differ between them

6.3 Need another test case! maybe for correlation, not strict selection

7. Implications for Systems Development

1. At some point, the control system will need to be modified to allow data taking while the antenna is activating, or alternatively, without requiring scan-based synchronization between the GBT's component devices. This is so that off-source data, which will be valuable for many types of observing, is not lost. Additionally, some high-frequency observations will only require seconds on each source, and the observe time will be largely dominated by latency in the scan startup and slew times. At these times, valuable information could be collected if individual devices could collect and store data.
2. A general system should be put in place so that quality checks can be performed on any device and/or observation, and the results stored along with the raw data. This should be externally configurable so that project scientists can own the data checks, not softare engineers. This means that the project scientists will be able to fine tune their quality checks without any software development work (consider a generalized case of ModificationRequest12C705). Additionally, this quality information should be accessible not only to quick look but to any downstream applications.
3. This general system should somehow be able to apply user-defined data quality rules in addition to rules defined by the project scientist.

Design of Flagging and Blanking for GBTIDL

This design information for flagging and blanking in GBTIDL assumes the following:

• Only data editing for spectral line data is to be implemented in GBTIDL, so the only data object to be operated on is the spectrum.
• Incoming datasets can be either raw or calibrated, so long as they contain spectra.
• "Blanking" in the following discussions refers to elimination of data objects which can be restored by reloading the dataset.
• "Flagging" refers to manual operations which can be reversibly executed by the user, the results of which are stored alongside the dataset.
• GBTIDL requires that incoming spectral line data be in frequency space (with FFTs already completed).

1. Blanking in GBTIDL

Blanking works on data in memory, replacing selected data values with NaN's. The "replace" procedure takes a "/blank" keyword to replace user-specified channel ranges with NaN values. The procedure will have the following structure:

  replace, bchan, echan, /zero, /blank

Note that the default behavior of "replace" is not to blank, but to interpolate the data values. More sophisticated blanking procedures can be implemented on top of "replace" in a straightforward way, if the user wishes to blank data based on selection criteria.

After blanking is applied, all other GBTIDL analysis procedures will handle the blanked data appropriately, and no additional user interaction is necessary. note what

As an example of special handling of blanked values, consider the "show" procedure in GBTIDL. "Show" will handle the special values by putting gaps in the plotted spectrum at the locations of blanked data. The "stats" command may apply blanking by simply ignoring any blanked channels in computing the rms. The "hanning" command will blank channels in the smoothed spectrum whose constituent channels are themselves blanked. In general, a procedure will have to know how to take the appropriate action when it encounters blanked data, and the appropriate action varies depending on the procedure.

replacing spectral intensities for a given set of channels with a special value recognized by the data analysis system. When the data analysis system encounters the special value, it must know to handle the requested operation in an appropriate manner. In the case of GBTIDL, the special value is "NaN", which means "Not a Number".

Not saying it isn't worthwhile, but the above seems to simply replicate the purpose of flagging. Is the intent of supporting what is described above to make operations run faster, or cut down on storage requirements, or what?

I had assumed that this level of blanking would provide an interpolated value, so that e.g. plots could be produced without auto-scaling being thrown off by RFI points, whereas flagging would show "gaps" in the spectrum. We should clarify the desired behavior.

2. Flagging

2.1 Setting Flags Based on Strict Selection

Flags can be set from the command line with the procedures flag and flagrec. The result of these procedures is a new entry in the flag file. The flag procedure will have the following structure:

  flag, scan, intnum=intnum, plnum=plnum, ifnum=ifnum, fdnum=fdnum, 
        bchan=bchan, echan=echan, idstring=idstring

and the flagrec procedure will have the following structure:

  flagrec, record, bchan=bchan, echan=echan, idstring=idstring

Examples: The following example shows how to flag a channel range for a small, select number of scans and integrations. Note that the scan parameter is required, but otherwise if a parameter is not specified, "all" is assumed. So in this example, all polarizations are flagged. flag, [18,19,20], int=[1,3], bchan=512, echan=514, idstring="RFI" The next example shows how to flag all channels for a given integration in one scan: flag, 15, int=3, idstring="spectrometer glitch" The next example shows how one could flag all data for the given three scans: flag, scans=[101,105,107] Finally, the next example shows how one might flag a record in a processed data file: flagrec, 15, idstring="Glitch"

2.2 Setting Flags Based on Correlation with Ancillary Data or Applied Algorithm

It is possible to develop more sophisticated blanking and flagging procedures, including procedures that automatically detect and flag bad data. For example, we could provide an "autoflag" procedure:

  autoflag, bflag, eflag, chan_clip, int_rms, idstring

Such a procedure would use the bflag and eflag parameters to flag channels at the beginning and end of all spectra. The chan_clip would flag, within each integration, channels that exceed the rms by the specified value, and the int_rms would flag integrations whose rms exceed the given value. One could imagine this routine becoming more sophisticated, if necessary.

Another flagging mode would use a visual aid, in the style of the classic AIPS flagger. A dynamic spectrum image could be used to represent the data, and the user could then interact with the image to generate flagging commands. Before this tool is developed, some assessment of its relative merit should be made.

Another flagging procedure would allow the user to specify flags according the frequency or velocity, rather than channel number.

I think it would be good to describe how to flag data on the basis of the antenna being off source here. Also, need to describe somewhere how header data will be modified correctly.

2.3 Undoing Flags

Flags can be unset by removing or commenting the flagging command from the ".flag" file. So, one way to unset flags is to edit the flag file directly. It will also be possible to unflag lines based on the string identifier, using a GBTIDL command unflag. The unflag command takes a single parameter, idstring, and it comments out all flagging commands that have that idstring.

  unflag, idstring

3. Storage of Flagging Information

3.1 Flag File Structure

would be useful to briefly review all of SDFITS's file usage here. For example, when "processed" data is written out, where does it go, and how does the flagging information go with it.

I think it is dangerous to expose the flagging information to the user in this way. If they go and edit the flag file directly, how do you know that header information remains up to date?

The flag file associated with a given sdfits file (e.g. "myfile.fits") will have the ".flag" suffix ("myfile.flag"). If no flag file is associated with a given sdfits file then it can be safely assumed that none of the data in that file are to be flagged. Flag files will be ASCII files with one flagging instruction per line. Each line will have the following entries:

records, scans, integrations, polarizations, IF's, beams, bchan, echan, idstring

The flagging file simply represents the parameters used in the flag or flagrec procedures. Note that the parametrization has some redundancy. If the user flags by record number, then the set of [scans, integrations, polarizations, IF's, beams] is not necessary. So when a record number or numbers are specified in a flag entry, this set of numbers is ignored.

3.2 Commenting out Flags

Flags can be commented out by placing a semicolon as the first character in the flag entry. The relevant flags are then not applied by the data analysis system.

3.3 Querying Flags

Because the flag file is a standard ASCII file, flags can be queried using standard Unix tools.

For example, to list all the flags with idstring="RFI" one could do this from the unix prompt: grep RFI mydata.flag There will also be tools in the gbtidl environment to list and query flags as follows: listids : list all the idstrings in the flag file attached to the input dataset listflags, idstring : list the flag details associated with all entries matching the given idstring

3.4 Selectively Using Flags from Retrieval and Averaging Procedures

By default, GBTIDL procedures that use flags will apply the flags when they are available. However, each procedure that may use flags should also have a /noflag keyword to allow the user to avoid the application of flags for a given operation. So for example:

  getnod, 15, /noflag

All procedures that retrieve data from disk, notably all the get* family of procedures, will need to have the /noflag option. The averaging procedures will also need the /noflag option.

4. Examples

   flag, scan=[18,19,20], int=[1,3], bchan=512, echan=514, idstring="RFI"

*,[18,19,20],[1,3],*,*,*,512,514,"RFI"

   flagrec, record=15, bchan=0, echan=8, idstring="bad channels"

15,*,*,*,*,*,0,8,"bad channels"

5. Issues not Addressed by the GBTIDL Scheme

The weighting of data required by averaging operations is a calibration issue as much as a flagging issue. So, weighting is not fully addressed in this document. However, the user should be aware of some potential issues.

Consider two reduced spectra, A and B, which resulted from an average of flagged data. In each of the two spectra, the individual channels have been flagged to different extents, so the final noise in each channel is different depending on how much of the raw data were flagged going into the average. For example, channels 0-10 in A may have been heavily flagged prior to averaging, and so they contain a higher noise than the other channels in A. If the observer then wishes to average A and B, the weighting in the average will be wrong because relative weights have not been stored for these spectra.

I think this document has to somewhere address weighting, and the correct handling of other header information (e.g. integration times). The first version of GBTIDL doesn't necessarily need to implement everything in the design, but the design needs to be thought through.

-- JimBraatz - Revised 12 Oct 2005 -- NicoleRadziwill - 18 Oct 2005