NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Software > ModificationRequest15C306

Changes | Index | Contents | Search | Statistics | Go

Software Fix for Spectrometer "Bad Lags"

Modification Request #15 (C3 2006)

• 1. Introduction
• 2. Background
• 3. Requirements
• 4. Design
• 5. Deployment Checklist
• 6. Test Plan
• Signatures
• CCC Discussion Area

1. Introduction

The "bad lag" problem with the spectrometer is causing observers to lose considerable amounts of data. We need a fix in place to help them recover their "lost" data.

2. Background

The spectrometer suffers from drop-outs in the values for some of the lags. While this problem is being fixed in hardware (see http://www.gb.nrao.edu/GBT/spectrometer/index.html), that fix will not be complete in the near future. To help observers with this problem, we will make a fix in the aips++ and sdfits fillers to allow users to replace the bad lag values with values which closely repreesnt reality.

3. Requirements

The fillers (aips++ and sdfits) should gain a new argument which lets users choose to fix the bad lags in their data. By default, this argument should be turned off.

There are two different types of "drop-outs", and fixing the two problems requires a slightly different approach. In each case, though, the data should be flagged to note that a given integration had lags altered. Additionally, it must be possible for the user to determine exactly which data were fixed.

• 1024 lag drop-outs: Here, an entire chip will report incorrect values. The best way to find these is to search in 1024 lag increments to determine if the mean of a given chip (1024 lags) is different from the mean of the 512 lags on either side of that chip by more than 6 sigma. If it is, the lags should be shifted so that the mean of the "bad lags" is equal to the mean of the 1024 lags on either side of it. The "bad lags" will then also be rescaled so that the rms in the bad segement is equal to the rms of the surrounding good segments. This should recover most or all of the "lost" information. There are 3 special cases. If the first 1024 lags are bad, then those lags will not be fixed and the entire integration will be flagged as bad. In all known cases of this, the offset of the bad segment is quite large and the existing code will find that the zero lag is unphysical and will already flag that integration based on that test alone. In checking the last 1024 lags, only the preceding 512 lags will be used to determine the mean and sigma to use in the check and possible fix of those lags. If adjacent 1024 lag segments are bad then the entire integration will be flagged as bad and that will be noted in the log file. Adjacent bad chips are noticed when the both the mean of the 1024 lags being tested and the mean of the trailing 512 lags differ from the mean of the leading 512 lags by more than 6 sigma. It is assumed that the leading 512 lags are not bad at that point.
  • Special cross-correlation case: This behavior was changed in a subsequent MR. Please see that MR for a description of that change. For cross-correlation lags, the zero-lag is not used to set the van Vleck correction or to scale the data (the zero-lags of the two related auto-correlation lags are used). Consequently, it is possible to adjust even the first 1024 lags when they are bad. In this case, the lags closest to zero may still contain some structure that is not dominated by noise and so only lags 512 to 1023 will be used to calculate the statistics that are then used to adjust the lags. If the mean of those 512 lags (the test lags) differs from the mean of the following 512 lags (the comparison lags) by more than 6 sigma and the mean of the comparison lags is closer to zero than the test lags then the entire first 1024 lags will be judged to be bad. The "bad lags" are shifted so that the mean of lags test lags is equal to the mean of the comparison lags. The "bad lags" are also all rescaled so that the rms of the test lags is equal to the rms of the comparison lags.
• Single drop-outs: Here one (or many) individual lag(s) will be bad. To catch these, the individual lag values need to be compared against the mean value for the 100 lags on each side. Any lag which value deviates from the mean of those surrouding lags by more than 6 sigma (this will have to be tested) would be replaced by the mean value. In this case, information from that lag will in fact be lost, but the overall data quality should not suffer significantly. Here, too, the first 200 lags (again, this value should be tested) should not be considered, as it would be difficult to know if a lag in that range is off. As the lag being tested approaches the last lag, only those lags that exist to the high lag side will be used in the test (so the full number of lags used in the test will be less than 200 lags and at the end only the 100 lags to the low lag side will be used in the test). If adjacent lags appear to be bad, then none of those apparently bad lags will be adjusted and the data will not be flagged as bad. A note to that effect will appear in the log. Adjacent lags that are signficantly different than the surrounding 200 lags are more likely to be real structure than a single bad lag. Any fix for the 1024 lag drop-outs will be applied first before any of the individual lags in that same section of 1024 lags are tested and fixed

4. Design

The changes needed to the aips++ filler and sdfits are very similar as seen by the user.

• There will be a new command line argument to sdfits and the aips++ filler (which will also be available in the dish.import command) called "fixbadlags" to control this behavior.
• When fixbadlags is true ("fixbadlags=T" in the aips++ filler "-fixbadlags" used in sdfits) then the lags will be fixed as discussed in section 3. Otherwise, the behavior will be as it is now - bad 1024-lag segments are flagged as bad, single drop-outs are not flagged.
  • In aips++, flagging proceeds by setting an element or elements in the flag vector. These can be unflagged in dish.
  • In sdfits, flagging here really means blanking the data - setting the data values to NaNs. Eventually sdfits will simply set an element in the data quality column to indicate that the data are bad or that they have been fixed, etc, and then downstream an analysis program such as gbtidl can choose to create flag rules based on that data quality column value. There are still several elements of that design that need to be worked out in conjunction with the data quality project that will be re-starting this summer. Consequently, it is premature to include those elements in this MR. For this MR, if the data are "fixed" there will be no indication of that in the sdfits file. If the data are "flagged as bad" the data are replaced with NaNs and that can not be undone after the fact.
• Each filler will write a log of fixed lags to an ascii file. This file will have the same format whether it is produced by sdfits or the aips++ filler. For sdfits, if the resulting fits file is <project>.raw.acs.fits then this log file of fixed lags will be <project>.raw.acs.fixed_lags. For aips++, if the resulting MS is <project>_ACS then then log of fixed lags will be <project>_ACS.fixed_lags.
• For each spectra with bad lags, the log file will contain: timestamp, scan, sampler, integration, phase_id, channels. Where timestamp is the scan's timestamp (the common name of the backend FITS files), sampler is the name of the sampler, integration is the integration number (counting from 0), phase_id is the switching phase number (counting from 0), and channels are the channels that have been flagged, e.g. (1024:2043) or (567,824,932) or (1024:2043,567,824,932) if there are both bad 1024-lag segements and single drop-outs in the same spectra.
• If a user wants to view the unfixed data, they will need to refill the data from scratch.

5. Deployment Checklist

• The aips++ and sdfits documentation needs to be updated to reflect this new option.
• Notification otherwise can be done through the cycle release notes.
• The sdfits documentation needs to be upgraded as follows, and similar documentation needss to be given to the aips page:

-fixbadlags
          Attempt to find and fix bad lags in ACS data.  The locations
          of the lags that were fixed are written to a log file.  The
          name of the log file is OUTPUT_PREFIX.MODE.acs.fixed_lags.
          This fix should only be used if you know that your data suffer
          from lag dropouts.  Full documentation on how the lags fixed is 
          applied is available at http://wiki.gb.nrao.edu/bin/view/Software/ModificationRequest15C306

    -sigmafactor=SIGMA_FACTOR
          This is part of the fixbadlags option.
          Lags are bad when they deviate from a comparison mean value
          by more than SIGMA_FACTOR times the standard deviation of the
          comparison data around that mean.  This defaults to 6. 

    -spikestart=SPIKE_START
          This is part of the fixbadlags option.
          Single bad lags (spikes or drop outs) are searched for starting
          with the SPIKE_START lag.  This defaults to 200.

6. Test Plan

We will test against a number of datasets with known problems and which are known to not have problems. The data will be examined in lag and in frequency space to insure the resultant data is useful.

Signatures

APPROVED: I acknowledge that my request is fully contained in this MR, and if the SDD delivers exactly what I specified, I will be happy.

ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.

Written	- BobGarwood - 5 Jun 2006
Checked	- PaulMarganian - 5 Jun 2006
Approved by Sponsor	- KarenONeil - 06 Jun 2006
Approved by CCC	- RichardPrestage - 16 Jun 2006
Accepted/Delivered by Sponsor	- KarenONeil - 13 Sep 06

Symbols:

• Use %X% if MR is not complete (will display )
• Use %Y% if MR iscomplete (will display )

CCC Discussion Area

-- KarenONeil - 27 Apr 2006