NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Software > ModificationRequest6C706

Changes | Index | Contents | Search | Statistics | Go

Change scaling for the Autocorrelation Spectra in the Spectrometer

Modification Request #6 (C07 2006)

• 1. Introduction
• 2. Background
• 3. Requirements
• 4. Design
• 5. Deployment Checklist
• 6. Test Plan
  • 6.1 Internal Testing
  • 6.2 Sponsor Testing
  • 6.3 Integration/Regression Tests
• Signatures
• CCC Discussion Area

1. Introduction

The GBT spectrometer records correlation data for both auto-correlation (correlations between the same sampler) and cross-correlations (correlations between different samplers). During their investigation of the cross-correlation for the spectrometer, Tim Robishaw and Carl Heiles found an error in the conversion from lags to spectra for the autocorrelation data as well. The result of this error is that the power scaling of the autocorrelation data is low by a factor of 2.

Here we propose to fix the error in the autocorrelation data. Please note that for all data which is calibrated (e.g. a noise cal was used to determine Tsys), this change is irrelevent, as the calibration will remove any apparant differences in the raw power spectrum. However observers should be aware that data filled before this change will have relative power levels that are 1/2 that from data filled after the change is made. Again, assuming the data is calibrated the difference will divide out in the result. If an observer wishes to have all of her raw data scaled to the same relative power level for all scans she should re-fill the older data using the new sdfits or aips++ fillers.

A more complete description of the problem is given below. To properly produce autocorrelation spectra from the GBT's spectrometer, the following steps should be followed (the cross-correlation case is described in a separate MR:

1. The lags are examined for bad lags and optionally adjusted as described in a previous MR.
2. The zero lag (first element in the vector of raw data values) is used to determine the first positive threshold level for that sampler
3. That sampler level is used to initialize a van Vleck correction interpolation table based on a memo by Fred Schwab.
4. The van Vleck correction is applied and the data are rescaled based on the uncorrected zero-lag (power) level.
5. A full correlation function is assembled containing 2*N lags (where N is the number elements in the original data) by reflecting the data around the zero lag and duplicating the last lag value to preserve a power of 2 in the total number (the subsequent fourier transform is fastest when the array being transformed is a power of 2). If the van Vleck corrected data are d(0), d(1), ..., d(n-1) then the whole correlation function is:
   • WCF = [d(n-1), d(n-1), d(n-2), ... d(1), d(0), d(1), ... d(n-2), d(n-1)]
6. The FFT of WCF is then taken. This results in 2*N frequency values and (N+1) independent values at frequencies 0 to +- N*delta_frequency. The imaginary part is zero to the precision of the calculation because WCF is symmetric about zero. The real part is also symmetric about zero.
7. sdfits and aips++ put the zero frequency value in a separate column in the output filled data set
8. The final spectrum is the sum of the powers at the positive and negative frequencies (recognizing that the power at -N*delta_f is equal to that at N*delta_f). This is a change from the current behavior in which the N positive frequencies (the value at -N*delta_frequency is equal to N*delta_frequency) are incorrectly stored as the filled spectrum associated with the measured lags. It is this change which we will implement in this MR.

2. Background

See the Robishaw and Heiles GBT memo #244, section 4.3.

3. Requirements

The documentation for the sdfits tool needs to be improved. A complete description of how sdfits arrives at each value in the output SDFITS file must be included in that improved documentation. Failure to fully finish the documentation improvements should not delay the release of the code changes described here if they are ready before the documentation work is done.

4. Design

The aips++ and sdfits fillers will be adjusted to do the correct step 8 as described above. This is equivalent to simply multiplying the values at positive frequencies (the current output spectra) by 2. Note that the zero-frequency value, stored separately by each filler, is not multiplied by 2.

5. Deployment Checklist

The sdfits and aips++ online documentation need to be modified to completely describe what is done with auto-correlation data to reduce the chance of confusion.

The sdfits release notes should note the change in the scaling of the raw, uncalibrated data.

GFM will also be affected by the new scaling of the power levels (and other changes made to the shared API). Although GFM is used as a Quick Look application, so this change should not upset observers, it should also be noted in the release notes as affecting GFM.

Finally, notice needs to be given to the GBT community alerting them of the change when it is released, presumably through the GBT email exploder.

6. Test Plan

6.1 Internal Testing

• Using the aips++ filler on auto-correlation data must produce the same result except for the factor of 2 in the scaling values.
• The sparrow sdfits and spectrometer unit tests must continue to pass. The change in iteration order through the data has been done first and any changes necessary in the unit tests have been made.
• Verify that GBTIDL does not care what order the data are found in the SDFITS file. This will be done by filling 2 scans (one position switched pair) with multiple IFs using the current released sdfits and then filling the same 2 scans to a separate SDFITS file using the filler modified as described in this MR. GBTIDL will be used to compare the two files integration by integration, polarization by polarization, and IF by IF to ensure that the values found are identical. GBTIDL will also be used to verify that the calibrated and averaged result for the the 2 scans does not depend on which input data set is used.
• Verify that GFM has not been adversely affected.

6.2 Sponsor Testing

• Using the aips++ filler on auto-correlation data must produce the same result except for the factor of 2 in the scaling values.
• The following is from e-mail received by Bob Garwood from Tim Robishaw on 13 Dec 2006 (the cross-correlation products were later found to fine as described in that MR).

           Bad news... the new version of SDFITS is not producing the correct 
           cross-correlation products.  Auto-correlations are fine.  

• Coross-correlation is fixed, now, too.

6.3 Integration/Regression Tests

• Regular regression testing should be done to verify that the standard auto-correlation cases do not show any new problems.

Signatures

APPROVED: I acknowledge that my request is fully contained in this MR, and if the SDD delivers exactly what I specified, I will be happy.

ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.

Written	- KarenONeil - 31 Oct 2006
Checked	- BobGarwood - 2 Nov 2006
Approved by Sponsor	- KarenONeil - 31 Oct 2006
Approved by CCC	- AmyShelton - 01 Dec 2006
Accepted/Delivered by Sponsor	- KarenONeil - 01 Feb 2007

Symbols:

• Use %X% if MR is not complete (will display )
• Use %Y% if MR iscomplete (will display )

CCC Discussion Area

-- KarenONeil - 31 Oct 2006