NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Software > PlanOfRecordC82005 > ModificationRequest16C705

Changes | Index | Contents | Search | Statistics | Go

Add Support for CCB to GFM

Modification Request #16 (C7 2005)

• 1. Introduction
• 2. Background
• 3. Requirements
  • 3.1 Specifications for Pointing and Focus Plugins
  • 3.2 Requirements for "DCR Plugin"
• 4. Design
  • 4.1 Overview
  • 4.2 New Base Classes
  • 4.3 Revising the DCR API
  • 4.4 CCB Specific
  • 4.5 GFM
    • 4.5.1 Pointing & Focus Plugin
    • 4.5.2 Generic DCR Plugin
• 5. Deployment Checklist
• 6. Test Plan
  • 6.1 Unit Testing
  • 6.2 System Testing
  • 6.3 Regression Testing
• Signatures
• CCC Discussion Area

1. Introduction

Since GFM is the real-time display tool for continuum data and the CCB is a new continuum backend, GFM must be updated to include data processing capabilities for the CCB.

2. Background

The FITS specification for the CCB is available at http://www.gb.nrao.edu/GBT/MC/doc/dataproc/gbtCCBFits/gbtCCBFits/gbtCCBFits.html.

The GBT FITS Monitor (GFM) provides sub-scan-based display and analysis of GBT data, either in real-time as the continuum data is being collected, or in an offline mode where it can be used to simply step through the sub-scans from an observation for continuum observations.

This MR should be implemented while keeping in mind (but not necessarily implementing) the data reduction vision that Frank created for all data reduction programs.

• The GFM bible (articulates the Vision thing-- not reqmts but, to bear in mind)
  • New Testament= http://wiki.gb.nrao.edu/bin/view/Data/WebHome?topic=GbtFitsMonitorKA_DCR
  • Old Testament= http://wiki.gb.nrao.edu/bin/view/Data/GbtFitsMonitorKA_DCR?topic=DCRDataDecoding

3. Requirements

GFM CCB support is meant to work both online and offline, within astrid, and within the stand-alone gfm application. I break the needed functionality down by plugin-- pointing/focus, and "dcr" (continuum). tipping plugin not presently specified. No calibration is presently specified-- it's all a form of Raw mode.

3.1 Specifications for Pointing and Focus Plugins

At present I specify only a raw mode for CCB processing. The goal is to form a quantity "y" which is used in GFM processing-- it is the "data" quantity that you are fitting to (ultimately the y axis of your GFM plot display, proportional to Antenna Temperature). The goal is to be able to use GFM online as well as offline to derive pointing and focus data with the GBT (thus, the requirements below must be implement at least for the pointing and focus plugin; it would be nice if the DCR plugin also displayed the data).

• There are up to 16 input ports to the CCB. We will select one for processing. Select the one which has the polarization and feed (in the SIG state) which the user desires. The field must be further narrowed by frequency. Frequency selection will be accomplished by the following mechanism
  • When the backend is CCB26_40, there will in addition to polarization be a "frequency selection menu" available under Tools->Options->Data Processing. For CCB26_40 the frequency options will be 27.75 GHz, 31.25 GHz, 34.75 GHz, 38.25 GHz. Amy will investigate making her own list of valid frequencies from the unique set of freqs in the IF table; if that is unduly complicated it is acceptable to hardware the above list for CCB26_40.
  • The choice of frequency will be either linked between plugins (eg point plugin, focus plugin), or will be independent between plugins-- this should follow the same scheme as is presently the case for the "polarization choice". (I think the selections are independent between plugins)
  • The frequency channel selection should be specifiable (for all plugins) from the user's dot-sparrow file.
• Associate antenna positions with each integration of the selected data in the usual fashion.
• Individual phases of each integration are flagged as overflow or no overflow. For a given integration, form the logical OR of all flags. flag the whole integration as overflowed or not; if it is overflowed, don't use it in subsequent processing. This information is in the OVRFLOW column of the CCB DATA table. Note that the OVRFLOW column has 2 dimensions, integration and phase, and what we will do here is: for each integration form the logical OR across all phases for that integration for an overall "integration has some overflowed data" flag.
• The CCB FITS file flags individual integrations as stable or not. Don't use integrations that aren't stable. These flags are in the USABLE column of the CCB DATA table.
• The CCB FITS file also flags individual integrations as CAL ON when the cals are on. Don't use any integrations where the CAL is ON. This information is in the CALA and CALB columns of the CCB DATA table.
• The previous three steps have identified one or more integrations which we don't want to use in subsequent processing. Note that in dropping these integrations we must not mess up the association of the data with antenna positions. So for instance if we have formed two arrays of N elements each-- one array of CCB data and one array of antenna positions-- if the three previous steps say we want to drop integrations 3, 7 and 105 of the CCB data array, then we must also drop elements 3, 7 and 105 of the antenna position array.
• Now form the actual data "y" to process:
  • If zero phase switches are active then there is only one phase in the data-- use that for y.
  • If one phase switch is active and one phase switch is not active...
    • If the non-active switch is in state "1", then y=p1-p0
    • if the non-active switch is in state "0", then y=p0-p1
  • If two phase switches are active, y=(p00+p11)/2 - (p01 +p10)/2
  • My terminology is as follows. pxy refers to the phase in which phase switch A is in state x and phase switch B is in state y. px refers to the phase which has the active phase switch in state x.
  • The values of the individual switches in each individual phase of an integration can be determined by the PHIA and PHIB columns of the CCBSTATE table.
  • Which switches were active can be determined in either of two ways:
    • The ACTPSW keyword of the primary Header, which has valid values of "A", "B", or "Both"
    • An active switch will have different values in different fows of the CCBSTATE table. So if all rows of PHIB have the same value in the CCBSTATE table, PHIB is not active. And if at least one row of PHIB has a different value from the others, then PHIB is active.
• Fit "y" for peak/focus offsets as usual. The integrations which have been flagged should neither be included in the data that are fitted, nor displayed in the peak/focus plot.

Further comment on the integration dropping... since you have potentially dropped samples from the y-axis (data) you should be careful in a) inferring the correct x-axis (time or antenna position offset) values; and b) fitting. I doubt that either of these assumes regularly spaced data in either axis but beware. A sensible way to do this is to first associate times and antenna positions with the full set of CCB data and then drop the CCB data samples that you don't want due to the reasons listed above (a cal diode was on, or the integration was not stable, or one or more phases of the integration were overflowed).

Don't forget to look up the correct (L1C etc) Tcal values. Except: we aren't using Tcals at this point so forget about it!

3.2 Requirements for "DCR Plugin"

Assume that the DCR plugin is becoming a "generic continuum plugin". It should display CCB data.

• You should be able to choose polarization and frequency in the same manner as is done in the pointing and focus plugins.
• You should also be able to choose the beam(s) (1,2) to show.
  • The frequency, beam, and polarization selections should follow the look/feel/philosophy of the equivalent selections for DCR data in the DCR plugin.
• User should be able to choose SIGNAL or REFERENCE phases in the same fashion as is done for DCR. In this case however SIG and REF are formed in the fashion above-- ie, if there are two phase switches switching you must average the two SIG phases appropriate given the phiA and phiB columns, etc.
• Note: it is possible that one frequency channel may exist for one beam but not another. If I have selected "Show all frequency channels" and "Beam 1, L poln" that is in principle 4 samplers but one could be missing (Beam 2, or R poln of Beam 1) may well have them all. That is ok. If one of the sought channels doesn't exist, GFM should simply print a warning like "WARNING: Beam 2, L polarization, 27.75 GHz data not found" and proceed. Best not to add a legend to the plot for this unfound channel.
• User should be able to choose whether to show: cal off integrations, XL cal on integration, YR cal on integrations, both cal on integrations. (Note: the CCB Data table records Acal and Bcal status; A and B are mapped to XL and YR via the primary header keyword "AisXL"). If all integrations are shown then individual integrations should not be distinguished from each other (visually or otherwise) on the basis of what cals were on. If cal on integrations are shown separately the points should not be connected by lines. These should be overplottable selections-- ie I should be able to see, all at once and in different colors: cal off integrations, L cal on integrations, and both cal on integrations. If that's not possible-- ie if you can only see one selection at a time-- then we also need a "all integrations" option.
• The user should, separately, be able to toggle whether to show stable integrations only, or all integrations (this is the USABLE column of the CCB data table). This should be a toggle: you get all the integrations, or only the stable ones. You can't plot both selections at once.
• Note: for some reason the DCR plugin presently only shows one phase at a time. I would think that either the nature of the buttons should be changed (pushing Sig/Off deselects all the others) or, all the checked phases should actually be shown. Ditto for polarizations.

4. Design

4.1 Overview

The outline of what the new backend classes will look like is shown in the backend overview graphic and then refined in backend details graphic. Additional helper classes can be found in the helper classes graphic. The existing DCR classes will have to be refactored to conform to the new design while maintaining backward compatibility; this effort will be helped by the unit tests for the existing DCR classes. The new CCB classes will essentially mirror the refactored DCR classes. No refactoring will be performed on the spectral line backend classes.

4.2 New Base Classes

Code common to both the DCR and CCB will be bubbled up from the various DCR classes to the appropriate base classes (ContinuumBackend, ContinuumInput, and possibly BackendData and Phase. Backend, SpectralLineInput, and Input will be skeleton base classes until the refactoring effort is extended to include spectral line backends.

4.3 Revising the DCR API

There are lots of files in the DCR API that don't have to be DCR specific. In fact, they should be generalized to use any available continuum backend. These files are: AzimuthScan.py, ElevationScan.py, FitDualGaussian.py, FitDualGaussianWithBase.py, FitFWSingleGaussianWithBase.py, FitPolynomial.py, FitPolynomialTests.py, FitSingleGaussian.py, FitSingleGaussianWithBase.py, FittingStrategy.py, FocusScan.py, OffsetScan.py, PointingScan.py, pointingTable.py. Any specific references to the DCR should be replaced with a call to a "backend", which will be of type ContinuumBackend. The fitting files should be placed in their own directory under sparrow/api called fitting. The C++ fitting library from DEAP will also be moved from DEAP to the new fitting directory. The *Scan files should be placed in a new directory under gbt/api called "scans". The file, Phase.py, should be placed in the new directory, "backend", mentioned in the previous section.

There are also files in the DCR API which can be removed because they are no longer used; these files are: Display.py, OffsetScanFactory.py, and TPoint.py.

There is one file that needs to be renamed. Input.py should be DcrInput.py.

4.4 CCB Specific

4.5 GFM

4.5.1 Pointing & Focus Plugin

These should be revised to point to the correct object (DCR or CCB) based upon which backend is present in the data. The CCB and the DCR will never be used at the same time due to physical restrictions. Other than that, the plugins should mostly use the generic stuff extracted to the backend API whose details are in the "Revising the DCR API" section above. So there are not many changes that have to happen here. Basically, anything DCR-specific should be generalized to any continuum backend. This is pretty much already done in the code, but DCR-specific variable names will have to change.

4.5.2 Generic DCR Plugin

This plugin needs to become the "Continuum" plugin and be able to handle both DCR and CCB data (and any other continuum backend for that matter). Again, this is pretty much set up to accept the new CCB classes but any DCR-specific references/variables should be generalized.

5. Deployment Checklist

• Email gbtlocal if this goes in as a patch rather than in a regular release.
• Include description of changes in release notes.
• Update GFM documentation (http://wiki.gb.nrao.edu/bin/view/Data/GbtFitsMonitor)
  • DCR plugin is now the Continuum plugin
  • Update any references that assume the DCR is the only continuum backend.

6. Test Plan

6.1 Unit Testing

The existing DCR unit tests should still work after the refactoring of the code base. Unit tests for the new CCB code will be created.

6.2 System Testing

Pointing offsets derived should be close to zero on the assumption that DCR peak scans 3 - 6 immediately preceeding (which can also be used for comparison), effectively installed them.

Before the end of December, additional testable data will certainly be collected. Notes to self (self is bsm)-- here are the data you should remember to acquire:

• two sequential peak/focuses w/DCR (so second is zero offset)
• CCB peak/focus with no flagging (rise/fall dt's of zero, no cals firing)
• heavily flagged CCB peak/focus (50% cal duty cycles, with flagging -- perhaps even with irregular cal pattern)

The idea here is to emphasize the integration-dropping and bring out any problems that might exist.

6.3 Regression Testing

There are two sets of regression tests, specified at GbtFitsMonitorRegressionExamples and GbtFitsMonitorBenchmarks, which will be run to ensure that there is no change to DCR data reduction behavior and that GFM will still process those projects correctly.

Also, these changes will be part of the regular regression tests on the telescope before release.

Signatures

APPROVED: I acknowledge that my request is fully contained in this MR, and if the SDD delivers exactly what I specified, I will be happy.

ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.

Written	- AmyShelton - 08 Dec 2005
Checked	- NicoleRadziwill - 08 Dec 2005
Approved by Sponsor	- BSM 10dec05
Approved by CCC	- RichardPrestage 19 Dec 2005
Accepted/Delivered by Sponsor	BSM 28mar06

Symbols:

• Use %X% if MR is not complete (will display )
• Use %Y% if MR iscomplete (will display )

CCC Discussion Area