NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Software > PlanOfRecordC32006 > ModificationRequest5C306

Changes | Index | Contents | Search | Statistics | Go

Puting Lab Spectrometer data into SDFITS/GBTIDL - Phase I

Modification Request #5 (C3 2006)

• 1. Introduction
• 2. Background
• 3. Requirements
• 4. Design
  • Robustness and missing auxillary files
  • Using backend FITS files for some values needed by =SDFITS=
• 5. Deployment Checklist
• 6. Test Plan
  • 6.1 Using Real GBT Spectrometer data.
  • 6.2 Using the Lab Spectrometer
  • 6.5 Test GBTIDL
  • 6.4 Test GFM
• Signatures
• CCC Discussion Area

1. Introduction

The Lab Spectrometer, sometimes referred to as the Test Spectrometer is a rump version of the GBT Spectrometer. It uses the same correlator chips, and is controlled in much the same way, via a VME Bit-3 interface to a linux host. The original purpose of the Lab Spectrometer was for ALMA antenna tests. It is now in the Electronics Lab, where the digital group uses it to troubleshoot Spectrometer issues, and receiver engineers and some scientists hope to use it to test receiver baselines while receivers are in the lab. For more details, see TestSpectrometer.

The above mentioned parties require data analysis software for many of the tests they would like to perform. This MR describes the modifications needed to the SDFITS program to fill Lab Spectrometer data into the sdfits format, so that it can then be read into our supported analysis tool, GBTIDL.

2. Background

The Lab Spectrometer produces FITS files identical to those of the GBT Spectrometer. However, in the Electronics Lab, the Lab Spectrometer will typically be run without the use of other M&C managers, so the resulting backend FITS files will be the only FITS files available to software analysis tools, i.e. there are no resulting ScanLog.fits file, nor GO, IF, Antenna FITS files.

The new GFM Spectrometer Engineering plugin, in conjunction with the scanlogsim program can read Lab Spectrometer data. However, GFM provides only Quick Look display capabilities, for raw lags and raw power, with only limited analytical functions, such as spectra differences, and spectra averaging. If more sophisticated analysis is needed, some other tool is needed.

For scientists or engineers who are willing or eager to write their own analysis tools, IDL, glish and python are all available to them. It would be ideal, however, if they could save time and energy by using the supported analysis tool for the GBT, GBTIDL.

The main obstacle blocking the use of GBTIDL with Lab Spectrometer data is the abscence of the above mentioned auxillary FITS files: ScanLog.fits, and GO, IF, LO1A/B, and Antenna FITS files. The import/export data format used by GBTIDL is SDFITS, which currently cannot fill data that are missing the above mentioned data files.

This MR will describe the modifications needed to the sdfits program so that it can produce SDFITS files which in turn can be imported into GBTIDL with enough information to support basic analysis procedures.

Note: The SDFITS format does not include the raw lags from the spectrometer. To do advanced analysis on raw lags (they can be just viewed with GFM), special tools will need to be written.

3. Requirements

• The sdfits program must be able to fill data when the project directory structure only contains:
  1. A ScanLog.fits file
  2. At least one Spectrometer FITS file in a Spectrometer directory
• The sdfits program is not responsible for creating a missing ScanLog.fits file. This can be created using the pre-existing scanlogsim application.
• The product of sdfits, when read into GBTIDL, must not introduce exceptions when performing the following basic operations:
  • filein
  • list
  • getrec, getscan, get
  • use of all plotter axes
• The product of sdfits, when read into GBTIDL, will not support high level calibration routines. This is due to the lack of IF, Feed and Polarization information in the sdfits file. Faking these values in the abscence of auxillary FITS files is beyond the scope of this MR, and GBTIDL users will have to write their own calibration routines, using the sampler name instead of the above mentioned variables. Some of the calibration routines not supported include:
  • gettp
  • getsigref
• The sdfits program must use default values for SDFITS keywords or columns when an auxillary FITS file is missing. An auxillary FITS file is defined as one of the following types:
  • GO
  • Antenna
  • LO1A or LO1B
  • IF
  • Receiver
• Certain specific values that are usually obtained from an auxillary FITS file can be obtained from the backend FITS file. If the required value is missing due to an absent auxillary FITS file, sdfits must obtain this value from the backend. These values include:
  • PROJID
  • OBJECT
  • SCAN
• When sdfits fills scans that are missing any auxillary FITS files, it must issue a single warning message. This message will include:
  • the files missing from which scans
• The following list of columns must accurately reflect the product of sdfits when missing auxillary files. The Notes column explains how the column's value must be derived.

Field	Available	Missing File	Notes
object			from backend file.
bandwid			from backend file.
tdim7
tunit7
ctype1			contstant: "FREQ-OBS", translated as "TOPO" in GBTIDL
crpix1			from backend file.
cdelt1			from backend file.
freqres			from backend file.
ctype2		GO
ctype3		GO
crval4		IF
observer		GO
obsid			available in backend file.
scan			avaialable in backend file.
obsmode		GO
frontend		Receiver
tcal		Receiver	1.0
veldef		LO1	"RADI-OBS"
restfreq		GO	bandwid/2
velocity		GO	0.0
equinox		GO
sampler
feed		IF	0
srfeed		IF	0
beamxoff		IF
beameoff		IF
sideband		IF
procsize		GO
procseqn		GO
laston		GO
lastoff		GO
timestamp
tambient		Antenna
pressure		Antenna
humidity		Antenna
date-obs
duration
exposure
tsys
data
lst		Antenna
crval1		LO1	bandwid/2
obsfreq		LO1	bandwid/2
crval2		Antenna
crval3		Antenna
azimuth		Antenna
elevation		Antenna
sig
cal
zerochan
vframe		LO1	"TOPO"
rvsys		LO1	0.0
projid			available in backend file.
sitelong		Antenna
sitelat		Antenna
siteelev		Antenna

4. Design

Robustness and missing auxillary files

The main bulk of the coding that needs to be done for this MR involves making sdfits and its supporting libraries robust to missing auxillary FITS files. Class by class, these changes include:

• sparrow/gbt/api/project/src/ProjectScans: This class needs to be refactored so that when the setProject method is used it does not raise an exception when the GO and Antenna FITS files are missing. Instead, this project should return a status that indicates whether any scans are indeed missing files.
• sparrow/gbt/api/project/ScanData: As above, whenever a query for a missing auxillary FITS file fails, the appropriate default value should be returned.
• Backend Classes (Spectrometer): These classes also need to be more robust when any auxillary FITS files are missing. All currently raised exceptions should be suppressed, and instead the appropriate default values (see above) should be used.

Using backend FITS files for some values needed by SDFITS

The above mentioned values, PROJID, OBJECT, SCAN, are currently retrieved from the scan's GO FITS file. When the ScanData object fails to retrieve these values, for whatever reason, it should then try to retrieve them from the backend.

5. Deployment Checklist

What has to get done to integrate this completely into the system. This checklist must be completed before Cycle Integration Testing begins.

• System: These changes should not affect other programs that also use the Sparrow libraries, including GFM.
• Computing: No communication needed with computing group.
• Documentation: What ever documentation is being supported for the Lab Spectrometer should include quick instructions on how to use scanlogsim and sdfits to import data into GBTIDL. Of course, the SDFITS wiki documentation will be updated.
• Training: The above mentioned documentation should be sufficient for the users.
• Notification: The M&C and sdfits release notes should note these changes: that sdfits is now robust to missing auxillary FITS files, and will only warn users of missing FITS files if the -debug option is set.

6. Test Plan

6.1 Using Real GBT Spectrometer data.

• First a copy of all data for a scan range of a trusted project (such as an old regression test) should be made.
• This copy will then have all auxillary FITS file removed, so that the project directory only contains projectCopy/Spectrometer/*.fits files.
• The scanlogsim application will then be used to create a ScanLog.fits file in the projectCopy directory.
• sdifts can then be used to fill the data from both the original directory and then the copy.
• The comparison of the two resulting SDFITS files should show that:
  • The PROJID, OBJECT, and SCAN values are the same.
  • All data only available in or resulting from the Spectrometer FITS files are the same.
  • All data only available in or resulting from Auxillary FITS files show default values for the stripped project.

6.2 Using the Lab Spectrometer

• The Lab Spectrometer will be used to create FITS files in project/Spectrometer. (note: the manager may write them to project/LabSpectrometer, in which case, a symbolic link can be made to redirect this to project/Spectrometer).
• Once all scans for the test have been completed, the scanlogsim application will then be used to create a ScanLog.fits file in the project directory.
• At this point, the GFM Spectrometer Engineering Plugin should be used to view the data to make sure that the data is suitable for the test.
• sdfits can then be used to fill this data.
• The resulting SDFITS file should be viewed (using fv) to ensure that the appropriate values are using default values, etc.

6.5 Test GBTIDL

The resulting SDFITS files from the previous two tests should be read into GBTIDL to see if they pass the requirments stated above.

6.4 Test GFM

Run regression tests on GFM to ensure that there are no unintended side effects. Regression data sets are available at: GbtFitsMonitorBenchmarks AmyShelton is willing to do/assist with this.

Signatures

APPROVED: I acknowledge that my request is fully contained in this MR, and if the SDD delivers exactly what I specified, I will be happy.

ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.

Written	- PaulMarganian - 05 Apr 2006
Checked	- AmyShelton - 06 Apr 2006
Approved by Sponsor	- RonMaddalena - 25 Apr 2006
Approved by CCC	- RonMaddalena - 25 Apr 2006
Accepted/Delivered by Sponsor	- RonMaddalena - 17 May 2006

Symbols:

• Use %X% if MR is not complete (will display )
• Use %Y% if MR iscomplete (will display )

CCC Discussion Area

-- PaulMarganian - 05 Apr 2006