NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Software > PlanOfRecordC62006 > ModificationRequest5C606

Changes | Index | Contents | Search | Statistics | Go

Integration of the Zpectrometer into GBTIDL

Modification Request 5 (C6 2006)

• 1. Introduction
• 2. Background
• 3. Requirements
• 4. Design
  • 4.1 New Classes
  • 4.2 New Data Containers
  • 4.3 Plotting data containers
  • 4.4 Index File
• 5. Deployment Checklist
• 6. Test Plan
  • 6.1 Internal Testing
  • 6.2 Sponsor Testing
  • 6.3 Integration/Regression Tests
• Signatures
• CCC Discussion Area

1. Introduction

The Zpectrometer is an ultra-wideband spectrometer being built by the University of Maryland for use with the GBT and its Ka-band receiver. During the Fall of 2006, this new backend will be undergoing Engineering tests at the GBT. During this time, data analysis software will be needed to support the expert users performing these tests. This software package to be used will be GBTIDL.

However, the data products produced by Zpectrometer are not suitable for import into GBTIDL. In fact, a number of data reduction steps that usually occur upstream of GBTIDL are needed in this case. To support data analysis needs during this test phase of the Zpectrometer, special modifications are needed to GBTIDL to import the Zpectrometer data and convert them into a form suitable for data reduction. That is the topic of this MR.

The data capture development is being split into three parts as illustrated in this figure.

• Zpectrometer Data Capture Diagram:
  

The top of the figure illustrates how data capture fits into the flow of data at the GBT. The backend devices and other managers produce a set of FITS files. Those FITS files are collected and used in the data capture process to produce an SDFITS file that can be consumed by a data analysis package like GBTIDL. The tool that does this at the GBT is also called "sdfits".

In the case of the Zpectrometer (illustrated in the box at the bottom of the figure), the data capture steps are still under development. During development, they have been split into 3 steps. In the first step, the sdfits tool will take the raw FITS file produced by the Zpectrometer and use the other M&C FITS files to produce a new FITS file that includes that information and follows the SDFITS convention. Then the product of the Sdfits-tool will be imported into GBTIDL so that custom IDL routines can perform the final calibration and export the spectra to the GBT SDFITS format. This MR describes the work necessary to modify GBTIDL to support these IDL routines.

2. Background

For more information on the Zpectrometer project, see the NRAO Zpectrometer Wiki.

The details on the creation of a M&C Manager for the Zpectrometer can be found at ModificationRequest1C306.

A list of the keywords required for the WASP data product can be found in dict.html.

The format of the Zpectrometer SDFITS file, along with a link to an example file with 1 row of data, can be found in Appendix A.

The work needed to modify the sdfits tool to support this is work is described in ModificationRequest4C406

3. Requirements

GBTIDL must support:

1. The import of files that the sdfits tool generates from Zpectrometer observations.
2. The navigation of imported data using methods identical in functionality, and similar in use to data navigation methods supported in GBTIDL for spectral line mode.
3. The production of IDL structures from imported data using methods identical in functionality, and similar in use to methods supported in GBTIDL for spectral line mode.
4. The updating of an imported file via the update method (for keeping an imported file that is being filled dynamically by hand in sync).
5. These specific columns in the INDEX file: INDEX, FILE, EXTENSION, ROW, MC_SCAN, SCAN, BEINDEX, SOURCE, TRCKBEAM, PROJECT, OBSERVER, PROCEDURE, DIODE, AZIMUTH, ELEVATION, LONGITUDE, EXPOSURE, OBSFREQ, and TIMESTAMP.
6. Basic visualization of IDL structures from imported data.

GBTIDL does not need to support:

1. Online features involving specifically the online data in /home/sdfits
2. Data export of the IDL structures produced from the files generated by the sdfits tool.

4. Design

The main thrust of the modifications needed to support the above listed requirments will be to create a class (and all required supporting classes) similar to the io_sdfits_line class. An object of this class will then provide all the same methods as io_sdfits_line for data import, navigation, and retrieval. Another import design issue is the new data containers that this new class provides.

4.1 New Classes

Attatched is an outdated UML diagram of the GBTIDL I/O Classes. This needs to be updated!

We will be inheriting from various abstract classes to create new classes that can handle the specific differences between SDFITS spectral line classes and the new file(s) produced by the Sdfits tool. Here is a chart describing these new classes; existing classes are in regular text, new classes are bold:

There are additional structures that will be needed, such as z_row_info, and z_sdfits_row_group.

4.2 New Data Containers

The IDL structures (data containers) retrieved from the GBTIDL I/O layer will be, appart from the inculsion of virtual columns, a straight translation of the rows of the imported file(s) produced by the Sdfits tool. This is in line with the similar design approach taken in ModificationRequest4C406, where additional columns to the Zpectrometer FITS file are automatically picked up and included in the product of the sdfits tool.

The structures returned from the mrdfits routine can be used as our new data containers. However, certain virtual columns (keywords) in the Zpectrometer FITS binary extension table will need to be carried over to each new data container.

Though this approach offers the huge advantage of keeping the definition of the new data container fluid, and avoids having to recode everytime new columns are added to the Zpectrometer FITS file, it does bring about one disadvantage. The current I/O design allows the arbitrary, simultaneous extraction of data containers from multiple files and extensions, regardless of data size (since the spectrum data containers use pointers). This complex set of code will need to be by-passed so that we may use directly the structures returned by mrdfits. Another minor issue is that if the user is working with multiple files, we will have to append on the subsequent arrays of structures returned by mrdfits, since this function can only be called on one file-extension at a time.

4.3 Plotting data containers

We should be able use the current GBTIDL plotter with the new data containers. A procedure will be provided which will copy relevent parts of the structure and associated lags into a temporary continuum data container which will then be displayed in the plotter. This is being done to provide quick looks at the lag data. No new plotter code will need to be written. The continuum data container is chosen because the plotter will plot that as value versus integer channel number - disabling the x-axis display options in the plotter (all of the frequency and velocity related operation of the plotter). The standard IDL plotting functions will also be available as always.

4.4 Index File

The requirements for the index file are described in the related sdfits tool MR. The index fields will be INDEX, FILE, EXTENSION, ROW, MC_SCAN, SCAN, BEINDEX, SOURCE, TRCKBEAM, PROJECT, OBSERVER, PROCEDURE, DIODE, AZIMUTH, ELEVATION, LONGITUDE, EXPOSURE, OBSFREQ, and TIMESTAMP.

5. Deployment Checklist

This development is taking place on a separate branch of GBTIDL code, therefore there is no need to integrate it into the system. In terms of training, the current GBTIDL documentation will give the sponsors and users the familiarity needed as background, but one-on-one support will be given when needed.

6. Test Plan

6.1 Internal Testing

Currently there are no example data sets for the developers to use for Internal Testing. These may become available from Sept. 18, 2006, on.

* Unit Tests: once example data sets are available, they will be used to construct various unit tests in the code branch where this development will take place. Unit tests will cover both API level classes, and the final Sdfits application.

* Simulator Tests: example data sets will hopefully be produced through use of the simulator and the Zpectrometer hardware in the Jansky Lab from Sept. 18, 2006, onward. These will most likely contain a lot of nonsense data, but the developers will have to work with what they get for Internal Testing.

6.2 Sponsor Testing

Although KarenONeil is the NRAO sponsor for this MR, AndrewBaker will be doing the calibration in GBTIDL during commissioning. Therefore, sponsor testing will involve both Karen and Andrew determining that the product of the modified Sdfits tool has sufficient information so that when it is imported into GBTIDL using the the methods described in ModificationRequest5C606, it can be calibrated.

It should also be noted that commissioning does not start until October 1, 2006, so this MR's status will never move beyond "Ready for Sponsor Testing" during Cycle 6.

Here's a guide to aid in sponsor testing: ZpectrometerGbtidlUsage

6.3 Integration/Regression Tests

This development will be taking place in a branch of Sparrow, therefore there is no need for Integration/Regression testing.

Signatures

APPROVED: I acknowledge that my request is fully contained in this MR, and if the SDD delivers exactly what I specified, I will be happy.

ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.

Written	- PaulMarganian - Sept 14, 2006
Checked	- BobGarwood - Sept 14, 2006
Approved by Sponsor	- KarenONeil - 19 Sep 2006
Approved by Sponsor	- AndrewBaker - 19 Sep 2006
Approved by CCC	- RonMaddalena - 2 Oct 2006
Accepted/Delivered by Sponsor	symbol - name - date

Symbols:

• Use %X% if MR is not complete (will display )
• Use %Y% if MR iscomplete (will display )

CCC Discussion Area

Both MR's involve the creation of a new software branch. The branch is only meant, for now, for a single customer -- those commissioning the Zpectrometer. Since it's a separate branch, with its own executables, there is no interaction of this work with existing software and our standard users. There are also statements to the affect that this branch will be in development for some months. The branch may or may not be collapsed into our production software but that decision won't happen for many months.

It's possible we may want to review this software at the time the branch is collapsed. But, until then, there shouldn't be any interactions or consequences of the work on any of our users or systems.

-- PaulMarganian - 06 Sep 2006

• dict.html:

Attachment:	Action:	Size:	Date:	Who:	Comment:
IDL_Design_v0_4.doc	action	707072	06 Sep 2006 - 15:24	PaulMarganian
dict.html	action	34147	12 Sep 2006 - 13:27	RamonCreager
ZpectDataCapture.jpg	action	42947	19 Sep 2006 - 12:29	PaulMarganian
ZpectDataCapture2.jpg	action	42947	19 Sep 2006 - 12:30	PaulMarganian