NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Data > Sdfits > SdfitsRedesign05

Changes | Index | Contents | Search | Statistics | Jump to Topic

SDFITS Redesign, 2005

• Introduction
• Compelling Events
• Design Queries and Proposals
  • 1. Unit Tests
  • 1.1 Benchmark Tests
  • 2. Scan Data
  • 3. Backend Classes
  • 4. Calibration
  • 5. Index Writer Library
  • 6. New Header Info
  • 7. Backwards compatibility
  • 8. Data Capture Phase II
    • A. Single Integration Dump
    • B. Waterfall Plot
    • C. Scan
    • D. Observation
• Plan
  • Unit Tests
  • Benchmark Tests
  • Scan Data
  • Data Capture Phase II
  • SDFITS product changes
  • Other Programs with which to Coordinate Changes

Introduction

The Sdfits filler is a python-based program which reduces GBT Engineering FITS files into a single FITS file of the SDFITS format. Sdfits is not only a part of the Sparrow source tree, but also uses many of the same pre-processing components within Sparrow as programs such as Gfm, Turtle, and gbt2class.

The main goal of this Redesign is not to change the Export Data Format (EDF) that is our current Sdfits product. This may happen in the distant future, and should require a good deal of colaboration between Astronomers internal and external to NRAO.

The main purpose of this Redesign effort is to set ourselves up properly for future requirments. This includes the continued use of Sdfits with GBTIDL, the online Sdfits filler, integration-level data reduction, Gfm/GBTIDL integration, the addition of supplimental info in the EDF, and more.

In this document, we will list more speciffically the the problems we hope to address with this redesign, and then discuss individual design topics.

Compelling Events

Compelling Events, not in order of priority:

• Difficulty in debugging - Scan Data class mis-uses exceptions.
• Adding new device (like the CCB) will be a huge pain in Scan Data class.
• Making changes is frightening because there are no Unit Tests.
• Cross polarization in spectrometer must be supported.
• Additional Info requested - need project scientest to arbitrate this.
• Conforming to WCS - who's asking for this? Does it need to be done now?
• Data Capture Phase II - for integration by integration updates, do we leverage off of sdfits?
• Online Sdfits - does this interact with Gfm (tied to above issue)?
• How far do we take backwards compatibility? Should users have to run an old version of sdfits to fill old data?
• Users can't use sdfits as advertised on large files without running into Memory Leak!
• Sdfits Index Writer must be kept in sync with GBTIDL code that creates Index File.

Design Queries and Proposals

As background, the classes which Sdfits uses in the Sparrow source tree can be viewed in UML format here. The latest code can be found in /home/sparrow/integration/sparrow. The relationships between Scan, Scan Data, Project and Project Data need to be checked.

1. Unit Tests

• This needs to be the first thing done so that we can see when we break things due to our refactoring.
• Because we don't have any current tests, we just say the way it works now is right...
• What to use for data? Perhaps the data used by the GBTIDL tutorials here.
• Start with the most basic classes, and move our way up: backends, then project and scan classes...
• Make sure zipped data is checked into CVS.
• We should get the fql unit tests working again.
• Who should adopt this orphaned program (fql)?

1.1 Benchmark Tests

• In GBTIDL we have successfully used nightly benchmark tests to ensure that changes do not cause inadvertent hits in performance.
• We should do something similar for the programs that here

2. Scan Data

• Next in priority, since defeciencies here our currently inhibiting development efforts
• Refactor use of exceptions. Offending Queries are:
  • EQUINOX
  • Antenna Positions Query'
  • Separate PROCSIZE & PROCSEQN from LASTON & LASTOFF

• Extreme number of methods in Scan Data the problem?
  • For the current design of the SDFITSWriter class, scan info is gathered by the Scan Data class from the FITS files when a new scan is encountered, and then as each row is written, the gathered info is then retrieved from Scan Data for each column.
  • For each column in the SDFITS format, there is a corresponding get* method.
  • As many as 13 of these columns have different values according to wether the sdfits data mode is 'ave','cal', or 'raw'. Example: 'Duration' column's value will be for the scan in 'ave' mode, for the integration in 'cal' mode, and for the state in 'raw' mode.
  • With 46 columns, about 6 keyword values, and the above explained redundancy for certain column according to data mode, that means as many as 63 neccessary get* methods (there are 77).
  • Each columns info is gathered by a separate gather* method. This accounts for the 52 gather* methods.
  • So, it seems that the number of methods in Scan Data isn't really the problem.
• Size of Scan Data definitely a problem:
  • at 2622 lines, the size is definitely a problem.
  • much of the get* methods can't be reduced.
  • the following methods could probably go directly into the backend classes:
    • gather(Scan,Integration,State)Durations
    • gather(Scan,Integration,State)Exposures
  • the following happen no where else in Sparrow code, should they have their own classes?:
    • 14 gather* methods that query the GO fits file
    • Lst and position queries from the Antenna fits file, along with complicated calculations of sampled antenna positions. Antenna class does exist, but not sure who uses it, and is very small.
  • a large amount of code is used for the (scan,integration,state) Center Frequency:
    • these involve queries of the IF and LO1A tables
    • they also involve quering the GO fits table to determine if this is a freq. switched scan.
    • the current backend classes have very similar calculations for the Scan Center Frequency.
    • push these calcs and queries down into the backend classes. This will be tricky: making sure that the two sets of code produce the same results.
• Name/Location of Scan Data is wrong.
  • Scan Data provides data that is specifically tailored for Sdfits.
  • The name of Scan Data and it's location in gbt/api/project is mis-leading
  • perhaps it should be in a new directory: gbt/api/sdfits with the new name of sdfitsData. Then the sdfits and gbt2class applications can use it.
• Actually, perhaps Scan Data can stay:
  • but it will be a thin class that is only responsible for retrieving data directly from FITS files.
  • the new sdfitsData class could be a child of this new Scan Data, and this is where all the specific formatting and tailoring for sdfits occurs.
  • split up sdfitsData into sdfitsAveData, sdfitsCalData, and sdfitsRawData
• Summary:
  • Refactor code into class heirarchy for different data modes: raw, cal, ave. See proposal here.
  • Refactor ScanData into subclasses:
    • ScanData is responsible for retrieveing raw info in fits files. Results from fits files are stored in a dictionary, rather then dozens of member variables.
    • sdfitsData class and it's children tailor this info for Sdfits format.
  • Push some code (Durations, Exposures) into special backend classes.
    • Each sdfits[Backend] class uses the same interface, so that there are no if statements switching off of backend in the sdfitsData classes.
  • No more need for any of the 77 get* methods!: just use the dictionary and the base class's get("key") method.
  • Place all sdfits classes in the gbt/app/sdfits directory, since these are all used speciffically for sdfits.
    • This does mean that gbt2class will be using another applications classes, instead of an api's
    • The real solution would be to filter out all the common elements between sdfits and gbt2class, and place them in some new api (gbt/api/dataAnalysis, or something like that).
    • But that would be a lot of work for a program (gbt2class) that does not have a very big user base and has not even left the beta stage.

3. Backend Classes

• Current state of Backend classes is dismal. Lots of duplicate code in separate classes. The code can be found in /home/sparrow/integration/sparrow/gbt/api
• The current classes can be viewed here.
• Refactor into a class heirarchy to faciliate addition of new backends.
  • A proposal for refactoring the backends can be viewed here.
  • Details for the proposed backend classes can be viewed here here.
  • Details for the proposed helpers to the backend classes can be viewed here.
  • All backends children of a virtual backend class
  • Further split up backends into spectral line backends and continuum backends
• Use new CCB backend as a test case.

4. Calibration

• The calibration done by sdfits in 'cal' and 'ave' mode is not currently reliable. For use with GBTIDL, for instance, 'raw' mode is recommended.
• GBTIDL has trusted calibration code.
• An investigation should be into how hard it would be to port the IDL code to Sparrow's python code base.
• Though this migration might be difficult, the benifits would be enormous:
  • Gfm would be able to use the new sdfitsData classes to produce calibrated spectral line plots without interacting with GBTIDL and/or SDFITS files.
  • Users of gbt2class would finally be able to put calibrated data into their class binaries directly
  • Fix an otherwise 'broken' aspect of the SDFITS tool!
• Of course a down side to this possible migration would be the maintanence of identical code in two separate places.

5. Index Writer Library

• Separate this from sdfits so that it could also be used by GBTIDL
• Solves problem of duplicated code.
• But where would this code live - sparrow or gbtidl? in either case, you've got a funny dependence of one on the other - and they are in separate CVS modules.
• Place in a third separate location? All by itself?

6. New Header Info

• All this is for the project scientist to decide: JimBraatz
• What is the new header info that needs to be added?
• Some of this might rely on M&C, or at least Astrid changes
• Examples:
  • IF Number - currently must be calculated in GBTIDL
  • Integration Number - something more convinient then looking at LST number
  • Frequencies for each spectral window - Observor's intent?
• Previous work related to this topic can be found at SdfitsVersionTwo.

7. Backwards compatibility

• perhaps the 'sdfits' entry point should first read the M&C version number from the GO fits file, and then determine what version of the underlying code to call for filling this data...

8. Data Capture Phase II

• We need to spend time thinking about this now, so that we know wether this will affect our design changes.
• For more background, see the Software Review Presentation.
• Here's the basic issue: for scan-based Quick Looks, it seems a waste not to use the online Sdfits filler - but for integration-based Quick Looks, would it be better to by-pass Sdfits?
• Integration-based: relying on files for communication at this time scale could be a very bad thing: a reason to use Grail and not online Sdfits.
• Exactly what information is needed for different stages of an observation: is this info available only through sdfits, or can we just use Grail?
• Is the concept of the 'current record' practical, or do we not need it, due to present code?
• Does Sdfits filler work before the scan is done? Yes it can! Test results show that the following are written and complete at the begining of a scan:
  • LO1
  • Reciever
  • IF
  • GO
• But the following are updated as scan progresses:
  • Antenna: ANTPF(GR)POS table updates as scan progresses - takes about 1.5 minutes for first rows to be written, and then size updates regularly. Joe thinks that flushing isn't happening ... is looking into it.
  • Spectrometer: file size updates exactly as data parameter increments after each integration dump, but the DATA table does not appear until the very end of the scan. This was fixed and patched in on July 5, 05. Tested and works...
• So, we could create a 'current record' that updates at the integration level, and that only needs to fill the last integration at the end of a scan, instead of all the integrations, like the current online filler.
• Divide up the different Quick Look applications that would use this, using Spectrometer as example:

A. Single Integration Dump

• As each integration is completed, the data must be made available for display.
• What other info is required (i.e. IF info for creating frequency axis?).
• What processing needs to be done on the integration (same as what Sdfits does?).
• How do we store the results ('current record')?
• This all could be done right now in Gfm, if we don't need all the sdfits auxillary data.

B. Waterfall Plot

• Project Scientist needs to decide if this is needed.
• Should be able to share much of the same functionality as the display for the Single Integration Dump, just buffers previous ones for the display.
• Or are the previous integration dumps stored in the 'current record'

C. Scan

• Project Scientist needs to decide if this is needed.
• Scan using the M&C definition - for an OffOn, this would be the Off or On part (subscans 1 or 2).
• Quick Look to display an average of all the scan's integrations?
• Latest scans can be processed using the Online SDFITS filler and GBTIDL.

D. Observation

• For example an OffOn observation consisting of two M&C scans ( Off, and then On!), subscans 1 & 2.
• GBTIDL code and the online filler can be leveraged right now to operate at the Observation level.
• Or, we could create the 'current record' filler for using as input to GBTIDL.

Plan

A productive meeting was held on July 7, with AmyShelton, MelindaMello, PaulMarganian. Some of the proposals above were a product of this meeting. Below is the plan of action to take for the next few weeks.

Under Construction

Unit Tests

• Do these first!
• Fix Spectral Processor. Done
• Test all other methods for SP. Done
• Divide up Dcr with Amy. DCR and DCRData Done
• Let Amy handle Spectrometer. Done
• Fix FQL tests.
• Create ScanData unit tests (and rest of Project api) before refactoring starts. Done
• Create unit tests for SDFITSWriter. Done

Benchmark Tests

• Create benchmark test class
• Implement class for testing ScanData and SDFITSWriter

Scan Data

• Fix current problems with ScanData:
  • Split up gatherProcInfo in anticipation of LASTON & LASTOFF keywords missing. Done
  • Refactor gatherEquinox in anticipation of it being missing. Done
  • Refactor reading of Antenna fits file so that it works in debug mode (i.e., does not rely on catching exception when looking for wrong table). Done
• Create ScanDataNew to replace ScanData, with unit tests.
• Create sdfits backend classes, again with unit tests.
• Create sdfitsData with unit tests.
• Create sdfits[Scan,Integration,State]Data classes, with unit tests.
• Once the above classes are solid, modify gbt2class and the SDFITSWriter class to use the three sdfits data classes for the different data modes.
• Replace ScanData with ScanDataNew.

Data Capture Phase II

• Work on this in parrallel.
• Create integration state engine. Done
• Create a class hierarchy with present online filler? Done but not implemented with current filler.
• Test Joes' fix to Antenna flushing. Done
• Fix Spectral Processor 'data' parameter Done
• Test Spectrometer with more Banks and higher data rates. Done - in general, filler can't keep up.

SDFITS product changes

• Bob Garwood has compiled a list. Wait for Jim Braatz.

Other Programs with which to Coordinate Changes

• gbt2class
• gbtstatus
• GFM
• Turtle (?)

-- PaulMarganian - 30 Jun 2005

Attachment:	Action:	Size:	Date:	Who:	Comment:
proposed_backend_heirarchy.jpg	action	127730	16 Dec 2005 - 19:59	AmyShelton
proposed_sampler_heirarchy.jpg	action	1829	16 Dec 2005 - 19:59	AmyShelton
proposed_sdfits_backend_classes.jpg	action	71314	16 Dec 2005 - 20:13	AmyShelton
sdfits_backend_classes.jpg	action	62980	16 Dec 2005 - 19:59	AmyShelton
sdfits_classes.jpg	action	77597	16 Dec 2005 - 20:00	AmyShelton
9_DataCapture_final.ppt	action	218624	05 Jul 2005 - 19:23	PaulMarganian
proposed_scan_data.jpg	action	118302	16 Dec 2005 - 20:00	AmyShelton
new_scan_data.jpg	action	90530	16 Dec 2005 - 20:01	AmyShelton
new_scan_data.vsd	action	296448	16 Dec 2005 - 20:15	AmyShelton
proposed_backend_heirarchy.vsd	action	251392	16 Dec 2005 - 20:15	AmyShelton
proposed_sampler_heirarchy.vsd	action	252928	16 Dec 2005 - 20:15	AmyShelton
proposed_scan_data.vsd	action	227328	16 Dec 2005 - 20:17	AmyShelton
proposed_sdfits_backend_classes.vsd	action	340480	16 Dec 2005 - 20:17	AmyShelton
sdfits_backend_classes.vsd	action	315392	16 Dec 2005 - 20:17	AmyShelton
sdfits_classes.vsd	action	479232	16 Dec 2005 - 20:18	AmyShelton