NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Software > PlanOfRecordC52007 > ModificationRequest15C507

Changes | Index | Contents | Search | Statistics | Go

Merge Zpectrometer code branch into main trunk

Modification Request #15 (C5 2007)

• 1. Introduction
• 2. Background
• 3. Requirements
• 4. Design
  • 4.1 Merging Zpectrometer-specific code back into the main branches of GBTIDL and sparrow
    • 4.1.1 Changes local to the zpectrometer branch of GBTIDL
    • 4.1.2 Changes local to the zpectrometer branch of sparrow
  • 4.2 New organization of /home/sdfits
• 5. Logistical details during testing and deployment
• 6. Deployment Checklist
• 7. Test Plan
  • 7.1 Internal Testing
  • 7.2 Sponsor Testing
  • 7.3 Integration/Regression Tests
• Signatures
• CCC Discussion Area

1. Introduction

In the fall of 2006, the GBTIDL and sparrow code trees were both branched to produce development code branches in support of the initial tests of the Zpectrometer on the GBT. Those changes need to be merged back into the main trunks so that only only the main trunk needs to be maintained.

The online sdfits daemon needs to fill Zpectrometer data after the code merge has taken place. A re-arrangement of the structure of /home/sdfits and related directories needs to happen at that time to make it easier to manage that area and provide flexibility to Zpectrometer data capture.

2. Background

At the time the two code trees were branched, there was active and rapid development of code necessary to support the Zpectrometer. That work was not closely tied to the regular development cycle of regular software CM&E work in Green Bank. In order to isolate the main code branches from frequent changes to support the Zpectrometer, the two code trees were branched. The development in these trees are described in this GBTIDL MR, and this SDFITS MR. That work has remained relatively stable over recent months and it has also proven to be isolated from almost all of the rest of the code in each affected package. In order that the continuing Zpectrometer work see the regular bug fixes and enhancements in the main branches without having to maintain two separate code trees, it is now time to merge the Zpectrometer-specific parts of the branches back in to the main truck.

The online sdfits daemon only fills DCR, SpectralProcessor, and Spectrometer data. Output fits files are written to /home/sdfits. The accompanying index files are also written to /home/sdfits as are the log files produced by the sdfits daemon. Temporary lock files are written to /home/sdfits-locks and empty world-writable flag files are created in /home/sdfits-flags. A symbolic link to each flag file is placed in /home/sdfits. The GBTIDL "online" procedure looks in /home/sdfits and opens the most recently modified spectral-line fits file in that directory. The flag file is needed so that users can flag on-line data (which exist in a read-only directory to prevent the user from accidentally changing it). At the time that the scheme of the flag file existing in a writable directory (/home/sdfits-flags) and symbolic link existing in /home/sdfits was designed, it was thought that the flag file would need to be created and destroyed periodically. That turns out not to be the case and so there is no reason why the empty flag file could not be created in the same directory with the output fits files. The write permissions of /home/sdfits and /home/sdfits-flags are identical and there is no reason to create the symbolic link from the directory containing the output FITS file to the true flag file in /home/sdfits-flags.

When filling Zpectrometer data in the branched version of sdfits, associated files (as indicated in the raw Zpectrometer FITS file) are copied over to the same directory where the output fits file is written.

There are an increasing number of files that are written, created, or copied when the online sdfits daemon runs (the output FITS files, the index files, the log file, the flag file, associated Zpectrometer files) such that /home/sdfits is very cluttered. The changes described here will put all of the files produced when filling data from the same project into a single directory in /home/sdfits using the project name as the name of that directory (e.g. /home/sdfits/AGBT07A_058_03).

For more general information on the Zpectrometer project, see the NRAO Zpectrometer Wiki.

3. Requirements

• The existing unit tests must continue to work as before after the Zpectrometer code has been merged in (unaffected by the presence of the Zpectrometer code).
• The existing Zpectrometer functionality must be present in the main trunk of GBTIDL and sparrow (sdfits).
• The online sdfits daemon must fill Zpectrometer data.

4. Design

4.1 Merging Zpectrometer-specific code back into the main branches of GBTIDL and sparrow

4.1.1 Changes local to the zpectrometer branch of GBTIDL

The new code in the following list will all be simply moved to the contrib directory in the main GBTIDL trunk. A new zgbtidl script will be created which will invoke init_zgbtidl. That will be done to auto-compile the zpectrometer routines only for those users that need them. Note that they will still be in the IDL path for regular GBTIDL users, they will simply not be compiled at startup (none of the routines in contrib are compiled at startup during normal GBTIDL use) unless zgbtidl is invoked.

The rows_index_section__define.pro changes will need to be examined carefully as it appears from the comments that things were removed to deal with zpectrometer data. Either a new class (e.g. zrows_index_section__define.pro) will need to be written (presumably just a simply copy from this) or perhaps this class could be made smarter so that it can switch internally between Zpectrometer data and regular GBTIDL data.

The zpectrometer unit tests should be included in the regular nightly GBTIDL unit tests.

• init_gbtidl : auto-compilation of zpectrometer routines
• contrib (all 3 of these are new)
  • zdc_to_dc.pro
  • zoshow.pro
  • zshow.pro
• plotter
  • show_header.pro : this is a bug fix already found in the main trunk
• toolbox/io - We will try and merge this code to avoid code bloat and enhance the class heiarchy as time permits.
  • fits__define.pro : added new type (J) to fits_type_to_idl_type
  • io_sdfits_z__define.pro : new
  • rows_index_section__define.pro : this will need to be handled carefully
  • z_index__define.pro : new
  • z_index_section__define.pro : new
  • z_row_info : new
  • z_sdfits__define.pro : new
  • z_sdfits_row_group__define.pro : new
  • zfits__define.pro : new
• tests
  • z_unit_tests.pro : new
  • data/io : all of these are new
    • JZTEST.raw.zpec.fits
    • JZTEST.raw.zpec.index
    • TAPI_06.raw.zpec.fits
    • TAPI_06.raw.zpec.index
    • zpec.scan1.fits
    • zpec.scan1.index
    • zpec.scan1and2.fits
    • zpec.scan1and2.index

4.1.2 Changes local to the zpectrometer branch of sparrow

Most of these are new and can be moved as is over to the main branch. A few will need more careful work in merging to ensure that all functionality remains intact. None of this work should take very much time.

• gbt/api/project/data/JZTEST (entire tree is new)
• gbt/api/project
  • src
    • Scan.py : addition of Zpectrometer to list of known backends
    • ScanData.py: new getProcName method and self.inProcName internal
    • init.py : new import of ZpectrometerScanData
    • ZpectrometerScanData.py : new
  • test
    • AllTests.py : addition of ZpectrometerScanDataTests
    • ScanDataTests.py : additional tests of getProcName
    • ZpectrometerScanDataTests.py : new
• gbt/api/zpecrometer: all new, move as is
• gbt/app/sdfits - We will try and merge this code to avoid code bloat and enhance the class heiarchy as time permits.
  • SDFITSWriter.py: this is where all of the careful merging needs to happen
  • sdfits.py: new "zpec" backend type and checks
  • ZBinaryTableHDU.py: new
  • ZPrimaryHDU.py
  • ZpectrometerIndexWriter.py
  • data: various data files differ, may or may not be relevant
  • test/AllTests: includes ZpectrometerIndexWriterTests
  • test/ZpectrometerIndexWriterTests.py: new

4.2 New organization of /home/sdfits

• All products of the online sdfits daemon will be in /home/sdfits/[project] where [project] is the name of the project being filled. The separate directories, /home/sdfits-locks and /home/sdfits-flags, are not necessary. The flag files will be created in the same directory as the output FITS files and the lock files are no longer used. The permissions of the project subdirectory should be the same as the parent, /home/sdfits.
• The SDFITS_DATA environment value now points to the root directory (still /home/sdfits) and not where the output file is.
• OnlineSdfits.SetProject needs to check on the existence of /home/sdfits/[project] and create it when necessary. The outfile and logfile.
• OnlineSdfits.createFlagFile should create the empty flag file in the same directory.
• The environment variables SDFITS_LOCK and SDFITS_FLAGS are no longer necessary. These may not even be set anywhere.
• The "Zpectrometer" needs to be added to the self.sdfitsBackends dictionary as 'Zpectrometer':'zpec'
• The GBTIDL function io_sdfits_line::get_online_files needs to look in all of the directories in /home/sdfits. This turns out to be relatively easy to do in IDL. FILE_SEARCH can be used to get a list of all fits files for each backend. FILE_INFO can be used to find modification times on all of those files in a single call. That array of times can be sorted and the most recently modified fits file for each backend can be found. Should that proved to be too slow, a fallback position would be to have the online sdfits daemon keep an ascii file up to date with the most recently modified FITS file for each backend type. GBTIDL could then read that file to determined what get_online_files should return.
• A 'zpec' keyword should be added to get_online_files to return the most recent online zpectrometer fits file.
• The disk cleanup utility that keeps the disk space used in /home/sdfits (on host datacapture) below a limit will have to be modified to handle the new directory structure. This code is in DiskClearner.py and diskclearn.py in sparrow/gbt/daemon/onlinesdfits.
• The GBTIDL online procedure should use the same SDFITS_DATA environment variable value used by the online sdfits daemon. This will need to be set in the GBTIDL startup script in use in Green Bank in the same way that the online sdfits daemon gets that environment variable (i.e. the variable itself is only defined in one place that both tools use).

5. Logistical details during testing and deployment

• SDFITS_DATA environment variable is set in either /home/gbt/etc/config/system.conf for the production system or at /home/sim/etc/config/system.conf for the simulator.
• Prior to release, all of /home/sdfits needs to be re-organized to follow the new structure. This can be done in a separate location and then copied over just prior to release.
• The data rate of the Zpectrometer should be no larger than that of the spectrometer and may often be less so the current space limits in /home/sdfits should be adequate. The nightly job tries to keep the total disk usage in /home/sdfits below about 42 GBytes. It does that by removing the oldest (by modification time) project's files and then re-checking the total disk usage. It continues to remove files until the usage is below the limit. It would be useful to modify the nightly job to log it's output somewhere (it currently just e-mails a notice to BobGarwood and PaulMarganian) and to also provide tools to extract statistics from those logs such as the mean amount (size) of data removed each night and the typical time that a project's files remain on disk before they are removed by the nightly job. These statistics are not a required part of this MR.

6. Deployment Checklist

• What documentation needs to be updated: The GBTIDL release notes will be updated to report that limited zpectrometer support is now available through code added to the contrib directory. The sdfits release note should say something similar. This is still not yet intended for end users and so there is little yet to say in the documentation.
• Notification Needed: all of the people who need to know these details are part of the process of writing and reviewing this MR. No additional notification other than what is described above regarding documentation is needed.
• A description of the changes from the user's point of view should be included in the GBTIDL and SDFITS release notes. Most users should not notice any difference although the change in directory structure should be pointed out so that users who do care about the details will know where the data can now be found.

7. Test Plan

Critical!! ChangeControlCommittee will be reviewing these. Please send a link to this MR to RonMaddalena along with a suggestion for two reviewers when it is ready for CCC review.

Don't forget to include/acquire any additional GBT test time needed outside integration/regression testing! Get your requests in early!

Important! If possible, you should conduct as many of your tests as possible in offline modes and/or with a simulator. We should constantly endeavor to minimize our use of telescope time for testing!

7.1 Internal Testing

• Unit Testing is already covered as part of the requirements
• Testing of the refactored online filler should be able to be first done with Unit Tests using the 'playback' mechanism used in sparrow/gbt/daemon/onlinesdfits.
• Testing of the refactored online filler should also be done using the M&C simulator.
• Testing of the refactored online filler should also be done using the real system, with the exception of the antenna (Antenna Simulator should be acceptable).

7.2 Sponsor Testing

The sponsor will run a variety of routines from Andrew Baker's zpectrometer suite to insure that all functionality is still in place. This will be done both on data filled by the online filler and data filled offline.

7.3 Integration/Regression Tests

* We might want to have someone from outside the GBTIDL and Monctrl groups test out the 'online' command in GBTIDL to make sure that there are no permissions issues. * Integration tests should simply insure the old sdfits code is not broken,

Signatures

APPROVED: I acknowledge that my request is fully contained in this MR, and if the SDD delivers exactly what I specified, I will be happy.

ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.

Written	- BobGarwood - 27 Aug 2007
Checked	- PaulMarganian - 27 Aug 2007
Approved by Sponsor	- KarenONeil - 28 Aug 2007; - AndrewBaker - 3 Sep 2007
Approved by CCC	- RonMaddalena - 10 Sep 2007
Accepted/Delivered by Sponsor	- AndrewBaker - 2 Oct 2007

Symbols:

• Use %X% if MR is not complete (will display )
• Use %Y% if MR is complete (will display )

CCC Discussion Area

-- BobGarwood - 27 Jul 2007