!Online Sdfits Design

Online Sdfits Design

• If there are errors coming from the sdfits program, try to replicate them by filling the same scan(s) by hand. If you get the same error messages, then the problem does not lie with onlinesdfits, but with sdfits and/or the datat that is being taken.
• If this log file doesn't reveal any info, try looking for a anything, most likely a Traceback in the log file in /home/gbt/etc/log/datacapture. If there were problems concerning onlinesdfits, like a stale lock file that needs to be removed, that info should reside here.

• If there are errors coming from the sdfits program, try to replicate them by filling the same scan(s) by hand. If you get the same error messages, then the problem does not lie with onlinesdfits, but with sdfits and/or the data that is being taken.
• If this log file doesn't reveal any info, try looking for anything, most likely a Traceback in the log file in /home/gbt/etc/log/datacapture. If there were problems concerning onlinesdfits, like a stale lock file that needs to be removed, that info should reside here.

Trouble Shooting

When you get a call in the middle of the night saying that gbtidl online isn't working or there's a problem with the online filler, how, generally, do you attack the problem? This hasn't happended in a long time, so I'm a bit rusty, but here's a quick brain dump:

• Start up gbtstatus so you can find out the project name and latest scan number.
• Go to the /home/sdfits/project.log file, and see what the errors say, if there are any.
• If there are errors coming from the sdfits program, try to replicate them by filling the same scan(s) by hand. If you get the same error messages, then the problem does not lie with onlinesdfits, but with sdfits and/or the datat that is being taken.
• If this log file doesn't reveal any info, try looking for a anything, most likely a Traceback in the log file in /home/gbt/etc/log/datacapture. If there were problems concerning onlinesdfits, like a stale lock file that needs to be removed, that info should reside here.
• Even if you don't know what the problem is, sometimes just restarting onlinesdfits, and waiting to see if it fills the next scan is all thats needed. To restart it, you can either ask the operator to do this (which they should be able to do easily), or you can:
  • log into datacapture as monctrl
  • source /home/gbt/gbt.bash
  • TaskMaster datacapture status
  • TaskMaster datacapture stop 1
  • TaskMaster datacapture start 1
  • TaskMaster datacapture status

%META:TOPICINFO{author="KarenONeil" date="1144890480" format="1.0" version="1.1"}% %META:TOPICPARENT{name="Sdfits"}%

!Online Sdfits Design

• !Online Sdfits Design
  • Introduction
  • GBTIDL
  • Architecture
  • Software
    • Overview
    • Classes
  • Products
    • Sdfits files
    • Index files
    • Flag Files
    • Lock files
    • Log Files

Introduction

Online Sdfits refers to the daemon, onlinesdfits responsible for filling all DCR, Spectral Processor, and Spectrometer data into Sdfits files. This filling happens at the end of each successfull scan.

GBTIDL

One of the main purposes of onlinesdfits is to provide the GBTIDL program access to current data, and provide the user with a quasi real-time display and data reduction environment. This is the reason why Index and Flag files are created. GBTIDL users cannot modify their FITS or Index files, but they can write to the Flag file, and thus are able to use all the flagging functionality in GBTIDL.

As GBTIDL evolves, the onlinesdfits filler will also have to adapt to meet GBTIDL's needs.

Architecture

• The daemon is named onlinesdfits
• It runs on host datacapture
• The daemon, like other M&C processes, runs as user monctrl.
• The lifecycle of the daemon is controled by TaskMaster, thus making its execution identical to other M&C daemons. TaskMaster reports to the CLEO Message screen whenever it stops or starts this process.
• The daemon writes to the local disk on datacapture, where there are three directories:
  • /home/sdfits : this is where the *.fits files are actually written, as well as the symbolic links to the flag files. Only monctrl has permission to write to this directory.
  • /home/sdfits-locks : this is where lock files are written to (see below). Users have write permissions here.
  • /home/sdfits-flags : this is where users have permission to write flag files (see below). /home/sdfits contains links, created by onlinesdfits, that point to this directory.
• /home/sdfits has only 65 Gigs available. A separate cronjob exists to clean this disk nightly and keep it under 40 Gigs.

Software

Overview

At the heart of the daemon onlinesdfits is a state engine that is capable of detecting scan transitions (when they begin and end), project transitions, and backends selected in the Scan Coordinator. When the end of a scan is detected, the sdfits program is called, via a system call, filling in the scan that just ended, as well as any scans that it may have missed while it was filling previous scans. A system call is used to avoid a memory leak in sdfits, which has since been fixed. This approach needs to be refactored to improve performance.

The state engine, communicating with M&C with Grail, provides the information needed to construct a command string for the sdfits program. This includes the current project, current scan, and backend used. The command used in the system call takes this form:

sdfits /home/gbtdata/project -backends=backend -scans=scan -append -debug /home/sdfits/project > /home/sdfits/sdfitsLog.txt

In addition to appending to an sdfits file to /home/sdfits, both an index file (for spectral line scans only) and a log file are updated at the end of each scan. When an sdfits file is first created, a symbolic link is created in /home/sdfits pointing to a blank GBTIDL flagging file created in /home/sdfits-flags. A lock file is also created and destroyed while the system call is blocking in /home/sdfits-locks. More on this below.

Classes

• Online Sdfits
  • sparrow/gbt/daemon/onlinesdfits.py - entry point a while loop for the daemon
  • sparrow/gbt/daemon/OnlineSdfits.py - main class for the online filler. Contains the state engine, and all logic for producing the correct command string for the sdfits system call. Also responsible for writing lock and flag files.
  • sparrow/gbt/daemon/FlagFileWriter.py - responsible for writing a blank GBTIDL Flag File.

• State Engine
  • sparrow/gbt/api/ygor/src/BackendStateEngine - part of a class heirarchy for catching various transitions of the M&C system. Uses Grail.

• Sdfits
  • sparrow/gbt/app/sdfits - this is where the SDFITSWriter, IndexWriter, and other classes are found that are responsible for the sdfits application.

Products

The online filler produces a number of different files:

Sdfits files

A FITS file is produced for each backend used in a project:

• Specrometer : project.raw.acs.fits
• Spectral Processor : project.raw.sp.fits
• DCR : project.raw.dcr.fits

To avoid the FITS files from being corrupted by an error with sdfits, backups are made before each new scan is filled. If the scan is not succesfully filled, the backup is copied back over to the original file. These backups all are labeled the same as above, except with the *.backup extension.

Index files

For spectral line data, that is, for the Spectrometer and Spectral Processor backends, GBTIDL compatible Index files are created and maintained along with the FITS file. These have the same form as the FITS files, except they have the *.index extension. GBTIDL can also produce Index files - these two pieces of code that must create identical products is something that must be managed carefully.

Flag Files

When a FITS file is first created by sdfits, it creates a link in /home/sdfits to an empty flag file, also created by sdfits in /home/sdfits-flags. This step is necessary so that GBTIDL users can modify the blank flag file (set a flag, for example), which they have permission to since it is in /home/sdfits-flags as opposed to /home/sdfits. Like the Index File, GBTIDL also contains code for producing flag files, and the maintanence of two pieces of code for producing the same product is troublesome.

Lock files

Before the system call to sdfits is made, a lock file is written to /home/sdfits-locks, of the form project.lock. This lock file is destroyed once the system call is done blocking. If onlinesdfits is ready to fill the latest scan, but there exists a project.lock file, it will wait until this lock file is removed before proceeding. There is a time-out period for this waiting, whereupon an excpetion will be raised and onlinesdfits will terminate.

This lock file can be used by any consumer of the FITS file in /home/sdfits, to know when the FITS file is in the process of being written to. The GBTIDL program formerly interacted with onlinesdfits through this lock file, but that practice has been terminated; the use of this lock file with onlinesdfits remains, however.

Log Files

• Each project has file in /home/sdfits named project.log. After every system call to sdfits, the results of the call are appended as a line to this file (ex: Wed Apr 12 20:59:23 2006 : Scan(s) 13, Backend: dcr written.)
• All processes run via TaskMaster, including onlinesdfits maintain a log file. This log file is timestamped, and can be found in /home/gbt/etc/log/datacapture. It contains all the stdout from the process, which, in this case, includes basic info regarding the state engine, as well as all the output from each sdfits call (this is why the -debug flag is used).

-- PaulMarganian - 13 Apr 2006