• • (PRM) - Are the Tcal values inserted by Sdfits just those values needed for the scans being currently filled? If so, do we need to allow the user some way of adding new Tcal's from the RX FITS file later (other then doing it by hand using the Tcal vector from "user observations"?
  • (PRM) - A related question - what about the usage of Sdfits with the append option? Does Sdfits only add those Tcal vector's currently not found in the database? Or, in the interests of keeping it as simple as possible, does Sdfits not touch the database when in append mode?

• • • Only Tcal's covered by the data being filled are inserted into the cal. database.
    • Sdfits does not support adding additional Tcal's after the data has been initially filled (append mode not supported).
    • The cal. database is implemented in a FITS file.

• • • All Tcal values for a given Rcvr are copied to the calibration database.
    • If new data is being added to an sdfits file, then if a new Rcvr is used, this data will also be copied to the cal. database.
    • The cal. database is implemented in a FITS file, separate from the Sdfits file.

• • (PRM) Again, the same issue regarding append mode, and adding more weather results later (see above).

• • (PRM) What kind of metadata needs to be stored along with the actual opacities in cal. database? Should we record what project/scans these opacities are associated with, for example?

• • • Only tau's covered by the data being filled are inserted into the cal. database.
    • Sdfits does not support adding additional tau's after the data has been initially filled (append mode not supported).
    • Interface with forecast tool is optimal: can be called directly from python, and return results similar to desired format.
    • The cal. database is implemented in a FITS file.

• • • Only tau's covered by the data being filled are inserted into the cal. database. Sdfit smust understand the bands being used and grab the time series of opacties vs. frequencies appropriate.
    • Interface with forecast tool is optimal: most likely the data needed will already exist in a file that simply needs to be found and parsed. The contents of the file most likely will be coefficients for polynomial fits fo past weather and look ahead predictions.
    • The cal. database is implemented in a FITS file, separate from the Sdfits file.

• • (PRM) - Are the elevations and frequencies placed in this array just those values that are applicable to the data being filled by Sdfits? If so, do we need any additional functionality as described for the Tcal's from the RX FITS file (see above)?

• • • Only aperture efficiencies for the freq's and elevations covered by the data being filled are inserted into the cal. database.
    • Sdfits does not support adding additional aperture efficiencies after the data has been initially filled (append mode not supported).

• • • Aperture efficiencies vary coarsly with frequency and elevation, so that it would not bloat the cal. database simply to record all ap. efficiency elevations, for a given frequency, and frequencies vary roughly per receiver.
    • With the above assumption, the ap. efficiencies may need to be written to the cal. database only once per project's receiver.

• • • Work Estimate: 5 FTE Days.

• • • Work Estimate: 2 FTE Days.

• • (PRM) - Are the Tcal values inserted by Sdfits just those values needed for the scans being currently filled? If so, do we need to allow the user some way of adding new Tcal's from the RX FITS file later (other then doing it by hand using the Tcal vector from "user observations"?
  • (PRM) - A related question - what about the usage of Sdfits with the append option? Does Sdfits only add those Tcal vector's currently not found in the database? Or, in the interests of keeping it as simple as possible, does Sdfits not touch the database when in append mode?
  • (PRM) - This applies to both Sdfits and GBTIDL items, and is closely related (I imagine) to Bob's question about expected data volumes. The actual format chosen for the database will affect work estimates:
    • FITS - this might be the easiest to implement, though I don't have a lot of experience with libraries for editing FITS files.
    • Relational DB (e.g. MySQL) - this wouldn't be to much of a challenge either, though I haven't seen any very good DB libraries for IDL.
    • ASCII file (e.g. the Index file) - this would take the most work, if the experience with the flagging file can be used as a metric.
  • PRM's work estimate, with best-case assumptions:
    • Only Tcal's covered by the data being filled are inserted into the cal. database.
    • Sdfits does not support adding additional Tcal's after the data has been initially filled (append mode not supported).
    • The cal. database is implemented in a FITS file.
    • Work Estimate: 2 FTE Days.

• • (PRM) This could be pretty straight forward if CLEO came with functions that could be called directly from python (e.g. Sdfits app) that give the tau in the format specified below. Or if these could be looked up in a relational DB. I haven't used the forecast tool in almost a year, so I'm not really sure what's possible right now. Things might take longer if the interaction was more complex (i.e. system calls, file searches, etc.).
  • (PRM) What are the granularities of the CLEO results? Will determining which results to use for a data's time span be complicated, or relatively straightforward?
  • (PRM) Again, the same issue regarding append mode, and adding more weather results later (see above).
  • PRM's work estimate, with best-case assumptions:
    • Only tau's covered by the data being filled are inserted into the cal. database.
    • Sdfits does not support adding additional tau's after the data has been initially filled (append mode not supported).
    • Interface with forecast tool is optimal: can be called directly from python, and return results similar to desired format.
    • The cal. database is implemented in a FITS file.
    • Work Estimate: 5 FTE Days.

• • (PRM) - Are the elevations and frequencies placed in this array just those values that are applicable to the data being filled by Sdfits? If so, do we need any additional functionality as described for the Tcal's from the RX FITS file (see above)?
  • (PRM) - Again, a work estimate would be dependent on what kind of interface the lookup table has, or how it is implemented. For instance, a relational DB containing the lookup values would probably be the simplest thing to work with for the Sdfits app, some other implementation (flat files), might take longer.
  • PRM's work estimate, with best-case assumptions:
    • Only aperture efficiencies for the freq's and elevations covered by the data being filled are inserted into the cal. database.
    • Sdfits does not support adding additional aperture efficiencies after the data has been initially filled (append mode not supported).
    • Lookup data is in a relational DB.
    • The cal. database is implemented in a FITS file.
    • Work Estimate: 5 FTE Days.

%META:TOPICINFO{author="JimBraatz" date="1196370120" format="1.0" version="1.1"}% %META:TOPICPARENT{name="JimBraatz"}%

Specifications for GBTIDL Calibration File

Jim Braatz

Overview

This document describes the specifications for a calibration database to be used with GBTIDL. The database is part of the infrastructure and implementation of an improved flux density calibration scheme in GBTIDL. (See the GBTIDL Calibration Design document.) The database will enable calibration using vector Tsys as opposed to scalar Tsys, the current mode. The database will also enable users to store user-determined opacity and aperture efficiency values, and apply them in GBTIDL calibration.

Calibration Database Specifications

Database Contents

• Tcal vector from the RX FITS file * A single entry that will always be present in the calibration database
  • Units in degrees K
  • Tcal must be stored as a function of frequency, for each feed and polarization
  • Tcal from the RX FITS file should be set by the sdfits program and not modified by users
  • Stored separately from user-determined Tcal values
• Tcal vector from user observations
  • Multiple entries are allowed, each covering different frequency ranges
  • Units in degrees K
  • Tcal must be stored as a function of frequency, for each feed and polarization
  • Must merge Tcal values if frequency range overlaps existing entries
• Zenith opacity
  • A 2-D array, a function of frequency and time
• Aperture Efficiency
  • A 2-D array, a function of frequency and elevation

Usage Requirements

• Allow user to add, delete, and inspect user-determined Tcal vectors
• Allow user to inspect Tcal vector from RX FITS file
• Allow user to add, delete, and inspect zenith opacity entries
• Allow user to add, delete, and inspect aperture efficiency entries
• Allow user to reset the Tcal, tau, and ap_eff settings to the default

SDFITS Enhancements

SDFITS (the program) should create the calibration database file and fill it with default values. All modifications should be internal, and no user interface enhancements to the sdfits program are necessary.

• Tcal come from the RX FITS file
• Default opacity values will be taken from weather conditions (cleo forecast)
• Default aperture efficiency will be filled from lookup tables
• No defaults are entered for user-determined Tcal. (If GBTIDL does not find a user-determined Tcal in the calibration database, RX FITS file Tcal values will be applied.)

GBTIDL Enhancements

GBTIDL will require procedures to read and write to the database, as follows:

  1. set_cal, tau=tau, ap_eff=ap_eff, tcal=tcal, /default

     This procedure writes new values into the calibration database.  The parameters tau, ap_eff, and tcal
     are used to enter new calibration information.  Each of these parameters is optional, but at least one 
     must be supplied unless the /default switch is set.  Each parameter can accept either a scalar 
     value, in which case the value is applied to all frequencies, times, and elevations, or it can accept
     a structure, defined as follows:

     tau    : tau.time, tau.freq[], tau.value[freq]

            tau.time is the starting time for which the specified tau values will be applied.
            tau.freq is an array of frequencies for which zenith opacity is specified in tau.value.
            tau.value is the array of opacity values, one for each frequency.  Opacity values used
                      in calibration will be interpolated from this curve by the calibration procedures.

     ap_eff : ap_eff.elv[], ap_eff.freq[], ap_eff.value[freq, elv]

            ap_eff.elv is an array of elevations for which the ap_eff is specified.
            ap_eff.freq is an array of frequencies for which the ap_eff is specified.
            ap_eff.value is the 2-d array of aperture efficiencies, one for each frequency
                       and elevation.

     tcal   : tcal.freq[], tcal.value[n_feed, n_pol, freq]

            tcal.freq is an array of frequencies for which the tcal is specified.
            tcal.value is the array of calibration temperatures, one for each frequency, feed, 
                       and polarization.


  2. get_cal(scan=scan, time=time, /tau, /ap_eff, /tcal)

     This function returns calibration values from the database.  Exactly one of the switches 
     (tau, ap_eff, or tcal) should be set.  If the user requests tau values and there are tau values
     in the database for more than one time, then the user can also specify either a time or a scan 
     number to the get_cal function.  If time or scan number are not specified, the entire tau array
     is returned.  The function returns a structure with the appropriate fields to be fed directly 
     back into set_cal.

-- JimBraatz - 29 Nov 2007