Current version: sdfits version 1.7 (2007/25/14)

Current version: sdfits version 1.10 (2007/11/14)

See also the current Release Notes.

See also the current Release Notes.

Current version: sdfits version 1.5 (2006/05/17)

Current version: sdfits version 1.7 (2007/25/14)

For latest changes to SDFITS formats, see also the Release Notes .

See also the current Release Notes.

The utilities that produce SDFITS are part of the GB DataPreprocessingAPI. The SDFITS format was chosen for three reasons. First, the format is recognized by many astronomers at various institutions. Second, because the format was identified as a result of discussions extending over a several year period, we were able to leverage the contributions of astronomers internal and external to NRAO on this data format, rather than starting from scratch. Third, we have experts on the subject within NRAO who were able to guide the development process.

The single-dish FITS convention (SDFITS) was chosen for three reasons. First, the format is recognized by many astronomers at various institutions. Second, because the format was identified as a result of discussions extending over a several year period, we were able to leverage the contributions of astronomers internal and external to NRAO on this data format, rather than starting from scratch. Third, we have experts on the subject within NRAO who were able to guide the development process.

• Support More Users. IDL users no longer have to write their own filler to use the package for data analysis.
• Program Checkouts. Already, one visiting observer was able to readily examine the validity of his program on the GBT by examining the SDFITS file output from a program checkout. This could grow into an even more helpful capability in the future.

• Support More Users. Users no longer have to write their own filler to use the package for data analysis systems not supported by NRAO.
• Program Checkouts. User are more readily able to examine the validity of their program on the GBT by examining the SDFITS file output. This could grow into an even more helpful capability in the future.

• Upgradability. GBT will now be able to join the multiple engineering FITS files into one astronomically meaningful data set, and thus are better positioned to adopt emerging standards (perhaps MBFITS).

• Upgradability. The facilities for joining the multiple engineering FITS files into one astronomically meaningful data set are now all grouped in one location and thus we are better positioned to adopt emerging standards.

Defining the Reference Channel

General

From DanaBalser: The FITS keyword CRVAL1 contains the observed frequency for reference pixel CRPIX1 with an increment of CDELT1. The SDFITS file conforms to the FITS standard where pixels correspond to the center of the channel; although the choice of CRPIX1 is completely arbitrary. For example, a CRPIX1 = 2049.00 means that the CRVAL1 in that case corresponds to the frequency at the mid-point of channel 2049.

sdfits is not available for download and installation outside of Green Bank.

Hanning Smoothing Always Off

Only 3 backends are supported by sdfits: Spectrometer, SpectralProcessor, and DCR.

Currently, Hanning smoothing is not applied to all data from the Spectrometer.

Use of any mode except raw is discouraged (e.g. cal and avg). These discouraged modes use calibration algorithms that are not well maintained. User's should instead rely on GBTIDL or their own calibration algorithms and work directly with the raw output from sdfits. These options may be improved in the future, possibly using GBTIDL methods for the calibration and averaging steps.

Documentation on the specific meaning of each column and how each value is constructed from the raw FITS file is missing. This will be improved soon. In the meantime, direct all questions about the details of sdfits to either BobGarwood or PaulMarganian.

The labeling of polarization and feed (beam) number for Ka data is incorrect except for the SIG=T rows.

The frequency values associated with LSFS scans are not correct except for the first integration.

The IF numbering associated with each unique frequency (and used by GBTIDL) is not always the same as that used in GFM.

Other Issues

Pulsar Observing

• Total Power Observations: The sdfits program is not capable of making inferences or performing computations between a pair of scans in a total power "off-on" sequence. (Note that sdfits does record basic beam/switched-beam and procedure/sequencing information, however).

The sdfits program does not correctly handle pulsar data even when it comes an otherwise supported backend.

Future Work

There is still more work to be done with sdfits and the underlying libraries it uses so that they can lend themselves fully to a core set of observation cases. For multi-IF observations, averaging of integrations will require the alignment of frequency axes. For multi-beam data, the automatic generation of beam positions from the beam offsets is desirable. We plan to document these details soon, so that these improvements can be made in the future.

Performance Testing

Comparison with AIPS++ Measurement Set

File Sizes

It is true that sdfits output files are significantly smaller than a "comparable" result from AIPS++. However, the reality is that the results are not comparable. The files resulting from sdfits are a "final delivery" product to data analysis software, whereas AIPS++ output carries much more information which can be reiterated upon by AIPS++ tools. Thus, the comparison is essentially bogus, and we will make no further attempt to compare file sizes.

Timings

Comparing run-times between sdfits and AIPS++ is slightly tricky. The sdfits program has three data processing modes. For each mode, a comparable AIPS++ processing sequence must be found.

• -mode=avg: For averaged (scan-level) data, the sdfits utility plus AIPS++ sdfits2ms is equivalent to gbtmsfiller followed by d.cal on all scans with proc=F followed by d.getc on each IF followed by saving the result with d.save.
• -mode=cal: For calibrated (integration-level) data, the sdfits utility plus AIPS++ sdfits2ms is equivalent to gbtmsfiller followed by d.cal on all scans with proc=F (proc=T will calibrate On/Off pairs).
• -mode=raw: For raw (state-level) data, the sdfits utility plus AIPS++ sdfits2ms is equivalent to running gbtmsfiller and filling all the data.

T_LBAND_26MAR03 (spectrometer data, 596MB)	-mode=avg	-mode=cal	-mode=raw
sdfits	sdfits : 21m 06s (13MB) sdfits2ms : 0m 03s sdfits total : 21m 09s	sdfits : 28m 20s (289MB) sdfits2ms : 0m 49s sdfits total : 29m 09s	sdfits : 32m 41s (578MB) sdfits2ms : 1m 45s sdfits total : 34m 26s
AIPS++	gbtmsfiller : 7m 35s cal 3m 44s getc + save : 5m 03s dish total : 16m 22s	gbtmsfiller : 7m 35s cal : 3m 44s dish total : 11m 19s	gbtmsfiller : 7m 35s

We have made no comparisons involving spectral processor or digital continuum receiver data.

IDL

GBTIDL

UsingIDL illustrates how to work with data from sdfits. In particular, we demonstrate how to work with integration- and state-level continuum data for generating maps.

GBTIDL reads and writes SDFITS files.

Defining the Reference Channel

From DanaBalser: The FITS keyword CRVAL1 contains the observed frequency for reference pixel CRPIX1 with an increment of CDELT1. The SDFITS file conforms to the FITS standard where pixels correspond to the center of the channel; although the choice of CRPIX1 is completely arbitrary. For example, a CRPIX1 = 2049.00 means that the CRVAL1 in that case corresponds to the frequency at the mid-point of channel 2049.

Hanning Smoothing Always Off

Currently, Hanning smoothing is not applied to all data from the Spectrometer.

• The AIPS++ SDFITS Website

• The AIPS++ SDFITS Website

View all sdfits official release notes here.

All sdfits release notes can be found in the list of official release notes.

These links are here for historical reasons so that the sdfits development team can locate them. They are not intended for end users.

Current version: sdfits version 1.4 (2006/03/15)

Current version: sdfits version 1.5 (2006/05/17)

Online Sdfits Design

Current version: sdfits version 1.3 (2005/11/15)

Current version: sdfits version 1.4 (2006/03/15)

Memory Leak

There is a memory leak somewhere in sdfits. Users of sdfits should avoid generating FITS files larger than about 500 MBytes. This is most likely to be a problem in raw mode. Users should consider splitting up their data into multiple FITS files using the scans argument.

Update: Melinda investigated this issue more in-depth in C2 2005. Details are at SDFITSMemoryLeakInvestigationResults.

Hanning Smoothing Always On

Hanning Smoothing Always Off

Currently, Hanning smoothing is applied to all data from the Spectrometer. In the future there will be an option that will allow the user to turn on or off Hanning smoothing.

Currently, Hanning smoothing is not applied to all data from the Spectrometer.

• The calculation of exposure ("EXPOSURE") is not removing blanking time. So presently, the values given for exposure times are actually duration times.

Current version: sdfits version 1.3 (2005/03/31)

Current version: sdfits version 1.3 (2005/11/15)

Current version: sdfits version 1.2 (2005/03/31)

Current version: sdfits version 1.3 (2005/03/31)

For latest changes to SDFITS formats, see also the Release Notes .

View Online Sdfits Filler Test Results

View Online Sdfits Filler Performance

Current version: sdfits version 1.1 (2004/05/27)

Current version: sdfits version 1.2 (2005/03/31)

View Redesign Efforts, '05

Update: Melinda investigated this issue more in-depth in C2 2005. Details are at SDFITSMemoryLeakInvestigationResults.

Frequencies and Velocities from sdfits.

The sdfits convention requires that there be a frequency-like axis. The sdfits tool always writes a frequency axis as the first axis of the DATA coordinate system. This axis is described by CTYPE1, = CRVAL1=, CDELT1, and CRPIX1 values in the FITS binary tables produced by sdfits (other sdfits convention writers may put that axis in a different order and that axis may be a velocity axis, see the references for more information). This section describes how to construct the frequency and corresponding velocity axis using the information provided by sdfits.

FITS pixels are always numbered from 1. The frequency at pixel (channel) i is

    f(i) = CRVAL1 + (i - CRPIX1) * CDELT1

This gives the frequency in the reference frame implied by the last 4 characters of the CTYPE1 value. Currently, sdfits always uses a CTYPE1 value of "FREQ-OBS" which means that this is the topocentric frequency.

The VELDEF column describes a velocity definition (first 4 characters) and a reference frame (second 4 characters). The reference frame is the frame that was doppler tracked for that row. If the reference frame is the same as that implied by CTYPE1 the doppler tracking was off for that data.

The VFRAME column contains the true (fully relativistic) velocity (in m/s) of the reference frame given in VELDEF with respect to the observer. For sdfits output, where the CTYPE1 frame is the observer's frame, shifting the observed frequencies to the doppler tracked reference frame is a simple doppler shift using the VFRAME value.

    f'(i) = f(i) * sqrt((c+VFRAME)/(c-VFRAME))

where c is the speed of light in m/s.

The sdfits output also contains an RVSYS column. This is the true, relativistic, velocity of the source (in m/s) relative to the observer. So, to get the frequencies in a frame at rest with respect to the source, use RVSYS in the previous equation.

Velocities at each channel can then be calculated according to the specific velocity definition desired.

   v_radio/c = 1 - f'/f_rest
   v_optical/c = f_rest/f' - 1
   v_true/c = (1 - (f/f_rest)^2) / (1 + (f/f_rest)^2)

Where the rest frequency, f_rest comes from the RESTFREQ column.

The true, relativistic, source velocity relative to the doppler tracked frame can be recovered from RVSYS and VFRAME.

   v_source = (RVSYS - VFRAME) / (1 - RVSYS*VFRAME / c^2)

And the source velocity in the two velocity definitions is:

   v_radio/c = 1 - sqrt((c-v_source)/(c+v_source))
   v_optical/c = sqrt((c+vsource)/(c-v_source)) - 1

Defining the Reference Channel

From DanaBalser: The FITS keyword CRVAL1 contains the observed frequency for reference pixel CRPIX1 with an increment of CDELT1. The SDFITS file conforms to the FITS standard where pixels correspond to the center of the channel; although the choice of CRPIX1 is completely arbitrary. For example, a CRPIX1 = 2049.00 means that the CRVAL1 in that case corresponds to the frequency at the mid-point of channel 2049.

Hanning Smoothing Always On

Currently, Hanning smoothing is applied to all data from the Spectrometer. In the future there will be an option that will allow the user to turn on or off Hanning smoothing.

Memory Leak

There is a memory leak somewhere in sdfits. Users of sdfits should avoid generating FITS files larger than about 500 MBytes. This is most likely to be a problem in raw mode. Users should consider splitting up their data into multiple FITS files using the scans argument.

Current version: sdfits version 1.1 (2004/05/26)

Current version: sdfits version 1.1 (2004/05/27)

Current version: sdfits version 1.1 (2004/05/26)

Most of the issues mentioned here are to be fixed shortly; see "Upcoming Improvements" which follows this section. At this time, the sdfits program and the underlying libraries it uses only provide averaged frequency axis information. That is, for each sampler of a scan the mean of all recorded LO frequency values is used in determining a frequency axis. This axis is subsequently used in all integrations and sub-integrations for that sampler. This is inadequate for cases where velocity shifts can be substantial from integration to integration, such as with on-the-fly mapping or multi-IF observations. This is also inadequate in the frequency-switched case, where the frequency offset for the "sig" phase is not zero. Beam-switched observing - a technique rarely used these days with the GBT - is also not supported by sdfits.

Beam-switched observing - a technique rarely used these days with the GBT - is not supported by sdfits.

The sdfits program is suited to single IF/single beam/total power observations (other than on-the-fly mapping), such as point source spectroscopy. Additionally, multi-beam observations are somewhat supported by sdfits. That is, beam offsets are provided, but sdfits does not use them to generate beam positions. Rather, the end-user must use the provided beam offsets to calculate beam positions manually (see the GBT Antenna Control Unit FITS File Specification for details). Additionally, support for multi-beam mapping is inadequate for the same reason as for on-the-fly mapping described above.

Multi-beam observations are somewhat supported by sdfits. That is, beam offsets are provided, but sdfits does not use them to generate beam positions. Rather, the end-user must use the provided beam offsets to calculate beam positions manually (see the GBT Antenna Control Unit FITS File Specification for details).

• Switched Power and Total Power Observations: The sdfits program is presently not capable of making inter-scan or intra-scan inferences or computations. It is ignorant of the relationship between integrations and sub-integrations in a beam-switched or frequency-switched scan. It is also ignorant of the relationship between a pair of scans in a total power "off-on" sequence. (Note that sdfits does record basic beam/switched-beam and procedure/sequencing information, however).

Upcoming Improvements

We are currently evaluating what has to be done to sdfits and the underlying libraries it uses in order to facilitate per-integration and per-phase frequency axis information. For frequency-switching, we are also evaluating what needs to be done to allow the separate averaging of "sig" and "ref" phases. These capabilities will ensure that sdfits is useful for on-the-fly mapping, multi-IF work, and frequency-switched observations. We plan to have these improvements in place around the C3/C4 development cycle boundary in mid-May, at which time we will re-announce sdfits and its new capabilities.

• Total Power Observations: The sdfits program is not capable of making inferences or performing computations between a pair of scans in a total power "off-on" sequence. (Note that sdfits does record basic beam/switched-beam and procedure/sequencing information, however).

Even after the upcoming improvements described above are in place, there is still more work to be done with sdfits and the underlying libraries it uses so that they can lend themselves fully to a core set of observation cases. For multi-IF observations, averaging of integrations will require the alignment of frequency axes. For multi-beam data, the automatic generation of beam positions from the beam offsets is desirable. We plan to document these details soon, so that these improvements can be made in the future.

There is still more work to be done with sdfits and the underlying libraries it uses so that they can lend themselves fully to a core set of observation cases. For multi-IF observations, averaging of integrations will require the alignment of frequency axes. For multi-beam data, the automatic generation of beam positions from the beam offsets is desirable. We plan to document these details soon, so that these improvements can be made in the future.

• Astronomer's IDL Library

The sdfits program is suited to single IF/single beam/total power observations (other than on-the-fly mapping), such as point source spectroscopy. Additionally, multi-beam observations are somewhat supported by sdfits. That is, beam offsets are provided, but sdfits does not use them to generate beam positions. Rather, the end-user must use the provided beam offsets to calculate beam positions manually. Additionally, support for multi-beam mapping is inadequate for the same reason as for on-the-fly mapping described above.

The sdfits program is suited to single IF/single beam/total power observations (other than on-the-fly mapping), such as point source spectroscopy. Additionally, multi-beam observations are somewhat supported by sdfits. That is, beam offsets are provided, but sdfits does not use them to generate beam positions. Rather, the end-user must use the provided beam offsets to calculate beam positions manually (see the GBT Antenna Control Unit FITS File Specification for details). Additionally, support for multi-beam mapping is inadequate for the same reason as for on-the-fly mapping described above.

Official Release Notes

View all sdfits official release notes here.

• Switched Power and Total Power Observations: The sdfits program is presently not capable of making inter-scan or intra-scan inferences or computations. It is ignorant of the relationship between integrations in a beam-switched or frequency-switched scan. It is also ignorant of the relationship between a pair of scans in a total power "off-on" sequence. (Note that sdfits does record basic beam/switched-beam and procedure/sequencing information, however).

Spectral Line Observing

Most of the issues mentioned here are to be fixed shortly; see "Upcoming Improvements" which follows this section. At this time, the sdfits program and the underlying libraries it uses only provide averaged frequency axis information. That is, for each sampler of a scan the mean of all recorded LO frequency values is used in determining a frequency axis. This axis is subsequently used in all integrations and sub-integrations for that sampler. This is inadequate for cases where velocity shifts can be substantial from integration to integration, such as with on-the-fly mapping or multi-IF observations. This is also inadequate in the frequency-switched case, where the frequency offset for the "sig" phase is not zero. Beam-switched observing - a technique rarely used these days with the GBT - is also not supported by sdfits.

The sdfits program is suited to single IF/single beam/total power observations (other than on-the-fly mapping), such as point source spectroscopy. Additionally, multi-beam observations are somewhat supported by sdfits. That is, beam offsets are provided, but sdfits does not use them to generate beam positions. Rather, the end-user must use the provided beam offsets to calculate beam positions manually. Additionally, support for multi-beam mapping is inadequate for the same reason as for on-the-fly mapping described above.

Continuum Observing

The sdfits program is well suited for continuum observing. For multi-beam mapping, there is the caveat that mapping would be hindered by the need to manually calculate beam positions. And, as with the spectral line case, beam-switched continuum observing is not supported by sdfits.

Other Issues

• The calculation of exposure ("EXPOSURE") is not removing blanking time. So presently, the values given for exposure times are actually duration times.

• Switched Power and Total Power Observations: The sdfits program is presently not capable of making inter-scan or intra-scan inferences or computations. It is ignorant of the relationship between integrations and sub-integrations in a beam-switched or frequency-switched scan. It is also ignorant of the relationship between a pair of scans in a total power "off-on" sequence. (Note that sdfits does record basic beam/switched-beam and procedure/sequencing information, however).

Upcoming Improvements

We are currently evaluating what has to be done to sdfits and the underlying libraries it uses in order to facilitate per-integration and per-phase frequency axis information. For frequency-switching, we are also evaluating what needs to be done to allow the separate averaging of "sig" and "ref" phases. These capabilities will ensure that sdfits is useful for on-the-fly mapping, multi-IF work, and frequency-switched observations. We plan to have these improvements in place around the C3/C4 development cycle boundary in mid-May, at which time we will re-announce sdfits and its new capabilities.

Future Work

Even after the upcoming improvements described above are in place, there is still more work to be done with sdfits and the underlying libraries it uses so that they can lend themselves fully to a core set of observation cases. For multi-IF observations, averaging of integrations will require the alignment of frequency axes. For multi-beam data, the automatic generation of beam positions from the beam offsets is desirable. We plan to document these details soon, so that these improvements can be made in the future.

• E-ALFA Working Group: SDFITS Formats
• Bob Garwood's SDFITS Summary at ADASS IX

Known Issues

Benefits of SDFITS

The utilities that produce SDFITS are part of the GB DataPreprocessingAPI. The SDFITS format was chosen for three reasons. First, the format is recognized by many astronomers at various institutions. Second, because the format was identified as a result of discussions extending over a several year period, we were able to leverage the contributions of astronomers internal and external to NRAO on this data format, rather than starting from scratch. Third, we have experts on the subject within NRAO who were able to guide the development process.

There are many benefits from being able to generate SDFITS files from GBT data:

• Support More Users. IDL users no longer have to write their own filler to use the package for data analysis.
• Program Checkouts. Already, one visiting observer was able to readily examine the validity of his program on the GBT by examining the SDFITS file output from a program checkout. This could grow into an even more helpful capability in the future.
• Basis for Collaboration. Since other observatories such as Arecibo employ similar formats, we will be able to collaborate with them on improvements in the future.
• Familiarity. Users may already be familiar with SDFITS, which will reduce the learning curve for understanding data coming off the GBT.
• Upgradability. GBT will now be able to join the multiple engineering FITS files into one astronomically meaningful data set, and thus are better positioned to adopt emerging standards (perhaps MBFITS).

Known Issues

Performance Testing

Performance Testing

Comparison with AIPS++ Measurement Set

Comparison with AIPS++ Measurement Set

File Sizes

File Sizes

Timings

Timings

sdfits : 21m 6s (13MB) sdfits2ms : ?m??s sdfits total : ??m??s

sdfits : 21m 06s (13MB) sdfits2ms : 0m 03s sdfits total : 21m 09s

sdfits2ms : ?m??s sdfits total : ??m??s

sdfits2ms : 0m 49s sdfits total : 29m 09s

sdfits2ms : ?m??s sdfits total : ??m??s

sdfits2ms : 1m 45s sdfits total : 34m 26s

gbtmsfiller : ??m??s cal ?m ?s getc + save : ?m??s dish total : ??m??s

gbtmsfiller : 7m 35s cal 3m 44s getc + save : 5m 03s dish total : 16m 22s

gbtmsfiller : ??m??s cal : ?m ?s dish total : ??m??s

gbtmsfiller : 7m 35s cal : 3m 44s dish total : 11m 19s

gbtmsfiller : ??m??s

gbtmsfiller : 7m 35s

Processing SDFITS Data

IDL

UsingIDL illustrates how to work with data from sdfits. In particular, we demonstrate how to work with integration- and state-level continuum data for generating maps.

CLASS

The preprocessing components validated for use to produce SDFITS files are also being used to generate a GBT to CLASS converter.

References

• The AIPS++ SDFITS Website

Core Team

• Known Issues
• Performance Testing
  • Comparison with AIPS++ Measurement Set
    • File Sizes
    • Timings

Known Issues

• Switched Power and Total Power Observations: The sdfits program is presently not capable of making inter-scan or intra-scan inferences or computations. It is ignorant of the relationship between integrations in a beam-switched or frequency-switched scan. It is also ignorant of the relationship between a pair of scans in a total power "off-on" sequence. (Note that sdfits does record basic beam/switched-beam and procedure/sequencing information, however).

Performance Testing

Comparison with AIPS++ Measurement Set

File Sizes

It is true that sdfits output files are significantly smaller than a "comparable" result from AIPS++. However, the reality is that the results are not comparable. The files resulting from sdfits are a "final delivery" product to data analysis software, whereas AIPS++ output carries much more information which can be reiterated upon by AIPS++ tools. Thus, the comparison is essentially bogus, and we will make no further attempt to compare file sizes.

Timings

Comparing run-times between sdfits and AIPS++ is slightly tricky. The sdfits program has three data processing modes. For each mode, a comparable AIPS++ processing sequence must be found.

• -mode=avg: For averaged (scan-level) data, the sdfits utility plus AIPS++ sdfits2ms is equivalent to gbtmsfiller followed by d.cal on all scans with proc=F followed by d.getc on each IF followed by saving the result with d.save.

• -mode=cal: For calibrated (integration-level) data, the sdfits utility plus AIPS++ sdfits2ms is equivalent to gbtmsfiller followed by d.cal on all scans with proc=F (proc=T will calibrate On/Off pairs).

• -mode=raw: For raw (state-level) data, the sdfits utility plus AIPS++ sdfits2ms is equivalent to running gbtmsfiller and filling all the data.

 sdfits        : 21m 6s (13MB)
 sdfits2ms     :  ?m??s
 sdfits total  : ??m??s

 sdfits        : 28m20s (289MB)
 sdfits2ms     :  ?m??s
 sdfits total  : ??m??s

 sdfits        : 32m41s (578MB)
 sdfits2ms     :  ?m??s
 sdfits total  : ??m??s

 gbtmsfiller   : ??m??s
 cal              ?m ?s
 getc + save   :  ?m??s
 dish total    : ??m??s

 gbtmsfiller   : ??m??s
 cal           :  ?m ?s
 dish total    : ??m??s

 gbtmsfiller   :  ??m??s

We have made no comparisons involving spectral processor or digital continuum receiver data.

%META:TOPICINFO{author="DavidFleming" date="1082034180" format="1.0" version="1.1"}% The GBT SDFITS Project