NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Software > IDLDataIO

Changes | Index | Contents | Search | Statistics | Go

IDL Data I/O API

• IDL Data I/O API
• Introduction
• How to Use the IDL Data I/O API in Development
  • Preconditions
  • 1. Working with one file at a time: io_sdfits->set_file, filename [/new_index]
  • 2. Reading in several files all at once : io_sdfits->set_project, dir, [/new_index]
  • 3. Reading in several files, one at a time (expert mode): io_sdfits->add_file, file
  • 4. Reading in spectral line data in real time : io_sdfits_line->set_online, filename
  • 5. Writing spectral line data to an SDFITS file : io_sdfits_writer->write_spectra, spectra
• Known Issues
  • Major
  • Minor
• Performance Benchmarking
• Resources
• Frequently Asked Questions
• Core Team

I want to get my data into IDL quick, so I can start analyzing it immediately. How do I do this?

Introduction

The primary role of the Data I/O API for GBT data processing in IDL is to produce and consume data containers, the IDL structures that users interact with directly. If you have coded your own IDL modules in the past, you may have created an IDL structure to hold your data and its related metadata, particularly if your modules deal with the same types of data repeatedly. This structure is your data container.

This API has been designed to provide to the expert user an abstraction of the data of interest: users should be able to read, write, and navigate through the data without concerning themselves with the details of the data format. Standard users who work with the Unipops emulation layer, called GUIDE, will not need to work with the Data I/O API directly.

The Data I/O commands will not change, even if the data format does change. Because the IDL modules that do the data processing depend on the data containers created by the Data I/O modules, the data processing code will be invariant with respect to typical changes in the data format.

All of the Data I/O modules are written in IDL. No additional languages, or utilities outside of IDL, have been used.

How to Use the IDL Data I/O API in Development

This is a description of how to use the API interface. It is by no means an exhaustive list of all methods available to the API user. The beginner should first try working with method #1, and then #2 if they need to read multiple files at a time.

Preconditions

• data must be in SDFITS format. see DATA.sdfits for instructions.
• we recommend that you use SDFITS in 'raw' data format.
• data files must all be contained in the same directory.
• the object described below, 'io_sdfits' is created from two different classes:
  • io_sdfits = obj_new("io_sdfits_line") ; for spectral line
  • io_sdfits = obj_new("io_sdfits_cntm") ; for continuum

1. Working with one file at a time: io_sdfits->set_file, filename [/new_index]

• create an 'io_sdfits_line/io_sdfits_cntm' object
• call 'set_file, file.fits', where filename is the full path name of the file
• this searches for the index file 'filename.index', and loads it
• if the index file is not found, or is found but is invalid a new index is created
• use of the keyword /new_index forces creation of a new index.
• that's it! but now you can't call 'add_file' with this object every again.
• you can now use 'list' and 'get_spectrums' to navigate and retrieve data

2. Reading in several files all at once : io_sdfits->set_project, dir, [/new_index]

• place all your sdfits files in one empty directory (the files must all be spectral line, or all cont., but not mixed)
• create an 'io_sdfits_line/io_sdfits_cntm' object
• call this object's 'set_project' method, using the above dir as the input
• this searches for the index file 'dir.index', and loads it
• if the index file is not found, or is found but is invalid a new index is created
• use of the keyword /new_index forces creation of a new index.
• you can now use 'list' and 'get_spectrums' to navigate and retrieve data containers

3. Reading in several files, one at a time (expert mode): io_sdfits->add_file, file

• create an 'io_sdfits_line/io_sdfits_cntm' object
• call 'set_file_path' where all data will be found: io_sdfits->set_file_path, path
• call 'add_file' to read data from that file: io_sdfits->add_file, file
• this method will look to see if this file has been added before
  • if it has, it does nothing
  • if it has not, it looks to see if the 'io_sdfits_index' file exists
    • if not, the index file is created
    • if it does exist, the index file is updated with the info from the new file
• you can now use 'list' and 'get_spectrums' to navigate and retrieve data containers, or call 'add_file' again
• to add another file, simpley call 'add_file' again.
• see the flowchart to see the logic of how it works.

4. Reading in spectral line data in real time : io_sdfits_line->set_online, filename

• create an 'io_sdfits_line' object (online mode only works with spectral line)
• make sure that your file really exists, i.e., that the online SDFITS filler seems to be working (see [[][online filler]]).
• use 'set_online' to connect to the online file and it's index file (created by the online filler).
• all subsequent calls to 'list', 'get_spectrums' and other methods internally call 'update', which reads the new lines in the index file into the objects memory. So after a scan has been completed, and this scan has been filled in the SDFITS file, your call to 'list' should list that new scan info.

5. Writing spectral line data to an SDFITS file : io_sdfits_writer->write_spectra, spectra

• create an 'io_sdfits_writer' object
• call 'set_file_path' : io_sdfits_writer->set_file_path, path
• set the name of the file to write to : io_sdfits_writer->set_output_file, outfile
• use 'write_spectra' to write spectral line data to SDFITS file set above: io_sdfits_writer->write_spectra, spectra
  • different extensions are appended to the output file as needed, determined by the data_container's data length
  • an index file 'file.index' is created and updated after writing
  • note that this index file should be distinct from 'infile.index', infile being the name of the file you are reading from, so reading/writing can take place in the same directory * you can now use 'list' and 'get_spectrums' to navigate and retrieve data containers from this SDFITS file you are writing to.

Known Issues

Last Modified Nov 22 2004

Major

• SDFITS size limits: trouble creating index files for files larger then 500 MB. Try using multiple small files.
• Writing Continua to SDFITS format currently not supported.

Minor

• Further testing needed for sdfits files not generated with our sdfits tool (/home/aips++/data/demo/dishdemo).

Performance Benchmarking

TBD

A table should be created here that displays performance times for various actions involving the I/O modules. The purpose of this table would be to track changes in performance, clearly mark performance bottle-necks, and illustrate the advantages of using the index file.

nereid.gb.nrao.edu

io_sdfits_cntm_index
rows : 528
size : ?

? s

nereid.gb.nrao.edu

io_sdfits_cntm_index
rows : 528
size : ?

? s

Resources

• View the UML for the IDL Data IO Class structure.
• View the flowchart for the add_file method.
• View the flowchart for the set_project method
• See the User's Refernence Documentation.
• See the Developer's Refernence Documentation.

Frequently Asked Questions

• Q: What can I search for using the 'list' method?
  • A: You can use the names of the index file columns as search keywords with the 'list' method.

• Q: My index file is huge, can I get a summary?
  • A: You can currently use these methods to get an idea of what's in your index file (use /unique keyword): list_index_header, get_index_files, get_index_projects, get_index_procedures, get_index_scans

• Q: Can I use an 'io_sdfits_writer' object the same way as an 'io_sdfits_line' object?
  • A: Yes, the 'io_sdfits_writer' has all the same functionality as 'io_sdfits_line', but with the additional functionality of being able to write to a singe file at once.

Core Team