• Version 1.0?
• Version 1.1?

• Version 1.0
• Version 1.1

Status Summary: On June 28, 2006 GBTIDL version 2.0.1 was released

Status Summary: On March 28, 2007 GBTIDL version 2.1.1 was released

GBTIDL users will find the GBTIDL home page on Sourceforge the most useful starting point for GBTIDL documentation, bug reporting and tracking and discussion.

GBTIDL users will find the GBTIDL home page on Sourceforge the most useful starting point for GBTIDL documentation, bug reporting and tracking and discussion. The information found on the GBTIDL wiki pages may be out of date.

• Version 2.1
  • Version 2.1.1 (a patch to version 2.1)

Status Summary: On May 17, 2006 GBTIDL version 2.0 was released

Status Summary: On June 28, 2006 GBTIDL version 2.0.1 was released

• • Version 2.0.1 (a patch to version 2.0)

Status Summary: On December 13, 2005 GBTIDL version 1.2.1 (a patch to 1.2) was released

Status Summary: On May 17, 2006 GBTIDL version 2.0 was released

GBTIDL users will find the GBTIDL home page on Sourceforge the most useful starting point for GBTIDL documentation, bug reporting and tracking and discussion.

GBTIDL users will find the GBTIDL home page on Sourceforge the most useful starting point for GBTIDL documentation, bug reporting and tracking and discussion.

• Version 2.0

Status Summary: On August 16, 2005 GBTIDL version 1.1.1 (a patch to 1.1) was released

Status Summary: On December 13, 2005 GBTIDL version 1.2.1 (a patch to 1.2) was released

As a result, all instances in which contributed code has been adapted for use within the GBTIDL package are documented at IDLCodeReuse. This page lists the modules within the GBTIDL product which are, in full or in part, taken from pre-existing code written by astronomers. In most cases, the changes center around adapting the module to use an updated GBT data model, adding GBT as the default position, or making changes so that the module can be documented automatically by IDLDOC, the program used to generate user's reference documentation and developer's documentation.

As a result, all instances in which contributed code has been adapted for use within the GBTIDL package are documented at IDLCodeReuse. This page lists the modules within the GBTIDL product which are, in full or in part, taken from pre-existing code written by astronomers. In most cases, the changes center around adapting the module to use an updated GBT data model, adding GBT as the default position, or making changes so that the module can be documented automatically by IDLDOC, the program used to generate user's reference documentation and developer's documentation.

• Data Container discussion - a discussion about various ways to use the data containers.
• Spectral Line Data Container? - the fields found in a spectrum data container
• Continuum Data Container - the fields found in a continuum data container
• Data Container Translation? - a mapping between three data storage mechanisms and the above

• Data Container discussion - a discussion about various ways to use the data containers.
• Spectral Line Data Container - the fields found in a spectrum data container
• Continuum Data Container - the fields found in a continuum data container
• Data Container Translation - a mapping between three data storage mechanisms and the above

The toolbox is the collection of routines that the end user is expected to use directly. These will share some global variables such as the public data container as described above. These routines are used in constructing frequently-used and general purpose data reduction recipes. Complete documentation is available which describes the contents of the toolbox and usage for each module.

The toolbox is the collection of routines that the end user is expected to use directly. These will share some global variables such as the public data container as described above. These routines are used in constructing frequently-used and general purpose data reduction recipes. Complete documentation is available which describes the contents of the toolbox and usage for each module.

• • Version 1.2.1 (a patch to version 1.2)

• Version 1.2

1. By 1/15/05: Project Plan updated to determine how much additional internal testing is required, when product can be beta tested by external reviewers.

1. By 1/15/05: Project Plan was updated to determine how much additional internal testing is required, when product can be beta tested by external reviewers.

1. Date TBD: Evaluation Release for external beta testers. This will cover frequency switched track, total power track, position switched single beam, and frqeuency switched wrap ("RAP") cases. The code will already have been reviewed internally by the project scientist who will have verified scientific accuracy and consistency, and coverage of GBT observing cases.
2. Date TBD: Iteration Period. Once results from the evaluation release are available, the project plan will be updated again to determine a final release date to the general public, based on what additional work is required. It is expected that GBT coders work closely with beta testers and the project scientist during this time.
3. Date TBD: Production Release. Package becomes available to GBT users.

1. April 2005: Evaluation Release for external beta testers.
2. May 2005: Iteration Period. Once results from the evaluation release were available, the project plan was updated again to determine a final release date to the general public, based on additional work required. It is expected that GBT coders work closely with beta testers and the project scientist during this time.
3. 5/31/05: Production Release. Package became available to GBT users.

• • Version 1.1.1 (a patch to version 1.1)

Status Summary: On July 6, 2005 GBTIDL version 1.1 was released

Status Summary: On August 16, 2005 GBTIDL version 1.1.1 (a patch to 1.1) was released

Status Summary: On May 3, 2005 GBTIDL version 0.0 was released to internal testers only. On May 31, 2005 version 1.0 will be released for general use.

Status Summary: On July 6, 2005 GBTIDL version 1.1 was released

• Version 1.1?

• Data Container discussion? - a discussion about various ways to use the data containers.

• Data Container discussion - a discussion about various ways to use the data containers.

• Continuum Data Container? - the fields found in a continuum data container
• Data Container Translation - a mapping between three data storage mechanisms and the above

• Continuum Data Container - the fields found in a continuum data container
• Data Container Translation? - a mapping between three data storage mechanisms and the above

The toolbox is the collection of routines that the end user is expected to use directly. These will share some global variables such as the public data container as described above. These routines are used in constructing frequently-used and general purpose data reduction recipes. Complete documentation is available which describes the contents of the toolbox and usage for each module.

The toolbox is the collection of routines that the end user is expected to use directly. These will share some global variables such as the public data container as described above. These routines are used in constructing frequently-used and general purpose data reduction recipes. Complete documentation is available which describes the contents of the toolbox and usage for each module.

See the release notes:

• Version 1.0?

• Shared Data Container? - holds fields common to spectral line and continuum containers
• Spectral Line Data Container? - holds fields unique to spectral lines
• Continuum Data Container? - holds fields unique to continua

• Data Container discussion? - a discussion about various ways to use the data containers.
• Spectral Line Data Container? - the fields found in a spectrum data container
• Continuum Data Container? - the fields found in a continuum data container

Functions in the toolbox sense what type of data is in a container and react accordingly.

Functions in the toolbox sense what type of data is in a container and react accordingly. There is no wiki description of the fields in the shared data container (users tended to find the distinction more confusing than useful so all fields found in each container are now shown in each table linked above).

GBTIDL users will find the GBTIDL home page on Sourceforge the most useful starting point for GBTIDL documentation, bug reporting and tracking and discussion.

Status Summary: On May 3, 2005 GBTIDL version 0.0 was released to internal testers only. On May 31, 2005 version 1.0 will be released for general use.

GBTIDL User's Guide?

See All Documentation

This page describes the GBTIDL project. To learn how to use the program, see the GBTIDL documentation

Setting Up to Use IDL?

The GBTIDL project is IN PROGRESS and the beta release date has not yet been announced. Please do not attempt to install or use any of the code until the proper time, or you can expect unexpected circumstances!

GBTIDL has been released to internal testers only. On May 31, 2005 version 1.0 of GBTIDL will be released for general use.

As a result, all instances in which contributed code has been adapted for use within the GBTIDL package are documented at IDLCodeReuse. This page lists the modules within the GBTIDL product which are, in full or in part, taken from pre-existing code written by astronomers. In most cases, the changes center around adapting the module to use an updated GBT data model, adding GBT as the default position, or making changes so that the module can be documented automatically by IDLDOC, the program used to generate user's reference documentation and developer's documentation.

As a result, all instances in which contributed code has been adapted for use within the GBTIDL package are documented at IDLCodeReuse. This page lists the modules within the GBTIDL product which are, in full or in part, taken from pre-existing code written by astronomers. In most cases, the changes center around adapting the module to use an updated GBT data model, adding GBT as the default position, or making changes so that the module can be documented automatically by IDLDOC, the program used to generate user's reference documentation and developer's documentation.

• Team should focus on frequency switched track, total power track, and position switched single beam. In response to the external committee, these three items will be included in the recipes to be supported in the first evaluation release.

• Team should focus on frequency switched track, total power track, and position switched single beam. In response to the external committee and considering the frequency of execution of observing types on the GBT, the recipes to be supported in the first evaluation release are frequency switched track, position switched single beam, nod (instead of total power track), and in-band frequency switching.

• Team should adopt Bania's modules as is. There was a suggestion that since Tom Bania's modules are accurate, well-tested, and trusted, they should be used as is. In following this recommendation, the team has used Bania's code directly or as a functional template wherever possible.

• Team should adopt Bania's modules as is. There was a suggestion that since Tom Bania's modules are accurate, well-tested, and trusted, they should be used as is. In following this recommendation, the team has used Bania's code directly or as a functional template wherever possible. When departures from this approach have proved necessary, it has been to provide additional scalability, maintainability, and security, or to aid in deployment. The greatest of these, however, is the requirement to write modules that will work well for all Standard Observing Modes. The approach that has been taken, then, is to start with Bania's modules and retrofit them to the updated GBT data model and data container management system, while preserving any algorithms which have already been extensively tested. We note that the Bania's package was the overall model and impetus for NRAO's GBTIDL project.

• Limited version should be ready for distribution to testers by December 1. Because of the revised overall timeline for the project, the date for the limited version (the evaluation release) is also being revised. Note that generalizing software for production use is a complex task that surpasses the bounds of simply preparing algorithms for data analysis. The software team must consider the build process, deployment of machines, supporting users who do not have their own IDL licenses, security, maintainability, and extensibility, among other things. This extensive list of concerns precludes any release to external reviewers for testing during the requested time frame. Additionally, the Project Scientist, RonMaddalena, is responsible for pre-testing the package to ensure that all major goals have been addressed. Because of the GBT support schedule over the winter, there is only limited time to perform this function, and this is more adequately being addressed in the updated project plans.

• Limited version should be ready for distribution to testers by December 1. Because of the revised overall timeline for the project, the date for the limited version (the evaluation release) is also being revised. Note that generalizing software for production use is a complex task that surpasses the bounds of simply preparing algorithms for data analysis. The software team must consider the build process, deployment of machines, supporting users who do not have their own IDL licenses, security, maintainability, and extensibility, among other things. We feel that it will be most productive if we address these issues before we attempt to release the product for any external testing to avoid initial frustrations and to ensure that what is being tested is representative of the future product. The development group does agree that comments from key external testers at a "pre-beta" stage are essential and are working to arrange this. The arrangements for external evaluation testing will be determined when the project plan is finalized in December.

• Team should adopt Bania's modules as is. There was much concern that since Bania's modules are accurate, well-tested, and trusted, they should be used as is. In following this recommendation, the team has used Bania's code directly or as a template wherever possible. Departures from this approach have been the result of several other considerations that are being recognized during development, which include scalability, maintainability, security and deployment issues. The greatest of these, however, is the requirement to write modules that will work well for all Standard Observing Modes. Historically, contributions from scientists work exceedingly well for the observing modes they were intended, but yield unexpected and incorrect results for similar observing modes with subtle differences. The approach that has been taken, then, is to start with Bania's modules and retrofit them to the updated GBT data model and data container management system, while preserving any algorithms which have already been extensively tested.

• Team should adopt Bania's modules as is. There was a suggestion that since Tom Bania's modules are accurate, well-tested, and trusted, they should be used as is. In following this recommendation, the team has used Bania's code directly or as a functional template wherever possible.

• Limited version should be ready for distribution to testers by December 1. Generalizing software for production use is a complex task that surpasses the bounds of simply preparing algorithms for data analysis. The software team must consider the build process, deployment of machines, supporting users who do not have their own IDL licenses, security, maintainability, and extensibility, among other things. This extensive list of concerns precludes any release to external reviewers for testing during the requested time frame. Additionally, the Project Scientist, RonMaddalena, is responsible for pre-testing the package to ensure that all major goals have been addressed. Because of the GBT support schedule over the winter, there is only limited time to perform this function, and so it is not feasible to bring this package to the external committee for review until the original mid-February date.

• Limited version should be ready for distribution to testers by December 1. Because of the revised overall timeline for the project, the date for the limited version (the evaluation release) is also being revised. Note that generalizing software for production use is a complex task that surpasses the bounds of simply preparing algorithms for data analysis. The software team must consider the build process, deployment of machines, supporting users who do not have their own IDL licenses, security, maintainability, and extensibility, among other things. This extensive list of concerns precludes any release to external reviewers for testing during the requested time frame. Additionally, the Project Scientist, RonMaddalena, is responsible for pre-testing the package to ensure that all major goals have been addressed. Because of the GBT support schedule over the winter, there is only limited time to perform this function, and this is more adequately being addressed in the updated project plans.

• Perception that this is an interim solution NRAO will throw away. This is not the case. There will always be GBT users who wish to perform their data analysis using IDL, and so the IDL package will have a lifetime. It is important to control the scope of development, however, so that software development resources can at some point shift their focus to longer-term issues such as imaging and pipeline processing.

• Perception that this is an interim solution NRAO will throw away. This is not the case. There will always be GBT users who wish to perform their data analysis using IDL, and so the IDL package will have a long lifetime, during which support will be provided by the GB Software Development Division. It is important to control the scope of development, however, so that software development resources can at some point shift their focus to longer-term issues such as imaging and pipeline processing.

• Online mode must operate in real time, and not one scan behind. This is due to several issues within the control system which must be addressed, but can not be done as part of the scope of this project. Note that all GBT "real-time" data displays only plot when all contents of a subscan are stored to FITS files.

• Online mode must operate in real time, and not one scan behind. This issue is external to IDL development, and will be addressed as soon as possible but not as a part of IDL development work. Note that it affects all real-time data displays in GB, which only plot when all contents of a subscan are stored to FITS files.

• Team should adopt Bania's modules as is. There was much concern that since Bania's modules are accurate, well-tested, and trusted, they should be used as is. However, there are several other considerations that all software development groups at NRAO must develop according to; these include scalability, maintainability, security and deployment issues. The greatest of these, however, is the requirement to write modules that will work well for all Standard Observing Modes. Historically, contributions from scientists work exceedingly well for the observing modes they were intended, but yield unexpected and incorrect results for similar observing modes with subtle differences. The approach that has been taken, then, is to start with Bania's modules and retrofit them to the updated GBT data model and data container management system, while preserving any algorithms which have already been extensively tested.

• Team should adopt Bania's modules as is. There was much concern that since Bania's modules are accurate, well-tested, and trusted, they should be used as is. In following this recommendation, the team has used Bania's code directly or as a template wherever possible. Departures from this approach have been the result of several other considerations that are being recognized during development, which include scalability, maintainability, security and deployment issues. The greatest of these, however, is the requirement to write modules that will work well for all Standard Observing Modes. Historically, contributions from scientists work exceedingly well for the observing modes they were intended, but yield unexpected and incorrect results for similar observing modes with subtle differences. The approach that has been taken, then, is to start with Bania's modules and retrofit them to the updated GBT data model and data container management system, while preserving any algorithms which have already been extensively tested.

The GBTIDL development project is being conducted to provide GBT users with a reliable and well-documented means to reduce GBT data which has been taken in a subset of the GBT Standard Observing Modes. The work is being heavily based on Tom Bania's GBTIDL package for reducing 3He data, which he developed in late 2003 and early 2004, but includes a revised data model and several design upgrades to yield a system which will handle the majority of GBT's Standard Observing Modes.

The GBTIDL development project is being conducted to provide GBT users with a reliable and well-documented means to reduce GBT data which has been taken in a subset of the GBT Standard Observing Modes. The work is being heavily based on Tom Bania's GBTIDL package for reducing 3He data, which he developed in late 2003 and early 2004, but includes a revised data model and several design upgrades to yield a system which will handle the majority of GBT's Standard Observing Modes. This documentation is being updated frequently as lessons are learned and code is modified; changes are currently being requested by the project scientist who is conducting first beta tests, and we expect that additional changes will be requested once external observers become more involved.

2.1 Goals

2.1 Phases

1. 8/15/04 - 9/30/04: Initial Design Phase. Ideas captured and documented; proof-of-concept coding conducted.
2. 10/1/04 - 11/15/04: Initial Development Phase.
3. 11/15/04 - 1/15/05: Early Beta Testing by RonMaddalena.
4. By 1/15/05: Project Plan updated to determine how much additional internal testing is required, when product can be beta tested by external reviewers.
5. 1/15/05 - TBD: Second Round Early Beta Testing by Ron.Maddalena.
6. Date TBD: Evaluation Release for external beta testers. This will cover frequency switched track, total power track, position switched single beam, and frqeuency switched wrap ("RAP") cases. The code will already have been reviewed internally by the project scientist who will have verified scientific accuracy and consistency, and coverage of GBT observing cases.
7. Date TBD: Iteration Period. Once results from the evaluation release are available, the project plan will be updated again to determine a final release date to the general public, based on what additional work is required. It is expected that GBT coders work closely with beta testers and the project scientist during this time.
8. Date TBD: Production Release. Package becomes available to GBT users.

2.2 Goals

2.2 Approach

2.3 Approach

2.3 Scope

2.4 Scope

2.4 Code Reuse

2.5 Code Reuse

Given a solid Science Data Model, it is possible to accomplish one of the other goals of IDL development at GB: being able to bring data into or out of IDL at any stage of data analysis will no loss of information.

Given a solid Science Data Model, it is possible to accomplish one of the other goals of IDL development at GB: being able to bring data into or out of IDL at any stage of data analysis will no loss of information. Until issues are resolved with the continuum aspects of SDFITS, it is not possible to save and export continuum data without any loss of information. This is a key issue that must be resolved before these utilities are made available to the public.

The user of GBTIDL operates on spectrum and continuum data objects. These data objects are sliced and diced in various ways Data containers are produced by the Data I/O modules, and manipulated by the functions in the toolbox.

The user of GBTIDL operates on spectrum and continuum data objects which are held in data containers. Data containers are produced by the Data I/O modules, and manipulated by the functions in the toolbox. A spectrum object consists of a shared data container plus a spectral line data container. A continuum object consists of a shared data container plus a continuum data container. These distinctions should be largely transparent to users operating at the GUIDE layer, in any case. The following links describe the contents of each container:

The data container? holds the data and associated information necessary to fully describe that data. For the moment, the data container? will be a simple IDL structure. The exact contents of a data container? are still quite fluid. Initially, the data containers? will simply be the contents of one row of an sdfits file as produced by the sdfits tool.

• Shared Data Container? - holds fields common to spectral line and continuum containers
• Spectral Line Data Container? - holds fields unique to spectral lines
• Continuum Data Container? - holds fields unique to continua
• Data Container Translation - a mapping between three data storage mechanisms and the above

Functions in the toolbox will be able to tell what type of data is in a container and react accordingly.

Functions in the toolbox sense what type of data is in a container and react accordingly.

The Public Data Container is a special instance of a data container. It exists as part of the upper functional layer in this package. Procedures which produce a data container (either by modifying an existing one or creating a new one) will leave that result in the public data container. The plotter will also "watch" this data container and update its display as discussed below.

The primary data container is a special instance of a data container. It exists as part of the upper functional layer in this package. Procedures which produce a data container (either by modifying an existing one or creating a new one) will leave that result in the primary data container. The plotter is attuned to the primary data container, and updates its display as the contents of the primary data container are manipulated by the user.

Include summary information about plotter here

Although extensive work is not intended for the IDL plotter, there are several features which are being integrated for astronomical ease of use, such as the ability to specify the beginning and end number of channels to drop. The user will be able to use the GUIDE plotter or the IDL plotter, depending on their level of expertise.

4.6 Plotter

4.6 Data Display

Include summary information about plotter here

Getting Started?

THIS PROJECT IS IN PROGRESS AND IS ONLY AT THE INTERNAL TEST STAGE. Please do not expect to use any of the stated functionality for your data reduction until the release in spring 2004 (possibly March or April but yet to be determined).

see the Old page; see All Files Related to IDL Project

The IDL project at GBT has an important role in long term data analysis development in Green Bank. Because IDL is a rapid development environment, this provides an excellent means of generating requirements for the contents of a single dish analysis package in an exploratory manner - with the added benefit of providing users with a solid means of reducing their data while long-term developments are underway. Another key

The IDL project at GBT has an important role in long term data analysis development in Green Bank. Because IDL is a rapid development environment, this provides an excellent means of generating requirements for the contents of a single dish analysis package in an exploratory manner - with the added benefit of providing users with a solid means of reducing their data while long-term developments are underway. A critical part of this project is to generate a data standard, which provides "a frame of reference that encourages confidence between interacting parties."

The data input/output module will allow clients to conveniently and abstractly retrieve/store desired spectral line and continuum data from/to files on disk. Initially, the input and output formats will be compatible with the SDFITS convention. The input will be the output of the sdfits tool. It may be desirable to have an internal format for ease use by programmers and because some information used internally in IDL may not translate well to SDFITS. If that is the case, it will be possible to preserve all of the astronomically interesting information in an SDFITS file. Data I/O will be generalized so as to decouple SDFITS-specific storage concerns from general purpose input/output functionality.

The foundational Data I/O functionality allows clients (not end users) to conveniently and abstractly retrieve and store desired spectral line and continuum data from or to files on disk. The primary client is the GUIDE layer itself. Initially, the input and output formats will be compatible with the SDFITS convention, where the input is any valid GBT SDFITS file (which are currently produced by the SDFITS generator). Additional data formats may be supported in the future, possibly by contributions from users. Data I/O has been generalized so as to decouple storage concerns specific to an export data format from general purpose input/output functionality. This means that if new or different import/export data formats are to be supported, the only changes required would be at the Data I/O level, with no changes to the algorithms themselves. This eliminates the possibility of introducing new errors when data formats are added or changed.

Also, the use of a read/write cache will be employed so as to minimize the number of actual accesses to disk. This cache will be optimized for typical usage scenarios, such as repeated access to the same data and anticipatory access to related data.

Immediate access to the most recent online data will be possible without any additional "filling" or other user-initiated translations of the data. Any such steps will happen automatically behind the scenes to be available to any user of this package.

Within the Data I/O functionality, the use of a read/write cache has been employed to minimize the number of actual accesses to disk. This cache is optimized for typical usage scenarios, such as repeated access to the same data and anticipatory access to related data.

Specifications for the index file? are available.

Specifications for the index file are available.

4.5 Spectrum and Continuum Data Objects

4.5 Spectrum and Continuum Data Objects; Data Containers

Need to change this from a "will do this" paragraph to a "here's what is needed and why.

The plotter will offer expected features such as those available with the dish plotter. It will also feature a "sync/hold" toggle. In sync mode, the plotter will plot the contents of the Public Data Container. In particular, as the Public Data Container changes, so too will the plot. In hold mode, the plotter will stop responding to changes in the Public Data Container and will simply display its current contents until it is released from hold. This is useful for stringing together a series of computations on the Public Data Container where only the final result is worth plotting; i.e. don't plot each and every transformation on the Public Data Container if doing so isn't desired. Finally, the plotter will permit manual interaction for plotting non-Data Container quantities such as x,y plots. This usage implies the setting of the hold mode.

The IDL project at GBT has an important role in long term data analysis development in Green Bank. Because IDL is a rapid development environment, this provides an excellent means of generating requirements for the contents of a single dish analysis package in an exploratory manner - with the added benefit of providing users with a solid means of reducing their data while long-term developments are underway. Another key

A Science Data Model is a description of the data objects and their interrelationships, which does not specify things like what the names of keywords should be, or what the output format will look like. From a single science data model, there can be several Export Data Formats (EDFs) produced which are representations of the Science Data Model. For example, an SDM for single dish analysis will effectively describe a spectrum and all its characteristics, and relate that spectrum to information about the observer and project that produced the spectrum. This spectrum may be exported in FITS format, or ASCII format, or any other format, but the structure and contents of the spectrum are consistent from one representation to the next. As new observing modes are developed and commissioned (e.g. for observing with bolometers), the SDM is augmented with backward compatibility in mind, ensuring that new modes do not break existing software.

Because algorithms are coded based on the SDM and not a particular export data format, a robust GB SDM is the key to being able to share software across NRAO and accomplish Observatory-wide software goals. So in addition to the benefit it will provide by establishing a clear structure to code to, for developers both internal and external to NRAO, producing the Science Data Model is a core activity for Observatory-wide software development.

Given a solid Science Data Model, it is possible to accomplish one of the other goals of IDL development at GB: being able to bring data into or out of IDL at any stage of data analysis will no loss of information.

It is important to be able to run any general IDL procedure in the context of GBTIDL. For example, if an astronomer does not like a fitting routine provided in GBTIDL, he or she should be able to override this function with another module. Also, items from GBTIDL should be useful within other IDL builds if one of its algorithms is desired for use elsewhere.

It is important to be able to run any general IDL procedure in the context of GBTIDL. For example, if an astronomer does not like a fitting routine provided in GBTIDL, he or she should be able to override this function with another module. Also, items from GBTIDL should be useful within other IDL builds if one of its algorithms is desired for use elsewhere. These two capabilities are critical for the GBTIDL product, since flexibility is a primary desire for many observers, and essential for expert IDL users.

Occasionally, an astronomer will be doing cutting edge research using observing modes which are not supported all the way through software, from the control system through data analysis. In the past, these people have written their own data analysis routines, many of which were done in IDL. To most effectively advance the state of the science, it should be straightforward to integrate the newly developed capabilities into the main GBTIDL distiribution. Software development staff would retain the responsibility for retrofitting algorithms to the GB SDM if required, and making changes for maintainability, reliability, and scalability.

There are two competing interests in designing this package. First, following the UniPOPS? approach, most commands should be executed with a small amount of typing and with few arguments. Arguments which are likely to be constant over several invocations (e.g. channels to drop when plotting data) should not need to be typed each time the command is invoked.

There were two competing interests in designing this package, both of which were required to be satisfied in some way. First, following the UniPOPS approach, most commands should be executed with a small amount of typing and with few arguments. Second, arguments which are likely to be constant over several invocations (e.g. channels to drop when plotting data) should not need to be typed each time the command is invoked.

There are several issues involving IDL global variables, however, that make this approach less appealing to programmers. There are some issues involving IDL system globals (e.g. their fixed size) that make them difficult to use and for which we are still exploring solutions. As a consequence, the GBTIDL package is being constructed in two layers. The low layer is a function library where all arguments are explicitly given in each call. Function names may also be somewhat longer so that they convey more complete information about their intended uses. A second layer, built on top of this, provides the ease of use that UniPOPS? users expect. This layer is fairly thin, and its main purpose is to seamlessly integrate the various globals and automatically used quantities following the UniPOPS? model.

There are several issues involving IDL global variables, however, that make the UniPOPS approach less appealing to programmers. There are some issues involving IDL system globals (e.g. their fixed size) that makes them difficult to use. As a consequence, the GBTIDL package is constructed in two layers. The low layer is a function library where all arguments are explicitly given in each call. Function names may also be somewhat longer so that they convey more complete information about their intended uses. A second layer, built on top of this, provides the ease of use that UniPOPS users expect. This layer is fairly thin, and its main purpose is to seamlessly integrate the various globals and automatically used quantities following the UniPOPS model.

By following this approach initially, we were able to defer some of the decisions regarding the best use of globals until later as we developed the low level functional library. It is also somewhat more likely that the low level library will be easier to maintain and it will be easier for users to substitute other functions in place of the ones provided by this package since these functions will be entirely self contained.

By following this approach initially, we were able to defer some of the decisions regarding the best use of globals until later as we developed the low level functional library. This approach also makes the low level library easier to maintain, and easier for users to substitute other functions in place of the ones provided by this package since these functions are entirely self contained.

The user of GBTIDL operates on spectrum and continuum data objects (also called data containers). Data containers are produced by the Data I/O modules, and manipulated by the functions in the toolbox.

The user of GBTIDL operates on spectrum and continuum data objects. These data objects are sliced and diced in various ways Data containers are produced by the Data I/O modules, and manipulated by the functions in the toolbox.

THIS PROJECT IS IN PROGRESS AND IS ONLY AT THE INTERNAL TEST STAGE. Please do not expect to use any of the stated functionality for your data reduction until the release in spring 2004 (possibly March or April but yet to be determined).

Why are we doing this, what is overall goal of project

The GBTIDL development project is being conducted to provide GBT users with a reliable and well-documented means to reduce GBT data which has been taken in a subset of the GBT Standard Observing Modes. The work is being heavily based on Tom Bania's GBTIDL package for reducing 3He data, which he developed in late 2003 and early 2004, but includes a revised data model and several design upgrades to yield a system which will handle the majority of GBT's Standard Observing Modes.

The project encompasses work on all of the following:

What project encompasses. SDFITS, toolbox, i/o, basic plotter functionality

• SDFITS, a GBT export data format which must be expanded to accommodate some of the Standard Observing Modes
• a GUIDE layer, which provides simple commands for operations like importing and exporting data, manipulating stacks, and performing arithmetic operations on stacks
• a Data I/O layer, used by GUIDE, which cleanly segments the data processing in the GUIDE layer and the toolbox from the data format (thus making it easier to change or add new import and export data formats)
• a Toolbox, which contains modules for operating on arrays, continuum objects and spectrum objects
• an IDL Plotter, which has been enhanced to include a few select features of use to astronomers

The GBTIDL package is being considered a "short-term" solution to helping users reduce their GBT data taken in several of the Observing.StandardObservingModes. It is important to note that it is "short-term" only in the sense that the formal development process is intended to end in the spring of 2004, so that NRAO software development staff can start working on longer-term data analysis issues such as imaging and pipeline processing. It is fully expected that the GBTIDL package will have its own lifetime, and will require maintenance and support from NRAO staff indefinitely. However, the NRAO software development staff will not actively pursue growing the IDL codebase to include newly developed observing modes, such as for the Penn Array. Further development of the package will rely on contributions from scientists both internal and external to NRAO.

The GBTIDL package is being considered a "short-term" solution to helping users reduce their GBT data taken in several of the Observing.StandardObservingModes. It is important to note that it is "short-term" only in the sense that the formal development process is intended to end in the spring of 2004, so that NRAO software development staff can start working on longer-term data analysis issues such as imaging and pipeline processing. It is fully expected that the GBTIDL package will have its own lifetime, and will require maintenance and support from NRAO staff indefinitely. However, the NRAO software development staff will not actively pursue growing the IDL codebase to include newly developed observing modes, such as for the Penn Array. Further development of the package will rely on contributions from scientists both internal and external to NRAO. In some cases, project scientists already have intentions to develop their prototype data analysis software in IDL, meaning that capabilities will be made available in IDL as a result. This will be managed on a case by case basis.

There are many IDL resources available for data reduction adyra

There are many IDL resources available for data reduction, some of which are part of the IDL distribution, and some which can be found in collections such as the IDL Astronomy User's Library. In addition, many astronomers have written their own IDL modules for special purpose data reduction. For the GBT in particular, astronomer Tom Bania developed his own package in IDL to reduce his 3He data, and shared this with the software development team as a basis for continued development. Because there is such a rich code base of IDL work that has already been done for astronomy, one of the key goals of the GBTIDL project has been to leverage as much existing code as possible instead of creating it from scratch.

IDLCodeReuse

As a result, all instances in which contributed code has been adapted for use within the GBTIDL package are documented at IDLCodeReuse. This page lists the modules within the GBTIDL product which are, in full or in part, taken from pre-existing code written by astronomers. In most cases, the changes center around adapting the module to use an updated GBT data model, adding GBT as the default position, or making changes so that the module can be documented automatically by IDLDOC, the program used to generate user's reference documentation and developer's documentation.

http://wwwlocal.gb.nrao.edu/GBT/DA/gbtidl/devel/user/index.html

3.1 Response to Workshop Reviewers

3.1 Response to Workshop Participants

• Limited timeframe for completion by February 15, 2005 inappropriate. The planning team for the IDL project at GB considered this carefully and decided that due to the holidays, other pressures on the external committee who so graciously are participating in the review of this project, and the need to deliver a solid product to observers, that the deadline would be relaxed.

• Limited timeframe for completion by February 15, 2005 inappropriate. The planning team for the IDL project at GB considered this carefully and decided that due to the holidays, other pressures on the external committee who so graciously are participating in the review of this project, and the need to deliver a solid product to observers, that the deadline would be relaxed. A project plan is being constructed in December 2004 to cover all the additional items determined necessary by the Project Scientist after early beta testing, and this will be used to establish an official release date.

• Need a clear block diagram of the architecture. Completed in November; see above.

• Need a clear block diagram of the architecture. Completed in November; see below.

• Team should adopt Bania's modules as is. There was much concern that since Bania's modules are accurate, well-tested, and trusted, they should be used as is. However, there are several other considerations that all software development groups at NRAO must develop according to; these include scalability, maintainability, security and deployment issues. The greatest of these, however, is the requirement to write modules that will work well for all Standard Observing Modes. Historically, contributions from scientists work exceedingly well for the observing modes they were intended, but yield unexpected and incorrect results for similar observing modes with subtle differences. The approach that has been taken, then, is to start with Bania's modules and retrofit them to

• Team should adopt Bania's modules as is. There was much concern that since Bania's modules are accurate, well-tested, and trusted, they should be used as is. However, there are several other considerations that all software development groups at NRAO must develop according to; these include scalability, maintainability, security and deployment issues. The greatest of these, however, is the requirement to write modules that will work well for all Standard Observing Modes. Historically, contributions from scientists work exceedingly well for the observing modes they were intended, but yield unexpected and incorrect results for similar observing modes with subtle differences. The approach that has been taken, then, is to start with Bania's modules and retrofit them to the updated GBT data model and data container management system, while preserving any algorithms which have already been extensively tested.

The initial production version of GBTIDL will be dependent upon a solid SDFITS format available for import into the IDL package.

The initial production version of GBTIDL will be dependent upon a solid SDFITS format available for import into the IDL package. Details for what additions are needed are currently at the main SDFITS page, SdfitsRequests and SdfitsVersionTwo. These are currently being distilled to generate a list of what additions are needed.

In addition to the comprehensive documentation that is available for observers, there are also several data reduction recipes available to show you how certain data analysis operations are carried out within IDL. This is an excellent place to begin to see examples of how the GUIDE layer and the toolbox can be used together to generate scientific results.

Talk about longevity of product; it is not intended to be a throwaway solution

The GBTIDL package is being considered a "short-term" solution to helping users reduce their GBT data taken in several of the Observing.StandardObservingModes. It is important to note that it is "short-term" only in the sense that the formal development process is intended to end in the spring of 2004, so that NRAO software development staff can start working on longer-term data analysis issues such as imaging and pipeline processing. It is fully expected that the GBTIDL package will have its own lifetime, and will require maintenance and support from NRAO staff indefinitely. However, the NRAO software development staff will not actively pursue growing the IDL codebase to include newly developed observing modes, such as for the Penn Array. Further development of the package will rely on contributions from scientists both internal and external to NRAO.

For a complete listing of all the wiki and non-wiki pages pertaining to this project and the GBTIDL package, please refer to IDLContents.

There are many IDL resources available for data reduction adyra

http://wwwlocal.gb.nrao.edu/GBT/DA/gbtidl/devel/user/index.html

• Package should be expert friendly, not novice friendly. The planning team agrees that the package should be friendly to experts, but does not agree that novice users should be overlooked.

• Package should be expert friendly, not novice friendly. The planning team agrees that the package should be friendly to experts, but does not agree that novice users should be overlooked. The intent is to make this package useful for all observers who require data analysis capabilities.

• Perception that this is an interim solution NRAO will throw away. This is not the case. There will always be GBT users who wish to perform their data analysis using IDL, and so the IDL package will have a lifetime. It is important to control the scope of development, however, so that resources can at some point shift their focus to

• Perception that this is an interim solution NRAO will throw away. This is not the case. There will always be GBT users who wish to perform their data analysis using IDL, and so the IDL package will have a lifetime. It is important to control the scope of development, however, so that software development resources can at some point shift their focus to longer-term issues such as imaging and pipeline processing.

• Online mode must operate in real time, and not one scan behind. This is due to several issues within the control system which must be addressed, but can not be done as part of the scope of this project.

• Online mode must operate in real time, and not one scan behind. This is due to several issues within the control system which must be addressed, but can not be done as part of the scope of this project. Note that all GBT "real-time" data displays only plot when all contents of a subscan are stored to FITS files.

• Team should adopt Bania's modules as is. There was much concern that Bania's modules were accurate, well-tested, and should be used as is. However,

• Team should adopt Bania's modules as is. There was much concern that since Bania's modules are accurate, well-tested, and trusted, they should be used as is. However, there are several other considerations that all software development groups at NRAO must develop according to; these include scalability, maintainability, security and deployment issues. The greatest of these, however, is the requirement to write modules that will work well for all Standard Observing Modes. Historically, contributions from scientists work exceedingly well for the observing modes they were intended, but yield unexpected and incorrect results for similar observing modes with subtle differences. The approach that has been taken, then, is to start with Bania's modules and retrofit them to

Put new block diagram here.

The index file, created or validated when a dataset is bound to a data i/o object, is stored in the present working directory from which the user launched IDL. This file contains a map of where the data is located within the input data file, so that you can import only the data you wish to manipulate into IDL, thus saving processing time.

The index file contains a map of where the data is located within the input data file, so that you can import only the data you wish to manipulate into IDL, thus saving processing time. It is created or validated when a dataset is bound to a data i/o object, and is stored in the present working directory from which the user launched IDL.

Specifications for the index file? are available.

Specifications for the index file? are available.

The toolbox is the collection of routines that the end user is expected to use directly. These will share some global variables such as the public data container as described above. These routines can be used in constructing frequently-used and general purpose data reduction recipes.

The toolbox is the collection of routines that the end user is expected to use directly. These will share some global variables such as the public data container as described above. These routines are used in constructing frequently-used and general purpose data reduction recipes. Complete documentation is available which describes the contents of the toolbox and usage for each module.

click here to download

%META:FILEATTACHMENT{name="IDLAPAR3.jpg" attr="" comment="Edited/Updated IDL App Architecture" date="1102356507" path="C:\IDLWorkshop\IDLAPAR3.jpg" size="138545" user="NicoleRadziwill" version="1.1"}%

Why are we doing this, what is overall goal of project

What project encompasses. SDFITS, toolbox, i/o, basic plotter functionality

Talk about longevity of product; it is not intended to be a throwaway solution

• The package should be available to reduce data while observations are in progress as well as at an observer's home institution. This will allow for near real-time checking on the integrity of recent observations as well as post processing of data taken hours, weeks, or months before.
• It should use import and export data formats that are 'standards'. This will allow the exchange of data between other astronomical analysis packages. Essentially, the user can overcome any limitations of the package by switching to systems that are better suited to their needs.
• The package should provide ready-made recipes for analyzing the standard GBT observing techniques, backends, observing frequencies. Yet it should allow users the ability to extend the system to non standard observing modes or the development of new data reduction techniques to standard observing modes.
• The system should leverage a typical users familiarity with other data analysis systems. That is, provide a familiar framework so that a new user of the GBT can use their past experiences in data analysis. This should accelerate the scientific productivity of the user and reduce the load on staff in teaching observers a totally new system.
• The devlopers will take advantage of the large IDL user base within the astronomical community. Various public and proivate libraries and algorithms will be folded into the project.
• The growth of the package should come not only from NRAO's efforts but also from algorithms submitted to the NRAO from the user community.
• The package should be developed in concert with other NRAO analysis systems. It should be developed in such a way that algorithms and code can be shared between different systems for different telescopes. The development of the system should use the experiences of other analysis systems as well as be a testing ground for concepts and algorithms that will end up in other developing systems.

• The package will be available to reduce data while observations are in progress as well as at an observer's home institution. This will allow for near real-time checking on the integrity of recent observations as well as post processing of data taken hours, weeks, or months before.
• It will use import and export data formats that are 'standards'. This will allow the exchange of data between other astronomical analysis packages. Essentially, the user can overcome any limitations of the package by switching to systems that are better suited to their needs.
• The package will provide ready-made recipes for analyzing the standard GBT observing techniques, backends, observing frequencies. Yet it should allow users the ability to extend the system to non standard observing modes or the development of new data reduction techniques to standard observing modes.
• The system will leverage a typical user's familiarity with other data analysis systems. That is, provide a familiar framework so that a new user of the GBT can use their past experiences in data analysis. This should accelerate the scientific productivity of the user and reduce the load on staff in teaching observers a totally new system.
• The developers will take advantage of the large IDL user base within the astronomical community. Various public and proivate libraries and algorithms will be folded into the project.
• The growth of the package should come not only from NRAO's efforts but also from algorithms submitted to the NRAO from the user community, and .
• The package will be developed in concert with other NRAO analysis systems. A key design goal is that it be developed in such a way that algorithms and code can be shared between different systems for different telescopes. The development of the system will use the experiences of other analysis systems as well as be a testing ground for concepts and algorithms that can end up in other developing systems.

• A GUI will not be developed; all interactivity will be at the command line.

• A GUI will not be developed; all interactivity will be at the command line. Limited additions will be made to the IDL plotter for better handling of astronomical results.

• Will not initially include advanced imaging algorithms or handle Bolometer data. Will be limited to data produced by the GBT DCR , Spectrometer, and Spectral processor. Processing of polarization data from these backends may also be limited at first.

• Will not initially include advanced imaging algorithms or handle bolometer data. Will be limited to data produced by the GBT DCR , Spectrometer, and Spectral processor. Processing of polarization data from these backends may also be limited at first.

• A complete real-time display will not be initially available. We already know from our IDL experiences that developing some form of real-time display is not out of the question for a latter development cycle.

• A complete real-time display will not be initially available. We already know from our IDL experiences that developing some form of real-time display is not out of the question for a future development cycle.

NOTE: Although the scope will be limited at first, we expect users will develop the package and extend its coverage into many of these areas.

NOTE: Although the scope will be limited at first, we expect that users will conduct their own development and extend the coverage of this package into many needed areas.

• The devlopers will take advantage of the large IDL user base within the astronomical community. Various public and proivate libraries and algorithms will be folded into the project.
• The growth of the package should come not only from NRAO's efforts but also from algorithms submitted to the NRAO from the user community.

• The package will emphasize a 'toolbox' approach of single-purpose analysis steps. A toolbox, when cmbined with IDL's scripting ability, flexibility, and rapid prototyping, should empower users to devlop quickly new, exploratory analysis algorithms.
• Since we are counting on user contributions to enhance the package, we are working on ways to collect contributions and ways to appraise whether they should be part of the core package and supported jointly by NRAO and the contributor, or as a library of user contributions that are provided by NRAO but not supported by NRAO. We are also determining the best ways to distribute these contributions to those that will use the package at their home institutions.

• Will not initially include advanced imaging algorithms or handle Bolometer data. Will be limited to data produced by the GBT DCR , Spectrometer, and Spectral processor. Processing of polarization data from these backends may also be limited at first.
• Data pipelining will also be outside the current scope of the project.
• A complete real-time display will not be initially available. We already know from our IDL experiences that developing some form of real-time display is not out of the question for a latter development cycle.

NOTE: Although the scope will be limited at first, we expect users will develop the package and extend its coverage into many of these areas.

• The IDL Data Reduction project is meant to fill a need of our observers for a basic, interactive single-dish data analysis system.
• The package should be available to reduce data while observations are in progress as well as at an observer's home institution. This will allow for near real-time checking on the integrity of recent observations as well as post processing of data taken hours, weeks, or months before.
• It should use import and export data formats that are 'standards'. This will allow the exchange of data between other astronomical analysis packages. Essentially, the user can overcome any limitations of the package by switching to systems that are better suited to their needs.
• The package should provide ready-made recipes for analyzing the standard GBT observing techniques, backends, observing frequencies. Yet it should allow users the ability to extend the system to non standard observing modes or the development of new data reduction techniques to standard observing modes.
• The system should leverage a typical users familiarity with other data analysis systems. That is, provide a familiar framework so that a new user of the GBT can use their past experiences in data analysis. This should accelerate the scientific productivity of the user and reduce the load on staff in teaching observers a totally new system.
• The package should be developed in concert with other NRAO analysis systems. It should be developed in such a way that algorithms and code can be shared between different systems for different telescopes. The development of the system should use the experiences of other analysis systems as well as be a testing ground for concepts and algorithms that will end up in other developing systems.

• We will not do real time interaction with the telescope, such as calculation of LPCs and LFCs

• A GUI will not be developed; all interactivity will be at the command line.
• There will be no real time interaction with the telescope, such as for determination of LPCs and LFCs.
• No support will be provided for pulsar data analysis.

3.1 Completeness and Integrity of SDFITS as Input Data

3.1 Response to Workshop Reviewers

On October 15, 2004, an IDL Design Workshop was held in Green Bank. A great amount of comments were received by the external participants (Tom Bania, Ed Murphy, Tapasi Ghosh, Tim Robishaw, Chris Salter); these were all considered and woven in to the work plan. A summary of the concerns, recommendations and responses follow:

3.2 A Robust Science Data Model

• Limited timeframe for completion by February 15, 2005 inappropriate. The planning team for the IDL project at GB considered this carefully and decided that due to the holidays, other pressures on the external committee who so graciously are participating in the review of this project, and the need to deliver a solid product to observers, that the deadline would be relaxed.

• Limited version should be ready for distribution to testers by December 1. Generalizing software for production use is a complex task that surpasses the bounds of simply preparing algorithms for data analysis. The