NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Software > ReviewReport2004

Changes | Index | Contents | Search | Statistics | Go

Report on GBT Software Review 30 August 2004

• Report on GBT Software Review 30 August 2004
  • 1. GBT Software development plan
    • Software Issues
    • IDL Support
    • Python Environment
    • GBT-ALMA Collaboration
    • SDM for Single Dish Data
    • Data Capture
    • Quick Look
    • Pipeline Processing
  • 2. Operations Issues
    • Queue based and interactive observing
    • Dynamic scheduling
    • Remote observing
    • Proposal handling
    • Management
  • 3. Review panel comments
    • Ease of Use
    • Issues of concern
    • Follow up review
  • 4. Conclusions

The panel would like to commend the GBT software team on the high quality and thoroughness of the presentations. The plans presented for future development, even given the limited resources available, appear sound and achievable. Since these plans are ambitious they involve some risk.

The following report is divided into two parts: first a summary of key elements of the development plan as discussed in the review meeting, and second, a discussion of specific issues and concerns raised by the panel.

1. GBT Software development plan

Software Issues

The following does not attempt to cover all elements of GBT software. The discussion here is only of those "hot topics" which were discussed explicitly in the review meeting.

IDL Support

IDL is popular with a segment of the GBT user community and is a powerful tool for interactive data exploration by expert users, who often develop their own software. Support for GBT data reduction in IDL will assist user data reduction in the short term, however it is important that this effort be carefully scoped to limit the software cost of trying to support multiple environments and to avoid false expectations on the part of users for future IDL support.

The scope of IDL support for GBT data reduction includes:

• Access to GBT data (the SD SDM) from the IDL environment
• A basic spectral line package, similar to Unipops

In particular, IDL will be used to help develop the Single Dish (SD) Science Data Model (SDM). Interfacing the SDM to at least two environments will help ensure that the SDM remains package neutral. Other non-NRAO packages, e.g., CLASS, may also receive limited support via export of GBT data to the proprietary data format used by the package.

Python Environment

To focus software development resources and simplify things for the user it is desirable to have a standard environment for data reduction and analysis. The standard environment for NRAO data reduction, including for GBT, is Python, with most of the data processing functionality residing in compiled language components which are invoked from Python (or some other application layer software) via a framework. Capabilities for reducing data from all fully supported GBT instruments should be available from within the standard Python-based environment. This includes classical spectral line reductions, even if a separate IDL package is also available. Most GBT observers would thus be able to get a single software distribution to reduce data taken with the GBT, regardless of the instrument or observing mode used.

The reasons for choosing Python as the user-level scripting language are too involved to go into in detail here. Python is open source and freely available, but offers high level language capabilities at least comparable to those provided by proprietary languages such as IDL. Pushing most of the functionality off into compiled language components isolates the bulk of the code from the user interface, and enables scalability. Python is (or will be) used elsewhere within NRAO for pipeline and interactive data processing, and is also widely used elsewhere within astronomy.

Compiled language components can be written in any widely available compiled language, e.g., C, C++, or FORTRAN, and can use any class library internally. Legacy code can be wrapped as components. The SDM will allow components from different origins to inter-operate.

GBT-ALMA Collaboration

ALMA includes a single dish capability which overlaps to some unknown degree with GBT. The ALMA requirements for single dish are not yet clear, at least not to the panel. We need to determine the ALMA requirements for single dish data processing and compare these to those for the GBT. A decision should then be made whether to produce a common single dish SDM for GBT and ALMA, and whether to collaborate on single dish calibration software.

SDM for Single Dish Data

The Science Data Model, or SDM, defines the logical content of a raw or processed observation dataset from an instrument. The SDM is the basis for all subsequent science data post-processing, and is also the form in which data is stored in the archive. When the SDM is represented in a specific data interchange format, e.g., FITS or XML, the result is an Export Data Format (EDF), which captures the information content of the SDM in a particular external data format. The same observation can be represented equivalently in multiple physical internal or external data formats.

GBT requires a SDM and EDF to define the data products produced by the telescope. This must support all current and future GBT instruments, including focal plane arrays and high time resolution pulsar data.

Whether to have combined or separate but complementary SDMs for interferometric and single dish data is TBD. Whether to have the same single dish SDM for GBT and ALMA is TBD but is strongly recommended. If the ALMA requirements for SD are greatly constrained then this may not be necessary, but this is considered unlikely.

A new single dish SDM capable of supporting multiple telescopes, instruments, and data reduction packages is potentially of interest outside NRAO. Production and use of such a data model would be of benefit to all concerned including the general user community. Other single dish observatories such as Arecibo should be invited to participate in the definition of such a science data model.

Data Capture

Data capture is the process of capturing the data stream from an instrument during an observation, and producing a raw observation dataset which is conformant to the SDM. Data capture takes information from the Telescope Domain and produces information for the Science Domain. Hence data capture is the primary interface between these two domains; information is assumed to be represented differently in the two domains, with data capture controlling the conversion. All subsequent "downstream" processing is performed in the Science Domain. Since facilities such as Quick Look need to operate while data is being taken, data capture is normally a near real time, stream and message oriented process. The end product of an observation is a verified raw observation dataset conformant to the SDM, suitable for the archive and for all subsequent post-processing.

Data capture will be implemented for GBT once the SDM has been defined. This will give GBT essentially the same architecture and dataflow as for the other NRAO telescopes, including ALMA. This may require modifying the GBT telescope control subsystems to output metadata in some structured form such as XML, rather than FITS as is currently used (FITS is good for a static export or interchange data format but generally not for anything upstream).

Quick Look

The purpose of Quick Look is to provide a "quick look" at newly acquired data from the perspective of the Science Domain, to provide an end-to-end check that things are working, that valid data is being obtained, and that it can be subsequently processed and used within some actual science program.

It is important to differentiate between telescope and instrument status and monitoring and science domain quick look. Both are important, but they are not the same thing. Telescope and instrument monitoring verify that the hardware is working correctly. Quick look is science domain (following data capture) and provides an immediate check on the validity of the data for subsequent post-processing. This generally requires some canned quick look software which is driven automatically by observing, but export of data to user environments (Python, IDL, Unipops, etc.) can also be useful to permit verification via ad-hoc processing performed by the observer.

Pipeline Processing

Since the standard data processing environment for the GBT will be Python it will be possible to implement pipelines within the Python environment as for ALMA and (eventually) the other NRAO telescopes.

The consensus of the review was that pipeline processing is most needed for GBT instruments which produce large volumes of data, e.g., focal plane imaging arrays, or pulsar surveys. Classical spectral line analysis is more interactive and deals with smaller amounts of data. However, even in this case automated production of spectra could potentially be useful for quick look and for VO data mining of the GBT archive.

2. Operations Issues

Queue based and interactive observing

With few exceptions (e.g. VLBI) Observing with the GBT is currently interactive; astronomers give instructions to the antenna and instruments directly from GUIs at observe time. In the future, queue-based observation scheduling will also be supported. In this mode, previously generated Scheduling Blocks may be selected from a queue and observed. In the near future, Scheduling Blocks will be used for both interactive and queue based observing.

Dynamic scheduling

In order to take advantage of periods of good weather, dynamic scheduling of high frequency observations will be required. The current scheme of primary high frequency observer and secondary low frequency observer will be replaced with a more elaborate system. Dynamic scheduling will include both queue based observing and interactive observing when astronomical judgments are needed at observe time. It is hoped that the ALMA dynamic scheduling software will be usable for the GBT.

Remote observing

The ability for observers to conduct their observations from locations other than the GBT control room is desirable and for dynamically scheduled projects essential for scientists not physically residing in Green Bank. Specification of the software needs for remote observing are on hold due to a lack of scientific input.

Proposal handling

Although the GBT presently has its own proposal handling mechanism, it is clear that they are prepared to move to a system common to all of the NRAO's instruments.

Management

Clearly there are good management mechanisms in place to advance the software development. There is a well-managed list of tasks to be achieved with an assessment of the effort needed to achieve these tasks. The six-week development cycle allows the software engineering staff to be re-assigned to different projects on successive cycles. This encourages the distribution of expertise within the group and provides a more challenging changing environment for the developers. The short cycle also allows quick adjustment to the changing needs of the GBT. There is good scientific direction of the overall priorities.

3. Review panel comments

Ease of Use

The panel supports continuing effort in the "Ease of Use" development, both to make observations easier for outside observers and to reduce the observing pressure on the GB staff. Some good design steps have clearly been made.

Issues of concern

• New capabilities vs supporting existing capabilities. Past policy has been that development of new and especially unique capabilities has priority over maintenance and support of existing capabilities in the assignment of software development resources. Clearly there has been major emphasis on developing new scientific capabilities and the cost of maintaining the existing features. The panel supports the decision to allocate a higher fraction of resources in the next few cycles to fixing known problems. In the longer term, this should be very beneficial - higher reliability and less time spent by software engineers fixing or working around known defects. The panel would like to see continuation of a better balance of these conflicting desires to optimize the total scientific output of the GBT and to ensure that the GBT software, and therefore the GBT itself, does not establish a reputation for unreliability.
• Limited scope of IDL project. The IDL project has the potential to become an unbounded commitment of software development resources. Since the long term software package used for the bulk of GBT data processing will be something else, the panel feels that there should be clearly stated bounds on the commitment to the IDL project. It is the opinion of the panel that there are two purposes of this IDL development, 1) to quickly develop data analysis tools needed and usable by a significant portion of the GBT users and 2) for prototyping of concepts for the official system to be developed and especially in developing a science data model. Jeff Mangum is organizing and effort to coordinate the ALMA and GBT science data models. The transition plan to the python/c based system should be planed in advance. The python/c based system development should be coordinated with similar efforts elsewhere in the observatory.
• Science Data Model (SDM) The panel strongly supports the development of a GBT SDM.
• ALMA - GBT connection Commonality with ALMA is desirable. For data analysis software, compatible science data models are required for the GBT and ALMA. The GBT model is yet to be specified and the ALMA model is still in the process of being clarified so the outcome is TBD. There is urgency for the GBT to have an SDM by the end of 2004.
• Getting continuous advice and feedback from astronomers The day-to-day feedback from the GB scientific staff to the software development team has been less than desirable. This has been especially true for visionary input from GBT users. A tighter relationship between the developers and end users of the GBT is desirable and may need an expansion of the scientific input beyond Green Bank.
• Conflict between interactive and queue based observing. Much of the scientific research being done with the GBT requires a great deal of interactivity between the observer and the telescope. On the other hand, high frequency observations need good weather which is sufficiently rare in Green Bank that the telescope needs to be dynamically scheduled to take advantage of the periods of exceptional weather. In the limit of maximally efficient use of the resources, dynamic scheduling would be queue based with a minimum of, or no, interactivity. The balancing of these needs needs a great deal of attention. However, both Phil and Richard have experience from other telescopes and smooth evolution to a good procedure is anticipated.
• Remote observing. The physical remoteness of the GBT from most of its users requires that most high frequency, i.e. dynamically scheduled projects, will need to be done in a remote mode. Remote observations put some constraints on the design of the interactive tools used in the observations such as intelligence in the client processes. Since the next generation of these tools are currently under development, we strongly recommend that the needs of remote observing be incorporated into their design. A high speed data link to GB will be imperative.
• Monitor Tool. The messages are not time tagged. This makes such messages much less useful for understanding the sequence of events. This deficiency should be remedied.

Follow up review

On a timescale of 6 months, a followup review should check on the progress, especially for the IDL-Python transition of the software and the SDM.

4. Conclusions

The GBT software development project appears to be well managed with realistic, if ambitious, goals. The plans for the GBT are compatible with the overall NRAO e2e goals while trying to optimize the scientific productivity of the GBT. The principal recommendation of the panel is that the GBT software team to continue to pay attention to the long-term consequences of the inevitable trades to be made between conflicting demands on the limited resources.

The top three concerns are:

• Development of a SDM for the GBT.

• More emphasis on maintenance and getting the most science out of the GBT with existing facilities.

• Bounding the commitment of resources to the IDL project.

Panel members:

 John Benson  
 Bill Cotton  (co-chair)
 Allen Farris 
 Ed Fomalont  
 Dale Frail   
 Gareth Hunt  
 Jeff Mangum  
 Tom Morgan   
 Scott Ransom 
 Ken Sowinski 
 Doug Tody    (co-chair)

14 September 2004