NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Software > Adass2006

Changes | Index | Contents | Search | Statistics | Go

ADASS 2006, Tucson

• BOFS
  • FITS BOF (moderated by William Pence)
• Oral Presentations
  • O1.1 Designing for Peta-Scale in the LSST Database (Invited)
  • O1.5 Astronomical Tiled Image Compression: How and Why
• Posters
  • P2.01 A Customizable Database Server
  • P2.14 Design Features of the Hera Data Analysis Service
  • P2.31 The AISRP Code and Algorithm Libary
  • P2.36 Using a Workflow Engine for Data Reduction
  • P2.38 AstroAsciiData - Table Handling in Python
  • P3.11 Astronomy, Linux & the Problem of Large File Support
  • P3.19 The NOAO NVO Portal: Client-side VO
  • P3.20 The NOAO VO Portal: Overall Design and Implementation
  • P3.42 A Unique Paradigm in Astronomical Grid Computing: The Astro- WISE Distributed Processing System
  • P3.44 Managing Astronomical Data in the Grid through cfitsio
  • P4.01 A Component-based Framework for Radio-astronomical Imaging Software Systems
  • P4.04 Agile Development Process: Delivering a Successful Data Management Platform
  • P4.19 The NOAO High-Performance Pipeline System: Architecture Overview

BOFS

FITS BOF (moderated by William Pence)

• Those interested in keeping up to date with FITS developments should subscribe to fits-bits.
• There are three new world coordinate system (WCS) papers (III, IV, and V) at various draft stages. More info on III and IV can be found at the FITS news page. FYI, Eric Greisen is one of the people working on paper V (on time coordinates).
• Those who create their own extension need to register that extension.
• The proposal for the FOREIGN extension is interesting. "It puts a FITS wrapper about an arbitrary file, allowing a file or tree of files to be wrapped up in FITS and later restored to disk... The motivation for this extension was to allow an implementation based on the FITS multi-extension mechanism to encapsulate and pass non-FITS data." See P2.17, which I have as a handout. Doug Tody is a co-author of poster.
• There was a discussion of some of the restrictions imposed by FITS, e.g. 8 character keyword names, 80 character line lengths. There are a few new keywords, which are being considered for addition to the FITS standard, that focus on these limitations. CONTINUE would allow one to create long lines. CONTINUE is supported by cfitsio. HEIRARCH would allow one to create long keyword names. HEIRARCH is supported by PyFits. Most BOF participants were in favor of some relaxation of restrictions. Some supported more radical changes than these.
• There was also some talk of a proposed image compression propsal and the pros and cons of borrowing from industry when it comes to compression software. FYI, Doug Tody is on the aforementioned proposal.
• Someone mentioned that databases are the data format of the future and not files. Later in the conference, there was much talk about placing data directly into databases.
• Jeroen de Jong (ESO) gave a presentation on FITS and XML. He talked about describing binary FITS table headers in XML.

Oral Presentations

O1.1 Designing for Peta-Scale in the LSST Database (Invited)

• Data Products: images, catalogs, alerts. All data products are archived.
• They will provide open data access for up to 300 simultaneous users.
• Derived data and processing rates - sustained (peak):
  • Processing: 105 (120) TFlops
  • I/O:
    • read: 60 GB/s
    • write: 6 GB/s
  • Bandwidth: 2.5 (10) Gb/s
  • Allowed 0.1% alert publication failure (alerts are one of the data products)
• Must have automated data quality. Must be robust and quickly "fixable." Features dual-redundant systems.
• Raw data, alerts, and meta data are sent from a base station to the archive. These data are processed at both places independently to minimize network traffic. Processed information is not exchanged. Uses existing NSF-funded networks for distribution and transfer of data.
• They project that their computing requirements are within supercomputing technology trends.
• Data is stored as pixels in a database. They are using an open source database for testing. They are also benchmarking two commercial databases - SQLServer and DB2
  • metadata - 675 million rows
  • source - 250 billion rows
  • object - 22 billion rows
• David Fleming is working for the NCSA on the problem of handling these large datasets.
• Plan on being VO-compliant.
• Pipeline components: image processing -> detection -> association <-> moving object
• They will be using a distributed file system. They are currently investigating existing implementations, e.g. Google.
• They will incorporate three tiers of storage: fast disk, slow disk and tape w/ cache.
• Google has petitioned to become a member of the LSST. Microsoft is already a member.
• Believe it or not, the SKA data needs are much grander.

O1.5 Astronomical Tiled Image Compression: How and Why

Posters

P2.01 A Customizable Database Server

• Called My Customizable Server or MCS.
• I wonder if this is a possible replacement for FQL. It supposedly allows one to treat a FITS file as a database.
• http://ross.iasfbo.inaf.it/mcs

P2.14 Design Features of the Hera Data Analysis Service

• Astronomical data analysis service. Enables researchers to immediately being analyzing astronomical data in FITS format over the Internet.
• http://heasarc.gsfc.nasa.gov/hera
• See handout for actual poster content.

P2.31 The AISRP Code and Algorithm Libary

• The AISRP Code and Algorithm Library is maintained and supported by the NASA Applied Information Science Research Program for use by the general scientific community. It is intended to serve as a public library, archive, and distribution center for code, algorithm descriptions, software packages, and associated literature and documentation.
• http://astrophysics.arc.nasa.gov/AISRPCodeLibraryServer/index.html

P2.36 Using a Workflow Engine for Data Reduction

• ESO Reflex: The ESO Reflex application provides a new approach to astronomical data reduction. The innovative integration of a modern graphical user interface and field-proven legacy data reduction algorithms gives the astronomer the best of both worlds: ease of use combined with accurate scientific results. ESO Reflex is based on the state-of-the-art graphical workflow engine Taverna. http://www.eso.org/sampo/reflex/
• Taverna: aims to provide a language and software tools to facilitate easy use of workflow and distributed compute technology within the eScience community. http://taverna.sourceforge.net/
• The Common Pipeline Library (CPL): comprises a set of ISO-C libraries that provide a comprehensive, efficient and robust software toolkit. It forms a basis for the creation of automated astronomical data-reduction tasks (known as "pipelines"). The CPL was developed to standardise the way VLT instrument pipelines are built, to shorten their development cycle and to ease their maintenance. However, it may be more generally applied to any similar application, and the details of the CPL code have been engineered in a way to make the library portable and flexible, as well as minimising external dependencies. http://www.eso.org/observing/cpl/
• EsoRex?: The ESO Recipe Execution Tool. It can list, configure and execute CPL-based recipes from the command line. http://www.eso.org/observing/cpl/esorex.html
• See handout for actual poster content.

P2.38 AstroAsciiData? - Table Handling in Python

• AstroAsciiData? is a Python module to handle ASCII tables. The idea to develop this Python module emerged from the fact that ASCII tables continue to be one of the most popular and widely used data exchange formats in astronomy and probably science in general. Moreover the Python language is developing into some kind of standard programming language in astronomy, and being able to handle this major data exchange format in a nice way within it would would be very advantageous.
• The development of the AstroAsciiData? module is taking place as part of the AstroLib? project, an open source effort to develop general astronomical utilities akin to those available in the IDL ASTRON package.
• AstroLib?: STScI? would like to help develop an ASTRON-like library for Python. These astronomical utility libraries should be built for Python analogous to the ASTRON library for IDL. http://www.scipy.org/AstroLib
  • ASTRON Contents: http://idlastro.gsfc.nasa.gov/contents.html
• http://www.stecf.org/software/PYTHONtools/astroasciidata

P3.11 Astronomy, Linux & the Problem of Large File Support

• PyFits? 1.1 - 64-bit support & compatible with NumPy?
• Author conducted comparison of FITS writers. We should get a copy of this poster.

P3.19 The NOAO NVO Portal: Client-side VO

• VORuby - A set of Ruby modules to help provide access to the Virtual Observatory (VO). Includes support for VOTable, VORegistry, VOEvent, WESIX (an online source extraction service) and more. http://rubyforge.org/projects/voruby/

P3.20 The NOAO VO Portal: Overall Design and Implementation

• Uses Ruby on Rails (implementation language) and PostgreSQL? (database engine).

P3.42 A Unique Paradigm in Astronomical Grid Computing: The Astro- WISE Distributed Processing System

• ASTRO-WISE: Astronomical Wide-field Imaging System for Europe, http://www.astro-wise.org/
• ASTRO-WISE contains a sophisticated distributed processing system that is a fully scalable, distributed processing system that will overcome the huge information avalanche in, for example, wide-field astronomical imaging surveys.
• Distributed Processing Unit (DPU) example - 100% Python, http://dpu.test.hpc.rug.astro-wise.org/status
• Federated File Server (FFS) example - 100% Python, http://ds.astro.rug.astro-wise.org:18000

P3.44 Managing Astronomical Data in the Grid through cfitsio

• New drivers for the cfitsio libraries that interact directly with the GRID filesystem through its different layers, from the grdiftp to the file catalog.

P4.01 A Component-based Framework for Radio-astronomical Imaging Software Systems

• Athol Kemball

P4.04 Agile Development Process: Delivering a Successful Data Management Platform

• They use JIRA too!
• They do functional testing (in addition to unit testing) using Fitnesse - a fully integrated standalone wiki, and acceptance testing framework. http://fitnesse.org/
• See handout for poster content.

P4.19 The NOAO High-Performance Pipeline System: Architecture Overview

• Users a Pipeline Description Language (PDL) - an XML grammar
• See handout for poster content.