NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Data > DishReengineeringNotes

Changes | Index | Contents | Search | Statistics | Jump to Topic

Dish Reengineering Notes

1. Introduction

What is DISH? Why do we want to reengineer it?

1.1 In Scope

Stage 1:

• The new command line environment for DISH will be Python instead of glish.
• All routines will be decomposed to operate on an SDrecord.
• SDrecords will be created from whatever data format is to be supported upon import.
• The operations will not occur ON a MeasurementSet, but a MeasurementSet can be produced as a means of data export.
• Data access will be cleanly segmented from algorithms (and this will be appropriately documented).
• We will apply the concept of applications, wrappers and libraries. Work will focus on gathering code for libraries together.
• Our build will not require all of aips++, just the required modules from the toolbox.

Stage 2:

• Based on functionally atomic DISH components operating on an SDrecord, an OO design will be refined
• Refactoring/reimplementation as necessary to support the OO design
• Algorithms will implement two interfaces: one for command-line access and the other through a GUI. The environment will be integrated (as per DEAP).

1.2 Out Of Scope

The next generation DISH will not do the following:

• Generic plotting, flagging and editing (this will instead inherited from the DEAP package, maintained separately)
• Examinations of data quality prior to analysis (the assumption is that other utilities are responsible for this)

1.3 Proofs of Concept

• Must call a new Python module from within glish. This will require a C-callable interface "bridge" which is obviously not a permanent solution, but will provide the ability to preserve dish from user's perspective while migrating the "guts". Steps:
  • Identify at least a handful of atomic operations on the wiki.
  • Refactor existing code (if necessary) to expose at least one atomic operation.
  • Extract/reimplement said operation as a stateless C function.
  • Expose new C function to python.
• Then, call that same Python module from within the DISH environment (should be exactly the same as previous)
• Once all DISH components are in Python, we can turn off the glish command line and turn on the python command line, eliminating the need for the glish-python bridge
• At this point, environment should perform better (no C middlemen)

2. Technical Basis

2.1 Design Constraints

• Move away from glish and to python (a la ALMA)
• Number crunching will be done in C/C++ (note that all Python MLab and Numeric number crunching is actually done in C below the surface)
• Has to be compatible with the concept of an algorithm repository, in other words:
• All code must be segmented in two layers, data access and algorithm. As an example, take a look in the /home/gbt7/sparrow/sparrow/src/gbt/python/src/algorithms directory. You will notice that, for example, RonsAlgorithms.py contains only the algorithms for calibrating the four scenarios, not any way to access the data. It assumes that the data has been accessed already.
• Should be able to build it without building all of the aips++ interferometry modules.

2.2 Approach

• Migrate code "bottom up" - first we will get the building blocks, the lowest common denominator of functionality, then identify best ways to knit them together

2.3 Other Technical Requirements

• Sparrow must be deployable independent from the M&C system and the rest of M&C software.

X. Issues to Resolve

• We also have to consider, somewhere, the ALMA vision of being able to provide calibrated data products in our archive. We will have to address this, and it would be powerful to have an answer to this question that is uniform regardless of the users' choice of export format (ie. encapsulate the algorithm properly).

-- NicoleRadziwill - 13 Jan 2004