GB . Data . DishDesign

-- BobGarwood - 10 Feb 2004

This is an attempt to record the design and functionality in dish. It has gone through several designs - with each step building on the previous and thereby creating a fair amount of confusion and pain for users as well as maintainers.

What does dish do?

Dish is a collection of functions that share a small bit of common state (like the data source and sink, a default SdRecord on which to operate, a plotter, and a few common controlling parameters). It has gone through several distinct "design" phases - many of which were not very well coordinated, especially with respect to what was already there so that the end result is a much more confusing place than it needed to be.

SdRecord operations

The bulk of the operations in dish work on SdRecords. Many of them work on a shared SdRecord named "globalscan1". Usually, globalscan1 will be the same as the most recently displayed SdRecord in the dish plotter. The types of operations are listed here and the specific functions are listed elsewhere. Note that dish is inconsistent in how it updates other header values in the process of performing these operations. In many case (e.g. averaging) things like the integration time and system temperature should be adjusted appropriately so that the result is as internally consistent as the data going into the operation.

Math

addition, subtraction, multiplication, division of two SdRecords - channel by channel
scaling of an SdRecord (multiplication or addition of a scalar quantity to all channels)
smoothing (boxcar, hanning, gaussian) - optionally with an appropriate reduction in number of channels (decimation)
averaging many SdRecords with a single call
average the two polarizations in a single SdRecord to produce a new SdRecord
fold a SdRecord on itself (useful for frequency switched observations)
get a sub-region (range of channel numbers)
get a single polarization

data editing

replacing "bad" channels with "better" values (clipping, interpolation)
non-destructive flagging (the values are hidden, not overridden)

data modelling (all optionally interactive with the plotter)

baseline fits (polynomials and sinusoids)
multiple gaussian component fitting (this needs to be flexible enough so that things like the relative positions or relative heights of all or some of the components can be fixed during the fit).
removing previously fit components (baseline, gaussian, etc)
galaxy profile modelling (re-implementation of Arecibo IDL routine)
users will want general modeling capabilities - something dish doesn't provide.

informational

header summary
coordinate conversions (e.g. equatorial to galactic, frequency to velocity, etc)
statistics over a region (moments, mean, rms, peak, etc)
display (plotter capabilities are described elsewhere)

data source and sink interactions

The original design for dish had a single type of object for data sources and sinks. It's called an SDIterator and it can be attached to a MeasurementSet or an simple aips++ table constructed from an SDFITS file. Unfortunately, for many bulk operations (operations where it is necessary to work on many, many scans in succession, e.g. calibration) it is faster to work directly with the MeasurementSet. Also, the aips++ imaging tool, used to turn calibrated data into an image, only works with a MeasurementSet. It was intended that an SDIterator could also be attached to an aips++ Image cube so that the individual spectra found in it could be operated transparently using the rest of dish, but that goal was never reached.

converting the data into something that an SDIterator can be attached to (e.g. the GBT filler)
opening a data source or sink
fetching SdRecords from a source
saving them to a sink
data selection (which produced a new SDIterator which only had access to data matching the selection)
summary tools for describing what was in the data sources and sinks
writing a SdRecord out in ASCII to a file. Some users find that convenient for their own code.

The dish plotter

The dish plotter is built on top of the pgplotter and so it encompasses all of the pgplot library functions. In addition, it has these capabilities:

displays SdRecords via a simple interface
x,y vector pairs
overlay of additional SdRecords
optional auto-scaling of x and y axes (or leaving them fixed)
zoom and unzoom through mouse interactions
cursor feedback
instant display of cursor position and data value under the cursor
x-axis conversion (frequency to velocity, units, reference frames)
annotation on the plotting surface
hard-copy
optional display of a zero line
optional identification of catalogued lines on displayed frequency axis

Calibration

The intent was to implement the draft SpecLineCalibration memo. In practice, several key details were never implemented. This is primarily available through a single calibration function that fetches the data from its source, decides what type it is, and does all of the steps (determine the cal spectrum, determine Tsys from cal spectrum and Tcal values, reduce reference and signal phase as appropriate, scale by Tsys and also optionally scale to the desired final units). It obviously needs more modularity. There is one calibration function built on top of this and that is primarly interface to the main calibration routine. The function of this higher-level function is to recognize procedures and process all scans in the procedure together. There is one calibration function not covered by this and that is the reduction of position switched data when the user wants to specify an arbitrary "off" spectrum. This is typically used in total power mapping where there are many mapping scans and a few reference scans to be used as "off" scans. The dish design handles this by requiring first that the user calibrate all of the scans as simple total power scans using the main calibration function and then use this secondary function to provide a list of on scans and one off scan to be applied to all of them.

calibrate scans automatically
calibrate specific on scans using a given off scan
optionally fit and remove baselines after calibration (its more efficient in the dish design to combine these steps)
optionally fold frequency switched data
optionally flip the sense of signal and reference indicated in the data (very useful early on when the switching state was often not right, less useful now)
get a set of efficiencies and opacities for a given elevation and frequency (could also include other factors here such as ambient temperature if they were modelled - the current implementation is from early guesses that Ron made).

Whole MeasurementSet operations

calibrate all scans using the dish calibration code
summarize the data
imaging

The dish imaging functions are front-ends to the aips++ imager tool. The imager tool has a large number of possible parameters and the dish imaging functions are intended to simply the use of the imager tool by limiting the amount of control the user has over the behavior of the tool and provide reasonable defaults to many imager controls that most single dish users won't want to change. Its not clear how useful that approach is since I seem to end up leading most dish users interested in imaging on to direct control of the aips++ imager - bypassing the dish imaging functions. It may be simply that these functions don't really encompass the most common imaging tasks.

Miscellaneous

The results manager

The results manager is the bulk of the dish GUI that appears when dish is started. It is intended as a place where results are stored for later reference - by name or simply by selecting them in the GUI. There is a list of variables on the list and some simple controls on the right. Values can be displayed (which happens automatically for SdRecords when they are selected), browsed, copied to the clipboard (which can then be copied from there to the dish calculator, one of the operations available from the dish GUI menubar), and deleted. Their name and associated description can also be changed. This got good receptions during the intial tests but in practice, users seldom us it. That is due to a few things, I think. Users are much more interested in simply typing commands than in using a GUI (something which continues to surprise me). Many of the functions developed after the GUI was developed don't make good use of the results manager and so results often don't make it into the results manager automatically (they can be added by hand). In a sequence of commands, each producing intermediate results, users often (reasonably) don't want to keep the intermidiate results and the results manager isn't flexible enough to allow that - so that either the designer of the code producing the many results chooses not to put anything into the result manager or the results manager quickly fills up with lots of results and its difficult for the user to find the truely interesting ones. Its all done in memory and so it can use up resources. Still, I think its a useful concept since it does allow one to keep track of what has gone on before and "undo" becomes simply find the previous result in the results manager and starting from there again.

Saving and restoring the current state

A few people use this. Most users are probably not aware of it. Dish can be set up to automatically save its state on exit and restore a previously saved state on start up. Users can also choose to save the state to any named filed at any time and restore from a previously savfed state at any time. Unipops has a similar, although much simpler, capability that Unipops users do appreciate and expect to work.

DishCensus

Topic DishDesign . { Edit | Attach | Ref-By | Printable | Diffs | r1.5 | > | r1.4 | > | r1.3 | More }

Revision r1.5 - 04 Mar 2004 - 19:26 GMT - BobGarwood Parents: TWikiUsers > BobGarwood	Content copyright © 1999-2007 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Data.DishDesign moved from lib.DishDesign on 04 Mar 2004 - 19:25 by BobGarwood - put it back


NRAO Home > Green Bank \| Wiki Topic: GB > Data > TWikiUsers > BobGarwood > DishDesign

Changes \| Index \| Contents \| Search \| Statistics \| Jump to Topic