GB . Data . GbtFitsMonitorContinuumRequests

Requests for Continuum Improvements

Main

Requests

Upgrades Strategy

Current GFM improvement falls into three main categories:

1. High-priority fixes for specific problems which affect on-going operations. Examples of this include problems with Tsys calculations, various issues related to which beam is where, inability to produce results from apparently good data, and so on. Genuine emergencies are handled under operational support. Important but not emergency problems are carried over to the next cycle, but are generally made the highest priority for that cycle.

2. Major processing / infrastructural improvements. The main ones currently under consideration are:

Improving the handling of multi-beam data.
Improving the baseline fitting strategies for peak/focus scans, and the treatment of focus scans for which there is insufficient baseline.
Implementing the "definitive" calibration algorithms (this is awaiting production of these).
Improvements to the application of, and user interface into data quality heuristics.

These form the bulk of the work in any given cycle, but progress depend upon resources being assigned to GFM in that cycle.

3. Low priority, generally "cosmetic" improvements. These are fitted in on an as-possible basis.

If a task has been selected for work in a specific cycle, a link will be made to the corresponding modification request.

Prioritization into, and within these categories is currently being performed by RichardPrestage. Please contact him if you have any comments or suggestions!

I have a proposal for an improvement...

If you have a request for a new feature for gfm, please look down the following lists, to make sure it is not already requested. If you don't see it, either add it at the bottom of the appropriate list, or email it to AmyShelton or RichardPrestage. New requests will be reviewed and prioritorized at the start of each cycle.

Useful other links

GbtFitsMonitorBeamSwitching describes proposed upgrades to multi-beam handling.
GbtFitsMonitorHeuristics describes proposed upgrades to the treatment of heuristics.
GbtFitsMonitorBaselines describes proposed upgrades to the handling of baselines, and short focus scans.
GbtFitsMonitorRegressionExamples contains a list of example scans which have caused difficulties in the past.
GbtFitsMonitorBenchmarks contains benchmarking results for the continuum plug-ins.

Requests

Click on the table headings to sort in descending order; click again to resort in ascending order Missing items have been completed...

Item	Priority	Active C3 2005	Task	Sponsor	Lead	Details	Status
02	4 - N/A	N	Complete DCR Plug-in	RichardPrestage	AmyShelton	A functional version exists, but substantial more work is required. See details for more info.
03	4 - N/A	N	Install "correct" calibration	RichardPrestage	AmyShelton	Once the definitive calibration memo becomes available, implement it. This should cover single feed, dual-feed and beam-switched observing modes. details
09	4 - N/A	N	Resolve menu options	RichardPrestage	AmyShelton	details
13	4 - N/A	N	Dynamic plug-ins	RichardPrestage	AmyShelton	details
14	4 - N/A	N	Loading progress bar	NicoleRadziwill	AmyShelton	For large data sets, it would be nice to have a progress bar that appears to show you how much more loading time you should expect.
15	4 - N/A	N	Average and difference f/b	RichardPrestage	AmyShelton	details
17	4 - N/A	N	Handle duplicate scan numbers	RichardPrestage	AmyShelton	details
20	4 - N/A	N	Prepare to use CCB	RichardPrestage	AmyShelton	details
21	2 - Medium	N	Add print capability & fix Postscript export bug	RichardPrestage	AmyShelton	If you save the plot as postscript, then print it (at least to ops4050) the axis labels are outside the plotted area. It would be nice to be able to plot directly to the printer, rather than to an intermediate file.
23	4 - N/A	N	Force data reduction when GO FITS File incomplete	RichardPrestage	AmyShelton	details
24	4 - N/A	N	Average and difference Polarizations	RichardPrestage	AmyShelton	details
25	3 - Low	N	Goodness of fit test	RichardPrestage	AmyShelton	Need an estimate of how good the fit is. See details for more info.
26	4 - N/A	N	Initial fit from cross-correlation	RichardPrestage	AmyShelton	details
28	4 - N/A	N	fit to median-filtered data	RichardPrestage	AmyShelton	details
31	4 - N/A	N	Sequential scan check	RichardPrestage	AmyShelton	details
33	4 - N/A	N	Optimum Filtering	RonMaddalena	TBD	details
40	1 - High	Y	Rationalize baselines strategy	RichardPrestage	AmyShelton	ModificationRequest10C804	Ready for sponsor testing in ashelton's sandbox.
41	4 - N/A	N	Convolve Gaussian with disk/Gaussian for extended sources	RichardPrestage	AmyShelton	If we can find some way of identifying extended sources, GFM could fit these. Really only necessary for planets?
44	4 - N/A	N	Continue past pathological scans	RichardPrestage	AmyShelton	See details for info.
48	4 - N/A	N	Change how GFM calculates center frequency.	YuriKovalev	AmyShelton	Currently, GFM uses the center frequency value stored in the IF Manager FITS file. This value is correct most of the time, but not all of the time. We need to find a way to calculate center frequency correctly all of the time.
Add new requests below this line (please check existing requests first!)

49	4 - N/A	N	Remove use of DefaultDirectory? from .sparrow file	RichardPrestage	AmyShelton	GFM should use getConfigValue from Sparrow's api/ygor to find the default data directory rather than from the .sparrow file.
50	4 - N/A	N	Heuristrics for offsets should not be based on a percentage change.	RonMaddalena	AmyShelton	Heuristrics for offsets should not be based on a percentage change since the expected values, on the average, should have a zero mean. For example, had a 0.01 and 0.04 offsets that weren't accepted while each was a super-superb result. Instead, should be based on the (absolute change / measured or theoretical halfwidth). Heights and half-widths, on the otherhand, don't have zero for a mean and can be based on a percentage change.
51	4 - N/A	N	Fix Ka Band data processing bug	RonMaddalena	AmyShelton
52	4 - N/A	N	Add timestamps to log messages	JimBraatz	AmyShelton
53	4 - N/A	N	Need to be able to specify the default polarization for the pointing and focus plugins via the .sparrow file	BrianMason	AmyShelton

1. Complete tipping plug-in.

A plug-in has been created, but it doesn't always get the correct answer. It also needs to be able to process data taken in beam-switched mode (processing it as if it were total power). Jim Condon has agreed to work with Amy on this. Some example tipping scans are TQBAND040229 Scans 28-35, and TPTCSRMP040418 Scans 7,9, 107, 108, 189.

2. Complete DCR plug-in

A plug-in exists, but it is not complete. In particular, it needs to be able to calculate Ta (beam-switched or non-beamswitched) as an alternative to showing individual phases. The X axis should be a simple time in seconds, with the start time indicated (in local time) in the plot title. The Y-axis values are often difficult to read (especially for counts). There is a potential offset in the phase times - see TREG_040515 scan 8 (this offset might be real).

3. Install "correct" calibration.

Once the new calibration memo is released, this should be implemented. This includes total power, dual-beam with no electronic beam-switching, and beam-switched observations. Should also at this time allow the possibility for a user to specific a single Tcal value they want to use (perhaps via a different choice of calibration algorithm?).

9. Resolve Menu Options

Currently peak and focus plugins get their options from a menu, while the DCR plugin gets them from a panel to the right of the plot. We should decide which is considered "best", and implement this for all plugins. In addition, some of the menu options are somewhat confusing, especially in dual poln and/or dual-feed receivers. For example, for dual-poln receivers, Dana would prefer the options to be "X/L", "Y/R" or "both" (three choices), rather than "X/L", "Y/R" and checking both means "both". For the current "DCR" plugin, since "/" in other places means "or", I find the "Sig / Cal" etc labels confusing. We should discuss and resolve these issues

13. Dynamic Plugins

There have been various requests to be able to change plug-ins on the fly, have gfm keep track of state between invocations, and so on. We should agree upon requirements/behaviour, and then have the resulting decisions implemented.

15. Average and Difference F/B results

The azimuth and elevation offsets currently generated are the average of the forward and backward azimuth scan, and the forward and backward elevation scans, respectively. In addition, it would be extremely useful to report the difference of the F/B scans. This is an extremely useful check on the consistency of the results (for example, in windy weather the forward and backward scans can differ dramatically). We should discuss the appropriate format for output.

17. Handle duplicate scan numbers.

It is possible to have duplicate scan numbers in a single project directory - see e.g. AGBT03C_002_001. gfm appears to pick up the first copy of the scan. Can we decide on some mechanism for identifying duplicate scans and handling them correctly?

20. Prepare to use CCB

When the CCB is available, we will probably want to peak and focus with that, rather than the DCR. We will also want a real-time display of its data. Discuss with Brian Mason what, if anything, is actually required, and by when. But we should budget effort for this if we want it.

23. Force data-reduction if GO file incomplete.

This is LOW PRIORITY so should only be tackled when there is nothing better to do. We occasionally get perfectly good data, but the GO FITS file is missing (or potentially, incomplete). In principle, in some cases GFM could continue and attempt to figure out from the data what sort of scan this is, and try to process it appropriately. Specifically:

Option A:

Should probably get the OBJECT name out of the Antenna FITS file (since that must be present).
If this should have been scan 2,3,4 of a "peak" observation process it according to the appropriate recipe, by taking quantities that are normally extracted from the GO FITS file from the file for the previous scan, but modifying the PROCSEQN number accordingly.
If this is not scan 2,3,4 of a peak, it is much harder. Possibilities are:
- this is the first scan of a peak. Check whether either the AZ extent is > 2 FWHM for the observing frequency (obtained presumably from the IF FITS file?). If it is, assume this is scan 1 of a peak, and process it according to the "default" rules (TPwCAL for X-band and below, switching scheme as indicated by DCR contents for Ku and above).
- If not any peak scan, may be a focus. Look for the subreflector Y extent to be > 10mm (some people like small extents; for any non-focus scan the y extent should be ~0mm within the limits of servo noise). If so, process as above.
- it doesn't appear to be a peak or a focus scan; ignore it (generic DCR plug-in should handle it).

Option B:

ignore the scan on first pass. Provide a dialog box like the "play" dialog box (one each for peak and focus) which would allow the user to manually reprocess the scan (in the case of "peak" a group of 4 scans), with the dialog box requesting whatever additional information might be necessary (shouldn't be much, since the invocation of this dialog already indicates the type of scan, etc).

24. Average and Difference Polarizations

For dual-polarization reduction (i.e. if the user has explicitly requested that both polarizations be processed) the results of the two polarizations should be processed completely independently, and then the resulting offets, FWHMs, etc, should be the average of the two fits (I believe, but am not sure, that the averaging may currently be being performed by averaging the data arrays for the two polns.) In addition to the average values, the differences between the two polarizations should be reported. The differences should be small, since the atmosphere is unpolarized, so large errors would be an excellent diagnostic of problems in the system. We might discuss this one further before any work commences.

25. Estimate of goodness of fit.

On occasion, gfm comes up with fits that are clearly "wrong" by eye, but do not fail the heuristic tests. An example is TQBAND040229 scan 2. Some sort of goodness of fit (chi-squared?) test is required to decide whether the resulting fit (baseline + gaussian) is realistic.

From BrianMason:

I think we need a new heuristic involving the signal to noise for 
peak and focus. sometimes GFM will fit what is clearly absolutely useless 
bogus data and get a gaussian which by chance has a reasonable width, say 
(this is worse of a problem for focus, b/c with peak at least you get the 
other scan cross-check), then proceed to apply a silly correction. it 
would be something like, rject data when

        fitted peak height
  ------------------------------------   < N
  mean absolute deviation of residuals

is this a) on the wish list ; b) already debated and discareded (why?); c) 
other? I am willing to sponsor it, some cycle soon.

26. Initial fit from a cross-correlation

For dual-beam data, the signature of the +/-ve gaussian separated by the expected beam-width is extremely specific. We should cross-correlate a template of this form with the data array, to obtain the initial guess for the location of the source. This should help locate weak sources in the presence of noisy baselines, rfi, etc. For example, see TPTCSRMP040129 scans 5 and 6 taken at Ku-band. We had the incorrect beam offsets and thus the pointing was in error by 2-3 arcmin in the azimuth scans. Even though the S/N ratio was good GFM did not fit the data.

28. Fit to median-filtered data.

Median-filtering is already used to obtain robust initial guesses in the presence of RFI. In extreme cases, it might be preferrable to fit to the median-filtered data as well. This should not be done by default, as median-filtering degrades resolution. Would need to be a user preference.

29. TPOINT plug-in

At some point, I'd still like a TPOINT plugin. This would be like the "play" option on a series of peaks (and/or focuses), but the output would be an ascii text file with columns AZ, EL, TAZ and TEL. Need to discuss further before implementation.

31. Check for sequential scan numbers

In offline mode, it is possible to accidentally select the first az scan of one peak, then the second az scan of a different peak (e.g. for two successive peaks scarting with scan 1, scans 1,6,7,8). Since a "peak" is an atomic observation, scan numbers should always be adjacent. gfm is already doing some checking, since it generally catches scans which are out of order. In the check for scans 2,3, and 4 of a peak, please check that the scan numbers are +1, +2, +3 from the first scan.

33. Optimum Filtering

I also suggest we add to GFM the ability to convolve the data with a beam profile of user-specified width (default being the expected beam width at the observing frequency) before fitting a Gaussian. This uses the technique of 'optimum' filtering to pull signal out of what standard Gaussian fitting would call unusable data. It's a technique we used very successfully at the 140-ft on bad weather days. (from rmaddale email, 5/29/04)

44. Continue past pathological scans

In the peak plug-in, if GFM hits a pathological scan (e.g. a corrupt DCR FITS file), it appears to throw an exception on that scan, and final processing of the data aborts. This means that if the 4th scan fails, the results from the previous three scans, which are enough to generate an answer, do not get processed. It would be nice if GFM would continue to do this processing, ignoring the pathological scan. The same approach applies if any scan throws an exception, not just the last one.

This is a rare event, so this request is low priority, but it does happen on occasion. An example is

/home/archive/test-data/tape-0001/TPTCSRMP031211 Scans 346-349.

Topic GbtFitsMonitorContinuumRequests . { Edit | Attach | Ref-By | Printable | Diffs | r1.139 | > | r1.138 | > | r1.137 | More }

Revision r1.139 - 22 May 2006 - 18:05 GMT - AmyShelton	Content copyright © 1999-2007 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Data.GbtFitsMonitorContinuumRequests moved from Data.GbtFitsMonitorRequests on 16 Feb 2005 - 16:07 by AmyShelton - put it back


NRAO Home > Green Bank \| Wiki Topic: GB > Data > GbtFitsMonitorContinuumRequests

Changes \| Index \| Contents \| Search \| Statistics \| Jump to Topic