NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Data > DataQualityNotesNicole

Changes | Index | Contents | Search | Statistics | Jump to Topic

Data Quality Notes

This is my scratch pad for ideas for Ron. Very disorganized brainstorming! All below represent suggestions only and are not prescriptions for exactly what should be done...(nmr)

We need a data quality diagostic system. We should start with Karen's needs at ModificationRequest9C803 which runs simple hardware and software checks to make sure the telescope system is in a "good state" at the beginning of an observation. However, it would be nice to be able to look at one screen and tell a) if the systems are running OK and b) if the output data is complete and analyzable.

20 Sep 2004:

Need to figure out a good way to tag whether error messages happen during observing or maintenance.

Frequency of Execution.

Trending. It's detecting if a piece of the system that impacts the observer is going bad.

• temp sensors
• pointing accuracy
• weather sensors

23 Jul 2004:

Need to start thinking about this kind of stuff:

• Also need some high level metrics that can replace subjective quality measures from ops logs
• MTBF, MTTR, more stuff?
• Also need to measure frequency of execution (FOE) of different configurations and scan types, incl. how do we tell if this is a pulsar observation (can sniff from configuration chosen?) -- this will also guide prioritization decisions for new features
• Then we can measure MTBF/MTTR with respect to FOE and that will give us a clear picture of how robust or unstable a part of the system is. Right now, we only "sense" the magnitude of problems based on problem reports. If for example no one uses the spectral processor for three months, and then all of a sudden it's getting used a lot, and there are issues, we may incorrectly assume that "all of a sudden the spectral processor has gone bad." Which of course it hasn't.
• Hmmm, maybe there are some applications for the kind of metrics you use in storm forecasting. Probability of Detection. False Alarm Ratio. More Details

Notes from June:

There are 3 areas:

1. Device Health/Quality & making sure system is in a "good state"
2. Raw Data Quality (individual FITS files produced) - "unit test" level with respect to data
3. Interpreted Data Quality - "integration test" level with respect to data

• Check to make sure ScanLog?.fits is complete and correct. If it has entries in it that do not correspond to real files, the following could occur:
  • GFM gives errors like when processing scan 29 of AGBT04A_019_02, 'Error processing scan 29 : database "IF" not defined in current context'
• When the system experiences "network glitches" during observing we need to be able to identify where the unexpected problems are.
• Device Health. We need a way to see in real time which devices are healthy and which ones are not. We need a way to tie in the dev_health.conf file.
• Look at this too http://www.eso.org/observing/dfo/quality/
• No GO FITS files (or other declaration of observers intent)
• All raw FITS files have to be "good"
• Is system using proper DDLs?
• Is system.conf OK?
• Is phase table consistent?
• Be able to guarantee that at least all cabling is in place, all backends are accessible, the correct M&C version is being run, and the antenna is no longer in simulator mode.
• What's the purpose of the spectral processor quick tests (ask Amy)? The spectrometer self tests (ask Karen/Rich Lacasse)
• How to store/archive quality info with data?
• Autoflagging - what tests should be run on all data, what flags added, and for what?? RFI and what else?

-- NicoleRadziwill - 25 May 2004

Device Health:

• We need to determine the health of component devices at a point in time.
• This includes everything that has a Manager + the single boards + the host computers + the network. Anything else?
• Keep in mind single board computers can get in funky states too. How do we tell if the remaining single boards are OK? The solution right now is just to reboot them. Should we even address this, since we want to get everything ported to Linux within months?
• How can we tell if the network is healthy? Is this even possible? Ask Joe Brandt.

Device Health (trends):

• We need to determine the health of a device over time, and detect changes that could indicate a component is going bad.
• Example: Paul's temperature sensor health monitor.
• May require a one-off solution for each device that should be monitored this way. That is, comparing the values from sensors that are symmetrically placed on the telescope is valid for temperature sensors, but will not be valid for accelerometers.

Errors in Raw Data (FITS Files):

• First we need to figure out the solution space for FITS files that can be produced.
• Generate a table, perhaps with the following columns: FITS file name, error in FITS file, symptom, reported by. For example, an error in the Antenna FITS file could be something like "NaNs present (for this keyword)" and the symptom might be "GFM fails to fit data and produces a 'division by zero' error". "Reported by" is the person who told you that this is a symptom of the error that you are describing.

Errors in Interpreted Data:

• I wouldn't even address this until the device & raw data issues are fully outlined. This is bound to be the

People to talk to:

• Joe Brandt can tell you everything you need to know about the antenna & antenna FITS.
• Melinda should be the one you talk to about health of the IF path and component devices along the IF path (ie optical drivers). She can tell you about all the problems that will negatively impact the ability to successfully configure the telescope.
• Ray Creager can help you out with quality issues in the telescope control system configuration files and DDLs, and problems in the underlying system that can be reflected as symptoms by grail.
• Amy can tell you about spectrometer self tests. She may be able to tell you about spectral processor quick tests, but if not, Karen might, or Mark will be able to when he gets back. Karen uses these self tests for something, but I don't know what.
• Ask Bob Garwood about things that the "filler" checks for. The filler is the piece of code that takes raw data from the individual FITS files and knits them together into a measurement set, which is the data format understood by aips++.
• The operators. Ask Bob Anderson who is on duty when, and come in and ask them questions. Operators are notorious for making incorrect cause and effect relationships, so beware. Double check

-- NicoleRadziwill - 29 Jun 2004