General
This procedure provides guidance for correcting problems associated with the Spectrometer. This page should be referred to directly, i.e., not any copy of it, because it is expected the instructions will be changing on a regular basis. For software related problems not described below, contact the Software Division. For hardware related problems, contact the Digital Group. See current callout lists for individuals responsible. Unless the observing situation/observing friend suggests otherwise, the callout person should be contacted. Refer to the current callout policy as needed.
Info
The character ^ is used below to emphasize spaces between dialog elements.
The Cleo TaskMaster application may also be used to complete the systemstop actions shown below.
Problems and Actions
| Problem | Action |
|---|
| A single data interrupt failure | Rerun the scan (even if the system has recovered, the message will not clear until another scan has been run) |
| A second data interrupt failure | reset |
| Multiple data interrupt failures | reboot |
| A DMA failure | Rerun the scan |
| Bad data, e.g. odd looking lags, steps or jumps in the ACF link to example plot planned | reset |
| Bad data continues after reset | self test |
| Spectrometer locks up, i.e., remains hung in a state despite repeated aborts or is not responsive | restart |
| System power and/or system monitor lights are found off | cycle power |
| Cannot log into earth either from the console or remotely | If possible call someone in computer division or software division, else as a last resort do a hard reboot |
| Spectrometer program (TaskMaster, Spectrometer or Transporter) will not die even using kill -9 | reboot |
| The command ps shows that a spectrometer program (TaskMaster, Spectrometer or Transporter) is not running | restart |
| Continued failures or other problems | Contact software division callout person(s) |
| Bad data keeps being generated in spite of repeated resets. | self test |
| Self test fails | Contact digital division callout person(s). |
| Restart or reboot fails | Contact software division callout person(s) |
| Continuous serial line errors (more than 1 per second) | reset; if that does not work, cycle power |
| Intermittent serial line errors (every few seconds) | Note in OpsLog and ignore |
New Spectrometer problems should be entered into this list by operators to mark the need for new actions. For diagnostic purposes, it's important that these actions be followed and fully recorded in OpsLog.
Action Definitions
These terms are used to describes what procedures should be followed given specific spectrometer problems. They
should also be used when describing problem/failure entries in OpsLog and when consulting callout personnel.
- reset
- From the CLEO Spectrometer Manager menu, select Reset parameters. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.
- self test
- From the CLEO Spectrometer window select Change Configuration..., then select Testing, then Next and then Finish. In the Testing tab select Test Using Interrupts. Be patient. At the conclusion of the test a window will pop up filled with tables of numbers. If the test failed then the table will contain non-zero values and a failure message will be generated. The observer will need to reconfigure the Spectrometer.
- systemstop
- Log into earth as monctrl and enter the commands listed below. The last command prints all programs being run by monctrl. One should not see TaskMaster, Spectrometer or Transporter running.
$ source ^ /home/gbt/gbt.bash
$ TaskMaster ^ earth ^ systemstop
$ ps ^ -u ^ monctrl
- systemstart
- Log into earth as monctrl and enter the commands listed below. The ps command prints all programs being run by monctrl. After the first ps command, one should not see TaskMaster, Spectrometer, spectrometer_init, or Transporter running. If spectrometer_init is running, wait a couple of minutes and run ps again. After the last ps command, one should see TaskMaster, Spectrometer and Transporter running. The observer will need to reconfigure the Spectrometer.
$ ps ^ -u ^ monctrl
$ source ^ /home/gbt/gbt.bash
$ TaskMaster ^ earth ^ systemstart ^ /home/gbt/etc/config/earthProc.conf
$ ps ^ -u ^ monctrl
- restart
- Do a systemstop followed by a systemstart.
- reboot
- Do a systemstop and then log into earth as root (either remotely or from earth's console). Note that the command listed below will end the login when it reboots the computer. After earth has finished rebooting, do a systemstart. The observer will need to reconfigure the Spectrometer.
$ sync; ^ init ^ 6
- hard reboot
- At the workstation earth, turn the power off using the small black switch on the front panel inside the door. Inside the spectrometer cabinet press the Reset button on the MVME167-32A. Following the reboots insure the program spectrometer_init is completed (see ps command noted above) and then do a systemstart. The observer will need to reconfigure the Spectrometer.
- cycle power
- In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Monitor to ON, and turn System Power to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.
-- DavidRose - 6 Feb 2008
Revision r1.20 - 06 Feb 2008 - 17:19 GMT - DavidRose
Parents: WebHome > TelescopeOperations
|
Content copyright © 1999-2007 by the contributing authors.
All material on this collaboration platform is the property of the contributing authors.
|
| |