-- DavidRose - 22 Mar 2004

-- DavidRose - 6 Feb 2008

hard reboot

At the workstation earth, turn the power off using the small black switch on the front panel inside the door. Inside the spectrometer cabinet press the Reset button on the MVME167-32A. Following the reboots insure the program spectrometer_init is completed (see ps command noted above) and then do a systemstart. The observer will need to reconfigure the Spectrometer.

hard reboot

At the workstation earth, turn the power off using the small black switch on the front panel inside the door. Inside the spectrometer cabinet press the Reset button on the MVME167-32A. Following the reboots insure the program spectrometer_init is completed (see ps command noted above) and then do a systemstart. The observer will need to reconfigure the Spectrometer.

cycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Monitor to ON, and turn System Power to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

cycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Monitor to ON, and turn System Power to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

-- DavidRose - 19 Mar 2004

-- DavidRose - 22 Mar 2004

hard reboot

At the workstation earth, turn the power off using the small black switch on the front panel inside the door. Inside the spectrometer cabinet press the Reset button on the MVME167. Following the reboots insure the program spectrometer_init is completed (see ps command noted above) and then do a systemstart. The observer will need to reconfigure the Spectrometer.

hard reboot

At the workstation earth, turn the power off using the small black switch on the front panel inside the door. Inside the spectrometer cabinet press the Reset button on the MVME167-32A. Following the reboots insure the program spectrometer_init is completed (see ps command noted above) and then do a systemstart. The observer will need to reconfigure the Spectrometer.

cycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Monitor to ON, and turn System Power to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

cycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Monitor to ON, and turn System Power to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

-- DavidRose - 08 Mar 2004

-- DavidRose - 19 Mar 2004

Info 

The character ^ is used below to emphasize spaces between dialog elements.

Info 

The character ^ is used below to emphasize spaces between dialog elements.

The Cleo TaskMaster application may also be used to complete the systemstop actions shown below.

Problems and Actions 

New Spectrometer problems should be entered into this list by operators to mark the need for new actions. For diagnostic purposes, it's important that these actions be followed and fully recorded in OpsLog.

Problems and Actions 

New Spectrometer problems should be entered into this list by operators to mark the need for new actions.

For purposes of diagnostics, it is important that these actions be followed and fully recorded in the OpsLog.

-- MarkClark - 05 Mar 2004

-- DavidRose - 08 Mar 2004

-- MarkClark - 04 Mar 2004

-- MarkClark - 05 Mar 2004

For purposes of diagnostics, it is important that these actions be followed and recorded in the OpsLog.

For purposes of diagnostics, it is important that these actions be followed and fully recorded in the OpsLog.

-- MarkClark - 02 Mar 2004

-- MarkClark - 04 Mar 2004

self test

From the CLEO Spectrometer window select Change Configuration..., then select Testing, then Next and then Finish. In the Testing tab select Test Using Interrupts. Be patient. At the conclusion of the test a window will pop up filled with tables of numbers. If the test failed then the table will contain non-zero values and a failure message will be generated.

self test

From the CLEO Spectrometer window select Change Configuration..., then select Testing, then Next and then Finish. In the Testing tab select Test Using Interrupts. Be patient. At the conclusion of the test a window will pop up filled with tables of numbers. If the test failed then the table will contain non-zero values and a failure message will be generated. The observer will need to reconfigure the Spectrometer.

This procedure provides guidance for correcting problems associated with the Spectrometer. For software related problems not described below, contact the Software Division. For hardware related problems, contact the Digital Group. See current callout lists for individuals responsible. Unless the observing situation/observing friend suggests otherwise, the callout person should be contacted. Refer to the current callout policy as needed.

This procedure provides guidance for correcting problems associated with the Spectrometer. This page should be referred to directly, i.e., not any copy of it, because it is expected the instructions will be changing on a regular basis. For software related problems not described below, contact the Software Division. For hardware related problems, contact the Digital Group. See current callout lists for individuals responsible. Unless the observing situation/observing friend suggests otherwise, the callout person should be contacted. Refer to the current callout policy as needed.

self test

From the CLEO Spectrometer window select Change Configuration..., then select Testing, then Next and then Finish. In the Testing tab select Test Using Interrupts. Be patient. At the conclusion of the test a window will pop up filled with tables of numbers. If the test failed then the table will contain non-zero values and a failure message will be generated.

hard reboot

At the workstation earth, turn the power off using the small black switch on the front panel inside the door. Inside the spectrometer cabinet press the Reset button on the MVME167. Following the reboots insure the program spectrometer_init is completed (see PS command noted above) and then do a systemstart. The observer will need to reconfigure the Spectrometer.

hard reboot

At the workstation earth, turn the power off using the small black switch on the front panel inside the door. Inside the spectrometer cabinet press the Reset button on the MVME167. Following the reboots insure the program spectrometer_init is completed (see ps command noted above) and then do a systemstart. The observer will need to reconfigure the Spectrometer.

cycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Power to ON, and turn System Monitor to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

cycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Monitor to ON, and turn System Power to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

-- DavidRose - 01 Mar 2004

For purposes of diagnostics, it is important that these actions be followed and recorded in the OpsLog.

-- MarkClark - 02 Mar 2004

The following procedures are used to resurrect the Spectrometer should it fail to function correctly, become hung in a particular state, or errors are noted (serial line, DMA, or load xilinx files errors for example). Use the procedures in the order prescribed unless directed otherwise.

This procedure provides guidance for correcting problems associated with the Spectrometer. For software related problems not described below, contact the Software Division. For hardware related problems, contact the Digital Group. See current callout lists for individuals responsible. Unless the observing situation/observing friend suggests otherwise, the callout person should be contacted. Refer to the current callout policy as needed.

Info 

The character ^ is used below to emphasize spaces between dialog elements.

should also be used in the operator's log and call-outs are required.

cycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Power to ON, and turn System Monitor to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

should also be used when describing problem/failure entries in OpsLog and when consulting callout personnel.

hard reboot

At the workstation earth, turn the power off using the small black switch on the front panel inside the door. Inside the spectrometer cabinet press the Reset button on the MVME167. Following the reboots insure the program spectrometer_init is completed and then do a systemstart. The observer will need to reconfigure the Spectrometer.

reset

From the CLEO Spectrometer Manager menu, select Reset parameters. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

reboot

Do a systemstop and then log into earth as root (either remotely or from earth's console). Note that the command listed below will end the login when it reboots the computer. After earth has finished rebooting, do a systemstart. The observer will need to reconfigure the Spectrometer.

systemstop

Log into earth as monctrl and enter the commands listed below. The last command prints all programs being run by monctrl. One should not see TaskMaster, Spectrometer or Transporter running.

$ sync; init 6

$ source ^ /home/gbt/gbt.bash $ TaskMaster? ^ earth ^ systemstop $ ps ^ -u ^ monctrl

reset

From the CLEO's Spectrometer Managers menu select Reset parameters. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

restart

Do a systemstop followed by a systemstart.

$ ps -u monctrl $ source /home/gbt/gbt.bash $ TaskMaster? earth systemstart /home/gbt/etc/config/earthProc.conf $ ps -u monctrl

$ ps ^ -u ^ monctrl $ source ^ /home/gbt/gbt.bash $ TaskMaster? ^ earth ^ systemstart ^ /home/gbt/etc/config/earthProc.conf $ ps ^ -u ^ monctrl

systemstop

Log into earth as monctrl and enter the commands listed below. The last command prints all programs being run by monctrl. One should not see TaskMaster, Spectrometer or Transporter running.

restart

Do a systemstop followed by a systemstart.

reboot

Do a systemstop and then log into earth as root (either remotely or from earth's console). Note that the command listed below will end the login when it reboots the computer. After earth has finished rebooting, do a systemstart. The observer will need to reconfigure the Spectrometer.

$ source /home/gbt/gbt.bash $ TaskMaster? earth systemstop $ ps -u monctrl

$ sync; ^ init ^ 6

hard reboot

At the workstation earth, turn the power off using the small black switch on the front panel inside the door. Inside the spectrometer cabinet press the Reset button on the MVME167. Following the reboots insure the program spectrometer_init is completed (see PS command noted above) and then do a systemstart. The observer will need to reconfigure the Spectrometer.

Problems and Actions 

cycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Power to ON, and turn System Monitor to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

Problems and Actions 

Info 

For software related problems contact the Software Division. For hardware related problems, contact the Digital Group. See current callout lists for individuals responsible for the Spectrometer. The callout person should be contacted unless the observing situation/observing friend dictates otherwise. Refer to the current callout policy as needed.

Recovery Procedure 1: Use to correct for odd looking lags, steps, or jumps in the ACF.

Recovery Procedure 2: Use when the system locks up (hung in a state, not responsive), if procedure 1 fails to help, or as directed by the callout person.

Recovery Procedure 3: Use if the Spectrometer's System Power and/or System Monitor lights are found off, if procedure 2 fails to help, or as directed by the callout person.

The character " ^ " is used below to emphasize spaces between dialog elements.

-- DavidRose - 01 Mar 2004

Glossary 

Action Definitions 

These terms are used to describes what procedures should be followed given specific spectrometer problems. They should also be used in the operator's log and call-outs are required.

cycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Power to ON, and turn System Monitor to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

reboot

Do a systemstop and then log into earth as root (either remotely or from earth's console). Note that the command listed below will end the login when it reboots the computer. Following the command insure the program spectrometer_init is completed and then do a systemstart. The observer will need to reconfigure the Spectrometer.

reboot

Do a systemstop and then log into earth as root (either remotely or from earth's console). Note that the command listed below will end the login when it reboots the computer. After earth has finished rebooting, do a systemstart. The observer will need to reconfigure the Spectrometer.

recycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Power to ON, and turn System Monitor to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

systemstart

Log into earth as monctrl and enter the commands listed below. The ps command prints all programs being run by monctrl. After the first ps command, one should not see TaskMaster, Spectrometer or Transporter running. After the last ps command, one should see all three programs running. The observer will need to reconfigure the Spectrometer.

systemstart

Log into earth as monctrl and enter the commands listed below. The ps command prints all programs being run by monctrl. After the first ps command, one should not see TaskMaster, Spectrometer, spectrometer_init, or Transporter running. If spectrometer_init is running, wait a couple of minutes and run ps again. After the last ps command, one should see TaskMaster, Spectrometer and Transporter running. The observer will need to reconfigure the Spectrometer.

Problems and Actions 

New Spectrometer problems should be entered into this list by operators to mark the need for new actions.

-- DavidRose - 5 Feb 2004

-- MarkClark - 01 Mar 2004

Glossary 

hard reboot

At the workstation earth, turn the power off using the small black switch on the front panel inside the door. Inside the spectrometer cabinet press the Reset button on the MVME167. Following the reboots insure the program spectrometer_init is completed and then do a systemstart. The observer will need to reconfigure the Spectrometer.

reboot

Do a systemstop and then log into earth as root (either remotely or from earth's console). Note that the command listed below will end the login when it reboots the computer. Following the command insure the program spectrometer_init is completed and then do a systemstart. The observer will need to reconfigure the Spectrometer.

            $ sync; init 6

recycle power

In the spectrometer cabinet, turn System Power to OFF, turn System Monitor to OFF, wait 5 seconds, turn System Power to ON, and turn System Monitor to ON. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

reset

From the CLEO's Spectrometer Managers menu select Reset parameters. This will take a couple of minutes to complete; one can verify it has started by the "initializing hardware" message. This action does not require reconfiguration of the software.

restart

Do a systemstop followed by a systemstart.

systemstart

Log into earth as monctrl and enter the commands listed below. The ps command prints all programs being run by monctrl. After the first ps command, one should not see TaskMaster, Spectrometer or Transporter running. After the last ps command, one should see all three programs running. The observer will need to reconfigure the Spectrometer.

            $ ps -u monctrl
            $ source /home/gbt/gbt.bash
            $ TaskMaster earth systemstart /home/gbt/etc/config/earthProc.conf
            $ ps -u monctrl

systemstop

Log into earth as monctrl and enter the commands listed below. The last command prints all programs being run by monctrl. One should not see TaskMaster, Spectrometer or Transporter running.

            $ source /home/gbt/gbt.bash
            $ TaskMaster earth systemstop
            $ ps -u monctrl

Alternate Recovery Procedure: This is the method suggested by J. Brandt and seems to work real well.

Alternate Recovery Procedure

Alternate Recovery Procedure: This is the method suggested by J. Brandt and seems to work real well.

Alternate Recovery Procedure

-- DavidRose - 21 Jan 2004

-- DavidRose - 5 Feb 2004

(serial line, DMA, or load xilinx files errors for example). Use the procedures in the order prescribed unless directed otherwise by the Digital Group.  

(serial line, DMA, or load xilinx files errors for example). Use the procedures in the order prescribed unless directed otherwise.

For Software related problems contact the Software Division (Mark Clark). For hardware related problems, contact the Digital Group (Holly Chen, Rich Lacasse, or John Ford). 

Aborting via the Scan Coordinator should be attempted first as this will occasionally correct a hung state problem.

For software related problems contact the Software Division. For hardware related problems, contact the Digital Group. See current callout lists for individuals responsible for the Spectrometer. The callout person should be contacted unless the observing situation/observing friend dictates otherwise. Refer to the current callout policy as needed.

correct for odd looking lags, steps, or jumps in the ACF.

correct for odd looking lags, steps, or jumps in the ACF.

the system locks up (hung in a state, not responsive), if procedure 1 fails to help, or as directed by the Digital Group.

the system locks up (hung in a state, not responsive), if procedure 1 fails to help, or as directed by the callout person.

procedure 2 fails to help, or as directed by the Digital Group.  

procedure 2 fails to help, or as directed by the callout person.

Recovery Procedure 1

Recovery Procedure 1

Recovery Procedure 2

Recovery Procedure 3

Recovery Procedure 2

Recovery Procedure 3

-- DavidRose - 03 Oct 2003

-- DavidRose - 21 Jan 2004

%META:TOPICPARENT{name="TemporaryGBTOpsArea"}%

%META:TOPICPARENT{name="TelescopeOperations"}%

%META:TOPICINFO{author="DavidRose" date="1065191888" format="1.0" version="1.1"}% %META:TOPICPARENT{name="TemporaryGBTOpsArea"}%

General

The following procedures are used to resurrect the Spectrometer should it fail to function correctly, become hung in a particular state, or errors are noted (serial line, DMA, or load xilinx files errors for example). Use the procedures in the order prescribed unless directed otherwise by the Digital Group.  

Info 

For Software related problems contact the Software Division (Mark Clark). For hardware related problems, contact the Digital Group (Holly Chen, Rich Lacasse, or John Ford). 

Aborting via the Scan Coordinator should be attempted first as this will occasionally correct a hung state problem.

Recovery Procedure 1: Use to correct for odd looking lags, steps, or jumps in the ACF.

Recovery Procedure 2: Use when the system locks up (hung in a state, not responsive), if procedure 1 fails to help, or as directed by the Digital Group.

Recovery Procedure 3: Use if the Spectrometer's System Power and/or System Monitor lights are found off, if procedure 2 fails to help, or as directed by the Digital Group.  

The character " ^ " is used below to emphasize spaces between dialog elements.

Recovery Procedure 1

Recovery Procedure 2

Recovery Procedure 3

-- DavidRose - 03 Oct 2003