1. The script that you have run has also switched the sparrow Sparrow version for you. Sometimes, the version of GBTIDL must be changed as well. Contact BobGarwood or PaulMarganian to find out if this is necessary. The current system for switching GBTIDL versions is identical to that used by M&C and sparrow. It simply involves switching a symbolic link. As monctrl:

1. The script that you have run has also switched the sparrow Sparrow version for you. Sometimes, the version of GBTIDL must be changed as well. However, this should only be done during a release. We can't switch GBTIDL versions during integration/regression tests as this would affect current users.. Contact BobGarwood or PaulMarganian to find out if this is necessary. The current system for switching GBTIDL versions is identical to that used by M&C and sparrow. It simply involves switching a symbolic link. As monctrl:

1. The script that you have run has also switched the sparrow Sparrow version for you. Sometimes, the version of GBTIDL must be changed as well. Contact BobGarwood or PaulMarganian to find out if this is necessary. The current system for switching GBTIDL versions is identical to that used by M&C and sparrow. It simply involves switching a symbolic link. As monctrl:
   • • cd /home/gbtidl
     • rm -f release
     • ln -s GBTIDl_RELEASE release (where GBTIDL_RELEASE is the latest version number, ex: 2.3.1)

Version Specific Notes:

Switching from 5.7 to 6.1:

1. The machines earth, wind and fire must be rebooted to run Redhat Enterprise 4 (RHE4)
   • Log into each machine as root and do:
     • mount /dev/hda3 /mnt
     • cp /mnt/boot/grub/grub.conf.6.1 /mnt/boot/grub/grub.conf
     • sync;sync;reboot
     • logout
2. Prior to running switchversions, copy _installation.conf.6.1 to _installation.conf and manually shutdown TaskMaster on vortex, tank, and algol.
   • for i in algol tank vortex; do TaskMaster $i systemstop; done
3. switchversions must be run from a system running the RHE4 operating system.
4. When asked to reset the spectrometer SBC, also swap the orange DCR cable from algol to tank (the one with the purple tag should be plugged into tank).
5. If running 6.1, cleo screens may only be run from a system running RHE4, such as tank or ariel (to be resolved soon).
6. The tipper is down, skip the reset tipper step.
7. The inclinometer is down, ignore the inclinometer startup failure.

Switching from 6.1 to 5.7:

1. The machines earth, wind and fire must be rebooted to run Redhat 9 (RH9)
   • Log into each machine as root and do (Note the difference from above is correct!):
     • cp /boot/grub/grub.conf.5.7 /boot/grub/grub.conf
     • sync;sync;reboot
     • logout
2. Prior to running switchversions, copy _installation.conf.5.7 to _installation.conf and manually shutdown TaskMaster on tank
   • TaskMaster tank systemstop
3. When asked to reset the spectrometer SBC, also swap the orange DCR cable from tank to algol (the one with the purple tag should be plugged into algol).
4. The tipper is down, skip the reset tipper step.
5. The inclinometer is down, ignore the inclinometer startup failure.

Version Specific Notes:

Switching from 5.7 to 6.1:

1. The machines earth, wind and fire must be rebooted to run Redhat Enterprise 4 (RHE4)
   • Log into each machine as root and do:
     • mount /dev/hda3 /mnt
     • cp /mnt/boot/grub/grub.conf.6.1 /mnt/boot/grub/grub.conf
     • sync;sync;reboot
     • logout
2. Prior to running switchversions, copy _installation.conf.6.1 to _installation.conf and manually shutdown TaskMaster on vortex, tank, and algol.
   • for i in algol tank vortex; do TaskMaster $i systemstop; done
3. switchversions must be run from a system running the RHE4 operating system.
4. When asked to reset the spectrometer SBC, also swap the orange DCR cable from algol to tank (the one with the purple tag should be plugged into tank).
5. If running 6.1, cleo screens may only be run from a system running RHE4, such as tank or ariel (to be resolved soon).
6. The tipper is down, skip the reset tipper step.
7. The inclinometer is down, ignore the inclinometer startup failure.

Switching from 6.1 to 5.7:

1. The machines earth, wind and fire must be rebooted to run Redhat 9 (RH9)
   • Log into each machine as root and do (Note the difference from above is correct!):
     • cp /boot/grub/grub.conf.5.7 /boot/grub/grub.conf
     • sync;sync;reboot
     • logout
2. Prior to running switchversions, copy _installation.conf.5.7 to _installation.conf and manually shutdown TaskMaster on tank
   • TaskMaster tank systemstop
3. When asked to reset the spectrometer SBC, also swap the orange DCR cable from tank to algol (the one with the purple tag should be plugged into algol).
4. The tipper is down, skip the reset tipper step.
5. The inclinometer is down, ignore the inclinometer startup failure.

1. When asked to run fixGoHang:
   • Be sure monctrl has antenna and other device security privileges
   • type: y
2. When asked to restart the IFManager type: y
3. When asked to restart the LO1 type: y

1. When asked to run fixGoHang:
   • Be sure monctrl has antenna and other device security privileges
   • type: y
2. When asked to restart the IFManager type: y
3. When asked to restart the LO1 type: y

1. If this is a release, please email the release notes to gbtlocal.

1. If this is a release, please email the release notes to gbtlocal AND update the main Software wiki page with links to the release notes (they go in the "Testing & Releases" table).

• • For users who need to continue working with sdfits during this test period, follow these steps to run the production version (note this only applies to sdfits, and not other applications):
    • $ source /home/sparrow/PRODUCTION_VERSION_NUMBER/sparrow.bash
    • $ sdfits

1. As soon as the time for the tests has been scheduled, an email should be sent to gbtlocal, warning users and other developers that this change is occuring: "GBT Testing is scheduled for date-time1 to date-time2. For this period, the following programs will be available, but running test versions, and users may wish to refrain from using them. The affected applications include:"
   • The GBT M&C System
   • astrid
   • gfm
   • turtle
   • gbtstatus
   • sdfits
   • logview
   • gbt2class
   • config tool

1. As soon as the time for the tests has been scheduled, an email should be sent to gbtlocal, warning users and other developers that this change is occuring: GBT Software Testing is scheduled for date-time1 to date-time2.  For this period, the following programs will be available, but running test versions, and users may wish to refrain from using them.  The affected applications include:
   • the GBT M&C System
   • astrid
   • gfm
   • turtle
   • gbtstatus
   • sdfits
   • logview
   • gbt2class
   • config tool

1. Log in as monctrl to a Linux machine (see exception notes above)

1. Login as monctrl to a Linux machine (see exception notes above).
2. Request that the operator add monctrl to the security gateway file with full permissions.

1. cd to /home/gbt
2. run ./switchVersions 6.1 (or the version number you want to switch to)

1. cd /home/gbt
2. Run ./switchVersions 6.1 (or the version number to which you want to switch)

• • verify that the new version listed is 6.1 (or the desired version) then hit the enter key.

1. When you are asked "Should I shut down the currently running system?"

• • Verify that the new version listed is 6.1 (or the desired version) then hit enter.

1. When you are asked "=Should I shut down the currently running system?="

• • Verify that there are no rogue processes. There should be no python, MMConverter or TaskMaster processes in the lists
    • If there any of these processes exist, note the PID and the machine that it is running on.
      • Kill the rogue processes using the following commands:
        • login to the machine where the process is running by typing:
          • ssh theMachineName
          • su monctrl
          • source /home/gbt/gbt.bash
          • kill -9 PID (where the PID is the process Id noted above)
      • type: n
      • Verify that there are no rogue processes
        • type: y

1. When asked if you want to change from current version to the version you selected, hit return for yes

• • Verify that there are no rogue processes. There should be no Python, MMConverter or TaskMaster processes in the lists. If there are none, type: y
  • If any of these processes are listed, type: n to the rogue processes query, and manually kill each one:
    • Note the process ID (PID) and the machine on which each rogue process is running.
    • Open another shell window, and kill each rogue processes using the following commands:
      • Login to the machine where the process is running by typing:
        • ssh <theMachineName> (where <theMachineName> is the machine noted above)
        • su monctrl
        • source /home/gbt/gbt.bash
      • kill -9 <PID> (where <PID> is the process ID noted above)
    • When there are no more rogue processes, type: y at the rogue processes query

1. When asked if you want to change from current version to the version you specified, hit enter for yes

1. When asked to update kernel drivers, hit enter for no

• • If there are failures restart these tasks
    • type: y

• • If there are failures, manually restart these tasks (except IFManager; that's handled in later part of script):
    • Login to the machine hosting a failed-to-start task by typing:
      • ssh <theMachineName> (where <theMachineName> is the machine on which the failed process is hosted - check /home/gbt/etc/config/system.conf)
      • su monctrl
    • Restart the failed task by typing:
      • source /home/gbt/gbt.bash
      • tm status to list hosted processes; take note of the process number for the failed process
      • tm start , where is the process number noted in last step
  • Once all failed-to-restart processes are manually restarted, type: y

• • Using the Reset box, reset the managers listed in by the script

• • Using the Reset box, reset the managers listed by the script

• • • from a Linux window type: telnet gbtipper
    • login in as monctrl (use the old monctrl password)

• • • From a Linux window type: telnet gbtipper
    • Login in as monctrl (use the old monctrl password)

• • Manually restart the Spectrometer singleboard in the equipment room by pressing the reset button on the Spectrometer single board.

1. When the sampler files are displayed (Its a LONG list), verify that all the samplers have files listed under them that are approximately equal to the current time.

• • Manually restart the Spectrometer singleboard in the equipment room by pressing the red reset button on the Spectrometer single board.

1. When the sampler files are displayed (it's a LONG list), verify that all the samplers have files listed under them with timestamps that are approximately equal to the current time.

• • Be sure monctrl has antenna and other device security priviledges

• • Be sure monctrl has antenna and other device security privileges

1. If this is a release, please email the release notes to gbtlocal.

• • config tool

1. As soon as the time for the tests has been scheduled, an email should be sent to gbtlocal, warning users and other developers that this change is occuring: "GBT Testing is scheduled for date-time1 to date-time2. For this period, the following programs will be available, but running test versions, and users may wish to refrain from using them. The affected applications include:"
   • The GBT M&C System
   • astrid
   • gfm
   • turtle
   • gbtstatus
   • sdfits
   • logview
   • gbt2class

Directions for Operators to Switch versions

Version Specific Notes:

Switching from 5.7 to 6.1:

1. The machines earth, wind and fire must be rebooted to run Redhat Enterprise 4 (RHE4)
   • Log into each machine as root and do:
     • mount /dev/hda3 /mnt
     • cp /mnt/boot/grub/grub.conf.6.1 /mnt/boot/grub/grub.conf
     • sync;sync;reboot
     • logout
2. Prior to running switchversions, copy _installation.conf.6.1 to _installation.conf and manually shutdown TaskMaster on vortex, tank, and algol.
   • for i in algol tank vortex; do TaskMaster $i systemstop; done
3. switchversions must be run from a system running the RHE4 operating system.
4. When asked to reset the spectrometer SBC, also swap the orange DCR cable from algol to tank (the one with the purple tag should be plugged into tank).
5. If running 6.1, cleo screens may only be run from a system running RHE4, such as tank or ariel (to be resolved soon).
6. The tipper is down, skip the reset tipper step.
7. The inclinometer is down, ignore the inclinometer startup failure.

Switching from 6.1 to 5.7:

1. The machines earth, wind and fire must be rebooted to run Redhat 9 (RH9)
   • Log into each machine as root and do (Note the difference from above is correct!):
     • cp /boot/grub/grub.conf.5.7 /boot/grub/grub.conf
     • sync;sync;reboot
     • logout
2. Prior to running switchversions, copy _installation.conf.5.7 to _installation.conf and manually shutdown TaskMaster on tank
   • TaskMaster tank systemstop
3. When asked to reset the spectrometer SBC, also swap the orange DCR cable from tank to algol (the one with the purple tag should be plugged into algol).
4. The tipper is down, skip the reset tipper step.
5. The inclinometer is down, ignore the inclinometer startup failure.

1. Log in as monctrl to any Linux machine

General Method to Switch versions:

1. Log in as monctrl to a Linux machine (see exception notes above)

1. run ./switchVersions 5.4 (or the version number you want to switch to)

1. run ./switchVersions 6.1 (or the version number you want to switch to)

• • verify that the new version listed is 5.4 (or the desired version) then hit the enter key.

• • verify that the new version listed is 6.1 (or the desired version) then hit the enter key.

1. for 5.4: Confirm the orange cable is connected to the front of the Dcr chassis. Follow the switchover procedure if it is not.
2. When all open processes are displayed for each machine (vortex, wind, fire etc):

1. When all open processes are displayed for each machine (gbtdata, wind, fire etc):

• • Manuully restart gbweather because the reset box does not work

• • Exceptions for 5.4 are:
    • RcvrPF_2 (all samplers)
    • MotorRack-RefridgeratorPowerSupply5-refridge
    • Rcvr40_52-Rcvr40_52-gregorian

• • Exceptions for 6.1 are:
    • PF2 samplers when PF1 is installed, and vice versa

• • • Inclinometer

1. When asked to verify that glish.d is running type: n
2. When asked to restart the IFManager type: n

1. When asked to restart the IFManager type: y

Directions for Operators to Switch from 5.4 to 5.3

• Step 4 ./switchVersions 5.3
• Step 7. Reverse the procedure specificed in this step.
• Step 10. You will also be asked this question for sparrow, type y
• Step 12. Say yes to the files:

       system.conf.5.3  
       Say 'NO' to all other 5.3 files.

Automated Solution

1. Log in as monctrl to any Linux machine
2. source /home/gbt/gbt.bash
3. cd to /home/gbt
4. ./switchVersions version-to-switch-to
5. When you are asked "Should I shut down the currently running system?" hit return, for yes
6. When all open processes are displayed, make sure that there are no "real" processes active (like python)
7. When asked if you want to change from current version to the version you selected, hit return for yes
8. You will also be asked this question for sparrow, say yes
9. When asked to remove message state cache, say yes
10. There will be a whole bunch of things to check manually, but the script should tell you what to do
    1. Be sure to restart gbweather manually, since the reset box does not work

Manual Solution

1. If you are switching from M&Cv4.5 to M&Cv4.6, be sure to Recable the receivers
2. Confirm current running version with the command gbtversion.

        $ gbtversion
        Current version of software running on the GBT is 4.2
        $
   

3. Close all user programs (GO and cleo).
4. As monctrl, stop the Solaris daemons.

        $ su - monctrl
        Password:
        $ source /home/gbt/gbt.bash
        $ cd /home/gbt
        $ ./ygor_system stop
   

5. Confirm all daemons exited:

        $ for h in earth wind fire vortex gemini virgo naiad
        > do
        > echo $h
        > ssh $h ps -u monctrl
        > echo ""
        > done | more
   

6. Confirm the targeted version exists.

        $ ls /home/gbtversions
        3.19 4.2 4.3
        $
   

7. As monctrl change the current running version of M&C, e.g.,.

        $ cd /home/gbt
        $ ./setVersion 4.3
        lrwxrwxrwx   1 monctrl  monctrl       27 Jul 26 10:36 bin -> /home/gbtversions/4.3/bin
        lrwxrwxrwx   1 monctrl  monctrl       28 Jul 26 10:36 exec -> /home/gbtversions/4.3/exec
        lrwxrwxrwx   1 monctrl  monctrl       29 Jul 26 10:36 glish -> /home/gbtversions/4.3/glish
        lrwxrwxrwx   1 monctrl  monctrl       27 Jul 26 10:36 lib -> /home/gbtversions/4.3/lib
        lrwxrwxrwx   1 monctrl  monctrl       27 Jul 26 10:36 tcl -> /home/gbtversions/4.3/tcl
        $
   

8. As monctrl change the current running version of Sparrow, e.g.,

        $ cd /home/sparrow
        $ rm -f release
        $ ln -s 4.3 release
   

9. Confirm current running version with the command gbtversion.

        $ gbtversion
        Current version of software running on the GBT is 4.3
        $
   

10. If you care to start out fresh, i.e., without carrying over cleo messages from the last version, then as monctrl remove the message cache file.

         $ rm /home/gbtlogs/Messages/vortex.stateCache
    

    You can also clean out all the log and core files too. These are located in /home/gbt/etc/log and /home/gbt/etc/cores respectively.

11. If there are config files to change, this is when to do it. Check in /home/gbt/etc/config for files with the appropriate version number appended to the name. For each file there must be two versions: one appended with the new version identifier and one appended with the current version identifier, e.g., windProc.conf.4.6 and windProc.conf.4.7, so either can be copied to windProc.conf.
12. As monctrl, start the daemons.

         $ cd /home/gbt
         $ ./ygor_system start
    

13. Check if all daemons initiated:

         $ cd /home/gbt
         $ ./ygor_system amy
    

14. Through cleo restart the single-board computers (SBC). From the CleoLauncher select Launch » Utilities / Tools » Reset Box .... In the Reset Box window press gbtas0, gbtaio2, gbtdcr, gbtspsbc, gbweather and sitetime.
15. Telnet to gbtipper and type MCReset() at the VxWorks prompt. Then ctrl-] and "quit".
16. Bring up cleo's Messages window and wait for all the programs to get reconnected. One can follow the restart better if the Messages window is configured such that Auto Remove is off, New Messages are Placed at the Top, Info messages are filtered out, and Removed Cleared is pressed fairly often. Press Refresh after setting the window up as you desire. Also one can bring up the Manager Control window to ensure all Manager come up healthy. Be patient. After the restart is completed, it is best to quit the Manager Control window.
17. If there is a new device which may affect the IFManager parameter set (e.g. a new receiver) then do this on vortex:

          TaskMaster vortex stop 4
          TaskMaster vortex start 4
    

18. Possible problems include that some daemons or SBCs need to be restarted individually because of the "strain" of everything trying to boot simultaneously, and the config files in /home/gbt/etc/config may not be compatible across versions. Individual daemons may be stopped and started via TaskMaster.
19. Do a "Reset Parameters" on the IF Rack and turn the LO1 off, then back on.
20. One must check if logging is active after a startup (/home/gbtlogs). Though a sampler2log may be up and running according to TaskMaster, it may have lost it connection to the Transporter and therefore not updating the log files. Check if the logs are updating and if not, restart the logging associated daemon.

        ls -lrt /home/gbtlogs/Spectrometer-Spectrometer-duty_cycles/
        (long list ...)
        -rw-rw-r--    1 monctrl  monctrl    201600 May 18 07:00 2004_05_18_01:03:17.fits
                                                            ^
                                                            |
                         (verify this time is updating------+ )                                 
    

21. On any machine, run the following (make sure you have permission to all devices):

         $ glish -l fixGoHang.g
    

22. Optional: Normally the glishd daemon running on victor does not have to be restarted. However...
    • To see if glishd is running on victor, ssh to victor and: sudo /sbin/run-glishd status
    • To stop glishd on victor, ssh to victor and: sudo /sbin/run-glishd stop
    • To start glishd on victor, ssh to victor and: sudo /sbin/run-glishd start
23. Update the M&C version number on the GBT Operations white board located in the GBT Control Room.

-- MarkClark - 12 Nov 2004

-- JoeBrandt - 09 Feb 2006

1. You will also be asked this question for sparrow, type n
   • Change the sparrow version to integration:
     • login in to a Linux box as monctrl
     • type: source /home/gbt/gbt.bash
     • type: cd /home/gbt
     • type: rm sparrow
     • type: ln -s /home/sparrow/integration sparrow
     • type: cd /home/sparrow
     • type: rm release
     • type: ln -s /home/sparrow/integration release

• • • RcvrPF_2 (all samplers)

Directions for Operators to Switch from 4.7 to 4.8

• Directions for Operators to Switch versions

Directions for Operators to Switch versions

1. ./switchVersions 4.8

1. run ./switchVersions 5.4 (or the version number you want to switch to)

• • verify that the new version listed is 4.8 then hit the enter key.

• • verify that the new version listed is 5.4 (or the desired version) then hit the enter key.

1. for 5.4: Confirm the orange cable is connected to the front of the Dcr chassis. Follow the switchover procedure if it is not.

• • The following files will be changed:
    • AntennaCharacterization.conf
    • Inclinometer.conf
    • system.conf
    • fireProc,conf
    • windProc.conf
    • vortexProc.conf. When switching the version back to 4.7, these 6 files are the only files that should be switched back to version 4.7. Say 'NO' to all other files.

• • Exceptions for 4.8 are:
    • RcvrPF_2/RcvrPF_2/powerSuppy
    • RcvrPF_2/RcvrPF_2/cryogenics
    • MotorRack/RefridgeratorPowerSupply5/refridge

• • Exceptions for 5.4 are:
    • MotorRack-RefridgeratorPowerSupply5-refridge
    • Rcvr40_52-Rcvr40_52-gregorian

Directions for Operators to Switch from 4.8 to 4.7

• Step 4 ./switchVersions 4.7
• Step 9. You will also be asked this question for sparrow, type y
• Step 11. Say yes to the files:

       AntennaCharacterization.conf.4.7
       Inclinometer.conf.4.7 
       system.conf.4.7 
       fireProc,conf.4.7
       windProc.conf.4.7 
       vortexProc.conf.4.7 
       Say 'NO' to all other 4.7 files.

Directions for Operators to Switch from 5.4 to 5.3

• Step 4 ./switchVersions 5.3
• Step 7. Reverse the procedure specificed in this step.
• Step 10. You will also be asked this question for sparrow, type y
• Step 12. Say yes to the files:

       system.conf.5.3  
       Say 'NO' to all other 5.3 files.

Operators Direction for Switching from 4.7 to 4.8

Directions for Operators to Switch from 4.7 to 4.8

ssh theMachineName su monctrl source /home/gbt/gbt.bash kill -9 PID (where the PID is the process Id noted above)

• • • • • • ssh theMachineName
          • su monctrl
          • source /home/gbt/gbt.bash
          • kill -9 PID (where the PID is the process Id noted above)

• • The following files will be changed: AntennaCharacterization.conf, Inclinometer.conf, system.conf, fireProc,conf, windProc.conf and vortexProc.conf.

• • The following files will be changed:
    • AntennaCharacterization.conf
    • Inclinometer.conf
    • system.conf
    • fireProc,conf
    • windProc.conf
    • vortexProc.conf.

type

• • • type: y

• • • type: =MCReset

• • • type: MCReset

Operators Direction for Switching from 4.8 to 4.7

Directions for Operators to Switch from 4.8 to 4.7

• • • • type: n
      • Verify that there are no rogue processes
        • type: y

1. You will also be asked this question for sparrow, say yes
2. When asked to remove message state cache, say yes
3. There will be a whole bunch of things to check manually, but the script should tell you what to do
   1. Be sure to restart gbweather manually, since the reset box does not work

1. You will also be asked this question for sparrow, type n
   • Change the sparrow version to integration:
     • login in to a Linux box as monctrl
     • type: source /home/gbt/gbt.bash
     • type: cd /home/gbt
     • type: rm sparrow
     • type: ln -s /home/sparrow/integration sparrow
     • type: cd /home/sparrow
     • type: rm release
     • type: ln -s /home/sparrow/integration release
2. When asked to remove message state cache, type: y
3. When asked about the config directory type: y
   • The following files will be changed: AntennaCharacterization.conf, Inclinometer.conf, system.conf, fireProc,conf, windProc.conf and vortexProc.conf. When switching the version back to 4.7, these 6 files are the only files that should be switched back to version 4.7. Say 'NO' to all other files.
4. When asked to flush portmaps type: y
5. When asked to start system type: y
6. Verify that there are no start up failures * If there are failures restart these tasks

   type

7. Bring up the CLEO manager control, message and reset box displays
   • Verify that all the managers are up
   • Using the Reset box, reset the managers listed in by the script
   • Reset the tipper by using the following commands:
     • from a Linux window type: telnet gbtipper
     • login in as monctrl (use the old monctrl password)
     • type: =MCReset NOTE: nothing happens, you will not be able to type in any other commands while the single board computer is rebooting. This is normal. The telnet session will eventaully time out.
   • Manually restart the Spectrometer singleboard in the equipment room by pressing the reset button on the Spectrometer single board.
   • Manuully restart gbweather because the reset box does not work
8. When the sampler files are displayed (Its a LONG list), verify that all the samplers have files listed under them that are approximately equal to the current time.
   • Exceptions for 4.8 are:
     • RcvrPF_2/RcvrPF_2/powerSuppy
     • RcvrPF_2/RcvrPF_2/cryogenics
     • MotorRack/RefridgeratorPowerSupply5/refridge
     • Tipper
   • type: n if all other samplers are producing files.
9. When asked to run fixGoHang:
   • Be sure monctrl has antenna and other device security priviledges
   • type: y
10. When asked to verify that glish.d is running type: n
11. When asked to restart the IFManager type: n
12. When asked to restart the LO1 type: y

       AntennaCharacterization.conf.4.7
       Inclinometer.conf.4.7 
       system.conf.4.7 
       fireProc,conf.4.7
       windProc.conf.4.7 
       vortexProc.conf.4.7 
       Say 'NO' to all other 4.7 files.

Operators Direction for Switching from 4.7 to 4.8

1. Log in as monctrl to any Linux machine
2. source /home/gbt/gbt.bash
3. cd to /home/gbt
4. ./switchVersions 4.8
5. When you are asked if this looks sane:
   • verify that the new version listed is 4.8 then hit the enter key.
6. When you are asked "Should I shut down the currently running system?"
   • type: y
   • It takes a few minutes to shut down the current version.
7. When all open processes are displayed for each machine (vortex, wind, fire etc):
   • Verify that there are no rogue processes. There should be no python, MMConverter or TaskMaster processes in the lists
     • If there any of these processes exist, note the PID and the machine that it is running on.
       • Kill the rogue processes using the following commands:
         • login to the machine where the process is running by typing:

                             ssh theMachineName
                             su monctrl
                             source /home/gbt/gbt.bash
                             kill -9 PID    (where the PID is the process Id noted above)
           

8. When asked if you want to change from current version to the version you selected, hit return for yes
9. You will also be asked this question for sparrow, say yes
10. When asked to remove message state cache, say yes
11. There will be a whole bunch of things to check manually, but the script should tell you what to do
    1. Be sure to restart gbweather manually, since the reset box does not work

1. ./switchVersions.sh version-to-switch-to

1. ./switchVersions version-to-switch-to

Solution

Automated Solution

1. Log in as monctrl to any Linux machine
2. source /home/gbt/gbt.bash
3. cd to /home/gbt
4. ./switchVersions.sh version-to-switch-to
5. When you are asked "Should I shut down the currently running system?" hit return, for yes
6. When all open processes are displayed, make sure that there are no "real" processes active (like python)
7. When asked if you want to change from current version to the version you selected, hit return for yes
8. You will also be asked this question for sparrow, say yes
9. When asked to remove message state cache, say yes
10. There will be a whole bunch of things to check manually, but the script should tell you what to do
    1. Be sure to restart gbweather manually, since the reset box does not work

Manual Solution

1. Normally the glishd daemon running on victor does not have to be restarted. However...

1. Optional: Normally the glishd daemon running on victor does not have to be restarted. However...

$ source /home/gbt/gbt.bash

-- MarkClark - 11 Nov 2004

-- MarkClark - 12 Nov 2004

1. (8/4/04) - If you are switching from M&Cv4.4 to M&Cv4.5, be sure to Recable the SSS
2. (9/18/04) - If you are switching from M&Cv4.5 to M&Cv4.6, be sure to Recable the receivers

1. If you are switching from M&Cv4.5 to M&Cv4.6, be sure to Recable the receivers

1. Confirm all daemons exited:

        $ for h in earth wind fire vortex gemini virgo naiad
        > do
        > echo $h
        > ssh $h ps -u monctrl
        > echo ""
        > done | more
   

1. As monctrl change the current running version of M&C.

1. As monctrl change the current running version of M&C, e.g.,.

1. As monctrl change the current running version of Sparrow.

1. As monctrl change the current running version of Sparrow, e.g.,

1. If there are config files to change, this is when to do it. Check in /home/gbt/etc/config for files with the appropriate version number appended to the name.
2. (9/18/04) - If you are switching from M&Cv4.5 to M&Cv4.6, be sure to copy the following files (check to see if you need to save the 4.5 versions before copying)

                fireProc.conf.4.6 to fireProc.conf, 
                system.conf.4.6 to system.conf, 
                devices.conf.4.6 to devices.conf, 
                SIBMap.conf.4.6 SIBMap.conf 
                /home/gbt/ygor_system.4.6 to /home/gbt/ygor_system
                /home/gbt/etc/startup/gbtaio3.startup.4.6 to /home/gbt/etc/startup/gbtaio3.startup
            
   

3. (10/8/04) - If you are switching from M&Cv4.6 to M&Cv4.7, be sure to copy the following files (check to see if you need to save the 4.6 versions before copying)

                fireProc.conf.4.7 to fireProc.conf 
   

1. If there are config files to change, this is when to do it. Check in /home/gbt/etc/config for files with the appropriate version number appended to the name. For each file there must be two versions: one appended with the new version identifier and one appended with the current version identifier, e.g., windProc.conf.4.6 and windProc.conf.4.7, so either can be copied to windProc.conf.

1. Through cleo restart the single-board computers (SBC). From the CleoLauncher select Launch » Utilities / Tools » Reset Box .... In the Reset Box window press gbtaio2, gbteio1, gbtdcr, gbtspsbc, gbtas0, gbweather and sitetime.

1. Check if all daemons initiated:

        $ cd /home/gbt
        $ ./ygor_system amy
   

2. Through cleo restart the single-board computers (SBC). From the CleoLauncher select Launch » Utilities / Tools » Reset Box .... In the Reset Box window press gbtas0, gbtaio2, gbtdcr, gbtspsbc, gbweather and sitetime.

1. Possible problems include that some daemons or SBCs need to be restarted individually because of the "strain" of everything trying to boot simultaneously, and the config files in /home/gbt/etc/config may not be compatible across versions. Individual daemons may be stopped and started via TaskMaster, e.g., on gemini:

        $ TaskMaster gemini status
        Number  pid     last action     Name                    Args            
        1       4176    Start           SpStorage                                       
        2       4177    Start           SpectralProcessorA                                      
        3       4178    Start           SpectralProcessorB                                      
        4       4179    Start           SpectralProcessor                                       
        $ TaskMaster gemini stop 1
        $ TaskMaster gemini status
        Number  pid     last action     Name                    Args            
        1       4176    Stop            SpStorage                                       
        2       4177    Start           SpectralProcessorA                                      
        3       4178    Start           SpectralProcessorB                                      
        4       4179    Start           SpectralProcessor                                       
        $ TaskMaster gemini start 1
        $ TaskMaster gemini status
        Number  pid     last action     Name                    Args            
        1       4252    Start           SpStorage                                       
        2       4177    Start           SpectralProcessorA                                      
        3       4178    Start           SpectralProcessorB                                      
        4       4179    Start           SpectralProcessor                                       
        $ 
   

1. Possible problems include that some daemons or SBCs need to be restarted individually because of the "strain" of everything trying to boot simultaneously, and the config files in /home/gbt/etc/config may not be compatible across versions. Individual daemons may be stopped and started via TaskMaster.

-- MarkClark - 22 Sep 2003

-- MarkClark - 11 Nov 2004

1. (10/8/04) - If you are switching from M&Cv4.6 to M&Cv4.7, be sure to copy the following files (check to see if you need to save the 4.6 versions before copying)

                fireProc.conf.4.7 to fireProc.conf 
   

1. If switching between 4.1 and 4.2, follow the instructions for reconfiging the analog filter and converter rack control cables

/home/gbt/ygor_system.4.6 to /home/gbt/ygor_system.

/home/gbt/ygor_system.4.6 to /home/gbt/ygor_system /home/gbt/etc/startup/gbtaio3.startup.4.6 to /home/gbt/etc/startup/gbtaio3.startup

1. (9/18/04) - If you are switching from M&Cv4.5 to M&Cv4.6, be sure to Recable the receivers

1. (9/18/04) - If you are switching from M&Cv4.5 to M&Cv4.6, be sure to copy the following files (check to see if you need to save the 4.5 versions before copying)

fireProc.conf.4.6 to fireProc.conf, system.conf.4.6 to system.conf, devices.conf.4.6 to devices.conf, SIBMap.conf.4.6 SIBMap.conf /home/gbt/ygor_system.4.6 to /home/gbt/ygor_system.

1. Do a "Reset Parameters" on both the IF Rack and the LO1.

1. Do a "Reset Parameters" on the IF Rack and turn the LO1 off, then back on.

1. If there are config files to change, this is when to do it. Check in /home/gbt/etc/config for files with the appropriate version number appended to the name.

1. If there are config files to change, this is when to do it. Check in /home/gbt/etc/config for files with the appropriate version number appended to the name.

1. Through cleo restart the single-board computers (SBC). From the CleoLauncher select Launch » Utilities / Tools » Reset Box .... In the Reset Box window press gbtaio2, gbteio1, gbtdcr, gbtspsbc, gbtas0, weather and sitetime.

1. Through cleo restart the single-board computers (SBC). From the CleoLauncher select Launch » Utilities / Tools » Reset Box .... In the Reset Box window press gbtaio2, gbteio1, gbtdcr, gbtspsbc, gbtas0, gbweather and sitetime.

1. (8/4/04) - If you are switching from M&Cv4.4 to M&Cv4.5, be sure to Recable the SSS

1. If there are config files to change, this is when to do it.

1. If there are config files to change, this is when to do it. Check in /home/gbt/etc/config for files with the appropriate version number appended to the name.

1. Through cleo restart the single-board computers (SBC). From the CleoLauncher select Launch » Utilities / Tools » Reset Box .... In the Reset Box window press gbtaio2, gbteio1, gbtdcr, gbtspsbc, gbtas0, and sitetime.

1. Through cleo restart the single-board computers (SBC). From the CleoLauncher select Launch » Utilities / Tools » Reset Box .... In the Reset Box window press gbtaio2, gbteio1, gbtdcr, gbtspsbc, gbtas0, weather and sitetime.

Current version of software running on the GBT is 3.3.0

Current version of software running on the GBT is 4.2

and the Spectrometer

     $ ssh earth
     $ TaskMaster earth systemstop
     $ exit

3.2.3 3.3.0 3.3.1 fits

3.19 4.2 4.3

$ ./setVersion 3.3.1 lrwxrwxrwx 1 monctrl monctrl 27 Jul 26 10:36 bin -> /home/gbtversions/3.3.1/bin lrwxrwxrwx 1 monctrl monctrl 28 Jul 26 10:36 exec -> /home/gbtversions/3.3.1/exec lrwxrwxrwx 1 monctrl monctrl 29 Jul 26 10:36 glish -> /home/gbtversions/3.3.1/glish lrwxrwxrwx 1 monctrl monctrl 27 Jul 26 10:36 lib -> /home/gbtversions/3.3.1/lib lrwxrwxrwx 1 monctrl monctrl 27 Jul 26 10:36 tcl -> /home/gbtversions/3.3.1/tcl

$ ./setVersion 4.3 lrwxrwxrwx 1 monctrl monctrl 27 Jul 26 10:36 bin -> /home/gbtversions/4.3/bin lrwxrwxrwx 1 monctrl monctrl 28 Jul 26 10:36 exec -> /home/gbtversions/4.3/exec lrwxrwxrwx 1 monctrl monctrl 29 Jul 26 10:36 glish -> /home/gbtversions/4.3/glish lrwxrwxrwx 1 monctrl monctrl 27 Jul 26 10:36 lib -> /home/gbtversions/4.3/lib lrwxrwxrwx 1 monctrl monctrl 27 Jul 26 10:36 tcl -> /home/gbtversions/4.3/tcl

$ rm release $ ln -s 3.3.1 release

$ rm -f release $ ln -s 4.3 release

Current version of software running on the GBT is 3.3.1

Current version of software running on the GBT is 4.3

and the Spectrometer

     $ ssh earth
     $ TaskMaster earth systemstart /home/gbt/etc/config/earthProc.conf
     $ exit

1. If there is a new device (e.g. receiver) then do this on vortex:

1. If there is a new device which may affect the IFManager parameter set (e.g. a new receiver) then do this on vortex:

    ls -lrt /home/gbtlogs/Spectrometer-Spectrometer-duty_cycles/
    (long list ...)
    -rw-rw-r--    1 monctrl  monctrl    201600 May 18 07:00 2004_05_18_01:03:17.fits
                                                        ^
                                                        |
                     (verify this time is updating------+ )                                 

1. As monctrl, start the Solaris daemons.

1. As monctrl, start the daemons.

1. If switching between 4.1 and 4.2, follow the instructions for reconfiging the analog filter and converter rack control cables

$ rm -rf release

$ rm release

1. Telnet to gbtipper and type MCReset() at the VxWorks prompt.

1. Telnet to gbtipper and type MCReset() at the VxWorks prompt. Then ctrl-] and "quit".

1. If there is a new device (e.g. receiver) then do this on vortex:

         TaskMaster vortex stop 4
         TaskMaster vortex start 4
   

1. Through CLEO restart the single-board computer (SBC) gbtaio1 again. This is a workaround for a bug in the IFRack (see project office #864).

1. Do a "Reset Parameters" on both the IF Rack and the LO1.

1. As monctrl change the current running version.

1. As monctrl change the current running version of M&C.

1. As monctrl change the current running version of Sparrow.

        $ cd /home/sparrow
        $ rm -rf release
        $ ln -s 3.3.1 release
   

1. Telnet to gbtipper and type MCReset() at the VxWorks prompt.

1. If there are config files to change, this is when to do it.

1. One must check if logging is active after a startup. Though a sampler2log may be up and running according to TaskMaster, it may have lost it connection to the Transporter and therefore not updating the log files. Check if the logs are updating and if not, restart the logging associated daemon.

1. One must check if logging is active after a startup (/home/gbtlogs). Though a sampler2log may be up and running according to TaskMaster, it may have lost it connection to the Transporter and therefore not updating the log files. Check if the logs are updating and if not, restart the logging associated daemon.

1. Confirm current running version with the command gbtversion.

1. Confirm current running version with the command gbtversion.

2. Close all user programs (GO and cleo).

3. As monctrl, stop the Solaris daemons.

1. Close all user programs (GO and cleo).
2. As monctrl, stop the Solaris daemons.

4. Confirm the targeted version exists.

1. Confirm the targeted version exists.

5. As monctrl change the current running version.

1. As monctrl change the current running version.

6. Confirm current running version with the command gbtversion.

1. Confirm current running version with the command gbtversion.

7. If you care to start out fresh, i.e., without carrying over cleo messages from the last version, then as monctrl remove the message cache file.

1. If you care to start out fresh, i.e., without carrying over cleo messages from the last version, then as monctrl remove the message cache file.

8. As monctrl, start the Solaris daemons.

1. As monctrl, start the Solaris daemons.

9. Through cleo restart the single-board computers (SBC). From the CleoLauncher select Launch » Utilities / Tools » Reset Box .... In the Reset Box window press gbtaio2, gbteio1, gbtdcr, gbtspsbc, gbtas0, and sitetime.

10. Bring up cleo's Messages window and wait for all the programs to get reconnected. One can follow the restart better if the Messages window is configured such that Auto Remove is off, New Messages are Placed at the Top, Info messages are filtered out, and Removed Cleared is pressed fairly often. Press Refresh after setting the window up as you desire. Also one can bring up the Manager Control window to ensure all Manager come up healthy. Be patient. After the restart is completed, it is best to quit the Manager Control window.

11. Possible problems include that some daemons or SBCs need to be restarted individually because of the "strain" of everything trying to boot simultaneously, and the config files in /home/gbt/etc/config may not be compatible across versions. Individual daemons may be stopped and started via TaskMaster, e.g., on gemini:

1. Through cleo restart the single-board computers (SBC). From the CleoLauncher select Launch » Utilities / Tools » Reset Box .... In the Reset Box window press gbtaio2, gbteio1, gbtdcr, gbtspsbc, gbtas0, and sitetime.
2. Bring up cleo's Messages window and wait for all the programs to get reconnected. One can follow the restart better if the Messages window is configured such that Auto Remove is off, New Messages are Placed at the Top, Info messages are filtered out, and Removed Cleared is pressed fairly often. Press Refresh after setting the window up as you desire. Also one can bring up the Manager Control window to ensure all Manager come up healthy. Be patient. After the restart is completed, it is best to quit the Manager Control window.
3. Possible problems include that some daemons or SBCs need to be restarted individually because of the "strain" of everything trying to boot simultaneously, and the config files in /home/gbt/etc/config may not be compatible across versions. Individual daemons may be stopped and started via TaskMaster, e.g., on gemini:

12. Through CLEO restart the single-board computer (SBC) gbtaio1 again. This is a workaround for a bug in the IFRack (see project office #864).

13. On any machine, run the following (make sure you have permission to all devices):

1. Through CLEO restart the single-board computer (SBC) gbtaio1 again. This is a workaround for a bug in the IFRack (see project office #864).
2. One must check if logging is active after a startup. Though a sampler2log may be up and running according to TaskMaster, it may have lost it connection to the Transporter and therefore not updating the log files. Check if the logs are updating and if not, restart the logging associated daemon.
3. On any machine, run the following (make sure you have permission to all devices):

14. Normally the glishd daemon running on victor does not have to be restarted. However...

1. Normally the glishd daemon running on victor does not have to be restarted. However...

1. Update the M&C version number on the GBT Operations white board located in the GBT Control Room.

15. Update the M&C version number on the GBT Operations white board located in the GBT Control Room.

-- AmyShelton - 20 Aug 2003

-- MarkClark - 22 Sep 2003

-- AmyShelton - 30 Jul 2003

15. Update the M&C version number on the GBT Operations white board located in the GBT Control Room.

-- AmyShelton - 20 Aug 2003

$ su monctrl

$ su - monctrl

%META:TOPICINFO{author="AmyShelton" date="1059597480" format="1.0" version="1.1"}%

Problem

How do I switch M&C software versions for the GBT?

Solution

1. Confirm current running version with the command gbtversion.

$ gbtversion
Current version of software running on the GBT is 3.3.0
$

2. Close all user programs (GO and cleo).

3. As monctrl, stop the Solaris daemons.

$ su monctrl
Password:
$ cd /home/gbt
$ ./ygor_system stop

$ ssh earth
$ TaskMaster earth systemstop
$ exit

4. Confirm the targeted version exists.

$ ls /home/gbtversions
3.2.3  3.3.0  3.3.1  fits
$

5. As monctrl change the current running version.

$ cd /home/gbt
$ ./setVersion 3.3.1
lrwxrwxrwx   1 monctrl  monctrl       27 Jul 26 10:36 bin -> /home/gbtversions/3.3.1/bin
lrwxrwxrwx   1 monctrl  monctrl       28 Jul 26 10:36 exec -> /home/gbtversions/3.3.1/exec
lrwxrwxrwx   1 monctrl  monctrl       29 Jul 26 10:36 glish -> /home/gbtversions/3.3.1/glish
lrwxrwxrwx   1 monctrl  monctrl       27 Jul 26 10:36 lib -> /home/gbtversions/3.3.1/lib
lrwxrwxrwx   1 monctrl  monctrl       27 Jul 26 10:36 tcl -> /home/gbtversions/3.3.1/tcl
$

6. Confirm current running version with the command gbtversion.

$ gbtversion
Current version of software running on the GBT is 3.3.1
$

7. If you care to start out fresh, i.e., without carrying over cleo messages from the last version, then as monctrl remove the message cache file.

$ rm /home/gbtlogs/Messages/vortex.stateCache

You can also clean out all the log and core files too. These are located in /home/gbt/etc/log and /home/gbt/etc/cores respectively.

8. As monctrl, start the Solaris daemons.

$ cd /home/gbt
$ ./ygor_system start

$ ssh earth
$ TaskMaster earth systemstart /home/gbt/etc/config/earthProc.conf
$ exit

9. Through cleo restart the single-board computers (SBC). From the CleoLauncher select Launch » Utilities / Tools » Reset Box .... In the Reset Box window press gbtaio2, gbteio1, gbtdcr, gbtspsbc, gbtas0, and sitetime.

10. Bring up cleo's Messages window and wait for all the programs to get reconnected. One can follow the restart better if the Messages window is configured such that Auto Remove is off, New Messages are Placed at the Top, Info messages are filtered out, and Removed Cleared is pressed fairly often. Press Refresh after setting the window up as you desire. Also one can bring up the Manager Control window to ensure all Manager come up healthy. Be patient. After the restart is completed, it is best to quit the Manager Control window.

11. Possible problems include that some daemons or SBCs need to be restarted individually because of the "strain" of everything trying to boot simultaneously, and the config files in /home/gbt/etc/config may not be compatible across versions. Individual daemons may be stopped and started via TaskMaster, e.g., on gemini:

$ TaskMaster gemini status
Number  pid     last action     Name                    Args            
1       4176    Start           SpStorage                                       
2       4177    Start           SpectralProcessorA                                      
3       4178    Start           SpectralProcessorB                                      
4       4179    Start           SpectralProcessor                                       
$ TaskMaster gemini stop 1
$ TaskMaster gemini status
Number  pid     last action     Name                    Args            
1       4176    Stop            SpStorage                                       
2       4177    Start           SpectralProcessorA                                      
3       4178    Start           SpectralProcessorB                                      
4       4179    Start           SpectralProcessor                                       
$ TaskMaster gemini start 1
$ TaskMaster gemini status
Number  pid     last action     Name                    Args            
1       4252    Start           SpStorage                                       
2       4177    Start           SpectralProcessorA                                      
3       4178    Start           SpectralProcessorB                                      
4       4179    Start           SpectralProcessor                                       
$ 

12. Through CLEO restart the single-board computer (SBC) gbtaio1 again. This is a workaround for a bug in the IFRack (see project office #864).

13. On any machine, run the following (make sure you have permission to all devices):

$ glish -l fixGoHang.g

14. Normally the glishd daemon running on victor does not have to be restarted. However...

• • To see if glishd is running on victor, ssh to victor and: sudo /sbin/run-glishd status
  • To stop glishd on victor, ssh to victor and: sudo /sbin/run-glishd stop
  • To start glishd on victor, ssh to victor and: sudo /sbin/run-glishd start

-- AmyShelton - 30 Jul 2003 %META:FORM{name="WebForm"}% %META:FIELD{name="TopicClassification" title="TopicClassification" value="NoDisclosure"}% %META:FIELD{name="OperatingSystem" title="OperatingSystem" value="OsLinux, OsSolaris"}% %META:FIELD{name="OsVersion" title="OsVersion" value=""}%