NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Knowledge > SwitchingMCVersions

Readme | Changes | Index | Search | Go

Problem

How do I switch M&C software versions for the GBT?

• Problem
• General Method to Switch versions:
• Version Specific Notes:
  • Switching from 5.7 to 6.1:
  • Switching from 6.1 to 5.7:

General Method to Switch versions:

1. As soon as the time for the tests has been scheduled, an email should be sent to gbtlocal, warning users and other developers that this change is occuring: GBT Software Testing is scheduled for date-time1 to date-time2.  For this period, the following programs will be available, but running test versions, and users may wish to refrain from using them.  The affected applications include:
   • the GBT M&C System
   • astrid
   • gfm
   • turtle
   • gbtstatus
   • sdfits
   • logview
   • gbt2class
   • config tool
   • For users who need to continue working with sdfits during this test period, follow these steps to run the production version (note this only applies to sdfits, and not other applications):
     • $ source /home/sparrow/PRODUCTION_VERSION_NUMBER/sparrow.bash
     • $ sdfits
2. If this is a release, please email the release notes to gbtlocal AND update the main Software wiki page with links to the release notes (they go in the "Testing & Releases" table).
3. Login as monctrl to a Linux machine (see exception notes above).
4. Request that the operator add monctrl to the security gateway file with full permissions.
5. source /home/gbt/gbt.bash
6. cd /home/gbt
7. Run ./switchVersions 6.1 (or the version number to which you want to switch)
8. When you are asked if this looks sane:
   • Verify that the new version listed is 6.1 (or the desired version) then hit enter.
9. When you are asked "=Should I shut down the currently running system?="
   • type: y
   • It takes a few minutes to shut down the current version.
10. When all open processes are displayed for each machine (gbtdata, wind, fire etc):
    • Verify that there are no rogue processes. There should be no Python, MMConverter or TaskMaster processes in the lists. If there are none, type: y
    • If any of these processes are listed, type: n to the rogue processes query, and manually kill each one:
      • Note the process ID (PID) and the machine on which each rogue process is running.
      • Open another shell window, and kill each rogue processes using the following commands:
        • Login to the machine where the process is running by typing:
          • ssh <theMachineName> (where <theMachineName> is the machine noted above)
          • su monctrl
          • source /home/gbt/gbt.bash
        • kill -9 <PID> (where <PID> is the process ID noted above)
      • When there are no more rogue processes, type: y at the rogue processes query
11. When asked if you want to change from current version to the version you specified, hit enter for yes
12. When asked to remove message state cache, type: y
13. When asked about the config directory type: y
14. When asked to flush portmaps type: y
15. When asked to update kernel drivers, hit enter for no
16. When asked to start system type: y
17. Verify that there are no start up failures
    • If there are failures, manually restart these tasks (except IFManager; that's handled in later part of script):
      • Login to the machine hosting a failed-to-start task by typing:
        • ssh <theMachineName> (where <theMachineName> is the machine on which the failed process is hosted - check /home/gbt/etc/config/system.conf)
        • su monctrl
      • Restart the failed task by typing:
        • source /home/gbt/gbt.bash
        • tm status to list hosted processes; take note of the process number for the failed process
        • tm start , where is the process number noted in last step
    • Once all failed-to-restart processes are manually restarted, type: y
18. Bring up the CLEO manager control, message and reset box displays
    • Verify that all the managers are up
    • Using the Reset box, reset the managers listed by the script
    • Reset the tipper by using the following commands:
      • From a Linux window type: telnet gbtipper
      • Login in as monctrl (use the old monctrl password)
      • type: MCReset NOTE: nothing happens, you will not be able to type in any other commands while the single board computer is rebooting. This is normal. The telnet session will eventaully time out.
    • Manually restart the Spectrometer singleboard in the equipment room by pressing the red reset button on the Spectrometer single board.
19. When asked to run fixGoHang:
    • Be sure monctrl has antenna and other device security privileges
    • type: y
20. When asked to restart the IFManager type: y
21. When asked to restart the LO1 type: y
22. When the sampler files are displayed (it's a LONG list), verify that all the samplers have files listed under them with timestamps that are approximately equal to the current time.
    • Exceptions for 6.1 are:
      • PF2 samplers when PF1 is installed, and vice versa
      • Tipper
      • Inclinometer
    • type: n if all other samplers are producing files.
23. The script that you have run has also switched the sparrow Sparrow version for you. Sometimes, the version of GBTIDL must be changed as well. However, this should only be done during a release. We can't switch GBTIDL versions during integration/regression tests as this would affect current users.. Contact BobGarwood or PaulMarganian to find out if this is necessary. The current system for switching GBTIDL versions is identical to that used by M&C and sparrow. It simply involves switching a symbolic link. As monctrl:
    • • cd /home/gbtidl
      • rm -f release
      • ln -s GBTIDl_RELEASE release (where GBTIDL_RELEASE is the latest version number, ex: 2.3.1)

Version Specific Notes:

Switching from 5.7 to 6.1:

1. The machines earth, wind and fire must be rebooted to run Redhat Enterprise 4 (RHE4)
   • Log into each machine as root and do:
     • mount /dev/hda3 /mnt
     • cp /mnt/boot/grub/grub.conf.6.1 /mnt/boot/grub/grub.conf
     • sync;sync;reboot
     • logout
2. Prior to running switchversions, copy _installation.conf.6.1 to _installation.conf and manually shutdown TaskMaster on vortex, tank, and algol.
   • for i in algol tank vortex; do TaskMaster $i systemstop; done
3. switchversions must be run from a system running the RHE4 operating system.
4. When asked to reset the spectrometer SBC, also swap the orange DCR cable from algol to tank (the one with the purple tag should be plugged into tank).
5. If running 6.1, cleo screens may only be run from a system running RHE4, such as tank or ariel (to be resolved soon).
6. The tipper is down, skip the reset tipper step.
7. The inclinometer is down, ignore the inclinometer startup failure.

Switching from 6.1 to 5.7:

1. The machines earth, wind and fire must be rebooted to run Redhat 9 (RH9)
   • Log into each machine as root and do (Note the difference from above is correct!):
     • cp /boot/grub/grub.conf.5.7 /boot/grub/grub.conf
     • sync;sync;reboot
     • logout
2. Prior to running switchversions, copy _installation.conf.5.7 to _installation.conf and manually shutdown TaskMaster on tank
   • TaskMaster tank systemstop
3. When asked to reset the spectrometer SBC, also swap the orange DCR cable from tank to algol (the one with the purple tag should be plugged into algol).
4. The tipper is down, skip the reset tipper step.
5. The inclinometer is down, ignore the inclinometer startup failure.

-- JoeBrandt - 09 Feb 2006