NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Software > PlanOfRecordC72006 > ModificationRequest4C706

Changes | Index | Contents | Search | Statistics | Go

GFM/Turtle Upgrade - Remove use of LPCs.conf file

Modification Request #4 (C07 2006)

• 1. Introduction
• 2. Background
• 3. Requirements
• 4. Design
  • Modifications to GFM
  • Modifications to Turtle
• 5. Deployment Checklist
• 6. Test Plan
  • 7.1 Simulator testing
  • 7.2 Functional testing on GBT
  • 7.3 Integration/Regression Tests
• Signatures
• CCC Discussion Area

1. Introduction

Interactions between GFM and Turtle that set the Local Pointing Corrections (LPCs) and Local Focus Corrections (LFCs) have been failing at an unacceptably frequent rate. This MR therefore details plans to correct the unreliable communication of LPCs and LFCs from GFM to Turtle.

2. Background

GBT observations are aided by observation directives such as Peak, Focus, AutoPeak, AutoFocus, etc. These provide Local Pointing Corrections (LPCs) and Local Focusing Corrections (LFCs) to the telescope, to provide the best possible pointing and focusing settings under the current observing conditions.

The computation of LFC and LPC values is performed by the GBT Fits Monitoring program GFM. These corrections are currently written by GFM to a data file, which is then picked up by Turtle (the Observing API daemon), which then updates the GBT M&C Antenna Manager with the values.

This arrangement, which is a result of the history behind the development of an observation management system, has a serious drawback in that the method by which GFM and Turtle communicate is not as reliable as one would like. It is all too common for observers and operators to report that while LPCs and LFCs have been written, Turtle was unable to upload the corrections to the Antenna Manager. Investigations by the Software Development Division into this problem have shown that the cause for this unreliability lies with quirks in the operation of the Networked File System (NFS). (It should be noted that NFS was not intended for real-time performance.)

3. Requirements

The interaction between GFM and Turtle should work reliably, using an interprocess communication (IPC) method that was designed for robust and timely exchange of data between applications.

4. Design

There are three main alternatives designs:

• Move all LPC/LFC duties to Turtle.
• Leave LPC/LFC generation in GFM, and move responsibility for updating the Antenna to GFM.
• Leave each program with its current responsibilities, but replace the IPC-through-file mechanism with something more reliable.

The first option is undesirable because GFM obtains parameters for the LPC/LFC generation from the observer. Since GFM is a user program and Turtle is a daemon, the generation of these corrections thus properly resides within GFM.

The second option is more reasonable; however, if GFM is to generate the corrections and also update the antenna manager, it must still have some way of handshaking with Turtle to allow Turtle to pause while this process is going on and proceed with the scheduling block once corrections are updated. In short, it must still handshake with GFM, using some sort of IPC method. This is in addition to the new functionality which must be added to GFM to communicate with the Antenna Manager.

This leaves the last alternative as the most attractive one. It would require the least refactoring of Turtle and GFM. The responsibilities of each piece of software would not change. All that is required is to replace the file IPC method with something else.

Many suitable alternatives to using files for IPCs exist, and any one of these may be used with equal success:

• Raw TCP sockets
• SOAP
• XML/RPC
• Pyro
• etc...

Of these alternatives, the most attractive is to use Pyro, a Python based IPC library, simply because Turtle is already a Pyro service (Pyro is used for IPC between Turtle and Astrid), and Turtle and GFM are both written in Python.

Modifications to GFM

• A pyro client module will be added.
• The code that writes LPC/LFC files will be removed: In PointingPlugin.py, remove function PointingPlugin::WriteLPCFile(). In FocusPlugin.py, remove function FocusPlugin::WriteLPCFile() (Even though this is the focus plugin, the function is still WriteLPCFile).
• In both these plugin modules, references to WriteLPCFile(), called in UpdateLPCs(), will be replaced by the appropriate call to the Pyro module added above.

Modifications to Turtle

Turtle currently pauses a scheduling block and waits until an LPC/LFC file is modified before proceeding. This code shall be removed. In its place a Pyro service will be added that waits for LPC/LFC updates. When these occur, some sort of synchronization mechanism will be used to allow Turtle to be notified. Turtle could, for instance, fall into a method that does not return until LPC/LFCs are received, or a time-out occurs.

Turtle's Antenna.py is where this appears to occur. Antenna member function UpdateCorrections() seems to be the function. Its caller drops into it for up to waittime seconds. The function waits for the file in a 1 second loop, creating an instance of LocalCorrections every time through and looking at the modification time of the file, as returned by LocalCorrections.GetMTime(). The function returns 1 if the file's modification time comes after the start time and the 'last_scan' field is equal to the scan number. It returns 0 otherwise. LocalCorrections reads the file every time it is created. The way forward seems to be to keep Antenna.UpdateCorrections(), but completely change its innards. The Pyro service would have to be running at all times in its own thread. When it receives corrections it can save them in a buffer/queue and assert a condition variable to break UpdateCorrections() out of its wait. Then UpdateCorrections() can do the same checks as before, except that it would only check the scan number, since there is no longer a file to check against the start time. Or, the calculations made by GFM could be time-stamped to obtain this time for this comparison.

5. Deployment Checklist

Very little needs to be done here: There is already support in place in the system for Pyro, and ideally the system should work exactly as before, only more reliably.

6. Test Plan

Testing the results of the work in this MR will be difficult, since the goal is not to introduce a new feature but make an old one more reliable. Testing will proceed in three stages:

• Simulator testing
• Functional testing on GBT
• Integration/Regression tests (by sponsor)

7.1 Simulator testing

The new infrastructure being installed by the modifications in this MR can be tested on the simulator for functionality. The simulator cannot obtain real data to compute LFC and LPC values. However, simulated values may be used to test the new Pyro client and server code, and the synchronization mechanism in Turdle.d. This involves making temporary changes to GFM to skip the computation of LPCs and LFCs and to pass on these simulated values. After the tests are completed, the changes to GFM are reversed.

7.2 Functional testing on GBT

This test ensures that the completed GFM and Turtle.d code functions as expected: LPCs and LFCs are computed and successfully transferred to the Antenna Manager. Turtle.d and Astrid run the Peak and Focus scans as before, with no changes in behavior. This test requires only that one each of the Peak and Focus procedures completes as expected to verify that old functionality remains intact.

7.3 Integration/Regression Tests

The main purpose of this MR was to increase the reliability of the transfers of LPCs and LFCs to the Antenna Manager. To see that this has actually been acomplished the system must be used repetitively, with enough iterations to ensure with some confidence that the reliability of the system has been improved. The SDD is looking at possible test dates in mid-November with test times long enough to perform repeated point and focus scans. If possible we will use scheduling blocks developed by Dana Balser and Frank Ghigo that perform all-sky pointing runs. Alternatively, we can simply create an observing script that contains an infinite while loop calling =AutoPeakFocus=.

In addition to addition to the "many iterations" test, it is important to test as far as possible that the new mechanism is robust against non-standard behavior. In addition, as well as correct behavior when the peak/focus has gone well, we need to ensure correct behavior when the peak has failed for some reason. While it is difficult to predict exceptional circumstances, we should as far as possible attempt to do so.

Possible test combinations might be as follows, checking in each case that the anticipated behavior correctly occurs:

• Start GFM online, perform a peak, go offline, perform a peak, go online again, perform a peak.
• Pause/Stop/Abort observations at various points within the peak.
• Run a peak. Reset the next scan number to cause the first scan of the next peak to match the first scan of the previous peak.
• Set the heuristics so that the peak will definitely fail. Check the results.
• Set the heuristics to exercise all combinations of user-prompting behavior.
• Any other pathological observer behavior anyone can think of.

Signatures

APPROVED: I acknowledge that my request is fully contained in this MR, and if the SDD delivers exactly what I specified, I will be happy.

ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.

Written	- RamonCreager - 23 Oct 2006
Checked	- AmyShelton - 24 Oct 2006
Approved by Sponsor	- FrankGhigo - 30 Oct 2006
Approved by CCC	- RichardPrestage - 02 Nov 2006
Accepted/Delivered by Sponsor	- FrankGhigo - 04 Dec 2006

Symbols:

• Use %X% if MR is not complete (will display )
• Use %Y% if MR iscomplete (will display )

CCC Discussion Area

-- RamonCreager - 12 Oct 2006