Currently the GBT antenna control system uses a jerk-limiting algorithm described in GBT Memo 203 to slew between scans. Recently Fred Schwab has described an improved algorithm which attempts to further minimize the slew trajectory. The newer algorithm is expected to require significant computing resources, exceeding the capabilities of the current system. This MR describes the effort to rehost the GBT antenna controller/antenna manager onto a more capable platform.
Currently the antenna controller (a.k.a the antenna manager) is hosted on a 167MHz sparc Ultra-1 system running Solaris. It is estimated the improved algorithm will be more computationally intensive. Therefore a key prerequisite is to rehost the antenna manager on a more capable linux-based platform.
- The GBT antenna controller is one of the older managers, and therefore uses some of the older API's for timing and thread management. As part of this initiative, calls to the old API's have been replaced with calls to the McTime and Threads libraries.
- The technique used to offset a periodic 1Hz task has been changed by adding an 'wakeup-offset' or 'phase-delay' to the McCondition timing class.
- The calculation of the equation of the equinoxes has been changed to use UTC midnight instead of the current time on system startup. (This emulates the prior operation of the system, since this calculation is updated just after UTC midnight.)
The antenna manager is targeted for deployment on a Intel Pentium-4 type system running Redhat linux. In order to maintain command timing, the program will run in the SCHED_FIFO scheduler class with real-time priority. The interface into the servo system is via 10base2 coaxial type 10Mbit Ethernet. I have two 3Comm PCI 3C900B cards in hand for this purpose.
The antenna manager is required to be synchronized with the servo system clocks in order to perform correctly. The current Solaris based system uses an 1-pulse-per-second input into the serial port, in conjunction with the ntp 1pps clock driver. On linux, a similar technique can be used via the shm-rtc.c utility. The Computing division will begin monitoring the time sychronization parameters on the new antenna host.
A 5 day run of monitoring the new system (algol) time offsets in milliseconds is shown below:
The red line above indicates the system time as measured against the 1PPS signal. The green and blue traces are system time offset with respect to the time servers sadira and arcturus. The point here is to show that the system time is stable to the sub-millisecond mark. The approximately two millisecond excursions can be addressed by changing some of the shm clock parameters.
The antenna manager is a real-time system. Commands generated by the antenna manager must be 'launched' at the appropriate time, so that the GBT CCU and SCU servo systems receive the command in time to act upon it. The command window is on the order of 20 milliseconds. As a side note, this window in not symmetrical, that is, the effect of commands exceeding the window in one direction are not the same as exceeding the window limit in the other direction. This is due to a problem in the servo code which has not been addressed. Fortunately, the conditions under which the timing is violated in one direction are very infrequent. This problem is detailed here.
The existing Solaris system is able to meet the command window deadline[1] by setting the task scheduling interval to 1000Hz, and placing the process at very high priority. Attempts to do this on Linux 2.4 kernels were less than impressive, largely due to a slower scheduling interval. The recent Redhat Enterprise 4 OS now has the newer 2.6 version kernel, with 1000Hz scheduling. Initial tests show that nearly identical timing performance can be achieved using the fast-scheduler and high priority approach.
In order to access the elevated scheduling priority, the antenna manager process must be "setuid root". To make this easier to configure, the Linux version will have the Starlink library statically linked.
[1] - I should note that there are exceptions to this statement. Perhaps the phase "no unexpected dropouts" is more precise. Since this is a soft-real-time system, deadlines are met most of the time.
3.5 FITS File Modifications
As requested by the CCC, a minor modification will be made to the antenna FITS file format. A new primary header keyword "WEATHSTA" will be added to indicate the source of weather information. The values will be one of "Weather_Station_1", "Weather_Station_2", or "User_Input".
In order to denote this change, the FITSVER keyword value will be changed to 2.10.
What has to get done to integrate this completely into the system. This checklist must be completed before Cycle Integration Testing begins.
- Communication with Computing group needed? Yes.
- This deployment will require the installation of a new rack mounted computer in the servo room, near the existing antenna control computer.
- The computer needs a 100Mbit port on the alidade room switch.
- Both old and new computers will be directly connected to the 10Mbit servo LAN.
- The computer division should begin monitoring the ntp status on algol, as it is done on virgo.
- What documentation needs to be updated?
- List of computers in use on the GBT.
- Coordinate with Ron to add algol tab to cleo-taskmaster application. (May be automatic.)
- Training Needed? Is this being released to staff astronomers or everyone right now? No specific training required. Full release.
- Notification Needed?
The following tests shall be conducted to verify proper operation:
- Using the Linux host to control the servo, record the network traffic on the servo LAN using virgo. Analyse the packet traffic for proper command timing (specifically the servo position command timing).
- Setup both the real antenna (with a Linux host) and the antenna simulator (with a sparc host) tracking a source in J2000 coordinates. (Disable dynamic corrections, or set them manually to the same values.) With this setup running, record 30 seconds of the azElCommands sampler stream. (Also record the time offset between the two systems.) Verify the az/el command values are within TBD of each other.
- Conduct pointing/focusing scans first with the Linux host, and compare results running with the sparc host.
- Perform out-of-focus beam maps and daisy pedal scans with sparc and linux hosts, and compare the results.
- Plot the residuals of each axis (i.e. MNT_AZ[sparc] vs. MNT_AZ[linux]) for az/el and RA/DEC columns. (Results below)
| Verification Matrix |
|---|
| Test Criteria | Result |
|---|
| Packet command timing periodic to a few msec |  |
| Az/El command stream values identical in RA/DEC track | |
| RA/DEC indicated values identical in AZ/EL track | |
| Pointing/Focusing scan results identical | |
There are two major numerical processing pipelines in the antenna: one transforms sky positions to servo commands, the other takes indicated servo position and transforms them back into sky coordinates. These tests were run multiple times, but the data shown below are from two scans. The first is a 25,000 second long RA/DEC track, which started around 123,65 (az/el) transited at an elevation of 74.5 degrees, and ended at 284,19.5 probing a range of elevations.
Using two antenna simulators, one running linux the other a sparc-solaris system. Scans were run with a common specified start time, tracking an RA/DEC position. By comparing the observed commanded AZ/EL positions (i.e the OBSC_AZ, OBSC_EL columns) performance of the sky to ground transformations can be compared with the existing system. Results of this run is shown below.
Below are plots of the delta between a sparc-based simulator and the Linux-based simulator, during a simultaneous observation:
| OBSC_AZ Commanded Position Delta During J2000 RA/DEC Track |
|---|
 |
| File: 2006_05_15_23:19:00.fits |
| OBSC_EL Commanded Position Delta During J2000 RA/DEC Track |
|---|
 |
| File: 2006_05_15_23:19:00.fits |
It should be noted that these graphs are a bit misleading! Direct inspection of these files with fv shows that the two columns are exactly the same. In either case, the test shows that even during an extended scan (25,000 seconds in length) commands generated are for our purposes, the same.
A second test running two simulators tracking a constant AZ/EL position was used to verify the indicated to sky position conversion pipeline. In this test, the resulting RA/DEC positions of a constant AZ/EL position were compared. Difference plots are shown below.
Note the next two plots were taken with the two systems tracking a constant az/el position.
| RAJ2000 Indicated Position Delta During AZ/EL TRACK |
|---|
 |
| File: 2006_05_16_14:03:00.fits |
| DECJ2000 Indicated Position Position Delta During AZ/EL TRACK |
|---|
 |
| File: 2006_05_16_14:03:00.fits |
The plot below shows the difference in refraction during an extended RA/DEC scan. The plot of the difference between the mount elevation (MNT_EL) is shown for informational purposes only, and is not necessarily expected to be identical.
| Refraction Delta During RA/DEC TRACK |
|---|
 |
| File: 2006_05_15_23:19:00.fits |
This plot appears a bit "noisy" because the refraction column data is based upon the indicated elevation (i.e. MNT_EL). Note the plot below shows the refraction noise correlates well with the indicated elevation "noise". The elevation rate went to zero as at the transit around the 50,000th sample. The elevation rate was highest near the end of the scan.
| Indicated Elevation Delta During RA/DEC TRACK |
|---|
 |
| File: 2006_05_15_23:19:00.fits |
This is confirmed implictly, for if these were not correct, the data would not be complete.
The following are plots of the difference in indicated position in mm and tilt difference in degrees of the two simulators during an RA/DEC track. To put the plots in perspective, 1 encoder bit corresponds to 0.012mm.
| Subreflector X Position Delta During RA/DEC TRACK |
|---|
 |
| File: 2006_05_15_23:19:00.fits |
| Subreflector Y Position Delta During RA/DEC TRACK |
|---|
 |
| File: 2006_05_15_23:19:00.fits |
| Subreflector Z Position Delta During RA/DEC TRACK |
|---|
 |
| File: 2006_05_15_23:19:00.fits |
| Subreflector X Tilt Delta During RA/DEC TRACK |
|---|
 |
| File: 2006_05_15_23:19:00.fits |
| Subreflector Z Tilt Delta During RA/DEC TRACK |
|---|
 |
| File: 2006_05_15_23:19:00.fits |
I also ran a focus scan, and logged data with the subreflector command samplers. The commands in servo coordinates are exactly identical.
FITS Header Comparisons
I've examined the antenna FITS headers under several circumstances, and they are exactly the same, with one notable exception: the LSTSTART keyword differs in the last two digits (i.e 1.213649922842640E+03 vs. 1.213649922842655E+03) or 15 femto-seconds. I've tracked this down to the SLALIB routine which computes the equation of the equinoxes. The equation of the equinoxes calculation (i.e. slaeqeqx()) returns a result which is very slightly different on Sparc vs. Linux. I must assume this is due to the architecture differences.
A number of live tests were conducted during integration tests, including several diasy-pedal scans, and pointings. Dana's log file and results are included below:
Proj: TINT_01
Use L-band to test AutoPeak's for Linux and Sparc versions
Scan Source Proc Version Comments
-------- --------- ------- ------- ---------------------------------------
29-32 2129+8453 Peak Linux LPCs (0.150, 0.029)
33-36 " " " LPCs (0.313, -0.088)
37-40 " " " LPCs (0.326, -0.038)
41-44 " " Sparc LPCs (0.265, -0.040)
45-48 " " " LPCs (0.192, -0.062)
49-52 " " " LPCs (0.187, xxxxxx)
No corrections detected on last El scans but otherwise okay.
Check daisy petal scans for Linux and Sparc versions
53 " Daisy Sparc LPCs not zero
54 " " Linux LPCs are all zero
55 " " "
56 " " Sparc
In addition, I have examined and compared computational differences in unit-tests at different levels of the system. Initially a systemmatic error was noted at a very small level. This was traced to a bug in the system, which has to do with how the equation of the equinoxes was computed on system startup. Typically this is calculated once a day, at UTC midnight. However on system startup the value was calculated based on the current time. I've changed this to always use UTC midnight.
APPROVED: I acknowledge that my request is fully contained in this MR, and if the SDD delivers exactly what I specified, I will be happy.
ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.
Symbols:
- Use
%X% if MR is not complete (will display 
)
- Use
%Y% if MR iscomplete (will display )
CCC Discussion Area
As well as confirming that the correct demands are sent to the antenna servo at the correct time, I think it is extremely important that we check that the correct "upstream" co-ordinate conversions are performed, since this determines where the observers "think" the antenna was pointed. Also, since mechanisms to generate the nHz loops have changed, we need to check very carefully that the synchronization / indexing is all still correct. If I understood correctly from conversations with Dana it is possible to "reset" the antenna simulator (and hopefully manager) clocks so that it is possible to run the "same" scan multiple times. I'd therefore like to suggest that the following tests be performed.
- Pick one or two reasonably demanding test scans. A "peak" and "focus" would probably actually do it. Choose a reasonable J2000 target so that the antenna is moving reasonably fast in both axes.
- Use a start time rather than asap, so that the scans start on the same second.
- Run two or three peaks and focusses with the current antenna manager, with the identical start time, and resetting the clock each time, so that each scan should be nominally identical.
- Compare the resultant FITS files. They should be "identical". The headers really should be identical; the data may be different due to round-off error, etc. So, the comparison may be a bit laborious. But each column (MNT_AZ, RA2000, OBSC_AZ, etc) should be identical across all files to some level of precision, hopefully << 1 arcsec.
- Having got a benchmark from running the old version of the antenna manager a few times, run the new version of the antenna manager a few times. Compare the antenna FITS files between old and new antenna manager versions. Again, they should be identical.
- Test comparisons should include all six subreflector axes for the focus scan (and one peak scan).
- Any discrepancies at greater than the ~0.3" (encoder resolution) level need to be explained and resolved.
I'm sorry I have only just got around to proposing this, and I realise it is a very large amount of work. But the antenna underpins everything, so I think it is important.
Richard,
The comments above about "resetting clocks" is not available. The systems all run synchronized with the site IRIG and 1PPS signals, so testing and comparisons must be done simultaneously. This is the approach I've taken in using the two simulators. In most all cases the FITS files are identical, with a couple exceptions. One I've noted, and reproduced outside of the antenna code is the calculation of the equation of the equinoxes, using a SLALIB routine, I get results which differ in the last couple digits. This causes the LSTSTART keyword to differ slightly (~15 femto-seconds).
The question could be asked "To what level should we expect identity?" With 64 bit floating point, you get about 15-16 significant digits. With 32 bit about 8-9 significant digits. Most of the differences (excluding the servo related ones) are on this level. The servo simulator is numerically noisy for a number of reasons. Internally, it is taking az/el, asynchronously closing a servo loop, and generating positions at 10 cycles/sidereal second. This in turn is resampled to 50 samples/solar second. Secondary optics are computed and sent over the network as 32 bit-floats, further complicating matters. So I would not expect two simulators to compute the same axis position, even with identical inputs. That is why I emphasize the comparison of commands.
I hope the added material addresses your concerns noted above.
JoeBrandt
-- JoeBrandt - 06 Apr 2006
Revision r1.19 - 17 May 2006 - 19:43 GMT - RichardPrestage
Parents: PlanOfRecordC32006
|
Content copyright © 1999-2007 by the contributing authors.
All material on this collaboration platform is the property of the contributing authors.
|
| |
t1.png 
