• Additional maintenance item noted by RichardPrestage (observed by ToneyMinter). On occasion the Antenna Manager will not write scan-related quantities (such as the position of the mid-point of the scan) to the antenna FITS header. This means, amongst other things, that GFM cannot process pointing data. This has been observed before, but to date only during PTCS testing of new versions of the antenna manager, and especially during large multi-segment scans. This needs to be incorporated into the above priority list.

• From John:
  1. Section 2.3.4 Add the ability to monitor servo command rate -- The hardware for this is in place to monitor the rate command voltages. The SMU should be able to do this with little effort. It could be done with no software changes by physically moving the current monitor points to voltage monitors.
  2. Antenna Servo Upgrades. If we anticipate needing to do this for 3mm or improved Ka or Q band, we should aim for installing a modernized (digital) version of the existing servo system during the track repair shutdown. It will take several weeks to get this done, and another lengthy shutdown would be a PR nightmare. We really want to upgrade the interlock system to improve our MTTR on the servo system.
  3. Allow system to run with E-Stop I agree with Richard that this is not a danger. We do not rely on M&C to protect the antenna or personnel from harm. It should be acceptable to have the system run normally with an e-stop pushed. What should not happen is that axes should not automatically reenable themselves when an e-stop condition goes away. That should take an act by the operator or other human.

1. 1. A possible further problem is poor servo performance found at zero or low acceleration but high velocity ("high" is anything faster than a few times the 15 arcseconds/sec celestial rate-- eg, 1.5 arcmin/second is too fast); however as Richard has suggested this might be due to decaying residuals from large-acceleration-induced tracking errors at the map edges, and not an independent problem.
   2. Priorities: For my purposes reducing scan start overheads is reasonably important. Investigating the performance while accelerating, for me, has about the priority reflected in the above proposal, however investigating transients induced by large accelerations (which persist into nominally zero-velocity parts of the scan) is more important than that. Alternately, if the problem is high velocity not transients from the acceleration, that is fairly important. The real point here is that using reasonably high and mostly constant velocities to map a small patch of sky is desirable. In any case I agree that scan start issues are the major ones now and impact most anything one does.

1. 1. A possible further problem is poor servo performance found at zero or low acceleration but high velocity ("high" is anything faster than a few times the nominal 15 arcseconds/sec celestial rate-- eg, 1.5 arcmin/second is too fast); however as Richard has suggested this might be due to decaying residuals from large-acceleration-induced tracking errors at the map edges, and not an independent problem.
   2. Priorities: Reducing scan start overheads is reasonably important. Investigating the performance while accelerating, for my purposes, has about the priority reflected in the above proposal, however investigating transients induced by large accelerations (which persist into nominally constant-velocity parts of the scan) is more important than that. Alternately, if the problem is high velocity not transients from the acceleration, that is fairly important. The real point here is that using reasonably high and mostly constant velocities to map a small patch of sky is desirable. In any case I agree that scan start issues are the major ones now and impact most anything one does.

• Proto-suggestions from Brian:
  1. A possible further problem is poor servo performance found at zero or low acceleration but high velocity ("high" is anything faster than a few times the 15 arcseconds/sec celestial rate-- eg, 1.5 arcmin/second is too fast); however as Richard has suggested this might be due to decaying residuals from large-acceleration-induced tracking errors at the map edges, and not an independent problem.
  2. Priorities: For my purposes reducing scan start overheads is reasonably important. Investigating the performance while accelerating, for me, has about the priority reflected in the above proposal, however investigating transients induced by large accelerations (which persist into nominally zero-velocity parts of the scan) is more important than that. Alternately, if the problem is high velocity not transients from the acceleration, that is fairly important. The real point here is that using reasonably high and mostly constant velocities to map a small patch of sky is desirable. In any case I agree that scan start issues are the major ones now and impact most anything one does.

• A note regarding VLBI phase-referencing. (from Frank): Referring to a memo on this topic by Wrobel, Walker and Benson (VLBA Scientific Memo #24), the recommended cycle time for 22 GHz observing is 60 seconds, and for 43 GHz 30 sec. The cycle time means the time to observe the calibrator, observe the source, and go back to the calibrator. And of course, the larger the on-source and on-cal time the better. We will probably satisfy these desires if we can do a one-degree move, including settling time, in 5 seconds. Of course if we can do a 2 degree move in 5 seconds, so much the better!

Draft 0.9 - still under construction.

Draft 1.0 will be made available for review

Draft 1.0 for internal GB staff review

7. Reader's Comments

please feel free to add comments/suggestions here!

• Find out what performance can actually be expected. (Joe notes that initial servo commissioning in August 2000, that elevation easily met specifications, but azimuth required several "tweaks" to meet specs. (i.e. the elevation servos have a higher 'design-factor', whereas the azimuth may be slightly under-powered.)

• Find out what performance can actually be expected.

5. Work Plans

5. Approaches to the route-finding algorithm

Our proposed solution is to use the new multi-segment scan, coupled with the ability to read a complete scan trajectory from a FITS file, to prototype different approaches to this problem. We can prototype different route-finding algorithms in a high-level language (e.g. python or matlab), outside the antenna manager, ensure these produce stable and valid trajectories, and download sample trajectories to test with the simulated, and then real antenna. This would allow us to test both the response of the servo, and potential effects in the feed-arm (e.g. ringing). Since we have extremely limited staff effort available to work on this, it would have to proceed at low priority on a "best efforts" basis.

6. Work Plans

The agreed high-level distribution of SDD effort for 2005 shows ~0.5 ftes for C2, C3 and C4 available for antenna trajectory related work. In advance of the completion of this analysis, some of the more straightforward tasks have already been allocated for C2. During C2 we should also review this document with a broard audience, and agree that the item list is complete, and the relative priorities broadly correct.

C3 2005

During C3 we should:

• Code and implement selected items from the "Other" category
• Perform data analysis on existing test data sets and provide recommendations.
• Follow up on following actions:
  • see whether we can provide Brian with the trajectory he needs (Joe w assistance from Richard).
  • Characterize current inter-scan latency (Joe with assistance from Richard).
  • Figure out what is going on with Q-band peaks (Richard)
  • Document expected servo performance (Joe).
  • Probe proposed changes to route-finding algorithms using multi-segment scan. Start off with antenna simulator, follow up with real antenna test time late March / Early April.

• Review existing documentation / knowledge to confirm what we should expected the response of the antenna should be to different demand trajectories (constant velocity, constant acceleration, etc).
• Formulate a framework for prototyping different algorithms via the multi-segment scan.
• Tackle the simpler items from the "other" list if it appears that effort is available.

C4 2005

Depends on outcome of C2/C3 activities

2.1.1 Scan-start trajectory problem.

2.1.1 Scan-start trajectory problem

2.1.2 Antenna servo performance while accelerating.

2.1.2 Antenna servo performance while accelerating

2.2.1 Failure to Find Trajectory.

2.2.1 Failure to Find Trajectory

2.2.2 Slow subscan startup.

2.2.2 Slow subscan startup

2.2.3 Intra-scan latency.

2.2.3 Intra-scan latency

2.3.2 Handling of PF Focus Positions

2.3.2 LPC tweaking during subscans

The antenna handles focus positions for Gregorian receivers differently than for PF receivers. Optimum focus values for G receivers are centered around zero while for PF they have a large positive value. This discrepancy should be resolved. (Originally reported in PO#985). This is the an active MR for C2 2005.

2.3.3 LPC tweaking during subscans

2.3.7 Handling of PF Focus Positions

The antenna handles focus positions for Gregorian receivers differently than for PF receivers. Optimum focus values for G receivers are centered around zero while for PF they have a large positive value. This discrepancy should be resolved. (Originally reported in PO#985). This is the an active MR for C2 2005.

2.3.8 Add the ability to monitor servo command rate.

2.3.4 Add the ability to monitor servo command rate

2.3.4 Failure to find "stopping" trajectory.

2.3.5 Failure to find "stopping" trajectory.

2.3.10 Need a "reset antenna parameters" config_tool script.

2.3.6 Need a "reset antenna parameters" config_tool script

2.3.9 Need to be able to start a scan even if it can't finish

2.3.7 Need to be able to start a scan even if it can't finish

2.3.3 Antenna Does Not Record "On-Source" Flag

2.3.8 Antenna Does Not Record "On-Source" Flag

2.3.9 Correct Asymmetry in Velocity Feed Forward Algorithm

2.3.5 Be Able to Run Antenna in Monitor Mode when Control Not Available

2.3.10 Be Able to Run Antenna in Monitor Mode when Control Not Available

2.3.6 Rate Mismatch in Servo Controller

2.3.11 Rate Mismatch in Servo Controller

There are checks in the CCU servo software to detect whether or not the axes are moving at the commanded rates. The controller hardware for elevation reduces the maximum rate, based on the number of motors faulted out. The rate mismatch check needs to be aware of the number of motors faulted and adjust the check accordingly. (Originally reported in PO#987). I think the change required is a software only change to the CCU. I would put this pretty low on the priority list.

There are checks in the CCU servo software to detect whether or not the axes are moving at the commanded rates. The controller hardware for elevation reduces the maximum rate, based on the number of motors faulted out. The rate mismatch check needs to be aware of the number of motors faulted and adjust the check accordingly. (Originally reported in PO#987). I think the change required is a software only change to the CCU.

2.3.11 Allow immediate on-source.

2.3.12 Allow immediate on-source.

2.3.12 Correct Asymmetry in Velocity Feed Forward Algorithm

Don't expect to do anything on these for a long term, but being aware they are coming down the pipe may affect current

We do not expect to do anything in the forseeable future, but being aware they are coming down the pipe may affect current

2.4.1 Completely replace current servo

2.4.1 Upgrade current servo/antenna control electronics

• PTCS would certainly like a "real-time" interface into the subreflector, similar to the dynamic corrections to the main drives, at some point in the mid-term (late 2005 to early 2006). Anything else?

• PTCS would certainly like a "real-time" interface into the subreflector, similar to the dynamic corrections to the main drives, at some point in the mid-term (late 2005 to early 2006). Various other modifications to the antenna manager have

Too many layers of trajectory generation, parameterization and regeneration between the user and the final demand to the servo may complicate the efficiency of the operations. Simplifying should be a consideration.

There are many layers of trajectory generation, parameterization and regeneration between the user and the final demand to the servo. Simplifying should be a consideration.

JOE: Describe the available data sets here, when they were taken, how configured, where to find them, & what we want to learn from them.

6. Test Plans

Signatures

APPROVED: I acknowledge that my request is fully contained in this MR, and if the SDD delivers exactly what I specified, I will be happy.

ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.

Status

Symbols:

• Use %X% if MR is not complete (will display )
• Use %Y% if MR iscomplete (will display )

CCC Discussion Area

-- NicoleRadziwill -- RichardPrestage -- JoeBrandt

-- NicoleRadziwill - 10 Feb 2005

JoeBrandt, RichardPrestage, NicoleRadziwill

Draft 0.9 - still under construction.

Draft 1.0 will be made available for review

current algorithms (which potentially can be addressed now), and fundamental limitations with the existing servo (which may need to be postponed to a later time).

current algorithms (which potentially can be addressed now), and more fundamental limitations with the performance of the antenna and/or servo (which are completely outside the scope of this project).

Our intent is to make draft 1.0 of this document available for wider review by scientific and technical staff. Once relative priorities of items is agreed, this document will lead to specific MRs to tackle individual items, after which these will be prioritized and scheduled through the standard mechanisms. A few of the other issues described below have already been scheduled for C2 2005.

2. Summary of Potential areas for improvement

2. Summary of Potential Areas of Improvement

The major performance problem at the moment is the so-called scan start trajectory problem; this impacts most scans that are attempted, and impacts data quality. This manifests itself in two ways: firstly, the antenna (as indicated by main-drive servo error) is not correctly on source for the first ~ 10 seconds of the scan. Secondly, it appears that the scan-start transition can sometimes (but not always) excites "ringing" in the feed-arm, which can take up to ~30 seconds to dampen out. The main scientific impact of this problem, again I think in priority order is:

The major performance problem at the moment is the so-called scan start trajectory problem; this impacts most scans that are attempted, and impacts data quality as well as observing efficiency. This manifests itself in two ways: firstly, the antenna (as indicated by main-drive servo error) is not correctly on source for the first ~ 10 seconds of the scan. Secondly, it appears that the scan-start transition can sometimes (but not always) excites "ringing" in the feed-arm, which can take up to ~30 seconds to dampen out. The main scientific impacts of this problem, again we believe in priority order are:

of around 30 seconds (Exact time estimate TBD; we need to document how long it really takes to slew one or two degrees on the sky).

of around 30 seconds (Exact time estimate TBD; we need to document how long it really takes to slew one or two degrees on the sky, including acceleration/deceleration, etc).

be used to measure efficiency.

be used to measure aperture efficiency.

start delay, which moves the scan-start transition. This can improve data quality at the direct expense of efficiency. This is not a particularly good solution for VLBI phase referencing, where it really is important to complete a cycle within ~30 seconds (confirm this with Frank) even at the expense of the duty cycle, or it becomes pointless.

"settletime" parameter, which moves the scan-start transition ahead of actual data acquisition. This can improve data quality at the direct expense of efficiency. This is not a particularly good solution for VLBI phase referencing, where it really is important to complete a cycle within ~30 seconds (confirm this with Frank) even at the expense of the duty cycle, or it becomes pointless.

I believe for Nodded spectral-line observations, we could cope with the overhead, with some marginal loss of data quality. Right now we're working around the problem for efficiency measurements (e.g. by averaging scans together); this seems ok at K-band, but somewhat problematic at Ka and Q-band.

For Nodded spectral-line observations, we could cope with the overhead, with some marginal loss of data quality. Right now we're working around the problem for efficiency measurements (e.g. by averaging scans together); this seems ok at K-band, but somewhat problematic at Ka and Q-band.

A second hypothesis that Joe had formed was that the actual commands were not being sent at the expected times. It may be that commands are somehow queuing up prior to a scan start, then being blasted out. There are many checks in the Antenna Manager to warn of this, but if it occurs somewhere in the OS layer it is not detectible. This is a primary motivation for investigating a linux-based Antenna Manager with an RTAI rt-net 'command launcher'. 3/1/05 Update: Joe can confirm that this aspect of timing has ruled out as a problem.

Yet another idea is to use 'reflective memory' (e.g. the same card we use on spectrometer) instead of ethernet. This has the elegance of reducing command latencies and would allow a full 50Hz command path, if necessary.

Currently there are no requirements for a full bandwidth command path, we just make note of the possibility should the requirement arise.

(An alternative hypothesis that Joe had formed was that the actual commands were not being sent at the expected times. The hypothesis was that commands were somehow queuing up prior to a scan start, then being blasted out. There are many checks in the Antenna Manager to warn of this, but if it occurs somewhere in the OS layer it is not detectible. Recent investigations appear to have ruled this out for now, but the potential would be a primary motivation for investigating a linux-based Antenna Manager with an RTAI rt-net 'command launcher'. An alternative idea would be to use 'reflective memory' (e.g. the same card we use on spectrometer) instead of ethernet. This has the elegance of reducing command latencies and would allow a full 50Hz command path, if necessary.Currently there are no requirements for a full bandwidth command path, we just make note of the possibility should the requirement arise.)

• Encourage use of settle_time parameter (needs to be couple with a "reset antenna parameters" config script.

• Encourage use of settle_time parameter (this would need to be coupled with a "reset antenna parameters" config script).

point of view as well; potential implications for data analysis software.

point of view as well; has potentially large implications for existing data analysis software.

This is not currently a major problem, but will become progressively important as we introduce more sophisticated

This is not currently a major problem, but will become progressively more important as we introduce more sophisticated

while data acquisition is underway (is this true?). Apart from the scan-start issue, the servo performance

while data acquisition is underway. Apart from the scan-start issue, the servo performance

complex scanning patterns, including so-called "daisy-petal" scans and others where the antenna is constantly accelerating. These employ supported, but not commonly used facilities in GO and Turtle.

complex scanning patterns, including so-called "daisy-petal" scans and others where the antenna is constantly accelerating. These employ supported, but not commonly used, facilities in GO and Turtle.

velocity is also poor. Bill states that this performance is significantly worse than what he expected from the underlying hardware, but I have no idea on what he bases this claim; we need to follow this up with high priority.

velocity is also poor. Bill states that this performance is significantly worse than what he expected from the underlying hardware.

I think we should avoid setting high-priority tasks on subjective statements like "...performance is significantly worse than what is expected from the underlying hardware...". Performance measures like tracking accuracy, response time, efficiency, stiction resistance etc., are not always synergistic. (e.g. One may have to give up on efficiency in favor of tracking accuracy.)

We do not currently know on what basis this statement is made; following up on what precisely is reasonably expected performance for the existing servo is a high priority task. Performance measures like tracking accuracy, response time, efficiency, stiction resistance etc., are not always synergistic. (e.g. One may have to give up on efficiency in favor of tracking accuracy.)

PTCS has "solved" this problem for now by providing shaped acceleration demands which result in good performance during the constant-velocity periods of the scan. This improved performance comes at the expense of

PTCS has "solved" this problem for now for their purposes by providing shaped acceleration demands which result in acceptable performance during the constant-velocity periods of the scan. This improved performance comes at the expense of

• There was some concern over command timing, but this has now been ruled out.
• Don't know if this is a real problem, since we don't know what performance to expect.

• We do not currently know if this is a "real" problem, since we don't know what performance to reasonably expect under changing acceleration.

• See what the SIMULINK modes thinks the antenna is capable of.

• See what the SIMULINK models thinks the antenna is capable of.

• New/upgrade to servo system hardware and/or servo control algorithm?

It may be that this is no solution to this problem, or that the solution requires extensive work to the servos, outside of the scope of this project.

This is probably the second most serious current issue, after the scan-start transition problem described above. This intermittent bug in the Antenna Manager impacts many mapping observations.

We believe this is probably the second most serious current issue, after the scan-start transition problem described above. This intermittent bug in the Antenna Manager impacts many mapping observations.

These issues can be partially addressed with a better host computer. In the near term virgo could be upgraded to an Ultra-5 (about 3x faster than virgo). A long term solution should be a linux machine (typical 3.2GHz P4 is 10-20x faster than virgo) running with an RTAI module to do the 'real-time command launching' (i.e getting the packets out on time). but we now think the packets are getting out on time, so this is no longer relevant? (The issue will be relavent when the antenna manager is ported over to linux. The current antenna manager host is a Solaris machine with special configuration to run the task scheduler at 1000Hz vs. 100Hz. When ported to linux, timing will again be an issue, but can addressed by the RTAI add-on.)

These issues can be partially addressed with a better host computer. In the near term virgo could be upgraded to an Ultra-5 (about 3x faster than virgo). A long term solution should be a linux machine (typical 3.2GHz P4 is 10-20x faster than virgo) running with an RTAI module to do the 'real-time command launching' (i.e getting the packets out on time). (The issue will be relavent when the antenna manager is ported over to linux. The current antenna manager host is a Solaris machine with special configuration to run the task scheduler at 1000Hz vs. 100Hz. When ported to linux, timing will again be an issue, but can addressed by the RTAI add-on.)

• Antenna doesn't know what the next subscan is going to be, so it simply comes to a stop. Once the next subscan is specified, then it starts moving again.

• Antenna doesn't know what the next subscan is going to be, so it simply returns to default behaviour (e.g. tracking

These are in my perceived order of cost/benefit trade-off, e.g. easiest, most useful items first. We should discuss this order.

These are in our believed order of cost/benefit trade-off, e.g. easiest, most useful items first. This order should certainly be reviewed.

2.3.7 Handling of PF Focus Positions

The antenna handles focus positions for Gregorian receivers differently than for PF receivers. Optimum focus values for G receivers are centered around zero while for PF they have a large positive value. This discrepancy should be resolved. (Originally reported in PO#985). This is the an active MR for C2 2005.

2.3.8 Add the ability to monitor servo command rate.

Extremely useful for diagnostic purposes. It may be possible to setup the servo monitor to record the rate command voltages.

2.3.4 Failure to find "stopping" trajectory.

• The solution is to re-write the route-finding algorithm.
• Add additional 'sane' processing when the failure occurs to minimize the overall impact.

2.3.10 Need a "reset antenna parameters" config_tool script.

2.3.9 Need to be able to start a scan even if it can't finish

2.3.4 Failure to find "stopping" trajectory.

• The solution is to re-write the route-finding algorithm.
• Add additional 'sane' processing when the failure occurs to minimize the overall impact.

2.3.7 Handling of PF Focus Positions

The antenna handles focus positions for Gregorian receivers differently than for PF receivers. Optimum focus values for G receivers are centered around zero while for PF they have a large positive value. This discrepancy should be resolved. (Originally reported in PO#985).

2.3.8 Add the ability to monitor servo command rate.

Handy for diagnostic purposes. It may be possible to setup the servo monitor to record the rate command voltages.

2.3.9 Need to be able to start a scan even if it can't finish

2.3.10 Need a "reset antenna parameters" config_tool script.

Modification Request #1 (C2 2005)

Problem Description, Spring 2005

niggling problems for some time. The

niggling problems for some time.

2.1.1 Scan-start trajectory problem.

2.1.1 Scan-start trajectory problem.

• VLBI phase-referencing: slewing two/from a nearby calibator to get a reference phase measurement. Typically, these

The major performance problem at the moment is the so-called scan start trajectory problem; this impacts most scans that are attempted, and impacts data quality. This manifests itself in two ways: firstly, the antenna (as indicated by main-drive servo error) is not correctly on source for the first ~ 10 seconds of the scan. Secondly, it appears that the scan-start transition can sometimes (but not always) excites "ringing" in the feed-arm, which can take up to ~30 seconds to dampen out. The main scientific impact of this problem, again I think in priority order is:

• VLBI phase-referencing: slewing to/from a nearby calibator to get a reference phase measurement. Typically, these

of around 30 seconds (TBD, document how long it really takes to slew one or two degrees on the sky).

of around 30 seconds (Exact time estimate TBD; we need to document how long it really takes to slew one or two degrees on the sky).

• NODDED spectral-line (or continuum) observations. Here the desire is to spend some time integrating with the source

• NODDED spectral-line (or continuum) observations: Here the desire is to spend some time integrating with the source

• Peak scans/Efficiency Measurements. Here we perform typically 30 second scans while rastering across the source. The

• Peak Scans/Efficiency Measurements: Here we perform typically 30 second scans while rastering across the source. The

I believe for Nodded spectral-line observations, we could cope with the overhead, with some marginal loss of data quality. Right now we're working around the problem for efficiency measurements (e.g. by averaging scans together); this seems ok at K-band, but somewhat problematic at Ka and Q-band.

I believe for Nodded spectral-line observations, we could cope with the overhead, with some marginal loss of data quality. Right now we're working around the problem for efficiency measurements (e.g. by averaging scans together); this seems ok at K-band, but somewhat problematic at Ka and Q-band.

Believed Cause: This is believed to be due to a weakness in the route-finding algorithm. The problem arises in that accelerations give rise to position error, and the algorithm places a period of excessive acceleration just prior to the start of a scan. In some cases however, trajectory preprocessor generates commands which are higher in magnitude and shorter in duration than one might expect. This is directly due to the algorithm's assumption of equal magnitude for begining (phase-I) and ending (phase-III) accelerations. If the algorithm was able to deal with different acceleration limits for slew start and end, this could be mitigated somewhat. This issue is discussed further in modification request 2C404.

Believed Cause: This is believed to be due to a weakness in the route-finding algorithm. The problem arises in that accelerations give rise to position error, and the algorithm places a period of excessive acceleration just prior to the start of a scan. In some cases however, the trajectory preprocessor generates commands which are higher in magnitude and shorter in duration than one might expect. This is directly due to the algorithm's assumption of equal magnitude for begining (phase-I) and ending (phase-III) accelerations. If the algorithm was able to deal with different acceleration limits for slew start and end, this could be mitigated somewhat. This issue is discussed further in modification request 2C404.

It should be noted that any acceleration will produce some degree of position error, this is just the nature of the current GBT servo design. The original servo specifications did not specify a margin of following accuracy, rather the emphasis was on good tracking accuracy at a constant rate.

It should be noted that any acceleration will produce some degree of position error. This is just the nature of the current GBT servo design. The original servo specifications did not specify a margin of following accuracy, rather the emphasis was on good tracking accuracy at a constant rate.

A second hypothesis that Joe has formed is that the actual commands are not being sent at the expected times. It may be that commands are somehow queuing up prior to a scan start, then being blasted out. There are many checks in the Antenna Manager to warn of this, but if it occurs somewhere in the OS layer it is not detectible. This is a primary motivation for investigating a linux-based Antenna Manager with an RTAI rt-net 'command launcher'.

A second hypothesis that Joe had formed was that the actual commands were not being sent at the expected times. It may be that commands are somehow queuing up prior to a scan start, then being blasted out. There are many checks in the Antenna Manager to warn of this, but if it occurs somewhere in the OS layer it is not detectible. This is a primary motivation for investigating a linux-based Antenna Manager with an RTAI rt-net 'command launcher'. 3/1/05 Update: Joe can confirm that this aspect of timing has ruled out as a problem.

Update: Joe can confirm that this aspect of timing has ruled out as a problem.

Currently there are no requirements for a full bandwidth command path, I just make note of the possibility should the requirement arise.

Currently there are no requirements for a full bandwidth command path, we just make note of the possibility should the requirement arise.

scanning strategies.

scanning strategies (such as those to be used with the Penn Array).

appears reasonable under these conditions (barring e.g. response to track discontinuities, which are considered a separate

appears reasonable under these conditions (barring, for example, response to track discontinuities, which are considered a separate

• there was some concern over command timing, but this has now been ruled out.

• There was some concern over command timing, but this has now been ruled out.

• find out what performance can actually be expected. (Joe notes that initial servo commissioning in August 2000, that elevation easily met specifications, but azimuth required several "tweaks" to meet specs. (i.e. the elevation servos have a higher 'design-factor', whereas the azimuth may be slightly under-powered.)
• Replace offending scan patterns with corresponding shaped-acceleration ones
• new/upgrade to servo system hardware and/or servo control algorithm?

• Find out what performance can actually be expected. (Joe notes that initial servo commissioning in August 2000, that elevation easily met specifications, but azimuth required several "tweaks" to meet specs. (i.e. the elevation servos have a higher 'design-factor', whereas the azimuth may be slightly under-powered.)
• See what the SIMULINK modes thinks the antenna is capable of.
• Replace offending scan patterns with corresponding shaped-acceleration ones.
• New/upgrade to servo system hardware and/or servo control algorithm?
• Be able to get commanded rate information out of servo, and see if instructions match what happens when an antenna motion is executed.

2.2.1 Failure to Find Trajectory.

2.2.1 Failure to Find Trajectory.

This is probably the second most serious current issue, after the scan-start transition problem described above. This intermittent bug in the Antenna Manager impacts many mapping observations.

elevation/position/rate combinations) this can occur multiple times. Can this be reproduced in the simulator?

elevation/position/rate combinations) this can occur multiple times.

Potential Solution:

• re-write route-finding algorithm
• use other methods to 'tweak' a previously legal trajectory into the final solution
• use multi-segment scans (only one starting trajectory to find).

Potential Solutions/Actions:

• See if this problem can be reproduced in the new full-telescope software simulator built in December 2004.
• Re-write route-finding algorithm
• Use other methods to 'tweak' a previously legal trajectory into the final solution
• Use multi-segment scans (only one starting trajectory to find).

Potential Solutions:

• speed up grail -> antenna downloads? (I've already modified turtle to parcel large parameters into 'chunks' of 2000 or less. This speeded up Grail considerably. This possibly could be optimized further.
• remove or modify antenna check on scan segments when parameter is passed down (e.g. do it in a separate thread, or not at all assuming turtle has already validated it.

Potential Solutions/Actions:

• Speed up grail -> antenna downloads? (I've already modified turtle to parcel large parameters into 'chunks' of 2000 or less. This speeded up Grail considerably. This possibly could be optimized further.)
• Remove or modify antenna check on scan segments when parameter is passed down (e.g. do it in a separate thread, or not at all assuming turtle has already validated it.) The issue here is one of expected response on a failure. For example if a ten-minute scan fails at nine minutes, should the antenna abort? If so, will the missing antenna positions for that sub-scan cause the whole sub-scan to be invalid? Otherwise I like the idea of run-time checking.

These issues can be addressed with a better host computer. In the near term virgo could be upgraded to an Ultra-5 (about 3x faster than virgo). A long term solution should be a linux machine (typical 3.2GHz P4 is 10-20x faster than virgo) running with an RTAI module to do the 'real-time command launching' (i.e getting the packets out on time). but we now think the packets are getting out on time, so this is no longer relevant? (The issue will be relavent when the antenna manager is ported over to linux. The current antenna manager host is a Solaris machine with special configuration to run the task scheduler at 1000Hz vs. 100Hz. When ported to linux, timing will again be an issue, but can addressed by the RTAI add-on.)

RMP - I have to think a faster machine is a stop-gap for an inefficient algorithm. We should consider the cost-benefit tradeoff of abandoning the scan check at scan start - why not just check that the starting point is legal, then check the rest of it later? The issue here is one of expected response on a failure. For example if a ten-minute scan fails at nine minutes, should the antenna abort? If so, will the missing antenna positions for that sub-scan cause the whole sub-scan to be invalid? Otherwise I like the idea of run-time checking.

These issues can be partially addressed with a better host computer. In the near term virgo could be upgraded to an Ultra-5 (about 3x faster than virgo). A long term solution should be a linux machine (typical 3.2GHz P4 is 10-20x faster than virgo) running with an RTAI module to do the 'real-time command launching' (i.e getting the packets out on time). but we now think the packets are getting out on time, so this is no longer relevant? (The issue will be relavent when the antenna manager is ported over to linux. The current antenna manager host is a Solaris machine with special configuration to run the task scheduler at 1000Hz vs. 100Hz. When ported to linux, timing will again be an issue, but can addressed by the RTAI add-on.)

Action: need to quantify what these latencies actually are, and how bad they are in normal observing (e.g. "peak" and "nodded" observations).

Action:

• Need to quantify what these latencies actually are, and how bad they are in normal observing (e.g. "peak" and "nodded" observations).

These are in my perceived order of cost/benefit trade-off, e.g. easiest, most useful items first. We should discuss this order.

These are in my perceived order of cost/benefit trade-off, e.g. easiest, most useful items first. We should discuss this order.

2.3.1 Antenna Handling of Beams C & 1

This issue was originally described at PO#977. The "C" and "1" beams were intended to be synonymous for single-beam receivers, but this is not the case in the current implementation, and becomes an annoyance during configuration. Currently "C" is valid for all receivers and beam "1" will be added for single beam receivers. The legal beam, according to Richard, should be both "1" and "C".

2.3.1 Antenna Handling of Beams C & 1 - ModificationRequest9C205

A simple (and elegant) fix for this is to include both definitions in the PFM database.

This issue was originally described at PO#977. The "C" and "1" beams were intended to be synonymous for single-beam receivers, but this is not the case in the current implementation, and becomes an annoyance during configuration. Currently "C" is valid for all receivers and beam "1" will be added for single beam receivers. A simple (and elegant) fix for this is to include both definitions in the PFM database.

• record the value of the On Source flag n the antenna position binary table, on a per-row basis.

• record the value of the On Source flag in the antenna position (ANTPOS) binary table, on a per-row basis.

• PTCS would certainly like a "real-time" interface into the subreflector, similar to the dynamic corrections to the main drives, at some point in the mid-term (late 2005 to early 2006). What else?

• PTCS would certainly like a "real-time" interface into the subreflector, similar to the dynamic corrections to the main drives, at some point in the mid-term (late 2005 to early 2006). Anything else?

C4 2004 - MR2C404; Release Notes

C4 2004 - MR2C404; Release Notes

C3 2003 - MRs available offline

C3 2003 - MRs available offline

4.1.1 Data Set 1

4.1.2 Data Set 2

C2 2005

Joe can confirm that this aspect of timing has ruled out as a problem. Yet another idea is to use 'reflective memory' (e.g. the same card we use on spectrometer) instead of ethernet. This has the elegance of reducing command latencies and would allow a full 50Hz command path.

Update: Joe can confirm that this aspect of timing has ruled out as a problem. Yet another idea is to use 'reflective memory' (e.g. the same card we use on spectrometer) instead of ethernet. This has the elegance of reducing command latencies and would allow a full 50Hz command path, if necessary.

Joe, you would need to fill in the details.

Currently there are no requirements for a full bandwidth command path, I just make note of the possibility should the requirement arise.

This is not currently major problem, but will become progressively important as we introduce more sophisticated

This is not currently a major problem, but will become progressively important as we introduce more sophisticated

complex scanning patterns, including so-called "daisy-petal" scans and others where the antenna is constantly accelerating. These employ supported, but not commonly used facilities in GO and Turtle.

complex scanning patterns, including so-called "daisy-petal" scans and others where the antenna is constantly accelerating. These employ supported, but not commonly used facilities in GO and Turtle.

During periods of acceleration, the antenna servo error can be large (~20 arc-seconds, significant fractions of a beam), causing this observing mode to be much less useful in practice than had been hoped. A more serious problem for Bill Cotton and Brian Mason is that, after periods of large acceleration, the performance in a period of constant (or nearly constant) velocity is also poor. Bill states that this performance is significantly worse than what he expected from the underlying hardware, but I have no idea on what he bases this claim; we need to follow this up with high priority.

During periods of acceleration, the antenna servo error can be large (~20 seconds, significant fractions of a beam), causing this observing mode to be much less useful in practice than had been hoped. A more serious problem for Bill Cotton and Brian Mason is that, after periods of large acceleration, the performance in a period of constant (or nearly constant) velocity is also poor. Bill states that this performance is significantly worse than expected from the underlying hardware, but I have no idea on what he bases this claim; we need to follow this up with high priority.

I think we should avoid setting high-priority tasks on subjective statements like "...performance is significantly worse than what is expected from the underlying hardware...". Performance measures like tracking accuracy, response time, efficiency, stiction resistance etc., are not always synergistic. (e.g. One may have to give up on efficiency in favor of tracking accuracy.)

• find out what performance can actually be expected. (Joe notes that elevation was "good", but azimuth needed to be "tweaked".

• find out what performance can actually be expected. (Joe notes that initial servo commissioning in August 2000, that elevation easily met specifications, but azimuth required several "tweaks" to meet specs. (i.e. the elevation servos have a higher 'design-factor', whereas the azimuth may be slightly under-powered.)

• new servo????

• new/upgrade to servo system hardware and/or servo control algorithm?

elevation/position/rate combinations) this can occur multiple times. Can this be reproduced in the simulator

elevation/position/rate combinations) this can occur multiple times. Can this be reproduced in the simulator?

• Failure in trajectory algorithm

• Failure in antenna manager to converge on a single trajectory. The antenna manager's route planning module expects that minor changes in trajectory inputs into the preprocessor algorithm should result in minor changes in the generated trajectory. However, in some cases the generated trajectory is vastly different, which leads to the failure.

• use other methods to 'tweak' a previously legal trajectory into the final solution

It can take the antenna a long time (many seconds) to digest a command to start moving. This is especially ovbious for a long duration scan, or a complex scan with many segments.

It can take the antenna a long time (many seconds) to digest a command to start moving. This is especially obvious for a long duration scan, or a complex scan with many segments.

• long time for antenna manager to check scan trajectory

• long time for antenna manager to check scan trajectory (due to rigorous checking)

• speed up grail -> antenna downloads?

• speed up grail -> antenna downloads? (I've already modified turtle to parcel large parameters into 'chunks' of 2000 or less. This speeded up Grail considerably. This possibly could be optimized further.

• only echo back the first few records to user interfaces.

• Limit scan segment parameter feedback. Only echo back the first few records (of thousands) to user interfaces, since few if any displays actually use the information.

These issues can be addressed with a better host computer. In the near term virgo could be upgraded to an Ultra-5 (about 3x faster than virgo). A long term solution should be a linux machine (typical 3.2GHz P4 is 10-20x faster than virgo) running with an RTAI module to do the 'real-time command launching' (i.e getting the packets out on time). but we now think the packets are getting out on time, so this is no longer relevant?

These issues can be addressed with a better host computer. In the near term virgo could be upgraded to an Ultra-5 (about 3x faster than virgo). A long term solution should be a linux machine (typical 3.2GHz P4 is 10-20x faster than virgo) running with an RTAI module to do the 'real-time command launching' (i.e getting the packets out on time). but we now think the packets are getting out on time, so this is no longer relevant? (The issue will be relavent when the antenna manager is ported over to linux. The current antenna manager host is a Solaris machine with special configuration to run the task scheduler at 1000Hz vs. 100Hz. When ported to linux, timing will again be an issue, but can addressed by the RTAI add-on.)

RMP - I have to think a faster machine is a stop-gap for an inefficient algorithm. We should consider the cost-benefit tradeoff of abandoning the scan check at scan start - why not just check that the starting point is legal, then check the rest of it later?

RMP - I have to think a faster machine is a stop-gap for an inefficient algorithm. We should consider the cost-benefit tradeoff of abandoning the scan check at scan start - why not just check that the starting point is legal, then check the rest of it later? The issue here is one of expected response on a failure. For example if a ten-minute scan fails at nine minutes, should the antenna abort? If so, will the missing antenna positions for that sub-scan cause the whole sub-scan to be invalid? Otherwise I like the idea of run-time checking.

• Antenna doesn't know what the next subscan is going to be, so it simply comes to a stop. Then it starts moving again.

• Antenna doesn't know what the next subscan is going to be, so it simply comes to a stop. Once the next subscan is specified, then it starts moving again.

content.

content. (A lightweight 'interlock' might be helpful to avoid unintentional cross-correcting.)

The solution is to re-write the route-finding algorithm.

• The solution is to re-write the route-finding algorithm.
• Add additional 'sane' processing when the failure occurs to minimize the overall impact.

If we were able to run the antenna in monitoring-mode only when antenna control is not available, we could recover a large proportion of maintenance time by being able to concurrently support productive astronomy (e.g. to record the position of the antenna during drift scans when the antenna is disabled). This is not currently possible if the E-stop is pressed. It would be possible to over-ride the check on the E-stops when the antenna drives are disabled, but there is some concern this could lead to unsafe operation (I guess I (RMP) still don't see this, as long as the check is only

If we were able to run the antenna in monitoring-mode only when antenna control is not available, we could recover a large proportion of maintenance time by being able to concurrently support productive astronomy (e.g. to record the position of the antenna during drift scans when the antenna is disabled). This is not currently possible if the E-stop is pressed. It would be possible to over-ride the check on the E-stops when the antenna drives are disabled, but there is some concern this could lead to unsafe operation (I guess I (RMP) still don't see this, as long as the check is only

An alternative would be to allow the generation of FITS files when the antenna is in monitor-only mode. Joe needs to expand on possibilities here.

An alternative would be to allow the generation of FITS files when the antenna is in monitor-only mode, or allow the overriding of ESTOPs only when the system is not under antenna manager control. This would prevent the inadvertent enabling of axes when an ESTOP is cleared.

There are checks in the CCU servo software to detect whether or not the axes are moving at the commanded rates. The controller hardware for elevation reduces the maximum rate, based on the number of motors faulted out. The rate mismatch check needs to be aware of the number of motors faulted and adjust the check accordingly. (Originally reported in PO#987).

There are checks in the CCU servo software to detect whether or not the axes are moving at the commanded rates. The controller hardware for elevation reduces the maximum rate, based on the number of motors faulted out. The rate mismatch check needs to be aware of the number of motors faulted and adjust the check accordingly. (Originally reported in PO#987). I think the change required is a software only change to the CCU. I would put this pretty low on the priority list.

Handy for diagnostic purposes

Handy for diagnostic purposes. It may be possible to setup the servo monitor to record the rate command voltages.

stop (or continue without the antenna) if the antenna hits a limit.

stop (or continue without the antenna) if the antenna hits a limit. (Note comments above about expected behaviors in these cases.)

2.3.12 Correct Asymmetry in Velocity Feed Forward Algorithm

Believed Cause: This is believed to be due to a failure in the route-finding algorithm. The problem arises in that accelerations give rise to position error, and the algorithm places a period of acceleration just prior to the start of a scan. If the algorithm was able to deal with different acceleration limits for slew start and end, this could be mitigated somewhat.

Believed Cause: This is believed to be due to a weakness in the route-finding algorithm. The problem arises in that accelerations give rise to position error, and the algorithm places a period of excessive acceleration just prior to the start of a scan. In some cases however, trajectory preprocessor generates commands which are higher in magnitude and shorter in duration than one might expect. This is directly due to the algorithm's assumption of equal magnitude for begining (phase-I) and ending (phase-III) accelerations. If the algorithm was able to deal with different acceleration limits for slew start and end, this could be mitigated somewhat. This issue is discussed further in modification request 2C404.

It should be noted that any acceleration will produce some degree of position error, this is just the nature of the current GBT servo design. The original servo specifications did not specify a margin of following accuracy, rather the emphasis was on good tracking accuracy at a constant rate.

Can Joe confirm that timing is ruled out as a problem

Joe can confirm that this aspect of timing has ruled out as a problem.

Antenna Performance/Data Quality

Scan-start trajectory problem.

2.1 Antenna Performance/Data Quality

2.1.1 Scan-start trajectory problem.

Antenna servo performance while accelerating.

2.1.2 Antenna servo performance while accelerating.

Efficiency

2.2 Efficiency

Failure to Find Trajectory.

2.2.1 Failure to Find Trajectory.

Slow subscan startup.

2.2.2 Slow subscan startup.

Intra-scan latency.

2.2.3 Intra-scan latency.

Other Potential Upgrades

2.3 Other Potential Upgrades

Antenna Handling of Beams C & 1

2.3.1 Antenna Handling of Beams C & 1

LPC tweaking during subscans

2.3.2 LPC tweaking during subscans

Antenna Does Not Record "On-Source" Flag

2.3.3 Antenna Does Not Record "On-Source" Flag

Failure to find "stopping" trajectory.

2.3.4 Failure to find "stopping" trajectory.

Be Able to Run Antenna in Monitor Mode when Control Not Available

2.3.5 Be Able to Run Antenna in Monitor Mode when Control Not Available

There is a very genuine desire to be able to run the antenna in monitoring-mode only when antenna control is not available (e.g. to record the position of the antenna during drift scans when the antenna is disabled). This is not currently possible if the E-stop is pressed. It would be possible to over-ride the check on the E-stops when the antenna drives are disabled, but there

If we were able to run the antenna in monitoring-mode only when antenna control is not available, we could recover a large proportion of maintenance time by being able to concurrently support productive astronomy (e.g. to record the position of the antenna during drift scans when the antenna is disabled). This is not currently possible if the E-stop is pressed. It would be possible to over-ride the check on the E-stops when the antenna drives are disabled, but there

Rate Mismatch in Servo Controller

2.3.6 Rate Mismatch in Servo Controller

Handling of PF Focus Positions

2.3.7 Handling of PF Focus Positions

Add the ability to monitor servo command rate.

2.3.8 Add the ability to monitor servo command rate.

Need to be able to start a scan even if it can't finish

2.3.9 Need to be able to start a scan even if it can't finish

Need a "reset antenna parameters" config_tool script.

2.3.10 Need a "reset antenna parameters" config_tool script.

Allow immediate on-source.

2.3.11 Allow immediate on-source.

Long Term Upgrades

2.4 Long Term Upgrades

Completely replace current servo

Long term PTCS upgrades

2.4.1 Completely replace current servo

2.4.2 Long term PTCS upgrades

Reduce multiple layers

2.4.3 Reduce multiple layers

Command Timing Data

4.1 Command Timing Data

4.1.1 Data Set 1

Data Set 1:

Data Set 2:

4.1.2 Data Set 2

The major performance problem at the moment is the so-called scan start trajectory problem. This manifests

The major performance problem at the moment is the so-called scan start trajectory problem. This manifests

Believed Cause: This is believed to be due to a failure in the route-finding algorithm. Joe, you would need to

Believed Cause: This is believed to be due to a failure in the route-finding algorithm. The problem arises in that accelerations give rise to position error, and the algorithm places a period of acceleration just prior to the start of a scan. If the algorithm was able to deal with different acceleration limits for slew start and end, this could be mitigated somewhat.

A second hypothesis that Joe has formed is that the actual commands are not being sent at the expected times. It may be that commands are somehow queuing up prior to a scan start, then being blasted out. There are many checks in the Antenna Manager to warn of this, but if it occurs somewhere in the OS layer it is not detectible. This is a primary motivation for investigating a linux-based Antenna Manager with an RTAI rt-net 'command launcher'.

Can Joe confirm that timing is ruled out as a problem Yet another idea is to use 'reflective memory' (e.g. the same card we use on spectrometer) instead of ethernet. This has the elegance of reducing command latencies and would allow a full 50Hz command path.

Joe, you would need to

Potential Solutions:

• Encourage use of settle_time parameter (needs to be couple with a "reset antenna parameters" config script.
• Re-write the route-finding algorithm. (could be prototyped using multi-segment scan).
• Perform these observation using a multi-segement scan (only one scan start). Attractive from an overhead point of view as well; potential implications for data analysis software.

Actions: what is the expected performance of the antenna; can I create scan patterns that will do what Bill/Brian need.

Believed Causes

• there was some concern over command timing, but this has now been ruled out.
• Don't know if this is a real problem, since we don't know what performance to expect.

Potential Solutions/Actions

• find out what performance can actually be expected. (Joe notes that elevation was "good", but azimuth needed to be "tweaked".
• Replace offending scan patterns with corresponding shaped-acceleration ones
• new servo????

There are three main issues:

There are four main issues:

Believed causes:

• Failure in trajectory algorithm

Potential Solution:

• re-write route-finding algorithm
• use multi-segment scans (only one starting trajectory to find).

Believed Causes:

• long time for grail to send many segments down to antenna manager
• long time for antenna manager to check scan trajectory
• long time for antenna manager to echo all parameters back to all user inerfaces

Potential Solutions:

• speed up grail -> antenna downloads?
• remove or modify antenna check on scan segments when parameter is passed down (e.g. do it in a separate thread, or not at all assuming turtle has already validated it.
• only echo back the first few records to user interfaces.

These issues can be addressed with a better host computer. In the near term virgo could be upgraded to an Ultra-5 (about 3x faster than virgo). A long term solution should be a linux machine (typical 3.2GHz P4 is 10-20x faster than virgo) running with an RTAI module to do the 'real-time command launching' (i.e getting the packets out on time). but we now think the packets are getting out on time, so this is no longer relevant?

RMP - I have to think a faster machine is a stop-gap for an inefficient algorithm. We should consider the cost-benefit tradeoff of abandoning the scan check at scan start - why not just check that the starting point is legal, then check the rest of it later?

Believed Cause:

• Antenna doesn't know what the next subscan is going to be, so it simply comes to a stop. Then it starts moving again.

Potential Solution:

• This issue can be addressed by a solution proposed at ModificationRequest12C404. This re-enforces the need for faster hardware, because multiple sub-scans would be checked at the start of the scan, and so the hardware upgrade is recommended prior to software changes.

Action: need to quantify what these latencies actually are, and how bad they are in normal observing (e.g. "peak" and "nodded" observations).

These are in my perceived order of cost/benefir trade-off, e.g. easiest, most useful items first. We should discuss this order.

Antenna Handling of Beams C & 1

This issue was originally described at PO#977. The "C" and "1" beams were intended to be synonymous for single-beam receivers, but this is not the case in the current implementation, and becomes an annoyance during configuration. Currently "C" is valid for all receivers and beam "1" will be added for single beam receivers. The legal beam, according to Richard, should be