| No | Date | Time | Project Code | Observer | Support Scientist | Instrument | Bug, Etc. | Priority | Time Lost | Status | Description | Response | Scripts |
|---|
| 1 | 2005-10-01 | 08:00 | AGBT04C_031 | Kondratdo | Ghigo | Antenna/Antenna Manager | Problem | Important | 04:00 | Assigned | Saturday morning after project 4C031 (Kondratdo) started, we had a problem which I believe was due to the subreflector sometimes being in the wrong focus position. The symptom was that a pointing observation would show a very weak or missing peak. If we did a focus observation just before doing a peak, then everything looked fine. But after a re-configuration if we did a peak, when we would not see the source. The dynamic corrections looked as if they were updating and there wasn't any peculiar large focus correction, according to the Cleo antenna and status screens. Looking at the antenna optics screen, apparently the subreflector Y-focus was sometimes far from the correct position. Eventually, after suggestions from Melinda, we turned off the dynamic corrections and then peak observations all looked ok. We finally re-booted the antenna characterization manager after which the peak observations were normal. But we lost over 3 hours of telescope time fiddling around and trying to track down the problem. | We have added a fix for this in the Observing API for the case of Gregorian receivers. We are still working on solving the PF case. Joe is currently investigating requirements. More information to follow. AmyShelton | |
| 2 | 2005-10-08 | 09:35 | TBIGNELL_02 | Carl Bignell | cbignell & tminter | Other | Problem | Important | | Closed | Using GBTIDL for some observations using the Spectral Processor. The online option always selected the data of the last observer instead of my data. Is it related to the fact my project started with a T instead of an A? | The online command by defaul looks for the most recent Spectrometer data; to specify Spectral Processor data, use the /sp keyword with the online command.
This has been modified in gbtidl-test so that the online command simply picks up the most recent data, regardless of backend type. Will be available in gbtidl upon the next release. | |
| 3 | 2005-10-08 | 09:35 | TBIGNELL_02 | Carl Bignell | cbignell & tminter | Other | Problem | Important | | Submitted | The sympton. Standard on-off data using the spectral processor in square mode: GBTIDL only saw one IF but the data in the file appears to be complete with two IFs. Scans in the mentioned project directory with this mode are #6-#14. Is this an issue with SDFITS or GBTIDL? | This is an issue with SDFITS, which GBTIDL uses as input. Using the sdfits filler by hand, I get only one IF, with no warnings. I have examined the raw data for scan 6, and this also tells me there is only one IF. We should discuss this difference. | |
| 4 | 2005-10-10 | 17:15 | AGBT05C_049_03 | Minter | Minter | Other | Problem | Urgent | | Closed | While observing I had astrid pop up a window that said that the ability to update the system (ie the GFM part of astrid was no longer allowed to make updates for me). This happened while in the middle of an observation. The operator had not taken control from me nor anyone else that we could find. Joe B. looked in the database and couldn't find anything funny. Turtle had not restarted itself. I don't know if this was an astrid bug or something with security so I put it here to start with. | I discussed this issue with Toney. We have ruled out a number of possibilities except for the chance that GFM exitted irregularly and in doing so cleared the Security table of Toney's entry (which is correct behavior). Unfortunately, this is a hard one to recreate. I have asked staff to keep an eye on this and report any further problems to me. AmyShelton | |
| 5 | 2005-10-11 | | AGBT05C_049_04 | Minter | Minter | Other | Enhancement | Desired | | Rejected | I had the Spectral Processor go into aborting because it had a FIFO error. This caused astrid to allow me to abort my script - which was good. However, the antenna, receiver, converter rack, etc. were still left in running. When the data taking backend aborts it should also have the scan coordinator abort all devices if it is the only data taking backend. This would make recoving from this problem be quicker. | The explicit policy and design of the Ygor telescope control system is 1) it only contravenes commands from the user if equipment or personnel are in danger, and 2) each device is independent and autonomous. The system does not know, and I would argue should not try to determine, which specific failures should cause an entire scan to be aborted and data collection stopped. Only the user knows which devices and/or events are critical to his or her scan and therefore decides whether to abort the scan. In lieu of automatically giving up on a scan, the system loudly signals the problem, even to the extent of putting the ScanCoordinator into aborting though not ending the scan. The user then has the option of ending the scan through the abort command. In this specific case, an alternate solution might be for Astrid to send an abort command to the telescope whenever the user chooses to terminate a scheduling block. (I have taken the liberty of placing this into the Astrid requests.) MarkClark | |
| 6 | 2005-10-20 | 0900 | T_Oct20 | Robishaw | O'Neil | Spectrometer | Problem | Urgent | ?? | Fixed | During observations Tim noticed that the scans he s running were taking an unusually long amount of time (15 minutes for ~7 minutes of data). When we looked into it, it seems that the time was spent with the spectrometer in "stopping". I beleive this is due to the spectrometer needed considerable time to write the data across the network to disk. We will test this problem tomorrow (21 Oct).
-Main.KarenONeil | See resolution of problem #16 JoeBrandt | |
| 7 | 2005-10-18 | 18:20 | GBT05C-044_01 | Devine | O'Neil | Computer | Problem | Urgent | 10 | Submitted | Titania froze and had to be rebooted. this happened after a copy & paste operation in astrid (a fact which may be completely irrelevent)
-Main.KarenONeil | | |
| 8 | 2005-10-19 | 16:08 | GBT05C-030 | Lockman | O'Neil | Spectral Processor | Problem | Important | 00:02 | Closed | Spectral processor hung in stopping. Aborted the scheduling block then ran two scans from the spectral processor aborting both scans. The fault cleared on the second abort. | Typical behavior when the the data collection interface between the SpectralProcessor's accumulator and the computer fails. Long-standing problem. The log shows it was in rack B and the first failure was an attempt to read 16408 bytes but only 12288 were received, then only 4096 on the next read, and then 0 from then on until recovery. MarkClark | |
| 9 | 2005-10-20 | 00:10 | GBT05C-002 | Kavars | O'Neil | Spectrometer | Problem | Important | 00:03 | Closed | Data interrupt failure Bank A and B. Started the next scan and it cleared. | | |
| 10 | 2005-10-20 | 09:15 | T_20OCT05 | Robishaw | O'Neil | Antenna/Antenna Manager | Human Error | Desired | 00:05 | Closed | After moving to access to change the receiver Tim started a peak. The source the telescope slewed to was at 85 in elevation which caused problems with the antenna being able to accomplish a peak. The antenna aborted not being able to keep up in azimuth due to a high velocity. | Currently it is the responsibility of the user to avoid sending the antenna inoperative commands. MarkClark | |
| 11 | 2005-10-20 | 12:18 | T_20OCT05 | Robishaw | O'Neil | Spectrometer | Problem | Important | 00:10 | Fixed | The spectrometer hung in stopping. I aborted the scan and it hung in aborting. | See problem resolution of report number 16. JoeBrandt | |
| 12 | 2005-10-22 | | AGBT05A_024 | Campbell | Fhigo | Other | Problem | Important | | Closed | GBTSTATUS database has the wrong values for RA and DEC when using a coordinate mode of apparent RA/DEC. RA is displayed in degrees rather than hours and the displayed DEC seems to be exactly the same as the RA. | There are 2 issues:
1. The RA value was incorrectly being
displayed for both the RA and DEC.
2. The apparent RA/DEC was not being displayed in the format the user expected, but was displayed as designed.
The bug (issue 1) has been fixed. The improvement(issue 2) has been added. Both of these items will be released in sparrow version 5.7
-Main.MelindaMello | |
| 13 | 2005-10-22 | | AGBT05A_024, AGM057 | Campbell, vlbi | O'Neil | CLEO | Problem | Urgent | 00:05 | Closed | Cleo is giving false errors and stating that a project id is invalid when it is not. Tonight was the first observing session for both projects listed, and no directory with these project ids existed in teh /home/gbtdata area. Yet in both cases CLEO reported that the chosend session id (with 01 at the end) was invalid. For project AGBT05A_024 the operator (Donna) changed project ids until cleo accepted one (which ended up being session 4). for the vlbi run we just ignored cleo's warning.
- KarenONeil | The warnings were legitimate.
CLEO checks the contents of /home/archive as well as /home/gbtdata for duplicate sessions. In "archive", one will find sessions 1, 2, and 3 for AGBT05A_024 so 4 is exactly what CLEO should accept. (Note: If the operator typed in AGBT05A_024 without a project ID, CLEO would have automatically suggested 04 as a session number. It would be grand if Astrid also had this capability.) Astrid doesn't check /home/archive and, therefore, it has no way of completely foreseeing someone has entered a duplicate session number.
According to the requirements document for project ID's (http://www.gb.nrao.edu/internal/gbarchiv/naming-convention.shtml), VLB sessions do not accept session numbers. Thus, AGM057 is the exact string that should have been entered as an ID. In the case of VLB observing, then, Astrid should force a blank session ID.
In both cases, I thought the CLEO warning messages were rather explicit -- the messages were: "WARNING WARNING WARNING\nDuplicate project already exists in ARCHIVE area. PLEASE CHANGE ID" and "Project code has a bad length - must be 6 characters\n". Any suggestions for changing these messages? - RonMaddalena | |
| 14 | 2005-10-23 | 03:20 | AGBT05A_024 | Campbell | O'Neil | Antenna/Antenna Manager | Problem | Important | 00:10 | Assigned | The observer wished to run a map on the moon. The procedure ran through two scans and then hung in stopping with the message: "Failed to generate a stop trajectory - (jmFailed - Specified trafectory is not feasible)"
The map, as sent through astrid, is below:
DecLatMap("Moon_SRP", Offset("J2000",0.333,0.0,cosv=False), Offset("J2000",0.0,1.0,cosv=False), Offset("J2000",0.167,0.0,cosv=False), 30.0)
- KarenONeil | This is (unfortunately) a known issue that under some conditions the antenna fails to generate trajectories. The issue is being studied. A possible workaround are:
* reduce/increase rates in map
* observe source at different LST (which usually changes az,el rates) JoeBrandt | |
| 15 | 2005-10-23 | 18:30 | AGBT05C_014_03 | Devine | O'Neil | Other | Problem | Important | 00:45 | Need More Info | The observer set up her project and began observing, and everything was fine. She then aborted the observation (due to observer error in the script) in the middle of a scan (scan #8). She then commented out the Peak, configure, etc. commands, fixed her error in the procedure and re-ran things. For some reason, the spectrometer then get removed (during the abort? I don't know when) and the DCR was run for then entire map. the problem fixed itself for the next map, as a configure was run automatically before it.
- KarenONeil | From the stated sequence of events, it is indeed not clear how the Spectrometer could have been dropped. We need the script so we can attempt to reproduce the event. MarkClark | |
| 16 | 2005-10-23 | 09:30 | T_23Oct05 | Robishaw | O'Neil | Spectrometer | Problem | Urgent | 03:00 | Fixed | The spectrometer started taking about 5minutes to write its data (from a 80s scan) to disk. Joe came in and made the switch for the spectrometer to write to local disk. This appeared to solve the problem, leading me to believe the culprit was the network. Eventaully Joe switched things back, and the problem remained gone. Joe would have much more info on this than I do. - KarenONeil | This problem was due to a bug in the spectrometer FITS writing software. The problem was due to a race condition which was sensitive to the amount of time of the actual file writing required. As file writing time was reduced either by writing to a local disk or improved writing speed due to better network performance, the race condition was triggered.
The details of the problem are are complex, but it amounts to one thread performing checks for additional data before another thread has signaled that there is no more data. (i.e Each thread thought it was the other thread's next move.) This bug has been fixed and patched into 5.6, as of local noon Oct 23. Testing showed that stopping nominally took 1-2 seconds for the 80 second scans. JoeBrandt
| |
| 17 | 2005-10-27 | 08:00 | AGBT05A_024 | B. Campbell | Maddalena | Other | Problem | Important | 00:15 | Closed | Attempts to start a scan via Astrid produced RPC errors between the Scan Coordinator and LO1 at which point the scan would abort. Yet, starting a scan with the ScanCoordinator? directly via CLEO worked. Thus, this might be an Astrid problem. But, the RPC error messages suggest to me this has more of a flavor of a communication/network problem and that it was just happenstance that CLEO was able to start the scan.
Since this was a radar run that only needed to track the moon, and Arecibo radar time was running out, and the Astrid/GO FITS files aren't needed, and Astrid was able to load the antenna primary segments before the abort, running through CLEO was sufficient. | This is a known problem with the system. On long scans, both the antenna and LO1 perform lots of computations prior to begining the scan. If the computations take more than 10 seconds, RPC errors can result.
The current work-around is to specify scans of less than 1 hour in length.
(The script referenced had a 2 hour 46
minute scan.) | Observing files are in : /home/astro-util/radar/luna_oct05/obs2380.turtle |