| No | Date | Time | Project Code | Observer | Support Scientist | Instrument | Bug, Etc. | Priority | Time Lost | Status | Description | Response | Scripts |
|---|
| 1 | 2005-07-02 | 17:30 | AGBT05B_021_01 | Lim | O'Neil | DCR | Problem | Important | 01:00 | Closed | There was a problem with the DCR. The DCR was not restarting with TaskMaster and there was a communication problem between Algol and the DCR itself. This meant that neither the config_tool nor astrid could be run, and the set-up had to be done manually. Total time lost was about 1 hour. | This was due to the DCR being set as switching master, but not actually wired in the switching signals selector as the master. Normally when config_tool or astrid are used, the DCR gets configured properly (with respect to the switching master).
I have installed patches so that the misconfiguration is no longer a 'fatal' error.
JoeBrandt | |
| 2 | 2005-07-05 | 16:30 | AGBT05B_03 | Lovell | Ghigo (Prestage) | DCR | Problem | Important | 00:30? | Closed | Apparently the NTP daemon was not running on Algol. This appeared to cause no error messages (at least none that I was aware of), but caused the data for PEAK scans to be strongly hysteretic (by ~1 arcmin).
We need to add a check similar to virgo's so that the Operators (and support staff) will know if the NTP daemon is not running | It appears that this was due to an error on my (jbrandt) part. I failed to restart the daemon after installing a patch Sunday morning. I am asking Chris to add this host the the list of ntp monitored hosts so that errors can be detected before causing problems with observing. JoeBrandt | |
| 3 | 2005-07-07 | 16:00 | AGBT05B_03 | Lovell | Prestage (Maddalena) | Other | Problem | Important | 00:30 | Closed | Joe knows the details. The Scan Coordinator had a bizarrely large value for the expected start time. All of the Scan Coordinator FITS file info (including the expected name of the GO FITS file) seemed correct. But GO created FITS files (and wrote the start time inside them) with the incorrect UT date. So, off-line packages couldn't process the data since the GO FITS files didn't have the expected names.
We ran down GO and restarted the Scan Co-ordinator, and everything was ok after that. | We are unable to reproduce this. I am closing it as a one-off error, but will reopen if it happens again. AmyShelton | |
| 4 | 2005-07-07 | 17:30 | AGBT05B_03 | Lovell | Prestage | LO1 | Problem | Desired | 0 | Closed | Five seconds before the end of many scans, the LO1 (A and B) would report:
Sig/Ref state incorrect in stepper (1 != 0).
This would clear at the end of the scan.
Both the observer and the operator seemed to think this was innocuous, so I've put down no time lost, but it looked a bit ominous to me (Richard). | This would be seen if the switching period was not an integer multiple of the scan length. In testing we were able to recreate the condition. We also confirmed that the scan ended (and backend FITS file closed) prior to the error condition. JoeBrandt | |
| 5 | 2005-07-07 | Many | AGBT05B_03 | Lovell | Prestage | DCR | Problem | Important | little | Closed | DCR does not appear to behave correctly on first TP scan after frequency switching has been in effect (with SP as the master). Very first integration has a huge value. This throws GFM off. Scan numbers, etc emailed to Amy. (Richard) | Joe investigated this issue and installed a patch after testing. Tests confirmed the problem was fixed. JoeBrandt | |
| 6 | 2005-07-24 | 07;30 | AGBT_05B_03 | Kearsley | O'Neil | Spectrometer | Problem | Other | 00:45 | Closed | The spectrometer was giving garbage data - spectra repeated every few 10s of megahaertz. Typically, this can be fixed via a "reset parameters" on the spectrometer manager. We tried this, to no avail. I then ran the "test using interrupts" and "test using polling" and got no errors, but the problem persisted.
After a few more reconfigures and a few more "rest parameters" and "revert to last legal", I tried testing the xilinx personailities (using the test on the cleo spectrometer window). This, combined with a manager restart, solved the problem, | The connection between the control software and the firmware over the single serial line has been and remains unreliable. This is why "reset" commands often clear up bad data problems, i.e., some configuration is only "kinda" loaded
and so the system requires a fresh start. I have seen cases where the firmware's configuration is not only incomplete, but also corrupted. In these cases something stronger than a reset is needed, e.g., a restart, but I do not understand why since we have tried to have a reset do everything that a restart does short of cycling the power. These events occur rarely enough that CCC has judged the effort required to redo the serial interface is not justified at this time. MarkClark | |
| 7 | 2005-07-24 | 10:15 | AGBT05B_052 | Kramer/Feldman | Maddalena | Spigot | Problem | Important | 00:45 | Submitted | The spigot was misbehaving and would produce histograms that had data values at either + or -3e14. The observer was using the old non-Python-script instructions and X so we couldn't snoop on what he was typing. We tried various things, like restarting and stopping the manager, running through the setup a few times, resetting the Spectrometer's parameters, etc. Called up Scott and Karen.
Scott asked us to turn on and off the manager and then used his scripts to set up the spigot. Then, all was fine.
Scott came up with three possibilities as to why things started working: (1)
turning the manager on/off; (2) Robert F. was using the old instructions and
typed in something wrong; or (3) cott's scripts use a different binary and, therefore, there may be something amiss with the binary that the old
| The symptoms described here suggest an incorrect setup. I don't know if this applies here, but I have had similar histograms when running the spigot checks when the input power was excessive. Users should use the Python scripts that Scott has provided. RamonCreager | |
| 8 | 2005-07-31 | | AGBT05B_025_02 | West/Blanton | O'Neil | Other | Problem | Important | 01:00 | Submitted | The observer has experienced some poor baselines during his runs. First, every 2-3 hours the baselines seem to go bad and then clear up quickly. Going through his subscans (integrations) I can see that he just has a couple of scans with a significant (4-8MHz) ripple in them. I can't see anything in the raw lag data causing this, I can only assume there is some instability in the system. Scan #17 in the dataset shows this well.
the second issue is addressed in the next row. | One possible cause is strong out-of-band intereference. Perhaps satellites or aircraft. But, difficult to know for sure. (R. Norrod) | |
| 9 | 2005-07-31 | 14:30 | AGBT05B_025_02 | West/Blanton | O'Neil | Other | Problem | Important | 02:00 | Submitted | The second issue is that for about 2 hours, from 14:30-16:40 the baselines are of poor quality compared to most of the rest of the data (but much better than the baselines in, e.g. the scan 17/18 pair). My only guess is that this is due to reflections of the sun off the dish, as the sun was about 20 degrees from the observer (at L-band) | The sun is definitely a possible cause. (R. Norrod) | |