| No | Date | Time | Project Code | Observer | Support Scientist | Instrument | Bug, Etc. | Priority | Time Lost | Status | Description | Response | Scripts |
|---|
| 1 | 2005-08-04 | | AGBT05B_020_01 | Morris | Minter | Other | Problem | Urgent | | Fixed | We saw several errors that eventually
was tracked back to grail not running properly. The errors were that the AntennaCharacterizations would continually die, astrid said that a script that had previously been verified would no longer verify, astrid (in the configuration part of the script) would not see changes made to scripts made outside of astrid, and astrid would balance the spectrometer to values appropriate for fast samplers even though slow samplers were being used.
Once grail was restarted all starting behaving well. | Grail had a problem with the way it handled subscribers to samplers, which, when coupled with a problem in the M&C Linux implementation of the sampler system, caused it to prematurely shut down sampler streams. This is definitely the problem that has affected AntennaCharacterization, and possibly explains the other problems as well. Both Grail and the Linux M&C sampler system have been fixed to solve this problem; Both of these fixes will be available on release 5.5. RamonCreager | |
| 2 | 2005-08-04 | | AGBT05B_020_01 | Morris | Minter | GFM | Problem | Urgent | | Fixed | GFM did not reduce the third scan of a pointing. In the left side where the scan number, source name etc. show up it had just 'n/a'. A message saying that the scan did not have a valid GO FITS file appeared.
The data were present and OK. | This is exactly the problem that the SDD was troubleshooting at the beginning of the cycle. It seems to be related to NFS. The turtle daemon is writing the GO FITS file out on wind and the GFM application (on the user's machine) does not find the file. We thought that we had fixed this by adding an "ls" before trying to read the file, but obviously for some reason this isn't taking care of the problem. To be honest, I'm not quite sure what else that I can do. I need ideas. AmyShelton
We believe this is fixed by changing the NFS caching settings on the NFS clients that look for changes in the directories being written by other NFS clients.
Main.JohnFord | |
| 3 | 2005-08-05 | | AGBT05B_020_02 | Morris | Minter | Other | Problem | Urgent | | Closed | We had problems with Grail the same as #1 above. We rebooted grail as soon as the Antenna Characterizations started failing often. | See item #1 | |
| 4 | 2005-08-05 | | AGBT05B_020_02 | Morris | Minter | GFM | Problem | Urgent | | Closed | We had GFM not reduce the 3rd scan of a pointing. This is the same as #2 above. | See item #2. | |
| 5 | 2005-08-04 | | AGBT05B_020_01 | Morris | Minter | Spectrometer | Problem | Urgent | | On Hold | The observer is reporting that some (but not many?) 1024 lag jumps are being seen from banks C and D from the observing run. These can be flagged and the majority of the data are OK.
The spectrometer had 16 input signals and the integration time was 20 seconds. | yes - the bad LTA cards were moved to banks C & D, as fewer folks use them.
Hopefully when the new cards are complete this problem will go away. KarenONeil | |
| 6 | 2005-08-05 | | AGBT05B_011_25 | Minter | Minter | Spectral Processor | Problem | Urgent | | Closed | The spectral processor is reporting that IF 1 is failing to balance for both Rack A and Rack B. However the astrid script successfully balanced these. | The balancing in astrid does not use the Spectral Processor's internal balance mechanism,so if the SP's last balance failed, and then astrid succeeds, this will not clear out the error message since the balancing was done, essentially, manually. Also note that the two mechanisms have different criteria. Astrid measures levels ignoring whether the cal if firing or not, whereas the SP only measures states when the cal is not firing. MarkClark | |
| 7 | 2005-08-05 | | AGBT05B_011_25 | Minter | Minter | Spectral Processor | Problem | Urgent | | Closed | The spectral processor had FIFO overflow errors for Rack B at the beginning of scan 3. Only data for Rack A was being written. Aborting the scan, running a dummy scan in the Spectral Processor clear the error. | Data connection between computer and hardware fails sometimes, always has. MarkClark | |
| 8 | 2005-08-06 | | AGBT05B_011_26 | Minter | Minter | Spectral Processor | Problem | Urgent | | Closed | The spectral processor had FIFO overflow errors for Rack A at the beginning of scan 14. Only data for Rack B was being written. Aborting the scan, running a dummy scan in the Spectral Processor clear the error. | Data connection between computer and hardware fails sometimes, always has. MarkClark | |
| 9 | 2005-08-12 | | AGBT05B_011_30 | Minter | Minter | LO1 | Problem | Urgent | | CCC Approved | LO1 is showing a warning. The LO1 cleo screen suggests it is coming from LO1A. However, no associatted error message is present. The data do not seem to be affected. | This is a non-reproducible, but continuing irritant. All the Managers from time to time, unpredictably, have a status which is not supported by its messages. Apparently, a message gets cleared, but somehow the status misses the fact and fails to be updated. This becomes more serious, the more that users depend on the status to decide whether they should check the messages or not. These false alarms will eventually cause some important event to be missed. In general, it makes Ygor appear (and act) unreliable. I am arguing that some time be set aside to nail this one down. MarkClark | |
| 10 | 2005-08-16 | | AGBT05B_011_31 | Minter | Minter | Spectral Processor | Problem | Urgent | | Closed | When configuring for the observations, the spectral processor became hung in aborting for some reason (not very obvious). The operator rebooted the spectral processor but the WavTek? and the spectral processor were not communicating properly. The operator called Mark Clark who was able to help the operator get the spectral processor back running properly. | The usual cause of the Spectral Processor being hung in Aborting is the single-board computer being inoperative. When rebooting the single-board computer used by the Spectral Processor (gbtspsbc), sometimes the WaveTek synthesizer grabs control of the IEEE-488 bus which hangs the software on gbtspsbc. This is indicated by a "WaveTek not responding" error which can be cleared by turning the WaveTek off and back on before rebooting gbtspsbc again. MarkClark | |
| 11 | 2005-08-17 | | AGBT05B_011_32 | Minter | Minter | CLEO | Problem | Urgent | | Fixed | The obsFreq parameters entry/display has disappeared from the Spectral Processor cleo window. This is needed for pulsar observations with the spectral processor to make sure that quick look data displays and real time displays (like lurk) work properly (which use these values). Although you can still get to these values through dev explorer it much more painful. | The observeFreq and actualObserveFreq parameters have been re-instated. RonMaddalena? | |
| 12 | 2005-08-17 | | AGBT05B_011_32 | Minter | Minter | Analog Filters/Rack | Other | Urgent | | Closed | Astrid was very slow in starting up as well as cleo. This was using Ariel. Are network slowness issues on the horizon again? | We are examining a number of slowdown reports. A reboot of ariel was done to pickup some of the recent nfs configuration changes. JoeBrandt | |
| 13 | 2005-08-19 | | AGBT05B_011_34 | Minter | Minter | Spectral Processor | Problem | Urgent | | Closed | We had recurring FIFO errors for Rack A and Rack B of the spectral processor this morning. After several attempts and several reboots the problem changed to having no "GO Pulse" in either Rack A or Rack B. Rich Lacasse was called in and he decided to do a hard reboot of gemini. After this the spectral processor worked just fine.
About 2 hours of time was lost. | Yet another known occasional problem, maybe hardware, maybe software, likely an interaction of the 2. | |
| 14 | 2005-08-20 | 02:00 | AGBT05B_054 | Lockman | O'Neil | LO1 | Problem | Urgent | 01:00 | Fixed | We were attempting to run in FS mode 1-0, which is what the LO1 screen said that we were in. The CLEO status screen, though, claimed we were in mode FS2-1, which is what was apparently recorded in the data, causing the data reduction programs to be very confused. We then tried to get the LO1 (via CLEO) to switch to mode 2-1, to at least make things consistent. But As soon as I would hit "prepare" in the LO1 window, the mode would revert back to FS1-0 within that window, while the status would continue to say 2-1. I am thouroughly baffled by this. - KarenONeil | According to your configuration, swtype = "fsw" and swmode = "sp". The values that get written to the GO FITS file in this case (via Turtle) are SWTCHSIG = "FSW12" and SWSTATE = "FSWITCH". However, I see that SWSTATE = "PSWTICHOFF", which does not seem correct. This might explain why the data reduction programs were not happy, but does not explain the CLEO/parameter-setting weirdness. AmyShelton
The FS01 and FS12 modes displayed by Cleo are really the same mode (i.e. frequency switched with two offsets). The only difference is FS01 indicates the first offset is zero. The configuration used had the first frequency offset as zero, so Cleo actually displayed the correct value.
This difference in nomenclature is being resolved by Ron, in coordination with Jim and Bob.
JoeBrandt
I patched in a fix to the Observing API to fix the setting of SWSTATE and SWTCHSIG. AmyShelton | |
| 15 | 2005-08-24 | | AGBT05B_007_06 | Minter | Minter | Computer | Problem | Desired | | Closed | During the configuration phase there was a noticably longer period between when a command was sent to the M&C system and when things went into Activating.
The time between when scan a stops and scan b starts is taking about 40 seconds. Looking back at previous runs of this project shows that the typicall time between scans is 20 seconds.
Is this due to the NFS being slow? | We are continuing to examine network configuation issues, but I don't think that was the cause. I did however note that the Antenna/LO1 had 117 records specifying a user-defined coordinate system. This was probably left over from an earlier observation.
The changes in Astrid for 5.6 will revise how ephemerides are handled, and should mitigate the problem of 'left-over' settings. JoeBrandt | |
| 16 | 2005-08-30 | | AGBT05B_007_10 | Minter | Minter | Spectral Processor | Problem | Desired | | Submitted | When I started to setup for the observations the config script failed because it could not set anything up in the Spectral Processor. We then found that NONE of the Spectral Processor managers were running - no processes existed on gemini. There were no error
messages or anything to indicate that the managers were not running!
Restarting the managers solved the problem. | | |
| 17 | 2005-09-01 | | AGBT05B_007_11 | tminter | tminter | Other | Problem | Desired | | Submitted | #15 above.
I am still seeing very large latency times between scans. They are 40 seconds or more when previously they have been half this value or less. So the suggested reasons for this given above do not seem to be correct and there must be something else going on. | | |