| No | Date | Project Code | Observer | Support Scientist | Description & Scheduling Blocks | Response |
|---|
| 1 | 14 Jan 2006 | AGBT05C_033 | Marko Krco | Dana Balser | We had problems with the sdfits filler (both online and offline). We were not able to reach anyone at the time. I was able to look at the data with Glish and it appeared to be okay. Below is a message sent out by the observer, Marko Krco, with the details. Bob Garwood has since replied and has looked into it. I submit this report for the record.
The next observer, Alberto Bolatto, also had problems with the online filler but the offline filler seemed to work fine.
Dana
To: koneil@nrao.edu, dbalser@nrao.edu
Cc: nradziwi@nrao.edu, pmargani@nrao.edu, Paul.F.Goldsmith@jpl.nasa.gov
Subject: sdfits filler problem for AGBT05C_033
Date: Sat, 14 Jan 2006 15:22:12 -0500
Hi folks,
Dana and I have been beating our heads against this problem for a few hours, and
we're lost. I'm making a frequency switched DecLat? map in L-band using 4
spectral windows (1 for HI and 3 for OH). The telescope "seems" to be fine,
that is we got no complaints and it seems like it was doing what it was supposed
to. But data reduction seems to be a problem. For completeness sake I'm going to
try and describe the problem in detail.
After the first scan (strip) of my map was completed I tried running online in
GBTIDL but it simply reverted to the previous user's observations. It wasn't
until I tried it again a while later that I got something like this:
Invalid extension header encountered
XTENSION keyword missing
Only 1 extensions processed
In the summary window I got a few scans, but each one seemed to have only 2 IFs
(where as there should be 4). What's even odder is that after a few minutes the
sdfits file (the one automatically generated by the online filler) quite simply
disappeared..and reappeared again several minutes later.
We then tried manually filling a file and we got something like this:
[pgoldsmi@despina ~/L204]$ sdfits -backends=acs -scans=7 -mode=raw
/home/gbtdata/AGBT05C_033_01/
Gathering project info ...
Scan 7 (spectrometer) sampler B21, integration 84, state [SIG, CAL OFF]
writtegsl: interp.c:37: ERROR: insufficient number of points for interpolation type
Default GSL error handler invoked.
Aborted
It still created a file though, and we tried loading it into GBTIDL...
GBTIDL -> filein, 'AGBT05C_033_01.raw.acs.fits'
Invalid extension header encountered
XTENSION keyword missing
Only 1 extensions processed
About to create Index...
Parsing scan info:
__________
XXXXXXXXX
Writing rows to index file:
__________
XXXXXXXXX
Index file created.
GBTIDL -> summary
Scan Source Vel Proc Sub RestF? nIF nInt nFd Az El
-------------------------------------------------------------------------------
7 L204 4.0 DecLatM? 0 1.420 2 120 1 162.4 37.1
Where again the number of IFs is wrong, but everything else looks right. We can
look at two of the IFs and see the data, but there's still two windows missing.
Now to make things even stranger, a couple of hours later I tried running online
in GBTIDL again (I hadn't done anything else to change anything as far as I can
remember) and it didn't complain. In the summary page there were now 4 IFs for
each scan. But when I tried looking at the scans with getfs I'd get an error
message stating that there weren't enough spectra or phases to complete the FS
(sorry I can't reproduce the error msg right now). Only 2 out of about 20 scans
did not have error messages, and those seemed to be fine. But the rest I
couldn't calibrate because of that error msg with getfs.
In hope that it might be just a problem with looking at online incoming data I
tried manually making an SDFITS file with the filler after the observing was
complete, but I got the same message as before. Upon loading the sdfits file
into GBTIDL the number of IFs again seemed to drop to 2.
So I was hoping someone could help...:)
thank you,
Marko Krco
| It was determined that sdfits was crashing when it had problems with the Vlan Vleck corrections - specifically in scan 7, integration 85, sampler B21. sdfits-test catches these errors and places NaNs? in the data, so the user was able to use this to fill their data. It looks like much of the other errors in GBTIDL were steaming from this initial problem. |
| 2 | 16 Jan 2006 | AGBT06A_043 & T_RCO8_060116 | Morgan & Maddalena | Ditto | Larry was working with L-band and then K-band. It'll be easier if I address the problems in the reverse order to when they occurred.
K-band problem:
---------------
Config-tool appeared to correct setup the system. Yet, Larry saw wrong frequencies come out of GBTIDL and I found that LO1A's frequency (and that reported by the IF Manager) was 2 GHz off from where it should have been (20 something vs. 22.2 GHZ). However, the LO1 coordinator's control parameter was set to 22.2 GHz. When Larry/Joe retyped the frequency into CLEO, they stumbled upon a way to get the correct frequency into the synthesizer. Even after this 'fix', Larry never saw his test line so it's not clear if the system was properly set up in the end. Joe continue to work on the problem.
We found two other problems that somewhat obfuscated the LO1 problem. Glen's Spectrometer tool is returning zeros for one of the phases for one sampler for AGBT06A_043_CCS. GBTIDL was garbling the plotting of the data for the same project. For example, channel numbers were running from -4000 to 4000, which is not possible. Bob G has already fixed the later problem, but not yet released the change.
L-band problem
--------------
While diagnosing the L-band problem (the system had a sky frequency of 1400 MHz and not the specified 1666 MHz), I didn't think to look at the LO1A synthesizer values -- I was working under the assumption the problem was a configuration problem similar to the PF1 issue from a couple of weeks ago. At one point, I used CLEO to change the configuration in various ways. In retrospect, and since the symptoms of the L-band problems were so similar to that at K-band, it was probably my stumbling retyping of the rest frequency via CLEO that 'fixed' the problem.
Larry will have to be the judge but he may have actually collected 90 minutes of good data during this part of his run.
PF2 checkout
------------
Like Glen, I too saw that the operator hadn't changed the focus to 500 mm but I caught that right off and instructed the daytime operator as to why this is needed. I've sent the rest of the operators a reminder of what to do for the different PF receivers.
Again, we had frequency labeling/setting problems but of a totally different, and still unexplained, nature. Since this was the 3rd project in a row where frequencies weren't coming out right, I judged that I should spend most of the checkout time on the problem. So, I only gave the receiver a partial 'checkout' and missed the balance problem that Glen reported.
I set up the system with two 200MHz IF's centered at 970 and 1160 MHz, so as to span the full receiver's band (See ~rmaddale/GBT/GeneralSetupFiles/PF2Config_turtle). Thus four samplers of which I expected LL IF1 and RR IF1 to be identical (except for RFI's polarization properties), likewise LL IF2 and RR IF2, while IF1 and IF2 should show the receiver bandpass edges at 910 and 1220 MHZ, respectively. Almost all systems (gfm-test, GBTIDL, Glen's tool, IF Manager) agreed on how each sampler should be labeled, which agreed with what I expected from how the system was wired.
Yet, the data showed LL IF1 data as being identical to LL IF2 while the data for RR IF1 identical to RR IF2. LL IF2 and RR IF1 showed the receiver bandpass edges occurring at drastically wrong frequencies while LL IF1 and RR IF2 showed the correct edges. It's as if the IF Manager has scrambled the labeling of polarizations and/or IF's. (Glen's tool also seems to improperly flip the frequency axis for one of the samplers). Mark and I have dismissed the hypothesis that the cabling file was wrong.
Ron
| Responses for three parts:
1. Although the LO1 rest frequency was correctly set as indicated by
configtool and cleo, the LO1 recorded a frequency some 2 GHz lower than
requested. (The frequency used to observe the calibrator.) Why the LO1 did not
properly accept the new frequency is not currently known. (JJB)
2. What exaclty is the problem? I'm afraid we need more info. (PRM)
3. We're looking into wether this problem occured with subsequent PF2 projects. Currently querying Glen Langston. If it has reappeared, we will request test time to look further into it. (PRM)
|
| 3 | 28 Jan 06 | T_PCO_06A_040 | Minter | Minter | We started getting RPC connection errors between the scan coordinator and the IF Rack. The IF Rack would stay in activating after every other device had gone into running and would lag the other devices by more than a few tens of seconds.
Mark Clark was called. He had the operator take the IF Rack out of the scan coordinator and then put it back in. This
didn't fix things.
The scan coordinator was then thought to be the culprit and was restarted. This didn't fix things either.
Now we tried restarting the IF Rack (Mark Clark also reports sacrificing a goat =) ). This didn't help things either.
Mark is now calling Joe Brandt. It is now thought that this could a PF manager being left on and flooding the system with messages. Things are now better. | The goat did not help, the turning off of the PF manager probably did because of its activity on the MCB rather than generating messages, finally, there was some problem with the message system discovered later which may have been a contributor. MarkClark |
| 4 | 28 Jan 06 | T_PCO_06a_040 | Minter | Minter | At 22200 MHz, lower K-band, IF Rack filter 5960-6040 you can get a maximum of only 1.7 Volts to the IF Rack. Some of the ODMs don't have enough power to balance well at 1 Volt target level. | |