(updated 2-Apr: Dicker, Mason, Aguirre, White, Whitehead, Bloss)

• Date MUSTANG removed from telescope: On or about Wednesday, 2-Apr. After the instrument is in the GB labs it will be reviewed by a joint UPenn/NRAO team to finalize the plan for accomplishing the tasks below. This work is expected to take 2-3 days.

• Date MUSTANG removed from telescope: Thursday, 3-Apr. Instrument will be transported to UPenn on 10-Apr.

• UPenn Laboratory tests of Array #3 – Array #3 needs some remediation work by NIST and wirebonding by either NASA or Princeton. The team agreed the Green Bank facilities and resource experience were not appropriate for the wirebonding work.

• UPenn Laboratory tests of Array #3 – Array #3 needs some muxes by NIST due in the next few weeks followed by some wirebonding by either NASA or Princeton. Preliminary tests of Array #3 will occur over late spring – early summer.

• Two drops of YGOR manager code releases and testing on MUSTANG over the summer (inc. housekeeping) – Over the course of the summer, there will be two drops of YGOR code updates that require testing on MUSTANG while it is cold. The team believes accommodating these tests with MUSTANG at UPenn should not be a problem if advance notice is provided and testing schedules met. An immediate effort to evaluate the issues surrounding VNC access between Green Bank and UPenn should begin to assure that networking/connectivity issues are resolved and the path is robust before the first round of testing. The upgrade to a real-time version of Linux is a significant undertaking and has been deferred for now, but if needed this change could be revisited later in the summer.

• Two drops of YGOR manager code releases and testing on MUSTANG over the summer (inc. housekeeping) – Over the course of the summer, there will be two drops of YGOR code updates that require testing on MUSTANG while it is cold. The team believes accommodating these tests with MUSTANG at UPenn should not be a problem if advance notice is provided and testing schedules met. An installation of YGOR on ‘Spider’ will occur during the week of 7-Apr. while MUSTANG is in GB. The upgrade to a real-time version of Linux is a significant undertaking and has been deferred for now, but if needed this change could be revisited later in the summer.

• Review and analysis of the air circulation issues/doughnut air circulation improvement – There is general consensus that improved air circulation around the circuit boards is advisable although the insulation added in December appears to have mitigated the problems in the tower. When MUSTANG is removed in April a plan will be discussed to add circulation and potentially a fan to move air through the doughnut. All agree a minimum of alterations to the housing is a major consideration in the plan. A plan for improved grounding of the tower will be created concurrent with this work.

• Review and analysis of the air circulation issues/doughnut air circulation improvement – There is general consensus that improved air circulation around the circuit boards is advisable although the insulation added in December appears to have mitigated the problems in the tower. As part of MUSTANG removal, a plan will be discussed to add circulation and potentially a fan to move air through the doughnut. All agree a minimum of alterations to the housing is a major consideration in the plan. A plan for improved grounding of the tower will be created concurrent with this work.

• Separate tower and crate into separate remotely controllable power supplies – Before MUSTANG leave GB, a set of specifications for a new dual power supply will be created. Having a separate supply for the crate and tower allows for independent rests if an error condition arises; a feature that would have been useful in the concluding observation season. GB engineers will procure and build the supply in consultation with UPenn and ship the completed, tested unit to UPenn for installation at a time of UPenn’s choosing. The existing supply will be brought to GB in the fall along with MUSTANG and added to the spares pool for the instrument. The YGOR Manager will need to be updated to provide independent commands for tower and crate.

• Separate tower and crate into separate remotely controllable power supplies – Before MUSTANG leaves GB, a set of specifications for a new dual power supply will be created. Having a separate supply for the crate and tower allows for independent rests if an error condition arises; a feature that would have been useful in the concluding observation season. GB engineers will procure and build the supply in consultation with UPenn and ship the completed, tested unit to UPenn for installation at a time of UPenn’s choosing. The existing supply will be brought to GB in the fall along with MUSTANG and added to the spares pool for the instrument. The YGOR Manager will need to be updated to provide independent commands for tower and crate.

• Define failure scenarios and resolution roles and strategies – The team will meet the week MUSTANG is removed to identify the potential failure modes, single points of failure, and strategies for recovery in the event of those failures.

• Define failure scenarios and resolution roles and strategies – Determination of problems with MUSTANG and recovery will follow the same basic procedure as the other GB receivers. The first task after any reported failure will be to determine if the failure is hardware or software and then bring in local or remote diagnostics assistance as necessary. Simon and Brian will be the escalation point for problems that cannot be resolved by local technical staff. Simon will be the resource for repair of any problems in the cold areas with warm area repairs undertaken by GB personnel under Simon’s supervision.

• DAQ computer installation & removal procedures; receiver installation and removal. – The existing computer occupies the space needed for holography receiver components. GB will procure a replacement computer in a smaller form factor (2-3 RU) so that it can be installed inside the crate. This moves the computer into the RFI protection of the crate and simplifies the interconnecting cabling. Special consideration is needed for the GPIB and PCI boards. GB Software engineers will need to participate in the loading and configuration, and testing of the new computer. A second computer with the same form factor will need to be procured and delivered to GB along with MUSTANG in the fall and remain available as a test bed and emergency backup computer to MUSTANG.

• DAQ computer installation & removal procedures; receiver installation and removal. – The existing computer occupies the space needed for holography receiver components. During removal of MUSTANG in April a review will determine if space can be made available in the crate for the 4 RU ‘Spider’ computer. In the event a 4 RU space is not possible, GB will procure a replacement computer in a smaller form factor (2-3 RU) so that it can be installed inside the crate. This moves the computer into the RFI protection of the crate and simplifies the interconnecting cabling. Special consideration is needed for the GPIB and PCI boards (voltages and form factor) along with the potential demands of a 16x16 array and a possible faster Lsync speed. GB Software engineers will need to participate in the loading and configuration, and testing of the new computer. If ‘Spider’ can not be accommodated in the crate, a second computer with the same form factor as the replacement will need to be procured in a subsequent year as a test bed and emergency backup computer to MUSTANG.

• Documentation – There remain some substantial ‘holes’ in the documentation of MUSTANG available to the GB engineers; particularly the firmware and tower details. Team members will continue to post documents to http://wiki.gb.nrao.edu/bin/view/Pennarray/MustangDocumentRepository and GB team will ultimately move them into the GB documentation and drawing archives.

• Documentation – There remain some substantial ‘holes’ in the documentation of MUSTANG available to the GB engineers; particularly the firmware and tower details. Team members will continue to post documents to http://wiki.gb.nrao.edu/bin/view/Pennarray/MustangDocumentRepository and GB team will ultimately move them into the GB documentation and drawing archives. A scan for the completeness of the documentation by all principal team members in needed. There's also a wiki page with notes on, and links to, firmware versions and files in connection with the address card firmware at http://wiki.gb.nrao.edu/bin/view/Pennarray/MarkIIIFirmwareVersions .

• Improved cooling information - Data mining of all cryogenic records of the receiver (a UPenn summer student/grad student project?) to better determine temperature features of receiver. Items include better estimates of time remaining in a cooling cycle as a function of elevation (how much evaporation, how much condensation). This information could be ultimately add into YGOR as a feature.

• Refrigeration pulse - Modify receiver to get rid of (or reduce by a magnitude) the 1.4 Hz pulse caused by refrigeration that must be corrected for in the data. Requires

• Two drops of YGOR manager code releases and testing on MUSTANG over the summer (inc. housekeeping) – Over the course of the summer, there will be two drops of YGOR code updates that require testing on MUSTANG while it is cold. The team believes accommodating these tests with MUSTANG at UPenn should not be a problem if advance notice is provided and testing schedules met. An immediate effort to evaluate the issues surrounding VNC access between Green Bank and UPenn should begin to assure that networking/connectivity issues are resolved and the path is robust before the first round of testing. The upgrade to a real-time version of Linux has been deferred for now, but if time allows could be added in later in the summer.

• Two drops of YGOR manager code releases and testing on MUSTANG over the summer (inc. housekeeping) – Over the course of the summer, there will be two drops of YGOR code updates that require testing on MUSTANG while it is cold. The team believes accommodating these tests with MUSTANG at UPenn should not be a problem if advance notice is provided and testing schedules met. An immediate effort to evaluate the issues surrounding VNC access between Green Bank and UPenn should begin to assure that networking/connectivity issues are resolved and the path is robust before the first round of testing. The upgrade to a real-time version of Linux is a significant undertaking and has been deferred for now, but if needed this change could be revisited later in the summer.

• Separate tower and crate into separate remotely controllable power supplies – Before MUSTANG leave GB, a set of specifications for a new dual power supply will be created. Having a separate supply for the crate and tower allows for independent rests if an error condition arises; a feature that would have been useful in the concluding observation season. GB engineers will procure and build the supply in consultation with UPenn and ship the completed, tested unit to UPenn for installation at a time of UPenn’s choosing. The existing supply will be brought to GB in the fall along with MUSTANG and added to the spares pool for the instrument.

• Separate tower and crate into separate remotely controllable power supplies – Before MUSTANG leave GB, a set of specifications for a new dual power supply will be created. Having a separate supply for the crate and tower allows for independent rests if an error condition arises; a feature that would have been useful in the concluding observation season. GB engineers will procure and build the supply in consultation with UPenn and ship the completed, tested unit to UPenn for installation at a time of UPenn’s choosing. The existing supply will be brought to GB in the fall along with MUSTANG and added to the spares pool for the instrument. The YGOR Manager will need to be updated to provide independent commands for tower and crate.

• DAQ computer installation & removal procedures; receiver installation and removal. – The existing computer occupies the space needed for holography receiver components. GB will procure a replacement computer in a smaller form factor (2-3 RU) so that it can be installed inside the crate. This moves the computer into the RFI protection of the crate and simplifies the interconnecting cabling. Special consideration is needed for the GPIB and PCI boards. GB Software engineers will need to participate in the loading and configuration, and testing of the new computer. The existing computer (Spider) will be shipped to GB along with MUSTANG in the fall and remain available as a test bed and emergency backup computer to MUSTANG.

• DAQ computer installation & removal procedures; receiver installation and removal. – The existing computer occupies the space needed for holography receiver components. GB will procure a replacement computer in a smaller form factor (2-3 RU) so that it can be installed inside the crate. This moves the computer into the RFI protection of the crate and simplifies the interconnecting cabling. Special consideration is needed for the GPIB and PCI boards. GB Software engineers will need to participate in the loading and configuration, and testing of the new computer. A second computer with the same form factor will need to be procured and delivered to GB along with MUSTANG in the fall and remain available as a test bed and emergency backup computer to MUSTANG.

%META:TOPICINFO{author="mbloss" date="1205431588" format="1.0" version="1.1"}% %META:TOPICPARENT{name="MustangProjectPlan"}%

MUSTANG Hardware Meeting 11-Mar-08

• Attendees
• Agenda
• MUSTANG work 4/1/08 – 10/1/08

Attendees

Agenda

• Introductions (if necessary)
• Meeting goals
  • Final list of work for the summer
  • General work timeframes
  • Logistics
  • What GB resources are needed and when?
• Review scope of work for summer
• Discuss special logistics and general timeframes for work
• Identify teams and launch work
• Action items

MUSTANG work 4/1/08 – 10/1/08

• Date MUSTANG removed from telescope: On or about Wednesday, 2-Apr. After the instrument is in the GB labs it will be reviewed by a joint UPenn/NRAO team to finalize the plan for accomplishing the tasks below. This work is expected to take 2-3 days.

• UPenn Laboratory tests of Array #3 – Array #3 needs some remediation work by NIST and wirebonding by either NASA or Princeton. The team agreed the Green Bank facilities and resource experience were not appropriate for the wirebonding work.

• Install and test potential new array at UPenn (Array #4) – A strong push is underway to get a fourth array of 256 pixels build and installed for the 2008 fall/winter observing season. If this effort is successful, delivery of the array would be in late summer and there will likely be some changes to the optics required. This work could cause a delay in having MUSTANG ready for the fall season. Since this work is currently speculative, it is not currently included in the plans.

• Laboratory tests to identify and mitigate low-elevation vibration sources. - After MUSTANG arrives at UPenn it will be cooled and tested at low elevation angles to validate the theory that the vibrational issues are a result of the telescope environment and not being induced from within MUSTANG. Upon completion of these tests a course of action to lessen these vibrational effects will be discussed with the team.

• Two drops of YGOR manager code releases and testing on MUSTANG over the summer (inc. housekeeping) – Over the course of the summer, there will be two drops of YGOR code updates that require testing on MUSTANG while it is cold. The team believes accommodating these tests with MUSTANG at UPenn should not be a problem if advance notice is provided and testing schedules met. An immediate effort to evaluate the issues surrounding VNC access between Green Bank and UPenn should begin to assure that networking/connectivity issues are resolved and the path is robust before the first round of testing. The upgrade to a real-time version of Linux has been deferred for now, but if time allows could be added in later in the summer.

• Cryogenics- install better hoses between the turret and the cryostat – GB engineers will research the location of the refrigerant leak. While at UPenn new hoses will be installed between the turret and the cryostat with improved routing (angle connections).

• Review and analysis of the air circulation issues/doughnut air circulation improvement – There is general consensus that improved air circulation around the circuit boards is advisable although the insulation added in December appears to have mitigated the problems in the tower. When MUSTANG is removed in April a plan will be discussed to add circulation and potentially a fan to move air through the doughnut. All agree a minimum of alterations to the housing is a major consideration in the plan. A plan for improved grounding of the tower will be created concurrent with this work.

• Separate tower and crate into separate remotely controllable power supplies – Before MUSTANG leave GB, a set of specifications for a new dual power supply will be created. Having a separate supply for the crate and tower allows for independent rests if an error condition arises; a feature that would have been useful in the concluding observation season. GB engineers will procure and build the supply in consultation with UPenn and ship the completed, tested unit to UPenn for installation at a time of UPenn’s choosing. The existing supply will be brought to GB in the fall along with MUSTANG and added to the spares pool for the instrument.

• Procedures & requirements for operating the pulse tube cooler on the GBT (eg, our special compressor; cold trapping the lines prior to the run etc) – This work is underway by GB engineering and will continue into the spring.

• Procedures for maintaining the fibers in the wrap that we use to get data to the DAQ computer. – Procedures have been added after the problems that occurred in November with MUSTANG installation. If subsequent installations prove difficult or unreliable, a revisiting of the fiber cabling strategy will be conducted.

• Define failure scenarios and resolution roles and strategies – The team will meet the week MUSTANG is removed to identify the potential failure modes, single points of failure, and strategies for recovery in the event of those failures.

• DAQ computer installation & removal procedures; receiver installation and removal. – The existing computer occupies the space needed for holography receiver components. GB will procure a replacement computer in a smaller form factor (2-3 RU) so that it can be installed inside the crate. This moves the computer into the RFI protection of the crate and simplifies the interconnecting cabling. Special consideration is needed for the GPIB and PCI boards. GB Software engineers will need to participate in the loading and configuration, and testing of the new computer. The existing computer (Spider) will be shipped to GB along with MUSTANG in the fall and remain available as a test bed and emergency backup computer to MUSTANG.

• Inventory of spares – List has been compiled and spares are in place with the exception of some new cables, flexlines, and a second PROM. UPenn team is working on obtaining these items. A spare power card with improved front connections is desired and is in process.

• Documentation – There remain some substantial ‘holes’ in the documentation of MUSTANG available to the GB engineers; particularly the firmware and tower details. Team members will continue to post documents to http://wiki.gb.nrao.edu/bin/view/Pennarray/MustangDocumentRepository and GB team will ultimately move them into the GB documentation and drawing archives.

• Date MUSTANG arrives at GB ready for installation – approx 15-Oct-08 (Delivery of Array #4 could cause a delay in having MUSTANG ready for the fall season.)

-- MartyBloss - 13 Mar 2008