NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Main > TWikiUsers > RichardPrestage
   Users | Groups | Offices | Changes | Index | Contents | Notify | Search | Jump to Topic:

GBT Lessons Learnt

Overview

I've been invited to give a talk on the GBT at the May 2006 SPIE. I need to get the talk finished by 21st April. Phil and I have been alternating on this since 2000, and Jay and Paul before that, so this will be the 5th GBT overview talk, given 8 years since Jay's initial overview talk. I've been explictly asked to report on "lessons learnt". I was also asked to do that in 2002; may of the lessons learnt are still the same, and I append below what I said then.

I'd be extremely interested in any comments on this revised 2006 version; many thanks to those who have made some already. Please feel free to comment/correct content or style (of cause, I can always change it back later smile


Lessons Learned - or just identified? - 2006 version

The GBT saw first light on August 22nd, 2000; after some early commissioning activities the subsequent twelve months were almost completely consumed by remedial construction activity. Significant commissioning activities re-commenced in July 2001 and the first regular science observations starting in August 2001. Routine operations at K-band (20 GHz) commenced in fall 2003. The three years it took the GBT go from first light to substantially complete, routine operation is I believe fairly typical of a large, national facility, although I have no firm evidence for this belief. Along the way, we have learned many lessons. The 2002 paper provided an earlier summary some of these, the list has been revised and expanded here.

The first three lessons may be particularly appropriate for future telescopes, such a extremely large optical telescopes, which may be based on existing designs for smaller facilities, many thousands of identical components, or have very complex systems.

Be vary wary of scaling up existing designs

Copying and downscaling an existing structure or design may be practical, provided that the structure really is smaller, or has lower loads, than the original. Copying and upscaling, or extrapolating, a structure without a thorough analysis is risky. Upscaling a structure by 10%, with appropriate analyses, may be reasonable. The design limitations or shortcomings of the original structure should be fully investigated before upscaling. If extrapolating a design results in a 100%, or even 50%, increase in loads on components, you would be well served to abandon the original design, and start over with a clean piece of paper.

Pay attention to fabrication, hardware prototyping and testing

Maintain a close connection with fabricating shops and their suppliers, and perform frequent inspections. You will gain an appreciation of the characteristics of the parts, which may affect performance of the telescope in ways which would not otherwise be caught. Communication with staff "on the shop floor" helps significantly with interpretation of specifications, and avoids guesswork.

As far as possible, prototype all critical components, and perform full-life testing on the resulting designs selected for production. For example, this approach caught numerous problems with the GBT actuators, which are replicated thousands of times.

Be very aware of system complexity

The GBT is an extremely flexible and versatile system, and this has already had huge scientific payoff. Some of the most exciting scientific output has come from using the equipment in ways that were never anticipated when the telescope was first designed. However, such complexity comes at a price. In the early days of commissioning the staff were almost overwhelmed by the shear number of different hardware configurations and observing modes which needed to be debugged. One reason why the GBT metrology system as originally conceived has been abandoned is that the system integration issues associated keeping the complete system running were prohibitively expensive.

As our ambitious become every larger, system complexity increases exponentially. The lesson learned is to stive to ensure that a complex system can be brought up in stages. Ensure that your extremely flexible correlator has a few simple modes that can be easily debugged and brought into useful operation without requiring a thorough checkout of the 999 other modes. However, design the system such that testing one correlator mode will largely validate the 100 similar associated modes, and so on.

Leave some flexibility and agility in your project plan

Don't have everyone working so flat out that they cannot respond to innovative ideas. This prevents either not responding, or having something done on such a shoe-string that it takes forver, needs major rework, etc.

Perform software prototyping wherever possible

Use of M&C system on 140ft was great, Tests of aips++ using 140ft data were not performed.

Provide multiple but consistent views into the system

CLEO versus GO versus Astrid, etc.

Adopt a two-tier release process

Describe Nicole's idea.

Prototypes will remain in the system forever

It was noted above that prototyping was a good thing. However, schedule slip will result in pressure to use the prototype system as the "real" system. Once the telescope is in use, it will be almost (although not completely) impossible to remove the protoype from operation; other software will be built to accomodate, or take advantage of idiosyncrasies of the prototype, and these will become embedded in the overall system. The lesson here then is that you should always be satisfied with the architecture and engineering of the prototype. It should be a prototype only in the sense that it provides limited functionality, not that it has been designed or implemented in a slipshod fashion.

Plan to replace one sub-system before all the rest are fully in use

The time between initial groundbreaking and routine operation of a large telescope can easily be ten years. In the current environment, parts which are fabricated and installed early in the process can become obsolete before they are in routine use. In addition, for many telescopes it seems to be the case that at least one sub-system does not live up to expectations, and has to be replaced. In the case of the GBT for example, we are having to perform a complete retrofit of the azimuth track. Be prepared for this to be the case for your telescope.

Pay sufficient attention to software

Both in the construction of the telescope itself, but also for subsequent instrumentation development, it has often been the case that some aspect of the hardware design was considered the challenging and exciting problem; a common phrase was that the rest is "just" software. However, software won't just “take care of itself”. It can often be on the critical path; the challenge of delivering effective and elegant software is every bit as real as that of developing challenging hardware solutions. Sound project and technical management of the software systems is as important as for the mechanical and electrical/electronic components.

Have some clearly defined observing modes fully implemented from day one

As soon as a telescope is in operation, the pressure to start doing science will be immense. In most cases, there will be some obvious area which can immediately be tackled; in the case of the GBT, the clean beam of the antenna allowed significant new HI science to be performed using instrumentation which was extremely simple to commission. However these observations, which should have been absolutely trivial to perform, were significantly complicated by difficult to use and incomplete observation control and data analysis software. The lesson is to have a bread-and-butter observing mode completely ready to go, including all aspects of hardware, control software and data reduction facilities. Being able to demonstrate early results with this combination, and being able to fall back on this to make good use of the telescope when more complex observing modes have run into problems, will be a great stress reliever for your staff.

Minimize the number of items to commission at any one time

For example, use software that has been well tested on other telescopes if possible. The experience of commissioning the GBT and DISH at the same time was not good. This worked towards our advantage with the receivers. Some of them had been tested and used on the 140 Foot telescope so when we ran into spectral baseline problems we had some experience with at least part of the system.

Be prepared for the lengthy commissioning process...

Having a few experienced staff on your team who have commissioned similar telescopes in the past can hugely speed up the commissioning process. They can diagnose subtle symptoms and problems which may have less experienced staff puzzled for months. On the other hand, no matter what people tell you about commissioning a telescope there are some things that you have to learn from experience. It will just be hard work and take time no matter how well prepared you happen to be. This is similar to trying to tell young people about some fact of life. Even though they understand your words they really do not understand until they experience it. I guess this is life.


Lessons Learnt - 2002 version

Overall, the GBT is performing extremely well, with the basic antenna performing better than we had expected for the development stage we have currently reached. On the other hand, the system integration and commissioning process is proceeding far more slowly than had originally, perhaps optimistically, been hoped. In hindsight, most of our problems (apart from mechanical issues with the antenna track) have been rather mundane, and might have been predicted in advance; nevertheless it is perhaps worth noting the main issues here, in the hope that this might be of some use to future telescope developers.


Left over random thoughts

Currently just random thoughts... Add yours here!


AstridTalkRMP.ppt: draft talk on Astrid

Miscellaneous List of Things To Do

Stuff that TWiki created for me

Personal Preferences (details in TWikiVariables)

Related topics

Attachment: sort Action: Size: Date: Who: Comment:
AstridTalkRMP.ppt action 672768 14 Sep 2005 - 22:20 RichardPrestage draft talk on AStrid
eir1.JPG action 1718982 18 Jan 2006 - 21:50 RichardPrestage  
eir2.JPG action 3486777 18 Jan 2006 - 21:51 RichardPrestage  
eir3.JPG action 3866728 18 Jan 2006 - 21:51 RichardPrestage  
eir4.JPG action 3632267 18 Jan 2006 - 21:51 RichardPrestage  
talkSummary.pdf action 82896 25 May 2006 - 14:01 RichardPrestage  
LargeProposalNotes.pdf action 28902 09 Jan 2007 - 17:21 RichardPrestage draft notes on large proposals

Topic RichardPrestage . { Edit | Attach | Ref-By | Printable | Diffs | r1.64 | > | r1.63 | > | r1.62 | More }
Revision r1.64 - 05 Jul 2007 - 20:23 GMT - RichardPrestage
Parents: TWikiUsers
Content copyright © 1999-2007 by the contributing authors.
All material on this collaboration platform is the property of the contributing authors.