NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Main > TWikiUsers > RichardPrestage

Users | Groups | Offices | Changes | Index | Contents | Notify | Search | Jump to Topic:

GBT Lessons Learnt

Overview

I'd be extremely interested in any comments on this revised 2006 version; many thanks to those who have made some already. Please feel free to comment/correct content or style (of cause, I can always change it back later

Lessons Learned - or just identified? - 2006 version

The GBT saw first light on August 22nd, 2000; after some early commissioning activities the subsequent twelve months were almost completely consumed by remedial construction activity. Significant commissioning activities re-commenced in July 2001 and the first regular science observations starting in August 2001. Routine operations at K-band (20 GHz) commenced in fall 2003. The three years it took the GBT go from first light to substantially complete, routine operation is I believe fairly typical of a large, national facility, although I have no firm evidence for this belief. Along the way, we have learned many lessons. The 2002 paper provided an earlier summary some of these, the list has been revised and expanded here.

The first three lessons may be particularly appropriate for future telescopes, such a extremely large optical telescopes, which may be based on existing designs for smaller facilities, many thousands of identical components, or have very complex systems.

Be vary wary of scaling up existing designs

Copying and downscaling an existing structure or design may be practical, provided that the structure really is smaller, or has lower loads, than the original. Copying and upscaling, or extrapolating, a structure without a thorough analysis is risky. Upscaling a structure by 10%, with appropriate analyses, may be reasonable. The design limitations or shortcomings of the original structure should be fully investigated before upscaling. If extrapolating a design results in a 100%, or even 50%, increase in loads on components, you would be well served to abandon the original design, and start over with a clean piece of paper.

Pay attention to fabrication, hardware prototyping and testing

Maintain a close connection with fabricating shops and their suppliers, and perform frequent inspections. You will gain an appreciation of the characteristics of the parts, which may affect performance of the telescope in ways which would not otherwise be caught. Communication with staff "on the shop floor" helps significantly with interpretation of specifications, and avoids guesswork.

As far as possible, prototype all critical components, and perform full-life testing on the resulting designs selected for production. For example, this approach caught numerous problems with the GBT actuators, which are replicated thousands of times.

Be very aware of system complexity

The GBT is an extremely flexible and versatile system, and this has already had huge scientific payoff. Some of the most exciting scientific output has come from using the equipment in ways that were never anticipated when the telescope was first designed. However, such complexity comes at a price. In the early days of commissioning the staff were almost overwhelmed by the shear number of different hardware configurations and observing modes which needed to be debugged. One reason why the GBT metrology system as originally conceived has been abandoned is that the system integration issues associated keeping the complete system running were prohibitively expensive.

As our ambitious become every larger, system complexity increases exponentially. The lesson learned is to stive to ensure that a complex system can be brought up in stages. Ensure that your extremely flexible correlator has a few simple modes that can be easily debugged and brought into useful operation without requiring a thorough checkout of the 999 other modes. However, design the system such that testing one correlator mode will largely validate the 100 similar associated modes, and so on.

Leave some flexibility and agility in your project plan

Don't have everyone working so flat out that they cannot respond to innovative ideas. This prevents either not responding, or having something done on such a shoe-string that it takes forver, needs major rework, etc.

Perform software prototyping wherever possible

Use of M&C system on 140ft was great, Tests of aips++ using 140ft data were not performed.

Provide multiple but consistent views into the system

CLEO versus GO versus Astrid, etc.

Adopt a two-tier release process

Describe Nicole's idea.

Prototypes will remain in the system forever

It was noted above that prototyping was a good thing. However, schedule slip will result in pressure to use the prototype system as the "real" system. Once the telescope is in use, it will be almost (although not completely) impossible to remove the protoype from operation; other software will be built to accomodate, or take advantage of idiosyncrasies of the prototype, and these will become embedded in the overall system. The lesson here then is that you should always be satisfied with the architecture and engineering of the prototype. It should be a prototype only in the sense that it provides limited functionality, not that it has been designed or implemented in a slipshod fashion.

Plan to replace one sub-system before all the rest are fully in use

The time between initial groundbreaking and routine operation of a large telescope can easily be ten years. In the current environment, parts which are fabricated and installed early in the process can become obsolete before they are in routine use. In addition, for many telescopes it seems to be the case that at least one sub-system does not live up to expectations, and has to be replaced. In the case of the GBT for example, we are having to perform a complete retrofit of the azimuth track. Be prepared for this to be the case for your telescope.

Pay sufficient attention to software

Both in the construction of the telescope itself, but also for subsequent instrumentation development, it has often been the case that some aspect of the hardware design was considered the challenging and exciting problem; a common phrase was that the rest is "just" software. However, software won't just “take care of itself”. It can often be on the critical path; the challenge of delivering effective and elegant software is every bit as real as that of developing challenging hardware solutions. Sound project and technical management of the software systems is as important as for the mechanical and electrical/electronic components.

Have some clearly defined observing modes fully implemented from day one

As soon as a telescope is in operation, the pressure to start doing science will be immense. In most cases, there will be some obvious area which can immediately be tackled; in the case of the GBT, the clean beam of the antenna allowed significant new HI science to be performed using instrumentation which was extremely simple to commission. However these observations, which should have been absolutely trivial to perform, were significantly complicated by difficult to use and incomplete observation control and data analysis software. The lesson is to have a bread-and-butter observing mode completely ready to go, including all aspects of hardware, control software and data reduction facilities. Being able to demonstrate early results with this combination, and being able to fall back on this to make good use of the telescope when more complex observing modes have run into problems, will be a great stress reliever for your staff.

Minimize the number of items to commission at any one time

For example, use software that has been well tested on other telescopes if possible. The experience of commissioning the GBT and DISH at the same time was not good. This worked towards our advantage with the receivers. Some of them had been tested and used on the 140 Foot telescope so when we ran into spectral baseline problems we had some experience with at least part of the system.

Be prepared for the lengthy commissioning process...

Having a few experienced staff on your team who have commissioned similar telescopes in the past can hugely speed up the commissioning process. They can diagnose subtle symptoms and problems which may have less experienced staff puzzled for months. On the other hand, no matter what people tell you about commissioning a telescope there are some things that you have to learn from experience. It will just be hard work and take time no matter how well prepared you happen to be. This is similar to trying to tell young people about some fact of life. Even though they understand your words they really do not understand until they experience it. I guess this is life.

Lessons Learnt - 2002 version

Overall, the GBT is performing extremely well, with the basic antenna performing better than we had expected for the development stage we have currently reached. On the other hand, the system integration and commissioning process is proceeding far more slowly than had originally, perhaps optimistically, been hoped. In hindsight, most of our problems (apart from mechanical issues with the antenna track) have been rather mundane, and might have been predicted in advance; nevertheless it is perhaps worth noting the main issues here, in the hope that this might be of some use to future telescope developers.

• The GBT is a highly ambitious project. Although the offset geometry, active surface, and other advanced design features added a great deal of complexity to the project, the benefits of these features are already apparent and very significant, and have fully justified the design and construction effort required.

• The track problem itself may be of interest to other large telescope projects. The fact that there is some uncertainty about the physics and engineering of our problem suggests that there is scope for more R&D in this field.

• To get the desired performance, many parts of our 100m structure need to be aligned to ~ 100μm. Having a strong metrology group has been crucial to acceptance testing and diagnosing and resolving telescope problems.

• Have a robust, reliable workhorse scientific receiver/backend combination, which is available from before day one.

• Software won't just “take care of itself”. It can often be on the critical path. Sound project and technical management of the software systems is as important as for the mechanical and electrical/electronic components.

• Plan for in detail and budget sufficient time for system integration tests, and ensure that you have an adequate number of commissioning scientists, with the required skills and expertise.

Left over random thoughts

• The GBT is fantastic! We should look upon our glass as overwhelmingly half full. But our operational style concentrates almost exclusively on faults and problems. There must be a better way, what is it?

• GBT is highly flexible, and that is good, but comes at a price - what is the lesson here?

• Had to start repairing the track even before we started operations, i.e. should plan that at least some component of your telescope will be ready for replacement, or simply didn't work as you expected, even before you are fully on the air.

• What lessons can we really learn from the pulsar spigot card? Not clear to me that if we had finished the hardware earlier, we would have had the infrastructure (disk space, computing staff, observing scripts) to fully support it any way. Or would we? Is the lesson just for people like me to accept things that are more chaotic than I would like, and for those who would like to drive faster to accept there are sometimes good reasons for a more methodical approach?

• Spigot card is a fascinating one, because it has produced some of the most high profile science done by the GBT. What if the double binary pulsar hadn't been discovered by Parkes? What if it had, but the spigot didn't exist? Same for Scott's Terzan 5 stuff....

• Isn't the real lesson from the spigot card just that you mustn't so overcommit your staff to the identified tasks that they have no time to respond to small, innovative, agile projects?

• Took five years from first light to stable operations. This is normal, I think. Can I find any metrics on this?

• University collaborations are good, but expensive in many ways....

• Metrology group was too disconnected from actually demonstrating real, usable capabilities on the real telescope. The approach of targetting the final performance with no intermediate milestones was too ambitious, and failed.

• Once you have an implementation (e.g. the antenna manager) in your operating system, it takes years to make changes to it. The point I'm getting at here is that the "division of concerns" was not honored, and now it is almost impossible to go back!

• Prototypes stick around for years, so you'd better be pretty happy with them...

• One stream of information to software development resources means you get functionality delivered to users faster! Work on building consensus and making formal, official work requests to ensure that the proper solutions are delivered at the right times

• Two-tiered software releases are useful (the first to the scientific support staff, so they can train themselves and identify how to train users, and the second to users - this ensures that releases to the user community are co-managed by software and science resources)

• Don't implement new tools without a long-term maintenance plan (e.g. GO, CLEO, glish, operators logs)

AstridTalkRMP.ppt: draft talk on Astrid

Miscellaneous List of Things To Do

Stuff that TWiki created for me

• Name: Richard Prestage
• Login Name: rprestag
• Email: rprestag@nrao.edu
• Phone: (304)-456-2222
• Department: Green Bank
• Location: GreenBankWV
• Comment:

Personal Preferences (details in TWikiVariables)

• Set RELEASEEDITLOCKCHECKBOX = checked="checked"
• Horizontal size of text edit box:
  • Set EDITBOXWIDTH = 70
• Vertical size of text edit box:
  • Set EDITBOXHEIGHT = 30
• Style of text edit box. width: 99% for full window width (default), width: auto to disable.
  • Set EDITBOXSTYLE = width: 99%
• Optionally write protect your home page: (set it to your WikiName)
  • Set ALLOWTOPICCHANGE =

Related topics

• TWikiPreferences has site-level preferences of GB.
• WebPreferences has preferences of the GB.Main web.
• TWikiUsers has a list of other TWiki users.

• talkSummary.pdf:

• LargeProposalNotes.pdf: draft notes on large proposals

Attachment:	Action:	Size:	Date:	Who:	Comment:
AstridTalkRMP.ppt	action	672768	14 Sep 2005 - 22:20	RichardPrestage	draft talk on AStrid
eir1.JPG	action	1718982	18 Jan 2006 - 21:50	RichardPrestage
eir2.JPG	action	3486777	18 Jan 2006 - 21:51	RichardPrestage
eir3.JPG	action	3866728	18 Jan 2006 - 21:51	RichardPrestage
eir4.JPG	action	3632267	18 Jan 2006 - 21:51	RichardPrestage
talkSummary.pdf	action	82896	25 May 2006 - 14:01	RichardPrestage
LargeProposalNotes.pdf	action	28902	09 Jan 2007 - 17:21	RichardPrestage	draft notes on large proposals