NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Dynamic > JCMTMeetingNotes

Changes | Index | Search | Go

Meeting Notes

• JCMT Telescope Overview
• JCMT Dynamic Scheduling Overview
• JCMT Dynamic Scheduling Lessons
• JCMT Dynamic Scheduling Details
• JCMT Decision Process
• JCMT Software Suite
• JCMT Authentication Issues
• Other Notes

JCMT Telescope Overview

• All instruments are typically on the telescope at all times, unless one is broken
• Observations are night time only - maintenence occurs duringthe day, except in the case of a default
• There is always an observer and telescope operator present during the observing
• They have only two proposal deadline, with proposals due approximately 3 months before the semester starts

JCMT Dynamic Scheduling Overview

• The observer must run the projects which are appropriate for the current weather queue - if the weather is right for their own project, then it can be run
• System is truly dynamic, and there is no need to stick to a given project for any length of time
• Observers get notified as soon as the observations are done, and must give notice within 1 day of observations if there is any problem not caught by the data quality checks
• Telescope operators were given a new title, new job description, and more money.
• Telescope operators make judgement calls as to when calibration and pointing observations should be done
• Proposals are ranked purely by the TAC ranking, and usually the top ranked proposals get most/all of their time completed, but not always (e.g. if a proposal needs good weather and has a very limited LST range it might not get all its time). Proposals are not typically carried over from one semester to the nect - if a proposal is not completed, too bad. The exceptions to this are specially marked proposals and monitoring projects. There are moved by hand to the next term's queue. (Any calibration which is specific to a project, e.g. if they have a certain source needed to be observed, or a non-standard method of calirbating, still gets charged to that project.

JCMT Dynamic Scheduling Lessons

• Bulk of the work for them was observer feedback and support work:
  • Data quality checks (done automatically and by telescope operators) was single most difficult task
  • Another difficult task is getting general information about observations to the PI/CoIs (e.g. " Why did you observe in this way?")
  • Going to a system with no PI/CoI involvement affects every aspect of the telescope
• Weather changes there fast enough that "precalculating" the schedule is not efficient for them
• They pointout that we must insure our data reduciton system is sufficient for PI/CoIs to quickly look at the data. Also, getting a good data reduction pipeline id vital for getting data reduced quickly and papers published (increases publication rates)
• Metadata was the easiest part of their system
• They beleive we will get 90% of the benefits of true dynamic scheduling with our Stage II proposal
• Projects with many SBs are complicated in that it can be difficult to make things efficient in terms of telescope time, and also to insure that sensible sky sections are observed at the same time. The best way of dealing with that (for JCMT) is to have the PI/CoIs only load some SBs at a time
• Calibrations and pointings get split amoungst all projects, with the time allocated based on the amount of time each project that uses the given calibration had. But, they would prefer the calibration time get charged to something called "calibration time" rather than a given project. this would keep people fromtrying to scimp on their calibration times, insuring the data when archived has good calibration. This is important, as many of the JCMT papers published are from the archives.
• SBs are built up primarily from templates, which significantly has reduced mistakes

JCMT Dynamic Scheduling Details

• Metadata for each SB has a repeat table. Repeat number is incremented down once something is run and accepted, but the obsrver or operator can re-increase
• For monitoring projects the program automatically increments the time after the completion of a successful SB
• Target of Oppotunity (TO) projects win over all others
• Engineering/test projects that need to be run are treated like any other SB
• The program is smart enough to worry about time - e.g. if a SB has two sources, one is up now and to be observed for two hours and the other is up in two hours, the program understanda this and lets the SB be run now.
• Program Returns SBs sorted by project, with only the top few SBs of each project displayed. This way (1) if a project has 100s of SBs it doesn't dominate the list, and (2) displaying too many SBs can take a very long time, which should be avoided.
• Biggest current problem is with files locks - the lock out the system anytime someone is uploading a file or the dyn. scheduling program is reading it. They need a 'smarter' system that only locks things out when a project of interest is uploading a file.
• Algorithm can even be run within a given project, to choose the best SB to be scheduled.
• From all SBs returned, they are ranked only by TAC rating
• Once a SB is run, the question is aksed "Do you want to accept or reject this?" If you accept, the time and repeats is incremente. If you say reject, you must exaplain why (and the time and repeats do not get incremented)
• At the end of the night the observer approve the summary of times charged to the projects, At this point the observer can also assign reasons for time gaps (e.g. weather, project problems, etc)
• Query tool has option to forget about time constraints in case of, e.g. lots of bad weather and a shortage of bad weatehr proposals in the queue. In this case, a project may run for more than 100% of its allocated time. Similarly, somone can call up proposals from teh previous semester to fill time.

JCMT Decision Process

• Queries use xml, mostly becaue individual programs can then be written in almost any language and can output to xml
• Server seperates xml into two things - "static" and "astrometric" information
  • "Static" information includes, e.g. desired value of tau, proposal id, time left on project, # repeats, any relevant date information, country queue, waveband. This information is entered into an SQL database
  • "Astrometric" information includes, e.g., is the source up? above correct elevation? is the seeing correct? (anything that make the source observable). This information also includes checks that the SB length will allow it to be completed. This is run just as a filtering code
• Static SQL database is first searched, and then filter program is run on astrometric information
• SQL search takes 5s when run correctly (up to 60s otherwise). Astrometric filter takes up to 30-60s
• Currently they are using psybase (?) logic for SQL database

JCMT Software Suite

1. Observing Tool
   • Lets observers construct SBs and upload them to the database
   • Basically builds the SBs for the observer, much like the long-planned top level gui for astrid
   • Its flexible so that investigaors do not need to know the details of the hardware. E.g. it will pick the best beam if more than one is available, so that the investigator won't say "Two years ago Beam 1 was best, so I'll use it again". Instead the system is smart and flexible enough to know which beam is best.
   • Includes a survey tool which allows the user to readily make 1,000s of SBs which are identical except for, e.g. the source name
   • Uses xml
   • Publicly distributed
   • Must be easy to use
2. Query Tool
   • Browese the databases to choose the best SBs to be run
   • Not publicly distributed, but available locally
3. Observing Process Tools
   1. Obs log: allows comments to be added to teh database during observations, and also flags gaps inobserving and allows observer/operator to add explanations
   2. Time Accounting Tool
      • Run at the end of each night
      • Gives summary of night's observations
      • Needed in part because observing tool time estimates are not 100% correct
   3. Scripts run by support staff
      • uploads datafiles
      • emails investigators that data was observed
   4. Web feedback form
      • For investigators to know what's going on
      • Also allows comments between investigators and project friend
      • Important, as project friend may not be around when project is run. This allows all communications between them to be accessible by anyone
      • Also allows investigators to give science level description of the project, which allows project friend to insure Sbs makes sense

JCMT Authentication Issues

• Use project level authentication rather than user authentication, which eliminates worries about Co-investigators being on more than one project with different PIs
• Knows all PI/Cois of a project, so if, e.g. the password is lost, everyone gets the new one
• Has a backdoor so that staff can access all projects
• Co-investigators can opt out of getting emails and information about project here
• Enforces accoutnability
• Don't use authentication for query system as it can only be run internally

Other Notes

• AmysJCMTNotes

-- KarenONeil - 01 Jun 2006