NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Dynamic > DSSSoftwareSuite

Changes | Index | Search | Go

Dynamic Scheduling Software Suite

• Overview
• Core Software Functionality
  • Project Management
  • Session Management
    • Session Metadata
    • Session Verification
  • Contact Management
    • Static Observer Information
    • Transient Observer Information
    • Calendars
  • Scheduling
    • Session Scheduling
    • Session Probability Forecast
  • Time Accounting Tool
  • Weather Information
    • Current Weather
    • Predicted Weather
  • RFI
  • Past Telescope Schedule(s)
  • M&C Configuration Checker Daemon
  • E-mail Daemon
  • VLB Schedule Interface
• Support Software Functionality
  • Interface to the Proposal Submission Tool Database
  • Proposal Handling Tool
  • Replacement program(s) for Current Telescope Scheduler Support Tools

Overview

There are four groups of users of the Dynamic Scheduling Software Suite: observers, support staff, GBT operators, and the Telescope Scheduler. Each of these user groups will require interfaces into the system. Many of the interfaces overlap and only differ in terms of the level of access provided to the project data and the details of this are described along with the SchedulingData. A high-level diagram depicting the overlapping regions of interest for each user group is shown below. The specifics of each piece of functionality are described later in this document.

Some interfaces are reserved for special use by the GBT operators and the Telescope Scheduler. GBT operators and the Telescope Scheduler must have access to the scheduling algorithm and the time accounting tool. Additionally, the Telescope Scheduler has access to proposal handling, project handling, and VLB scheduling functionality. The specifics of each piece of functionality are described later in this document.

The code base for the Dynamic Scheduling Software Suite is comprised of more than just user interfaces. A good bit of code is devoted to "under the hood" functionality. Some of the "under the hood" functions have user interfaces associated with them and some do not.

Currently, many of the components which comprise the software suite are not yet under development. We have started developing the following components: Project Management, Session Management, Static Observer Information, and Session Scheduling.

Core Software Functionality

Project Management

In short, a project is the scientific proposal, with its metadata, which is submitted through the Proposal Submission Tool and which then may eventually be allocated telescope time. Additionally, the DSS must also support maintenance. As a simplification, maintenance time is treated as a project in the DSS and throughout this document. The primary difference between scientific project and maintenance project is the maintenance project data if not entered through the PST but instead through software provided by the DSS.

All groups of users with scheduled projects have web access with differing permissions to their project metadata. Access to project information will be administrated through the TurboGears authentication mechanism. Observers will visit a project web page to view their project metadata. Observers will also get feedback about the execution state of their science program (in terms of percentage complete), how their observations fit into the schedule, the execution state of individual sessions within their projects, and contact information for their project's friend via their project web page. The details of access for project metadata are discussed and outlined in the context of the SchedulingData. Also, the ProjectsPage provides an excellent outline of the metadata associated with each project as well as groupings of like information. The groupings on the ProjectsPage are being used directly in the implementation of the user interface to project management. In summary, the groupings are as follows: project data, project observation history, and project communication history. The "Session Information" on the ProjectsPage is discussed below in the Session Metadata section of this document; sessions are contained within projects.

Project metadata originates from both the Proposal Submission Tool (PST) database and the Telescope Scheduler. The PST contains items such as the project names, indicators tying a project to thesis work, and the maxiumum hours projects can be allocated in any given semester. The Telescope Scheduler provides project science grades and allocated times for each science grade. Based upon known project metadata, we have created the following schema for the portion of the Dynamic Scheduling database devoted to storing project metata.

Session Management

All projects are broken into one or more sessions, where each session is uniquely defined by a given set of criteria (e.g. LST/UT, weather, hardware). All observations which an investigator wishes to pursue for the project must be contained within one or more sessions. And it is the time for these individual sessions which the Dynamic Scheduling System will actually schedule.

Session Metadata

As with project management, all groups of users have access in one form or another to session metadata. The details of access for session metadata are discussed and outlined in the context of the SchedulingData. Also, the ProjectsPage (see "Session Information") provides an excellent outline of the metadata associated with each session in a project. The listing of session metadata on the ProjectsPage is being used directly in the implementation of the user interface to session management.

Currently, it is unclear who or what will be creating the sessions for a particular project. There are three proposals for session creation currently on the table and the final decision will be made by Fall 2007. The implementation of session metadata access will be directly influenced by the final decision among these three proposals or perhaps some future fourth proposal.

• Proposal 1 - Have the Proposal Submission Tool gather the information needed to automatically create sessions on the observer's behalf. With this proposal, session verification can be done at the interface level, i.e. verify the information as it is entered.
• Proposal 2 - Impose a minimum data entry requirement on the Proposal Submission Tool and have the observer (after a project is accepted) enter the information needed to automatically create sessions. With this proposal, session verification can be done by the session creation software, i.e. verify the information before creating the session.
• Proposal 3 - This proposal is similar to proposal 2 but differs in how sessions are created. This proposal would impose a minimum data entry requirement on the Proposal Submission Tool and have the observer create (after a project is accepted) their own sessions. With this proposal, session verification can be done at the interface level, i.e. verify the information as it is entered.

Based upon known session metadata, we have created the following schema for the portion of the Dynamic Scheduling database devoted to storing session metadata. Note that the project metadata shown above is also shown here to highlight the relationship between projects and sessions.

Session Verification

Session verification is required to ensure that the metadata for a particular session is correct and consistent. All data required by the DSS for scheduling purposes should be filled in by the project investigators. If any data is missing, the session is not runnable and the user will be notified.

Contact Management

Because the GBT Dynamic Scheduling System relies on the observer to conduct observations, observer contact information is a crucial component of the software system. There are three types of contact information: (1) static contact information, (2) transient contact information, and (3) observer availability, i.e. calendars.

Static contact information refers to items which change infrequently, e.g. an observer's permanent address. Transient contact information refers to items with a known and relatively short-lived duration, e.g. an observer's hotel phone number for a week-long conference. Both types of contact information are required in order to ensure that dynamic scheduling works well and the Telescope Scheduler and GBT operators can contact observers as part of their administrative duties.

Calendars indicate when the observer is not available. In other words, the calendars for the Dynamic Scheduling System are black-out calendars and indicate when the observer cannot conduct observations. We recognize that the success of this project rests largely on our ability to engage and inform our users at each stage of dynamic scheduling. We will provide observers with the utmost flexibility when updating their availability calendars. These calendars will be examined when choosing candidate sessions for each Telescope Period, allowing observers to continually make changes to their blackout dates.

The Dynamic Scheduling software combines static contact information, transient contact information, and calendars together to provide an easy to read display which shows not only the raw information, but also indicates who should be contacted first for a particular session and indicate if possibly more than one observer will be available for the given Telescope Period.

Static Observer Information

The static observer information database schema directly uses the NRAO-wide Users Database. The schema allows for multiple mailing addresses, phone numbers, and e-mail addresses to reference a single record in the person and organization tables.

Transient Observer Information

The transient observer information is also contained within the NRAO-wide Users Database schema, but is flagged as transient by associating dates to entries, i.e. dates to an observer's phone number entry. The person and phone tables shown below are part of the NRAO-wide Users Database and we will use that information "as is." The other tables will be "value added" tables which are part of the Dynamic Scheduling database only.

Calendars

The schema and design of calendars is under development. Nominally, the schema for the calendar information could be a single table which would contain a foreign key to the person table among its fields. The calendar software will display observers who should/should not be contacted on certain days/times and how to contact them, along with an order for whom should be contacted first if more than one observer will be available

Scheduling

Thirty minutes before the end of a given Telescope Period (TP), the operators will query the Dynamic Scheduling Software (DSS) to determine what the next session should be. Both the GBT operators and the Telescope Scheduler must have access to scheduling functionality. All users have access to session scheduling forecasts.

Session Scheduling

The basic rules which the DSS will follow are described completely in the SelectionRules, but are summarized here. All sessions are examined to see what can be scheduled based off the current time (LST/UT), available hardware, observer availability (if no observer is available, a project cannot run), and to insure there is still allocated time left for the project/session. Also, the scheduling software determines if there are any fixed time or window projects which must be run at this time. If there are, then the fixed time or window project is chosen. If not, the weather and time of day (LST/UT) are examined to eliminate all sessions for which the weather is not good enough for the session to be scheduled, and which the current time is inappropriate. Once these elimination rules are applied, the remaining sessions are ranked based on a number of criteria (the most important of which are observing efficiency and stringency). The telescope operator is presented with a ranked list of sessions from which to choose the next project/session to be observed.

Because the GBT operator makes the final decision on the session which will be run during the current telescope period, it is crucial that the ranked results of the current scheduler program be presented to the operator in a lucid form that will facilitate the operator's decision. All sessions that can scheduled at the chosen LST should be displayed ranked with auxiliary information such as LST constraints, min and max run times, day/night transitions, ranking coefficients as well as other session data as needed, e.g. observer on site. As part of the visualization, the GBT operator may choose to overlay these candidate sessions with a "future" look. For example, the GBT operator may wish to overlay the sessions that can scheduled now, with sessions that can scheduled in 2 hours with different weather conditions. Such a future look will enable the GBT operator to properly determine the appropriate telescope period. In addition to examining sessions for the current telescope period, all upcoming fixed and window sessions must be presented to the operator at least 12 hours in advance of their start time.

Session Probability Forecast

The session scheduling probability software operates in much the same manner as the real session scheduler, but uses predicted rather than current weather and makes other simplifying assumptions, e.g. there is no unexpected hardware issue.

Ideas/Issues for probability forecast:

• 24-hour prediction based on the weather forecast
• Predicted weather changes have a very good chance of actually happening, but probably will not occur exactly when it is predicted to occur.
• Give advanced notice to users of their probability of being scheduled in the next 24/48/week period.
• Use predicted weather to determine ranking of sessions for the next 24 hours. The sessions that are expected to be scheduled are notified that their probability is very good if the predicted weather occurs.
• Since weather rankings are continuous, it is difficult to access the relative rankings of projects. For instance if wind is 1 mph less than predicted, this may cause a different session than predicted to be scheduled.
• Additional minimal weighting for previously predicted sessions.

Time Accounting Tool

The design of the time accounting tool is still in development. The purpose of the tool is to keep track of the amount of time each session has used in its allocation.

Weather Information

Weather information is used for session scheduling, but is also available to the users of the Dynamic Scheduling System. Both current and predicted weather information will be accessible through web pages.

Current Weather

Current weather information will be obtained from several site weather stations through web pages.

Predicted Weather

Predicted weather will be drawn through NOAAWeatherForecastData and CLEO's weather forecast program through web pages.

RFI

The software suite plays an important part in the overall RFIPlan. The following is a list of important functionality:

• To facilitate cooperation with groups with whom we do not have a formal agreement, but whom are willing to limit emissions at certain frequencies when it is possible for them to do so, the output from the software suite will be displayed on a public website, and will include the frequency ranges of every scheduled project (in GHz).
  • This requirement needs to be modified once we have a reasonable idea of the predictability of weather for the GBT and know if it is feasible to write a semi-accurate telescope schedule and, if it is possible, how far in advance this schedule can be written. At a minimum, we should be able to post a schedule of the prime focus and other feed changes, at least within a few days. At best we may have a rough schedule that can be seen by all other users of the spectrum.
• The software suite needs an easy means to mark unexpected RFI problems. This needs to be a database which can both be changed on the fly (e.g. when RFI is seen or remedied) and which also can have dates, times, and frequency entered into it. If a project is begun and then canceled due to unexpected RFI, that fact, along with all critical parameters needed to define the RFI, will be both logged into the operator's log and e-mailed to the RFI group.
• If RFI is present in an observing band, we need two different types of indicators. The first would indicate that a project has been negatively affected by the RFI and the observer wishes not to try and run the project until the RFI has been mitigated. The second is simply a warning that there is known RFI in that band and that observers of the band need to be warned of its presence.
• When RFI is seen by the observers, the GBT telescope operators need to be able to readily look at the GBT RFI monitoring system (to be deployed before dynamic scheduling begins). If the RFI can also be seen on the monitor, that fact should be noted also in the software suite. When the RFI is then no longer seen on the monitor, the frequency should be unmarked as existing (but will remain in the RFI database for reference later). If the RFI cannot be seen on the monitor, the telescope operators will wait for word from either a member of the RFI group or one of the staff support scientists before unblocking the frequency in the DSS database.
• In the cases where we have full cooperation with an individual or group (e.g. amateur radio beacons), and that group cannot easily turn off its signal with only 30 minutes notice, we need to do the following:
  • Have an easy to understand (and update) list of all frequencies, times signatures, and power levels, of signals for which we have full cooperation;
  • Have a ready means for the observer to tell us that he needs coordination with a given frequency(s). This would best be done through a tool which feeds that fact directly to the Dynamic Scheduling Software (DSS) (e.g. a query in the Proposal Submission Tool);
  • Once the need for coordination is registered with the database, the RFI/EMI group (rfigrp) will be added to all scheduling e-mails for that project
  • When the project moves to high on its probability list, its parameters will change and it will become a Fixed Window project, with a three day window, and the RFI group will notify the person(s) responsible for the interfering signal about the schedule.
  • The person(s) responsible for the interfering signal will be asked to check the current GBT schedule before turning on the signal during the Fixed Window.
• In the cases where we have full cooperation with an individual or group and that group can easily turn off the signal with only 30 minutes notice, we need to do the following:
  • Have an easy to understand (and update) list of all frequencies, times signatures, and power levels, of signals for which we have full cooperation;
  • Have a ready means for the observer to tell us that he needs coordination with a given frequency(s). This would best be done through a tool which feeds that fact directly to the software suite.
  • Once the need for coordination is registered with the database, the RFI/EMI group (rfigrp) will be added to all scheduling e-mails for that project.
  • When the project is selected by the telescope operator, the operator will see an indicator noting that he or she will need to call the person responsible for the transmitter and request that he or she turn the transmitter off for the upcoming TP. The observer should be contacted before the person responsible for the transmitter to ensure the observer is indeed available for observing.

The RFI group is creating a RfiDataBase which contains known RFI sources, which will be extremely useful in implementing the functionality above.

Past Telescope Schedule(s)

All users will have access to past telescope schedules. These schedules will be an important communication tool between the NRAO and GBT observers. Observers will know what sessions got scheduled when and the information may even be used when observers are assessing their own future scheduling likelihoods. Among other things, past telescope schedules will include weather history.

M&C Configuration Checker Daemon

The purpose of the M&C configuration checker software is to (1) looks for additions and changes to cabling files and (2) verify configurations and modify the DSS database "configuration valid" field accordingly. This is the component of the system responsible for determining the availability of the hardware required by each session. The hardware availability is one of the factors used during the scheduling process of candidate sessions. Sessions are only eligible to be scheduled when the hardware necessary to complete the session is available. Currently, there is never a Telescope Period when all receivers are available so this component of the DSS is quite important. Additionally, hardware may fail occasionally and make sessions ineligible for scheduling. We expect to have more than 200 sessions per semester, discounting backlogged projects and carry over sessions from previous semesters. The GBT hardware configuration is currently determined by a cabling file and a device health file. Whenever either of these files is modified, the M&C Configuration Checker will evaluate each sessions ability to run and update the database appropriately with a Boolean field used directly by the scheduling algorithm to select candidate sessions for a given Telescope Period.

E-mail Daemon

The DSS will incorporate a system of e-mail notifications intended to communicate significant events to the various users of the system. Each part of the DSS may generate one or more e-mails to the same person for any number of reasons, e.g. there is a change in project or session metadata. However, we don't want to overburden the user with e-mails.

In our current concept of the e-mail notifications, each subsystem of the DSS will be responsible the content of the message. It is the e-mail daemon's responsibility to collate the various messages within a single e-mail. We envision this process will also include individual date and time stamps for each message which will be used to winnow the messages. For example, if a session is missing data and the user was notified yesterday about the problem, we may not want to send this message again today. Alternatively, we could also give the users of the DSS the flexibility to determine if they receive individual e-mails or a single daily digest e-mail, as is done with most e-mail lists. In either case, we will allow users the option of selecting the types of e-mails they receive. The only exception to this would be for primary investigators; they must receive all e-mails pertinent to their project(s). It is important to note that many of these e-mail triggered events are also reflected on the project web page.

E-mail Triggers:

• E-mail notification to the primary investigator and any subscribers (e.g. a project friend) to that information:
  • Changes to any project or session data
  • Incomplete session data
  • A session can not be scheduled
  • Significant changes to the probability of the allocation being scheduled
    • If a project's observing status (probability) changes due to the running of the DSS (e.g. observing or priority has changed), that change will be automatically reflected on the project's web page, a page which provides or provides links to, all pertinent information for a given project and its sessions (and within a DSS database). As with all changes to the project web page, this change will generate an automatic e-mail to the interested investigators and to the Project Friend, informing them that there has been a schedule change and they should examine the project web pages to determine what the change is. It should also remind the investigators to update their contact information if it looks likely their project will be observed. The caveat is that the DSS should not overburden the investigators with e-mails due to small tweaks in the Session priorities. The exact mechanism for determining when an investigator/observer receives e-mail from the DSS will be determined through planned tests of the DSS.
  • Changes to the hardware availability required by the session
  • Notification of time accounting issues
  • RFI issues
  • Results of refereeing process
• E-mails to the Primary Investigator only:
  • The project has been approved, the project name and password with a link to DSS website. The PI may disseminate the password to COPIs/observers of their project.
• E-mail to RFI group:
  • unexpected RFI
• E-mail to Telescope Scheduler:
  • A Telescope Period was deemed unsuccessful

VLB Schedule Interface

One of the main goals of the DSS is to achieve efficient scheduling of VLB. We must provide adequate support in the software suite for this type of project, especially in terms of the Telescope Scheduler interface.

Support Software Functionality

Interface to the Proposal Submission Tool Database

The NRAO Proposal Submission Tool (PST) is a package for creating and submitting proposals for NRAO telescopes. Once proposals are reviewed and accepted, the proposal information must migrate from the PST to the GBT DSS each semester. Therefore, there will need to be some sort of communication mechanism linking the PST database and the DSS database. There are two possibilities for this communication. If the DSS has access to the PST database, it can access the information directly. If the DSS cannot access the PST database, the DSS will need an interface (presumably provided by the group responsible for the PST, the e2e Division. The e2e division is also responsible for the NRAO-wide User Database which has query servlet interfaces that provide user info in XML format. Something similar to the PST database would be desirable if we don't get full access to the database itself.

Proposal Handling Tool

This is a tool to update the results from the refereeing and Proposal Selection Committee process (e.g. science grade, session hours, weather grades etc.) and is tabled for the moment until we better understand the requirements.

Replacement program(s) for Current Telescope Scheduler Support Tools

Detailed information on CarlBignell's current suite of support tools can be found at SchedulingSoftware.