NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Dynamic > DSSSimulations

Changes | Index | Search | Go

DSS Simulations

• Introduction
• Goals
• Simulation Inputs
  • Projects
  • Weather
    • Wind
    • Tsys
    • NOAA Forecast Data
    • Additional Factors
  • Thermal Transition Boundaries
  • Telescope State
  • Activities
  • RFI
  • Operator
• Simulation Outputs
  • Telescope Period Decision Tracking
  • Classical Telescope Schedule
• Process
  • Automated Vs. Manual Simulations
    • Manual Simulations: someone gets to play Operator
    • Automated Simulations: simulating the Operator
  • I. Simulation Initialization
  • II. Decide next Telescope Period
  • III. View Reports
• Software Suite Elements
  • Necessary Elements
  • Superflous Elements
  • Additional Elements
• Intermediate Steps

Introduction

I got pretty confused at yesterdays meeting. It seems like we are on the verge of being able to write software, but we need to figure out a plan of attack. Melinda is querying a test database using our selection rules with some assumed inputs (like Weather Grade) and displaying the results as plots of the resultant allocations. This is very cool (to use a technical term), but she keeps asking a question which I can't answer: "how should these results be formatted or displayed?". That lead to me (us?) asking, 'what do these results tell us'? I still can't answer that question, so I'm now trying to back up and ask: 'what are the final tests that we want to make'?

I've started this wiki page as just a scratch pad for figuring out what kind of tests, or simulations, we want to be able to run before the Sep. 2007 test time. Perhaps if we can clearly define these, then we can determine the intermediate tests we want to run in the short term (now), and answer questions like the ones stated above.

Below is essentially a proposal for a framework of simulations of the DSS before the Sep. 2007 test time. Karen brought up the point that the production system will need a simulation mode as well (as most SDD software products have), but I'm not trying to address this issue in particular, though it should be kept in mind during design of the DSS.

Goals

The main goal before the Sep. 2007 test time would be to simulate entire Semesters using Dynamic Scheduling Algorithms and see:

• what the frequentyly encountered problems were
• wether or not high frequency weather was properly used
• wether or not projects were completed in a timely manner
• what happens at the begining and end of a semester
• what kinds of general trends appear
• what affect being 'on site' really has
• what happens if we change or add/remove rules and priorities -Main.KarenONeil

Simulation Inputs

It seems like most of the necessary input can be simulated. Note that all of the below mentioned should be predetermined, as in entries in a database. One of the reasons to not use 'randomized at run-time' inputs would be that our simulations would be deterministic: if the same operator decisions were made using the same inputs, then the resultant schedule for the semester should be the same.

Projects

MelindaMello has put togethor a database that contains a pre-observing DSS representation of semester 06A. Much of the project metadata discussed in SchedulingData are represented in this database, which currently includes tables for Project, Allocation, and more. See DSSTestDatabase for more details on how the data was generated. See DSSTestCode for details on what has been implemented and example queries.

Here are some of the pre-observing inputs that still need to be put in this database:

• User Info:
  • Can copy basic User Contact Info from Carl's database
  • Can start a simulation of who was 'on site' for a given semester using the Reservations database
  • Will just have to make up other things to do a full simulation: user not available dates, etc.
• Fixed Time and Fixed Window dates
• Activities
  • This includes maintanence and commisioning times. How exactly are these represented in our database?

Weather

This is one of the main inputs used for creating a list of canidate allocations at the start of a Telescope Period. For our simulations, we can use a combination of real historical wind speeds, historical forecasts for Tsys, and other factors if need be.

Wind

We can simulate the wind using real wind data from the past. See NOAA Forecasts for more details on the existing software for pulling up sampler2log data.

Tsys

We have no real data for Tsys, but we can create tables of predicted Tsys (0-12 hours old) for the past to use for simulations. This would mean using cleo forecasts to write out the results to a file, and then dumping the Tsys Average values into a database table to use for the simulations. See TsysPredictions for more info.

NOAA Forecast Data

All the raw data form the NOAA forecasts are available to us in a database. Again, see NOAA Forecasts for more details.

Additional Factors

What about the fact that when deciding the next TP, an operator may use all of the above weather inputs, but may also look out the window or at the radar to make their decision. Is this also something that should/could be simulated? It seems to me that we want to ignore that for our simulations. Instead, what would be interesting is to see how often the window or radar images give us information that contradicts the predicted/measured weather -Main.KarenONeil

PRM - I think I agree with Karen's comment about ignoring this factor when doing fully automated simulations (see discussion below on Process). And I think that this is also what Karen means, but let me try and reword it. It would be good, if the simulation were not in automated mode, to be able to let the operator input additional weather info (looking out the window, or at the radar screen). In this way, we could see the affect of the operator overriding what the forecasts may be predicting for the weather.

Thermal Transition Boundaries

This can easily be determined from the sunrise/sunset times for a given date.

Telescope State

It seems like we could simulate the state of the telescope for a given semester: another words, when were certain receivers up or down. Or when did the Spectrometer break and how long did it take to fix it? Did the telephone lines go down for a day (not that that would ever happen).

But how to do this? Might mean a lot of trolling through operator logs or the resource calendar.

The best method would be to not worry as much about what happened historically, but instead try and understand how this would be done with just the DSS observing database in hand, and knowing something about the average number of maintenance days and commissioning runs in a given trimester. -Main.KarenONeil

PRM - I agree with this statement in regards to maintancence days and commissioning runs. And for regular activities, like receiver swapping, this should also be straightforward. My concerns were mostly about simulating hardware failures. It's easy enough to make up stuff, like saying the Spectrometer was out for 2 days in the third week of the semester, but how do we make sure that the sum of all these breakdowns is realistic for a semester. Probably not a huge deal though.

Activities

We need to simulate when maintanence days want to be scheduled (put them in as fixed windows). This should be easy. What about commissioning? This should be straightforward as well. We can always pull one of the old commissioning schedules off the wiki if we want, or make it up. -Main.KarenONeil

Also, as mentioned in the above Projects section, how are Activities represented in the Project database.

RFI

I was feeling good about everything until I remembered this.

How do we simulate a semesters worth of RFI? I imagine we need something that tells us 'from this time to this time there was bad RFI in this band, badly affecting these lines'. Even if this is good enough, how can we create it? Are there records from previous semesters we can view?

If nothing else I can go through the saved emails I have from RFI and we can use that to simulate a semester's worth of problems. -Main.KarenONeil

Operator

The operator is the human being who will be making many decsisions at the beginning of each TP. A true test of the system would include a human operator - another words we won't want to simulate this person. However, as I'll discuss below, we may want to try simulating the operators desicions for other reasons. With the Operator simulated, we could run fully automated simulations.

Simulation Outputs

As the simulation progresses, a number of data points will need to be tracked. Many of the things that need to be recorded during simulation will also need to be tracked in the production system.

Telescope Period Decision Tracking

For both simulations, and in the production system, it will be desirable to look back at a certain Telescope Period and understand how the decision was made to run a particular allocation. What would be sufficient to answer this question?

One easy enough thing to do is save off the inputs and outputs of the Scheduler Tool. Some of these inputs would be 'snapshots' of current conditions (telescope state, weather, etc.). The outputs would at least include the prioitized canidate allocations the tool presented to the operator.

But how much more info about the Allocations? It seems like the Allocation variables that change should be listed here, like percent complete. But this could be a slippery slope. How do you track change in the database?

In addition, other operator actions will need to be recorded. If the operator contacts the observer for the top allocation choice, and the observer can't be contacted, this needs to recorded in a machine-readable format, etc. Is having the operator change the users availability to not availabile for that telescope period sufficient?

It seems like we can split the determination of a TP into two parts: the use of the Scheduler Tool, and the actions taken aftwerwards. For instance, the operator may run the Scheduler Tool given a certain endpoint for the next TP, not like the results, and run it again with the TP endpoint changed slightly. I don't understand the problem that you are trying to solve. I thought one of the goals of the simulation was to run it manually with no Telescope periods specified. The user/operator would see allocations that could be with the current time/LST. Part of the simulation is to determine what information needs to be displayed to enable the user to readily make intelligent choices. KAren has suggested (at least in theory), going through the simulation this way will help us determine the "rules/logic/intellignce" choices. After we determine this, we can then consider putting the login in the DSS Scheduler.

So, with the above discussion, what would the necessary tables look like?

Classical Telescope Schedule

Again, for both simulations and production, one thing that will be desirable during and after a semester is the ability to view what has been run so far in the format of what is called a 'classical schedule'. This should be easy enough. Given our current terminology, two different tables could be written to:

• Telescope Period - these can include more then one session
  • tpId (unique auto increment integer)
  • start datetime
  • end datetime
  • total time (= end - start)
• Session - these are allocations scheduled on the telescope during a TP
  • sessionId (unique auto increment integer)
  • allocationId
  • start datetime
  • end datetime
  • total time (= end - start)

Process

How would these simulations actually run? Below I try and flesh that out. I first propose the running of simulation in two different modes (automated vs. manual), then I discuss the actual steps in running the simulation.

Automated Vs. Manual Simulations

It seems to me, and from Karen's comments, that being able to run the simulations in two different modes would be very useful. Automated simulations would be very fast and would provide deterministic results, while manual simulations provide us a means of seeing the affects of human decisions.

Manual Simulations: someone gets to play Operator

It seems to me that a good test of the DSS would be for the user of the simulation to be acting as Operator in 'condensed time' for an entire semester: a decision is made for the next TP, that TP elapses in 'condensed time' rather then in real time, and the Operator has to decide again for the next TP, and so on until the semester ends.

Is this practical? Well, how many TPs would have to be decided?

4 months * 28 days * 24 hours / 4 hours per TP = 672 Telescope Periods

If the operator takes about 5 minutes per decision, then:

672 * 5 = 3360 minutes = 56 hours = ~1.5 FTE weeks.

So, it would take about 2 weeks for one person to simulate an entire semester. Eventually, I think we'll want to do this.

By inserting a human being at the point in the simulation, we can see the affects of all the decisions and actions taken by the Operator. These actions include, but aren't limited to:

• Changing the length of the Telescope Period
• Turning on & off filters in the selection rules to get different canidate Allocations
• Making separate judgements on the weather
• Failing to contact an 'available' observer (we can pretend that this happens)
• What else?

Automated Simulations: simulating the Operator

We could speed things up by 'simulating' the operator: the operator always picks the top priority allocation, the user can always be contacted, the TP is shrunk or extending appropriately, and we just let the simulation run until an error is encountered that requires human intervention. This might be a quick and dirty way of running tests.

But it also provides a means for creating deterministic simulations. If consistent rules are used for the simulated operator, and whatever human intervention is recorded, then, along with the other simulated inputs, there seems no reason why the same initial conditions (inputs) should not yield the same results (the final semesters schedule).

But it would be valuable if we can do this for a period of time, then go back to full interaction mode, and be able to switch back and forth like that. E.g. We could do the simulation for a trimester in automatic mode and then look and see where an operator's decision could have been valuable or could have changed things. Then we re-reun the simulation for the first, say, month, go into non-automatic mode and make some selesctions that were different thaen the automatic mode, then go back into automatic mode for awhile. A system like this could be very valuable. - KarenONeil

Yes, I agree with Karen's idea above. The use of Automated Mode gives us the ability to 'create' a semester in progress.

I. Simulation Initialization

• Create a new database? Or just finish populating Melinda's database - KarenONeil

Let me explain more fully what I meant here. It seems to me that it would be valuable not to overwrite previous simulations (unless you explicity need/want to). In other words, after I run the simulation for an entire semester, I have altered the fields in many of the pre-observing tables in the database, and created large amounts of records on how the semesters schedule was derived. This all serves as a record of the 'experiment', and may need to be archived for further analysis in the future.

So, all I'm trying to propose here is that every new simulation starts off with it's own pre-observing database. This could simply be a copy of the database that Melinda is working on right now (DSSTestDatabase), or we could get more sophisticated (not sure this is necessary). In this way, we simply avoid overwriting our results.

• What else needs to be setup?
• If running in Automatic mode, give the timestamps that you want to run in this mode (i.e., run from the start of the semester to the 4th week).

II. Decide next Telescope Period

If the simulation is in Manual mode, or has just finished a round of Automated simulation over a period of time in the semester, then the 'user' must pretend to be the Operator, and decide what will be scheduled for the next Telescope Period.

• 'current time' in simulation is choosen so as to be appropriate for the simulation data.
• inputs displayed for operator (simulation user):
  • list of allocation candidates (as LST ranges?)
  • thermal transition boundaries
  • weather information, including past hours measured weather, forecasts, etc.
  • fixed time and fixed windows
  • what else is needed for operator to make decision?
• operator makes decision and contacts user
  • Should all users be able to be reached? We can randomly select some user-dates where they can't be reached to simulate real observing
• all decisions and other factors saved to maintain state of simulation
• does TP get interrupted? Due to:
  • changing weather
  • RFI
  • device breakdown
• if TP is not interrupted, advance clock to end of TP so that next TP can be determined
• 'Branch' simulation if desired, so other simulations can be run using different decisions/conditions. In practice, this could mean making a full copy of the current database, so that a different simulation could use this database, effectively taking off from where the 'Branch' was made.
• view reports on state of simulation

III. View Reports

What reports are these? Whatever help us answer the questions posed in the Goals section above. These don't need to be the pretty web pages that are required for the production system, just something that displayes information adequately to answer the questions. See also the discussion above concerning outputs from the simulator.

Examples:

• A view of what has been scheduled so far this semester, as a 'classical schedule'.
• Project data: top copmleted projects, allocations, etc.
• Decisions: for a given TP, what were the top allocation canidates, and what were the inputs that colored the operator's choices.

Software Suite Elements

What needs to be implemented in the Software Suite to support these simulations, what can be ignored, and what is needed that is not specified there (either as 'throw away' software, or as part of future production code).

Necessary Elements

Superflous Elements

Additional Elements

Intermediate Steps

So, if the above is the final goal, what can we do now to get there? Here we need I think to understand the full list of software which we believe we will be using, and then we can break down that list into what we should simulate first. -- KarenONeil

-- PaulMarganian - 29 Nov 2006