NRAO Home  >  Green Bank  |  Wiki Topic:    GB > Data > GbtFitsMonitorContinuumRequests > GbtFitsMonitorBenchmarks

Changes | Index | Contents | Search | Statistics | Jump to Topic

GFM Benchmarks

The intent of this page is to provide benchmarking information and test results output for comparison with future versions of gfm. Currently, this refers only to continuum peak/focus fitting.

Target is elapsed time less than one second per scan for dual-beam observations, delivered into the hands of the Observer running on the "typical" observer's workstation under "typical" load conditions.

• GFM Benchmarks
  • Results
  • Notes
  • Conclusions
• Test Data Sets
  • DatasetA: TPTCSRMP031229 Scans 5 - 404
  • DatasetB: TPTCSDSB040225 All scans

Results

Notes

Test 1 This test was done with gfm running on titania, while the Observer was running the full observing interface on this machine (many CLEO screens, GO, IARDS, their own invocation of GFM). The display was remote on pleiades (my windows laptop running Exceed). It therefore should represent a pretty typical to slightly extreme example of real-world usage of GFM. Peak cpu usage by gfm never seemed to exceed around 60%, and performance was pretty uniform throughout the test.

Test 2 This test was run directly logged in to naiad. I was the only interactive user; the antenna characterization manager and associated infrastructure was running on naiad, but the machine seemed basically idle. I ran gfm with the main window minimized, and only "play" window open. gfm seemed to get ~95% of the cpu through-out the test.

Test 3 This test was run remotely logged in to naiad from pleiades (my windows laptop). naiad was otherwise idle as for test2. I had the main gfm window open, and carried on with other windows applications while the application was running.

Test 4 Run on idle naiad, display remoted back to pleiades, open. Aborted scan, purpose was to generate test output.

Test 5 Run on an idle thalassa, with the plotting window open (thalassa is a dual-cpu machine, but the individual cpu's seem to be the same class as in naiad). Very analagous to test 2.

Test 6 As test 5 but for gfm-test.

Test 7 As test 6 but with DCR plugin enabled also.

Test 8 As test 6, but only DCR plugin.

Test 9 As test 5 but for gfm-test (version 3).

Test 10 As test 5 but for gfm-test on naiad (version 4).

Test 11 As test 7 but for gfm-test on naiad (version 4).

Test 12 As test 5 but for gfm-test on naiad (version 4).

Test 13 As test 7 but for gfm v4.3 on naiad.

Test 14 As test 5 but for gfm v4.3 on naiad.

Test 15 As test 7 but for gfm-test C4 v1 on naiad.

Test 16 As test 5 but for gfm-test C4 v1 on naiad.

Conclusions

I believe that Test 1 is a pretty good "real-world" example of the performance a typical observer might see in real-time use. For test2, gfm appears to be compute-bound; from a comparison between test2 and test3 it appears that driving a remote or local display doesn't add a lot of overhead. Note that I deliberately ran this test with the "old" version of gfm; a new version might be faster. My conclusions:

• We need at least a factor of six speed improvement over the current gfm in normal use (above tests were all single beam only.
• It looks like this is mainly dominated by the data processing (although maybe I'm missing something).
• The 4/16/04 gfm-test improvements have resulted in a ~20% speed improvement.
• From the DCR plugin-only results, current cpu usage for single-beam peak/focus seems to be split ~50:50 between data access/ preprocessing, and gaussian fitting.
• Running the DCR plugin in parallel with the peak/focus plugins will be an unacceptable overhead unless the data access time is significantly decreased.

Since we don't want to restrict gfm to a dedicated, special purpose machine, the ideal solution would be a x6 time improvement in cpu utilization. Conceivable alternatives would be a factor of three improvement and a dedicated existing machine, or no improvement and run it on a 3x faster new machine.

Test Data Sets

DatasetA: TPTCSRMP031229 Scans 5 - 404

This is an X-band all-sky pointing run. Here is the observing log. Dynamic corrections were turned on. Scan 1 has a missing GO fits file, so I've excluded the first peak; observations continue after scan 404, but ~400 scans seemed enough. These are all comparatively bright sources, and the weather was good, so this is an "easy" dataset. The 05/12/04 production version of gfm is capable of reducing all of these scans with no errors. Here are the results. Note that there are a few scans which have corrupted data:

• Scans 65 through 77 show RFI. gfm does a good job of processing these.
• Scan 119 was part of an aborted observation.
• Scan 127 has a large baseline shift which screws up the fit (although again gfm does a good job under the circumstances).
• Scans 170 through 179 have huge RFI.
• Scans 275 and 276 have strange -ve "dropouts".

DatasetB: TPTCSDSB040225 All scans

This is a combined Ku, K and Q-band commissioning run to perform out-of-focus beam-maps. Here is the observing log. Here are the results. This is a pretty challenging test of gfm. It has it all; weak sources at Q-band, aborted scans, raster maps that the peak/focus modules have to ignore, etc. I have not recorded timing information.

-- AmyShelton - 13 May 2004