• See the bug list.

• Bug Tracker
• Requested Features Tracker

• See a presentation of DEAP at PyCon 2004 - a Python conference organized to highlight significant advances in the Python development community!

• See a presentation of DEAP at PyCon 2004

-- AmyShelton - 24 Feb 2004

-- AmyShelton - 19 Mar 2004

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of viewing the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accommodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive read-only plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP is available as an Open Source product via Source Forge.

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of viewing the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accommodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive read-only plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP is available as an Open Source product via Source Forge.

DEAP is written in Python and presents two user interfaces - a command line window and a plot window. The GUI portion of DEAP utilizes wxPython and features a custom widget, created by the NRAO, that integrates PGPlot into wxWindows. PGPlot is a C-callable, device-independent graphics package for making simple scientific graphs and has been used in radio astronomy software for many years to produce publication-quality plots. See the "Technical Features and Enhancements" section of this page for a listing of DEAP’s current features.

DEAP is written in Python and presents two user interfaces - a command line window and a plot window. The GUI portion of DEAP utilizes wxPython and features a custom widget, created by the NRAO, that integrates PGPlot into wxWindows. PGPlot is a C-callable, device-independent graphics package for making simple scientific graphs and has been used in radio astronomy software for many years to produce publication-quality plots. See the "Technical Features and Enhancements" section of this page for a listing of DEAP’s current features.

XY data sets are entered into DEAP via the command line window. The X data and Y data are simply lists of numbers. In fact, all data entered into DEAP and extracted from DEAP are formatted as lists at the command line. The command line window for DEAP is the Python command line plus a set of functions through which the user may interact with DEAP. For example, the user may add a new XY plot to DEAP via the AddXYPlot command that accepts two arguments - a list of x data and a list of y data. The command line enables DEAP commands to be executed one at a time or as scripts (via Python's execfile function). Many commands that are available via the command line are also available via drop-down menus or toolbar items in the plot window. One feature of note is the integration of both the command line window and the plot window, meaning that command issued through one window are reflected both in DEAP and in the other window.

XY data sets are entered into DEAP via the command line window. The X data and Y data are simply lists of numbers. In fact, all data entered into DEAP and extracted from DEAP are formatted as lists at the command line. The command line window for DEAP is the Python command line plus a set of functions through which the user may interact with DEAP. For example, the user may add a new XY plot to DEAP via the AddXYPlot command that accepts two arguments - a list of x data and a list of y data. The command line enables DEAP commands to be executed one at a time or as scripts (via Python's execfile function). Many commands that are available via the command line are also available via drop-down menus or toolbar items in the plot window. One feature of note is the integration of both the command line window and the plot window, meaning that command issued through one window are reflected both in DEAP and in the other window.

DEAP currently runs on Linux and Windows, but plans are underway to allow its use on Mac OS X.

DEAP currently runs on Linux and Windows, but plans are underway to allow its use on Mac OS X.

• See a presentation of DEAP at PyCon 2004 - a Python conference organized to highlight significant advances in the Python development community!

• See a presentation of DEAP at PyCon 2004 - a Python conference organized to highlight significant advances in the Python development community!

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of viewing the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accommodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive read-only plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net.

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of viewing the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accommodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive read-only plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP is available as an Open Source product via Source Forge.

DEAP is written in Python and presents two user interfaces - a command line window and a plot window. The GUI portion of DEAP utilizes wxPython and features a custom widget, created by the NRAO, that integrates PGPlot into wxWindows. PGPlot is a C-callable, device-independent graphics package for making simple scientific graphs and has been used in radio astronomy software for many years to produce publication-quality plots (see http://www.astro.caltech.edu/~tjp/pgplot for more information regarding PGPlot). The following is a summary of DEAP’s current features:

DEAP is written in Python and presents two user interfaces - a command line window and a plot window. The GUI portion of DEAP utilizes wxPython and features a custom widget, created by the NRAO, that integrates PGPlot into wxWindows. PGPlot is a C-callable, device-independent graphics package for making simple scientific graphs and has been used in radio astronomy software for many years to produce publication-quality plots. See the "Technical Features and Enhancements" section of this page for a listing of DEAP’s current features.

XY data sets are entered into DEAP via the command line window. The X data and Y data are simply lists of numbers. In fact, all data entered into DEAP and extracted from DEAP are formatted as lists at the command line. The command line window for DEAP is the Python command line plus a set of functions through which the user may interact with DEAP. For example, the user may add a new XY plot to DEAP via the AddXYPlot command that accepts two arguments - a list of x data and a list of y data. The command line enables DEAP commands to be executed one at a time or as scripts (via Python's execfile function). Many commands that are available via the command line are also available via drop-down menus or toolbar items in the plot window. One feature of note is the integration of both the command line window and the plot window, meaning that command issued through one window are reflected both in DEAP and in the other window.

The plot window is focused on handling user interactions with the XY data sets. These interactions include zooming and flagging. Flagging refers to associating an integer value with a data point. By default, all data points in an XY data set are flagged with a value of 0. The user associates flag values with data points by lassoing them with the mouse. Because the flag values are simply integers, the user is free to associate any meaning to the flag values that he/she desires. Flagging is important in working with radio astronomy data because it is an important means of highlighting data contaminated with radio frequency interference or RFI.

DEAP currently runs on Linux and Windows, but plans are underway to allow its use on Mac OS X.

2. Technical Features and Enhancements

• Exporting of plots to a variety of graphical formats: BMP, GIF, Postscript, color Postscript

• Exporting of plots to a variety of graphical formats: BMP, JPG, PNG, Postscript, color Postscript

Perhaps DEAP's features can be better explained by highlighting some of the features above. XY data sets are entered into DEAP via the command line window. The X data and Y data are simply lists of numbers. In fact, all data entered into DEAP and extracted from DEAP are formatted as lists at the command line. The command line window for DEAP is the Python command line plus a set of functions through which the user may interact with DEAP. For example, the user may add a new XY plot to DEAP via the AddXYPlot command that accepts two arguments - a list of x data and a list of y data. The command line enables DEAP commands to be executed one at a time or as scripts (via Python's execfile function). Many commands that are available via the command line are also available via drop-down menus or toolbar items in the plot window. One feature of note is the integration of both the command line window and the plot window, meaning that command issued through one window are reflected both in DEAP and in the other window.

The plot window is focused on handling user interactions with the XY data sets. These interactions include zooming and flagging. Flagging refers to associating an integer value with a data point. By default, all data points in an XY data set are flagged with a value of 0. The user associates flag values with data points by lassoing them with the mouse. Because the flag values are simply integers, the user is free to associate any meaning to the flag values that he/she desires. Flagging is important in working with radio astronomy data because it is an important means of highlighting data contaminated with radio frequency interference or RFI.

DEAP currently runs on Linux and Windows, but plans are underway to allow its use on Mac OS X.

2. Technical Features and Enhancements

• Need some

• See the bug list

• See the bug list.

None

• DEAP Project Page
• PGPlot
• See a presentation of DEAP at PyCon 2004 - a Python conference organized to highlight significant advances in the Python development community!

Generic Data Flagging & Editing Presentation for February 2004 Software Review

Data Extraction and Analysis Program (DEAP)

Tool for Generic Data Flagging & Editing

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of viewing the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accommodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive read-only plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net. The DEAP website on Source Forge is currently under development as of this proposal and will be fully available for those interested in learning more about DEAP and downloading their own copy before PyCon 2004.

1. Introduction

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of viewing the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accommodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive read-only plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net.

DEAP currently runs on Linux and Windows, but plans are underway to allow its use on Mac Os X.

DEAP currently runs on Linux and Windows, but plans are underway to allow its use on Mac OS X.

2. Technical Features and Enhancements

• Need some

3. Known Issues

• See the bug list

4. Performance Testing

No performance testing yet.

5. References

None

-- AmyShelton - 15 Jan 2004

Core Team

-- AmyShelton - 24 Feb 2004

Note: This is currently formatted for submittal to Py Con 2004. I will fix the formatting after the proposal has been submitted on January 15.

Author Name:

Amy Shelton

Contact Information:

National Radio Astronomy Observatory
Post Office Box 2
Green Bank, WV 24944
(304) 456-2277
ashelton@nrao.edu

Requested timeslot:

30 minutes

Author Background:

Amy Shelton has worked for the National Radio Astronomy Observatory for the last 5 years, the last 3 of which were spent with the Software Development Division. She holds a Bachelors of Science in Electrical Engineering from the University of Cincinnati and a Masters of Software Engineering from the University of Maryland University College.

Summary of proposed presentation:

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of viewing the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accomodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive read-only plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net. The DEAP website on Source Forge is currently under development as of this proposal and will be fully available for those interested in learning more about DEAP and downloading their own copy before Py Con 2004.

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of viewing the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accommodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive read-only plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net. The DEAP website on Source Forge is currently under development as of this proposal and will be fully available for those interested in learning more about DEAP and downloading their own copy before PyCon 2004.

• Interactive, multicolor XY plotting with zooming, error bars, and data flagging capabilities.
• XY plots allow choice of line type (including no line), choice of marker type for data point representation, auto-scaling.
• Creation of titles, axes labels, and annotations.
• Plotting of multiple data sets on the same plot.
• Saving/Retrieving of plots and plot information from binary files.
• Exporting of plots to a variety of graphical formats: BMP, GIF, Postscript, color Postscript.
• Data analysis operations available for data sets include calculation of the mean, calculation of the variance, Gaussian fitting, linear fitting, and polynomial fitting.

• Interactive, multicolor XY plotting with zooming, error bars, and data flagging capabilities
• XY plots allow choice of line type (including no line), choice of marker type for data point representation and auto-scaling
• Creation of titles, axes labels, and annotations
• Plotting of multiple data sets on the same plot
• Saving/Retrieving of plots and plot information from binary files
• Exporting of plots to a variety of graphical formats: BMP, GIF, Postscript, color Postscript
• Data analysis operations available for data sets include calculation of the mean, calculation of the variance, Gaussian fitting, linear fitting, and polynomial fitting

XY data sets are entered into DEAP via the command line window. The X data and Y data are simply lists of numbers. In fact, all data entered into DEAP and extracted from DEAP are formatted as lists at the command line. The command line window for DEAP is the Python command line plus a set of functions through which the user may interact with DEAP. For example, the user may add a new XY plot to DEAP via the AddXYPlot command that accepts two arguments - a list of x data and a list of y data. The command line enables DEAP commands to be executed one at a time or as scripts (via Python's execfile function). Many commands that are available via the command line are also available via drop-down menus or toolbar items in the plot window. One important issue to note is that both the command line window and the plot window are fully integrated meaning that command issued through one window are reflected both in DEAP and in the other window.

Perhaps DEAP's features can be better explained by highlighting some of the features above. XY data sets are entered into DEAP via the command line window. The X data and Y data are simply lists of numbers. In fact, all data entered into DEAP and extracted from DEAP are formatted as lists at the command line. The command line window for DEAP is the Python command line plus a set of functions through which the user may interact with DEAP. For example, the user may add a new XY plot to DEAP via the AddXYPlot command that accepts two arguments - a list of x data and a list of y data. The command line enables DEAP commands to be executed one at a time or as scripts (via Python's execfile function). Many commands that are available via the command line are also available via drop-down menus or toolbar items in the plot window. One feature of note is the integration of both the command line window and the plot window, meaning that command issued through one window are reflected both in DEAP and in the other window.

Presentation outline:

1. Motivation for creation of DEAP and applicability to generic data flagging and editting
2. Overview of DEAP functionality
   1. XY plotting
   2. data flagging
   3. data analysis
3. DEAP tutorial through example - importing sample data set, flagging data, fitting data to polynomial and Gaussian
   1. Use of the command line window
   2. Use of the plot window
   3. Saving and/or exporting plots
4. Future development

-- AmyShelton - 13 Jan 2004

-- AmyShelton - 15 Jan 2004

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of view the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accomodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive (read-only) plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net. The DEAP website on Source Forge is currently under development as of this proposal and will be fully available for those interested in learning more about DEAP and downloading their own copy before Py Con 2004.

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of viewing the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accomodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive read-only plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net. The DEAP website on Source Forge is currently under development as of this proposal and will be fully available for those interested in learning more about DEAP and downloading their own copy before Py Con 2004.

• Command line (allowing full access to numeric python)

• Command line (allowing full access to numeric python)

Comments in italics. May require some editing to keep this under the size limit.

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. At present, plans are being prepared to reengineer the single-dish data analysis system currently in use (a procedurally-written, single-user, single-tier application) into a layered and fully object-oriented application.

To do this, a generic tool that allows users to view and analyze their data either before importing that data into a data reduction package or as a component that extends the data reduction package is required. Such a generic tool would not necessarily replace commercial data reduction packages, but instead would extend the functionality offered by those tools while giving observers a tool that could also be used standalone.

I think DEAP can be a bit better motivated. First, explain that data reduction is the process of producing publishable scientific plots from raw radio data, and that this involves an iterative, highly interactive process of viewing the data in detail, editing it (flagging), and analyzing it (fitting). Second, explain how no existing tools really fill all these needs. There are good tools for producing publishable plots (pgplot), there are good tools for producing interactive (read-only) plots (tkplot or qwt), and there are good tools for doing data analysis (numeric, gsl). No existing free tools really combine all these features, and no existing tools provide quality interactive graphical data editing. Further, even the commercial tools tend to be a little weak in one or more of these areas, so DEAP really is a uniquely useful application.

The fulfillment of this need is the motivation and purpose behind the development of the Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for graphing and analyzing many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net. The DEAP website on Source Forge is currently under development as of this proposal and will be fully available for those interested in learning more about DEAP and downloading their own copy before Py Con 2004.

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. Astronomers refer to the conversion of raw radio astronomy data into publishable science as data reduction. Data reduction is an interactive, highly iterative process of view the data in detail, editing it (data flagging), and analyzing it (data fitting). Currently, there are no existing tools that accomodate data viewing, data flagging, and data fitting within the same application. There are good tools for producing publication quality plots (PGPlot), for producing interactive (read-only) plots (tkplot or qwt), and for doing data analysis (numeric, gsl). However, no existing free tools solidly combine all these features, and no existing tools provide quality interactive graphical data editing. The creation of a single application that marries these needs is the motivation for Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the data reduction needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for viewing, flagging, and fitting many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net. The DEAP website on Source Forge is currently under development as of this proposal and will be fully available for those interested in learning more about DEAP and downloading their own copy before Py Con 2004.

Comments in italics. May require some editing to keep this under the size limit.

I think DEAP can be a bit better motivated. First, explain that data reduction is the process of producing publishable scientific plots from raw radio data, and that this involves an iterative, highly interactive process of viewing the data in detail, editing it (flagging), and analyzing it (fitting). Second, explain how no existing tools really fill all these needs. There are good tools for producing publishable plots (pgplot), there are good tools for producing interactive (read-only) plots (tkplot or qwt), and there are good tools for doing data analysis (numeric, gsl). No existing free tools really combine all these features, and no existing tools provide quality interactive graphical data editing. Further, even the commercial tools tend to be a little weak in one or more of these areas, so DEAP really is a uniquely useful application.

DEAP is written in Python and presents two user interfaces - a command line window and a plot window. The GUI portion of DEAP utilizes wxWindows and features a custom widget, created by the NRAO, that integrates PGPlot into wxWindows. PGPlot is a C-callable, device-independent graphics package for making simple scientific graphs and has been used in radio astronomy software for many years to produce publication-quality plots (see http://www.astro.caltech.edu/~tjp/pgplot for more information regarding PGPlot). The following is a summary of DEAP’s current features:

DEAP is written in Python and presents two user interfaces - a command line window and a plot window. The GUI portion of DEAP utilizes wxPython and features a custom widget, created by the NRAO, that integrates PGPlot into wxWindows. PGPlot is a C-callable, device-independent graphics package for making simple scientific graphs and has been used in radio astronomy software for many years to produce publication-quality plots (see http://www.astro.caltech.edu/~tjp/pgplot for more information regarding PGPlot). The following is a summary of DEAP’s current features:

• Command line

• Command line (allowing full access to numeric python)

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. The current data analysis system at Green Bank lacks a generic tool that allows users to view and analyze their data either before importing that data into a data reduction package or as a component that extends the data reduction package. Such a generic tool would not necessarily replace commercial data reduction packages, but instead would extend the functionality offered by those tools while giving observers a tool that could also be used standalone. The fulfillment of this need is the motivation and purpose behind the development of the Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for graphing and analyzing many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net. The DEAP website on Source Forge is currently under development as of this proposal and will be fully available for those interested in learning more about DEAP and downloading their own copy before Py Con 2004.

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. At present, plans are being prepared to reengineer the single-dish data analysis system currently in use (a procedurally-written, single-user, single-tier application) into a layered and fully object-oriented application.

DEAP is written in Python and presents two user interfaces - a command line window and a plot window. The GUI portion of DEAP utilizes wxWindows and features a custom widget, created by the NRAO, that integrates PGPlot into wxWindows. PGPlot is a C-callable, device-independent graphics package for making simple scientific graphs and has been used in radio astronomy software for many years (see http://www.astro.caltech.edu/~tjp/pgplot for more information regarding PGPlot). The following is a summary of DEAP’s current features:'

To do this, a generic tool that allows users to view and analyze their data either before importing that data into a data reduction package or as a component that extends the data reduction package is required. Such a generic tool would not necessarily replace commercial data reduction packages, but instead would extend the functionality offered by those tools while giving observers a tool that could also be used standalone.

The fulfillment of this need is the motivation and purpose behind the development of the Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for graphing and analyzing many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net. The DEAP website on Source Forge is currently under development as of this proposal and will be fully available for those interested in learning more about DEAP and downloading their own copy before Py Con 2004.

DEAP is written in Python and presents two user interfaces - a command line window and a plot window. The GUI portion of DEAP utilizes wxWindows and features a custom widget, created by the NRAO, that integrates PGPlot into wxWindows. PGPlot is a C-callable, device-independent graphics package for making simple scientific graphs and has been used in radio astronomy software for many years to produce publication-quality plots (see http://www.astro.caltech.edu/~tjp/pgplot for more information regarding PGPlot). The following is a summary of DEAP’s current features:

%META:TOPICINFO{author="AmyShelton" date="1074028080" format="1.0" version="1.1"}% %META:TOPICPARENT{name="SoftwareReviewFeb04"}%

Generic Data Flagging & Editing Presentation for February 2004 Software Review

Note: This is currently formatted for submittal to Py Con 2004. I will fix the formatting after the proposal has been submitted on January 15.

Author Name:

Amy Shelton

Contact Information:

National Radio Astronomy Observatory
Post Office Box 2
Green Bank, WV 24944
(304) 456-2277
ashelton@nrao.edu

Requested timeslot:

30 minutes

Author Background:

Amy Shelton has worked for the National Radio Astronomy Observatory for the last 5 years, the last 3 of which were spent with the Software Development Division. She holds a Bachelors of Science in Electrical Engineering from the University of Cincinnati and a Masters of Software Engineering from the University of Maryland University College.

Summary of proposed presentation:

Astronomers from around the world come to the Green Bank site of the National Radio Astronomy Observatory (NRAO) to perform radio astronomy observations with the Robert C. Byrd Green Bank Telescope (GBT). Thus, there is a pressing need for a generic data display and analysis toolkit that will provide radio astronomers with a versatile mechanism for viewing and interpreting their observation data. The current data analysis system at Green Bank lacks a generic tool that allows users to view and analyze their data either before importing that data into a data reduction package or as a component that extends the data reduction package. Such a generic tool would not necessarily replace commercial data reduction packages, but instead would extend the functionality offered by those tools while giving observers a tool that could also be used standalone. The fulfillment of this need is the motivation and purpose behind the development of the Data Extraction and Analysis Program, or DEAP. Even though DEAP has been developed with the needs of the NRAO in mind, all assumptions regarding the nature of the data entered into the program were kept at a minimum. This means that DEAP is intended to be useful for graphing and analyzing many types of data, not just astronomical data. Hence, DEAP will be made available as an Open Source product via Source Forge at http://deap.sourceforge.net. The DEAP website on Source Forge is currently under development as of this proposal and will be fully available for those interested in learning more about DEAP and downloading their own copy before Py Con 2004.

DEAP is written in Python and presents two user interfaces - a command line window and a plot window. The GUI portion of DEAP utilizes wxWindows and features a custom widget, created by the NRAO, that integrates PGPlot into wxWindows. PGPlot is a C-callable, device-independent graphics package for making simple scientific graphs and has been used in radio astronomy software for many years (see http://www.astro.caltech.edu/~tjp/pgplot for more information regarding PGPlot). The following is a summary of DEAP’s current features:'

• Interactive, multicolor XY plotting with zooming, error bars, and data flagging capabilities.
• XY plots allow choice of line type (including no line), choice of marker type for data point representation, auto-scaling.
• Creation of titles, axes labels, and annotations.
• Plotting of multiple data sets on the same plot.
• Saving/Retrieving of plots and plot information from binary files.
• Exporting of plots to a variety of graphical formats: BMP, GIF, Postscript, color Postscript.
• Data analysis operations available for data sets include calculation of the mean, calculation of the variance, Gaussian fitting, linear fitting, and polynomial fitting.
• Command line
• User interactive plot window

XY data sets are entered into DEAP via the command line window. The X data and Y data are simply lists of numbers. In fact, all data entered into DEAP and extracted from DEAP are formatted as lists at the command line. The command line window for DEAP is the Python command line plus a set of functions through which the user may interact with DEAP. For example, the user may add a new XY plot to DEAP via the AddXYPlot command that accepts two arguments - a list of x data and a list of y data. The command line enables DEAP commands to be executed one at a time or as scripts (via Python's execfile function). Many commands that are available via the command line are also available via drop-down menus or toolbar items in the plot window. One important issue to note is that both the command line window and the plot window are fully integrated meaning that command issued through one window are reflected both in DEAP and in the other window.

The plot window is focused on handling user interactions with the XY data sets. These interactions include zooming and flagging. Flagging refers to associating an integer value with a data point. By default, all data points in an XY data set are flagged with a value of 0. The user associates flag values with data points by lassoing them with the mouse. Because the flag values are simply integers, the user is free to associate any meaning to the flag values that he/she desires. Flagging is important in working with radio astronomy data because it is an important means of highlighting data contaminated with radio frequency interference or RFI.

DEAP currently runs on Linux and Windows, but plans are underway to allow its use on Mac Os X.

Presentation outline:

1. Motivation for creation of DEAP and applicability to generic data flagging and editting
2. Overview of DEAP functionality
   1. XY plotting
   2. data flagging
   3. data analysis
3. DEAP tutorial through example - importing sample data set, flagging data, fitting data to polynomial and Gaussian
   1. Use of the command line window
   2. Use of the plot window
   3. Saving and/or exporting plots
4. Future development

-- AmyShelton - 13 Jan 2004