Please refer to the BusinessOfficeSystem project page for non-software specific information on the Business Office System (BOS).
During the initial design phase of the BOS project an analysis of requirements was done to give us a stepping stone for our choice of design pattern (implementation technology), classes, and database schema. The results were the completion of the diagram below.
Throughout the BOS, common patterns were used for the relationships between models, controllers, templates, widgets, etc. Much of what was done can also be seen in the TurboGears book and documentation. Here is an attempt to elucidate these patterns through an example of the dreded 'foobar' object.
Suppose there exists an object in the BOS, named 'foobar', that has two variables, 'foo' and 'bar', and that is part of the 'module' module (as opposed to a real object, like the 'item' object in the 'TransactionManagement' module). The object's Class, table, and the relationship between the two could be found in the bos/BOS/bos/model.py file:
foobar_table = Table( ... )
class foobar(object): ...
assign_mapper(session.context, Foobar, foobar_table, ... )
A controller for this object is defined in the appropriate module. So, in our example, there exists a bos/BOS/bos/subcontrollers/module/foobar.py. In this file, exists the definition of the class FoobarController. The instantiation of the controller takes place in the main controllers file, bos/BOS/bos/controllers.py:
foobar = FoobarController()
The class FoobarController must create methods for each url that will be associated with the object 'foobar'. And what are these url's? Well, they all follow this basic pattern:
/foobar/operationFoobar/foobarId
Where foobarId is the Primary Key of the foobar object being operated on, and an example of an operation might be 'edit', or 'save'.
To elaborate off of our foobar example, imagine that we need a web-page for the following:
- viewing the contents of a foobar object.
- editing the contents of a foobar object, w/ save and cancel buttons.
This example would give rise to the following methods in the class FoobarController:
- def viewFoobar(self, *args):
In this case, the args list would contain one element, the foobar id (ex: /foobar/viewFoobar/1). This controller would be responsible for quering for this foobar object, and passing on the object, and any other info, to the appropriate template (see below), which would simply display the 'foo' and 'bar' variables, and include an 'edit' button, which would point to '/foobar/editFoobar/1' (next controller). For example:
@expose(template="bos.templates.module.foobar")
def editFoobar(self, *args):
f = Foobar.get(args[0])
title = "View Foobar "+args[0]
return (foobar=f, title=title)
- def editFoobar(self, *args):
Called by a url such as '/foobar/editFoobar/1', again the args list would just contain the id used to query the appropriate foobar object. However, the values of this object would be put in a dictionary, where the keyword/value pairs specify the contents of the form used to edit this object. The form used to edit this object may be imported from the bos/BOS/bos/widgets/module/widgets.py file (more on widgets below). For our example:
@expose(template="bos.templates.module.form")
def editFoobar(self, *args):
foo = kwargs["foo"]
bar = kwargs["bar"]
f = Foobar.get(args[0])
values = { 'foo' : f.getFoo(), 'bar' : f.getBar() }
title = "Edit Foobar " + args[0]
action = "/foobar/saveFoobar/" + args[0]
cancel = "/foobar/viewFoobar"
form = widgets.editFoobar_form
return dict(form=form, title=title, values=values, action=action, cancel=cancel)
- def saveFoobar(self, *args, **kwargs):
Called by a url such as '/foobar/saveFoobar/1', again the args list would just contain the id used to query the appropriate foobar object. However, the kwargs would be a dictionary, where the keyword/value pairs specify the contents passed from the form used to edit this object. In addition, the validators (again, see next section) assocaited with the form used are called here in the decorator. After the appropriate changes are made to the object, the control is redirected back to the '/foobar/viewFoobar/1' screen:
@error_handler(editFoobar)
@validate(form=widgets.editFoobar_form)
def saveFoobar(self, *args, **kwargs):
foo = kwargs["foo"]
bar = kwargs["bar"]
f = Foobar.get(args[0])
f.setFoo(foo)
f.setBar(bar)
cherrypy.HTTPRedirect("/foobar/viewFoobar/"+args[0])
In many cases the use of templates in the BOS are very straightforward, such as the use of the 'foobar' template by the FoobarController.viewFoobar method. However, the use of widgetLists with forms is worth describing here. For our example, a form is needed to edit the two variables of a foobar object. This could all be described in the bos/BOS/bos/widgets/module/widgets.py file. First, a list of widgets defined in a widget list are created, and validators are used so that error messages appear if blanks are left in the text fields:
class FoobarFields(widgets.WidgetsList)
foo = widgets.TextField("foo", validators=validators.NotEmpty)
bar = widgets.TextField("bar", validators=validators.NotEmpty)
Then the TableForm class is subclassed so that a custom template can be used to display our Cancel button:
class EditForm(widgets.TableForm):
template = "bos.widgets.module.templates.edit_form"
Finally, the Foobar form is instantiated using the above classes:
editFoobar_form = EditForm(fields=FoobarFields(), submit_text="Save")
Part of the Business Office System (BOS), called the User Management module (or BOS-UM), has been created to manage information (i.e. mailing addresses, phone numbers, employee offices, etc.) for NRAO employees, users, and affiliate organizations. Users of the BOS are members of the NRAO business offices, other NRAO employees, and external customers. These users are split up into two user groups, administrators and users. The administrators are limited to a few NRAO employees in the business offices, Computing divisions and the Green Bank Software Development division. Members of the administrator group have complete access the BOS user management module, while members of the users group can only access their own information.
The database schema is based upon the current NRAO-wide Users Database. It contains all of the information contained in the original NRAO-wide Users Database as well as additional information which will expand the potential uses of the database. Through collaboration with the e2e division, we have been able to fit the requirements for the BOS into a new proposed NRAO-wide Users Database design, shown below. This design allows for multiple mailing addresses, phone numbers, and email addresses to reference a single record in the person and organization tables. We chose the name of the organization table (rather than the current term, institution) to encompass companies, primary schools, elementary schools, and high schools in addition to universities. The userAuthentication table serves as a place holder for user accounts, which are technology dependent.
The tables in the proposed schema above cover most of the functionality required for the BOS-UM, however a few additional tables are needed. These tables, shown below, contain information on the location of NRAO employees and are also used in the BOS Reservation Management module.
The intent of an NRAO-wide User Database is to create a central repository which will contain contact information for NRAO facility users. In reality, the NRAO deals with many more people than just users of our telescopes. For example, the NRAO-wide Users Database will be used by the BOS to store contact information for all types of visitors to our facilities - not just observers who will use our telescopes. Other software products will also make use of the NRAO-wide Users Database. The Dynamic Scheduling Software is such a product. For each GBT observer, the DSS will need to keep track of prioritized phone lists that are only validate for a specified date range. For details on how Dynamic Scheduling makes use of the NRAO-wide Users Database, take a look at the DSSSoftwareSuite.
There are a large number of tables in the Transaction module. For this reason, the below schema does not include any tables linked to in other modules, such as person, organization, address, email, phone, and site tables.
The classes found in bos/BOS/bos/model_trans.py do not all simply have a flat one-to-one correspondence to the tables in the database schema. This is mostly due to the shared behavior of the different Transactions (Invoices, Receipts, etc.). The below UML describes the classes for the Transactions module. Note that attributes are not listed (only operations are shown) since these attributes are almost without exception derivable from the database schema.
The Transaction hierarchy deserves special attention here. The first thing to note is that sqlalchemy only supports class heirarchies in it's alpha release at the time of writing. We choose to not adpot one of the several solutions available in the alpha release. Instead, the Invoice, InterdepartmentalSale, and Receipt classes map directly to their corresponding tables, while the abstract Transaction class maps to no table. However, the Transaction class is still used throughout the Transaction module for a variety of purposes; objects of this class are actually created when needed - the data displayed in the manage transaction screen is actually derived from a list of Transaction objects. In addition, there are several templates which are used for all types of transactions, and the fact that the Transaction class contains all the shared operations means that no distinctions in the template are needed.
One would expect that in addition to the current structure, the Receipt class would have been abstracted, with A/R Receipt and G/L Receipt subclasses deriving from them. One could reasonably ask why this was not done. It turns out that in the requirements for the BOS, Invoices and Interdepartmental Sales are almost always treated separately. Thus, there are separate tables for these classes, and separate web pages as well (/invoice/editInvoice vs. /intDeptSale/editIntDeptSale). However, the two types of receipts are sometimes treated separately, and sometimes treated identically. This ambiquity, and the fact that the two types could be easily distinguished simply through the state of the invoice_id field lead us to the decision to use just one table and one class for all Receipts.
The controllers in the transaction module follow a fairly straightforward one-to-one relationship with most of the classes found in the module.
There is only one line in the Transactions MR that mentions validation of account numbers: it just states that the accounts presented as valid account numbers must have been checked within the last 24 hours. So, this leads us free to make this sub-system as complicated and robust as we see fit. Here we present an outline of a system that we believe will be easy to develop, yet will keep account numbers in the BOS validated while informing interested parties on how this is being done.
There are two main components to this subsystem. The first is the downloading of account numbers from People Soft, and the second is reconciliation of these valid account numbers with those that already exist in the BOS:
Every night, Chuck Beverage (MIS division) runs a 'Nightly' through the 'Scheduler' in People Soft that dumps all the
current valid accounts into a text file. The contents of this text file are fully described in the appendix of the Project Charter. Currently, these files are being written to /home/groups/BOS, using the bos user account.
In bos/BOS/bos/utils exists a class, AccountReconciler, which is responsible for reconciling the accounts found in the text file dumped from People Soft and the accounts currently in the BOS account table. Due to the permissions set on the nightly downloads, the AccountReconciler must be run as user bos. This process can be described in the following steps:
- Temporary Accounts Table - The AccountReconciler first determines if the text file exists and is young enough (currently not working). If so, it then simply parses this text file and loads the results into the validAccount table described in the schema. The location of the text file is set in a config file that is passed to the AccountReconciler on instantiation. In addition, the original text file is moved to the same directory with the timestamp appended to the name, so that the next nightly will not append to the original file.
- Determination of Deprecated, Undeprecated and New Account Numbers - The next step involves comparing the new contents of the validAccount table and the current contents of the account table:
- Deprecated & Undeprecated accounts - first the account table is run through, row by row. If the account number does not appear in the validAccount table, then this is a newly deprecated account, and is set to be invalid. If an invalid account, by contrast, does appear in the validAccount table, it is newly marked as valid and is considered 'undeprecated'.
- New Accounts - Then the validAccount table is run through, row by row. If an account does not appear in the account table, it is considered a new account and added.
- Notifications - Wether the AccountReconciler finishes successfully or not, the emailNotifier class is used to notify all subscribers of the results. Subscribers are set in the config file.
Other things to cover.
The live BOS server runs on the Green Bank web server gollum. Since gollum is exposed to the outside world, security is very tight on this machine. It does not connect to any NFS mounts, although the BOS installation directories are accessible from other computers via /home/bos.nrao.edu/active. You must use the linux user bos to write to this directory. Due to the security restrictions on gollum, a local python installation must be used to run the BOS server (/home/bos.nrao.edu/active/smeagol). Source /home/bos.nrao.edu/active/smeagol/BosPython.bash or BosPython.csh to use smeagol.
Perform the following steps to install a new release.
- cd /home/bos/integration/bos Note: where integration is the version to be tagged
- Verify that there are no locally modified files in bos:
- export CVSROOT=/home/gbt2/repository
- cvs status
- Tag the bos version
- cvs tag release_1_0 (assuming new version is 1.0)
- cvs tag -b release_1_0_patches
- su bos
- cd /home/bos.nrao.edu/active/bos
- mkdir BOSRELEASE (BOSRELEASE = 1.0 in this case)
- cd BOSRELEASE
- First type $ export CVSROOT=/home/gbt2/repository
- For creating a new integration (using 'trunk'): $ cvs checkout -P bos
- For creating a new release (using a 'branch'): cvs checkout -P -r release_1_0_patches bos
- cd bos/BOS
- cp ../release/bos/BOS/prod.cfg .
- cp ../release/bos/BOS/accountReconciler.cfg .
- edit prod.cfg to specify the database connection by setting the sqlalchemy.dburi
- sqlalchemy.dburi="mysql://username:password@hostname:port/databasename"
- cd /home/bos.nrao.edu/active/bos
- rm release
- ln -s BOSRELEASE release
- Restart the bos server
- ssh gollum
- sudo /etc/init.d/bos restart
- su bos
- cd /home/bos.nrao.edu/active/bos/release
- cp the files you've updated from your sandbox to their corresponding directories.
- cvs commit the files you've updated
- Restart the bos server
- ssh gollum
- sudo /etc/init.d/bos restart
- Go test it at bos.nrao.edu!
There are several ways to deploy BOS behind an apache web server:
-
mod_rewrite
-
mod_proxy
-
mod_python
Each has advantages and drawbacks. mod_rewrite and mod_proxy both work by having Apache forward requests to the CherryPy server port. These two methods are easy to set up and apparently are fast enough for production purposes. Apache can be set up to handle all static requests (such as requests for file downloads, etc.) leaving CherryPy to handle only the dynamic content. The drawbacks to these two approaches are:
- CherryPy is receiving requests from
localhost, therefore you must set up special headers to get the actual Host header of the request.
- Apache is not responsible for keeping BOS running. Some sort of independent task, such as
cron or supervisor (see man page) must be used instead. Alternatively Apache's error handling mechanism can be customized to launch a CGI script on error; this CGI script could re-launch BOS (M. Ramm, K. Dangoor, G. Sayfan, 2007, pg. 362).
- There is some overhead in the forwarding process.
mod_python provides a more appealing way to deploy BOS because it is slightly faster than the other two methods, and Apache starts/re-starts the BOS process. The disadvantages are that Apache runs multiple processes (of BOS?), something that probably does not affect BOS but is good to know, and that it is harder to configure and get running right (Ramm, 2007, pg 363).
The BOS team chose the mod_python option, and indeed it has proven to be difficult to set up so far. The directions in the Ramm book are obsolete, so we chose to use the directions on the TurbogGears web site. We encounterded problems immediately, even with a simple, do-nothing TurboGears application. Further troubleshooting and web searches revealed that the problem was that Python and Apache are using two different versions of Expat. Apache is using 1.95.7 and python is using 1.95.8. This causes the mod_python session to terminate with a segmentation fault (sig 11). This is where things stand as of 06/05/07.
Our options at present are:
- Upgrade the system Expat on
colossus to match the one used by the Python in /opt/services/python. This is unfortunately a much more involved upgrade than one would think, because the new version of Expat requires a newer version of the C library. This is a core library used by just about everything else, which means the entire system must be upgraded. I've offered to allow goauld to be upgraded (to RHE5) but Chris prefers that Wolfgang do these things and Wolfgang is out of the country for the next 3 weeks.
- "Downgrade" python to compile with the 1.95.7 version of Expat. This would require investigating how to reconfigure the build for the Python installation on colossus. This may be our best option if we wish to proceed with
mod_python
- Use one of the other methods.
After going with option 2 we ran into other issues with loading anything in the static directory where all our JavaScript, CSS, and Images are located in the source tree. Also, the server seemed to be very slow for unknown reasons. Online research (TG Web site) indicated that mod_python was not the recommended way to go due to the performance issues that we seemed to be experiencing. At this time we decide to attempt option 3 with mod_proxy.
- Wolfgang recompile apache on colossus to enable mod_proxy, which comes standard with apache2.
- We added the following to the apache configuration httpd.conf
<VirtualHost *>
AddDefaultCharset utf-8
ServerAdmin you@yoursite.com
ServerName www.yoursite.com
ProxyPreserveHost On
DocumentRoot /export/raid/www/nrao
# this prevents the follow URL path from being proxied
# ProxyPass /static !
# setup the proxy
<Proxy *>
Order allow,deny
Allow from all
</Proxy>
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
</VirtualHost>
- We started the cherrypy server on colossus port 8080 (the test server runs on 8086)
- Point your browser to http://colossus.gb.nrao.edu Done!
- Log on to colossus as
monctrl:
- source
/home/gbt7/galahad/McPython.bash
- cd to the top of test source tree,
/home/bos, and into the test version that you wish to run. For example, 0.2/bos/BOS.
- Make sure the source is updated with CVS to the version of the code that you want to run.
- If you are checking out a new pre-release
- Edit the
def.cfg to point to the "bos" MySQL database and set server.socket_port = 8086
- If new pre-release or changes made to models
- Make sure that the sponsors are okay with data loss.
- Drop all tables in the "bos" MySQL database
- run
tg-admin sql create
- run
tg-admin shell InitTestDB.py
- run
nohup python start-bos.py &
- Test with a browser: point to http://colossus.gb.nrao.edu:8086 and ensure that the BOS login screen appears.
- log out of colossus
- Log onto colossus as
monctrl:
- Get the pid for the python process running the test server.
ps axf | grep start-bos.py
- Kill the process.
ps should have displayed three PIDs:
[monctrl@colossus ~]$ps axf | grep start-bos.py
14821 pts/1 S+ 0:00 \_ grep start-bos.py
13752 ? S 0:07 python start-bos.py
13756 ? Sl 1:24 \_ /home/gbt7/galahad/bin/python start-bos.py
Ignore the grep PID. Kill (with default signal) the PID for the second python (the one with the full path specifier). The other will go as well. In this case:
kill 13756
Revision r1.20 - 01 Oct 2007 - 14:48 GMT - MikeMcCarty
Parents: WebHome > SoftwareDocs
|
Content copyright © 1999-2007 by the contributing authors.
All material on this collaboration platform is the property of the contributing authors.
|
| |
