Penn Array Native Manager

Modification Request #3C406 (C4 2006)

NOTE: Work In Progress!

Penn Array Native Manager

1. Introduction

The existing Penn Array Receiver (PAR) host interfaces with the PAR hardware via two serial ports and a PCI-based card which uses DMA to capture multiple data streams into memory for processing. The PAR software is currently based upon a java-based framework known as IRC. The software that controls the specific hardware used in the Penn Array project is known as the "Mark III" software. The "House Keeping" program is a standalone java program that controls and monitors the cyrostat of the Penn Array Receiver.

The IRC provides many functions to support device management and data streaming. It also includes a mechanism for plotting various data streams. NRAO additions to the Mark-III are a FITS archiver, and a rudimentary custom interface to the ygor-based receiver manager.

The intent of this MR is to document the initial design of a native ygor-based manager (YPAR), which will replace the existing IRC/Mark III software.

2. Background

2.1 In the Most Basic Terms, How Does the PAR Work?

The PAR is arranged as a set of elements in a 8x9 matrix, with eight "columns" and 9 "rows". Each column has a dedicated digital feedback (DFB) card which services all rows in a given column, on a time-multiplexed basis. All of the DFB data-streams are linked to the NIST PCI card (poorly named) which takes the data, and writes into the host's memory buffers using DMA. The data ends up in an interleaved fashion (see example) in the host buffers. System configuration is performed by writing values to the host serial ports.

2.2 References:

Along with the entire PAR wiki, the following links have relavent information:

3. Requirements

The number of NIST channels shall be configurable.
The mapping of "virtual" pixels to physical pixels shall be configuable.
Co-adding to reduce data output rate shall be available.
The YPAR software must parse the NIST card datastreams, and make the result available for objects to subscribe.
Control parameters must be made available to control the device fully. (See section on parameters.)
The YPAR must be able to write FITS files during scans in the existing PAR format.
It should not be necessary to run the manager as root.
Special consideration should be given to quick-look/monitoring issues.
The system shall be able to emulate the NIST PCI card hardware to provide a rough capability to run scans without the actual hardware.
The system must provide an interface to the existing JAVA housekeeping program or replace its functionality.
The system must provide an interface to the existing tuning algorithms or replace their functionality.
The software must initialize the hardware to a known state based on configuration file and default values.
Be able to specify the following values as parameters:
- ADDR and DFB NSTEP
- NIST channel mask
- NIST card shift delay
- LSYNC
- NSAMPLES
- SETTLE
- physical columns to NIST card mapping (i.e. net effect of DFB, mux and NIST card wiring)
- ADDR row lookup table
- ADDR and DFB row delta (LUT step is different in RAW mode?)
- ADDR and DFB row start
- ADDR enable switching
- ADDR enable driver
- ADDR reference voltage
- stream mode (i.e. standard, raw, empty)
- bit mode (i.e. 16/22)
- DFB mode (signal generator, normal feedback, last DAC value, default DAC value)
- signal generator start, stop , hold, step, drop, signal shape(?)
- DFB general reset
- DFB reset for clock frame counter, master, signal generator
- DFB status request
- DFB stream header rate (default = 5)
- DFB transfer rate (default = 32)
- DFB start enable (?)
- DFB SAE scale factor, coadded scale factor
- DFB Thresholding:
  - proportional gain
  - target lock point
  - prediction factor
  - threshold
- DFB and Feedback/Bias/PreAmp target dacValue
- DFB default DAC value
- Feedback/Bias/PreAmp channel
- frame rate
- clock divider
- clock start
- clock stop
- IF card frame sync delay?
- CoAdder? Properties
  - RMS of DAC and SAE
  - CoAdding? based on: * number of samples * time to coAdd samples in microseconds
  - time stamp used for coAdded data: avg, last, first
- Raw Frame dump to file
  - file dir/name
- basic manager parameters (asap, starttime, ...)

Additional Properties (Configurable through a start up configuration file):
- Ramp Bias:
  - Start count, counts
  - Finish count, counts
  - Step size, counts
  - Delta time, seconds
  - Bias card address, none
  - Bias card channel, none
- Squid Tuning params:
  - bias DAC, counts
  - feedback1, counts
  - feedback2, counts
  - deltaT, seconds
  - bias card address, none
  - bias card channel, none
  - feedback card address, none
  - feedback card channel, none
- NIST Card
  - channel mask, mask
  - shift delay, clocks
- DFB Card
  - NIST channel number
  - card address (not used)
  - crate slot location (i dont think we need this main.MelindaMello)
  - mux column (i dont think we need this main.MelindaMello)
  - name
- Bias/PreAmp Card
  - card address
  - channel
  - dacValue
- Interface card
  - input 0 sample rate
  - input 1 sample rate
  - input 2 sample rate
  - input 3 sample rate
  - card address (not used)
  - nist channel number
  - crate slot location (i dont think we need this main.MelindaMello)
  - mux col (i dont think we need this main.MelindaMello)
- Address card
  - LUT
  - LUT address params (start, delta)
  - NSTEP
  - card address
- Crate Serial port
  - baud rate
- Tower Serial port
  - baud rate
- Low-level commands
  - start all ( MelindaMello I moved this to the requirements section..."must initialize hardware to known state")

Modal Commands:
- Start/Stop Clock (MelindaMello, I'm not sure why these are separated, I duplicated these in the parameter list above)
- Reset DFB's
- Tune?

4. Design

4.1 Architecture

Using a four-layer architecture pattern, we can divide the complete system into a view layer; an application model; a domain model; and an infrastructure model. The view layer deals with presentation details to the user (i.e. gadgets and display items like graphs or text fields) and is not part of the YPAR. The application layer mediates between the external interface components and translates the messages that they understand into messages understood by the objects in the domain model. The domain layer represents the custom internals of the system. The Infrastructure layer consists of resource allocation and mediation constructs, and may also contain items which are supporting elements, such as the base Manager, Sampler and Messaging components.

The Ygor manager fulfills much of the application model layer. It provides a standardized state model for running scans, and presents a mechanism to communicate with the presentation layer through the Panel interface.

Basic Object Architecture

Figure 1

4.1.1 The Domain Model

I've chosen to divide the domain model into at least three major parts: elemental processing; subsequent processing; and control.

4.1.1.1 Elemental Processing

The goal of elemental processing is to capture data from the system (with implied real-time requirements), and produce complete frames with associated pixels and configuration information. In this part of the system, processing steps are relatively invariant, so I think this portion of the system can be tightly coupled to make the system as efficient as possible. Starting with the NIST card, data frames are DMA'ed into shared memory buffers. The user-level application is triggered each time a new buffer is available for processing. At this point the data is still interleaved together, before processing/parsing, the data is first de-interleaved. A buffer may contain a frame which spans two buffers, so there is a problem of partial buffers. A bit of bookkeeping is required to partiion buffers properly. Once complete frames are availble, the separated streams can be parsed.

The basic 'atom' of data is represented by the DataElement object, which can be either a Pixel or a data from the Interface card. These atomic units of data are grouped together in a temporal sense by the Frame object. Multiple Frames may reference configuration state information via the FrameSet object. The FrameSet acts as an association object between frames of data and an instance of configuration information, with the intent to reduce memory footprint.

is the Frame-set/Frame/Pixel construct. A Pixel contains (TBD) information, as well as a physical row/column. A Frame associates a number of Pixels with a common timestamp, frame count, and Frame-set. A Frame-set associates one or more Frames with an instance of configuration data. ( Interface card data is treated as a special case of Pixel data? )

4.1.1.2 Subsequent Processing

Here the stages of processing and the goals are more diverse. One primary goal is to deliver data to a FITS writing endpoint, to record the sensor data. However there may be other parallel processes (such as a real-time monitor) and tuning. In this stage the publish-subscribe (AKA the Observer) pattern is used to distribute the data streams, and the strategy/algorithm patterns are used to allow any number of variations on processing. This dual pattern allows the definition of consistent interfaces, while allowing the algorithms to vary independent of the Distributor and DataStream.

The Distributor is the 'publisher' portion of the publish-subscribe pattern. It may have zero or more DataStream subscribers, with each subscription having a number of attributes like coadding rate, and event filters for Manager state events and configuration information updates. Each subscriber has an associated strategy for how to handle the data received. A Fits writing strategy would either directly write or indirectly manage the writing of YPAR FITS files. A monitoring strategy may simply pass data onto a gui application, and therfore be uninterested in Manager states changes. For efficientcy, coadding is implemented by the Distributor, and coadded data streams of the same rate are served by common CoAdder objects.

4.1.1.3 Control

Here the central theme is the Ygor Manager, and software abstractions of the PAR hardware components. The Panel presents the standard Ygor Control interface to the rest of the system; the manager tracks scan state, device settings, and overall status.

Each major hardware component has a software object abstraction, coupled with a resource-management object. For example the Tower and Crate objects handle resource management and arbitrate access to shared resources (e.g. access to the serial line). The abstraction of hardware by the FeedbackCard, DFBCard, InterfaceCard, and Timing objects is useful for value conversion (e.g. something that is logically a boolean value might be mapped to a bit pattern); emulation of a 'read-back' even though the hardware does not support it, and simulation when devices are not present. These card objects are considered to be aggregated into the Crate, because the cards are considered to be 'part-of' the Crate. [1]

The NIST card is treated separate from the Crate, simply because it does not share common resources. It is treated directly because the card (and it's interface) are tightly coupled to the inner workings of the Parser subsystem.

A key item to note is that in the YPAR, there is a concept of 'requested' configuration and 'indicated' configuration values. Periodically the NIST card data contains data packets which indicate the actual hardware settings. If a value is requested to be changed, the command is sent via serial line without any acknowlegement. But the state is not considered 'changed' until the NIST card data indicates the change. This presents a transient situation during which the requested and indicated configuration may be different.

Notes:

[1] At this point I would not advocate making each hardware component a sub-manager, it just does not fit. Instead I would make the number of hardware items run-time configurable (e.g. number of DFB's).

4.2 Issues/Risks:

Time-stamping (Each Frame is tagged with a free-running counter tag which must be aligned/resolved with system time to be able to record an absolute time.)
Parsing performance
Partial buffers
Working around (known) firmware oddities

4.2.1 Time-stamping

Currently time-stamping is done by examination of the frame count, and then tied to an absolute time by computing an offset from a previously time-tagged frame count. Also a periodic stream-core message is included in the data, which contains a 64-bit counter, which runs at a 50 MHz rate. A problem here is that none of these clocks are sync'ed to a standard, so small pertubations will be seen on every re-sync'ing of the frame count if the clock frequency is not exact.

4.2.2 Parsing Performance

Depending upon the system configuration, it may be a challenge to process all of the data at the higher Frame rates. It should be noted that the testbed "spider" has half the memory of the actual PAR ("green").

4.2.3 Partial Buffer Handling

Since the data is basically asynchronous to the NIST card buffers, some special care should be taken during parsing to splice frames between buffers. The problem is sort of a re-pagenation problem. Buffer remnants of incomplete frames must be saved and prepended to the next buffer.

4.2.4 Working Around Firmware Bugs

TBD.

4.3 Significant Differences between YPAR and IRC/Mark-III

The number of kernel-based buffers will be configurable from the start. Mark-III currently only permits two.
The YPAR will be careful to avoid stability problems, like the hard to reliably reproduced garbage collection hangs.
The YPAR will be server-only, running as a daemon. User-interfaces, plotting packages, etc. will be built from a mixture of Cleo, Astrid and GBTIDL toolkits.

4.4 Patterns Used

Ygor Manager-Controller pattern
The Observer pattern in concert with the Strategy pattern will be used to internally provide a publish-subscribe mechanism for data flow at the later stages of processing through the manager.
Other minor design patterns used include the state pattern; the iterator, and the singleton.

5. YPAR Parameters

YPAR Low Level Control Parameters

Interface	Parameter	Feedbk	Pixel Addressable	Col Addressable	Dependencies	Range (min.. max)	Default Value	NOTES
Crate(CLOCK)	clock divider		NA	NA	lsync, #DFB/Interface card, dsync (TBF)	>31 (TBF what makes sense)
Crate(CLOCK)	clock start/stop		NA	NA		start/stop (bimodal)
Crate(DFB)	bit mode	Yes	No	No		16,22	16
Crate(DFB)	coAdd scale factor[32] (B)	Yes	Yes	Yes		0..31	0
Crate(DFB)	default dac value		Yes	Yes		0..16383	1
Crate(DFB)	enable all timing?							TBF
Crate(DFB)	general reset		No	Yes		true/false (1 shot)	FALSE
Crate(DFB)	prediction factor[32]	Yes[32]	Yes	Yes		0..2 (float)	0	9
Crate(DFB)	proportional gain[32] (alpha)	Yes[32]	Yes	Yes		-256..255	255
Crate(DFB)	SAE scale factor[32] (C)	Yes	Yes	Yes		-13,29	8
Crate(DFB)	sig gen hold	Yes	Yes	Yes		0..31	30	18
Crate(DFB)	sig gen shape /drop	Yes	Yes	Yes		triangle=0,sawtooth=1	triangle	18
Crate(DFB)	sig gen start	Yes	Yes	Yes				18
Crate(DFB)	sig gen step	Yes	Yes	Yes		0..16383	1	18
Crate(DFB)	sig gen stop		Yes	Yes		0..16383	16383	19
Crate(DFB)	signal mode/DFB active		Yes	Yes		siggen,feedback,lastVal,DACVal	feedback
Crate(DFB)	target[32]	Yes[32]	Yes	Yes		0...4095	2048	6
Crate(DFB)	threshold[32]	Yes[32]	Yes	Yes		0..655355	0
Crate(DFB)	start enable	Yes	No	Yes		true/false	TRUE
Crate(DFB) Crate(INTFC)	reset sig gen		No	Yes		true/false (1 shot)	FALSE	3
Crate(DFB) Crate(INTFC)	lsync	Yes	NA	No	clock divider, #DFB/Interface card, dsync (TBF)	1..1048575		16
Crate(DFB) Crate(INTFC)	number of samples	Yes	No	Yes		1..1048575 or 1..lsyncdiv – settle -3		7
Crate(DFB) Crate(INTFC)	request status	Yes	No	No		true/false (1 shot)	FALSE	5
Crate(DFB) Crate(INTFC)	reset clock frame counter		No	Yes		true/false (1 shot)	FALSE
Crate(DFB) Crate(INTFC)	reset master		No	Yes		true/false (1 shot)	FALSE	4
Crate(DFB) Crate(INTFC)	settle	Yes	No	Yes		1..1048575
Crate(DFB) Crate(INTFC)	stream header rate	Yes	No	No			5	5
Crate(DFB) Crate(INTFC)	stream mode	Yes	No	No		standard, raw, empty	standard	5 and 6
Crate(DFB) Crate(INTFC)	transfer rate / DSYNC	Yes	No	Yes		32..4095	32
Crate(INTFC)	frame sync delay		NA	NA		enable/disable	disable
Crate(INTFC)	num row selectsf/frame		NA	NA				13 and 14
Crate(INTFC)	raw mode #frames		NA	NA	dfb mode			13
Crate(INTFC)	raw mode skip	Yes	NA	NA	dfb mode			13
Crate(INTFC)	sample rates (qty 4)		NA	NA		0..255
Crate(PREAMP),Crate(BIAS)	dac value		Yes	Yes		0..65535
NA	CoAdd mode		TBF	TBF		numberofsamplers/time	numberofsamples
NA	CoAdd number of samp		TBF	TBF		TBF	TBF
NA	CoAdd rate		TBF	TBF	lsync,clock divider,#DFB/Interface card,dsync (TBF)	TBF	TBF
NA	CoAdd RMS of dac/sae		TBF	TBF		enable/disable	disable
NA	CoAdd timestamp mode		TBF	TBF		last,first, avg	TBF
NA	DFB to NIST channel #							15
NA	raw frame to file		NA	NA		enable/disable	disable
NA	raw frame to file filename		NA	NA		STRING of valid path and filename	/home/gbtlogs/par/PAR_RAW.TIMESTRING
PCI	channel mask		NA	NA	DFBS in cabling file	1..(2 pow 8 )	all enabled
PCI or SW only(TBF)	shift delay		NA	NA		0..50
Tower(ADDR)	enable driver		NA	NA		on/off	on
Tower(ADDR)	enable switching		NA	NA		on/off	on
Tower(ADDR)	reference voltage		NA	NA		0..16383
Tower(ADDR) Crate(DFB)	lut start	Yes	No	No	nstep, lutdelta	0..31	0	11
Tower(ADDR) Crate(DFB)	lut delta	Yes	No	No	nstep, lutstart	0..31	1	11 and 12
Tower(ADDR) Crate(DFB)	lut entries[32]	Yes[32]	NA	NA		1..32	1,2,...32	10
Tower(addr) Crate(DFB)	nstep	Yes	No	No	length of lut entries, lutstart, lutdelta	1..32	9	11
????	frame rate

Notes for YPAR Low Level Parameters

1.Must check what dfb commands should be sent to interface card. Some in ctl spec make no sense

2.Most commands share a register with other commands, with each command assigned to different bits of a word.

Because we don't have read back capabilities in the hardware, we can't use the MCBMultiBit? type of algorithm

that we use in YGOR. This will either cause us to create a lot of getParameters() calls in the activate

mgr methods which generate dependencies(MC verify), or some other method to account for this

3.. Interface card does not have sig gen yet its in control spec.

4. resets the fpga

5. Interface card does not send streams, why is it listed in ctrl spec !!

6. This is the ADC lock point.

7. firmware coadding

8. legally possible to address per col

9. Value is changed from float to nearest bit value

10. In tower individual entries are sent 1 at a time. For DFB=per dfb/chan

11. legally possible to address per col, but makes no sense to do so

12. LUTSTART is possibly different in raw mode? (TBF)

13. In ctl spec..not in IRC? TBF

14. Not used for DFB

15. Each DFB and Interface card is assigned a NIST ch #

16. TBF what makes sense for min max ranges

17. Raw Mode Steps Skipped is divided between stream core and stream config!

18. Feedback is only available for 1 value, not sure which LUT it corresponds to

19. Ctrl spec has 2 “sig gen step” feedbacks, i think 1 should be “sig gen stop”

20. Unless otherwise noted values are integers

21. Legal channels for channel addressable parameters are 0..7

Feedback Parameters without Control

Name

SyncFault

card address of stream origination

firmware ID

clock high

clock low

frame counter

# command echo words

NOTES:

This sheet contains feedback parameters with no associated control parameter. Feedback parameters are returned via DFB cards only.

Interface card does not provide feedback streams.

6. Deployment Checklist

What has to get done to integrate this completely into the system. This checklist must be completed before Cycle Integration Testing begins.

Communication with Computing group needed?
What documentation needs to be updated?
Training Needed? Is this being released to staff astronomers or everyone right now?
Notification Needed?

7. Test Plan

A detailed plan is not yet available, however, it would be advantageous to be able to run both IRC and YPAR in parallel. This may be possible with some extra effort. It would permit a direct comparison of processed data between the two systems, thereby confirming correct processing, or exposing problems.

Written	symbol - name - date
Checked	symbol - name - date
Approved by Sponsor	symbol - name - date
Approved by CCC	symbol - name - date
Accepted/Delivered by Sponsor	symbol - name - date

Symbols:

Use %X% if MR is not complete (will display )
Use %Y% if MR iscomplete (will display )

CCC Discussion Area

-- JoeBrandt - 07 Jun 2006

Attachment:	Action:	Size:	Date:	Who:	Comment:
frame_pixel_construct.png	action	9734	13 Jun 2006 - 20:13	JoeBrandt

Topic ModificationRequest3C406 . { Edit | Attach | Ref-By | Printable | Diffs | r1.13 | > | r1.12 | > | r1.11 | More }

Revision r1.13 - 02 Aug 2006 - 20:38 GMT - JoeBrandt Parents: PlanOfRecordC42006	Content copyright © 1999-2007 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.