Fix Inclinometer Data Collection Problems

Modification Request #1 (C2 2007)

1. Introduction

The Inclinometer consists of four tilt-sensitive devices, a terminal server and manager, documented in the original MR. There have been reports of problems with the recorded inclinometer data, which are noted as:

The device data timestamp is either duplicated for two different data samples, or filled in as NaN values
The device data fields are filled as Nan values
Increased accuracy of time-tagging is desired.
Decreased error rates are desired.
Communication errors such as a non-response for a measurement query, or a garbled message.

2. Background

The Inclinometer devices communicate via a serial line connected into a network-based terminal server. An M&C manager communicates with the inclinometer via a socket. The device protocol is of the request-reply type, each data sample is reported only after a request is received. Since the inclinometers are not synchronized with the host system, the manager requests and receives data at twice the integration rate specified to the device. This assures that data samples are not missed. Each unique data sample (not the same as each request-reply) is marked with a sequence number. This helps to identify and timestamp the data, and filter out duplicate samples.

3. Requirements

3.1 Fix or minimize the problems described in section 1.
3.2 Modify manager to support acquisition of data at independent rates on pairs of inclinometers.

4. Findings

The section describes debugging work and findings to date.

4.1 Network Related Issues:

The inclinometer interfaces with a network based terminal server. Initially the inclinometer manager was using TCP for communications, but this presented a few problems:

The terminal server in TCP configuration often sends a NULL at the end of a received message
The controlling computer's operating system sometimes combined packets (commands) into a single packet, which in turn would cause data loss on the serial line. Setting the TCP_NODELAY option on the socket should have resolved this behavior. Tests indicated that it did not.

4.2 Device Timestamp Generation

The existing code sometimes produces either duplicate timestamps, or timestamps where the interval is twice the expected value. This was caused by the algorithm used to timestamp the data. It was using an estimate of the integration interval which at times resolved to the incorrect interval. (See section 5.2.3 below.)

4.3 Scheduler and Other Timing Issues

Each device query is a serial operation, which takes 20-23 milliseconds. With four devices on a single serial link, integration times of l00ms are not possible.

5. Mitigation Efforts

A number of changes have been made to the inclinometer code and are detailed in the sections below.

5.1 Hardware Reorganization

Initially four inclinometers shared a single serial link. In order to get better throughput, the inclinometers were paired together on two links, one for each elevation bearing assembly. This allows the two sets of devices to be queried in parallel, doubling the effective serial link bandwidth.

5.2 Manager Reorganization/Changes

The original device interface was written in python and called from C++. In order to maximize throughput, the code was replaced by a set of C++ classes, and threads were dedicated to each pair of inclinometers. (The python based module was removed because python is not re-entrant and could not be used with the revised threading model.)

5.2.1 Device Interface

The terminal server has been reconfigured to use UDP, which reduced the communications latency and placed the timing closer to the intent of the Inclinometer manager.

5.2.2 Sampler Reorganization

The samplers originally had all four inclinometers sampled into a single sampler. This presented the problem of collation, and NAN-filling when inclinometer errors occurred. The current version has each data value broken out into a separate sampler. This normalizes the data format, and meshes well with the log_collator design. (The log_collator eases the merging of sampler-streams.)

5.2.3 Time Stamp Algorithm

Periodically the host sends a re-sync command to adjust the start of integration to a 'well-estimated' value. This is used as an epoch from which sample timing is extrapolated from the sequence number. A modification has been put into place which instead of estimating the current integration interval from the host clock, instead uses the sequence number produced by the inclinometer. By counting the number of sequence number progressions (0..15,0..15,etc.) and thereby creating a 'sample number'. (i.e. the number of integrations since the most recent clock resynchronization.) Each time a re-sync command is sent, a sync_time is calculated as shown below, and the sample_number is reset to zero.

lastSyncTime = time_sync_cmd_sent + median_network_delay/2 + message_transmit_delay - integration_time/2
sample_number = number_of_progressions * 16 + sequence_number

The data time tag is then given by:

sample_time = lastSyncTime + sample_number * integration_time

5.3 Manager Independent Rate Changes

Parameters have been reorganized to support independent acquisition rates on pairs of accelerometers. Each pair of inclinometers (i.e the X,Y pairs on each elevation bearing) has a parameter to control the inclinometer acquisition rate. The old parameter integrationPeriod has been changed to integrationPeriodOne (for the X1,Y1 inclinometers) and integrationPeriodTwo for the X2,Y2 inclinometers.

The check routine on each parameter has been changed to verify the value is an even number of milliseconds, because the inclinometer devices can only support rates divisible by two.

6. Communications Error Rates

Although a number of changes have been made, one pair of inclinometers continues to produce 2-10 times the number of errors of the other pair. The cause of the errors are not precisely known, they simply appear as either (1) no response for a measurement request or (2) a garbled reply. No resolution to this problem has been found.

7. Deployment Checklist

What has to get done to integrate this completely into the system. This checklist must be completed before Cycle Integration Testing begins.

Communication with Computing group needed? No
What documentation needs to be updated? Inclinometer Documentation
Training Needed? Is this being released to staff astronomers or everyone right now? Released for PTCS staff use
Notification Needed? No

8. Test Plan

8.1 Internal Testing

To verify the requirement in 3.2 (independent acquisition rates) follow the procedure:

connect to the device using devexplorer
set the parameter integrationPeriodOne to 1000 (milliseconds)
set the parameter integrationPeriodTwo to 500 (milliseconds)
verify the x1data, y1data, x2data, and y2data samplers are running.
examine data from each sampler stream. x1data and y1data sample_time values should be very close to 1 second apart. (Note use the sample_time field, not DMJD.)
examine data from x2data and y2data. The x2data and y2data sample_time values should be very close to 0.5 seconds apart.
Set both integrationPeriodOne and integrationPeriodTwo to 166.
Repeat the procedure above to confirm all inclinometer sample_time fields are approximately 166ms apart.

8.2 Sponsor Testing

The best test will be the result of analysis of the data, comparing data from the accelerometers and servo systems. This has been performed twice by Kim.

8.3 Integration/Regression Tests

Verify proper operation by checking the sampled data:

verify the sample_time and DMJD fields track each other on average, with a cyclic variation of about +1 and -2 integration periods.
verify all data streams are updating properly by checking for log files in the following directories:
- /home/gbtlogs/Inclinometer-Inclinometer-x1data
- /home/gbtlogs/Inclinometer-Inclinometer-y1data
- /home/gbtlogs/Inclinometer-Inclinometer-x2data
- /home/gbtlogs/Inclinometer-Inclinometer-y2data
- /home/gbtlogs/Inclinometer-Inclinometer-echoData

Signatures

APPROVED: I acknowledge that my request is fully contained in this MR, and if the SDD delivers exactly what I specified, I will be happy.

ACCEPTED: I acknowledge that I have validated the completed code according to the acceptance tests, and I am happy with the results.

Written	- JoeBrandt - 12 Apr 2007
Checked	- PaulMarganian - 19 Apr 2007
Approved by Sponsor	- JohnFord - 19 Apr 2007
Approved by CCC	- RonMaddalena - 4 May 2007
Accepted/Delivered by Sponsor	- JohnFord - 31 May 2007

Symbols:

Use %X% if MR is not complete (will display )
Use %Y% if MR iscomplete (will display )

CCC Discussion Area

-- JoeBrandt - 22 Feb 2007

Topic ModificationRequest1C207 . { Edit | Attach | Ref-By | Printable | Diffs | r1.17 | > | r1.16 | > | r1.15 | More }

Revision r1.17 - 31 May 2007 - 15:40 GMT - JohnFord Parents: PlanOfRecordC22007	Content copyright © 1999-2007 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.


NRAO Home > Green Bank \| Wiki Topic: GB > Software > PlanOfRecordC22007 > ModificationRequest1C207

Changes \| Index \| Contents \| Search \| Statistics \| Go