Micro Monitor Software

Introduction
The NSLS control monitor is a real-time software that drives the micro processors at the NSLS facility. It performs hardware control, data acquisition, error reporting and closed loop algorithms in real-time. It provides a standard interface to the high level programs on the host computers irrespective of the differences at the hardware equipment level. This paper describes the various functional blocks constituting the monitor software.

Micro Hardware Configuration
The micro system at NSLS uses a VME crate with a Motorola 680x0 series or a Power PC single board computer in the ethernet controller, 1 megabyte battery backed-up memory and a GPLS (General Purpose Light Source) board which provides extra timers, serial ports, bus-interrupter module, video display generator, software selectable switches, diagnostic LEDs and a real time clock with up to 0.1 micro second resolution. Further hardware I/O interface requirement for any system is dependent on the equipment under control. Wherever feasible, commercially available boards are used. The hardware I/O interfaces include ADC, DAC and Bit-IO cards in the VME crate. Instruments with GPIB and RS-232 interfaces are also supported. Using Kinetic system VME to Camac interface boards, a single micro can control and monitor instruments within 90 meters distance. Two subsystems monitor 500 to 1000 ADC signals. Since all the I/O boards cannot be housed in a single VME crate, bus repeaters have been used; they are also used to map a memory segment of one micro into another micro for high data transfer rate. Remote VME crates have been interconnected using bus repeaters with fiber optic cable. Some micros have two CPUs in one crate. The master CPU interfaces with the workstations and the slave CPU collects data and performs time-consuming calculations.

Micro Software Objectives
The monitor software consists of a set of system and device-specific application tasks and interrupt handlers. The prime objectives in developing the software are: (1) present a standard equipment interface to the host level application programs, (2) provide an easy environment for developing equipment-specific modules in the micro, (3) define a clear interface between the system and device control tasks and (4) incorporate all the functions necessary for handling standard hardware devices.

Logical Device Abstraction
The complexity and diversity of the various hardware used in the facility makes each micro unique at the hardware control level. However to achieve the above mentioned objectives, it is necessary to define a standard set of commands for control and data acquisition and present a standard and uniform hardware interface to the host level applications. The monitor views all hardware signals associated with any micro as a set of logical devices with definite attributes. A primitive set of device types are analog and digital input types (for read-only signals) and analog and digital output types (for settings only). Composite types are derived by combining one or more primitive types. Array types represent data blocks or character strings encountered in some hardware units (example: RS-232 interface). Table 1. lists the NSLS types and their attributes with examples. A hardware can be represented by a primitive or a composite type. A device with multiple signals can be represented by a group of primitive devices or by a composite device. A logical device need not necessarily represent a hardware. There can be soft devices to represent variables that may be used to control an algorithm or to represent a result of a calculation (example the UVLIFETIME) or a global pseudo device which in turn controls or gets data from a group of hardware signals. A standard device format (NSLS format) is used for the logical devices. This format contains a number of fields for storing various information about a logical device. The typical fields are read-back, set-point, tolerance, set-point limits, alarm limits for analog devices. Commands and states are typical fields for digital state devices. There are fields to indicate the error status, lock/unlock condition, error-report-control, command-process state etc. The format also supports calibration arrays to hold calibration constants and unit information and up to four data arrays. The data arrays can be used to send or receive character strings or to hold a ramp table for a power supply. The important fields in the format is the CONFIGURATION MASK which defines the device type. The mask specifies the applicable fields and commands for a device.

Standard routines for updating/retrieving the device parameters are used. Every logical device is assigned a meaningful name (such as UVDIPOLE) which is unique in the entire system. The logical device name list is added to the device information base in the host level file server at the installation time of a micro system. The host level applications access a device by the logical name. The high level utility libraries translate the name to the micro node name and the device record index and sends the message to the appropriate micro.

Device Database
This is a key element of the micro system. The data base is a collection of logical device records and it resides in the processor memory. The logical devices and their types are defined and meaningful names are assigned to them by the microcode programmer who has a complete knowledge of the control requirements of the hardware associated with a microsystem. Default values for the parameters and states and calibration arrays are set up in the database. In addition to the device parameters, the database includes the hardware information (hardware I/O interface board addresses, channel numbers, I/O bit(s) etc., or the list of devices in case of a group device). Both application tasks and system tasks access the database via access routines which prevent simultaneous access of a record by multiple tasks.

Software Components
(1) Real-time Kernel, (2) System modules and (3) Device Application modules. It is arranged in logical layers as shown in Fig. 3. The real-time kernel provides the device drivers for the CPU and ethernet hardware, thereby making the monitor portable to a different CPU board with minimum effort. The system software hides all the real-time kernel specifics and the system hardware and provides services for real-time control by the device application tasks. The interface between the system and application tasks has been defined in such a way that any modification at the system level will not require rewriting of the micro application tasks.

Fig. 4 illustrates the interactions of the various modules and the device database and the data flow.

Real-time Kernel
The processor runs VxWorks, a real-time kernel developed by Wind River Systems. This is a fast memory-resident kernel with the interrupt latency time less than 10-12 usec for a 68020-based 20 MHz CPU. The kernel supports multitask control, dynamic memory allocation and de-allocation, interrupt handling utilities, intertask communication via event semaphores, message queuing etc. The VxWorks ethernet clerk (Network package) provides the standard socket level abstraction. Software development is carried out in an HP UNIX development system using VxWorks libraries.

Monitor System Module
This module is standard for all micros and consists of multiple tasks and interrupt handlers. It provides system timing ( 1 millisec resolution) and uses the VxWorks primitives to synchronize and coordinate the activities of various modules described below.

Initialization Module
This is the first task that runs on power-up or reset. The task initializes the necessary system hardware (timers, serial port, video chip etc.), sets up interrupt handlers and spawns the device application tasks.

Monitor System Module
This has the highest priority among all the tasks. Commands are received in standard format from the workstation and translated into the logical record format that can be understood by the application tasks. The communication model is based on the standard server-client paradigm. Every micro has a server which receives messages from any node on the network. The client software in the micro is similar to the workstation ethernet software and provides the micro to micro communication facility. The real-time communication uses the connectionless UDP protocol with a message size of 1024 bytes per packet. The UDP protocol provides a higher data rate than TCP/IP but it is the user's responsibility to make it fail-safe. In the NSLS system, a lot of efforts have been put in to provide reliable communication.

All messages have a fixed message header which has been designed with the necessary fields for identifying error conditions. All headers start with the size of the header, version number, source and destination node names, and the message function code, process id number, message sequence number, etc. It also contains the number of individual device command packets in the message. The device packets follow the message header. The device packet have fields for the packet size, version number in addition to device control code and data. Future modifications on the headers or device packet are very easy and will be backward compatible. A handshake is provided for every message transaction. Integrity of messages are checked and duplicate and out-of-sequence replies are identified and appropriate actions taken. The client initiates a request and the server replies to it with the data. It is the client's responsibility to retry if a message is lost. The number of retries is 5 and the time-out period is around 400 millisec. The communication module checks the source and destination node names, message function code, compatibility between the message size and the number of devices etc. If an error is detected, the entire message is discarded and a negative acknowledgement is returned to the client. Valid messages are disassembled into individual packets and passed on to the command decoder. The decoder returns a reply for each packet which is the requested data or acknowledgement. All the individual device reply packets are assembled in the same order as received and returned to the workstation. If a message size exceeds the packet size, it is broken into small segments and the the header will provide the number of segments and the segment number. Handshake is provided for every segment and the segments are assembled into a single message before processing the message.

During the upgrade phase, rigorous tests were carried out and the communication proved to be robust. Handling multiple requests in one ethernet message dramatically improves the micro response at the host level and also reduces the network traffic.

Command Decoder
This module is responsible for processing the device command packet and generating a device reply. Since the device packet follows a standard format irrespective of the hardware control, the system software can check each packet for errors before passing on to the device control tasks. The decoder module performs checks for the validity of the specified record in the micro, the command code and the size of the parameters if any. Every device reply packet has a command acceptance status field. Invalid packets are returned to the communication module with an error code.

The commands are divided into two types: READ and SET commands. The read commands allow the host level programs to acquire any parameter of interest or a selected combination of parameters or the entire logical device record. There are commands to read the data arrays and the calibration arrays. The read commands are handled at the monitor level and the reply is constructed using the current values in the logical records since they are updated at rates 10Hz or even higher in certain micros by the device data-acquisition module. There are more than 20 set commands to control analog settings (set-points, set-point limits, alarm limits, tolerance), digital states, resetting a device, locking/unlocking a device etc. There are commands to set data and calibration arrays. The monitor uses the Configuration Mask before updating the device record. A set-point command for a "digital output only" device is automatically discarded. The analog set-point is clamped at the appropriate limit if it is out of range. If a device is locked, then the command is ignored. All the device records have a copy in the battery-backed up ram. The new parameters are also updated in the back-up records. The device application tasks are kicked off and the type of the command is notified for further low level control. (Normally only the new set-point, digital commands like On/Off, device Reset and new data arrays are of interest to the device control tasks).

Error and Alarm Reporting
The system module provides services for checking alarm conditions and error reporting. The error checking includes detection of out-of-tolerance, out-of alarm limits, failure to control a digital device, time-out conditions, fault status like water flow error, RF trip etc. The error messages are sent asynchronously to an Error Handler which itself is a micro system. The error handler generates a scrolling error message display (with name, time-stamp and error message type) on a console in the Control room and alerts the operator with a beep. Error flags are latched in the device records to prevent error storms. A new Set or Reset or an explicit Error reset command will unlatch the error. At this point, if the device is no longer in error, an ERROR RESET message is sent to the error-handler micro. The error handler saves the error messages in memory (a 4 megabyte non-volatile memory). An error logger which is running in background on a workstation periodically (once in 5 seconds) collects the error information and stores it in a disk for post-mortem analysis. The micro has the option to inhibit error reporting on an individual device basis or on a micro basis using a software selectable switch on the GPLS board. This feature is necessary when an equipment is under test or when it is known to be malfunctioning.

Display Management
The GPLS board can generate a Cable TV compatible ASCII display. There are four switch selectable hardware pages. One hardware page can multiplex up to 8 software selectable pages. The page switching is accomplished either by a push-button panel attached to the micro or remotely by an operator command. The displays are generated both by the system and the device tasks. Some micros have permanent TV channels for continuous display of parameters; others go to multiplexors and are switched to the TV by operator commands. Since the displays are generated locally by micros, very little of the network bandwidth is used. These memory mapped displays provide a great diagnostic aid for the real-time programmers.

AC Fail Interrupt Module and Watch-dog Timer
All the micros have AC power-fail interrupt module. The monitor traps the AC fail interrupt and invokes an application routine which has the option to save some crucial parameters (within 5 millisec) before the system crashes. The watch-dog timer has the ability to restart a micro when it is caught in a loop. This feature should be used with discretion.

Diagnostic Statistics
The important system parameters such as message traffic, transmission errors, statistics of various interrupts, spurious bus interrupts, total number of logical records, the time duration since last reset/power-up etc. are saved in memory by the monitor. A low priority task provides a display of this data every 500 millisec. The information can also be retrieved from a workstation for analysis purposes.

Device Application Module
This module performs the low-level hardware control. In the monitor development environment, these are just a collection of subfunctions which are invoked when the appropriate device tasks are in run state or when the application related interrupts occur. The device initialization module is invoked by the monitor when the micro starts up. All the device records and hardware I/O interfaces are initialized and the equipment state is restored to the previous state using the battery-backed up parameters or optionally to the default state. When a new version is installed or records are deleted/added, the default state is used. When new commands arrive, the monitor notifies the device tasks. The device controls may be as simple as setting/clearing bit(s) in a bit I/O card or setting a DAC or may involve controlling a group of devices synchronously or sequentially or may require a complicated software algorithm. The job is accomplished by using appropriate device drivers. The task monitors the hardware signals periodically at 10 Hz or more and updates the device database. The micro that monitors the XRAY beam positions (xray orbit) reads approximately 200 ADC signals (using multiplexed ADC boards) at 550 Hz using a single CPU. The device module checks for errors when the devices are not in the command execution state and requests the monitor services for error reporting. The tasks also generate application-specific video displays. The tasks can request the system module for periodic scheduling (the minimum sleep period is 2 millisec). The system module generates periodic interrupts at 1, 2, 4 and 8 millisec intervals for applications purposes. Some applications use these for triggered data acquisition or for ramping set-points. In addition, the system provides services for interrupt connections from external stimuli or from the slaves on the VME bus.

Some micros use the client services to send control commands asynchronously to other micros. The micro controlling the main magnets (dipoles, quads and sextupoles) sends the energy value to the RF micros and the Trim magnet micros when it ramps the dipole set-point. The RF and Trim magnet micros use the preloaded ramp tables to ramp their set-points synchronously without the intervention of any workstation. A smooth knob control with a guaranteed 10 Hz rate is also implemented by a micro system using the micro to micro communication utility.

Conclusion
The VME micros provide very quick response at the device control level and reliable and fast communications with workstations. The multiple commands in one message reduces a lot of network traffic. Storing the parameters in the battery backed up memory has been very useful to restore the equipment state following a power-dip or AC failure. The video displays and micro to micro communication facility have been found very useful in daily storage ring operations.

For information please contact: Susila Ramamoorthy susila@bnl.gov