Observing Operations | Reviews | Survey Management

Sloan Digital Sky Survey
Review of Observing Systems and Survey Operations

Data Acquisition System
Don Petravick
April 11, 2000

Description and Functionality 


The SDSS data acquisition system consists of hardware, software, and methods for maintaining the hardware and software. Conceptually, the system sits between the data path of the instruments and the "observers software".
The system was specified and procured by early 1992. The system is built of common components and shares much Unix software with the rest of the survey. The system was delivered to Apache Point Observatory in 1996.

The physical specification of the system is summarized as:

  • A computer system for the 2.5 m telescope whose primary components are:
    • An SGI Challenge computer plus peripherals serving as a host
    • Two VME crates plus disk and tape peripherals to service the photometric part of the imaging camera.
    • A VME crate plus disk peripherals to service the astrometric part of the imaging camera.
    • A VME system plus disk for the Spectrograph.
  • A computer system for the Photometric Telescope whose primary components are:
    • An SGI Challenge computer plus peripherals serving as a host
    • A VME system plus disk and tape peripherals
  • Supporting equipment at Apache Point Observatory, consisting of:
    • A simulator sufficient to drive one optical fibre, and to simulate any survey instrument.
    • A spares kit.
  • A lab at Fermilab, consisting of:
    • An SGI Challenge computer
    • A simulator setup
    • Sufficient VME hardware to test one unit of throughput.
The function of the SDSS data acquisition system is
  • To acquire all data from the Imager, the Photometric Telescope, and the twin Spectrographs.
  • To verify framing of the images.
  • To display the images in real time on video monitors as an operational aide.
  • To spool a night's worth of images, making them available to the "op" software.
  • To make a number of control primitives available to the "op" software.
  • For "photometric" ccds in the imager
    • To write the data to redundant tape drives
    • To histogram all the data to produce a "flat field" vector for almost all frames.
    • To scan the images for stars, producing a "postage stamp" file for almost all frames.
    • To make sufficient images available to a Unix host computer for online QA.
    • To make all summary data available to a Unix host computer for online QA and recording.
  • For the Astrometric CCDs in the imager camera
    • Calculate a "sky" value for each "division" in in the physical CCD.
    • To locate stars in the astrometric data, place the stars in a "postage stamp" file.
    • To locate stars in the "focus loop" ccd and produce information sufficient for a focus servo.
    • To make sufficient images available to a Unix host computer for online QA.
    • To make all summary data available to a Unix host computer for online QA and recording.
The system does not:
  • Command the survey instruments.
  • Command the telescope control computer or any type of motion control systems.
  • Implement the observing policies of the survey.
  • Implement any decisions on the quality of the data or the condition of the instruments.
The software for the VME system is constructed on top of VxWorks 5.1.1. The software is aware of the hardware features of all VME cards. As such it is constrained to run unchanged only in a VME environment which is hardware compatable with the delivered system.

The Unix DA software is written to IRIX. The software includes a IRIX device driver for the PT link, a high speed (for its day) non-standard network connecting the Unix systems and the VME crates.


Degree of completion 


We believe the system meets its functional specifications

The system was delivered to Apache Point Observatory in September of 1996. Prior to delivery, the system was unit tested by means of a full scale hardware simulator. The simulator was constructed a Fermilab. Many defects were discovered and removed from the system using this test structure.

After delivery, an integration effort, involving the data interface to each instrument was completed. The data acquisition system was also run for a substantial amount of time at APO with the full simulator.

As operation of the survey has ramped up, other efforts have been made to explain the system, resolve problems and fix defects in the system by the FNAL group.


Correctness 


The system was unit tested before and after its deployment to Apache Point Observatory. It performs all functions specified in its functional requirements document. There are features in the DA system to check data integrity from the VCI+ to the tape in test mode.

There is an ongoing effort to resolve problems which are detected as higher levels of the system are delivered, and as the Survey undergoes its test year. An example of a recent problem resolution is the completion of a project to work around a defect in the CIPRICO RIMFIRE 3563 board in March, 2000. The root cause is an unexpected type of behavior in the Rimfire board when a particular type of tape error occurs.

Pending issues are:

  • Resolve occasional crashes of the astrometric boards.
  • Deploy the "video services" card and retire the VIGRA card.
  • Center the sky histograms for the photometric CCDS more quickly when the sky moves.
Lifetime


A component sparing and lifetime project was begun in FY1999 and completed except as noted in March, 2000. The purpose of the project was to provide for an additional five years of lifetime for the system. (The original system was to last until the year 2001). All hardware and software components were considered. Notable activities of the life time project were:
  • Replacement of the the SGI Crimson host computers with SGI challenge computers.
  • Upgrade of the IRIX operating system to IRIX 6.5.
  • Upgrade of the VxWorks 5.1.1 cross compiler to gcc cygnus-2.2.3.
  • Upgrade of the "pool" disks and real time file system to 50GB per MVME167.
  • Upgrade of the tape drives to DLT4000.
  • Construction of seven complete VCI+ cards and three partial cards.
  • Construction of a replacement for the "vigra" video cards, which are no longer manufactured. These cards are not yet deployed.
  • Consideration of the lifetime of the MVME167 boards. These boards have a battery whose lifetime is a function of the temperature and the number of power-on hours.
  • Consideration if the lifetime of the LEDS in the "foxi" senders. LEDS have a lifetime which is a function of operational hours and operational temperature. Ten Spare FOXI transmitter modules have been procured.
  • The hardware spares kit was evaluated and augmented.

Beyond the sparing kit and sparing methods we rely on the FNAL equipment pool for additional pairs of the CIPRICO Rimfire boards and the PTE link.

The Integrated Systems development department has four full time staff members who were involved in the development of the system (Bakken, Holmgren, Petravick, Rechenmacher).  In addition, a fifth person (Nicinski) has accepted an offer to return to the department. While none of these persons have the SDSS as a full time assignment, the department has been able to staff and run its SDSS related projects. The department has a clear mandate from its management to provide for the long term support of this system.

Having completed the sparing project, the planned activity for FY2000 is to stand by and fix bugs and help resolve problems. The department had made several trips to APO and performed substantial work at Fermilab doing this work.




Review of Observing Systems and Survey Operations
Apache Point Observatory
April 25-27, 2000