Imaging Camera Overview
Connie Rockosi
April 11, 2000

Introduction

The SDSS imaging camera was delivered to APO in October 1997 and completely reassembled in the clean room in December of that year. It saw its first light on the 2.5m in May 1998. We have since fixed the cause of a few electronics problems that showed up in the original data, and have tried to make the instrument as it is installed at the telescope easy to care for and robust. A technical description of the design of instrument is far outside the scope of this review, and can be found in Gunn et al. (AJ 116, 3040. 1998).

Operational Description

The CCDs are mounted off the back side of the second imaging corrector lens, and that lens is carried mechanically by a stainless steel ring that attaches to the telescope on kinematic mounts. The CCD electronics and liquid nitrogen cooling system and the rest of the support electronics are back behind the CCDs within the circumference of the steel ring. The whole instrument is roughly cylindrical with a diameter of about a meter and a depth 2/3 that at the high point of the back cover, and weighs approximately 700lbs. Surrounding the camera is an open rectangular structure we call the saddle which is mounted to the telescope independently from the camera and carries the weight of the 400lbs of powersupplies and liquid nitrogen supply resevoirs that are connected to the camera through cables and vacuum-jacketed hoses. The imager is mounted to the telescope on the central rotator bearing when it is taking data and is stored on its cart in a small enclosure on the telescope's rotating floor when it is not. Figure 1 shows the imager mounted on the back of the telescope, between the two green spectrograph bodies. The back cover is off, exposing some of its maze of plumbing and cables.

The 54 CCDs in the imager are housed in eight independent dewars, each with its own LN2 resevoir, autofill plumbing, powersupply and data output to the DA system. The dewars are evacuated to a few 10^-7 Torr when cold and contain the CCDs mounted on their optical benches, preamplifier and signal-conditioning circuitry for the CCD clocking signals and a 200ml LN2 resevoir and the associated thermal regulation system for the detectors. The double-correlated sampling and 16-bit digitization circuitry for each CCD is mounted in an enclosure on the back of its dewar, and the parallelized data for all the CCDs in a dewar is brought out on fiber. All the CCDs are clocked synchronously, and the clock waveforms are derived from a master 8MHz clock. The digital signals are carried around the instrument on RS485 busses, and converted to and if necessary retransmitted from 5V CMOS where they are used. Figure 2 is a schematic diagram of the focal plane layout. Figure 3 is a drawing of a single dewar, showing the detectors on their optical benches, the associated signal-processing electronics in their chassis on the back, and the LN2 resevoir in the center.

Each dewar runs from a set of power supplies kept in the chassis on the saddle, and the voltages they generate are monitored in hardware for under- or overvoltage conditions. If a supply is 10% beyond its nominal value the entire set of supplies for that dewar is shut down and must be swapped out or otherwise tended to. Operating procedure is to swap out the module if the problem is in the supply, with more detailed debugging done off-line during the day

Everything in the instrument is controlled by Forth code running on a Hitachi H8/532-based microcontroller made by Triangle Digital Systems, Ltd. Communication to the outside world is over a 9600-baud RS232 serial connection. The microcontroller manages the configuration of a slave exactly like itself that is busy full-time generating the clocking signals for the CCDs. Through its I/O ports the micro also controls the shutter and the LN2 autofill, sets the DACs that control the 21 operating voltages for each CCD, and monitors the state of the instrument. The health and safety monitoring is quite extensive and is an integral part of the instrument's robustness and safety; it will be discussed below. The Forth code is put in an EPROM on the microcontroller, and the code is kept under version control. The micro can return the value of a version string stored in the EPROM, and the same string is the tag used to retrieve the corresponding Forth source from the Survey's CVS repository.

The CCDs are thermally connected to the LN2 resevoirs through conductive straps that bring them to thermal equilibrium somewhat below their desired operating temperature. A proportional control loop uses a temperature sensor and heater resistor to keep each CCD at its operating temperature. The LN2 resevoirs will hold for about two hours, and are filled once an hour from the 10L intermediate storage dewars on the saddle on command from the microprocessor. The autofill system runs on pressure gradients, and the LN2 system is kept at about 10psi. To fill the resevoirs in the dewars, solenoid valves open on the vent side of each resevoir's plumbing and LN2 flows down the pressure gradient and into the resevoirs. If the microprocessor for some reason does not properly control the fill system, the low-level control electronics automatically triggers a fill if a resevoir goes empty. The 10L dewars are filled on the same principle from a standard 180L 22psi supply dewar on the rotating floor. The control is less sophisticated and is not done through the microprocessor. A 180L dewar lasts 24 to 36 hours.

The optical benches that mount all the CCDs in each dewar are located in the focal plane on kinematic mounts. The mounts work by allowing titanium balls to roll along stainless steel rods in each corner of the rectangular optical benches. By changing the relative rotation of the stainless steel rails between the corners, the optical benches are constrained in all dimensions. The mounts are accurate and repeatable, but because the contact area between the balls and rods is small they are susceptible to damage from the forces transmitted to them during normal handling of the instrument. For this reason, the optical benches are latched away from the kinematic mounts when the camera is not observing. The motion toward and away from the focal plane is accomplished by inflating or contracting a set of bellows in each dewar, and control is through a valve that switches the hoses for the bellows between high-purity 40psi N2 to inflate them and a source of vacuum to contract them. Moving the mechanical benches on and off the kinematics is part of the process of instrument change for the imager. Figure 4 is a photograph of one dewar's CCD mounting hardware. The optical bench is face-up on the left. The rods for the kinematic mounts are visible in the four corners, and at the bottom the picture is looking down on one of the latches that holds the bench off the kinematics for handling.

Because of design constraints this motion will only work if the instrument is vertical, and it is possible to damage the kinematic mounts if the optical benches are moved with gravity pulling them toward the ends of the dewars. It is thus very important that the optical benches be properly on their kinematic mounts when the imager is on the telescope and it is slewed away from zenith, and that they are latched off their kinematic mounts when the imager is taken off the telescope before it is rolled back into its enclosure.

Because the imager does not know its own orientation, the microprocessor does not have control over the motion of the optical benches. Instead, there is four-signal interface between the the PLC and the hardware in the imager that controls the valves for the bellows. Two signals come from the PLC to the imager to command motion and two signals are returned to the PLC from the imager to report status. The PLC can command the optical benches to be either on the kinematic mounts (unlatched) for observing or latched away for safe handling of the camera, and the PLC interlock logic will only pass this request if the telescope altitude is above 88 degrees. The bellows valve controller can measure the pressure in each dewar's bellows hose and compare it against the supply sources of 40psi N2 and vacuum, and knows which one it expects to match given the commanded position. If the pressure for all the dewars matches the correct supply, then the control hardware will claim to believe the optical benches are latched or unlatched and assert the appropriate status line to the PLC. The microprocessor can see both status signals and report them through its RS232 interface. There is an additional check on the instrument's orientation from a tiltmeter mounted inside the camera. It produces a binary output which changes state when it moves through an angle 2 degrees from vertical. This output is incorporated into the bellows valve control logic, and a latch or unlatch command will be refused if the tiltmeter indicates that the instrument is not vertical. The micro has access to this bit so it can be interrogated if a latch command fails. The micro can also report the values of the bellows pressure for all eight dewars, and the value of the absolute pressure in the vacuum supply.

All this information, plus gauge readings of the supplies and a gauge mounted on the side of the imager that reads the pressure in one dewar's bellows hose directly, can be made available during instrument change. There is a toggle-switch control box that can be plugged into the control cable in place of the PLC interface to move the optical benches directly. As a last resort there is a ball valve that can switch the source of the 40psi N2 supply hose between 40psi and vacuum. This can be used to move the optical benches by taking advantage of the fact that with the control cable disconnected the bellows valves will always switch the hoses to the 40psi supply input to the camera. This behavior of the valves is a precaution to keep the optical benches from moving if the instrument should lose power while pointed at an arbitrary position on the sky.

There is a single cable bundle that goes to the imager which remains attached when the instrument is on the telescope and in its enclosure. It carries AC UPS power, LN2 from the 180L supply dewar, water-glycol for the heat exchangers, compressed air for the shutter actuators, N2 for the autofill pressurization, vacuum and high-grade N2 for the optical bench bellows motion, the signals from the PLC to control the optical bench motion, the optical fibers that carry the RS232 communication with the camera and the 10 optical fibers that bring the digitized pixel data out to the data acquisition system. Any signal brought to the instrument on copper originates at the imager's rotating rack and is carefully referenced to the instrument's master ground. The run from the rack up to the instrument is as short as is practical for both noise and lightning safety considerations.

Health and Safety Monitoring

The imager is kept ready to take data at all times so that it can always take advantage of good observing conditions. It is also an essential part of the Survey and as such it should be as difficult as possible for any harm to come to it. It is also important that it not require an inordinate amount of attention from the observers and APO staff so that they do not spend too much of their time and energy on hardware support, and so that the instrument will be safe over long periods of inattention when blizzards, dust storms, overturned trucks on the Sunspot Highway, etc., keep its normal caretakers from their rounds.

The microcontroller that runs the imager has access to the important information about every subsystem. All the voltage levels are brought to a 10-bit A/D converter built into the Hitachi processor through an analog multiplexor, and the CMOS binary status information is all available on the micro's input ports. The micro runs a continuous background process that loops through the subsystems and looks at each piece of status information. It compares its measurement against a nominal value and tolerance, if applicable, and if it is outside the acceptable range a corresponding error flag is set. There is a Forth command that has the micro go through the list of error flags and return a list of all the errors and some information on why the flags were set. This list of errors is available all the time, updated roughly every four minutes as the micro cycles through its list of things to be checked. We have tried to make the list exhaustive so that any failure mode that can adversely effect the data or harm the instrument will be caught by the micro and made available to the outside world through a single command. The micro will not volunteer any information as a safeguard against clogging RS232 communication, and to cut down on chatter will return only the standard Forth "ok" if it does not detect any errors. There are other commands to return the values of status measurements that the micro records as it goes through the monitoring loop, and these are designed to provided long-term performance monitoring of the instrument. A complete list of everything checked by the micro is appended to the end of this document.

For most error conditions that the micro can see, it is sufficient that it report the error when it is asked. The data may be junk or the instrument out of service for a day or two if it goes undetected, but there is no threat of permanent harm to the instrument. There is a subset of error conditions, however, that if unrectified could result in damage to the imager. Since it is impossible to require that there always be someone on-site to handle such situations the imager responds to these on its own, whether or not its error status is ever queried. It does so by doing the only thing that is guaranteed to be safe, which is to shut itself off. It does so by allowing a watchdog timer to expire, which in turn opens a relay contact to cut off incoming AC power and turns the instrument off. Normally the imager resets this timer every time it executes the monitoring process after checking that none of the designated errors are present. The monitoring loop is done as part of the high-level functioning of the microprocessor, so if it hangs or crashes and stays down the watchdog timer will run out. This is alleviated by a second watchdog with a much shorter timeout that the micro must actively reset in the same monitoring process; when this timer expires it executes a cold reset of the micro which should reboot it and restart the monitoring process before the instrument power shuts off.

The following errors will result in the watchdog timer shutting down power to the instrument:

-- Any of the CCD amplifier VDD voltages too high

-- No flow in the water-glycol input lines to the imager from the chiller. This is measured by closure of a switch contact in a flow sensor mounted in the input line to the instrument.

-- Saddle powersupply chassis temperature above 45 C. This is quite possible to reach if the chiller is not running.

-- CCD temperatures above 40 degrees C. They can (and will) get there if the detectors start at room temperature and are clocked continuously for an hour or so with the LN2 system not running.

-- Instrument air or outer chassis temperature above 40 C.

Current status

The last piece of functionality that has not yet been implemented is the PLC and IOP interface for controlling the optical benches. We have so far left the control cable unplugged and positioned the optical benches by using the valve in the rack to force them to move. The imager's side of the control electronics for the bellows valves, the micro's access to the status bits and tiltmeter and the optical isolation and surge suppression for the signals in the PLC interface were finished before the instrument was shipped. They have been tested and used to some limited extent while we worked in the clean room at Princeton and at APO, and exercised with the beginnings of the PLC interface in the last few months. The imager is otherwise a reasonably self-contained system, but the optical bench control requires the coordination of many normally-independent parts of 2.5m operations so there are sure to be software and perhaps hardware modifications to the imager necessary to make the process work smoothly and transparently.

The only outstanding problem preventing the imager from taking good data that that we would like to keep for the Survey is an intermittent noise problem with the u' CCD in dewar 2. Other problems that need to be addressed as soon as possible are a recent problem in the leading astrometric dewar that causes CCD heater power supplies to trip their monitor circuits, and some bug or leak in the Forth code that occasionally crashes the imager which has begun to show up more often and less predictably now that we leave the instrument running all the time between dark runs.

The scientific requirements document states the design goals of the instrument, and we have some of the data to see how well those goals are met.

There are requirements on the FWHM across the array for a given free-air seeing and on the magnitude of a 5-sigma point source in all 5 bandpasses, but these are difficult to assign entirely to the imager since they depend so heavily on the imager quality delivered by the telescope. It is clear that we have to be sure that the alignment of the detectors in the focal plane is not a significant contribution to the image widths either from focus mismatch or distortion.

There is a QE/throughput requirement specified on the minimum number of electrons that the imager should measure from a 20th magnitude object, and that is straightfoward to look at. The requirements are: 1300 e- in u', 9000 e- in each of r' and g', 6500 e- in i' and 1500 e- in z'; these are 75% of the predicted values. The initial flux calibration done by the reduction pipelines is output as the flux in DN that would be measured from a 20th magnitude object in each frame in each detector. From the reduction of some data taken in February 2000 and multiplying by the gains to get electrons, the mean value for this 20th magnitude flux parameter for each CCD is listed below. The dispersion around this mean is negligible.

u1 1820	g1 10128	r1 9405	i1 7281	z1 1484
u2 1796	g2 10185	r2 9171	i2 7981	z2 1531
u3 1909	g3 10404	r3 9977	i3 7478	z3 1323
u4 1830	g4 10922	r4 9587	i4 7530	z4 1281
u5 1938	g5 10643	r5 9705	i5 6775	z5 1522
u6 1771	g6 10821	r6 9698	i6 7211	z6 1411

This is not yet coallated for commissioning data to check for long-term effects, but we are easily meeting the minimum requirement in all but z', and not missing to badly in most cases.

Work To Do

The PLC interface to the optical bench control is well underway and will be commissioned as part of the MCP-controlled instrument change. The direct gauge reading of the pressure to one dewar's bellows gives anyone working with the new interface a double-check that the optical benches are in the correct configuration to safely move the instrument and/or the telescope. This should make it possible to debug the new optical bench control system with minimum chance of damage to the instrument.

The noise problem in CCD u2 and the source of the microprocessor crashes need to be understood and fixed as soon as possible.

We are still finding little details that need to be worked out in the monitoring system, though it has worked quite well and done its job of saving the instrument from its builders on more than one occasion. We have twice now had the camera shut itself off and warm up, taking it out of commission for several days, for an error that went away before it could do any harm. The chiller and pumps that send water-glycol to the heat exchangers in the instrument are not run off a UPS and there is a delay of many minutes between the loss of utility power at the observatory and when the generator switches on. If the microprocessor's monitor process checks the flow in the coolant lines in that interval, it will right now declare that a dangerous error condition exists and allow the watchdog timer to run out and shut down the instrument. The micro will announce this fact if asked before the timer runs out, but the last several power failures have come in bad weather when no-one who would normally catch the problem was on site. It will take some time for the instrument to get dangerously hot with the chiller off so it seems reasonable to give the generator a little breathing space and change the coolant flow error to be an error only if there is no coolant flow for some predetermined amount of time. This should be safe particularly given that so many of the other errors that cause the imager to shut itself down are also related to thermal monitoring.

Long-Term Robustness

The primary function of the imager is to produce good data for the survey, and one piece of monitoring and checkout code that has yet to be implemented is something that will look at the performance of the detectors and decide if the camera is giving acceptable data. We have access to the statistical information for each frame that is provided by the on-line analysis code running in the DA system and this would allow performance monitoring to go on as a normal part of observing. We could also, if necessary, write special code that looked in detail at designated bias data and perform some tests. We know the nominal readnoise and other characteristics of the detectors and can compare new data against these to look for any discrepancies. Automated procedures for checking the CCD data would make testing the imager after it was worked on much easier and more reliable, and will hopefully prevent any data getting to FNAL that later had to be rejected because of some problem with the instrument.

The process by which a new version of the Forth code for the micro is accepted needs to be specified more concretely, probably by a set of commands that will exhaustively check the imager's normal functions so that the observers can be certain that it will be able to take data. The current procedure is for someone knowledgeable to install the new firmware and exercise the changes and be available when the imager is run in case problems are uncovered; hardly a long-term solution to the problem.

Mamoru Doi of the University of Tokyo and some of his colleagues built a monochrometer to measure and track the absolute quantum efficiency and other characteristics of the imager as part of the calibration of Survey photometry. It is housed on top of the imager's enclosure and provides a means of illuminating the CCDs to check things like gain and linearity, though taking the data for such tests is right now very time-consuming and can only be done sporadically. Once the calibration system gives us this detailed information about the imager's performance it will be incumbent on us to make use of it to be sure that the instrument takes uniform data over the length of the Survey.

The CCDs were originally grouped in independent dewars partly as a precaution so that a catastrophic problem with the vacuum in one of the dewars would only effect a subset of the detectors. However, the dewars all behaved quite poorly when we tried to leave them evacuated and the detectors cold for more than about two weeks. This would have been unacceptable for Survey operations since the camera can usefully take data for most of a lunar cycle. After finally deciding that the problem was not a leak, an ion pump was installed in the camera last summer. It is mounted on a manifold that connects all eight dewars in the camera and the pump is left on and open to the dewars at all times.

The eight dewars are now all part of the same, much larger vacuum system, and the ion pump turns out to have altered its characteristics enough that we need to go back and make sure that it is as we want it and fix it where necessary. It is possible that we trap in the getter and on the cold outside of the LN2 resevoirs quite a lot of the debris that we want to take out with the ion pump. If enough gas remains in the dewars there is significant danger to the CCDs as the instrument warms up, since the CCDs remain cold long after the LN2 resevoirs and the getter come back to room temperature and release all the gas they collected while they were cold. The return to room temperature can be controlled to eliminate danger to the CCDs, but the instrument was designed to be safe when it is turned off unattended so that the microprocessor's control of the watchdog timer can be used to respond to dangerous error conditions. We need to be sure that it is not now just as dangerous for the instrument to turn itself off.

Maintenance

The imager needs a new full 180L dewar every 24 to 36 hours. Things like the 40psi supply pressure and vacuum for the optical bench motion are checked as part of setup for normal observing, and can be maintained by the daystaff or by the observers if necessary.

If it turns out that running for long periods with the ion pump on leaves a substantial amount of gas trapped in the instrument, then it will be necessary to warm up the imager during full moon and put it on the vacuum pump to clean it up. This is somewhat complicated by the fact that it can only be done with the imager on the back of the telescope, immobilizing it for of order 24 hours.

The vacuum-jacketed lines need to be pumped when they begin to reduce the efficiency of LN2 transfer. It looks like it will have to be done three or four times per year.

We will have to decide what to do about changing the getter in the dewars now that the ion pump is installed. Previously it looked like it would have to be changed once a year during the summer monsoon season.

It is expected that any in-depth repair or maintenance work on the imager will be done by its builders with support from the Survey staff at APO.

Appendix: List of information monitored by the imager's microprocessor

The value of the 23 clock, amplifier and signal-processing voltages for each of the 54 CCDs in the imager.
Temperature sensor readings for each CCD
Voltage put across the heater resistor used to regulate the CCD temperature for 24 of the 30 photometric CCDs
Temperature of the LN2 resevoir for each dewar
Temperature in both power supply chassis on the saddle
Temperature of the air inside the instrument
Temperature of the main steel ring supporting the corrector and the rest of the camera
Current position of the optical benches as reported by the control hardware
Pressure in the hose that goes to each of the eight dewars' bellows
Pressure of the vacuum source for the optical bench system
Current position of the shutter blades and an error if they are not open if they have been commanded open.
Pressure in both 10L autofill supply dewars on the saddle
Success or failure in setting the DACs that control the CCD voltages
Whether or not the micro has rebooted since the the last acknowledged reboot Successful or unsuccessful completion of the last scheduled fill of the dewar resevoirs
Empty condition in one or more of the dewar LN2 resevoirs
Tiltmeter output status, vertical or not.

 

---

Review of Observing Systems and Survey Operations
Apache Point Observatory
April 25-27, 2000