General SDSS Site Operations Responsibilities

Past:  Site construction and infrastructure development, on-site staff buildup and training, receive deliverables and support integration and commissioning.

Present:  Finish receipt and integration of operational systems, provide direct technical and admin support to remote and visiting Project staff during commissioning and handovers, complete recruitment and training, capture and systematize engineering and operational documentation, start survey data acquisition.

Future:  Conduct Survey observing and spectroscopic plug plate processing, maintain SDSS facilities and site infrastructure, ramp-down SDSS operations as Survey nears completion.

Site Physical Description

APO located in the Sacramento Mountains at 2800 meters elevation near Sunspot, New Mexico, near the National Solar Observatory facilities at Sacramento Peak.  The APO campus sits on 7 acres of land leased from the U.S. Forest Service.

Observing facilities include the SDSS 2.5-meter survey telescope, a 0.5-meter telescope which is used to calibrate survey photometry, a
3.5-meter telescope used for general visible and IR imaging and spectroscopy, and a 1-meter telescope owned by NMSU.  The site also
contains operations and support systems, buildings, and dormitories. 23,000 sq. ft. under roof (about half for SDSS).

On-site Staffing and Organization

Site operations staff comprised of about 25 full- or part-time individuals who conduct the observing and provide technical and
administrative support to the site, telescopes, and instruments. Many of the support staff divide their time between the SDSS and 3.5-m
projects; the SDSS observers, engineers and technicians specialize in SDSS.  About 17 FTEs work on-site on SDSS, plus a few "commuters" from SDSS institutions.

[org chart and staff profile to be attached]

Off-site Project Resources and Interfaces

Scientific and engineering support is provided to survey operations through arrangements with SDSS partner institutions.  During
construction and commissioning, support has been ~3000 person-days/year at the site, plus remote support furnished by e-mail
or telephone.  This is tapering down.

Operational Notes and Plans

Commissioning Support and Handover - Operations staff work closely with off-site and visiting project staff during pre-ship, installation, check out, commissioning.  This helps to get jobs done plus operational perspective value added, with critical information transferred to site staff.

Observing - 2.5-m Telescope will be on the sky whenever possible during dark (and grey?) time.  Operational models for efficient observing, safety, and skill-mix require 2 Observers per night (requires minimum of 6 Observers total) for Survey observing, and possibly 1 Observer plus engineer for engineering time.  Observers are scientists/technologists and assume the role of PIs for the daily/nightly runs on the telescopes.  Activities include running site planning tools, participating in daily telecon with off-site SDSS scientists, performing nightly evaluation of data products, making tactical observing decisions, helping coordinate plug-plate staging, and writing and sending observing logs.  Observers are expected to participate in SDSS science programs, and will be assigned occasional TDY at FNAL to work with data processing scientists (and vice versa).

Photometric Telescope - In operation during all survey observing, probably about 0.5 FTE at site, TBD.  In steady state, it is expected that the PT will require minimal nighttime operational support.  During the first year or two, the site oberving staff will need to be augmented by an observer furnished by JHU.

Plug Plates - FNAL provides drilling specifications to UWashington, where the plates are fabricated and then shipped to APO.  APO manages receipt, processing, storage, and inventory at site. On-site activities include plate database maintenance, plate receipt and storage, Q/A monitoring, staging, plugging and unplugging, fiber mapping, and fiber harness and cartridge use and maintenance.

Data Tapes - FedEx'd to FNAL next day (except Sunday), with a backup copy kept at the site.

On-line Q/A and Observer Utility Programs - Currently being developed, maintained, used - Observers enhancing and maintaining Project-provided utilities with support from developers.

Standard Operations Procedures - Check lists, staff training and process development, Q/A tools - provide written accountability for
project activities and processes that directly contribute to data quality and uniformity, and observing efficiency.

Monitoring, Trending - Periodic measurement of telescope and instrument performance uniformity important to survey data quality,
and for prevention of h/w degradation.  Telescope and Instrument engineering responsibility in conjunction with developers.

Scheduled Routine Maintenance and Safety Inspections - Operations must be safe for site and equipment, and facilities expected to kept
like-new for operational lifetime.

Weekend and Holiday Coverage - Observing is scheduled 365 days/year, weekend day support by "skeleton" crew, regular day staff available informally on-call nights, weekends, and holidays.

Trouble Response, Three-tier approach, Problem Reporting System - Problems at night are addressed first by Observers, then by phone to
operations day staff, then next day to on- and off-site support engineers and scientists.  Formal computer-based problem reporting
system, some automated error sensing systems.

SDSS Observers

Qualifications, numbers, schedule
•   Recent Ph.D.s, generalists, science motivated and capable.
•   Six in baseline plan, work in pairs.
•   20-21 nights/lunation covered, day and engineering time supported on a limited basis.

Survey science - Observers are scientific participants in the Survey and work on science projects in collaborations and as part of data evaluation  responsibilities.

Training, tools -  Observers participate in the development of training procedures, checklists, software utilities, and documentation associated with observing operations.

 Procedures - Observers are participating in the development of operating procedures in an on-going activity as part of the commissioning process.

 System Commissioning -  Observers developing system commissioning plans with developers.  These plans will be used to verify that the various sub-systems asscociated with survey operations at the site perform to specifications.

Infrastructure

On-site and off-site housing - Twelve semi-private dormitory beds for short-term, long-term house and trailer at NSO, and short-term rental overflow at NSO.  Meeting and symposia facilities available at site, at Sunspot Visitor Center, and in Cloudcroft.  Provide housing, work space, and meeting arrangements for project visitors.  ~4000 person-days of site visitors housed in past year, >70% SDSS.

Telecom and network - T1 to Internet through Las Cruces, vBNS thereafter; LAN is 100Base-T, switched.  Backup T1 connection to
Sunspot.  Two incoming modems.  Site provides, monitors, and improves site telecommunication locally and globally.

Documentation - Engineering drawings, documentation and manuals delivered to site by developers and integrated into documentation
archive by APO staff.

Seeing & meteorological - Seeing monitor (DIMM) in use.  Also have system to provide site-wide monitoring and data display/archiving of site weather including wind direction and speed, temperature, dew point, dust, clouds.  Work with local public and gov't agencies to
enact and enforce "dark sky" outdoor lighting.

Human health and safety - OSHA requirements, existing site safety plan and designated safety officer.  Periodic outside audits
coordinated with NMSU, FNAL, NOAO, VLA safety officials.

Power - 208v, 600 amp commercial service, distributed through multiple hybrid surge suppression.  Critical equipment on UPSs, plus
SDSS camera isolated with transformer and motor-generator.  Provide and maintain automatic backup diesel generator, and portable
gas-driven generators.

Cryogenics and vacuum - LN2 furnished to levels required by instrumentation, on-site staff transport from local vendor.  Vacuum
pumping and leak-testing stations provided on site.

Utility gases, liquids - Provide dry clean compressed air, LPG, hot/cold water, dry nitrogen, LCO2, helium, maintain underground
coolant system.

Tools, test equipment, components stock - acquire and maintain hand tools, bolts and screws, electronic components and test equipment,
etc. to level that rarely limits ability to work or observe.

Optics handling, cleaning, aluminizing - On-site staff handle, clean, and aluminize smaller mirrors.  Primary mirrors recoated at KPNO.

Computer systems - FNAL-furnished DAQ hardware systems for SDSS, operational and utility computers a mixture of Sun, Macintosh,
Silicon Graphics, PC, DEC; ~70 IP addresses.  Help maintain and manage SDSS computer systems, acquire and manage site utility systems.

Preventive maintenance & spares - Scheduled PM performed to check lists, critical and long lead time items spared on site.

Observing metrics from Q1/2000

[to be provided in next draft]

Miscellaneous

• Local government and community relations
• 3.5-m interfaces

Attachments

•  Site layout
•  Staff org chart, profile
•  WBS listing
•  Q1 observing metrics, detailed