Digital Sky Survey
April 25 to 27, 2000
Prepared by the Review Team
Jim Crocker, Chair
Version 2, May 31, 2000
and overall impressions
This document is the report of
impressions and recommendations written by the Review Team for the Operations
Readiness Review of the Sloan Digital Sky Survey (SDSS) conducted from April 25
to 27, 2000 at Apache Point Observatory (APO).
The overall impression is very
positive. All project personnel
have every reason to be very proud of their accomplishments.
The project has confirmed the basic concept of the Survey and, even
though the actual Survey has not officially started, already has some impressive
science achievements to its credit.
The Review Team was extremely
impressed with the progress the SDSS Project has made to date.
The imaging camera and spectrographs appear to be performing at or near
survey levels and are producing impressive scientific results.
The facilities for instrument handling and configuration seem
professionally engineered and well suited to safely perform the required task.
The ability to acquire and process the staggering amount of astronomical
data is striking. The high level of
professionalism and dedication of the project personnel is widely evident.
The Project already has some
impressive achievements to its credit, including not only the highest redshift
quasar found to date but a large enough sample of quasars to make an estimate of
the quasar luminosity function at redshifts higher than 3.5; the discovery of
several brown dwarfs; and the identification of enough halo stars to identify
tidal remnants. These early results
are widely known within the astronomical community and already demonstrate the
types of discoveries that will be enabled by such large, uniformly selected
samples of objects. The dataset
that will be provided by the complete Sloan survey will have a profound and
lasting impact on many fields of astrophysics.
The overall impression from the
review is that the engineering effort is well under control with a fairly well
defined set of issues to be completed in order to begin full-scale survey
operations. There were no glaring
mechanical issues that had not been addressed already, in some fashion, by the
engineering team. With the
exception of some issues to do with image quality, the telescope performance
appears to be quite good and instrument performance meets specification.
The telescope interlock system is well thought out with only a few
remaining issues to do with instrument interlocks.
Several efficiency related hardware enhancements were presented and the
scope and implementation of these improvements seems reasonable.
for all aspects of the project was provided to the Review Team via www posting
previous to the review. More
in-depth information was provided in presentations at the review for all aspects
of the project except the data processing that is performed at Fermilab and the
Project Management structure and process. A
separate review of the data processing performed at Fermilab will be conducted
at Fermilab before the end of July. Paper
copies of the www posted information and the presentations were provided to the
Review Team at the review.
The initial review planning,
schedule, and logistics for housing, meals, and meetings were performed by the
SDSS project. The actual review
details and preparation of this report were performed by the Review Team.
In addition to attendance and
discussion at the presentations, the Review Team organized themselves into three
breakout sessions, roamed the observatory watching operations and interacting
with Observers, met several times in executive session, and presented a
close-out session at the conclusion of the review.
The Review Team organization and subjects of
the breakout sessions were:
Systems Requirements & Science Commissioning
(Comments in section 5, Q&As in section 8.1)
Review Team Chair
2) Observers and Observatory Operations (Comments in section 6, Q&As in section 8.2)
3) Hardware Systems and Their Maintenance (Comments in section 7, Q&As in section 8.3)
note about duplication and negativity
The Review Team has taken the
position that this report should serve as a management tool to aid the Project
in the very difficult transition from construction/commissioning to production
is emphasized here because this project has set very stringent quality,
quantity, and uniformity goals to achieve in a relatively short period of time.
The "production" pressure in this project will be a constant
source of conflict adding to the already arduous task of managing such an
ambitious effort. To that end, we
have assumed the fundamental question asked of us is not, What is right?
although a list of what is right would be very long, but, What is wrong? and
recommendations on approaches to address these issues.
To the credit of the project, there follow more than 20 pages of What is
wrong? We say "to the credit
of the project" because essentially all of the items listed were raised by
project personnel, most with proposed solutions.
The lists of items appear
longer than they really are because there is duplication of items from different
Team Members but from different perspectives.
The duplications remain to preserve the different perspectives in the
hope they will be useful to produce a more effective solution.
The size of the observing staff is a good example.
It is clear the observing staff is not large enough to accomplish all the
work required. One perspective is
the number of people required, another is how to most effectively use the
valuable resource represented by the high-level of personnel now on staff, and
another is the skill mix that may be suitable after some years of operations.
The questions and answers are
included as recorded to preserve the original context.
This has resulted in more duplication but is retained to preserve the
additional insight provided by the different perspectives of the sources.
While there are many issues to
address, as would be expected in such an ambitious endeavor, one must keep in
mind we are addressing the last few % of data quality and processes related to
observing efficiency, not any fundamental flaw in either concept or execution.
Again, the Review Team is very
positively impressed with the accomplishments to date and predicts a high
probability of success.
This report is largely a
compilation of writings by the Review Team members, questions asked by the
Review Team members, and answers to these questions written by Project staff
within a few hours. Section and
subsection numbering and headings have been added to facilitate organization and
referencing, however, the original text is retained to preserve the tone and
context of the original writings.
Sections 5, 6, and 7 of this
report correspond to the breakout session numbers 1, 2, and 3 above and
contain the Review Team member's writings organized by breakout session.
Section 8 is a list of the Review Team members.
The original hard copy report
contained a section of the questions and answers organized by breakout session
and a section about an Email
discussion conducted by two reviewers a few days after the review about the
performance versus efficiency controversy. It also contained a section
with comments by one reviewer, who was also a reviewer for the
secondary mirror damage investigation, about the justification for replacing the
mount with the new design and a section of one reviewer's list of work items.
These sections are not included in the on-line version of the report.
to the Review Team
The construction phase of the
Sloan Digital Sky Survey is complete, the testing phase is nearly complete, and
commissioning operations have been underway for some time.
Science quality data has been obtained from the commissioning operations,
analyzed, and published. The Survey
is now making the transition from commissioning to routine operations. This
transition is not complete and there is much to be done as we shift our focus
from building and testing equipment to making the equipment reliable and
efficient. At the same time,
we are building a scientific staff, which will bring to fruition the goals that
we established a decade ago. These
goals were to image about 10,000 square degrees of the Northern Galactic Cap; to
select from those images a million objects, primarily galaxies and quasars; to
obtain spectra of those objects; and to create a catalog of those images and
spectra, first for the astronomers from the participating institutions and then
for the astronomy community-at-large.
We are planning two reviews of
the Survey in order to help us set our course for the next five years. The first
review, which will be held at Apache Point, from April 25 to 27, will examine
whether the Observing Systems and the Observatory Support will be able to
sustain the five years of observing and produce the quality of data that is
needed to achieve our scientific objectives. The second review will be held at
Fermilab, the site of the SDSS data processing and distribution, prior to the
end of July. In that review we will ask the reviewers to examine whether the
data processing operations can process the data in a timely manner and
effectively distribute it to the collaboration and then the astronomy community
at large in accordance with our distribution plan developed in cooperation with
the National Science Foundation.
The specific charge for the
review of the Observing Systems and Observatory support is contained in three
sets of questions, which we would like this Review Team to answer:
"Will the Observing Systems, with the proposed improvements, be
ready to support the five-year survey? Will
they, as you find them on April 25, be sufficient to achieve our scientific
As of April 25, the data do meet, apart from possible questions of image
quality, the quality requirements. The
project is far short of being able to produce data in the required quantities.
Solving that problem will require a substantial investment in real-time
software, and it is not clear whether adequate resources have been identified.
"Has the observing staff been prepared to support a five-year
survey? In particular has the SDSS Management given them the tools and training
to carry out the survey? Is the size of the observing staff sufficient to
support a five-year survey?"
The observing staff appears to have the commitment, competence, and
training required, apart possibly from issues of quality assessment of the data.
It is not clear they have the tools required to achieve adequate
efficiency of operation. The
observing staff needs to be augmented by one, but in addition, closer attention
needs to be paid to optimizing the range of skill levels, the way shifts are
scheduled, and the effective use of the observers.
"Does the project have sufficient technical personnel at APO and the
participating institutions to support the maintenance and continuous improvement
of efficiency and reliability? Given the geographic dispersion of people will
our management plan work?"
We were not given enough management information to assess the answer to
this question. Better project management is a prerequisite for evaluating
We are requesting more information and time to review this for the July
Requirements and Science Commissioning
Status and Production Tracking
versus quantity conflict management
In any production activity, the
desire to improve the product will be in conflict with the desire to produce a
larger quantity of data. It will be
extremely important to define "good enough" and to resist requirements
creep. The expansion of the science
utility of the survey beyond its current state is evident and future expansion,
while noble in ideal, may place at the risk the ability to achieve the necessary
volume within the available resources.
for tracking production progress, conic progress charts
Transitioning into the
production mode of the survey will be challenging. As we reviewed the presentations, the challenges being
experienced in the transition from commissioning to operation were evident.
More detailed and thoughtful
analysis will be necessary to identify the areas of efficiency or performance
improvements required to achieve the necessary production rates.
It must focus attention on the trades that will be required between
performance improvements and efficiency improvements.
These items are at different ends of the project time horizon and need to
be viewed in harmony.
The Review Team concluded that
a top-level metric would be helpful in both focusing the goals of the survey as
well as tracking the progress. Such
a tool would assist in identifying where actions are required to ensure the end
result meets the scientific requirements within the allocated budget.
The example charts shown in Figures 1 and 2 could be used to track the
progress of the survey over the funded five years.
The survey objectives are expressed as minimum accomplishments for survey
success. The lower level bounds the
science projects enabled with the currently achieved system performance over
square degrees of sky in Figure 1 and number of redshifts in Figure 2. The upper level is what additional science projects would be
enabled with slightly improved performance and a more optimistic assessment of
sky coverage or number of spectra. By
plotting the progress in these areas, everyone associated with the project can
tell at a glance the progress to date.
At this juncture, the team
suggests that more attention should be applied to improving the operational
efficiency of the survey even if it is at the risk of delaying or eliminating
additional performance improvements.
1. Example Survey Progress Tracking Tool
2. Example Survey Progress Tracking Tool
Some data of the caliber
required for the survey have already been obtained, even at this early stage of
operation. However, we also heard
about many improvements that could be made in both hardware and software that
would improve performance, reliability, efficiency, and safety.
What was not presented to us was a systems level overview of relative
importance of these various improvement projects when measured against the
scientific requirements of the survey, a method for establishing the priorities
among the different projects, an integrated schedule for completing them, or a
mapping of projects onto available budget and manpower resources.
The tools for managing projects were not in evidence during our visit,
and so we are not able to determine the extent to which they are being used. However, in a project of this complexity, given the
distributed nature of the staff and resources, and the competing and probably
conflicting claims for regular observations and continued enhancements to the
facility, the use of project management tools is essential.
Management of software
development is notoriously more difficult than the management of hardware
projects. Not surprisingly, it is
our impression that the real-time software for supporting operations is
converging more slowly than the non-real-time software and hardware components
of the system. Since a very high
level of efficiency as measured in terms of time on the sky is required in order
to achieve the project goals in a period of five years, resources will have to
be identified for continued software development.
Quantitative goals for efficiency should be established, and priorities
for new software should be determined in order to achieve those goals as early
in the project as resources permit. Improved
automation in instrument set up, more efficient instrument changes, and
real-time quality assessment of data would all enhance throughput.
Safety must be a major concern
for any astronomical observatory. Small
groups of people are working in remote locations around massive moving
machinery. Equipment worth up to a
million dollars or even more must be operated, handled, and repaired, often late
at night when people are tired. Severe
environmental conditions, including lightning storms or blizzards that prevent
access for days at a time, are common. The
staff of the SDSS is very aware of these issues and has taken some steps to
address them. We particularly commend the staff for scheduling an external
review of safety by experts from other organizations. However, we recommend that
priority be given to establishing clear and enforced procedures for tagging and
lockout of systems that may present dangers to personnel; to providing systems
for lightning protection and specifying procedures for disconnecting key
equipment and taking other appropriate steps during storms; and to creating an
environment that encourages safety consciousness on the part of staff and
visitors and facilitates the reporting of safety problems.
We are concerned about external
access to the SDSS computer network. Hacking
is an increasingly serious problem for all networked systems, and observatories
are frequently targets. Appropriate
security measures should be put in place that allow reasonably convenient access
by external staff but that provide a substantial challenge to unauthorized
users. This is an area that will
have to be continuously reviewed and updated.
safety, Instrument-change risk
The current plan of operations
calls for changing between imaging and spectroscopy during the night as image
quality and cloud cover change. The
overhead of this change is substantial and each instrument change involves some
degree of risk. Steps should be
taken to: 1) speed up the process
of changing of instruments; 2) make the process as risk free as possible
including possibly limiting the number of instrument changes per night to one
and/or restricting how late at night they can occur; and 3) accumulating data
that enable simulations to determine whether such instrument changes actually
enhance throughput. For example, if seeing is typically strongly variable over
hourly intervals during the night, it may be better to wait out an hour of bad
seeing rather than switch to spectroscopy on photometric nights.
Some primitive forecasting tools may be helpful; if the jet stream is
over the site or a front is moving through, for example, good seeing is very
unlikely, and the night should probably be dedicated to spectroscopy.
of Science Requirements for the SDSS
A detailed evaluation of the
science requirements for the SDSS is available in Scientific Requirements and
Scientific Commissioning for the SDSS, written and/or edited by M. Strauss.
This document outlines the scope of the project, its scientific goals,
and provides quality specifications for the data to be cataloged.
The Review Team is truly impressed at the degree to which most of these
goals have been met or exceeded. For
example, the 100mas astrometric accuracy greatly exceeds the 180mas
"Drop-Dead" requirement and already meets the goal required for proper
motion determinations. Throughput
goals for the imager are reached for all but one CCD.
The spectrographs essentially meet or exceed all criteria tested thus
far, including throughput, resolution, wavelength coverage, and stability.
These and other technical feats
attest to exceptional talent, effort, and dedication on the part of all
The one survey criterion that
is still far from adequate is overall efficiency.
If the project is to achieve its stated imaging and spectroscopic goals,
it must achieve a collective change in perspective from one of
"building" to one of "producing". This change in perspective also necessitates a change
in attitude and management philosophy. When
evaluating a proposed modification in hardware or procedure, consideration must
be given not only to cost (in dollars) and benefit (in data quality), but also
to the project downtime required to implement the modification.
Once the 5-year clock begins to run, every hour that the telescope is not
on the sky is another portion of a stripe that is not imaged, or a field that
will not produce spectra. This
reality poses a real dilemma for most astronomers, who are not accustomed to
thinking in terms of production. As
research scientists, it is our nature to push an idea or technology to its
physical limits. Yielding to that
tendency at this point in the SDSS must be very critically weighed against the
impact of the decision on the eventual science that is obtainable from the data.
A case in point is the current desire to improve the secondary mirror
support to potentially improve an astrometric accuracy that already meets or
exceeds all projected goals. However,
the quandary is evident across the project: from image quality to CCD
performance to calibration requirements.
for evaluating observing efficiency versus performance
In delving into this question,
it is the opinion of the Review Team that the SDSS scientific/ management staff
has not yet implemented a firm mechanism for evaluating the scientific impact of
technical and operational decisions that must be made.
For example, the Strauss document fails to quantify the scientific
compromise that would result from failing to reach ANY of the technical
specifications. It is our
recommendation that a small group of scientists from the member institutions
with deep interests in the project should be assembled to carefully consider the
scientific goals of the project and justify the technical requirements and real
coverage which are motivated by that science.
This input would be an essential ingredient to the setting of the
performance benchmarks by which progress will be judged.
The group should also evaluate the scientific impact of proposed upgrades
in hardware or changes in operational policy.
At a minimum, this process would weigh possible improvements in data
quality against the likely reductions in sky coverage.
In some cases, tradeoffs of performance vs. instrument safety might also
need to be considered. Final
decisions on such proposals would, of course, be made by the Project Director on
the basis of the recommendation of that group as well as the advice of the
Project Scientist and Project Manager. A
starting point for this scientific advisory committee might be the existing SOC.
Another reviewer comments:
The Observing efficiency versus
performance section 5.3.3 highlighted the need for a committee of survey
scientists that can trade performance and coverage against scientific goals.
That point is relevant to a question that Jim Gunn asked us about
near-term tactics. Jim wanted to know if they should shut down for a couple of
months and fix everything or continue to have a mix of science verification data
gathering and improvements. Our
point should be to emphasize that the project needs a process whereby they can
answer such questions for themselves. That
process requires adequate management information and the benchmarks to assess
impact on science. If such
questions are handed to the science committee for evaluation, their advice then
informs the director's decision, mediated by the Project Scientist.
The existence of such an apparatus would engage the scientific partners
and allow the kind of trades we've been advocating.
efficiency versus observing staff
The required increase in survey
efficiency also drives a need for better organization of effort within the
mountain staff. Because the Review
Team was not presented with an overall organization chart of personnel, it was
not possible to accurately evaluate lines of communication or the adequacy of
the staff to address the outstanding technical and operational problems.
Clearly, this must improve. Timetables
must be generated for the completion of key outstanding software and hardware
projects. Manpower must be identified and their time budgeted.
Safety procedures must be solidified, followed, and monitored.
The existing talent mix must be evaluated in the context of a production
operation, and adjustments made where necessary.
In short, a "top-down" approach must be adopted, where the
primary benchmark is progress toward completion of the survey.
as resource, Lead Observer
An important resource which
appears to be largely overlooked is the team of Observers.
Perhaps more than any other group at this point, these individuals will
be essential in realizing the project’s potential.
As Ph.D.-level scientists, they are an expensive team with a great deal
of talent to offer toward reaching the overall efficiency goals.
Indeed, it is clear to this Review Team that the Observers also take a
personal stake in the success of the project.
The current organization, however, does not provide an effective means
for them to exercise their talents and education or a good path for their ideas
to be communicated to the higher-level scientific and managerial staff where
they might be implemented. The
result is reduced survey productivity and a disillusioned observing staff.
It is the very strong opinion of the Review Team that a Head Observer
should be hired, among whose duties would be the management and organization of
the Observers, the setting of their schedules, their hiring and training, and
the setting of their duties. He/she
would also monitor the overall performance of the team, fill in for
sickness/vacation, and coordinate observing procedures with relevant management
personnel. The Head Observer would
act not only as a conduit for raising important suggestions and concerns of the
Observers, but also as a filter for the many competing influences that they
and Observatory Operations
The SDSS survey is in a
transition mode between commissioning and operations. The Review Team feels that it is important that this
transition occur and SDSS moves into regular operations with high efficiency.
Two specific questions related to this are:
What are the major impediments to efficient operations?
Who makes the list of things to be done and who decides priorities and
of responsibility to Observers
During the commissioning phase,
these issues were best addressed by engineering and project people.
However, moving now into an operations mode, it is felt that the people
charged with delivering the "product", i.e. the Observers, should now
play a more prominent role. This is
particularly true if one considers a chart showing % complete survey versus
time, which is reflected in the strategy that SDSS needs to adopt over the
coming years. The Observers group has direct responsibility flowing from
this, to play a leading role.
to observing efficiency
The Observers group was asked
one by one to identify the major impediments to efficient completion of the
survey from their experience with nightly operations. The following major issues
It is felt that with the
current problems and inefficiencies, there was no way the spectroscopic portion
of the survey could be completed in any reasonable time scale.
The particular problems mentioned were: guide fiber acquisition speed,
poor pointing model, stable focus, and extra (possibly unnecessary)
Quality Assessment Tools
The lack of these prevents a
real-time assessment of the data quality and completeness. This means the
exposure times are probably being extended unnecessarily longer to guarantee
good quality data.
and troubleshooting experience
A lack of proper documentation
and troubleshooting experience has meant that problems often take longer to fix
than they should.
The complicated operation of
the building roll-off and roll-on requires they be more conservative with
observing conditions since the SDSS telescope is more exposed and vulnerable in
case of a sudden weather change.
rather than involved
Since the Observers are now
playing a leading role in the completion of the SDSS survey, their involvement
in the process of problem solving and follow-up needs to change. The
issues/problems they have raised which effect the efficiency of their
observations are of critical importance with the SDSS moving into an Operations
mode. A greater involvement in the
Prioritization, follow-up, and deadlines for improvements and problem fixes
should be given to the Observers.
This process requires closer
consultation and collaboration with Engineering. One comment we heard repeated several times was that the
Observers are not internal to this process.
They are "informed" rather than "involved".
One way of improving this situation might be to provide greater overlap
with daytime engineering operations. Both
for the evening hand-over and start up, but also in the morning when engineering
work is being planned. Some type of
regular formal meetings between Observers and Engineering to plan and prioritize
should be organized.
In the data production phase of
the survey, the observers represent a key asset for efficient and successful
completion. They were selected to
have research experience and interest in the survey data. That level of expertise is highly valuable during the current
commissioning stage, during which they can identify, verify, and occasionally
implement improvements to operational efficiency, particularly at the software
and user interface level. Several
personnel issues related to this group should be addressed in order to assure
their productivity and longevity with the project.
Staffing Schedule cannot be maintained
In the documents provided for
the review and in the presentations and discussions, it was clear that the
current staffing schedule is not maintainable over the long run.
In order to avoid burnout, two objectives must be realized in the near
term, without compromising safety and with minimal impact on operational
efficiency. One is to significantly
reduce or eliminate the frequency of "shift slipping", in which
schedules are changed among full night, early evening, late evening, and days.
The observers are currently showing the signs of fatigue that come to any
shift workers that have to alternate shifts frequently.
The other is to allow more substantial overlap with day workers, with
benefits to on-site operations as discussed above. More extensive overlap with consortium scientists during
working hours is another significant advantage.
Several creative alternatives to the current paired shift assignments are
possible, and should be explored.
Observer, off-shift time recommended
The observers themselves
suggested two additional observers should be added to the staff.
We recommend that the current complement be augmented by one, in order to
create the position of Lead Observer. The
primary purpose is effective management of this key scientific resource.
A major goal of the position is to give the observers a more significant
voice in the scientific planning and execution of the survey. This individual will interact with (and preferably be part
of) the project management team, assuring that the priorities as articulated by
the observers for investments to enhance survey efficiency are considered
carefully in the course of project prioritization.
This lead person will communicate the priorities and status of the
project to the observers, coordinate their scheduling, plan their non-shift
observatory projects, guide their skill enhancement and career development, and
personally serve on shift, to be both an effective group leader and to create
more off-shift time for the observers.
of responsibility to Observers
With a strong site-based cadre
of observers, responsibility for tactical decision-making on execution of the
survey should devolve to the site. Following
the models of queue observing at ESO and WIYN, the imaging and spectroscopic
goals for each full lunation would be developed by the science planning team,
then transmitted to APO, where the observers would make the nightly choices
about the details of program execution. The
observers still desire frequent communication with the off-site survey team
members. This level of delegation
of responsibility is appropriate for the skill level of the observers and the
nature of the "production" phase of the Survey.
As the survey progresses and
data acquisition becomes a more nearly routine activity, it will be worthwhile
to consider a change of skill mix if the necessity arises to fill a vacant
observer position. Ph.D.-level
research experience is a worthwhile prerequisite for data quality assurance and
assessment of short- and long-term impact on science of instrument and telescope
anomalies. Such a background will
be less critical for telescope, instrument, and facility operations when these
functions are mature. At that time,
Survey management should consider replacement with a more traditional
"telescope operator" profile. For
the remaining, relatively short duration of the survey, that person could be
assigned nearly exclusively to operating shifts, further freeing the observers
to concentrate more on data quality, pipeline applications, and the option of
longer blocks of non-work hours to make progress on science projects.
3, Observatory Operations, Engineering - Software
The operational software
appears to be in good shape overall to support initial observing.
There is clearly additional functionality needed and many rough edges to
be addressed to provide software to support efficient operations that are
maintainable through five years of production data taking.
Observer Interfaces, dedicated status displays
Many different software
sub-systems have been integrated to support operations.
We agree with the assessment that a layer of software integration,
including user friendly interfaces, and easy to organize and use status
displays, is needed to support 5 years of operations.
We encourage the survey to find the resources to implement a first
version of this over the next few months. Since
these are the panels that are in daily use by the Observing staff, the
requirements should be identified through watching their use of the software
during observations and discussion of their needs and ideas towards more
We recommend the addition or
allocation of dedicated status screens in the control room to show existing and
enhanced status displays.
functionality of Observer's Programs (IOP, SOP,
MOP), companion observer
There is a need for increased
functionality and stability of the xOP programs and for real-time quality
analysis of the data. For increased
operational efficiency, both in the short and long term, the Observers must have
continued interaction and the opportunity to work with the developers
responsible for this software. The
quality analysis programs should be configured to ensure they do not perturb the
main observing process. If the
speed of the networks allow, perhaps there is an opportunity for a
"companion observer" at Fermilab to help with QA during observing.
to transition the focus from development to production operations
We were not presented with a
prioritized list of software tasks, assignments, and schedule.
We recommend that one be posted. We
were not presented with any plan for the transition from development of the
software to an operations focus and any accompanying transfer of knowledge and
responsibility. We recommend that
such a plan be developed to address operational use of the software for the next
management and documentation
We endorse a long-term
commitment to configuration management and to provide operational and reference
documentation. We encourage
including documentation in configuration management.
We recommend including regular inspections (code and documentation) in
the configuration management plan.
for MCP and real-time programming
We agree with the concern and
recommendation to identify someone with long-term
responsibility for the maintenance and support of the Motion Control Processor.
We recommend the "intermittent axis-motion problems" be fixed.
Attention should continue to be paid to this software and if problems
persist effort identified to do the necessary rewrite of the PVT code.
Concern that the MCP code is not interpolating all TCC commands correctly
should be understood. We identified
a lack of real-time programming expertise at APO and recommend that some be
acquired to provide backup support for the experts at Fermilab.
We encourage the software
project management to maintain awareness of those parts of the software that
have only one person with any knowledge of the internals or one person who has
contributed to the code.
and schedule for upgrades and maintenance
A schedule for deployment of
the semi-automated instrument-change system should be posted and the necessary
resources and priorities assigned.
Plans for incremental upgrade
of the data acquisition system to address those parts becoming unsupportable
seem appropriate. Sufficient time
should be planned for the integration and testing of changes.
We recommend construction and
posting of a calendar of all the maintenance tasks with their time and resource
Systems and Their Maintenance
In general, the reviewers feel
that priorities should be balanced so as to:
guarantee the safety of personnel and the instrumentation,
improve the image performance and instrumentation robustness,
improve operational efficiency.
The overall goal should be to,
as expeditiously as possible, bring the instrumentation to a level of
performance consistent with a finalized set of scientific objectives.
An aggressive, yet plausible, timeline should be established for this
objective so as to guarantee the timely completion of the survey.
Listed below is a summary of
key improvements and enhancements that, in some way, lead to improved safety,
performance, or efficiency. This
list is not intended to be all-inclusive but rather to emphasize the issues the
reviewers felt were most important.
Related Mechanical Issues
Catastrophic risk to the
instrumentation due to lightning
should be mitigated further via implementation of proper telescope grounding and
the installation of some sort of formal early warning system.
As it stands now, the instrumentation is particularly susceptible when
the enclosure is rolled off.
Safe handling of the spectroscopic
corrector should be ensured through completion of the dedicated handling
cart and storage area.
The completion of the interlock
system for safe engagement of the imager
optical benches needs to be ironed out.
The understanding is that engagement must be performed with the telescope
near the zenith to prevent misalignment, and possible damage, of the kinematic
There is an issue with
potential outgassing of the camera getter
in the event of a camera warm-up. The
engineering team has planned to remove the getter during the summer shutdown.
We agree with this line of thinking.
A wind screen shock absorber is
needed to prevent damage to the secondary support system in the event of a
The frequency of recoating the primary and secondary should be minimized
to mitigate the risk to the optics.
Related Mechanical Issues
There is uncertainty regarding
the source and severity of the image degradation that is observed in the
telescope. Before the relatively
risky operation is undertaken to ship the secondary mirror and corrector to the
Lick Observatory for characterization, the project should gather enough image
data to confidently characterize the problem and investigate other possible
sources of the degradation. In
particular, thermometry should be implemented as soon as possible to investigate
in particular any effect of the following on local seeing:
accuracy of the enclosure temperature control,
localized heat sources,
temperature of the wind baffle relative to the air, and the possibility
of a temperature gradient across the baffle.
This should be done before the
removal of the secondary mirror and common corrector for characterization.
The Max Plank Institute has agreed to lend the SDSS a thermal imaging
camera for thermal studies.
Related Mechanical Issues
It is not clear whether a
manual or automated instrument-change system is most beneficial.
Whichever method is implemented, the method must:
be kept absolutely as simple as possible,
be thoroughly debugged, and
be implemented as a routine, standard procedure with the users thoroughly
trained in that procedure.
The throughput of the
Photometric Telescope must be improved to keep pace with the rest of the
The mechanical spare parts list
should be established and the critical spares kept on the site.
Spectroscopic corrector safety
latches should be implemented to minimize the inconvenience of using the special
Problems with the DIMM should
8. The Review Team
Crocker, Chair Johns
Rohloff Max Plank
Institute for Astrophysics
University of Arizona
NIST/University of Maryland