Observing Operations | Reviews | Survey Management

Sloan Digital Sky Survey
Review of Data Processing and Distribution Operations


Technical Summary of the Photometric Pipeline
Jill Knapp and Robert Lupton
Princeton University
June 20, 2000


 

Dear Reader:

We are beginning, or perhaps tiptoeing around, writing up the technical
paper describing the Photometric Pipeline.  The following outline is
a summary of what Photo does:
 

PHOTO paper: Outline
====================
 

Draft of June 20 2000
 
 

The first two chapters are background and expected to get into few details.
 

1. Introduction
===============

  The SDSS
    Filters, CCD mosaic, data taking
    CCD photometry: particular problems and characteristics
    TDI mode
    pixel scale
    strips and stripes
    lateral overlaps
    expected PSF
    time to cross array
    expected/observed sky brightness
    expected PS sensitivities
    expected surface brightness sensitivities
    saturation
    position calibration: Photo <-> Astrom
    photometric calibration: Photo <-> MTpipe
 

 
  What Photo has to do
  Data rate
  Execution time
  Goals have been met
  Outline of the rest of the paper
  Plans for distribution of code
  useful URLs
 

  Tables:  Filters, nominal wavelengths, widths, PS sensitivities

    Sky coverage, size of a frame, pixel size, lateral overlap area,
      time to cross array, integration time

  Figures: pretty pic of camera, or diagram of filter layout

    pretty color pic of a piece of the sky
 
 

2. Photo: Overall Architecture
==============================
 

  DA system and astroline
  Motivation for architecture
    Frames
      needs:
    PSP
      needs:
    SSC
  (I have put it in this order because we need to explain why we need
   PSP and in turn why we need SSC)
  Necessary inputs to Frames
  Data file formats and nomenclature
 

  Table: list of inputs:
    postage stamps
    darks
    bias
    flats
    fang & gang files
    astroline stamps
    astrom stars
    astrom and ptpipe trans structures and files
 
 
  Figures: data flow through photometric pipelines
 

3. SSC
======

  Inputs from astroline
  cutting frames and overlaps
  aligning frames to make a field
   ("data startup cost" goes here?)
  cutting postage stamps
    definition of different kinds of postage stamp
  outputs

  Table: inputs to SSC
   outputs from SSC
   (do we need this?)
 
 

4. PSP-1: Overview and Functionality
====================================

  Tasks of PSP
  Inputs to PSP
  Outputs from PSP
  Analysis of idB files: smoothing/averaging lengths
       overscan
       flat field
       sky
  Postage stamps: different types and their purpose
    PSF
    wings
    frame stars
    ghosts
  Wings of bright stars
  diffraction spikes
  ghosts (?)

  Tables: Inputs
   Outputs

  Figures: data flow through PSP
  montage of postage stamps
  variation of sky & flat field on a typical night
 

5. PSP-2: PSF
======================

**Note: this will probably be a separate paper**

  The problem
    Optics
    Seeing
    Photometry
  Modeling the PSF
    Optics, seeing and spatial variation
    stationarity
    basis functions
    KL decomposition
    required number of stamps per frame
    Stokes parameters
    rejection
    using modelled PSF
    aperture corrections
    PSF widths
  Interpolation of PSF over a run
  Dealing with too few postage stamps
  Science: some remarks about seeing
    atmospheric modeling
    promise further analysis

Table: PSF decomposition: terms and coefficients

Figures: Aperture corrections vs time: "before" and "after"
  Model vs real PSF across frame, and residuals
  Variation of PSF width across camera
  Variation of PSF width as a function of time
         QU plots
         width plots
 

6. Frames-1  Introduction
=========================

  Inputs and outputs (general)
  A walk through Frames
    what gets done where
    what gets done when
  data problems
    pixel size and band limit
    bad columns
    CRs
    bleed trails
  wings, ghosts
  variable PSFs
  Image analysis
 

  Tables:  Inputs
    Outputs (summary)

  Figures:  a frame of raw data
     the same frame corrected
     data flow through Frames
 

7. Frames-2: Correction of data defects
=======================================

  Introduction
    description of a frame
    saturation levels in each filter
    application of bias
    application of flat
  Bad columns
    typical number of bad columns
    origin of bad column map
    correcting for bad column
  Cosmic rays
    finding cosmics
    correcting for cosmics
    number of cosmics in a typical frame
  Bleed trails
  Serial register artifacts
  Outputs: corrected frames

  Figures: performance of interpolation across columns
    "before" and "after" of
      cosmic ray
      bad column
      bleed trail
 

7. Frames-3: Finding and Combining Objects
==========================================

  First pass: find bright objects
    saturated stars (already known)
    subtract bright objects
    wings of saturated stars
    (?exploiting wings to measure magnitude?)
  Second pass: finding objects to the limit
    psf filtering
    find objects
    binning/smoothing
    finding more objects
    blank sky objects
  Growing objects
  Combining filters
  Blank sky objects
  Definition of atlas images
  Outputs
    atlas images
    (compression of atlas images)
    object subtracted binned frames
    masks

  Tables:  Limits
     (faintest & brightest PSs, lowest and highest
      surface brightness)
    How much sky is lost
    Outputs

  Figures: `before' and 'after' subtraction of saturated star with wings
     Frame with atlas image boundaries and centroids of all objects
     Example atlas images
     Frame reconstructed from Atlas Images
 

8. Frames-4: Measure Objects
============================
 

  (understood: everything below is followed by "& uncertainties")
  sky subtraction: global vs local
  sub-pixellization
  centroiding, sinc-shifting and astrometric offsets between bands
  sector array and radial profiles
  objects at frame edges
  processing flags
  photometry
    psf
    aperture
    3" aperture
    radial profile
    isophotal
    luptitudes and procedure for "upper limits"
    Petrosian magnitudes
    Petrosian radii
  shapes
    ellipticities
    position angles
    Stokes parameters
    isophotal shape
  model fits
    definition
    library
    psf convolution
    likelihoods
    model magnitudes and model parameters
    model colors
  star/galaxy separation
  **this may be a separate paper**
  roughness parameter(s)
  PSF fraction

  Figures: map of sector array
  PSF vs model vs aperture magnitudes for point sources
  deV/exponential/psf/likelihood prism diagrams vs magnitude
  sample color color and color magnitude diagrams

  Tables: Photo outputs
   Photo flags
 

9. Frames-5: Astrometric and Photometric Calibration
====================================================

This should be short.  There should be a reference to whatever photometric
calibration paper we have/will have.

  Astrometric calibration:
  Anomalous refraction
  Map of CCD array and position information
    astrometric trans structures (w reference to ASTROM paper)
    centroiding and transfer objects
  Photometric calibration:
    photometric trans structures
      calibration stars
      final calibration

  Figures: sample color-color and color-magnitude diagrams
    color pictures of some galaxies
    star and galaxy counts
 

10. Frames-6: Deblending
========================

**note: this will be a separate paper, already drafter**

  Statement of the problem
    Definition of a blended object
    decomposition algorithm
    problem objects

  Tables: number of blended objects as a function of magnitude

  Figures: images of several examples of parents and children
    schematic to show how the algorithm works
 

11. Frames-7: Moving Objects
=============================

  The problem
    expected proper motions
    very fast objects
    moving objects as contaminants
  finding moving objects
  flags
  prospects for future (Kuiper belt etc)

  Figures: velocities showing asteroids
    (color-magnitude plots of asteroids)
    (color-color plots of asteroids)

12. Photo Summary: Outputs and Performance
==========================================

  Outputs
  Processing flags
  performance on several platforms
    profiles: memory, timing

  Tables: Measured parameters
   Output parameters
   Flags
   Output formats

  Figures: timing/memory diagrams
 

13. Testing and QA
==================

  History
  tracking problems
  testing PSP: simulations
  simulations: description
  HST fields, including the Groth strip
  clusters
  overlaps: lateral, half field, repeat

  The QA test suite: tracking Photo's performance
 

  Tables: known regions observed

  Figures: simulated images
    scan of globular cluster
    scan of HST field
 

14. Summary
===========

  Photo performance: accuracy, completeness, speed
  Information produced
  list of web sites for more information
 

15. Future
==========

  Coaddition
  cosubtraction
  atlas image analysis

  SDSS boilerplate etc
  plans for releasing the code