Observing Operations | Reviews | Survey Management

Sloan Digital Sky Survey
Review of Data Processing and Distribution Operations

Technical Summary of the Photometric Pipeline
Jill Knapp and Robert Lupton
Princeton University
June 20, 2000


Dear Reader:

We are beginning, or perhaps tiptoeing around, writing up the technical
paper describing the Photometric Pipeline.  The following outline is
a summary of what Photo does:

PHOTO paper: Outline

Draft of June 20 2000

The first two chapters are background and expected to get into few details.

1. Introduction

  The SDSS
    Filters, CCD mosaic, data taking
    CCD photometry: particular problems and characteristics
    TDI mode
    pixel scale
    strips and stripes
    lateral overlaps
    expected PSF
    time to cross array
    expected/observed sky brightness
    expected PS sensitivities
    expected surface brightness sensitivities
    position calibration: Photo <-> Astrom
    photometric calibration: Photo <-> MTpipe

  What Photo has to do
  Data rate
  Execution time
  Goals have been met
  Outline of the rest of the paper
  Plans for distribution of code
  useful URLs

  Tables:  Filters, nominal wavelengths, widths, PS sensitivities

    Sky coverage, size of a frame, pixel size, lateral overlap area,
      time to cross array, integration time

  Figures: pretty pic of camera, or diagram of filter layout

    pretty color pic of a piece of the sky

2. Photo: Overall Architecture

  DA system and astroline
  Motivation for architecture
  (I have put it in this order because we need to explain why we need
   PSP and in turn why we need SSC)
  Necessary inputs to Frames
  Data file formats and nomenclature

  Table: list of inputs:
    postage stamps
    fang & gang files
    astroline stamps
    astrom stars
    astrom and ptpipe trans structures and files
  Figures: data flow through photometric pipelines

3. SSC

  Inputs from astroline
  cutting frames and overlaps
  aligning frames to make a field
   ("data startup cost" goes here?)
  cutting postage stamps
    definition of different kinds of postage stamp

  Table: inputs to SSC
   outputs from SSC
   (do we need this?)

4. PSP-1: Overview and Functionality

  Tasks of PSP
  Inputs to PSP
  Outputs from PSP
  Analysis of idB files: smoothing/averaging lengths
       flat field
  Postage stamps: different types and their purpose
    frame stars
  Wings of bright stars
  diffraction spikes
  ghosts (?)

  Tables: Inputs

  Figures: data flow through PSP
  montage of postage stamps
  variation of sky & flat field on a typical night

5. PSP-2: PSF

**Note: this will probably be a separate paper**

  The problem
  Modeling the PSF
    Optics, seeing and spatial variation
    basis functions
    KL decomposition
    required number of stamps per frame
    Stokes parameters
    using modelled PSF
    aperture corrections
    PSF widths
  Interpolation of PSF over a run
  Dealing with too few postage stamps
  Science: some remarks about seeing
    atmospheric modeling
    promise further analysis

Table: PSF decomposition: terms and coefficients

Figures: Aperture corrections vs time: "before" and "after"
  Model vs real PSF across frame, and residuals
  Variation of PSF width across camera
  Variation of PSF width as a function of time
         QU plots
         width plots

6. Frames-1  Introduction

  Inputs and outputs (general)
  A walk through Frames
    what gets done where
    what gets done when
  data problems
    pixel size and band limit
    bad columns
    bleed trails
  wings, ghosts
  variable PSFs
  Image analysis

  Tables:  Inputs
    Outputs (summary)

  Figures:  a frame of raw data
     the same frame corrected
     data flow through Frames

7. Frames-2: Correction of data defects

    description of a frame
    saturation levels in each filter
    application of bias
    application of flat
  Bad columns
    typical number of bad columns
    origin of bad column map
    correcting for bad column
  Cosmic rays
    finding cosmics
    correcting for cosmics
    number of cosmics in a typical frame
  Bleed trails
  Serial register artifacts
  Outputs: corrected frames

  Figures: performance of interpolation across columns
    "before" and "after" of
      cosmic ray
      bad column
      bleed trail

7. Frames-3: Finding and Combining Objects

  First pass: find bright objects
    saturated stars (already known)
    subtract bright objects
    wings of saturated stars
    (?exploiting wings to measure magnitude?)
  Second pass: finding objects to the limit
    psf filtering
    find objects
    finding more objects
    blank sky objects
  Growing objects
  Combining filters
  Blank sky objects
  Definition of atlas images
    atlas images
    (compression of atlas images)
    object subtracted binned frames

  Tables:  Limits
     (faintest & brightest PSs, lowest and highest
      surface brightness)
    How much sky is lost

  Figures: `before' and 'after' subtraction of saturated star with wings
     Frame with atlas image boundaries and centroids of all objects
     Example atlas images
     Frame reconstructed from Atlas Images

8. Frames-4: Measure Objects

  (understood: everything below is followed by "& uncertainties")
  sky subtraction: global vs local
  centroiding, sinc-shifting and astrometric offsets between bands
  sector array and radial profiles
  objects at frame edges
  processing flags
    3" aperture
    radial profile
    luptitudes and procedure for "upper limits"
    Petrosian magnitudes
    Petrosian radii
    position angles
    Stokes parameters
    isophotal shape
  model fits
    psf convolution
    model magnitudes and model parameters
    model colors
  star/galaxy separation
  **this may be a separate paper**
  roughness parameter(s)
  PSF fraction

  Figures: map of sector array
  PSF vs model vs aperture magnitudes for point sources
  deV/exponential/psf/likelihood prism diagrams vs magnitude
  sample color color and color magnitude diagrams

  Tables: Photo outputs
   Photo flags

9. Frames-5: Astrometric and Photometric Calibration

This should be short.  There should be a reference to whatever photometric
calibration paper we have/will have.

  Astrometric calibration:
  Anomalous refraction
  Map of CCD array and position information
    astrometric trans structures (w reference to ASTROM paper)
    centroiding and transfer objects
  Photometric calibration:
    photometric trans structures
      calibration stars
      final calibration

  Figures: sample color-color and color-magnitude diagrams
    color pictures of some galaxies
    star and galaxy counts

10. Frames-6: Deblending

**note: this will be a separate paper, already drafter**

  Statement of the problem
    Definition of a blended object
    decomposition algorithm
    problem objects

  Tables: number of blended objects as a function of magnitude

  Figures: images of several examples of parents and children
    schematic to show how the algorithm works

11. Frames-7: Moving Objects

  The problem
    expected proper motions
    very fast objects
    moving objects as contaminants
  finding moving objects
  prospects for future (Kuiper belt etc)

  Figures: velocities showing asteroids
    (color-magnitude plots of asteroids)
    (color-color plots of asteroids)

12. Photo Summary: Outputs and Performance

  Processing flags
  performance on several platforms
    profiles: memory, timing

  Tables: Measured parameters
   Output parameters
   Output formats

  Figures: timing/memory diagrams

13. Testing and QA

  tracking problems
  testing PSP: simulations
  simulations: description
  HST fields, including the Groth strip
  overlaps: lateral, half field, repeat

  The QA test suite: tracking Photo's performance

  Tables: known regions observed

  Figures: simulated images
    scan of globular cluster
    scan of HST field

14. Summary

  Photo performance: accuracy, completeness, speed
  Information produced
  list of web sites for more information

15. Future

  atlas image analysis

  SDSS boilerplate etc
  plans for releasing the code

Submit comments and questions to the SDSS Helpdesk. Last updated 04/01/14