SDSS DAS User's Guide

Caveats

The DAS webserver will time out for queries taking longer than 20 minutes to process. However, the script looking for your data will still continue to run. The page you see after submitting a query will have instructions on what to do if that happens.

Here is an indication of how long it takes to process queries:

   # files      time
  requested
  ---------     ----
     100        5 sec
    1000       60 sec
    2000      135 sec
    5000      660 sec
   10000      >20 min (times out)

Directory Formats

The location of files on disk at FNAL is described in the Data Model Directory Structure document. It is up to you to define the root directory for data you download to your machine. The data files you retrieve from the DAS (either as a .tar, .tar.gz, or .zip file archive file or as an rsync command) will be organized into subdirectories under your root directory, mimicking the directory structure at FNAL.

The Web Interface

The web interface to the DR6 DAS is configured for public access to the DR6 data.

The steps to use this interface are outlined on the interface:

1. Select one or more data products:

   Use the buttons to select what kind of files you want. If you don't know what the abbreviations mean, click on the short file description to see the data model description of the file type.

   For each file type, the table on the right lists which parameters you need to specify in your upload file to locate files of this type. So, for example, if you want to retrieve fpC files, your upload file must contain run, rerun, camCol, filter, field for each object.

2. Create the upload file: The upload file you create contains a list of objects. For each of the objects, you must specify the necessary parameters as given in the table, based on the data products you selected in the previous step. There are four formats that the DAS understands. It uses the file specification (.csv, .tbl, .par, .fit, or .fits) to guess the file format, so please name your files appropriately. Here are sample files for:
   • Comma Separated Value (imaging.csv) for imaging data. This contains run, rerun, camcol, field, and filter, so it can be used to get fpC files, for example. Note that the first line is a comma separated list of the parameter names. This is mandatory.
   • Comma Separated Value (spectro.csv) for spectro data. This contains plate, mjd, fiberID, and sprerun (the spectro rerun, always 25 for DR6), so it could be used to get spSpec files.
   • Table File (imaging.tbl) for imaging data. This contains field, run, rerun, and camCol, so it can be used to get tsObj files. To get fpC or fpBIN files, you would need to add a "filter" column.
   • Table File (spectro.tbl) for spectro data. This contains the same information as the spectro.cvs file above. Note that the columns are in a different order. This does not matter -- as long as the information in the column header agrees with the information in the data for the column!
   • ASCII Parameter File (imaging.par) for imaging data. This contains run, rerun, camcol, field, filter.
   • ASCII Parameter File (spectro.par) for spectro data. This contains plate, mjd, and rerun, and can be used to get spPlate files.
   • FITS Binary Table (.fit) for imaging data. This contains run, rerun, camcol, field, and filter information, and can be used to get fpC files. If you use it to get fpObjc files, the filter information will be ignored.
   • FITS Binary Table (.fit) for spectro data. This contains plate, mjd, fiberID, and rerun information, and can be used to get spSpec files.
3. Select the file to upload: Once you have a file, either type in its name, or browse to find it on your local computer.
4. Select the transfer format:

   The recommended option is for the DAS to create an rsync include file, to be used with rsync (see our rsync help page), or a wget (http client) include file which will copy over all the files you requested directly via a command line interface from our rsync or http server. Since the rsync and wget commands require no overhead for combining the requested files into an archive (.tar, .tar.gz, or .zip) file and since it only copies files which do not exist or have changed on your local machine, it is the preferred method for transferring large numbers of files. Due to this consideration, any request for more than 10 files will default to the rsync option. See our rsync help page) for help installing rsync.

   There are three further options for transferring the data, which are, however, only available for small data sets (up to 10 files) to prevent large requests from locking up the DAS. UNIX users will likely wish to use either a .tar or a .tar.gz archive file containing the files requested, and Windows users will probably want a .zip archive file.

5. Submit: Self-explanatory.

What happens next?

If you chose the .tar, .tar.gz, or .zip transfer format options -- and if you requested 10 or fewer files -- a script runs which looks for all of the files you requested, makes an archive of them, prepares a listing of the files, and presents you with a web page where you can download the archive. It will look something like this:

SDSS Data Archive Server Download Results (DR6)

Use your browser to download the results file. /webscratch/DAS_27303_1096658512/imaging.tar.gz

(Note that any files generated under /export/data/www/data/webscratch/DAS_27303_1096658512 will have a
limited lifetime -- typically 24 hours.)

The file size is 7194872 bytes.

SDSS wrapper report Fri Oct  1 14:21:57 CDT 2004
Version v16_80 
Total number of bytes to be written:  7192974
        2403437 1356/40/corr/3/fpC-001356-r3-0086.fit.gz
        2398552 1356/40/corr/3/fpC-001356-r3-0087.fit.gz
        2390985 1356/40/corr/3/fpC-001356-r3-0088.fit.gz

If you chose the rsync transfer format option, or if you requested more than 10 files, you will obtain a page describing how you can transfer them to your computer using rsync by using the rsync include file the DAS creates:

SDSS Data Archive Server Results (dr6)
You have requested X FITS, postscript, and/or gif files.

Use rsync to download the files (see our rsync help page).

rsync will mirror the data archive directories containing the files
you have requested on your disk.

In the directory where you would like the directories containing your data to be placed, 
save this link as sdss-rsync.lis
and type:

        rsync -vtrLPR rsync://dr6@rsync.sdss.org/DR6/ ./ --include-from=sdss-rsync.lis

• Type the password when prompted after issuing the rsync command.
• First set an environment variable RSYNC_PASSWORD to the password, then issue the rsync command
• First save the password in a file (on UNIX-like systems, the file must not be readable by anyone but the user), then tell rsync its location with the option --password-file=FILENAME

What if it does not work?

• your query or upload file
• the timestamp and directory name at the top of the results page (a line like Begin access on Fri Feb 28 11:46:45 CST 2003 at /webscratch/DAS_21769_1046454405
• the URL of the particular DAS webserver you are using
• the version of DP that the DAS webserver was using; this is listed at the bottom of the DAS query form (e.g., "DP Version: dp v14_207;")

What Data Are Available?

See also the imaging and spectro root directories for more information.

Do I have to use the Web Interface?

No, there are other ways to get at the data. One way is to use the rsync command from your computers command line as described above using an appropriate include file like sdss-rsync.lis. In addition, other methods of access are described on the Data Access Page.