SDSS DAS User's Guide
The DAS webserver will time out for queries taking longer than 20
minutes to process. However, the script looking for your data will
still continue to run. The page you see after submitting a query will
have instructions on what to do if that happens.
Here is an indication
of how long it takes to process queries:
# files time
100 5 sec
1000 60 sec
2000 135 sec
5000 660 sec
10000 >20 min (times out)
The above timing estimates assumes that the rsync or wget
method of downloading the files was selected; selecting tar.gz, tar,
or zip methods take substantially longer to process and are therefore
limited to queries requesting 10 or fewer files.
The location of files on disk at FNAL is described in the Data Model Directory Structure
document. It is up to you to define the root directory for data you
download to your machine. The data files you retrieve from the DAS
(either as a .tar, .tar.gz, or .zip file archive file or as an rsync command) will be organized into
subdirectories under your root directory, mimicking the directory
structure at FNAL.
The Web Interface
The web interface to the
DR6 DAS is configured for public access to the DR6 data.
The steps to use this interface are outlined on the interface:
- Select one or more data products:
Use the buttons to select
what kind of files you want. If you don't know what the abbreviations
mean, click on the short file description to see the data model
description of the file type.
For each file type, the table on the right lists which
parameters you need to specify in your upload file to locate files
of this type. So, for example, if you want to retrieve fpC files,
your upload file must contain run, rerun,
camCol, filter, field for each object.
- Create the upload file:
The upload file you create contains a list of objects. For each
of the objects, you must specify the necessary parameters as given in
the table, based on the data products you selected in the previous
step. There are four formats that the DAS understands. It uses the
file specification (.csv, .tbl, .par, .fit, or .fits) to guess the
file format, so please name your files appropriately. Here are sample
- Comma Separated Value
(imaging.csv) for imaging data. This contains run, rerun, camcol,
field, and filter, so it can be used to get fpC files, for example.
Note that the first line is a comma separated list of the parameter names.
This is mandatory.
- Comma Separated Value
(spectro.csv) for spectro data. This contains plate, mjd,
fiberID, and sprerun (the spectro rerun, always 25 for DR6), so it
could be used to get spSpec files.
- Table File (imaging.tbl) for
imaging data. This contains field, run, rerun, and camCol, so it can
be used to get tsObj files. To get fpC or fpBIN files, you would need
to add a "filter" column.
- Table File (spectro.tbl) for
spectro data. This contains the same information as the spectro.cvs
file above. Note that the columns are in a different order. This
does not matter -- as long as the information in the column header
agrees with the information in the data for the column!
- ASCII Parameter File (imaging.par)
for imaging data. This contains run, rerun, camcol, field,
- ASCII Parameter File (spectro.par)
for spectro data. This contains plate, mjd, and rerun, and can
be used to get spPlate files.
- FITS Binary Table (.fit) for
imaging data. This contains run, rerun, camcol, field, and filter
information, and can be used to get fpC files. If you use it to get
fpObjc files, the filter information will be ignored.
- FITS Binary Table (.fit) for
spectro data. This contains plate, mjd, fiberID, and rerun information,
and can be used to get spSpec files.
- Select the file to upload: Once you have a file, either
type in its name, or browse to find it on your local computer.
- Select the transfer format:
The recommended option is for the DAS to create an rsync include
file, to be used with rsync (see our rsync help
page), or a wget (http client) include file which will copy over
all the files you requested directly via a command line interface from
our rsync or http server.
Since the rsync and wget commands require no overhead for combining the
requested files into an archive (.tar, .tar.gz, or .zip) file and
since it only copies files which do not exist or have changed on your
local machine, it is the preferred method for transferring large
numbers of files. Due to this consideration, any request for more
than 10 files will default to the rsync option. See our rsync help page) for help installing rsync.
There are three further options for transferring the data,
which are, however, only available for small data sets (up to 10
files) to prevent large requests from locking up the DAS. UNIX
users will likely wish to use either a .tar or a .tar.gz archive file
containing the files requested, and Windows users will probably want a
.zip archive file.
- Submit: Self-explanatory.
What happens next?
If you chose the .tar, .tar.gz, or .zip transfer format options --
and if you requested 10 or fewer files -- a script runs which
looks for all of the files you requested, makes an archive of them,
prepares a listing of the files, and presents you with a web page
where you can download the archive. It will look something like this:
SDSS Data Archive Server Download Results (DR6)
Use your browser to download the results file. /webscratch/DAS_27303_1096658512/imaging.tar.gz
(Note that any files generated under /export/data/www/data/webscratch/DAS_27303_1096658512 will have a
limited lifetime -- typically 24 hours.)
The file size is 7194872 bytes.
SDSS wrapper report Fri Oct 1 14:21:57 CDT 2004
Total number of bytes to be written: 7192974
(NOTE: The above is example output. Actual results files are removed
from our system daily.)
If you chose the rsync transfer format option, or if you
requested more than 10 files, you will obtain a page describing
how you can transfer them to your computer using rsync by using the
rsync include file the DAS creates:
SDSS Data Archive Server Results (dr6)
You have requested X FITS, postscript, and/or gif files.
The rsync password you will need is "dr6". There are three ways to specify the password:
rsync to download the files (see our rsync help page).
rsync will mirror the data archive directories containing the files
you have requested on your disk.
In the directory where you would like the directories containing your data to be placed,
save this link as sdss-rsync.lis
rsync -vtrLPR rsync://email@example.com/DR6/ ./ --include-from=sdss-rsync.lis
- Type the password when prompted after issuing the rsync command.
- First set an environment variable RSYNC_PASSWORD to the password, then issue the rsync command
- First save the password in a file (on UNIX-like systems, the
file must not be readable
by anyone but the user), then tell rsync its location with
the option --password-file=FILENAME
What if it does not work?
Please run one of the examples to see if the server is functional.
Then please check your file to confirm you have it formatted
correctly. Submit your problem to
the SDSS helpdesk and
include the following:
We keep usage logs for each access, sorted by date, and this
information will help us track down trouble.
- your query or upload file
- the timestamp and directory name at the top of the results page
(a line like
Begin access on Fri Feb 28 11:46:45 CST 2003
- the URL of the particular DAS webserver you are using
- the version of DP that the DAS webserver was using;
this is listed at the bottom of the DAS query form
(e.g., "DP Version: dp v14_207;")
What Data Are Available?
Sky coverage for DR6 is described on the coverage page.
See also the imaging
and spectro root
directories for more information.
Do I have to use the Web Interface?
No, there are other ways to get at the data. One way is to use the
rsync command from your computers command line as described above using an appropriate
include file like sdss-rsync.lis. In
addition, other methods of access are described on the Data Access Page.
Last modified: Thu Jun 14 18:17:23 CEST 2007