Go to sdss.org
DR7 Home
Where to Start
Data
   Database (CAS)
   Files (DAS)
   Value-Added
   Tutorials
   Older Releases
About DR7
   Sky Coverage
   Instruments
   Data Processing
   Algorithms
Help
   FAQ
   Glossary
   Known Issues
   DR Papers
   Helpdesk
   Search

Rsync Help for SDSS DAS Users

Users of the SDSS Data Archive Server (DAS) may use rsync, an open source utility that provides fast incremental file transfer. Rsync is available for download at the rsync website. (Note that wget and other http-based mass download tools can be used to accomplish the same task, as described on the DAS homepage.) Here we provide detailed instructions for installation and use of rsync on a variety of platforms.

The same instructions apply for mirroring all or parts of the data archive server through each of the rsync URLs

  • rsync://user@rsync.sdss.org/imaging
  • rsync://user@rsync.sdss.org/spectro
  • rsync://user@rsync.sdss.org/pt
  • rsync://user@rsync.sdss.org/nightly
  • rsync://user@rsync.sdss.org/unprocessed
with password sdss.

BSD/Unix/Linux

Rsync is often installed by default on BSD/Unix/Linux systems. It is included in Mac OS X, for instance. If it is not installed on your system, or you would like to upgrade to the latest version, you can find links to the available downloads. We recommend downloading and compiling the source, rather than retrieving the binary.

The source can be downloaded directly from the rsync download area. You will need to gunzip and untar the file if your browser does not do so automatically. A directory called rsync-2.5.6 is created; just cd into that directory, and execute
./configure
This will create a Makefile and config.h appropriate for your system. Then type
make
to compile the code. Finally, to put rsync in your path, either add it by hand, or type
make install.

At this point, you are ready to use rsync. You can use DAS access page to generate a file that can be used by rsync to download the data set you can specify:

  1. Follow the "Upload a user generated list of spectroscopic fibers" or "Upload a user generated list of imaging fields" depending on whether you want spectroscopic or imaging data.
  2. Select a data release and supply a table describing the data you want, as instructed on the form.
  3. Follow the "this form" link in the "Download selection" section of the resultant page.
  4. Select the filters (for imaging) and file types you want, and check "rsync" for the download method.
  5. Submit the form, and save the result. The default name of the saved file will be of the form sdss-rsync-XXXXX.lis.
This file contains a listing of all the data files to be pulled over by rsync. Then, change the working directory to where you want the downloaded directory tree to be placed and execute
rsync -vtrLPRH --copy-unsafe-links --include-from=sdss-rsync-XXXXX.lis rsync://user@rsync.sdss.org/imaging/ ./
or
rsync -vtrLPRH --copy-unsafe-links --include-from=sdss-rsync-XXXXX.lis rsync://user@rsync.sdss.org/spectro/ ./
to retrieve your files. (If you use rsync for mirroring the archive, you can create your own include file, or point rsync directly to a specific directory if only one directory is to be downloaded.) You will be prompted for a password, which is dr7. If you will be downloading often, you can set an environment variable of this password by adding
setenv RSYNC_PASSWORD XXXX
to your .cshrc file, or
export RSYNC_PASSWORD=XXXX
to your .bashrc file.

NOTE:If you are behind a firewall, you will need to
setenv RSYNC_PROXY host:port
(or the equivalent export in non-csh-like shells) to point to your institution's proxy server and port number.

Windows

Windows users cannot download and run rsync directly. You will first have to install cygwin, a linux emulator. It works with all non-beta, non "release candidate", ix86 versions of Windows since Windows 95, with the exception of Windows CE. Cygwin is available for download here, free of charge. Click on the "Install Now" link, and a file called setup.exe will be downloaded. Running this will allow you to install Cygwin.

Because rsync is not included with Cygwin, you will also need to download and install rsync. Use your web browser to download the rsync source code. Open a Cygwin session, cd to the location of the source, and follow the instructions in the Unix section above to install rsync.

Macintosh

Mac users with OSX can follow the instructions in the Unix section above to download and install rsync. Users with OS9.x will unfortunately not be able to use this utility.


Last modified: Thu Jun 14 18:10:45 CEST 2007