Commit fe1a280f authored by Lisa Gerhardt's avatar Lisa Gerhardt Committed by Rollin Thomas

Adding missing data transfer pages

parent 34b340b1
# Using bbcp at NERSC
Bbcp is a point-to-point network file copy application with excellent
network transfer rates. The application was originally written for
transferring large files of the data-intensive High-Energy Physics
community. Bbcp is available on the NERSC Data Transfer Nodes.
## Requirements
To transfer files into/out of NERSC using bbcp, you need an SSH
client. For most Unix-like systems (Linux//MacOS/Cygwin), the command
ssh is sufficient and the bbcp executable. You can download the bbcp
executable from the [maintainer's
site](http://www.slac.stanford.edu/~abh/bbcp/).
## Usage
All example commands below are executed on your local machine, not the
NERSC machine:
The syntax of bbcp is similar to the syntax of scp, with some special
options specifying how to run ssh/bbcp.
Get a file from Data Transfer Node:
```shell
bbcp -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" "user_name@dtn01.nersc.gov:/remote/path/file" /local/path
```
Send a file to Data Transfer Node:
```shell
bbcp -T "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp" /local/path/file "user_name@dtn01.nersc.gov:/remote/path/"
```
Get a file from an outside host to NERSC:
```shell
bbcp -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /path/to/bbcp/on/remote/host" "user_name@remote.host.com:/remote/path/file" /local/path
```
Send a file from NERSC to an outside host:
```shell
bbcp -T "ssh -x -a -oFallBackToRsh=no %I -l %U %H /path/to/bbcp/on/remote/host" /local/path/file "user_name@remote.host.com:/remote/path/"
```
Note the difference between "-S" and "-T" option, "-S" means the source (where the data come from), "-T" means the target (where the data goes to).
In case you get the following error or similar, add the "-z" option to your command line (right after bbcp).
```shell
bbcp: Accept timed out on port 5031
bbcp: Unable to allocate more than 0 of 8 data streams.
Killed by signal 15.
```
# GridFTP Data Transfer
GridFTP is a command line service for parallel movement of data. You
may find it easier to use [Globus](globus.md), which uses the same
underlying gridFTP but adds reliability, performance, and ease of use.
GridFTP is a command line service for parallel movement of data. For
most transfers, we recommend [Globus](globus.md), which uses the same
underlying gridFTP but adds reliability, performance, and ease of
use. However, users occasionally find the flexibility of GridFTP
useful.
## Availability
......
# Using scp at NERSC
Secure Copy (scp) is used to securely transfer files between two hosts
using the Secure Shell (ssh) protocol.
!!! tip
Scp is suggested for smaller files (<~10GB), otherwise use
[Globus](../services/globus.md).
To transfer files into/out of NERSC using scp, you need an ssh
client. On Linux/Unix and MacOS these should be installed by default,
but on Windows you will need a GUI tool such as WinSCP.
## Usage
Get a file from Data Transfer Node
```shell
scp user_name@dtn01.nersc.gov:/remote/path/myfile.txt /local/path
```
Send a file to Data Transfer Node
```shell
scp /local/path/myfile.txt user_name@dtn01.nersc.gov:/remote/path
```
Use a pre-existing ssh key (like one made by sshproxy)
```shell
scp -i ~/.ssh/nersc user_name@dtn01.nersc.gov:/remote/path/myfile.txt /local/path
```
### Using tar+ssh
When you want to transfer many small files in a directory, we
recommend [Globus](../services/globus.md). If you don't wish to use
Globus, you can consider using ssh piped with tar.
Send a directory to Data Transfer Node:
```shell
tar cz /loca/path/dirname | ssh user_name@dtn01.nersc.gov tar zxv -C /remote/path
```
Get a directory from Data Transfer Node:
```shell
ssh user_name@dtn01.nersc.gov tar cz /remote/path/dirname | tar zxv -C /local/path
```
\ No newline at end of file
......@@ -125,6 +125,8 @@ nav:
- CVMFS : services/cvmfs.md
- Databases: services/databases.md
- GridFTP: services/gridftp.md
- Scp: services/scp.md
- Bbcp: services/bbcp.md
- Programming:
- Build Tools:
- programming/build-tools/index.md
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment