NAME

bbftp - transfer files using compression and parallel streams

SYNOPSIS

bbftp -v

bbftp [Options] [-u RemoteUsername] -i ControlFile [RemoteHost]

bbftp [Options] [-u RemoteUsername] -e ControlCommands [RemoteHost]

DESCRIPTION

Use the bbftp command to transfer files between the local host and a remote host. In order to get better performance on a loaded Wide Area Network than transferring with the standard ftp command, use the bbftp command. bbftp has been designed to take advantage of the RFC 1323 and uses multiple streams is order to speed up transfer. It also implements an automatic retry in case of failure of commands contained in the ControlFile or in the ControlCommands.

bbftp allows two methods of connection to the remote host, a direct connection on the control port of the remote bbftpd daemon or the ability to remotely start a bbftpd daemon through an ssh tunnel. For the first method, the bbftpd daemon has to be started (by inetd or as a standalone daemon) on the remote host and all user information (username, password) are transmitted encrypted to the daemon; for the second the bbftpd binary has to be accessible somewhere on the remote host and all control data are transmitted through the ssh tunnel.

A third additionnal authenticate mode allows to use certificates to log on. This mode is based on the Grid Security Infrastructure and requires Globus software to be installed. The client side needs a certificate to identify itself and the daemon needs a host certificate.

The behaviour of bbftp is controled by the ControlFile which contains the commands to be executed (option -i) or by commands separated by semicolons (option -e). The format of these commands are given in CONTROL COMMANDS.

bbftp may be used in one of the following ways:

% bbftp

to print a short help

% bbftp -v

to write the version of the software and default values to standard output.

% bbftp [Options] -i ControlFile [-u RemoteUsername] [RemoteHost]

% bbftp [Options] -e ControlCommands [-u RemoteUsername] [RemoteHost]

to request the execution of commands contained in the control file ControlFile or the ControlCommands using RemoteUsername on RemoteHost.

Depending on the Options a password for the RemoteUsername on RemoteHost may be requested interactively (see OPTIONS section, for a full description of the Options).

The RemoteUsername value is ignored in the certificate authentication mode. The remote user is automatically detected by the daemon using the Globus gridmap file.

OPTIONS

Options can be separated into two classes, those describing the way bbftp connects to the bbftpd daemon and those modifying the behaviour of the control commands.

CONNECTION OPTIONS

-u

Use this option to specify the user on the remote host. In SSH mode (see below) this option is optional: if not specified, the user in SSH config file will be used, or the local user if not in the SSH config file. When compiled with the certificates authentication module, the use of this option will determine the way the authentication is done: if -u is used, the classical user/password will be used, otherwise the certificate will be used.

-s

Use this option to use ssh to remotely start a bbftpd daemon. It usually starts the binary "bbftpd -s", but this can be changed througth the -E option.

-S

Same as -s but ask ssh to run without asking a question (no password, no host key checking). It implies the usage of the identity file (for those familiar to ssh, this in done in setting the ssh options BatchMode to yes and the option StrictHostKeyChecking to no).

-E Server command to run

This option has to be used if the binary to be started on the remote host is not the default one (usually "bbftpd -s"). This option also implies the usage of the ssh mode (no need to add the -s option)

-I SSH identity file

This option has to be used if the ssh identity file is not the default one (usually $HOME/.ssh/identity). This option also implies the usage of the ssh mode (no need to add the -s option)

-L SSH command

If your ssh command is not the default one (usually "ssh -q"), use this option to change it. This option also implies the usage of the ssh mode (no need to add the -s option)

-P Private authentication string

If your client have been compiled with a private authentication schema, this option allow to pass an arbitrary string to the authentication module. You can determine if your client is using a private authentication module with the -v option.

-w Control port number

Use this option to change the control port for connection to on the RemoteHost. This option is meaningless in ssh mode.

PROTOCOL OPTIONS

-D[min:max] Domain for ephemeral ports

Use this option to use protocol V2 (non-passive mode) instead of V3 (passive mode). min and max are the range of ports used for data sockets. If min and max are not specified, free ports will be used (or built-in ports at compilation time).

BEHAVIOUR OPTIONS

-b

Use this option to run bbftp in the background after all interactive requests.

-c

Use this option to gzip the data during transmission. Compression and uncompression are done "on the fly" and do not require any additionnal disk space. Do not use it if the files to transmit are not compressible, as this will only lead to a waste of CPU and time. This option can be overridden by the control command setoption gzip

-f ErrorFile

Use this option to redirect the data generated on the standard error to ErrorFile

-g Globus service name

Use this option in certificate mode if the server does not use a host server. The service name will be the subject of the certificate the server is actually using.

-m

Use to special output on file transfer. The ouput is in the following format:

Direction NumberOfBytes NumberOfSeconds BuffersizePerStream SendWinSize RecvWinSize NumberOfStreams Compression

BuffersizePerStream, SendWinSize, RecvWinSize are expressed in KiloBytes

For example :

get 10000 10 256 256 256 2 gzip

means that 10000 Bytes have been transfered in 10 seconds in compressed mode using 2 streams, a buffer size per stream of 256 KBytes, a TCP send window size of 256 KBytes and a TCP receive window size of 256 KBytes.

This option is useful when trying to choose the best parameters between two sites.

When this option is set -V, -W and -t options are not available.

-n

Use this option to simulate the transfer. All commands will be executed but no data will be transfered with the get, mget put and mput commands. Output is the same as a real transfer. OutputFile

-o OutputFile

Use this option to redirect the data generated on the standard output to OutputFile

-q

Use this option to mark packets for QBSS (QBone Scavenger Service)

-p NumberOfParallelStreams

Use this option to increase the number of streams to use during the file transfer. Default is 1. This option can be overridden by the control command setnbstream

-r NumberOfTries

Use this option to change the number of tries to use when a transfer fails. The default is usually 5.

-R .bbftprc file

After a successful connection to the daemon the client is going to execute all control commands located in the "$HOME/.bbftprc" file. The location of this file can be changed with this option. Take care, not all control commands are allowed in the .bbftprc file (See CONTROL COMMANDS to know the authorized one) If this option is not provided and "$HOME/.bbftprc" does not exists, the client tries to execute all commands in "/etc/bbftp.conf".

-t

Use this option to have a timestamp on all output (overridden by -m option).

-V

Use this option to set the client in verbose mode (overridden by -m option).

-W

Use this option to print warnings to stderr (overridden by -m option).

CONTROL COMMANDS

The control commands are either contained by an ASCII file ( -i option) or written on the bbftp line ( -e option). They can be divided into two classes, the "File related commands" and the "Behaviour commands".

All "Behaviour commands" may be put in a .bbftprc file, but all "File related commands" are forbiden in that file.

IMPORTANT NOTE

Under the RFIO mode (see setoption remoterfio and setoption localrfio ) all file related commands have to be given in absolute mode.

FILE RELATED COMMANDS

cd "RemoteDirectory"

Change the current directory of the daemon on the remote host.

If the RemoteDirectory is given in relative mode (not beginning by a /), it is supposed to be relative to the directory where the daemon is currently running. After the first connection, the current directory is the home directory of the RemoteUsername .

The client keeps in mind the current remote directory so in case of broken connection during a transfer, it can reset the current directory of the daemon to the correct directory.

If the daemon has been set in RFIO mode (see setoption remoterfio ) this option is unavailable.

get " RemoteFile LocalFile"

Transfer the remote file RemoteFile to the local host with the name LocalFile. If the local file already exists it is overwritten (only in case of successful transfer if the setoption tmpfile has been set). Under some circumstances (No space left on device, Access denied, File is a directory ...), no retry is done and the next command is processed.

If the RemoteFile is given in relative mode (not beginning by a /), it is supposed to be relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the beginning). If the LocalFile is given in relative mode (not beginning by a /) the file is created relative to the directory where the bbftp command is running (which may have been changed with the lcd command).

get " RemoteFile LocalDir/"

Transfer the remote file RemoteFile to the local host with the name RemoteFile (without path) in the LocalDir directory.

If the local file already exists it is overwritten (only in case of successful transfer if the setoption tmpfile has been set). Under some circumstances (No space left on device, Access denied, File is a directory ...), no retry is done and the next command is processed.

If the RemoteFile is given in relative mode (not beginning by a /), it is supposed to be relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the beginning).

If the LocalDir is given in relative mode (not beginning by a /) the file is created relative to the directory where the bbftp command is running (which may have been changed with the lcd command).

get " RemoteFile"

Transfer the remote file RemoteFile to the local host with the name RemoteFile.

If the RemoteFile is given in relative mode (not beginning by a /), it is supposed to be relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the beginning) and created on the local host relative to the directory where the bbftp command is running (which may have been changed with the lcd command).

lcd " LocalDirectory"

Change the current directory on the local host. If the LocalDirectory is given in relative mode (not beginning by a /), it is supposed to be relative to the directory where the client is currently.

If the client has been set into RFIO mode (see setoption localrfio ) this option is unavailable.

mget " RemoteFiles LocalDirectory"

Expand the RemoteFiles on the remote machine and do a "get" for each file name thus produced. Files are transferred into the LocalDirectory.

mget " RemoteFiles"

Expand the RemoteFiles on the remote machine and do a "get" for each file name thus produced. Files are transferred into the local working directory, which can be changed with the lcd command.

mkdir " RemoteDirectory"

Create the directory RemoteDirectory on the remote host. If the directory already exist no retry is done and the next command of the file is processed. If the RemoteDirectory is given in relative mode (not beginning by a /) the directory is created relative to the current remote directory.

If one directory in the given path does not exist the command will fail if the setoption nocreatedir is set. If the setoption createdir has been set all unexisting directories will be created.

mput " LocalFiles RemoteDirectory"

Expand wild cards in the list of local files given as arguments and do a "put" for each file in the resulting list. Files are transfered into the RemoteDirectory.

mput " LocalFiles"

Expand wild cards in the list of local files given as arguments and do a "put" for each file in the resulting list. Files are transfered into the current remote directory which can be changed with the cd command.

put " LocalFile RemoteFile"

Transfer the local file LocalFile to the remote host with the name RemoteFile. If the remote file already exists it is overwritten. Under some circumstances (No space left on device, Access denied ...), no retry is done and the next command of the file is processed.

If the LocalFile is given in relative mode (not beginning by a /) the file is supposed to be relative to the directory where the bbftp command is running (which may have been changed with the lcd command).

If the RemoteFile is given in relative mode (not beginning by a /), it is created relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the beginning).

put " LocalFile RemoteDir/"

Transfer the local file LocalFile to the remote host with the name LocalFile (without the path) in the RemoteDir directory

If the remote file already exists it is overwritten. Under some circumstances (No space left on device, Access denied ...), no retry is done and the next command of the file is processed.

If the LocalFile is given in relative mode (not beginning by a /) the file is supposed to be relative to the directory where the bbftp command is running (which may have been changed with the lcd command).

If the RemoteDir is given in relative mode (not beginning by a /), it is created relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the beginning).

put " LocalFile"

Transfer the local file LocalFile to the remote host with the name LocalFile.

If the LocalFile is given in relative mode (not beginning by a /) the file is supposed to be relative to the directory where the bbftp command is running (which may have been changed with the lcd command) and created relative to the current directory on the remote host (which is set to the home directory of the RemoteUsername at the beginning).

dir " RemoteDir"

If used with '-m', lists the content of the remote directory. Each line has the following format:

LINE: <LINK_CHAR><TYPE_CHAR><SP><FILE_NAME>

LINK_CHAR: 'l' if file is a link else blank

TYPE_CHAR: 'd' if file is a directory, 'f' if it is a reguler file or 'u' if it is an invalid link

SP: ' '

FILE_NAME: STRING

If -m is not used, the result will be only 'OK' or 'FAILED'. This can be used to test the existence of a directory without printing its content.

rm " RemoteFile"

Deletes the remote file RemoteFile

stat " RemoteFile"

If used with '-m', print statistics about the file or the directory in the following format:

device inode mode size block_size blocks uid gid (access time) (modification time) (change time)

df " RemoteDir"

If used with '-m', print statistics about the file system that contains the directory RemoteDir in the following format:

total_blocks blocks_available block_size total_inode inode_available filename_max_length

BEHAVIOUR COMMANDS

setoption " Option"

To negate an option just add "no" before the option (ie setoption nocreatedir). The options are the following :

createdir

All file-related commands will create missing directories if needed (default createdir).

gzip

All file transfers will be compressed using the gzip algorythm (default nogzip).

keepaccess

The access time and modify time will be kept on each file transferred (default keepaccess).

keepmode

The file mode will be kept on each file transferred (default keepmode).

localrfio

All local files will be created with RFIO functions (default nolocalrfio).

remoterfio

All remote files will be created with RFIO functions (default noremoterfio).

qbss

All the packets will be marked for QBSS (default noqbss).

tmpfile

All files will be created under a temporary name (FileName.bbftp.tmp.HostName.Pid) and renamed to the correct file name if transfer is successful (default tmpfile)

setbuffersize " Buffersize"

Set the size in Kbytes of the buffer used for reading or writing the files. This command set the local and remote buffer size. (Each stream will use the same buffer size)

setlocalcos " LocalCos"

Set the local COS to the value specified by LocalCos. This COS will be used for further rfio funtions. It is used if the setoption localrfio has been set and if the file is a HPSS file. A value of 0 allows to select the COS according to the file size. A negative value allows to not set the COS. The default value is 0.

setlocalumask " LocalUmask"

Set the local umask to the value specified by LocalUmask. This umask will be used for further i/o funtions. The LocalUmask has to be given in OCTAL

setnbstream " NumberOfParallelStreams"

Set the number of parallel streams to NumberOfParallelStreams. This number will be used for further transfer commands.

setremotecos " RemoteCos"

Set the remote COS to the value specified by RemoteCos. This COS will be used for further rfio funtions. It is used if the setoption remoterfio has been set and if the file is a HPSS file. A value of 0 allows to select the COS according to the file size. A negative value allows to not set the COS. The default value is 0.

setremoteumask " RemoteUmask"

Set the remote umask to the value specified by RemoteUmask. This remote umask will be used for further i/o funtions. The RemoteUmask has to be given in OCTAL

setrecvwinsize " WindowSize"

Set size in Kbytes of the receive TCP window of each stream of the bbftpd daemon. This also set the send window size of the client to the same value.

setsendwinsize " WindowSize"

Set size in Kbytes of the send TCP window of each stream of the bbftpd daemon. This also set the receive window size of the client to the same value.

setackto " Acknowledge time-out" Set time-out (in seconds) to wait for an acknowledge. Default value is 100

setrecvcontrolto " Input control time-out"

Set time-out (in seconds) to wait while reading on the control socket. Default value is 180

setsendcontrolto " Output control time-out"

Set time-out (in seconds) to wait while writing on the control socket. Default value is 180

setdatato " Data time-out"

Set time-out (in seconds) to wait while reading on the data socket. Default value is 300

NOTES If the option tmpfile is used then if the new file ( RemoteFile for a put or LocalFile for a get) did not exist before, bbftp ensures that the file transfer was correct if the file exists.

In case of an already existing file, if the size, the last access and modification time are correct (if option keepaccess has been set) bbftp ensures that the file transfer was correct.

TURL NOTATION

The TURL notation can be used with commands relatives to files and directory.

If you use this notation, you should not specify the server name in command.

BBFTP support 3 types of TURL:

Remote TURL

bbftp://bbftp_servername/path_to_file_or_directory

The path to file can be relative or absolute (start with '/')

For example:

"mkdir bbftp://bbftp.domain/dir"

is the command to create a directory dir on the remote bbftp server under the user's home directory

"cd bbftp://bbftp.domain//tmp"

is the command to change into /tmp on the remote bbftp server

Local TURL

file://path_to_file

The path to file can be relative or absolute (start with '/')

For example:

"put file://file bbftp://bbftp.domain//tmp/file"

is the command to transfer the file file located on the local host server under the current directory to the remote host bbftp.domain under the /tmp directory

Local rfio TURL

rfio://path_to_file

The path to file can be relative or absolute (start with '/')

For example:

"put rfio:///path/to/MSS/file bbftp://bbftp.domain//tmp/file"

is the command to transfer a file retrived via the rfio path /path/to/MSS/file into the remote host bbftp.domain under the /tmp directory

When the 'rfio://' notation is used, the option 'localrfio' is no longer necessary.

Remote TURL with rfio

It is possible to use rfio on the remote side using mixed TURL like:

bbftp://bbftp_servername/rfio://path_to_rfio_file

When the 'bbftp://bbftp_servername/rfio://' notation is used, the option 'remoterfio' is no longer necessary.

RETURN VALUES

The following exit values are returned:

"0"

if all commands were successfuly executed

">0"

if one command failed.

It may happend that a non-zero value is returned even if all files were correctly transfered, if during one transfer a retry was needed. This will be corrected in future releases.

MESSAGES AND ERRORS

All informative messages are written to the standard ouput (or to the OutputFile ). All error messages are written to the standard error (or to the ErrorFile ).

WARNING

The bbftp client version 2.0.0 is unable to talk with a daemon in release 1.x.x.

The rfioxxx or xxxrfio commands are no longer supported, use instead the options localrfio or remoterfio in conjunction with put and get commands to obtain the same result.

RESULT FILE

If the -i option was used a result file will be created in the same directory as the ControlFile. Its name is ControlFile with the extension ".res". It contains the same lines as the ControlFile plus the keyword "OK", in case of success, or "FAILED", in case of failure.

If the -e option was used and the -V option was not used, the software will print the command executed plus the keyword "OK", in case of success, or "FAILED", in case of failure to standard output.

CONNECTION EXAMPLES

bbftp -i ctrlfile -u jon -p 5 -c cchost.in2p3.fr

means that bbftp is going to connect to remote host cchost.in2p3.fr using username jon. If the connection is successful then the commands in ctrlfile will be executed. All transfer commands will use five streams and gzip compression.

bbftp -i ctrlfile -u phg -s cchost.in2p3.fr

means that bbftp is going to start a remote bbftp via sshd on host cchost.in2p3.fr using username phg. ssh will first try an RSAAuthentication if it is allowed by cchost.in2p3.fr; otherwise ssh will ask for a password for user phg on cchost.in2p3.fr. Then the sshd on cchost.in2p3.fr will log user phg and try to start the command "bbftpd -s"

bbftp -i ctrlfile -u jon -E '/tmp/bbftpd -s' cchost.in2p3.fr

Same behaviour as preceding, except that the remote command will be "/tmp/bbftpd -s"

bbftp -i ctrlfile -u gilles -S cchost.in2p3.fr

means that bbftp is going to start using ssh a remote bbftpd on host cchost.in2p3.fr using username gilles. ssh will try an RSAAuthentication if it is allowed by cchost.in2p3.fr, otherwise the connection will be broken.

bbftp -e 'setrecvwinsize 1024 ; put file1 file2' -u jon cchost.in2p3.fr

means that bbftp is going to connect to remote host cchost.in2p3.fr using jon username. If the connection is successful then the commands setrecvwinsize 1024 and put file1 file2 will be executed. All tranfer commands will use one stream.

bbftp -e 'put file1 file2' cchost.in2p3.fr

means (in the certification authentication mode) that bbftp is going to connect to remote host cchost.in2p3.fr using a certificate. The remote user will be detected by the daemon which will check for the certificate provided and will accept or not the connection.

Using SSH to start a BBFTPD daemon linked with dynamic libraries

If you have linked the daemon with dynamic libraries with -L/path/to/lib option, you need to specify this location in $LD_LIBRARY_PATH. To be taken into account by SSHD, this environment variable must be modified in the $HOME/.ssh/environment file.

See your SSH or SSHD manual for more details.

EXAMPLES

User jon want to transfer files from host localhost to remotehost on the account bbrdist. The bbrdist account has /home/babar/bbrdist as default directory on remotehost but has no subdirectories. We are going to study a control file in order to understand bbftp behaviour (we do not care here about the connection method; see CONNECTION EXAMPLES for that).

User jon on the local host is on the /home/babar/jon directory and has the following control file (all lines have a number which must not exists but which are there just for clarity) :

1 setnbstream 20

2 setremoteumask 022

3 setoption nocreatedir

4 put /home/babar/jon/f1 /home/babar/bbrdist/newfiles/f1

5 setoption createdir

6 put /home/babar/jon/f1 /home/babar/bbrdist/newfiles/f1

7 setnbstream 5

8 setrecvwinsize 1024

9 setoption gzip

10 put /home/babar/jon/f2 /home/babar/bbrdist/newfiles/f2

Command 1 just sets the number of parallel streams to 20 for further get or put commands.

Command 2 sets the remote umask for further put commands.

Command 3 indicates that no directory has to be created on further put or get commands.

Command 4 tries to send the local file /home/babar/jon/f1 to /home/babar/bbrdist/newfiles/f1. This command will fail because bbrdist has no subdirectory and directory creation is inhibed.

Command 5 resets the createdir option.

Command 6 will be successful (if the connection does not break), because the creation of the directory /home/babar/bbrdist/newfiles has been authorized by the createdir option.

Command 7 reduces the number of streams to 5

Command 8 sets the receive TCP window size to 1024 Kbytes on remotehost and the send TCP window size to 1024 Kbytes on localhost.

Command 9 sets the gzip option for further get or put commands.

Command 10 will transfer file /home/babar/jon/f2 to /home/babar/bbrdist/newfiles/f2 with 5 streams in compressed mode.

AUTHORS

bbftp was developed by Gilles Farrache. It is now maintained by Lionel Schwarz at IN2P3 Computing Center , Villeurbanne (FRANCE).

CONTRIBUTORS

Tim Adye (Idea and implementation of ssh mode)

Gilles Gallot (Mutli-IP addresses support, secondary groups support, port on various systems and bug fixes)

Andrew Goodney (Port to Darwin)

Paola Grosso (Idea and implementation of the -q client option)

Petr Holub (Port to Windows cygwin)

Dan Schrager (Idea and implementation of the -D client option)

Rod Walker & Kostas Georgiou (Idea and implementation of the -g client option)

Shuwei Ye (Bug fix)

BUGS

Send bugs / comments to bbftp@in2p3.fr

SEE ALSO

bbftpd(1).