SSH Tectia

scpg3

scpg3 — Secure Shell file copy client - Generation 3

Synopsis

scpg3 [options...]
[ src_profile: | [user@] src_host [#port]: ] src_file...
[ dst_profile: | [user@] dst_host [#port]: ] dst_file_or_dir

Description

scpg3 is used to securely copy files over the network. scpg3 launches ssh-broker-g3 to provide a secure transport using the Secure Shell version 2 protocol. ssh-broker-g3 will ask for passwords or passphrases if they are needed for authentication. scpg3 uses the configuration specified in the ssh-broker-config.xml file.

Any filename may contain a host, user, and port specification to indicate that the file is to be copied to or from that host ( [user@] host [#port]). If no username is given, the local username is assumed. If no port is given, the default Secure Shell port 22 is assumed. Alternatively, a connection profile defined in the ssh-broker-config.xml file (profile) can be given.

When defining a connection profile in the scpg3 command, notice that SSH Tectia client tools for z/OS deduces the meaning of the argument differently depending on its format. If there is an @ sign in the given attribute value, SSH Tectia client tools for z/OS always interprets it to be <username@hostname>, i.e. not a profile.

Also, if there are dots in a profile name (for example host.x.example.com, the dots need to be escaped on command line. Enter host\.x\.example\.com, instead. Otherwise the profile name is taken as a host name and the current Windows user name is used for logging in.

Copies between two remote hosts are permitted. The remote host(s) must be running a Secure Shell version 2 server with the sftp-server subsystem enabled.

The host parameter can optionally be enclosed in square brackets ([]) to allow the use of semicolons. The file argument can contain simple wild cards: asterisk (*) for any number of any characters, and question mark (?) for any one character.

For information on special characters in file names, see the section called “Filename Support”.

Options

The following command-line parameters can be used to further specify the scpg3 options.

-a [arg]

Transfers files using the ASCII mode, that is, newlines will be converted on the fly. For transfers between SSH Tectia on z/OS and other hosts, this also enables automatic ASCII-EBCDIC conversions. See the ascii command in the section called “Commands”.

If the server does not advertise the newline convention, you can give it a hint by giving an argument after -a. The default is to set the destination newline convention, but you can specify either one by prefixing the argument with src: or dest: for source or destination convention, respectively. The available conventions are dos, unix, and mac, using \r\n, \n, and \r as newlines, respectively. An example is shown below:

$ scpg3 -asrc:unix -adest:dos src_host:src_file dest_host:dest_file
-b buffer_size_bytes

Defines the maximum buffer size for one read/write request (default: 32768 bytes).

-B, --batch-mode

Uses batch mode. Fails authentication if it requires user interaction on the terminal.

Using batch mode requires that you have previously saved the server host key on the client and set up a non-interactive method for user authentication (for example, host-based authentication or public-key authentication without a passphrase).

-d

Forces target to be a directory.

-D, --debug=LEVEL

Sets the debug level. LEVEL is a number from 0 to 99, where 99 specifies that all debug information should be displayed. This should be the first argument on the command line.

[Note]Note

The debug level can be set only when the scpg3 command starts the Connection Broker. This option has no effect in the command if the Connection Broker is already running.

-I, --interactive

Prompts whether to overwrite an existing destination file (does not work with -B).

-N max_requests

Defines the maximum number of read/write requests sent in parallel (default: 10).

-O, --offset=r<offset> | w<offset> | l<length> | t<length>

Sets offset. Offset r<offset> specifies the start offset in the source file. Offset w<offset> specifies the start offset in the destination file. Length l<length> specifies the amount of data to be copied. Truncate length t<length>, if given, specifies the length to which the destination file is truncated or expanded after the file data has been copied.

-p

Preserves the file permissions and the timestamps when both the source and the destination are on Unix filesystems (including z/OS USS). Preserves the timestamps but not the file permissions, if either one, the source or the destination is on Windows. If the destination is on z/OS MVS, none will be preserved.

-P port

Connects to this Secure Shell port on the remote machine (default: 22).

-q

Uses quiet mode (only fatal errors are shown).

-Q

Does not show progress indicator.

-r

Recurses subdirectories.

-u, --unlink-source

Removes source files after copying (file move).

-v, --verbose

Uses verbose mode (equal to -D 2).

--fips

Performs the checksums using the FIPS cryptographic library.

--force-lower-case

Destination filename will be converted to lowercase characters.

--overwrite=yes | no

Selects whether to overwrite existing destination file(s) (default: yes).

--password= PASSWORD | file://PASSWORDFILE | extprog://PROGRAM

Sets user password that the client will send as a response to password authentication. The PASSWORD can be given directly as an argument to this option (not recommended), or you can enter a path to a file containing the password, or a path to a program or a script that outputs the password.

[Caution]Caution

Supplying the password on the command line is not a secure option. For example, in a multi-user environment, the password given directly on the command line is trivial to recover from the process table. You should set up a more secure way to authenticate. For non-interactive batch jobs, it is more secure to use public-key authentication without a passphrase, or host-based authentication. At a minimum, use a file or a program to supply the password.

--prefix=PREFIX

Adds prefix to filename during the file transfer. The prefix is removed after the file has been successfully transferred.

--progress-display= no | bar | line

Chooses the mode of displaying the progress during a file transfer operation. The default is bar, which shows a progress bar. Option line shows the progress information according to the settings made in the --progress-line-format option.

--progress-line-format=FORMAT_STRING

Chooses what information will be shown on the progress line. Use this option when --progress-display=line. Select the contents for the progress line using the same conversion specification options as with --statistics-format below.

--progress-line-interval=seconds

Defines how often the progress information is updated in line mode. The interval is given in seconds, and the default is 60 seconds.

--statistics=no | yes | simple | bytes

Chooses the style of the statistics to be shown after a file transfer operation. The options mean:

no - no statistics will be shown. This is the default.

yes - detailed statistics will be shown. You can configure the contents with the statistics-format option. The default statistics contents are:

Default settings:                    Render for example this:
"Source: %c:%g\n"                    user@host1#22:/path/to/source/file
"Source parameters: %e\n"            X=TEXT, C=ISO8859-1,D=IBM.1047
"Destination: %C:%G\n"               user@host2#22:/path/to/destination/file
"Destination parameters: %E\n"       NONE
"File size: %s bytes\n"              123456 bytes
"Transferred: %t bytes\n"            123456 bytes
"Rate: %RB/s\n"                      345kiB/s
"Start: %xy-%xt-%xd %xh:%xm:%xs\n"   2008-12-19 13:10:56
"Stop: %Xy-%Xt-%Xd %Xh:%Xm:%Xs\n"    2008-12-19 13:23:30
"Time: %y\n"                         00:12:34

simple - simple one-line statistics will be shown. You can configure the contents with the statistics-format option. The default statistics contents are:

Default settings:                   Render for example this:  
"%f | %TB | %RB/s | TOC: %z\n"      file | 2.8kiB |  72kiB/s | TOC: 00:00:38

bytes - basic statistics reporting the transferred bytes will be shown. You can configure the contents with the statistics-format option. The default statistics contents are:

Default settings:                             
"Transferred %t bytes, file: '%f' -> '%F'\n"  

Render for example this:  
Transferred 12345 bytes, file: 'file1' -> 'file2'
--statistics-format=FORMAT STRING

Chooses the format and the contents of the statistics. Use this option when --statistics=yes|simple|bytes. Select the contents for the statistics using the following conversion specifications:

%c      - source connection: user@host#port or profile
%g      - /path/to/source/file
%f      - source file name
%e      - source parameters (file transfer and dataset parameters)
%C      - destination connection: user@host#port or profile
%G      - /path/to/destination/file
%F      - destination file name
%E      - destination parameters (file transfer and dataset parameters)
%s      - file size in bytes
%S      - file size as "XXyB" (B, kiB, MiB or GiB)
%t      - transfer size in bytes
%T      - transfer size as "XXyB" (B, kiB, MiB or GiB)
%p      - transfer percentage
%q      - transfer rate in bit/s
%Q      - transfer rate as "XXyb/s" (b/s, kib/s, Mib/s, Gib/s)
%r      - transfer rate in bytes/s
%R      - transfer rate as "XXyB/s" (B/s, kiB/s, MiB/s, GiB/s)
%D      - current date in format: Wed Jan 28 2009 17:11:52 +0200
%D(FMT) - current date
%x      - start date in format: Wed Jan 28 2009 17:11:52 +0200
%x(FMT) - start date
%X      - end date in format: Wed Jan 28 2009 17:11:52 +0200
%X(FMT) - end date
%y      - elapsed time
%Y      - time remaining
%z      - ETA or, if transfer has finished, TOC
%Z      - string "ETA" or, if transfer has finished, "TOC"
%Z(eta)(toc) - string "eta" or, if transfer has finished, "toc"

Above, (FMT) indicates the conversion specifications used in the strftime(3) POSIX function. Note that the supported conversion specifications vary slightly between operating systems. See your operating system documentation for details. For POSIX related specifications, see also IEEE Std 1003.1, 2004 Edition.

For example on Linux, the following conversion specifications can be used:

%a The abbreviated weekday name according to the current locale.
%A The full weekday name according to the current locale.
%b The abbreviated month name according to the current locale.
%B The full month name according to the current locale.
%c The preferred date and time representation for the current locale.
%C The century number (year/100) as a 2-digit integer.
%d The day of the month as a decimal number (range 01 to 31).
%D Equivalent to %m/%d/%y. 
%e Like %d, the day of the month as a decimal number, but a leading zero is 
   replaced by a space.
%F Equivalent to %Y-%m-%d (the ISO 8601 date format).
%G The ISO 8601 year with century as a decimal number. The 4-digit year 
   corresponding to the ISO week number (see %V). This has the same format 
   and value as %y, except that if the ISO week number belongs to the 
   previous or next year, that year is used instead.
%g Like %G, but without century, that is, with a 2-digit year (00-99).
%h Equivalent to %b.
%H The hour as a decimal number using a 24-hour clock (range 00-23).
%I The hour as a decimal number using a 12-hour clock (range 01-12).
%j The day of the year as a decimal number (range 001-366).
%k The hour (24-hour clock) as a decimal number (range 0-23);
   single digits are preceded by a blank. (See also %H.)
%l The hour (12-hour clock) as a decimal number (range 1-12);
   single digits are preceded by a blank. (See also %I.)
%m The month as a decimal number (range 01-12).
%M The minute as a decimal number (range 00-59).
%p Either `AM' or `PM' according to the given time value, or the 
   corresponding strings for the current locale. Noon is treated as `pm' 
   and midnight as `am'.
%r The time in a.m. or p.m. notation. In the POSIX locale this is equivalent 
   to %I:%M:%S%p.
%R The time in 24-hour notation (%H:%M). 
   For a version including the seconds, see %T below.
%S The second as a decimal number (range 00-60). (The range is up to 60 to 
   allow for occasional leap seconds.)
%T The time in 24-hour notation (%H:%M:%S).
%u The day of the week as a decimal, range 1 to 7, Monday being 1. See
   also %w.
%U The week number of the current year as a decimal number, range 00-53, 
   starting with the first Sunday as the first day of week 01. 
   See also %V and %W.
%V The ISO 8601:1988 week number of the current year as a decimal number, 
   range 01-53, where week 1 is the first week that has at least 4 days in 
   the current year, and with Monday as the first day of the week. 
   See also %U and %W.
%w The day of the week as a decimal, range 0-6, Sunday being 0. See also %u.
%W The week number of the current year as a decimal number, range 00-53, 
   starting with the first Monday as the first day of week 01.
%x The preferred date representation for the current locale without the time.
%X The preferred time representation for the current locale without the date.
%y The year as a decimal number without a century (range 00-99).
%Y The year as a decimal number including the century.
%z The time-zone as hour offset from GMT. Required to emit RFC 822
   conformant dates (using "%a, %d %b %Y %H:%M:%S %z").
%Z The time zone or name or abbreviation.

Other special characters in the format strings are:

\n - line feed
\t - horizontal tab
\\ - backslash
%% - a literal % sign

--streaming=ext | yes | no | force

Defines streaming for file transfer. On z/OS, ext is the default and enables direct MVS dataset access. All files will use extended streaming, if not otherwise set with "--streaming".

The --streaming=ext option requires also the --checksum=no option, because if checksums are calculated, the file transfer uses staging, which excludes streaming.

Option yes means regular streaming will be used, but files smaller than buffer_size_bytes are not transferred using streaming. Use force with small files.

An alternative way to activate extended streaming is to define SSH_SFTP_STREAMING_MODE=ext and SSH_SFTP_CHECKSUM_MODE=no as environment variables.

--checksum=no | yes | md5 | sha1 | md5-force | sha1-force | checkpoint

Defines whether MD5 or SHA-1 checksums or a separate checkpoint database are used to determine the point in the file where file transfer can be resumed. The default is no, meaning that checksums are not used if "--checksum" is not specified.

With option yes, SHA1 checksums are used in FIPS mode, and MD5 checksums are used otherwise. Note however, that files smaller than buffer_size_bytes are not checked. Use md5-force or sha1-force with small files. Use checkpointing when transferring large files one by one.

-W, --whole-file

Does not try incremental checks. By default (if this option is not given), incremental checks are tried. This option can only be used together with the --checksum option.

--checkpoint=s<seconds>

Time interval between checkpoint updates (default: 10 seconds). This option can only be used when --checksum=checkpoint.

--checkpoint=b<bytes>

Byte interval between checkpoint updates (default: 10 MB). This option can only be used when --checksum=checkpoint.

--src-site=PARAM

Uses the specified site parameters with the source files. See the site command in the section called “Commands”.

--dst-site=PARAM

Uses the specified site parameters with the destination files. See the site command in the section called “Commands”.

--append

Appends data to the end of the destination file.

--binary

Uses binary transfer mode. If the server is SSH Tectia Server for IBM z/OS, the server is requested not to perform ASCII to EBCDIC conversion, and the file is transferred using the Stream format. You can use the --src-site and --dst-site options to change the values.

--sunique

Stores datasets with unique names. In case more than one of the transferred datasets have the same name, this feature adds a sequencial number to the end of the repeated dataset name, for example: DS.version, DS.version1, and DS.version2.

-V, --version

Displays program version and exits.

-h, --help, -?

Displays a short summary of command-line options and exits.

Filename Support

Different operating systems allow different character sets in filenames. On Unix, some of the special characters are allowed in filenames, but on Windows, the following characters are not allowed:

\/ : * ? " < > |

When you use the scpg3 command to copy files with special characters (for example unixfilename*?".txt) from a Unix server to Windows, you need to provide the files with new names that are acceptable on Windows. Enter the commands in the following format:

$ scpg3 user@unixserver:"unixfilename~*~?\".txt" windowsfilename.txt

The general rule is to follow your platform specific syntax when you enter filenames containing special characters as arguments to the scpg3 command.

SSH Tectia fully supports filenames containing only ASCII characters. Filenames containing characters from other character sets are not guaranteed to work.

Using Wildcards

The scpg3 command supports * and ? as wildcards.

The wildcards can be used both on the remote and the local side in the commands. The following example command will copy all text files (*.txt) from all subdirectories of directory dir2 whose names begin with the prefix data- into the current local directory ( . ):

$ scpg3 -r user@server:"dir2/data-*/*.txt" .

Note that on Unix, the characters * and ? can appear also in the filenames. So it is necessary to use escape characters to distinguish the wildcards from the characters belonging to a filename. See more information in the section called “Escaping Special Characters”.

Escaping Special Characters

Some special characters that are used in filenames in different operating system, may have a special meaning in the SSH Tectia commands. Note also that the meaning can be different in various parts of the file transfer system.

In the scpg3 command, the following characters have a special meaning, and they need to be escaped in commands that take filenames as arguments:

* asterisk is a wildcard for any number of any characters

? question mark is a wildcard for any single character

"" quotation marks placed around strings that are to be taken 'as is'

\ backslash is an escape character on Unix

~ tilde is an escape character on Windows.

The escape character tells the scpg3 command to treat the next character "as is" and not to assume any special meaning for it. The escape character is selected according to the operating system of the local machine.

Note that the \ and ~ characters are special characters themselves, and if they are present in the filename, escape characters must be placed in front of them, too. Therefore, if you need to enter a filename containing \ in Unix or ~ in Windows to the scpg3 command, add the relevant escape character to it:

\\ on Unix

~~ on Windows

See the examples below to learn how the escape characters are used in the SSH Tectia scpg3 command, and how to enter filenames with special characters in different operating systems.

Examples of filenames in the scpg3 command:

The following filenames are valid in Unix, but they need escape characters in the commands:

file|name.txt 
file-"name".txt 
file?name.txt 
file*name.txt 
file\name.txt 
file - name.txt 
file~name.txt 

When using the scpg3 command on Unix, in certain cases several escape characters are needed, as they escape one another. Enter the above mentioned filenames in the following formats:

file\|name.txt     or  "file|name.txt" 
file-\"name\".txt 
file\\\?name.txt   or  "file\?name.txt" 
file\\\*name.txt   or  "file\*name.txt" 
file\\\\name.txt   or  "file\\\name.txt" 
file\ -\ name.txt  or  "file - name.txt" 
file~name.txt 

Example commands on Unix:

$ scpg3 user@server:file\\\*name.txt .
$ scpg3 user@server:file\ -\ name.txt .

Environment Variables

scpg3 uses the following environment variables:

SSH_SFTP_CHECKSUM_MODE=no|md5|md5-force|sha1|sha1-force|checkpoint

Defines the default checksum mode for sftpg3 and scpg3 commands. Checksums are used to determine the point in the file where file transfer can be resumed if it gets interrupted.

no - checksums are not used; the file is always transferred from the beginning until EOF. This prevents staging in z/OS.

md5 - MD5 checksums are used

md5-force - MD5 checksums are forced

sha1 - SHA1 checksums are used

sha1-force - SHA1 checksums are forced

checkpoint - a separate checkpoint database is used.

SSH_DEBUG_FMT

This environment variable can be used to specify the format of the debug messages.

For more information, see SSH_DEBUG_FMT in the sftpg3 description.

SSH_SFTP_OVERWRITE=yes|no

If this variable is set to yes (default), the default behavior is to overwrite existing files. If set to no, the default behavior is not to overwrite existing files.

SSH_SFTP_SHOW_BYTE_COUNT=yes|no

If this variable is set to yes, the number of transferred bytes is shown after successful file transfer. Also the names of source and destination files are shown. The default is no.

SSH_SFTP_SMF_TYPE=yes|no

If this variable is set to TYPE119, file transfers create SMF records of type 119.

SSH_SFTP_STATISTICS=yes|no|simple|bytes

If this variable is set to yes (default is no), detailed statistics are shown after a file transfer operation. Option simple outputs a one-line set of information. Option bytes outputs basic statistics reporting the transferred bytes. See the description of command --statistics above for instructions on defining the contents of the statistics to be displayed.

SSH_SFTP_STREAMING_MODE=yes|no|ext

Defines the default streaming mode to be used with sftpg3 and scpg3 commands.

no - streaming is not used.

yes - standard streaming is used.

ext - extended streaming is used.

Files

In addition to the files used by ssh-broker-g3, scpg3 uses the following files:

$HOME/.ssh2/ssh_ftadv_config

This is the user-specific file that contains a list of file transfer profiles, which furnish file transfer attributes to be used when processing local files. For more information, see File Transfer Profiles.

/opt/tectia/etc/ssh_ftadv_config

This is the global (system-wide) file that contains a list of file transfer profiles, which furnish file transfer attributes to be used when processing local files. For more information, see File Transfer Profiles.

Exit Values

scpg3 returns the following values based on the success of the operation:

0     Operation was successful. 
1     Internal error. 
2     Connection aborted by the user.
3     Destination is not a directory, but a directory was specified by the user. 
4     Connecting to the host failed. 
5     Connection lost. 
6     File does not exist. 
7     No permission to access file. 
11    Some non-fatal errors occured during a directory operation.
101   Wrong command-line arguments specified by the user. 

Examples

Copy files from your local system to a remote Unix system:

$ scpg3 localfile user@remotehost:/dst/dir/

Copy files from your local system to a remote Windows system:

$ scpg3 localfile user@remotehost:/C:/dst/dir/

Copy files from a remote system to your local disk:

$ scpg3 user@remotehost:/src/dir/srcfile /dst/dir/dstfile

Copy files from one remote system to another using connection profiles defined in the ssh-broker-config.xml file:

$ scpg3 profile1:/src/dir/srcfile profile2:/dst/dir/dstfile