How to Perform Unattended Server to Server File Transfers: Unix to Unix

Use SSH (Secure Shell) keys and SCP (Secure Copy) to perform secure, unattended, server to server file transfers between Unix servers.

filetransfer

Setting Up ID Accounts

ID Types and Audits

The ID types used in secure, unattended file transfers are complex in order to create an audit trail. Holding IDs and File Transfer IDs cannot be traced back to a single user so each type of ID is given limited functions.

  • (Regular) User ID: Is traceable to a single user. Can log in, run processes, send files. Do not use for unattended file transfers.
  • Holding ID: Is not traceable to a single user. Can run processes. Cannot log in or send files.
  • File Transfer ID: Is not traceable to a single user. Can send files. Cannot log in or run processes.

Setting Up SSH Keys for File Transfer ID

Secure Shell (SSH) is a program used to provide encryption during file transfers using SCP or SFTP (Secure File Transfer Protocol). SSH can use public/private key pairs (with or without a password) to authenticate file transfers.

How does it work? First, you set up a public/private key pair. See the following procedure for steps. Then, when you are ready to transfer files, the File Transfer ID on the receiving server sends the public key to the source server. Using the public key, the source server sends an encrypted challenge back to the receiving server. The receiving server uses the private key to decrypt the challenge and respond to it.

Note: If one or more of the servers does not use OpenSSH, this won’t work. For more information, see the Open SSH site.

Note: This procedure works only when using the default key file name .ssh/id_dsa. If necessary, change the file name.

  1. Run the procedure using the Holding ID for which the keys are being generated. On the source server, as the Holding ID, generate the key pair. Enter:
    • For Solaris: /opt/common/bin/ssh-keygen -t dsa
    • For Linux: /usr/bin/ssh-keygen -t dsa

    This generates a key pair in ~/.ssh. You will be prompted for a password. Click Enter to indicate you don’t want to use a password. (Don’t set up a password because in an unattended file transfer, no one will be available to enter the password. However, because the private key is not password-protected, be sure to take extra care to protect it. For example, don’t copy the private key to other directories.)

  2. Using the password for the File Transfer ID, copy the public key to the File Transfer ID account on the receiving server. Enter:cd ~/.ssh
    scp id_dsa.pub filetransferid@receiving server:.ssh/authorized_keys
  3. Note: This overwrites any keys already in the authorized keys file. If you need to transfer files from more than one Holding ID to this File Transfer ID, use your own UserID to transfer the keys and concatenate them together.

    You can now scp from the source server to the receiving server without answering a password request.

Transferring Files

  1. Use the Holding ID’s cron job process to transfer the file from the source server to the receiving server.
  2. You can use this sample script:

    30 1 * * * /opt/common/bin/scp source_file filetransferid@receivingserver:/path/receiving_file > /path/transfer_error_log 2>&1

    This cron entry in the Holding ID account will attempt to copy the source_file every night at 1:30 AM and record any errors that occur in the file /path/transfer_error_log. (This error file is on the source server.) You can add to the sample script if you need to record the transfer in a log file, rename the file with the date once the transfer is done, archive and remove old files, etc.

Advertisements

Usage of Crontab on unix

Using Cron

To use cron for tasks meant to run only for your user profile, add entries to your own user’s crontab file. Start the crontab editor from a terminal window:

crontab -e

Edit the crontab using the format described in the next sections. Save your changes. (Exiting without saving will leave your crontab unchanged.)

Note that a great source of information about the format can be found at:

man 5 crontab

Commands that normally run with administrative privileges (i.e. they are generally run using sudo) should be added to the root user’s crontab (instead of the user’s crontab):

 sudo crontab -e

Crontab Sections

Each of the sections is separated by a space, with the final section having one or more spaces in it. No spaces are allowed within Sections 1-5, only between them. Sections 1-5 are used to indicate when and how often you want the task to be executed. This is how a cron job is laid out:

minute (0-59), hour (0-23, 0 = midnight), day (1-31), month (1-12), weekday (0-6, 0 = Sunday), command

01 04 1 1 1 /usr/bin/somedirectory/somecommand

The above example will run /usr/bin/somedirectory/somecommand at 4:01am on January 1st plus every Monday in January. An asterisk (*) can be used so that every instance (every hour, every weekday, every month, etc.) of a time period is used. Code:

01 04 * * * /usr/bin/somedirectory/somecommand

The above example will run /usr/bin/somedirectory/somecommand at 4:01am on every day of every month.

Comma-separated values can be used to run more than one instance of a particular command within a time period. Dash-separated values can be used to run a command continuously. Code:

01,31 04,05 1-15 1,6 * /usr/bin/somedirectory/somecommand

The above example will run /usr/bin/somedirectory/somecommand at 01 and 31 past the hours of 4:00am and 5:00am on the 1st through the 15th of every January and June.

The “/usr/bin/somedirectory/somecommand” text in the above examples indicates the task which will be run at the specified times. It is recommended that you use the full path to the desired commands as shown in the above examples. Enter which somecommand in the terminal to find the full path to somecommand. The crontab will begin running as soon as it is properly edited and saved.

You may want to run a script some number of times per time unit. For example if you want to run it every 10 minutes use the following crontab entry (runs on minutes divisible by 10: 0, 10, 20, 30, etc.)

*/10 * * * * /usr/bin/somedirectory/somecommand

which is also equivalent to the more cumbersome

0,10,20,30,40,50 * * * * /usr/bin/somedirectory/somecommand

Crontab Options

  • The -l option causes the current crontab to be displayed on standard output.
  • The -r option causes the current crontab to be removed.
  • The -e option is used to edit the current crontab using the editor specified by the EDITOR environment variable.

After you exit from the editor, the modified crontab will be checked for accuracy and, if there are no errors, installed automatically. The file is stored in /var/spool/cron/crontabs but should only be edited via the crontab command.

Enable User Level Cron

If the /etc/cron.allow file exists, then users must be listed in it in order to be allowed to run the crontab command. If the /etc/cron.allow file does not exist but the /etc/cron.deny file does, then users must not be listed in the /etc/cron.deny file in order to run crontab.

In the case where neither file exists, the default on current Ubuntu (and Debian, but not some other Linux and UNIX systems) is to allow all users to run jobs with crontab.

No cron.allow or cron.deny files exist in a standard Ubuntu install, so all users should have cron available by default, until one of those files is created. If a blank cron.deny file has been created, that will change to the standard behavior users of other operating systems might expect: cron only available to root or users in cron.allow.

Note, userids on your system which do not appear in /etc/shadow will NOT have operational crontabs, if you desire to enter a user in /etc/passwd, but NOT /etc/shadow that user’s crontab will never run. Place an entry in /etc/shadow for the user with a * for the password crypt,ie:

joeuser:*:15169::::::

Further Considerations

Crontab commands are generally stored in the crontab file belonging to your user account (and executed with your user’s level of permissions). If you want to regularly run a command requiring administrative permissions, edit the root crontab file:

sudo crontab -e

Depending on the commands being run, you may need to expand the root users PATH variable by putting the following line at the top of their crontab file:

PATH=/usr/sbin:/usr/bin:/sbin:/bin

It is sensible to test that your cron jobs work as intended. One method for doing this is to set up the job to run a couple of minutes in the future and then check the results before finalising the timing. You may also find it useful to put the commands into script files that log their success or failure, for example:

echo "Nightly Backup Successful: $(date)" >> /tmp/mybackup.log

For more information, see the man pages for cron and crontab (man is detailed on the BasicCommands page). If your machine is regularly switched off, you may also be interested in at and anacron, which provide other approaches to scheduled tasks. For example, anacron offers simple system-wide directories for running commands hourly, daily, weekly, and monthly. Scripts to be executed in said times can be placed in/etc/cron.hourly//etc/cron.daily//etc/cron.weekly/, and /etc/cron.monthly/. All scripts in each directory are run as root, and a specific order to running the scripts can be specified by prefixing the scripts’ filenames with numbers (see the man page for run-parts for more details). Although the directories contain periods in their names, run-parts will not accept a file name containing a period and will fail silently when encountering them (bug #38022). Either rename the file or use a symlink (without a period) to it instead (see, for example, python + cron without login? and Problems with Hourly Cron Job).

Troubleshooting and Common Problems

Edits to a user’s crontab and jobs that are run on their behalf are all logged by default to /var/log/syslog and that’s the first place to check if things are not running as you expect.

When adding a new entry to a blank crontab, forgetting to add a newline at the end is a common source for the job not running. If the last line in the crontab does not end with a newline, no errors will be reported at edit or runtime, but that line will never run. See man crontab for more information. This has already been suggested as a bug.

If a user was not allowed to execute jobs when their crontab was last edited, just adding them to the allow list won’t do anything. The user needs to re-edit their crontab after being added to cron.allow before their jobs will run.

When creating a crontab for the root user, the user name must be specified as a parameter after the date/time parameters. Accidentally including the user name that way in a user-specific crontab will result in trying to run the user’s name as a command, rather than what was expected.

Entries in cron may not run with the same environment, in particular the PATH, as you expect them to. Try using full paths to files and programs if they’re not being located as you expect.

The “%” character is used as newline delimiter in cron commands. If you need to pass that character into a script, you need to escape it as “\%”.

If you’re having trouble running a GUI application using cron, see the GUI Applications section below.

Advanced Crontab

The Crontabs discussed above are user crontabs. Each of the above crontabs is associated with a user, even the system crontab which is associated with the root user. There are two other types of crontab.

Firstly, as mentioned above anacron uses the run-parts command and /etc/cron.hourly/etc/cron.weekly, and /etc/cron.monthlydirectories. However anacron itself is invoked from the /etc/crontab file. This file could be used for other cron commands, but probably shouldn’t be. Here’s an example line from a ficticious /etc/crontab:

00 01 * * * rusty /home/rusty/rusty-list-files.sh

This would run Rusty’s command script as user rusty from his home directory. However, it is not usual to add commands to this file. While an experienced user should know about it, it is not recommended that you add anything to /etc/crontab. Apart from anything else, this could cause problem if the /etc/crontab file is affected by updates! Rusty could lose his command.

The second type of crontab is to be found in /etc/cron.d. Within the directory are small named crontabs. The directory is often used by packages, and the small crontabs allows a user to be associated with the commands in them.

Instead of adding a line to /etc/crontab which Rusty knows is not a good idea, Rusty might well add a file to /etc/cron.d with the name rusty, containing his cron line above. This would not be affected by updates but is a well known location.

When would you use these alternate crontab locations? Well, on a single user machine or a shared machine such as a school or college server, auser crontab would be the way to go. But in a large IT department, where several people might look after a server, then /etc/cron.d is probably the best place to install crontabs – it’s a central point and saves searching for them!

You may not need to look at /etc/crontab or /etc/cron.d, let alone edit them by hand. But an experienced user should perhaps know about them and that the packages that he/she installs may use these locations for their crontabs.

Special strings

Cron also offers some special strings:

  • string meaning
    @reboot Run once, at startup.
    @yearly Run once a year, “0 0 1 1 *”.
    @annually (same as @yearly)
    @monthly Run once a month, “0 0 1 * *”.
    @weekly Run once a week, “0 0 * * 0”.
    @daily Run once a day, “0 0 * * *”.
    @midnight (same as @daily)
    @hourly Run once an hour, “0 * * * *”.

Usage: “@reboot /path/to/execuable1” will execute /path/to/executable1 when the system starts. See “man 5 crontab” for more info.