How to Perform Unattended Server to Server File Transfers: Unix to Unix

Use SSH (Secure Shell) keys and SCP (Secure Copy) to perform secure, unattended, server to server file transfers between Unix servers.

filetransfer

Setting Up ID Accounts

ID Types and Audits

The ID types used in secure, unattended file transfers are complex in order to create an audit trail. Holding IDs and File Transfer IDs cannot be traced back to a single user so each type of ID is given limited functions.

  • (Regular) User ID: Is traceable to a single user. Can log in, run processes, send files. Do not use for unattended file transfers.
  • Holding ID: Is not traceable to a single user. Can run processes. Cannot log in or send files.
  • File Transfer ID: Is not traceable to a single user. Can send files. Cannot log in or run processes.

Setting Up SSH Keys for File Transfer ID

Secure Shell (SSH) is a program used to provide encryption during file transfers using SCP or SFTP (Secure File Transfer Protocol). SSH can use public/private key pairs (with or without a password) to authenticate file transfers.

How does it work? First, you set up a public/private key pair. See the following procedure for steps. Then, when you are ready to transfer files, the File Transfer ID on the receiving server sends the public key to the source server. Using the public key, the source server sends an encrypted challenge back to the receiving server. The receiving server uses the private key to decrypt the challenge and respond to it.

Note: If one or more of the servers does not use OpenSSH, this won’t work. For more information, see the Open SSH site.

Note: This procedure works only when using the default key file name .ssh/id_dsa. If necessary, change the file name.

  1. Run the procedure using the Holding ID for which the keys are being generated. On the source server, as the Holding ID, generate the key pair. Enter:
    • For Solaris: /opt/common/bin/ssh-keygen -t dsa
    • For Linux: /usr/bin/ssh-keygen -t dsa

    This generates a key pair in ~/.ssh. You will be prompted for a password. Click Enter to indicate you don’t want to use a password. (Don’t set up a password because in an unattended file transfer, no one will be available to enter the password. However, because the private key is not password-protected, be sure to take extra care to protect it. For example, don’t copy the private key to other directories.)

  2. Using the password for the File Transfer ID, copy the public key to the File Transfer ID account on the receiving server. Enter:cd ~/.ssh
    scp id_dsa.pub filetransferid@receiving server:.ssh/authorized_keys
  3. Note: This overwrites any keys already in the authorized keys file. If you need to transfer files from more than one Holding ID to this File Transfer ID, use your own UserID to transfer the keys and concatenate them together.

    You can now scp from the source server to the receiving server without answering a password request.

Transferring Files

  1. Use the Holding ID’s cron job process to transfer the file from the source server to the receiving server.
  2. You can use this sample script:

    30 1 * * * /opt/common/bin/scp source_file filetransferid@receivingserver:/path/receiving_file > /path/transfer_error_log 2>&1

    This cron entry in the Holding ID account will attempt to copy the source_file every night at 1:30 AM and record any errors that occur in the file /path/transfer_error_log. (This error file is on the source server.) You can add to the sample script if you need to record the transfer in a log file, rename the file with the date once the transfer is done, archive and remove old files, etc.

Advertisements

Usage of Crontab on unix

Using Cron

To use cron for tasks meant to run only for your user profile, add entries to your own user’s crontab file. Start the crontab editor from a terminal window:

crontab -e

Edit the crontab using the format described in the next sections. Save your changes. (Exiting without saving will leave your crontab unchanged.)

Note that a great source of information about the format can be found at:

man 5 crontab

Commands that normally run with administrative privileges (i.e. they are generally run using sudo) should be added to the root user’s crontab (instead of the user’s crontab):

 sudo crontab -e

Crontab Sections

Each of the sections is separated by a space, with the final section having one or more spaces in it. No spaces are allowed within Sections 1-5, only between them. Sections 1-5 are used to indicate when and how often you want the task to be executed. This is how a cron job is laid out:

minute (0-59), hour (0-23, 0 = midnight), day (1-31), month (1-12), weekday (0-6, 0 = Sunday), command

01 04 1 1 1 /usr/bin/somedirectory/somecommand

The above example will run /usr/bin/somedirectory/somecommand at 4:01am on January 1st plus every Monday in January. An asterisk (*) can be used so that every instance (every hour, every weekday, every month, etc.) of a time period is used. Code:

01 04 * * * /usr/bin/somedirectory/somecommand

The above example will run /usr/bin/somedirectory/somecommand at 4:01am on every day of every month.

Comma-separated values can be used to run more than one instance of a particular command within a time period. Dash-separated values can be used to run a command continuously. Code:

01,31 04,05 1-15 1,6 * /usr/bin/somedirectory/somecommand

The above example will run /usr/bin/somedirectory/somecommand at 01 and 31 past the hours of 4:00am and 5:00am on the 1st through the 15th of every January and June.

The “/usr/bin/somedirectory/somecommand” text in the above examples indicates the task which will be run at the specified times. It is recommended that you use the full path to the desired commands as shown in the above examples. Enter which somecommand in the terminal to find the full path to somecommand. The crontab will begin running as soon as it is properly edited and saved.

You may want to run a script some number of times per time unit. For example if you want to run it every 10 minutes use the following crontab entry (runs on minutes divisible by 10: 0, 10, 20, 30, etc.)

*/10 * * * * /usr/bin/somedirectory/somecommand

which is also equivalent to the more cumbersome

0,10,20,30,40,50 * * * * /usr/bin/somedirectory/somecommand

Crontab Options

  • The -l option causes the current crontab to be displayed on standard output.
  • The -r option causes the current crontab to be removed.
  • The -e option is used to edit the current crontab using the editor specified by the EDITOR environment variable.

After you exit from the editor, the modified crontab will be checked for accuracy and, if there are no errors, installed automatically. The file is stored in /var/spool/cron/crontabs but should only be edited via the crontab command.

Enable User Level Cron

If the /etc/cron.allow file exists, then users must be listed in it in order to be allowed to run the crontab command. If the /etc/cron.allow file does not exist but the /etc/cron.deny file does, then users must not be listed in the /etc/cron.deny file in order to run crontab.

In the case where neither file exists, the default on current Ubuntu (and Debian, but not some other Linux and UNIX systems) is to allow all users to run jobs with crontab.

No cron.allow or cron.deny files exist in a standard Ubuntu install, so all users should have cron available by default, until one of those files is created. If a blank cron.deny file has been created, that will change to the standard behavior users of other operating systems might expect: cron only available to root or users in cron.allow.

Note, userids on your system which do not appear in /etc/shadow will NOT have operational crontabs, if you desire to enter a user in /etc/passwd, but NOT /etc/shadow that user’s crontab will never run. Place an entry in /etc/shadow for the user with a * for the password crypt,ie:

joeuser:*:15169::::::

Further Considerations

Crontab commands are generally stored in the crontab file belonging to your user account (and executed with your user’s level of permissions). If you want to regularly run a command requiring administrative permissions, edit the root crontab file:

sudo crontab -e

Depending on the commands being run, you may need to expand the root users PATH variable by putting the following line at the top of their crontab file:

PATH=/usr/sbin:/usr/bin:/sbin:/bin

It is sensible to test that your cron jobs work as intended. One method for doing this is to set up the job to run a couple of minutes in the future and then check the results before finalising the timing. You may also find it useful to put the commands into script files that log their success or failure, for example:

echo "Nightly Backup Successful: $(date)" >> /tmp/mybackup.log

For more information, see the man pages for cron and crontab (man is detailed on the BasicCommands page). If your machine is regularly switched off, you may also be interested in at and anacron, which provide other approaches to scheduled tasks. For example, anacron offers simple system-wide directories for running commands hourly, daily, weekly, and monthly. Scripts to be executed in said times can be placed in/etc/cron.hourly//etc/cron.daily//etc/cron.weekly/, and /etc/cron.monthly/. All scripts in each directory are run as root, and a specific order to running the scripts can be specified by prefixing the scripts’ filenames with numbers (see the man page for run-parts for more details). Although the directories contain periods in their names, run-parts will not accept a file name containing a period and will fail silently when encountering them (bug #38022). Either rename the file or use a symlink (without a period) to it instead (see, for example, python + cron without login? and Problems with Hourly Cron Job).

Troubleshooting and Common Problems

Edits to a user’s crontab and jobs that are run on their behalf are all logged by default to /var/log/syslog and that’s the first place to check if things are not running as you expect.

When adding a new entry to a blank crontab, forgetting to add a newline at the end is a common source for the job not running. If the last line in the crontab does not end with a newline, no errors will be reported at edit or runtime, but that line will never run. See man crontab for more information. This has already been suggested as a bug.

If a user was not allowed to execute jobs when their crontab was last edited, just adding them to the allow list won’t do anything. The user needs to re-edit their crontab after being added to cron.allow before their jobs will run.

When creating a crontab for the root user, the user name must be specified as a parameter after the date/time parameters. Accidentally including the user name that way in a user-specific crontab will result in trying to run the user’s name as a command, rather than what was expected.

Entries in cron may not run with the same environment, in particular the PATH, as you expect them to. Try using full paths to files and programs if they’re not being located as you expect.

The “%” character is used as newline delimiter in cron commands. If you need to pass that character into a script, you need to escape it as “\%”.

If you’re having trouble running a GUI application using cron, see the GUI Applications section below.

Advanced Crontab

The Crontabs discussed above are user crontabs. Each of the above crontabs is associated with a user, even the system crontab which is associated with the root user. There are two other types of crontab.

Firstly, as mentioned above anacron uses the run-parts command and /etc/cron.hourly/etc/cron.weekly, and /etc/cron.monthlydirectories. However anacron itself is invoked from the /etc/crontab file. This file could be used for other cron commands, but probably shouldn’t be. Here’s an example line from a ficticious /etc/crontab:

00 01 * * * rusty /home/rusty/rusty-list-files.sh

This would run Rusty’s command script as user rusty from his home directory. However, it is not usual to add commands to this file. While an experienced user should know about it, it is not recommended that you add anything to /etc/crontab. Apart from anything else, this could cause problem if the /etc/crontab file is affected by updates! Rusty could lose his command.

The second type of crontab is to be found in /etc/cron.d. Within the directory are small named crontabs. The directory is often used by packages, and the small crontabs allows a user to be associated with the commands in them.

Instead of adding a line to /etc/crontab which Rusty knows is not a good idea, Rusty might well add a file to /etc/cron.d with the name rusty, containing his cron line above. This would not be affected by updates but is a well known location.

When would you use these alternate crontab locations? Well, on a single user machine or a shared machine such as a school or college server, auser crontab would be the way to go. But in a large IT department, where several people might look after a server, then /etc/cron.d is probably the best place to install crontabs – it’s a central point and saves searching for them!

You may not need to look at /etc/crontab or /etc/cron.d, let alone edit them by hand. But an experienced user should perhaps know about them and that the packages that he/she installs may use these locations for their crontabs.

Special strings

Cron also offers some special strings:

  • string meaning
    @reboot Run once, at startup.
    @yearly Run once a year, “0 0 1 1 *”.
    @annually (same as @yearly)
    @monthly Run once a month, “0 0 1 * *”.
    @weekly Run once a week, “0 0 * * 0”.
    @daily Run once a day, “0 0 * * *”.
    @midnight (same as @daily)
    @hourly Run once an hour, “0 * * * *”.

Usage: “@reboot /path/to/execuable1” will execute /path/to/executable1 when the system starts. See “man 5 crontab” for more info.

20 Linux System Monitoring Tools Every SysAdmin Should Know

Need to monitor Linux server performance? Try these built-in commands and a few add-on tools. Most Linux distributions are equipped with tons of monitoring. These tools provide metrics which can be used to get information about system activities. You can use these tools to find the possible causes of a performance problem. The commands discussed below are some of the most basic commands when it comes to system analysis and debugging server issues such as:

  1. Finding out bottlenecks.
  2. Disk (storage) bottlenecks.
  3. CPU and memory bottlenecks.
  4. Network bottlenecks.

#1: top – Process Activity Command

The top program provides a dynamic real-time view of a running system i.e. actual process activity. By default, it displays the most CPU-intensive tasks running on the server and updates the list every five seconds.

Fig.01: Linux top command

Fig.01: Linux top command

Commonly Used Hot Keys

The top command provides several useful hot keys:

Hot Key Usage
t Displays summary information off and on.
m Displays memory information off and on.
A Sorts the display by top consumers of various system resources. Useful for quick identification of performance-hungry tasks on a system.
f Enters an interactive configuration screen for top. Helpful for setting up top for a specific task.
o Enables you to interactively select the ordering within top.
r Issues renice command.
k Issues kill command.
z Turn on or off color/mono

=> Related: How do I Find Out Linux CPU Utilization?

#2: vmstat – System Activity, Hardware and System Information

The command vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity.
# vmstat 3
Sample Outputs:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 2540988 522188 5130400    0    0     2    32    4    2  4  1 96  0  0
 1  0      0 2540988 522188 5130400    0    0     0   720 1199  665  1  0 99  0  0
 0  0      0 2540956 522188 5130400    0    0     0     0 1151 1569  4  1 95  0  0
 0  0      0 2540956 522188 5130500    0    0     0     6 1117  439  1  0 99  0  0
 0  0      0 2540940 522188 5130512    0    0     0   536 1189  932  1  0 98  0  0
 0  0      0 2538444 522188 5130588    0    0     0     0 1187 1417  4  1 96  0  0
 0  0      0 2490060 522188 5130640    0    0     0    18 1253 1123  5  1 94  0  0

Display Memory Utilization Slabinfo

# vmstat -m

Get Information About Active / Inactive Memory Pages

# vmstat -a
=> Related: How do I find out Linux Resource utilization to detect system bottlenecks?

#3: w – Find Out Who Is Logged on And What They Are Doing

w command displays information about the users currently on the machine, and their processes.
# w username
# w vivek

Sample Outputs:

 17:58:47 up 5 days, 20:28,  2 users,  load average: 0.36, 0.26, 0.24
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    10.1.3.145       14:55    5.00s  0.04s  0.02s vim /etc/resolv.conf
root     pts/1    10.1.3.145       17:43    0.00s  0.03s  0.00s w

#4: uptime – Tell How Long The System Has Been Running

The uptime command can be used to see how long the server has been running. The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.
# uptime
Output:

 18:02:41 up 41 days, 23:42,  1 user,  load average: 0.00, 0.00, 0.00

1 can be considered as optimal load value. The load can change from system to system. For a single CPU system 1 – 3 and SMP systems 6-10 load value might be acceptable.

#5: ps – Displays The Processes

ps command will report a snapshot of the current processes. To select all processes use the -A or -e option:
# ps -A
Sample Outputs:

  PID TTY          TIME CMD
    1 ?        00:00:02 init
    2 ?        00:00:02 migration/0
    3 ?        00:00:01 ksoftirqd/0
    4 ?        00:00:00 watchdog/0
    5 ?        00:00:00 migration/1
    6 ?        00:00:15 ksoftirqd/1
....
.....
 4881 ?        00:53:28 java
 4885 tty1     00:00:00 mingetty
 4886 tty2     00:00:00 mingetty
 4887 tty3     00:00:00 mingetty
 4888 tty4     00:00:00 mingetty
 4891 tty5     00:00:00 mingetty
 4892 tty6     00:00:00 mingetty
 4893 ttyS1    00:00:00 agetty
12853 ?        00:00:00 cifsoplockd
12854 ?        00:00:00 cifsdnotifyd
14231 ?        00:10:34 lighttpd
14232 ?        00:00:00 php-cgi
54981 pts/0    00:00:00 vim
55465 ?        00:00:00 php-cgi
55546 ?        00:00:00 bind9-snmp-stat
55704 pts/1    00:00:00 ps

ps is just like top but provides more information.

Show Long Format Output

# ps -Al
To turn on extra full mode (it will show command line arguments passed to process):
# ps -AlF

To See Threads ( LWP and NLWP)

# ps -AlFH

To See Threads After Processes

# ps -AlLm

Print All Process On The Server

# ps ax
# ps axu

Print A Process Tree

# ps -ejH
# ps axjf
# pstree

Print Security Information

# ps -eo euser,ruser,suser,fuser,f,comm,label
# ps axZ
# ps -eM

See Every Process Running As User Vivek

# ps -U vivek -u vivek u

Set Output In a User-Defined Format

# ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
# ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
# ps -eopid,tt,user,fname,tmout,f,wchan

Display Only The Process IDs of Lighttpd

# ps -C lighttpd -o pid=
OR
# pgrep lighttpd
OR
# pgrep -u vivek php-cgi

Display The Name of PID 55977

# ps -p 55977 -o comm=

Find Out The Top 10 Memory Consuming Process

# ps -auxf | sort -nr -k 4 | head -10

Find Out top 10 CPU Consuming Process

# ps -auxf | sort -nr -k 3 | head -10

#6: free – Memory Usage

The command free displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel.
# free
Sample Output:

            total       used       free     shared    buffers     cached
Mem:      12302896    9739664    2563232          0     523124    5154740
-/+ buffers/cache:    4061800    8241096
Swap:      1052248          0    1052248

=> Related: :

  1. Linux Find Out Virtual Memory PAGESIZE
  2. Linux Limit CPU Usage Per Process
  3. How much RAM does my Ubuntu / Fedora Linux desktop PC have?

#7: iostat – Average CPU Load, Disk Activity

The command iostat report Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems (NFS).
# iostat
Sample Outputs:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 	06/26/2009
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.50    0.09    0.51    0.03    0.00   95.86
Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              22.04        31.88       512.03   16193351  260102868
sda1              0.00         0.00         0.00       2166        180
sda2             22.04        31.87       512.03   16189010  260102688
sda3              0.00         0.00         0.00       1615          0

=> Related: : Linux Track NFS Directory / Disk I/O Stats

#8: sar – Collect and Report System Activity

The sar command is used to collect, report, and save system activity information. To see network counter, enter:
# sar -n DEV | more
To display the network counters from the 24th:
# sar -n DEV -f /var/log/sa/sa24 | more
You can also display real time usage using sar:
# sar 4 5
Sample Outputs:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 		06/26/2009
06:45:12 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
06:45:16 PM       all      2.00      0.00      0.22      0.00      0.00     97.78
06:45:20 PM       all      2.07      0.00      0.38      0.03      0.00     97.52
06:45:24 PM       all      0.94      0.00      0.28      0.00      0.00     98.78
06:45:28 PM       all      1.56      0.00      0.22      0.00      0.00     98.22
06:45:32 PM       all      3.53      0.00      0.25      0.03      0.00     96.19
Average:          all      2.02      0.00      0.27      0.01      0.00     97.70

=> Related: : How to collect Linux system utilization data into a file

#9: mpstat – Multiprocessor Usage

The mpstat command displays activities for each available processor, processor 0 being the first one. mpstat -P ALL to display average CPU utilization per processor:
# mpstat -P ALL
Sample Output:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in)	 	06/26/2009
06:48:11 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
06:48:11 PM  all    3.50    0.09    0.34    0.03    0.01    0.17    0.00   95.86   1218.04
06:48:11 PM    0    3.44    0.08    0.31    0.02    0.00    0.12    0.00   96.04   1000.31
06:48:11 PM    1    3.10    0.08    0.32    0.09    0.02    0.11    0.00   96.28     34.93
06:48:11 PM    2    4.16    0.11    0.36    0.02    0.00    0.11    0.00   95.25      0.00
06:48:11 PM    3    3.77    0.11    0.38    0.03    0.01    0.24    0.00   95.46     44.80
06:48:11 PM    4    2.96    0.07    0.29    0.04    0.02    0.10    0.00   96.52     25.91
06:48:11 PM    5    3.26    0.08    0.28    0.03    0.01    0.10    0.00   96.23     14.98
06:48:11 PM    6    4.00    0.10    0.34    0.01    0.00    0.13    0.00   95.42      3.75
06:48:11 PM    7    3.30    0.11    0.39    0.03    0.01    0.46    0.00   95.69     76.89

=> Related: : Linux display each multiple SMP CPU processors utilization individually.

#10: pmap – Process Memory Usage

The command pmap report memory map of a process. Use this command to find out causes of memory bottlenecks.
# pmap -d PID
To display process memory information for pid # 47394, enter:
# pmap -d 47394
Sample Outputs:

47394:   /usr/bin/php-cgi
Address           Kbytes Mode  Offset           Device    Mapping
0000000000400000    2584 r-x-- 0000000000000000 008:00002 php-cgi
0000000000886000     140 rw--- 0000000000286000 008:00002 php-cgi
00000000008a9000      52 rw--- 00000000008a9000 000:00000   [ anon ]
0000000000aa8000      76 rw--- 00000000002a8000 008:00002 php-cgi
000000000f678000    1980 rw--- 000000000f678000 000:00000   [ anon ]
000000314a600000     112 r-x-- 0000000000000000 008:00002 ld-2.5.so
000000314a81b000       4 r---- 000000000001b000 008:00002 ld-2.5.so
000000314a81c000       4 rw--- 000000000001c000 008:00002 ld-2.5.so
000000314aa00000    1328 r-x-- 0000000000000000 008:00002 libc-2.5.so
000000314ab4c000    2048 ----- 000000000014c000 008:00002 libc-2.5.so
.....
......
..
00002af8d48fd000       4 rw--- 0000000000006000 008:00002 xsl.so
00002af8d490c000      40 r-x-- 0000000000000000 008:00002 libnss_files-2.5.so
00002af8d4916000    2044 ----- 000000000000a000 008:00002 libnss_files-2.5.so
00002af8d4b15000       4 r---- 0000000000009000 008:00002 libnss_files-2.5.so
00002af8d4b16000       4 rw--- 000000000000a000 008:00002 libnss_files-2.5.so
00002af8d4b17000  768000 rw-s- 0000000000000000 000:00009 zero (deleted)
00007fffc95fe000      84 rw--- 00007ffffffea000 000:00000   [ stack ]
ffffffffff600000    8192 ----- 0000000000000000 000:00000   [ anon ]
mapped: 933712K    writeable/private: 4304K    shared: 768000K

The last line is very important:

  • mapped: 933712K total amount of memory mapped to files
  • writeable/private: 4304K the amount of private address space
  • shared: 768000K the amount of address space this process is sharing with others

=> Related: : Linux find the memory used by a program / process using pmap command

#11 and #12: netstat and ss – Network Statistics

The command netstat displays network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. ss command is used to dump socket statistics. It allows showing information similar to netstat. See the following resources about ss and netstat commands:

#13: iptraf – Real-time Network Statistics

The iptraf command is interactive colorful IP LAN monitor. It is an ncurses-based IP LAN monitor that generates various network statistics including TCP info, UDP counts, ICMP and OSPF information, Ethernet load info, node stats, IP checksum errors, and others. It can provide the following info in easy to read format:

  • Network traffic statistics by TCP connection
  • IP traffic statistics by network interface
  • Network traffic statistics by protocol
  • Network traffic statistics by TCP/UDP port and by packet size
  • Network traffic statistics by Layer2 address

Fig.02: General interface statistics: IP traffic statistics by network interface

Fig.02: General interface statistics: IP traffic statistics by network interface

Fig.03 Network traffic statistics by TCP connection

Fig.03 Network traffic statistics by TCP connection

#14: tcpdump – Detailed Network Traffic Analysis

The tcpdump is simple command that dump traffic on a network. However, you need good understanding of TCP/IP protocol to utilize this tool. For.e.g to display traffic info about DNS, enter:
# tcpdump -i eth1 'udp port 53'
To display all IPv4 HTTP packets to and from port 80, i.e. print only packets that contain data, not, for example, SYN and FIN packets and ACK-only packets, enter:
# tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
To display all FTP session to 202.54.1.5, enter:
# tcpdump -i eth1 'dst 202.54.1.5 and (port 21 or 20'
To display all HTTP session to 192.168.1.5:
# tcpdump -ni eth0 'dst 192.168.1.5 and tcp and port http'
Use wireshark to view detailed information about files, enter:
# tcpdump -n -i eth1 -s 0 -w output.txt src or dst port 80

#15: strace – System Calls

Trace system calls and signals. This is useful for debugging webserver and other server problems. See how to use to trace the process and see What it is doing.

#16: /Proc file system – Various Kernel Statistics

/proc file system provides detailed information about various hardware devices and other Linux kernel information. See Linux kernel /proc documentations for further details. Common /proc examples:
# cat /proc/cpuinfo
# cat /proc/meminfo
# cat /proc/zoneinfo
# cat /proc/mounts

17#: Nagios – Server And Network Monitoring

Nagios is a popular open source computer system and network monitoring application software. You can easily monitor all your hosts, network equipment and services. It can send alert when things go wrong and again when they get better. FAN is “Fully Automated Nagios”. FAN goals are to provide a Nagios installation including most tools provided by the Nagios Community. FAN provides a CDRom image in the standard ISO format, making it easy to easilly install a Nagios server. Added to this, a wide bunch of tools are including to the distribution, in order to improve the user experience around Nagios.

18#: Cacti – Web-based Monitoring Tool

Cacti is a complete network graphing solution designed to harness the power of RRDTool’s data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices. It can provide data about network, CPU, memory, logged in users, Apache, DNS servers and much more. See how to install and configure Cacti network graphing tool under CentOS / RHEL.

#19: KDE System Guard – Real-time Systems Reporting and Graphing

KSysguard is a network enabled task and system monitor application for KDE desktop. This tool can be run over ssh session. It provides lots of features such as a client/server architecture that enables monitoring of local and remote hosts. The graphical front end uses so-called sensors to retrieve the information it displays. A sensor can return simple values or more complex information like tables. For each type of information, one or more displays are provided. Displays are organized in worksheets that can be saved and loaded independently from each other. So, KSysguard is not only a simple task manager but also a very powerful tool to control large server farms.

Fig.05 KDE System Guard

Fig.05 KDE System Guard {Image credit: Wikipedia}

See the KSysguard handbook for detailed usage.

#20: Gnome System Monitor – Real-time Systems Reporting and Graphing

The System Monitor application enables you to display basic system information and monitor system processes, usage of system resources, and file systems. You can also use System Monitor to modify the behavior of your system. Although not as powerful as the KDE System Guard, it provides the basic information which may be useful for new users:

  • Displays various basic information about the computer’s hardware and software.
  • Linux Kernel version
  • GNOME version
  • Hardware
  • Installed memory
  • Processors and speeds
  • System Status
  • Currently available disk space
  • Processes
  • Memory and swap space
  • Network usage
  • File Systems
  • Lists all mounted filesystems along with basic information about each.

Fig.06 The Gnome System Monitor application

Fig.06 The Gnome System Monitor application

Bonus: Additional Tools

A few more tools:

  • nmap – scan your server for open ports.
  • lsof – list open files, network connections and much more.
  • ntop web based tool – ntop is the best tool to see network usage in a way similar to what top command does for processes i.e. it is network traffic monitoring software. You can see network status, protocol wise distribution of traffic for UDP, TCP, DNS, HTTP and other protocols.
  • Conky – Another good monitoring tool for the X Window System. It is highly configurable and is able to monitor many system variables including the status of the CPU, memory, swap space, disk storage, temperatures, processes, network interfaces, battery power, system messages, e-mail inboxes etc.
  • GKrellM – It can be used to monitor the status of CPUs, main memory, hard disks, network interfaces, local and remote mailboxes, and many other things.
  • vnstat – vnStat is a console-based network traffic monitor. It keeps a log of hourly, daily and monthly network traffic for the selected interface(s).
  • htop – htop is an enhanced version of top, the interactive process viewer, which can display the list of processes in a tree form.
  • mtr – mtr combines the functionality of the traceroute and ping programs in a single network diagnostic tool.

Siebel schema export and import from one Siebel DB server to other

This document would assist the user to export the Siebel data base dump from one environment and import the same into another environment.  The source and target database server have Oracle as their databases.

Step 1:

Exporting siebel database from source DB server environment:

Connect to the source DB server as oracle user.

[oracle@celbssdb6 ~]$ expdp system/password@DBname schemas=SIEBEL dumpfile=SBL_DEV_ddmmyyyy.dmp logfile=exp_SBL_DEV_ddmmyyyy.log compression=ALL parallel=4&

<The user need not specify the system password in the expdp command.  Without password when the command is run, it would prompt the user to enter the system password>

 Step 2:

Copy the DB dump file to the target DB server environment:

scp the file to target DB server as oracle user to the path /app/oracle/admin/<DBname>/<folder>

Step 3:

Connect to sqlplus on the target DB server as the user system.

Run the query

SQL> select * from siebel.dba_directories where directory_name=’DATA_PUMP_DIR’;

OWNER                          DIRECTORY_NAME

—————————— ——————————

DIRECTORY_PATH

——————————————————————————–

SYS                            DATA_PUMP_DIR

/app/oracle/admin/<DBname>/<folder>/

SQL>drop user SIEBEL cascade;

SQL>commit;

SQL>exit

Drop the user siebel only if the table space and index space on the source and target environment are same.

And re-create it again:

DROP USER SIEBEL CASCADE;

—–

CREATE USER SIEBEL
IDENTIFIED BY <password>
DEFAULT TABLESPACE SIEBEL_DATA
TEMPORARY TABLESPACE TEMP
PROFILE DEFAULT
ACCOUNT UNLOCK;
— 4 Roles for SIEBEL
GRANT SELECT_CATALOG_ROLE TO SIEBEL;
GRANT SCHEDULER_ADMIN TO SIEBEL;
GRANT SSE_ROLE TO SIEBEL;
GRANT TBLO_ROLE TO SIEBEL;
ALTER USER SIEBEL DEFAULT ROLE ALL;
— 2 Tablespace Quotas for SIEBEL
ALTER USER SIEBEL QUOTA UNLIMITED ON SIEBEL_DATA;
ALTER USER SIEBEL QUOTA UNLIMITED ON SIEBEL_INDEX;

—–

Step 4:

Stop the siebel server, gateway server and SWSE on the target environment.

Step 5:

Import the database dump file on the target DB server as oracle user.

impdp system/password@DBname schemas=SIEBEL dumpfile=SBL_DEV_ddmmyyyy.dmp logfile=imp_SBL_DEV_ddmmyyyy.log parallel=4

<The user need not specify the system password in the imppdp command.  Without password when the command is run, it would prompt the user to enter the system password>

After DB import change the password of SIEBEL user to the original password in the target environment.  If the password for the SIEBEL is same for both source and target DB server then leave it.

Step 6:

Start the SWSE, gateway and siebel servers on the target environment.

Step 7:

Run a full compile on the target environment and deploy the SRF & generate browser scripts.

Step 8:

Verify the database updates and changes in the target environment.

BI Publisher integration with Siebel

CLIENT INSTALLATION PREREQUISITIES:

  • Siebel Tools
  • Siebel Client
  • Microsoft .NET Framework. Required as a prerequisite for installing Oracle BI Publisher.
  • Microsoft word
  • Oracle BI Publisher Desktop for generating templates

SERVER INSTALLATION PREREQUISITIES:

  • Siebel Server
  • Siebel Web Client
  • Java Development Kit (version 1.5 or higher)
  • Siebel Application Object Manager (AOM) 

STEPS REQUIRED FOR BI PUBLISHER SERVER SETUP:

  • Download BI publisher installable from Oracle Website and install it on server,
  • Creating user in BI publisher server,
  • Define the Outbound web service for BI Publisher Server,
  • Copying JAR Files to the BI Server,
  • Create JAR Files,
  • Copying Fonts for Use with Siebel Reports,
  • Enable Siebel Reports Component,
  • Configure JVM parameters in Siebel Web Client,
  • Configure DLL Paremeters in Siebel Web Client

INSTALL BI PUBLISHER SERVER

BI Publisher Server can be installed on the same machine where Siebel Server is installed or it can be installed on a different machine, too.

Follow the installation steps as provided in Admin document for installing, starting and stopping the BI Publisher Server.

To test BI Server:

a)      Open the URL of BI publisher server.

b)      Login using Administrator/Administration credentials.

If you are able to successfully login in BI Server, BI is installed properly.

CREATING USER IN BI PUBLISHER SERVER

a)      Login in BI server using Administrator/Administration credentials.

b)      Go in Admin tab

c)      SelectSecurityCenter-> Users->Create User (For eg. Testsiebel)

d)      Assign 6 roles to that user namely;

    1.  BI Publisher Administrator
    2.  BI Publisher Developer
    3.  BI Publisher Scheduler
    4.  BI Publisher Template Designer
    5.  BI Publisher Excel Analyzer
    6.  BI Publisher Online Analyzer

DEFINE OUTBOUND WEB SERVICE FOR BI PUBLISHER SERVER

a)        Login to Siebel Web Client

b)       Go To Administration – Web Services – > Outbound Web Services

c)        In the Name Field query for “PublicReportService”.

d)       Change the Address of Web Service to include BI Publisher server details as below:

http://<host.domain&gt;:<port>/xmlpserver/services/PublicReportService

where:

■host is the full path for your BI Publisher Server

■port is the port the BI Publisher Server uses

For example:

 http://host:port/xmlpserver/services/PublicReportService

e)        Click on “Clear Cache” button.

COPYING JAR FILES

 1) Copy the following files from $SIEBEL_HOME/CLASSES to:

OC4J_HOME/j2ee/home/applications/xmlpserver/xmlpserver/WEB-INF/lib

XSLFunctions.JAR

SiebelCustomXMLP.JAR

SiebelCustomXMLP_SIA.JAR

2) Enable external file references by performing the following tasks:

a) Log in to the BI Publisher Server.

b) Click the Admin tab, and then select Runtime Configuration and Properties.

c) Change the Default value for the Disable External Reference attribute to FALSE.

 3) Restart OC4J.

CREATING JAR FILES

To configure the Siebel Developer Web Client for to create JAR files for Siebel BIPublisher Report, follow below steps:

1. In the Siebel client application Classes installation folder (<installation folder>Classes), create an xmlp.jar package file by performing the following:

a) Copy all the XMLP-related JAR files to an empty directory.

b) Change to the empty directory (type: cd <empty directory name>)

c) Execute the following command:

jar cvf XMLP.jar *.*

2. Edit the siebel.cfg file to add the following, if not already implemented:

[XMLPReports]

XdoDir = /xmlp/templates/

ReportOutputDir = /xmlp/reports/

ReportDataDir = /xmlp/data/

3. Edit the siebel.cfg file to add content similar to the following:

[XMLPJvmSubsys]

FullName = XMLPJvmSubsys

Description = XMLP Java Business Service Subsystem Parameters

SubsysType = JVMSubSys

C:PROGRA~1Siebel8.1WEBCLI~1classesSiebelXMLP.jar;C:PROGRA~1Siebel8.1WEBCLI~1classesXMLP.jar;C:PROGRA~1Siebel8.1WEBCLI~1classessiebel.jar;

C:PROGRA~1Siebel8.1WEBCLI~1classesXSLFunctions.jar;C:PROGRA~1Siebel

8.1WEBCLI~1classesSiebelCustomXMLP.jar

COPYING FONTS

On the Siebel Server machine where the XMLP Report Server Component is enabled, copy the fonts in the C:/Fonts directory to the $JRE_HOME/fonts directory

 ENABLE SIEBEL REPORT COMPONENT

1)   Log in to the Siebel application with system administrator privileges.

2)   Navigate to Administration – Server Configuration, Enterprises, and then Component Groups.

3)   In the Component Groups list, select XMLP Report, and then click Enable.

4)   Click the Synchronize view tab, and then in the Component list, select XMLP Report Server.

5)  Change the Parameters of component “BIP User Name” and “BIP Password” to the

login credentials you created in step 2 in BI Publisher server If you have LDAP Running on ecom, you must provide LDAP User at above parameters. Same user must be created at BI Publisher Server.

6) Click Synchronize.

7)Restart the Siebel Server

 CONGIFURING JVM PARAMETERS

 To configure the JVM Classpath for UNIX using srvrmgr

 1) Run the following command to connect to the Siebel Server Manager:

 srvrmgr /g <gateway machine name:port number> /s <xmlp siebel server name> /e <enterprise> /u <user name> /p <password>

where:

gateway machine name: port number is the name of the physical machine on which the Siebel

Gateway Name Server is running and the port number on which the Gateway Name Server is

listening.

xmlp siebel server name is the name of the Siebel Server on which the Siebel BI Publisher

Reports Server is enabled.

user name is the login name of the administrator.

password is the password for the administrator.

For example, you might use the following command:

srvrmgr /g sdchs20n358:2330 /s schs20n359 /e Siebel /u SADMIN /p MSSQL

 

2) Run the following command to configure the XMLPJvmSubsys parameter:

change param

CLASSPATH=${SIEBEL_HOME}/classes/SiebelXMLP.jar:${SIEBEL_HOME}/classes/

xdoparser.jar:${SIEBEL_HOME}/classes/fix6312772.jar:${SIEBEL_HOME}/classes/

xdocore.jar:${SIEBEL_HOME}/classes/xmlparserv2-904.jar:${SIEBEL_HOME}/classes/

versioninfo.jar:${SIEBEL_HOME}/classes/share.jar:${SIEBEL_HOME}/classes/

jewt4.jar:${SIEBEL_HOME}/classes/jdbc12.jar:${SIEBEL_HOME}/classes/

i18nAPI_v3.jar:${SIEBEL_HOME}/classes/collections.jar:${SIEBEL_HOME}/classes/

bipres.jar:${SIEBEL_HOME}/classes/bicmn.jar:${SIEBEL_HOME}/classes/

Siebel.jar:${SIEBEL_HOME}/classes/XSLFunctions.jar:${SIEBEL_HOME}/classes/

SiebelCustomXMLP.jar for named subsystem XMLPJvmSubsys

3) Restart the Siebel Server.

 CONFIGURING JVM DLL PARAMETERS

To configure the JVM DLL Name value for Siebel BI Publisher Reports

    1. Navigate to the Administration – Server Configuration screen > Enterprises > Profile Configuration view.

2. In the Profile Configuration list, select XMLPJvmSubsys.

     3 In the JVM DLL Name field in the Profile Parameters subview, enter

“JRE_HOME/lib/sparc/client/libjvm.so”

VERIFY JVM CLASSPATH AND JVM DLL SETTINGS:

 1. Log in to the Siebel Server Manager at the enterprise level.

2. Use the following command to verify the JVM Classpath and JVM DLL Name profile parameters are set correctly:

list params for named subsystem XMLPJvmSubsys show PA_ALIAS, PA_VALUE(2048)

PROCESS OF CREATING REPORTS:

 1) GENERATE SAMPLE XML

    •    Create IO. IO name must start with “BIP”.
      • Go to Administration – BIP Reports
      • In “Sample Data File Generation”, select the earlier created IO and click “Generate Sample XML” button.
      • One xml data file would be created with IO name in $SIEBEL_HOME/XMLP/Data folder. (Each time you add extra xml tag in IO and you want to include that tag (field) in the template, you need to generate sample xml of that IO)

2) CREATE THE REPORT

    • Go in “Administration – BIP Reports->Report Template Registration”, create a new report.
      • Specify the IO Name in “Report Template IO” in the below applet.
      • Select the template name. Browse to a location in your machine where template is stored.
      • Select the XLIFF File name. Browse to a location in your machine where XLIFF File is stored.
      • Select the “Output Type”.
      • Click “Upload File”. (Each time you change the template, you need to Upload files to the server).

3) REGISTER THE TEMPLATE

    • Go in “Administration- BIP Reports – > View Association”, select the view name where you want to show the report.
    • Select the report name in below template.