How to connect remote linux server from local linux server with ssh

Using the SSH protocol and by sharing the public key, you can connect and authenticate to remote servers and services. With SSH keys, you can connect to remote server from local server.
Let us consider we have 2 linux machine local server and remote server.

Step 1 . Check if you have already key pair 

Before starting let check if you have public and private in your local server for the user local-user.

[local-user@ip-xx-xx-xx ~]$ ls -l ~/.ssh
-rw-------. 1 local-user local-user  401 Nov 19 09:13 authorized_keys
-rw-------. 1 local-user local-user 3243 Nov 19 17:36 id_rsa
-rw-r--r--. 1 local-user local-user  743 Nov 19 17:36 id_rsa.pub
-rw-r--r--. 1 local-user local-user  803 Nov 19 17:32 known_hosts

If you able to see id_rsa and id_rsa.pub then that means you have already generated key pair otherwise you need to generate key pair.

Step 2. Generate the public and private key pair

Run the following command to generate the ssh key pair.

[local-user@ip-172-31-52-47 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/local-user/.ssh/id_rsa):[ Hit Enter key ]
Created directory '/home/local-user/.ssh'.
Enter passphrase (empty for no passphrase):[ Hit Enter key ]
Enter same passphrase again:[ Hit Enter key ]
Your identification has been saved in /home/local-user/.ssh/id_rsa.
Your public key has been saved in /home/local-user/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Pz5p8KbvF/kcgOTSak45/ZGkE5saGVLZD36U1okLtg4 local-user@ip-172-31-52-47.ec2.internal
The key's randomart image is:
+---[RSA 2048]----+
|         o   + . |
|        o ** = o |
|       . O .     |
|      . E B *    |
|       .SX xB +  |
|        X.B = .  |
|       + *o+ = . |
|        o.*.o o  |
|        .*+o     |
+----[SHA256]-----+

Now you will be able to see public and private key in the /home/local-user/.ssh/ directory.

Step 3 : Share the public key to remote server

Copy and install the public key using ssh-copy-id command

$ ssh-copy-id -i remote-user@<ip-address-of-remote-server>
    remote-user@<ip-address-of-remote-server> password:
   

Note : You can also direct copy the content of public key and pate into authorized_keys (~/.ssh/authorized_keys) file of remote server.
 
Step 4. Access remote server with SSH

Now try logging into the machine, with "ssh remote-user@<ip-address-of-remote-server>" from  local server with local-user.

******************** END ********************

5 ways to count the lines of a file in Linux

There are multiple ways to count the number of lines of a file in Linux. In our daily life we need to count number of lines of a csv file, text file etc and the most popular command we use "wc -l". In this article i will show you 5 different ways to find number of line along with wc -l

Let us consider we have 2 sample file name as sample1.txt and sample2.txt which having 10 and 3 lines as following. I will show you different examples to get the number of lines.

Contents of sample1.txt

[~]$ cat sample1.txt

One
Two
Three
four
five
six This is longest line
seven
eight
nine
ten

Contents of sample2.txt

[~]$ cat sample2.txt
One
Two Longest Line
Three

The wc (word count) command is very popular in Unix/Linux to find number of lines count, word counts, byte and characters count in a file. The syntax of wc command is as following.

wc  [OPTION]... [FILE]...

Where OPTION are as below:

  -c, --bytes            print the byte counts
  -m, --chars            print the character counts
  -l, --lines            print the newline counts
      --files0-from=F    read input from the files specified by
                           NUL-terminated names in file F;
                           If F is - then read names from standard input
  -L, --max-line-length  print the length of the longest line
  -w, --words            print the word counts

[~]$ wc sample1.txt sample2.txt

10 14 70 sample1.txt
 3  5 27 sample2.txt
13 19 97 total

In the above example first column shows number of line , 2nd column shows words and 3rd column shows number of chars. Last row shows the total counts of all files.

Lets us have a look of different ways to find the number of lines in a file , we will use sample1.txt in our demo.

1) Counting lines with WC command

[~]$ cat sample1.txt | wc -l
10

or

[~]$ wc -l sample1.txt
10 sample1.txt

2) Counting lines with sed command
sed uses "=" operator for line numbering and "$" gives the last count of numbering which is total number of lines.

[~]$ sed -n "$=" sample1.txt
10

3) grep command with -c option:

To count all non-empty lines or non-blank line.

[~]$ grep -c "." sample2.txt
3

To count all line including blank or empty lines.

[~]$ grep -c ".*" sample2.txt
4

or

[~]$ grep -c "^" sample2.txt
4

4). Counting line with awk:

awk with NR variable gives the line numbers and by printing NR with the end block it gives the line number of the last line which is nothing but a total number of lines in file.

[~]$ awk 'END {print NR}' sample1.txt
10

5). Counting lines with perl:

End block as in awk we can use also with perl. In Perl "$." gives the number of lines.

[~]$ perl -lne 'END {print $.}' sample1.txt
10

Note:

To find the longest line character count we can use wc with -L option. As shown below in sample1.txt file the longest line having 24 character where as in sample2.txt it is 16.

[~]$ wc -L sample1.txt sample2.txt
24 sample1.txt
16 sample2.txt
24 total

Have a look on couple of heck of counting lines.

Trick 01:

cat -n sample1.txt | tail -n 1 | cut -f1

Explanation : "cat -n sample1.txt" keeps a number line in file and pipe "tail -n1" gives the first row from bottom which includes line number and last line content. Then pipe "cut -f1" take out the first field of last line(last line number) which is nothing but the total count of lines.

Trick 02:

Using the while loop reading file line by line and increasing counter.

#!/bin/bash

count=0
while read
do
  ((count=$count+1))
done <sample1.txt

echo $count

***End***

Split command in Linux/Unix

Split command is very useful when you are managing large file . Consider you have a csv file with millions of records and its taking too much time to open. In this case we can split file into small pieces and can access it easily in any GUI.

The default size for each split file is 1000 lines, and default PREFIX is "x". However we can split file based of number of lines or bytes and can change the prefix as well. In this article i will show you how to use split command with examples.

Let us consider we have a file testfile.csv with 1342 records.

[~]$ cat testfile.csv | wc -l
1342

1). Split simple example :

As you can see below split command split file testfile.csv in 2 pieces with default prefix x. testfile.csv file having total 1342 records hence by default it split first file name as xaa with default 1000 line and second file name as xab with remaining records 342.

[~]$ split testfile.csv

[~]$ ls
testfile.csv  xaa  xab

[~]$ cat xaa | wc -l
1000

[~]$ cat xab | wc -l
342

2) Split file with specific number of lines:

We can use -l option with split command to achieve specific number of line into split files. Let us we want to split file with 500 records for each then use following command.

[~]$ split -l 500 testfile.csv

[~]$ ls
testfile.csv  xaa  xab  xac

[~]$ cat xaa | wc -l
500

[~]$ cat xab | wc -l
500

[~]$ cat xac | wc -l
342

3) Split file with a specific prefix:

If we want to use our own prefix  "NEW" in split files use the following command.

[~]$ split -l 500 testfile.csv NEW

[~]$ ls
NEWaa  NEWab  NEWac  testfile.csv

4) Split file with numeric suffix:

We can append our own numeric suffix like 00,01,02... instead default xa,xb,xc .... with -d option as following.

[~]$ split -l 50 -d testfile.csv NEW

[~]$ ls
NEW00  NEW02  NEW04  NEW06  NEW08  NEW10  NEW12  NEW14  NEW16  NEW18  NEW20  NEW22  NEW24  NEW26
NEW01  NEW03  NEW05  NEW07  NEW09  NEW11  NEW13  NEW15  NEW17  NEW19  NEW21  NEW23  NEW25  testfile.csv

By default numeric suffix has 2 digits and you may need to increase the number of digits if split files crossing more than 100 files. In that case you will get following "suffixes exhausted" message and you may loose some split files after NEW99.

[~]$ split -l 10 -d testfile.csv NEW
split: output file suffixes exhausted

[~]$ ls
NEW00  NEW05  NEW10  NEW15  NEW20  NEW25  NEW30  NEW35  NEW40  NEW45  NEW50  NEW55  NEW60  NEW65  NEW70  NEW75  NEW80  NEW85  NEW90  NEW95  testfile.csv
NEW01  NEW06  NEW11  NEW16  NEW21  NEW26  NEW31  NEW36  NEW41  NEW46  NEW51  NEW56  NEW61  NEW66  NEW71  NEW76  NEW81  NEW86  NEW91  NEW96
NEW02  NEW07  NEW12  NEW17  NEW22  NEW27  NEW32  NEW37  NEW42  NEW47  NEW52  NEW57  NEW62  NEW67  NEW72  NEW77  NEW82  NEW87  NEW92  NEW97
NEW03  NEW08  NEW13  NEW18  NEW23  NEW28  NEW33  NEW38  NEW43  NEW48  NEW53  NEW58  NEW63  NEW68  NEW73  NEW78  NEW83  NEW88  NEW93  NEW98
NEW04  NEW09  NEW14  NEW19  NEW24  NEW29  NEW34  NEW39  NEW44  NEW49  NEW54  NEW59  NEW64  NEW69  NEW74  NEW79  NEW84  NEW89  NEW94  NEW99

To overcome this you can increase number of digits in suffix by using -a option as following.

[~]$ split -l 10 -a 3 -d testfile.csv NEW

[~]$ ls
NEW000  NEW007  NEW014  NEW021 .........  NEW099  NEW100  NEW101  .........  NEW132

5) Split file with 4000 bytes output:

We can use -b option with desired number of size.

[~]$ split -b4000 testfile.csv
              (or)
[~]$ split -b4k testfile.csv

[~]$ ls -ltr x*
-rw-rw-r-- 1 mukesh mukesh 3888 Oct  7 21:14 xae
-rw-rw-r-- 1 mukesh mukesh 4096 Oct  7 21:14 xad
-rw-rw-r-- 1 mukesh mukesh 4096 Oct  7 21:14 xac
-rw-rw-r-- 1 mukesh mukesh 4096 Oct  7 21:14 xab
-rw-rw-r-- 1 mukesh mukesh 4096 Oct  7 21:14 xaa

6) Split file with 2 files of equal length:

We can use -n option in place of -l as following to achieve specific number of file of same records.

[~]$ split -n 2 -d testfile.csv NEW

[~]$ ls
NEW00  NEW01  testfile.csv

[~]$ cat NEW00 | wc -l
670

[~]$ cat NEW01 | wc -l
672

[~]$ cat testfile.csv | wc -l
1342

In above example the expected count should be 671 into each NEW00 and NEW01 but its not. If anyone could explain me it would be appreciated.

***End***

Cron Job In Linux | Scheduling job with crontab in Linux

Cron job are used to schedule commands to be executed periodically. You can setup commands or scripts, which will repeatedly run at a set time. Cron is one of the most useful tool in Linux or UNIX like operating systems. The cron service (daemon) runs in the background and constantly checks the
/etc/crontab file, and /etc/cron.*/ (cron.d/,cron.daily/,cron.hourly/,cron.monthly/,cron.weekly/) directories. It also checks the /var/spool/cron/ directory. Each user can have their own crontab file.

Other than crontab there are 2 more services to schedule the job the "at" command and "batch" command. The "at" command is used to schedule a one-time task at a specific time. The batch command is used to schedule a one-time task to be executed when the systems load average drops below 0.8.

Install or create or edit my own cron jobs:

To edit your crontab file, type the following command at the UNIX / Linux shell prompt:

$ crontab -e

Syntax of crontab (field description)

The syntax is:

    1 2 3 4 5 /path/to/command arg1 arg2

OR

    1 2 3 4 5 /root/backup-script.sh

Where,

1: Minute (0-59)
2: Hours (0-23)
3: Day (0-31)
4: Month (0-12 [12 == December])
5: Day of the week(0-7 [7 or 0 == sunday])

/path/to/command - Script or command name to schedule

Easy to remember format:

Examples of Run backup cron job script:
If you wish to run a backup-script.sh daily at 2.00 AM then first install your cronjob by running the following command as following .
$ crontab -e
append the following entry at the end

0 2 * * * /root/backup-script.sh

save and close the crontab file . The backup-script.sh will run every day at 2.00 AM.

More examples:
To run /path/to/script.sh every 5 min , Enter following command:

*/5 * * * * /path/to/script.sh

To run /path/to/script.sh five minutes after midnight, every day, Enter following:

5 0 * * * /path/to/script.sh

To run /path/to/script.sh at 3:15 PM on the first of every month, Enter following:

15 14 1 * * /path/to/script.sh

To run /path/script.sh at 10 PM on weekdays(Mon,Tue,Wed,Thu,Fri), Enter following:

0 22 * * 1-5 /scripts/script.sh

To run /path/myscripts/script.pl at 23 minutes after midnight, 2am, 4am ..., everyday, Enter:

23 0-23/2 * * * /path/myscripts/script.pl

How do I use operators?

An operator allows you to specifying multiple values in a field. There are four operators:

The asterisk (*) :

This operator specifies all possible values for a field. For example, an asterisk in the hour time field would be equivalent to every hour or an asterisk in the month field would be equivalent to every month.

The comma (,) :

This operator specifies a list of values, for example: "1,5,10,15,20, 25".

The dash (-) :

This operator specifies a range of values, for example: "5-15" days , which is equivalent to typing "5,6,7,8,9,....,13,14,15" using the comma operator.

The separator (/) :

This operator specifies a step value, for example: "0-23/" can be used in the hours field to specify command execution every other hour. Steps are also permitted after an asterisk, so if you want to say every two hours, just use */2.

How To Send Email From Linux Command Line

In Linux the mail command is an essential in our practical life. There is lot of instance when we want to send mail from Unix/Linux server to our outlook or personal email inbox. For example in Linux server we have generated csv file from database and want to pull this in our window machine. In this case we can use mail command to send file in our mailbox direct from Linux rather using FTP to pull report in local window machine.

Also mail command is useful in our automation script in which we want to generate some kind of report and send to its in recipients email box.

Before starting we should make sure we have installed mailx command in our system. If not installed run the following commands.

# For ubuntu/debian

$ sudo apt-get install heirloom-mailx

# For fedora/centos

$ sudo yum install mailx

Usage of mail command:
Once installation done you are ready to use mail command from command line. Following are the couple of examples .

1. Simple mail :
Run the following command and hit enter then you can write message for email body and for new line just hit enter and write message.
Once message complete press Ctrl+D and it would display EOT at the end.

Check the below message in recipient email if you not getting mail in inbox then check in spam/junk folder.

$ mail -s "This is subject" xyz@example.com
Hi there
This is a simple email body
bye!
EOT

2. Redirect message into mail from a file:

We can send email body message from a file by redirecting file as following.

$ mail -s "This is subject" xyz@example.com < /home/mukesh/message/message-body.txt

3. Message body in echo :

We can write the body message with echo as following.

$ echo -e "This is message body \nHere is second line." | mail -s "This is subject" xyz@example.com

4. Send mail to multiple recipients:

To send email to multiple recipient just separate the email id's  by comma.

echo -e "This is message body \nHere is second line." | mail -s "This is subject" xyz@example.com,abc@example.com

5. Using CC and BCC option:

Use -c to add CC and -b to add BCC

$ echo -e "This is message body" | mail -s "This is subject" -c ccrecipient@example.com xyz@example.com

$ echo -e "This is message body" | mail -s "This is subject" -c ccrecipient@example.com -b bccrecipient@example.com xyz@example.com

6. Specifying from Email:
If you wish to put sender email for recipient use -r option as following.

$ echo "This is message body" | mail -s "This is Subject" -r "Mukesh<fromemail@example.com>" recipient@example.com

7.Email with attachment ( Important ):

To attach  file use -a option as following.

echo "This is Message Body" | mail -s "This is Subject" -a /home/mukesh/message/sampleFile.csv recipient@example.com

echo "This is Message Body" | mail -s "This is Subject" -r "Mukesh<fromemail@example.com>" -a /home/mukesh/message/sampleFile.csv recipient@example.com

**** End****

How to find the length of a string variable in UNIX.

You may need to find the length of string or size of string variable in your shell script program. Here are the 5 best ways through which you can acheive it.

Above picture may clear you whole story if not please go in details as following.


Consider we have a variable name VAR and it stores a string "Hello" which length is 5.
VAR="Hello"

1) echo : In the bash we can use echo command as following .

VAR="Hello"
echo ${#VAR}
5

2) echo with wc : Second method is echo with wc with -c option as following.

VAR="Hello"
echo -n $VAR | wc -m
5

or

echo -n $VAR | wc -c
5

Note:
Where -m print the character counts and -c print the byte counts .

3) printf with wc : 3rd method is to use printf with wc as following.

printf $VAR | wc -c
5

or

printf $VAR | wc -m
5

4) echo with awk : 4th method is echo with awk as following.

echo $VAR | awk '{print length ;}'
5

5) expr : 5th method to use expr as following.

expr length $VAR
5

or

expr $VAR : '.*'
5

Watch video in detail

How To Install NRPE On Redhat or CentOS Linux Server

Need personal assistance on Nagios? Please contact me at immukesh72@gmail.com at very nominal charges

Step 01: Install NRPE Plugins in Nagios monitoring host

wget http://downloads.sourceforge.net/project/nagios/nrpe-2.x/nrpe-2.15/nrpe-2.15.tar.gz

tar -zxvf nrpe-2.15.tar.gz

cd nrpe-2.15

./configure --enable-command-args --with-nagios-user=nagios --with-nagios-group=nagios --with-ssl=/usr/bin/openssl --with-ssl-lib=/usr/lib/x86_64-linux-gnu

Now build and install NRPE and its xinetd startup script with these commands:

make all
sudo make install
sudo make install-xinetd
sudo make install-daemon-config

  
Step 01: On client server that you want to monitor, install the EPEL repository:
    NRPE packages and plugins are available under epel yum repository

sudo yum install epel-release
or
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

RPM (Red Hat Package Manager) is an default open source and most popular package management utility for Red Hat based systems like (RHEL, CentOS and Fedora). The tool allows system administrators and users to install, update, uninstall, query, verify and manage system software packages in Unix/Linux operating systems.
   
Now install Nagios Plugins and NRPE:

yum --enablerepo=epel -y install nrpe nagios-plugins

update private IP address of your Nagios server in allowed host

sudo vi /etc/nagios/nrpe.cfg
allowed_hosts=127.0.0.1,10.132.224.168

Add new commands or update existing commands like below in same file.

 command[check_root_disk]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
 command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20

Step 02: Start NRPE Service:
Reboot nrpe once to read new configuration, Also configure nrpe to auto start on system boot.

service nrpe start
chkconfig nrpe on

Step 03: Test NRPE from Nagios server
Login to Nagios server and run the following command. On successful connection it will print the version of nrpe package.

/usr/local/nagios/libexec/check_nrpe -H 10.132.224.168
NRPE v2.15

Need personal assistance on Nagios? Please contact me at immukesh72@gmail.com at very nominal charges

How to Install Nagios on RHEL,CentOS or Fedora Linux Operating System.

In this tutorial i will tell you how to install Nagios Core 4.1.1 and Nagios Plugins 2.1.1 on RHEL Linux server. Following all steps were performed and tested on RHEL 7.3 Linux Server.

Step 01 : Disable SELinux
Disable the SELinux by running the following command with root user.

$ setenforce 0

Basically this a security feature of the Linux kernel. It is designed to protect the server against misconfigurations and/or compromised daemons.

Note: setenforce is a command line utility that is used to switch the mode SELinux is running in from enforcing to permissive and vice versa without requiring a reboot.

Modify /etc/selinux/config and change enforcing to disabled

Need personal assistance on Nagios? Please contact me at immukesh72@gmail.com at very nominal charges

Step 02: Install the required packages
Make sure you have installed the following packages on your Linux system before continuing.
Apache 2
PHP
GCC compiler and development libraries
GD development libraries

If you did not install then please run the following commands:

sudo yum install httpd php php-cli gcc glibc glibc-common gd gd-devel net-snmp openssl-devel wget unzip -y

Note:
GCC compiler and development libraries :
GNU Compiler Collection. gcc, formerly known as the GNU C Compiler, compiles multiple languages (C, C++, Objective-C, Ada, FORTRAN, and Java) to machine code.

GD development libraries :
The GD library is a graphics drawing library that provides tools for manipulating image data. In Shopp, the GD library is used to process images for generating gallery preview and thumbnail size images automatically.

net-snmp:
Simple Network Management Protocol (SNMP) is a widely used protocol for monitoring the health and welfare of network equipment (eg. routers), computer equipment and even devices like UPSs.
And helps Nagios for notification emails.

openssl-devel:
OpenSSL is a toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols. It is also a general-purpose cryptography library.

php-cli : PHP Command Line Interface

Step 03: Create Account Information
Create the Nagios user

  useradd nagios

Create a new nagcmd group for allowing external commands to be submitted through the web interface.

  groupadd nagcmd

Add both the nagios user and the apache user to the group nagcmd.

  usermod -a -G nagcmd nagios
  usermod -a -G nagcmd apache

Where :
The usermod command modifies the system account files to reflect the changes that are specified on the command line.
-a, --append
           Add the user to the supplementary group(s). Use only with the -G option.

-G, --groups GROUP1[,GROUP2,...[,GROUPN]]]
           A list of supplementary groups which the user is also a member of.
         
Step 04: Download and Install Nagios
Run the following commands to download and extract Nagios and Nagios plugins in /tmp directory.

cd /tmp

wget https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.1.1.tar.gz

wget http://www.nagios-plugins.org/download/nagios-plugins-2.1.1.tar.gz

tar zxf nagios-4.1.1.tar.gz

tar zxf nagios-plugins-2.1.1.tar.gz

Go to nagios-4.1.1 directory and start compiling Nagios.
    cd nagios-4.1.1

Run the Nagios configure script, passing the name of the group you created earlier like so:

./configure --with-command-group=nagcmd

Compile the Nagios source code.

make all

Install binaries, init script, sample config files and set permissions on the external command directory.

make install
make install-init
make install-config
make install-commandmode

Configure the Web Interface

make install-webconf

Same way install the Nagios plugin as well.

cd /tmp/nagios-plugins-2.1.1

./configure --with-nagios-user=nagios --with-nagios-group=nagios --with-openssl

make all

make install

Step 05: Creating a password for nagiosadmin
We now need to create a password for the nagiosadmin user. This will be used to login to your core web GUI.

htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

Step 06: Start Nagios and Apache server

service httpd start
service nagios start (or) /etc/init.d/nagios start

Step 07: Login to the Web Interface
Enter the following url into the browser, replace your public IP of nagios server. It will ask to enter username and password. In this configuration we have username as nagiosadmin and password also as nagiosadmin.

http://<public_ip_of_nagios_server>/nagios/

Step 08: Troubleshooting
If you cannot access the Nagios web page, it may be related to your firewall rules. The following command will open up port 80 on your Nagios Core machine.

firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --reload

If you don't have firewall-cmd then try the following rule into your iptables

iptables -A INPUT -p tcp --dport http -j ACCEPT
iptables -A INPUT -p tcp --dport https -j ACCEPT
iptables-save > /etc/sysconfig/iptables
/etc/init.d/iptables restart

If you are still unable to access the web GUI, your web server may be only listening on IPv6. Modify /etc/httpd/conf/httpd.conf
and look for the part that says 'Listen :80'. Modify it to be 'Listen 0.0.0.0:80'. Then, run service httpd restart.

Need personal assistance on Nagios? Please contact me at immukesh72@gmail.com at very nominal charges

Useful Linux Commands - Part 01

1. Command to grep a process and kill

We can use xargs as following. Consider we have a process sleep.sh
 ps -ef  | grep sleep.sh | awk  '{print$2}' | xargs kill -9
 ps -ef  | grep sleep.sh | grep -v 'grep' | awk  '{print$2}' | xargs kill -9


2. Command to list top 10 large sized file in /var directory

sudo du -a /var | sort -n -r | head -n 10

3. Consider demo.csv file with following content.
mukesh@ubuntu:~/Desktop$ cat demo.csv
sno,Item,cost
1,Tea,20
2,Burger,25
3,Ice cream,30
4,Candy,15

Problem 01: Write a command to iterate all the items available in demo.csv file.

cat demo.csv | awk -F ',' '{print $2}' | grep -nv [0-9]
1:Item
2:Tea
3:Burger
4:Ice cream
5:Candy

Problem 02 : Write command to sum up the the cost of items in demo.csv

$ awk -F ',' '{s+=$3}END{print s}' demo.csv
awk -F ',' '{total = total + $3}END{print "Total Amount collected = "total}' demo.csv
Total Amount collected = 90

4. Write a for loop from 1 to 5 in single command

Syntax : for i in {1..5}; do COMMAND-HERE; done
$ for i in {1..5}; do echo $1; done
1
2
3
4
5

5. Write command to all the files in /home/mukesh/Desktop directory.

$ for i in /home/mukesh/Desktop/* ; do echo "$i"; done

6. Write command to grep 2 more lines after the pattern match

$ grep -A2 Tea demo.csv
1,Tea,20
2,Burger,25
3,Ice cream,30

7. Write command to grep 2 more lines before the pattern match

$ grep -B2 Ice demo.csv
1,Tea,20
2,Burger,25
3,Ice cream,30

8. Command to print date 1 day before and and after in YYYY-MM-DD format

refdate=`date +"%Y-%m-%d"`
1 day before date:
refdate=`date --date="1 day before" +%Y-%m-%d`

$ date --date="1 days ago"
$ date --date="1 day ago"
$ date --date="yesterday"
$ date --date="-1 day"

$ date --date="next day"

9. Command to sed the multiple patterns

 sed 's/ab/~~/g; s/bc/ab/g; s/~~/bc/g' file.txt > file_tmp.txt

10. list all files which were created in last 1 Hrs

find <search_dirname> -cmin -60 # creation time
find <search_dirname> -mmin -60 # modification time
find <search_dirname> -amin -60 # access time

11. Command to find all files modified,accessed and permission changed on the 7th of June, 2007:

$ find . -type f -newermt 2007-06-07 ! -newermt 2007-06-08
$ find . -type f -newerat 2008-09-29 ! -newerat 2008-09-30
$ find . -type f -newerct 2008-09-29 ! -newerct 2008-09-30

12. find the files which were created on more/less than 30 days and created exact on 30th day ago

we can use -ctime option with find command as following
    More than 30 days ago: -ctime +30
    Less than 30 days ago: -ctime -30
    Exactly 30 days ago: -ctime 30

13. Command to grep multiple pattern

grep  -e '2015-12-02' -e '2015-12-03' dateFile.txt
grep  '2015-12-02\|2015-12-03' dateFile.txt

14. Command to display cpu info of your linux/Unix system

 Following command will list number of processor on the server and its details.
 cat /proc/cpuinfo

15. How to list top memory consuming process.

 Top + Press Shift  m

16. Command to check linux version

Note : CentOS is based on Redhat Enterprise Linux, while Ubuntu/Ubuntu Server has its roots in Debian
# For ubuntu
 cat /etc/os-release

    root@ubuntu:/etc/ansible# cat /etc/os-release
    NAME="Ubuntu"
    VERSION="12.04.5 LTS, Precise Pangolin"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu precise (12.04.5 LTS)"
    VERSION_ID="12.04"

# For centos
 cat /etc/system-release
 cat /etc/redhat-release
  CentOS release 6.5 (Final)

Top Daily Use Linux/Unix Commands

pwd
cat
less
cd
mkdir
ls
cp
mv
head
tail
wc
grep
kill
nohup
df
du
zip
tar
find
date

Top Useful Network Monitoring Commands

Useful commands for network monitoring

Ping  (Unix/Windows)
Traceroute  (Unix/Windows)
Arp (Unix/Windows)
Curl and wget (Unix/ Windows)
Netstat (Unix/Windows)
Whois (Unix/ Windows)
SSH (Unix/Linux/Windows)
TCPDump (Unix/Linux/Windows)
Ngrep (Unix/Linux/Windows)
NMAP (Unix/Windows)
Netcat (Windows/Unix)
Lsof (Unix/Windows)
IPtraf (Linux)

Ping (Unix/Window):
Ping is very basic and important command. Ping sends an ICMP ECHO_REQUEST packet to the specified host. If the host responds, you get an ICMP packet back. You can “ping” an IP address to see if a machine is alive. If there is no response, you know something is wrong. It’s also used to check the “speed” or latency time for said network connection. It’s a command that exists on all OS’s that support TCP/IP and it’s one of those basics you should know. Following is the ping examle.

$ ping HostName/IP Address
$ ping google.com
$ ping 192.168.182.132

You can specify the count of ECHO_REQUEST packets to be sent while ping a host. For this we can use parameter '-c' as following.

$ ping -c 4 192.168.182.132
PING 192.168.182.132 (192.168.182.132) 56(84) bytes of data.
64 bytes from 192.168.182.132: icmp_req=1 ttl=64 time=0.281 ms
64 bytes from 192.168.182.132: icmp_req=2 ttl=64 time=0.495 ms
64 bytes from 192.168.182.132: icmp_req=3 ttl=64 time=0.459 ms
64 bytes from 192.168.182.132: icmp_req=4 ttl=64 time=0.250 ms

--- 192.168.182.132 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.250/0.371/0.495/0.107 ms

Traceroute  (Unix/Windows):
Traceroute command is a very useful network diagnostic tool. Traceroute displays each host that a packet travels through(devices, switches, routers, computers) as it tries to reach its destination. In case of trouble it can give us an idea at which point problem is there while moving packets to its destination.

Traceroute (Unix):
~$ traceroute 192.168.182.132
traceroute to 192.168.182.132 (192.168.182.132), 30 hops max, 60 byte packets
 1  ubuntu.local (192.168.182.132)  0.302 ms  0.316 ms  0.304 ms

tracert( Window ):
C:\Users\Mukesh.Kumar>tracert google.com

Tracing route to google.com [216.58.220.46]
over a maximum of 30 hops:
  1     *        *        *     Request timed out.
  2    48 ms    56 ms    49 ms  10.210.0.82
  3    55 ms    23 ms    41 ms  10.210.0.86
  4    25 ms    26 ms    29 ms  125.17.150.37
  5    62 ms    42 ms    38 ms  182.79.234.221
  6    49 ms     *       59 ms  72.14.242.178
  7    56 ms    46 ms    34 ms  66.249.94.73
  8    44 ms    36 ms     *     209.85.255.43
  9    57 ms    50 ms    29 ms  maa03s18-in-f14.1e100.net [216.58.220.46]

Trace complete.

Note : The Unix "traceroute" uses UDP datagrams rather than ICMP to perform a similar function link ping.

Arp (Unix/Windows) :
Using the arp command allows you to display and modify the Address Resolution Protocol (ARP) cache. An ARP cache is a simple mapping of IP addresses to MAC addresses.
Example:
C:\Users\Mukesh.Kumar>arp -a 192.168.182.132

Interface: 192.168.182.1 --- 0x1e
  Internet Address      Physical Address      Type
  192.168.182.132       00-0c-29-08-b7-93     dynamic
 
Wget and curl (Unix/ Windows):
    This command allow to download files or entire webpage.both are command line tools that can download contents from FTP, HTTP and HTTPS
    both can send HTTP POST requests
    both support HTTP cookies
    both are designed to work without user interaction, like from within scripts
    both are fully open source and free software
    both support metalink

    Note : curl supports FTP, FTPS, Gopher, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT,
      LDAP, LDAPS, FILE, POP3, IMAP, SMB/CIFS, SMTP, RTMP and RTSP.    
      Wget only supports HTTP, HTTPS and FTP

Netstat (Unix/Windows):
Netstat prints information about the Linux networking subsystem. By default, netstat displays a list of open sockets.  If you don't specify any address families, then the active sockets of  all  configured address families will be printed.  The type of information printed is controlled by the first argument,as follows:
Netstat( Unix ):
   --route , -r
       Display the kernel routing tables. See the description in route(8) for details.  netstat -r and route -e produce the same output.

   --groups , -g
       Display multicast group membership information for IPv4 and IPv6.

   --interfaces, -i
       Display a table of all network interfaces.

   --masquerade , -M
       Display a list of masqueraded connections.

   --statistics , -s
       Display summary statistics for each protocol.
'

Whois (Unix/ Windows) :
Network command used to consult domain data. Mainly data like the domain owner, it’s expire time, configured registries, contact data, etc. are consulted. It’s very recommendable to use it to contact domain administrators or for service migration instances, such as email or webpage migrations.

In order to use ‘whois’ on Windows, you should download the software linked on the following URL: https://technet.microsoft.com/en-us/sysinternals/whois.aspx

You can also consult them using services such as https://www.whois.net/ on your browser.

SSH (Unix/Linux/Windows):
SSH, or Secure Shell, is a protocol used to securely log onto remote systems. It is the most common way to access remote Linux and Unix-like servers.

$ ssh remote-host

If remote user name is diffrence then you can use username also
$ ssh user-name@remote-host

If you wish to execute a command on remote system, you can specify it after the host-name.
$ ssh remote_host command_to_run
$ ssh mukesh@ubuntu-server.com ls

Note : To use SSH on Windows we recommend using Putty. http://www.putty.org/

TCPDump (Unix/Linux/Windows):
tcpdump is a most powerful and widely used command-line packets sniffer or package analyzer tool which is used to capture or filter TCP/IP packets that received or transferred over a network on a specific interface. It is available under most of the Linux/Unix based operating systems. tcpdump also gives us a option to save captured packets in a file for future analysis. It saves the file in a pcap format, that can be viewed by tcpdump command or a open source GUI based tool called Wireshark (Network Protocol Analyzier) that reads tcpdump pcap format files.

Ngrep (Unix/Linux/Windows):
This takes the potency of the ‘grep’ command to the Net. It’s basically a tcpdump with text subchain filters in real time. It’s an HTTP, SMTP, DNS and other protocol communication packets filter. It has a very powerful filtering system over regular expressions and it’s usually used to process files generated by tcpdump, Wireshark, etc.

NMAP (Unix/Windows):
The Nmap aka Network Mapper is an open source and a very versatile tool for Linux system/network administrators. Nmap is used for exploring networks, perform security scans, network audit and finding open ports on remote machine. It scans for Live hosts, Operating systems, packet filters and open ports running on remote hosts.

Netcat (Windows/Unix):
Netcat or nc is a networking utility for debugging and investigating the network.

This utility can be used for creating TCP/UDP connections and investigating them. The biggest use of this utility is in the scripts where we need to deal with TCP/UDP sockets.

Lsof (Unix/Windows):
It’s a tool that’s useful for identifying which files a process is using or keeping open. In the case of Unix environments, a file is also a network connection, so it’s useful to know which ports are open during a specific running process, something which can prove to be extremely useful in some cases.
It can also be used to know how many files a single process has open. It doesn’t have anything to do with the network, but we’re sure it’ll be useful for you anyway. Lsof is one of those tools you should know about.

IPtraf (Linux):
A specialized network command which obtains traffic statistics. It has an ncurses interface (text) to analyze the traffic that goes through an interface in real time. Very useful if you see anomalies on your device and you need to see and inspect the traffic coming through it.

Shell Script Programs For Beginners - SET 02 ( Case and Loop )

1). Switch case Example 01
2). Switch case Example 02
3). Switch case Example 03
4). For loop Example01 : Simple for loop.
5). For Loop Example02 : List all content of current directory.
6). For loop Example03: Executing unix command in loop.
7). While loop Example 01: simple while loop
8). Until loop Example01: Simple until loop
9). For loop with break keyword
10). For loop with continue keyword

1) Switch case Example 01

#!/bin/bash
#
#Script Name : caseExample01.sh 
#Description : Script will read a value from user and choose accordingly choice.
#
echo "Enter the number from 1 to 4"

read x

case $x in 
    1) echo "One";;
    2) echo "Two";;
    3) echo "Three";;
    4) echo "Four";;
    *) echo "Wrong input, Please enter number 1 to 4";;
esac

Output:
Case 1 : if x=2
Two
Case 2 : if x=4
Four
Case 3 : if x=0 or 5 or 10 or d
Wrong input, Please enter number 1 to 4

2) Switch case Example02

#!/bin/bash
#
#Script Name : caseExample02.sh 
#Description : Script will read a value from user and choose accordingly choice.
#               Keeping default '*)' in between rather keeping it at end and observe the output of program.
#

echo "Enter the number from 1 to 4"
read x
case $x in 
    1) echo "One";;
    2) echo "Two";;
    3) echo "Three";;
    *) echo "Wrong input, Please enter number 1 to 4";;
    4) echo "Four";;
esac

Output:
Case 1 : if x=1
One
Case 2 : if x=4
Wrong input, Please enter number 1 to 4
Case 3 : if x=0 or 5 or 10 or d
Wrong input, Please enter number 1 to 4

If you observe the above program it is clear that the case which are places after default case those does not come under consideration of program.

3) Switch case Example 03

#!/bin/bash
#
#Script Name : caseExample03.sh 
#Description : Script will read a value from user and choose accordingly choice.
#               We can use expression also in case.
#

case `expr 20 + 5` in 
    10) echo "Ten";;
    25) echo "Twenty Five";;
    30) echo "Thirty";;
    *) echo "Wrong input";;
esac

Output:
Twenty Five

Loop:
Loops are very powerful tool to perform a set of instructions repeatedly until a certain condition is reached.
For,While and Until are the popular loop in shell program . Following are the example of each kind of loops.

4) For loop Example01 : Simple for loop

#!/bin/bash
#
#Script Name : for01.sh
#Description : Script showing demo of a simple for loop
#

for val in a b c d e f g h i j
do
        echo $val
done

5) For Loop Example02 : List all content of current directory

#!/bin/sh
#
#Script Name : for02.sh
#Description : Script showing demo of a for loop in which program listing all the contents of current directory.

for val in *
do
  echo "$val"
done

Remember * is Wildcard which will consider all the contents of current directory hence the above program will loop all the file and directory available in current directory.

6) For loop Example03: Executing unix command in loop

#!/bin/sh
#
#Script Name : for03.sh
#Description : Script showing demo of a for loop in which program is executing unix commands

for val in date cal exit who

do
   "$val"
done

Output:
Mon Dec 12 22:05:57 IST 2016
   December 2016
Su Mo Tu We Th Fr Sa
             1  2  3
 4  5  6  7  8  9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

Note : Replcae "$val" to echo "$val" in above program and observe the output.

While Loop:

In 'while' loop, we give one control command to control the iteration of the loop
A set of commands will be executed as long as the control command's exit status is 0
When the control command's exit status becomes 1, the 'while' loop is quit.

7) While loop Example 01: simple while loop

#!/bin/sh
#
#Script Name : while01.sh
#Description : A simple while loop demo

a=0

while [ $a -lt 10 ]
do
   echo -ne "$a\t"
   a=`expr $a + 1`
done

Output:
0       1       2       3       4       5       6       7       8       9

Note : In above program i used -n and -e option where -n option is used to disable trailing new line character.
and -e option is used to enable escape sequences and \t is used for tab.

Until loop:
A until loop is used to repeat set of commands as long as the control command's exit status is 1
When the control command's exit status becomes 0, the loop is get quit.

8) Until loop Example01: Simple until loop

#!/bin/sh
#
#Script Name : until01.sh
#Description : A simple until loop demo


echo  "Type 'end' to exit  "
read var
until  [  "$var"  =  "end"  ]
do
    `echo $var`
     read  var
done

echo  "End of the script and exiting"

Output:
Type 'end' to exit
date
Mon Dec 12 23:13:00 IST 2016
end
End of the script and exiting

Break and Continue Keywords:
In shell scripts we can use break and continue keywords only in the looping statements (i.e., while , until and for).

9) For loop with break keyword
In the following example as soon variable i become equal to 5 the loop breaks and quit.

#!/bin/sh
#
#Script Name : break.sh
#Description : A simple break keyword in a loop


for i in 1 2 3 4 5 6 7 8
do
    echo $i
    if [ $i -eq 5 ]
    then
        break
    fi

done

Output:
1 2 3 4 5

10) For loop with <b>continue</b> keyword
Following example showing the a demo of 'continue' keyword in which it print which are not equal to 5, as soon controller reach to 5 continue keyword move it to the next number.

#!/bin/sh
#
#Script Name : continue.sh
#Description : A simple continue keyword in a loop

for i in 1 2 3 4 5 6 7 8

do
#   echo $i
    if [ $i -ne 5 ]
    then
        echo $i
    else
        continue
    fi

done

Output:
1 2 3 4 6 7 8

File System in UNIX

The UNIX file system is different from the MS-DOS file system. The disk space allotted to a UNIX file system is made of blocks of usually 512 bytes. All the blocks belonging to a file system are divided into following four parts

Boot Block  : Occupies the beginning of the root file system. This block contains a program called
                       bootstrap loader. Bootstrap loader executes when the host machine is booted.

Super Block : Super block contains the state of the file system ie its size, where to find the free space on the file system, how many files it can store etc.

Inode Block(Table) : It follows the super block and gives the internal representation of the file. The information related to these files is stored in a table known as Inode Table on the disk. For each file, there is an inode entry in the table. Each entry is approx 64 bytes and contains details such as

•     Owner of the file
•     Type of the file
•     File access permissions
•     File access time
•     Size of the file
•     Date and time of last access etc

   
Data Block : Data block contains the actual file contents. An allocated block can belong to only one file in the file system.(Size of blocks can vary from 512 bytes to 4K)


Hierarchy of Unix File System:

The Unix file system follows a tree structure with root represented by '/'.
Every non-leaf node is a directory, containing some other files and subdirectories
Every leaf nodes can be empty directories, regular files or special device files.

The main features of a Unix file system:

•     Hierarchical structure
•     Create and delete files
•     Dynamic growth of files
•     Protection of file data
•     Treats peripheral devices as files
•     Keeps track of files using i-node numbers.
•     Information of files is kept in the i-node block.

File access permissions can be set for 3 classes of users:

•     File Owner
•     File Owner’s Group
•     Others

Permissions on file:
Read (r), Write (w), Execute (x) permissions can be set for each file.

/bin        : Executable System Utilities (sh, cp, rm etc.)
/lib         : Operating System and Programming Libraries.
/dev       : Device Related Files
/etc        : System Configuration files and databases. 
/tmp      : System Scratch Files
/usr       : Home Directories of all users

   ***************************END**************************

Unix System Architecture

There 4 major components in the Unix system.

Kernal : Monitors and controls the resources of a computer and allocates them among its users in an
              optimal manner.
Shell    : Provides a processing environment for the user programs and acts like a command translator
Utilities : Programs which are used for development purposes
User Applications : Programs written by the user

The above 4 components can be understood better by a layer approach as following:

Layer 1 (Hardware ) : 

This layer is the hardware over which the different OS layers are built. Hardware is not a part of the Unix OS.

Layer 2 (Kernal) :

This layer is the Kernel, which is the most important component of the Unix system. It is a collection of programs which directly communicate with the hardware. It is that part of the Unix system that is loaded into memory first when the system is booted. It manages the system resources, allocates them between users and the processes, decides priorities of different processes etc.

Layer 3 (User Application/Commands): 

This layer contains programs that interact with the Kernel by invoking some system-calls. Basically these programs are Unix commands like wc, grep, find etc.

Layer 4 (Shell):

This layer is called Shell. Technically another Unix command, it is the interpreter of user requests. It takes a command from the user, deciphers it, and communicates with the Kernel to see that the command is executed. It is actually the interface between the user and the kernel, which effectively insulates the user from the knowledge of kernel functions. It also has a programming capability of its own.

The kernel and shell together make the Unix system work. The shell provides a processing environment to the user programs.

Unix/Linux Shell Variables

Variables in Unix/Linux Shell

To process our data/information, data must be kept in computers RAM memory. RAM memory is divided into small locations, and each location had unique number called memory location/address, which is used to hold our data. Programmer can give a unique name to this memory location/address called memory variable or variable (Its a named storage location that may take different values, but only one at a time).

In Unix/Linux(Shell), there are two types of variable:

1)  System variables : 

    Created and maintained by Linux itself. This type of variable defined in CAPITAL LETTERS.

    You can see system variables by giving command like $ set, There is a long list of system variable

    so if you interested to see all the system variable then you should redirect the all system variable

    into a text file by using command as "set > systemVariable.txt". Now check systemVariable.txt in

    your current directory.

Some of the important System variables are as following:
   

System Variable	Description
BASH=/bin/bash	System Shell Name
BASH_VERSION=4.2.25(1)-release	System Shell Version Name
COLUMNS=80	Number of Column for your screen
LINES=34	Number of Lines for your screen
HOME=/home/mukesh	Your Shell's Home Directory
LOGNAME=mukesh	Logging Name of Current System
OSTYPE=Linux	Our Operating System Type
PATH=/usr/bin:/sbin:/bin:/usr/sbin	Our System's Path Setting
PS1=[\u@\h \W]\$	Our Prompt Settings
SHELL=/bin/bash	Our Shell Name
PWD=/home/mukesh/Documents	Our Current Working Directory
USERNAME=mukesh	Username who is currently log in into system

   
   
    Note :Some of the above settings can be different in your Unix/Linux environment as per your system.
    You can print any of the above variables contains as follows:

    $ echo $USERNAME
    $ echo $BASH_VERSION



2) User defined variables : 

•   Created and maintained by user. This type of variable defined in lower letters.
•   User-defined variable can be assigned a value.
•   The value of a named variable can be retrieved by preceding the variable name with a ‘$’ sign
•   Used extensively in shell-scripts.
•   Used for reading data, storing and displaying it.

Accepting Data and Displaying It on Screen:


•  To accept input data from user you can use read command.
•  In shell scripts , read is used for accepting data from the user and storing it for other processing.
•  Example : $ read yourname

Type command "read yourname" and hit enter. Type your name and hit enter again. So now your name has been stored into user defined varaible name yourname.

Displaying Value:
To display the value you can use the echo command.
So to display your name which you have stored into a variable yourname type the following command.

  echo $yourname