What does 2>&1 mean in a shell script?

I am working on a change in a shell script that I used to work with my ex-intern last year. I came across a syntax that I forgot what did it mean when it was added into or pre-existed in the shell script. I did a quick search online to get some information, and I think it is good to share the information in my blog.

There is a reply that I found in stackoverflow.com with highest numbers of ticks. It means many people agreed with the explanation given by the person who replied to the post. Here is the information,

File descriptor 0 is the standard input (stdin)
File descriptor 1 is the standard output (stdout)
File descriptor 2 is the standard error (stderr)
> means redirection, send to as a whole complete file, overwriting target if exists.
>> means redirections, send in addition to would append to the target if exists.
& indicates that what follows and precedes is a file descriptor and not a filename.
The >& syntax redirects a stream to another file descriptor, or it can consider as a redirect merger operator.

Samples

I can redirect stdout to stderr by using the below syntax,

echo test 1>&2
echo test >&2

References: https://stackoverflow.com/questions/818255/in-the-shell-what-does-21-mean

How to use SSH to connect to a remote server

To establish a connection with a remote machine depending on the operation system you are running, there are two most commonly used protocols:

• Secure Shell (SSH) for Linux based machine.
• Remote desktop protocal (RDP) for Windows based machine.

This protocols use client and server applications to establish a remote connection. It allows you to gain access and remotely manage other machines. Today’s topic focuses on Linux based machine and I am using Ubuntu 16.04.

How to install an OpenSSH Client

For Linux Ubuntu, you can install the openssh-server to enable the SSH on Ubuntu client machine. For Windows machine, you can install PuTTY or any other client of your choice.

sudo apt-get install openssh-server

OpenSSH defaults

• TCP port: 22
• OpenSSH server config file is called sshd_config which is located at /etc/ssh/

How to install an OpenSSH Server

In order to accept SSH connections, a machine needs to have a server-side part of the SSH. First, you can check if OpenSSH server is available on the Ubuntu server machine of the remote computer that needs to accept SSH connections, you can try to connect to the localhost.

ssh localhost

Ubuntu machine that without the SSH server installed, the screen may shows:

username@host:~$ ssh localhost
ssh: connect to host localhost port 22: Connection refused username@host:~$

The port 22 is not established, therefore the connection is refused. Then, you can install the SSH server,

sudo apt-get install openssh-server ii

You can check if SSH server is running after the installation is completed on the Ubuntu machine by using command below:

sudo service ssh status

The screenshot above show the status is active and running currently. Then, we can move back to the client machine which can be our local machine to try the command ssh to the remote server machine.

How to connect via SSH

Open the terminal from your machine and run command: ssh username@host_ip_address

Key in the password and start connecting. If you are connected for the first time, an ECDSA key fingerprint is required, so just follow the instruction on the terminal and then, you are connected to the remote server. If the default port has changed, you can specify the port behind the ip address with -p 9876

Firewall rules

Next you may want to look at the firewall setting in the server machine to ensure the port is listening, not blocked by firewall and forwarded incorrectly.

Creating a Linux service

I think creating a Linux service is a fairly easy task and it allows us to write a program and turn it into a service using systemd. The service can start or stop using a terminal or GUI (Graphical User Interface) for Windows. I am going to use terminal for this topic here as we are going to create a Linux service to start running my program.

Another reason I want to create a service to run my program is it allows my program to be restarted if it terminates due to unforeseen reasons.

How to begin?

I started with create a script file with .service file extension. I got a sample copy from my colleague and began to modify the script. I have to be careful enough with the directory path. While I thought it may just a directory change from /opt/ to /home/ for the WorkingDirectory and ExecStart in the script, it did not turn out working when I enabled and started the service. I will share what the stupid mistaken I made was.

Once you have the script file ready, saved and copy it to the following directory, /etc/systemd/system. You cannot do a copy-paste of the document into this directory path.

sudo cp mytest_service.service /etc/systemd/system

Permission and Executable File

Next, add the file permission to the root user and make the file executable. You can check on the command, chmod for details setting of file permission for user, guest and other. Otherwise, to set the file to be executable, the below command does the work.

sudo chmod +x mytest_service.service

Enable and Start Service

Once you have the .service file ready in the said directory and make it executable, I think it should be good enough for the next step which is to enable to the service and start the service.

sudo systemctl enable mytest_service.service
sudo systemctl start mytest_service.service

Created symlink from /etc/systemd/system/multi-user.target.wants/mytest_service.service to /etc/systemd/system/mytest_service.service.

Upon running the above command to enable the service, the symlink is created as shown above.

If you wish to know whether the service is started correctly, you can use the following command to check the running status.

sudo systemctl status mytest_service.service

Checking the status will give us few information which we want know, whether the service is active, running or failed with error. The error message serves an important keyword for us to search online for solutions.

If you search on the Internet, you may find that some command is using service instead of systemctl. service is a fairly simple wrapper which only supports limited actions such as start, stop and check the status of the service. For more complex tasks, actual command to be used is systemctl.

For more reading about systemctl and service command, you can find it in this link.

What is the error I received?

The “Active” status on the screen shows “failed” when I ran the command to start the service. It showed an error code 200/CHDIR. I googled the error and found out this error indicated the path is not found or accessible at the time the service is attempting to run.

Since, I have set the file permission earlier on, the access right to the file should be granted with executable file. Hence, it should be my path wrong.

Having my program saved at home directory, I missed out to include my full home directory path in the “WorkingDirectory” and “ExecStart”. Use “pwd” in the terminal, it is a command line to print the current working directory. Hence, it helps to get a correct path to allow the service to execute the program.

/home/myname/program/run.sh
##/home/program/run.sh

Being a SysOps Explorer

Into my second time writing about my experience doing some of the SysOps’ work. I found that I cannot use scp command to transfer a file from my Linux machine to the virtual machine.

scp command
scp (secure copy) is a command line utility that allows you securely copy files and directories between two locations, example from your local machine to a remote server or vice versa, or even two remote locations from your local machine.

Syntax:

scp [OPTION] [user@]SRC_HOST:]file1 [user@]DEST_HOST:]file2

• Option detail can refer to this link. Mainly, it is for ssh configuration, ssh port, recursive copy.
• SRC_HOST is the source file’s IP.
• DEST_HOST is the destination file’s IP.
  

You will use the scp command from your local machine. If you want to copy a file from your local machine to a remote server, the command can be something like,

scp file.txt remote_username@remote_server_ip:/remote/directory

remote_username@remove_server_ip's password:
file.txt                             100%    0     0.0KB/s   00:00

It requires the user to enter password of the remote server before the transfer process starts. How about the other way round, copying a file from remote server to local machine? Do this at your local machine,

scp remote_username@remote_server_ip:/remote/file.txt /local/directory

It will prompt user to enter password too. So, what if want to transfer from two remote locations? Run this in your local machine,

scp user1@host1.com:/files/file.txt user2@host2.com:/files

There is an useful guide here.

I tried my luck but failed to connect. It showed timed out and cannot connect to port 22. Port 22 is a default listening port for SSH on the remote server. Then, I sought help from my IT guy again and after a few troubleshooting, we realized that the virtual machine network was NAT. It has to be changed to use “Bridged”.

Some instant notes of network access option,

The above shows me why I cannot run the scp command from my local machine. For NAT, it allows access from guest machine (virtual machine) to external network. In other words, from the virtual machine it allows to go to Internet but no communications between host machine (local machine) and guest machine. Changing the network to Bridged networking allows bi-directional communication.

Then, I tried again and it worked. It was a great experience to know about the network setting in a virtual machine. Besides that, my IT guy also told me that I can start the virtual machine with headless state, which using less memory, whenever I did not need the interface.

SysOps By Trials and Errors

During a discussion with my big boss and two other colleagues, my big boss mentioned about the sales account manager of MongoDB did not come back to him for the MongoDB Enterprise’s quotation. There is a project may need to use the enterprise version due to some security reasons. He, then, suggested me to try out the Percona Server for MongoDB.

According to the Percona’s website, “Percona Server for MongoDB is our free and open-source drop-in replacement for MongoDB Community Edition. It offers all the features and benefits of MongoDB Community Edition, plus additional enterprise-grade functionality.”

Below is the features comparison made by Percona and available at their website.

Since I am curious too, I started to try on setting up a virtual machine on my Linux machine on Friday. My IT guy told me to use VBox (VirtualBox) and so I did. I used the installer to complete the installation. It can be done using command lines too.

sudo apt-get update
sudo apt-get install virtualbox-6.0

For VBox installation, you can refer to this link.

Just I was about to think of where I can download the image disk for Ubuntu, he sent me a message to inform me where I can get them. Alternatively, you can get it from this link.

The setup and installation of the Ubuntu 16.04 were all done in the VBox using the .vdi image. I did not recall there was any complications during the Ubuntu installation. It was straight-forward all the way.

Next, I needed to get the Percona Server installed. I registered to the Percona’s website to obtain a copy of the PDF document which contains the installation guide and features’ setups. It is an useful documentation for me to setup the server. On Monday, by trials and errors, I installed latest Percona Server, configured it and ran the service in the virtual machine.

There are plenty features share in the documentation with guide to implement them. I completed the configuration to use Percona Memory Engine for storage engine. I am not sure whether I configured it correctly especially the virtual machine is running on 1GB memory. I set the Percona memory to be running at 3GB. It is something I need to re-visit after this.

Besides that, I did try to enable the authorization mode. Immediately, after it was enabled, I tried to launch my company’s product using default authentication method and the system returned errors because there are some databases not authorized to be used. Authorization and authentication are different, even I have created an user credential in MongoDB to access those databases. It is also a good topic to re-visit.

It gave me an experience being a day or two as a system engineer, or most people called them as SysOps nowadays. Although, it was not a full cycle of SysOps, the experience of installation and setup Ubuntu in the virtual machine, followed by installation of Percona Server and Robo 3T for MongoDB and lastly configurations and using the Percona server with my company’s product was so great that I wanted to share it here, today.

I always have the special privilege being a woman in the industry to have men to do this dirty job, but when a woman tries her hand to work on it, it is a beautiful piece of art!

Great thanks to my colleagues who are willing to help me and guide me through this trials and errors. At least, I did it for myself once!

Unix: Shebang(#!)

It is called a shebang or a “bang” line. It is nothing but the absolute path to the Bash interpreter. It consists of a number sign and an exclamation point character (#!), followed by the full path to the interpreter such as /bin/bash. All scripts under Linux execute using the interpreter specified on a first line.

Actually, it is usually implemented as a symbolic link pointing to the executable for whichever shell is the system shell.

On Linux or other Unix-based systems, there is a choice of multiple shells. The shell is responsible for interpreting commands.

Bash is the most common shell used as a default shell for users in Linux systems. A script may specify #!/bin/bash on the first line, meaning that the script should always be run with bash, rather than another shell. /bin/sh is an executable representing the system shell.

Convert dos to Unix

For some of us may use Windows based editor to create the scripts that to be used in Linux server. There is a need to convert the dos file (created in Windows OS) to Unix file. The reason is files created in DOS/Windows use carriage return (\r) and line feed (\n) for line endings. However, files in Unix/Linux solely use line feed. When transferring a file from one system to another, make sure to convert the files.

Converting Files on Linux

There are several ways you can transfer files to use the appropriate line endings. From an Internet search, I found a few ways to convert the files:

• dos2unix command
• unix2dos command
• sed command – stream editor command to remove the carriage return line endings.
• tr command – The tr command is a command line utility for translating or deleting characters.
• Vi/Vim text editor – remove carriage return line endings from files in DOS format using the Vi/Vim text editor.
• perl one-liner command – Perl one-liner command to delete all \r line endings. Pearl on-liners are scripts that fit in a single line of code.

Based on the online resource that I found, when a Linux/Unix machine has installed the dos2unix tool, it calls the below command to convert a DOS file into Unix file.

dos2unix [file_name]

To convert a Unix file to DOS file, it uses below command:

unix2dos [file_name]

More commands and syntax for the other ways listed above can refer to the below link. Besides that, make sure the script is an executable file before start using it. To change the file permission, please check out the CHMOD command. I am not going to cover it in this blog for now, however, I have provided a link to check out the CHMOD syntax.

References

• https://phoenixnap.com/kb/convert-dos-to-unix
• https://www.computerhope.com/unix/uchmod.htm
• https://linuxize.com/post/chmod-command-in-linux/

Ubuntu: How to check your laptop is running on SSD or HDD?

Before I begin, the abbreviation of SSD is solid-state drive, and HDD is hard disk drive. What is the difference between the two is SSDs have no moving mechanical components, unlike the HDDs contain spinning disks, read/write heads. SSDs are more resistant to physical shock, run silently, have quicker access time and lower latency.

Anyway, this is not what I wish to write for today. I want to share a command I used just now to check the laptop that I am having is using SSD or HDD. Since the laptop is running on Ubuntu OS, instantly I searched for the command to put in the Terminal.

The command which I used was, 
cat /sys/block/sda/queue/rotational

If the output returns 1 then it means it is using HDD, if returns output 0 then it means it is using SSD.