Table Of Contents

This Page

A Brief Introduction to UNIX

Preface

  • Words in UNIX are case sensitive - foo, Foo, & FOO are 3 different words (this used not to be the case on OSX - still?)
  • Root is a special user, the superuser who has administrative powers
  • Users have fewer privileges than those of root
  • Every file, directory, and executable programme has permissions - what you as a user can read/write/execute depends on your user permissions for the file
  • For help on commands try $ man <command> or $ <command> --help (or a variation thereof -h --help)
  • The most important key is $ <TAB> (autocomplete)
  • By default, if a programme executes without error nothing is printed to the screen, and a new prompt is returned

The Shell

A shell or command line interpreter provides a command line interface to the UNIX operating system. This document assumes you are using the bash shell. Bash is the default shell in OSX and most flavours of Linux. If you are unsure which shell is being used, issue the command:

[cymon@gyra temp]$ echo $SHELL
/bin/bash

Recommended reading: Newham, C. & Rosenblatt, B. 2005. Learning the bash shell (3rd edn.). O’Reilly, Cambridge.

Commands:

[cymon@spiro ~]$ sort -n phonelist > phonelist.sorted
.
command optional argument positional argument I/O redirect
sort -n phonelist > phonelist.sorted

Issuing Commands

Some common commands

  • $ ls  <path> → list contents
  • $ cd  <path> → change directory
  • $ cp <path> <path> → copy path to path
  • $ pwd → present working directory
  • $ mkdir <path> → make directory
  • $ mv <path> <path> → move path
  • $ rm <path> → remove path
  • $ rmdir <path to dir> → remove directory
  • $ who → who is logged in
  • $ id → who am I
  • $ env → list environment
  • $ touch  <path> → create empty file
  • $ cat <file>[<file>] → catenate file(s)
  • $ which <command> → path to first occurence of <command> in $PATH
  • $ locate <command> → find something
  • $ chmod <permissions> <path> → change permission of <path>
  • $ chgrp <group> <path> → change group of <path>
  • $ file <path> → determine file type
  • $ find <path> <expression> → search for files in a directory hierarchy
  • $ sed {expression} <input file> → stream editor for filtering and transforming text
  • $ awk <pattern> {action} → pattern scanning and text processing language
  • $ exit → exit shell

Wildcards

wildcard matches
? any single character
* Any string of characters
[*set*] Any character in set
[!*set] Any character not in set

Examples:

[cymon@spiro temp]$ ls
file4  file44.pdf  file5.pdf  file5.text  file6
[cymon@spiro temp]$ ls file?.pdf
file5.pdf
[cymon@spiro temp]$ ls file??.pdf
file44.pdf
[cymon@spiro temp]$ ls file?.*
file5.pdf  file5.text
[cymon@spiro temp]$ ls file[4-6]
file4  file6
[cymon@spiro temp]$ ls file[!4-5]*
file6
[cymon@spiro temp]$ ls file[a-z].*
ls: cannot access file[a-z].*: No such file or directory

Expansions

[cymon@spiro temp]$ ls *.{pdf,text}
file44.pdf  file5.pdf  file5.text
[cymon@spiro dirongyra]$ echo b{ed,olt,ar}s
beds bolts bars
[cymon@spiro dirongyra]$ echo b{ar{d,n,k},ed}s
bards barns barks beds

Autocomplete

Everytime you press <TAB> the shell will attempt to guess what you want to write, i.e., it tries to autocomplete your command.

If there is more that one possible completion of the command, a second <TAB> will list all the possible completions.

Examples:

[cymon@spiro ~]$ cd /h<TAB>ome/c<TAB>ymon/w<TAB>ork/

In each case after entering h, c, and w there was only one possible completion (there are no other directories in /home that begin with c apart from cymon and there is only one directory in /home/cymon that begins with a w), so the ome, ymon and ork were complete automatically.

[cymon@spiro ~]$ ls -l /usr/l<TAB><TAB>

This time there is more than one possible completion:

lib/   lib32/ lib64/ local/
[cymon@spiro ~]$ ls -l /usr/li<TAB>b<TAB>
lib/   lib32/ lib64/

If you enter and i plus <TAB it will automatically extend with a b but no further because there are 3 possible completions of lib. If you then enter a / and then <TAB this happens:

[cymon@spiro ~]$ ls -l /usr/lib/<TAB>
Display all 2334 possibilities? (y or n)

So there are 2334 files/directories with /usr/lib - at this stage you will probably want to enter n unless you want to list all 2334.

Possible completions depend on the command you are trying to execute, but note that is not always so clever. Here we try and cd (change directory) to a file, we enter t<TAB> and it autocompletes to touch_brain a file:

[cymon@spiro ~]$ ls -l t<TAB>ouch_brain
-rw-rw-r-- 1 cymon cymon 1416 2011-03-24 16:44 touch_brain
[cymon@spiro ~]$ cd touch_brain
-bash: cd: touch_brain: Not a directory

However, if the name you are trying to complete is an executable, only executable files will be listed - moreover, only executables that are in your $PATH (see below)

[cymon@spiro unix-course]$ un<TAB>
unalias               unattended-upgrades   unexpand              unicode_stop          unique_list_items.py  unlink                unpack200             unzip
uname                 uncompress            unflatten             uniconvertor          unix_chkpwd           unlzma                unset                 unzipsfx
unattended-upgrade    unconfined            unicode_start         uniq                  unix_update           unopkg                until
[cymon@spiro unix-course]$ una<TAB>
unalias              uname                unattended-upgrade   unattended-upgrades
[cymon@spiro unix-course]$ uname

History

The shell will record your commands. To use a previous command use the ↑ key.

[cymon@spiro temp]$ history
60  ls
61  ll
62  cd ..
<and 997 others>

[cymon@spiro temp]$ echo bang!
bang!
[cymon@spiro temp]$ history
<skip 998 entries>
1063  echo bang!
1064  history
[cymon@spiro temp]$

press up arrow ↑ twice at prompt

[cymon@spiro temp]$ echo bang!
bang!
[cymon@spiro temp]$ history
<skip 996 entries>
1063  echo bang!
1064  history
1065  echo bang!
1066  history
[cymon@spiro temp]$

Control keys

Control keys - those that you type holding down both the CONTROL (CTRL) and hitting a second key - also have special effects in the shell.

Control Key Notation Description
CTRL-C ^C stop current command
CTRL-D ^D end of input
CTRL-Z ^Z suspend current command

Filesystem structure

May vary between Linux/OSX/BSD and between flavours of Linux.

Common directories on Linux:

System directories - only used by root and priviliged users - users may be able to execute or read programmes in these directories but will not be able to write to or alter files here:

  • / - root directory

  • /root - root’s home directory

  • /proc - system files

  • /sbin - superuser executable programs and utilities

  • /bin - binary executable programs that all users need

  • /etc - configuration files

  • /tmp - temporary work files

  • /dev - device files that control drives

  • /var - user specific files

  • /opt - other options

  • /usr - pronounced ‘user’ and contains unix commands and utilities
    • /bin - binary executable programs
    • /lib - program libraries
    • /sbin - more executable programs and Linux utilities for administrative purposes
    • /doc - documentation
    • /src - source code to programs

User directories - users can use this space for whatever purpose:

  • /home/<user name> - the users home directory
  • /usr/local - general purpose area, possible access depending on configuration

In addition there maybe any number of additional directories below the root directory, e.g. boot, CA, dev, export, lib64, media, misc, mnt, selinux, share, srv, state, sys, tftpboot, tmp...


Permissions

The level of the user’s privilege determines what each user can do with a file.

Every file on your Linux system, including directories, is owned by a specific user and group. Therefore, file permissions are defined separately for users, groups, and others.

Ownership

User:
The username of the person who owns the file, typically the user who created it.
Group:
The usergroup that owns the file. All users beloning to the group have the same permissions.
Other:
A user who is not the owner of the file and doesn’t belong in the same group the file does, i.e. everyone else.

Changing ownership

Only the root / superuser can change ownership of a file. But if you are the owner of a file you can change the group ownership:

[cymon@gyra temp]$ ls -l
total 44
drwxr-xr-- 2 cymon biouser  4096 Apr 15 12:58 adirectory
-rwxrw-r-- 1 cymon litol    9393 Apr 15 12:58 afile
-rwxr-xr-- 1 cymon wheel   25270 Apr 15 12:58 anotherfile
[cymon@gyra temp]$ chgrp biouser afile
[cymon@gyra temp]$ ls -l afile
-rwxrw-r-- 1 cymon biouser 9393 Apr 15 12:58 afile

Types of permission

Read:
  • On a file → open and read
  • On a directory → list the contents or the directory
Write:
  • On a file → modify
  • On a directory → add, remove, and rename files in the directory
Execute:
  • On a file → execute the file
  • On a directory → access files in the directory

Some confusions:

Even though a user has write permission on a file (ie its contents can be modified), the user can only reaname or delete the file if the user has the write permission on the directory in which the file is contained.

The execute bit allows the user to enter the directory, but the contents can only be listed if the user has read permissions to the directory.

Viewing permissions

[cymon@gyra temp]$ ls -l
total 44
drwxr-xr-- 2 cymon biouser  4096 Apr 15 12:58 adirectory
-rw-r--r-- 1 cymon litol    9393 Apr 15 12:58 afile
-rw------- 1 cymon wheel   25270 Apr 15 12:58 anotherfile

Explanation:

permissions links user group size date name
drwxr-xr-- 2 cymon biouser 4096 Apr 15 12:58 adirectory
-rw-r--r-- 1 cymon litol 9393 Apr 15 12:58 afile
-rw------- 1 cymon wheel 25270 Apr 15 12:58 anotherfile

Permissions:

  • r can read
  • w can write
  • x can execute
  • - no permission

type can be:

  • d = directory
  • - = regular file
  • l = symbolic link
type user group others
d rwx r-x r--

Changing permissions

Symbolically:

Who:

  • u user/owner
  • g group
  • o other
  • a all

Change how:

  • + add this permission
  • - remove this permission
  • = set exactly this permission

What permission:

  • r read
  • w write
  • x execute

Examples:

[cymon@gyra temp]$ ls -l
total 44
drwxr-xr-- 2 cymon biouser  4096 Apr 15 12:58 adirectory
-rw-r--r-- 1 cymon litol    9393 Apr 15 12:58 afile
-rw------- 1 cymon wheel   25270 Apr 15 12:58 anotherfile
[cymon@gyra temp]$ chmod g+r anotherfile
[cymon@gyra temp]$ ls -l anotherfile
-rw-r----- 1 cymon wheel 25270 Apr 15 12:58 anotherfile
[cymon@gyra temp]$ chmod o+r anotherfile
[cymon@gyra temp]$ ls -l anotherfile
-rw-r--r-- 1 cymon wheel 25270 Apr 15 12:58 anotherfile
[cymon@gyra temp]$ chmod a+x anotherfile
[cymon@gyra temp]$ ls -l anotherfile
-rwxr-xr-x 1 cymon wheel 25270 Apr 15 12:58 anotherfile
[cymon@gyra temp]$ chmod o-x anotherfile
[cymon@gyra temp]$ ls -l anotherfile
-rwxr-xr-- 1 cymon wheel 25270 Apr 15 12:58 anotherfile

Numerically:

Values:

  • 4 = read (r)
  • 2 = write (w)
  • 1 = execute (x)
  • 0 = no permission (-)

Examples:

[cymon@gyra temp]$ ls -l
total 44
drwxr-xr-- 2 cymon biouser  4096 Apr 15 12:58 adirectory
-rw-r--r-- 1 cymon litol    9393 Apr 15 12:58 afile
-rw------- 1 cymon wheel   25270 Apr 15 12:58 anotherfile

To set permission to -rw-r-----: user is rw- 4+2+0=6; group is r-- 4+0+0=4; others is --- 0+0+0=0 => numerical value 640

[cymon@gyra temp]$ chmod 640 anotherfile
[cymon@gyra temp]$ ls -l anotherfile
-rw-r----- 1 cymon wheel 25270 Apr 15 12:58 anotherfile

To set permissions to -rwxr-xr--: user is rwx 4+2+1=7; group is r-x 4+0+1=5; others is r-- 4+0+0=4 => numerical code 754

[cymon@gyra temp]$ chmod 754 anotherfile
[cymon@gyra temp]$ ls -l anotherfile
-rwxr-xr-- 1 cymon wheel 25270 Apr 15 12:58 anotherfile

Profile, environment variables, and $PATH

Profile

Your profile defines your starting shell enviromnent when you login to your account. Some of your environment will be set as default by the system adminsistrator, while customisations can be made by editing the file ~/.bash_profile which is read on login.

Only edit this file if you know what you are doing - you could render your account effectively unusable

Environment variables

The UNIX environment defines a number of variables. Some of these are set by default to the operating system and others can be set by the user or when executing software.

To see a list of currently set environment variables, issue the command:

[cymon@gyra temp]$ env
BIOROLL=/opt/bio
HOSTNAME=gyra.ualg.pt
TERM=xterm
SHELL=/bin/bash
WISECONFIGDIR=/usr/local/src/wise2.2.0/wisecfg
HISTSIZE=1000
(and many others)

To access the value of a variable, prepend the variable name with a dollar sign:

[cymon@gyra temp]$ echo $HOSTNAME
gyra.ualg.pt

The most important enviroment variable is PATH.

[cymon@spiro ~]$ echo $PATH
/home/cymon/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/lib:/usr/X11R6/bin:.

The PATH variable determines where in the filesystem the OS looks for an executable programme. When you issue a command e.g. echo the OS searches, in order, the directories in your PATH until if find an executable call echo and then executes it. If it fails to find an executable of that name it will return a command not found.

Example:

[cymon@spiro ~]$ uname --help
Usage: uname [OPTION]...
Print certain system information.  With no OPTION, same as -s.
... etc
[cymon@spiro ~]$ locate uname
/bin/uname
/usr/lib/klibc/bin/uname
[cymon@spiro ~]$ which uname
/bin/uname

So, although there two executable programmes called uname, if a full path to the version required is not given, the PATH will be searched for uname and the version in /bin executed.

Should you want to execute the version of uname in /usr/lib/klibc you would need to specify the full path to overide the default PATH:

[cymon@spiro ~]$ /usr/lib/klibc/bin/uname

Defining enviroment variables

$ export VARNAME=new value

Example:

[cymon@gyra temp]$ export CHEESE=cheddar
[cymon@gyra temp]$ echo $CHEESE
cheddar

Be careful not to over-write important system default variables:

[cymon@gyra temp]$ export PATH=
[cymon@gyra temp]$ echo $PATH

[cymon@gyra temp]$ ls
-bash: ls: No such file or directory

Aliases

Some command aliases are system specified, others can be defined by the user. If you use a particular command often, consider defining simple alias for it (be sure not to an alias that is the name of an already defined executable):

[cymon@spiro ~]$ alias
alias cp='cp -r'
alias ev='evince'
alias grep='grep --color -n'
alias l1='ls -1'
alias la='ls -alh'
alias ll='ls -thor'
alias ls='ls --color=auto'
alias lsd='ls -d */'
alias of='ooffice'
alias scp='scp -r'
alias sv='seaview'
alias tw='texworks'
alias vi='vim'
[cymon@spiro ~]$ alias gyra="ssh gyra.ualg.pt"
[cymon@spiro ~]$ gyra
Last login: Sat Apr 16 11:43:02 2011 from 10.10.40.121
Rocks 5.3 (Rolled Tacos)
Profile built 16:35 28-Oct-2010

Kickstarted 17:57 28-Oct-2010
[cymon@gyra ~]$

Custom aliases are traditionally defined in ~/.bashrc which is read at login.


Input and Output

Standard I/O

Each program that is invoked has three standard I/0 channels set to the terminal:

  1. standard input is form the keyboard
  2. standard output to the screen
  3. standard error to the screen

Standard input and output can come from a file and be redirected to a file, respectively, as necessary. Moreover, you can pipe the standard output of one program into the standard input of another makeing a pipline.

Example:

The program cat invoked without arguments simply copies standard input to standard output:

[cymon@spiro temp]$ cat
**This is me typing into standard input...**
This is me typing into standard input... <printed on standard output>
^D
[cymon@spiro temp]$

cat invoked with a single filename will output the file contents on standard out i.e. the screen:

[cymon@spiro temp]$ cat file1
1. These are the contents of file 1

Invoked with 2 or more filenames will concatenate all contents of the files (in order) and print them on standard out:

[cymon@spiro temp]$ cat file1 file2
1. These are the contents of file 1
2. These are the contents of file 2!!

I/O redirection

The standard output stream may then be re-directed to another file:

[cymon@spiro temp]$ cat file1 file2 > file3
[cymon@spiro temp]$ cat file3
1. These are the contents of file 1
2. These are the contents of file 2!!

The general form of redirection to and from files is:

command < infile > outfile

You can also append to a file:

[cymon@spiro temp]$ echo "adding this to the bottom of file3" >> file3
[cymon@spiro temp]$ cat file3
1. These are the contents of file 1
2. These are the contents of file 2!!
adding this to the bottom of file3

The standard output stream can also be redirect to the standard input of a different program using a pipe sybolised by |:

[cymon@spiro unix-course]$ seq 1 4
1
2
3
4
[cymon@spiro unix-course]$ seq 1 4 |sort -nr
4
3
2
1

The command tac (the reverse of cat) prints standard input in reverse order of lines:

[cymon@spiro temp]$ cat file3
1. These are the contents of file 1
2. These are the contents of file 2!!
adding this to the bottom of file3
[cymon@spiro temp]$ cat file3 | tac
adding this to the bottom of file3
2. These are the contents of file 2!!
1. These are the contents of file 1

Chaining pipes into a pipline - this pipline of 9 commands prints the top 10 most used commands of my last 1000 commands recorded in history:

[cymon@spiro ~]$ history | awk '{print $2}' | awk 'BEGIN {FS="|"}{print $1}' | sort | uniq -c | sort -n | tail | sort -nr
241 ls
222 cd
 87 sphinx-build
 71 vi
 46 ll
 25 rm
 25 du
 23 exit
 14 ssh
 14 java


Copying files to an from a UNIX machine

(If you are using a Microsoft Window OS see these instructions.)

Copy to using scp (secure copy)

Copy the file touch_brain to my /home directory ~ on gyra.ualg.pt:

[cymon@spiro ~]$ scp touch_brain gyra.ualg.pt:~
touch_brain                                             100% 1416     1.4KB/s   00:00
[cymon@spiro ~]$

To copy entire directories, use the -r (recursive) flag:

[cymon@spiro temp]$ ls -al mydirectory
total 8
drwxrwxr-x 2 cymon cymon 4096 2011-04-16 12:08 .
drwxr-xr-x 3 cymon cymon 4096 2011-04-16 12:08 ..
-rw-rw-r-- 1 cymon cymon    0 2011-04-16 12:08 file1
-rw-rw-r-- 1 cymon cymon    0 2011-04-16 12:08 file2
-rw-rw-r-- 1 cymon cymon    0 2011-04-16 12:08 file3
[cymon@spiro temp]$ scp -r mydirectory gyra.ualg.pt:~/work
file2                                                                               100%    0     0.0KB/s   00:00
file1                                                                               100%    0     0.0KB/s   00:00
file3                                                                               100%    0     0.0K
[cymon@spiro temp]
[cymon@spiro temp]$ ssh gyra.ualg.pt
[cymon@gyra ~]$ ls -l  work/mydirectory
total 0
-rw-r--r-- 1 cymon wheel 0 Apr 16 12:12 file1
-rw-r--r-- 1 cymon wheel 0 Apr 16 12:12 file2
-rw-r--r-- 1 cymon wheel 0 Apr 16 12:12 file3

Copy from using scp

[cymon@gyra work]$ ls -l dirongyra
total 0
-rw-r--r-- 1 cymon wheel 0 Apr 16 12:18 file4
-rw-r--r-- 1 cymon wheel 0 Apr 16 12:18 file5
-rw-r--r-- 1 cymon wheel 0 Apr 16 12:18 file6
[cymon@spiro temp]$ scp gyra.ualg.pt:~/work/dirongyra/file4 .
file4                                                                               100%    0     0.0KB/s   00:00
[cymon@spiro temp]$ ls -l
total 0
-rw-r--r-- 1 cymon cymon 0 2011-04-16 12:16 file4
[cymon@spiro temp]$ scp -r gyra.ualg.pt:~/work/dirongyra .
file4                                                                               100%    0     0.0KB/s   00:00
file5                                                                               100%    0     0.0KB/s   00:00
file6                                                                               100%    0     0.0KB/s   00:00
[cymon@spiro temp]$ ls -l dirongyra
total 0
-rw-r--r-- 1 cymon cymon 0 2011-04-16 12:17 file4
-rw-r--r-- 1 cymon cymon 0 2011-04-16 12:17 file5
-rw-r--r-- 1 cymon cymon 0 2011-04-16 12:17 file6
[cymon@spiro temp]$

If copying to or from an account with a different user name to the machine on which the scp command is issued, you will need to prepend the machine name with the name of the user account, e.g.

[cymon@spiro temp]$ scp bob@gyra.ualg.pt:~/work/dirongyra/file4 .

rsync

If you need to transfer large amounts of data regularly, you should consider using rsync which synchronizes files and directories from one location to another while minimising the amount of data transferred - i.e. it mirrors data between locations. It’s a very powerful tool - see man rsync for details.

Line-endings

Characters defining line-endings and carrige-returns in Windows and Unix text files differ. The Window transfer software should convert to Unix line-endings, but, for whatever reason, this still occurs occasionally.

The line terminator expected for each file format is:

unix LF only (each line ends with an LF character)
dos CRLF (each line ends with two characters, CR then LF)
mac CR only (each line ends with a CR character)
CR is carriage return (return cursor to left margin), which is Ctrl-M or ^M or hex 0D.
LF is linefeed (move cursor down), which is Ctrl-J or ^J or hex 0A. Sometimes, LF is written as NL (newline).
Mac OS version 9 and earlier use mac line endings, while Mac OS X and later use unix line endings.
[cymon@spiro ~]$ file windows-file.text
windows-file.text: ASCII text, with CRLF line terminators
[cymon@spiro ~]$ dos2unix windows-file.text
windows-file.text
[cymon@spiro ~]$ file windows-file.text
windows-file.text: ASCII text

windows-file.text now has unix line-endings


Text editors

There are two big UNIX text editors: vi(m) and emacs. Both require some effort to learn to use efficiently, but if you are going to use UNIX, it’s well worth the effort.

If you just want to get something written occassionally, use nano - it’s self-explanatory.