Bash commands

Commands descriptions

  • ls [options] [path]“ - list directory contents
    • ls -lahF - long format, all files, human readable format, files indicators
  • file [path]“ - determine file type
  • man <command name>“ - access to manual pages
    • man -k <search term> - find required search term in man pages
    • press / to start search inside man page and n to search next result
  • Files manipulations
    • mkdir [options] <directory>“ - make directories
      • mkdir -p - create parent directories
      • mkdir -v - verbose output
    • rmdir [options] <directory>“ - remove empty directories. Flags similar to mkdir
    • touch [options] <filename>“ - change file timestamp. Can be used to create blank file
    • cp [options] <source> <destination>“ - copy file or directory
      • cp -r <source> <destination> - copy some folder
      • cp -n ... - do not overwrite existing file
    • mv [options] <source> <destination>“ - move a file or directory
    • rm [options] <file>“ - remove vile or directory
      • rm -r <directory> - remove non empty directory
      • rm -i <file> - interactive mode
      • rm -d <path> - remove empty directories
  • cat [files]“ - concatenate and print files
    • cat file1 file2 > out_file - concatenate two files
  • less <file>“ - display file with navigation. space - next page, b - previous page
  • chmod [permissions] [path]“ — change file modes or Access Control Lists
    • chmod g+x - add execute permissions to the group
    • chmod u-w - revoke read permissions for the user
    • chmod ug+rx - add read and execute permissions to the user and group
    • chmod 755 - all rights for user, read and execute only for group and other
    • chmod 644 - read/write for user, read only for group and other
  • Filters
    • head [-number of lines to print] [path]“ - display first lines of a file
    • tail [-number of lines to print] [path]“ - display the last part of a file
    • sort [-options] [path]“ - sort lines of text files
    • nl [-options] [path]“ - line numbering filter
      • nl -s '. ' -w 10 - add some formating
    • wc [-options] [path]“ - word, line, character, and byte count
    • cut [-options] [path]“ - cut out selected portions of each line of a file
      • cut -f 1 -d ' ' - get first column from the file, if columns separated by spaces.
      • cut -f 1,2 -d ' ' - get first and second columns
    • sed <expression> [path]“ - stream editor. basic expression - s/search/replace/g
      • sed 's/oranges/bananas/g' mysampledata.txt - replace oranges with bananas
    • uniq [options] [path] - report or filter out repeated lines in a file
    • tac [path] - concatenate and print files in reverse (reversed cat).
  • diff [options] <files> - compare two files
  • top - display and update sorted information about processes
  • ps - process status
  • date - display current date.
    • date +%F - display in format year-month-day
  • xargs - construct argument list(s) and execute utility
  • find - walk a file hierarchy
    • find . -type f - find all files
    • find . -type d - find all directories
    • find . -size +200M -exec ls -sh {} \ - find and display all files more than 200M
    • find . -mtime -1 - find all recently changed files
  • tar - manipulate tape archives
    • tar -zcvf mytar.tar.gz * - add all files in folder to the archive
    • tar -zxvf mytar.tar.gz - extract files from archive
  • awk ‘pattern { action }’ - pattern-directed scanning and processing language

Wildcards

  • * - represents zero or more characters
  • ? - represents a single character
  • [] - represents a range of characters
    • ls [sv]* - looking for a files begins with s or v
    • ls *[0-9]* - set by using a hyphen
    • ls [^a-z]* - caret means not

Permissions

For files: - r read - you may view the contents of the file. - w write - you may change the contents of the file. - x execute - you may execute or run the file if it is a program or script.

For directories: - r - you have the ability to read the contents of the directory (ie do an ls) - w - you have the ability to write into the directory (ie create files and directories) - x - you have the ability to enter that directory (ie cd)

Set for:

  • u user
  • g group
  • o other
  • a all

In case octal mask using, each number represent [ugo].

grep/egrep

  • . (dot) - a single character.
  • ? - the preceding character matches 0 or 1 times only.
  • * - the preceding character matches 0 or more times.
  • + - the preceding character matches 1 or more times.
  • {n} - the preceding character matches exactly n times.
  • {n,m} - the preceding character matches at least n times and not more than m times.
  • [agd] - the character is one of those included within the square brackets.
  • [^agd] - the character is not one of those included within the square brackets.
  • [c-f] - the dash within the square brackets operates as a range. In this case it means either the letters c, d, e or f.
  • () - allows us to group several characters to behave as one.
  • | (pipe symbol) - the logical OR operation.
  • ^ - matches the beginning of the line.
  • $ - matches the end of the line.

flags:

  • -n - display line number as well at the output
  • -c - display quantity of matched lines

Piping and redirection

  • > - STDOUT to a file. cat filename > out
  • < - STDIN from a file. wc -l < filename
  • STDIN and STDOUT. wc -l < barry.txt > myoutput
  • 2> - redirect STDERR
  • ls -l > out 2>&1 redirect STDERR to STDOUT and STDOUT to a file
  • | - pipe. Output of program to the next one. Ex: ls | head -3 | tail -1
  • save the output from the command to the variable - lines=\`cat $1 | wc -l\`

Foreground and Background Jobs

  • program_name & - run program in background
  • ctrl+z - pause the current foreground process and move it into the background
  • jobs - show background jobs
  • fg <job number> - return job to foreground

bash scripting

  • assign variable var_name=value
  • refer a variable $var_name
  • $0 - The name of the script.
  • $1 - $9 - Any command line arguments given to the script. $1 is the first argument, $2 the second and so on.
  • $# - How many command line arguments were given to the script.
  • $* - All of the command line arguments.

if statements:

if [ <some test> ]
then
  <commands>
else
  <another>
fi

while loop:

while [ <some test> ]
do
  <commands>
done

for loops:

# for value in {1..5}
for var in <list>
do
  <commands>
done

By category

Create ssh key

ssh-keygen -t rsa -b 4096 -f ~/.ssh/key_name
cat ~/.ssh/key_name.pub | xclip -sel clip

Tmux simple example

When you close ssh session, scripts that were called from ssh user may be closed. To handle this situation you may run scripts under the sudo. Or use tmux. What we should do:

  • Connect to VPN, required server ssh username@host_ip
  • Install tmux - sudo apt-get install tmux
  • Open new tmux session tmux new -s session_name
  • Run desired script.
  • Detach session with hotkey ctrl+b ++ d (Means press ctrl + b first and after the d)
  • You may reconnect at this point to the server
  • List all tmux session tmux ls
  • Connect to chosen session with tmux a -t session_name
  • Kill session from itself if not required any more ctrl+b ++ x

Additional notes:

  • In case of mouse scrolling not works - inside tmux type tmux set-option -g mouse on
  • If you want to be sure that tmux session will not be stopped - you may open new window under the sudo sudo tmux new -s window_name and after inside change the user su - username
  • Here exist quite full cheat sheet for tmux
  • Copy from tmux screen can be some way inconvinient - so it’s better to store output in some file: ./script_name | tee -a logs.txt
  • In case you want to copy something - you may just highlight by mouse required region, and to past press ctrl+b ++ ]

Tmux is very powerful tool with many other capabilities. For example you may work in one session with your team.

Another hints

ubuntu-bash-hints.sh (Source)

# show running process by name with headers
$ ps aux | egrep "python|PID"

### la
# sizes of files
$ ls -lah
# colored
$ ls -G
# count quantity of files inside folder
$ ls -1 | wc -l

### sizes
# check free disk space
$ df -h
# get folder size
$ du -hs
# get folders size inside another one
$ du -h --max-depth=1 some_folder

### search
# search recursive by folders
$ grep -r "/api/Profile/active_home_devices" static/
# search by file content
$ grep -R 'GCModel' SemanticProcessor/
# find by filename
$ find ~ -name readme.txt

### zipping/unzipping
# unzip tar.gz file
$ tar -xvzf filename.tar.gz
# zip something
$ zip archive_name source_name
# unzip something
$ unzip archive_name

# execute output of some command(assume it in text file)
$ $(cat filename)

# get the location of the executable link
$ readlink -f /usr/bin/java
/usr/lib/jvm/java-8-oracle/jre/bin/java

# display part of the file
$ cat filename | less
$ tail -200 filename
$ head -200 filename
$ head -n 1000 filename | tail
# display to stdout
$ echo SOMETHING

# get list of all listened ports
$ sudo netstat -peanut

# check memory every 1 second
$ watch -n 1 free -m

# as per tutorial: http://manpages.ubuntu.com/manpages/trusty/man1/enca.1.html
# get coding of document
$ enca filename
# change encoding of doc
$ enconv filename

# merge all files in folder to one
$ cat folder_path/* > new_file

# copy content of some text file(ssh for example)
$ cat ~/.ssh/id_rsa.pub | xclip -sel clip
# the same for mac
$ pbcopy < ~/.ssh/id_rsa.pub

# get source code of some package
http://askubuntu.com/questions/167468/where-can-i-find-the-source-code-of-ubuntu

# get dependencies
$ readelf -d some_exe

# find location of installed files
$ dpkg --listfiles libqt4-dev

# remove range of folders starts with numbers
$ rm -r output/logs/{17..26}*

# send stdout to file and display it to the bash at the same time
$ ./some_file.sh | tee -a logs.txt

# stop first running docker instance
# all runing| second line| container name    | stop container by name
$ docker ps | sed -n 2p  | awk '{print $NF}' | xargs docker stop

# get version of CUDA
$ dpkg -l | grep cuda

# rename some folders by pattern
# find required folders  | split folder by '/' or '_' |
# based on split generate new name | call `mv` command with `system ()` flag
# note that splaces added in quotes
find . -type d -name "celeb*" |  awk -F '/|_' '{system ("mv " $0 " " $2"_mobile_"$3)}'