missing semester in computer science lecture2

missing semester in CS

Shell Tools and Scripting.Part

This lecture shows us how to use bash as a scripting language along with some shell tools.

Shell Scripting

To assign variables in bash, we use the syntax foo=bar.We should notice that we can not usefoo = bar since it is interpreted as calling the foo program with the arguments = and bar.

Here are the examples.

(base) myfile@zhaoxuanhaodeMacBook-Pro ~ % foo=bar
(base) myfile@zhaoxuanhaodeMacBook-Pro ~ % echo "$foo"
bar
# prints bar
(base) myfile@zhaoxuanhaodeMacBook-Pro ~ % echo '$foo'
$foo
# prints $foo

bash supports control flow techniques including if cas while and for.

mcd(){
	mkdir -p "$1"
	cd "$1"
}

Here $1 is the first argument to the script/function.

$0 # Name of the script
$1 to $9 # Arguments to the script.$1 is the first argument and so on
$@ All the arguments
$# Number of arguments
$? Return code of the previous command
$$ Process identification number of the current script
!! - Entire last command, including arguments. A common pattern is to execute a command only for it to fail due to missing permissions; you can quickly re-execute the command with sudo by doing sudo !!
$_ - Last argument from the last command. If you are in an interactive shell, you can also quickly get this value by typing Esc followed by . or Alt+.

Exit codes can be used to conditionally execute commands using && and ||.

The true program will always have a 0 return code and the false command will always have a 1 return code.Here are some examples:

false || echo "Oops, fail"
# Oops, fail

true || echo "Will not be printed"
#

true && echo "Things went well"
# Things went well

false && echo "Will not be printed"
#

true ; echo "This will always run"
# This will always run

false ; echo "This will always run"
# This will always run

And here are some other syntax:

convert image.{png,jpg}
# Will expand to
convert image.png image.jpg

cp /path/to/project/{foo,bar,baz}.sh /newpath
# Will expand to
cp /path/to/project/foo.sh /path/to/project/bar.sh /path/to/project/baz.sh /newpath

# Globbing techniques can also be combined
mv *{.py,.sh} folder
# Will move all *.py and *.sh files


mkdir foo bar
# This creates files foo/a, foo/b, ... foo/h, bar/a, bar/b, ... bar/h
touch {foo,bar}/{a..h}
touch foo/x bar/y
# Show differences between files in foo and bar
diff <(ls foo) <(ls bar)
# Outputs
# < x
# ---
# > y

Note that scripts need not necessarily be written in bash to be called from the terminal. For instance, here’s a simple Python script that outputs its arguments in reversed order:

# !/usr/local/bin/python (shebang line)
import sys
for arg in reversed(sys.argv[1:]):
    print(arg)

Now the kernel knows to execute it with a python interpreter because of the shebang line.

The differences between the shell functions and scripts are:

Functions have to be in the same language as the shell, while scripts can be written in any language. This is why including a shebang for scripts is important.
Functions are loaded once when their definition is read. Scripts are loaded every time they are executed. This makes functions slightly faster to load, but whenever you change them you will have to reload their definition.
Functions are executed in the current shell environment whereas scripts execute in their own process. Thus, functions can modify environment variables, e.g. change your current directory, whereas scripts can’t. Scripts will be passed by value environment variables that have been exported using export
As with any programming language, functions are a powerful construct to achieve modularity, code reuse, and clarity of shell code. Often shell scripts will include their own function definitions.

Shell Tools

finding files

All UNIX-like systems come packaged with find, a great shell tool to fine files.find will recursively search for files matching some criteria.

# Find all directories named src
find . -name src -type d
# Find all python files that have a folder named test in their path
find . -path '*/test/*.py' -type f
# Find all files modified in the last day
find . -mtime -1
# Find all zip files with size in range 500k to 10M
find . -size +500k -size -10M -name '*.tar.gz'



# be careful when using syntaxs below:

# Delete all files with .tmp extension
find . -name '*.tmp' -exec rm {} \;
# Find all PNG files and convert them to JPG
find . -name '*.png' -exec convert {} {}.jpg \;

Finding code

For now, know that grep has many flags that make it a very versatile tool. Some I frequently use are -C for getting Context around the matching line and -v for inverting the match, i.e. print all lines that do not match the pattern. For example, grep -C 5 will print 5 lines before and after the match. When it comes to quickly searching through many files, you want to use -R since it will Recursively go into directories and look for files for the matching string.

# Find all python files where I used the requests library
rg -t py 'import requests'
# Find all files (including hidden files) without a shebang line
rg -u --files-without-match "^#!"
# Find all matches of foo and print the following 5 lines
rg foo -A 5
# Print statistics of matches (# of matched lines and files )
rg --stats PATTERN