当前位置:网站首页>MIT lecture on Computing Science shell tools and scripts

MIT lecture on Computing Science shell tools and scripts

2020-12-08 08:14:33 osc_ 0atwpkmx

In this lecture , We will introduce the use of bash As a basic knowledge of scripting language , As well as some Shell Tools , These tools cover some of the most common tasks that you will perform continuously on the command line .

Shell Script

up to now , We have seen how in Shell Execute commands in and pipe them together . however , in many instances , You will need to execute a series of commands and control flow expressions such as conditions or loops .

Shell Scripting is the next step in complexity . Most shells have their own scripting language , It contains variables , Control flow and its syntax . send Shell The reason why scripts are different from other scripting languages is that , It is optimized to perform with Shell Related tasks . therefore , Create a command pipeline , Saving results to a file and reading from standard input are Shell Primitives in scripts , It's easier to use than a common scripting language . In this section , We will focus on bash Script , Because it's the most common script .

To be in bash Distribution variables in , Please use grammar foo=bar And by accessing the value of the variable $foo. Please note that , this foo = bar Will not work , Because it's interpreted as foo Using parameter = And the calling program bar. Usually , stay shell Script , The space character will perform parameter splitting . At the beginning of use , This behavior can be confusing , So be sure to check .

bash You can use the ' and " Separator to define , But they're not equivalent . Separated by ' A string is a literal string , Does not replace variable values , And then " Separate strings instead of .

 
foo=bar
echo "$foo"
# prints bar
echo '$foo'
# prints $foo

Like most programming languages ,bash Support flow control technology , Include if,case,while and for. Again ,bash Functions that take parameters and operate on them . This is a create directory and put it cd Examples of functions placed in the directory .

mcd () {
    mkdir -p "$1"
    cd "$1"
}

 

this $1 Is the script / First argument to function . Unlike other scripting languages ,bash Use various special variables to reference parameters , Error codes and other related variables . Here are some of the lists . stay here You can find a more comprehensive list .

  • $0 - Script name
  • $1 To $9- Script variables .$1 Is the first parameter , And so on .
  • $@ - All the arguments
  • $# - The number of arguments
  • $? - The return code of the previous command
  • $$ - The process ID of the current script (PID)
  • !!- The whole last command , Including the parameters . A common pattern is to execute commands only so that they fail due to lack of permissions . You can use the sudo Quickly re execute the command sudo !!
  • $_- The last parameter from the last command . If you're in an interactive shell, You can also quickly type this value Esc, then .

 

Commands usually use STDOUT, Erroneous passage STDERR And return code to return the output , To report errors in a more script friendly way . Return code or exit status is a script / The way orders convey the way they are executed . value 0 It usually means that everything is normal . differ 0 Any value of indicates that an error has occurred .

Exit code can be used to use &&( And operators ) and ||( Or operator ) Carry out the order conditionally , They are all A short circuit Operator . You can also use semicolons to separate commands on the same line ;. The true The program always has a 0 Return code and false Orders always have 1 Return codes . Let's look at some examples

false || echo "Oops, fail"
# Oops, fail

true || echo "Will not be printed"
#

true && echo "Things went well"
# Things went well

false && echo "Will not be printed"
#

true ; echo "This will always run"
# This will always run

false ; echo "This will always run"
# This will always run

Another common pattern is to want the output of a command as a variable . This can be done by Command substitution To complete . Whenever you place $( CMD ) Its time CMD, It will all execute , Get the output of the command and replace it in place . for example , If you do for file in $(ls),shell Will be called first ls, And then iterate over these values . A similar function rarely known is Process replacement ,<( CMD ) It will carry out CMD Output and place it in a temporary file , and <() Replace with the name of the file . When the command expects the value to pass through the file instead of STDIN On delivery , It's very useful . for example ,diff <(ls foo) <(ls bar) Will be displayed dirsfoo The difference between the files in and  bar.

Because it's a huge dump of information , So let's take a look at an example that shows some of these features . It will traverse us for grep The parameters provided by the string ,foobar If you can't find it , It is attached to the file as a comment .

 

#!/bin/bash

echo "Starting program at $(date)" # Date will be substituted

echo "Running program $0 with $# arguments with pid $$"

for file in "$@"; do
    grep foobar "$file" > /dev/null 2> /dev/null
    # When pattern is not found, grep has exit status 1
    # We redirect STDOUT and STDERR to a null register since we do not care about them
    if [[ $? -ne 0 ]]; then
        echo "File $file does not have any foobar, adding one"
        echo "# foobar" >> "$file"
    fi
done

In comparison , We tested whether $? be equal to 0.Bash Many of these comparisons have been made - You can find a detailed list of these in the online help page of test. stay bash When making a comparison in , Try double brackets [[ ]], Instead of simple brackets [ ]. Although it's not easy to make mistakes , But the chances of making mistakes are low sh. stay here A more detailed explanation can be found .

When you start the script , You usually want to provide similar parameters .Bash You can simplify this process by performing file name extension , Extended expression . These technologies are often referred to as shells wildcard .

  • wildcard - Whenever you want to perform some kind of wildcard match , Can be used separately ? and * Match one or any number of characters . for example , Given the file foo,foo1,foo2,foo10 and bar, This command rm foo? Will delete foo1 and foo2 and rm foo* All... Will be deleted , but bar.
  • Curly braces {}– As long as there is a common substring in a series of commands , You can use curly braces to put bash Automatic extension . When moving or converting files , It's very convenient .
convert image.{png,jpg}
# Will expand to
convert image.png image.jpg

cp /path/to/project/{foo,bar,baz}.sh /newpath
# Will expand to
cp /path/to/project/foo.sh /path/to/project/bar.sh /path/to/project/baz.sh /newpath

# Globbing techniques can also be combined
mv *{.py,.sh} folder
# Will move all *.py and *.sh files


mkdir foo bar
# This creates files foo/a, foo/b, ... foo/h, bar/a, bar/b, ... bar/h
touch {foo,bar}/{a..h}
touch foo/x bar/y
# Show differences between files in foo and bar
diff <(ls foo) <(ls bar)
# Outputs
# < x
# ---
# > y

To write bash Scripts can be tricky and not intuitive . image shellcheck Tools like this can help you sh / bash Error found in script .

Please note that , Scripts don't have to use bash Write to call from the terminal . for example , This is an easy one Python Script , It outputs its parameters in reverse order :

#!/usr/local/bin/python
import sys
for arg in reversed(sys.argv[1:]):
    print(arg)

 

The kernel knows how to use python Interpreter, not shell Command to execute the script , Because we include... At the top of the script shebang That's ok . A good practice is to use the env The command to write shebang That's ok , The command will be parsed to any location in the system , This increases the portability of scripts . To resolve the location ,env Will make use of PATH The environment variables we introduced in the first lesson . For this example ,shebang The line looks like #!/usr/bin/env python.

What you should bear in mind shell Some of the differences between functions and scripts are :

  • Functions must use the same language as the shell , And scripts can be written in any language . That's why scripts are included shebang Why it's so important .
  • After reading the function definition , Function will be loaded once . The script loads every time it executes . This makes the function load a little faster , But whenever you change them , Must reload their definitions .
  • Function in the current shell Execution in the environment , The script is executed in its own process . therefore , Function can modify environment variables , for example , Change the current directory , Scripts can't . The script will be passed by the value environment variable exported using the following command export
  • Like any programming language , Functions are a powerful construct , It can be modularized , Code reuse and Shell Code clarity . Usually ,shell The script will contain its own function definition .

 

Shell tools

Find out how to use commands

At this point , You may want to know how to find in the aliasing section , Such as the parameters of the command ls -l,mv -i and mkdir -p. More generally , Give a command , How do you find out what it does and its different options ? You can always start using Google Search for , But because of UNIX Before StackOverflow, So there's a built-in way to get this information .

As we are Shell What we saw in the lecture , The first-order method is to use -hor--help Flag calls the command . A more detailed method is to use man command . Manual man abbreviation , Provides a man page for the command you specify ( It's called a manual page ). for example ,man rm Will output rm The behavior of command and the sign it takes , Include -i The logo we showed earlier . actually , up to now , What I've been linking for each command is for that command Linux Online version of the online help page . If a developer writes a man page entry and includes it in the installation process , Even if you install a non-native command, it will have a man page entry . For interactive tools ( For example, based on ncurses Tools for ), You can usually use :help Command or type to access help for commands in a program ?.

Sometimes , The online help page may provide an overly detailed description of the command , This makes it difficult for you to understand the flags used in common use cases / grammar . TLDR page It's a good complementary solution , Focus on giving command example use cases , So that you can quickly determine which option to use . for instance , I found myself referring back to tldr page tar and ffmpeg Ways are often better than online help pages .

 

Find files

One of the most common repetitive tasks faced by every programmer is to find files or directories . All similar to UNIX All of our systems come with find, This is a good way to find files shell Tools .find Will recursively search for files that meet certain criteria . Some examples :

# Find all directories named src
find . -name src -type d
# Find all python files that have a folder named test in their path
find . -path '*/test/*.py' -type f
# Find all files modified in the last day
find . -mtime -1
# Find all zip files with size in range 500k to 10M
find . -size +500k -size -10M -name '*.tar.gz'

In addition to listing documents ,find You can also perform operations on files that match your query . This property is useful for simplifying tasks that can be very monotonous . 

# Delete all files with .tmp extension
find . -name '*.tmp' -exec rm {} \;
# Find all PNG files and convert them to JPG
find . -name '*.png' -exec convert {} {}.jpg \;

 

Even though find It's everywhere , But its grammar is sometimes hard to remember . for example , It's easy to find files that match certain patterns ,PATTERN It has to be carried out find -name '*PATTERN*'( perhaps -iname If you want pattern matching to be case insensitive ). You can start building aliases for these scenarios , however Shell Part of philosophy is , It's a good thing to explore alternatives . please remember ,shell One of the best features of is that you're just calling the program , So you can find ( Even write it yourself ) Alternative programs for some programs . for example ,fd Yes, it's simple , Fast and user friendly alternatives find. It provides some nice default values , For example, color output , Default regular expression matches and Unicode Support . in my opinion , It also has a more intuitive grammar . for example , Syntax for finding patterns PATTERN yes fd PATTERN.

Most people agree that find,fd And it's good , But some of you may want to know how efficient it is to find a file every time and compile an index or database for quick search . that locate For what . locate Use the database to update with updatedb. In most systems ,updatedb It's updated every day through cron. therefore , The trade-off between the two is speed and freshness . and find, Similar tools can also be used, such as file size , Modify attributes such as time or file permissions to find files , and locate Use only the file name . stay here You can find a deeper comparison .

Look for code

It's useful to find files by name , But usually you want to use the file Content To search . A common case is to want to search all the files that contain a pattern , And the location of these patterns in these files . To achieve this , Most of them are similar to UNIX All of our systems provide grep A general tool , Used to match patterns in input text . grep It's a very valuable shell tool , We're going to talk more about .

For now , You know it grep With a lot of signs , This makes it a very general tool . Some of the things I use a lot are -C Used to get ç ontext Match lines and places -v stay v Listen to the game , That is to print all those lines No Matching patterns . for example ,grep -C 5 It will be printed before and after the game 5 That's ok . When it comes to searching through many files quickly , You have to use -R, Because it will [R ecursively Entry directory , And find the matching string file .

however grep -R There are many ways to improve , for example .git Use more CPU Support &c Ignore folders .grep Many alternatives have been developed , Include ack,ag and rg. They're all great , Provides almost the same functionality . For now rg, In view of its rapidity and intuition , I insist on using ripgrep(). Some examples :

# Find all python files where I used the requests library
rg -t py 'import requests'
# Find all files (including hidden files) without a shebang line
rg -u --files-without-match "^#!"
# Find all matches of foo and print the following 5 lines
rg foo -A 5
# Print statistics of matches (# of matched lines and files )
rg --stats PATTERN

lookup shell command

up to now , We've seen how to find files and code , But as you start in shell Spend more time in , You may want to find a specific command that you type at some point in time . The first thing to know is this , Typing the up arrow will take you back to the previous command , If you continue to press the order , You'll be browsing the shell history .

The history The command will allow you to programmatically access your Shell Historical record . It will make your shell The history is printed to standard output . If you want to search here , We can pass that output to the pipeline grep And search for patterns . history | grep find Will print containing substrings “ find” The order of .

In most shell in , You can use Ctrl+R Do a backward search of history . After pressing Ctrl+R, You can type a substring to match the command in the history . Keep pressing it , You will cycle through the Games in the history . You can also use zsh in Of UP / DOWN The arrow turns it on .Ctrl+R Use fzf The binding comes with a nice addition . fzf Is a general fuzzy finder that can be used with many commands . ad locum , It's used to vaguely match your history , And present the results in a convenient and intuitive way .

Another really cool technique I like about history Automatic suggestions based on history . By the first fish shell introduce , This feature uses the latest command you type ( Share a common prefix with it ) Dynamic automatic completion of the current shell command . Can be in zsh Enable it in , This is for your shell It's a great life skill .

Last , One thing to remember is , If you start with a leading space, the command will not be added to your Shell In history . When you enter a command with a password or other sensitive bits of information , It's very convenient . If you make the mistake of not adding leading spaces , You can always edit .bash_history Or delete the entry manually .zhistory.

 

Directory navigation

up to now , We've assumed that you're already where you need to perform these operations . however , How to browse the catalog quickly ? You can do this in many simple ways , For example, write shell Alias or use ln -s Create symbolic links , But the truth is , Developers have now found very clever and complex solutions .

Same as the theme of this course , You usually want to optimize for common situations . It can be done through things like fasd And other tools to find frequent and / Or the latest files and directories autojump.Fasd adopt frequency ( namely frequency and Recently ) Do... On files and directories ranking . By default , Add a command , You can use this command to quickly use newest The substring of the directory . for example , If you often go to , You can use jump there . Use auto jump , You can use to make the same changes to the directory .fasdzcd/home/user/files/cool_projectz coolj cool

More sophisticated tools can quickly get an overview of the directory structure :tree,broot Or even a fully fledged file manager nnn or ranger.

 

To write bash function  marco,polo Then do the following . Whenever executed marco Current working directory , You should save the current directory in some way , And then in execution polo, No matter in which directory ,polo Should be cd Return to the execution Directory marco. For the convenience of debugging , You can write code into a file ,marco.sh And then, through execution, define ( again ) Load into Shell in source marco.sh.

 

Suppose you have a command that rarely fails . To debug it , You need to capture its output , But failure to run can be time-consuming . Write a bash Script , This script will run the following script , Until it fails , The error and its output are captured in the standard file , And print everything at the end . If you can also report how many runs the script failed to run , You can add points .

 #!/usr/bin/env bash

 n=$(( RANDOM % 100 ))

 if [[ n -eq 42 ]]; then
    echo "Something went wrong"
    >&2 echo "The error was using magic numbers"
    exit 1
 fi

 echo "Everything went according to plan"

 

  1. As we cover lectures find Of -exec It's very powerful to perform operations beyond the files we're searching for . however , If we want to be right all What does the document do , For example, create a zip file , What should I do ? up to now , You've seen that the command will start from the parameters and STDIN Get input from both . When passing commands , We will STDOUT Connect to STDIN, But some orders ( for example ,tar Get the parameters from the input ). To eliminate this disconnection , Yes xargs A command will use STDIN Execute the command as an argument . for example ls | xargs rm The files in the current directory will be deleted .

    Your task is to write a command , This command recursively finds all the... In the folder HTML file , And use them to compress . Please note that , Even if there are spaces in the file ( Tips : Check -d Whether the sign exists xargs), Your command should also work .

    If you are using macOS, Please note the default BSDfind And GNU coreutils in Contains BSD Different . You can -print0 stay find and -0 Use the logo on xargs. As macOS user , You should be aware of macOS The accompanying command-line utility may be associated with GNU The corresponding program is different ; If you like , Sure Use brew install GNU edition .

  2. ( senior ) Write a command or script to recursively find the most recently modified file in the directory . More generally , Can you list all the files by their latest degree ?

 

 

 

 

版权声明
本文为[osc_ 0atwpkmx]所创,转载请带上原文链接,感谢
https://chowdera.com/2020/12/20201208080404605y.html