diff options
| -rw-r--r-- | saved-articles/an introduction to unix.txt | 1913 |
1 files changed, 0 insertions, 1913 deletions
diff --git a/saved-articles/an introduction to unix.txt b/saved-articles/an introduction to unix.txt deleted file mode 100644 index 302c268..0000000 --- a/saved-articles/an introduction to unix.txt +++ /dev/null @@ -1,1913 +0,0 @@ ---- -title: Oliver | An Introduction to Unix -date: 2015-03-13T14:08:55Z -source: http://www.oliverelliott.org/article/computing/tut_unix/#100UsefulUnixCommands -tags: #cmdline - ---- - -**Everybody Knows How to Use a Computer, but Not Everyone Knows How to Use the Command Line. Yet This is the Gateway to Doing Anything and Everything Sophisticated with a Computer and the Most Natural Starting Place to Learn Programming** -_by Oliver; Jan. 13, 2014_ - - - -## Introduction - -I took programming in high school, but I never took to it. This, I strongly believe, is because it wasn't taught right—and teaching it right means starting at the beginning, with unix. The reason for this is three-fold: _(1)_ it gives you a deeper sense of how a high-level computer works (which a glossy front, like Windows, conceals); _(2)_ it's the most natural port of entry into all other programming languages; and _(3)_ it's super-useful in its own right. If you don't know unix and start programming, some things will forever remain hazy and mysterious, even if you can't put your finger on exactly what they are. If you already know a lot about computers, the point is moot; if you don't, then by all means start your programming education by learning unix! - -A word about terminology here: I'm in the habit of horrendously confusing and misusing all of the precisely defined words "[Unix][1]", "[Linux][2]", "[The Command Line][3]", "[The Terminal][4]", "[Shell Scripting][5]", and "[Bash][6]." Properly speaking, _unix_ is an operating system while _linux_ refers to a closely-related family of unix-based operating systems, which includes commercial and non-commercial distributions [1]. (Unix was not free under its developer, AT&T, which caused the unix-linux schism.) The _command line_, as [Wikipedia][3] says, is: - -> ... a means of interacting with a computer program where the user issues commands to the program in the form of successive lines of text (command lines) ... The interface is usually implemented with a command line shell, which is a program that accepts commands as text input and converts commands to appropriate operating system functions. - -So what I mean when I proselytize for "unix", is simply that you learn how to punch commands in on the command line. The _terminal_ is your portal into this world. Here's what my mine looks like: - -![image][7] - - -There is a suite of commands to become familiar with—[The GNU Core Utilities][8] ([wiki entry][9])—and, in the course of learning them, you learn about computers. Unix is a foundational piece of a programming education. - -In terms of bang for the buck, it's also an excellent investment. You can gain powerful abilities by learning just a little. My coworker was fresh out of his introductory CS course, when he was given a small task by our boss. He wrote a full-fledged program, reading input streams and doing heavy parsing, and then sent an email to the boss that began, _"After 1.5 days of madly absorbing perl syntax, I completed the exercise..."_ He didn't know how to use the command-line at the time, and now [a print-out of that email hangs on his wall][10] as a joke—and as a monument to the power of the terminal. - -You can find ringing endorsements for learning the command line from all corners of the internet. For instance, in the excellent course [Startup Engineering (Stanford/Coursera)][11] Balaji Srinivasan writes: - -> A command line interface (CLI) is a way to control your computer by typing in commands rather than clicking on buttons in a graphical user interface (GUI). Most computer users are only doing basic things like clicking on links, watching movies, and playing video games, and GUIs are fine for such purposes. -> -> But to do industrial strength programming - to analyze large datasets, ship a webapp, or build a software startup - you will need an intimate familiarity with the CLI. Not only can many daily tasks be done more quickly at the command line, many others can only be done at the command line, especially in non-Windows environments. You can understand this from an information transmission perspective: while a standard keyboard has 50+ keys that can be hit very precisely in quick succession, achieving the same speed in a GUI is impossible as it would require rapidly moving a mouse cursor over a profusion of 50 buttons. It is for this reason that expert computer users prefer command-line and keyboard-driven interfaces. - -To provide foreshadowing, here are some things you can do in unix: - -* make or rename 100 folders or files _en masse_ -* find all files of a given extension or any file that was created within the last week -* log onto a computer remotely and access its files with ssh -* copy files to your computer directly over the network (no external hard drive necessary!) with [rsync][12] -* run a [Perl][13] or [Python][14] script -* run one of the many programs that are only available on the command line -* see all processes running on your computer or the space occupied by your folders -* see or change the permissions on a file -* parse a text file in any way imaginable (count lines, swap columns, replace words, etc.) -* soundly encrypt your files or communications with [gpg2][15] -* run your own web server on the [Amazon cloud][16] with [nginx][17] -What do all of these have in common? All are hard to do in the [GUI][18], but easy to do on the command line. It would be remiss not to mention that unix is not for everything. In fact, it's not for a lot of things. Knowing which language to use for what is usually a matter of common sense. However, when you start messing about with computers in any serious capacity, you'll bump into unix very quickly—and that's why it's our starting point. - -Is this the truth, the whole truth, and nothing but the truth, [so help my white ass][19]? I believe it is, but also my trajectory through the world of computing began with unix. So perhaps I instinctively want to push this on other people: do it the way I did it. And, because it occupies a bunch of my neuronal real estate at the moment, I could be considered brainwashed :-) - -* * * - - -[1] Still confused about unix vs linux? [Refer to the full family tree][20] and these more precise definitions from Wikipedia: -**_unix_**: _a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, developed in the 1970s at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and others_ -**_linux_**: _a Unix-like and mostly POSIX-compliant computer operating system assembled under the model of free and open-source software development and distribution [whose] defining component ... is the Linux kernel, an operating system kernel first released [in] 1991 by Linus Torvalds_ ↑ - -## 100 Useful Unix Commands - -This article is an introduction to unix. It aims to teach the basic principles and neglects to mention many of the utilities that give unix superpowers. To learn about those, see [100 Useful Unix Commands][21]. - -## Getting Started: Opening the Terminal - -If you have a Mac, navigate to Applications > Utilities and open the application named "Terminal": - -![image][22] - -If you have a PC, _abandon all hope, ye who enter here_! Just kidding—partially. None of the native Windows shells, such as [cmd.exe][23] or [PowerShell][24], are unix-like. Instead, they're marked with hideous deformities that betray their ignoble origin as grandchildren of the MS-DOS command interpreter. If you didn't have a compelling reason until now to quit using PCs, here you are [1]. Typically, my misguided PC friends don't use the command line on their local machines; instead, they have to `ssh` into some remote server running Linux. (You can do this with an ssh client like [PuTTY][25], [Chrome's Terminal Emulator][26], or [MobaXterm][27], but don't ask me how.) On Macintosh you can start practicing on the command line right away without having to install a Linux distribution [2] (the Mac-flavored unix is called [Darwin][28]). - -For both Mac and PC users who want a bona fide Linux command line, one easy way to get it is in the cloud with [Amazon EC2][16] via the [AWS Free Tier][29]. If you want to go whole hog, you can download and install a Linux distribution—[Ubuntu][30], [Mint][31], [Fedora][32], and [CentOs][33] are popular choices—but this is asking a lot of non-hardcore-nerds (less drastically, you could boot Linux off of a USB drive or run it in a virtual box). - -* * * - - -[1] I should admit, you can and should get around this by downloading something like [Cygwin][34], whose homepage states: _"Get that Linux feeling - on Windows"_ ↑ -[2] However, if you're using Mac OS rather than Linux, note that OS does not come with the [GNU coreutils][9], which are the gold standard. [You should download them][35] ↑ - -## The Definitive Guides to Unix, Bash, and the Coreutils - -Before going any further, it's only fair to plug the authoritative guides which, unsurprisingly, can be found right on your command line: - - $ man bash - $ info coreutils - -(The $ at the beginning of the line represents the terminal's prompt.) These are good references, but overwhelming to serve as a starting point. There are also great resources online: although these guides, too, are exponentially more useful once you have a small foundation to build on. - -## The Unix Filestructure - -All the files and _directories_ (a fancy word for "folder") on your computer are stored in a hierarchical tree. Picture a tree in your backyard upside-down, so the trunk is on the top. If you proceed downward, you get to big branches, which then give way to smaller branches, and so on. The trunk contains everything in the sense that everything is connected to it. This is the way it looks on the computer, too, and the trunk is called the _root directory_. In unix it's represented with a slash: - -/ - -The root contains directories, which contain other directories, and so on, just like our tree. To get to any particular file or directory, we need to specify the _path_, which is a slash-delimited address: - -/dir1/dir2/dir3/some_file - -Note that a full path always starts with the root, because the root contains everything. As we'll see below, this won't necessarily be the case if we specify the address in a _relative_ way, with respect to our current location in the filesystem. - -Let's examine the directory structure on our Macintosh. We'll go to the root directory and look down just one level with the unix command tree. (If we tried to look at the whole thing, we'd print out every file and directory on our computer!) We have: - -![image][36] - - -While we're at it, let's also look at the directory /Users/username, which is the specially designated [_home directory][37]_ on the Macintosh: - -![image][38] - - -One thing we notice right away is that the Desktop, which holds such a revered spot in the GUI, is just another directory—simply the first one we see when we turn on our computer. - -If you're on Linux rather than Mac OS, the directory tree might look less like the screenshot above and more like this: - -![image][39] - - -The naming of these folders is not intuitive, but you can read about the role of each one [here][40]. I've arbitrarily traced out the path to /var/log, a location where some programs store their log files. - -If the syntax of a unix path looks familiar, it is. A webpage's [URL][41], with its telltale forward slashes, looks like a unix path with a domain prepended to it. This is not a coincidence! For a simple static website, its structure on the web is determined by its underlying directory structure on the server, so navigating to: - -http://www.example.com/abc/xyz - -will serve you content in the folder _websitepath/abc/xyz_ on the host's computer (i.e., the one owned by _example.com_). Modern dynamic websites are more sophisticated than this, but it's neat to reflect that the whole word has learned this unix syntax without knowing it. - -To learn more, see the [O'Reilly discussion of the unix file structure][42]. - -## The Great Trailing Slash Debate - -Sometimes you'll see directories written with a trailing slash, as in: - -dir1/ - -This helpfully reminds you that the entity is a directory rather than a file, but on the command line using the more compact _dir1_ is sufficient. There are a handful of unix commands which behave slightly differently if you leave the trailing slash on, but this sort of extreme pedantry isn't worth worrying about. - -## Where Are You? - Your _Path_ and How to Navigate through the Filesystem - -When you open up the terminal to browse through your filesystem, run a program, or do anything, _you're always somewhere_. Where? You start out in the designated _home directory_ when you open up the terminal. The home directory's path is preset by a global variable called HOME. Again, it's /Users/username on a Mac. - -As we navigate through the filesystem, there are some conventions. The _current working directory (cwd)_—whatever directory we happen to be in at the moment—is specified by a dot: - -. - -Sometimes it's convenient to write this as: - -./ - -which is not to be confused with the root directory: - -/ - -When a program is run in the cwd, you often see the syntax: - - $ ./myprogram - -which emphasizes that you're executing a program from the current directory. The directory one above the cwd is specified by two dots: - -.. - -With the trailing slash syntax, that's: - -../ - -A tilde is shorthand for the home directory: - -~ - -or: - -~/ - -To see where we are, we can _print working directory_: - - $ pwd - -To move around, we can _change directory_: - - $ cd /some/path - -By convention, if we leave off the argument and just type: - - $ cd - -we will go home. To _make directory_—i.e., create a new folder—we use: - - $ mkdir - -As an example, suppose we're in our home directory, /Users/username, and want to get one back to /Users. We can do this two ways: - - $ cd /Users - -or: - - $ cd .. - -This illustrates the difference between an _absolute path_ and a _relative path_. In the former case, we specify the complete address, while in the later we give the address with respect to our _cwd_. We could even accomplish this with: - - $ cd /Users/username/.. - -or maniacally seesawing back and forth: - - $ cd /Users/username/../username/.. - -if our primary goal were obfuscation. This distinction between the two ways to specify a path may seem pedantic, but it's not. Many scripting errors are caused by programs expecting an absolute path and receiving a relative one instead or vice versa. Use relative paths if you can because they're more portable: if the whole directory structure gets moved, they'll still work. - -Let's mess around. We know cd with no arguments takes us home, so try the following experiment: - - $ echo $HOME # print the variable HOME - /Users/username - $ cd # cd is equivalent to cd $HOME - $ pwd # print working directory shows us where we are - /Users/username - - $ unset HOME # unset HOME erases its value - $ echo $HOME - - $ cd /some/path # cd into /some/path - $ cd # take us HOME? - $ pwd - /some/path - -What happened? We stayed in /some/path rather than returning to /Users/username. The point? There's nothing magical about _home_—it's merely set by the variable HOME. More about variables soon! - -## Gently Wading In - The Top 10 Indispensable Unix Commands - -Now that we've dipped one toe into the water, let's make a list of the 10 most important unix commands in the universe: - -1. pwd -2. ls -3. cd -4. mkdir -5. echo -6. cat -7. cp -8. mv -9. rm -10. man -Every command has a help or _manual_ page, which can be summoned by typing man. To see more information about pwd, for example, we enter: - - $ man pwd - -But pwd isn't particularly interesting and its man page is barely worth reading. A better example is afforded by one of the most fundamental commands of all, ls, which lists the contents of the _cwd_ or of whatever directories we give it as arguments: - - $ man ls - -The man pages tend to give TMI (too much information) but the most important point is that commands have _flags_ which usually come in a _one-dash-one-letter_ or _two-dashes-one-word_ flavor: - - command -f - command --flag - -and the docs will tell us what each option does. You can even try: - - $ man man - -Below we'll discuss the commands in the top 10 list in more depth. - -## ls - -Let's go HOME and try out [ls][43] with various flags: - - $ cd - $ ls - $ ls -1 - $ ls -hl - $ ls -al - -Some screen shots: - -![image][44] - - - -![image][45] - - -First, vanilla ls. We see our files—no surprises. And ls -1 merely displays our files in a column. To show the human-readable, long form we stack the -h and -l flags: - -ls -hl - -This is equivalent to: - -ls -h -l - -Screenshot: - -![image][46] - - -This lists the owner of the file; the group to which he belongs (_staff_); the date the file was created; and the file size in human-readable form, which means bytes will be rounded to kilobytes, gigabytes, etc. The column on the left shows _permissions_. If you'll indulge mild hyperbole, this simple command is already revealing secrets that are well-hidden by the GUI and known only to unix users. In unix there are three spheres of permission—_user_, _group_, and _other/world_—as well as three particular types for each sphere—_read_, _write_, and _execute_. Everyone with an account on the computer is a unique _user_ and, although you may not realize it, can be part of various groups, such as a particular lab within a university or team in a company. To see yourself and what groups you belong to, try: - - $ whoami - $ groups - -(To see more information about a user, [finger][47] his username.) A string of dashes displays permission: - - --------- - rwx------ - rwxrwx--- - rwxrwxrwx - -This means, respectively: no permission for anybody; read, write, execute permission for only the user; _rwx_ permission for the user and anyone in the group; and _rwx_ permission for the user, group, and everybody else. Permission is especially important in a shared computing environment. You should internalize now that two of the most common errors in computing stem from the two _P_ words we've already learned: _paths and permissions_. The command chmod, which we'll learn later, governs permission. - -If you look at the screenshot above, you see a tenth letter prepended to the permission string, e.g.: - -This has nothing to do with permissions and instead tells you about the type of entity in the directory: _d_ stands for directory, _l_ stands for symbolic link, and a plain dash denotes a file. - -The -a option in: - -ls -al - -lists _all_ files in the directory, including [_dotfiles][48]_. These are files that begin with a dot and are hidden in the GUI. They're often system files—more about them later. Screenshot: - -![image][49] - - -Note that, in contrast to ls -hl, the file sizes are in pure bytes, which makes them a little hard to read. - -A general point about unix commands: they're often robust. For example, with ls you can use an arbitrary number of arguments and it obeys the convention that an asterisk matches anything (this is known as [file _globbing_][50], and I think of it as the prequel to _regular expressions_). Take: - - $ ls . dir1 .. dir2/*.txt dir3/A*.html - -This monstrosity would list anything in the _cwd_; anything in directory _dir1_; anything in the directory one above us; anything in directory _dir2_ that ends with _.txt_; and anything in directory _dir3_ that starts with _A_ and ends with _.html_. You get the point. - -## Single Line Comments in Unix - -Anything prefaced with a # —that's _pound-space_—is a comment and will not be executed: - - $ # This is a comment. - $ # If we put the pound sign in front of a command, it won't do anything: - $ # ls -hl - -Suppose you write a line of code on the command line and decided you don't want to execute it. You have two choices. The first is pressing _Cntrl-c_, which serves as an "abort mission." The second is jumping to the beginning of the line (_Cntrl-a_) and adding the pound character. This has an advantage over the first method that the line will be saved in bash history (discussed below) and can thus be retrieved and modified later. - -In a script, pound-special-character (like _#!_) is sometimes interpreted (see below), so take note and include a space after # to be safe. - -## The Primacy of Text Files, Text Editors - -As we get deeper into unix, we'll frequently be using text editors to edit code, and viewing either data or code in text files. When I got my hands on a computer as a child, I remember text editors seemed like the most boring programs in the world (compared to, say, 1992 [Prince of Persia][51]). And text files were on the bottom of my food chain. But the years have changed me and now I like nothing better than a clean, unformatted _.txt_ file. It's all you need! If you store your data, your code, your correspondence, your book, or almost anything in _.txt_ files with a systematic structure, they can be parsed on the command line to reveal information from many facets. Here's some advice: do all of your text-related work in a good text editor. Open up clunky [Microsoft Word][52], and you've unwittingly spoken a demonic incantation and summoned the beast. Are these the words of a lone lunatic dispensing [hateration][53]? No, because on the command line you can count the words in a text file, search it with [grep][54], input it into a Python program, et cetera. However, a file in Microsoft Word's proprietary and unknown formatting is utterly unusable. - -Because text editors are extremely important, some people develop deep relationships with them. My co-worker, who is a [Vim][55] aficionado, turned to me not long ago and said, "You know how you should think about editing in Vim? _As if you're talking to it._" On the terminal, a ubiquitous and simple editor is [nano][56]. If you're more advanced, try [Vim][55] or [Emacs][57]. Not immune to my co-worker's proselytizing, I've converted to Vim. Although it's sprawling and the learning curve can be harsh—Vim is like a programming language in itself—you can do a zillion things with it. I put together a quick and dirty Vim wiki [here][58]. - -On the GUI, there are many choices: [Sublime][59], [Aquamacs][60], [Smultron][61], etc. I used to use Smultron until I found, unforgivably, that the spacing of documents when you looked at them in the editor and on the terminal was different. I hear good things about Sublime and Aquamacs. - -_Exercise_: Let's try making a text file with nano. Type: - - $ nano file.txt - -and make the following three-row two-column file: (It's _Cntrl-o_ to save and _Cntrl-x_ to exit.) - -## _echo_ and _cat_ - -More essential commands: echo prints the _string_ passed to it as an argument, while cat prints the _contents_ of files passed to it as arguments. For example: - - $ echo joe - $ echo "joe" - -would both print _joe_, while: - - $ cat file.txt - -would print the contents of _file.txt_. Entering: - - $ cat file.txt file2.txt - -would print out the contents of both _file.txt_ and _file2.txt_ concatenated together, which is where this command gets its slightly confusing name. - -Finally, a couple of nice flags for these commands: - - $ echo -n "joe" # suppress newline - $ echo -e "joetjoenjoe" # interpret special chars ( t is tab, n newline ) - $ cat -n file.txt # print file with line numbers - -## _cp_, _mv_, and _rm_ - -Finishing off our top 10 list we have cp, mv, and rm. The command to make a copy of a file is cp: - - $ cp file1 file2 - $ cp -R dir1 dir2 - -The first line would make an identical copy of _file1_ named _file2_, while the second would do the same thing for directories. Notice that for directories we use the -R flag (for _recursive_). The directory and everything inside it are copied. - -_Question_: what would the following do? - - $ cp -R dir1 ../../ - -_Answer_: it would make a copy of _dir1_ up two levels from our current working directory. - -To rename a file or directory we use mv: - - $ mv file1 file2 - -In a sense, this command also moves files, because we can rename a file into a different path. For example: - - $ mv file1 dir1/dir2/file2 - -would move _file1_ into _dir1/dir2/_ and change its name to _file2_, while: - - $ mv file1 dir1/dir2/ - -would simply move _file1_ into _dir1/dir2/_ or, if you like, rename _./file1_ as _./dir1/dir2/file1_. - -Finally, rm removes a file or directory: - - $ rm file # removes a file - $ rm -r dir # removes a file or directory - $ rm -rf dir # force removal of a file or directory - # (i.e., ignore warnings) - -## Variables in Unix - -To declare something as a variable use an equals sign, with no spaces. Let's declare _a_ to be a variable: - - $ a=3 # This syntax is right (no whitespace) - $ a = 3 # This syntax is wrong (whitespace) - -bash: a: command not found - -Once we've declared something as a variable, we need to use _$_ to access its value (and to let bash know it's a variable). For example: - - $ a=3 - $ echo a - a - $ echo $a - 3 - -So, with no _$_ sign, bash thinks we just want to echo the string _a_. With a _$_ sign, however, it knows we want to access what the variable _a_ is storing, which is the value _3_. Variables in unix are loosely-typed, meaning you don't have to declare something as a string or an integer. - - $ a=3 # a can be an integer - $ echo $a - 3 - - $ a=joe # or a can be a string - $ echo $a - joe - - $ a="joe joe" # Use quotes if you want a string with spaces - $ echo $a - joe joe - -We can declare and echo two variables at the same time, and generally play fast and loose, as we're used to doing on the command line: - - $ a=3; b=4 - $ echo $a $b - 3 4 - $ echo $a$b # mesh variables together as you like - 34 - $ echo "$a$b" # use quotes if you like - 34 - $ echo -e "$at$b" # the -e flag tells echo to interpret t as a tab - 3 4 - -You should also be aware of how bash treats double vs single quotes. As we've seen, if you want to use a string with spaces, you use double quotes. If you use double quotes, any variable inside them will be expanded, the same as in Perl. If you use single quotes, everything is taken literally and variables are not expanded. Here's an example: - - $ var=5 - $ joe=hello $var - -bash: 5: command not found - - $ joe="hello $var" - $ echo $joe - hello 5 - - $ joe='hello $var' - $ echo $joe - hello $var - -An important note is that often we use variables to store _paths_ in unix. Once we do this, we can use all of our familiar directory commands on the variable: - - $ d=dir1/dir2/dir3 - $ ls $d - $ cd $d - - $ d=.. # this variable stores the directory one above us (relative path) - $ cd $d/.. # cd two directories up - -## Escape Sequences - -[Escape sequences][62] are important in every language. When bash reads _$a_ it interprets it as whatever's stored in the variable _a_. What if we actually want to echo the string _$a_? To do this, we use as an escape character: - - $ a=3 - $ echo $a - 3 - $ echo $a - $a - $ echo "$a" # use quotes if you like - $a - -What if we want to echo the slash, too? Then we have to escape the escape character (using the escape character!): - - $ echo \$a # escape the slash and the dollar sign - $a - -This really comes down to parsing. The slash helps bash figure out if your text is a plain old string or a variable. It goes without saying that you should avoid special characters in your variable names. In unix we might occasionally fall into a parsing tar-pit trap. To avoid this, and make extra sure bash parses our variable right, we can use the syntax _${a}_ as in: - - $ echo ${a} - 3 - -When could this possibly be an issue? Later, when we discuss scripting, we'll learn that _$n_, where _n_ is a number, is the _n_th argument to our script. If you were crazy enough to write a script with 11 arguments, you'd discover that bash interprets a=$11 as a=$1 (the first argument) concatenated with the string 1 while a=${11} properly represents the eleventh argument. This is getting in the weeds, but FYI. - -Here's a more practical example: - - $ a=3 - $ echo $a # variable a equals 3 - 3 - $ echo $apple # variable apple is not set - - $ echo ${a}pple # this describes the variable a plus the string "pple" - 3pple - -## Global Variables in Unix - -In general, it is the convention to use capital letters for global variables. We've already learned about one: HOME. We can see all the variables set in our shell by simply typing: - - $ set - -Some basic variables deserve comment: - -* HOME -* PS1 -* TMPDIR -* EDITOR -* DISPLAY -HOME, as we've already seen, is the path to our home directory (preset to /Users/username on Macintosh). PS1 sets the shell's prompt. For example: - - $ PS1=':-) ' - -changes our prompt from a dollar-sign into an emoticon, as in: - -![image][63] - - -On your computer there is a designated temporary directory and its path is stored in TMPDIR. Some commands, such as sort, which we'll learn later, surreptitiously make use of this directory to store intermediate files. At work, we have a shared computer system and occasionally this common directory $TMPDIR will run out of space, causing programs trying to write there to fail. One solution is to simply set TMPDIR to a different path where there's free space. EDITOR sets the default text editor (you can invoke it by pressing _Cntrl-x-e_). And DISPLAY is a variable related to the [X Window System][64]. - -Many programs rely on their own agreed-upon global variables. For example, if you're a Perl user, you may know that Perl looks for modules in the directory whose path is stored in PERL5LIB. Python looks for its modules in PYTHONPATH; R looks for packages in R_LIBS; Matlab uses MATLABPATH; awk uses AWKPATH; C++ looks for libraries in LD_LIBRARY_PATH; and so on. These variables don't exist in the shell by default. A program will make a system call and look for the variable. If the user has had the need or foresight to define it, the program can make use of it. - -## The _PATH_ - -The most important global variable of all is the PATH. This is _the_ PATH, as distinct from _a_ path, a term we've already learned referring to a location in the filesystem. The PATH is a colon-delimited list of directories where unix will look for executable programs when you enter something on command line. If your program is in one of these directories, you can run it from any location by simply entering its name. If the program is not in one of these directories, you can still run it, of course, but you'll have to include its path. - -Let's revisit the idea of a command in unix. What's a command? It's nothing more than a program sitting in a directory somewhere. So, if ls is a program, where is it? Use the command which to see its path: - - $ which ls # on my work computer - /bin/ls - - $ which ls # on my home Mac - /usr/local/Cellar/coreutils/8.20/libexec/gnubin/ls - -For the sake of argument, let's say I download an updated version of the ls command, and then type ls in my terminal. What will happen—will the old ls or the new ls execute? The PATH comes into play here because it also determines priority. When you enter a command, unix will look for it in each directory of the PATH, from first to last, and execute the first instance it finds. For example, if: - -PATH=/bin/dir1:/bin/dir2:/bin/dir3 - -and there's a command named _ls_ in both /bin/dir1 and /bin/dir2, the one in /bin/dir1 will be executed. - -Let's see what your PATH looks like. Enter: - - $ echo $PATH - -For example, here's a screenshot of the default PATH on Ubuntu: - -![image][65] - - -To emphasize the point again, all the programs in the directories specified by your PATH are all the programs that you can access on the command line by simply typing their names. - -The PATH is not immutable. You can set it to be anything you want, but in practice you'll want to augment, rather than overwrite, it. By default, it contains directories where unix expects executables, like: - -* /bin -* /usr/bin -* /usr/local/bin -Let's say you have just written the command /mydir/newcommand. If you're not going to use the command very often, you can invoke it using its full path every time you need it: - - $ /mydir/newcommand - -However, if you're going to be using it frequently, you can just add /mydir to the PATH and then invoke the command by name: - - $ PATH=/mydir:$PATH # add /mydir to the front of PATH - highest priority - $ PATH=$PATH:/mydir # add /mydir to the back of PATH - lowest priority - $ newcommand # now invoking newcommand is this easy - -This is a frequent chore in unix. If you download some new program, you will often find yourself updating the PATH to include the directory containing its binaries. How can we avoid having to do this every time we open the terminal for a new session? We'll discuss this below when we learn about _.bashrc_. - -If you want to shoot yourself in the foot, you can vaporize the PATH: - - $ unset PATH # not advisable - $ ls # now ls is not found - -bash: ls: No such file or directory - -but this is not advisable, save as a one-time educational experience. - -## Links - -While we're on the general subject of paths, let's talk about [_symbolic links][66]_. If you've ever used the _Make Alias_ command on a Macintosh (not to be confused with the unix command alias, discussed below), you've already developed intuition for what a link is. Suppose you have a file in one folder and you want that file to exist in another folder simultaneously. You could copy the file, but that would be wasteful. Moreover, if the file changes, you'll have to re-copy it—a huge ball-ache. Links solve this problem. A link to a file is a stand-in for the original file, often used to access the original file from an alternate file path. It's not a copy of the file but, rather, points to the file. - -To make a symbolic link, use the command [ln][67]: - - $ ln -s /path/to/target/file mylink - -This produces: - - mylink --> /path/to/target/file - -in the cwd, as ls -hl will show. Note that removing _mylink_: - - $ rm mylink - -does not affect our original file. - -If we give the target (or source) path as the sole argument to ln, the name of the link will be the same as the source file's. So: - - $ ln -s /path/to/target/file - -produces: - - file --> /path/to/target/file - -Links are incredibly useful for all sorts of reasons—the primary one being, as we've already remarked, if you want a file to exist in multiple locations without having to make extraneous, space-consuming copies. You can make links to directories as well as files. Suppose you add a directory to your PATH that has a particular version of a program in it. If you install a newer version, you'll need to change the PATH to include the new directory. However, if you add a link to your PATH and keep the link always pointing to the most up-to-date directory, you won't need to keep fiddling with your PATH. The scenario could look like this: - - $ ls -hl myprogram - current -> version3 - version1 - version2 - version3 - -(where I'm hiding some of the output in the long listing format.) In contrast to our other examples, the link is in the same directory as the target. Its purpose is to tell us which version, among the many crowding a directory, we should use. - -Another good practice is putting links in your home directory to folders you often use. This way, navigating to those folders is easy when you log in. If you make the link: - - ~/MYLINK --> /some/long/and/complicated/path/to/an/often/used/directory - -then you need only type: - - $ cd MYLINK - -rather than: - - $ cd /some/long/and/complicated/path/to/an/often/used/directory - -Links are everywhere, so be glad you've made their acquaintance! - -## What is Scripting? - -By this point, you should be comfortable using basic utilities like echo, cat, mkdir, cd, and ls. Let's enter a series of commands, creating a directory with an empty file inside it, for no particular reason: - - $ mkdir tmp - $ cd tmp - $ pwd - /Users/oliver/tmp - $ touch myfile.txt # the command touch creates an empty file - $ ls - myfile.txt - $ ls myfile_2.txt # purposely execute a command we know will fail - ls: cannot access myfile_2.txt: No such file or directory - -What if we want to repeat the exact same sequence of commands 5 minutes later? _Massive bombshell_—we can save all of these commands in a file! And then run them whenever we like! Try this: - - $ nano myscript.sh - -and write the following: - - # a first script - mkdir tmp - cd tmp - pwd - touch myfile.txt - ls - ls myfile_2.txt - -Gratuitous screenshot: - -![image][68] - - -This file is called a _script_ (_.sh_ is a typical suffix for a shell script), and writing it constitutes our first step into the land of bona fide computer programming. In general usage, a script refers to a small program used to perform a niche task. What we've written is a recipe that says: ** - -* create a directory called "tmp" -* go into that directory -* print our current path in the file system -* make a new file called "myfile.txt" -* list the contents of the directory we're in -* specifically list the file "myfile_2.txt" (which doesn't exist) -** This script, though silly and useless, teaches us the fundamental fact that all computer programs are ultimately just lists of commands. - -Let's run our program! Try: - - $ ./myscript.sh - -bash: ./myscript.sh: Permission denied - -_WTF!_ It's dysfunctional. What's going on here is that the file permissions are not set properly. In unix, when you create a file, the default permission is _not executable_. You can think of this as a brake that's been engaged and must be released before we can go (and do something potentially dangerous). First, let's look at the file permissions: - - $ ls -hl myscript.sh - -rw-r--r-- 1 oliver staff 75 Oct 12 11:43 myscript.sh - -Let's change the permissions with the command chmod and execute the script: - - $ chmod u+x myscript.sh # add executable(x) permission for the user(u) only - $ ls -hl myscript.sh - -rwxr--r-- 1 oliver staff 75 Oct 12 11:43 myscript.sh - - $ ./myscript.sh - /Users/oliver/tmp/tmp - myfile.txt - ls: cannot access myfile_2.txt: No such file or directory - -Not bad. Did it work? Yes, it did because it's printed stuff out and we see it's created tmp/myfile.txt: - - $ ls - myfile.txt myscript.sh tmp - $ ls tmp - myfile.txt - -An important note is that even though there was a cd in our script, if we type: - - $ pwd - /Users/oliver/tmp - -we see that we're still in the same directory as we were in when we ran the script. Even though the script entered /Users/oliver/tmp/tmp, and did its bidding, we stay in /Users/oliver/tmp. Scripts always work this way—where they go is independent of where we go. - -If you're wondering why anyone would write such a pointless script, you're right—it would be odd if we had occasion to repeat this combination of commands. There are some more realistic examples of scripting below. - -## File Suffixes in Unix - -As we begin to script it's worth following some file naming conventions. We should use common sense suffixes, like: - -* _.txt_ \- for text files -* _.html_ \- for html files -* _.sh_ \- for shell scripts -* _.pl_ \- for Perl scripts -* _.py_ \- for Python scripts -* _.cpp_ \- for c++ code -and so on. Adhering to this organizational practice will enable us to quickly scan our files, and make searching for particular file types easier [1]. As we saw above, commands like ls and find are particularly well-suited to use this kind of information. For example, list all text files in the cwd: - - $ ls *.txt - -List all text files in the cwd and below (i.e., including child directories): - - $ find . -name "*.txt" - - - -* * * - - -[1] An astute reader noted that, for commands—as opposed to, say, html or text files—using suffixes is not the best practice because it violates the principle of encapsulation. The argument is that a user is neither supposed to know nor care about a program's internal implementation details, which the suffix advertises. You can imagine a program that starts out as a shell script called _mycommand.sh_, is upgraded to Python as _mycommand.py_, and then is rewritten in C for speed, becoming the binary _mycommand_. What if other programs depend on _mycommand_? Then each time _mycommand_'s suffix changes they have to be rewritten—a big problem. Although I make this sloppy mistake in this article, that doesn't excuse you! [Read the full argument][69] ↑ - -## The Shebang - -We've left out one important detail about scripting. How does unix know we want to run a bash script, as opposed to, say, a Perl or Python script? There are two ways to do it. We'll illustrate with two simple scripts, a bash script and a Perl script: - - $ cat myscript_1.sh # a bash script - echo "hello kitty" - - $ cat myscript_1.pl # a Perl script - print "hello kittyn"; - -The first way to tell unix which program to use to interpret the script is simply to say so on the command line. For example, we can use bash to execute bash scripts: - - $ bash ./myscript_1.sh # use bash for bash scripts - hello kitty - -and perl for Perl scripts: - - $ perl ./myscript_1.pl # use Perl for Perl scripts - hello kitty - -But this won't work for a Perl script: - - $ ./myscript_1.pl # this won't work - ./myscript_1.pl: line 1: print: command not found - -And if we purposefully specify the wrong language, we'll get errors: - - $ bash ./myscript_1.pl # let's purposefully do it backwards - ./myscript_1.pl: line 1: print: command not found - - $ perl ./myscript_1.sh - String found where operator expected at ./myscript_1.sh line 1, - near "echo "hello kitty"" - (Do you need to predeclare echo?) - syntax error at ./myscript_1.sh line 1, near "echo "hello kitty"" - Execution of ./myscript_1.sh aborted due to compilation errors. - -The second way to specify the proper interpreter—and the better way, which you should emulate—is to put it in the script itself using a [_shebang_][70]. To do this, let's remind ourselves where bash and perl reside on our system. On my computer, they're here: - - $ which perl - /usr/bin/perl - - $ which bash - /bin/bash - -although perl could be somewhere else on your machine (bash should be in /bin by convention). The _shebang_ specifies the language in which your script is interpreted according to the syntax #! followed by the path to the language. It should be the first line of your script. Note that it's not a comment even though it looks like one. Let's add shebangs to our two scripts: - - $ cat myscript_1.sh - #!/bin/bash - echo "hello kitty" - - $ cat myscript_1.pl - #!/usr/bin/perl - print "hello kittyn"; - -Now we can run them without specifying the interpreter in front: - - $ ./myscript_1.sh - hello kitty - $ ./myscript_1.pl - hello kitty - -However, there's _still_ a lingering issue and it has to do with [portability][71], an important software principle. What if perl is in a different place on your machine than mine and you copy my scripts and try to run them? The path will be wrong and they won't work. The solution to this issue is courtesy of a neat trick using [env][72]. We can amend our script to be: - - $ cat myscript_1.pl - #!/usr/bin/env perl - print "hello kittyn"; - -Of course, this assumes you have a copy of env in /usr/bin, but this is usually a correct assumption. What env does here is to use whatever your environmental variable for perl is—i.e., the perl that's first in your PATH. - -This is a useful practice even if you're not sharing scripts. Suppose you've updated your version of perl and there's a newer copy than /usr/bin/perl. You've appropriately updated your PATH such that the directory containing the updated perl comes before /usr/bin. If you have env in your shebang, you're all set. However, if you've _hardwired_ the old path in your shebang, your script will run on the old perl [1]. - -The question that the shebang resolves—which program will run your script?—reminds us of a more fundamental distinction between [interpreted languages][73] and [compiled languages][74]. The former are those like bash, Perl, and Python, where you can cat a script and look inside it. The later, like C++, require [_compilation][75]_, the process whereby code is translated into machine language (the result is sometimes called a _binary_). This can be done with a command line utility like [g++][76]: - - $ g++ MyProgram.cpp -o MyProgram - -Compiled programs, such as the unix utilities themselves, tend to run faster. Don't try to cat a binary, such as ls, or it will spew out gibberish: - - $ cat $( which ls ) # don't do this! - - - -* * * - - -[1] Of course, depending on circumstances, you may very well want to stick with the old version of Perl or whatever's running your program. An update can have unforeseen consequences and this is the motivation for tools like [virtualenv][77] (Python), whose docs remind us: "_If an application works, any change in its libraries or the versions of those libraries can break the application_" ↑ - -## _bash_ - -We've thrown around the term _bash_ a few times but we haven't defined it. To do so, let's examine the special command, sh, which is more primitive than bash and came before it. To quote Wikipedia and the manual page: - -> The Bourne shell (sh) is a shell, or command-line interpreter, for computer operating systems. The shell is a command that reads lines from either a file or the terminal, interprets them, and generally executes other commands. It is the program that is running when a user logs into the system ... Commands can be typed directly to the running shell or can be put into a file and the file can be executed directly by the shell - -As it describes, sh is special because it's both a command interpreter and a command itself (usually found at /bin/sh). Put differently, you can run _myscript_ as: - - $ sh ./myscript - -or you can simply type: - - $ sh - -to start an interactive sh shell. If you're in this shell and run: - - $ ./myscript - -without specifying an interpreter or using a shebang, your script will be interpreted by sh by default. On most computers, however, the default shell is no longer sh but bash (usually located at /bin/bash). To mash up Wikipedia and the manual page: - -> The **B**ourne-**A**gain **SH**ell (bash) a Unix shell written by Brian Fox for the GNU Project as a free software replacement for the Bourne shell. bash is an sh-compatible command language interpreter that executes commands read from the standard input or from a file ... There are some subtle differences between bash and traditional versions of sh - -Like sh, bash is a command you can either invoke on a script or use to start an interactive bash shell. Read more on Stackoverflow: [Difference between sh and bash][78]. - -Which shell are you using right now? Almost certainly bash, but if you want to double check, there's a neat command [given here][79] to display your shell type: - - $ ps -p $$ - -There are more exotic shells, like [Z shell][80] and [tcsh][81], but they're beyond the scope of this article. - -## _chmod_ - -Let's take a closer look at how to use [chmod][82]. Remember the three domains: - -* _u_ \- user -* _g_ \- group -* _o_ \- other/world -and the three types of permission: -* _r_ \- read -* _w_ \- write -* _x_ \- execute -we can mix and match these how we like, using a plus sign to grant permissions according to the syntax: - -chmod entity+permissiontype - -or a minus sign to remove permissions: - -chmod entity-permissiontype - -E.g.: - - $ chmod u+x myfile # make executable for you - $ chmod g+rxw myfile # add read write execute permissions for the group - $ chmod go-wx myfile # remove write execute permissions for the group - # and for everyone else (excluding you, the user) - -You can also use _a_ for "all of the above", as in: - - $ chmod a-rwx myfile # remove all permissions for you, the group, - # and the rest of the world - -If you find the above syntax cumbersome, there's a numerical shorthand you can use with chmod. The only two I have memorized are _777_ and _755_: - - $ chmod 777 myfile # grant all permissions (rwxrwxrwx) - $ chmod 755 myfile # reserve write access for the user, - # but grant all other permissions (rwxr-xr-x) - -Read more about the numeric code [here][83]. In general, it's a good practice to allow your files to be writable by you alone, unless you have a compelling reason to share access to them. - -## _ssh_ - -In addition to chmod, there's another command it would be remiss not to mention. For many people, the first time they need to go to the command line, rather than the GUI, is to use the [Secure Shell (ssh)][84] protocol. Suppose you want to use a computer, but it's not the computer that's in front of you. It's a different computer in some other location—say, at your university, your company, or on the [Amazon cloud][16]. [ssh][85] is the command that allows you to log into a computer remotely over the network. Once you've sshed into a computer, you're in its shell and can run commands on it just as if it were your personal laptop. To ssh, you need to know your user name, the address of the host computer you want to log into, and the password [1]. The basic syntax is: - -ssh username@host - -For example: - - $ ssh username@myhost.university.edu - -If you're trying to ssh into a private computer and don't know the hostname, use its IP address (_username@IP-address_). - -ssh also allows you to run a command on the remote server without logging in. For instance, to list of the contents of your remote computer's home directory, you could run: - - $ ssh username@myhost.university.edu "ls -hl" - -Cool, eh? Moreover, if you have ssh access to a machine, you can copy files to or from it with the utility [rsync][12]—a great way to move data without an external hard drive. - -The file: - -~/.ssh/config - -determines ssh's behavior and you can create it if it doesn't exist (the dot in the name _.ssh_ confers invisibility—[see the discussion about dotfiles below][86]). On your own private computer, you can ssh into selected servers without having to type in a password by updating this configuration file. To do this, generate [rsa][87] ssh [keys][88]: - - $ mkdir -p ~/.ssh - $ cd ~/.ssh - $ ssh-keygen -t rsa -f localkey - -This will create two files on your computer, a public key: - -~/.ssh/localkey.pub - -and a private key: - -~/.ssh/localkey - -You can share your public key, but _do not give anyone your private key!_ Suppose you want to ssh into _myserver.com_. Normally, that's: - - $ ssh myusername@myserver.com - -Instead of doing this, add these lines to your _~/.ssh/config_ file: - - Host Myserver - HostName myserver.com - User myusername - IdentityFile ~/.ssh/localkey - -Next, cat your public key and paste it into: - -~/.ssh/authorized_keys - -on the remote machine (i.e., the _myserver.com_ computer). Now on your local computer, you can ssh into _myserver.com_ without a password: - - $ ssh Myserver - -You can also use this technique to push to [github.com][89] [2], without having to punch your password in each time, by pasting your public key into: - -Settings > SSH Keys > Add SSH Key - -on GitHub (read the [official tutorial][90]). - -If this is your first encounter with ssh, you'd be surprised how much of the work of the world is done by ssh. It's worth reading the extensive man page, which gets into matters of computer security and cryptography. - -* * * - - -[1] The host also has to enable ssh access. On Macintosh, for example, it's disabled by default, but you can turn it on, as instructed [here][91] ↑ -[2] As you get deeper into the game, tracking your scripts and keeping a single, stable version of them becomes crucial. [Git][92], a vast subject for [another tutorial][93], is the neat solution to this problem and the industry standard for version control. On the web [GitHub][94] provides free hosting of script repositories and connects to the command line via the git interface ↑ - -## Saving to a File; Stdout and Stderr - -To save to a file in unix, use an angle bracket: - - $ echo joe > junk.txt # save to file - $ cat junk.txt - joe - -To append to the end of a pre-existing file, use a double angle bracket: - - $ echo joe >> junk.txt # append to already-existing file - $ cat junk.txt - joe - joe - -Returning to our first script, [_myscript.sh][95]_, let's save the output to a file: - - $ ./myscript.sh > out.txt - mkdir: cannot create directory 'tmp': File exists - ls: cannot access myfile_2.txt: No such file or directory - - $ cat out.txt - /Users/oliver/tmp/tmp - myfile.txt - -This is interesting: _out.txt_ has its output. However, not everything went into _out.txt_, because some error messages were echoed to the console. What's going on here is that there are actually two [output streams][96]: _stdout_ (standard out) and _stderr_ (standard error). Look at the following figure from Wikipedia: - -![image][97] - -(Image credit: [Wikipedia: Standard streams][96]) - -Proper output goes into stdout while errors go into stderr. The syntax for saving stderr in unix is 2> as in: - - $ # save the output into out.txt and the error into err.txt - $ ./myscript.sh > out.txt 2> err.txt - $ cat out.txt - /Users/oliver/tmp/tmp - myfile.txt - $ cat err.txt - mkdir: cannot create directory 'tmp': File exists - ls: cannot access myfile_2.txt: No such file or directory - -When you think about it, the fact that output and error are separated is supremely useful. At work, sometimes we parallelize heavily and run 1000 instances of a script. For each instance, the error and output are saved separately. The 758th job, for example, might look like this: - -./myjob --instance 758 > out758.o 2> out758.e - -(I'm in the habit of using the suffixes _.o_ for output and _.e_ for error.) With this technique we can quickly scan through all 1000 _.e_ files and check if their size is 0. If it is, we know there was no error; if not, we can re-run the failed jobs. Some programs are in the habit of echoing run statistics or other information to stderr. This is an unfortunate practice because it muddies the water and, as in the example above, would make it hard to tell if there was an actual error. - -Output vs error is a distinction that many programming languages make. For example, in C++ writing to stdout and stderr is like this: - - cout << "some output" << endl; - cerr << "some error" << endl; - -In Perl it's: - - print STDOUT "some outputn"; - print STDERR "some errorn"; - -In Python it's: - - import sys - sys.stdout.write("some outputn") - sys.stderr.write("some errorn") - -and so on. - -## More on Stdout and Stderr; Redirection - -For the sake of completeness, we should note that you can redirect standard error to standard output and vice versa. Let's make sure we get the syntax of all things pertaining to stdout and stderr right: - - 1> # save stdout to (plain old > also works) - 2> # save stderr to - -as in: - - $ ./myscript.sh 1> out.o 2> out.e - $ ./myscript.sh > out.o 2> out.e # these two lines are identical - -What if we want to choose where things will be printed from _within_ our script? Then we can use the following syntax: - - &1 # standard out stream - &2 # standard error stream - -Let's examine five possible versions of our _Hello Kitty_ script: - - #!/bin/bash - # version 1 - echo "hello kitty" - - #!/bin/bash - # version 2 - echo "hello kitty" > somefile.txt - - #!/bin/bash - # version 3 - echo "hello kitty" > &1 - - #!/bin/bash - # version 4 - echo "hello kitty" > &2 - - #!/bin/bash - # version 5 - echo "hello kitty" > 1 - -Here's how they work: - -* _version 1_ \- echo "hello kitty" to stdout -* _version 2_ \- echo "hello kitty" to the file somefile.txt -* _version 3_ \- same as version 1 -* _version 4_ \- echo "hello kitty" to sterr -* _version 5_ \- echo "hello kitty" to _the file named 1_ -This illustrates the point of the ampersand syntax: it distinguishes between the output streams and files named _1_ or _2_. Let's try running script version 4 as a sanity check to make sure these scripts are working as expected: - - $ # output saved to file but error printed to console - $ ./hellokitty.sh > junk.txt - hello kitty - -_hello kitty_ is indeed stderr because it's echoed to the console, not saved into _junk.txt_. - -This syntax makes it easy to see how we could, e.g., redirect the standard error to standard output: - - $ ./somescript.sh 2> &1 # redirect stderr to stdout - -I rarely have occasion to do this and, although it's not something you need in your introductory unix toolkit, it's good to know. - -## Conditional Logic - -Conditional Logic is a universal feature of programming languages. The basic idea is, _if_ this condition, _then_ do something. It can be made more complex: _if_ this condition, _then_ do something; _else if_ that condition, _then_ do another thing; _else_ (if any other condition), _then_ do yet another thing. Let's see how to implement this in bash: - - $ a=joe - $ if [ $a == "joe" ]; then echo hello; fi - hello - -or: - - $ a=joe - $ if [ $a == "joe" ]; then echo hello; echo hello; echo hello; fi - hello - hello - hello - -The structure is: - -if [ _condition_ ]; then ... ; fi - -Everything between the words then and fi (_if_ backwards in case you didn't notice) will execute if the condition is satisfied. In other languages, this block is often defined by curly brackets: _{ }_. For example, in a Perl script, the same code would be: - - #!/usr/bin/env perl - - my $a="joe"; - - if ( $a eq "joe" ) - { - print "hellon"; - print "hellon"; - print "hellon"; - } - -In bash, _if_ is if, _else_ is else, and _else if_ is elif. In a script it would look like this: - - #!/bin/bash - - a=joe - - if [ $a == "joe" ]; then - echo hello; - elif [ $a == "doe" ]; then - echo goodbye; - else - echo "ni hao"; - fi - -You can also use a case statement to implement conditional logic. See an example of that [here][98]. - -Although I said in the intro that unix is the best place to start your computer science education, I have to admit that the syntax for _if-then_ logic is somewhat unwieldy—even unfriendly. Bash is a bad teaching language for conditional logic, [arrays][99], [hashes][100], etc. But that's only because its element is not heavy-duty programming with lots of functions, numerical operations, sophisticated data structures, and logic. Its mastery is over the quick and dirty, manipulating files and directories, and doing system stuff. I still maintain it's the proper starting point because of its wonderful tools, and because knowing its fundamentals is a great asset. Every language has its place in the programming ecosystem. Back in College, I stumbled on a physics book called [_The Tiger and the Shark: Empirical Roots of Wave-Particle Dualism_][101] by Bruce Wheaton. The book had a great epigraph: - -> It is like a struggle between a tiger and a shark, -each is supreme in his own element, -but helpless in that of the other. -_J.J. Thomson, 1925_ - -In our context, this would read: bash is supreme on the command line, but not inside of a script. - -## File Test Operators; Return or Exit Status - -_File Test Operators_ and _exit status_ are two completely different topics, but since they both go well with if statements, I'll discuss them here. File Test Operators are things you can stick in an if statement to give you information about a file. Two common problems are _(1)_ checking if your file exists and _(2)_ checking if it's non-zero size: Let's create two files, one empty and one not: - - $ touch emptyfile # create an empty file - $ echo joe > nonemptyfile # create a non-empty file - -The operator _-e_ tests for existence and _-s_ tests for non-zero-ness: - - $ file=emptyfile - $ if [ -e $file ]; then echo "exists"; if [ -s $file ]; then echo "non-0"; fi; fi - exists - - $ file=nonemptyfile - $ if [ -e $file ]; then echo "exists"; if [ -s $file ]; then echo "non-0"; fi; fi - exists - non-0 - -Read The Linux Documentation Project's discussion of file test operators [here][102]. - -Changing the subject altogether, you may be familiar with the idea of a return value in computer science. Functions can return a value upon completion. In unix, commands also have a return value or _exit code_, queryable with: - -$? - -This is usually employed to tell the user whether or not the command successfully executed. By convention, successful execution returns 0. For example: - - $ echo joe - joe - $ echo $? # query exit code of previous command - 0 - -Let's see how the exit code can be useful. We'll make a script, _test_exitcode.sh_, such that: - - $ cat test_exitcode.sh - #!/bin/bash - sleep 10 - -This script just pauses for 10 seconds. First, we'll let it run and then we'll interrupt it using _Cntrl-c_: - - $ ./test_exitcode.sh; # let it run - $ echo $? - 0 - - $ ./test_exitcode.sh; # interrupt it - ^C - $ echo $? - 130 - -The non-zero exit code tells us that it's failed. Now we'll try the same thing with an if statement: - - $ ./test_exitcode.sh - $ if [ $? == 0 ]; then echo "program succeeded"; else echo "program failed"; fi - program succeeded - - $ ./test_exitcode.sh; - ^C - $ if [ $? == 0 ]; then echo "program succeeded"; else echo "program failed"; fi - program failed - -In research, you might run hundreds of command-line programs in parallel. For each instance, there are two key questions: _(1)_ Did it finish? _(2)_ Did it run without error? Checking the exit status is the way to address the second point. You should always check the program you're running to find information about its exit code, since some use different conventions. Read The Linux Documentation Project's discussion of exit status [here][103]. - -_Question_: What's going on here? - - $ if echo joe; then echo joe; fi - joe - joe - -This is yet another example of bash allowing you to stretch syntax like silly putty. In this code snippet, - -echo joe - -is run, and its successful execution passes a _true_ return code to the if statement. So, the two _joe_s we see echoed to the console are from the statement to be evaluated and the statement inside the conditional. We can also invert this formula, doing something if our command fails: - - $ outputdir=nonexistentdir # set output dir equal to a nonexistent dir - $ if ! cd $outputdir; then echo "couldnt cd into output dir"; fi - -bash: pushd: nonexistentdir: No such file or directory - couldnt cd into output dir - - $ mkdir existentdir # make a test directory - $ outputdir=existentdir - $ if ! cd $outputdir; then echo "couldnt cd into output dir"; fi - $ # no error - now we're in the directory existentdir - -Did you follow that? (! means logical NOT in unix.) The idea is, we try to cd but, if it's unsuccessful, we echo an error message. This is a particularly useful line to include in a script. If the user gives an output directory as an argument and the directory doesn't exist, we exit. If it does exist, we cd into it and it's business as usual: - -if ! cd $outputdir; then echo "[error] couldn't cd into output dir"; exit; fi - -Without this line, the script will run in whatever directory it's in if cd fails. Once in lab, I was running a script that didn't have this kind of protection. The output directory wasn't found and the script starting making and deleting files in the wrong directory. It was powerfully uncool! - -We can implement similar constructions using the && and || operators rather than an if statement. Let's see how this works by making some test files: - - $ touch file{1..4} - $ ls - file1 file2 file3 file4 - -The && operator will chug through a chain of commands and keep on going until one of the commands fails, as in: - - $ ( ls file1 ) && ( ls file2 ) && ( ls file3 ) && ( ls file4 ) - file1 - file2 - file3 - file4 - - $ ( ls file1 ) && ( ls file2 ) && ( ls fileX ) && ( ls file4 ) - file1 - file2 - ls: cannot access fileX: No such file or directory - -In contrast, the || operator will proceed through the command chain and _stop_ after the first successful one, as in: - - $ ( ls file1 ) || ( ls file2 ) || ( ls file3 ) || ( ls file4 ) - file1 - - $ ( ls fileX ) || ( ls fileY ) || ( ls fileZ ) || ( ls file4 ) - ls: cannot access fileX: No such file or directory - ls: cannot access fileY: No such file or directory - ls: cannot access fileZ: No such file or directory - file4 - -## Basic Loops - -In programming, loops are a way of performing operations iteratively. Loops come in different flavors, but the _for loop_ and _while loop_ are the most basic. In bash, we can implement a for loop like this: - - $ for i in 1 2 3; do echo $i; done - 1 - 2 - 3 - -The structure is: - -for _variable_ in _list_; do ... ; done - -Put anything you like in the list: - - $ for i in 1 2 hello; do echo $i; done - 1 - 2 - hello - -Many other languages wouldn't let you get away with combining data types in the iterations of a loop, but this is a recurrent bash theme: it's fast; it's loose; it's malleable. - -To count from 1 to 10, try: - - $ for i in {1..10}; do echo -n "$i "; done; echo - 1 2 3 4 5 6 7 8 9 10 - -But if we can just write: - - $ echo {1..10} - -why do we need a loop here? Loops really come into their own in bash when—no surprise!—we're dealing with files, paths, and commands. For example, to loop through all of the text files in the cwd, use: - - $ for i in *.txt; do echo $i; done - -Although this is nearly the same as: - - $ ls *.txt - -the former construction has the advantage that we can stuff as much code as we like in the block between do and done. Let's make a random directory structure like so: - - $ mkdir -p myfolder{1..3}/{X,Y} - -We can populate it with token files (fodder for our example) via a loop: - - $ j=0; for i in myfolder*/*; do echo "*** "$i" ***"; touch ${i}/a_${j}.txt ${i}/b_${j}.txt; ((j++)); done - -In bash, ((j++)) is a way of incrementing j. We echo $i to get some visual feedback as the loop iterates. Now our directory structure looks like this: - -![image][104] - - -To practice loops, suppose we want to find any file that begins with _b_ in any subfolder and make a symbolic link to it from the cwd: - - $ for i in myfolder*/*/b*; do echo "*** "$i" ***"; ln -s $i; done - -As we learned above, a link is not a copy of a file but, rather, a kind of pointer that allows us to access a file from a path other than the one where it actually resides. Our loop yields the links: - - b_0.txt -> myfolder1/X/b_0.txt - b_1.txt -> myfolder1/Y/b_1.txt - b_2.txt -> myfolder2/X/b_2.txt - b_3.txt -> myfolder2/Y/b_3.txt - b_4.txt -> myfolder3/X/b_4.txt - b_5.txt -> myfolder3/Y/b_5.txt - -allowing us to access the _b_ files from the cwd. - -I can't overstate all the heroic things you can do with loops in bash. Suppose we want to change the extension of any text file that begins with _a_ and resides in an _X_ subfolder from _.txt_ to _.html_: - - $ for i in myfolder*/X/a*.txt; do echo "*** "$i" ***"; j=$( echo $i | sed 's|.txt|.html|' ); echo $j; mv $i $j; echo; done - -But I've jumped the gun! This example features three things we haven't learned yet: command substitution, piping, and sed. You should revisit it after reading those sections, but the idea is that the variable _j_ stores a path that looks like our file's but has the extension replaced. And you see that a knowledge of loops is like a stick of dynamite you can use to blow through large numbers of files. - -Here's another contrived example with these yet-to-be-discussed techniques: - - $ for i in $( echo $PATH | tr ":" " " ); do echo "*** "$i" ***"; ls $i | head; echo; done | less - -Can you guess what this does? It shows the first ten commands in each folder in our PATH—not something you'd likely need to do, but a demonstration of the fluidity of these constructions. - -If we want to run a command or script in parallel, we can do that with loops, too. [gzip][105] is a utility to compress files, thereby saving hard drive space. To compress all text files in the cwd, in parallel, do: - - $ for i in *.txt; do { echo $i; gzip $i & }; done - -But I've gotten ahead of myself again. We'll leave the discussion of this example to the section on processes. - -The structure of a while loop is: - -while _condition_; do ... ; done - -I use while loops much less than for loops, but here's an example: - - $ x=1; while ((x <= 3)); do echo $x; ((x++)); done - 1 - 2 - 3 - -The while loop can also take input from a file. Suppose there's a file _junk.txt_ such that: - - $ cat junk.txt - 1 - 2 - 3 - -You can iterate over this file as such: - - $ while read x; do echo $x; done < junk.txt - 1 - 2 - 3 - -## Arguments to a Script - -Now that we've covered basic [control flow][106], let's return to the subject of scripting. An important question is, how can we pass arguments to our script? Let's make a script called _hellokitty.sh_: - - #!/bin/bash - - echo hello - -Try running it: - - $ chmod 755 hellokitty.sh - $ ./hellokitty.sh - hello - -We can change it to the following: - - #!/bin/bash - - echo hello $1 - -Now: - - $ ./hellokitty.sh kitty - hello kitty - -In bash $1 represents the first argument to the script, $2 the second, and so on. If our script is: - - #!/bin/bash - - echo $0 - echo hello $1 $4 - -Then: - - $ ./hellokitty.sh my sweet kitty cat - ./hellokitty.sh - hello my cat - -In most programming languages, arguments passed in on the command line are stored as an array. Bash stores the _n_th element of this array in the variable $_n_. $0 is special and refers to the name of the script itself. - -For casual scripts this suits us well. However, as you go on to write more involved programs with many options, it becomes impractical to rely on the position of an argument to determine its function in your script. The proper way to do this is using _flags_ that can be deployed in arbitrary order, as in: - -command --flag1 1 --flag2 1 --flag3 5 - -or, in short form: - -command -f1 1 -f2 1 -f3 5 - -You can do this with the command [getopts][107], but it's sometimes easier just to write your own options parser. Here's a sample script called [_test_args][108]_. Although a case statement would be a good way to handle numerous conditions, I'll use an if statement: - - #!/bin/bash - - helpmessage="This script showcases how to read arguments" - - ### get arguments - # while input array size greater than zero - while (($# > 0)); do - if [ "$1" == "-h" -o "$1" == "-help" -o "$1" == "--help" ]; then - shift; - echo "$helpmessage" - exit; - elif [ "$1" == "-f1" -o "$1" == "--flag1" ]; then - # store what's passed via flag1 in var1 - shift; var1=$1; shift - elif [ "$1" == "-f2" -o "$1" == "--flag2" ]; then - shift; var2=$1; shift - elif [ "$1" == "-f3" -o "$1" == "--flag3" ]; then - shift; var3=$1; shift - # if unknown argument, just shift - else - shift - fi - done - - ### main - # echo variable if not empty - if [ ! -z $var1 ]; then echo "flag1 passed "$var1; fi - if [ ! -z $var2 ]; then echo "flag2 passed "$var2; fi - if [ ! -z $var3 ]; then echo "flag3 passed "$var3; fi - -This has some things we haven't seen yet: - -* $# is the size of our input argument array -* shift pops an element off of our array (the same as in Perl) -* exit exits the script -* -o is logical OR in unix -* -z checks if a variable is empty -The code loops through the argument array and keeps popping off elements until the array size is zero, whereupon it exits the loop. For example, one might run this script as: - - $ ./test_args --flag1 x -f2 y --flag3 zzz - flag1 passed x - flag2 passed y - flag3 passed zzz - -To spell out how this works, the first argument is _\--flag1_. Since this matches one of our checks, we shift. This pops this element out of our array, so the first element, $1, becomes _x_. This is stored in the variable _var1_, then there's another shift and $1 becomes _-f2_, which matches another condition, and so on. - -The flags can come in any order: - - $ ./test_args --flag3 x --flag1 zzz - flag1 passed zzz - flag3 passed x - - $ ./test_args --flag2 asdf - flag2 passed asdf - -We're brushing up against the outer limits of bash here. My prejudice is that you usually shouldn't go this far with bash, because its limitations will come sharply into focus if you try to do too-involved scripting. Instead, use a more friendly language. In Perl, for example, the array containing inputs is @ARGV; in Python, it's sys.argv. Let's compare these common scripting languages: - -| ----- | -| **Bash** | **Perl** | **Python** | **Description** | -| $0 | $0 | sys.argv[0] | Name of Script Itself | -| $* | | | String Containing All Input Arguments | -| ("$@") | @ARGV | sys.argv | Array or List Containing All Input Arguments [1] | -| $1 | $ARGV[0] | sys.argv[1] | First Argument | -| $2 | $ARGV[1] | sys.argv[2] | Second Argument | - -Perl has a [Getopt][109] package that is convenient for reading arguments, and Python has an even better one called [argparse][110]. Their functionality is infinitely nicer than bash's, so steer clear of bash if you're going for a script with lots of options. - -* * * - - -[1] The distinction between $* and $@ is knotty. Dive into these subtleties [on Stackoverflow][111] ↑ - -## Multi-Line Comments, Multi-Line Strings in Bash - -Let's continue in the realm of scripting. You can do a multi-line comment in bash with an if statement: - - # multi-line comment - if false; then - echo hello - echo hello - echo hello - fi - -(Yes, this is a bit of a hack!) - -Multi-line strings are handy for many things. For example, if you want a help section for your script, you can do it like this: - - cat <<_EOF_ - - Usage: - - $0 --flag1 STRING [--flag2 STRING] [--flag3 STRING] - - Required Arguments: - - --flag1 STRING This argument does this - - Options: - - --flag2 STRING This argument does that - --flag3 STRING This argument does another thing - - _EOF_ - -How does this syntax work? Everything between the __EOF__ tags comprises the string and is printed. This is called a [_Here Document][112]_. Read The Linux Documentation Project's discussion of Here Documents [here][113]. - -## Source and Export - -_Question_: If we create some variables in a script and exit, what happens to those variables? Do they disappear? The answer is, yes, they do. Let's make a script called _test_src.sh_ such that: - - $ cat ./test_src.sh - #!/bin/bash - - myvariable=54 - echo $myvariable - -If we run it and then check what happened to the variable on our command line, we get: - - $ ./test_src.sh - 54 - $ echo $myvariable - -The variable is undefined. The command [source][114] is for solving this problem. If we want the variable to persist, we run: - - $ source ./test_src.sh - 54 - $ echo $myvariable - 54 - -and—voilà!—our variable exists in the shell. An equivalent syntax for sourcing uses a dot: - - $ . ./test_src.sh # this is the same as "source ./test_src.sh" - 54 - -But now observe the following. We'll make a new script, _test_src_2.sh_, such that: - - $ cat ./test_src_2.sh - #!/bin/bash - - echo $myvariable - -This script is also looking for _$myvariable_. Running it, we get: - - $ ./test_src_2.sh - -Nothing! So _$myvariable_ is defined in the shell but, if we run another script, its existence is unknown. Let's amend our original script to add in an export: - - $ cat ./test_src.sh - #!/bin/bash - - export myvariable=54 # export this variable - echo $myvariable - -Now what happens? - - $ ./test_src.sh - 54 - $ ./test_src_2.sh - -Still nothing! Why? Because we didn't source _test_src.sh_. Trying again: - - $ source ./test_src.sh - 54 - $ ./test_src_2.sh - 54 - -So, at last, we see how to do this. If we want access on the shell to a variable which is defined inside a script, we must source that script. If we want _other_ scripts to have access to that variable, we must source plus export. - -## Dotfiles (_.bashrc_ and _.bash_profile_) - -Dotfiles are simply files that begin with a dot. We can make a test one as follows: - - $ touch .test - -Such a file will be invisible in the GUI and you won't see it with vanilla ls either. (This works the same way for directories.) The only way to see it is to use the list _all_ option: - -ls -al - -or to list it explicitly by name. This is useful for files that you generally want to keep hidden from the user or discourage tinkering with. - -Many programs, such as bash, [Vim][55], and [Git][92], are highly configurable. Each uses dotfiles to let the user add functionality, change options, switch key bindings, etc. For example, here are some of the dotfiles files each program employs: - -* bash - _.bashrc_ -* vim - _.vimrc_ -* git - _.gitconfig_ -The most famous dotfile in my circle is _.bashrc_ which resides in HOME and configures your bash. Actually, let me retract that: let's say _.bash_profile_ instead of _.bashrc_ (read about the difference [here][115]). In any case, the idea is that this dotfile gets executed as soon as you open up the terminal and start a new session. It is therefore ideal for setting your PATH and other variables, adding functions ([like this one][116]), creating _aliases_ (discussed below), and doing any other setup related chore. For example, suppose you download a new program into /some/path/to/prog and you want to add it to your PATH. Then in your _.bash_profile_ you'd add: - - export PATH=/some/path/to/prog:$PATH - -Recalling how export works, this will allow any programs we run on the command line to have access to our amended PATH. Note that we're adding this to the front of our PATH (so, if the program exists in our PATH already, the existing copy will be superseded). Here's an example snippet of my setup file: - - PATH=/apps/python/2.7.6/bin:$PATH # use this version of Python - PATH=/apps/R/3.1.2/bin:$PATH # use this version of R - PATH=/apps/gcc/4.6.0/bin/:$PATH # use this version of gcc - export PATH - -There is much ado about _.bashrc_ (read _.bash_profile_) and it inspired one of the greatest unix blog-post titles of all time: [_Pimp my .bashrc][117]_—although this blogger is only playing with his prompt, as it were. As you go on in unix and add things to your _.bash_profile_, it will evolve into a kind of fingerprint, optimizing bash in your own unique way (and potentially making it difficult for others to use). - -If you have multiple computers, you'll want to recycle much of your program configurations on all of them. My co-worker uses a nice system I've adopted where the local and global aspects of setup are separated. For example, if you wanted to use certain aliases across all your computers, you'd put them in a global settings file. However, changes to your PATH might be different on different machines, so you'd store this in a local settings file. Then any time you change computers you can simply copy the global files and get your familiar setup, saving lots of work. A convenient way to accomplish this goal of a unified shell environment across all the systems you work on is to put your dotfiles on a server, like [GitHub][94] or [Bitbucket][118], you can access from anywhere. This is exactly what I've done and you can [get the up-to-date versions of my dotfiles on GitHub][119]. - -Here's a sketch of how this idea works: in HOME make a _.dotfiles/bash_ directory and populate it with your setup files, using a suffix of either _local_ or _share_: - - $ ls -1 .dotfiles/bash/ - bash_aliases_local - bash_aliases_share - bash_functions_share - bash_inirun_local - bash_paths_local - bash_settings_local - bash_settings_share - bash_welcome_local - bash_welcome_share - -When _.bash_profile_ is called at the startup of your session, it sources all these files: - - # the directory where bash configuration files reside - INIT_DIR="${HOME}/.dotfiles/bash" - - # to make local configurations, add these files into this directory: - # bash_aliases_local - # bash_paths_local - # bash_settings_local - # bash_welcome_local - - # this line, e.g., protects the functionality of rsync by only turning on the below if the shell is in interactive mode - # In particular, rsync fails if things are echo-ed to the terminal - [[ "$-" != *i* ]] && return - - # bash welcome - if [ -e "${INIT_DIR}/bash_welcome_local" ]; then - cat ${INIT_DIR}/bash_welcome_local - elif [ -e "${INIT_DIR}/bash_welcome_share" ]; then - cat ${INIT_DIR}/bash_welcome_share - fi - - #--------------------LOCAL------------------------------ - # aliases local - if [ -e "${INIT_DIR}/bash_aliases_local" ]; then - source "${INIT_DIR}/bash_aliases_local" - echo "bash_aliases_local loaded" - fi - - # settings local - if [ -e "${INIT_DIR}/bash_settings_local" ]; then - source "${INIT_DIR}/bash_settings_local" - echo "bash_settings_local loaded" - fi - - # paths local - if [ -e "${INIT_DIR}/bash_paths_local" ]; then - source "${INIT_DIR}/bash_paths_local" - echo "bash_paths_local loaded" - fi - - #---------------SHARE----------------------------- - # aliases share - if [ -e "${INIT_DIR}/bash_aliases_share" ]; then - source "${INIT_DIR}/bash_aliases_share" - echo "bash_aliases_share loaded" - fi - - # settings share - if [ -e "${INIT_DIR}/bash_settings_share" ]; then - source "${INIT_DIR}/bash_settings_share" - echo "bash_settings_share loaded" - fi - - # functions share - if [ -e "${INIT_DIR}/bash_functions_share" ]; then - source "${INIT_DIR}/bash_functions_share" - echo "bash_functions_share loaded" - fi - -A word of caution: echoing things in your _.bash_profile_, as I'm doing here, can be dangerous and break the functionaly of utilities like scp and rsync. However, we protect against this with the cryptic line near the top. - -Taking care of bash is the hard part. Other programs are less of a chore because, even if you have different programs in your PATH on your home and work computers, you probably want everything else to behave the same. To accomplish this, just drop all your other configuration files into your _.dotfiles_ repository and link to them from your home directory: - - .gitconfig -> .dotfiles/.gitconfig - .vimrc -> .dotfiles/.vimrc - -## Working Faster with Readline Functions and Key Bindings - -If you've started using the terminal extensively, you might find that things are a bit slow. Perhaps you need some long command you wrote yesterday and you don't want to write the damn thing again. Or, if you want to jump to the end of a line, it's tiresome to move the cursor one character at a time. Failure to immediately solve these problems will push your productivity back into the stone age and you may end up swearing off the terminal as a Rube Goldberg-ian dystopia. So—enter keyboard shortcuts! - -The backstory about shortcuts is that there are two massively influential text editors, [Emacs][57] and [Vim][55], whose users—to be overdramatic—are divided into two warring camps. Each program has its own conventions for shortcuts, like jumping words with your cursor, and in bash they're Emacs-flavored by default. But you can toggle between either one: - - $ set -o emacs # Set emacs-style key bindings (this is the default) - $ set -o vi # Set vi-style key bindings - -Although I prefer Vim as a text-editor, I use Emacs key bindings on the command line. The reason is that in Vim there are multiple modes (normal mode, insert mode, command mode). If you want to jump to the front of a line, you have to switch from insert mode to normal mode, which breaks up the flow a little. In Emacs there's no such complication. Emacs commands usually start with the _Control_ key or the _Meta_ key (usually _Esc_). Here are some things you can do: - -* _Cntrl-a_ \- jump cursor to beginning of line -* _Cntrl-e_ \- jump cursor to end of line -* _Cntrl-k_ \- delete to end of line -* _Cntrl-u_ \- delete to beginning of line -* _Cntrl-w_ \- delete back one word -* _Cntrl-y_ \- paste (yank) what was deleted with the above shortcuts -* _Cntrl-r_ \- reverse-search history for a given word -* _Cntrl-c_ \- kill the process running in the foreground; don't execute current line on the command line -* _Cntrl-z_ \- suspend the process running in the foreground -* _Cntrl-l_ \- clear screen. (this has an advantage over the unix command clear in that it works in the Python, MySQL, and other shells) -* _Cntrl-d_ \- [end of transmission][120] (in practice, often synonymous with quit - e.g., exiting the Python or MySQL shells) -* _Cntrl-s_ \- freeze screen -* _Cntrl-q_ \- un-freeze screen -_These are supremely useful!_ I use these numerous times a day. (On the Mac, the first three even work in the Google search bar!) The first bunch of these fall under the umbrella of [_ReadLine Functions][121]_ (read GNU's extensive documentation [here][122]). There are actually tons more, and you can see them all by entering: - - $ bind -P # show all Readline Functions and their key bindings - $ bind -l # show all Readline Functions - -Four of the most excellent Readline Functions are: - -* _forward-word_ \- jump cursor forward a word -* _backward-word_ \- jump cursor backward a word -* _history-search-backward_ \- scroll through your bash history backward -* _history-search-forward_ \- scroll through your bash history forward -For the first two—which are absolutely indispensable—you can use the default Emacs way: -* _Meta-f_ \- jump forward one word -* _Meta-b_ \- jump backward one word -However, reaching for the _Esc_ key is a royal pain in the ass—you have to re-position your hands on the keyboard. This is where _key-binding_ comes into play. Using the command bind, you can map a Readline Function to any key combination you like. Of course, you should be careful not to overwrite pre-existing key bindings that you want to use. I like to map the following keys to these Readline Functions: - -* _Cntrl-forward-arrow_ \- forward-word -* _Cntrl-backward-arrow_ \- backward-word -* _up-arrow_ \- history-search-backward -* _down-arrow_ \- history-search-forward -In my _.bash_profile_ (or, more accurately, in my global bash settings file) I use: - - # make cursor jump over words - bind '"e[5C": forward-word' # control+arrow_right - bind '"e[5D": backward-word' # control+arrow_left - - # make history searchable by entering the beginning of command - # and using up and down keys - bind '"e[A": history-search-backward' # arrow_up - bind '"e[B": history-search-forward' # arrow_down - -(although these may not work universally [1].) How does this cryptic symbology translate into these particular keybindings? There's a neat trick you can use, to be revealed in the next section. - -_Tip_: On Mac, you can move your cursor to any position on the line by holding down _Option_ and clicking your mouse there. I rarely use this, however, because it's faster to make your cursor jump via the keyboard. - -* * * - - -[1] If you have trouble getting this to work on OS's terminal, try [iTerm2][123] instead, as described [here][124] ↑ - -## More on Key Bindings, the ASCII Table, _Control-v_ - -Before we get to the key binding conundrum, let's review [ASCII][125]. This is, simply, a way of mapping every character on your keyboard to a numeric code. As Wikipedia puts it: - -> The American Standard Code for Information Interchange (ASCII) is a character-encoding scheme originally based on the English alphabet that encodes 128 specified characters—the numbers 0-9, the letters a-z and A-Z, some basic punctuation symbols, some control codes that originated with Teletype machines, and a blank space—into the 7-bit binary integers. - -For example, the character _A_ is mapped to the number _65_, while _q_ is _113_. Of special interest are the _control characters_, which are the representations of things that cannot be printed like _return_ or _delete_. Again [from Wikipedia][126], here is the portion of the ASCII table for these control characters: - -| ----- | -| **Binary** | **Oct** | **Dec** | **Hex** | **Abbr** | [**a]** | [**b]** | [**c]** | **Name** | -| 000 0000 | 000 | 0 | 00 | NUL | ␀ | ^@ | - -[1]: http://en.wikipedia.org/wiki/Unix -[2]: http://en.wikipedia.org/wiki/Linux -[3]: http://en.wikipedia.org/wiki/Command-line_interface -[4]: http://en.wikipedia.org/wiki/Terminal_emulator -[5]: http://en.wikipedia.org/wiki/Shell_script -[6]: http://en.wikipedia.org/wiki/Bourne-again_shell -[7]: http://www.oliverelliott.org/static/article/img/terminal_591.png -[8]: http://www.gnu.org/software/coreutils/ -[9]: http://en.wikipedia.org/wiki/GNU_Core_Utilities -[10]: /static/img/letter_600.jpg -[11]: https://class.coursera.org/startup-001 -[12]: http://ss64.com/bash/rsync.html -[13]: http://www.perl.org -[14]: http://www.python.org -[15]: https://www.gnupg.org/index.html -[16]: http://aws.amazon.com/ec2/ -[17]: http://nginx.org/ -[18]: http://en.wikipedia.org/wiki/Graphical_user_interface -[19]: http://www.youtube.com/watch?v=WiX7GTelTPM -[20]: http://upload.wikimedia.org/wikipedia/commons/c/cd/Unix_timeline.en.svg -[21]: /article/computing/ref_unix/ -[22]: http://www.oliverelliott.org/static/article/img/terminal_119.png -[23]: http://en.wikipedia.org/wiki/Cmd.exe -[24]: http://en.wikipedia.org/wiki/Windows_PowerShell -[25]: http://www.putty.org -[26]: https://chrome.google.com/webstore/detail/secure-shell/pnhechapfaindjhompbnflcldabbghjo?hl=en -[27]: http://mobaxterm.mobatek.net -[28]: http://en.wikipedia.org/wiki/Darwin_(operating_system) -[29]: http://aws.amazon.com/free/ -[30]: http://www.ubuntu.com/download -[31]: http://www.linuxmint.com -[32]: https://getfedora.org/ -[33]: http://www.centos.org -[34]: https://www.cygwin.com/ -[35]: /article/computing/tips_mac/#InstalltheGNUCoreutils -[36]: http://www.oliverelliott.org/static/article/img/root_dir_structure.png -[37]: http://en.wikipedia.org/wiki/Home_directory -[38]: http://www.oliverelliott.org/static/article/img/home_dir_structure.png -[39]: http://www.oliverelliott.org/static/article/img/dir_struct_1125.png -[40]: http://www.thegeekstuff.com/2010/09/linux-file-system-structure/ -[41]: http://en.wikipedia.org/wiki/Uniform_resource_locator -[42]: http://www.e-reading.biz/htmbook.php/orelly/unix2.1/lrnunix/ch03_01.htm -[43]: http://ss64.com/bash/ls.html -[44]: http://www.oliverelliott.org/static/article/img/ls.png -[45]: http://www.oliverelliott.org/static/article/img/ls1.png -[46]: http://www.oliverelliott.org/static/article/img/lshl.png -[47]: http://unixhelp.ed.ac.uk/CGI/man-cgi?finger -[48]: http://en.wikipedia.org/wiki/Hidden_file_and_hidden_directory -[49]: http://www.oliverelliott.org/static/article/img/lsal.png -[50]: http://en.wikipedia.org/wiki/Glob_%28programming%29 -[51]: http://macintoshgarden.org/games/prince-of-persia -[52]: http://en.wikipedia.org/wiki/Microsoft_Word -[53]: http://www.youtube.com/watch?v=znlFu_lemsU -[54]: http://en.wikipedia.org/wiki/Grep -[55]: http://www.vim.org -[56]: http://www.nano-editor.org/ -[57]: http://www.gnu.org/software/emacs/ -[58]: /article/computing/wik_vim/ -[59]: http://www.sublimetext.com/ -[60]: http://aquamacs.org/ -[61]: http://www.peterborgapps.com/smultron/ -[62]: http://en.wikipedia.org/wiki/Escape_character -[63]: http://www.oliverelliott.org/static/article/img/bash_prompt_426.png -[64]: http://en.wikipedia.org/wiki/X_Window_System -[65]: http://www.oliverelliott.org/static/article/img/thepath_410.png -[66]: http://en.wikipedia.org/wiki/Symbolic_link -[67]: http://ss64.com/bash/ln.html -[68]: http://www.oliverelliott.org/static/article/img/myscript_634.png -[69]: https://www.talisman.org/~erlkonig/documents/commandname-extensions-considered-harmful.shtml -[70]: http://en.wikipedia.org/wiki/Shebang_(Unix) -[71]: http://en.wikipedia.org/wiki/Software_portability -[72]: http://en.wikipedia.org/wiki/Env -[73]: http://en.wikipedia.org/wiki/Interpreted_language -[74]: http://en.wikipedia.org/wiki/Compiled_language -[75]: http://en.wikipedia.org/wiki/Compiler -[76]: http://gcc.gnu.org -[77]: https://virtualenv.pypa.io/en/latest/ -[78]: http://stackoverflow.com/questions/5725296/difference-between-sh-and-bash -[79]: http://www.cyberciti.biz/tips/how-do-i-find-out-what-shell-im-using.html -[80]: http://en.wikipedia.org/wiki/Z_shell -[81]: http://en.wikipedia.org/wiki/Tcsh -[82]: http://ss64.com/bash/chmod.html -[83]: http://en.wikipedia.org/wiki/Chmod -[84]: http://en.wikipedia.org/wiki/Secure_Shell -[85]: http://www.ss64.com/bash/ssh.html -[86]: /article/computing/tut_unix/#Dotfilesbashrcandbash_profile -[87]: http://en.wikipedia.org/wiki/RSA_(cryptosystem) -[88]: http://en.wikipedia.org/wiki/Public-key_cryptography -[89]: https://github.com/ -[90]: https://help.github.com/articles/generating-ssh-keys/ -[91]: /article/computing/tips_mac/#sshintoYourMac -[92]: http://git-scm.com/ -[93]: /article/computing/wik_git/ -[94]: https://github.com -[95]: /static/article/example/myscript.html -[96]: http://en.wikipedia.org/wiki/Standard_streams -[97]: http://www.oliverelliott.org/static/article/img/Stdstreams-notitle.svg.png -[98]: http://bash.cyberciti.biz/guide/The_case_statement -[99]: http://en.wikipedia.org/wiki/Array_data_structure -[100]: http://en.wikipedia.org/wiki/Hash_table -[101]: http://www.amazon.com/The-Tiger-Shark-Empirical-Wave-Particle/dp/0521358922 -[102]: http://www.tldp.org/LDP/abs/html/fto.html -[103]: http://tldp.org/LDP/abs/html/exit-status.html -[104]: http://www.oliverelliott.org/static/article/img/lsdirtree_234.jpg -[105]: http://ss64.com/bash/gzip.html -[106]: http://en.wikipedia.org/wiki/Control_flow -[107]: http://wiki.bash-hackers.org/howto/getopts_tutorial -[108]: /static/article/example/test_args.html -[109]: http://perldoc.perl.org/Getopt/Long.html -[110]: https://docs.python.org/2/howto/argparse.html -[111]: http://stackoverflow.com/questions/12314451/accessing-bash-command-line-args-vs -[112]: http://en.wikipedia.org/wiki/Here_document -[113]: http://www.tldp.org/LDP/abs/html/here-docs.html -[114]: http://ss64.com/bash/source.html -[115]: http://www.joshstaiger.org/archives/2005/07/bash_profile_vs.html -[116]: http://www.virtualblueness.net/linux-gazette/109/marinov.html -[117]: http://zxvf-linux.blogspot.com/2013/05/pimp-my-bashrc.html -[118]: https://bitbucket.org -[119]: https://github.com/gitliver/.dotfiles -[120]: http://en.wikipedia.org/wiki/End-of-transmission_character -[121]: http://en.wikipedia.org/wiki/GNU_Readline -[122]: http://tiswww.case.edu/php/chet/readline/readline.html -[123]: http://iterm2.com/ -[124]: /article/computing/tips_mac/#InstalliTerm2 -[125]: http://en.wikipedia.org/wiki/ASCII -[126]: http://en.wikipedia.org/wiki/ASCII#ASCII_control_characters |