May the
source be with you,
and remember the KISS principle ;-)
Softpanorama: (slightly skeptical) Open Source Software Educational Society
Computer science related content (under the Open Content License) for self-study. Online resource for students in underfunded schools and universities with a special emphasis on KISS("Keep It Simple Stupid") approach to software engineering, algorithms Perl and other scripting languages, computer security, Sysadmin Certification, extremely useful but undervalued tools ( Orthodox file managers, Folding Xedit-style editors) , the importance of using the best available computer books for self-study (especially books by Donald Knuth), open source software development problems, and the problem of information overload.
| Recommended Books | Recommended Links | |||||
| Scripts Collections |
Note: Highly recommended shell site is SHELLdorado by Heiner Steven.
This collection of links is oriented on students. As David Korn recently said:
There are many people who use UNIX or Linux who IMHO do not understand UNIX. UNIX is not just an operating system, it is a way of doing things, and the shell plays a key role by providing the glue that makes it work. The UNIX methodology relies heavily on reuse of a set of tools rather than on building monolithic applications. Even perl programmers often miss the point, writing the heart and soul of the application as perl script without making use of the UNIX toolkit.
I recommend ksh93 (especially Tksh implementation) as your open source shell. And I think that the most valuable feature of any shell is pipes support. From the programming language design point of view the main advantage of ksh93 is more powerful piping constructs and that's make ksh93 superior to Perl in small scripts. See also Softpanorama Perl bibliography, Perl links (Perl can be considered as an extension of the ksh+awk and in case of larger applications can and should be used along or instead of shell scripts). For NT an interesting and pretty new possibility is to use Jscript. That provide an advantage of using the same language for HTML scripting and shell programming. I do not recommend VBScript but your mileage may vary.
Because ksh93 better supports pipes I strongly prefer ksh93 over bash. IMHO ksh93 is the most interesting shell that is still within human abilities to learn (Perl and z-shell might be actually beyond human abilities ;-). And it can be used both in Windows environment and Unix, although in windows environment it's much less powerful and stable. But my students do amazing things with ksh93 in Windows environment. You probably need to buy a book to learn this shell -- I was not able to find a decent free tutorial on the Web for ksh93. I would like to stress that both Born-shell and Korn-shell are were designed by original Unix developers (Born shell is based on Mashey shell which in turn was an extension of the original Thompson shell which in turn was derivative of Multix shell), very talented people and are under-appreciated as languages. Especially such things as pipes, named pipes, regular expressions and evaluation mechanisms (backquote) are grossly under-appreciated by many users and system administrators. Even good startup files are a problem :-(.
I would like to reiterate that I prefer ksh93 over bash. The ksh93 syntax is a superset of the original Bourne Shell, and therefore pretty much a superset of the Bourne Again Shell. In the ksh93, the typeset command provides a number of extra features for things such as presenting non-decimal numbers, and converting from upper case text to lower case and vice versa. A very cool feature of ksh93 is `|&` which is a way of getting a pipe to a background process (co-processes). You can also call it a bi-directional pipe. You can then use `read` and `print` to send and receive stuff from the process started with `|&`. For example, if I need to do floating point math, I can start `bc` in the background with `|&`. Interesting. with zsh, |& is a pipe which takes both stdout and stderr, similar to >& and >>& for redirection. It's handy, especially when you need to use a program with a rather verbose --help option. Bash doesn't recognise |& at all :-(. Consider the following example derived from discussion after Slashdot David Korn interview:
x="hello"
echo "goodbye" | read x
echo $x
In both bash and pdksh, running the script would incorrectly return
"hello", because the "read x" would be run in a subshell
with its own environment. The subshell would then close without affecting the
instance of x in the main shell. In ksh, however, putting 'read' at the end of a
pipeline does not create a subshell, so running the script would return
"goodbye" just as the naive reader of the code might expect. Scripts
that rely on "echo $string | read x y junk" are
convenient to parse space-separated lists, and of course these statements will
not work in Bash (you have to do a few backtick/sed operations). Actually, you
can still do it bash using shell builtins. Save off the current argument list
($*) if necessary, and then:
set -- $string
x=$1; shift
y=$1; shift
junk="$*"
As you can see the ksh implementation is much better.
The prev. paragraph illustrates my main conviction that the most important concept traditionally associated with shells is piping -- I urge you to learn this concept as deeply as possible -- actually this is a pretty powerful programming paradigm. And that is the major weakness of Perl -- it does not support piping and coroutines as first level constructs -- you cannot write a loop that takes data from the pipe like in ksh93, not you can write a function that serve a stage is the pipeline. Other than that Perl almost killed shell scripting for non-trivial automation.
From the point of view of command line level friendliness, tcsh is probably the best out of the box. But you can compensate for some neat features like tcsh 'autolist' functionality in bash too: possibilities are listed after an ambiguous completion on tab (bash requires two tab's), eg:
% ls /usr/l[tab] lib/ libexec/ local/.
This can be set via readline. In /etc/inputrc or ~/.inputrc, put "set show-all-if-ambiguous on". This'll cause bash (which uses readline) to list all possibilities after a single tab. Check the readline man page for other nifty stuff, including the reverse-search-history (default Control-R).
I would like to stress the importance of use of a good file manager along with shell and I recommend to use an Orthodox file manager -- see OFM page -- Midnight Commander is a perfect interactive shell for Linux and other Unixes -- highly recommended. See also my on-line book about Orthodox file managers at ../OFM/Ofm_00.shtml.
Other useful links: Editors Links, Unix/Linux links. Overview of books about shells is in Shell bibliography. Highly recommended shell site is SHELLdorado. You should probably visit it first. More or less useful dotfiles sometimes can be found at dotfiles.com home. Check it out.
Dr. Nikolai Bezroukov
Daniel Robbins' three-part series on bash programming on developerWorks: Part 1, Part 2, and Part 3.
developerWorks Linux Using Bash shell scripts for function testing
"Function testing is the phase during a development cycle in which the software application is tested to ensure that the functionality is working as desired and that any errors in the code are properly handled. It is usually done after the unit testing of individual modules, and before a more thorough system test of the entire product under load/stress conditions."
"There are many testing tools in the marketplace that offer a lot of functionality to help with the testing efforts. However, they need to be obtained, installed, and configured, which could take up valuable time and effort. Bash can help to speed things along."
Functions and aliases in bash" also http://www.linuxgazette.com/issue53/eyler.html
Variable Mangling in Bash with String Operators LG #57 by Pat Eyler (see also Variable Mangling in Bash with String Operators)
Have you ever wanted to change the names of many files at once? How about using a default value for a variable if it has no value? These and many other options are available to you through string operators in bash and other bourne shell derived shells.
String operators allow you to manipulate the contents of a variable without having to write your own shell functions to do so. They are provided through 'curly brace' syntax. Any variable can be displayed like this ${foo} without changing its meaning. This functionality is often used to protect a variable name from surrounding characters.
bash-2.02$ export foo=foo
bash-2.02$ echo ${foo}bar # foo exists so this works
foobar
bash-2.02$ echo $foobar # foobar doesn't exist, so this fails
bash-2.02$
By the end of this article, you'll be able to use it for a whole lot more.
There are three kinds of variable substitution:I'll talk about the first two and leave command substitution for another article. ... If you're interested in more hints about bash (or other stuff I've written about), please take a look at my home page.
- Pattern Matching,
- Substitution,
- Command Substitution.
Column - Ed Schaefer - The Shell Corner -- Each month UnixReview.com will publish a selected script in this column. I will evaluate the submissions and award each winner $100.00 courtesy...
Welcome to the inaugural edition of the Shell Corner column. The premise of Shell Corner is simple: You, the Unix professional send in your favorite Unix shell script and each month UnixReview.com will publish a selected script in this column. I will evaluate the submissions and award each winner $100.00 courtesy of UnixReview.com. What could be more simple?
Joe Casad, UnixReview.com senior editor, wanted a "shell guru" to author this column, but ended up with me instead. My experience is mostly as a Unix business applications programmer and not as a systems administrator. I feel my attitude is more minimalist than most, and my script evaulations are certain to reflect this. I enjoy scripts that are (relatively) easy to read and well documented. If I have trouble reading a script, chances are I'll quickly go to the next one.
What are the limitations on script submissions? I can think of no limitation, other than length. Use any of the Unix tools such as Perl, awk, sed, etc, as well as high level languages, such as C or C++. I'll even struggle through assembler if you will.... ... ...
# tolower.sh: convert file names to lower case # in the current working directory # Choose either all the files in a directory or a command-line list if [ "$#" -gt 0 ]; then filelist="$@" # just the files on command line else filelist=`ls` # all files fi for file in $filelist; do # Use the grep command to determine if the file has an upper case letter # Determine the destination of the mv command by down shifting all the # letters in the file name. Command substituting an echo of the # file name to the translate filter, tr, performs the downshift if echo "$file"|grep [A-Z] > /dev/null; then mv "$file" `echo "$file"|tr "[A-Z]" "[a-z]"` fi done
Bourne Shell Programming by Robert P. Sayle Chapters:
About.com: Mailing From Scripts(Jul 29, 2000) About.com: Calling Commands When Booting [init script](Jul 29, 2000) Linux Gazette: The Deep, Dark Secrets of Bash(Jul 08, 2000) Linuxnewbie.org: Bash Programming Cheat Sheet(Jun 14, 2000) IBM developerWorks: Bash by example, Part 2(Apr 09, 2000) Linux Gazette: Introduction to Shell Scripting--The Basics(Apr 02, 2000) shellscript.org: Share UNIX/Linux scripts/applets(Apr 00, 2000) IBM developerWorks: Bash by example, Part 1(Mar 30, 2000) Ext2: Shell Scripting Part One(Feb 27, 2000)
*** Of Unix shells and environment variables -- Unix shells and commands use shell variables. Understanding what these variables are, and how they're created and deployed, can be very useful. This month, Mo Budlong gives you a rundown on shell environment variables, and explains how you can get around some of their limitations. (2,200 words)
**** [June 26, 2000] Unix KornShell memento -- good reference [96K]
**** [June 26, 2000] Bash module for the Dotfile Generator -- reference manual --great !!!
[Apr. 30, 2000] Put the path in your xterm titlebar by James Andrews
export PS1=$PS1"\[\e]0;\H:\w\a\]" (bash) This alters the prompt to print special escape characters, which xterm uses to set its titlebar. If you are going into a deep directory tree and the path is getting very long, then using the title bar space can be an attractive idea.A csh version is
alias cd 'chdir \!*;echo -n "^[]0;"`hostname`":" $cwd^G'.^[ is an escape character; use ctrl V then escape . ^G is a bell.
[Mar 4, 2000] Slashdot Articles AT&T's Korn Shell Source Code Released KSH93 source was released.
Henk Langeveld writes, "This announcement can be found at kornshell.com: March 1, 2000: I am happy to annouce the the 'i' point release of ksh93 is now available for download. For the first time, source is available as well as binaries for several architectures. If you build binaries for new architectures, and send them to us, we can add them to the download site. The download page has been completely revised in a manner that hopefully will be easier to use. ksh93 is part of the ast-open package. tksh (ksh with tk support) is also part of this package. -- David Korn As a long-term fan I'm glad to see the korn shell now being released under a new license. The license is quite non-standard, and does include some restrictions (changes can only be distributed as patches), but as far as I can see it does allow anyone to bundle binaries with their products. The distribution format is quite non-standard. The research group at AT&T has their own packaging system, built around nmake. "
[Mar 4, 2000] Adam's UNIX shells page
[Jan 3, 2000] Portable Unix Shell Environment - shell scripts, Korn shell, Perl, porting
The Portable Unix Shell Environment (PUSE) is a set of Korn shell login environment files and about 80 Korn shell, Perl and Expect scripts that have been ported to several versions of Unix. It is distributed at no charge as open source software. The scripts, which can be used independently of the login environment files, include general utility scripts and systems administration scripts.
The Portable Unix Shell Environment allows the user to, for example:
- Substitute one string for another in all the text files in a directory hierarchy.
- List all the subdirectories in one or more directories, including symbolic links to directories.
- Copy a file to multiple remote hosts, with backup of the target file on each host.
- List the city or region for a telephone area code or vice-versa.
- Use a single set of Korn shell login environment files (.profile, .kshrc) in a heterogeneous Unix network.
Useful information from Solaris Advanced User's Guide
SMUG What Is It -- interesting book
Every few years, Robelle publishes a book of tips for Hewlett-Packard computer users. For obscure and historical reasons, these have been called "SMUG" proceedings, where SMUG stands for Seldom Met Users Group.During the San Antonio Interex Conference of 1982, Robelle gave away copies of the SMUG II Microproceedings, which contained unpublished and unsuspected data on the HP 3000, including the first publication of Alfredo Rego's classic: "How to Read A Manual." Our supply of these little blue books was quickly exhausted, but they were reprinted and shipped world-wide.
[July 18, 1999] Novice Bash Tip Issue 14
[July 18, 1999] bash String Manipulations Issue 18
[July 18, 1999] Bash Tip
OK, here's a fun little ditty that I recently came across that I've really fallen in love with :-) It's for all of those times that you find yourself either scooting between two directories or when you've been working in one directory, momentarily 'cd' to another, and then need to get back to wherever it was that you originally were at. (How about that for a run on sentence... :-) I'm sure that I could do better with a bit of a running start and a good tail wind...)
Anyway, I'm sure that most all of you know that the 'cd' command -- a shell built-in function -- in BASH will automatically wisk you back to your home directory if it's invoked without an argument. That is, if you simply type in:
% cd at the prompt, you'll suddenly find yourself back in your home directory from anywhere in the system.So... this is a great little thing to remember for all you absentminded folks that have a tendency to wander a bit. You know... some poor freshman with a note pinned to his/her shirt: "If you find me, please return me to my $HOME: my address is 'cd'".Anyway, for all you BASH shell users out there, here's a bit of handiness that goes a step beyond this and lets you find your way back to the directory that you previously were in.
For instance, suppose that you're tooling along on some seriously cool Tcl/Tk proggie and you realize that a bit of code from one of the demo apps might come in handy for some little problem you've been having. No problem mon... You just do the 'ol
% cd /usr/lib/tk and have a look around for the code you've been looking for. OK, now you've found it and everything's groovy (I just heard on the radio the other day, BTW, that "groovy" is "cool" once again -- you know, no one is using it, so it's "hip" once again, because prettysoon everyone will be using it... I know, I know... I'm starting to ramble. Better check the note my wife pinned to my shirt this morning... :-) anyway, where were we...Oh yeah, we had just stopped by the 'ol TK demo subdir to do a bit of GPL'd plagiarism and need to get back to GO. Well, there's a VERY EASY way to do this that isn't at all intuitively obvious -- unless you've already read through all 80 pages of the BASH manual page and then if you have -- what are you doing hanging around here!!?
Well, to get back, you simply type in:
% cd ~- that's a 'cd' with one of those squiggly lookin' things that lives up in the upper left hand corner of the keyboard (a tilde I believe :-) and a "dash" (or minus sign).Do this and Presto! you're back in the 'ol neighborhood again!
Seriously Mucho Cool, eh?
Thought you'd like this one :-) And if you act now, you'll get at no extra cost a set of 8 Ginzu knives... They slice... they dice... they.... (Hmmm... I think it must be time for my medicine again :-)
Yeah, seriously, this one is pretty helpful. By now, you know all about using the 'chvt' program that comes with the 'kbd' distribution -- use this one to flit from virtual terminal to virtual terminal. Being able to do a bunch of stuff at the same time is very cool and using 'chvt' lets you keep you hands on the keyboard while you move from VT to VT. This same basic principle lets you oscillate from one directory to another as you're working. Do the 'ol "cd `-" once and you're back to the directory you started in. Do it a second time and you're back to the other directory.
Another helpful thing that you can do with this is set it up as an alias of some kind -- use any shortcut alias you'd like. Now 'cd ~-' isn't all that hard to write, but it's often easier to make up a quick alias. I use:
alias cdup='cd ~-'so that I can type in 'cdup' and I'll 'cd' back up to wherever it was that I was last at.How about that, eh?
You know, you have to admit that the Linux Gazette can never be accused to being vaporware but, truth be told, it occasionally is gaseousware I mean, where else could you find an entire two pages of verbal meanderings that boil down to, "if you type 'cd ~-' you'll change to the directory you were last in."
Think of it as getting your money's worth...
Seriously, for those of you who might be interested, this is an example of TILDE EXPANSION which is a feature of the BASH shell. Check the manual page for more info.
Enjoy! :-)
John
[July 16, 1999] WebRSH WebRSH is a CGI program which implements a general purpose web-based computing shell. It provides a command line interface, file manager, text editor, and a few other basic things, while being highly configurable and adaptable to various tasks. It can replace, to some extent, rsh, telnet, and ftp daemons. WebRSH is implemented in Perl, and runs on both UNIX and Win32 platforms.
[May 20, 1999] The Ignorant's Guide to Shell Programming
Here is a very interesting posting by John Mashey that summarize shell history till mid 80th very well:
From: mash@mips.UUCP (John Mashey) Newsgroups: net.unix-wizards Subject: Re: Shell history, true facts, but long Date: 18 Mar 86 09:05:12 GMT At Todd Williams' request:
1) In the Beginning, there was the shell, /bin/sh, by Ken Thompson. I can't remember what version this appeared in, but it was well before my time, i.e., it sure looked and acted like a shell before late 73. For more info on the early days, see DMR's paper in the 2nd UNIX BLTJ. Almost nothing was builtin (on purpose), but there were separate if/goto commands. Goto acted by seeking the file pointer it shared with its parent shell. For a single-stream command-line interpreter, most, if not all of the fundamental (in my opinion) ideas were already present. See reference [1].
2) During the period 1973-1976, lots of people hacked on that shell in various and sundry ways. USG (UNIX Support Group at Murray Hill) added some things, and so did we:
3) The "PWB Shell" first appeared in mid-1975. It derived from a set of requirements and suggestions from me in early 1975 in trying to do serious shell programming. In mid-1975, the shell acquired variables, including 3 that were derived from per-process data. This is where the idea of more generalised path- searching came in. Some of the existing support commands were beefed up, including if/else/endif. Most of this work was done by Alan Glasser or Dick Haight. I ended up taking this over, doing performance analysis, moving most of the control structure commands into the shell, adding switch,, while, etc. etc. By this time there was serious shell programming in massive amounts. This had mostly stabilised by early/mid 1976, although a few more features were added in early 1977. For various reasons this came to be called the "Mashey shell", a term I universally abhorred, since it was fundamentally still a Thompson shell with selected additions and tunings by various of us, constrained to be mostly upward compatible, leading to some things that I found less than esthetic. Even less than esthetic was the title of a later internal Bell Labs course: "Bourne Shell Programming for Mashey Shell Programmers". See [2] and [3].
4) The Bourne shell work started either in early 1976, or maybe late 1975. The first version was VERY different; for example, it used $ as the escape character instead of \. During 1976 there was a long sequence of modification, dialog, discussion, with various players in shifting alliances [like the game Diplomacy]. Besides the Computing Research crew, SRB himself, and me, this is when (as I recall) Dave Korn started getting involved (as heavy earlier user and extender of Bourne sh). To summarise, some of the Bourne shell's fundamentals came from the Thompson shell, but some were new. Some of the added semantics, and a few pieces of syntax came either from the PWB shell, or by agreement on what it ought to be, given that everything was changing anyway. [For example, the environment idea was a much-needed generalisation of the restricted PWB variables; this came from DMR, SRB, and I.] This did become the standard shell, although it took PWBers a while to convert, since they had huge investments in existing shell procedures. [Not that many had been written in the standard V6 or earlier shells.] See [4]. As a minor tidbit, how many 68K programmers out there have cursed the shell's trapping of memory fault to grow it's storage? I admit to causing this, having goaded SRB into it by moaning at the speed of the then-current Bourne shell vs the faster PWB shell.
5) I haven't tracked the exact influences on the C-shell. However, I believe that certain pieces were at least derived from the PWB shell's documentation (implementation, if not always exact syntax, of control cmds): if-the-else- endif, break, breakaw, onintr, while-end. Ask Bill where he got everything. Finally, note that there have been a number of other interesting and important shells around, but I'm tired, and I think more people understand the later work.
[1] K. Thompson, "The UNIX Command Language", in Structured Programming, Infotech State of the Art Report, Nicholson House, Maidenhead, Berkshire, England: Infotech International Ltd (March 1975), 375-384.
Classic Thompson-style paper of maximal insights per word, incl: "Many familiar computing 'concepts' are missing from UNIX. Files have no records. There are no access methods. User programs contain no system buffers. There are no file types. These concepts fill a much needed gap...
[2] J. R. Mashey, "Using a Command Language as a High-Level Programming Language", Proc. 2nd Int. Conf on Software Engineering, IEEE (Oct 13-15, 1976), 169-176.
Includes some usage data; state of the world in PWBland in mid-1976.
[3] T. A. Dolotta, J. R. Mashey, "Using a Command Language as the Primary Programming Tool", in Command Language Directions: Proc. IFIP Working Conference on Command Languauges, Sept 10-14, 1979, D. Beech, Ed, North- Holland, AMsterdam, 1980.
3 years later; more data; Bourne shell(called here nsh) included.
[4] S. R. Bourne, "An Introduction to the UNIX Shell", BSTJ 57, No 6, Part 2 (Jul-Aug 1978), 2797-2822.
-john mashey UUCP: {decvax,ucbvax,ihnp4}!decwrl!mips!mash
DDD: 408-720-1700
USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086
Other classic historical papers:
An Introduction to the UNIX Shell (sh) by S. R. Bourne -- the author of Bourne Shell -- the most interesting Unix shell language that was further developed in ksh, bash, zsh. HTMLized by Era Eriksson
An Introduction to the C shell by William Joy.
The new Korn Shell -- ksh93 by by David G. Korn, Charles J. Northrup, and Jeffery Korn The paper provides a concise feature list of ksh93 and authored by the authors of ksh93. Ksh93 was influenced by Perl and Tcl. Opening ksh93 on March 2000 reminds me of the article posted a while ago on the original Unix source code being available. Something like "Open source in history"? It's kind of sad that it took this long for one of the best shells to be released.
Arrays
ksh93 provides one-dimensional indexed and associative arrays. An array element is referenced as varName[subscript]. Indexed arrays use arithmetic expressions for subscripts. This permits computation within the subscript expression. The statement varName[3+8] for example, references the 11th element of the indexed array. (Arithmetic expressions are described more fully below).
The elements of an indexed array can be initialized from a list using the varName=(....) command. This provides a convenient notation for initializing an array to contain the names of files in a given directory. The number of entries in the array describes the number of files found. As an example, consider the following statement to initialize the entries indexed array with the names of files found in the current directory: entries=(*)
An associative array uses arbitrary strings for subscripts. We could, for example, create a state tax associative array and reference elements by the state name. This works even for space separated tokens within the string, such as New Jersey.
typeset -A StateTax StateTax[New Jersey]=0.06 print ${StateTax[New Jersey]}Several special positional parameter expansions are provided for array processing. Using ${varName[@]} refers to all elements of the array. The subscripts of an array can be referenced with ${!varName[@]}. The notation ${#varName[@]} provides the number of elements within the array. Elements within a numeric subscript range can be referenced using ${varName[@]:offset:length}. This special notation works with both indexed and associative arrays.
Arrays are used throughout the example lsc script. We define video as an associative array with capability names from the terminfo database as subscripts. The definition of video is provided as a compound assignment for an associative array.
video=( [bold]=$(tput bold) [reset]=$(tput reset) [reverse]=$(tput reverse) )Each element is assigned a value from the standard output of a tput execution for the capability name. For example, video[bold] is the terminfo sequence for bold lettering. Similarly, video[reverse] will provide reverse video output.
Using the notation $(command) will cause command to execute in a subshell of the current ksh. In many instances, ksh will not actually fork/exec a subshell when command is a built-in or a shell function. (Built-in functions are described below).
Expanded Name Space
In ksh93 a variable is defined by a name=value pair. The variable name space is hierarchical with . (dot) delimiters. The expanded name space permits an aggregate definition for a variable.
... .... ....Functions:
ksh93 provides two methods for function definitions. The formats are given as:
function name { body } name() { body }The second function format is provided for compatibility with POSIX standards. The primary distinction is that of variable name scope. In a POSIX function, a variable definition has global scope. In the following POSIX function bar, variable foo is redefined to a value of 6.
typeset foo=5 bar() { typeset foo=6 echo $foo } bar 6 echo $foo 6Variable definitions in ksh93 functions have local scope. In the following ksh93 function bar, a local variable foo is defined and has precedence over the global variable foo.
typeset foo=5 function bar { typeset foo=6 echo $foo } bar 6 echo $foo 5Discipline Functions
ksh93 provides active variables through a series of discipline functions. From the shell level, you can write get, set, and unset disciplines. Through the KornShell Development Kit, you can also add disciplines unique to your environment.
When a variable is referenced, as in $foo, ksh will invoke the get discipline associated with foo. The default discipline is to simply return the current value associated with foo. From the shell level, you can define a foo.get discipline function.
The set discipline is called when a value is assigned to a variable. Within the set discipline, the special variable .sh.name is the name of the variable whose value is being set.
On line 31 of lsc, we define a max_stringSize.get discipline function. Every reference to ${max_stringSize} will result in this function being executed. The value of the special .sh.value variable is the value returned from the discipline.
printf Statement
In ksh93, a printf statement is available following the ANSI C printf definition. This permits formatting specifications to be applied to each argument. To appreciate the differences between the standard print and printf statements, consider how you would output the contents of the entries array (from the lsc example), one per line. The standard print statement would display the file names as space-separated tokens on a single line. Using the printf statement with a "%s\n" format, however, would produce the desired results.
Arithmetic Commands
ksh93 statements of the form (( expression )) are called arithmetic commands. Arithmetic commands return True when the value of the enclosed expression is non-zero, and False when the expression evaluates to zero. The construct $((expression)) can be used as a word or part of a word. It is replaced by the value of expression.
In the lsc example, line 38, we evaluate the value of the discipline function using:
(( .sh.value = $(strlenList ${entries[@]}) + 3 ))ksh93 will evaluate the expression, which includes an assignment to the .sh.value variable. Note that the:
$(strlenList ${entries[@]})will invoke the strlenList built-in function and return the maximum width of the strings (given as element values) in the entries[] array. We add 3 to this value for formatting purposes.
ANSI C Strings
An ANSI C string is defined by preceding the single-quoted string with a $. For example, $'*' is the literal asterisk, *. With ANSI C strings, all characters between the single quotes retain their literal meaning, except for escape sequences. An escape sequence is introduced by the escape character \.
ANSI C string support provides an essential feature for shell programmers. Consider, for example, having to process variables with embedded tabs in their values. Without ANSI C string support, we would not be able to effectively test the value of the variable for embedded tabs. As an example, consider the following script:
print "foo\tbar" > /tmp/foobar read aline < /tmp/foobar if [[ "${aline}" == "foo\tbar" ]] then print TRUE fiThe comparison (see Conditional Commands, below) will fail. We can replace the conditional with ANSI C strings and ensure proper functionality. The example above should be rewritten as:
print "foo\tbar" > /tmp/foobar read aline < /tmp/foobar if [[ "${aline}" == $'foo\tbar' ]] then print TRUE fiOn line 45 of Listing 1, we must test to see if the directory is empty. The preceding entries=(*) in an empty directory will set the entries variable to the literal asterisk if no files are found.
Conditional Commands
A conditional command in ksh93 evaluates a test-expression and returns either True or False. Conditional commands can be used as part of an ``Or list'' (||), ``And list'' (&&), or as part of an if-elif-else command. Conditional commands have the format:
[[ test-expression ]]When used in conjunction with an ``And list'', ksh93 evaluates the test-expression and will execute the ``And component'' only if the test-expression evaluates to True. We use a conditional command as part of an ``And list'' such that the return statement will be executed only if the test-expression is True.
[[ ${entries[0]} == $'*' ]] && return 2Iteration Control
The for command has two formats. The traditional format is provided to iterate on each word in a list. The format is:
for variableName [ in word-list ] do compound-list doneAn arithmetic for command has been provided that is very similar to the C programming language for statement. The format is:
for (( initExpr ; condition ; loopExpr )) do compound-list doneThe initExpression is evaluated by ksh prior to executing the for command. The condition is then evaluated prior to each iteration of compound-list. If the condition is non-zero, then ksh executes the compound-list. The loopExpression is evaluated at the end of each iteration.
Name Referencing
A new typeset option has been added for name referencing. Using typeset -n nameReference=variableName will associate nameReference with variableName. A special alias, nameref, is provided as the equivalent for typeset -n. A shell script may use the reference name to refer to the variable name. Name referencing provides a convenient mechanism to pass the name of compound variables, or arrays, to ksh functions. This is more efficient than passing the variable's content.
In the lsc example, function setOutput must add the directory entries to the appropriate row and column. We could have defined separate functions named addToRow and addToColumn for this purpose. The main body of the functions, however, would be equivalent. Instead, we opted to write a single function addToCell that uses a nameref to the cell type passed as a parameter.
The addToCell function accepts three arguments, of which the first two are required. The first argument is the cell type and must be either row or col. We create a nameref using the local variable cell to be equivalent to the cell type specified. A reference to ${cell.index} would therefore be equivalent to ${row.index} or ${col.index}.
FPATH
ksh functions are not inherited across invocations of ksh. A child shell process, for example, does not have access to the functions defined within the parent ksh invocation. This has historically limited the re-usability of ksh functions. As a solution, ksh93 will search the colon-separated list of directories given by the FPATH variable value, for an executable file with the same name as the function. In the lsc example, we can eliminate the last statement:
lsc "${@}"The FPATH can then be set to the directory containing the lsc file. From the shell level, we can now call lsc. ksh93 will load the lsc script and will call the lsc function with the command line arguments specified. Note that the supporting functions defined in the lsc script are available to the lsc function.
A function autoload feature is provided, in which an auto-loaded function definition is loaded and retained within the ksh93 environment upon the first reference to the function name. This provides better performance since the search and load steps are eliminated for subsequent references.
***** SHELLdorado -- Excellent site by by Heiner Steven. Very cute name that catches the fact that the shell is a crown jewel of Unix ;-) IMHO this is the best shell-related site. Actively maintained. Highly recommended !
***+ kshweb.html-- a good site by Dana French
***+ Open Directory - Computers Operating Systems Unix Shell -- a decent site. Could be better...
*** dotfiles.com home -- several dot files of very uneven (often low) quality. Still better than nothing...
Etc
Actually not all UNIXes contain tutorials on shells. Solaris does not :-(. AIX tutorial is not bad. The best is probably from Digital Unix. See below. There is no good free tutorial of ksh93 and here you probably will be better off buying a book. O'Reilly Learning Korn Shell is available on CD and is probably the best introductory book for ksh88 (not ksh93) available in HTML. It's definitely better than any of the offerings below...
| {****} Bash
Features: Overview Documentation for Bash
Bash Reference Manual - Table of Contents -- Edition 2.2, for bash Version 2.02. 1 April 1998 |
Command and Shell User's Guide -- good reference from Digital . See also Digital UNIX |
UXP/V User's Guide -- contains good Shell Tutorial (Copyright FUJITSU LIMITED 1997) |
Shell Scripts and Awk on the CUED Teaching System |
Linux.com
Learn HOWTOs - Advanced Bash-Scripting HOWTO by Mendel Cooper
--Advanced Bash-Scripting HOWTO: A guide to shell scripting, using Bash.
Nice tutorial
The Solaris 2.7 Common Desktop Environment: Desktop KornShell User's Guide Answerbook is available online (the dtksh is a ksh93 with Motif extensions).
{***+} Bourne Shell Programming by Robert Sayle Online book. Pretty decent.[link updated Jan 29, 2000. ]
Bourne Shell Programming teaches UNIX users how to harness the power of the shell. The book assumes that the reader has at least a general knowledge of UNIX including its commands, syntax, and operation. It also assumes that the reader also understands simple programming techniques inherent to most programming languages. The book does not provide instruction on the basics of UNIX. It instead builds upon these basics by showing how to combine them with the shell's programming facilities. The goal is to train users to employ these techniques at both the command line and within scripts so that the operating system becomes a tool instead of a hindrance.This book was written over the course of a number of years mainly because the author switched career paths. The change consequently resulted in a text tested against two different versions of UNIX. Some examples are shown in Sun Solaris 2.4 while others are given from Linux Slackware 3.2. Readers are cautioned to check their local operating system's manual pages on any command demonstrated for proper syntax and operation.
The author realizes he is prone to error and respectfully requests any corrections be forwarded by email. Suggestions on improving this book are also welcome as are offers to publish it through traditional channels.
For notes on the format of this book, the reader is directed to the conventions listed in the book's foreword.
This book was derived from a class created and presented at ARINC Inc, Newport Beach, CA.
AIX Bourne shell tutorial from UCF Computer Services AIX on-line manual. There are several mirrors for AIX docs online AIX Version 4.3 System User's Guide Operating System and Devices see also mirror About This Guide
[June 15, 1999] Idaho State University Basic UNIX Tutorial -- good; not only potato can grow in Idaho :-)
[November 17, 1998] Shell Quoting Guidelines V 1.2 by Tim Maher
[November 17, 1998] How to write a shell script
Knowledge Base - In Unix what is chsh and how do I use it to change my shell
Changing your login session's look and feel using aliases and configuration files
This is an outdated documentation for bash 1.14. Of course the ksh93 is a preferable shell. Bash 1.xx has disappointing globbing capabilities. But since version 2 of bash, if I drop a "shopt -s extglob" in my .bashrc file, I get those cool globbing features back. Very cool.
Edition 1.14, for bash Version 1.14. May 1994
UNIX Bourne Shell Programming contains a good introduction to the Bourne Shell and has a chapter about KSH Programming, too. download Zip-archive with the book (116 KB).
APPENDIXES A - sh B - test C - expr D - ftp E - cc F - f77 G - lint H - cb I - ar J - time K - ksh
Quick lessons on shell programming
The new Korn Shell -- ksh93 by by David G. Korn, Charles J. Northrup, and Jeffery Korn The paper provides a concise feature list of ksh93. The paper is authored by the authors of ksh93.
- Good Shell Coding Practices Part 1: Handling Command Line Arguments. by Heiner STEVEN <heiner.steven@odn.de> the author of famous Shelldorado site
- Good Shell Coding Practices Part2. Temporary files and signal handling
- Sending files as Mail Attachments -- very good !
docs.sun.com man pages section 1 User Commands - ksh
comp.unix.shell -- the only Usenet group about shells
Note: see first Unix FAQshell Index -- it's probably more up-to-date than this document
Reference cards:
Reference manuals
The best implementation of ksh93 is Tksh that allow to use TCL on top of ksh93:
Tksh is an implementation of the Tcl C library written on top of the library for the new KornShell (ksh93). Tksh emulates the behavior of Tcl by using the API that is provided for extending ksh93, which is similar to the Tcl library in that it allows access to variables, functions and other state of the interpreter. This implementation allows Tcl libraries such as Tk to run on top of ksh93 unchanged, making it possible to use shell scripts in place of Tcl scripts. ksh93 is well suited for use with Tk because it is backward compatible with sh, making it both easy to learn and easy to extend existing scripts to provide a graphical user interface. Tksh is not yet another port of Tk to another language -- it allows Tcl scripts to run without modification using the ksh93 internals.
As stated on the AT&T software download selections page:
KSH-93 provides the following features, many of which were also inherent in KSH-88:
Documentation for KSH-93 consists of an "Introduction to KSH-93", "Compatibility with the Bourne Shell" and a manual page and a README file. In addition, the "New KornShell Command and Programming Language," book is available from Prentice Hall.
- Enhanced Command Re-entry Capability: The KSH-93 history function records commands entered at any shell level and stores them, up to a user-specified limit, even after you log off. This allows you to re-enter long commands with a few keystrokes - even those commands you entered yesterday. The history file allows for eight bit characters in commands and supports essentially unlimited size histories.
- In-line Editing: In "sh", the only way to fix mistyped commands is to backspace or retype the line. KSH-93 allows you to edit a command line using a choice of EMACS-TC or "vi" functions. You can use the in-line editors to complete filenames as you type them. You may also use this editing feature when entering command lines from your history file. A user can capture keystrokes and rebind keys to customize the editing interface.
- Extended I/O Capabilities: KSH-93 provides several I/O capabilities not available in "sh", including the ability to:
- specify a file descriptor for input and output
- start up and run co-processes
- produce a prompt at the terminal before a read
- easily format and interpret responses to a menu
- echo lines exactly as output without escape processing
- format output using printf formats.
- read and echo lines ending in "
\ ". - Improved performance: KSH-93 executes many scripts faster than the System V Bourne shell. A major reason for this is that many of the standard utilities are built-in. To reduce the time to initiate a command, KSH-93 allows commands to be added as built-ins at run time on systems that support dynamic loading such as System V Release 4.
- Arithmetic: KSH-93 allows you to do integer arithmetic in any base from two to sixty-four. You can also do double precision floating point arithmetic. Almost the complete set of C language operators are available with the same syntax and precedence. Arithmetic expressions can be used to as an argument expansion or as a separate command. In addition there is an arithmetic for command that works like the for statement in C.
- Arrays: KSH-93 supports both indexed and associative arrays. The subscript for an indexed array is an arithmetic expression, whereas, the subscript for an associative array is a string.
- Shell Functions and Aliases: Two mechanisms - functions and aliases - can be used to assign a user-selected identifier to an existing command or shell script. Functions allow local variables and provide scoping for exception handling. Functions can be searched for and loaded on first reference the way scripts are.
- Substring Capabilities: KSH-93 allows you to create a substring of any given string either by specifying the starting offset and length, or by stripping off leading or trailing substrings during parameter substitution. You can also specify attributes, such as upper and lower case, field width, and justification to shell variables.
- More pattern matching capabilities: KSH-93 allows you to specify extended regular expressions for file and string matches.
- KSH-93 uses a hierarchal name space for variables. Compound variables can be defined and variables can be passed by reference. In addition, each variable can have one or more disciplines associated with it to intercept assignments and references.
- Improved debugging: KSH-93 can generate line numbers on execution traces. Also, I/O redirections are now traced. There is a DEBUG trap that gets evaluated after each command so that errors can be localized.
- Job Control: On systems that support job control, including System V Release 4, KSH-93 provides a job-control mechanism almost identical to that of the BSD "csh", version 4.1. This feature allows you to stop and restart programs, and to move programs between the foreground and the background.
- Added security: KSH-93 can execute scripts which do not have read permission and scripts which have the setuid and/or setgid set when invoked by name, rather than as an argument to the shell. It is possible to log or control the execution of setuid and/or setgid scripts. The noclobber option prevents you from accidentally erasing a file by redirecting to an existing file.
- KSH-93 can be extended by adding built-in commands at run time. In addition, KSH-93 can be used as a library that can be embedded into an application to allow scripting.
I see no active development for the Korn shell since 1993. Anyway Korn shell is a more powerful instrument than C-shell. Two major versions exist ksh88 and ksh93. The latest (Ksh93) version is more powerful than bash 2.0 and is the best commercial shell available. Ksh93 is now free for educational and private use. The only shell that can complete with ksh93 feature wise is probably zsh. It seems that Windows 95/NT version is also available -- see also David Korn U/Win project homepage
ksh93 is still probably the best shell around. In the absense of a better one, as long as you apply the right magic to your .shrc you can get around some limitations. To get some decent command line editting do a 'set -o emacs', to get the arrow keys etc to work
alias __A=`/bin/echo "\020"` # Up alias __B=`/bin/echo "\016"` # Down alias __C=`/bin/echo "\006"` # Right alias __D=`/bin/echo "\002"` # Left alias __H=`/bin/echo "\001"` # Home alias __p=`/bin/echo "\004"` # Delete alias __q=`/bin/echo "\005"` # End alias __z=`/bin/echo "\017"` # Clear
(The previous works on Solaris, it might need some tweaking for different forms of echo) Unfortunately, I don't know of an easy way to get TAB to do filename completion (the default is ESC ESC in ksh93).
Public domain Korn Shell is closer to ksh88 than ksh93. It has most of the ksh88 features, not much of the ksh93 features, and a number of its own features.
Now sh is often implemented as a special compilation of existing Ksh.
Bash 2.0 was de-facto standard among open source shells until opening of ksh93. But generally FSF dropped the ball and the product was stalled. That's a shame. Now I definitely prefer ksh93.
There are two major versions of BASH v.1.14.7 and v. 2.04. The latter is a recommended version as it's more compatible with ksh93 and contains a lot of important enhancements. At last it does has process substitution in pipes ( > (command) and < (command) ).
Each command argument of the form
<(list),>(list)or=(list)is subject to process substitution. In the case of the<and>forms, the shell will run process list asynchronously, connected to a named pipe (FIFO). The name of this pipe will become the argument to the command. If the form with>is selected then writing to this file will provide input for list. If<is used, then the file passed as an argument will be a named pipe connected to the output of the list process. For example,
paste <(cut -f1 file1) <(cut -f3 file2) | tee >(process1) >(process2) >/dev/null
cuts fields 1 and 3 from the files file1 and file2 respectively,pastes the results together, and sends it to the processes process1 and process2. Note that the file, which is passed as an argument to the command, is a system pipe so programs that expect tolseek(2)on the file will not work. Also note that the previous example can be more compactly and efficiently written as:
paste <(cut -f1 file1) <(cut -f3 file2) >>(process1) >>(process2)The shell uses pipes instead of FIFOs to implement the latter two process substitutions in the above example.
If
=is used, then the file passed as an argument will be the name of a temporary file containing the output of the list process. This may be used instead of the<form for a program that expects tolseek(2)on the input file.
Other important enhancements include (from the FAQ):
- new `time' reserved word to time pipelines, shell built-ins, and shell functions
- one-dimensional arrays with a new compound assignment statement,
- appropriate expansion constructs and modifications to some of the built-ins (read, declare, etc.)
- new quoting syntaxes for ANSI-C string expansion and locale-specific string translation
- new expansions to do substring extraction, pattern replacement, and indirect variable expansion
- new built-ins: `disown' and `shopt'
- new variables: HISTIGNORE, SHELLOPTS, PIPESTATUS, DIRSTACK, GLOBIGNORE, MACHTYPE, BASH_VERSINFO
- special handling of many unused or redundant variables removed (e.g., $notify, $glob_dot_filenames, $no_exit_on_failed_exec)
- dynamic loading of new built-in commands; many loadable examples provided
- new prompt expansions: \a, \e, \n, \H, \T, \@, \v, \V
- history and aliases available in shell scripts
- new readline variables: enable-keypad, mark-directories, input-meta, visible-stats, disable-completion, comment-begin
- new readline commands to manipulate the mark and operate on the region
- new readline emacs mode commands and bindings for ksh-88 compatibility
- updated and extended built-ins
- new DEBUG trap
- expanded (and now documented) restricted shell mode
When you login, the BASH shell only reads the .bash_profile file. Check that the line source $HOME/.bashrc is in your .bash_profile file. This ensures that the .bashrc file is read each time you login or start a subshell.