| CONTENTS |
Summary BoxThere are many specialized forms of editing that happen frequently enough that they sometimes want to be saved into a script. Examples of this kind of thing include:
In addition, there are a number of programs that provide some useful ways of modifying files but that you don't normally think of as editors:
This chapter covers all that and more. TOR |
One of the problems with fold is that it breaks text at an arbitrary column position even if that position happens to be in the middle of a word. It's a pretty primitive utility, designed to keep long lines from printing off the edge of a line printer page, and not much more.
fmt can do a better job because it thinks in terms of language constructs like paragraphs. fmt wraps lines continuously, rather than just folding the long ones. It assumes that paragraphs end at blank lines.
You can use fmt for things like neatening lines of a mail message or a file that you're editing with vi (Section 17.28). (Emacs has its own built-in line-neatener.) It's also great for shell programming and almost any place you have lines that are too long or too short for your screen.
To make this discussion more concrete, let's imagine that you have the following paragraph:
Most people take their Emo Phillips for granted. They figure, and not without some truth, that he is a God-given right and any government that considers itself a democracy would naturally provide its citizens with this sort of access. But what too many of this Gap-wearing, Real World-watching generation fail to realize is that our American forefathers, under the tutelage of Zog, the wizened master sage from Zeta-Reticuli, had to fight not only the godless and effete British for our system of self-determined government, but also avoid the terrors of hynpo-death from the dark and unclean Draco-Repitilians.
To prepare this text for printing, you'd like to have all the lines be about 60 characters wide and remove the extra space in the lines. Although you could format this text by hand, GNU fmt can do this for you with the following command line:
% fmt -tuw 60 my_file
The -t option, short for --tagged-paragraph mode, tells fmt to preserve the paragraph's initial indent but align the rest of the lines with the left margin of the second line. The -u option, short for --uniform-spacing, squashes all the inappropriate whitespace in the lines. The final option, -w, sets the width of the output in characters. Like most UNIX commands, fmt sends its output to stdout. For our test paragraph, fmt did this:
Most people take their Emo Phillips for granted. They figure, and not without some truth, that he is a God-given right and any government that considers itself a democracy would naturally provide its citizens with this sort of access. But what too many of this Gap-wearing, Real World-watching generation fail to realize is that our American forefathers, under the tutelage of Zog, the wizened master sage from Zeta-Reticuli, had to fight not only the godless and effete British for our system of self-determined government, but also avoid the terrors of hynpo-death from the dark and unclean Draco-Repitilians.
There is one subtlety to fmt to be aware of: fmt expects sentences to end with a period, question mark, or exclamation point followed by two spaces. If your document isn't marked up according to this convention, fmt can't differentiate between sentences and abbreviations. This is a common "gotcha" that appears frequently on Usenet.
|
There are a few different versions of fmt, some fancier than others. In general, the program assumes the following:
Paragraphs have blank lines between them.
If a line is indented, the indentation should be preserved.
The output lines should be about 70 characters wide. Some have a command-line option to let you set this. For example, fmt -132 (or on some versions, fmt -l 132) would reformat your file to have lines with no more than 132 characters on each.
It reads files or standard input. Lines will be written to standard output.
Go to http://examples.oreilly.com/upt3 for more information on: fmt
The GNU fmt is on the CD-ROM [see http://examples.oreilly.com/upt3]. There are also a couple of free versions available. Many versions of fmt have options for other structured data. The -p option (Section 21.4) reformats program source code. (If your fmt doesn't have -p, the recomment (Section 21.4) script uses standard fmt with sed to do the same thing.) The -s option breaks long lines at whitespace but doesn't join short lines to form longer ones.
Alternatively, you can make your own (Section 21.3) simple (and a little slower) version with sed and nroff. If you want to get fancy (and use some nroff and/or tbl coding), this will let you do automatically formatted text tables, bulleted lists, and much more.
JP, TOR, and JJ
fmt (Section 21.2) is hard to do without once you've learned about it. Unfortunately, it's not available in some versions of Unix. You can get the GNU version from the CD-ROM [see http://examples.oreilly.com/upt3], but it's also relatively easy to emulate with sed (Section 37.4) and nroff. Using those two utilities also lets you take advantage of the more sophisticated formatting and flexibility that sed and nroff macros can give you. (If you're doing anything really fancy, like tables with tbl,[1] you might need col or colcrt to clean up nroff's output.)
Here's the script:
Go to http://examples.oreilly.com/upt3 for more information on: fmt.sh
#!/bin/sh sed '1i\ .ll 72\ .na\ .hy 0\ .pl 1' $* | nroff
The reason this is so complicated is that, by default, nroff makes some assumptions you need to change. For example, it assumes an 11-inch page (66 lines) and will add blank lines to a short file (or the end of a long file). The quick-and-dirty workaround to this is to manually put the nroff request .pl 1 (page length 1 line) at the top of the text you want to reformat. nroff also tends to justify lines; you want to turn this off with the .na request. You also want to turn off hyphenation (.hy 0), and you may want to set the line length to 72 instead of nroff's default 65, if only for consistency with the real fmt program. All these nroff requests get inserted before the first line of input by the sed 1i command.
A fancier script would take a -nn line-length option and turn it into a .ll request for nroff, etc.
Another solution to consider is Damian Conway's Text::Autoformat Perl module. It has some very sophisticated heurestics to try to figure out how text should be formatted, including bulleted and numbered lists. In its simplest form, it can be used to read from stdin and write to stdout, just as a standard Unix utility would do. You can invoke this module from the command line like this:
% perl -MText::Autoformat -e 'autoformat' < your_file_here
By default, autoformat formats only one paragraph at a time. This behavior can be changed by altering the invocation slightly:
% perl -MText::Autoformat -e 'autoformat({all =>1})'
The manpage for this module even suggests a way into integrate this into vi:
map f !Gperl -MText::Autoformat -e'autoformat'
TOR and JJ
Lines in a program's comment block usually start with one or more special characters, like this:
# line 1 of the comment # line 2 of the comment # line 3 of the comment ...
It can be a hassle to add more text to one of the comment lines in a block, because the line can get too long, which requires you to fold that line onto the next line, which means you have to work around the leading comment character(s).
The fmt (Section 21.2) program neatens lines of a text file. But the standard fmt won't help you "neaten" blocks of comments in a program: it mixes the comment characters from the starts of lines with the words. (If your fmt has the -p option, it handles this problem; there's an example below.) The recomment script is fmt for comment blocks. It's for people who write shell, awk, C, or almost any other kind of program with comment blocks several lines long.
Go to http://examples.oreilly.com/upt3 for more information on: recomment
recomment reads the lines that you feed its standard input. It looks at the first line and figures out what characters you're using to comment the line (see the $cchars variable for a list typically SPACEs, TABs, #, or *). recomment then strips those comment characters off each line, feeds the remaining block of text to the fmt utility, and uses sed (Section 34.1) to add the comment characters again.
I usually use recomment from inside vi, with filter-through (Section 17.18) commands like:
!}recomment reformat to the next blank line 5!!recomment reformat this line and the next 4
Normally, recomment lets fmt choose the width of the comment block (72 characters, typically). To get another width, you can do one of the following:
Give the width on the command line, like this:
recomment -50
Set an environment variable named CBLKWID. Give the maximum width, in characters, for the comment text. For example, in the C shell, use:
% setenv CBLKWID 50
recomment isn't perfect, but it's usually much better than nothing! Here's the part of the script that does the work. The first two commands get the comment character(s) and count their length. The next three commands strip the comment characters, clean up the remaining comment text, and add the same comment characters to the start of all reformatted lines:
-n Section 34.3, expr Section 36.22, cut Section 21.14
# Get comment characters used on first line; store in $comment: comment=`sed -n "1s/^\([$cchars]*\).*/\1/p" $temp` # Count number of characters in comment character string: cwidth=`expr "$comment" : '.*'` # Re-format the comment block. If $widopt set, use it: cut -c`expr $cwidth + 1`- < $temp | # Strip off comment leader fmt $widopt | # Re-format the text, and sed "s/^/$comment/" # put back comment characters
When the expr command in backquotes (Section 28.14) is expanded, it makes a command like cut -c4-.
Some versions of fmt (like the one on the CD-ROM [see http://examples.oreilly.com/upt3]) have a -p option that does the same thing. Unlike the automatic system in recomment, you have to tell fmt -p what the prefix characters are but then it will only reformat lines with that prefix character For example, here's the start of a C++ program. The prefix character is *:
% cat load.cc /* * This file, load.cc, reads an input * data file. * Each input line is added to a new node * of type struct Node. */ ... % fmt -p '*' load.cc /* * This file, load.cc, reads an input data file. Each input line is * added to a new node of type struct Node. */ ...
JP
When you're saving or resending a Usenet article or mail message, you might want to the remove header lines (Subject:, Received:, and so on). This little script will handle standard input, one or many files. It writes to standard output. Here are a few examples:
With saved messages, at a shell prompt:
mail Section 1.21
% behead msg* | mail -s "Did you see these?" fredf
To save an article from a pipe without a header, from a program (here, the old readnews) that can't cut off headers itself:
What now? [ynq] s- | behead > filename
Here's the script, adapted a little from the original by Arthur David Olson:
Go to http://examples.oreilly.com/upt3 for more information on: behead
#! /bin/sh case $# in 0) exec sed '1,/^$/d' ;; *) for i do sed '1,/^$/d' "$i" done ;; esac
The script relies on the fact that news articles and mail messages use a blank line to separate the header from the body of the message. As a result, the script simply deletes the text from the beginning up to the first blank line.

Go to http://examples.oreilly.com/upt3 for more information on: dd
dd provides an unexpectedly easy answer. Let's say you wanted to delete the first 100 characters in a file. Here's the command that will do the trick (assuming of course that you give dd a filename with the if= option or data from a pipe):
% dd bs=100 skip=1
Or you could try:
% dd bs=1 skip=100
dd normally reads and writes data in 512-byte blocks; the input block size can be changed with the ibs= option, and the output block size with obs=. Use bs= to set both. skip= sets the number of blocks to skip at the start of the file.
Why would you want to do this? Section 21.9 gives an interesting example of reading text from standard input and writing it to a series of smaller files. Section 21.13 shows even more uses for dd.
TOR
Do you have a printer that starts each line too close to the left margin? You might want to indent text to make it look better on the screen or a printed page. Here's a Perl script that does that. It reads from files or standard input and writes to standard output. The default indentation is 5 spaces. For example, to send a copy of a file named graph to the lp printer, indented 12 spaces:
% offset -12 graph | lp
Here's the Perl script that does the job:
Go to http://examples.oreilly.com/upt3 for more information on: offset
#!/usr/local/bin/perl if ($ARGV[0] =~ /-[0-9]+/) { ($indent = $ARGV[0]) =~ s/-//; shift @ARGV; } else { $indent = 5; } while (<>) { print " " x $indent, $_; }
If there's an indentation amount in the first command-line argument, the dash is stripped and the value stored, then that argument is shifted away. Then a loop steps through the remaining arguments, if any (otherwise standard input is read) and outputs their text preceded by spaces. The script uses the Perl operator "string" x n, which outputs the string (in this case, a single space) n times. The Perl $_ operator contains the current input line.
JP
Here's an awk script, written by Greg Ubben, that centers text across an 80-character line. If your system understands #! (Section 36.3), this script will be passed directly to awk without a shell. Otherwise, put this into a Bourne shell wrapper (Section 35.19).
Go to http://examples.oreilly.com/upt3 for more information on: center
#!/usr/bin/awk -f { printf "%" int(40+length($0)/2) "s\n", $0 }
For each input line, the script builds a printf command with a width specification just wide enough to center the line (which awk holds in $0). For instance, a line 60 characters wide would give a value of int(40+60/2), which is 70. That makes the following printf command:
printf %70s\n, $0
Because %s prints a string right-justified, that command gives a 10-character indentation (70 minus 60) on an 80-character line. The right end of the line is also 10 characters (80 minus 70) from the right edge of the screen.
In vi, you can use a filter-through (Section 17.18) command to center lines while you're editing. Or just use center from the command line. For example:
% center afile > afile.centered % sort party_list | center | lp
JP
Most versions of Unix come with a program called split whose purpose is to split large files into smaller files for tasks such as editing them in an editor that cannot handle large files, or mailing them if they are so big that some mailers will refuse to deal with them. For example, let's say you have a really big text file that you want to mail to someone:
% ls -l bigfile -r--r--r-- 1 jik 139070 Oct 15 21:02 bigfile
Running split on that file will (by default, with most versions of split) break it up into pieces that are each no more than 1000 lines long:
wc Section 16.6
% split bigfile % ls -l total 283 -r--r--r-- 1 jik 139070 Oct 15 21:02 bigfile -rw-rw-r-- 1 jik 46444 Oct 15 21:04 xaa -rw-rw-r-- 1 jik 51619 Oct 15 21:04 xab -rw-rw-r-- 1 jik 41007 Oct 15 21:04 xac % wc -l x* 1000 xaa 1000 xab 932 xac 2932 total
Note the default naming scheme, which is to append "aa", "ab", "ac", etc., to the letter "x" for each subsequent filename. It is possible to modify the default behavior. For example, you can make split create files that are 1500 lines long instead of 1000:
% rm x?? % split -1500 bigfile % ls -l total 288 -r--r--r-- 1 jik 139070 Oct 15 21:02 bigfile -rw-rw-r-- 1 jik 74016 Oct 15 21:06 xaa -rw-rw-r-- 1 jik 65054 Oct 15 21:06 xab
You can also get it to use a name prefix other than "x":
% rm x?? % split -1500 bigfile bigfile.split. % ls -l total 288 -r--r--r-- 1 jik 139070 Oct 15 21:02 bigfile -rw-rw-r-- 1 jik 74016 Oct 15 21:07 bigfile.split.aa -rw-rw-r-- 1 jik 65054 Oct 15 21:07 bigfile.split.ab
Although the simple behavior described above tends to be relatively universal, there are differences in the functionality of split on different Unix systems. There are four basic variants of split as shipped with various implementations of Unix:
A split that understands only how to deal with splitting text files into chunks of n lines or less each.
A split, usually called bsplit, that understands only how to deal with splitting nontext files into n-character chunks.
A split that splits text files into n-line chunks, or nontext files into n-character chunks, and tries to figure out automatically whether it's working on a text file or a nontext file.
A split that does either text files or nontext files but needs to be told explicitly when it is working on a nontext file.
The only way to tell which version you've got is to read the manual page for it on your system, which will also tell you the exact syntax for using it.
The problem with the third variant is that although it tries to be smart and automatically do the right thing with both text and nontext files, it sometimes guesses wrong and splits a text file as a nontext file or vice versa, with completely unsatisfactory results. Therefore, if the variant on your system is (3), you probably want to get your hands on one of the many split clones out there that is closer to one of the other variants (see below).
Variants (1) and (2) listed above are OK as far as they go, but they aren't adequate if your environment provides only one of them rather than both. If you find yourself needing to split a nontext file when you have only a text split, or needing to split a text file when you have only bsplit, you need to get one of the clones that will perform the function you need.
Go to http://examples.oreilly.com/upt3 for more information on: split
Variant (4) is the most reliable and versatile of the four listed, and it is therefore what you should go with if you find it necessary to get a clone and install it on your system. There are several such clones in the various source archives, including the free BSD Unix version. GNU split is on the CD-ROM [see http://examples.oreilly.com/upt3]. Alternatively, if you have installed perl (Section 41.1), it is quite easy to write a simple split clone in perl, and you don't have to worry about compiling a C program to do it; this is an especially significant advantage if you need to run your split on multiple architectures that would need separate binaries. The Perl code for a binary split program follows:
#!/usr/bin/perl -w -- # Split text or binary files; jjohn 2/2002 use strict; use Getopt::Std; my %opts; getopts("?b:f:hp:ts:", \%opts); if( $opts{'?'} || $opts{'h'} || !-e $opts{'f'}){ print <<USAGE; $0 - split files in smaller ones USAGE: $0 -b 1500 -f big_file -p part. OPTIONS: -? print this screen -h print this screen -b <INT> split file into given byte size parts -f <TXT> the file to be split -p <TXT> each new file to begin with given text -s <INT> split file into given number of parts USAGE exit; } my $infile; open($infile, $opts{'f'}) or die "No file given to split\n"; binmode($infile); my $infile_size = (stat $opts{'f'})[7]; my $block_size = 1; if( $block_size = $opts{'b'} ){ # chunk file into blocks }elsif( my $total_parts = $opts{'s'} ){ # chunk file into N parts $block_size = int ( $infile_size / $total_parts) + 1; }else{ die "Please indicate how to split file with -b or -s\n"; } my $outfile_base = $opts{'p'} || 'part.'; my $outfile_ext = "aa"; my $offset = 0; while( $offset < $infile_size ){ my $buf; $offset += read $infile, $buf, $block_size; write_file($outfile_base, $outfile_ext++, \$buf); } #--- subs ---# sub write_file { my($fname, $ext, $buf) = @_; my $outfile; open($outfile, ">$fname$ext") or die "can't open $fname$ext\n"; binmode($outfile); my $wrote = syswrite $outfile, $$buf; my $size = length($$buf); warn "WARN: wrote $wrote bytes instead of $size to $fname$ext\n" unless $wrote == $size; }
Although it may seem somewhat complex at first glance, this small Perl script is cross-platform and has its own small help screen to describe its options. Briefly, it can split files into N-sized blocks (given the -b option) or, with -s, create N new segments of the original file. For a better introduction to Perl, see Chapter 42.
If you need to split a nontext file and don't feel like going to all of the trouble of finding a split clone to handle it, one standard Unix tool you can use to do the splitting is dd (Section 21.6). For example, if bigfile above were a nontext file and you wanted to split it into 20,000-byte pieces, you could do something like this:
for Section 35.21, > Section 28.12
$ ls -l bigfile -r--r--r-- 1 jik 139070 Oct 23 08:58 bigfile $ for i in 1 2 3 4 5 6 7 #[2] > do > dd of=x$i bs=20000 count=1 2>/dev/null #[3] > done < bigfile $ ls -l total 279 -r--r--r-- 1 jik 139070 Oct 23 08:58 bigfile -rw-rw-r-- 1 jik 20000 Oct 23 09:00 x1 -rw-rw-r-- 1 jik 20000 Oct 23 09:00 x2 -rw-rw-r-- 1 jik 20000 Oct 23 09:00 x3 -rw-rw-r-- 1 jik 20000 Oct 23 09:00 x4 -rw-rw-r-- 1 jik 20000 Oct 23 09:00 x5 -rw-rw-r-- 1 jik 20000 Oct 23 09:00 x6 -rw-rw-r-- 1 jik 19070 Oct 23 09:00 x7
JIK and JJ
Go to http://examples.oreilly.com/upt3 for more information on: csplit
Like split (Section 21.9), csplit lets you break a file into smaller pieces, but csplit (context split) also allows the file to be broken into different-sized pieces, according to context. With csplit, you give the locations (line numbers or search patterns) at which to break each section. csplit comes with System V, but there are also free versions available.
Let's look at search patterns first. Suppose you have an outline consisting of three main sections that start on lines with the Roman numerals I., II., and III.. You could create a separate file for each section by typing:
% csplit outline /I./ /II./ /III./ 28 number of characters in each file 415 . 372 . 554 . % ls outline xx00 outline title, etc. xx01 Section I xx02 Section II xx03 Section III
This command creates four new files (outline remains intact). csplit displays the character counts for each file. Note that the first file (xx00) contains any text up to but not including the first pattern, and xx01 contains the first section, as you'd expect. This is why the naming scheme begins with 00. (If outline had begun immediately with a I., xx01 would still contain Section I, but in this case xx00 would be empty.)
If you don't want to save the text that occurs before a specified pattern, use a percent sign as the pattern delimiter:
% csplit outline %I.% /II./ /III./ 415 372 554 % ls outline xx00 Section I xx01 Section II xx02 Section III
The preliminary text file has been suppressed, and the created files now begin where the actual outline starts (the file numbering is off, however).
Let's make some further refinements. We'll use the -s option to suppress the display of the character counts, and we'll use the -f option to specify a file prefix other than the conventional xx:
% csplit -s -f part. outline /I./ /II./ /III./ % ls outline part.00 part.01 part.02 part.03
There's still a slight problem, though. In search patterns, a period is a metacharacter (Section 32.21) that matches any single character, so the pattern /I./ may inadvertently match words like Introduction. We need to escape the period with a backslash; however, the backslash has meaning both to the pattern and to the shell, so in fact, we need either to use a double backslash or to surround the pattern in quotes (Section 27.12). A subtlety, yes, but one that can drive you crazy if you don't remember it. Our command line becomes:
% csplit -s -f part. outline "/I\./" /II./ /III./
You can also break a file at repeated occurrences of the same pattern. Let's say you have a file that describes 50 ways to cook a chicken, and you want each method stored in a separate file. The sections begin with headings WAY #1, WAY #2, and so on. To divide the file, use csplit's repeat argument:
% csplit -s -f cook. fifty_ways /^WAY/ "{49}"
This command splits the file at the first occurrence of WAY, and the number in braces tells csplit to repeat the split 49 more times. Note that a caret (^) (Section 32.5) is used to match the beginning of the line and the C shell requires quotes around the braces (Section 28.4). The command has created 50 files:
% ls cook.* cook.00 cook.01 ... cook.48 cook.49
Quite often, when you want to split a file repeatedly, you don't know or don't care how many files will be created; you just want to make sure that the necessary number of splits takes place. In this case, it makes sense to specify a repeat count that is slightly higher than what you need (the maximum is 99). Unfortunately, if you tell csplit to create more files than it's able to, this produces an "out of range" error. Furthermore, when csplit encounters an error, it exits by removing any files it created along the way. (A bug, if you ask me.) This is where the -k option comes in. Specify -k to keep the files around, even when the "out of range" message occurs.
csplit allows you to break a file at some number of lines above or below a given search pattern. For example, to break a file at the line that is five lines below the one containing Sincerely, you could type:
% csplit -s -f letter. all_letters /Sincerely/+5
This situation might arise if you have a series of business letters strung together in one file. Each letter begins differently, but each one begins five lines after the previous letter's Sincerely line. Here's another example, adapted from AT&T's Unix User's Reference Manual:
% csplit -s -k -f routine. prog.c '%main(%' '/^}/+1' '{99}'
The idea is that the file prog.c contains a group of C routines, and we want to place each one in a separate file (routine.00, routine.01, etc.). The first pattern uses % because we want to discard anything before main. The next argument says, "Look for a closing brace at the beginning of a line (the conventional end of a routine) and split on the following line (the assumed beginning of the next routine)." Repeat this split up to 99 times, using -k to preserve the created files.[4]
The csplit command takes line-number arguments in addition to patterns. You can say:
% csplit stuff 50 373 955
to create files split at some arbitrary line numbers. In that example, the new file xx00 will have lines 1-49 (49 lines total), xx01 will have lines 50-372 (323 lines total), xx02 will have lines 373-954 (582 lines total), and xx03 will hold the rest of stuff.
csplit works like split if you repeat the argument. The command:
% csplit top_ten_list 10 "{18}"
breaks the list into 19 segments of 10 lines each.[5]
DG
The tr command is a character translation filter, reading standard input (Section 43.1) and either deleting specific characters or substituting one character for another.
The most common use of tr is to change each character in one string to the corresponding character in a second string. (A string of consecutive ASCII characters can be represented as a hyphen-separated range.)
For example, the command:
$ tr 'A-Z' 'a-z' < file Berkeley version
will convert all uppercase characters in file to the equivalent lowercase characters. The result is printed on standard output.
In fact, a frequent trick I use tr for is to convert filenames from all uppercase to all lowercase. This comes up when you're dealing with files from MS-DOS or VMS that you are copying on to a Unix filesystem. To change all the files in the current directory to uppercase, try this from a Bash or Bourne shell prompt:
$ for i in `ls`; do mv $i `echo $i | tr [A-Z] [a-z]`; done
Of course, you need to be careful that there are no files that have the same name regardless of case. The GNU mv can be passed the -i flag that will make the program prompt you before overwriting an existing file. If you want to uppercase filenames, simply flip the arguments to tr. You can even apply this to an entire branch of a file system by sticking this in a find command. First, create a small shell script that can downcase a file and call it downcase:
#!/bin/sh mv $1 `echo $1 | tr [A-Z] [a-z]`
Now you can really do some damage with find:
$ find /directory/to/be/affected -exec 'downcase' '{}' ';'
Obviously, running this programming on random directories as root is not recomended, unless you're looking to test your backup system.
Go to http://examples.oreilly.com/upt3 for more information on: tr
In the System V version of tr, square brackets must surround any range of characters. That is, you have to say [a-z] instead of simply a-z. And of course, because square brackets are meaningful to the shell, you must protect them from interpretation by putting the string in quotes. The GNU tr, on the web site, is basically the System V version.
If you aren't sure which version you have, here's a test. Both tr examples below will convert the lowercase letters a through z to an uppercase A, but that's not what we're testing here. The Berkeley version also converts the input [ ] to A characters because [ ] aren't treated as range operators:
% echo '[ ]' | tr '[a-z]' A AA Berkeley version % echo '[ ]' | tr '[a-z]' A [ ] System V version
There's one place you don't have to worry about the difference between the two versions: when you're converting one range to another range, and both ranges have the same number of characters. For example, this command works in both versions:
$ tr '[A-Z]' '[a-z]' < file both versions
The Berkeley tr will convert a [ from the first string into the same character [ in the second string, and the same for the ] characters. The System V version uses the [ ] characters as range operators. In both versions, you get what you want: the range A-Z is converted to the corresponding range a-z. Again, this trick works only when both ranges have the same number of characters.
The System V version also has a nice feature: the syntax [a*n], where n is some digit, means that the string should consist of n repetitions of character "a." If n isn't specified or is 0, it is taken to be some indefinitely large number. This is useful if you don't know how many characters might be included in the first string.
As described in Section 17.18, this translation (and the reverse) can be useful from within vi for translating a string. You can also delete specific characters. The -d option deletes from the input each occurrence of one or more characters specified in a string (special characters should be placed within quotation marks to protect them from the shell). For instance, the following command passes to standard output the contents of file with all punctuation deleted (and is a great exercise in shell quoting (Section 27.12)):
$ tr -d ",.\!?;:\"\'`" < file
The -s (squeeze) option of tr removes multiple consecutive occurrences of the same character in the second argument. For example, the command:
$ tr -s " " " " < file
will print on standard output a copy of file in which multiple spaces in sequence have been replaced with a single space.
We've also found tr useful when converting documents created on other systems for use under Unix. For example, as described in Section 1.8, tr can be used to change the carriage returns at the end of each line in a Macintosh text file into the newline Unix expects. tr allows you to specify characters as octal values by preceding the value with a backslash, so the following command does the trick:
$ tr '\015' '\012' < file.mac > file.unix
The command:
$ tr -d '\015' < pc.file
will remove the carriage return from the carriage return/newline pair that a PC file uses as a line terminator. (This command is also handy for removing the excess carriage returns from a file created with script (Section 37.7).)
TOR, JP, and JJ
Email transport systems were originally designed to transmit characters with a seven-bit encoding like ASCII. This meant they could send messages with plain English text but not "binary" text, such as program files or graphics (or non-English text!), that used all of an eight-bit byte. Usenet (Section 1.21), the newsgroup system, was transmitted like email and had its same seven-bit limitations. The solution which is still used today is to encode eight-bit text into characters that use only the seven low bits.
The first popular solution on Unix-type systems was uuencoding. That method is mostly obsolete now (though you'll still find it used sometimes); it's been replaced by MIME encoding. The next two sections cover both of those though we recommend avoiding uuencode like the plague.
The uuencode utility encodes eight-bit data into a seven-bit representation for sending via email or on Usenet. The recipient can use uudecode to restore the original data. Unfortunately, there are several different and incompatible versions of these two utilities. Also, uuencoded data doesn't travel well through all mail gateways partly because uuencoding is sensitive to changes in whitespace (space and TAB) characters, and some gateways munge (change or corrupt) whitespace. So if you're encoding text for transmission, use MIME instead of uuencode whenever you can.
To create an ASCII version of a binary file, use the uuencode utility. For instance, a compressed file (Section 15.6) is definitely eight-bit; it needs encoding.
A uuencoded file (there's an example later in this article) starts with a begin line that gives the file's name; this name comes from the first argument you give the uuencode utility as it encodes a file. To make uuencode read a file directly, give the filename as the second argument. uuencode writes the encoded file to its standard output. For example, to encode the file emacs.tar.gz from your ~/tarfiles directory and store it in a file named emacs.tar.gz.uu:
% uuencode emacs.tar.gz ~/tarfiles/emacs.tar.gz > emacs.tar.gz.uu
You can then insert emacs.tar.gz.uu into a mail message and send it to someone. Of course, the ASCII-only encoding takes more space than the original binary format. The encoded file will be about one-third larger.[6]
If you'd rather, you can combine the steps above into one pipeline. Given only one command-line argument (the name of the file for the begin line), uuencode will read its standard input. Instead of creating the ~/tarfiles/emacs.tar.gz, making a second uuencoded file, then mailing that file, you can give tar the "filename" so it writes to its standard output. That feeds the archive down the pipe:[7]
mail Section 1.21
% tar cf - emacs | gzip | uuencode emacs.tar.gz | \ mail -s "uuencoded emacs file"<EMAIL_ADDRESS>What happens when you receive a uuencoded, compressed tar file? The same thing, in reverse.  Let's say you call this file mailstuff. How do you get the original files back? Use the following sequence of commands:
% uudecode mailstuff % gunzip emacs.tar.gz % tar xf emacs.tar
The uudecode command searches through the file, skipping From:, etc., until it sees its special begin line; it decodes the rest of the file (until the corresponding end line) and creates the file emacs.tar.gz. Then gunzip recreates your original tar file, and tar xf extracts the individual files from the archive.
Again, though, you'll be better off using MIME encoding whenever you can.
When MIME (Multipurpose Internet Mail Extensions) was designed in the early 1990s, one main goal was robust email communications. That meant coming up with a mail encoding scheme that would work on all platforms and get through all mail transmission paths.
Some text is "mostly ASCII": for instance, it's in a language like German or French that uses many ASCII characters plus some eight-bit characters (characters with a octal value greater than 177). The MIME standard allows that text to be minimally encoded in a way that it can be read fairly well without decoding: the quoted-printable encoding. Other text is full binary either not designed for humans to read, or so far from ASCII that an ASCII representation would be pointless. In that case, you'll want to use the base64 encoding.
Go to http://examples.oreilly.com/upt3 for more information on: mimencode, mailto
Most modern email programs automatically MIME-encode files. Unfortunately, some aren't too smart about it. The Metamail utilities come with a utility called mimencode (also named mmencode) for encoding and decoding MIME formats. Another Metamail utility, mailto, encodes and sends MIME messages directly but let's use mimencode, partly because of the extra control it gives you.
By default, mimencode reads text from standard input, uses a base64 encoding, and writes the encoded text to standard output. If you add the -q option, mimencode uses quoted-printable encoding instead.
Unlike uuencoded messages, which contain the filename in the message body, MIME-encoded messages need information in the message header (the lines "To:", "From:", etc.). The mail utility (except an older version) doesn't let you make a message header. So let's do it directly: create a mail header with cat > (Section 11.2), create a mail body with mimencode, and send it using a common system mail transfer agent, sendmail. (You could automate this with a script, of course, but we're just demonstrating.) The MIME standard header formats are still evolving; we'll use a simple set of header fields that should do the job. Here's the setup. Let's do it first in three steps, using temporary files:
$ cat > header From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Subject: base64-encoded smallfile MIME-Version: 1.0 Content-Type: application/octet-stream; name="smallfile.tar.gz" Content-Transfer-Encoding: base64 CTRL-d $ tar cf - smallfile | gzip | mimencode > body $ cat header body | /usr/lib/sendmail -t
The cat > command lets me create the header file by typing it in at the terminal; I could have used a text editor instead. One important note: the header must end with a blank line. The second command creates the body file. The third command uses cat to output the header, then the body; the message we've built is piped to sendmail, whose -t option tells it to read the addresses from the message header. You should get a message something like this:
Date: Wed, 22 Nov 2000 11:46:53 -0700 Message-Id<EMAIL_ADDRESS>From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Subject: base64-encoded smallfile MIME-Version: 1.0 Content-Type: application/octet-stream; name="smallfile.tar.gz" Content-Transfer-Encoding: base64 H4sIACj6GzoAA+1Z21YbRxb1c39FWcvBMIMu3A0IBWxDzMTYDuBgrxU/lKSSVHF3V6erGiGv rPn22edU3wRIecrMPLgfEGpVV53LPvtcOktcW6au3dnZ2mrZcfTkb7g6G53O7vb2k06ns7G3 06HPzt7uDn/Sra1N/L+32dnd29ve3tjD+s3Nna0novN3CHP/yqyTqRBPfk+U+rpknUnlf0Oc ...
Your mail client may be able to extract that file directly. You also can use mimencode -u. But mimencode doesn't know about mail headers, so you should strip off the header first. The behead (Section 21.5) script can do that. For instance, if you've saved the mail message in a file msg:
$ behead msg | mimencode -u > smallfile.tar.gz
Extract (Section 39.2) smallfile.tar.gz and compare it to your original smallfile (maybe with cmp). They should be identical.
If you're planning to do this often, it's important to understand how to form an email header and body properly. For more information, see relevant Internet RFCs (standards documents) and O'Reilly's Programming Internet Email by David Wood.
JP and ML
Besides the other uses of dd (Section 21.6) we've covered, you also can use this versatile utility to convert:
fixed length to variable-length records (conv=unblock), and the reverse (conv=block)
uppercase to lowercase (conv=lcase), and the reverse (conv=ucase)
the byte order of every pair of bytes (conv=swab)
ASCII to EBCDIC and the reverse (conv=ebcdic, conv=ibm). If you're converting old IBM tapes, you'll need to know the tape's blocking factor. And if the tape has multiple files on it, you'll have to use the tape device name that allows "no rewind on close" (Section 38.5) to read past the first file.
The cbs= option must be used to specify a conversion buffer size when using block and unblock and when converting between ASCII and EBCDIC. The specified number of characters are put into the conversion buffer. For ascii and unblock conversion, trailing blanks are trimmed and a newline is added to each buffer before it is output. For ebcdic, ibm, and block, the input is padded with blanks up to the specified conversion buffer size.
TOR
A nifty command called cut lets you select a list of columns or fields from one or more files.
You must specify either the -c option to cut by column or -f to cut by fields. (Fields are separated by tabs unless you specify a different field separator with -d. Use quotes (Section 27.12) if you want a space or other special character as the delimiter.)
In some versions of cut, the column(s) or field(s) to cut must follow the option immediately, without any space. Use a comma between separate values and a hyphen to specify a range (e.g., 1-10,15 or 20,23 or 50-).
The order of the columns and fields is ignored; the characters in each line are always output from first to last, in the order they're read from the input. For example, cut -f1,2,4 produces exactly the same output as cut -f4,2,1. If this isn't what you want, try perl (Section 41.1) or awk (Section 20.10), which let you output fields in any order.
cut is incredibly handy. Here are some examples:
Find out who is logged in, but list only login names:
who Section 2.8
% who | cut -d" " -f1
Extract usernames and real names from /etc/passwd (Section 22.3):
% cut -d: -f1,5 /etc/passwd
Cut characters in the fourth column of file, and paste them back as the first column in the same file:
% cut -c4 file | paste - file
Section 21.18 covers the cut counterpart, paste.
As was mentioned, you can use awk or perl to extract columns of text. Given the above task to extract the fifth and first fields fields of /etc/passwd, you can use awk:
% awk -F: '{print $5, "=>", $1}' /etc/passwd
An often forgotten command-line option for perl is -a, which puts perl in awk compatibility mode. In other words, you can get the same field-splitting behavior right from the command line:
% perl -F: -lane 'print $F[4], "=>", "$F[0]"' /etc/passwd
In the line above, perl is told about the field separator in the same way awk is, with the -F flag. The next four options are fairly common. The -l option removes newlines from input and adds a newline to all print statements. This is a real space saver for "one-line wonders," like the one above. The -a flag tells perl to split each line on the indicated field separator. If no field separator is indicated, the line is split on a space character. Each field is stored in the global array @F. Remember that the first index in a Perl array is zero. The -n option encloses the Perl code indicated by the -e to be wrapped in a loop that reads one line at a time from stdin. This little Perl snippet is useful if you need to do some additional processing with the contents of each field.
TOR, DG, and JJ
The pr command (Section 45.6) is famous for printing a file neatly on a page with margins at top and bottom, filename, date, and page numbers. It can also print text in columns: one file per column or many columns for each file.
The -t option takes away the heading and margins at the top and bottom of each page. That's useful when "pasting" data into columns with no interruptions.
The -m option reads all files on the command line simultaneously and prints each in its own column, like this:
% pr -m -t file1 file2 file3
The lines The lines The lines
of file1 of file2 of file3
are here are here are here
... ... ...
pr may use TAB characters between columns. If that would be bad, you can pipe pr's output through expand. Many versions of pr have a -sX option that sets the column separator to the single character X.
By default, pr -m doesn't put filenames in the heading. If you want that, use the -h option to make your own heading. Or maybe you'd like to make a more descriptive heading. Here's an example using process substitution to compare a directory with its RCS (Section 39.5) subdirectory:
% pr -m -h "working directory compared to RCS directory" <(ls) <(ls RCS) 2000-11-22 23:57 working directory compared to RCS directory Page 1 0001.sgm 0001.sgm,v 0002.sgm 0002.sgm,v 0007.sgm 0007.sgm,v 0008.sgm 0008.sgm,v ...
(The heading comes from the GNU version of pr. Later examples in this article use a different version with a different heading format.)
An option that's a number will print a file in that number of columns. For instance, the -3 option prints a file in three columns. The file is read, line by line, until the first column is full (by default, that takes 56 lines). Next, the second column is filled. Then, the third column is filled. If there's more of the file, the first column of page 2 is filled and the cycle repeats:
% pr -3 file1 Nov 1 19:44 1992 file1 Page 1 Line 1 here Line 57 here Line 115 here Line 2 here Line 58 here Line 116 here Line 3 here Line 59 here Line 117 here ... ... ...
The columns aren't balanced if the file will fit into one column, the other columns aren't used. You can change that by adjusting -l, the page length option; see the section below.
Do you want to arrange your data across the columns, so that the first three lines print across the top of each column, the next three lines are the second in each column, and so on, like this?
% pr -l1 -t -3 file1 Line 1 here Line 2 here Line 3 here Line 4 here Line 5 here Line 6 here Line 7 here Line 8 here Line 9 here ... ... ...
Use the -l1 (page length 1 line) and -t (no title) options. Each "page" will be filled by three lines (or however many columns you set). You have to use -t; otherwise, pr will silently ignore any page lengths that don't leave room for the header and footer. That's just what you want if you want data in columns with no headings.
If you want headings too, pipe the output of pr through another pr:
% pr -l1 -t -3 file1 | pr -h file1 Nov 1 19:48 1992 file1 Page 1 Line 1 here Line 2 here Line 3 here Line 4 here Line 5 here Line 6 here Line 7 here Line 8 here Line 9 here ... ... ...
The -h file1 puts the filename into the heading.
Also see paste (Section 21.18). Of course, programming languages like awk (Section 20.10) and perl (Section 41.1) can also make text into columns.
JP
Go to http://examples.oreilly.com/upt3 for more information on: column
Another column-making program, besides cols and pr (Section 21.15), is the creatively named utility column. It tries to determine the terminal width, which you can override with the -c option (-c 132, for example, gives 132 columns: handy for printing on wide line-printer paper.) The -x option fills columns before rows similar to pr with its -n option and cols -d.
What makes column different from the others is its -t option. This reads input data that's already in columns and rebalances the columns into a table with variable-width columns. Say what? This is easiest to see with an example, and the column(1) manual page has a good one.
If you'd like to add column headings to ls -l output, it can be a pain to try to make headings that each take the same number of characters as the data below them. For instance, the first field on each line, the permissions, takes 10 characters, but if you want to use the heading "PERM", which takes only 4 characters, you need to balance it by adding 6 spaces after. Using column -t, you can balance these automatically. Here's an example. The first command is plain ls -l. In the second and third examples, I use sed 1d (Section 34.1) to delete the total n line from ls, and subshells (Section 24.4) to make both commands use the same standard output; this is important only in the third command, where I pipe the combined stdout to column for balancing:
; Section 28.16, > Section 28.12
$ ls -lo total 1644 -r--r--r-- 1 jpeek 1559177 Sep 19 1999 ORA_tifs.tgz -rw-rw-r-- 1 jpeek 4106 Oct 21 1999 UPT_Russian.jpg -rw-rw-r-- 1 jpeek 101944 Nov 19 09:30 london_dusk-livesights.xwd.gz dr-xr-xr-x 2 jpeek 4096 Dec 12 1999 me $ (echo "PERM LINKS OWNER SIZE MON DY TM/YR NAME"; \ > ls -lo | sed 1d) PERM LINKS OWNER SIZE MON DY TM/YR NAME -r--r--r-- 1 jpeek 1559177 Sep 19 1999 ORA_tifs.tgz -rw-rw-r-- 1 jpeek 4106 Oct 21 1999 UPT_Russian.jpg -rw-rw-r-- 1 jpeek 101944 Nov 19 09:30 london_dusk-livesights.xwd.gz dr-xr-xr-x 2 jpeek 4096 Dec 12 1999 me $ (echo PERM LINKS OWNER SIZE MONTH DAY HH:MM/YEAR NAME; \ > ls -lo | sed 1d) | column -t PERM LINKS OWNER SIZE MONTH DAY HH:MM/YEAR NAME -r--r--r-- 1 jpeek 1559177 Sep 19 1999 ORA_tifs.tgz -rw-rw-r-- 1 jpeek 4106 Oct 21 1999 UPT_Russian.jpg -rw-rw-r-- 1 jpeek 101944 Nov 19 09:30 london_dusk-livesights.xwd.gz dr-xr-xr-x 2 jpeek 4096 Dec 12 1999 me
My feeble attempt in the second example took a lot of trial-and-error to get the right spacing, and I still had to cram DY over the tiny sixth column and TM/YR over the seventh. In the third example, column automatically adjusted the column width to compensate for the HH:MM/YEAR heading. Unfortunately, the long filename london_dusk-livesights.xwd.gz ran off the right edge (past column 80, my window width) but there was nothing column could do in this case because the combined header+columns were just too wide.
JP
As we were writing this book, I decided to make a list of all the articles and the numbers of lines and characters in each, then combine that with the description, a status code, and the article's title. After a few minutes with wc -l -c (Section 16.6), cut (Section 21.14), sort (Section 22.1), and join (Section 21.19), I had a file that looked like this:
% cat messfile 2850 2095 51441 ~BB A sed tutorial 3120 868 21259 +BB mail - lots of basics 6480 732 31034 + How to find sources - JIK's periodic posting ...900 lines... 5630 14 453 +JP Running Commands on Directory Stacks 1600 12 420 !JP With find, Don't Forget -print 0495 9 399 + Make 'xargs -i' use more than one filename
Yuck. It was tough to read: the columns needed to be straightened. The column (Section 21.16) command could do it automatically, but I wanted more control over the alignment of each column. A little awk (Section 20.10) script turned the mess into this:
% cat cleanfile 2850 2095 51441 ~BB A sed tutorial 3120 868 21259 +BB mail - lots of basics 6480 732 31034 + How to find sources - JIK's periodic posting ...900 lines... 5630 14 453 +JP Running Commands on Directory Stacks 1600 12 420 !JP With find, Don't Forget -print 0495 9 399 + Make 'xargs -i' use more than one filename
Here's the simple script I used and the command I typed to run it:
% cat neatcols { printf "%4s %4s %6s %-4s %s\n", \ $1, $2, $3, $4, substr($0, index($0,$5)) } % awk -f neatcols messfile > cleanfile
You can adapt that script for whatever kinds of columns you need to clean up. In case you don't know awk, here's a quick summary:
The first line of the printf, between double quotes ("), specifies the field widths and alignments. For example, the first column should be right-aligned in 4 characters (%4s). The fourth column should be 4 characters wide left-adjusted (%-4s). The fifth column is big enough to just fit (%s). I used string (%s) instead of decimal (%d) so awk wouldn't strip off the leading zeros in the columns.
The second line arranges the input data fields onto the output line. Here, input and output are in the same order, but I could have reordered them. The first four columns get the first four fields ($1, $2, $3, $4). The fifth column is a catch-all; it gets everything else. substr($0, index($0,$5)) means "find the fifth input column; print it and everything after it."
JP
Go to http://examples.oreilly.com/upt3 for more information on: cut+paste
Do you ever wish you could paste two (or even three) files side by side? You can, if you have the paste program (or the public-domain implementation on the disc).
For example, to create a three-column file from files x, y, and z:
$ paste x y z > file
To make paste read standard input, use the - option, and repeat - for every column you want. For example, to make an old ls (which lists files in a single column) list files in four columns:
$ ls | paste - - - -
The "standard input" option is also handy when used with cut (Section 21.14). You can cut data from one position on a line and paste it back on another.
The separate data streams being merged are separated by default with a tab, but you can change this with the -d option. Unlike the -d option to cut, you need not specify a single character; instead, you can specify a list of characters, which will be used in a circular fashion.
The characters in the list can be any regular character or the following escape sequences:
newline
tab
backslash
empty string
Use quoting (Section 27.12), if necessary, to protect characters from the shell.
There's also a -s option that lets you merge subsequent lines from one file. For example, to merge each pair of lines onto a single line:
$ paste -s -d"\t\n" list
Let's finish with one nice place to use process substitution, if your shell has it. You can use cut to grab certain columns from certain files, then use process substitution to make "files" that paste will read. Output those "files" into columns in any order you want. For example, to paste column 1 from file1 in the first output column, and column 3 from file2 in the second output column:
paste <(cut -f1 file1) <(cut -f3 file2)
If none of the shells on your system have process substitution, you can always use a bunch of temporary files, one file per column.
TOR, DG, and JP
If you've worked with databases, you'll probably know what to do with the Unix join command; see your online manual page. If you don't have a database (as far as you know!), you still probably have a use for join: combining or "joining" two column-format files. join searches certain columns in the files; when it finds columns that match one another, it "glues the lines together" at that column. This is easiest to show with an example.
I needed to summarize the information in thousands of email messages under the MH mail system. MH made that easy: it has one command (scan) that gave me almost all the information I wanted about each message and also let me specify the format I needed. But I also had to use wc -l (Section 16.6) to count the number of lines in each message. I ended up with two files, one with scan output and the other with wc output. One field in both lines was the message number; I used sort (Section 22.1) to sort the files on that field. I used awk '{print $1 "," $2}' to massage wc output into comma-separated fields. Then I used join to "glue" the two lines together on the message-number field. (Next I fed the file to a PC running dBASE, but that's another story.)
Here's the file that I told scan to output. The columns (message number, email address, comment, name, and date sent) are separated with commas (,):
0001,andrewe@isc.uci.edu,,Andy Ernbaum,19901219 0002,bc3170x@cornell.bitnet,,Zoe Doan,19910104 0003,zcode!postman@uunet.uu.net,,Head Honcho,19910105 ...
Here's the file from wc and awk with the message number and number of lines:
0001,11 0002,5 0003,187 ...
The following join command then joined the two files at their first columns (-t, tells join that the fields are comma-separated):
% join -t, scanfile wcfile
The output file looked like this:
0001,andrewe@isc.uci.edu,,Andy Ernbaum,19901219,11 0002,bc3170x@cornell.bitnet,,Zoe Doan,19910104,5 0003,zcode!postman@uunet.uu.net,,Head Honcho,19910105,187 ...
Go to http://examples.oreilly.com/upt3 for more information on: join
join can do a lot more than this simple example shows. See your online manual page. The GNU version of join is on the CD-ROM [see http://examples.oreilly.com/upt3].
JP
Go to http://examples.oreilly.com/upt3 for more information on: uniq
uniq reads a file and compares adjacent lines (which means you'll usually want to sort the file first to be sure identical lines appear next to each other). Here's what uniq can do as it watches the input lines stream by:
With the -u option, the output gets only the lines that occur just once (and weren't repeated).
The -d option does the opposite: the output gets a single copy of each line that was repeated (no matter how many times it was repeated).
(The GNU version also has a -D option. It's like -d except that all duplicate lines are output.)
The default output (with no options) is the union of -u and -d: only the first occurrence of a line is written to the output file; any adjacent copies of a line (second, third, etc.) are ignored.
The output with -c is like the default, but each line is preceded by a count of how many times it occurred.
|
Three more options control how comparisons are done:
-n ignores the first n fields of a line and all whitespace before each. A field is defined as a string of nonwhitespace characters (separated from its neighbors by whitespace).
+n ignores the first n characters. Fields are skipped before characters.
-w n in the GNU version compares no more than n characters in each line.
GNU uniq also has -i to make comparisons case-insensitive. (Upper- and lowercase letters compare equal.)
uniq is often used as a filter. See also comm (Section 11.8), sort (Section 22.1), and especially sort -u (Section 22.6).
So what can you do with all of this?
To send only one copy of each line from list (which is typically sorted) to output file list.new:
uniq list list.new
To show which names appear more than once:
sort names | uniq -d
To show which lines appear exactly three times, search the output of uniq -c for lines that start with spaces before the digit 3 and have a tab after. (This is the way GNU uniq -c makes its output lines, at least.) In the example below, the space is marked by Λ?; the TAB is marked by tab:
grep Section 13.1
sort names | uniq -c | grep "^Λ?*3tab"
The lines don't have to be sorted; they simply have to be adjacent. For example, if you have a log file where the last few fields are repeated, you can have uniq "watch" those fields and tell you how many times they were repeated. Here we'll skip the first four fields and get a count of how many times the rest of each line was repeated:
$ cat log Nov 21 17:20:19 powerd: down 2 volts Nov 21 17:20:27 powerd: down 2 volts Nov 21 17:21:15 powerd: down 2 volts Nov 21 17:22:48 powerd: down 2 volts Nov 21 18:18:02 powerd: up 3 volts Nov 21 19:55:03 powerd: down 2 volts Nov 21 19:58:41 powerd: down 2 volts $ uniq -4 -c log 4 Nov 21 17:20:19 powerd: down 2 volts 1 Nov 21 18:18:02 powerd: up 3 volts 2 Nov 21 19:55:03 powerd: down 2 volts
JP and DG
Every now and then you come across something and say, "Gee, that might come in handy someday, but I have no idea for what." This might happen to you when you're browsing at a flea market or garage sale; if you're like us, it might happen when you're browsing through public domain software.
Go to http://examples.oreilly.com/upt3 for more information on: rot
Which brings us to the rot program. rot basically just rotates text columns and rows. For example, the first column below shows an input file. The other three columns show the same file fed through rot once, twice, and three times:
|
$ cat file |
$ rot file |
$ rot file | rot |
$ rot file | rot | rot |
|---|---|---|---|
abcde |
54321 |
5 |
e |
1 |
a |
4 |
d |
2 |
b |
3 |
c |
3 |
c |
2 |
b |
4 |
d |
1 |
a |
5 |
e |
edcba |
12345 |
Now let's compare combinations of rot and tail -r (Section 42.1):
|
$ cat file |
$ rot file |
$ rot file | tail -r |
$ tail -r file | rot |
|---|---|---|---|
abcde |
54321 |
e |
12345 |
1 |
a |
d |
a |
2 |
b |
c |
b |
3 |
c |
b |
c |
4 |
d |
a |
d |
5 |
e |
54321 |
e |
rot rotates the text 90 degrees. tail -r turns the text "upside down" (last line in becomes the first line out, and so forth).
rot can also rotate the output of banner to print down a page instead of across. By now, we hope you have an idea of what rot can do!
JP and LM
[1] [The combination of tbl, nroff, and col can make ASCII tables in a few quick steps. The tables aren't sexy, but they can be quite complex. They can be emailed or printed anywhere and, because they're plain text, don't require sophisticated viewing software or equipment. tbl is a powerful way to describe tables without worrying about balancing columns or wrapping text in them. And if you want nicer-looking output, you can feed the same tbl file to groff. JP]
[2] To figure out how many numbers to count up to, divide the total size of the file by the block size you want and add one if there's a remainder. The jot program can help here.
[3] The output file size I want is denoted by the bs or "block size" parameter to dd. The 2>/dev/null (Section 36.16, Section 43.12) gets rid of dd's diagnostic output, which isn't useful here and takes up space.
[4] In this case, the repeat can actually occur only 98 times, since we've already specified two arguments and the maximum number is 100.
[5] Not really. The first file contains only nine lines (1-9); the rest contain 10. In this case, you're better off saying split -10 top_ten_list.
[6] If so, why bother gzipping? Why not forget about both gzip and uuencode? Well, you can't. Remember that tar files are binary files to start with, even if every file in the archive is an ASCII text file. You'd need to uuencode a file before mailing it, anyway, so you'd still pay the 33 percent size penalty that uuencode incurs. Using gzip minimizes the damage.
[7] With GNU tar, you can use tar czf - emacs | uuencode .... That's not the point of this example, though. We're just showing how to uuencode some arbitrary data.
| CONTENTS |


Canoemates: A Story of the Florida Reef and Everglades
by Kirk Munroe
Chapter XXVIII: “What Became of Quorum and the Canoes”
Additional Information
- Year Published: 1892
- Language: English
- Country of Origin: United States of America
- Source: Munroe, K. (1892). Canoemates: A Story of the Florida Reef and Everglades. New York, NY: Harper & Brothers.
-
Readability:
- Flesch–Kincaid Level: 9.5
- Word Count: 1,651
- Genre: Adventure
- ✎ Cite This
Downloads
WHEN Worth and Quorum were left alone they sat for some time discussing the mystery of the smoke, and whether or not they had better begin unloading the boats and preparing camp. Worth advised against this. He hoped the others would discover a better camping place than that. He also thought that perhaps they might return with news that would necessitate their leaving the island in a hurry. As he complained of being very hungry, Quorum got out the biscuit bag, and they each took a hardtack from it. It was while they were eating these that the sound of a loud “gobble, gobble, gobble,” came from the bushes, apparently but a few rods from where they sat.
Worth’s hunting instinct was at once aroused, and slipping a couple of shells into his gun, he whispered: “You sit still, Quorum, and I’ll have that fellow in a minute. My! But he must be a big one!”
Then he stepped noiselessly to the shore, and silently disappeared among the trees. Quorum sat with his back to the water, watching the spot where his young companion had entered the forest, and listening eagerly for the expected shot.
All at once a slight jar of the boat caused him to start; but before he could turn his head it was enveloped in a thick fold of cloth that effectually prevented his seeing or calling out. In a few seconds two active forms had bound his hands and feet, and slid him into the bottom of the boat, where he lay blinded, helpless, and nearly smothered. One of his captors picked up the biscuit bag from which the prisoner had just been eating, and tossed it ashore with a low laugh.
In the mean time two others had been unfastening the canoes, and dragging them cautiously backward through the opening cut in the bushes to the channel, where lay the craft in which they had come. It was a large and well-shaped cypress dugout, capable of holding a dozen men. In less than three minutes from the time of Quorum’s capture it was being poled rapidly but silently along through the twilight shadows, with the stolen boats in tow.
At a point about half a mile from the island these were skillfully concealed in a clump of tall grasses, and Quorum was bundled into the dugout. A choking sound from beneath the cloth that enveloped his head caused one of the strange canoemen to loosen it somewhat, so as to facilitate the prisoner’s breathing. Then, propelled by four pairs of lusty young arms, the dugout shot away up one of the watery lanes leading directly into the heart of the ’Glades.
An hour later it was run ashore on one of the numerous islands whose purple outlines had so charmed the observers from the top of the mound. Here it was greeted by the barking of dogs and the sound of many voices. The thongs that bound Quorum’s legs were cut, he Was lifted to his feet, and, led by two of his captors, he was made to walk for some distance. At length he was halted, his wrists were unbound, and the cloth that enveloped his head was snatched from it.
The bewildered negro was instantly confronted by such a glare of firelight that for a minute his eyes refused to perform their duty. As he stood clumsily rubbing them, he heard a titter of laughter and the subdued sound of talking.
Until that moment he had not known who had made him prisoner, nor why he had been carried off; and even now the second part of the question remained as great a mystery as ever. There was no doubt, however, that,for some purpose or other, he had been captured by a scouting party of Seminoles, and though Quorum had met individuals of this tribe while cruising on the reef, he had never visited one of their camps nor been in their power. He therefore gazed about him with considerable trepidation, and wondered what was going to be done with him.
As he did not recognize any of the dusky faces gathered in the firelight, he was amazed when one of the men, addressing him in broken English, said:
“How, Quor’m! How!  You hongry? Eat sofkee. Good, plenty.”
At the same time the speaker pointed to a smoking kettle of something that a squaw had just lifted from the fire and set close to the negro. A great wooden spoon was thrust into it, and its odor was most appetizing. Having fasted since early morning, Quorum was very hungry. Not only this, but under the circumstances he would have eaten almost anything his entertainers chose to set before him rather than run the risk of offending them. Therefore, without waiting for a second invitation, he squatted beside the kettle of sofkee, and began sampling its contents with the huge spoon. To his surprise, he bad never in his life tasted a more delicious stew. After the first mouthful, he had no hesitation in eating Such a meal as made even the Indians, among whom a large eater is considered worthy of respect, regard him with envious admiration.
It is no wonder that Quorum found this Indian food palatable, for the Seminole squaws are notable cooks, and sofkee is the tribal dish. It is a stew of venison, turtle, or some other meat, potatoes, corn, beans, peppers, and almost anything else that is at hand. It is thickened with coontie starch, and a kettleful of it is always to be found over one of the village fires, at the disposal of every hungry comer. The one drawback to its perfect enjoyment, according to a white man’s fastidious taste, is that, besides the sofkee, the wooden spoon with which it is eaten is equally at the disposal of all comers, and is in almost constant use. This fact was not known to Quorum at the time of his introduction to sofkee. If it had been, it would hardly have lessened his relish of the meal, for Quorum was too wise to be fastidious.
He was so refreshed by his supper, as well as emboldened by the fact that no one seemed inclined to harm him, that something of his natural aggressiveness returned. After laying the sofkee spoon down, he turned to the Indian who had already spoken to him, and said:
“Why fo’ yo’ call me Quor’m? I ’ain’t hab no ‘quaintance wif you.”
For answer the Indian only said, “Tobac, you got um, Quor’m?”
“Yes, sah. Tobac? I got er plenty ob him back yonder in de boat wha’ yo’ tuk me frum. Why fo’ yo’ treat a ’spectable colored gen’l’man dish yer way, anyhow? Wha’ yo’ mean by playin’ sich tricks on him, an’ on de white mans wha’ trabblin’ in he comp’ny?”
While speaking the negro had mechanically produced his black pipe, and instead of answering his questions, the Indian said: “Tobac. You no got urn. Me got um, plenty. You take um, smoke um, bimeby talk heap.”
With this he handed a plug of tobacco to the negro, who understood the action, if he had not fully comprehended the words that accompanied it. As he cut off a pipeful and carefully crumbled it in his fingers, he began to think that his position was not such a very unpleasant one, after all. He only wished he could imagine his fellow explorers as being half so comfortable as he was at that moment. Realizing from the Indian’s last remark that there would be no talk until after the smoke, he assumed as comfortable a position as possible, and gazed curiously about him.
The little village, or camp, of half a dozen huts, was nearly hidden in the black shadows of the forest trees that surrounded it on all sides. Its huts were built of poles, supporting roofs of palmetto thatch, and were open at the sides. Each was provided with a raised floor of split poles, thickly covered with skins, and every hut contained one or more cheesecloth sleeping canopies. Each hut had also several rifles and other hunting gear hanging in it, while canoe masts, sails, paddles, and push poles leaned against its walls.
The men, who lay smoking on the furs inside the huts, or stretched in comfortable attitudes on the ground outside, were tall, clean-limbed, athletic-looking fellows clad in turbans of bright colors, gay calico shirts, and moccasins of deerskin; the women wore immense necklaces of beads, calico jackets, and long skirts, but were barefooted and bareheaded; and the children were clad precisely like their elders, with the exception of the turbans, which are denied to the boys and young men until they reach the age of warriors. Besides the Indians, Quorum saw that the camp was occupied by numbers of fowls, dogs, and small black pigs, that roamed through it at will. Everybody and everything in it, animals as well as humans, looked contented and well fed.
At length Quorum’s smoke was finished, and he knocked the ashes from his pipe. As if this were a signal, the Indian men laid aside their pipes, and it was evident that the time for talking had arrived.


{% extends "layout-meta-forward.html" %} {% block page_title %} Home Office - Visa and Immigration {% endblock %} {% block content %} {% include "includes/phase_banner_alpha.html" %} {% include "includes/outgoing-header.html" %} {% include "includes/applicant-details.html" %} Email sending Waiting {% endblock %}
Waiting


Javascript Arrays
Please fix all problems in the javascript file
"js/app.js"
DO NOT open your console


My Codepen -
Code samples and projects from FreeCodeCamp.
Love Lab -
Designed & developed front end of the website (HTML5/.Net Razor MVC, CSS3/LESS, Javascript, jQuery).
ReferralClix -
Designed & developed front end of the website (HTML5, CSS3, Javascript, jQuery); Developed scrolling Javascript animation on the "How it Works" page; designed all graphics.
Kathy Kaehler -
Developed, implemented, and managed Wordpress and customized template; designed all graphics and logo; reworked data structure of the site for ease-of-use for client.
Voloe -
Designed and developed front end of the website (HTML5, CSS3, Javascript, jQuery, AJAX); designed all graphics and logo; assisted a team with implementing .NET


Modules¶
Workflow¶
Here is a description of the main script workflow with error and succes out.
Nominal case
- N1: Service started and arguments analysis
- E1: Missing argument. Return an error and log it
- E2: Bad value for an argument. Return and error and log it
- N2: Connexion to current cost
- E3: Unable to connect to current cost. Send a message over the network to inform that it is not possible to connect to current cost and log it. Return to step N3.
- N3: Waited for a message from current cost
- E4: If no message received after 30 seconds, send a message over the network to inform that there is a problem with current cost and log it. Return to step N4.
- N4: XML message received and analyzed
- E5: Incorrect message. Log this message and return to step N4. (to be defined)
- N5: Creation of a network message looking like {variableID: ..., date: ..., message: ...} and send this message over the network. Return to step N4.
- E6: Problem during message sending. Retry and log this error.
Alternative cases
- A1: USB port disconnected.
- E7: Log this error, send an error message over the network and retry to reconnect to the USB port. If USB port reconnected, return to step N2.
Test plan¶
Here is a description of test made using behave to validate this software.
Feature: Current Cost data collection
As a end-user,
I want to see my current cost information about my consumption,
in order to reduce my energy consumption
Scenario: Missing argument
When we launch currentcost script without important argument
Then we should see an error message on screen
Scenario: Problem with current cost connexion without RabbitMQ
When we start currentcost with bad port without rabbitmq
Then we should see currentcost is unreachable in log
Scenario: Problem with RabbitMQ credential
When we start currentcost with bad port and bad rabbitmq credential
Then we should see currentcost is unreachable in log
And we should see rabbitmq error in log
Scenario: Problem with current cost connexion with RabbitMQ activated
When we start currentcost with bad port with rabbitmq
Then we should see currentcost is unreachable in log
And we should receive a message saying that current cost is unreachable
Scenario: Current cost disconnected
Given current cost does not send any message
When we launch currentcost script and reach the timeout limit
Then we should get informed that current cost does not send messages
And we should see current cost does not send any message in log
Scenario: Problem with USB port
Given current cost is connected and currentcost script is launched
When we disconnect USB port
Then we should receive a message saying that current cost is disconnected
And we should see currentcost is disconnected in log
Scenario: Problem with current cost message
Given current cost is connected and script is launched
When current cost send incorrect message
Then we should get informed that current cost send incorrect message
And we should see incorrect message error in log
Scenario: Nominal case instant consumption
Given current cost is connected and script is launched
When current cost send instant consumption
Then we should receive instant consumption over the network
@prod
Scenario: Problem with current cost connexion without RabbitMQ
When we start currentcost with bad port without rabbitmq with log
Then we should see currentcost is unreachable in /var/log
@prod
Scenario: Nominal case historical consumption
Given current cost is connected and script is launched
Then we should receive historical consumption over the network
Utils module¶
- This project contains 2 mains modules:
- Utils contains all useful functions and Variables for this project,
- RabbitMQ_messager send message to RabbitMQ or display it in stdout.
Globals and function needs for this project.
- currentcost.utils.argument_parser()[source]¶
Method that parse arguments from command line, return an error in case of missing parameter or return arguments with their value.
Returns: dict – Dict containing arguments.
- currentcost.utils.data_validator(data, variable_name, site_name)[source]¶
Analyse data from currentcost and return according TOPIC and MESSAGE.
Parameters: - data – XML string that contain data.
- variable_name (str.) – Name of the Variable.
- site_name (str.) – Name of the Site.
Returns: str – Topic of the message (error or success) and Message containing error description or data sent by CC.
RabbitMQ_messager module¶
Method that send message on RabbitMQ or display it on stdout.
- class currentcost.messager.RabbitMQMessager(username, password, host)[source]¶
This class send message to RabbitMQ or in stdout following user choice.
- consume(topic, callback)[source]¶
Method that wait for a message on RabbitMQ topics channels.
Parameters: - topic (str.) – Channel where to send message (RabbitMQ filter).
- callback (Function.) – Function that compute result of message (async).
- send(topic, message, out=<open file '<stdout>', mode 'w' at 0x7fae9d0ce1e0>)[source]¶
Method that send a message with a topic.
Parameters: - topic (str.) – Channel where to send message (RabbitMQ filter).
- message (str.) – Message to send to RabbitMQ.
- out (StringIO.) – Stdout redirection in case of test (default: stdout).
- send_message(topic, site_name, var_name, data, out=<open file '<stdout>', mode 'w' at 0x7fae9d0ce1e0>)[source]¶
Format parameter into json string for phase platform.
Parameters: - topic (str.) – Channel where to send message (RabbitMQ filter).
- site_name (str.) – Name of the site.
- var_name (str.) – Variable name.
- data (str.) – Data send to send.
- out (StringIO.) – Stdout redirection in case of test (default: stdout).


Angular JS
08764335768
Welcome !
Todo Application
Notepad Application
Bootstrap Modal
Login Application
Upload File
In-line Application
Nav Menu
Switch Menu


vacuumdb - garbage-collect and analyze a PostgreSQL database
vacuumdb [connection-option...] [option...] [ --table | -t table [( column [,...] )] ]... [dbname]
vacuumdb [connection-option...] [option...] --all | -a
vacuumdb is a utility for cleaning a PostgreSQL database. vacuumdb will also generate internal statistics used by the PostgreSQL query optimizer.
vacuumdb is a wrapper around the SQL command VACUUM(7). There is no effective difference between vacuuming and analyzing databases via this utility and via other methods for accessing the server.
vacuumdb accepts the following command-line arguments:
-a
--all
Vacuum all databases.
[-d] dbname
[--dbname=]dbname
Specifies the name of the database to be cleaned or analyzed. If this is not specified and -a (or --all) is not used, the database name is read from the environment variable PGDATABASE. If that is not set, the user name specified for the connection is used.
-e
--echo
Echo the commands that vacuumdb generates and sends to the server.
-f
--full
Perform (lqfull(rq vacuuming.
-F
--freeze
Aggressively (lqfreeze(rq tuples.
-q
--quiet
Do not display progress messages.
-t table [ (column [,...]) ]
--table=table [ (column [,...]) ]
Clean or analyze table only. Column names can be specified only in conjunction with the --analyze or --analyze-only options. Multiple tables can be vacuumed by writing multiple -t switches.
Tip
If you specify columns, you probably have to escape the parentheses from the shell. (See examples below.)
-v
--verbose
Print detailed information during processing.
-V
--version
Print the vacuumdb version and exit.
-z
--analyze
Also calculate statistics for use by the optimizer.
-Z
--analyze-only
Only calculate statistics for use by the optimizer (no vacuum).
--analyze-in-stages
Only calculate statistics for use by the optimizer (no vacuum), like --analyze-only. Run several (currently three) stages of analyze with different configuration settings, to produce usable statistics faster.
This option is useful to analyze a database that was newly populated from a restored dump or by pg_upgrade. This option will try to create some statistics as fast as possible, to make the database usable, and then produce full statistics in the subsequent stages.
-?
--help
Show help about vacuumdb command line arguments, and exit.
vacuumdb also accepts the following command-line arguments for connection parameters:
-h host
--host=host
Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket.
-p port
--port=port
Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections.
-U username
--username=username
User name to connect as.
-w
--no-password
Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.
-W
--password
Force vacuumdb to prompt for a password before connecting to a database.
This option is never essential, since vacuumdb will automatically prompt for a password if the server demands password authentication. However, vacuumdb will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.
--maintenance-db=dbname
Specifies the name of the database to connect to discover what other databases should be vacuumed. If not specified, the postgres database will be used, and if that does not exist, template1 will be used.
PGDATABASE
PGHOST
PGPORT
PGUSER
Default connection parameters
This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 31.14, (lqEnvironment Variables(rq, in the documentation).
In case of difficulty, see VACUUM(7) and psql(1) for discussions of potential problems and error messages. The database server must be running at the targeted host. Also, any default connection settings and environment variables used by the libpq front-end library will apply.
vacuumdb might need to connect several times to the PostgreSQL server, asking for a password each time. It is convenient to have a ~/.pgpass file in such cases. See Section 31.15, (lqThe Password File(rq, in the documentation for more information.
To clean the database test:
$ vacuumdb test
To clean and analyze for the optimizer a database named bigdb:
$ vacuumdb --analyze bigdb
To clean a single table foo in a database named xyzzy, and analyze a single column bar of the table for the optimizer:
$ vacuumdb --analyze --verbose --table *(Aqfoo(bar)*(Aq xyzzy
VACUUM(7)


Contact an adviser
Explore the different ways to get in touch with an adviser. You can also send us feedback about our service.
Contact an advisor
Webchat
Chat with an adviser using webchat - 8am to 10pm, 7 days a week.
Phone
Call 0800 100 900 to speak to an adviser - 8am to 10pm, 7 days a week. Calls are free from landlines and most mobile numbers. Find out more about call charges and freephone numbers.
You can speak to an adviser, by choosing option 5 for advice in a different language, using an interpreter.
Web form
Use our web form to ask your question or query and we'll get back to you as soon as possible.
Call back
Tell us your number and we'll call you back.
Face-to-face advice
Call 0800 100 900 to find out if you're eligible for a face-to-face appointment with an adviser.
Text
Text us on 07766 413219 for information like telephone numbers or addresses. We can't open pictures sent by text.
Post
Contact us by post at:
National Careers Service
PO Box 1331
Newcastle Upon Tyne
NE99 5EB
Feedback
We welcome feedback. Complete our feedback form to make a suggestion or a comment.
Technical support
Complete our technical support form if you're in England and need technical help to use any of the tools on the website. If you prefer to speak with someone, call 0800 100 900 (8am to 10pm, 7 days a week).


EXCLUSIVE: Nick Cannon: The Hood Doesn't Listen to Eminem, Whites are Guests in Hip Hop
by Jessica ToddIn this flashback, Nick Cannon shared his thoughts on Conway's reaction to Lord Jamar saying that white people, including Eminem, are guests in the house of hip-hop. After pointing out that Eminem is in his top-five rappers list, Nick went on to speak about how he agrees with Lord Jamar, saying, "Which is a fact." Nick also agreed with DJ Vlad telling Conway that the hood didn't listen to Eminem, which led to a conversation about teenage girls deciding who is a huge superstar because they are a large market that buys merchandise and tickets to shows. To hear Nick's full statement, hit the above clip.


Toggle navigation
Home
About
Taxonomy Status
Curation
Spider Behavior Vocabulary
Resources
Current work area: data cleanup and grouping/sorting of terms in progress.
Current List
.
Original set of collected terms, ordered by location in source volumes. Minimal data cleaning:
Raw Source List
.
Notes
Each notebook has multiple sheets (tabs).
Behavior
contains terms that are the main focus
Web Morphology
terms for web shapes and parts - maybe a separate vocabulary.
Not Spider Specific
behavior terms that are more general, should be terms in more general ontologies.
Chemistry
mostly spider pheromones, should be added to the ChEBI ontology.
Qualities
terms that are not processes or independent continuents, but their placement is still controversial.


DPROFPP(1) Perl Programmers Reference Guide DPROFPP(1) NAME dprofpp - display perl profile data SYNOPSIS dprofpp [-a|-z|-l|-v|-U] [-d] [-s|-r|-u] [-q] [-F] [-I|-E] [-O cnt] [-A] [-R] [-S] [-g subroutine] [-G <regexp> [-P]] [-f <regexp>] [profile] dprofpp -T [-F] [-g subroutine] [profile] dprofpp -t [-F] [-g subroutine] [profile] dprofpp -G <regexp> [-P] [profile] dprofpp -p script [-Q] [other opts] dprofpp -V [profile] DESCRIPTION The dprofpp command interprets profile data produced by a profiler, such as the Devel::DProf profiler. Dprofpp will read the file tmon.out and display the 15 subroutines which are using the most time. By default the times for each subroutine are given exclusive of the times of their child subroutines. To profile a Perl script run the perl interpreter with the -d switch. So to profile script test.pl with Devel::DProf use the following: $ perl5 -d:DProf test.pl Then run dprofpp to analyze the profile. The output of dprofpp depends on the flags to the program and the version of Perl you're using. $ dprofpp -u Total Elapsed Time = 1.67 Seconds User Time = 0.61 Seconds Exclusive Times %Time Seconds #Calls sec/call Name 52.4 0.320 2 0.1600 main::foo 45.9 0.280 200 0.0014 main::bar 0.00 0.000 1 0.0000 DynaLoader::import 0.00 0.000 1 0.0000 main::baz The dprofpp tool can also run the profiler before analyzing the profile data. The above two commands can be executed with one dprofpp command. $ dprofpp -u -p test.pl Consult "PROFILE FORMAT" in Devel::DProf for a description of the raw profile. OUTPUT Columns are: %Time Percentage of time spent in this routine. #Calls Number of calls to this routine. sec/call Average number of seconds per call to this routine. Name Name of routine. CumulS Time (in seconds) spent in this routine and routines called from it. ExclSec Time (in seconds) spent in this routine (not including those called from it). Csec/c Average time (in seconds) spent in each call of this routine (including those called from it). OPTIONS -a Sort alphabetically by subroutine names. -d Reverse whatever sort is used -A Count timing for autoloaded subroutine as timing for *::AUTOLOAD. Otherwise the time to autoload it is counted as time of the subroutine itself (there is no way to separate autoload time from run time). This is going to be irrelevant with newer Perls. They will inform "Devel::DProf" when the "AUTOLOAD" switches to actual subroutine, so a separate statistics for "AUTOLOAD" will be collected no matter whether this option is set. -R Count anonymous subroutines defined in the same package separately. -E (default) Display all subroutine times exclusive of child subroutine times. -F Force the generation of fake exit timestamps if dprofpp reports that the profile is garbled. This is only useful if dprofpp determines that the profile is garbled due to missing exit timestamps. You're on your own if you do this. Consult the BUGS section. -I Display all subroutine times inclusive of child subroutine times. -l Sort by number of calls to the subroutines. This may help identify candidates for inlining. -O cnt Show only cnt subroutines. The default is 15. -p script Tells dprofpp that it should profile the given script and then interpret its profile data. See -Q. -Q Used with -p to tell dprofpp to quit after profiling the script, without interpreting the data. -q Do not display column headers. -r Display elapsed real times rather than user+system times. -s Display system times rather than user+system times. -T Display subroutine call tree to stdout. Subroutine statistics are not displayed. -t Display subroutine call tree to stdout. Subroutine statistics are not displayed. When a function is called multiple consecutive times at the same calling level then it is displayed once with a repeat count. -S Display merged subroutine call tree to stdout. Statistics are displayed for each branch of the tree. When a function is called multiple (not necessarily consecutive) times in the same branch then all these calls go into one branch of the next level. A repeat count is output together with combined inclusive, exclusive and kids time. Branches are sorted with regard to inclusive time. -U Do not sort. Display in the order found in the raw profile. -u Display user times rather than user+system times. -V Print dprofpp's version number and exit. If a raw profile is found then its XS_VERSION variable will be displayed, too. -v Sort by average time spent in subroutines during each call. This may help identify candidates for inlining. -z (default) Sort by amount of user+system time used. The first few lines should show you which subroutines are using the most time. -g "subroutine" Ignore subroutines except "subroutine" and whatever is called from it. -G <regexp> Aggregate "Group" all calls matching the pattern together. For example this can be used to group all calls of a set of packages -G "(package1::)|(package2::)|(package3::)" or to group subroutines by name: -G "getNum" -P Used with -G to aggregate "Pull" together all calls that did not match -G. -f <regexp> Filter all calls matching the pattern. -h Display brief help and exit. -H Display long help and exit. ENVIRONMENT The environment variable DPROFPP_OPTS can be set to a string containing options for dprofpp. You might use this if you prefer -I over -E or if you want -F on all the time. This was added fairly lazily, so there are some undesirable side effects. Options on the commandline should override options in DPROFPP_OPTS--but don't count on that in this version. BUGS Applications which call _exit() or exec() from within a subroutine will leave an incomplete profile. See the -F option. Any bugs in Devel::DProf, or any profiler generating the profile data, could be visible here. See "BUGS" in Devel::DProf. Mail bug reports and feature requests to the perl5-porters mailing list at<EMAIL_ADDRESS>Bug reports should include the output of the -V option. FILES dprofpp - profile processor tmon.out - raw profile SEE ALSO perl, Devel::DProf, times(2) perl v5.14.2 2014-09-30 DPROFPP(1)


I have learned many things till now and some of them are Video Creation, Video Editing, Basics of C Language, HTML5, Basic properties of CSS etc.. I also know basic game development. I am learning more things and that will be soon updated on the site.
➲ I have made this site just using HTML and CSS.
➲ I haven't used any third party library like jQuery, BootStrap, Materialize CSS.
➲ Icons are provided by Flaticon.
➲ This website is inspired by Creative Shivam.
➲ This website is for my portfolio projects and I will be soon uploading them here and this website is also a part of my portfolio.


Add New User
Enter values for username, password, and full name to create a new user in the table.
username:
password:
full name:


About
Dozens of countries around the world are deciding whether or not to develop their shale gas and tight oil resources. However, extracting these energy sources poses environmental risks, especially to water.
Drilling and hydraulic fracturing requires up to 25 million liters of water per well, meaning shale resources can be hard to develop where freshwater is hard to find.
The risks and impacts specific to surface and groundwater availability have been thinly documented to date. This tool and the associated report:
- Identify priority locations worldwide where freshwater management will be most critical if shale is developed.
- Reveal potential business risks associated with freshwater availability to companies and build the case for corporate water stewardship and early source-water assessment.
Scope
This tool shares information that can create dialog among water users from industry, government, and civil society in river basins worldwide. It does not attempt to identify risks to water quality from shale resource development, nor does it assess the oil and gas industry’s water management practices.
Definitions
- Baseline water stress: The ratio of total water withdrawals to available renewable supply in an area. In high-stress areas, 40 percent or more of the available supply is withdrawn every year. In extremely high-stress areas, that number goes up to 80 percent or higher. A higher percentage means more water users are competing for limited supplies. See the high and extremely high-stress areas highlighted in red and dark red on the maps. For more detailed information, please see Aqueduct’s Global Maps 2.0 metadata document.
- Hydraulic fracturing: A method of extraction for shale gas and tight oil resources. Fluid is pumped at high pressure down a well to create cracks in low-permeability geological formations. Natural gas and oil then flows from the cracks back into the well.
- Shale play: Part of a shale basin that can be commercially extracted.
- Shale gas: Natural gas deposit found in shale reservoirs, which are between ten and many thousands of times less permeable than conventional natural gas reservoirs.
- Tight oil: Oil trapped in fine-grained sedimentary rocks with extremely low permeability, such as shale, sandstone, or carbonate
- Technically recoverable resource: Shale oil or gas deposit that can be extracted with current technology, but does not consider economic viability.
Report Highlights
Read: Global Shale Gas Development: Water Availability and Business Risks
Global Shale Gas Development analyzes water availability across all potentially commercial shale resources worldwide, and finds that limited freshwater supplies could pose challenges to shale resource development on every continent.
-
38 percent of the world's shale resources face high to extremely high water stress or arid conditions.
-
386 million people live on land above shale plays - increased competition for water and public concern over hydraulic fracturing is more likely in densely populated areas.
WRI assessed water stress in the 20 countries with the largest shale resources. For shale gas, plays in 40% of those countries face high to extremely high water stress: China, Algeria, Mexico, South Africa, Libya, Pakistan, Egypt, and India.
WRI also analyzed water availability for each shale play in 11 countries already developing – or more likely to develop – shale resources: Algeria, Argentina, Australia, Canada, China, Mexico, Poland, Saudi Arabia, South Africa, the United Kingdom, and the United States.
-
China is shale rich, but water stressed. It has the world's largest commercially viable shale reserves, but over 60 percent of them face high water stress or arid conditions.
-
Argentina is shale rich and less water stressed. It has the world's second-largest technically recoverable gas reserves, but 71% of resources face low to medium stress.
-
The United Kingdom, is Europe's third-largest natural gas producer, but is also densely populated, and 34% of resources face high water stress or arid conditions.
Included in the Report
Six indicators and associated business risks for shale development: Water stress, water-supply variation among months of the year, drought severity, groundwater depletion rates, largest water user, population density, and depth of shale reserve.
Data Sources
Reccomendations
Governments, companies, and civil society can help protect water security while minimizing business risks by:
- Conducting water risk assessments to understand local water availability and reduce business risk
- Increasing transparency and engaging with local regulators, communities, and industry to minimize uncertainty
- Ensuring adequate water governance to guarantee water security and reduce regulatory and reputational risks
- •Minimizing freshwater use and engaging in corporate water stewardship to reduce impacts on water availability
Read the full report.
Countries
Eight of the top 20 countries with the largest shale gas technically recoverable resources (TRR) face arid conditions or an average of high to extremely high baseline water stress where the shale is located. China, Mexico, and South Africa stand out, ranking very highly based on the size of their resources and exposure to baseline water stress.
|
Country |
Estimated resources (trillion cubic feet) |
Average water stress over shale play area |
| 1 |
China |
1,115 |
High |
| 2 |
Argentina |
802 |
Low to Medium |
| 3 |
Algeria |
707 |
Arid & Low Water Use |
| 4 |
Canada |
573 |
Low to Medium |
| 5 |
United States |
567 |
Medium to High |
| 6 |
Mexico |
545 |
High |
| 7 |
Australia |
437 |
Low |
| 8 |
South Africa |
390 |
High |
| 9 |
Russian Federation |
287 |
Low |
| 10 |
Brazil |
245 |
Low |
| 11 |
Venezuela |
167 |
Low |
| 12 |
Poland |
148 |
Low to Medium |
| 13 |
France |
137 |
Low to Medium |
| 14 |
Ukraine |
128 |
Low to Medium |
| 15 |
Libya |
122 |
Arid & Low Water Use |
| 16 |
Pakistan |
105 |
Extremely High |
| 17 |
Egypt, Arab Rep. |
100 |
Arid & Low Water Use |
| 18 |
India |
96 |
High |
| 19 |
Paraguay |
75 |
Medium to High |
| 20 |
Colombia |
55 |
Low |
Data
Download the Global Shale Resources Geodatabase from West Virginia University.
Download Aqueduct Global Water Risk Atlas.