Prototype (R package).
Model and code are based on examples in Gelman et al. (2014) and McElreath (2020).
This prototype is related to a previous simulation study.
The distributions and parameters in the simulation study were chosen so that the generated data is comparable to certain data of interest. The prototype is meant to be used for roughly the same kind of data. Please refer to this report for details.
The package documentation can be accessed via the package website.
First install dependencies.
To get a working C++ toolchain on macOS Big Sur (version 11.2.1 or higher), install Xcode from the App Store or install Xcode command line tools in a shell:
% xcode-select --install
Follow the instructions for cmdstanr. The macOS R toolchain installer referred to in the RStan documentation is then not needed.
Next, install the following R packages and their dependencies in this order:
The rethinking package needs to be installed from GitHub.
The prototype can then be installed from GitHub with:
devtools::install_github("https://github.com/jmeydam/countcheck.git",
build_vignettes = FALSE)
We simulate count data for 1000 observational units, assuming a certain known data-generating process.
The data-generating process is free of anomalies, and all counts exceeding their respective upper control limits (UCLs) are false positives. Other things being equal it is desirable to have as few false positives as possible.
When we compare the results for the partial-pooling and complete pooling models we tend to find that using a Bayesian hierarchical model with partial pooling far fewer new counts exceed their respective upper control limits.
d <- countcheck(random_seed = 200807)
#> This is cmdstanr version 0.3.0
#> - Online documentation and vignettes at mc-stan.org/cmdstanr
#> - CmdStan path set to: /Users/jens/.cmdstanr/cmdstan-2.26.0
#> - Use set_cmdstan_path() to change the path
#> Compiling Stan program...
#> Running MCMC with 4 sequential chains, with 1 thread(s) per chain...
#>
#> Chain 1 Iteration: 1 / 4000 [ 0%] (Warmup)
#> Chain 1 Iteration: 100 / 4000 [ 2%] (Warmup)
#> Chain 1 Iteration: 200 / 4000 [ 5%] (Warmup)
#> Chain 1 Iteration: 300 / 4000 [ 7%] (Warmup)
#> Chain 1 Iteration: 400 / 4000 [ 10%] (Warmup)
#> Chain 1 Iteration: 500 / 4000 [ 12%] (Warmup)
#> Chain 1 Iteration: 600 / 4000 [ 15%] (Warmup)
#> Chain 1 Iteration: 700 / 4000 [ 17%] (Warmup)
#> Chain 1 Iteration: 800 / 4000 [ 20%] (Warmup)
#> Chain 1 Iteration: 900 / 4000 [ 22%] (Warmup)
#> Chain 1 Iteration: 1000 / 4000 [ 25%] (Warmup)
#> Chain 1 Iteration: 1100 / 4000 [ 27%] (Warmup)
#> Chain 1 Iteration: 1200 / 4000 [ 30%] (Warmup)
#> Chain 1 Iteration: 1300 / 4000 [ 32%] (Warmup)
#> Chain 1 Iteration: 1400 / 4000 [ 35%] (Warmup)
#> Chain 1 Iteration: 1500 / 4000 [ 37%] (Warmup)
#> Chain 1 Iteration: 1600 / 4000 [ 40%] (Warmup)
#> Chain 1 Iteration: 1700 / 4000 [ 42%] (Warmup)
#> Chain 1 Iteration: 1800 / 4000 [ 45%] (Warmup)
#> Chain 1 Iteration: 1900 / 4000 [ 47%] (Warmup)
#> Chain 1 Iteration: 2000 / 4000 [ 50%] (Warmup)
#> Chain 1 Iteration: 2001 / 4000 [ 50%] (Sampling)
#> Chain 1 Iteration: 2100 / 4000 [ 52%] (Sampling)
#> Chain 1 Iteration: 2200 / 4000 [ 55%] (Sampling)
#> Chain 1 Iteration: 2300 / 4000 [ 57%] (Sampling)
#> Chain 1 Iteration: 2400 / 4000 [ 60%] (Sampling)
#> Chain 1 Iteration: 2500 / 4000 [ 62%] (Sampling)
#> Chain 1 Iteration: 2600 / 4000 [ 65%] (Sampling)
#> Chain 1 Iteration: 2700 / 4000 [ 67%] (Sampling)
#> Chain 1 Iteration: 2800 / 4000 [ 70%] (Sampling)
#> Chain 1 Iteration: 2900 / 4000 [ 72%] (Sampling)
#> Chain 1 Iteration: 3000 / 4000 [ 75%] (Sampling)
#> Chain 1 Iteration: 3100 / 4000 [ 77%] (Sampling)
#> Chain 1 Iteration: 3200 / 4000 [ 80%] (Sampling)
#> Chain 1 Iteration: 3300 / 4000 [ 82%] (Sampling)
#> Chain 1 Iteration: 3400 / 4000 [ 85%] (Sampling)
#> Chain 1 Iteration: 3500 / 4000 [ 87%] (Sampling)
#> Chain 1 Iteration: 3600 / 4000 [ 90%] (Sampling)
#> Chain 1 Iteration: 3700 / 4000 [ 92%] (Sampling)
#> Chain 1 Iteration: 3800 / 4000 [ 95%] (Sampling)
#> Chain 1 Iteration: 3900 / 4000 [ 97%] (Sampling)
#> Chain 1 Iteration: 4000 / 4000 [100%] (Sampling)
#> Chain 1 finished in 14.9 seconds.
#> Chain 2 Iteration: 1 / 4000 [ 0%] (Warmup)
#> Chain 2 Iteration: 100 / 4000 [ 2%] (Warmup)
#> Chain 2 Iteration: 200 / 4000 [ 5%] (Warmup)
#> Chain 2 Iteration: 300 / 4000 [ 7%] (Warmup)
#> Chain 2 Iteration: 400 / 4000 [ 10%] (Warmup)
#> Chain 2 Iteration: 500 / 4000 [ 12%] (Warmup)
#> Chain 2 Iteration: 600 / 4000 [ 15%] (Warmup)
#> Chain 2 Iteration: 700 / 4000 [ 17%] (Warmup)
#> Chain 2 Iteration: 800 / 4000 [ 20%] (Warmup)
#> Chain 2 Iteration: 900 / 4000 [ 22%] (Warmup)
#> Chain 2 Iteration: 1000 / 4000 [ 25%] (Warmup)
#> Chain 2 Iteration: 1100 / 4000 [ 27%] (Warmup)
#> Chain 2 Iteration: 1200 / 4000 [ 30%] (Warmup)
#> Chain 2 Iteration: 1300 / 4000 [ 32%] (Warmup)
#> Chain 2 Iteration: 1400 / 4000 [ 35%] (Warmup)
#> Chain 2 Iteration: 1500 / 4000 [ 37%] (Warmup)
#> Chain 2 Iteration: 1600 / 4000 [ 40%] (Warmup)
#> Chain 2 Iteration: 1700 / 4000 [ 42%] (Warmup)
#> Chain 2 Iteration: 1800 / 4000 [ 45%] (Warmup)
#> Chain 2 Iteration: 1900 / 4000 [ 47%] (Warmup)
#> Chain 2 Iteration: 2000 / 4000 [ 50%] (Warmup)
#> Chain 2 Iteration: 2001 / 4000 [ 50%] (Sampling)
#> Chain 2 Iteration: 2100 / 4000 [ 52%] (Sampling)
#> Chain 2 Iteration: 2200 / 4000 [ 55%] (Sampling)
#> Chain 2 Iteration: 2300 / 4000 [ 57%] (Sampling)
#> Chain 2 Iteration: 2400 / 4000 [ 60%] (Sampling)
#> Chain 2 Iteration: 2500 / 4000 [ 62%] (Sampling)
#> Chain 2 Iteration: 2600 / 4000 [ 65%] (Sampling)
#> Chain 2 Iteration: 2700 / 4000 [ 67%] (Sampling)
#> Chain 2 Iteration: 2800 / 4000 [ 70%] (Sampling)
#> Chain 2 Iteration: 2900 / 4000 [ 72%] (Sampling)
#> Chain 2 Iteration: 3000 / 4000 [ 75%] (Sampling)
#> Chain 2 Iteration: 3100 / 4000 [ 77%] (Sampling)
#> Chain 2 Iteration: 3200 / 4000 [ 80%] (Sampling)
#> Chain 2 Iteration: 3300 / 4000 [ 82%] (Sampling)
#> Chain 2 Iteration: 3400 / 4000 [ 85%] (Sampling)
#> Chain 2 Iteration: 3500 / 4000 [ 87%] (Sampling)
#> Chain 2 Iteration: 3600 / 4000 [ 90%] (Sampling)
#> Chain 2 Iteration: 3700 / 4000 [ 92%] (Sampling)
#> Chain 2 Iteration: 3800 / 4000 [ 95%] (Sampling)
#> Chain 2 Iteration: 3900 / 4000 [ 97%] (Sampling)
#> Chain 2 Iteration: 4000 / 4000 [100%] (Sampling)
#> Chain 2 finished in 15.7 seconds.
#> Chain 3 Iteration: 1 / 4000 [ 0%] (Warmup)
#> Chain 3 Iteration: 100 / 4000 [ 2%] (Warmup)
#> Chain 3 Iteration: 200 / 4000 [ 5%] (Warmup)
#> Chain 3 Iteration: 300 / 4000 [ 7%] (Warmup)
#> Chain 3 Iteration: 400 / 4000 [ 10%] (Warmup)
#> Chain 3 Iteration: 500 / 4000 [ 12%] (Warmup)
#> Chain 3 Iteration: 600 / 4000 [ 15%] (Warmup)
#> Chain 3 Iteration: 700 / 4000 [ 17%] (Warmup)
#> Chain 3 Iteration: 800 / 4000 [ 20%] (Warmup)
#> Chain 3 Iteration: 900 / 4000 [ 22%] (Warmup)
#> Chain 3 Iteration: 1000 / 4000 [ 25%] (Warmup)
#> Chain 3 Iteration: 1100 / 4000 [ 27%] (Warmup)
#> Chain 3 Iteration: 1200 / 4000 [ 30%] (Warmup)
#> Chain 3 Iteration: 1300 / 4000 [ 32%] (Warmup)
#> Chain 3 Iteration: 1400 / 4000 [ 35%] (Warmup)
#> Chain 3 Iteration: 1500 / 4000 [ 37%] (Warmup)
#> Chain 3 Iteration: 1600 / 4000 [ 40%] (Warmup)
#> Chain 3 Iteration: 1700 / 4000 [ 42%] (Warmup)
#> Chain 3 Iteration: 1800 / 4000 [ 45%] (Warmup)
#> Chain 3 Iteration: 1900 / 4000 [ 47%] (Warmup)
#> Chain 3 Iteration: 2000 / 4000 [ 50%] (Warmup)
#> Chain 3 Iteration: 2001 / 4000 [ 50%] (Sampling)
#> Chain 3 Iteration: 2100 / 4000 [ 52%] (Sampling)
#> Chain 3 Iteration: 2200 / 4000 [ 55%] (Sampling)
#> Chain 3 Iteration: 2300 / 4000 [ 57%] (Sampling)
#> Chain 3 Iteration: 2400 / 4000 [ 60%] (Sampling)
#> Chain 3 Iteration: 2500 / 4000 [ 62%] (Sampling)
#> Chain 3 Iteration: 2600 / 4000 [ 65%] (Sampling)
#> Chain 3 Iteration: 2700 / 4000 [ 67%] (Sampling)
#> Chain 3 Iteration: 2800 / 4000 [ 70%] (Sampling)
#> Chain 3 Iteration: 2900 / 4000 [ 72%] (Sampling)
#> Chain 3 Iteration: 3000 / 4000 [ 75%] (Sampling)
#> Chain 3 Iteration: 3100 / 4000 [ 77%] (Sampling)
#> Chain 3 Iteration: 3200 / 4000 [ 80%] (Sampling)
#> Chain 3 Iteration: 3300 / 4000 [ 82%] (Sampling)
#> Chain 3 Iteration: 3400 / 4000 [ 85%] (Sampling)
#> Chain 3 Iteration: 3500 / 4000 [ 87%] (Sampling)
#> Chain 3 Iteration: 3600 / 4000 [ 90%] (Sampling)
#> Chain 3 Iteration: 3700 / 4000 [ 92%] (Sampling)
#> Chain 3 Iteration: 3800 / 4000 [ 95%] (Sampling)
#> Chain 3 Iteration: 3900 / 4000 [ 97%] (Sampling)
#> Chain 3 Iteration: 4000 / 4000 [100%] (Sampling)
#> Chain 3 finished in 15.7 seconds.
#> Chain 4 Iteration: 1 / 4000 [ 0%] (Warmup)
#> Chain 4 Iteration: 100 / 4000 [ 2%] (Warmup)
#> Chain 4 Iteration: 200 / 4000 [ 5%] (Warmup)
#> Chain 4 Iteration: 300 / 4000 [ 7%] (Warmup)
#> Chain 4 Iteration: 400 / 4000 [ 10%] (Warmup)
#> Chain 4 Iteration: 500 / 4000 [ 12%] (Warmup)
#> Chain 4 Iteration: 600 / 4000 [ 15%] (Warmup)
#> Chain 4 Iteration: 700 / 4000 [ 17%] (Warmup)
#> Chain 4 Iteration: 800 / 4000 [ 20%] (Warmup)
#> Chain 4 Iteration: 900 / 4000 [ 22%] (Warmup)
#> Chain 4 Iteration: 1000 / 4000 [ 25%] (Warmup)
#> Chain 4 Iteration: 1100 / 4000 [ 27%] (Warmup)
#> Chain 4 Iteration: 1200 / 4000 [ 30%] (Warmup)
#> Chain 4 Iteration: 1300 / 4000 [ 32%] (Warmup)
#> Chain 4 Iteration: 1400 / 4000 [ 35%] (Warmup)
#> Chain 4 Iteration: 1500 / 4000 [ 37%] (Warmup)
#> Chain 4 Iteration: 1600 / 4000 [ 40%] (Warmup)
#> Chain 4 Iteration: 1700 / 4000 [ 42%] (Warmup)
#> Chain 4 Iteration: 1800 / 4000 [ 45%] (Warmup)
#> Chain 4 Iteration: 1900 / 4000 [ 47%] (Warmup)
#> Chain 4 Iteration: 2000 / 4000 [ 50%] (Warmup)
#> Chain 4 Iteration: 2001 / 4000 [ 50%] (Sampling)
#> Chain 4 Iteration: 2100 / 4000 [ 52%] (Sampling)
#> Chain 4 Iteration: 2200 / 4000 [ 55%] (Sampling)
#> Chain 4 Iteration: 2300 / 4000 [ 57%] (Sampling)
#> Chain 4 Iteration: 2400 / 4000 [ 60%] (Sampling)
#> Chain 4 Iteration: 2500 / 4000 [ 62%] (Sampling)
#> Chain 4 Iteration: 2600 / 4000 [ 65%] (Sampling)
#> Chain 4 Iteration: 2700 / 4000 [ 67%] (Sampling)
#> Chain 4 Iteration: 2800 / 4000 [ 70%] (Sampling)
#> Chain 4 Iteration: 2900 / 4000 [ 72%] (Sampling)
#> Chain 4 Iteration: 3000 / 4000 [ 75%] (Sampling)
#> Chain 4 Iteration: 3100 / 4000 [ 77%] (Sampling)
#> Chain 4 Iteration: 3200 / 4000 [ 80%] (Sampling)
#> Chain 4 Iteration: 3300 / 4000 [ 82%] (Sampling)
#> Chain 4 Iteration: 3400 / 4000 [ 85%] (Sampling)
#> Chain 4 Iteration: 3500 / 4000 [ 87%] (Sampling)
#> Chain 4 Iteration: 3600 / 4000 [ 90%] (Sampling)
#> Chain 4 Iteration: 3700 / 4000 [ 92%] (Sampling)
#> Chain 4 Iteration: 3800 / 4000 [ 95%] (Sampling)
#> Chain 4 Iteration: 3900 / 4000 [ 97%] (Sampling)
#> Chain 4 Iteration: 4000 / 4000 [100%] (Sampling)
#> Chain 4 finished in 16.4 seconds.
#>
#> All 4 chains finished successfully.
#> Mean chain execution time: 15.7 seconds.
#> Total execution time: 63.2 seconds.
#> 1000 vector or matrix parameters hidden. Use depth=2 to show them.
#> mean sd 5.5% 94.5% n_eff Rhat4
#> alpha 0.04905132 0.001360069 0.04697336 0.05128713 7247.988 0.9998833
Result:
# Counts exceeding UCLs
# *********************
# ucl_true_theta:
sum(d$y_new - d$ucl_true_theta > 0)
#> [1] 4
# ucl_nopool:
sum(d$y_new - d$ucl_nopool > 0)
#> [1] 68
# ucl_complpool:
sum(d$y_new - d$ucl_complpool > 0)
#> [1] 74
# ucl_partpool:
sum(d$y_new - d$ucl_partpool > 0)
#> [1] 18
d[d$y_new - d$ucl_partpool > 0,
c("n", "y", "n_new", "y_new",
"true_theta", "theta_partpool",
"ucl_true_theta", "ucl_partpool")]
#> n y n_new y_new true_theta theta_partpool ucl_true_theta ucl_partpool
#> 109 24 0 12 3 0.05812885 0.024302886 3.5 2.5
#> 228 54 1 27 4 0.11591158 0.027967910 8.5 3.5
#> 364 95 5 48 8 0.10194831 0.050417683 12.5 7.5
#> 374 100 2 50 6 0.07644745 0.026326791 10.5 5.5
#> 399 110 1 55 5 0.01644587 0.016491387 4.5 4.5
#> 462 136 2 68 8 0.04883827 0.020516222 9.5 5.5
#> 501 157 9 79 12 0.08478296 0.054769164 14.5 11.5
#> 529 171 5 86 9 0.05595134 0.032173221 11.5 8.5
#> 598 210 1 105 5 0.03900614 0.009289763 10.5 4.5
#> 613 219 2 110 6 0.01705977 0.013208173 6.5 5.5
#> 635 239 12 120 14 0.06846729 0.049739447 17.5 13.5
#> 698 289 9 145 16 0.04752106 0.032966185 15.5 11.5
#> 751 336 3 168 7 0.01771915 0.011626066 8.5 6.5
#> 805 410 39 205 33 0.15066465 0.089261026 48.5 31.5
#> 894 612 26 306 25 0.05070100 0.042676439 27.5 24.5
#> 935 802 54 401 51 0.09019851 0.066278225 54.5 42.5
#> 967 1040 26 520 25 0.03408030 0.025680542 30.5 24.5
#> 974 1124 73 562 55 0.07536090 0.064351810 62.5 54.5
Using the dataset provided in the package, the same result (here without the true_theta values) can be obtained with:
e <- simdat
d1 <- countcheck(unit = e$unit,
n = e$n,
y = e$y,
n_new = e$n_new,
y_new = e$y_new,
random_seed = 200807)
#> Compiling Stan program...
#> Running MCMC with 4 sequential chains, with 1 thread(s) per chain...
#>
#> Chain 1 Iteration: 1 / 4000 [ 0%] (Warmup)
#> Chain 1 Iteration: 100 / 4000 [ 2%] (Warmup)
#> Chain 1 Iteration: 200 / 4000 [ 5%] (Warmup)
#> Chain 1 Iteration: 300 / 4000 [ 7%] (Warmup)
#> Chain 1 Iteration: 400 / 4000 [ 10%] (Warmup)
#> Chain 1 Iteration: 500 / 4000 [ 12%] (Warmup)
#> Chain 1 Iteration: 600 / 4000 [ 15%] (Warmup)
#> Chain 1 Iteration: 700 / 4000 [ 17%] (Warmup)
#> Chain 1 Iteration: 800 / 4000 [ 20%] (Warmup)
#> Chain 1 Iteration: 900 / 4000 [ 22%] (Warmup)
#> Chain 1 Iteration: 1000 / 4000 [ 25%] (Warmup)
#> Chain 1 Iteration: 1100 / 4000 [ 27%] (Warmup)
#> Chain 1 Iteration: 1200 / 4000 [ 30%] (Warmup)
#> Chain 1 Iteration: 1300 / 4000 [ 32%] (Warmup)
#> Chain 1 Iteration: 1400 / 4000 [ 35%] (Warmup)
#> Chain 1 Iteration: 1500 / 4000 [ 37%] (Warmup)
#> Chain 1 Iteration: 1600 / 4000 [ 40%] (Warmup)
#> Chain 1 Iteration: 1700 / 4000 [ 42%] (Warmup)
#> Chain 1 Iteration: 1800 / 4000 [ 45%] (Warmup)
#> Chain 1 Iteration: 1900 / 4000 [ 47%] (Warmup)
#> Chain 1 Iteration: 2000 / 4000 [ 50%] (Warmup)
#> Chain 1 Iteration: 2001 / 4000 [ 50%] (Sampling)
#> Chain 1 Iteration: 2100 / 4000 [ 52%] (Sampling)
#> Chain 1 Iteration: 2200 / 4000 [ 55%] (Sampling)
#> Chain 1 Iteration: 2300 / 4000 [ 57%] (Sampling)
#> Chain 1 Iteration: 2400 / 4000 [ 60%] (Sampling)
#> Chain 1 Iteration: 2500 / 4000 [ 62%] (Sampling)
#> Chain 1 Iteration: 2600 / 4000 [ 65%] (Sampling)
#> Chain 1 Iteration: 2700 / 4000 [ 67%] (Sampling)
#> Chain 1 Iteration: 2800 / 4000 [ 70%] (Sampling)
#> Chain 1 Iteration: 2900 / 4000 [ 72%] (Sampling)
#> Chain 1 Iteration: 3000 / 4000 [ 75%] (Sampling)
#> Chain 1 Iteration: 3100 / 4000 [ 77%] (Sampling)
#> Chain 1 Iteration: 3200 / 4000 [ 80%] (Sampling)
#> Chain 1 Iteration: 3300 / 4000 [ 82%] (Sampling)
#> Chain 1 Iteration: 3400 / 4000 [ 85%] (Sampling)
#> Chain 1 Iteration: 3500 / 4000 [ 87%] (Sampling)
#> Chain 1 Iteration: 3600 / 4000 [ 90%] (Sampling)
#> Chain 1 Iteration: 3700 / 4000 [ 92%] (Sampling)
#> Chain 1 Iteration: 3800 / 4000 [ 95%] (Sampling)
#> Chain 1 Iteration: 3900 / 4000 [ 97%] (Sampling)
#> Chain 1 Iteration: 4000 / 4000 [100%] (Sampling)
#> Chain 1 finished in 17.0 seconds.
#> Chain 2 Iteration: 1 / 4000 [ 0%] (Warmup)
#> Chain 2 Iteration: 100 / 4000 [ 2%] (Warmup)
#> Chain 2 Iteration: 200 / 4000 [ 5%] (Warmup)
#> Chain 2 Iteration: 300 / 4000 [ 7%] (Warmup)
#> Chain 2 Iteration: 400 / 4000 [ 10%] (Warmup)
#> Chain 2 Iteration: 500 / 4000 [ 12%] (Warmup)
#> Chain 2 Iteration: 600 / 4000 [ 15%] (Warmup)
#> Chain 2 Iteration: 700 / 4000 [ 17%] (Warmup)
#> Chain 2 Iteration: 800 / 4000 [ 20%] (Warmup)
#> Chain 2 Iteration: 900 / 4000 [ 22%] (Warmup)
#> Chain 2 Iteration: 1000 / 4000 [ 25%] (Warmup)
#> Chain 2 Iteration: 1100 / 4000 [ 27%] (Warmup)
#> Chain 2 Iteration: 1200 / 4000 [ 30%] (Warmup)
#> Chain 2 Iteration: 1300 / 4000 [ 32%] (Warmup)
#> Chain 2 Iteration: 1400 / 4000 [ 35%] (Warmup)
#> Chain 2 Iteration: 1500 / 4000 [ 37%] (Warmup)
#> Chain 2 Iteration: 1600 / 4000 [ 40%] (Warmup)
#> Chain 2 Iteration: 1700 / 4000 [ 42%] (Warmup)
#> Chain 2 Iteration: 1800 / 4000 [ 45%] (Warmup)
#> Chain 2 Iteration: 1900 / 4000 [ 47%] (Warmup)
#> Chain 2 Iteration: 2000 / 4000 [ 50%] (Warmup)
#> Chain 2 Iteration: 2001 / 4000 [ 50%] (Sampling)
#> Chain 2 Iteration: 2100 / 4000 [ 52%] (Sampling)
#> Chain 2 Iteration: 2200 / 4000 [ 55%] (Sampling)
#> Chain 2 Iteration: 2300 / 4000 [ 57%] (Sampling)
#> Chain 2 Iteration: 2400 / 4000 [ 60%] (Sampling)
#> Chain 2 Iteration: 2500 / 4000 [ 62%] (Sampling)
#> Chain 2 Iteration: 2600 / 4000 [ 65%] (Sampling)
#> Chain 2 Iteration: 2700 / 4000 [ 67%] (Sampling)
#> Chain 2 Iteration: 2800 / 4000 [ 70%] (Sampling)
#> Chain 2 Iteration: 2900 / 4000 [ 72%] (Sampling)
#> Chain 2 Iteration: 3000 / 4000 [ 75%] (Sampling)
#> Chain 2 Iteration: 3100 / 4000 [ 77%] (Sampling)
#> Chain 2 Iteration: 3200 / 4000 [ 80%] (Sampling)
#> Chain 2 Iteration: 3300 / 4000 [ 82%] (Sampling)
#> Chain 2 Iteration: 3400 / 4000 [ 85%] (Sampling)
#> Chain 2 Iteration: 3500 / 4000 [ 87%] (Sampling)
#> Chain 2 Iteration: 3600 / 4000 [ 90%] (Sampling)
#> Chain 2 Iteration: 3700 / 4000 [ 92%] (Sampling)
#> Chain 2 Iteration: 3800 / 4000 [ 95%] (Sampling)
#> Chain 2 Iteration: 3900 / 4000 [ 97%] (Sampling)
#> Chain 2 Iteration: 4000 / 4000 [100%] (Sampling)
#> Chain 2 finished in 18.5 seconds.
#> Chain 3 Iteration: 1 / 4000 [ 0%] (Warmup)
#> Chain 3 Iteration: 100 / 4000 [ 2%] (Warmup)
#> Chain 3 Iteration: 200 / 4000 [ 5%] (Warmup)
#> Chain 3 Iteration: 300 / 4000 [ 7%] (Warmup)
#> Chain 3 Iteration: 400 / 4000 [ 10%] (Warmup)
#> Chain 3 Iteration: 500 / 4000 [ 12%] (Warmup)
#> Chain 3 Iteration: 600 / 4000 [ 15%] (Warmup)
#> Chain 3 Iteration: 700 / 4000 [ 17%] (Warmup)
#> Chain 3 Iteration: 800 / 4000 [ 20%] (Warmup)
#> Chain 3 Iteration: 900 / 4000 [ 22%] (Warmup)
#> Chain 3 Iteration: 1000 / 4000 [ 25%] (Warmup)
#> Chain 3 Iteration: 1100 / 4000 [ 27%] (Warmup)
#> Chain 3 Iteration: 1200 / 4000 [ 30%] (Warmup)
#> Chain 3 Iteration: 1300 / 4000 [ 32%] (Warmup)
#> Chain 3 Iteration: 1400 / 4000 [ 35%] (Warmup)
#> Chain 3 Iteration: 1500 / 4000 [ 37%] (Warmup)
#> Chain 3 Iteration: 1600 / 4000 [ 40%] (Warmup)
#> Chain 3 Iteration: 1700 / 4000 [ 42%] (Warmup)
#> Chain 3 Iteration: 1800 / 4000 [ 45%] (Warmup)
#> Chain 3 Iteration: 1900 / 4000 [ 47%] (Warmup)
#> Chain 3 Iteration: 2000 / 4000 [ 50%] (Warmup)
#> Chain 3 Iteration: 2001 / 4000 [ 50%] (Sampling)
#> Chain 3 Iteration: 2100 / 4000 [ 52%] (Sampling)
#> Chain 3 Iteration: 2200 / 4000 [ 55%] (Sampling)
#> Chain 3 Iteration: 2300 / 4000 [ 57%] (Sampling)
#> Chain 3 Iteration: 2400 / 4000 [ 60%] (Sampling)
#> Chain 3 Iteration: 2500 / 4000 [ 62%] (Sampling)
#> Chain 3 Iteration: 2600 / 4000 [ 65%] (Sampling)
#> Chain 3 Iteration: 2700 / 4000 [ 67%] (Sampling)
#> Chain 3 Iteration: 2800 / 4000 [ 70%] (Sampling)
#> Chain 3 Iteration: 2900 / 4000 [ 72%] (Sampling)
#> Chain 3 Iteration: 3000 / 4000 [ 75%] (Sampling)
#> Chain 3 Iteration: 3100 / 4000 [ 77%] (Sampling)
#> Chain 3 Iteration: 3200 / 4000 [ 80%] (Sampling)
#> Chain 3 Iteration: 3300 / 4000 [ 82%] (Sampling)
#> Chain 3 Iteration: 3400 / 4000 [ 85%] (Sampling)
#> Chain 3 Iteration: 3500 / 4000 [ 87%] (Sampling)
#> Chain 3 Iteration: 3600 / 4000 [ 90%] (Sampling)
#> Chain 3 Iteration: 3700 / 4000 [ 92%] (Sampling)
#> Chain 3 Iteration: 3800 / 4000 [ 95%] (Sampling)
#> Chain 3 Iteration: 3900 / 4000 [ 97%] (Sampling)
#> Chain 3 Iteration: 4000 / 4000 [100%] (Sampling)
#> Chain 3 finished in 17.8 seconds.
#> Chain 4 Iteration: 1 / 4000 [ 0%] (Warmup)
#> Chain 4 Iteration: 100 / 4000 [ 2%] (Warmup)
#> Chain 4 Iteration: 200 / 4000 [ 5%] (Warmup)
#> Chain 4 Iteration: 300 / 4000 [ 7%] (Warmup)
#> Chain 4 Iteration: 400 / 4000 [ 10%] (Warmup)
#> Chain 4 Iteration: 500 / 4000 [ 12%] (Warmup)
#> Chain 4 Iteration: 600 / 4000 [ 15%] (Warmup)
#> Chain 4 Iteration: 700 / 4000 [ 17%] (Warmup)
#> Chain 4 Iteration: 800 / 4000 [ 20%] (Warmup)
#> Chain 4 Iteration: 900 / 4000 [ 22%] (Warmup)
#> Chain 4 Iteration: 1000 / 4000 [ 25%] (Warmup)
#> Chain 4 Iteration: 1100 / 4000 [ 27%] (Warmup)
#> Chain 4 Iteration: 1200 / 4000 [ 30%] (Warmup)
#> Chain 4 Iteration: 1300 / 4000 [ 32%] (Warmup)
#> Chain 4 Iteration: 1400 / 4000 [ 35%] (Warmup)
#> Chain 4 Iteration: 1500 / 4000 [ 37%] (Warmup)
#> Chain 4 Iteration: 1600 / 4000 [ 40%] (Warmup)
#> Chain 4 Iteration: 1700 / 4000 [ 42%] (Warmup)
#> Chain 4 Iteration: 1800 / 4000 [ 45%] (Warmup)
#> Chain 4 Iteration: 1900 / 4000 [ 47%] (Warmup)
#> Chain 4 Iteration: 2000 / 4000 [ 50%] (Warmup)
#> Chain 4 Iteration: 2001 / 4000 [ 50%] (Sampling)
#> Chain 4 Iteration: 2100 / 4000 [ 52%] (Sampling)
#> Chain 4 Iteration: 2200 / 4000 [ 55%] (Sampling)
#> Chain 4 Iteration: 2300 / 4000 [ 57%] (Sampling)
#> Chain 4 Iteration: 2400 / 4000 [ 60%] (Sampling)
#> Chain 4 Iteration: 2500 / 4000 [ 62%] (Sampling)
#> Chain 4 Iteration: 2600 / 4000 [ 65%] (Sampling)
#> Chain 4 Iteration: 2700 / 4000 [ 67%] (Sampling)
#> Chain 4 Iteration: 2800 / 4000 [ 70%] (Sampling)
#> Chain 4 Iteration: 2900 / 4000 [ 72%] (Sampling)
#> Chain 4 Iteration: 3000 / 4000 [ 75%] (Sampling)
#> Chain 4 Iteration: 3100 / 4000 [ 77%] (Sampling)
#> Chain 4 Iteration: 3200 / 4000 [ 80%] (Sampling)
#> Chain 4 Iteration: 3300 / 4000 [ 82%] (Sampling)
#> Chain 4 Iteration: 3400 / 4000 [ 85%] (Sampling)
#> Chain 4 Iteration: 3500 / 4000 [ 87%] (Sampling)
#> Chain 4 Iteration: 3600 / 4000 [ 90%] (Sampling)
#> Chain 4 Iteration: 3700 / 4000 [ 92%] (Sampling)
#> Chain 4 Iteration: 3800 / 4000 [ 95%] (Sampling)
#> Chain 4 Iteration: 3900 / 4000 [ 97%] (Sampling)
#> Chain 4 Iteration: 4000 / 4000 [100%] (Sampling)
#> Chain 4 finished in 17.6 seconds.
#>
#> All 4 chains finished successfully.
#> Mean chain execution time: 17.7 seconds.
#> Total execution time: 71.2 seconds.
#> 1000 vector or matrix parameters hidden. Use depth=2 to show them.
#> mean sd 5.5% 94.5% n_eff Rhat4
#> alpha 0.04905132 0.001360069 0.04697336 0.05128713 7247.988 0.9998833
Result:
d1[d1$y_new - d1$ucl_partpool > 0,
c("n", "y", "n_new", "y_new",
"true_theta", "theta_partpool",
"ucl_true_theta", "ucl_partpool")]
#> n y n_new y_new true_theta theta_partpool ucl_true_theta ucl_partpool
#> 109 24 0 12 3 NA 0.024302886 NA 2.5
#> 228 54 1 27 4 NA 0.027967910 NA 3.5
#> 364 95 5 48 8 NA 0.050417683 NA 7.5
#> 374 100 2 50 6 NA 0.026326791 NA 5.5
#> 399 110 1 55 5 NA 0.016491387 NA 4.5
#> 462 136 2 68 8 NA 0.020516222 NA 5.5
#> 501 157 9 79 12 NA 0.054769164 NA 11.5
#> 529 171 5 86 9 NA 0.032173221 NA 8.5
#> 598 210 1 105 5 NA 0.009289763 NA 4.5
#> 613 219 2 110 6 NA 0.013208173 NA 5.5
#> 635 239 12 120 14 NA 0.049739447 NA 13.5
#> 698 289 9 145 16 NA 0.032966185 NA 11.5
#> 751 336 3 168 7 NA 0.011626066 NA 6.5
#> 805 410 39 205 33 NA 0.089261026 NA 31.5
#> 894 612 26 306 25 NA 0.042676439 NA 24.5
#> 935 802 54 401 51 NA 0.066278225 NA 42.5
#> 967 1040 26 520 25 NA 0.025680542 NA 24.5
#> 974 1124 73 562 55 NA 0.064351810 NA 54.5
In a realistic scenario it is not possible to assess performance by comparison with “true” values. The true values of theta are generally not known, and the probability distributions assumed for the Bayesian hierarchical model (the no-pooling model) may in fact be inappropriate.
On the other hand, in a realistic scenario it is often possible to assess performance by investigating individual cases.
Select data from data frame d (example above) for the report:
countcheck_df <- select_for_report(d)
Result:
str(countcheck_df)
#> 'data.frame': 18 obs. of 3 variables:
#> $ y_new : int 8 16 51 3 4 5 5 6 6 7 ...
#> $ ucl_partpool: num 5.5 11.5 42.5 2.5 3.5 4.5 4.5 5.5 5.5 6.5 ...
#> $ unit : int 462 698 935 109 228 399 598 374 613 751 ...
We also need a data frame with unit master data. Here, we just generate some data:
units_tmp <- sort(unique(countcheck_df$unit))
unit_df <- data.frame(
unit = units_tmp,
unit_name = paste("Unit",
units_tmp),
unit_url = paste0("http://domain/units/",
units_tmp,
".html"),
unit_group_name = paste("Group",
rep(1:5,
length.out = length(units_tmp)))
)
Result:
str(unit_df)
#> 'data.frame': 18 obs. of 4 variables:
#> $ unit : int 109 228 364 374 399 462 501 529 598 613 ...
#> $ unit_name : chr "Unit 109" "Unit 228" "Unit 364" "Unit 374" ...
#> $ unit_url : chr "http://domain/units/109.html" "http://domain/units/228.html" "http://domain/units/364.html" "http://domain/units/374.html" ...
#> $ unit_group_name: chr "Group 1" "Group 2" "Group 3" "Group 4" ...
Generate report as R string:
report <- html_report(
countcheck_list = list(
list(df = countcheck_df, caption = "KPI 1")
),
unit_df = unit_df,
title = "Report",
table_width_px = 500,
column_headers = c(
group = "Group",
count = "Count",
ucl = "UCL",
unit = "Unit",
name = "Name"),
charset = "utf-8",
lang = "en",
home_url = "https://github.com/jmeydam/countcheck"
)
Save HTML report to disk:
> sink("report.html")
> cat(report)
> sink()


Ethan White
is an Associate Professor in the Department of Biology and the
Ecology Center at Utah State University. He is
a Moore
Investigator in Data-Driven Discovery and a recipient of the
National Science Foundation CAREER "Young Investigators" Award. He
is a proponent of open and reproducible science and serves on
the Data Carpentry
steering committee,
the Impactstory board of
directors, and the editorial boards
of PLOS ONE
and PeerJ.
Advisory Council


“Far far away, behind the word mountains, far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove right at the coast of the Semantics, a large language ocean.”
“Far from the countries Vokalia and Consonantia, there live the blind texts. Separated they live in Bookmarksgrove right at the coast of the Semantics, a large language ocean.”
We Want Funds to Support More of Our Projects
There are so many Vanurable girls out there that need your support. Your Donations can help someone.


Preparation time
less than 30 mins
Cooking time
over 2 hours
Serves
Serves 6-8
A delicately strawberry and rose flavoured dessert, designed to impress. Perfect for a special occasion.
For this recipe you will need two 20cm/8in cake tins.
For the jelly, put the strawberries in a large metal bowl with the sugar and 100ml/3½fl oz water. Cover with foil to seal well, then place over a pan of simmering water for 40 minutes. Strain off the liquid into a large jug or bowl and reserve the solids.
Soak the gelatine in cold water for 5 minutes to soften. Squeeze excess water from gelatine, then add to the liquid strawberry mixture, stirring to dissolve. Add the rosewater and stir to combine. Pour into a tray. Cool, then chill in the fridge for 4 hours or until set. Cut half of the jelly into cubes for garnish and then set aside the other half.
For the crème pâtissière, using a balloon whisk, whisk the egg yolks, sugar, flour and cornflour together in a large bowl. Combine the milk and vanilla in a saucepan and heat over medium-low heat until almost boiling, then, whisking constantly, add to the egg mixture. Pour into a clean pan, place over a low heat and cook, stirring, until mixture is very thick. Strain through a sieve into a bowl and cover the surface closely with cling film to prevent a skin forming. Cool for 10 minutes, then chill until ready to use.
For the sponge, preheat the oven to 180C/160C Fan/Gas 4. Lightly grease two 20cm/8in cake tins and line with baking parchment.
In a large mixing bowl, beat the butter and sugar together until pale and fluffy. Gradually add the eggs a little at a time, scraping down the side of the bowl after each addition. Add the orange extract, mix in half the flour, then fold in the rest.
Divide the mixture between the prepared tins and bake for 15-20 minutes until cooked through. Remove and cool on a wire rack.
For the rose cream, whisk the cream, adding the rosewater as you go, until soft peaks form when the whisk is removed. Fold in the jam, being careful not to over-mix.
For the garnish, reduce the oven to 110C/90C Fan/Gas ¼. Line a baking tray with parchment.
In a large clean bowl, whisk the egg whites until soft peaks form when the whisk is removed. Whisk in the sugar a spoon at a time and continue to whisk into you have a glossy mixture. Spread on the baking tray in a circle the right size to sit on top of the finished trifle. Bake for 15 minutes.
To serve, spread one of the sponges with the reserved strawberry juice pulp, and a little of the crème pâtissière. Sandwich the sponges together and cut into cubes.
Place half the cubed sponge in a large serving bowl. Sprinkle with the halved strawberries, reserve a few for garnish. Melt half the remaining jelly with a little hot water. Pour over the sponge and leave to cool and set in the fridge for 2 hours.
Add a small layer of crème pâtissière, some of the rose cream and the remaining the cubed sponge. Melt the rest of the jelly, pour it over, and leave to set again.
Smooth the top with the rose cream. Top with the meringue and garnish with strawberry jelly cubes, strawberries and rose petals.


Boilerplate is any text that is or can be reused in new contexts or applications without being greatly changed from the original.
Boiler plate
originally referred to the maker's label used to identify the builder of steam boilers.(1)
In the field of printing, the term dates back to the early 1900s. From the 1890s onwards, printing plates of text for widespread reproduction such as advertisements or syndicated columns were cast or stamped in steel (instead of the much softer and less durable lead alloys used otherwise) ready for the printing press and distributed to newspapers around the United States. By analogy, they came to be known as 'boilerplates'. One large supplier to newspapers of this kind of boilerplate was the Western Newspaper Union, which supplied ready to print stories which contained national or international news
to papers with smaller geographic footprints, which could include advertisements preprinted next to the conventional content.(2)
Some companies also sent out press releases as boilerplate so that they had to be printed as written. The modern equivalent is the press release boilerplate, or boiler
, a paragraph or two that describes the company and its products.
In contractual law, the term boilerplate language
describes the parts of a contract that are considered standard.
Boilerplate language may also exist in precreated form letters on the Internet for such things as issues to be broached by a politician based on an issue ad, requests that a cable or satellite operator add a cable network to a system, prewritten complaints about a product that is opposed by groups that create such letters, or online petitions. The person requesting the action usually only needs to type or sign his or her name at the end of the prewritten greeting and body.
In computer programming, boilerplate is the sections of code that have to be included in many places with little or no alteration. Such boilerplate is particularly salient when the programmer must include a lot of code for minimal functionality. The need for boilerplate can be reduced through highlevel mechanisms.
A related phenomenon, bookkeeping code, is code that is not part of the business logic but is interleaved with it in order to keep data structures updated or able to handle secondary aspects of the program.
Boiler plate, Oxford English Dictionary,2nd Edition (online).(subscription required)
Building that was home to boilerplate to become luxury condos. Retrieved


Looking for your containers safety? contact us.
Plot no. 119/120,
SUSREZ Diamond Park, Sachin, Surat-394230
GALA NO. 35, Bombay Timber Market,
Signal Hill Avenue, Darukhana, Mumbai-4000010
(Telephone)
+91-22-23746946
+91-22-23713829
(A.O.H)
+91-22-24112434
+91-22-24110058
(24x7)
+91-9833134013
(Operational)
(Technical)
(Stepney)


| |
|
The latest edition of this best-selling favorite is
loaded with vital information on Linux, Darwin, and BSD.
Unix Power Tools 3rd Edition now offers
more coverage of bash, zsh, and other new shells, along with discussions
about modern utilities and applications. Several sections focus on security
and Internet access.  Introduction |
| |
|
|
Section 1.1. What's Special About Unix? |
| |
|
|
Section 1.2. Power Grows on You |
| |
|
|
Section 1.3. The Core of Unix |
| |
|
|
Section 1.4. Communication with Unix |
| |
|
|
Section 1.5. Programs Are Designed to Work Together |
| |
|
|
Section 1.6. There Are Many Shells |
| |
|
|
Section 1.7. Which Shell Am I Running? |
| |
|
|
Section 1.8. Anyone Can Program the Shell |
| |
|
|
Section 1.9. Internal and External Commands |
| |
|
|
Section 1.10. The Kernel and Daemons |
| |
|
|
Section 1.11. Filenames |
| |
|
|
Section 1.12. Filename Extensions |
| |
|
|
Section 1.13. Wildcards |
| |
|
|
Section 1.14. The Tree Structure of the Filesystem |
| |
|
|
Section 1.15. Your Home Directory |
| |
|
|
Section 1.16. Making Pathnames |
| |
|
|
Section 1.17. File Access Permissions |
| |
|
|
Section 1.18. The Superuser (Root) |
| |
|
|
Section 1.19. When Is a File Not a File? |
| |
|
|
Section 1.20. Scripting |
| |
|
|
Section 1.21. Unix Networking and Communications |
| |
|
|
Section 1.22. The X Window System |
| |
| |
|
|
Chapter 2. Getting Help |
| |
|
|
Section 2.1.  Searching Online Manual Pages |
| |
|
|
Section 2.5. How Unix Systems Remember Their Names |
| |
|
|
Section 2.6. Which Version Am I Using? |
| |
|
|
Section 2.7. What tty Am I On? |
| |
|
|
Section 2.8. Who's On? |
| |
|
|
Section 2.9.   What Happens When You Log In |
| |
|
|
Section 3.2. The Mac OS X Terminal Application |
| |
|
|
Section 3.3. Shell Setup Files Which, Where, and Why |
| |
|
|
Section 3.4. Login Shells, Interactive Shells |
| |
|
|
Section 3.5. What Goes in Shell Setup Files? |
| |
|
|
Section 3.6. Tip for Changing Account Setup: Keep a Shell Ready |
| |
|
|
Section 3.7. Use Absolute Pathnames in Shell Setup Files |
| |
|
|
Section 3.8. Setup Files Aren't Read When You Want? |
| |
|
|
Section 3.9. Gotchas in set prompt Test |
| |
|
|
Section 3.10. Automatic Setups for Different Terminals |
| |
|
|
Section 3.11. Terminal Setup: Testing TERM |
| |
|
|
Section 3.12. Terminal Setup: Testing Remote Hostname and X Display |
| |
|
|
Section 3.13. Terminal Setup: Testing Port |
| |
|
|
Section 3.14. Terminal Setup: Testing Environment Variables |
| |
|
|
Section 3.15. Terminal Setup: Searching Terminal Table |
| |
|
|
Section 3.16. Terminal Setup: Testing Window Size |
| |
|
|
Section 3.17. Terminal Setup: Setting and Testing Window Name |
| |
|
|
Section 3.18. A .cshrc.$HOST File for Per Host Setup |
| |
|
|
Section 3.19. Making a "Login" Shell |
| |
|
|
Section 3.20. RC Files |
| |
|
|
Section 3.21.  Writing a Simple Manpage with the -man Macros |
| |
| |
|
|
Chapter 4. Interacting with Your Environment |
| |
|
|
Section 4.1. Basics of Setting the Prompt |
| |
|
|
Section 4.2. Static Prompts |
| |
|
|
Section 4.3. Dynamic Prompts |
| |
|
|
Section 4.4. Simulating Dynamic Prompts |
| |
|
|
Section 4.5. C-Shell Prompt Causes Problems in vi, rsh, etc. |
| |
|
|
Section 4.6. Faster Prompt Setting with Built-ins |
| |
|
|
Section 4.7. Multiline Shell Prompts |
| |
|
|
Section 4.8. Session Info in Window Title or Status Line |
| |
|
|
Section 4.9. A "Menu Prompt" for Naive Users |
| |
|
|
Section 4.10. Highlighting and Color in Shell Prompts |
| |
|
|
Section 4.11. Right-Side Prompts |
| |
|
|
Section 4.12. Show Subshell Level with $SHLVL |
| |
|
|
Section 4.13. What Good Is a Blank Shell Prompt? |
| |
|
|
Section 4.14. dirs in Your Prompt: Better Than $cwd |
| |
|
|
Section 4.15. External Commands Send Signals to Set Variables |
| |
|
|
Section 4.16. Preprompt, Pre-execution, and Periodic Commands |
| |
|
|
Section 4.17. Running Commands When You Log Out |
| |
|
|
Section 4.18. Running Commands at Bourne/Korn Shell Logout |
| |
|
|
Section 4.19.  Getting the Most out of Terminals, xterm, and X Windows |
| |
|
|
Section 5.1. There's a Lot to Know About Terminals |
| |
|
|
Section 5.2. The Idea of a Terminal Database |
| |
|
|
Section 5.3. Setting the Terminal Type When You Log In |
| |
|
|
Section 5.4. Querying Your Terminal Type: qterm |
| |
|
|
Section 5.5.  Checklist: Terminal Hangs When I Log In |
| |
|
|
Section 5.7. Find Out Terminal Settings with stty |
| |
|
|
Section 5.8.  Working with xterm and Friends |
| |
|
|
Section 5.10. Login xterms and rxvts |
| |
|
|
Section 5.11. Working with Scrollbars |
| |
|
|
Section 5.12. How Many Lines to Save? |
| |
|
|
Section 5.13. Simple Copy and Paste in xterm |
| |
|
|
Section 5.14. Defining What Makes Up a Word for Selection Purposes |
| |
|
|
Section 5.15. Setting the Titlebar and Icon Text |
| |
|
|
Section 5.16. The Simple Way to Pick a Font |
| |
|
|
Section 5.17. The xterm Menus |
| |
|
|
Section 5.18. Changing Fonts Dynamically |
| |
|
|
Section 5.19. Working with xclipboard |
| |
|
|
Section 5.20. Problems with Large Selections |
| |
|
|
Section 5.21. Tips for Copy and Paste Between Windows |
| |
|
|
Section 5.22. Running a Single Command with xterm -e |
| |
|
|
Section 5.23.   Defining Keys and Button Presses with xmodmap |
| |
|
|
Section 6.2. Using xev to Learn Keysym Mappings |
| |
|
|
Section 6.3. X Resource Syntax |
| |
|
|
Section 6.4. X Event Translations |
| |
|
|
Section 6.5. Setting X Resources: Overview |
| |
|
|
Section 6.6. Setting Resources with the -xrm Option |
| |
|
|
Section 6.7. How -name Affects Resources |
| |
|
|
Section 6.8. Setting Resources with xrdb |
| |
|
|
Section 6.9. Listing the Current Resources for a Client: appres |
| |
|
|
Section 6.10.  Directory Organization |
| |
|
|
Section 7.1. What? Me, Organized? |
| |
|
|
Section 7.2. Many Homes |
| |
|
|
Section 7.3. Access to Directories |
| |
|
|
Section 7.4.  Private (Personal) Directories |
| |
|
|
Section 7.6. Naming Files |
| |
|
|
Section 7.7. Make More Directories! |
| |
|
|
Section 7.8. Making Directories Made Easier |
| |
| |
|
|
Chapter 8. Directories and Files |
| |
|
|
Section 8.1. Everything but the find Command |
| |
|
|
Section 8.2. The Three Unix File Times |
| |
|
|
Section 8.3. Finding Oldest or Newest Files with ls -t and ls -u |
| |
|
|
Section 8.4. List All Subdirectories with ls -R |
| |
|
|
Section 8.5. The ls -d Option |
| |
|
|
Section 8.6. Color ls |
| |
|
|
Section 8.7. Some GNU ls Features |
| |
|
|
Section 8.8. A csh Alias to List Recently Changed Files |
| |
|
|
Section 8.9. Showing Hidden Files with ls -A and -a |
| |
|
|
Section 8.10. Useful ls Aliases |
| |
|
|
Section 8.11. Can't Access a File? Look for Spaces in the Name |
| |
|
|
Section 8.12. Showing Nonprintable Characters in Filenames |
| |
|
|
Section 8.13. Counting Files by Types |
| |
|
|
Section 8.14. Listing Files by Age and Size |
| |
|
|
Section 8.15. newer: Print the Name of the Newest File |
| |
|
|
Section 8.16. oldlinks: Find Unconnected Symbolic Links |
| |
|
|
Section 8.17. Picking a Unique Filename Automatically |
| |
| |
|
|
Chapter 9. Finding Files with find |
| |
|
|
Section 9.1. How to Use find |
| |
|
|
Section 9.2. Delving Through a Deep Directory Tree |
| |
|
|
Section 9.3. Don't Forget -print |
| |
|
|
Section 9.4. Looking for Files with Particular Names |
| |
|
|
Section 9.5. Searching for Old Files |
| |
|
|
Section 9.6. Be an Expert on find Search Operators |
| |
|
|
Section 9.7. The Times That find Finds |
| |
|
|
Section 9.8. Exact File-Time Comparisons |
| |
|
|
Section 9.9. Running Commands on What You Find |
| |
|
|
Section 9.10. Using -exec to Create Custom Tests |
| |
|
|
Section 9.11. Custom -exec Tests Applied |
| |
|
|
Section 9.12. Finding Many Things with One Command |
| |
|
|
Section 9.13. Searching for Files by Type |
| |
|
|
Section 9.14. Searching for Files by Size |
| |
|
|
Section 9.15. Searching for Files by Permission |
| |
|
|
Section 9.16. Searching by Owner and Group |
| |
|
|
Section 9.17. Duplicating a Directory Tree |
| |
|
|
Section 9.18. Using "Fast find" Databases |
| |
|
|
Section 9.19. Wildcards with "Fast find" Database |
| |
|
|
Section 9.20. Finding Files (Much) Faster with a find Database |
| |
|
|
Section 9.21. grepping a Directory Tree |
| |
|
|
Section 9.22. lookfor: Which File Has That Word? |
| |
|
|
Section 9.23. Using Shell Arrays to Browse Directories |
| |
|
|
Section 9.24. Finding the (Hard) Links to a File |
| |
|
|
Section 9.25. Finding Files with -prune |
| |
|
|
Section 9.26. Quick finds in the Current Directory |
| |
|
|
Section 9.27. Skipping Parts of a Tree in find |
| |
|
|
Section 9.28. Keeping find from Searching Networked Filesystem |
| |
| |
|
|
Chapter 10. Linking, Renaming, and Copying Files |
| |
|
|
Section 10.1. What's So Complicated About Copying Files |
| |
|
|
Section 10.2. What's Really in a Directory? |
| |
|
|
Section 10.3. Files with Two or More Names |
| |
|
|
Section 10.4. More About Links |
| |
|
|
Section 10.5. Creating and Removing Links |
| |
|
|
Section 10.6. Stale Symbolic Links |
| |
|
|
Section 10.7. Linking Directories |
| |
|
|
Section 10.8. Showing the Actual Filenames for Symbolic Links |
| |
|
|
Section 10.9. Renaming, Copying, or Comparing a Set of Files |
| |
|
|
Section 10.10. Renaming a List of Files Interactively |
| |
|
|
Section 10.11.  Copying Directory Trees with cp -r |
| |
|
|
Section 10.13. Copying Directory Trees with tar and Pipes |
| |
| |
|
|
Chapter 11. Comparing Files |
| |
|
|
Section 11.1. Checking Differences with diff |
| |
|
|
Section 11.2. Comparing Three Different Versions with diff3 |
| |
|
|
Section 11.3. Context diffs |
| |
|
|
Section 11.4. Side-by-Side diffs: sdiff |
| |
|
|
Section 11.5. Choosing Sides with sdiff |
| |
|
|
Section 11.6. Problems with diff and Tabstops |
| |
|
|
Section 11.7. cmp and diff |
| |
|
|
Section 11.8. Comparing Two Files with comm |
| |
|
|
Section 11.9. More Friendly comm Output |
| |
|
|
Section 11.10. make Isn't Just for Programmers! |
| |
|
|
Section 11.11.  Showing What's in a File |
| |
|
|
Section 12.1. Cracking the Nut |
| |
|
|
Section 12.2. What Good Is a cat? |
| |
|
|
Section 12.3. "less" is More |
| |
|
|
Section 12.4. Show Nonprinting Characters with cat -v or od -c |
| |
|
|
Section 12.5. What's in That Whitespace? |
| |
|
|
Section 12.6. Finding File Types |
| |
|
|
Section 12.7. Squash Extra Blank Lines |
| |
|
|
Section 12.8. How to Look at the End of a File: tail |
| |
|
|
Section 12.9. Finer Control on tail |
| |
|
|
Section 12.10. How to Look at Files as They Grow |
| |
|
|
Section 12.11. GNU tail File Following |
| |
|
|
Section 12.12. Printing the Top of a File |
| |
|
|
Section 12.13. Numbering Lines |
| |
| |
|
|
Chapter 13. Searching Through Files |
| |
|
|
Section 13.1. Different Versions of grep |
| |
|
|
Section 13.2. Searching for Text with grep |
| |
|
|
Section 13.3. Finding Text That Doesn't Match |
| |
|
|
Section 13.4. Extended Searching for Text with egrep |
| |
|
|
Section 13.5. grepping for a List of Patterns |
| |
|
|
Section 13.6. Approximate grep: agrep |
| |
|
|
Section 13.7. Search RCS Files with rcsgrep |
| |
|
|
Section 13.8. GNU Context greps |
| |
|
|
Section 13.9. A Multiline Context grep Using sed |
| |
|
|
Section 13.10. Compound Searches |
| |
|
|
Section 13.11. Narrowing a Search Quickly |
| |
|
|
Section 13.12. Faking Case-Insensitive Searches |
| |
|
|
Section 13.13. Finding a Character in a Column |
| |
|
|
Section 13.14. Fast Searches and Spelling Checks with "look" |
| |
|
|
Section 13.15. Finding Words Inside Binary Files |
| |
|
|
Section 13.16. A Highlighting grep |
| |
| |
|
|
Chapter 14. Removing Files |
| |
|
|
Section 14.1. The Cycle of Creation and Destruction |
| |
|
|
Section 14.2. How Unix Keeps Track of Files: Inodes |
| |
|
|
Section 14.3. rm and Its Dangers |
| |
|
|
Section 14.4. Tricks for Making rm Safer |
| |
|
|
Section 14.5. Answer "Yes" or "No" Forever with yes |
| |
|
|
Section 14.6. Remove Some, Leave Some |
| |
|
|
Section 14.7.  Safer File Deletion in Some Directories |
| |
|
|
Section 14.9. Safe Delete: Pros and Cons |
| |
|
|
Section 14.10. Deletion with Prejudice: rm -f |
| |
|
|
Section 14.11. Deleting Files with Odd Names |
| |
|
|
Section 14.12. Using Wildcards to Delete Files with Strange Names |
| |
|
|
Section 14.13.  Using unlink to Remove a File with a Strange Name |
| |
|
|
Section 14.15. Removing a Strange File by its i-number |
| |
|
|
Section 14.16. Problems Deleting Directories |
| |
|
|
Section 14.17. Deleting Stale Files |
| |
|
|
Section 14.18. Removing Every File but One |
| |
|
|
Section 14.19. Using find to Clear Out Unneeded Files |
| |
| |
|
|
Chapter 15. Optimizing Disk Space |
| |
|
|
Section 15.1. Disk Space Is Cheap |
| |
|
|
Section 15.2. Instead of Removing a File, Empty It |
| |
|
|
Section 15.3. Save Space with "Bit Bucket" Log Files and Mailboxes |
| |
|
|
Section 15.4. Save Space with a Link |
| |
|
|
Section 15.5. Limiting File Sizes |
| |
|
|
Section 15.6. Compressing Files to Save Space |
| |
|
|
Section 15.7. Save Space: tar and compress a Directory Tree |
| |
|
|
Section 15.8. How Much Disk Space? |
| |
|
|
Section 15.9. Compressing a Directory Tree: Fine-Tuning |
| |
|
|
Section 15.10. Save Space in Executable Files with strip |
| |
|
|
Section 15.11.  Spell Checking, Word Counting, and Textual Analysis |
| |
|
|
Section 16.1. The Unix spell Command |
| |
|
|
Section 16.2. Check Spelling Interactively with ispell |
| |
|
|
Section 16.3. How Do I Spell That Word? |
| |
|
|
Section 16.4. Inside spell |
| |
|
|
Section 16.5. Adding Words to ispell's Dictionary |
| |
|
|
Section 16.6. Counting Lines, Words, and Characters: wc |
| |
|
|
Section 16.7. Find a a Doubled Word |
| |
|
|
Section 16.8. Looking for Closure |
| |
|
|
Section 16.9. Just the Words, Please |
| |
| |
|
|
Chapter 17. vi Tips and Tricks |
| |
|
|
Section 17.1. The vi Editor: Why So Much Material? |
| |
|
|
Section 17.2. What We Cover |
| |
|
|
Section 17.3. Editing Multiple Files with vi |
| |
|
|
Section 17.4. Edits Between Files |
| |
|
|
Section 17.5. Local Settings for vi |
| |
|
|
Section 17.6. Using Buffers to Move or Copy Text |
| |
|
|
Section 17.7.  Using Search Patterns and Global Commands |
| |
|
|
Section 17.9. Confirming Substitutions in vi |
| |
|
|
Section 17.10. Keep Your Original File, Write to a New File |
| |
|
|
Section 17.11. Saving Part of a File |
| |
|
|
Section 17.12. Appending to an Existing File |
| |
|
|
Section 17.13. Moving Blocks of Text by Patterns |
| |
|
|
Section 17.14. Useful Global Commands (with Pattern Matches) |
| |
|
|
Section 17.15. Counting Occurrences; Stopping Search Wraps |
| |
|
|
Section 17.16. Capitalizing Every Word on a Line |
| |
|
|
Section 17.17. Per-File Setups in Separate Files |
| |
|
|
Section 17.18.  Be Careful with vi -r Recovered Buffers |
| |
|
|
Section 17.21. Shell Escapes: Running One UnixCommand While Using Another |
| |
|
|
Section 17.22. vi Compound Searches |
| |
|
|
Section 17.23. vi Word Abbreviation |
| |
|
|
Section 17.24. Using vi Abbreviations as Commands (Cut and Paste Between
vi's) |
| |
|
|
Section 17.25. Fixing Typos with vi Abbreviations |
| |
|
|
Section 17.26. vi Line Commands Versus Character Commands |
| |
|
|
Section 17.27. Out of Temporary Space? Use Another Directory |
| |
|
|
Section 17.28. Neatening Lines |
| |
|
|
Section 17.29. Finding Your Place with Undo |
| |
|
|
Section 17.30. Setting Up vi with the .exrc File |
| |
| |
|
|
Chapter 18. Creating Custom Commands in vi |
| |
|
|
Section 18.1.  Save Time and Typing with the vi map Commands |
| |
|
|
Section 18.3.  Keymaps for Pasting into a Window Running vi |
| |
|
|
Section 18.6. Protecting Keys from Interpretation by ex |
| |
|
|
Section 18.7. Maps for Repeated Edits |
| |
|
|
Section 18.8. More Examples of Mapping Keys in vi |
| |
|
|
Section 18.9. Repeating a vi Keymap |
| |
|
|
Section 18.10. Typing in Uppercase Without CAPS LOCK |
| |
|
|
Section 18.11. Text-Input Mode Cursor Motion with No Arrow Keys |
| |
|
|
Section 18.12.  File-Backup Macros |
| |
| |
|
|
Chapter 19. GNU Emacs |
| |
|
|
Section 19.1. Emacs: The Other Editor |
| |
|
|
Section 19.2. Emacs Features: A Laundry List |
| |
|
|
Section 19.3. Customizations and How to Avoid Them |
| |
|
|
Section 19.4. Backup and Auto-Save Files |
| |
|
|
Section 19.5. Putting Emacs in Overwrite Mode |
| |
|
|
Section 19.6. Command Completion |
| |
|
|
Section 19.7. Mike's Favorite Timesavers |
| |
|
|
Section 19.8. Rational Searches |
| |
|
|
Section 19.9. Unset PWD Before Using Emacs |
| |
|
|
Section 19.10. Inserting Binary Characters into Files |
| |
|
|
Section 19.11. Using Word-Abbreviation Mode |
| |
|
|
Section 19.12. Directories for Emacs Hacks |
| |
|
|
Section 19.13. An Absurd Amusement |
| |
| |
|
|
Chapter 20. Batch Editing |
| |
|
|
Section 20.1. Why Line Editors Aren't Dinosaurs |
| |
|
|
Section 20.2. Writing Editing Scripts |
| |
|
|
Section 20.3. Line Addressing |
| |
|
|
Section 20.4. Useful ex Commands |
| |
|
|
Section 20.5. Running Editing Scripts Within vi |
| |
|
|
Section 20.6. Change Many Files by Editing Just One |
| |
|
|
Section 20.7. ed/ex Batch Edits: A Typical Example |
| |
|
|
Section 20.8.  Quick Reference: awk |
| |
|
|
Section 20.11. Versions of awk |
| |
| |
|
|
Chapter 21. You Can't Quite Call This Editing |
| |
|
|
Section 21.1. And Why Not? |
| |
|
|
Section 21.2. Neatening Text with fmt |
| |
|
|
Section 21.3. Alternatives to fmt |
| |
|
|
Section 21.4. Clean Up Program Comment Blocks |
| |
|
|
Section 21.5. Remove Mail/News Headers with behead |
| |
|
|
Section 21.6. Low-Level File Butchery with dd |
| |
|
|
Section 21.7. offset: Indent Text |
| |
|
|
Section 21.8. Centering Lines in a File |
| |
|
|
Section 21.9. Splitting Files at Fixed Points: split |
| |
|
|
Section 21.10. Splitting Files by Context: csplit |
| |
|
|
Section 21.11. Hacking on Characters with tr |
| |
|
|
Section 21.12. Encoding "Binary" Files into ASCII |
| |
|
|
Section 21.13. Text Conversion with dd |
| |
|
|
Section 21.14. Cutting Columns or Fields |
| |
|
|
Section 21.15. Making Text in Columns with pr |
| |
|
|
Section 21.16. Make Columns Automatically with column |
| |
|
|
Section 21.17. Straightening Jagged Columns |
| |
|
|
Section 21.18. Pasting Things in Columns |
| |
|
|
Section 21.19. Joining Lines with join |
| |
|
|
Section 21.20. What Is (or Isn't) Unique? |
| |
|
|
Section 21.21. Rotating Text |
| |
| |
|
|
Chapter 22. Sorting |
| |
|
|
Section 22.1. Putting Things in Order |
| |
|
|
Section 22.2.  Changing the sort Field Delimiter |
| |
|
|
Section 22.4. Confusion with Whitespace Field Delimiters |
| |
|
|
Section 22.5. Alphabetic and Numeric Sorting |
| |
|
|
Section 22.6.   Job Control |
| |
|
|
Section 23.1. Job Control in a Nutshell |
| |
|
|
Section 23.2. Job Control Basics |
| |
|
|
Section 23.3. Using jobs Effectively |
| |
|
|
Section 23.4. Some Gotchas with Job Control |
| |
|
|
Section 23.5. The "Current Job" Isn't Always What You Expect |
| |
|
|
Section 23.6. Job Control and autowrite: Real Timesavers! |
| |
|
|
Section 23.7. System Overloaded? Try Stopping Some Jobs |
| |
|
|
Section 23.8. Notification When Jobs Change State |
| |
|
|
Section 23.9.  Disowning Processes |
| |
|
|
Section 23.12. Linux Virtual Consoles |
| |
|
|
Section 23.13. Stopping Remote Login Sessions |
| |
| |
|
|
Chapter 24. Starting, Stopping, and Killing Processes |
| |
|
|
Section 24.1. What's in This Chapter |
| |
|
|
Section 24.2. fork and exec |
| |
|
|
Section 24.3. Managing Processes: Overall Concepts |
| |
|
|
Section 24.4. Subshells |
| |
|
|
Section 24.5. The ps Command |
| |
|
|
Section 24.6. The Controlling Terminal |
| |
|
|
Section 24.7. Tracking Down Processes |
| |
|
|
Section 24.8. Why ps Prints Some Commands in Parentheses |
| |
|
|
Section 24.9. The /proc Filesystem |
| |
|
|
Section 24.10. What Are Signals? |
| |
|
|
Section 24.11. Killing Foreground Jobs |
| |
|
|
Section 24.12.  Printer Queue Watcher: A Restartable Daemon Shell Script |
| |
|
|
Section 24.14. Killing All Your Processes |
| |
|
|
Section 24.15. Killing Processes by Name? |
| |
|
|
Section 24.16.  Processes Out of Control? Just STOP Them |
| |
|
|
Section 24.18. Cleaning Up an Unkillable Process |
| |
|
|
Section 24.19.  The Process Chain to Your Window |
| |
|
|
Section 24.21. Terminal Windows Without Shells |
| |
|
|
Section 24.22. Close a Window by Killing Its Process(es) |
| |
| |
|
|
Chapter 25. Delayed Execution |
| |
|
|
Section 25.1. Building Software Robots the Easy Way |
| |
|
|
Section 25.2. Periodic Program Execution: The cron Facility |
| |
|
|
Section 25.3. Adding crontab Entries |
| |
|
|
Section 25.4. Including Standard Input Within a cron Entry |
| |
|
|
Section 25.5. The at Command |
| |
|
|
Section 25.6. Making Your at Jobs Quiet |
| |
|
|
Section 25.7. Checking and Removing Jobs |
| |
|
|
Section 25.8. Avoiding Other at and cron Jobs |
| |
|
|
Section 25.9.  System Performance and Profiling |
| |
|
|
Section 26.1. Timing Is Everything |
| |
|
|
Section 26.2. Timing Programs |
| |
|
|
Section 26.3. What Commands Are Running and How Long Do They Take? |
| |
|
|
Section 26.4. Checking System Load: uptime |
| |
|
|
Section 26.5. Know When to Be "nice" to Other Users...and When Not To |
| |
|
|
Section 26.6. A nice Gotcha |
| |
|
|
Section 26.7.  Shell Interpretation |
| |
|
|
Section 27.1.  How the Shell Executes Other Commands |
| |
|
|
Section 27.3. What's a Shell, Anyway? |
| |
|
|
Section 27.4. Command Evaluation and Accidentally Overwriting Files |
| |
|
|
Section 27.5. Output Command-Line Arguments One by One |
| |
|
|
Section 27.6. Controlling Shell Command Searches |
| |
|
|
Section 27.7. Wildcards Inside Aliases |
| |
|
|
Section 27.8. eval: When You Need Another Chance |
| |
|
|
Section 27.9. Which One Will bash Use? |
| |
|
|
Section 27.10. Which One Will the C Shell Use? |
| |
|
|
Section 27.11. Is It "2>&1 file" or "> file 2>&1"? Why? |
| |
|
|
Section 27.12. Bourne Shell Quoting |
| |
|
|
Section 27.13. Differences Between Bourne and C Shell Quoting |
| |
|
|
Section 27.14. Quoting Special Characters in Filenames |
| |
|
|
Section 27.15. Verbose and Echo Settings Show Quoting |
| |
|
|
Section 27.16. Here Documents |
| |
|
|
Section 27.17. "Special" Characters and Operators |
| |
|
|
Section 27.18. How Many Backslashes? |
| |
| |
|
|
Chapter 28. Saving Time on the Command Line |
| |
|
|
Section 28.1. What's Special About the Unix Command Line |
| |
|
|
Section 28.2. Reprinting Your Command Line with CTRL-r |
| |
|
|
Section 28.3. Use Wildcards to Create Files? |
| |
|
|
Section 28.4. Build Strings with { } |
| |
|
|
Section 28.5. String Editing (Colon) Operators |
| |
|
|
Section 28.6. Automatic Completion |
| |
|
|
Section 28.7. Don't Match Useless Files in Filename Completion |
| |
|
|
Section 28.8. Repeating Commands |
| |
|
|
Section 28.9. Repeating and Varying Commands |
| |
|
|
Section 28.10. Repeating a Command with Copy-and-Paste |
| |
|
|
Section 28.11. Repeating a Time-Varying Command |
| |
|
|
Section 28.12. Multiline Commands, Secondary Prompts |
| |
|
|
Section 28.13. Here Document Example #1: Unformatted Form Letters |
| |
|
|
Section 28.14. Command Substitution |
| |
|
|
Section 28.15. Handling Lots of Text with Temporary Files |
| |
|
|
Section 28.16. Separating Commands with Semicolons |
| |
|
|
Section 28.17. Dealing with Too Many Arguments |
| |
|
|
Section 28.18. Expect |
| |
| |
|
|
Chapter 29. Custom Commands |
| |
|
|
Section 29.1. Creating Custom Commands |
| |
|
|
Section 29.2. Introduction to Shell Aliases |
| |
|
|
Section 29.3. C-Shell Aliases with Command-Line Arguments |
| |
|
|
Section 29.4. Setting and Unsetting Bourne-Type Aliases |
| |
|
|
Section 29.5. Korn-Shell Aliases |
| |
|
|
Section 29.6. zsh Aliases |
| |
|
|
Section 29.7. Sourceable Scripts |
| |
|
|
Section 29.8. Avoiding C-Shell Alias Loops |
| |
|
|
Section 29.9. How to Put if-then-else in a C-Shell Alias |
| |
|
|
Section 29.10. Fix Quoting in csh Aliases with makealias and quote |
| |
|
|
Section 29.11. Shell Function Basics |
| |
|
|
Section 29.12. Shell Function Specifics |
| |
|
|
Section 29.13. Propagating Shell Functions |
| |
|
|
Section 29.14. Simulated Bourne Shell Functions and Aliases |
| |
| |
|
|
Chapter 30. The Use of History |
| |
|
|
Section 30.1. The Lessons of History |
| |
|
|
Section 30.2. History in a Nutshell |
| |
|
|
Section 30.3. My Favorite Is !$ |
| |
|
|
Section 30.4. My Favorite Is !:n* |
| |
|
|
Section 30.5. My Favorite Is ^^ |
| |
|
|
Section 30.6. Using !$ for Safety with Wildcards |
| |
|
|
Section 30.7. History by Number |
| |
|
|
Section 30.8. History Substitutions |
| |
|
|
Section 30.9. Repeating a Cycle of Commands |
| |
|
|
Section 30.10. Running a Series of Commands on a File |
| |
|
|
Section 30.11. Check Your History First with :p |
| |
|
|
Section 30.12. Picking Up Where You Left Off |
| |
|
|
Section 30.13. Pass History to Another Shell |
| |
|
|
Section 30.14. Shell Command-Line Editing |
| |
|
|
Section 30.15. Changing History Characters with histchars |
| |
|
|
Section 30.16. Instead of Changing History Characters |
| |
| |
|
|
Chapter 31. Moving Around in a Hurry |
| |
|
|
Section 31.1. Getting Around the Filesystem |
| |
|
|
Section 31.2. Using Relative and Absolute Pathnames |
| |
|
|
Section 31.3. What Good Is a Current Directory? |
| |
|
|
Section 31.4.  Saving Time When You Change Directories: cdpath |
| |
|
|
Section 31.6. Loop Control: break and continue |
| |
|
|
Section 31.7. The Shells' pushd and popd Commands |
| |
|
|
Section 31.8. Nice Aliases for pushd |
| |
|
|
Section 31.9. Quick cds with Aliases |
| |
|
|
Section 31.10. cd by Directory Initials |
| |
|
|
Section 31.11. Finding (Anyone's) Home Directory, Quickly |
| |
|
|
Section 31.12.  Automatic Setup When You Enter/Exit a Directory |
| |
| |
|
|
Chapter 32. Regular Expressions (Pattern Matching) |
| |
|
|
Section 32.1. That's an Expression |
| |
|
|
Section 32.2. Don't Confuse Regular Expressions with Wildcards |
| |
|
|
Section 32.3. Understanding Expressions |
| |
|
|
Section 32.4. Using Metacharacters in Regular Expressions |
| |
|
|
Section 32.5. Regular Expressions: The Anchor Characters ^ and $ |
| |
|
|
Section 32.6. Regular Expressions: Matching a Character with a Character
Set |
| |
|
|
Section 32.7. Regular Expressions: Match Any Character with . (Dot) |
| |
|
|
Section 32.8. Regular Expressions: Specifying a Range of Characters with
[...] |
| |
|
|
Section 32.9. Regular Expressions: Exceptions in a Character Set |
| |
|
|
Section 32.10. Regular Expressions: Repeating Character Sets with * |
| |
|
|
Section 32.11. Regular Expressions: Matching a Specific Number of Sets with
\ { and \ } |
| |
|
|
Section 32.12. Regular Expressions: Matching Words with \ < and \ > |
| |
|
|
Section 32.13. Regular Expressions: Remembering Patterns with \ (, \ ), and
\1 |
| |
|
|
Section 32.14. Regular Expressions: Potential Problems |
| |
|
|
Section 32.15. Extended Regular Expressions |
| |
|
|
Section 32.16. Getting Regular Expressions Right |
| |
|
|
Section 32.17. Just What Does a Regular Expression Match? |
| |
|
|
Section 32.18. Limiting the Extent of a Match |
| |
|
|
Section 32.19. I Never Meta Character I Didn't Like |
| |
|
|
Section 32.20. Valid Metacharacters for Different Unix Programs |
| |
|
|
Section 32.21. Pattern Matching Quick Reference with Examples |
| |
| |
|
|
Chapter 33. Wildcards |
| |
|
|
Section 33.1. File-Naming Wildcards |
| |
|
|
Section 33.2. Filename Wildcards in a Nutshell |
| |
|
|
Section 33.3. Who Handles Wildcards? |
| |
|
|
Section 33.4. What if a Wildcard Doesn't Match? |
| |
|
|
Section 33.5. Maybe You Shouldn't Use Wildcards in Pathnames |
| |
|
|
Section 33.6. Getting a List of Matching Files with grep -l |
| |
|
|
Section 33.7.  The sed Stream Editor |
| |
|
|
Section 34.1. sed Sermon^H^H^H^H^H^HSummary |
| |
|
|
Section 34.2. Two Things You Must Know About sed |
| |
|
|
Section 34.3. Invoking sed |
| |
|
|
Section 34.4.  Order of Commands in a Script |
| |
|
|
Section 34.7. One Thing at a Time |
| |
|
|
Section 34.8. Delimiting a Regular Expression |
| |
|
|
Section 34.9. Newlines in a sed Replacement |
| |
|
|
Section 34.10. Referencing the Search String in a Replacement |
| |
|
|
Section 34.11. Referencing Portions of a Search String |
| |
|
|
Section 34.12. Search and Replacement: One Match Among Many |
| |
|
|
Section 34.13. Transformations on Text |
| |
|
|
Section 34.14. Hold Space: The Set-Aside Buffer |
| |
|
|
Section 34.15. Transforming Part of a Line |
| |
|
|
Section 34.16. Making Edits Across Line Boundaries |
| |
|
|
Section 34.17. The Deliberate Scrivener |
| |
|
|
Section 34.18. Searching for Patterns Split Across Lines |
| |
|
|
Section 34.19. Multiline Delete |
| |
|
|
Section 34.20. Making Edits Everywhere Except... |
| |
|
|
Section 34.21. The sed Test Command |
| |
|
|
Section 34.22. Uses of the sed Quit Command |
| |
|
|
Section 34.23.  Shell Programming for the Uninitiated |
| |
|
|
Section 35.1. Writing a Simple Shell Program |
| |
|
|
Section 35.2. Everyone Should Learn Some Shell Programming |
| |
|
|
Section 35.3. What Environment Variables Are Good For |
| |
|
|
Section 35.4. Parent-Child Relationships |
| |
|
|
Section 35.5. Predefined Environment Variables |
| |
|
|
Section 35.6. The PATH Environment Variable |
| |
|
|
Section 35.7. PATH and path |
| |
|
|
Section 35.8. The DISPLAY Environment Variable |
| |
|
|
Section 35.9. Shell Variables |
| |
|
|
Section 35.10. Test String Values with Bourne-Shell case |
| |
|
|
Section 35.11. Pattern Matching in case Statements |
| |
|
|
Section 35.12. Exit Status of Unix Processes |
| |
|
|
Section 35.13. Test Exit Status with the if Statement |
| |
|
|
Section 35.14. Testing Your Success |
| |
|
|
Section 35.15. Loops That Test Exit Status |
| |
|
|
Section 35.16. Set Exit Status of a Shell (Script) |
| |
|
|
Section 35.17. Trapping Exits Caused by Interrupts |
| |
|
|
Section 35.18. read: Reading from the Keyboard |
| |
|
|
Section 35.19. Shell Script "Wrappers" for awk, sed, etc. |
| |
|
|
Section 35.20. Handling Command-Line Arguments in Shell Scripts |
| |
|
|
Section 35.21. Handling Command-Line Arguments with a for Loop |
| |
|
|
Section 35.22. Handling Arguments with while and shift |
| |
|
|
Section 35.23. Loop Control: break and continue |
| |
|
|
Section 35.24. Standard Command-Line Parsing |
| |
|
|
Section 35.25. The Bourne Shell set Command |
| |
|
|
Section 35.26. test: Testing Files and Strings |
| |
|
|
Section 35.27. Picking a Name for a New Command |
| |
|
|
Section 35.28. Finding a Program Name and Giving Your Program Multiple
Names |
| |
|
|
Section 35.29. Reading Files with the . and source Commands |
| |
|
|
Section 35.30. Using Shell Functions in Shell Scripts |
| |
| |
|
|
Chapter 36. Shell Programming for the Initiated |
| |
|
|
Section 36.1. Beyond the Basics |
| |
|
|
Section 36.2. The Story of : # #! |
| |
|
|
Section 36.3. Don't Need a Shell for Your Script?  Making #! Search the PATH |
| |
|
|
Section 36.5. The exec Command |
| |
|
|
Section 36.6. The Unappreciated Bourne Shell ":" Operator |
| |
|
|
Section 36.7. Parameter Substitution |
| |
|
|
Section 36.8. Save Disk Space and Programming: Multiple Names for a Program |
| |
|
|
Section 36.9. Finding the Last Command-Line Argument |
| |
|
|
Section 36.10. How to Unset All Command-Line Parameters |
| |
|
|
Section 36.11. Standard Input to a for Loop |
| |
|
|
Section 36.12. Making a for Loop with Multiple Variables |
| |
|
|
Section 36.13. Using basename and dirname |
| |
|
|
Section 36.14. A while Loop with Several Loop Control Commands |
| |
|
|
Section 36.15. Overview: Open Files and File Descriptors |
| |
|
|
Section 36.16. n>&m: Swap Standard Output and Standard Error |
| |
|
|
Section 36.17. A Shell Can Read a Script from Its Standard Input, but... |
| |
|
|
Section 36.18. Shell Scripts On-the-Fly from Standard Input |
| |
|
|
Section 36.19. Quoted hereis Document Terminators: sh Versus csh |
| |
|
|
Section 36.20. Turn Off echo for "Secret" Answers |
| |
|
|
Section 36.21. Quick Reference: expr |
| |
|
|
Section 36.22. Testing Characters in a String with expr |
| |
|
|
Section 36.23. Grabbing Parts of a String |
| |
|
|
Section 36.24. Nested Command Substitution |
| |
|
|
Section 36.25. Testing Two Strings with One case Statement |
| |
|
|
Section 36.26. Outputting Text to an X Window |
| |
|
|
Section 36.27. Shell Lockfile |
| |
| |
|
|
Chapter 37. Shell Script Debugging and Gotchas |
| |
|
|
Section 37.1. Tips for Debugging Shell Scripts |
| |
|
|
Section 37.2. Bourne Shell Debugger Shows a Shell Variable |
| |
|
|
Section 37.3. Stop Syntax Errors in Numeric Tests |
| |
|
|
Section 37.4.  Quoting and Command-Line Parameters |
| |
|
|
Section 37.6. How Unix Keeps Time |
| |
|
|
Section 37.7. Copy What You Do with script |
| |
|
|
Section 37.8. Cleaning script Files |
| |
|
|
Section 37.9.  Backing Up Files |
| |
|
|
Section 38.1. What Is This "Backup" Thing? |
| |
|
|
Section 38.2. tar in a Nutshell |
| |
|
|
Section 38.3. Make Your Own Backups |
| |
|
|
Section 38.4.  How to Make Backups to a Local Device |
| |
|
|
Section 38.6. Restoring Files from Tape with tar |
| |
|
|
Section 38.7. Using tar to a Remote Tape Drive |
| |
|
|
Section 38.8. Using GNU tar with a Remote Tape Drive |
| |
|
|
Section 38.9. On-Demand Incremental Backups of a Project |
| |
|
|
Section 38.10. Using Wildcards with tar |
| |
|
|
Section 38.11. Avoid Absolute Paths with tar |
| |
|
|
Section 38.12. Getting tar's Arguments in the Right Order |
| |
|
|
Section 38.13. The cpio Tape Archiver |
| |
|
|
Section 38.14. Industrial Strength Backups |
| |
| |
|
|
Chapter 39. Creating and Reading Archives |
| |
|
|
Section 39.1. Packing Up and Moving |
| |
|
|
Section 39.2. Using tar to Create and Unpack Archives |
| |
|
|
Section 39.3. GNU tar Sampler |
| |
|
|
Section 39.4. Managing and Sharing Files with RCS and CVS |
| |
|
|
Section 39.5. RCS Basics |
| |
|
|
Section 39.6. List RCS Revision Numbers with rcsrevs |
| |
|
|
Section 39.7. CVS Basics |
| |
|
|
Section 39.8.  Software Installation |
| |
|
|
Section 40.1. /usr/bin and Other Software Directories |
| |
|
|
Section 40.2. The Challenges of Software Installation on Unix |
| |
|
|
Section 40.3. Which make? |
| |
|
|
Section 40.4. Simplifying the make Process |
| |
|
|
Section 40.5. Using Debian's dselect |
| |
|
|
Section 40.6. Installing Software with Debian's Apt-Get |
| |
|
|
Section 40.7. Interruptable gets with wget |
| |
|
|
Section 40.8. The curl Application and One-Step GNU-Darwin Auto-Installer
for OS X |
| |
|
|
Section 40.9. Installation with FreeBSD Ports |
| |
|
|
Section 40.10. Installing with FreeBSD Packages |
| |
|
|
Section 40.11. Finding and Installing RPM Packaged Software |
| |
| |
|
|
Chapter 41. Perl |
| |
|
|
Section 41.1. High-Octane Shell Scripting |
| |
|
|
Section 41.2. Checking your Perl Installation |
| |
|
|
Section 41.3. Compiling Perl from Scratch |
| |
|
|
Section 41.4. Perl Boot Camp, Part 1: Typical Script Anatomy |
| |
|
|
Section 41.5. Perl Boot Camp, Part 2: Variables and Data Types |
| |
|
|
Section 41.6. Perl Boot Camp, Part 3: Branching and Looping |
| |
|
|
Section 41.7. Perl Boot Camp, Part 4: Pattern Matching |
| |
|
|
Section 41.8. Perl Boot Camp, Part 5: Perl Knows Unix |
| |
|
|
Section 41.9. Perl Boot Camp, Part 6: Modules |
| |
|
|
Section 41.10. Perl Boot Camp, Part 7: perldoc |
| |
|
|
Section 41.11. CPAN |
| |
|
|
Section 41.12. Make Custom grep Commands (etc.) with Perl |
| |
|
|
Section 41.13. Perl and the Internet |
| |
| |
|
|
Chapter 42. Python |
| |
|
|
Section 42.1. What Is Python? |
| |
|
|
Section 42.2. Installation and Distutils |
| |
|
|
Section 42.3. Python Basics |
| |
|
|
Section 42.4.  What About Perl? |
| |
| |
| |
|
Part VIII: Communication and Connectivity |
| |
|
|
Chapter 43. Redirecting Input and Output |
| |
|
|
Section 43.1. Using Standard Input and Output |
| |
|
|
Section 43.2.  Send (Only) Standard Error Down a Pipe |
| |
|
|
Section 43.4. Problems Piping to a Pager |
| |
|
|
Section 43.5. Redirection in C Shell: Capture Errors, Too? |
| |
|
|
Section 43.6. Safe I/O Redirection with noclobber |
| |
|
|
Section 43.7. The ( ) Subshell Operators |
| |
|
|
Section 43.8. Send Output Two or More Places |
| |
|
|
Section 43.9. How to tee Several Commands into One Place |
| |
|
|
Section 43.10. Redirecting Output to More Than One Place |
| |
|
|
Section 43.11. Named Pipes: FIFOs |
| |
|
|
Section 43.12. What Can You Do with an Empty File? |
| |
| |
|
|
Chapter 44. Devices |
| |
|
|
Section 44.1. Quick Introduction to Hardware |
| |
|
|
Section 44.2. Reading Kernel Boot Output |
| |
|
|
Section 44.3. Basic Kernel Configuration |
| |
|
|
Section 44.4. Disk Partitioning |
| |
|
|
Section 44.5. Filesystem Types and /etc/fstab |
| |
|
|
Section 44.6. Mounting and Unmounting Removable Filesystems |
| |
|
|
Section 44.7. Loopback Mounts |
| |
|
|
Section 44.8. Network Devices ifconfig |
| |
|
|
Section 44.9. Mounting Network Filesystems NFS, SMBFS |
| |
|
|
Section 44.10. Win Is a Modem Not a Modem? |
| |
|
|
Section 44.11. Setting Up a Dialup PPP Session |
| |
|
|
Section 44.12. USB Configuration |
| |
|
|
Section 44.13. Dealing with Sound Cards and Other Annoying Hardware |
| |
|
|
Section 44.14.  Printing |
| |
|
|
Section 45.1. Introduction to Printing |
| |
|
|
Section 45.2. Introduction to Printing on Unix |
| |
|
|
Section 45.3. Printer Control with lpc |
| |
|
|
Section 45.4. Using Different Printers |
| |
|
|
Section 45.5. Using Symbolic Links for Spooling |
| |
|
|
Section 45.6. Formatting Plain Text: pr |
| |
|
|
Section 45.7. Formatting Plain Text: enscript |
| |
|
|
Section 45.8. Printing Over a Network |
| |
|
|
Section 45.9. Printing Over Samba |
| |
|
|
Section 45.10. Introduction to Typesetting |
| |
|
|
Section 45.11. A Bit of Unix Typesetting History |
| |
|
|
Section 45.12. Typesetting Manpages: nroff |
| |
|
|
Section 45.13. Formatting Markup Languages troff, LATEX,
HTML, and So On |
| |
|
|
Section 45.14. Printing Languages PostScript, PCL, DVI, PDF |
| |
|
|
Section 45.15. Converting Text Files into a Printing Language |
| |
|
|
Section 45.16. Converting Typeset Files into a Printing Language |
| |
|
|
Section 45.17. Converting Source Files Automagically Within the Spooler |
| |
|
|
Section 45.18. The Common Unix Printing System (CUPS) |
| |
|
|
Section 45.19. The Portable Bitmap Package |
| |
| |
|
|
Chapter 46. Connectivity |
| |
|
|
Section 46.1. TCP/IP IP Addresses and Ports |
| |
|
|
Section 46.2. /etc/services Is Your Friend |
| |
|
|
Section 46.3. Status and Troubleshooting |
| |
|
|
Section 46.4. Where, Oh Where Did That Packet Go? |
| |
|
|
Section 46.5. The Director of Operations: inetd |
| |
|
|
Section 46.6. Secure Shell (SSH) |
| |
|
|
Section 46.7. Configuring an Anonymous FTP Server |
| |
|
|
Section 46.8. Mail SMTP, POP, and IMAP |
| |
|
|
Section 46.9. Domain Name Service (DNS) |
| |
|
|
Section 46.10. Dynamic Host Configuration Protocol (DHCP) |
| |
|
|
Section 46.11. Gateways and NAT |
| |
|
|
Section 46.12. Firewalls |
| |
|
|
Section 46.13. Gatewaying from a Personal LAN over a Modem |
| |
| |
|
|
Chapter 47. Connecting to MS Windows |
| |
|
|
Section 47.1. Building Bridges |
| |
|
|
Section 47.2. Installing and Configuring Samba |
| |
|
|
Section 47.3. Securing Samba |
| |
|
|
Section 47.4. SWAT and GUI SMB Browsers |
| |
|
|
Section 47.5. Printing with Samba |
| |
|
|
Section 47.6. Connecting to SMB Shares from Unix |
| |
|
|
Section 47.7. Sharing Desktops with VNC |
| |
|
|
Section 47.8. Of Emulators and APIs |
| |
|
|
Section 47.9.  Security Basics |
| |
|
|
Section 48.1. Understanding Points of Vulnerability |
| |
|
|
Section 48.2. CERT Security Checklists |
| |
|
|
Section 48.3. Keeping Up with Security Alerts |
| |
|
|
Section 48.4. What We Mean by Buffer Overflow |
| |
|
|
Section 48.5. What We Mean by DoS |
| |
|
|
Section 48.6. Beware of Sluggish Performance |
| |
|
|
Section 48.7. Intruder Detection |
| |
|
|
Section 48.8. Importance of MOTD |
| |
|
|
Section 48.9. The Linux proc Filesystem |
| |
|
|
Section 48.10. Disabling inetd |
| |
|
|
Section 48.11. Disallow rlogin and rsh |
| |
|
|
Section 48.12. TCP Wrappers |
| |
| |
|
|
Chapter 49. Root, Group, and User Management |
| |
|
|
Section 49.1. Unix User/Group Infrastructure |
| |
|
|
Section 49.2. When Does a User Become a User |
| |
|
|
Section 49.3. Forgetting the root Password |
| |
|
|
Section 49.4. Setting an Exact umask |
| |
|
|
Section 49.5. Group Permissions in a Directory with the setgid Bit |
| |
|
|
Section 49.6. Groups and Group Ownership |
| |
|
|
Section 49.7. Add Users to a Group to Deny Permissions |
| |
|
|
Section 49.8. Care and Feeding of SUID and SGID Scripts |
| |
|
|
Section 49.9. Substitute Identity with su |
| |
|
|
Section 49.10. Never Log In as root |
| |
|
|
Section 49.11. Providing Superpowers with sudo |
| |
|
|
Section 49.12. Enabling Root in Darwin |
| |
|
|
Section 49.13. Disable logins |
| |
| |
|
|
Chapter 50. File Security, Ownership, and Sharing |
| |
|
|
Section 50.1. Introduction to File Ownership and Security |
| |
|
|
Section 50.2. Tutorial on File and Directory Permissions |
| |
|
|
Section 50.3. Who Will Own a New File? |
| |
|
|
Section 50.4. Protecting Files with the Sticky Bit |
| |
|
|
Section 50.5. Using chmod to Change File Permission |
| |
|
|
Section 50.6. The Handy chmod = Operator |
| |
|
|
Section 50.7.   A Directory That People Can Access but Can't List |
| |
|
|
Section 50.11. Juggling Permissions |
| |
|
|
Section 50.12. File Verification with md5sum |
| |
|
|
Section 50.13. Shell Scripts Must Be Readable and (Usually) Executable |
| |
|
|
Section 50.14. Why Can't You Change File Ownership? |
| |
|
|
Section 50.15. How to Change File Ownership Without chown |
| |
| |
|
|
Chapter 51. SSH |
| |
|
|
Section 51.1. Enabling Remote Access on Mac OS X |
| |
|
|
Section 51.2. Protecting Access Through SSH |
| |
|
|
Section 51.3. Free SSH with OpenSSH |
| |
|
|
Section 51.4. SSH Problems and Solutions |
| |
|
|
Section 51.5. General and Authentication Problems |
| |
|
|
Section 51.6. Key and Agent Problems |
| |
|
|
Section 51.7. Server and Client Problems |
| |
| |
| |
|
Glossary |
| |
Copyright
Copyright © 2003, 1997, 1993 O'Reilly & Associates, Inc. All
rights reserved.
Printed in the United States of America.
Published by O'Reilly & Associates, Inc., 1005 Gravenstein
Highway North, Sebastopol, CA 95472.
O'Reilly & Associates books may be purchased for educational,
business, or sales promotional use. Online editions are also available for most
titles (http://safari.oreilly.com).
For more information contact our corporate/institutional sales department:
800-998-9938 or
<EMAIL_ADDRESS>Nutshell Handbook, the Nutshell Handbook logo, and the
O'Reilly logo are registered trademarks of O'Reilly & Associates, Inc. Many of
the designations used by manufacturers and sellers to distinguish their products
are claimed as trademarks. Where those designations appear in this book, and
O'Reilly & Associates, Inc. was aware of a trademark claim, the designations
have been printed in caps or initial caps. The association between the image of
a hand drill and the topic of Unix is a trademark of O'Reilly & Associates, Inc.
While every precaution has been taken in the preparation of
this book, the publisher and the author assume no responsibility for errors or
omissions, or for damages resulting from the use of the information contained
herein.
How to Use This Book
This section refers to conventions used in the print book and
explains how they were modified for the Safari version. The numbers in the
following images correspond to the list below.
-
Summary Boxes. You'll see
gray shaded summary boxes all through the book. (On Safari, the Summary Boxes
are bordered sidebars with the title "Summary Box.") They summarize a topic
and point you to articles with examples and further explanation.
-
Article/Section Number.
The first two digits indicate in which chapter the article resides; the last
two digits indicate the number of the article within that chapter. The article
number is used to refer to this article in all cross-references throughout the
book. (On Safari, Article numbers correspond to Section numbers.)
-
Cross-Reference in a Sentence.
To find out more about the topic displayed in gray type (On Safari, this text
is displayed in boldface.), see the article referenced by the number in
parentheses immediately following the term.
-
Cross-Reference in a Code
Example. When a cross-reference occurs in an example, the
cross-referenced text and related article number appear in the left margin.
(On Safari, these cross-references appear above the code example.)
-
Globe If you don't want to
type this script into a file yourself, or if we're talking about a C program
that isn't shown, you can download it from the book's web site. See the
Preface for full details on the content available for download. (Online
version available at
http://examples.oreilly.com/upt3)
-
Screw. Be careful with
this feature, or you might get screwed.
-
Pushpin. A note to keep in
mind, or a helpful tip.
-
Bomb. A bomb icon in the
margin is a cross-reference to another article that explains the possible
trouble you might encounter using the tip or script in the current article.
(You can think of the bomb as a cross-referenced screw.) (On Safari, the Bomb
appears above the paragraph it refers to.)
-
Author's Initials. The
author's full name is listed in the Preface.
Preface
A Book for Browsing
Technical books can be boring. But this is not an ordinary
technical book! This book is like an almanac, a news magazine, and a hypertext
database all rolled into one. Instead of trying to put the topics in perfect
order and expecting you to start at the beginning, then read through to the
end we hope that you'll browse. Start anywhere. Read what you want. (That's
not quite true. First, you should read this Preface and the pages before it
titled How to Use This Book. They will help you
get the most out of your time with this book. Next, you may want to skim through
the Unix fundamentals in
Chapter 1.
Then read what you want.)
Like an Almanac
The book is full of practical information. The main purpose
isn't to teach you concepts (though they're in here). We've picked a lot of
common problems, and we'll show you how to solve them.
Even though it's not designed to be read in strict order, the
book is organized into chapters with related subject matter. If you want to find
a specific subject, the table of contents is still a good place to start. In
addition, several of the chapters contain shaded boxes.
These are like small tables of contents on a particular subject, which might be
even more limited than the scope of the chapter itself. Use the
Index when you're trying to find a specific
piece of information instead of a general group of articles about a topic.
Like a News Magazine
This book has short articles.
Most show a problem and a solution in one page or less. The articles are
numbered within each chapter. Not all articles are "how-to" tips. Some articles
have background information and concepts.
Like a Hypertext Database
Each article doesn't define all the concepts and words used.
Instead, it gives you "links" that let you get more information
if you need it. It's easy to get more
information when you need it, but just skip the link if you don't.
Unix Power Tools uses two kinds of links: those
in a sentence and those in the margin. For examples, see the pages before this
Preface titled How to Use This Book.
Programs on the Web
The book describes scripts and freely available programs that
are available on the web site. An article about a program or file that's on the
web site will have a globe icon next to it, like this. To get one of these
programs, use our visit the web site:
-
http://www.oreilly.com/catalog/upt3/
About Unix Versions
There are lots of similarities between different versions of
Unix. But it's almost impossible to write a book that covers every detail of
every version correctly. Where we know there might be big differences or
problems, we'll print a note in the text. Other places, we're forced to use
"weasel words" like "Some versions of XXX will do...," without telling you
exactly which versions. When you see those
weasel words, what can you do?
-
If the command or feature won't destroy anything when it
doesn't work, try it! For instance, don't experiment with
rm, the command that removes files. But
cat, a command that shows files, probably
won't hurt anything if some feature doesn't work with your version.
-
Look at the online manual
or check your vendor's latest printed
manuals. However, even these can be wrong. For instance, your system
administrator may have installed a local version of a command that works
differently but not updated the online documentation. Be careful with
"generic" manuals, the kind you buy at a bookstore; there are lots of versions
of Unix, and the manual may not match your version closely enough.
-
Ask your system administrator or another "guru" for help
before you use a command that might be dangerous.
Cross-References
If a cross-reference is to a single word for example, a
command name like this: tar the cross
reference is probably to an article that introduces that command. Cross
references to phrases like this: from a parent
process to child process are to an article that explains more about the
concept or problem printed in gray.
Cross references don't necessarily give a complete list of
all articles about a topic. We've tried to pick one or a few articles that give
the best information. For a more complete list, use the Index.
What's New in the Third Edition
There have been some big changes in Unix since we wrote the
first edition in the early 1990s, and there's been a surprising number of
changes since the second edition, released in the late 1990s. Well over half of
the articles have been revised, and we've expanded our coverage of the so-called
small Unix flavors: Linux, FreeBSD, Mac OS X's Darwin, and so on.
A major change to this edition was the addition of several
new topics relevant to today's connected world, including protecting your
machine from attack and several articles related to Internet protocols. We've
also added chapters with coverage of two of the more popular languages used in
Unix: Perl and Python.
Typefaces and Other Conventions
- Italic
-
Is used for the names of all Unix utilities, switches,
directories, and filenames and to emphasize new terms and concepts when they
are first introduced. It's also used in programs and examples to explain
what's happening or what's been left out at the . . . marks.
- Bold
-
Is used occasionally within text to make words easy to find
just like movie stars' names in the People section of your local newspaper.
- Constant width
-
Is used for sample code fragments and examples. A reference
in text to a word or item used in an example or code fragment is also shown in
constant width font.
Constant width bold
-
Is used in examples to show commands or text that would be
typed in literally by the user.
- Constant width italic,
bold italic
-
Are used in code fragments and examples to show variables
for which a context-specific substitution should be made. (The variable
filename, for example, would be replaced by some actual filename.)
- function(n)
-
Is a reference to a manual page in Section
n of the Unix programmer's manual. For
example, getopt(3) refers to a page called
getopt in Section 3.
- %
-
Is the C-shell prompt.
- $
-
Is the Bourne-shell prompt.
- :-)
-
Is a "smiley face" that means "don't take this seriously."
The idea started on
Usenet and spread.
- & . . .
-
Stands for text (usually computer output) that's been
omitted for clarity or to save space.
- CTRL
-
Starts a control character. To create CTRL-d, for example,
hold down the "control" key and press the "d" key. Control characters are not
case sensitive; "d" refers to both the upper- and lowercase letter. The
notation ^D also means CTRL-d. Also, you'll sometimes see the key
sequence in bold (for example, CTRL-d is used when we want to make it clear
exactly what you should type.
- Λ?
-
Is used in some examples to represent a space charaΛ?cter.
- TAB
-
Is used in some examples to represent a TAB character.
The Authors
This book is the effort of several authors who have
contributed to one edition or another since the first edition was released. Much
of the material for the first and second edition came from three authors: Jerry
Peek, Tim O'Reilly, and Mike Loukides. Their work is still present, though
edited for current times. This third edition brought in four new authors, who
edited the previous material, in addition to contributing new articles: Shelley
Powers, Steven Champeon, Deborah Hooker, and Joe Johnston.
In addition, we also had several other authors contribute to
all three editions either people who originally posted a good tip to Usenet,
authors of Nutshell Handbooks who let us take material from their books, or
authors of software packages who let us take a few paragraphs from README files
or other documentation.
Here's a list of authors and their initials:
- AD
|
- Angus Duggan
|
- JIK
|
- Jonathan I. Kamens
|
- AF
|
- AEleen Frisch
|
- JM
|
- Jeff Moskow
|
- AN
|
- Adrian Nye
|
- JP
|
- Jerry Peek
|
- BA
|
- Brandon S. Allbery
|
- JJ
|
- Joe Johnston
|
- BB
|
- Bruce Barnett
|
- JS
|
- John Strang
|
- BR
|
- Bill Rosenblatt
|
- LK
|
- Lar Kaufman
|
- CT
|
- Chris Torek
|
- LL
|
- Linda Lamb
|
- DC
|
- Debra Cameron
|
- LM
|
- Linda Mui
|
- DD
|
- Dale Dougherty
|
- LW
|
- Larry Wall
|
- DG
|
- Daniel Gilly
|
- MAL
|
- Maarten Litmaath
|
- DH
|
- Dave Hitz
|
- ML
|
- Mike Loukides
|
- DJPH
|
- Deborah Hooker
|
- MS
|
- Mike Stansbery
|
- DL
|
- Don Libes
|
- RS
|
- Randal Schwartz
|
- DR
|
- Daniel Romike
|
- SP
|
- Shelley Powers
|
- DS
|
- Daniel Smith
|
- SG
|
- Simson Garfinkel
|
- EK
|
- Eileen Kramer
|
- SC
|
- Steve Champeon
|
- EP
|
- Eric Pearce
|
- SW
|
- Sun Wu
|
- GS
|
- Gene Spafford
|
- TC
|
- Tom Christiansen
|
- GU
|
- Greg Ubben
|
- TOR
|
- Tim O'Reilly
|
- HS
|
- Henry Spencer
|
- UM
|
- Udi Manber
|
The Fine Print
Where we show an article from an author on Usenet, that
person may not have thought of the idea originally, but may just be passing on
something he or she learned. We attribute everything we can.
Request for Comments
Please tell us about any errors you find in this book or ways
you think it could be improved. Our U.S. mail address, phone numbers, and
electronic mail address are as follows:
O'Reilly & Associates, Inc.
1005 Gravenstein Highway North
Sebastopol, CA 95472
(800) 998-9938 (in the United States or Canada)
(707) 829-0515 (international/local)
(707) 829-0104 (fax)
<EMAIL_ADDRESS>(email)
Acknowledgments for the First Edition
This book wouldn't exist without Ron Petrusha. As the
technical book buyer at Golden-Lee, a major book distributor, he discovered us
soon after we started publishing Nutshell Handbooks in the mid-1980s. He was one
of our early boosters, and we owed him one. So when he became an editor at
Bantam (whose computer-book publishing operations were later acquired by Random
House), we took him seriously when he started asking if there was anything we
could do together.
At first nothing seemed to fit, since by that time we were
doing pretty well as a publisher. We needed to find something that we could do
together that might sell better than something that either company might do
alone. Eventually, Ron suggested that we copublish a Unix book for Bantam's
"Power Tools" series. This made sense for both of us. It gave Bantam access to
our Unix expertise and reputation, and it gave us a chance to learn from Bantam
about the mass market bookstore trade, as well as build on their successful
"Power Tools" series.
But what would the book contain? There were two features of
Bantam's original DOS Power Tools that we
decided to emulate: its in-depth treatment of under-documented system features
and its large collection of freely available scripts and utilities. However, we
didn't want to write yet another book that duplicated the format of many others
on the market, in which chapters on each of the major Unix tools follow one
another in predictable succession. Our goal was certainly to provide essential
technical information on Unix utilities, but more importantly, to show how the
utilities can be combined and used to solve common (and uncommon) problems.
Similarly, because we were weary of the multitude of endless
tutorial books about Unix utilities, we wanted to keep the tone brisk and to the
point. The solution I came up with, a kind of "hypertext in print," actually
owes a lot to Dale Dougherty. Dale has been working for several years on
hypertext and online information delivery, and I was trying to get him to work
with me on this project. So I tried to imagine the kind of book that he might
like to create. (We have a kind of friendly rivalry, in which we try to leapfrog
each other with ideas for new and better books!) Dale's involvement never went
far beyond the early brainstorming stage, but the book still bears his indirect
stamp. In some of the first books he wrote for me, he introduced the idea that
sidebars asides that illuminate and expand on the topic under discussion
could be used effectively in a technical book. Well, Dale, here's a book that's
nothing but sidebars!
Dale, Mike Loukides, and I worked out the basic outline for
the book in a week or two of brainstorming and mail exchanges. We thought we
could throw it together pretty quickly by mining many of our existing books for
the tips and tricks buried in them. Unfortunately, none of us was ever able to
find enough time, and the book looked to be dying a slow death. (Mike was the
only one who got any writing done.) Steve Talbott rescued the project by
insisting that it was just too good an idea to let go; he recruited Jerry Peek,
who had just joined the company as a writer and Unix consultant/tools developer
for our production department.
Production lost the resulting tug of war, and Jerry plunged
in. Jerry has forgotten more Unix tips and tricks than Mike, Dale, or I ever
knew; he fleshed out our outline and spent a solid year writing and collecting
the bulk of the book. I sat back in amazement and delight as Jerry made my ideas
take shape. Finally, though, Jerry had had enough. The book was just too big,
and he'd never signed on to do it all alone! (It was about 1,000 pages at that
point, and only half done.) Jerry, Mike, and I spent a week locked up in our
conference room, refining the outline, writing and cutting articles, and
generally trying to make Jerry feel a little less like Sisyphus.
From that point on, Jerry continued to carry the ball, but
not quite alone, with Mike and I playing "tag team," writing and editing to fill
in gaps. I'm especially grateful to Mike for pitching in, since he had many
other books to edit and this was supposed to be "my" project. I am continually
amazed by the breadth of Mike's knowledge and his knack for putting important
concepts in perspective.
Toward the end of the project, Linda Mui finished up another
book she was working on and joined the project, documenting many of the freely
available utilities that we'd planned to include but hadn't gotten around to
writing up. Linda, you really saved us at the end!
Thanks also to all the other authors, who allowed us to use
(and sometimes abuse!) their material. In particular, we're grateful to Bruce
Barnett, who let us use so much of what he's written, even though we haven't yet
published his book, and Chris Torek, who let us use many of the gems he's posted
to the Net over the years. (Chris didn't keep copies of most of these articles;
they were saved and sent in by Usenet readers, including Dan Duval, Kurt J. Lidl,
and Jarkko Hietaniemi.)
Jonathan Kamens and Tom Christiansen not only contributed
articles but read parts of the book with learned and critical eyes. They saved
us from many a "power goof." If we'd been able to give them enough time to read
the whole thing, we wouldn't have to issue the standard disclaimer that any
errors that remain are our own. H. Milton Peek provided technical review and
proofreading. Four sharp-eyed Usenet readers helped with debugging: Casper Dik
of the University of Amsterdam, Byron Ratzikis of Network Appliance Corporation,
Dave Barr of the Population Research Institute, and Duncan Sinclair.
In addition to all the acknowledged contributors, there are
many unacknowledged ones people who have posted questions or answers to the
Net over the years and who have helped to build the rich texture of the Unix
culture that we've tried to reflect in this book. Jerry also singles out one
major contributor to his own mastery of Unix. He says: "Daniel Romike of
Tektronix, Inc. (who wrote
Section 28.5 and
Section 30.8 in the early 1980s, by the way) led the first Unix workshop I
attended. He took the time to answer a ton of questions as I taught myself Unix
in the early 1980s. I'm sure some of the insights and neat tricks that I thought
I've figured out myself actually came from Dan instead."
James Revell and Bryan Buus scoured "the Net" for useful and
interesting free software that we weren't aware of. Bryan also compiled most of
the software he collected so we could try it out and gradually winnow down the
list.
Thanks also to all of the authors of the software packages we
wrote about and included on the CD! Without their efforts, we wouldn't have had
anything to write about; without their generosity in making their software free
in the first place, we wouldn't be able to distribute hundreds of megabytes of
software for the price of a book.
Jeff Moskow of Ready-to-Run Software solved the problem we
had been putting off to the end: that of packaging up all the software for the
original disk, porting it to the major Unix platforms, and making it easy to
install. This was a much bigger job than we'd anticipated, and we could never
have done it without Jeff and the RTR staff. We might have been able to
distribute source code and binaries for a few platforms, but without their
porting expertise, we could never have ported all these programs to every
supported platform. Eric Pearce worked with RTR to pre-master the software for
CD-ROM duplication, wrote the installation instructions, and made sure that
everything came together at the end! (Eric, thanks for pitching in at the last
minute. You were right that there were a lot of details that might fall through
the cracks.)
Edie Freedman worked with us to design the format of the book
quite an achievement considering everything we wanted the format to do! She
met the challenge of presenting thousands of inline cross-references without
distracting the reader or creating a visual monstrosity. What she created is as
attractive as it is useful a real breakthrough in technical book design, and
one that we plan to use again and again!
Lenny Muellner was given the frightful task of implementing
all of our ideas in troff no mean feat, and
one that added to his store of grey hair.
Eileen Kramer was the copyeditor, proofreader, and critic who
made sure that everything came together. For a thousand-plus page book with
multiple authors, it's hard to imagine just how much work that was.
Ellie Cutler wrote the index; Chris Reilley created the
illustrations. Additional administrative support was provided by Bonnie Hyland,
Donna Woonteiler, and Jane Appleyard.
Tim O'Reilly
Acknowledgments for the Second Edition
After teaching myself about Unix for the past 15 years, I'm
off to graduate school in Computer Science. Frank Willison, O'Reilly's
Editor-in-Chief, fit this project into the summer between leaving my position at
ORA and starting school. Frank didn't just give me something to do in the
summer: the royalties should help to pay for my coursework. (So, buy this book
and support a student! ;-)) Gigi Estabrook edited this edition and
fielded my zillions of questions along the way. Many thanks to Gigi, Frank, and
ORA's Production staff. Clairemarie Fisher O'Leary and Nancy Wolfe Kotary shared
the jobs of production editor and project manager. Madeleine Newell and Kismet
McDonough-Chan provided production support. Sheryl Avruch, Nicole Gipson Arigo,
and Danny Marcus provided quality control checks. Lenny Muellner provided
extensive troff assistance and technical
support. Chris Reilley created the technical illustrations.
When time was short, I got expert advice from Arnold Robbins,
the maintainer of the GNU gawk utility, and
coauthor of O'Reilly's sed & awk, Second
Edition. He reviewed parts of the book and gave me thorough comments.
I'd also like to thank all the readers who took a moment to
send us comments and corrections. I read every message, and the ideas in them
made a big difference in this second edition. Three peoples' comments were
extensive enough to mention specially. Ted Timar spotted problems that showed
his deep knowledge of Unix. I'm glad he still found the book useful enough to
read it and to spot goofs in some of our hairier tips. Andrew T. Young sent
two long email messages: one a few years ago and another after I contacted him.
He caught plenty of techno-goofs and also sent fixes for them. Andy doesn't know
just Unix: his background in English helped to sharpen a few rough spots in our
folksy writing style. Finally, Greg Ubben sent a 15-page (!) email message that
took me most of a week to work through. When I tracked him down, three years
after writing his message, he was even more helpful. Greg wrote enough to make
into a small book and, in fact, agreed to write a few new articles, too. He's
an expert in sed and regular expressions (and
Unix) who taught me a lot in our month of email messages back and forth. I
deeply appreciate all that he's given to this book's readers.
Jerry Peek,
<EMAIL_ADDRESS>Acknowledgments for the Third Edition
Though much of this book is new material or has been newly
updated for changes in Unix, there is a core that remains from previous
editions. The fact that this material has continued to be fresh, useful, and
relevant through the years is a testament to the abilities technical and
writing of the original authors. This includes Tim O'Reilly and Jerry Peek,
among others previously mentioned, who contributed to past additions. We, the
authors of this current edition, thank you. We had a number of terrific
reviewers comment on this version of the text. We appreciate the work of Dave
Carrano, Chris DiBona, Schuyler Erle, Jeff Kawski, Werner Klauser, Adam Langley,
Arnold Robbins, Jaron Rubenstein, Kevin Schmidt, Jay Sekora, Joe Sloan, Nat
Torkington, and Jay Ts.
In addition, I would like to thank those who contribute their
time and efforts on Unix systems, particularly the open source versions of Unix
such as FreeBSD, Linux, and now Darwin.
Shelley Powers
I'd just like to thank you all for inviting me to contribute
to a book that helped me learn Unix a long time ago. It's nice to be able to
give something back, given how much the book helped me back in 1994 when I was
just another Unix newbie.
Steven Champeon
Thank you, Amy and Joel, for the input and review and just
for putting up with me through it, and Jasper, for being my strength when I
needed it.
Deborah Hooker