Next-Gen system container management
INDEX | Category: LINUX | Series: CONTAINERS PART 1 |
Disclaimer!
The why?
So, lately i’ve been deeply involved with *nix’s namespaces and cgrpups. I am trying to build ths tool called: tent, which basically at high level is a chroot clone but with modern flare. So it has some automatic mounting points, user directora manupilations etc…
Along the way, i got to learn quite a lot how linux handles containers, especially with liblxc. The way i was going with it was by trying to understand how Canonical’s lxd
(lxc
) works. And as usual, in order to remeber what i learned, i decided to document it, so here it goes.
Linux Containers
Linux containers (LXC) is an operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a single control host (LXC host). It does not provide a virtual machine, but rather provides a virtual environment that has its own CPU, memory, block I/O, network, etc. space and the resource control mechanism. This is provided by the namespaces and cgroups features in the Linux kernel on the LXC host. It is similar to a chroot, but offers much more isolation.
LXD/LXC
LXD (pronounced “Lex-Dee”) is a system container manager build on top of Linux Containers (LXC). The goal of LXD is to provide an experience similar to a virtual machine but through containerization rather than hardware virtualization. Compared to Docker for delivering applications, LXD offers nearly full operating-system functionality with additional features such as snapshots, live migrations, and storage management. It also offers a simple REST API, network and storage management, project views and easy clustering to dozen of hosts…
LXD is an open source system container manager, developed by the team behind LXC and supportet by Canonical, written in Go and that’s been around for quite some time now. Containers are created from images with prebuilt images available for most Linux distributions. Multiple hosts can easily be clustered together to form one large virtual host, exposing the exact same API as a single host would. Storage pools and networks can also be created and managed through LXD and resources can be segmented into projects.
LXC vs siblings
Compared to other container syblings, LXD offers plenty of admin-appealing features, such as:
- Secured using unprivileged containers, resource restrictions and much more
- Scalable up to thousands of compute nodes
- Intuitive with a straightforward CLI
- Numerous images from which to pull
- Cross-host container and image transfer support
- Advanced resource control for cpu, memory, network I/O, block I/O, disk usage and kernel resources
- Device passthrough for USB, GPU, unix character and block devices, NICs, disks and paths
- Network management for bridge creation and configuration, cross-host tunnels, and more
- Storage management support for multiple storage backends, storage pools and storage volumes
- Uses LXC through liblxc and its Go binding to create and manage the containers
Installation
The first thing to be done is the installation of LXD. As this tool is found in the standard repositories, it can be installed from the package manager of your distro:
arch: pacman -S lxd
debian: apt install lxd
fedora: dnf install lxd
alpine: apk add lxd
Once installed, you’ll need to add your user to the lxd group and log-out and back in:
Configutation
To go through the initial LXD configuration, run:
You will then be asked a number of questions. Unless you need a specific configuration for your setup, the defaults should work just fine. Those questions are:
Would you like to use LXD clustering? (yes/no) [default=no]: no
Do you want to configure a new storage pool? (yes/no) [default=yes]: yes
Name of the new storage pool [default=default]: default
Name of the storage backend to use (btrfs, ceph, dir, lvm, zfs) [default=zfs]: zfs
Create a new ZFS pool? (yes/no) [default=yes]: yes
Would you like to use an existing block device? (yes/no) [default=no]: no
Size in GB of the new loop device (1GB minimum) [default=15GB]: 15GB
Would you like to connect to a MAAS server? (yes/no) [default=no]: no
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like LXD to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
Verify that the lxc client is talking to the LXD daemon: lxc list
and you’lle get an output like:
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
Managing containers
The syntax is as follows to create your first container:
The easiest way is to use a built-in remote image server.
You can get a list of built-in image servers with:
and the output will be:
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| NAME | URL | PROTOCOL | AUTH TYPE | PUBLIC | STATIC |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| images | https://images.linuxcontainers.org | simplestreams | none | YES | NO |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| local (default) | unix:// | lxd | file access | NO | YES |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu | https://cloud-images.ubuntu.com/releases | simplestreams | none | YES | YES |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu-daily | https://cloud-images.ubuntu.com/daily | simplestreams | none | YES | YES |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
The one we are interested is the first column, images
, so to list all available images for various Linux distro, notice the :
at the end of the command:
Each image contains a basic operating system (for example a linux distribution) and some other LXD-related information.
Let’s say you want to launch a container using the Ubuntu image:
Where CONTAINER_NAME
is the name assigned to your container. To get the name of the newly created contaners, list all containers with lxc list
.
If it already exists then we start it with:
Accessing the Container
If you want to access the shell of your new container (so you can start developing with it), issue the command:
We’re not just limited using lxc exec as a method of dropping into a container, however. Should we need to run a one-off command, we can do so directly:
Do note, however, that chaining commands with &&
or |
operands does take some small tweaking of the command, using double quotes "
.
Stop or delete the Container
To stop the container, issue the commands:
Or if you want to delete:
Managing virtual machines
Starting with LXD 4.0, virtual machines are suported natively,and thanks to a built-in agent, they behave almost just like containers 🤯 🤯.
Creating a VM
Creating a VM is as simple as the following command. It is the standard lxc launch
command, with the addition of the --vm
option to create a virtual machine (instead of a system container). We specify the default profile (whichever base configuration you use in your LXD installation). Depending on your computer’s specifications, it takes a few seconds to launch the container/vm, and then less than 10 seconds for the VM to boot up and receive the IP address from your network.
or
now, running lxc list
will output:
+--------+---------+------+------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+--------+---------+------+------+-----------------+-----------+
| ubuntu | RUNNING | | | VIRTUAL-MACHINE | 0 |
+--------+---------+------+------+-----------------+-----------+
Accessing the VM
You can see your VM boot with:
(detach with ctrl+a-q)
Once booted, VMs with the agent built-in will also respond to:
By installing the LXD agent inside the LXD VM, we can run the usual LXD commands such as lxc exec
, lxc file
, etc. Here is how to get a shell, either using the built-in alias lxc shell
, or lxc exec
to get a shell with the non-root account of the container images.
Example: WINDOWS 10
It is possible to build Windows images for LXD, but this process is currently very very manual and involves either a bunch of custom raw.qemu flags to get a temporary graphical console to perform the install or using a separate QEMU process to prepare the image.
The basic steps are:
- Grab a [Windows ISO image](https://www.microsoft.com/en-ca/software-download/windows10ISO) from Microsoft
- Grab the [latest virtio drivers](https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.173-9/virtio-win.iso) for Windows
- Create an empty VM with beefier CPU/RAM and SecureBoot disabled: `lxc init win10 --empty --vm -c security.secureboot=false -c limits.cpu=4 -c limits.memory=4GB`
- Grow its root disk to a reasonable size: `lxc config device override win10 root size=20GB`
- Enable temporary graphical output and pass the install and drivers media: `echo -n '-device virtio-vga -vnc :1 -drive file=/path/to/Win10_1909_English_x64.iso,index=0,media=cdrom,if=ide -drive file=/path/to/virtio-win-0.1.173.iso,index=1,media=cdrom,if=ide' | lxc config set win10 raw.qemu - `
- Start the VM: `lxc start win10`
- Immediately attach using `lxc console win10` and hit ESC until presented with a boot menu
- Select Boot Manager and then the `QM00001` drive. Then hit ENTER a few times to answer an invisible boot prompt.
- Use a VNC client to connect to `localhost:1`, you’ll see the installer boot
- During the installation, load the virtio-scsi driver to access the hard disk
- Once installed and configured to allow RDP/SSH access, you can remove the various workarounds with: `lxc config unset win10 raw.qemu`
- Boot the system to confirm it all works, install all the other drivers from the virtio drive, then run the Windows sysprep tool
- Finally publish your VM as an image with: `lxc publish win10 --alias win10`
Files
When configuring any container, being able to run commands is only part of the picture. We also need to be able to update and add files. While we could do this by dropping into a shell on our container and working, but LXD also offers us some more streamlined options for working with files on our containers.
Direct file acces
We can directly edit (using the default $EDITOR
) a specific file in the container:
Or completely delete file with:
Copying/Moving from and to container
Copy from an instance to host by using pull
Pull a folder with:
Copy from host to instance by using push
Push a folder with:
Profiles
When we create a container, where and how it’s created and configured is dependent upon its profile. Up until now, we’ve been using the default profile to assign the storage pool, network, and other information to our containers:
But we can create specialized profiles to suit our needs. Let’s start by creating a YFS pool:
From here, we can use the default profile as a base by creating our zfs-test storage pool via the copy command:
Let’s now update our new profile to use the appropriate storage pool:
root:
path: /
pool: zfs-test
type: disk
Now let’s launch a new container based on this profile:
We’re not just limited to the existing configurations presented in the default profile, however. We can make a quick NEW_PROFILE
profile in only a few commands:
To add the profile to our container, we can just use:
Let’s also remove the existing default profile from our INSTANCE_NAME
container:
As you may have guessed by now, many of the lxc profile commands follow the same conventions as storage and config. Remember, should we ever need a refresher, we can view a full list with:
Snapshots
Rarely when working with containers are we working with one-off, individualized configurations. One of the benefits of containers, after all, is the ability to launch many of them very quickly. We don’t want these to be stock, unconfigured containers, however. And while we can enroll our containers in a configuration management system, we are more likely to launch our containers based on preconfigured images. These preconfigured images, in turn, are generally created from a configured container. Snapshots can use any naming convention you desire, but is good practise to keep to version numbers. So let’s say this snapshot is version 1.0 of our container:
To confirm, we can use the lxc info command, which lists all snapshots associated with the defined container:
We can now create a duplicate of this container with lxc copy:
We could also use cp in place of copy. Notice, however, when we view our current list of containers, that our new INSTANCE_NAME_BACKUP container exists, but is not automatically started. We can remedy this with a lxc start:
Finally, we can remove any unneeded snapshots with the lxc delete command, the same command we use to remove containers themselves. Instead of listing only the container name, however, we’ll list the name and version in the same format we used when copying the snapshot:
Other features
You can attach nic and disk devices to them, grow their CPU & RAM, take snapshots, move them between hosts, use them in a cluster, publish them as images, … VM should pretty much all behave as you would expect a LXD instance, just a bit slower because you’re dealing with a full virtual machine.


Implements empirical Bayes approaches to genotype polyploids from next generation sequencing data while accounting for allelic bias, overdispersion, and sequencing error. The main function is flexdog(), which allows the specification of many different genotype distributions. An experimental function that takes into account varying levels of relatedness is implemented in mupdog(). Also provided are functions to simulate genotypes, rgeno(), and read-counts, rflexdog(), as well as functions to calculate oracle genotyping error rates, oracle_mis(), and correlation with the true genotypes, oracle_cor(). These latter two functions are useful for read depth calculations. Run browseVignettes(package = "updog") in R for example usage. See also Gerard et al. (2018) <doi:10.1101/281550> for details on the implemented methods.
| Version: | 1.0.1 |
| Depends: | R (≥ 3.4.0) |
| Imports: | Rcpp (≥ 0.12.16), RcppArmadillo, assertthat, foreach, doParallel, ggplot2, ggthemes, stringr |
| LinkingTo: | Rcpp, RcppArmadillo |
| Suggests: | covr, testthat, SuppDists, Rmpfr, CVXR, knitr, rmarkdown, tidyverse |
| Published: | 2018-07-27 |
| Author: | David Gerard [aut, cre] |
| Maintainer: | David Gerard <gerard.1787 at gmail.com> |
| BugReports: | http://github.com/dcgerard/updog/issues |
| License: | GPL-3 |
| NeedsCompilation: | yes |
| Citation: | updog citation info |
| Materials: | README NEWS |
| CRAN checks: | updog results |
| Reference manual: | updog.pdf |
| Vignettes: |
Oracle Calculations Parallelized Genotyping using Updog Simulate Next-Generation Sequencing Data Example Use of Updog |
| Package source: | updog_1.0.1.tar.gz |
| Windows binaries: | r-devel: updog_1.0.1.zip, r-release: updog_1.0.1.zip, r-oldrel: updog_1.0.1.zip |
| OS X binaries: | r-release: updog_1.0.1.tgz, r-oldrel: updog_1.0.1.tgz |
| Old sources: | updog archive |
Please use the canonical form https://CRAN.R-project.org/package=updog to link to this page.


| Class | Description |
|---|---|
| AuthToken |
The AuthToken class creates an authentication token using
the
ClientLogin API.
|
| CaptchaInformation |
Represents CAPTCHA information retruned by the client login API.
|
| DfpHttpSender |
DfpHttpSender acts as a custom HTTP transport layer which
allows for access to the MessageContext before it is processed. |
| DfpService |
DfpService provides a class which defines a service
that can be instantiated through reflection. |
| DfpService.V201208 |
The service class for v201208.
|
| DfpService.V201211 |
The service class for v201211.
|
| DfpService.V201302 |
The service class for v201302.
|
| DfpService.V201306 |
The service class for v201306.
|
| DfpService.V201308 |
The service class for v201308.
|
| DfpService.V201311 |
The service class for v201311.
|
| DfpServiceFactory |
The
DfpServiceFactory generates service stubs with an
Axis EngineConfiguration which maps the HTTP transport to the
class DfpHttpSender . |
| DfpServiceLogger |
The class
DfpServiceLogger allows logging of
outgoing and incoming SOAP XML (DfpService )
messages as API calls are executed. |
| DfpUser |
The
DfpUser class contains all needed information to
authenticate a user as well as generate a valid service for that user. |
| DfpVersion |
For each active version of the DoubleClick For Publishers API, there should exist
a sub-class which implements the abstract class.
|
| ServiceAccountant |
The
ServiceAccountant class acts as an accountant for metrics
extracted from the request and response from a SOAP call. |
| ServiceAccountantManager |
The
ServiceAccountantManager manages all instances of ServiceAccountant by reference to the javax.xml.rpc.Service
hashcode. |
| ServiceAccountantRecord |
A record for purposes of metric accounting.
|
| Exception | Description |
|---|---|
| AuthTokenException |
An exception thrown when the ClientLogin API was unable to successfully
generate a token.
|


Comparatives are formed by adding -er to most adjectives that have one syllable (small/smaller) and adjectives that end in -y (pretty/prettier). If an adjective has two or more syllables, usually "more" or "less" comes before it (more comfortable, more peaceful).
Superlatives are formed by adding -est to most adjectives that have one syllable (small/smallest) and adjectives that end in -y (pretty/prettiest). If an adjective has two or more syllables, usually "most" or "least" comes before it (most comfortable, most peaceful).
There are also some adjectives that use irregular forms for comparatives and superlatives. Irregular means they don't follow the predictable pattern.
| Adjective | Comparative | Superlative |
|---|---|---|
| good | better | best |
| bad | worse | worst |
| little | less | least |
| much | more | most |
| far | farther/further | farthest/furthest |
Other common forms:
| as + adjective + as | He is as intelligent/nice/tall as her. (They are the same.) |
| the same (as) | I have the same shirt as you./ Our shirts are the same. |
| similar to | He looks similar to his father. |
| different from | He looks nothing like his brother. They are different from each other. |
| More | Less | Same |
|---|---|---|
|
Russia is colder than Mexico. For most adjectives that are only one or two syllables in length, add -er + than. |
Mexico is not as cold as Russia. Mexico is less cold than Russia. |
Russian history is just as fascinating as Mexican history. |
|
Earth is bigger than Mars. For some adjectives, double the final consonant and add -er + than. |
Mars is not as big as Earth. Mars is less big than Earth. |
According to some people, Mars is just as beautiful as Earth. |
|
The girl is hungrier than the boy. For adjectives ending in y, change the y to an i and add -er + than. |
The boy is not as hungry as the girl. The boy is less hungry than the girl. |
The boy is just as hungry as the girl |
|
The sirloin tip roast is more expensive than the chicken breast For most adjectives that are more than two syllables in length, add more before and keep the base form of the adjective + than. |
The chicken breast is not as expensive as the sirloin tip roast. The chicken breast is less expensive than the sirloin top roast. |
The chicken breast is just as delicious as the sirloin tip roast. |
| Most | Others Compared to the Most | The Most Compared to Others | Least |
|---|---|---|---|
|
Antarctica is the coldest place. For most adjectives that are only one or two syllables in length, add the + –est. |
Other places are not as cold as Antarctica. |
Antarctica is the coldest of all places. Antarctica is the coldest of any place. |
Antarctica is the least populated place. |
|
Jupiter is the biggest planet. For some adjectives, double the final consonant and add the + -est. |
Other planets are not as big as Jupiter. |
Jupiter is the biggest of all places. Jupiter is the biggest of any place. |
Mars has the least colorful landscape. |
|
Algebra is the easiest subject. For adjectives ending in y, change the y to an i and add the + -est. |
Other subjects are not as easy as Algebra. |
Algebra is the easiest of all subjects. Algebra is the easiest of any subject. |
Algebra is the least difficult subject. |
|
Top Sirloin is the most expensive meat. For most adjectives that are more than two syllables in length, add the + most before and keep the base form of the adjective. |
Other meats are not as expensive as Top Sirloin. |
Top Sirloin is the most expensive of all meats. Top Sirloin is the most expensive of any meat. |
Chicken breast is the least expensive. |
Comparatives compare TWO people, places, or things. Use the word "than". (He is taller than her.) (This book is more interesting than that book.)
Superlatives compare a group of THREE or more people, places, or things. Superlatives often have the word "the" before them. (She was the best student in the whole class.)
We can use comparative and superlative forms in simple sentences in much the same way as we use the standard adjective forms.
When asking for a comparison of two, it is common to ask the question and list the two alternatives. The list usually comes at the end of the question, but it can sometimes come at the front.
Examples:
Question: Who is smarter, John or Peter?
Answer: John is smarter.
Question: Out of mathematics and economics, which is harder?
Answer: Mathematics is harder.
Question: Which is farther from Honolulu, Los Angeles or San Francisco?
Answer: Los Angeles is a little further.
Question: Is it better to go to a store or to shop on-line?
Answer: I prefer going to a store.
When asking for a comparison among three or more alternatives, it is common to ask the question and state the range of choice. Again, the statement of the range of alternatives usually comes at the end of the question, but it can come at the front.
Examples:
Question: Who is the oldest of your children?
Answer: Tina is (the oldest).
Question: Who is the greatest basketball player of all time?
Answer: Michael Jordan is.
Question: Out of all the EU countries, which has the lowest birth rate?
Answer: Germany does.
Question: Which of your children lives farthest away from you?
Answer: My oldest son lives the furthest away, in London.
We can use comparatives and superlatives with adverbs as well.
Examples:
John speaks very softly.
Mary speaks more softly (than John).
Frances speaks the most softly (of all three of them).
He thought sadly about the wreckage of his car.
He thought more sadly about the wreckage of his home.
But he thought most sadly about the wreckage of his marriage.
Sometimes you speak more kindly to strangers than to your friends.
For more information on comparative and superlative forms and practice exercises, go to https://www.ego4u.com/en/cram-up/grammar/adjectives-adverbs


| Name: MR.MOHAMMED | Reg No: 05-9SMHC17 |
|---|---|
| Age / Sex:27/M | Contact No:<PHONE_NUMBER> |
| Marital Status:SINGLE | Date:7/9/17 |
| Occupation: SPARE PARTS INCHARGE | Dr. PJFP |
Address: NATIVE OF MYSORE IN BALORE 1 YR |
DIAGNOSIS- |
ITCHING FROM 3 DAYS.
L: GROINS LOWER ABDOMEN, RT. SHIN AREA.
S: ITCHING.
H/O OF ONE MONTH BACK SAME PROBLM IN MYSORE.
M:< EVENING,
COLD WATER.
FEELS ITCHY MORE AFTER SCRATCHING. NO BLEEDING. AT TIMES NIGHT FEELS MORE ITCHY. BUT EVENING IS MORE.
BRIGHT LIGHTS CANT TAKE AND GETS HEADACHE ON LEFT ABOVE EYEBROW.
PAST H/O:
FAMILY H/
MOM 45: DM,
DAD 52: GASTRITIS
Y.SIS 25: BP
P.GFATHER : DIED OF CANCER. 20 YEARS
PERSONAL H/O:
DIET: MIXED. LESS APP. EATS TWICE A DAY.
DESIRE: MASALA
AVERSION/ AGG: BRINJAL
THIRST : 5 LTS / DAY. THIRSTY
URINE: 4/2
MOTION: REG FREE
SWEAT: NS
LIKES COLD WEATHER ONLY. HOT WEATHER FEELS SKIN BLACKISH LIKES COLD
FAN MUST
DEPENDS ON WEATHER WANTS COVERINGS.
RX:
NAT MUR 200 – 5 DOSES WITH WATER B/FOOD MORNING,
APIS 200 – 4 -4-4-4 B/FOOD. WITH WATER.
CARC 1M – 1 DOSE STAT.
CALC SULPH 6X – 4-4-4-4.
15/9/17: nt feeling better. No reddish color but itching is much. >cold water.
Rx: ars.iod30
Merc sol 30
Ac flour 30
Lach 30.
Medo 1m 2 doses.


| GIF and PNG Images for Math Symbols | ||
| Mirrors > Home > GIF and PNG Images for Math Symbols |
| Contents of this page |
Works in ALL browsers I have tried!
—Ian G. Clark, Ph.D. |
Over time these images may be updated, including a change of their names.
|  |
renders the symbol "". The ALT attribute provides accessibility for people using text-only browsers such as w3m, links, and lynx. The TITLE attribute is optional and causes a "tool tip" with the symbol name to appear when hovering over the symbol. The WIDTH and HEIGHT (which is always 19) are optional but make the page load faster with fewer browser jumps while rendering the page. You can see the WIDTH and HEIGHT in the tool tip that appears when you pass your cursor over the symbol chart above, or you can click on the symbol and inspect its properties in your browser. If you use WIDTH and HEIGHT make sure you get them right, for otherwise the symbol will be distorted.
The STYLE="margin-bottom:-4px" attribute shifts the symbol vertically so that it lines up with the text. On a page with many symbols, we would ordinarily put this attribute in a CSS directive, as you can see from the source code of this web page. If you use this CSS directive, be aware that images that aren't math symbols must have the attribute STYLE="margin-bottom:0px" to override the CSS; again, you can see that done on this web page.
When you finish editing your web page, verify that its syntax is correct with W3C's on-line HTML Validator [external]. Just because it "looks right" on your particular browser version doesn't mean it will display correctly on others.
There are a couple of ways you can mix the symbol images with the rest of your page. One way is to use special symbols mixed with the normal browser text. Here is an example of a formula from mathematical logic, "For any x, there exists a y not equal to x." The HTML source code looks like this (for simplicity, we omitted the optional but recommended WIDTH and HEIGHT attributes):
|
xy yx |
Spaces or line breaks between the bracketed tags affect the display. The only such space in this example is between the two y's. Here is the result (on a colored background to demonstrate the transparency):
| This is what it looks like on a browser with the font size set to 12: | This is what it looks like on a browser with the font size set to 10: | This is what it looks like on your browser right now: | |
|
Depending on your browser settings, this may look OK. But if you set the font size to something other than the usual default 12 point size (that the symbols were designed for), the characters may not align properly.
Often a better approach is to use images for the formula's letters as well. This will guarantee that all browsers will see the same thing. Also, the letters then become "math italic," which is different from regular italic. Here is the HTML source code:
|
  |
and here is the result, independent of your browser font setting:
It is possible to build two-dimensional formulas by using table cells to determine the placement of symbols. For example,
|

 |
looks like this:
|
In the code above, the parentheses and integral sign were stretched using the HEIGHT attribute, while preserving the original WIDTH.
|
The square root of 2 is irrational: ` sqrt 2 e/ QQ ` , where ` QQ ` is the set of rational numbers. |
then programatically generate HTML code that displays like this:
| The square root of 2 is irrational: , where is the set of rational numbers. |
The overall idea is very simple. You define mnemonics for your math symbols, such as sqrt for the square root surd. When the program encounters these mnemonics enclosed in backticks (`), it replaces the mnemonics with the HTML code you specified for them.
I will show you how to do this by exploiting the web-page generation feature built into a program called "Metamath". Note that Metamath was designed for a completely different purpose, and you must follow the instructions below carefully to coax it into accomplishing our goal. (In principle a specialized program could be written that would be simpler to use, but I'll leave that task for someone else if they are interested. The method described here is one I use myself, and I thought it would be useful to share it.)
First, download and compile the Metamath program according to the instructions on the Metamath Home Page. For Windows, you can download the pre-compiled version metamath.exe (0.5MB) instead. Then create an input file for Metamath with a text editor (Notepad or equivalent), using the following example to guide you. You may want to copy and test this example to make sure everything works. The lines you will customize are shown in red. The 5 lines shown in black should not be modified!
|
$( DO NOT EDIT ANY LINE WITH A "$" ON IT! $) $c dummy $. $( LIST YOUR MNEMONICS BELOW. $) $c QQ 2 e/ sqrt $. $( DEFINE YOUR MNEMONICS BELOW. $) $( $t htmldef "QQ" as ""; htmldef "2" as ""; htmldef "e/" as "  "; htmldef "sqrt" as ""; htmldef "dummy" as ""; $) $( TYPE YOUR TEXT TO BE TRANSLATED BELOW. $) $( The square root of 2 is irrational: ` sqrt 2 e/ QQ ` , where ` QQ ` is the set of rational numbers. $) dummy $a dummy $. |
Call this file "dummy.mm". Here is an explanation of this file. Read it carefully.
|
htmldef "QQ" as ""; |
Once you build up a collection of the "htmldef" lines you need, you can use them over and over. The Metamath master database file set.mm (7MB) has inside of it many "htmldef" lines pre-defined for you that you can clip out if you like. It also references some custom GIFs with names beginning with an underscore; you probably don't care about those but if you want them they can be found in the Metamath Proof Explorer download.
Next run the Metamath program, giving it the commands "read dummy.mm", "show statement dummy /html", and "exit". (On Windows, put the files "metamath.exe" and "dummy.mm" in the same folder. Click on "metamath.exe" to run the program.) The session will look like this, with the commands you will type shown in red:
|
Metamath - Version 0.07x 25-Apr-05
Type HELP for help, EXIT to exit. MM> read dummy.mm Reading source file "dummy.mm"... 12 lines (653 characters) were read from "dummy.mm". The source has 3 statements; 1 are $a and 0 are $p. No errors were found. However, proofs were not checked. Type VERIFY PROOFS * if you want to check them. MM> show statement dummy /html Creating HTML file "dummy.html"... 5 typesetting statements were read from "dummy.mm". MM> exit |
The Metamath program thinks it is generating a web page for the Metamath site, and there is a lot of extra stuff in its output file "dummy.html" that is irrelevant for our purposes. To extract the relevant HTML code, open "dummy.html" in your text editor, search for the string "Description: ", and copy the text after it and up to the string "	". Paste it into your HTML file. For our example, the code section you will copy looks like this:
|
The square root of 2 is irrational:   , where  is the set of rational numbers. |
When pasted into your HTML file, it will display like this:
| The square root of 2 is irrational: , where is the set of rational numbers. |
Each individual symbol image provided here is an n x 19 pixel transparent GIF (or PNG). The width n is the width of the symbol plus (usually) a pixel of padding on each side. If we call the bottom row 1, then roman letter "I" is aligned between rows 5 and 15 inclusive. These numbers were chosen because they will properly align the symbols with a 12-point browser fonts Times New Roman and Arial.
Here is how to use The GIMP to extract a symbol from the symbol chart and create a transparent GIF from it. We show the exact process down to each individual keystroke and mouse click, so even if you have never used The GIMP before you should have no trouble with this. We assume you have saved the symbol chart into your directory as the file symbols.gif (in the Mozilla browser, right-click on it and select "Save Image...").
Note: the "webify" plug-in must be installed in The GIMP. This happens automatically with a full install of The GIMP on RedHat 7.2.
The following instructions are for the Gnome window manager. They should be similar in KDE. You can also type "gimp" at the bash command prompt in a terminal emulation window.
Done! You have now created a transparent GIF for your symbol.
But this isn't just a problem with Internet Explorer. The font designers for the fonts used by non-Microsoft browsers seem to have gone off on their own inconsistent path rather than just avail themselves of the LaTeX fonts that already exist and are in the public domain. Here is a comparison of a few of the symbols used on this site, showing the LaTeX version, our bitmaps, and the Mozilla/Firefox version. To me, the Unicode characters just don't look as good as the LaTeX font or even the bitmaps on this page. The character sizes, shapes, and weights seem awkward and oddly inconsistent: the vee and wedge are too dark, the angle brackets too light, the subset-equals squished, the aleph too small, the script P missing (requiring the wreath product P as a compromise), no special italic for math letters, intersection and union not symmetrical, the dot in less-than-dot not centered, etc. The unsmoothed version looks rough, unpleasant, and amateurish, as if it was done quickly with no regard to its appearance or even legibility: the angle brackets are asymmetrical with a spurious dot on the left bracket, and the big union is missing a leg, making it literally unreadable! With non-ClearType smoothing, some characters are smoothed and others aren't in a seemingly arbitrary way.
I don't think it's just my eccentric taste. Based on the web logs, even Mozilla/Firefox users use the GIF version of the pages almost exclusively, in spite of their much slower speed and jagged bitmaps. Of course this site defaults to the GIF version, but it also makes it easy to try the Unicode version, with a link on every proof page. Almost without exception, anyone who tries the Unicode version will look at a page or two, then go back to the GIF version.
Another effort to standardize mathematics in browsers is the MathML [external] project, but currently it requires special browsers or plug-ins.
Therefore to make the site universally accessible, the only option at the present time seems to be the use of GIF (or PNG) images for the symbols. But I could not find suitable images on the web - there always seemed to be problems ranging from copyright issues to blurry fonts clipped from PDF screen displays. For example, compare our symbols to the ones in this collection [external], which has 181 symbols [external] with oddly inconsistent quality and font weights. The collection is accompanied by a restrictive license [external] that will set you back $50 [external] for commercial use. (Hopefully you'll find our public domain collection to be a better deal!)
So, I created this bi-level (1-bit monochrome) interpretation of the standard fonts that come with the LaTeX typesetting program, to fit within the limited resolution of a 12-point browser font. I made them public domain to eliminate any concern with copyright issues.
Ultimately I hope that Unicode and MathML will make these obsolete. But unfortunately it might be several years before all standard mathematical symbols are universally available by default in web browsers.
To convert from the GIF to the PNG format, I used the program gif2png [external] with the command "gif2png -Os xxx.gif".
All images and text displayed on this web page are placed in the public domain by the author, per the Creative Commons Public Domain Dedication [external].
| For search engines | This page was last revised on
2-Oct-2015. Your comments are welcome: Norman Megill |
W3C HTML validation [external] |


Reducing Food Waste in the Restaurant Industry
WINTER 2016 — I took part in the Accenture Innovation Challenge and the World Wildlife Fund in a team of 3 to conceptualize our vision of using mobile technology to reduce food waste in the restaurant industry.
In order to combat food waste, we developed Bamboo. Bamboo is a mobile platform in which restaurants can notify local customers of leftover and excess prepared food that they are willing to sell at a discounted rate at the end of a business day.
2016 Accenture Innovation Challenge Semi-Finalist.
01 Approach
Problem Statement
World Wildlife Fund supports the USDA and EPA’s Food Waste Reduction Goal, which aims to reduce food waste by 50% by 2030.
Food waste has environmental and humanitarian implications, hurting both the environment and people. Food production is responsible for 1/3 of all greenhouse gas emissions. World needs at least 50% more food by 2050 to feed 9 billion people. 84% of food waste is disposed of. [Source: 2014 BSR paper prepared for the Food Waste Reduction Alliance]
Idea Generation
1. Inbound Logistics
Source locally
Source reduction via improved analytics
Encourage more efficient shipping practices (vacuum-seal)
2. Food Preparation
Avoid over-trimming meats/vegetables
Develop storage system to maximize shelf life
3. Post-Preparation
Smaller portions to reduce waste
Repurpose left-overs (jams, canned goods)
Sell/donate extra prepared food
Out of the 3 potential areas of food production, post-production offers the biggest opportunity for innovation.
02 Bamboo
Idea
Bamboo is an app that allows restaurants to alert customers in the area that they have excess food available at a discount at the end of a business day
Business Model
College students have the buying power to sustain Bamboo: We would like to launch Bamboo at selected colleges aimed at reducing Boston’s annual 4 million pounds of food waste.
Stakeholders
1. WWF
Receives 2.5% transaction fee
Increases publicity/funding surrounding outreach efforts
2. Restaurants
Recover revenue lost from excess production
Increase business and publicity
3. Customers
Aimed at college students
Have a new option for affordable food
4. Global Community
Reduces food waste
Moves towards maximizing food utility
03 Result
Role
I conceptualized the idea for a mobile service to reduce food waste, and designed the mockups and vision for the app.
Result
We pitched the idea to Accenture consultants, and qualified for Round 3, the Semi-Final round, out of 200+ teams. We were invited to interview for an Consulting Analyst internship following the challenge.
Team
Kevin Shum, Ellie Klose & Xiaoxi Wang
We also collaborated with various Accenture consultants who offered advice and guidance throughout the challenge.
Technologies
Case Study
Market Research
UI/UX


Aria2 Integration Settings
Host
Secure Connection
Port
Secret
Leave it blank if is not specified.
Enable Capture
Capture File Size
If enabled, downloading a file with size larger than this value is automatically captured by Aria2.
File Size
K
M
G
T
File Types
These files are
always
handled by Aria2.
e.g rar,zip
Sites Whitelist
Downloads from these site are
always
handled by Aria2,
regardless of size or type
.
e.g microsoft.*, *google*
Sites Blacklist
Downloads from these site are
always
ignored by Aria2,
regardless of size or type
.
e.g microsoft.*, *google*
Protocols Whitelist
Only with these protocols are handled by Aria2
based on size or type
.
e.g http,https,ftp
Save


ThisTom
Home
About
Projects
Contact
Hi, I'm Tom.
I do stuff.
Hello there!
This page is currently undergoing construction.
This is a link!
Hello there!
This page is currently undergoing construction.
Hello there!
This page is currently undergoing construction.


{% extends 'blog/base.html' %} {% load staticfiles %} {% block content %}
We're experiencing some internal server problem. Please try again later.


AUTO-Py Suite
Open
Overview
(current)
Edit
Generate
Templates
Merge
Manage Extracts
Merge Files
Extract Locker
Streamline
Overview
Share
Export


Problem:
Lets assume you need to model a school. You would need to list all the classes first. Then each class would have different sections(like Class X-A, class X-B.. etc). Under each class you would have n students. This is a simple 3 level tree. So when a user selects class X, then he would be displayed with all the sections in it. And when he selects a section we would list all the students in it. The below is how it would look and the same is available under examples folder


Genome completeness before and after genome finishing. (A) Genome of dolphin rhabdovirus (DRV). (B) Genome of red fox fecal rhabdovirus (RFFRV). (C) Genome of python nidovirus (PNV). Indicated is contig size after initial genome assembly (red) and after genome finishing (blue). Striped line at the 5′ end of PNV indicates putative unresolved 5′ end.


Participate in UNIQUE experiences!

       Help raise funds for AMAZING causes!
   Featured Experiences
See more snippets like this online store item at
$64.99
This is a short description. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
$74.99
This is a short description. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
$84.99
This is a short description. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
$94.99
This is a short description. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
$94.99
This is a short description. Lorem ipsum dolor sit amet, consectetur adipiscing elit.


|
||||||||||
| PREV NEXT | FRAMES NO FRAMES |
| Packages that use StrBuilder | |
|---|---|
| org.apache.commons.lang.text |
Provides classes for handling and manipulating text, partly as an extension to java.text . |
| Uses of StrBuilder in org.apache.commons.lang.text |
|---|
| Methods in org.apache.commons.lang.text that return StrBuilder | |
|---|---|
StrBuilder |
StrBuilder.append(boolean value)
Appends a boolean value to the string builder. |
StrBuilder |
StrBuilder.append(char ch)
Appends a char value to the string builder. |
StrBuilder |
StrBuilder.append(char[] chars)
Appends a char array to the string builder. |
StrBuilder |
StrBuilder.append(char[] chars,
int startIndex,
int length)
Appends a char array to the string builder. |
StrBuilder |
StrBuilder.append(double value)
Appends a double value to the string builder using String.valueOf . |
StrBuilder |
StrBuilder.append(float value)
Appends a float value to the string builder using String.valueOf . |
StrBuilder |
StrBuilder.append(int value)
Appends an int value to the string builder using String.valueOf . |
StrBuilder |
StrBuilder.append(long value)
Appends a long value to the string builder using String.valueOf . |
StrBuilder |
StrBuilder.append(Object obj)
Appends an object to this string builder. |
StrBuilder |
StrBuilder.append(StrBuilder str)
Appends another string builder to this string builder. |
StrBuilder |
StrBuilder.append(StrBuilder str,
int startIndex,
int length)
Appends part of a string builder to this string builder. |
StrBuilder |
StrBuilder.append(String str)
Appends a string to this string builder. |
StrBuilder |
StrBuilder.append(StringBuffer str)
Appends a string buffer to this string builder. |
StrBuilder |
StrBuilder.append(StringBuffer str,
int startIndex,
int length)
Appends part of a string buffer to this string builder. |
StrBuilder |
StrBuilder.append(String str,
int startIndex,
int length)
Appends part of a string to this string builder. |
StrBuilder |
StrBuilder.appendAll(Collection coll)
Appends each item in a collection to the builder without any separators. |
StrBuilder |
StrBuilder.appendAll(Iterator it)
Appends each item in an iterator to the builder without any separators. |
StrBuilder |
StrBuilder.appendAll(Object[] array)
Appends each item in an array to the builder without any separators. |
StrBuilder |
StrBuilder.appendFixedWidthPadLeft(int value,
int width,
char padChar)
Appends an object to the builder padding on the left to a fixed width. |
StrBuilder |
StrBuilder.appendFixedWidthPadLeft(Object obj,
int width,
char padChar)
Appends an object to the builder padding on the left to a fixed width. |
StrBuilder |
StrBuilder.appendFixedWidthPadRight(int value,
int width,
char padChar)
Appends an object to the builder padding on the right to a fixed length. |
StrBuilder |
StrBuilder.appendFixedWidthPadRight(Object obj,
int width,
char padChar)
Appends an object to the builder padding on the right to a fixed length. |
StrBuilder |
StrBuilder.appendln(boolean value)
Appends a boolean value followed by a new line to the string builder. |
StrBuilder |
StrBuilder.appendln(char ch)
Appends a char value followed by a new line to the string builder. |
StrBuilder |
StrBuilder.appendln(char[] chars)
Appends a char array followed by a new line to the string builder. |
StrBuilder |
StrBuilder.appendln(char[] chars,
int startIndex,
int length)
Appends a char array followed by a new line to the string builder. |
StrBuilder |
StrBuilder.appendln(double value)
Appends a double value followed by a new line to the string builder using String.valueOf . |
StrBuilder |
StrBuilder.appendln(float value)
Appends a float value followed by a new line to the string builder using String.valueOf . |
StrBuilder |
StrBuilder.appendln(int value)
Appends an int value followed by a new line to the string builder using String.valueOf . |
StrBuilder |
StrBuilder.appendln(long value)
Appends a long value followed by a new line to the string builder using String.valueOf . |
StrBuilder |
StrBuilder.appendln(Object obj)
Appends an object followed by a new line to this string builder. |
StrBuilder |
StrBuilder.appendln(StrBuilder str)
Appends another string builder followed by a new line to this string builder. |
StrBuilder |
StrBuilder.appendln(StrBuilder str,
int startIndex,
int length)
Appends part of a string builder followed by a new line to this string builder. |
StrBuilder |
StrBuilder.appendln(String str)
Appends a string followed by a new line to this string builder. |
StrBuilder |
StrBuilder.appendln(StringBuffer str)
Appends a string buffer followed by a new line to this string builder. |
StrBuilder |
StrBuilder.appendln(StringBuffer str,
int startIndex,
int length)
Appends part of a string buffer followed by a new line to this string builder. |
StrBuilder |
StrBuilder.appendln(String str,
int startIndex,
int length)
Appends part of a string followed by a new line to this string builder. |
StrBuilder |
StrBuilder.appendNewLine()
Appends the new line string to this string builder. |
StrBuilder |
StrBuilder.appendNull()
Appends the text representing null to this string builder. |
StrBuilder |
StrBuilder.appendPadding(int length,
char padChar)
Appends the pad character to the builder the specified number of times. |
StrBuilder |
StrBuilder.appendSeparator(char separator)
Appends a separator if the builder is currently non-empty. |
StrBuilder |
StrBuilder.appendSeparator(char standard,
char defaultIfEmpty)
Append one of both separators to the builder If the builder is currently empty it will append the defaultIfEmpty-separator Otherwise it will append the standard-separator The separator is appended using append(char) . |
StrBuilder |
StrBuilder.appendSeparator(char separator,
int loopIndex)
Appends a separator to the builder if the loop index is greater than zero. |
StrBuilder |
StrBuilder.appendSeparator(String separator)
Appends a separator if the builder is currently non-empty. |
StrBuilder |
StrBuilder.appendSeparator(String separator,
int loopIndex)
Appends a separator to the builder if the loop index is greater than zero. |
StrBuilder |
StrBuilder.appendSeparator(String standard,
String defaultIfEmpty)
Appends one of both separators to the StrBuilder. |
StrBuilder |
StrBuilder.appendWithSeparators(Collection coll,
String separator)
Appends a collection placing separators between each value, but not before the first or after the last. |
StrBuilder |
StrBuilder.appendWithSeparators(Iterator it,
String separator)
Appends an iterator placing separators between each value, but not before the first or after the last. |
StrBuilder |
StrBuilder.appendWithSeparators(Object[] array,
String separator)
Appends an array placing separators between each value, but not before the first or after the last. |
StrBuilder |
StrBuilder.clear()
Clears the string builder (convenience Collections API style method). |
StrBuilder |
StrBuilder.delete(int startIndex,
int endIndex)
Deletes the characters between the two specified indices. |
StrBuilder |
StrBuilder.deleteAll(char ch)
Deletes the character wherever it occurs in the builder. |
StrBuilder |
StrBuilder.deleteAll(String str)
Deletes the string wherever it occurs in the builder. |
StrBuilder |
StrBuilder.deleteAll(StrMatcher matcher)
Deletes all parts of the builder that the matcher matches. |
StrBuilder |
StrBuilder.deleteCharAt(int index)
Deletes the character at the specified index. |
StrBuilder |
StrBuilder.deleteFirst(char ch)
Deletes the character wherever it occurs in the builder. |
StrBuilder |
StrBuilder.deleteFirst(String str)
Deletes the string wherever it occurs in the builder. |
StrBuilder |
StrBuilder.deleteFirst(StrMatcher matcher)
Deletes the first match within the builder using the specified matcher. |
StrBuilder |
StrBuilder.ensureCapacity(int capacity)
Checks the capacity and ensures that it is at least the size specified. |
StrBuilder |
StrBuilder.insert(int index,
boolean value)
Inserts the value into this builder. |
StrBuilder |
StrBuilder.insert(int index,
char value)
Inserts the value into this builder. |
StrBuilder |
StrBuilder.insert(int index,
char[] chars)
Inserts the character array into this builder. |
StrBuilder |
StrBuilder.insert(int index,
char[] chars,
int offset,
int length)
Inserts part of the character array into this builder. |
StrBuilder |
StrBuilder.insert(int index,
double value)
Inserts the value into this builder. |
StrBuilder |
StrBuilder.insert(int index,
float value)
Inserts the value into this builder. |
StrBuilder |
StrBuilder.insert(int index,
int value)
Inserts the value into this builder. |
StrBuilder |
StrBuilder.insert(int index,
long value)
Inserts the value into this builder. |
StrBuilder |
StrBuilder.insert(int index,
Object obj)
Inserts the string representation of an object into this builder. |
StrBuilder |
StrBuilder.insert(int index,
String str)
Inserts the string into this builder. |
StrBuilder |
StrBuilder.minimizeCapacity()
Minimizes the capacity to the actual length of the string. |
StrBuilder |
StrBuilder.replace(int startIndex,
int endIndex,
String replaceStr)
Replaces a portion of the string builder with another string. |
StrBuilder |
StrBuilder.replace(StrMatcher matcher,
String replaceStr,
int startIndex,
int endIndex,
int replaceCount)
Advanced search and replaces within the builder using a matcher. |
StrBuilder |
StrBuilder.replaceAll(char search,
char replace)
Replaces the search character with the replace character throughout the builder. |
StrBuilder |
StrBuilder.replaceAll(String searchStr,
String replaceStr)
Replaces the search string with the replace string throughout the builder. |
StrBuilder |
StrBuilder.replaceAll(StrMatcher matcher,
String replaceStr)
Replaces all matches within the builder with the replace string. |
StrBuilder |
StrBuilder.replaceFirst(char search,
char replace)
Replaces the first instance of the search character with the replace character in the builder. |
StrBuilder |
StrBuilder.replaceFirst(String searchStr,
String replaceStr)
Replaces the first instance of the search string with the replace string. |
StrBuilder |
StrBuilder.replaceFirst(StrMatcher matcher,
String replaceStr)
Replaces the first match within the builder with the replace string. |
StrBuilder |
StrBuilder.reverse()
Reverses the string builder placing each character in the opposite index. |
StrBuilder |
StrBuilder.setCharAt(int index,
char ch)
Sets the character at the specified index. |
StrBuilder |
StrBuilder.setLength(int length)
Updates the length of the builder by either dropping the last characters or adding filler of unicode zero. |
StrBuilder |
StrBuilder.setNewLineText(String newLine)
Sets the text to be appended when a new line is added. |
StrBuilder |
StrBuilder.setNullText(String nullText)
Sets the text to be appended when null is added. |
StrBuilder |
StrBuilder.trim()
Trims the builder by removing characters less than or equal to a space from the beginning and end. |
| Methods in org.apache.commons.lang.text with parameters of type StrBuilder | |
|---|---|
StrBuilder |
StrBuilder.append(StrBuilder str)
Appends another string builder to this string builder. |
StrBuilder |
StrBuilder.append(StrBuilder str,
int startIndex,
int length)
Appends part of a string builder to this string builder. |
StrBuilder |
StrBuilder.appendln(StrBuilder str)
Appends another string builder followed by a new line to this string builder. |
StrBuilder |
StrBuilder.appendln(StrBuilder str,
int startIndex,
int length)
Appends part of a string builder followed by a new line to this string builder. |
boolean |
StrBuilder.equals(StrBuilder other)
Checks the contents of this builder against another to see if they contain the same character content. |
boolean |
StrBuilder.equalsIgnoreCase(StrBuilder other)
Checks the contents of this builder against another to see if they contain the same character content ignoring case. |
String |
StrSubstitutor.replace(StrBuilder source)
Replaces all the occurrences of variables with their matching values from the resolver using the given source builder as a template. |
String |
StrSubstitutor.replace(StrBuilder source,
int offset,
int length)
Replaces all the occurrences of variables with their matching values from the resolver using the given source builder as a template. |
boolean |
StrSubstitutor.replaceIn(StrBuilder source)
Replaces all the occurrences of variables within the given source builder with their matching values from the resolver. |
boolean |
StrSubstitutor.replaceIn(StrBuilder source,
int offset,
int length)
Replaces all the occurrences of variables within the given source builder with their matching values from the resolver. |
protected String |
StrSubstitutor.resolveVariable(String variableName,
StrBuilder buf,
int startPos,
int endPos)
Internal method that resolves the value of a variable. |
protected boolean |
StrSubstitutor.substitute(StrBuilder buf,
int offset,
int length)
Internal method that substitutes the variables. |
|
||||||||||
| PREV NEXT | FRAMES NO FRAMES |


Anime¶
-
class
Sakurajima.models.base_models.
Anime
(data_dict: dict, network, api_url: str)¶ Wraps all the relevant data for an anime like anime_id (called as detail_id by AniWatch backend), title, airing date etc. Use the get_episodes method to get a list of available episodes
-
add_recommendation
(recommended_anime_id: int)¶ Adds the user’s reccomendation for the anime.
- Parameters
recommended_anime_id (int) – The aniwatch anime ID of the anime you want to recommend.
- Returns
[description]
- Return type
[type]
-
get_chronicle
(page=1)¶ Gets the chronicle for the anime, a chronicle tracks the user’s watch history for a particular anime.
- Parameters
page (int, optional) – The page of the chronicle you want to get, defaults to 1
- Returns
A list of ChronicleEntry objects where each object has details like date.
- Return type
list[ChronicleEntry]
-
get_complete_object
()¶ Gets the current anime object but with complete attributes. Sometimes, the Anime object that is returned by the API does not contain values for all the attributes that the Anime object has. Use this method to get an object with the maximum amount of data. This method is almost never required but is for edge cases
- Returns
An Anime object with values for as many attributes as possible.
- Return type
-
get_dict
()¶ Gets the JSON response in the form of a dictionary that was used to initialize the object.
- Returns
A dictionary of the JSON response
- Return type
dict
-
get_episodes
()¶ Gets a list of all available episodes of the anime.
- Returns
An EpisodeList object, an EpisodeList object is very similar to a normal list. You can access specific indexes using the “[]” syntax. In addition to this, an EpisodeList has few convinience methods like
get_episode_by_number(episode_number)
which returns an episode with the provided episode number.- Return type
EpisodeList
-
get_media
()¶ Gets the anime’s associated media from aniwatch.me
- Returns
A Media object that has attributes like
opening
,osts
.- Return type
Media
-
get_recommendations
()¶ Gets the recommendations for the anime.
- Returns
A list of RecommendationEntry objects where each object represents a single recommendation.
- Return type
list[RecommendationEntry]
-
get_relations
()¶ Gets the relation of the anime.
- Returns
A Relation object that contains all the details of the relation.
- Return type
Relation
-
mark_as_completed
()¶ Marks the anime as “completed” on the user’s aniwatch anime list.
- Returns
True if the operation is successful, False if an error occurs.
- Return type
bool
-
mark_as_dropped
()¶ Marks the anime as “dropped” on the user’s aniwatch anime list.
- Returns
True if the operation is successful, False if an error occurs.
- Return type
bool
-
mark_as_on_hold
()¶ Marks the anime as “on hold” on the user’s aniwatch anime list.
- Returns
True if the operation is successful, False if an error occurs.
- Return type
bool
-
mark_as_plan_to_watch
()¶ Marks the anime as “plan to watch” on the user’s aniwatch anime list.
- Returns
True if the operation is successful, False if an error occurs.
- Return type
bool
-
mark_as_watching
()¶ Marks the anime as “watching” on the user’s aniwatch anime list
- Returns
True if the operation is successful, False if an error occurs.
- Return type
bool
-
rate
(rating: int)¶ Set the user’s rating for the anime on aniwatch.
- Parameters
rating (int) – The rating you want to set, should be between 1 to 10. Rate 0 to remove the user’s rating for the anime
- Returns
True if the operation is successful, False if an error occurs.
- Return type
bool
-
remove_from_list
()¶ Removes the anime from the user’s aniwatch anime list.
- Returns
True if the operation is successful, False if an error occurs.
- Return type
bool
-


In spacecraft operations, The Launch and Early Orbit phase (LEOP) is one of the most critical phases of a mission. Spacecraft operations engineers take control of the satellite after it separates from the launch vehicle up to the time when the satellite is safely positioned in its final orbit. During this period, operations staff works 24 hours a day to activate, monitor and control the various subsystems of the satellite, including the deployment of any satellite appendages (antennas, solar array, reflector, etc.), and undertake critical orbit and attitude control manoeuvres. For geostationary satellites, the launch vehicle typically carries the spacecraft to Geostationary Transfer Orbit, or GTO. From this elliptical orbit, the LEOP generally includes a sequence of apogee engine firings to reach the circular geostationary orbit. In spacecraft operations, The Launch and Early Orbit phase (LEOP) is one of the most critical phases of a mission. Spacecraft operations engineers take control of the satellite after it separates from the launch vehicle up to the time when the satellite is safely positioned in its final orbit. During this period, operations staff works 24 hours a day to activate, monitor and control the various subsystems of the satellite, including the deployment of any satellite appendages (antennas, solar array, reflector, etc.), and undertake critical orbit and attitude control manoeuvres. For geostationary satellites, the launch vehicle typically carries the spacecraft to Geostationary Transfer Orbit, or GTO. From this elliptical orbit, the LEOP generally includes a sequence of apogee engine firings to reach the circular geostationary orbit. In spacecraft operations, The Launch and Early Orbit phase (LEOP) is one of the most critical phases of a mission. Spacecraft operations engineers take control of the satellite after it separates from the launch vehicle up to the time when the satellite is safely positioned in its final orbit. During this period, operations staff works 24 hours a day to activate, monitor and control the various subsystems of the satellite, including the deployment of any satellite appendages (antennas, solar array, reflector, etc.), and undertake critical orbit and attitude control manoeuvres. For geostationary satellites, the launch vehicle typically carries the spacecraft to Geostationary Transfer Orbit, or GTO. From this elliptical orbit, the LEOP generally includes a sequence of apogee engine firings to reach the circular geostationary orbit. In spacecraft operations, The Launch and Early Orbit phase (LEOP) is one of the most critical phases of a mission. Spacecraft operations engineers take control of the satellite after it separates from the launch vehicle up to the time when the satellite is safely positioned in its final orbit. During this period, operations staff works 24 hours a day to activate, monitor and control the various subsystems of the satellite, including the deployment of any satellite appendages (antennas, solar array, reflector, etc.), and undertake critical orbit and attitude control manoeuvres. For geostationary satellites, the launch vehicle typically carries the spacecraft to Geostationary Transfer Orbit, or GTO. From this elliptical orbit, the LEOP generally includes a sequence of apogee engine firings to reach the circular geostationary orbit. In spacecraft operations, The Launch and Early Orbit phase (LEOP) is one of the most critical phases of a mission. Spacecraft operations engineers take control of the satellite after it separates from the launch vehicle up to the time when the satellite is safely positioned in its final orbit. During this period, operations staff works 24 hours a day to activate, monitor and control the various subsystems of the satellite, including the deployment of any satellite appendages (antennas, solar array, reflector, etc.), and undertake critical orbit and attitude control manoeuvres. For geostationary satellites, the launch vehicle typically carries the spacecraft to Geostationary Transfer Orbit, or GTO. From this elliptical orbit, the LEOP generally includes a sequence of apogee engine firings to reach the circular geostationary orbit. In spacecraft operations, The Launch and Early Orbit phase (LEOP) is one of the most critical phases of a mission. Spacecraft operations engineers take control of the satellite after it separates from the launch vehicle up to the time when the satellite is safely positioned in its final orbit. During this period, operations staff works 24 hours a day to activate, monitor and control the various subsystems of the satellite, including the deployment of any satellite appendages (antennas, solar array, reflector, etc.), and undertake critical orbit and attitude control manoeuvres. For geostationary satellites, the launch vehicle typically carries the spacecraft to Geostationary Transfer Orbit, or GTO. From this elliptical orbit, the LEOP generally includes a sequence of apogee engine firings to reach the circular geostationary orbit.


Class: ActiveRecord::Encryption::KeyGenerator
| Relationships & Source Files | |
| Inherits: | Object |
| Defined in: | activerecord/lib/active_record/encryption/key_generator.rb |
Overview
Utility for generating and deriving random keys.
Class Method Summary
Instance Attribute Summary
- #hash_digest_class readonly
Instance Method Summary
-
#derive_key_from(password, length: key_length)
Derives a key from the given password.
-
#generate_random_hex_key(length: key_length)
Returns a random key in hexadecimal format.
-
#generate_random_key(length: key_length)
Returns a random key.
- #key_derivation_salt private
- #key_length private
Constructor Details
.new(hash_digest_class: ActiveRecord::Encryption.config.hash_digest_class) ⇒ KeyGenerator
# File 'activerecord/lib/active_record/encryption/key_generator.rb', line 11
def initialize(hash_digest_class: ActiveRecord::Encryption.config.hash_digest_class) @hash_digest_class = hash_digest_class end
Instance Attribute Details
#hash_digest_class (readonly)
[ GitHub ]# File 'activerecord/lib/active_record/encryption/key_generator.rb', line 9
attr_reader :hash_digest_class
Instance Method Details
#derive_key_from(password, length: key_length)
Derives a key from the given password. The key will have a size in bytes of :length
(configured Cipher
‘s length by default)
The generated key will be salted with the value of ActiveRecord::Encryption.key_derivation_salt
# File 'activerecord/lib/active_record/encryption/key_generator.rb', line 38
def derive_key_from(password, length: key_length) ActiveSupport::KeyGenerator.new(password, hash_digest_class: hash_digest_class) .generate_key(key_derivation_salt, length) end
#generate_random_hex_key(length: key_length)
Returns a random key in hexadecimal format. The key will have a size in bytes of :length
(configured Cipher
‘s length by default)
Hexadecimal format is handy for representing keys as printable text. To maximize the space of characters used, it is good practice including not printable characters. Hexadecimal format ensures that generated keys are representable with plain text
To convert back to the original string with the desired length:
[ value ].pack("H*")
# File 'activerecord/lib/active_record/encryption/key_generator.rb', line 30
def generate_random_hex_key(length: key_length) generate_random_key(length: length).unpack("H*")[0] end
#generate_random_key(length: key_length)
Returns a random key. The key will have a size in bytes of :length
(configured Cipher
‘s length by default)
# File 'activerecord/lib/active_record/encryption/key_generator.rb', line 16
def generate_random_key(length: key_length) SecureRandom.random_bytes(length) end
#key_derivation_salt (private)
[ GitHub ]# File 'activerecord/lib/active_record/encryption/key_generator.rb', line 44
def key_derivation_salt @key_derivation_salt ||= ActiveRecord::Encryption.config.key_derivation_salt end
#key_length (private)
[ GitHub ]# File 'activerecord/lib/active_record/encryption/key_generator.rb', line 48
def key_length @key_length ||= ActiveRecord::Encryption.cipher.key_length end


Welcome to openDACs! Our aim to provide simple-to-follow instructions on how to build reliable precision electronics using off-the-shelf components. This site is not intended for electronics experts; rather, we hope to assist nonexperts working in materials science, condensed matter physics, and other fields in meeting their electronics needs at an economical price–with respect both to time and money. The main platforms we will use are the versatile Arduino series of microcontrollers and evaluation boards available from Analog Devices.


Changelog
Will welcome if anyone wants to backfill prior changelogs.
1.8.8 - 2016-08-10
#47 Support for gulp-data
#46 Fix undefined template local "filename"
#46 Document
file
and
filename
locals.
#45 Started CHANGELOG.md


Use this option to view the key with which the active document was encrypted or decrypted. If no key is available for the active document (and if the icon is grayed out), this menu option will not be offered. If there is a key available, it is located in the internal key store.
The key will be displayed in the dialog Key Display, assuming it is a one-line key, as is the case in all the encryption algorithms in CrypTool, with the exception of the Hill encryption algorithm.
If you click the button Copy Key in the dialog Key Display the actually shown / used key is stored into the internal key storage.
Shortcuts
| Toolbar: |


The Games Forum
provides the same rules we use and also cites the source The
Official World Encyclopedia of Sports and Games, Diagram Visual
Information Ltd 1979.
Roger Eastman implemented a Pig dice game with optimal play as
part of his iDice 2.0 dice game compilation for the Macintosh.
Michael Plasmeier implemented a Pig dice game PLAYPIG for TI-83+/84+ graphing calculators.
Justin Boyan and Andrew Moore applied reinforcement learning to a
variation of Pig where a roll of 1 scores one point. In this acyclic
variation, you can't find yourself in the same situation twice. Such
a game can be solved with dynamic programming (computing the best
moves from the end of the game backwards to the beginning). This Pig
variation description is available as HTML,PDF and PostScript
(see reference below).
Durango Bill's
partial analysis of two dice Pig. - A hold value analysis is
performed assuming one is trying to reach 100 in N turns.
However, the analysis doesn't provide a policy for what N should be.
"In practice (in a game) you as a player have to determine how many
turns you should use to try to get to 100." This analysis is more
complex than Knizia's odds-based analysis. Although there are regions
in our Pig state space where the optimal player is clearly trying to
reach the goal in one or two turns, the true optimal policy is more
complex than "win in N turns" over much of the state space. We
conjecture that the solution for this version of Pig would be
qualitatively similar to our own, and thus more complex than this
analysis indicates.
SKUNK
is like NRICH's Pig variant but limited to 5 turns with the maximum
score winning. This page presents classroom teaching activities.
Note: We have also solved this variation of Pig. The number of dice
rolls is irrelevant to optimal play or score maximization.
In R.C. Bell's Discovering Dice and Dominoes (ISBN
0-85263-532-X, 1980), a variant is described where double 1's simply
ends the turn with no score. That is double 1's are like rolling a
single 1.
Kyle Bradshaw's Pigalator applet
features complex four-player tournament rules with doubles doubled as in Frey's variant (but with re-roll required), double 1's eliminating all points and doubling game stakes, and all points eliminated if 100 is reached exactly.
Two Dice, Doubles are Bad: Doubles lose the turn total and end the turn. All other rolls accumulate their sum to the turn total. This variant, "Piggy", appears in "Kids Travel: A Backseat Survival Kit" by the editors of Klutz Press, 1994, p. 11.
Two Pig-Dice:Pass the
Pigs® Milton Bradley / Winning Moves game played with "pig" dice.
Also known as PigMania®
(David Moffat Enterprises, 1977). Rolls are scored depending on
whether pigs land on their feet, sides, backs, etc. in various
combinations.
Two Non-standard Dice, Doubles are Bad: Ivars Peterson
describes a Pig variant called Piggy with non-standard dice that
produce the same distribution of rolls but with a different
probability for rolling doubles. ("Weird Dice", Muse Magazine,
May/June 2000, p. 18). Related curricular resources are available at
the Teacher
Resource Exchange.
Five Dice, 1 is Bad: This pig dice game
from Koplow Games, is not to
be confused with Parker Brother's Pig Dice®. The goal score is
200. One 1 reduces a player's turn total by 10. Two through four 1's loses
the turn total and ends the turn. Five 1's loses the turn total and
score, and ends the turn.
One can also order individual pig dice
from Koplow Games, which are
standard dice with the 1's replaced by pictures of pigs.
Todd W. Neller and Clifton G.M. Presser. Pigtail: A
Pig Addendum, The UMAP Journal 26(4) (2005), pp. 443-458. An addendum with solutions to many Pig variants.
Todd W. Neller and Clifton G.M. Presser. Practical Play of the Dice Game Pig, The UMAP Journal 31(1) (2010), pp. 5-19. Evaluation of human-playable policies with respect to optimal play.
Bell, Robert Charles. 1979. Board and Table Games from Many
Civilizations. Revised ed. New York: Dover Publications,
Inc. Excellent game history reference.
Bellhouse, David R. 1999. Il campanile statistico: What I did on my summer
holidays. Chance 12 (1): 48-50.
Bellman, Richard. 1957. Dynamic Programming. Princeton, NJ: Princeton University
Press. Bellman's optimality equation is the basis for the value iteration technique used to solve Pig.
Bertsekas, D.P. 1987. Dynamic Programming: Deterministic and
Stochastic Models. Englewood Cliffs, NJ: Prentice-Hall. A more
recent treatment of Bellman's optimality equation.
Brutlag, Dan. 1994. Choice and chance in life: The game of "skunk". Mathematics
Teaching in the Middle School 1 (1): 28-33. Describes a pig variant
as a tool for teaching probabilistic concepts at the middle school
level. Unfortunately incorporates the "gambler's fallacy" in
teaching.
Diagram Visual Information Ltd. 1979. The Official World Encyclopedia of Sports
and Games. London: Paddington Press. Additional source for our Pig
rules
Falk, Ruma, and Maayan Tadmor-Troyanski. 1999. THINK: A game of choice
and chance. Teaching Statistics 21 (1): 24-27. Partial analysis of
Brutlag's SKUNK game. We have performed a more complete analysis,
computing optimal scoring play, and demonstrating that single-decision
odds-based analysis does not yield optimal scoring play.
Fendel, Dan, Diane Resek, Lynne Alper, and Sherry Fraser. 1997. The
Game of Pig. Teacher's Guide. Interactive Mathematics Program, Year
1. Berkeley, CA: Key Curriculum Press. An excellent teacher's
resource which uses the game of Pig to teach probabalistic reasoning
at the secondary education level.
Frey, Skip. 1975. How to Win at Dice Games. North Hollywood, CA: Wilshire
Book Co. 1997. Reprint. Source for a 2-dice Pig variant
Reiner
Knizia. Dice Games Properly Explained. Elliot Right-Way Books,
Brighton Road, Lower Kingswood, Tadworth, Surrey, KT20 6TD U.K., 1999.
The rules for Pig and many other dice games can be found here.
Knizia makes an odds-based argument for Pig tactics which I call the
"Hold at 20" policy. He then goes on to point out that there are
situations where one should deviate from this policy. Knizia made me
curious about what optimal Pig play would look like.
Tom
M. Mitchell. Machine Learning. McGraw-Hill, New York,
1997. A good advanced undergraduate or graduate text on the machine
learning in the field of artificial intelligence. Chapter 13 gives an
overview of Reinforcement Learning and covers convergence properties
relevant to value iteration.
Parlett, David. 1991. A History of Card Games. New York: Oxford University
Press. Excellent reference for card game history.
Parlett, David. 1999. The Oxford History of Board Games. New York: Oxford University
Press. Excellent reference for board game history.
Scarne, John. 1945. Scarne on Dice. Harrisburg, PA: Military
Service Publishing Co. 1980. 2nd ed. New York: Crown Publishers,
Inc. The earliest published rules for our game of Pig
Shi, Yixun. 2000. The game PIG: Making decisions based on
mathematical thinking. Teaching Mathematics and Its Applications 19
(1): 30-34. Source for another 2-dice Pig variant


Xapian Administrator’s Guide¶
Table of contents
- Xapian Administrator’s Guide
- Introduction
- Databases
- Backup Strategies
- Inspecting a database
- Database maintenance
- Compacting a database
- Merging databases
- Checking database integrity
- Fixing corrupted databases
- Converting a chert database to a glass database
- Converting a pre-1.1.4 chert database to a chert database
- Converting a flint database to a chert database
- Converting a quartz database to a flint database
- Converting a 0.9.x flint database to work with 1.0.y
Introduction¶
This document is intended to provide general hints, tips and advice to administrators of Xapian systems. It assumes that you have installed Xapian on your system, and are familiar with the basics of creating and searching Xapian databases.
The intended audience is system administrators who need to be able to perform general management of a Xapian database, including tasks such as taking backups and optimising performance. It may also be useful introductory reading for Xapian application developers.
The document is up-to-date for Xapian version 1.4.1.
Databases¶
Xapian databases hold all the information needed to perform searches in a set of tables. The default database backend for the 1.4 release series is called glass. The default backend for the 1.2 release series was called chert, and this is also supported by 1.4.
Glass Backend¶
The following table always exists:
- The postlist table holds a list of all the documents indexed by each term in the database (postings), and also chunked streams of the values in each value slot.
The following table exists by default, but you can choose not to have it:
- The termlist table holds a list of all the terms which index each document, and also the value slots used in each document. Without this, some features aren’t supported - see Xapian::DB_NO_TERMLIST for details.
And the following optional tables exist only when there is data to store in them:
- The docdata table holds the document data associated with each document in the database. If you never set any term positions, this table won’t exist.
- The position table holds a list of all the word positions in each document which each term occurs at. If you never set positional data, this table won’t exist.
- The spelling table holds data for suggesting spelling corrections.
- The synonym table holds a synonym dictionary.
Each of the tables is held in a separate file with extension .glass (e.g. postlist.glass), allowing an administrator to see how much data is being used for each of the above purposes.
The .glass file actually stores the data, and is structured as a tree of blocks, which have a default size of 8KB (though this can be set, either through the Xapian API, or with some of the tools detailed later in this document).
Changing the blocksize may have performance implications, but it is hard to know whether these will be positive or negative for a particular combination of hardware and software without doing some profiling.
The .baseA and .baseB files you may remember if you’ve worked Xapian database backends no longer exist in glass databases - the information about unused blocks is stored in a freelist (itself stored in unused blocks in the .glass file, and the other information is stored in the iamglass file.
Glass also supports databases stored in a single file - currently these only support read operations, and have to be created by compacting an existing glass database.
Chert Backend¶
The following tables always exist:
- The postlist holds a list of all the documents indexed by each term in the database, and also chunked streams of the values in each value slot.
- The record holds the document data associated with each document in the database.
- The termlist holds a list of all the terms which index each document, and also the value slots used in each document.
And the following optional tables exist only when there is data to store in them:
- The position holds a list of all the word positions in each document which each term occurs at.
- The spelling holds data for suggesting spelling corrections.
- The synonym holds a synonym dictionary.
Each of the tables is held in a separate file, allowing an administrator to see how much data is being used for each of the above purposes. It is not always necessary to fully populate these tables: for example, if phrase searches are never going to be performed on the database, it is not necessary to store any positionlist information.
If you look at a Xapian database, you will see that each of these tables
actually uses 2 or 3 files. For example, for a “chert” format database the
termlist table is stored in the files termlist.baseA
, termlist.baseB
and termlist.DB
.
The .DB
file actually stores the data, and is structured as a tree of
blocks, which have a default size of 8KB (though this can be set, either
through the Xapian API, or with some of the tools detailed later in this
document).
The ”.baseA” and ”.baseB” files are used to keep track of where to start looking for data in the ”.DB” file (the root of the tree), and which blocks are in use. Often only one of the ”.baseA” and ”.baseB” files will be present; each of these files refers to a revision of the database, and there may be more than one valid revision of the database stored in the ”.DB” file at once.
Changing the blocksize may have performance implications, but it is hard to tell whether these will be positive or negative for a particular combination of hardware and software without doing some profiling.
Atomic modifications¶
Xapian ensures that all modifications to its database are performed atomically. This means that:
- From the point of view of a separate process (or a separate database object in the same process) reading the database, all modifications made to a database are invisible until the modifications is committed.
- The database on disk is always in a consistent state.
- If the system is interrupted during a modification, the database should always be left in a valid state. This applies even if the power is cut unexpectedly, as long as the disk does not become corrupted due to hardware failure.
Committing a modification requires several calls to the operating system to make it flush any cached modifications to the database to disk. This is to ensure that if the system fails at any point, the database is left in a consistent state. Of course, this is a fairly slow process (since the system has to wait for the disk to physically write the data), so grouping many changes together will speed up the throughput considerably.
Many modifications can be explicitly grouped into a single transaction, so that lots of changes are applied at once. Even if an application doesn’t explicitly protect modifications to the database using transactions, Xapian will group modifications into transactions, applying the modifications in batches.
Note that it is not currently possible to extend Xapian’s transactions to cover multiple databases, or to link them with transactions in external systems, such as an RDBMS.
Finally, note that it is possible to compile Xapian such that it doesn’t make modifications in an atomic manner, in order to build very large databases more quickly (search the Xapian mailing list archives for “DANGEROUS” mode for more details). This isn’t yet integrated into standard builds of Xapian, but may be in future, if appropriate protections can be incorporated.
Single writer, multiple reader¶
Xapian implements a “single writer, multiple reader” model. This means that, at any given instant, there is only permitted to be a single object modifying a database, but there may (simultaneously) be many objects reading the database at once.
Xapian enforces this restriction using by having a writer lock the database.
Each Xapian database directory contains a lock file named
flintlock
(we’ve kept the same name as flint used, since the locking
technique is the same).
This lock-file will always exist, but will be locked using fcntl()
when the
database is open for writing. Because of the semantics of fcntl()
locking,
for each WritableDatabase opened we spawn a child process to hold the lock,
which then exec-s cat
, so you will see a cat
subprocess of any writer
process in the output of ps
, top
, etc.
If a writer exits without being given a chance to clean up (for example, if the
application holding the writer is killed), the fcntl()
lock will be
automatically released by the operating system. Under Microsoft Windows, we
use a different locking technique which doesn’t require a child process, but
also means the lock is released automatically when the writing process exits.
Revision numbers¶
Xapian databases contain a revision number. This is essentially a count of the number of modifications since the database was created, and is needed to implement the atomic modification functionality. It is stored as a 32 bit integer, so there is a chance that a very frequently updated database could cause this to overflow. The consequence of such an overflow would be to throw an exception reporting that the database has run out of revision numbers.
This isn’t likely to be a practical problem, since it would take nearly a year
for a database to reach this limit if 100 modifications were committed every
second, and no normal Xapian system will commit more than once every few
seconds. However, if you are concerned, you can use the xapian-compact
tool to make a fresh copy of the database with the revision number set to 1.
The revision number of each table can be displayed by the xapian-check
tool.
Network file systems¶
Xapian should work correctly over a network file system. However, there are various potential issues with such file systems, so we recommend extensive testing of your particular network file system before deployment.
Be warned that Xapian is heavily I/O dependent, and therefore performance over a network file system is likely to be slow unless you’ve got a very well tuned setup.
Xapian needs to be able to lock a file in a database directory when modifications are being performed. On some network files systems (e.g., NFS) this requires a lock daemon to be running.
Which database format to use?¶
As of release 1.4.0, you should generally use the glass format (which is now the default).
Support for the pre-1.0 quartz format (deprecated in 1.0) was removed in 1.1.0. See below for how to convert a quartz database to a flint one.
The flint backend (the default for 1.0, and still supported by 1.2.x) was removed in 1.3.0. See below for how to convert a flint database to a chert one.
The chert backend (the default for 1.2) is still supported by 1.4.x, but deprecated - only use it if you already have databases in this format; and plan to migrate away.
Can I put other files in the database directory?¶
If you wish to store meta-data or other information relating to the Xapian database, it is reasonable to wish to put this in files inside the Xapian database directory, for neatness. For example, you might wish to store a list of the prefixes you’ve applied to terms for specific fields in the database.
Current Xapian backends don’t do anything which will break this technique, so as long as you don’t choose a filename that Xapian uses itself, there should be no problems. However, be aware that new versions of Xapian may use new files in the database directory, and it is also possible that new backend formats may not be compatible with the technique. And of course you can’t do this with a single-file glass database.
Backup Strategies¶
Summary¶
- The simplest way to perform a backup is to temporarily halt modifications, take a copy of all files in the database directory, and then allow modifications to resume. Read access can continue while a backup is being taken.
- If you have a filesystem which allows atomic snapshots to be taken of directories (such as an LVM filesystem), an alternative strategy is to take a snapshot and simply copy all the files in the database directory to the backup medium. Such a copy will always be a valid database.
- Progressive backups are not easily possible; modifications are typically spread throughout the database files.
Detail¶
Even though Xapian databases are often automatically generated from source data which is stored in a reliable manner, it is usually desirable to keep backups of Xapian databases being run in production environments. This is particularly important in systems with high-availability requirements, since re-building a Xapian database from scratch can take many hours. It is also important in the case where the data stored in the database cannot easily be recovered from external sources.
Xapian databases are managed such that at any instant in time, there is at least one valid revision of the database written to disk (and if there are multiple valid revisions, Xapian will always open the most recent). Therefore, if it is possible to take an instantaneous snapshot of all the database files (for example, on an LVM filesystem), this snapshot is suitable for copying to a backup medium. Note that it is not sufficient to take a snapshot of each database file in turn - the snapshot must be across all database files simultaneously. Otherwise, there is a risk that the snapshot could contain database files from different revisions.
If it is not possible to take an instantaneous snapshot, the best backup strategy is simply to ensure that no modifications are committed during the backup procedure. While the simplest way to implement this may be to stop whatever processes are used to modify the database, and ensure that they close the database, it is not actually necessary to ensure that no writers are open on the database; it is enough to ensure that no writer makes any modification to the database.
Because a Xapian database can contain more than one valid revision of the database, it is actually possible to allow a limited number of modifications to be performed while a backup copy is being made, but this is tricky and we do not recommend relying on it. Future versions of Xapian are likely to support this better, by allowing the current revision of a database to be preserved while modifications continue.
Progressive backups are not recommended for Xapian databases: Xapian database files are block-structured, and modifications are spread throughout the /database file. Therefore, a progressive backup tool will not be able to take a backup by storing only the new parts of the database. Modifications will normally be so extensive that most parts of the database have been modified, however, if only a small number of modifications have been made, a binary diff algorithm might make a usable progressive backup tool.
Inspecting a database¶
When designing an indexing strategy, it is often useful to be able to check the contents of the database. Xapian includes a simple command-line program, xapian-delve, to allow this (prior to 1.3.0, xapian-delve was usually called delve, though some packages were already renaming it).
For example, to display the list of terms in document “1” of the database “foo”, use:
xapian-delve foo -r 1
It is also possible to perform simple searches of a database. Xapian includes
another simple command-line program, quest
, to support this. quest
is
only able to search for un-prefixed terms, the query string must be quoted to
protect it from the shell. To search the database “foo” for the phrase “hello
world”, use:
quest -d foo '"hello world"'
If you have installed the “Omega” CGI application built on Xapian, this can also be used with the built-in “godmode” template to provide a web-based interface for browsing a database. See Omega’s documentation for more details on this.
Database maintenance¶
Compacting a database¶
Xapian databases normally have some spare space in each block to allow new information to be efficiently slotted into the database. However, the smaller a database is, the faster it can be searched, so if there aren’t expected to be many further modifications, it can be desirable to compact the database.
Xapian includes a tool called xapian-compact
for compacting databases.
This tool makes a copy of a database, and takes advantage of
the sorted nature of the source Xapian database to write the database out
without leaving spare space for future modifications. This can result in a
large space saving.
The downside of compaction is that future modifications may take a little longer, due to needing to reorganise the database to make space for them. However, modifications are still possible, and if many modifications are made, the database will gradually develop spare space.
There’s an option (-F
) to perform a “fuller” compaction. This option
compacts the database as much as possible, but it violates the design of the
Btree format slightly to achieve this, so it is not recommended if further
modifications are at all likely in future. If you do need to modify a “fuller”
compacted database, we recommend you run xapian-compact
on it without -F
first.
While taking a copy of the database, it is also possible to change the blocksize. If you wish to profile search speed with different blocksizes, this is the recommended way to generate the different databases (but remember to compact the original database as well, for a fair comparison).
Merging databases¶
When building an index for a very large amount of data, it can be desirable to
index the data in smaller chunks (perhaps on separate machines), and then
merge the chunks together into a single database. This can be performed using
the xapian-compact
tool, simply by supplying several source database paths.
Normally, merging works by reading the source databases in parallel, and writing the contents in sorted order to the destination database. This will work most efficiently if excessive disk seeking can be avoided; if you have several disks, it may be worth placing the source databases and the destination database on separate disks to obtain maximum speed.
The xapian-compact
tool supports an additional option, --multipass
,
which is useful when merging more than three databases. This will cause the
postlist tables to be grouped and merged into temporary tables, which are then
grouped and merged, and so on until a single postlist table is created, which
is usually faster, but requires more disk space for the temporary files.
Checking database integrity¶
Xapian includes a command-line tool to check that a database is
self-consistent. This tool, xapian-check
, runs through the entire database,
checking that all the internal nodes are correctly connected. It can also be
used on a single table, for example, this command will check the termlist table
of database “foo”:
xapian-check foo/termlist.DB
Fixing corrupted databases¶
The “xapian-check” tool is capable of fixing corrupted databases in certain limited situations. Currently it only supports this for chert, where it is capable of:
- Regenerating a damaged
iamchert
file (if you’ve lost yours completely just create an invalid one, e.g. withtouch iamchert
).- Regenerating damaged or lost base files from the corresponding DB files. This was developed for the scenario where the database is freshly compacted but should work provided the last update was cleanly applied. If the last update wasn’t actually committed, then it is possible that it will try to pick the root block for the partial update, which isn’t what you want. If you are in this situation, come and talk to us - with a testcase we should be able to make it handle this better.
To fix such issues, run xapian-check like so:
xapian-check /path/to/database F
Converting a chert database to a glass database¶
This can be done using the copydatabase
example program included with Xapian.
This is a lot slower to run than xapian-compact
, since it has to perform the
sorting of the term occurrence data from scratch, but should be faster than a
re-index from source data since it doesn’t need to perform the tokenisation
step. It is also useful if you no longer have the source data available.
The following command will copy a database from “SOURCE” to “DESTINATION”, creating the new database at “DESTINATION” as a chert database:
copydatabase SOURCE DESTINATION
By default copydatabase will renumber your documents starting with docid 1.
If the docids are stored in or come from some external system, you should
preserve them by using the --no-renumber
option:
copydatabase --no-renumber SOURCE DESTINATION
Converting a pre-1.1.4 chert database to a chert database¶
The chert format changed in 1.1.4 - at that point the format hadn’t been finalised, but a number of users had already deployed it, and it wasn’t hard to write an updater, so we provided one called xapian-chert-update which makes a copy with the updated format:
xapian-chert-update SOURCE DESTINATION
It works much like xapian-compact
so should take a similar amount of time
(and results in a compact database). The initial version had a few bugs, so
use xapian-chert-update from Xapian 1.2.5 or later.
The xapian-chert-update
utility was removed in Xapian 1.3.0, so you’ll need
to install Xapian 1.2.x to use it.
Converting a flint database to a chert database¶
It is possible to convert a flint database to a chert database by installing
Xapian 1.2.x (since this has support for both flint and chert)
using the copydatabase
example program included with Xapian. This is a
lot slower to run than xapian-compact
, since it has to perform the
sorting of the term occurrence data from scratch, but should be faster than a
re-index from source data since it doesn’t need to perform the tokenisation
step. It is also useful if you no longer have the source data available.
The following command will copy a database from “SOURCE” to “DESTINATION”, creating the new database at “DESTINATION” as a chert database:
copydatabase SOURCE DESTINATION
By default copydatabase
will renumber your documents starting with docid 1.
If the docids are stored in or come from some external system, you should
preserve them by using the --no-renumber
option (new in Xapian 1.2.5):
copydatabase --no-renumber SOURCE DESTINATION
Converting a quartz database to a flint database¶
It is possible to convert a quartz database to a flint database by installing
Xapian 1.0.x (since this has support for both quartz and flint)
and using the copydatabase
example program included with Xapian. This is a
lot slower to run than xapian-compact
, since it has to perform the
sorting of the term occurrence data from scratch, but should be faster than a
re-index from source data since it doesn’t need to perform the tokenisation
step. It is also useful if you no longer have the source data available.
The following command will copy a database from “SOURCE” to “DESTINATION”, creating the new database at “DESTINATION” as a flint database:
copydatabase SOURCE DESTINATION
Converting a 0.9.x flint database to work with 1.0.y¶
In 0.9.x, flint was the development backend.
Due to a bug in the flint position list encoding in 0.9.x which made flint
databases non-portable between platforms, we had to make an incompatible
change in the flint format. It’s not easy to write an upgrader, but you
can convert a database using the following procedure (although it might
be better to rebuild from scratch if you want to use the new UTF-8 support
in Xapian::QueryParser
, Xapian::Stem
, and
Xapian::TermGenerator
).
Run the following command in your Xapian 0.9.x installation to copy your 0.9.x flint database “SOURCE” to a new quartz database “INTERMEDIATE”:
copydatabase SOURCE INTERMEDIATE
Then run the following command in your Xapian 1.0.y installation to copy your quartz database to a 1.0.y flint database “DESTINATION”:
copydatabase INTERMEDIATE DESTINATION