. They can either be present in the original
input file (except that the Nib and
2Bit formats are incapable of containing
X
s), or added by using an
xmask
or
nmask
action in the
sequence specifier.
LASTZ can also be configured to tolerate the other IUPAC-IUB ambiguity codes
(B, D, H, K, M, R, S, V, W,
and Y
), and to recognize
a special user-specified separator character.
Many database sequences contain N
s to represent bases for which
the actual nucleotide is not known (at least, not known with any level of
confidence). N
s (or better, X
s) can also be used to
mask out regions that have previously been identified as being of no interest,
and therefore should not be aligned. And unfortunately, there is also a
tradition of using strings of X
s or N
s to splice
together multiple sequences to gain efficiency when dealing with programs that
were limited to operating on a single sequence.
Although splicing was useful in BLASTZ, it is no longer needed for LASTZ.
Since LASTZ can handle multiple target sequences (via the
multiple
action in the target
file’s sequence specifier), it is preferred that users not
resort to splicing.
If splicing is necessary, the preferred method is to specify a
separator character to tell LASTZ explicitely
where the splices have occurred.
Replacing BLASTZ with LASTZ in an existing
pipeline may still involve spliced sequences, so LASTZ’s default
interpretation of non-ACGT characters is the same as BLASTZ’s:
X
s are excluded from the alignment seeding stage, and are so
severely penalized by alignment scoring that they will not normally
appear in
any alignment. N
s are also excluded from seeding, and are
penalized about the same as a transversion mismatch. Specifically, any
substitution with X
is scored as −1000, and any substitution
with anything else (other than A
, C
, G
,
or T
) is scored as −100.
Note that you have to put "enough" X
s or N
s between
the sequences so that no alignment block will cross the splice. This can be
tricky, since gap scoring is only dependent on the length of the gap and not on
the characters in the gap. So if a gap the same length as the splice is not
penalized more than the y-drop setting, the
alignment may hop the splice. As a rough guideline, a splice length of 50 is
usually enough with the default settings, but this is not guaranteed.
This default treatment of non-ACGT characters also works well when
X
s or N
s are used to mask out regions that should not
be aligned. However, it is inappropriate when the sequences contain
N
s to represent ambiguous bases. To handle this case, LASTZ
provides the ‑‑ambiguous=n
option,
which causes substitutions with N
to be scored as zero.
Additionally, the ‑‑ambiguous=iupac
option causes the other IUPAC-IUB ambiguity codes
(B, D, H, K, M, R, S, V, W,
and Y
) to be treated this
same as an ambiguous N
. The two ‑‑ambiguous
options
also allow you to specify rewards and penalties for matches and mismatches
involving ambiguous characters.
In either case, non-ACGT characters are ignored during the seeding stage.
Only seed words that consist entirely of A
, C
,
G
, and/or T
are involved in seeding, even if the
non-ACGT characters occur in "don't-care" positions in the seed pattern.
The score values described above can be changed if a
scoring file is specified. The −1000 score
is called bad_score
and the −100 score is called
fill_score
. Further, which character is considered "bad" (by
default this is X
) can also be specified in the scoring file, and
can actually be different between the target and query. Throughout this
document, when we refer to the character X
appearing in a DNA
sequence, we generally mean the character specified as "bad", which defaults to
X
.
Splicing, or more correctly separation, can also be accomplished by
placing a specific character between subsequences, then using the
separator=<character>
action. LASTZ will then break the sequence into the prescribed subsequences
and prevent any alignment from crossing the boundaries.
Quantum DNA sequences are different: they use an
arbitrary, user-defined alphabet of symbols, so the above-mentioned special
treatments for N
and X
do not apply. The default
"bad" character for quantum sequences is the null byte (00
hexadecimal), which is not even allowed in the sequence; however it can be
changed to one of the valid alphabet symbols via the scoring file. There is
no analog of ambiguous N
s for quantum sequences, as typically
every symbol has some level of ambiguity.
Often the names in the input sequence files are inconvenient for downstream processing, or create problems with certain output formats. This is further complicated by the fact that some input formats (most notably Nib) do not contain sequence names, so in those cases a name must be derived from the filename. LASTZ provides several choices for naming the input sequences. These alternatives are mutually exclusive; only one can be used at a time for a particular input file.
Internally, LASTZ handles the naming task in two phases. First, it creates a full header for the sequence. If the input format provides a name or header, that becomes the full header. Otherwise, the full header is constructed from the file name.
In the second phase, LASTZ shortens the full header to a nickname. If the full
header starts with a file name, any path prefix is removed, and commonly-used
file extension suffixes are also removed (.fa
, .fasta
,
.nib
, .2bit
). Then by default, LASTZ uses the first
word (composed of characters other than whitespace, vertical bar, or colon) of
the remaining string as the sequence name. Thus a
FASTA header like
"> ~someuser/human/hg18/chr1.fa Human Chromosome 1
"
is shortened to simply chr1
.
The actions
nameparse=darkspace
and nameparse=alphanum
in the
sequence specifier change how the first word is
determined. darkspace
(i.e., "non-whitespace") narrows the set of terminating characters
to allow vertical bars and colons to appear in the word, while
alphanum
widens it so the word is restricted to only alphabetic,
numeric, and underscore characters. Path prefixes
and file extensions are still removed.
The default shortening is often adequate. For example, consider the following
FASTA file. By default, the names will be 000007_3133_3729
and
000015_3231_1315
.
>000007_3133_3729 length=142 uaccno=FX9DQEU13H5YZN ACCCGAAAGAGAAACAGCTTCCCCCCCTGTCCCGAGGGATATCAAGTAGTTTGTTGGCTA GGCTGATATTGGGGCCTTCCGCTAGAGTCGGCGCCCGCGCCTACGAGTCCCCCCCACCCC CCACCCCCACAGCGGGTTATCC >000015_3231_1315 length=190 uaccno=FX9DQEU13HUTXE TTGTTGAGTCGGATGAGAATAGCAAGTGCAGTCAACGGCAATGTGCTGGGTTAGTACAAC ...
However, the user may find it more convenient to use the accession numbers. To
accomplish this, she can use the
nameparse=tag:uaccno=
action. LASTZ
will look for the tag string uaccno=
in each header and read the
name from the characters that follow it, up to the first character that is not
alphabetic, numeric, or an underscore. In this case the sequence names would be
FX9DQEU13H5YZN
and FX9DQEU13HUTXE
. If the tag string
is not found in the full header for a particular sequence, the default
shortening is used instead.
Now consider this FASTA file:
>gi|197102135|ref|NM_001133512.1| Pongo abelii ... GCGCGCGTCTCCGTCAGTGTACCTTCTAGTCCCGCCATGGCCGCTCTCACCCGGGACCCC CAGTTCCAGAAGCTGCAGCAATGGTACCGCGAGCACGGCTCCGAGCTGAACCTGCGCCGC ... >gi|169213872|ref|XM_001716177.1| PREDICTED: Homo sapiens ... ATGTCTGAGGAGGTAGGATTTGATGCAGGAGGGAGGATCTGGTGCACTTATAAGGATCTG GGTCTGTCAGTGTCAGAGAAGGTAGGATCTGGCCCTGGTATGAGGATCTGGATCTGTCAG ... >gi|34784771|gb|BC006342.2| Homo sapiens ... GGGTGGGAGGACGCTACTCGCTGTGGTCGGCCATCGGACTCTCCATTGCCCTGCACGTGG GTTTTGACAACTTCGAGCAGCTGCTCTCGGGGGCTCACTGGATGGACCAGCACTTCCGCA ...
In this case the default action does not do what we want (all sequences would
be named gi
). The action nameparse="tag:gi|"
gives
us the names 197102135
, 169213872
, and
34784771
. (Note the quotes; this is necessary to prevent the
command-line shell from interpreting |
as a pipe character.)
Observe that a tag of ref|
will not work, because the third
sequence would need gb|
instead.
Sometimes it is more convenient just to assign a specific name. This can be
done with the
nickname=<name>
action. For example, using the target and query file specifiers
~someuser/human/hg18/chr1.nib[nickname=human]
and
~someuser/human/ponAbe2/chr1.nib[nickname=orang]
, the output
will show the sequences as human
and orang
rather
than calling them both chr1
.
If <name>
contains the substring {number}
,
the nickname will contain the number of the sequence within the file. This is
particularly useful when there is more than one sequence in the file.
If you want to do away with name mangling entirely, you can use the action
nameparse=full
. This uses the full
header as the sequence name. But note that if it contains spaces, the
resulting alignment files may not be readable by downstream tools.
The above discussion applies to ordinary DNA sequences in FASTA, Nib, or
2Bit format. HSX index files
are handled differently: by default LASTZ uses the name from the index as-is,
without shortening it,
and the various nameparse
actions are not
allowed. The nickname
action can be used,
but is generally not
necessary since you can store the names you want directly in the index.
Note that if the
subset=<names_file>
action is
used, the names in the <names_file>
must match the mangled
(or indexed) names.
For FASTA files, more complicated name mangling can be performed using standard
Unix command-line tools. In the second example above, we could pipe the input
through sed
a couple times to shorten each name to the NCBI
accession numbers NM_001133512.1
, XM_001716177.1
,
and BC006342.2
.
cat query_file.fa \ | sed "s/>.*ref\|/>/g" \ | sed "s/>.*gb\|/>/g" \ | lastz target /dev/stdin
Seeds are short near-matches between the target and query sequences, where "short" typically means less than 20 bp. Early alignment programs used exact matches (e.g. of length 12) as seeds, but spaced seeds can improve sensitivity when the sequences are diverged.
A spaced seed pattern is a list of positions, in a short word, where
a seed may contain mismatches. For example, consider the seed pattern
1100101111
. A 1
indicates a match is
required in this position, and a 0
indicates a mismatch is allowed
(effectively it is a "don't care" position). As the example below shows, using
this seed pattern, the seed word GTAGCTTCAC
hits twice in the
sequence ACGTGACATCACACATGGCGACGTCGCTTCACTGG
.
target: ACGTGACATCACACATGGCGACGTCGCTTCACTGG (mis)match: ||XX|X|||| ||X||||||| query: GTAGCTTCAC GTAGCTTCAC pattern:<PHONE_NUMBER><PHONE_NUMBER>
Spaced seeds have been shown to be more sensitive than exact match seeds, with little change in specificity. This is most advantageous when the sequences have lower similarity, such as human vs. mouse or chicken. Which seed pattern is best depends on the sequences being compared. See [Buhler 2003] for a discussion of spaced seeds and how to design them.
LASTZ’s seeding options give the “user” many choices. The intent is that these will be selected by some program (hence the quote marks around “user”), but they are available from the command line for anyone.
--seed=match<length>
1
s, 0
s, and T
s, where a 1
indicates that a match is required in that position, a 0
indicates
that a mismatch is allowed, and a T
indicates that a mismatch is
allowed only if it is a transition (A↔G or C↔T).
--seed=<pattern>The default seed is
‑‑seed=1110100110010101111
, which is the same
12-of-19 seed used as the default in BLASTZ.
0
s and T
s, it is
implemented internally as a half-weight seed, which uses much less memory
(the same amount as a normal seed pattern half as long). Additionally,
‑‑seed=half<length>
can be used as shorthand to specify a
space-free half-weight seed (i.e., all T
s).
1
in a spaced seed, or any
position in an N-mer match) is allowed to be a transition instead of a true
match. ‑‑notransition
disables this. Alternatively,
‑‑transition=2
allows any two match positions to be
transitions.
‑‑filter
option imposes additional requirements on the number
of transversions and matches in a valid seed. This is especially useful in
conjunction with half-weight patterns. For example,
--seed=TTT0T00TT00T0T0TTTT --filter=2,15specifies the same pattern as the default seed, but allows the twelve
T
positions to be matches or transitions, requires at least
fifteen matches total (among the 19 positions), and allows at most two
transversions. Note that the transversions can only occur in the
0
positions, since the T
positions allow only matches
or transitions.
And although there are seven 0
positions, five of
them must contain matches or transitions since only two transversions are
allowed.
--twins=[<minsep>..]<maxsep>The distance between the hits (the number of bases between the end of the first hit and the beginning of the second) must be at least
<minsep>
but not more than <maxsep>
.
If <minsep>
is omitted, zero is used (which means the
twin seeds may be adjacent but not overlap). Negative values can
be used; for example ‑‑twins=‑5..10
means the twins can overlap
by as much as 5 bases or can have as much as 10 bases between them.
Sometimes, the only answer you want from an aligner is whether a query has any strong alignments to the target or not. For example, you may want to know which reads in a sequencing run have no alignment with a reference genome. In this case, if a read aligns to a thousand different places on a particular chromosome, you aren't interested in learning where — all you want to know is whether it aligned or not.
The ‑‑anyornone
option is designed
for such cases, and can significantly improve alignment speed. Once any
qualifying alignment has been found, processing for the current query is
halted. The alignment is reported to the output, and then we immediately begin
processing the next query. A qualifying alignment is one that would normally
be output given the other parameter settings; for example it satisfies the
scoring thresholds (‑‑hspthresh
and/or ‑‑gappedthresh
) and any
back-end filters.
To get a list of reads that have at least one "good" alignment with a reference sequence, you could do something like this:
lastz <reference> <reads> --anyornone \ --step=10 --seed=match12 --notransition --exact=20 --noytrim \ --match=1,5 --ambiguous=n \ --filter=coverage:90 --filter=identity:95 \ --format=general:name2
This option slightly changes the usual processing order described in the Overview. Instead of performing gap-free extension on all seeds, collecting them into a list of HSPs, and then performing gapped extension, each HSP is gap-extended and back-end filtered immediately. This avoids wasted work to perform complete early stage processing on hits that will just be abandoned as soon as the first qualifying alignment is found.
The default configuration of gapped extension in LASTZ is to end the alignment where the score would be the highest. This means that any prefix or suffix of the alignment will have a non-negative score. While this is appropriate for alignments that lie somewhere in the middle of two long sequences, it is not desirable when an alignment is near the end of one or both sequences, which happens quite often when aligning short reads.
Consider the following alignment of a 50-base query to a chromosome target, and
suppose we are using ‑‑match=1,5
,
‑‑gap=6,1
,
‑‑filter=identity:97
, and
‑‑filter=coverage:95
. The entire
alignment as shown has 97.9% identity (46/47) and 100% coverage. However, the
first five bases (AGAAC
vs. AGAAG
) have a negative
score: four matches at +1 each and one mismatch at −5 gives a score
of −1 for this prefix. The highest scoring alignment is from positions
6 through 50, for a score of 33 (the entire alignment scores only 32). If
we stop the alignment at the highest score, coverage drops to 90%, and the
alignment is discarded. The overall result is that we will discard reads that
we don't want to, and we will see a bias against mismatches near the ends of
reads. (Note that this anomaly arises because the alignment is terminated
abruptly by the end of the sequence rather than normally by a low-scoring
region; also the ‑‑filter=coverage
option is more commonly used
with short reads than with longer sequences.)
target: ... CTTAGAACGGTAGATACTTGTATAAT---CGAGGGGGTTATTTTGTACAAATGACT ... ||||X|||||||||||||||||| |||||||||||||||||||||||| query: AGAAGGGTAGATACTTGTATAATCAACGAGGGGGTTATTTTGTACAAATG ↑ ↑<PHONE_NUMBER>1234567890123456789012345678901234567890
To avoid this behavior, use the
‑‑noytrim
option when aligning short
reads. This causes LASTZ to refrain from trimming such alignments back to the
highest-scoring location. Specifically, if the
gapped extension process encounters the end of the
sequence, it will keep that as the end of the alignment. In this case a
negatively-scoring prefix or suffix will be kept as long as it does not score
worse than the ‑‑ydrop
value.
In some applications, e.g. when assembling reads into contigs, we want to
determine how sequence ends overlap each other. For example, in case 1 below,
the starting portion of the query overlaps the ending portion of the target by
30 bases, and both sequences extend beyond each other in opposite directions.
We call this situation "shingling" (like shingles on a rooftop), and the
shingle
field of the General output
format provides a measurement of it. A positive value indicates that the
starting portion of the query overlaps the ending portion of the target (case
1), while a negative value indicates the roles are reversed (case 2). If
neither of these cases occurs (e.g. if either sequence fails to extend beyond
the other), an NA
is reported.
Case 1 (shingle = +30):
target_end 3 2 1 ↓<PHONE_NUMBER>09876543210987654321 target: ... GACGGCGGCTAACACATTGTGTTGXACGTACCATAACCAA ||||||X|||||||||XX||X|||||| query: AACACAGTGTGTTGCAACTATCATAACATTAAACTTTAGA ...<PHONE_NUMBER>12345678901234567890 ↑ 1 2 3 query_start
Case 2 (shingle = −30):
target_start ↓ 1 2 3<PHONE_NUMBER>12345678901234567890 target: TCCCTAATAAATCTTAAGTGCGATCCGCAGCGAGGTGTAC ... ||||X|||||||||X||||||||X|| query: ... TGGCGCCTGTAGTCTAAGAAATCTTAATTGCGATCCACAC<PHONE_NUMBER>09876543210987654321 3 2 1 ↑ query_end
Note that the value reported has no relation to the number of bases that align in that region, nor is it an indication that the alignment extends all the way to the start or end of the sequences. The shingle value is just evidence that the proper registration of the two reads is to overlap them by the given value — information that an assembler might use in assembling those reads into a contig.
Target capsule files are provided to improve run-time memory utilization when multiple CPU cores on the same computer are running LASTZ with the same target sequence. They permit the lion’s share of the large internal data structures to be shared between the processes. This allows more copies of LASTZ to be run simultaneously with less physical memory, which can improve the throughput, for example, when mapping a large set of reads to a single (large) reference sequence.
To create a capsule file, use a command like this:
lastz <target> --writecapsule=<capsule_file> [<seeding_options>]Applicable seeding options are
‑‑seed
,
‑‑step
,
‑‑maxwordcount
,
and ‑‑word
.
To use the capsule file, run LASTZ like this:
lastz --targetcapsule=<capsule_file> <query> [<other_options>]No additional effort on the part of the user is required to handle sharing of the capsule data between separate runs. Nearly all options are allowed; however the seeding options
‑‑seed
,
‑‑step
,
‑‑maxwordcount
,
and ‑‑word
are not allowed, since these (or their byproducts) are already stored in the
capsule file. Further, ‑‑masking
is not allowed, because it would require modifying both the target sequence and
the target seed word position table, which are contained in the capsule.
Internally LASTZ asks the operating system to directly map the capsule file
into the running program’s memory space
in a read-only fashion. Multiple running instances can map
the same file; each instance will have its own virtual addresses for the
capsule data, but the physical memory is shared. There is no requirement for
more than one instance to actually use the capsule simultaneously. Running
a single copy of lastz
with ‑‑targetcapsule
will work
fine, and in fact there may be a small speed improvement compared to running
the same alignment without a capsule.
The downside of this technique is that the capsule files are very large and are also machine-dependent. For example, the file for human chromosome 1 is about 1.4 Gb. Note that attempts to run a capsule built on a mismatched computer are detected and rejected.
Scoring inference is an automated method for determining appropriate substitution scores and/or gap penalties directly from the sequences being aligned. The resulting scoring parameters can be saved to a file and/or used immediately to align the sequences. Generally these depend mostly on the species rather than particular regions, so once a suitable scoring set has been obtained for a pair of species, the inference does not need to be performed for each alignment run. In this section we give a brief overview of the inference process; see [Harris 2007] for a more detailed description.
Inference is achieved by computing the probability of each of the 18 different alignment events (gap open, gap extend, and 16 substitutions). These probabilities are estimated from alignments of the sequences. Of course, at first we don't have alignments, so we start by using a generic scoring set to create alignments, infer scores from those, then realign, and so on, until the scores stabilize or "converge". Ungapped alignments are performed until the substitution scores converge, then gapped alignments are performed (holding the substitution scores constant) until the gap penalties converge.
To have LASTZ infer scoring parameters, use
a suitably enabled build of LASTZ (see below), and specify
the ‑‑infer
or
‑‑inferonly
options. (The latter
will stop after inferring the parameters, without performing the final
alignment.) Settings for the inference process can be specified in a
control file included with these options.
The ‑‑infscores
option causes the
inferred scoring parameters to be written out to a separate file. If no
<output_file>
is specified, it is written to the header
of the alignment output file, as a comment. As a last resort, if no alignment
is performed the scoring set is written to stdout
. The parameters
are written in the same format used to input scoring
sets.
Usually it is undesirable to use all alignment blocks for inference. Blocks with a high substitution rate (low identity) are likely to be false positives. On the other hand, blocks with few substitutions (high identity) will be found regardless of what scoring parameters are used. Thus it is desirable to base the inference only on statistics from a mid-range of identity. By default the middle 50% is used (that is, the 25th through 75th percentile from the identity distribution), but this can be changed in the control file.
lastz_D
). Moreover,
the technique used to infer gap penalties has not yet been shown to select good
values, so the author recommends that users only employ inference for
substitution scores. To encourage these recommendations, the scoring inference
code is blocked from operation in the integer scoring version of LASTZ
(lastz
), and gap penalty inference is blocked in both versions.
Special build options are available to defeat the blocks; contact the author
if you are interested.
Though LASTZ provides several filtering options (e.g.
‑‑filter=identity
,
‑‑filter=continuity
,
‑‑filter=coverage
,
‑‑filter=nmatch
,
‑‑filter=nmismatch
,
‑‑filter=ngap
and
‑‑filter=cgap
),
sometimes these
are not sufficient for the task at hand. But in many cases it is still possible
to perform the desired filtering by using the
‑‑format=general
option in conjunction
with a simple
awk,
perl, or
python script. Here we show one such
example, using awk.
Suppose we want to filter alignments by length, discarding anything shorter than 500 bp, and that we need AXT output for downstream processing. We can have LASTZ output whatever columns are necessary to reproduce AXT and use awk to perform the filtering and reconstruct an AXT file.
Looking at the UCSC AXT specification, the corresponding --format=general fields are shown in the table below. Note that when determining which fields are needed for a given format, care has to be taken to make sure to get the correct start and end fields. Different formats count from zero instead of one, and some count reverse-strand positions along the plus strand. The interval coordinates section provides more detail about possible numbering schemes.
| AXT field | field for ‑‑format=general |
| Alignment number | (none) |
| Chromosome (primary organism) | name1 |
| Alignment start (primary organism) | start1 |
| Alignment end (primary organism) | end1 |
| Chromosome (aligning organism) | name2 |
| Alignment start (aligning organism) | start2 |
| Alignment end (aligning organism) | end2 |
| Strand (aligning organism) | strand2 |
| Blastz score | score |
| Sequence line (primary assembly) | text1 |
| Sequence line (aligning assembly) | text2 |
Then we can perform our filtered alignment with a series of commands like this:
lastz target.fa query.fa \ --format=general:name1,start1,end1,name2,start2,end2,strand2,score,text1,text2 \ | grep -v "^#" \ | awk '{ if ($3-$2+1 >= 500) print $0 }' \ | awk 'BEGIN { n=-1;} { print ++n,$1,$2,$3,$4,$5,$6,$7,$8; print $9; print $10; print ""; }' \ > filtered.axtThe grep command discards the line containing column headers.
The first awk command computes the alignment length in the target, and if it is
at least 500, copies the line to the output. $3
is
end1
and $2
is start1
. Since these
represent a closed interval, we have to add 1 to get the length. $0
represents the entire input line.
The second awk command converts the alignment from a single line into four lines
required for AXT. We use an awk counter, n
, to create the
alignment number field. The other fields are copied from the fields output by
LASTZ.
For many alignment problems it is desirable to ignore alignments that consist soley of genomic repeats. For this reason, most finished genomic assemblies are soft-masked — bases that are part of identified repeats are in lowercase. LASTZ’s seeding stage avoids seed hits in lowercase, and thereby avoids finding alignments that are solely repeats.
With the wider availability of less-expensive DNA sequencing and custom assemblies, it is more common for users to have unannotated sequences. The following describes how LASTZ can be used to crudely identify duplications and soft-mask the original sequence. This process is called self-masking.
The self-masking process works by looking for alignments of the sequence with itself, and makes use of the dynamic masking feature to reduce computational time. The sequence is split into overlapping fragments, on the fly, which are then aligned against the entire sequence. As duplications are discovered they are marked as such and removed from the seeding process.
A command to perform self-masking would look like this, where critter.fa contains the sequence to be masked.
cat critter.fa \ | ../tools/fasta_fragments.py --fragment=200 --step=100 \ | lastz critter.fa[multiple,unmask,nameparse=darkspace] /dev/stdin --masking=3 \ --progress+masking=10K \ --format=none --outputmasking+:soft=critter.masking.dat \ --notransition
In that command, lastz is given the whole “critter” as its target sequence, and overlapping 200bp fragments of the critter as the queries.
The multiple
action tells lastz to
allow more than one sequence in the file, and the
unmask
action tells lastz to
ignore any softmasking that may be present in the file. (If your sequence
already has had some masking performed, and you want to keep that, omit
unmask
.) The
nameparse=darkspace
action tells
lastz to extract the first non-whitespace string from the sequence header line.
This is necessary to ensure that the final step
(fasta_softmask_intervals
) will see the same sequence names in the
masked intervals file as those in the sequence file.
The ‑‑masking=3
option enables
dynamic masking, which will mark any reference base appearing in 3 or more
alignments. Since the fragments overlap by a factor of two, we expect every
base will appear in two trivial alignments. Any more than that would be caused
by a duplication elsewhere.
The ‑‑progress+masking
option causes lastz to give you a progress report after every 10 thousand
fragments. These reports come to the console (stderr) and look like this:
(16.933s) processing query 50,001: critter_21299501, masked 8,920,893/51,304,566 (17.4%)
The ‑‑format=none
option inhibits the
normal alignment output and
‑‑format=outputmasking+:soft
tells lastz to write the final masked intervals to a file.
The final line (‑‑notransition
in this example) is whatever alignment scoring parameters you want to use.
What is appropriate will depend on the level of divergence you want to allow in
the masked duplications.
The command to apply the masking intervals to the fasta file will look like this:
cat critter.fa \ | ../tools/fasta_softmask_intervals.py --origin=1 critter.masking.dat \ > critter.softmasked.fa
There are many occasions when you have a general idea of where the alignments you are interested in are, and it seems computationally wasteful to align entire sequences just to find a relatively few alignments. For instance, you may have identified some alignments using fast, high sensitivity settings and now want to look for alignments with higher divergence in the remaining regions. Or you may have previously found alignments but did not collect all the fields you needed. Or perhaps you used some tool other than LASTZ to identify regions where you want to focus your search. Or you may have gapped alignments from some other tool, and want to compare them to LASTZ's alignments in the same subsequences.
There are many ways to solve such a problem, and LASTZ provides several options to support these needs. Here we describe and critique several different approaches.
Sequence masking. LASTZ can use masking to eliminate the possibility of alignments in (or not in) a given list of intervals. So we can create two files, containing the desired intervals in the target and query, then run one LASTZ job.
The disadvantage of this solution is that you may get alignments between unintended interval pairs (for example, an alignment between the first target interval and the fifth query interval). Some post-proceesing would be necessary to eliminate these. Moreover, LASTZ will be spending a lot of its time looking for alignments in those unintended interval pairs.
Separate files. The simplest solution, conceptually, is to preprocess the sequences to extract the intervals of interest into separate files, then run each pair of files as a separate LASTZ job.
The disadvantages of this solution are numerous. There is extra I/O involved in splitting the files, and extra overhead in repeatedly launching LASTZ. Further, depending on your needs, post-processing may be necessary to map alignment positions back to the original sequences.
Subranges and subsets. The separate files solution can be improved upon by letting LASTZ mimic the file splitting, internally. This can be accomplished with the subrange and subset actions, while still running each as a separate lastz job.
This improves upon external file separation by eliminating some I/O, and eliminating the need to post-process to map positions. However, it still suffers the extra overhead of repeatedly launching LASTZ.
Alignment chores.
Another solution is to use an alignment chores file
(specified with the
‑‑chores=<file>
option). The
chores file corresponds directly to the interval pairs of interest. Complete
alignment is performed over the regions defined by each pair (subject to
whatever other options have been set), and is blocked from extending beyond the
region.
There is little downside to this solution. The reported results will be the same as for the post-processed separate files solution, or subrange/subset solution (but with minor variations such as shifting of equally-scoring gap placements). There is some computational waste in the chores solution, but this is much less costly than the repeated launching.
In most cases, a chores file will be preferred over an anchors file.
Anchor segments.
Another solution is to use an anchor segments file
(specified with the
‑‑segments=<file>
option). The
anchors file does not correspond directly to the interval pairs of
interest. Instead, you will need to have same-length intervals in target and
query. Typically this will be a single point somewhere in the region.
Alignment is anchored at this point. Gapped alignment is performed in both
directions from that point, and is not restricted to the region.
The main disadvantage of this solution is lack of versatility. It is most appropriate when used in conjunction with an external anchor-identifying method.
It can also be useful in cases where you want to run ungapped alignment with a different scoring scheme than for gapped alignment. An ungapped run of LASTZ creates the anchors (segments) file, and a second run uses those as anchors for ungapped alignment with a different scoring scheme.
‑‑allgappedbounds
option can be
used to revert to the bounding criteria used in BLASTZ.
B, D, H, K, M, R, S, V, W,
and
Y
) in fasta sequences but was unclear about how these were scored.
Since we feel the user should be aware of how these bases are treated, LASTZ
rejects them by default. The
‑‑ambiguous=iupac
option permits them
but treats them the same as an ambiguous N
. This is discussed in
Non-ACGT Characters.
During the gapped extension stage, LASTZ processes the anchors in order of score (highest scoring anchor is extended first, and so on). As anchors are extended, a list of bounding alignments is constructed. These correspond to paths in the DP matrix. Bounding alignments created for higher-scoring anchors are used to bound the possible DP paths that lower-scoring anchors can take. This prevents alignments from crossing each other.
In BLASTZ, every gapped extension became a bound, and this was originally the default behavior in LASTZ, through release 1.1.52. However, this caused LASTZ to miss some alignments which it should have found. The failure case occured as follows. A high-scoring anchor is extended but fails to meet the score threshold. But it gets added as a bound. Then the extension of a lower-scoring anchor is prevented from crossing or intersecting with that path, and it too gets discarded even though it might score highly enough. This could occur, for example, when two extensions would (in the absence of each other) share the same tail, and the higher-scoring of the two has a lower-scoring anchor.
The correction for this is to only use alignments as bounds if they satisfy the
score threshold. This corrected behavior is now the default in LASTZ (as of
release 1.02.00). The
‑‑allgappedbounds
option can be
used to revert to the bounding criteria used in BLASTZ.
| Release | Date | Changes |
|---|---|---|
| 1.0.1 | Jul/28/2008 | Initial release. |
| 1.0.5 | Aug/2/2008 |
Fixed a bug that in some cases caused a bus error when interpolated
alignments (e.g. ‑‑inner= …) were used with multiple
queries.
|
Added xmask=<file> and nmask=<file>
file masking actions.
| ||
| 1.0.21 | Sep/9/2008 |
Fixed a bug involving the default value for ‑‑gappedthresh
(a.k.a. L ) when ‑‑exact is used. The bug caused the
gapped threshold to be inordinately low, allowing undesirable alignment blocks
to make it to the output file.
|
Fixed a bug whereby Xs and Ns were treated as desirable substitutions when
unit scores (e.g. ‑‑match= …) were used.
| ||
Re-implemented ‑‑twins= …. The previous implementation
improperly truncated the left-extension of HSPs. The new implementation is
slower and uses more memory.
| ||
Added ‑‑census=<file> . The census counts the number of
times each base in the target sequence is part of an alignment block.
Previously, ‑‑census produced a census only if the output format
was LAV (the census is a special stanza in a LAV file). Otherwise the option
was ignored. Now, if a file is specified a census is written to that file.
The format of lines in the census is
<name> <position> <count> .
The position is one-based, and the count is limited to 255.
In situtations where 255 is too limiting, | ||
Added ‑‑format=<differences> , to support Galaxy. All
differences (gaps and runs of mismatches) are reported, one per line.
| ||
Added ‑‑anchors=<file> (eventually this was renamed to
‑‑segments=<file> ), giving the user the ability to bypass
the seeding and gap-free extension stages.
| ||
Changed default gap penalties for unit scores (e.g.
‑‑match= …) to be relative to mismatch score (instead of
match score).
| ||
Made the <start>#<length> file subrange action
better at checking errors, and also allowed <length> to use
units such as M and K.
| ||
| Sped up program exit by no longer freeing dynamically allocated memory. | ||
| 1.1.0 | Dec/5/2008 | Improved x-drop extension to better handle suboptimal HSPs. Left-extension now starts at the right end of the seed (rather than the left end). This reduces the chance that the extended region (the combination of left and right extensions) will score less than some subinterval. |
| Changed coverage filtering so that it is relative to whichever sequence is shortest. Previously it was always relative to the query. | ||
Changed defaults for xdrop and ydrop when ‑‑match scoring is
used.
| ||
| Interpolation now uses the xdrop value from the main alignment. Previously it used the ydrop value to match BLASTZ, but we have decided that was a bug in BLASTZ. | ||
Added general output format.
| ||
Added ‑‑maxwordcount .
| ||
Added ‑‑notrivial .
| ||
Corrected problem with ‑‑subset action, which wasn't using
mangled sequence names.
| ||
| Fixed problem in writing LAV m- and x-stanzas. | ||
| Blocked the use of scoring inference in the integer build, and blocked gap scoring inference in all builds. | ||
| Changed much of the syntax for options and actions. The newer syntax is clearer and more consistent than the older. The older is still supported by the program so that existing scripts will still work, but it is not documented. | ||
Changed reporting of duplicated options from
can't understand "<option>" to
duplicated or conflicting option "<option>" .
| ||
Added ‑‑format=rdotplot option.
| ||
| 1.1.25 | Feb/5/2009 | Fixed a bug that caused some gapped extensions to be terminated prematurely. In some cases this also allowed a nearby low-scoring alignment to "piggyback" onto the remainder of a terminated alignment, gaining enough in score to pass the score threshold. |
| Added support for target capsule files. | ||
Added support for ‑‑format=cigar .
| ||
Added the <center>^<length> sequence interval
specifier.
| ||
Corrected the behavior of ‑‑exact regarding lowercase and
non-ACGT characters. ‑‑exact now considers, e.g., a lowercase A
to be a match for an uppercase A. Further, any non-ACGT characters now stop
the match.
| ||
| Improved detection and reporting of memory allocation overflow. Two problems were fixed as part of this: (1) allocation of single blocks larger than 2 Gb was being rejected even on platforms that could support larger blocks, and (2) an allocation overflow problem which could cause a segfault for target sequences longer than about 1 Gb (these require allocation of a block larger than 4 Gb). | ||
Changed the behavior when encountering an empty sequence in a file with
many sequences. Previously this was reported as an error, and the program
halted. Now it is reported as a warning (to stderr ), and the
program continues.
| ||
Added the ‑‑output option. In some batch systems, it is
difficult to redirect stdout into a file, so this option allows
the user to do it directly.
| ||
Removed ‑‑quantum and ‑‑code options, replacing
them with the quantum and quantum=<code_file>
sequence specifier actions. This is in preparation for allowing a quantum
target sequence.
| ||
| 1.1.50 | Mar/16/2009 |
Fixed two problems with exact-match extension. First, when both target and
query used the multiple sequence specifier action, exact match
extension was able to skip the boundary between sequences (this problem was
introduced in 1.1.25). Second, when the exact match should have extended to
the end of the sequence, it was being cut short by 1 bp (on either end). The
latter problem was only evident for ‑‑nogapped ; a gapped entension
recovered the additional bases.
|
Fixed several problems with ‑‑segment=<file> . First, if
the file contained more than 4,000 segments, on some platforms the program would
segfault. Second, if a sequence subrange was being used, the limit test
comparing the segment interval to the subrange was incorrect. Third (if the
user was lucky enough to avoid the first two problems), if a segment was on the
negative strand it was improperly mapped to the subrange.
| ||
Added ‑‑noytrim to prevent y-drop mismatch shadow, improving
LASTZ’s ability to align short reads.
| ||
Set the default gapped extension score threshold to inherit the lowest HSP score in the
case where ‑‑hspthresh=top<basecount> or
‑‑hspthresh=top<percentage>% is used but
‑‑gappedthresh=<score> is not (and gapped extension is
performed). Previously this case was trapped by a low level routine and the
alignment was halted.
| ||
Fixed a problem with the start2+ field of
‑‑format=general . The position was left blank for alignments on
the + strand.
| ||
Fixed a problem in which ‑‑writecapsule was rejected if
‑‑seed=match<length> was used.
| ||
| Fixed a problem related to name mangling which caused an "internal error" to be reported. | ||
| Fixed a problem whereby single-symbol identifiers were not recognized in quantum code files. | ||
End of sequence limit checking for <start>#<length>
and <center>^<length> sequence specifier actions is
now "soft". If the resulting interval is beyond the end of the sequence it is
truncated.
| ||
Changed how ‑‑format=cigar reports alignments on the negative
strand. Apparently there is no complete spec for CIGAR format. Matching what
I see output by exonerate for certain cases is the best I can do.
| ||
| Quantum code files can now specify probabilities as fractions. This gives a clearer representation for motif-like sequences derived from a multiple alignment. | ||
Added cigar field for ‑‑format=general .
| ||
Added shingle field for ‑‑format=general .
| ||
Added the ‑‑rdotplot=<file> option.
| ||
The ‑‑notrivial option now works with the multiple
sequence specifier action.
| ||
Added ‑‑markend .
| ||
Added nameparse=darkspace .
| ||
| Modifed the build process to accomodate the Solaris platform. | ||
| 1.1.52 | Mar/24/2009 | Fixed a bug that occurred when ydrop was less than the penalty for a one-base gap (the sum of open and extend penalties). In this case, a bug in the initialization of the DP matrix resulted in no gapped alignments ever being found. |
| Fixed a problem with the combination of ‑‑recoverseeds and ‑‑exact. Recovered seeds were cut short by one base on the left end. | ||
Added ‑‑format=segments option. This was later replaced by
‑‑writesegments .
| ||
Added a workaround in the source code for what appears to be a bug in gcc
4.3.2 (see http://gcc.gnu.org/bugzilla/show_bug.cgi?id=37861). Without the
workaround, the build fails with this message:
quantum.c: In function 'generate_dna_ball': quantum.c:347: error: array subscript is above array boundsThe workaround uses an ifdef that specifically targets gcc 4.3.2. | ||
| 1.02.00 | Jan/12/2010 |
Relaxed the rejection of some output formats, which was too aggressive.
Specifically, runs with ‑‑tableonly were rejected because of
output format, even though no output would be generated in that format.
|
Added the ability to set the ‑‑maxwordcount option as a
percentage. Also, ‑‑maxwordcount=<limit> now allows
<limit> to be 1. Previously it was not allowed to be less
than 2.
| ||
The scoring matrix used during x-drop extension now reflects the use
of ‑‑ambiguous=n . Previously, this matrix was not affected by
‑‑ambiguous=n ,
and N-vs-N matches and N-vs-other matches were scored as -100 (more
specifically, as fill_score ) during gap-free extension. This
caused LASTZ to miss some HSPs, usually those containing an N-vs-N match, since
the HSP was terminated at that match and didn't meet the score threshold. This
has been corrected.
| ||
| Added support for HSX indexes, to support random access into FASTA files. This improves the speed of aligning a single read (from a file of half a million) by a factor of about 12. | ||
Added ‑‑softmask=<mask_file> file action to permit
soft masking of specified intervals. Also added masking of the
interval complements —
‑‑xmask=keep:<mask_file> ,
‑‑nmask=keep:<mask_file> , and
‑‑softmask=keep:<mask_file> . These make it easier to
restrict alignment to several specified intervals of a sequence.
| ||
Enabled the use of ‑‑filter=[<transv>,]<matches>
for non-halfweight seeds. Previously, ‑‑filter had only been
tested for half-weight seeds, but was erroneously prohibited for
all seeds (instead of just prohibiting non-halfweight seeds). Further, it
was not properly implemented for seed-only output (‑‑nogfextend
‑‑nogapped ). These have all been corrected, and ‑‑filter
is now available for all seed types.
Also corrected the behavior of
Also changed the behavior when the | ||
Added a compile-time directive compileForWindows to make
appropriate behavioral adjustments for running on a Windows machine.
Currently this only affects the handling of file paths. To activate it,
the user must add -DcompileForWindows to the definition of
definedForAll in
.../lastz‑distrib‑X.XX.XX/src/Makefile .
| ||
Fixed chaining of seed hits. Previously, if ‑‑nogfextend and
‑‑chain were used together, nothing was output. This was due to
the fact that unextended seeds had no scores, and the chaining algorithm only
reports chains with positive score. This has been corrected by calculating
scores (as the sum of substitution scores) over anchor segments whenever (a)
the segments have not had scores computed for them, and (b) scores are required
for later processing.
This change may also affect (for the better) the results of gapped extension
when either | ||
Changed ‑‑format=segments to
‑‑writesegments=<file> .
| ||
| Added M-mismatch extension. | ||
Added the replacement of {number} in sequence nicknames.
| ||
Added support for continuity reporting and
filtering .
| ||
Added support for match count
filtering .
| ||
Fixed a bug in handling subrange actions for nib files. The problem occurred
when the subrange action was of the form <start>.. and
<start> was even. That is, no <end> was
specified, and LASTZ is supposed to use the remainder of the sequence. LASTZ
miscalculated the length of the interval, making it one base longer. If the
actual full sequence length was odd, this resulted in an extra T
being appended to the sequence data. If the full sequence length was even,
LASTZ quit, reporting that it was unable to read the sequence. Note that this
only happened for .nib files, only when <start> when even,
and only when no <end> was specified.
| ||
Added the subsample action.
| ||
Added the ‑‑anyornone option.
| ||
Added ‑‑allgappedbounds .
| ||
Fixed a bug in exact and mistmach extension and queries using the
multiple action. It was possible for an HSP to cover parts of
two different queries.
| ||
| Fixed an overflow bug in the chaining algorithm. Due to numerical overflow, very high scoring HSPs were treated as negatively scoring, and thus were not included in final chains. With default scoring values, overflow was caused by the equivalent of an exact match of about 22Mbases. This problem also existed in BLASTZ. | ||
| Added support for output in SAM format. | ||
| Corrected dotplot output. Previously, some of the coordinate values were inconsistent and off by one. | ||
Added ‑‑progress=[<N>] .
This existed as an unadvertized option in earlier versions of the program, as
‑‑debug=queryprogress=<N> . It has now been promoted to a
first class option.
| ||
Added ‑‑ambiguous=iupac and changed ‑‑ambiguousn to
‑‑ambiguous=n . the former is still supported, but not advertized.
| ||
Column headers for ‑‑format=general now match the command-line
keywords. Previously, all related keywords shared the same column header.
For example, keywords start2 , zstart2 ,
start2+ and zstart2+ all produced the same column
header, start2 , in the output file.
Also added | ||
Now using inttypes.h macros for sized-types. This is to satisfy some
additional type-checking pickiness that appears to have added to gcc version
4.2.1. In the unlikely even that a compiler doesn't support inttypes.h, the
compile-time definition override_inttypes can be used.
| ||
Added nmatch , nmismatch , ngap ,
cgap and cigarx fields for
‑‑format=general .
| ||
Added ‑‑format=mapping , a shortcut for typical fields for
‑‑format=general for mapping reads.
| ||
| 1.02.11 | Aug/21/2010 |
Fixed the cigarx field for ‑‑format=general , so
that a run length of 1 is omitted for indels.
|
Fixed the behavior of ‑‑recoverseeds , which was failing to
recover many HSPs when seed denisty was high. This was due to left extension
being blocked by other seeds on that same hash-equivalent diagonal. Left
extension is now unblocked when ‑‑recoverseeds is enabled.
| ||
Changed/corrected how the ‑‑segment option handles wildcard names
when the multiple action in used. To support this, the
rewind command was added to the segments file format.
| ||
Sequence masking actions (softmask , xmask and
nmask ) are now allowed for the multiple action.
| ||
| Command-line arguments beginning with two unicode non-breaking hyphens are now recognized. Since these are used in some places within this README file, it is natural for a user to copy them to the command line. Previously these were not recognized, which led to a somewhat confusing error message. | ||
| Fixed detection and reporting of improper gap penalties. Because the first base in a gap is penalized as open+extend, open can be zero or negative as long as that sum is strictly positive. Previously, a sum of zero was permitted, and a negative sum was misleadingly reported as a problem with the open penalty. Now, the sum must be strictly positive, and when it isn't the message more accurately describes the problem. | ||
Fixed the implementation of ‑‑self with regard to mirror-image
pairs. Previously, alignments were internally restricted to be above the main
diagonal in the ungapped stage only. The mirrored twins were created prior to
the gapped stage, and the gapped stage operated on the full set of anchors.
This had two undesirable effects -- there was little computational savings, and
the resulting set of alignments could be assymetrical (due to small variations
in gap positioning). This behavior has been changed so that the above-diagonal
restriction occurs throughout the alignment process and mirrored twins are
created just prior to output.
| ||
| 1.02.16 | Nov/2/2010 |
Fixed a problem with ‑‑self , introduced in 1.02.11. The problem
manifested itself on 64-bit CPUs, with an error message indicating it was
attempting to allocate 17 billion bytes for edit_script_copy. This has been
corrected.
|
|
Corrected a problem in LAV output, in which the d stanza
reported an incorrect value for K or L (the ungapped and gapped soring
thresholds) when they were not equal to each other. Which value was reported
incorrectly depends on nuances of the compiler and could differ by platform.
Alignments were not affected. | ||
| Changed the error message when a fasta file contains bad characters. The previous message caused confusion when the bad character happened to be punctuation. Now the error message explicitely describes the offending character (comma, ampersand, etc.). | ||
Added ‑‑format=blastn .
| ||
Added idfrac , id% , blastid% ,
covfrac , cov% , confrac ,
con% , ncolumn , and npair fields for
‑‑format=general .
| ||
Added start..end+zoom% subrange specifier.
| ||
| 1.02.23 | Jan/10/2011 | Fixed a problem that occurred if the gap extension penalty was set to zero. This caused a divide by zero (which is reported in different ways on different platforms) and the program crashed. This has been corrected by trapping the offending division. However, the fix increases memory usage. Moreover, it is highly likely to cause truncated alignments. It’s not clear that there is any useful reason to set gap extension to zero. |
Added ‑‑format=rdotplot+score and
‑‑rdotplot+score=<file> .
| ||
Improved ‑‑masking=<count> so that it can allow a count
threshold greater than 254.
| ||
Fixed a problem with ‑‑scores=<scoring_file> . When the
<scoring_file> defined score values for N ,
those scores were not honored during the ungapped seed extension stage.
| ||
Fixed problems with ‑‑ambiguous=n and
‑‑ambiguous=iupac . These were
incorrectly penalizing substitutions between non-ambiguous nucleotides
(A, C, G, or T ) and ambiguous ones (N, B, D, H, K, M, R, S,
V, W, or Y ). This has been corrected to honor the original
intent, which was clearly to score these as zero.
However, for users who desire the previous behavior, a substitution penalty can now be specified with each of these options. To match the previous behavior, a penalty of twice the gap extension should be used. A later change history item is also relevant. | ||
Added ‑‑queryhsplimit=<n> .
| ||
| 1.02.27 | Jan/31/2011 |
Added ‑‑outputmasking=<file> .
|
| 1.02.37 | Mar/31/2011 |
Added ‑‑outputmasking:soft=<file> .
|
| Added example of filtering with shell commands. | ||
| Changed the interpretation of comments in sequence name files. Previously, the first # was considered a comment. The implemenation predated the author’s familiarity with Illumina read names (which contain a #). In order to still allow lines that contain a read name and a comment, a # is not considered a comment unless it is is preceded by whitespace or the start of the line. | ||
Changed the behavior of
‑‑queryhsplimit=<n> to
better match user expectations. Previously the limit was applied separately
for each strand of the query. Moreover, HSPs discovered before the limit was
reached were still passed downstream for further processing.
This has all been changed so that the limit applies to the combined total of HSPs for query, and if the limit is reached (exceeded), all HSPs for the read are discarded and no downstream processing is performed. | ||
Fixed a bug involving the ngap and cgap fields for
‑‑format=general . These fields were only reported correctly if
the continuity or ncolumn fields were also requested.
Otherwise, the value reported represented the contents of unitialized memory.
| ||
Added filtering options
‑‑filter=nmismatch:0..<max> ,
‑‑filter=ngap:0..<max> ,
and ‑‑filter=cgap:0..<max> .
Also changed the option name for match count filtering to
| ||
| 1.02.40 | Apr/7/2011 |
Added ‑‑outputmasking+=<file>
and ‑‑outputmasking+:soft=<file> .
|
Added
‑‑progress+masking=[<N>] .
This existed as an unadvertized option in earlier versions of the program, as
‑‑debug=queryprogress+masking=<N> . It has now been promoted
to a first class option.
| ||
| Added an example of how to create a soft-masked sequence by self-masking. | ||
| 1.03.00 | Jul/14/2011 |
When a subrange was used, the wrong denominator
was used to compute coverage. The denominator
used was the length of the subrange instead of the entire sequence. This
adversely affected both the
‑‑filter=coverage filter and the
coverage output field. This has been corrected
to use the length of the entire sequence.
|
Added the
separator=<character>
action, allowing the user to specify a character which alignments will not
cross. See also
Non-ACGT Characters, Splicing, and Separation.
| ||
| Added support for reading FASTQ files. Quality values do not participate in alignment, but are copied to alignment output when appropriate. | ||
Added ‑‑format=general fields
nucs1 ,
nucs2
(the entire target or query nucleotides sequence),
quals1 and
quals2
(the target or query base-call quality sequence).
| ||
Fixed a minor problem with the ‑‑format=general fields
cov% and con% . Those fields were being written with
an extra tab character preceeding them. This had a detrimental affect on
downstream parsers that required tabs as separators (parsers that interpreted
whitespace as separators were not affected).
| ||
Added ‑‑readgroup=<tags> ,
allowing the specification of tags for SAM's ‑RG header line.
| ||
Added
‑‑allocate:target=<bytes>
and
‑‑allocate:query=<bytes> .
These allow the user to predict the amount of memory needed to store target
or query sequence data, which in some instances can resolve memory overuse
(it saves LASTZ from incrementally predicting the amount of memory needed).
For consistency,
| ||
Added ‑‑include=<file> ,
allowing command-line arguments to be read from a text file.
| ||
|
Updated the Yasra shortcuts. Some options that
improved alignment read mapping had not previously been included in the
Yasra definitions, because these options did not exist when the Yasra
shortcuts were originally defined.
To allow backward compatibility, the shortcuts now permit specification of a particular version of LASTZ. See the description of the shortcuts for details. | ||
| 1.03.02 | Jul/19/2011 |
Fixed a bug in ‑‑format=axt and
‑‑format=axt+ , which caused every
alignment to be reported twice. The bug had been introduced in version
1.02.28 (not present in 1.02.27, present in 1.02.37).
|
| 1.03.34 | Apr/12/2013 |
Fixed a problem with ‑‑self and ‑‑format=lav ,
introduced in 1.02.11, which caused lastz to segfault.
|
Fixed a bug in ‑‑writecapsule . When the target was larger than
≈1 billion bp, an internal sanity check triggered incorrectly, stopping
the program and reporting "internal error writing to" the capsule file.
| ||
Fixed a bug related to ‑‑nogfextend . If no
‑‑gappedthresh was set, the gapped threshold incorrectly was set
to 0 instead of the correct default of 3000. This has been corrected.
| ||
The match count filter now allows the count to be specified as a percentage of
the query length
(‑‑filter=nmatch:<min>% ).
| ||
Added ‑‑format=general fields
number1 ,
number2 ,
number and
znumber
(sequence and alignment numbers).
| ||
Added ‑‑format=general fields
qalign1 and
qalign2
(quality sequences in alignment order).
| ||
Corrected rdotplot output when the
query files contains more than one sequence. Previously, the header line
containing sequence names was only written once, at the beginning of the file.
Now it is written once for each query sequence.
| ||
Added a warning when a scores file is used
(‑‑scores ), with the scale of the
scoring matrix substantially different from the default scoring matrix, and the
user hasn't set the hsp threshold or gapped threshold. This is a common
mistake and often results in no alignments being found.
The warning looks something like this: WARNING. Scores file may warrant setting of thresholds absent from scores.txt. Minimum match score is 10, for matrix entry (A,A). This may not work well with default --gappedthresh=3000. | ||
Added ‑‑queryhspbest=<n> .
| ||
Added ‑‑querydepth=<n> .
| ||
Added alignment chores files,
‑‑chores=<file> option,
chores=<file> action, and
chore field for
‑‑format=general .
See Aligning Many Subintervals.
| ||
Fixed a bug related to the
nickname=<name> action. If
the corresponding sequence file was in
2Bit format, the nickname wasn't used and sequence
names were copied from the sequence file. This has been corrected.
| ||
Added ‑‑help=defaults and
‑‑show=defaults .
| ||
|
Fixed a problem which caused runaway memory allocation of the traceback row
buffer. The problem was discovered when an alignment of a 72-bp read to a
reference genome needed to allocate 170 million rows (about 700M bytes). This
has been corrected.
It is not clear whether this had any affect on the alignments produced. In the examples used for testing and debugging, alignments were not affected. The negative affect was memory requirements and possibly runtime. However, the cause of the problem was incorrect determination of bounding alignments when performing gapped alignment backwards from the anchor. So it is possible that this could have caused a desirable alignment to have been missed, truncated, or to contain suboptimal gap placement. | ||
Corrected the behavior when
‑‑anyornone was used with
‑‑nogapped . Previously this failed
with a message indicating an internal error ("gapped_extend was given a NULL
traceback pointer.").
| ||
Score thresholds can now make use of units (e.g.
‑‑hspthresh=5K instead of
‑‑hspthresh=5000 ).
| ||
Detection of trivial self-alignments has been improved for cases where the
multiple action is used.
| ||
The implementation of ‑‑self has been
reworked so that it no longer reads the input file twice. As a result,
‑‑self now supports a file piped into stdin.
| ||
Added an additional build (lastz_32 ) to address aligning to whole
genomes larger than 2 gigabases.
| ||
Changed the option names for identity, continuity and coverage filtering to
‑‑filter=identity:<min>[..<max>] ,
‑‑filter=continuity:<min>[..<max>] ,
and
‑‑filter=coverage:<min>[..<max>] .
This change was made to achieve consistency with the other back-end filtering
options.
The older options, | ||
| 1.03.46 | Oct/2/2013 |
Added the namejoin action, to allow
better handling of input files that have spaces in sequence names (e.g. Illumina
casava version 1.8 fastq files).
|
| Greatly improved speed for the use case where the target contains a large number of sequences (e.g. 100 thousand exons). This was essentially a bug which had no effect on accuracy. A data structure was being repeatly allocated and erased, and was much larger than was needed (e.g. 12 Mbytes per query), and since all writes were cache misses, this ended up being very significant. | ||
| Added the allowBackToBackGaps build option. Previous versions of LASTZ (and BLASTZ) did not consider alignments in which an insert was immediately adjacent to a delete (or vice versa). | ||
Fixed a problem in ‑‑format=differences
that was inadvertantly introduced in 1.03.34. A failsafe check for an unhandled
case was added in that version, and ‑‑format=differences wasn't
being handled. Unfortunately this prevented this format from being usable.
This has been corrected.
| ||
| 1.03.52 | Jan/14/2014 |
Corrected a bug that occured when the
‑‑inner option is used with the
multiple action. With this
combination lastz could report alignments straddling two sequences. This is
now prevented.
This bug has existed in all previous versions of lastz. It was not in blastz since blastz did not provide the ability to have multiple sequences in memory. |
| 1.03.54 | Jan/28/2014 |
Modified ‑‑ambiguous=n and
‑‑ambiguous=iupac to allow a reward
for matches to be specified in addition to the penalty for mismatches.
An earlier change history item is also relevant. |
| 1.03.66 | Jan/19/2015 | Fixed an error that would cause a segfault. The causitive conditions at the user level are not well characterized. In the discovered example both target and query sequences contained 1K bp bursts of similar but highly diverged low complexity sequence, but the specific relationship (if there is one) between this feature and the failure are not known. Internally the problem resulted from an unsigned index variable attempting to become negative. |
| Fixed an error that would cause a segfault if fasta queries were piped from stdin and the first query was empty. | ||
| Added a sanity check to scoring inference. It is possible that, for a candidate scoring set, the enforcement of identity filter settings (min_identity and max_identity) leaves the inference with no alignments from which to infer scores. This condition should now result in a failure message. | ||
Changed how back-to-back gaps are represented in
lav format , to match the way
BLASTZ represented them. A zero-length segment is now written
to the file, separating the two gaps. It has been discovered that at least one
of the lav-processing tools in the Miller Lab suite expects to have such a
segment.
| ||
Fixed an error which prevented
‑‑progress[=<N>]
being reported for empty sequences.
| ||
| 1.03.73 | Jul/8/2015 |
Eliminated alignments that begin or end with gaps. Such alignments do not make
sense biologically.
Earlier versions could report a gap at either end of an alignment if the alignment was very close to an alignment found earlier in the process. This is a failure in the logic that prevents alignments from crossing (or overlapping) and/or assumptions made in extending HSPs when the anchor point is very close to a previously-found alignment. The current solution truncates the alignment by trimming away any end gaps and rescoring. |
| 1.04.00 | Mar/12/2017 |
Corrected a bug involving chaining. Previously, if
‑‑chain
and
[multiple]
were used together, the chaining algorithm incorrectly considered all sequences
together as a single entity, and found a single chain across all sequences.
This has been corrected, and the chaining algorithm is now performed independently for each sequence. |
|
Implemented a workaround for parsing conflicts between
sequence specifier actions and shells that use square
brackets for filename expansion.
In such shells appending any action to a filename, such as
| ||
| Fixed an error that would cause a segfault when there are more than about 32 million HSPs. The program now detects this error case and suggests ways to avoid the situation. | ||
| 1.04.03 | Nov/14/2019 |
Fixed a problem with blastn format when
‑‑nogapped was used. The q.end
column was incorrect (it had the query length concatentated to it).
For more details see issue 3 and issue 20 at the lastz github repository. |
There was an error in LAV output when an adaptive HSP threshold was used.
The HSP threshold is adaptive if
‑‑hspthresh=top<basecount>
or
‑‑hspthresh=top<percentage>%
is used. In this case, forward-strand alignments were not properly tagged as
forward strand. This has been corrected.
For more details see issue 14 at the lastz github repository. | ||
A potential overflow error with
‑‑inner was corrected. A 32-bit
absolute value function was used on 33-bit signed inputs. No occurrences of
this error have been reported; however, the error could have occurred and gone
unnoticed. I have not investigated in detail, but it probably could only have
occurred on alignments with coordinates more than 2Gbp apart.
This has been corrected. | ||
|
Additional self-tests have been added.
See the "testing" section in | ||
| Several changes were made to the source code to placate compiler warnings. These changes have no affect on the executable. | ||
| 1.04.15 | Aug/27/2021 |
Fixed a bug in which the setting for
‑‑queryhspbest
was ignored. For more details see
issue 28 at the lastz
github repository.
|
| Added several sanity checks to prevent possible segfaults. | ||
| Added PAF format. | ||
| Updated the python scripts in the tools directory to be compatible with python3. The older scripts have been moved to the tools.python2 directory. | ||
| Added code to properly distinguish between char and int types in parsing input files (some compilers treat chars as signed, other treat them as unsigned). For more details see issue 23 at the lastz github repository. | ||
| Added Makefile.warnings. | ||
| Some changes were made to the source code to placate compiler warnings. These changes have no affect on the executable. |
Bellman R (1957). Dynamic Programming. Princeton University Press, Princeton, NJ.
Buhler J, Keich U, Sun Y (2003). Designing seeds for similarity search in genomic DNA. Proc. 7th Annual International Conference on Research in Computational Molecular Biology (RECOMB '03), pp. 67-75.
Chiaromonte F, Yap VB, Miller W (2002). Scoring pairwise genomic sequence alignments. Pacific Symposium on Biocomputing 7:115-126.
Cock PJA, Fields CJ, Goto N, Heuer ML, Rice PM (2009). The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants. Nucleic Acids Research 38:1767-1771.
Gusfield D (1997). Algorithms on strings, trees and sequences. Cambridge University Press, Cambridge, pp. 244.
Harris RS (2007). Improved pairwise alignment of genomic DNA. Ph.D. thesis, Pennsylvania State University.
Li H et al. (2009). The Sequence Alignment/Map (SAM) format and SAMtools. Bioinformatics 25:2078-2079.
Myers EW, Miller W (1989). Approximate matching of regular expressions. Bull. Math. Biol. 51:5-37.
Zhang Z, Berman P, Miller W (1998). Alignments without low-scoring regions. J. Comput. Biol. 5:197-210.
Thanks for to Haibao Tang for contributing an example implemention for BLASTN output.
Bob Harris and Cathy Riemer


fun onPlayerPause(player: AudioPlayer): Unit
Fired by Lavaplayer when the AudioPlayer is paused.
This sends a PLAYER_PAUSED
event, wrapped in a Dispatch Response, across a WebSocketChannel.
Note: This sends the same event as when the player is resumed, but the data is set to true
(for "is paused").


A helper class to create Custom Post Types.
It handles:
- Creation of the post type (with a prefix by default)
- Adding templates within the plugin for singles and archives (using setupTemplates method).
- Allows basic settings to be set through chaining
Note: To make sure all translations are ready this should be called in an init action!
Usage (ordinary mode):
new \LengthOfRope\Treehouse\Register\PostType('products', array(
'labels' => array(
'name' => __('Products'),
'singular_name' => __('Product')
),
'public' => true,
'has_archive' => true,
));
Usage (chained mode):
\LengthOfRope\Treehouse\Register\PostType::factory('products')
->setLabels(__('Products'), __('Product'))
->setPublic(true)
->setHasArchive(true);
Methods summary
public
|
#
__construct( string $slug, array $args = array(), string $slugPrefix = 'th_' )
Setup a new posttype
Parameters
- $slug
- The unique slug for this post type.
- $args
- $slugPrefix
|
public static
|
#
factory( string $slug, array $args = array(), string $slugPrefix = 'th_' )
Factory to allow chaining
Factory to allow chaining
Parameters
- $slug
- The unique slug for this post type.
- $args
- $slugPrefix
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setLabel( string $label )
A plural descriptive name for the post type marked for translation.
A plural descriptive name for the post type marked for translation.
Parameters
- $label
- Descriptive name (plural)
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setLabels( string|array $labels, string|false $singularName = false )
An array of labels for this post type.
By default, post labels are used for non-hierarchical post types and page labels for hierarchical ones.
Default: if empty, 'name' is set to value of 'label', and 'singular_name' is set to value of 'name'.
An array of labels for this post type.
By default, post labels are used for non-hierarchical post types and page labels for hierarchical ones.
Default: if empty, 'name' is set to value of 'label', and 'singular_name' is set to value of 'name'.
Parameters
- $labels
- If (plural) string, and singularName is also used, auto-generates all labels
- $singularName
- If string and labels is plural string, auto-generates all labels
Returns
See
https://codex.wordpress.org/Function_Reference/register_post_type
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setDescription( string $description )
A short descriptive summary of what the post type is.
A short descriptive summary of what the post type is.
Parameters
- $description
- Summary of what the post type is.
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setPublic( boolean $boolean )
Controls how the type is visible to authors (show_in_nav_menus, show_ui) and
readers (exclude_from_search, publicly_queryable).
Default: false
'true' - Implies exclude_from_search: false,
publicly_queryable: true,
show_in_nav_menus: true, and
show_ui: true.
The built-in types attachment, page, and post are similar to this.
'false' - Implies exclude_from_search: true,
publicly_queryable: false,
show_in_nav_menus: false, and
show_ui: false.
The built-in types nav_menu_item and revision are similar to this.
Best used if you'll provide your own editing and viewing interfaces (or none at all).
Controls how the type is visible to authors (show_in_nav_menus, show_ui) and
readers (exclude_from_search, publicly_queryable).
Default: false 'true' - Implies exclude_from_search: false, publicly_queryable: true, show_in_nav_menus: true, and show_ui: true. The built-in types attachment, page, and post are similar to this. 'false' - Implies exclude_from_search: true, publicly_queryable: false, show_in_nav_menus: false, and show_ui: false. The built-in types nav_menu_item and revision are similar to this. Best used if you'll provide your own editing and viewing interfaces (or none at all).
If no value is specified for exclude_from_search, publicly_queryable, show_in_nav_menus, or show_ui,
they inherit their values from public.
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setExcludeFromSearch( boolean $boolean )
Whether to exclude posts with this post type from front end search results.
Default: value of the opposite of public argument
'true' - site/?s=search-term will not include posts of this post type.
'false' - site/?s=search-term will include posts of this post type.
Whether to exclude posts with this post type from front end search results.
Default: value of the opposite of public argument 'true' - site/?s=search-term will not include posts of this post type. 'false' - site/?s=search-term will include posts of this post type.
Note: If you want to show the posts's list that are associated to taxonomy's terms, you must set
exclude_from_search to false (ie : for call site_domaine/?taxonomy_slug=term_slug or
site_domaine/taxonomy_slug/term_slug). If you set to true, on the taxonomy page (ex: taxonomy.php) WordPress
will not find your posts and/or pagination will make 404 error...
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setPubliclyQueryable( boolean $boolean )
Whether queries can be performed on the front end as part of parse_request().
Default: value of public argument
Whether queries can be performed on the front end as part of parse_request().
Default: value of public argument
Note: The queries affected include the following (also initiated when rewrites are handled) - ?post_type={post_type_key} - ?{post_type_key}={single_post_slug} - ?{post_type_query_var}={single_post_slug}
Note: If query_var is empty, null, or a boolean FALSE, WordPress will still attempt to interpret it (4.2.2)
and previews/views of your custom post will return 404s.
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setShowUI( boolean $boolean )
Whether to generate a default UI for managing this post type in the admin.
Default: value of public argument
'false' - do not display a user-interface for this post type
'true' - display a user-interface (admin panel) for this post type
Whether to generate a default UI for managing this post type in the admin.
Default: value of public argument 'false' - do not display a user-interface for this post type 'true' - display a user-interface (admin panel) for this post type
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setShowInAdminBar( boolean $boolean )
Whether to make this post type available in the WordPress admin bar.
Default: value of the show_in_menu argument
Whether to make this post type available in the WordPress admin bar.
Default: value of the show_in_menu argument
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setHasArchive( boolean $boolean )
Enables post type archives. Will use $post_type as archive slug by default.
Default: false
Enables post type archives. Will use $post_type as archive slug by default.
Default: false
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setSupports( array|false $supports )
An alias for calling add_post_type_support() directly. As of 3.5, boolean false can be passed as value
instead of an array to prevent default (title and editor) behavior.
Default: title and editor
An alias for calling add_post_type_support() directly. As of 3.5, boolean false can be passed as value
instead of an array to prevent default (title and editor) behavior.
Default: title and editor
Possible default values:
- title
- editor
- author
- thumbnail
- excerpt
- trackbacks
- custom-fields
- comments
- revisions
- page-attributes
- post-formats
Parameters
- $supports
- Array of supported items
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setCapabilityType( string|array $capabilityType )
The string to use to build the read, edit, and delete capabilities. May be passed as an array to allow for
alternative plurals when using this argument as a base to construct the capabilities,
e.g. array('story', 'stories') the first array element will be used for the singular capabilities and
the second array element for the plural capabilities, this is instead of the auto generated version if no
array is given which would be "storys". The 'capability_type' parameter is used as a base to construct
capabilities unless they are explicitly set with the 'capabilities' parameter.
Default: "post"
The string to use to build the read, edit, and delete capabilities. May be passed as an array to allow for
alternative plurals when using this argument as a base to construct the capabilities,
e.g. array('story', 'stories') the first array element will be used for the singular capabilities and
the second array element for the plural capabilities, this is instead of the auto generated version if no
array is given which would be "storys". The 'capability_type' parameter is used as a base to construct
capabilities unless they are explicitly set with the 'capabilities' parameter.
Default: "post"
Parameters
- $capabilityType
- String or array with capability types
Returns
See
https://codex.wordpress.org/Function_Reference/register_post_type
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setCapabilities( array $capabilities )
An array of the capabilities for this post type.
Default: capability_type is used to construct
An array of the capabilities for this post type.
Default: capability_type is used to construct
By default, seven keys are accepted as part of the capabilities array:
- edit_post, read_post, and delete_post These three are meta capabilities, which are then generally mapped to corresponding primitive capabilities depending on the context, for example post being edited/read/deleted and the user or role being checked. Thus these capabilities would generally not be granted directly to users or roles.
- edit_posts - Controls whether objects of this post type can be edited.
- edit_others_posts - Controls whether objects of this type owned by other users can be edited. If the post type does not support an author, then this will behave like edit_posts.
- publish_posts - Controls publishing objects of this post type.
- read_private_posts - Controls whether private objects can be read.
Note 1: those last four primitive capabilities are checked in core in various locations.
There are also eight other primitive capabilities which are not referenced directly in core, except in
map_meta_cap(), which takes the three aforementioned meta capabilities and translates them into one or more
primitive capabilities that must then be checked against the user or role, depending on the context.
These additional capabilities are only used in map_meta_cap(). Thus, they are only assigned by default if
the post type is registered with the 'map_meta_cap' argument set to true (default is false).
- read - Controls whether objects of this post type can be read.
- delete_posts - Controls whether objects of this post type can be deleted.
- delete_private_posts - Controls whether private objects can be deleted.
- delete_published_posts - Controls whether published objects can be deleted.
- delete_others_posts - Controls whether objects owned by other users can be can be deleted. If the post type does not support an author, then this will behave like delete_posts.
- edit_private_posts - Controls whether private objects can be edited.
- edit_published_posts - Controls whether published objects can be edited.
- create_posts - Controls whether new objects can be created
Parameters
- $capabilities
- Array with capabilities
Returns
See
https://codex.wordpress.org/Function_Reference/register_post_type
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setMapMetaCap( boolean $boolean )
Whether to use the internal default meta capability handling.
Default: null
Whether to use the internal default meta capability handling.
Default: null
Note: If set it to false then standard admin role can't edit the posts types. Then the edit_post capability
must be added to all roles to add or edit the posts types.
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setHierarchical( boolean $boolean )
Whether the post type is hierarchical (e.g. page). Allows Parent to be specified. The 'supports' parameter
should contain 'page-attributes' to show the parent select box on the editor page.
Default: false
Whether the post type is hierarchical (e.g. page). Allows Parent to be specified. The 'supports' parameter
should contain 'page-attributes' to show the parent select box on the editor page.
Default: false
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setRegisterMetaBoxCB( function $callback )
Provide a callback function that will be called when setting up the meta boxes for the edit form.
The callback function takes one argument $post, which contains the WP_Post object for the currently edited post.
Do remove_meta_box() and add_meta_box() calls in the callback.
Default: None
Provide a callback function that will be called when setting up the meta boxes for the edit form.
The callback function takes one argument $post, which contains the WP_Post object for the currently edited post.
Do remove_meta_box() and add_meta_box() calls in the callback.
Default: None
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setTaxonomies( array $taxonomies )
An array of registered taxonomies like category or post_tag that will be used with this post type. This can
be used in lieu of calling register_taxonomy_for_object_type() directly. Custom taxonomies still need to be
registered with register_taxonomy().
Default: no taxonomies
An array of registered taxonomies like category or post_tag that will be used with this post type. This can
be used in lieu of calling register_taxonomy_for_object_type() directly. Custom taxonomies still need to be
registered with register_taxonomy().
Default: no taxonomies
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setRewrite( boolean|array $rewrite )
Triggers the handling of rewrites for this post type. To prevent rewrites, set to false.
Default: true and use $post_type as slug
Triggers the handling of rewrites for this post type. To prevent rewrites, set to false.
Default: true and use $post_type as slug
$args array 'slug' => string Customize the permalink structure slug. Defaults to the $post_type value. Should be translatable. 'with_front' => bool Should the permalink structure be prepended with the front base. (example: if your permalink structure is /blog/, then your links will be: false->/news/, true->/blog/news/). Defaults to true 'feeds' => bool Should a feed permalink structure be built for this post type. Defaults to has_archive value. 'pages' => bool Should the permalink structure provide for pagination. Defaults to true
Note: If registering a post type inside of a plugin, call flush_rewrite_rules() in your activation and
deactivation hook (see Flushing Rewrite on Activation below). If flush_rewrite_rules() is not used, then
you will have to manually go to Settings > Permalinks and refresh your permalink structure before your
custom post type will show the correct structure.
Parameters
- $rewrite
- array with rewrite stuff
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setQueryVar( boolean|string $value )
Sets the query_var key for this post type.
Default: true - set to $post_type
'false' - Disables query_var key use. A post type cannot be loaded at /?{query_var}={single_post_slug}
'string' - /?{query_var_string}={single_post_slug} will work as intended.
Sets the query_var key for this post type.
Default: true - set to $post_type 'false' - Disables query_var key use. A post type cannot be loaded at /?{query_var}={single_post_slug} 'string' - /?{query_var_string}={single_post_slug} will work as intended.
Note: The query_var parameter has no effect if the ‘publicly_queryable’ parameter is set to false. query_var
adds the custom post type’s query var to the built-in query_vars array so that WordPress will recognize it.
WordPress removes any query var not included in that array.
If set to true it allows you to request a custom posts type (book) using this: example.com/?book=life-of-pi
If set to a string rather than true (for example ‘publication’), do: example.com/?publication=life-of-pi
Parameters
- $value
- True or false or string
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setCanExport( boolean $boolean )
Can this post_type be exported.
Default: true
Can this post_type be exported.
Default: true
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setShowInRest( boolean $boolean )
Whether to expose this post type in the REST API.
Default: false
Whether to expose this post type in the REST API.
Default: false
Parameters
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setRestBase( string $restBase )
The base slug that this post type will use when accessed using the REST API.
Default: $post_type
The base slug that this post type will use when accessed using the REST API.
Default: $post_type
Parameters
- $restBase
- Base slug for REST API
Returns
|
public
LengthOfRope\Treehouse\Register\PostType
|
#
setRestControllerClass( string $restControllerClass )
An optional custom controller to use instead of WP_REST_Posts_Controller. Must be a subclass
of WP_REST_Controller.
Default: WP_REST_Posts_Controller
An optional custom controller to use instead of WP_REST_Posts_Controller. Must be a subclass
of WP_REST_Controller.
Default: WP_REST_Posts_Controller
Parameters
- $restControllerClass
- Controller class
Returns
|
public
|
#
setupRoutes( string $coreFile )
Setup template routes for this Post Type.
Setup template routes for this Post Type.
If theme does no provide the following files:
- single-{slug}.php
- archive-(slug}.php
This will check the plugin/ theme if the file exists in this location and serves that file
- /templates/single-{slug}.php
- /templates/archive-{slug}.php
Parameters
- $coreFile
- The FILE variable of the plugin/ theme location
|


The biggest and greatest Pop songs from the 90s. Nothing else needed!
Lots of hiphop songs to sing along to. Come sing with us!
A sweet selection of Indian and Bollywood songs, all in indian..
Prepare your throats.
Lots of hiphop songs to sing along to. Come sing with us!
Prepare your throats. Sing in 8-Bit! Like, share and subscribe!
The biggest and greatest Pop songs from the 90s. Nothing else needed!
A sweet selection of Indian and Bollywood songs, all in indian..


I was interested to read Don't Let Yourself Get Unitized, I guess because he refers to problems with JUnit, and unittest is based on JUnit, and I don't like unittest that much.
But the article turns into a curmudgeonly rant and his underlying resentment of unit testing comes out:
Consider the goals and tenents of unit testing:
- Very small "units" are tested
- Testing is almost always done of individual components in isolation from other components
- Mocking strengthens the isolation aspect
- The code and the tests are almost always written by the same person
Taken together this means that unit tests are testing the lowest level pieces of your code, each in turn and in isolation from all other pieces, and the definition of the tests and the code are done by the same person.
This sort of testing catches what I consider "low hanging fruit". It catches problems-in-the-small. It'll find individual methods or classes which don't match what the unit tests say should happen.
This is a good thing and provides very valuable feedback on the correctness of your code. But keep in mind it _only_ catches low hanging fruit. By design, unit testing is supposed to be easy, and to consider individual small pieces of a system in isolation. Because of this, by its very nature, unit testing does not consider the _composition_ of a system, only its individual parts. Unit tests never check the interconnections of an application, it never checks how they are wired together.
In my experience, the interconnections and "wiring" of an app is where most of the complexity of the application lies. The wiring defines your design, and if considered at a high enough level it can even be considered to capture your architecture. How information flows across many software layers and between many components really define what an application does. And the very definition of Unit Testing is that it does not test these aspects of an application. Unit testing ignores information flow across software layers and components, ignores how classes and objects are interrelated and are put together into larger designs and architecture. This means that unit tests can catch simple errors in individual pieces of code, but says nothing whatsoever about your system's design or architecture. And what makes or breaks an application really is the overall design and architecture. The design and architecture captures your system's performance, it's memory use, the "end-to-end" correctness from the user's inputs out to whatever servers you might be using, and the round trip back again. How all the wiring interconnects shows the true system behavior, and it is in this area where the toughest bugs and problems lie, and where people sweat blood to get things right. Writing individual components in isolation is easy. It's hooking them together into a cohesive whole that's hard - and unit tests only pass judgement on the individual parts in isolation, not the whole.
Getting one component to act "correctly" in a system is almost always a pretty trivial exercise. Writing one component in isolation is not the difficult part of computer programming. Any single small component of a system is generally easy to code. The hard part of development comes in getting all of the components of a system to work together - to get the wiring right. Unit tests can verify that each of your individual components does what you the developer thinks it should do. But by its very definition, unit testing cannot check the more complex "wiring" - and the wiring is where most of our design, development, and debugging time goes into.
There's also some weird digs at Martin Fowler (and I guess all consultants), which seems out of the blue and a little mean. The guy has a chip on his shoulder. But anyway, I'll respond with my thinking on unit tests.
First: Getting one component to act "correctly" in a system is almost always a pretty trivial exercise. Sure, getting it to act "correctly" is easy. Getting a component to act correctly without scare quotes is much harder.  When he's talking about "correctly" I think he really means "according to some spec that was delivered to the programmer." This is a symptom of defensive programming, where the programmer isn't at fault if there's a bug in the spec, or a bug in the larger design, or what-have-you. A responsible programmer cares about the success of the larger project, and judges correctness based on the utility and reliability of their code in that larger project, so there is no "correct" that does not include the system.
But another error is in how he views "units". A unit test is small and isolated, and tests a small amount of code. But a "small amount of code" is a matter of perspective. I shudder to think about the amount of code involved in the Python expression a = some_dict.copy(). There's the Python compiler, and VM, and the dictionary implementation, and the code behind classes and types, and underneath that is libc and the kernel and who knows what. And the combinations! Most of those pieces have multiple implementations; alternate dictionary-like objects, alternate VMs, many underlying platforms. We have created ourselves a Tower of Babel, an incredible monument to abstraction. But God hasn't struck us down... well, we have been "cursed" with a multitude of languages, but our efforts have not collapsed.
And yet despite all that underlying code, that one line is too small to be unit tested. So how did we get there? For the purpose of what we're testing we ignore the code and trust the abstractions. All those pieces underneath it are mature and well tested, having undergone years of development and testing by a large number of developers.
When I create a system, I have to build those same kinds of abstractions, reliable pieces that I can trust. When I later create code that depends on those libraries, I test that code, I do not test the library.
But unlike foundational code like libc, I can't put years of effort into building my abstractions. I have deadlines, and anyway I don't actually want to work on a single project for years at a time, all the way through to some time-based maturity, suffering through an unreliable interim and using QA departments and other systems that require bureaucracy.
Unit testing is in this way circular. Unit testing allows me to bring something to maturity faster, to practically will it to maturity. Along the way I have to make compromises -- I have to keep my code decoupled, and I have to write for testability. But I make those compromises because they allow me to move beyond my code, to build up things that don't need to be constantly revisited and retested according to context.
This is circular, because only with this foundation can I move to testing higher-level code in "isolation". Because of the Stable Dependencies Principle, I can't build something reliable and mature on top of something that is unreliable and immature. I can only make my code "isolated" if I have real trust that my underlying code is well defined and functional, that I can ignore it the way I ignore libc and the virtual machine. And I can only do that with unit tests.
I agree with most of what you have to say - I think most people who use unit-tests realize that you can't use them instead of having a well-understood QA process. Also, it's my understanding that unit tests are quite good at prevents previously found bugs from showing up again. When a defect is reported, _if possible?, you write a test for it and then it's added to the regression cycle.
I suspect a lot of people shudder when they hear the idea of programmers writing tests. But one of the benefits has to be that as a programmer, you really have to think about what you're doing before you do it. What is this one thing supposed to do. Before unit testing was popular, we required our developers to write function 'tombstones' (documentation about the name, who wrote it when, what it does, valid results, etc. etc.) before writing the code so that one would consider the atomic parts necessary. It helped.
ill agree that Spille shoots himself in the foot by ranting heavily and slathering on the quasi-ad-hominem stuff, but I like the points he makes at the end for integration tests, regression tests, and non-functional tests. I think we dont hear so much about those things on the one hand because they are so "old school" and we supposedly all know them already, but I think moreso because those techniques are hard and cant be easily transmitted into one-page soundbyte-types of articles and tutorials. and the rule of the day seems to be the technologies and techniques that can be learned by any newbie in half a day are the only ones that gain any kind of popularity.
the testing style i am familiar with on pretty much an intuitive basis, is a couple of unit tests just for highly algorithmic units of code, such as elaborate data structures or synchronization objects, and after that a lot of integration testing, functional testing, and regression testing, with a varied mix of automated test runners and manual tests, as is appropriate. so far, ive never felt so inclined to write unit tests for all those components that are obviously trivial (trivial to me, that is), which pretty much prove their functionality after i inspect their progress within some full-system tests (i.e. components where if they had any kind of flaw, theres no way the app could possibly work).
But, I am trying to "see the light" for a more unit-testing-fundamental approach and beginning to semi-begrudgingly write some unit tests for code i usually wouldnt bother with, and in fact it did reveal some dumb little non-bug-but-not-quite-correct behavior almost right out of the gate. But I'm also inclined to throw into the testing framework a combination of unit tests and the more integrative tests Ive already written, like multithreaded load-killer type tests and other front-to-back types of stuff. Since unittest, being minimal (and yes not that great i agree) just gives you "testXXXX", why even call it "unittest" ? you can put whatever kinds of tests you want in there.
# mike bayer
I had this impression.. let me state it: this is a java guy.
I don't mean he is using java I mean he is committed to the idea that if a thing is simple it is not valuable. The problem with unittest in python is a different beast, people actually want it to be more simple. Go figure.
 If I can't unit test the composition of components, how am I supposed to check if something work? Should I automate my tests by getting some slave who endlessly type the same input or should I use some kind of automated interface..mh.. like.. well.. a xUnit framework?
Also, he clearly states that unit testing "says nothing whatsoever about your system's design or architecture", which imo is wrong. The ease of testing is a direct heir of good design.


[§27-54] Data collection relating to the well-being of Hawaii's children and families. (a) The Hawaii performance partnerships board with the assistance of the center on the family at the University of Hawaii at Manoa, shall establish key indicators and data sets relevant to the health, education, and socioeconomic well-being of Hawaii's children and families.
(b) The Hawaii performance partnerships board may request and shall receive from every department, division, board, bureau, commission, or other agency of the State and its political subdivisions, cooperation and assistance in the performance of its duties relating to results and performance accountability, decision-making, and budgeting. [L 2001, c 274, §1]
Revision Note
Section was enacted as an addition to chapter 304 but is renumbered to this chapter pursuant to §23G-15.
Cross References
Departmental data sharing (improving educational and workforce outcomes), see §27-7.