Links
Table of Contents
Rimu Tips
Click the Edit icon ✎ in the examples below to enable live editing of the Rimu source; click the Preview icon ⬄ to toogle output between HTML and Preview.
HTML is valid Rimu
You are free to mix any valid HTML into a Rimu document — no
special syntax is required (in this sense Rimu can be thought of as an
HTML superset). Just remember that included HTML is only processed if
safeMode
is zero.
Converting Rimu to PDF
First convert Rimu markup to HTML using the rimuc tool and then convert the HTML to PDF. Here are a few options for converting HTML to PDF:
- Use Libre Office Writer or Microsoft Word to open the HTML document then export it to a PDF file.
- Use a command-line application such as wkhtmltopdf to convert HTML to PDF.
- Google's Chrome web browser has an option to print HTML to a PDF file.
Using Rimu Macros
Macros provide a simple, consistent mechanism for extending the Rimu markup vocabulary.
Examples
| Rimu Markup |
|
| HTML | Preview
Fun with MacrosHeads up! See the Rimu Playground for documentation and hands-on experimentation. Note This document is not yet finished. |
- The
{sidebar}
macro shows how you can inject CSS styles into a Division Block (you can also use use HTML
- Add a new block style. The following example puts a green border
around blocks with the
green-border
class: .green-border Green is the color of envy.
- Create new themes and tweak existing themes.
The style
element must not contain any blank lines because
Rimu HTML block elements are terminated with a blank line
Themes
A theme is a layout styling variation. The
--layouts
option supports a number of built-in
themes. You can create your own themes
by including custom CSS style elements.
Here is an example theme named redheads:
{--theme!.*\bredheads\b.*}.+skip

The HTML style
element will be included in the output when the
--theme
macro value includes the word redheads
, for example:
rimuc --layout sequel --theme redheads document.rmu
You can combine themes by specifying more than one theme, for example:
rimuc --layout sequel --theme "graystone redheads" document.rmu
If you specify multiple themes their CSS styles will cascade in the order they are processed in the Rimu source documents.
Admonition, sidebar and verse styles
The rimuc command --layout
option includes CSS classes for styling
paragraphs and Division blocks whose content sits outside the main narrative
flow.
The CSS class names are injected using the Block Attibutes element or by appending them to the opening block delimiter.
important
, note
, tip
and warning
styles
The important
, note
, tip
and warning
styles render messages and
admonitions. The optional notitle
style class can be used to hide the
admonition's title.
Examples
| Rimu Markup | HTML | Preview
|
Admonition styled paragraph. Admonition styled paragraph without a title. Admonition styled Division Block. Neque a massa. Nunc mauris tempor. Pede mauris sed. Scelerisque feugiat massa alias. Admonition styled Division Block without a title. Neque a massa. Nunc mauris tempor. Pede mauris sed. Scelerisque feugiat massa alias. |
sidebar
style
The sidebar
style renders text boxes containing digressions and observations
related to the main narrative flow.
Examples
| Rimu Markup | HTML | Preview
|
verse
style
The verse
style renders passages of text that retain line breaks.
See also Citing quotes and verses
and Controlling text alignment.
Examples
| Rimu Markup | HTML | Preview
|
Verse styled paragraph. Neque a massa. Porttitor consectetuer commodo. Centered verse styled Division BlockNeque a massa. Porttitor consectetuer commodo. Nunc mauris tempor. Pede mauris sed. Scelerisque feugiat massa alias. |
Using Macro, Replacement and Quote definitions in Safe Mode
Macro, Replacement and Quote definitions are not processed in Safe Mode. To apply Macro, Replacement or Quote definitions to Rimu markup that is processed in Safe Mode you need to load the definitions with a separate API call, for example:
Rimu.render(trusted_rimu_definitions, {safeMode: 0});
var html = Rimu.render(untrusted_rimu_markup, {safeMode: 5});
Mathematical formulae
The rimuc command supports MathJax LaTeX
and MathML mathematical formulas. To include MathJax support use
the rimuc command --mathjax
option with the --layout
option.
Examples
| Rimu Markup |
|
| HTML Preview |
A LaTeX inline formula: \(\sum_{i=0}^n i^2 = \frac{n^2+n}m{2}\) A LaTeX block formula: \[\sum_{i=0}^n i^2 = \frac{n^2+n}m{2}\] A MathML inline formula: A MathML block formula: |
- Macro expansion has been disabled in LaTeX formulas to stop
{i=0}
and{2}
in the LaTeX formualae from being mistaken for Rimu macro invocations.
See the latest MathJax documentation to learn how to use MathJax.
Conditional Inclusion
Macro invocations can be used to conditionally include and exclude Rimu source text. Here are some examples:
// Include script if `foo` value is non-blank.
{foo!}
// Skip paragraph if `foo` is blank.
{foo=}.+skip
Display this paragraph if macro `foo` is non-blank.
// Conditional inclusion of arbitrary blocks of Rimu source.
// Include some CSS if both `foo` and `bar` are non-blank.
// Works because a Division Block does not emit div tags if they
// have no HTML attributes.
{foo=}.+skip
{bar=}.+skip
..

..
// Inline examples.
This line is included {foo=} if the value of foo is blank.
This line is included {foo!} if the value of foo is not blank.
This line is included {foo=one} if the value of foo is 'one'.
This line is included {foo=two} if the value of foo is 'two'.
This line is included {foo!\d+} if the value of foo is not a number.
Use Comments to unconditionally exclude text.
Be careful when mixing conditional and simple macros on the same line, for example:
{foo!}{multi-line-macro}
Because the simple macro is expanded before the conditional macro only the
first line of the expanded {multi-line-macro}
will be subject to {foo!}
exclusion. Use the +skip
block option attribute to exclude the entire
{multi-line-macro}
, for example:
{foo=}.+skip
{multi-line-macro}
Tables
Create tables using Block HTML elements (Rimu has no dedicated syntax for tables but you can use macros to simplify table creation).
HTML blocks must start at the left margin and end with a blank line.
The rimuc --layout
option includes the bordered
CSS class for
styling bordered tables.
Examples
| Rimu Markup |
|
||
| HTML | Preview
|
The use of inline HTML for tables instead of a Rimu-specific table syntax was a deliberate design decision:
- A large number of formatting options (e.g. for headers and footers; cell widths, heights, alignment and spans) are required to accommodate all but the simplest of tables. This makes for a complex and noisy syntax — precisely what we're trying to avoid.
- Coming up with a syntax that is both recognizably tabular and easily written is difficult — imagine trying to write the above example in a way that maintains the side-by-side cell layout.
- Editing tabular layouts is unwieldy — text editors are designed to edit line-oriented text.
- At first blush tables come across as a hugely useful feature, but most documentation makes surprising little use of tables.
See also Use macros to create tables.
Use macros to create tables
Creating tables with raw HTML is repetitive, time consuming and difficult to read.
Parametrized macros can be used to generate row oriented tables:
Table macro example 1
| Rimu Markup |
|
||||||||||||||||
| HTML | Preview
|
Table macro example 2
| Rimu Markup |
|
||||||||||||
| HTML | Preview
|
- Block HTML elements must be followed by a blank line.
- The macro invocations emit HTML (block) elements and must be contained on a single line otherwise they will be processed as paragraphs.
- Double-dollar macro parameters are necessary because text formatting is not expanded and special characters are not escaped inside Block HTML elements.
Macros can also be used to simplify columnar text layouts:
Columnar text layouts
| Rimu Markup |
|
|||||
| HTML | Preview
Two column table
Three column table
|
Image links
Use raw HTML or a macro:
Image links
| Rimu Markup |
|
| HTML | Preview
Inline citations
The following example shows how macros can be used to generate inline citations.
Inline citations
| Rimu Markup |
|
| HTML | Preview
Purus aliquam mauris a aliquam orci [LFB42] . Massa sit sit. Penatibus. Maecenas at tellus [LFB28] . Sapien quam mauris. Non a pede in. |
The {LBF}
parameterised macro cites numbered sections from a publication using the
HTML abbr
element
to create tooltips.
Meta Macros
You can define macros which, when invoked, generate other macros.
The Footnotes macros example generates footnote links and footnotes. The Table of contents macros example generates document section headers, links and table of contents entries from a section identifier and title.
Footnotes macros
| Rimu Markup |
|
| HTML | Preview
Table of contents macros
| Rimu Markup |
|
| HTML | Preview
The backslash character is used to continue macro definition lines that end with a single-quote.
Macro tips
- Use the
|
character entity to display a|
character inside a macro parameter value. - Closing block delimiters must be explicit and cannot be sourced from a macro invocation (because Rimu elements are recognized before macro expansion).
- A macro invocation can be rendered verbatim in a code block by
disabling macro expansion. For example:
.-macros `` {example-42} ``
- Macro definition elements cannot be prefixed with an Inclusion/Exclusion macro (see Conditional Inclusion).
- Macro invocations in macro definition values are expanded when the
macro is declared, not when it is invoked. You can defer evaluation
until the macro is invoked by escaping macro invocations or (in the
case of multi-line definitions) by using the
-macros
Block Attributes option. For example this Rimu markup:{m1} = 'foo' {m2} = '{m1} \{m1}' {m2} {m1} = 'bar' {m2}
Generates:

foo foo

foo bar

List item counters
In addition to the verse and sidebar CSS styles the rimuc tool --layout
option also includes:
dl-counter
,ol-counter
andul-counter
CSS classes for counting labeled, numbered and bullet list items.{--dl-counter}
,{--ol-counter}
and{--ul-counter}
counter macros.- The
dl-numbered
CSS class for numbering labeled list items.
Examples
| Rimu Markup | HTML | Preview
|
Numbered labeled list
labeled list items. Counted unordered list
There are unordered list items. Counted ordered list
There are ordered list items. |
Append arbitrary elements to list items
An attached Division Block can be used to group and append arbitrary block elements to a list item.
Examples
| Rimu Markup | HTML | Preview
|
|
This works because if a Fenced Block immediately follows a list item without any intervening blank lines it will be rendered with the list item.
Applying Block Attributes to list item elements
Block Attribute elements within a list are applied to the next list item or attached list item element. This facilitates styling and linking within lists.
Examples
| Rimu Markup | HTML | Preview
|
|
Section numbering
Use the rimuc --section-numbers
option with the --layout
option
to number top level h2 and h3 sub-sections. For example:
rimuc --layout sequel --section-numbers mydoc.rmu
The rimuc --section-numbers
option is a shortcut for
--prepend "{--section-numbers} = 'yes'"
.
Nested Fenced Blocks
Fenced blocks of the same type can be nested by using differing numbers of delimiter characters to distinguish between them.
Examples
| Rimu Markup | HTML | Preview
|
|
List termination
List termination can be forced by following the list by two or more blank lines. This comes in handy if you want to follow one list immediately by another or if you want to separate an Indented paragraph from a preceding list.
Syntax surprises
Here are some situations that generate unexpected results.
Nested list item ambiguity
For readability Rimu list items can be optionally indented. If an Indented paragraph looks like a list item it will be rendered as a list item (because Lists take precedence over Indented paragraphs). To render an indented list item as an Indented paragraph change it to a fenced code block.
Examples
| Rimu Markup | HTML | Preview
|
|
Labelled list and delimited block ambiguity
If the first line of a delimited block ends with two colons it will be mistaken for a labelled list. For example, enclosing a labelled list in an HTML block comment. The problem arises because lists take precedence over delimited blocks. The work-around is to put the opening block delimiter on a separate line.
Examples
| Rimu Markup | HTML | Preview
|
|
This issue only applies to HTML Block, Macro Definition and Normal Paragraph delimited blocks.
URLs
- To include a
[
character in a Markdown URL or image captions, use a[
entity instead of the[
character.The caption in the following example displays as
the [Main] section
(if the[
character were used the caption would beMain] section
):[the [Main] section](#main)
- To include a
)
character in a Markdown URL , use a)
entity instead of the)
character. For example:[wolves](#the_wolf_(lupus))
- To include a
>
character in a Rimu URL or image captions, use a>
entity instead of the>
character.The caption in the following example displays as

(if the>
character were used the caption would be<Header
):<#header|>
Braces in macro patterns
You need to escape closing brace characters in Inclusion and Exclusion macro patterns. For example:
{foo=x{2\}}
Escaping the brace serves two purposes: It excludes {2}
from being
seen as a Simple macro invocation and it stops the brace character
from being seen as the Inclusion macro's closing brace
Escaped macro definitions
Macro definitions can be escaped by prefixing \
(zero-width space character followed by backslash character). Simply prefixing a
backslash is insufficent because the unescaped definition will be mismatched
downstream.
Examples
| Rimu Markup | HTML | Preview
|
{macro} = 'escaped definition' |
Preserving line breaks
There are three ways to preserve line breaks:
- Terminate lines with a Rimu line break (a space followed by a backslash character).
- Terminate lines with an HTML break element (

). - Apply the
preserve-breaks
class to a Rimu block element and compile with therimuc
command--layout
option.
Examples
| Rimu Markup | HTML | Preview
|
Urna aliquam pid Urna aliquam pid Aenean enim cum sed Augue non egestas amet? |
Controlling printer page breaks
The rimuc tool --layout
option includes page-break
and
no-page-break
CSS classes which apply page-break-before:always
and
page-break-inside:avoid
properties respectively:
.sidebar no-page-break
..
Page breaks are avoided inside
this sidebar.
..
.page-break
### A header that starts on a new page
Controlling text alignment
The rimuc tool --layout
option includes align-left
,
align-center
and align-right
CSS classes to apply the CSS
text-align
property. Use them to control the horizontal alignment
of flowed inline elements (e.g. text, inline images):
Examples
| Rimu Markup | HTML | Preview
|
Enim odio ultrices Enim odio ultrices Enim odio ultrices |
Citing quotes and verses
The rimuc command --layout
option includes the CSS cite class
for attributing quotes and verses. For example:
Examples
| Rimu Markup | HTML | Preview
|
Quotation:
Verse: Urna aliquam pid Aenean enim cum sed Augue non egestas amet? A. N. Mouse |
An example letter
Here's an example of a personal letter:
Examples
| Rimu Markup |
|
| HTML | Preview
24 Milky Way 17th June 2016 Mr. A. N. Mouse Dear Andrew Mattis pellentesque in lundium lectus penatibus turpis enim enim in, mid velit. Eu lundium. Lundium! Porttitor! Porttitor purus lacus et odio nisi amet in magna penatibus odio amet? In urna rhoncus! Nec scelerisque enim in turpis pid, phasellus cras, rhoncus, egestas aliquet lundium! Lundium auctor vel natoque enim. Yours sincerely |
- The font parameters are set by enclosing the entire contents in a Division Block.
- If you want to control the print margins put something like this at
the start of the document:

General purpose macro processor
You can use Rimu as a general purpose macro text processor. This is achieved by enveloping the text to be processed in a Division Block. For example:
// Macro definitions.
{sans-font} = 'Arial, Helvetica, sans-serif'
{primary-color} = '#527bbd'
{primary-background} = 'white'
{text-color} = '#333333'
// Only expand macros.
.+macros -specials -spans -container
..
body {
font-family: {sans-font};
font-size: 14px;
line-height: 20px;
color: {text-color};
background-color: {primary-background};
margin: 1em;
max-width: 50em;
}
h1, h2, h3, h4, h5, h6 {
margin: 10px 0;
color: {primary-color};
}
..
- Use
rimuc
to expand the macros, for example:rimuc --output main.css main.rmu
- This technique can be combined with conditional macro expansion and all the other Rimu macro features.
Passing arbitrary text to the output
To pass arbitrary blocks of text directly to the output disable all processing options. For example:
// Disable all processing inside Division Block.
.-macros -spans -specials -container
..
function toggleToc() {
document.getElementById("toc").classList.toggle('toc-visible');
}
window.onclick = toggleToc
..
To enable only macro expansion drop the -macros
option
(c.f. General purpose macro processor).
Building websites with Rimu
Take a look at hindsite if you want to build websites from Rimu source.
hindsite is a fast, lightweight static website generator. It builds static websites from Rimu or Markdown source documents.
Keyword comments
It's often useful to categorise comments by a keword e.g. todo, fixme. The following Rimu replacement definition implements keyword comments (it's in the example .rimurc file).
/\\?@(deprecated|fixme|important|note|todo|warning)\b.*/ = ''
- A keyword comment comprises a keyword plus any text following it on the same line.
- Keyword comments are ommitted from the Rimu HTML output.
- Prefixing the keyword with an
@
character is simply a convention to minimise ambiguity.
Examples
| Rimu Markup | HTML Preview |
|
Tincidunt proin lorem Duis ipsum nam eget vitae ut consequat quisque duis mollis magna arcu. |


Load balancing¶
When a filter needs to acquire a connection to a host in an upstream cluster, the cluster manager uses a load balancing policy to determine which host is selected. The load balancing policies are pluggable and are specified on a per upstream cluster basis in the configuration. Note that if no active health checking policy is configured for a cluster, all upstream cluster members are considered healthy.
Supported load balancers¶
Weighted round robin¶
This is a simple policy in which each healthy upstream host is selected in round robin order. If weights are assigned to endpoints in a locality, then a weighted round robin schedule is used, where higher weighted endpoints will appear more often in the rotation to achieve the effective weighting.
Weighted least request¶
The least request load balancer uses different algorithms depending on whether any of the hosts have weight greater than 1.
all weights 1: An O(1) algorithm which selects two random healthy hosts and picks the host which has fewer active requests (Research has shown that this approach is nearly as good as an O(N) full scan). This is also known as P2C (power of two choices). The P2C load balancer has the property that a host with the highest number of active requests in the cluster will never receive new requests. It will be allowed to drain until it is less than or equal to all of the other hosts.
all weights not 1: If any host in the cluster has a load balancing weight greater than 1, the load balancer shifts into a mode where it uses a weighted round robin schedule in which weights are dynamically adjusted based on the host’s request load at the time of selection (weight is divided by the current active request count. For example, a host with weight 2 and an active request count of 4 will have a synthetic weight of 2 / 4 = 0.5). This algorithm provides good balance at steady state but may not adapt to load imbalance as quickly. Additionally, unlike P2C, a host will never truly drain, though it will receive fewer requests over time.
Note
If all weights are not 1, but are the same (e.g., 42), Envoy will still use the weighted round robin schedule instead of P2C.
Ring hash¶
The ring/modulo hash load balancer implements consistent hashing to upstream hosts. The algorithm is based on mapping all hosts onto a circle such that the addition or removal of a host from the host set changes only affect 1/N requests. This technique is also commonly known as “ketama” hashing. A consistent hashing load balancer is only effective when protocol routing is used that specifies a value to hash on. The minimum ring size governs the replication factor for each host in the ring. For example, if the minimum ring size is 1024 and there are 16 hosts, each host will be replicated 64 times. The ring hash load balancer does not currently support weighting.
When priority based load balancing is in use, the priority level is also chosen by hash, so the endpoint selected will still be consistent when the set of backends is stable.
Note
The ring hash load balancer does not support locality weighted load balancing.
Maglev¶
The Maglev load balancer implements consistent hashing to upstream hosts. It uses the algorithm described in section 3.4 of this paper with a fixed table size of 65537 (see section 5.3 of the same paper). Maglev can be used as a drop in replacement for the ring hash load balancer any place in which consistent hashing is desired. Like the ring hash load balancer, a consistent hashing load balancer is only effective when protocol routing is used that specifies a value to hash on.
In general, when compared to the ring hash (“ketama”) algorithm, Maglev has substantially faster table lookup build times as well as host selection times (approximately 10x and 5x respectively when using a large ring size of 256K entries). The downside of Maglev is that it is not as stable as ring hash. More keys will move position when hosts are removed (simulations show approximately double the keys will move). With that said, for many applications including Redis, Maglev is very likely a superior drop in replacement for ring hash. The advanced reader can use this benchmark to compare ring hash versus Maglev with different parameters.
Random¶
The random load balancer selects a random healthy host. The random load balancer generally performs better than round robin if no health checking policy is configured. Random selection avoids bias towards the host in the set that comes after a failed host.
Original destination¶
This is a special purpose load balancer that can only be used with an original destination cluster. Upstream host is selected based on the downstream connection metadata, i.e., connections are opened to the same address as the destination address of the incoming connection was before the connection was redirected to Envoy. New destinations are added to the cluster by the load balancer on-demand, and the cluster periodically cleans out unused hosts from the cluster. No other load balancing type can be used with original destination clusters.
Original destination host request header¶
Envoy can also pick up the original destination from a HTTP header called
x-envoy-orignal-dst-host.
Please note that fully resolved IP address should be passed in this header. For example if a request has to be
routed to a host with IP address <IP_ADDRESS> at port 8888, the request header value should be set as
<IP_ADDRESS>:8888
.
Panic threshold¶
During load balancing, Envoy will generally only consider healthy hosts in an upstream cluster. However, if the percentage of healthy hosts in the cluster becomes too low, Envoy will disregard health status and balance amongst all hosts. This is known as the panic threshold. The default panic threshold is 50%. This is configurable via runtime as well as in the cluster configuration. The panic threshold is used to avoid a situation in which host failures cascade throughout the cluster as load increases.
Note that panic thresholds are per-priority. This means that if the percentage of healthy nodes in a single priority goes below the threshold, that priority will enter panic mode. In general it is discouraged to use panic thresholds in conjunction with priorities, as by the time enough nodes are unhealthy to trigger the panic threshold most of the traffic should already have spilled over to the next priority level.
Priority levels¶
During load balancing, Envoy will generally only consider hosts configured at the highest priority level. For each EDS LocalityLbEndpoints an optional priority may also be specified. When endpoints at the highest priority level (P=0) are healthy, all traffic will land on endpoints in that priority level. As endpoints for the highest priority level become unhealthy, traffic will begin to trickle to lower priority levels.
Currently, it is assumed that each priority level is over-provisioned by a (hard-coded) factor of 1.4. So if 80% of the endpoints are healthy, the priority level is still considered healthy because 80*1.4 > 100. As the number of healthy endpoints dips below 72%, the health of the priority level goes below 100. At that point the percent of traffic equivalent to the health of P=0 will go to P=0 and remaining traffic will flow to P=1.
Assume a simple set-up with 2 priority levels, P=1 100% healthy.
| P=0 healthy endpoints | Percent of traffic to P=0 | Percent of traffic to P=1 |
|---|---|---|
| 100% | 100% | 0% |
| 72% | 100% | 0% |
| 71% | 99% | 1% |
| 50% | 70% | 30% |
| 25% | 35% | 65% |
| 0% | 0% | 100% |
If P=1 becomes unhealthy, it will continue to take spilled load from P=0 until the sum of the health P=0 + P=1 goes below 100. At this point the healths will be scaled up to an “effective” health of 100%.
| P=0 healthy endpoints | P=1 healthy endpoints | Traffic to P=0 | Traffic to P=1 |
|---|---|---|---|
| 100% | 100% | 100% | 0% |
| 72% | 72% | 100% | 0% |
| 71% | 71% | 99% | 1% |
| 50% | 50% | 70% | 30% |
| 25% | 100% | 35% | 65% |
| 25% | 25% | 50% | 50% |
As more priorities are added, each level consumes load equal to its “scaled” effective health, so P=2 would only receive traffic if the combined health of P=0 + P=1 was less than 100.
| P=0 healthy endpoints | P=1 healthy endpoints | P=2 healthy endpoints | Traffic to P=0 | Traffic to P=1 | Traffic to P=2 |
|---|---|---|---|---|---|
| 100% | 100% | 100% | 100% | 0% | 0% |
| 72% | 72% | 100% | 100% | 0% | 0% |
| 71% | 71% | 100% | 99% | 1% | 0% |
| 50% | 50% | 100% | 70% | 30% | 0% |
| 25% | 100% | 100% | 35% | 65% | 0% |
| 25% | 25% | 100% | 25% | 25% | 50% |
To sum this up in pseudo algorithms:
load to P_0 = min(100, health(P_0) * 100 / total_health)
health(P_X) = 140 * healthy_P_X_backends / total_P_X_backends
total_health = min(100, Σ(health(P_0)...health(P_X))
load to P_X = 100 - Σ(percent_load(P_0)..percent_load(P_X-1))
Zone aware routing¶
We use the following terminology:
- Originating/Upstream cluster: Envoy routes requests from an originating cluster to an upstream cluster.
- Local zone: The same zone that contains a subset of hosts in both the originating and upstream clusters.
- Zone aware routing: Best effort routing of requests to an upstream cluster host in the local zone.
In deployments where hosts in originating and upstream clusters belong to different zones Envoy performs zone aware routing. There are several preconditions before zone aware routing can be performed:
- Both originating and upstream cluster are not in panic mode.
- Zone aware routing is enabled.
- The originating cluster has the same number of zones as the upstream cluster.
- The upstream cluster has enough hosts. See here for more information.
The purpose of zone aware routing is to send as much traffic to the local zone in the upstream cluster as possible while roughly maintaining the same number of requests per second across all upstream hosts (depending on load balancing policy).
Envoy tries to push as much traffic as possible to the local upstream zone as long as roughly the same number of requests per host in the upstream cluster are maintained. The decision of whether Envoy routes to the local zone or performs cross zone routing depends on the percentage of healthy hosts in the originating cluster and upstream cluster in the local zone. There are two cases with regard to percentage relations in the local zone between originating and upstream clusters:
- The originating cluster local zone percentage is greater than the one in the upstream cluster. In this case we cannot route all requests from the local zone of the originating cluster to the local zone of the upstream cluster because that will lead to request imbalance across all upstream hosts. Instead, Envoy calculates the percentage of requests that can be routed directly to the local zone of the upstream cluster. The rest of the requests are routed cross zone. The specific zone is selected based on the residual capacity of the zone (that zone will get some local zone traffic and may have additional capacity Envoy can use for cross zone traffic).
- The originating cluster local zone percentage is smaller than the one in upstream cluster. In this case the local zone of the upstream cluster can get all of the requests from the local zone of the originating cluster and also have some space to allow traffic from other zones in the originating cluster (if needed).
Note that when using multiple priorities, zone aware routing is currently only supported for P=0.
Locality weighted load balancing¶
Another approach to determining how to weight assignments across different zones and geographical locations is by using explicit weights supplied via EDS in the LocalityLbEndpoints message. This approach is mutually exclusive with the above zone aware routing, since in the case of locality aware LB, we rely on the management server to provide the locality weighting, rather than the Envoy-side heuristics used in zone aware routing.
When all endpoints are healthy, the locality is picked using a weighted round-robin schedule, where the locality weight is used for weighting. When some endpoints in a locality are unhealthy, we adjust the locality weight to reflect this. As with priority levels, we assume an over-provision factor (currently hardcoded at 1.4), which means we do not perform any weight adjustment when only a small number of endpoints in a locality are unhealthy.
Assume a simple set-up with 2 localities X and Y, where X has a locality weight of 1 and Y has a locality weight of 2, L=Y 100% healthy.
| L=X healthy endpoints | Percent of traffic to L=X | Percent of traffic to L=Y |
|---|---|---|
| 100% | 33% | 67% |
| 70% | 33% | 67% |
| 69% | 32% | 68% |
| 50% | 26% | 74% |
| 25% | 15% | 85% |
| 0% | 0% | 100% |
To sum this up in pseudo algorithms:
health(L_X) = 140 * healthy_X_backends / total_X_backends
effective_weight(L_X) = locality_weight_X * min(100, health(L_X))
load to L_X = effective_weight(L_X) / Σ_c(effective_weight(L_c))
Note that the locality weighted pick takes place after the priority level is picked. The load balancer follows these steps:
- Pick priority level.
- Pick locality (as described in this section) within priority level from (1).
- Pick endpoint using cluster specified load balancer within locality from (2).
Locality weighted load balancing is configured by setting locality_weighted_lb_config in the cluster configuration and providing weights in LocalityLbEndpoints via load_balancing_weight.
This feature is not compatible with load balancer subsetting, since it is not straightforward to reconcile locality level weighting with sensible weights for individual subsets.
Load Balancer Subsets¶
Envoy may be configured to divide hosts within an upstream cluster into subsets based on metadata attached to the hosts. Routes may then specify the metadata that a host must match in order to be selected by the load balancer, with the option of falling back to a predefined set of hosts, including any host.
Subsets use the load balancer policy specified by the cluster. The original destination policy may not be used with subsets because the upstream hosts are not known in advance. Subsets are compatible with zone aware routing, but be aware that the use of subsets may easily violate the minimum hosts condition described above.
If subsets are configured and a route
specifies no metadata or no subset matching the metadata exists, the subset load balancer initiates
its fallback policy. The default policy is NO_ENDPOINT
, in which case the request fails as if
the cluster had no hosts. Conversely, the ANY_ENDPOINT
fallback policy load balances across all
hosts in the cluster, without regard to host metadata. Finally, the DEFAULT_SUBSET
causes
fallback to load balance among hosts that match a specific set of metadata.
Subsets must be predefined to allow the subset load balancer to efficiently select the correct subset of hosts. Each definition is a set of keys, which translates to zero or more subsets. Conceptually, each host that has a metadata value for all of the keys in a definition is added to a subset specific to its key-value pairs. If no host has all the keys, no subsets result from the definition. Multiple definitions may be provided, and a single host may appear in multiple subsets if it matches multiple definitions.
During routing, the route’s metadata match configuration is used to find a specific subset. If there is a subset with the exact keys and values specified by the route, the subset is used for load balancing. Otherwise, the fallback policy is used. The cluster’s subset configuration must, therefore, contain a definition that has the same keys as a given route in order for subset load balancing to occur.
This feature can only be enabled using the V2 configuration API. Furthermore, host metadata is only
supported when using the EDS discovery type for clusters. Host metadata for subset load balancing
must be placed under the filter name "envoy.lb"
. Similarly, route metadata match criteria use
the "envoy.lb"
filter name. Host metadata may be hierarchical (e.g., the value for a top-level
key may be a structured value or list), but the subset load balancer only compares top-level keys
and values. Therefore when using structured values, a route’s match criteria will only match if an
identical structured value appears in the host’s metadata.
Examples¶
We’ll use simple metadata where all values are strings. Assume the following hosts are defined and associated with a cluster:
| Host | Metadata |
|---|---|
| host1 | v: 1.0, stage: prod |
| host2 | v: 1.0, stage: prod |
| host3 | v: 1.1, stage: canary |
| host4 | v: 1.2-pre, stage: dev |
The cluster may enable subset load balancing like this:
---
name: cluster-name
type: EDS
eds_cluster_config:
eds_config:
path: '.../eds.conf'
connect_timeout:
seconds: 10
lb_policy: LEAST_REQUEST
lb_subset_config:
fallback_policy: DEFAULT_SUBSET
default_subset:
stage: prod
subset_selectors:
- keys:
- v
- stage
- keys:
- stage
The following table describes some routes and the result of their application to the cluster. Typically the match criteria would be used with routes matching specific aspects of the request, such as the path or header information.
| Match Criteria | Balances Over | Reason |
|---|---|---|
| stage: canary | host3 | Subset of hosts selected |
| v: 1.2-pre, stage: dev | host4 | Subset of hosts selected |
| v: 1.0 | host1, host2 | Fallback: No subset selector for “v” alone |
| other: x | host1, host2 | Fallback: No subset selector for “other” |
| (none) | host1, host2 | Fallback: No subset requested |
Metadata match criteria may also be specified on a route’s weighted clusters. Metadata match criteria from the selected weighted cluster are merged with and override the criteria from the route:
| Route Match Criteria | Weighted Cluster Match Criteria | Final Match Criteria |
|---|---|---|
| stage: canary | stage: prod | stage: prod |
| v: 1.0 | stage: prod | v: 1.0, stage: prod |
| v: 1.0, stage: prod | stage: canary | v: 1.0, stage: canary |
| v: 1.0, stage: prod | v: 1.1, stage: canary | v: 1.1, stage: canary |
| (none) | v: 1.0 | v: 1.0 |
| v: 1.0 | (none) | v: 1.0 |
Example Host With Metadata¶
An EDS LbEndpoint
with host metadata:
---
endpoint:
address:
socket_address:
protocol: TCP
address: <IP_ADDRESS>
port_value: 8888
metadata:
filter_metadata:
envoy.lb:
version: '1.0'
stage: 'prod'
Example Route With Metadata Criteria¶
An RDS Route
with metadata match criteria:
---
match:
prefix: /
route:
cluster: cluster-name
metadata_match:
filter_metadata:
envoy.lb:
version: '1.0'
stage: 'prod'


| Name: mr.chandrashekaran | Reg No: 03-7SMHC19 |
|---|---|
| Age / Sex: 42/m | Contact No:9986714126 |
| Marital Status:ml- 19 years | Date:06/7/19 |
| Occupation:manager automobile industry. | Dr. pjfp |
Address: native of near chittor pallipat, In blore since 1999 Diploma in mechanical eng. |
DIAGNOSIS- |
Rt. Knee pain < climbing. From 2 months.
Left back pain lower . < travelling
Headache since 10 years. Before mild. < strain,
Gastric issues. More flatus. < outside food.
Likes warmth.
Parents divorced. Had issues with them.
Sweats less must work a lot.
120/80 mmhg….
Rx:
Lycopodium 200 – 4-0-4
Robinia 200 – 4-4-4 b/food
Nux vom 200/ ars 200 night b/food
Thuja200 – 4 aftn b.food.
Rhus tox /bryonia1m – 4-4-4-4 disc.


Every company needs to have vinyl banners for advertising and events. These can come in a different variety including thick solid vinyl or mesh vinyl for construction sites, or windy areas.
All vinyl banners are full color, high definition graphics. We make it easy and charge only by the square foot and not the number of colors you need.
We do run bulk discounts and also specials for returning customers.
For a more accurate quote, please email us for more details


--- layout: post title: Books date: '2005-11-12T00:00:00-05:00' tags: - '2005' tumblr_url: http://poetry.liabogoev.com/post/49084956392/books ---
A portal to a magical world,
In the pages of a book.
The gift of letters and words awaits,
If you would just look…
Knights, queens, and dragons,
Fairies, and pixies, and leprechauns,
Things not even imaginable,
Waiting for an open door, a chance to escape,
So many things lie undiscovered,
So many places we could go,
Catch them before they’re gone,
Think of everything you could know…


Overview Installation VTL reference Configuration Architecture Java reference Velosurf FAQ |
InstallationYou should be familiar with Velocity and the concept of Webapp to easily understand the following. Building the Velosurf archiveThe last release is included in the /bin directory, but if you want to re-build the library, you need to have ant installed.
Configuring the toolboxThe following instructions detail how to use Velosurf as a Velocity tool using the VelocityViewServlet and the toolbox.xml mechanism. First, you need to set up the Velocity toolbox in your Webapp. This involves:
Then, you can set up the Velosurf tools in /WEB-INF/toolbox.xml, like: <?xml version="1.0"?> <toolbox> <!-- http query parameters tool: You can either use org.apache.velocity.tools.view.tools.ParameterParser or velosurf.tool.HttpQueryTool. If you are using the autofetch feature, the Velosurf version is mandatory. --> <tool> <key>query</key> <scope>request</scope> <class>org.apache.velocity.tools.view.tools.ParameterParser</class> </tool> <!-- database --> <tool> <key>db</key> <scope>request</scope> <class>velosurf.tools.VelosurfTool</class> </tool> </toolbox> Configuring VelosurfVelosurf will search for a default configuration file named '/WEB-INF/velosurf.xml'. Its goal is to gather all SQL code. If you wish to change its name, you can set the appropriate context-param in your Webapp, with 'velosurf.config' as key and the target file as value. By default, the database will be reverse enginered and each table becomes an entity, each column an attribute. But this file lets you fully customize entities and attributes. You can start with the minimal configuration, that only specify the database connection parameters: <?xml version="1.0"?> <database user='login' password='password' url='database_url'> </database> You can then check your installation with a very simple template that checks that you can see your tables. |


Saturday and Sunday, 10am - 4pm
Take in the bustle of the grand space as you pile in pastries from the adjoining bakery, fluffy pancakes drenched in maple syrup, or eggs whatever way. Choose between fresh juices or cocktails to wash it all down.
Tuesday - Sunday, 11am - 3pm
For £25 its brunch becomes bottomless and includes unlimited prosecco and mimosas and a selection of cocktails.
Friday - Sunday, 12pm - 5pm
A class act, Chiltern Firehouse (discreetly) enables weekend extravagance.
Saturday and Sunday, 12pm - 3pm
Best brunch for... Brunch classics with a contemporary twist (the bellied pork in brioche is a must try).
Wednesday - Sunday, 11am - 5pm
The Black Penny doesn’t do passable - it’s straight up excellent. There are scuffed tables and a library of books that give it a properly homely feel, and everything on their extensive brunch menu is very good.


Gateway
BOT
13-May-19 05:56 PM
[telegram] <zahaliya> cannt stop falling
[telegram] <Sergio> This is normal, no binance working BTC mooning and bots pushing down the price (re @zahaliya: cannt stop falling)
[telegram] <TheoHx> it is pretty dismal, i assume bots are selling at a loss
[telegram] <TheoHx> bots and people will also oversell to create a sell chain reaction and buy back up
[telegram] <kapscedric> there is a bug in new agama wallet i can login in notary election with public key i dont now if i can vote but i can login with public key of other i dont try to vote with other vote token
[telegram] <kapscedric> i do that by error i past the public key by error and i was login in notary election
[telegram] <TheoHx> I just want the price to go back to where I bought it and just sell, KMD price is too risky
[telegram] <TheoHx> In all honesty, worst performing coin I’ve ever bought. But I still love the project. Tough situation
[telegram] <Regnar> Did you have VOTE coins? (re @kapscedric: i do that by error i past the public key by error and i was login in notary election)
[telegram] <Regnar> Or access to the vote coins associated with your public key?
[telegram] <kapscedric> yes i vote with my wif key but i have acces to other vote token with public key of other
[telegram] <Regnar> Hmm okay i see what you mean. I don't have any VOTE to see if it works, but I was able to access the vote area and shown the VOTE dust I have left over (re @kapscedric: yes i vote with my wif key but i have acces to other vote token with public key of other)
[telegram] <Regnar> Just asked in Discord "users can login using R address, that is watch-only mode. they can't send funds using that. they need to login using seed/WIF to send funds out" (re @kapscedric: yes i vote with my wif key but i have acces to other vote token with public key of other)
[telegram] <kapscedric> ha ok so no problem we are safe thanks you
[telegram] <Regnar> Yup! Np thanks for bringing it up!


Options for you step.
Your current steps, you can order it.
Preview for your current steps.
impress js editor. Thanks for use it!


Police abandon posts outside Bolivia’s presidential palace
PHNOM PENH: Cambodia has freed opposition leader Kem Sokha from house arrest more than two years after he was arrested and charged with treason, but he remains banned from politics and from leaving the country, a court said on Sunday.
The easing of restrictions on Kem Sokha comes a day after Sam Rainsy, a co-founder of their now banned Cambodian National Rescue Party (CNRP), flew into Malaysia from self-imposed exile in Paris, aiming to return to Cambodia.
The Phnom Penh Municipal Court said in a statement Kem Sokha could leave his house, but that he could not engage in political activity or leave the country.
“Under the new requirements, he can travel anywhere but can’t leave Cambodia ... This is because he has been cooperative with the authorities,” Justice Ministry spokesman Chin Malin told Reuters.


pmem::obj::experimental::slice - provides interface to access sequence of objects.
More...
#include <libpmemobj++/experimental/slice.hpp>
|
| | slice (Iterator begin, Iterator end) |
| | Constructor taking two RandomAccess iterators which define a range. More...
|
| |
|
| slice (const slice &other) noexcept=default |
| | Defaulted copy constructor.
|
| |
|
slice & | operator= (const slice &other) noexcept=default |
| | Defaulted assignment operator.
|
| |
|
iterator | begin () const noexcept |
| | Returns iterator to the beginning of the range.
|
| |
|
iterator | end () const noexcept |
| | Returns iterator to the end of the range.
|
| |
|
reverse_iterator | rend () const noexcept |
| | Returns reverse iterator to the end.
|
| |
|
reverse_iterator | rbegin () const noexcept |
| | Returns reverse iterator to the beginning.
|
| |
| reference | at (size_type idx) |
| | Element access operator. More...
|
| |
| reference | operator[] (size_type idx) |
| | Element access operator. More...
|
| |
template<typename Iterator>
class pmem::obj::experimental::slice< Iterator >
pmem::obj::experimental::slice - provides interface to access sequence of objects.
◆ slice()
template<typename Iterator>
Constructor taking two RandomAccess iterators which define a range.
- Exceptions
-
| std::out_of_range | if it_end < it_begin. |
◆ at()
template<typename Iterator>
Element access operator.
- Exceptions
-
| std::out_of_range | if idx is greater or equal to size. |
◆ operator[]()
template<typename Iterator>
Element access operator.
No bounds checking is performed.
The documentation for this class was generated from the following file:


LaundryTime, a website of more than 1 million students can use the laundry service of their residence.
Daniel Carballal, Stefan Dierauf, Carla Diví and Germán García are exchange students of Kungliga Tekniska Högskolan (KTH) and this website is a project from one course for Master of Computer Science: Interaction Programming and Dynamic Web. This project is an opportunity to practice the basics of well-architected GUI programming by practicing the Model-View-Controller paradigm on a larger task that includes real data and persistence. Also for practice interactive software development in a group and reflect on the experience.
|
Daniel Carballal Daniel Carballal is a student from University of Illinois at Urbana-Champaign(USA). He was born in Madrid(Spain) 6 of October of 1995, but he grew up in Urbana(Illinois, USA). |
Stefan Dierauf Stefan Dierauf is a student from University of Washington(USA). He was born in Portland(Oregon, USA) 28 of October of 1993. |
Carla Diví Carla Diví is a student from FIB-UPC(Facultat d'informàtica de Barcelona - Universitat Politècnica de Catalunya) from Barcelona(Spain). She was born in Barcelona(Spain) 25 of July of 1994. |
Germán García Germán García is a student from UPV(Universitat Politècnica de València) from Valencia(Spain). He was born in València(Spain). |


WITNESS LIST Appropriations-S/C on Regulatory Committee February 9, 2005-8:00A Budget Against: Biedrzycki, Carol (Texas Ratepayers Organization to Save Energy) On: Bailey Taylor, Shelia (State Office of Administrative Hearings) Bordelon, Rod (Office of Public Insurance Counsel) Chapman, Randy (Texas Legal Services Center) Farrow, Dale Beebe (Texas Board of Professional Engineers) Hendricks, Cathy L. (Self and Texas Board of Architectual Examiners) Lanford, Lane (Public Utility Commission of Texas) McClellan, Suzi (Office of Public Utility Cousel) Montemayor, Jose (Texas Department of Insurance) Shipe, Bob (Texas Workers' Compensation Committee) Thomas, Stephen (Texas Residential Construction Commission) Treacy, William (Texas State Board of Public Accountancy)


Similitude Project Charter. To create an interactive, graphical editor and simulator for digital circuits suitable for use in courses such as MUN's 3861.
Functional requirements: As a user I will be able to:
- Add components to a circuit diagram.
- Components are things like inputs, outputs, and gates, adders.
- Some components have a fixed set of input or ports.
- Some components have a variable number of input (or output?) ports.
- Move components around a circuit diagram.
- Rotate components (4 directions).
- Delete components from a circuit diagram.
- Add probes for debugging.
- Add links
- Delete links
- Move an endpoint of a link
- Automatic layout of links or components and links
- Create, edit, and use, save, compound (user-defined) components.
- Using compound components could be
- black box (no display of internals, no stopping for internal events)
- white box (display internals, stop during simulation)
- Server stuff
- Save circuit diagrams to the server.
- Login
- Create a new login name and password.
- Recover my password.
- Access library
- Upload/download circuits (and compound components) and test inputs.
- Open circuit diagrams from disk files.
- Edit the input values of input components
- Value includes bits (at least 3 values), sequences of bits, an error value
- Input signals are functions from time to values.
- Also expected outputs.
- Simulate the circuit for a period of time with animation.
- Step by step.
- Continuous (real-time slowed down)
- Instant
- View the values of inputs and outputs graphically (wave form, binary, octal, hex, decimal).
- Question: Run time or static type checking.
Nonfunctional requirements
- Servers could be written in Java 1.8 or Haxe 3 and run on the the JavaSE 1.8 platform.
- Should run on tablets as well as desk/laptop computers.
- Client could be written in Haxe 3 or TypeScript.


Incubator
Resources
Please Enter Your Information
Remember Me
Login
I forgot my password
I want to register
Retrieve Password
Enter your email and to receive instructions
Send Me!
Back to login
New User Registration
Enter your details to begin:
I accept the
User Agreement
Reset
Register
Back to login


Junior Full-stack Developer with a focus on front-end development. Currently going through Software development Bootcamp with UNC, and it is no joke.I've learned alot about programming and how to create amazing things in many different ways, but I think my best take away is I found a passion in being able to design and create challenging applications.
Improve and/or re-establish strong customer relations by troubleshooting iOS devices and providing a complete solution for customers based on their mobile needs. Mostly troubleshooting mobile devices and making sure the customer leaves with their mobile issues resolved and with a better understanding of their product.
Troubleshoot Internet and TV based servers remotely to insure customers with service are updated and records are maintained of property status. Mostly working with commercial properties (hotels,hospitals ...etc) and making sure to work on clients issue in a timely method due to high priority SLA standards. Keep clean notes and track record of communication and property status in Salesforce database.
Support sales team with administrative needs and responding to customer inquiries in a timely matter. Basically keeping good contact with sales team and customer, adhere to customers needs and recommending the most suitable product for them. Also selling a car from time to time.
Lead a team of mobile technicians to meet sales and repair quotas by implementing strategies and strong communication on mobile hardware repairs. Repairing and reporting customers hardware issues to head of Sprint ACS and keeping a team of mobile technicians on the same page as far as repair methods and repair quotas.
Apart from being a web developer, Socializing is something i enjoy doing anytime i can sit down with friends and have a good discussion is a win for me, I tend to enjoy inspiring others by uncovering their passions and motivating them to better themselves.
The beautiful artwork in this portfolio is from the artist "AndreasLie". I am a big fan of art and enjoy the pieces made by him, it reminds me that there should always be a deep appreciation for animals and their habitats.


The service is temporarily unavailable, please try again later.
Click here to return to the homepage.


Machine Learning Pipelines

<button class="btn btn-link bookmark-button "
aria-pressed="false"
data-bookmark-id="NoahB,block-v1:IRONHACK+DAFT+201906_MIA+type@vertical+block@e503a035cc0d4dabadea829648d27949"
data-bookmarks-api-url="/api/bookmarks/v1/bookmarks/">
Bookmark this page
</button>

Lesson Goals

In this lesson you will learn about:

Machine Learning pipelines.

Why you need them.

How to implement them in Scikit-learn.

Introduction

In fact you have seen machine learning pipelines in the previous lessons and labs though we haven't formally introuced the concept. What is a pipeline? In general, a pipeline is a complex process formed by subprocesses, so that the output of one subprocess is consumed as input by the following subprocess. Below is a simple example of using two Unix commands as a pipeline. The output of the first command is passed to the second command as its input by using the pipeline (|) connector:

$ cat /etc/mtab | grep sda5

The first command produces the content of the file /etc/mtab, which you do not see in the console because the content is consumed as input by the second command
grep. The eventual output you see is what
grep produces, a line containing the string
sda5:

/dev/sda5 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0

Although the execution of this example may produce a different output in different computers, because the /etc/mtab file usually differs from one computer to another, the key point here is that the output of the first command (or subprocess) is automatically fed as input to the second command (or subprocess) in a transparent way. We only see the output produced by the last command (or subprocess) of the pipeline.

Containing only two subprocesses, this small example provides an idea of how a minimalist pipeline works. A real Machine Learning pipeline may contain as many subprocesses as needed, as long as you relay the output of each subprocess to the next one.

Machine Learning Pipelines

We now transfer this concept to the domain of Machine Learning. In ML, the term pipeline is somewhat misleading because it sounds like ML is a one-way flow of data. On the contrary, however, ML pipelines are cyclic and iterative. Typically, one iteration of the ML pipeline consists the following subprocesses:

Feature selection - select which feature set to be included in the ML model

Training set formation - split the dataset into a training set and a testing set

Training - use the training set to training the ML model

Estimate test set - use the trained ML model to estimate the test set

Evaluation - compare the estimated test set with the real test set

Several iterations of ML pipelines are usually needed in order to improve/optimize you ML model. In each iteration, you can experiment different feature sets, different ML algorithms, or both. You save the state of your pipeline in each iteration. You keep performing as many iterations as you want but you should stop at a point when your further time investment is not rewarded by signficant performance gains of your ML model.

All sounds good. But isn't that the same as conduting ML analysis with the same dataset but different algorithms???

No they are not the same. You may have encountered the experience that when you perform the same ML analysis repeatedly, the outputs are different every time even if you are using exactly the same feature set and ML algorithm. This happens because when you split the train-test dataset each time, the randomization algorithm makes things different and your model performance scores become different. This makes it impossible to compare your model performance from iteration to iteration. Therefore, it's a good idea to take advantage of ML pipelines which makes sure you won't run into such troubles.

Scikit-learn Pipelines

The module
sklearn.pipeline contains the class Pipeline that we can use to implement a Machine Learning pipeline. This class allows the implementation of a sequence of chained transformations that must end with an estimation. In other words, it creates a chain of transformers followed by an estimator.

One advantage this implementation has is you do not need to build the pipeline every time it is going to be used: the whole pipeline can be serialized and saved to a file on its first use, and we can read the file an example below to show how this is done.

The Lymphography Application

A lymphography is an X-ray image of a person's lymphatic system, including but not limited to the lymph nodes. Imagine you are provided with the lymphography data of 148 lymphographies obtained at a hospital from patients with metastases and malign lymphs, and you are asked to build a Machine Learning based system for diagnosing new patients based on the data of the 148 cases. How would you proceed? In this lesson we show you how to proceed, walking you through a process of 8 phases, that uses and exemplifies Machine Learning pipelines implemented in Scikit-learn. In your future Machine Learning projects, this 8 phase process can be reused and adapted, so that you do not have to start from zero, and you have an idea of how the whole process looks. So, let's get started!

We will build a Machine Learning based application to diagnose metastases and malign lymph. For this purpose we will use the public
Lymphography dataset available at the Machine Learning Repository at UCI (University of California at Irvine). We will break the process down into 8 phases:

Load Dataset

Slice Dataset

Create Pipeline

Train

Evaluate

Save the Pipeline

Load the Pipeline

Classify new instances

We will explain what each stage does and its implementation, providing code snippets for each phase. At the end we will put all the snippets together and we will provide the whole Python program. Let's get started!

Phase 1: Load Dataset

For reading the CSV file into a Pandas dataframe, we use this function again (which we have already used in previous lessons):

import pandas.io
def readCsvToDataFrame(path, row_with_column_names):
theDataFrame= pandas.read_csv(path, header=row_with_column_names)
return theDataFrame

And we invoke like this:

dataset = readCsvToDataFrame(DATASETPATH, None)

DATASETPATH is a constant that contains the path to the CSV file. It is usually a good idea to inspect the structure of the object just read and to preview a few lines. This is a handy function that you can use for the inspection:

def show_df_info(dataframe):
# get the data type
print(type(dataframe))
print("amount of entries is %s" % dataframe.size)
print("dimensions= %i" % dataframe.ndim)
print("shape is ", end="")
print(dataframe.shape)
print("axes: ", end="")
print(dataframe.axes)
print("data types of columns:")
print(dataframe.dtypes)
print("features: %s" % dataframe.columns)

Then, we would invoke the function and preview a few lines:

show_df_info(dataset)
print(dataset.head(5))

Phase 2: Slice the Dataset

It is a good idea to encapsulate the details of the slicing that will vary from one software application to another, inside a function: that makes the rest of the code more reusable. In our example, we use this function:

def sliceDataFrame(df):
# remove 2 instances with class label "normal find"
df= df.drop(df[df[0]==1].index)
# remove 4 instances with class label "fibrosis"
df = df.drop(df[df[0] == 4].index)
# iloc arguments: range of rows, range of columns
# class 'pandas.core.series.Series'
return df.iloc[:, 1:], df.iloc[:, 0]

Note that in the slicing we have removed the instances of classes "normal find" and "fibrosis," because there are not enough of them to perform 5-fold cross validation. Then, using
iloc, attributes have been selected, starting at column with index 1 and finishing at the last column. Also, using
iloc, the class labels (target values) have been selected from the first column (index 0).

Now we will invoke the function and then do some inspection and previewing of the dataframe afterwards:

training_instances, class_labels = sliceDataFrame(dataset)
show_df_info(training_instances)
# preview the data
print(training_instances.head(5))
print()
print(class_labels.head(5))

Phase 3: Create the Pipeline

This is the code snippet that creates the Scikit-learn pipeline:

from sklearn import tree, pipeline, preprocessing
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
cart_model = tree.DecisionTreeClassifier()
pipe = pipeline.Pipeline(steps=[
("feature_selection", SelectKBest(chi2, k=8)),
("scale", preprocessing.StandardScaler()),
("CART", cart_model)])

After the required imports, a CART model is created (as you may remember, this is something we did in a previous lesson), and then a pipeline is created with the following steps as argument:

steps=[
("feature_selection", SelectKBest(chi2, k=8)),
("scale", preprocessing.StandardScaler()),
("CART", cart_model)]

The first argument indicates the name of the step , in this case "feature_selection". The second argument is the invocation that implements the step, in this case:
SelectKBest(chi2, k=8).
SelectKBest() will evaluate each feature using chi-squared, and will return the best 8 features out of a total of 18 available features (this is called dimensionality reduction, which you can find more information about in our previous lesson on "Feature Extraction and Engineering").

The second step of the pipeline is "scale", and it scales the dataset resulting from the feature selection step, so that the domain of all features lies in the range [0, 1]. This is accomplished with this invocation:

preprocessing.StandardScaler()

The third step in the pipeline "CART" and it applies the Machine Learning trained model to the scaled dataset output by the previous step.

Phase 4: Train

In the previous phase we created the pipeline, now we train it with a single invocation:

pipe.fit(training_instances, y=class_labels)

Note that the fit() method is applied to all the steps in the pipeline, although there is a single invocation at the pipeline level.

Phase 5: Evaluation

We will evaluate the pipeline using the same method that we used in previous lessons: we run cross-validation and use as quality metric the average of the resulting scores. This is how you do it:

# return value is array of scores
scores = cross_val_score(pipe, training_instances, class_labels, cv=5)
# use as quality metric the average CV score
meanCvAccuracy = scores.mean()
print("Mean CV accuracy= %f" % meanCvAccuracy)

This is the output produced:

Mean CV accuracy= 0.760952

The interpretation of the output is that, based on the tests performed, the model is expected to provide the correct diagnose in 76% of new cases.

Phase 6: Save the Pipeline

Our pipeline is a chain of 3 steps, but we do not have to save them independently, we can save the whole pipeline with a single invocation:

joblib.dump(pipe, PIPELINEPATH, compress = 1)

PIPELINE is a constant that contains the path to the file where the pipeline will be stored.

Phase 7: Load the Pipeline

Next time we need the pipeline, we do not have to build it again, we can reuse the pipeline we saved. It is read in the following code snippet:

pipeline_loaded = joblib.load(PIPELINEPATH)

Phase 8: Classify New Instances

We can use the pipeline we just loaded to classify new problem instances. In the following code, a new instance is created in variable vector, then it is processed by the pipeline, and the final classification result is printed:

import numpy
# create new random problem instance
vector = numpy.random.randint(0, 8, size=18)
print(vector)
result = pipeline_loaded.predict([vector,])
print("class label is %i" % result)

This is the output to the console:

[0 1 0 5 4 7 1 3 4 2 2 2 0 2 7 1 1 2]
class label is 3

The Whole Program

If we glue all the phases together, this is the resulting program:

import numpy
from sklearn import datasets, utils
from sklearn.model_selection import cross_val_score
import pandas.io
from sklearn import tree, pipeline, preprocessing
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
from sklearn.externals import joblib
#
# DECLARATION PART
#
PIPELINEPATH= "ser_pipeline.pickle"
DATASETPATH= "lymphography.data"
# if there is no header row with the column/attribute names, use the constant None for row_with_column_names
def readCsvToDataFrame(path, row_with_column_names):
theDataFrame= pandas.read_csv(path, header=row_with_column_names)
return theDataFrame
def show_df_info(dataframe):
# get the data type
print(type(dataframe))
print("amount of entries is %s" % dataframe.size)
print("dimensions= %i" % dataframe.ndim)
print("shape is ", end="")
print(dataframe.shape)
print("axes: ", end="")
print(dataframe.axes)
print("data types of columns:")
print(dataframe.dtypes)
print("features: %s" % dataframe.columns)
def sliceDataFrame(df):
# remove 2 instances with class label "normal find"
df= df.drop(df[df[0]==1].index)
# remove 4 instances with class label "fibrosis"
df = df.drop(df[df[0] == 4].index)
# iloc arguments: range of rows, range of columns
# class 'pandas.core.series.Series'
return df.iloc[:, 1:], df.iloc[:, 0]
#
# PROGRAM BODY
#
## PHASE 1: LOAD DATASET
dataset= readCsvToDataFrame(DATASETPATH, None)
show_df_info(dataset)
print(dataset.head(5))
## PHASE 2: SLICE DATASET
training_instances, class_labels= sliceDataFrame(dataset)
show_df_info(training_instances)
# preview the data
print(training_instances.head(5))
print()
print(class_labels.head(5))
## PHASE 3: CREATE PIPELINE
cart_model= tree.DecisionTreeClassifier()
pipe= pipeline.Pipeline(steps=[("feature_selection", SelectKBest(chi2, k=8)), ("scale", preprocessing.StandardScaler()), ("CART", cart_model)])
## PHASE 4: TRAIN
# fit all stages of the pipeline
pipe.fit(training_instances, y=class_labels)
## PHASE 5: EVALUATE
# return value is array of scores
scores = cross_val_score(pipe, training_instances, class_labels, cv=5)
# use as quality metric the average CV score
meanCvAccuracy= scores.mean()
print("Mean CV accuracy= %f" % meanCvAccuracy)
## PHASE 6: SAVE PIPELINE
# the whole pipeline in one single file
joblib.dump(pipe, PIPELINEPATH, compress = 1)
## PHASE 7: LOAD THE PIPELINE
# read the file and deserialize the pipeline
pipeline_loaded = joblib.load(PIPELINEPATH)
## PHASE 8: CLASSIFY NEW INSTANCES
# create new random problem instance
vector= numpy.random.randint(0, 8, size=18)
print(vector)
result= pipeline_loaded.predict([vector,])
print("class label is %i" % result)
print("--- end of execution ---")

Summary

In this lesson, you have learned what a Machine Learning pipeline is, its advantages, and the procedure to create one using Scikit-learn. You can reproduce the 8 phases shown here on your future Machine Learning projects, and adapt and reuse the code snippets. Now, you are ready to go!


Astar tetnet Dusty Network faucet
Please use https://eth.dusty.astar.network RPC to connect
to the Dusty testnet.
Request 0.02 PLD


For our final performance assessment in pre-calculus honors, we have to model a ferris wheel
We will model how the location of a person on the ferris wheel depends on time
Presentation
Amit
Amos
Kevin


sphinx.ext.inheritance_diagram
– 包含继承图¶
0.6 新版功能.
此扩展允许您包含继承图，通过 Graphviz extension
呈现。
它添加了这个指令:
-
.. inheritance-diagram::
¶ 该指令有一个或多个参数，每个参数都给出一个模块或类名。班级名称可以是不合格的;在这种情况下，它们被存在于当前描述的模块中（参见
py:module
）。For each given class, and each class in each given module, the base classes are determined. Then, from all classes and their base classes, a graph is generated which is then rendered via the graphviz extension to a directed graph.
This directive supports an option called
parts
that, if given, must be an integer, advising the directive to keep that many dot-separated parts in the displayed names (from right to left). For example,parts=1
will only display class names, without the names of the modules that contain them.在 2.0 版更改: The value of for
parts
can also be negative, indicating how many parts to drop from the left. For example, if all your class names start withlib.
, you can give:parts: -1
to remove that prefix from the displayed node names.The directive also supports a
private-bases
flag option; if given, private base classes (those whose name starts with_
) will be included.You can use
caption
option to give a caption to the diagram.在 1.1 版更改: Added
private-bases
option; previously, all bases were always included.在 1.5 版更改: Added
caption
optionIt also supports a
top-classes
option which requires one or more class names separated by comma. If specified inheritance traversal will stop at the specified class names. Given the following Python module:""" A / \ B C / \ / \ E D F """ class A: pass class B(A): pass class C(A): pass class D(B, C): pass class E(B): pass class F(C): pass
If you have specified a module in the inheritance diagram like this:
.. inheritance-diagram:: dummy.test :top-classes: dummy.test.B, dummy.test.C
any base classes which are ancestors to
top-classes
and are also defined in the same module will be rendered as stand alone nodes. In this example class A will be rendered as stand alone node in the graph. This is a known issue due to how this extension works internally.If you don’t want class A (or any other ancestors) to be visible then specify only the classes you would like to generate the diagram for like this:
.. inheritance-diagram:: dummy.test.D dummy.test.E dummy.test.F :top-classes: dummy.test.B, dummy.test.C
在 1.7 版更改: Added
top-classes
option to limit the scope of inheritance graphs.
例子¶
以下是实现该指令的内部 InheritanceDiagram
类的不同继承图。
全名:
.. inheritance-diagram:: sphinx.ext.inheritance_diagram.InheritanceDiagram
仅显示类名:
.. inheritance-diagram:: sphinx.ext.inheritance_diagram.InheritanceDiagram
:parts: 1
Stopping the diagram at sphinx.util.docutils.SphinxDirective
(the
highest superclass still part of Sphinx), and dropping the common left-most
part (sphinx
) from all names:
.. inheritance-diagram:: sphinx.ext.inheritance_diagram.InheritanceDiagram
:top-classes: sphinx.util.docutils.SphinxDirective
:parts: -1
配置¶
-
inheritance_graph_attrs
¶ A dictionary of graphviz graph attributes for inheritance diagrams.
For example:
inheritance_graph_attrs = dict(rankdir="LR", size='"6.0, 8.0"', fontsize=14, ratio='compress')
-
inheritance_node_attrs
¶ A dictionary of graphviz node attributes for inheritance diagrams.
For example:
inheritance_node_attrs = dict(shape='ellipse', fontsize=14, height=0.75, color='dodgerblue1', style='filled')
-
inheritance_edge_attrs
¶ A dictionary of graphviz edge attributes for inheritance diagrams.
-
inheritance_alias
¶ Allows mapping the full qualified name of the class to custom values (useful when exposing the underlying path of a class is not desirable, e.g. it’s a private class and should not be instantiated by the user).
For example:
inheritance_alias = {'_pytest.Magic': 'pytest.Magic'}


Horrific state of tar-blackened lungs belonging to 30-year chain smoker
WARNING: GRAPHIC CONTENT Medics had removed the organ from a 52-year-old brain dead man who had requested his organs be donated but they were in such an appalling condition they were rejected for a transplant
by Lorraine KingShocking footage shows the tar-blackened lungs of a 52-year-old man who had been a chain smoker for 30 years.
The viral video viewed more than 25millions times shows medics removing the organs from the brain dead man after his request to donate them.
But instead of being a healthy pink colour they are heavily blackened from decades of tobacco residue accumulation.
The clip - called the ‘best anti-smoking ad ever’ by social media users - shows Doctor Chen Jingyu and his transplant team inspecting the lungs at Wuxi People’s Hospital in Jiangsu, China.
Dr Chen, who is a leading lung transplant surgeon and also vice president of the facility, said the donor was declared brain dead and his organs were donated.
However, the medic quickly realised after harvesting that they would not be able to give the lungs to a patient in need on the waiting list, citing issues such as lung calcification, bullous lung disease and pulmonary emphysema - all thought to be the result of three decades of smoking.
Using the hashtag ‘jieyan’ - Mandarin for ‘quit smoking’ - Doctor Chen wrote: “Many smokers in this country have lungs which look like this.
“Our team decided to reject these lungs for transplant.
“If you’re a heavy smoker, your lungs may not be accepted even if you choose to donate them after death.
“Look at these lungs - do you still have the courage to smoke?”
Doctor Chen later told local media: “The patient didn’t undergo a CT scan before his death. He was declared brain dead, and his lungs were donated shortly after that.
“Initial oxygenation index tests were OK, but when we harvested the organs, we realised we wouldn’t be able to use them.
“We Chinese love smoking. It would be impractical to say that we wouldn’t accept the lungs of all smokers, but there are strict standards.
“These include lungs under 60 years of age in a patient who has only recently been declared medically dead; minor infections in the lungs and relatively clean chest X-rays are also acceptable.
“If the above conditions are met, we would consider transplanting the lungs.”
According to a 2018 study by China’s Center for Disease Control and Prevention, 26.6 percent of the country’s population above the age of 15 are smokers.
In 2017, the World Health Organization reported that worldwide tobacco use causes 7 million deaths per year.


!Iason Heine: Go and meet Priest Wood in the Temple of Einhasad, traveller! He'll be able to set up a meeting with the Witness of Dawn.


TAuthorizationRule class
TAuthorizationRule represents a single authorization rule. A rule is specified by an action (required), a list of users (optional), a list of roles (optional), a verb (optional), and a list of IP rules (optional). Action can be either 'allow' or 'deny'. Guest (anonymous, unauthenticated) users are represented by question mark '?'. All users (including guest users) are represented by asterisk '*'. Authenticated users are represented by '@'. Users/roles are case-insensitive. Different users/roles are separated by comma ','. Verb can be either 'get' or 'post'. If it is absent, it means both. IP rules are separated by comma ',' and can contain wild card in the rules (e.g. '<IP_ADDRESS>, 192.122.*.*')
Located in /Security/TAuthorizationRule.php (line 32)
TComponent | --TAuthorizationRule
Constructor.
Inherited From TComponent
TComponent::addParsedObject()
TComponent::attachEventHandler()
TComponent::canGetProperty()
TComponent::canSetProperty()
TComponent::createdOnTemplate()
TComponent::detachEventHandler()
TComponent::evaluateExpression()
TComponent::evaluateStatements()
TComponent::getEventHandlers()
TComponent::getSubProperty()
TComponent::hasEvent()
TComponent::hasEventHandler()
TComponent::hasProperty()
TComponent::raiseEvent()
TComponent::setSubProperty()
TComponent::__get()
TComponent::__set()
Documentation generated on Fri, 03 Jun 2011 16:22:53 +0200 by phpDocumentor 1.3.0RC4


It's easy to expose a REST endpoint with Spring:
@RestController public class SampleController { @GetMapping(path = "/data", produces = {MediaType.APPLICATION_JSON_VALUE, MediaType.APPLICATION_XML_VALUE}) public SampleData sampleData() { return new SampleData(new String[]{"sample1", "sample2"}); } }
Internally Spring converts the SampleData
object using a HttpMessageConverter
.
Spring brings a set of default converters and you can customize your own one. For instance, when the SampleData
looks like:
@Data @AllArgsConstructor public class SampleData { private String[] data; }
...the default XML conversion ends up with a following structure:
<SampleData> <data> <data>sample1</data> <data>sample2</data> </data> </SampleData>
But you may prefer a different structure:
<SampleData> <data>sample1</data> <data>sample2</data> </SampleData>
To achieve this with Spring Boot (you need spring-boot-starter-web
dependency and @EnableAutoConfiguration
annotation on your application configuration), you have just to register a converter bean:
@Bean public MappingJackson2XmlHttpMessageConverter mappingJackson2XmlHttpMessageConverter() { return new MappingJackson2XmlHttpMessageConverter( new Jackson2ObjectMapperBuilder().defaultUseWrapper(false).createXmlMapper(true).build() ); }
We could end here as things are so easy with Spring Boot - the converter is registered and used in the needed order automatically.
As a note it is worth to mention another option to customize message converters with Spring Boot - a WebMvcConfigurer
configuration. The interface comes actually from Spring MVC, but its usage from within a Spring Boot application has some specifics:
@Configuration class Config implements WebMvcConfigurer { @Override public void configureMessageConverters(List<HttpMessageConverter<?>> converters) { converters.add(0, new MappingJackson2XmlHttpMessageConverter( new Jackson2ObjectMapperBuilder().defaultUseWrapper(false).createXmlMapper(true).build() )); } }
Spring Boot web auto-configuration (concretely WebMvcAutoConfiguration
) brings a default configurer which means we can register as many custom configurers we want to without danger to overwrite the default configuration. That's why we can overwrite the configureMessageConverters
method without losing the default message converters, but we have to put it at the beginning of the list to take over from the already included defaults (by converting the fist applicable converter will be used).
If we have to stick with Spring MVC only, to register a converter bean is just no more enough.
One option is to use your own WebMvcConfigurer
as we already seen above. But the situation without Spring Boot is slightly different: Spring MVC default web configuration (enabled by annotating a configuration with @EnableWebMvc
) collects instances of WebMvcConfigurer
into a composite where all the converters live in a common list. This means, overwriting the configureMessageConverters
method would remove the default converters which is, usually, undesirable. Here, the method will do the job (again, you have to put the custom converter at the beginning of the list):
@Configuration class MessageConvertersConfig implements WebMvcConfigurer { @Override public void extendMessageConverters(List<HttpMessageConverter<?>> converters) { converters.add(0, new MappingJackson2XmlHttpMessageConverter( new Jackson2ObjectMapperBuilder().defaultUseWrapper(false).createXmlMapper(true).build() )); } }
Another options is to overwrite the entire configuration (assumes removing @EnableWebMvc
from your codebase). This is very similar to the previous solution:
@Configuration class WebConfig extends WebMvcConfigurationSupport { @Override protected void extendMessageConverters(List<HttpMessageConverter<?>> converters) { converters.add(0, new MappingJackson2XmlHttpMessageConverter( new Jackson2ObjectMapperBuilder().defaultUseWrapper(false).createXmlMapper(true).build() )); } }
As you can see, there is more to be aware of when working without convenience of using Spring Boot, but it's still pretty straightforward.
And that's it. I hope this helped a bit and wish you happy message converting!
You can find working examples on my Github.


Naturalist and broadcaster David Attenborough to get Indira Gandhi Peace Prize for 2019
“Sir Attenborough had tirelessly worked to awaken humankind.”
by PTIThe Indira Gandhi Prize for Peace, Disarmament and Development for 2019 will be conferred on renowned naturalist and broadcaster Sir David Attenborough.
His name was selected for the prize by an international jury chaired by former president Pranab Mukherjee, the Indira Gandhi Memorial Trust announced on Tuesday.
“Few individuals have come to be as identified with the well being of our planet, of all living creatures, and their relationship with human beings, as Sir David,” the Trust said in a statement.
The 2019 prize, it added, is awarded to Sir Attenborough for a lifetime of doing more to reveal the wonders of the natural world to us than perhaps any other individual.
Suman Dubey, secretary of the trust, said Sir Attenborough had tirelessly worked to “awaken humankind to the need, to preserve and protect the biodiversity on our planet, to live in a sustainable and harmonious way with all life, and to stop being what he has called: ‘a plague on earth’.”


Hello my dear, Before I introduce myself, I wish to inform you that this letter is not a hoax mail and I urge you to treat it serious. I am a Director in Procurement Department with Petroleum Ministry here in U. A. E. I obtained your email while searching for a reliable person, who could assist me in receiving transfer of a supposed contract awarded funds. This fund came as a result of over estimated contract awarded sums executed by foreign contractors in the Petroleum Ministry. The concerned Ministry has approved these over invoiced funds for payment. The contracts which the over invoiced sums was created from has been executed and commissioned. What I am about to receive now, is the over estimated funds which was added during the process of obtaining the contracts for my own interest. This is a normal deal that goes in my Ministry by top officials. On my part, all modalities have been worked out in ensuring a smooth conclusion of the contract payment to your bank account within the next few days. My department approves payments to contractors, and I happen to head the department. All I want from you is to receive these funds on my behalf as a Sub-Let Company, because as Government Official I cannot collect the funds directly from my Ministry, neither I am allowed by Law to operate/ run foreign bank accounts. If you are trustworthy and can assist me in receiving the fund, do not hesitate to respond back to me immediately. Please note that there is no risk involved in receiving the over estimated contract funds in your account. I want you to state in percentage what you shall have for participating in using your bank account to secure the funds. As soon as you indicate your interest, further details and the amount involved shall be given to you once I receive response from you. I hope I can trust you to treat this proposal with utmost confidentiality. Looking forward to hearing from you soonest. Best regards, Engineer Waheed Kamal.


SUBROUTINE DLATDF( IJOB, N, Z, LDZ, RHS, RDSUM, RDSCAL, IPIV, $ JPIV ) * * -- LAPACK auxiliary routine (version 3.1) -- * Univ. of Tennessee, Univ. of California Berkeley and NAG Ltd.. * November 2006 * * .. Scalar Arguments .. INTEGER IJOB, LDZ, N DOUBLE PRECISION RDSCAL, RDSUM * .. * .. Array Arguments .. INTEGER IPIV( * ), JPIV( * ) DOUBLE PRECISION RHS( * ), Z( LDZ, * ) * .. * * Purpose * ======= * * DLATDF uses the LU factorization of the n-by-n matrix Z computed by * DGETC2 and computes a contribution to the reciprocal Dif-estimate * by solving Z * x = b for x, and choosing the r.h.s. b such that * the norm of x is as large as possible. On entry RHS = b holds the * contribution from earlier solved sub-systems, and on return RHS = x. * * The factorization of Z returned by DGETC2 has the form Z = P*L*U*Q, * where P and Q are permutation matrices. L is lower triangular with * unit diagonal elements and U is upper triangular. * * Arguments * ========= * * IJOB (input) INTEGER * IJOB = 2: First compute an approximative null-vector e * of Z using DGECON, e is normalized and solve for * Zx = +-e - f with the sign giving the greater value * of 2-norm(x). About 5 times as expensive as Default. * IJOB .ne. 2: Local look ahead strategy where all entries of * the r.h.s. b is choosen as either +1 or -1 (Default). * * N (input) INTEGER * The number of columns of the matrix Z. * * Z (input) DOUBLE PRECISION array, dimension (LDZ, N) * On entry, the LU part of the factorization of the n-by-n * matrix Z computed by DGETC2: Z = P * L * U * Q * * LDZ (input) INTEGER * The leading dimension of the array Z. LDA >= max(1, N). * * RHS (input/output) DOUBLE PRECISION array, dimension N. * On entry, RHS contains contributions from other subsystems. * On exit, RHS contains the solution of the subsystem with * entries acoording to the value of IJOB (see above). * * RDSUM (input/output) DOUBLE PRECISION * On entry, the sum of squares of computed contributions to * the Dif-estimate under computation by DTGSYL, where the * scaling factor RDSCAL (see below) has been factored out. * On exit, the corresponding sum of squares updated with the * contributions from the current sub-system. * If TRANS = 'T' RDSUM is not touched. * NOTE: RDSUM only makes sense when DTGSY2 is called by STGSYL. * * RDSCAL (input/output) DOUBLE PRECISION * On entry, scaling factor used to prevent overflow in RDSUM. * On exit, RDSCAL is updated w.r.t. the current contributions * in RDSUM. * If TRANS = 'T', RDSCAL is not touched. * NOTE: RDSCAL only makes sense when DTGSY2 is called by * DTGSYL. * * IPIV (input) INTEGER array, dimension (N). * The pivot indices; for 1 <= i <= N, row i of the * matrix has been interchanged with row IPIV(i). * * JPIV (input) INTEGER array, dimension (N). * The pivot indices; for 1 <= j <= N, column j of the * matrix has been interchanged with column JPIV(j). * * Further Details * =============== * * Based on contributions by * Bo Kagstrom and Peter Poromaa, Department of Computing Science, * Umea University, S-901 87 Umea, Sweden. * * This routine is a further developed implementation of algorithm * BSOLVE in [1] using complete pivoting in the LU factorization. * * [1] Bo Kagstrom and Lars Westin, * Generalized Schur Methods with Condition Estimators for * Solving the Generalized Sylvester Equation, IEEE Transactions * on Automatic Control, Vol. 34, No. 7, July 1989, pp 745-751. * * [2] Peter Poromaa, * On Efficient and Robust Estimators for the Separation * between two Regular Matrix Pairs with Applications in * Condition Estimation. Report IMINF-95.05, Departement of * Computing Science, Umea University, S-901 87 Umea, Sweden, 1995. * * ===================================================================== * * .. Parameters .. INTEGER MAXDIM PARAMETER ( MAXDIM = 8 ) DOUBLE PRECISION ZERO, ONE PARAMETER ( ZERO = 0.0D+0, ONE = 1.0D+0 ) * .. * .. Local Scalars .. INTEGER I, INFO, J, K DOUBLE PRECISION BM, BP, PMONE, SMINU, SPLUS, TEMP * .. * .. Local Arrays .. INTEGER IWORK( MAXDIM ) DOUBLE PRECISION WORK( 4*MAXDIM ), XM( MAXDIM ), XP( MAXDIM ) * .. * .. External Subroutines .. EXTERNAL DAXPY, DCOPY, DGECON, DGESC2, DLASSQ, DLASWP, $ DSCAL * .. * .. External Functions .. DOUBLE PRECISION DASUM, DDOT EXTERNAL DASUM, DDOT * .. * .. Intrinsic Functions .. INTRINSIC ABS, SQRT * .. * .. Executable Statements .. * IF( IJOB.NE.2 ) THEN * * Apply permutations IPIV to RHS * CALL DLASWP( 1, RHS, LDZ, 1, N-1, IPIV, 1 ) * * Solve for L-part choosing RHS either to +1 or -1. * PMONE = -ONE * DO 10 J = 1, N - 1 BP = RHS( J ) + ONE BM = RHS( J ) - ONE SPLUS = ONE * * Look-ahead for L-part RHS(1:N-1) = + or -1, SPLUS and * SMIN computed more efficiently than in BSOLVE [1]. * SPLUS = SPLUS + DDOT( N-J, Z( J+1, J ), 1, Z( J+1, J ), 1 ) SMINU = DDOT( N-J, Z( J+1, J ), 1, RHS( J+1 ), 1 ) SPLUS = SPLUS*RHS( J ) IF( SPLUS.GT.SMINU ) THEN RHS( J ) = BP ELSE IF( SMINU.GT.SPLUS ) THEN RHS( J ) = BM ELSE * * In this case the updating sums are equal and we can * choose RHS(J) +1 or -1. The first time this happens * we choose -1, thereafter +1. This is a simple way to * get good estimates of matrices like Byers well-known * example (see [1]). (Not done in BSOLVE.) * RHS( J ) = RHS( J ) + PMONE PMONE = ONE END IF * * Compute the remaining r.h.s. * TEMP = -RHS( J ) CALL DAXPY( N-J, TEMP, Z( J+1, J ), 1, RHS( J+1 ), 1 ) * 10 CONTINUE * * Solve for U-part, look-ahead for RHS(N) = +-1. This is not done * in BSOLVE and will hopefully give us a better estimate because * any ill-conditioning of the original matrix is transfered to U * and not to L. U(N, N) is an approximation to sigma_min(LU). * CALL DCOPY( N-1, RHS, 1, XP, 1 ) XP( N ) = RHS( N ) + ONE RHS( N ) = RHS( N ) - ONE SPLUS = ZERO SMINU = ZERO DO 30 I = N, 1, -1 TEMP = ONE / Z( I, I ) XP( I ) = XP( I )*TEMP RHS( I ) = RHS( I )*TEMP DO 20 K = I + 1, N XP( I ) = XP( I ) - XP( K )*( Z( I, K )*TEMP ) RHS( I ) = RHS( I ) - RHS( K )*( Z( I, K )*TEMP ) 20 CONTINUE SPLUS = SPLUS + ABS( XP( I ) ) SMINU = SMINU + ABS( RHS( I ) ) 30 CONTINUE IF( SPLUS.GT.SMINU ) $ CALL DCOPY( N, XP, 1, RHS, 1 ) * * Apply the permutations JPIV to the computed solution (RHS) * CALL DLASWP( 1, RHS, LDZ, 1, N-1, JPIV, -1 ) * * Compute the sum of squares * CALL DLASSQ( N, RHS, 1, RDSCAL, RDSUM ) * ELSE * * IJOB = 2, Compute approximate nullvector XM of Z * CALL DGECON( 'I', N, Z, LDZ, ONE, TEMP, WORK, IWORK, INFO ) CALL DCOPY( N, WORK( N+1 ), 1, XM, 1 ) * * Compute RHS * CALL DLASWP( 1, XM, LDZ, 1, N-1, IPIV, -1 ) TEMP = ONE / SQRT( DDOT( N, XM, 1, XM, 1 ) ) CALL DSCAL( N, TEMP, XM, 1 ) CALL DCOPY( N, XM, 1, XP, 1 ) CALL DAXPY( N, ONE, RHS, 1, XP, 1 ) CALL DAXPY( N, -ONE, XM, 1, RHS, 1 ) CALL DGESC2( N, Z, LDZ, RHS, IPIV, JPIV, TEMP ) CALL DGESC2( N, Z, LDZ, XP, IPIV, JPIV, TEMP ) IF( DASUM( N, XP, 1 ).GT.DASUM( N, RHS, 1 ) ) $ CALL DCOPY( N, XP, 1, RHS, 1 ) * * Compute the sum of squares * CALL DLASSQ( N, RHS, 1, RDSCAL, RDSUM ) * END IF * RETURN * * End of DLATDF * END


  Quam fuga eaque sapiente perspiciatis, id consequuntur.
|
|
|
|
|
|
|
Features All Joined Together in One App
|
|
|
SYNC
|
|
Lorem ipsum dolor sit amet, consectetur adipisicing elit.
|
|
Learn More
|
|
|
SHARE
|
|
Lorem ipsum dolor sit amet, consectetur adipisicing elit.
|
|
Learn More
|
|
|
STATS
|
|
Lorem ipsum dolor sit amet, consectetur adipisicing elit.
|
|
Learn More
|
|
|
|
|
RESPONSIVE LAYOUT
|
|
|
Tanjin is one of nearly 2,000 people to have landed on Italy's southernmost.
|
|
|
RETINA READY
|
|
|
Tanjin is one of nearly 2,000 people to have landed on Italy's southernmost.
|
|
|
|
VARIOUS SKINS
|
|
|
Tanjin is one of nearly 2,000 people to have landed on Italy's southernmost.
|
|
|
GREAT PERFORMANCE
|
|
|
Tanjin is one of nearly 2,000 people to have landed on Italy's southernmost.
|
|
|
|
USER FRIENDLY MENUS
|
|
|
Tanjin is one of nearly 2,000 people to have landed on Italy's southernmost.
|
|
|
SOCIAL SHARE
|
|
|
Tanjin is one of nearly 2,000 people to have landed on Italy's southernmost.
|
|
|
|
Interested in our App
Muriel Bowser's says move is unrelated to charges that encryption caused failures in Metro.
REQUEST A QUOTE
|
|
|
|
|


{% extends '_index.html' %} {% import '_macros_res_imports.html' as m_res_import with context %} {% import '_macros_compat.html' as m_compat with context %}


--- title: 'Generating Multiple PDF at Once: A use case of Rmarkdown render in a purrr loop' author: Package Build date: '2022-01-06' slug: [] categories: [] tags: [] subtitle: '' summary: '' authors: [] lastmod: '2022-01-06T15:04:53Z' featured: no image: caption: '' focal_point: '' preview_only: no projects: [] ---
As my previous post about using docx template to create PDF with Rmarkdown files had quite a success, people might be interested by more content on how I use Rmarkdown files in my daily activities. Indeed, I have a couple of more interesting use cases to show and among them, generating multiple pdf documents at once is one of my favorite.
Here is some context: for some modules that I teach, I am assessing students on their ability to write short research papers and I require them to submit a PDF document. PDFs have a lot of advantages however it is difficult to annotate them to provide feedback to students. For this reason, I am creating a PDF report for each student containing my feedback on each section of their research paper.
Now, I could write a document for each student and save it as PDF but it appears much more efficient to write my comments in a spreadsheet and then to generate the PDF report for all the students at the same time!
The trick to create multiple PDF documents from a Rmd file is not mine, I just applied what the fantastic Alison Hill presented in her slides “Made with YAML, strings, and glue. An R Markdown valentine for you. If you are interested in doing it yourself, this presentation is amazing and far more useful than this blog post alone. However, it is always good to have different use cases and I am sure that mine can be interesting somehow.
Enough talk, time to code!
To make this example more interesting, let’s make a table with some of my favorite TV show characters and some of their quotes:
| fullname | quote_1 | quote_2 | quote_3 |
|---|---|---|---|
| Michael Scott | Sometimes I’ll start a sentence and I don’t even know where it’s going. I just hope I find it along the way. | I’m not superstitious, but I am a little stitious. | Would I rather be feared or loved? Easy. Both. I want people to be afraid of how much they love me. |
| Leslie Knope | We have to remember what’s important in life: friends, waffles, and work. Or waffles, friends, work. But work has to come third. | What I hear when I’m being yelled at is people caring really loudly at me. | There’s nothing we can’t do if we work hard, never sleep, and shirk from all other responsibilities in our lives. |
| Jake Peralta | Fine, but in protest, I’m walking over there extremely slowly! | I wasn’t hurt that badly. The doctor said all my bleeding was internal. That’s where the blood’s supposed to be. | I appealed to their sense of teamwork and camaraderie with a rousing speech that would put Shakespeare to shame. |
The table has to be saved as “data_batch.csv” to be used in the batch processing.
Now, we need a Rmd template document called “index.Rmd” which will read the data file, use each row and produce a PDF with these personalized data. Here is the YAML of this “index.Rmd” file:
---
title: "Batch PDF processing - `r params$fullname` quotes"
output: pdf_document
params:
fullname: "TV Show Character"
---
In this “index.Rmd”, the key is the use of the YAML option params:
with the column header to iterate on (e.g., fullname) and a null value that will be replaced by the name of each TV show character (e.g., “TV Show Character”). Note, the title of the PDF includes a variable reference to the characters’ name with an inline code.
It is also important to read the data file in this “index.Rmd” and to filter with params$fullname
in order to keep only the row corresponding to the character of the iteration.
Then, two solutions to input the data in the text. The first and most natural solution would be use inline r code as follow:
The second solution is the one introduced by Alison, using the package {epoxy} created by the genious Garrick Aden-Buie. This package allows to glue field of the data very easily in R chunks. This is a bit overkill in this use case as there are only 4 columns in the data but it would be very useful in a text having many different variables.
In a new R script called “_render_batch.R”, read the data file in an object here called “data_batch”.
Then, with a purrr::walk()
function, we can iterate over each TV show character (i.e., each row) to produce a PDF that includes their corresponding quotes.
data_batch <- readr::read_csv("data_batch.csv")
purrr::walk(
.x = data_batch$fullname,
~ rmarkdown::render(
input = "index.Rmd",
output_file = glue::glue("PDF output - {.x} quotes.pdf"),
params = list(fullname = {.x})
)
)
Basically, purrr::walk()
has two arguments:
.x
a list or vector containing each element to iterate over (e.g., TV show character names).f
a function used for each element of the previous list or vectorWhile .x
is naturally data_batch$fullname
containing all the characters’ names from the data file previously read, .f
is the function rmarkdown::render()
which will get the template, replace params$fullname
with the value of .x
and knit the document. Q.E.D!
Note the essential use of glue::glue()
to include the value of .x
in the name of the PDF file obtained in order not not overwrite the same document for each value of .x
!