module type S2 =sig
..end
type ('a, 'e)
m
type 'a
t
val all : ('a, 'e) m t ->
('a t, 'e) m
all cs
perfoms all computations in cs
and returns a
list of results in the same order. The order of
evaluation is unspecified.val all_ignore : ('a, 'e) m t ->
(unit, 'e) m
all_ignore cs
performs all computations in cs
in an
unspecified order, and discards the results.val sequence : (unit, 'e) m t ->
(unit, 'e) m
sequence cs
performs all computations in cs
from left
to right.val map : 'a t ->
f:('a -> ('b, 'e) m) ->
('b t, 'e) m
map xs ~f
returns a container where n
'th element is a
result of computation f x_n
, where x_n
is the n
'th
element of the input container. It is unspecified, in
which order the computations are evaluated, and whether
all computations are performed.val iter : 'a t ->
f:('a -> (unit, 'e) m) ->
(unit, 'e) m
iter xs ~f
performs f x
for each x
in xs
in the
left to right order.val fold : 'a t ->
init:'b ->
f:('b -> 'a -> ('b, 'e) m) ->
('b, 'e) m
fold xs ~init:s0 ~f
folds f
over xs
in the given monad.
Effectively computes a chain:
f s0 x0 >>= fun s1 -> f s1 x1 >>= fun s2 -> ... f sN xN
Except that the computation uses a constant stack size.
val fold_left : 'a t ->
init:'b ->
f:('b -> 'a -> ('b, 'e) m) ->
('b, 'e) m
fold_left
is a synonym for fold
.val fold_right : 'a t ->
f:('a -> 'b -> ('b, 'e) m) ->
init:'b -> ('b, 'e) m
fold_right xs ~f ~init:s0
folds f
over xs
from right to
left in the given monad.
Effectively computes a chain:
f x_N s0 >>= fun s1 -> f x_(N-1) s1 >>= fun s2 -> ... f x0 s_N
Except that the computation uses a constant stack size.
val reduce : 'a t ->
f:('a -> 'a -> ('a, 'e) m) ->
('a option, 'e) m
reduce xs ~f
same as fold
except that the initial
state is obtained from the first element of the
container, i.e., computes a sequence
f x0 x1 >>= fun s1 -> f s1 x2 >>= fun s2 -> ... f sN xN
val exists : 'a t ->
f:('a -> (bool, 'e) m) ->
(bool, 'e) m
exists xs ~f
returns a computation that results in
true
iff there exists an element x
in xs
such that
f x
evaluates to true
val for_all : 'a t ->
f:('a -> (bool, 'e) m) ->
(bool, 'e) m
for_all xs ~f
returns a computation that results in
true
iff for all x
in xs
f x
evaluates to true
.val count : 'a t ->
f:('a -> (bool, 'e) m) ->
(int, 'e) m
count xs ~f
returns a computation that results to a
number of elements of xs
for which f
evaluates to
true
. The order of application of f
is unspecified.val map_reduce : (module Monads.Std.Monoid.S with type t = 'a) ->
'b t ->
f:('b -> ('a, 'e) m) ->
('a, 'e) m
map_reduce (module Monoid) xs ~f
a composition of map
and reduce
. Effectively the same as map xs ~f
and then
reduce in Monoid
except that no intermediate
collections are created.val find : 'a t ->
f:('a -> (bool, 'e) m) ->
('a option, 'e) m
find xs ~f
returns the first element x
of xs
for
wich f x
evaluates to true
.val find_map : 'a t ->
f:('a -> ('b option, 'e) m) ->
('b option, 'e) m
find_map xs ~f
returns the first computation f x
for
x
in xs
which will result in non None
.val filter : 'a t ->
f:('a -> (bool, 'e) m) ->
('a t, 'e) m
filter xs ~f
returns a computation that contains all
the elements of xs
for which f
evaluated to true
. The
order of the elements is the same.val filter_map : 'a t ->
f:('a -> ('b option, 'e) m) ->
('b t, 'e) m
filter_map xs ~f
is a partial mapping from xs
to a
collection ys
, such that all the elements of xs
for which
f
returned Some
value are mapped, while the rest are
omitted.


MD Doc Paragraph • GitLab uses "GitLab Flavored Markdown" (GFM). It extends the standard Markdown in a few significant ways to add some useful functionality. You can use GFM in the following areas: comments, issues, merge requests, milestones, snippets and more.
MD Doc Small Paragraph • GitLab uses "GitLab Flavored Markdown" (GFM). It extends the standard Markdown in a few significant ways to add some useful functionality. You can use GFM in the following areas: comments, issues, merge requests, milestones, snippets and more.
UI Monospace
UI Small Monospace
MD Doc Paragraph • GitLab uses "GitLab Flavored Markdown" (GFM). It extends the standard Markdown in a few significant ways to add some useful functionality. You can use GFM in the following areas: comments, issues, merge requests, milestones, snippets and more.
MD Doc Paragraph • GitLab uses "GitLab Flavored Markdown" (GFM). It extends the standard Markdown in a few significant ways to add some useful functionality. You can use GFM in the following areas: comments, issues, merge requests, milestones, snippets and more.
MD Doc Paragraph • GitLab uses "GitLab Flavored Markdown" (GFM). It extends the standard Markdown in a few significant ways to add some useful functionality. You can use GFM in the following areas: comments, issues, merge requests, milestones, snippets and more.
MD Doc Paragraph • GitLab uses "GitLab Flavored Markdown" (GFM). It extends the standard Markdown in a few significant ways to add some useful functionality. You can use GFM in the following areas: comments, issues, merge requests, milestones, snippets and more.
MD Doc Paragraph • GitLab uses "GitLab Flavored Markdown" (GFM). It extends the standard Markdown in a few significant ways to add some useful functionality. You can use GFM in the following areas: comments, issues, merge requests, milestones, snippets and more.
MD Doc Blockquote • GitLab uses "GitLab Flavored Markdown" (GFM). It extends the standard Markdown in a few significant ways to add some useful functionality. You can use GFM in the following areas: comments, issues, merge requests, milestones, snippets and more.


Legalize Multiple Channel Ops Per Channel
Written on 2023-05-02.
Context
Objective
Whenever an XLS channel has multiple sends/receives, the XLS compiler currently relies on the mutual exclusion pass to merge these operations. If the operations cannot be proven to be mutually exclusive or if they are not in the same token level, it is an error and many legal programs fail to compile. The goal is to compile more programs that have multiple sends/receives on the same channel.
Background
One common way for the mutual exclusion pass to fail is with a pipelined loop:
if (p) {
send(chan, value0);
} else {
#pragma hls_pipeline_init_interval 1
for (int i = 0; i < 4; i++) {
send(chan, value1);
}
}
The XLSCC frontend1 will make the for loop a separate proc with channels that pass necessary state in and out. The "outside" proc will pause forward progress until the loop proc completes, so the two sends are mutually exclusive. However, after proc inlining, the predicates to the sends will be a function of the proc state that requires induction to prove mutual exclusion. To see why this is, it's helpful to understand how proc inlining works. The proc inlining pass creates a new container proc that allows all the inlined procs to tick simultaneously. This is implemented via an "activation network"- a series of boolean values encoded in the proc state that follow the token network of each proc. Each proc has a root activation that corresponds to the proc token parameter- this root activation is initialized as true, and activations flow down the network until something like a waiting blocking receive on an internal channel occurs. When a side-effecting operation does not occur within a container tick, all dependent operations will not be activated and will wait until the next container tick. This allows inlined procs to independently make progress on their execution. It also means that inlined proc ticks are decoupled from container proc ticks.
Channel operations in different procs will have activations be a portion of their predicates, but proving properties about these activations in general requires reasoning across multiple proc ticks. The mutual exclusion pass is not able to prove properties over time and will fail to merge the sends in the example above.
Currently, XLS treats multiple channel operations on the same channel as an error as a design choice to enforce that full throughput at II=1 is achievable. As a result, compilation will fail. It is important to note that although this problem exists for any channel, it is especially common when working with RAMs, especially single-port RAMs. The example above could come from code like
if (p) {
mem[x] = value0;
} else {
#pragma hls_pipeline_init_interval 1
for (int i = 0; i < 4; i++) {
value1 += mem[(x + 1)%MEM_SIZE];
}
}
Each memory access gets lowered to a pair of send/receives on a request and response channel. For single-port RAMs, read and write channels will also be merged. It is therefore critical for programs using RAMs that multiple sends/receives work.
Design
Overview
Ultimately, we are asking the mutual exclusion pass to do too much. To merge operations like the sends in the example above, analysis must prove arbitrarily complex properties of the program, including those that change over time. Instead of relying on mutual exclusion analysis, it seems better to allow multiple sends and treat the mutually exclusive operations as a special case that allows for optimization. Concretely, the proposal is:
- Do not merge mutually exclusive channel operations before scheduling.
-
When multiple channel operations occur on a channel, IR lowering should replace the channel for those operations with new internal channels. An adapter proc should be added that accepts multiple channel operations on the new "generated by the compiler pass" side and forwards them to the original IO channel.
1. One adapter variant will be an arbiter. If multiple channel operations occur in a cycle, the pipeline stalls until they all complete. Priority for the adapter will be determined by token level. If operations that are not ordered with respect to each other activate at the same time, it will be an assertion failure. Optionally, the compiler can impose an arbitrary order to unordered operations (following the token ordering where one exists) via a topo sort.
1. Another adapter variant will be a "mutex assumed" variant used only when the channel operations are mutually exclusive. The XLS→outside signals will be OR'd and the outside→XLS signals will be fanned out. An assertion will be added checking that at most one channel operation is active in a cycle2.
-
These adapters should be separate procs and codegen'd as their own Verilog module. This hierarchy is useful for tracking the cost of the arbiter variant and facilitating later replacement of arbiter variants with mutex assumed variants. The MVP can initially inline the adapters until multi-proc codegen is supported.
Minimum Viable Product
The goal is to support the example in the Background section (and similar examples) ASAP. The steps roughly follow:
- Write a proc generator for the arbiter variant of adapter.
-
When there are multiple channel operations on the same channel, a scheduling pass3 will add new channels and an adapter. The pass will move the channel operations' channel ids over to the new channels and connect the adapter's send/recv to the original channel. New channels to send the predicate before the operation and receive a completion after the operation will be added and connected.
1. The adapter will arbitrate between multiple operations in priority order.
2. Priority order will be determined by token dependencies. The adapter will have runtime checks that all active receives are ordered with respect to each other. Optionally, impose an arbitrary order to unordered operations via a topo sort.
3. Channel operations from different procs will be required to be mutually exclusive, enforced by a runtime check.
4. Initially, use proc inlining on the adapter.
-
After the adapter is added and connected by the pass, scheduling constraints will only apply to the operation in the adapter. The original send/receives will be freely scheduled, although after inlining the predicate sends (before the operation) and completion receives (after the operation) will implicitly order the operations with respect to the scheduling constraint.
Details
This proposal suggests extending the current XLS behavior which are, roughly, "multiple channel operations must be formally proven to be mutually exclusive and on the same token level, and then must be merged into a single operation". There are trade-offs in choosing how far to extend.
Single Channel Operation per Tick with True Predicate
One case is to allow multiple channel operations per proc tick, but require that only one of them have a true predicate. The current XLS behavior requires the operations to be on the same token level and to be formally proven to have mutually exclusive predicates. This could be extended to not require formal proof and instead provide a runtime check (assertion). This corresponds to the mutex assumed adapter variant described earlier and is a very incremental extension of the existing behavior. One problem with assuming all channel operations are mutually exclusive (even if you can't prove it) is that they aren't mutually exclusive. We could make it an error in the interpreter and JIT to have multiple channel operations per tick, but it currently isn't an error and seems potentially useful to allow eventually. There it seems like the mutually exclusive case should be an optimization of the more general case where multiple operations fire, and only applied when desired.
Multiple Channel Operations per Tick with True Predicate
When there are multiple channel operations per tick firing, there is a new problem to consider: how to order the operations. There is no ambiguity when there is a total ordering on operations. It would be safe for the MVP to require a total ordering. However, the XLSCC frontend specifies token dependencies as loosely as possible4, so it is a normal occurrence for there not to be a total ordering on channel operations.
If multiple channel operations aren't totally ordered, you can choose a static
or dynamic order. On a subset of operations with no ordering with respect to
each other, you could imagine choosing randomly or going round robin. However,
choosing a static order seems simplest for MVP, and dynamic options seem of
little utility. One way to choose this static order is to use the order given by
XLS's TopoSort()
5.
An alternative to worrying about the order is to legalize channel operations by adding a new channel per extra operation. This is equivalent to the current approach without the adapter- instead, the adapter (if needed) becomes the responsibility of the external integrator. This option might be useful in cases where the adapter orders things based on the data (for memory traffic, you could imagine performing all writes before all reads, or vice versa, and forwarding data written in the previous cycle if read in the current cycle).
Channel Operations in Different Procs
Channel operations in different procs have no direct token relationship. There
is an ordering imposed by channel operations over control channels, but this
relationship is indirect. TopoSort()
only operates on one proc or function at
a time.
We could require that multiple sends/receives occur within the same proc, although XLSCC produces channel operations in different procs for the example given at the start and many other interesting examples.
We could build a global token graph, where each proc's token parameter has an edge to some root token and each proc's next token feeds into a global next token (analogous to proc inlining). Edges will be added where procs send/receive to each other on internal channels. Note that the analysis may be tricky in cases where multiple parallel send/receives occur on these internal channels. It may be desirable to disallow multiple channel operations on internal channels, or have a variant of channel that disallows multiple channel operations to keep analysis sane.
Scheduling
For MVP, it seems simplest to require all channel operations on a given channel be scheduled in the same cycle. This will sometimes cause scheduling to fail and may interact poorly with II>1 passes, but is a useful starting point. Putting channel ops on the same channel in different cycles could allow for better QoR (especially with II>1) than could be realized if channel ops were required to be merged.
If channel operations on the same channel are scheduled in different cycles, the adapters need to delay earlier channel operations until the last cycle containing one of the multiple channel operations67.
Long-Term Enhancements
Multi-Proc Codegen
The adapter procs should ideally be codegen'd as separate Verilog modules, which will require multi-proc codegen.
Optimization for Mutually Exclusive Operations
Post-codegen analysis may be needed to determine if operations are mutually exclusive. For example, mutual exclusion properties may be data dependent and depend on the context a block is instantiated in. Third party formal tools may be useful in proving these properties.
Multi-proc codegen makes it straightforward to switch an adapter from arbitrating to mutex assumed- generate both variants and have a Verilog config set which variant should be used for each instance after the fact.
Arbitrary Scheduling of Multiple Channel Operations
Wavefront matching channel operations across cycles seems to mean that scheduling channel operations in different cycles will not save anything. This is because the extra state you need to delay a channel operation. If analysis can determine that the wavefronts don't need to be matched (in general, this could be data dependent!), then you could remove this extra state.
If wavefronts do need to be matched, care needs to be taken to drive channel operations in the correct order in the presence of stalls.
IR-level testing should run on scheduled IR
The pass described here is a scheduling pass, and hence performed during codegen. The priority chosen by the pass when there is not a total order on the channel operations will impact program behavior, so ideally it would be consistent when running IR JIT.
One way to resolve this is to combine IR opt + codegen into a single driver. That can produce opt IR, scheduled IR, or RTL. Alternatively, you could make scheduling its own distinct step, or move scheduling passes into opt and only do block conversion and RTL conversion in codegen. Regardless, the IR jit should be running the scheduled IR to ensure equivalent program behavior.
Alternatives considered
Improve Proc Inlining
In practice, many of the failures we've seen in identifying mutually exclusive channel operations are a result of proc inlining. Proc inlining moves channel operation predicates into the proc state, but this doesn't seem necessary for many of the examples that fail to compile.
The XLSCC frontend generally produces procs that transfer execution to one proc at a time. They generally look something like:
arguments = receive(input_channel, pred=is_init_state);
state = WORK_STATE;
// do stuff
send(output_channel, pred=is_done_state);
if (is_done_state) {
state = INIT_STATE;
}
In this case, the proc activation state can be condensed into something like a
program counter. Each value of the program counter would correspond to a
different inlined proc's activation and channel operations would have their
predicates be strengthened with p' = p & (pc == MY_PROC_ID)
. As a result,
channel operations in different inlined procs would be trivially mutually
exclusive.
The analysis to determine if this optimization can be performed safely could be as simple as trying it and performing a deadlock analysis. However, even if the optimization is safe, it may not be an optimization- multiple procs ticking in parallel may be desired for performance reasons.
Analysis to check that execution is transferred could look like:
-
Find places where procs give control away. These are found by unrolling by 2 and looking for all blocking (send, recv) pairs where the receive is downstream of the send in the token graph (with nothing in-between). If the send's predicate implies the receive's predicate, the proc gives control to another proc on the send's channel until it is returned via the receive's channel. If this property holds for every proc→proc channel within a proc, we can say the proc "always extinguishes".
-
Find procs that come out of initialization "extinguished". There must be at least one proc (generally, the top proc) where this is not true. Other procs that begin with a blocking receive with predicate true on initialization are initialized as "extinguished". Every other side-effecting operation should be downstream of this receive.
The subset of procs that are initialized as extinguished and always extinguish can be inlined as described above.
It's unclear that this approach will allow mutual exclusion to succeed for more complicated scenarios, e.g. multiple sends/recvs in nested loops in different orders. It also seems generally unscalable to rely on passes being aware of mutual exclusion.
It is worth noting that this may be a generically useful improvement to proc inlining. Having predicates that are easier to reason about could assist optimization, and the state representation of proc activation would be more compact.
It is also worth noting that non-blocking channel operations will not be defined as extinguishing. The notion of extinguishing could be generalized to include procs whose state isn't updated observably unless a receive completes. Similarly, in our discussions Remy described some potentially simpler ways of identifying procs that can be run one at a time with a PC. Regardless of what analysis is used to identify situations where proc inlining can execute procs mutually exclusively, the shortcomings of the approach seem to be that 1) proc inlining and other passes need to be aware of how powerful the mutual exclusion pass is, which doesn't scale well; and 2) it’s not clear that this resolves all the cases where mutual exclusion can fail, even in proc inlining (proc inlining adds state to track multiple proc↔proc channel operations, so mutual exclusion could fail within an inlined proc if this obfuscates the predicate).
Use a proof assistant that supports induction
A more powerful proof assistant has the danger of increasing runtime or timing out unpredictably (Z3 is already pushing our boundaries in this regard). Even though the unrolling required to prove mutual exclusion seems bounded in most cases, it isn’t clear what sort of runtime we should expect.
It seems better to treat powerful proof assistants as an infrequent, high-effort final optimization step. For example, in the proposed design the arbiter variants could be replaced with mutex assumed variants if an expensive proof of mutual exclusion can be found.
-
The example presented here is particular to the XLSCC frontend. DSLX scopes channels to procs, so a similar example in DSLX would not have multiple procs sending/receiving the same channel. Instead, an internal channel would communicate proc→proc and the top proc would forward it to the IO channel. If the top proc manages the send predicate, it is likely that mutual exclusion will be able to merge the channel operations. However, if the child proc manages the predicate, DSLX could have the same problems with proc inlining that XLSCC does.
Of course, non-mutually-exclusive multiple sends/receives are not allowed in either frontend currently and this proposal would enable that for both frontends. ↩
-
In the general case where channel operations are scheduled in different cycles, this will need to be a temporal assertion. ↩
-
A "scheduling pass" is a pass that operates on the IR after scheduling but before block conversion. Importantly, we want this pass to run after the mutual exclusion pass has a chance to run and after scheduling for II>1 has occurred. ↩
-
As of writing, XLSCC does not specify any token relationships, it emits IR with every side effecting operation taking the proc’s root token and gathers all resulting tokens into an after-all for the next token. The TokenDependencyPass uses data dependencies to add minimal token relationships. ↩
-
It is often desirable for DSLX interpretation, IR interpretation/JIT, and RTL simulation to all have identical output. One reason is to support fast verification at higher levels of abstraction. Then a LEC chain from IR to RTL can prove that the high-level coverage applies to the RTL.
TopoSort()
(and many other ways of choosing an order) will be unstable across the different levels of abstraction, which is a potential problem. The DSLX interpreter should have a "safe" mode or throw a warning when multiple channel operations happen in a single tick. IR interpretation/JIT should be performed on IR with the token graph reflecting the order chosen by the channel legalization pass. ↩ -
If you have send(c0) -> recv(c1) -> send(c10) each scheduled in different cycles and recv(c1) blocks until the first send(c0) goes through, you’ll get deadlock. ↩
-
Like mutual exclusion analysis, a proof that a send in (tick, cycle) (t[n], c[n]) implies the predicate in (t[n-i], c[n+j]) is false could allow for early sends and simpler logic. ↩


Improved fork of fs-extra
with extra [sic] features (and semicolons!)fsxt
provides support for node.js 10 and above.
npm install --save fsxt
Or install with your preferred package manager (yarn, pnpm, ...)
fsxt
is a drop-in replacement for the node.js core fs
module. All methods in fs
can be used in their standard forms in fsxt, with some
improvements.
You don't ever need to include the original fs
module again:
const fs = require('fs'); // this is no longer necessary
you can now do this:
const fs = require('fsxt');
or if you prefer to make it clear that you're using fsxt
and not fs
, you may want to name your
fs
variable fsxt
like so:
const fsxt = require('fsxt');
you can also keep both, but it's redundant:
const fs = require('fs');
const fsxt = require('fsxt');
All the improvements from mz/fs
are included,
which also includes improvements from graceful-fs
.
Most methods are async by default, returning a Promise that resolves to the method's result, or rejects if the operation fails.
Sync methods on the other hand will throw if an error occurs, and directly return the resulting value to the caller if the operation succeeds.
You can also use the methods in the legacy node.js form, passing a callback as the last parameter,
as a function that takes (Error, <result>)
parameters, but it's not recommended, and those
variants may be removed in a later (major) version.
Additionally, see notes on fs.read()
& fs.write()
if you're using
either method.
Example use:
const fs = require('fsxt');
// Async with promises:
fs.copy('/tmp/myfile', '/tmp/mynewfile')
.then(() => console.log('success!'))
.catch(err => console.error(err));
// Async with callbacks:
fs.copy('/tmp/myfile', '/tmp/mynewfile', err => {
if (err) return console.error(err);
console.log('success!');
});
// Sync:
try {
fs.copySync('/tmp/myfile', '/tmp/mynewfile');
console.log('success!');
} catch (err) {
console.error(err);
}
// Async/Await:
async function copyFiles () {
try {
await fs.copy('/tmp/myfile', '/tmp/mynewfile');
console.log('success!');
} catch (err) {
console.error(err);
}
}
copyFiles();
The documentation is available at https://uwx-node-modules.github.io/fsxt/typedoc/globals.html.
You can also find the documentation for just the methods added in fsxt that aren't in fs-extra at https://uwx-node-modules.github.io/fsxt/typedoc-fsxt-only/globals.html.
The docs don't include descriptions for methods inherited from fs-extra, but you can find them here:
The core node.js fs
module methods are also
available; although the node.js documentation doesn't show Promise overloads of the async methods,
you can find them at:
If you want to watch for changes to files or directories, then you should use chokidar.
Do you want to hack on fsxt? Well, you probably shouldn't. Still, you can go ahead and send a PR.
Please, no changes to anything in the lib
folder; the contents of that folder are almost entirely
verbatim from fs-extra, so they should be submitted upstream.
fsxt uses the Google Style. It's pretty.
fsxt contains like at least 4 tests that pass.
npm run lint
: runs eslintnpm run unit
: runs the unit testsnpm test
: runs both the linter and the testsIf you run the tests on the Windows and receive a lot of symbolic link EPERM
permission errors,
it's because on Windows you need elevated privilege to create symbolic links. You can either run the
tests as Administrator or run node testlite
to test only fsxt-exclusive methods, which doesn't
include symbolic links
Licensed under MIT. Full license text available at LICENSE.txt
fs-extra
is copyright (c) 2011-2017 JP Richardson
fsxt
is copyright © 2016-2018 uwx, some rights reserved.
Parts of the documentation were taken from other modules and the Node.js fs
module.
Relevant licenses are included at the following locations:
fs-extra
and fsxt
are not endorsed by or affiliated with Joyent or the Node.js Foundation.
fsxt
is not endorsed by or affiliated with JP Richardson.
Generated using TypeDoc


{% extends "regsys/adminBase.html" %} {% block content %}
| Section Number | Course Name | Year | Semester | Weekday | Start | End | Current Enrollment | Max Enrollment | Faculty | Building | Update Info |
|---|---|---|---|---|---|---|---|---|---|---|---|
| {{s.section_no}} | {{s.course_no}} | {{s.year}} | {{s.semester}} | {{s.weekday}} | {{s.start}} | {{s.end}} | {{s.current_enrollment}} | {{s.max_enrollment}} | {{s.staff_id}} | {{s.building_code}} | Update |


Deontay Wilder vs. Tyson Fury
01 Dec 2018 - Deontay Wilder vs. Tyson Fury
Deontay Wilder vs. Tyson Fury was a professional boxing fight that took place on December 1, 2018, at the Staples Center in Los Angeles, California. Undefeated defending WBC heavyweight champion Deontay Wilder faced undefeated challenger and former WBA (Super), WBO, IBF, IBO, The Ring, and lineal heavyweight champion Tyson Fury. The fight ended in a controversial split draw with the scorecard at 115–111 for Wilder, 114–112 for Fury (which was incorrectly announced as 114–110), and 113–113.


Orgazmo (1998) ***1/2 out of *****
Cast: Trey Parker, Matt Stone, Dian Bachar, Michael Dean Jacobs, Ron Jeremy, Lloyd Kaufman Written and Directed by: Trey Parker Running Time: 94 minutes
Crude, low-budget comedy from Matt Stone and Trey Parker, the creators of Comedy Central's smash hit "South Park". Filmed before "South Park" became a big hit, but released after Stone and Parker's summer 1998 effort BASEketball, Orgazmo tells the tale of Joe Young (Trey Parker), a mormon on a trip to Hollywood who gets mixed up in the adult film industry after knocking on the wrong door. Eventually, Young becomes both a superstar in the business, as well as a crime fighting superhero.
Orgazmo is certainly crude (it's rated NC-17), but nothing in the film really offended me in any way. It's definately low-brow humor from start to finish, but even low-brow humor can be done well. Of course I could have done without seeing the "T-Rex" character or the rampant shots of naked male ass... which yes, I understand is part of the overall joke of the film, but still...
You'd probably never think that a parody of both porn and of martial arts films would be cohesive enough to be entertaining, but Orgazmo manages to pull it off. Sure with these kinds of movies most of the jokes are hit and miss, but enough of the jokes work here, therefore making it a pretty entertaining flick for fans of Parker and Stone. [NC-17]
The Jacksonville Film Journal -- Film Reviews by Chuck Dowling URL:http://users.southeast.net/~chuckd21/<EMAIL_ADDRESS>© 1999 of The Jacksonville Film Journal. No reviews may be reprinted without permission.
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Home
Add radio station
Genres
News
Pop
Classic Hits
80s
Contact Us
Irish Radio Stations
{% block content %} {% endblock %}


Text-Only RSS Reader
Exit
storeavailable:
no
appState:
zero
lastReadLink:
lastReadLink
Feed Status:
feedStatus
landscape
portrait
You can get the code for this application at:
https://github.com/jessemonroy650/3rdParty-RSS-Examples
Add a Feed
Add Feed
Cancel
Feed
feedStatus:
feedStatus
getData
getFeeds
Feed Status:
feedStatus
debug2:
Created by Jesus Monroy, Jr. © 2016


Maksim, developer of ClickHouse.
1. High Level System Architecture.
2. CI/CD Pipeline.
3. Introspection.
4. Abstractions and Algorithms.
5. Libraries.
6. JIT compilation. Dynamic dispatch.
Column-oriented storage — data is physically stored by columns.
Only necessary columns are read from disk during query.
Better compression because of data locality.
Vectorized query execution — data is processed in blocks. Block contains multiple columns with max_block_size rows (65536 by default).
Each column is stored as a vector of primitive data types or their combination:
1. Better utilization for CPU caches and pipeline.
2. Data is processed using SIMD instructions.
Numeric columns — PODArray. Almost the same as std::vector.
1. Use our Allocator with support of realloc.
2. No additional memset during resize.
3. Padding with 15 bytes at the end.
Nullable columns contain data column and UInt8 column bitmask is element null.
Array columns contain data column and UInt64 column with offsets.
Const column contain 1 constant value.
Main class is IColumn.
Polymorphic type that can be part of interfaces.
Declares methods that all concrete column types need to support.
In most of the functions unwrapped to concrete column type.
class IColumn
{
...
[[nodiscard]] virtual Ptr filter(const Filter & filt, ssize_t result_size_hint) const = 0;
[[nodiscard]] virtual Ptr permute(const Permutation & perm, size_t limit) const = 0;
virtual void insertFrom(const IColumn & src, size_t n);
virtual void insertRangeFrom(const IColumn & src, size_t start, size_t length) = 0;
...
}
1. Functional, Integration tests.
2. Run all tests with sanitizers (ASAN, TSAN, MSAN).
3. Fuzzers (data types, compression codecs).
4. AST fuzzer.
5. Stress tests (Our special TSAN stress)..
6. Performance tests.
Part of CI/CD pipeline.
Runs for each commit in pull request.
Runs for each commit in the master branch.
Write test in special XML configuration.
<test>
<substitutions>
<substitution>
<name>func</name>
<values>
bitCount
bitNot
abs
...
</values>
</substitution>
<substitution>
<name>expr</name>
<values>
<value>number</value>
<value>toUInt32(number)</value>
...
</values>
</substitution>
</substitutions>
<query>SELECT {func}({expr}) FROM numbers(100000000) FORMAT Null</query>
</test>
Collect different statistics during each performance test run. Can be useful for later debugging.
Processor metrics (CPU cycles, cache misses same as perf-stat).
ClickHouse specific profile events (read bytes from disk, transferred over network, etc).
Helps us find performance regressions.
Nice tool that can help to find places where performance can be improved.
1. Try different allocators, different libraries.
2. Try different compiler options (loop unrolling, inline threshold)
3. Enable AVX/AVX2/AVX512 for build.
Query should not be short, because otherwise it measures nothing.
Query should not be very long. Performance of long queries can be more affected by random external factors.
It is also important to write a query in a special way to check some function, or feature in isolation.
1. Run two server versions simultaneously on the same machine before commit and after commit.
2. Run performance tests against two servers.
3. Measure medians for performance tests runs. Also collect all other possible statistics like CPUBranchMisses, CPUCacheMisses.
4. Use statistical methods to compute the largest difference in median query run T that we can observe even if nothing has changed.
5. Using D difference of medians and T decide if changes in performance are significant.
https://clickhouse.com/blog/testing-the-performance-of-click-house/If T is greater than 10 percent, the query is unstable.
For such a query we cannot detect any changes in performance because even if nothing has changed query time can be affected.
Should we just delete tests with unstable queries?
Several examples why query can be unstable:
1. Excessive allocations.
2. Large copying.
3. Bad written query.
4. External factors (disks, network, background activities, bugs in kernel).
We should only delete/rewrite tests with bad written queries. Other cases require deep investigation.
<test>
<query>
SELECT detectLanguageUnknown(SearchPhrase)
FROM hits_100m_single
LIMIT 500000
FORMAT Null
</query>
<query>
SELECT detectCharset(SearchPhrase)
FROM hits_100m_single
LIMIT 500000
FORMAT Null
</query>
</test>
Functions detectCharset, detectLanguageUnknown inner loop:
for (size_t i = 0; i < offsets.size(); ++i)
{
const UInt8 * str = data.data() + offsets[i - 1];
const size_t str_len = offsets[i] - offsets[i - 1] - 1;
HashMap<UInt16, UInt64, DefaultHash<UInt16>, 4> model;
/// Calculate string ngrams
calculateStats(str, str_len, model);
/// Compare ngrams with known encodings dictionaries and get result value
std::string_view result_value = matchEncoding(model, encodings_frequency);
/// Write result value into column
writeResult(result_column, result_value);
}
Functions detectCharset, detectLanguageUnknown inner loop:
for (size_t i = 0; i < offsets.size(); ++i)
{
const UInt8 * str = data.data() + offsets[i - 1];
const size_t str_len = offsets[i] - offsets[i - 1] - 1;
HashMapWithStackMemory<UInt16, UInt64, DefaultHash<UInt16>, 4> model;
/// Calculate string ngrams
calculateStats(str, str_len, model);
/// Compare ngrams with known encodings dictionaries and get result value
std::string_view result_value = matchEncoding(model, encodings_frequency);
/// Write result value into column
writeResult(result_column, result_value);
}
<test>
<query>
SELECT toString(rand()) IN ('41577', '83972', /*big_array*/);
</query>
<query>
SELECT (rand(), rand()) IN ((17258, 93148), /*big array of tuples*/);
</query>
</test>
Constant evaluation during query processing analyze part.
std::pair<Field, std::shared_ptr<const IDataType>>
evaluateConstantExpression(const ASTPtr & node, ContextPtr context)
{
/// Derive AST expression data type and evaluate it into constant Field
}
Constant evaluation during query processing analyze part.
class ASTLiteral : public ASTWithAlias
{
public:
explicit ASTLiteral(Field value_) : value(std::move(value_)) {}
Field value;
...
}
std::pair<Field, std::shared_ptr<const IDataType>>
evaluateConstantExpression(const ASTPtr & node, ContextPtr context)
{
/// Derive AST expression data type and evaluate it into constant Field
if (ASTLiteral * literal = node->as<ASTLiteral>())
return {literal->value, applyVisitor(FieldToDataType(), literal->value)};
}
Performance improvements after change:
For other performance tests, which construct big IN sections as part of their internal implementation we also saw improvement.
Collect ProfileEvents for each query:
RealTimeMicroseconds, UserTimeMicroseconds, SystemTimeMicroseconds, SoftPageFaults, HardPageFaults using getrusage system call.
https://man7.org/linux/man-pages/man2/getrusage.2.htmlCollect ProfileEvents for each query:
Collect :taskstats from procFS.
OSCPUVirtualTimeMicroseconds, OSCPUWaitMicroseconds (when /proc/thread-self/schedstat is available)
OSIOWaitMicroseconds (when /proc/thread-self/stat is available)
OSReadChars, OSWriteChars, OSReadBytes, OSWriteBytes (when /proc/thread-self/io is available)
We also support Netlink interface.
Collect ProfileEvents for each query:
Hardware specific counters CPU cache misses, CPU branch mispredictions using perf_event_open system call.
https://man7.org/linux/man-pages/man2/perf_event_open.2.htmlCollect ProfileEvents for each query:
Different ClickHouse specific metrics FileOpen, DiskReadElapsedMicroseconds, NetworkSendBytes.

Binary must be compiled with -fasynchronous-unwind-tables.
Send signal and collect traces in the signal handler, then put them in a special system table allowing for introspection.
1. There can be many edge cases during stack unwinding when .eh_frame section can contain broken unwind instructions (bug in compiler, bug in hand written assembly). LLVM libunwind will crash.
2. Unwind library should be signal safe, no allocations during traces collection.
Currently using a patched fork of LLVM libunwind.
Check all threads current stack trace from system.stack_trace
WITH arrayMap(x -> demangle(addressToSymbol(x)), trace) AS all
SELECT thread_name, thread_id, query_id, arrayStringConcat(all, '\n') AS res
FROM system.stack_trace LIMIT 1 FORMAT Vertical;
Row 1:
──────
thread_name: clickhouse-serv
thread_id: 125441
query_id:
res: pthread_cond_wait
std::__1::condition_variable::wait(std::__1::unique_lock&)
BaseDaemon::waitForTerminationRequest()
DB::Server::main(/*arguments*/)
Poco::Util::Application::run()
DB::Server::run()
Poco::Util::ServerApplication::run(int, char**)
mainEntryClickHouseServer(int, char**)
main
__libc_start_main
_start
Generate flamegraph of query execution
./clickhouse-client --query="SELECT
arrayStringConcat(
arrayMap(x -> concat(
splitByChar('/', addressToLine(x))[-1],
'#',
demangle(addressToSymbol(x))),
trace),
';') AS stack,
count(*) AS samples
FROM system.trace_log
WHERE (trace_type = 'Real') AND (query_id = '344b07d9-9d7a-48f0-a17e-6f5f6f3d61f5')
GROUP BY trace" | flamegraph.pl
For high performance systems interfaces must be determined by algorithms
Top-down approach does not work.
High-performance system must be designed concentrating on doing a single task efficiently.
Designed from hardware capabilities.
ClickHouse was designed to efficiently FILTER and GROUP BY data that fits in RAM.
There is no silver bullet, or best algorithm for any task.
Try to choose the fastest possible algorithm/algorithms for your specific task.
Performance must be evaluated on real data.
Most of the algorithms are affected by data distribution.
Complex task can be viewed as number of small tasks.
Such small tasks can also be viewed as special cases that can be optimized.
For any task there are dozens of different algorithms that can be combined together (Example Sorting, Aggregation).
Each algorithm can be tuned later using different low-level optimizations (Data layout, Specializations, SIMD instructions, JIT compilation).
High level design desigion — data must be processed not only by multiple threads, but by multiple servers.
Core component is the HashTable framework.
Different HashTable for different types of keys (Special StringHashTable for Strings).
Additional specializations for Nullable, LowCardinality
Tuned a lot of low-level details, like allocations, structures layout in memory, batch multiple operations to avoid virtual function calls.
Added JIT compilation for special cases.
Added cache of hash-table sizes.
Optimizing performance is about trying different approaches.
Most of the time without any results.
Each problem can have a lot degrees of freedom. For example Sorting:
Stable / nonstable?
External / in RAM?
With limit/without limit?
Is the data already almost sorted?
What about data distribution? How many unique values?
Can we use vectorized sorting algorithms?
Can we allocate additional memory?
Insert into ClickHouse main storage engine MergeTree
Must be very fast.
Data ingestion in ClickHouse is one of the main selling points.
We just sort a block of data that was inserted in ClickHouse by PRIMARY KEY and then write to disk. That is it.
void stableGetPermutation(const Block & block, const SortDescription & description,
IColumn::Permutation & out_permutation)
{
size_t size = block.rows();
out_permutation.resize(size);
for (size_t i = 0; i < size; ++i)
out_permutation[i] = i;
auto sort_desc = getColumnsWithSortDescription(block, description);
auto cmp = PartialSortingLess(sort_desc);
std::stable_sort(out_permutation.begin(), out_permutation.end(), cmp);
}
ALWAYS_INLINE int compare(size_t lhs, size_t rhs) const
{
int res = 0;
for (const auto & elem : columns)
{
auto nulls_direction = elem.description.nulls_direction;
res = elem.column->compareAt(lhs, rhs, *elem.column, nulls_direction);
res *= elem.description.direction;
if (res != 0)
break;
}
return res;
}
Worst thing for column databases is to process elements in rows.
class ColumnVector : IColumn
{
...
int compareAt(size_t n, size_t m,
const IColumn & rhs_, int nan_direction_hint) const override
{
auto rhs_value = assert_cast<const Self &>(rhs_).data[m];
return CompareHelper<T>::compare(data[n], rhs_value, nan_direction_hint);
}
...
};
Solution:
For a single column it is easy, we can just call getPermutation method directly. We need to change interface to support stable sort.
For multiple columns need to change interface of updatePermutation method to also support stable sort.
Update permutation method sort data only for equal ranges (initially 1 pair with values 0 and block size), and update equal ranges array. For the next column updatePermutation call will finish sorting only for these equal ranges.
Update IColumn base interface. Also update all subclasses.
Additional special cases for Nullable, LowCardinality, Sparse columns.
class IColumn {
...
virtual void getPermutation(PermutationSortDirection direction,
PermutationSortStability stability, size_t limit, int nan_direction_hint,
Permutation & res) const = 0;
virtual void updatePermutation(PermutationSortDirection direction,
PermutationSortStability stability, size_t limit, int nan_direction_hint,
Permutation & res, EqualRanges & equal_ranges) const = 0;
}
Performance results for random data.
Performance results for production data.
INSERT INTO test_hits_insert
SELECT UserID, WatchID, CounterID
FROM hits_100m_single
Was: 22.043 sec (4.54 million rows/s., 90.73 MB/s.)
Now: 11.268 sec (8.87 million rows/s., 177.50 MB/s.)
If someone on the Internet says my algorithm is fastest we will try it in ClickHouse.
Always try to find interesting algorithms, and solutions.
ClickHouse/contrib$ ls | grep -v "cmake" | wc -l
95
1. Different algorithms for parsing floats, json (multiple libraries).
2. A lot of integrations.
3. Embedded storages.
4. LLVM for JIT compilation.
5. libcxx (C++ standard library).
Almost in any library our CI system finds bugs. We report them to maintainers.
We also have a lot of library forks with a lot of changes. For example POCO, LLVM libunwind.
We are not afraid of adding additional contrib. Our CI system will do the job.
We have high-performance IO Framework, HashTable Framework, Different Data Structures, Vectorized Engine.
Should we make ClickHouse a library?
Vote for it.
JIT compilation can transform dynamic configuration into static configuration.
Not all functions can be easily compiled, not all algorithms can be easily compiled.
Has its own costs (compilation time, memory, maintenance).
But can greatly improve performance in special cases.
Compile evaluation of multiple expressions. Example: SELECT a + b * c + 5 FROM test_table;
Compile special cases for GROUP BY. Example: SELECT sum(a), avg(b), count(c) FROM test_table;
Compile comparator in ORDER BY. Example: SELECT * FROM test_table ORDER BY a, b, c;
In all cases we transform dynamic configuration into static.
My presentation from CPP Russia 2021 JIT in ClickHouse:
Compile evaluation of multiple expressions. Example: SELECT a + b * c + 5 FROM test_table;
void aPlusBMulitplyCPlusConstant(
int64_t * a,
int64_t * b,
int64_t * c,
int64_t constant,
int64_t * result,
size_t size)
{
for (size_t i = 0; i < size; ++i)
{
*result = (*a) + (*b) * (*c) + constant;
++a;
++b;
++c;
++result;
}
}
.LBB0_8: # %vector.body
vmovdqu (%r11,%rax,8), %ymm1
vmovdqu (%r9,%rax,8), %ymm3
vmovdqu 32(%r11,%rax,8), %ymm2
vmovdqu 32(%r9,%rax,8), %ymm4
vpsrlq $32, %ymm3, %ymm5
vpsrlq $32, %ymm1, %ymm6
vpmuludq %ymm1, %ymm5, %ymm5
vpmuludq %ymm6, %ymm3, %ymm6
vpmuludq %ymm1, %ymm3, %ymm1
vpsrlq $32, %ymm4, %ymm3
vpmuludq %ymm2, %ymm3, %ymm3
vpaddq %ymm5, %ymm6, %ymm5
vpsllq $32, %ymm5, %ymm5
vpaddq %ymm5, %ymm1, %ymm1
vpsrlq $32, %ymm2, %ymm5
vpmuludq %ymm2, %ymm4, %ymm2
vpaddq (%r14,%rax,8), %ymm1, %ymm1
vpmuludq %ymm5, %ymm4, %ymm5
vpaddq %ymm3, %ymm5, %ymm3
vpsllq $32, %ymm3, %ymm3
vpaddq %ymm3, %ymm2, %ymm2
vpaddq 32(%r14,%rax,8), %ymm2, %ymm2
vpaddq %ymm0, %ymm1, %ymm1 /// in ymm0 there is constant 5. vpbroadcastq (%rbp), %ymm0
vmovdqu %ymm1, (%r10,%rax,8)
vpaddq %ymm0, %ymm2, %ymm2
vmovdqu %ymm2, 32(%r10,%rax,8)
addq $8, %rax
cmpq %rax, %r8
ClickHouse distributed as portable binary.
We use the old instruction set SSE4.2.
For AVX, AVX2, AVX512 instructions need to use runtime instructions specialization using CPUID.
In addition a lot of companies bring us SIMD optimizations (ContentSquare, Intel), before most such optimizations were disabled during compilation time.
It is important that compilers can vectorize even complex loops. We can rely on this.
Main idea apply compiler flags to some functions, to compile it with AVX, AVX2, AVX512
Then in runtime check CPUID and execute specialized function.
For example for clang:
# define BEGIN_AVX512F_SPECIFIC_CODE \
_Pragma("clang attribute push(__attribute__((target(\"sse,sse2,sse3,ssse3,sse4,\
popcnt,avx,avx2, avx512f\"))), apply_to=function)")
\
# define BEGIN_AVX2_SPECIFIC_CODE \
_Pragma("clang attribute push(__attribute__((target(\"sse,sse2,sse3,ssse3,sse4,\
popcnt, avx,avx2\"))), apply_to=function)") \
\
# define END_TARGET_SPECIFIC_CODE \
_Pragma("clang attribute pop")
Usage for standalone functions:
DECLARE_DEFAULT_CODE (
int funcImpl() {
return 1;
}
) // DECLARE_DEFAULT_CODE
DECLARE_AVX2_SPECIFIC_CODE (
int funcImpl() {
return 2;
}
) // DECLARE_DEFAULT_CODE
/// Dispatcher function
int dispatchFunc() {
#if USE_MULTITARGET_CODE
if (isArchSupported(TargetArch::AVX2))
return TargetSpecific::AVX2::funcImpl();
#endif
return TargetSpecific::Default::funcImpl();
}
Standalone functions are not interesting.
#define AVX512_FUNCTION_SPECIFIC_ATTRIBUTE __attribute__((target("sse,sse2,sse3,\
ssse3,sse4,popcnt, avx,avx2,avx512f"))) \
\
#define AVX2_FUNCTION_SPECIFIC_ATTRIBUTE __attribute__((target("sse,sse2,sse3,ssse3,\
sse4,popcnt, avx,avx2"))) \
Insert specific attribute before function name.
We also need to generate functions with different names. Ideally with suffixes like SSE42, AVX2, AVX512.
int /*Insert specific attribute here*/ testFunctionImpl(int value)
{
return value;
}
int AVX2_FUNCTION_SPECIFIC_ATTRIBUTE testFunctionImplAVX2(int value)
{
return value;
}
Split function into header (MULTITARGET_FUNCTION_HEADER), name and body (MULTITARGET_FUNCTION_BODY) to insert specific attribute before name.
MULTITARGET_FUNCTION_AVX2_SSE42(
MULTITARGET_FUNCTION_HEADER(int),
/*We need to insert specific attribute here*/ testFunctionImpl,
MULTITARGET_FUNCTION_BODY((int value)
{
return value;
}))
Our infrastructure is ready.
Now how to find places where SIMD optimizations can be applied?
Make build in our performance tests for AVX, AVX2, AVX512.
If some performance tests will run much faster, we find place for dynamic dispatch.
Wrap such place into our macros.
Find a place using our performance tests.
template <typename Value>
void NO_INLINE addManyImpl(const Value * __restrict ptr, size_t start, size_t end)
{
ptr += start;
size_t count = end - start;
const auto * end_ptr = ptr + count;
/// Loop
T local_sum{};
while (ptr < end_ptr)
{
Impl::add(local_sum, *ptr);
++ptr;
}
Impl::add(sum, local_sum);
}
Wrap function in our dynamic dispatch macro.
MULTITARGET_FUNCTION_AVX2_SSE42(
MULTITARGET_FUNCTION_HEADER(
template <typename Value>
void NO_SANITIZE_UNDEFINED NO_INLINE
), addManyImpl,
MULTITARGET_FUNCTION_BODY((const Value * __restrict ptr, size_t start, size_t end)
{
ptr += start;
size_t count = end - start;
const auto * end_ptr = ptr + count;
/// Loop
T local_sum{};
while (ptr < end_ptr)
{
Impl::add(local_sum, *ptr);
++ptr;
}
Impl::add(sum, local_sum);
}))
Dispatch function based on CPUID:
template <typename Value>
void NO_INLINE addMany(const Value * __restrict ptr, size_t start, size_t end)
{
#if USE_MULTITARGET_CODE
if (isArchSupported(TargetArch::AVX2))
{
addManyImplAVX2(ptr, start, end);
return;
}
else if (isArchSupported(TargetArch::SSE42))
{
addManyImplSSE42(ptr, start, end);
return;
}
#endif
addManyImpl(ptr, start, end);
}
Loop in unary functions.
static void vector(const ArrayA & a, ArrayC & c)
{
/// Loop Op::apply is template for operation
size_t size = a.size();
for (size_t i = 0; i < size; ++i)
c[i] = Op::apply(a[i]);
}
Loop in unary functions.
MULTITARGET_FUNCTION_WRAPPER_AVX2_SSE42(
MULTITARGET_FH(static void NO_INLINE),
vectorImpl,
MULTITARGET_FB((const ArrayA & a, ArrayC & c) /// NOLINT
{
/// Loop Op::apply is template for operation
size_t size = a.size();
for (size_t i = 0; i < size; ++i)
c[i] = Op::apply(a[i]);
}))
roundDuration optimized even better than others using special AVX2 instructions.
For AVX2 we use such optimizations a lot.
For AVX512 currently we do not apply a lot of such optimizations. It potentially could decrease performance of other system parts.
Latest Intel processors like Rocket Lake and Ice Lake fix this issue. We can detect such processors in runtime and then use optimizations.
https://stackoverflow.com/questions/56852812/simd-instructions-lowering-cpu-frequency
We have AVX, AVX2, AVX512 pull requests that periodically build ClickHouse to check if there are some places to improve.
https://github.com/ClickHouse/ClickHouse/pull/34071
1. CI/CD infrastructure, especially performance tests, must be the core component of a high performance system.
2. Without deep introspection it is hard to investigate issues with performance.
3. For high performance systems interfaces must be determined by algorithms.
4. Add specializations for special cases.
5. Tune your performance using low-level techniques (Data layout, JIT compilation, Dynamic Dispatch).


--- layout: post title: "Sentiment Analysis?" subtitle: "Using Recurrent Neural Network" date: 2019-06-17 background: '/img/posts/reviews_ex.png' author: "Khaing Win" ---
In this project, I constructed a recurrent neural network (RNN) for the purpose of determining the sentiment of a review using the IMDB movie dataset. The model was created using Amazon's SageMaker
service. This model is then deployed and a simple web app is created which then interacts with the deployed model. Although the model was trained on movie reviews, the testing worked well on any kind of reviews.
Human language is composed of sequence of words, and the next word depends on the previous words in a sentence. When dealing with sequential information, RNN is more accurate.
How would the machine learning algorithm understand english language? or any other language?
We have to somehow find a way to turn the language into numerical representations.
Here, I used a dataset of movie reviews, accompanied by sentiment labels, whether the review is positive or negative.
First, we'll pass in words (which are our inputs) to an embedding layer.
Word embeddings work better than one-hot encoding since it's more efficient. Think about it: we have tens of thousands of words, we'll need a more efficient representation for our input data. That's why we need an embedding layer than one-hot encoded vectors.
We can actually train an embedding with the Skip-gram Word2Vec model
and use those embeddings as input. However, we are just going to use an embedding layer and let our network learn a different embedding table on its own.
In this case, the embedding layer is for the purpose of dimensionality reduction, rather than for learning semantic representations.
After input words are passed to an embedding layer, the new embeddings will be passed to LSTM cells.
The LSTM cells will add recurrent connections to the network.
This will give us the ability to include information about the sequence of words in the movie review data.
Finally, the LSTM outputs will go to a sigmoid output layer.
1
and negative review is labeled as 0
.0
-1
.Check out your sentiment on a review in my next post.
[SENTIMENT](sentiment-deploy.html)


Preparation time
30 mins to 1 hour
Cooking time
30 mins to 1 hour
Serves
Serves 4
For the kumquat marmalade, place a heavy-based pan on a moderate heat, and when hot, add the sunflower oil, followed by the red onion. Fry for 2-3 minutes until the onion has softened.
Add the cinnamon to the pan and continue frying for a further two minutes.
Tip in the kumquats and cook for 3-4 minutes, until just beginning to soften.
Add the brown sugar and pour over the vinegar.
Cover, and simmer for 8-10 minutes, until the kumquats have completely softened and lost their shape.
Remove the lid and continue boiling for a further 6-8 minutes until the juices have cooked down and the marmalade has thickened.
Remove the pan from the heat and leave to cool.
Preheat the oven to 220C/425F/Gas 7.
Season the cod with salt and black pepper.
Heat a frying pan until hot then add the olive oil and cod, skin-side down.
Cook for two minutes then place in the oven for 5-8 minutes until cooked through.
Meanwhile, to make the green pepper dressing, place the peppers and water into a food processor and blend to a purée.
Strain the mixture through a tea towel into a clean pan.
Place the pan on the heat and bring to a boil.
Reduce the heat and simmer for five minutes until the mixture has thickened.
Place the micro cress and fennel tops into a bowl and toss with the olive oil, salt and black pepper, to taste.
To serve, spoon the kumquat marmalade into the centre of four plates. Place the cod on top and sprinkle the micro cress around the plate. Finish with a drizzle of green pepper dressing.


An Interactive Property Information Platform that allows users to query for information on property
Benben will enable users to monitor and manage updates on their property records.


This is a simple wrapper for the yakmo K-Means library (developed by Naoki Yoshinaga, see http://www.tkl.iis.u-tokyo.ac.jp/~ynaga/yakmo/). It performs fast and robust (orthogonal) K-Means.
| Version: | 0.1.1 |
| Imports: | Rcpp (≥ 0.11.6), checkmate (≥ 1.5.1), BBmisc (≥ 1.9) |
| LinkingTo: | Rcpp |
| Suggests: | knitr, testthat |
| Published: | 2015-08-04 |
| Author: | Aydin Demircioglu |
| Maintainer: | Aydin Demircioglu <aydin.demircioglu at ini.rub.de> |
| License: | GPL-2 | GPL-3 [expanded from: GPL (≥ 2)] |
| Copyright: | Original GPL-2 C++ code by Naoki Yoshinaga (see http://www.tkl.iis.u-tokyo.ac.jp/~ynaga/yakmo/) |
| NeedsCompilation: | yes |
| CRAN checks: | yakmoR results |
| Reference manual: | yakmoR.pdf |
| Vignettes: |
yakmoR |
| Package source: | yakmoR_0.1.1.tar.gz |
| Windows binaries: | r-devel: yakmoR_0.1.1.zip, r-release: yakmoR_0.1.1.zip, r-oldrel: yakmoR_0.1.1.zip |
| OS X binaries: | r-release: yakmoR_0.1.1.tgz, r-oldrel: yakmoR_0.1.1.tgz |
| Old sources: | yakmoR archive |
Please use the canonical form https://CRAN.R-project.org/package=yakmoR to link to this page.


By Tay Yang Shun
For tutorial group 8, here's your real tutor
def odd(x):
if (x + 3) % 2 == 0:
return True
else:
return False
Redundant (and a little weird) conversion of
the parity of the input variable.
def odd(x):
if ((x % 2) == 1):
return True
elif ((x % 2) == -1):
return True
else:
return False
Redundant brackets and the result of modulo %
with a positive number (2 in this case) in Python will be a non-negative value.
Hence, the second elif
block is redundant.
def odd(x):
return x % 2 == 1
if x % 2 == 1:
return True
else:
return False
Putting code after the return
statement
does not cause it to be executed.
def odd(x):
if x % 2 == 0:
return False
else:
return True
By Samantha See
def odd(x):
if x % 2 != 0:
return True
else:
return False
By Xia Xiao Bo
def new_odd(x):
return odd(x)
def odd(x):
if (x + 3) % 2 == 0:
return True
else:
return False
Not supposed to reuse the code from the previous question!
def new_odd(x):
if x % 2 != 0:
return True
else:
return False
Blatantly ignoring the instructions and still used if/else
!
def new_odd(x):
while x % 2 > 0:
return True
else:
return False
while
doesn't really make sense here.
def new_odd(x):
return x%2 > 0
By Ang Yen Ling
def new_odd(x):
return x % 2 == 1
By Chua Wei Lin
def number_of_digits(x):
if (x / 1) < 10:
return 1
elif (x / 1) < 100:
return 2
elif (x / 1) < 1000:
return 3
elif (x / 1) < 10000:
return 4
elif (x / 1) < 100000:
return 5
elif (x / 1) < 1000000:
return 6
elif (x / 1) < 10000000:
return 7
elif (x / 1) < 100000000:
return 8
elif (x / 1) <<PHONE_NUMBER>:
return 9
elif (x / 1) <<PHONE_NUMBER>0:
return 10
Doesn't cover cases where x >=<PHONE_NUMBER>0
.
def number_of_digits(x):
import math
if x == 0:
return 1
elif x != 1000:
return int(math.log(x, 10)) + 1
else:
return round(math.log(x, 10)) + 1
Never use import
within a function.
Using log
is a viable solution. Here's a possible answer using
the convenient log10
function from the math
package:
import math
def number_of_digits(x):
return int(math.log10(x), 1) + 1 if x != 0 else 1
def number_of_digits(x):
return len(str(x))
There's a more fundamental way of doing
this without converting to a string.
def number_of_digits(x):
if x < 10:
return 1
else:
return 1 + digits(x // 10)
Solution using recursion. We will go through this again soon.
def bigger_sum(a, b, c):
if a > b > c:
return ( a*a + b*b )
elif b > a > c:
return ( b*b + a*a)
elif a > c > b:
return ( a*a + c*c )
elif b > c > a:
return ( b*b + c*c )
elif c > b > a:
return ( c*c + b*b )
elif c > a > b:
return ( c*c + a*a )
Redundant cases implemented. We don't have to
check the order of the two bigger numbers.
def bigger_sum(a, b, c):
if b > a and c > a:
return (b ** 2) + (c ** 2)
elif a > b and c > b:
return (a ** 2) + (c ** 2)
else:
return (a ** 2) + (b ** 2)
By Foo Li Wei
Last else
block doesn't need
conditions because it is the remaining case.
def bigger_sum(a, b, c):
if a < b and a < c:
return (b ** 2 + c ** 2)
elif b < a and b < c:
return (a ** 2 + c ** 2)
else:
return (a ** 2 + b ** 2)
By Melvin Tan Jun Keong
Brackets are redundant because the exponential operator **
has higher
precedence over the addition/substraction.
def bigger_sum(a, b, c):
return round((a**2 + b**2 + c**2 - (min(a, b, c)) ** 2), 6)
By Kelly Seah
round()
is used to handle floating point errors in double values arithmetic. Actually, the code is correct even without the usage of round()
.
def is_leap_year(year):
return (year % 400 == 0) or \
(year % 4 == 0 and not year % 100 == 0)
Short and elegant solution.
Note the \
is used there to allow code to be written across multiple lines.


essential garlic oil diallyl molitor compounds sulfide toxic disulfide adults insects oils concentrations {border-style .–. larvae px;} used = . mg mL #ffffff;} Tenebrio activity adult different insect rate toxicity control lethal trisulfide (), (.%), (Coleoptera: Diallyl Sci. pupa pupae stored values P < .), effects time (t = ., . ± . h, using , = ., Garlic castaneum Allium CHS Entomol. Sitophilus allyl applied concentration constituents followed zeamais (LC estimated pest respiration = ., Linnaeus Prod. P < .) Res. Tribolium disulfide, larva, products px; repellency tetrasulfide {font-family Fig. Food Tenebrionidae) Universidade application composition control. developmental df = ) font-size grain insecticidal larva larvae, molitor. obtained oil, pests test °C (L.) Brasil Chem. Federal Insecticidal Lethal Respiration SAS Stored Table [], body chemical double; gas identified kill level methyl molitor, mortality plant produced respectively. sativum showed solution stages temperature total -vinyl-[H]-,-dithiin Chemical Departamento Figure Mortality Pest Repellency action beetle, border-color border-width column dimethyl dishes effect evaluated exposed exposure food high index indicated main mealworm observed period physiological repellent responses solid; took treatment unknown volatile wheat {font-style {font-weight −), ./j.-...x ./ps. Control Econ. Manag. Petri Physiol. Serrão Toxicity Unknown Viçosa Zanuncio activities allicin analyses based bioassays bold bran caused commercially components compound courier; detector determined em; induced low mass number placed potential production products. pt;} pupae, results stock studies h, mL uL


--- layout: post title: Funny computer jargons date: type: post published: false status: private categories: [] tags: [] meta: blogger_blog: moresandeep.blogspot.com blogger_author<EMAIL_ADDRESS>blogger_permalink: /feeds/2599631319575757995/posts/default/7496287827629444856 author: login: Sandeep email: '' display_name: Sandeep first_name: '' last_name: '' ---


Loading...
- 1、Camera device
- 2、Speaker and MIC device
- 3、System
- iOS Safari's Version above 11.1.2
- iOS weChat not Support, pls replace with IOS Safari or Chrome
- Android Chrome's Version above 63
- Android weChat
- Mac Chrome's Version above 60
- Mac Safari's Version above 11
- Windows Chrome's Version above 60
- Windows QQ Browser's Version above 10 (Speed mode)
- Windows Sogou Browser's Version above 8.6
- Windows 360 Browser's Version above 12 (Speed mode)
- Windows Edge Browser's Version above 81
- Hot Swap is not supported. If replace the equipment, please visit again and do not replace the equipment during the meeting.
- Download Windows Chrome , Android Chrome ,
IOS Chrome, MAC Chrome
帮助
云会议
18:23:00


EffectiveBrowsing
Rate the current website with one of these categories. Thank you.
Select one
Informations
Work
Chat
Entertainment
Shopping/Finance
Don't know
Rate Website
Not now


--- layout: post current: post cover: false navigation: True title: Angular and Bulma date: 2018-01-30 10:00:00 tags: angular bulma class: post-template subclass: 'post' author: dave ---
I recently wanted to include the Bulma CSS library into my Angular project. I didn't just want to include the library in my project, I wanted to be able to include just pieces of the library and maybe override a thing or two. This post will explain how to include Bulma into your Angular projects using SASS.
You'll need to install angular-cli
first so that you can create an Angular project.
$ npm install -g angular-cli
Creating a new anuglar project is easy with angular-cli
. Let'g go ahead and use the ng new
command to create a new angular project. We will specify that we want to use SASS
for our CSS.
$ ng new angular-bulma --style=scss
I really don't like keeping my sass files at the root of my src
directory. So we'll create a sass
directory to hold our sass files and then move our styles.scss
into it.
$ cd src
$ mkdir sass
$ mv styles.scss sass
Now that we have moved the location of our main sass file, we'll have to update our angular-cli.json
configuration file to reflect that change. Search for styles
and replace it with the following:
"styles": [
"sass/styles.scss"
],
Now the fun begins. Lets go ahead and install Bulma
$ npm install bulma --save
You will be able to find Bulma sass files under /node_modules/bulma/sass
. We'll first import the initial-variables
sass file. After that we will import the main Bulma sass file.
@import "~bulma/sass/utilities/initial-variables";
@import "~bulma/bulma";
We'll add a button to our page. Find the app.component.html
file and add a button.
Success
Let's test it out. Start your app.
$ npm start
Go to your page by going here. You should see Bulma working now with a nice stylized button.
Lastly, we are going to override one of Bulma's color variables. Let's go ahead and add to our styles.scss
file. Before the Bulma import, lets add a color override and set the primary color to red.
// Customization
$blue: hsl(217, 10%, 53%);
$primary: $red;
Refresh the page and now you should see more of a grey color for the button than the default blue color.


This split pane component width is in percent. min-width
(0px
in this exampe) will be respected
Re-size the window and move the divider to see the effect!
This split pane component width is in percent. min-width
(0px
in this exampe) will be respected


Thai SEC Mulls Tweaking Existing Crypto Regulations to Boost Growth
by Martin YoungAsian nations such as Thailand are wasting no time in preparing their economies for a future that involves crypto assets. China is full steam ahead on blockchain technology, and Thailand is making plans to open up further to the digital asset industry with amendments to existing regulations.
Paving The Way For Thai Crypto Industry
According to reports in local media today, Thailand’s Securities and Exchange Commission plans to amend its digital asset business regulations next year. The overall aim is to facilitate the growth of crypto assets while protecting investors from unnecessary risk.
Secretary-general of the SEC, Ruenvadee Suwanmongkol, stated that they wish to study exiting regulations for any stumbling blocks that may hinder digital asset businesses within the kingdom.
“Laws should not be outdated and should serve market needs, especially for new digital asset products, and be competitive with the global market. We need to explore any possible obstacles.”
The current royal decree on crypto businesses came into effect in May 2018 listing four types of secondary business intermediaries which were crypto exchanges, brokerage firms, dealers, and token portal service providers or ICO portals.
As it currently stands exchanges, brokers and dealers are required to apply for licenses from the Finance Ministry, while ICOs must be approved by the SEC.
Ms Ruenvadee added that there have been many applications for ICOs in Thailand but they must first satisfy regulatory approval. She continued to state that investor protection was also a priority.
“The legislation also aims to protect investors from risk of fraud and deception by dishonest persons, money laundering and exploitation of digital assets to facilitate illegal financial transactions, while ensuring regulatory clarity to facilitate legitimate uses of digital assets,”
Digital Asset Licenses Awarded
According to the SEC, five crypto companies have been granted crypto-asset exchange licenses. These include Bitkub Online Co Ltd and Satang Pro Corporation which are currently operating.
In a move that shocked local markets earlier this year, Bitcoin Co unexpectedly closed the county’s most popular exchange, BX Thailand. It was reported at the time that regulators in the military-appointed government were demanding unreasonable amounts of transaction data from the firm and its clients.
That leaves two remaining portals Bitherb Co Ltd and Huobi Thailand Co Ltd which have yet to commence operations. The regulator has also authorized official licenses to three crypto brokers, Coin TH Co Ltd, bitherb.net and Bitazza Co Ltd.
The move has been seen as a step forward for Thailand which has been increasingly looking towards China for policy direction.
Will Thailand become a crypto hub in Asia? Add your comments below.
Image via Bitcoinist Media Library
The Rundown


Utopia::Controller
A middleware which loads controller classes and invokes functionality based on the requested path.
Nested
Definitions
CONTROLLER_RB = 'controller.rb'.freeze
The controller filename.
def initialize(app, root: Utopia::default_root, base: Controller::Base)
Implementation
def initialize(app, root: Utopia::default_root, base: Controller::Base)
@app = app
@root = root
@controller_cache = Concurrent::Map.new
@base = base
end
def lookup_controller(path)
Fetch the controller for the given relative path. May be cached.
Implementation
def lookup_controller(path)
@controller_cache.fetch_or_store(path.to_s) do
load_controller_file(path)
end
end
def load_controller_file(uri_path)
Loads the controller file for the given relative url_path.
Implementation
def load_controller_file(uri_path)
base_path = File.join(@root, uri_path.components)
controller_path = File.join(base_path, CONTROLLER_RB)
# puts "load_controller_file(#{path.inspect}) => #{controller_path}"
if File.exist?(controller_path)
klass = Class.new(@base)
# base_path is expected to be a string representing a filesystem path:
klass.const_set(:BASE_PATH, base_path.freeze)
# uri_path is expected to be an instance of Path:
klass.const_set(:URI_PATH, uri_path.dup.freeze)
klass.const_set(:CONTROLLER, self)
klass.class_eval(File.read(controller_path), controller_path)
# We lock down the controller class to prevent unsafe modifications:
klass.freeze
# Create an instance of the controller:
return klass.new
else
return nil
end
end
def invoke_controllers(request)
Invoke the controller layer for a given request. The request path may be rewritten.
Implementation
def invoke_controllers(request)
request_path = Path.from_string(request.path_info)
# The request path must be absolute. We could handle this internally but it is probably better for this to be an error:
raise ArgumentError.new("Invalid request path #{request_path}") unless request_path.absolute?
# The controller path contains the current complete path being evaluated:
controller_path = Path.new
# Controller instance variables which eventually get processed by the view:
variables = request.env[VARIABLES_KEY]
while request_path.components.any?
# We copy one path component from the relative path to the controller path at a time. The controller, when invoked, can modify the relative path (by assigning to relative_path.components). This allows for controller-relative rewrites, but only the remaining path postfix can be modified.
controller_path.components << request_path.components.shift
if controller = lookup_controller(controller_path)
# Don't modify the original controller:
controller = controller.clone
# Append the controller to the set of controller variables, updates the controller with all current instance variables.
variables << controller
if result = controller.process!(request, request_path)
return result
end
end
end
# Controllers can directly modify relative_path, which is copied into controller_path. The controllers may have rewriten the path so we update the path info:
request.env[Rack::PATH_INFO] = controller_path.to_s
# No controller gave a useful result:
return nil
end


Student population
Page 72 in the ISGAAmsterdam has a sizeable student population. Together, the UvA (University of Amsterdam) and VU University Amsterdam have over 50,000 students. The Universities of Applied Sciences (hogescholen or hbo) combined have approximately 55,000 students. Economics is the most popular course for the hogescholen, while Behavioural and Social Sciences attract the most students at the Amsterdam UvA and VU Universities.
Higher Education
There are two types of higher education institutions in the Netherlands, Universities and Universities of Applied Sciences (hogescholen or hbo). Hogescholen prepare students for specific professions and tend to be more practically oriented than universities, which focus more on theory and generating knowledge through research. Generally speaking, an hbo degree provides its graduates with sufficient skills to practise the profession they were trained for, without any additional training
Bachelor – Master
In the last decade, the system of higher education has been drastically reorganised to comply with the British or American model. This means that a university education has been subdivided into a broader, three-year bachelor’s degree and a one-year master’s degree. At hogescholen the bachelor’s degree takes four years while in some cases graduates will be able to continue their studies for one or two years for a master’s degree. The aim of this European reorganisation is to make the system of higher education more transparent throughout Europe and to make it easier to study in another country.


{% extends "base.html" %} {% block content %} Change password New password: Confirm password: Password and confirmation do not match. Password: Incorrect password Update {% endblock %}


This is a web-based interface that allows you to use X-Cash without running a full X-Cash node. However, because this convenience
comes at a cost: it is extremely difficult to securely deliver its code to your browser. This means that there is
considerable risk in using this service for large amounts!
It is recommended that you treat this service as you would treat your
actual wallet, and not store very large amounts in it. For long-term storage of X-Cash you should create a cold wallet using
X-Cash Address or similar.


If you don't have a motorbike, we can help you rent one with the option of a driver to take you around for the day. See pricing.
The trip includes sleeping accommodations and local food.
$70/day with your own motorbike
$80/day for motorbike + driver
Includes gas


|
|||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object com.izforge.izpack.installer.WebAccessor
public class WebAccessor
Dialogs for password authentication and firewall specification, when needed, during web installation.
| Nested Class Summary | |
|---|---|
private class |
WebAccessor.MyDialogAuthenticator
Authenticates via dialog when needed. |
| Field Summary | |
|---|---|
private int |
contentLength
|
private javax.swing.JDialog |
dialog
|
private javax.swing.JLabel |
errorLabel
|
private java.lang.Exception |
exception
|
private javax.swing.JTextField |
hostField
|
private java.io.InputStream |
iStream
|
private javax.swing.JTextField |
nameField
|
private java.lang.Thread |
openerThread
|
private java.awt.Component |
parent
|
private javax.swing.JPasswordField |
passField
|
private javax.swing.JPanel |
passwordPanel
|
private javax.swing.JTextField |
portField
|
private javax.swing.JLabel |
promptLabel
|
private javax.swing.JPanel |
proxyPanel
|
private java.lang.Object |
soloCancelOption
|
private boolean |
tryProxy
|
private java.lang.String |
url
|
| Constructor Summary | |
|---|---|
WebAccessor()
Not yet Implemented: placeholder for headless installs. |
|
WebAccessor(java.awt.Component parent)
Create a WebAccessor that prompts for proxies and passwords using a JDialog. |
| Method Summary | |
|---|---|
int |
getContentLength()
|
private javax.swing.JPanel |
getPasswordPanel()
|
private javax.swing.JPanel |
getProxyPanel()
Only to be called after an initial error has indicated a connection problem |
java.lang.String |
getUrl()
|
java.io.InputStream |
openInputStream(java.net.URL url)
Opens a URL connection and returns it's InputStream for the specified URL. |
void |
setUrl(java.lang.String url)
|
private void |
startOpening(java.net.URL url)
|
| Methods inherited from class java.lang.Object |
|---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Field Detail |
|---|
private java.lang.Thread openerThread
private java.io.InputStream iStream
private java.lang.Exception exception
private java.lang.Object soloCancelOption
private java.awt.Component parent
private javax.swing.JDialog dialog
private boolean tryProxy
private javax.swing.JPanel passwordPanel
private javax.swing.JLabel promptLabel
private javax.swing.JTextField nameField
private javax.swing.JPasswordField passField
private javax.swing.JPanel proxyPanel
private javax.swing.JLabel errorLabel
private javax.swing.JTextField hostField
private javax.swing.JTextField portField
private java.lang.String url
private int contentLength
| Constructor Detail |
|---|
public WebAccessor()
java.lang.UnsupportedOperationException
public WebAccessor(java.awt.Component parent)
parent
- determines the frame in which the dialog is displayed; if the parentComponent
has no Frame, a default Frame is used| Method Detail |
|---|
public java.io.InputStream openInputStream(java.net.URL url)
url
- the url to open the stream to.
private void startOpening(java.net.URL url)
private javax.swing.JPanel getProxyPanel()
private javax.swing.JPanel getPasswordPanel()
public java.lang.String getUrl()
public void setUrl(java.lang.String url)
public int getContentLength()
|
|||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |


Intercepting Invocations
CORBA brokers provide support for interception of GIOP requests and replies both in the server and client sides. Although the interception model is heavily influenced by the CORBA Portable Interceptors specification, OiL does not stick to it. Like other aspects of the OiL's API, it priories a simpler model more adequate to Lua than the defined by the CORBA standard.
Interceptors
Invocation interception is done by interceptor objects, which can be register in the broker as a client interceptor or a server interceptor. Client interceptors intercept invocations issued to remote servants, while server interceptors intercept invocations received from remote applications.
A client interceptor is an object that provides either methods sendrequest
or receivereply
.
The first method is called before an invocation request is sent to a remote servant, and the second is called after a reply is received from a remote servant.
On the other hand, a server interceptor is an object that provides either methods receiverequest
or sendreply
, which are called before an invocation request is processed by the server, and after a reply is sent to the corresponding invoker, respectively.
Each one of these methods receives a single parameter, which is a table representing the invocation.
The structure and information contained in this table varies accordingly to the method being called.
However, for a single invocation issued or received, the table received by method sendrequest
and receivereply
are always the same.
The same is true for methods receiverequest
and sendreply
, which means that the parameter received by the former method is the same parameter passed to the corresponding invocation of the later.
This also means that any field stored in this table during the execution of the first method will be available during the execution of the other for a single invocation.
For an example, consider the code below that measures the time each invocation takes to conclude.
local socket = require "socket" TimeProfiler = {} function TimeProfiler:receiverequest(request) request.start = socket.gettime() end function TimeProfiler:sendreply(reply) local time = socket.gettime() - reply.start print(string.format("opreation '%s' lasted %d seconds", reply.operation, time)) end
Client Interceptors
The fields provided by table request
of operation sendrequest
of the client interceptor are:
request_id |
number | [read-only] | Number that identifies the pending request over the connection used to communicate with the remote ORB. Note that this identifier is not unique across different connections, and it may be reused by other requests after this one is completed (i.e. replied). |
response_expected |
boolean | Flag that indicates if the server must send a response for this request. | |
reference |
table | [read-only] | The complete IOR structure (see CORBA::IOP::IOR in CORBA specs.) that identifies the servant the request is sent to. |
object_key |
string | [occasional] | Key of the object the request is addressed to. | If the ORB is not able to decode an IOR profile, this field is
profile_tag |
number | [occasional, read-only] | The identification of the kind of the IOR profile used to establish a connection with the servant's ORB.
If the ORB is not able to establish a connection, this field is nil . |
profile_data |
table | [occasional, read-only] | The decoded profile data used to establish a connection with the servant's ORB.
If the ORB is not able to establish a connection, this field is nil .
The structure of this profile depends on the value of the profile_tag .
For IIOP profiles (profile_tag == 0 ), this table follows the structure of CORBA::IIOP::ProfileBody_1_0 . |
interface |
object | [read-only] | Complete description of the interface of the servant being invoked. |
interface_name |
string | [read-only] | Name of the the interface of the servant being invoked. |
operation |
object | Description of the operation being invoked.
To change the operation being invoked on the remote servant, the application must set a new operation description to this field, which can be obtained from the local Interface Repository (see field types ) |
|
operation_name |
string | [read-only] | Name of the operation being invoked. |
parameters |
table | List of the parameters passed to the operation being invoked. | |
service_context |
table | [optional] | Set this value to define the list of service context data to be marshaled in the request.
See CORBA::IOP::ServiceContextList in CORBA specs. |
success |
boolean | [optional] | Set this value to cancel request:
true cancels the request and indicates to the invocation was successful;
false cancels the request and indicates to the invocation raised an exception. |
results |
table | [optional] | Set this value to provide the result values of a canceled request. This value must be a list of the values to be returned (in case of a successful invocation) or a list with a single value indicating the raised exception or error (in case of a failed invocation). |
reply_service_context |
table | [optional] | Set this value to define the list of service context to be added to the reply of a cancelled request.
See CORBA::IOP::ServiceContextList in CORBA specs. |
forward_reference |
table | [optional] | Set this value to forward this request to another reference.
This value must be a complete IOR structure (see CORBA::IOP::IOR in CORBA specs.). |
The fields added to table request
of operation receivereply
of the client interceptor are (all other fields provided on the corresponding sendrequest
operation are still available):
success |
boolean | Indication of success of the invocation.
true indicates the invocation was successful;
false indicates the invocation raised an exception. |
|
results |
table | List of the values to be returned (in case of a successful invocation) or a list with a single value indicating the raised exception or error (in case of a failed invocation). | |
reply_service_context |
table | The list of service context received with the reply of the request.
See CORBA::IOP::ServiceContextList in CORBA specs. |
Server Interceptors
The fields provided by table request
of operation receiverequest
of the client interceptor are:
request_id |
number | [read-only] | Number that identifies the pending request over the connection used to communicate with the remote ORB. Note that this identifier is not unique across different connections, and it may be reused by other requests after this one is completed (i.e. replied). |
response_expected |
boolean | [read-only] | Flag that indicates if the server must send a response for this request. |
servant |
table | [occasional, read-only] | The implementation of the object request is addressed to.
If the ORB cannot find a servant with the provided object_key then this field is nil . |
object_key |
string | Key of the object the request is addressed to. | |
interface |
object | [read-only] | Complete description of the interface of the servant being invoked.
If the ORB cannot find a servant with the provided object_key then this field is nil . |
interface_name |
string | [occasional, read-only] | Name of the the interface of the servant being invoked.
If the ORB cannot find a servant with the provided object_key then this field is nil . |
operation |
object | [occasional] | Description of the operation being invoked.
If the ORB cannot find a servant with the provided object_key or the operation requested is not defined in the servant's interface then this field is nil .
To change the operation being invoked on the remote servant, the application must set a new operation description to this field, which can be obtained from the local Interface Repository (see field types ) |
operation_name |
string | [read-only] | Name of the operation being invoked. |
parameters |
table | [occasional] | List of the parameters passed to the operation being invoked.
If the ORB cannot find a servant with the provided object_key or the operation requested is not defined in the servant's interface then this field is nil . |
service_context |
table | [optional] | Set this value to define the list of service context data to be marshaled in the request.
See CORBA::IOP::ServiceContextList in CORBA specs. |
success |
boolean | [optional] | Set this value to cancel request:
true cancels the request and indicates to the invocation was successful;
false cancels the request and indicates to the invocation raised an exception. |
results |
table | [optional] | Set this value to provide the result values of a canceled request. This value must be a list of the values to be returned (in case of a successful invocation) or a list with a single value indicating the raised exception or error (in case of a failed invocation). |
reply_service_context |
table | [optional] | Set this value to define the list of service context to be added to the reply of a cancelled request.
See CORBA::IOP::ServiceContextList in CORBA specs. |
forward_reference |
table | [optional] | Set this value to forward this request to another reference.
This value must be a complete IOR structure (see CORBA::IOP::IOR in CORBA specs.). |
The fields added to table request
of operation sendreply
of the client interceptor are (all other fields provided on the corresponding receiverequest
operation are still available):
success |
boolean | Indication of success of the invocation.
true indicates the invocation was successful;
false indicates the invocation raised an exception. |
|
results |
table | List of the values to be returned (in case of a successful invocation) or a list with a single value indicating the raised exception or error (in case of a failed invocation). | |
reply_service_context |
table | The list of service context received with the reply of the request.
See CORBA::IOP::ServiceContextList in CORBA specs. |
|
forward_reference |
table | [optional] | Set this value to forward this request to another reference.
This value must be a complete IOR structure (see CORBA::IOP::IOR in CORBA specs.). |
Adding Information to Invocations
GIOP messages for requests and replies has the special field service_context
, which can be used to store additional information to be send with the invocation.
Basically, the service_context
is a sequence of structures, each containing a tag number and a octet sequence that is used to store arbitrary data.
Interceptors can define the value of field service_context
in the parameter of methods sendrequest
or sendreply
.
Similarly, the value of field service_context
can be accessed through the parameter of methods receiverequest
or receivereply
.
To promote interoperability with other CORBA ORBs, the information contained in field service_context
is usually encoded using CORBA's CDR.
To help this, OiL brokers provide methods newencoder
()
and newdecoder
(stream)
.
The example below, illustrates how to use interceptors to add authentication information in an invocation.
require "oil" oil.main(function() local broker = oil.init() print("Type the your username") local Username = io.read() print("Type the password") local Password = io.read() local AuthID = broker:loadidl"struct AuthID { string user; string pswd; };" local encoder = broker:newencoder() encoder:put({user=UserName,pswd=Password}, AuthID) local AuthData = encoder:getdata() local Authenticator = {} function Authenticator:sendrequest(request) request.service_context[#request.service_context + 1] = { context_id = 404, context_data = AuthData, } end broker:setinterceptor(Authenticator, "client") ... end)Server
require "oil" oil.main(function() local broker = oil.init() local AuthID = broker:loadidl[[ struct AuthID { string user; string pswd; }; ]] local Validator = {} function Validator:receiverequest(request) local valid for _, context in ipairs(request.service_context) do if context.context_id == 404 then local decoder = broker:newdecoder(context.context_data) local authid = decoder:get(AuthID) valid = (authid.user == "admin") and (authid.pswd == "admin") end end if not valid then -- cancel request with an exception request.success = false request[1] = broker:newexcept{ "CORBA::NO_PERMISSION", minor_code_value = 0, } end end broker:setinterceptor(Validator, "server") ... end)
Multithreading Issues
OiL guarantees that for each single invocation both methods of the interceptor is invoked in the same thread.
Moreover, this thread is the same that initiated the invocation (in case of a client interceptor) or the thread that will execute the invocation (in case of a server interceptor).
However, while an invocation is being executed, others can be issued and the interceptor be invoked again, but it will execute in a different thread.
Therefore, you can use the current thread (oil.tasks.current
) to identify an invocation.
On the server side, the ORB may reuse a thread that executed a previous invocation that finished to execute other invocations.
Multiple Interceptors
Although CORBA brokers only allow one interceptor to be defined on the client side and one on the server side, if you need to define a chain of interceptors, you can register an interceptor that delegates interception to a chain of interceptors.
The following example illustrates the implementation of a interception chain that does not guarantee the order the interceptors are invoked.
The implementation use class loop.object.Publisher
of LOOP library.
require "oil" require "loop.object.Publisher" oil.main(function() local broker = oil.init() InterceptionChain = loop.object.Publisher{ Interceptor_01, Interceptor_02, Interceptor_03, ... } broker:setinterceptor(InterceptionChain, "server") ... end)