Github
Github username
Connect
Your account is too recent. We are unable to verify you as a developer.


I'm currently a structural engineer in training, but I like to dabble in code in my free time. This website was made from scratch by me.
I have both engineering and coding skills.
Find me on GitHub! @Eliz964


--- layout: post title: Simple document download categories: [] tags: serviceapi dotnet-serviceapi-client status: publish type: post published: true meta: {} ---
In a previous post I showed how to patch the ServiceAPI TrimClient to fix a problem with downloading child documents (e.g. record revisions). A customer (Mark from WA) pointed out a way to download a file to a stream without having to write it to a file.
The following code downloads a document using a stream. I then save to a file but it could also be passed on as a stream or byte array to another system.
Stream stream = trimClient.Get<Stream>(string.Format("Record/{0}/File/Document", 1843)); using (FileStream fs = File.Create("d:\\junk\\myfile.xml")) { stream.CopyTo(fs); }
Below is the code to use to get a record revision as a stream.
Stream stream = trimClient.Get<Stream>(string.Format("Record/{0}/RecordRevision/{1}", 1843, 12));
Two side benefits of this approach are:


[Part IV.] TELEVISION RECOVERY SYSTEM
Revision Note
This part, designated as part III by L 2009, c 183, §6, is redesignated as part IV pursuant to §23G-15.
[§339D-21] Applicability. The recycling provisions of this part shall apply only to covered televisions used and discarded in this State by a covered entity. [L 2009, c 183, pt of §6]


The next three boxes should look identical. (If the second and third look different, then margin collapsing through empty elements is broken. Go to these tests for more details.)


pulseaudio
Section: User Commands (1)Updated: User
Index Return to Main Contents
NAME
pulseaudio - The PulseAudio Sound SystemSYNOPSIS
pulseaudio [options]pulseaudio --help
pulseaudio --version
pulseaudio --dump-conf
pulseaudio --dump-modules
pulseaudio --dump-resample-methods
pulseaudio --cleanup-shm
pulseaudio --start
pulseaudio --kill
DESCRIPTION
PulseAudio is a networked low-latency sound server for Linux, POSIX and Windows systems.OPTIONS
- -h | --help
- Show help.
- --version
- Show version information.
- --dump-conf
- Load the daemon configuration file daemon.conf (see below), parse remaining configuration options on the command line and dump the resulting daemon configuration, in a format that is compatible with daemon.conf.
- --dump-modules
- List available loadable modules. Combine with -v for a more elaborate listing.
- --dump-resample-methods
- List available audio resamplers.
- --cleanup-shm
- Identify stale PulseAudio POSIX shared memory segments in /dev/shm and remove them if possible. This is done implicitly whenever a new daemon starts up or a client tries to connect to a daemon. It should normally not be necessary to issue this command by hand. Only available on systems with POSIX shared memory segments implemented via a virtual file system mounted to /dev/shm (e.g. Linux).
- --start
- Start PulseAudio if it is not running yet. This is different from starting PulseAudio without --start which would fail if PA is already running. PulseAudio is guaranteed to be fully initialized when this call returns. Implies --daemon.
- -k | --kill
- Kill an already running PulseAudio daemon of the calling user (Equivalent to sending a SIGTERM).
- --check
- Return 0 as return code when the PulseAudio daemon is already running for the calling user, or non-zero otherwise. Produces no output on the console except for errors to stderr.
- --system[=BOOL]
- Run as system-wide instance instead of per-user. Please note that this disables certain features of PulseAudio and is generally not recommended unless the system knows no local users (e.g. is a thin client). This feature needs special configuration and a dedicated UNIX user set up. It is highly recommended to combine this with --disallow-module-loading (see below).
- -D | --daemonize[=BOOL]
- Daemonize after startup, i.e. detach from the terminal.
- --fail[=BOOL]
- Fail startup when any of the commands specified in the startup script default.pa (see below) fails.
- --high-priority[=BOOL]
- Try to acquire a high Unix nice level. This will only succeed if the calling user has a non-zero RLIMIT_NICE resource limit set (on systems that support this), or we're called SUID root (see below), or we are configure to be run as system daemon (see --system above). It is recommended to enable this, since it is only a negligible security risk (see below).
- --realtime[=BOOL]
- Try to acquire a real-time scheduling for PulseAudio's I/O threads. This will only succeed if the calling user has a non-zero RLIMIT_RTPRIO resource limit set (on systems that support this), or we're called SUID root (see below), or we are configure to be run as system daemon (see --system above). It is recommended to enable this only for trusted users, since it is a major security risk (see below).
- --disallow-module-loading[=BOOL]
- Disallow module loading after startup. This is a security feature since it disallows additional module loading during runtime and on user request. It is highly recommended when --system is used (see above). Note however, that this breaks certain features like automatic module loading on hot plug.
- --exit-idle-time=SECS
- Terminate the daemon when idle and the specified number of seconds passed.
- --module-idle-time=SECS
- Unload autoloaded modules when idle and the specified number of seconds passed.
- --scache-idle-time=SECS
- Unload autoloaded samples from the cache when the haven't been used for the specified number of seconds.
- --log-level[=LEVEL]
- If an argument is passed, set the log level to the specified value, otherwise increase the configured verbosity level by one. The log levels are numerical from 0 to 4, corresponding to error, warn, notice, info, debug. Default log level is notice, i.e. all log messages with lower log levels are printed: error, warn, notice.
- -v
- Increase the configured verbosity level by one (see --log-level above). Specify multiple times to increase log level multiple times.
- --log-target={auto,syslog,stderr}
- Specify the log target. If set to auto (which is the default), then logging is directed to syslog when --daemonize is passed, otherwise to STDERR.
- --p | --dl-search-path=PATH
- Set the search path for dynamic shared objects (plugins).
- --resample-method=METHOD
- Use the specified resampler by default (See --dump-resample-methods above for possible values).
- --use-pid-file[=BOOL]
- Create a PID file. If this options is disabled it is possible to run multiple sound servers per user.
- --no-cpu-limit[=BOOL]
- Do not install CPU load limiter on platforms that support it. By default, PulseAudio will terminate itself when it notices that it takes up too much CPU time. This is useful as a protection against system lockups when real-time scheduling is used (see below). Disabling this meachnism is useful when debugging PulseAudio with tools like valgrind(1) which slow down execution.
- --disable-shm[=BOOL]
- PulseAudio clients and the server can exchange audio data via POSIX shared memory segments (on systems that support this). If disabled PulseAudio will communicate exclusively over sockets. Please note that data transfer via shared memory segments is always disabled when PulseAudio is running with --system enabled (see above).
- -L | --load="MODULE ARGUMENTS"
- Load the specified plugin module with the specified arguments.
- -F | --file=FILENAME
- Run the specified script on startup. May be specified multiple times to specify multiple scripts to be run in order. Combine with -n to disable loading of the default script default.pa (see below).
- -C
- Open a command interpreter on STDIN/STDOUT after startup. This may be used to configure PulseAudio dynamically during runtime. Equivalent to --load=module-cli.
- -n
- Don't load default script file default.pa (see below) on startup. Useful in conjunction with -C or --file.
FILES
~/.pulse/daemon.conf, /etc/pulse/daemon.conf: configuration settings for the PulseAudio daemon. If the version in the user's home directory does not exist the global configuration file is loaded. See pulse-daemon.conf(5) for more information.~/.pulse/default.pa, /etc/pulse/default.pa: the default configuration script to execute when the PulseAudio daemon is started. If the version in the user's home directory does not exist the global configuration script is loaded. See default.pa(5) for more information.
~/.pulse/client.conf, /etc/pulse/client.conf: configuration settings for PulseAudio client applications. If the version in the user's home directory does not exist the global configuration file is loaded. See pulse-client.conf(5) for more information.
SIGNALS
SIGINT, SIGTERM: the PulseAudio daemon will shut down (Same as --kill).SIGHUP: dump a long status report to STDOUT or syslog, depending on the configuration.
SIGUSR1: load module-cli, allowing runtime reconfiguration via STDIN/STDOUT.
SIGUSR2: load module-cli-protocol-unix, allowing runtime reconfiguration via a AF_UNIX socket. See pacmd(1) for more information.
UNIX GROUPS AND USERS
Group pulse-rt: if the PulseAudio binary is marked SUID root, then membership of the calling user in this group decides whether real-time and/or high-priority scheduling is enabled. Please note that enabling real-time scheduling is a security risk (see below).Group pulse-access: if PulseAudio is running as a system daemon (see --system above) access is granted to members of this group when they connect via AF_UNIX sockets. If PulseAudio is running as a user daemon this group has no meaning.
User pulse, group pulse: if PulseAudio is running as a system daemon (see --system above) and is started as root the daemon will drop priviliges and become a normal user process using this user and group. If PulseAudio is running as a user daemon this user and group has no meaning.
REAL-TIME AND HIGH-PRIORITY SCHEDULING
To minimize the risk of drop-outs during playback it is recommended to run PulseAudio with real-time scheduling if the underlying platform supports it. This decouples the scheduling latency of the PulseAudio daemon from the system load and is thus the best way to make sure that PulseAudio always gets CPU time when it needs it to refill the hardware playback buffers. Unfortunately this is a security risk on most systems, since PulseAudio runs as user process, and giving realtime scheduling priviliges to a user process always comes with the risk that the user misuses it to lock up the system -- which is possible since making a process real-time effectively disables preemption.To minimize the risk PulseAudio by default does not enable real-time scheduling. It is however recommended to enable it on trusted systems. To do that start PulseAudio with --realtime (see above) or enabled the appropriate option in daemon.conf. Since acquiring realtime scheduling is a priviliged operation on most systems, some special changes to the system configuration need to be made to allow them to the calling user. Two options are available:
On newer Linux systems the system resource limit RLIMIT_RTPRIO (see setrlimit(2) for more information) can be used to allow specific users to acquire real-time scheduling. This can be configured in /etc/security/limits.conf, a resource limit of 9 is recommended.
Alternatively, the SUID root bit can be set for the PulseAudio binary. Then, the daemon will drop root priviliges immediately on startup, however retain the CAP_NICE capability (on systems that support it), but only if the calling user is a member of the pulse-rt group (see above). For all other users all capababilities are dropped immediately. The advantage of this solution is that the real-time priviliges are only granted to the PulseAudio daemon -- not to all the user's processes.
Alternatively, if the risk of locking up the machine is considered too big to enable real-time scheduling, high-priority scheduling can be enabled instead (i.e. negative nice level). This can be enabled by passing --high-priority (see above) when starting PulseAudio and may also be enabled with the approriate option in daemon.conf. Negative nice levels can only be enabled when the appropriate resource limit RLIMIT_NICE is set (see setrlimit(2) for more information), possibly configured in /etc/security/limits.conf. A resource limit of 31 (corresponding with nice level -11) is recommended.
ENVIRONMENT VARIABLES
The PulseAudio client libraries check for the existance of the following environment variables and change their local configuration accordingly:$PULSE_SERVER: the server string specifying the server to connect to when a client asks for a sound server connection and doesn't explicitly ask for a specific server.
$PULSE_SINK: the symbolic name of the sink to connect to when a client creates a playback stream and doesn't explicitly ask for a specific sink.
$PULSE_SOURCE: the symbolic name of the source to connect to when a client creates a record stream and doesn't explicitly ask for a specific source.
$PULSE_BINARY: path of PulseAudio executable to run when server auto-spawning is used.
$PULSE_CLIENTCONFIG: path of file that shall be read instead of client.conf (see above) for client configuration.
These environment settings take precedence -- if set -- over the configuration settings from client.conf (see above).
AUTHORS
The PulseAudio Developers <mzchyfrnhqvb (at) 0pointer (dot) net>; PulseAudio is available from http://pulseaudio.org/SEE ALSO
pulse-daemon.conf(5), default.pa(5), pulse-client.conf(5), pacmd(1)
Index
- NAME
- SYNOPSIS
- DESCRIPTION
- OPTIONS
- FILES
- SIGNALS
- UNIX GROUPS AND USERS
- REAL-TIME AND HIGH-PRIORITY SCHEDULING
- ENVIRONMENT VARIABLES
- AUTHORS
- SEE ALSO
This document was created by man2html, using the manual pages.
Time: 05:29:09 GMT, December 24, 2015


Nuclear Weapon Site Credits and Thanks
Last changed 3 January 2005
This is a list of giving brief mention of those who have contributed
to this site. From some, the contribution has been major, from some it
has been minor. Since it is difficult or impossible to fairly list
everyone by the magnitude of their contribution, I will instead opt
for an alphabetical list.
Those who have contributed specifically to the Nuclear Weapons Frequently Asked Questions (aka NWFAQ) are credited separately in the introductory section of the NWFAQ.
A special thanks to those who have chosen to remain anonymous.
- Gary Au
- Lauren Bayliss
- Nicholas Badenhorst
- Professor Jeremy Bernstein
- Jack Boureston
- Guy Brown
- Richard K. Brown
- Dr. Robert R. Brownlee
- Brian Cathcart
- Sharon Cisneros
- Brian Davis
- Eric Erpelding
- Barnaby Finch
- Gabriel A. Gardiner
- "Gator"
- Gennadi Gorelik
- Andy Gillette
- Ron Gray
- Andre Gsponer
- Chuck Hansen
- Stewart Henderson
- C. Nicholas Hill
- Jim Kajpust
- Josh Knauer
- Peter Kuran
- Jim Lawson
- Michael Levi
- Matthew E. Massey
- Paul McDonell
- Karl Z. Morgan
- Howard Morland
- Sidd(hartha) Mukherjee
- John Pasley
- John Pike
- Mungo Poore
- "RayPix" (Ray Giguere)
- Dale Reid
- Prof. Seasonov
- Howard Seguine
- Yogi Shan
- Dr. S.K. Sikka
- Antonello Sotgiu
- Peter Vogel
- Gregory Walker
- Mark Sublette
- Matthias Warkus
- Prof. Raymond G. Wilson
If I have slighted anyone who feels they should be included, please contact me.
--Carey Sublette


--- layout: page title: "About" header-img: "img/transit.jpg" group: "navigation" order: 2 ---
What: Music reviews, mostly from the worlds of hip-hop and R&B, with an emphasis on quality artists and projects that are underrepresented or underappreciated.
Who: Just a passionate music fan, not an artist in any sense.
Where: Here on your computer or smartphone, brought to you by my resident coding nerd.
When: Several times a week (call it 3 or so), but without a schedule.
Why: See The Grand Scheme for more.
Contact Us: Hit the little mail button


The National Incident-Based Reporting System
Toggle navigation
(NIBRS) Pre-Certification Tool
Home
Resources
Tool Limitations
Test Files
About
Signed in as
Sign Out


We make it simple to launch in the cloud and scale up as you grow—whether you’re running one virtual machine or ten thousand.
By signing up you agree to the Terms of Service. Terms of Service


This page contains Section 4.2 of the following article:
High-Level Specifications: Lessons from
Industry This section was written by Brannon Batson, who was then an Intel engineer. It was preceded by a section describing the use of TLA+ and TLC at Compaq. Batson and some other Compaq engineers moved from Compaq to Intel.
4.2 IntelWe now describe the use of TLA+ and TLC by B[rannon] B[atson] and his colleagues at Intel. The actual specifications are proprietary and have not been viewed by anyone outside Intel.
4.2.1 Overview of the ProblemDesigning a complex system starts with a problem statement and an appropriate set of boundary conditions. A component of a computer system is initially represented abstractly as a black box, with assertions about its functionality and with some guidelines on performance, cost, and complexity. The engineering process involves iteratively refining this abstract model into lower-level models. Each lower-level model is a representation of the design at a certain level of abstraction, and it has a specific purpose. Some models are meant to evaluate tradeoffs between scope, performance, cost, and complexity. Others carry the design down to the low level of detail needed to manufacture the component.The engineering process therefore creates multiple representations of a design. Validation entails checking these multiple representations against one another. Designers of digital systems have good tools and methods for validating mid-level functional models, written in a hardware description language (HDL), against lower-level models such as circuit net-lists. However, they have not had as much success checking the higher-level functional representations of the design against one another, and against the initial problem statement and functional assertions. For some components, there is an intuitive correlation between the high-level notions of correctness and the HDL model; such components tend not to be difficult to validate. Other components, like multiprocessor cache-coherence protocols, are sufficiently complex that checking the HDL model against the problem statement is quite challenging. We need formal techniques from the world of mathematics to perform this high-level validation. Although formal methods are based on mathematics, engineers view them differently from the way mathematicians do. To engineers, formal verification is simply another imperfect validation tool (albeit a powerful one). A TLA+ specification is only an abstraction of the actual system, and model checking can usually validate the specification only for a highly restricted set of system parameters. Validating the specification therefore cannot guarantee that there are no errors in the system. For engineers, formal verification is a way of finding bugs, not of proving correctness. The main benefit of applying TLA+ to engineering problems comes from the efficiency of the TLC model checker in reaching high levels of coverage and finding bugs. A secondary benefit we have encountered is the ability of TLA+ and TLC to provide good metrics for the complexity of a design. Complexity is a major consideration in evaluating design tradeoffs. However, unlike performance or cost, engineers have not historically had a good way to quantify algorithmic complexity before attempting to validate a design. TLA+ encourages designers to specify the design abstractly, suppressing lower-level details, so the length of the specification provides a measure of a design's complexity. TLC reports the size of the reachable state space, providing another measure of complexity. Experience and intuition will always have a place in evaluating complexity, but TLA+ and TLC provide robust and impartial input to the evaluation. Having this input early in the design process is of considerable value.
4.2.2 Designing with TLA+The core group at Intel started using TLA+ at Compaq while working on the Alpha EV7 and EV8 multiprocessor projects described above. From that experience, the Alpha engineers learned that multiprocessor cache-coherence protocols are an ideal candidate for formal methods because most of the protocol bugs can be found at a high level of abstraction. They also learned that the true value of TLA+ and TLC would be realized when (a) they were applied early enough in the design to provide implementation feedback, and (b) the implementation was based directly on the specification that had been verified. On the EV8 project, the TLA+ specification was completed before the design was stable, and it provided important feedback to the designers.When the engineers from the Alpha group joined Intel, they began applying their experience in writing TLA+ specifications when collaborating with other Intel engineers on cache-coherence protocols for future Intel products. Intel engineers are now using TLA+ as an integral part of the design process for the protocols that are under study.
Whiteboard PhaseDesigning one cache-coherence protocol from scratch provided the engineers with the opportunity to evaluate TLA+ as a prototyping platform for complex algorithms. Work on this protocol started by exploring the design space on a whiteboard for about two months. In this phase, basic message sequencing was determined, as were some coarse notions of what state had to be recorded at the protocol endpoints. A basic direction was set, based on the guidelines for functionality, performance, and cost.
Because of their background, engineers tend to visualize an algorithm
in terms of a particular implementation. They are better at
gauging implementation complexity than at measuring algorithmic
complexity. One benefit of having engineers write formal
specifications is that it helps them learn how to think about a
protocol abstractly, independent of implementation details. We found
that, even in the whiteboard phase of the protocol design, the Intel
engineers were able to make some judgments on complexity by asking
themselves, The whiteboard phase produced a general understanding of the protocol philosophy, an understanding of the constraints placed on the communication medium, the basic message flows, and coarse ideas on what state needed to be maintained. The next step was to introduce the rigor of a formal specification.
TLA+ Scratchpad PhaseThe TLA+ scratchpad phase of the project involved formally describing the abstract system, with appropriate state variables representing high-level components. This phase took about two months, starting with the initial design of the protocol. The difficulty lay not in the use of TLA+ — engineers frequently learn new programming languages—but rather in (a) determining the layer of abstraction and (b) exploring the protocol's corner cases. Task (a) is where TLA+ forces engineers to think about the protocol abstractly, which they often find unnatural. Their ability to think abstractly improves with experience writing TLA+ specifications. Task (b) is inevitable when documenting a protocol formally, as it forces the designers to explore the corner cases. During the scratchpad phase, the designers had to return to the whiteboard a few times when they encountered new race cases while writing the specification.The actions that formed the major blocks of the specification were chosen early; very few changes were made later. The Intel engineers adopted a methodology used in the earlier Alpha specifications, in which the decomposition of high-level named actions is based on classifying the protocol messages that they process. This methodology has led to fairly readable specifications, since it means that each action changes only a few local state variables. It encouraged the protocol specifications to be designed in a modular way, which also enabled the inter-module interfaces in the specification to be similar to their low-level counterparts in the implementation.
Running TLCThe initial week or so of running TLC was spent finding and fixing typographical errors and type mismatch problems. This time could probably have been shortened by doing more syntax checking when writing the specification, which is what one often does when programming.The next four weeks saw a continuous process of running TLC, finding bugs, fixing them, and re-running TLC. During this phase of the project, many assumptions and assertions about the protocol were brought into question. This had the effect of educating the engineers about the protocol they had designed. We have found that TLC can be a useful learning tool if we use in-line assertions and global invariants to check everything we think is true. The Intel engineers were able to develop an intuitive understanding of the correctness of the protocol by developing meaningful global invariants and having TLC check them. If an assertion or invariant fails, TLC generates a counterexample that is useful for visualizing a difficult race case. These counterexamples are such a powerful teaching aid that the Intel engineers have developed tools to translate the TLC output into nicely formatted protocol flow diagrams that are easier to read.
Another useful feature of the TLC model checker is its coverage
checking. TLC can print the number of times each action was
The performance of the model checker was sufficient to debug a large
protocol specification. The engineers determined a base configuration
that would
4.2.3 Optimizing with TLCOnce the initial protocol specification was successfully checked by TLC, the Intel engineers were able to use it as a test bed for exploring optimizations. TLA+ is an ideal language to explore changes because its expressiveness usually allows the new version to be written quickly. Model checking the modified specification with TLC not only checks functional correctness, but it also measures any increase in the state space. Such an increase implies additional algorithmic complexity. The engineers spent several months exploring additions to the protocol, testing them with TLC. As a general rule, they would consider adopting only those optimizations that did not appreciably expand the state space. The insight that TLA+ and TLC gave into the complexity of modifications to the protocol was invaluable in iterating towards an optimal solution that adequately weighed algorithmic complexity along with factors like cost and performance.A significant optimization was later made to the protocol. This optimization followed the normal design cycle described above, though on a compressed schedule. With the original design yielding a good starting point, the entire cycle (whiteboard phase, TLA+ coding, and verification with TLC) was done within six weeks. This modification was accomplished by a recent college graduate with an undergraduate degree in engineering. He was able to learn TLA+ well enough within a matter of weeks to do this work.
4.2.4 Feedback on TLA+ syntaxThe feedback we have received from engineers about the TLA+ language has been mostly positive. Engineers are usually able to pick up and understand a specification within a couple of days. One mistake we made was to present TLA+ to hardware designers as similar to a programming language. This led to some frustration. A better approach seems to be to describe TLA+ as being like a hardware description language. Engineers who design digital systems are well acquainted with methods for specifying finite-state machines, with the associated restrictions of allowing a primed variable to be assigned a value only once within a conjunction, not allowing a primed variable to appear in a conjunction before the assignment of its value, etc. To an engineer, TLA+ looks like a language for specifying finite-state machines.
While writing the protocol specification at Intel, BB was impressed by
the ease of specifying complex data structures in TLA+ as sets and
tuples. The part of the specification that described and manipulated
data structures was a small part of the complete protocol
specification. This compact specification of For the algorithmic specification, TLA+ naturally encourages nested disjunctions of conjunctions (known to engineers as sums of products of expressions). This method for specifying Boolean formulas has both advantages and disadvantages. One advantage is that it allows expressive comment blocks and assertions to be inserted in-line with a nested conjunct. A disadvantage is that this tends to lead to large specifications. The engineers are experimenting with the use of TLA+ operators to encode large blocks of regular Boolean disjunctions as truth tables, which engineers find more natural to work with.
|
||


>
This is part of the old website. New website begins at fontforge.github.io
Are you a web developer? Help us migrate this page on Github
FontForge supports a plugin mechanism for extending its functionality.
Plugins are shared libraries that can be loaded at runtime. When fontforge starts up (actually, when it loads its preferences) it will automatically load all files that look like shared libraries and that live in the two directories:
/usr/local/share/fontforge/plugins
$(PREFIX)/share/fontforge/plugins
)
~/.FontForge/plugins
Most of what I once expected to do with plugins I now do with python scripts.
Currently none of my pre-built packages contain any plugins. If you want to play with them you must grab a source distribution, build it, and install it. Having done that type the following:
$ cd plugins $ make install
FontForge currently supports two types of plugins:
(Actually I suppose there is a third type. When it loads a plugin FF calls an initialization routine. I suppose the plugin could just start running and doing things without ever returning to fontforge).
I once expected to add
(but I did that in python instead).
When FontForge loads a plugin it calls a routine int
FontForgeInit(void)
. This routine is responsible for registering the
plugin with FontForge, and then returning to allow FontForge to continue.
FontForge provides two registration routines:
AddEncoding(char *name,int (*enc_to_uni)(int), int
(*uni_to_enc)(int))
This will return 0 if it failed (if you tried to replace a built in encoding for example), 1 if it added the encoding, 2 if it replaced a previous plugin with that name.
AddScriptingCommand(char *name, void (*UserScriptFunc)(Context *),
int needs_font)
This will return 0 if it failed (if you tried to replace a built in scripting command for example), 1 if it added the command, 2 if it replaced a previous plugin with that name.
You may register more than one thing in from FontForgeInit
.
For example, many encodings have an EUC form and an ISO-2022 form and it
makes sense for one plugin to handle both.
FontForgeInit
should return 1 if it succeeds, and 0 if it fails.
If it fails FontForge will dlclose() the library.
When you write your plugin you should include "plugins.h" from the fontforge directory.
Look in the plugins subdirectory for an example on how to create, build & install a plugin.


What We Are Here To Promote
We make strong but breathable clothing
Our Pattented formular is designed so that when you are out and riding you are able to alow air in the top to keep your body fresh. however in the cold winter months it also keeps you warm by traping small areas of your body heat to stop any unwanted heat from esaping all you need to do is do the zip up.
You can use our clothes in any sport.
The product is useable in many sports from cold weather sports too warm weather sports. Like we said above the type of material we use is breathable, the zip opens and closes to allow the person using the product to control the heat inside the top. When in the colder climates the top should be ziped up to allow the product to use its addaptive pockets to keep main areas of the body warm, and in warmer temperatures use the zips to control how much of the heat you want to disperce into the surronding enviroment.


Hi, I'm Yibo (Jacky) Zhang
photo taken in Vancouver, Canada
I am a PhD student in the Department of Computer Science at Stanford University, where I am fortunate to be advised by
Sanmi Koyejo.
I am broadly interested in
- Machine Learning: broadly in understanding and designing machine learning algorithms, especially from a
fundamental perspective.
- Optimization: non-convex optimization, combinatorial optimization, stochastic optimization, submodular optimization.
- Algorithms: approximation algorithms, heuristic algorithms.
Due to uncovered reasons, Yibo Zhang (张益博) used an academic pseudonyms: Jacky Y. Zhang.


Home page
There is not controller in this page
When "debug" is activate the pages without controller will show an error.


All of the code described in these documents are available via using the Concurrent Versions System (CVS) software. While somewhat more complicated than simply having a tar file available from a website, this allows users to immediately update their code to reflect code updates and bug fixes. As of February 2004 the code is available to both the public and to members of the LCA: However, the public version is missing some features that are available in the LCA version, such as star formation and feedback and some cooling models.
Before installing enzo, you must ensure that CVS is installed on the target system. This is most likely true for any unix or unix-like system. Just type 'cvs' at a command prompt and if a help screen doesn't appear, contact your system administrator. To use CVS, first set the CVSROOT variable:
setenv CVSROOT :pserver:login@jbpc.ucsd.edu:/usr/local/cvs_lca
Where login is your user login. Contact James Bordner for access to the CVS repository. You may also have to set CVS_RSH to use ssh instead of rsh:
setenv CVS_RSH ssh
These setenv commands assume that you use c shell or tc shell. If you use bash or ksh, you're on your own. After setting the environmental variables, log in to the server:
cvs login
[enter password at prompt]
Then check out the code:
cvs checkout enzo-code
This will then download the entire enzo bundle to a directory called 'enzo-code'. You must have about 10 megabytes of disk space available for the source, and an additional 40 megabytes or so for the code to have space to compile.
You only have to do this checkout once to create a local copy of the repository. After the initial code checkout you merely have to go into the enzo-code directory and type:
cvs update -d
to get the latest version of the code. It will merge changes/bugfixes made by the Enzo code development group with your local copy.
To see what modifications have been made you can use:
cvs history
This command can also be used to remove these changes in order to recover the code to a previous state.


Presentation Bridge Configuration
Add New Bridge
ID:
Name:
Control Password:
Observe Password:
Enabled
Allow Text Listeners
Allow Image Listeners
Allow Stage Display
Logo:
Foreground Color (Text):
Background Color:
Font:
Helvetica
Monospace
Add
Cancel
Update
Cancel
Delete
Change Config Password:
Old Password:
New Password:
Change Config Password


Request a ride and you’ll be on your way in minutes. Request. Ride. Repeat.
From knowing the color of your driver’s car to our 24/7 Trust & Safety Team, we got you.
Ride with us and you’ll see why 9 out of 10 rides end with five stars.
Lyft's new emblem, Amp, is the secret to smooth pickups. Lighting up dashboards nationwide, the device makes it easy for passengers and drivers to find each other.
In an animated short film by Academy Award-winner John Kahrs, a lonely widow in historic South Chicago is empowered to start sharing the ride-and sharing her life, too.
Launching on TV and online. "Ride on the Bright Side" shows how Lyft continues to prioritize happy drivers, short ETAs, and safety.


Domain object DACL privilege escalation
A privilege escalation is possible from the Exchange Windows permissions (EWP) security group to compromise the entire prepared Active Directory domain.
DISCLAIMER: This issue has been responsibly disclosed to MSRC in October 2017 and after a few back and forth emails, they decided it was working as intended and would not fix it.
When preparing Exchange 2010/2013/2016 installation in Shared permissions (default) or RBAC split permissions, some ACEs are positioned on the domain object for the Exchange Windows Permissions security group. This happens during the "Setup /PrepareAD" command.
Two ACEs on the domain object are missing the INHERIT_ONLY_ACE bit in the Flags field.
This is their SDDL representation:
(OA;CI;DTWD;;4828cc14-1437-45bc-9b07-ad6f015e5f28;<SID of EWP>) (OA;CI;DTWD;;bf967aba-0de6-11d0-a285-00aa003049e2;<SID of EWP>)
Notice the missing 'IO' (inherit_only) flag. As a matter of fact, INHERITED_OBJECT_TYPE ACE effectively apply to the object itself when the inherit_only flag is not set.
This is how they are documented on Technet, so it is definitely missing that information:
Account ACE type Inheritance Permissions On property/ Applies to Comments
Exchange Windows Permissions Allow ACE All WriteDACL / user
Exchange Windows Permissions Allow ACE All WriteDACL / inetOrgPerson
The issue can be confirmed in the "Effective Access" tab of the domain object when specifying EWP.
Coincidentally, INHERITED_OBJECT_TYPE ACE created with DSA console always have the IO flag. Creating a 'self and specific descendents', INHERITED_OBJECT_TYPE ACE is only possible programmatically and only viewable programmatically.
Technical consequence
The Allow permission to WriteDACL is granted to the Trustee on the domain object itself and not only on its user/inetOrgPerson descendants, effectively giving complete control to anyone with the Exchange Windows Permissions SID in its token.
Expected behavior
The Allow permission to WriteDACL should only be given on child objects matching the INHERITED_OBJECT_TYPE Guid. It is supposed to only apply to user or inetOrgPerson classes. This is not consistent with the other INHERITED_OBJECT_TYPE ACEs on the domain object for the same Trustee, which all have the INHERIT_ONLY_ACE bit in the Flags field.
Security Consequence
A privilege escalation is possible from the Exchange Windows permissions (EWP) security group to compromise the entire prepared Active Directory domain. Just set the Ds-Replication-Get-Changes-All extended right on the domain object. This is enough to use a DRSUAPI replication tool ala DCSync to get all the domain secrets (Kerberos keys, hashes, etc.).
You can control the SID of the EWP group from Organization management, from Account operators or from any Exchange server. Interestingly, the RBAC system is proxified through EWP so a privilege escalation is possible from the Active Directory Permissions RBAC role to compromise the entire prepared Active Directory domain.
Proof of concept 1: Set-Acl (or with DSA console)
From an Organization Management member account, add yourself to Exchange Windows Permissions. This is possible by default (see the above technet page).
$id = [Security.Principal.WindowsIdentity]::GetCurrent()
$user = Get-ADUser -Identity $id.User
Add-ADGroupMember -Identity "Exchange Windows Permissions" -Members $user
RELOG to update groups in your token then give yourself Ds-Replication-Get-Changes-All extended right on the domain object.
$acl = get-acl "ad:DC=test,DC=local"
$id = [Security.Principal.WindowsIdentity]::GetCurrent()
$user = Get-ADUser -Identity $id.User
$sid = new-object System.Security.Principal.SecurityIdentifier $user.SID
# rightsGuid for the extended right Ds-Replication-Get-Changes-All
$objectguid = new-object Guid 1131f6ad-9c07-11d1-f79f-00c04fc2dcd2 $identity = [System.Security.Principal.IdentityReference] $sid
$adRights = [System.DirectoryServices.ActiveDirectoryRights] "ExtendedRight" $type = [System.Security.AccessControl.AccessControlType] "Allow"
$inheritanceType = [System.DirectoryServices.ActiveDirectorySecurityInheritance] "None"
$ace = new-object System.DirectoryServices.ActiveDirectoryAccessRule $identity,$adRights,$type,$objectGuid,$inheritanceType
$acl.AddAccessRule($ace) Set-acl -aclobject $acl "ad:DC=test,DC=com"
Proof of concept 2: RBAC Add-ADPermission
The following Powershell code, executed by a user account with the RBAC role Active Directory Permissions, sets the Ds-Replication-Get-Changes-All extended right on the domain object.
$id = [Security.Principal.WindowsIdentity]::GetCurrent() Add-ADPermission "DC=test,DC=local" -User $id.Name -ExtendedRights Ds-Replication-Get-Changes-All
Workaround fix
Use the Fix-DomainObjectDACL.ps1 Powershell script in this repository. Read it, test it, use it at your own risk.
By default, it checks the two faulty ACE, this can be done by any user. Use with -Fix switch with Domain Admins privileges to set the InheritOnly missing flags. Use with -Restore switch to restore the two previously fixed ACEs to their original, vulnerable state. An SDDL backup is made everytime the script is run.
Another option is doing it manually with LDP. Bind as a Domain Admin account with the usual precautions, as you will change the domain object DACL. Locate the 2 faulty ACE in the DACL of the domain object: you can sort by Trustee to see Exchange Windows Permissions, there will only be 2 ACE with "Write DACL,..." Rights. Check the 'Inherit only' box in their ACE flags field.


Ocean [as53], ocean, in The Inner Sea
Last updated Turn #48
Ocean [as53], ocean, in The Inner Sea
------------------------------------------------------------------------
15: Travelling Wave [hp7g], ship, 2 hulls, 4 rowing ports, 4% damaged,
15: arrived from Ocean [as54], owner:
15: Stewart [k0u], Pen, with 18 sailors
15: Jonathan [h9k], Pen
16: Travelling Wave [hp7g], ship, 2 hulls, 4 rowing ports, 4% damaged,
16: departed for Ocean [as52].
Routes leaving Ocean:
North, to Ocean [ar53], 6 days
East, to Ocean [as54], 6 days
South, to Ocean [at53], 6 days
West, to Ocean [as52], 6 days


LAYING A GHOST.
TO the ignorance of men in our age in this particular and mysterious part of philosophy and religion,--namely, the communication between spirits and men,--not one scholar out of ten thousand, though otherwise of excellent learning, knows anything of it, or the way how to manage it. This ignorance breeds fear and abhorrence of that which otherwise might be of incomparable benefit to mankind."
Such is the concluding paragraph of "An Account of an Apparition, attested by the Rev. Wm. Ruddell, Minister at Launceston, in Cornwall," 1665.
A schoolboy was haunted by Dorothy Dingley; we know not why, but the boy pined. He was thought to be in love; but when, at the wishes of his friends, the parson questioned him, he told him of his ghostly visitor, and he took the parson to the field in which he was in the habit of meeting the apparition; and the reverend gentleman himself saw the spectral Dorothy, and afterwards he showed her to the boy's father and mother. Then comes the story of the laying. "The next morning being Thursday, I went out very early by myself, and walked for about an hour's space in meditation and prayer in the field next adjoining to the Quartiles. Soon after five, I stepped over the stile into the disturbed field, and had not gone above thirty or forty paces before the ghost appeared at the further stile. I spoke to it with a loud voice, in some such sentences as the way of these dealings directed me; whereupon it approached, but slowly, and when I came near it it moved not. I spoke again, and it answered again in a voice which was neither very audible nor intelligible. I was not the least terrified, therefore I persisted until it spake again and gave me satisfaction. But the work could not be finished at this time; wherefore the same evening, an hour after sunset, it met me again, near the same place, and after a few words on each side it quietly vanished, and neither doth appear since, nor ever will more to any man's disturbance." [a]
[a] "Historical Survey of Cornwall," C. S. Gilbert.


List 1 Clear
List 2 Clear
List 3 Clear
List 4 Clear
List 5 Clear
List 6 Clear
List 7 Clear
Upload File
Export Image


Clustering - RDD-based API
Clustering is an unsupervised learning problem whereby we aim to group subsets of entities with one another based on some notion of similarity. Clustering is often used for exploratory analysis and/or as a component of a hierarchical supervised learning pipeline (in which distinct classifiers or regression models are trained for each cluster).
The spark.mllib
package supports the following models:
- K-means
- Gaussian mixture
- Power iteration clustering (PIC)
- Latent Dirichlet allocation (LDA)
- Bisecting k-means
- Streaming k-means
K-means
K-means is one of the
most commonly used clustering algorithms that clusters the data points into a
predefined number of clusters. The spark.mllib
implementation includes a parallelized
variant of the k-means++ method
called kmeans||.
The implementation in spark.mllib
has the following parameters:
- k is the number of desired clusters. Note that it is possible for fewer than k clusters to be returned, for example, if there are fewer than k distinct points to cluster.
- maxIterations is the maximum number of iterations to run.
- initializationMode specifies either random initialization or initialization via k-means||.
- runs This param has no effect since Spark 2.0.0.
- initializationSteps determines the number of steps in the k-means|| algorithm.
- epsilon determines the distance threshold within which we consider k-means to have converged.
- initialModel is an optional set of cluster centers used for initialization. If this parameter is supplied, only one run is performed.
Examples
The following code snippets can be executed in spark-shell
.
In the following example after loading and parsing data, we use the
KMeans
object to cluster the data
into two clusters. The number of desired clusters is passed to the algorithm. We then compute Within
Set Sum of Squared Error (WSSSE). You can reduce this error measure by increasing k. In fact the
optimal k is usually one where there is an “elbow” in the WSSSE graph.
Refer to the KMeans
Scala docs and KMeansModel
Scala docs for details on the API.
import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}
import org.apache.spark.mllib.linalg.Vectors
// Load and parse the data
val data = sc.textFile("data/mllib/kmeans_data.txt")
val parsedData = data.map(s => Vectors.dense(s.split(' ').map(_.toDouble))).cache()
// Cluster the data into two classes using KMeans
val numClusters = 2
val numIterations = 20
val clusters = KMeans.train(parsedData, numClusters, numIterations)
// Evaluate clustering by computing Within Set Sum of Squared Errors
val WSSSE = clusters.computeCost(parsedData)
println(s"Within Set Sum of Squared Errors = $WSSSE")
// Save and load model
clusters.save(sc, "target/org/apache/spark/KMeansExample/KMeansModel")
val sameModel = KMeansModel.load(sc, "target/org/apache/spark/KMeansExample/KMeansModel")
All of MLlib’s methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate JavaRDD
class. You can convert a Java RDD to a Scala one by
calling .rdd()
on your JavaRDD
object. A self-contained application example
that is equivalent to the provided example in Scala is given below:
Refer to the KMeans
Java docs and KMeansModel
Java docs for details on the API.
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.mllib.clustering.KMeans;
import org.apache.spark.mllib.clustering.KMeansModel;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
// Load and parse data
String path = "data/mllib/kmeans_data.txt";
JavaRDD<String> data = jsc.textFile(path);
JavaRDD<Vector> parsedData = data.map(s -> {
String[] sarray = s.split(" ");
double[] values = new double[sarray.length];
for (int i = 0; i < sarray.length; i++) {
values[i] = Double.parseDouble(sarray[i]);
}
return Vectors.dense(values);
});
parsedData.cache();
// Cluster the data into two classes using KMeans
int numClusters = 2;
int numIterations = 20;
KMeansModel clusters = KMeans.train(parsedData.rdd(), numClusters, numIterations);
System.out.println("Cluster centers:");
for (Vector center: clusters.clusterCenters()) {
System.out.println(" " + center);
}
double cost = clusters.computeCost(parsedData.rdd());
System.out.println("Cost: " + cost);
// Evaluate clustering by computing Within Set Sum of Squared Errors
double WSSSE = clusters.computeCost(parsedData.rdd());
System.out.println("Within Set Sum of Squared Errors = " + WSSSE);
// Save and load model
clusters.save(jsc.sc(), "target/org/apache/spark/JavaKMeansExample/KMeansModel");
KMeansModel sameModel = KMeansModel.load(jsc.sc(),
"target/org/apache/spark/JavaKMeansExample/KMeansModel");
The following examples can be tested in the PySpark shell.
In the following example after loading and parsing data, we use the KMeans object to cluster the data into two clusters. The number of desired clusters is passed to the algorithm. We then compute Within Set Sum of Squared Error (WSSSE). You can reduce this error measure by increasing k. In fact the optimal k is usually one where there is an “elbow” in the WSSSE graph.
Refer to the KMeans
Python docs and KMeansModel
Python docs for more details on the API.
from numpy import array
from math import sqrt
from pyspark.mllib.clustering import KMeans, KMeansModel
# Load and parse the data
data = sc.textFile("data/mllib/kmeans_data.txt")
parsedData = data.map(lambda line: array([float(x) for x in line.split(' ')]))
# Build the model (cluster the data)
clusters = KMeans.train(parsedData, 2, maxIterations=10, initializationMode="random")
# Evaluate clustering by computing Within Set Sum of Squared Errors
def error(point):
center = clusters.centers[clusters.predict(point)]
return sqrt(sum([x**2 for x in (point - center)]))
WSSSE = parsedData.map(lambda point: error(point)).reduce(lambda x, y: x + y)
print("Within Set Sum of Squared Error = " + str(WSSSE))
# Save and load model
clusters.save(sc, "target/org/apache/spark/PythonKMeansExample/KMeansModel")
sameModel = KMeansModel.load(sc, "target/org/apache/spark/PythonKMeansExample/KMeansModel")
Gaussian mixture
A Gaussian Mixture Model
represents a composite distribution whereby points are drawn from one of k Gaussian sub-distributions,
each with its own probability. The spark.mllib
implementation uses the
expectation-maximization
algorithm to induce the maximum-likelihood model given a set of samples. The implementation
has the following parameters:
- k is the number of desired clusters.
- convergenceTol is the maximum change in log-likelihood at which we consider convergence achieved.
- maxIterations is the maximum number of iterations to perform without reaching convergence.
- initialModel is an optional starting point from which to start the EM algorithm. If this parameter is omitted, a random starting point will be constructed from the data.
Examples
In the following example after loading and parsing data, we use a GaussianMixture object to cluster the data into two clusters. The number of desired clusters is passed to the algorithm. We then output the parameters of the mixture model.
Refer to the GaussianMixture
Scala docs and GaussianMixtureModel
Scala docs for details on the API.
import org.apache.spark.mllib.clustering.{GaussianMixture, GaussianMixtureModel}
import org.apache.spark.mllib.linalg.Vectors
// Load and parse the data
val data = sc.textFile("data/mllib/gmm_data.txt")
val parsedData = data.map(s => Vectors.dense(s.trim.split(' ').map(_.toDouble))).cache()
// Cluster the data into two classes using GaussianMixture
val gmm = new GaussianMixture().setK(2).run(parsedData)
// Save and load model
gmm.save(sc, "target/org/apache/spark/GaussianMixtureExample/GaussianMixtureModel")
val sameModel = GaussianMixtureModel.load(sc,
"target/org/apache/spark/GaussianMixtureExample/GaussianMixtureModel")
// output parameters of max-likelihood model
for (i <- 0 until gmm.k) {
println("weight=%f\nmu=%s\nsigma=\n%s\n" format
(gmm.weights(i), gmm.gaussians(i).mu, gmm.gaussians(i).sigma))
}
All of MLlib’s methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate JavaRDD
class. You can convert a Java RDD to a Scala one by
calling .rdd()
on your JavaRDD
object. A self-contained application example
that is equivalent to the provided example in Scala is given below:
Refer to the GaussianMixture
Java docs and GaussianMixtureModel
Java docs for details on the API.
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.mllib.clustering.GaussianMixture;
import org.apache.spark.mllib.clustering.GaussianMixtureModel;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
// Load and parse data
String path = "data/mllib/gmm_data.txt";
JavaRDD<String> data = jsc.textFile(path);
JavaRDD<Vector> parsedData = data.map(s -> {
String[] sarray = s.trim().split(" ");
double[] values = new double[sarray.length];
for (int i = 0; i < sarray.length; i++) {
values[i] = Double.parseDouble(sarray[i]);
}
return Vectors.dense(values);
});
parsedData.cache();
// Cluster the data into two classes using GaussianMixture
GaussianMixtureModel gmm = new GaussianMixture().setK(2).run(parsedData.rdd());
// Save and load GaussianMixtureModel
gmm.save(jsc.sc(), "target/org/apache/spark/JavaGaussianMixtureExample/GaussianMixtureModel");
GaussianMixtureModel sameModel = GaussianMixtureModel.load(jsc.sc(),
"target/org.apache.spark.JavaGaussianMixtureExample/GaussianMixtureModel");
// Output the parameters of the mixture model
for (int j = 0; j < gmm.k(); j++) {
System.out.printf("weight=%f\nmu=%s\nsigma=\n%s\n",
gmm.weights()[j], gmm.gaussians()[j].mu(), gmm.gaussians()[j].sigma());
}
In the following example after loading and parsing data, we use a GaussianMixture object to cluster the data into two clusters. The number of desired clusters is passed to the algorithm. We then output the parameters of the mixture model.
Refer to the GaussianMixture
Python docs and GaussianMixtureModel
Python docs for more details on the API.
from numpy import array
from pyspark.mllib.clustering import GaussianMixture, GaussianMixtureModel
# Load and parse the data
data = sc.textFile("data/mllib/gmm_data.txt")
parsedData = data.map(lambda line: array([float(x) for x in line.strip().split(' ')]))
# Build the model (cluster the data)
gmm = GaussianMixture.train(parsedData, 2)
# Save and load model
gmm.save(sc, "target/org/apache/spark/PythonGaussianMixtureExample/GaussianMixtureModel")
sameModel = GaussianMixtureModel\
.load(sc, "target/org/apache/spark/PythonGaussianMixtureExample/GaussianMixtureModel")
# output parameters of model
for i in range(2):
print("weight = ", gmm.weights[i], "mu = ", gmm.gaussians[i].mu,
"sigma = ", gmm.gaussians[i].sigma.toArray())
Power iteration clustering (PIC)
Power iteration clustering (PIC) is a scalable and efficient algorithm for clustering vertices of a
graph given pairwise similarities as edge properties,
described in Lin and Cohen, Power Iteration Clustering.
It computes a pseudo-eigenvector of the normalized affinity matrix of the graph via
power iteration and uses it to cluster vertices.
spark.mllib
includes an implementation of PIC using GraphX as its backend.
It takes an RDD
of (srcId, dstId, similarity)
tuples and outputs a model with the clustering assignments.
The similarities must be nonnegative.
PIC assumes that the similarity measure is symmetric.
A pair (srcId, dstId)
regardless of the ordering should appear at most once in the input data.
If a pair is missing from input, their similarity is treated as zero.
spark.mllib
’s PIC implementation takes the following (hyper-)parameters:
k
: number of clustersmaxIterations
: maximum number of power iterationsinitializationMode
: initialization model. This can be either “random”, which is the default, to use a random vector as vertex properties, or “degree” to use normalized sum similarities.
Examples
In the following, we show code snippets to demonstrate how to use PIC in spark.mllib
.
PowerIterationClustering
implements the PIC algorithm.
It takes an RDD
of (srcId: Long, dstId: Long, similarity: Double)
tuples representing the
affinity matrix.
Calling PowerIterationClustering.run
returns a
PowerIterationClusteringModel
,
which contains the computed clustering assignments.
Refer to the PowerIterationClustering
Scala docs and PowerIterationClusteringModel
Scala docs for details on the API.
import org.apache.spark.mllib.clustering.PowerIterationClustering
val circlesRdd = generateCirclesRdd(sc, params.k, params.numPoints)
val model = new PowerIterationClustering()
.setK(params.k)
.setMaxIterations(params.maxIterations)
.setInitializationMode("degree")
.run(circlesRdd)
val clusters = model.assignments.collect().groupBy(_.cluster).mapValues(_.map(_.id))
val assignments = clusters.toList.sortBy { case (k, v) => v.length }
val assignmentsStr = assignments
.map { case (k, v) =>
s"$k -> ${v.sorted.mkString("[", ",", "]")}"
}.mkString(", ")
val sizesStr = assignments.map {
_._2.length
}.sorted.mkString("(", ",", ")")
println(s"Cluster assignments: $assignmentsStr\ncluster sizes: $sizesStr")
PowerIterationClustering
implements the PIC algorithm.
It takes an JavaRDD
of (srcId: Long, dstId: Long, similarity: Double)
tuples representing the
affinity matrix.
Calling PowerIterationClustering.run
returns a
PowerIterationClusteringModel
which contains the computed clustering assignments.
Refer to the PowerIterationClustering
Java docs and PowerIterationClusteringModel
Java docs for details on the API.
import org.apache.spark.mllib.clustering.PowerIterationClustering;
import org.apache.spark.mllib.clustering.PowerIterationClusteringModel;
JavaRDD<Tuple3<Long, Long, Double>> similarities = sc.parallelize(Arrays.asList(
new Tuple3<>(0L, 1L, 0.9),
new Tuple3<>(1L, 2L, 0.9),
new Tuple3<>(2L, 3L, 0.9),
new Tuple3<>(3L, 4L, 0.1),
new Tuple3<>(4L, 5L, 0.9)));
PowerIterationClustering pic = new PowerIterationClustering()
.setK(2)
.setMaxIterations(10);
PowerIterationClusteringModel model = pic.run(similarities);
for (PowerIterationClustering.Assignment a: model.assignments().toJavaRDD().collect()) {
System.out.println(a.id() + " -> " + a.cluster());
}
PowerIterationClustering
implements the PIC algorithm.
It takes an RDD
of (srcId: Long, dstId: Long, similarity: Double)
tuples representing the
affinity matrix.
Calling PowerIterationClustering.run
returns a
PowerIterationClusteringModel
,
which contains the computed clustering assignments.
Refer to the PowerIterationClustering
Python docs and PowerIterationClusteringModel
Python docs for more details on the API.
from pyspark.mllib.clustering import PowerIterationClustering, PowerIterationClusteringModel
# Load and parse the data
data = sc.textFile("data/mllib/pic_data.txt")
similarities = data.map(lambda line: tuple([float(x) for x in line.split(' ')]))
# Cluster the data into two classes using PowerIterationClustering
model = PowerIterationClustering.train(similarities, 2, 10)
model.assignments().foreach(lambda x: print(str(x.id) + " -> " + str(x.cluster)))
# Save and load model
model.save(sc, "target/org/apache/spark/PythonPowerIterationClusteringExample/PICModel")
sameModel = PowerIterationClusteringModel\
.load(sc, "target/org/apache/spark/PythonPowerIterationClusteringExample/PICModel")
Latent Dirichlet allocation (LDA)
Latent Dirichlet allocation (LDA) is a topic model which infers topics from a collection of text documents. LDA can be thought of as a clustering algorithm as follows:
- Topics correspond to cluster centers, and documents correspond to examples (rows) in a dataset.
- Topics and documents both exist in a feature space, where feature vectors are vectors of word counts (bag of words).
- Rather than estimating a clustering using a traditional distance, LDA uses a function based on a statistical model of how text documents are generated.
LDA supports different inference algorithms via setOptimizer
function.
EMLDAOptimizer
learns clustering using
expectation-maximization
on the likelihood function and yields comprehensive results, while
OnlineLDAOptimizer
uses iterative mini-batch sampling for online
variational
inference
and is generally memory friendly.
LDA takes in a collection of documents as vectors of word counts and the following parameters (set using the builder pattern):
k
: Number of topics (i.e., cluster centers)optimizer
: Optimizer to use for learning the LDA model, eitherEMLDAOptimizer
orOnlineLDAOptimizer
docConcentration
: Dirichlet parameter for prior over documents’ distributions over topics. Larger values encourage smoother inferred distributions.topicConcentration
: Dirichlet parameter for prior over topics’ distributions over terms (words). Larger values encourage smoother inferred distributions.maxIterations
: Limit on the number of iterations.checkpointInterval
: If using checkpointing (set in the Spark configuration), this parameter specifies the frequency with which checkpoints will be created. IfmaxIterations
is large, using checkpointing can help reduce shuffle file sizes on disk and help with failure recovery.
All of spark.mllib
’s LDA models support:
describeTopics
: Returns topics as arrays of most important terms and term weightstopicsMatrix
: Returns avocabSize
byk
matrix where each column is a topic
Note: LDA is still an experimental feature under active development. As a result, certain features are only available in one of the two optimizers / models generated by the optimizer. Currently, a distributed model can be converted into a local model, but not vice-versa.
The following discussion will describe each optimizer/model pair separately.
Expectation Maximization
Implemented in
EMLDAOptimizer
and
DistributedLDAModel
.
For the parameters provided to LDA
:
docConcentration
: Only symmetric priors are supported, so all values in the providedk
-dimensional vector must be identical. All values must also be $> 1.0$. ProvidingVector(-1)
results in default behavior (uniformk
dimensional vector with value $(50 / k) + 1$topicConcentration
: Only symmetric priors supported. Values must be $> 1.0$. Providing-1
results in defaulting to a value of $0.1 + 1$.maxIterations
: The maximum number of EM iterations.
Note: It is important to do enough iterations. In early iterations, EM often has useless topics, but those topics improve dramatically after more iterations. Using at least 20 and possibly 50-100 iterations is often reasonable, depending on your dataset.
EMLDAOptimizer
produces a DistributedLDAModel
, which stores not only
the inferred topics but also the full training corpus and topic
distributions for each document in the training corpus. A
DistributedLDAModel
supports:
topTopicsPerDocument
: The top topics and their weights for each document in the training corpustopDocumentsPerTopic
: The top documents for each topic and the corresponding weight of the topic in the documents.logPrior
: log probability of the estimated topics and document-topic distributions given the hyperparametersdocConcentration
andtopicConcentration
logLikelihood
: log likelihood of the training corpus, given the inferred topics and document-topic distributions
Online Variational Bayes
Implemented in
OnlineLDAOptimizer
and
LocalLDAModel
.
For the parameters provided to LDA
:
docConcentration
: Asymmetric priors can be used by passing in a vector with values equal to the Dirichlet parameter in each of thek
dimensions. Values should be $>= 0$. ProvidingVector(-1)
results in default behavior (uniformk
dimensional vector with value $(1.0 / k)$)topicConcentration
: Only symmetric priors supported. Values must be $>= 0$. Providing-1
results in defaulting to a value of $(1.0 / k)$.maxIterations
: Maximum number of minibatches to submit.
In addition, OnlineLDAOptimizer
accepts the following parameters:
miniBatchFraction
: Fraction of corpus sampled and used at each iterationoptimizeDocConcentration
: If set to true, performs maximum-likelihood estimation of the hyperparameterdocConcentration
(akaalpha
) after each minibatch and sets the optimizeddocConcentration
in the returnedLocalLDAModel
tau0
andkappa
: Used for learning-rate decay, which is computed by $(\tau_0 + iter)^{-\kappa}$ where $iter$ is the current number of iterations.
OnlineLDAOptimizer
produces a LocalLDAModel
, which only stores the
inferred topics. A LocalLDAModel
supports:
logLikelihood(documents)
: Calculates a lower bound on the provideddocuments
given the inferred topics.logPerplexity(documents)
: Calculates an upper bound on the perplexity of the provideddocuments
given the inferred topics.
Examples
In the following example, we load word count vectors representing a corpus of documents. We then use LDA to infer three topics from the documents. The number of desired clusters is passed to the algorithm. We then output the topics, represented as probability distributions over words.
Refer to the LDA
Scala docs and DistributedLDAModel
Scala docs for details on the API.
import org.apache.spark.mllib.clustering.{DistributedLDAModel, LDA}
import org.apache.spark.mllib.linalg.Vectors
// Load and parse the data
val data = sc.textFile("data/mllib/sample_lda_data.txt")
val parsedData = data.map(s => Vectors.dense(s.trim.split(' ').map(_.toDouble)))
// Index documents with unique IDs
val corpus = parsedData.zipWithIndex.map(_.swap).cache()
// Cluster the documents into three topics using LDA
val ldaModel = new LDA().setK(3).run(corpus)
// Output topics. Each is a distribution over words (matching word count vectors)
println(s"Learned topics (as distributions over vocab of ${ldaModel.vocabSize} words):")
val topics = ldaModel.topicsMatrix
for (topic <- Range(0, 3)) {
print(s"Topic $topic :")
for (word <- Range(0, ldaModel.vocabSize)) {
print(s"${topics(word, topic)}")
}
println()
}
// Save and load model.
ldaModel.save(sc, "target/org/apache/spark/LatentDirichletAllocationExample/LDAModel")
val sameModel = DistributedLDAModel.load(sc,
"target/org/apache/spark/LatentDirichletAllocationExample/LDAModel")
Refer to the LDA
Java docs and DistributedLDAModel
Java docs for details on the API.
import scala.Tuple2;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.mllib.clustering.DistributedLDAModel;
import org.apache.spark.mllib.clustering.LDA;
import org.apache.spark.mllib.clustering.LDAModel;
import org.apache.spark.mllib.linalg.Matrix;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
// Load and parse the data
String path = "data/mllib/sample_lda_data.txt";
JavaRDD<String> data = jsc.textFile(path);
JavaRDD<Vector> parsedData = data.map(s -> {
String[] sarray = s.trim().split(" ");
double[] values = new double[sarray.length];
for (int i = 0; i < sarray.length; i++) {
values[i] = Double.parseDouble(sarray[i]);
}
return Vectors.dense(values);
});
// Index documents with unique IDs
JavaPairRDD<Long, Vector> corpus =
JavaPairRDD.fromJavaRDD(parsedData.zipWithIndex().map(Tuple2::swap));
corpus.cache();
// Cluster the documents into three topics using LDA
LDAModel ldaModel = new LDA().setK(3).run(corpus);
// Output topics. Each is a distribution over words (matching word count vectors)
System.out.println("Learned topics (as distributions over vocab of " + ldaModel.vocabSize()
+ " words):");
Matrix topics = ldaModel.topicsMatrix();
for (int topic = 0; topic < 3; topic++) {
System.out.print("Topic " + topic + ":");
for (int word = 0; word < ldaModel.vocabSize(); word++) {
System.out.print(" " + topics.apply(word, topic));
}
System.out.println();
}
ldaModel.save(jsc.sc(),
"target/org/apache/spark/JavaLatentDirichletAllocationExample/LDAModel");
DistributedLDAModel sameModel = DistributedLDAModel.load(jsc.sc(),
"target/org/apache/spark/JavaLatentDirichletAllocationExample/LDAModel");
Refer to the LDA
Python docs and LDAModel
Python docs for more details on the API.
from pyspark.mllib.clustering import LDA, LDAModel
from pyspark.mllib.linalg import Vectors
# Load and parse the data
data = sc.textFile("data/mllib/sample_lda_data.txt")
parsedData = data.map(lambda line: Vectors.dense([float(x) for x in line.strip().split(' ')]))
# Index documents with unique IDs
corpus = parsedData.zipWithIndex().map(lambda x: [x[1], x[0]]).cache()
# Cluster the documents into three topics using LDA
ldaModel = LDA.train(corpus, k=3)
# Output topics. Each is a distribution over words (matching word count vectors)
print("Learned topics (as distributions over vocab of " + str(ldaModel.vocabSize())
+ " words):")
topics = ldaModel.topicsMatrix()
for topic in range(3):
print("Topic " + str(topic) + ":")
for word in range(0, ldaModel.vocabSize()):
print(" " + str(topics[word][topic]))
# Save and load model
ldaModel.save(sc, "target/org/apache/spark/PythonLatentDirichletAllocationExample/LDAModel")
sameModel = LDAModel\
.load(sc, "target/org/apache/spark/PythonLatentDirichletAllocationExample/LDAModel")
Bisecting k-means
Bisecting K-means can often be much faster than regular K-means, but it will generally produce a different clustering.
Bisecting k-means is a kind of hierarchical clustering. Hierarchical clustering is one of the most commonly used method of cluster analysis which seeks to build a hierarchy of clusters. Strategies for hierarchical clustering generally fall into two types:
- Agglomerative: This is a “bottom up” approach: each observation starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy.
- Divisive: This is a “top down” approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy.
Bisecting k-means algorithm is a kind of divisive algorithms. The implementation in MLlib has the following parameters:
- k: the desired number of leaf clusters (default: 4). The actual number could be smaller if there are no divisible leaf clusters.
- maxIterations: the max number of k-means iterations to split clusters (default: 20)
- minDivisibleClusterSize: the minimum number of points (if >= 1.0) or the minimum proportion of points (if < 1.0) of a divisible cluster (default: 1)
- seed: a random seed (default: hash value of the class name)
Examples
Refer to the BisectingKMeans
Scala docs and BisectingKMeansModel
Scala docs for details on the API.
import org.apache.spark.mllib.clustering.BisectingKMeans
import org.apache.spark.mllib.linalg.{Vector, Vectors}
// Loads and parses data
def parse(line: String): Vector = Vectors.dense(line.split(" ").map(_.toDouble))
val data = sc.textFile("data/mllib/kmeans_data.txt").map(parse).cache()
// Clustering the data into 6 clusters by BisectingKMeans.
val bkm = new BisectingKMeans().setK(6)
val model = bkm.run(data)
// Show the compute cost and the cluster centers
println(s"Compute Cost: ${model.computeCost(data)}")
model.clusterCenters.zipWithIndex.foreach { case (center, idx) =>
println(s"Cluster Center ${idx}: ${center}")
}
Refer to the BisectingKMeans
Java docs and BisectingKMeansModel
Java docs for details on the API.
import java.util.Arrays;
import java.util.List;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.mllib.clustering.BisectingKMeans;
import org.apache.spark.mllib.clustering.BisectingKMeansModel;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
List<Vector> localData = Arrays.asList(
Vectors.dense(0.1, 0.1), Vectors.dense(0.3, 0.3),
Vectors.dense(10.1, 10.1), Vectors.dense(10.3, 10.3),
Vectors.dense(20.1, 20.1), Vectors.dense(20.3, 20.3),
Vectors.dense(30.1, 30.1), Vectors.dense(30.3, 30.3)
);
JavaRDD<Vector> data = sc.parallelize(localData, 2);
BisectingKMeans bkm = new BisectingKMeans()
.setK(4);
BisectingKMeansModel model = bkm.run(data);
System.out.println("Compute Cost: " + model.computeCost(data));
Vector[] clusterCenters = model.clusterCenters();
for (int i = 0; i < clusterCenters.length; i++) {
Vector clusterCenter = clusterCenters[i];
System.out.println("Cluster Center " + i + ": " + clusterCenter);
}
Refer to the BisectingKMeans
Python docs and BisectingKMeansModel
Python docs for more details on the API.
from numpy import array
from pyspark.mllib.clustering import BisectingKMeans, BisectingKMeansModel
# Load and parse the data
data = sc.textFile("data/mllib/kmeans_data.txt")
parsedData = data.map(lambda line: array([float(x) for x in line.split(' ')]))
# Build the model (cluster the data)
model = BisectingKMeans.train(parsedData, 2, maxIterations=5)
# Evaluate clustering
cost = model.computeCost(parsedData)
print("Bisecting K-means Cost = " + str(cost))
Streaming k-means
When data arrive in a stream, we may want to estimate clusters dynamically,
updating them as new data arrive. spark.mllib
provides support for streaming k-means clustering,
with parameters to control the decay (or “forgetfulness”) of the estimates. The algorithm
uses a generalization of the mini-batch k-means update rule. For each batch of data, we assign
all points to their nearest cluster, compute new cluster centers, then update each cluster using:
\begin{equation}
c_{t+1} = \frac{c_tn_t\alpha + x_tm_t}{n_t\alpha+m_t}
\end{equation}
\begin{equation}
n_{t+1} = n_t + m_t
\end{equation}
Where $c_t$
is the previous center for the cluster, $n_t$
is the number of points assigned
to the cluster thus far, $x_t$
is the new cluster center from the current batch, and $m_t$
is the number of points added to the cluster in the current batch. The decay factor $\alpha$
can be used to ignore the past: with $\alpha$=1
all data will be used from the beginning;
with $\alpha$=0
only the most recent data will be used. This is analogous to an
exponentially-weighted moving average.
The decay can be specified using a halfLife
parameter, which determines the
correct decay factor a
such that, for data acquired
at time t
, its contribution by time t + halfLife
will have dropped to 0.5.
The unit of time can be specified either as batches
or points
and the update rule
will be adjusted accordingly.
Examples
This example shows how to estimate clusters on streaming data.
Refer to the StreamingKMeans
Scala docs for details on the API.
And Refer to Spark Streaming Programming Guide for details on StreamingContext.
import org.apache.spark.mllib.clustering.StreamingKMeans
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.streaming.{Seconds, StreamingContext}
val conf = new SparkConf().setAppName("StreamingKMeansExample")
val ssc = new StreamingContext(conf, Seconds(args(2).toLong))
val trainingData = ssc.textFileStream(args(0)).map(Vectors.parse)
val testData = ssc.textFileStream(args(1)).map(LabeledPoint.parse)
val model = new StreamingKMeans()
.setK(args(3).toInt)
.setDecayFactor(1.0)
.setRandomCenters(args(4).toInt, 0.0)
model.trainOn(trainingData)
model.predictOnValues(testData.map(lp => (lp.label, lp.features))).print()
ssc.start()
ssc.awaitTermination()
Refer to the StreamingKMeans
Python docs for more details on the API.
And Refer to Spark Streaming Programming Guide for details on StreamingContext.
from pyspark.mllib.linalg import Vectors
from pyspark.mllib.regression import LabeledPoint
from pyspark.mllib.clustering import StreamingKMeans
# we make an input stream of vectors for training,
# as well as a stream of vectors for testing
def parse(lp):
label = float(lp[lp.find('(') + 1: lp.find(')')])
vec = Vectors.dense(lp[lp.find('[') + 1: lp.find(']')].split(','))
return LabeledPoint(label, vec)
trainingData = sc.textFile("data/mllib/kmeans_data.txt")\
.map(lambda line: Vectors.dense([float(x) for x in line.strip().split(' ')]))
testingData = sc.textFile("data/mllib/streaming_kmeans_data_test.txt").map(parse)
trainingQueue = [trainingData]
testingQueue = [testingData]
trainingStream = ssc.queueStream(trainingQueue)
testingStream = ssc.queueStream(testingQueue)
# We create a model with random clusters and specify the number of clusters to find
model = StreamingKMeans(k=2, decayFactor=1.0).setRandomCenters(3, 1.0, 0)
# Now register the streams for training and testing and start the job,
# printing the predicted cluster assignments on new data points as they arrive.
model.trainOn(trainingStream)
result = model.predictOnValues(testingStream.map(lambda lp: (lp.label, lp.features)))
result.pprint()
ssc.start()
ssc.stop(stopSparkContext=True, stopGraceFully=True)
As you add new text files with data the cluster centers will update. Each training
point should be formatted as [x1, x2, x3]
, and each test data point
should be formatted as (y, [x1, x2, x3])
, where y
is some useful label or identifier
(e.g. a true category assignment). Anytime a text file is placed in /training/data/dir
the model will update. Anytime a text file is placed in /testing/data/dir
you will see predictions. With new data, the cluster centers will change!


Gov. Dickson appoints 32 permanent secretaries four months to end of tenure
by Cletus UkpongThe Bayelsa State governor, Seriake Dickson, has appointed a new head for the state civil service.
Mr Dickson has also appointed 32 new permanent secretaries “to fill existing vacancies in the civil service,” a statement said.
The new acting head of service, Biobelemoye Charles-Onyeama, is said to be the most senior permanent secretary in Bayelsa civil service.
Mrs Charles-Onyeama’s appointment followed the retirement of the former head of service, Luka Obiri.
The appointments were announced on Tuesday through a statement from Mr Dickson’s spokesperson, Fidelis Soriwei.
According to the statement, “The appointments are based on seniority in service, meritorious service and fair spread among the Local Government Areas of the state”.
The new permanent secretaries would be sworn in on Friday by Mr Dickson.
Mr Dickson’s second term ends in February next year.
The governor’s party, the Peoples Democracy Party, was trounced in the Saturday’s governorship election in the state by the All Progressives Congress.
Below is a list of the new permanent secretaries as announced by the governor’s spokesperson, Mr Soriwei:
1. Ujile Ayi Helen
2. Mark Isaac Ikasuote
3. Ikelemo Nancy Waibite
4. Yaweh Help Jane
5. Mieyefa Emily
6. Singabele Oyinbrakemi Joyce
7. Oweimo Ebieme William
8. Gowon Toruyouyei
9. Eniekezimene Daniel
10. Tamaraodubo Thomas
Akpokabowei
11. Orubebe Timilaemi
12. Oguara Juliana D. A.
13. Omubo Wakiente Onokuma
14. Benwari Clarence Barimugha
15. Jokori James
16. Ekinabhari Dumatein Stephanie
17. Dr Asiam Blessing Ikuru
18. Patience Oliemen Erica Abah 19. Amasuomo Hannah Binafeigha
20. Keku Okobakiye Alaboigoni
21. Liverpool Walton Akpobolokemi
22. Alegi Ebiotu
23. Dickson Hitler
24. Ewhrudjakpo Christopher O
25. Koroye Tonye Enebimere
26. Freston Akpor
27. Imolade-Adeh Ayebatonye
28. Tukuru Patimidi Roland
29. Amokiese Beneovie Onumajira
30. Weribakumo Amaitari
31. Waripamo Martin Amaitari
32. Ndoni-Gabriel Ebiere Flora


"LUD "(1.) The fourth son of Shem (Genesis 10:22; 1 Chronicles 1:17), ancestor of the Lydians probably. (2.) One of the Hamitic tribes descended from Mizraim (Genesis 10:13), a people of Africa (Ezekiel 27:10; 30:5), on the west of Egypt. The people called Lud were noted archers (Isaiah 66:19; comp. Jeremiah 46:9).
"LUD "(1.) The fourth son of Shem (Genesis 10:22; 1 Chronicles 1:17), ancestor of the Lydians probably.
(2.) One of the Hamitic tribes descended from Mizraim (Genesis 10:13), a people of Africa (Ezekiel 27:10; 30:5), on the west of Egypt. The people called Lud were noted archers (Isaiah 66:19; comp. Jeremiah 46:9).