API and code to convert text into indexable/searchable tokens. Covers {@link org.apache.lucene.analysis.Analyzer} and related classes.
Lucene, an indexing and search library, accepts only plain text input.
Applications that build their search capabilities upon Lucene may support documents in various formats – HTML, XML, PDF, Word – just to name a few. Lucene does not care about the Parsing of these and other document formats, and it is the responsibility of the application using Lucene to use an appropriate Parser to convert the original format into plain text before passing that plain text to Lucene.
Plain text passed to Lucene for indexing goes through a process generally called tokenization. Tokenization is the process of breaking input text into small indexing elements – tokens. The way input text is broken into tokens heavily influences how people will then be able to search for that text. For instance, sentences beginnings and endings can be identified to provide for more accurate phrase and proximity searches (though sentence identification is not provided by Lucene).
In some cases simply breaking the input text into tokens is not enough – a deeper Analysis may be needed. Lucene includes both pre- and post-tokenization analysis facilities.
Pre-tokenization analysis can include (but is not limited to) stripping HTML markup, and transforming or removing text matching arbitrary patterns or sets of fixed strings.
There are many post-tokenization steps that can be done, including (but not limited to):
The analysis package provides the mechanism to convert Strings and Readers into tokens that can be indexed by Lucene. There are four main classes in the package from which all analysis processes are derived. These are:
The synergy between {@link org.apache.lucene.analysis.Analyzer} and {@link org.apache.lucene.analysis.Tokenizer} is sometimes confusing. To ease this confusion, some clarifications:
Lucene Java provides a number of analysis capabilities, the most commonly used one being the StandardAnalyzer. Many applications will have a long and industrious life with nothing more than the StandardAnalyzer. However, there are a few other classes/packages that are worth mentioning:
Analysis is one of the main causes of performance degradation during indexing. Simply put, the more you analyze the slower the indexing (in most cases). Perhaps your application would be just fine using the simple WhitespaceTokenizer combined with a StopFilter. The benchmark/ library can be useful for testing out the speed of the analysis process.
Applications usually do not invoke analysis – Lucene does it for them:
However an application might invoke Analysis of any text for testing or for any other purpose, something like:
Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY Analyzer analyzer = new StandardAnalyzer(matchVersion); // or any other analyzer TokenStream ts = analyzer.tokenStream("myfield", new StringReader("some text goes here")); OffsetAttribute offsetAtt = ts.addAttribute(OffsetAttribute.class); try { ts.reset(); // Resets this stream to the beginning. (Required) while (ts.incrementToken()) { // Use {@link org.apache.lucene.util.AttributeSource#reflectAsString(boolean)} // for token stream debugging. System.out.println("token: " + ts.reflectAsString(true)); System.out.println("token start offset: " + offsetAtt.startOffset()); System.out.println(" token end offset: " + offsetAtt.endOffset()); } ts.end(); // Perform end-of-stream operations, e.g. set the final offset. } finally { ts.close(); // Release resources associated with this stream. }
Selecting the "correct" analyzer is crucial for search quality, and can also affect indexing and search performance. The "correct" analyzer differs between applications. Lucene java's wiki page AnalysisParalysis provides some data on "analyzing your analyzer". Here are some rules of thumb:
Creating your own Analyzer is straightforward. Your Analyzer can wrap existing analysis components — CharFilter(s) (optional), a Tokenizer, and TokenFilter(s) (optional) — or components you create, or a combination of existing and newly created components. Before pursuing this approach, you may find it worthwhile to explore the analyzers-common library and/or ask on the<EMAIL_ADDRESS>mailing list first to see if what you need already exists. If you are still committed to creating your own Analyzer, have a look at the source code of any one of the many samples located in this package.
The following sections discuss some aspects of implementing your own analyzer.
When {@link org.apache.lucene.document.Document#add(org.apache.lucene.index.IndexableField) document.add(field)} is called multiple times for the same field name, we could say that each such call creates a new section for that field in that document. In fact, a separate call to {@link org.apache.lucene.analysis.Analyzer#tokenStream(java.lang.String, java.io.Reader) tokenStream(field,reader)} would take place for each of these so called "sections". However, the default Analyzer behavior is to treat all these sections as one large section. This allows phrase search and proximity search to seamlessly cross boundaries between these "sections". In other words, if a certain field "f" is added like this:
document.add(new Field("f","first ends",...); document.add(new Field("f","starts two",...); indexWriter.addDocument(document);
Then, a phrase search for "ends starts" would find that document. Where desired, this behavior can be modified by introducing a "position gap" between consecutive field "sections", simply by overriding {@link org.apache.lucene.analysis.Analyzer#getPositionIncrementGap(java.lang.String) Analyzer.getPositionIncrementGap(fieldName)}:
Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY Analyzer myAnalyzer = new StandardAnalyzer(matchVersion) { public int getPositionIncrementGap(String fieldName) { return 10; } };
By default, all tokens created by Analyzers and Tokenizers have a {@link org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute#getPositionIncrement() position increment} of one. This means that the position stored for that token in the index would be one more than that of the previous token. Recall that phrase and proximity searches rely on position info.
If the selected analyzer filters the stop words "is" and "the", then for a document containing the string "blue is the sky", only the tokens "blue", "sky" are indexed, with position("sky") = 1 + position("blue"). Now, a phrase query "blue is the sky" would find that document, because the same analyzer filters the same stop words from that query. But also the phrase query "blue sky" would find that document.
If this behavior does not fit the application needs, a modified analyzer can be used, that would increment further the positions of tokens following a removed stop word, using {@link org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute#setPositionIncrement(int)}. This can be done with something like the following (note, however, that StopFilter natively includes this capability by subclassing FilteringTokenFilter}:
public TokenStream tokenStream(final String fieldName, Reader reader) { final TokenStream ts = someAnalyzer.tokenStream(fieldName, reader); TokenStream res = new TokenStream() { CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); PositionIncrementAttribute posIncrAtt = addAttribute(PositionIncrementAttribute.class); public boolean incrementToken() throws IOException { int extraIncrement = 0; while (true) { boolean hasNext = ts.incrementToken(); if (hasNext) { if (stopWords.contains(termAtt.toString())) { extraIncrement++; // filter this word continue; } if (extraIncrement>0) { posIncrAtt.setPositionIncrement(posIncrAtt.getPositionIncrement()+extraIncrement); } } return hasNext; } } }; return res; }
Now, with this modified analyzer, the phrase query "blue sky" would find that document. But note that this is yet not a perfect solution, because any phrase query "blue w1 w2 sky" where both w1 and w2 are stop words would match that document.
A few more use cases for modifying position increments are:
"Flexible Indexing" summarizes the effort of making the Lucene indexer pluggable and extensible for custom index formats. A fully customizable indexer means that users will be able to store custom data structures on disk. Therefore an API is necessary that can transport custom types of data from the documents to the indexer.
Classes {@link org.apache.lucene.util.Attribute} and {@link org.apache.lucene.util.AttributeSource} serve as the basis upon which the analysis elements of "Flexible Indexing" are implemented. An Attribute holds a particular piece of information about a text token. For example, {@link org.apache.lucene.analysis.tokenattributes.CharTermAttribute} contains the term text of a token, and {@link org.apache.lucene.analysis.tokenattributes.OffsetAttribute} contains the start and end character offsets of a token. An AttributeSource is a collection of Attributes with a restriction: there may be only one instance of each attribute type. TokenStream now extends AttributeSource, which means that one can add Attributes to a TokenStream. Since TokenFilter extends TokenStream, all filters are also AttributeSources.
Lucene provides seven Attributes out of the box:
| {@link org.apache.lucene.analysis.tokenattributes.CharTermAttribute} | The term text of a token. Implements {@link java.lang.CharSequence} (providing methods length() and charAt(), and allowing e.g. for direct use with regular expression {@link java.util.regex.Matcher}s) and {@link java.lang.Appendable} (allowing the term text to be appended to.) |
| {@link org.apache.lucene.analysis.tokenattributes.OffsetAttribute} | The start and end offset of a token in characters. |
| {@link org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute} | See above for detailed information about position increment. |
| {@link org.apache.lucene.analysis.tokenattributes.PayloadAttribute} | The payload that a Token can optionally have. |
| {@link org.apache.lucene.analysis.tokenattributes.TypeAttribute} | The type of the token. Default is 'word'. |
| {@link org.apache.lucene.analysis.tokenattributes.FlagsAttribute} | Optional flags a token can have. |
| {@link org.apache.lucene.analysis.tokenattributes.KeywordAttribute} | Keyword-aware TokenStreams/-Filters skip modification of tokens that return true from this attribute's isKeyword() method. |
Class
)
of an Attribute as an argument and returns an instance. If an Attribute of the same type was previously added, then
the already existing instance is returned, otherwise a new instance is created and returned. Therefore TokenStreams/-Filters
can safely call addAttribute() with the same Attribute type multiple times. Even consumers of TokenStreams should
normally call addAttribute() instead of getAttribute(), because it would not fail if the TokenStream does not have this
Attribute (getAttribute() would throw an IllegalArgumentException, if the Attribute is missing). More advanced code
could simply check with hasAttribute(), if a TokenStream has it, and may conditionally leave out processing for
extra performance.
In this example we will create a WhiteSpaceTokenizer and use a LengthFilter to suppress all words that have only two or fewer characters. The LengthFilter is part of the Lucene core and its implementation will be explained here to illustrate the usage of the TokenStream API.
Then we will develop a custom Attribute, a PartOfSpeechAttribute, and add another filter to the chain which utilizes the new custom attribute, and call it PartOfSpeechTaggingFilter.
public class MyAnalyzer extends Analyzer { private Version matchVersion; public MyAnalyzer(Version matchVersion) { this.matchVersion = matchVersion; } {@literal @Override} protected TokenStreamComponents createComponents(String fieldName, Reader reader) { return new TokenStreamComponents(new WhitespaceTokenizer(matchVersion, reader)); } public static void main(String[] args) throws IOException { // text to tokenize final String text = "This is a demo of the TokenStream API"; Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY MyAnalyzer analyzer = new MyAnalyzer(matchVersion); TokenStream stream = analyzer.tokenStream("field", new StringReader(text)); // get the CharTermAttribute from the TokenStream CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class); try { stream.reset(); // print all tokens until stream is exhausted while (stream.incrementToken()) { System.out.println(termAtt.toString()); } stream.end(); } finally { stream.close(); } } }In this easy example a simple white space tokenization is performed. In main() a loop consumes the stream and prints the term text of the tokens by accessing the CharTermAttribute that the WhitespaceTokenizer provides. Here is the output:
This is a demo of the new TokenStream API
createComponents()
method in our analyzer needs to be changed:
{@literal @Override} protected TokenStreamComponents createComponents(String fieldName, Reader reader) { final Tokenizer source = new WhitespaceTokenizer(matchVersion, reader); TokenStream result = new LengthFilter(true, source, 3, Integer.MAX_VALUE); return new TokenStreamComponents(source, result); }Note how now only words with 3 or more characters are contained in the output:
This demo the new TokenStream APINow let's take a look how the LengthFilter is implemented:
public final class LengthFilter extends FilteringTokenFilter { private final int min; private final int max; private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); /** * Build a filter that removes words that are too long or too * short from the text. */ public LengthFilter(boolean enablePositionIncrements, TokenStream in, int min, int max) { super(enablePositionIncrements, in); this.min = min; this.max = max; } {@literal @Override} public boolean accept() throws IOException { final int len = termAtt.length(); return (len >= min && len <= max); } }
In LengthFilter, the CharTermAttribute is added and stored in the instance
variable termAtt
. Remember that there can only be a single
instance of CharTermAttribute in the chain, so in our example the
addAttribute()
call in LengthFilter returns the
CharTermAttribute that the WhitespaceTokenizer already added.
The tokens are retrieved from the input stream in FilteringTokenFilter's
incrementToken()
method (see below), which calls LengthFilter's
accept()
method. By looking at the term text in the
CharTermAttribute, the length of the term can be determined and tokens that
are either too short or too long are skipped. Note how
accept()
can efficiently access the instance variable; no
attribute lookup is necessary. The same is true for the consumer, which can
simply use local references to the Attributes.
LengthFilter extends FilteringTokenFilter:
public abstract class FilteringTokenFilter extends TokenFilter { private final PositionIncrementAttribute posIncrAtt = addAttribute(PositionIncrementAttribute.class); private boolean enablePositionIncrements; // no init needed, as ctor enforces setting value! public FilteringTokenFilter(boolean enablePositionIncrements, TokenStream input){ super(input); this.enablePositionIncrements = enablePositionIncrements; } /** Override this method and return if the current input token should be returned by {@literal {@link #incrementToken}}. */ protected abstract boolean accept() throws IOException; {@literal @Override} public final boolean incrementToken() throws IOException { if (enablePositionIncrements) { int skippedPositions = 0; while (input.incrementToken()) { if (accept()) { if (skippedPositions != 0) { posIncrAtt.setPositionIncrement(posIncrAtt.getPositionIncrement() + skippedPositions); } return true; } skippedPositions += posIncrAtt.getPositionIncrement(); } } else { while (input.incrementToken()) { if (accept()) { return true; } } } // reached EOS -- return false return false; } /** * {@literal @see #setEnablePositionIncrements(boolean)} */ public boolean getEnablePositionIncrements() { return enablePositionIncrements; } /** * Iftrue
, this TokenFilter will preserve * positions of the incoming tokens (ie, accumulate and * set position increments of the removed tokens). * Generally,true
is best as it does not * lose information (positions of the original tokens) * during indexing. * *When set, when a token is stopped * (omitted), the position increment of the following * token is incremented. * *
NOTE: be sure to also * set org.apache.lucene.queryparser.classic.QueryParser#setEnablePositionIncrements if * you use QueryParser to create queries. */ public void setEnablePositionIncrements(boolean enable) { this.enablePositionIncrements = enable; } }
PartOfSpeechAttribute
. First we need to define the interface of the new Attribute:
public interface PartOfSpeechAttribute extends Attribute { public static enum PartOfSpeech { Noun, Verb, Adjective, Adverb, Pronoun, Preposition, Conjunction, Article, Unknown } public void setPartOfSpeech(PartOfSpeech pos); public PartOfSpeech getPartOfSpeech(); }
Now we also need to write the implementing class. The name of that class is important here: By default, Lucene
checks if there is a class with the name of the Attribute with the suffix 'Impl'. In this example, we would
consequently call the implementing class PartOfSpeechAttributeImpl
.
This should be the usual behavior. However, there is also an expert-API that allows changing these naming conventions: {@link org.apache.lucene.util.AttributeSource.AttributeFactory}. The factory accepts an Attribute interface as argument and returns an actual instance. You can implement your own factory if you need to change the default behavior.
Now here is the actual class that implements our new Attribute. Notice that the class has to extend {@link org.apache.lucene.util.AttributeImpl}:
public final class PartOfSpeechAttributeImpl extends AttributeImpl implements PartOfSpeechAttribute { private PartOfSpeech pos = PartOfSpeech.Unknown; public void setPartOfSpeech(PartOfSpeech pos) { this.pos = pos; } public PartOfSpeech getPartOfSpeech() { return pos; } {@literal @Override} public void clear() { pos = PartOfSpeech.Unknown; } {@literal @Override} public void copyTo(AttributeImpl target) { ((PartOfSpeechAttribute) target).setPartOfSpeech(pos); } }
This is a simple Attribute implementation has only a single variable that
stores the part-of-speech of a token. It extends the
AttributeImpl
class and therefore implements its abstract methods
clear()
and copyTo()
. Now we need a TokenFilter that
can set this new PartOfSpeechAttribute for each token. In this example we
show a very naive filter that tags every word with a leading upper-case letter
as a 'Noun' and all other words as 'Unknown'.
public static class PartOfSpeechTaggingFilter extends TokenFilter { PartOfSpeechAttribute posAtt = addAttribute(PartOfSpeechAttribute.class); CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); protected PartOfSpeechTaggingFilter(TokenStream input) { super(input); } public boolean incrementToken() throws IOException { if (!input.incrementToken()) {return false;} posAtt.setPartOfSpeech(determinePOS(termAtt.buffer(), 0, termAtt.length())); return true; } // determine the part of speech for the given term protected PartOfSpeech determinePOS(char[] term, int offset, int length) { // naive implementation that tags every uppercased word as noun if (length > 0 && Character.isUpperCase(term[0])) { return PartOfSpeech.Noun; } return PartOfSpeech.Unknown; } }
Just like the LengthFilter, this new filter stores references to the attributes it needs in instance variables. Notice how you only need to pass in the interface of the new Attribute and instantiating the correct class is automatically taken care of.
Now we need to add the filter to the chain in MyAnalyzer:
{@literal @Override} protected TokenStreamComponents createComponents(String fieldName, Reader reader) { final Tokenizer source = new WhitespaceTokenizer(matchVersion, reader); TokenStream result = new LengthFilter(true, source, 3, Integer.MAX_VALUE); result = new PartOfSpeechTaggingFilter(result); return new TokenStreamComponents(source, result); }Now let's look at the output:
This demo the new TokenStream APIApparently it hasn't changed, which shows that adding a custom attribute to a TokenStream/Filter chain does not affect any existing consumers, simply because they don't know the new Attribute. Now let's change the consumer to make use of the new PartOfSpeechAttribute and print it out:
public static void main(String[] args) throws IOException { // text to tokenize final String text = "This is a demo of the TokenStream API"; MyAnalyzer analyzer = new MyAnalyzer(); TokenStream stream = analyzer.tokenStream("field", new StringReader(text)); // get the CharTermAttribute from the TokenStream CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class); // get the PartOfSpeechAttribute from the TokenStream PartOfSpeechAttribute posAtt = stream.addAttribute(PartOfSpeechAttribute.class); try { stream.reset(); // print all tokens until stream is exhausted while (stream.incrementToken()) { System.out.println(termAtt.toString() + ": " + posAtt.getPartOfSpeech()); } stream.end(); } finally { stream.close(); } }The change that was made is to get the PartOfSpeechAttribute from the TokenStream and print out its contents in the while loop that consumes the stream. Here is the new output:
This: Noun demo: Unknown the: Unknown new: Unknown TokenStream: Noun API: NounEach word is now followed by its assigned PartOfSpeech tag. Of course this is a naive part-of-speech tagging. The word 'This' should not even be tagged as noun; it is only spelled capitalized because it is the first word of a sentence. Actually this is a good opportunity for an exercise. To practice the usage of the new API the reader could now write an Attribute and TokenFilter that can specify for each word if it was the first token of a sentence or not. Then the PartOfSpeechTaggingFilter can make use of this knowledge and only tag capitalized words as nouns if not the first word of a sentence (we know, this is still not a correct behavior, but hey, it's a good exercise). As a small hint, this is how the new Attribute class could begin:
public class FirstTokenOfSentenceAttributeImpl extends AttributeImpl implements FirstTokenOfSentenceAttribute { private boolean firstToken; public void setFirstToken(boolean firstToken) { this.firstToken = firstToken; } public boolean getFirstToken() { return firstToken; } {@literal @Override} public void clear() { firstToken = false; } ...
{@link org.apache.lucene.analysis.CharFilter} is designed to allow you to pre-process input like a FilterReader would, but also preserve the original offsets associated with those characters. This way mechanisms like highlighting still work correctly. CharFilters can be chained.
Example:
public class MyAnalyzer extends Analyzer { {@literal @Override} protected TokenStreamComponents createComponents(String fieldName, Reader reader) { return new TokenStreamComponents(new MyTokenizer(reader)); } {@literal @Override} protected Reader initReader(String fieldName, Reader reader) { // wrap the Reader in a CharFilter chain. return new SecondCharFilter(new FirstCharFilter(reader)); } }


Marvel's God of Thunder Wields Mjolnir in Artgerm's Thor #1 Variant
by Berek BigosWith Jason Aaron wrapping up his final story arc for the Thunder God, Marvel is already preparing for Donny Cates and Nic Klein to helm Thor's next adventure. The publisher has now revealed Stanley "Artgerm" Lau's variant cover for Thor #1 -- which arrives in comic shops New Year's Day.
Since Aaron's grand finale has yet to reach its conclusion, Marvel has remained quite secretive regarding what fans can expect when Cates and Klein take over the series. That said, it comes as no surprise that Artgerm's variant doesn't include any spoilers or hint at an unknown plot element from Thor's upcoming relaunch. Rather, it is simply a stunning image of Thor with his trusty hammer Mjolnir in hand.
Click the button below to start this article in quick view.
During CBR's recent discussion with Donny Cates, we were offered a closer look at what this new creative team has in store for readers. Amongst the handful of details discussed, Cates mentioned that he plans to delve into Thor's mental state as the new All-Father and discover what it means to be a leader.
Those eagerly anticipating the relaunch can read the Asgardian's current series, King Thor, featuring writer Jason Aaron and artist Esad Ribic. With the first three issues available now, this four-issue limited series promises to wrap up Thor's current story arc this December while also laying the groundwork for his next creative team.
Written by Donny Cates and illustrated by Nic Klein, Thor #1 goes on sale Jan. 1, 2020 from Marvel Comics.
https://schema.orgBreadcrumbListListItem1Homehttps://www.cbr.com/ListItem2Comicshttps://www.cbr.com/category/comics/ListItem3Comic Newshttps://www.cbr.com/category/comics/news/ListItem4Marvel's God of Thunder Wields Mjolnir in Artgerm's Thor #1 Varianthttps://www.cbr.com/thor-1-2020-artgerm-variant/


Hi my name is Sura. I’m a front-end developer. My passion for art and technology have defined my career and education goals. I obtained a degree in Computer Science.  However, as much as I value technology I don't rely completely on it. I prefer face to face interactions.
You can learn more about my porfessional profile on Linkedin
I’m a self taught artist. In my spare time I love to paint and draw. I have been enjoying exploring art since the age of 5.
I still remember as a child the joy I felt to see people paint with colors, it distracted me from the hardship of undergoing war and civil unrest when I lived in Iraq.
I like to paint about my dreams, my emotions and I encourage everyone to do it, to try and explore the artist in you.
Let life inspire you.
You can explore my facebook page Follow Your Passion
I’m a passionate individual and I am a committed to making positive impact on the community through voluntary work. Besides pursuing my career in technology, I have enjoyed volunteering with different NGO's such as United Nations, Salvation Army, World Food Program and recently Sick Kids. I also enjoy running for a cause like the Terry Fox Marathon for cancer research.
HTML, CSS, responsive design, Javascript and AJAX, Project management.
Pencil and paper, watercolor or acrylics paints, Photoshop, Gimp, inkscape, Atom, Sublime, webstorm, serge, Github
I'm bi-lingual I speak fluently English and Arabic.
I would love to hear from you if you need any of my servies. Let’s get in touch
If you need any other task that wasn't listed get in touch with me at<EMAIL_ADDRESS>and I'm happy to work it out with you or guide you to some one who might be able to help you.
Want to get in touch? Fill out the form below to send me a message and I will get back to you as soon as possible!


User Support for Statistical Disclosure Control Tools
The 2023 User Group meeting took place online, 10:00 - 12:30 C.E.T., on February the 23-rd.
The meeting agenda and the slides of the presentations, as well as the minutes of the meeting are available here. There were 35 participants and five presentations concerning both the status/progress of the sdc-tools presented by developers and several practical solutions/applications presented by users.
On September 20, 2022, a SDC Useer Group Workshop took place in Paris, at INSEE. Being a hybrid meeting, it was also possible to attend the workshop online. The workshop concentrated on practical implementations of SDC in statistical production and dissemination. There were 7 presentations given by different users coming from France, Germany, Ireland, Norway, Poland and Iceland. More details can be found here: List of accepted abstracts (pdf), Agenda (pdf) and Slides of presentations (pdf).
The 2022 SDC User Group meeting took place online on February 25. There were 36 participants from 18 different countries. There were presentations by the developers teams on the status quo of the different SDC tools, a discussion on the pros and cons of the different LP-solvers needed for some of the approaches in τ-argus (see slides) and a presentation of a new initiative by INSEE to write an R-wrapper for τ-argus called rtauargus
(see here for the sources).


--- layout: project categories: projects client: Strive client_url: client_color: video_title: "We are Strive" video_id: 292995702 video_thumb: strive.png featured: true ---
Director — Cole Mueller
Producer — Vitascope Arts
Editor — Cole Mueller
VFX — Richard Cabrera


Psalms 119:58
I intreated thy favour with my whole heart: be merciful unto me according to thy word.


[ARTICLE XX.--1933]
Section 1. The terms of the President and Vice-President shall end at noon on the 20th day of January, and the terms of Senators and Representatives at noon on the 3rd day of January, of the years in which such terms would have ended if this article had not been ratified; and the terms of their successors shall then begin.


Emulation
Mode:
disconnected
abortingRelayHosting
abortingRelayConnection
stoppingHosting
startingHosting
disconnectingFromHost
disconnectingFromRelayForHosting
disconnectingFromRelayForConnection
connectingToHost
connectingToRelayForHosting
connectingToRelayForConnection
connectingToRelayForCallback
awaitingTokenConnection
awaitingToken
connectedToHost
hostingDirectly
connectedToRelayForHosting
connectedToRelayForConnection
Online:
Offline:
NEED SWIRLY LIGHT
EMULATE GATEWAY:
Not Started
Unknown Nat
Public Nat
Private Nat
EMULATE PEER:
disconnected
abortingRelayHosting
abortingRelayConnection
stoppingHosting
startingHosting
disconnectingFromHost
disconnectingFromRelayForHosting
disconnectingFromRelayForConnection
connectingToHost
connectingToRelayForHosting
connectingToRelayForConnection
connectingToRelayForCallback
awaitingTokenConnection
awaitingToken
connectedToHost
hostingDirectly
connectedToRelayForHosting
connectedToRelayForConnection
EMULATE RELAY:
No Relay
Idle
Waiting for host
Connected to host
EMULATE ROLE:
Not Playing
Guest
Host
Relay
Solo
Playing:
In:
Join
Relay
Stop Relay
Download
Stop Hosting
Host
Host
Quit
MUD
Storage
Connection
Settings
About
Create
Upload
Username:
NAT Status:
Peer ID:
Join
Give your connect string to people you want to invite
Connect to relay
Your Connect string:
Waiting for protocol listener
Your Connect string:
Your peer does not have public access so you must use a relay
Ask someone to start relaying
They will send you the relay connect string
Paste the relay connect string above and click "Connect to relay"
Give your relay connect string to the private peer's owner
Relay
Host Peer:
Relay connect string:
Paste the private peer's id above
Click Relay
Copy the relay connect string and send it to the private peer's owner
Run Command
Name:
Peer Port:
Peer ID:
Activate
Copy
World name:
Generate download
DELETING
Users
Extensions
Upload extension
Save
Cancel
Name:
Password:
admin
default
Name
Hash
Upload new version
View extension
, please close the page


Match Play. Red trails 3-7 to 11. How should Red play 43?
pips Red=142 White=135
(CorrectAnswer= 0/0, ErrorScore= 0)
This position came up in the on-line match between USBGF members and Neil Kazaross. My choice was to play bar/22, 9/5 keeping the fifth point in the prime slotted but this turns out to be a bad error and bar/22, 8/4 is the correct play.
This leaves only two shots at the blot on the 9-pt instead of four at the blot on the 8-pt. Given that Red has no anchor he needs to play conservatively and therefore bar/22, 8/4 is the right approach.
Bar/22, 13/9 is way too loose and is in blunder territory. As noted above Red needs to play cautiously in this position despite the score.
This leaves only two shots at the blot on the 9-pt instead of four at the blot on the 8-pt. Given that Red has no anchor he needs to play conservatively and therefore bar/22, 8/4 is the right approach.
Bar/22, 13/9 is way too loose and is in blunder territory. As noted above Red needs to play cautiously in this position despite the score.
1. Cubeful 2-ply bar/22 8/4 Eq.: +0.275 0.529 0.193 0.015 - 0.471 0.213 0.013 2. Cubeful 2-ply bar/22 9/5 Eq.: +0.213 (-0.062) 0.518 0.189 0.016 - 0.482 0.232 0.014 3. Cubeful 2-ply bar/22 13/9 Eq.: +0.181 (-0.095) 0.519 0.175 0.016 - 0.481 0.264 0.026 4. Cubeful 0-ply bar/22 6/2 Eq.: -0.225 (-0.500) 0.443 0.135 0.010 - 0.557 0.338 0.042 5. Cubeful 0-ply bar/22 7/3 Eq.: -0.810 (-1.085) 0.327 0.096 0.007 - 0.673 0.460 0.097


|
The MatrixMaybe if your college philosophy class had featured insane kung fu stunts, car chases and digital FX, you might have scored that coveted 2.9 grade point average."The Matrix" is one of those truly bizarre phenomena that restore your faith in humanity. Released in 1999 with little hype by the then-unknown writer-director team of Andy and Larry Wachowski, "The Matrix" appropriately snuck into the American zeitgeist like a computer virus in the brain. The movie was an unlikely combination of Hong Kong action flick, Philip K Dick sci-fi, second-year philosophy, Gnosticism and leather, intellectual riddles and Keanu Reeves. (Spoilers for the first "Matrix" follow.) The premise goes something like this: The world as we know it is not real.  The machines discovered that bodies die when their minds atrophy, so they created the Matrix, a massive virtual reality simulation designed to keep the human race placated in a not-so-gilded cage. A few individuals with powerful minds act as freedom fighters in this brave new world; they have learned the unreality of the Matrix and have seen the ugly truth of the real world. Because they know the Matrix is unreal, they can bend the laws of physics when they are inside — providing a ready made excuse for the truly unbelievable fight sequences. The Wachowski brothers had been raised on a steady diet of Japanese animation and Hong Kong action movies, and it really, really shows. The first Matrix featured action sequences unlike anything the cinema had seen before or since... Until the "Matrix Reloaded" debuted in 2003, anyway. The action sequences in the Matrix films feature a mind-bending array of impossible stunts and time-warping perceptions. The most famous effect, known as "bullet time," involved slowing the action mid-fight so that you could see the flight of bullets in midair. And they did it with style — our heroes Neo (Reeves), Trinity (Carrie-Ann Moss) and Morpheus (Laurence Fishburne) were not just decked out in the oh-so-'80s camo of "Rambo" and "Predator" vintage. One of the perks of transcendent gnosis is apparently the freedom to whip up whatever wardrobe you prefer, which involves using startling amounts of leather and vinyl for maximum hipness. But for all the glitz and "holy shit" visuals, the movie's cult success was driven in large part by the concepts.  The Gnostic worldview is strongly represented throughout the movie, which coincides with the Philip K Dick-style paranoia and questioning of reality. Grant Morrison's counter-culture comic book The Invisibles was kept on the set during the filming of the first movie, and its influence is also clearly visible throughout. In an unreal universe, your mind is the most powerful tool, a concept frequently referenced throughout the trilogy. While movie reviewers robotically fixated on Neo as a dot-com hero of the 1990s, they frequently missed some of the broader philosophical points, including the extremely unsubtle message that Neo has much more in common with Jesus Christ than Kevin Mitnick. The Matrix was also influential in more disturbing ways. The movie featured an extraordinary amount of gunplay, even by the most jaded action film standards, leading a variety of weak-minded psychos to blame the Matrix for their questionable behavior. The highest-profile case was Columbine high school massacre in 1999, in which Eric Harris and Dylan Klebold armed themselves to the teeth, donned black trench coats and gunned down a couple dozen classmates, killing 15 of them. They claimed to have been influenced by the Matrix, as did quite a few other crazy killers. Some even tried to use the movie in their legal defense, explaining that they killed people because they realized after watching the movie that nothing was real. After "The Matrix" became a surprise cult hit, grossing upwards of $170 million in the U.S. and hundreds of millions more overseas, sequels followed as surely as night follows day. The Wachowskis had always planned to continue the story in some form (it was originally planned as a comic book), but the movie's shocking success allowed them to start thinking bigger. Although they had decided the plot details long before the first movie was even released, they approached the sequels with a fresh eye toward two major issues. Their first goal was to top the special effects innovations in the original movie, and their second goal was to figure out how to do the sequels and still pull in a profit. The original Matrix cost about $63 million to make; its sequels would approximately double that, even with cost-saving measures in place, including shooting both films back to back during a marathon six-month production that saw cast members die, marry, become pregnant and break bones performing the mind-blowing stunts. The two sequels were scheduled for back-to-back releases, with "The Matrix: Reloaded" appearing in May 2003 and "The Matrix: Revolutions" set for a November 2003 debut. "Reloaded" upped the ante on action with a numbing freeway chase, featuring an incredible martial arts battle on top of a speeding semi truck and the spectacle of Keanu Reeves in a courtyard fighting hundreds of clones of his nemesis, Agent Smith, a computer virus. The movie also upped the philosophical ante with several dramatic (but not entirely unexpected) revelations about the nature of the Matrix and the dynamics driving the characters into their respective roles, as well as brain candy such as some fun insider references to the Knights Templar and the true secret of the Holy Grail. The contents of "The Matrix: Revolutions" remain a closely guarded secret, with only a sparse trailer after the 20-minute credit sequence in "Reloaded" to provide any clues. It would be nice to think that pop culture could produce a truly paradigm-shattering, world-changing evolution in philosophical thought, but you'll have to wait until November to find out. UPDATE: "The Matrix: Revolutions" has been released.  At the risk of spoiling the "surprise" for the three people in the world who didn't see it coming, the concluding chapter of the Matrix trilogy descends into the sort of depressingly predictable Christian Crucifixion equivalency that the original movie would have scoffed at. Neo as Jesus Christ? Yeah, wow, that's original. Gnostic, my ass... They could have at least made Trinity into the Christ figure and invited a little controversy. Between Neo, Frodo and Mel Gibson's new Jesus, that's three Christ-on-a-stick Hollywood blockbusters in one 12 month period, which is surely some sort of sign that the end is near...
|


High-Level API¶
Overview¶
The High-Level API makes it easy to rapidly:
Prepare samples.
Hypertune/ train queues of models.
Feed appropriate data/ parameters into those models.
Evaluate model performance with metrics and plots.
It does so by wrapping and bundling together the methods of the Low-Level API. The table below demonstrates how the high-level entities abstract abstract the low-level entities. While this abstraction eliminates many steps to enable rapid model prototyping, it comes at a cost of customization.
High-level object |
Groups together or creates the following objects |
|---|---|
|
Dataset, File, Image, Tabular, Label, Featureset, Splitset, Foldset, Folds, Labelcoder, Encoderset, Featurecoders. |
|
Functions to build, train, predict, and evaluate a machine learning model. |
|
Algorithm, Hyperparamset, Hyperparamcombos, Queue, Job, Jobset, Result. |
Prerequisites¶
If you’ve already completed the instructions on the Installation page, then let’s get started.
[4]:
import aiqc
from aiqc import datum
1. Pipeline¶
a) Tabular Dataset¶
Tabular/ delimited/ flat-file Dataset.Tabular
can be created from either Pandas DataFrames or flat files (CSV/ TSV or Parquet).
Let’s grab one of AIQC’s built-in datasets from the datum
module that we imported above. This module is described in the ‘Built-In Examples - Datasets’ section of the documentation.
[3]:
df = datum.to_pandas(name='iris.tsv')
The Pipeline
process starts with raw data. A Dataset object is generated from that data and prepared for training based on the parameters the user provides to the Pipeline.make
method. To get started, set the dataFrame_or_filePath
equal to the dataframe we just fetched. It’s the only argument that’s actually required so
Import any scikit-learn encoders that you want to use to encode labels and/ or features. Any encoders that you pass in will need to be instantiated with the attributes you want them to use.
Reference the
Encoderset
section of the low-level API for more detail on how to include/ exclude specificFeatureset
columns by name/dtype. Thefeature_encoders
argument seen below takes a list of dictionaries as input, where each dictionary contains the**kwargs
for aFeaturecoder
.
[4]:
from sklearn.preprocessing import OneHotEncoder, PowerTransformer
Rather than wrangling your data with many lines of data science code, just set the arguments below and AIQC takes care of the rest: stratification (including continuous dtypes), validation splits, cross-validation folds, and dtype/column specific encoders to be applied on-read.
Don’t use
fold_count
unless your (total sample count / fold_count) still gives you an accurate representation of your sample population. You can try it with the ‘iris_10x.tsv’ datum.
[5]:
splitset = aiqc.Pipeline.Tabular.make(
dataFrame_or_filePath = df
, dtype = None
, label_column = 'species'
, features_excluded = None
, size_test = 0.24
, size_validation = 0.12
, fold_count = None
, bin_count = None
, label_encoder = OneHotEncoder(sparse=False)
, feature_encoders = [{
"sklearn_preprocess": PowerTransformer(method='box-cox', copy=False)
, "dtypes": ['float64']
}]
, size_window = None
, size_shift = None
)
___/ featurecoder_index: 0 \_________
=> The column(s) below matched your filter(s) and were ran through a test-encoding successfully.
['sepal_length', 'sepal_width', 'petal_length', 'petal_width']
=> Done. All feature column(s) have encoder(s) associated with them.
No more Featurecoders can be added to this Encoderset.
b) Sequence Dataset¶
The sequence dataset is a 3 dimensional structure intended for multi-observations per sample to enable time series analysis.
In order to perform supervised learning on sequence files, you’ll need both a Dataset.Sequence
and a Dataset.Tabular
:
Dataset.Sequence
is created from a homogenous 3d NumPy array.Dataset.Tabular
is created as seen in the section above. It must contain 1 row per sample.Then a
Splitset
is constructed using:The
Label
of theDataset.Tabular
.The
Featureset
of theDataset.Sequence
.
[5]:
df = datum.to_pandas('epilepsy.parquet')
[6]:
label_df = df[['seizure']]
[7]:
seq_ndarray3D = df.drop(columns=['seizure']).to_numpy().reshape(1000,178,1)
[8]:
from sklearn.preprocessing import StandardScaler
[9]:
seq_splitset = aiqc.Pipeline.Sequence.make(
seq_ndarray3D = seq_ndarray3D
, seq_feature_encoders = [{
"sklearn_preprocess": StandardScaler()
, "columns": '0'
}]
, tab_DF_or_path = label_df
, tab_label_column = 'seizure'
, size_test = 0.22
, size_validation = 0.12
)
⏱ Validating Sequences 🧬: 100%|████████████| 1000/1000 [00:00<00:00, 310183.70it/s]
⏱ Ingesting Sequences 🧬: 100%|█████████████████| 1000/1000 [00:10<00:00, 99.52it/s]
___/ featurecoder_index: 0 \_________
=> Info - System overriding user input to set `sklearn_preprocess.copy=False`.
This saves memory when concatenating the output of many encoders.
=> The column(s) below matched your filter(s) and were ran through a test-encoding successfully.
['0']
=> Done. All feature column(s) have encoder(s) associated with them.
No more Featurecoders can be added to this Encoderset.
c) Image Dataset¶
AIQC also supports image data and convolutional analysis.
In order to perform supervised learning on image files, you’ll need both a Dataset.Image
and a Dataset.Tabular
:
Dataset.Image
can be created from either a folder of images or a list of urls. The Pillow library is used to normalize images ingested into AIQC. Each image must be the same size (dimensions) and mode (colorscale).Dataset.Tabular
is created as seen in the section above. It must contain 1 row per image.Then a
Splitset
is constructed using:The
Label
of theDataset.Tabular
.The
Featureset
of theDataset.Image
.
Again, we’ll use the built-in data found in the datum
module that we imported above.
[6]:
df = datum.to_pandas(name='brain_tumor.csv')
image_urls = datum.get_remote_urls(manifest_name='brain_tumor.csv')
[7]:
img_splitset = aiqc.Pipeline.Image.make(
folderPath_or_urls = image_urls
, pillow_save = {}
, tabularDF_or_path = df
, tabular_dtype = None
, label_column = 'status'
, label_encoder = None
, size_test = 0.30
, size_validation = None
, fold_count = 4
, bin_count = None
)
🖼 Validating Images 🖼: 100%|███████████████████████| 80/80 [00:16<00:00, 4.83it/s]
🖼 Ingesting Images 🖼: 100%|████████████████████████| 80/80 [00:09<00:00, 8.36it/s]
2. Experiment¶
As seen in the Compatibility Matrix, the only library supported at this point in time is Keras
as it is the most straightforward for entry-level users.
You can find great examples of machine learning cookbooks on this blog: MachineLearningMastery.com “Multi-Label Classification”
[8]:
import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.callbacks import History
Model Functions¶
When we define our models, we’ll do so by wrapping each phase in the following functions:
fn_build
contains the topology/ layers.fn_train
specifies the samples and how the model should run.Optional and automatically determined:
fn_optimize
,fn_predict
andfn_lose
.
You can name the functions below whatever you want, but do not change their predetermined arugments (e.g. features_shape
, **hp
, model
, etc.). These items are used behind the scenes to pass the appropriate data, parameters, and models into your training jobs.
Because these are functions, we can even play with the topology as a parameter! As demonstrated by the if (hp['extra_layer'])
line below.
Put a placeholder anywhere you want to try out different hyperparameters:
hp['<some_variable_name>']
. You’ll get a chance to define the hyperparameters in a minute.
fn_build
¶
[9]:
def fn_build(features_shape, label_shape, **hp):
model = Sequential()
model.add(Dense(units=features_shape[0], activation='relu', kernel_initializer='he_uniform'))
model.add(Dropout(hp['dropout_size']))
if (hp['extra_layer']):
model.add(Dense(units=hp['neuron_count'], activation='relu', kernel_initializer='he_uniform'))
model.add(Dropout(hp['dropout_size']))
model.add(Dense(units=label_shape[0], activation='softmax', name='output'))
return model
fn_train
¶
[10]:
def fn_train(model, loser, optimizer, samples_train, samples_evaluate, **hp):
model.compile(
optimizer=optimizer,
loss='categorical_crossentropy',
metrics=['accuracy']
)
model.fit(
samples_train["features"]
, samples_train["labels"]
, validation_data = (
samples_evaluate["features"]
, samples_evaluate["labels"]
)
, verbose = 0
, batch_size = 3
, epochs = hp['epoch_count']
, callbacks=[History()]
)
return model
Reference the low-level API documentation for information on the custom ‘early stopping’ callbacks AIQC makes available.
Hyperparameters¶
The hyperparameters
below will be automatically fed into the functions above as **kwargs
via the **hp
argument we saw earlier.
For example, wherever you see hp['neuron_count']
, it will pull from the key:value pair "neuron_count": [9, 12]
seen below. Where model A will have 9 neurons and model B will have 12 neurons.
[11]:
hyperparameters = {
"neuron_count": [9, 12]
, "extra_layer": [True, False]
, "dropout_size": [0.10, 0.20]
, "epoch_count": [50]
, "learning_rate": [0.01]
}
Then pass these functions into the Algorithm
.
The library
and analysis_type
help handle the model and its output behind the scenes. Current analysis types include: ‘classification_multi’, ‘classification_binary’, and ‘regression’.
Experiment.make()
¶
Now it’s time to bring together the data and logic into an Experiment
.
[13]:
queue = aiqc.Experiment.make(
library = "keras"
, analysis_type = "classification_multi"
, fn_build = fn_build
, fn_train = fn_train
, splitset_id = splitset.id
, repeat_count = 2
, hide_test = False
, hyperparameters = hyperparameters
, foldset_id = None
, fn_lose = None #automated.
, fn_optimize = None #automated.
, fn_predict = None #automated.
, pick_count = None
, pick_percent = None
)
[14]:
queue.run_jobs()
🔮 Training Models 🔮: 100%|████████████████████████████████████████| 16/16 [01:11<00:00, 4.45s/it]
For more information on visualization of performance metrics, reference the Visualization & Metrics documentation.


--- layout: default ---
The Google group is a public mailing list that anyone can join to ask questions about or otherwise discuss Jsonnet. We also use the list for release announcements and other relevant communications to Jsonnet users.
There is also an IRC channel: #jsonnet on the freenode network (chat.freenode.net).
All Jsonnet implementations are licensed under the Apache 2.0 license.
If you wish to contribute a patch to Jsonnet, do keep us informed as we may be able to help. Please create a fork on Github and do your work there. Before we can merge your pull request, we need you to sign either the Google individual or corporate contributor license agreement (CLA), unless you are a Google employee, intern, or contractor. CLAs are a common requirement when patching open source software projects owned by large companies.


Register An Account
First Name:
Last Name:
Email:
Username:
Phone Number:
PPS Number:
Password:
Confirm Password:
Register
Log In
Chat
Live Chat
Message
Send
Close


WELCOME TO MY PORTFOLIO!!!
Iam a student at Moringa school. I'm taking a class on progrmming since iam interested in making the world a better place ofcourse with the help of computers. Times are changing very fast and with time we should also understand the future of technology and how to develop various parts of the country and the world. Here's a link of one website that I have worked on it is still a working progress.


Framingham data
Table of Contents
#title Framingham data
<contents>
Description
Planned as a 20 years cohort study of residents aged 30-59 in Framingham town, Massachusetts, in 1948. Here: Data of 1406 persons (age > 44) and CHOL not missing at exam 1.
Variable Names
In order from left to right: 'data.frame': 1406 obs. of 9 variables:
| Variable | Explanation (Unit) |
|---|---|
| ID | subject id |
| SEX | gender (1 for males, 2 for females) |
| AGE | Age in years |
| FRW | ``Framingham relative weight'' (pct.) at baseline (52-222; 11 persons have missing values) |
| SBP | systolic blood pressure at baseline mmHg (90-300) |
| DBP | diastolic blood pressure at baseline mmHg (50-160) |
| CHOL | cholesterol at baseline mg/100ml (96-430) |
| CIG | cigarettes per day at baseline (0-60; 1 person has missing value) |
| CHD | 0 if no ``coronary heart disease'' during follow-up, 1 if ``coronary heart disease'' at baseline (prevalent cases), x=2-10 if ``coronary heart disease'' was diagnosed at follow-up no. x |
Download
Format || Download text, space separated | Framingham.txt


Congratulations! Your browser has the necessary requirements to play the game!
Please be patient. We still need connect
...
Connected players:
You can contact the game master using the chat window below.