Sphinx-4 Application Programmer's Guide |
This tutorial shows you how to write Sphinx-4 applications. We will use the HelloDigits demo as an example to show how a simple application can be written. We will then proceed to a more complex example. Consequently, this tutorial is divided into the following parts:
We will look at a very simple Sphinx-4 speech application, the HelloDigits demo. This application recognizes connected digits. As you will see, the code is very simple. The harder part is understanding the configuration, but we will guide you through every step of it. Lets look at the code first.
All the source code of the HelloDigits demo is in one short file
sphinx4/demo/sphinx/hellodigits/HelloDigits.java
:
/* * Copyright 2004 Carnegie Mellon University. * Portions Copyright 2004 Sun Microsystems, Inc. * Portions Copyright 2004 Mitsubishi Electric Research Laboratories. * All Rights Reserved. Use is subject to license terms. * * See the file "license.terms" for information on usage and * redistribution of this file, and for a DISCLAIMER OF ALL * WARRANTIES. * */ package demo.sphinx.hellodigits; import edu.cmu.sphinx.frontend.util.Microphone; import edu.cmu.sphinx.recognizer.Recognizer; import edu.cmu.sphinx.result.Result; import edu.cmu.sphinx.util.props.ConfigurationManager; import edu.cmu.sphinx.util.props.PropertyException; import java.io.File; import java.io.IOException; import java.net.URL; /** * A simple HelloDigits demo showing a simple speech application * built using Sphinx-4. This application uses the Sphinx-4 endpointer, * which automatically segments incoming audio into utterances and silences. */ public class HelloDigits { /** * Main method for running the HelloDigits demo. */ public static void main(String[] args) { try { URL url; if (args.length > 0) { url = new File(args[0]).toURI().toURL(); } else { url = HelloDigits.class.getResource("hellodigits.config.xml"); } ConfigurationManager cm = new ConfigurationManager(url); Recognizer recognizer = (Recognizer) cm.lookup("recognizer"); Microphone microphone = (Microphone) cm.lookup("microphone"); /* allocate the resource necessary for the recognizer */ recognizer.allocate(); /* the microphone will keep recording until the program exits */ if (microphone.startRecording()) { System.out.println ("Say any digit(s): e.g. \"two oh oh four\", " + "\"three six five\"."); while (true) { System.out.println ("Start speaking. Press Ctrl-C to quit.\n"); /* * This method will return when the end of speech * is reached. Note that the endpointer will determine * the end of speech. */ Result result = recognizer.recognize(); if (result != null) { String resultText = result.getBestResultNoFiller(); System.out.println("You said: " + resultText + "\n"); } else { System.out.println("I can't hear what you said.\n"); } } } else { System.out.println("Cannot start microphone."); recognizer.deallocate(); System.exit(1); } } catch (IOException e) { System.err.println("Problem when loading HelloDigits: " + e); e.printStackTrace(); } catch (PropertyException e) { System.err.println("Problem configuring HelloDigits: " + e); e.printStackTrace(); } catch (InstantiationException e) { System.err.println("Problem creating HelloDigits: " + e); e.printStackTrace(); } } }
edu.cmu.sphinx.recognizer.Recognizer
edu.cmu.sphinx.result.Result
edu.cmu.sphinx.util.props.ConfigurationManager
The Recognizer
is the main class any application should
interact with (refer also to the architecture diagram above).
The Result
is returned by the Recognizer to the application
after recognition completes. The ConfigurationManager
creates the entire Sphinx-4 system according to the configuration specified
by the user.
Lets look at the main()
method. The first few
lines creates the URL of the XML-based configuration file.
A ConfigurationManager
is then created using that URL.
The ConfigurationManager then reads in the
file internally. Since the configuration file specifies the components
"recognizer"
and "microphone"
(we will look
at the configuration file next),
we perform a lookup()
in the ConfigurationManager to obtain these components. The allocate()
method of the Recognizer is then called to allocate the resources
need for the recognizer. The Microphone class is used for capturing live
audio from the system audio device. Both the Recognizer and the Microphone
is configured as specified in the configuration file.
Once all the necessary components are created, we can start running the demo.
The program first turns on the Microphone
(microphone.startRecording()
).
After the microphone is turned on successfully,
the program enters a loop that repeats the following. It tries to recognize
what the user is saying, using the
Recognizer.recognize()
method. Recognition stops when
the user stops speaking, which is detected by the endpointer built into
the front end by configuration. Once an utterance is recognized,
the recognized text, which is returned by the method
Result.getBestResultNoFiller()
, is printed out.
If the Recognizer recognized nothing (i.e., result
is
null), then it will print out a message saying that. Finally, if the
demo program cannot turn on the microphone in the first place, the
Recognizer will be deallocated, and the program exits.
It is generally a good practice to call the method
deallocate()
after the work is done to release all the
resources.
Note that several exceptions were being caught. First of all,
the IOException is thrown by the constructor
of the ConfigurationManager and the Recognizer.allocate() method.
The PropertyException
is thrown again by the constructor, and by the
lookup()
method, of the ConfigurationManager.
These exceptions should be caught and handled appropriately.
Hopefully, by this point, you will have some idea of how to write a simple Sphinx-4 application. We will now turn to the harder part, understanding the various components necessary to create a connected-digits recognizer. These components are specified in the configuration file, which we will now explain in depth.
In this section, we will explain the various Sphinx-4 components that are used for the HelloDigits demo, as specified in the configuration file. We will look at each section of the config file in depth. If you want to learn about the format of these configuration files, please refer to the document Sphinx-4 Configuration Management.
The lines below define the frequently tuned properties. They are located at the top of the configuration file so that they can be edited quickly.
<!-- ******************************************************** --> <!-- frequently tuned properties --> <!-- ******************************************************** --> <property name="logLevel" value="WARNING"/> <property name="absoluteBeamWidth" value="-1"/> <property name="relativeBeamWidth" value="1E-80"/> <property name="wordInsertionProbability" value="1E-36"/> <property name="languageWeight" value="8"/> <property name="frontend" value="epFrontEnd"/> <property name="recognizer" value="recognizer"/> <property name="showCreations" value="false"/>
The lines below define the recognizer component that performs speech recognition. It defines the name and class of the recognizer, Recognizer. This is the class that any application should interact with. If you look at the javadoc of the Recognizer class, you will see that it has two properties, 'decoder' and 'monitors'. This configuration file is where the value of these properties are defined.
<!-- ******************************************************** --> <!-- word recognizer configuration --> <!-- ******************************************************** --> <component name="recognizer" type="edu.cmu.sphinx.recognizer.Recognizer"> <property name="decoder" value="decoder"/> <propertylist name="monitors"> <item>accuracyTracker </item> <item>speedTracker </item> <item>memoryTracker </item> </propertylist> </component>
We will explain the monitors later. For now, lets look at the decoder.
<component name="decoder" type="edu.cmu.sphinx.decoder.Decoder"> <property name="searchManager" value="searchManager"/> </component>The decoder component is of class
edu.cmu.sphinx.decoder.Decoder
. Its property 'searchManager'
is set to the component 'searchManager', defined as:
<component name="searchManager" type="edu.cmu.sphinx.decoder.search.SimpleBreadthFirstSearchManager"> <property name="logMath" value="logMath"/> <property name="linguist" value="flatLinguist"/> <property name="pruner" value="trivialPruner"/> <property name="scorer" value="threadedScorer"/> <property name="activeListFactory" value="activeList"/> </component>The searchManager is of class
edu.cmu.sphinx.decoder.search.SimpleBreadthFirstSearchManager
.
This class performs a simple breadth-first search through the search graph
during the decoding process to find the best path. This search manager
is suitable for small to medium sized vocabulary decoding.
The logMath property is the
log math that is used for calculation of scores during the search process.
It is defined as having the log base of 1.0001. Note that typically the
same log base should be used throughout all components, and therefore
there should only be one logMath definition in a configuration file:
<component name="logMath" type="edu.cmu.sphinx.util.LogMath"> <property name="logBase" value="1.0001"/> <property name="useAddTable" value="true"/> </component>The linguist of the searchManager is set to the component 'flatLinguist' (which we will look at later), which again is suitable for small to medium sized vocabulary decoding. The pruner is set to the 'trivialPruner':
<component name="trivialPruner" type="edu.cmu.sphinx.decoder.pruner.SimplePruner"/>which is of class
edu.cmu.sphinx.decoder.pruner.SimplePruner
.
This pruner performs simple absolute beam and relative beam pruning
based on the scores of the tokens.
The scorer of the searchManager is set to the component 'threadedScorer',
which is of class
edu.cmu.sphinx.decoder.scorer.ThreadedAcousticScorer
.
It can use multiple threads (usually one per CPU) to score the tokens
in the active list. Scoring is one of the most time-consuming step of the
decoding process. Tokens can be scored independently of each other,
so using multiple CPUs will definitely speed things up.
The threadedScorer is defined as follows:
<component name="threadedScorer" type="edu.cmu.sphinx.decoder.scorer.ThreadedAcousticScorer"> <property name="frontend" value="${frontend}"/> <property name="isCpuRelative" value="true"/> <property name="numThreads" value="0"/> <property name="minScoreablesPerThread" value="10"/> <property name="scoreablesKeepFeature" value="true"/> </component>The 'frontend' property is the front end from which features are obtained. For details about the other properties of the threadedScorer, please refer to javadoc for ThreadedAcousticScorer. Finally, the activeListFactory property of the searchManager is set to the component 'activeList', which is defined as follows:
<component name="activeList" type="edu.cmu.sphinx.decoder.search.PartitionActiveListFactory"< <property name="logMath" value="logMath"/< <property name="absoluteBeamWidth" value="${absoluteBeamWidth}"/< <property name="relativeBeamWidth" value="${relativeBeamWidth}"/< </component<It is of class
edu.cmu.sphinx.decoder.search.PartitionActiveListFactory
.
It uses a partitioning algorithm to select the top N highest scoring
tokens when performing absolute beam pruning. The 'logMath' property
specifies the logMath used for score calculation, which is the same
LogMath used in the searchManager. The property 'absoluteBeamWidth'
is set to the value given at the very top of the configuration file using
${absoluteBeamWidth}
. The same is for 'relativeBeamWidth'.
<component name="flatLinguist" type="edu.cmu.sphinx.linguist.flat.FlatLinguist"> <property name="logMath" value="logMath"/> <property name="grammar" value="jsgfGrammar"/> <property name="acousticModel" value="wsj"/> <property name="wordInsertionProbability" value="${wordInsertionProbability}"/> <property name="languageWeight" value="${languageWeight}"/> </component>It also uses the logMath that we've seen already. The grammar used is the component called 'jsgfGrammar', which is a BNF-style grammar:
<component name="jsgfGrammar" type="edu.cmu.sphinx.jsapi.JSGFGrammar"> <property name="grammarLocation" value="resource:/demo.sphinx.helloworld.HelloWorld!/demo/sphinx/hellodigits/"/> <property name="dictionary" value="dictionary"/> <property name="grammarName" value="digits"/> <property name="logMath" value="logMath"/> </component>
JSGF grammars are defined in JSAPI. The class that translates JSGF into a form that
Sphinx-4 understands is
edu.cmu.sphinx.jsapi.JSGFGrammar
(NOTE: that this link to the javadoc also
describes the limitations of the current implementation).
The property 'grammarLocation' can take two kinds of values.
If it is a URL, it specifies the URL of the directory where JSGF grammar files
are to be found. Otherwise, it is interpreted as resource locator.
In our example, the HelloDigits demo is being deployed as a JAR file.
The 'grammarLocation' property is therefore used to specify
the location of the resource
digits.gram
within the JAR file. Note that it is not necessary to the JAR file
within which to search. The system searches the JAR file in which the class
demo.sphinx.helloworld.HelloWorld
resides.
The 'grammarName' property specifies the grammar to use when creating
the search graph. 'logMath' is the same log math as the other components.
The 'dictionary' is the component that maps words to their phonemes.
It is almost always the dictionary of the
acoustic model, which lists all the words that were used to trained
the acoustic model:
<component name="dictionary" type="edu.cmu.sphinx.linguist.dictionary.FastDictionary"> <property name="dictionaryPath" value="resource:/edu.cmu.sphinx.model.acoustic.TIDIGITS_8gau_13dCep_16k_40mel_130Hz_6800Hz!/edu/cmu/sphinx/model/acoustic/dictionary"/> <property name="fillerPath" value="resource:/edu.cmu.sphinx.model.acoustic.TIDIGITS_8gau_13dCep_16k_40mel_130Hz_6800Hz!/edu/cmu/sphinx/model/acoustic/fillerdict"/> <property name="addSilEndingPronunciation" value="false"/> <property name="wordReplacement" value="<sil>"/> <property name="allowMissingWords" value="true"/> </component>
The locations of these dictionary files are specified using the Sphinx-4 resource mechanism. In short, this mechanism locates the JAR file in which a class resides, and looks into that JAR file for the desired resource. The syntax is:
resource:/{name of class to locate the JAR file}!{location in the JAR file of the desired resource}Take the 'dictionaryPath' property, for example. The "name of class to locate the JAR file" is "
edu.cmu.sphinx.model.acoustic.TIDIGITS_8gau_13dCep_16k_40mel_130Hz_6800Hz
", while the "location in the JAR file of the desired resource" is "/edu/cmu/sphinx/model/acoustic/dictionary
".
The dictionary for filler words like "BREATH" and "LIP_SMACK" is the file
fillerdict
.
For details about the other properties, please refer to the javadoc for FastDictionary.
The next important property of the flatLinguist is the acoustic model, which is defined as:
<component name="tidigits" type="edu.cmu.sphinx.model.acoustic.TIDIGITS_8gau_13dCep_16k_40mel_130Hz_6800Hz"> <property name="loader" value="sphinx3Loader"/> </component> <component name="sphinx3Loader" type="edu.cmu.sphinx.model.acoustic.TIDIGITSLoader"> >property name="logMath" value="logMath"/> </component>
'tidigits' stands for the TIDIGITS acoustic models. In Sphinx-4,
different acoustic models are represented by classes that implement the
AcousticModel interface. The implementation class, together with the
actual model data files, are packaged in a JAR file, which is included
in the classpath of the Sphinx-4 application. This JAR file for the TIDIGITS
models is called TIDIGITS_8gau_13dCep_16k_40mel_130Hz_6800Hz.jar
,
and is in the sphinx4/lib
directory. Inside this JAR file
is a class called edu.cmu.sphinx.model.acoustic.TIDIGITS_8gau_13dCep_16k_40mel_130Hz_6800Hz
, which implements the AcousticModel interface.
This class will automatically load up all the actual model data files when
it is loaded. The loader of this AcousticModel is called
TIDIGITSLoader
, and it also is inside the JAR file.
As a programmer, all you need to do is to specify the class of the
AcousticModel, and the loader of the AcousticModel, as shown above
(note that if you are using the TIDIGITS model in other applications,
these lines should be the same, except that you might have called your
'logMath' component something else).
| Vocabulary Size | Word Insertion Probability | Language Weight |
| Digits (11 words - TIDIGITS) | 1E-36 | 8 |
| Small (80 words - AN4) | 1E-26 | 7 |
| Medium (1000 words - RM1) | 1E-10 | 7 |
| Large (64000 words - HUB4) | 0.2 | 10.5 |
<!-- ******************************************************** --> <!-- The frontend configuration --> <!-- ******************************************************** --> <component name="frontEnd" type="edu.cmu.sphinx.frontend.FrontEnd"> <propertylist name="pipeline"> <item>microphone </item> <item>premphasizer </item> <item>windower </item> <item>fft </item> <item>melFilterBank </item> <item>dct </item> <item>liveCMN </item> <item>featureExtraction </item> </propertylist> </component> <!-- ******************************************************** --> <!-- The live frontend configuration --> <!-- ******************************************************** --> <component name="epFrontEnd" type="edu.cmu.sphinx.frontend.FrontEnd"> <propertylist name="pipeline"> <item>microphone </item> <item>speechClassifier </item> <item>speechMarker </item> <item>nonSpeechDataFilter </item> <item>premphasizer </item> <item>windower </item> <item>fft </item> <item>melFilterBank </item> <item>dct </item> <item>liveCMN </item> <item>featureExtraction </item> </propertylist> </component>As you might notice, the only different between these two front ends is that the live front end (epFrontEnd) has the additional components
speechClassifier
, speechMarker
and
nonSpeechDataFilter
. These three components make up the
default endpointer of Sphinx-4. Below is a listing of all the components
of both front ends, and those properties which have values different from
the default:
<component name="speechClassifier" type="edu.cmu.sphinx.frontend.endpoint.SpeechClassifier"> <property name="threshold" value="13"/> </component> <component name="nonSpeechDataFilter" type="edu.cmu.sphinx.frontend.endpoint.NonSpeechDataFilter"/> <component name="speechMarker" type="edu.cmu.sphinx.frontend.endpoint.SpeechMarker" > <property name="speechTrailer" value="50"/> </component> <component name="premphasizer" type="edu.cmu.sphinx.frontend.filter.Preemphasizer"/> <component name="windower" type="edu.cmu.sphinx.frontend.window.RaisedCosineWindower"> </component> <component name="fft" type="edu.cmu.sphinx.frontend.transform.DiscreteFourierTransform"/> <component name="melFilterBank" type="edu.cmu.sphinx.frontend.frequencywarp.MelFrequencyFilterBank"> </component> <component name="dct" type="edu.cmu.sphinx.frontend.transform.DiscreteCosineTransform"/> <component name="batchCMN" type="edu.cmu.sphinx.frontend.feature.BatchCMN"/> <component name="liveCMN" type="edu.cmu.sphinx.frontend.feature.LiveCMN"/> <component name="featureExtraction" type="edu.cmu.sphinx.frontend.feature.DeltasFeatureExtractor"/> <component name="microphone" type="edu.cmu.sphinx.frontend.util.Microphone"> <property name="msecPerRead" value="10"/> <property name="closeBetweenUtterances" value="false"/> </component>Lets explain some of the properties set here that have values different from the default. The property 'threshold' of the SpeechClassifier specifies the minimum difference between the input signal level and the background signal level in order that the input signal is classified as speech. Therefore, the smaller this number, the more sensitive the endpointer, and vice versa. The property 'speechTrailer' of the SpeechMarker specifies the length of non-speech signal to be included after the end of speech to make sure that no speech signal is lost. Here, it is set at 50 milliseconds. The property 'msecPerRead' of the Microphone specifies the number of milliseconds of data to read at a time from the system audio device. The value specified here is 10ms. The property 'closeBetweenUtterances' specifies whether the system audio device should be released between utterances. It is set to false here, meaning that the system audio device will not be released between utterances. This is set as so because on certain systems (Linux for one), closing and reopening the audio does not work too well.
<component name="accuracyTracker" type="edu.cmu.sphinx.instrumentation.AccuracyTracker"> <property name="recognizer" value="${recognizer}"/> <property name="showAlignedResults" value="false"/> <property name="showRawResults" value="false"/> </component> <component name="memoryTracker" type="edu.cmu.sphinx.instrumentation.MemoryTracker"> <property name="recognizer" value="${recognizer}"/> <property name="showSummary" value="false"/> <property name="showDetails" value="false"/> </component> <component name="speedTracker" type="edu.cmu.sphinx.instrumentation.SpeedTracker"> <property name="recognizer" value="${recognizer}"/> <property name="frontend" value="${frontend}"/> <property name="showSummary" value="true"/> <property name="showDetails" value="false"/> </component>
The various knobs of these monitors mainly control whether statistical information about accuracy, speed and memory usage should be printed out. Moreover, the monitors monitor the behavior of a recognizer, so they need a reference to the recognizer that they are monitoring.
HelloDigits uses a very small vocabulary and a guided grammar. What if you want to use a larger vocabulary, and there is no guided grammar for your application? One way to do it would be to use what is known as a language model, which describes the probability of occurrence of a series of words. The HelloNGram demo shows you how to do this with Sphinx-4.
The source code for the HelloNGram demo is exactly the same as that of the HelloDigits demo, except for the names of the demo class. The demo runs exactly the same way: it keeps listening to and recognizes what you say, and when it was detected the end of an utterance, it will show the recognition result.
Sphinx-4 supports the n-gram language model (both ascii and binary versions) generated by the Carnegie Mellon University Statistical Language Modeling toolkit. The input file is a long list of sample utterances. Using the occurrence of words and sequences of words in this input file, a language model can be trained. The resulting trigram language model file is hellongram.trigram.lm.
<!-- ******************************************************** --> <!-- frequently tuned properties --> <!-- ******************************************************** --> <property name="absoluteBeamWidth" value="500"/> <property name="relativeBeamWidth" value="1E-80"/> <property name="absoluteWordBeamWidth" value="20"/> <property name="relativeWordBeamWidth" value="1E-60"/> <property name="wordInsertionProbability" value="1E-16"/> <property name="languageWeight" value="7.0"/> <property name="silenceInsertionProbability" value=".1"/> <property name="frontend" value="epFrontEnd"/> <property name="recognizer" value="recognizer"/> <property name="showCreations" value="false"/>The above lines defines frequently tuned properties. They are located at the top of the configuration file so that they can be edited quickly.
<component name="recognizer" type="edu.cmu.sphinx.recognizer.Recognizer"> <property name="decoder" value="decoder"/> <propertylist name="monitors"> <item>accuracyTracker </item> <item>speedTracker </item> <item>memoryTracker </item> <item>recognizerMonitor </item> </propertylist> </component>The above lines define the recognizer component that performs speech recognition. It defines the name and class of the recognizer. This is the class that any application should interact with. If you look at the javadoc of the Recognizer class, you will see that it has two properties, 'decoder' and 'monitors'. This configuration file is where the value of these properties are defined.
<component name="decoder" type="edu.cmu.sphinx.decoder.Decoder"> <property name="searchManager" value="wordPruningSearchManager"/> <property name="featureBlockSize" value="50"/> </component>The decoder component is defined to be of class
edu.cmu.sphinx.decoder.Decoder
. Its property 'searchManager'
is set to the component 'wordPruningSearchManager':
<component name="wordPruningSearchManager" type="edu.cmu.sphinx.decoder.search.WordPruningBreadthFirstSearchManager"> <property name="logMath" value="logMath"/> <property name="linguist" value="lexTreeLinguist"/> <property name="pruner" value="trivialPruner"/> <property name="scorer" value="threadedScorer"/> <property name="activeListManager" value="activeListManager"/> <property name="growSkipInterval" value="0"/> <property name="checkStateOrder" value="false"/> <property name="buildWordLattice" value="false"/> <property name="acousticLookaheadFrames" value="1.7"/> <property name="relativeBeamWidth" value="${relativeBeamWidth}"/> </component>The searchManager is of class
edu.cmu.sphinx.decoder.search.WordPruningBreadthFirstSearchManager
. It is better than the SimpleBreadthFirstSearchManager
for larger vocabulary recognition. This class also performs a simple
breadth-first search through the search graph, but at each frame it also
prunes the different types of states separately. The logMath property is the
log math that is used for calculation of scores during the search process.
It is defined as having the log base of 1.0001. Note that typically the
same log base should be used throughout all components, and therefore
there should only be one logMath definition:
<component name="logMath" type="edu.cmu.sphinx.util.LogMath"> <property name="logBase" value="1.0001"/> <property name="useAddTable" value="true"/> </component>The linguist of the searchManager is set to the component 'lexTreeLinguist' (which we will look at later), which again is suitable for large vocabulary recognition. The pruner is set to the 'trivialPruner':
<component name="trivialPruner" type="edu.cmu.sphinx.decoder.pruner.SimplePruner"/>which is of class
edu.cmu.sphinx.decoder.pruner.SimplePruner
.
This pruner performs simple absolute beam and relative beam pruning
based on the scores of the tokens.
The scorer of the searchManager is set to the component 'threadedScorer',
which is of class
edu.cmu.sphinx.decoder.scorer.ThreadedAcousticScorer
.
It can use multiple threads (usually one per CPU) to score the tokens
in the active list. Scoring is one of the most time-consuming step of the
decoding process. Tokens can be scored independently of each other,
so using multiple CPUs will definitely speed things up.
The threadedScorer is defined as follows:
<component name="threadedScorer" type="edu.cmu.sphinx.decoder.scorer.ThreadedAcousticScorer"> <property name="frontend" value="${frontend}"/> <property name="isCpuRelative" value="true"/> <property name="numThreads" value="0"/> <property name="minScoreablesPerThread" value="10"/> <property name="scoreablesKeepFeature" value="true"/> </component>The 'frontend' property is the front end from which features are obtained. For details about the other properties of the threadedScorer, please refer to javadoc for ThreadedAcousticScorer. Finally, the 'activeListManager' property of the wordPruningSearchManager is set to the component 'activeListManager', which is defined as follows:
<component name="activeListManager" type="edu.cmu.sphinx.decoder.search.SimpleActiveListManager"> &propertylist name="activeListFactories"> <item>standardActiveListFactory</item> <item>wordActiveListFactory</item> <item>wordActiveListFactory</item> <item>standardActiveListFactory</item> <item>standardActiveListFactory</item> <item>standardActiveListFactory</item> </propertylist> </component> <component name="standardActiveListFactory" type="edu.cmu.sphinx.decoder.search.PartitionActiveListFactory"> <property name="logMath" value="logMath"/> <property name="absoluteBeamWidth" value="${absoluteBeamWidth}"/> <property name="relativeBeamWidth" value="${relativeBeamWidth}"/> </component> <component name="wordActiveListFactory" type="edu.cmu.sphinx.decoder.search.PartitionActiveListFactory"> <property name="logMath" value="logMath"/> <property name="absoluteBeamWidth" value="${absoluteWordBeamWidth}"/> <property name="relativeBeamWidth" value="${relativeWordBeamWidth}"/> </component>The SimpleActiveListManager is of class
edu.cmu.sphinx.decoder.search.SimpleActiveListManager
.
Since the word-pruning search manager performs pruning on different
search state types separately, we need a different active list for each
state type. Therefore, you see different active list factories being listed
in the SimpleActiveListManager, one for each type. So how do we know which
active list factory is for which state type? It depends on the 'search order'
as returned by the search graph (which in this case is generated by the
LexTreeLinguist). The search state order and active list factory used here are:
| State Type | ActiveListFactory |
| LexTreeNonEmittingHMMState | standardActiveListFactory |
| LexTreeWordState | wordActiveListFactory |
| LexTreeEndWordState | wordActiveListFactory |
| LexTreeEndUnitState | standardActiveListFactory |
| LexTreeUnitState | standardActiveListFactory |
| LexTreeHMMState | standardActiveListFactory |
<component name="lexTreeLinguist" type="edu.cmu.sphinx.linguist.lextree.LexTreeLinguist"> <property name="logMath" value="logMath"/> <property name="acousticModel" value="wsj"/> <property name="languageModel" value="trigramModel"/> <property name="dictionary" value="dictionary"/> <property name="addFillerWords" value="false"/> <property name="fillerInsertionProbability" value="1E-10"/> <property name="generateUnitStates" value="false"/> <property name="wantUnigramSmear" value="true"/> <property name="unigramSmearWeight" value="1"/> <property name="wordInsertionProbability" value="${wordInsertionProbability}"/> <property name="silenceInsertionProbability" value="${silenceInsertionProbability}"/> <property name="languageWeight" value="${languageWeight}"/> </component>
For details about the LexTreeLinguist, please refer to the Javadocs of the LexTreeLinguist. In general, the LexTreeLinguist is the one to use for large vocabulary speech recognition, and the FlatLinguist is the one to use for small vocabulary speech recognition. The LexTreeLinguist has a lot of properties that can be set, but the ones that are must be set are the 'logMath', the 'acousticModel', the 'languageModel', and the 'dictionary'. These properties are the necessary sources of information for the LexTreeLinguist to build the search graph. The rest of the properties are for controlling the speed and accuracy performance of the linguist, and you can read more about them in the Javadocs of the LexTreeLinguist.
The 'acousticModel' is where the LexTreeLinguist obtains the HMM for the words or units. For the HelloNGram demo, it is defined as:
<component name="wsj" type="edu.cmu.sphinx.model.acoustic.WSJ_8gau_13dCep_16k_40mel_130Hz_6800Hz"> <property name="loader" value="wsjLoader"/> </component> <component name="wsjLoader" type="edu.cmu.sphinx.model.acoustic.WSJLoader"> <property name="logMath" value="logMath"/> </component>
'wsj' stands for the Wall Street Journal acoustic models. In Sphinx-4,
different acoustic models are represented by classes that implement the
AcousticModel interface. The implementation class, together with the
actual model data files, are packaged in a JAR file, which is included
in the classpath of the Sphinx-4 application. The JAR file for the WSJ
models is called WSJ_8gau_13dCep_16k_40mel_130Hz_6800Hz.jar
,
and is in the sphinx4/lib
directory. Inside this JAR file
is a class called edu.cmu.sphinx.model.acoustic.WSJ_8gau_13dCep_16k_40mel_130Hz_6800Hz
, which implements the AcousticModel interface.
This class will automatically load up all the actual model data files when
it is loaded. The loader of this AcousticModel is called
WSJLoader
, and it also is inside the JAR file. As a programmer,
all you need to do is to specify the class of the AcousticModel, and the
loader of the AcousticModel, as shown above (note that if you are using
the WSJ model in other applications, these lines should be the same,
except that you might have called your 'logMath' component something else).
<component name="trigramModel" type="edu.cmu.sphinx.linguist.language.ngram.SimpleNGramModel"> <property name="location" value="resource:/demo.sphinx.hellongram.HelloNGram!/demo/sphinx/hellongram/hellongram.trigram.lm"/> <property name="logMath" value="logMath"/> <property name="dictionary" value="dictionary"/> <property name="maxDepth" value="3"/> <property name="unigramWeight" value=".7"/> </component>
The language model is generated by the CMU Statistical Language Modeling Toolkit. It is in text format, which can be loaded by the SimpleNGramModel class. For this class, you also need to specify the dictionary that you are using, which is the same as the one used by the lexTreeLinguist. Same for 'logMath' (note that the same logMath component should be used throughout the system). The 'maxDepth' property is 3, since this is a trigram language model. The 'unigramWeight' should normally be set to 0.7.
The last important component of the LexTreeLinguist is the 'dictionary', which is defined as follows:
<component name="dictionary" type="edu.cmu.sphinx.linguist.dictionary.FastDictionary"> <property name="dictionaryPath" value= value="resource:/edu.cmu.sphinx.model.acoustic.WSJ_8gau_13dCep_16k_40mel_130Hz_6800Hz!/edu/cmu/sphinx/model/acoustic/dict/cmudict.0.6d"/> <property name="fillerPath" value="resource:/edu.cmu.sphinx.model.acoustic.WSJ_8gau_13dCep_16k_40mel_130Hz_6800Hz!/edu/cmu/sphinx/model/acoustic/dict/fillerdict"/> <property name="addSilEndingPronunciation" value="false"/> <property name="wordReplacement" value="<sil>"/> </component>
As you might realize, it is using the dictionary inside the JAR file of the
Wall Street journal acoustic model. The main dictionary for words is the
edu/cmu/sphinx/model/acoustic/dict/cmudict.0.6d
file
inside the JAR file, and the dictionary for filler words
like "BREATH" and "LIP_SMACK" is
edu/cmu/sphinx/model/acoustic/dict/fillerdict
. You can
inspect the contents of a JAR file by (assuming your JAR file is called
myJar.jar)
jar tvf myJar.jarYou can see the contents of the WSJ JAR file by:
sphinx4> jar tvf lib/WSJ_8gau_13dCep_16k_40mel_130Hz_6800Hz.jar 0 Fri May 07 10:50:26 EDT 2004 META-INF/ 219 Fri May 07 10:50:24 EDT 2004 META-INF/MANIFEST.MF 0 Fri May 07 10:50:20 EDT 2004 edu/ 0 Fri May 07 10:50:20 EDT 2004 edu/cmu/ 0 Fri May 07 10:50:20 EDT 2004 edu/cmu/sphinx/ 0 Fri May 07 10:50:20 EDT 2004 edu/cmu/sphinx/model/ 0 Fri May 07 10:50:24 EDT 2004 edu/cmu/sphinx/model/acoustic/ 262 Fri May 07 10:50:16 EDT 2004 edu/cmu/sphinx/model/acoustic/WSJLoader.class 688 Fri May 07 10:50:16 EDT 2004 edu/cmu/sphinx/model/acoustic/WSJ_8gau_13dCep_16k_40mel_130Hz_6800Hz.class 0 Fri May 07 10:50:24 EDT 2004 edu/cmu/sphinx/model/acoustic/cd_continuous_8gau/ 0 Fri May 07 10:50:24 EDT 2004 edu/cmu/sphinx/model/acoustic/dict/ 0 Fri May 07 10:50:22 EDT 2004 edu/cmu/sphinx/model/acoustic/etc/ 1492 Fri May 07 10:50:22 EDT 2004 edu/cmu/sphinx/model/acoustic/README 5175518 Fri May 07 10:50:22 EDT 2004 edu/cmu/sphinx/model/acoustic/cd_continuous_8gau/means 132762 Fri May 07 10:50:24 EDT 2004 edu/cmu/sphinx/model/acoustic/cd_continuous_8gau/mixture_weights 2410 Fri May 07 10:50:24 EDT 2004 edu/cmu/sphinx/model/acoustic/cd_continuous_8gau/transition_matrices 5175518 Fri May 07 10:50:22 EDT 2004 edu/cmu/sphinx/model/acoustic/cd_continuous_8gau/variances 353 Fri May 07 10:50:24 EDT 2004 edu/cmu/sphinx/model/acoustic/dict/alpha.dict 4718935 Fri May 07 10:50:24 EDT 2004 edu/cmu/sphinx/model/acoustic/dict/cmudict.0.6d 373 Fri May 07 10:50:22 EDT 2004 edu/cmu/sphinx/model/acoustic/dict/digits.dict 204 Fri May 07 10:50:22 EDT 2004 edu/cmu/sphinx/model/acoustic/dict/fillerdict 5654967 Fri May 07 10:50:24 EDT 2004 edu/cmu/sphinx/model/acoustic/etc/WSJ_clean_13dCep_16k_40mel_130Hz_6800Hz.4000.mdef 2641 Fri May 07 10:50:22 EDT 2004 edu/cmu/sphinx/model/acoustic/etc/WSJ_clean_13dCep_16k_40mel_130Hz_6800Hz.ci.mdef 375 Fri May 07 10:50:22 EDT 2004 edu/cmu/sphinx/model/acoustic/etc/variables.def 1797 Fri May 07 10:50:22 EDT 2004 edu/cmu/sphinx/model/acoustic/license.terms 719 Fri May 07 10:50:24 EDT 2004 edu/cmu/sphinx/model/acoustic/model.propsThe locations of the dictionary files with the JAR file are specified using the Sphinx-4 resource mechanism. In short, this mechanism locates the JAR file in which a class resides, and looks into that JAR file for the desired resource. The general syntax is:
resource:/{name of class to locate the JAR file}!{location in the JAR file of the desired resource}Take the 'dictionaryPath' property, for example. The "name of class to locate the JAR file" is "
edu.cmu.sphinx.model.acoustic.WSJ_8gau_13dCep_16k_40mel_130Hz_6800Hz
", while the "location in the JAR file of the desired resource" is "/edu/cmu/sphinx/model/acoustic/dict/cmudict.0.6d
". This
gives the string: "resource:/edu.cmu.sphinx.model.acoustic.WSJ_8gau_13dCep_16k_40mel_130Hz_6800Hz!/edu/cmu/sphinx/model/acoustic/dict/cmudict.0.6d"
.
For details about the other properties, please refer to the javadoc for FastDictionary.
The rest of the configuration file, which includes the front end configuration and the configuration of the monitors, are the same as in the HelloDigits demo. Therefore, please refer to those sections for explanations. This concludes the walkthrough of the simple HelloNGram example.
As you can see from the above examples, the Recognizer returns a Result object
which provides the recognition results.
The Result object essentially contains all the paths during the recognition
search that have reached the final state (or end of sentence, usually denoted
by ""). They are ranked by the
ending score of the path, and the one with the highest score is the best
hypothesis. Moreover, the Result also contains all the active paths (that
have not reached the final state) at the end of the recognition. Usually,
one would call the Result.getBestResultNoFiller
method to
obtain a string of the best result that has no filler words like "++SMACK++".
This method first attempts to return the best path that has reached the
final state. If no paths have reached the final state, it returns the best
path out of the paths that have not reached the final state.
If you only want to return those paths that have reached the final state,
you should call the method Result.getBestFinalResultNoFiller
. For example, the HelloWorld demo uses this method to avoid treating
any partial sentence in the grammar as the result.
There are other methods in the Result object that can give you
more information, e.g., the N-best results. You will also notice that there are
a number of methods that return Tokens. Tokens are objects along a search
path that record where we are at the search, and the various scores at that
particular location. For example, the Token object has a getWord
method that tells you which word the search is in. For details about the
Token object please refer to the javadoc for Token. For details about the Result object,
please refer to the
javadoc for Result.


Season
Buyer
Reference no
Job no
STO no.
Order Type
Issue Date
Delivery Date
John Cooper
Simple
SECTION
MATERIAL NO.
COLOR
GSM
DIA
ORDER QTY(IN KGS) OR COLLAR/CUFF(IN PCS)
(%)LOSS
FABRICATION
FAB-ACRONYM
Body
0%
Comment
Add more row
Save


The content of this tutorial is an extension of the Tutorial for compiling NCNN library.
When we successfully compile the NCNN library normally, we can use the tools in it to convert our models into NCNN format models (*.param
and *.bin
). This model can be used for various mobile terminal deployments.
But when we need to deploy the model in a webpage, we need to compile the model and the C++ program that calls it into WebAssembly format. Before compiling the program, we must first compile the NCNN library into the form of WebAssembly.
The content of this article is the specific steps for compiling the NCNN library into WebAssembly. The following steps have been tested on Ubuntu 18.04.
Install Node.js
Before starting, we need to install the latest version of node.js as a dependency in the environment. You can find the installation tutorial on its official Github. I chose its recommended installation location.
VERSION=v14.15.1
DISTRO=linux-x64
sudo mkdir -p /usr/local/lib/nodejs
sudo tar -xJvf node-$VERSION-$DISTRO.tar.xz -C /usr/local/lib/nodejs
This step is to extract the nodejs package to the recommended location. Next, run ~/.profile
in terminal, and paste the content below to the end of the pop-up file.
# Nodejs
VERSION=v14.15.1
DISTRO=linux-x64
export PATH=/usr/local/lib/nodejs/node-$VERSION-$DISTRO/bin:$PATH
Run . ~/.profile
in terminal to refresh the profile.
After the installation is over, you can run node -v
in terminal to check whether the installation is successful.
Build NCNN with WASM
First, let’s clone the NCNN repository. This folder should be different from the one we used before(in the previous article), because they have different usage.
git clone https://github.com/Tencent/ncnn.git
cd ncnn
Then clone the key library emsdk for compiling WASM.
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
Install and activate it.
./emsdk install latest
./emsdk activate latest
Then, return to the parent folder(master of ncnn), set the emsdk as a source.
cd ..
source emsdk/emsdk_env.sh
Observe the output of the terminal here, you will find that the currently used nodejs is a built-in library in emsdk, and its version is old. Therefore, we need to re-add the latest version of nodejs that we installed before to the path.
export PATH=/usr/local/lib/nodejs/node-v14.15.1-linux-x64/bin:$PATH
You can run node -v
to check the version again.
Next, prepare to compile the NCNN library.
mkdir build && cd build
If you are compiling for a browser that supports experimental WebAssembly features such as SIMD and SSE2, please use the following code.
cmake -DCMAKE_TOOLCHAIN_FILE=../emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DNCNN_SIMPLEOMP=ON -DNCNN_BUILD_TESTS=ON ..
If it is compiled for browsers such as Safari in iOS without SIMD SSE2, replace it with the following code.
cmake -DCMAKE_TOOLCHAIN_FILE=../emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DNCNN_OPENMP=OFF -DNCNN_SIMPLEOMP=OFF -DNCNN_RUNTIME_CPU=OFF -DNCNN_SSE2=OFF -DNCNN_AVX2=OFF -DNCNN_BUILD_TESTS=ON ..
Then start make.
make
After the make ends, use the built-in test case of NCNN to check whether the compilation is successful.
With SIMD SSE2:
TESTS_EXECUTABLE_LOADER=node TESTS_EXECUTABLE_LOADER_ARGUMENTS="--experimental-wasm-simd;--experimental-wasm-threads;--experimental-wasm-bulk-memory" ctest --output-on-failure -j 2
Without SIMD SSE2:
TESTS_EXECUTABLE_LOADER=node ctest --output-on-failure -j 2
If all the tests are passed, generate the install file.
make install
So far, the NCNN with WebAssembly has been compiled, and the next step is to introduce its usage.
Compile C++ code
First, we need to write a C++ program that calls the NCNN model. The relevant files I have written have been uploaded in the facemask-detection repository, and I will compile it based on this.
Clone it to the specified folder.
cd ~/Documents
git clone https://github.com/waittim/facemask-detection.git
cd facemask-detection
Note that if you are compiling a version without SIMD and SSE2, please delete the following content in CMakeLists.txt.
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fopenmp -s USE_PTHREADS=1 -s PTHREAD_POOL_SIZE=15")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fopenmp -s USE_PTHREADS=1 -s PTHREAD_POOL_SIZE=15")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fopenmp -s USE_PTHREADS=1 -s PTHREAD_POOL_SIZE=15")
And change target_link_libraries(yolo ncnn pthread)
to be target_link_libraries(yolo ncnn)
.
Next, create an empty folder named ncnn in it, and copy the two files under build/install/
under ncnn master to this ncnn folder.
mkdir ncnn
Copy the emsdk folder under ncnn master and paste it to the master folder of facemask-detection.
At this time, I recommend to use node -v
to ensure that the version of nodejs is correct.
Finally, use emsdk to compile.
emcmake cmake
emmake make
After compiling, you can get yolo.js
, yolo.wasm
,
yolo.worker.js
. These are the files needed to call WASM format functions in the JavaScript environment.
The complete CMakeLists.txt content is as follows:
project(facemask-detection)
cmake_minimum_required(VERSION 3.10)
set(CMAKE_BUILD_TYPE release)
set(ncnn_DIR "${CMAKE_CURRENT_SOURCE_DIR}/ncnn/lib/cmake/ncnn")
find_package(ncnn REQUIRED)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -s FORCE_FILESYSTEM=1 -s INITIAL_MEMORY=256MB -s EXIT_RUNTIME=1")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -s FORCE_FILESYSTEM=1 -s INITIAL_MEMORY=256MB -s EXIT_RUNTIME=1")
set(CMAKE_EXECUTBLE_LINKER_FLAGS "${CMAKE_EXECUTBLE_LINKER_FLAGS} -s FORCE_FILESYSTEM=1 -s INITIAL_MEMORY=256MB -s EXIT_RUNTIME=1")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fopenmp -s USE_PTHREADS=1 -s PTHREAD_POOL_SIZE=15")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fopenmp -s USE_PTHREADS=1 -s PTHREAD_POOL_SIZE=15")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fopenmp -s USE_PTHREADS=1 -s PTHREAD_POOL_SIZE=15")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -sEXPORTED_FUNCTIONS=['_yolo_ncnn'] --preload-file<EMAIL_ADDRESS>set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -sEXPORTED_FUNCTIONS=['_yolo_ncnn'] --preload-file<EMAIL_ADDRESS>set(CMAKE_EXECUTBLE_LINKER_FLAGS "${CMAKE_EXECUTBLE_LINKER_FLAGS} -sEXPORTED_FUNCTIONS=['_yolo_ncnn'] --preload-file<EMAIL_ADDRESS>add_executable(yolo yolo.cpp)
target_link_libraries(yolo ncnn pthread)
Note: Thanks to NCNN creator nihui for her help in this process.


Mastering Algorithms with C
By Kyle Loudon
Slots : 1
Table of Contents
[
SYMBOL
] [
A
] [
B
] [
C
] [
D
] [
E
] [
F
] [
G
] [
H
] [
I
] [
J
] [
K
] [
L
] [
M
] [
N
] [
O
] [
P
] [
Q
] [
R
] [
S
] [
T
] [
U
] [
V
] [
W
] [
X
]
fact function
factorial
2nd
3rd
facttail function
farthest-pair problems
feedback
Fibonacci heaps
FIFO (first-in, first-out)
file allocation, linked
final permutation
finishing times
first-in, first-out (FIFO)
frame management
2nd
3rd
4th
frame pointer
free_frame function
2nd
function calls
activation records
C
execution of
2nd
function pointers
2nd
function tables
2nd
functional relationship
Top


One of the last aspects I wanted to explore and get working on my MangoPi MQ-Pro. For this experiment, like my previous endeavor trying out Bluetooth on my MangoPi MQ-Pro, I used the Ubuntu 22.10 RISC-V image for the SiPeed LicheeRV Dock image.
No extra configuration is needed to get GPIO working with this image on the MangoPi MQ-Pro. I tried out three different ways of accessing the GPIO pins; using legacy sysfs-based access, using the userspace GPIO driver, and lastly using lgpio from Python.
First of all, I physically connected a LED to the GPIO pins of my MangoPi MQ-Pro. I connected a blue LED to pin 18 of the GPIO array, with a 1k resistor between it.
To use the sysfs-based approach to access the GPIO pins, you first need to find out the pin numbers the MangoPi MQ-Pro uses internally. This is a mapping table for this:
| GPIO | Pin | Pin | GPIO |
|---|---|---|---|
| 1 | 2 | ||
| 205 | 3 | 4 | |
| 204 | 5 | 6 | |
| 39 | 7 | 8 | 40 |
| 9 | 10 | 41 | |
| 117 | 11 | 12 | 37 |
| 118 | 13 | 14 | |
| 32 | 15 | 16 | 33 |
| 17 | 18 | 110 | |
| 19 | 20 | ||
| 21 | 22 | 65 | |
| 23 | 24 | ||
| 25 | 26 | 111 | |
| 145 | 27 | 28 | 144 |
| 42 | 29 | 30 | |
| 43 | 31 | 32 | 64 |
| 44 | 33 | 34 | |
| 38 | 35 | 36 | 34 |
| 113 | 37 | 38 | 35 |
| 39 | 40 | 36 |
So, if you have connected an output (LED) to pin 18, we need to access this using pin 110. With the following command we can start interacting with this exact pin, these will have to be ran as root:
# echo 110 >/sys/class/gpio/export
# echo out >/sys/class/gpio/gpio110/direction
# cat /sys/class/gpio/gpio110/direction
To make the LED light up, you can execute this command:
# echo 1 > /sys/class/gpio/gpio110/value
And to turn it off again:
# echo 0 > /sys/class/gpio/gpio110/value
When you’re done playing around, you can remove the resources to this pin:
# echo 110 >/sys/class/gpio/unexport
To make the LED blink, you can create a simple script:
while true
do
echo 0 > /sys/class/gpio/gpio110/value
sleep 1
echo 1 > /sys/class/gpio/gpio110/value
sleep 1
done
To make the same LED turn on as in the previous step, you can execute:
# gpioset gpiochip0 110=1
And to turn if off again:
gpioset gpiochip0 110=0
And a script to make it blink:
while true
do
gpioset gpiochip0 110=0
sleep 1
gpioset gpiochip0 110=1
sleep 1
done
To use the GPIO pins from Python, we will have to install a package:
# apt update
# apt install python3-lgpio
You can create a script name blink.py
to make the GPIO pin blink:
import time
import lgpio
LED = 110
# open the gpio chip and set the LED pin as output
h = lgpio.gpiochip_open(0)
lgpio.gpio_claim_output(h, LED)
try:
while True:
# Turn the GPIO pin on
lgpio.gpio_write(h, LED, 1)
time.sleep(1)
# Turn the GPIO pin off
lgpio.gpio_write(h, LED, 0)
time.sleep(1)
except KeyboardInterrupt:
lgpio.gpio_write(h, LED, 0)
lgpio.gpiochip_close(h)
When executing the script by executing python3 blink.py
, the LED should blink.


The Beginning... On September 24th, a team of young adults filled with high spirits, formed a unit for a purpose after successfully accomplishing a memorable act of kindness. We gathered together to plan, execute and pray over our first Dry Run named #OperationsLunchBag. This occasion involved us organizing lunch bags with homemade sandwiches, a bag of chips and an 8oz bottle of water, along with a target to feed 50 individuals. In addition to our intention, we chose 125th and Lexington, to actively execute our mission due to the extensive amount of homelessness happening on a daily basis. As we approached an end to our event only lasting 20 minutes, we were left with notable statements full of gratitude such as “May I please have two?” “Thank you, you’re truly blessed” “God Bless you and you’re family” “Hey, we’ll be here every day” and “Tatiana: They are more people by the FDR. Come I’ll show you”.
“Home is a notion that only nations of the homeless fully appreciate and only the uprooted comprehend.” - Wallace StegnerReflecting on what we were able to do in just 20 minutes for 50 individuals, we were left propelled, inspirited and inspired to continue servicing our community. Now as a team, we have pledged to make each minute count by committing our time to make a world a peaceful place. With our forthcoming belief of giving, we are determined to create higher objectives in the number of individuals we are able to comfort, provide and supply. In conclusion, Feed. Clothe. Pray is simply here to serve in countless ways.


Mobie-UI is Bootstrap based UI-library to create mobile Apps using front-end framework like Vue, Angular and React.
Your browser does not support iframes.
2020 © Copyright by Bootcatch.com


About
Projects
Contact
Resume
Ado Ibori
Web Developer
I tell computers what to do. My github is linked below.
Projects
Pathfinding Visualization Tool
Sorting Algorithm Visualization
Persistent Bug Tracker
Monte Carlo Pi Estimator
MLK Tribute Page
Loading…


Samsung Galaxy A51 nets Bluetooth certification
by MichailThe Samsung Galaxy A51 has been collecting the required certification from various agencies for the past weeks ahead of what appears to be an imminent launch. We recently also got out first renders of the phone which revealed it will boast a centered punch-hole display and L-shaped quad camera setup.
The latest clue comes from the Bluetooth certification agency which signals that we are closer to the official release. The device passed with the SM-AF15F/N model numbers and the only notable info that we got from the listing is that the phone will feature Bluetooth 5.0 support.
Previously, the A51 passed through Geekbench which revealed it will ship with the Exynos 9611 chipset just like the regular A50 alongside 4GB RAM and Android 10. More recently, we saw another certification which revealed the A51 will ship with a 4,000 mAh battery.
According to other rumors, the phone will feature a 6.5-inch FHD+ display with a 32MP selfie camera and UD fingerprint reader. The quad-camera setup will be headlined by a 48MP main shooter alongside a 12MP ultrawide camera, 12MP telephoto module and 5MP depth sensor. The phone is expected to go on sale in the early days of 2020.


|
OR SET IT URSELF Type hex color code or color name Note: won't automatically change text color to fit background |
Note: won't automatically change text color to fit background OR SET IT URSELF Type hex color code or color name Note: won't automatically change text color to fit background |
|---|
Copyright (c) 2019 Organization of Dpzk


{% extends "base-full.html" %} {% load staticfiles %} {% block title %}Sunlight Academy Coming Soon!{% endblock %} {% block css %} {{ block.super }} {% endblock %} {% block content %}
We will send you an email notification when the training portal is live.
Until then, check out some of Sunlight's other great tools.
Scout is a rapid notification service allows anyone to create customized email or text alerts on actions Congress takes on an issue or a specific bill.
Party Time documents the Congressional Fundraising Circuit.
Influence Explorer connects the dots of political contributions on the federal and state level.
Open States aggregates legislative data directly from state legislatures.
Politiwidgets makes inserting information on a Member of Congress into a blog post as easy as embedding a YouTube video.


My major is fine art and own unique personality is advantages. I studied character design and cartoons myself. and beacause I studied programming in ssafy, also knew how to deal with Python, Django, Web, JavaScript and Vuejs in programming.
- 영화 추천 서비스 플랫폼
- 삼성 소프트웨어 아카데미 교육생 취업 다이어리
- 향수 맞춤 추천 서비스 플랫폼
- 스토리텔링 여행 테마 추천 서비스 플랫폼
My major in college is Oriental painting: I have won awards in various contests, and even after graduation, I went to graduate school and participated in exhibitions. I draw a picture of a combination of ink and coloring. I have drawn many pictures without any restrictions. These trainings have stimulated my imagination.
I am a self-taught cartoonist with high-level of understanding of design. My design range is quite wide from people to any living things. Furthermore, I have experience of character design for new entrepreneurs(2017) and of the face and body painting for the festival visitors.(2015)
There was an annual overseas art program for education at the university. I applied for three times and once went on a scholarship. Thanks to university I could visit to Cambodia, Vietnam, and China . It was a great experience to see and learn new things: to learn culture and to broaden your perspective on painting and perception as you get to know people's food and life.
I learned the basics and algorithms of the python, the web (html, css) and database management through django. I also worked on a weekly project and created a simple movie recommendation site.
I like to draw personal pictures even when I have a job related to painting. I enjoy playing games with VR and other devices, watching movies, listening to music, cooking, and so on. I have a hobby without distinguishing between indoor and outdoor activities, but I usually like to spend time indoors. I like to watch Netflix indoors on rainy days, eat delicious food, and play games with my favorite people; My enjoying lists in Netflix are cute cartoons like Tuca & Bertie and Aggressive Letzco. I enjoy playing RPG games alone, but when I play together also enjoy games such as Overwatch and Maple Story.
Outdoor, I enjoy to go to comics cafe, Cinema, travel. My favorite comics is like fantasy, action, and comedy, and sometimes even have romance. I like almost every movie genre, but I do not like movies that have poor plot of story, so I don’t watch horror movies much. my favorite genre is SF, and I recently have a hobby in musical movies. I would like to see orchestra performances that play my favorite songs if I have a chance later. I love going to karaoke for sing, go to tasty restaurants and I always keep my own list of tasty restaurants in mind.


Asgard Framework - {{.Subtitle}}
{{include "layouts/header"}}
{{include "layouts/sidebar"}}
{{include "layouts/top"}}
{{template "content" .}}
{{template "script" .}}


PERLFAQ6
Section: Perl Programmers Reference Guide (1)Updated: 2009-08-15
Index Return to Main Contents
NAME
perlfaq6 - Regular ExpressionsDESCRIPTION
This section is surprisingly small because the rest of the FAQ is littered with answers involving regular expressions. For example, decoding a URL and checking whether something is a number are handled with regular expressions, but those answers are found elsewhere in this document (in perlfaq9: ``How do I decode or create those %-encodings on the web'' and perlfaq4: ``How do I determine whether a scalar is a number/whole/integer/float'', to be precise).How can I hope to use regular expressions without creating illegible and unmaintainable code?
Three techniques can make regular expressions maintainable and understandable.- Comments Outside the Regex
-
Describe what you're doing and how you're doing it, using normal Perl
comments.
# turn the line into the first word, a colon, and the # number of characters on the rest of the line s/^(\w+)(.*)/ lc($1) . ":" . length($2) /meg;
- Comments Inside the Regex
-
The "/x" modifier causes whitespace to be ignored in a regex pattern
(except in a character class), and also allows you to use normal
comments there, too. As you can imagine, whitespace and comments help
a lot.
"/x" lets you turn this:
s{<(?:[^>'"]*|".*?"|'.*?')+>}{}gs;
into this:
s{ < # opening angle bracket (?: # Non-backreffing grouping paren [^>'"] * # 0 or more things that are neither > nor ' nor " | # or else ".*?" # a section between double quotes (stingy match) | # or else '.*?' # a section between single quotes (stingy match) ) + # all occurring one or more times > # closing angle bracket }{}gsx; # replace with nothing, i.e. delete
It's still not quite so clear as prose, but it is very useful for describing the meaning of each part of the pattern.
- Different Delimiters
-
While we normally think of patterns as being delimited with "/"
characters, they can be delimited by almost any character. perlre
describes this. For example, the "s///" above uses braces as
delimiters. Selecting another delimiter can avoid quoting the
delimiter within the pattern:
s/\/usr\/local/\/usr\/share/g; # bad delimiter choice s#/usr/local#/usr/share#g; # better
I'm having trouble matching over more than one line. What's wrong?
Either you don't have more than one line in the string you're looking at (probably), or else you aren't using the correct modifier(s) on your pattern (possibly).There are many ways to get multiline data into a string. If you want it to happen automatically while reading input, you'll want to set $/ (probably to '' for paragraphs or "undef" for the whole file) to allow you to read more than one line at a time.
Read perlre to help you decide which of "/s" and "/m" (or both) you might want to use: "/s" allows dot to include newline, and "/m" allows caret and dollar to match next to a newline, not just at the end of the string. You do need to make sure that you've actually got a multiline string in there.
For example, this program detects duplicate words, even when they span line breaks (but not paragraph ones). For this example, we don't need "/s" because we aren't using dot in a regular expression that we want to cross line boundaries. Neither do we need "/m" because we aren't wanting caret or dollar to match at any point inside the record next to newlines. But it's imperative that $/ be set to something other than the default, or else we won't actually ever have a multiline record read in.
$/ = ''; # read in whole paragraph, not just one line while ( <> ) { while ( /\b([\w'-]+)(\s+\1)+\b/gi ) { # word starts alpha print "Duplicate $1 at paragraph $.\n"; } }
Here's code that finds sentences that begin with ``From '' (which would be mangled by many mailers):
$/ = ''; # read in whole paragraph, not just one line while ( <> ) { while ( /^From /gm ) { # /m makes ^ match next to \n print "leading from in paragraph $.\n"; } }
Here's code that finds everything between START and END in a paragraph:
undef $/; # read in whole file, not just one line or paragraph while ( <> ) { while ( /START(.*?)END/sgm ) { # /s makes . cross line boundaries print "$1\n"; } }
How can I pull out lines between two patterns that are themselves on different lines?
You can use Perl's somewhat exotic ".." operator (documented in perlop):
perl -ne 'print if /START/ .. /END/' file1 file2 ...
If you wanted text and not lines, you would use
perl -0777 -ne 'print "$1\n" while /START(.*?)END/gs' file1 file2 ...
But if you want nested occurrences of "START" through "END", you'll run up against the problem described in the question in this section on matching balanced text.
Here's another example of using "..":
while (<>) { $in_header = 1 .. /^$/; $in_body = /^$/ .. eof; # now choose between them } continue { $. = 0 if eof; # fix $. }
How do I match XML, HTML, or other nasty, ugly things with a regex?
(contributed by brian d foy)If you just want to get work done, use a module and forget about the regular expressions. The "XML::Parser" and "HTML::Parser" modules are good starts, although each namespace has other parsing modules specialized for certain tasks and different ways of doing it. Start at CPAN Search ( http://search.cpan.org ) and wonder at all the work people have done for you already! :)
The problem with things such as XML is that they have balanced text containing multiple levels of balanced text, but sometimes it isn't balanced text, as in an empty tag ("<br/>", for instance). Even then, things can occur out-of-order. Just when you think you've got a pattern that matches your input, someone throws you a curveball.
If you'd like to do it the hard way, scratching and clawing your way toward a right answer but constantly being disappointed, beseiged by bug reports, and weary from the inordinate amount of time you have to spend reinventing a triangular wheel, then there are several things you can try before you give up in frustration:
- *
- Solve the balanced text problem from another question in perlfaq6
- *
- Try the recursive regex features in Perl 5.10 and later. See perlre
- *
- Try defining a grammar using Perl 5.10's "(?DEFINE)" feature.
- *
- Break the problem down into sub-problems instead of trying to use a single regex
- *
- Convince everyone not to use XML or HTML in the first place
I put a regular expression into $/ but it didn't work. What's wrong?
$/ has to be a string. You can use these examples if you really need to do this.If you have File::Stream, this is easy.
use File::Stream; my $stream = File::Stream->new( $filehandle, separator => qr/\s*,\s*/, ); print "$_\n" while <$stream>;
If you don't have File::Stream, you have to do a little more work.
You can use the four-argument form of sysread to continually add to a buffer. After you add to the buffer, you check if you have a complete line (using your regular expression).
local $_ = ""; while( sysread FH, $_, 8192, length ) { while( s/^((?s).*?)your_pattern// ) { my $record = $1; # do stuff here. } }
You can do the same thing with foreach and a match using the c flag and the \G anchor, if you do not mind your entire file being in memory at the end.
local $_ = ""; while( sysread FH, $_, 8192, length ) { foreach my $record ( m/\G((?s).*?)your_pattern/gc ) { # do stuff here. } substr( $_, 0, pos ) = "" if pos; }
How do I substitute case insensitively on the LHS while preserving case on the RHS?
Here's a lovely Perlish solution by Larry Rosler. It exploits properties of bitwise xor on ASCII strings.
$_= "this is a TEsT case"; $old = 'test'; $new = 'success'; s{(\Q$old\E)} { uc $new | (uc $1 ^ $1) . (uc(substr $1, -1) ^ substr $1, -1) x (length($new) - length $1) }egi; print;
And here it is as a subroutine, modeled after the above:
sub preserve_case($$) { my ($old, $new) = @_; my $mask = uc $old ^ $old; uc $new | $mask . substr($mask, -1) x (length($new) - length($old)) } $string = "this is a TEsT case"; $string =~ s/(test)/preserve_case($1, "success")/egi; print "$string\n";
This prints:
this is a SUcCESS case
As an alternative, to keep the case of the replacement word if it is longer than the original, you can use this code, by Jeff Pinyan:
sub preserve_case { my ($from, $to) = @_; my ($lf, $lt) = map length, @_; if ($lt < $lf) { $from = substr $from, 0, $lt } else { $from .= substr $to, $lf } return uc $to | ($from ^ uc $from); }
This changes the sentence to ``this is a SUcCess case.''
Just to show that C programmers can write C in any programming language, if you prefer a more C-like solution, the following script makes the substitution have the same case, letter by letter, as the original. (It also happens to run about 240% slower than the Perlish solution runs.) If the substitution has more characters than the string being substituted, the case of the last character is used for the rest of the substitution.
# Original by Nathan Torkington, massaged by Jeffrey Friedl # sub preserve_case($$) { my ($old, $new) = @_; my ($state) = 0; # 0 = no change; 1 = lc; 2 = uc my ($i, $oldlen, $newlen, $c) = (0, length($old), length($new)); my ($len) = $oldlen < $newlen ? $oldlen : $newlen; for ($i = 0; $i < $len; $i++) { if ($c = substr($old, $i, 1), $c =~ /[\W\d_]/) { $state = 0; } elsif (lc $c eq $c) { substr($new, $i, 1) = lc(substr($new, $i, 1)); $state = 1; } else { substr($new, $i, 1) = uc(substr($new, $i, 1)); $state = 2; } } # finish up with any remaining new (for when new is longer than old) if ($newlen > $oldlen) { if ($state == 1) { substr($new, $oldlen) = lc(substr($new, $oldlen)); } elsif ($state == 2) { substr($new, $oldlen) = uc(substr($new, $oldlen)); } } return $new; }
How can I make \w match national character sets?
Put "use locale;" in your script. The \w character class is taken from the current locale.How can I match a locale-smart version of /[a-zA-Z]/?
You can use the POSIX character class syntax "/[[:alpha:]]/" documented in perlre.No matter which locale you are in, the alphabetic characters are the characters in \w without the digits and the underscore. As a regex, that looks like "/[^\W\d_]/". Its complement, the non-alphabetics, is then everything in \W along with the digits and the underscore, or "/[\W\d_]/".
How can I quote a variable to use in a regex?
The Perl parser will expand $variable and @variable references in regular expressions unless the delimiter is a single quote. Remember, too, that the right-hand side of a "s///" substitution is considered a double-quoted string (see perlop for more details). Remember also that any regex special characters will be acted on unless you precede the substitution with \Q. Here's an example:
$string = "Placido P. Octopus"; $regex = "P."; $string =~ s/$regex/Polyp/; # $string is now "Polypacido P. Octopus"
Because "." is special in regular expressions, and can match any single character, the regex "P." here has matched the <Pl> in the original string.
To escape the special meaning of ".", we use "\Q":
$string = "Placido P. Octopus"; $regex = "P."; $string =~ s/\Q$regex/Polyp/; # $string is now "Placido Polyp Octopus"
The use of "\Q" causes the <.> in the regex to be treated as a regular character, so that "P." matches a "P" followed by a dot.
What is /o really for?
(contributed by brian d foy)The "/o" option for regular expressions (documented in perlop and perlreref) tells Perl to compile the regular expression only once. This is only useful when the pattern contains a variable. Perls 5.6 and later handle this automatically if the pattern does not change.
Since the match operator "m//", the substitution operator "s///", and the regular expression quoting operator "qr//" are double-quotish constructs, you can interpolate variables into the pattern. See the answer to ``How can I quote a variable to use in a regex?'' for more details.
This example takes a regular expression from the argument list and prints the lines of input that match it:
my $pattern = shift @ARGV; while( <> ) { print if m/$pattern/; }
Versions of Perl prior to 5.6 would recompile the regular expression for each iteration, even if $pattern had not changed. The "/o" would prevent this by telling Perl to compile the pattern the first time, then reuse that for subsequent iterations:
my $pattern = shift @ARGV; while( <> ) { print if m/$pattern/o; # useful for Perl < 5.6 }
In versions 5.6 and later, Perl won't recompile the regular expression if the variable hasn't changed, so you probably don't need the "/o" option. It doesn't hurt, but it doesn't help either. If you want any version of Perl to compile the regular expression only once even if the variable changes (thus, only using its initial value), you still need the "/o".
You can watch Perl's regular expression engine at work to verify for yourself if Perl is recompiling a regular expression. The "use re 'debug'" pragma (comes with Perl 5.005 and later) shows the details. With Perls before 5.6, you should see "re" reporting that its compiling the regular expression on each iteration. With Perl 5.6 or later, you should only see "re" report that for the first iteration.
use re 'debug'; $regex = 'Perl'; foreach ( qw(Perl Java Ruby Python) ) { print STDERR "-" x 73, "\n"; print STDERR "Trying $_...\n"; print STDERR "\t$_ is good!\n" if m/$regex/; }
How do I use a regular expression to strip C style comments from a file?
While this actually can be done, it's much harder than you'd think. For example, this one-liner
perl -0777 -pe 's{/\*.*?\*/}{}gs' foo.c
will work in many but not all cases. You see, it's too simple-minded for certain kinds of C programs, in particular, those with what appear to be comments in quoted strings. For that, you'd need something like this, created by Jeffrey Friedl and later modified by Fred Curtis.
$/ = undef; $_ = <>; s#/\*[^*]*\*+([^/*][^*]*\*+)*/|("(\\.|[^"\\])*"|'(\\.|[^'\\])*'|.[^/"'\\]*)#defined $2 ? $2 : ""#gse; print;
This could, of course, be more legibly written with the "/x" modifier, adding whitespace and comments. Here it is expanded, courtesy of Fred Curtis.
s{ /\* ## Start of /* ... */ comment [^*]*\*+ ## Non-* followed by 1-or-more *'s ( [^/*][^*]*\*+ )* ## 0-or-more things which don't start with / ## but do end with '*' / ## End of /* ... */ comment | ## OR various things which aren't comments: ( " ## Start of " ... " string ( \\. ## Escaped char | ## OR [^"\\] ## Non "\ )* " ## End of " ... " string | ## OR ' ## Start of ' ... ' string ( \\. ## Escaped char | ## OR [^'\\] ## Non '\ )* ' ## End of ' ... ' string | ## OR . ## Anything other char [^/"'\\]* ## Chars which doesn't start a comment, string or escape ) }{defined $2 ? $2 : ""}gxse;
A slight modification also removes C++ comments, possibly spanning multiple lines using a continuation character:
s#/\*[^*]*\*+([^/*][^*]*\*+)*/|//([^\\]|[^\n][\n]?)*?\n|("(\\.|[^"\\])*"|'(\\.|[^'\\])*'|.[^/"'\\]*)#defined $3 ? $3 : ""#gse;
Can I use Perl regular expressions to match balanced text?
(contributed by brian d foy)Your first try should probably be the "Text::Balanced" module, which is in the Perl standard library since Perl 5.8. It has a variety of functions to deal with tricky text. The "Regexp::Common" module can also help by providing canned patterns you can use.
As of Perl 5.10, you can match balanced text with regular expressions using recursive patterns. Before Perl 5.10, you had to resort to various tricks such as using Perl code in "(??{})" sequences.
Here's an example using a recursive regular expression. The goal is to capture all of the text within angle brackets, including the text in nested angle brackets. This sample text has two ``major'' groups: a group with one level of nesting and a group with two levels of nesting. There are five total groups in angle brackets:
I have some <brackets in <nested brackets> > and <another group <nested once <nested twice> > > and that's it.
The regular expression to match the balanced text uses two new (to Perl 5.10) regular expression features. These are covered in perlre and this example is a modified version of one in that documentation.
First, adding the new possesive "+" to any quantifier finds the longest match and does not backtrack. That's important since you want to handle any angle brackets through the recursion, not backtracking. The group "[^<>]++" finds one or more non-angle brackets without backtracking.
Second, the new "(?PARNO)" refers to the sub-pattern in the particular capture buffer given by "PARNO". In the following regex, the first capture buffer finds (and remembers) the balanced text, and you need that same pattern within the first buffer to get past the nested text. That's the recursive part. The "(?1)" uses the pattern in the outer capture buffer as an independent part of the regex.
Putting it all together, you have:
#!/usr/local/bin/perl5.10.0 my $string =<<"HERE"; I have some <brackets in <nested brackets> > and <another group <nested once <nested twice> > > and that's it. HERE my @groups = $string =~ m/ ( # start of capture buffer 1 < # match an opening angle bracket (?: [^<>]++ # one or more non angle brackets, non backtracking | (?1) # found < or >, so recurse to capture buffer 1 )* > # match a closing angle bracket ) # end of capture buffer 1 /xg; $" = "\n\t"; print "Found:\n\t@groups\n";
The output shows that Perl found the two major groups:
Found: <brackets in <nested brackets> > <another group <nested once <nested twice> > >
With a little extra work, you can get the all of the groups in angle brackets even if they are in other angle brackets too. Each time you get a balanced match, remove its outer delimiter (that's the one you just matched so don't match it again) and add it to a queue of strings to process. Keep doing that until you get no matches:
#!/usr/local/bin/perl5.10.0 my @queue =<<"HERE"; I have some <brackets in <nested brackets> > and <another group <nested once <nested twice> > > and that's it. HERE my $regex = qr/ ( # start of bracket 1 < # match an opening angle bracket (?: [^<>]++ # one or more non angle brackets, non backtracking | (?1) # recurse to bracket 1 )* > # match a closing angle bracket ) # end of bracket 1 /x; $" = "\n\t"; while( @queue ) { my $string = shift @queue; my @groups = $string =~ m/$regex/g; print "Found:\n\t@groups\n\n" if @groups; unshift @queue, map { s/^<//; s/>$//; $_ } @groups; }
The output shows all of the groups. The outermost matches show up first and the nested matches so up later:
Found: <brackets in <nested brackets> > <another group <nested once <nested twice> > > Found: <nested brackets> Found: <nested once <nested twice> > Found: <nested twice>
What does it mean that regexes are greedy? How can I get around it?
Most people mean that greedy regexes match as much as they can. Technically speaking, it's actually the quantifiers ("?", "*", "+", "{}") that are greedy rather than the whole pattern; Perl prefers local greed and immediate gratification to overall greed. To get non-greedy versions of the same quantifiers, use ("??", "*?", "+?", "{}?").An example:
$s1 = $s2 = "I am very very cold"; $s1 =~ s/ve.*y //; # I am cold $s2 =~ s/ve.*?y //; # I am very cold
Notice how the second substitution stopped matching as soon as it encountered ``y ''. The "*?" quantifier effectively tells the regular expression engine to find a match as quickly as possible and pass control on to whatever is next in line, like you would if you were playing hot potato.
How do I process each word on each line?
Use the split function:
while (<>) { foreach $word ( split ) { # do something with $word here } }
Note that this isn't really a word in the English sense; it's just chunks of consecutive non-whitespace characters.
To work with only alphanumeric sequences (including underscores), you might consider
while (<>) { foreach $word (m/(\w+)/g) { # do something with $word here } }
How can I print out a word-frequency or line-frequency summary?
To do this, you have to parse out each word in the input stream. We'll pretend that by word you mean chunk of alphabetics, hyphens, or apostrophes, rather than the non-whitespace chunk idea of a word given in the previous question:
while (<>) { while ( /(\b[^\W_\d][\w'-]+\b)/g ) { # misses "`sheep'" $seen{$1}++; } } while ( ($word, $count) = each %seen ) { print "$count $word\n"; }
If you wanted to do the same thing for lines, you wouldn't need a regular expression:
while (<>) { $seen{$_}++; } while ( ($line, $count) = each %seen ) { print "$count $line"; }
If you want these output in a sorted order, see perlfaq4: ``How do I sort a hash (optionally by value instead of key)?''.
How can I do approximate matching?
See the module String::Approx available from CPAN.How do I efficiently match many regular expressions at once?
( contributed by brian d foy )Avoid asking Perl to compile a regular expression every time you want to match it. In this example, perl must recompile the regular expression for every iteration of the "foreach" loop since it has no way to know what $pattern will be.
@patterns = qw( foo bar baz ); LINE: while( <DATA> ) { foreach $pattern ( @patterns ) { if( /\b$pattern\b/i ) { print; next LINE; } } }
The "qr//" operator showed up in perl 5.005. It compiles a regular expression, but doesn't apply it. When you use the pre-compiled version of the regex, perl does less work. In this example, I inserted a "map" to turn each pattern into its pre-compiled form. The rest of the script is the same, but faster.
@patterns = map { qr/\b$_\b/i } qw( foo bar baz ); LINE: while( <> ) { foreach $pattern ( @patterns ) { if( /$pattern/ ) { print; next LINE; } } }
In some cases, you may be able to make several patterns into a single regular expression. Beware of situations that require backtracking though.
$regex = join '|', qw( foo bar baz ); LINE: while( <> ) { print if /\b(?:$regex)\b/i; }
For more details on regular expression efficiency, see Mastering Regular Expressions by Jeffrey Freidl. He explains how regular expressions engine work and why some patterns are surprisingly inefficient. Once you understand how perl applies regular expressions, you can tune them for individual situations.
Why don't word-boundary searches with \b work for me?
(contributed by brian d foy)Ensure that you know what \b really does: it's the boundary between a word character, \w, and something that isn't a word character. That thing that isn't a word character might be \W, but it can also be the start or end of the string.
It's not (not!) the boundary between whitespace and non-whitespace, and it's not the stuff between words we use to create sentences.
In regex speak, a word boundary (\b) is a ``zero width assertion'', meaning that it doesn't represent a character in the string, but a condition at a certain position.
For the regular expression, /\bPerl\b/, there has to be a word boundary before the ``P'' and after the ``l''. As long as something other than a word character precedes the ``P'' and succeeds the ``l'', the pattern will match. These strings match /\bPerl\b/.
"Perl" # no word char before P or after l "Perl " # same as previous (space is not a word char) "'Perl'" # the ' char is not a word char "Perl's" # no word char before P, non-word char after "l"
These strings do not match /\bPerl\b/.
"Perl_" # _ is a word char! "Perler" # no word char before P, but one after l
You don't have to use \b to match words though. You can look for non-word characters surrounded by word characters. These strings match the pattern /\b'\b/.
"don't" # the ' char is surrounded by "n" and "t" "qep'a'" # the ' char is surrounded by "p" and "a"
These strings do not match /\b'\b/.
"foo'" # there is no word char after non-word '
You can also use the complement of \b, \B, to specify that there should not be a word boundary.
In the pattern /\Bam\B/, there must be a word character before the ``a'' and after the ``m''. These patterns match /\Bam\B/:
"llama" # "am" surrounded by word chars "Samuel" # same
These strings do not match /\Bam\B/
"Sam" # no word boundary before "a", but one after "m" "I am Sam" # "am" surrounded by non-word chars
Why does using $&, $`, or $' slow my program down?
(contributed by Anno Siegel)Once Perl sees that you need one of these variables anywhere in the program, it provides them on each and every pattern match. That means that on every pattern match the entire string will be copied, part of it to $`, part to $&, and part to $'. Thus the penalty is most severe with long strings and patterns that match often. Avoid $&, $', and $` if you can, but if you can't, once you've used them at all, use them at will because you've already paid the price. Remember that some algorithms really appreciate them. As of the 5.005 release, the $& variable is no longer ``expensive'' the way the other two are.
Since Perl 5.6.1 the special variables @- and @+ can functionally replace $`, $& and $'. These arrays contain pointers to the beginning and end of each match (see perlvar for the full story), so they give you essentially the same information, but without the risk of excessive string copying.
Perl 5.10 added three specials, "${^MATCH}", "${^PREMATCH}", and "${^POSTMATCH}" to do the same job but without the global performance penalty. Perl 5.10 only sets these variables if you compile or execute the regular expression with the "/p" modifier.
What good is \G in a regular expression?
You use the "\G" anchor to start the next match on the same string where the last match left off. The regular expression engine cannot skip over any characters to find the next match with this anchor, so "\G" is similar to the beginning of string anchor, "^". The "\G" anchor is typically used with the "g" flag. It uses the value of "pos()" as the position to start the next match. As the match operator makes successive matches, it updates "pos()" with the position of the next character past the last match (or the first character of the next match, depending on how you like to look at it). Each string has its own "pos()" value.Suppose you want to match all of consecutive pairs of digits in a string like ``1122a44'' and stop matching when you encounter non-digits. You want to match 11 and 22 but the letter <a> shows up between 22 and 44 and you want to stop at "a". Simply matching pairs of digits skips over the "a" and still matches 44.
$_ = "1122a44"; my @pairs = m/(\d\d)/g; # qw( 11 22 44 )
If you use the "\G" anchor, you force the match after 22 to start with the "a". The regular expression cannot match there since it does not find a digit, so the next match fails and the match operator returns the pairs it already found.
$_ = "1122a44"; my @pairs = m/\G(\d\d)/g; # qw( 11 22 )
You can also use the "\G" anchor in scalar context. You still need the "g" flag.
$_ = "1122a44"; while( m/\G(\d\d)/g ) { print "Found $1\n"; }
After the match fails at the letter "a", perl resets "pos()" and the next match on the same string starts at the beginning.
$_ = "1122a44"; while( m/\G(\d\d)/g ) { print "Found $1\n"; } print "Found $1 after while" if m/(\d\d)/g; # finds "11"
You can disable "pos()" resets on fail with the "c" flag, documented in perlop and perlreref. Subsequent matches start where the last successful match ended (the value of "pos()") even if a match on the same string has failed in the meantime. In this case, the match after the "while()" loop starts at the "a" (where the last match stopped), and since it does not use any anchor it can skip over the "a" to find 44.
$_ = "1122a44"; while( m/\G(\d\d)/gc ) { print "Found $1\n"; } print "Found $1 after while" if m/(\d\d)/g; # finds "44"
Typically you use the "\G" anchor with the "c" flag when you want to try a different match if one fails, such as in a tokenizer. Jeffrey Friedl offers this example which works in 5.004 or later.
while (<>) { chomp; PARSER: { m/ \G( \d+\b )/gcx && do { print "number: $1\n"; redo; }; m/ \G( \w+ )/gcx && do { print "word: $1\n"; redo; }; m/ \G( \s+ )/gcx && do { print "space: $1\n"; redo; }; m/ \G( [^\w\d]+ )/gcx && do { print "other: $1\n"; redo; }; } }
For each line, the "PARSER" loop first tries to match a series of digits followed by a word boundary. This match has to start at the place the last match left off (or the beginning of the string on the first match). Since "m/ \G( \d+\b )/gcx" uses the "c" flag, if the string does not match that regular expression, perl does not reset pos() and the next match starts at the same position to try a different pattern.
Are Perl regexes DFAs or NFAs? Are they POSIX compliant?
While it's true that Perl's regular expressions resemble the DFAs (deterministic finite automata) of the egrep(1) program, they are in fact implemented as NFAs (non-deterministic finite automata) to allow backtracking and backreferencing. And they aren't POSIX-style either, because those guarantee worst-case behavior for all cases. (It seems that some people prefer guarantees of consistency, even when what's guaranteed is slowness.) See the book ``Mastering Regular Expressions'' (from O'Reilly) by Jeffrey Friedl for all the details you could ever hope to know on these matters (a full citation appears in perlfaq2).What's wrong with using grep in a void context?
The problem is that grep builds a return list, regardless of the context. This means you're making Perl go to the trouble of building a list that you then just throw away. If the list is large, you waste both time and space. If your intent is to iterate over the list, then use a for loop for this purpose.In perls older than 5.8.1, map suffers from this problem as well. But since 5.8.1, this has been fixed, and map is context aware - in void context, no lists are constructed.
How can I match strings with multibyte characters?
Starting from Perl 5.6 Perl has had some level of multibyte character support. Perl 5.8 or later is recommended. Supported multibyte character repertoires include Unicode, and legacy encodings through the Encode module. See perluniintro, perlunicode, and Encode.If you are stuck with older Perls, you can do Unicode with the "Unicode::String" module, and character conversions using the "Unicode::Map8" and "Unicode::Map" modules. If you are using Japanese encodings, you might try using the jperl 5.005_03.
Finally, the following set of approaches was offered by Jeffrey Friedl, whose article in issue #5 of The Perl Journal talks about this very matter.
Let's suppose you have some weird Martian encoding where pairs of ASCII uppercase letters encode single Martian letters (i.e. the two bytes ``CV'' make a single Martian letter, as do the two bytes ``SG'', ``VS'', ``XX'', etc.). Other bytes represent single characters, just like ASCII.
So, the string of Martian ``I am CVSGXX!'' uses 12 bytes to encode the nine characters 'I', ' ', 'a', 'm', ' ', 'CV', 'SG', 'XX', '!'.
Now, say you want to search for the single character "/GX/". Perl doesn't know about Martian, so it'll find the two bytes ``GX'' in the ``I am CVSGXX!'' string, even though that character isn't there: it just looks like it is because ``SG'' is next to ``XX'', but there's no real ``GX''. This is a big problem.
Here are a few ways, all painful, to deal with it:
# Make sure adjacent "martian" bytes are no longer adjacent. $martian =~ s/([A-Z][A-Z])/ $1 /g; print "found GX!\n" if $martian =~ /GX/;
Or like this:
@chars = $martian =~ m/([A-Z][A-Z]|[^A-Z])/g; # above is conceptually similar to: @chars = $text =~ m/(.)/g; # foreach $char (@chars) { print "found GX!\n", last if $char eq 'GX'; }
Or like this:
while ($martian =~ m/\G([A-Z][A-Z]|.)/gs) { # \G probably unneeded print "found GX!\n", last if $1 eq 'GX'; }
Here's another, slightly less painful, way to do it from Benjamin Goldberg, who uses a zero-width negative look-behind assertion.
print "found GX!\n" if $martian =~ m/ (?<![A-Z]) (?:[A-Z][A-Z])*? GX /x;
This succeeds if the ``martian'' character GX is in the string, and fails otherwise. If you don't like using (?<!), a zero-width negative look-behind assertion, you can replace (?<![A-Z]) with (?:^|[^A-Z]).
It does have the drawback of putting the wrong thing in $-[0] and $+[0], but this usually can be worked around.
How do I match a regular expression that's in a variable? ,
(contributed by brian d foy)We don't have to hard-code patterns into the match operator (or anything else that works with regular expressions). We can put the pattern in a variable for later use.
The match operator is a double quote context, so you can interpolate your variable just like a double quoted string. In this case, you read the regular expression as user input and store it in $regex. Once you have the pattern in $regex, you use that variable in the match operator.
chomp( my $regex = <STDIN> ); if( $string =~ m/$regex/ ) { ... }
Any regular expression special characters in $regex are still special, and the pattern still has to be valid or Perl will complain. For instance, in this pattern there is an unpaired parenthesis.
my $regex = "Unmatched ( paren"; "Two parens to bind them all" =~ m/$regex/;
When Perl compiles the regular expression, it treats the parenthesis as the start of a memory match. When it doesn't find the closing parenthesis, it complains:
Unmatched ( in regex; marked by <-- HERE in m/Unmatched ( <-- HERE paren/ at script line 3.
You can get around this in several ways depending on our situation. First, if you don't want any of the characters in the string to be special, you can escape them with "quotemeta" before you use the string.
chomp( my $regex = <STDIN> ); $regex = quotemeta( $regex ); if( $string =~ m/$regex/ ) { ... }
You can also do this directly in the match operator using the "\Q" and "\E" sequences. The "\Q" tells Perl where to start escaping special characters, and the "\E" tells it where to stop (see perlop for more details).
chomp( my $regex = <STDIN> ); if( $string =~ m/\Q$regex\E/ ) { ... }
Alternately, you can use "qr//", the regular expression quote operator (see perlop for more details). It quotes and perhaps compiles the pattern, and you can apply regular expression flags to the pattern.
chomp( my $input = <STDIN> ); my $regex = qr/$input/is; $string =~ m/$regex/ # same as m/$input/is;
You might also want to trap any errors by wrapping an "eval" block around the whole thing.
chomp( my $input = <STDIN> ); eval { if( $string =~ m/\Q$input\E/ ) { ... } }; warn $@ if $@;
Or...
my $regex = eval { qr/$input/is }; if( defined $regex ) { $string =~ m/$regex/; } else { warn $@; }
REVISION
Revision: $Revision$Date: $Date$
See perlfaq for source control details and availability.
AUTHOR AND COPYRIGHT
Copyright (c) 1997-2009 Tom Christiansen, Nathan Torkington, and other authors as noted. All rights reserved.This documentation is free; you can redistribute it and/or modify it under the same terms as Perl itself.
Irrespective of its distribution, all code examples in this file are hereby placed into the public domain. You are permitted and encouraged to use this code in your own programs for fun or for profit as you see fit. A simple comment in the code giving credit would be courteous but is not required.
Index
- NAME
- DESCRIPTION
-
- How can I hope to use regular expressions without creating illegible and unmaintainable code?
- I'm having trouble matching over more than one line. What's wrong?
- How can I pull out lines between two patterns that are themselves on different lines?
- How do I match XML, HTML, or other nasty, ugly things with a regex?
- I put a regular expression into $/ but it didn't work. What's wrong?
- How do I substitute case insensitively on the LHS while preserving case on the RHS?
- How can I make \w match national character sets?
- How can I match a locale-smart version of /[a-zA-Z]/?
- How can I quote a variable to use in a regex?
- What is /o really for?
- How do I use a regular expression to strip C style comments from a file?
- Can I use Perl regular expressions to match balanced text?
- What does it mean that regexes are greedy? How can I get around it?
- How do I process each word on each line?
- How can I print out a word-frequency or line-frequency summary?
- How can I do approximate matching?
- How do I efficiently match many regular expressions at once?
- Why don't word-boundary searches with \b work for me?
- Why does using $&, $`, or $' slow my program down?
- What good is \G in a regular expression?
- Are Perl regexes DFAs or NFAs? Are they POSIX compliant?
- What's wrong with using grep in a void context?
- How can I match strings with multibyte characters?
- How do I match a regular expression that's in a variable? ,
- REVISION
- AUTHOR AND COPYRIGHT
This document was created by man2html, using the manual pages.
Time: 05:29:08 GMT, December 24, 2015


The <go-footer>
component should only be used once, at the root level component.
It should be placed inside a <go-layout>
and be given the attribute of
<go-layout-footer>
.
This component will be rendered at the end of the main content on ever page.
The go-layout uses the go-layout-footer
for content projection.
This should be applied to a <go-footer>
element to
add content to the end of the page. The content will be centered on the page.
For the sake of this example, we'll be referencing the layout of this application because it's a living example of the entire layout component and family. It is also being shown inside of the layout component because it's required to be inside it.


--- layout: page title: Projects permalink: /projects/index.html ---
I'm currently in the process of syncing my personal projects to this page. You can find a complete list of my personal projects here:
{% include purple-icon-github.html username=project.github %}
{% endif %} {% if project.gitlab %}{% include gitlab-fox.html username=project.gitlab %}
{% endif %}{% include link.html link=project.link link_name=project.link %}
{% if project.image %}{% endif %}


Lorem Ipsum is simply dummy text of the printing and typesetting industry.
Lorem Ipsum is simply dummy text of the printing and typesetting industry.
Lorem Ipsum is simply dummy text of the printing and typesetting industry.
Lorem Ipsum is simply dummy text of the printing and typesetting industry.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Nulla vitae elit libero, a pharetra augue mollis interdum.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Nulla vitae elit libero, a pharetra augue mollis interdum.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Nulla vitae elit libero, a pharetra augue mollis interdum.
Get discounts on pricing and services.
Contact Us$0/month
$100/month
$500/month
98% 5 star reviews in the "People" business community.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.