Tsearch2 - Introduction
[Online version] of this document is available.
The tsearch2 module is available to add as an extension to the PostgreSQL database to allow for Full Text Indexing. This document is an introduction to installing, configuring, using and maintaining the database with the tsearch2 module activated.
Please, note, tsearch2 module is fully incompatible with old tsearch, which is deprecated in 7.4 and will be obsoleted in 8.0.
USING TSEARCH2 AND POSTGRESQL FOR A WEB BASED SEARCH ENGINE
This documentation is provided as a short guide on how to quickly get up and running with tsearch2 and PostgreSQL, for those who want to implement a full text indexed based search engine. It is not meant to be a complete in-depth guide into the full ins and outs of the contrib/tsearch2 module, and is primarily aimed at beginners who want to speed up searching of large text fields, or those migrating from other database systems such as MS-SQL.
The README.tsearch2 file included in the contrib/tsearch2 directory contains a brief overview and history behind tsearch. This can also be found online [right here].
Further in depth documentation such as a full function reference, and user guide can be found online at the [tsearch documentation home].
ACKNOWLEDGEMENTS
Robert John Shepherd originally wrote this documentation for the previous version of tsearch module (v1) included with the postgres release. I took his documentation and updated it to comply with the tsearch2 modifications.
Robert's original acknowledgements:
"Thanks to Oleg Bartunov for taking the time to answer many of my questions regarding this module, and also to Teodor Sigaev for clearing up the process of making your own dictionaries. Plus of course a big thanks to the pair of them for writing this module in the first place!"
I would also like to extend my thanks to the developers, and Oleg Bartunov for all of his direction and help with the new features of tsearch2.
OVERVIEW
MS-SQL provides a full text indexing (FTI) system which enables the fast searching of text based fields, very useful for websites (and other applications) that require a results set based on key words. PostgreSQL ships with a contributed module called tsearch2, which implements a special type of index that can also be used for full text indexing. Further more, unlike MS' offering which requires regular incremental rebuilds of the text indexes themselves, tsearch2 indexes are always up-to-date and keeping them so induces very little overhead.
Before we get into the details, it is recommended that you have installed and tested PostgreSQL, are reasonably familiar with databases, the SQL query language and also understand the basics of connecting to PostgreSQL from the local shell. This document isn't intended for the complete PostgreSQL newbie, but anyone with a reasonable grasp of the basics should be able to follow it.
INSTALLATION
Starting with PostgreSQL version 7.4 tsearch2 is now included in the contrib directory with the PostgreSQL sources. contrib/tsearch2 is where you will find everything needed to install and use tsearch2. Please note that tsearch2 will also work with PostgreSQL version 7.3.x, but it is not the module included with the source distribution. You will have to download the module separately and install it in the same fashion.
I installed the tsearch2 module to a PostgreSQL 7.3 database from the contrib directory without squashing the original (old) tsearch module. What I did was move the modules tsearch src driectory into the contrib tree under the name tsearchV2.
Step one is to download the tsearch V2 module :
[http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/] (check Development History for latest stable version !)
tar -zxvf tsearch-v2.tar.gz mv tsearch2 $PGSQL_SRC/contrib/ cd $PGSQL_SRC/contrib/tsearch2
If you are installing from PostgreSQL version 7.4 or higher, you can skip those steps and just change to the contrib/tsearch2 directory in the source tree and continue from there.
As of May 9, 2004 there is a source patch available for tsearch2. The patch provides changes to the pg_ts_ configuration tables to allow for easy dump and restore of a database containing tsearch2. The patch is available here : [http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/regprocedure_7.4.patch.gz]
To apply this patch, download the mentioned file and place it in your postgreSQL source tree ($PGSQL_SRC). This patch is not required for tsearch2 to work. I would however, highly recommend it as it makes the backup and restore procedures very simple.
cd $PGSQL_SRC gunzip regprocedure_7.4.patch.gz patch -b -p1 < regprocedure_7.4.patch
If you have a working version of tsearch2 in your database, you do not need to re-install the tsearch2 module. Just apply the patch and run make. This patch only affects the tsearch2.sql file. You can run the SQL script found : [right here] This script will make the modifications found in the patch, and update the fields from the existing data. From this point on, you can dump and restore the database in a normal fashion. Without this patch, you must follow the instructions later in this document for backup and restore.
This patch is only needed for tsearch2 in PostgreSQL versions 7.3.x and 7.4.x. The patch has been applied to the sources for 8.0.x.
When you have your source tree for tsearch2 ready, you can continue with the regular building and installation process
gmake gmake install gmake installcheck
That is pretty much all you have to do, unless of course you get errors. However if you get those, you better go check with the mailing lists over at http://www.postgresql.org or http://openfts.sourceforge.net/ since its never failed for me.
If you ever need to revert this patch, and go back to the unpatched version of tsearch2, it is simple if you followed the above patch command. The -b option creates a backup of the original file, so we can just copy it back.
cd $PGSQL_SRC/contrib/tsearch2 cp tsearch.sql.in.orig tsearch.sql.in make
If you need the patched version again, just follow the patch instructions again.
The directory in the contib/ and the directory from the archive is called tsearch2. Tsearch2 is completely incompatible with the previous version of tsearch. This means that both versions can be installed into a single database, and migration the new version may be much easier.
NOTE: the previous version of tsearch found in the contrib/tsearch directory is depricated. Although it is still available and included within PostgreSQL version 7.4. It will be removed in version 8.0.
ADDING TSEARCH2 FUNCTIONALITY TO A DATABASE
We should create a database to use as an example for the remainder of this file. We can call the database "ftstest". You can create it from the command line like this:
#createdb ftstest
If you thought installation was easy, this next bit is even easier. Change to the PGSQL_SRC/contrib/tsearch2 directory and type:
psql ftstest < tsearch2.sql
The file "tsearch2.sql" holds all the wonderful little goodies you need to do full text indexing. It defines numerous functions and operators, and creates the needed tables in the database. There will be 4 new tables created after running the tsearch2.sql file : pg_ts_dict, pg_ts_parser, pg_ts_cfg, pg_ts_cfgmap are added.
You can check out the tables if you like:
#psql ftstest ftstest=# \d List of relations Schema | Name | Type | Owner --------+--------------+-------+---------- public | pg_ts_cfg | table | kopciuch public | pg_ts_cfgmap | table | kopciuch public | pg_ts_dict | table | kopciuch public | pg_ts_parser | table | kopciuch (4 rows)
You may need to grant permissions to use on pg_ts_dict, pg_ts_parser, pg_ts_cfg, pg_ts_cfgmap tables to let non-superuser works with tsearch2. GRANT SELECT should be enough for search-only access.
TYPES AND FUNCTIONS PROVIDED BY TSEARCH2
The first thing we can do is try out some of the types that are provided for us. Lets look at the tsvector type provided for us:
SELECT 'Our first string used today'::tsvector; tsvector --------------------------------------- 'Our' 'used' 'first' 'today' 'string' (1 row)
The results are the words used within our string. Notice they are not in any particular order. The tsvector type returns a string of space separated words.
SELECT 'Our first string used today first string'::tsvector; tsvector ----------------------------------------------- 'Our' 'used' 'first' 'today' 'string' (1 row)
Notice the results string has each unique word ('first' and 'string' only appear once in the tsvector value). Which of course makes sense if you are searching the full text ... you only need to know each unique word in the text.
Those examples were just casting a text field to that of type tsvector. Lets check out one of the new functions created by the tsearch2 module.
The function to_tsvector has 3 possible signatures:
to_tsvector(oid, text); to_tsvector(text, text); to_tsvector(text);
We will use the second method using two text fields. The overloaded methods provide us with a way to specifiy the way the searchable text is broken up into words (Stemming process). Right now we will specify the 'default' configuration. See the section on TSEARCH2 CONFIGURATION to learn more about this.
SELECT to_tsvector('default', 'Our first string used today first string'); to_tsvector -------------------------------------------- 'use':4 'first':2,6 'today':5 'string':3,7 (1 row)
The result returned from this function is of type tsvector. The results came about by this reasoning: All of the words in the text passed in are stemmed, or not used because they are stop words defined in our configuration. Each lower case morphed word is returned with all of the positons in the text.
In this case the word "Our" is a stop word in the default configuration. That means it will not be included in the result. The word "first" is found at positions 2 and 6 (although "Our" is a stop word, it's position is maintained). The word(s) positioning is maintained exactly as in the original string. The word "used" is morphed to the word "use" based on the default configuration for word stemming, and is found at position 4. The rest of the results follow the same logic. Just a reminder again ... the order of the 'word' position in the output is not in any kind of order. (ie 'use':4 appears first)
If you want to view the output of the tsvector fields without their positions, you can do so with the function "strip(tsvector)".
SELECT strip(to_tsvector('default', 'Our first string used today first string')); strip -------------------------------- 'use' 'first' 'today' 'string'
If you wish to know the number of unique words returned in the tsvector you can do so by using the function "length(tsvector)"
SELECT length(to_tsvector('default', 'Our first string used today first string')); length -------- 4 (1 row)
Lets take a look at the function to_tsquery. It also has 3 signatures which follow the same rational as the to_tsvector function:
to_tsquery(oid, text); to_tsquery(text, text); to_tsquery(text);
Lets try using the function with a single word :
SELECT to_tsquery('default', 'word'); to_tsquery ----------- 'word' (1 row)
I call the function the same way I would a to_tsvector function, specifying the 'default' configuration for morphing, and the result is the stemmed output 'word'.
Lets attempt to use the function with a string of multiple words:
SELECT to_tsquery('default', 'this is many words'); ERROR: Syntax error
The function can not accept a space separated string. The intention of the to_tsquery function is to return a type of "tsquery" used for searching a tsvector field. What we need to do is search for one to many words with some kind of logic (for now simple boolean).
SELECT to_tsquery('default', 'searching|sentence'); to_tsquery ---------------------- 'search' | 'sentenc' (1 row)
Notice that the words are separated by the boolean logic "OR", the text could contain boolean operators &,|,!,() with their usual meaning.
You can not use words defined as being a stop word in your configuration. The function will not fail ... you will just get no result, and a NOTICE like this:
SELECT to_tsquery('default', 'a|is¬|!the'); NOTICE: Query contains only stopword(s) or doesn't contain lexem(s), ignored to_tsquery ----------- (1 row)
That is a beginning to using the types, and functions defined in the tsearch2 module. There are numerous more functions that I have not touched on. You can read through the tsearch2.sql file built when compiling to get more familiar with what is included.
INDEXING FIELDS IN A TABLE
The next stage is to add a full text index to an existing table. In this example we already have a table defined as follows:
CREATE TABLE tblMessages ( intIndex int4, strTopic varchar(100), strMessage text );
We are assuming there are several rows with some kind of data in them. Any data will do, just do several inserts with test strings for a topic, and a message. here is some test data I inserted. (yes I know it's completely useless stuff ;-) but it will serve our purpose right now).
INSERT INTO tblMessages VALUES ('1', 'Testing Topic', 'Testing message data input'); INSERT INTO tblMessages VALUES ('2', 'Movie', 'Breakfast at Tiffany\'s'); INSERT INTO tblMessages VALUES ('3', 'Famous Author', 'Stephen King'); INSERT INTO tblMessages VALUES ('4', 'Political Topic', 'Nelson Mandella is released from prison'); INSERT INTO tblMessages VALUES ('5', 'Nursery rhyme phrase', 'Little jack horner sat in a corner'); INSERT INTO tblMessages VALUES ('6', 'Gettysburg address quotation', 'Four score and seven years ago' ' our fathers brought forth on this' ' continent a new nation, conceived in' ' liberty and dedicated to the proposition' ' that all men are created equal'); INSERT INTO tblMessages VALUES ('7', 'Classic Rock Bands', 'Led Zeppelin Grateful Dead and The Sex Pistols'); INSERT INTO tblMessages VALUES ('8', 'My birth address', '18 Sommervile road, Regina, Saskatchewan'); INSERT INTO tblMessages VALUES ('9', 'Joke', 'knock knock : who\'s there?' ' I will not finish this joke'); INSERT INTO tblMessages VALUES ('10', 'Computer information', 'My computer is a pentium III 400 mHz' ' with 192 megabytes of RAM');
The next stage is to create a special text index which we will use for FTI, so we can search our table of messages for words or a phrase. We do this using the SQL command:
ALTER TABLE tblMessages ADD COLUMN idxFTI tsvector;
Note that unlike traditional indexes, this is actually a new field in the same table, which is then used (through the magic of the tsearch2 operators and functions) by a special index we will create in a moment.
The general rule for the initial insertion of data will follow four steps:
1. update table 2. vacuum full analyze 3. create index 4. vacuum full analyze
The data can be updated into the table, the vacuum full analyze will reclaim unused space. The index can be created on the table after the data has been inserted. Having the index created prior to the update will slow down the process. It can be done in that manner, this way is just more efficient. After the index has been created on the table, vacuum full analyze is run again to update postgres's statistics (ie having the index take effect).
UPDATE tblMessages SET idxFTI=to_tsvector('default', strMessage); VACUUM FULL ANALYZE;
Note that this only inserts the field strMessage as a tsvector, so if you want to also add strTopic to the information stored, you should instead do the following, which effectively concatenates the two fields into one before being inserted into the table:
UPDATE tblMessages SET idxFTI=to_tsvector('default',coalesce(strTopic,'') ||' '|| coalesce(strMessage,'')); VACUUM FULL ANALYZE;
Using the coalesce function makes sure this concatenation also works with NULL fields.
We need to create the index on the column idxFTI. Keep in mind that the database will update the index when some action is taken. In this case we _need_ the index (The whole point of Full Text INDEXINGi ;-)), so don't worry about any indexing overhead. We will create an index based on the gist function. GiST is an index structure for Generalized Search Tree.
CREATE INDEX idxFTI_idx ON tblMessages USING gist(idxFTI); VACUUM FULL ANALYZE;
After you have converted all of your data and indexed the column, you can select some rows to see what actually happened. I will not display output here but you can play around yourselves and see what happened.
The last thing to do is set up a trigger so every time a row in this table is changed, the text index is automatically updated. This is easily done using:
CREATE TRIGGER tsvectorupdate BEFORE UPDATE OR INSERT ON tblMessages FOR EACH ROW EXECUTE PROCEDURE tsearch2(idxFTI, strMessage);
Or if you are indexing both strMessage and strTopic you should instead do:
CREATE TRIGGER tsvectorupdate BEFORE UPDATE OR INSERT ON tblMessages FOR EACH ROW EXECUTE PROCEDURE tsearch2(idxFTI, strTopic, strMessage);
Before you ask, the tsearch2 function accepts multiple fields as arguments so there is no need to concatenate the two into one like we did before.
If you want to do something specific with columns, you may write your very own trigger function using plpgsql or other procedural languages (but not SQL, unfortunately) and use it instead of tsearch2 trigger.
You could however call other stored procedures from within the tsearch2 function. Lets say we want to create a function to remove certain characters (like the @ symbol from all text).
CREATE FUNCTION dropatsymbol(text) RETURNS text AS 'select replace($1, \'@\', \' \');' LANGUAGE SQL;
Now we can use this function within the tsearch2 function on the trigger.
DROP TRIGGER tsvectorupdate ON tblmessages; CREATE TRIGGER tsvectorupdate BEFORE UPDATE OR INSERT ON tblMessages FOR EACH ROW EXECUTE PROCEDURE tsearch2(idxFTI, dropatsymbol, strMessage); INSERT INTO tblmessages VALUES (69, 'Attempt for dropatsymbol', 'Test@test.com');
If at this point you receive an error stating: ERROR: Can't find tsearch config by locale
Do not worry. You have done nothing wrong. And tsearch2 is not broken. All that has happened here is that the configuration is setup to use a configuration based on the locale of the server. All you have to do is change your default configuration, or add a new one for your specific locale. See the section on TSEARCH2 CONFIGURATION.
SELECT * FROM tblmessages WHERE intindex = 69; intindex | strtopic | strmessage | idxfti ----------+--------------------------+---------------+----------------------- 69 | Attempt for dropatsymbol |<EMAIL_ADDRESS>| 'test':1 'test.com':2 (1 row)Notice that the string content was passed throught the stored procedure dropatsymbol. The '@' character was replaced with a single space ... and the output from the procedure was then stored in the tsvector column.
This could be useful for removing other characters from indexed text, or any kind of preprocessing needed to be done on the text prior to insertion into the index.
QUERYING A TABLE
There are some examples in the README.tsearch2 file for querying a table. One major difference between tsearch and tsearch2 is the operator ## is no longer available. Only the operator @@ is defined, using the types tsvector on one side and tsquery on the other side.
Lets search the indexed data for the word "Test". I indexed based on the the concatenation of the strTopic, and the strMessage:
SELECT intindex, strtopic FROM tblmessages WHERE idxfti @@ 'test'::tsquery; intindex | strtopic ----------+--------------- 1 | Testing Topic (1 row)
The only result that matched was the row with a topic "Testing Topic". Notice that the word I search for was all lowercase. Let's see what happens when I query for uppercase "Test".
SELECT intindex, strtopic FROM tblmessages WHERE idxfti @@ 'Test'::tsquery; intindex | strtopic ----------+---------- (0 rows)
We get zero rows returned. The reason is because when the text was inserted, it was morphed to my default configuration (because of the call to to_tsvector in the UPDATE statement). If there was no morphing done, and the tsvector field(s) contained the word 'Text', a match would have been found.
Most likely the best way to query the field is to use the to_tsquery function on the right hand side of the @@ operator like this:
SELECT intindex, strtopic FROM tblmessages WHERE idxfti @@ to_tsquery('default', 'Test | Zeppelin'); intindex | strtopic ----------+-------------------- 1 | Testing Topic 7 | Classic Rock Bands (2 rows)
That query searched for all instances of "Test" OR "Zeppelin". It returned two rows: the "Testing Topic" row, and the "Classic Rock Bands" row. The to_tsquery function performed the correct morphology upon the parameters, and searched the tsvector field appropriately.
The last example here relates to searching for a phrase, for example "minority report". This poses a problem with regard to tsearch2, as it doesn't index phrases, only words. But there is a way around which doesn't appear to have a significant impact on query time, and that is to use a query such as the following:
SELECT intindex, strTopic FROM tblmessages WHERE idxfti @@ to_tsquery('default', 'gettysburg & address') AND strMessage ~* '.*men are created equal.*'; intindex | strtopic ----------+------------------------------ 6 | Gettysburg address quotation (1 row) SELECT intindex, strTopic FROM tblmessages WHERE idxfti @@ to_tsquery('default', 'gettysburg & address') AND strMessage ~* '.*something that does not exist.*'; intindex | strtopic ----------+---------- (0 rows)
Of course if your indexing both strTopic and strMessage, and want to search for this phrase on both, then you will have to get out the brackets and extend this query a little more.
TSEARCH2 CONFIGURATION
Some words such as "and", "the", and "who" are automatically not indexed, since they belong to a pre-existing dictionary of "Stop Words" which tsearch2 does not perform indexing on. If someone needs to search for "The Who" in your database, they are going to have a tough time coming up with any results, since both are ignored in the indexes. But there is a solution.
Lets say we want to add a word into the stop word list for english stemming. We could edit the file :'/usr/local/pgsql/share/english.stop' and add a word to the list. I edited mine to exclude my name from indexing:
- Edit /usr/local/pgsql/share/english.stop - Add 'andy' to the list - Save the file.
When you connect to the database, the dict_init procedure is run during initialization. And in my configuration it will read the stop words from the file I just edited. If you were connected to the DB while editing the stop words, you will need to end the current session and re-connect. When you re-connect to the database, 'andy' is no longer indexed:
SELECT to_tsvector('default', 'Andy'); to_tsvector ------------ (1 row)
Originally I would get the result :
SELECT to_tsvector('default', 'Andy'); to_tsvector ------------ 'andi':1 (1 row)
But since I added it as a stop word, it would be ingnored on the indexing. The stop word added was used in the dictionary "en_stem". If I were to use a different configuration such as 'simple', the results would be different. There are no stop words for the simple dictionary. It will just convert to lower case, and index every unique word.
SELECT to_tsvector('simple', 'Andy andy The the in out'); to_tsvector ------------------------------------- 'in':5 'out':6 'the':3,4 'andy':1,2 (1 row)
All this talk about which configuration to use is leading us into the actual configuration of tsearch2. In the examples in this document the configuration has always been specified when using the tsearch2 functions:
SELECT to_tsvector('default', 'Testing the default config'); SELECT to_tsvector('simple', 'Example of simple Config');
The pg_ts_cfg table holds each configuration you can use with the tsearch2 functions. As you can see the ts_name column contains both the 'default' configurations based on the 'C' locale. And the 'simple' configuration which is not based on any locale.
SELECT * from pg_ts_cfg; ts_name | prs_name | locale -----------------+----------+-------------- default | default | C default_russian | default | ru_RU.KOI8-R simple | default | (3 rows)
Each row in the pg_ts_cfg table contains the name of the tsearch2 configuration, the name of the parser to use, and the locale mapped to the configuration. There is only one parser to choose from the table pg_ts_parser called 'default'. More parsers could be written, but for our needs we will use the default.
There are 3 configurations installed by tsearch2 initially. If your locale is set to 'en_US' for example (like my laptop), then as you can see there is currently no dictionary configured to use with that locale. You can either set up a new configuration or just use one that already exists. If I do not specify which configuration to use in the to_tsvector function, I receive the following error.
SELECT to_tsvector('learning tsearch is like going to school'); ERROR: Can't find tsearch config by locale
We will create a new configuration for use with the server encoding 'en_US'. The first step is to add a new configuration into the pg_ts_cfg table. We will call the configuration 'default_english', with the default parser and use the locale 'en_US'.
INSERT INTO pg_ts_cfg (ts_name, prs_name, locale) VALUES ('default_english', 'default', 'en_US');
We have only declared that there is a configuration called 'default_english'. We need to set the configuration of how 'default_english' will work. The next step is creating a new dictionary to use. The configuration of the dictionary is completlely different in tsearch2. In the prior versions to make changes, you would have to re-compile your changes into the tsearch.so. All of the configuration has now been moved into the system tables created by executing the SQL code from tsearch2.sql
Lets take a first look at the pg_ts_dict table
ftstest=# \d pg_ts_dict Table "public.pg_ts_dict" Column | Type | Modifiers -----------------+--------------+----------- dict_name | text | not null dict_init | regprocedure | dict_initoption | text | dict_lexize | regprocedure | not null dict_comment | text | Indexes: pg_ts_dict_pkey primary key btree (dict_name)
The dict_name column is the name of the dictionary, for example 'simple', 'en_stem' or 'ru_stem'. The dict_init column is a text representation of a stored procedure to run for initialization of that dictionary, for example 'snb_en_init(text)' or 'snb_ru_init(text)'. The initial configuration of tsearch2 had the dict_init and dict_lexize columns as type oid. The patch mentioned in the Installation Notes changes these types to regprocedure. The data inserted, or updated can still be the oid of the stored procedure. The representation is just different. This makes backup and restore procedures much easier for tsearch2. The dict_init option is used for options passed to the init function for the stored procedure. In the cases of 'en_stem' or 'ru_stem' it is a path to a stopword file for that dictionary, for example '/usr/local/pgsql/share/english.stop'. This is however dictated by the dictionary. ISpell dictionaries may require different options. The dict_lexize column is another OID of a stored procedure to the function used to lexize, for example 'snb_lexize(internal, internal, integer)'. The dict_comment column is just a comment.
Next we will configure the use of a new dictionary based on ISpell. We will assume you have ISpell installed on you machine. (in /usr/local/lib)
There has been some confusion in the past as to which files are used from ISpell. ISpell operates using a hash file. This is a binary file created by the ISpell command line utility "buildhash". This utility accepts a file containing the words from the dictionary, and the affixes file and the output is the hash file. The default installation of ISPell installs the english hash file english.hash, which is the exact same file as american.hash. ISpell uses this as the fallback dictionary to use.
This hash file is not what tsearch2 requires as the ISpell interface. The file(s) needed are those used to create the hash. Tsearch uses the dictionary words for morphology, so the listing is needed not spellchecking. Regardless, these files are included in the ISpell sources, and you can use them to integrate into tsearch2. This is not complicated, but is not very obvious to begin with. The tsearch2 ISpell interface needs only the listing of dictionary words, it will parse and load those words, and use the ISpell dictionary for lexem processing.
I found the ISPell make system to be very finicky. Their documentation actually states this to be the case. So I just did things the command line way. In the ISpell source tree under languages/english there are several files in this directory. For a complete description, please read the ISpell README. Basically for the english dictionary there is the option to create the small, medium, large and extra large dictionaries. The medium dictionary is recommended. If the make system is configured correctly, it would build and install the english.has file from the medium size dictionary. Since we are only concerned with the dictionary word listing ... it can be created from the /languages/english directory with the following command:
sort -u -t/ +0f -1 +0 -T /usr/tmp -o english.med english.0 english.1
This will create a file called english.med. You can copy this file to whever you like. I placed mine in /usr/local/lib so it coincides with the ISpell hash files. You can now add the tsearch2 configuration entry for the ISpell english dictionary. We will also continue to use the english word stop file that was installed for the en_stem dictionary. You could use a different one if you like. The ISpell configuration is based on the "ispell_template" dictionary installed by default with tsearch2. We will use the OIDs to the stored procedures from the row where the dict_name = 'ispell_template'.
INSERT INTO pg_ts_dict (SELECT 'en_ispell', dict_init, 'DictFile="/usr/local/lib/english.med",' 'AffFile="/usr/local/lib/english.aff",' 'StopFile="/usr/local/pgsql/share/contrib/english.stop"', dict_lexize FROM pg_ts_dict WHERE dict_name = 'ispell_template');
Now that we have a dictionary we can specify it's use in a query to get a lexem. For this we will use the lexize function. The lexize function takes the name of the dictionary to use as an argument. Just as the other tsearch2 functions operate. You will need to stop your psql session and start it again in order for this modification to take place.
SELECT lexize('en_ispell', 'program'); lexize ----------- {program} (1 row)
If you wanted to always use the ISpell english dictionary you have installed, you can configure tsearch2 to always use a specific dictionary.
SELECT set_curdict('en_ispell');
Lexize is meant to turn a word into a lexem. It is possible to receive more than one lexem returned for a single word.
SELECT lexize('en_ispell', 'conditionally'); lexize ----------------------------- {conditionally,conditional} (1 row)
The lexize function is not meant to take a full string as an argument to return lexems for. If you passed in an entire sentence, it attempts to find that entire sentence in the dictionary. Since the dictionary contains only words, you will receive an empty result set back.
SELECT lexize('en_ispell', 'This is a senctece to lexize'); lexize -------- (1 row) If you parse a lexem from a word not in the dictionary, then you will receive an empty result. This makes sense because the word "tsearch" is not in the english dictionary. You can create your own additions to the dictionary if you like. This may be useful for scientific or technical glossaries that need to be indexed. SELECT lexize('en_ispell', 'tsearch'); lexize -------- (1 row)
This is not to say that tsearch will be ignored when adding text information to the the tsvector index column. This will be explained in greater detail with the table pg_ts_cfgmap.
Next we need to set up the configuration for mapping the dictionay use to the lexxem parsings. This will be done by altering the pg_ts_cfgmap table. We will insert several rows, specifying to use the new dictionary we installed and configured for lexizing within tsearch2. There are several type of lexims we would be concerned with forcing the use of the ISpell dictionary.
INSERT INTO pg_ts_cfgmap (ts_name, tok_alias, dict_name) VALUES ('default_english', 'lhword', '{en_ispell,en_stem}'); INSERT INTO pg_ts_cfgmap (ts_name, tok_alias, dict_name) VALUES ('default_english', 'lpart_hword', '{en_ispell,en_stem}'); INSERT INTO pg_ts_cfgmap (ts_name, tok_alias, dict_name) VALUES ('default_english', 'lword', '{en_ispell,en_stem}');
We have just inserted 3 records to the configuration mapping, specifying that the lexem types for "lhword, lpart_hword and lword" are to be stemmed using the 'en_ispell' dictionary we added into pg_ts_dict, when using the configuration ' default_english' which we added to pg_ts_cfg.
There are several other lexem types used that we do not need to specify as using the ISpell dictionary. We can simply insert values using the 'simple' stemming process dictionary.
INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'url', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'host', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'sfloat', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'uri', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'int', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'float', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'email', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'word', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'hword', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'nlword', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'nlpart_hword', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'part_hword', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'nlhword', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'file', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'uint', '{simple}'); INSERT INTO pg_ts_cfgmap VALUES ('default_english', 'version', '{simple}');
Our addition of a configuration for 'default_english' is now complete. We have successfully created a new tsearch2 configuration. At the same time we have also set the new configuration to be our default for en_US locale.
SELECT to_tsvector('default_english', 'learning tsearch is like going to school'); to_tsvector -------------------------------------------------- 'go':5 'like':4 'learn':1 'school':7 'tsearch':2 SELECT to_tsvector('learning tsearch is like going to school'); to_tsvector -------------------------------------------------- 'go':5 'like':4 'learn':1 'school':7 'tsearch':2 (1 row)
Notice here that words like "tsearch" are still parsed and indexed in the tsvector column. There is a lexem returned for the word becuase in the configuration mapping table, we specify words to be used from the 'en_ispell' dictionary first, but as a fallback to use the 'en_stem' dictionary. Therefore a lexem is not returned from en_ispell, but is returned from en_stem, and added to the tsvector.
SELECT to_tsvector('learning tsearch is like going to computer school'); to_tsvector --------------------------------------------------------------------------- 'go':5 'like':4 'learn':1 'school':8 'compute':7 'tsearch':2 'computer':7 (1 row)
Notice in this last example I added the word "computer" to the text to be converted into a tsvector. Because we have setup our default configuration to use the ISpell english dictionary, the words are lexized, and computer returns 2 lexems at the same position. 'compute':7 and 'computer':7 are now both indexed for the word computer.
You can create additional dictionary lists, or use the extra large dictionary from ISpell. You can read through the ISpell documents, and source tree to make modifications as you see fit.
In the case that you already have a configuration set for the locale, and you are changing it to your new dictionary configuration. You will have to set the old locale to NULL. If we are using the 'C' locale then we would do this:
UPDATE pg_ts_cfg SET locale=NULL WHERE locale = 'C';
That about wraps up the configuration of tsearch2. There is much more you can do with the tables provided. This was just an introduction to get things working rather quickly.
ADDING NEW DICTIONARIES TO TSEARCH2
To aid in the addition of new dictionaries to the tsearch2 module you can use another additional module in combination with tsearch2. The gendict module is included into tsearch2 distribution and is available from gendict/ subdirectory.
I will not go into detail about installation and instructions on how to use gendict to it's fullest extent right now. You can read the README.gendict ... it has all of the instructions and information you will need.
BACKING UP AND RESTORING DATABASES THAT FEATURE TSEARCH2
Never rely on anyone elses instructions to backup and restore a database system, always develop and understand your own methodology, and test it numerous times before you need to do it for real.
The backup and restore procedure has changed over time. This is not meant to be the bible for tsearch2 back up and restore. Please read all sections so you have a complete understanding of some backup and restore issues. Please test your own procedures, and do not rely on these instructions solely.
If you come accross some issues in your own procedures, please feel free to bring the question up on the Open-FTS, and PostgreSQL mailing lists.
ORIGINAL BACKUP PROCEDURES
Originally, tsearch2 had problems when using the pg_dump, and or the pg_dumpall utilities. The problem lies within the original use of OIDs for column types. Since OIDs are not consistent accross pg_dumps, when you reload the data values into the pg_ts_dict table, for example, those oids no longer point to anything. You would then end up trying to use a "broken" tsearch2 configuration.
The solution was to backup and restore a database using the tsearch2 module into small unique parts, and then load them in the correct order. You would have to edit the schema and remove the tsearch stored procedure references in the sql file. You would have to load your global objects, then the tsearch2 objects. You had to re-create the tsearch module before restoring your schema so no conflicts would arise. Then you could restore your data (all schemas, and types needed for the data were now available).
The original backup instructions were as follows
1) Backup any global database objects such as users and groups (this step is usually only necessary when you will be restoring to a virgin system)
pg_dumpall -g > GLOBALobjects.sql
2) Backup the full database schema using pg_dump
pg_dump -s DATABASE > DATABASEschema.sql
3) Backup the full database using pg_dump
pg_dump -Fc DATABASE > DATABASEdata.tar
The original restore procedures were as follows
1) Create the blank database
createdb DATABASE
2) Restore any global database objects such as users and groups (this step is usually only necessary when you will be restoring to a virgin system)
psql DATABASE < GLOBALobjects.sql
3) Create the tsearch2 objects, functions and operators
psql DATABASE < tsearch2.sql
4) Edit the backed up database schema and delete all SQL commands which create tsearch2 related functions, operators and data types, BUT NOT fields in table definitions that specify tsvector types. If your not sure what these are, they are the ones listed in tsearch2.sql. Then restore the edited schema to the database
psql DATABASE < DATABASEschema.sql
5) Restore the data for the database
pg_restore -N -a -d DATABASE DATABASEdata.tar
If you get any errors in step 4, it will most likely be because you forgot to remove an object that was created in tsearch2.sql. Any errors in step 5 will mean the database schema was probably restored wrongly.
Issues with this procedure
As I mentioned before, it is vital that you test out your own backup and restore procedures. These procedures were originally adopted from this document's orignal author. Robert John Shepherd. It makes use of the pg_dump custom archive functionality. I am not that familiar with the formatting output of pg_dump, and using pg_restore. I have always had the luxury of using text files (Everything is DATABASE.sql).
One issue not forseen in the case of using a binary dump is the when you have added more than the default tsearch2 configurations. Upon reload of the data it will fail due to duplicate primary keys. If you load the tsearch2 module, and then delete the data loaded by tsearch2 into the configuration tables, the data will restore. The configurations are incorrect because you can not remove the data using OID references from the custom archive.
It would be very simple to fix this problem if the data was not in an archive format. I do believe all of your data would have been restored properly and you can get things working fairly easy. All one would have to do is create the configurations as in the tsearch2.sql file. And then create your custom configurations again.
I have read in the pg_dump man page that if the tar archive format is used, it is possible to limit which data is restored using pg_restore. If anyone has more experience with pg_dump archives, and pg_restore. Please feel free to test and contribute your procedure(s).
CURRENT BACKUP AND RESTORE PROCEDURES
Currently a backup and restore of a database using the tsearch2 module can be quite simple. If you have applied the patch mentioned in the installation instructions prior to tsearch2 installation. This patch removes the use of the oid column. The text representation for the stored procedures used are dumped with the data and the restoration of the data works seemlessly.
1) to backup the database
pg_dump DATABASE > DATABASE.sql
1) to restore the database
createdb DATABASE psql -d DATABASE -f DATABASE.sql
This procedure is now like any normal backup and restore procedure. I can say whether this has been proven using the pg_dump archive, and restoring with pg_restore. In theory there should be no problems with any format after the patch is applied.
This restoration procedure should never be an issue with the patch applied to version 8.0 of PostgreSQL. Only versions 7.3 and 7.4 are affected. You can avoid any troubles by applying the patch prior to installation, or running the SQL script provided to live database before backup and restoring is done.


Right-Now Weather Dashboard
Search
---
Temperature
Humidity
Wind Speed
5 Day Forecast:
Date
Temperature
Humidity
Date
Temperature
Humidity
Date
Temperature
Humidity
Date
Temperature
Humidity
Date
Temperature
Humidity


Bill Gates WinHEC 2003 Keynote
Here. "With the next round of Windows, which we call Longhorn, this idea of the completely new presentation system and the standard user interface guideline requiring that rich graphics capability will take a leap forward and fully exploit the extra graphics capabilities. In fact, the way the experience will work is that people with even today's hardware will want to move up to the advanced graphics capabilities."


In this Decision making section , when a developer writing code he spifices one or more conditions to be checked or tested by the program , if that following condition is true then it excutes then a statement is executed or the other statement is executed if it false.
C programming langauge provides the following types of Decision making statement :
if statement
In this expression ,if the condition is true then the statement inside the 'if ' condition will get excuted otherwise it is skipped.
Syntax :
if(condition){ statement; }
if else statement
In this expression , if statement can be followed by an optional else statement. If the first condition faills ,then the else statement is executed.
Syntax :
if(condition){ statement; }else{ statemnet; }
nested if else if
It is the next version of the if else statemnet , in this expression we have multiple if else conditions.It goes checking the condition until the correct condition is found and excutes that statement.
if(condition){ statement; }else if(condition){ statement; } // we can have multiple condition . . . . else{ statement; }
switch statement
It is an alternative to nested if else statement which allows us to check value of the user input or some data with multiple data for given single varible.Here, We can define various statements in the multiple cases for the different values of a single variable.
switch(expression){ case value1 : //statement break; case value2: //statement; break; . . . // If no cases match then the default case is executed default : //statement break; }


JavaScript's Beauty
JavaScript is designed to work well across browsers and devices. It's compatible with the latest devices. Our tablets, notebooks, and smart phones all use some type of browser in order to view web applications. Therefore, it is beneficial to have a web application built using a programming language such as Javascript so that we can easily view/access a web application, in all of its beauty, across all devices.
JavaScript is also designed to look into databases and present that data (e.g. pricing for various products, part numbers). Companies, as opposed to individuals, which need to gather and understand their own data find this type of programming language very useful.
Node.js is a recently developed software platform which allows developers to create a high-performance web application using one language: JavaScript. This makes the work of a developer easier because they are only working with / editing one language. It also reduces development time which benefits the end-client and/or user (you).
JavaScript makes static HTML5/CSS dyanmic. HTML5 and CSS create the structure and style of a web page and JavaScript creates movement; that is, animations, 3-D graphics, and games within that web page. Basically, JavaScript is written or embedded in teh HTML page, where it can interact with that page's Document Object Model, thereby allowing the page to perform in a more dynamic and interactive way.
JavaScript is a fast programming language.
For programmers, JavaScript has the added benefit of being accessible offline. This is helpful in conducting work in remote locations.
JavaScript's Criticisms
JavaScript is vulnerable to cross-site scripting (XSS). This vulnerability allows attackers to manipulate websites and return un-intended and malicious script to visitors of the site.


Feel Your Journal
Rememberly is so easy on the eyes that you don't even miss the feel of a physical journal.
Organized and Fast
All your entries are just a few taps away, organized and intuitive.
Read Along Mode
Rememberly offers a dedicated reading mode for focused reading experience.


Introduced in: 14 Aug 2008 by asceth<EMAIL_ADDRESS>Authors: asceth (machinist@asceth.com).
hostport() = {host, string(), port, integer()}
Tuple describing a host and port to connect to
memcached_connection() = hostport() | socket()
memcached_key() = list() | atom()
socket() = {socket, port()}
Tuple describing an existing socket
| add/3 | Associate Bytes with Key if Key isn't set already in memcached. |
| add/5 | Associate Bytes with Key using Flags and Expire options if Key isn't set already in memcached. |
| delete/2 | Delete a key from memcached. |
| delete/3 | Delete a key from memcached after Time seconds. |
| get/2 | Return value associated with Key. |
| replace/3 | Associate Bytes with Key if Key is set already in memcached. |
| replace/5 | Associate Bytes with Key using Flags and Expire options if Key is set already in memcached. |
| set/3 | Associate Bytes with Key. |
| set/5 | Associate Bytes with Key using Flags and Expire options. |
| stats/1 | Delete a key from memcached. |
add(X1::memcached_connection(), Key::memcached_key(), Bytes::any()) -> ok | {error, not_stored}
Associate Bytes with Key if Key isn't set already in memcached.
add(X1::memcached_connection(), Key::memcached_key(), Flags::integer(), Expire::integer(), Bytes::any()) -> ok | {error, not_stored}
Associate Bytes with Key using Flags and Expire options if Key isn't set already in memcached.
delete(X1::memcached_connection(), Key::memcached_key()) -> ok | {error, not_found}
Delete a key from memcached
delete(X1::memcached_connection(), Key::memcached_key(), Time::integer()) -> ok | {error, not_found}
Delete a key from memcached after Time seconds
get(X1::memcached_connection(), Key::memcached_key() | [Key::memcached_key()]) -> [any()]
Return value associated with Key. Will automatically convert back to erlang terms. Key can be a single key or a list of keys.
replace(X1::memcached_connection(), Key::memcached_key(), Bytes::any()) -> ok | {error, not_stored}
Associate Bytes with Key if Key is set already in memcached.
replace(X1::memcached_connection(), Key::memcached_key(), Flags::integer(), Expire::integer(), Bytes::any()) -> ok | {error, not_stored}
Associate Bytes with Key using Flags and Expire options if Key is set already in memcached.
set(X1::memcached_connection(), Key::memcached_key(), Bytes::any()) -> ok | {error, not_stored}
Associate Bytes with Key.
set(X1::memcached_connection(), Key::memcached_key(), Flags::integer(), Expire::integer(), Bytes::any()) -> ok | {error, not_stored}
Associate Bytes with Key using Flags and Expire options.
stats(X1::memcached_connection()) -> [string()]
Delete a key from memcached
Generated by EDoc, Jan 15 2009, 17:20:37.


FROM:MR.DAN WAHILI. AFRICAN DEVELOPEMENT BANK BURKINA FASO (ADB) OUAGADOUGOU-BURKINA FASO. DEAR FRIEND, IT WILL BE A SURPRISE FOR YOU TO RECEIVE THIS MAIL. WELCOME THIS LETTER IN THE NAME OF ALLAH. I AM MR.DAN WAHILI,THE DIRECTOR OF THE ACCOUNTS & AUDITING DEPT.AT THE AFRICAN DEVLOPEMENT BANK OUAGADOUGOU-WEST AFRICA. WITH DUE RESPECT I HAVE DECIDED TO CONTACT YOU ON A BUSINESS TRANSACTION THAT WILL BE BENEFICIAL TO BOTH OF US. AT THE BANK'S LAST ACCOUNTS/AUDITING EVALUATIONS,MY STAFFS CAME ACROSS AN OLD ACCOUNT WHICH WAS BEING MAINTAINED BY A FOREIGN CLIENT WHO WE LEARNT WAS AMONG THE DECEASED PASSENGERS OF AN AIRLINE CRASH ON 6TH NOV. 2003.SINCE THE DECEASED WAS UNABLE TO RUN THIS ACCOUNT SINCE HIS DEATH.THE ACCOUNT HAS REMAINED DORMANT WITHOUT THE KNOWLEDGE OF HIS FAMILY SINCE IT WAS PUT IN A SAFE DEPOSIT ACCOUNT IN THE BANK FOR FUTURE INVESTMENT BY THE CLIENT. SINCE HIS DEMISE,NOBODY,NOT EVEN THE MEMBERS OF HIS FAMILY HAVE APPLIED FOR CLAIMS OVER THIS FUND AND IT HAS BEEN IN THE SAFE DEPOSIT ACCOUNT UNTIL WE DISCOVERED THAT IT CANNOT BE CLAIMED SINCE OUR CLIENT IS A FOREIGN NATIONAL AND WE ARE SURE THAT HE HAS NO NEXT OF KIN HERE TO FILE CLAIMS OVER THE MONEY.AS THE DIRECTOR OF THE DEPT,THIS DISCOVERY WAS BROUGHT TO MY OFFICE SO AS TO DECIDE WHAT IS TO BE DONE.WITH THE FEW PERSONEL IN MY DEPT,WE DECIDED TO SEEK WAYS THROUGH WHICH TO TRANSFER THIS MONEY OUT OF THE BANK AND OUT OF THE COUNTRY TOO.   WE WILL PLEASE WISH YOU KEEP THIS TRANSACTION SECRET AS WE ARE HOPING TO RETIRE WITH OUR SHARE OF THIS MONEY AT THE END OF TRANSACTION WHICH WILL BE WHEN THIS MONEY IS SAFELY IN YOUR ACCOUNT. WE WILL THEN COME OVER TO YOUR COUNTRY FOR SHARING ACCORDING TO THE PREVIOUSLY AGREED PERCENTAGES.YOU MIGHT EVEN HAVE TO ADVISE US ON POSSIBILITIES OF INVESTMENT IN YOUR COUNTRY OR ELSEWHERE OF OUR CHOICE.MAY ALLAH HELP YOU TO HELP US TO A RESTIVE RETIREMENT.AMIN. PLEASE FOR FURTHER INFORMATIONS AND ENQUIRIES FEEL FREE TO CONTACT ME THROUGH MY NUMBER 00 226 78 84 92 06 FOR ORAL DISCUSSION. AM AWAITING FOR YOUR URGENT RESPONSE!!! THANKS AND REMAIN BLESSED. MR.DAN WAHILI _________________________________________________________________ MSN Hotmail : créez votre adresse e-mail gratuite & à vie ! http://www.msn.fr/newhotmail/Default.asp?Ath=f


{{title|safe}}
Date
Update View
Toggle Dropdown
Choose View Type
Location
Sensor and Location
Activity


It would be able to provide appreciable assistance to the blind
ZIP code, for example, would contain only the characters 0-9. This fact would reduce the number of possible identifications.
This can help us further in automatic mail sorting, automatic processing of administrative files and in other fields.
Easy to use, no documentation required!
Put an image, video, animation, or anything else in the screen!
As always, this theme is free to download and use for any purpose!
Since this theme is MIT licensed, you can use it commercially!
Alphabetic recognition refers to the ability to visually recognize letters of the alphabet. This includes the ability to name each letter and match that letter name with its written form, both upper and lower case, and in both manuscript and cursive. Alphabet recognition also includes the ability to distinguish between similar looking letters, like C/G, M/W, d/b and p/q. It is foundational to any alphabetic based language.
Rihab Eddahech
Computer Science Student Engineer with a good technical background in software design and development. Strong teamwork, collaboration and leadership skills. Confident, friendly and easy to get along with.
Abyr Barhoumi
Amet consetetur sadipscing elitrdiam nonumy eirmod tempor invidunt ut labore et dolore.
Established in 1984, ENSI (the National School of Computer Science) is one of the most prestigious area of engineering school specializing in computer Tunisia.
ENSI is a regional class engineering college where teaching and research - with relevance to the practical world as a guiding principle - continue to be its main goal. The mission of the ENSI is to advance knowledge and educate students in science, technology and other fields of study that will best serve Tunisia in the 21st century. The diplomas obtained in ENSI include Engineering degree in Computer Science , Degree of Master of Science and PhD degree in Computer Science .


In the last section, I ended my little network stack at this point:
// Something that can be fetched from the API
protocol Fetchable: Decodable {
static var apiBase: String { get } // The part of the URL for this type
}
// A client that fetches things from the API
final class APIClient {
let baseURL = URL(string: "https://www.example.com")!
let session: URLSession = URLSession.shared
// Fetch any Fetchable type given an ID, and return it asynchronously
func fetch<Model: Fetchable>(_ model: Model.Type, id: Int,
completion: @escaping (Result<Model, Error>) -> Void)
{
// Construct the URLRequest
let url = baseURL
.appendingPathComponent(Model.apiBase)
.appendingPathComponent("\(id)")
let urlRequest = URLRequest(url: url)
// Send it to URLSession
let task = session.dataTask(with: urlRequest) { (data, _, error) in
if let error = error {
completion(.failure(error))
} else if let data = data {
let result = Result { try JSONDecoder().decode(Model.self, from: data) }
completion(result)
}
}
task.resume()
}
}
This can decode any Fetchable model from an API endpoint that has a URL something like https://<base>/<model>/<id>
. That’s pretty good, but we can do a lot better. A natural first question is “how do I test it?” It relies explicitly on URLSession, which is very hard to test against. A natural approach would be to create a protocol to mock URLSession.

The basic premise of a mock is to build a test object that mimics some other object you want to replace. That encourages you to design a protocol that very closely matches the existing interface, and then your “mock object” will also closely match that interface. This makes your “real” object, the protocol, and the mock evolve tightly in lockstep, and it cuts off opportunities for more powerful protocols that aren’t tied to one implementation. If the only reason you can imagine using a protocol is for testing, then you’re not getting all you could out of it. Protocols can be so much more.
So my goal isn’t to “mock” URLSession, but to abstract the functionality I need. What I want is to map a URLRequest to Data, asynchronously:1
// A transport maps a URLRequest to Data, asynchronously
protocol Transport {
func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
}
Notice that nothing about that says “HTTP server over the network.” Anything that can map a URLRequest to Data asynchronously is fine. It could be a database. It could be static unit test data. It could be flat files. It could be different routes depending on the scheme.
Now comes the power of retroactive modeling. I can extend URLSession to be a Transport:
extension URLSession: Transport {
func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
{
let task = self.dataTask(with: request) { (data, _, error) in
if let error = error { completion(.failure(error)) }
else if let data = data { completion(.success(data)) }
}
task.resume()
}
}
And then anything that requires a Transport can use a URLSession directly. No need for wrappers or adapters. It just works, even though URLSession is a Foundation type and Apple doesn’t know anything about my Transport protocol. A few lines of code and it just works, without giving up any of the power of URLSession.
With that in place, APIClient
can use Transport rather than URLSession.
final class APIClient { let baseURL = URL(string: "https://www.example.com")! let transport: Transport init(transport: Transport = URLSession.shared) { self.transport = transport } // Fetch any Fetchable type given an ID, and return it asynchronously func fetch<Model: Fetchable>(_ model: Model.Type, id: Int, completion: @escaping (Result<Model, Error>) -> Void) { // Construct the URLRequest let url = baseURL .appendingPathComponent(Model.apiBase) .appendingPathComponent("\(id)") let urlRequest = URLRequest(url: url) // Send it to the transport transport.send(request: urlRequest) { data in let result = Result { try JSONDecoder().decode(Model.self, from: data.get()) } completion(result) } } }
By using a default value in init
, callers can still use the APIClient()
syntax if they want the standard network transport.
Transport is a lot more powerful than just “a URLSession mock.” It’s a function that converts URLRequests into Data. That means it can be composed. I can build a Transport that wraps other Transports. For example, I can build a Transport that adds headers to every request.
// Add headers to an existing transport
final class AddHeaders: Transport
{
let base: Transport
var headers: [String: String]
init(base: Transport, headers: [String: String]) {
self.base = base
self.headers = headers
}
func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
{
var newRequest = request
for (key, value) in headers { newRequest.addValue(value, forHTTPHeaderField: key) }
base.send(request: newRequest, completion: completion)
}
}
let transport = AddHeaders(base: URLSession.shared,
headers: ["Authorization": "..."])
Now, rather than having every request deal with authorization, that can be centralized to a single Transport transparently. If the authorization token changes, then I can update a single object, and all future requests will get the right headers. But this is still unit testable (even the AddHeaders part). I can swap in whatever lower-level Transport I want.
This means I can extend existing systems in a really flexible way. I can add encryption or logging or caching or priority queues or automatic retries or whatever without intermingling that with the actual network layer. I can tunnel all the network traffic over a custom VPN protocol (I’ve done exactly that with a system like this), all without losing the ability to unit test. So yes, I get mocks, yes, I get unit testing, but I get so much more.
For completeness, here’s a “mock” Transport, but it’s probably the least interesting thing we can do with this protocol.
// A transport that returns static values for tests
enum TestTransportError: Swift.Error { case tooManyRequests }
final class TestTransport: Transport {
var history: [URLRequest] = []
var responseData: [Data]
init(responseData: [Data]) { self.responseData = responseData }
func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void) {
history.append(request)
if !responseData.isEmpty {
completion(.success(responseData.removeFirst()))
} else {
completion(.failure(TestTransportError.tooManyRequests))
}
}
}
And I still haven’t used an associated type or a Self requirement. Transport doesn’t need any of that. It’s not even generic.
Common currency
The split between a APIClient.fetch
, which is generic, and Transport.send
, which is not, is a common structure that I look for. Transport.send
operates on a small set of concrete types: URLRequest in, Data out. When you’re working with a small set of concrete types, then composition is easy. Anything that can generate a URLRequest or can consume Data can participate. APIClient.fetch
converts Data into any kind of generic Fetchable. When angle-brackets and associated types start creeping in, the code becomes more expressive, but harder to compose because you have to make sure all the types line up.
The power of the Internet is that it mostly operates on just one type: the packet. It doesn’t care what’s in the packet or what the packet “means.” It just moves packets from one place to another; packets in, packets out. And that’s why the Internet is so flexible, and the equipment that makes it work can be implemented by numerous vendors in wildly different ways, and they can all work together.
At each layer above the network layer, additional context and meaning is applied to the information. It’s interpreted as user information or commands to execute or video to display. That’s composition, gluing together independent layers, each with their own concerns. When designing protocols, I try to employ the same approach. Particularly at the lowest layers I look for common, concrete types to work with. URL and URLRequest. Data and Int. Simple functions like () -> Void
. As I move up the stack, then greater meaning is applied to the data in the form of model types and the like. That means it’s easy to write Transports and many different things can use Transports. And that’s the goal.
This network stack still is nowhere near as flexible and powerful as I want. But now it can fetch a wide variety of model types from a particular type of API in a very composable and testable way. That’s great progress. For some very simple APIs, it might even be done. There’s no need to make it more flexible for its own sake. But I think we’ll quickly find more features we need to add.
Next time, I’ll jump back up to the very top of the stack, to the models, and show where a PAT (protocol with associated type) can really shine.
-
Throughout this series, whenever it’s unambiguous, I’ll refer to
Result<Value, Error>
as just “Value.” ↩


Starting with an Array Stack that is initially empty, illustrate the evolution of the stack as the following operations are performed.
For this exercise, array size is assumed to be:
- Doubled when array becomes full
- Halved when array becomes less than one third full
- Drag elements to move them around in the stack.
- Drop the trash can on any element to delete it.
- Drag the end of the stack to resize it.
- Press the Check button to check your answer.
Do it again!


tracks
albums
genres
artists
podcasts
playlists
podcast name
podcast publisher
description
Recent Searches
recent search
recent search
recent search
Results for Podcasts
podcast name
artist
podcast name
artist
podcast name
artist
podcast name
artist
podcast name
artist
podcast name
artist
podcast name
artist
podcast name
artist


I purchased 1.1 grams of 6x Salvia divinorum extract from an online seller. Cost was about $7. Didn't get to smoke it as soon as I had hoped, but several weeks later, finally got a chance to. I had been interested in trying salvia simply to experience something new - curiosity of where else my mind could go. I had read many reports here, and from a few friends, so I had certain expectations. Here's the details of my first experience.
Started off with some gin and juice (just barely got a buzz) and some candles and good music. Friends came over, and I loaded a bowl of salvia, the extract is real leaf with extracted salvinorin concentrate 'infused' in it, as best as I understand. Smoked the small bowl (.1g?) in a chillum. The smoke was hotter and harsher than anything I had ever hit before, but I learned that part of that had to do with the pipe we used. Some of my friends also mentioned how harsh the smoke was, but it was still manageable. It burns quickly - I took several quick hits and held as long as I could, which wasn't long due to my reaction to the smoke. I passed the pipe, and my friend smoked a bowl.
At this point, a slow high came over me, not a quick rush, as I had read about. A pleasant feeling (relaxed, slightly disoriented, content, aware) crept in. The best way to describe is to say that it was as if I had smoked marijuana several hours ago and had the lingering effects. I was slightly surprised/dismayed that I wasn't having a stronger reaction. No reality-melting yet. My friends seemed to be reacting similarly, but it may have been stronger for some of them.
This state was maintained even after smoking my 2nd and 3rd bowls. After the 3rd or 4th, the bag had been finished, and the feelings got a little bit stronger. I noticed loss of my sense of time - a song would begin playing, and then when I later heard the end of it, I felt as if an hour had passed. Visual hallucinations were very minimal and subtle. Every object in my field of vision at one point seemed to be a brick in a wall, maybe because we were listening to pink floyd. This was subtle and didn't last long. I seemed to be able to control whether or not I was all out 'hallucinating' or just slightly 'high-ish'. With marijuana, I seem to have no control over how high I am, but on this trip with salvia, I seemed to be in control of myself and in control of how much I wanted to hallucinate. As my friends got up to leave, I was again surprised that only a few minutes had passed, not a few hours. I snapped out of hallucinating and showed them out the door.
I then returned to the couch and settled in. I closed my eyes and let myself go. I found that darker was better, because there was more for me to see when I could see nothing at all. I kept my eyes closed and began to see and sense things. The entire time, I was aware that there was nothing in front of me but black, and yet I was sensing colors, shapes, and patterns. I didn't actually see them, but I imagined them and sensed that they were right there in front of me. Very enjoyable, as they danced and changed along with the music. I felt that even with my eyes closed, my field of vision was too small. I was frustrated that my [black] field of vision was so narrow - I wanted to be able to see the nothingness which seemed so colorful all around me, not just directly in front of me. I wanted to know more, understand more, have broader vision. I appreciated that the salvia was helping me to expand my understanding, while making me want to expand it even more.
Occasionally (every few minutes, which felt like every few hours), I would open my eyes and vaguely see the apartment, and I went back to feeling 'just high-ish' rather than hallucinating. When I closed my eyes again, new shapes and colors would 'form'. Everything but the colors seemed to melt away. It was just me, sliding through the waving shapes - no couch, no floor, no apartment. 30 minutes later I got up and got in bed. Getting up and walking brought back almost to normal feeling, although I felt somewhat weightless. Laying down in bed brought on a new wave of light hallucinations, and at some point I must have fallen asleep. No effects were noticed whatsoever the next morning.
BOTTOM LINE
smoked .4g of 6x extract in a chillum
noticed 2 levels - high-ish, and hallucinating
began as high-ish, progressed to hallucinating
I controlled the mindset I was in, partly by closing my eyes
hallucinating was subtle, controllable, not overwhelming
Pleasant feeling, colors, shapes, slight change in 'reality'
Minimal loss of motor skills
fleeting sense of time and location
time from smoking, to falling asleep (still feeling it) = 1.5hrs?
From my expectations of this drug, I was underwhelmed - BUT it was very enjoyable nonetheless. Just not as intense as what I was expecting. I will definitely try salvia again. Although I would be happy repeating this experience, next time I will try a stronger extract, or I will smoke more, to try to have a more intense response to it.
Salvia pros:
legal
cheap
good hallucinations
in control
no 'hangover'
cons:
not intense (reality didn't melt)
harsh smoke
Experience Reports are the writings and opinions of the individual authors who submit them.
Some of the activities described are dangerous and/or illegal and none are recommended by Erowid.


I wrote up instructions on using toppcloud (update: renamed Silver Lining) with Django. They are up on the site (where they will be updated in the future), but I’ll drop them here too…
Creating a Layout
First thing you have to do (after installing toppcloud of course) is create an environment for your new application. Do that like:
$ toppcloud init sampleapp
This creates a directory sampleapp/ with a basic layout. The first thing we’ll do is set up version control for our project. For the sake of documentation, imagine you go to bitbucket and create two new repositories, one called sampleapp and another called sampleapp-lib (and for the examples we’ll use the username USER).
We’ll go into our new environment and use these:
$ cd sampleapp
$ hg clone http://bitbucket.org/USER/sampleapp src/sampleapp
$ rm -r lib/python/
$ hg clone http://bitbucket.org/USER/sampleapp-lib lib/python
$ mkdir lib/python/bin/
$ echo "syntax: glob
bin/python*
bin/activate
bin/activate_this.py
bin/pip
bin/easy_install*
" > lib/python/.hgignore
$ mv bin/* lib/python/bin/
$ rmdir bin/
$ ln -s lib/python/bin bin
Now there is a basic layout setup, with all your libraries going into the sampleapp-lib repository, and your main application in the sampleapp repository.
Next we’ll install Django:
$ source bin/activate
$ pip install Django
Then we’ll set up a standard Django site:
$ cd src/sampleapp
$ django-admin.py sampleapp
Also we’d like to be able to import this file. It’d be nice if there was a setup.py file, and we could run pip -e src/sampleapp, but django-admin.py doesn’t create that itself. Instead we’ll get that on the import path more manually with a .pth file:
$ echo "../../src/sampleapp" > lib/python/sampleapp.pth
Also there’s the tricky $DJANGO_SETTINGS_MODULE that you might have had problems with before. We’ll use the file lib/python/toppcustomize.py (which is imported everytime Python is started) to make sure that is always set:
$ echo "import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'sampleapp.settings'
" > lib/python/toppcustomize.py
Also we have a file src/sampleapp/sampleapp/manage.py, and that file doesn’t work quite how we’d like. Instead we’ll put a file into bin/manage.py that does the same thing:
$ rm sampleapp/manage.py
$ cd ../..
$ echo '#!/usr/bin/env python
from django.core.management import execute_manager
from sampleapp import settings
if __name__ == "__main__":
execute_manager(settings)
' > bin/manage.py
$ chmod +x bin/manage.py
Now, if you were just using plain Django you’d do something like run python manage.py runserver. But we’ll be using toppcloud serve instead, which means we have to set up the two other files toppcloud needs: app.ini and the runner. Here’s a simple app.ini:
$ echo '[production]
app_name = sampleapp
version = 1
runner = src/sampleapp/toppcloud-runner.py
' > src/sampleapp/toppcloud-app.ini
$ rm app.ini
$ ln -s src/sampleapp/toppcloud-app.ini app.ini
The file must be in the "root" of your application, and named app.ini, but it’s good to keep it in version control, so we set it up with a symlink.
It also refers to a "runner", which is the Python file that loads up the WSGI application. This looks about the same for any Django application, and we’ll put it in src/sampleapp/toppcloud-runner.py:
$ echo 'import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
' > src/sampleapp/toppcloud-runner.py
Now if you want to run the application, you can:
$ toppcloud serve .
This will load it up on http://localhost:8080, and serve up a boring page. To do something interesting we’ll want to use a database.
Setting Up A Database
At the moment the only good database to use is PostgreSQL with the PostGIS extensions. Add this line to app.ini:
service.postgis =
This makes the database "available" to the application. For development you still have to set it up yourself. You should create a database sampleapp on your computer.
Next, we’ll need to change settings.py to use the new database configuration. Here’s the lines that you’ll see:
DATABASE_ENGINE = '' # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
DATABASE_NAME = '' # Or path to database file if using sqlite3.
DATABASE_USER = '' # Not used with sqlite3.
DATABASE_PASSWORD = '' # Not used with sqlite3.
DATABASE_HOST = '' # Set to empty string for localhost. Not used with sqlite3.
DATABASE_PORT = '' # Set to empty string for default. Not used with sqlite3.
First add this to the top of the file:
import os
Then you’ll change those lines to:
DATABASE_ENGINE = 'postgresql_psycopg2'
DATABASE_NAME = os.environ['CONFIG_PG_DBNAME']
DATABASE_USER = os.environ['CONFIG_PG_USER']
DATABASE_PASSWORD = os.environ['CONFIG_PG_PASSWORD']
DATABASE_HOST = os.environ['CONFIG_PG_HOST']
DATABASE_PORT = ''
Now we can create all the default tables:
$ manage.py syncdb
Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
...
Now we have an empty project that doesn’t do anything. Let’s make it do a little something (this is all really based on the Django tutorial).
$ manage.py startapp polls
Django magically knows to put the code in src/sampleapp/sampleapp/polls/ — we’ll setup the model in src/sampleapp/sampleapp/polls/models.py:
from django.db import models
class Poll(models.Model):
question = models.CharField(max_length=200)
pub_date = models.DateTimeField('date published')
class Choice(models.Model):
poll = models.ForeignKey(Poll)
choice = models.CharField(max_length=200)
votes = models.IntegerField()
And activate the application by adding 'sampleapp.polls' to INSTALLED_APPS in src/sampleapp/sampleapp/settings.py. Also add 'django.contrib.admin' to get the admin app in place. Run manage.py syncdb to get the tables in place.
You can try toppcloud serve . and go to /admin/ to login and see your tables. You might notice all the CSS is broken.
toppcloud serves static files out of the static/ directory. You don’t actually put static in the URLs, these files are available at the top-level (unless you create a static/static/ directory). The best way to put files in there is generally symbolic links.
For Django admin, do this:
$ cd static
$ ln -s ../lib/python/django/contrib/admin/media admin-media
Now edit src/sampleapp/sampleapp/settings.py and change ADMIN_MEDIA_PREFIX to '/admin-media'.
(Probably some other links should be added.)

You still don’t really have an application, but the rest is mere "programming" so have at it!
Automatically generated list of related posts:
- Why toppcloud (Silver Lining) will not be agnostic I haven’t received a great deal of specific feedback on...
- toppcloud renamed to Silver Lining After some pondering at PyCon, I decided on a new...
- Silver Lining: More People! OK… so I said before Silver Lining is for collaborators...
- Core Competencies, Silver Lining, Packaging I’ve been leaning heavily on Ubuntu and Debian packages for...
- Pronouncing “Django” I’m not saying this to anyone in particular, but I’ve...
Not being a fanboy but damn that’s a lot of work for something so simple.
With the code you made my work so simple.Thanks for the information.


Some quick example text to build on the card title and make up the bulk of the card's content.
Probability: XX.XX %


|
MS&E 311 Optimization Winter 2013-2014 |
| About Optimization |
Optimization is so large a subject that it cannot adequately be treated in the short amount time available in one quarter of an academic year. In this course, we shall restrict our attention mainly to some aspects of nonlinear programming and discuss linear programming as a special case. Among the many topics that will not be covered in this course are integer programming, network programming, and stochastic programming.
Optimization often goes by the name Mathematical Programming (MP). The latter name tends to be used in conjunction with finite-dimensional optimization problems, which in fact are what we shall be studying here. The word "Programming" should not be confused with computer programming which in fact it antedates. As originally used, the term refers to the timing and magnitude of actions to be carried out so as to achieve a goal in the best possible way. Mathematical Programming is one of the central quantitative decision models in Management Science and Operations Research. Highlights of topics are Information Aggregation, Economic Equilibrium, Pricing Model, Core of Game, Financial Decision and Risk Management, Sparse and Low Rank Opimization and its Computations, which you would learn during the process of the course.
| Course Outline |
| Course Requirements |
What background is needed for MS&E 311? This is an advanced master or doctoral-level Core course in the MS&E Department. No prior optimization background is required (although it should do no harm to have some). In this sense, it is an introductory course, but it is not intended to be an elementary course. Students who have taken courses such as MS&E 211 (or MS&E 310) will see some repetition of material. This is unavoidable, but MS&E 311 is intended to be more theoretical and advanced than MS&E 211.
Students in this course will be expected to possess a firm background in the following mathematical subjects: multivariate differential calculus; basic concepts of analysis; linear algebra and some matrix theory. Familiarity with computers and computer programming might also be useful. Above all, it is essential to have a tolerance for mathematical discourse plus an ability to follow - and sometimes devise one's own - mathematical proofs. These play a much larger role in the course than computer work.


--- layout: layout title: Projects ---
Currently ordered by a vague combination of chronology and preference, this page represents a list of my creative and academic projects since 2015.


{% load static %}
The goal of the player is to have a hand worth a value that exceeds the value of the dealer's hand. However, having a worth more than 21 will cause the player to lose (bust).
Once both the player and the dealer have taken their last card, their the values of their hands are compared. If the value of the player's hand is higher than the dealer's then the player wins, and vice versa.
The player can choose the following actions once if neither player nor dealer has a blackjack.
"Split" - this action is only allowed when the player's hand containers two cards of the same value. The player will split their hand into two and acquire and card for each hand. Doing so will require the player to double their bet. The resulting hands are then played independently.
"Hit" - the player takes another card. This can be repeated while the value of the player's hand does not exceed 21.
"Double" - the player will double their bet, acquire another card and will stand.
"Stand" - the player takes no card and keeps their hand as is.
Once there are no more player actions, the face-down card of the dealer will be revealed. If required, the dealer will acquire addition cards until their hand has a value of 17 or higher. If the dealer's hand is worth more than 21, then the remaining players will have won.
If the first two cards of the player has a combined value of 21, then their hand is considered a "blackjack". Having such a starting hand will automatically result in a win for the player unless the dealer also has a "blackjack". In the event that both player and dealer have "blackjack", the outcome will be a tie.


VoidyBootstrap
Version 2.1
A Bootstrap 3 blog theme for the Pelican static site generator.
VoidyBootstrap is a Pelican theme that aims to be mobile friendly, responsive, flexible and easily customisable. It can be used in single column mode, or responsive 2 column mode with sidebar.
While the theme provides lots of customisation options, care has been taken to make it usable right out of the box with minimal configuration and sensible defaults -- albeit with the ubiquitous Bootstrap look and feel. But that's kind of the idea with this theme -- start with bog standard Bootstrap, then customise as much or as little as you like.
On larger screens, with the sidebar enabled, you get a clean and responsive 2 column layout. At the top there's a navbar containing page links, and jumbotron area. The default sidebar has category links and optional tag-cloud. Font Awesome 4 is used for icons.
VoidyBootstrap is functional but deliberately minimal in terms of design. It can be used as is, if your design need are modest and all you want is basic theme that "just works", with all the common features you'd expect from a blog theme. But for those who do want to tinker and customise, it can be used as a starting point for creating custom Bootstrap-based websites. This theme has many features designed to facilitate creating a custom design on top of the default Bootstrap base it provides. Regardless of whether you use few or many of the customisation features provided, what you get is a carefully crafted Pelican theme that tries to accommodate common blogging needs and provide ways to easily accomplish common customisations. It strives to be as minimal, flexible and unopinionated as possible, while still being as usable and simple as possible.
Customised CSS, JavaScript and other modifications can be easily incorporated using configuration settings, and content can be added to the base layout (without modifying existing templates) by strategically adding template fragments.
Installation
Clone this repo, then in your pelicanconf.py set the THEME variable to point to it:
THEME = 'path/to/voidy-bootstrap/'
Out of the box what you get is the stock standard Bootstrap 3 look and feel. To customise things see below.
Example Settings
All of these are optional, but here is a basic example of common settings you might want to configure.
The following should be set in pelicanconf.py:
SITESUBTITLE ='Sub-title that goes underneath site name in jumbotron.' SITETAG = "Text that's displayed in the title on the home page." # Extra stylesheets, for bootstrap overrides or additional styling. STYLESHEET_FILES = ("pygment.css", "voidybootstrap.css",) # Put taglist at end of articles, and use the default sharing button implementation. CUSTOM_ARTICLE_FOOTERS = ("taglist.html", "sharing.html", ) CUSTOM_SCRIPTS_ARTICLE = "sharing_scripts.html" # Default sidebar template. Omit this setting for single column mode without sidebar. SIDEBAR = "sidebar.html" SOCIAL = (('Google+', 'http://plus.google.com/userid', 'fa fa-google-plus-square fa-fw fa-lg'), ('Twitter', 'https://twitter.com/username', 'fa fa-twitter-square fa-fw fa-lg'), ('LinkedIn', 'http://linkedin-url', 'fa fa-linkedin-square fa-fw fa-lg'), ('BitBucket', 'http://bitbucket.org/username', 'fa fa-bitbucket-square fa-fw fa-lg'), ('GitHub', 'http://github.com/username', 'fa fa-github-square fa-fw fa-lg'), )
The following are probably better suited for publishconf.py:
FEED_DOMAIN = SITEURL FEED_ALL_ATOM = 'feeds/all.atom.xml' DISQUS_SITENAME = "disqussitename" GOOGLE_ANALYTICS = "UA-xxxxxxxxxx" GOOGLE_ANALYTICS_SITEID = "gasiteid" TWITTER_USERNAME = "twitterusername"
Basic Configuration and Customisation
By default all required CSS and JavaScript files are downloaded from CDNs. Intentionally little is supplied in the theme's static/css. Just a pygment.css file, and a sample css file (voidybootstrap.css) with very minimal styling is provided as a starting point.
The simplest way to customise things is to override the standard Bootstrap styles as necessary. The theme intentionally avoids loading any additional stylesheets by default, but this can easily be done by using the STYLESHEET_FILES setting. STYLESHEET_FILES is a list for specifying additional stylesheets that will be loaded by the base.html template after the main Bootstrap CSS file. Place any CSS stylesheet files you may require in the static/css directory and add the filenames to the STYLESHEET_FILES setting in pelicanconf.py
For example, if you've placed your own CSS styling in a stylesheet called custom.css (possibly using voidybootstrap.css as a starting point), then you would need something like this in your pelicanconf.py:
STYLESHEET_FILES = ("pygment.css", "custom.css",)
Custom Bootstrap
As an alternative, or in addition, you can easily replace the standard Bootstrap file with a customised one. The BOOTSTRAP_STYLESHEET setting is provided for this. The default behaviour -- if this variable is not set -- is that a standard bootstrap.min.css will be used from a CDN.
To use a different (i.e. customised) Bootstrap stylesheet, set BOOTSTRAP_STYLESHEET to the filename of a stylesheet to use instead.
This could be a customised Bootstrap stylesheet compiled manually from the Bootstrap source files, or perhaps one obtained from an online source.
For example, you could use the Bootstrap customizer to create your own customised Bootstrap CSS file. Place that file in static/css and set BOOTSTRAP_STYLESHEET to its filename.
Similarly, a Bootswatch theme can be easily integrated. Select a theme and download the files. Place all the necessary CSS files in static/css. Set BOOTSTRAP_STYLESHEET to the filename of the main Bootstrap CSS file, and specify any additional CSS files in the STYLESHEET_FILES list.
Further Customisation
Like any other Pelican theme, you can just take the templates provided and modify them to your liking. However, if you're happy with the overall layout and just want to add/subtract things here are there, this theme provides "hooks" to do just that by setting specific variables in your pelicanconf.py and/or providing your own template fragments. For simple customisations, this can make is relatively straightforward to isolate modifications and easily keep up to date with any upstream changes.
These customisation "hooks" are settings that can be configured in pelicanconf.py to point to filenames of custom template fragments. These template fragments will either add or replace content in the default layout provided by the theme. For example, if pelicanconf.py contains the line:
CUSTOM_FOOTER = "custom/footer.html"
Then the bottom of every page will contain whatever is in the template file located at:
templates/includes/custom/footer.html
See the "Custom Includes" section below for further details.
Note that these template fragments must be relative to the theme's templates/includes/ directory.
Standard Settings
VoidyBootstrap honors the following standard Pelican settings:
- SITEURL
- SITENAME
- SITESUBTITLE
- LINKS
- SOCIAL
- TWITTER_USERNAME
- FEED_DOMAIN
- FEED_ALL_ATOM
- FEED_ALL_RSS
- DISQUS_SITENAME
- GOOGLE_ANALYTICS
For the most part they should work as expected, although some may behave slightly differently than in the default Pelican theme (see next section).
Settings
This theme supports the following configuration settings. All are optional.
- SITESUBTITLE
- Sub-title -- displayed in jumbotron.
- SITETAG
- Text that will be placed in the title on the home page.
- DEFAULT_METADESC
- Default value for HTML meta description tag. Individual articles and pages can specify their own meta description by using the theme's custom description metadata tag.
- BOOTSTRAP_STYLESHEET
- Bootstrap CSS file to use instead of default, as described above.
- STYLESHEET_URLS
- A list of URLS for additional stylesheets that should be pulled in by the base.html template, like CSS files from a CDN. Useful for adding the bootstrap theme CSS, for example.
- STYLESHEET_FILES
- An list of filenames (relative to the /theme/css/ directory) for additional stylesheets that should be pulled in by the base.html template (after any STYLESHEET_URLS entries).
- SKIP_DEFAULT_CSS
- No default CSS files at all will be used if this is set to True. In this case the only stylesheets used will be those specified in STYLESHEET_FILES and STYLESHEET_URLS. This setting is provided to give more control over which specific Bootstrap (and Font Awesome) CSS files are used (i.e. it allows a specific version to be configured through settings). But it means for things to work properly at a minimum the Bootstrap and Font Awesome need to be configured using the STYLESHEET_* settings.
- SKIP_DEFAULT_JS
- Don't load any default JavaScript in the base template. If this is set to True, at a minimum jQuery and the Bootstrap JavaScript must be provided via JAVASCRIPT_FILES and/or JAVASCRIPT_URLS.
- SKIP_DEFAULT_NAVIGATION
- Don't display navbar at the top of the page. (If required a custom navigation template can be configured via CUSTOM_SITE_HEADERS.)
- JAVASCRIPT_FILES and JAVASCRIPT_URLS
- Similar to the STYLESHEET_* settings, these allow arbitrary JavaScript files to be loaded. They will be referenced at the bottom of base.html, after any default script files.
- ARCHIVES_URL
- URL of archives page. Default is archives.html. If you're modifying ARCHIVES_SAVE_AS in your pelicanconf.py then you'll probably need to change this setting as well.
- TWITTER_USERNAME
- Set to a valid Twitter username to enable the twitter sharing button.
- TWITTER_CARD
- If set to True, Twitter Card meta-data will be added to article pages. If this is enabled, TWITTER_USERNAME must also be set.
- OPEN_GRAPH
- Set to True to enable Facebook Open Graph meta-properties.
- OPEN_GRAPH_FB_APP_ID
- Facebook App ID.
- OPEN_GRAPH_ARTICLE_AUTHOR
- Value for Open Graph article:author property, which will be set on article pages.
- OPEN_GRAPH_IMAGE
- Default value for Open Graph og:image property on index pages.
- DEFAULT_SOCIAL_IMAGE
- Default value for Open Graph og:image property (and Twitter card image, if enabled) on articles and pages. The custom social_image metadata tag can be used to specify a per-article (or per-page) value, which will always take precedence
- FAVICON
- Allows an alternative favicon filename to be specified.
Sidebar Settings
As of version 2.0 of VoidyBootstrap, the sidebar is optional. To enable the default sidebar, add the following line to your pelicanconf.py:
SIDEBAR = "sidebar.html"
This will give you a simple sidebar with social media links and a list of Categories, which can be customised using the options described in this section.
Optionally, if you'd like to use the LINKS setting or include a list of tags in the default sidebar, add the following to pelicanconf.py after SIDEBAR:
CUSTOM_SIDEBAR_MIDDLES = ("sb_links.html", "sb_taglist.html", )
As of Pelican 3.6 tag cloud support has been removed from Pelican, so if you'd like a proper tag cloud in the sidebar you'll need to configure the tag_cloud plugin and add the following to pelicanconf.py:
CUSTOM_SIDEBAR_MIDDLES = ("sb_tagcloud.html", )
The "sidebar" is an area where many sites will require something specific and it's unlikely any particular implementation will satisfy everyone all the time. However, there are things that commonly appear in sidebars (e.g. author bio, categories, tag cloud, etc), so the theme includes a default sidebar template that provides a fairly typical sidebar implementation. The default sidebar has a few customisation settings so that common things can be added and configured via settings in pelicanconf.py (as described in this section), but it's intended more as a starting point for custom implementations rather than an attempt to satisfy all possible use cases.
For those who might want something completely different in a sidebar, just create your own sidebar template (possibly using includes/sidebar.html as a starting point), and then set SIDEBAR to point to it. This will completely replace the theme's default sidebar with whatever is in your custom sidebar template.
But for those happy enough to stick with the default sidebar, the following settings are available to customise it:
- SOCIAL
- Social media links to display in sidebar. This option is handled a bit differently than in the default Pelican theme. This should be a list/tuple where each element is a tuple with 3 elements: (name, URL, Font Awesome icon class). (See the "Example Settings" section above for an example.) The last element (icon class) can be omitted, in which case a generic icon will be used instead.
- LINKS
- Optional list of arbitrary links to display in the sidebar. Each element must be a tuple with 2 elements: (link title, URL). For this option to work with the default sidebar, "sb_links.html" needs to be added to CUSTOM_SIDEBAR_MIDDLES (as shown above).
- SIDEBAR_HIDE_CATEGORIES
- A list of categories is displayed in the sidebar by default. Set this option to True to disable this category list.
- SIDEBAR_SIZE
- The number of columns in the Bootstrap grid the sidebar should take up. The default is 3.
See also CUSTOM_SIDEBAR_TOP, CUSTOM_SIDEBAR_BOTTOM and CUSTOM_SIDEBAR_MIDDLES below.
Custom Includes
The following settings (all optional), if specified, should be set to filenames of appropriate template fragments that will be included at strategic points from the primary templates. They will override or add to existing sections of content. All filename paths must be relative to the theme's templates/includes directory.
Note that in the following setting names, an "*" (asterisk) represents a page type, possible values of which are: INDEX, ARTICLE, PAGE, CATEGORY, TAG, AUTHOR, ARCHIVES.
(Also note that in Version 2.0 of this theme, many settings were renamed, and many new ones added. See CHANGES.rst.)
- CUSTOM_SITE_HEADERS
- List of templates that will replace the default site header area (i.e. the jumbotron area).
- CUSTOM_HEADER_*
- Add custom content after the site headers, before the main container, and outside any container dev (and therefore before any content columns).
- CUSTOM_CONTAINER_TOP_*
- Similar to CUSTOM_HEADER_*, but inside the main container div. Not inside any row or column, so any content here will span across the top of both columns (in 2 column mode).
- CUSTOM_CONTENT_TOP_*
- Template fragment that will be inserted at the top of the content column, before anything else.
- CUSTOM_CONTENT_BOTTOM_*
- These will be included right at the bottom of the content column.
- CUSTOM_CONTAINER_BOTTOM_*
- These will be included at the bottom of the main container, after the columns. Content will end up inside a container but outside of columns, spanning across the bottom, below both columns (in 2 column mode).
- CUSTOM_ARTICLE_HEADERS
- List of templates that will replace the default article header provided in includes/article_header.html on article pages only.
- CUSTOM_INDEX_ARTICLE_HEADERS
- List of templates that will replace the default article header provided in includes/article_header.html on index pages only.
- CUSTOM_ARTICLE_PRECONTENT
- Template fragment that will be inserted just before the start of the article body text (after any headers, image and standfirst).
- CUSTOM_ARTICLE_FOOTERS and CUSTOM_PAGE_FOOTERS
- List of templates that will included at the bottom of articles/pages, after the body text but before the comments. Can be used to configure any appropriate content, like sharing buttons, taglist, etc.
- CUSTOM_FOOTER
- Footer template to be included by base.html. Anything here (e.g. copyright text) will appear between footer tags at the bottom of every page.
- CUSTOM_SCRIPTS_ARTICLE and CUSTOM_SCRIPTS_PAGE and CUSTOM_SCRIPTS_ARCHIVE
- Template fragment for any additional javascript code specific to articles, pages and archives respectively (useful for things like social media sharing code). Will be included at the bottom of pages, just before the closing body tag. Set to includes/sharing_scripts.html to use the default provided implementation.
- CUSTOM_SCRIPTS_BASE
- As above, but will be included on every page.
- CUSTOM_SIDEBAR_TOP
- Included by sidebar.html at the top of the sidebar. Provides a convenient place for an "about" blurb, for example.
- CUSTOM_SIDEBAR_BOTTOM
- Included by sidebar.html at the bottom of the sidebar.
- CUSTOM_SIDEBAR_MIDDLES
- List of templates that will be included by sidebar.html after Categories but before CUSTOM_SIDEBAR_BOTTOM. Provided mainly as an easy way to use the default tag cloud implementation. Realistically, if you're doing more customisations than this in your sidebar, a better option is just to create your own implementation of sidebar.html and use this instead (as described in "Sidebar Settings").
- CUSTOM_INDEX_META
- Included by index.html in the head section. Can be used to add extra HTML meta tags to index pages, for example.
- CUSTOM_HTML_HEAD
- Included by base.html in the head section.
Custom Metadata Tags
This theme supports the following (optional) custom metadata tags for use in articles and pages.
- description
- Can be used in pages and articles to provide a value for the HTML meta description tag, and social meta data (i.e. Open Graph).
- standfirst
- Text for a summary/intro paragraph that will be placed at the start of an article. This paragraph will be given a CSS class of "standfirst" so that additional styles can be applied.
- social_image
- Set to an image filename (relative to {{ SITEURL }}/images/) to provide a value for an article's og:image meta property.
- image
- Set to an image filename (relative to {{ SITEURL }}/images/) to display a (responsive) "featured image" at the top of an article, underneath any standfirst.
- schema_type
- Pages only. A schema.org itemtype for the page. Default is "WebPage".
- javascript
- Pages only. Filename of a JavaScript file (relative to theme/js/ directory) to load for this page.
Security Considerations
By default, this theme obtains various dependency files (styles, scripts, etc) from CDNs. In an attempt to follow current security best practices, files linked from CDNs are retrieved via HTTPS, and use integrity attributes.
In addition, if your server has a Content-Security-Policy (CSP) header configured, it should take into account the CDNs used by the theme.
For example, a policy in Nginx configuration syntax that covers the external resources used by this theme might look something like:
add_header Content-Security-Policy "default-src 'none' ; img-src 'self' ; font-src 'self' https://maxcdn.bootstrapcdn.com ; form-action 'self' ; frame-ancestors 'none' ; media-src 'self' ; script-src 'self' https://ajax.googleapis.com https://maxcdn.bootstrapcdn.com https://oss.maxcdn.com ; style-src 'self' https://maxcdn.bootstrapcdn.com " ;
Author
Screenshot
License
Licensed under the MIT License
Comments


{% extends "base.html" %} {% block content %}
This example demonstrates how to assign a form to a form group. As a prerequisite, ensure that you have created a form group and set the office ID on this form group before proceeding.
{% if show_doc %}Documentation about this example.
{% endif %}API methods used: FormGroups:GetFormGroups and FormLibraries::GetFormLibraryForms.
{% endblock %}