L
- label typepublic class DocumentReader<L>
extends java.lang.Object
| Modifier and Type | Field and Description |
|---|---|
protected java.io.BufferedReader |
in
Reader used to read in document text.
|
protected boolean |
keepOriginalText
Whether to keep source text in document along with tokenized words.
|
protected TokenizerFactory<? extends HasWord> |
tokenizerFactory
Tokenizer used to chop up document text into words.
|
| Constructor and Description |
|---|
DocumentReader()
Constructs a new DocumentReader without an initial input source.
|
DocumentReader(java.io.Reader in)
Constructs a new DocumentReader using a PTBTokenizerFactory and keeps the original text.
|
DocumentReader(java.io.Reader in,
TokenizerFactory<? extends HasWord> tokenizerFactory,
boolean keepOriginalText)
Constructs a new DocumentReader that will read text from the given
Reader and tokenize it into words using the given Tokenizer.
|
| Modifier and Type | Method and Description |
|---|---|
static java.io.BufferedReader |
getBufferedReader(java.io.Reader in)
Wraps the given Reader in a BufferedReader or returns it directly if it
is already a BufferedReader.
|
boolean |
getKeepOriginalText()
Returns whether created documents will store their source text along with tokenized words.
|
java.io.Reader |
getReader()
Returns the reader for the text input source of this DocumentReader.
|
static java.io.Reader |
getReader(java.io.File file)
Returns a Reader that reads in the given file.
|
static java.io.Reader |
getReader(java.io.InputStream in)
Returns a Reader that reads in the given InputStream.
|
static java.io.Reader |
getReader(java.lang.String text)
Returns a Reader that reads in the given text.
|
static java.io.Reader |
getReader(java.net.URL url)
Returns a Reader that reads in the given URL.
|
TokenizerFactory<? extends HasWord> |
getTokenizerFactory()
Returns the tokenizer used to chop up text into words for the documents.
|
protected BasicDocument<L> |
parseDocumentText(java.lang.String text)
Creates a new Document for the given text.
|
BasicDocument<L> |
readDocument()
Reads the next document's worth of text from the reader and turns it into
a Document.
|
protected java.lang.String |
readNextDocumentText()
Reads the next document's worth of text from the reader.
|
static java.lang.String |
readText(java.io.Reader in)
Returns everything that can be read from the given Reader as a String.
|
void |
setKeepOriginalText(boolean keepOriginalText)
Sets whether created documents should store their source text along with tokenized words.
|
void |
setReader(java.io.Reader in)
Sets the reader from which to read and create documents.
|
void |
setTokenizerFactory(TokenizerFactory<? extends HasWord> tokenizerFactory)
Sets the tokenizer used to chop up text into words for the documents.
|
protected java.io.BufferedReader in
protected TokenizerFactory<? extends HasWord> tokenizerFactory
protected boolean keepOriginalText
public DocumentReader()
setReader(java.io.Reader)
before trying to read any documents.
Uses a PTBTokenizer and keeps original text.public DocumentReader(java.io.Reader in)
in
- The Readerpublic DocumentReader(java.io.Reader in, TokenizerFactory<? extends HasWord> tokenizerFactory, boolean keepOriginalText)
public java.io.Reader getReader()
public void setReader(java.io.Reader in)
public TokenizerFactory<? extends HasWord> getTokenizerFactory()
public void setTokenizerFactory(TokenizerFactory<? extends HasWord> tokenizerFactory)
public boolean getKeepOriginalText()
public void setKeepOriginalText(boolean keepOriginalText)
public BasicDocument<L> readDocument() throws java.io.IOException
readNextDocumentText()
and passes it to parseDocumentText(java.lang.String)
to create the document.
Subclasses may wish to override either or both of those methods to handle
custom formats of document collections and individual documents
respectively. This method can also be overridden in its entirety to
provide custom reading and construction of documents from input text.java.io.IOException
protected java.lang.String readNextDocumentText() throws java.io.IOException
java.io.IOException
protected BasicDocument<L> parseDocumentText(java.lang.String text)
public static java.io.BufferedReader getBufferedReader(java.io.Reader in)
public static java.lang.String readText(java.io.Reader in) throws java.io.IOException
java.io.IOException
public static java.io.Reader getReader(java.lang.String text)
public static java.io.Reader getReader(java.io.File file) throws java.io.FileNotFoundException
java.io.FileNotFoundException
public static java.io.Reader getReader(java.net.URL url) throws java.io.IOException
java.io.IOException
public static java.io.Reader getReader(java.io.InputStream in)


My name is Burak. I came from Turkey one month ago. I like photography and playing drums. Back in Turkey, I used to play in a Led Zeppelin cover band and we kicked ass! I also played with other bands but it was just for money. I'd like to add that I am actually a lawyer. After practicing law for 13 years, I had finally enough and wanted to continue doing something I'm interested in and earning my living along the way.
Aaand here I am in Canada! Oh by the way, I'm a proud satanist and I appreciate Canada for letting me practice my religion to the fullest..So HAIL SATAN!! and Justin Trudeau I guess..
And I am the one and only TC.
I divorced all of my wives when they were 33 years-old.
I'm certain that I posses healing powers.
I do my own stunts in my movies.
Most importantly, I AM A SCIENTOLOGIST!!
Join the Church of Scientology and we shall thrive together! (we also need you to update our web page)..


Ghana-Burkina Faso railway progresses steadily - Deputy Minister
By Kodjo Adams/Haruna Abdulai, GNA
Accra, Nov. 29, GNA - Mr Andy Appiah-Kubi, the Deputy Minister for Railways Development, has said construction of the Ghana-Burkina Faso railway line and other railway networks within the country was progressing steadily.
He said feasibility studies had already been commissioned for the 595km Kumasi-Paga line being undertaken by Mesers Vision Consult (Ghana), Gauff Ingeneres (Germany), and ILF Consult (Germany),adding that studies on the Tamale-Paga line was completed.
The Deputy Minister said this at the second rail dialogue dubbed " Developing a Greenfield Railway project, the example of Ghana-Burkina Interconnectivity Project: Lessons, Challenges and the Way Forward.
The Minister said Transaction Advisors made up of TEAM engineering of Italy and Vision Consult were currently studying the section of the rail line from Mpakadan to Tamale, through Yendi and Paga to Ouagadougou.
He said a joint committee of experts from the two countries was established in Ouagadougou on January 16, 2018 through a Memorandum of Understanding to facilitate the implementation of the project.
Mr Kubi said the phase one of the Transaction Advisor’s Report was reviewed and approved by the joint Committee on April 15, 2019 in Bolgatanga, Ghana.
The joint committee also validated pre-feasibility report, which included the preliminary alignment of the project, on October 9, 2019 in Manga, Burkina Faso.
The preliminary alignment phase is completed whilst the phase two the procurement process is also in progress.
The Deputy Minister revealed that the two countries had signed a treaty on the December 14, 2018 relating to the joint implementation of the project.
The Treaty, which established the guiding principles and financing model for the project, is before Parliament for Ratification.
He said the project provided opportunities across regions and Nations adding that the collaboration had gone on a long way to strengthen the bilateral relations between the two countries.
Mr Appiah-Kubi noted that the two countries were working together to address any pitfalls that might arise during the project.
He called on other West African countries to join Ghana and Burkina Faso in building a strong rail network around the continent.
Some of the participants expressed satisfaction with the spate of the project, hoping to see its completion to ease transportation in the country and appealed to government to settle outstanding compensations to avoid the delay of the project.
GNA


Going through katas I came across a few problems that got me stuck. Sometimes I needed someone else to push me the right direction. Sometimes I was able to work the solution out myself. This is going to be an example of a second case. It was when I worked on kata on build-in functions. After relatively easily completing previous ones, I was quite confident about that one too. For some reason I thought I will only need to call one of the build-in functions, like join(), and the kata will be completed.
When I started solving it I realized that it wouldn’t be as easy as I anticipated. I was trying to find a solution for some time and than I’ve noticed I’m actually just trying to guess instead using some of the problem solving techniques. And let me tell you, it took me a few hours to come to this conclusion.
So I restarted my work, sit to it with a fresh mind, and started with writing a pseudocode. After that, in shockingly short period of time, I also wrote a working function…
Was that a very elegant way of solving the problem? I’m not sure. I’m actually not sure if I have managed to solve any problem in an elegant way so far:) I’m just happy I solved them in the first place.
Problem solving techniques
Since the beginning of foundations I was quite skeptical about problem solving process. When I first read about it somewhere in sprint 1, I thought it was quite obvious. We are all adults and we solved a few problems in our lives before, so I suppose we all should have our own processes. Well, as good (or not) as they work in every day life, they are not always applicable to code writing. And that’s pretty much what I took out of this situation. There is a reason why we are being introduced to every single module in this course.
Looking at the problem solving techniques, I only used a few of them so far. I really like pseudo coding and I use it every time. I also test my functions in the console and use debugger, so naturally I read the error messages too. And of course, ask uncle Google each time I don’t know a solution to my problem (not just while coding;)) I tent not too use help of peers nor coaches too much, although it happened a few times.
Eventually I’d like to start improving my process, as this feels like a natural technique to me. At the moment I don’t have robust process, but I’m sure that will change once in the bootcamp.


Name
Send
ID
Name
Phone
Email
Delete
{{range .contacts}}
{{.ID}}
{{.Name}}
{{.Phone}}
{{.Email}}
Delete
{{end}}
Add Contact


workbox.expiration. CacheExpiration
The CacheExpiration
class allows you define an expiration and / or limit on the number of responses stored in a
Cache
.
Constructor
CacheExpiration
new CacheExpiration(cacheName, config)
To construct a new CacheExpiration instance you must provide at least one of the config
properties.
Parameter |
|||||||
|---|---|---|---|---|---|---|---|
|
cacheName |
string Name of the cache to apply restrictions to. |
||||||
|
config |
Object Values in
|
Methods
expireEntries
expireEntries()
Expires entries for the given cache and given criteria.
isURLExpired
isURLExpired(url) returns boolean
Can be used to check if a URL has expired or not before it's used.
This requires a look up from IndexedDB, so can be slow.
Note: This method will not remove the cached entry, call
expireEntries()
to remove indexedDB and Cache entries.
Parameter |
|
|---|---|
|
url |
string |
- Returns
-
boolean
updateTimestamp
updateTimestamp(url)
Update the timestamp for the given URL. This ensures the when removing entries based on maximum entries, most recently used is accurate or when expiring, the timestamp is up-to-date.
Parameter |
|
|---|---|
|
url |
string |


Source code for lenstronomy.GalKin.galkin_model
from lenstronomy.GalKin.numeric_kinematics import NumericKinematics
from lenstronomy.GalKin.analytic_kinematics import AnalyticKinematics
__all__ = ['GalkinModel']
[docs]class GalkinModel(object):
"""
this class handles all the kinematic modeling aspects of Galkin
Excluded are observational conditions (seeing, aperture etc)
Major class to compute velocity dispersion measurements given light and mass models
The class supports any mass and light distribution (and superposition thereof) that has a 3d correspondance in their
2d lens model distribution. For models that do not have this correspondence, you may want to apply a
Multi-Gaussian Expansion (MGE) on their models and use the MGE to be de-projected to 3d.
The computation follows Mamon&Lokas 2005.
The class supports various types of anisotropy models (see Anisotropy class).
Solving the Jeans Equation requires a numerical integral over the 3d light and mass profile (see Mamon&Lokas 2005).
This class (as well as the dedicated LightModel and MassModel classes) perform those integral numerically with an
interpolated grid.
The cosmology assumed to compute the physical mass and distances are set via the kwargs_cosmo keyword arguments.
d_d: Angular diameter distance to the deflector (in Mpc)
d_s: Angular diameter distance to the source (in Mpc)
d_ds: Angular diameter distance from the deflector to the source (in Mpc)
The numerical options can be chosen through the kwargs_numerics keywords
interpol_grid_num: number of interpolation points in the light and mass profile (radially). This number should
be chosen high enough to accurately describe the true light profile underneath.
log_integration: bool, if True, performs the interpolation and numerical integration in log-scale.
max_integrate: maximum 3d radius to where the numerical integration of the Jeans Equation solver is made.
This value should be large enough to contain most of the light and to lead to a converged result.
min_integrate: minimal integration value. This value should be very close to zero but some mass and light
profiles are diverging and a numerically stable value should be chosen.
These numerical options should be chosen to allow for a converged result (within your tolerance) but not too
conservative to impact too much the computational cost. Reasonable values might depend on the specific problem.
"""
def __init__(self, kwargs_model, kwargs_cosmo, kwargs_numerics=None, analytic_kinematics=False):
"""
:param kwargs_model: keyword arguments describing the model components
:param kwargs_cosmo: keyword arguments that define the cosmology in terms of the angular diameter distances involved
:param kwargs_numerics: numerics keyword arguments
:param analytic_kinematics: bool, if True uses the analytic kinematic model
"""
if kwargs_numerics is None:
kwargs_numerics = {'interpol_grid_num': 200, # numerical interpolation, should converge -> infinity
'log_integration': True,
# log or linear interpolation of surface brightness and mass models
'max_integrate': 100,
'min_integrate': 0.001} # lower/upper bound of numerical integrals
if analytic_kinematics is True:
anisotropy_model = kwargs_model.get('anisotropy_model')
if not anisotropy_model == 'OM':
raise ValueError('analytic kinematics only available for OsipkovMerritt ("OM") anisotropy model.')
self.numerics = AnalyticKinematics(kwargs_cosmo=kwargs_cosmo, **kwargs_numerics)
else:
self.numerics = NumericKinematics(kwargs_model=kwargs_model, kwargs_cosmo=kwargs_cosmo, **kwargs_numerics)
self._analytic_kinematics = analytic_kinematics
[docs] def check_df(self, r, kwargs_mass, kwargs_light, kwargs_anisotropy):
"""
checks whether the phase space distribution function of a given anisotropy model is positive.
Currently this is implemented by the relation provided by Ciotti and Morganti 2010 equation (10)
https://arxiv.org/pdf/1006.2344.pdf
:param r: 3d radius to check slope-anisotropy constraint
:param theta_E: Einstein radius in arc seconds
:param gamma: power-law slope
:param a_ani: scaled transition radius of the OM anisotropy distribution
:param r_eff: half-light radius in arc seconds
:return: equation (10) >= 0 for physical interpretation
"""
dr = 0.01 # finite differential in radial direction
r_dr = r + dr
sigmar2 = self.numerics.sigma_r2(r, kwargs_mass, kwargs_light, kwargs_anisotropy)
sigmar2_dr = self.numerics.sigma_r2(r_dr, kwargs_mass, kwargs_light, kwargs_anisotropy)
grav_pot = self.numerics.grav_potential(r, kwargs_mass)
grav_pot_dr = self.numerics.grav_potential(r_dr, kwargs_mass)
self.numerics.delete_cache()
return r * (sigmar2_dr - sigmar2 - grav_pot + grav_pot_dr) / dr


Line:
Go!
Show/Hide Demos
Documentation
|
Unit Tests
# Not sure what to do now? Enter your mips code here # and hit step (to run one line at a time) or # hit run (to run them all at once) # Keep an eye on the register and stack tracker # to see what changes are being made to them. # If you want to preload the stack or some registers # with data, you can click on them to change edit them. # If you would like more infomation, check out the user # guide linked in the menu.


| 024:035 Khan | : |
Allah is the Light of the heavens and the earth. The parable of His Light is as (if there were) a niche and within it a lamp, the lamp is in glass, the glass as it were a brilliant star, lit from a blessed tree, an olive, neither of the east (i.e. neither it gets sun-rays only in the morning) nor of the west (i.e. nor it gets sun- rays only in the afternoon, but it is exposed to the sun all day long), whose oil would almost glow forth (of itself), though no fire touched it. Light upon Light! Allah guides to His Light whom He wills. And Allah sets forth parables for mankind, and Allah is All-Knower of everything. |
| 024:035 Maulana | : |
Allah is the light of the heavens and the earth. A likeness of His light is as a pillar on which is a lamp -- the lamp is in a glass, the glass is as it were a brightly shining star -- lit from a blessed olive-tree, neither eastern nor western, the oil whereof gives light, though fire touch it not -- light upon light. Allah hides to His light whom He pleases. And Allah sets forth parables for men, and Allah is Knower of all things -- |
| 024:035 Pickthal | : |
Allah is the Light of the heavens and the earth. The similitude of His light is as a niche wherein is a lamp. The lamp is in a glass. The glass is as it were a shining star. (This lamp is) kindled from a blessed tree, an olive neither of the East nor of the West, whose oil would almost glow forth (of itself) though no fire touched it. Light upon light. Allah guideth unto His light whom He will. And Allah speaketh to mankind in allegories, for Allah is Knower of all things. |
| 024:035 Rashad | : |
GOD is the light of the heavens and the earth. The allegory of His light is that of a concave mirror behind a lamp that is placed inside a glass container.  The fuel thereof is supplied from a blessed oil-producing tree, that is neither eastern, nor western. Its oil is almost self-radiating; needs no fire to ignite it. Light upon light. GOD guides to His light whoever wills (to be guided). GOD thus cites the parables for the people. GOD is fully aware of all things. |
| 024:035 Sarwar | : |
God is the light of the heavens and the earth. A metaphor for His light is a niche in which there is a lamp placed in a glass. The glass is like a shining star which is lit from a blessed olive tree that is neither eastern nor western. Its oil almost lights up even though it has not been touched by the fire. It is light upon light. God guides to His light whomever He wants. God uses various metaphors. He has the knowledge of all things. |
| 024:035 Shakir | : |
Allah is the light of the heavens and the earth; a likeness of His light is as a niche in which is a lamp, the lamp is in a glass, (and) the glass is as it were a brightly shining star, lit from a blessed olive-tree, neither eastern nor western, the oil whereof almost gives light though fire touch it not-- light upon light-- Allah guides to His light whom He pleases, and Allah sets forth parables for men, and Allah is Cognizant of all things. |
| 024:035 Sherali | : |
ALLAH is the light of the heavens and the earth. HIS light is as if there were a lustrous niche, wherein is a lamp. The lamp is inside a glass-globe. The globe is, as it were, a glittering star. The lamp is lit from the oil of a blessed tree -an olive - neither of the East nor of the West, whose oil well-nigh would shine forth even though fire touched it not. Light upon light ! ALLAH guides to HIS light whomsoever HE pleases. And ALLAH sets forth parables for men, and ALLAH knows all things full well. |
| 024:035 Yusufali | : |
Allah is the Light of the heavens and the earth. The Parable of His Light is as if there were a Niche and within it a Lamp: the Lamp enclosed in Glass: the glass as it were a brilliant star: Lit from a blessed Tree, an Olive, neither of the east nor of the west, whose oil is well-nigh luminous, though fire scarce touched it: Light upon Light! Allah doth guide whom He will to His Light: Allah doth set forth Parables for men: and Allah doth know all things. |
Concepts/Themes Covered: أركان الإسلام > التوحيد > توحيد الله تعالي > صفات الله تعالى > النور Pillars of Islam> Islamic> Oneness of Allah> His Glory's Characteristics> The Light أركان الإسلام > التوحيد > توحيد الله تعالي > صفات الله تعالى > نور السماوات والأرض Pillars of Islam> Islamic> Oneness of Allah> His Glory's Characteristics> The Light of the Heaven and Earth أركان الإسلام > التوحيد > توحيد الله تعالي > مشيئته Pillars of Islam> Islamic> Oneness of Allah> His Will أركان الإسلام > (محمد (صلى الله عليه وسلم > أخلاقه وصفاته Pillars of Islam> The Blessed Muhammad(PBUH)> His Manners and Attributes
This tool uses eight English parallel translations for the Holy Quran imported from Mr. Clay Smith. You can also visit Qurany Concepts for browsing the Quran themes or topics. If you want to search the Holy Quran for keywords, please visit Qurany Keywords.


It was 3:00 PM today when I loaded up my bowl with the salvia 6x extract I had received on this fine 420 holiday. I took in a hit and held it until my exhale revealed no smoke. As I laid back I began to feel a little weird, looking around my dimly lit room. This was not going to be enough! This is what I had heard in my head. I quickly loaded up another hit, thinking I didn't have much time. I breathe in the harsh smoke, setting down my pipe as I lay back on my bed.
I can't really say how the trip started, my memory is weak of the whole event. Before I knew it though, I was part of an extremely complex system of gears, like I was in some kind of bizarre factory. I found my hands straight up in the air as if I was turning a large gear, when I realized what I was doing I was shocked! Still, I kept turning the gears, as I WAS a gear and there was nothing I could do to stop my hands from taking part in this! My room was looking extremely crazy, and as I glanced to my ceiling fan something odd happened.
It was like my legs were really long and liquefied and were running up into my dimmed light of the fan. I couldn't really tell if my legs were flowing up into the fan or if they were pouring down from the fan, maybe a combination of both. I was hearing alot of voices, like alot of people screaming, but one voice stood out from the others. It was kind of a higher voice, coming from behind me to the right, and it was saying 'He doesn't even know what he's doing!!' over and over again. 'Oh yes I do,' I thought to myself. 'I just smoked some salvia!' is the next thing that finally popped into my head.  I then laid back down to attempt a 'break through'.
As I laid, my vision went back a little bit and viewed my body as a whole. I was in complete blackness and my body had a thick white outline to it. I was trying so hard to let go, and the further I let go, my body would begin turning from the white to green, filling in radiantly. The green was a weird energy/spirit and I was doing by best to let it overcome me. I remember the voice continuing to speak to me but I found it hard to remember exactly what it was saying. Unfortunately I do not think I achieved my 'break through' as my body never got completely covered with the 'green' and I suddenly came off my trip.
Overall it was a very interesting experience that I enjoyed. I plan on attempting to get back to this 'multiverse' very soon!
Experience Reports are the writings and opinions of the individual authors who submit them.
Some of the activities described are dangerous and/or illegal and none are recommended by Erowid.


(Deprecated API)
(Deprecated API)
(Deprecated API)
Includes enterprise licensing and support
Two common types of spatial queries used in Maps mash-ups are proximity searches and bounding box queries. Proximity searches return all results within a certain distance of a location. Bounding box queries return all results in a bounding box of four points defined by its four corners. These types of searches are useful in a wide variety of mash-ups. For instance, a store finder on a website typically allows the user to show all stores within a certain radius. Or it may want to show all stores that are in the current map boundaries. While intuitively simple, these searches can be computationally difficult.
In this tutorial we will show how to implement these kinds of geospatial searches
using the open source project GeoModel
in Python hosted on Google App Engine (GAE). We will build a demo app
PubSchools
, which displays public schools
using bounding box and proximity searches. Developers familiar with Python will
get the most out of this article.
The tutorial is organized into the following sections:
GeoModel
and GeocellsPubSchools
Demo A typical use case of geospatial queries is find the closest 20 locations
that are within a radius of 25 miles to the center of the bounnding box given by (lat_t, lng_t)
.
This kind of proximity query requires calculation of relative distances between latitude/longitudes
of pre-defined and pre-loaded geo locations and the target latitude/longitude so that the distance algorithm
can filter by a cut-off distance and order the results by distance. The actual
SQL query looks like this:
SELECT id, ( 3959 * acos( cos( radians(lat_t) ) * cos( radians( lat ) ) * cos( radians( lng ) - radians(lng_t) ) + sin( radians(lat_t) ) * sin( radians( lat ) ) ) ) AS distance FROM Stores HAVING distance < 25 ORDER BY distance
where Stores
is a table that holds the pre-defined and pre-loaded entities
of interest with geo locations such as stores or schools, whose fields lat
and lng
contain
their respective latitude and longitude (See detail here).
Since relative distances must be calculated based on user input of target location given by (lat_t, lng_t)
,
it results in a rather expensive query every time a new set of lat/lng is submitted as input of a new request.
Similarly a bounding box query might look like this in SQL:
SELECT * FROM Stores WHERE lat > (lat_t - d) AND lat < (lat_t + d) AND lng > (lng_t - d) AND lng < (lng_t + d)
where (lat_t, lng_t)
represents target coordinate and d
a fixed size for bounding box. While this type of bounding box query can be implemented with proper
indexes in SQL database, it presents a difficulty for the App Engine Datastore, as described in the
next section.
Google App Engine (GAE) enables you to build web applications on the same scalable systems that power Google applications. App Engine applications are easy to build, easy to maintain, and easy to scale as your traffic and data storage needs grow. With App Engine, there are no servers to maintain: You just upload your application, and it's ready to serve to your users.
Although Google App Engine has an SQL-like query language, GQL, its Datastore differs significantly from typical relational database (see Under the Cover of the App Engine Datastore). For example, while a simple query like the following is possible:
db.GqlQuery("SELECT * FROM Stores ORDER BY <some property> DESC")
GQL does not support table 'joins' nor 'having' clause commands. These syntax restrictions make it
difficult to construct a SQL like 'having distance < 25'
. This makes it impossible to have
a query by proximity in App Engine.
Bounding box queries are also not supported in App Engine because there need to be two inequalities on two independent properties. App Engine is optimized for performing lookups of single properties and lookups are implemented via indexes over many servers. When a query needs to compare values of two independent properties, it would have to spawn two lookups that would have to be executed as two separate processes on physically separate servers but then merging and sorting returned dataset over two servers would be problematic at minimum and in general not optimized for high performance. This makes it impossible to have a scalable solution for this type of queries.
These limitations on queries prevent us from running either proximity or bounding box queries in App Engine.
To implement these types of searches, we need to use another mechanism by introducing a concept called
geocell and a GeoModel
class.
GeoModel
and Geocells
GeoModel
is an open source project that aims to
provide a generalized solution for performing basic indexing and querying of geospatial data in
Google App Engine. This implementation introduces a concept called geocell
,
which is a coordinate system that divides the world up into rectangular cells. The idea is inspired by similar
concepts in geohashing. See detailed explanation of geocell
in later section but for now think of it as a rectangular region at various zoom levels.
The GeoModel
class implements a base class that comes with two key interfaces. Developers
can define application-specific entities by extending GeoModel
class and call
the following two methods without having to do much coding.
@staticmethod def proximity_fetch(query, center, max_results=10, max_distance=0):
The proximity_fetch()
method is statically defined at class level
with the following list of parameters:
query
: an object of db.Query
type, which can be used to
specify additional filtering and sorting on entities like stores or schools (more on this in
PubSchools
example below)center
: the center point of search based on user inputmax_results
: max number of results to returnmax_distance
radius of proximity search
Proximity query starts with a geocell that contains the center
point and queries
that geocell. It then bubbles its way up to adjacent geocells of next resolutions within
a cut-off distance supplied by the search radius. Here a geocell is basically a rectangular
box labeled by a hexadecimal string, the length of which represents its resolution.
See the sample geocells in the image on the left.
@staticmethod def bounding_box_fetch(query, bbox, max_results=1000, cost_function=None):
The bounding_box_fetch()
method is statically defined at class level
with the following list of parameters:
query
: an object of db.Query
type, which can be
used to specify additional filteringbbox
: an object of geotypes.Box
type representing a bounding boxmax_results
: max number of results to return
The parameter cost_function
is an optional resource calculation that accepts two arguments,
num_cells
and resolution
of each geocell to search, and returns the
'cost' calculated in number for querying against this number of cells at the given resolution.
The bounding box query starts off with a list of geocells inside the input bounding box and then runs queries against each geocell for relevant entities, merges them, and returns them in a list.
As you may see, both proximity and bounding box searches rely on a key concept of
geocell in GeoModel
. The next section will explain how this works.
In GeoModel
a geocell is labeled by a hexadecimal string that defines
a two dimensional rectangular region inside the [-90,90] x [-180,180] latitude/longitude
space. Resolution of a geocell is measured by the length of its string label. We start
out with 16 geocells labeled by hexdecimal number with a single digit and subsequently
divide each geocell into 16 more geocells labeled by hexdecimal strings with more digits.
This kind of hierarchical structure of geocells is similar to
geohash in that any prefix of a geocell
is considered its ancestor, with geocell[:-1]
being geocell's immediate
parent cell. As resolution increases, label of a geocell gets longer and the area covered
by the geocell more granular. For most practical purposes, at very high resolutions (e.g.
precision of 13 digits), geocells can be treated as single points.
To calculate the rectangle of a given geocell string, for example '78a', first divide the [-90,90] x [-180,180] latitude/longitude space evenly into a 4x4 grid as shown in Diagram 1 below:
Note that the point (0, 0) is at the intersection of grid cells 3, 6, 9 and c. Each cell is divided into additional 4x4 grid. For the example of geostring '78a', cell '7' further divides as shown in Diagram 2 above, and should represent the sub-rectangle from (-45, 90) to (0, 180).
If we continue to re-divide cell '78' into 4x4 grid of sub-rectangles, we will arrive at cell '78a'. In general this is the approach to map a geostring to a particular geocell:
PubSchools
Demo
The PubSchools
demo app demonstrates how
to extend GeoModel
and implement geospatial search for public schools. Follow the link
to check out the demo and enter searches (e.g. New York, NY or 94040) to see a list of
public schools. The sample source code is available
here.
To define your own application-specific class for your own entities, you extend from GeoModel
.
Here is how PubSchools
class entends from Geo Model:
class PublicSchool(GeoModel): """A location-aware model for public school entities. """ school_id = db.StringProperty() name = db.StringProperty() address = db.StringProperty() city = db.StringProperty() state = db.StringProperty() zip_code = db.IntegerProperty() enrollment = db.IntegerProperty() phone_number = db.StringProperty() locale_code = db.IntegerProperty() school_type = db.IntegerProperty() school_level = db.IntegerProperty() grades_taught = db.ListProperty(int)
The application-specific properties of PubSchools
class such as
address
, name
, or grades_taught
allow additional
query possibilities but the essential functionality of geospatial queries (namely proximity
and bounding box searches) is inherited from GeoModel
.
Let's say the user enters a query with location info such as 'New York, NY' or
'1600 Pennsylvania Ave NW, Washington, DC.' The PubSchools
demo app will
geocode the address first. If the query geocodes to something resembling an address or an intersection,
a proximity search is done; otherwise a bounding box search is done on the geocoded area's extents.
(see front.js
for more detail). The demo app will then make an Ajax call using doSearch
function as follows.
<script type="text/javascript"> if (proximitySearch) { doSearch(updateObject(commonOptions, { type: 'proximity', center: bounds.getCenter() })); } else { doSearch(updateObject(commonOptions, { type: 'bounds', bounds: bounds })); } ... function doSearch(options) { ... // Perform proximity or bounds search. g_currentSearchXHR = $.ajax({ url: '/s/search', type: 'get', data: searchParameters, dataType: 'json', ... } </script>
The doSearch
function sends a request to an endpoint 's/search'
as shown in the code snippet above. More detailed query parameters such as type of search
(bounding box or proximity) and center
or bbox
boundary are all wrapped inside the
searchParameters
. Requests get processed by PubSchools
's service handler
service.py.
The Python code snippet is given as follows.
# Perform proximity or bounds fetch. if query_type == 'proximity': results = PublicSchool.proximity_fetch(base_query, center, max_results=max_results, max_distance=max_distance) elif query_type == 'bounds': results = PublicSchool.bounding_box_fetch(base_query, bounds, max_results=max_results)
Here PubSchools
calls either proximity_fetch()
or bounding_box_fetch()
static method from its parent class GeoModel
and returns results in JSON back to a callback function
of the originating Ajax call, which parses out the results and renders them onto Google Map as markers.
Typical JSON data returned by 's/search'
service is shown below.
{"status": "success", "results": [{"phone_number": "(908) 436-6855", "city": "ELIZABETH", "name": "ELIZABETH HIGH", "school_level": 3, "enrollment": 5279, "highest_grade_taught": 12, "lowest_grade_taught": 9, "school_type": 1, "locale_code": 21, "state": "NJ", "address": "600 PEARL STREET", "lat": 40.6603361, "lng": -74.2116865, "school_id": "340459005478", "zip_code": 7202}]}
Once JSON data are successfully returned to the callback function, the entities inside the data get parsed out and
rendered properly on the map as markers. Detailed info shows in a pop-up info window when a marker is clicked.
See front.js
for more detail on success: function(obj)
callback function.
Note that data of public schools with location info like latitude and longitude have to be loaded into
App Engine Datastore of PubSchools
demo beforehand in order for
geospatial queries to return results. This is done via the appcfg.py upload_data
utility.
We will skip the detail here but source code and info can be found in the
batch folder.
Having covered geospatial queries, App Engine Datastore, GeoModel
,
and internal design of PubSchools
demo, you can put it all together and see the app
in action. Here's a screenshot of PubSchools
but take a test drive of the live demo at
PubSchools
and experiment with various
queries, and toggle between map and list views.
It is straightforward to set up the maps UI in this demo. As shown below,
we include jquery library, set up a Google Maps API key, and include the front.js
.
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script> <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jqueryui/1.7.2/jquery-ui.min.js"></script> <script type="text/javascript" src="http://www.google.com/jsapi?key=<YOUR_API_KEY>"></script> <script type="text/javascript" src="/static/lib/jquery.scrollTo-1.4.2-min.js"></script> <script type="text/javascript" src="/static/lib/microtemplate.js"></script> <script type="text/javascript" src="/static/js/front.js"></script> <script type="text/javascript"> google.load('maps', '2'); google.setOnLoadCallback(init); </script>
The setOnLoadCallback
sets init
as callback function
of onLoad complete. The init
function in turn calls initMap
and
initUI
functions.
/** * On body/APIs ready callback. */ function init() { initMap(); initUI(); } /** * Creates the Google Maps API instance. */ function initMap() { map = new google.maps.Map2($('#map').get(0)); map.setCenter(new google.maps.LatLng(39,-96), 4); geocoder = new google.maps.ClientGeocoder(); // anything besides default will not work in list view map.setUIToDefault(); }
The initMap
function loads Google Maps into the 'map'
div,
centers it around lat/lon (39, -96)
in this case, and sets it to default view
while initUI
sets up various other UI components and event listeners
(see front.js
for more detail) to get ready for user queries. When user enters query and clicks on
the Search button, an Ajax request is constructed and submitted according to the data flow
outlined above. The JSON results returned from the '/s/search'
service endpoint
are rendered on map UI by callback function.
For other approaches to geospatial queries in Google App Engine, see a geohash-based geodatastore project.
Another possible backend solution for spatial queries is the Google Maps Data API. Recently, it added proximity and bounding box searches as well as attribute search. See Maps Data API for more info and stay tuned for search capabiity in upcoming revisions.


freeCodeCamp Survey Form
Thank you for taking the time to help us improve the platform
Name
Email
Age (optional)
Which option best describes your current role?
Select current role
Student
Full time Job
Full time learner
Prefer not to say
Others
Would you recommend freeCodeCamp to a friend?
Definitely
Maybe
Not sure
What is your favorite feature of freeCodeCamp?
Select an option
Challenges
Projects
Community
Open source
What would you like to see improved? (Check all that apply)
Front-end Projects
Back-end Projects
Data Visualization
Challenges
Open Source Community
Gitter help rooms
Videos
City Meetups
Wiki
Forum
Additional Courses
Any comments or suggestions?
Enter your comment here...
Submit


FAT12 Filesystem Explorer
Upload a valid disk image in the FAT12 format to explore its file and directory hierarchy.


|
Unishox
A hybrid encoder for Short Unicode Strings
|
Main code of Unishox2 Compression and Decompression library. More...
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <stdint.h>
#include <limits.h>
#include "unishox2.h"
Macros | |
| #define | NICE_LEN 5 |
| Minimum length to consider as repeating sequence. | |
| #define | RPT_CODE ((2 << 5) + 26) |
| Set (USX_NUM - 2) and vertical code (26) for encoding repeating letters. | |
| #define | TERM_CODE ((2 << 5) + 27) |
| Set (USX_NUM - 2) and vertical code (27) for encoding terminator. | |
| #define | LF_CODE ((1 << 5) + 7) |
| Set (USX_SYM - 1) and vertical code (7) for encoding Line feed \n. | |
| #define | CRLF_CODE ((1 << 5) + 8) |
| Set (USX_NUM - 1) and vertical code (8) for encoding \r\n. | |
| #define | CR_CODE ((1 << 5) + 22) |
| Set (USX_NUM - 1) and vertical code (22) for encoding \r. | |
| #define | TAB_CODE ((1 << 5) + 14) |
| Set (USX_NUM - 1) and vertical code (14) for encoding \t. | |
| #define | NUM_SPC_CODE ((2 << 5) + 17) |
| Set (USX_NUM - 2) and vertical code (17) for space character when it appears in USX_NUM state \r. | |
| #define | UNI_STATE_SPL_CODE 0xF8 |
| Code for special code (11111) when state=USX_DELTA. | |
| #define | UNI_STATE_SPL_CODE_LEN 5 |
| Length of Code for special code when state=USX_DELTA. | |
| #define | UNI_STATE_SW_CODE 0x80 |
| Code for switch code when state=USX_DELTA. | |
| #define | UNI_STATE_SW_CODE_LEN 2 |
| Length of Code for Switch code when state=USX_DELTA. | |
| #define | SW_CODE 0 |
| Switch code in USX_ALPHA and USX_NUM 00. | |
| #define | SW_CODE_LEN 2 |
| Length of Switch code. | |
| #define | TERM_BYTE_PRESET_1 0 |
| Terminator bit sequence for Preset 1. Length varies depending on state as per following macros. | |
| #define | TERM_BYTE_PRESET_1_LEN_LOWER 6 |
| Length of Terminator bit sequence when state is lower. | |
| #define | TERM_BYTE_PRESET_1_LEN_UPPER 4 |
| Length of Terminator bit sequence when state is upper. | |
| #define | USX_OFFSET_94 33 |
| Offset at which usx_code_94 starts. | |
| #define | SAFE_APPEND_BITS(exp) |
| This is a safe call to append_bits() making sure it does not write past olen. More... | |
| #define | SAFE_APPEND_BITS2(olen, exp) |
| Macro used in the main compress function so that if the output len exceeds given maximum length (olen) it can exit. More... | |
| #define | SECTION_COUNT 5 |
| The list of veritical codes is split into 5 sections. Used by readVCodeIdx() | |
| #define | DEC_OUTPUT_CHAR(out, olen, ol, c) |
| Macro to ensure that the decoder does not append more than olen bytes to out. More... | |
| #define | DEC_OUTPUT_CHARS(olen, exp) |
| Macro to ensure that the decoder does not append more than olen bytes to out. More... | |
Typedefs | |
| typedef unsigned char | byte |
| byte is unsigned char | |
Functions | |
| void | init_coder () |
| int | append_bits (char *out, int olen, int ol, byte code, int clen) |
| int | append_switch_code (char *out, int olen, int ol, byte state) |
| Appends switch code to out depending on the state (USX_DELTA or other) | |
| int | append_code (char *out, int olen, int ol, byte code, byte *state, const byte usx_hcodes[], const byte usx_hcode_lens[]) |
| Appends given horizontal and veritical code bits to out. | |
| int | encodeCount (char *out, int olen, int ol, int count) |
| Encodes given count to out. | |
| int | encodeUnicode (char *out, int olen, int ol, int32_t code, int32_t prev_code) |
| Encodes the unicode code point given by code to out. prev_code is used to calculate the delta. | |
| int32_t | readUTF8 (const char *in, int len, int l, int *utf8len) |
| Reads UTF-8 character from in. Also returns the number of bytes occupied by the UTF-8 character in utf8len. | |
| int | matchOccurance (const char *in, int len, int l, char *out, int olen, int *ol, byte *state, const byte usx_hcodes[], const byte usx_hcode_lens[]) |
| int | matchLine (const char *in, int len, int l, char *out, int olen, int *ol, struct us_lnk_lst *prev_lines, byte *state, const byte usx_hcodes[], const byte usx_hcode_lens[]) |
| byte | getBaseCode (char ch) |
| char | getNibbleType (char ch) |
| int | append_nibble_escape (char *out, int olen, int ol, byte state, const byte usx_hcodes[], const byte usx_hcode_lens[]) |
| Starts coding of nibble sets. | |
| long | min_of (long c, long i) |
| Returns minimum value of two longs. | |
| int | append_final_bits (char *const out, const int olen, int ol, const byte state, const byte is_all_upper, const byte usx_hcodes[], const byte usx_hcode_lens[]) |
| Appends the terminator code depending on the state, preset and whether full terminator needs to be encoded to out or not . | |
| int | unishox2_compress_lines (const char *in, int len, UNISHOX_API_OUT_AND_LEN(char *out, int olen), const byte usx_hcodes[], const byte usx_hcode_lens[], const char *usx_freq_seq[], const char *usx_templates[], struct us_lnk_lst *prev_lines) |
| int | unishox2_compress (const char *in, int len, UNISHOX_API_OUT_AND_LEN(char *out, int olen), const byte usx_hcodes[], const byte usx_hcode_lens[], const char *usx_freq_seq[], const char *usx_templates[]) |
| int | unishox2_compress_simple (const char *in, int len, char *out) |
| int | readBit (const char *in, int bit_no) |
| int | read8bitCode (const char *in, int len, int bit_no) |
| int | readVCodeIdx (const char *in, int len, int *bit_no_p) |
| int | readHCodeIdx (const char *in, int len, int *bit_no_p, const byte usx_hcodes[], const byte usx_hcode_lens[]) |
| int | getStepCodeIdx (const char *in, int len, int *bit_no_p, int limit) |
| Returns the position of step code (0, 10, 110, etc.) encountered in the stream. | |
| int32_t | getNumFromBits (const char *in, int len, int bit_no, int count) |
| Reads specified number of bits and builds the corresponding integer. | |
| int32_t | readCount (const char *in, int *bit_no_p, int len) |
| Decodes the count from the given bit stream at in. Also updates bit_no_p. | |
| int32_t | readUnicode (const char *in, int *bit_no_p, int len) |
| int | writeUTF8 (char *out, int olen, int ol, int uni) |
| Write given unicode code point to out as a UTF-8 sequence. | |
| int | decodeRepeat (const char *in, int len, char *out, int olen, int ol, int *bit_no, struct us_lnk_lst *prev_lines) |
| Decode repeating sequence and appends to out. | |
| char | getHexChar (int32_t nibble, int hex_type) |
| Returns hex character corresponding to the 4 bit nibble. | |
| int | unishox2_decompress_lines (const char *in, int len, UNISHOX_API_OUT_AND_LEN(char *out, int olen), const byte usx_hcodes[], const byte usx_hcode_lens[], const char *usx_freq_seq[], const char *usx_templates[], struct us_lnk_lst *prev_lines) |
| int | unishox2_decompress (const char *in, int len, UNISHOX_API_OUT_AND_LEN(char *out, int olen), const byte usx_hcodes[], const byte usx_hcode_lens[], const char *usx_freq_seq[], const char *usx_templates[]) |
| int | unishox2_decompress_simple (const char *in, int len, char *out) |
Variables | |
| byte | usx_sets [][28] |
| This 2D array has the characters for the sets USX_ALPHA, USX_SYM and USX_NUM. Where a character cannot fit into a byte, 0 is used and handled in code. More... | |
| byte | usx_code_94 [94] |
| byte | usx_vcodes [] |
| Vertical codes starting from the MSB. More... | |
| byte | usx_vcode_lens [] |
| Length of each veritical code. More... | |
| byte | usx_freq_codes [] = {(1 << 5) + 25, (1 << 5) + 26, (1 << 5) + 27, (2 << 5) + 23, (2 << 5) + 24, (2 << 5) + 25} |
| Vertical Codes and Set number for frequent sequences in sets USX_SYM and USX_NUM. First 3 bits indicate set (USX_SYM/USX_NUM) and rest are vcode positions. | |
| const int | UTF8_MASK [] = {0xE0, 0xF0, 0xF8} |
| Not used. | |
| const int | UTF8_PREFIX [] = {0xC0, 0xE0, 0xF0} |
| Not used. | |
| byte | is_inited = 0 |
| global to indicate whether initialization is complete or not | |
| unsigned int | usx_mask [] = {0x80, 0xC0, 0xE0, 0xF0, 0xF8, 0xFC, 0xFE, 0xFF} |
| Mask for retrieving each code to be encoded according to its length. | |
| const byte | count_bit_lens [5] = {2, 4, 7, 11, 16} |
| Length of bits used to represent count for each level. | |
| const int32_t | count_adder [5] = {4, 20, 148, 2196, 67732} |
| Cumulative counts represented at each level. | |
| const byte | count_codes [] = {0x01, 0x82, 0xC3, 0xE4, 0xF4} |
| Codes used to specify the level that the count belongs to. | |
| const byte | uni_bit_len [5] = {6, 12, 14, 16, 21} |
| Length of bits used to represent delta code for each level. | |
| const int32_t | uni_adder [5] = {0, 64, 4160, 20544, 86080} |
| Cumulative delta codes represented at each level. | |
| byte | usx_vsections [] = {0x7F, 0xBF, 0xDF, 0xEF, 0xFF} |
| Used by readVCodeIdx() for finding the section under which the code read using read8bitCode() falls. | |
| byte | usx_vsection_pos [] = {0, 4, 8, 12, 20} |
| Used by readVCodeIdx() for finding the section vertical position offset. | |
| byte | usx_vsection_mask [] = {0x7F, 0x3F, 0x1F, 0x0F, 0x0F} |
| Used by readVCodeIdx() for masking the code read by read8bitCode() | |
| byte | usx_vsection_shift [] = {5, 4, 3, 1, 0} |
| Used by readVCodeIdx() for shifting the code read by read8bitCode() to obtain the vpos. | |
| byte | usx_vcode_lookup [36] |
| byte | len_masks [] = {0x80, 0xC0, 0xE0, 0xF0, 0xF8, 0xFC, 0xFE, 0xFF} |
Main code of Unishox2 Compression and Decompression library.
This file implements the code for the Unishox API function
defined in unishox2.h
| #define DEC_OUTPUT_CHAR | ( | out, | |
| olen, | |||
| ol, | |||
| c | |||
| ) |
Macro to ensure that the decoder does not append more than olen bytes to out.
| #define DEC_OUTPUT_CHARS | ( | olen, | |
| exp | |||
| ) |
Macro to ensure that the decoder does not append more than olen bytes to out.
| #define SAFE_APPEND_BITS | ( | exp | ) |
This is a safe call to append_bits() making sure it does not write past olen.
| #define SAFE_APPEND_BITS2 | ( | olen, | |
| exp | |||
| ) |
Macro used in the main compress function so that if the output len exceeds given maximum length (olen) it can exit.
| anonymous enum |
Enum indicating nibble type - USX_NIB_NUM means ch is a number '0' to '9',
USX_NIB_HEX_LOWER means ch is between 'a' to 'f',
USX_NIB_HEX_UPPER means ch is between 'A' to 'F'
| int append_bits | ( | char * | out, |
| int | olen, | ||
| int | ol, | ||
| byte | code, | ||
| int | clen | ||
| ) |
Appends specified number of bits to the output (out)
If maximum limit (olen) is reached, -1 is returned
Otherwise clen bits in code are appended to out starting with MSB
| byte getBaseCode | ( | char | ch | ) |
Returns 4 bit code assuming ch falls between '0' to '9',
'A' to 'F' or 'a' to 'f'
| char getNibbleType | ( | char | ch | ) |
Gets 4 bit code assuming ch falls between '0' to '9',
'A' to 'F' or 'a' to 'f'
| void init_coder | ( | ) |
Fills the usx_code_94 94 letter array based on sets of characters at usx_sets
For each element in usx_code_94, first 3 msb bits is set (USX_ALPHA / USX_SYM / USX_NUM)
and the rest 5 bits indicate the vertical position in the corresponding set
| int matchLine | ( | const char * | in, |
| int | len, | ||
| int | l, | ||
| char * | out, | ||
| int | olen, | ||
| int * | ol, | ||
| struct us_lnk_lst * | prev_lines, | ||
| byte * | state, | ||
| const byte | usx_hcodes[], | ||
| const byte | usx_hcode_lens[] | ||
| ) |
This is used only when encoding a string array Finds the longest matching sequence from the previous array element to the beginning of the string array.
If a match is found and it is longer than NICE_LEN, it is encoded as a repeating sequence to out
This is also used for Unicode strings
This is a crude implementation that is not optimized. Assuming only short strings
are encoded, this is not much of an issue.
| int matchOccurance | ( | const char * | in, |
| int | len, | ||
| int | l, | ||
| char * | out, | ||
| int | olen, | ||
| int * | ol, | ||
| byte * | state, | ||
| const byte | usx_hcodes[], | ||
| const byte | usx_hcode_lens[] | ||
| ) |
Finds the longest matching sequence from the beginning of the string.
If a match is found and it is longer than NICE_LEN, it is encoded as a repeating sequence to out
This is also used for Unicode strings
This is a crude implementation that is not optimized. Assuming only short strings
are encoded, this is not much of an issue.
| int readHCodeIdx | ( | const char * | in, |
| int | len, | ||
| int * | bit_no_p, | ||
| const byte | usx_hcodes[], | ||
| const byte | usx_hcode_lens[] | ||
| ) |
Decodes the horizontal code from the given bitstream at in
depending on the hcodes defined using usx_hcodes and usx_hcode_lens
Returns the horizontal code index or 99 if match could not be found.
Also updates bit_no_p with how many ever bits used by the horizontal code.
| int32_t readUnicode | ( | const char * | in, |
| int * | bit_no_p, | ||
| int | len | ||
| ) |
Decodes the Unicode codepoint from the given bit stream at in. Also updates bit_no_p
When the step code is 5, reads the next step code to find out the special code.
| int readVCodeIdx | ( | const char * | in, |
| int | len, | ||
| int * | bit_no_p | ||
| ) |
Decodes the vertical code from the given bitstream at in
This is designed to use less memory using a 36 byte buffer
compared to using a 256 byte buffer to decode the next 8 bits read by read8bitCode()
by splitting the list of vertical codes.
Decoder is designed for using less memory, not speed.
Returns the veritical code index or 99 if match could not be found.
Also updates bit_no_p with how many ever bits used by the vertical code.
| int unishox2_compress | ( | const char * | in, |
| int | len, | ||
| UNISHOX_API_OUT_AND_LEN(char *out, int olen) | , | ||
| const unsigned char | usx_hcodes[], | ||
| const unsigned char | usx_hcode_lens[], | ||
| const char * | usx_freq_seq[], | ||
| const char * | usx_templates[] | ||
| ) |
Comprehensive API for compressing a string
Presets are available for the last four parameters so they can be passed as single parameter.
See USX_PSET_* macros. Example call:
unishox2_compress(in, len, out, olen, USX_PSET_ALPHA_ONLY);
| [in] | in | Input ASCII / UTF-8 string |
| [in] | len | length in bytes |
| [out] | out | output buffer - should be large enough to hold compressed output |
| [in] | olen | length of 'out' buffer in bytes. Can be omitted if sufficient buffer is provided |
| [in] | usx_hcodes | Horizontal codes (array of bytes). See macro section for samples. |
| [in] | usx_hcode_lens | Length of each element in usx_hcodes array |
| [in] | usx_freq_seq | Frequently occuring sequences. See USX_FREQ_SEQ_* macros for samples |
| [in] | usx_templates | Templates of frequently occuring patterns. See USX_TEMPLATES macro. |
| int unishox2_compress_lines | ( | const char * | in, |
| int | len, | ||
| UNISHOX_API_OUT_AND_LEN(char *out, int olen) | , | ||
| const unsigned char | usx_hcodes[], | ||
| const unsigned char | usx_hcode_lens[], | ||
| const char * | usx_freq_seq[], | ||
| const char * | usx_templates[], | ||
| struct us_lnk_lst * | prev_lines | ||
| ) |
More Comprehensive API for compressing array of strings
See unishox2_compress() function for parameter definitions.
This function takes an additional parameter, i.e. 'prev_lines' - the usx_lnk_lst structure
See -g parameter in test_unishox2.c to find out how this can be used.
This function is used when an array of strings need to be compressed
and stored in a compressed array of bytes for use as a constant in other programs
where each element of the array can be decompressed and used at runtime.
| int unishox2_compress_simple | ( | const char * | in, |
| int | len, | ||
| char * | out | ||
| ) |
Simple API for compressing a string
| [in] | in | Input ASCII / UTF-8 string |
| [in] | len | length in bytes |
| [out] | out | output buffer - should be large enough to hold compressed output |
| int unishox2_decompress | ( | const char * | in, |
| int | len, | ||
| UNISHOX_API_OUT_AND_LEN(char *out, int olen) | , | ||
| const unsigned char | usx_hcodes[], | ||
| const unsigned char | usx_hcode_lens[], | ||
| const char * | usx_freq_seq[], | ||
| const char * | usx_templates[] | ||
| ) |
Comprehensive API for de-compressing a string
Presets are available for the last four parameters so they can be passed as single parameter.
See USX_PSET_* macros. Example call:
unishox2_decompress(in, len, out, olen, USX_PSET_ALPHA_ONLY);
| [in] | in | Input compressed bytes (output of unishox2_compress functions) |
| [in] | len | length of 'in' in bytes |
| [out] | out | output buffer - should be large enough to hold de-compressed output |
| [in] | olen | length of 'out' buffer in bytes. Can be omitted if sufficient buffer is provided |
| [in] | usx_hcodes | Horizontal codes (array of bytes). See macro section for samples. |
| [in] | usx_hcode_lens | Length of each element in usx_hcodes array |
| [in] | usx_freq_seq | Frequently occuring sequences. See USX_FREQ_SEQ_* macros for samples |
| [in] | usx_templates | Templates of frequently occuring patterns. See USX_TEMPLATES macro. |
| int unishox2_decompress_lines | ( | const char * | in, |
| int | len, | ||
| UNISHOX_API_OUT_AND_LEN(char *out, int olen) | , | ||
| const unsigned char | usx_hcodes[], | ||
| const unsigned char | usx_hcode_lens[], | ||
| const char * | usx_freq_seq[], | ||
| const char * | usx_templates[], | ||
| struct us_lnk_lst * | prev_lines | ||
| ) |
More Comprehensive API for de-compressing array of strings
This function is not be used in conjuction with unishox2_compress_lines()
See unishox2_decompress() function for parameter definitions.
Typically an array is compressed using unishox2_compress_lines() and
a header (.h) file is generated using the resultant compressed array.
This header file can be used in another program with another decompress
routine which takes this compressed array as parameter and index to be
decompressed.
| int unishox2_decompress_simple | ( | const char * | in, |
| int | len, | ||
| char * | out | ||
| ) |
Simple API for decompressing a string
| [in] | in | Input compressed bytes (output of unishox2_compress functions) |
| [in] | len | length of 'in' in bytes |
| [out] | out | output buffer for ASCII / UTF-8 string - should be large enough |
| byte len_masks[] = {0x80, 0xC0, 0xE0, 0xF0, 0xF8, 0xFC, 0xFE, 0xFF} |
Mask for retrieving each code to be decoded according to its length
Same as usx_mask so redundant
| byte usx_code_94[94] |
Stores position of letter in usx_sets. First 3 bits - position in usx_hcodes Next 5 bits - position in usx_vcodes
| byte usx_sets[][28] |
This 2D array has the characters for the sets USX_ALPHA, USX_SYM and USX_NUM. Where a character cannot fit into a byte, 0 is used and handled in code.
| byte usx_vcode_lens[] |
Length of each veritical code.
| byte usx_vcode_lookup[36] |
Vertical decoder lookup table - 3 bits code len, 5 bytes vertical pos code len is one less as 8 cannot be accommodated in 3 bits
| byte usx_vcodes[] |
Vertical codes starting from the MSB.


ROZ HAS A BLOG
Portfolio
Contact
About
Version Control Systems
Blog Post Title
Blog Post Title
Blog Post Title
Blog Post Title
Blog Post Title