Granite
Amber is a web framework written in the Crystal language.
This project is to provide an ORM in Crystal.
Installation
Add this library to your projects dependencies along with the driver in
your shard.yml
. This can be used with any framework but was originally
designed to work with the amber framework in mind. This library will work
with kemal or any other framework as well.
dependencies:
granite:
github: amberframework/granite
# Pick your database
mysql:
github: crystal-lang/crystal-mysql
sqlite3:
github: crystal-lang/crystal-sqlite3
pg:
github: will/crystal-pg
Next you will need to create a config/database.yml
You can leverage environment variables using ${}
syntax.
mysql:
database: "mysql://username:password@hostname:3306/database_${AMBER_ENV}"
pg:
database: "postgres://username:password@hostname:5432/database"
sqlite:
database: "sqlite3:./config/${DB_NAME}.db"
Or you can set the DATABASE_URL
environment variable. This will override the config/database.yml
Usage
Here is an example using Granite Model
require "granite/adapter/mysql"
class Post < Granite::Base
adapter mysql
field name : String
field body : String
timestamps
end
You can disable the timestamps for SqlLite since TIMESTAMP is not supported for this database:
require "granite/adapter/sqlite"
class Comment < Granite::Base
adapter sqlite
table_name post_comments
field name : String
field body : String
end
id, created_at, updated_at
The primary key is automatically created for you and if you use timestamps
they will be
automatically updated for you.
Here are the MySQL field definitions for id, created_at, updated_at
id BIGINT NOT NULL AUTO_INCREMENT
# Your fields go here
created_at TIMESTAMP
updated_at TIMESTAMP
PRIMARY KEY (id)
Custom Primary Key
For legacy database mappings, you may already have a table and the primary key is not named id
or Int64
.
We have a macro called primary
to help you out:
class Site < Granite::Base
adapter mysql
primary custom_id : Int32
field name : String
end
This will override the default primary key of id : Int64
.
Natural Keys
For natural keys, you can set auto: false
option to disable auto increment insert.
class Site < Granite::Base
adapter mysql
primary code : String, auto: false
field name : String
end
UUIDs
For databases that utilize UUIDs as the primary key, the primary
macro can be used again with the auto: false
option. A before_create
callback can be added to the model to randomly generate and set a secure UUID on the record before it is saved to the database.
class Book < Granite::Base
require "uuid"
adapter mysql
primary ISBN : String, auto: false
field name : String
before_create :assign_isbn
def assign_isbn
@ISBN = UUID.random.to_s
end
end
Bulk Insertions
Import
Note: Imports do not trigger callbacks automatically. See Running Callbacks.
Each model has an import
class level method to import an array of models in one bulk insert statement.
models = [
Model.new(id: 1, name: "Fred", age: 14),
Model.new(id: 2, name: "Joe", age: 25),
Model.new(id: 3, name: "John", age: 30),
]
Model.import(models)
update_on_duplicate
The import
method has an optional update_on_duplicate
+ columns
params that allows you to specify the columns (as an array of strings) that should be updated if primary constraint is violated.
models = [
Model.new(id: 1, name: "Fred", age: 14),
Model.new(id: 2, name: "Joe", age: 25),
Model.new(id: 3, name: "John", age: 30),
]
Model.import(models)
Model.find!(1).name # => Fred
models = [
Model.new(id: 1, name: "George", age: 14),
]
Model.import(models, update_on_duplicate: true, columns: %w(name))
Model.find!(1).name # => George
NOTE: If using PostgreSQL you must have version 9.5+ to have the on_duplicate_key_update feature.
ignore_on_duplicate
The import
method has an optional ignore_on_duplicate
param, that takes a boolean, which will skip records if the primary constraint is violated.
models = [
Model.new(id: 1, name: "Fred", age: 14),
Model.new(id: 2, name: "Joe", age: 25),
Model.new(id: 3, name: "John", age: 30),
]
Model.import(models)
Model.find!(1).name # => Fred
models = [
Model.new(id: 1, name: "George", age: 14),
]
Model.import(models, ignore_on_duplicate: true)
Model.find!(1).name # => Fred
batch_size
The import
method has an optional batch_size
param, that takes an integer. The batch_size determines the number of models to import in each INSERT statement. This defaults to the size of the models array, i.e. only 1 INSERT statement.
models = [
Model.new(id: 1, name: "Fred", age: 14),
Model.new(id: 2, name: "Joe", age: 25),
Model.new(id: 3, name: "John", age: 30),
Model.new(id: 3, name: "Bill", age: 66),
]
Model.import(models, batch_size: 2)
# => First SQL INSERT statement imports Fred and Joe
# => Second SQL INSERT statement imports John and Bill
Running Callbacks
Since the import
method runs on the class level, callbacks are not triggered automatically, they have to be triggered manually. For example, using the Item class with a UUID primary key:
require "uuid"
class Item < Granite::Base
adapter mysql
table_name items
primary item_id : String, auto: false
field item_name : String
before_create :generate_uuid
def generate_uuid
@item_id = UUID.random.to_s
end
end
items = [
Item.new(item_name: "item1"),
Item.new(item_name: "item2"),
Item.new(item_name: "item3"),
Item.new(item_name: "item4"),
]
# If we did `Item.import(items)` now, it would fail since the item_id wouldn't get set before saving the record, violating the primary key constraint.
# Manually run the callback on each model to generate the item_id.
items.each(&.before_create)
# Each model in the array now has a item_id set, so can be imported.
Item.import(items)
# This can also be used for a single record.
item = Item.new(item_name: "item5")
item.before_create
item.save
Note: Manually running your callbacks is mainly aimed at bulk imports. Running them before a normal .save
, for example, would run your callbacks twice.
SQL
To clear all the rows in the database:
Post.clear #truncate the table
Find All
posts = Post.all
if posts
posts.each do |post|
puts post.name
end
end
Find First
post = Post.first
if post
puts post.name
end
post = Post.first! # raises when no records exist
Find
post = Post.find 1
if post
puts post.name
end
post = Post.find! 1 # raises when no records found
Find By
post = Post.find_by :slug, "example_slug"
if post
puts post.name
end
post = Post.find_by! :slug, "foo" # raises when no records found
Insert
post = Post.new
post.name = "Granite Rocks!"
post.body = "Check this out."
post.save
Update
post = Post.find 1
post.name = "Granite Really Rocks!"
post.save
Delete
post = Post.find 1
post.destroy
puts "deleted" unless post
Queries
The where clause will give you full control over your query.
All
When using the all
method, the SQL selected fields will always match the
fields specified in the model.
Always pass in parameters to avoid SQL Injection. Use a ?
in your query as placeholder. Checkout the Crystal DB Driver
for documentation of the drivers.
Here are some examples:
posts = Post.all("WHERE name LIKE ?", ["Joe%"])
if posts
posts.each do |post|
puts post.name
end
end
# ORDER BY Example
posts = Post.all("ORDER BY created_at DESC")
# JOIN Example
posts = Post.all("JOIN comments c ON c.post_id = post.id
WHERE c.name = ?
ORDER BY post.created_at DESC",
["Joe"])
First
It is common to only want the first result and append a LIMIT 1
to the query.
This is what the first
method does.
For example:
post = Post.first("ORDER BY posts.name DESC")
This is the same as:
post = Post.all("ORDER BY posts.name DESC LIMIT 1").first
Relationships
One to Many
belongs_to
and has_many
macros provide a rails like mapping between Objects.
class User < Granite::Base
adapter mysql
has_many :posts
field email : String
field name : String
timestamps
end
This will add a posts
instance method to the user which returns an array of posts.
class Post < Granite::Base
adapter mysql
belongs_to :user
field title : String
timestamps
end
This will add a user
and user=
instance method to the post.
For example:
user = User.find 1
user.posts.each do |post|
puts post.title
end
post = Post.find 1
puts post.user
post.user = user
post.save
In this example, you will need to add a user_id
and index to your posts table:
CREATE TABLE posts (
id BIGSERIAL PRIMARY KEY,
user_id BIGINT,
title VARCHAR,
created_at TIMESTAMP,
updated_at TIMESTAMP
);
CREATE INDEX 'user_id_idx' ON posts (user_id);
Many to Many
Instead of using a hidden many-to-many table, Granite recommends always creating a model for your join tables. For example, let's say you have many users
that belong to many rooms
. We recommend adding a new model called participants
to represent the many-to-many relationship.
Then you can use the belongs_to
and has_many
relationships going both ways.
class User < Granite::Base
has_many :participants
field name : String
end
class Participant < Granite::Base
belongs_to :user
belongs_to :room
end
class Room < Granite::Base
has_many :participants
field name : String
end
The Participant class represents the many-to-many relationship between the Users and Rooms.
Here is what the database table would look like:
CREATE TABLE participants (
id BIGSERIAL PRIMARY KEY,
user_id BIGINT,
room_id BIGINT,
created_at TIMESTAMP,
updated_at TIMESTAMP
);
CREATE INDEX 'user_id_idx' ON TABLE participants (user_id);
CREATE INDEX 'room_id_idx' ON TABLE participants (room_id);
has_many through:
As a convenience, we provide a through:
clause to simplify accessing the many-to-many relationship:
class User < Granite::Base
has_many :participants
has_many :rooms, through: participants
field name : String
end
class Participant < Granite::Base
belongs_to :user
belongs_to :room
end
class Room < Granite::Base
has_many :participants
has_many :users, through: participants
field name : String
end
This will allow you to find all the rooms that a user is in:
user = User.first
user.rooms.each do |room|
puts room.name
end
And the reverse, all the users in a room:
room = Room.first
room.users.each do |user|
puts user.name
end
Errors
All database errors are added to the errors
array used by Granite::Validators with the symbol ':base'
post = Post.new
post.save
post.errors[0].to_s.should eq "ERROR: name cannot be null"
Callbacks
There is support for callbacks on certain events.
Here is an example:
require "granite/adapter/pg"
class Post < Granite::Base
adapter pg
before_save :upcase_title
field title : String
field content : String
timestamps
def upcase_title
if title = @title
@title = title.upcase
end
end
end
You can register callbacks for the following events:
Create
- before_save
- before_create
- save
- after_create
- after_save
Update
- before_save
- before_update
- save
- after_update
- after_save
Destroy
- before_destroy
- destroy
- after_destroy
Migration
migrator
providesdrop
,create
anddrop_and_create
methods
class User < Granite::Base
adapter mysql
field name : String
end
User.migrator.drop_and_create
# => "DROP TABLE IF EXISTS `users`;"
# => "CREATE TABLE `users` (id BIGSERIAL PRIMARY KEY, name VARCHAR(255));"
User.migrator(table_options: "ENGINE=InnoDB DEFAULT CHARSET=utf8").create
# => "CREATE TABLE ... ENGINE=InnoDB DEFAULT CHARSET=utf8;"
Contributing
- Fork it ( https://github.com/amberframework/granite/fork )
- Create your feature branch (git checkout -b my-new-feature)
- Commit your changes (git commit -am 'Add some feature')
- Push to the branch (git push origin my-new-feature)
- Create a new Pull Request
Running tests
Granite uses Crystal's built in test framework. The tests can be run with $ crystal spec
.
The test suite depends on access to a PostgreSQL, MySQL, and SQLite database to ensure the adapters work as intended.
Docker setup
There is a self-contained testing environment provided via the docker-compose.yml
file in this repository.
After you have docker installed do the following to run tests:
First run
Subsequent runs
Cleanup
If you're done testing and you'd like to shut down and clean up the docker dependences run the following:
Local setup
If you'd like to test without docker you can do so by following the instructions below:
- Install dependencies with
$ crystal deps
- Update .env to use appropriate ENV variables, or create appropriate databases.
 TO 'granite'@'localhost' WITH GRANT OPTION;
- Export
.env
with$ source .env
$ crystal spec


MulensModel.event module¶
-
class
MulensModel.event.
Event
(datasets=None, model=None, coords=None)¶ Bases:
object
Combines a microlensing model with data. Allows calculating chi^2 and making a number of plots.
- Arguments :
datasets
:MulensData
or list ofMulensData
objectsDatasets that will be linked to the event. These datasets will be used for chi^2 calculation, plotting etc.
model
Model
Microlensing model that will be linked to the event. In order to get chi^2 for different sets of model parameters you should keep a single
Model
instance and change parameters for this model (i.e., do not provide separateModel
instances).
coords
: str,Coordinates
, or astropy.SkyCoordCoordinates of the event. If str, then needs format accepted by astropy.SkyCoord e.g.,
'18:00:00 -30:00:00'
.
The datasets can be in magnitude or flux spaces. When we calculate chi^2 we do it in magnitude or flux space depending on value of
chi2_fmt
attribute. If dataset is in magnitude space and model results in negative flux, then we calculate chi^2 in flux space but only for the epochs with negative model flux.-
plot_model
(**kwargs)¶ Plot the model light curve in magnitudes. See
MulensModel.model.Model.plot_lc()
for details.
-
plot_data
(**kwargs)¶ Plot the data scaled to the model. See
MulensModel.model.Model.plot_data()
for details.
-
plot_residuals
(**kwargs)¶ Plot the residuals (in magnitudes) of the model. See
MulensModel.model.Model.plot_residuals()
for details.
-
plot_trajectory
(**kwargs)¶ Plot the trajectory of the source. See : py:func:MulensModel.model.Model.plot_trajectory() for details.
-
plot_source_for_datasets
(**kwargs)¶ Plot source positions for all linked datasets. See
MulensModel.model.Model.plot_source_for_datasets()
for details.
-
get_ref_fluxes
(data_ref=None, fit_blending=None)¶ Get source and blending fluxes for the reference dataset. See
MulensModel.model.Model.get_ref_fluxes()
for details.
-
get_chi2
(fit_blending=None)¶ Calculates chi^2 of current model by fitting for source and blending fluxes.
- Parameters :
- fit_blending: boolean, optional
If True, then the blend flux is a free parameter. If False, the blend flux is fixed at zero. Default is the same as
MulensModel.fit.Fit.fit_fluxes()
.
- Returns :
- chi2: float
Chi^2 value
-
get_chi2_for_dataset
(index_dataset, fit_blending=None)¶ Calculates chi^2 for a single dataset
- Parameters :
- index_dataset: int
index that specifies for which dataset the chi^2 is requested
- fit_blending: boolean, optional
Are we fitting for blending flux? If not then blending flux is fixed to 0. Default is the same as
MulensModel.fit.Fit.fit_fluxes()
.
- Returns :
- chi2: float
chi2 for dataset[index_dataset].
-
get_chi2_per_point
(fit_blending=None)¶ Calculates chi^2 for each data point of the current model by fitting for source and blending fluxes.
- Parameters :
- fit_blending: boolean, optional
Are we fitting for blending flux? If not then blending flux is fixed to 0. Default is the same as
MulensModel.fit.Fit.fit_fluxes()
.
- Returns :
- chi2: list of np.ndarray
Chi^2 contribution from each data point, e.g.
chi2[data_num][k]
returns the chi2 contribution from the k-th point of dataset data_num.
- Example :
Assuming
event
is instance of Event class to get chi2 for 10-th point point of 0-th dataset.chi2 = event.get_chi2_per_point() print(chi2[0][10])
-
chi2_gradient
(parameters, fit_blending=None)¶ Calculate chi^2 gradient (also called Jacobian), i.e., \(d chi^2/d parameter\).
- Parameters :
- parameters: str or list, required
Parameters with respect to which gradient is calculated. Currently accepted parameters are:
t_0
,u_0
,t_eff
,t_E
,pi_E_N
, andpi_E_E
. The parameters for which you request gradient must be defined in py:attr:~model.- fit_blending: boolean, optional
Are we fitting for blending flux? If not then blending flux is fixed to 0. Default is the same as
MulensModel.fit.Fit.fit_fluxes()
.
- Returns :
- gradient: float or np.ndarray
chi^2 gradient
-
reset_best_chi2
()¶ Reset
best_chi2
attribute and its parameters (best_chi2_parameters
).
-
property
coords
¶ see
Coordinates
-
property
datasets
¶ a list of
MulensData
instances.
-
property
data_ref
¶ Reference data set for scaling the model fluxes to (for plotting). May be set as a
MulensData
object or an index (int). Default is the first data set.
-
property
best_chi2
¶ float
The smallest value returned by
get_chi2()
.
-
property
best_chi2_parameters
¶ dict
Parameters that gave the smallest chi2.
-
property
sum_function
¶ str
Function used for adding chi^2 contributions. Can be either ‘math.fsum’ (default value) or ‘numpy.sum’. The former is slightly slower and more accurate, which may be important for large datasets.
-
clean_data
()¶ masks outlying datapoints. Not Implemented.
-
estimate_model_params
()¶ estimates model parameters without fitting them. Not Implemented.


mat-radio
provides the same functionality as a native
<input type="radio">
enhanced with Material Design
styling and animations.
@Input() disabled: boolean
Whether the radio group is
disabled
@Input() labelPosition: 'before' | 'after'
Whether the
labels should appear after or before the radio-buttons. Defaults to
'after'
@Output() change: EventEmitter<MatRadioChange>
Event emitted when the group value changes. Change events are only
emitted when the value changes due to user interaction with a radio
button (the same behavior as
<input type-"radio">
).


{% avocado load field "variants.sift.score" as sift %} {% avocado load field "variants.polyphen2.score" as polyphen2 %}
Deleterous prediction rating for SNPs. Since these ratings are specific to SNPs, INDELs will not be affected if a rating is selected.


Examples
It's our goal to create a wide variety of example of how Tether
can be used. Here's what we have so far, please send a PR with
any examples you might create.
Beginner
- simple: A simple example to get you started
- out-of-bounds: How to hide the element when it would
otherwise be offscreen
- pin: How to pin the element so it never goes offscreen
- enable-disable: How to enable and disable the Tether
in JavaScript
Advanced
- content-visible: Demonstrates using the
'visible'
targetModifier
to align an element with the visible portion of another.
- dolls: A performance test to show several dozen elements,
each tethered to the previous. Try dragging the top left tether.
- element-scroll: Demonstrates using the
'scroll-handle'
targetModifier
to align an element with the scrollbar of an element.
- scroll: Demonstrates using the
'scroll-handle'
targetModifier
to align an element with the body's scroll handle.
- viewport: Demonstrates aligning an element with the
viewport by using the
'visible'
targetModifier
when tethered to the body.


We're sorry but this application doesn't work properly without JavaScript enabled. Please enable it to continue.


This is not the end! this is just skyfall
just for fun!!
Section 1: Basic info
Name: John Doe
Place: India
Section 2: Resume
John's Resume
Section 3: Webs
John's github profile
John's linkedin profile
Section 3: External videos
A youtube video
Description
A youtube video inserted in your webpage with embed link
Section 5: Everything Else
Cricket
(playing more than watching)
Football
(watching)
books
(reading)
movies
(somewhat)
music
(listening)


Introduction
Virat Kohli is an Indian cricketer who currently captains the India national team. A right-handed top-order batsman, Kohli is regarded as one of the best batsmen in the world. He plays for Royal Challengers Bangalore in the Indian Premier League (IPL), and has been the team's captain since 2013. Since October 2017, he has been the top-ranked ODI batsman in the world and is currently 1st in Test rankings with 928 points. Among Indian batsmen, Kohli has the best ever Test rating (937 points), ODI rating (911 points) and T20I rating (897 points).
Kohli has been the recipient of many awards such as the Sir Garfield Sobers Trophy (ICC Cricketer of the Year) in 2017 and 2018; ICC Test Player of the Year 2018; ICC ODI Player of the Year in 2012, 2017 and 2018 and Wisden Leading Cricketer in the World in 2016, 2017 and 2018.
Kohli was appointed the vice-captain of the ODI team in 2012 and handed over the Test captaincy following Mahendra Singh Dhoni's Test retirement in 2014. In early 2017, he became the limited-overs captain as well after Dhoni stepped down from the position. In ODIs, Kohli has the second highest number of centuries and the highest number of centuries in run-chases in the world.


{% extends 'base.html' %} {% load humanize %} {% block content %}
Name: {{ package.name }}
Date of Birth: {{ package.dob }}
ID: {{ package.tax_id }}
Farm Size: {{ package.land_area }} hectares
Family Members: {{ package.num_family }}
Expected Payments
Loss of Harvest: $1,450 x {{ package.land_area }} hectares = $ {{ package.get_loss_of_harvest_compensation|intcomma }}
Food Security: $80 x {{ package.num_family }} family members = $ {{ package.get_num_family_compensation|intcomma }}
Final Compensation: $1,450 x {{ package.land_area }} hectares x 5 years x 1.1725 inflation adjustment = $ {{ package.get_final_compensation|intcomma }}
Verified Payments:
| Date | Lost Harvest | Food Security | Improvements | Final Compensation | Total |
|---|---|---|---|---|---|
| {{ key }} | {% for payment in value.payments %}{% if payment.payment_type == 'lost_harvest' %}$ {{ payment.value|intcomma }}{% endif %}{% endfor %} | {% for payment in value.payments %}{% if payment.payment_type == 'food_security' %}$ {{ payment.value|intcomma }}{% endif %}{% endfor %} | {% for payment in value.payments %}{% if payment.payment_type == 'improvements' %}$ {{ payment.value|intcomma }}{% endif %}{% endfor %} | {% for payment in value.payments %}{% if payment.payment_type == 'final_compensation' %}$ {{ payment.value|intcomma }}{% endif %}{% endfor %} | $ {{ value.total|intcomma }} |


--- layout: post title: "Multiprocessing in python" date: 2017-08-25 comments: true archive: false tags: multiprocessing-python excerpt: Multiprocessing in python ---
In this blog post I'll try to cover the difference between the
functionality found in the different functions that the
multiprocessing
package offers in python. The multiprocessing
package offers the ability to work with multiple threads and if you
wish with multiplce cores. First I'll try to cover the multiple core
scenario and explain the difference between map, map_async
, imap,
imap_unorderd
, apply, apply_async
. In a later blog post I'll try to
cover the multithreaded scenario. First I'll start with
map/map_async
and imap/imap_unorderd
.
map
operates on function that accepts only one arguement, it is
concurrent, it blocks your code, meaning nothing else can be executed
until its finished and finally it returns your results in an ordered
manner. map
consumes your iterable by converting the iterable to a
list assuming it isn't already, breaking it into chunks, and
sending those chunks to the worker processes in the Pool
. Breaking the
iterable into chunks performs better than passing each item in the
iterable between processes one item at a time - particularly if the
iterable is large. However, turning the iterable into a list in order
to chunk it can have a very high memory cost, since the entire list
will need to be kept in memory.
map_async
on the other hand can operate on a function
accepting multiple arguements it is not concurrent, it allows excution
of code while it is still running, and finally it doesn't return your
results in an ordered manner.
imap
doesn't turn the iterable you give it into a list, nor does break
it into chunks by default. It will iterate over the iterable one
element at a time, and send them each to a worker process. This means
you don't take the memory hit of converting the whole iterable to a
list, but it also means the performance is slower for large iterables,
because of the lack of chunking. This can be mitigated by passing a
chunksize
argument larger than default of 1, however.
The other major difference between imap/imap_unordered
and
map/map_async
, is that with imap/imap_unordered
, you can start
receiving results from workers as soon as they're ready, rather than
having to wait for all of them to be finished. With map_async
, an
AsyncResult
is returned right away, but you can't actually retrieve
results from that object until all of them have been processed, at
which points it returns the same list that map
does, basically map
is actually implemented internally as map_async(...).get()
. There's
no way to get partial results, you either have the entire result, or
nothing.
imap
and imap_unordered
both return iterables right away. With imap
,
the results will be yielded from the iterable as soon as they're
ready, while still preserving the ordering of the input iterable. With
imap_unordered
, results will be yielded as soon as they're ready,
regardless of the order of the input iterable.
To summarize the key differences between imap/imap_unordered
and
map/map_async
are:
apply
sends a single task off to a worker process, and then blocks
until it's complete. apply_async
sends a single task off to a work
process, and then immediately returns an AsyncResult object, which can
be used to wait for the task to finish and retrieve the
result. apply
is implemented by simply calling
apply_async(...).get()
. With apply_async
, if an exception occurs
inside of your calling function, you won't know about it unless you
explicitly call Pool['process_x'].get()
on the failing AsyncResult
object, which would require iterating over all of Pool
. With
map_async the exception will be raised if you call
~async_result.get()
- no iteration required. map_async
has
built-in chunking functionality, which will make your code perform
noticeably better if arguement list is very large.
Here is a final table summarizing all of those properties and diffferencies:
| Multi-args | Concurrence | Blocking | Ordered-results | |||||
|---|---|---|---|---|---|---|---|---|
| map | no | yes | yes | yes | ||||
| map-async | no | yes | no | yes | ||||
| apply | yes | no | yes | no | ||||
| apply-async | yes | yes | no | no | ||||
| imap | no | yes | yes | yes | ||||
| imap-async | no | yes | no | no |


fopencookie
—open a stream with custom callbacks#include <stdio.h> FILE *fopencookie(const void *cookie, const char *mode, cookie_io_functions_t functions);
Description
fopencookie
creates a FILE
stream where I/O is performed using
custom callbacks. The callbacks are registered via the structure:
typedef ssize_t (*cookie_read_function_t)(void *_cookie, char *_buf, size_t _n); typedef ssize_t (*cookie_write_function_t)(void *_cookie, const char *_buf, size_t _n); typedef int (*cookie_seek_function_t)(void *_cookie, off_t *_off, int _whence); typedef int (*cookie_close_function_t)(void *_cookie);
typedef struct { cookie_read_function_t *read; cookie_write_function_t *write; cookie_seek_function_t *seek; cookie_close_function_t *close; } cookie_io_functions_t;
The stream is opened with mode treated as in fopen
. The
callbacks functions.read and functions.write may only be NULL
when mode does not require them.
functions.read should return -1 on failure, or else the number of
bytes read (0 on EOF). It is similar to read
, except that
cookie will be passed as the first argument.
functions.write should return -1 on failure, or else the number of
bytes written. It is similar to write
, except that cookie
will be passed as the first argument.
functions.seek should return -1 on failure, and 0 on success, with
_off set to the current file position. It is a cross between
lseek
and fseek
, with the _whence argument interpreted in
the same manner. A NULL functions.seek makes the stream behave
similarly to a pipe in relation to stdio functions that require
positioning.
functions.close should return -1 on failure, or 0 on success. It
is similar to close
, except that cookie will be passed as the
first argument. A NULL functions.close merely flushes all data
then lets fclose
succeed. A failed close will still invalidate
the stream.
Read and write I/O functions are allowed to change the underlying
buffer on fully buffered or line buffered streams by calling
setvbuf
. They are also not required to completely fill or empty
the buffer. They are not, however, allowed to change streams from
unbuffered to buffered or to change the state of the line buffering
flag. They must also be prepared to have read or write calls occur on
buffers other than the one most recently specified.
Returns
NULL
is returned, and errno
will be set to EINVAL if a
function pointer is missing or mode is invalid, ENOMEM if the
stream cannot be created, or EMFILE if too many streams are already
open.
Portability
funopen
interface from BSD.
Supporting OS subroutines required: sbrk
.


Review Wizard Status Report for December 2009
News
Polygon Library Accepted
- Boost 1.40 Released
- New Libraries: None Revised Libraries: Accumulators, Asio, Circular Buffer, Filesystem, Foreach, Function, Fusion, Hash, Interprocess, Intrusive, MPL, Program Options, Proto, Python, Serialization, Unordered, Xpressive
Geometry Library Accepted
- Boost 1.41 Released
- New Libraries: Property Tree Revised Libraries: DateTime, Filesystem, Iostreams, Math, Multi-index Containers, Proto, Python, Regex, Spirit, System, Thread, Unordered, Utility, Wave, Xpressive
MSM Library Review Underway
Constrained Value Review - Review Result still Pending
Older Issues
The Time Series Library, accepted in August 2007, has not yet been submitted to SVN.
The Floating Point Utilities Library, has not yet been submitted to SVN. It is slated to be integrated with the Boost.Math library.
The Switch Library, accepted provisionally in January 2008, has not yet been submitted for mini-review and full acceptance.
The Phoenix Library, accepted provisionally in September 2008, has not yet been submitted for mini-review and full acceptance.
For libraries that are still waiting to get into SVN, please get them ready and into the repository. The developers did some great work making the libraries, so don't miss the chance to share that work with others.
General Announcements
As always, we need experienced review managers. The review queue has been growing substantially but we have had few volunteers, so manage reviews if possible and if not please make sure to watch the review schedule and participate. Please take a look at the list of libraries in need of managers and check out their descriptions. In general review managers are active boost participants or library contributors. If you can serve as review manager for any of them, email Ron Garcia or John Phillips, "garcia at osl dot iu dot edu" and "phillips at mps dot ohio-state dot edu" respectively.
We are also suffering from a lack of reviewers. While we all understand time pressures and the need to complete paying work, the strength of Boost is based on the detailed and informed reviews submitted by you. A recent effort is trying to secure at least five people who promise to submit reviews as a precondition to starting the review period. Consider volunteering for this and even taking the time to create the review as early as possible. No rule says you can only work on a review during the review period.
A link to this report will be posted to www.boost.org. If you would like us to make any modifications or additions to this report before we do that, please email Ron or John.
If you're a library author and plan on submitting a library for review in the next 3-6 months, send Ron or John a short description of your library and we'll add it to the Libraries Under Construction below. We know that there are many libraries that are near completion, but we have hard time keeping track all of them. Please keep us informed about your progress.
The included review queue isn't a classic queue. It is more an unordered list of the libraries awaiting review. As such, any library in the queue can be reviewed once the developer is ready and a review manager works with the wizards and the developer to schedule a review. It is not FIFO.
Review Queue
- Lexer
- Shifted Pointer
- Logging
- Log
- Join
- Pimpl
- Task
- Endian
- Conversion
- Sorting
- GIL.IO
- AutoBuffer
- String Convert
- Move
- Containers
- Interval Containers
- Type Traits Extensions
- Interthreads
- Bitfield
- Lockfree
Lexer
| Author: | Ben Hanson |
|---|---|
| Review Manager: | Eric Neibler |
| Download: | Boost Vault |
| Description: | A programmable lexical analyser generator inspired by 'flex'. Like flex, it is programmed by the use of regular expressions and outputs a state machine as a number of DFAs utilising equivalence classes for compression. |
Shifted Pointer
| Author: | Phil Bouchard |
|---|---|
| Review Manager: | Needed |
| Download: | Boost Vault |
| Description: | Smart pointers are in general optimized for a specific resource (memory usage, CPU cycles, user friendliness, ...) depending on what the user need to make the most of. The purpose of this smart pointer is mainly to allocate the reference counter (or owner) and the object itself at the same time so that dynamic memory management is simplified thus accelerated and cheaper on the memory map. |
Logging
| Author: | John Torjo |
|---|---|
| Review Manager: | Gennadiy Rozental |
| Download: | http://torjo.com/log2/ |
| Description: | Used properly, logging is a very powerful tool. Besides aiding debugging/testing, it can also show you how your application is used. The Boost Logging Library allows just for that, supporting a lot of scenarios, ranging from very simple (dumping all to one destination), to very complex (multiple logs, some enabled/some not, levels, etc). It features a very simple and flexible interface, efficient filtering of messages, thread-safety, formatters and destinations, easy manipulation of logs, finding the best logger/filter classes based on your application's needs, you can define your own macros and much more! |
Log
| Author: | Andrey Semashev |
|---|---|
| Review Manager: | Needed |
| Download: | Boost Vault |
| Description: | The library is aimed to help adding logging features to applications. It provides out-of-box support for many widely used capabilities, such as formatting and filtering based on attributes, sending logs to a syslog server or to Windows Event Log, or simply storing logs into files. It also provides basic support for the library initialization from a settings file. The library can also be used for a wider range of tasks and implement gathering and processing statistical information or notifying user about application events. |
Join
| Author: | Yigong Liu |
|---|---|
| Review Manager: | Needed |
| Download: | http://channel.sourceforge.net/ |
| Description: | Join is an asynchronous, message based C++ concurrency library based on join calculus. It is applicable both to multi-threaded applications and to the orchestration of asynchronous, event-based applications. It follows Comega's design and implementation and builds with Boost facilities. It provides a high level concurrency API with asynchronous methods, synchronous methods, and chords which are "join-patterns" defining the synchronization, asynchrony, and concurrency. |
Pimpl
| Author: | Vladimir Batov |
|---|---|
| Review Manager: | Needed |
| Download: | http://www.ddj.com/cpp/205918714 (documentation)
|
| Description: | The Pimpl idiom is a simple yet robust technique to minimize coupling via the separation of interface and implementation and then implementation hiding. This library provides a convenient yet flexible and generic deployment technique for the Pimpl idiom. It's seemingly complete and broadly applicable, yet minimal, simple and pleasant to use. |
Task
| Author: | Oliver Kowalke |
|---|---|
| Review Manager: | Needed |
| Download: | |
| Description: | Formerly called Thread Pool The library provides: * thread creation policies:
|
Endian
| Author: | Beman Dawes |
|---|---|
| Review Manager: | Needed |
| Download: | http://mysite.verizon.net/beman/endian-0.10.zip |
| Description: |
Conversion
| Author: | Vicente Botet |
|---|---|
| Review Manager: | Needed |
| Download: | |
| Description: | Generic explicit conversion between unrelated types.
|
Sorting
| Author: | Steven Ross |
|---|---|
| Review Manager: | Needed |
| Download: | |
| Description: | A grouping of 3 templated hybrid radix/comparison-based sorting algorithms that provide superior worst-case and average-case performance to std::sort: integer_sort, which sorts fixed-size data types that support a rightshift (default of >>) and a comparison (default of <) operator. float_sort, which sorts standard floating-point numbers by safely casting them to integers. string_sort, which sorts variable-length data types, and is optimized for 8-bit character strings. All 3 algorithms have O(n(k/s + s)) runtime where k is the number of bits in the data type and s is a constant, and limited memory overhead (in the kB for realistic inputs). In testing, integer_sort varies from 35% faster to 8X as fast as std::sort, depending on processor, compiler optimizations, and data distribution. float_sort is roughly 7X as fast as std::sort on x86 processors. string_sort is roughly 2X as fast as std::sort. |
GIL.IO
| Author: | Christian Henning |
|---|---|
| Review Manager: | Needed |
| Download: | GIL Google Code Vault |
| Description: | I/O extension for boost::gil which allows reading and writing of/in various image formats ( tiff, jpeg, png, etc ). This review will also include the Toolbox extension which adds some common functionality to gil, such as new color spaces, algorithms, etc. |
AutoBuffer
| Author: | Thorsten Ottosen |
|---|---|
| Review Manager: | Needed |
| Download: | Here |
| Description: | Boost.AutoBuffer provides a container for efficient dynamic, local buffers. Furthermore, the container may be used as an alternative to std::vector, offering greater flexibility and sometimes better performance. |
String Convert
| Author: | Vladimir Batov |
|---|---|
| Review Manager: | Needed |
| Download: | |
| Description: | The library takes the approach of boost::lexical_cast in the area of string-to-type and type-to-string conversions, builds on the past boost::lexical_cast experience and advances that conversion functionality further to additionally provide:
It is an essential tool with applications making extensive use of configuration files or having to process/prepare considerable amounts of data in, say, XML, etc. |
Move
| Author: | Ion Gaztanaga |
|---|---|
| Review Manager: | Needed |
| Download: | http://svn.boost.org/svn/boost/sandbox/move/ and online documentation at http://svn.boost.org/svn/boost/sandbox/move/libs/move/doc/html/index.html |
| Description: | In C++0x, move semantics are implemented with the introduction of rvalue references. They allow us to implement move() without verbosity or runtime overhead. Boost.Move is a library that offers tools to implement those move semantics not only in compilers with rvalue references but also in compilers conforming to C++03. |
Containers
| Author: | Ion Gaztanaga |
|---|---|
| Review Manager: | Needed |
| Download: | http://www.boostpro.com/vault/index.php?action=downloadfile&filename=boost.move.container.zip&directory=Containers& |
| Documentation: | http://svn.boost.org/svn/boost/sandbox/move/libs/container/doc/html/index.html |
| Description: | Boost.Container library implements several well-known containers, including STL containers. The aim of the library is to offers advanced features not present in standard containers or to offer the latest standard draft features for compilers that comply with C++03. |
Interval Containers Library
| Author: | Joachim Faulhaber |
|---|---|
| Download: | |
| Documentation: | http://herold-faulhaber.de/boost_itl/doc/libs/itl/doc/html/index.html |
| Review Manager: | Needed |
| Description: | The Interval Template Library (Itl) provides intervals and two kinds of interval containers: Interval_sets and interval_maps. Interval_sets and maps can be used just as sets or maps of elements. Yet they are much more space and time efficient when the elements occur in contiguous chunks: intervals. This is obviously the case in many problem domains, particularly in fields that deal with problems related to date and time. Interval containers allow for intersection with interval_sets to work with segmentation. For instance you might want to intersect an interval container with a grid of months and then iterate over those months. Finally interval_maps provide aggregation on associated values, if added intervals overlap with intervals that are stored in the interval_map. This feature is called aggregate on overlap. It is shown by example: typedef set<string> guests; interval_map<time, guests> party; guests mary; mary.insert("Mary"); guests harry; harry.insert("Harry"); party += make_pair(interval<time>::rightopen(20:00, 22:00),mary); party += make_pair(interval<time>::rightopen_(21:00, 23:00),harry); // party now contains [20:00, 21:00)->{"Mary"} [21:00, 22:00)->{"Harry","Mary"} //guest sets aggregated on overlap [22:00, 23:00)->{"Harry"} As can be seen from the example an interval_map has both a decompositional behavior (on the time dimension) as well as a accumulative one (on the associated values). |
Type Traits Extensions
| Author: | Frederic Bron |
|---|---|
| Review Manager: | Needed |
| Download: | |
| Description: | The purpose of the addition is to add type traits to detect if types T and U are comparable in the sense of <, <=, >, >=, == or != operators, i.e. if t<u has a sens when t is of type T and u of type U (same for <=, >, >=, ==, !=). The following traits are added: is_equal_to_comparable<T,U> is_greater_comparable<T,U> is_greater_equal_comparable<T,U> is_less_comparable<T,U> is_less_equal_comparable<T,U> is_not_equal_to_comparable<T,U> The names are based on the corresponding names of the standard template library (<functional> header, section 20.3.3 of the standard). The code has the following properties: * returns true if t<u is meaningful and returns a value convertible to bool * returns false if t<u is meaningless. * fails with compile time error if t<u is meaningful and returns void (a possibility to avoid compile time error would be to return true with an operator, trick but this has little sens as returning false would be better) |
InterThreads
| Author: | Vicente J. Botet Escriba |
|---|---|
| Review Manager: | Needed |
| Download: | |
| Description: |
(thread_decorator and thread_specific_shared_ptr) are based on the original implementation of threadalert written by Roland Schwarz. Boost.InterThreads extends Boost.Threads adding thread setup/cleanup decorator, thread specific shared pointer, thread keep alive mechanism and thread tuples. |
Bitfield
| Author: | Vicente J. Botet Escriba |
|---|---|
| Review Manager: | Needed |
| Download: | http://svn.boost.org/svn/boost/sandbox/bitfield with documentation available at http://svn.boost.org/svn/boost/sandbox/bitfield/libs/integer/doc/index.html |
| Description: |
|
Lockfree
| Author: | Tim Blechmann |
|---|---|
| Review Manager: | Needed |
| Download: | |
| Documentation: | |
| Description: | boost.lockfree provides implementations of lock-free data structures. lock-free data structures can be accessed by multiple threads without the necessity of blocking synchronization primitives such as guards. lock-free data structures can be used in real-time systems, where blocking algorithms may lead to high worst-case execution times, to avoid priority inversion, or to increase the scalability for multi-processor machines.
|
Libraries under development
Persistent
| Author: | Tim Blechmann |
|---|---|
| Description: | A library, based on Boost.Serialization, that provides access to persistent objects with an interface as close as possible to accessing regular objects in memory.
Please let us know of any libraries you are currently developing that you intend to submit for review. |
See http://svn.boost.org/trac/boost/wiki/LibrariesUnderConstruction for a current listing of libraries under development.


So I’m safely in my hotel room in San Francisco now - beaten, bruised and so, so tired.
SF seems nice - when I was a kid, I always imagined that when I went to another country, the sky would be a different colour, or the air would taste different - Vietnam held up it’s end of the bargain on the air when I was there this year - but SF seems like Sydney with wider streets and everybody going in the wrong direction. I’m told the water is not so good for me - so I’ll have to hunt down some bottled stuff later on.
I’m too ruined to be excited about WWDC yet, but that should hit tomorrow night at some point (or maybe my excitement gland is broken?) - either way, it appears not everyone is as keen as I am to break their machines with Leopard when (if?) we get the installer discs.
I’d show you pictures of the room and how fantastic I look when I’m tired, but I forgot the cable for the camera. Anyhow, I’m going to go rest before dinner - lots of sightseeing to do tomorrow! If you’re here and want to have a drink, send me an email, or iChat me.


Setup
First get the code from the opas-f-server repo
Docker
the simplest way to set up your Open-Pentesting-And-Security-Framework using docker. First step is to copy the ‘local_settings.template.py’ file to ‘local_settings.py’ and edit it to fit your needs.
docker-compose up
Docker with Postgres database
TODO
Without Docker (Linux)
Database
- install needed software
sudo apt install libpq-dev postgresql postgresql-contrib
- setup database
sudo su - postgres
psql
CREATE DATABASE opasf_db;
CREATE USER opasf_user WITH PASSWORD 'somesecurepassword';
GRANT ALL PRIVILEGES ON DATABASE opasf_db TO opasf_user;
\q
exit
- update ‘local_settings.py’
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'opasf_db',
'USER': 'opasf_user',
'PASSWORD': 'somesecurepassword',
'HOST': 'localhost',
'PORT': '',
}
}
- optionally enable postgresql systemd service
systemctl enable postgresql
Open-Pentesting-And-Security-Framework
- install git, virtualenv (optional)
sudo apt install git virtualenv
- clone code
git clone https://codeberg.org/OPAS-F/opas-f-server.git
- cd into directory
cd opas-f-server
- enter virtual environment
source bin/activate
- install requirements
pip install -r requirements.txt
pip install django[argon2]
- prepare settings
cp local_settings.template.py local_settings.py
- optionally: edit settings to your need
- migrate database
python manage.py migrate
- if you want to enable admin interface
python manage.py collectstatic
- create super user
python manage.py createsuperuser
- start gunicorn
gunicorn opas_f_server.wsgi
- alternativly use systemd for gunicorn
- paste the code below into ‘/etc/systemd/system/gunicorn.service’
systemctl enable gunicorn
systemctl start gunicorn
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=cysec
Group=www-data
WorkingDirectory=/usr/local/src/opas-f-server
ExecStart=/usr/local/bin/gunicorn --timeout 180 --workers 4 --bind <IP_ADDRESS>:8000 opas_f_server.wsgi:application --pythonpath /usr/local/src/opas-f-server
[Install]
WantedBy=multi-user.target
Setup Bughunting auto disclosure
Todo: explain how to set up without docker
If you use docker, you do not need to configure anything here.
Docker Hidden-Service
We created a simple script that allows you to easily create V3 - Hidden-Services that are running the api server. If you did not already create a ‘local_settings.py’ file, you should create one by reading steps above.
For the hidden services to be created run sh run_as_tor_hidden_service_v3.sh
. After they have been created and started, you need to change the ‘ALLOWED_HOSTS’ setting in the ‘local_settings.py’ file to your new hidden service addresses (they end with .onion). The script will output the generated addresses.
Docker Hidden-Service with Postgres database
run docker-compose -f docker-compose-tor-postgres.yml up
. Configure your local_settings.py and volume paths in ‘docker-compose-tor-postgres.yml’ to fit your needs.
Next Steps
Now your server should be up and running, hopefully ;). If you need some help, feel free to ask for help at our matrix room.
(TODO: add room identifier)
You are now using the server with our predefined settings. Some tips of how to change settings see the settings page
Happy Hacking!


Thanks to a grant from the Gordon and Betty Moore Foundation, I’ve enjoyed the Julia Summer of Code 2015 program administered by the NumFOCUS and a travel to the JuliaCon 2015 at Boston. During this program, I have created several packages about data structures and algorithms for sequence analysis, mainly targeted for bioinformatics. Even though Julia had lots of practical packages for numerical computing on floating-point numbers, it lacked efficient and compact data structures that are fundamental in bioinformatics.
Recent development of high-throughput DNA sequencers has enabled to sequence massive numbers of DNA fragments (known as reads) from biological samples within a day. The first step of sequence analysis is locating positions of these fragments in other long reference sequence, then we can detect genetic variants or gene expressions based on the result. This step is called sequence mapping or aligning, and because reference sequences are most commonly genome-scale (about 3.2 billions length for human), a full-text search index is used to speed up this alignment process. This kind of full-text search index is implemented in many bioinformatics tools, most notably bowtie2 and BWA, whose papers are cited thousands of times.
The main focus of my project was creating a full-text search index in Julia that is easy to use and efficient in practical applications. In the course towards this destination, I’ve created several packages that are useful as a building block for other data structures. I’m going to introduce you these packages in this post.
IntArrays.jl is a package for arrays of unsigned integer.
So, is it useful? Yes, it is! This is because the IntArray
type implemented in this package can store integers as small space as possible.
The IntArray
type has a type parameter w
that represents the number of bits required to encode elements in an array.
For example, if each element is an integer between 0 and 3, you only need to use two bits to encode it and w
can be set to 2 or greater.
These 2-bit integers are packed into a buffer and therefore the array consumes only one fourth of the space compared to the usual array.
The following is a case of a byte sequence of [0x01, 0x03, 0x02, 0x00]
:
index: 1 2 3 4 byte sequence (hex): 0x01 0x03 0x02 0x00 byte sequence (bin): 0b00000001 0b00000011 0b00000010 0b00000000 packed sequence (w=2): 01 11 10 00 in-memory layout: 00101101
The full type definition is IntArray{w,T,n}
, where w
is the number of bits
for each element as I explained, T
is the type of elements, and n
is the
dimension of the array. This type is a subtype of the AbstractArray{T,n}
and
will behave like a familiar array; allocation, random access and update are
supported. IntVector
and IntMatrix
are also defined as type aliases like
Vector
and Matrix
, respectively.
Here is an example:
julia> IntArray{2,UInt8}(2, 3)
2x3 IntArrays.IntArray{2,UInt8,2}:
0x00 0x00 0x01
0x00 0x00 0x03
julia> array = IntVector{2,UInt8}(6)
6-element IntArrays.IntArray{2,UInt8,1}:
0x00
0x00
0x03
0x03
0x02
0x00
julia> array[1] = 0x02
0x02
julia> array
6-element IntArrays.IntArray{2,UInt8,1}:
0x02
0x00
0x03
0x03
0x02
0x00
julia> sort!(array)
6-element IntArrays.IntArray{2,UInt8,1}:
0x00
0x00
0x02
0x02
0x03
0x03
And the memory footprint of IntArray
is much smaller:
julia> sizeof(IntVector{2,UInt8}(1_000_000))
250000
julia> sizeof(Vector{UInt8}(1_000_000))
1000000
Since packing and unpacking integers in a buffer require additional operations,
there are overheads in operations and IntArray
is often slower than Array
.
I’ve tried to keep this discrepancy as small as possible, but the IntArray
is
about 4-5 times slower when sorting it:
julia> array = rand(0x00:0x03, 2^24);
julia> sort(array); @time sort(array);
0.488779 seconds (8 allocations: 16.000 MB)
julia> iarray = IntVector{2}(array);
julia> sort(iarray); @time sort(iarray);
2.290878 seconds (18 allocations: 4.001 MB)
If you have a great idea to improve the performance, please let me know!
The next package is IndexableBitVectors.jl.
You must be familiar with the BitVector
type in the standard library; types defined in my package is a static but indexable version of it.
Here “indexable” means that a query to ask the number of bits between an arbitrary range can be answered in constant time.
If you are already familiar with succinct data structures, you may know this is an important building block of other succinct data structures like wavelet trees, LOUDS, etcetera.
The package exports two variants of such bit vectors: SucVector
and RRR
.
SucVector
is simpler and faster than RRR
, but RRR
is compressible and will be smaller if 0/1 bits are localized in a bit vector.
Both types split a bit vector into blocks and cache the number of bits up to the position.
In SucVector
, the extra space is about 1/4 bits per bit, so it will become ~25% larger than the original bit vector.
The most important query operation over these data structures would be the rank1(bv, i)
query, which counts the number of 1 bits within bv[1:i]
. Owing to the cached bit counts, we can finish the rank operation in constant time:
julia> using IndexableBitVectors
julia> bv = bitrand(2^30);
julia> function myrank1(bv, i) # count ones by loop
r = 0
for j in 1:i
r += bv[j]
end
return r
end
myrank1 (generic function with 1 method)
julia> myrank1(bv, 2^29); @time myrank1(bv, 2^29);
0.714866 seconds (6 allocations: 192 bytes)
julia> sbv = SucVector(bv);
julia> rank1(sbv, 2^29); @time rank1(sbv, 2^29); # much faster!
0.000003 seconds (6 allocations: 192 bytes)
julia> rrr = RRR(bv);
julia> rank1(rrr, 2^29); @time rank1(rrr, 2^29); # much faster, too!
0.000004 seconds (6 allocations: 192 bytes)
The select1(bv, j)
query is also useful in many cases, which locates the
j
-th 1 bit in the bit vector bv
. For example, if a set of positive
integers is represented in this bit vector, you can efficiently query the
j
-th smallest member in the set.
Let’s see the internal representation of SucVector
to understand the magic.
A bit vector is separated into large blocks:
type SucVector <: AbstractIndexableBitVector
blocks::Vector{Block}
len::Int
end
Each large block contains 256 bits and consists of four small blocks which contain 64 bits respectively, a large block stores global 1s’ count up to the starting position of it and a small block stores local 1s’ count staring from the beginning position of its parent large block. Bits itself are stored in four bit chunks corresponding to small blocks:
immutable Block
# large block
large::UInt32
# small blocks
# the first small block is used for 8-bit extension of the large block
# hence, 40 (= 32 + 8) bits are available in total
smalls::NTuple{4,UInt8}
# bit chunks (64bits × 4 = 256bits)
chunks::NTuple{4,UInt64}
end
Since the bit count of the first small block is always zero, we can exploit
this space to extend the cache of the large block (red frame). When running
the rank1(bv, i)
query, it first picks a large and small block pair that the
i
-th bit belongs to and then adds their cached bit counts, finally counts
remaining 1 bits in a chunk on the fly.
As I mentioned, this data structure can be used as a building block of various data structures. The next package I’m going to introduce is one of them.
You may already know about the wavelet
tree, which supports the rank
and select queries like SucVector
and RRR
, but elements are not
restricted to 0/1 bits. In fact, the rank and select queries are available
on arbitrary unsigned integers. The wavelet tree can be thought as a
generalization of indexable bit vectors in this respect. What I’ve implemented
is not the well-known wavelet tree, a variant of it called “wavelet matrix”.
You can find an implementation and a link to a paper at
WaveletMatrices.jl.
According to the authors of the paper, the wavelet matrix is “simpler to build,
simpler to query, and faster in practice than the levelwise wavelet tree”.
The WaveletMatrix
type takes three type parameters: w
, T
, and B
. w
and T
are analogous to those of IntArray{w,T,n}
, and B
is a type of
indexable bit vector.
julia> using WaveletMatrices
julia> wm = WaveletMatrix{2}([0x00, 0x01, 0x02, 0x03])
4-element WaveletMatrices.WaveletMatrix{2,UInt8,IndexableBitVectors.SucVector}:
0x00
0x01
0x02
0x03
julia> wm[3]
0x02
julia> rank(0x02, wm, 2)
0
julia> rank(0x02, wm, 3)
1
julia> xs = rand(0x00:0x03, 2^16);
julia> wm = WaveletMatrix{2}(xs); # 2-bit encoding
julia> sum(xs[1:2^15] .== 0x03)
8171
julia> rank(0x03, wm, 2^15)
8171
The details of the data structure and algorithms are relatively simple but beyond the scope of this post. For people who are interested in this data structure, the paper I mentioned above and my implementation would be helpful. There are more operations that the wavelet matrix can run efficiently and those operations will be added in the future.
80% of sequence analysis in bioinformatics is about sequence search, which includes pattern search, homologous gene search, genome comparison, short-read mapping, and so on. The FM-Index is often regarded as one of the most efficient indices for full-text search, and I’ve implemented it in the FMIndexes.jl package. Thanks to the packages I’ve introduced so far, the code of it looks really simple. For example, counting the number of occurrences of a given pattern in a text can be written as follows (slightly simplified for explanatory purpose):
function count(query, index::FMIndex)
sp, ep = 1, length(index)
# backward search
i = length(query)
while sp ≤ ep && i ≥ 1
char = convert(UInt8, query[i])
c = index.count[char+1]
sp = c + rank(char, index.bwt, sp - 1) + 1
ep = c + rank(char, index.bwt, ep)
i -= 1
end
return length(sp:ep)
end
A unique property of the FM-Index is that an index itself is just a permutation of characters of an original text and counts of characters contained in it. This permutation is called Burrows-Wheeler transform (also known as BWT), and the permuted text is stored in a wavelet matrix (or a wavelet tree) in order to efficiently count the number of characters within a specific region. Therefore, the space required to index a text is often smaller than that of other full-text indices (actually, in practice, efficiently finding positions of a query needs auxiliary data as well). Moreover, this transform is bijective, and thus the original text can be restored from an index.
Building an index for full-text search is ridiculously simple: just passing a sequence to a constructor:
julia> using FMIndexes
julia> fmindex = FMIndex("abracadabra");
The FMIndex
type supports two main queries: count
and locate
. The
count(query, index)
query literally counts the number of occurrences of the
query
string and the locate(query, index)
locates starting positions of the
query
. In order to restore the original text, you can use the restore
function. Here is a simple usage:
julia> count("a", fmindex)
5
julia> count("abra", fmindex)
2
julia> locate("a", fmindex) |> collect
5-element Array{Any,1}:
11
8
1
4
6
julia> locate("abra", fmindex) |> collect
2-element Array{Any,1}:
8
1
julia> bytestring(restore(fmindex))
"abracadabra"
As an example, for bioinformaticians, let’s try several queries on a chromosome. You also need to install the Bio.jl package to efficiently parse a FASTA file. The next script reads a chromosome from a FASTA file, build an FM-Index, and then serialize it into a file for later use (I love the serializers of Julia, they are available for free!):
index.jl
using Bio.Seq
using IntArrays
using FMIndexes
# encode a DNA sequence with 3-bit unsigned integers;
# this is because a reference genome has five nucleotides: A/C/G/T/N.
function encode(seq)
encoded = IntVector{3,UInt8}(length(seq))
for i in 1:endof(seq)
encoded[i] = convert(UInt8, seq[i])
end
return encoded
end
# read a chromosome from a FASTA file
filepath = ARGS[1]
record = first(open(filepath, FASTA))
println(record.name, ": ", length(record.seq), "bp")
# build an FM-Index
fmindex = FMIndex(encode(record.seq))
# save it in a file
open(string(filepath, ".index"), "w+") do io
serialize(io, fmindex)
end
OK, then create an index for chromosome 22 of human (you can download it from here):
$ julia4 index.jl chr22.fa
chr22: 50818468bp
$ ls -lh chr22.fa.index
-rw-r--r--+ 1 kenta staff 74M 9 26 06:30 chr22.fa.index
After construction finished (this will take several minutes), read the index in REPL:
julia> using FMIndexes
julia> fmindex = open(deserialize, "chr22.fa.index");
Now that you can execute queries to search a DNA fragment:
julia> using Bio.Seq
julia> count(dna"GACTTTCAC", fmindex) # this DNA fragment hits at 111 locations
111
julia> count(dna"GACTTTCACTTT", fmindex) # this hits at 3 locations
3
julia> locate(dna"GACTTTCACTTT", fmindex) |> collect # the loci of these hits
3-element Array{Any,1}:
36253071
47308573
34159872
julia> count(dna"GACTTTCACTTTCCC", fmindex) # found a unique hit!
1
julia> locate(dna"GACTTTCACTTTCCC", fmindex) |> collect
1-element Array{Any,1}:
36253071
julia> @time locate(dna"GACTTTCACTTTCCC", fmindex); # this can be located in 32 μs!
0.000032 seconds (5 allocations: 192 bytes)
This locus, chr22:36253071, is the starting position of the APOL1 gene.
My aim of having created these packages was to prove that it is practicable to implement high-performance data structures for bioinformatics in Julia. I’m pretty sure that it is true, but it may be skeptical to others. So, I’m going to prove it by writing useful and performant applications using these packages. Now I’m working on FMM.jl, which aligns massive amounts of DNA fragments to a genome sequence using the FM-Index and other algorithms. This is still a work in progress, there would be many bugs and unusual cases I should care about, but its performance is not so bad compared to other implementations.
The BioJulia project is also under active development. The packages I made are intended to work with the Bio.jl package. If you are interested in the BioJulia project, we really welcome your contributions!


Limitations (JavaTM IDL/JavaTM RMI over IIOP) |
This document lists some of the limitations regarding JavaTM IDL and JavaTM RMI over IIOP technology in this release of the JavaTM 2 Platform Standard Edition (J2SETM). The limitations are outlined below.
The Official Specifications for CORBA Support in J2SE 5.0 document provides information outlining with which OMG Specifications this implementation of the Java IDL and Java RMI over IIOP technologies comply.
LIMITATIONS:
When a J2SETM 1.3.x client talks to a server written in J2SE 1.4 or higher using RMI-IIOP and an unchecked or RemoteException is thrown by the server, the client will hang. Similarly, when the server is using J2SE 1.3.x and the client is using J2SE 1.4 or higher, the client will receive a CORBA UNKNOWN exception with minor code 0.This issue is currently being addressed at the OMG in the Java to IDL RTF as issue 4795: http://www.omg.org/issues/java2idl-rtf.html#Issue4795.
In J2SE 1.4.1 and higher: If one side is using J2SE <IP_ADDRESS>, and the other side is using J2SE 1.4.1 or higher on the other side, this type of communication will work properly.
If an IOR has multiple profiles, the first IIOP profile is used for the invocation.
This release of J2SE platform does not allow multiple ORB instantiations in persistent servers. If the persistent server is written using the Java RMI over IIOP APIs, instantiating JNDI's InitialContext() creates an ORB. To avoid creating multiple ORBs in this setting, you can set the java.naming.ORB system property to an ORB that is already initialized.
Locally-optimized stubs, those with the local optimization generated by a different IDL-to-Java compiler than idlj, may be used with the J2SE 1.4 or higher ORB. However, Portable Interceptors will not be invoked when taking the locally-optimized stub branch. This is due to an issue with the OMG IDL to Java language mapping and is being addressed under OMG issue 4701.The J2SE 1.4 or higher rmic compiler generates locally optimized stubs when used in conjunction with the -iiop option. However, the J2SE 1.4 and higher implementation of javax.rmi.CORBA.Util always returns false for isLocal. The result is that the stubs never use the local optimization. This is due to an issue with the OMG Java to IDL language mapping and is addressed under OMG issue 3754.
The J2SE 1.4 and higher ORB does use a locally-optimized transport for both Java IDL and Java RMI-IIOP when it detects that the client and server are running sharing the same ORB.
As per the CORBA 2.3.1 specification, there can be two choices for Thread Policy for the Portable Object Adapter (POA): The ORB Controlled Model and the Single Thread Model. J2SE 5.0 does not support the Single Thread Model.
If the super-type of a class has evolved to become custom-marshalled and writes optional, custom data, then that class cannot be sent between J2SE 1.3.x and J2SE 1.4 or higher. One example of this is that java.util.Calendar in J2SE 1.4 or higher cannot be sent back to J2SE 1.3.x.


The most common problem in everyday work is time management and organization. Gathering all the information and peaces from your client, colleges, employers, you name it. That can be quite a productivity killer. We would like to know how much of the problem that is to you and how you are dealing with it.
we promise it will not take more then 5 min of your time. We know that time is valuable.


Production Services
Datachecks
Datacheck Catalog
Run Datachecks
Production Services
Roll up, roll up, we've got all the luverly
datachecks
you'll ever need!


We have taken the main city of FURLIN (known for it's rich history of fur trading). Our troops are hunting the enemy high command. We know that they are hiding in a BUNKER in a remote part of the city.
We have successfully reproduced the MARK IV machine, however we do not know what today's wheel settings are. Therefore, we need to use our machine to crack though an encoded message we received today to find the wheel positions.
JSON QUESTION... The encoded message is in the code attribute of the JSON data. The message contains clue words found in the above text. Once decrypted, what is the decoded message and what are the wheel positions? (extra points are given for finding all the possible wheel positions that will successfully decode this message).
Email<EMAIL_ADDRESS>with your answer, along with public access to your code repository. All submissions will be dated by email receipt. We will contact all entrants with details of the winner. We will also publish the winner on this site and the Eden website.
Thanks for playing the Eden Minisculus Challenge!


{% extends "admin_layout.html" %} {% block nav %} Dashboard Expenses Income Loans Accounts Users Loans Give Loan Get Loan {% endblock %} {% block header %}
− £{{ loan.0.amount|abs|currencyformat }} {{ loan.0.description }}
£{{ loan.0.amount|currencyformat }} {{ loan.0.description }}


jquery UI
download it from :
click Here to download
Transform
Sample
Date Picker Example
Progress bar Example
Slider :
slider


TR(1) User Commands TR(1) NAME tr - translate or delete characters SYNOPSIS tr [OPTION]... SET1 [SET2] DESCRIPTION Translate, squeeze, and/or delete characters from standard input, writing to standard output. -c, -C, --complement use the complement of SET1 -d, --delete delete characters in SET1, do not translate -s, --squeeze-repeats replace each input sequence of a repeated character that is listed in SET1 with a single occurrence of that character -t, --truncate-set1 first truncate SET1 to length of SET2 --help display this help and exit --version output version information and exit SETs are specified as strings of characters. Most represent themselves. Interpreted sequences are: \NNN character with octal value NNN (1 to 3 octal digits) \\ backslash \a audible BEL \b backspace \f form feed \n new line \r return \t horizontal tab \v vertical tab CHAR1-CHAR2 all characters from CHAR1 to CHAR2 in ascending order [CHAR*] in SET2, copies of CHAR until length of SET1 [CHAR*REPEAT] REPEAT copies of CHAR, REPEAT octal if starting with 0 [:alnum:] all letters and digits [:alpha:] all letters [:blank:] all horizontal whitespace [:cntrl:] all control characters [:digit:] all digits [:graph:] all printable characters, not including space [:lower:] all lower case letters [:print:] all printable characters, including space [:punct:] all punctuation characters [:space:] all horizontal or vertical whitespace [:upper:] all upper case letters [:xdigit:] all hexadecimal digits [=CHAR=] all characters which are equivalent to CHAR Translation occurs if -d is not given and both SET1 and SET2 appear. -t may be used only when translating. SET2 is extended to length of SET1 by repeating its last character as necessary. Excess characters of SET2 are ignored. Only [:lower:] and [:upper:] are guaranteed to expand in ascending order; used in SET2 while translating, they may only be used in pairs to specify case conversion. -s uses SET1 if not translating nor deleting; else squeezing uses SET2 and occurs after translation or dele‐ tion. AUTHOR Written by Jim Meyering. REPORTING BUGS Report tr bugs to<EMAIL_ADDRESS>GNU coreutils home page: <http://www.gnu.org/software/coreutils/> General help using GNU software: <http://www.gnu.org/gethelp/> Report tr translation bugs to <http://translationproject.org/team/> COPYRIGHT Copyright © 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. SEE ALSO The full documentation for tr is maintained as a Texinfo manual. If the info and tr programs are properly installed at your site, the command info coreutils 'tr invocation' should give you access to the complete manual. GNU coreutils 8.12.197-032bb September 2011 TR(1)


STAGE 2: Now you are the victim Jane who received the email below. If you point on the link with your mouse you will see that there is a SID included. Click on it to see what happens.
Mail From : admin@webgoatfinancial.com


Filter Example
In most cases, tag is a simple and useful design to select message you want. For example:
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID_EXAMPLE");
consumer.subscribe("TOPIC", "TAGA || TAGB || TAGC");
The consumer will recieve messages that contains TAGA or TAGB or TAGC. But the limitation is that one message only can have one tag, and this may not work for sophisticated scenarios. In this case, you can use SQL expression to filter out messages.
Principle
SQL feature could do some calculation through the properties you put in when sending messages. Under the grammars defined by RocketMQ, you can implement some interesting logic. Here is an example:
------------ | message | |----------| a > 5 AND b = 'abc' | a = 10 | --------------------> Gotten | b = 'abc'| | c = true | ------------ ------------ | message | |----------| a > 5 AND b = 'abc' | a = 1 | --------------------> Missed | b = 'abc'| | c = true | ------------
Grammars
RocketMQ only defines some basic grammars to support this feature. You could also extend it easily.
- Numeric comparison, like
>
,>=
,<
,<=
,BETWEEN
,=
; - Character comparison, like
=
,<>
,IN
; IS NULL
orIS NOT NULL
;- Logical
AND
,OR
,NOT
;
Constant types are:
- Numeric, like 123, 3.1415;
- Character, like ‘abc’, must be made with single quotes;
NULL
, special constant;- Boolean,
TRUE
orFALSE
;
Usage constraints
Only push consumer could select messages by SQL92. The interface is:
public void subscribe(final String topic, final MessageSelector messageSelector)
Producer example
You can put properties in message through method putUserProperty
when sending.
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
producer.start();
Message msg = new Message("TopicTest",
tag,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET)
);
// Set some properties.
msg.putUserProperty("a", String.valueOf(i));
SendResult sendResult = producer.send(msg);
producer.shutdown();
Consumer example
Use MessageSelector.bySql
to select messages through SQL92 when consuming.
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_4");
// only subsribe messages have property a, also a >=0 and a <= 3
consumer.subscribe("TopicTest", MessageSelector.bySql("a between 0 and 3");
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
Leave a Comment


BVNDIT drop sultry red 'Dumb' MV!
by germainejBVNDIT have dropped their sultry red music video for "Dumb"!
"Dumb" is an upbeat Latin pop song about sending messages to someone who doesn't seem to get the signals, and it's the title track of BVNDIT's first mini album 'BE!'. The MV features the BVNDIT members against a red backdrop and sheer curtains.
Watch BVNDIT's "Dumb" MV above, and let us know what you think in the comments below.


Home/Login
About CC Creators
Edit Your Profile
See Your Class Cards
>>Currently on Class Cards Results page
Thank you for visting Class Cards by
Awesome 201
Copyright 2018


[gecode-users] Constraint Hierachy
Christian Schulte
cschulte at kth.se
Mon Feb 2 10:14:51 CET 2009
No, it does not. You need to be more detailled of what you want to do.
Christian
--
Christian Schulte, www.ict.kth.se/~cschulte/
From: users-bounces at gecode.org [mailto:users-bounces at gecode.org] On Behalf Of Jeffrey
Sent: Sunday, February 01, 2009 10:21 AM
To: gecode at gecode.org
Subject: [gecode-users] Constraint Hierachy
Does Gecode support constraint hierarchy?
If not, how can I implemented one on Gecode?
_____
Yahoo!香港提供網上安全攻略，教你如何防範黑客! <http://hk.promo.yahoo.com/security/> 了解更多
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ps.uni-sb.de/pipermail/users/attachments/20090202/0474796c/attachment.htm>
More information about the gecode-users
mailing list


La Bamba (1987)
Grade: 73
"La Bamba" is a surprisingly good biography of ill-fated Mexican-American rock and roll star Ritchie Valens (Lou Diamond Phillips). While sometimes overdramatized, the film provides a good character study, not only of Valens, but of his troubled "black sheep" half-brother Bob (Esai Morales).
The story begins with Valens already a teenager, working as a day laborer with his strong willed but loving mother (Rosana De Soto) and family friend (and telegenic beauty) Rosie (Elizabeth Pena). Thanks partly to Bob's success as a drug dealer, Valens is able to attend a nice high school, where he meets WASP blonde Donna (Danielle von Zerneck).
He also joins a Rock and Roll band, with his energy and talent attracting the notice of Del-Fi talent scout Joe Pantoliano. Soon, Valens has a nationwide hit single and is touring the country. But while his career is soaring, his half-brother, now married to Rosie, becomes violently embittered over his success.
The most fun for me when watching this film was the casting of contemporary rock artists as their more famous 1950s counterparts. Marshall Crenshaw, who had always been compared to Buddy Holly, finally gets to play Buddy Holly. Stray Cats' guitarist Brian Setzer is an ideal match for rockabilly singer Eddie Cochran. The group Los Lobos dubs Phillip's impersonation of Valens. Carlos Santana performs the musical score. Valens' real life mother and half-brother have cameos. By the way, the real Ritchie Valens can be found in the fifties B-film "Go, Johnny, Go".
"La Bamba" is not a perfect film. Too much is made of Valens not speaking Spanish. He even has the line "Yo no speako Espa~ol", which strains credibility given the fact that everyone else in his family seems to speak Spanish fluently. A more obvious weakness is Morales' bad boy act, which recurs even more often than Ritchie's prophetic nightmare of two planes colliding. Also, the audience is led to believe that the tragic plane crash that killed Holly, Valens, and novelty singer Big Bopper was caused by the breaking of Valens' talisman necklace.
<EMAIL_ADDRESS>http://members.tripod.com/~Brian_Koller/movis.html
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


# File lib/chronic/season.rb, line 12 def self.find_next_season(season, pointer) lookup = [:spring, :summer, :autumn, :winter] next_season_num = (lookup.index(season) + 1 * pointer) % 4 lookup[next_season_num] end
# File lib/chronic/season.rb, line 7 def initialize(start_date, end_date) @start = start_date @end = end_date end
# File lib/chronic/season.rb, line 18 def self.season_after(season) find_next_season(season, +1) end
# File lib/chronic/season.rb, line 22 def self.season_before(season) find_next_season(season, -1) end


It is a three year long program and has been specifically designed for aspirants looking for a career in computers. The course covers all aspects of computers right from the basic fundamentals of computers to database systems & advanced courses like C++, Java etc.
BSc Computer Science Eligibility To be eligible for this program, you must have completed 10+2 with the minimum 45%- 60% marks. This requirement may vary from college to college.
Career prospects:
On the successful completion of the program, one can start working as: Systems analyst, Information technology manager, Software programmer, Network programmer, Cyber Security Manager, Computer Graphics, Database manager
List of BSc Computer Science Subjects & Syllabus
Fundamentals of Computer Organization & Embedded Systems, Introduction to Database Management Systems, Introduction to programming using Python, Advanced Programming using Python, Discrete Mathematics, Computer Graphics, C++ Programming, Java Programming, Database Management Systems, Software Engineering


This write up explains how i solved the challenge proposed by CakeCTF2022. The challenge was to get the flag in a binary made with the Nim (https://nim-lang.org/) program language.
tl;dr
If you want to know the answer right away, just put a break point in the sym.join_main_42 function, and check the registers and your flag will be there.
The challenge itself was not complex however, I had never written anything in Nim. So right after downloading the challenge binary, I went to look at some of Nim’s documentation and created a simple example, a guess number:
import strutils
echo "Whats the number: "
let guess = parseInt(stdin.readline)
let number = 1337
if guess < number:
echo "guess to low"
elif guess > number:
echo "guess to high"
else:
echo "WoW!"
The Nim language at Syntax level looks like Python but you can generate a binary in C. And with the same code you generate JavaScript, very cool.
Looking at the code made by me, and put on Radare, I could see a few things:
1 - Hard coded strings have the @ in front; 2 - There is a main, and a NimMain functions 3 - Other functions named Nim
4 - NimMainInner sometime is Key to find the application logic.
The following images illustrate how my code generated, inside Radare and the strings that are in my code like Whats the number:
The command r2 -AA -d will Analyze the binary and debug
The command iz will dump all strings
The next step is to put a breakpoint in main and check what it is calling:
The command db (function name) will set a break point radare2
The command dc will run the program
The command pdf will print the “disassemble” of main funcion
As can be seen in the previous image, the main function calls another function named sym.NimMain.
When analyzing what has in sym.NimMan, I could see that the calls usually go through an instruction and call the sym.NimMainInner function, and after that call the sym.NimMainModule
The next step was essential to solve the CTF, I realized that all Nim function calls are inside the sym.NimMainInner call. So the functions I used in my example are called here:
Solving the nimrev Challenger
When downloading the binary and running it straight away, we noticed that the sym.readLine function is called, as it waits for user input, as I did in the example. However the string Wrong.. is displayed. And we know that to search for these strings we need the char @:
We know that, we have to look for the sym.NimMainModule to check the functions used and thus try to understand some of the logic.
Looking at the output of the sym.NimMainInner call, we notice that after reading our input with sym.readLine several other functions are called and after calling the newSeq function (https://nim-lang.org/docs/system.html#newSeq)
several mov byte operations are performed and a function called eqStrings is called before executing the function which will show Wrong or Correct ( echoBinSafe )
But before calling the string comparison, a function is called first ( sym.join_main_42 ) and is probably responsible for the check, let’s check.
When analyzing each function present:
sym.resizeString sym.appendString_2 sym.nimCharToStr
When analyzing the nimCharToStr function it is possible to observe that a “new” string is dynamically created, for comparison with our input, before calling the eqStrings function. Therefore, we must check the values of the registers before the comparison and so we get our flag ;)
The drr command shows all the registers em try to “long to bytes” for you ;)
To check the veracity of the flag, we run the binary again and pass the flag
That’s it for today, Nim is a very cool language and I learned a lot from this challenge. It’s basic but maybe someone hasn’t done it using radare and this can be a mini guide for those who want to understand more about Radare ( https://rada.re /n/ ) and the Nim Language ( https://nim-lang.org/ ).
Thanks O/


Chibuzor Anthony Obi, also known as 'Tonee C' is a rap artist currently studying
Mechanical Engineering at the univerity of Portharcourt.He is known for his versatility in the rap,hip hop and afro-trap genre of music
and also for his lyrical prowess.Also known as 'Lyrical gentleman' which was coined from a song on his mix-tape titled 'Lyrical Gentleman'
Tonee C is one of portharcourt's finest upcoming rappers.


Deed-Coins:
Badges:
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer posuere erat.
This card has supporting text below as a natural lead-in to additional content.


Chapter 6 Feature attributes
Feature attributes refer to the properties of features (“things”) that do not describe the feature’s geometry. Feature attributes can be derived from geometry (e.g. length of a LINESTRING
, area of a POLYGON
) but they can also refer to completely different properties, such as
- the name of a street or a county,
- the number of people living in a country,
- the type of a road
- the soil type in a polygon from a soil map.
- the opening hours of a shop
- the body weight of an animal
- the NO\(_2\) concentration measured at an air quality monitoring station
Although temporal properties of features are no less fundamental than their spatial properties, the simple feature access standard and consequently the sf
package does not give time a similar role as space; more on that in chapter 4.
Most sf
objects will contain both geometries and attributes for features. The geometric operations described in the previous chapter (5) operate on geometries only, and may occasionally add attributes, but will not modify attributes present.
In all these cases, attribute values remain unmodified. At first sight, that looks rather harmless. But if we look into a simple case of replacing a county boundary with a county centroid, as in
library(sf)
library(dplyr)
system.file("gpkg/nc.gpkg", package="sf") %>%
read_sf() %>%
st_transform(32119) %>%
select(BIR74, SID74, NAME) %>%
st_centroid() %>%
head(n = 1) -> x # save as x
#> Warning in st_centroid.sf(.): st_centroid assumes attributes are constant over
#> geometries of x
st_geometry(x)[[1]]
#> POINT (385605 3e+05)
we receive a warning. This warning is justified for the first two variables shown (total births and number of SID disease cases, 1974) which, as such, are not associated with a feature whose geometry is POINT (385605.4 300303.5)
. The third variable, NAME
is however still the county name for the point indicated, but the point geometry no longer is the county geometry.
6.1 Attribute-geometry relationships
Changing the feature geometry without changing the feature attributes does change the feature, since the feature is characterised by the combination of geometry and attributes. Can we, ahead of time, predict whether the resulting feature will still meaningfully relate to the attribute data when we replace all geometries for instance with their convex hull or centroid? It depends.
Take the example of a road, represented by a LINESTRING
, which has an attribute property road width equal to 10 m. What can we say about the road width of an arbitray subsectin of this road? That depends on whether the attribute road length describes, for instance the road width everywhere, meaning that road width is constant along the road, or whether it describes an aggregate property, such as minimum or average road width. In case of the minimum, for an arbitrary subsection of the road one could still argue that the minimum road width must be at least as large as the minimum road width for the whole segment, but it may no longer be the minimum for that subsection. This gives us two “types” for the attribute-geometry relationship (AGR):
- constant the attribute value is valid everywhere in or over the geometry
- aggregate the attribute is an aggregate, a summary value over the geometry
For polygon data, typical examples of constant AGR are
- land use for a land use polygon
- rock units or geologic strata in a geological map
- soil type in a soil map
- elevation class in a elevation map that shows elevation as classes
- climate zone in a climate zone map
Typical examples for the aggregate AGR are
- population, either as number of persons or as population density
- other socio-economic variables, summarised by area
- total emission of pollutants by region
- block mean NO\(_2\) concentrations, as e.g. obtained by block kriging or a dispersion model that predicts areal means
A third type of AGR is that where an attribute identifies a feature geometry. The example above is county NAME
: the name identifies the county, and is still the county NAME
for any sub-area.
- identity the attribute value uniquely identifies the geometry as a whole, there are no other geometries with the same value
Arbitrary sub-areas will lose the identity property but becomes a constant attribute. An example is:
- any point inside a county is still part of the county and must have the same value for county name, but it does not longer represent the (entire) geometry corresponding to that county.
We can specify the AGR of an attribute in an sf
object by st_set_agr
:
nc <- system.file("gpkg/nc.gpkg", package="sf") %>%
read_sf() %>%
st_transform(32119)
nc1 <- nc %>% select(BIR74, SID74, NAME) %>%
st_set_agr(c(BIR74 = "aggregate", SID74 = "aggregate", NAME = "identity"))
This helps to get rid of warnings that a particular attribute is assumed to be constant over a geometry, if it already is. The following no longer generates a warning
nc1 %>% select(NAME) %>%
st_centroid() %>%
head(1)
#> Simple feature collection with 1 feature and 1 field
#> Attribute-geometry relationship: 1 constant, 0 aggregate, 0 identity
#> geometry type: POINT
#> dimension: XY
#> bbox: xmin: 386000 ymin: 3e+05 xmax: 386000 ymax: 3e+05
#> projected CRS: NAD83 / North Carolina
#> # A tibble: 1 x 2
#> NAME geom
#> <chr> <POINT [m]>
#> 1 Ashe (385605 3e+05)
and also changes AGR for NAME
from identity
to constant
when replacing the geometry with the geometry’s centroid:
nc1 %>% select(NAME) %>%
st_centroid() %>%
st_agr()
#> NAME
#> constant
#> Levels: constant aggregate identity
Identifying attribute-geometry relationships, and warning against their absence is a first and simple implementation of the notion that the types of phenomena we encounter in spatial data science (like objects, fields, and aggregations) are not identified by their geometrical representations (points, lines, polygons, rasters). Making the wrong assumptions here easily leads to meaningless analysis results (Stasch et al. 2014,Scheider et al. (2016)).
6.2 Spatial join
Spatial joins are similar to regular (left or inner) joins, where the join criterion is not equality of one or more fields, but a spatial predicate, such as that two records have intersecting geometries. As an example, we can create a join between two tables,
a = st_sf(a = 1:2, geom = st_sfc(st_point(c(0,0)), st_point(c(1,1))))
b = st_sf(b = 3:4, geom = st_sfc(st_linestring(rbind(c(2,0), c(0,2))), st_point(c(1,1))))
st_join(a, b)
#> Simple feature collection with 3 features and 2 fields
#> geometry type: POINT
#> dimension: XY
#> bbox: xmin: 0 ymin: 0 xmax: 1 ymax: 1
#> CRS: NA
#> a b geom
#> 1 1 NA POINT (0 0)
#> 2 2 3 POINT (1 1)
#> 2.1 2 4 POINT (1 1)
st_join(a, b, left = FALSE)
#> Simple feature collection with 2 features and 2 fields
#> geometry type: POINT
#> dimension: XY
#> bbox: xmin: 1 ymin: 1 xmax: 1 ymax: 1
#> CRS: NA
#> a b geom
#> 2 2 3 POINT (1 1)
#> 2.1 2 4 POINT (1 1)
st_join(b, a)
#> Simple feature collection with 2 features and 2 fields
#> geometry type: GEOMETRY
#> dimension: XY
#> bbox: xmin: 0 ymin: 0 xmax: 2 ymax: 2
#> CRS: NA
#> b a geom
#> 1 3 2 LINESTRING (2 0, 0 2)
#> 2 4 2 POINT (1 1)
We see that unless left = FALSE
, we get all elements (and geometries) from the first argument, augmented with fields of the second argument when geometries match. The example shows the case where there are two geometries matching to point (1,1). The spatial join predicate function can be freely chosen, e.g. from the binary predicates listed in section 5.1.2.
When we match two sets of polygons, it may be a bit of a mess to go through all the many matches. One way out of this is to only provide the match with the largest overlap with the target geometry, obtained by adding argument largest = TRUE
. An example of this is shown (visually) in figure 6.1.
6.3 Aggregate and summarise
Package sf
provides sf
methods for stats::aggregate
and dplyr::summarise
. Both do essentially the same:
- given a grouping predicate (for
summarise
, obtained fromgroup_by
) - given an aggregation function
- aggregate the selected attributes using this function, per group
- aggregate in addition the geometries.
- if
do_union
isTRUE
(the default), union the aggregated geometries.
Unioning aggregated geometries dissolves for instance internal polygon boundaries, which otherwise would lead to invalid MULTIPOLYGON
errors in subsequent analysis, or plotting of potentially unwanted internal polygon boundaries. Figure 6.2 illustrates this.
6.4 Intersections
Suppose we have two datasets with different geometries and attributes (left figure 6.3), and we want to compute their intersections:
p1 = st_polygon(list(rbind(c(0,0), c(4,0), c(4,4), c(0,4), c(0,0))))
d1 = st_sf(a = c(3,1), geom = st_sfc(p1, p1 + c(4, 0)))
d2 = st_sf(b = c(4), geom = st_sfc(p1 * .75 + c(3, 2)))
What will the intersection of these two objects give?
(i = st_intersection(d1, d2))
#> Warning: attribute variables are assumed to be spatially constant throughout all
#> geometries
#> Simple feature collection with 2 features and 2 fields
#> geometry type: POLYGON
#> dimension: XY
#> bbox: xmin: 3 ymin: 2 xmax: 6 ymax: 4
#> CRS: NA
#> a b geom
#> 1 3 4 POLYGON ((3 4, 4 4, 4 2, 3 ...
#> 2 1 4 POLYGON ((4 2, 4 4, 6 4, 6 ...
plot(d1, xlim = c(0,8), ylim = c(0, 6), col = NA, border = 1, reset = FALSE)
plot(d2, col = NA, border = 'red', add = TRUE, lwd = 2)
plot(d1, xlim = c(0,8), ylim = c(0, 6), col = NA, border = 1, lwd = 2, reset = FALSE)
plot(d2, col = NA, border = 'red', add = TRUE, lwd = 3)
plot(st_geometry(i), add = TRUE, col = grey(c(.7,.9)), , border = 'green', lwd = 1)
As we see, this gives the areas of intersection, along with the corresponding attributes for both contributing objects, and a warning that attributes were assumed to be spatially constant. Although this may be convenient in some cases, it may be entirely meaningless in others. For instance, in case attribute b
in object d2
represents the number of people living in d2
, then after the intersection we end up with twice as many people living over a smaller area.
As seen in section 5.5, computing intersections easily leads to errors caused by invalid geometries. Setting precision (section 5.4) may prevent this.
6.5 Area-weighted interpolation
Suppose we want to combine geometries and attributes of two datasets such that we get attribute values of the first dataset summarised for the geometries of the second. There are various ways we can go for this. The simplest one, building on the previous example, would be to obtain for the geometry of d2
the attribute of d1
that has the largest overlap with d2
. This is obtained by
st_join(d2, d1, largest = TRUE)
#> Warning: attribute variables are assumed to be spatially constant throughout all
#> geometries
#> Simple feature collection with 1 feature and 2 fields
#> geometry type: POLYGON
#> dimension: XY
#> bbox: xmin: 3 ymin: 2 xmax: 6 ymax: 5
#> CRS: NA
#> b a geom
#> 1 4 1 POLYGON ((3 2, 6 2, 6 5, 3 ...
Another option would be to summarise the attribute, e.g. taking its mean, regardless the amount of overlap. This is obtained by
aggregate(d1, d2, mean)
#> Simple feature collection with 1 feature and 1 field
#> geometry type: POLYGON
#> dimension: XY
#> bbox: xmin: 3 ymin: 2 xmax: 6 ymax: 5
#> CRS: NA
#> a geometry
#> 1 2 POLYGON ((3 2, 6 2, 6 5, 3 ...
A third option is to apply area-weighted interpolation, meaning that we interpolate (average) the variable by taking into account the respective area contributions of overlap (Goodchild and Lam 1980,Do, Thomas-Agnan, and Vanhems (2015a),Do, Thomas-Agnan, and Vanhems (2015b)). This is done e.g. by
d3 = st_sfc(p1 * .75 + c(3, 2), p1 * .75 + c(3,3))
st_interpolate_aw(d1, d3, extensive = FALSE)$a
#> Warning in st_interpolate_aw.sf(d1, d3, extensive = FALSE): st_interpolate_aw
#> assumes attributes are constant over areas of x
#> [1] 1.67 1.67
st_interpolate_aw(d1, d3, extensive = TRUE)$a
#> Warning in st_interpolate_aw.sf(d1, d3, extensive = TRUE): st_interpolate_aw
#> assumes attributes are constant over areas of x
#> [1] 0.625 0.312
6.5.1 Spatially intensive and extensive variables
The difference between the two examples for area-weighted interpolation is how the final weighted sum (value times area of intersection) is normalised: by the target area (extensive), or by the sum of the area covered (intensive, extensive = FALSE
). Spatially intensive variables are variables for which the value, when we split an area, does not principally change. An example might be temperature, elevation, or population density. Spatially extensive variables are variables for which the value is also split, according to the area. Examples are population (amount), or area.
6.6 Exercises
- Add a variable to the
nc
dataset bync$State = "North Carolina"
. Which value should you attach to this variable for the attribute-geometry relationship (agr)? - Create a new
sf
object from the geometry obtained byst_union(nc)
, and assign"North Carolina"
to the variableState
. Whichagr
can you now assign to this attribute variable? - Use
st_area
to add a variable with namearea
tonc
. Compare thearea
andAREA
variables in thenc
dataset. What are the units ofAREA
? Are the two linearly related? If there are discrepancies, what could be the cause? - Is the
area
variable intensive or extensive? Is its agr equal toconstant
,identity
oraggregate
? - Find the name of the county that contains
POINT(-78.34046 35.017)
- Find the names of all counties with boundaries that touch county
Sampson
. - List the names of all counties that are less than 50 km away from county
Sampson
.
References
Do, Van Huyen, Christine Thomas-Agnan, and Anne Vanhems. 2015a. “Accuracy of Areal Interpolation Methods for Count Data.” Spatial Statistics 14: 412–38. doi:10.1016/j.spasta.2015.07.005.
Do, Van Huyen, Christine Thomas-Agnan, and Anne Vanhems. 2015b. “Spatial Reallocation of Areal Data: A Review.” Rev. Econ. Rég. Urbaine 1/2: 27–58. https://www.tse-fr.eu/sites/default/files/medias/doc/wp/mad/wp_tse_397_v2.pdf.
Goodchild, Michael F, and Nina Siu Ngan Lam. 1980. Areal Interpolation: A Variant of the Traditional Spatial Problem. Department of Geography, University of Western Ontario London, ON, Canada.
Scheider, Simon, Benedikt Gräler, Edzer Pebesma, and Christoph Stasch. 2016. “Modeling Spatiotemporal Information Generation.” International Journal of Geographical Information Science 30 (10). Taylor & Francis: 1980–2008. doi:10.1080/13658816.2016.1151520.
Stasch, Christoph, Simon Scheider, Edzer Pebesma, and Werner Kuhn. 2014. “Meaningful Spatial Prediction and Aggregation.” Environmental Modelling & Software 51: 149–65. doi:10.1016/j.envsoft.2013.09.006.


ng-href navigation
- This type of navigation uses a dynamically created href attribute in combination with AngularJs's routing.