Sometime tweets come with geographic coordinates attached as metadata.
Showing this position on a map along with the language used to write the
tweet, give us the opportunity to build interesting visualizations depicting
the languages of this world.
birdwtch
geotagged tweets on a map,
language color-coded


June 2016 - August 2016
IVY
Internship building tests for AngularJS site
New York City
November 2015 - March 2016
General Assembly
Web Development Immersive
New York City
JavaScript / jQuery
Ruby on Rails / Git / AJAX
HTML5 / CSS3 / SQL / Bootstrap
Since graduating I've been keeping busy spending lots of
time working to learn AngularJS, React.js and Node.js better.


pg_basebackup - take a base backup of a PostgreSQL cluster
pg_basebackup is used to take base backups of a running PostgreSQL database cluster. These are taken without affecting other clients to the database, and can be used both for point-in-time recovery (see Section 24.3, (lqContinuous Archiving and Point-in-Time Recovery (PITR)(rq, in the documentation) and as the starting point for a log shipping or streaming replication standby servers (see Section 25.2, (lqLog-Shipping Standby Servers(rq, in the documentation).
pg_basebackup makes a binary copy of the database cluster files, while making sure the system is put in and out of backup mode automatically. Backups are always taken of the entire database cluster; it is not possible to back up individual databases or database objects. For individual database backups, a tool such as pg_dump(1) must be used.
The backup is made over a regular PostgreSQL connection, and uses the replication protocol. The connection must be made with a superuser or a user having REPLICATION permissions (see Section 20.2, (lqRole Attributes(rq, in the documentation), and pg_hba.conf must explicitly permit the replication connection. The server must also be configured with max_wal_senders set high enough to leave at least one session available for the backup.
There can be multiple pg_basebackups running at the same time, but it is better from a performance point of view to take only one backup, and copy the result.
pg_basebackup can make a base backup from not only the master but also the standby. To take a backup from the standby, set up the standby so that it can accept replication connections (that is, set max_wal_senders and hot_standby, and configure host-based authentication). You will also need to enable full_page_writes on the master.
Note that there are some limitations in an online backup from the standby:
h'-04'(buh'+03'c
The backup history file is not created in the database cluster backed up.
h'-04'(buh'+03'c
There is no guarantee that all WAL files required for the backup are archived at the end of backup. If you are planning to use the backup for an archive recovery and want to ensure that all required files are available at that moment, you need to include them into the backup by using -x option.
h'-04'(buh'+03'c
If the standby is promoted to the master during online backup, the backup fails.
h'-04'(buh'+03'c
All WAL records required for the backup must contain sufficient full-page writes, which requires you to enable full_page_writes on the master and not to use a tool like pg_compresslog as archive_command to remove full-page writes from WAL files.
The following command-line options control the location and format of the output.
-D directory
--pgdata=directory
Directory to write the output to. pg_basebackup will create the directory and any parent directories if necessary. The directory may already exist, but it is an error if the directory already exists and is not empty.
When the backup is in tar mode, and the directory is specified as - (dash), the tar file will be written to stdout.
This option is required.
-F format
--format=format
Selects the format for the output. format can be one of the following:
p
plain
Write the output as plain files, with the same layout as the current data directory and tablespaces. When the cluster has no additional tablespaces, the whole database will be placed in the target directory. If the cluster contains additional tablespaces, the main data directory will be placed in the target directory, but all other tablespaces will be placed in the same absolute path as they have on the server.
This is the default format.
t
tar
Write the output as tar files in the target directory. The main data directory will be written to a file named base.tar, and all other tablespaces will be named after the tablespace OID.
If the value - (dash) is specified as target directory, the tar contents will be written to standard output, suitable for piping to for example gzip. This is only possible if the cluster has no additional tablespaces.
-r rate
--max-rate=rate
The maximum transfer rate of data transferred from the server. Values are in kilobytes per second. Use a suffix of M to indicate megabytes per second. A suffix of k is also accepted, and has no effect. Valid values are between 32 kilobytes per second and 1024 megabytes per second.
The purpose is to limit the impact of pg_basebackup on the running server.
This option always affects transfer of the data directory. Transfer of WAL files is only affected if the collection method is fetch.
-R
--write-recovery-conf
Write a minimal recovery.conf in the output directory (or into the base archive file when using tar format) to ease setting up a standby server.
-T olddir=newdir
--tablespace-mapping=olddir=newdir
Relocate the tablespace in directory olddir to newdir during the backup. To be effective, olddir must exactly match the path specification of the tablespace as it is currently defined. (But it is not an error if there is no tablespace in olddir contained in the backup.) Both olddir and newdir must be absolute paths. If a path happens to contain a = sign, escape it with a backslash. This option can be specified multiple times for multiple tablespaces. See examples below.
If a tablespace is relocated in this way, the symbolic links inside the main data directory are updated to point to the new location. So the new data directory is ready to be used for a new server instance with all tablespaces in the updated locations.
--xlogdir=xlogdir
Specifies the location for the transaction log directory. xlogdir must be an absolute path. The transaction log directory can only be specified when the backup is in plain mode.
-x
--xlog
Using this option is equivalent of using -X with method fetch.
-X method
--xlog-method=method
Includes the required transaction log files (WAL files) in the backup. This will include all transaction logs generated during the backup. If this option is specified, it is possible to start a postmaster directly in the extracted directory without the need to consult the log archive, thus making this a completely standalone backup.
The following methods for collecting the transaction logs are supported:
f
fetch
The transaction log files are collected at the end of the backup. Therefore, it is necessary for the wal_keep_segments parameter to be set high enough that the log is not removed before the end of the backup. If the log has been rotated when it*(Aqs time to transfer it, the backup will fail and be unusable.
s
stream
Stream the transaction log while the backup is created. This will open a second connection to the server and start streaming the transaction log in parallel while running the backup. Therefore, it will use up two slots configured by the max_wal_senders parameter. As long as the client can keep up with transaction log received, using this mode requires no extra transaction logs to be saved on the master.
-z
--gzip
Enables gzip compression of tar file output, with the default compression level. Compression is only available when using the tar format.
-Z level
--compress=level
Enables gzip compression of tar file output, and specifies the compression level (1 through 9, 9 being best compression). Compression is only available when using the tar format.
The following command-line options control the generation of the backup and the running of the program.
-c fast|spread
--checkpoint=fast|spread
Sets checkpoint mode to fast or spread (default) (see Section 24.3.3, (lqMaking a Base Backup Using the Low Level API(rq, in the documentation).
-l label
--label=label
Sets the label for the backup. If none is specified, a default value of (lqpg_basebackup base backup(rq will be used.
-P
--progress
Enables progress reporting. Turning this on will deliver an approximate progress report during the backup. Since the database may change during the backup, this is only an approximation and may not end at exactly 100%. In particular, when WAL log is included in the backup, the total amount of data cannot be estimated in advance, and in this case the estimated target size will increase once it passes the total estimate without WAL.
When this is enabled, the backup will start by enumerating the size of the entire database, and then go back and send the actual contents. This may make the backup take slightly longer, and in particular it will take longer before the first data is sent.
-v
--verbose
Enables verbose mode. Will output some extra steps during startup and shutdown, as well as show the exact file name that is currently being processed if progress reporting is also enabled.
The following command-line options control the database connection parameters.
-d connstr
--dbname=connstr
Specifies parameters used to connect to the server, as a connection string. See Section 31.1.1, (lqConnection Strings(rq, in the documentation for more information.
The option is called --dbname for consistency with other client applications, but because pg_basebackup doesn*(Aqt connect to any particular database in the cluster, database name in the connection string will be ignored.
-h host
--host=host
Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket. The default is taken from the PGHOST environment variable, if set, else a Unix domain socket connection is attempted.
-p port
--port=port
Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections. Defaults to the PGPORT environment variable, if set, or a compiled-in default.
-s interval
--status-interval=interval
Specifies the number of seconds between status packets sent back to the server. This allows for easier monitoring of the progress from server. A value of zero disables the periodic status updates completely, although an update will still be sent when requested by the server, to avoid timeout disconnect. The default value is 10 seconds.
-U username
--username=username
User name to connect as.
-w
--no-password
Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.
-W
--password
Force pg_basebackup to prompt for a password before connecting to a database.
This option is never essential, since pg_basebackup will automatically prompt for a password if the server demands password authentication. However, pg_basebackup will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.
Other options are also available:
-V
--version
Print the pg_basebackup version and exit.
-?
--help
Show help about pg_basebackup command line arguments, and exit.
This utility, like most other PostgreSQL utilities, uses the environment variables supported by libpq (see Section 31.14, (lqEnvironment Variables(rq, in the documentation).
The backup will include all files in the data directory and tablespaces, including the configuration files and any additional files placed in the directory by third parties. But only regular files and directories are copied. Symbolic links (other than those used for tablespaces) and special device files are skipped. (See Section 49.3, (lqStreaming Replication Protocol(rq, in the documentation for the precise details.)
Tablespaces will in plain format by default be backed up to the same path they have on the server, unless the option --tablespace-mapping is used. Without this option, running a plain format base backup on the same host as the server will not work if tablespaces are in use, because the backup would have to be written to the same directory locations as the original tablespaces.
pg_basebackup works with servers of the same or an older major version, down to 9.1. However, WAL streaming mode (-X stream) only works with server version 9.3 and later.
To create a base backup of the server at mydbserver and store it in the local directory /usr/local/pgsql/data:
$ pg_basebackup -h mydbserver -D /usr/local/pgsql/data
To create a backup of the local server with one compressed tar file for each tablespace, and store it in the directory backup, showing a progress report while running:
$ pg_basebackup -D backup -Ft -z -P
To create a backup of a single-tablespace local database and compress this with bzip2:
$ pg_basebackup -D - -Ft | bzip2 > backup.tar.bz2
(This command will fail if there are multiple tablespaces in the database.)
To create a backup of a local database where the tablespace in /opt/ts is relocated to ./backup/ts:
$ pg_basebackup -D backup/data -T /opt/ts=$(pwd)/backup/ts
pg_dump(1)


Consul is a very popular service discovery and DNS solution. Dinivas can create Consul cluster and managed them for you.


export { default as ERROR } from "./error";
export { ERRORTYPES } from "./error";
Copyright 2015-2021 JC Fisher
The library includes functions for logic, math, text, lookup, date/time, aggregation, arrays, objects, finance, statistics and other utilities.
export { default as ERROR } from "./error";
export { ERRORTYPES } from "./error";
export { default as PARSE } from "./parser";
export { default as parse } from "./parser";
export { default as WALKER } from "./walker";
export { defaultConfig as WALKERCONFIGDEFAULT } from "./walker";
export { fpConfig as WALKERCONFIGFP } from "./walker";
export { jsConfig as WALKERCONFIGJS } from "./walker";
export { default as COMPILE } from "./compile";
export { default as compile } from "./compile";
export { default as RUN } from "./run";
export { default as run } from "./run";
export { default as BRANCH } from "./branch";
export { default as IF } from "./branch";
export { default as IFS } from "./branch";
export { default as branch } from "./branch";
export { default as ifs } from "./branch";
export { default as if } from "./branch";
export { default as CHOOSE } from "./choose";
export { default as choose } from "./choose";
export { default as SWITCH } from "./switch";
export { default as switch } from "./switch";
export { default as AND } from "./and";
export { default as and } from "./and";
export { default as NAND } from "./nand";
export { default as nand } from "./nand";
export { default as OR } from "./or";
export { default as or } from "./or";
export { default as NOR } from "./nor";
export { default as nor } from "./nor";
export { default as XOR } from "./xor";
export { default as xor } from "./xor";
export { default as NOT } from "./not";
export { default as not } from "./not";
eq
returns true when the first and second arguments are equivalent. String comparision is case insensitive.
export { default as EQ } from "./eq";
export { default as eq } from "./eq";
ne
returns true when the first and second arguments are not equivalent. String comparision is case insensitive.
export { default as NE } from "./ne";
export { default as ne } from "./ne";
export { default as GT } from "./gt";
export { default as gt } from "./gt";
export { default as GTE } from "./gte";
export { default as gte } from "./gte";
export { default as LT } from "./lt";
export { default as lt } from "./lt";
export { default as LTE } from "./lte";
export { default as lte } from "./lte";
export { default as IDENTITY } from "./identity";
export { default as identity } from "./identity";
ifempty
returns the second argument when the first argument is blank or the third argument
export { default as IFBLANK } from "./ifblank";
export { default as ifBlank } from "./ifblank";
export { default as ifblank } from "./ifblank";
ifempty
returns the second argument when the first argument is empty or the third argument
export { default as IFEMPTY } from "./ifempty";
export { default as ifEmpty } from "./ifempty";
export { default as ifempty } from "./ifempty";
ifempty
returns the second argument when the first argument is an error or the third argument
export { default as IFERROR } from "./iferror";
export { default as ifError } from "./iferror";
export { default as iferror } from "./iferror";
export { default as IFNA } from "./ifna";
export { default as ifNA } from "./ifna";
export { default as ifna } from "./ifna";
export { default as ISARRAY } from "./isarray";
export { default as isArray } from "./isarray";
export { default as isarray } from "./isarray";
export { default as ISASYNCFUNCTION } from "./isasyncfunction";
export { default as isAsyncFunction } from "./isasyncfunction";
export { default as isasyncfunction } from "./isasyncfunction";
export { default as ISBLANK } from "./isblank";
export { default as isBlank } from "./isblank";
export { default as isblank } from "./isblank";
export { default as ISBOOLEAN } from "./isboolean";
export { default as isBoolean } from "./isboolean";
export { default as isboolean } from "./isboolean";
export { default as ISDATE } from "./isdate";
export { default as isDate } from "./isdate";
export { default as isdate } from "./isdate";
export { default as ISEMAIL } from "./isemail";
export { default as isEmail } from "./isemail";
export { default as isemail } from "./isemail";
export { default as ISEMPTY } from "./isempty";
export { default as isEmpty } from "./isempty";
export { default as isempty } from "./isempty";
export { default as ISERROR } from "./iserror";
export { default as isError } from "./iserror";
export { default as iserror } from "./iserror";
export { default as ISEVEN } from "./iseven";
export { default as isEven } from "./iseven";
export { default as iseven } from "./iseven";
export { default as ISFALSY } from "./isfalsy";
export { default as isFalsy } from "./isfalsy";
export { default as isfalsy } from "./isfalsy";
export { default as ISFUNCTION } from "./isfunction";
export { default as isFunction } from "./isfunction";
export { default as isfunction } from "./isfunction";
export { default as ISLEAPYEAR } from "./isleapyear";
export { default as isLeapYear } from "./isleapyear";
export { default as isleapyear } from "./isleapyear";
export { default as ISLOWERCASE } from "./islowercase";
export { default as isLowerCase } from "./islowercase";
export { default as islowercase } from "./islowercase";
export { default as ISOBJECT } from "./isobject";
export { default as isObject } from "./isobject";
export { default as ISNA } from "./isna";
export { default as isNA } from "./isna";
export { default as isna } from "./isna";
export { default as ISNAN } from "./isnan";
export { default as isNaN } from "./isnan";
export { default as isnan } from "./isnan";
isnumber
returns true when the value is JavaScript number type and not NaN and not infinite.
export { default as ISNUMBER } from "./isnumber";
export { default as isNumber } from "./isnumber";
export { default as isnumber } from "./isnumber";
export { default as ISODD } from "./isodd";
export { default as isOdd } from "./isodd";
export { default as isodd } from "./isodd";
export { default as ISOWEEKNUM } from "./isoweeknum";
export { default as ISOWeekNum } from "./isoweeknum";
export { default as isoweeknum } from "./isoweeknum";
export { default as ISPROMISE } from "./ispromise";
export { default as isPromise } from "./ispromise";
export { default as ispromise } from "./ispromise";
export { default as ISREF } from "./isref";
export { default as isRef } from "./isref";
export { default as isref } from "./isref";
export { default as ISTEXT } from "./istext";
export { default as isText } from "./istext";
export { default as ISTRUTHY } from "./istruthy";
export { default as isTruthy } from "./istruthy";
export { default as istruthy } from "./istruthy";
export { default as ISUPPERCASE } from "./isuppercase";
export { default as isUpperCase } from "./isuppercase";
export { default as isuppercase } from "./isuppercase";
export { default as ISURL } from "./isurl";
export { default as isURL } from "./isurl";
export { default as isurl } from "./isurl";
export { default as ISWHOLENUMBER } from "./iswholenumber";
export { default as isWholeNumber } from "./iswholenumber";
export { default as iswholenumber } from "./iswholenumber";
export { default as MINUS } from "./minus";
export { default as minus } from "./minus";
export { default as PLUS } from "./plus";
export { default as plus } from "./plus";
export { default as ADD } from "./add";
export { default as add } from "./add";
export { default as SUBTRACT } from "./subtract";
export { default as subtract } from "./subtract";
export { default as MULTIPLY } from "./multiply";
export { default as multiply } from "./multiply";
export { default as DIVIDE } from "./divide";
export { default as divide } from "./divide";
export { default as ABS } from "./abs";
export { default as abs } from "./abs";
export { default as ACOS } from "./acos";
export { default as acos } from "./acos";
export { default as ACOSH } from "./acosh";
export { default as acosh } from "./acosh";
export { default as ACOT } from "./acot";
export { default as acot } from "./acot";
export { default as ACOTH } from "./acoth";
export { default as acoth } from "./acoth";
export { default as ASIN } from "./asin";
export { default as asin } from "./asin";
export { default as ASINH } from "./asinh";
export { default as asinh } from "./asinh";
export { default as ATAN } from "./atan";
export { default as atan } from "./atan";
export { default as ATAN2 } from "./atan2";
export { default as atan2 } from "./atan2";
export { default as ATANH } from "./atanh";
export { default as atanh } from "./atanh";
export { default as COS } from "./cos";
export { default as cos } from "./cos";
export { default as DEGREES } from "./degrees";
export { default as degrees } from "./degrees";
export { default as MOD } from "./mod";
export { default as mod } from "./mod";
export { default as PI } from "./pi";
export { default as pi } from "./pi";
export { default as POWER } from "./power";
export { default as power } from "./power";
export { default as ROUND } from "./round";
export { default as round } from "./round";
export { default as ROUNDUP } from "./roundup";
export { default as roundUp } from "./roundup";
export { default as roundup } from "./roundup";
export { default as SIN } from "./sin";
export { default as sin } from "./sin";
export { default as TAN } from "./tan";
export { default as tan } from "./tan";
export { default as TAU } from "./tau";
export { default as tau } from "./tau";
export { default as TRUNC } from "./trunc";
export { default as trunc } from "./trunc";
export { default as WITHIN } from "./within";
export { default as withIn } from "./within";
export { default as within } from "./within";
export { default as CHAR } from "./char";
export { default as char } from "./char";
export { default as CAMELCASE } from "./camelcase";
export { default as camelCase } from "./camelcase";
export { default as camelcase } from "./camelcase";
export { default as CODE } from "./code";
export { default as code } from "./code";
export { default as CONCATENATE } from "./concatenate";
export { default as concatenate } from "./concatenate";
export { default as EXACT } from "./exact";
export { default as exact } from "./exact";
export { default as FIND } from "./find";
export { default as find } from "./find";
export { default as JOIN } from "./join";
export { default as join } from "./join";
export { default as LEFT } from "./left";
export { default as left } from "./left";
export { default as LEN } from "./len";
export { default as len } from "./len";
export { default as LOWER } from "./lower";
export { default as lower } from "./lower";
export { default as NUMBERVALUE } from "./numbervalue";
export { default as numberValue } from "./numbervalue";
export { default as numbervalue } from "./numbervalue";
export { default as PARSEBOOL } from "./parsebool";
export { default as parseBool } from "./parsebool";
export { default as parsebool } from "./parsebool";
export { default as PARSEDATE } from "./parsedate";
export { default as parseDate } from "./parsedate";
export { default as parsedate } from "./parsedate";
export { default as PARSEQUERY } from "./parsequery";
export { default as parseQuery } from "./parsequery";
export { default as parsequery } from "./parsequery";
export { default as PROPER } from "./proper";
export { default as proper } from "./proper";
export { default as REPLACE } from "./replace";
export { default as replace } from "./replace";
export { default as RIGHT } from "./right";
export { default as right } from "./right";
export { default as REPT } from "./rept";
export { default as rept } from "./rept";
search
returns the index of a value inside a string with wildcard support for single characters (?) and multiple characters (*).
export { default as SEARCH } from "./search";
export { default as search } from "./search";
export { default as SNAKECASE } from "./snakecase";
export { default as snakeCase } from "./snakecase";
export { default as snakecase } from "./snakecase";
export { default as SUBSTITUTE } from "./substitute";
export { default as substitute } from "./substitute";
export { default as SUBSTITUTEALL } from "./substituteAll";
export { default as substituteAll } from "./substituteAll";
export { default as substituteall } from "./substituteAll";
surroundKeys
returns a new object where the keys have been wrapped with start and end strings.
export { default as SURROUNDKEYS } from "./surroundKeys";
export { default as surroundKeys } from "./surroundKeys";
export { default as surroundkeys } from "./surroundKeys";
export { default as SPLIT } from "./split";
export { default as split } from "./split";
export { default as TEXT } from "./text";
export { default as text } from "./text";
export { default as TRIM } from "./trim";
export { default as trim } from "./trim";
export { default as UPPER } from "./upper";
export { default as upper } from "./upper";
hlookup
searches the first row and returns the value from the found column at a given row.
export { default as HLOOKUP } from "./hlookup";
export { default as hlookup } from "./hlookup";
export { default as INCLUDES } from "./includes";
export { default as includes } from "./includes";
export { default as NOTINCLUDES } from "./notincludes";
export { default as notIncludes } from "./notincludes";
export { default as notincludes } from "./notincludes";
export { default as INDEX } from "./index";
export { default as index } from "./index";
lookup
searches an array and returns the value found or (optionally) the value at the same index in a second array.
export { default as LOOKUP } from "./lookup";
export { default as lookup } from "./lookup";
match
searches an array and returns the found index with support for wildcard and range queries.
export { default as MATCH } from "./match";
export { default as match } from "./match";
vlookup
searches the first column and returns the value from the found row at a given column.
export { default as VLOOKUP } from "./vlookup";
export { default as vlookup } from "./vlookup";
export { default as DATE } from "./date";
export { default as date } from "./date";
export { default as DATEVALUE } from "./datevalue";
export { default as datevalue } from "./datevalue";
export { default as DATEDIF } from "./datedif";
export { default as datedif } from "./datedif";
export { default as DAY } from "./day";
export { default as day } from "./day";
export { default as DAYS360 } from "./days360";
export { default as days360 } from "./days360";
edate
returns the serial number for a date that is a given number of months before or after a given date.
export { default as EDATE } from "./edate";
export { default as edate } from "./edate";
export { default as EOMONTH } from "./eomonth";
export { default as eomonth } from "./eomonth";
export { default as HOUR } from "./hour";
export { default as hour } from "./hour";
export { default as MINUTE } from "./minute";
export { default as minute } from "./minute";
export { default as MONTH } from "./month";
export { default as month } from "./month";
export { default as NOW } from "./now";
export { default as now } from "./now";
export { default as SECOND } from "./second";
export { default as second } from "./second";
export { default as TODAY } from "./today";
export { default as today } from "./today";
export { default as TIME } from "./time";
export { default as time } from "./time";
export { default as TIMEVALUE } from "./timevalue";
export { default as timevalue } from "./timevalue";
export { default as YEAR } from "./year";
export { default as year } from "./year";
export { default as YEARFRAC } from "./yearfrac";
export { default as yearfrac } from "./yearfrac";
export { default as AVERAGE } from "./average";
export { default as average } from "./average";
export { default as MIN } from "./min";
export { default as min } from "./min";
export { default as MAX } from "./max";
export { default as max } from "./max";
export { default as QUERY } from "./query";
export { default as query } from "./query";
export { default as SUM } from "./sum";
export { default as sum } from "./sum";
export { default as ACCRINT } from "./accrint";
export { default as accrint } from "./accrint";
export { default as FV } from "./fv";
export { default as fv } from "./fv";
export { default as NPER } from "./nper";
export { default as nper } from "./nper";
export { default as NPV } from "./npv";
export { default as npv } from "./npv";
export { default as PMT } from "./pmt";
export { default as pmt } from "./pmt";
export { default as CUMIPMT } from "./cumipmt";
export { default as cumipmt } from "./cumipmt";
ipmt
returns the portion of the periodic payment which is interest for a fixed rate loan or annuity.
export { default as IPMT } from "./ipmt";
export { default as ipmt } from "./ipmt";
export { default as PV } from "./pv";
export { default as pv } from "./pv";
export { default as BIN2DEC } from "./bin2dec";
export { default as bin2dec } from "./bin2dec";
export { default as DEC2BIN } from "./dec2bin";
export { default as dec2bin } from "./dec2bin";
export { default as OCT2DEC } from "./oct2dec";
export { default as oct2dec } from "./oct2dec";
export { default as FILTER } from "./filter";
export { default as filter } from "./filter";
export { default as FLATTEN } from "./flatten";
export { default as flatten } from "./flatten";
export { default as MAP } from "./map";
export { default as map } from "./map";
export { default as PLUCK } from "./pluck";
export { default as pluck } from "./pluck";
export { default as REDUCE } from "./reduce";
export { default as reduce } from "./reduce";
export { default as SOME } from "./some";
export { default as some } from "./some";
export { default as SORT } from "./sort";
export { default as sort } from "./sort";
export { default as UNFLATTEN } from "./unflatten";
export { default as unFlatten } from "./unflatten";
export { default as unflatten } from "./unflatten";
export { default as UNIQUE } from "./unique";
export { default as unique } from "./unique";
export { default as CHANGED } from "./changed";
export { default as changed } from "./changed";
export { default as DIFF } from "./diff";
export { default as diff } from "./diff";
export { default as CLEAN } from "./clean";
export { default as clean } from "./clean";
export { default as GET } from "./get";
export { default as get } from "./get";
export { default as SELECT } from "./select";
export { default as select } from "./select";
export { default as KEYS } from "./keys";
export { default as keys } from "./keys";
export { default as ADDRESS } from "./address";
export { default as address } from "./address";
export { default as ASSIGN } from "./assign";
export { default as assign } from "./assign";
export { default as BASE } from "./base";
export { default as base } from "./base";
export { default as CELLINDEX } from "./cellindex";
export { default as cellIndex } from "./cellindex";
export { default as cellindex } from "./cellindex";
export { default as CEILING } from "./ceiling";
export { default as ceiling } from "./ceiling";
export { default as COLUMN } from "./column";
export { default as column } from "./column";
export { default as COLUMNLETTER } from "./columnletter";
export { default as columnLetter } from "./columnletter";
export { default as columnletter } from "./columnletter";
export { default as COLUMNNUMBER } from "./columnnumber";
export { default as columnNumber } from "./columnnumber";
export { default as columnnumber } from "./columnnumber";
export { default as DECODEBASE64 } from "./decodebase64";
export { default as decodeBase64 } from "./decodebase64";
export { default as decodebase64 } from "./decodebase64";
export { default as DECODEJWT } from "./decodejwt";
export { default as decodeJWT } from "./decodejwt";
export { default as decodejwt } from "./decodejwt";
entries
returns returns an array of a given object’s
own enumerable property [key, value] pairs.
export { default as ENTRIES } from "./entries";
export { default as entries } from "./entries";
export { default as EVEN } from "./even";
export { default as even } from "./even";
export { default as FLOOR } from "./floor";
export { default as floor } from "./floor";
export { default as GROUP } from "./group";
export { default as group } from "./group";
export { default as GUID } from "./guid";
export { default as guid } from "./guid";
export { default as HASH } from "./hash";
export { default as hash } from "./hash";
export { default as INT } from "./int";
export { default as int } from "./int";
export { default as INDEX2COL } from "./index2col";
export { default as index2Col } from "./index2col";
export { default as index2col } from "./index2col";
export { default as INDEX2ROW } from "./index2row";
export { default as index2Row } from "./index2row";
export { default as index2row } from "./index2row";
export { default as INTERSECT } from "./intersect";
export { default as intersect } from "./intersect";
export { default as INTERPOLATE } from "./interpolate";
export { default as interpolate } from "./interpolate";
interpolator
creates a function that accepts a percentage distance between a and b, used by interpolate.
export { default as INTERPOLATOR } from "./interpolator";
export { default as interpolator } from "./interpolator";
export { default as N } from "./n";
export { default as n } from "./n";
export { default as NUMBERS } from "./numbers";
export { default as numbers } from "./numbers";
export { default as OVERLAP } from "./overlap";
export { default as overlap } from "./overlap";
export { default as REF } from "./ref";
export { default as ref } from "./ref";
export { default as SERIAL } from "./serial";
export { default as serial } from "./serial";


A Pure Java Browser Made With JavaFX Technology
Download Sorce Learn MoreImplements the latest JavaFX Technology.
Loading Progress Indicator Shows the Loading Progress of Web Pages.
Open Multiple Closable Tabs to Browse the Internet.
Navigate Browsing History Forward and Backward.
Supports HTML5 Technology.
Simple User-Friendly Interface.
This is a simple browser project developed using JavaFX Technology.
This will be a good starter to create an advanced browser using JavaFX technology.
With this project, you can learn about;
All Rights Reserved | Copyright © 2015 - Buddhilive Developers


This cheek blush blends into skin for a better-than-natural-looking flush thanks to its lightweight and superblendable gel texture.
This hydrating lip balm reacts to your body's chemistry to transform into a unique pink shade that enhances your natural lip color.
This mascara lengthens and volumizes your lashes. It's also sweatproof and clump-free, so your look will always stays polished.
Specializing in lightweight, long-wearing makeup and refreshing skincare, Birchbox beauty brand ARROW is designed to keep up with your on-the-go, active life—whether that means going straight from pilates to brunch or just powering through an action-packed, appointment-filled day. Each breathable product creates a fresh, natural, beautiful effect, so you can feel cool and confident from an A.M. gym session to a late-night work session.
LOVE IT! This definitely did it's job! It was perfect for those "I didn't get much sleep" days. Actually everyday.
OBSESSED! It’s nice to have a lipstick that matches my mood. It’s quick and easy to put on, like chapstick, but adds subtle color. Love this!
BEST LENGTHENING! Hands down the best Mascara for lengthening as well as great coverage. Love love love!
COOLING! Love this cheek tint so much! It's a lovely color and it feel so cool and delightful on my skin!
877.487.7272
info@arrowbeauty.com


Data handling overview
In a BPM-based application, use:
- Business variables for information that:
- is mostly used by Bonita
- has meaning after a process instance is archived
- is used at several places in the process
- is read by a process to orientate its flow
- is shared across several process instances
- is displayed in an application page
- Task-level process variables for:
- intermediate computation results (for example a connector result to be used as input of the next connector)
- information that is private to the process, not used elsewhere in the application
- Pool-level process variables for information that:
- can be read by the process to orientate its flow
- can be used at several places in the process
- is private to the process, not used elsewhere in the application
- External databases, with connectors and custom data types, for information that:
- is used by multiple applications besides Bonita
Key benefits of using business data:
- Control of indexing strategy
- Definition of custom queries to efficiently read the data
- Full control of lifecycle of the data through the set of processes that handle it (add, update and delete)
- Improved performance compared with process variables or external databases
- Easier application maintenance
An application page accesses business data using the bdm REST API. A page also contains page variables, which have no meaning outside the page.
The data that is passed from a form to a process is defined using a contract. Define a contract for process instantiation and for each human task. These contracts are part of the process definition. A form has access to business data and documents using the context](/bonita_doc/bonita/7.10/application-and-process-design/define-access-control-on-business-objects/contracts-and-contexts) and the . A form also contains form variables, which have no meaning outside the form.


The world around us is populated by independent objects. If an object's behavior is influenced by its history, then it is said to have state. For example, a bank account has state if it reports a new balance after a successful withdrawal operation. The bank account we defined in the previous chapter do not have state, because the withdrawal operation returned a new bank account instead of updating the current balance. This is not how bank accounts in the real world works!
To create bank account object that behave exactly like its real world counterpart, we need an operation that can change the value associated with its local variable. Slogan provides the assignment operator for this purpose. Assignment is accomplished by the statement:
var = new_value
Using assignment, we will define a new bank account abstraction that support three fundamental operations – balance
for
checking the current balance, deposit
for adding money to the current balance and withdraw
for getting money out of
the account.
function make_bank_account(balance)
where balance > 0
{ function deposit(amount)
balance = balance + amount
function withdraw(amount)
where balance - amount > 0
balance = balance - amount
^(message)
| 'balance -> balance
| 'deposit -> deposit
| 'withdraw -> withdraw }
As it can now update its internal state, a single instance of bank account is able to keep track of the running balance over a series of transactions:
let account = make_bank_account(100)
account.deposit(10)
account.balance
// 110
account.withdraw(80)
account.balance
// 30
Assignment helps a lot in improving the modularity of a system. To understand this better, consider a function that
adds the number 2
to its argument. For the purpose of reporting some metrics, it should also log the number of times
it was called. Without assignment, this is how we would define and use this function:
function add2(x, n)
{ showln("add2: ", n+1)
x+2:n+1 }
let fn = 0
let r:fn2 = add2(10, fn)
//> add2: 1
r
// 12
let r:fn3 = add2(21, fn2)
//> add2: 2
r
// 23
The user of add2
only cares about the result of calling the function, i.e x
added with 2
.
But now he has to worry about the implementation details of add2
and define extra variables to correctly keep track of the
metrics. If the caller is careless about the second argument, the metrics log will become wrong and useless.
The implementation of add2
in not modular because it leaks its internals to other parts of the program.
Assignment allows us to easily fix the problem. We will maintain the count in a changing variable visible only to add2
:
let add2 =
let (call_count = 0)
function (x)
{ call_count = call_count + 1
showln("add2: ", call_count)
x + 2 }
add2(10)
//> add2: 1
// 12
add2(21)
//> add2: 2
// 23
As we saw in the preceding section, assignment helps to model objects in the real world in a modular fashion. But this convenience comes with a price. Once we introduce assignment, it becomes harder to reason about the functions we write. As long as we don't do assignments, two invocations of the same function with the same arguments will produce the same result. In other words, functions without assignment are computing mathematical functions with easily verifiable properties. This style of programming is known as functional programming.
Consider a simple function that decrements a number by a given amount. Let us first write a functional version and how see how simple reasoning techniques can be applied to verify its correctness.
function make_dec(balance)
^(amount) balance - amount
Does the function returned by make_dec
work as expected? Is it possible to verify its correctness without actually running it?
For instance, consider the function returned by the call make_dec(100)
:
let d = make_dec(100)
What will be the result of calling d(20)
followed by a call to d(45)
?
It will produce 80
and 55
. We can verify this by simply substituting the
variables balance
and amount
in the function body with their respective values. We can "expand" the function call
to the following steps:
d(20)
=> balance - amount
=> 100 - 20
=> 80
d(45)
=> balance - amount
=> 100 - 45
=> 55
We are able to substitute the variable names with their values because the variables are guaranteed not to change. In other words, they are said to be referentially transparent.
Now let us look at a new version of the decrementor function that depend on assignment:
function make_dec(balance)
^(amount)
{ balance = balance - amount
balance }
Will the simple reasoning based on value substitution work for this new implementation? Let us find out by manually evaluating the same sequence of calls as before:
d(20)
=> { balance = balance - amount; balance }
=> { balance = 100 - 20; 100 }
=> 100
d(45)
=> { balance = balance - amount; balance }
=> { balance = 100 - 20; 100 }
=> 100
Our reasoning is obviously wrong1. If we actually run the code, we will get 80
and 35
as results. Once assignment is brought in we loose the ability to do such simple reasoning about functions. We can no
longer assume that a variable is just a name for a value. It now reflects on the underlying computer architecture. The variable has become
a reference to a location in the computer's memory, which can change. The programming model that makes extensive use of assignment is known
as imperative programming.
Assignment also affects the way how two objects are compared for equality. Without assignment two objects that were equal when they were created continues to be so for the rest of their lives. This is because their internal state never changes. In fact, a smart compiler can even optimize on storage by putting both objects at the same location in memory and doing a fast equality check by just comparing memory addresses. But once we add assignment to the language, for any equality check to be reliable, the structure of both objects has to be inspected.
Assignment tends to be an expensive operation in modern multi-core chips with multiple levels of caches. In functional programming variables are initialized once and they never changes. This leads to efficient caching of frequently accessed values at the hardware level. When a new value is assigned, the cached values has to be flushed and written back to the main memory. This might make a computation with assignments costlier than a purely functional one.
Assignment is also more susceptible to bugs because the order of doing assignments is important. The situation becomes more complicated once we have multiple threads of execution in the same program. Slogan provide elegant solutions for managing state in concurrent programs. This will be the topic of the next two chapters.
Earlier in this book, we learned how to do iterations with recursive function calls.
The named letfn
construct is a generalization
on this to enable local iterations. The iterations we did so far were functional because they don't have to change state
to keep track of the loops boundary conditions. For example, consider the following loop. On each iteration, a new value is created and passed to
the recursive call to loop
. No mutation of existing state is involved.
letfn loop (i = 0)
when (i <= 2)
{ showln(i)
loop(i + 1) }
//> 0
1
2
If you have programmed in an imperative language like C or Java, you must be more familiar with writing this kind of iterations using the
for
or while
statements. Though both styles will work for simple loops like the one above,
some algorithms can be implemented
efficiently and cleanly only in an imperative style. Slogan provides the for
looping construct to meet such
a requirement. The preceding example can be rewritten using for
as shown below:
for (i = 0 to 2)
showln(i)
//> 0
1
2
Counting downwards is accomplished by,
for (i = 2 downto 0)
showln(i)
//> 2
1
0
You are allowed to have explicit control over the looping counter and condition:
for (i = 5; i < 15; i+3)
showln(i)
//> 5
8
11
14
The for
loop construct is an expression in Slogan. It returns the value of the last expression
evaluated in its body.
let s = for (j = 0, i = 5; i < 15; i+3)
{ j = j + i
j }
s
// 38
You can exit early from the loop by calling the break
expression. This expression returns
false
by default. Break
can be asked to return another value,
by calling it like a function and passing the required value as argument.
for (i = 0; i < 10; i + 1)
when (i == 5) break
// false
for (i = 0; i < 10; i + 1)
when (i == 5) break(i + 1)
// 6
The continue
expression allows control to return to the top of the loop. Just like
break
, an optional argument can be passed to continue
.
This will become the value of the loop if continue
happens to be the last expression
evaluated in its body.
for (j = 0, i = 0; i < 10; i + 1)
if (remainder(i, 3) == 0) continue
else { j = j + i; j }
// false
for (j = 0, i = 0; i < 10; i + 1)
if (remainder(i, 3) == 0) continue(j)
else { j = j + i; j }
// 27
for (j = 0, i = 0; i < 10; i + 1)
if (remainder(i, 2) == 0) continue
else { j = j + i; j }
// 25
If your style of programming demands more imperative control structures like the for
loop, you can
add them to the language! We will figure out how to do this and more in the last chapter of Part I.
Once we have assignment in our language, we also have mutable data structures. For instance, it is now possible to have functions that change the values in the head or tail of a pair:
Other data structures like lists, arrays and hash tables can also be mutated:
let xs = [10, 20, 30]
xs[1] = xs[0] + xs[2]
xs
// [10, 40, 30]
let ys = #[10, 20, 30]
ys[1] = ys[1] * ys[2]
ys
// #[10, 600, 30]
let zs = #{'name:'alice, 'age:3}
zs['age] = 2
zs;
// #{name: alice, age: 2}
// for faster updates, use type-specific mutators/accessors
// instead of the generic [] operator.
list_set(xs, 0, "hello")
xs
// [hello, 40, 30]
array_set(ys, 0, \a)
ys
// #[\a, 600, 30]
hashtable_set(zs, 'name, 'bob)
zs
// #{name: bob, age: 2}
In this section, we will see how to define new mutable data structures. We will define a queue, a sequence in which items are inserted at one end (called the rear of the queue) and deleted from the other end (the front). The queue data structure will support the following operations:
make_queue – construct and return an empty queue
front_queue – select and return the first item in the queue
insert_queue – insert a new item at the rear of the queue
delete_queue – remove the first item in the queue
is_empty_queue – check if the queue is empty
These operations will internally use the following functions to access and modify the pointers to front and rear of the queue. Two lists will represent the front and rear parts and they enable constant time lookup and insertion operations.
function front_ptr(queue) head(queue)
function rear_ptr(queue) tail(queue)
function set_front_ptr(queue, item) set_head(queue, item)
function set_rear_ptr(queue, item) set_tail(queue, item)
The queue constructor return a pair of empty lists.
function make_queue() []:[]
A queue is considered empty if its front part is the empty list.
function is_empty_queue(q) is_empty(front_ptr(q))
Insertion should check for two conditions. If the queue is empty a list is created with the new item and both the front an rear pointers point to this list. If the queue is not empty, the rear pointer is updated to point to this list. Then the rear pointer is "truncated" to make sure the next insertion need not traverse all the elements in the queue.
function insert_queue(q, item) let (new_entry = [item]) if (is_empty_queue(q)) { set_front_ptr(q, new_entry) set_rear_ptr(q, new_entry) } else { set_tail(rear_ptr(q), new_entry) set_rear_ptr(q, new_entry) }
To select the item at the front of the queue, we return the head of the pair indicated by the front pointer.
function front_queue(q)
if (is_empty_queue(q))
error("front called on an empty queue")
else
head(front_ptr(q))
To delete the item at the front of the queue, we merely modify the front pointer so that it now points at the second item in the queue.
function delete_queue(q)
if (is_empty_queue(q))
error("delete called on an empty queue")
else
set_front_ptr(q, tail(front_ptr(q)))
Now it's time to see the queue data structure in action!
let q = make_queue()
insert_queue(q, 1)
insert_queue(q, 2)
front_queue(q)
// 1
delete_queue(q)
insert_queue(q, 3)
front_queue(q)
// 2
delete_queue(q)
front_queue(q)
// 3
Each insert and delete operation mutates the internal state of the queue. An advantage of this is that, we are able to do all operations in constant time. On the downside, we lose a version of the queue with an insert or delete operation. There is no way to "go back" and see the contents of the queue at point past in time. We also cannot easily share this queue across simultaneously executing computations (threads or tasks) without adding code for data synchronization. For applications that require these properties, we need a queue that is immutable.
In the pure functional model that disallow assignments, the best we can do is to get amortized constant-time insert and delete operations.
That is, any sequence of n
insert and delete operations takes a total time that is proportional to some constant times n
.
Any individual operation might not be constant-time, however.
The functional queue is also represented as a pair of lists, one for the front and the other for the rear.
At any instant, the queue content is given by
append(front, reverse(rear))
.
An element is inserted by adding it to the head of rear
and deleted by removing it from
the head of front
. To make this representation work, each element in rear
has to
be moved to front
sooner or later. This should be done only occasionally, when front
becomes empty. Then rear
is reversed and moved to front
. The reverse takes time
proportional to the length of rear
, i.e., to the number of elements it reverses. Each
element that goes through the queue is passed exactly once from rear
to front
.
Allocating the execution time of reverse
to each element therefore gives a constant time per element.
This is why the queue is amortized.
Having described how our immutable queue works, let us move on to its implementation. The queue constructor needs to accept the
front
and rear
lists as optional arguments. This is because each insert and delete operation has to create
a new queue, instead of mutating the existing one. We will also define a helper function called
check_queue
whose purpose is to reverse and move rear
to front
if front
is empty.
function make_queue(f = [], r = []) f:r
function check_queue(q)
if (is_empty(front_ptr(q)))
make_queue(reverse(rear_ptr(q)))
else q
The insert_queue
function adds the new item to the head of rear and makes sure the queue is ready to return the next item
from its front
. Similarly, delete_queue
also ensures the required invariant after removing the
head of front
.
function insert_queue(q, item)
check_queue(make_queue(front_ptr(q), item:rear_ptr(q)))
function delete_queue(q)
check_queue(make_queue(tail(front_ptr(q)), rear_ptr(q)))
The program below show operations on the queue. As this queue is immutable, we need to explicitly keep track of instances to reach a desired final state.
let q = insert_queue(insert_queue(make_queue(), 1), 2)
front_queue(q)
// 1
front_queue(delete_queue(insert_queue(delete_queue(q), 3)))
//3
The functions that implemented the immutable queue has effectively over-written the imperative queue implementation. Is it possible to have both implementations in the same program, while maintaining a single "protocol" for operations on queues? The obvious solution is to implement the queues as two closures, one imperative and the other functional.
Slogan provide a convenient way to describe closure objects, without having to explicitly define and return a "message-handler" function.
This is accomplished by the module
expression. A module
definition
has the following syntax:
module <name> (<export_list>) <definitions>
The export_list
contain the names from definitions
that must be visible outside the module. These are essentially
the "messages" an instance of the module can respond to.
Let us define the imperative queue as a module:
module mutable_queue(make, is_empty:is_empty_queue,
front, insert, delete)
{ function front_ptr(queue) head(queue)
function rear_ptr(queue) tail(queue)
function set_front_ptr(queue, item) set_head(queue, item)
function set_rear_ptr(queue, item) set_tail(queue, item)
function make() []:[]
function is_empty_queue(q) is_empty(front_ptr(q))
function insert(q, item)
let (new_entry = [item])
if (is_empty_queue(q))
{ set_front_ptr(q, new_entry)
set_rear_ptr(q, new_entry) }
else
{ set_tail(rear_ptr(q), new_entry)
set_rear_ptr(q, new_entry) }
function front(q)
if (is_empty_queue(q))
error("front called on an empty queue")
else
head(front_ptr(q))
function delete(q)
if (is_empty_queue(q))
error("delete called on an empty queue")
else
set_front_ptr(q, tail(front_ptr(q))) }
The functions make
, is_empty_queue
, front
, insert
and delete
are exported and made visible to the consumers of the module. These names constitute the public interface
of the module. The functions front_ptr
, rear_ptr
,
set_front_ptr
and set_rear_ptr
are used in the implementation of the module and need not be published to
the outside world.
Note that for all public functions, except for is_empty_queue
we have omitted the _queue
suffix.
This is because the module name makes the extra suffix redundant. We retained the suffix for is_empty_queue
so that
it won't conflict with the is_empty
built-in function used in its implementation. But note that we export
is_empty_queue
as is_empty
. This achieved by the export specification – is_empty:is_empty_queue
.
The exported name will not conflict with the global is_empty
because it is scoped by the module name.
This is how we would use the new mutable_queue
module:
let mq = mutable_queue
let q = mq.make()
mq.insert(q, 1)
mq.insert(q, 2)
mq.front(q)
// 1
mq.delete(q)
mq.insert(q, 3)
mq.front(q)
// 2
mq.delete(q)
mq.front(q)
// 3
Modules
are first-class objects in Slogan. They can be assigned to variables, they can be passed as arguments to functions and
they can be dynamically created and returned from functions. This is why we could assign the shorter alias mq
to refer to
the mutable_queue
module.
We shall put the immutable queue into its own module as well:
module immutable_queue(make, is_empty:is_empty_queue,
front, insert, delete)
{ function front_ptr(queue) head(queue)
function rear_ptr(queue) tail(queue)
function make(f = [], r = []) f:r
function check(q)
if (is_empty(front_ptr(q)))
make(reverse(rear_ptr(q)))
else q
function is_empty_queue(q) is_empty(front_ptr(q))
function front(q)
if (is_empty_queue(q))
error("front called on an empty queue")
else head(front_ptr(q))
function insert(q, item)
check(make(front_ptr(q), item:rear_ptr(q)))
function delete(q)
check(make(tail(front_ptr(q)), rear_ptr(q))) }
This is how we would use the immutable_queue
module:
let imq = immutable_queue
let q = imq.insert(imq.insert(imq.make(), 1), 2)
imq.front(q)
// 1
imq.front(imq.delete(imq.insert(imq.delete(q), 3)))
// 3
Now we have two separate queue modules that share a single protocol. How can we package these two together so that users can choose between the mutable and immutable implementations as they wish? One way to do this is to put them into a "super" module as shown below:
module queue(mutable_queue, immutable_queue)
{ module mutable_queue(...) { ... }
module immutable_queue(...) { ... } }
Another option is to make use of the namespace
facility:
namespace queue
{ module mutable_queue(...) { ... }
module immutable_queue(...) { ... } }
All members in a namespace
are public and are accessible via the ::
operator.
let mq = queue::mutable_queue
let imq = queue::immutable_queue
The primary purpose of a namespace is to arrange modules and other namespaces into logical groups. Just like modules, namespaces are first-class objects in Slogan.
1We did not substitute balance
before the assignment operator
because the expression 100 = 100 - 20
does not make sense and cannot be evaluated.


Liverpool boss Jurgen Klopp cannot repeat Man City ‘mistake’ in crunch title clash
The sides face off in their first showdown of the season on Sunday with the Reds holding a six-point advantage
by Darren Wells, https://www.mirror.co.uk/authors/darren-wells/Liverpool boss Jurgen Klopp has been warned not to make the same mistake against Man City - or face the consequences in the title race.
The Reds have opened up a six-point lead on their rivals ahead of the clash between the two sides at Anfield on Sunday.
Last season Liverpool were seven points clear when they travelled to the Etihad in January, but were beaten as City caught them up to defend their crown.
Klopp's men had let City off the hook in a dull contest at Anfield earlier in the campaign, which former Reds midfielder Jamie Redknapp feels was crucial.
"Liverpool cannot make the same mistake they made last season when Manchester City left Anfield with a 0-0 draw," Redknapp told the Mail .
"Jurgen Klopp's men were passive last October. In hindsight, we can see how costly it was that they only picked up one point rather than three.
"This Sunday, they should go for the win and strike a psychological blow."
Despite the draw, Liverpool were perhaps fortunate to take anything from that fixture, with City's Riyad Mahrez missing a penalty in the dying minutes.
The return fixture saw City win 2-1 by the finest of margins, as John Stones's clearance off the line stayed out of the net by millimetres, and proved significant to their surge for the title.
There have already been subtle jibes between both managers ahead of Sunday's match, after Klopp hit back at Pep Guardiola's claims that Sadio Mane was a diver.
The Senegalese forward was booked for simulation against Aston Villa on Saturday, before heading in an injury-time winner.
City boss Guardiola said: "It's happened a lot for Liverpool in the last years [scoring late on]. It's because he's a special talent.
"Sometimes he's diving, sometimes he has this talent to score incredible goals in the last minute. He's a talent."
Klopp took umbrage with his adversary, responding: "Sadio is not a diver. Yes there was a situation in the Aston Villa game where he felt contact and went down.
“Maybe it was not a penalty but there was contact.
“It's not like he jumped over a leg or something. All the other penalties were penalties because he was there.
“If something like that had happened for Manchester City I'm sure they would have wanted a penalty too.”
The teams are both involved in respective Champions League ties this week, before Sunday's clash which kicks off at 4:30pm (GMT).


Man dies following M50 crash
by Jack QuannA motorcyclist has died following a serious road crash on the M50 in Dublin.
It happened at the merging of the off ramp and M50 at Junction 5 Southbound at around 7:45am on Tuesday.
The 29-year-old man was taken to Connolly Hospital Blanchardstown, where he has since passed away.
A post mortem is due to take place at a later date.
The road has fully re-opened since.
Gardaí are continuing to appeal for anyone who may have witnessed the incident, particularly any road users who may have camera footage, to contact Finglas Garda station on 01-666-7500 or the Garda Confidential Line at 1800-666-111.


FOI release
How the department ensures the safety and security of pupils' personal data.
Now that the department has control over Social Work and Children’s Law does this mean any teacher/anyone at the local education department can access personal data?
How is personal data being protected and kept private from members of council departments and which data is kept private (unless ordered otherwise by a judge) and whom exactly has the right to access any/all our data as and when they want to?
This department has the lead responsibility for children’s policy in England. Professionals working in children’s services in local authorities ( LAs ) in England may have access to personal information about children which it holds where necessary and appropriate for the performance of their functions. LAs will have their own organisational policies around the handling of personal information.
All processing of personal data will need to comply with the Data Protection Act and other relevant legislation. There are also statutory provisions requiring Schools and LAs to share specified personal information with the department.
If a child is at risk of harm there may be a clear and legitimate purpose to share information. Sharing of information when there is concern about a child’s welfare will enable professionals to consider jointly how to proceed in the best interests of the child and to safeguard children more generally. In deciding whether there is a need to share information professionals should consider their legal obligations, including whether they have a duty of confidentiality to the child.
Where there is such a duty, the professional may lawfully share information if the child consents or if there is a public interest of sufficient force. This must be judged by the professional on the facts of each case. Where there is a clear likelihood of a child suffering significant harm the public interest test will almost certainly be satisfied.
The Information Commissioner’s Office ( ICO ) also provides guidance to public bodies and to individuals on the handling of personal data and when such data can be shared lawfully between professionals.


Built for Speed
These apps make use of lightweight resources to greatly improve loading speed and performance.
Full Customization
This collection of apps supports theming of multiple components.
Made for Android
These apps have quality design and are meant to look good running on the latest devices.
Fornax
A collection of flat wallpapers.
Paragraphical Expander
An essay-typing app that doubles, triples, or quadruples the spaces while replacing words with longer phrases to increase the size.
Card Calculator
A basic calculator that shows a detailed equation history in a list of cards.
Simple Browser
A material design browser app, focusing on the implementation of tabs on smaller screens.
Simple Camera
A small camera app in material design.
Charge Screen
A screen that shows the battery life whenever your phone is plugged in to charge.
Felix Launcher
A dark-themed launcher, built to save battery life on older devices.
List Launcher
A launcher based on a list of cards.
Simple Browser
A material design browser app, focusing on the implementation of tabs on smaller screens.
Simple Camera
A small camera app in material design.
Felix Launcher
A dark-themed launcher, built to save battery life on older devices.
Card Calculator
A basic calculator that shows a detailed equation history in a list of cards.
Charge Screen
A screen that shows the battery life whenever your phone is plugged in to charge.
List Launcher
A launcher based on a list of cards.
Fornax
A collection of flat wallpapers.
Paragraphical Expander
An essay-typing app that doubles, triples, or quadruples the spaces while replacing words with longer phrases to increase the size.


In case you have not seen this very cool presentation done at the TED conference, check this out there : Photosynth. This is a project I would do anything to work on it. They are reconstructing the Notre-Dame Cathedral with image from flicker. Can you imagine that! Unbelievable! If you still have time to check another video, check the Tony Robbins presentation. I have read his book some years ago (his name in the french book is Anthony Robbins, translator have this crazy idea that name can be translated, by example : Buster in the Mythbuster show is called Jimmy) and he had help me a lot finding what I wanted to do in life. He is very entertaining and motivating.
Rise of Ruby
2 years ago, someone told me Ruby will be the “next big thing”. I told him “no way”. C# will gain more and more popularity and ASP.NET 2.0 will totally rules. God, I was wrong. After 2 months of Ruby On Rails, I must admit it’s here to stay. The ease of use of the Rails framework is impressive. In 30 minutes I have managed to create a little web application to keep track of my weight with complete support for creating/editing/deleting as well as listing all items. This was the first minute, the remaining 29 minutes was modifying the design and doing some CSS.
If you have never seen the coding of a blog engine by a ruby on rails programmer, then you must absolutely check this screencast. There is a lot more screencasts that will encourage you to go further on the Ruby On Rails website. And a bonus for all of you who are already into ruby : RailsCasts
Getting rid of intellisense addiction
For a month now, I have been trying to learn ruby and the Ruby On Rails framework. The biggest difficulty so far was programming without intellisense. Honestly, I’m loving this difficulty. I reading a lot more the documentation that before and find this really helpful for learning more rapidly. But the coding is a lot more slower. But, this is a personal project and I have no deadline (and you can get a project done really quick with Rails, believe me !).
Now, if only I can get rid of the mouse …
I will use Windows Live Writer
Some months ago (when I was blogging on WDevs) I was using Windows Live Writer a lot. The only problem I had, was to configure it to actually see my blog. And today I have decided to give it a second try. And, wow, no problem at all. Everything is working perfectly. There is also some interesting plug-in that will enhance my experience (ex : Syntax Highlighting and Flicker4Writer).
This, has made my day. :) Windows Live Writer
My first continuous integration server is complete !
I have completed configuring my first continuous integration server for an ASP.NET Web Application. Here is the workflow:
- Get the latest source from SourceSafe.
- Compile of course. The first compilation is done with MSBuild.
- Second compilation with devenv.exe (Visual Studio). For some reason, MSBuild does not compile deployment and Setup project.
- Run unit test with MSTest.
- Run Simian to detect code duplication.
- Delete previous log file.
- Deploy the web application.
- Copy Web.Config to the web application folder.
- Create database and execute all the database scripts.
- Start Selenium Remote Control and execute all functional tests.
The tools used are:
- CruiseControl.NET
- sqlcmd
- NAnt
- Visual Studio 2005
- Selenium Remote Control
- MSBuild
- MSTest
- Simian
I have been through a lot of problem during the setup, but how fascinating it was! I was in constant learning.
Getting started with Selenium IDE - Part 1
As promised, I will return back everything I have learned about Selenium. I invite you to go the OpenQA web site for additional information. If you stick with me for this series of article, you will get a great quick start to create good acceptances tests. Selenium IDE (Integrated Development Environment) is a Firefox extension that records your actions on a page. Then you can execute them back with the IDE or with other version of Selenium (Selenium Remote Control is one of them). First step : Download Firefox Second step: Install Selenium IDE extension on your browser
- Start Firefox and navigate to the Selenium IDE home page.
- Click the download link. This will install the IDE.
(If , for some reason, the Selenium IDE can’t replay your test, switch to a previous version.)
Third step : Record your first test
- In Firefox, go to Tools\Selenium IDE.
- Make sure the record button is pressed.
- Navigate on the web application.
- Press the green arrow (start button).
- Your first test is done.
Now, maybe the replay has not worked and you got some errors. This will be covered in the next post when I will show you how to add others commands to your Selenium test.
Here is some features found in the interface:
- There is the baseURL text field. Suppose you have two times the same web site, you want to have this attribute because you will avoid hard coding the path in the test. Simply specific another baseURL and the test will be performed on it.
- Run/Walk/Step. This is basically the speed at witch the test will be executed. However, Step will require you to press the “Step” button to go line by line. Very useful.
- Of course you can save the test, juste press Ctrl-S or go to File/Save Test.
- Table/Source tabs. Basically, a test is just an html file (not entirely true, you will see later), composed of a table element, and 3 columns (each one has a specific purpose).
- Play with Selenium Test Runner. This will launch a test runner for your test (will be covered later).
A lot more is possible and this is only the beginning. If you find missing information, just leave a comment, and I will update the post. Selenium IDE Getting Started Series
Blog of the day
And this blog is : Patrick Dussud’s Blog
Very informative! This is the kind of thing I have overlooked when learning .NET and was very important (at least to impress colleague). And who is the most appropriate to explain the birth of the CLR than the Lead Architect of the CLR (Common Language Runtime) himself.
Sys is undefined
Lastly I have got to play a lot with AJAX (ATLAS) and have been pleasantly surprised by the quality and the ease of use of the product. Really easy to use and “a lot of support” for it. For every problem there is a solution and those seem to be resolved fast by the community.
The first really big error so far, is the ‘Sys’ is undefined one. There seems to have a problem with the anonymous user being disable for the web site (in particular the script folder). Well, I’m sure that I wont allow anonymous users to access critical data, so adding a new location in the Web.Config for this specific folder with solve the problem.
Here is the lines to add :
<location path=”ScriptResource.axd”>
<system.web>
<authorization>
<allow users=”*”/>
</authorization>
</system.web>
</location>
There can be also the fact that the resource handler is not set, so adding the following line in the Web.Config, should also solve the problem :
<add verb=”GET” path=”ScriptResource.axd” type=”Microsoft.Web.Handlers.ScriptResourceHandler” validate=”false”/>
Should I learn Eiffel
Ever wanted to learn a new programming language? Today, I installed today the IDE that came with Eiffel (see link) and have been really surprised by the syntax. And the fact that is considered an academic language make it really interesting. I will keep you informed of my progress.
Easier check if an item is in the list of items on a ListControl
Every time I set the SelectedValue of a DropdownList in ASP.NET, I was checking if the value existed in the control and if it doesn’t, I usually set a default one (creating a new item with a SelectedIndex of -1). I realized that if you set the FormattingEnabled to true, the SelectedIndex will be set to -1. Thus removing the need for the this check.
This property is found on the ListControl class, with the Dropdownlist inherit.
Happy coding !


--- layout: page ---
Home › Support › Give
Donation Amount: (Currency: USD)
Repeat Donation: Do not repeatWeeklyMonthly x
Mailing Address:
5526 W. 13400 S. #352 Herriman UT 84096
"Each of you should give what you have decided in your heart to give, not reluctantly or under compulsion, for God loves a cheerful giver."


Change the following image to a grape whose width is 300.
Go to Exercise 1. The name of anchor "a1" is:


Able Player is a fully accessible cross-browser HTML5 media player designed from the ground up to meet the diverse needs and preferences of all users.
Following are examples of Able Player in action. All demos use the "2020" skin unless otherwise indicated.
The audio examples feature songs by Terrill Thompson, Able Player's creator and lead developer. Check out Terrill's music site for more listening, and to support his work.
The video examples feature clips from videos produced by The DO-IT Center at the University of Washington. Additional videos are available on the DO-IT Video website, which uses Able Player.


Django-CookieSessions v0.1.2 documentation
Install it from the Python Package Index
easy_install cookiesession
Comment out, or delete django.contrib.sessions.middleware.SessionMiddleware from settings.py’s MIDDLEWARE_CLASSES setting.
Add cookiesession’s middleware to your settings.py MIDDLEWARE_CLASSES
MIDDLEWARE_CLASSES = (
'django.middleware.common.CommonMiddleware',
#'django.contrib.sessions.middleware.SessionMiddleware',
'cookiesessions.middleware.CookieSessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
)
Optional. In order to use the management commands you need to add cookiesession to settings.py’s INSTALLED_APPS
In settings.py (or local_settings.py) set SESSION_COOKIE_DOMAIN, SESSION_COOKIE_NAME, or SESSION_COOKIE_PATH as necessary (see the Django session docs for more info).
Developers will want to override the SESSION_COOKIE_DOMAIN with <IP_ADDRESS> and access their local development server as http://<IP_ADDRESS>:8000/
Cookies are inherently insecure and can’t be trusted. Users have access to the values and can do what ever they want with them. So how could you ever put important data in a cookie?
There’s a difference between “important” and “secret”. Django only stores two pieces of data about the user in the session cookie: the user’s id and the authentication backend used for authentication. Neither of these bits of information are “secret” in that knowing them can lead to security breaches. They are “important” in that we need them to retrieve the user’s information.
However with cookies, a user could easily modify the user id stored in the cookie and suddenly be a different user. That means we need a way to detect a modification to the cookie.
The process is pretty straight forward:
When the user has logged in and makes a new request, we reverse the steps to verify the cookie hasn’t been tampered with:


Review The Spanish Prisoner Written and Directed by David Mamet Starring: Campbell Scott, Steve Martin, Rebecca Pigeon, Ben Gazzara Approx 120 min.
No one is what you think they are. That is the basis of The Spanish Prisoner, a new film by Pulitzer Prize winning playwright and sometime director David Mamet. This psychological thriller has the makings of a big-time, whamm-o production, but still comes off as a rather small, soon to be forgotten picture. Which is too bad because it has a lot of bright spots. Belive it, Steve Martin is a bad guy, a rich guy, and not too funny. He is the mystery man who charms his way into techno-inventor Campbell Scott's life, and attempts to ruin it.
Scott's character, Joe Ross, has made a brilliant invention for his company (the details of which we never learn) but begins to feel slighted that he will not be monetarily rewarded for it. As his disappointment grows he becomes entangled in a web of mystery that is spun by enigmatic millionaire Martin (Jimmy) and a secretary for the company, Susua, played by Rebecca Pigeon (writer/director Mamet's real-life bride).
The Spanish Prisoner begins to take on the feel of David Fincher's The Game, as the lead characters from both are surrounded by uncertainty (as is indeed the audience) and they lose control of the events in their lives. But Michael Douglas portrayed a sense of frustration and helplessness that Scott was either unable to, or instructed not to convey to the viewer. As things get more and more out of hand, I never saw pain, frustration or terror in Scott's face. Though it is a good performance, the strength of The Spanish Prisoner is found in the extremely "Mametian" script, the refreshing performance by Martin, and the suspense and mystery that I, personally, could not help but fall into.
There are, of course, a number of coincidental happenings that allow the story to continue, and they, as much as anything else, make the film completely unoriginal, but it still mangages to deliver. The devices are just crafty enough to make it work. But I have a feeling that you will need to go out and see The Spanish Prisoner rather soon if you want to catch it on the big screen. It just has that feel.
*** out of **** Stars Copyright (C) 1998, Nick Amado
Comments, complaints, questions, email me at<EMAIL_ADDRESS>The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Cox’s Bazar is a District under Chittagong Division, which is famous for its longest unbroken sandy sea beach. It is located 150 km south of the industrial port- Chittagong. Cox’s Bazar is considered as having the longest sea beach in the world, with a total of 121 kilometer long. The name Cox’s Bazar was derived from its founder, Captain Cox. He founded the very attractive beach in 1798. Then the Cox’s Bazar beach started only as a small port and health resort. Though the beach is considered to be the longest beach in the world, it has been the least crowded among the other beaches. Here, visitors can enjoy the relaxing breeze of the Bay of Bengal Sea and the peacefulness of the place. The Cox’s bazar beach certainly has the finest leisure it can offer to each of its visitors. People can take a timely stroll along the lengthened stretch of the beach and enjoy the view of the amazing seascape. Visitors can also enjoy water sport activities like scuba diving, surfing, and try some boat rides. There are also other wonderful places to see around the Cox’s beach bazar. One of these is the beautiful Himchari waterfall, located 18 kilometer south of Cox’s Bazar sea beach. This is the most visited place in the district, with its beautiful refreshing green hills and the wonderful water fall, where visitors can bathe and swim apart from the sea itself. Another attraction is the Moheshkhali Island near the Cox’s beach bazar coast. The island can be reached via speedboat. Here, visitors can view the breath-taking view of the island. With the bright green color of its mangrove forest that will surely amaze everyone who come to see the place. This is definitely the next best thing to explore after one’s beach trip. Cox’s Beach Bazar really is the place best recommended for people who seeks for serene, soberness and relaxing vacation. Every Cox’s Beach Bazar trip will surely please and satisfy everyone who visits here.


Father of ex-police minister granted bail over alleged hit-and-run
by The New DailyEx-NSW police minister Troy Grant’s father, who is accused of killing a pedestrian while drink driving at more than twice the legal limit, will spend Christmas with his family after being released on bail.
Kenneth Wayne Grant appeared in Burwood Local Court on Monday, where he was granted conditional bail by magistrate Lisa Stapleton.
The 70-year-old, who previously lived alone, must report to Gladesville police station on a weekly basis, live with his son Troy in Chiswick and cannot drink alcohol.
Ms Stapleton said Mr Grant had no criminal history, strong community ties, no criminal associations and was a “man of impeccable character”.
However, she said Mr Grant had allegedly committed a “serious offence” that resulted in a death, and police had a strong case against him.
If Mr Grant been refused bail, the retired police officer would potentially have been in custody for more than 12 months with a custodial sentence likely to follow, Ms Price said, adding he posed only a “slight flight risk”.
The court also heard Mr Grant might have cognitive impairment issues requiring investigation relating to his culpability.
Mr Grant was arrested on Sunday after allegedly hitting Tony Greenfield, 62, with his ute about 11.30pm on November 30 at Flat Road in the Maitland suburb of Bolwarra.
The scientist, a friend of Mr Grant’s, died of his injuries in Newcastle’s John Hunter Hospital.
The ute that allegedly struck him had left the scene. A short time later, police spotted the gold Mazda and attempted to stop it, but the driver ignored them and a pursuit began.
The low-speed chase through local streets ended on Maitland’s High Street after the vehicle mounted the footpath, detectives say.
Police have previously said Mr Grant returned a positive roadside breath test and a subsequent analysis returned a reading of 0.108.
Mr Grant was initially charged with a mid-range drink-driving offence and driving recklessly while not stopping for police.
After further investigations, he was charged with dangerous driving occasioning death and failing to stop offences.
His son Troy – who was NSW police minister from April 2015-March 2019 – broke down outside court on Monday, saying his father had “offered to trade his own life” to bring back Mr Greenfield.
“That’s not possible,” Troy Grant said.
“The only thing we can do is try to get his family and everyone the answers as to how and why this tragedy has occurred and that journey starts today.”
Mr Grant said his father was devastated by what had happened.
“I’m devastated. All he’s talked about is what a terrific person Tony and his family were. He can’t speak high enough of him,” he said.
Mr Grant is next due to face Maitland Local Court on December 18.
-with AAP


Travis Encrypter
×
You've entered an invalid input. Please try again.
Your Github Repository URL
https://github.com/
Variable to be Encrypted
Encrypt
×
The algorithim used to encrypt these values is non-deterministic. That means that encrypting the same value multiple times is unlikely to produce the same output. Any of the outputs produced below will work.


Preparation time
30 mins to 1 hour
Cooking time
over 2 hours
Serves
Serves 4-6
Dietary
Melted white chocolate, mango, passion fruit and caramel… what’s not to love about this easy pavlova recipe?
For the meringue, preheat the oven to 140C/280F/Gas 1.
Place the egg whites into a clean glass bowl and whisk with a hand whisk until soft peaks are formed when the whisk is removed from the bowl. Add half of the sugar and whisk until firm, then fold in the remainder of the sugar until combined.
Gently fold in the cornflour and vinegar with a spoon until well combined; then set aside.
Line a baking tray with non-stick baking parchment. With a large spoon, make a nest shape with the meringue. Place in the oven to cook for 10 minutes before turning the heat down to 120C250F/Gas ¼ and cooking for a further hour. Then turn the oven off and leave the meringue overnight in the residual heat. Then remove from the oven and leave to cool completely.
For the filling, whisk the cream in a bowl until soft peaks form when the whisk is removed from the bowl. Spread the white chocolate in the centre of the meringue nest and fill with the whipped cream. Sprinkle over the mango and spoon over the passion fruit pulp.
For the caramel, place the sugar and water into a non-stick pan and put over a medium heat. Allow to caramelise, then remove from the heat and set aside to cool for 1-2 minutes. Using a spoon, drizzle the caramel over the top of the pavlova. Serve immediately.


This document describes the architecture and other implementation/correctness aspects of the Manifest Committer
The protocol and its correctness are covered in Manifest Committer Protocol.
The Manifest committer is a committer for work which provides performance on ABFS for “real world” queries, and performance and correctness on GCS.
This committer uses the extension point which came in for the S3A committers. Users can declare a new committer factory for abfs://
and gcs://
URLs. It can be used through Hadoop MapReduce and Apache Spark.
| Term | Meaning |
|---|---|
| Committer | A class which can be invoked by MR/Spark to perform the task and job commit operations. |
| Spark Driver | The spark process scheduling the work and choreographing the commit operation. |
| Job | In MapReduce. the entire application. In spark, this is a single stage in a chain of work |
| Job Attempt | A single attempt at a job. MR supports multiple Job attempts with recovery on partial job failure. Spark says “start again from scratch” |
| Task | a subsection of a job, such as processing one file, or one part of a file |
| Task ID | ID of the task, unique within this job. Usually starts at 0 and is used in filenames (part-0000, part-001, etc.) |
| Task attempt (TA) | An attempt to perform a task. It may fail, in which case MR/spark will schedule another. |
| Task Attempt ID | A unique ID for the task attempt. The Task ID + an attempt counter. |
| Destination directory | The final destination of work. |
| Job Attempt Directory | A temporary directory used by the job attempt. This is always underneath the destination directory, so as to ensure it is in the same encryption zone as HDFS, storage volume in other filesystems, etc. |
| Task Attempt directory | (also known as “Task Attempt Working Directory”). Directory exclusive for each task attempt under which files are written |
| Task Commit | Taking the output of a Task Attempt and making it the final/exclusive result of that “successful” Task. |
| Job Commit | aggregating all the outputs of all committed tasks and producing the final results of the job. |
The purpose of a committer is to ensure that the complete output of a job ends up in the destination, even in the presence of failures of tasks.
For Hive’s classic hierarchical-directory-structured tables, job committing requires the output of all committed tasks to be put into the correct location in the directory tree.
The committer built into hadoop-mapreduce-client-core
module is the FileOutputCommitter
.
The Manifest Committer is a higher performance committer for ABFS and GCS storage for jobs which create file across deep directory trees through many tasks.
It will also work on hdfs://
and indeed, file://
URLs, but it is optimized to address listing and renaming performance and throttling issues in cloud storage.
 It will also have the performance problems of copying rather than moving all the generated data.
Although it will work with MapReduce there is no handling of multiple job attempts with recovery from previous failed attempts.
A Manifest file is designed which contains (along with IOStatistics and some other things)
Task attempts are committed by:
No renaming takes place —the files are left in their original location.
The directory treewalk is single-threaded, then it is O(directories)
, with each directory listing using one or more paged LIST calls.
This is simple, and for most tasks, the scan is off the critical path of of the job.
Statistics analysis may justify moving to parallel scans in future.
Job Commit consists of:
_SUCCESS
file with the same format as the S3A committer (for testing; use write and rename for atomic save)The job commit phase supports parallelization for many tasks and many files per task, specifically:
O(files)
on Google cloud storage, and also on ABFS when OAuth authentication is used.Optional scan of all ancestors …if any are files, delete.
getFileStatus()
on the path. Not found: create directory, add entry and those of all parent paths Found and is directory: add entry and those of all parent paths Found and is file: delete. then create as before.Efficiently handling concurrent creation of directories (or delete+create) is going to be a troublespot; some effort is invested there to build the set of directories to create.
Files are renamed in parallel.
A pre-rename check for anything being at that path (and deleting it) will be optional. With spark creating new UUIDs for each file, this isn’t going to happen, and saves HTTP requests.
Optional scan of all committed files and verify length and, if known, etag. For testing and diagnostics.
This solution is necessary for GCS and should be beneficial on ABFS as listing overheads are paid for in the task committers.
A key goal is to keep the manifest committer isolated and neither touch the existing committer code nor other parts of the hadoop codebase.
It must plug directly into MR and Spark without needing any changes other than already implemented for the S3A Committers
PathOutputCommitterFactory
.As a result of this there’s a bit of copy and paste from elsewhere, e.g. org.apache.hadoop.util.functional.TaskPool
is based on S3ACommitter’s org.apache.hadoop.fs.s3a.commit.Tasks
.
The_SUCCESS
file MUST be compatible with the S3A JSON file. This is to ensure any existing test suites which validate S3A committer output can be retargeted at jobs executed by the manifest committer without any changes.
When? Proposed: heartbeat until renaming finally finishes.
We would want to stop the entire job commit. Some atomic boolean “abort job” would need to be checked in the processing of each task committer thread’s iteraton through a directory (or processing of each file?) Failures in listing or renaming will need to be escalated to halting the entire job commit. This implies that any IOE raised in asynchronous rename operation or in a task committer thread must:
commitJob()
callIf a job commit stage is using a thread pool for per-task operations, e.g. loading files, that same thread pool MUST NOT be used for parallel operations within the per-task stage.
As every JobStage
is executed in sequence within task or job commit, it is safe to share the same thread pool across stages.
In the current implementation, there is no parallel “per manifest” operation in job commit other than for actually loading the files. The operations to create directories and to rename files are actually performed without performing parallel processing of individual manifests.
Directory Preparation: merge the directory lists of all manifests, then queue for creation the (hopefully very much smaller) set of unique directories.
Rename: iterate through all manifests and queue their renames into a pool for renaming.
The lifespan of thread pools is constrained to that of the stage configuration, which will be limited to within each of the PathOutputCommitter
methods to setup, commit, abort and cleanup.
This avoids the thread pool lifecycle problems of the S3A Committers.
This was a failure in terasorting where many tasks each generated many files; the full list of files to commit (and the etag of every block) was built up in memory and validated prior to execution.
The manifest committer assumes that the amount of data being stored in memory is less, because there is no longer the need to store an etag for every block of every file being committed.
Combine all lists of directories to create and eliminate duplicates.
The implementation architecture reflects lessons from the S3A Connector.
The committer collects duration statistics on all the operations it performs/invokes against filesystems. * Those collected during task commit are saved to the manifest (excluding the time to save and rename that file) * When these manifests are loaded during job commit, these statistics are merged to form aggregate statistics of the whole job. * Which are saved to the _SUCCESS
file * and to any copy of that file in the directory specified by mapreduce.manifest.committer.summary.report.directory
, if set. to be saved. * The class org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.ManifestPrinter
can load and print these.
IO statistics from filsystems and input and output streams used in a query are not collected.
When invoking the ManifestCommitter
via the PathOutputCommitter
API, the following attributes are added to the active (thread) context
| Key | Value |
|---|---|
ji |
Job ID |
tai |
Task Attempt ID |
st |
Stage |
These are also all set in all the helper threads performing work as part of a stage’s execution.
Any store/FS which supports auditing is able to collect this data and include in their logs.
To ease backporting, all audit integration is in the single class org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.AuditingIntegration
.


Class FreeClimbCommon
FreeClimbCommon is the base class from which all classes that are built to represent objects returned from the FreeClimb API are built upon. A user of the SDK should never directly interact with a FreeClimbCommon object.
Inheritance
Inherited Members
Namespace: com.freeclimb.api
Assembly: freeclimb-cs-sdk.dll
Syntax
[JsonObject]
public class FreeClimbCommon : IFreeClimbCommon, IEquatable<FreeClimbCommon>
Properties
| Improve this Doc View SourcegetDateCreated
Retrieve Time and date when this resource was created.
Declaration
public DateTime getDateCreated { get; }
Property Value
| Type | Description |
|---|---|
| System.DateTime | Time and date when this resource was created. |
getDateUpdated
Retrieve Time and date when this resource was last updated.
Declaration
public DateTime getDateUpdated { get; }
Property Value
| Type | Description |
|---|---|
| System.DateTime | Time and date when this resource was last updated. |
getRevision
Retrieve the number of revisions this object has undergone.
Declaration
public int getRevision { get; }
Property Value
| Type | Description |
|---|---|
| System.Int32 | The number of revisions this object has undergone. |
getUri
Retrieve URI where this resource can be found.
Declaration
public string getUri { get; }
Property Value
| Type | Description |
|---|---|
| System.String | URI where this resource can be found. |
Methods
| Improve this Doc View SourceEqual(FreeClimbCommon, FreeClimbCommon)
Helper method to deep compare two FreeClimbCommon instances.
Declaration
public static bool Equal(FreeClimbCommon a, FreeClimbCommon b)
Parameters
| Type | Name | Description |
|---|---|---|
| FreeClimbCommon | a | FreeClimbCommon instance one. |
| FreeClimbCommon | b | FreeClimbCommon instance two. |
Returns
| Type | Description |
|---|---|
| System.Boolean | True is a == b, otherwise false. |
Equals(FreeClimbCommon)
Compares the current instance with another FreeClimbCommon object.
Declaration
public bool Equals(FreeClimbCommon other)
Parameters
| Type | Name | Description |
|---|---|---|
| FreeClimbCommon | other | FreeClimbCommon object. |
Returns
| Type | Description |
|---|---|
| System.Boolean | true if the current object is equal to the other parameter; otherwise, false. |
Equals(Object)
Compares the current instance with another object of the same type.
Declaration
public override bool Equals(object obj)
Parameters
| Type | Name | Description |
|---|---|---|
| System.Object | obj | Object of type FreeClimbCommon. |
Returns
| Type | Description |
|---|---|
| System.Boolean | true if the current object is equal to the other parameter; otherwise, false. |
Overrides
GetHashCode()
Serves as the default hash function.
Declaration
public override int GetHashCode()
Returns
| Type | Description |
|---|---|
| System.Int32 | A hash code for the current object. |
Overrides
Operators
| Improve this Doc View SourceEquality(FreeClimbCommon, FreeClimbCommon)
FreeClimbCommon equality operator.
Declaration
public static bool operator ==(FreeClimbCommon a, FreeClimbCommon b)
Parameters
| Type | Name | Description |
|---|---|---|
| FreeClimbCommon | a | FreeClimbCommon object. |
| FreeClimbCommon | b | FreeClimbCommon object. |
Returns
| Type | Description |
|---|---|
| System.Boolean | true if equal otherwise false. |
Inequality(FreeClimbCommon, FreeClimbCommon)
FreeClimbCommon inequality operator.
Declaration
public static bool operator !=(FreeClimbCommon a, FreeClimbCommon b)
Parameters
| Type | Name | Description |
|---|---|---|
| FreeClimbCommon | a | FreeClimbCommon object. |
| FreeClimbCommon | b | FreeClimbCommon object. |
Returns
| Type | Description |
|---|---|
| System.Boolean | true if not equal otherwise false. |