As I continue to hack away at the design of Made Mistakes and improve on things, I’ve tried to release a simplified Jekyll starter after each major revision. Just my way of giving back to the community.
Below are all of the Jekyll starters I’ve open sourced thus far. Each “theme” contains _layouts
, _includes
, assets, and sample pages to get you started. For the most part they share a lot of similarities (especially in _config.yml
), which should make switching and updating easier.
All themes and starters are fully compatible with GitHub Pages.
Skinny Bones
A Jekyll starter used to build the latest incarnation of Made Mistakes.
- Minimal design meant to be hacked up and customized
- Built in support for Sass and data files (requires Jekyll 2.x)
- Off canvas menu
- Tile based archives
- Optional table of contents, share links, Disqus comments modules, and more
Minimal Mistakes
A minimally designed responsive Jekyll theme for text heavy blogs.
- Minimal design
- Built in support for Sass and data files (requires Jekyll 2.x)
- Author sidebar
- Automatic table of contents on single posts via Kramdown
So Simple
A simple and clean responsive Jekyll theme for words and large photographs.
- Minimal design
- Simple search (searches by post title)
- Built in support for Sass and data files (requires Jekyll 2.x)
HPSTR
A responsive Jekyll blog theme designed to give your site a modern and somewhat cliché feel.
- Modern, subtle design
- Animated main menu
- Built in support for Sass and data files (requires Jekyll 2.x)
License
These themes are free and open source software, distributed under the MIT License. Feel free to to modify them to suit your needs.


#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import sys
import json
from py4j.java_gateway import java_import
from pyspark import since, keyword_only
from pyspark.sql.column import _to_seq
from pyspark.sql.readwriter import OptionUtils, to_str
from pyspark.sql.types import StructType, StructField, StringType
from pyspark.sql.utils import ForeachBatchFunction, StreamingQueryException
__all__ = ["StreamingQuery", "StreamingQueryManager", "DataStreamReader", "DataStreamWriter"]
[docs]class StreamingQuery(object):
"""
A handle to a query that is executing continuously in the background as new data arrives.
All these methods are thread-safe.
.. versionadded:: 2.0.0
Notes
-----
This API is evolving.
"""
def __init__(self, jsq):
self._jsq = jsq
@property
@since(2.0)
def id(self):
"""Returns the unique id of this query that persists across restarts from checkpoint data.
That is, this id is generated when a query is started for the first time, and
will be the same every time it is restarted from checkpoint data.
There can only be one query with the same id active in a Spark cluster.
Also see, `runId`.
"""
return self._jsq.id().toString()
@property
@since(2.1)
def runId(self):
"""Returns the unique id of this query that does not persist across restarts. That is, every
query that is started (or restarted from checkpoint) will have a different runId.
"""
return self._jsq.runId().toString()
@property
@since(2.0)
def name(self):
"""Returns the user-specified name of the query, or null if not specified.
This name can be specified in the `org.apache.spark.sql.streaming.DataStreamWriter`
as `dataframe.writeStream.queryName("query").start()`.
This name, if set, must be unique across all active queries.
"""
return self._jsq.name()
@property
@since(2.0)
def isActive(self):
"""Whether this streaming query is currently active or not.
"""
return self._jsq.isActive()
[docs] @since(2.0)
def awaitTermination(self, timeout=None):
"""Waits for the termination of `this` query, either by :func:`query.stop()` or by an
exception. If the query has terminated with an exception, then the exception will be thrown.
If `timeout` is set, it returns whether the query has terminated or not within the
`timeout` seconds.
If the query has terminated, then all subsequent calls to this method will either return
immediately (if the query was terminated by :func:`stop()`), or throw the exception
immediately (if the query has terminated with exception).
throws :class:`StreamingQueryException`, if `this` query has terminated with an exception
"""
if timeout is not None:
if not isinstance(timeout, (int, float)) or timeout < 0:
raise ValueError("timeout must be a positive integer or float. Got %s" % timeout)
return self._jsq.awaitTermination(int(timeout * 1000))
else:
return self._jsq.awaitTermination()
@property
@since(2.1)
def status(self):
"""
Returns the current status of the query.
"""
return json.loads(self._jsq.status().json())
@property
@since(2.1)
def recentProgress(self):
"""Returns an array of the most recent [[StreamingQueryProgress]] updates for this query.
The number of progress updates retained for each stream is configured by Spark session
configuration `spark.sql.streaming.numRecentProgressUpdates`.
"""
return [json.loads(p.json()) for p in self._jsq.recentProgress()]
@property
def lastProgress(self):
"""
Returns the most recent :class:`StreamingQueryProgress` update of this streaming query or
None if there were no progress updates
.. versionadded:: 2.1.0
Returns
-------
dict
"""
lastProgress = self._jsq.lastProgress()
if lastProgress:
return json.loads(lastProgress.json())
else:
return None
[docs] def processAllAvailable(self):
"""Blocks until all available data in the source has been processed and committed to the
sink. This method is intended for testing.
.. versionadded:: 2.0.0
Notes
-----
In the case of continually arriving data, this method may block forever.
Additionally, this method is only guaranteed to block until data that has been
synchronously appended data to a stream source prior to invocation.
(i.e. `getOffset` must immediately reflect the addition).
"""
return self._jsq.processAllAvailable()
[docs] @since(2.0)
def stop(self):
"""Stop this streaming query.
"""
self._jsq.stop()
[docs] def explain(self, extended=False):
"""Prints the (logical and physical) plans to the console for debugging purpose.
.. versionadded:: 2.1.0
Parameters
----------
extended : bool, optional
default ``False``. If ``False``, prints only the physical plan.

# We should print it in the Python process.
print(self._jsq.explainInternal(extended))
[docs] def exception(self):
"""
.. versionadded:: 2.1.0
Returns
-------
:class:`StreamingQueryException`
the StreamingQueryException if the query was terminated by an exception, or None.
"""
if self._jsq.exception().isDefined():
je = self._jsq.exception().get()
msg = je.toString().split(': ', 1)[1] # Drop the Java StreamingQueryException type info
stackTrace = '\n\t at '.join(map(lambda x: x.toString(), je.getStackTrace()))
return StreamingQueryException(msg, stackTrace, je.getCause())
else:
return None
[docs]class StreamingQueryManager(object):
"""A class to manage all the :class:`StreamingQuery` StreamingQueries active.
.. versionadded:: 2.0.0
Notes
-----
This API is evolving.
"""
def __init__(self, jsqm):
self._jsqm = jsqm
@property
def active(self):
"""Returns a list of active queries associated with this SQLContext
.. versionadded:: 2.0.0
Examples
--------
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()
>>> sqm = spark.streams
>>> # get the list of active streaming queries
>>> [q.name for q in sqm.active]
['this_query']
>>> sq.stop()
"""
return [StreamingQuery(jsq) for jsq in self._jsqm.active()]
[docs] def get(self, id):
"""Returns an active query from this SQLContext or throws exception if an active query
with this name doesn't exist.
.. versionadded:: 2.0.0
Examples
--------
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()
>>> sq.name
'this_query'
>>> sq = spark.streams.get(sq.id)
>>> sq.isActive
True
>>> sq = sqlContext.streams.get(sq.id)
>>> sq.isActive
True
>>> sq.stop()
"""
return StreamingQuery(self._jsqm.get(id))
[docs] @since(2.0)
def awaitAnyTermination(self, timeout=None):
"""Wait until any of the queries on the associated SQLContext has terminated since the
creation of the context, or since :func:`resetTerminated()` was called. If any query was
terminated with an exception, then the exception will be thrown.
If `timeout` is set, it returns whether the query has terminated or not within the
`timeout` seconds.
If a query has terminated, then subsequent calls to :func:`awaitAnyTermination()` will
either return immediately (if the query was terminated by :func:`query.stop()`),
or throw the exception immediately (if the query was terminated with exception). Use
:func:`resetTerminated()` to clear past terminations and wait for new terminations.
In the case where multiple queries have terminated since :func:`resetTermination()`
was called, if any query has terminated with exception, then :func:`awaitAnyTermination()`
will throw any of the exception. For correctly documenting exceptions across multiple
queries, users need to stop all of them after any of them terminates with exception, and
then check the `query.exception()` for each query.
throws :class:`StreamingQueryException`, if `this` query has terminated with an exception
"""
if timeout is not None:
if not isinstance(timeout, (int, float)) or timeout < 0:
raise ValueError("timeout must be a positive integer or float. Got %s" % timeout)
return self._jsqm.awaitAnyTermination(int(timeout * 1000))
else:
return self._jsqm.awaitAnyTermination()
[docs] def resetTerminated(self):
"""Forget about past terminated queries so that :func:`awaitAnyTermination()` can be used
again to wait for new terminations.
.. versionadded:: 2.0.0
Examples
--------
>>> spark.streams.resetTerminated()
"""
self._jsqm.resetTerminated()
[docs]class DataStreamReader(OptionUtils):
"""
Interface used to load a streaming :class:`DataFrame <pyspark.sql.DataFrame>` from external
storage systems (e.g. file systems, key-value stores, etc).
Use :attr:`SparkSession.readStream <pyspark.sql.SparkSession.readStream>` to access this.
.. versionadded:: 2.0.0
Notes
-----
This API is evolving.
"""
def __init__(self, spark):
self._jreader = spark._ssql_ctx.readStream()
self._spark = spark
def _df(self, jdf):
from pyspark.sql.dataframe import DataFrame
return DataFrame(jdf, self._spark)
[docs] def schema(self, schema):
"""Specifies the input schema.
Some data sources (e.g. JSON) can infer the input schema automatically from data.
By specifying the schema here, the underlying data source can skip the schema
inference step, and thus speed up data loading.
.. versionadded:: 2.0.0
Parameters
----------
schema : :class:`pyspark.sql.types.StructType` or str
a :class:`pyspark.sql.types.StructType` object or a DDL-formatted string
(For example ``col0 INT, col1 DOUBLE``).
Notes
-----
This API is evolving.
Examples
--------
>>> s = spark.readStream.schema(sdf_schema)
>>> s = spark.readStream.schema("col0 INT, col1 DOUBLE")
"""
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
if isinstance(schema, StructType):
jschema = spark._jsparkSession.parseDataType(schema.json())
self._jreader = self._jreader.schema(jschema)
elif isinstance(schema, str):
self._jreader = self._jreader.schema(schema)
else:
raise TypeError("schema should be StructType or string")
return self
[docs] def option(self, key, value):
"""Adds an input option for the underlying data source.
.. versionadded:: 2.0.0
Notes
-----
This API is evolving.
Examples
--------
>>> s = spark.readStream.option("x", 1)
"""
self._jreader = self._jreader.option(key, to_str(value))
return self
[docs] def options(self, **options):
"""Adds input options for the underlying data source.
.. versionadded:: 2.0.0
Notes
-----
This API is evolving.
Examples
--------
>>> s = spark.readStream.options(x="1", y=2)
"""
for k in options:
self._jreader = self._jreader.option(k, to_str(options[k]))
return self
[docs] def load(self, path=None, format=None, schema=None, **options):
"""Loads a data stream from a data source and returns it as a
:class:`DataFrame <pyspark.sql.DataFrame>`.
.. versionadded:: 2.0.0
Parameters
----------
path : str, optional
optional string for file-system backed data sources.
format : str, optional
optional string for format of the data source. Default to 'parquet'.
schema : :class:`pyspark.sql.types.StructType` or str, optional
optional :class:`pyspark.sql.types.StructType` for the input schema
or a DDL-formatted string (For example ``col0 INT, col1 DOUBLE``).
**options : dict
all other string options
Notes
-----
This API is evolving.
Examples
--------
>>> json_sdf = spark.readStream.format("json") \\
... .schema(sdf_schema) \\
... .load(tempfile.mkdtemp())
>>> json_sdf.isStreaming
True
>>> json_sdf.schema == sdf_schema
True
"""
if format is not None:
self.format(format)
if schema is not None:
self.schema(schema)
self.options(**options)
if path is not None:
if type(path) != str or len(path.strip()) == 0:
raise ValueError("If the path is provided for stream, it needs to be a " +
"non-empty string. List of paths are not supported.")
return self._df(self._jreader.load(path))
else:
return self._df(self._jreader.load())
[docs] def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None,
allowComments=None, allowUnquotedFieldNames=None, allowSingleQuotes=None,
allowNumericLeadingZero=None, allowBackslashEscapingAnyCharacter=None,
mode=None, columnNameOfCorruptRecord=None, dateFormat=None, timestampFormat=None,
multiLine=None, allowUnquotedControlChars=None, lineSep=None, locale=None,
dropFieldIfAllNull=None, encoding=None, pathGlobFilter=None,
recursiveFileLookup=None, allowNonNumericNumbers=None):
"""
Loads a JSON file stream and returns the results as a :class:`DataFrame`.
`JSON Lines <http://jsonlines.org/>`_ (newline-delimited JSON) is supported by default.
For JSON (one record per file), set the ``multiLine`` parameter to ``true``.
If the ``schema`` parameter is not specified, this function goes
through the input once to determine the input schema.
.. versionadded:: 2.0.0
Parameters
----------
path : str
string represents path to the JSON dataset,
or RDD of Strings storing JSON objects.
schema : :class:`pyspark.sql.types.StructType` or str, optional
an optional :class:`pyspark.sql.types.StructType` for the input schema
or a DDL-formatted string (For example ``col0 INT, col1 DOUBLE``).
Other Parameters
----------------
Extra options
For the extra options, refer to
`Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-json.html#data-source-option>`_
in the version you use.
.. # noqa
Notes
-----
This API is evolving.
Examples
--------
>>> json_sdf = spark.readStream.json(tempfile.mkdtemp(), schema = sdf_schema)
>>> json_sdf.isStreaming
True
>>> json_sdf.schema == sdf_schema
True
"""
self._set_opts(
schema=schema, primitivesAsString=primitivesAsString, prefersDecimal=prefersDecimal,
allowComments=allowComments, allowUnquotedFieldNames=allowUnquotedFieldNames,
allowSingleQuotes=allowSingleQuotes, allowNumericLeadingZero=allowNumericLeadingZero,
allowBackslashEscapingAnyCharacter=allowBackslashEscapingAnyCharacter,
mode=mode, columnNameOfCorruptRecord=columnNameOfCorruptRecord, dateFormat=dateFormat,
timestampFormat=timestampFormat, multiLine=multiLine,
allowUnquotedControlChars=allowUnquotedControlChars, lineSep=lineSep, locale=locale,
dropFieldIfAllNull=dropFieldIfAllNull, encoding=encoding,
pathGlobFilter=pathGlobFilter, recursiveFileLookup=recursiveFileLookup,
allowNonNumericNumbers=allowNonNumericNumbers)
if isinstance(path, str):
return self._df(self._jreader.json(path))
else:
raise TypeError("path can be only a single string")
[docs] def orc(self, path, mergeSchema=None, pathGlobFilter=None, recursiveFileLookup=None):
"""Loads a ORC file stream, returning the result as a :class:`DataFrame`.
.. versionadded:: 2.3.0
Other Parameters
----------------
Extra options
For the extra options, refer to
`Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-orc.html#data-source-option>`_
in the version you use.
.. # noqa
Examples
--------
>>> orc_sdf = spark.readStream.schema(sdf_schema).orc(tempfile.mkdtemp())
>>> orc_sdf.isStreaming
True
>>> orc_sdf.schema == sdf_schema
True
"""
self._set_opts(mergeSchema=mergeSchema, pathGlobFilter=pathGlobFilter,
recursiveFileLookup=recursiveFileLookup)
if isinstance(path, str):
return self._df(self._jreader.orc(path))
else:
raise TypeError("path can be only a single string")
[docs] def parquet(self, path, mergeSchema=None, pathGlobFilter=None, recursiveFileLookup=None,
datetimeRebaseMode=None, int96RebaseMode=None):
"""
Loads a Parquet file stream, returning the result as a :class:`DataFrame`.
.. versionadded:: 2.0.0
Parameters
----------
path : str
the path in any Hadoop supported file system
Other Parameters
----------------
Extra options
For the extra options, refer to
`Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#data-source-option>`_.
in the version you use.
.. # noqa
Examples
--------
>>> parquet_sdf = spark.readStream.schema(sdf_schema).parquet(tempfile.mkdtemp())
>>> parquet_sdf.isStreaming
True
>>> parquet_sdf.schema == sdf_schema
True
"""
self._set_opts(mergeSchema=mergeSchema, pathGlobFilter=pathGlobFilter,
recursiveFileLookup=recursiveFileLookup,
datetimeRebaseMode=datetimeRebaseMode, int96RebaseMode=int96RebaseMode)
if isinstance(path, str):
return self._df(self._jreader.parquet(path))
else:
raise TypeError("path can be only a single string")
[docs] def text(self, path, wholetext=False, lineSep=None, pathGlobFilter=None,
recursiveFileLookup=None):
"""
Loads a text file stream and returns a :class:`DataFrame` whose schema starts with a
string column named "value", and followed by partitioned columns if there
are any.
The text files must be encoded as UTF-8.
By default, each line in the text file is a new row in the resulting DataFrame.
.. versionadded:: 2.0.0
Parameters
----------
paths : str or list
string, or list of strings, for input path(s).
Other Parameters
----------------
Extra options
For the extra options, refer to
`Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-text.html#data-source-option>`_
in the version you use.
.. # noqa
Notes
-----
This API is evolving.
Examples
--------
>>> text_sdf = spark.readStream.text(tempfile.mkdtemp())
>>> text_sdf.isStreaming
True
>>> "value" in str(text_sdf.schema)
True
"""
self._set_opts(
wholetext=wholetext, lineSep=lineSep, pathGlobFilter=pathGlobFilter,
recursiveFileLookup=recursiveFileLookup)
if isinstance(path, str):
return self._df(self._jreader.text(path))
else:
raise TypeError("path can be only a single string")
[docs] def csv(self, path, schema=None, sep=None, encoding=None, quote=None, escape=None,
comment=None, header=None, inferSchema=None, ignoreLeadingWhiteSpace=None,
ignoreTrailingWhiteSpace=None, nullValue=None, nanValue=None, positiveInf=None,
negativeInf=None, dateFormat=None, timestampFormat=None, maxColumns=None,
maxCharsPerColumn=None, maxMalformedLogPerPartition=None, mode=None,
columnNameOfCorruptRecord=None, multiLine=None, charToEscapeQuoteEscaping=None,
enforceSchema=None, emptyValue=None, locale=None, lineSep=None,
pathGlobFilter=None, recursiveFileLookup=None, unescapedQuoteHandling=None):
r"""Loads a CSV file stream and returns the result as a :class:`DataFrame`.
This function will go through the input once to determine the input schema if
``inferSchema`` is enabled. To avoid going through the entire data once, disable
``inferSchema`` option or specify the schema explicitly using ``schema``.
Parameters
----------
path : str or list
string, or list of strings, for input path(s).
schema : :class:`pyspark.sql.types.StructType` or str, optional
an optional :class:`pyspark.sql.types.StructType` for the input schema
or a DDL-formatted string (For example ``col0 INT, col1 DOUBLE``).
.. versionadded:: 2.0.0
Other Parameters
----------------
Extra options
For the extra options, refer to
`Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-csv.html#data-source-option>`_
in the version you use.
.. # noqa
Notes
-----
This API is evolving.
Examples
--------
>>> csv_sdf = spark.readStream.csv(tempfile.mkdtemp(), schema = sdf_schema)
>>> csv_sdf.isStreaming
True
>>> csv_sdf.schema == sdf_schema
True
"""
self._set_opts(
schema=schema, sep=sep, encoding=encoding, quote=quote, escape=escape, comment=comment,
header=header, inferSchema=inferSchema, ignoreLeadingWhiteSpace=ignoreLeadingWhiteSpace,
ignoreTrailingWhiteSpace=ignoreTrailingWhiteSpace, nullValue=nullValue,
nanValue=nanValue, positiveInf=positiveInf, negativeInf=negativeInf,
dateFormat=dateFormat, timestampFormat=timestampFormat, maxColumns=maxColumns,
maxCharsPerColumn=maxCharsPerColumn,
maxMalformedLogPerPartition=maxMalformedLogPerPartition, mode=mode,
columnNameOfCorruptRecord=columnNameOfCorruptRecord, multiLine=multiLine,
charToEscapeQuoteEscaping=charToEscapeQuoteEscaping, enforceSchema=enforceSchema,
emptyValue=emptyValue, locale=locale, lineSep=lineSep,
pathGlobFilter=pathGlobFilter, recursiveFileLookup=recursiveFileLookup,
unescapedQuoteHandling=unescapedQuoteHandling)
if isinstance(path, str):
return self._df(self._jreader.csv(path))
else:
raise TypeError("path can be only a single string")
def table(self, tableName):
"""Define a Streaming DataFrame on a Table. The DataSource corresponding to the table should
support streaming mode.
.. versionadded:: 3.1.0
Parameters
----------
tableName : str
string, for the name of the table.
Returns
--------
:class:`DataFrame`
Notes
-----
This API is evolving.
Examples
--------
>>> spark.readStream.table('input_table') # doctest: +SKIP
"""
if isinstance(tableName, str):
return self._df(self._jreader.table(tableName))
else:
raise TypeError("tableName can be only a single string")
[docs]class DataStreamWriter(object):
"""
Interface used to write a streaming :class:`DataFrame <pyspark.sql.DataFrame>` to external
storage systems (e.g. file systems, key-value stores, etc).
Use :attr:`DataFrame.writeStream <pyspark.sql.DataFrame.writeStream>`
to access this.
.. versionadded:: 2.0.0
Notes
-----
This API is evolving.
"""
def __init__(self, df):
self._df = df
self._spark = df.sql_ctx
self._jwrite = df._jdf.writeStream()
def _sq(self, jsq):
from pyspark.sql.streaming import StreamingQuery
return StreamingQuery(jsq)
[docs] def outputMode(self, outputMode):
"""Specifies how data of a streaming DataFrame/Dataset is written to a streaming sink.
.. versionadded:: 2.0.0
Options include:
* `append`: Only the new rows in the streaming DataFrame/Dataset will be written to
the sink
* `complete`: All the rows in the streaming DataFrame/Dataset will be written to the sink
every time these are some updates
* `update`: only the rows that were updated in the streaming DataFrame/Dataset will be
written to the sink every time there are some updates. If the query doesn't contain
aggregations, it will be equivalent to `append` mode.
Notes
-----
This API is evolving.
Examples
--------
>>> writer = sdf.writeStream.outputMode('append')
"""
if not outputMode or type(outputMode) != str or len(outputMode.strip()) == 0:
raise ValueError('The output mode must be a non-empty string. Got: %s' % outputMode)
self._jwrite = self._jwrite.outputMode(outputMode)
return self
[docs] def option(self, key, value):
"""Adds an output option for the underlying data source.
.. versionadded:: 2.0.0
Notes
-----
This API is evolving.
"""
self._jwrite = self._jwrite.option(key, to_str(value))
return self
[docs] def options(self, **options):
"""Adds output options for the underlying data source.
.. versionadded:: 2.0.0
Notes
-----
This API is evolving.
"""
for k in options:
self._jwrite = self._jwrite.option(k, to_str(options[k]))
return self
[docs] def partitionBy(self, *cols):
"""Partitions the output by the given columns on the file system.
If specified, the output is laid out on the file system similar
to Hive's partitioning scheme.
.. versionadded:: 2.0.0
Parameters
----------
cols : str or list
name of columns
Notes
-----
This API is evolving.
"""
if len(cols) == 1 and isinstance(cols[0], (list, tuple)):
cols = cols[0]
self._jwrite = self._jwrite.partitionBy(_to_seq(self._spark._sc, cols))
return self
[docs] def queryName(self, queryName):
"""Specifies the name of the :class:`StreamingQuery` that can be started with
:func:`start`. This name must be unique among all the currently active queries
in the associated SparkSession.
.. versionadded:: 2.0.0
Parameters
----------
queryName : str
unique name for the query
Notes
-----
This API is evolving.
Examples
--------
>>> writer = sdf.writeStream.queryName('streaming_query')
"""
if not queryName or type(queryName) != str or len(queryName.strip()) == 0:
raise ValueError('The queryName must be a non-empty string. Got: %s' % queryName)
self._jwrite = self._jwrite.queryName(queryName)
return self
[docs] @keyword_only
def trigger(self, *, processingTime=None, once=None, continuous=None):
"""Set the trigger for the stream query. If this is not set it will run the query as fast
as possible, which is equivalent to setting the trigger to ``processingTime='0 seconds'``.
.. versionadded:: 2.0.0
Parameters
----------
processingTime : str, optional
a processing time interval as a string, e.g. '5 seconds', '1 minute'.
Set a trigger that runs a microbatch query periodically based on the
processing time. Only one trigger can be set.
once : bool, optional
if set to True, set a trigger that processes only one batch of data in a
streaming query then terminates the query. Only one trigger can be set.
continuous : str, optional
a time interval as a string, e.g. '5 seconds', '1 minute'.
Set a trigger that runs a continuous query with a given checkpoint
interval. Only one trigger can be set.
Notes
-----
This API is evolving.
Examples
--------
>>> # trigger the query for execution every 5 seconds
>>> writer = sdf.writeStream.trigger(processingTime='5 seconds')
>>> # trigger the query for just once batch of data
>>> writer = sdf.writeStream.trigger(once=True)
>>> # trigger the query for execution every 5 seconds
>>> writer = sdf.writeStream.trigger(continuous='5 seconds')
"""
params = [processingTime, once, continuous]
if params.count(None) == 3:
raise ValueError('No trigger provided')
elif params.count(None) < 2:
raise ValueError('Multiple triggers not allowed.')
jTrigger = None
if processingTime is not None:
if type(processingTime) != str or len(processingTime.strip()) == 0:
raise ValueError('Value for processingTime must be a non empty string. Got: %s' %
processingTime)
interval = processingTime.strip()
jTrigger = self._spark._sc._jvm.org.apache.spark.sql.streaming.Trigger.ProcessingTime(
interval)
elif once is not None:
if once is not True:
raise ValueError('Value for once must be True. Got: %s' % once)
jTrigger = self._spark._sc._jvm.org.apache.spark.sql.streaming.Trigger.Once()
else:
if type(continuous) != str or len(continuous.strip()) == 0:
raise ValueError('Value for continuous must be a non empty string. Got: %s' %
continuous)
interval = continuous.strip()
jTrigger = self._spark._sc._jvm.org.apache.spark.sql.streaming.Trigger.Continuous(
interval)
self._jwrite = self._jwrite.trigger(jTrigger)
return self
[docs] def foreach(self, f):
"""
Sets the output of the streaming query to be processed using the provided writer ``f``.
This is often used to write the output of a streaming query to arbitrary storage systems.
The processing logic can be specified in two ways.
#. A **function** that takes a row as input.
This is a simple way to express your processing logic. Note that this does
not allow you to deduplicate generated data when failures cause reprocessing of
some input data. That would require you to specify the processing logic in the next
way.
#. An **object** with a ``process`` method and optional ``open`` and ``close`` methods.
The object can have the following methods.
* ``open(partition_id, epoch_id)``: *Optional* method that initializes the processing
(for example, open a connection, start a transaction, etc). Additionally, you can
use the `partition_id` and `epoch_id` to deduplicate regenerated data
(discussed later).
* ``process(row)``: *Non-optional* method that processes each :class:`Row`.
* ``close(error)``: *Optional* method that finalizes and cleans up (for example,
close connection, commit transaction, etc.) after all rows have been processed.
The object will be used by Spark in the following way.
* A single copy of this object is responsible of all the data generated by a
single task in a query. In other words, one instance is responsible for
processing one partition of the data generated in a distributed manner.
* This object must be serializable because each task will get a fresh
serialized-deserialized copy of the provided object. Hence, it is strongly
recommended that any initialization for writing data (e.g. opening a
connection or starting a transaction) is done after the `open(...)`
method has been called, which signifies that the task is ready to generate data.
* The lifecycle of the methods are as follows.
For each partition with ``partition_id``:
... For each batch/epoch of streaming data with ``epoch_id``:
....... Method ``open(partitionId, epochId)`` is called.
....... If ``open(...)`` returns true, for each row in the partition and
batch/epoch, method ``process(row)`` is called.
....... Method ``close(errorOrNull)`` is called with error (if any) seen while
processing rows.
Important points to note:
* The `partitionId` and `epochId` can be used to deduplicate generated data when
failures cause reprocessing of some input data. This depends on the execution
mode of the query. If the streaming query is being executed in the micro-batch
mode, then every partition represented by a unique tuple (partition_id, epoch_id)
is guaranteed to have the same data. Hence, (partition_id, epoch_id) can be used
to deduplicate and/or transactionally commit data and achieve exactly-once
guarantees. However, if the streaming query is being executed in the continuous
mode, then this guarantee does not hold and therefore should not be used for
deduplication.
* The ``close()`` method (if exists) will be called if `open()` method exists and
returns successfully (irrespective of the return value), except if the Python
crashes in the middle.
.. versionadded:: 2.4.0
Notes
-----
This API is evolving.
Examples
--------
>>> # Print every row using a function
>>> def print_row(row):
... print(row)
...
>>> writer = sdf.writeStream.foreach(print_row)
>>> # Print every row using a object with process() method
>>> class RowPrinter:
... def open(self, partition_id, epoch_id):
... print("Opened %d, %d" % (partition_id, epoch_id))
... return True
... def process(self, row):
... print(row)
... def close(self, error):
... print("Closed with error: %s" % str(error))
...
>>> writer = sdf.writeStream.foreach(RowPrinter())
"""
from pyspark.rdd import _wrap_function
from pyspark.serializers import PickleSerializer, AutoBatchedSerializer
from pyspark.taskcontext import TaskContext
if callable(f):
# The provided object is a callable function that is supposed to be called on each row.
# Construct a function that takes an iterator and calls the provided function on each
# row.
def func_without_process(_, iterator):
for x in iterator:
f(x)
return iter([])
func = func_without_process
else:
# The provided object is not a callable function. Then it is expected to have a
# 'process(row)' method, and optional 'open(partition_id, epoch_id)' and
# 'close(error)' methods.
if not hasattr(f, 'process'):
raise AttributeError("Provided object does not have a 'process' method")
if not callable(getattr(f, 'process')):
raise TypeError("Attribute 'process' in provided object is not callable")
def doesMethodExist(method_name):
exists = hasattr(f, method_name)
if exists and not callable(getattr(f, method_name)):
raise TypeError(
"Attribute '%s' in provided object is not callable" % method_name)
return exists
open_exists = doesMethodExist('open')
close_exists = doesMethodExist('close')
def func_with_open_process_close(partition_id, iterator):
epoch_id = TaskContext.get().getLocalProperty('streaming.sql.batchId')
if epoch_id:
epoch_id = int(epoch_id)
else:
raise RuntimeError("Could not get batch id from TaskContext")
# Check if the data should be processed
should_process = True
if open_exists:
should_process = f.open(partition_id, epoch_id)
error = None
try:
if should_process:
for x in iterator:
f.process(x)
except Exception as ex:
error = ex
finally:
if close_exists:
f.close(error)
if error:
raise error
return iter([])
func = func_with_open_process_close
serializer = AutoBatchedSerializer(PickleSerializer())
wrapped_func = _wrap_function(self._spark._sc, func, serializer, serializer)
jForeachWriter = \
self._spark._sc._jvm.org.apache.spark.sql.execution.python.PythonForeachWriter(
wrapped_func, self._df._jdf.schema())
self._jwrite.foreach(jForeachWriter)
return self
[docs] def foreachBatch(self, func):
"""
Sets the output of the streaming query to be processed using the provided
function. This is supported only the in the micro-batch execution modes (that is, when the
trigger is not continuous). In every micro-batch, the provided function will be called in
every micro-batch with (i) the output rows as a DataFrame and (ii) the batch identifier.
The batchId can be used deduplicate and transactionally write the output
(that is, the provided Dataset) to external systems. The output DataFrame is guaranteed
to exactly same for the same batchId (assuming all operations are deterministic in the
query).
.. versionadded:: 2.4.0
Notes
-----
This API is evolving.
Examples
--------
>>> def func(batch_df, batch_id):
... batch_df.collect()
...
>>> writer = sdf.writeStream.foreachBatch(func)
"""
from pyspark.java_gateway import ensure_callback_server_started
gw = self._spark._sc._gateway
java_import(gw.jvm, "org.apache.spark.sql.execution.streaming.sources.*")
wrapped_func = ForeachBatchFunction(self._spark, func)
gw.jvm.PythonForeachBatchHelper.callForeachBatch(self._jwrite, wrapped_func)
ensure_callback_server_started(gw)
return self
[docs] def start(self, path=None, format=None, outputMode=None, partitionBy=None, queryName=None,
**options):
"""Streams the contents of the :class:`DataFrame` to a data source.
The data source is specified by the ``format`` and a set of ``options``.
If ``format`` is not specified, the default data source configured by
``spark.sql.sources.default`` will be used.
.. versionadded:: 2.0.0
Parameters
----------
path : str, optional
the path in a Hadoop supported file system
format : str, optional
the format used to save
outputMode : str, optional
specifies how data of a streaming DataFrame/Dataset is written to a
streaming sink.
* `append`: Only the new rows in the streaming DataFrame/Dataset will be written to the
sink
* `complete`: All the rows in the streaming DataFrame/Dataset will be written to the
sink every time these are some updates
* `update`: only the rows that were updated in the streaming DataFrame/Dataset will be
written to the sink every time there are some updates. If the query doesn't contain
aggregations, it will be equivalent to `append` mode.
partitionBy : str or list, optional
names of partitioning columns
queryName : str, optional
unique name for the query
**options : dict
All other string options. You may want to provide a `checkpointLocation`
for most streams, however it is not required for a `memory` stream.
Notes
-----
This API is evolving.
Examples
--------
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()
>>> sq.isActive
True
>>> sq.name
'this_query'
>>> sq.stop()
>>> sq.isActive
False
>>> sq = sdf.writeStream.trigger(processingTime='5 seconds').start(
... queryName='that_query', outputMode="append", format='memory')
>>> sq.name
'that_query'
>>> sq.isActive
True
>>> sq.stop()
"""
self.options(**options)
if outputMode is not None:
self.outputMode(outputMode)
if partitionBy is not None:
self.partitionBy(partitionBy)
if format is not None:
self.format(format)
if queryName is not None:
self.queryName(queryName)
if path is None:
return self._sq(self._jwrite.start())
else:
return self._sq(self._jwrite.start(path))
def toTable(self, tableName, format=None, outputMode=None, partitionBy=None, queryName=None,
**options):
"""
Starts the execution of the streaming query, which will continually output results to the
given table as new data arrives.
The returned :class:`StreamingQuery` object can be used to interact with the stream.
.. versionadded:: 3.1.0
Parameters
----------
tableName : str
string, for the name of the table.
format : str, optional
the format used to save.
outputMode : str, optional
specifies how data of a streaming DataFrame/Dataset is written to a
streaming sink.
* `append`: Only the new rows in the streaming DataFrame/Dataset will be written to the
sink
* `complete`: All the rows in the streaming DataFrame/Dataset will be written to the
sink every time these are some updates
* `update`: only the rows that were updated in the streaming DataFrame/Dataset will be
written to the sink every time there are some updates. If the query doesn't contain
aggregations, it will be equivalent to `append` mode.
partitionBy : str or list, optional
names of partitioning columns
queryName : str, optional
unique name for the query
**options : dict
All other string options. You may want to provide a `checkpointLocation`.
Notes
-----
This API is evolving.
For v1 table, partitioning columns provided by `partitionBy` will be respected no matter
the table exists or not. A new table will be created if the table not exists.
For v2 table, `partitionBy` will be ignored if the table already exists. `partitionBy` will
be respected only if the v2 table does not exist. Besides, the v2 table created by this API
lacks some functionalities (e.g., customized properties, options, and serde info). If you
need them, please create the v2 table manually before the execution to avoid creating a
table with incomplete information.
Examples
--------
>>> sdf.writeStream.format('parquet').queryName('query').toTable('output_table')
... # doctest: +SKIP
>>> sdf.writeStream.trigger(processingTime='5 seconds').toTable(
... 'output_table',
... queryName='that_query',
... outputMode="append",
... format='parquet',
... checkpointLocation='/tmp/checkpoint') # doctest: +SKIP
"""
self.options(**options)
if outputMode is not None:
self.outputMode(outputMode)
if partitionBy is not None:
self.partitionBy(partitionBy)
if format is not None:
self.format(format)
if queryName is not None:
self.queryName(queryName)
return self._sq(self._jwrite.toTable(tableName))
def _test():
import doctest
import os
import tempfile
from pyspark.sql import SparkSession, SQLContext
import pyspark.sql.streaming
os.chdir(os.environ["SPARK_HOME"])
globs = pyspark.sql.streaming.__dict__.copy()
try:
spark = SparkSession.builder.getOrCreate()
except py4j.protocol.Py4JError: # noqa: F821
spark = SparkSession(sc) # noqa: F821
globs['tempfile'] = tempfile
globs['os'] = os
globs['spark'] = spark
globs['sqlContext'] = SQLContext.getOrCreate(spark.sparkContext)
globs['sdf'] = \
spark.readStream.format('text').load('python/test_support/sql/streaming')
globs['sdf_schema'] = StructType([StructField("data", StringType(), True)])
globs['df'] = \
globs['spark'].readStream.format('text').load('python/test_support/sql/streaming')
(failure_count, test_count) = doctest.testmod(
pyspark.sql.streaming, globs=globs,
optionflags=doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE | doctest.REPORT_NDIFF)
globs['spark'].stop()
if failure_count:
sys.exit(-1)
if __name__ == "__main__":
_test()


Code by Shai Sachs at https://shaisachs.github.io/ecodays/
Background image courtesy of Annie Spratt on Unsplash


Never let your plants dry out.
Get a reminder sent to your phone when it's time to water your plants.
Always know when you need to water your plants.
Never worry about your plant's needs again.
Share your latest flower pictures.
Post updates to your favorite social networks, Facebook, Instagram and more.
A well-watered plant is a happy plant! We help keep your plants happy and healthy.
Consectetur adipiscing elit duis tristique sollicitudin. A diam maecenas sed enim ut sem. Massa sed elementum tempus egestas sed. Quam adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus. Lorem ipsum dolor sit amet consectetur. Amet risus nullam eget felis eget nunc lobortis.


Deployment¶
Congress has two modes for deployment: single-process and multi-process. If you are interested in test-driving Congress or are not concerned about high-availability, the single-process deployment is best because it is easiest to set up. If you are interested in making Congress highly-available you want the multi-process deployment.
In the single-process version, you run Congress as a single operating-system process on one node (i.e. container, VM, physical machine).
In the multi-process version, you start with the 3 components of Congress (the API, the policy engine, and the datasource drivers). You choose how many copies of each component you want to run, how you want to distribute those components across processes, and how you want to distribute those processes across nodes.
Section Configuration Options describes the common configuration options for both single-process and multi-process deployments. After that HA Overview and HA Deployment describe how to set up the multi-process deployment.
Configuration Options¶
In this section we highlight the configuration options that are specific to Congress. To generate a sample configuration file that lists all available options, along with descriptions, run the following commands:
$ cd /path/to/congress
$ tox -egenconfig
The tox command will create the file etc/congress.conf.sample
, which has
a comprehensive list of options. All options have default values, which
means that even if you specify no options Congress will run.
The options most important to Congress are described below, all of which appear under the [DEFAULT] section of the configuration file.
drivers
- The list of permitted datasource drivers. Default is the empty list. The list is a comma separated list of Python class paths. For example: drivers = congress.datasources.neutronv2_driver.NeutronV2Driver,congress.datasources.glancev2_driver.GlanceV2Driver
datasource_sync_period
- The number of seconds to wait between synchronizing datasource config from the database. Default is 0.
enable_execute_action
- Whether or not congress will execute actions. If false, Congress will never execute any actions to do manual reactive enforcement, even if there are policy statements that say actions should be executed and the conditions of those actions become true. Default is True.
One of Congress’s new experimental features is distributing its services across multiple services and even hosts. Here are the options for using that feature.
bus_id
- Unique ID of DSE bus. Can be any string. Defaults to ‘bus’. ID should be same across all the processes of a single congress instance and should be unique across different congress instances. Used if you want to create multiple, distributed instances of Congress and can be ignored if only one congress instance is deployed as single process in rabbitMQ cluster. Appears in the [dse] section.
Here are the most often-used, but standard OpenStack options. These are specified in the [DEFAULT] section of the configuration file.
auth_strategy
- Method for authenticating Congress users. Can be assigned to either ‘keystone’ meaning that the user must provide Keystone credentials or to ‘noauth’ meaning that no authentication is required. Default is ‘keystone’.
verbose
- Controls whether the INFO-level of logging is enabled. If false, logging level will be set to WARNING. Default is true. Deprecated.
debug
- Whether or not the DEBUG-level of logging is enabled. Default is false.
transport_url
- URL to the shared messaging service. It is not needed in a single-process Congress deployment, but must be specified in a multi-process Congress deployment.
[DEFAULT]
transport_url = rabbit://<rabbit-userid>:<rabbit-password>@<rabbit-host-address>:<port>
HA Overview¶
Some applications require Congress to be highly available. Some applications require a Congress Policy Engine (PE) to handle a high volume of queries. This guide describes Congress support for High Availability (HA) High Throughput (HT) deployment.
Please see the OpenStack High Availability Guide for details on how to install and configure OpenStack for High Availability.
HA Types¶
Warm Standby¶
Warm Standby is when a software component is installed and available on the secondary node. The secondary node is up and running. In the case of a failure on the primary node, the software component is started on the secondary node. This process is usually automated using a cluster manager. Data is regularly mirrored to the secondary system using disk based replication or shared disk. This generally provides a recovery time of a few minutes.
Active-Active (Load-Balanced)¶
In this method, both the primary and secondary systems are active and processing requests in parallel. Data replication happens through software capabilities and would be bi-directional. This generally provides a recovery time that is instantaneous.
Congress HAHT¶
Congress provides Active-Active for the Policy Engine and Warm Standby for the Datasource Drivers.
Run N instances of the Congress Policy Engine in active-active configuration, so both the primary and secondary systems are active and processing requests in parallel.
One Datasource Driver (DSD) per physical datasource, publishing data on oslo-messaging to all policy engines.
+-------------------------------------+ +--------------+
| Load Balancer (eg. HAProxy) | <----+ Push client |
+----+-------------+-------------+----+ +--------------+
| | |
PE | PE | PE | all+DSDs node
+---------+ +---------+ +---------+ +-----------------+
| +-----+ | | +-----+ | | +-----+ | | +-----+ +-----+ |
| | API | | | | API | | | | API | | | | DSD | | DSD | |
| +-----+ | | +-----+ | | +-----+ | | +-----+ +-----+ |
| +-----+ | | +-----+ | | +-----+ | | +-----+ +-----+ |
| | PE | | | | PE | | | | PE | | | | DSD | | DSD | |
| +-----+ | | +-----+ | | +-----+ | | +-----+ +-----+ |
+---------+ +---------+ +---------+ +--------+--------+
| | | |
| | | |
+--+----------+-------------+--------+--------+
| |
| |
+-------+----+ +------------------------+-----------------+
| Oslo Msg | | DBs (policy, config, push data, exec log)|
+------------+ +------------------------------------------+
Details¶
- Datasource Drivers (DSDs):
- One datasource driver per physical datasource
- All DSDs run in a single DSE node (process)
- Push DSDs: optionally persist data in push data DB, so a new snapshot can be obtained whenever needed
- Warm Standby:
- Only one set of DSDs running at a given time; backup instances ready to launch
- For pull DSDs, warm standby is most appropriate because warm startup time is low (seconds) relative to frequency of data pulls
- For push DSDs, warm standby is generally sufficient except for use cases that demand sub-second latency even during a failover
- Policy Engine (PE):
- Replicate policy engine in active-active configuration.
- Policy synchronized across PE instances via Policy DB
- Every instance subscribes to the same data on oslo-messaging
- Reactive Enforcement:
All PE instances initiate reactive policy actions, but each DSD locally
selects a leader to listen to. The DSD ignores execution requests
initiated by all other PE instances.
- Every PE instance computes the required reactive enforcement actions and initiates the corresponding execution requests over oslo-messaging
- Each DSD locally picks a PE instance as leader (say the first instance the DSD hears from in the asymmetric node deployment, or the PE instance on the same node as the DSD in a symmetric node deployment) and executes only requests from that PE
- If heartbeat contact is lost with the leader, the DSD selects a new leader
- Each PE instance is unaware of whether it is a leader
- Node Configurations:
- Congress supports the Two Node-Types (API+PE nodes, all-DSDs) node configuration because it gives reasonable support for high-load DSDs while keeping the deployment complexities low.
- Local Leader for Action Execution:
- Local Leader: every PE instance sends action-execution requests, but each receiving DSD locally picks a “leader” to listen to
- Because there is a single active DSD for a given data source, it is a natural spot to locally choose a “leader” among the PE instances sending reactive enforcement action execution requests. Congress supports the local leader style because it avoids the deployment complexities associated with global leader election. Furthermore, because all PE instances perform reactive enforcement and send action execution requests, the redundancy opens up the possibility for zero disruption to reactive enforcement when a PE instance fails.
- API:
- Each node has an active API service
- Each API service routes requests for the PE to its associated intranode PE
- Requests for any other service (eg. get data source status) are routed to the Datasource and/or Policy Engine, which will be fielded by some active instance of the service on some node
- Load balancer:
- Layer 7 load balancer (e.g. HAProxy) distributes incoming API calls among the nodes (each running an API service).
- load balancer optionally configured to use sticky session to pin each API caller to a particular node. This configuration avoids the experience of going back in time.
- External components (load balancer, DBs, and oslo messaging bus) can be made highly available using standard solutions (e.g. clustered LB, HA rabbitMQ)
Performance Impact¶
- Increased latency due to network communication required by multi-node deployment
- Increased reactive enforcement latency if action executions are persistently logged to facilitate smoother failover
- PE replication can achieve greater query throughput
Cautions and Limitations¶
- Replicated PE deployment is new in the Newton release and a major departure from the previous model. As a result, the deployer may be more likely to experience unexpected issues.
- In the Newton release, creating a new policy requires locking a database table. As a result, it should not be deployed with a database backend that does not support table locking (e.g., Galera). The limitation is expected to be removed in the Ocata release.
- Different PE instances may be out-of-sync in their data and policies (eventual consistency). The issue is generally made transparent to the end user by configuring the load balancer to make each user sticky to a particular PE instance. But if a user reaches a different PE instance (say because of load balancer configuration or because the original instance went down), the end user reaches a different instance and may experience out-of-sync artifacts.
HA Deployment¶
Overview¶
This section shows how to deploy Congress with High Availability (HA). For an architectural overview, please see the HA Overview.
An HA deployment of Congress involves five main steps.
- Deploy messaging and database infrastructure to be shared by all the Congress nodes.
- Prepare the hosts to run Congress nodes.
- Deploy N (at least 2) policy-engine nodes.
- Deploy one datasource-drivers node.
- Deploy a load-balancer to load-balance between the N policy-engine nodes.
The following sections describe each step in more detail.
Hosts Preparation¶
Congress should be installed on each host expected to run a Congress node. Please follow the directions in separate install instructions to install Congress on each host, skipping the local database portion.
In the configuration file, a transport_url
should be specified to use the
RabbitMQ messaging service configured in step 1.
For example:
[DEFAULT]
transport_url = rabbit://<rabbit-userid>:<rabbit-password>@<rabbit-host-address>:5672
In addition, the replicated_policy_engine
option should be set to True
.
[DEFAULT]
replicated_policy_engine = True
All hosts should be configured with a database connection that points to the shared database deployed in step 1, not the local address shown in separate install instructions.
For example:
[database]
connection = mysql+pymysql://root:<database-password>@<shared-database-ip-address>/congress?charset=utf8
Datasource Drivers Node¶
In this step, we deploy a single datasource-drivers node in warm-standby style.
The datasource-drivers node can be started directly with the following command:
$ python /usr/local/bin/congress-server --datasources --node-id=<unique_node_id>
A unique node-id (distinct from all the policy-engine nodes) must be specified.
For warm-standby deployment, an external manager is used to launch and manage the datasource-drivers node. In this document, we sketch how to deploy the datasource-drivers node with Pacemaker .
See the OpenStack High Availability Guide for general usage of Pacemaker and how to deploy Pacemaker cluster stack. The guide also has some HA configuration guidance for other OpenStack projects.
Prepare OCF resource agent¶
You need a custom Resource Agent (RA) for DataSoure Node HA. The custom RA is
located in Congress repository, /path/to/congress/script/ocf/congress-datasource
.
Install the RA with following steps.
$ cd /usr/lib/ocf/resource.d
$ mkdir openstack
$ cd openstack
$ cp /path/to/congress/script/ocf/congress-datasource ./congress-datasource
$ chmod a+rx congress-datasource
Configuring the Resource Agent¶
You can now add the Pacemaker configuration for Congress DataSource Node resource. Connect to the Pacemaker cluster with the crm configure command and add the following cluster resources. After adding the resource make sure commit the change.
primitive ds-node ocf:openstack:congress-datasource \
params config="/etc/congress/congress.conf" \
node_id="datasource-node" \
op monitor interval="30s" timeout="30s"
Make sure that all nodes in the cluster have same config file with same name and
path since DataSource Node resource, ds-node
, uses config file defined at
config parameter to launch the resource.
The RA has following configurable parameters.
- config: a path of Congress’s config file
- node_id(Option): a node id of the datasource node. Default is “datasource-node”.
- binary(Option): a path of Congress binary Default is “/usr/local/bin/congress-server”.
- additional_parameters(Option): additional parameters of congress-server
Policy Engine Nodes¶
In this step, we deploy N (at least 2) policy-engine nodes, each with an associated API server. This step should be done only after the Datasource Drivers Node is deployed. Each node can be started as follows:
$ python /usr/local/bin/congress-server --api --policy-engine --node-id=<unique_node_id>
Each node must have a unique node-id specified as a commandline option.
For high availability, each node is usually deployed on a different host. If
multiple nodes are to be deployed on the same host, each node must have a
different port specified using the bind_port
configuration option in the
congress configuration file.
Load-balancer¶
A load-balancer should be used to distribute incoming API requests to the N policy-engine (and API service) nodes deployed in step 3. It is recommended that a sticky configuration be used to avoid exposing a user to out-of-sync artifacts when the user hits different policy-engine nodes.
HAProxy is a popular load-balancer for this purpose. The HAProxy section of the OpenStack High Availability Guide has instructions for deploying HAProxy for high availability.


WA Police on the hunt for Bentley child sex attacker and warns people to be vigilant
WA Police has stressed it will “leave no stone unturned” in the investigation into the sexual assault of a 12-year-old girl in Bentley.
It is alleged two men with covered faces grabbed the young girl in a park on Mackay Street and sexually assaulted her between 6pm-11pm on Thursday.
Today WA Police Minister Michelle Roberts said the incident was deeply concerning.
“Everyone does need to be vigilant. Know where your children are and take all safety precautions,” she said.
“Of course everyone should be able to walk the streets safely but unfortunately there are predators out there and my advice is take caution. Police will leave no stone unturned in this investigation, they're taking it very seriously.”
Detectives from the Sex Crime Division as well as specialists are involved in the active investigation.
Detective Superintendent in Charge of the Major Crime Division Robert Anderson said he did not believe there was a “massive risk” to the public at this stage, adding the alleged incident appeared to be isolated.
“Perth is an exceptionally safe place to live, yes of course there's always a risk to people,” he said.
“As a parent I would urge all children, all parents to take care and due diligence around your surroundings, but is there a massive risk to the public at this stage? We're looking into that but at this stage I would say no.”
Anyone with information is asked to call Crime Stoppers 131 444.


{{define "Navbar"}}
{{end}} {{define "body"}}Inner Hearth Yoga offers alignment-focused hatha yoga classes in our snug studio nestled in the heart of Point Breeze.
Whether you’re looking for an asana challenge or total rest and relaxation, we’ve got a class that will fit your needs. No matter what class you choose, you’ll get individual attention from our skilled and attentive teachers to help you deepen and focus your practice.
If you want to Stretch and Strengthen, try:
If you want to Rest and Restore, try:
For expecting mothers and new mothers:


While we all want to see cryptocurrencies succeed, many investment experts think that most cryptocurrencies will eventually be completely valueless. Unlike physical assets with intrinsic scarcity or shares of companies or fiat currency with enforced scarcity, anyone can take the source code of an existing cryptocurrency, make some improvements to it, and issue a new cryptocurrency without contributing any value back to the original cryptocurrency.
Of course, as an open-source project, anyone will be able to use the Nyzo source to create a new blockchain, and we think that is a wonderful thing. However, we also want for the Nyzo cryptocurrency to have long-term, sustainable value. While a particular cryptocurrency may not typically have an intrinsic value, the community and the technology surrounding the currency do have such value. We think it is only fair to tie the value of the original currency to the value of the community and the technology, and we think this is the best way to ensure that Nyzo will maintain its value in the long run.
Our proposal is simple: if technological advancements render the current Nyzo system or blockchain obsolete, and a new blockchain needs to be released, all the coins in that system should be derived from original Nyzo coins. There should be no mining and no generation of new coins that are not tied to original Nyzo coins. This does not mean that all future generations of Nyzo will have to have the same number of coins as the current generation. For instance, if the second generation has 200 million coins instead of 100 million, then each first-generation Nyzo coin that is traded in will be rewarded with two second-generation Nyzo coins.
The sustainability program will work as follows. Any coins transferred to address
00000000
The sustainability address (00000000
While this method of transferring assets to the next-generation blockchain does impose an additional implementation complexity on the next-generation blockchain, such is a small price to pay for continuation of blockchain assets beyond obsolescence of the technology that originally created those assets. Applying this transitively, we can imagine a fifteenth generation of Nyzo enabling global commerce 200 years from now, with every last coin in that system having a lineage traceable back to the Genesis block of the original Nyzo blockchain.


 The deal lasts until the end of Black Friday, but you'll have to act fact. The machine usually costs $233 but is down to $100 for the rest of the day.
Custom coffee
Nespresso VertuoPlus Coffee and Espresso Maker
Intelligent blending
This coffee and Espresso maker scans your coffee blends to deliver a customized blending experience.
The Nespresso VertuoPlus Coffee and Espresso maker works with Nespresso capsules, opening your coffee break to a wide variety of blends and flavors. This machine scans each blend's barcode and adjusts the brewing parameters to match each blend. It supports five different cup sizes, so you can create a customized blend that's just the right size for you.
Don't miss out on today's newest and best Black Friday deals
Amazon's Fire Tablet is the one Black Friday purchase you NEED to make!
This bundle includes both the device and a collection of Nespresso capsules. To make a cup of coffee or Espresso, you just place a capsule in the machine and press a single button.
We may earn a commission for purchases using our links. Learn more.


...
Packages
Standard library
Other packages
Sub-repositories
These packages are part of the Go Project but outside the main Go tree.
They are developed under looser compatibility requirements than the Go core.
Install them with "go get".
- benchmarks — benchmarks to measure Go as it is developed.
- blog — blog.golang.org's implementation.
- build — build.golang.org's implementation.
- crypto — additional cryptography packages.
- debug — an experimental debugger for Go.
- image — additional imaging packages.
- mobile — experimental support for Go on mobile platforms.
- net — additional networking packages.
- sys — packages for making system calls.
- text — packages for working with text.
- tools — godoc, goimports, gorename, and other tools.
- tour — tour.golang.org's implementation.
- exp — experimental and deprecated packages (handle with care; may change without warning).
These services can help you find Open Source packages provided by the community.


The Checker has passed you applicant,The draft certificate had been generated
The formal certificate is issued and its hash has been merged to the blockchain transaction
Each auditor has checked and signed for blockchain transaction which contains the your certificate.
the certificates was published to peer to peer network successfully. It will valid after miners' confirmation
Loading


Login / Sign Up
New Delhi
Gurugram
Lucknow
Jaipur
Hyderabad
Bangalore
Pune
Mumbai
Chennai
Kolkata
Chandigarh
Indore
Laundry
Horologist
Saloon/Spa
Fitness
Cook
Recommended Services
Home Decor
Fitness Trainer
Doctor at Home
Popular Search
Dance Jockey
Carpenter
Painter
Keep in touch
© Copyright NeedIt 2018. Template Design by Piyush Sahay


CBOR converter
Input
JSON
hex
base64
→→→ to →→→
{ "type": "here" }
Output
commented
diagnostic
hex
base64
js
JSON
Copy to input
Powered by
node-cbor
.
Source


|
|
||||||
Our Clients |
| New York Life | ||
| We've written and produced a wide range of programming for their diverse financial offerings. |
| Pfizer | ||
| We worked with Pfizer for a number of years on many global webcasts and on marketing videos. |
| Early Advantage | ||
| We produce their home-based product, do commercial photography and create on-line movies to increase their reach and appeal to the youth market. |
| GE | ||
| We have produced many movies for GE that explain and amplify their on-line offerings. |
| Sony Pictures | ||
| We shoot Super 35mm movie footage for their feature projects including special effect matte shots. |
| ABC | ||
| ABC Sports called on us to shoot their commercial when they wanted to extend their brand to a line of clothing and other merchandise. |
| USA Network | ||
| USA Network calls on us then they need motion picture photography on projects such as Murder in Greenwich. |
| PBS | ||
| We produce programming for PBS including the award-winning film Wisdom for Life, which expains the West's tradition of reason and how this contrasts with fundamentalism. |
| QVC Network | ||
| When QVC needed a commercial campaign to drive sales with their top jewelry pitcher, with called on us to film their campaign in Bevery Hills. |
|
|
|
|
|||||
|
© Copyright Brand Applause 2007. All rights reserved.
|


Introducing Lens.js
I've been in the web game for quite a while. Started in grade school on Angelfire, moved up from HTML 2.0, through 3.2 and the dHTML days. I learned Javascript. I learned CSS 2.0, then 2.1, then 3.0
I survived the browser wars.
I've been writing the same Javascript over and over again, from project to project. It's wasteful, and it's sad, but no more. Now it's time to unleash lens.js.
There are lots of frameworks out there, but too many of them assume you don't know what you are doing and attempt to isolate you from the browser. They erect a big old software wall and say you can play in here, but not out there..
I want a framework that lets me experiment, to try out new things.
To that end, lens.js is built to:
- stay out of the way
- help, not hinder
- provide small, sharp tools
If you want to get involved, head over to Github and check out the code.
To get started using lens.js, check out the documentation.


But, not all calculators use ‘E’ to describe very large or very small numbers. Desmos provides an online scientific calculator 1 that anyone can use. If you try to add, subtract, multiply, or divide two numbers with more than 9 digits, Desmos will automatically write them in scientific notation. Instead of using the capital E, Desmos writes scientific notation the same way that we've seen.
in-context


Culture
Variety concert in St Cronan’s Church, Roscrea
Proceeds will go to the Peter McVerry Trust and to church funds, north Tipperary circuit of the Methodist Church
by Eoin KelleherHas Tipperary got talent?
Indeed it has, and those who attended the recent variety concert in St Cronan’s Church of Ireland, Roscrea, saw and heard ample evidence of this.
Wonderful foot-tapping music from Roscrea Comhaltas Ceoltoirí lifted the hearts of the large audience.
Carol Thompson, the leader, noted that performers ranged in age from 7 to 77, with representation from all parts of the community. A highlight was the solo from Tadhg Maher. The Cloughjordan Community Choir, under the hand of John Armitage, featured in both parts of the programme, bringing sing-along songs from yesteryear, together with American spirituals and an amusing song about a cat and mouse!
Mellow Tones, conducted by Liz Powell, warmed the hearts of people from southern Africa, with a song from their part of the world. We were reminded about the major issue of the twenty-first century, climate change, as Liz coached us to join in singing a piece on this theme.
Musical offering were interspersed with dialogues from Play It By Ear, a duo who had people thinking about the meaning of Christmas.
Proceeds will go to the Peter McVerry Trust and to church funds (north Tipperary circuit of the Methodist Church).


Open Issues
warning
Bugs To be Disscused
list of All bugs that are Reported Tbd or Nab(Not a Bug)
IssueId
Project Name
Title
Assign to
Status
More info..
{{value}}
priority_high
More Info
close
close bug
check
Solved
warning
reassigned
warning
Solved


Package imagingbook.calibration.zhang
Class NonlinearOptimizerAnalytic
- java.lang.Object
-
- imagingbook.calibration.zhang.NonlinearOptimizer
-
- imagingbook.calibration.zhang.NonlinearOptimizerAnalytic
-
public class NonlinearOptimizerAnalytic extends NonlinearOptimizer
Nonlinear optimizer based on the Levenberg-Marquart method, where the Jacobian matrix is calculated from analytic expressions (obtained with MatLab). Part of this implementation is based on code from OpenIMAJ by J. Hare: Copyright (c) 2011, The University of Southampton and the individual contributors. All rights reserved (see https://github.com/openimaj/openimaj/).
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class imagingbook.calibration.zhang.NonlinearOptimizer
NonlinearOptimizer.ValueFun
-
-
Field Summary
-
Fields inherited from class imagingbook.calibration.zhang.NonlinearOptimizer
camParLength, M, modelPts, N, obsPts, viewParLength
-
-
Constructor Summary
Constructors Modifier Constructor Description protected
NonlinearOptimizerAnalytic(java.awt.geom.Point2D[] modelPts, java.awt.geom.Point2D[][] obsPts)
-
Method Summary
-
Methods inherited from class imagingbook.calibration.zhang.NonlinearOptimizer
getFinalCamera, getFinalViews, makeObservedVector, optimize
-
-
-
-
Constructor Detail
-
NonlinearOptimizerAnalytic
protected NonlinearOptimizerAnalytic(java.awt.geom.Point2D[] modelPts, java.awt.geom.Point2D[][] obsPts)
-
-


On 21 November 2019, 9 new articles were published in 7 journals.
Gun Violence in the United States in 2017 and the Role of Mental Illness
Revisiting the Relationship Between Corruption and Innovation in Developing and Emerging Economies
Recent Legal Developments: Criminal Justice Decisions of the U.s. Supreme Court, 2018 Term
Online Deception and Situations Conducive to the Progression of Non-Payment Fraud
The Risk and Needs of Cyber-Dependent Offenders Sentenced in the United States
Grade-Level Distinctions in Student Threats of Violence
Small Wars, Big Data: the Information Revolution in Modern Conflict


WQMY
Christmas Village to be Nov. 9-11 at State Fairgrounds (10/18/2012)
Oct 18, 2012 . The 52nd annual Christmas Village will be held Nov. 9-11 at the Tennessee State Fairgrounds. More than 250 merchants will offer a variety of .
http://www.mc.vanderbilt.edu/myvumc/?article=13396
| blood pressure measurement listen nurse/metlife 505 independence parkway chesapeake va/ sky tv broadband phone |
|
|---|---|
| speedup windowsxp startup annoyances | Williamsport |
| Branding | MyNetworkTV christmas village state fairgrounds tennessee |
| dominican ladies in new york | free bb screen lock: 29 (stop pretending by sona somes) & ideas working with children-DT 45.3 (UHF) |
| nfl quarterbacks 6 1 or shorter | see article |
| Owner | New Age Media, LLC Nashville Expo Center |
| First air date | December 30, 1988 |
| asus p5e-vm do lga775 matx motherboard | MYNetworkTV |
| examples of repetition in literature | WOLF-TV, air bag light hyundai |
| peter brown west virgina | WILF (1988-2006) |
| Former channel number(s) | Annual Christmas Village Brings Holiday Shopping Extravaganza ... |
| height information on landmarks | one-two go flight attendants (1988-1998) where is vanuatu in the pacific / campaign david vitter aegis higher ground (1998-2006) |
| first care family physicians fort wayne | 50 gas tank of 2003 pt cruiser 420 kW (WOLF-DT3) |
| simple network two computers | 243 m 488 m (WOLF-DT3) |
| horoscope for taurus 2010 | 52075 73375 (WOLF-DT3) |
| Transmitter coordinates | cat litter dust free saint clair county high school (WOLF-DT3) |
nashville's 51st annual christmas village to take place november 10 ...
51st annual Christmas Village, a shopping tradition in Middle Tennessee. In the heart of. Nashville, the Tennessee State Fairgrounds are transformed into a .
http://www.christmasvillage.org/getattachment/a59c3823-32b6-4c5e-af4a-e442e25e92da/CHRISTMAS_VILLAGE_Release_2011.pdf.aspx
TicketsNashville.com - Going Local ! - Christmas Village
TN State Fairgrounds 500 Wedgewood Avenue Nashville, TN 37203. Directions Click here to visit the TN State Fairgrounds website for detailed directions.
http://www.ticketsnashville.com/websales/pages/VenueListPage.aspx?rguid=261c94f6-ab10-4f2b-85eb-e1c5c05af712&
Contents |
[south american noni juice] History
Nashville Christmas Village Shopping Information
It's beginning to look a lot like Christmas at the Tennessee State Fairgrounds in . Just a short distance from downtown Nashville, Christmas Village is fun, .
http://www.vacationsmadeeasy.com/NashvilleTN/articles/NashvilleChristmasVillageShoppingInformation.cfm
Christmas Village - NowPlayingNashville.com
Nashville's oldest and most prestigious consumer show with more than 250 merchants and 25000 shoppers from all over the U.S. participate in this premier .
http://www.nowplayingnashville.com/event/detail/441620195/Christmas_Village
Tennessee State Fairgrounds | NowPlayingNashville.com
For nearly a century, the Tennessee State Fairgrounds has served as the host . More than one million people come to the Tennessee State Fairgrounds each .
http://www.nowplayingnashville.com/venue/detail/30666/Tennessee_State_Fairgrounds
Tickets & DIrections | Christmas Village Details | Nashville, TN
Learn more about ticket prices and get directions to Christmas Village 2010. . See Location Map below or the TN State Fairgrounds Website . See Site Map for .
http://www.christmasvillage.org/Tickets_Directions
Christmas Village | Christmas Shopping Event | Nashville, TN
Christmas Village, which will be held November 9-11, 2012 at the Tennessee State Fairgrounds, is one of the oldest charitable events in Nashville. Merchants .
http://www.christmasvillage.org/
Nashville Expo Center - Events
Events. The Tennessee State Fairgrounds welcomes over 277 events annually. We are home to the prestigious Christmas Village, Hunter's Custom Auto Show, .
http://www.nashvilleexpocenter.org/expo/events/index.asp
Christmas Village - NowPlayingNashville.com
Christmas Village 2011 marks the 51st year of this annual event. It was created in 1961 by the Nashville Pi Beta Phi Alumnae Club to raise money for its .
http://www.nowplayingnashville.com/event/detail/441342633/Christmas_Village
Christmas Village at Tennessee State Fairgrounds and Expo Center ...
Nov 11, 2012 . More than 260 merchants from all over the U.S. participate in this premier consumer show. A variety of unique seasonal and gift items are .
http://www.eventcrazy.com/Nashville-TN/events/details/352401-Christmas-Village
Nashville, TN Pi Beta Phi Alumnae Club Website | christmas-village
Today Christmas Village is a four-day event and is recognized as a premier consumer show in the southeast. Housed at the Tennessee State Fairgrounds in six .
http://www.nashvillepibetaphialumnae.org/#!__christmas-village
Christmas Village - Nashville, Tennessee - Yahoo! Travel
Aug 5, 2010 . Christmas Village at the Tennessee State Fairgrounds is one of the city's oldest holiday traditions. Held each year ... More. Christmas Village .
http://travel.yahoo.com/p-travelguide-2741398-christmas_village_nashville-i
257. Christmas Village | 365 Days of Nashville - Music City
Nov 8, 2010 . http://www.christmasvillage.org : TN State Fairgrounds : 500 Wedgewood Avenue : Nashville, TN 37203. Christmas Village 2010 .
http://365nashville.com/2010/11/08/257-christmas-village/
[sch nste arsch welt] Digital channels
| Channel | Name | peavey cl-2 cluster pa speakers | facts on sir francis drake | Programming |
|---|---|---|---|---|
| 53.1 | christmas village state fairgrounds tennessee DT | alberta treasury b ranch | france units for middle school students | Main christmas village state fairgrounds tennessee programming / MyNetworkTV |
| 53.2 | WOLF DT | what's up with danity kane | cody arens dragon wars | Simulcast of family health plan cost |
| 53.3 | WSWB DT | 480i | 4:3 | Simulcast of campaign david vitter aegis higher ground |
Tennessee State Fairgrounds Master Plan Phase 1 - Nashville.gov
Feb 18, 2013 . Tennessee State Fair. Nashville Flea Market. Fairgrounds Speedway Nashville. Music City Brawl Stars Roller Derby. Christmas Village .
http://www.nashville.gov/Portals/0/SiteContent/Planning/docs/fairgrounds/TN%20State%20Fairgrounds%20Phase%201%20Master%20Plan%20FINAL%202013_02_18.pdf
[speedup windowsxp startup annoyances] References
[swimming costume size 20] External links
- nfl quarterbacks 6 1 or shorter
- cat litter dust free
- dual path ground drive
- saint row online game
- dundee real estate investment trust
- citybuilding games powered by vbulletin
|
|


TwilioCarrierCallerType
public enum TwilioCarrierCallerType: String
Available types of carrier caller
-
The carrier caller is not available
Declaration
Swift
case Unavailable = "unavailable"
-
The carrier caller is a business number
Declaration
Swift
case Business = "business"
-
The carrier caller is a consumer number
Declaration
Swift
case Consumer = "consumer"


{% include head.html %}
{% include blog/header.html %}
Webdev Notes
Webdev notes and thoughts.
{{ content | checked }}
{% include blog/sidebar.html %}
{% include blog/footer.html %}


Phyloland package models a space colonization process mapped onto a phylogeny, it aims at estimating limited dispersal and ecological competitive exclusion in a Bayesian MCMC statistical phylogeographic framework (please refer to phyloland-package help for details.)
| Version: | 1.3 |
| Imports: | ape |
| Published: | 2014-09-13 |
| Author: | Louis Ranjard, Marie Paturel |
| Maintainer: | Louis Ranjard <l.ranjard at auckland.ac.nz> |
| License: | GPL-2 |
| NeedsCompilation: | yes |
| In views: | Phylogenetics |
| CRAN checks: | phyloland results |
| Reference manual: | phyloland.pdf |
| Package source: | phyloland_1.3.tar.gz |
| Windows binaries: | r-devel: phyloland_1.3.zip, r-release: phyloland_1.3.zip, r-oldrel: phyloland_1.3.zip |
| OS X binaries: | r-release: phyloland_1.3.tgz, r-oldrel: phyloland_1.3.tgz |
| Old sources: | phyloland archive |
Please use the canonical form https://CRAN.R-project.org/package=phyloland to link to this page.


Home
About Us
Contact
Wishing Well
Please check email for new password
Login
Email
Password
Sign Up
Forgot Password


Demonstrates the application of a custom layer.
Description
This example demonstrates how to write a custom layer for tfjs-layers.
We build a custom activation layer called 'Antirectifier' which outputs two
channels for each input, one with just the positive signal, and one with just
the negative signal. The example shows exercises the call
and
computeOutputShape
overides of the Layer
.
Input
Output


THREE.scene
.color
.mesh
Toggle Dropdown
plane
circle
ring
cube
cylinder
sphere
torus
torusknot
tetrahedron
octahedron
dodecahedron
icosahedron
custom text
.quantity
THREE.mesh
.position.x
.position.y
.position.z
.rotation.x
.rotation.y
.rotation.z


{% extends "base.html" %} {% block app %}
If you are not logged in, you might need to be logged in to see this page. If you are logged in, and believe you should have access to this page, contact a website administrator for further support.


Dear CP_NAME,
Thank you for your inquiry:
CP_COMMENTS
We will get back to you within 24 hours. If you want to add any comments, feel free to reply to this email.
Thank you for considering us,
Eduardo
Cooking Point
cookingpoint.es
(+34) 910 115 154


SplitPane
ion-split-pane
ion-split-pane
SplitPane is a component that makes it possible to create multi-view layout. Similar to iPad apps, SplitPane allows UI elements, like Menus, to be displayed as the viewport increases.
If the devices screen size is below a certain size, the SplitPane will collapse and the menu will become hidden again. This is especially useful when creating an app that will be served over a browser or deployed through the app store to phones and tablets.
Usage
To use SplitPane, simply add the component around your root component. In this example, we'll be using a sidemenu layout, similar to what is provided from the sidemenu starter template.
<ion-split-pane>
<!-- our side menu -->
<ion-menu [content]="content">
<ion-header>
<ion-toolbar>
<ion-title>Menu</ion-title>
</ion-toolbar>
</ion-header>
</ion-menu>
<!-- the main content -->
<ion-nav [root]="root" main #content></ion-nav>
</ion-split-pane>
Here, SplitPane will look for the element with the main
attribute and make
that the central component on larger screens. The main
component can be any
Ionic component (ion-nav
or ion-tabs
) except ion-menu
.
Setting breakpoints
By default, SplitPane will expand when the screen is larger than 768px.
If you want to customize this, use the when
input. The when
input can
accept any valid media query, as it uses matchMedia()
underneath.
<ion-split-pane when="(min-width: 475px)">
<!-- our side menu -->
<ion-menu [content]="content">
....
</ion-menu>
<!-- the main content -->
<ion-nav [root]="root" main #content></ion-nav>
</ion-split-pane>
SplitPane also provides some predefined media queries that can be used.
<!-- could be "xs", "sm", "md", "lg", or "xl" -->
<ion-split-pane when="lg">
...
</ion-split-pane>
| Size | Value | Description |
|---|---|---|
xs |
(min-width: 0px) |
Show the split-pane when the min-width is 0px (meaning, always) |
sm |
(min-width: 576px) |
Show the split-pane when the min-width is 576px |
md |
(min-width: 768px) |
Show the split-pane when the min-width is 768px (default break point) |
lg |
(min-width: 992px) |
Show the split-pane when the min-width is 992px |
xl |
(min-width: 1200px) |
Show the split-pane when the min-width is 1200px |
You can also pass in boolean values that will trigger SplitPane when the value or expression evaluates to true.
<ion-split-pane [when]="isLarge">
...
</ion-split-pane>
class MyClass {
public isLarge = false;
constructor(){}
}
Or
<ion-split-pane [when]="shouldShow()">
...
</ion-split-pane>
class MyClass {
constructor(){}
shouldShow(){
if(conditionA){
return true
} else {
return false
}
}
}
Input Properties
| Attr | Type | Details |
|---|---|---|
| enabled | boolean |
If |
| when | string | boolean |
When the split-pane should be shown. Can be a CSS media query expression, or a shortcut expression. Can also be a boolean expression. |
Output Events
| Attr | Details |
|---|---|
| ionChange | Expression to be called when the split-pane visibility has changed |
Sass Variables
| Property | Default | Description |
|---|---|---|
$split-pane-ios-side-min-width |
$split-pane-side-min-width |
Minimum width of the split-pane's side pane |
$split-pane-ios-side-max-width |
$split-pane-side-max-width |
Maximum width of the split-pane's side pane |
$split-pane-ios-border |
$hairlines-width solid $list-ios-border-color |
Border style of the side pane |
| Property | Default | Description |
|---|---|---|
$split-pane-md-side-min-width |
$split-pane-side-min-width |
Minimum width of the split-pane's side pane |
$split-pane-md-side-max-width |
$split-pane-side-max-width |
Maximum width of the split-pane's side pane |
$split-pane-md-border |
1px solid $list-md-border-color |
Border style of the side pane |
| Property | Default | Description |
|---|---|---|
$split-pane-wp-side-min-width |
$split-pane-side-min-width |
Minimum width of the split-pane's side pane |
$split-pane-wp-side-max-width |
$split-pane-side-max-width |
Maximum width of the split-pane's side pane |
$split-pane-wp-border |
1px solid $list-wp-border-color |
Border style of the side pane |


ECB meeting accounts: Plea was made for patience, “Wait and see posture”
by Eren SengezerA plea was made for patience to allow the measures taken in September to work through the economy, supporting a wait and see posture at the current juncture, the accounts of the European Central Bank's (ECB) October policy meeting showed on Thursday.
The shared currency's reaction to the ECB statement has been muted so far with the EUR/USD pair extending its sideways grind below the 1.11 handle.
Key takeaways
"Strong call for unity of the Governing Council was made at the October 23-24 policy meeting."
"Frank discussions are necessary but it's important to form a consensus, unite behind the inflation aim."
"Economic data raised the question as to whether weakness would continue for longer than anticipated in September."
"Wide agreement that more info is needed to reassess inflation outlook, the impact of ECB measures."
"Measures should be allowed more time to unfold; confident they will support inflation."


| download_who {WHOmortality} | R Documentation |
This function downloads the raw WHO mortality data based on the links on the webpage https://www.who.int/healthinfo/statistics/mortality_rawdata/en/
download_who(dest, data = c("docs", "mort", "pop", "ccode", "all"))
dest |
a |
data |
to specify which files to download, takes values of "docs", "mort", "pop", "ccode", "all". Default value downloads the documentation files |
The data
parameter allows the user to download a specific set of data files.
The docs
downloads the documentation files associated with the raw WHO mortality data, this includes the "documentation.zip", "notes.zip", and "availability.zip" files
The mort
downloads the two ICD-10 mortality .zip files, "Morticd10_part1.zip" and "Morticd10_part2.zip"
The pop
downloads the WHO population data, "pop.zip"
The ccode
downloads the country code table "country_codes.zip"
The all
downloads all the abovementioned .zip files at once
the same value as the download.file
## Not run: download_who(dest = getwd(), data = "ccode") ## End(Not run)


I'm A Celeb's Jacqueline Jossa drops Love Island confession as she removes wedding ring
Jacqueline Jossa admitted on last night's I'm A Celebrity that she'd "love" to do Love Island - despite being married to Dan Osborne
by Irishmirror.ieJacqueline Jossa dropped a bombshell on last night's I'm A Celebrity that will no doubt have husband Dan Osborne in a bit of a tizz.
The married ex-EastEnders star gasped excitedly as she and her fellow campmates walked on to the set for the live trial and made her announcement.
Hosts Ant McPartlin and Declan Donnelly had taken a leaf out of ITV's other flagship programme Love Island's book, turning the jungle into a mini villa for the gruelling live task.
And Jacqueline was the most excited to see the Love Island set - complete with hot tub, fire pit and kitchen - practically dancing on as the Caroline Flack-fronted show's theme music blared out.
As she skipped in behind Roman Kemp, Jac shouted out: "WAHOO!"
"Oh my god," she gasped as she took in the surroundings.
"I've always wanted to do Love Island!"
Ant laughed as she filed in behind the other campmates, telling Dec: "She's a big fan!"
Jac's confession seemed to slip out after she notably removed her wedding ring from her left hand.
While some of the celebs appear to be taking off their jewellery to do the tasks, plenty of them are still wearing their wedding bands in camp and in the Bush Telegraph.
She was spotted without the distinctive ring on after her Bushtucker trial with Ian Wright, and again during last night's Love Island segment.
Other stars have kept their wedding rings on inside the camp, including Andy Whyment - who is married to wife Nicola - and Cliff Parisi, who is with BBC producer Tara Wyer.
It comes after Jacqueline's marriage with Dan was rocked in May 2018 when photos emerged of Dan looking incredibly close to Love Island star Gabby Allen on a yacht off the coast of Spain while Jac was heavily pregnant back home in the UK.
She and Dan temporarily split and he moved out of the family home, citing their "constant arguments" and tense atmosphere.
While Dan denied the rumours of an affair and they reconciled, the pair soon saw their marriage rocked again when eight months later rumours emerged that Dan had locked lips with Love Island star Alexandra Cane on a wild night out.
Although Dan rejected the fresh rumours, Gabby waded into the debate in March, telling the Sun that Jac "deserves better" amid the scandal.
Awkwardly, one of Jacqueline's closest friends inside the I'm A Celeb camp is Gabby's ex Myles Stephenson, who called it quits with the Love Island star in August.
A source told the Sun that Myles wouldn't shy away from discussing the Dan/Gabby situation with Jac.
"Gabby and Myles discussed the whole Dan situation during their relationship, and even though both Dan and Gabby always denied anything went on between them, there are a lot of things that Jacqueline might not know," they said.
"Myles would be willing to tell Jacqueline everything - and answer any questions she might have."
Jacqueline and Dan have had many ups and downs in their relationship, with Dan telling new! magazine earlier this year that their marriage was plagued by a "negative energy".
"I just want to be happy and I want Jac to be happy," the ex-TOWIE hunk said at the time, "Whether that's with me or not.
"It's more like a really bad tension in the house and it's getting worse. We're losing our friendship. When I feel it's getting too much, I'll get out of the house and go to the gym or something."
*I'm A Celebrity continues tonight at 9pm on ITV


We know that a suppressor/silencer reduces damage for some weapons, could you include that somewhere in the detailed weapon info?
Topicstarter
Guest
Suppressor doesn't reduce damage just ROF of some weapons (only noticed it on a couple pistols). it MAY reduce damage over long distances though but i am not certain about that.
BrainFooLong
Administrator
In the next update comes more detailed information about each attachment and its impact to each weapon.
Greetz


@*
* Copyright 2023 HM Revenue & Customs
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*@
@import tags._
@import scala.util.Try
@import controllers.auth.requests.UserRequest
@import scala.concurrent.ExecutionContext
@import components.{ErrorSummary, InputRadios, Link, Button}
@this(
main: MainView,
form: FormWithCSRF,
errorSummary: ErrorSummary,
inputRadios: InputRadios,
link: Link,
button: Button
)
@(addressSelectorForm: Form[models.dto.AddressSelectorDto], recordSet: models.addresslookup.RecordSet, typ: AddrType,
postcode: String, filter: Option[String])(implicit request: UserRequest[_], messages: play.api.i18n.Messages)
@title = {
@if(addressSelectorForm.hasErrors){@messages("error.message.error") } @typ.ifIs(
residential = messages("label.select_your_address"),
postal = messages("label.select_an_address")
)
}
@main(
pageTitle = title.toString,
showBackLink = true,
yourProfileActive = true
) {
@errorSummary(addressSelectorForm, Some("radio-0"))
@form(action = controllers.address.routes.AddressSelectorController.onSubmit(typ), args = (Symbol("class")) -> "select-address-form", (Symbol("id")) -> "select-address-form") {
@inputRadios(
form = addressSelectorForm,
legend = typ.ifIs( residential = messages("label.select_your_address"), postal = messages("label.select_an_address")),
name = "addressId",
items = recordSet.addresses.zipWithIndex.map{
case(ad, index) =>
RadioItem(
content = Text(s"""${ad.address.lines.mkString(", ")}, ${ad.address.town.getOrElse("")}, ${ad.address.postcode}"""),
id = Some("radio-" + index),
value = Some(ad.id),
checked = addressSelectorForm("addressId").value.contains(ad.id)
)
}
)
@link(controllers.address.routes.UpdateAddressController.onPageLoad(typ).url, "label.enter_your_address_if_it_is_not_shown")
@button("global.label.continue_button", id="submitAddressSelector")
}
}


Job One
Start Time:
Minutes to Run:
1
2
3
4
5
6
7
8
9
10
15
20
25
30
Days to Run
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
Sunday
Job Two
Start Time:
Minutes to Run:
1
2
3
4
5
6
7
8
9
10
15
20
25
30
Days to Run
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
Sunday


When you click on the upload button, a json file will be uploaded to your bucket. Check the console for the filename which was used.
Further clicks on the upload button will upload data to the same file as the first time.


Huffman Encoding Visualizer
Home
About
Huffman Encoding Visualizer
Download
Upload
Type to see graph...
Type to see bits, or
drop encoded file here
...
Compression Percentage


The World Wide Web Consortium (W3C) is the main international standards organization for the World Wide Web. Founded in 1994 and currently led by Tim Berners-Lee, the consortium is made up of member organizations that maintain full-time staff working together in the development of standards for the World Wide Web. As of 21 October 2019, W3C had 443 members. W3C also engages in education and outreach, develops software, and serves as an open forum for discussion about the Web.
The World Wide Web Consortium (W3C) was founded in 1994 by Tim Berners-Lee after he left the European Organization for Nuclear Research (CERN) in October, 1994. It was founded at the Massachusetts Institute of Technology Laboratory for Computer Science (MIT/LCS) with support from the European Commission, the Defense Advanced Research Projects Agency (DARPA), which had pioneered the ARPANET, one of the predecessors to the Internet. It was located in Technology Square until 2004, when it moved, with CSAIL, to the Stata Center.
The organization tries to foster compatibility and agreement among industry members in the adoption of new standards defined by the W3C. Incompatible versions of HTML are offered by different vendors, causing inconsistency in how web pages are displayed. The consortium tries to get all those vendors to implement a set of core principles and components that are chosen by the consortium.
It was originally intended that CERN host the European branch of W3C; however, CERN wished to focus on particle physics, not information technology. In April 1995, the French Institute for Research in Computer Science and Automation (INRIA) became the European host of W3C, with Keio University Research Institute at SFC (KRIS) becoming the Asian host in September 1996. Starting in 1997, W3C created regional offices around the world. As of September 2009, it had eighteen World Offices covering Australia, the Benelux countries (Netherlands, Luxembourg, and Belgium), Brazil, China, Finland, Germany, Austria, Greece, Hong Kong, Hungary, India, Israel, Italy, South Korea, Morocco, South Africa, Spain, Sweden, and, as of 2016, the United Kingdom and Ireland.
In October 2012, W3C convened a community of major web players and publishers to establish a MediaWiki wiki that seeks to document open web standards called the WebPlatform and WebPlatform Docs.
In January 2013, Beihang University became the Chinese host.
HTML is the language for describing the structure of Web pages. HTML gives authors the means to:
• Publish online documents with headings, text, tables, lists, photos, etc.
• Retrieve online information via hypertext links, at the click of a button.
• Design forms for conducting transactions with remote services, for use in searching for information, making reservations, ordering products, etc.
• Include spread-sheets, video clips, sound clips, and other applications directly in their documents.
HTML is the standard markup language for documents designed to be displayed in a web browser. It can be assisted by technologies such as Cascading Style Sheets (CSS) and scripting languages such as JavaScript.
Web browsers receive HTML documents from a web server or from local storage and render the documents into multimedia web pages. HTML describes the structure of a web page semantically and originally included cues for the appearance of the document.
HTML elements are the building blocks of HTML pages. With HTML constructs, images and other objects such as interactive forms may be embedded into the rendered page. HTML provides a means to create structured documents by denoting structural semantics for text such as headings, paragraphs, lists, links, quotes and other items. HTML elements are delineated by tags, written using angle brackets. Tags such as and directly introduce content into the page. Other tags such as
surround and provide information about document text and may include other tags as sub-elements. Browsers do not display the HTML tags, but use them to interpret the content of the page.
HTML can embed programs written in a scripting language such as JavaScript, which affects the behavior and content of web pages. Inclusion of CSS defines the look and layout of content. The World Wide Web Consortium (W3C), former maintainer of the HTML and current maintainer of the CSS standards, has encouraged the use of CSS over explicit presentational HTML since 1997.
XHTML is a variant of HTML that uses the syntax of XML, the Extensible Markup Language. XHTML has all the same elements (for paragraphs, etc.) as the HTML variant, but the syntax is slightly different. Because XHTML is an XML application, you can use other XML tools with it (such as XSLT, a language for transforming XML content).
Cascading Style Sheets (CSS) is a style sheet language used for describing the presentation of a document written in a markup language such as HTML. CSS is a cornerstone technology of the World Wide Web, alongside HTML and JavaScript.
CSS is designed to enable the separation of presentation and content, including layout, colors, and fonts. This separation can improve content accessibility, provide more flexibility and control in the specification of presentation characteristics, enable multiple web pages to share formatting by specifying the relevant CSS in a separate .css file which reduces complexity and repetition in the structural content as well as enabling the .css file to be cached to improve the page load speed between the pages that share the file and its formatting.
Separation of formatting and content also makes it feasible to present the same markup page in different styles for different rendering methods, such as on-screen, in print, by voice (via speech-based browser or screen reader), and on Braille-based tactile devices. CSS also has rules for alternate formatting if the content is accessed on a mobile device.
The name cascading comes from the specified priority scheme to determine which style rule applies if more than one rule matches a particular element. This cascading priority scheme is predictable.
The CSS specifications are maintained by the World Wide Web Consortium (W3C). Internet media type (MIME type) text/css is registered for use with CSS by RFC 2318 (March 1998). The W3C operates a free CSS validation service for CSS documents..
In addition to HTML, other markup languages support the use of CSS including XHTML, plain XML, SVG, and XUL.
WebFonts is a technology that enables people to use fonts on demand over the Web without requiring installation in the operating system. W3C has experience in downloadable fonts through HTML, CSS2, and SVG. Until recently, downloadable fonts have not been common on the Web due to the lack of an interoperable font format. The WebFonts effort plans to address that through the creation of an industry-supported, open font format for the Web (called "WOFF").