When building Transactional Data Store applications, the architecture decisions involve application startup (running recovery) and handling system or application failure. For details on performing recovery, see the Recovery procedures.
Recovery in a database environment is a single-threaded procedure, that is, one thread of control or process must complete database environment recovery before any other thread of control or process operates in the Berkeley DB environment.
Performing recovery first marks any existing database environment as "failed" and then removes it, causing threads of control running in the database environment to fail and return to the application. This feature allows applications to recover environments without concern for threads of control that might still be running in the removed environment. The subsequent re-creation of the database environment is serialized, so multiple threads of control attempting to create a database environment will serialize behind a single creating thread.
One consideration in removing (as part of recovering) a database environment which may be in use by another thread, is the type of mutex being used by the Berkeley DB library. In the case of database environment failure when using test-and-set mutexes, threads of control waiting on a mutex when the environment is marked "failed" will quickly notice the failure and will return an error from the Berkeley DB API. In the case of environment failure when using blocking mutexes, where the underlying system mutex implementation does not unblock mutex waiters after the thread of control holding the mutex dies, threads waiting on a mutex when an environment is recovered might hang forever. Applications blocked on events (for example, an application blocked on a network socket, or a GUI event) may also fail to notice environment recovery within a reasonable amount of time. Systems with such mutex implementations are rare, but do exist; applications on such systems should use an application architecture where the thread recovering the database environment can explicitly terminate any process using the failed environment, or configure Berkeley DB for test-and-set mutexes, or incorporate some form of long-running timer or watchdog process to wake or kill blocked processes should they block for too long.
Regardless, it makes little sense for multiple threads of control to simultaneously attempt recovery of a database environment, since the last one to run will remove all database environments created by the threads of control that ran before it. However, for some applications, it may make sense for applications to have a single thread of control that performs recovery and then removes the database environment, after which the application launches a number of processes, any of which will create the database environment and continue forward.
There are four ways to architect Berkeley DB Transactional Data Store applications. The one chosen is usually based on whether or not the application is comprised of a single process or group of processes descended from a single process (for example, a server started when the system first boots), or if the application is comprised of unrelated processes (for example, processes started by web connections or users logged into the system).
The first way to architect Transactional Data Store applications is as a single process (the process may or may not be multithreaded.)
When this process starts, it runs recovery on the database environment and then opens its databases. The application can subsequently create new threads as it chooses. Those threads can either share already open Berkeley DB DB_ENV and DB handles, or create their own. In this architecture, databases are rarely opened or closed when more than a single thread of control is running; that is, they are opened when only a single thread is running, and closed after all threads but one have exited. The last thread of control to exit closes the databases and the database environment.
This architecture is simplest to implement because thread serialization is easy and failure detection does not require monitoring multiple processes.
If the application's thread model allows processes to continue after thread failure, the DB_ENV->failchk() method can be used to determine if the database environment is usable after thread failure. If the application does not call DB_ENV->failchk(), or DB_ENV->failchk() returns DB_RUNRECOVERY, the application must behave as if there has been a system failure, performing recovery and re-creating the database environment. Once these actions have been taken, other threads of control can continue (as long as all existing Berkeley DB handles are first discarded).
Note that by default DB_ENV->failchk() will only notify the
calling thread that the database environment is unusable.
However, you can optionally cause DB_ENV->failchk() to broadcast
this to other threads of control by using the
--enable-failchk_broadcast
flag when you
compile your Berkeley DB library. If this option is turned
on, then all threads of control using the database
environment will return
DB_RUNRECOVERY
when they attempt to obtain a mutex lock. In this
situation, a DB_EVENT_FAILCHK_PANIC or
DB_EVENT_MUTEX_DIED event will also be
raised. (You use DB_ENV->set_event_notify() to examine events).
The second way to architect Transactional Data Store applications is as a group of related processes (the processes may or may not be multithreaded).
This architecture requires the order in which threads of control are created be controlled to serialize database environment recovery.
In addition, this architecture requires that threads of control be monitored. If any thread of control exits with open Berkeley DB handles, the application may call the DB_ENV->failchk() method to detect lost mutexes and locks and determine if the application can continue. If the application does not call DB_ENV->failchk(), or DB_ENV->failchk() returns that the database environment can no longer be used, the application must behave as if there has been a system failure, performing recovery and creating a new database environment. Once these actions have been taken, other threads of control can be continued (as long as all existing Berkeley DB handles are first discarded).
The easiest way to structure groups of related processes is to first create a single "watcher" process (often a script) that starts when the system first boots, runs recovery on the database environment and then creates the processes or threads that will actually perform work. The initial thread has no further responsibilities other than to wait on the threads of control it has started, to ensure none of them unexpectedly exit. If a thread of control exits, the watcher process optionally calls the DB_ENV->failchk() method. If the application does not call DB_ENV->failchk() or if DB_ENV->failchk() returns that the environment can no longer be used, the watcher kills all of the threads of control using the failed environment, runs recovery, and starts new threads of control to perform work.
The third way to architect Transactional Data Store applications is as a group of related processes that rely on DB_ENV->failchk() broadcasting to inform other threads and processes that recovery is required. DB_ENV->failchk() broadcasting is not enabled by default for the DB library, but using broadcasting means that a watcher process is not required. Instead, if DB_ENV->failchk() fails then all other threads and processes operating in that environment will also be notified of that failure so that they can know to run recovery.
To enable DB_ENV->failchk() broadcasting use the
--enable-failchk_broadcast
flag when you
configure the library. On Windows, use
HAVE_FAILCHK_BROADCAST
when you compile
the library.
If DB_ENV->failchk() broadcasting is enabled for your library and a thread of control encounters a failure when DB_ENV->failchk() is run, then all other threads and processes operating in that environment will be notified. If a failure is broadcast, then threads and processes will receive DB_RUNRECOVERY when they attempt to preform any one of a range of activities, including:
When entering a DB API.
When locking a mutex.
When performing disk or network I/O.
Threads and processes that are monitoring events will also receive DB_EVENT_FAILCHK_PANIC or DB_EVENT_MUTEX_DIED. You use DB_ENV->set_event_notify() to examine events.
The fourth way to architect Transactional Data Store applications is as a group of unrelated processes (the processes may or may not be multithreaded). This is the most difficult architecture to implement because of the level of difficulty in some systems of finding and monitoring unrelated processes. There are several possible techniques to implement this architecture.
One solution is to log a thread of control ID when a new Berkeley DB handle is opened. For example, an initial "watcher" process could run recovery on the database environment and then create a sentinel file. Any "worker" process wanting to use the environment would check for the sentinel file. If the sentinel file does not exist, the worker would fail or wait for the sentinel file to be created. Once the sentinel file exists, the worker would register its process ID with the watcher (via shared memory, IPC or some other registry mechanism), and then the worker would open its DB_ENV handles and proceed. When the worker finishes using the environment, it would unregister its process ID with the watcher. The watcher periodically checks to ensure that no worker has failed while using the environment. If a worker fails while using the environment, the watcher removes the sentinel file, kills all of the workers currently using the environment, runs recovery on the environment, and finally creates a new sentinel file.
The weakness of this approach is that, on some systems, it is difficult to determine if an unrelated process is still running. For example, POSIX systems generally disallow sending signals to unrelated processes. The trick to monitoring unrelated processes is to find a system resource held by the process that will be modified if the process dies. On POSIX systems, flock- or fcntl-style locking will work, as will LockFile on Windows systems. Other systems may have to use other process-related information such as file reference counts or modification times. In the worst case, threads of control can be required to periodically re-register with the watcher process: if the watcher has not heard from a thread of control in a specified period of time, the watcher will take action, recovering the environment.
The Berkeley DB library includes one built-in implementation of this approach, the DB_ENV->open() method's DB_REGISTER flag:
If the DB_REGISTER flag is set, each process opening the database environment first checks to see if recovery needs to be performed. If recovery needs to be performed for any reason (including the initial creation of the database environment), and DB_RECOVER is also specified, recovery will be performed and then the open will proceed normally. If recovery needs to be performed and DB_RECOVER is not specified, DB_RUNRECOVERY will be returned. If recovery does not need to be performed, DB_RECOVER will be ignored.
Prior to the actual recovery beginning, the DB_EVENT_REG_PANIC event is set for the environment. Processes in the application using the DB_ENV->set_event_notify() method will be notified when they do their next operations in the environment. Processes receiving this event should exit the environment. Also, the DB_EVENT_REG_ALIVE event will be triggered if there are other processes currently attached to the environment. Only the process doing the recovery will receive this event notification. It will receive this notification once for each process still attached to the environment. The parameter of the DB_ENV->set_event_notify() callback will contain the process identifier of the process still attached. The process doing the recovery can then signal the attached process or perform some other operation prior to recovery (i.e. kill the attached process).
The DB_ENV->set_timeout() method's DB_SET_REG_TIMEOUT flag can be set to establish a wait period before starting recovery. This creates a window of time for other processes to receive the DB_EVENT_REG_PANIC event and exit the environment.
There are three additional requirements for the DB_REGISTER architecture to work:
First, all applications using the database environment must specify the DB_REGISTER flag when opening the environment. However, there is no additional requirement if the application chooses a single process to recover the environment, as the first process to open the database environment will know to perform recovery.
Second, there can only be a single DB_ENV handle per database environment in each process. As the DB_REGISTER locking is per-process, not per-thread, multiple DB_ENV handles in a single environment could race with each other, potentially causing data corruption.
Third, the DB_REGISTER implementation does not explicitly terminate processes using the database environment which is being recovered. Instead, it relies on the processes themselves noticing the database environment has been discarded from underneath them. For this reason, the DB_REGISTER flag should be used with a mutex implementation that does not block in the operating system, as that risks a thread of control blocking forever on a mutex which will never be granted. Using any test-and-set mutex implementation ensures this cannot happen, and for that reason the DB_REGISTER flag is generally used with a test-and-set mutex implementation.
A second solution for groups of unrelated processes is also based on a "watcher process". This solution is intended for systems where it is not practical to monitor the processes sharing a database environment, but it is possible to monitor the environment to detect if a thread of control has failed holding open Berkeley DB handles. This would be done by having a "watcher" process periodically call the DB_ENV->failchk() method. If DB_ENV->failchk() returns that the environment can no longer be used, the watcher would then take action, recovering the environment.
The weakness of this approach is that all threads of control using the environment must specify an "ID" function and an "is-alive" function using the DB_ENV->set_thread_id() method. (In other words, the Berkeley DB library must be able to assign a unique ID to each thread of control, and additionally determine if the thread of control is still running. It can be difficult to portably provide that information in applications using a variety of different programming languages and running on a variety of different platforms.)
A third solution for groups of unrelated processes is a hybrid of the two above. Along with implementing the built-in sentinel approach with the the DB_ENV->open() methods DB_REGISTER flag, the DB_FAILCHK flag can be specified. When using both flags, each process opening the database environment first checks to see if recovery needs to be performed. If recovery needs to be performed for any reason, it will first determine if a thread of control exited while holding database read locks, and release those. Then it will abort any unresolved transactions. If these steps are successful, the process opening the environment will continue without the need for any additional recovery. If these steps are unsuccessful, then additional recovery will be performed if DB_RECOVER is specified and if DB_RECOVER is not specified, DB_RUNRECOVERY will be returned.
Since this solution is hybrid of the first two, all of the requirements of both of them must be implemented (will need "ID" function, "is-alive" function, single DB_ENV handle per database, etc.)
The described approaches are different, and should not be combined. Applications might use either the DB_REGISTER approach, the DB_ENV->failchk() or the hybrid approach, but not together in the same application. For example, a POSIX application written as a library underneath a wide variety of interfaces and differing APIs might choose the DB_REGISTER approach for a few reasons: first, it does not require making periodic calls to the DB_ENV->failchk() method; second, when implementing in a variety of languages, is may be more difficult to specify unique IDs for each thread of control; third, it may be more difficult determine if a thread of control is still running, as any particular thread of control is likely to lack sufficient permissions to signal other processes. Alternatively, an application with a dedicated watcher process, running with appropriate permissions, might choose the DB_ENV->failchk() approach as supporting higher overall throughput and reliability, as that approach allows the application to abort unresolved transactions and continue forward without having to recover the database environment. The hybrid approach is useful in situations where running a dedicated watcher process is not practical but getting the equivalent of DB_ENV->failchk() on the DB_ENV->open() is important.
Obviously, when implementing a process to monitor other threads of control, it is important the watcher process' code be as simple and well-tested as possible, because the application may hang if it fails.


Water
The water industry has a challenging task to provide potable water and disposal
services to a wide variety of users throughout the UK. The Met.Office provides
services to all parts of the industry from water suppliers to sewage engineers.
In this industry the weather affects both the demand for the product and the
availability of supplies thus it is imperative that the operating companies
have an accurate site specific service tailored for their needs.
A range of services is available:
MIST a system of providing current weather information direct to your PC.
Weather Sensitivity Analysis a process for relating business variables to the weather.
Forecasts available in a variety of timescales.
MORECS a data based system giving soil moisture content and thus the relationship between rainfall and available water.
Rainfall Programmes
For further information contact:
Mary Clarke
Tel.No. +44 (0)1344 856692
Postal Address: Met.Office, Room S1/2, Sutton House, London Road, Bracknell, Berks. RG12 2SY. United Kingdom
or simply fill in our Response Form.
Ian Watkins |<EMAIL_ADDRESS>| Last updated: 28 September 1995 | © Crown Copyright 1995


##Academic Publications
Process Based Evaluation of Computer Generated Poetry
Stephen McGregor, Matthew Purver, and Geraint Wiggins
INLG Workshop on Computational Creativity in Natural Language Generation
September 2016
Modeling Metaphor Perception with Distributional Semantics Vector Space Models
Kat R. Agres, Stephen McGregor, Karolina Rataj, Matthew Purver, Geraint A. Wiggins
ESSLLI Workshop on Computational Creativity, Concept Invention, and General Intelligence
August 2016
Words, Concepts, and the Geometry of Analogy
Stephen McGregor, Matthew Purver, and Geraint Wiggins
Workshop on Semantic Spaces at the Intersection of NLP, Physics and Cognitive Science
June 2016
From Distributional Semantics to Conceptual Spaces: A Novel Computational Method for Concept Creation
Stephen McGregor, Kat Agres, Matthew Purver, and Geraint A. Wiggins
Jounral of Aritificial General Intelligence
December 2015
Conceptualizing Creativity: From Distributional Semantics to Conceptual Spaces
Kat Agres, Stephen McGregor, Matthew Purver, and Geraint Wiggins
Sixth International Conference on Computational Creativity
July 2015
Metaphor, Meaning, Computers and Consciousness
Stephen McGregor, Matthew Purver, and Geraint Wiggins
8th AISB Symposium on Computing and Philosophy: The Significance of Metaphor and Other Figurative Modes of Expression and Thought
April 2015
How Many Robots Does It Take? Creativity, Robots and Multi-Agent Systems
Stephen McGregor, Mariano Mora McGinty, and Sascha Griffiths
Second AISB Workshop on Computational Creativity
April 2015
Computational Creativity: A Philosophical Approach, and and Approach to Philosophy
Stephen McGregor, Matthew Purver, and Geraint Wiggins
Fifth International Conference on Computational Creativity
June 2014
Considering the Law as an Evaluative Mechanism for Computational Creativity
Stephen McGregor
AISB Workshop on Computational Creativity
April 2014


Uses of Interface
com.opengamma.strata.calc.runner.CalculationFunction
-
Packages that use CalculationFunction Package Description com.opengamma.strata.calc.runner The calculation runner.com.opengamma.strata.measure.bond Base package for calculation functions.com.opengamma.strata.measure.capfloor Calculation functions for Ibor cap/floor products.com.opengamma.strata.measure.cms Calculation functions for constant maturity swap (CMS) products.com.opengamma.strata.measure.credit Calculation functions for credit products.com.opengamma.strata.measure.deposit Calculation functions for deposit products.com.opengamma.strata.measure.dsf Calculation functions for DSF products.com.opengamma.strata.measure.fra Calculation functions for FRA products.com.opengamma.strata.measure.fx Calculation functions for FX products.com.opengamma.strata.measure.fxopt Calculation functions for FX option products.com.opengamma.strata.measure.index Calculation functions for index products.com.opengamma.strata.measure.payment Calculation functions for payment products.com.opengamma.strata.measure.security Calculation functions for futures products.com.opengamma.strata.measure.swap Calculation functions for swap products.com.opengamma.strata.measure.swaption Calculation functions for swaption products. -
-
Uses of CalculationFunction in com.opengamma.strata.calc.runner
Methods in com.opengamma.strata.calc.runner that return CalculationFunction Modifier and Type Method Description default <T extends CalculationTarget>
CalculationFunction<? super T>CalculationFunctions. getFunction(T target)
Gets the function that handles the specified target.CalculationFunction<CalculationTarget>
CalculationTask. getFunction()
Gets the function that will calculate the value.Methods in com.opengamma.strata.calc.runner that return types with arguments of type CalculationFunction Modifier and Type Method Description <T extends CalculationTarget>
Optional<CalculationFunction<? super T>>CalculationFunctions. findFunction(T target)
Finds the function that handles the specified target.Methods in com.opengamma.strata.calc.runner with parameters of type CalculationFunction Modifier and Type Method Description static CalculationFunctions
CalculationFunctions. of(CalculationFunction<?>... functions)
Obtains an instance from the specified functions.static CalculationTask
CalculationTask. of(CalculationTarget target, CalculationFunction<? extends CalculationTarget> function, CalculationParameters parameters, List<CalculationTaskCell> cells)
Obtains an instance that will calculate the specified cells.static CalculationTask
CalculationTask. of(CalculationTarget target, CalculationFunction<? extends CalculationTarget> function, CalculationTaskCell... cells)
Obtains an instance that will calculate the specified cells.Method parameters in com.opengamma.strata.calc.runner with type arguments of type CalculationFunction Modifier and Type Method Description static CalculationFunctions
CalculationFunctions. of(List<? extends CalculationFunction<?>> functions)
Obtains an instance from the specified functions.static CalculationFunctions
CalculationFunctions. of(Map<Class<?>,? extends CalculationFunction<?>> functions)
Obtains an instance from the specified functions. -
Uses of CalculationFunction in com.opengamma.strata.measure.bond
Classes in com.opengamma.strata.measure.bond that implement CalculationFunction Modifier and Type Class Description class
BillTradeCalculationFunction<T extends SecuritizedProductPortfolioItem<Bill> & Resolvable<ResolvedBillTrade>>
Perform calculations on a singleBillTrade
orBillPosition
for each of a set of scenarios.class
BondFutureOptionTradeCalculationFunction<T extends SecuritizedProductPortfolioItem<BondFutureOption> & Resolvable<ResolvedBondFutureOptionTrade>>
Perform calculations on a singleBondFutureOptionTrade
orBondFutureOptionPosition
for each of a set of scenarios.class
BondFutureTradeCalculationFunction<T extends SecuritizedProductPortfolioItem<BondFuture> & Resolvable<ResolvedBondFutureTrade>>
Perform calculations on a singleBondFutureTrade
orBondFuturePosition
for each of a set of scenarios.class
CapitalIndexedBondTradeCalculationFunction<T extends SecuritizedProductPortfolioItem<CapitalIndexedBond> & Resolvable<ResolvedCapitalIndexedBondTrade>>
Perform calculations on a singleCapitalIndexedBondTrade
orCapitalIndexedBondPosition
for each of a set of scenarios.class
FixedCouponBondTradeCalculationFunction<T extends SecuritizedProductPortfolioItem<FixedCouponBond> & Resolvable<ResolvedFixedCouponBondTrade>>
Perform calculations on a singleFixedCouponBondTrade
orFixedCouponBondPosition
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.capfloor
Classes in com.opengamma.strata.measure.capfloor that implement CalculationFunction Modifier and Type Class Description class
IborCapFloorTradeCalculationFunction
Perform calculations on a singleIborCapFloorTrade
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.cms
Classes in com.opengamma.strata.measure.cms that implement CalculationFunction Modifier and Type Class Description class
CmsTradeCalculationFunction
Perform calculations on a singleCmsTrade
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.credit
Classes in com.opengamma.strata.measure.credit that implement CalculationFunction Modifier and Type Class Description class
CdsIndexTradeCalculationFunction
Perform calculations on a singleCdsIndexTrade
for each of a set of scenarios.class
CdsTradeCalculationFunction
Perform calculations on a singleCdsTrade
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.deposit
Classes in com.opengamma.strata.measure.deposit that implement CalculationFunction Modifier and Type Class Description class
TermDepositTradeCalculationFunction
Perform calculations on a singleTermDepositTrade
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.dsf
Classes in com.opengamma.strata.measure.dsf that implement CalculationFunction Modifier and Type Class Description class
DsfTradeCalculationFunction<T extends SecuritizedProductPortfolioItem<Dsf> & Resolvable<ResolvedDsfTrade>>
Perform calculations on a singleDsfTrade
orDsfPosition
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.fra
Classes in com.opengamma.strata.measure.fra that implement CalculationFunction Modifier and Type Class Description class
FraTradeCalculationFunction
Perform calculations on a singleFraTrade
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.fx
Classes in com.opengamma.strata.measure.fx that implement CalculationFunction Modifier and Type Class Description class
FxNdfTradeCalculationFunction
Perform calculations on a singleFxNdfTrade
for each of a set of scenarios.class
FxSingleTradeCalculationFunction
Perform calculations on a singleFxSingleTrade
for each of a set of scenarios.class
FxSwapTradeCalculationFunction
Perform calculations on a singleFxSwapTrade
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.fxopt
Classes in com.opengamma.strata.measure.fxopt that implement CalculationFunction Modifier and Type Class Description class
FxSingleBarrierOptionTradeCalculationFunction
Perform calculations on an FX single barrier option trade for each of a set of scenarios.class
FxVanillaOptionTradeCalculationFunction
Perform calculations on an FX vanilla option trade for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.index
Classes in com.opengamma.strata.measure.index that implement CalculationFunction Modifier and Type Class Description class
IborFutureOptionTradeCalculationFunction<T extends SecuritizedProductPortfolioItem<IborFutureOption> & Resolvable<ResolvedIborFutureOptionTrade>>
Perform calculations on a singleIborFutureOptionTrade
orIborFutureOptionPosition
for each of a set of scenarios.class
IborFutureTradeCalculationFunction<T extends SecuritizedProductPortfolioItem<IborFuture> & Resolvable<ResolvedIborFutureTrade>>
Perform calculations on a singleIborFutureTrade
orIborFuturePosition
for each of a set of scenarios.class
OvernightFutureTradeCalculationFunction<T extends SecuritizedProductPortfolioItem<OvernightFuture> & Resolvable<ResolvedOvernightFutureTrade>>
Perform calculations on a singleOvernightFutureTrade
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.payment
Classes in com.opengamma.strata.measure.payment that implement CalculationFunction Modifier and Type Class Description class
BulletPaymentTradeCalculationFunction
Perform calculations on a singleBulletPaymentTrade
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.security
Classes in com.opengamma.strata.measure.security that implement CalculationFunction Modifier and Type Class Description class
GenericSecurityPositionCalculationFunction
Perform calculations on a singleGenericSecurityPosition
for each of a set of scenarios.class
GenericSecurityTradeCalculationFunction
Perform calculations on a singleGenericSecurityTrade
for each of a set of scenarios.class
SecurityPositionCalculationFunction
Perform calculations on a singleSecurityPosition
for each of a set of scenarios.class
SecurityTradeCalculationFunction
Perform calculations on a singleSecurityTrade
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.swap
Classes in com.opengamma.strata.measure.swap that implement CalculationFunction Modifier and Type Class Description class
SwapTradeCalculationFunction
Perform calculations on a singleSwapTrade
for each of a set of scenarios. -
Uses of CalculationFunction in com.opengamma.strata.measure.swaption
Classes in com.opengamma.strata.measure.swaption that implement CalculationFunction Modifier and Type Class Description class
SwaptionTradeCalculationFunction
Perform calculations on a singleSwaptionTrade
for each of a set of scenarios.
-


A data estate can be various and complex. For every application you have in production, you have many copies of that application running in lower environments. And not only do you have numerous lower environments, but the data needs of each environment can be notably different.
Consider the following example from a Delphix customer (before they had Delphix):
Acme Finance has three major applications that are interdependent: Asset Management, Wealth Management, and Investment Banking. Each of these applications use a different RDBMS, as well (Oracle, MSSQL, and ASE).
Acme Finance has the following environments:
- Dev - an insane amount of copies of each application using synthetic and subsetted data sets.
- Test - many different levels of testing, each with different testing needs that vary greatly in complexity. Functional can perhaps be performed in isolation on subsetted data, but integration and system testing require all the application datasets to be in sync and have referential integrity. The size of the data for these environments limits the amount of parallel testing environments they can have, and the complexity of coordinating data refreshes across three different database technolgies and teams severly limits the frequency that the testing environments can be provisioned with fresh non-polluted data. Data-related defects commonly make it to production.
- Perf - One environment. Uses full production datasets. The size and complexity create the same constraints experinced in system and integration testing. Perf typically gets one run in a testing cycle to identify any potential performance tweaks.
- Staging/UAT - One environment. Uses full production datasets. The size and complexity create the same constraints experinced in system and integration testing. Users commonly encounter errors due to data pollution.
- There are also BreakFix, Analytics, and AI/ML environments.
All of these discrete datasets throughout their enterprise, and to add insult to injury, they don’t mask because it is too hard to accomplish!
By leveraging Delphix, you can bring data into your delivery pipeline and automate the way you protect, build, and deliver it as easy as you do code.
In our example, we show the Delphix Dynamic Data Platform
integrated with Jenkins
to:
- Get the latest data from production
- Create a masked copy of production data so it is clean for use in lower environments
- Reset/drop production usernames and passwords
- Create parallel data preparation environments
- Execute subset jobs against data preparation environments**
- Bookmark the data in those environments so that the data is now available via automation and self-Service
** Subsetting is simulated in this example. Delphix does not subset data, but many of our customers use their existing subsetting products to perform this functionality in the manner described above.
While we wait for the remainder of the Data Pipeline
build job to finish, we will go over some of the details of what is actually happening.
The format of this scene will differ a little bit, as you will execute steps as we go along and explain some things.
We use myriad techniques to accomplish the data automation in the Pipeline for demonstration and educational purposes. In practice, the user could leverage any method they prefer.
YOUR STEPS FOR THIS SCENE:
- Click the
Jenkins
bookmark. - Click on the
Data Pipeline
- Click on the job at the top of the list.
This screen shows us the various stages of the
Data Pipeline
. TheData Pipeline
is a declarative Jenkins Pipeline. The Pipeline is defined by a file (Jenkinsfile
) that is stored with our application code in version control. - OPTIONAL: The relevant contents of the
Jenkinsfile
have been included in images on this page for convenience, but you can clone the actual file in your environment by executingcd ~/git git clone git@tooling:/var/lib/jenkins/data-pipeline.git
-
Let’s inspect the
Jenkinsfile
.In the image below, several sections of the Jenkinsfile have been collapsed for clarity.
The main sections of the Jenkinsfile
include:
- environment - This section establishes environment variables that are set and/or read during the exection of the Pipeline.
- stage(<name>) - Each of these correspond to named stages you saw in the job view in your web browser
The stages of the Jenkinsfile
:
- Sync Delphix with Prod - This stage exists for aesthetics of the pipeline to better show the flow. Syncing the production data with Delphix is executed in the next stage in combination with the refresh.
- Refresh Secure Pipeline Master - Jenkins executes a Golang binary (
snap_prod_refresh_mm
) which sends REST calls to the DDDP to:- Snapshot the production database
- Refresh the
Patients Masked Master
data pod with the latest data from production and mask it according to the defined policy.
- Update Database Users and Passwords - This phase is a placeholder for where you could specify calls to any external tools you may have to reset application and database usernames and passwords.
- Prepare Data and Make Available For Use - This stage executes six near identical parallel stages.
One stage is expanded in the image below:
- Jenkins executes a kotlin binary (Delphix Automation Framework (DAF)) which sends REST calls to the DDDP to spinup a data pod.
- Jenkins executes a subsetting job (simulated)
- Jenkins executes a python script (dx_jetstream_container.py) which sends REST calls to Delphix to bookmark the
Patients Masked Master
and to make that bookmark available to Delphix Self Service.
Now, feel free to explore around until your Data Pipeline
job completes.
Just be sure to not click Save or Apply,
and you should be able to stay clear of doing any real damage.
Once the Data Pipeline
job has completed, you are ready for the next scene.


{include h} {include forum_header} {foreach $forum as $key => $v} {$v.name} {/foreach} {include b_menu} {include f}


Class Thing
Class that represents all things in the game.
Inherited Members
Namespace: Fibula.Common
Assembly: Fibula.Common.dll
Syntax
public abstract class Thing : IThing, ILocatable, IContainedThing, IEquatable<IThing>
Constructors
| Improve this Doc View SourceThing()
Initializes a new instance of the Thing class.
Declaration
public Thing()
Properties
| Improve this Doc View SourceCanBeMoved
Gets a value indicating whether this thing can be moved.
Declaration
public abstract bool CanBeMoved { get; }
Property Value
| Type | Description |
|---|---|
| System.Boolean |
CarryLocation
Gets the location where this thing is being carried at, if any.
Declaration
public abstract Location? CarryLocation { get; }
Property Value
| Type | Description |
|---|---|
| System.Nullable<Location> |
Location
Gets this thing's location.
Declaration
public virtual Location Location { get; }
Property Value
| Type | Description |
|---|---|
| Location |
ParentContainer
Gets or sets the parent container of this thing.
Declaration
public IThingContainer ParentContainer { get; set; }
Property Value
| Type | Description |
|---|---|
| IThingContainer |
ThingId
Gets the id of this thing.
Declaration
public abstract ushort ThingId { get; }
Property Value
| Type | Description |
|---|---|
| System.UInt16 |
UniqueId
Gets the unique id of this item.
Declaration
public Guid UniqueId { get; }
Property Value
| Type | Description |
|---|---|
| System.Guid |
Methods
| Improve this Doc View SourceDescribeForLogger()
Provides a string describing the current thing for logging purposes.
Declaration
public abstract string DescribeForLogger()
Returns
| Type | Description |
|---|---|
| System.String | The string to log. |
Equals(IThing)
Indicates whether the current object is equal to another object of the same type.
Declaration
public bool Equals(IThing other)
Parameters
| Type | Name | Description |
|---|---|---|
| IThing | other | The other object to compare against. |
Returns
| Type | Description |
|---|---|
| System.Boolean | True if the current object is equal to the other parameter, false otherwise. |
InvokePropertyChanged(String)
Invokes the ThingChanged event on this thing.
Declaration
public void InvokePropertyChanged(string propertyName)
Parameters
| Type | Name | Description |
|---|---|---|
| System.String | propertyName | The name of the property. |
ToString()
Declaration
public override string ToString()
Returns
| Type | Description |
|---|---|
| System.String |
Overrides
Events
| Improve this Doc View SourceThingChanged
Event to invoke when any of the properties of this thing have changed.
Declaration
public event OnThingStateChanged ThingChanged
Event Type
| Type | Description |
|---|---|
| OnThingStateChanged |


{% extends "layout.html" %} {% block page_title %} Registered Traveller {% endblock %} {% block content %} Back Sign out Which visa do you have? Select which of these applies to you Tier 1 Tier 2 Tier 5 (excluding creative or sporting concessions) Sole representative Minister of religion Civilian contractor Continue {% endblock %}


Display
- src
- utilities
- display.scss
Class list
- .flex
- .block
- .inline
- .inline-block
- .absent
- .invisible
- .flex-(breakpoint)
- .block-(breakpoint)
- .inline-(breakpoint)
- .inline-block-(breakpoint)
- .absent-(breakpoint)
- .invisible-(breakpoint)
Description
Use the display utility classes to override the display type for an element.
Absent
The .absent class is an abstraction on display: none;. Reduce the width of your browser to see the example in action.
small breakpoint
medium breakpoint
large breakpoint

small breakpoint

medium breakpoint

large breakpoint

Invisible
Use the .invisible class to set visibility: hidden; on an element. Reduce the width of your browser to see the example in action.


Message Box
Server:
Port:
Server Status: Not Connected
Subscribe to a Topic:
Message:
Publish Topic:
Course:
Your Name:


Lorem ipsum dolor amet roof party enamel pin adaptogen slow-carb dreamcatcher. Pitchfork letterpress four loko wayfarers snackwave. Kickstarter church-key try-hard quinoa vice helvetica tattooed. Tofu hexagon keffiyeh bicycle rights, chillwave shabby chic freegan whatever literally woke mixtape viral. Single-origin coffee fanny pack fingerstache vice. Pickled craft beer hell of, coloring book quinoa viral sartorial. Man braid YOLO kale chips, cardigan pour-over leggings wolf.
Lorem ipsum dolor amet roof party enamel pin adaptogen slow-carb dreamcatcher. Pitchfork letterpress four loko wayfarers snackwave. Kickstarter church-key try-hard quinoa vice helvetica tattooed. Tofu hexagon keffiyeh bicycle rights, chillwave shabby chic freegan whatever literally woke mixtape viral. Single-origin coffee fanny pack fingerstache vice. Pickled craft beer hell of, coloring book quinoa viral sartorial. Man braid YOLO kale chips, cardigan pour-over leggings wolf.


I have always been able to pick things up quicker than most. The technology world has been something that has interested me since I was a little kid. I can pick up electronics and know how to work them without reading about it. I still learn something new everyday but when it comes to developing or working with technology in general it seems to come natural.
You know the person in high school that wasnt part of just one group of people? The one that was friends with all the groups? That was me. I have always been able to find something in common with people from all walks of life. I think that has helped me in my professional life as a consultant. You can transform yourself and find a way to connect with someone.
Easy going but always on task. I have learned how to manage people and time to make the workplace less stressful even during close deadlines. I am a firm believer of happy people will work harder and more efficient.


Hi there! Welcome to my website!
For the best experience I recommend using Google Chrome and a powerful PC
Ok, so...
I really like computers!
Well in fact...
I enjoy computer programming and learning new programming languages in my free time.
I am specially experienced in...
C/C++,
.NET
(C#, F#, Windows Forms and ASP.NET MVC),
HTML5 / JavaScript,
...and some experience in Java, Python and Go.
But really, I can easily adapt to any framework, language or tool needed for the job at hand.
I also like videogames! Hey, why don't you check out this quick game I made:
I am self-taught, and enjoy learning new stuff like...
I also like to work out, running and play the
And ..., I guess that's all for now!
I hope you enjoyed knowing more about me!
Hey, wanna see something cool?
Use a spacebar or arrow keys to navigate


| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 64 1:function bot has 25 lines of code should be kept below 20 lines of code. |
| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 92 1:type signature for aumentaVelocidade has 6 arguments should be less than 5 |
| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 114 1:type signature for estabilizaVelocidade has 6 arguments should be less than 5 |
| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 206 1:type signature for colideT4Batota has 6 arguments should be less than 5 |
| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 209 1:type signature for colideT4 has 7 arguments should be less than 5 |
| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 223 1:type signature for vaiMorrer has 6 arguments should be less than 5 |
| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 229 1:type signature for vaiColidir has 7 arguments should be less than 5 |
| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 242 1:type signature for apontaPara has 6 arguments should be less than 5 |
| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 258 1:type signature for contrariaPara has 6 arguments should be less than 5 |
| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 259 1:function contrariaPara has 30 lines of code should be kept below 20 lines of code. |
| Warning |
BotHugo183.hs:SrcLoc "BotHugo183.hs" 348 1:type signature for percorreHistorico has 7 arguments should be less than 5 |
| Warning |
OracleT3.hs:SrcLoc "OracleT3.hs" 104 1:type signature for colideLocalAcc has 6 arguments should be less than 5 |
| Warning |
OracleT3.hs:SrcLoc "OracleT3.hs" 172 1:function temParede has 28 lines of code should be kept below 20 lines of code. |
| Warning |
OracleT3.hs:SrcLoc "OracleT3.hs" 368 1:function unitTests has 48 lines of code should be kept below 20 lines of code. |
| Warning |
OracleT3.hs:SrcLoc "OracleT3.hs" 503 1:function evaluationTests has 47 lines of code should be kept below 20 lines of code. |
| Warning |
OracleT4.hs:SrcLoc "OracleT4.hs" 93 1:type signature for lancaNitros has 6 arguments should be less than 5 |
| Warning |
OracleT5.hs:SrcLoc "OracleT5.hs" 203 1:function glossTempo has 22 lines of code should be kept below 20 lines of code. |
| Warning |
OracleT5.hs:SrcLoc "OracleT5.hs" 250 1:function joga has 63 lines of code should be kept below 20 lines of code. |
| Warning |
Tarefa3_2017li1g183.hs:SrcLoc "Tarefa3_2017li1g183.hs" 101 1:type signature for colideLocalAcc has 6 arguments should be less than 5 |
| Warning |
Tarefa3_2017li1g183.hs:SrcLoc "Tarefa3_2017li1g183.hs" 169 1:function temParede has 23 lines of code should be kept below 20 lines of code. |
| Warning |
Tarefa3_2017li1g183.hs:SrcLoc "Tarefa3_2017li1g183.hs" 326 1:function unitTests has 38 lines of code should be kept below 20 lines of code. |
| Warning |
Tarefa4_2017li1g183.hs:SrcLoc "Tarefa4_2017li1g183.hs" 76 1:type signature for lancaNitros has 6 arguments should be less than 5 |
| Critical |
OracleT1.hs:SrcLoc "OracleT1.hs" 20 1:StandaloneDeriving language extension is not enabled. Please add {-# LANGUAGE StandaloneDeriving #-} pragma at the top of your module. |


Vote
Apply To Contest
View All Offices
View All Parties
View Candidates
Profile
Admin
Log Out
Apply To Be A Candidate
Office
*
-- Select Office Name --
President
Governor
Senate
Federal House Of Representative
Local Government Chairman
Councillor
State House of Representative
Select Political Party
*
-- Select Political Party --
All Progressive Congress (APC)
People's Democratic Party (PDP)
African Action Congress (AAC)
All Progressive Grand Alliance (APGA)
Alliance for Democracy (AD)
Action Democratic Party (ADP)
African Democratic Congress (ADC)
All People's Party (APP)
Democratic Alternative (DA)
Communist Party of Nigeria (CPN)
Apply


A script is a standard Python file including special comments that are interpreted by slave during the conversion. The only mandatory comment is #main. It separates the script initialization from the main script section. Everything after the #main comment is executed in a separate thread.
Here is a simple script example that scans the power of a vna and record the average transmission :
fig, ax = subplots()
power = linspace(1e-1,0.3,100)
out = ones_like(power)*nan
#main
for index,p in enumerate(power):
#pause?
# Change parameter
vna1.SetPower(power=10*log10(p))
# Acquire data
time.sleep(1)
data = vna1.fetch()
out[index] = mean(abs(data.Sij)**2)
# Display data
ax.clear()
ax.plot(power,out)
#draw
#abort?
#looptime?
This comment signals the start of the cript main body. Everything after this comment is wrapped in a function that will be run in a separate thread. This comment is mandatory and must appear before the other comments listed below.
This line checks if the user asked for a pause. If yes it will wait until the user resumes or aborts the script.
This line inserts a Python break statement if the user aborts the script. Multiple #abort? should be used to exit nested loops.
Displays the given string in the slave window.
Displays the time ellapsed between two calls to the function.
Tell Matplotlib to redraw figures.


Exercise #3
Instructions:
create a grammatically sound sentence by choosing a word in each
column
. A word tile will turn green when the chosen word is correct. Red when it is wrong.
Create a simple sentence in present tense.
washing
always
eating.
I
washen
my hands
before
eat.
wash
during
eaten.
See answer
I wash my hands before eating.
Finish


Jonathan Sharman
Usable Security
Matthew Bernhard, Jonathan Sharman, Claudia Ziegler Acemyan, Philip Kortum, Dan S. Wallach, and J. Alex Halderman. 2019. On the Usability of HTTPS Deployment. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Paper 310, 10 pages. DOI:
https://doi.org/10.1145/3290605.3300540
High-performance Computing
Sharman, J. (2017). Exploring Tradeoffs in Parallel Implementations of Futures in C++ (master's thesis). Rice University, Houston, TX.
In 2016, I participated in the Co-Design Summer School at Los Alamos National Lab, where I collaborated with a team of physicists and computer scientists to develop 2COOL, a mini-app for handling data distribution and parallelization of smoothed particle hydrodynamics code. With two other computer science students and four physics students, we generated this nifty simulation of a binary neutron star merger.
Geochronology
Sharman, GR, Sharman, JP, Sylvester, Z. detritalPy: A Python‐based toolset for visualizing and analysing detrital geo‐thermochronologic data. Depositional Rec. 2018; 4: 202– 215.
https://doi.org/10.1002/dep2.45. In the top 10% of most-downloaded papers in Depositional Record, 2018-2019.


Welcome! This is Binary BootCamp reminding you there are only 10 people in the world.
Those who understand binary and those who don't.
BY: Kareem St. Louis
<EMAIL_ADDRESS>This is a STEM based APP designed to familiarize or remind yourself about binary numbers
On the following pages you'll find tips and tricks to help you from 0010 to 2.
CREDIT TO; text2speech.org for our voice and Google images for our charts.
The binary number system is a base-two system, meaning it uses two distinct digits – 0 and 1.
The decimal number system we’re all familiar with is a base-ten system, meaning it uses
ten distinct digits – 0 and 1, but also 2, 3, 4, 5, 6, 7, 8 and 9.
Counting in the decimal system, when all ten digits have been exhausted, the next number
is represented as ’10’. In the binary system, it’s the exact same. After 0 and 1 comes 10.
the binary system counts up in a logical order, and whenever a number
made up of entirely 1’s is reached, an extra digit is added. This is exactly the same thing
that happens in the decimal system when a number made up of entirely 9’s is reached.
However, binary numbers in a computing context are always represented in 8 digits or less
8 binary digits forming one byte. The total quantity of distinct numbers that can be
represented in 8 digits is 2^8 = 256. 1 through 255 as well as 0.
So 255 in binary is 11111111.
Lets do a little recap.
Binary numbers and arithmetic let you represent any amount you want using just two
Decimal 1 is binary 0001
Decimal 3 is binary 0011
digits: 0 and 1. Each digit "1" in a binary number represents a power of two, and each "0" represents zero
0001 is 2 to the zero power, or 1
0010 is 2 to the 1st power, or 2
When you see a number like "0101" you can figure out what it means by adding the powers of 2:
0101 = 0 + 4 + 0 + 1 = 5
I hope you got all that feel free to start this Tour over if not.
Now just one more thing.
ADDITION
Adding two binary numbers together is like adding decimal numbers,
except 1 + 1 = 10 (in binary, that is), so you have to carry the one to the next column:
0001
+ 0100
----
0101 (no carries to get this)
0001
+ 0001
----
0010 (1 plus 1 is 10, carry the 1 to the next column)
0011
+ 0011
----
0110 (1 + 1 = 10, so carry; then 1 + 1 + 1 = 11, so carry again)
0011
+ 0101
----
1000 (carry in every column here)
Subtraction is harder. Don't worry about it.
QUIZ
1. What is 5 in binary?
2. What is 0001+0001 in binary?
3. What is 0100+0101 in Decimal?
4. What is 1111 in Decimal?
5. What is 2 + 1010 in Binary?
Question 1: The correct answer is the Answer : 0101 .
Question 2: The correct answer is Answer : 0010 .
Question 3: The correct answer is Answer : 9.
Question 4: The correct answer is Answer : 15.
Question 5: The correct answer is Answer : 1100 .


Markov Chain Text Generator
Generates text using Markov chains based on the text sample provided
Reset
Play
Build model
Generate


10 Mar 2015
Context: England crashed out of the Cricket World Cup by losing to (no disrespect intended) a smaller team (Bangladesh) and their coach started talking about looking at data which, as expected, hasn't gone down well.
I found a few key quotes in this article that illustrate the need to balance measurements with sensibility:
- "England have, under Moores, known the price of everything but the value of nothing."
- "But the data helped imprison them. ...The data, doubtless, meant the batting order could never be adjusted to take advantage of or respond to a changed set of circumstances. The data said this and that and turned intelligent cricketers - for such we assume they must be – into morons."
"...the data doesn’t mean nearly as much as you think it does there comes a point when your theory has to be reconsidered."
and finally
"It’s tough to measure this sort of thing, however, so instead we assume that what’s important is what can be measured. That isn’t the case. It leads to a bad place."
These quotes succintly contextualize several facets of data analysis:
- the push-pull relationship between analysis and action
- asking the right questions
- measuring the right things
- utilizing domain expertise
- understanding stakeholders and their latent needs
- communicating and acting on insights
- knowing when, where and how to rely on data
Amidst the cacophony of complex statistics, models and a variety of tools, it is important to remember that ALL organizations are results-driven.
It is already hard enough to translate complex technical insights to a broader audience that is juggling several priorities. Despite this, a data analyst is responsible for utilizing the available technical resources and organizational strategies in delivering the results that are demanded by businesses/organizations.
To that end, the context of analytics usage is paramount.
07 Oct 2014
I've been taking a class in Statistical Analysis with Prof. Yuri Balasanov and we recently covered the concept of a Random Walk.
Random Walks are interesting to study because they expose the frailty of our grasp over probabilistic thinking. They are also interesting from a philosophical point of view. A minor class discussion point moved me immensely and inspired me to write this post to illustrate the power of choices.
What is a Random Walk?
In very simple layman's terms, a random walk is a succession of random steps with a probability associated with each outcome. This is typically illustrated using a (fair) coin toss.
Let's illustrate these points and test our intuition using using a fair coin i.e a coin whose probability of Heads = probability of Tails = 0.5.
Expt 1: If I flipped the coin 100,000 times, what probability will the tails converge at?
 As the graph below shows, over a large enough sample, the convergence is at 0.5.
Graph 1: Convergence of tails
So far, so good.
Expt 2: Let's play a game. I give you a dollar each time the coin lands as a tail and take away a dollar each time the coin lands as heads. If we flipped the coin a million times, how much money do you think you will make?
Given the results of the first experiment, most people would guess that the profits and losses would net themselves out and we would end up with a zero profit.
Graph 2: Random Walks
But in reality, there are several possibilities as illustrated by the graph above. These graphs illustrate that there are several possible cumulative outcomes each independent of the past. This is the core illustration of a Random Walk, and this is the point at which I had a mini "Aha!" moment. This is also what Nassim Taleb refers to as the concept of Alternate Histories in his book Fooled by Randomness.
These basic set of graphs illustrate two fundamentally simple, yet profound points:
A random walk is not mean converging. In simple English: While the probability of an individual event may be 50-50, a cumulative outcome will vary significantly from the expected average outcome. This has significant implications in financial markets, but also in day to day activities such as individual performances, individual choices etc.
The graphs demonstrate possible outcomes from time t = 0 to t = INF. Some graphs consistently go up, others go down while yet others oscillate. Any time an individual plays this game, s/he is already pinned to a single path of outcomes. However, and this was my fundamental insight, at any time t = N, an individual ALWAYS HAS A CHOICE to play or not to play.
Graph 3: Illustration of Choice
This is because at any time t = N, though there are several paths and whilst an outcome cannot deviate from an individual path, the player always has a choice to play for the next outcome or to not play. This has the impact of changing the player's path from the one he is on to one he invents anew through choice and an equiprobable outcome. In effect, a player cannot switch paths, but can invent a new path for himself through his choices.
From a philosophical standpoint, I wonder if this is what we call destiny.
20 Sep 2014
I conducted an R Workshop at the University of Chicago. This post contains the materials and code adapted from a Slidify presentation. The full presentation and code can be found on this repo.
Agenda
- Installing and Loading Packages
- Reading Data into R - using jsonlite package
- Reshaping Data (Wide to Long format) - using reshape2 package
- Data Manipulation - using dplyr package
- Data Visualization - using ggplot2 package
Installing Packages
- install.packages("packagename")
- library(packagename)
Exercise 1: Install and load these packages into the R environment
- httr
- jsonlite
- dplyr
- ggplot2
Reading Data into R
Exercise 2: Fetch your IP address
Data Manipulation
- dplyr
- Key features
- tbl_df()
- %>% chaining operator
- Verbs
- select
- filter
- arrange
- mutate
- summarise
Exercise 3a: Come up with a metric for the sports dataset and use dplyr to generate it
Exercise 3b: Explain why the metric is relevant
Data Visualization
- ggplot2: Grammar of Graphics
- Plots as objects
- Layering aesthetics and options
Exercise 4: Visualize the sports dataset metric
Summary
- Installing and Loading packages in R
- Reading data
- read.csv
- jsonlite package
- Data munging using dplyr
- Visualization using ggplot2
- Metrics
- Context
- Relevance
- Communication
Resources
R, Statistics
Visualization:
25 Aug 2014
Hey there! My name is Narayanan "Naveen" Venkataraman. I am a student at The University of Chicago studying Analytics. My github contains code for the projects I'm currently working on including: classroom work, writing portfolio and open source contributions.
R projects
- Currency Arbitrage
- R For Excel Users - In partnership with John Taveras


A synthetic dataset that describes pizza sales for a pizza place somewhere in
the US. While the contents are artificial, the ingredients used to make the
pizzas are far from it. There are 32 different pizzas that fall into 4
different categories: classic
(classic pizzas: 'You probably had one like
it before, but never like this!'), chicken
(pizzas with chicken as a major
ingredient: 'Try the Southwest Chicken Pizza! You'll love it!'), supreme
(pizzas that try a little harder: 'My Soppressata pizza uses only the finest
salami from my personal salumist!'), and, veggie
(pizzas without any meats
whatsoever: 'My Five Cheese pizza has so many cheeses, I can only offer it in
Large Size!').
pizzaplace
A tibble with 49574 rows and 7 variables:
The ID for the order, which consists of one or more pizzas
at a give date
and time
A character representation of the order
date,
expressed in the ISO 8601 date format (YYYY-MM-DD)
A character representation of the order
time,
expressed as a 24-hour time the ISO 8601 extended time format
(hh:mm:ss)
The short name for the pizza
The size of the pizza, which can either be S
,
M
, L
, XL
(rare!), or XXL
(even rarer!);
most pizzas are available in the S
, M
, and L
sizes
but exceptions apply
The category or type of pizza, which can either be
classic
, chicken
, supreme
, or veggie
The price of the pizza and the amount that it sold for (in USD)
Each pizza in the dataset is identified by a short name
. The following
listings provide the full names of each pizza and their main ingredients.
Classic Pizzas:
classic_dlx
: The Classic Deluxe Pizza (Pepperoni, Mushrooms, Red
Onions, Red Peppers, Bacon)
big_meat
: The Big Meat Pizza (Bacon, Pepperoni, Italian Sausage,
Chorizo Sausage)
pepperoni
: The Pepperoni Pizza (Mozzarella Cheese, Pepperoni)
hawaiian
: The Hawaiian Pizza (Sliced Ham, Pineapple, Mozzarella
Cheese)
pep_msh_pep
: The Pepperoni, Mushroom, and Peppers Pizza
(Pepperoni, Mushrooms, and Green Peppers)
ital_cpcllo
: The Italian Capocollo Pizza (Capocollo, Red
Peppers, Tomatoes, Goat Cheese, Garlic, Oregano)
napolitana
: The Napolitana Pizza (Tomatoes, Anchovies, Green
Olives, Red Onions, Garlic)
the_greek
: The Greek Pizza (Kalamata Olives, Feta Cheese,
Tomatoes, Garlic, Beef Chuck Roast, Red Onions)
Chicken Pizzas:
thai_ckn
: The Thai Chicken Pizza (Chicken, Pineapple, Tomatoes,
Red Peppers, Thai Sweet Chilli Sauce)
bbq_ckn
: The Barbecue Chicken Pizza (Barbecued Chicken, Red
Peppers, Green Peppers, Tomatoes, Red Onions, Barbecue Sauce)
southw_ckn
: The Southwest Chicken Pizza (Chicken, Tomatoes, Red
Peppers, Red Onions, Jalapeno Peppers, Corn, Cilantro, Chipotle Sauce)
cali_ckn
: The California Chicken Pizza (Chicken, Artichoke,
Spinach, Garlic, Jalapeno Peppers, Fontina Cheese, Gouda Cheese)
ckn_pesto
: The Chicken Pesto Pizza (Chicken, Tomatoes, Red
Peppers, Spinach, Garlic, Pesto Sauce)
ckn_alfredo
: The Chicken Alfredo Pizza (Chicken, Red Onions,
Red Peppers, Mushrooms, Asiago Cheese, Alfredo Sauce)
Supreme Pizzas:
brie_carre
: The Brie Carre Pizza (Brie Carre Cheese, Prosciutto,
Caramelized Onions, Pears, Thyme, Garlic)
calabrese
: The Calabrese Pizza (‘Nduja Salami, Pancetta,
Tomatoes, Red Onions, Friggitello Peppers, Garlic)
soppressata
: The Soppressata Pizza (Soppressata Salami, Fontina
Cheese, Mozzarella Cheese, Mushrooms, Garlic)
sicilian
: The Sicilian Pizza (Coarse Sicilian Salami, Tomatoes,
Green Olives, Luganega Sausage, Onions, Garlic)
ital_supr
: The Italian Supreme Pizza (Calabrese Salami,
Capocollo, Tomatoes, Red Onions, Green Olives, Garlic)
peppr_salami
: The Pepper Salami Pizza (Genoa Salami, Capocollo,
Pepperoni, Tomatoes, Asiago Cheese, Garlic)
prsc_argla
: The Prosciutto and Arugula Pizza (Prosciutto di San
Daniele, Arugula, Mozzarella Cheese)
spinach_supr
: The Spinach Supreme Pizza (Spinach, Red Onions,
Pepperoni, Tomatoes, Artichokes, Kalamata Olives, Garlic, Asiago Cheese)
spicy_ital
: The Spicy Italian Pizza (Capocollo, Tomatoes, Goat
Cheese, Artichokes, Peperoncini verdi, Garlic)
Vegetable Pizzas
mexicana
: The Mexicana Pizza (Tomatoes, Red Peppers, Jalapeno
Peppers, Red Onions, Cilantro, Corn, Chipotle Sauce, Garlic)
four_cheese
: The Four Cheese Pizza (Ricotta Cheese, Gorgonzola
Piccante Cheese, Mozzarella Cheese, Parmigiano Reggiano Cheese, Garlic)
five_cheese
: The Five Cheese Pizza (Mozzarella Cheese, Provolone
Cheese, Smoked Gouda Cheese, Romano Cheese, Blue Cheese, Garlic)
spin_pesto
: The Spinach Pesto Pizza (Spinach, Artichokes,
Tomatoes, Sun-dried Tomatoes, Garlic, Pesto Sauce)
veggie_veg
: The Vegetables + Vegetables Pizza (Mushrooms,
Tomatoes, Red Peppers, Green Peppers, Red Onions, Zucchini, Spinach, Garlic)
green_garden
: The Green Garden Pizza (Spinach, Mushrooms,
Tomatoes, Green Olives, Feta Cheese)
mediterraneo
: The Mediterranean Pizza (Spinach, Artichokes,
Kalamata Olives, Sun-dried Tomatoes, Feta Cheese, Plum Tomatoes, Red Onions)
spinach_fet
: The Spinach and Feta Pizza (Spinach, Mushrooms,
Red Onions, Feta Cheese, Garlic)
ital_veggie
: The Italian Vegetables Pizza (Eggplant, Artichokes,
Tomatoes, Zucchini, Red Peppers, Garlic, Pesto Sauce)
11-5
Other Datasets:
countrypops
,
exibble
,
gtcars
,
sp500
,
sza
#> Rows: 49,574 #> Columns: 7 #> $ id <chr> "2015-000001", "2015-000002", "2015-000002", "2015-000002", "20… #> $ date <chr> "2015-01-01", "2015-01-01", "2015-01-01", "2015-01-01", "2015-0… #> $ time <chr> "11:38:36", "11:57:40", "11:57:40", "11:57:40", "11:57:40", "11… #> $ name <chr> "hawaiian", "classic_dlx", "mexicana", "thai_ckn", "five_cheese… #> $ size <chr> "M", "M", "M", "L", "L", "L", "L", "M", "M", "M", "S", "S", "S"… #> $ type <chr> "classic", "classic", "veggie", "chicken", "veggie", "supreme",… #> $ price <dbl> 13.25, 16.00, 16.00, 20.75, 18.50, 20.75, 20.75, 16.50, 16.50, …


Go to the source code of this file.
Macros | |
| #define | MAIN_CLOCK_FREQUENCY 12000000 |
| #define | ACLK_FREQUENCY (LFXT1_FREQUENCY/LFXT1_FREQUENCY_DIVIDER) |
| #define | CCRn_VALUE_FOR_10_mSEC ACLK_FREQUENCY/100 |
| #define | CYCLES_PER_PERIOD ((MAIN_CLOCK_FREQUENCY/MAIN_CLOCK_DIVIDER)/ACLK_FREQUENCY) |
| #define | CYCLES_AVAILABLE_FOR_TEST (CYCLES_PER_PERIOD*CCRn_VALUE_FOR_10_mSEC) |
| #define | FREQUENCY_COUNT (CYCLES_AVAILABLE_FOR_TEST/NUMBER_OF_CYLES_TO_INCREMENT_COUNTER) |
| #define | FREQUENCY_COUNT_MAX (FREQUENCY_COUNT*(100 + PERCENT_FREQUENCY_DRIFT))/100 |
| #define | FREQUENCY_COUNT_MIN (FREQUENCY_COUNT*(100 - PERCENT_FREQUENCY_DRIFT))/100 |
| #define MAIN_CLOCK_FREQUENCY 12000000 |
| #define ACLK_FREQUENCY (LFXT1_FREQUENCY/LFXT1_FREQUENCY_DIVIDER) |
| #define CCRn_VALUE_FOR_10_mSEC ACLK_FREQUENCY/100 |
Referenced by IEC60730_OSCILLATOR_TEST_testOsc().
| #define CYCLES_PER_PERIOD ((MAIN_CLOCK_FREQUENCY/MAIN_CLOCK_DIVIDER)/ACLK_FREQUENCY) |
| #define CYCLES_AVAILABLE_FOR_TEST (CYCLES_PER_PERIOD*CCRn_VALUE_FOR_10_mSEC) |
| #define FREQUENCY_COUNT (CYCLES_AVAILABLE_FOR_TEST/NUMBER_OF_CYLES_TO_INCREMENT_COUNTER) |
| #define FREQUENCY_COUNT_MAX (FREQUENCY_COUNT*(100 + PERCENT_FREQUENCY_DRIFT))/100 |
Referenced by IEC60730_OSCILLATOR_TEST_testOsc().
| #define FREQUENCY_COUNT_MIN (FREQUENCY_COUNT*(100 - PERCENT_FREQUENCY_DRIFT))/100 |
Referenced by IEC60730_OSCILLATOR_TEST_testOsc().


|
**** Under Siege |
|
|
Nationality: Generic American™ Training/Specialty: ex-SEAL (expert in: martial arts, weapons and tactics, explosives), Silver Star, Purple Heart with cluster, Navy Cross KILLS: 22 (!) Broken Bones |
|
| Elbows: 1 Right | Knees: |
| Back: | Neck: |
| Wrist: 1 Right | Ankles: |
| Fingers: | |
|
Notable Painful Injuries: I-Beam falls through chest, Shoulder pushed through high-power bandsaw, Tommy Lee Jones' right eye pushed in then knife rammed to the hilt in the top of his skull |
|
|
One-liners: Gary Busey: "Watch it, Ryback, we still have a week together." Seagal: "I guess that means I won't get to see you go through puberty." |
|
|
Comments: Appearance of Dagger-grip knife-fighting. |


A simple demo of Beta-Binomial Bayesian Update
Introduction
One of the most useful concepts that I’ve learned from work is about the beta-binomial model and its bayesian update process. Hence I would like to share this knowledge so that more people can benefit from it.
It is a very easy-to-use technique that can help us overcome the issue of data sparsity and give us a realistic estimation of a probability.
In the context of e-Business, we can use this technique to predict the conversion rate or click-through-rate (number of clicks/number of impressions).
Read More  →


Preparation time
less than 30 mins
Cooking time
30 mins to 1 hour
Serves
Serves 6-8
Dietary
Parkin is the perfect sweet ginger traybake, ideal for bonfire parties and Halloween.
For the parkin, preheat the oven to 140C/275F/Gas 1 and butter a 20cm/8in square cake tin.
Sieve the flour, salt, ginger, nutmeg and mixed spice together into a large bowl. Mix in the oat flakes.
Put the syrup, treacle, butter and sugar into a small saucepan and melt over a gentle heat. Bring up to a simmer but do not boil. When the butter and sugar have melted, pour into the flour mixture and stir to combine.
Mix in the beaten egg and milk to create a soft, almost pouring consistency batter.
Pour into the buttered tin and bake for 50 minutes, or until firm in the centre.
Remove from the oven and leave to cool in the tin for 5-10 minutes before turning out and cutting into squares.
For the sauce, place all the ingredients in a saucepan, bring to the boil and simmer until a syrup consistency is formed.
Serve a square of parkin on each plate, then spoon over the syrup. Finish with a scoop of ice cream.


|
=appointment['Name']?> Dose: =dose?> When: = convertDateToString(appointment['Date'])?> @ =convert24to12hr(appointment['Time'])?> Where: =appointment['Location']?> Brand: =appointment['VaccineBrand']?> Notes: =appointment['Notes']?> |
| } else {?> ID: =id?> }?> |


Click
Select single item
alt
Disable selection box to select text on the screen
+ Click
Add or remove single item to selection
⇧ + Drag
Add items to selection
⇧ + Click
Select items in a range
⇧ + R
Move start of range selection
⇧ + + Drag
Remove items from selection


INVENTORY
table.
| Objective | Monitoring Report Used | |
| Step 1: | Determining the problem category | MONREPORT.DBSUMMARY |
| Step 2: | Performing detailed diagnosis appropriate locking problems | MONREPORT.DBSUMMARY |
| Step 3: | Identifying the scope of the SQL statements involved | MONREPORT.LOCKWAIT |


New Media Club is a student run organization at RIT focused on holding events for those interested in New Media and bringing students, professors and alumni together outside of a classroom setting.
pssst... try clicking and dragging to draw ;)
JOIN US
For our next meeting
We are designers, developers, artists, and creators at the Rochester Institute of Technology in Rochester, NY. We strongly believe in including all in our pursuit for better design.
One of the main goals of New Media Club is to provide opportunities for mentorship, which strengthens our community and reinforces our bonds with our large and growing network of alumni. We create mentorship groups yearly so that our students can get to know those from different backgrounds and year levels.
We also hold mentorship events occasionally and encourage groups to meet up and get to know their peers outside of the lab. Our goal is to have students build connections stronger than just classmates and teach the next generation of students our tips and tricks.
More About Our Mentorship Program →Support our club through our web store. We offer stickers, t-shirts and other NMC swag for you to rep in the lab or the real world. Thank you!


Non Drug Management of Irritable Bowel Syndrome
- Avoid foods rich in sugars, soda, caffeine, alcohol
-
Eat whole meals, natural foods, fruits and vegetables
-
Try Low FODMAP diet. It's been shown to improve symptoms.Foods that have high FODMAP
(F- fermentable; O- oligosaccharides; D- disaccharides; M-monosaccharides; A-and; P- polyols)
include the following (should be avoided): wheat, barley, onions, leek, garlic, peas, cashews, milk, custard,
ice-cream, yoghurt, apples, pears, mangoes, cherries, watermelons, honey, asparagus, apricots, cauliflower, artificially
sweetened chewing gum)
-
Eat deliberately and slowly
-
Some people find benefits from avoiding milk and gluten products
-
Destress yourself via meditation, massage and exercise
-
Try probiotics ie good bacteria
-
Peppermint oil
-
Acupuncture
-
Increase physical activity and try eliminating gluten and or lactose
-
Talk to a health professional


Constructor Summary
| Public Constructor | ||
| public |
constructor(separator: string, name: string, storage: Storage) |
Method Summary
| Public Methods | ||
| public |
Register a value, such as a string, a number, an array, an object or a function. |
|
| public |
Register a service factory, which will be called to return the service instance. |
|
| public |
Returns a namespace name. |
|
| public |
Returns a module namespace. |
|
| public |
Register a service constructor,
which will be invoked with |
|
| public |
value(name: string, dependencies: array, definition: string | number | object | array | function): function Register a value, such as a string, a number, an array, an object or a constructor. |
Public Constructors
Public Methods
public const(name: string, definition: string | number | object | array | function): function source
Register a value, such as a string, a number, an array, an object or a function.
public factory(name: string, dependencies: array, definition: function): function source
Register a service factory, which will be called to return the service instance. Any function's value will be registered as a singleton.
public namespace(name: string): Namespace source
Returns a module namespace.
Params:
| Name | Type | Attribute | Description |
| name | string | Module namespace. Optional. |
public service(name: string, dependencies: array, definition: function): function source
Register a service constructor,
which will be invoked with new
to create the service instance.
Any type which was registered as a service is singleton.
public value(name: string, dependencies: array, definition: string | number | object | array | function): function source
Register a value, such as a string, a number, an array, an object or a constructor. Note: If passed value is function type, it will be treated as constructor and every time when it's injected, new instance will be created.


{{ 'home.view-experiment.enrollment-over-time.unique-enrolled.text' | translate }}
67
{{ 'home.view-experiment.enrollment-over-time.excluded.text' | translate: { numbers: 50 } }}
{{ 'home.view-experiment.enrollment-over-time.classes-enrolled.text' | translate }}
12
{{ 'home.view-experiment.enrollment-over-time.excluded.text' | translate: { numbers: 12 } }}


MyHome
Home
(current)
Lights
Temperature
Garage
Lights
Kitchen
Bedroom
leaving Room
Toilet
Garage
Temperature
Temperature :
Humidity
Humidity :
Garage door
Close
Open