Installation¶
Pip Install
pip install tamr_toolbox[all]
Optional Features
Some features of Tamr-Toolbox require additional dependencies. You can opt-in to these features during installation. By including tamr_toolbox[all]
in the pip installation command, you will install the dependencies required for all optional features. To include dependencies required for one or more optional features use tamr_toolbox[feature_1, feature_2]
. A minimal installation is achieved by omitting the []
entirely.
In some cases you may already have a version of the library installed and would prefer to use that instead. Or perhaps you would like better control over what version you are installing. If doing so, please use at least minimum version of the library specified below. You will then want to install a version of tamr-toolbox without that library included (such as the version with no optional features) so that it does not attempt to change your existing version.
All optional features (suggested)
Install instructions:
pip install tamr_toolbox[all]
No optional features
Install instructions:
pip install tamr_toolbox
Optional Feature: Google Translate
Install instructions:
pip install tamr_toolbox[translation]
Required for Translation Enrichment
Library: GoogleTranslate (Tamr-toolbox uses version == 2.0.1)
Note: You will additionally need your own google API key in order to use translation capabilities.
Optional Feature: Mock API Testing
Install instructions:
pip install tamr_toolbox[testing]
Required for Testing
Library: Responses (Tamr-Toolbox uses version == 0.10.14)
Optional Feature: Pandas dataframes
Install instructions:
pip install tamr_toolbox[pandas]
Required for DataFrame I/O
Library: Pandas (Tamr-Toolbox uses version >= 0.21.0)
Optional Feature: Slack Notifications
Install instructions:
pip install tamr_toolbox[slack]
Required for Slack
Library: Slack Client (Tamr-Toolbox uses version >= 2.7.2)
Offline installation
Download tamr_toolbox
and its dependencies on a machine with the same operating system and python version as your target system, that has online access to PyPI:
pip download tamr_toolbox[all] -d tamr-toolbox-requirements
zip -r tamr-toolbox-requirements.zip tamr-toolbox-requirements
Deliver the .zip
file to the target machine where you want tamr-toolbox
installed. You can do this via email, cloud drives, scp
or any other mechanism.
Finally, install tamr-toolbox
from the saved dependencies:
unzip tamr-toolbox-requirements.zip
pip install --no-index --find-links=tamr-toolbox-requirements tamr_toolbox[all]
If you are not using a virtual environment, you may need to specify the --user
flag if you get permissions errors:
pip install --user --no-index --find-links=tamr-toolbox-requirements tamr_toolbox[all]


{% extends 'admin_base.html' %} {% load filetransfers %} {% block title %}Make:good - Admin!{% endblock %} {% block content %}
Cath: here you can upload background images for the homepage. The image will will picked at random for each user/visit. The maximum image size is a little under 1mb. You should compress the image as much as you can bear to do so, as this will speed up page load times for your users. If page load times are a problem we can look in to a couple of different solutions.


The file MotifGroups.txt
This
parameter file in
Database subdirectories specifies
MotifGroups.
A MotifGroup is a collection
of Motifs expected to match at the same
position in a retrovirus. It may include Motifs from different virus genera. It belongs to a SubGene .
MotifGroups.txt contains a number of singleparameters,
one for each MotifGroup. The key of the parameter is the name of the
MotifGroup.
The contents of the parameter must contain at least two
blank-separated fields:
- The name of the SubGene
to which the MotifGroup belongs.
- A range specifier. It
specifies the acceptable range for the distance between the hotspot of
a Motif hit by a Motif belonging to this MotifGroup, and the hotspot of
the SubGene hit in which it is included.
There may also be an arbitrary number of subsequent blank-separated
fields consisting of a MotifGroup name and a
range specifier separated by a ':' (no blanks allowed) which
specify the acceptable range between the hotspot of a Motif hit by a
Motif belonging to this MotifGroup, and
the hotspot of a Motif hit by a Motif belonging to another MotifGroup
in
the same SubGene hit.


vignettes/calibration_diagnostics.Rmd
calibration_diagnostics.Rmd
This report aims to:
The data used here come from a systematic review by Whittaker et al. (2021) looking into global patterns of submicroscopic P. falciparum infection - see here for further information on this dataset. Let’s start by plotting the raw data:
# plot data and model
plot1 <- PCR_micro_age_whittaker2021 %>%
ggplot() + theme_bw() +
geom_abline(aes(slope = 1, intercept = 0), linetype = "dotted") +
geom_point(aes(x = PCR_Prev, y = Micro_Prev, color = Age_Group)) +
scale_color_manual(values = simplegen_cols()) +
xlab("Prevalence (PCR)") + ylab("Prevalence (microscopy)")
plot1
We can see that prevalence by microscopy is always below the 1:1 line, in other words, microscopy is less sensitive than PCR over the entire prevalence range. The relationship also appears to have a curve, with the two measures agreeing at both very high and very low prevalence, but with microscopy being significantly below PCR prevalence at intermediate values. Nevertheless, let’s start by fitting a simple linear relationship to each age group to get a feel for the overall slope:
# fit linear model to each age group
slope_df <- mapply(function(x) {
mod <- lm(x$Micro_Prev ~ x$PCR_Prev + 0)
data.frame(Age_Group = x$Age_Group[1], slope = mod$coefficients[1])
}, split(PCR_micro_age_whittaker2021, f = PCR_micro_age_whittaker2021$Age_Group), SIMPLIFY = FALSE) %>%
bind_rows()
# add fits to plot
plot2 <- plot1 +
geom_abline(aes(slope = slope, intercept = 0, color = Age_Group),
linetype = "dashed", data = slope_df)
plot2
# print slope
slope_df
## Age_Group slope
## x$PCR_Prev...1 0-5 0.7352078
## x$PCR_Prev...2 5-15years 0.6177060
## x$PCR_Prev...3 15+ 0.4601922
This is not an excellent fit to the data, but it makes it clear that microscopy sensitivity decreases (relative to PCR) as we look at older age groups.
We will compare this against model output produced by taking a range of mosquito relative population densities, implying a range of EIRs. Measured values include the prevalence by both microscopy and PCR in age groups that correspond to the data (0-5 years, 5-15 years, and over 15 year olds). For each simulation, model output is run for 20 years and prevalences are taken as the average over the final 10 years.
# load precomputed model output
# code to produce this output: R_ignore/calibration/calibration_micro_PCR1.R
df_model <- readRDS("../inst/extdata/calibration_micro_PCR1.rds")
plot2 +
geom_line(aes(x = PCR, y = Micro, color = Age_Group), data = df_model)
We can see that our model captures the overall pattern in the data, but it fails to capture the differences between the age groups. This implies that we need a stronger relationship between detectability and age in our model, such that older age groups contain many sub-microscopic infections. As we only model detectability changing as a function of exposure and not age, we need this to be an exposure-driven effect in the current setup. However, any changes also need to make sense in terms of age-incidence patterns (see here).
We also find that the model is too linear, having a slight upwards curve at very high prevalence but nothing like what we see in the data. At extremely high transmission it is likely that the vast majority of infections are asymptomatic, but based on the patterns above we would also expect them to be microscopy-positive, rather than purely sub-patent. Hence, we perhaps need to revisit the implied connection between clinical status and detectability in our model.


{% extends "base.html" %} {% from "_formhelpers.html" import form_errors %} {% block breadcrumb %}
If you edit your address here, you have to verify your account again. You will be logged out after submitting this form.


Susan Bellamore Damour
Democrat
Retired - 2017 to present ------------------------------------------------------------- 2009 - 2017: Regional Administrator, U.S. General Services Administration, Rocky Mountain Region, Obama Administration
B.A. Colorado Woman's College
80220
Throughout my career, I have been fortunate to have positions where I am able to advocate for Coloradans to promote their full participation in the democracy. In my college years, I worked on civil rights issues. My support of women's equality and women's reproductive rights is demonstrated by my involvement in the effort to pass The Equal Rights Amendment and serve on the Board of Rocky Mountain Planned Parenthood. To promote women's full participation in Colorado, I was the Executive Director of The Women's Foundation of Colorado and served on the Board of the Women's Bean Project - both organizations serving low-income women on their path to self-sufficiency. My service on the staffs of a governor, a member of the U.S. House, U.S. Senate and two presidents gave me a chance to make the lives of all Coloradans better - everyday if I did my job well, I helped open doors to services provided by my government. As a reluctant retiree, I have volunteered at the Library to ensure that "The People's University" is open to all and able to provide services and materials to enrich its patrons' lives. I have organized volunteers to elect political candidates. I have gone door to door canvasing. I have given classes helping people better understand both sides of the issues on the Colorado ballot. To support a greater voice for citizens in national politics, I served on the Board of Colorado Common Cause and became the first straight person on the Colorado Human Rights Campaign Board - seeking equality for the LBGTQ community, and I was honored to be the first Anglo on the Latino Community Foundation of Colorado's Board, recognizing the growing and vibrant Hispanic population throughout the state and helping to meet their needs. My life has been richly textured by these experiences and serving on the commission will give me a chance to help shape a dynamic, fair and unbiased division of the state's population into congressional districts that will well represent us in the decade ahead. As a senior member of Congressman David Skaggs' staff in 1991, I got first hand knowledge of the process as the district was being reconfigured and the decisions affected the constituents and the Congressman.
Employment and Volunteer Political Activities: Governor's Staff, Richard D. Lamm - 1975-1977; 1982-1987. Congressional Staff, Congressman David E. Skaggs - 1991-1998 Regional Administrator, US General Services Administration - 1998-2001; 2009-2017 Staff member, U.S. Congressman and Senator Mark Udall - 2008 Volunteer, Colorado Attorney General Phil Weiser 2018 to present Volunteer, State Senator Chris Hansen, 2019 to present Volunteer, Denver Democratic Party, 2019 to present Volunteer, Andrew Romanoff for U.S. Senate, Colorado, 2019-2020 ------------------------------------------------------------------ Donations: 2017: Perlmutter for Colorado Governor 2018: Phil Weiser for Colorado Attorney General Jason Crow for U.S. House of Representatives - CD6 2019: Andrew Romanoff for U.S. Senate Chris Hansen for State House - HD6 Amy McGrath for U.S. Senate - Kentucky 2020: Michael Blake for CD15 - New York Give Green for U.S. President Joe Biden Biden Victory Fund for Joe Biden for President Diane M. Bush for U.S. House of Representatives -CD3 - Colorado Denver Democratic Party - Colorado Blue Momentum - Colorado State Senate small donor fund
2016-2019: Board Member, Friends of Chamber Music, Denver, CO 2000-present: Member, Women's Forum of Colorado, Inc. Denver, CO 1987-present: Member, Historic Montclair Community Association, Denver, CO 2016-2019: Member, Denver Mile HIgh Club, Denver, CO 2013-2019: Member, Committee on Aging, Rose Community Foundation, Denver, CO 2017-present: Member, Growth Committee, Denver Public Library Friends Foundation, Denver CO
I would like to serve on the commission to help create a carefully constructed compromise that proportions the congressional districts to best serve the constituents of Colorado in the U.S. House of Representatives.
It is most important to create trust and respect for all members of the group first and foremost. Valuing fellow commissioners and the varying perspectives they bring to the table is essential to the group being able to reach the successful goal. I believe that by reaching out, listening and learning from my colleagues without bias, I can be most successful. Being fair and impartial is a matter of personal character, which I hope I would bring to the table as well.
Through careful research, past experience, consulting with others of differing views and being eternally curious, I have tried to bring as much information to the table to make the best decisions. I also try to think how my current decisions will hold true going forward - one, five or ten years out - not just solve a problem in the immediate. Finally, I work to develop options, so that if Plan A doesn't work, we can move to Plan B and beyond - I work to be a multi-tiered thinker, realizing that complex problems need complex solutions that can be implemented as straightforwardly as possible.
female


Problem Definition
Tracking 3-dimensional objects in a two dimensional frame is a difficult task. For example, moving from frame to frame, the algorithm must not only detect objects of interest in each frame, but also associate current positions with past positions. The difficulty is compounded by many facets of 3d projection onto a two dimensional plane, such as occlusion and emergence of objects between frames. Here, we present an implementation that can largely track the movements of objects in two different datasets -- one with bats flying in and out of view, and the second with fish swimming in a tank.
Method and Implementation
The method I implemented for object tracking is largely agnostic between bat and fish datasets. The algorithm implements both kalman filters to estimate centers and the Hungarian algorithm to handle frame-by-frame assignment.
Object Detection
The algorithm begins by detecting centers in the first frame of the video. In the bat dataset this is provided to us. In the case of the fish dataset, fish are outlined in red. To find the center for each fish I first binarize the image by setting any pixel with red values > 100 and green levels < 75. I then performed dilation followed by erosion on the binary image to open up contours. Contours were detected using the opencv library. Centers were calculated for each contour using image moments. Because there were quite a bit of overlapping centers, due to erroneous contour detecting, only centers with a euclidean distance greater than 10, between already logged centers, were returned. It is important to note fish detection was not perfect, and in order to get a stronger contour, using my naive color filtering approach resulted in a fair number of false positive fishes.
Object Tracking
Once the first centers were isolated, trajectories, kalman filters, and
velocities for each center were initialized. Trajectories were constructed as
a vector of vectors containing points to observed centers. Kalman filters were
initialized with 1e-4 process and measurement noise, and a 4 x 4 transition
matrix that considered x and y positions along with x and y velocities. To
better estimate the cost of assigning a past center to a current center,
calculated velocities were used during distance matrix construction.
After the tracking variables had been initialized, the algorithm moved on to
the second frame of the dataset. Centers were again extracted in a dataset
dependent manner, and a distance matrix was calculted by finding the euclidean
distance between past points summed with their past velocity and new centers.
However, because no assignment has taken place between time frames yet,
velocities for the first frame are zero.
Once a distance matrix is constructed, the matrix is scanned for any past
centers with a minimum distance greater than 70. These centers are very far
from any current observed center, and it is likely they left the video in the
previous frame, and their associated trajectories are terminated.
Centers at the current frame are also scanned. If current centers display a
minimum distance greater than 70, it is assumed these centers represent new
objects to the video, and they have new trajectories and kalman filters
initialized.
After center filtering occurs, a new distance matrix is constructed that only
contains centers that passed the initial pruning step. This new cost matrix
is fed into the hungarian algorithm, from a third party library, to find the
past centers that best match with current centers. If the number of rows in
the matrix is greater than the number of columns, that means we have more
past centers than current centers. It is assumed this represents an occlusion
event, where two objects overlap. Therefore, a measurement for the unassigned
past center is estimated using its Kalman filter. In the case when the number
of columns is greater than the number of rows, we have more current centers
than past centers. It was decided this likely represents an emergence event.
Whereby a previously occluded object emerges from behind another object. In
this case, we treat the center as a new object, and initialize a new
trajectory. Creating a whole new trajectory might be cause for concern because
it is possible we will be starting a trajectory for a previously tracked
object (e.g. two separate objects occluded and then separated). However,
because we use kalman filters during occlusion events to estimate centers,
we don't actually lose trajectories during occlusion. Thus, we can be
reasonably confident we are not starting a new trajectory for a previously
tracked object.
Once centers have been assigned, either through Hungarian matching with
observed centers or estimated Kalman centers, trajectories are updated,
current velocities are calculated, and Kalman filters are looped. This process
repeats the for every frame left in the video.
The algorithm to analyze the bat dataset is called by the function analyze_bats(). Similarly, to analyze the fish dataset, the analyze_fish() function is called. Neither require parameters as file location is assumed. Helper methods are well documented in the file tracking.hpp. However, their definitions are left out of this report for the sake of brevity.
Experiments
No real experiments were conducted. Trajectories were plotted next to observed centers and binary images to visualize tracking success. No quantitavie measures were taken, and results are largely qualitative and based on the eye test.
Results
List your experimental results. Provide examples of input images and output images. If relevant, you may provide images showing any intermediate steps. If your work involves videos, do not submit the videos but only links to them.
Results | ||
| Source Image | Trajectories | |
| Bats | ||
| Fish |
Discussion
While some erroneous trajectories do take place during tracking, I believe by
and large the tracking algorithm implemented here is largely successful in
tracking the images both bat and fish movement. Specifically, the tracking of
fish trajectories is particularly successful.
The bat dataset provided many difficulties for tracking: individuals would leave
the frame, other indivuals would spontaneously enter, and the smaller bats far
in the distance provided dense clumps of difficult to track bats. The most
successful cases, in terms of dealing with occluded or appearing objects, were
close bats with longer trajectories. The algorithm's ability to correctly deal
with occluded objects is directly related to how long the trajectory has been
tracked before occlusion. Longer trajectories are well tracked, but shorter
trajectories are sometimes lost. The length of occlusion also affects how well
trajectory tracking occurs. Objects that are occluded longer can display poor
tracking where projected centers remain at the point of occlusion rather than
following the occluded trajectory. As a plus, the pruning stage before center
assignment seems to have an exceptionally high positive predictive value for
new bats entering the scene and old bats leaving the scene. It should be noted
that some spurious trajectories are randomly tracked in the bats. The exact
cause of this is still a mystery, thoguh it could be due to a combination of
necessitating as many assignment as possible during the hungarian algorithm.
When the algorithm to track bats was applied to fish, the results were largely exceptional. Occluded objects were tracked well, and predicted centers seem to track individuals even when they are not outlined in red. The largest contributor to poor performance during the fish dataset appears to be object detection. Many false positive arise during tracking, which is largely due to a generous green-level threshold to produce more robust contours. It is relatively common for false-postive centers to interfere with true centers and their trajectories; however, trackin is still largely successful.
Conclusions
For the first pass at a tracking algorithm, I believe I was largely successful. However, improvements can definitely be made. There is an implicit assumption in the algorithm that occlusion and separation events cannot occur at the same time. This is due to the fact that occlusion and appearance events are determined by comparing the number of rows and columns in the filtered distance matrix. A vast improvement in trackign could likely occurr if iterations or modifications occurred during center assignment. For example, modifying the hungarian algorithm to throw out centers if it greatly reduces the cost of center assignment as a whole, would help identify centers that have occluded or appeared in a single frame. Further, improvements could be made in kalman filter intialization. For example, smaller bats in the lower right hand corner of videos, likely display a greater noise-to-signal ratio in their center measurements than larger bats. Initializing filters depending on the number of centers nearby, might help capture this increased noise.
Credits and Bibliography
Implementation for the hungarian algorithm was taken from a third party library hosted here.
This was an individual assignment with no additional group members


Answer TRUE or FALSE.
A IARRAY[IT] is another name for a IARRAY[NOTIT].
A binary tree is either empty, or else a root node with two binary trees as children.
A BST (binary search tree) is a binary tree where, for every node N, all nodes in the left subtree of N have key values less than the key value of N, and all nodes in the right subtree of N have key values greater than the key value of N.


Both boxes should move infinitely from left to right.
The blue version actually consists of two different boxes (light and dark). The transition between is triggered via webkitAnimationEnd event and a simple handler code.
Red vs blue discrepancy delay: -- ms.


2 writes to ExperimentalFeatures
System.Management.Automation (2)
engine\Modules\ModuleCmdletBase.cs (2)
2539
manifestInfo.
ExperimentalFeatures
= new ReadOnlyCollection<ExperimentalFeature>(expFeatureList);
3173
newManifestInfo.
ExperimentalFeatures
= manifestInfo.ExperimentalFeatures;
1 reference to ExperimentalFeatures
System.Management.Automation (1)
engine\Modules\ModuleCmdletBase.cs (1)
3173
newManifestInfo.ExperimentalFeatures = manifestInfo.
ExperimentalFeatures
;


If login form does not appear — Change API ERVER & APPLICATION KEY / SECRET.
In order to use this demo your application must have:
© 1999 – RingCentral™, Inc. All rights reserved.


|
ConfigurableRequirementsInterface.php
|
<?php
/*
* This file is part of the Symfony package.
*
* (c) Fabien Potencier<EMAIL_ADDRESS>*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/
namespace Symfony\Component\Routing\Generator;
/**
* ConfigurableRequirementsInterface must be implemented by URL generators that
* can be configured whether an exception should be generated when the parameters
* do not match the requirements. It is also possible to disable the requirements
* check for URL generation completely.
*
* The possible configurations and use-cases:
* - setStrictRequirements(true): Throw an exception for mismatching requirements. This
* is mostly useful in development environment.
* - setStrictRequirements(false): Don't throw an exception but return an empty string as URL for
* mismatching requirements and log the problem. Useful when you cannot control all
* params because they come from third party libs but don't want to have a 404 in
* production environment. It should log the mismatch so one can review it.
* - setStrictRequirements(null): Return the URL with the given parameters without
* checking the requirements at all. When generating a URL you should either trust
* your params or you validated them beforehand because otherwise it would break your
* link anyway. So in production environment you should know that params always pass
* the requirements. Thus this option allows to disable the check on URL generation for
* performance reasons (saving a preg_match for each requirement every time a URL is
* generated).
*
* @author Fabien Potencier<EMAIL_ADDRESS>* @author Tobias Schultze <http://tobion.de>
*/
interface ConfigurableRequirementsInterface
{
/**
* Enables or disables the exception on incorrect parameters.
* Passing null will deactivate the requirements check completely.
*/
public function setStrictRequirements(?bool $enabled);
/**
* Returns whether to throw an exception on incorrect parameters.
* Null means the requirements check is deactivated completely.
*
* @return bool|null
*/
public function isStrictRequirements();
}


10-bit Clock Widget uses 10 binary dots and their color, to represent current time in 12-hour format. In the following image, first 4 dots (before separator line) represents hour (0-11), other 6 dots represent minutes (0-59) and color of these dots denotes A.M./P.M. as defined in widget settings. Furthermore, dim dots are 0s and bright dots are 1s, so in following image, first 4 dots denote 1010 (10) and other 6 dots denote 010011 (19).


{% extends "base.html" %} {% comment %} Copyright 2010 Google Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. {% endcomment %} {% load i18n %} {% block content %}
utcnow debug value:
| Currently: | {{ utcnow }} |
| Previously: | {{ utcbefore }} |


Exploring Identity Through
Cuban Sound and Russian Art:
Only a city like New York could give life to Culture Links. Blending the city’s diversity into unique multi-cultural
experiences that are expressed through the arts and technology, Culture Links provides New Yorkers with a
sophisticated space to explore identity and culture.
Event Information
WHEN
Monday, December 18th
7:00 pm - 8:30 pm
WHERE
Caelum Gallery
526 W 26th St #315, New York, NY 10001
An Unforgettable Evening
Below are the scheduled events for the evening.
-
Silent Auction
Auction of paintings and mix media artwork by Moscow's own
Viktoriya Basina
Victoriya Basina -
Live Music
Live Afro Cuban rhythms
David Oquendo
David Oquendo. -
Emcee
Entertainment by Kevin Thompson
XX
Kevin Thompson -
Cocktails
Open bar serving delicious Vodka cocktails
-
Hors d'ouvres
Cuban appetizers
-
9.00 AM
Breakfast
-
10.00 AM
Advanced SVG Animations
By Karl Groves -
11.00 AM
Presenting Work with Confidence
By Sarah Dransner -
12.00 AM
Keynote on UX & UI Design
By Ned Stark -
1.00 PM
The End
-
9.00 AM
Breakfast
-
10.00 AM
Advanced SVG Animations
By Karl Groves -
11.00 AM
Presenting Work with Confidence
By Sarah Dransner -
12.00 AM
Keynote on UX & UI Design
By Ned Stark -
1.00 PM
The End
David Oquendo
MUSICIAN
Born in Havana, Cuba, David Oquendo is a self taught musician who absorbed the essence of the Afro Cuban rhythmical vernacular from his native city. Mr. Oquendo has worked with numerous artists such as Paquito D’Rivera, Marc Anthony, Johnny Ventura, Ray Barreto. He is a guitarist, singer, composer, arranger and bassist, and has performed and recorded in Cuba, Panama , Dominican Republic, Austria, Canada Greece and the U.S. He is founder and director of the Afro Cuban folklore group “Raices Habaneras”, which has been performing every Sunday since 1996. He was the musical director and producer for “The Cuban Rumba All Stars".
Viktoriya Basina
ARTIST
Viktoriya Basina’s work is inspired by folk and sacred art of various cultures, in particular by the Russian icons as well as by contemporary urban landscape, analytical art, and human psychology. These themes, along with addressing the conflict between collectively inherited archetypes, and an existential world of a human being, preoccupy her pursuit for making mixed-media imagery in a contemporary context.
Kevin Thompson
EMCEE
Kevin Thompson’s work is inspired by folk and sacred art of various cultures, in particular by the Russian icons as well as by contemporary urban landscape, analytical art, and human psychology. These themes, along with addressing the conflict between collectively inherited archetypes, and an existential world of a human being, preoccupy her pursuit for making mixed-media imagery in a contemporary context.
Join Us
Join us for this one of a kind evening of culture, soul and social impact. All proceeds collected from the evening are tax-deductible and will support innovative education initiatives in Harlem, home to one of the city’s most underserved education communities.
Location
Infinito Studio Gallery, located in Tribeca.
79 Leonard St, New York, NY 10013
(Between Church and Broadway)
Nearby Trains:
1 to Franklin Street Station
N, Q, R, W to Canal Street Station
Nearby Buses:
M55
X10, X10B, X11, X12, X17, 19
Techrow Fund
The TechRow Fund is a 501c3 that supports and develops innovative education programs that support Harlem’s district 5 through immersive technologies. Founded in 2016, the Fund has supported Columbia University’s public programs at the Zuckerman Institute, District 5’s STEM expo and is a member of the Harlem Renaissance Educational Pipeline. The Fund has reached over 300 children with VR programming dedicated towards improving learning experiences and education outcomes.


|
|
|
|
|
DOCKET NUMBER |
PLAINTIFF TRIBE |
PAGE NUMBER |
|
101
|
The Citizen Band of Potawatomi Indians of Oklahoma, et al.
Findings of Fact
Opinion of the Commission
Final Order
|
1
6
9-a
|
|
75
|
Tillamook Band of Tillamooks, Naalem (Na-ha-lum) Band of Tillamooks, Clatsop Tribe, Kathlamet Band of Chinooks, Nue-Quee-Clah-We-Muck Tribe, the Confederated Tribes of Siletz Indians, Confederated Tribes of the Grand Ronde Community, Oregon, and portions and descendants of all such tribes and bands
Order Allowing Fees to the Attorneys for the Tillamook Band of Tillamooks and the Naalem (Ne-ha-lum) Band of Tillamooks, and Allowing the Claim of those Attorneys for Certain Sums Advanced and Paid by them to and for James Hatfield, an expert witness
|
10
|
|
264
|
The Confederated Tribes of the Umatilla Indian Reservation
Findings of Fact
Opinion of the Commission
Interlocutory Order
|
14
104
171-a
|
|
29-L, 29-M, 29-O, 29-P, Consolidated with, 15-N, 15-O, 15-R, 128, 309, 124-B, 254, 314-B, 310 |
Hannahville Indian Community, et al
Opinion Petitioners Motion to Admit Additional Evidence
Order Denying the Admission of Additional Evidence in Docket No. 29-I (Consolidated with Docket Nos. 216, 15-L et al.), Docket No. 29-J (Consolidated with Docket Nos. 217, 15-K et al.), and Docket No. 29-K (Consolidated with Docket Nos. 146, 15-M et al.)
|
172
203-a
|
|
29-D, 29-E |
Hannahville Indians, et al.
Opinion on Defendant's Motions to Dismiss
Order Denying Petitioners' Motion to Insert the Name of "Albert Mackety" as Party Plaintiff and Granting Defendant's Motions to Dismiss the Petitions
|
204
220-a
|
|
46 |
The Nooksack Tribe of Indians on relation of Joseph Louis, Chairman of the General Council
Findings of Fact in the Matter of Application by Attorney for Allowance of Attorney Fee and Expenses Incurred in the Prosecution of the Claim
Order Allowing Attorney Fee and Reimbursable Expenses
|
221
225-a
|
|
18-B, 18-N
|
Minnesota Chippewa Tribe, White Earth Band, Leech Lake Band, Mille Lac Band, Ed Wilson, James Davis, John Corbow, William Morell, Harold Emerson, Joseph Morrison, Ole Same, Monroe Skinaway, Eugene Reynolds, Frank La Rose, Joseph Monroe, Archie Libby and John Squirrel
Additional Findings of Fact
Opinion of the Commission
Interlocutory Order, with Respect to Docket No. 18-B and Final Order, with Respect to Docket No. 18-N
|
226
294
328-a
|
|
29-K 146 15-M 40-K
|
Hannahville Indian Community, Wilson, Michigan, Forest County Potawatomi Community, Crandon, Wisconsin, et al. Citizen Band of Potawatomi Indians of Oklahoma, et al. Potawatomie Tribe of Indians, the Prairie Band of the Potawatomie Tribe of Indians, et al. Robert Dominic, et al., as the representatives and on behalf of all members by blood of the Ottawa Tribe of Indians
Commission's Opinion Accompanying Order Denying Petitioners' Motion in Docket No. 29-K, and for other purposes
Order Dismissing the Petition in Docket No. 29-K and for other purposes
Order Denying Petitioners in Docket No. 29-K "Motion to Amend Findings of Fact of the Commission, Dated June 30, 1958"
|
329
359-a
359-d
|
|
18-U
|
Minnesota Chippewa Tribe, et al.
Findings of Fact
Opinion of the Commission
Order Granting Petitioners' Motion for Summary Judgment
|
360
368
374-a
|
|
256 Consolidate With 124-D, 124-E, 124-F
|
The Miami Tribe of Oklahoma, also known as the Miami Tribe Harley T. Palmer, Frank C. Pooler and David Leonard, as representatives of the Miami Tribe and all members thereof Ira S. Godfroy, et al., Miami Indians of Indiana
Findings of Fact
Opinion of the Commission
Interlocutory Order
|
375
434
483-a
|
|
11-A |
Otoe and Missouria Tribe of Indians
Order Allowing Reimbursement of Attorney Expenses
|
483-c
|
|
248
|
The Seminole Nation
Findings of Fact
Opinion of the Commission
Final Award
|
484
505
517-a
|
|
146 15-M 40-K
|
Citizen Band of Potawatomi Indians of Oklahoma, et al. The Prairie Band of the Potawatomi Tribe of Indians, et al. Robert Dominic, et al., as the representatives and on behalf of all members by blood of the Ottawa Tribe of Indians
Additional Findings of Fact
Opinion of the Commission
Second Interlocutory Order
|
518
549
569-a
|
|
96
|
The Citizen Band of Potawatomi Indian of Oklahoma
Supplemental Findings of Fact
Opinion of the Commission
Interlocutory Order
|
570
585
607-a
|
|
67 124 (Consolidated)
|
The Miami Tribe of Oklahoma, also known as the Miami Tribe Harley T. Palmer, Frank D. Pooler and David Leonard, representatives of the Miami Tribe and all members thereof
Opinion of the Commission
Order Allowing Attorneys' Reimbursable Expenses
|
608
617-a
|
|
88, 330, 330-A
|
The Southern Paiute Nation, et al.
Findings of Fact
Opinion of the Commission
Final Judgment
Order Allowing Attorney Fees
|
618
647
650-a
650-b
|
|
351, 351-A
|
The Chemehuevi Tribe of Indians, the Chemehuevi Tribe of Indians by Dan Eddy
Findings of Fact
Opinion of the Commission
Final Judgment
|
651
673
676-a
|
|
303
|
The Ottawa Tribe and Guy Jennison, Bronson Edwards and Gene Jennison, as representatives of the Ottawa Tribe
Amended Final Award
|
677
|
|
327
|
The Confederated Bands of the Ute Indians
Additional Findings of Fact on the Stipulated Settlement of Offsets
Opinion of the Commission
Final Judgment
|
679
704
706-a
|
|
349
|
Ute Indian Tribe of the Uintah and Ouray Reservation, an Indian Reorganization Act Corporation, for and on behalf of the Uncompahgre Band of Ute Indians
Findings of Fact on the Stipulated Settlement of Claims and Offsets
Opinion of the Commission
Final Judgment
|
707
725
728-a
|
|
157
|
Shoshone Tribe of Indians of the Wind River Reservation, Wyoming
Findings of Fact on Compromise of the Claim Set Forth in Docket No. 157
Opinion of the Commission
Final Judgment
|
729
744
746-a
|
|


Protect your cryptographic keys by storing them away from data, in a secured key vault. Ensure there is a gap between threat vectors that have access to application or data, and attack vectors that have potential access to crypto keys.
Keys should remain in the key vault at all times. If the cryptographic keys are unavailable, any compromised data is unusable by a thief.
Any crypto key used must be protected from unauthorized access. Define a key lifecycle so that it is rekeyed, and then removed from use on a prescribed schedule. The lifecycle you define for security of crypto keys will tell you when to remove the key from use.
When your application uses multiple keys, their makeup must be independent, and not related. All keys must be stored away from the data, so that a compromise will not result in the theft of the key and data.
AWS encourages the use of a hardware security module (HSM). Such an appliance provides secure key storage and cryptographic operations in the cloud. This is an overview of steps you will take to get started with the HSM.
Set up the HSM environment. Setup your account in the administrators group and generate your access key. Create an IAM policy to use CloudHSM and attach it to users. Setup private and public subnets, IAM role, EC2 instance, and security group in your VPC environment. Now generate SSH key pair and setup the CLI tools.
Provision the HSM. Use the CLI to create an HSM from your control instance with the create-hsm
command. You will need to supply subnet_id
, public_key_file
, iam_role_arn
, and syslog_ip_address
from the previous step.
Configure the HSM. With the CLI, use list-hsms
and describe-hsm
to get the HSM ENI Identifier and IP Address. Apply the security group through the EC2 console. In the CLI use initialize-hsm
and required parameters.
Connect the HSM. Connect the on-premises HSM appliances to your HSM VPC.
Configure the HSM client. Download and install a Linux or Windows client with required options for the appliance.
Configure HA. For information about high availability and load balancing options and setup, see http://docs.aws.amazon.com/cloudhsm/latest/userguide/.
Integrate CloudHSM with the application. For information about using transparent data encryption with third-party software applications, see http://docs.aws.amazon.com/cloudhsm/latest/userguide/. A complete sample application is provided at the link.


Tutorial¶
Table of Contents¶
Extended Tutorial¶
A simple example¶
# also: _hourly, _daily, _weekly, or _yearly
from autots.datasets import load_monthly
df_long = load_monthly(long=True)
from autots import AutoTS
model = AutoTS(
forecast_length=3,
frequency='infer',
ensemble='simple',
max_generations=5,
num_validations=2,
)
model = model.fit(df_long, date_col='datetime', value_col='value', id_col='series_id')
# Print the description of the best model
print(model)
Import of data¶
There are two shapes/styles of pandas.DataFrame
which are accepted.
The first is long data, like that out of an aggregated sales-transaction table containing three columns identified to .fit()
as date_col {pd.Datetime}, value_col {the numeric or categorical data of interest}, and id_col {id string, if multiple series are provided}
.
Alternatively, the data may be in a wide format where the index is a pandas.DatetimeIndex
, and each column is a distinct data series.
If horizontal style ensembles are used, series_ids/column names will be coerced to strings.
You can tailor the process in a few ways…¶
The simplest way to improve accuracy is to increase the number of generations max_generations=15
. Each generation tries new models, taking additional time but improving the accuracy. The nature of genetic algorithms, however, means there is no consistent improvement for each generation, and large number of generations will often only result in minimal performance gains.
Another approach that may improve accuracy is to set ensemble='all'
. Ensemble parameter expects a single string, and can for example be 'simple,dist'
, or 'horizontal'
. As this means storing more details of every model, this takes more time and memory.
A handy parameter for when your data is expected to always be 0 or greater (such as unit sales) is to set no_negatives=True
. This forces forecasts to be greater than or equal to 0.
A similar function is constraint=2.0
. What this does is prevent the forecast from leaving historic bounds set by the training data. In this example, the forecasts would not be allowed to go above max(training data) + 2.0 * st.dev(training data)
, as well as the reverse on the minimum side. A constraint of 0
would constrain forecasts to historical mins and maxes.
Another convenience function is drop_most_recent=1
specifing the number of most recent periods to drop. This can be handy with monthly data, where often the most recent month is incomplete.
drop_data_older_than_periods
provides similar functionality but drops the oldest data to speed up the process on large datasets.
remove_leading_zeroes=True
is useful for data where leading zeroes represent a process which has not yet started.
When working with many time series, it can be helpful to take advantage of subset=100
. Subset specifies the interger number of time series to test models on, and can be useful with many related time series (1000’s of customer’s sales). Usually the best model on a 100 related time series is very close to that tested on many thousands (or more) of series.
Subset takes advantage of weighting, more highly-weighted series are more likely to be selected. Weighting is used with multiple time series to tell the evaluator which series are most important. Series weights are assumed to all be equal to 1, values need only be passed in when a value other than 1 is desired. Note for weighting, larger weights = more important.
Probably the most likely thing to cause trouble is having a lot of NaN/missing data. Especially a lot of missing data in the most recent available data.
Using appropriate cross validation (backwards
especially if NaN is common in older data but not recent data) can help.
Dropping series which are mostly missing, or using prefill_na=0
(or other value) can also help.
What to Worry About¶
There are some basic things to beware of that can commonly lead to poor results:
Bad data (sudden drops or missing values) in the most recent data is the single most common cause of bad forecasts here. As many models use the most recent data as a jumping off point, error in the most recent data points can have an oversized effect on forecasts.
Misrepresentative cross-validation samples. Models are chosen on performance in cross validation. If the validations don’t accurately represent the series, a poor model may be chosen. Choose a good method and as many validations as possible.
Anomalies that won’t be repeated. Manual anomaly removal can be more effective than any automatic methods. Along with this, beware of a changing pattern of NaN occurrences, as learned FillNA may not longer apply.
Artifical historical events, a simple example being sales promotions. Use of regressors is the most common method for dealing with this and may be critical for modeling these types of events.
What you don’t need to do before the automated forecasting is any typical preprocessing. It is best to leave it up to the model selection process to choose, as different models do better with different types of preprocessing.
Validation and Cross Validation¶
Cross validation helps assure that the optimal model is stable over the dynamics of a time series. Cross validation can be tricky in time series data due to the necessity of preventing data leakage from future data points.
Firstly, all models are initially validated on the most recent piece of data. This is done because the most recent data will generally most closely resemble the forecast future.
With very small datasets, there may be not be enough data for cross validation, in which case num_validations
may be set to 0. This can also speed up quick tests.
Note that when num_validations=0
one evaluation is still run. It’s just not cross validation. num_validations
is the number of cross validations to be done in addition.
In general, the safest approach is to have as many validations as possible, as long as there is sufficient data for it.
Here are the available methods:
Backwards cross validation is the safest method and works backwards from the most recent data. First the most recent forecast_length samples are taken, then the next most recent forecast_length samples, and so on. This makes it more ideal for smaller or fast-changing datasets.
Even cross validation slices the data into equal chunks. For example, num_validations=3
would split the data into equal, progressive thirds (less the original validation sample). The final validation results would then include four pieces, the results on the three cross validation samples as well as the original validation sample.
Seasonal validation is supplied as 'seasonal n'
ie 'seasonal 364'
. This is a variation on backwards
validation and offers the best performance of all validation methods if an appropriate period is supplied.
It trains on the most recent data as usual, then valdations are n
periods back from the datetime of the forecast would be.
For example with daily data, forecasting for a month ahead, and n=364
, the first test might be on May 2021, with validation on June 2020 and June 2019, the final forecast then of June 2021.
Similarity automatically finds the data sections most similar to the most recent data that will be used for prediction. This is the best general purpose choice but currently can be sensitive to messy data.
Custom allows validations of any type. If used, .fit() needs validation_indexes
passed - a list of pd.DatetimeIndex’s, tail of forecast_length of each is used as test (which should be of the same length as num_validations
+ 1).
backwards
, even
and seasonal
validation all perform initial evaluation on the most recent split of data. custom
performs initial evaluation on the first index in the list provided, while similarity
acts on the closest distance segment first.
Only a subset of models are taken from initial validation to cross validation. The number of models is set such as models_to_validate=10
.
If a float in 0 to 1 is provided, it is treated as a % of models to select.
If you suspect your most recent data is not fairly representative of the whole, it would be a good idea to increase this parameter.
However, increasing this value above, say, 0.35
(ie 35%) is unlikely to have much benefit, due to the similarity of many model parameters.
While NaN values are handled, model selection will suffer if any series have large numbers of NaN values in any of the generated train/test splits.
Most commonly, this may occur where some series have a very long history, while others in the same dataset only have very recent data.
In these cases, avoid the even
cross validation and use one of the other validation methods.
Another Example:¶
Here, we are forecasting the traffice along Interstate 94 between Minneapolis and St Paul in Minnesota. This is a great dataset to demonstrate a recommended way of including external variables - by including them as time series with a lower weighting.
Here weather data is included - winter and road construction being the major influencers for traffic and will be forecast alongside the traffic volume. These additional series carry information to models such as RollingRegression
, VARMAX
, and VECM
.
Also seen in use here is the model_list
.
from autots import AutoTS
from autots.datasets import load_hourly
df_wide = load_hourly(long=False)
# here we care most about traffic volume, all other series assumed to be weight of 1
weights_hourly = {'traffic_volume': 20}
model_list = [
'LastValueNaive',
'GLS',
'ETS',
'AverageValueNaive',
]
model = AutoTS(
forecast_length=49,
frequency='infer',
prediction_interval=0.95,
ensemble=['simple', 'horizontal-min'],
max_generations=5,
num_validations=2,
validation_method='seasonal 168',
model_list=model_list,
transformer_list='all',
models_to_validate=0.2,
drop_most_recent=1,
n_jobs='auto',
)
model = model.fit(
df_wide,
weights=weights_hourly,
)
prediction = model.predict()
forecasts_df = prediction.forecast
# prediction.long_form_results()
Probabilistic forecasts are available for all models, but in many cases are just data-based estimates in lieu of model estimates.
upper_forecasts_df = prediction.upper_forecast
lower_forecasts_df = prediction.lower_forecast
Model Lists¶
By default, most available models are tried. For a more limited subset of models, a custom list can be passed in, or more simply, a string, one of 'probabilistic', 'multivariate', 'fast', 'superfast', or 'all'
.
A table of all available models is below.
On large multivariate series, DynamicFactor
and VARMAX
can be impractically slow.
Deployment and Template Import/Export¶
Take a look at the production_example.py
Many models can be reverse engineered with (relative) simplicity outside of AutoTS by placing the choosen parameters into Statsmodels or other underlying package.
Following the model training, the top models can be exported to a .csv
or .json
file, then on next run only those models will be tried.
This allows for improved fault tolerance (by relying not on one, but several possible models and underlying packages), and some flexibility in switching models as the time series evolve.
One thing to note is that, as AutoTS is still under development, template formats are likely to change and be incompatible with future package versions.
# after fitting an AutoTS model
example_filename = "example_export.csv" # .csv/.json
model.export_template(example_filename, models='best',
n=15, max_per_model_class=3)
# on new training
model = AutoTS(forecast_length=forecast_length,
frequency='infer', max_generations=0,
num_validations=0, verbose=0)
model = model.import_template(example_filename, method='only') # method='add on'
print("Overwrite template is: {}".format(str(model.initial_template)))
Running Just One Model¶
While the above version of deployment, with evolving templates and cross_validation on every run, is the recommended deployment, it is also possible to run a single fixed model.
Coming from the deeper internals of AutoTS, this function can only take the wide
style data (there is a long_to_wide function available).
Data must already be fairly clean - all numerics (or np.nan).
This will run Ensembles, and as such is generally recommended over loading the models directly. Subsidiary models use the sklearn format.
from autots import load_daily, model_forecast
df = load_daily(long=False) # long or non-numeric data won't work with this function
df_forecast = model_forecast(
model_name="AverageValueNaive",
model_param_dict={'method': 'Mean'},
model_transform_dict={
'fillna': 'mean',
'transformations': {'0': 'DifferencedTransformer'},
'transformation_params': {'0': {}}
},
df_train=df,
forecast_length=12,
frequency='infer',
prediction_interval=0.9,
no_negatives=False,
# future_regressor_train=future_regressor_train2d,
# future_regressor_forecast=future_regressor_forecast2d,
random_seed=321,
verbose=0,
n_jobs="auto",
)
df_forecast.forecast.head(5)
The model.predict()
of AutoTS class runs the model given by three stored attributes:
model.best_model_name,
model.best_model_params,
model.best_model_transformation_params
If you overwrite these, it will accordingly change the forecast output.
Metrics¶
There are a number of available metrics, all combined together into a ‘Score’ which evaluates the best model. The ‘Score’ that compares models can easily be adjusted by passing through custom metric weights dictionary. Higher weighting increases the importance of that metric, while 0 removes that metric from consideration. Weights must be numbers greater than or equal to 0. This weighting is not to be confused with series weighting, which effects how equally any one metric is applied to all the series.
metric_weighting = {
'smape_weighting': 5,
'mae_weighting': 2,
'rmse_weighting': 2,
'made_weighting': 0.5,
'mage_weighting': 1,
'mle_weighting': 0,
'imle_weighting': 0,
'spl_weighting': 3,
'containment_weighting': 0,
'contour_weighting': 1,
'runtime_weighting': 0.05,
}
model = AutoTS(
forecast_length=forecast_length,
frequency='infer',
metric_weighting=metric_weighting,
)
It is best to use several metrics for several reasons. The first is to avoid overfitting - a model that does well on many metrics is less likely to be overfit. Secondly, forecasts often have to meet multiple expectations. Using a composite score allows balancing the quality of point forecast, quality of probabilistic forecast, overestimation or underestimation, visual fit, and speed of runtime.
Some metrics are scaled and some are not. MAE, RMSE, MAGE, MLE, iMLE are unscaled and accordingly in multivariate forecasting will favor model performance on the largest scale input series.
Horizontal style ensembles use metric_weighting
for series selection, but only the values passed for mae, rmse, made, mle, imle, contour, spl
. If all of these are 0, mae is used for selection.
Accordingly it may be better to reduce the use ofsmape
, containment
, and mage
weighting when using these ensembles. With univariate models, runtime for overall won’t translate to runtime inside a horizontal ensemble.
sMAPE
is Symmetric Mean Absolute Percentage Loss and is generally the most versatile metric across multiple series as it is scaled. It doesn’t handle forecasts with lots of zeroes well.
SPL
is Scaled Pinball Loss, sometimes called Quantile Loss, and is the optimal metric for optimizing upper/lower quantile forecast accuracies.
Containment
measures the percent of test data that falls between the upper and lower forecasts, and is more human readable than SPL. Also called coverage_fraction
in other places.
MLE
and iMLE
are Mean Logarithmic Error inspired by the mean squared log error
. They are used to target over or underestimation with MAE of the penalized direction and log(error) for the less-penalized (and less outlier sensitive) direction.
MLE
penalizes an under-predicted forecast greater than an over-predicted forecast.
iMLE
is the inverse, and penalizes an over-prediction more.
MAGE
is Mean Absolute aGgregate Error which measures the error of a rollup of the forecasts. This is helpful in hiearchial/grouped forecasts for selecting series that have minimal overestimation or underestimation when summed.
Contour
is designed to help choose models which when plotted visually appear similar to the actual. As such, it measures the % of points where the forecast and actual both went in the same direction, either both up or both down, but not the magnitude of that difference. It is more human-readable than MADE for this information.
This is similar to but faster than MDA (mean directional accuracy) as contour evaluates no change as a positive case.
MADE
is (Scaled) Mean Absolute Differential Error. Similar to contour, it measures how well similar a forecast changes are to the timestep changes in the actual. Contour measures direction while MADE measures magnitude. Equivalent to ‘MAE’ when forecast_length=1. It is better for optimization than contour.
The contour and MADE metrics are useful as they encourages ‘wavy’ forecasts, ie, not flat line forecasts. Although flat line naive or linear forecasts can sometimes be very good models, they “don’t look like they are trying hard enough” to some managers, and using them favors non-flat forecasts that (to many) look like a more serious model.
import matplotlib.pyplot as plt
model = AutoTS().fit(df)
prediction = model.predict()
prediction.plot(
model.df_wide_numeric,
series=model.df_wide_numeric.columns[2],
remove_zeroes=False,
start_date="2018-09-26",
)
plt.show()
model.plot_per_series_mape(kind="pie")
plt.show()
model.plot_per_series_error()
plt.show()
model.plot_generation_loss()
plt.show()
if model.best_model_ensemble == 2:
model.plot_horizontal_per_generation()
plt.show()
model.plot_horizontal_transformers(method="fillna")
plt.show()
model.plot_horizontal_transformers()
plt.show()
model.plot_horizontal()
plt.show()
if "mosaic" in model.best_model["ModelParameters"].iloc[0].lower():
mosaic_df = model.mosaic_to_df()
print(mosaic_df[mosaic_df.columns[0:5]].head(5))
if False: # slow
model.plot_backforecast(n_splits="auto", start_date="2019-01-01")
Hierarchial and Grouped Forecasts¶
Hiearchial and grouping refer to multivariate forecast situations where the individual series are aggregated. A common example of this is product sales forecasting, where individual products are forecast and then also aggregated for a view on demand across all products. Aggregation combines the errors of individual series, however, potentially resulting in major over- or -under estimation of the overall demand. Traditionally to solve this problem, reconciliation is used where a top-level and lower-level forecasts are averaged or otherwise adjusted to produce a less exaggerated final result.
Unfortunately, any reconciliation approach is inherently sub-optimal. On real world data with optimized forecasts, the error contributions of individual series and the direction of the error (over- or under- estimate) are usually unstable, not only from forecast to forecast but from timestep to timestep inside each forecast. Thus reconciliation often reassigns the wrong amount of error to the wrong place.
The suggestion here for this problem is to target the problem from the beginning and utilize the MAGE
metric across validations.
This assesses how well the forecasts aggregate, and when used as part of metric_weighting drives model selection towards forecasts that aggregate well.
MAGE
assesses all series present, so if very distinct sub-groups are present, it may be sometimes necessary to model those groups in separate runs.
Additionally, MLE
or iMLE
can be used if either underestimation or overestimation respectively has been identified as a problem.
Ensembles¶
Ensemble methods are specified by the ensemble=
parameter. It can be either a list or a comma-separated string.
simple
style ensembles (labeled ‘BestN’ in templates) are the most recognizable form of ensemble and are the simple average of the specified models, here usally 3 or 5 models.
distance
style ensembles are two models spliced together. The first model forecasts the first fraction of forecast period, the second model the latter half. There is no overlap of the models.
Both simple
and distance
style models are constructed on the first evaluation set of data, and run through validation along with all other models selected for validation.
Both of these can also be recursive in depth, containing ensembles of ensembles. This recursive ensembling can happen when ensembles are imported from a starting template - they work just fine, but may get rather slow, having lots of models.
horizontal
ensembles are the type of ensembles for which this package was originally created.
With this, each series gets its own model. This avoids the ‘one size does not fit all’ problem when many time series are in a dataset.
In the interest of efficiency, univariate models are only run on the series they are needed for.
Models not in the no_shared
list may make horizontal ensembling very slow at scale - as they have to be run for every series, even if they are only used for one.
horizontal-max
chooses the best series for every model. horizontal
and horizontal-min
attempt to reduce the number of slow models chosen while still maintaining as much accuracy as possble.
A feature called horizontal_generalization
allows the use of subset
and makes these ensembles fault tolerant.
If you see a message no full models available
, however, that means this generalization may fail. Including at least one of the superfast
or a model not in no_shared
models usually prevents this.
These ensembles are choosen based on per series accuracy on mae, rmse, contour, spl
, weighted as specified in metric_weighting
.
horizontal
ensembles can contain recursive depths of simple
and distance
style ensembles but horizontal
ensembles cannot be nested.
mosaic
enembles are an extension of horizontal
ensembles, but with a specific model choosen for each series and for each forecast period.
As this means the maximum number of models can be number of series * forecast_length
, this obviously may get quite slow.
Theoretically, this style of ensembling offers the highest accuracy.
They are much more prone to over-fitting, so use this with more validations and more stable data.
Unlike horizontal
ensembles, which only work on multivariate datasets, mosaic
can be run on a single time series with a horizon > 1.
One thing you can do with mosaic
ensembles if you only care about the accuracy of one forecast point, but want to run a forecast for the full forecast length, you can convert the mosaic to horizontal for just that forecast period.
import json
from autots.models.ensemble import mosaic_to_horizontal, model_forecast
# assuming model is from AutoTS.fit() with a mosaic as best_model
model_params = mosaic_to_horizontal(model.best_model_params, forecast_period=0)
result = model_forecast(
model_name="Ensemble",
model_param_dict=model_params,
model_transform_dict={},
df_train=model.df_wide_numeric,
forecast_length=model.forecast_length,
)
result.forecast
Installation and Dependency Versioning¶
pip install autots
Some optional packages require installing Visual Studio C compilers if on Windows.
On Linux systems, apt-get/yum (rather than pip) installs of numpy/pandas may install faster/more stable compilations.
Linux may also require sudo apt install build-essential
for some packages.
You can check if your system is using mkl, OpenBLAS, or none with numpy.show_config()
. Generally recommended that you double-check this after installing new packages to make sure you haven’t broken the LINPACK connection.
Requirements:¶
Python >= 3.6
numpy
>= 1.20 (Sliding Window in Motif and WindowRegression)
pandas
>= 1.1.0 (prediction.long_form_results())
gluonts incompatible with 1.1, 1.2, 1.3
sklearn
>= 0.23.0 (PoissonReg)
>= 0.24.0 (OrdinalEncoder handle_unknown)
>= 1.0 for models effected by "mse" -> "squared_error" update
>? (IterativeImputer, HistGradientBoostingRegressor)
statsmodels
>= 0.13 ARDL and UECM
scipy.uniform_filter1d (for mosaic-window ensemble only)
scipy.stats (anomaly detection, Kalman)
scipy.signal (ScipyFilter)
scipy.spatial.cdist (Motifs)
Of these, numpy and pandas are critical. Limited functionality should exist without scikit-learn.
* Sklearn needed for categorical to numeric, some detrends/transformers, horizontal generalization, numerous models, nan_euclidean distance
Full functionality should be maintained without statsmodels, albeit with fewer available models.
Prophet, Greykite, and mxnet/GluonTS are packages which tend to be finicky about installation on some systems.
pip install autots['additional']
Optional Packages¶
requests
psutil
holidays
prophet
gluonts (requires mxnet)
mxnet (mxnet-mkl, mxnet-cu91, mxnet-cu101mkl, etc.)
tensorflow >= 2.0.0
lightgbm
xgboost
tensorflow-probability
fredapi
greykite
matplotlib
pytorch-forecasting
neuralprophet
scipy
arch
Tensorflow, LightGBM, and XGBoost bring powerful models, but are also among the slowest. If speed is a concern, not installing them will speed up ~Regression style model runs.
Safest bet for installation:¶
venv, Anaconda, or Miniforge with some more tips here.
# create a conda or venv environment
conda create -n timeseries python=3.9
conda activate timeseries
python -m pip install numpy scipy scikit-learn statsmodels lightgbm xgboost numexpr bottleneck yfinance pytrends fredapi --exists-action i
python -m pip install pystan prophet --exists-action i # conda-forge option below works more easily, --no-deps to pip install prophet if this fails
python -m pip install tensorflow
python -m pip install mxnet --no-deps # check the mxnet documentation for more install options, also try pip install mxnet --no-deps
python -m pip install gluonts arch
python -m pip install holidays-ext pmdarima dill greykite --exists-action i --no-deps
# install pytorch
python -m pip install --upgrade numpy pandas --exists-action i # mxnet likes to (pointlessly seeming) install old versions of numpy
python -m pip install autots --exists-action i
mamba install scikit-learn pandas statsmodels prophet numexpr bottleneck tqdm holidays lightgbm matplotlib requests xgboost -c conda-forge
pip install mxnet --no-deps
pip install yfinance pytrends fredapi gluonts arch
pip install intel-tensorflow scikit-learn-intelex
mamba install spyder
mamba install autots -c conda-forge
conda install pytorch torchvision torchaudio cpuonly -c pytorch
conda install pytorch-forecasting -c conda-forge
pip install neuralprophet
GPU support, Linux only. CUDA versions will need to match package requirements. Mixed CUDA versions may cause crashes if run in same session.
nvidia-smi
mamba activate base
mamba install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0 nccl # install in conda base
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/ # NOT PERMANENT unless add to ./bashrc make sure is for base env, mine /home/colin/mambaforge/lib
mamba create -n gpu python=3.8 scikit-learn pandas statsmodels prophet numexpr bottleneck tqdm holidays lightgbm matplotlib requests -c conda-forge
mamba activate gpu
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
pip install mxnet-cu112 --no-deps
pip install gluonts tensorflow neuralprophet pytorch-lightning pytorch-forecasting
mamba install spyder
mamba
and conda
commands are generally interchangeable. conda env remove -n env_name
Intel conda channel installation (sometime faster, also, more prone to bugs)¶
https://software.intel.com/content/www/us/en/develop/tools/oneapi/ai-analytics-toolkit.html
# create the environment
mamba create -n aikit37 python=3.7 intel-aikit-modin pandas statsmodels prophet numexpr bottleneck tqdm holidays lightgbm matplotlib requests tensorflow dpctl -c intel
conda config --env --add channels conda-forge
conda config --env --add channels intel
conda config --env --get channels
# install additional packages as desired
python -m pip install mxnet --no-deps
python -m pip install gluonts yfinance pytrends fredapi
mamba update -c intel intel-aikit-modin
python -m pip install autots
# OMP_NUM_THREADS, USE_DAAL4PY_SKLEARN=1
Speed Benchmark¶
from autots.evaluator.benchmark import Benchmark
bench = Benchmark()
bench.run(n_jobs="auto", times=3)
bench.results
Caveats and Advice¶
Mysterious crashes¶
Usually mysterious crashes or hangs (those without clear error messages) occur when the CPU or Memory is overloaded.
UnivariateRegression
is usually the most prone to these issues, removing it from the model_list may help (by default it is not included in most lists for this reason).
Try setting n_jobs=1
or an otherwise low number, which should reduce the load. Also test the ‘superfast’ naive models, which are generally low resource consumption.
GPU-accelerated models (Tensorflow in Regressions and GluonTS) are also more prone to crashes, and may be a source of problems when used.
If problems persist, post to the GitHub Discussion or Issues.
Rebooting between heavy uses of multiprocessing can also help reduce the risk of crashing in future model runs.
Series IDs really need to be unique (or column names need to be all unique in wide data)¶
Pretty much as it says, if this isn’t true, some odd things may happen that shouldn’t.
Also if using the Prophet model, you can’t have any series named ‘ds’
Short Training History¶
How much data is ‘too little’ depends on the seasonality and volatility of the data.
Minimal training data most greatly impacts the ability to do proper cross validation. Set num_validations=0
in such cases.
Since ensembles are based on the test dataset, it would also be wise to set ensemble=None
if num_validations=0
.
Adding regressors and other information¶
future_
regressor, to make it clear this is data that will be know with high certainy about the future.
Such data about the future is rare, one example might be number of stores that will be (planned to be) open each given day in the future when forecast sales.
Generally using regressors is very helpful for separating ‘organic’ and ‘inorganic’ patterns.
‘Inorganic’ patterns refers to human business decisions that effect the outcome and can be controlled.
A very common example of those is promotions and sales events.
The model can learn from the past promotion information to then anticpate the effects of the input planned promotion events.
Simulation forecasting, described below, is where multiple promotional plans can be tested side-by-side to evaluate effectiveness.
Only a handful of models support adding regressors, and not all handle multiple regressors.
The way to provide regressors is in the wide
style as a pd.Series/pd.Dataframe with a DatetimeIndex.
Don’t know the future? Don’t worry, the models can handle quite a lot of parallel time series, which is another way to add information. Additional regressors can be passed through as additional time series to forecast as part of df_long. Some models here can utilize the additional information they provide to help improve forecast quality. To prevent forecast accuracy for considering these additional series too heavily, input series weights that lower or remove their forecast accuracy from consideration.
an example of regressors:
from autots.datasets import load_monthly
from autots.evaluator.auto_ts import fake_regressor
from autots import AutoTS
long = False
df = load_monthly(long=long)
forecast_length = 14
model = AutoTS(
forecast_length=forecast_length,
frequency='infer',
validation_method="backwards",
max_generations=2,
)
future_regressor_train2d, future_regressor_forecast2d = fake_regressor(
df,
dimensions=4,
forecast_length=forecast_length,
date_col='datetime' if long else None,
value_col='value' if long else None,
id_col='series_id' if long else None,
drop_most_recent=model.drop_most_recent,
aggfunc=model.aggfunc,
verbose=model.verbose,
)
model = model.fit(
df,
future_regressor=future_regressor_train2d,
date_col='datetime' if long else None,
value_col='value' if long else None,
id_col='series_id' if long else None,
)
prediction = model.predict(future_regressor=future_regressor_forecast2d, verbose=0)
forecasts_df = prediction.forecast
print(model)
For models here in the lower level api, confusingly, regression_type=”User” must be specified as well as passing future_regressor. Why? This allows the model search to easily try both with and without the regressor, because sometimes the regressor may do more harm than good.
Simulation Forecasting¶
Simulation forecasting allows for experimenting with different potential future scenarios to examine the potential effects on the forecast.
This is done here by passing known values of a future_regressor
to model .fit
and then running .predict
with multiple variations on the future_regressor
future values.
By default in AutoTS, when a future_regressor
is supplied, models that can utilize it are tried both with and without the regressor.
To enforce the use of future_regressor for simulation forecasting, a few parameters must be supplied as below. They are: model_list, models_mode, initial_template
.
from autots.datasets import load_monthly
from autots.evaluator.auto_ts import fake_regressor
from autots import AutoTS
df = load_monthly(long=False)
forecast_length = 14
model = AutoTS(
forecast_length=forecast_length,
max_generations=2,
model_list="regressor",
models_mode="regressor",
initial_template="random",
)
# here these are random numbers but in the real world they could be values like weather or store holiday hours
future_regressor_train, future_regressor_forecast = fake_regressor(
df,
dimensions=2,
forecast_length=forecast_length,
drop_most_recent=model.drop_most_recent,
aggfunc=model.aggfunc,
verbose=model.verbose,
)
# another simulation of regressor
future_regressor_forecast_2 = future_regressor_forecast + 10
model = model.fit(
df,
future_regressor=future_regressor_train,
)
# first with one version
prediction = model.predict(future_regressor=future_regressor_forecast, verbose=0)
forecasts_df = prediction.forecast
# then with another
prediction_2 = model.predict(future_regressor=future_regressor_forecast_2, verbose=0)
forecasts_df_2 = prediction_2.forecast
print(model)
Note, this does not necessarily force the model to place any great value on the supplied features.
It may be necessary to rerun multiple times until a model with satisfactory variable response is found,
or to try with a subset of the regressor model list like ['FBProphet', 'GLM', 'ARDL', 'DatepartRegression']
.
Event Risk Forecasting and Anomaly Detection¶
Anomaly (or Outlier) Detection is historic and Event Risk Forecasting is forward looking.
Event Risk Forecasting Generate a risk score (0 to 1, but usually close to 0) for a future event exceeding user specified upper or lower bounds.
Upper and lower limits can be one of four types, and may each be different.
None (no risk score calculated for this direction)
Float in range [0, 1] historic quantile of series (which is historic min and max at edges) is chosen as limit.
- A dictionary of {“model_name”: x, “model_param_dict”: y, “model_transform_dict”: z, “prediction_interval”: 0.9} to generate a forecast as the limits
Primarily intended for simple forecasts like SeasonalNaive, but can be used with any AutoTS model
a custom input numpy array of shape (forecast_length, num_series)
import numpy as np
from autots import (
load_daily,
EventRiskForecast,
)
from sklearn.metrics import multilabel_confusion_matrix, classification_report
forecast_length = 6
df_full = load_daily(long=False)
df = df_full[0: (df_full.shape[0] - forecast_length)]
df_test = df[(df.shape[0] - forecast_length):]
upper_limit = 0.95 # --> 95% quantile of historic data
# if using manual array limits, historic limit must be defined separately (if used)
lower_limit = np.ones((forecast_length, df.shape[1]))
historic_lower_limit = np.ones(df.shape)
model = EventRiskForecast(
df,
forecast_length=forecast_length,
upper_limit=upper_limit,
lower_limit=lower_limit,
)
# .fit() is optional if model_name, model_param_dict, model_transform_dict are already defined (overwrites)
model.fit()
risk_df_upper, risk_df_lower = model.predict()
historic_upper_risk_df, historic_lower_risk_df = model.predict_historic(lower_limit=historic_lower_limit)
model.plot(0)
threshold = 0.1
eval_lower = EventRiskForecast.generate_historic_risk_array(df_test, model.lower_limit_2d, direction="lower")
eval_upper = EventRiskForecast.generate_historic_risk_array(df_test, model.upper_limit_2d, direction="upper")
pred_lower = np.where(model.lower_risk_array > threshold, 1, 0)
pred_upper = np.where(model.upper_risk_array > threshold, 1, 0)
model.plot_eval(df_test, 0)
multilabel_confusion_matrix(eval_upper, pred_upper).sum(axis=0)
print(classification_report(eval_upper, pred_upper, zero_division=1)) # target_names=df.columns
A limit specified by a forecast can be used to use one type of model to judge the risk of another production model’s bounds (here ARIMA) being exceeded. This is also useful for visualizing the effectivness of a particular model’s probabilistic forecasts.
Using forecasts as a limit is also a common method of detecting anomalies in historic data - looking for data points that exceeded what a forecast would have expected.
Forecast_length effects how far ahead each forecast step is. Larger is faster, smaller means tighter accuracy (only the most extreme outliers are flagged).
predict_historic
is used for looking back on the training dataset. Use eval_periods
to look at only a portion.
lower_limit = {
"model_name": "ARIMA",
"model_param_dict": {'p': 1, "d": 0, "q": 1},
"model_transform_dict": {},
"prediction_interval": 0.9,
}
Anomaly Detection
Multiple methods are available, including use of forecast_params
which can be used to analyze the historic deviations of an AutoTS forecasting model.
Holiday detection may also pick up events or ‘anti-holidays’ ie days of low demand. It won’t pick up holidays that don’t usually have a significant impact.
from autots.evaluator.anomaly_detector import AnomalyDetector
from autots.datasets import load_live_daily
# internet connection required to load this df
wiki_pages = [
"Standard_deviation",
"Christmas",
"Thanksgiving",
"all",
]
df = load_live_daily(
long=False,
fred_series=None,
tickers=None,
trends_list=None,
earthquake_min_magnitude=None,
weather_stations=None,
london_air_stations=None,
gov_domain_list=None,
weather_event_types=None,
wikipedia_pages=wiki_pages,
sleep_seconds=5,
)
params = AnomalyDetector.get_new_params()
mod = AnomalyDetector(output='multivariate', **params)
mod.detect(df)
mod.plot()
mod.scores # meaning of scores varies by method
# holiday detection, random parameters
holiday_params = HolidayDetector.get_new_params()
mod = HolidayDetector(**holiday_params)
mod.detect(df)
# several outputs are possible, you'll need to subset results from multivariate inputs
full_dates = pd.date_range("2014-01-01", "2024-01-01", freq='D')
prophet_holidays = mod.dates_to_holidays(full_dates, style="prophet")
mod.plot()
A Hack for Passing in Parameters (that aren’t otherwise available)¶
There are a lot of parameters available here, but not always all of the options available for a particular parameter are actually used in generated templates. Usually, very slow options are left out. If you are familiar with a model, you can try manualy adding those parameter values in for a run in this way… To clarify, you can’t usually add in entirely new parameters in this way, but you can often pass in new choices for existing parameter values.
Run AutoTS with your desired model and export a template.
Open the template in a text editor or Excel and manually change the param values to what you want.
Run AutoTS again, this time importing the template before running .fit().
There is no guarantee it will choose the model with the given params- choices are made based on validation accuracy, but it will at least run it, and if it does well, it will be incorporated into new models in that run (that’s how the genetic algorithms work).
Categorical Data¶
Categorical data is handled, but it is handled crudely. For example, optimization metrics do not currently include any categorical accuracy metrics. For categorical data that has a meaningful order (ie ‘low’, ‘medium’, ‘high’) it is best for the user to encode that data before passing it in, thus properly capturing the relative sequence (ie ‘low’=1, ‘medium’=2, ‘high’=3).
Custom and Unusual Frequencies¶
Data must be coercible to a regular frequency. It is recommended the frequency be specified as a datetime offset as per pandas documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects Some models will support a more limited range of frequencies.
Using the Transformers independently¶
The transformers expect data only in the wide
shape with ascending date.
The simplest way to access them is through the GeneralTransformer.
This takes dictionaries containing strings of the desired transformers and parameters.
Inverse_transforms get confusing. It can be necessary to inverse_transform the data to get predictions back to a usable space.
Some inverse_transformer only work on ‘original’ or ‘forecast’ data immediately following the training period.
The DifferencedTransformer is one example.
It can take the last N value of the training data to bring forecast data back to original space, but will not work for just ‘any’ future period unconnected to training data.
Some transformers (mostly the smoothing filters like bkfilter
) cannot be inversed at all, but transformed values are close to original values.
from autots.tools.transform import transformer_dict, DifferencedTransformer
from autots import load_monthly
print(f"Available transformers are: {transformer_dict.keys()}")
df = load_monthly(long=long)
# some transformers tolerate NaN, and some don't...
df = df.fillna(0)
trans = DifferencedTransformer()
df_trans = trans.fit_transform(df)
print(df_trans.tail())
# trans_method is not necessary for most transformers
df_inv_return = trans.inverse_transform(df_trans, trans_method="original") # forecast for future data
Note on ~Regression Models¶
The Regression models are WindowRegression, RollingRegression, UnivariateRegression, MultivariateRegression, and DatepartRegression. They are all different ways of reshaping the time series into X and Y for traditional ML and Deep Learning approaches. All draw from the same potential pool of models, mostly sklearn and tensorflow models.
DatepartRegression is where X is simply the date features, and Y are the time series values for that date.
WindowRegression takes an
n
preceeding data points as X to predict the future value or values of the series.RollingRegression takes all time series and summarized rolling values of those series in one massive dataframe as X. Works well for a small number of series but scales poorly.
MultivariateRegression uses the same rolling features as above, but considers them one at a time, features for series
i
are used to predict next step for seriesi
, with a model trained on all data from all series. This model is now often called by the community a “global forecasting ML model”.UnivariateRegression is the same as MultivariateRegression but trains an independent model on each series, thus not capable of learning from the patterns of other series. This performs well in horizontal ensembles as it can be parsed down to one series with the same performance on that series.
Currently MultivariateRegression
has the (slower) option to utilize a stock GradientBoostingRegressor with quantile loss for probabilistic estimates, while others utilize point to probabilistic estimates.
Models¶
Model |
Dependencies |
Optional Dependencies |
Probabilistic |
Multiprocessing |
GPU |
Multivariate |
Experimental |
Use Regressor |
|---|---|---|---|---|---|---|---|---|
ConstantNaive |
||||||||
LastValueNaive |
||||||||
AverageValueNaive |
True |
|||||||
SeasonalNaive |
||||||||
GLS |
statsmodels |
True |
||||||
GLM |
statsmodels |
joblib |
True |
|||||
ETS - Exponential Smoothing |
statsmodels |
joblib |
||||||
UnobservedComponents |
statsmodels |
True |
joblib |
True |
||||
ARIMA |
statsmodels |
True |
joblib |
True |
||||
VARMAX |
statsmodels |
True |
True |
|||||
DynamicFactor |
statsmodels |
True |
True |
True |
||||
DynamicFactorMQ |
statsmodels |
True |
True |
|||||
VECM |
statsmodels |
True |
True |
|||||
VAR |
statsmodels |
True |
True |
True |
||||
Theta |
statsmodels |
True |
joblib |
|||||
ARDL |
statsmodels |
True |
joblib |
True |
||||
FBProphet |
prophet |
True |
joblib |
True |
||||
GluonTS |
gluonts, mxnet |
True |
yes |
True |
True |
|||
RollingRegression |
sklearn |
lightgbm, tensorflow |
sklearn |
some |
True |
True |
||
WindowRegression |
sklearn |
lightgbm, tensorflow |
sklearn |
some |
True |
True |
||
DatepartRegression |
sklearn |
lightgbm, tensorflow |
sklearn |
some |
True |
|||
MultivariateRegression |
sklearn |
lightgbm, tensorflow |
True |
sklearn |
some |
True |
True |
|
UnivariateRegression |
sklearn |
lightgbm, tensorflow |
sklearn |
some |
True |
|||
Univariate/MultivariateMotif |
scipy.distance.cdist |
True |
joblib |
|||||
SectionalMotif |
scipy.distance.cdist |
sklearn |
True |
True |
True |
|||
MetricMotif, SeasonalityMotif |
True |
|||||||
NVAR |
True |
blas/lapack |
True |
|||||
RRVAR, MAR, TMF |
True |
|||||||
LATC |
True |
|||||||
NeuralProphet |
neuralprophet |
nyi |
pytorch |
yes |
True |
|||
PytorchForecasting |
pytorch-forecasting |
True |
pytorch |
yes |
True |
|||
ARCH |
arch |
True |
joblib |
True |
||||
Cassandra |
scipy |
True |
True |
True |
||||
KalmanStateSpace |
True |
|||||||
MotifSimulation |
sklearn.metrics.pairwise |
True |
joblib |
True |
True |
|||
Greykite |
(deprecated) |
True |
joblib |
True |
nyi |
|||
TensorflowSTS |
(deprecated) |
True |
yes |
True |
True |
|||
TFPRegression |
(deprecated) |
True |
yes |
True |
True |
True |
||
ComponentAnalysis |
(deprecated) |
True |
True |
_ |
nyi = not yet implemented
deprecated models are not actively maintained but updates may be requested in issues


Get the file _basename.
Get the file content type.
Get the file directory _name.
Get the file extension.
Get the file object.
Get the ID.
Check if job is locally available.
Set if the job is locally available.
Get the file _name.
Set a new file _name.
Get the file _name proper.
Get the file _path.
Set a new file _path.
Create a new life event.
Sends an email.
Email options
// my_tunnel.js
tunnel.run(function (job, nest) {
job.email({
subject: "Test email from pug template",
to<EMAIL_ADDRESS>template: __dirname + "./template_files/my_email.pug"
});
});
// template_files/my_email.pug
h1="Example email!"
p="Got job ID " + job.getId()
tunnel.run(function (job, nest) {
job.email({
subject: "Test email with hard-coded plain-text",
to<EMAIL_ADDRESS>text: "My email body!"
});
});
tunnel.run(function (job, nest) {
job.email({
subject: "Test email with hard-coded html",
to<EMAIL_ADDRESS>html: "
My email body!
"
});
});
Function to call to fail a job while in a tunnel.
Get the job object as JSON with circular references removed.
Get the entire job property object.
Get the type of a property.
job.setPropertyValue("My Job Number", 123456);
console.log(job.getPropertyType("My Job Number"));
// "number"
Get the value of a property if it has been previously set.
Check if job is a file.
Check if job is a folder.
Add a message to the log with this job as the actor.
0 = debug, 1 = info, 2, = warning, 3 = error
Log message
Moves a file to a nest. This is an asynchronous method which provides a callback on completion.
The nest object the job will be sent to.
The callback provides the updated instance of the job. Depending on the nest it was sent to, it may have been cast to a new job type. This is helpful in case you need the remote path to the job once it has been uploaded to S3, for example.
tunnel.run((job, nest) => {
console.log("Found job " + job.name);
job.move(my_s3_bucket, (s3_job) => {
// Uploaded
console.log("Uploaded to " + s3_job.path); // https://mybucket.s3.amazonaws.com/myfile.pdf
});
});
Packs the job instance and file together in a zip. Returns a PackJob in the parameter of the callback.
job.pack(function(packJob){
packJob.move(packed_folder_nest);
});
Deletes the local file.
Rename the job file to a new _name.
Attach job specific data to the job instance.
job.setPropertyValue("My Job Number", 123456);
console.log(job.getPropertyValue("My Job Number"));
// 123456
Class _name for logging.
Transfer a job to another tunnel directly.
Unpacks a packed job. Returns a the original unpacked job in the parameter of the callback.
packedJob.unpack(function(unpackedJob){
console.log("Unpacked", unpackedJob.name);
unpackedJob.move(unpacked_folder);
packedJob.remove();
});
Generated using TypeDoc
FileJob constructor.


Principal Software Engineer @ Red Hat Storage
GitHub, IRC: lpabon
We implement programming (or design) by contract as a Hoare Triple:
where {P} is the precondition before executing command C, and {Q} is the postcondition.
http://artofunittesting.com/definition-of-a-unit-test/
You could build the entire car and then test
test_chair()
{
assert(supports a person);
assert(move foard);
assert(move back);
assert(move left);
assert(move right);
assert(lean back N degrees without falling back);
}
// Replace human with robot
test_chair()
{
assert(robot arm push down on chair with X Newtons);
assert(robot arm move chair forward 0.5m);
assert(robot arm move chair back 0.5m);
assert(robot arm move chair left 0.5m);
assert(robot arm move chair right 0.5m);
assert(robot arm lean chair back 25 degrees);
}
test_cushion() { ... }
test_arm() { ... }
test_leg() { ... }
test_leaning_assembly { ... }
test_lift { ... }
test_wheel()
{
Mock chair leg holding wheel();
exepect wheel pin to appear in Mock chair leg();
insert wheel into mock leg();
assert(simulate weight);
assert(clockwise rotation);
assert(counter-clockwise rotation);
assert(connector to mock chair leg);
}
C Unit Test Framework
GitHub: https://github.com/lpabon/cmockery2
#include <stdio.h>
#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <inttypes.h>
#include <cmockery/cmockery.h>
void null_test_success(void **state) {
printf("Hello test world!\n");
}
int main(int argc, char* argv[]) {
const UnitTest tests[] = {
unit_test(null_test_success),
};
return run_tests(tests, "hello");
}
$ gcc -o hello_test \
hello_test.c \
`pkg-config --libs --cflags cmockery2`
$ ./hello_test
[==========] Running 1 test(s).
[ RUN ] null_test_success
Hello test world!
[ OK ] null_test_success
[==========] 1 test(s) run.
[ PASSED ] 1 test(s).
[ FAILED ] 0 test(s).
[ REPORT ] Created hello_xunit.xml report
int add(int a, int b) {
return a+b;
}
int add(int a, int b);
void test_add(void **state) {
assert_true(add(0,0) == 0);
assert_true(add(1,0) == 1);
assert_true(add(0,1) == 1);
assert_false(add(1,1) == 0);
}
int main(int argc, char* argv[]) {
const UnitTest tests[] = {
unit_test(test_add),
};
return run_tests(tests, "add");
}
$ gcc -o add_test \
add.c \
add_test.c \
`pkg-config --libs --cflags cmockery2`
$ ./add_test
[==========] Running 1 test(s).
[ RUN ] test_add
[ OK ] test_add
[==========] 1 test(s) run.
[ PASSED ] 1 test(s).
[ FAILED ] 0 test(s).
[ REPORT ] Created add_xunit.xml report
#include <cmockery/pbc.h>
Post- and preconditions for applications
These macros are just macros on top of assert()
These semantics are only be enabled during debug builds and during unit test runs
_dict_set()
static int32_t
_dict_set (dict_t *this, char *key, data_t *value, gf_boolean_t replace) {
...
}
static int32_t
_dict_set (dict_t *this, char *key, data_t *value, gf_boolean_t replace) {
REQUIRE(key != NULL);
REQUIRE(this != NULL);
REQUIRE(value != NULL);
REQUIRE(1000000 > strlen(key));
REQUIRE(NULL != THIS);
REQUIRE(NULL != THIS->ctx);
REQUIRE(NULL != THIS->ctx->dict_pair_pool);
INVARIANT(_dict_invariant(this));
...
ENSURE(val == _dict_lookup(this, key));
}
_dict_invariant()
static void
_dict_invariant (dict_t *this) {
REQUIRE(this != NULL);
REQUIRE(this->refcount > 0);
REQUIRE(this->lock != 0);
REQUIRE(this->members != NULL);
}
Cmockery provides support for overriding system calls like assert()
and memory management functions
Provided by <cmockery/cmockery_override.h>
which must the last #include
in the application
Here is an example application function calling assert
#include <cmockery/cmockery_override.h>
void increment_value(int * const value) {
assert(NULL != value);
(*value) ++;
}
Here we can test the assert is triggered
void increment_value_assert(void **state) {
expect_assert_failure(increment_value(NULL));
}
int main(int argc, char *argv[]) {
const UnitTest tests[] = {
unit_test(increment_value_assert),
};
return run_tests(tests, "assert_module");
}
malloc(), calloc(), free(), realloc()
functionsa.c
static int
a(int x, int y) {
return double(x) + y;
}
Place new mock double()
in a_mock.c
and build the test:
gcc -o a_test a_test.c a.c a_mock.c `pkg-config --libs --cflags cmockery2`
Test
static void
test_a( void **state ) {
// Tells cmockery that the function 'double' will have a
// parameter 'value' with the value of '2'
expect_value(double, value, 2);
// Return '4' from the function 'double'
will_return(double, 4);
// Make sure that a() returns 8
assert_int_equal(a(2, 4), 8);
}
Mock double()
int
double( int value ) {
// Check the argument is as expected
check_expected(value);
// Return what we had expressed in the test
return((int)mock());
}
Prepare the environment
$ sudo yum -y install cmockery2 cmockery2-devel
Download the lab
$ git clone https://github.com/lpabon/hashtest.git
Add the following includes
#include <cmockery/pbc.h>
#include <cmockery/cmockery_override.h>
Add REQUIRE and ENSURE to all API functions. Add REQUIRE() after variable initialization, and ENSURE() before last return.
Open hash_test.c and add unit tests to hash_create(). You first may want to start by testing any REQUIRE()s. You can then test that NULL is returned when size is less than 1
Rename strdup to test_strdup as follows:
#include <cmockery/pbc.h>
#include <cmockery/cmockery_override.h>
#ifdef UNIT_TESTING
#define strdup test_strdup
#endif
Create a new file called hash_mock.c and add the mock function test_strdup to that file. You can now use this function to test. Check the values passed to the mock function. By using mocks, you can now send back NULL from the mock function.


| PhotoLine Tip: Remove red eyes |
| 1. You have
a photo where the eyes of a person got red because of the flash of your
camera.
2. To change the color of the eyes, select the tool Replace Colors from the Tool Bar. 3. Setup the
Tool Options to about 60% Tolerance and select Click Color
-> Foreground Color. This function replaces the color under the mouse
when you click into the image by the foreground color. So your next step
should be to setup the foreground color to your new eye color (brown, blue
or green).
4. Use a little
unsharp pen and draw around the red part of the eyes and you will get a
nice photo.
|
| ««Previous Tip <Contents> Next Tip»» |
| Please email questions and suggestions on these tips to Gerhard Huber. |


Preparation time
overnight
Cooking time
10 to 30 mins
Serves
Serves 4
Prepare the mixture in advance, then just a few minutes frying is all it takes for a delicious light meal of hot crisp croquettes served with sharp, crunchy tuna salad.
For the croquettes, melt the butter in a medium saucepan over a medium heat, add the ham and leave to cook gently for a minute or two to soften.
Stir in 110g/4oz of the flour and cook for one minute, then very gradually stir in the milk, a little at a time, beating really well between each addition so that the mixture becomes silky smooth.
Increase the heat slightly, bring to the boil, and leave to cook gently, whisking constantly, for five minutes, to cook out the flour.
Stir in the chicken and one teaspoon of salt, or to taste, then stir in the hard-boiled egg.
Scrape the mixture into a bowl, press a sheet of cling film onto the surface, and chill in the fridge, for at least six hours but ideally overnight, until really firm.
When you are ready to make the croquettes, lightly flour the palms of your hands and for each croquette shape 1½ tablespoons of the chilled mixture into a ball, and then roll into a cork-shaped barrel. You should make about 24 croquettes. Refrigerate again for 30 minutes.
Meanwhile for the salad, remove and discard the outer leaves of the lettuce if necessary. Break the remainder into approximately 7cm/3in pieces, wash and dry well, in a salad spinner if you have one.
Spread the leaves over four medium-sized plates and arrange the tomatoes, onion and asparagus on top.
Scatter the flaked tuna over the centre of each plate.
Whisk the red wine vinegar and olive oil together with the salt to taste and drizzle over the salads.
Garnish with the hard-boiled eggs and green olives.
Heat some oil for deep-frying to 190C/375F. (Caution: hot oil can be dangerous. Never leave unattended.)
Put the remaining 75g/3oz flour, egg and breadcrumbs into three separate shallow trays or bowls.
Take the croquettes from the fridge and dip them four or five at a time into the flour, then the beaten egg and then the breadcrumbs. Lower them into the hot oil and cook for not quite two minutes until crisp and lightly golden-brown. You can tell when they are done because as they cook they begin to make a different noise in the fryer.
Lift out with a slotted spoon and set aside on plenty of kitchen paper to drain briefly. Serve while they are still hot.
To serve, pile the hot croquettes straight onto a plate with a spoonful of salad alongside.


--- layout: null ---
Graphic: Metro Ideas Project Sources: UnifiEd courtesy of Hamilton County Department of Education; Tennessee Department of Education.


Meet the Team
PROJECT LEAD
Brandon Mulas
UX DESIGNER
Kelly Cubitt
WEB UI DEVELOPER
Richard Wang
Shaun Riutta
WEB REACT I
Nicholas Sandomeno
Robert Mitchell
WEB REACT II
Alejandro E. Rodriguez La Rosa
BACKEND DEVELOPER
Pedro Prieto
DATA ENGINEER
Tuan Ky
Keiandra King
MACHINE LEARNING ENGINEER
Brandon Mulas
Andrea Christelle
Pierre Nelson


see
Alias to `run-mailcap`'s view.
An alias to a `run-mailcap`'s action print.
More information: home page
- See action can be used to view any file (usually image) on default mailcap explorer:
see filename
- Using with `run-mailcap`:
run-mailcap --action=view filename


Store
You can buy items.
Buy 1 item
Buy 10 items
Buy 100 items
Admin
Store backend system.
Connected:
Consuming from queue:
Worker name:
Pending jobs
(from queue)
Completed jobs


Thiruvananthapuram is a major tourist centre, known for the Padmanabhaswamy Temple, the beaches of Kovalam and Varkala, the backwaters of Poovar and Anchuthengu and its Western Ghats tracts of Ponmudi and the Agastyamala
Famous for its Lion and Deer Safari, Neyyar Wildlife Sanctuary is a treat for nature enthusiasts. The park houses a crocodile farm and elephant rehabilitation centre. Ask for a guided tour of the sanctuary which is quite informative.
One of the 108 Divya Desams, Sree Padmanabhaswamy Temple is located in Thiruvananthapuram, the capital city of Kerala. Embellished in gold-plated covering, the temple is opened only to Hindu followers. Dedicated to Lord Padmanabha; one of the avatars of Lord Vishnu, the Padmanabhaswamy Temple is one of the principal centres of Vaishnava worship in the dharma of Vaishnavism.
Epicentre of tourist flow, Kovalam Beach needs no introduction. This water delight seizes the top position in the list of tourist places in Trivandrum. Often crowded with both local and foreign tourists, it is located 13km away from the city. Kovalam Beach is the treasure chest of multitude of adventure activities. Along with the sessions of para-sailing, diving and wind surfing, the beach also allows you to go on a leisure walk and offers a playful swimming time. Tourists have a choice to opt for lounge chairs to just sit and gaze at the most extraordinary sunrises and sunsets. The gleaming blue water and silver sands offers a quality time to enjoy nature at its best.
Napier Museum is certainly one of the most visited tourist places in Trivandrum. Situated in the Museum compound, ground floor of this place houses India’s oldest zoological garden. Another important feature is Sree Chithra Art Gallery, established by Maharaja Chithira Thirunalin in 1935.The gallery contains epic works from much famed Raja Ravi Varma and Nicholas Roerich.