public static class R.string
public static int abc_action_bar_home_description
public static int abc_action_bar_up_description
public static int abc_action_menu_overflow_description
public static int abc_action_mode_done
public static int abc_activity_chooser_view_see_all
public static int abc_activitychooserview_choose_application
public static int abc_capital_off
public static int abc_capital_on
public static int abc_search_hint
public static int abc_searchview_description_clear
public static int abc_searchview_description_query
public static int abc_searchview_description_search
public static int abc_searchview_description_submit
public static int abc_searchview_description_voice
public static int abc_shareactionprovider_share_with
public static int abc_shareactionprovider_share_with_application
public static int abc_toolbar_collapse_description
public static int app_name
public static int status_bar_notification_info_overflow
public static int vertical_form_stepper_form_confirm_button
public static int vertical_form_stepper_form_continue
public static int vertical_form_stepper_form_current_step_uncompleted_alert
public static int vertical_form_stepper_form_discard
public static int vertical_form_stepper_form_discard_cancel
public static int vertical_form_stepper_form_discard_question
public static int vertical_form_stepper_form_down_next
public static int vertical_form_stepper_form_down_previous
public static int vertical_form_stepper_form_info_will_be_lost
public static int vertical_form_stepper_form_last_step
public static int vertical_form_stepper_form_sending_data_message


Project name is required.
BU/BP is required.
Project Manager is required.
Project Supervisor is required.
Project Member is required.
Project owner is required.
Start date is required.
Finish Date is required.
Project Detail is required.


APTITUDE-RUN-STATE(1) Command-line reference APTITUDE-RUN-STATE(1) NAME aptitude-run-state-bundle - unpack an aptitude state bundle and invoke aptitude on it SYNOPSIS aptitude-run-state-bundle [<options>...] <input-file> [<program> [<arguments>...]] DESCRIPTION aptitude-run-state-bundle unpacks the given aptitude state bundle created by aptitude-create-state-bundle(1) to a temporary directory, invokes <program> on it with the supplied <arguments>, and removes the temporary directory afterwards. If <program> is not supplied, it defaults to aptitude(8). OPTIONS The following options may occur on the command-line before the input file. Options following the input file are presumed to be arguments to aptitude. --append-args Place the options that give the location of the state bundle at the end of the command line when invoking <program>, rather than at the beginning (the default is to place options at the beginning). --help Display a brief usage summary. --prepend-args Place the options that give the location of the state bundle at the beginning of the command line when invoking <program>, overriding any previous --append-args (the default is to place options at the beginning). --no-clean Do not remove the unpacked state directory after running aptitude. You might want to use this if, for instance, you are debugging a problem that appears when aptitude's state file is modified. When aptitude finishes running, the name of the state directory will be printed so that you can access it in the future. This option is enabled automatically by --statedir. --really-clean Delete the state directory after running aptitude, even if --no-clean or --statedir was supplied. --statedir Instead of treating the input file as a state bundle, treat it as an unpacked state bundle. For instance, you can use this to access the state directory that was created by a prior run with --no-clean. --unpack Unpack the input file to a temporary directory, but don't actually run aptitude. SEE ALSO aptitude-create-state-bundle(1), aptitude(8), apt(8) AUTHOR Daniel Burrows<EMAIL_ADDRESS>Author. COPYRIGHT Copyright 2007 Daniel Burrows. This manual page is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This manual page is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. aptitude-run-state-bundle 0.6. 11/05/2012 APTITUDE-RUN-STATE(1)


Image to ZPL
Drop image here
Image Rotation
Normal
Right (CW)
Left (CCW)
Inverted
Black Threshold
1..99
No Trim
ZPL Format
Z64
ACS
Original Image:
Rendered Image:


Download ANTLR
Looking for ANTLR v3?
The latest version of ANTLR is 4.1, released June 30, 2013. As of 4.1, we have a Java and C# target, but C++ is next on deck now that we have finalized the API. [Unfortunately, we had to push C++ target off to unknown future date]
ANTLR v4 is written in ANTLR v3.5 and StringTemplate 4.0.7. In antlr-4.1-complete.jar, you'll find everything you need to run the ANTLR tool and make its generated parsers work. Please see CHANGES.txt and README.txt.
ANTLRWorks2 and plugins
ANTLRWorks 2. This IDE is a sophisticated editor for ANTLR v3/v4 grammars as well as StringTemplate templates. It can run the ANTLR tool to generate recognizers and can run the TestRig (grun on command line) to test grammars. To integrate ANTLR-generated recognizers into your application, you will still need at least the runtime library shown below.
Also we have a basic ANTLR v4 plugin for Intellij 12. It does syntax highlighting and syntax error checking. You get a navigation window too. Sample usage showing Java grammar to the right. See the installation instructions.
Edgar Espina has created an eclipse plugin for ANTLR v4. Features: Advanced Syntax Highlighting, Automatic Code Generation (on save), Manual Code Generation (through External Tools menu), Code Formatter (Ctrl+Shift+F), Syntax Diagrams, Advanced Rule Navigation between files (F3), Quick fixes.
Java Target
- Complete ANTLR 4.1 Java binaries jar. Complete ANTLR 4.1 tool, Java runtime and ST 4.0.7, which lets you run the tool and the generated code.
- ANTLR 4.1 distribution. Everything you need to build the tool and Java runtime from source if you use ant or mvn.
- ANTLR 4.1 Java runtime binaries jar. Only what's needed for building and executing parsers/lexers generated in Java.
C# Target
Currently still 4.0 not 4.1. Should be out July 7.Next steps
Your next steps are probably:


| Icon |
|---|
| Name |
| Chevron |
New York City’s streetscape shapes the way residents experience the city. Manhattan Community Board 1 (CB1) encompasses some of the city’s most complex streets and sidewalks. This map displays the existing street furniture (four of eighteen total documented features) in the CB1 in order to analyze the existing conditions of sidewalks. Use the right hand sidebar to view the location and density of each feature.
The data provided on this site is for informational and planning purposes only.
Absolutely no accuracy or completeness guarantee is implied or intended. All information on this map is subject to such variations and corrections as might result from a complete title search and/or accurate field survey.
Community Board data courtesy of New York City Department of City Planning
MTA Subway data courtesy of the CUNY Mapping Service at the Center for Urban Research
Sidewalk feature data courtesy of NYU Planning Fellow, Cammie Flippen and Pace University undergraduate students


Chat analysis
20 Feb 2017Chat analysis (google hangout and whatsapp)
little experiment I did last year, using R NLP librarires and hangout and whatsapp data.
GitHub Link
Getting/Cleaning hangout data
Hangout data can be downloaded using google takeaway. The data that I used and tested is from 2014-2015 timeframe. JSON file provided by google is very huge and requires knowledge of data format to extract the conversation so I used https://bitbucket.org/dotcs/hangouts-log-reader/ tool to extract the convertion
after processing the output of ‘hangout_reader.py’ I had the file in this format:
2014-02-18 16:44:22: <1author-1> message
2014-02-18 16:44:33: <2author-2> message
Getting/Cleaning whatsapp data
Getting whatsapp data was straight forward. Just emailed the conversation to my inbox. only problem was that the format slightyly changed in between 2014 and 2015. so I had to use two different parsing code.
2014 format
09:53, 21 Jan - 1author-1: message
09:57, 24 Jan - 2author-2: message
2015 format
20/05/2015, 1:18 PM - 1author-1: message
20/05/2015, 1:18 PM - 2author-2: message
Final data format
Final data was exported to individual CSV files (cleanChat.c) with following columns
- year
- month
- day
- hour
- minute
- author
- message
- source
Data input
datain.R : to input data
WordCloud
wordcloud.R : Generates a basic wordcloud libraries
- slam
- tm
- wordcloud
GGplot2
ggplot2.R : Generates a basic graph library
- ggplot2
Other details
R version : 3.2.5
further work
will add more experiments soon.


{% extends 'main/base.html' %} {% block content %}
Please write your name, email address and a message below if you have any questions.


Try out our super special all-time favorite milk tea with your choice of YUMMY toppings!
Order now!
Thanks! We've received your order:
Total = $
Please enter a number of drinks, select a size and sugar level to submit your order!
*Toppings are optional.


List of available pages
Users section
Builds section
Skills section
Items section
Build - Gems conection section
Build - Cube items conection section
Build - Items conection section
Build - Skills conection section
Vote - Build - User conection section


What data can I see on Pivot?
The median GPA and grade distribution of declared students and the 10 most commonly taken courses by students
How will it support my work?
Pivot supports a wide range of UW roles, such as advisers who are helping students applying to a capacity-constrained major and departmental admins who make curriculum decisions.
Before you get started
- Pivot uses box plots to visualize the GPA data. New to box plots? Visit About Pivot.
- Recent changes in the data may not be reflected because Pivot provides aggregated, historical data.
- Remember GPA is just one of many factors that goes into admission decisions.


API
Core API
The API comprises two classes,
Connection
and
Request
,
along with a collection of data types
.
Connection Pooling
Tedious does not provide pooling of connections. For a connection pooling solution, see tedious-connection-pool.


Grotesque
This group features most of the early (19th century to early 20th)
sans-serif designs. Influenced by
Didone
serif fonts of the period and signpainting traditions,
these were often quite
solid, bold designs suitable for headlines and advertisements.
The early sans-serif typefaces
often did not feature a
lowercase
or italics, since they were not needed for such uses.
They were sometimes released by
width,
with a range of widths from
extended
to normal
to condensed,
with each style different,
meaning to modern eyes they can look quite
irregular
and eccentric.
Grotesque fonts have limited variation
of stroke width,
often none perceptible in
capitals.
The terminals of curves are usually
horizontal,
and many have a spurred
“G” and an “R”
with a curled leg.
Capitals tend to be of relatively
uniform width.
Cap height
and ascender height
are generally the same
to create a more regular effect in texts such as
titles
with many capital letters,
and descenders are often short for tighter linespacing.
Most avoid having a true italic
in favour of a more restrained oblique or sloped design,
although at least some did have true
italics.
Neo-Grotesque
As the name implies,
these modern designs consist of a direct
evolution of grotesque types.
They are relatively straightforward in appearance
with limited width
variation.
Unlike earlier grotesque designs, many were issued
in extremely
large and versatile families
from the time of release, making them easier to use for
body text.
Similar to grotesque typefaces,
neogrotesques
often feature capitals of uniform width
and a quite “folded-up”
design,
in which strokes (for example on the “c”) are
curved all the way
round to end on a perfect horizontal or vertical.


Fix self retain cycles #66
Merged
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
de1ec06
Merge pull request #1 from PerfectlySoft/master
califrench 3c31172
Refactored the JSONEncode and JSONDecode class to be named JSONEncode…
califrench 1e2fb56
Merge pull request #2 from PerfectlySoft/master
califrench 8d08732
Documented some methods of the JSONEncoder class
califrench d14ba1b
Merge remote-tracking branch 'origin/master'
califrench efbfbbf
Updated the code base to use weak or unowned capture groups for refer…
califrench 29345ea
Merge remote-tracking branch 'PerfectlySoft/master'
califrench
Jump to file or symbol
Failed to load files and symbols.
Oops, something went wrong.


In the example we gave in the previous post, it may seem that Pex chose random numbers as inputs for the Triang() method but it is not. But also its not all possible values for the inputs.
Actually, Pex generates test inputs by analyzing your program code, so it is called whitebox test generation (as opposed to blackbox test generation). For every statement in the code, Pex will eventually try to create a test input that will reach that statement. Pex will do a case analysis for every conditional branch in the code—for example, if statements, assertions, and all operations that can throw exceptions.
In other words, the number of test inputs that Pex generates depends on the number and possible combinations of conditional branches in the code (if interested to know more about that, search for symbolic execution). Pex operates in a feedback loop: it executes the code multiple times and learns about the program behavior by monitoring the control and data flow.
After each run, Pex does the following:
The test is executed again with the new inputs, and the process repeats. On each run, Pex might discover new code and dig deeper into the implementation. In this way, Pex explores the behavior of the code.
Because our code doesn’t have any conditions that test zero length sides, Pex generated zero an input and also shows that our program is defective (because it considers a triangle with (0,0,0) as equilateral and (1,0,1) as isosceles). If we added the following lines of code after declaring triOut and before doing anything.
// A quick confirmation that it's a valid triangle
if (Side1 <= 0 || Side2 <= 0 || Side3 <= 0)
{
triOut = 4;
return (triOut);
}
You will get a different set of test inputs from Pex that reveal more code path combinations.
We mentioned before that pex generates test input by performing a symoblic analysis of the code under test. You can use the method GetPathConditionString of the PexSymbolicValue class to obtain a textual representation of the current path condition, a predicate (condition) that characterizes an execution path. The ToString method of PexSymbolicValue class gives you a textual representation of how a value was derived from the test input provided. To do so, add reference to Pex Framework dll (located at “C:\Program Files\Microsoft Moles\PublicAssemblies\Microsoft.Pex.Framework.dll”). Add using Microsoft.Pex.Framework; to your code then add the following code anywhere in your code to get details about the code path at that point.
PexObserve.ValueForViewing("Condition", PexSymbolicValue.GetPathConditionString());
PexObserve.ValueForViewing("Return Value", PexSymbolicValue.ToString(
Triang(1, 1, 1)) );
Here I added it into the Triang() method we used before. Run Pex and you will see the new columns added to the results populated with conditions that led to .
ToRawString method and GetRawPathConditionString method return expressions representing symbolic values and the path condition, formatted as S-expressions.
Method PexObserve.ValueForViewing can also be used to display the value picked by Pex for any variable at any point in your code.
The same engine Pex uses is now available as part of Code Digger, a Visual Studio 2012 extension. We will talk about it in a future post.


Classes
Alignment
BigExample
Borders
CellComments
CellTypes
CreateCells
CreateDateCells
EventExample
FrillsAndFills
HyperlinkFormula
MergedCells
NewLinesInCells
NewSheet
NewWorkbook
OfficeDrawing
OfficeDrawingWithGraphics
Outlines
ReadWriteWorkbook
RepeatingRowsAndColumns
SplitAndFreezePanes
WorkingWithFonts
ZoomSheet |


refresh
UK Covid Data
The below data is provided as a live link from the API at
coronavirus.data.gov.uk
Cumulative Deaths
Cumulative Cases


{{stats.name}}
Player since {{stats.playerSince | ogDate}}
Ranking: {{stats.rank}}
{{stats.wins}}
Total wins
{{stats.losses}}
Total losses
{{stats.avgPointsPerGame | realNumber}}
Avg PPG
{{stats.winRate | winRate}}
Win Rate
{{stats.longestWinningStreak}} Games
{{stats.longestLosingStreak}} Games
{{stats.weeklyGameCount}} Games |
{{stats.weeklyWins}} Wins |
{{stats.weeklyLosses}} Losses |
{{stats.weeklyWinRate | winRate}} Win Rate |
{{game.date | smartDate}}
{{game.winnerName | shortFullName}} won against {{game.loserName | shortFullName}}


At this point we have resolved most of our objections to the original Student_info
structure, so it is worth reviewing what we've done:
class Student_info { public: std::string name() const { return n; } bool valid() const { return !homework.empty(); } // as defined in §9.2.1/157, and changed to read into n instead of name std::istream& read(std::istream&); double grade() const; // as defined in §9.2.1/158 private: std::string n; double midterm, final; std::vector<double> homework; }; bool compare(const Student_info&, const Student_info&);
Users can change the state of a Student_info
object only by calling the read
member
function. They cannot reach inside an object and directly change any of the data members.
Instead, we provide operations that hide our implementation. Finally, all the operations
on Student_info
objects are logically gathered together.


Durable stitch and Quality finish. This T-shirt is stitched for higher durability using the best technology in the industry. Manufactured from Cotton fabric, this T-shirt is very smooth and soft making it comfortable to wear during all Seasons. This fabric is Durable, Odorless and passed through Anti fading treatment which ensures the T-shirt color to be intact even after repeated washes.
Inkast Denim Co. offers a mix of classic as well as contemporary denim styles in a variety of washes. Designed for all casual occasions, Inkast offers the authentic denim look with high emphasis on quality, craftsmanship and fit.
street style store brand, Symbol is built on the pillars of quality, reliability and affordability, offering a range of lifestyle essentials that help you look good every day. Symbol brings a range of classic chinos a variety of colors that keep you subtle & comfortable. Built for varsatility, our essential pair of chinos find place in every man’s wardrobe. This pair of Relaxed fit 100% cotton chinos are styled with slant pockets on the side and a buttoned welt pocket at the back. Wear them to work with a classic shirt or keep it casual with a polo


Instructions
Follow the instructions below to get your Google Drive file link
Open Google Drive
- Right click on a file.
- Select "Get link" and click "Copy link"
- Paste the link here and click “Generate Direct Link” button
Google Direct Download Link Generator
Google Drive link
Paste your Google Drive File link below
Audio Embed Link
If your link was an Audio file, use the code snippet to embed it on a web page.
Video/Doc Embed Link
If your link was an Video or Document, use the code snippet to embed it on a web page.


Welcome to the redeveloped Sentinel Hotspots monitoring system.
The Commonwealth and other copyright holders advise that the information available from this website is to be used as a management tool only. The information is able to indicate relative temperature differences or "hotspots", but is not intended to be able to indicate the cause of or identify the nature of such "hotspots".
The user needs to be aware that information on this website is dependent on the availability and functioning of satellites. The quality and geo-positional accuracy of the information cannot be guaranteed. Information may therefore be outdated, inaccurate or incomplete at the time of using this website due to circumstances such as unavailability of or delay in data supply from satellites, inability to obtain precise data caused by the inherent spatial inaccuracies of satellite sensors or inability to obtain complete data due to the prevailing weather conditions at the time of over-flight.
No reliance or actions must therefore be made on this information without seeking prior professional, scientific and technical advice to determine the identity and actual cause of the "hotspot" and without carrying out on the ground surveillance and investigations.
To the extent permitted by law, the Commonwealth and other copyright holders (including their employees and consultants) exclude all liability to any person for any consequences, including but not limited to all losses, damages, costs, expenses and any other compensation, arising directly or indirectly from using this website (in part or in whole) and any information or material contained in it. any consequences, including but not limited to all losses, damages, costs, expenses and any other compensation, arising directly or indirectly from using this website (in part or in whole) and any information or material contained in it.
By clicking "I Agree", you are acknowledging and accepting this Disclaimer Agreement.


Hey!
I'M Pranav Agarwal. I own a Youtube channel I just started my youtube
carrier. so to watch my videos you can click the above image.the name of my
channel is
Robodue. This channel is all about technology in which you can learn to code get to
know about some cool products we do unboxing of some cool products
DIY inventions, life hacks some simple Electronics Circuit and anything you
can think off. so visit my channel and keep learning. and dont forget to like all
my Video's and do hit the Subscribe button to be a part of my small
family . I hope you guys will get to know something through my video's
and at last i want to say thank you guys .


In this notebook, we will learn how to solve the regression problem of predicting flight delays, using decision trees and random forests.
The main goals of this project are the following:
Prediction is very difficult, especially if it’s about the future. (Niels Bohr)
Decision trees are a very popular approach to prediction problems. Decision trees can be trained from both categorical and numerical features, to perform classification and regression. They are the oldest and most well-studied types of predictive analytics. In many analytics packages and libraries, most algorithms are devoted either to address classification or regression problems, and they include for example support vector machines (SVM), neural networks, naïve Bayes, logistic regression, and deep learning...
In general, classification refers to the problem of predicting a label, or category, like spam/not spam, rainy/sunny/mild, for some given data. Regression refers to predicting a numeric quantity like salary, temperature, delay time, product's price. Both classification and regression involve predicting one (or more) values given one (or more) other input values. They require labelled data to perform a training phase, which builds the statistical model: they belong to supervised learning techniques.
To understand how regression and classification operate, it is necessary to briefly define the terms that describe their input and output.
Assume that we want to predict the temperature of tomorrow given today's weather information. The weather information is a loose concept. For example, we can use many variables to express today's weather such as:
These variables are called features or dimensions.
Each variable can be quantified. For example, high and low temperatures are measured in degrees Celsius, humidity can be measured as a fraction between 0 and 1, and weather type can be labeled cloudy
, rainy
or clear
...
So, the weather today can be expressed by a list of values: 11.4, 18.0, 0.64, 20, cloudy
.
Each feature is also called a predictor
. Together, they constitute a feature vector
.
A feature whose domain is a set of categories is called categorical feature. In our example, outlook
is a categorical feature. A feature whose values are numerical is called numerical feature. In our example, temperature
is a numerical feature.
Finally, tomorrow's temperature, that is what we want to predict, is called target feature.
The first question that you might ask is: "Why Decision trees and not another approach?"
Well, the literature shows that the family of algorithms known as decision trees can naturally handle both categorical and numeric features. The training process is easy to understand. The model is easy to interpret. They are robust to outliers in the data, meaning that a few extreme and possibly erroneous data points should not affect the tree at all. The model can be trained in parallel easily. The accuracy is comparable to other methods... In short, there are lots of advantages when using decision trees with respect to other methods!
The way we use a tree model is very simple to understand. We can say that this process "mimics" the way humans take decisions. For example, to decide whether to play football or not, a natural question would be "does it rain now?". If yes, the decision is no
. If it's sunny, the condition is favorable to play football. A second natural question could be: "is it windy?". If no, then you may want to stay at home because otherwise it is going to be too hot. Otherwise, a third plausible question could be: "do we have enough people?". If no, then there's no point playing. Otherwise, time to play!
Using a decision tree allows to follow a similar process to that described above (see the image below). Given a new input, the algorithm traverses the tree in a such a way that the input satisfies the condition of each node until reaching a leaf one. The value of the leaf node is the decision.
The tree model in the figure is built from historical information concerning many past days. The feature predictor
contains three features: Rain, Is_Windy, Enough_People.
An example of the training data is as follows:
| Rain | Is_Windy | Enough_People | Play |
|---|---|---|---|
| Yes | Yes | No | No |
| No | No | No | No |
| No | Yes | Yes | Yes |
| No | No | Yes | No |
As you can see, in the training data, we know the values of predictors and we also know the corresponding answer: we have the ground truth.
One limitation of decision trees is that it's easy to incur in overfitting problems. In other words, the model is too fit to the training data, it is too precise and not general enough. So, when testing the quality of predictions with different testing sets, accuracy could fluctuate. To overcome this limitation, the tree can be pruned after it is built, or even be pruned during the training process. Another approach is building a Random Decision Forest
.
A Random Decision Forest
, as its name implies, is a forest of random Decision trees. Each tree element is built randomly from the training data. Randomization generally applies to:
To take a decision, the forest "asks" all trees about their prediction, and then chooses the outcome which is the most voted.
Every day, in US, there are thousands of flights departures and arrivals: unfortunately, as you may have noticed yourself, flight delays are not a rare event!! Now, given historical data about flights in the country, including the delay information that was computed a-posteriori (so the ground truth is available), we want to build a model that can be used to predict how many minutes of delay a flight might experience in the future. This model should provide useful information for the airport to manage better its resources, to minimize the delays and their impact on the journey of its passengers. Alternatively, astute passengers could even use the model to choose the best time for flying, such as to avoid delays.
The data we will use in this notebook has been collected by the RITA (Research and Innovative Technology Administration), and it contains details facets about each air flight that happened in the US between 1987 and 2008. It includes 29 variables such as the origin airport, the destination airport, the scheduled departure time, day, month, the arrival delay... For more information, please visit the following link, that provides a lot of detail on the data. Our goal is to build a model to predict the arrival delay
.
For our project, we can follow a simple methodology:
Now it's time to apply the simple methodology outlined in section 2.3 on the use case of this notebook.
The data has 29 features, that can be either categorical or numerical. For example, the src_airport
(source airport) is categorical: there exist no comparison operator between airport names. We can not say "SGN is bigger than NCE". The departure is numerical, for which a comparison operator exists. For instance, "flight departing before 6PM" can be express by "departure_time < 1800".
In this use case, most features are numerical, except carier
, flight_number
, cancelled
, cancelation_code
and diverted
.
The data contains a header, that is useless in building the statistical model. In addition, we already know the data schema, so we can safely neglect it. Note that there are some features with missing values in some lines of the dataset. The missing values are marked by "NA". These values can cause problems when processing and can lead to unexpected results. Therefore, we need to remove the header and replace all "NA" values by empty values, such as they can be interpreted as null values.
As we have seen already, there are multiple ways to manipulate data:
Next, we will focus on using DataFrames. However, to use DataFrames, the data must be clean (no invalid values). That means we cannot create DataFrame directly from the "RAW" data. Instead, we will first create an RDD from RAW data, produce a new, clean RDD, then transform it to a DataFrame and work on it. The RDD cleaned_data
is an RDD[String]. We need to transform it to RDD[(TypeOfColumn1, TypeOfColumn2,.., TypeOfColumn29)] then call a function to create a DataFrame from the new RDD.
Let's prepare for the cleaning step: Loading the data into an RDD.
First, we need to import some useful python modules for this notebook.
import os
import sys
import re
from pyspark import SparkContext
from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql import Row
from pyspark.sql.functions import *
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib as mpl
import pandas as pd
import numpy as np
import pyspark.sql.functions as func
import matplotlib.patches as mpatches
import warnings
warnings.filterwarnings("ignore")
# to start testing, we can focus on a single year
input_path = "/datasets/airline/1994.csv"
raw_data = sc.textFile(input_path)
from matplotlib.colors import ListedColormap
dark2_colors = [(0.10588235294117647, 0.6196078431372549, 0.4666666666666667),
(0.8509803921568627, 0.37254901960784315, 0.00784313725490196),
(0.4588235294117647, 0.4392156862745098, 0.7019607843137254),
(0.9058823529411765, 0.1607843137254902, 0.5411764705882353),
(0.4, 0.6509803921568628, 0.11764705882352941),
(0.9019607843137255, 0.6705882352941176, 0.00784313725490196),
(0.6509803921568628, 0.4627450980392157, 0.11372549019607843),
(0.10588235294117647, 0.6196078431372549, 0.4666666666666667),
(0.8509803921568627, 0.37254901960784315, 0.00784313725490196),
(0.4588235294117647, 0.4392156862745098, 0.7019607843137254),
(0.9058823529411765, 0.1607843137254902, 0.5411764705882353),
(0.4, 0.6509803921568628, 0.11764705882352941),
(0.9019607843137255, 0.6705882352941176, 0.00784313725490196),
(0.6509803921568628, 0.4627450980392157, 0.11372549019607843)]
cmap_set1 = ListedColormap(['#4daf4a', '#377eb8', '#e41a1c' ])
dark2_cmap=ListedColormap(dark2_colors)
print("number of rows before cleaning:", raw_data.count())
# extract the header
header = raw_data.first() # take(1) returns a list while first gives string
# replace invalid data with NULL and remove header
cleaned_data = (raw_data\
# filter out the header
.filter(lambda x: x != header)
# replace the missing values with empty characters
.map(lambda x: x.replace('NA', ''))
)
print("number of rows after cleaning:", cleaned_data.count())
Now the data is clean, valid and can be used to create DataFrame. First, we will declare the data schema for the DataFrame. By doing that, we can specify the name and data type of each column.
sqlContext = SQLContext(sc)
# Declare the data schema
# see http://stat-computing.org/dataexpo/2009/the-data.html
# for more information
airline_data_schema = StructType([ \
#StructField( name, dataType, nullable)
StructField("year", IntegerType(), True), \
StructField("month", IntegerType(), True), \
StructField("day_of_month", IntegerType(), True), \
StructField("day_of_week", IntegerType(), True), \
StructField("departure_time", IntegerType(), True), \
StructField("scheduled_departure_time", IntegerType(), True), \
StructField("arrival_time", IntegerType(), True), \
StructField("scheduled_arrival_time", IntegerType(), True), \
StructField("carrier", StringType(), True), \
StructField("flight_number", StringType(), True), \
StructField("tail_number", StringType(), True), \
StructField("actual_elapsed_time", IntegerType(), True), \
StructField("scheduled_elapsed_time", IntegerType(), True), \
StructField("air_time", IntegerType(), True), \
StructField("arrival_delay", IntegerType(), True), \
StructField("departure_delay", IntegerType(), True), \
StructField("src_airport", StringType(), True), \
StructField("dest_airport", StringType(), True), \
StructField("distance", IntegerType(), True), \
StructField("taxi_in_time", IntegerType(), True), \
StructField("taxi_out_time", IntegerType(), True), \
StructField("cancelled", StringType(), True), \
StructField("cancellation_code", StringType(), True), \
StructField("diverted", StringType(), True), \
StructField("carrier_delay", IntegerType(), True), \
StructField("weather_delay", IntegerType(), True), \
StructField("nas_delay", IntegerType(), True), \
StructField("security_delay", IntegerType(), True), \
StructField("late_aircraft_delay", IntegerType(), True)\
])
To "convert" an RDD to DataFrame, each element in the RDD must be a list of column values that match the data schema.
# convert each line into a tuple of features (columns)
cleaned_data_to_columns = cleaned_data.map(lambda l: l.split(","))\
.map(lambda cols:
(
int(cols[0]) if cols[0] else None,
int(cols[1]) if cols[1] else None,
int(cols[2]) if cols[2] else None,
int(cols[3]) if cols[3] else None,
int(cols[4]) if cols[4] else None,
int(cols[5]) if cols[5] else None,
int(cols[6]) if cols[6] else None,
int(cols[7]) if cols[7] else None,
cols[8] if cols[8] else None,
cols[9] if cols[9] else None,
cols[10] if cols[10] else None,
int(cols[11]) if cols[11] else None,
int(cols[12]) if cols[12] else None,
int(cols[13]) if cols[13] else None,
int(cols[14]) if cols[14] else None,
int(cols[15]) if cols[15] else None,
cols[16] if cols[16] else None,
cols[17] if cols[17] else None,
int(cols[18]) if cols[18] else None,
int(cols[19]) if cols[19] else None,
int(cols[20]) if cols[20] else None,
cols[21] if cols[21] else None,
cols[22] if cols[22] else None,
cols[23] if cols[23] else None,
int(cols[24]) if cols[24] else None,
int(cols[25]) if cols[25] else None,
int(cols[26]) if cols[26] else None,
int(cols[27]) if cols[27] else None,
int(cols[28]) if cols[28] else None
))
To train our model, we use the following features: year
, month
, day_of_month
, day_of_week
, scheduled_departure_time
, scheduled_arrival_time
, arrival_delay
, distance
, src_airport
, dest_airport
.
# create dataframe df
df = (sqlContext.createDataFrame(data=cleaned_data_to_columns, schema=airline_data_schema)
.select(
['year', 'month', 'day_of_month', 'day_of_week', 'scheduled_departure_time',
'scheduled_arrival_time', 'arrival_delay', 'distance', 'src_airport', 'dest_airport', 'carrier'
])
.cache()
)
Next, we will go over a series of simple queries on our data, to explore it and compute statistics. These queries directly map to the questions you need to answer.
NOTE: finding the right question to ask is difficult! Don't be afraid to complement the questions below, with your own questions that, in your opinion, are valuable ways to inspect data. This can give you extra points!
Basic queries:
Statistic on flight volume: this kind of statistics are helpful to reason about delays. Indeed, it is plausible to assume that "the more flights in an airport, the higher the probability of delay".
Statistic on the fraction of delayed flights
num_src_airport = df.select('src_airport').distinct().count()
num_dest_airport = df.select('dest_airport').distinct().count()
print("number of origin airports:", num_src_airport)
print("number of destination airports:", num_src_airport)
num_carrier = df.select('carrier').distinct().count()
print("the number distinct carriers:", num_carrier)
print("the number of night flights:", df[df.scheduled_departure_time > 1800].count())
from IPython.display import display
statistic_month = df.groupBy('month').count().collect()
#statistic_day_of_week.show()
pdf = pd.DataFrame(data=statistic_month, columns=['month', 'num_flights'])
pdf = pdf.sort_values(['month'])
str_month = ['Jan','Feb','Mar','Apr','May','Jun',
'Jul','Aug','Sep', 'Oct', 'Nov','Dec']
pdf['MonthName']=pd.Series(str_month, pdf.index)
display(pdf[['MonthName', 'num_flights']])
#map_into_month = {1: 'Jan', 2: 'Feb', 3: 'Mar', 4: 'Apr', 5: 'May', 6: 'Jun',
#7: 'Jul', 8: 'Aug', 9: 'Sep', 10: 'Oct', 11: 'Nov', 12: 'Dec'}
#month_label = pdf['month'].map(lambda m: map_into_month[m])
#print(pdf.to_html())
plt.figure(figsize=(15, 9))
plt.xlabel("Months", fontsize=16, fontstyle='oblique')
plt.ylabel("Number of flights", fontsize=16, fontstyle='oblique')
plt.title('Figure 1: The change of number flights over months', fontsize=20, fontweight='semibold')
plt.grid(True,which="both",ls="-")
#plt.xticks(pdf['month'], month_label)
plt.tight_layout()
plt.plot(pdf['month'], pdf['num_flights'], color=dark2_colors[7], linewidth=5)
plt.fill_between(pdf['month'], pdf['num_flights'], 340000, color=dark2_colors[7], alpha=0.3)
plt.xticks(pdf['month'], str_month, fontsize=18)
plt.show()
len(pdf['month'])
statistic_day_of_week = df.groupBy('day_of_week').count().collect()
#statistic_day_of_week.show()
pdf = pd.DataFrame(data=statistic_day_of_week, columns=['day_of_week', 'num_flights'])
#pdf = pdf.sort_values(['day_of_week'])
map_int_into_day = { 1:"Mon", 2:"Tue", 3:"Wed", 4:"Thu", 5:"Fri", 6:"Sat", 7:"Sun" }
day_of_week_label = pdf['day_of_week'].map(lambda i: map_int_into_day[i])
pdf['DayName']=pd.Series(day_of_week_label, pdf.index)
display(pdf[['DayName', 'num_flights']])
plt.figure(figsize=(15, 9))
plt.xlabel("Days of week", fontsize=16, fontstyle='oblique')
plt.ylabel("Number of flights", fontsize=16, fontstyle='oblique')
plt.title('Figure 2: The change of number flights over days of week', fontsize=22, fontweight='bold')
plt.grid(True, axis='y',ls=":")
# plot bar chart
plt.bar(pdf['day_of_week'], pdf['num_flights'], color=dark2_colors[0], align='center')
plt.xticks(pdf['day_of_week'], day_of_week_label, fontsize=18)
plt.show()
statistic_day_of_month = df.groupBy('day_of_month').count().collect()
pdf = pd.DataFrame(data=statistic_day_of_month, columns=['day_of_month', 'num_flights'])
pdf = pdf.sort_values(['day_of_month'])
display(pdf)
# plot
plt.figure(figsize=(16, 8))
plt.xlabel("Day of month", fontsize=16, fontstyle='oblique')
plt.ylabel("Number of flights", fontsize=16, fontstyle='oblique')
plt.title("Figure 3: The change of number of flights over days of month", fontsize=22, fontweight='bold')
plt.grid(True, axis='y', ls=':')
plt.bar(pdf['day_of_month'], pdf['num_flights'], color=dark2_colors[8])
plt.xticks(pdf['day_of_month'], fontsize=18)
plt.show()
statistic_hour_of_day = df.groupBy('scheduled_departure_time').count().collect()
#statistic_hour_of_day.show()
pdf = pd.DataFrame(data=statistic_hour_of_day, columns=['dep_time', 'num_flights'])
pdf = pdf.sort_values(['dep_time'])
print(display(pdf.describe()))
#plot char chart
plt.figure(figsize=(16, 8))
plt.xlabel('Departure time', fontsize=16, fontstyle='oblique')
plt.ylabel('Number of flights', fontsize=16, fontstyle='oblique')
plt.title('Figure 4: The change of number flights over hours of day', fontsize=22, fontweight='bold')
plt.grid(True, which='both', ls='-')
plt.bar(pdf['dep_time'], pdf['num_flights'])
plt.show()
# consider outbound flights
stat_src = (df
.groupBy(df.src_airport)
.agg(func.count('*').alias('count1'))
)
# consider inbound flights
stat_dest = (df
.groupBy(df.dest_airport)
.agg(func.count('*').alias('count2'))
)
#print(stat_src.take(10))
#print(stat_dest.take(10))
# full join the statistic of inbound flights and outbound flights
stat_airports = stat_src.join(stat_dest, stat_src.src_airport == stat_dest.dest_airport, how='inner')
# TOP 20 BUSIEST AIRPORTS
stat_airport_traffic = (stat_airports
# define the new column `total`
# which has values are equal to the sum of `count1` and `count2`
.withColumn('total', stat_airports['count1'] + stat_airports['count2'])
# select top airpoint in termns of number of flights
.select(['dest_airport', 'total']).orderBy(desc('total'))
)
#display(stat_airport_traffic.show())
pdf = pd.DataFrame(data=stat_airport_traffic.take(20), columns=['dest_airport', 'total'])
bins = range(0, len(pdf['total']))
plt.figure(figsize=(15, 9))
plt.xlabel('Airport', fontsize=16, fontstyle='oblique')
plt.ylabel('Number of flights', fontsize=16, fontstyle='oblique')
plt.title('Figure 5: Number of flights per airport', fontsize=22, fontweight='bold')
plt.grid(True, which='both')
plt.bar(bins, pdf['total'], align='center', color=dark2_colors[11])
plt.xticks(bins, pdf['dest_airport'])
plt.show()
stat_carrier = (df
.groupBy(df.carrier)
.agg(func.count('*').alias('count'))
.orderBy(desc('count'))
)
display(pd.DataFrame(stat_carrier.take(20), columns=['carrier', 'count']))
Statistics on the percentage of delayed flights
# 'year', 'month', 'day_of_month', 'day_of_week', 'scheduled_departure_time',
# 'scheduled_arrival_time', 'arrival_delay', 'distance', 'src_airport', 'dest_airport', 'carrier'
# create new column that marks whether the flights are delay
df_with_delay = df.withColumn('is_delay', when(df['arrival_delay'] >= 15, 1).otherwise(0))
# create a new column that indicates the scheduled departure time in hour
# (ignore the part of minute)
delay_per_hour = df_with_delay.withColumn('hour', round(df.scheduled_departure_time/100, 0))
# group by year and hour
statistic_delay_hour = delay_per_hour.groupBy(['year', 'hour'])
# calculate the delay ratio and create a new column
delay_ratio_per_hour = statistic_delay_hour.agg(
(func.sum('is_delay')/func.count('*')).alias('delay_ratio')
)
# order the result by hour
delay_ratio_per_hour = (
delay_ratio_per_hour
.orderBy('hour')
.select(['hour', 'delay_ratio'])
)
pdf_delay_ratio_per_hour = pd.DataFrame(data=delay_ratio_per_hour.collect(), columns=['hour', 'delay_ratio'])
# plot a bar chart
plt.figure(figsize=(13, 7))
plt.grid(True, which='both', ls='-')
plt.xlabel('Hour of day', fontsize=16, fontstyle='oblique')
plt.ylabel('Delay ratio', fontsize=16, fontstyle='oblique')
plt.title('Figure 4: The ratio of delay over hours in day', fontsize=22, fontweight='bold')
plt.bar(pdf_delay_ratio_per_hour['hour'], pdf_delay_ratio_per_hour['delay_ratio'], color=dark2_colors[13])
plt.show()
mean_delay_per_hour = statistic_delay_hour.agg(
(func.mean('arrival_delay')).alias('mean_delay')
)
mean_delay_per_hour = (
mean_delay_per_hour
.orderBy('hour')
.select(['hour','mean_delay'])
)
pdf_mean_delay_per_hour = pd.DataFrame(data=mean_delay_per_hour.collect(), columns=['hour', 'mean_delay'])
plt.figure(figsize=(13, 7))
plt.grid(True, which='both', ls='-')
plt.xlabel('Hour', fontsize=16, fontstyle='oblique')
plt.ylabel('Mean delay', fontsize=16, fontstyle='oblique')
plt.title('Figure 5: The delay over hours in day', fontsize=22, fontweight='bold')
plt.bar(pdf_mean_delay_per_hour['hour'], pdf_mean_delay_per_hour['mean_delay'], color=dark2_colors[3])
plt.show()
With data of year 1994, the flight from 3AM to 4AM often depart earlier than in their schedule. The flights in the morning have less delay then in the afternoon and evening.
So, an attentive student should notice here that we have somehow a problem with the definition of delay! Next, we will improve how to represent and visualize data to overcome this problem.
#pdf2 = pd.DataFrame(data=mean_delay_per_hour.collect())
plt.figure(figsize=(13, 7))
plt.xlabel("Hours", fontsize=16, fontstyle='oblique')
plt.ylabel("Ratio of delay", fontsize=16, fontstyle='oblique')
plt.title('Figure 6: The ratio of delay over hours in day', fontsize=22, fontweight='bold')
plt.grid(True,which="both",ls="-")
bars = plt.bar(pdf_delay_ratio_per_hour['hour'], pdf_delay_ratio_per_hour['delay_ratio'], align='center', edgecolor = "black")
for i in range(0, len(bars)):
color = 'red'
if pdf_mean_delay_per_hour['mean_delay'][i] < 0:
color = 'lightgreen'
elif pdf_mean_delay_per_hour['mean_delay'][i] < 2:
color = 'green'
elif pdf_mean_delay_per_hour['mean_delay'][i] < 4:
color = 'yellow'
elif pdf_mean_delay_per_hour['mean_delay'][i] < 8:
color = 'orange'
bars[i].set_color(color)
patch1 = mpatches.Patch(color='lightgreen', label='Depart earlier')
patch2 = mpatches.Patch(color='green', label='delay < 2 minutes')
patch3 = mpatches.Patch(color='yellow', label='delay < 4 minutes')
patch4 = mpatches.Patch(color='orange', label='delay < 8 minutes')
patch5 = mpatches.Patch(color='red', label='delay >= 8 minutes')
plt.legend(handles=[patch1, patch2, patch3, patch4, patch5], loc='left',
borderaxespad=0.)
plt.show()
In the new figure (Figure 6), we have more information in a single plot. The flights in 3AM to 4AM have very low probability of being delayed, and actually depart earlier than their schedule. In contrast, the flights in the 4PM to 8PM range have higher chances of being delayed: in more than 50% of the cases, the delay is 8 minutes or more.
This example shows us that the way representing results are also important.
pdArrivDelay = pd.DataFrame(df.select(['arrival_delay', ]).collect(), columns=['Delay'])
# Plot The distribution of flight delays
plt.figure(figsize=(14, 7))
plt.xlabel("Arrival delay in minutes", fontsize=16, fontstyle='oblique')
plt.ylabel("Probability", fontsize=16, fontstyle='oblique')
plt.title('Figure 7: Arrival Delay Distribution', fontsize=22, fontweight='bold')
plt.grid(True, axis='y',ls="-")
plt.hist(pdArrivDelay.Delay.dropna(), bins=400, normed=1, alpha=0.75, color=dark2_colors[2])
plt.xlim(-25,150)
plt.show()
####### The changes of delay ratio over days of month ########
# calculate the delay ratio in each day of month
statistic_day_of_month = (
df_with_delay
.groupBy('day_of_month')
# is_dealy is either 0 or 1
.agg(func.sum('is_delay')/func.count('*')).alias('delay_ratio')
# order by day_of_month
.orderBy('day_of_month')
)
# collect data and plot
pdf_day_of_month = pd.DataFrame(data=statistic_day_of_month.collect(), columns=['day_of_month', 'delay_ratio'])
display(pdf_day_of_month.describe())
plt.figure(figsize=(14, 7))
plt.xlabel("Day of month", fontsize=16, fontstyle='oblique')
plt.ylabel("Mean delay", fontsize=16, fontstyle='oblique')
plt.title('Figure 8: The ratio of delay over days of month', fontsize=22, fontweight='bold')
plt.grid(True, axis='y',ls=":")
plt.bar(pdf_day_of_month['day_of_month'], pdf_day_of_month['delay_ratio'], color=dark2_colors[9])
plt.show()
####### The changes of delay ratio over days of week ########
# calculate the delay ratio in each day of week
statistic_day_of_week = (
df_with_delay
.groupBy('day_of_week')
.agg(func.sum('is_delay')/func.count('*')).alias('delay_ratio')
# order by day_of_month
.orderBy('day_of_week')
)
# collect data and plot
pdf_day_of_week = pd.DataFrame(data=statistic_day_of_week.collect(), columns=['day_of_week', 'ratio'])
map_int_into_day = { 1:"Mon", 2:"Tue", 3:"Wed", 4:"Thu", 5:"Fri", 6:"Sat", 7:"Sun" }
day_of_week_label = pdf_day_of_week['day_of_week'].map(lambda i: map_int_into_day[i])
plt.figure(figsize=(14, 7))
plt.xlabel("Day of the week", fontsize=16, fontstyle='oblique')
plt.ylabel("Ratio", fontsize=16, fontstyle='oblique')
plt.title('Figure 9: The ratio of delay over days of week', fontsize=22, fontweight='bold')
plt.grid(True, axis='y',ls=":")
plt.xticks(pdf_day_of_week['day_of_week'], day_of_week_label, fontsize = 20)
plt.bar(pdf_day_of_week['day_of_week'], pdf_day_of_week['ratio'], align='center', color=dark2_colors[11])
plt.show()
####### The changes of delay ratio over months ########
# calculate the delay ratio in month
statistic_month = (
df_with_delay.groupBy('month').agg(func.sum('is_delay')/func.count('*').alias('delay_ratio'))
)
# collect data and plot
pdf_month = pd.DataFrame(data=statistic_month.collect(), columns=['month', 'ratio'])
map_into_month = {1: 'Jan', 2: 'Feb', 3: 'Mar', 4: 'Apr', 5: 'May', 6: 'Jun',
7: 'Jul', 8: 'Aug', 9: 'Sep', 10: 'Oct', 11: 'Nov', 12: 'Dec'}
month_label = pdf_month['month'].map(lambda m: map_into_month[m])
plt.figure(figsize=(14, 7))
plt.xlabel("Month", fontsize=16, fontstyle='oblique')
plt.ylabel("Ratio", fontsize=16, fontstyle='oblique')
plt.title('Figure 10: The ratio of delay over months', fontsize=22, fontweight='bold')
plt.grid(True, axis='y',ls=":")
plt.xticks(pdf_month['month'], month_label, fontsize=18)
plt.bar(pdf_month['month'], pdf_month['ratio'], align='center', color=dark2_colors[12])
plt.show()
We are ready now to draw some observations from our data, even if we have only looked at data coming from a year worth of flights:
Putting things together, we can have a global picture of the whole year!
df_with_delay = df.withColumn('is_delay', when(df["arrival_delay"] >= 15, 1).otherwise(0))
statistic_day = df_with_delay.groupBy(['year','month', 'day_of_month', 'day_of_week'])\
.agg((func.sum('is_delay')/func.count('*')).alias('delay_ratio'))
# assume that we do statistic on year 1994
statistic_day = statistic_day\
.orderBy('year', 'month', 'day_of_month', 'day_of_week')
pdf = pd.DataFrame(data=statistic_day.collect(), columns=
['year','month', 'day_of_month', 'day_of_week', 'delayRatio'])
pdf.describe()
fig = plt.figure(figsize=(25,14))
plt.xlabel("Weeks/Months in year", fontsize=16, fontstyle='oblique')
plt.ylabel("Day of weeks (1:Monday -> 7 :Sunday)", fontsize=16, fontstyle='oblique')
plt.title('Figure 11: The change of number flights over days in year', fontsize=22, fontweight='bold')
ax = fig.add_subplot(1,1,1)
rec_size = 0.3
from matplotlib.patches import Rectangle
import datetime
num_days = len(pdf.year)
ax.patch.set_facecolor('gray')
ax.set_aspect('equal', 'box')
ax.xaxis.set_major_locator(plt.NullLocator())
ax.yaxis.set_major_locator(plt.NullLocator())
for i in range(0, num_days):
# extract information from the result
year = pdf.year[i]
month = pdf.month[i]
day_of_month = pdf.day_of_month[i]
day_of_week = pdf.day_of_week[i]
day_of_year= datetime.date(year=year, month=month, day=day_of_month).timetuple()
week_of_year = datetime.date(year=year, month=month, day=day_of_month).isocalendar()[1]
# dealing with the week of the previous year
if week_of_year == 52 and month == 1:
week_of_year = 0
# the coordinate of a day in graph
X = week_of_year*rec_size
Y = day_of_week*rec_size
# use different colors to show the delay ratio
cMap = ListedColormap(['white', 'palegreen', 'greenyellow', 'yellow','gold', 'orange', 'tomato', 'brown', 'maroon'])
if pdf.delayRatio[i] <= 0.086802:
color = 'white'
elif pdf.delayRatio[i] <= 0.106950:
color = 'palegreen'
elif pdf.delayRatio[i] <= 0.130625:
color = 'greenyellow'
elif pdf.delayRatio[i] <= 0.152415:
color = 'yellow'
elif pdf.delayRatio[i] <= 0.180998:
color = 'gold'
elif pdf.delayRatio[i] <= 0.210556:
color = 'orange'
elif pdf.delayRatio[i] <= 0.246090:
color = 'tomato'
elif pdf.delayRatio[i] <= 0.315542:
color = 'brown'
else:
color = 'maroon'
rect = plt.Rectangle((X - rec_size/2.0, Y - rec_size/2.0), rec_size, rec_size,
alpha=1, facecolor=color, edgecolor='whitesmoke')
ax.add_patch(rect)
# drawing boders to separate months
if day_of_month <= 7:
rect2 = plt.Rectangle((X -rec_size/2.0, Y - rec_size/2.0), 0.01, rec_size,
alpha=1, facecolor='black')
ax.add_patch(rect2)
if day_of_month == 1:
rect2 = plt.Rectangle((X - rec_size/2.0, Y - rec_size/2.0), rec_size, 0.01,
alpha=1, facecolor='black')
ax.add_patch(rect2)
ax.autoscale_view()
plt.show()
fig = plt.figure(figsize=(15, 1))
ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.1])
norm = mpl.colors.Normalize(vmin=0, vmax=43.0963)
plt.title('% of delay ratio')
cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cMap,
norm=norm,
orientation='horizontal')
# Importing Airport names
airportInfo = pd.read_csv("support/airports.csv")
display(airportInfo[:10])
airportNames = airportInfo.drop(['lat', 'long', 'country', 'state', 'airport'],1)
display(airportNames[:10])
####### The delay ratio of the top 20 busiest airports ########
K = 20
# extract top_20_airports from stat_airport_traffic
top_20_airports = [item[0] for item in stat_airport_traffic.take(K)]
# select the statistic of source airports
statistic_ratio_delay_airport = (
df_with_delay
# select only flights that depart from one of top 20 ariports
.filter(df_with_delay.src_airport.isin(top_20_airports))
# group by source airport
.groupBy('src_airport')
# calculate the delay ratio
.agg(func.sum('is_delay')/func.count('*').alias('delay_ratio'))
# sort by name of airport
.orderBy(['src_airport'])
)
top_20_airportsPD = pd.DataFrame(statistic_ratio_delay_airport.take(20), columns=['src_airport', 'delay_ratio'])
top_20_airportsRenamed = pd.merge(top_20_airportsPD, airportNames, left_on='src_airport', right_on='iata', how='inner')
display(top_20_airportsRenamed.drop('iata', 1))
# collect data and plot
pdf_ratio_delay_airport = pd.DataFrame(data=statistic_ratio_delay_airport.collect())
pdf_top_20_airport_volume = pd.DataFrame(data=stat_airport_traffic.take(K), columns=['src_airport', 'total'])
pdf_top_20_airport_volume = pdf_top_20_airport_volume.sort_values(by='src_airport')
#print(pdf_top_20_airport_volume)
index = np.arange(len(top_20_airports))
bar_width = 0.35
opacity = 0.4
fig = plt.figure(figsize=(20,10))
ax = fig.add_subplot(1,1,1)
ax2 = ax.twinx()
plt.axis('normal')
ax.set_xlabel("Airport", fontsize=22, fontstyle='oblique')
ax.set_ylabel("Flight volume", fontsize=22, fontstyle='oblique')
ax2.set_ylabel("Ratio of delay", fontsize=22, fontstyle='oblique')
plt.xticks(index + bar_width, top_20_airportsRenamed.city)
plt.setp(ax.get_xticklabels(), rotation=40, fontsize=22, ha='right')
plt.title('Figure 12: Flight volume & Delay ratio for Airports', fontsize=26, fontweight='bold')
plt.grid(True, axis='y',ls=":")
bar = ax.bar(index, pdf_top_20_airport_volume['total'],
bar_width, color=dark2_colors[1],
label='flight volume')
bar2 = ax2.bar(index + 1.5*bar_width, pdf_ratio_delay_airport[1], bar_width,
align='center', color=dark2_colors[7],
label='Delay ratio')
lines, labels = ax.get_legend_handles_labels()
lines2, labels2 = ax2.get_legend_handles_labels()
ax2.legend(lines + lines2, labels + labels2, loc=0, prop={'size':25})
plt.tight_layout()
plt.show()
K = 20
# extract top_20_carriers from stat_carrier
top_20_carriers = [item[0] for item in stat_carrier.take(K)]
statistic_ratio_delay_carrier = (
df_with_delay
# select only flights that belong from one of top 20 carriers
.filter(df_with_delay.carrier.isin(top_20_carriers))
# group by carriers
.groupBy('carrier')
# calculate the delay ratio
.agg(func.sum('is_delay')/func.count('*').alias('delay_ratio'))
# sort by name of airport
.orderBy(['carrier'])
)
top_20_carriers_delay = pd.DataFrame(statistic_ratio_delay_carrier.take(20), columns=['carrier', 'delay_ratio'])
display(top_20_carriers_delay)
# Importing Carrier names
carrierInfo = pd.read_csv("support/carriers.csv")
carrierInfo.Description = carrierInfo.Description.map(lambda x:x.split(',')[0])
carrierInfo.Description = carrierInfo.Description.map(lambda x:x.split('.')[0])
display(carrierInfo[:10])
# collect data and translate names
pdf_ratio_delay_carrier = pd.DataFrame(data=statistic_ratio_delay_carrier.collect())
pdf_top_20_carrier_volume = pd.DataFrame(data=stat_carrier.take(20), columns=['carrier', 'count'])
pdf_top_20_carrier_volume = pdf_top_20_carrier_volume.sort_values(by='carrier')
pdf_top_20_carrier_volume = pd.merge(pdf_top_20_carrier_volume, carrierInfo, left_on='carrier', right_on='Code')
pdf_top_20_carrier_volume = pdf_top_20_carrier_volume.drop(['Code'],1)
display(pdf_top_20_carrier_volume)
# Plot top 20 carriers
top_20_carriers.sort()
index = np.arange(len(top_20_carriers))
bar_width = 0.35
opacity = 0.4
fig = plt.figure(figsize=(20,12))
ax = fig.add_subplot(1,1,1)
ax2 = ax.twinx()
plt.axis('normal')
ax.set_xlabel("Carrier", fontsize=22, fontstyle='oblique')
ax.set_ylabel("Flight volume", fontsize=22, fontstyle='oblique')
ax2.set_ylabel("Ratio of delay", fontsize=22, fontstyle='oblique')
plt.xticks(index + bar_width, pdf_top_20_carrier_volume.Description)
plt.setp(ax.get_xticklabels(), rotation=40, fontsize=22, ha='right')
plt.title('Figure 13: Delay ratio and flight volume of Carriers', fontsize=26, fontweight='bold')
plt.grid(True,which="both",ls=":")
bar = ax.bar(index, pdf_top_20_carrier_volume['count'],
bar_width, color=dark2_colors[6],
label='flight volume')
bar2 = ax2.bar(index + 1.5*bar_width, pdf_ratio_delay_carrier[1], bar_width,
align='center', color=dark2_colors[7],
label='Delay ratio')
lines, labels = ax.get_legend_handles_labels()
lines2, labels2 = ax2.get_legend_handles_labels()
ax2.legend(lines + lines2, labels + labels2, loc=0)
plt.tight_layout()
plt.show()
Now that we have a good grasp on our data and its features, we will focus on how build a statistic model. Note that the features we can decide to use, to train our model, can be put in two groups:
weather
(for example querying a public API)is_weekend
, is_holiday
, season
, in_winter
,...In this notebook, we will focus on the following predictors: year
, month
, day_of_month
, day_of_week
, scheduled_departure_time
, scheduled_arrival_time
, carrier
, is_weekend
, distance
, src_airport
, dest_airport
. Among them, is_weekend
is an implicit feature. The rest are explicit features.
The target feature is arrival_delay
.
Currently, MLLIB only supports building models from RDDs. It is important to read well the documentation and the MLLib API, to make sure to use the algorithms in an appropriate manner:
LabelledPoint
. This data structure has 2 components: label
and predictor vector
. label
is the value of target feature in the current record. predictor vector
is a vector of values of type Double
. As such, we need to map each value of each categorical feature to a number. In this project, we choose a naïve approach: map each value to a unique index. maxBins
bins (32 by default). With categorical features, each distinct value fits in its own bin. IMPORTANT: MLLIB requires that no categorical feature have more than maxBins
distinct values.Among the selected features, src_aiport
, dest_airport
, carrier
and distance
have missing values. Besides, the first three of them are categorical features. That means, in order to use them as input features of MLLIB, the values of these features must be numerical. We can use a naïve approach: map each value of each feature to a unique index.
# select distinct source airports and map values to index
# sort the airport by their frequency descending
# so the most common aiport will be on the top
stat_src = (
df
.groupBy('src_airport')
.agg(func.count('*').alias('count'))
.orderBy(desc('count'))
)
# extract the airport names from stat_src
src_airports = [item[0] for item in stat_src.collect()]
num_src_airports = len(src_airports)
src_airports_idx = range(0, num_src_airports)
map_src_airport_to_index = dict(zip(src_airports, src_airports_idx))
# test the dictionary
print('Index of most popular SOURCE airport (ORD, matches data from figure 5): ', map_src_airport_to_index['ORD'])
print('Index of 3rd most SOURCE popular irport (ATL): ', map_src_airport_to_index['ATL'])
#print(map_src_airport_to_index)
# select distinct destination airports and map values to index
# sort the airport by their frequency descending
# so the most common aiport will be on the top
stat_dest = (
df
.groupBy('dest_airport')
.agg(func.count('*').alias('count'))
.orderBy(desc('count'))
)
dest_airports = [item[0] for item in stat_dest.collect()]
num_dest_airports = len(dest_airports)
num_dest_airports_idx = range(0, num_dest_airport)
map_dest_airports_to_index = dict(zip(dest_airports, num_dest_airports_idx))
# test the dictionary
print('Index of most popular DESTINATION airport (ORD, matches data from figure 5): ', map_dest_airports_to_index['ORD'])
print('Index of 3rd most DESTINATION popular irport (ATL): ', map_dest_airports_to_index['ATL'])
#print(map_dest_airports_to_index)
stat_carrier = (
df
.groupBy('carrier')
.agg(func.count('*').alias('count'))
.orderBy(desc('count'))
)
# select distinct carriers and map values to index
# sort carriers by their frequency descending
# so the most common aiport will be on the top
carriers = [item[0] for item in stat_carrier.collect()]
num_carriers = len(carriers)
map_carriers_to_index = dict(zip(carriers, range(0, num_carrier)))
print(map_carriers_to_index)
We use a simple strategy for filling in the missing values: replacing them with the most common value of the corresponding feature.
In the previous question, when constructing the dictionary for categorical features, we also sort their statistical information in a such way that the most common value of each feature are placed on the top.
Note that, feature is_weekend
has the most common value set to 0
(that is, no
the day is not a weekend).
def most_common(groupBy):
return (df
.groupBy(groupBy)
.agg(func.count('*').alias('count'))
.orderBy(desc('count'))
).first()[0]
the_most_common_month = most_common('month')
print("The most common month:", the_most_common_month)
the_most_common_day_of_month = most_common('day_of_month')
the_most_common_day_of_week = most_common('day_of_week')
print("The most common day of month:", the_most_common_day_of_month)
print("The most common day of week:", the_most_common_day_of_week)
the_most_common_s_departure_time = most_common('scheduled_departure_time')
the_most_common_s_arrival_time = most_common('scheduled_arrival_time')
print("The most common scheduled departure time:", the_most_common_s_departure_time)
print("The most common scheduled arrival time:", the_most_common_s_arrival_time)
# calculate mean distance
mean_distance = df.agg(func.mean('distance')).first()[0]
print("mean distance:", mean_distance)
# calculate mean arrival delay
mean_arrival_delay = df.agg(func.mean('arrival_delay')).first()[0]
print("mean arrival delay:", mean_arrival_delay)
As known from section 3.4, there are 225 different origin airports and 225 different destination airports, more than the number of bins in default configuration. So, we must set maxBins
>= 225.
Recall, in this project we focus on decision trees. One way to think about our task is that we want to predict the unknown arrival_delay
as a function combining several features, that is:
arrival_delay = f(year, month, day_of_month, day_of_week, scheduled_departure_time, scheduled_arrival_time, carrier, src_airport, dest_airport, distance, is_weekend)
When categorical features contain corrupt data (e.g., missing values), we proceed by replacing corrupt information with the most common value for the feature. For numerical features, in general, we use the same approach as for categorical features; in some cases, we repair corrupt data using the mean value of the distribution for numerical features (e.g., we found the mean for delay
and distance
, by answering questions above).
The original data is split randomly into two parts with ratios 70% for training and 30% for testing.
from pyspark.mllib.tree import DecisionTree, DecisionTreeModel
from pyspark.mllib.util import MLUtils
from pyspark.mllib.regression import LabeledPoint
def is_valid(value):
return value != "NA" and len(value) > 0
data = cleaned_data\
.map(lambda line: line.split(','))\
.map(lambda values:
LabeledPoint(
int(values[14]) if is_valid(values[14]) else mean_arrival_delay, # arrival delay
[
int(values[0]), # year
int(values[1]) if is_valid(values[1]) else the_most_common_month, # month
int(values[2]) if is_valid(values[2]) else the_most_common_day_of_month, # day of month
int(values[3]) if is_valid(values[3]) else the_most_common_day_of_week, # day of week
int(values[5]) if is_valid(values[5]) else the_most_common_s_departure_time, # scheduled departure time
int(values[7]) if is_valid(values[7]) else the_most_common_s_arrival_time, # scheduled arrival time
# if the value is valid, map it to the corresponding index
# otherwise, use the most common value
map_carriers_to_index[values[8]] if is_valid(values[8]) \
else map_carriers_to_index[carriers[0]], # carrier
map_src_airport_to_index[values[16]] if is_valid(values[16]) \
else map_src_airport_to_index[src_airports[0]], # src_airport
map_dest_airports_to_index[values[17]] if is_valid(values[17]) \
else map_dest_airports_to_index[dest_airports[0]], # destination_airport
int(values[18]) if is_valid(values[18]) else mean_distance, # distance
1 if is_valid(values[3]) and int(values[3]) >= 6 else 0, # is_weekend
]
)
)
# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])
trainingData = trainingData.cache()
testData = testData.cache()
trainingData.take(10)
`DecisionTree.trainRegressor(, categoricalFeaturesInfo=, impurity=, maxDepth=, maxBins=)`.
Where,
# declare information of categorical features
# format: feature_index : number_distinct_values
categorical_info = {6 : num_carriers, 7: num_src_airports , 8: num_dest_airports, 10: 2}
# Train a DecisionTree model.
model = DecisionTree.trainRegressor(trainingData,
categoricalFeaturesInfo=categorical_info,
impurity='variance', maxDepth=12, maxBins=255)
print('Model Number of Nodes: ', model.numNodes())
print('Model Depth: ', model.depth())
# Evaluate model on test instances and compute test error
predictions = model.predict(testData.map(lambda x: x.features))
labelsAndPredictions = testData.map(lambda lp: lp.label).zip(predictions)
testMSE = labelsAndPredictions.map(
lambda p: np.power(p[0] - p[1], 2)).mean()
print('Test Mean Squared Error = ' + str(testMSE))
print('Test RMSE = ' + str(np.sqrt(testMSE)))
testBias = labelsAndPredictions.map(lambda x: x[0]-x[1]).mean()
print('Bias of the model: ', testBias)
Next, we use MLLib to build a more powerful model: random forests. In what follows, use the same predictors defined and computed above to build a decision tree, but this time use them to build a random decision forest.
from pyspark.mllib.tree import RandomForest, RandomForestModel
# Train a RandomForest model.
forest_model = RandomForest.trainRegressor(trainingData, categoricalFeaturesInfo=categorical_info,
numTrees=10, impurity='variance', maxDepth=12, maxBins=255)
random_forest_predictions = forest_model.predict(testData.map(lambda x: x.features))
forestLabelsAndPredictions = testData.map(lambda lp: lp.label).zip(random_forest_predictions)
testMSE = forestLabelsAndPredictions.map(lambda p: np.power(p[0] - p[1], 2)).mean()
print('Random Forest -> Test Mean Squared Error = ' + str(testMSE))
print('Random Forest -> Test RMSE = ' + str(np.sqrt(testMSE)))
testBias = labelsAndPredictions.map(lambda x: x[0]-x[1]).mean()
print('Random Forest -> Bias of the model: ', testBias)
import time
for numTrees in [15]:
for maxDepth in [20]:
for maxBins in [300]:
try:
t_start = time.time()
forest_model = RandomForest.trainRegressor(trainingData, categoricalFeaturesInfo=categorical_info,
numTrees=numTrees, impurity='variance', maxDepth=maxDepth, maxBins=maxBins)
random_forest_predictions = forest_model.predict(testData.map(lambda x: x.features))
forestLabelsAndPredictions = testData.map(lambda lp: lp.label).zip(random_forest_predictions)
testMSE = forestLabelsAndPredictions.map(lambda p: np.power(p[0] - p[1], 2)).mean()
t_end = time.time()
print('Test with forest [numTrees = %f, maxDepth = %f, maxBins = %f] MSE = %f, time = %f' % \
(num.Trees, maxDepth, maxBins, testMSE, t_end-t_start))
except:
print('Test with forest [numTrees = %f, maxDepth = %f, maxBins = %f] -> RUNTIME ERROR' % \
(num.Trees, maxDepth, maxBins)
As you may have noticed, the performance of our statistical models is somehow questionable! What are we missing here? Why is that even using state-of-the-art approaches give poor results?
In what follows, we will try to address some of the limitations of the present Notebook, and provide additional data that might help.
In the HDFS file system you have used for running the Notebook, you will notice that there are several other years available (in addition to 1994), which could be used to train a statistical model with more data. In the end, we're playing with "Big Data", hence one might think that feeding more training data to the algorithm should help!
raw_data = sc.textFile("/datasets/airline/2000.csv") + \
sc.textFile("/datasets/airline/2001.csv")
print("number of rows before cleaning:", raw_data.count())
# extract the header
header = raw_data.first()
# replace invalid data with NULL and remove header
cleaned_data = (raw_data.filter(lambda x: x != header).map(lambda x: x.replace('NA', '')))
# create dataframe df
df = (sqlContext.createDataFrame(data=cleaned_data_to_columns, schema=airline_data_schema)
.select(['year', 'month', 'day_of_month', 'day_of_week', 'scheduled_departure_time',
'scheduled_arrival_time', 'arrival_delay', 'distance', 'src_airport', 'dest_airport', 'carrier'])
.cache())
stat_src = (df.groupBy('src_airport').agg(func.count('*').alias('count')).orderBy(desc('count')))
# extract the airport names from stat_src
src_airports = [item[0] for item in stat_src.collect()]
num_src_airports = len(src_airports)
src_airports_idx = range(0, num_src_airports)
map_src_airport_to_index = dict(zip(src_airports, src_airports_idx))
stat_dest = (df.groupBy('dest_airport').agg(func.count('*').alias('count')).orderBy(desc('count')))
dest_airports = [item[0] for item in stat_dest.collect()]
num_dest_airports = len(dest_airports)
num_dest_airports_idx = range(0, num_dest_airport)
map_dest_airports_to_index = dict(zip(dest_airports, num_dest_airports_idx))
stat_carrier = (df.groupBy('carrier').agg(func.count('*').alias('count')).orderBy(desc('count')))
carriers = [item[0] for item in stat_carrier.collect()]
num_carriers = len(carriers)
map_carriers_to_index = dict(zip(carriers, range(0, num_carrier)))
def most_common(groupBy):
return (df.groupBy(groupBy).agg(func.count('*').alias('count')).orderBy(desc('count'))).first()[0]
the_most_common_month = most_common('month')
the_most_common_day_of_month = most_common('day_of_month')
the_most_common_day_of_week = most_common('day_of_week')
the_most_common_s_departure_time = most_common('scheduled_departure_time')
the_most_common_s_arrival_time = most_common('scheduled_arrival_time')
mean_distance = df.agg(func.mean('distance')).first()[0]
mean_arrival_delay = df.agg(func.mean('arrival_delay')).first()[0]
# Code ommited. We just rerun all the NB with the csv from 2001 and collected this image
You might think that the flight delays do not only depend on the source airport, destination airport, departure time, etc... as we assumed. They also depend on other features such as the weather, the origin country, the destination city,... To improve the prediction quality, we should consider these features too.
There are some other datasets that related to this use case:
# Trying new features!
def is_holiday(day, month):
if month == 12 and day >= 20 or month == 1 and day <= 10: return 1 #Christmas holiday
if month >= 5 and day > 20 and month <= 8 and day <= 10: return 1 #Summer holiday
return 0
def is_winter(day, month):
if month == 12 and day >= 15 or month <= 2 and day <= 17: return 1
return 0
airportTrafficPD = pd.DataFrame(stat_airport_traffic.collect(), columns=['airport', 'n_flights'])
mean_n_flights = airportTrafficPD.n_flights.mean()
map_nflights_to_airport = dict(zip(airportTrafficPD.airport, airportTrafficPD.n_flights))
# Recreate the data with the new features
data_extended = cleaned_data\
.map(lambda line: line.split(','))\
.map(lambda values:
LabeledPoint(
int(values[14]) if is_valid(values[14]) else mean_arrival_delay, # arrival delay
[
int(values[0]), # year
int(values[1]) if is_valid(values[1]) else the_most_common_month, # month
int(values[2]) if is_valid(values[2]) else the_most_common_day_of_month, # day of month
int(values[3]) if is_valid(values[3]) else the_most_common_day_of_week, # day of week
int(values[5]) if is_valid(values[5]) else the_most_common_s_departure_time, # scheduled departure time
int(values[7]) if is_valid(values[7]) else the_most_common_s_arrival_time, # scheduled arrival time
# if the value is valid, map it to the corresponding index
# otherwise, use the most common value
map_carriers_to_index[values[8]] if is_valid(values[8]) \
else map_carriers_to_index[carriers[0]], # carrier
map_src_airport_to_index[values[16]] if is_valid(values[16]) \
else map_src_airport_to_index[src_airports[0]], # src_airport
map_dest_airports_to_index[values[17]] if is_valid(values[17]) \
else map_dest_airports_to_index[dest_airports[0]], # destination_airport
int(values[18]) if is_valid(values[18]) else mean_distance, # distance
1 if is_valid(values[3]) and int(values[3]) >= 6 else 0, # is_weekend
#--------------------------- NEW FEATURES ----------------------------
is_holiday(int(values[2]), int(values[1])), # is holiday
is_winter(int(values[2]), int(values[1])), # is winter
int(map_nflights_to_airport[values[16]]) if values[16] in map_nflights_to_airport\
else int(mean_n_flights),
#---------------------------------------------------------------------
]
)
)
# Split the data into training and test sets (30% held out for testing)
(trainingData_ext, testData_ext) = data_extended.randomSplit([0.7, 0.3])
trainingData_ext = trainingData_ext.cache()
testData_ext = testData_ext.cache()
trainingData_ext.take(10)
# Model with added implicit features
model_extended = DecisionTree.trainRegressor(trainingData_ext,
categoricalFeaturesInfo=categorical_info,
impurity='variance', maxDepth=12, maxBins=255)
print('Model Number of Nodes: ', model_extended.numNodes())
print('Model Depth: ', model_extended.depth())
# Evaluate model on test instances and compute test error
predictions_ext = model_extended.predict(testData_ext.map(lambda x: x.features))
labelsAndPredictions_ext = testData_ext.map(lambda lp: lp.label).zip(predictions_ext)
testMSE_ext = labelsAndPredictions_ext.map(
lambda p: np.power(p[0] - p[1], 2)).mean()
print('Test Mean Squared Error = ' + str(testMSE_ext))
print('Test RMSE = ' + str(np.sqrt(testMSE_ext)))
| Model | Nº Nodes | Test MSE | Test RMSE |
|---|---|---|---|
| Original | 8045 |<PHONE_NUMBER>84 | 22.1687978538 |
| Extended (Original + isWinter + isHoliday) | 8067 |<PHONE_NUMBER>5 | 22.1619184943 |
| Extended2 (Extended + airport_nFlights) | 8073 |<PHONE_NUMBER>93 | 22.0950921359 |
The important thing is to not stop questioning.
Curiosity has its own reason for existence. (Albert Einstein) In this nNotebook, we've had an overview about Decision Trees, Random Forests and how to use them. We also insisted on a simple methodology to adopt when dealing with a Data Science problem. The main take home messages should be:
Feature selection is a difficult, delicate and important task. In this project, the student was heavily guided. However, we invite to exercise with additional features, for example external ones related to weather conditions.
Parameter tuning requires a deep understanding of the algorithm used to build a statistical model. In general, to reduce computational cost, several techniques introduce parameters that, if tuned properly, can lead to tremendous time savings.
Flight delays project: http://ddowey.github.io/cs109-Final-Project/FlightDelay-ProjectFinal.html


SKPP developed a new abstraction and related requirements for least privilege in separation kernels, and a mean by which a Target of Evaluation(TOE) could ensure the adequacy of its trusted subjects(with respect to the basic abstract policy). It developed the configuration vector abstraction by which the basic abstract policy and other requirements for configuration could be provided to the TOE. It developed specifications for a configuration tool to be used by security administrators to prepare configuration vector for submission to the TOE. And finally, the authoring team developed concepts and requirements for dynamic, runtime changes to the security configuration of the TOE.
A separation kernel is an executive that serves a simple purpose: it controls all of the physical resources of a system(i.e., hardware, firmware, software); it constructs different types of resources abstractions from the physical resources, while leaving other physical resources in their original form; it exports a subset of these resources; it allocates the exported resources to partitions; and it controls information flow between the partitions and between exported resources. The exported resources, partitions, and allowed flows are specified in a configuration vector input to the separation kernel during its initialization.
There are three kinds of flows: partition-to-partition(P2P) flow and subject-to-resource(S2R) flow.
The SKPP was written to enforce a compound security policy, or PIFP, with requirements at the gross partition level as well as at the granularity of individual subjects and resources.
SKPP uses compound security policy to bring two benefits:
A TOE implementation can encode the compound polices however it likes for efficiency, and cache the related decisions, as long as these encodings and caches are performed by the TSF or its initialization routines, In fact, with suitable hardware support, runtime software checks may not be needed at all, e.g., if all of the flows of each runtime application are verified during initialization to conform to the policy, and those verifications are cached in the form of hardware segment descriptors. The kernel could then pass the descriptors to application in the form of application start-up date, and no further software checks would be required.
The original security policy for the SKPP was fairly simple, but strict: for a flow be ALLOWED, it must be permitted by both the P2P policy rules and the S2R policy rules.
This policy was revised to accommodate vendors who wished to bifurcate the compound policy, allowing either or both policies to be active in a given system.
Based on a desire for the SKPP S2R policy to be consistent with DAC policies in other systems, the SKPP management team decided that the S2R policy should be structured with a set of ordered rules by which the priority of authorization would be determined. This policy includes notions of explicit denial of access, and "don't care" mode of access, such that each mode(read, write) in the policy matrix would have one of three values (allowed, denied, or null).
This policy allows an override of the P2P policy by including anything other null in the S2R entry.
The final policy simplified the ordered set of rules, although it is still more complicated and less restrictive than the original.
In the S2R policy the S2R rules override the P2P rules everywhere except where there is a null entry in the S2R rule set. Note that the S2R policy is defined to reference the P2P rules, regardless of whether the P2P policy itself is active.
The original policy should be implemented as the default in the SKPP-based system. Since the S2R matrix cannot be ignored in the original policy, the set of secure reachable states(or the set of access allowed) is a subset of those allowed by the final policy: so the original policy can be considered to be at least as secure as the final policy. Of course, having implemented the original policy as a default, a vendor could also choose to allow the final policy to be selected as a configuration-vector option.
Another significant concern with separation kernels is the identification of trusted subjects. Information stakeholders need assurance that all trusted subjects in deployed systems are suitable trustworthy, and that they are confined with respect to the principle of least policy. To support the analysis of trusted subjects, the SKPP requires the


Test and familiarize yourself with the sensors and signal
processing needed for your robot to detect acoustic and
optical signals and interface with the Arduino Uno. The
Acoustic sub-team will work on the microphone circuit, which
will detect a 660 Hz whistle blow, signifying the start of
the maze mapping competition. The Optical sub-team will work
on the IR sensor circuit, which will detect nearby robots
emitting IR at 6.08 kHz and distinguish them from decoys
emitting at 18 kHz.
Prelab
Review documentation for the analog-to-digital converter on the ATmega328 (Arduino microcontroller)
Find a website to generate the 660 Hz tone for testing
Design simple analog amplifying and filtering circuits
Sub-Teams
To begin, we split into two groups of two. Each group progressed
through the lab individually, as described below.
Acoustic Team = Kenneth and Tyler
Optical Team = Brian and Eric
FFT Analysis
Both the Acoustic and Optical parts of this lab used Fast Fourier
Transforms (FFT) to determine the frequency components of the sampled
input signal from the sensors. For pure sinusoids, this is not so
difficult, as one can simply read off the period in the time domain.
However, for more complicated signals, it is not necessarily clear what
frequencies are present, and certainly not clear if a given frequency
is present. To make this process easier, a signal can be inspected in
the frequency domain.
In the frequency domain, we can observe a signals “spectrum”.
A spectrum is an interpretation of a signal which assigns a magnitude
(an importance) to a given frequency. The standard way to compute this
is with a Discrete Fourier Transform (DFT):
The DFT equation
The DFT computes a change of basis, assigning the weight X(k) to a
given radial frequency (2*pi*k/N) by computing the inner product of the
original signal with the complex exponential at that frequency.
Note that to compute one X(k) value, we need N multiplications, and
N - 1 additions. To get useful information about a signal of size N, you
generally need N values of k. This makes the DFT computation an O(n^2)
operation, which can get very slow for large values of N.
To speed this process up, the Fast Fourier Transform (FFT) is used.
The FFT is an algorithm for computing the DFT efficiently—the best
speed up is achieved when N is a power of 2. The algorithm uses a
“divide-and-conquer” method to compute the DFT. It first isolates the
even and odd terms of the original DFT sum. Then it evaluates these
sums as two independent FFTs of length N/2.
The FFT divide-and-conquer approach
FFT Butterfly diagram
A key to this process is that the odd sum gets transformed into a N/2
FFT by pulling out an overall constant phase. This process of splitting
up into two FFTs can continue down all the way until you have reached a
series of FFTs, each acting on only two values. This splitting action
occurs log2(N) - 1 times. At each stage, there are N multiplications and
additions, so our computational complexity has decreased to
O(N * log2(N)).
The FFT algorithm was implemented using Open Music Labs Arduino FFT
Library. By default, the library takes in an analog signal from Arduino
pin A0, samples it at 256 equally spaced intervals, and returns an array
with 128 values, which represent the magnitude of the frequency content
for frequenices up to half of the samping frequency. Since the FFT of a real signal is
symmetric over zero, only half of the outputs are unique, and thus 128
bins are returned (not 256).
The FFT code can be modified to alter the performance of the ADC
(information from the ADC section of the ATmega328 datasheet [page 305]).
The important ADC characteristics are:
10-bit resolution
ADC clock is dependent on chosen prescalar, which is set based upon last 3 bits of ADCSRA register
The default FFT code sets the ADC prescaler = 32
Prescalar = 128 -> maps to clock of 125 kHz
Prescalar = 64 -> maps to clock of 250 kHz
Prescalar = 32 -> maps to clock of 1 MHz
Prescalar = 16 -> maps to clock of 2 MHz
Prescalar = 8 -> maps to clock of 4 MHz
Prescalar = 4 -> maps to clock of 8 MHz
NOTE - at frequencies higher than 250 kHz the 10-bit accuracy degrades
A normal conversion takes 13 ADC clock cycles
Sampling frequency = clock rate divided by number of cycles
Fs = clock rate/cycles
Fs = 125000/13 = 9615 Hz
To test and visualize the data, we used the example code "fft_adc_serial,"
which prints the FFT magnitude data to the serial monitor. We then copied
the data into Excel, and wrote a simple MATLAB script to read in and plot
the data.
Acoustic Team
GOAL: use an Arduino and the FFT library to detect a 660 Hz whistle blow
First, we built the recommended circuit from the datasheet to connect the microphone to the Arduino
1 × 1 μF Capacitor
1 × 3.3 kΩ resistor
Circuit from datasheet
Circuit on breadboard
Results from oscilloscope
We used the previously mentioned website to generate a 660 Hz tone
Results shown by the oscilloscope indicate a response, although the waveform amplitude is relatively small
DC bias ~3mV
Peak-to-peak ~40mV
That is with the source right next to the microphone
Clearly, we need to amplify the output
Testing the FFT Library
As mentioned earlier, we use the Open Labs Arduino FFT Library for calculating the FFT
To increase the accuracy of our readings, we altered the ADC prescaler value
Original: prescaler = 32 -> maps to clock of 1 MHz
Modified: prescaler = 128 -> maps to clock of 125 kHz
The original code had:
First set line: ADCSRA = 0xe5
Second set line: ADCSRA = 0xf5
Our modified code is:
void setup() {
Serial.begin(115200); // use the serial port
TIMSK0 = 0; // turn off timer0 for lower jitter - delay() and millis() killed
ADCSRA = 0xe7; // set the adc to free running mode, set prescaler=128 (default e5=32)
ADMUX = 0x40; // use adc0
DIDR0 = 0x01; // turn off the digital input for adc0
}
void loop() {
while(1) { // reduces jitter
cli(); // UDRE interrupt slows this way down on arduino1.0
for (int i = 0 ; i < 512 ; i += 2) { // save 256 samples
while(!(ADCSRA & 0x10)); // wait for adc to be ready
ADCSRA = 0xf7; // restart adc --> set prescaler=f7=128 (default f5=32)
Next, we calculated which bin we would expect to find the 660 Hz signal in
ADC clock = 125 kHz
ADC sampling = 13 clock cycles per read
ADC sampling = 125 kHz/13 = 9,615.38 Hz
Number of Samples = 256
Bin Size = 9,615.38/256 = 37.56 Hz/bin
Bin for 660 Hz = 660/37.56 = 17.57 --> Bin 18
Experimentally, we found that 660 Hz is right on the border between Bins 18 and 19
Thus, in our integrated detection code we check for a large magnitude in BOTH of these bins
Building a Microphone Amplifier Circuit
For our circuit, we added an active low pass filter with a gain of -10 dB
First, we added two large (680 kΩ) resistors to bias the op amp's inverting input to mid-rail (2.5V)
Next, we biased the op amp's non-inverting input to mid-rail using two 10 kΩ resistors, so that we only amplified the differential changes due to the microphone
Lastly, we added a parallel RC tank which acts as a low pass filter with a cutoff frequency of around 742 Hz
Because there was no 550 pF capacitor in lab, we added a parallel combination of a 220 pF and a 330 pF capacitor to give us the desired cutoff frequency
Circuit diagram
Breadboard circuit
Breadboard circuit
Testing the Microphone - Amplifier Circuit
Circuit testing
To make sure the microphone amplifier circuit was working properly, we ran it through the following tests:
660 Hz tone
680 Hz tone
640 Hz tone
660+760 Hz tone
660+560 Hz tone
For each test, we played one or more distinct tones from device(s) 5 cm away from the microphone
With the audio playing, we ran the FFT code and recorded the data
Once we finished, we graphed the results
The data shows that we are successfully able to differentiate between various tones
Microphone amplifier circuit FFT test results
Integrating the Microphone Functionality into Software
We created a function "detect_660hz"
Runs FFT function with prescaler = 128
Checks bins 18 AND 19
If magnitudes are both greater than a defined threshold -> returns TRUE
/**
* [detect_660hz]
* OUTPUT = returns TRUE if a 660hz signal is detected by the Arduino,
* & FALSE otherwise
* Uses THRESHOLD to determine whether the tone is playing or not
*/
bool detect_660hz(){
byte * fft_log_out = get_fft_bins();
if (fft_log_out[17] > THRESHOLD && fft_log_out[18] > THRESHOLD) //bins 18 & 19, but 0 based so subtract 1
return true;
else
return false;
}
To test "detect_660hz", we wrote a simple sketch
Every time through the loop, "detect_660hz" is run
To prevent a stray audio source from setting off our robot, a counter is used
If and only if "detect_660hz" returns true 6 times in a row --> then the robot will start
We chose to implement an active band-pass filter with amplification in order to filter out DC and low frequency signals while also filtering out the higher frequency decoy signals
Received IR signals in the pass band (from other robots’ IR hats) are amplified for better ADC quantization range, as well as to detect them from further distances
Cut-off frequencies were intended to be 5 kHz and 7 kHz
Our capacitor and resistor values were:
R1 = 10 Ω
C1 = 2.3 μF
R2 = 3.2 kΩ
C2 = 10 nF
This resulted in cut-off frequencies of 4.97 kHz and 6.92 kHz
The gain of the filter in the pass band was intended to be R2/R1 = 320
Actual results:
Peak Gain: 75 or 19 dB
Bode Plot
Shows bandpass filter with 3 dB bandwidth 700 Hz to 10 kHz
18kHz signal is half the power of the 6 kHz signal after active filter
Harmonics of the 18 kHz signal that would alias into the 6 kHz bins come out of the filter with 1/10 the magnitude of a 6 kHz signal
Band-pass filter schematic (from Professor Shealy, ECE2100 course notes)
IR filter bode plot
FFT Analysis
The default example code sets the ADC in free running mode and polls for each ADC conversion
ADC prescaler = 32
NOTE - this ADC setup is different from the one used by the Acoustic Team
Arduino clock frequency = 16 MHz
Thus, sampling frequency = 38.4 kHz
This is quick enough for the purpose of detecting IR signals, since we are sampling above the Nyquist rate, and the highest frequency IR signal we will see is 18 kHz
However, we may still see harmonics of these signals that are aliased to lower frequencies, since a sinusoidal input to a diode will cause harmonics
Given our sampling rate and decoy frequency, we expect these aliased signals to appear at frequencies that are filtered out by our band pass filter
The FFT code outputs 128 bins
Frequencies of interest are in bins 41 and 42
This corresponds to 6.01 kHz and 6.16 kHz
Results
Silence
Spectrum is pretty sparse
There are some small peaks around 8 kHz and 13 kHz
IR Hat
There is a large peak at 6160 Hz
Bins 41 and 42 correspond to the 6 kHz IR hat signal
When the peaks in these bins are above a threshold, then the IR hat signal is detected
IR Decoy
There is a peak corresponding to 18 kHz on the spectrum
No frequencies are aliased to the bins around 6 kHz
Thus, we can ignore the 18 kHz peak and only observe bins 41 and 42 for IR hat detection
IR Hat + IR Decoy
There are peaks at 6 kHz and 18 kHz
Ignore the 18 kHz bins
Observe bins 41 and 42 for IR hat detection
Concurrent Testing
Our concurrent testing software works as follows
Idle until 660 Hz tone is detected
When microphone circuit detects 660 Hz, turn on YELLOW LED and advance in code


Yellowstone National Park
Native Wildlife
Find Animals all over the park!
Here are just some of the amazing wildlife you may spot in Yellowstone
- Grizzly Bear
- Bison
- Black Bear
- Bobcat
- Cougar
- Coyote
- Gray Wolf
- Elk
- Bighorn Sheep
- Moose
- Mule Deer
- Mountain Goat
- Learn more about the Wildlife here


About Tove
Best known as the creator of the Moomins, Tove Jansson (1914 – 2001) was a Swedish – speaking Finnish cartoonist, painter, illustrator, graphic artist and writer. She was born in Helsinki in a Swedish family of artists – her mother a graphic designer and illustrator, and her father a sculptor. She studied painting both in Finland and abroad while working as an illustrator and drawing cartoons for satirical and political magazines before leaning into her love of nature, animals and changing seasons by charting the adventures of the tightly – knit Moomin family in series of 8 books.
Moomins
Moomins are a peace – loving philosophical family consisting of Moominmamma, Moominpappa and their son the Moomintroll + their many friends. Conceived in the 1940’s as a series of children’s books the Moomins are now a global phenomenon making their creator Tove Jansson one of the most succesful authors of children’s books of all time. Moomin books are very influenced by Tove’s own life, it doesn't take a genius to trace Mamma and Pappa Jansson in Moominmamma and Moominpappa or Tove’s partner Tuulikki Pietilä as Too – ticky.
Work outside Moomins
Although Tove is mainly known for her Moomin books she is also a polymathic artist, as brilliant with oils and acrylics as a black pen and a comic strip. Tove struggled for many years with her desire to be taken seriously as a painter rather than as an illustrator. Her best known paintings are the large frescoes made for Helsinki City Hall in 1947, the frescoes depict party scenes in countryside and city, in one fresco Tove has even painted herself sitting next to a little Moomintroll watching her lover dance with someone else.
My favorite Moomin books:
- The Moomins and The Great Flood (1945)
- Finn Family Moomintroll (1954)
- Moominland Midwinter (1957)
Tove's adult novels:
- The Summer Book (1972)
- Sun City (1974)
- The True Deceiver (1982)
- The Field of Stones (1984)
- Fair Play (1989)


0% complete
Address and Identity verification.
Following question will determine your residency
Option to upload forms go here somewhere
Vision Screening
Exam scheduling
Schedule the Road skills test or Knowledge test (writter exam)
Click the continue button to show form progression.
Review and complete your application


<wsk-draggable class="language" ng-repeat="language in languages"
data-to-drag="language"
drop-zone="language.type">\language.name\
</wsk-draggable>
<wsk-draggable class="language" ng-repeat="language in languages"
data-to-drag="language"
drop-zone="language.type">\language.name\
</wsk-draggable>
<wsk-draggable class="language" ng-repeat="language in languages"
data-to-drag="language"
drop-zone="language.type">\language.name\
</wsk-draggable>


{% extends "base.html" %} {% block h1_title %}External authentication cancelled{% endblock %} {% block title %}External authentication cancelled{% endblock %} {% block content %}
Unfortunately, external authentication didn't succeed.
If this was unexpected, you can try again. If you need any assistance, please email {{ SUPPORT_EMAIL_NAME }} for support.
{% endblock %}