@* * Copyright 2023 HM Revenue & Customs * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. *@ @import config._ @import views.html.helpers._ @import views.html.helper.form @import uk.gov.hmrc.govukfrontend.views.html.components._ @import views.ViewUtils.titleBuilder @this(bcMain: bcMain, formHelper: FormWithCSRF, govukDetails : GovukDetails, govukBackLink : GovukBackLink, govukButton : GovukButton) @(service: String, isAgent: Boolean, businessDetails: models.ReviewDetails, backLink: Option[String])(implicit request: Request[_], messages: Messages, appConfig: ApplicationConfig) @headingText = @{ isAgent match { case true => messages("business-review.agent.header") case false => messages("business-review.user.header") } } @headingSecondaryText = @{ isAgent match { case true => messages("bc.business-registration.text.agent", service.toUpperCase) case false => messages("bc.business-registration.text.client", service.toUpperCase) } } @businessDetailsMatchTextQuestion = @{ businessDetails.directMatch match { case true => messages("business-review.invalid-business.header") case false => messages("business-review.invalid-address.header") } } @businessDetailsMatchText = @{ businessDetails.directMatch match { case true => messages("business-review.invalid-business.text") case false => messages("business-review.invalid-address.text") } } @contentDetails = {
@businessDetailsMatchText
@messages("business-review.registered-with-companies-house.text") @messages("business-review.registered-with-companies-house.text.link").
@messages("business-review.sole-trader.text") @messages("business-review.sole-trader.text.link").
@businessDetails.businessName
@businessDetails.businessAddress.line_1
@businessDetails.businessAddress.line_2
@businessDetails.businessAddress.line_3.map { line_3 => @line_3
}
@businessDetails.businessAddress.line_4.map { line_4 => @line_4
}
@businessDetails.businessAddress.postcode.map { postcode => @postcode
}
@appConfig.getSelectedCountry(businessDetails.businessAddress.country)


algorithms
Contents
2.6. algorithms#
Some notes on algorithms following the book Introduction to algorithms and based on UVA’s course.
2.6.1. asymptotics#
Big-O
big-oh: O(g): functions that grow no faster than g - upper bound, runs in time less than g
\(f(n) \leq c\cdot g(n)\) for some c, large n
set of functions s.t. there exists c,k>0, 0 ≤ f(n) ≤ c*g(n), for all n > k
big-theta: Θ(g): functions that grow at the same rate as g
big-oh(g) and big-theta(g) - asymptotic tight bound
big-omega: Ω(g): functions that grow at least as fast as g
f(n)≥c*g(n) for some c, large n
Example: f = 57n+3
O(n^2) - or anything bigger
Θ(n)
Ω(n^.5) - or anything smaller
input must be positive
We always analyze the worst case run-time
little-omega: omega(g) - functions that grow faster than g
little-o: o(g) - functions that grow slower than g
we write f(n) ∈ O(g(n)), not f(n) = O(g(n))
They are all reflexive and transitive, but only Θ is symmetric.
Θ defines an equivalence relation.
add 2 functions, growth rate will be \(O(max(g_1(n)+g_2(n))\) (same for sequential code)
recurrence thm:\( f(n) = O(n^c) => T(n) = 0(n^c)\)
\(T(n) = a*T(n/b) + f(n)\)
\(c = log_b(a)\)
over bounded number of elements, almost everything is constant time
2.6.2. recursion#
moving down/right on an NxN grid - each path has length (N-1)+(N-1)
we must move right N-1 times
ans = (N-1+N-1 choose N-1)
for recursion, if a list is declared outside static recursive method, it shouldn’t be static
generate permutations - recursive, add char at each spot
think hard about the base case before starting
look for lengths that you know
look for symmetry
n-queens - one array of length n, go row by row
2.6.3. dynamic programming#
//returns max value for knapsack of capacity W, weights wt, vals val
int knapSack(int W, int wt[], int val[])
int n = wt.length;
int K[n+1][W+1];
//build table K[][] in bottom up manner
for (int i = 0; i <= n; i++)
for (int w = 0; w <= W; w++)
if $(i==0 \vert \vert w==0)$ // base case
K[i][w] = 0;
else if (wt[i-1] <= w) //max of including weight, not including
K[i][w] = max(val[i-1] + K[i-1][w-wt[i-1]], K[i-1][w]);
else //weight too large
K[i][w] = K[i-1][w];
return K[n][W];
2.6.4. min-cut / max-cut#
2.6.5. hungarian#
assign N things to N targets, each with an associated cost
2.6.6. max-flow#
A list of pipes is given, with different flow-capacities. These pipes are connected at their endpoints. What is the maximum amount of water that you can route from a given starting point to a given ending point?
2.6.7. sorting#
you can assume w.l.o.g. all input numbers are unique
sorting requires Ω(nlog n) (proof w/ tree)
considerations: worst case, average, in practice, input distribution, stability (order coming in is preserved for things with same keys), in-situ (in-place), stack depth, having to read/write to disk (disk is much slower), parallelizable, online (more data coming in)
adaptive - changes its behavior based on input (ex. bubble sort will stop)
<IP_ADDRESS>. comparised-based#
<IP_ADDRESS>.1. bubble sort#
keep swapping adjacent pairs
for i=1:n-1
if a[i+1]<a[i]
swap(a,i,i+1)
have a flag that tells if you did no swaps - done
number of passes ~ how far elements are from their final positions
O(n^2)
<IP_ADDRESS>.2. odd-even sort#
swap even pairs
then swap odd pairs
parallelizable
<IP_ADDRESS>.3. selection sort#
move largest to current position for i=1:n-1 for j=1:n x = max(x,a[j]) jmax = j swap(a,i,j)
0(n^2)
<IP_ADDRESS>.4. insertion sort#
insert each item into lists for i=2:n insert a[i] into a[1..(i-1)] shift
O(n^2), O(nk) where k is max dist from final position
best when alsmost sorted
<IP_ADDRESS>.5. heap sort#
insert everything into heap
kepp remove max
can do in place by storing everything in array
can use any height-balanced tree instead of heap
traverse tree to get order
ex. B-tree: multi-rotations occur infrequently, average O(log n) height
0(n log n)
<IP_ADDRESS>.6. smooth sort#
adaptive heapsort
collection of heaps (each one is a factor larger than the one before)
can add and remove in essentially constant time if data is in order
<IP_ADDRESS>.7. merge sort#
split into smaller arrays, sort, merge
T(n) = 2T(n/2) + n = 0(n log n)
stable, parallelizable (if parallel, not in place)
<IP_ADDRESS>.8. quicksort#
split on pivot, put smaller elements on left, larger on right
O(n log n) average, O(n^2) worst
O(log n) space
<IP_ADDRESS>.9. shell sort#
generalize insertion sort
insertion-sort all items i apart where i starts big and then becomes small
sorted after last pass (i=1)
O(n^2), O(n^(3/2), … unsure what complexity is
no matter what must be more than n log n
not used much in practice
<IP_ADDRESS>. not comparison-based#
<IP_ADDRESS>.1. counting sort#
use values as array indices in new sort
keep count of number of times at each index
for specialized data only, need small values
0(n) time, 0(k) space
<IP_ADDRESS>.2. bucket sort#
spread data into buckets based on value
sort the buckets
O(n+k) time
buckets could be trees
<IP_ADDRESS>.3. radix sort#
sort each digit in turn
stable sort on each digit
like bucket sort d times
0(d*n time), 0(k+n) space
<IP_ADDRESS>.4. meta sort#
like quicksort, but 0(nlogn) worst case
run quicksort, mergesort in parallel
stop when one stops
there is an overhead but doesn’t affect big-oh analysis
ave, worst-cast = O(n log n)
<IP_ADDRESS>. sorting overview#
in exceptional cases insertion-sort or radix-sort are much better than the generic QuickSort / MergeSort / HeapSort answers.
merge a and b sorted - start from the back
2.6.8. searching#
binary sort can’t do better than linear if there are duplicates
if data is too large, we need to do external sort (sort parts of it and write them back to file)
write binary search recursively
use low<= val and high >=val so you get correct bounds
binary search with empty strings - make sure that there is an element at the end of it
“a”.compareTo(“b”) is -1
we always round up for these
finding minimum is Ω(n)
pf: assume an element was ignored, that element could have been minimum
simple algorithm - keep track of best so far
thm: n/2 comparisons are necessary because each comparison involves 2 elements
thm: n-1 comparisons are necessary - need to keep track of knowledge gained
every non-min element must win atleast once (move from unkown to known)
find min and max
naive solution has 2n-2 comparison
pairwise compare all elements, array of maxes, array of mins = n/2 comparisons
check min array, max array = 2* (n/2-1)
3n/2-2 comparisons are sufficient (and necessary)
pf: 4 categories (not tested, only won, only lost, both)
not tested-> w or l =n/2 comparisons
w or l -> both = n/2-1
therefore 3n/2-2 comparisons necessary
find max and next-to-max
thm: n-2 + log(n) comparisons are sufficient
consider elimination tournament, pairwise compare elements repeatedly
2nd best must have played best at some point - look for it in log(n)
selection - find ith largest integer
repeatedly finding median finds ith largest
finding median linear yields ith largest linear
T(n) = T(n/2) + M(n) where M(n) is time to find median
quickselect - partition around pivot and recur
average time linear, worst case O(n^2)
median in linear time - quickly eliminate a constant fraction and repeat
partition into n/5 groups of 5
sort each group high to low
find median of each group
compute median of medians recursively
move groups with larger medians to right
move groups with smaller medians to left
now we know 3/10 of elements larger than median of medians
3/10 of elements smaller than median of medians
partition all elements around median of medians
recur like quickselect
guarantees each partition contains at most 7n/10 elements
T(n) = T(n/5) + T(7n/10) + O(n) -> f(x+y)≥f(x)+f(y)
T(n) ≤ T(9n/10) + O(n) -> this had to be less than T(n)
2.6.9. computational geometry#
range queries
input = n points (vectors) with preprocessing
output - number of points within any query rectangle
1D
range query is a pair of binary searches
O(log n) time per query
O(n) space, O(n log n) preprocessing time
2D
subtract out rectangles you don’t want
add back things you double subtracted
we want rectangles anchored at origin
nD
make regions by making a grid that includes all points
precompute southwest counts for all regions - different ways to do this - tradeoffs between space and time
O(log n) time per query (after precomputing) - binary search x,y
polygon-point intersection
polygon - a closed sequence of segments
simple polygon - has no intersections
thm (Jordan) - a simple polygon partitions the plane into 3 regions: interior, exterior, boundary
convex polygon - intersection of half-planes
polytope - higher-dimensional polygon
raycasting
intersections = odd - interior, even - exterior
check for tangent lines, intersecting corners
O(n) time per query, O(1) space and time
convex case
preprocessing
find an interior point p (pick a vertext or average the vertices)
partition into wedges (slicing through vertices) w.r.t. p
sort wedges by polar angle
query
find containing wedge (look up by angle)
test interior / exterior
check triangle - cast ray from p to point, see if it crosses edge
O(log n) time per query (we binary search the wedges)
O(n) space and O(n log n) preprocessing time
non-convex case
preprocessing
sort vertices by x
find vertical slices
partition into trapezoids (triangle is trapezoid)
sort slice trapezoids by y
query
find containing slice
find trapezoid in slice
report interior/ exterior
O(log n) time per query (two binary searches)
O(n^2) space and O(n^2) preprocessing time
convex hull
input: set of n points
output: smallest containing convex polygon
simple solution 1 - Jarvis’s march
simple solution 2 - Graham’s scan
mergehull
partition into two sets - computer MergeHull of each set
merge the two resulting CHS
pick point p with least x
form angle-monotone chains w.r.t p
merge chains into angle-sorted list
run Graham’s scan to form CH
T(n) = 2T(n/2) + n = 0(n log n)
generalizes to higher dimensions
parallelizes
quickhull (like quicksort)
find right and left-most points
partition points along this line
find points farthest from line - make quadrilateral
eliminate all internal points
recurse on 4 remaining regions
concatenate resulting CHs
O(n log n) expected time
O(n^2) worst-case time - ex. circle
generalizes to higher dim, parallelizes
lower bound - CH requires Ω(n log n) comparisons
pf - reduce sorting to convex hull
consider arbitrary set of x_i to be sorted
raise the x_i to the parabola (x_i,x_i^2) - could be any concave function
compute convex hull of the parabola - all connected and line on top
from convex hull we can get sorted x_i => convex hull did sorting so at least n log n comparisons
corollary - Graham’s scan is optimal
Chan’s convex hull algorithm
assume we know CH size m=h
partitoin points into n/m sets of m each
convex polygon diameter
Voronoi diagrams - input n points - takes O(nlogn) time to compute
problems that are solved
Voronoi cell - the set of points closer to any given point than all others form a convex polygon
generalizes to other metrics (not just Euclidean distance)
a Voronoi cell is unbounded if and only if it’s point is on the convex hull
corollary - convex hull can be computed in linear time
Voronoi diagram has at most 2n-5 vertices and 3n-6 edges
every nearest neighbor of a point defines an edge of the Voronoi diagram
corollary - all nearest neighbors can be computed from the Voronoi diagram in linear time
corollary - nearest neighbor search in O(log n) time using planar subdivision search (binary search in 2D)
connection points of neighboring Voronoi diagram cells form a triangulation (Delanuay triangulation)
a Delanuay triangulation maximizes the minimum angle over all triangulations - no long slivery triangles
Euclidean minimum spanning tree is a subset of the Delanuay triangulation (can be computed easily)
calculating Voronoi diagram
discrete case / bitmap - expand breadth-first waves from all points
time is O(bitmap size)
time is independent of #points
intersecting half planes
Voronoi cell of a point is intersection of all half-planes induced by the perpendicular bisectors w.r.t all other points
use intersection of convex polygons to intersect half-planes (nlogn time per cell)
can be computed in nlogn total time
idea divide and conquer
merging is complex
sweep line using parabolas


Pormodoro sounded quiet promising because I wanted to see what I could achieve without the distractions or interuptions from family members. So I decided to treat the next 30 mins like a experiment to find out if my mind would drift or lose focus engaged with the amount of reading and information put before us in the this sprint.
What I found interesting is one, I should always study on a Monday to Friday while the children are at school. Two that my mind didn't feel as inundated with code or all the stresses and I could quiet my mind and focus at the one task at hand. I felt a little more clearer and restful and not so stressed about completing this sprint. I knew that I would complete it.
Locking the bedroom door defintely worked to minimise some distration. While Reading through the various pieces of information I found the articles on the habits of successful the idea of starting small. Luckily, having started this cohort with 8 extra weeks up my sleeve this allowed me plenty of time to really focus and concentrate unhurriedly on the tasks before me. I have found personally steping back from study for a few days has helped me alot. Moving forward on the following Monday I quickly formulated my future approach on Phase zero.
I would defintely like to attempt meditation, reading through the information I also had never knew about the benefits. It taught me alot about myself my strengths and weaknesses, I like the idea having a mind of clarity in todays world and as a mother of three is easy to forget the importance of sanity, unplugging and taking time for yourself. I also liked the youtube video about forget big change, start with a tiny habit. Ten different ways to celebrate, the energy and the ideolgy of the presenter and the message he was trying to convey.


Supply chain analytics is the second product in our company’s arsenal. Here at our company we use various supply chain softwares to provide insights helping make better decisions that add surplus to the company.
We provide dashboards to our clients that lets them monitor many metrics at once. They can quickly check the health of the accounts or see correlations between different reports. Dashboards are easy to create, customize, share, interact with and they help in finding out the KPI’s, for better decisions.
The analytics provided will be fruitful for the customer in understanding risks, it helps in providing a better image of the future, so that planning can happen accurately, and can lead to the achievement of a lean supply chain and a greater Return on Investment.
To provide the best possible solutions for our customers we believe in staying up to date with the latest technological advances that are going to impact the way our world produces, processes, transports and trades.
Block chain is the future of secure and trusted Supply Chains.
At Cultivo, we have been persistently working on producing a highly secured, reliable and stable block chain that helps in creating an environment for the development of a transparent world.
We present to you a decentralized block chain that helps our clients update and track each intricate detail of their product data. The decentralized network facilitates a sense of trust amongst the stakeholders of a supply chain as well as the credibility of the product for end consumers.
The block chain has a high security rating, making alteration by any outsiders impossible. Such security helps in resolving the uncertainty related to third party dealers.


In the past, you would probably only really check if a website used https if you are purchasing something, or maybe if you were logging in. In the last few years, there has (rightly) been a large movement to better ensure privacy and security. Why not have every site and every page be encrypted? That’s a good question, and one Google has already answered for itself. Not only will Google start shaming websites that are not https, they will also give an SEO boost to those that are. If your customer’s peace of mind wasn’t enough, you now have more than enough incentive to make the switch.
I recently decided to do this for Explorable Places, for the reasons above and also in preparation of supporting Stripe and credit card payments. While it is simple to make this change, there is a few things worth learning (and documenting) and some snags you may run into.
What is SSL/HTTPS?
Otherwise know as SSL (secure socket layer), HTTPS is a protocol, in the same realm as regular ole HTTP or FTP, that is encrypted and was developed by Netscape. It listens on port 443 as opposed to port 80. You’ll need just a couple things to enable it and start using in on your site: You’ll need a certificate file and a private key file.
Without going into too much detail, the certificate is signed by an authority that the browser trusts (it only trusts known certificate authorities). I’ll get into signing in a bit though. In addition to being signed, the certificate also contains a public key that will be sent to the browsers. When you make a request to the server from your browser, it will encrypt that data with that public key. The data in that request can only be decrypted with the private key residing on the server.
So the server decrypts the request with its private key and does whatever it needs to with it. It’ll then return a response that it encrypts with said key again. This response in turn can only be decrypted with the public key. It’s actually quite nice how they work together.
Now it’s important to point out that the data being sent to the browser is encrypted, but that doesn’t mean it’s secure since all it takes to decrypt it is the public key, which anyone has access to. But all data sent to it, that is locked down. 🔒
Back to signing the certificate. You could create your own certificate and sign it yourself (as opposed to a known authority) and use that. Many people do this when developing locally or for apps within a private network. But to public facing apps, self-signed certificates can’t be trusted since nobody knows you and you could be up to no good.
So you’ll need to purchase one, which will run from very cheap with no paperwork (insures for very little, good for personal sites), to very expensive and lots of paperwork (insures for a lot). The latter requires you to prove your business is who they say they are, and can allow you to include that name next to the URL in the browser. This also looks very nice, so it may be worth it to you if security is particularly a big deal.
I went with just the plain old domain verification option, which can be issued immediately with no paperwork, I just have to prove that I own the domain. This can be done with an email that is associated with the domain, easy peasy.
How do I get one
Once you’ve decided on what you want and purchased it, you’ll need to request the certificate from the issuing company. To do this, you’ll need a certificate signing request (CSR) file. This can be created with many tools available around the internet, most of which cost money. Or you could do it on the command line with openSSL:
This will create a token with 2048 bit encryption, and output your CSR and your private KEY file which you will need for later. When you enter this command, it will prompt you through a set of questions, the only one worth noting is the name
field, which should be your domain name (if you are doing a domain verification certificate like me).
When finished, submit this CSR to the issuing company and answer whatever other questions they have (it’s very easy), and you will get issued your certificate (CRT) file within a few minutes. You should now have your certificate .crt
and private key .key
files.
Intermediate Certificate
Depending on the type of certificate you purchase, you may be required to include an intermediate certificate. An intermediate certificate lies between the domain certificate (yours) and the root certificate of the issuing authority. This is done to ensure its authenticity and maintain the “Chain of Trust”™ because it’s insecure to include the root certificate itself. To include this, you will need to chain them together. Chaining is nothing more than file concatenation.
You will be able to just download an intermediate certificate from the issuer as there is no customization or CSR needed for them, everyone gets the same one. Then either in a text editor (with no formatting) or through the command line, concatenate the files with your domain certificate first and intermediate second.
 This includes uploading the chained certificate (or regular domain certificate if you did not need to chain) and the KEY file from the beginning. You can put these anywhere. I have a linux server and put the certificate in /etc/ssl/certs
and the key in /etc/ssl/private
.
Then in your nginx configuration file (or where ever your server config is), you will need to tell it where those files are. You will also need to tell the server to listen on port 443. For nginx and unicorn this will look something like:
If you use rails, you’ll likely use force_ssl
in your controllers. I use it in the application controller to force https for all routes. It relies on the X-Forwarded-Proto
header to work and is why rails needs it.
Then just restart your server, for nginx: sudo /etc/init.d/nginx restart
Now when you visit your site with https://
, it should work and you should see the little green lock. If not, you may want to troubleshoot with the issuing company to get it resolved. Also, you will need to make sure ALL assets you serve from the page (js, css, images, fonts, etc) are served via SSL as well, or the browser will warn the user, or possibly not work at all.
Welcome to the secure side! 🔐
Now I just need to wait for Github to allow for https of github pages with a custom domain and I can revisit this article.


Superstar Kim Kardashian leads the celebrity fashion scene in luxury faux fur
Kim Kardashian and other well-known faces are re-cycling their wardrobes and bringing back past fashion including tweed, fur and mum jeans
by Rachael Bletchly, https://www.facebook.com/dailymirror/Elasticated waists are back in fashion... and millions of women of a certain age are breathing a sigh of relief.
And they are not the only former style duds to reappear as the height of chic. Mum jeans, tweed suits, jelly shoes, big knickers and headscarves have all had their moment.
Stylist Martine Alexander said: “Even when fashion comes full circle, there will always be a modern twist.”
So it’s time to have a rummage in your wardrobe. Here’s what else is hot again...
It comes after their half-sister sister Kylie Jenner bagged a huge deal with Coty.
Khloe Kardashian celebrated herself and sisters Kim and Kourtney Kardashian as the "OG trio."
The sassy nickname was unveiled on Instagram on Tuesday night in light of younger half-sibling Kylie Jenner , 22, inflating her personal bank balance to around the tune of $1.6 million - following a huge business deal.
Not one of Kylie's older sisters made reference to the huge achievement but Khloe, 35, used to opportunity to remind fans that she Kim, 39, and Kourtney, 40, were the original Kardashians.
She illustrated the caption with a photo of the threesome posing in pastel coloured bodysuits from a recent photoshoot for their perfume collaboaration.
On her stories Khloe did post footage of unboxing Kylie's Christmas make-up collection as well as Kim's cosmetic kit that was a partnership with her long-time make-up artist Mario Dedivanovic.
Kylie is selling off a majority 51 per cent share in her company Kylie Cosmetics to beauty giant Coty in a deal worth around $600 million.
The daughter of Caitlyn Jenner's deal with Coty was announced on Monday and means she'll be able reach more customers across the world thanks to the company's international infrastructure.


Tom
sent you a file.You are in the place to get the file.
{{original_filename}}
{{file_size}}
▼ Download!


Seattle Seasons
Greeting will display here.
Date will display here.
Click a season to see the weather.
Spring
Summer
Fall
Winter


AFC Champions League & AFC Cup 2020: Qualifiers opponents of Buriram United, Bengaluru FC, Chennai City predicted
by HarigovindQuite recently, the Asian Football Confederation (AFC) announced which countries and clubs have been deemed eligible for the 2020 AFC Champions League and AFC Cup, and Twitter channel FootyRankings have predicted the qualifiers opponents for various teams.
As per FootyRankings, for the AFC Champions League, the various footballing associations are split into two regions:
- West Region which consists of the associations from the West Asian Football Federation (WAFF), the Central Asian Football Association (CAFA), and the South Asian Football Federation (SAFF).
- East Region which consists of the associations from the ASEAN Football Federation (AFF) and the East Asian Football Federation (EAFF).
In each region, there are four groups in the group stage, including a total of 12 direct slots, with the 4 remaining slots filled through play-offs.
The top 12 associations in each region as per the AFC rankings are eligible to enter the AFC Champions League.
Accordingly, Thai League 1 Runners-up Buriram United will face V.League 1 runners-up Ho Chi Minh City. Singapore club Tampines Rovers will face the league champions in Indonesia, while Philippines Football League champions Ceres-Negros will take on Myanmar-based Shan United.
FootyRankings further predicts that I-League Champions Chennai City will lock horns with Bahraini Premier League winners Al-Riffa, while in the 2020 AFC Cup, Indian Super League winners Bengaluru FC will have to face the winners of a preliminary match between Bhutan-based Paro FC and Sri Lanka’s Defenders FC.
For more details, check out the Twitter thread right below:


Sample application for the Bosch XDK device that shows raw numeric sensor data in real time. The data is gathered by the XDK device and transferred over Bluetooth Low Energy to the application running on the smartphone. The app displays the data in a simple numeric table.
The following three screenshots show how the app looks like initially, when it's connected, and when it's started showing sensor data.
You can browse the source code for this example at github.
The file index.html contains the HTML markup of the app and the app.js contains the logic.
This example runs in the Evothings Viewer on Android or iOS.
Yoyu need an iOS device or an Android device with support for Bluetooth 4.0 (which includes BLE). For Android version 4.3 or later is needed.
Obviously you need an XDK device from a retail distributor.
In order to be able to build and flash the device with the device side code, you also need to download and install the XDK Workbench 1.6.0 which is available for Windows 7 and later (not Linux or OSX).
Bosch has made available several guides on how to install and get started with the Eclipse based XDK Workbench. However, below you find the specific steps you need to just get the firmware built and flashed:
Use the Copy button on the Bosch Sensors example in the Examples tab in Evothings Studio, to get your own copy of the code.
Fire up the XDK IDE and connect the XDK device using a USB cord. Turn it on. Hopefully it will be discovered (I failed first due to a faulty cord) and then you will get the green icon and the "Flash" and "Boot" buttons appear, something like this:
Then you select "New Project" from the IDE, and then under "C/C++", pick "Makefile Project with Existing Code":
Find the xdk-firmware directory and import the code.
Now you should be able to build and flash your XDK device, either using the "Flash" button or by first building, producing a .bin file under the debug directory, and then flashing that file. Log should show something similar to this:
Success!


STATS CLUSTER SIL Extension Command
Calculate cluster silhouette statistics for a clustering analysis already carried out.
STATS CLUS SIL
CLUSTER = variable*
VARIABLES = clustering variables*
DISSIMILARITY = EUCLID** or ABSDIFF or MINKOWSKI or GOWER or MAXIMUM
MINKOWSKIPOWER = number
NEXTBEST = variable
SILHOUETTE = variable
/OPTIONS
RENUMBERORDINAL = NO** or YES
VARWEIGHTS = numbers
MISSING = RESCALE** or IGNORE or OMIT
/OUTPUT
HISTOGRAM = YES* * or NO
ORIENTATION = HORIZONTAL** or VERTICAL
THREEDBAR = YES** or NO
THREEDCOUNTS = YES or NO**
/HELP
* Required
** Default
STATS CLUSTER SIL /HELP displays this help and does nothing else.
STATS CLUS SIL CLUSTER=clusternumber VARIABLES=x1 x2 x3
DISSIMILARITY=ABSDIFF NEXTBEST = altclus SILHOUETTE = sil.
This command requires the data in memory and takes time that is proportional to the square of the number of cases. With larger datasets, you may want to carry this analysis out on a random sample of the data.
CLUSTER specifies the variable containing the cluster number for each case. Gaps in the numbering, such as might occur if the cluster results are sampled, are treated as clusters of size 0.
VARIABLES lists the variables to be used for calculating silhouettes. These would normally be the variables used to compute the clustering, but this is not required.
DISSIMILARITY specifies the function used to calculate the distance between cases.
NEXTBEST and SILHOUETTE specify variables for the casewise output. The NEXTBEST variable holds the number of the next best cluster. The SILHOUETTE variable holds the silhouette statistics.
RENUMBERORDINAL causes ordinal variables to be recoded as above. It is always used for GOWER, but it can be used with any measure.
VARWEIGHTS can specify weights for each of the clustering variables. If used, each variable difference is multiplied by the corresponding weight. As many weights must be specified as there are clustering variables.
MISSING specifies how missing values are handled in case comparisions. If a variable's contribution to the measure cannot be computed because either case has a missing value, there are three choices.
HISTOGRAM specifies whether a histogram of silhouette scores for each cluster is produced. By default, the histograms are panelled horizontally.
ORIENTATION = VERTICAL causes the histograms to be stacked. Since all the histograms appear on one row, VERTICAL may be needed if there are many clusters.
THREEDBAR = YES produces a 3-d bar chart showing the average silhouette value by assigned cluster and the next best clusters.
THREEDCOUNTS produces a 3-d bar chart showing for each assigned cluster the percentage distribution of next best clusters.
© Copyright IBM Corp. 1989, 2014


The time has come for your to face reality. Your parents won't always be there to make you food. Congratulations! Welcome to the start of adulthood. If you have some prior experience in the kitchen, then you're already off to great start. If not, then don't worry! We're here to help. Learning to cook for yourself is an essential part of growing up. Go through each of the subject matter below and I'm sure you'll come out knowing more.
If you didn't notice, there's lot of dangerous items in the kitchen. Cooking is pretty much chemistry lab except you can eat what you're making. So, the same rules apply:
To become more effective in the kicthen, you should know the proper uses of each knife. This will not only prolong the life of your knives but would also prevent injuries. Below you can find a quick guide to the everyday essential kives.
A paring knife is a good all-around knife. It is ideally a sharp blade often used for chopping small items like fruit and herbs. During apple pie season, it really comes in handy when peeling all those apples.
Similar to the paring knife, a chef's knife is a multipurpose knife in the kitchen. It is approximately 8 to 10 inches long. A chef's knife rounded tip makes it ideal for rock chopping and its straight edge is ideal for tap chopping. For most people, the chef knife is the most used knife in the kitchen.
A bread knife, as the name implies, is most useful when slicing bread. Bread knives are also useful when slicing pineapple! Definitely keep this beautiful tool in mind during the summer months.
A Santoku knife is very similar in use as the chef's knife. The Santoku knife is really usefule when slicing and dicing in the kicthen. This knife originated from Japan and is very efficient.
First, to anyone who is in doubt, yes you need a cutting board. Do NOT cut on your countertop. The kind of cutting board you use affects the sanitation process of your board. It could also affect the lifetime of your knife.
Plastic cutting boards are cheaper than wooden cutting boards. They are easier to clean than their wooden counterpart. Thus, many would argue that they are cleaner than the other kinds of cutting board. However, some people fail to realize that plastic cutting boards tend to be ridden with gashes from knife cuts. Bacteria can grow within these gashes, thereby contamonating the board itself.
Wooden cutting boards tend to be more expensive but lasts longer than plastic cutting boards. Depending on the kind of wood that the board is made off, the lifetime of your knife and board can vary. Cutting boards made out of hardwoods like walnut and maple will dull your knife faster but will be more resistant to scratches and cuts. On the other hand, boards made out of softer woods wold dull your knife less but could sustain cuts faster. Just like in the plastic boards, bacteria grows within the cuts in the wood.
Because plastic and wood boards have different characteristics, their care instrcution would also be different. Plastic boards can be put in the dishwasher and cleaned with high temperature water. On the other hand, wood cutting boards will be ruined by the dishwasher
If cleaning boards by hand, whether it is plastic or wood, you should do the following:


BLOG POST
tag
GIT
sketch out your own contributon calendar
TAG
Click below to start drawing your very own contribution calendar.
GIT
Configure the repository, download and push!
Name
Email
Download


import pygame import game from player import Player # This fil save all constants variables # Screen variables screen_width = 1080 screen_height = 720 # Game variables game_name = "Shooter_Blanck" background_default = pygame.image.load('assets/bg.jpg') player_image_sprite = pygame.image.load('assets/player.png') bullet_image_sprite = pygame.image.load('assets/projectile.png') mummy_image_sprite = pygame.image.load('assets/mummy.png') rotate_image_angle_bullet = 180 default_mummy_posX = 1000 default_mummy_posY = screen_height - (265 - (Player(game).rect.width/4))


Detecting anomalies - a simple approach
During my work I have a lot to do with the analysis of time series. These are mainly signal data which have to be evaluated accordingly. Predictions have to be made, which give a conclusion on how a certain sensor value will behave in e.g. 30 days. This is only one of many applications of forecasting methods.
Today, however, I would like to devote myself to another, equally exciting topic. As an analyst, I am not always interested in the future, but also in the past. For example, I’m interested in certain progressions, constellations, correlations or even outliers of data points.
Outliers can occur in many use cases: Some are real and some are not. Maybe it is a sensor error or an error in the posting of stock items. Or it is actually an outlier that is a real deviation from the target. Such application cases are manifold:
A produced part is not within the specification
The memory consumption of data centers deviates from normal
Or fraud control by comparing the behaviour of a user
A very useful definition of outliers can be found in the Engineering statistics handbook
An outlier is an observation that lies an abnormal distance from other values in a random sample from a population. In a sense, this definition leaves it up to the analyst (or a consensus process) to decide what will be considered abnormal. Before abnormal observations can be singled out, it is necessary to characterize normal observations.
There are many ways to identify outliers. The simplest is probably the graphical evaluation of progressions using a box plot or a time series. However, when it comes to checking data in large quantities and time-critically, such methods reach their limits. You need procedures that can be automated.
In my contribution today I want to develop and test such a procedure.
How can we now implement such an outlier detection. A common method is the determination by the inter-quantile range. By definition, inter-quantile range is:
In descriptive statistics, the interquartile range (IQR), also called the midspread, middle 50%, or H‑spread, is a measure of statistical dispersion, being equal to the difference between 75th and 25th percentiles, or between upper and lower quartiles, IQR = Q3 − Q1. wikipedia; 2020-06-14
In order to determine these, we form the corresponding quantiles and then use them to calculate the IQR
x <- runif(1000, 5.0, 7.5)
quantile <- stats::quantile(x, prob = c(0.25, 0.75), na.rm = TRUE)
quantile[[2]] - quantile[[1]]
## [1] 1.265083
R also offers us with the stats package a function to calculate the value:
iqr <- stats::IQR(x)
iqr
## [1] 1.265083
In order to define the corresponding limits for outliers, a definition must be made of the value from which a data point is declared as an outlier. We use the following definition which is also used in the tsoutliers package:
limits <- quantile + 1.5 * iqr * c(-1, 1)
limits
## 25% 75%
## 3.713498 8.773831
Our function should evaluate the column col of a dataframe and then output a new column outlier. This column is then indexed with TRUE or FALSE whether the corresponding value is an outlier.
detect_outliers <- function(df, col) {
col_enquo <- rlang::enquo(col)
x <- df %>% dplyr::pull(!! col_enquo)
quantile <- stats::quantile(x, prob = c(0.25, 0.75), na.rm = TRUE)
iqr <- stats::IQR(x)
limits <- quantile + 1.5 * iqr * c(-1, 1)
tibble::tibble(Value = x) %>%
dplyr::mutate(lower_limit = limits[[1]],
upper_limit = limits[[2]],
outlier = dplyr::case_when(Value > upper_limit |
Value < lower_limit ~ TRUE,
TRUE ~ FALSE)) %>%
dplyr::bind_cols(df %>% dplyr::select(- !! col_enquo))
}
Let’s test our function on some sample data. Two outliers are built into the data set with 1002 points. One has the value -5, the other 20.
dataframe <- tibble::tibble(Index = 1:1002,
Value = c(-5, runif(1000, 5.0, 7.5), 20))
outlier_df <- detect_outliers(dataframe, "Value")
outlier_df
## # A tibble: 1,002 x 5
## Value lower_limit upper_limit outlier Index
## <dbl> <dbl> <dbl> <lgl> <int>
## 1 -5 3.84 8.68 TRUE 1
## 2 6.08 3.84 8.68 FALSE 2
## 3 5.47 3.84 8.68 FALSE 3
## 4 6.49 3.84 8.68 FALSE 4
## 5 5.22 3.84 8.68 FALSE 5
## 6 5.66 3.84 8.68 FALSE 6
## 7 5.40 3.84 8.68 FALSE 7
## 8 6.66 3.84 8.68 FALSE 8
## 9 5.48 3.84 8.68 FALSE 9
## 10 5.81 3.84 8.68 FALSE 10
## # ... with 992 more rows
Let us look at the result graphically again. In green we see the data points defined as outliers. The two horizontal lines mark the corresponding limits for the evaluation.
outlier_df %>%
ggplot2::ggplot(aes(x = Index)) +
ggplot2::geom_point(aes(y = Value, color = outlier)) +
ggplot2::geom_line(aes(y = lower_limit)) +
ggplot2::geom_line(aes(y = upper_limit)) +
ggplot2::theme_light()
As we can see the two outliers are correctly classified by our outlier detection.
Our chosen data behaves stationary. What if we want to examine data with a trend for outliers? Does our method still work?
dataframe_trend <- tibble::tibble(Index = 1:1002,
Value = c(30, diffinv(rnorm(999, mean = 0, sd = 5)), 0))
outlier_df_trend <- detect_outliers(dataframe_trend, "Value")
outlier_df_trend %>%
ggplot2::ggplot(aes(x = Index)) +
ggplot2::geom_point(aes(y = Value, color = outlier)) +
ggplot2::geom_line(aes(y = lower_limit)) +
ggplot2::geom_line(aes(y = upper_limit)) +
ggplot2::theme_light()
As we can see, our method fails with more complex processes. To solve the problem we need a different approach which I will describe in another post.
My goal in this post was to show that outlier detection can be implemented relatively easily. It is important to note that the underlying data plays a decisive role.


About Me
Alison Lubar teaches high school English by day and yoga by night.
Her life's work, aside from wordsmithing, has evolved into
bringing mindfulness practices, and sometimes even poetry, to
young people. Her work has been published by or appeared in SWWIM
Every Day, trampset, The Esthetic Apostle, Lady Blue Literary Arts
Journal, Cathexis Northwest Press, High Shelf, Gravitas, great
weather for MEDIA, Toho, Rowan University’s Glassworks, Giovanni’s
Room anthology
queerbook, Fearsome Critters' Quaranzine, Apiary Magazine, and antonym lit.
Alison graduated from the University of Delaware in 2007, with a
double major in Philosophy and Italian literature. In 2009, while
teaching at University City High School in West Philadelphia, she
received her MS Ed with a concentration in Urban Education from
the University of Pennsylvania. In 2016, after completing a 95hr
Kids' Yoga Teacher Training, she finished her RYT 200 with Anjali
Power Yoga, where she teaches power vinyasa classes. Alison also
currently teaches at a public high school in New Jersey, where she
has created courses in philosophy, mindfulness, & yoga, and
developed an restorative practices program.
Though she is Californian by birth, Alison currently lives in New
Jersey with her partner and their respective dogs. She hopes to
one day, with her pack, return Westward.
CONNECT
LinkedIn:
https://www.linkedin.com/in/alison-lubar-0210b31b3/
GitHub:
https://github.com/theoriginalison
Email<EMAIL_ADDRESS>or use the form on the contact page.
Resume
Phone:<PHONE_NUMBER> Location: New Jersey & Philadelphia Click here to download Alison's resume.


--- title: Support heading: Support description: Send your questions, comments, bugs, or enhancements. ---
Make sure to check the known bugs and existing enhancement list before submitting.
Creates a GitHub issue<EMAIL_ADDRESS>Private message: blakegearindev@gmail.com


For the Huffman tree shown below, what is the corresponding sequence represented by question? Type its representation as a series of answer.
ANS
To decode a letter, trace the path down the tree to the letter. Add a 0 for each left branch or 1 for right branch, until you get to the leaf node. For example, the code for Z is 1+1+1+1+0+0.
To find the corresponding letter for a code, trace a path through the tree according to the 0s (left) or 1s (right) until you get to a letter. For example, for 1110 you would go left (1), left (1), left (1), and then right (0) to find the letter C.


On 19 August 2018, 1 new articles were published in 1 journals.
Note: although this article was published on 11 August 2018, the publisher only added it to the journal RSS feed on 19 August 2018, which is why it is shown in the list of articles for that date.


{{define "content"}}
A failover request has been submitted to the constellation. It will take a few seconds to determine if the request was successful. In {{.RefreshTime}} seconds the Pod's view will automatically be reloaded.
In the meantime you can load a different page without interrupting the failover process.
{{end}}


Welcome to the third project of the Machine Learning Engineer Nanodegree! In this notebook, some template code has already been provided for you, and it will be your job to implement the additional functionality necessary to successfully complete this project. Sections that begin with 'Implementation' in the header indicate that the following block of code will require additional functionality which you must provide. Instructions will be provided for each section and the specifics of the implementation are marked in the code block with a 'TODO'
statement. Please be sure to read the instructions carefully!
In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question X' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.
Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.
In this project, you will analyze a dataset containing data on various customers' annual spending amounts (reported in monetary units) of diverse product categories for internal structure. One goal of this project is to best describe the variation in the different types of customers that a wholesale distributor interacts with. Doing so would equip the distributor with insight into how to best structure their delivery service to meet the needs of each customer.
The dataset for this project can be found on the UCI Machine Learning Repository. For the purposes of this project, the features 'Channel'
and 'Region'
will be excluded in the analysis — with focus instead on the six product categories recorded for customers.
Run the code block below to load the wholesale customers dataset, along with a few of the necessary Python libraries required for this project. You will know the dataset loaded successfully if the size of the dataset is reported.
# Import libraries necessary for this project
import numpy as np
import pandas as pd
from IPython.display import display # Allows the use of display() for DataFrames
# Import supplementary visualizations code visuals.py
import visuals as vs
# Pretty display for notebooks
%matplotlib inline
# Load the wholesale customers dataset
try:
data = pd.read_csv("customers.csv")
data.drop(['Region', 'Channel'], axis = 1, inplace = True)
print "Wholesale customers dataset has {} samples with {} features each.".format(*data.shape)
except:
print "Dataset could not be loaded. Is the dataset missing?"
In this section, you will begin exploring the data through visualizations and code to understand how each feature is related to the others. You will observe a statistical description of the dataset, consider the relevance of each feature, and select a few sample data points from the dataset which you will track through the course of this project.
Run the code block below to observe a statistical description of the dataset. Note that the dataset is composed of six important product categories: 'Fresh', 'Milk', 'Grocery', 'Frozen', 'Detergents_Paper', and 'Delicatessen'. Consider what each category represents in terms of products you could purchase.
# Display a description of the dataset
display(data.describe())
To get a better understanding of the customers and how their data will transform through the analysis, it would be best to select a few sample data points and explore them in more detail. In the code block below, add three indices of your choice to the indices
list which will represent the customers to track. It is suggested to try different sets of samples until you obtain customers that vary significantly from one another.
# TODO: Select three indices of your choice you wish to sample from the dataset
indices = [154, 181, 338]
# Create a DataFrame of the chosen samples
samples = pd.DataFrame(data.loc[indices], columns = data.keys()).reset_index(drop = True)
print "Chosen samples of wholesale customers dataset:"
display(samples)
import seaborn as sns
percentiles_data = 100*data.rank(pct=True)
percentiles_samples = percentiles_data.iloc[indices]
sns.heatmap(percentiles_samples, annot=True)
Consider the total purchase cost of each product category and the statistical description of the dataset above for your sample customers.
Hint: Examples of establishments include places like markets, cafes, delis, wholesale retailers, among many others. Avoid using names for establishments, such as saying "McDonalds" when describing a sample customer as a restaurant. You can use the mean values for reference to compare your samples with. The mean values are as follows:
Knowing this, how do your samples compare? Does that help in driving your insight into what kind of establishments they might be?
Answer:
The establishment with indice 154 has an annual spending much lower than average belonging to the lower 25 percentile for all the features. This shows that it could be a rather small business. We also see that the majority of its annual spending covers "Fresh" products followed by "Grocery". This can be the indicator that the establishement is a small restaurant.
The establishment with indice 181 has an annual spending much higher than average in all of the features. It's in the upper 75 percentile for all category of products which makes me think that it might be a very big hotel or for a chain of hotels.
The establishment with indice 338 has an annual spending overall lower than the mean except for the products belonging to the "Frozen" category which is actually pretty high with an annual spending of 15601 and therefore belonging in the upper 75 percentile ot the dataset. Considering the Frozen feature, it is actually about 2 std away from the mean on the upper side although all the other features are rather low. This observation can lead us to believe that this establishment might be a market that sells frozen product.
One interesting thought to consider is if one (or more) of the six product categories is actually relevant for understanding customer purchasing. That is to say, is it possible to determine whether customers purchasing some amount of one category of products will necessarily purchase some proportional amount of another category of products? We can make this determination quite easily by training a supervised regression learner on a subset of the data with one feature removed, and then score how well that model can predict the removed feature.
In the code block below, you will need to implement the following:
new_data
a copy of the data by removing a feature of your choice using the DataFrame.drop
function.sklearn.cross_validation.train_test_split
to split the dataset into training and testing sets.test_size
of 0.25
and set a random_state
.random_state
, and fit the learner to the training data.score
function.# TODO: Make a copy of the DataFrame, using the 'drop' function to drop the given feature
def score_feature(data, feature):
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
new_data = data.drop([feature], axis=1)
# TODO: Split the data into training and testing sets(0.25) using the given feature as the target
# Set a random state.
all_scores = []
for i in range (100):
X_train, X_test, y_train, y_test = train_test_split(new_data, data[feature], test_size=0.25)
# TODO: Create a decision tree regressor and fit it to the training set
regressor = DecisionTreeRegressor()
regressor.fit(X_train, y_train)
# TODO: Report the score of the prediction using the testing set
all_scores.append(regressor.score(X_test, y_test))
score = np.mean(all_scores)
print "For feature \"{}\", the coefficient of determination is {}".format(feature, score)
for key in data:
score_feature(data, key)
Hint: The coefficient of determination, R^2
, is scored between 0 and 1, with 1 being a perfect fit. A negative R^2
implies the model fails to fit the data. If you get a low score for a particular feature, that lends us to beleive that that feature point is hard to predict using the other features, thereby making it an important feature to consider when considering relevance.
Answer: I didn't set the random state because the coefficient of determination varied greatly when I changed it so I instead calculated the mean of scores over 100 iterations to have a more general idea.
I first attempted to predict "Fresh" and the coefficient of determination was -0.708230948053. The score being negative, it shows that no correlation with the other features has been found and that the model fails to fit the data. This shows that "Fresh" provides unique information that cannot be predicted by other features, therefore the feature "Fresh" should be necessary for identifying customers' spending habits.
To get a better understanding of the dataset, we can construct a scatter matrix of each of the six product features present in the data. If you found that the feature you attempted to predict above is relevant for identifying a specific customer, then the scatter matrix below may not show any correlation between that feature and the others. Conversely, if you believe that feature is not relevant for identifying a specific customer, the scatter matrix might show a correlation between that feature and another feature in the data. Run the code block below to produce a scatter matrix.
# Produce a scatter matrix for each pair of features in the data
pd.plotting.scatter_matrix(data, alpha = 0.3, figsize = (14,8), diagonal = 'kde');
import seaborn as sns; sns.set()
corr = data.corr()
mask = np.zeros_like(corr)
mask[np.triu_indices_from(mask, 1)] = True
with sns.axes_style("white"):
ax = sns.heatmap(corr, mask=mask, annot=True, fmt='+.3f')
Hint: Is the data normally distributed? Where do most of the data points lie? You can use corr() to get the feature correlations and then visualize them using a heatmap(the data that would be fed into the heatmap would be the correlation values, for eg: data.corr()
) to gain further insight.
Answer:
The data appears to have a highly positively skewed distribution with all the features, showing the presence of outliers with very high annual spendings for all features compared to the average. Indeed, most of the data points are near zero. This biais could impact the algorithm performance and therefore it is important to apply some kind of normalisation to make the features normally distributed.
We can see that "Grocery" and "Detergents_Paper" exhibit the highest correlation pair with 0.925. We also see some correlation between "Grocery" and "Milk" (0.728); "Milk" and "Detergents_Paper" (0.662) but the correlation is relatively mild. There are no obvious correlation pair with "Fresh", "Frozen" and "Delicatessen".
This confirms what we observed previously, the coefficient of determination for predicting "Fresh" with the other features was -0.708230948053. This score showed no obvious correlation with the other features which corroborates our observation in the scatter matrix where no clear pattern can be observed with any of the features.
In this section, you will preprocess the data to create a better representation of customers by performing a scaling on the data and detecting (and optionally removing) outliers. Preprocessing data is often times a critical step in assuring that results you obtain from your analysis are significant and meaningful.
If data is not normally distributed, especially if the mean and median vary significantly (indicating a large skew), it is most often appropriate to apply a non-linear scaling — particularly for financial data. One way to achieve this scaling is by using a Box-Cox test, which calculates the best power transformation of the data that reduces skewness. A simpler approach which can work in most cases would be applying the natural logarithm.
In the code block below, you will need to implement the following:
log_data
after applying logarithmic scaling. Use the np.log
function for this.log_samples
after applying logarithmic scaling. Again, use np.log
.# TODO: Scale the data using the natural logarithm
log_data = np.log(data.copy())
# TODO: Scale the sample data using the natural logarithm
log_samples = np.log(samples)
# Produce a scatter matrix for each pair of newly-transformed features
pd.plotting.scatter_matrix(log_data, alpha = 0.3, figsize = (14,8), diagonal = 'kde');
After applying a natural logarithm scaling to the data, the distribution of each feature should appear much more normal. For any pairs of features you may have identified earlier as being correlated, observe here whether that correlation is still present (and whether it is now stronger or weaker than before).
Run the code below to see how the sample data has changed after having the natural logarithm applied to it.
# Display the log-transformed sample data
display(log_samples)
Detecting outliers in the data is extremely important in the data preprocessing step of any analysis. The presence of outliers can often skew results which take into consideration these data points. There are many "rules of thumb" for what constitutes an outlier in a dataset. Here, we will use Tukey's Method for identfying outliers: An outlier step is calculated as 1.5 times the interquartile range (IQR). A data point with a feature that is beyond an outlier step outside of the IQR for that feature is considered abnormal.
In the code block below, you will need to implement the following:
Q1
. Use np.percentile
for this.Q3
. Again, use np.percentile
.step
.outliers
list.NOTE: If you choose to remove any outliers, ensure that the sample data does not contain any of these points!
Once you have performed this implementation, the dataset will be stored in the variable good_data
.
# For each feature find the data points with extreme high or low values
for feature in log_data.keys():
# TODO: Calculate Q1 (25th percentile of the data) for the given feature
Q1 = np.percentile(log_data[feature], 25)
# TODO: Calculate Q3 (75th percentile of the data) for the given feature
Q3 = np.percentile(log_data[feature], 75)
# TODO: Use the interquartile range to calculate an outlier step (1.5 times the interquartile range)
step = 1.5*(Q3-Q1)
# Display the outliers
print "Data points considered outliers for the feature '{}':".format(feature)
display(log_data[~((log_data[feature] >= Q1 - step) & (log_data[feature] <= Q3 + step))])
# OPTIONAL: Select the indices for data points you wish to remove
outliers = [65, 66, 75, 128, 154]
# Remove the outliers, if any were specified
good_data = log_data.drop(log_data.index[outliers]).reset_index(drop = True)
outliers
list to be removed, explain why.Hint: If you have datapoints that are outliers in multiple categories think about why that may be and if they warrant removal. Also note how k-means is affected by outliers and whether or not this plays a factor in your analysis of whether or not to remove them.
Answer:
The following data points were found to be outliers in more than one feature based on the above definition [65, 66, 75, 128, 154]. I think those points should be removed from the dataset as they might induce wrong biaises to the model. Indeed because those are outliers for multiple feature this shows a specific spending behavior that is not representative of the masse. Therefore I added them to the outliers
list above. I kept the data points that were flagged as outliers for only one feature because removing them all would mean taking out a consequent amount of data points (42 in total), representing 9.5% of the data this would generally not be recommended without a strong justification. One other possibility could have been to increase the step size in Tukey's method to find the extreme outliers.
In this section you will use principal component analysis (PCA) to draw conclusions about the underlying structure of the wholesale customer data. Since using PCA on a dataset calculates the dimensions which best maximize variance, we will find which compound combinations of features best describe customers.
Now that the data has been scaled to a more normal distribution and has had any necessary outliers removed, we can now apply PCA to the good_data
to discover which dimensions about the data best maximize the variance of features involved. In addition to finding these dimensions, PCA will also report the explained variance ratio of each dimension — how much variance within the data is explained by that dimension alone. Note that a component (dimension) from PCA can be considered a new "feature" of the space, however it is a composition of the original features present in the data.
In the code block below, you will need to implement the following:
sklearn.decomposition.PCA
and assign the results of fitting PCA in six dimensions with good_data
to pca
.log_samples
using pca.transform
, and assign the results to pca_samples
.# TODO: Apply PCA by fitting the good data with the same number of dimensions as features
from sklearn.decomposition import PCA
pca = PCA(n_components=6).fit(good_data)
# TODO: Transform log_samples using the PCA fit above
pca_samples = pca.transform(log_samples)
# Generate PCA results plot
pca_results = vs.pca_results(good_data, pca)
print pca_results['Explained Variance'].cumsum()
Hint: A positive increase in a specific dimension corresponds with an increase of the positive-weighted features and a decrease of the negative-weighted features. The rate of increase or decrease is based on the individual feature weights.
Answer:
| Dimension | Explained Variance | Cumulative Explained Variance | Discussion |
|---|---|---|---|
| 1 | 0.4430 | 0.4430 | Mainly "Detergents_Paper", "Milk" and "Grocery" together negatively-weighted. This corresponds to day-to-day household spendings which might represent retailers. |
| 2 | 0.2638 | 0.7068 | Mainly "Fresh", "Frozen" and "Delicatessen" together negatively-weighted. This corresponds to mainly food spendings which might represent restaurants, canteens. |
| 3 | 0.1231 | 0.8299 | Mainly "Fresh" being negatively-weighted and "Delicatessen" being positively-weighted. This corresponds mainly to fresh and ethnic food spendings which might represent small familial restaurants, organic/ethnic food shops. |
| 4 | 0.1012 | 0.9311 | Mainly "Frozen" being positively-weighted and "Delicatessen" being negatively-weighted. This corresponds mainly to frozen and ethnical food spendings which might represent convinient stores. |
| 5 | 0.0485 | 0.9796 | Mainly "Milk" being positively-weighted and "Detergent_Paper" being negatively-weighted. |
| 6 | 0.0204 | 1.0000 | Mainly "Grocery" being positively-weighted and "Milk"being negatively-weighted. |
Run the code below to see how the log-transformed sample data has changed after having a PCA transformation applied to it in six dimensions. Observe the numerical value for the first four dimensions of the sample points. Consider if this is consistent with your initial interpretation of the sample points.
# Display the log-transformed sample data
display(log_samples)
# Display sample log-data after having a PCA transformation applied
display(pd.DataFrame(np.round(pca_samples, 4), columns = pca_results.index.values))
When using principal component analysis, one of the main goals is to reduce the dimensionality of the data — in effect, reducing the complexity of the problem. Dimensionality reduction comes at a cost: Fewer dimensions used implies less of the total variance in the data is being explained. Because of this, the cumulative explained variance ratio is extremely important for knowing how many dimensions are necessary for the problem. Additionally, if a signifiant amount of variance is explained by only two or three dimensions, the reduced data can be visualized afterwards.
In the code block below, you will need to implement the following:
good_data
to pca
.good_data
using pca.transform
, and assign the results to reduced_data
.log_samples
using pca.transform
, and assign the results to pca_samples
.# TODO: Apply PCA by fitting the good data with only two dimensions
pca = PCA(n_components=2).fit(good_data)
# TODO: Transform the good data using the PCA fit above
reduced_data = pca.transform(good_data)
# TODO: Transform log_samples using the PCA fit above
pca_samples = pca.transform(log_samples)
# Create a DataFrame for the reduced data
reduced_data = pd.DataFrame(reduced_data, columns = ['Dimension 1', 'Dimension 2'])
Run the code below to see how the log-transformed sample data has changed after having a PCA transformation applied to it using only two dimensions. Observe how the values for the first two dimensions remains unchanged when compared to a PCA transformation in six dimensions.
# Display sample log-data after applying PCA transformation in two dimensions
display(pd.DataFrame(np.round(pca_samples, 4), columns = ['Dimension 1', 'Dimension 2']))
A biplot is a scatterplot where each data point is represented by its scores along the principal components. The axes are the principal components (in this case Dimension 1
and Dimension 2
). In addition, the biplot shows the projection of the original features along the components. A biplot can help us interpret the reduced dimensions of the data, and discover relationships between the principal components and original features.
Run the code cell below to produce a biplot of the reduced-dimension data.
# Create a biplot
vs.biplot(good_data, reduced_data, pca)
Once we have the original feature projections (in red), it is easier to interpret the relative position of each data point in the scatterplot. For instance, a point the lower right corner of the figure will likely correspond to a customer that spends a lot on 'Milk'
, 'Grocery'
and 'Detergents_Paper'
, but not so much on the other product categories.
From the biplot, which of the original features are most strongly correlated with the first component? What about those that are associated with the second component? Do these observations agree with the pca_results plot you obtained earlier?
"Detergents_Paper", "Grocery" and "Milk" are the features most strongly correlated with the first component because most of their magnitude can be described along the first component axis.
"Delicatessen", "Frozen" and "Fresh" are the features most strongly correlated with the second component because most of their magnitude can be described along the second component axis.
We see that the observations support the results obtained in the pca_results plot for dimension 1 and dimension 2 as we can clearly see that the projection of the vectors onto each dimension correspond to the PC plane with original features projections.
In this section, you will choose to use either a K-Means clustering algorithm or a Gaussian Mixture Model clustering algorithm to identify the various customer segments hidden in the data. You will then recover specific data points from the clusters to understand their significance by transforming them back into their original dimension and scale.
Hint: Think about the differences between hard clustering and soft clustering and which would be appropriate for our dataset.
Answer:
K-Means advantages:
Gaussian Mixture Model advantages:
The main criteria I would take into account to choose between these two algorithms are the speed VS second order information desired and the underlying structure of the data.
Since our dataset is quite small and scalability would not be an issue I would use the Gaussian Mixture Model because ranking the data points with their likelihood (soft classification) gives more flexibility and insight about the result than K-means (hard classification).
Depending on the problem, the number of clusters that you expect to be in the data may already be known. When the number of clusters is not known a priori, there is no guarantee that a given number of clusters best segments the data, since it is unclear what structure exists in the data — if any. However, we can quantify the "goodness" of a clustering by calculating each data point's silhouette coefficient. The silhouette coefficient for a data point measures how similar it is to its assigned cluster from -1 (dissimilar) to 1 (similar). Calculating the mean silhouette coefficient provides for a simple scoring method of a given clustering.
In the code block below, you will need to implement the following:
reduced_data
and assign it to clusterer
.reduced_data
using clusterer.predict
and assign them to preds
.centers
.pca_samples
and assign them sample_preds
.sklearn.metrics.silhouette_score
and calculate the silhouette score of reduced_data
against preds
.score
and print the result.# TODO: Apply your clustering algorithm of choice to the reduced data
from sklearn.mixture import GaussianMixture
from sklearn.metrics import silhouette_score
from sklearn.cluster import KMeans
'''
In the function, silhouette(i), you have hard-coded the value of n_components to be 2.
This is why you are getting approximately the same silhouette score in each iteration.
'''
def silhouette(i, n):
clusterer = GaussianMixture(n_components=n).fit(reduced_data)
# clusterer = KMeans(n_clusters=2).fit(reduced_data)
# TODO: Predict the cluster for each data point
preds = clusterer.predict(reduced_data)
# TODO: Find the cluster centers
centers = clusterer.means_
# centers = clusterer.cluster_centers_
# TODO: Predict the cluster for each transformed sample data point
sample_preds = clusterer.predict(pca_samples)
# TODO: Calculate the mean silhouette coefficient for the number of clusters chosen
score = silhouette_score(reduced_data, preds)
print "- Silhouette coefficient for {} clusters: {}".format(n, score)
return preds, centers, sample_preds
n=10
for i in range(1,10):
preds, centers, sample_preds = silhouette (i, n)
i += 1
n -= 1
Answer:
The best silhouette score is with 2 clusters : 0.422324682646
Once you've chosen the optimal number of clusters for your clustering algorithm using the scoring metric above, you can now visualize the results by executing the code block below. Note that, for experimentation purposes, you are welcome to adjust the number of clusters for your clustering algorithm to see various visualizations. The final visualization provided should, however, correspond with the optimal number of clusters.
# Display the results of the clustering from implementation
vs.cluster_results(reduced_data, preds, centers, pca_samples)
Each cluster present in the visualization above has a central point. These centers (or means) are not specifically data points from the data, but rather the averages of all the data points predicted in the respective clusters. For the problem of creating customer segments, a cluster's center point corresponds to the average customer of that segment. Since the data is currently reduced in dimension and scaled by a logarithm, we can recover the representative customer spending from these data points by applying the inverse transformations.
In the code block below, you will need to implement the following:
centers
using pca.inverse_transform
and assign the new centers to log_centers
.np.log
to log_centers
using np.exp
and assign the true centers to true_centers
.# TODO: Inverse transform the centers
log_centers = pca.inverse_transform(centers)
# TODO: Exponentiate the centers
true_centers = np.exp(log_centers)
# Display the true centers
segments = ['Segment {}'.format(i) for i in range(0,len(centers))]
true_centers = pd.DataFrame(np.round(true_centers), columns = data.keys())
true_centers.index = segments
display(true_centers)
display(np.exp(good_data).describe())
Hint: A customer who is assigned to 'Cluster X'
should best identify with the establishments represented by the feature set of 'Segment X'
. Think about what each segment represents in terms their values for the feature points chosen. Reference these values with the mean values to get some perspective into what kind of establishment they represent.
Answer: The value for "Milk", "Grocery" and "Detergents_Paper" in segment 0 (cluster 0) belong to the upper percentile 75 of the dataset and could therefore help identify the establishments such as big and small shops/retailers reselling primarely groceries, dairies and household products.
The value for "Fresh" and "Frozen" in segment 1 (Cluster 1) are above the median of the dataset and could help identify the establishments such as restaurants/canteens needing primarely fresh products but also need some frozen food.
Run the code block below to find which cluster each sample point is predicted to be.
# Display the predictions
for i, pred in enumerate(sample_preds):
print "Sample point", i, "predicted to be in Cluster", pred
Answer:
The sample point 0 corresponds to the establishment index 154 and was estimated to be a small restaurant. It is indeed consistent with the classification in the Cluster 1 that I above described.
The sample point 1 corresponds to the establishment index 181. Unfortunately this data point was removed from the dataset because it was considered an outlier in a previous section. I estimated this customer to be a big Hotel or a chain of hotels as it was spending much more in all features than the average. It has been classified in the Cluster 0 that I described as representing mainly establishments being big and small shops/retailers. Being an outlier makes this categorization complicated to interpret without having a closer look to the data.
The sample point 2 corresponds to the establishment index 338 and was estimated to be a market that sells frozen product. It was classified in the Cluster 1. It is consistent with the observation that the Cluster 1 includes establishments with higher spendings on frozen products than the ones in Cluster 0.
In this final section, you will investigate ways that you can make use of the clustered data. First, you will consider how the different groups of customers, the customer segments, may be affected differently by a specific delivery scheme. Next, you will consider how giving a label to each customer (which segment that customer belongs to) can provide for additional features about the customer data. Finally, you will compare the customer segments to a hidden variable present in the data, to see whether the clustering identified certain relationships.
Companies will often run A/B tests when making small changes to their products or services to determine whether making that change will affect its customers positively or negatively. The wholesale distributor is considering changing its delivery service from currently 5 days a week to 3 days a week. However, the distributor will only make this change in delivery service for customers that react positively.
Hint: Can we assume the change affects all customers equally? How can we determine which group of customers it affects the most?
Answer:
We can run A/B tests separately for each segment where a certain percentage of customers are offered the 3 days a week option (group A) instead of the 5 days a week option (group B). A representative proportions of customers from each segments should be tested for the new option. This would allow to determine if there is a correlation between how well the new option is received. At the end of a defined period, it would be possible to compare the revenu made by the distributor and determine if customers reacted positively or negatively depending on segments and groups (A or B).
Additional structure is derived from originally unlabeled data when using clustering techniques. Since each customer has a customer segment it best identifies with (depending on the clustering algorithm applied), we can consider 'customer segment' as an engineered feature for the data. Assume the wholesale distributor recently acquired ten new customers and each provided estimates for anticipated annual spending of each product category. Knowing these estimates, the wholesale distributor wants to classify each new customer to a customer segment to determine the most appropriate delivery service.
Hint: A supervised learner could be used to train on the original customers. What would be the target variable?
Answer:
Now that we have customer segments fairly well described, we can engineer those segments as new features for the data. Because there are only two segments, we can apply this to a binary supervised learner. We can consider the feature "segment" the target variable (labels), split the data into training and testing sets and train the supervised learner on the training set. The learner should be tested for its performance and if the performance is satisfying, then the learner should be able to predict with a known accuracy in which new customers should belong only with their features (which are the provided estimates for anticipated annual spending of each product category).
At the beginning of this project, it was discussed that the 'Channel'
and 'Region'
features would be excluded from the dataset so that the customer product categories were emphasized in the analysis. By reintroducing the 'Channel'
feature to the dataset, an interesting structure emerges when considering the same PCA dimensionality reduction applied earlier to the original dataset.
Run the code block below to see how each data point is labeled either 'HoReCa'
(Hotel/Restaurant/Cafe) or 'Retail'
the reduced space. In addition, you will find the sample points are circled in the plot, which will identify their labeling.
# Display the clustering results based on 'Channel' data
vs.channel_results(reduced_data, outliers, pca_samples)
Answer:
Segment 0 (Cluster 0) could help identify the establishments such as big and small shops/retailers reselling primarely groceries, dairies and household products.
Segment 1 (Cluster 1) could help identify the establishments such as restaurants/canteens needing primarely fresh products but also need some frozen food, dairies and groceries.
Here segment 0 corresponds to "Retailer" and segment 1 corresponds to "Hotel/Restaurant/Cafe". Those classifications are consistent with the previous definition I gave of the customer segments.
Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to
File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.


|
Elektra
0.8.23
|
A plugin which takes care about initializing the haskell run-time.
This plugin on its own provides a minimalistic implementation to test the basic functionality, but acts as a base for developing further haskell plugins. To be precise, the following files are required and should use the following naming scheme, with <PLUGIN_NAME>
being the name of this plugin (haskell
in this case):
<PLUGIN_NAME>.cabal.in
which is the cabal build file for the plugin, ensure that:hs-source-dirs:
is set to "@CMAKE_CURRENT_SOURCE_DIR@"
build-depends:
includes libelektra-haskell
for the haskell bindingsbuild-type:
is set to Custom
@CABAL_CUSTOM_SETUP@
is included somewhere in case the default Setup.hs is usedtestmod_<PLUGIN_NAME>.c
which includes the tests for the pluginUse the cmake command add_haskell_plugin which can be used by including LibAddHaskellPlugin. This command will take care about the proper linking of your haskell plugin. Furthermore it uses the c wrapper provided by this plugin so this doesn't have to be done again.
By default plugins get built in a cabal sandbox shared with all other Haskell plugins- and the Haskell bindings to speed up compilation by compiling commonly used dependencies just once. Dependencies are automatically resolved according to the cabal build configuration.
Currently the Haskell plugin only executes tests written in C and not directly in Haskell.
Unfortunately the haskell runtime itself leaks some memory, but as haskell plugins are generally executed in an isolated process it should not matter too much.


REVIEW: Her Majesty Mrs Brown (1997)
The attraction of love is a powerful thing. In the tale of Queen Victoria and an extravagant veteran servant, awkwardly titled "Mrs Brown," we are reminded of a value lesson. For when the word love is mentioned, we immediately think of marriages and intimate relationships. But the love and attraction highlighted in Mrs Brown is not the kind found in sex or marriage; rather in the wonder and bondage of a true friendship.
During the lengthy reign of Queen Victoria (Judi Dench), her majesty went through a period of deep grief and depression after the death of her beloved husband Albert. Disappearing from public view and perception, the secretive and distressed Queen has lost confidence in her role as leader of the country.
Entering her life is John Brown (Billy Connolly), a loyal yet outspoken servant that devotes his life to giving the Queen confidence and protection at any cost. Over the following couple of years the two become immersed in friendship and drawn together with a powerful attraction. The Queen is not an attractive woman, but her power and stand in society is, and although both of them realize and even consider the costs of having an affair, the intimate boundaries of their friendship prevents them from doing so. Behind her back, the Queen is dubbed as "Mrs. Brown."
Whilst this relationship is taking place, England is not benefiting from the Queen's seemingly eternal absence from public view. So as the monarchy beings to fall into crisis, Prime Minister Disraeli (Anthony Sher) realizes that Brown is the key to refreshing faith in the crown.
Mrs. Brown can surely be considered as an elegant and graceful film, but it quickly finds itself trying too hard to make us care for the relationship between the Queen and Brown, which ultimately falls flat on its face as soon as we are witness to the Queen's uninspiring and emotionless personality. The film's dialogue doesn't give Judi Dench any chance to get into her distressing role, so naturally when the Queen enters the scene Mrs Brown takes a tumble into an almost non existent level of interest.
Billy Connolly, responsible for picking up the pieces of a fairly weak film, is an absolute revelation and refreshes the audience with a touch of well placed humor and a diligent presence on the screen. For those of you who don't know, he is actually a stand up comedian - so if you thought that casting him as John Brown would be like casting Jerry Seinfeld as Hamlet, think again.
The finest part of Mrs Brown's awkwardly crafted collection of scenes is just near the end, yet ironically the final scenes of the film are among the worst. They seem incredibly rushed and hastily put together; so as the lights in the cinema were becoming brighter I was left gazing at the credits asking myself - "Did I miss something?"
Another main problem with the film, and call me what you will, is that it's just not interesting enough. I don't have anything against historical and periodic stories, but Mrs. Brown proved too bland and often much too dull to keep me eagerly watching. There are two ways that the production crew could have addressed this problem - either alter history to suit the film's needs (which, thankfully, they did not do) or to change the focus of the story to a more interesting character or setting. So in this case, I would have been delighted if Mrs. Brown had shown us much more of Billy Connolly's intriguing character rather than monotonous Queen who doesn't tantalize any climatic taste buds.
But overall, Mrs Brown is an entertaining period film that could have been much more than what it amounted to be, and the main problem lies in the relationship between the Queen and Brown that never really works. More often then not the film appears to be emotionless and cold; which should never be related to a film so caught up in love and loyalty.
2 = stars out of 5
For a film originally made for British TV, the idiot box would be an ideal place to showcase this slightly disappointing drama.
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


module Distributed: sig
.. end
This module provides modules to create distribtued computations.
Distributed comutations are described using the
Distributed.Process
.
Distributed.Process
provides a monadic interface to describe distributed computations.
Author(s): essdotteedot [<essdotteedot[at]gmail[dot]com>]
Version: 0.4.0
Some nomenclature :
- Node : A node corresponds to a operating system process. There can be many nodes on a
single machine.
- Process : A process corresponds to a light weight thread (i.e., user space cooperative threads).
There can be many processes on a single Node.
module Node_id: sig
.. end
This module provides a type representing a node id.
module Process_id: sig
.. end
This module provides a type representing a process id.
module type Nonblock_io = sig
.. end
Abstract type which can perform monadic concurrent IO.
module type Message_type = sig
.. end
The abstract type representing the messages that will be sent between processes.
module type Process = sig
.. end
A unit of computation which can be executed on a local or remote host, is monadic.
module Make (
I
:
Nonblock_io
)
(
M
:
Message_type
)
: Process
with type message_type = M.t and type 'a io = 'a I.t and type logger = I.logger


<<< BACK <<<
STM-030: EtherTrust
Authors
Grishchenko, Ilya and Maffei, Matteo and Schneidewind, Clara
Venue and Year
, 2018
Original Work
EtherTrust: Sound Static Analysis of Ethereum bytecode
Defense Modality
Subtractive
Core Method(s)
static analysis
Input-Output Mapping
bytecode => report
Targeted Contracts
Any Ethereum
Threat Model
Vulnerable contract, malicious contract, or both
both
Targeted Vulnerabilities