A few weeks ago I posed a code challenge that was extremely open ended. It left a lot of elements to the imagination of the reader of the problem to research, determine, or just make up. In that same sense, here is the question that I posed on Monday, September the 18th. The code challenge as I was saying wasn’t some simple, solve it in 5 minutes, type of thing. This was something where we humans actually have real problems to solve in the world. So here we go, this is going to get seriously intense (and long cuz’ I’m gonna describe every bit of my process). I’ll start with a quick recap of the problem.
Given that roadways have three primary functions they’re used for: active transport, storage or parking, and nil or no use. Solve for what percentage of time a roadway experiences each. Details are further elaborated on here:
Ask: Well what type of road is it, how many lanes, how many parking spots? What is the layout?
Answer: Here are three types of roads specifically to solve for in a first attempt. These are specified just to have a starting point.
NOTE: I added pictures of the road types by using Street Mix. If you’re interested in road alignments then this tool might consume a few hours of your day, so beware!
- Country roadway: Two lanes, no parking spaces, and no area to pull off on. Each lane has a capacity of ~800-1200 cars per hour. Traffic on the particular road is about 8000 per day.
Note: I added a tram to the picture, but it is not there for calculation reasons. It’s there merely because I wanted to have a tram there. The focus on this is still entirely the public roadway to the left of the tram.)
- City arterial, 6-lanes: Two lanes each way, one lane parking on each side. Each lane has a theoretical capacity of about ~700-1000 cars per hour (that lane changing brings down the number). Traffic usually is about 12,000 per day.
NOTE: The travel lane in the center, even though it has an arrow for one way, is actually bi-directional. It’s very similar to a lot of the older town center organized neighborhoods as seen in Brooklyn in New York, in Ballard or a number of other neighborhoods in Seattle, and many in Portland. Almost every major US city has neighborhoods with small streets like this, often an indicator of a time when it wasn’t assumed you put your person property (a car) in the road and the road was actually considered public domain by the general populace. But I digress, back to the coding problem.
- Neighborhood street, 3-lanes: Single travel lane in the center, one parking lane on either side. Ability to move about ~500 cars per hour but only experiences about ~1500 cars per day.
The challenge here far exceeds merely some algorithm to solve for X, but instead ponders what would need to be included to find transport use (driving or in movement), storage (parked car), and no car, no movement, and nobody parked anywhere to be seen (and empty road).
My Solution
Key known things at this point.
- There are three roads we want to solve for: country roadway, city arterial, and neighborhood street.
- Respectively each has ~800-1200 cars per hour, ~700-1000 cars per hour, and the last specifically has ~500 cars per hour capacity but only about ~1500 per day.
- Respectively each has 2 lanes, 6 lanes, and 3 lanes. Along with this a few other facts here and there.
- We have no idea how many cars can park on or near one of these roads. We need to determine that.
- We have no idea when these cars arrive or are not arriving. So maybe we’ll just determine how long it takes for them to go by.
Let’s just set some scope and guidelines for making these calculations.
- The roadways we’ll measure will be for exactly 1 kilometer.
- The roadways, for initial simplicity of creating a solution, will not have any interruptive side roads, lights, stop signs, or such, but will just be merely 1 kilometer long without interruptions.
- For parked cars, I’m going to say the average length of a car is equal to a Toyota Camry. That used to be one of the most sold cars in the US, and is a pretty in between size. It’s not a huge truck, but it also isn’t a Mini Cooper! The dimensions for a 2017 Toyota Camry are 191″ L x 72″ W x 58″ H. Let’s imagine that in the front and rear of the car we need at least some space, so I’ll calculate that at half a meter in front and half a meter in back. So 191 inches is 4.851 meters (that’s based on 39.3701 inches per meter), plus the space in front and back, gives us 5.851 meters. For the width, the Camry is 72 inches giving us 1.828 meters width, and a height of 58 inches doesn’t really matter in these calculations. In summary, our parking places will need to be 1.828 meters wide by 5.851 meters in length.
- I wrote the capacity of each roadway per hour, and for these calculations I’ll continue to assume the capacity per hour is what the road actually experiences.
- For the time period to determine parked, moving, or empty space, I’ll start working with a 24 hour period.
That gives me a few points now to really start working with. Let’s also make the following rules:
- I’ll refer to parking, or storage of a car on the road simply as storage or S from now on.
- For a moving car, truck, bus, or other vehicle in use for transport I’ll refer to simply as transport, transporting, or T.
- For road space that is not in use and has no vehicles, I’ll call that dead space, or D.
Now let’s work with this added information combined with the assumptions I’ve drawn up.
S | Storage
S should be easy to determine as I only need to determine what can fit on the side of a road. The country road is easy, since there is no parking. The other two I wrote there is a lane of parking on each side of the street. For those two, that would be 1 kilometer divided by the 5.851 meters, then multipled by 2 since there are two sides to the road. There are a thousand meters in a kilometer so our initial work looks like this.
S = 1000 / 5.851 * 2.
Which gives us<PHONE_NUMBER>92 parking spaces per side, I’ll round down just to have a clean 170 spots per roadside, giving us 340 parking sports per kilometer of roadway. This should suffice on both of the roads listed above.
I also added time above, because time is fundamental element to determine the actual metrics above, but that leaves me with a question of how frequently to spaces rotate.
TO THE GOOGLES
I did some research here about parking in various cities to get an idea for turnover rate. I specifically searched for parking turnover in seattle
and parking turnover in portland
. Holy smokes I hit the jackpot of data right off with those. My google-fu is on point. If you ever were curious if someone was actually thinking about where, when, how, and what we can park, rest assured there is a massive amount of thought that goes into it.
Portland Links:
- State of Parking
- Public Parking in The City of Portland
- PBOT page with many of the above and other links
- Central City Parking Occupancy and Turnover Analysis
Seattle Links:
I then went about reading through them all and then realized, this is vastly more complex then it needs to be for the immediate moment. With that conclusion, I realized it ought to just be a variable and I’ll make one up upon executing the function.
There’s one more thing that just popped into my head, to help get an actual measurement of things I’d also need to know what percent of the spots stay empty over time. I’ll measure the number of parking spots empty per hour. This is so we can also determine later how much empty parking lends to unused roadway space.
With that, here we go.
- Time Being Measured in Hours = T
- Parking Space (Length in meters) = P
- Roadway Distance (Length in meters) = R
- Turnover Rate (Cars Parked Per Hour) = H
- Empty Spots (Average # Per Hour) = E
Where S is the total time cars are parked in parking spots on the roadway.
S = ((R / P) - E) * H * T
T | Transport / Transporting
Whew, I’ve got S, next up is T for the number of vehicles moving by per hour. I actually provided the cars per hour on the roadway, but that leaves us to determine how much of the roadway is actually being used to move that number of cars. Let’s try to get a decent ballpark figure.
For the country road I stated that it has 8000 cars per day. For the arterial road the route has about 12,000 per day, and for the neighborhood street there are only about 1500. I also stated the throughput of each road, so let’s use the country road as an example and work from that.
I originally stated a 800-1200 cars per hour, with 8000 traveling through the road per day. A day is 24 hours of course. Let’s work with the idea we’re getting maximum throughput of 1200 cars per hour. If there is 8000 cars per day, at 1200 per hour, we could divide the total per day by the hours like 8000 / 1200 = hours it takes to have 8000 cars travel the road. Which in this case, is 6.666666666 hours. How devilish!
The equation then looks something like this.
- The total cars traversing the road = D
- The total cars that can travel through the road per hour = H
Where T is the total hours in which transporting is being done.
T = D / H
D | Dead Space
Alright, so this equation is going to take into account the time left and is going to also get super tricky. More on its trickiness after I post the solution though. For now, soffice to say the total dead space is the remainder hours when cars aren’t traversing and also the dead time of cars not parked. At least, that’s what I’ll go with for now.
- Transport or Transporting (the result of the above T equation) = T
- Empty parking spots per hour, value available from determinant in the S equation as = E
Where D is the hours of dead space on the road.
D = T + E
Alright it’s time to sling some code now that the pieces are collected.
CODE!
First things first, I’m going to put in place a test for each of these functions that I’ll need. These tests are by no means a paramount example of testing, nor do they test much of anything besides the extreme happy path, but they’ll be perfect for getting started to verify the functions we’ll write. The tests also will provide the basis in which to extend past these calculations in the future and really start to extrapolate this into a project that gasp may be a real usable thing.
I’ve created a thrashed.go
and thrashed_test.go
file for this initial bit of coding. The first test I’ve added in the test file is the test for what I’ll call the Storage function.
func TestStorage(test *testing.T) {
var t float32 = 24 /* 24 hours in a day. */
var p float32 = 5.851 /* Meters a parking space needs to be */
var r float32 = 1000 /* Meters of roadway distance in calculation */
var h float32 = 1.8 /* Cars parked per hour in a single parking spot. */
var e float32 = 8 /* Empty parking spots per hour for the measured distance. */
if Storage(t, p, r, h, e) != 7037.753 {
test.Error("Failure to determine correct storage result.")
}
}
I’m a big fan of tests being a good source of documentation and how to use a set of functions or libraries. With that in mind I’ve added a few comments to define what each variable is. In the thrashed.go
file I dive into implementing the calculation for storage.
func Storage(TimeBeingMeasured float32, ParkingSpace float32, RoadwayDistance float32, TurnoverRate float32, EmptySpots float32) float32 {
return ((RoadwayDistance / ParkingSpace) - EmptySpots) * TurnoverRate * TimeBeingMeasured
}
I’m not happy with this example, as I wrote out every variable so that they’re self descriptive. But it leaves the function signature a bit long. Even though it won’t be self descriptive, I’m going to turn these into acronyms for now.  For now, just for now, I’m going to refactor the storage function to look like this.
func Storage(tbm float32, ps float32, rd float32, tr float32, es float32) float32 {
return ((rd / ps) - es) * tr * tbm
}
Am I happier with that? Not really, but it at least fits on the screen. Anyway, on to the next metric to measure. Following good practice, a test first.
func TestTransport(t *testing.T) {
var d int = 8000 /* Total cars traveling the roadway for the test time frame. */
var h int = 1200 /* The number of cars that can travel the roadway per hour. */
result := Transport(d, h)
fmt.Println(result)
if result != 6.6666665 {
t.Error("Failure to determine correct transportation throughput.")
}
}
Then I implement it with this.
func Transport(TotalCars int, CarsPerHour int) float32 {
return float32(TotalCars) / float32(CarsPerHour)
}
Now for the dead space test.
func TestDeadSpace(test *testing.T) {
var t int = 23 /* Transport or transporting (the result of the "Transport" equation). */
var e int = 2 /* Empty parking sports per hour. This value is what would be the same \
as the E value in the Storage function. */
result := DeadSpace(t, e)
if result != 25 {
test.Error("Nope that isn't the correct amount of dead space.")
}
}
Then implementation of it.
func DeadSpace(t int, e int) int {
return t + e
}
With those functions in place I can now, finally, after all this work move toward an initial answer! Now I’m going to write a set of tests, one for each of the percentages that are requested in the original code challenge. One test for the storage, transport, and dead space percentages. For these tests, I’m doing a nice easy split of storage, transport, and dead space; 50, 75, and 75. That was I can calculate it out in my head if need be.
func TestGimmeTheStoragePercent(t *testing.T) {
result := GimmeTheSolutionAlready(50, 75, 75)
if result[0] != 25 {
t.Error("The storage percentage is not correct.")
}
}
func TestGimmeTheTransportPercent(t *testing.T) {
result := GimmeTheSolutionAlready(50, 75, 75)
if result[1] != 37.5 {
t.Error("The transport percentage is not correct.")
}
}
func TestGimmeTheDeadspacePercent(t *testing.T) {
result := GimmeTheSolutionAlready(50, 75, 75)
if result[2] != 37.5 {
t.Error("The dead space percentage is not correct.")
}
}
My implementation that I then put together looks like this.
func GimmeTheSolutionAlready(storage float64, transport float64, deadspace float64) []float64 {
var quotient = (storage + transport + deadspace) / 100
var storagePercent = storage / quotient
var transportPercent = transport / quotient
var deadspacePercent = deadspace / quotient
var result []float64
result = append(result, storagePercent)
result = append(result, transportPercent)
result = append(result, deadspacePercent)
return result
}
Did I do it right? Here’s the Github Repo! Whew, I’ll check it a bit later, but happy for someone else to point out an error or code mishap! Overall though, this is my first brutally long rough draft of code with a few unit tests and this whole blog entry written!
I’m also interested in what else you think should go into this mix. I’m likely to turn this into a CLI just for fun and would like to add additional complexities into it, so that when I implment the CLI it can create the actual story of what this tells us about a roadway.
Until next code challenge, or ya know whenever, happy coding!
References Collection:


SDrive_t Struct Reference
List of all members.
|
Public Attributes |
| HANDLE | handle |
| unsigned int | type |
| | handle to the drive.
|
| unsigned int | bufferSize |
| | CD-ROM, DVD-ROM, HDD, SSD, unknown.
|
| unsigned int | maxLASize |
| | size of the drive's cache in bytes.
|
| unsigned int | maxLACount |
| | maximum size of a look-ahead read.
|
| unsigned int | maxSeekBack |
| | maximum number of reads to look-ahead.
|
| unsigned int | bytesPerSector |
| | maximum number of bytes to read-back.
|
| unsigned int | sectorsPerCluster |
| | number of bytes per sector.
|
| unsigned int | bytesPerCluster |
| | number of sectors per cluster.
|
Detailed Description
Definition at line 82 of file FSDiskServ_W32.cpp.
Member Data Documentation
The documentation for this struct was generated from the following file:


Id = 18
Nombre Capitulo = Mr. Goldenfold's dream
Tipo = Dream
Dimension = Dimension C-137
Personaje = ['https://rickandmortyapi.com/api/character/63', 'https://rickandmortyapi.com/api/character/80', 'https://rickandmortyapi.com/api/character/221', 'https://rickandmortyapi.com/api/character/246', 'https://rickandmortyapi.com/api/character/304', 'https://rickandmortyapi.com/api/character/305', 'https://rickandmortyapi.com/api/character/306', 'https://rickandmortyapi.com/api/character/396']
Numero Personaje = ['63', '80', '221', '246', '304', '305', '306', '396']
URL Locacion = https://rickandmortyapi.com/api/location/18


Hello World!
This is a basic template for starting your WebApp project
Open File Selector New Mode
Close Window


Handwritten digits classification with MDP and scikits.learn¶
CodeSnippet
You can download all the code on this page from the code snippets directory
If you have the library scikits.learn installed on your machine, MDP will automatically wrap the algorithms defined there, and create a new series of Nodes, whose name ends with ScikitsLearnNode.
In this example, we show how it is possible to merge MDP nodes and flows and a scikits.learn classifier to perform handwritten digit classification.
One of the nice features of scikits.learn is that it provides access to several classical dataset. First of all, we will load the digits dataset:
>>> import mdp
>>> import numpy
>>>
>>> import sklearn as sl
>>> from sklearn import datasets
>>>
>>> digits = datasets.load_digits()
>>> images = digits.images.astype('f')
>>> labels = digits.target
>>> data = digits.images.reshape((images.shape[0],
... numpy.prod(images.shape[1:])))
and divide it in a training and a test set:
>>> # number of digits to be used as training data (2/3 of total)
>>> ntrain = images.shape[0] // 3 * 2
>>> train_data = [data[:ntrain, :]]
>>> train_data_with_labels = [(data[:ntrain, :], labels[:ntrain])]
>>> test_data = data[ntrain:, :]
>>> test_labels = labels[ntrain:]
These are a few examples of the 8 x 8 images in the database:
For our handwritten digits classification application, we build a flow that performs these steps:
- reduce the dimensionality of the data from 8 x 8 = 64 to 25 using Principal Component Analysis
- expand the data in the space of polynomials of degree 3, i.e., augment the pixel data (x_1, x_2, …) with monomial of order 2 and three, like x_1 * x_2^2 or x_1 * x_3 . This is a common trick to transform a linear algorithm (in this case, the one in step 4) in a non-linear algorithm
- reduce the dimensionality of the data again, keeping 99% of the variance in the expanded data
- perform Fisher Discriminant Analysis (FDA), a supervised algorithm that finds a projection of the data that maximizes the variance between labels, and minimize the variance within labels. In other words, the algorithm tries to form well separated clusters for each label.
- classify the digit using the Support Vector Classification algorithm defined in scikits.learn
In the application, the data type is set to single precision to spare memory:
>>> flow = mdp.Flow([mdp.nodes.PCANode(output_dim=25, dtype='f'),
... mdp.nodes.PolynomialExpansionNode(3),
... mdp.nodes.PCANode(output_dim=0.99),
... mdp.nodes.FDANode(output_dim=9),
... mdp.nodes.SVCScikitsLearnNode(kernel='rbf')], verbose=True)
Note how it is possible to set parameters in the the scikits.learn algorithms simply by using the corresponding keyworg argument. In this case, we use Radial Basis Function kernels for the classifier.
We’re ready to train our algorithms on the training data set:
>>> flow.train([train_data, None, train_data,
... train_data_with_labels, train_data_with_labels])
Training node #0 (PCANode)
Training finished
Training node #1 (PolynomialExpansionNode)
Training finished
Training node #2 (PCANode)
Training finished
Training node #3 (FDANode)
Training finished
Training node #4 (SVCScikitsLearnNode)
Training finished
Close the training phase of the last node
>>> # print the final state of the nodes
>>> print repr(flow)
Flow([PCANode(input_dim=64, output_dim=25, dtype='float32'),
PolynomialExpansionNode(input_dim=25, output_dim=3275, dtype='float32'),
PCANode(input_dim=3275, output_dim=646, dtype='float32'),
FDANode(input_dim=646, output_dim=9, dtype='float32'),
SVCScikitsLearnNode(input_dim=9, output_dim=9, dtype='float32')])
Finally, we can execute the application on the test data set, and compute the error rate:
>>> # set the execution behavior of the last node to return labels
>>> flow[-1].execute = flow[-1].label
>>>
>>> # get test labels
>>> prediction = flow(test_data)
>>> # percent error
>>> error = ((prediction.flatten() != test_labels).astype('f').sum()
... / (images.shape[0] - ntrain) * 100.)
>>> print 'percent error:', error
percent error: 3.33889816361
One can probably do better than 3.3 percent error using a larger non-linear space, using more PCA components, or using another classifier. Have fun exploring the parameters!


What you write here is for checksum generation only, not an archive. It will be deleted, so please store it elsewhere!
Payment should be made to this address:
This script should be saved and should be shared with all the participants before a payment is made, so they may validate the authenticity of the address, it will also be used later to release the bitcoins.
Transaction This should be filled out and immediately signed by the SELLER
"INPUTS" is the amount paid by the BUYER.
Multisig Redeem Script:
(80 byte limit, 40 bytes recommended)
When using this option you may enter a hex string or address into the address field on the output tab.
The settings page can be used to select alternative networks of which you can retrieve your unspent outputs and broadcast a signed transaction into.
Enter the addresses/amounts payments will go to from the Multisig address:
BTC Address of Purchase Price Recipient (Seller):
Purchase Price:
USD $:
BTC Address of Arbitrator:
Arbitration Fixed Rate:
USD $ 0.10
BTC Address of Facilitator:
Facilitator's Fixed Bonus:
USD $ 5.00
Checksum for Contract:
Enter the details of inputs you wish to spend.
Optimized Transaction Fee
USD $:
The transaction below has been generated and encoded. It can be broadcast once it has been signed by two parties.
Size: 0bytes
Address
Amount
After reviewing the above, if everything looks perfect to you, go ahead and sign it with your private key:
There is a problem with one or more of your inputs, please check and try again
This transaction has been signed once and can be retrieved by your counter-party for final signature:
Congratulation!
Luckily, your counter-party has prepared and signed your transaction already. Check it out carefully and if it's all good, sign it.
Of course, if you prefer, you can continue with the above form and then have the counter-party provide the second signature.
Address
Amount
After reviewing the above, if everything looks as it should, go ahead and sign it with your private key:
.
There is a problem with one or more of your inputs, please check and try again
Settings making coinb.in even better!
These simple settings below make coinb.in one of the most decentralized wallets in the world.
Network:
Select which network you'd like to use for key pair generation.
Pub:
Priv:
Script Hash (multisig):
HD Pub:
HD Priv:
You will not be able to automatically broadcast or retreive your unspent outputs from coinb.in when using this setting and will need to use your desktop client instead, however everything else such as creating key pairs, addresses, transaction generation and signing will continue to function normally.
Broadcast:
Select the network you wish to broadcast the transaction via
Unspent outputs:
Select the network you wish to retreive your unspent inputs from
This page uses javascript by "Coinb.in" and others to generate your addresses and sign your transactions within your browser. This means we never receive your private keys. You can easily verify this yourself, right here and now if you like, by examining the data traffic using your browser's web developer/console.
"Coinb.in" has been around forever, is intelligently put together, is much used and loved by bitcoiners worldwide and you won't find a bad thing written about it.
Checkabid/AltConSys uses the *public* keys and scripts only and stores them on a server briefly to allow "stateful" transactions that are safe but anonymous, transparent, convenient and intuitive. Once transactions are concluded they get deleted. No log-in is required, no email address. We don't care who you are or what you do.
You can add a public key when creating a 2-of-3 multi signature address and for a low fee your abitrator will help with the recovery of the funds should any disputes arise.
Should a dispute arise please contact the below address for further information
(for disputes)
Reset Keys / Start a New Transaction
If you push below RESET button all keys of this transaction will be replaced by a new set.
If you are far into the transaction this may inconvenience your contract partner!
Enter the other party's public key here
Your contract partner should have sent you their public key. Enter this public key here:
Warning High Fee!
Please be aware that you have created a transaction with what seems to be a very high fee of BTC!


In Jigsaw each served URI is bound to an object generating content. This object is mapped to a FramedResource (FileResource, DirectoryResource...) which in turn is associated with a ProtocolFrame (usually the HTTPFrame or any subclass) instance, which in turn is created manually (with JigAdmin 1.0 or 2.0) or automatically by an indexer.
This document has the following sections:
Inside Jigsaw, a resource is a full Java object, containing only information that the raw Resource (a file, a directory...) can provide (eg, for a file, the size, last modification date...)
To be available on the server, a resource MUST be associated with a ProtocolFrame implementing the protocol available on that particular server. An instance of Jigsaw (ie: the java process) can support multiple servers, each server having the possibility of implementing different protocols and may or may not share the same set of resources.
The list of all Resources is available.
A Frame is a full Java Object, containing all the information needed to serve this Resource using a specific Protocol (eg, HTTPFrame for HTTP).
The list of all Frames is available.
A Filter is a full Java Object, associated to a Frame, that can modify the Request and/or the Reply. For example the Authentication is handled by a special filter, the GenericAuthFilter (Note, as this filter is protocol dependant, it is placed on the protocol frame).
The list of all Filters is available.
An indexers, placed on a Container, will be in charge of creating its sons resources. It will create Resource of a special kind depending, for example, on the extension of the filename ("html" for an html page, "png" for a PNG image file....). Or place a specific indexer for cgi on a directory named "cgi-bin". In Jigsaw-2.0, the indexer is not only in charge of creating the resource, it has also to put the right protocol frames (and other frames if necessary) on the created resource.
An indexer have two main part:
In the extensions, you have to specify how to index files or leaf Resources. The default extension are mapped to FileResource, html, gif, png, txt... In the 2.0 version, an HTTPFrame is added to the Resource.
A tutorial about the setup of indexers is available, it helps understanding how it works.
All the basic Resources, such as FileResource and DirectoryResource, were heavily linked to HTTP as all the resources served were extensions protocols that are not HTTP-related, we propose this new version of the Resource:
Where (1) and (2) are ResourceFrames. A Resource is now a very basic thing, containing only information that the raw Resource can provide (e.g., for a file, the size, last modification date, creation date if available, etc.), then, attached to that Resource, we find the ResourceFrames that extend Resource (they are handled the same way) and contain information about the Resource they are attached to.
To serve a resource using a protocol -- for instance, HTTP -- the Resource will have a protocol ResourceFrame, HTTPFrame, that contains all the information needed to serve this Resource using HTTP. This frame is like the old version of HTTPResource, but it contains more information than the previous version.
The filters are now divided in two categories: the filters on the Resource and the filters on the protocol Frames. The usual filtering scheme used in the previous version of Jigsaw is still valid. The main difference is that filters are no longer attached to the Resource itself but to its protocol frame. ResourceFrames can also have frames.
Other kind of frames can be attached, like RDF frame for metadata, PICS frame to rate this resource, etc...
The new inheritance tree is:
more complex, but more flexible than the previous version.
This RSM contains a hashtable associating a key (unique indentifier of a ResourceContainer) and a StoreEntry. The StoreEntry contains the store of the resource sons and a hashtable associating the identifier of the sons of the resource and the ResourceReference of those resources.
The ResourceReference is used like this:
� ResourceReference rr; � .... � try { ���� Resource res = rr.lock(); ���� .... � } catch (InvalidResourceException ex) { ��� /* InvalidResource means that the resource has been deleted */ ���� .... � } finally { ��� rr.unlock(); � } � ...If the resource has been garbage-collected, the rr.lock() will load it again, and during the lock, it is guaranteed that the resource won't be deleted, unloaded or modified by someone else. This allows safe concurrent modification access to this resource.
Now the container is no longer responsible for the management of its son; it only has a key to the StoreEntry, which contains its sons. To get its own store, the resource has to ask its parent for the StoreEntry that contains it.
The following picture show a Jigsaw resources tree (relative to the URL /archives/index.html), where root and archives are DirectoryResource (root is the root resource) and index.html is a FileResource. F1, F2 and F3 are filters (ResourceFilter subclass instance).
In the following description, Jigsaw receive an HTTP GET request for /archives/index.html. To handle the incomming request, Jigsaw will go through the following steps:
root lookup(ls,lr) ��� -> HTTPFrame1 lookup(ls,lr) ������� -> F1 lookup(ls,lr) ������� -> HTTPFrame1 lookupDirectory(ls,lr) ��� -> archives lookup(ls,lr) ������� -> HTTPFrame2 lookup(ls,lr) ����������� -> F2 lookup(ls,lr) ����������� -> HTTPFrame2 lookupDirectory(ls,lr) ������� -> index.html lookup(ls,lr) ����������� -> HTTPFrame3 lookup(ls,lr) ��������������� -> F3 lookup(ls,lr) ��������������� -> HTTPFrame3 lookupFile(ls,lr) => index.html
2) Call the ingoingFilter method of filters. Request is the incomming request.
F1 ingoingFilter(Request) F2 ingoingFilter(Request) F3 ingoingFilter(Request)
3) Perform the request (GET) on the resource found at lookup time.
index.html perform(Request) ��� -> HTTPFrame3 perform(Request) ������� -> HTTPFrame3 get(Request) ����������� -> HTTPFrame3 getFileResource(Request) => Reply
F3 outgoingFilter(Request, Reply) F2 outgoingFilter(Request, Reply) F1 outgoingFilter(Request, Reply)


mformat(1) mformat(1) Name mformat - add an MSDOS filesystem to a low-level formatted floppy disk Note of warning This manpage has been automatically generated from mtools's texinfo docu‐ mentation, and may not be entirely accurate or complete. See the end of this man page for details. Description The mformat command is used to add an MS-DOS file system to a low-level formatted diskette. Its syntax is: mformat [-t cylinders|-T tot_sectors] [-h heads] [-s sectors] [-f size] [-1] [-4] [-8] [-v volume_label] [-F] [-S sizecode] [-X] [-2 sectors_on_track_0] [-3] [-0 rate_on_track_0] [-A rate_on_other_tracks] [-M software_sector_size] [-N serial_number] [-a] [-C] [-H hidden_sectors] [-I fsVersion] [-r root_sectors] [-L fat_len] [-B boot_sector] [-k] [-m media_descriptor] [-K backup_boot] drive: Mformat adds a minimal MS-DOS file system (boot sector, FAT, and root directory) to a diskette that has already been formatted by a Unix low- level format. The following options are supported: (The S, 2, 1 and M options may not exist if this copy of mtools has been compiled without the USE_2M option) The following options are the same as for MS-DOS's format command: Options v Specifies the volume label. A volume label identifies the disk and can be a maximum of 11 characters. If you omit the -v switch, mformat will assign no label to the disk. f Specifies the size of the DOS file system to format. Only a cer‐ tain number of predefined sizes are supported by this flag; for others use the -h/-t/-s flags. The following sizes are supported: 160 160K, single-sided, 8 sectors per track, 40 cylinders (for 5 1/4 DD) 180 160K, single-sided, 9 sectors per track, 40 cylinders (for 5 1/4 DD) 320 320K, double-sided, 8 sectors per track, 40 cylinders (for 5 1/4 DD) 360 360K, double-sided, 9 sectors per track, 40 cylinders (for 5 1/4 DD) 720 720K, double-sided, 9 sectors per track, 80 cylinders (for 3 1/2 DD) 1200 1200K, double-sided, 15 sectors per track, 80 cylinders (for 5 1/4 HD) 1440 1440K, double-sided, 18 sectors per track, 80 cylinders (for 3 1/2 HD) 2880 2880K, double-sided, 36 sectors per track, 80 cylinders (for 3 1/2 ED) t Specifies the number of tracks on the disk. T Specifies the number of total sectors on the disk. Only one of these 2 options may be specified (tracks or total sectors) h The number of heads (sides). s Specifies the number of sectors per track. If the 2m option is given, number of 512-byte sector equivalents on generic tracks (i.e. not head 0 track 0). If the 2m option is not given, number of physical sectors per track (which may be bigger than 512 bytes). 1 Formats a single side (equivalent to -h 1) 4 Formats a 360K double-sided disk (equivalent to -f 360). When used together with -the 1 switch, this switch formats a 180K disk 8 Formats a disk with 8 sectors per track. MS-DOS format's q, u and b options are not supported, and s has a differ‐ ent meaning. The following options are specific to mtools: F Format the partition as FAT32. S The size code. The size of the sector is 2 ^ (sizecode + 7). X formats the disk as an XDF disk. See section XDF, for more details. The disk has first to be low-level formatted using the xdfcopy utility included in the fdutils package. XDF disks are used for instance for OS/2 install disks. 2 2m format. The parameter to this option describes the number of sectors on track 0, head 0. This option is recommended for sectors bigger than normal. 3 don't use a 2m format, even if the current geometry of the disk is a 2m geometry. 0 Data transfer rate on track 0 A Data transfer rate on tracks other than 0 M software sector size. This parameter describes the sector size in bytes used by the MS-DOS file system. By default it is the physi‐ cal sector size. N Uses the requested serial number, instead of generating one auto‐ matically a If this option is given, an Atari style serial number is gener‐ ated. Ataris store their serial number in the OEM label. C creates the disk image file to install the MS-DOS file system on it. Obviously, this is useless on physical devices such as flop‐ pies and hard disk partitions, but is interesting for image files. H number of hidden sectors. This parameter is useful for formatting hard disk partition, which are not aligned on track boundaries (i.e. first head of first track doesn't belong to the partition, but contains a partition table). In that case the number of hidden sectors is in general the number of sectors per cylinder. This is untested. I Sets the fsVersion id when formatting a FAT32 drive. In order to find this out, run minfo on an existing FAT32 drive, and mail me about it, so I can include the correct value in future versions of mtools. c Sets the size of a cluster (in sectors). If this cluster size would generate a FAT that too big for its number of bits, mtools automatically increases the cluster size, until the FAT is small enough. d Sets the number of FAT copies. Default is 2. This setting can also be specified using the MTOOLS_NFATS environment variable. r Sets the size of the root directory (in sectors). Only applicable to 12 and 16 bit FATs. This setting can also be specified using the MTOOLS_DIR_LEN environment variable. L Sets the length of the FAT. B Use the boot sector stored in the given file or device, instead of using its own. Only the geometry fields are updated to match the target disks parameters. k Keep the existing boot sector as much as possible. Only the geom‐ etry fields and other similar file system data are updated to match the target disks parameters. K Sets the sector number where the backup of the boot sector should be stored (only relevant on FAT32). m Use a non-standard media descriptor byte for this disk. The media descriptor is stored at position 21 of the boot sector, and as first byte in each FAT copy. Using this option may confuse DOS or older mtools version, and may make the disk unreadable. Only use if you know what you are doing. To format a diskette at a density other than the default, you must supply (at least) those command line parameters that are different from the default. Mformat returns 0 on success or 1 on failure. It doesn't record bad block information to the Fat, use mbadblocks for that. See Also Mtools' texinfo doc Viewing the texi doc This manpage has been automatically generated from mtools's texinfo docu‐ mentation. However, this process is only approximative, and some items, such as crossreferences, footnotes and indices are lost in this transla‐ tion process. Indeed, these items have no appropriate representation in the manpage format. Moreover, not all information has been translated into the manpage version. Thus I strongly advise you to use the original texinfo doc. See the end of this manpage for instructions how to view the texinfo doc. * To generate a printable copy from the texinfo doc, run the follow‐ ing commands: ./configure; make dvi; dvips mtools.dvi * To generate a html copy, run: ./configure; make html A premade html can be found at `http://www.gnu.org/software/mtools/man‐ ual/mtools.html' * To generate an info copy (browsable using emacs' info mode), run: ./configure; make info The texinfo doc looks most pretty when printed or as html. Indeed, in the info version certain examples are difficult to read due to the quot‐ ing conventions used in info. mtools-4.0.17 29Jun11 mformat(1)


Hello
How's it going? My name is Justin
I'm a Creative Technologist
I work at Google, Creative Lab (NYC)
I'm from England
- I like
-
- Actionscript
- Ambient Music
- Animation
- Cycling
- Design
- Drawing
- Drumming
- Generative Art
- Illustration
- Javascript
- Open Frameworks
- Processing
- Programming
- Physics
- Spicy Food
- Strong Coffee
- Typography
- WebGL


class: center, middle .title[Lecture 2: Covariance and correlation] .author[Shane Elipot] .institution[The Rosenstiel School of Marine and Atmospheric Science, University of Miami] .coauthor[] .institution[] .date[] .note[] .footnote[Created with [{Remark.js}](http://remarkjs.com/) using [{Markdown}](https://daringfireball.net/projects/markdown/) + [{MathJax}](https://www.mathjax.org/)] --- exclude: true name: foreword class: left, .toc[[✧](#toc)] ## Foreword --- name: references class: left, .toc[[✧](#toc)] ## References [1] Bendat, J. S., & Piersol, A. G. (2011). *Random data: analysis and measurement procedures* (Vol. 729). John Wiley & Sons. [2] Thomson, R. E., & Emery, W. J. (2014). *Data analysis methods in physical oceanography*. Newnes. [dx.doi.org/10.1016/B978-0-12-387782-6.00003-X](http://dx.doi.org/10.1016/B978-0-12-387782-6.00003-X) [3] Taylor, J. (1997). *Introduction to error analysis, the study of uncertainties in physical measurements*. [4] Press, W. H. et al. (2007). *Numerical recipes 3rd edition: The art of scientific computing*. Cambridge university press. [5] Kanji, G. K. (2006). *100 statistical tests*. Sage. [6] von Storch, H. and Zwiers, F. W. (1999). *Statistical Analysis in Climate Research*, Cambridge University Press --- name: toc class: left, .toc[[✧](#toc)] # Lecture 2: Outline 1. [Covariance & correlation](#covariance) 2. [Lagged covariance & correlation](#laggedcovariance) 3. [A quick look at "A leisurely look at the bootstrap, the jackknife, and cross-validation"](#bootstrap) 4. [Covariance and correlation of bivariate variables](#bivariate) --- exclude: true name: introduction class: center, middle, .toc[[✧](#toc)] # 1. Introduction --- exclude: true class: left, .toc[[✧](#toc)] ## Introduction --- name: covariance class: center, middle, .toc[[✧](#toc)] # 1. Covariance & Correlation --- class: left, .toc[[✧](#toc)] ## Covariance (definitions) Whereas we previously dealt with a single r.v. `$x$`, we now deal with two r.vs., `$x$` and `$y$`. In particular, we are interested in evaluating how much they covary, possibly to make some statement about a causation from one variable to the other, perhaps explained by a dynamical relationship. Perhaps the first quantity to consider is the *covariance of $x$ and $y$*: `$$ \text{Cov}(x,y) = C_{xy} \equiv E[(x-\mu_x)(y-\mu_y)] $$` The *variance of $x$* is a particular case of covariance when $y$ and $x$ are the same r.v.: `$$ \text{Cov}(x,x) = C_{xx} \equiv E[(x-\mu_x)(x-\mu_x)] = E[(x-\mu_x)^2] = \text{Var}(x) = \sigma^2_x $$` --- class: left, .toc[[✧](#toc)] ## Covariance (definitions) `$$ \begin{eqnarray} C_{xy} & = & E[(x-\mu_x)(y-\mu_y)] = E[xy] - \mu_x \mu_y \cr & = & \int^{+\infty}_{-\infty} (x-\mu_x)(y-\mu_y)p(x,y)\, dx dy \cr \end{eqnarray} $$` `$p(x,y)$` is called the *joint probability density function* or *joint PDF*. If `$p(x,y) = p(x) p(y)$` where $p(x)$ and $p(y)$ are the PDFs of $x$ and $y$, respectively, then it is said that $x$ and $y$ are *independent*. Just like we were able to build histograms from samples $X_n$ of a single r.v. $x$, we can also build 2D histograms from pairs of samples $(X_n,Y_n)$ in order to estimate the joint PDF. --- class: left, .toc[[✧](#toc)] ## Correlation Since the r.v. $x$ and $y$ can be of different nature or magnitude, we can consider the normalized covariance, that is the *correlation* between $x$ and $y$ `$$ \rho_{xy} = \frac{C_{xy}}{\sqrt{C_{xx}C_{yy}}} = \frac{C_{xy}}{\sigma_x \sigma_y} = \frac{E[(x-\mu_x)(y-\mu_y)]}{\left\{E[(x-\mu_x)^2]E[(y-\mu_y)^2]\right\}^{1/2}} $$` Since we have the property that `$|C_{xy}|\leq \sigma_x \sigma_y$`, the correlation is a number between `$-1$` and `$1$`. If $\rho_{xy} = 0$ it is said that the variable $x$ and $y$ are *uncorrelated*. If two r.vs. are independent `$[p(x,y) = p(x) p(y)]$` **then** they are also uncorrelated, but if two r.vs. are uncorrelated, they are not necessarily independent (i.e. maybe `$p(x,y) \neq p(x) p(y)$`) --- class: left, .toc[[✧](#toc)] ## Covariance & Correlation: estimation Just like `$s^2_x$` is an unbiased estimate of `$\sigma^2_x$`, an unbiased estimate of the covariance is `$$ s_{xy} = \widehat{C}_{xy} = \frac{1}{N-1}\sum_{n=1}^N (X_n-\overline{X})(Y_n-\overline{Y}) $$` and an estimate of `$\rho_{xy}$` is `$$ \begin{eqnarray} r_{xy} & = & \widehat{\rho}_{xy} = \frac{s_{xy}}{\sqrt{s^2_{x}s^2_{y}}} = \frac{\sum_{n=1}^N(X_n-\overline{X})(Y_n-\overline{Y})} {\left[\sum_{n=1}^N (X_n-\overline{X})^2\sum_{n=1}^N(Y_n-\overline{Y})^2\right]^{1/2}}\cr \end{eqnarray} $$` `$r_{xy}$` is called the *Pearson's correlation coefficient*. It measures the relative strength of a *linear relationship* between $x$ and $y$ (see Lecture 3). A *nonlinear relationship* or *noise* will make `$r_{xy}$` tends to zero. --- class: left, .toc[[✧](#toc)] ## Correlation: significance Since `$x$` and `$y$` are r.vs., `$r_{xy}$` is also a r.v. with a given distribution. Typically, one wants to test if `$r_{xy}$` is different from zero. [Kanji (2006), reference [5]](#references) recommends to calculate the following test statistic (test 12) `$$ t = \frac{r_{xy}}{\sqrt{1-r_{xy}^2}}\sqrt{N-2} \sim t(0,N-2) $$` which follows the Student's t-distribution with `$N-2$` degrees of freedom. --- class: left, .toc[[✧](#toc)] ## Correlation: confidence interval Alternatively, one may want to derive CIs for `$r_{xy}$` by using the Fisher transformed variable: `$$ w = \tanh^{-1} r_{xy} = \frac{1}{2}\ln\left(\frac{1+r_{xy}}{1-r_{xy}}\right) \sim \mathcal{N}\left[\frac{1}{2}\ln\left(\frac{1+r_{xy}}{1-r_{xy}}\right),\frac{1}{\sqrt{N-3}}\right] $$` Since `$r_{xy} = \tanh w$`, the CI derived for `$w$` can be used to calculate CIs for `$r_{xy}$`. The test statistic `$$ z = \frac{w - w_0}{1/\sqrt{N-3}} \sim \mathcal{N}(0,1) $$` can be used to test the null hypotheses that `$\rho_{xy} = \tanh w_0$`, see test 13 of [Kanji (2006), reference [5]](#references). --- class: left, .toc[[✧](#toc)] ## Correlation: effective degrees of freedom A crucial aspect of testing for the significance of `$\rho_{xy}$` is to use the right value for `$N$`, which is the size of your sample if your data points are independent. Otherwise, a lower value of $N$, known as the *effective degree of freedom*, needs to be considered. We will explore these aspects later and during the practical this afternoon. --- class: left, .toc[[✧](#toc)] ## Serial correlation: example Let's consider a realization `$X_{1},X_{2},\ldots,X_{N}$` of a time series `$x(t)$`, and calculate the correlations `$\rho(X_n,X_{n+1})$`: .center[
] Since `$r_{xy} \neq 0$`, `$X_n$` and `$X_{n+1}$` are correlated, hence they are not independent samples. It is said that $x$ is *serially correlated*, thus the number of degrees of freedom is less than $N$. The variance and bias of correlation estimates involving `$x$` are likely to be affected. --- class: left, .toc[[✧](#toc)] ## Spearman correlation The Pearson's correlation coefficient is not the only correlation coefficient. The *rank correlation coefficient* or *Spearman correlation coefficient* is `$$ r_s = 1 - \frac{6\sum_{n=1}^N (R_{x;n} - R_{y;n})^2}{N(N^2-1)} $$` where `$R_{x;n}$` is the rank of the data sample `$X_n$`. -- As an example, with `$N=3$` `$$ \begin{eqnarray} X_n & \quad R_{x;n} \quad & Y_n \quad & R_{y;n} \quad & R_{x;n} - R_{y;n} \cr 4 & 3 & 1 & 1 & 2 \cr -1 & 1 & 12 & 3 & -2 \cr 3 & 2 & 3 & 2 & 0 \cr \end{eqnarray} $$` -- `$r_s$` measures if the relationship between `$x$` and `$y$` is monotonically increasing (`$r_s$>0`) or decreasing (`$r_s<0$`). For large `$N$`, `$r_s \approx \mathcal{N}(0,1/\sqrt{N-1})$` under `$H_0: r_s = 0$`. --- class: left, .toc[[✧](#toc)] ## Pearson vs Spearman .center[
] --- class: left, .toc[[✧](#toc)] ## Covariance: multivariate case If you are dealing with more than two r.vs., let's say `$P$` variables `$x_1, x_2,\ldots,x_P$` for which you have `$n = 1,2,\ldots,N$` samples, you need to build the *covariance matrix* .center[ `$\mathbf{C}_{xx} \equiv \left[\matrix{C_{x_1x_1} & C_{x_1x_2} & \dots & C_{x_1x_P}\cr C_{x_2x_1} & C_{x_2x_2} & \dots & C_{x_2x_P}\cr \vdots & \vdots & \cdots & \vdots \cr C_{x_Px_1} & C_{x_Px_2} & \dots & C_{x_Px_P}\cr }\right]$` ] or the *correlation matrix* .center[ `$\mathbf{\rho}_{xx} \equiv \left[\matrix{\rho_{x_1x_1} & \rho_{x_1x_2} & \dots & \rho_{x_1x_P}\cr \rho_{x_2x_1} & \rho_{x_2x_2} & \dots & \rho_{x_2x_P}\cr \vdots & \vdots & \cdots & \vdots \cr \rho_{x_Px_1} & \rho_{x_Px_2} & \dots & \rho_{x_Px_P}\cr }\right]$` ] Methods to study these matrices will be covered in lecture 5 (Eigen techniques). --- class: left, .toc[[✧](#toc)] ## Covariance: multivariate case; estimate If you have `$P$` r.vs. `$x_1, x_2,\ldots,x_P$` for which you have `$n = 1,2,\ldots,N$` samples, typically arranged in a `$N \times P$` data matrix `$\mathbf{X} = \left[\mathbf{X}_1,\, \mathbf{X}_2,\, \cdots ,\,\mathbf{X}_P\right] =\left[\matrix{X_1(1) & X_2(1) & \dots & X_P(1)\cr X_1(2) & X_2(2) & \cdots & X_P(2)\cr \vdots & \vdots & \cdots & \vdots \cr X_1(N) & X_2(N) & \cdots & X_P(N)\cr}\right]$` The estimate of the covariance matrix is `$\widehat{\mathbf{C}}_{xx} = \frac{1}{N-1} \mathbf{X}^\mathrm{T}\mathbf{X} = \left[\matrix{\widehat{C}_{x_1x_1} & \widehat{C}_{x_1x_2} & \dots & \widehat{C}_{x_1x_P}\cr \widehat{C}_{x_2x_1} & \widehat{C}_{x_2x_2} & \dots & \widehat{C}_{x_2x_P}\cr \vdots & \vdots & \cdots & \vdots \cr \widehat{C}_{x_Px_1} & \widehat{C}_{x_Px_2} & \dots & \widehat{C}_{x_Px_P}\cr }\right]$` --- class: left, .toc[[✧](#toc)] ## Correlation: beware! It is not because you find a zero or non significant correlation between two r.vs. that there is no formal or dynamical relationship between the two! .center[
] --- class: left, .toc[[✧](#toc)] ## Correlation: beware! This animation shows the correlation coefficient between two sinusoid signals F and G as a function of phase lag. .center[
] Gif by
Divergentdata
-
Own work
,
CC BY-SA 4.0
,
Link
--- name: laggedcovariance class: center, middle, .toc[[✧](#toc)] # 2. Lagged Covariance & Correlation --- class: left, .toc[[✧](#toc)] ### Lagged covariance & correlation functions We now generalize the concept of covariance by considering two r.vs. for which the samples are ordered, maybe as a function of time `$t$` (or of space). In this case, the samples are realizations of time series. -- The covariance statistic presented earlier is a special case of the *(cross-)covariance function*, function of *lag* `$\tau$` `$$ C_{xy}(\tau) = E\{[x(t)-\mu_x][y(t+\tau)-\mu_y]\} $$` If `$y\equiv x$`, `$C_{xx}(\tau)$` is called the *auto-covariance function* of `$x$`. -- Note that here we have assumed that the population means `$\mu_x$` and `$\mu_y$` are constant with time, in which case it is said that `$x(t)$` and `$y(t)$` are stationary time series. --- class: left, .toc[[✧](#toc)] ### Lagged covariance & correlation functions In statistics and engineering sciences the names can be different. In particular, in Matlab, the following function is called the *cross-correlation function* `$$ R_{xy}(\tau) = E\{x(t) y(t+\tau)\} $$` which is similar to the covariance function but without subtracting the means. As such `$$ C_{xy}(\tau) = R_{xy}(\tau) - \mu_x \mu_y $$` -- With such naming convention, `$R_{xx}(\tau)$` is the *auto-correlation function*. --- class: left, .toc[[✧](#toc)] ### Lagged covariance & correlation functions The auto-covariance and auto-correlation functions are even functions of `$\tau$` (i.e. symmetric around `$0$`): `$$ \begin{eqnarray} C_{xx}(-\tau) & = & C_{xx}(\tau) \cr R_{xx}(-\tau) & = & R_{xx}(\tau) \cr \end{eqnarray} $$` The cross-covariance and cross-correlation functions are neither odd nor even, but satisfies `$$ \begin{eqnarray} C_{xy}(-\tau) & = & C_{yx}(\tau) \cr R_{xy}(-\tau) & = & R_{yx}(\tau) \cr \end{eqnarray} $$` -- Beware! Check the conventions of the softwares you use! In Matlab, the cross-correlation function is defined as `$E[x(t+\tau)y(t)] = R_{xy}(-\tau) = R_{yx}(\tau)$`. --- class: left, .toc[[✧](#toc)] ## Lagged correlation coefficient The *lagged correlation coefficient* is `$$ \rho_{xy}(\tau) = \frac{C_{xy}(\tau)}{\sqrt{C_{xx}(0) C_{yy}(0)}} = \frac{C_{xy}(\tau)}{\sigma_x \sigma_y} $$` -- It can be challenging but let's try not to confuse correlation function and lagged correlation. -- You may want to call `$\rho_{xx}(\tau) = \frac{C_{xx}(\tau)}{C_{xx}(0)}$` the *lagged auto-correlation coefficient*, but it is usually called the autocorrelation function (which should be used for `$R_{xx}(\tau)$`). -- Distributions under null hypotheses for `$\rho_{xy}(\tau)$` or `$\rho_{xx}(\tau)$` are difficult to come by and require to make several assumptions (see practical this afternoon). --- class: left, .toc[[✧](#toc)] ## Lagged covariance function estimates Consider two r.vs. `$x$` and `$y$` for which we have `$N$` samples separated by constant intervals `$\Delta t$`. One estimate of the cross-covariance function at lags `$\tau_k = k \Delta t$` is `$$ \widehat{C}^{(1)}_{xy}(k\Delta t) = \frac{1}{N-|k|} \sum^{N-|k|}_{n=1} (X_n - \overline{X})(Y_{n+k} - \overline{Y}) $$` where `$k = 0,\pm 1,\pm 2,\ldots,\pm (N-1)$` are the possible lags. It can be shown that this estimator is unbiased (if the population means are known rather than estimated). -- There exists another estimator of `$C_{xy}(\tau)$` which is `$$ \widehat{C}^{(2)}_{xy}(k\Delta t) = \frac{1}{N} \sum^{N-|k|}_{n=1} (X_n - \overline{X})(Y_{n+k} - \overline{Y}) $$` which is typically called the "biased" estimator but has a smaller random error compared to `$\widehat{C}^{(1)}_{xy}$`. See practical this afternoon. --- class: left, .toc[[✧](#toc)] ## Lagged correlation estimate An estimate of the lagged correlation coefficient is `$$ \widehat{\rho}_{xy}(k\Delta t) = \frac{\widehat{C}^{(2)}_{xy}(k\Delta t)}{\left[\widehat{C}^{(2)}_{xx}\widehat{C}^{(2)}_{yy}\right]^{1/2}} = \frac{\sum^{N-|k|}_{n=1} (X_n - \overline{X})(Y_{n+k} - \overline{Y})}{\left[\sum_{n=1}^N (X_n-\overline{X})^2 \sum_{n=1}^N (Y_n-\overline{Y})^2 \right]^{1/2}} $$` where the use of `$\widehat{C}^{(2)}_{xy}$` cancels out the factor `$N$` in the formulas for the variances. This estimator is accessed in Matlab using the scale option `$\texttt{'coeff'}$` as in ```ruby [rhox,lags] = xcov(Y,X,'coeff'); ``` or ```ruby [rhox,lags] = xcorr(Y-mean(Y),X-mean(X),'coeff'); ``` --- class: center, .toc[[✧](#toc)] ### Auto-correlation as a measure of memory Lag `$m$` scatter plots and cor. coeff. `$\rho(X_n,X_{n+m})$` for Agulhas transport
--- class: center, .toc[[✧](#toc)] ### Auto-correlation as a measure of memory Lagged autocorrelation coefficient for Agulhas transport time series. With `$N=808$`, possible lags are `$\tau = -807,\ldots,807$`
--- class: center, .toc[[✧](#toc)] ### Auto-correlation as a measure of memory Lagged autocorrelation coefficient for Agulhas transport time series. With `$N=808$`, possible lags are `$\tau = -807,\ldots,807$`
--- class: left, .toc[[✧](#toc)] ### Lagged correlation for time delay The lagged correlation or covariance function can be used to determine a time delay between two signals. Let's assume that a transmitted signal (a time series) `$x(t)$` is received as `$$ y(t) = \alpha x(t-\tau_0) + n(t) $$` where `$\alpha$` is an *attenuation factor*, `$\tau_0 = d/c$` is a constant time delay equal to let's say a distance `$d$` divided by a propagation velocity `$c$`, and `$n(t)$` is an added noise uncorrelated with `$x(t)$`. -- It is easily shown that `$$ R_{xy}(\tau) = \alpha R_{xx}(\tau-\tau_0) $$` -- Since the maximum of `$R_{xx}(\tau)$` is for `$\tau=0$` , the peak value of `$R_{xy}(\tau)$` occurs when `$\tau=\tau_0$` which allows us to determine the constant time delay. --- class: center, .toc[[✧](#toc)] ## Lagged correlation for time delay
--- class: center, .toc[[✧](#toc)] ## Lagged correlation for time delay .left-column[
] .right-column[
Example from [DiNezio et al. 2009](http://dx.doi.org/10.1175/2008JPO4001.1) investigating the relationship between Gulf Stream transport through the Florida Strait and wind stress curl. ] --- class: left, .toc[[✧](#toc)] ### Auto-correlation and effective degrees of freedom The concept of *effective degrees of freedom* and *(de)correlation time scale* are intimately linked. If `$N$` is the number of evenly distributed samples at interval `$\Delta t$`, then the number of *effective degrees of freedom* is `$$ N_{eff} = \frac{N\Delta t}{T_0} = \frac{T}{T_0} $$` where `$T$` is the length of your time series and `$T_0$` is the *decorrelation time scale*, also referred to as an *integral time scale*. `$N_{eff}$` gives you the number of effectively independent samples in your data. -- If your samples are independent, the decorrelation time scale is the time step of your time series `$\Delta t$` and `$N_{eff} = N$`. There is not only one way of computing `$N_{eff}$` or estimating `$T_0$`. In fact, it depends on the statistics for which these will be used. --- class: left, .toc[[✧](#toc)] ### Decorrelation time scale One common definition of *the decorrelation or integral time scale* for the r.v. `$x$` is (e.g. Thompson and Emery (2014), section 3.15.2) `$$ T_0 = \frac{1}{C_{xx}(0)}\int_{-\infty}^{+\infty} C_{xx}(\tau)\,d\tau = \frac{2}{C_{xx}(0)}\int_{0}^{+\infty} C_{xx}(\tau)\,d\tau $$` A graphical interpretation of `$T_0$` is given in Figure 3.13 of Emery and Thompson (2014) : `$T_0 \times {C_{xx}(0)} = \int_{-\infty}^{+\infty} C_{xx}(\tau)\,d\tau$` .center[
] --- class: left, .toc[[✧](#toc)] ### Decorrelation time scale An estimate of `$T_0$` is obtained by applying the trapezoidal integration formula for the integral, i.e. `$$ \widehat{T_0} = \frac{2}{s^2_x} \sum_{n=0}^{M-1} \frac{\widehat{C}_{xx}[(n+1)\Delta t]+\widehat{C}_{xx}[n\Delta t]}{2} \Delta t $$` where `$s^2_x = \widehat{C}_{xx}(0)$` is the sample variance estimate of `$x$` and `$M$` is the index where the summation is stopped. -- Note that Thompson and Emery (2014), section 5.3.5, give an alternate formula `$$ \widehat{T_0} = \frac{1}{s^2_x} \sum_{n=0}^{M-1} \frac{\widehat{C}_{xx}[(n+1)\Delta t]+\widehat{C}_{xx}[n\Delta t]}{2} \Delta t $$` where they omitted the factor `$2$` and essentially integrated only one side of the auto-covariance function. This is also the formula typically used in Lagrangian studies. (And this is what I use). --- class: left, .toc[[✧](#toc)] ### Decorrelation time scale There are several sources of error entering formulas for `$T_0$`: estimate of the variance, estimate of the autocovariance, and truncation of the trapezoidal integration. The truncation of the integration may be where the estimated autocovariance function reaches a constant value but this may not happen in practice. Another possibility is to integrate up to the first zero crossing or up to where the autocorrelation is insignificant. See practical session this afternoon. .center[
] --- class: left, .toc[[✧](#toc)] ### Auto-correlation and effective degrees of freedom Once you have obtained an integral time scale `$T_0$` and calculated the effective degrees of freedom `$N_{eff}$`, you may use this parameter to calculate CIs for sample means. -- The formulas given previously do not seem to be appropriate when `$N_{eff}$` is needed to assess estimators of auto-covariance/correlation or cross-covariance/correlation. In these cases, Von Storch and Zwiers (1999), reference [[6]](#references), gives the respective two formulas `$$ \begin{eqnarray} \frac{T_0}{\Delta t} & = & 1 + 2\sum_{k=1}^{+\infty}\rho^2_{xx}(k\Delta t)\cr \frac{T_0}{\Delta t} & = & 1 + 2\sum_{k=1}^{+\infty}\rho_{xx}(k\Delta t)\rho_{yy}(k\Delta t)\cr \end{eqnarray} $$` assuming you have a constant time step `$\Delta t$` for your time series. --- name: bootstrap class: middle, center, .toc[[✧](#toc)] ## 3. A quick look at "A leisurely look at the bootstrap, the jackknife, and cross-validation" by Efron and Gong (1983) --- class: left, .toc[[✧](#toc)] ## The bootstrap and the jackknife So far, we have used theoretical or assumed distributions of our data in order to derive standard errors and CIs of our parameter estimates. However, in many cases, like for the correlation coefficient, it is impossible to express the variance in closed form. In addition, we are often stuck with one sample `$(X_1,X_2,\ldots,X_n)$` from one experiment in order to investigate the `$x$` population. On top of this, our samples may not be independent and we run into the issue of estimating the effective degrees of freedom. -- The idea of the *bootstrap* and *jackknife*, combined with Monte Carlo methods, is to resample your original sample with *replacements for the bootstrap*, and with *deletions for the jackknife*. These methods are relatively easy to implement with our fast and modern computers. --- class: left, .toc[[✧](#toc)] ## The bootstrap : principle (1) Let's assume you are investigating a univariate or multivariate r.v. `$x$` for which you want to estimate a statistic `$\phi$` with estimator `$\widehat{\phi}$`, using a sample `$(X_1,X_2,\ldots,X_n)$`. -- First, you draw randomly a *bootstrap sample* `$(X^*_1,X^*_2,\ldots,X^*_N)$` of the same size as your original sample. As an example, if your original sample if of size $N=3$, i.e. `$(X_1,X_2,X_3)$`, a bootstrap sample with replacement may be `$(X^*_1,X^*_2,X^*_3) = (X_1,X_2,X_1)$`. -- From the bootstrap sample, you calculate a bootstrap replication estimate of your statistic `$\widehat{\phi}^*$`. -- You repeat this operation `$B$` times to obtain `$B$` bootstrap replications `${\widehat{\phi}^*}^{(1)},{\widehat{\phi}^*}^{(2)},\ldots,{\widehat{\phi}^*}^{(B)}$`. -- You finally calculate (estimate) the variance of your replications as `$$ \text{Var}[\widehat{\phi}^*] = \frac{1}{B-1}\sum_{b=1}^B ({\widehat{\phi}^*}^{(b)} - {\widehat{\phi}^*}^{(.)})^2 \quad \text{where} \quad {\widehat{\phi}^*}^{(.)} = \frac{\sum_b {\widehat{\phi}^*}^{(b)}}{B} $$` --- class: left, .toc[[✧](#toc)] ## The bootstrap : principle (2) You can now use `$\text{Var}[\widehat{\phi}^*]$` to derive a standard error of your estimate as `$\sqrt{\text{Var}[\widehat{\phi}^*]}$` or use the estimated distribution of your bootstrap replications `${\widehat{\phi}^*}^{(1)},{\widehat{\phi}^*}^{(2)},\ldots,{\widehat{\phi}^*}^{(B)}$` to calculate CIs. .underline[Example] : Bootstrap correlations between between Agulhas jet and boundary transports with `$B=1000$`. .center[
] --- class: left, .toc[[✧](#toc)] ## The jackknife : principle (1) Let's assume again that you are investigating a univariate or multivariate r.v. `$x$` for which you want to estimate a statistic `$\phi$` with estimator `$\widehat{\phi}$`, using a sample `$(X_1,X_2,\ldots,X_n)$`. -- A *jackknife sample* is obtained by deleting `$J$` data points from your original sample of `$N$` points. The number of such sample that can be obtained is the number of permutations of `$N$` objects taken `$J$` at a time which is `$N!/(N-J)!$`. If you have `$N$` data points there are `$N$` permutations for the "delete-1" jackknife. The number of of possible jackknife samples can become very large so that a subset need to be choosen. -- The formula for the "delete-1" jackknife variance of your statistic estimate is `$$ \text{Var}[\widehat{\phi}^J] = \frac{N-1}{N}\sum_{j=1}^N ({\widehat{\phi}^*}^{(j)} - {\widehat{\phi}^*}^{(.)})^2 \quad \text{where} \quad {\widehat{\phi}^*}^{(.)} = \frac{\sum_j {\widehat{\phi}^*}^{(j)}}{N} $$` --- class: left, .toc[[✧](#toc)] ## The jackknife : principle (2) The jackknife method can be also used to estimate the bias of your estimator as `$$ \widehat{b}_J[\widehat{\phi}] = (N-1)({\widehat{\phi}^*}^{(.)} - \widehat{\phi}) $$` so that the *jackknife estimate* of your statistic is `$$ \widehat{\phi}_J = N\widehat{\phi} - (N-1){\widehat{\phi}^*}^{(.)} $$` -- There are some important details to the bootstrap and the jackknife methods. Please see [Thompson and Emery](#references) , [Tichelaar and Ruff (1989)]([http://dx.doi.org/10.1029/89EO00156) and [Efron and Gong (1983)](http://dx.doi.org/10.1080/00031305.1983.10483087). --- name: bivariate class: middle, center, .toc[[✧](#toc)] # 4. Covariance and correlation of bivariate variables --- class: left, .toc[[✧](#toc)] ## Bivariate random variable What if your process of interest, or your data, is a "vector variable"? It could be an ocean current, an atmospheric wind, the position of a drifter etc. Such variables are called *bivariate* variables in the statistics litterature, and are also treated as r.vs. -- Here we will use the formalism of time series, that is we assume that we have samples indexed along an axis `$t$` that represents time. -- Let's consider a first bivariate variable `$z(t)$` with Cartesian components `$x(t)$` and `$y(t)$` (east-west and north-south). -- We can arrange the components into a `$1 \times 2$` vector function of time `$$ \mathbf{z}(t) = \left[ \begin{matrix} x(t) & y(t) \end{matrix}\right] \quad \text{or} \quad \mathbf{z}_n = \begin{bmatrix} x_n & y_n \end{bmatrix} $$` or alternatively use a complex-valued notation `\[ z(t) = x(t) + i y(t) = |z(t)| e^{i \arg{(z)}},\]` where `$i = \sqrt{-1}$` and `$\arg{(z)}$` is the *complex argument* (or polar angle) of `$z$` in the interval `$[-\pi,+\pi]$`. --- class: left, .toc[[✧](#toc)] ## The mean of bivariate Data The sample mean of the vector time series `$\mathbf{z}(t)$` is also a vector, `$$ \begin{eqnarray} \widehat{\mu}_\mathbf{z} & = & \overline{\mathbf{z}} \equiv \frac{1}{N}\sum_{n=1}^{N}\mathbf{z}_n = \begin{bmatrix} \frac{1}{N}\sum_{n=1}^{N}{x}_n & \frac{1}{N}\sum_{n=1}^{N}{y}_n \end{bmatrix}\cr & = & \begin{bmatrix} \overline{x} & \overline{y} \end{bmatrix} = \begin{bmatrix} \widehat{\mu}_x & \widehat{\mu}_y \end{bmatrix} \cr \end{eqnarray} $$` that consists of the *sample means* of the `$x$` and `$y$` components of `$z$`. -- Using the complex notation, the sample mean is `$$ \overline{z} = \overline{x} + i\,\overline{y} $$` which is complex number. --- class: left, .toc[[✧](#toc)] ## The variance of bivariate variable The *variance* of the vector-valued times series `$\mathbf{z}_n$` is not a scalar or a vector, it is a `$2\times 2$` *matrix* `$$ \mathbf{\Sigma}_{z} \equiv \frac{1}{N-1}\sum_{n=1}^{N}\left(\mathbf{z}_n-\overline{\mathbf{z}}\right)^T\left(\mathbf{z}_n-\overline{\mathbf{z}}\right) $$` where “`$T$`” is the *matrix transpose*, `$\mathbf{z}_n =\begin{bmatrix} x_n & y_n \end{bmatrix}$`, `$\mathbf{z}_n^T =\begin{bmatrix} x_n \\ y_n \end{bmatrix}$`. Carrying out the matrix multiplication leads to `$$ \mathbf{\Sigma}_{z} = \frac{1}{N-1}\sum_{n=1}^{N} \begin{bmatrix} \left(x_n-\overline{x}\right)^2 & \left(x_n-\overline{x}\right)\left(y_n-\overline{y}\right)\\ \left(y_n-\overline{y}\right)\left(x_n-\overline{x}\right) &\left(y_n-\overline{y}\right)^2 \end{bmatrix} = \begin{bmatrix} s^2_x & s_{xy} \\ s_{yx} & s^2_{y} \end{bmatrix} $$` The diagonal elements of `$\mathbf{\Sigma}_{z}$` are the sample variances, while the off-diagonal gives the sample covariance between `$x_n$` and `$y_n$`. Note that the two off-diagonal elements are identical, `$s_{yx}=s_{xy}$`. --- class: left, .toc[[✧](#toc)] ## The variance of bivariate variable The variance of a complex r.v., `$z = x + i y$`, needs a definition: `$$ \text{Var}[z] = E[(z - \mu_z)^*(z - \mu_z)] $$` where `$(.)^*$` means the *complex conjuguate*, i.e. `$z^* = x - iy$`. -- Substituting and expanding the previous expression gives `$$ \begin{eqnarray} \text{Var}[z] & = & E[(x-\mu_x)^2] + E[(y-\mu_y)^2] \cr & = & \text{Var}[x] + \text{Var}[y] \end{eqnarray} $$` The variance of a "physical vector" written as a complex r.v. is a single **real** number which is the sum of the variance of its components. It is different from the `$2 \times 2$` matrix of variances and covariances of its components seen in the previous slide. -- If `$z$` represents an ocean current, `$\text{Var}[z]$` is proportional to the average eddy kinetic energy (density) `$KE = (
+
)/2$` --- class: left, .toc[[✧](#toc)] ## The covariance of bivariate variables In addition to `$z(t)$`, let's consider a second bivariate variable `$w(t)$` with Cartesian components `$g(t)$` and `$h(t)$`. -- The cross covariance function at zero lag, or simply the covariance between the components of `$z$` and `$w$` can be formed as `$$ \begin{eqnarray} E\left\{ \mathbf{z}^\mathrm{T}\mathbf{w} \right\} & = & E\left\{ \begin{bmatrix} (x - \mu_x)(g-\mu_g) & (x-\mu_x)(h-\mu_h) \\ (y-\mu_y)(g-\mu_g) & (y-\mu_y)(h-\mu_h) \end{bmatrix}\right\} \\ & = & \begin{bmatrix} C_{xg} & C_{xh} \\ C_{yg} & C_{yh} \end{bmatrix} \end{eqnarray} $$` -- The meaning of each individual entry of this matrix is obvious but the interpretation of all of them at once may be less so. --- class: left, .toc[[✧](#toc)] ## The covariance of bivariate variables Alternatively, let's use the complex representations of `$z$` and `$w$` `$$ \begin{eqnarray} z & = & x + i y \cr w & = & g + i h \cr \end{eqnarray} $$` but assume for simplicity that `$\mu_z = \mu_w = 0$`. -- A definition of the covariance between these two complex r.v. is `$$ \begin{eqnarray} C_{z w} & = & E[ z^* w ] \\ & = & E[ (x - i y)(g + i h) ] \\ & = & E[x g] + E[y h] + i( E[ x h]+ E[-y g] ). \end{eqnarray} $$` -- Beware that the definition of the covariance between complex r.v. may differ. Sometimes it is defined as `$E[ z w^* ]$`. Our convention here is the same as Matlab. --- class: left, .toc[[✧](#toc)] ## The covariance of bivariate variables Simple geometry reveals that the real part of the complex covariance is the expectation of the dot product [or inner product, noted `$(\boldsymbol{\cdot})$`] while the imaginary part is the expectation of the magnitude of the vector cross product [or outer product, noted `$(\boldsymbol{\times})$`]: `$$ \begin{eqnarray} C_{z w} & = & E[x g] + E[y h] + i( E[ x h]+ E[-y g] )\\ & = & E[\mathbf{z} \boldsymbol{\cdot} \mathbf{w}] + iE[||\mathbf{z} \boldsymbol{\times} \mathbf{w}||] \\ & = & E[|z||w| \cos \theta ] + i E[|z||w| \sin \theta] \end{eqnarray} $$` where `$\theta$` is the instantaneous geometric angle in the Cartesian plane between `$\mathbf{z}$` and `$\mathbf{w}$`, measured positively counterclockwise relative to the geometric angle of `$\mathbf{z}$`. --- class: left, .toc[[✧](#toc)] ## The covariance of bivariate variables `$$ C_{z w} = E[|z||w| \cos \theta ] + i E[|z||w| \sin \theta] $$` .center[
] --- class: left, .toc[[✧](#toc)] ## The covariance of bivariate variables `$$ C_{zw} = E[|z||w| \cos \theta ] + i E[|z||w| \sin \theta] $$` The covariance of `$z$` and `$w$` is a complex number which argument, or phase, is `$$ \text{Arg}[C_{z w}] = \text{atan} \left\{ \frac{E[|z||w| \sin \theta]}{E[|z||w| \cos \theta ]} \right\} \neq E[\theta]. $$` -- The absolute value of `$C_{z w}$` is clearly a measure of the covariance between the two bivariate variables, but its phase is not `$E[\theta(t)]$`: it is not the expectation, or mean, of the geometric angle between `$\mathbf{z}$` and `$\mathbf{w}$`. -- However, `$\text{Arg}[C_{z w}]$` can still be seen as an indication of the relative angle of covariance. If `$\text{Arg}[C_{z w}] = 0$` this indicates that the covariance occurs with $\mathbf{z}$ and $\mathbf{w}$ aligned and pointing in the same direction, and if `$\text{Arg}[C_{z w}] = \pi/2$` this indicates that the covariance occurs with `$\mathbf{w}$` at right angle counterclockwise from `$\mathbf{z}$`. --- class: left, .toc[[✧](#toc)] ## The correlation of bivariate variables From the complex covariance, we can define the complex correlation between `$\mathbf{z}$` and `$\mathbf{w}$` as `$$ \rho_{z w} \equiv \frac{C_{z w}}{\sqrt{C_{z z} C_{w w}}} = |\rho_{z w}|e^{i\text{Arg}[\rho_{z w}]} $$` Since `$C_{z z}$` and `$C_{w w}$` are real numbers, the phase of `$\rho_{z w}$` is the same phase as the phase of `$C_{z w}$`: `$\text{Arg}[\rho_{z w}] = \text{Arg}[C_{z w}]$` -- Note that the "vector" regression model for `$w$` based on the covariance between `$z$` and `$w$` is (see Lecture 3) `$$ w_m(t) = \frac{C_{z w}}{C_{z z}}z(t) = \frac{|C_{z w}|}{C_{z z}}e^{i\text{Arg}[C_{z w}]} z(t) $$` which shows that the regressed vector `$w_m(t)$` is rotated counterclockwise by `$\text{Arg}[C_{z w}]$` from the direction of `$z(t)$`. This is valid after removing the means. --- class: left, .toc[[✧](#toc)] ## The correlation of bivariate variables .left-column[
Panel c is the correlation `$\rho$` between the 10-m wind at 170E, 0N and 10-m winds at all other locations from ECMWF reanalyses. The real part of `$\rho$` is shown in panel a and the imaginary part is shown in panel b as a pure imaginary number so that c = a + b.] .righ-column[
] --- class: left, .toc[[✧](#toc)] ## The complementary of a vector The *complementary of a vector* `$\mathbf{z}$`, noted `$\mathbf{z}^c$`, is, using complex notation `$$ z^c \equiv z^* = x - i y $$` -- Geometrically, it is the vector `$\mathbf{z}$` flipped with respect to the real axis. If a bivariate time series represents a vector rotating in one direction, then its complementary is rotating in the opposite direction. .center[
] --- class: left, .toc[[✧](#toc)] ## The complementary correlation of bivariate variables The *complementary covariance* between `$\mathbf{z}$` and `$\mathbf{w}$` is `$$ \begin{eqnarray} C_{z^c w} & = & E[ (z^*)^* w ] \\ & = & E[ (x + i y)(g + i h) ] \\ & = & E[x g] - E[y h] + i( E[ x h]+ E[y g] \end{eqnarray} $$` The *complementary or reflectional correlation* is `$$ \rho_{z^c w} \equiv \frac{C_{z^c w}}{\sqrt{C_{z^c z^c} C_{w w}}} $$` where `$C_{z^c z^c} = C_{z z}$` --- class: left, .toc[[✧](#toc)] ## Auto-covariance and variance ellipses The special cases of standard and complementary auto covariances of a physical vector with itself can be related to the concept of *variance or standard deviation ellipses*. -- For a bivariate r.v. `$z$`, the fixed angle `$\theta_M$` between the real axis positive direction and the so-called major axis of the standard deviation ellipse is `$$ \theta_M = \frac{1}{2}\text{Arg}\left[ C_{z^c z}\right] = \frac{1}{2}\arctan\,\left[ \frac{2E[xy]}{E[x^2] - E[y^2]} \right], $$` while `$\theta_M+\pi/2$` defines the direction of the so-called minor axis. --- class: left, .toc[[✧](#toc)] ## Auto-covariance and variance ellipses The variances `$a^2_v$` and `$b^2_v$` of the bivariate variable along the major and minor axes, respectively, are given by `$$ \begin{eqnarray} a^2_v & = & \frac{1}{2}\left[ C_{z z} + |C_{z^c z}|\right]\\ & = & \frac{1}{2} \left\{ E[x^2] + E[y^2] + \sqrt{(E[x^2] - E[y^2])^2 + 4(E[xy])^2} \right\}, \end{eqnarray} $$` `$$ \begin{eqnarray} b^2_v & = & \frac{1}{2}\left[C_{z z} - |C_{z^c z}|\right]\\ & = & \frac{1}{2} \left\{ E[x^2] + E[y^2] - \sqrt{(E[x^2] - E{y^2})^2 + 4(E[xy])^2} \right\}. \end{eqnarray} $$` Anticipating Lecture 5, `$a^2_v$` and `$b^2_v$` are the 2 eigen values of the `$2 \times 2$` cross covariance matrix of the Cartesian component, while the associated eigen vectors are `$[\cos \theta_M,\, \sin \theta_M]$` and `$[-\sin \theta_M,\, \cos \theta_M]$`. --- class: left, .toc[[✧](#toc)] ## Auto-covariance and variance ellipses The ratio of the absolute value of the complementary auto correlation to the absolute value of the auto correlation is a measure of the absolute *linearity* of the standard deviation ellipse `$$ |\lambda| = \frac{C_{z^c z}}{|C_{zz}|} = \frac{a_v^2-b_v^2}{a_v^2+b_v^2}. $$` If `$|\lambda| = 1$` the ellipse is flat and if `$|\lambda| = 0$` the ellipse is a circle. `$\lambda$` is a relative of the more commonly known eccentricity parameter of the ellipse which is `$$ e = \sqrt{1-\frac{b_v^2}{a_v^2}}. $$` See [Lilly and Gascard (2006)](http://www.nonlin-processes-geophys.net/13/467/2006/) and [Lilly J, Olhede S. (2010)](http://dx.doi.org/10.1109/TSP.2009.2031729). --- class: left, .toc[[✧](#toc)] ## Standard deviation or variance ellipses If your complex signal is `$z(t) = e^{i\theta}[a \cos(t) + i \sin(t)]$` which represents an ellipse, then the major axis and minor semi-axis of the standard deviation ellipse are `$a_v = \sqrt{a^2/2}
] --- class: left, .toc[[✧](#toc)] ## Standard deviation or variance ellipses .underline[Example]: Mean currents (blue) and standard deviation ellipses (light blue) of near bottom currents from the RAPID WAVE experiment. Orange axes show topography gradient. See [Hughes et al. 2013](http://dx.doi.org/10.1175/JTECH-D-12-00149.1) .center[
] --- class: left, .toc[[✧](#toc)] ## Standard deviation or variance ellipses .underline[Example]: Complementary auto correlation of 10-m winds at all locations from ECMWF reanalyses. .center[
] --- class: left, .toc[[✧](#toc)] ## Standard deviation or variance ellipses The standard ellipse is a statistical description of the variance of a bivariate variable and does not mean that any underlying variability is actually elliptical in time. Standard deviation ellipses can be computed from a number of individual pairs of Cartesian components of a bivariate quantity without these points actually forming consecutive time series. -- .underline[Example]: Variance ellipses from drifters from [Lumpkin and Johnson 2013](http://dx.doi.org/10.1002/jgrc.20210) .center[
] --- class: left, .toc[[✧](#toc)] ## Practical session Please download data at the following link: Please download the Matlab code at the following link: Make sure you have installed and tested the free jLab Matlab toolbox from Jonathan Lilly at [www.jmlilly.net/jmlsoft.html](https://www.jmlilly.net/jmlsoft.html) --- exclude: true class: left, .toc[[✧](#toc)] ```ruby plot(t,x) ``` First Header | Second Header ------------ | ------------- Content from cell 1 | Content from cell 2 Content in the first column | Content in the second column ~~this~~


Use this command to place the current selection on the clipboard and delete it from the file. Cutting data to the clipboard replaces the contents previously stored there.
This command is unavailable if the file is read only, or there is no data currently selected. If the window is in overtype mode and/or read only mode you cannot cut -- HexEdit prompts you to turn off these modes. Alternatively, you can copy to the clipboard using the Copy command which will not modify the file.
HexEdit stores the data to the clipboard in two formats. The first format is a binary format that allows non-text characters (such as character zero) to be cut and pasted within HexEdit. It is also the same format that the Microsoft Developer Studio uses in its binary editor.
HexEdit also cuts the bytes to the clipboard as ASCII text, which allows other text-based programs to retrieve them. However, in this case some non-text characters (in particular, character zero) are lost. If HexEdit’s character display is currently EBCDIC then the characters are converted from EBCDIC to ASCII before being placed on the clipboard as text but any invalid EBCDIC characters are ignored.
Shortcuts
Toolbar:
Keys: Ctrl+X


TABLE OF CONTENTS
A. GS1 Related B. GS2 Related POLITY 1. Give verdict copy in the vernacular: Kovind 2. Before polls were called, Gujarat pushed terror Bill INTERNATIONAL AFFAIRS/BILATERAL RELATIONS 1. India turns down Pak. offer of talks on transit trade to Afghanistan HEALTH ISSUES 1. Introduce eggs, milk in mid-day meal, ICDS diet: NHRC C. GS3 Related ECONOMICS 1. India to leapfrog 30 places in business ease rank ENVIRONMENT AND ECOLOGY 1. Four Asian vulture species now on highest protection list D. GS4 Related E. Prelims Fact F. UPSC Prelims Practice Questions G. UPSC Mains Practice Questions
Nothing here for Today!!!
1. Give verdict copy in the vernacular: Kovind
In news:
- President Ram Nath Kovind suggested that a system be evolved for providing certified copies of High Court judgments translated in local or regional languages of the litigants.
- Copies of the translated judgments could be provided in 24 or 36 hours after the verdict was pronounced.
- The President pointed out that the High Courts delivered judgments in English, “but we are a country of diverse languages. The litigant may not be conversant with English and the finer points of the judgment may escape him or her. The litigating parties will thus be dependent on the lawyer or another person to translate the judgment. This can add to time and cost”.
- It was important not only to take justice to the people but also to make it understandable to the litigating parties in a language they know
2. Before polls were called, Gujarat pushed terror Bill
In news:
- Days before the Gujarat Assembly elections were announced, the State government renewed its bid to revive the controversial Gujarat Control of Terrorism and Organised Crime (GCTOC) Bill, 2015, which authorises police officers to tap phones and submit the conversations as evidence in court.
- The Bill includes provisions to file a charge sheet within 180 days instead of the usual 90 and proposes stricter conditions for granting bail.
Inter-ministerial review:
- The Union Home Ministry has sent the Bill for inter-ministerial consultations, the fifth time since it was first conceived in 2003 when Prime Minister Narendra Modi was the Chief Minister of Gujarat.
- Former President Pranab Mukherjee had refused to clear the Bill, following which the Home Ministry recalled it in January 2016.
- The Ministry asked the State government for clarifications, and Gujarat had sent a fresh proposal this September.
Earlier attempts:
- The earlier GUJCOC Bill was reintroduced as GCTOC Bill in 2015. The same year the Information and Technology Ministry objected to the provisions that allowed “authorisation of interception of telephonic conversations and their admissibility as evidence in the court of law”.
- As per norms, agencies such as the Research & Analysis Wing (RAW), the Directorate of Revenue Intelligence, the Enforcement Directorate and the police can tap phones only with the permission of the Union Home Secretary.
- The Bill in its earlier form was rejected thrice by two former Presidents — the late A.P.J. Abdul Kalam in 2004 and by Pratibha Patil in 2008 and 2009.
- One of the provisions to which objections were raised by the Centre was Section 25, which provided immunity to the State from any legal action for “anything which is in good faith done or intended to be done in pursuance of this Act.”
Category: INTERNATIONAL AFFAIRS/BILATERAL RELATIONS
1. India turns down Pak. offer of talks on transit trade to Afghanistan
In news:
- India has rejected an offer from Pakistan for talks on transit trade to Afghanistan.
- According to sources, the offer was made by the Pakistan Chief of Army Staff General Qamar Javed when he met President Ashraf Ghani on October 1 in Kabul.
- During discussions on the renewal of the Afghanistan-Pakistan Transit Trade Agreement (APTTA), that lapsed in 2015, President Ghani repeated concerns that trade with India over the Wagah border had been blocked by Pakistan, despite being agreed to in the APTTA.
- Days later, Indian officials at the SCO Afghanistan-Contact group meeting in Moscow, told the Afghan delegation that it would not take up the offer for talks.
- Pakistan has consistently refused to allow any Indian goods to travel over land Afghanistan, insisting that India use the sea-route via Karachi.
- Exasperated by Pakistan’s refusal to allow the truck trade, President Ghani has said repeatedly that he would cut off Pakistan’s access to Central Asia and issued a decree last week cancelling permission for Pakistani trucks to transit through Afghanistan.
New routes:
India and Afghanistan are now working on strengthening alternative routes, including the air cargo corridor launched in June this year, and the Chabahar sea route.
1. Introduce eggs, milk in mid-day meal, ICDS diet: NHRC
In news:
- Stressing that food being served under the mid-day meal scheme and Integrated Child Development Scheme (ICDS) do not guarantee the protein intake as envisaged, the National Human Rights Commission (NHRC) has recommended to state governments and the Ministry of Women and Child Development to introduce eggs and milk as part of the meal in all states.
- The recommendation was made during a conference organised by the NHRC on the implementation of the National Food Security Act, 2013 on Friday.
- Malnutrition continues to haunt India despite economic growth and the mid day meal scheme and the ICDS are the two biggest nutrition programmes.
- While the meal is supplied in schools, the ICDS aims at providing supplementary nutrition through anganwadi centres to children aged below 6 years, besides pregnant and lactating women.
The recommendations of the NHRC
- For the ICDS scheme, it has been recommended that adequate food testing facilities be created preferably at the district level by all states through the public health department, universities and colleges.
- For the meal scheme, it was recommended that social audit and evaluation by a third party be conducted annually for each district and the reports be published on state portals.
- It also said that no child should be deprived of the meal because of non-linking of Aadhaar.
- The NHRC also questioned the state governments whether they have strictly followed Supreme Court orders banning contractors in supplying nutrition under ICDS and also whether adequate funds have been made available for full implementation of the Maternity Benefit Scheme or not.
- Moreover, experts said cash transfers instead of hot cooked meals may be counterproductive and must not be encouraged as cash can be used for other expenditure.
- Citing Kerala and Tamil Nadu as examples, the states were asked to resolve infrastructural gaps in terms of buildings, kitchens, drinking water supply and toilets within three years, by sourcing funds from different schemes and departments.
1. India to leapfrog 30 places in business ease rank
In news:
- India will leapfrog 30 places to the 100th position out of 190 countries in the World Bank’s Doing Business Report, high-level sources have confirmed.
- The report is expected to be released on October 31 — “India will hit a century.”
Key Point:
- This huge jump in the country’s ranking is thanks to reforms in areas such as ‘starting a business’, ‘dealing with construction permits’, and ‘resolving insolvency’, where it was placed a lowly 155, 185 and 136 respectively last year.
- India was ranked a poor 130 overall last year, up by just one place from 131 the previous year.
- On the future prospects for India, the source said: If India maintains this momentum, it can jump to a rank in double digits next year,” adding that Mumbai and Delhi — the two cities covered in the Report — had responded well to the government’s call for improvement. The Department of Industrial Policy and Promotion systematically worked with the line ministries and State governments to “get things done on the ground.
- The World Bank’s ‘distance to frontier’ score — which “measures the distance of each economy to the ‘frontier’ that represents the best performance observed on each of the indicators across all economies in the Doing Business sample since 2005 – showed that India’s score gradually improved from 48.77 in 2010 to 55.27 in 2017.
Category: ENVIRONMENT AND ECOLOGY
1. Four Asian vulture species now on highest protection list
In news:
- Several species of vultures, including four that have India on their migratory routes, were awarded the highest protection by the Convention on the Conservation of Migratory Species of Wild Animals.
- The whale shark, which inhabits the Indian Ocean, got global protection too. However, the proposal to extend additional protection to the chinkara or Indian gazelle was withdrawn.
- The convention in Manila concluded with approvals for protection of 34 species in submissions made by 24 countries from Asia, Africa, the Americas, Europe and Oceania.
- The Asian vultures that are set to get collaborative international protection are the red-headed vulture, white-rumped vulture, Indian vulture and slender-billed vulture.
- Threats: They are faced with threats such as poisoning, hunting, collision with electricity cables and habitat degradation.
- A subspecies of the black noddy, the yellow bunting and the lesser and great grey shrike are the other avians on the protected list.
- Lions, chimpanzees, giraffes and leopards were marked out as species that needed additional protection.
Marine Species:
- Widespread over-fishing is driving many shark species, including the whale shark, to extinction. India is among 121 nations whose waters are home to sharks threatended with near extinction. The major threats are bycatch in nets and vessel strikes.
- Proposals for conservation of the blue shark and common guitarfish have also been accepted.
- A resolution to develop and manage protected area networks within the ASEAN region has been adopted.
- Proposals submitted by Mongolia to protect two of Central Asia’s rarest species, Przewalski’s horse and the Gobi bear, also got the nod.
- The Caspian seal has also been identified for conservation. It is the only marine mammal found in the world’s largest inland sea, where its migration is prompted by ice formation and foraging.
About the Convention on the Conservation of Migratory Species of Wild Animals:
- The Convention on the Conservation of Migratory Species of Wild Animals — more commonly abbreviated to just the Convention on Migratory Species (CMS) or the Bonn Convention — aims to conserve terrestrial, marine and avian migratory species throughout their range. It is an international treaty, concluded under the aegis of the United Nations Environment Programme, concerned with the conservation of wildlife and habitats on a global scale
- Since the Convention’s entry into force, its membership has grown steadily to include over 120 Parties from Africa, Central and South America, Asia, Europe and Oceania.
- The Convention was signed in 1979 in Bad Godesberg, a suburb of Bonn (hence the name), and entered into force in 1983. The depositary is the government of the Federal Republic of Germany.
- The CMS is the only global and UN-based intergovernmental organization established exclusively for the conservation and management of terrestrial, aquatic and avian migratory species throughout their range. CMS and its daughter agreements determine policy and provide further guidance on specific issues through their Strategic Plans, Action Plans, resolutions, decisions and guidelines. All maintain on their websites a list of all decisions taken, guidelines issues and Action Plans adopted by the Member States.
Nothing here for Today!!!
|
Nothing here for Today!!! |
F. Practice Questions for UPSC Prelims Exam
Question 1. Consider the following statement with reference to Brain corals
- Brain corals are slow-growing species of coral which often act as foundations for reefs.
- Brain corals do not possess biological brains
- Brain corals are found in Caribbean, Atlantic and Pacific Oceans
Identify the correct statement from the options given below
- 1 and 3
- 2 and 3
- 1 and 2
- All are correct
See
Question 2. Consider the following statements.
- Any ordinary law needs to be passed by both the Houses.
- Money bills need to be passed only by the Lok Sabha.
- In case of a disagreement between the two houses on ordinary bills, a joint sitting is called that is presided by the President.
Select the correct answer using the codes below
- 1 and 2 Only
- 1 and 3 only
- 1 only
- All are correct
See
Question 3. ‘Project Dantak’ refers to
- The construction of road and telecommunications network by India’s Border Roads Organisation, in Bhutan
- The Ministry of Health and Family Welfare’s programme to set up dental clinics through PPP mode in every district
- The Income Tax Department’s initiative to e-verify large cash deposits made during the window provided for depositing high-denomination currency notes with banks
- An India Post virtual museum showcasing all philatelic items associated with India’s Men in Uniform
Question 4. Consider the following statements with reference to the ‘Visiting Advanced Joint Research (VAJRA) Faculty Scheme’,
- It was launched by the Ministry of Human Resource Development
- It will offer adjunct/visiting faculty assignments to eligible candidates overseas including Non-resident Indians (NRI).
- Overseas Citizens of India (OCI) have been excluded from the purview of this scheme.
Select the correct answer using the codes below
- 1 and 2 only
- 1 and 3 only
- 3 only
- 2 only
Question 5. NNP at factor cost is
- GNP at market price – depreciation
- NNP at market price – Indirect taxes
- NNP at market prices+ subsidies
- NNP at market price- indirect taxes+ subsidies
Question 6. The Architectural Splendour of Hampi is located in which of these river basins?
- Godavari-Vaigai basin
- Cauvery-Penna basin
- Krishna-Tungabhadra basin
- Mahanadi-Bhima basin
G. UPSC Mains Practice Questions
GS Paper I
- “The Himalayas are highly prone to landslides.”Discuss the causes and suggest suitable measures of mitigation.
GS Paper IV
- What do you understand by the following terms in the context of public service? a) Integrity b) Perseverance c) Spirit of service d) Commitment e) Courage of conviction f) Personal opinion
Also, check previous Daily News Analysis
“Proper Current Affairs preparation is the key to success in the UPSC- Civil Services Examination. We have now launched a comprehensive ‘Current Affairs Webinar’. Limited seats available. Click here to Know More.”
Enroll for India’s Largest All-India Test Series


DelphiDabbler Code Snippets Database v2
Documentation
Contents
The
DelphiDabbler Code Snippets Database v2
collection is documented in the following files:
collection-format.html
Description of the format of all files in the collection.
Start here
.
source-code-format.html
Detailed description of how source code files are formatted for each type of snippet in the collection.
reml.html
Information about the REML mini markup language used to format snippet descriptions etc.


-
From the command prompt, cd into the application's directory.
-
Run skyux build to build the SPA into the dist/ folder.
-
Copy the contents of the dist/ folder into a directory with the SPA name that you specified when you created the SPA. For example, if you named the SPA my-spa when you ran skyux new, change the dist/ folder to my-spa/.
To host your SPA at the root without a subfolder, set the base property to / in the app configuration section in skyuxconfig.json.
-
Host the my-spa/ directory on your preferred web server.
-
In a browser, enter https://example.com/my-spa to view your SPA.
SKY UX uses HTML5 push states for routing, so you must either configure your host to redirect all requests to the root index.html or enable the useHashRouting configuration option.


Visualizing the
Base Rate Fallacy
Robert is a man of tiny stature from the United States who wears glasses and likes solving puzzles. He is more likely to be, (a) A truck driver, or (b) A computer science student at Stanford?
If your mind was drawn to the answer (b), this is the base rate fallacy at work here. You probably figured that the description of Robert wouldn’t fit in too well with your image or stereotype of truck drivers. But if you would to take a step back and think about the bigger picture - there are clearly way more truck drivers in the United States, than students in a certain university in California. Hence, it follows that Robert is more likely to be a truck driver!
The vivid and oddly specific description has led you to neglect any statistical or general information - what is known as base rate information (and hence the name of this cognitive error). Your mind did not attempt to incorporate any prior knowledge into your reasoning, as it had never occurred to be relevant, and simply embraced any causal explanations.
Judging someone's profession doesn't seem like much of a big deal. But what happens if our reliance on our intuitive judgments can potentially pose real-life consequences? Let's take a closer look at an example that you might be able to relate to - health screenings.  You are well aware that breast cancer is the most common cancer among women, and that an early diagnosis can increase the chance of recovery. So why not make that trip down to the clinic? Better to be safe than sorry, right? Plus, a mammography will only incorrectly detect breast cancer on a healthy person 9% of the time, which does seem pretty reliable.
So let's say you got your mammogram done, but unfortunately, you are surprised by a positive result. Here’s the important question: given the accuracy of the mammogram test, what is the probability that you actually have breast cancer?

Let’s visualize 1 dotfor every 1,000 women, for a total of 100 dots.
For the sake of example, let’s assume breast cancer affects 1% of the population.
The test is rather accurate; it always correctly identifies those who have the disease. However, roughly 9% of mammograms will generate a false positive, meaning that the person tests positive but does not actually have the disease.
Hence, the results of the screening are as follows: 1,000 women are correctly found to have breast cancer. But of the remaining 99,000 healthy ones, around 8,910 are incorrectly identified to have breast cancer.
Putting it all together, we can calculate the probability of an inaccurate positive result: a total of 9,910 women will be revealed to have breast cancer, but only 1,000 of them actually carry the disease. In other words, the probability of you having breast cancer is a mere 10%!
Despite the accuracy of the test, a bulk of the people who test positive in the mammogram examination do not actually have the disease! If you have tried experimenting with the sliders in the visualization, you might have noticed a trend - increasing the prevalence of the disease reduces the proportion of false positives.
Hence, what is responsible for this seemingly counter-intuitive phenomenon is simply our failure to recognize the base rate - which in this case, is the low rate of breast cancer in the population. No matter how reliable the test is, the rare occurrence of the disease in the population will always guarantee a sizeable proportion of false positive results.
This is important as mass screenings could result in overdiagnosis of patients and unnecessary invasive follow-up treatments like surgery and chemotherapy. That is why doctors only recommend health screenings on patients with high risk factors, such as age or a family history of the disease (i.e. increasing the base rate and hence the accuracy of the test).
In the medical profession, a common saying beaten into the heads of budding doctors is: "When you hear hoofbeats, think of horses not zebras." In the context of their practice, it cautions medical students not to make any rare diagnoses before considering the more common diseases. It's easy to fall prey to the base-rate fallacy (even two-thirds of doctors fail to judge accurately the efficacy of health screenings), but hopefully even a rudimentary understanding of it will help you arrive at proper judgments, conclusions and make smarter decisions.
So whether you are estimating the odds of success of your first business venture, the likelihood of your child getting into Havard, or the chance of landing a spot at Google with your computer science degree, be sure to ignore any specific information, and focus on the base rate - remember to hold your zebras, and utilize your horse sense instead.