stack.js | |
|---|---|
BOSSTell your JavaScript functions what to do. Copyright and Licensecopyright: (c) 2011 by Kris Walker (kris@kixx.name). Unless otherwise indicated, all source code is licensed under the MIT license. See MIT-LICENSE for details. Boss provides a task queue constructor which runs an array of asynchronous functions in serial. This is cool for test, build, and deployment scripts as well as simplifying complex serial operations in your JavaScript programs. InstallationInstallation in Node.js is pretty easy if you have npm installed.
And then in your program do
OR you could just clone this repo and drop it into a location of your choosing and then do
OR simply copy the
Whatever. You got the idea. | |
Code Docs | |
exports.createStack( functions, [context] )Creates a linked stack of functions. Params
Returns A function which accepts a callback function to be invoked when the stack is completed. Contract
| |
| Stack 'em up; starting from the end and moving to the front. Create the wrapped continuation function for each function in the array, and link it into the stack at the same time. When invoking the wrapped function, first, create the next() function to be passed to the next linked function in the stack. Each next() function privides the link which makes the returned stack a linked list of continuation functions. Inside the next() function, we force execution in the next turn of the event loop with process.nextTick(). Inside the next event loop, a gaurd is in place to be sure the callback is called once, and only once. Then we check for an error passed to next(), and if it exists, invoke the callback with the error and terminate the stack. After checking for an error, if we are not at the end of the stack, invoke the next linked function, but, if we are at the end of the stack, then invoke the callback instead. After the next() function is created, invoke the linked stack function, catching any uncaught errors and passing them to the callback, terminating the stack. | exports.createStack = function createStack(functions, context) {
var stack = null;
functions.reverse().forEach(function (fn) {
var continuation = stack;
stack = function (callback) {
var resolved = false;
function next(err) {
process.nextTick(function () {
if (resolved) { return; }
resolved = true;
if (err) { return callback.call(context, err); }
if (continuation) { return continuation(callback); }
callback.call(context);
});
}
try {
fn.call(context, next)
} catch (err) {
resolved = true;
callback.call(context, err)
}
};
});
return stack
}; |
UsageSimple example (but kinda pointless):
Async example (more useful)
Each function in the stack is guaranteed be called in a new turn of the event loop.
Error handling
RoadmapMaybe add constructors for 'Dict' and 'List' objects which would provide a "bottom up" way to run tasks in parallel and execute a callback when they all complete. | |


Volume:
Desktop: Left-click + drag to rotate, scroll to zoom,
right-click + drag to pan.
Touch: One finger drag to rotate, pinch to zoom, two finger drag to pan.
This is a WebGL viewer for Connectomics data, and can render both the microscopy acquired image volume in combination with a set of neuron traces. The DIADEM NC Layer 1 Axons and reference traces are the NC Layer 1 Axons dataset, courtesy De Paola et al. 2006, included as part of the DIADEM Challenge. Get the code on GitHub!
You can upload a TIFF stack or multi-page TIFF from your computer, or fetch a multi-page TIFF from a remote URL (e.g., Dropbox, Google Drive). When fetching a TIFF from a URL the remote site must support Cross-Origin Resource Sharing for this site to fetch the data. If fetching from a URL fails, you can download the file from the URL yourself and upload it. Supported TIFF Formats are single-channel 8 and 16bit unsigned int images. 16bit images may appear pixelated, as WebGL2 does not support filtering on these textures.


[name]
A loader for loading a JSON resource. Unlike the [page:JSONLoader], this one make use of the .type attributes of objects to map them to their original classes.
Example
// instantiate a loader
var loader = new THREE.ObjectLoader();
// assuming we loaded a JSON structure from elsewhere
var object = loader.parse( a_json_object );
scene.add( object );
[example:webgl_loader_msgpack]
Constructor
[name]( [page:LoadingManager manager] )
[page:LoadingManager manager] — The [page:LoadingManager loadingManager] for the loader to use. Default is [page:LoadingManager THREE.DefaultLoadingManager].
Creates a new [name].
Properties
Methods
[method:null load]( [page:String url], [page:Function onLoad], [page:Function onProgress], [page:Function onError] )
[page:String url] — required
[page:Function onLoad] — Will be called when load completes. The argument will be the loaded [page:Object3D object].
[page:Function onProgress] — Will be called while load progresses. The argument will be the XmlHttpRequest instance, that contain .[page:Integer total] and .[page:Integer loaded] bytes.
[page:Function onError] — Will be called when load errors.
Begin loading from url and call onLoad with the parsed response content.
[method:Object3D parse]( [page:Object json] )
[page:Object json] — required. The JSON source to parse
Parse a JSON content and return a threejs object.
[method:null setCrossOrigin]( [page:String value] )
[page:String value] — The crossOrigin string to implement CORS for loading the url from a different domain that allows CORS.
Source
[link:https://github.com/mrdoob/three.js/blob/master/src/[path].js src/[path].js]


{/'CD Execute bash command' | translate/} {/'CD docker pull image' | translate/} {/'Do not disclose this command' | translate/}


|
FlatBuffers
An open source project by FPL.
|
Responsible for building up and accessing a FlatBuffer formatted byte array (via ByteBuffer). More...
Responsible for building up and accessing a FlatBuffer formatted byte array (via ByteBuffer).
Public Member Functions | |
| void | Add< T > (T[] x) |
| Add an array of type T to the buffer (aligns the data and grows if necessary). More... | |
| void | Add< T > (Span< T > x) |
| Add a span of type T to the buffer (aligns the data and grows if necessary). More... | |
| void | AddBool (bool x) |
Add a bool to the buffer (aligns the data and grows if necessary). More... | |
| void | AddByte (byte x) |
Add a byte to the buffer (aligns the data and grows if necessary). More... | |
| void | AddDouble (double x) |
Add a double to the buffer (aligns the data and grows if necessary). More... | |
| void | AddFloat (float x) |
Add a float to the buffer (aligns the data and grows if necessary). More... | |
| void | AddInt (int x) |
Add an int to the buffer (aligns the data and grows if necessary). More... | |
| void | AddLong (long x) |
Add a long to the buffer (aligns the data and grows if necessary). More... | |
| void | AddOffset (int off) |
| Adds an offset, relative to where it will be written. More... | |
| void | AddSbyte (sbyte x) |
Add a sbyte to the buffer (aligns the data and grows if necessary). More... | |
| void | AddShort (short x) |
Add a short to the buffer (aligns the data and grows if necessary). More... | |
| void | AddUint (uint x) |
Add an uint to the buffer (aligns the data and grows if necessary). More... | |
| void | AddUlong (ulong x) |
Add an ulong to the buffer (aligns the data and grows if necessary). More... | |
| void | AddUshort (ushort x) |
Add an ushort to the buffer (aligns the data and grows if necessary). More... | |
| void | Clear () |
| Reset the FlatBufferBuilder by purging all data that it holds. More... | |
| StringOffset | CreateString (string s) |
Encode the string s in the buffer using UTF-8. More... | |
| StringOffset | CreateUTF8String (Span< byte > chars) |
| Creates a string in the buffer from a Span containing a UTF8 string. More... | |
| VectorOffset | CreateVectorOfTables< T > (Offset< T >[] offsets) |
| Creates a vector of tables. More... | |
| VectorOffset | EndVector () |
| Writes data necessary to finish a vector construction. More... | |
| void | Finish (int rootTable) |
Finalize a buffer, pointing to the given root_table . More... | |
| void | Finish (int rootTable, string fileIdentifier) |
Finalize a buffer, pointing to the given rootTable . More... | |
| void | FinishSizePrefixed (int rootTable) |
Finalize a buffer, pointing to the given root_table , with the size prefixed. More... | |
| void | FinishSizePrefixed (int rootTable, string fileIdentifier) |
Finalize a buffer, pointing to the given rootTable , with the size prefixed. More... | |
| FlatBufferBuilder (int initialSize) | |
| Create a FlatBufferBuilder with a given initial size. More... | |
| FlatBufferBuilder (ByteBuffer buffer) | |
| Create a FlatBufferBuilder backed by the pased in ByteBuffer More... | |
| byte [] | SizedByteArray () |
A utility function to copy and return the ByteBuffer data as a byte[] . More... | |
Protected Member Functions | |
| void | Finish (int rootTable, bool sizePrefix) |
Finalize a buffer, pointing to the given root_table . More... | |
| void | Finish (int rootTable, string fileIdentifier, bool sizePrefix) |
Finalize a buffer, pointing to the given rootTable . More... | |
Properties | |
| ByteBuffer | DataBuffer [get] |
| Get the ByteBuffer representing the FlatBuffer. More... | |
| bool | ForceDefaults [get, set] |
| Gets and sets a Boolean to disable the optimization when serializing default values to a Table. More... | |
|
inline |
Create a FlatBufferBuilder with a given initial size.
| initialSize | The initial size to use for the internal buffer. |
|
inline |
Create a FlatBufferBuilder backed by the pased in ByteBuffer
| buffer | The ByteBuffer to write to |
|
inline |
Add an array of type T to the buffer (aligns the data and grows if necessary).
| T | The type of the input data |
| x | The array to copy data from |
| T | : | struct |
|
inline |
Add a span of type T to the buffer (aligns the data and grows if necessary).
| T | The type of the input data |
| x | The span to copy data from |
| T | : | struct |
|
inline |
Add a bool
to the buffer (aligns the data and grows if necessary).
| x | The bool to add to the buffer. |
|
inline |
Add a byte
to the buffer (aligns the data and grows if necessary).
| x | The byte to add to the buffer. |
|
inline |
Add a double
to the buffer (aligns the data and grows if necessary).
| x | The double to add to the buffer. |
|
inline |
Add a float
to the buffer (aligns the data and grows if necessary).
| x | The float to add to the buffer. |
|
inline |
Add an int
to the buffer (aligns the data and grows if necessary).
| x | The int to add to the buffer. |
|
inline |
Add a long
to the buffer (aligns the data and grows if necessary).
| x | The long to add to the buffer. |
|
inline |
Adds an offset, relative to where it will be written.
| off | The offset to add to the buffer. |
|
inline |
Add a sbyte
to the buffer (aligns the data and grows if necessary).
| x | The sbyte to add to the buffer. |
|
inline |
Add a short
to the buffer (aligns the data and grows if necessary).
| x | The short to add to the buffer. |
|
inline |
Add an uint
to the buffer (aligns the data and grows if necessary).
| x | The uint to add to the buffer. |
|
inline |
Add an ulong
to the buffer (aligns the data and grows if necessary).
| x | The ulong to add to the buffer. |
|
inline |
Add an ushort
to the buffer (aligns the data and grows if necessary).
| x | The ushort to add to the buffer. |
|
inline |
Reset the FlatBufferBuilder by purging all data that it holds.
|
inline |
Encode the string s
in the buffer using UTF-8.
| s | The string to encode. |
|
inline |
Creates a string in the buffer from a Span containing a UTF8 string.
| chars | the UTF8 string to add to the buffer |
|
inline |
Creates a vector of tables.
| offsets | Offsets of the tables. |
| T | : | struct |
|
inline |
Writes data necessary to finish a vector construction.
|
inlineprotected |
Finalize a buffer, pointing to the given root_table
.
| rootTable | An offset to be added to the buffer. |
| sizePrefix | Whether to prefix the size to the buffer. |
|
inline |
Finalize a buffer, pointing to the given root_table
.
| rootTable | An offset to be added to the buffer. |
|
inlineprotected |
Finalize a buffer, pointing to the given rootTable
.
| rootTable | An offset to be added to the buffer. |
| fileIdentifier | A FlatBuffer file identifier to be added to the buffer before root_table . |
| sizePrefix | Whether to prefix the size to the buffer. |
|
inline |
Finalize a buffer, pointing to the given rootTable
.
| rootTable | An offset to be added to the buffer. |
| fileIdentifier | A FlatBuffer file identifier to be added to the buffer before root_table . |
|
inline |
Finalize a buffer, pointing to the given root_table
, with the size prefixed.
| rootTable | An offset to be added to the buffer. |
|
inline |
Finalize a buffer, pointing to the given rootTable
, with the size prefixed.
| rootTable | An offset to be added to the buffer. |
| fileIdentifier | A FlatBuffer file identifier to be added to the buffer before root_table . |
|
inline |
A utility function to copy and return the ByteBuffer data as a byte[]
.
|
get |
Get the ByteBuffer representing the FlatBuffer.
This is typically only called after you call Finish()
. The actual data starts at the ByteBuffer's current position, not necessarily at 0
.
|
getset |
Gets and sets a Boolean to disable the optimization when serializing default values to a Table.
In order to save space, fields that are set to their default value don't get serialized into the buffer.


Data Scientist
Product Owner for a neural network IP that automates incident triage for a 98% reduction in investigation time. Transformed and deployed the neural network IP for new, data-ready customers within a six-week timeframe. Deployed a machine learning-enabled solution for optimizing client contracts.
Data Engineer
Built a prototype for an all-purpose data processing architecture using Python and SQL designed for centralizing disparate data sources across an organization. Created Python-based automations for handling human resource data from MS Excel for a 90% reduction in man-hours during audits. Upgraded analytics tools for monitoring Business Operations using SQL and integrated through Talend.
Project Manager
Managed successful end-to-end deployment of multiple application enhancements and system upgrades across multiple project teams. Enhanced customer relationships through partnership building, risk management, and communication. Ensured all projects were compliant to ITIL Change and Release Management standards.
albertyn.s-carpio@outlook.com


In 2019, Eventmaker celebrates its 10th anniversary, a great opportunity for a retrospective...
10.000
events organized with our platform
in
22
countries
France / United States / Switzerland / Spain / Turkey / Rwanda / Luxembourg / Sweden / Qatar / Norway / Austria / Ukraine / Hong Kong / Morocco / South Korea / Lithuania / Netherlands / Italy / Ireland / Denmark / Mexico / United Kingdom / United Arab Emirates / Algeria / Ivory Coast
Driven by
24.000
event managers
Totalizing
13 Million
of registrations
...flashback on our story...
In
2017
we became Eventmaker.io
A company held at
95%
by its managers
A growing workforce of
1>20
event makers
2 Million
of lines of <code> written carefully
more than
20
advanced features for your events
-Qr code
-Check-in
-Lists & guest management
-Online registration
-Online payments
-Customizable websites
-Personal space
-Emails & Campaigns
-Dedicated iPad application
-CRM integration
-Robust API
-100% multi-lingual
And many more to discover..
5
Location changes
Cergy pontoise Rue du Château-d’Eau - Paris 10 Rue des Poissonniers - Paris 18 Rue René Clair - Paris 18 Rue de Ponthieu - Paris 8 Rue Laffitte - Paris 9
It is also.. 90 People hired2 Couples formed7 Babies4 Businesses created100 Years of accumulated experience
And everything
thanks to you
and we will celebrate soon...
The year has just begun, let's stay connected 2019 - All rights reserved


| PostgreSQL 9.0.10 Documentation | ||||
|---|---|---|---|---|
| Prev | Up | Chapter 51. Index Access Method Interface Definition | Next |
Each index access method is described by a row in the pg_am system catalog (see Section 45.3). The principal contents of a pg_am row are references to pg_proc entries that identify the index access functions supplied by the access method. The APIs for these functions are defined later in this chapter. In addition, the pg_am row specifies a few fixed properties of the access method, such as whether it can support multicolumn indexes. There is not currently any special support for creating or deleting pg_am entries; anyone able to write a new access method is expected to be competent to insert an appropriate row for themselves.
To be useful, an index access method must also have one or more operator families and operator classes defined in pg_opfamily, pg_opclass, pg_amop, and pg_amproc. These entries allow the planner to determine what kinds of query qualifications can be used with indexes of this access method. Operator families and classes are described in Section 35.14, which is prerequisite material for reading this chapter.
An individual index is defined by a pg_class entry that describes it as a physical relation, plus a pg_index entry that shows the logical content of the index — that is, the set of index columns it has and the semantics of those columns, as captured by the associated operator classes. The index columns (key values) can be either simple columns of the underlying table or expressions over the table rows. The index access method normally has no interest in where the index key values come from (it is always handed precomputed key values) but it will be very interested in the operator class information in pg_index. Both of these catalog entries can be accessed as part of the Relation data structure that is passed to all operations on the index.
Some of the flag columns of pg_am have nonobvious implications. The requirements of amcanunique are discussed in Section 51.5. The amcanmulticol flag asserts that the access method supports multicolumn indexes, while amoptionalkey asserts that it allows scans where no indexable restriction clause is given for the first index column. When amcanmulticol is false, amoptionalkey essentially says whether the access method allows full-index scans without any restriction clause. Access methods that support multiple index columns must support scans that omit restrictions on any or all of the columns after the first; however they are permitted to require some restriction to appear for the first index column, and this is signaled by setting amoptionalkey false. amindexnulls asserts that index entries are created for NULL key values. Since most indexable operators are strict and hence cannot return TRUE for NULL inputs, it is at first sight attractive to not store index entries for null values: they could never be returned by an index scan anyway. However, this argument fails when an index scan has no restriction clause for a given index column. In practice this means that indexes that have amoptionalkey true must index nulls, since the planner might decide to use such an index with no scan keys at all. A related restriction is that an index access method that supports multiple index columns must support indexing null values in columns after the first, because the planner will assume the index can be used for queries that do not restrict these columns. For example, consider an index on (a,b) and a query with WHERE a = 4. The system will assume the index can be used to scan for rows with a = 4, which is wrong if the index omits rows where b is null. It is, however, OK to omit rows where the first indexed column is null. Thus, amindexnulls should be set true only if the index access method indexes all rows, including arbitrary combinations of null values. An index access method that sets amindexnulls may also set amsearchnulls, indicating that it supports IS NULL and IS NOT NULL clauses as search conditions.


InNerve2k17 Registration
PR Team Authentication
vpn_key
PR TEAM CODE
school
College Name
Register Teams
|| ARMY INSTITUTE OF TECHNOLOGY , PUNE ||
InNerve2k17
Registration Form
Event Type
Hackathon
Game of Codes
account_circle
Team Name
account_circle
Team Leader Name
school
phone
Telephone Number
email
Email
Register
|| ARMY INSTITUTE OF TECHNOLOGY , PUNE ||


NEWS ALERT:
- Nov 2012:Intersect ENT Announces FDA Approval PROPEL Mini, Allowing More Chronic Sinusitis Patients to Benefit from Localized Drug Delivery
- Jan 2013:Warren Man With Chronic Sinusitus Breathes Easy After Implant Surgery
- Jan 2013:Intersect ENT Initiates U.S. Clinical Study of a Novel In-Office Treatment for Chronic Sinusitis Patients
- Mar 2013:Sinus “Stent” Helps Patients Breathe Easier
- Nov 2012:Intersect ENT Announces FDA Approval PROPEL Mini, Allowing More Chronic Sinusitis Patients to Benefit from Localized Drug Delivery
- Jan 2013:Warren Man With Chronic Sinusitus Breathes Easy After Implant Surgery
- Jan 2013:Intersect ENT Initiates U.S. Clinical Study of a Novel In-Office Treatment for Chronic Sinusitis Patients
- Mar 2013:Sinus “Stent” Helps Patients Breathe Easier


--- layout: topic permalink: "/topic/versioncontrol/" title: Introduction to version control topic-tag: versioncontrol show-resources: true ---
Most open-source platforms are developed using public repositories for code and data. The most common of these include Github, Gitlab, and Bitbucket. Utilizing such repositories will be central to the development of any open-source SDG NRP. Platform code itself will commonly be maintained on such a repository, and some platforms will also take advantage of hosting services.


Winter winds and winter snows slowed the passage of trade across lands where they did not end it until spring, and for every three pigeons sent by merchants, two fell to hawks or weather, but where ice did not cover the rivers, ships still sailed, and rumor flew faster than lightnings. A thousand rumors, each throwing off a thousand seeds that sprouted and grew in snow and ice as in fertile soil.
At Tar Valon, some stories said, great armies had clashed, and the streets ran with blood, and rebel Aes Sedai had stuck the head of Elaida a’Roihan on a pike. No; Elaida had closed her hand, and those who survived among the rebels groveled at Elaida’s feet. There had been no rebels, no division of the White Tower. It was the Black Tower that had been broken, by Aes Sedai designs and Aes Sedai power, and Asha’man hunted Asha’man across the nations. The White Tower had shattered the Sun Palace in Cairhien, and the Dragon Reborn himself was bound now to the Amyrlin Seat, her puppet and her tool. Some tales said Aes Sedai had been bound to him, bound to the Asha’man, yet few believed that, and those few were ridiculed.
Artur Hawkwing’s armies had returned to reclaim his long-dead empire, and the Seanchan were sweeping all before them, even to driving the Dragon Reborn from Altara in defeat. The Seanchan had come to serve him. No; he had cast the Seanchan into the sea, destroying their army utterly. They had carried the Dragon Reborn away, to kneel before their Empress. The Dragon Reborn was dead, and there was as much celebration as mourning, as many tears as cries of joy.
Across the nations the stories spread like spiderweb laid upon spiderweb, and men and women planned the future, believing they knew truth. They planned, and the Pattern absorbed their plans, weaving toward the future foretold.


Twenty One Pilots is my favourite band because their help me through get through the hardest and rough times in my life they have this cool vibe that when i listen to them when life gets hard it always seems to me that they are telling me "WHEN LIFE GETS TOUGH THE TOUGHER GETS GOING". Personally I listen to them because they have always been a part of my life since their songs some times feel like they are talking about most teenagers lives. Twenty One Pilots are super popular, but they’re an unspoken element in the world. Most people don't talk about them,even as they quietly sing the chorus to the top’s most popular song “Stressed Out” a Grammy award wining song .
Tyler Joseph, the founder and lead vocalist of the american duo Twenty One Pilots was born on December the 1st 1988 in Columbus Ohio. He had three siblings a brother named Jay and the other Zack and a sister named Madison. His mother was a teacher and then became a basketball coach at the Olentangy Orange High School.
Joshua William “Josh” Dun is an American musician. He is currently the drummer for the band Twenty One Pilots. He was born in Columbus, Ohio, on June 18, 1988, along with two sisters (Ashley and Abigail) and a brother (Jordan), and taught himself how to play drums when he was young. According to Dun, he was not really allowed to listen to music as a child but would sneak down to the music store once a week and ask for suggestions from the people there. Dun worked at the Guitar Center for three years, also working with Twenty One Pilots’ former drummer Chris Salih.
Nick Thomas was born in Columbus, Georgia to Helen, a post master and Issac Thomas, a businessman. Nick grew up loving the arts and was consistently involved in theatre plays and local talent shows in Alabama where he grew up. He briefly studied acting at John Casablanca studios in Jacksonville, FL (2000) where he was completing one of his tours during his 10 year U.S. Navy career. He was an immediate "viral sensation", racking up over 30 million views worldwide on social media as the angry father in the short "She's Gonna Be Grounded for Life". Nick continues to work in Hollywood commercially as well as theatrical.
Chris Salih (born September 18, 1985) is the former drummer of Twenty One Pilots. Along with Nick Thomas, the former bassist, Salih left the band in 2011 due to personal reasons. Salih posted his farewell post on May 8, 2011 on Facebook.


Parr Family
Create Event
S
M
Tu
W
Th
F
S
×
Create a new event
Parr Family
Name:
Time:
:00
Helen
Bob
Violet
Dash
Jack-Jack
Month:
Day:
Edit event
Parr Family
Name:
Time:
:00
Helen
Bob
Violet
Dash
Jack-Jack
Month:
Day:


Bulk carrier on fire. Update Nov 15.
by Mikhail VoytenkoBulk carrier YONG SHUN reported fire on board at 0715 UTC Nov 14 in 10 16N 107 05E some 6 nm south of Vung Tau Vietnam, while en route from Hong Kong to Phu My, Vietnam. As of 1000 UTC the ship is cruising in legs or drifting, with offshore tug HAI DUONG 89 (IMO 9625188) at her side.
UPDATE 1430 UTC: Fire in cargo hold is reportedly, under control, but not yet extinguished. The ship is loaded with some 5000 tons of scrap originated in China, so most probably, scrap ignited spontaneously, like it was in a number of similar accidents in Japan. 18 crew reported safe.
UPDATE 0500 UTC Nov 15: As of 0300 UTC, fire in Hold 3 was still on, though under control. Coast Guard ship joined rescue operation, dosing bulker with water from her cannons. 18 crew reported safe and uninjured.
Bulk carrier YONG SHUN, IMO 8748397, dwt 15000, built 2009, flag Sierra Leone, manager CHENG TAI INT’L SHIPPING CO LT, HK.
Bulk carrier on fire since afternoon Nov 14 in South China sea, withholding details, waiting for updates.
Details for subscribers only.
Annual subscription $750. For annual subscription, contact<EMAIL_ADDRESS>Subscribe if you want to receive accidents news ASAP and in full.


| Class | Applies to | Description |
|---|---|---|
{{ item.class }} |
{{ item.applies }} | {{ item.description }} |
Care should be taken to only include at most one primary and one secondary navigation per page. Using multiple navigations with the same style can cause user confusion.
Forcing a navigation to scroll is an established pattern on mobile devices, so it may be appropriate to use it in that context. Wrapping tends to make more sense on larger screens, where the user isn’t forced to scroll passed a ton of navigation chrome.
{% header h3 | Default %}Full width
Wrapped
Full width
Wrapped


Commitment & Experience
Do I need experience to join?
- No experience is required! We will teach you everything you need to learn to be a competitive sailor!
When are practices?
- Practices are generally 3-5 hours every Saturday and Sunday.
- In addition, we try to have weekday practices at the convenience of our members.
Do I have to go to all the practices?
- No! We know that classes/ research (if you're a grad student) come first. Go to as many practices as you like/ have time for.
- That said, the more you practice, the quicker you'll learn.
- We do, however, require sailors to average one practice per week to maintain regatta eligibility.
When are regattas?
- Regattas are typically on the weekends and you can expect to be gone all day or all weekend depending on the location.
How do I get to practices and regattas?
- We carpool to all of our practices and events.
- Teammates with cars are encouraged to drive and are reimbursed for gas.
- When we do not have enough vehicles for a regatta, we rent fully-insured vehicles.
Do I need to come to meetings?
- Yes. Meetings have important info on regattas, practices, and activities.
- To be an active team member make sure to come the weekly meeting every Wednesday night at 8PM, in the MU Fielder Room.
I’m not sure if sailing’s for me.
- Don’t worry! We give prospective members two free practices before they have to commit.
Gear
Am I expected to bring my own gear?
 However, the team has a limited supply of extremely used foul weather gear that we loan to new sailors. We also have loaner life-jackets but they are not recommended for racing.
- There are many places to find affordable sailing gear, but if you can’t buy actual sailing gear try finding substitutes! (Dry fit, surf booties, and gardening gloves are great substitutes!)
If I don’t have any sailing gear what do I need to bring to practice?
- Bring a bathing suit, towel, water, snacks, and a change of clothes. Plan to get wet.
Financial Information
Is the team a charitable organization?
- Yes. We are a 501©3 organization.
What are the member dues?
- Undergraduates: $125 to Davis Sailing Team, $15 to Lake Washington Sailing Club.
- *Undergraduates in their first quarter on the team who do not wish to participate in regattas: $35 to Davis Sailing Team, $15 to Lake Washington Sailing Club.
- *Undergraduates who join or renew when there are no regattas remaining in the quarter: $35 to Davis Sailing Team, $15 to Lake Washington Sailing Club.
- Graduates: $35 to Davis Sailing Team, $15 to Lake Washington Sailing Club.
What are the regatta fees?
- In-state regatta fees (lodging, transportation, boats, etc.) are included for undergraduates paying full dues.
- Graduate students and undergraduates paying reduced dues are not elligible to attend ICSA regattas.
Is financial aid available?
- Yes! If you have financial concerns about joining our team, don't hessitate to talk with us! We'll do our best to work something out.
Prospective Freshmen
What role can my sailing play in the admissions process?
- Unfortunately, a desire to sail for UC Davis cannot assist you in the admissions process.
Can I get a scholarship for sailing?
- No. Unfortunately our governing body, Inter-collegiate Sailing Association (ICSA) does not allow any sailor to receive a sailing related scholarship at any school.
How cool are you guys?
Sailing Aspects
What types of boats do we sail?
- We sail the Collegiate Flying Junior (CFJ.), which is a 14-foot, 2 person dinghy.
- Additionally, we own a single-crew, Olympic-class Laser and a 2-4 person Hobie Cat 16.
Will I get to chose whether I get to skipper or crew?
- Yes. We let our sailors chose which position they prefer. We encourage all sailors to be proficient in
both positions. However, we encourage people to certain positions depending on their size, skill and enthusiasm.
- About a third of our skippers joined the team without sailing experience.
- *Skippers must be approved by a board member before sailing with non-skippers (without an instructor).
How much do I need to know about racing?
- Nothing! We will teach you everything that you need to know. That said, if you have race experience, that’s awesome!(Please teach us)
Teams
How do you make varsity?
- The varsity team is chosen by skill. But don’t get discouraged if you haven’t sailed before. Half our varsity learned in college and already have the skills to race at the varsity level.
What is the difference between JV and Varsity?
- The JV and Varsity teams practice, travel and race together. The only difference between the team is in scoring and eligibility. There are both varsity only and JV only regattas the team attends.
What is the difference between a student organization and a club or varsity sport?
- A student organization is student run, meaning we have no overseeing administrator. All our officers are students elected by our peers and are very aware of the students’ budget, time, commitment and needs. The officers also gain valuable leadership and resume experience which they would not receive on a varsity sport.
- As student organization, we are not only able to attend regattas without university restrictions but also maintain our equipment more efficiently. We cannot have our funding cut and we can receive donations.
Team Leadership
Who is in charge of the team?
- A group of six students are elected to be the officers of the team. It is a great leadership opportunity and looks great on a resume.
Do we have a coach?
- Our head coach is our captain, who is both an officer and a student.
- All of our skippers have experience instructing fellow teammates. In addition,
several of us have experience instructing both youth and adult sailing classes.
Who designed this awesome website?
- Our V.P. for Equipment and Graduate Student Advisor, Sean Wu, who is a skipper/ crew and the team’s resident rocket scientist, pilot, … basically he does everything.
- Updated in October 2018 by Isabel Kalogiannis, Senior Computer Science Major and second year sailor on the Davis Sailing team.
Return to Top


| random_port {pbdZMQ} | R Documentation |
Generate a valid, random TCP port.
random_port(min_port = 49152, max_port = 65536) random_open_port(min_port = 49152, max_port = 65536, max_tries = 100)
min_port, max_port |
The minimum/maximum value to be generated. The minimum should not be below 49152 and the maximum should not exceed 65536 (see details). |
max_tries |
The maximum number of times a random port will be searched for. |
By definition, a TCP port is an unsigned short, and so it can not exceed 65535. Additionally, ports in the range 1024 to 49151 are (possibly) registered by ICANN for specific uses.
random_port()
will simply generate a valid, non-registered
tcp port. random_unused_port()
will generate a port
that is available for socket connections.
random_open_port()
finds a random port not already bound
to an endpoint.
Drew Schmidt
"The Ephemeral Port Range" by Mike Gleason. http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html
random_port()


Art of Conversation
The past year had taken a heavy toll on Steve, but he didn't appreciate seeing proof in the mirror. His usually sharp blue eyes looked hazy and drawn tonight. A dark stubble was shrouding his strong jaw and dimpled chin. Around his temples, the gray highlights were advancing, making their way deeper into his thicket of coarse black hair. Although his female colleagues insisted the gray only accentuated his bookish appeal, Langdon knew better.
Time will go here
Stats will go here
Styling


public class Distribution<E> extends java.lang.Object implements Sampler<E>, ProbabilityDistribution<E>
Counters
. Smoothed counters reserve probability mass for unseen
items, so queries for the probability of unseen items will return a small
positive amount. Normalization is L1 normalization:
totalCount()
should always return 1.
A Counter passed into a constructor is copied. This class is Serializable.| Modifier and Type | Method and Description |
|---|---|
static <E> Distribution<E> |
absolutelyDiscountedDistribution(Counter<E> counter,
int numberOfKeys,
double discount) |
void |
addToKeySet(E o)
Insures that object is in keyset (with possibly zero value)
|
E |
argmax() |
boolean |
containsKey(E key) |
static <E> Distribution<E> |
distributionFromLogisticCounter(Counter<E> cntr)
Maps a counter representing the linear weights of a multiclass
logistic regression model to the probabilities of each class.
|
static <E> Distribution<E> |
distributionWithDirichletPrior(Counter<E> c,
Distribution<E> prior,
double weight)
Returns a Distribution that uses prior as a Dirichlet prior
weighted by weight.
|
E |
drawSample()
Exactly the same as sampleFrom(), needed for the Sampler interface.
|
E |
drawSample(java.util.Random random)
A method to draw a sample, providing an own random number generator.
|
static <E> Distribution<E> |
dynamicCounterWithDirichletPrior(Counter<E> c,
Distribution<E> prior,
double weight)
Like normalizedCounterWithDirichletPrior except probabilities are
computed dynamically from the counter and prior instead of all at once up front.
|
boolean |
equals(Distribution<E> distribution) |
boolean |
equals(java.lang.Object o) |
double |
getCount(E key)
Returns the current count for the given key, which is 0 if it hasn't
been
seen before.
|
Counter<E> |
getCounter() |
static <E> Distribution<E> |
getDistribution(Counter<E> counter)
Creates a Distribution from the given counter.
|
static <E> Distribution<E> |
getDistributionFromLogValues(Counter<E> counter)
Creates a Distribution from the given counter, ie makes an internal
copy of the counter and divides all counts by the total count.
|
static <E> Distribution<E> |
getDistributionFromPartiallySpecifiedCounter(Counter<E> c,
int numKeys)
Assuming that c has a total count < 1, returns a new Distribution using the counts in c as probabilities.
|
static <E> Distribution<E> |
getDistributionWithReservedMass(Counter<E> counter,
double reservedMass) |
int |
getNumberOfKeys() |
static <E> Distribution<E> |
getPerturbedDistribution(Counter<E> wordCounter,
java.util.Random r) |
static <E> Distribution<E> |
getPerturbedUniformDistribution(java.util.Collection<E> s,
java.util.Random r) |
double |
getReservedMass() |
static <E> Distribution<E> |
getUniformDistribution(java.util.Collection<E> s) |
static <E> Distribution<E> |
goodTuringSmoothedCounter(Counter<E> counter,
int numberOfKeys)
Creates a Good-Turing smoothed Distribution from the given counter.
|
static <E> Distribution<E> |
goodTuringWithExplicitUnknown(Counter<E> counter,
E UNK)
Creates a Good-Turing smoothed Distribution from the given counter without
creating any reserved mass-- instead, the special object UNK in the counter
is assumed to be the count of "UNSEEN" items.
|
int |
hashCode() |
java.util.Set<E> |
keySet() |
static <E> Distribution<E> |
laplaceSmoothedDistribution(Counter<E> counter,
int numberOfKeys)
Creates an Laplace smoothed Distribution from the given counter, ie adds one count
to every item, including unseen ones, and divides by the total count.
|
static <E> Distribution<E> |
laplaceSmoothedDistribution(Counter<E> counter,
int numberOfKeys,
double lambda)
Creates a smoothed Distribution using Lidstone's law, ie adds lambda (typically
between 0 and 1) to every item, including unseen ones, and divides by the total count.
|
static <E> Distribution<E> |
laplaceWithExplicitUnknown(Counter<E> counter,
double lambda,
E UNK)
Creates a smoothed Distribution with Laplace smoothing, but assumes an explicit
count of "UNKNOWN" items.
|
double |
logProbabilityOf(E key)
Returns the natural logarithm of the object's probability
|
static void |
main(java.lang.String[] args)
For internal testing purposes only.
|
double |
probabilityOf(E key)
Returns the normalized count of the given object.
|
E |
sampleFrom()
Returns an object sampled from the distribution using Math.random().
|
E |
sampleFrom(java.util.Random random)
Returns an object sampled from the distribution using a self-provided
random number generator.
|
static <E> Distribution<E> |
simpleGoodTuring(Counter<E> counter,
int numberOfKeys)
Creates a Distribution from the given counter using Gale & Sampsons'
"simple Good-Turing" smoothing.
|
java.lang.String |
toString() |
java.lang.String |
toString(java.text.NumberFormat nf) |
double |
totalCount() |
public E drawSample()
drawSample
in interface Sampler<E>
public E drawSample(java.util.Random random)
drawSample
in interface ProbabilityDistribution<E>
public java.lang.String toString(java.text.NumberFormat nf)
public double getReservedMass()
public int getNumberOfKeys()
public java.util.Set<E> keySet()
public boolean containsKey(E key)
public double getCount(E key)
get
that casts
and extracts the primitive value.key
- The key to look up.public static <E> Distribution<E> getDistributionFromPartiallySpecifiedCounter(Counter<E> c, int numKeys)
public static <E> Distribution<E> getUniformDistribution(java.util.Collection<E> s)
s
- a Collection of keys.public static <E> Distribution<E> getPerturbedUniformDistribution(java.util.Collection<E> s, java.util.Random r)
s
- a Collection of keys.public static <E> Distribution<E> getPerturbedDistribution(Counter<E> wordCounter, java.util.Random r)
public static <E> Distribution<E> getDistribution(Counter<E> counter)
public static <E> Distribution<E> getDistributionWithReservedMass(Counter<E> counter, double reservedMass)
public static <E> Distribution<E> getDistributionFromLogValues(Counter<E> counter)
public static <E> Distribution<E> absolutelyDiscountedDistribution(Counter<E> counter, int numberOfKeys, double discount)
public static <E> Distribution<E> laplaceSmoothedDistribution(Counter<E> counter, int numberOfKeys)
public static <E> Distribution<E> laplaceSmoothedDistribution(Counter<E> counter, int numberOfKeys, double lambda)
public static <E> Distribution<E> laplaceWithExplicitUnknown(Counter<E> counter, double lambda, E UNK)
counter
- the counter to normalizelambda
- the value to add to each countUNK
- the UNKNOWN symbolpublic static <E> Distribution<E> goodTuringSmoothedCounter(Counter<E> counter, int numberOfKeys)
public static <E> Distribution<E> goodTuringWithExplicitUnknown(Counter<E> counter, E UNK)
counter
- the counterUNK
- the unknown symbolpublic static <E> Distribution<E> simpleGoodTuring(Counter<E> counter, int numberOfKeys)
public static <E> Distribution<E> distributionWithDirichletPrior(Counter<E> c, Distribution<E> prior, double weight)
weight
- multiplier of prior to get "pseudo-count"public static <E> Distribution<E> dynamicCounterWithDirichletPrior(Counter<E> c, Distribution<E> prior, double weight)
weight
- multiplier of prior to get "pseudo-count"public static <E> Distribution<E> distributionFromLogisticCounter(Counter<E> cntr)
public E sampleFrom()
public E sampleFrom(java.util.Random random)
public double probabilityOf(E key)
probabilityOf
in interface ProbabilityDistribution<E>
public double logProbabilityOf(E key)
logProbabilityOf
in interface ProbabilityDistribution<E>
public E argmax()
public double totalCount()
public void addToKeySet(E o)
o
- object to put in keysetpublic boolean equals(java.lang.Object o)
equals
in class java.lang.Object
public boolean equals(Distribution<E> distribution)
public int hashCode()
hashCode
in class java.lang.Object
public java.lang.String toString()
toString
in class java.lang.Object
public static void main(java.lang.String[] args)


Data: [{"title":"Choice 1","value":"1"},{"title":"Choice 2","value":"2"}]
Empty Item Title: No Choice
Label: Choices
Selected: [[stringify(selected)]]


Something went wrong during the build!
This page should never show :( The real website replaces this placeholder if everything goes well.


| Char | English | Locale | ℓ | script l | script l | ℘ | script upper P | p de weierstrass | ⅀ | double struck n ary summation | ⅅ | double struck italic upper D | mayúscula D | ⅆ | double struck italic d | d | ⅇ | double struck italic e | e | ⅈ | double struck italic i | i | ⅉ | double struck italic j | ℼ | double struck pi | ℽ | double struck gamma | ℾ | double struck upper Gamma | ℿ | double struck upper Pi | 𝚤 | italic dotless i | cursiva dotless i | 𝚥 | italic dotless j | cursiva dotless j |
|---|


Last updated: Sep 14th, 2018
Career Buddy: Joy Corsi
Visual
10% Auditory
50% Visual
40% Tactile
Version 1
My name is Ryan Becker
I am a junior web developer
I've currently been playing around with Javascript and SQL
Version 2
Hi My name is Ryan Becker. I’m working on my coding skills and foundation at CNM's DeepDive that will help me develop games that others can enjoy. Are you working on developing games?
Version 3
Hi My name is Ryan Becker and I’m a junior web developer. I’m looking for employment with a tech company who creates vibrant concise websites. I am actively networking by attending community events weekly to make connections in the tech industry. Do you know of any reputable companies looking for junior devs?
My buddy Joy and I chose these two photos to represent how our path through this bootcamp is very strict and set in stone initially. And our path opens to the edge of our imaginations once we make it out in the real world!
What:
Get into a career that is both fulfilling and pays well.
How:
By developing coding skills and foundation.
Why:
Developing games that others can enjoy.
In my life I try to achieve balance by spending time with family and my two dogs. Every week I attend church and dinner with my sister's family. I also spend a chunk of my weekend over at my parents helping in the yard and around the house. When a project gets hard or I get frustrated, I take some time to play with my dogs. They love playing tug of war as well as general rough housing! Even a short break spent with them can make all the difference.
Playing forbidden island with my group was a lot fun! Marlon was very patient and thoughtful when it came to moves and possible outcomes. I was happy with my ability to explain the instructions to the group with Caleb's help. Caleb was awesome at making sure we adhered to the water rising mechanic and moving the levels up. Chelsea did a good job of sifting through and arriving at the best option when all of us had different ideas of how to advance the game with someone's turn. I'm happy to say we won! Both in game and as a group! I feel we all have a better understanding of each other and how to communicate possible ideas.
This week started out overwhelming but as each day passed I found that I was more and more comfortable as well as confident. The lecture portions can be confusing as Dylan moves pretty fast but by Friday I found myself better at taking notes. George also reminded me that I have access to Dylan's past lecture notes so I will definitely be looking back on some of those to add more to my notes. I am a little nervous about setting up the database/SQL portion of the data-design project but I will make it through! I am very much looking forward to next week!
This week I really struggled with the functions assignment. I tried to spend a good amount of time watching tutorials on YouTube but I am still struggling with them as far the homework assignment was concerned I hope there are more assignments to offset how badly I did on this one. The snap challenges regarding functions have also been hard. I will be spending some significant time outside class trying to understand them better.
This week started out pretty rough with functions but as the week is coming to a close I am feeling very good about databases. Writing class objects with accessors, mutators and constructors turned out to be a lot easier than I thought it would be. Same thing with actually inserting data into my databases. The snap challenge on Friday was the most confident I have felt about one! It felt great to help a few of my classmates as well. I am feeling great about both projects, the PWP and data design. Bring on week 3!
This week has been a lot fun with regards to Captsone! The submission process was very straight forward and I was impressed with the variety of the ideas submitted. Voting was also very easy and I feel lucky enough to put on the most fun idea, Nerd Nook! I'am also very happy with my teammates, they are all outgoing and always willing to help. Our planning process was methodical, but I am already seeing that we work well together! I am stoked to see how the project unfolds!
I am a little sad to not see us take the data design project further. It's been an interesting project and I've felt confident with it. I am eagerly awaiting feedback on it. The CSS lecture and challenge has been hard. Coming into the class I thought the CSS would be the easiest thing to nail down but boy was I wrong! I also failed to follow instructions on the challenge, which didn't help but I do feel it was a good exercise as I still manipulated the content quite a bit and ended up with a pretty decent result.
It's great to finally be working on our capstone project! We've mad a lot of progress in a short amount of time. I am still feeling a little overwhelmed with the knowledge base. But at least the introduction of new concepts have slowed down a little bit. I am really appreciating my fellow class mates as they are always ready and willing to help or answer questions. This week I attended two after school meet ups and both were really interesting! Especially the first one about publishing apps to the two major app stores.
I enjoy coding because it's creative and allows me to solve problems. I'm also learning to love the tech community as a whole, everyone I've met is friendly and approachable. I also love coding because it allows me to interface with technology which has always been a major passion of mine. I bought a computer specifically for this class and as each week passes I love it more and more.
It's hard to believe that we are half way! It seems as though time is speeding up! Capstone has been going well, most of the group has been able to clear all errors from unit testing. We will hopefully be past unit testing by mid next week. I attended two after school activities this week. Dylan's talk on net neutrality was very interesting. And on Wednesday I attended the soft opening of the UNM rain forrest collaboration center. We performed a quick design related activity with partners and I very much enjoyed it! I met some great contacts and spent some time with a fellow capstone group member. I am initially feeling overwhelmed with JavaScript, but hopefully after a few weeks I'll be feeling better just like PHP. My PWP is going well but I need to put some more time into it. I have had a few migraines this week, I think its been due to stress as we have been discussing post-bootcamp a lot more. I am confident in my abilities but it's starting to feel like a short time until graduation! I hope I am ready!
This week I am enjoying coding because there are so many ways to accomplish different tasks. I learned this week that I constantly need to be studying and learning new languages as popularity and demand can change so much year to year. It's a plus and minus! Those who keep moving moving will do great, opposed to becoming stagnant and avoiding change.
This week has absolutely flown by! They seem to be speeding up! My group and I have made fantastic project on our capstone. We are moving fast through APIs! I have been really appreciating all the in class time the bridge has been giving us for group work. Some of my group members live pretty far, so it's really key for us maximize the group time to discuss questions and top level ideas. I appreciated the check in with Karen and Dan, it's amazing to have so much support. Personally I need to start networking more aggressively the last 4 weeks. I have business cards ready for next week and I look forward to handing a few out. I have found a great friend with Marlon, he is generally interested in attending meet ups and it makes it way easier as I sometimes tend to be quiet around new people. Things are great this week!
Stemming off of last week reflection, I really love how general "coding" is today. A few weeks back it seemed really daunting and overwhelming but as I get closer to the end, I'm seeing an immense world of opportunities. I just need to pick something and dive in!
This week has been productive! I had an excellent meeting with George in regards to the progress on my personal website. I am very pleased with the progress I've made and the direction of my site. George was pleased as well and provided great feedback. Our capstone group is chugging along with our project. We burned our API test tickets! Working with insomnia was some what confusing, but it is a cool tool to use when checking APIs. This week I had another bad migraine, I know stress and tension can be major contributors but I feel as thought I lacked the adequate hydration on Tuesday and Wednesday. I'll be sure to drink more next week!
This week I love coding because yet another guest speaker told us to pick a specialization! It's super stressful thinking about needing to be a jack of all trades or be knowledgeable in EVERYTHING! I will be good with knowing a little of everything and picking a specialty.
This week has flown by! I cannot believe we have just two more weeks. This week I made excellent progress on my personal website. I had my layout and colors done last week but my contact from was non-functional so I configured that this week and I'm happy to say everything works 100% and all I have left now is to write some content itself! I am feeling great for next week's deadline of Thursday. This week included interview day on Friday. My technical interview was first at 8:30am. Right out the gate! Dylan ended up being my technical interviewer so that made me even more nervous. I did not do great and Dylan agreed, he said he was expecting better. But I'm trying my best to chalk it up to a positive experience with things to improve for next time. I guess I'm happy it wasn't with an actual employer. My HR interview followed two hours later and it went really well. I'm very proud of my previous experience and discussing it comes pretty easy. I learned so much working for a small family business and I think it translated well. I am eager to hear what Mia reports of my interview! It is truly shocking that we have two weeks left...It's pretty fitting that Friday's prodev lab included a presentation about imposter syndrome. I am feeling like I probably need to take more classes and expand my knowledge base. I don't necessarily feel like an imposter but I do feel like I still don't understand some of the advanced and theoretical topics. I will absolutely be applying to some dev shops but I am also okay with continuing my education whether it's at UNM or CNM.
This week I am absolutely loving my personal website. It's really exciting to see code turn into something beautiful and functional. I am discovering more and more that even though CSS & Bootstrap is a pains sometimes, I enjoy design and and might be better off sticking with front end development or design.
Personal website projects were due this week and I am so pleased with how it turned out! I am very eager to hear feedback from George. My group and I have made significant progress this week on our project. We have most of the pages loaded into angular and hopefully we'll get more done this weekend. We practiced our presentation for demo day on Friday. It was a great exercise, I was impressed how much we improved with just two iterations.
This week I am enjoying coding because it has introduced to me to some wonderful people. Not only have I met some really nice people at community events, but my capstone teammates are wonderful and I look forward to making more fun memories!
Graduation day is finally here! It's been a long journey but I am happy to be moving on to the next chapter! I have truly enjoyed my time and deep dive and I am extremely thankful for all the support and guidance along the way. The bridge has set an excellent tone for the bootcamp as a whole and I wish the best of luck to Dylan & George. My capstone teammates have been so helpful and I am happy to say I've made some excellent friends for life. Samething with our cohort as a whole, it's been a pleasure working with everyone.
This week I am leaving with an excellent foundation and I cannot wait to continue my education and further help the tech community as a whole!


 Well, not this time. Post Consumer Brands and Hostess have just announced they are introducing a frickin' Twinkies cereal to grocery stores in December.
Twinkies Cereal replicates the distinctive taste and familiar oblong shape of the golden-colored snack cake, delivering delightful sweetness in every bite...
"In developing a cereal version of the iconic Twinkies, our top priority was focused on delivering the great Twinkies flavor in each bite," said Josh Jans, Brand Manager of Cereal Partnerships at Post Consumer Brands. "Knowing that taste remains the No. 1 purchase driver of cereal, we conducted multiple consumer tests, and the new Twinkies Cereal delivered. We think fans will find that it not only tastes great with milk but also outside the bowl."
What did we do to deserve this?!
images via Post Hostess Twinkies/CNBC


FROM THE DESK OF REV DR Homas Ella NIGERIA NATIONAL PETROLEUM CORPORATION 21 VICTORIA ISLAND LAGOS ATTENTION PERSONAL AND CONFIDENTIAL Having consulted with my colleagues and based on the information Gathered from the Nigerian Chambers Of Commerce And Industry, I have the privilege to request your assistance to transfer the sum of (15,500.000.00) FIFITEEN MILLION, FIVE HUNDRED THOUSAND UNITED STATES DOLLARS, into your accounts. The above sum resulted from an over-invoiced contract, executed, Commissioned and paid for about three years (3) ago by a foreign contractor. This action was however intentional and since then the fund has been in a suspense account at The Central Bank Of Nigeria Apex Bank. We are now ready to transfer the fund overseas and that is where you Come in. It is important to inform you that as civil servants, we are Forbidden to operate a foreign account; that is why we require your assistance. The total sum will be shared as follows: 65% for us, 30% for you and 5% for local and international expenses incidental to the transfer. The transfer is risk free on both sides. I am an accountant with the Nigerian National Petroleum Corporation (NNPC). If you find this proposal acceptable,we shall require the following documents:(1) Your banker name and address (2) Your private telephone and fax numbers ? for confidentiality and easy communication. (c) Your full name and address. Alternatively we will furnish you with an application form which you will file and apply directly to the Central Bank of Nigeria, where this money was deposited, along with a breakdown explaining, comprehensively what we require of you. The business will take us (14) working days to accomplish. Await your urgent response. Yours Sincerely Rev. Dr. THomas Ella.
|


TAKEOVER
Supported kindly by
Win cool stuff
Tweet about the evening using the hashtag #WiTLincs to be in with a chance to win one of these sweet prizes!
Lighting Talks!
Talks last 5 mins (you will be warned when you have 1 min left)
Topics can be anything! No limits!

#WiTLincs


Passenger - YB Move was a project that aimed to make the lives of bus users significantly easier by allowing them to use their mobile phone as their ticket. The secondary aim was to improve efficiency of the boarding process.
The aim of the Passenger ticketing platform was to create simple apps that allowed customers to purchase tickets with ease while reducing boarding times for the buses. Passenger ticketing was created from the begining to be highly configurable and applicable to many bus operators.
The ticketing app ensures speed of boarding by using a bold, animated, yet simple pin that the drivers can verify at a glance, no more searching paper tickets for a date or secret code to validate a ticket.
During the creation of this project I setup jenkins server to frequently test the build, distribute the build internally or to clients and finally to release it to the Google Play Store. The Jenkins setup also included setup of automated instrumentation and unit tests as well as Jacoco code coverage, lint and checkstyle reports. An Artifactory server was also leveraged to allow for reusable components such as UI widgets or API modules to be reused on any future project.
Technologies used:
"Left my wallet at home, no money and no cards. But down loaded the app as bus was coming down the road and got a ticket within seconds."
Paul Chandler


Overview
This has been my first real weekend of study and practice. My head hurts! But now I'm getting familiar with the command-line, git, git branching, and github.
I've pushed my first few commits up to github, and made my github.io website repository and published a very basic home page and this blog post.
What's your take on the DBC/EDA experience?
Afer watching the Fireside chat with Shereef: Kitchen vs Table I was surprised to see how similar the conversation and expectations were between the DBC and the EDA programme, and that this fireside chat was 4 years ago. I felt at ease after listening to the chat and the Q&A session where many of my own questions were addressed.
What are your impressions?
My impressions are that the foundations and philosophy of EDA are well founded and based on much experience by the likes of DBC and the EDA over the last few years. I feel inspired that so much thought and care has been put into putting this programme on, and appreciate the honesty about the expectations to keep up, put in the mahi, be kind, and be honest with each other and ourselves.
How do you see yourself engaging with this type of culture?
I'll be throwing myself in boots and all and "trusting the process" and myself to make this an awesome, life-changing, experience.
The focus on soft skills, time management, self-awareness, team work and interpersonal relationships will be a good challenge for me, as I certainly know I tend to lean into the technical sides of any work.
Have your expectations of EDA changed? If so, how?
I always expected it would be about culture and self-learning, and this has been shown to be the case with the balance of tech vs human-skills
I'm realising now the level of work and commitment that will be required is even more than I had expected, but after working through a few challenges already this weekend, I'm feeling confident I'll succeed.
When I attended the "Day in the life of Dev Academy" last year, I did not see the depth of structure and support around the students that I have seen since prepping for Orientation and now working through the first Sprint. I'm impressed and feel both the challenge on me and the support from the EDA crew, and know that I'll be drawing on the cohort for support and strength, as well as being part of that support for those around me.
Are you excited to participate in this kind of learning environment? Does it make you nervous?
Yes, excited and nervous. But after this weekend, knowing that I took the time to learn, to go back to basics, and that I know things today that I didn’t know on Friday, has given me a boost in enthusiasm and confidence in myself.
Nervous that this is way more work than I'm used to in my current job, but excited to become a better person by working through it all, with those around me.
It is only these challenges that make us renew ourselves and realise what we are capable of.


Classes
Engines
Engines::PluginMigrator
Engines::RailsExtensions
Engines::RailsExtensions::ActiveRecord
Engines::RailsExtensions::Dependencies
Engines::RailsExtensions::Migrations
Engines::RailsExtensions::PublicAssetHelpers
Engines::RailsExtensions::RailsInitializer
Engines::RailsExtensions::Routing
Engines::RailsExtensions::Templates
Engines::RailsExtensions::Templates::ActionView
Engines::RailsExtensions::Templates::Layout
Engines::RailsExtensions::Templates::MailTemplates
Engines::Testing
Module
Plugin
PluginList
Rails


Intro.Rmd
Assume that there is a data science team working on developing a classification algorithm that at a Bond Object level can predict
PULL_TEST_STATUS
based on the time-series data in the Sample Objects. It is currently a very time-consuming task for this team to access the data they need. These JSON files land in a single directory on a Network File Share (NFS) and there are tens of thousands of files generated per day. Thus, this team can only get data for a specificBOND_GROUP_ID
, for example, by loading data for a particular time range into memory and then filtering. They are looking to you to help provide a new way to access this data in a schema which facilitates their modeling efforts.
As a data engineer, I want to make it as easy as possible for the data science team to query data by timestamp for a given bond_group_id so they can predict the failure of pull_test_status given sample data and anticipate, identify, and eliminate failures.
Inputs:
Constraints:
Requirements:
A few ideas come to mind for how we’ll store the data:
Star Schema: Since our data are nested, we can create dimension tables for files and fact tables for bonds and samples.
files_table
- file_id
- file_* (metadata)
bonds_table
- file_id
- bond_id
- bond_* (metadata)
samples_table:
- (optional): file_id // optional bc each sample_id is unique to bond_id
- bond_id
- sample_id
- sample_* (metadata)
Cons: Users would have to join the data to do their analysis. Joins can be messy (especially with time-series data).
Model-ready dataset: Prepare it for modeling (flat tables):
samples_table:
sample_id
bond_id
bond_metadata* (replicated each row per sample_id)
file_id
file_metadata (replicated each row per bond_id)
Flat tables are easiest for doing machine learning. They’re inefficient for storage, however.
Other ideas:
A few storage approaches come to mind:
Pros/cons of database
Pros:
Cons:
Pros/cons of parquet files
Pros:
Cons:
Pros/cons of avro: similar to parquet, but more easily handles file systems which change schema over time.
Pros/cons of language-native file systems
Pros:
.Rda
, .RDS
and .fst
file types (see post). That, and the data is efficiently stored if it’s going to be used specifically by R users.Cons:
I’ve decided to design using a few different design approaches (reviewed below). I wrote an R package to handle the data processing. The bonds
package has multiple functions to help with ETL.
library(bonds)
#> Loading required package: arrow
#>
#> Attaching package: 'arrow'
#> The following object is masked from 'package:utils':
#>
#> timestamp
#> Loading required package: DBI
#> Loading required package: dplyr
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
#> Loading required package: fs
#> Loading required package: glue
#>
#> Attaching package: 'glue'
#> The following object is masked from 'package:dplyr':
#>
#> collapse
#> Loading required package: logger
#> Loading required package: stringr
#> Loading required package: jsonlite
#> Loading required package: tidyjson
#>
#> Attaching package: 'tidyjson'
#> The following object is masked from 'package:jsonlite':
#>
#> read_json
#> Loading required package: tidyr
#> Loading required package: tictoc
#> Loading required package: RSQLite
#> Loading required package: pool
#> Loading required package: here
#> here() starts at /home/rstudio/tesla/bonds
I created a function etl_all_files()
that does the ELT:
Here are the logs for the output:
> etl_all_files()
[1] 1
INFO [2021-10-29 05:14:41] Parsing JSON
INFO [2021-10-29 05:14:44] Cleaning data
INFO [2021-10-29 05:14:45] Loading to sql
0.812 sec elapsed
INFO [2021-10-29 05:14:45] Save as pqt all
0.359 sec elapsed
INFO [2021-10-29 05:14:46] Save as pqt by group_id
0.426 sec elapsed
INFO [2021-10-29 05:14:46] Finished file1
[1] 2
INFO [2021-10-29 05:14:46] Parsing JSON
INFO [2021-10-29 05:14:49] Cleaning data
INFO [2021-10-29 05:14:49] Loading to sql
INFO [2021-10-29 05:14:49] Updating [data.files] from file 2
INFO [2021-10-29 05:14:49] Updating [data.bonds] from file 2
INFO [2021-10-29 05:14:49] Updating [data.samples] from file 2
0.5 sec elapsed
INFO [2021-10-29 05:14:50] Save as pqt all
0.2 sec elapsed
INFO [2021-10-29 05:14:50] Save as pqt by group_id
0.942 sec elapsed
INFO [2021-10-29 05:14:51] Finished file2
[1] 3
INFO [2021-10-29 05:14:51] Parsing JSON
INFO [2021-10-29 05:14:54] Cleaning data
INFO [2021-10-29 05:14:54] Loading to sql
INFO [2021-10-29 05:14:54] Updating [data.files] from file 3
INFO [2021-10-29 05:14:54] Adding rework_trigger_msg AS TEXT to [data.bonds]
INFO [2021-10-29 05:14:54] Updating [data.bonds] from file 3
INFO [2021-10-29 05:14:54] Updating [data.samples] from file 3
0.522 sec elapsed
INFO [2021-10-29 05:14:55] Save as pqt all
0.191 sec elapsed
INFO [2021-10-29 05:14:55] Save as pqt by group_id
0.469 sec elapsed
INFO [2021-10-29 05:14:55] Finished file3
[1] 4
INFO [2021-10-29 05:14:55] Parsing JSON
INFO [2021-10-29 05:14:59] Cleaning data
INFO [2021-10-29 05:14:59] Loading to sql
INFO [2021-10-29 05:14:59] Updating [data.files] from file 4
INFO [2021-10-29 05:14:59] Updating [data.bonds] from file 4
INFO [2021-10-29 05:14:59] Updating [data.samples] from file 4
0.659 sec elapsed
INFO [2021-10-29 05:15:00] Save as pqt all
0.26 sec elapsed
INFO [2021-10-29 05:15:00] Save as pqt by group_id
0.536 sec elapsed
INFO [2021-10-29 05:15:01] Finished file4
[1] 5
INFO [2021-10-29 05:15:01] Parsing JSON
INFO [2021-10-29 05:15:04] Cleaning data
INFO [2021-10-29 05:15:04] Loading to sql
INFO [2021-10-29 05:15:04] Updating [data.files] from file 5
INFO [2021-10-29 05:15:04] Updating [data.bonds] from file 5
INFO [2021-10-29 05:15:04] Updating [data.samples] from file 5
0.385 sec elapsed
INFO [2021-10-29 05:15:04] Save as pqt all
0.123 sec elapsed
INFO [2021-10-29 05:15:04] Save as pqt by group_id
0.572 sec elapsed
INFO [2021-10-29 05:15:05] Finished file5
[1] 6
INFO [2021-10-29 05:15:05] Parsing JSON
INFO [2021-10-29 05:15:07] Cleaning data
INFO [2021-10-29 05:15:08] Loading to sql
INFO [2021-10-29 05:15:08] Updating [data.files] from file 6
INFO [2021-10-29 05:15:08] Updating [data.bonds] from file 6
INFO [2021-10-29 05:15:08] Updating [data.samples] from file 6
0.489 sec elapsed
INFO [2021-10-29 05:15:08] Save as pqt all
0.128 sec elapsed
INFO [2021-10-29 05:15:09] Save as pqt by group_id
0.412 sec elapsed
INFO [2021-10-29 05:15:09] Finished file6
[1] 7
INFO [2021-10-29 05:15:09] Parsing JSON
INFO [2021-10-29 05:15:11] Cleaning data
INFO [2021-10-29 05:15:11] Loading to sql
INFO [2021-10-29 05:15:11] Updating [data.files] from file 7
INFO [2021-10-29 05:15:11] Updating [data.bonds] from file 7
INFO [2021-10-29 05:15:11] Updating [data.samples] from file 7
0.355 sec elapsed
INFO [2021-10-29 05:15:12] Save as pqt all
0.098 sec elapsed
INFO [2021-10-29 05:15:12] Save as pqt by group_id
0.411 sec elapsed
INFO [2021-10-29 05:15:12] Finished file7
[1] 8
INFO [2021-10-29 05:15:12] Parsing JSON
INFO [2021-10-29 05:15:15] Cleaning data
INFO [2021-10-29 05:15:15] Loading to sql
INFO [2021-10-29 05:15:15] Updating [data.files] from file 8
INFO [2021-10-29 05:15:15] Updating [data.bonds] from file 8
INFO [2021-10-29 05:15:15] Updating [data.samples] from file 8
0.367 sec elapsed
INFO [2021-10-29 05:15:16] Save as pqt all
0.107 sec elapsed
INFO [2021-10-29 05:15:16] Save as pqt by group_id
0.47 sec elapsed
INFO [2021-10-29 05:15:16] Finished file8
[1] 9
INFO [2021-10-29 05:15:16] Parsing JSON
INFO [2021-10-29 05:15:20] Cleaning data
INFO [2021-10-29 05:15:20] Loading to sql
INFO [2021-10-29 05:15:20] Updating [data.files] from file 9
INFO [2021-10-29 05:15:20] Updating [data.bonds] from file 9
INFO [2021-10-29 05:15:20] Updating [data.samples] from file 9
0.462 sec elapsed
INFO [2021-10-29 05:15:20] Save as pqt all
0.206 sec elapsed
INFO [2021-10-29 05:15:21] Save as pqt by group_id
0.501 sec elapsed
INFO [2021-10-29 05:15:21] Finished file9
[1] 10
INFO [2021-10-29 05:15:21] Parsing JSON
INFO [2021-10-29 05:15:24] Cleaning data
INFO [2021-10-29 05:15:24] Loading to sql
INFO [2021-10-29 05:15:24] Updating [data.files] from file 10
INFO [2021-10-29 05:15:24] Updating [data.bonds] from file 10
INFO [2021-10-29 05:15:24] Updating [data.samples] from file 10
0.52 sec elapsed
INFO [2021-10-29 05:15:25] Save as pqt all
0.21 sec elapsed
INFO [2021-10-29 05:15:25] Save as pqt by group_id
0.469 sec elapsed
INFO [2021-10-29 05:15:25] Finished file10
[1] 11
INFO [2021-10-29 05:15:25] Parsing JSON
INFO [2021-10-29 05:15:28] Cleaning data
INFO [2021-10-29 05:15:28] Loading to sql
INFO [2021-10-29 05:15:28] Updating [data.files] from file 11
INFO [2021-10-29 05:15:28] Updating [data.bonds] from file 11
INFO [2021-10-29 05:15:28] Updating [data.samples] from file 11
0.409 sec elapsed
INFO [2021-10-29 05:15:29] Save as pqt all
0.21 sec elapsed
INFO [2021-10-29 05:15:29] Save as pqt by group_id
0.571 sec elapsed
INFO [2021-10-29 05:15:30] Finished file11
[1] 12
INFO [2021-10-29 05:15:30] Parsing JSON
INFO [2021-10-29 05:15:32] Cleaning data
INFO [2021-10-29 05:15:32] Loading to sql
INFO [2021-10-29 05:15:32] Updating [data.files] from file 12
INFO [2021-10-29 05:15:32] Updating [data.bonds] from file 12
INFO [2021-10-29 05:15:32] Updating [data.samples] from file 12
0.328 sec elapsed
INFO [2021-10-29 05:15:33] Save as pqt all
0.099 sec elapsed
INFO [2021-10-29 05:15:33] Save as pqt by group_id
0.447 sec elapsed
INFO [2021-10-29 05:15:33] Finished file12
[1] 13
INFO [2021-10-29 05:15:33] Parsing JSON
INFO [2021-10-29 05:15:37] Cleaning data
INFO [2021-10-29 05:15:37] Loading to sql
INFO [2021-10-29 05:15:37] Updating [data.files] from file 13
INFO [2021-10-29 05:15:37] Updating [data.bonds] from file 13
INFO [2021-10-29 05:15:37] Updating [data.samples] from file 13
0.476 sec elapsed
INFO [2021-10-29 05:15:37] Save as pqt all
0.144 sec elapsed
INFO [2021-10-29 05:15:37] Save as pqt by group_id
0.361 sec elapsed
INFO [2021-10-29 05:15:38] Finished file13
[1] 14
INFO [2021-10-29 05:15:38] Parsing JSON
INFO [2021-10-29 05:15:41] Cleaning data
INFO [2021-10-29 05:15:42] Loading to sql
INFO [2021-10-29 05:15:42] Updating [data.files] from file 14
INFO [2021-10-29 05:15:42] Updating [data.bonds] from file 14
INFO [2021-10-29 05:15:42] Updating [data.samples] from file 14
0.539 sec elapsed
INFO [2021-10-29 05:15:42] Save as pqt all
0.114 sec elapsed
INFO [2021-10-29 05:15:42] Save as pqt by group_id
0.69 sec elapsed
INFO [2021-10-29 05:15:43] Finished file14
[1] 15
INFO [2021-10-29 05:15:43] Parsing JSON
INFO [2021-10-29 05:15:46] Cleaning data
INFO [2021-10-29 05:15:46] Loading to sql
INFO [2021-10-29 05:15:46] Updating [data.files] from file 15
INFO [2021-10-29 05:15:46] Updating [data.bonds] from file 15
INFO [2021-10-29 05:15:46] Updating [data.samples] from file 15
0.374 sec elapsed
INFO [2021-10-29 05:15:47] Save as pqt all
0.113 sec elapsed
INFO [2021-10-29 05:15:47] Save as pqt by group_id
0.399 sec elapsed
INFO [2021-10-29 05:15:47] Finished file15
There are two schemas:
files
, bonds
, and samples
table. This is saved out as both sqlite and parquetbonds
is joined to samples
Set up SQL connection for star schema:
db_path = here::here('output/data.db')
# pool <- dbPool(RSQLite::SQLite(), db=db_path)
con <- DBI::dbConnect(RSQLite::SQLite(), db=db_path)
Explore schema of tables:
dbGetQuery(con, 'PRAGMA table_info(files)')
#> cid name type notnull dflt_value pk
#> 1 0 file_id INTEGER 0 NA 0
#> 2 1 start_date_time REAL 0 NA 0
#> 3 2 uut_serial_number TEXT 0 NA 0
#> 4 3 station_id TEXT 0 NA 0
#> 5 4 bond_count INTEGER 0 NA 0
dbGetQuery(con, 'PRAGMA table_info(bonds)')
#> cid name type notnull dflt_value pk
#> 1 0 file_id INTEGER 0 NA 0
#> 2 1 bond_id INTEGER 0 NA 0
#> 3 2 process_program TEXT 0 NA 0
#> 4 3 zone_number INTEGER 0 NA 0
#> 5 4 wire_number INTEGER 0 NA 0
#> 6 5 bond_number INTEGER 0 NA 0
#> 7 6 bond_group_id TEXT 0 NA 0
#> 8 7 date_time REAL 0 NA 0
#> 9 8 wire_feed INTEGER 0 NA 0
#> 10 9 pull_test_status TEXT 0 NA 0
#> 11 10 program_rev TEXT 0 NA 0
#> 12 11 cell_id_mod TEXT 0 NA 0
#> 13 12 brick_id_mod INTEGER 0 NA 0
#> 14 13 cell_polarity TEXT 0 NA 0
#> 15 14 cp_id_mod TEXT 0 NA 0
#> 16 15 trace_sample_count INTEGER 0 NA 0
#> 17 16 rework_trigger_msg TEXT 0 NA 0
dbGetQuery(con, 'PRAGMA table_info(samples)')
#> cid name type notnull dflt_value pk
#> 1 0 file_id INTEGER 0 NA 0
#> 2 1 bond_id INTEGER 0 NA 0
#> 3 2 sample INTEGER 0 NA 0
#> 4 3 dfrm REAL 0 NA 0
#> 5 4 driv INTEGER 0 NA 0
#> 6 5 freq REAL 0 NA 0
#> 7 6 phas REAL 0 NA 0
#> 8 7 us INTEGER 0 NA 0
#> 9 8 volt INTEGER 0 NA 0
#> 10 9 alcpt REAL 0 NA 0
You can write SQL to read the data directly:
select *
from files
limit 10
| file_id | start_date_time | uut_serial_number | station_id | bond_count |
|---|---|---|---|---|
| 1 |<PHONE_NUMBER> | 097192 | M364110709 | 888 |
| 2 |<PHONE_NUMBER> | 103118 | M364111210 | 888 |
| 3 |<PHONE_NUMBER> | 145928 | M364111202 | 888 |
| 4 |<PHONE_NUMBER> | 108025 | M364111210 | 888 |
| 5 |<PHONE_NUMBER> | 097231 | M364111107 | 888 |
| 6 |<PHONE_NUMBER> | 144167 | M364120804 | 888 |
| 7 |<PHONE_NUMBER> | 103279 | AST215060003 | 888 |
| 8 |<PHONE_NUMBER> | 111696 | M362100702 | 888 |
| 9 |<PHONE_NUMBER> | 143550 | M364111205 | 888 |
| 10 |<PHONE_NUMBER> | 097298 | M364111205 | 888 |
I saved the star schema out as a parquet files, one parquet file per file_id. There are 15 files per bonds/
, files/
and samples/
folders.
dir_output <- "/home/rstudio/tesla/bonds/output"
dir_pqt <- here::here(dir_output, "pqt")
dir_pqt_all <- here::here(dir_pqt, "all")
dir_pqt_grpid <- here::here(dir_pqt, "grpid")
fs::dir_ls(dir_pqt_all)
#> /home/rstudio/tesla/bonds/output/pqt/all/bonds
#> /home/rstudio/tesla/bonds/output/pqt/all/files
#> /home/rstudio/tesla/bonds/output/pqt/all/samples
Inspecting the files
folder:
fs::dir_ls(fs::path(dir_pqt_all, 'files'))
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00001.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00002.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00003.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00004.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00005.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00006.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00007.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00008.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00009.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00010.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00011.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00012.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00013.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00014.parquet
#> /home/rstudio/tesla/bonds/output/pqt/all/files/part-00015.parquet
I wrote a function to read the parquet files:
load_pqt(dir_pqt_all, 'files')
#> # A tibble: 15 × 5
#> file_id start_date_time uut_serial_number station_id bond_count
#> <int> <dttm> <chr> <chr> <int>
#> 1 1 2016-01-06 14:04:36 097192 M364110709 888
#> 2 2 2016-01-08 04:30:57 103118 M364111210 888
#> 3 3 2016-01-22 08:22:05 145928 M364111202 888
#> 4 4 2016-01-10 05:39:49 108025 M364111210 888
#> 5 5 2016-01-06 17:09:08 097231 M364111107 888
#> 6 6 2016-01-21 07:25:25 144167 M364120804 888
#> 7 7 2016-01-11 17:52:28 103279 AST215060003 888
#> 8 8 2016-01-11 19:07:48 111696 M362100702 888
#> 9 9 2016-01-20 13:09:59 143550 M364111205 888
#> 10 10 2016-01-06 22:46:12 097298 M364111205 888
#> 11 11 2016-01-21 21:58:52 148533 M364111204 888
#> 12 12 2016-01-29 13:42:09 156205 AST215060001 888
#> 13 13 2016-01-20 13:04:16 143581 M364111107 888
#> 14 14 2016-01-26 01:07:40 148985 M364111209 889
#> 15 15 2016-01-29 12:27:41 167333 M364110707 888
I also wrote an interface to allow the readers to filter the parquet files based on the file name. If I rename the parquet files based on the datetime of the file_id, this would enable the data scientists to filter to the time range very easily.
load_pqt(dir_pqt_all, 'files', range=4:7)
#> # A tibble: 4 × 5
#> file_id start_date_time uut_serial_number station_id bond_count
#> <int> <dttm> <chr> <chr> <int>
#> 1 4 2016-01-10 05:39:49 108025 M364111210 888
#> 2 5 2016-01-06 17:09:08 097231 M364111107 888
#> 3 6 2016-01-21 07:25:25 144167 M364120804 888
#> 4 7 2016-01-11 17:52:28 103279 AST215060003 888
Read in the bonds table:
df_bonds <- load_pqt(dir_pqt_all, 'bonds')
df_bonds %>%
head()
#> # A tibble: 6 × 17
#> file_id bond_id process_program zone_number wire_number bond_number
#> <int> <int> <chr> <int> <int> <int>
#> 1 1 1 Z06_TS_FP_KS_3600_1_1_4 6 1 1
#> 2 1 9 Z06_TS_FP_KS_3600_1_1_4 6 1 2
#> 3 1 17 Z06_TS_FP_KS_3600_1_1_4 6 2 1
#> 4 1 25 Z06_TS_FP_KS_3600_1_1_4 6 2 2
#> 5 1 33 Z06_TS_FP_KS_3600_1_1_4 6 3 1
#> 6 1 41 Z06_TS_FP_KS_3600_1_1_4 6 3 2
#> # … with 11 more variables: bond_group_id <chr>, date_time <dttm>,
#> # wire_feed <int>, pull_test_status <chr>, program_rev <chr>,
#> # cell_id_mod <chr>, brick_id_mod <int>, cell_polarity <chr>,
#> # cp_id_mod <chr>, trace_sample_count <int>, rework_trigger_msg <chr>
If the DS team cares to model a specific group_id with samples, I partitioned the files by group_id in output/pqt/grpid/<group_id>
:
fs::dir_ls(dir_pqt_grpid)
#> /home/rstudio/tesla/bonds/output/pqt/grpid/12mil_CP
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Anode-OPT3-300(12)
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Anode_V313_V3_Minus
#> /home/rstudio/tesla/bonds/output/pqt/grpid/CP-Rev3-300(12)
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Cathode-Rev3-300(12)
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Cathode_V313_V2_Minus
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Ni-CP-Rev6-300(12)
For example, you can see how some file_ids contained a group_id
and others didn’t:
fs::dir_ls(dir_pqt_grpid, recurse=T) %>%
head(10)
#> /home/rstudio/tesla/bonds/output/pqt/grpid/12mil_CP
#> /home/rstudio/tesla/bonds/output/pqt/grpid/12mil_CP/part-00007.parquet
#> /home/rstudio/tesla/bonds/output/pqt/grpid/12mil_CP/part-00012.parquet
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Anode-OPT3-300(12)
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Anode-OPT3-300(12)/part-00001.parquet
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Anode-OPT3-300(12)/part-00002.parquet
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Anode-OPT3-300(12)/part-00003.parquet
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Anode-OPT3-300(12)/part-00004.parquet
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Anode-OPT3-300(12)/part-00005.parquet
#> /home/rstudio/tesla/bonds/output/pqt/grpid/Anode-OPT3-300(12)/part-00006.parquet
The data scientists can easily load in all the samples for a bond_id. This table replicates the metadata for each bond_id across each trace sample. This makes the data easily to model.
load_pqt(dir_root=dir_pqt_grpid, tab='12mil_CP') %>%
head(20)
#> # A tibble: 20 × 24
#> file_id bond_id process_program zone_number wire_number bond_number
#> <int> <int> <chr> <int> <int> <int>
#> 1 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 2 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 3 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 4 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 5 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 6 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 7 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 8 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 9 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 10 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 11 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 12 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 13 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 14 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 15 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 16 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 17 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 18 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 19 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> 20 7 8 NTS_2000CELL_KS_ASTERION… 99 1 2
#> # … with 18 more variables: bond_group_id <chr>, date_time <dttm>,
#> # wire_feed <int>, pull_test_status <chr>, program_rev <chr>,
#> # cell_id_mod <chr>, brick_id_mod <int>, cell_polarity <chr>,
#> # cp_id_mod <chr>, trace_sample_count <int>, SAMPLE <int>, DFRM <dbl>,
#> # DRIV <int>, FREQ <dbl>, PHAS <dbl>, US <int>, VOLT <int>, ALCPT <dbl>
The following snippets explore the star schema data:
df_bonds <- load_pqt(dir_root = dir_pqt_all, 'bonds')
df_samples <- load_pqt(dir_root = dir_pqt_all, 'samples')
# 7 bond_group_ids
df_bonds %>%
summarize(n_distinct(bond_group_id))
#> # A tibble: 1 × 1
#> `n_distinct(bond_group_id)`
#> <int>
#> 1 7
# sample: need to up/downsample
# filter to the failed samples, save those in one file
# extreme class imbalance
df_bonds %>%
count(pull_test_status)
#> # A tibble: 2 × 2
#> pull_test_status n
#> <chr> <int>
#> 1 Fail 1
#> 2 Pass 13320
# there are 3548 bonds
df_bonds %>%
summarize(n_distinct(bond_id))
#> # A tibble: 1 × 1
#> `n_distinct(bond_id)`
#> <int>
#> 1 3548
# there are up to 4600 samples per bond_id.
df_samples %>%
count(bond_id, sort=T)
#> # A tibble: 3,548 × 2
#> bond_id n
#> <int> <int>
#> 1 1408 4645
#> 2 1628 4550
#> 3 1562 4295
#> 4 1342 4200
#> 5 1364 4200
#> 6 1386 4200
#> 7 1430 4200
#> 8 1452 4200
#> 9 1474 4200
#> 10 1496 4200
#> # … with 3,538 more rows
# what's the relationship between bonds and samples?
# multiple bond_ids
df_bonds %>%
group_by(bond_group_id) %>%
summarize(n_distinct(bond_id))
#> # A tibble: 7 × 2
#> bond_group_id `n_distinct(bond_id)`
#> <chr> <int>
#> 1 12mil_CP 888
#> 2 Anode_V313_V3_Minus 433
#> 3 Anode-OPT3-300(12) 758
#> 4 Cathode_V313_V2_Minus 430
#> 5 Cathode-Rev3-300(12) 690
#> 6 CP-Rev3-300(12) 1247
#> 7 Ni-CP-Rev6-300(12) 228
# what's the primary key of the bonds table?
# file_id, bond_id
count(df_bonds)
#> # A tibble: 1 × 1
#> n
#> <int>
#> 1 13321
df_bonds %>%
distinct(file_id, bond_id) %>%
count()
#> # A tibble: 1 × 1
#> n
#> <int>
#> 1 13321
# what is the primary key of bonds table without file_id?
# bond_id, bond_group_id, and date_time make a PK
df_bonds %>%
distinct(bond_id, bond_group_id, date_time) %>%
count()
#> # A tibble: 1 × 1
#> n
#> <int>
#> 1 13321
This section shows how a DS could model the data using the schema I’ve provided.
There are two simple ways to model the data:
Upon inspecting the data, it appears there’s severe class imbalance (only one Fail
per 13321 bonds). This data would be nearly impossible to model as is, so either we’d need to gather more data or would have to do up-sample the samples
for the Fail status, downsample the samples
for the Pass status, or some mixture of both.
Since only Ni-CP-Rev6-300(12)
has a Fail
test status, we’ll build a model on that group_id.
df_bonds %>%
filter(pull_test_status == 'Fail') %>%
pull(bond_group_id)
#> [1] "Ni-CP-Rev6-300(12)"
Load data:
df_ni <- load_pqt(dir_root = dir_pqt_grpid, tab='Ni-CP-Rev6-300(12)')
There are 189 trace samples with label pull_test_status=Fail
.
df_ni %>%
group_by(pull_test_status) %>%
count()
#> # A tibble: 2 × 2
#> # Groups: pull_test_status [2]
#> pull_test_status n
#> <chr> <int>
#> 1 Fail 189
#> 2 Pass 326403
I perform classification with tidymodels, comparing both logistic regression (glm) with decision trees (xgboost). I query only a few numeric variables to keep things simple:
library(tidymodels)
#> Registered S3 method overwritten by 'tune':
#> method from
#> required_pkgs.model_spec parsnip
#> ── Attaching packages ────────────────────────────────────── tidymodels 0.1.4 ──
#> ✓ broom 0.7.9 ✓ recipes 0.1.17
#> ✓ dials 0.0.10 ✓ rsample 0.1.0
#> ✓ ggplot2 3.3.5 ✓ tibble 3.1.5
#> ✓ infer 1.0.0 ✓ tune 0.1.6
#> ✓ modeldata 0.1.1 ✓ workflows 0.2.4
#> ✓ parsnip 0.1.7 ✓ workflowsets 0.1.0
#> ✓ purrr 0.3.4 ✓ yardstick 0.0.8
#> ── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
#> x glue::collapse() masks dplyr::collapse()
#> x purrr::discard() masks scales::discard()
#> x dplyr::filter() masks stats::filter()
#> x recipes::fixed() masks stringr::fixed()
#> x purrr::flatten() masks jsonlite::flatten()
#> x dplyr::lag() masks stats::lag()
#> x recipes::step() masks stats::step()
#> • Use suppressPackageStartupMessages() to eliminate package startup messages
library(xgboost)
#>
#> Attaching package: 'xgboost'
#> The following object is masked from 'package:dplyr':
#>
#> slice
# Convert pull_test_status to be 0-1
df_model <- df_ni %>%
mutate(
pull_test_status = as.factor(pull_test_status)
) %>%
select(
pull_test_status,
zone_number,
wire_number,
DRIV,
PHAS,
VOLT)
# explore distribution of the target variable
with(df_model, table(pull_test_status))
#> pull_test_status
#> Fail Pass
#> 189 326403
set.seed(100)
# Put 3/4 of the data into the training set
data_split <- initial_split(df_model,
prop = .7,
strata = pull_test_status)
# Create dataframes for the two sets:
train_data <- training(data_split)
test_data <- testing(data_split)
# specify target variable and training data
# Downsample the data: https://themis.tidymodels.org/reference/step_downsample.html
# recipes are for how data will be modeled and processed
y_recipe <- recipe(pull_test_status ~ ., data = train_data) %>%
themis::step_downsample(pull_test_status, under_ratio = 1)
#> Registered S3 methods overwritten by 'themis':
#> method from
#> bake.step_downsample recipes
#> bake.step_upsample recipes
#> prep.step_downsample recipes
#> prep.step_upsample recipes
#> tidy.step_downsample recipes
#> tidy.step_upsample recipes
#> tunable.step_downsample recipes
#> tunable.step_upsample recipes
# ensure downsampling is done right: this proves i sampled
# a small number of Pass variables
y_recipe %>%
prep(training = train_data, retain = TRUE) %>%
bake(new_data = NULL) %>%
with(., table(pull_test_status))
#> pull_test_status
#> Fail Pass
#> 129 129
# Specify model types
mod_glm <- logistic_reg() %>%
set_engine('glm') %>%
set_mode('classification')
# fit a model with downsampeld data
downsampled_data <- y_recipe %>%
prep(training = train_data, retain=T) %>%
bake(new_data=NULL)
mod_glm %>%
fit(pull_test_status ~ ., data=downsampled_data)
#> Warning: glm.fit: algorithm did not converge
#> Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
#> parsnip model object
#>
#> Fit time: 12ms
#>
#> Call: stats::glm(formula = pull_test_status ~ ., family = stats::binomial,
#> data = data)
#>
#> Coefficients:
#> (Intercept) zone_number wire_number DRIV PHAS VOLT
#> 233.2565 -23.5407 -0.7302 -0.7556 -14.9039 6.3849
#>
#> Degrees of Freedom: 257 Total (i.e. Null); 252 Residual
#> Null Deviance: 357.7
#> Residual Deviance: 1.184e-08 AIC: 12
Compare GLM to XGB:
# Specify model types
mod_glm <- logistic_reg() %>%
set_engine('glm') %>%
set_mode('classification')
mod_xgb <- boost_tree() %>%
set_engine("xgboost") %>%
set_mode("classification")
# Create the workflows
wflow_glm <- workflow() %>%
add_recipe(y_recipe) %>%
add_model(mod_glm)
wflow_xgb <- workflow() %>%
add_recipe(y_recipe) %>%
add_model(mod_xgb)
# Define cross-validation
cv_folds <- vfold_cv(train_data, v = 5, strata = pull_test_status)
# Model the data, saving out the resutls
results_glm <-
wflow_glm %>%
fit_resamples(
resamples = cv_folds,
metrics = metric_set(
recall, precision, f_meas,
accuracy, kap,
roc_auc, sens, spec),
control = control_resamples(
save_pred = TRUE)
)
#> ! Fold1: preprocessor 1/1, model 1/1: glm.fit: algorithm did not converge, glm.fi...
#> ! Fold2: preprocessor 1/1, model 1/1: glm.fit: algorithm did not converge, glm.fi...
#> ! Fold3: preprocessor 1/1, model 1/1: glm.fit: algorithm did not converge, glm.fi...
#> ! Fold4: preprocessor 1/1, model 1/1: glm.fit: algorithm did not converge, glm.fi...
#> ! Fold5: preprocessor 1/1, model 1/1: glm.fit: algorithm did not converge, glm.fi...
# results_glm$.notes %>% as.character()
results_xgb <- wflow_xgb %>%
fit_resamples(
resamples = cv_folds,
metrics = metric_set(
recall, precision, f_meas,
accuracy, kap,
roc_auc, sens, spec),
control = control_resamples(
save_pred = TRUE)
)
# results_xgb$.notes %>% as.character()
GLM/XGBoost results - both models seem comparable, but seem suspiciously good:
results_glm %>%
collect_metrics()
#> # A tibble: 8 × 6
#> .metric .estimator mean n std_err .config
#> <chr> <chr> <dbl> <int> <dbl> <chr>
#> 1 accuracy binary 0.991 5 0.00271 Preprocessor1_Model1
#> 2 f_meas binary 0.136 5 0.0356 Preprocessor1_Model1
#> 3 kap binary 0.135 5 0.0356 Preprocessor1_Model1
#> 4 precision binary 0.0751 5 0.0215 Preprocessor1_Model1
#> 5 recall binary 0.986 5 0.0138 Preprocessor1_Model1
#> 6 roc_auc binary 0.991 5 0.00694 Preprocessor1_Model1
#> 7 sens binary 0.986 5 0.0138 Preprocessor1_Model1
#> 8 spec binary 0.991 5 0.00271 Preprocessor1_Model1
results_xgb %>%
collect_metrics()
#> # A tibble: 8 × 6
#> .metric .estimator mean n std_err .config
#> <chr> <chr> <dbl> <int> <dbl> <chr>
#> 1 accuracy binary 0.988 5 0.00146 Preprocessor1_Model1
#> 2 f_meas binary 0.0920 5 0.0164 Preprocessor1_Model1
#> 3 kap binary 0.0910 5 0.0164 Preprocessor1_Model1
#> 4 precision binary 0.0485 5 0.00906 Preprocessor1_Model1
#> 5 recall binary 1 5 0 Preprocessor1_Model1
#> 6 roc_auc binary 0.999 5 0.000184 Preprocessor1_Model1
#> 7 sens binary 1 5 0 Preprocessor1_Model1
#> 8 spec binary 0.988 5 0.00146 Preprocessor1_Model1
Assess predictions - we notice a lot of overfitting in the model. There’s like a single feature driving everything (sign of feature leakage):
preds_glm <- results_glm %>%
collect_predictions()
preds_xgb <- results_xgb %>%
collect_predictions()
preds_glm %>%
conf_mat(pull_test_status, .pred_class)
#> Truth
#> Prediction Fail Pass
#> Fail 127 2111
#> Pass 2 226374
# GBM predicts 100% of the failed classes (likely overfit)
preds_xgb %>%
conf_mat(pull_test_status, .pred_class)
#> Truth
#> Prediction Fail Pass
#> Fail 129 2745
#> Pass 0 225740
# Roc Curves is upside down (likely because of some bug with how
# the target variable is labeled)
preds_glm %>%
group_by(id) %>%
roc_curve(pull_test_status, .pred_Pass) %>%
autoplot()
# ROC curve
preds_xgb %>%
group_by(id) %>%
roc_curve(pull_test_status, .pred_Pass) %>%
autoplot()
Let’s see what features come up as important when we train on the full data and predict on a test set.
last_fit_xgb <- last_fit(wflow_xgb,
split = data_split,
metrics = metric_set(
recall, precision, f_meas,
accuracy, kap,
roc_auc, sens, spec)
)
# Prediction metrics look good
last_fit_xgb %>%
collect_metrics()
#> # A tibble: 8 × 4
#> .metric .estimator .estimate .config
#> <chr> <chr> <dbl> <chr>
#> 1 recall binary 1 Preprocessor1_Model1
#> 2 precision binary 0.0408 Preprocessor1_Model1
#> 3 f_meas binary 0.0783 Preprocessor1_Model1
#> 4 accuracy binary 0.986 Preprocessor1_Model1
#> 5 kap binary 0.0772 Preprocessor1_Model1
#> 6 sens binary 1 Preprocessor1_Model1
#> 7 spec binary 0.986 Preprocessor1_Model1
#> 8 roc_auc binary 0.999 Preprocessor1_Model1
# Explore
last_fit_xgb %>%
collect_predictions() %>%
conf_mat(pull_test_status, .pred_class)
#> Truth
#> Prediction Fail Pass
#> Fail 60 1412
#> Pass 0 96506
# Explore feature importance
library(vip)
#>
#> Attaching package: 'vip'
#> The following object is masked from 'package:utils':
#>
#> vi
last_fit_xgb %>%
pluck(".workflow", 1) %>%
extract_fit_parsnip() %>%
vip(num_features = 20)
Volt is the top predictor. There’s likely something about volt and wire number that’s driving “perfect” predictions. Let’s look at the distribution of VOLT compared to the target variable.
# downsampled data is the train data
downsampled_data %>%
group_by(pull_test_status) %>%
skimr::skim()
| Name | Piped data |
| Number of rows | 258 |
| Number of columns | 6 |
| _______________________ | |
| Column type frequency: | |
| numeric | 5 |
| ________________________ | |
| Group variables | pull_test_status |
Variable type: numeric
| skim_variable | pull_test_status | n_missing | complete_rate | mean | sd | p0 | p25 | p50 | p75 | p100 | hist |
|---|---|---|---|---|---|---|---|---|---|---|---|
| zone_number | Fail | 0 | 1 | 4.00 | 0.00 | 4.00 | 4 | 4 | 4 | 4.00 | ▁▁▇▁▁ |
| zone_number | Pass | 0 | 1 | 2.56 | 1.29 | 1.00 | 1 | 2 | 4 | 4.00 | ▅▆▁▁▇ |
| wire_number | Fail | 0 | 1 | 72.00 | 0.00 | 72.00 | 72 | 72 | 72 | 72.00 | ▁▁▇▁▁ |
| wire_number | Pass | 0 | 1 | 37.67 | 23.66 | 1.00 | 17 | 35 | 57 | 81.00 | ▇▆▅▆▅ |
| DRIV | Fail | 0 | 1 | 1090.91 | 132.66 | 204.00 | 1125 | 1125 | 1125 | 1125.00 | ▁▁▁▁▇ |
| DRIV | Pass | 0 | 1 | 1097.05 | 118.58 | 204.00 | 1125 | 1125 | 1125 | 1125.00 | ▁▁▁▁▇ |
| PHAS | Fail | 0 | 1 | -0.05 | 0.18 | -0.49 | 0 | 0 | 0 | 0.49 | ▁▁▇▁▁ |
| PHAS | Pass | 0 | 1 | 0.01 | 0.21 | -1.47 | 0 | 0 | 0 | 0.49 | ▁▁▁▇▁ |
| VOLT | Fail | 0 | 1 | 107.75 | 15.94 | 6.00 | 106 | 113 | 114 | 116.00 | ▁▁▁▁▇ |
| VOLT | Pass | 0 | 1 | 122.57 | 16.18 | 7.00 | 125 | 125 | 127 | 131.00 | ▁▁▁▁▇ |
ggplot(downsampled_data, aes(x = VOLT, y = pull_test_status)) +
geom_point()
# wire number is driving the prediction
ggplot(downsampled_data,
aes(x = wire_number, y = pull_test_status)) +
geom_point(alpha = .1)
ggplot(downsampled_data,
aes(x = wire_number, y = VOLT, color=pull_test_status)) +
geom_point(alpha = .1)
with(downsampled_data, table(pull_test_status, wire_number))
#> wire_number
#> pull_test_status 1 2 4 5 6 7 8 9 10 11 12 13 14 15 16
#> Fail 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
#> Pass 2 2 5 1 1 1 2 2 3 3 2 1 3 3 1
#> wire_number
#> pull_test_status 17 19 21 22 24 25 26 27 28 29 30 31 34 35 36
#> Fail 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
#> Pass 4 2 6 2 1 1 1 1 1 4 3 3 3 1 3
#> wire_number
#> pull_test_status 37 38 42 44 46 47 49 50 52 54 55 56 57 58 60
#> Fail 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
#> Pass 2 2 3 2 2 1 1 2 3 3 4 2 4 2 2
#> wire_number
#> pull_test_status 61 63 64 66 68 69 70 72 73 74 75 77 78 79 80
#> Fail 0 0 0 0 0 0 0 129 0 0 0 0 0 0 0
#> Pass 1 2 3 2 1 1 1 1 1 3 2 3 1 2 1
#> wire_number
#> pull_test_status 81
#> Fail 0
#> Pass 1
downsampled_data %>%
group_by(pull_test_status, VOLT) %>%
count() %>%
arrange(desc(VOLT)) %>%
pivot_wider(names_from=pull_test_status, values_from=n)
#> # A tibble: 43 × 3
#> # Groups: VOLT [43]
#> VOLT Pass Fail
#> <int> <int> <int>
#> 1 131 2 NA
#> 2 130 22 NA
#> 3 129 8 NA
#> 4 127 8 NA
#> 5 126 17 NA
#> 6 125 40 NA
#> 7 124 13 NA
#> 8 123 2 NA
#> 9 122 2 NA
#> 10 121 2 NA
#> # … with 33 more rows
Wire number 72 seems to be the only place where there are failures, where VOLT is in a range between 112 and 116.
Ideas:
data/pqt/12mil_CP/part-20211020_152030-00001.parquet
. This would allow the DS team to easily choose dates/times and read in that data. (I’d create a function similar to the range
functionality, but filters parquet files between two dates.)I considered these alternatives for parsing json:
Other solutions: - Looked at airbyte and considered building an airbyte connector?
After considering the above, and playing around with a few common imports, I might try dask + prefect.


Welcome to Bookmarkly
Please sign in
×
Oops!
Invalid username or password.
Email address
Password
Sign in


Level
Segment
Circle
State 1
|
2
|
3
General View
Domains: Side View
Circles: Top View
Perspective
Deep Go
Random
|
State
|
+rebuild
Highlight Chunks
End State
Save State
Enable Mouse Selection
Animate
orbit
|
sequence
Prev
|
Next