Overview
The Omics Discovery Index (OmicsDI) provides dataset discovery across a heterogeneous, distributed group of genomics, proteomics and metabolomics data resources spanning eight repositories in two continents and six organisations, including both open and controlled access data resources. The resource provides a short description of every dataset: accession, description, sample/data protocols biological evidences, publication, etc. The search capabilities offer a unique resource to search for omics datasets. This fact converts the OmicsDI in the first resource worldwide to provide search capabilities through multi-omics experiments.
Major Partners
ProteomeXchange
The ProteomeXchange Consortium is a collaboration of currently three major mass spectrometry proteomics data repositories, PRIDE at EMBL-EBI in Cambridge (UK), PeptideAtlas at ISB in Seattle (US), and MASSive at UCSD (US), offering a unified data deposition and discovery strategy across all three repositories. ProteomeXchange is a distributed database infrastructure; the potentially very large raw data component of the data is only held at the original submission database, while the searchable metadata is centrally collected and indexed. All ProteomeXchange data is fully open after release of the associated publication.
MetabolomeXchange
MetabolomeXchange is a collaboration of 4 major metabolomics repositories, with a total of 10 partners contributing. MetabolomeXchange was inspired by and is implementing similar coordination strategies to ProteomeXchange. The founding partners are MetaboLights at EMBL-EBI(UK), Metabolomics Repository Bordeaux(FR), Golm Metabolome Database and the Metabolomics Workbench (US). The Metabolomics Workbench is a NIH funded collaboration of 6 Regional Comprehensive Metabolomics Resource Cores. MetabolomeXchange started accepting metadata submissions in summer 2014, and reached 200 public datasets in March 2015.
The European Genome-Phenome Archive
The European Genome-Phenome Archive (EGA) provides a service for the permanent archiving and distribution of personally identifiable genetic and phenotypic data resulting from biomedical research projects. Data at EGA was collected from individuals whose consent agreements authorise data release only for specific research use to bona fide researchers. Strict protocols govern how information is managed, stored and distributed by the EGA project. The EGA comprises a public metadata section, allowing searching and identifying relevant studies, and the controlled access data section. Access to the data section for a particular study is only granted after validation of a research proposal through the relevant ethics approval.
Current Databases
The original OmicsDI project provides access to eleven different databases with proteomics, genomics, transcriptomics and metabolomics data from Europe and Unite States (see updated list ). The original list includes: PRIDE (proteomics, UK), PeptideAtlas (proteomics, US), MassIVE (proteomics, US), GPMDB (proteomics, Canada), Metaboligths (metabolomics, UK), MetabolomeWorkbench (metabolomics, US), GNPS (metabolomics, USA), MetabolomeExpress (metabolomics, Australia), ArrayExpress and GEO (transcriptomics, genomics, UK), ExpressionAtlas (UK), EGA (genomics, UK).
The project is open for new partners and databases (please contact us: omicsdi-support@ebi.ac.uk). If you are interested in the architecture and the metadata that the resource should provide see section 2.3 and 2.4 in "Specifications" of OmicsDI.
OmicsDI web application
The main goal of OmicsDI project is to have a way to search interesting datasets across omics repositories. The main web application and web service (see here) allow the user to search and navigate through the OmicsDI datasets. The OmicsDI web application offers two main access modes: (i) using the home page navigation panels or (ii) the search box.
Navigate the data from home page
The OmicsDI home page provides different blocks to navigate through the datasets, some of them are: 2D WordCloud; the species/organism/diseases bubble chart, repo/omics pie chart, Latest datasets, Most accessed datasets, Datasets per year. All the charts allow the user to search the data using the specific attribute. These boxes also act as an statistic component of the resource : for example the pie chart shows how many datasets for each repository and omics the resource contains.
A TagCloud or WordCloud is a visual representation for metadata, typically used to depict keyword metadata (tags) on datasets, or to visualize free form text. The WordCloud is build using the more frequently words for every database/repository. The OmicsDI WordCloud can be consider as a two dimensional term representation where the user can select the database and the field they want to look for: description vs database. The user can click the highlight word in the wordcloud to search for this term in the resource.
The bubble chart block allows the users to navigate the data using three main categories: Tissues, Organisms, and Diseases. The user can click in the bubble and it will be redirected to the search using the clicked term.
The Repo/Omics pie chart and the Omics vs Year bar chart allow the users navigate the data using the omics categories (metabolomics, transcriptomics, proteomics and genomics). The user can click a bar or the pie and it will be redirected to the search using the clicked term.
The Latest Datasets and Most accessed datasets blocks provide a list of the datasets by the tow categories.
Searching datasets
The main search box in OmicsDI allows the user to search datasets using different keywords. The main search redirects the user to the browser page where the user can see the results of the search (see section 3.3).
The OmicsDI search page is better that most of the partners (proteomeXchange, metabolomeXchange)
searching the data because the data is also indexed using cross-references to other databases.
Searching using publication details
The user can use the PubMed identifier, Title, Authors or even terms from the publication abstract.
Searching using biological Evidences
The search box allows the end-users to search data using biological evidences such as the list of the proteins identified in the proteomics experiment or the metabolite reported in the metabolomics experiment. For example if the user search for 3-methyl-2-oxobutanoic in the resource it will found one dataset in Metaboligths and five in Metabolome workbench that identified the current molecule.
Refining the Search results
The search results can be filter or refine using different categories, filters or terms. The OmicsDI web application supports at the moment nine different refinements: (1) omics type, (2) repository/database, (3) organisms, (4) tissue, diseases, (5) modifications (proteomics), (6) instruments and platforms, (7)publication data, (8) technology type.
The refine filters works in combination, if the user set two filters they will act at the same time.
Dataset View
The dataset View shows the information for every dataset in the resource. It contains four main information components: (1) the dataset description, (2) the sample/data protocol, (3) the publication information, and (4) the similar datasets.
OmicsDI Architecture
The OmicsDI project design is a modular architecture with three main components (Figure 1): (i) Database/Repository Adapters, (ii) EBI Search Indexer, (iii) OmicsDI Application. In summary, the Database/Repository Adapters provides all the libraries and readers to translate the original repository/databases information into a universal and highly customizable XML file for each dataset (see section 5.1). The EBI Search Indexer is a Lucene-based framework that enables to index all the metadata and biological evidences for each dataset (see section 5.2). The OmicsDI application component provides a MongoDB, web-services and web application for search and access the different datasets.
Database/Repository Adapters
The database/repository Adapter components are a set of readers and libraries that translate the data from the original repositories to a common XML-based file. The OmicsDI-XML is a fully customizable XML file used to represent the data from the different datasets, it contains as mandatory for each dataset:
- Dataset ID
- Dataset Title
- Type of experiment
- Publication date
- Submitter details
EBI Search Indexer
In summary, the indexing pipeline:
- Based on Apache Lucene
- An automatized indexing cycle takes place on daily basis.
- Check every night, for every domain if new data is available.
- The process benefits from the indexing parallelization.
- All domains together: ~750GB, more than 300GB indexes
- While the software releases are less frequent, the data to index is checked dail


Cybernetics Resolution
Home
Please fill out the following details
Name:
Phone no:
Select City:
--None Selected--
Delhi
Ghaziabad
Noida
Gurgaon
Faridabad
Gautam Budh Nagar
Select Area:
--None Selected--
Submit


Meme game: Here's what desi Twitterati will tell their kids and so should you
by Entertainment DeskNow, we’ve all seen those “I’m going to tell my kids” memes flooding social media. It’s become evident that the current generation plans on feeding their children with hilarious misrepresentations of the world as it is right now, and well, have a lot of them.
We are talking about the generation that came up with memes in the first place; the generation that coined the terms ‘nibbas’ and ‘nibbis’, the very generation that gave a whole new definition to Iqbal’s Shaheens and made sure Pooja behaved. So there’s no denying that this generation is bound to twist history more than there forefathers ever could with their history books.
Let’s just take a look at how misinformed our future generations are going to be.
POLITICAL MISINFORMATION
Panama is an Island and no one argues.
Well, how else could we have discovered that the amount of rain is directly proportional to the amount of water?
CELEBRITY MISINFORMATION
You’ll be lying if you said he wasn’t
For all the 90’s kids, no wand could beat the power of the pencil
Okay, this is just inhumane
Because he is..
Who even came up with these?
Because Pooja won more hearts with her attention seeking techniques than any Kardashian ever could.
CORPORATE MISINFORMATION
Lets just say we upgrade entrepreneurs like Apple upgrades its phones..
This is how it actually happened.
SPORTS MISINFORMATION
Aren’t they adorable?
And last but not the least..


ngPluralize
is a directive that displays messages according to en-US localization rules.
These rules are bundled with angular.js and the rules can be overridden
(see Angular i18n dev guide). You configure ngPluralize directive
by specifying the mappings between
plural categories and the strings to be displayed.
There are two plural categories in Angular's default en-US locale: "one" and "other".
While a pural category may match many numbers (for example, in en-US locale, "other" can match any number that is not 1), an explicit number rule can only match one number. For example, the explicit number rule for "3" matches the number 3. You will see the use of plural categories and explicit number rules throughout later parts of this documentation.
You configure ngPluralize by providing 2 attributes: count
and when
.
You can also provide an optional attribute, offset
.
The value of the count
attribute can be either a string or an Angular expression; these are evaluated on the current scope for its binded value.
The when
attribute specifies the mappings between plural categories and the actual
string to be displayed. The value of the attribute should be a JSON object so that Angular
can interpret it correctly.
The following example shows how to configure ngPluralize:
<ng-pluralize count="personCount" when="{'0': 'Nobody is viewing.', 'one': '1 person is viewing.', 'other': '{} people are viewing.'}"> </ng-pluralize>
In the example, "0: Nobody is viewing."
is an explicit number rule. If you did not
specify this rule, 0 would be matched to the "other" category and "0 people are viewing"
would be shown instead of "Nobody is viewing". You can specify an explicit number rule for
other numbers, for example 12, so that instead of showing "12 people are viewing", you can
show "a dozen people are viewing".
You can use a set of closed braces({}
) as a placeholder for the number that you want substituted
into pluralized strings. In the previous example, Angular will replace {}
with
{{personCount}}
. The closed braces {}
is a placeholder
for {{numberExpression}}.
The offset
attribute allows further customization of pluralized text, which can result in
a better user experience. For example, instead of the message "4 people are viewing this document",
you might display "John, Kate and 2 others are viewing this document".
The offset attribute allows you to offset a number by any desired value.
Let's take a look at an example:
<ng-pluralize count="personCount" offset=2 when="{'0': 'Nobody is viewing.', '1': '{{person1}} is viewing.', '2': '{{person1}} and {{person2}} are viewing.', 'one': '{{person1}}, {{person2}} and one other person are viewing.', 'other': '{{person1}}, {{person2}} and {} other people are viewing.'}"> </ng-pluralize>
Notice that we are still using two plural categories(one, other), but we added three explicit number rules 0, 1 and 2. When one person, perhaps John, views the document, "John is viewing" will be shown. When three people view the document, no explicit number rule is found, so an offset of 2 is taken off 3, and Angular uses 1 to decide the plural category. In this case, plural category 'one' is matched and "John, Marry and one other person are viewing" is shown.
Note that when you specify offsets, you must provide explicit number rules for numbers from 0 up to and including the offset. If you use an offset of 3, for example, you must provide explicit number rules for 0, 1, 2 and 3. You must also provide plural strings for plural categories "one" and "other".
<ng-pluralize count="{string|expression}" when="{string}" [offset="{number}"]> </ng-pluralize>
<ANY ng-pluralize count="{string|expression}" when="{string}" [offset="{number}"]> ... </ANY>
count – {string|expression} –
The variable to be bounded to.
when – {string} –
The mapping between plural category to its correspoding strings.
offset(optional) – {number=} –
Offset to deduct from the total number.

 Person 1:<input type="text" ng-model="person1" value="Igor" />
 Person 2:<input type="text" ng-model="person2" value="Misko" />
 Number of People:<input type="text" ng-model="personCount" value="1" />
  Without Offset: <ng-pluralize count="personCount" when="{'0': 'Nobody is viewing.', 'one': '1 person is viewing.', 'other': '{} people are viewing.'}"> </ng-pluralize>


Introduction
Cascading Style Sheets (CSS) is a style sheet language used for describing the presentation of a document written in a markup language like HTML.[1] CSS is a cornerstone technology of the World Wide Web, alongside HTML and JavaScript.
Selector
In CSS, selectors declare which part of the markup a style applies to by matching tags and attributes in the markup itself. Selectors may apply to the following:
- all elements of a specific type, e.g. the second-level headers h2
- elements specified by attribute, in particular:
- id: an identifier unique within the document
- class: an identifier that can annotate multiple elements in a document
- elements depending on how they are placed relative to others in the document tree.
Classes and IDs are case-sensitive, start with letters, and can include alphanumeric characters, hyphens and underscores. A class may apply to any number of instances of any elements. An ID may only be applied to a single element.
Pseudo-classes are used in CSS selectors to permit formatting based on information that is not contained in the document tree. One example of a widely used pseudo-class is :hover, which identifies content only when the user "points to" the visible element, usually by holding the mouse cursor over it. It is appended to a selector as in a:hover or #elementid:hover. A pseudo-class classifies document elements, such as :link or :visited, whereas a pseudo-element makes a selection that may consist of partial elements, such as ::first-line or ::first-letter.[6]
Selectors may be combined in many ways to achieve great specificity and flexibility.[7] Multiple selectors may be joined in a spaced list to specify elements by location, element type, id, class, or any combination thereof. The order of the selectors is important. For example, div .myClass {color: red;} applies to all elements of class myClass that are inside div elements, whereas .myClass div {color: red;} applies to all div elements that are in elements of class myClass.
Declaration Block
A declaration block consists of a list of declarations in braces. Each declaration itself consists of a property, a colon (:), and a value. If there are multiple declarations in a block, a semi-colon (;) must be inserted to separate each declaration.[9]
Properties are specified in the CSS standard. Each property has a set of possible values. Some properties can affect any type of element, and others apply only to particular groups of elements.[10]
Values may be keywords, such as "center" or "inherit", or numerical values, such as 200px (200 pixels), 50vw (50 percent of the viewport width) or 80% (80 percent of the parent element's width). Color values can be specified with keywords (e.g. "red"), hexadecimal values (e.g. #FF0000, also abbreviated as #F00), RGB values on a 0 to 255 scale (e.g. rgb(255, 0, 0)), RGBA values that specify both color and alpha transparency (e.g. rgba(255, 0, 0, 0.8)), or HSL or HSLA values (e.g. hsl(000, 100%, 50%), hsla(000, 100%, 50%, 80%)).[11]
Non-zero numeric values representing linear measures must include a length unit, which is either an alphabetic code or abbreviation, as in 200px or 50vw; or a percentage sign, as in 80%. Some units – cm (centimetre); in (inch); mm (millimetre); pc (pica); and pt (point) – are absolute, which means that the rendered dimension does not depend upon the structure of the page; others – em (em); ex (ex) and px (pixel) – are relative, which means that factors such as the font size of a parent element can affect the rendered measurement. These eight units were a feature of CSS 1[12] and retained in all subsequent revisions. The proposed CSS Values and Units Module Level 3 will, if adopted as a W3C Recommendation, provide seven further length units: ch; Q; rem; vh; vmax; vmin; and vw.[13]
Use
Before CSS, nearly all presentational attributes of HTML documents were contained within the HTML markup. All font colors, background styles, element alignments, borders and sizes had to be explicitly described, often repeatedly, within the HTML. CSS lets authors move much of that information to another file, the style sheet, resulting in considerably simpler HTML. For example, headings (h1 elements), sub-headings (h2), sub-sub-headings (h3), etc., are defined structurally using HTML. In print and on the screen, choice of font, size, color and emphasis for these elements is presentational. Before CSS, document authors who wanted to assign such typographic characteristics to, say, all h2 headings had to repeat HTML presentational markup for each occurrence of that heading type. This made documents more complex, larger, and more error-prone and difficult to maintain. CSS allows the separation of presentation from structure. CSS can define color, font, text alignment, size, borders, spacing, layout and many other typographic characteristics, and can do so independently for on-screen and printed views. CSS also defines non-visual styles, such as reading speed and emphasis for aural text readers. The W3C has now deprecated the use of all presentational HTML markup.[14]

 Chapter 1.

Using CSS, the same element can be coded using style properties instead of HTML presentational attributes:

 Chapter 1.

The advantages of this may not be immediately clear (since the second form is actually more verbose), but the power of CSS becomes more apparent when the style properties are placed in an internal style element or, even better, an external CSS file. For example, suppose the document contains the style element:

rather than by laboriously going through the document and changing the color for each individual h1 element. The styles can also be placed in an external CSS file, as described below, and loaded using syntax similar to:
<link href="path/to/file.css" rel="stylesheet" type="text/css">
Sources
CSS information can be provided from various sources. These sources can be the web browser, the user and the author. The information from the author can be further classified into inline, media type, importance, selector specificity, rule order, inheritance and property definition. CSS style information can be in a separate document or it can be embedded into an HTML document. Multiple style sheets can be imported. Different styles can be applied depending on the output device being used; for example, the screen version can be quite different from the printed version, so that authors can tailor the presentation appropriately for each medium. The style sheet with the highest priority controls the content display. Declarations not set in the highest priority source are passed on to a source of lower priority, such as the user agent style. The process is called cascading. One of the goals of CSS is to allow users greater control over presentation. Someone who finds red italic headings difficult to read may apply a different style sheet. Depending on the browser and the web site, a user may choose from various style sheets provided by the designers, or may remove all added styles and view the site using the browser's default styling, or may override just the red italic heading style without altering other attributes.
Specifity
Specificity refers to the relative weights of various rules.[15] It determines which styles apply to an element when more than one rule could apply. Based on specification, a simple selector (e.g. H1) has a specificity of 1, class selectors have a specificity of 1,0, and ID selectors a specificity of 1,0,0. Because the specificity values do not carry over as in the decimal system, commas are used to separate the "digits"[16] (a CSS rule having 11 elements and 11 classes would have a specificity of 11,11, not 121). Thus the following rules selectors result in the indicated specificity:
Consider this HTML fragment:
<!DOCTYPE html>

<meta charset="utf-8">

 To demonstrate specificity

In the above example, the declaration in the style attribute overrides the one in the


You walk into a large room with tables in the middle. The way the tables are arranged, it seems as if they serve as wall-less offices. Just not nearly as fancy as you would find at a tech firm.
You are looking at an exciting nicknack on someone's desk when a hand grabs your shoulder. You turn quickly to see a man smiling at you. "Can I help You?" He asks.
You look at his name tag. It says Branch Manager. Melanie sent me! You quickly say, as if you are confessing to a crime. "Really? Come have a seat in my office."


School Application Form
Name:
School:
Gender:
Male
Female
State of Origin:
Select
Abuja
Kaduna
Kano
Phone:
Email:
JAMB:
WAEC:
First Choice:
Select
Computer Science
Mathematics
Microbiology
Second Choice:
Select
Biology
Law
Medicine


The problem
The Vertex Coloring problem can be described like this:
Find a function \(f: V \rightarrow \{1, \dots, n\}, n \in \mathbb{N}\) such that:
\(\forall e=\{v_1, v_2\} \in E: f(v_1) \neq f(v_2)\).
Minimize \(n\).
If \(n\) is minimal for \(G\), it is called the chromatic number \(\chi(G)\).
What does that mean?
You have a graph. Then you take pencils and color the vertices such all vertices that are connected are not of the same color.
Example
All of the following graphs show valid thee colorings:
[caption id="attachment_69831" align="aligncenter" width="500"] Valid thee colorings of one graph
Source: Wikipedia[/caption]
Interesting facts
- $1 \leq \chi(G) \leq |V|$: You need at least one color and you could color all vertices with a different color
- $\chi(K_n) = n$: All complete graphs with $n$ vertices need exactly $n$ colors. One color for each vertex.
- Every planar graph can be colored with 4 colors (see four color theorem).
- Determining if a graph can be colored with 2 colors is equivalent to determining whether or not the graph is bipartite. This can be checked in polynomial time. You simply start with one vertex, give it color 1 and all adjacent vertices color 2. Then all adjacent vertices of color 2 have to have color 1, ...
- Vertex Coloring is in $\mathcal{NPC}$.
- All triangle-free planar graphs can be 3-colored. You can get this coloring in linear time (source).
Algorithms
I think learning from errors is important. This is the reason why I share the following algorithm that do not work.
First WRONG try: Fix adjacent vertices
[caption id="attachment_69891" align="aligncenter" width="512"] A vertex coloring algorithm that does not work[/caption]
The time complexity of this algorithm is in \(\mathcal{O}(|V|^2)\). This should make you suspicious, as Vertex Coloring is in \(\mathcal{NPC}\). So if it was correct, it would solve the P vs. NP problem which is worth a million dollars.
But an example why it doesn't work is better. Just try it for the following graph:
[caption id="attachment_69921" align="aligncenter" width="512"] Example that does not work with the provided algorithm[/caption]
Second WRONG try: Fix current vertex
[caption id="attachment_69971" align="aligncenter" width="512"] Another vertex coloring algorithm[/caption]
This algorithm gives a valid coloring, but the coloring is not minimal.
Example:
When you apply the algorithm the the graph below, you will get a coloring with four colors. But obviously, it is possible to color it with three colors. [caption id="attachment_69951" align="aligncenter" width="512"] Graph that can be vertex-colored with 3 colors[/caption]
Brute force
It's always a good idea to think about brute force algorithms. On the one hand, they are simple. So you can intuitively understand why they are correct and see their time / space complexity. On the other hand, you can use them for sanity checks of better algorithms for small problem instances.
Here is a brute force algorithm for the vertex coloring problem [caption id="attachment_70001" align="aligncenter" width="512"] Brute force a minimal vertex coloring[/caption]
You need \(\sum_{i=2}^n i^n\) steps at maximum. Wow. This is MUCH. Even when you only want to check if \(i\) colors are enough, you need \(i^n\).
You could use the second algorithm to get a better upper bound and try the next smaller ones. As soon as you don't get valid colorings, you know that the number of colors you've used in the last valid coloring was the minimum number.
Zero-knowledge protocol
Vertex coloring is relevant for so called "zero-knowledge protocols". This is a method by which one party (the prover) can prove to another party (the verifier) that a given statement is true, without conveying any additional information apart from the fact that the statement is indeed true.
A zero-knowledge proof must satisfy three properties:
- Completeness: if the statement is true, the honest verifier (that is, one following the protocol properly) will be convinced of this fact by an honest prover.
- Soundness: if the statement is false, no cheating prover can convince the honest verifier that it is true, except with some small probability.
- Zero-knowledge: if the statement is true, no cheating verifier learns anything other than this fact. This is formalized by showing that every cheating verifier has some ''simulator'' that, given only the statement to be proven (and no access to the prover), can produce a transcript that "looks like" an interaction between the honest prover and the cheating verifier.
Source: Wikipedia
Another great example is the following:
Imagine your friend is color-blind. You have two billiard balls; one is red, one is green, but they are otherwise identical. To your friend they seem completely identical, and he is skeptical that they are actually distinguishable. You want to prove to him (I say "him" as most color-blind people are male) that they are in fact differently-colored. On the other hand, you do not want him to learn which is red and which is green. Here is the proof system.  You can see the balls at this point, but you don't tell him which is which. Your friend then puts both hands behind his back. Next, he either switches the balls between his hands, or leaves them be, with probability 1/2 each. Finally, he brings them out from behind his back. You now have to "guess" whether or not he switched the balls. By looking at their colors, you can of course say with certainty whether or not he switched them. On the other hand, if they were the same color and hence indistinguishable, there is no way you could guess correctly with probability higher than $\frac{1}{2}$.  Furthermore, the proof is "zero-knowledge" because your friend never learns which ball is green and which is red; indeed, he gains no knowledge about how to distinguish the balls.
Source: mathoverflow.net
Lets make this more concrete. Say you want to authenticate somebody. This person is identified as the "only" person who knows a three-coloring of a big graph. This makes him/her special. If you simply asked him "what's the three coloring for your graph?" he would no longer be the only person who knows the three coloring.
So you want to get sure that he knows a three coloring without getting it.


{% load crispy_forms_tags %} {% load dict %}
Enter lists of HGNC symbols, Entrez IDs, or ENSEMBL gene IDs separated by spaces or line break. If any allow-list gene is given, only variants in a allow-listed gene are shown. Variants in block-listed genes are excluded. Entries are case-sensitive.
Enter lists of genomic regions, separated by spaces or line break. An entry has the format 1:1,000,000-2,000,000. This restricts the query to the listed regions.


Although I’m not a huge fan of Harry Potter, I absolutely LOVE Eliezer Yudkowsky’s Harry Potter and the Methods of Rationality.
Eliezer hasn’t updated the fanfic since Sept 7, 2011 and a lot of folks had thought it was dead, but he just returned with a vengeance a couple of days ago with the latest installment and a promise to keep going:
“We’re now set for a roll of updates that should last until Ch. 84 or longer!”
However, there’s one thing that keeps bothering me…
Spoiler Alert Ahead
If you haven’t read the latest chapter (or the book for that matter) go and take care of business first.
http://hpmor.com/chapter/78
Okay, now that you’re up to date, here’s my gripe: why were the Chaotics wearing their green tinted glasses so early on, allowing the Dragons to see them?
Besides potentially tipping off any scouting enemies, there’s a more practical reason why they shouldn’t be wearing shades until the last moment - it’s just weird to see everything in green! It’s not like there are paparazzi out to snap photos of them with blinding flash, so in my opinion there’s no sane reason to put on the sunglasses that early. Guess Eliezer just wanted Harry to not win one for a change.
Anyways, I’m sure there’s some sort of forum where people discuss this sort of thing but I’ve got better things to do with my time. That was sarcasm since I just spent 10 minutes writing this up but hey, just had to get it off my chest. Plus, I’m supposed to write about everything I read now.


--- layout: page title: "About" description: "What we do." header-img: "img/about-bg.png" ---
For more information about image. Read Fractal flame


Import/export data as mumodo-like objects
mumodoIO.py: Functions to for stream- and interval- frame I/O
Streamframes and Intervalframes are the two types of objects that mumodo uses for analysis. Both are Pandas DataFrames. Instead of subclassing Pandas DataFrames, we simply format our dataframes in a specific way.
Streamframes are like time series. They contain a series of timed, typed events. The events can be of different types in the same series. The index of the DataFrame is the timestamp of the event. Thus, slices of the DataFrame correspond to time intervals directly. Streamframes are created primarily from our own XIO files, but can also be saved in other formats (e.g. CSV, Praat point tiers).
Intervalframes are tiers of annotation intervals, such as those found in Praat and ELAN. They have three columns (start_time, end_time, text) This module makes use of the tgt [textgrid tools] package in order to import Praat textgrids. Because each tier of a textgrid is an intervalframe itself, the respective function returns a dictionary of Intervalframes, with the names of the tiers as keys.
Import data for one sensor out of a XIOFile and return a StreamFrame indexed with timestamps. By default, the timestamps are made relative. Optionally, and offset can be added to relative timestamps or the raw timestamps can be kept.
Arguments: filepath – Path + filename of the XIOFile to be imported. sensorname – Name of the sensor to be imported. window_size, with_fields, without_fields, discard_duplicates – Parameters for quantizing (see Quantize
function)
start_time, end_time, relative – Parameters for xio_quicklinegen. timestamp_offset – is zero by default and makes timestamps relative
Set to any non_integer value, timestamp_offset will leave the timestamps raw. Any non-zero integer value will be added as offset to the relative timestamps
Save many streamframes to a single XIOFile.
Arguments: framedict – a dict of streamframes, keys are the
sensornames
filepath – Path + filename of the file to be written. sensorname – Name of the sensor.
Import a textgrid and return a dict of IntervalFrames.
Each tier in the textgrid becomes an IntervalFrame (Pandas DataFrame) The Intervals by default are tokenized into start_time, end_time and text columns. The points (for point tiers) are tokenized into time and mark columns.
Arguments: filepath – Path + filename of the TextGrid file to be imported.
Keyword Arguments: asobjects – If True, then values are intervalobjects (as defined in
package tgt, instead of tokenizing into start_time etc. IntervalFrame has only one column with these objects.
encoding – character encoding to read the textgrid file
Write a dict of IntervalFrames in a textgrid-File.
Arguments: framedict – Dictionary of dataframes. The keys become tier
names in the textgrid file
filepath – Path + filename of the file to be written.
Keyword arguments: encoding: character encoding to save textgrid file
Quantize frames within a fixed time window
Series of typed, timed events can often be grouped by sensorname (or namespace) to create complete frames of data. For example, skeleton/lefthand and skeleton/righthand may have been logged as different events (lines), but should belong together, as frames of the sensor ‘skeleton’. This function performs this operation. This results in the quantization of the timestamps of the individual events into less timestamps for frames.
The function is a Python generator that yields dictionaries. Each dictionary is a frame, with keys corresponding to individual fields in the frame and their values. The function can be directly input to a Pandas Dataframe constructor (as the data argument)
Arguments: rows – iterable with dictionaries as items. They represent parsed
I/O events. These dictionaries’ keys should be:[‘sensorname’, ‘fieldname’, ‘time’, ‘type’,’value’]
sensorname – the name of the sensor by which to group
Keyword arguments: window_size – maximum time in ms to consider as one frame. Start a
new frame after this time has passed
Create an interval frame from an inc_reco file
Creates a dictionary with an intervalframe per chunk. The dictionary keys are the times of the chunks
Arguments:
filepath - the path to the inc_reco file
Kwargs:
encoding – the encoding of the file
Convert a Pointier into a StreamFrame
Converts a PointTier imported from a textgrid into a Streamframe
Arguments: pointtier – The PoinTier to convert
Convert a StreamFrame into a dictionary of PointTiers
Converts a Streamframe into a dictionary of PointTiers. The column names of the stream frame becomes the dictionary keys The resulting dictionary can then be saved into a Praat TextGrid with those Point Tiers.
Arguments: streamframe – the input streamframe
Keyword Arguments: columns – list of columns to convert. Each column must exist
in the streamframe


{{ end}}
{{ .Summary }}
I'm a student at the University of Minnesota - Twin Cities. I enjoy developing backend API services, tinkering with network configurations, and managing my homelab of servers during my freetime. When I'm not in class or out rockclimbing, you can probably find me wrangling code.
I'm always looking for new opportunities; if it sounds like a good fit, let's talk!
{{ .Summary }}
{{ .Summary }}


EXPERIENCE+EXPLORE
Movies In Theaters
Movie Actors and Actresses
Mobile
New DVD Releases
Special Offers


Essex Police said the victim, a woman in her 20s, was approached by a man at 6.30pm in Jaywick Lane, Jaywick, on Tuesday November 19.

Essex Police have described the man as in his 30s, white, with dark hair. He was wearing a grey jacket.
Polce are keen to speak to anyone who either lives or was drivng in the area, has CCTV-enabled doorbell or dash cam footage. They would also like to speak to anyone who was in the area at the time and saw a man matching the description or anyone acting suspiciously.
Anyone with information regarding the incident is asked to contact Essex Police's Major Crime Team on 101, quoting crime reference 42/183965/19.


- generateReg() - Method in class emulator.core.cpu.Flags
-
Combines the flagbits into a single byte.
- get() - Method in class emulator.core.cpu.Register
-
- getBit(byte, int) - Static method in interface emulator.Utils
-
Gives the state of a bit numbering from 0-7
- getCarry() - Method in class emulator.core.cpu.Flags
-
- getCycle() - Method in class emulator.core.cpu.CPU
-
- getHalfCarry() - Method in class emulator.core.cpu.Flags
-
- getHighByte() - Method in class emulator.core.cpu.Register
-
- getInterrupt() - Method in class emulator.core.cpu.Flags
-
- getLCDC() - Method in class emulator.core.gpu.Screen
-
- getLowByte() - Method in class emulator.core.cpu.Register
-
- getScrollX() - Method in class emulator.core.gpu.Screen
-
- getScrollY() - Method in class emulator.core.gpu.Screen
-
- getSign() - Method in class emulator.core.cpu.Flags
-
- getSubtract() - Method in class emulator.core.cpu.Flags
-
- getSurface() - Method in class emulator.core.gpu.GPU
-
- getZero() - Method in class emulator.core.cpu.Flags
-
- GPU - Class in emulator.core.gpu
-
Handles writing pixels to the main display.
- GPU(Memory) - Constructor for class emulator.core.gpu.GPU
-
Initializes a display window as well as
sets the video related registers to default
values.
- GUI - Class in emulator.core.gpu
-
Creates a window for the program.
- GUI(Screen) - Constructor for class emulator.core.gpu.GUI
-
Initializes window settings and creates
a surface to draw to.


public class SolverModel
extends java.lang.Object
Deals with all operations upon the values and candidates of the Cells, in
respect to their location and status within the 'Sudoku' grid.
Many different algorithms are used in order to solve a variety of different
sudoku puzzles, some easy and some a little tougher. This is the 'heart' of
the SudokuSolver.
Many algorithms are self contained, and attempt to use the same code for both Rows and Columns. Some methods grew larger and needed to be split into multiple methods (i.e. PointingPairs - Large If statement conditions), for ease of maintainability and explanation.
Collection tends to refer to a Row, Column or Block.
Each method returns a 'Result', containing it's success, any cells affected
and any message attached to the success. More can be seen on this in the
JavaDoc for 'Result.class'.
| Constructor and Description |
|---|
SolverModel()
Constructor prepares class for use by making new instances of
'allCollections', which holds all rows, columns and blocks as ordered
lists.
|
| Modifier and Type | Method and Description |
|---|---|
java.lang.String |
appendCellInfo(Cell cell)
Appends common information to a StringBuffer to be set and returned in
the result.
|
void |
defineBlocks()
Similar to defineRowsAndColumns, yet involving slightly different use of
nested for loops, as each block starts, continues for 3 cells, then moves
down a row (or across a column, depending how you build it.
|
void |
defineRowsAndColumns()
Through the use of a nested for loop, gets what would be each Cell in a
'Row' or 'Column' (depdning on 'direction' of i and j) and adds them into
the overall list of collections (Row, Column, Block).
|
java.util.LinkedList<java.util.LinkedList<Cell>> |
getAllCollections() |
Cell[][] |
getGrid() |
Result |
hiddenSingles()
Hidden Singles in Sudoku looks for any cells that have only one candidate
left in their candidates list.
|
Result |
nakedPairsAndTriples()
The technique of nakedPairsAndTriples relies upon finding Cells who have
only two or three values, shared between one another, within a
collection.
|
boolean |
nakedPairsAndTriplesRemoveCandidates(java.util.LinkedList<Cell> currentCollection,
java.util.LinkedList<java.lang.Integer> toRemove,
java.util.LinkedList<Cell> doNotTouch)
Part of the nakedPairsAndTriples algorithm, we first go through each
candidate for potential removal and with this look at each cell in the
collection.
|
Result |
nakedSingles()
Standard Sudoku technique involving looking through each Cell in the grid
in a particular collection and finding those with a value that appears
within that Cell and only that Cell in that one particular collection,
even if it belongs to another collection that has many cells looking for
this candidate.
|
Result |
pointingPairs(int mod)
Checks for pairs within a row or column that 'point'
towards other Cells that hold candidates to be removed,
as the pair itself hold the candidates to be removed.
|
boolean |
pointingPairsPairUp(java.util.LinkedList<Cell> currentBlock,
int modifier,
int candidate,
int i,
boolean useFirst)
Part of pointingPairs - removed to break up the code
and make it easier to read and maintain.
|
boolean |
pointingPairsStripCandidates(int modifier,
Cell firstCell,
int candidate)
In order to split larger chunks of
code up from pointingPairs() and make
it more maintainable, there is this method
that strips the candidates from the other
Cells in the Row/Column should the previous
tests and validations have passed.
|
boolean |
pointingPairsValidateCell(Cell firstCell,
Cell cellCheck,
int modifier)
Checks that the Cell being passed does not belong
on the same Row or Column as the Pair, ensuring that
candidates are not removed from the pair or that
candidates are valid (i.e.
|
void |
printLines()
My original way of viewing the steps and results taken as the Sudoku
puzzle solves.
|
Result |
removeCollectionCandidates()
Removes candidates from each Cell based on what candidates currently
exist in the grid.
|
void |
setCells(int[][] lines)
What values, including 0, to set to particular Cells to build the current
Sudoku puzzle.
|
boolean |
solved()
Checks all Cells in the puzzle.
|
public SolverModel()
Constructor prepares class for use by making new instances of 'allCollections', which holds all rows, columns and blocks as ordered lists.
public void setCells(int[][] lines)
What values, including 0, to set to particular Cells to build the current Sudoku puzzle.
lines
- - Cells as read in by FileHandler.public void defineRowsAndColumns()
Through the use of a nested for loop, gets what would be each Cell in a 'Row' or 'Column' (depdning on 'direction' of i and j) and adds them into the overall list of collections (Row, Column, Block).
public void defineBlocks()
Similar to defineRowsAndColumns, yet involving slightly different use of nested for loops, as each block starts, continues for 3 cells, then moves down a row (or across a column, depending how you build it. This example starts and iterates over rows).
By using multiple nested for loops, and knowing that each block is size
of 3, we can increment through the Cells relatively easy.
For example, we know that Block 2 (last on top Row), starts at, and
includes, cell (0, 6), which would be reached by rowStart = n *
Block_Spaces, or, 2 * Block_Spaces = 6. From there, it's a simple case of
iterating through the grid to grab the rows in each block below the
starting block when we know where to start and that no block goes more
than +3 in either rows or columns from the start.
public java.util.LinkedList<java.util.LinkedList<Cell>> getAllCollections()
public Result removeCollectionCandidates()
Removes candidates from each Cell based on what candidates currently
exist in the grid.
Searches the grid and adds existing values to a list, then goes through
this list and removes any cells that hold them as candidates in each type
of collection.
public Result nakedSingles()
Standard Sudoku technique involving looking through each Cell in the grid
in a particular collection and finding those with a value that appears
within that Cell and only that Cell in that one particular collection,
even if it belongs to another collection that has many cells looking for
this candidate.
Due to only one cell available for this candidate, it is only possible
for this cell to be this value, and so it is set to the cell.
public Result nakedPairsAndTriples()
The technique of nakedPairsAndTriples relies upon finding Cells who have
only two or three values, shared between one another, within a
collection.
We can be assured that if we find two cells in the collection who share
the two same candidates, then the values must belong to these cells and
no others, so those candidates are removed from other cells in the
collection.
In the case of finding three cells with three values, the same applies.
However, the same is also true if we find a cell that may have three
candidates, shared between two cells with only two candidates (e.g. {3,
6, 9}, {6,9}, {3,9}). Even with this in case, we know that out of all
cells, only these three can really contain these values, as other cells
have other options. Once again, this leads to removing, this time three,
candidates from the other cells in the collection.
The other possibility is that three cells appear with three candidates
split between them, but no one cell has three candidates to itself. My
algorithm here is not strong enough handle these situations. (e.g. {3,4},
{3,9}, {4,9} in the same collection).
public boolean nakedPairsAndTriplesRemoveCandidates(java.util.LinkedList<Cell> currentCollection, java.util.LinkedList<java.lang.Integer> toRemove, java.util.LinkedList<Cell> doNotTouch)
Part of the nakedPairsAndTriples algorithm, we first go through each
candidate for potential removal and with this look at each cell in the
collection.
For each cell, if it is unsolved, is not one of the Cells not to be
altered and that the amount of candidates to remove is equal to the
amount of cells not to be altered, remove the candidate from the cell.
The check of size of the list of Cells not to be altered versus the amount of candidates to be removed is very important. It is this check that ensures if we are removing anything, it is no more and no less than we have pairs. If it were otherwise, it would mean that the values do not work as pairs/triples.
currentCollection
- - Current Row/Column/BlocktoRemove
- - candidates for potential removaldoNotTouch
- - Cells not to be alteredpublic Result pointingPairs(int mod)
Checks for pairs within a row or column that 'point' towards other Cells that hold candidates to be removed, as the pair itself hold the candidates to be removed.
A 'pair' is looked for along the row or a column or each individual Block. If they are found to have matching candidates, and only those have the candidate in the individual block, we know we can remove the candidates from all other cells in the row/column, that the cells appear in, outside of the box. They are 'pointing' at cells that need candidates removed, hence the name.
The 'mod' parameter is a modifier that should either be a 1 (rows) or 3 (columns). Using simple arithmetic, and knowing that the size of a sudoku grid is always 9x9, a block is always 3x3 and there is also a static amount of Cells, columns, rows and blocks, the behaviour of the method can be altered using this modifier.
mod
- - A modifier, either 1 or 3, representing rows
or columns, respectively.public boolean pointingPairsPairUp(java.util.LinkedList<Cell> currentBlock, int modifier, int candidate, int i, boolean useFirst)
Part of pointingPairs - removed to break up the code
and make it easier to read and maintain.
Checks if the two Cell to potentially be paired up
in the Row/Column both don't have a value and share the
candidate expected.
Use first refers to whether the first Cell in the Row/Column
was selected, or if the second one was. The if statement
handles both of these outcomes with an OR statement, so
that if either of these is true, then it may indeed be
a pair.
currentBlock
- - The block to be checkedmodifier
- - rows/columns modifiercandidate
- - What candidate to be expected and comparedi
- - the location of the Cells being compared in the grid, when used
with the modifieruseFirst
- - whether the first or second Cell is the paired Cell
to be checked, based on which was available.public boolean pointingPairsValidateCell(Cell firstCell, Cell cellCheck, int modifier)
Checks that the Cell being passed does not belong on the same Row or Column as the Pair, ensuring that candidates are not removed from the pair or that candidates are valid (i.e. contained only in that one row/column and not any others in the block).
firstCell
- - First cell for pairing checkcellCheck
- - Cell to validate whether the
candidate is in the single Row/Column or elsewhere
in the block.modifier
- - Rows/Columns modifierpublic boolean pointingPairsStripCandidates(int modifier, Cell firstCell, int candidate)
In order to split larger chunks of code up from pointingPairs() and make it more maintainable, there is this method that strips the candidates from the other Cells in the Row/Column should the previous tests and validations have passed.
modifier
- firstCell
- candidate
- public Result hiddenSingles()
Hidden Singles in Sudoku looks for any cells that have only one candidate left in their candidates list. If they only have one candidate option, it means they must be this candidate without a doubt (presuming correct solutions to this point).
public boolean solved()
Checks all Cells in the puzzle. If each one of them has a value (based on
a counter increasing on each positive value found, then the method
returns true.
MAX_CELLS = 81, the full Sudoku grid cells.
public Cell[][] getGrid()
public java.lang.String appendCellInfo(Cell cell)
Appends common information to a StringBuffer to be set and returned in the result. In particular, information concerning cells and their location.
cell
- - current cell modifiedpublic void printLines()
My original way of viewing the steps and results taken as the Sudoku
puzzle solves. Left in for sake of re-usability should the GUI need to be
removed.
Prints out the current status of the board, with dividers for the Blocks
and underscores for unsolved cells. Does not display candidates.


Psalms 74:9
We see not our signs: there is no more any prophet: neither is there among us any that knoweth how long.


One
Two
Three
Four
Five
Six
Seven
Eight
One
Two
Three
Four
Five
Six
Seven
Eight
One
Two
Three
Four
Five
Six
Seven
Eight
Nine
Ten
Eleven
Twelve


Ocean [ax33], ocean, in The Inner Sea
Last updated Turn #80
Ocean [ax33], ocean, in The Inner Sea
------------------------------------------------------------------------
4: The Savvy Aardvark [ha5r], ship, 2 hulls, 1 rowing port, 6 sails,
4: 9% damaged, arrived from Ocean [ax34], owner:
4: Phantasm [n4b], Imperial Empire, duke, with two peasants, eight sailors
4: A storm batters the ship and causes some damage.
5: The Savvy Aardvark [ha5r], ship, 2 hulls, 1 rowing port, 6 sails,
5: 10% damaged, departed for Ocean [ax32].
14: It has become quite windy.
Routes leaving Ocean:
North, to Ocean [aw33], 6 days
East, to Ocean [ax34], 6 days
South, to Ocean [ay33], 6 days
West, to Ocean [ax32], 6 days
It is raining.
It is windy.


---
layout: feedback
title: Colton Hagan - 2016-02-24 18:50:42
date: 2016-02-24 18:50:42.000000000 -06:00
type: feedback
published: true
status: publish
categories: []
tags: []
meta:
_feedback_email: 'a:2:{s:2:"to";a:1:{i:0;s:21:"broberts_az@yahoo.com";}s:7:"message";s:390:"Name:
Colton Hagan
Email<EMAIL_ADDRESS>Website:
http://na
Comment: We have a ruby site that''s almost done.
Need help.<PHONE_NUMBER>. Call me.
Time: February 24, 2016 at 6:50
pm
IP Address: <IP_ADDRESS>
Contact Form URL: http://bob-roberts.net/contact/
Sent by an unverified visitor to your site.";}'
_feedback_extra_fields: a:0:{}
author:
---
We have a ruby site that's almost done. Need help.<PHONE_NUMBER>. Call me.
AUTHOR: Colton Hagan
AUTHOR EMAIL<EMAIL_ADDRESS>AUTHOR URL: http://na
SUBJECT: Email from Blog
IP: <IP_ADDRESS>
Array
(
[1_Name] => Colton Hagan
[2_Email] =><EMAIL_ADDRESS>[3_Website] => na
[4_Comment] => We have a ruby site that's almost done. Need help.<PHONE_NUMBER>. Call me.
)


Using Drill Metastore
Drill 1.17 introduces the Drill Metastore which stores the table schema and table statistics. Statistics allow Drill to better create optimal query plans.
The Metastore is a beta feature and is subject to change. In particular, the SQL commands and Metastore format may change based on your experience and feedback.
Note
In Drill 1.17, Metastore supports only tables in Parquet format. The feature is disabled by default. In Drill 1.18, Metastore supports all format plugins (except MaprDB) for the file system plugin. The feature is still disabled by default.
Drill Metastore introduction
One of the main advantages of Drill is schema-on-read. But Drill can’t handle some cases with this approach, there are the issues related to schema evolution or ambiguous schema.
Significant benefits of schema-aware execution:
- At Planning time:
- Better scope for planning optimizations.
- Proper estimation of column widths since types are known, hence more accurate costing.
- Graceful early exit if certain data type validations fail.
- At Runtime:
SchemaChange
exceptions avoidance. All minor fragments will have a common understanding of the schema.
Reading the data along with its statistics metadata helps to build more efficient plans and optimize query execution:
- Crucial for optimal join planning, 2-phase aggregation vs 1-phase aggregation planning, selectivity estimation of filter conditions, parallelization decisions.
Taking into account the above points, existing query processing can be improved by:
- storing table schema and reusing it;
- collecting, storing and reusing table statistics to improve query planning.
One of the main steps to resolve all these goals is providing the framework for metadata management named hereafter as Drill Metastore.
Enabling Drill Metastore
To use the Drill Metastore, you must enable it at the session or system level with one of the following commands:
SET `metastore.enabled` = true;
ALTER SYSTEM SET `metastore.enabled` = true;
Alternatively, you can enable the option in the Drill Web UI at http://<drill-hostname-or-ip-address>:8047/options
.
Computing and storing table metadata to Drill Metastore
Once you enable the Metastore, the next step is to populate it with data. Metastore entries are optional. If you query a table without a Metastore entry, Drill works with that table just as if the Metastore was disabled. In Drill, only add data to the Metastore when doing so improves query performance. In general, large tables benefit from statistics more than small tables do.
Unlike Hive, Drill does not require you to declare a schema. Instead, Drill infers the schema by scanning your table
in the same way as it is done during regular select and computes some metadata like MIN
/ MAX
column values and
NULLS_COUNT
designated as “metadata” to be able to produce more optimizations like filter push-down, etc. If
planner.statistics.use
option is enabled, this command will also calculate and store table statistics into Drill
Metastore.
Configuration
To configure the Metastore, create $DRILL_HOME/conf/drill-metastore-override.conf
file.
This is a HOCON-format file, just like drill-override.conf
. All Metastore configuration properties should reside in
drill.metastore
namespace.
Metastore Implementations
Drill Metastore offers an API that allows for any number of implementations. See metastore-api module docs for a description of the API.
The default implementation is the Iceberg Metastore based on Iceberg tables that provides support of transactions and concurrent writes. It resides on the file system specified in Metastore configuration.
To specify custom Metastore implementation, place the JAR which has the implementation of
org.apache.drill.metastore.Metastore
interface into classpath and indicate custom class in the
drill.metastore.implementation.class
config property.
The default value is the following:
drill.metastore: {
implementation.class: "org.apache.drill.metastore.iceberg.IcebergMetastore"
}
Metastore Components
The Drill 1.17 version of the Metastore stores metadata about tables: the table schema and table statistics. The Metastore is an active subproject of Drill, See DRILL-6552 for more information.
Table Metadata
Table Metadata includes the following info:
- Table schema, column name, type, nullability, scale and precision if available, and other info. For details please refer to Schema provisioning.
- Table statistics are of two kinds:
- Summary statistics:
MIN
,MAX
,NULL count
, etc. - Detail statistics: histograms,
NDV
, etc.
- Summary statistics:
Schema information and summary statistics also computed and stored for table segments, files, row groups and partitions.
The detailed metadata schema is described here.
You can try out the metadata to get a sense of what is available, by using the
Inspect the Metastore using INFORMATION_SCHEMA
tables tutorial.
Every table described by the Metastore may be a bare file or one or more files that reside in one or more directories.
If a table consists of a single directory or file, then it is non-partitioned. The single directory can contain any number of files.
Larger tables tend to have subdirectories. Each subdirectory is a partition and such a table are called “partitioned”.
Please refer to Exposing Drill Metastore metadata through INFORMATION_SCHEMA
tables
for information, how to query partitions and segments metadata.
A traditional database divides tables into schemas and tables. Drill can connect to any number of data sources, each of which may have its own schema. As a result, the Metastore labels tables with a combination of (plugin configuration name, workspace name, table name). Note that if before renaming any of these items, you must delete table’s Metadata entry and recreate it after renaming.
Using schema provisioning feature with Drill Metastore
The Drill Metastore holds both schema and statistics information for a table. The ANALYZE
command can infer the table
schema for well-defined tables (such as many Parquet tables). Some tables are too complex or variable for Drill’s
schema inference to work well. For example, JSON tables often omit fields or have long runs of nulls so that Drill
cannot determine column types. In these cases, you can specify the correct schema based on your knowledge of the
table’s structure. You specify a schema in the ANALYZE
command using the
Schema provisioning syntax.
Please refer to Provisioning schema for Drill Metastore for examples of usage.
Schema priority
Drill uses metadata during both query planning and execution. Drill gives you multiple ways to provide a schema.
When you run the ANALYZE TABLE
command, Drill will use the following rules for the table schema to be stored in the Metastore. In priority order:
- A schema provided in the table function.
- A schema file, created with
CREATE OR REPLACE SCHEMA
, in the table root directory. - Schema inferred from file data.
To plan a query, Drill requires information about your file partitions (if any) and about row and column cardinality. Drill does not use the provided schema for planning as it does not provide this metadata. Instead, at plan time Drill obtains metadata from one of the following, again in priority order:
- The Drill Metastore, if available.
- Inferred from file data. Drill scans the table’s directory structure to identify partitions. Drill estimates row counts based on the file size. Drill uses default estimates for column cardinality.
At query execution time, a schema tells Drill the shape of your data and how that data should be converted to Drill’s SQL types. Your choices for execution-time schema, in priority order, are:
- With a table function:
- specify an inline schema
- specify the path to the schema file.
- With a schema file, created with
CREATE OR REPLACE SCHEMA
, in the table root directory. - Using the schema from the Drill Metastore, if available.
- Infer the schema directly from file data.
Related Session/System Options
The Metastore provides a number of options to fit your environment. The default options are fine in most cases.
The options are set via ALTER SYSTEM SET
, SET
(it is an alias for ALTER SESSION SET
) or the Drill Web console.
In general, admin should set the options via ALTER SYSTEM
so that they take effect for all users.
Setting options at the session level is an advanced topic.
- metastore.enabled
Enables Drill Metastore usage to be able to store table metadata during ANALYZE TABLE commands execution and to be able
to read table metadata during regular queries execution or when querying some INFORMATION_SCHEMA tables. Default is
false
. - metastore.metadata.store.depth_level
Specifies the most-specific metadata kind to be collected with more general metadata kinds. Same options as the level option above. Default is
'ALL'
. - metastore.retrieval.retry_attempts
If you run the
ANALYZE TABLE
command at the same time as queries run, then the query can read incorrect or corrupt statistics. Drill will reload statistics and replan the query. This option specifies the maximum number of retry attempts. Default is5
. - metastore.metadata.fallback_to_file_metadata
Allows using file metadata cache for the case when required metadata is absent in the Metastore.
Default is
true
. - metastore.metadata.use_schema
The
ANALYZE TABLE
command infers table schema as it gathers statistics. This option tells Drill to use that schema information while planning the query. Disable this option if Drill has inferred the schema incorrectly, or schema will be provided separately (see CREATE OR REPLACE SCHEMA). Default istrue
. - metastore.metadata.use_statistics
Enables the Drill query planner to use table and column statistics stored in the Metastore. Default is
true
. Enableplanner.statistics.use
to be able to use statistics during query planning. - drill.exec.storage.implicit.last_modified_time.column.label
Sets the implicit column name for the last modified time (
lmt
) column. Used when producing Metastore analyze. You can set the last modified time column name to custom name when current column name clashes which column name present in the table. If your table contains a column name with the same name as an implicit column, the implicit column takes priority and shadows column from the table. Default islmt
. - drill.exec.storage.implicit.row_group_index.column.label
Sets the implicit column name for the row group index (
rgi
) column. Used when producing Metastore analyze. You can set row group index column name to custom name when current column name clashes which column name present in the table. If your table contains a column name with the same name as an implicit column, the implicit column takes priority and shadows column from the table. Default isrgi
. - drill.exec.storage.implicit.row_group_length.column.label
Sets the implicit column name for the row group length (
rgl
) column. Used when producing Metastore analyze. You can set row group length column name to custom name when current column name clashes which column name present in the table. If your table contains a column name with the same name as an implicit column, the implicit column takes priority and shadows column from the table. Default isrgl
. - drill.exec.storage.implicit.row_group_start.column.label
Sets the implicit column name for the row group start (
rgs
) column. Used when producing Metastore analyze. You can set row group start column name to custom name when current column name clashes which column name present in the table. If your table contains a column name with the same name as an implicit column, the implicit column takes priority and shadows column from the table. Default isrgs
.
Analyzing a table
You create Metastore metadata by running the ANALYZE TABLE
command.
The first time you run it, the Metastore will infer the schema and (depending on which options you have selected), populate statistics.
Tables change over time. To keep the Metastore metadata up-to-date, you must periodically run ANALYZE TABLE
again
on each changed table.
When you do ANALYZE TABLE
a second time, Drill will attempt to update statistics, called “incremental analysis”.
Incremental analysis will compute metadata only for files and partitions changed since the last analysis and reuse actual metadata from the Metastore where possible.
The command will return the following message if table statistics are up-to-date:
ANALYZE TABLE `lineitem` REFRESH METADATA;
|-------|---------------------------------------------------------|
| ok | summary |
|-------|---------------------------------------------------------|
| false | Table metadata is up to date, analyze wasn't performed. |
|-------|---------------------------------------------------------|
Table schemas evolve over time. If your table adds (or removes) columns, run ANALYZE TABLE
with the new set of
columns. Drill will perform a full table analysis.
General Information
Metadata usage
Drill uses the Metastore in several places. When you run a query with multiple directories, files or Parquet row groups,
Drill will use statistics to “prune” the scan. That is, to identify those directories, files or row groups that
do not contain data that your query needs. If you add new files or directories and do not rerun ANALYZE TABLE
,
then Drill will assume that existing metadata is invalid and wouldn’t use it. Periodically rerun ANALYZE TABLE
so
that Drill can use table metadata when possible.
Exposing Drill Metastore metadata through INFORMATION_SCHEMA
tables
Drill exposes some Metastore tables metadata through INFORMATION_SCHEMA
tables.
Note, that Metastore metadata will be exposed to the INFORMATION_SCHEMA
only if Metastore is enabled. If it is disabled, info
tables won’t contain Metastore metadata.
TABLES
table includes the set of tables on which you have run ANALYZE TABLE
.
Description of Metastore-specific columns:
| Column name | Type | Nullable | Description |
|---|---|---|---|
TABLE_SOURCE |
VARCHAR | YES | Table data type: PARQUET , CSV , JSON |
LOCATION |
VARCHAR | YES | Table location: /tmp/nation |
NUM_ROWS |
BIGINT | YES | Total number of rows in all files of the table. Null if not known |
LAST_MODIFIED_TIME |
TIMESTAMP | YES | Timestamp of the most-recently modified file within the table. Updated on each ANALYZE TABLE run. |
The COLUMNS
table describes the columns within each table. Only those columns listed in the COLUMNS
clause of the
ANALYZE TABLE
statement appear in this table.
| Column name | Type | Nullable | Description |
|---|---|---|---|
COLUMN_DEFAULT |
VARCHAR | YES | Column default value. |
COLUMN_FORMAT |
VARCHAR | YES | Usually applicable for date time columns: yyyy-MM-dd . See Format for Date, Time Conversion. |
NUM_NULLS |
BIGINT | YES | Number of rows which contain nulls for this column. |
MIN_VAL |
VARCHAR | YES | Minimum value of the column. For example: '-273' . |
MAX_VAL |
VARCHAR | YES | Maximum value of the column. For example: '100500' . |
NDV |
FLOAT8 | YES | Number of distinct values in column. |
EST_NUM_NON_NULLS |
FLOAT8 | YES | Estimated number of non null values. |
IS_NESTED |
BIT | NO | If column is nested. Nested columns are extracted from columns with struct type. |
A table can be divided into directories, called “partitions”. The PARTITIONS
table contains an entry for each directory.
| Column name | Type | Nullable | Description |
|---|---|---|---|
TABLE_CATALOG |
VARCHAR | YES | Table catalog (currently we have only one catalog): DRILL . |
TABLE_SCHEMA |
VARCHAR | YES | Table schema: dfs.tmp . |
TABLE_NAME |
VARCHAR | YES | Table name: nation . |
METADATA_KEY |
VARCHAR | YES | Top level segment key, the same for all nested segments and partitions: part_int=3 . |
METADATA_TYPE |
VARCHAR | YES | SEGMENT or PARTITION . Partition here corresponds to “Drill partition”, though segment corresponds to data parts like partitions in general case, for example, Hive partition. |
METADATA_IDENTIFIER |
VARCHAR | YES | Current metadata identifier: part_int=3/part_varchar=g . It is unique value for segment or partition within the table. |
PARTITION_COLUMN |
VARCHAR | YES | Partition column name: part_varchar . |
PARTITION_VALUE |
VARCHAR | YES | Partition column value: g . |
LOCATION |
VARCHAR | YES | Segment location, null for partitions: /tmp/nation/part_int=3 . |
LAST_MODIFIED_TIME |
TIMESTAMP | YES | Last modification time. |
Limitations of the 1.17 release
- Applies to tables stored as Parquet files and only when stored in the
DFS
storage plugin. - Disabled by default. You must enable this feature through the
metastore.enabled
system/session option.
Limitations of the 1.18 release
- Applies to all file system storage plugin formats except for MaprDB.
Cheat sheet of ANALYZE TABLE
commands
- Add a new table with
ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA
command. - When table data (but not schema) changes, run
ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA
command. - When the table schema changes, run
ANALYZE TABLE dfs.tmp.lineitem COLUMNS (col1, col2, ...) REFRESH METADATA
command. - If partitions are added or removed, run
ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA
command. - Remove table metadata by submitting
ANALYZE TABLE dfs.tmp.lineitem DROP METADATA
command.
Tutorial
Examples throughout this topic use the files and directories described in the following section Directory and File Setup
.
Directory and File Setup
The following examples are written for local file system, but Drill Metastore supports collecting metadata for tables placed in any any of Drill’s supported file systems. The examples work for both embedded and distributed Drill modes.
Download TPC-H sf1 tables and unpack archive to desired file system.
Set up storage plugin for desired file system, as described here: Connecting Drill to a File System.
Create lineitem directory in /tmp/
and two subdirectories under /tmp/lineitem
named s1
and s2
and copy there table data:
mkdir /tmp/lineitem
mkdir /tmp/lineitem/s1
mkdir /tmp/lineitem/s2
cp TPCH/lineitem /tmp/lineitem/s1
cp TPCH/lineitem /tmp/lineitem/s2
Query the directory /tmp/lineitem
:
SELECT count(*) FROM dfs.tmp.lineitem;
|----------|
| EXPR$0 |
|----------|
| 12002430 |
|----------|
1 row selected (0.291 seconds)
Notice that the query plan contains a group scan with usedMetastore = false
:
00-00 Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8410
00-01 Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8409
00-02 DirectScan(groupscan=[selectionRoot = file:/tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = false, ...
Compute table metadata and store in the Drill Metastore
Enable Drill Metastore:
SET `metastore.enabled` = true;
The above command enables the Metastore for just this one session.
Run the ANALYZE TABLE command on the table, whose metadata should be computed and stored into the Drill Metastore:
ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
|------|-------------------------------------------------------------|
| ok | summary |
|------|-------------------------------------------------------------|
| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
|------|-------------------------------------------------------------|
1 row selected (32.257 seconds)
The output of this command provides the status of the command execution and its summary.
Now that we’ve collected table metadata, we can use it when we query the table, by checking the usedMetastore=true
entry in ParquetGroupScan
:
00-00 Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.1 rows, 2.1 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8560
00-01 Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {2.0 rows, 2.0 cpu, 1.0 io, 0.0 network, 0.0 memory}, id = 8559
00-02 DirectScan(groupscan=[selectionRoot = /tmp/lineitem, numFiles = 12, usedMetadataSummaryFile = false, usedMetastore = true, ...
Perform incremental analysis
Rerun ANALYZE TABLE command on the lineitem
table:
ANALYZE TABLE dfs.tmp.lineitem REFRESH METADATA;
|-------|---------------------------------------------------------|
| ok | summary |
|-------|---------------------------------------------------------|
| false | Table metadata is up to date, analyze wasn't performed. |
|-------|---------------------------------------------------------|
1 row selected (0.249 seconds)
Inspect the Metastore using INFORMATION_SCHEMA tables
Run the following query to inspect lineitem
table metadata from TABLES
table stored in the Metastore:
SELECT * FROM INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME='lineitem';
|---------------|--------------|------------|------------|--------------|---------------|----------|-----------------------|
| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | TABLE_SOURCE | LOCATION | NUM_ROWS | LAST_MODIFIED_TIME |
|---------------|--------------|------------|------------|--------------|---------------|----------|-----------------------|
| DRILL | dfs.tmp | lineitem | TABLE | PARQUET | /tmp/lineitem | 12002430 | 2016-09-28 03:22:58.0 |
|---------------|--------------|------------|------------|--------------|---------------|----------|-----------------------|
1 row selected (0.157 seconds)
To obtain columns with their types and descriptions within the lineitem
table, run the following query:
SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
|---------------|--------------|------------|-----------------|------------------|----------------|-------------|-------------------|--------------------------|------------------------|-------------------|-------------------------|---------------|--------------------|---------------|--------------------|-------------|---------------|-----------|--------------|---------------------------------------------|-----------|-------------------|-----------|
| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | COLUMN_NAME | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE | DATA_TYPE | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS | MIN_VAL | MAX_VAL | NDV | EST_NUM_NON_NULLS | IS_NESTED |
|---------------|--------------|------------|-----------------|------------------|----------------|-------------|-------------------|--------------------------|------------------------|-------------------|-------------------------|---------------|--------------------|---------------|--------------------|-------------|---------------|-----------|--------------|---------------------------------------------|-----------|-------------------|-----------|
| DRILL | dfs.tmp | lineitem | dir0 | 1 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | 0 | s1 | s2 | null | null | false |
| DRILL | dfs.tmp | lineitem | l_orderkey | 2 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | 0 | 1 | 6000000 | 1499876.0 | 1.200243E7 | false |
| DRILL | dfs.tmp | lineitem | l_partkey | 3 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | 0 | 1 | 200000 | 199857.0 | 1.200243E7 | false |
...
| DRILL | dfs.tmp | lineitem | l_comment | 17 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | 0 | Tiresias | zzle? slyly final platelets sleep quickly. | 4586320.0 | 1.200243E7 | false |
|---------------|--------------|------------|-----------------|------------------|----------------|-------------|-------------------|--------------------------|------------------------|-------------------|-------------------------|---------------|--------------------|---------------|--------------------|-------------|---------------|-----------|--------------|---------------------------------------------|-----------|-------------------|-----------|
17 rows selected (0.187 seconds)
The sample lineitem
table has two partitions. The PARTITIONS
table contains an entry for each directory:
SELECT * FROM INFORMATION_SCHEMA.`PARTITIONS` WHERE TABLE_NAME='lineitem';
|---------------|--------------|------------|--------------|---------------|---------------------|------------------|-----------------|------------------|-----------------------|
| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | METADATA_KEY | METADATA_TYPE | METADATA_IDENTIFIER | PARTITION_COLUMN | PARTITION_VALUE | LOCATION | LAST_MODIFIED_TIME |
|---------------|--------------|------------|--------------|---------------|---------------------|------------------|-----------------|------------------|-----------------------|
| DRILL | dfs.tmp | lineitem | s2 | SEGMENT | s2 | `dir0` | s2 | /tmp/lineitem/s2 | 2016-09-28 03:22:58.0 |
| DRILL | dfs.tmp | lineitem | s1 | SEGMENT | s1 | `dir0` | s1 | /tmp/lineitem/s1 | 2016-09-28 03:22:58.0 |
|---------------|--------------|------------|--------------|---------------|---------------------|------------------|-----------------|------------------|-----------------------|
2 rows selected (0.149 seconds)
Drop table metadata
Once we are done exploring metadata we can drop the metadata for the lineitem
table.
Table metadata may be dropped using ANALYZE TABLE DROP METADATA
command:
ANALYZE TABLE dfs.tmp.lineitem DROP METADATA;
|------|----------------------------------------|
| ok | summary |
|------|----------------------------------------|
| true | Metadata for table [lineitem] dropped. |
|------|----------------------------------------|
1 row selected (0.291 seconds)
Collect metadata for specific set of columns
Next let’s gather metadata for a subset of the columns in the lineitem
table. You would do this to allow
Drill to optimize WHERE
conditions on certain columns. Also, if file size or the number of columns grows large, it
can take too long to gather all statistics. Instead you can speed up analysis by gathering statistics only for
selected columns: those actually used in the WHERE
clause.
For the case when metadata for several columns should be computed and stored into the Metastore, the following command may be used:
ANALYZE TABLE dfs.tmp.lineitem COLUMNS(l_orderkey, l_partkey) REFRESH METADATA;
|------|-------------------------------------------------------------|
| ok | summary |
|------|-------------------------------------------------------------|
| true | Collected / refreshed metadata for table [dfs.tmp.lineitem] |
|------|-------------------------------------------------------------|
1 row selected (94.1 seconds)
Now, check, that metadata is collected only for specified columns (MIN_VAL
, MAX_VAL
, NDV
, etc.), but all
columns are present:
SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='lineitem';
|---------------|--------------|------------|-----------------|------------------|----------------|-------------|-------------------|--------------------------|------------------------|-------------------|-------------------------|---------------|--------------------|---------------|--------------------|-------------|---------------|-----------|---------|---------|-----------|-------------------|-----------|
| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | COLUMN_NAME | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE | DATA_TYPE | CHARACTER_MAXIMUM_LENGTH | CHARACTER_OCTET_LENGTH | NUMERIC_PRECISION | NUMERIC_PRECISION_RADIX | NUMERIC_SCALE | DATETIME_PRECISION | INTERVAL_TYPE | INTERVAL_PRECISION | COLUMN_SIZE | COLUMN_FORMAT | NUM_NULLS | MIN_VAL | MAX_VAL | NDV | EST_NUM_NON_NULLS | IS_NESTED |
|---------------|--------------|------------|-----------------|------------------|----------------|-------------|-------------------|--------------------------|------------------------|-------------------|-------------------------|---------------|--------------------|---------------|--------------------|-------------|---------------|-----------|---------|---------|-----------|-------------------|-----------|
| DRILL | dfs.tmp | lineitem | dir0 | 1 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | 0 | s1 | s2 | null | null | false |
| DRILL | dfs.tmp | lineitem | l_orderkey | 2 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | 0 | 1 | 6000000 | 1499876.0 | 1.200243E7 | false |
| DRILL | dfs.tmp | lineitem | l_partkey | 3 | null | YES | INTEGER | null | null | 0 | 2 | 0 | null | null | null | 11 | null | 0 | 1 | 200000 | 199857.0 | 1.200243E7 | false |
...
| DRILL | dfs.tmp | lineitem | l_comment | 17 | null | YES | CHARACTER VARYING | 65535 | 65535 | null | null | null | null | null | null | 65535 | null | null | null | null | null | null | false |
|---------------|--------------|------------|-----------------|------------------|----------------|-------------|-------------------|--------------------------|------------------------|-------------------|-------------------------|---------------|--------------------|---------------|--------------------|-------------|---------------|-----------|---------|---------|-----------|-------------------|-----------|
17 rows selected (0.183 seconds)
Provisioning schema for Drill Metastore
Directory and File Setup
Ensure you have configured the file system storage plugin as described here: Connecting Drill to a File System.
Set store.format
to csvh
:
SET `store.format`='csvh';
|------|-----------------------|
| ok | summary |
|------|-----------------------|
| true | store.format updated. |
|------|-----------------------|
Create a text table based on the sample /tpch/nation.parquet
table from cp
plugin:
CREATE TABLE dfs.tmp.text_nation AS
(SELECT *
FROM cp.`/tpch/nation.parquet`);
|----------|---------------------------|
| Fragment | Number of records written |
|----------|---------------------------|
| 0_0 | 25 |
|----------|---------------------------|
Query the table text_nation
:
SELECT typeof(n_nationkey),
typeof(n_name),
typeof(n_regionkey),
typeof(n_comment)
FROM dfs.tmp.text_nation
LIMIT 1;
|---------|---------|---------|---------|
| EXPR$0 | EXPR$1 | EXPR$2 | EXPR$3 |
|---------|---------|---------|---------|
| VARCHAR | VARCHAR | VARCHAR | VARCHAR |
|---------|---------|---------|---------|
Notice that the query plan contains a group scan with usedMetastore = false
:
00-00 Screen : rowType = RecordType(ANY EXPR$0, ANY EXPR$1, ANY EXPR$2, ANY EXPR$3): rowcount = 1.0, cumulative cost = {25.1 rows, 109.1 cpu, 2247.0 io, 0.0 network, 0.0 memory}, id = 160
00-01 Project(EXPR$0=[TYPEOF($0)], EXPR$1=[TYPEOF($1)], EXPR$2=[TYPEOF($2)], EXPR$3=[TYPEOF($3)]) : rowType = RecordType(ANY EXPR$0, ANY EXPR$1, ANY EXPR$2, ANY EXPR$3): rowcount = 1.0, cumulative cost = {25.0 rows, 109.0 cpu, 2247.0 io, 0.0 network, 0.0 memory}, id = 159
00-02 SelectionVectorRemover : rowType = RecordType(ANY n_nationkey, ANY n_name, ANY n_regionkey, ANY n_comment): rowcount = 1.0, cumulative cost = {24.0 rows, 93.0 cpu, 2247.0 io, 0.0 network, 0.0 memory}, id = 158
00-03 Limit(fetch=[1]) : rowType = RecordType(ANY n_nationkey, ANY n_name, ANY n_regionkey, ANY n_comment): rowcount = 1.0, cumulative cost = {23.0 rows, 92.0 cpu, 2247.0 io, 0.0 network, 0.0 memory}, id = 157
00-04 Scan(table=[[dfs, tmp, text_nation]], groupscan=[EasyGroupScan [... schema=null, usedMetastore=false...
Compute table metadata and store in the Drill Metastore
Enable Drill Metastore:
SET `metastore.enabled` = true;
Specify table schema when running ANALYZE
query:
ANALYZE TABLE table(dfs.tmp.`text_nation` (type=>'text', fieldDelimiter=>',', extractHeader=>true,
schema=>'inline=(
`n_nationkey` INT not null,
`n_name` VARCHAR not null,
`n_regionkey` INT not null,
`n_comment` VARCHAR not null)'
)) REFRESH METADATA;
|------|----------------------------------------------------------------|
| ok | summary |
|------|----------------------------------------------------------------|
| true | Collected / refreshed metadata for table [dfs.tmp.text_nation] |
|------|----------------------------------------------------------------|
Inspect the Metastore using INFORMATION_SCHEMA tables
Run the following query to inspect text_nation
table schema stored in the Metastore:
SELECT COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.`COLUMNS` WHERE TABLE_NAME='text_nation';
|-------------|-------------------|
| COLUMN_NAME | DATA_TYPE |
|-------------|-------------------|
| n_nationkey | INTEGER |
| n_name | CHARACTER VARYING |
| n_regionkey | INTEGER |
| n_comment | CHARACTER VARYING |
|-------------|-------------------|
Ensure that this schema is applied to the table:
SELECT typeof(n_nationkey),
typeof(n_name),
typeof(n_regionkey),
typeof(n_comment)
FROM dfs.tmp.text_nation
LIMIT 1;
|--------|---------|--------|---------|
| EXPR$0 | EXPR$1 | EXPR$2 | EXPR$3 |
|--------|---------|--------|---------|
| INT | VARCHAR | INT | VARCHAR |
|--------|---------|--------|---------|
select sum(n_nationkey) from dfs.tmp.`text_nation`;
|--------|
| EXPR$0 |
|--------|
| 300 |
|--------|
Query plan contains schema from the Metastore and group scan with usedMetastore = true
:
00-00 Screen : rowType = RecordType(ANY EXPR$0): rowcount = 1.0, cumulative cost = {45.1 rows, 287.1 cpu, 2247.0 io, 0.0 network, 0.0 memory}, id = 3129
00-01 Project(EXPR$0=[$0]) : rowType = RecordType(ANY EXPR$0): rowcount = 1.0, cumulative cost = {45.0 rows, 287.0 cpu, 2247.0 io, 0.0 network, 0.0 memory}, id = 3128
00-02 StreamAgg(group=[{}], EXPR$0=[SUM($0)]) : rowType = RecordType(ANY EXPR$0): rowcount = 1.0, cumulative cost = {44.0 rows, 286.0 cpu, 2247.0 io, 0.0 network, 0.0 memory}, id = 3127
00-03 Scan(table=[[dfs, tmp, text_nation]], groupscan=[EasyGroupScan ... schema=..., usedMetastore=true]]) ...


Browse movies and add reviews to moives you have watched.
Browse tv shows and add reviews to tv shows you have watched.
Browse documentaries and add reviews to documentaries you have watched.


Before Change
// We need to terminate the worker here instead of from the master
// so we can guarantee that the router won"t submit any more tasks
// on it.
worker_handle.__ray_terminate__.remote()
async def set_traffic(self, endpoint, traffic_dict):
logger.debug("Setting traffic for endpoint %s to %s", endpoint,
traffic_dict)
After Change
backend_replica_tag = backend_tag + ":" + replica_tag
if backend_replica_tag not in self.replicas:
return
del self.replicas[backend_replica_tag]
// We need this lock because we modify worker_queue here.
async with self.flush_lock:
old_queue = self.worker_queues[backend_tag]


| CONTENTS | PREV | NEXT | Java Remote Method Invocation |
A simple bootstrap name server is provided for storing named references to remote objects. A remote object reference can be stored using the URL-based methods of the classjava.rmi.Naming
.For a client to invoke a method on a remote object, that client must first obtain a reference to the object. A reference to a remote object is usually obtained as a parameter or return value in a method call. The RMI system provides a simple bootstrap name server from which to obtain remote objects on given hosts. The
java.rmi.Naming
class provides Uniform Resource Locator (URL) based methods to look up, bind, rebind, unbind, and list the name-object pairings maintained on a particular host and port.


ASU TESOL Professional Certificate
Saralis Rivera | Teaching Portfolio
Teaching Philosophy
1. Edited Reading/Writing Lesson Plan with Technology
2. Technology-Enriched Reading/Writing Lesson Plan
3. Edited Listening/Speaking Lesson Plan with Technology
4. Technology-Enriched Listening/Speaking Lesson Plan
5. Grammar Lesson Plan
6. Pronunciation Lesson Plan
7. Capstone 1 Lesson Plan
8. Original Reading/Writing/Grammar Lesson Plan
9. Original Listening/Speaking/Pronunciation Lesson Plan
10. Technology-Enriched Lesson Plan
Toolbox Tip 1
Toolbox Tip 2


By DORREN CARVAJAL and SUZANNE DALEY
The initiative of LICT is part of the government's plan to create 30,000 direct and 120,000 indirect jobs in the IT and ITES companies under LICT Project in which the World Bank is providing US$ 70 million credit for implementing the Leveraging ICT for growth, Employment and Governance project (LICT) of Bangladesh Computer Council (BCC) under the ICT Division of Posts, Telecommunication and IT Ministry.


body{ text-align: center; }
Here is how we create all of the levels for our game. Having already created small, cropped versions of our images for inclusion in the game, we coded a GUI to help make the process easier. By clicking with the mouse, and using several keyboard controls, we switch between the different elements and can place them onto the Java matrix (the red dot is where the element will go). The GUI associates each element with a numeric identifier, and then its position with another numeric identifier (so we can find where the element goes later). At the end of placing all the elements where we want it to be, we can export all the data on the elements to a matrix on numbers with one key stroke. Finally, another Java file reads the raw data from the GUI and turn it into code that can be incorporated into the Pebble Watch application. From here, the main application can be coded!


lem-in
src
paths
paths_print.c
Go to the documentation of this file.
1
/* ************************************************************************** */
3
/* */
4
/* ::: :::::::: */
5
/* paths_print.c :+: :+: :+: */
6
/* +:+ +:+ +:+ */
7
/* By: user<EMAIL_ADDRESS>+#+ +:+ +#+ */
8
/* +#+#+#+#+#+ +#+ */
9
/* Created: 2020/10/06 19:15:58 by user #+# #+# */
10
/* Updated: 2020/10/10 23:36:02 by user ### ########.fr */
11
/* */
12
/* ************************************************************************** */
13
14
#include "
paths.h
"
15
16
void
paths_print
(
const
t_paths
*paths,
const
t_graph
*graph)
17
{
18
int
i;
19
t_link
*link;
20
21
i = 0;
22
while
(i < paths->npaths)
23
{
24
link = paths->
arr
[i]->
head
;
25
while
(link)
26
{
27
ft_putstr
(
graph_id2name
(graph, link->
content
));
28
link = link->
next
;
29
ft_putstr
(link ?
" -> "
:
"\n"
);
30
}
31
i++;
32
}
33
}
s_link::content
int content
The integer value at this link.
Definition:
list.h:35
graph_id2name
const char * graph_id2name(const t_graph *graph, int id)
Returns the name of the node with a given id.
Definition:
graph_id2name.c:16
ft_putstr
void ft_putstr(char const *s)
paths_print
void paths_print(const t_paths *paths, const t_graph *graph)
Prints the paths.
Definition:
paths_print.c:16
s_list::head
t_link * head
The first link in the list.
Definition:
list.h:53
paths.h
s_graph
A graph.
Definition:
graph.h:96
s_paths::arr
t_list ** arr
An array of paths.
Definition:
paths.h:43
s_paths
A data structing containing the paths in the graph.
Definition:
paths.h:39
s_link::next
struct s_link * next
A pointer to the next link.
Definition:
list.h:36
s_link
A link in in a doubly linked list of integers.
Definition:
list.h:33
Generated by
1.8.16


Student
Staff
Student Registration
Student ID:
Firstname:
Lastname:
Address:
Email:
Password:
Password confirm:
Date Of Birth:
Phone Number:
Already have an account?
Log in
Staff Registration
Staff ID:
Firstname:
Lastname:
Email:
Password:
Password Confirm:
Qualifications:
Expertise:
Phone Number:
Already have an account?
Log in


Welcome NAME !
Log Out
Home
My Facets
Add New Facet
Edit a Facet
History
Choose a Group:
Group 1
Group 2
MyStore
Choose a Time Period:
From:
To:
Choose a Facet:
Facet 1
black_friday
New arrival
Choose Statistics:
Likes
Comments
Posts
Welcome to
RadFa
! Discover the reasonance of your words beyond your posts and your groups!!


land scenario development scenarios change New spatial LULC Recent new England rate use future Trends conservation allocation land-use simulations stakeholders OpenUrl protection Figure Impact planning probability Wildlife Areas alternative areas landscape developed protected Environmental NELF cover simulated simulation wildlife View observed Thompson Yankee city forest increase natural process year Table flood project scenarios. using Cosmopolitan Explorer Global Growing Trends. increased local resource resulted tool Conservation Department Download Fish based co-designed development, example, impacts participatory patterns qualitative region scenario. terms translating zones Action Connected CrossRef Management National Open approach area consequences data-hide-link-title=""> data-icon-position="" figure habitat high models modifications narrative outcomes subregions. tab transition used Forest Forest, Island Landscape Maine McBride Priority Rhode Scenario defined delineated development. different envisions forests greater key patch scenario, time water Areas. Communities Dinamica Fisheries Flood Futures Hampshire Harvard Modelling Research Scenarios State Survey Trends, agriculture and, change. changes compared data designed differences environmental explore important loss low make meet model potential priorities priority quantitative range represent residential simulate specific subregions transitions urban &; (bar (i.e., Areas, Connecticut Core Density Development, Division Drinking Ecological Emergency Federal Foster Indeed, Lambert Modeling Olofsson Overall, Plan. Soil States Thompson, United agricultural available cellular centers climate community conditions covers despite difference distribution divergent diverse economic expansion forested futures greatest included including inline innovation land-cover landscape. multiple narratives online pattern planning, plans population popup predictor priorities. private rates regional relative services shape social socio-economic table: trends understanding utility way wetlands


Sam Bridges is ready to deliver your Podtoid questions
Ask away, m'kay
by CJ Andriessen[Update: So you might have noticed we didn't have a show this week. Well, long story short, food poisoning. Very, very bad food poisoning. So we're moving community questions to this week's show and giving you another chance to ask us what's on your mind.]
What a busy week for video games. Big releases, big announcements, and big reviews capped off an interesting and certainly memorable October. Now there are just one or two more big releases left in 2019. Will they be good? Will they suck? We can't answer those questions but the Podtoid crew can answer any other questions you have for us on this week's show. So ask away in the comments below.


|
LOC
0.2
Count lines of code
|
Class responsible for parsing a file language code. More...
#include <streamParser.hh>
Public Member Functions | |
| streamParser () | |
| Class constructor. | |
| ~streamParser () | |
| Class destructor. | |
| parserType | getType (void) |
| Obtain the parser type. More... | |
| Public Member Functions inherited from parser | |
| parser () | |
| Class constructor. | |
| virtual | ~parser () |
| Class destructor. | |
| virtual void | parse (file *sourceFile) |
| Parse a source file. More... | |
Additional Inherited Members | |
| Static Protected Member Functions inherited from parser | |
| static std::size_t | findToken (std::string_view token, std::string_view where, bool sensitive) |
| Search a comment token in string. More... | |
| Protected Attributes inherited from parser | |
| language * | p_lang |
| Pointer to a language object. | |
| file * | p_iFile |
| Current file being processed. | |
| std::ifstream | iSourceFile |
| Stream to read file. | |
| statistics | iStats |
| Internal statistics for the current parsed file. | |
Class responsible for parsing a file language code.
|
inlinevirtual |