assigned
by None at 2022-10-31 14:13:42.closed
by None at 2022-10-31 14:18:17.closed
by None at 2022-10-31 17:29:00.closed
by None at 2022-10-31 17:29:01.head_ref_force_pushed
by Christina Chortaria at 2022-10-31 12:56:58.deployed
by None at 2022-10-31 12:58:47.@dependabot rebase
mentioned
by Dependabot at 2022-10-31 12:57:32.head_ref_force_pushed
by None at 2022-10-31 12:58:06.deployed
by None at 2022-10-31 13:11:58.deployed
by None at 2022-10-31 13:22:20.referenced
by Christina Chortaria at 2022-10-31 13:48:01.merged
by Christina Chortaria at 2022-10-31 13:48:01.closed
by Christina Chortaria at 2022-10-31 13:48:01.head_ref_deleted
by Christina Chortaria at 2022-10-31 13:48:02.assigned
by Bess Sadler at 2022-10-31 18:14:49.These are still working as expected, I am going to consider these resolved.
closed
by James R. Griffin III at 2022-10-31 18:45:48.referenced
by Eliot Jordan at 2022-10-31 19:07:34.merged
by Eliot Jordan at 2022-10-31 19:07:34.closed
by Eliot Jordan at 2022-10-31 19:07:34.head_ref_deleted
by Eliot Jordan at 2022-10-31 19:07:35.closed
by Eliot Jordan at 2022-10-31 19:07:36.labeled
by Eliot Jordan at 2022-10-31 19:19:54.assigned
by Eliot Jordan at 2022-10-31 19:20:08.assigned
by Shaun Ellis at 2022-10-31 19:51:36.Should the "For more information on harmful content please see the PUL Statement on Harmful Content" statement be appended for each resource or will the metadata have it when it's needed?
This part of the statement isn't in the example record metadata.
@m-ichelleliu We're no longer generating JP2s for Cantaloupe. You can remove that from the doc.
mentioned
by None at 2022-10-31 19:05:50.subscribed
by None at 2022-10-31 19:05:50.review_requested
by None at 2022-10-31 20:04:07.deployed
by None at 2022-10-31 18:09:32.ready_for_review
by Jane Sandberg at 2022-10-31 18:11:58.closed
by Kevin Reiss at 2022-10-31 04:46:18.referenced
by Christina Chortaria at 2022-10-31 22:51:57.merged
by Christina Chortaria at 2022-10-31 22:51:57.closed
by Christina Chortaria at 2022-10-31 22:51:57.head_ref_deleted
by Christina Chortaria at 2022-10-31 22:51:58.deployed
by None at 2022-10-31 22:09:59.referenced
by Christina Chortaria at 2022-10-31 22:54:31.merged
by Christina Chortaria at 2022-10-31 22:54:32.closed
by Christina Chortaria at 2022-10-31 22:54:32.head_ref_deleted
by Christina Chortaria at 2022-10-31 22:54:32.assigned
by Jane Sandberg at 2022-10-31 21:56:16.closed
by Christina Chortaria at 2022-10-31 22:54:33.referenced
by Vickie Karasic at 2022-10-31 18:40:02.merged
by Vickie Karasic at 2022-10-31 18:40:02.closed
by Vickie Karasic at 2022-10-31 18:40:03.Closed in favor of #144
assigned
by Christina Chortaria at 2022-10-31 13:56:15.deployed
by None at 2022-10-31 15:22:17.referenced
by Jane Sandberg at 2022-10-31 15:28:49.merged
by Jane Sandberg at 2022-10-31 15:28:49.closed
by Jane Sandberg at 2022-10-31 15:28:49.head_ref_deleted
by Jane Sandberg at 2022-10-31 15:28:50.closed
by Jane Sandberg at 2022-10-31 15:28:50.labeled
by Christina Chortaria at 2022-10-31 16:19:48.head_ref_force_pushed
by Jane Sandberg at 2022-10-31 16:28:49.head_ref_force_pushed
by Jane Sandberg at 2022-10-31 16:34:49.ready_for_review
by Jane Sandberg at 2022-10-31 16:44:57.referenced
by Christina Chortaria at 2022-10-31 17:15:18.merged
by Christina Chortaria at 2022-10-31 17:15:18.closed
by Christina Chortaria at 2022-10-31 17:15:18.head_ref_deleted
by Christina Chortaria at 2022-10-31 17:15:19.closed
by Christina Chortaria at 2022-10-31 17:15:19.Coverage decreased (-0.02%) to 94.514% when pulling ad1a6aca92b66460ad50bb8673ae2333430856bd on dependabot/bundler/nokogiri-1.13.9 into 490d53c50674f5267f2e63f5795243750488d780 on main.
@dependabot rebase
deployed
by None at 2022-10-31 13:49:12.mentioned
by Dependabot at 2022-10-31 13:49:23.head_ref_force_pushed
by None at 2022-10-31 13:49:57.deployed
by None at 2022-10-31 13:58:06.referenced
by Christina Chortaria at 2022-10-31 14:12:26.merged
by Christina Chortaria at 2022-10-31 14:12:26.closed
by Christina Chortaria at 2022-10-31 14:12:26.head_ref_deleted
by Christina Chortaria at 2022-10-31 14:12:27.Example records to test in catalog qa https://catalog-qa.princeton.edu/catalog/9947826143506421 constituent https://catalog-qa.princeton.edu/catalog/9968683243506421 https://catalog-qa.princeton.edu/catalog/9967734313506421 https://catalog-qa.princeton.edu/catalog/9956597633506421 related records https://catalog-qa.princeton.edu/catalog/9970446223506421 https://catalog-qa.princeton.edu/catalog/9938524743506421 https://catalog-qa.princeton.edu/catalog/99101599253506421 cdl https://catalog-qa.princeton.edu/catalog/99116299583506421
deployed
by None at 2022-10-31 15:33:47.referenced
by Jane Sandberg at 2022-10-31 15:42:56.merged
by Jane Sandberg at 2022-10-31 15:42:56.closed
by Jane Sandberg at 2022-10-31 15:42:57.head_ref_deleted
by Jane Sandberg at 2022-10-31 15:42:58.closed
by Jane Sandberg at 2022-10-31 15:42:58.assigned
by Hector Correa at 2022-10-31 13:39:32.@mccalluc do these changes make more sense to you?
head_ref_force_pushed
by None at 2022-10-31 12:39:44.review_dismissed
by None at 2022-10-31 14:39:38.head_ref_force_pushed
by None at 2022-10-31 14:39:38.mentioned
by Chuck McCallum at 2022-10-31 14:40:43.subscribed
by Chuck McCallum at 2022-10-31 14:40:43.head_ref_force_pushed
by None at 2022-10-31 14:59:43.review_requested
by None at 2022-10-31 15:02:57.referenced
by None at 2022-10-31 15:15:08.merged
by None at 2022-10-31 15:15:08.closed
by None at 2022-10-31 15:15:08.head_ref_deleted
by None at 2022-10-31 15:15:09.referenced
by None at 2022-10-31 12:36:51.referenced
by None at 2022-10-31 12:39:43.referenced
by None at 2022-10-31 14:39:37.referenced
by None at 2022-10-31 14:59:42.closed
by None at 2022-10-31 15:15:09.referenced
by None at 2022-10-31 15:15:10.referenced
by Hector Correa at 2022-10-31 17:39:51.merged
by Hector Correa at 2022-10-31 17:39:51.closed
by Hector Correa at 2022-10-31 17:39:51.head_ref_deleted
by Hector Correa at 2022-10-31 17:39:52.head_ref_force_pushed
by None at 2022-10-31 15:02:15.head_ref_force_pushed
by None at 2022-10-31 15:04:57.head_ref_force_pushed
by None at 2022-10-31 15:15:32.referenced
by Hector Correa at 2022-10-31 17:46:04.merged
by Hector Correa at 2022-10-31 17:46:04.closed
by Hector Correa at 2022-10-31 17:46:04.head_ref_deleted
by Hector Correa at 2022-10-31 17:46:05.assigned
by None at 2022-10-31 12:39:00.closed
by Hector Correa at 2022-10-31 17:46:05.referenced
by Hector Correa at 2022-10-31 18:02:33.merged
by Hector Correa at 2022-10-31 18:02:33.closed
by Hector Correa at 2022-10-31 18:02:33.head_ref_deleted
by Hector Correa at 2022-10-31 18:02:34.Legacy DataSpace record: https://dataspace.princeton.edu/handle/88435/dsp01h415pd457 Migration spec: https://github.com/pulibrary/pdc_describe/blob/main/spec/system/cytoskeletal_form_submission_spec.rb
This is the datacite record as produced by our migration spec:
<?xml version="1.0"?>
<resource xsi:schemaLocation='http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4/metadata.xsd' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='http://datacite.org/schema/kernel-4'>
<identifier identifierType='DOI'>10.34770/r2dz-ys12</identifier>
<creators>
<creator>
<creatorName>Taylor, Jenny A</creatorName>
<givenName>Jenny A</givenName>
<familyName>Taylor</familyName>
</creator>
<creator>
<creatorName>Bratton, Benjamin P</creatorName>
<givenName>Benjamin P</givenName>
<familyName>Bratton</familyName>
</creator>
<creator>
<creatorName>Sichel, Sophie R</creatorName>
<givenName>Sophie R</givenName>
<familyName>Sichel</familyName>
</creator>
<creator>
<creatorName>Blair, Kris M</creatorName>
<givenName>Kris M</givenName>
<familyName>Blair</familyName>
</creator>
<creator>
<creatorName>Jacobs, Holly M</creatorName>
<givenName>Holly M</givenName>
<familyName>Jacobs</familyName>
</creator>
<creator>
<creatorName>DeMeester, Kristen E</creatorName>
<givenName>Kristen E</givenName>
<familyName>DeMeester</familyName>
</creator>
<creator>
<creatorName>Kuru, Erkin</creatorName>
<givenName>Erkin</givenName>
<familyName>Kuru</familyName>
</creator>
<creator>
<creatorName>Gray, Joe</creatorName>
<givenName>Joe</givenName>
<familyName>Gray</familyName>
</creator>
<creator>
<creatorName>Biboy, Jacob</creatorName>
<givenName>Jacob</givenName>
<familyName>Biboy</familyName>
</creator>
<creator>
<creatorName>VanNieuwenhze, Michael S</creatorName>
<givenName>Michael S</givenName>
<familyName>VanNieuwenhze</familyName>
</creator>
<creator>
<creatorName>Vollmer, Waldemar</creatorName>
<givenName>Waldemar</givenName>
<familyName>Vollmer</familyName>
</creator>
<creator>
<creatorName>Grimes, Catherine L</creatorName>
<givenName>Catherine L</givenName>
<familyName>Grimes</familyName>
</creator>
<creator>
<creatorName>Shaevitz, Joshua W</creatorName>
<givenName>Joshua W</givenName>
<familyName>Shaevitz</familyName>
</creator>
<creator>
<creatorName>Salama, Nina R</creatorName>
<givenName>Nina R</givenName>
<familyName>Salama</familyName>
</creator>
</creators>
<titles>
<title>
Distinct cytoskeletal proteins define zones of enhanced cell wall
synthesis in Helicobacter pylori
</title>
</titles>
<publisher>Princeton University</publisher>
<resourceType resourceTypeGeneral='Dataset'/>
<publicationYear>2019</publicationYear>
<relatedIdentifiers>
<relatedIdentifier relationType='IsIdenticalTo' relatedIdentifierType='ARK'>ark:/88435/dsp01h415pd457</relatedIdentifier>
<relatedIdentifier relationType='IsCitedBy' relatedIdentifierType='arXiv'>https://www.biorxiv.org/content/10.1101/545517v1</relatedIdentifier>
<relatedIdentifier relationType='IsCitedBy' relatedIdentifierType='DOI'>https://doi.org/10.7554/eLife.52482</relatedIdentifier>
</relatedIdentifiers>
<version>1</version>
<rightsList>
<rights rightsURI='https://creativecommons.org/licenses/by/4.0/' rightsIdentifier='CC BY'>Creative Commons Attribution 4.0 International</rights>
</rightsList>
<descriptions>
<description descriptionType='Other'>
Helical cell shape is necessary for efficient stomach colonization by
Helicobacter pylori, but the molecular mechanisms for generating helical
shape remain unclear. We show that the helical centerline pitch and radius
of wild-type H. pylori cells dictate surface curvatures of considerably
higher positive and negative Gaussian curvatures than those present in
straight- or curved-rod bacteria. Quantitative 3D microscopy analysis of
short pulses with either N-acetylmuramic acid or D-alanine metabolic
probes showed that cell wall growth is enhanced at both sidewall curvature
extremes. Immunofluorescence revealed MreB is most abundant at negative
Gaussian curvature, while the bactofilin CcmA is most abundant at positive
Gaussian curvature. Strains expressing CcmA variants with altered
polymerization properties lose helical shape and associated positive
Gaussian curvatures. We thus propose a model where CcmA and MreB promote
PG synthesis at positive and negative Gaussian curvatures, respectively,
and that this patterning is one mechanism necessary for maintaining
helical shape. This dataset includes structured illumination fluorescence
microscopy images (SIM) and their associated cell shape reconstructions,
phase contrast micrographs, and transmission electron micrographs. See the
README.txt for detailed description of the strains and conditions
represented in each data file.
</description>
</descriptions>
</resource>
Posted to #digitalopendataandresearch channel for approval on 31 Oct 2022
oops: ERB syntax sneaked through...
convert_to_draft
by Chuck McCallum at 2022-10-31 20:14:08.Last commit should fix CI, but I should go back, review the issue, and confirm that this does what we need.
convert_to_draft
by None at 2022-10-31 19:45:54.head_ref_force_pushed
by James R. Griffin III at 2022-10-31 13:55:11.head_ref_force_pushed
by James R. Griffin III at 2022-10-31 17:14:22.head_ref_force_pushed
by James R. Griffin III at 2022-10-31 17:33:14.head_ref_force_pushed
by James R. Griffin III at 2022-10-31 17:34:53.head_ref_force_pushed
by James R. Griffin III at 2022-10-31 17:37:15.head_ref_force_pushed
by James R. Griffin III at 2022-10-31 17:44:14.ready_for_review
by James R. Griffin III at 2022-10-31 17:44:19.review_requested
by James R. Griffin III at 2022-10-31 18:40:35.convert_to_draft
by James R. Griffin III at 2022-10-31 19:07:10.review_dismissed
by James R. Griffin III at 2022-10-31 19:22:07.head_ref_force_pushed
by James R. Griffin III at 2022-10-31 19:22:07.referenced
by Christina Chortaria at 2022-10-31 18:24:17.merged
by Christina Chortaria at 2022-10-31 18:24:17.closed
by Christina Chortaria at 2022-10-31 18:24:17.head_ref_deleted
by Christina Chortaria at 2022-10-31 18:24:18.assigned
by Jane Sandberg at 2022-10-31 16:48:20.unassigned
by Shaun Ellis at 2022-10-31 19:39:10.ready_for_review
by None at 2022-10-31 18:31:10.review_requested
by None at 2022-10-31 20:03:35.deployed
by None at 2022-10-31 20:05:54.More examples: https://findingaids.princeton.edu/catalog/LAE006 (not abid) https://findingaids.princeton.edu/catalog/C1513 https://findingaids.princeton.edu/catalog/C1515 https://findingaids.princeton.edu/catalog/C1532 https://findingaids.princeton.edu/catalog/C1571 https://findingaids.princeton.edu/catalog/C0910 (mixture) https://findingaids.princeton.edu/catalog/C1381 (mixture)
closed
by Kevin Reiss at 2022-10-31 04:52:09.closed
by Kevin Reiss at 2022-10-31 04:52:18.closed
by Kevin Reiss at 2022-10-31 04:52:24.


Locksmith Dorothy:
Why have you come to see me?
"Nikola thought you might be able to repair these broken metal pieces."


Pig Cookbook
- Overview
-
Performance Enhancers
- Use Optimization
- Use Types
- Project Early and Often
- Filter Early and Often
- Reduce Your Operator Pipeline
- Make Your UDFs Algebraic
- Implement the Aggregator Interface
- Drop Nulls Before a Join
- Take Advantage of Join Optimizations
- Use the Parallel Features
- Use the LIMIT Operator
- Prefer DISTINCT over GROUP BY/GENERATE
- Compress the Results of Intermediate Jobs
- Combine Small Input Files
Overview
This document provides hints and tips for pig users.
Performance Enhancers
Use Optimization
Pig supports various optimization rules which are turned on by default. Become familiar with these rules.
Use Types
If types are not specified in the load statement, Pig assumes the type of =double= for numeric computations. A lot of the time, your data would be much smaller, maybe, integer or long. Specifying the real type will help with speed of arithmetic computation. It has an additional advantage of early error detection.
--Query 1 A = load 'myfile' as (t, u, v); B = foreach A generate t + u; --Query 2 A = load 'myfile' as (t: int, u: int, v); B = foreach A generate t + u;
The second query will run more efficiently than the first. In some of our queries with see 2x speedup.
Project Early and Often
Pig does not (yet) determine when a field is no longer needed and drop the field from the row. For example, say you have a query like:
A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); C = join A by t, B by x; D = group C by u; E = foreach D generate group, COUNT($1);
There is no need for v, y, or z to participate in this query. And there is no need to carry both t and x past the join, just one will suffice. Changing the query above to the query below will greatly reduce the amount of data being carried through the map and reduce phases by pig.
A = load 'myfile' as (t, u, v); A1 = foreach A generate t, u; B = load 'myotherfile' as (x, y, z); B1 = foreach B generate x; C = join A1 by t, B1 by x; C1 = foreach C generate t, u; D = group C1 by u; E = foreach D generate group, COUNT($1);
Depending on your data, this can produce significant time savings. In queries similar to the example shown here we have seen total time drop by 50%.
Filter Early and Often
As with early projection, in most cases it is beneficial to apply filters as early as possible to reduce the amount of data flowing through the pipeline.
-- Query 1 A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); C = filter A by t == 1; D = join C by t, B by x; E = group D by u; F = foreach E generate group, COUNT($1); -- Query 2 A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); C = join A by t, B by x; D = group C by u; E = foreach D generate group, COUNT($1); F = filter E by C.t == 1;
The first query is clearly more efficient than the second one because it reduces the amount of data going into the join.
One case where pushing filters up might not be a good idea is if the cost of applying filter is very high and only a small amount of data is filtered out.
Reduce Your Operator Pipeline
For clarity of your script, you might choose to split your projects into several steps for instance:
A = load 'data' as (in: map[]); -- get key out of the map B = foreach A generate in#k1 as k1, in#k2 as k2; -- concatenate the keys C = foreach B generate CONCAT(k1, k2); .......
While the example above is easier to read, you might want to consider combining the two foreach statements to improve your query performance:
A = load 'data' as (in: map[]); -- concatenate the keys from the map B = foreach A generate CONCAT(in#k1, in#k2); ....
The same goes for filters.
Make Your UDFs Algebraic
Queries that can take advantage of the combiner generally ran much faster (sometimes several times faster) than the versions that don't. The latest code significantly improves combiner usage; however, you need to make sure you do your part. If you have a UDF that works on grouped data and is, by nature, algebraic (meaning their computation can be decomposed into multiple steps) make sure you implement it as such. For details on how to write algebraic UDFs, see the Pig UDF Manual and Aggregate Functions.
A = load 'data' as (x, y, z) B = group A by x; C = foreach B generate group, MyUDF(A); ....
If MyUDF is algebraic, the query will use combiner and run much faster. You can run explain command on your query to make sure that combiner is used.
Implement the Aggregator Interface
If your UDF can't be made Algebraic but is able to deal with getting input in chunks rather than all at once, consider implementing the Aggregator interface to reduce the amount of memory used by your script.If your function is Algebraic and can be used on conjunction with Accumulator functions, you will need to implement the Accumulator interface as well as the Algebraic interface. For more information, see the Pig UDF Manual and Accumulator Interface.
Drop Nulls Before a Join
With the introduction of nulls, join and cogroup semantics were altered to work with nulls. The semantic for cogrouping with nulls is that nulls from a given input are grouped together, but nulls across inputs are not grouped together. This preserves the semantics of grouping (nulls are collected together from a single input to be passed to aggregate functions like COUNT) and the semantics of join (nulls are not joined across inputs). Since flattening an empty bag results in an empty row (and no output), in a standard join the rows with a null key will always be dropped.
This join
A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); C = join A by t, B by x;
is rewritten by Pig to
A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); C1 = cogroup A by t INNER, B by x INNER; C = foreach C1 generate flatten(A), flatten(B);
Since the nulls from A and B won't be collected together, when the nulls are flattened we're guaranteed to have an empty bag, which will result in no output. So the null keys will be dropped. But they will not be dropped until the last possible moment.
If the query is rewritten to
A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); A1 = filter A by t is not null; B1 = filter B by x is not null; C = join A1 by t, B1 by x;
then the nulls will be dropped before the join. Since all null keys go to a single reducer, if your key is null even a small percentage of the time the gain can be significant. In one test where the key was null 7% of the time and the data was spread across 200 reducers, we saw a about a 10x speed up in the query by adding the early filters.
Take Advantage of Join Optimizations
Regular Join Optimizations
Optimization for regular joins ensures that the last table in the join is not brought into memory but streamed through instead. Optimization reduces the amount of memory used which means you can avoid spilling the data and also should be able to scale your query to larger data volumes.
To take advantage of this optimization, make sure that the table with the largest number of tuples per key is the last table in your query. In some of our tests we saw 10x performance improvement as the result of this optimization.
small = load 'small_file' as (t, u, v); large = load 'large_file' as (x, y, z); C = join small by t, large by x;
Specialized Join Optimizations
Optimization can also be achieved using fragment replicate joins, skewed joins, and merge joins. For more information see Specialized Joins.
Use the Parallel Features
You can set the number of reduce tasks for the MapReduce jobs generated by Pig using two parallel features. (The parallel features only affect the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block.)
You Set the Number of Reducers
Use the set default parallel command to set the number of reducers at the script level.
Alternatively, use the PARALLEL clause to set the number of reducers at the operator level. (In a script, the value set via the PARALLEL clause will override any value set via "set default parallel.") You can include the PARALLEL clause with any operator that starts a reduce phase: COGROUP, CROSS, DISTINCT, GROUP, JOIN (inner), JOIN (outer), and ORDER BY.
The number of reducers you need for a particular construct in Pig that forms a MapReduce boundary depends entirely on (1) your data and the number of intermediate keys you are generating in your mappers and (2) the partitioner and distribution of map (combiner) output keys. In the best cases we have seen that a reducer processing about 1 GB of data behaves efficiently.
Let Pig Set the Number of Reducers
If neither "set default parallel" nor the PARALLEL clause are used, Pig sets the number of reducers using a heuristic based on the size of the input data. You can set the values for these properties:
- pig.exec.reducers.bytes.per.reducer - Defines the number of input bytes per reduce; default value is 1000*1000*1000 (1GB).
- pig.exec.reducers.max - Defines the upper bound on the number of reducers; default is 999.
The formula, shown below, is very simple and will improve over time. The computed value takes all inputs within the script into account and applies the computed value to all the jobs within Pig script.
#reducers = MIN (pig.exec.reducers.max, total input size (in bytes) / bytes per reducer)
Examples
In this example PARALLEL is used with the GROUP operator.
A = LOAD 'myfile' AS (t, u, v); B = GROUP A BY t PARALLEL 18; ...
In this example all the MapReduce jobs that get launched use 20 reducers.
SET default_parallel 20; A = LOAD ‘myfile.txt’ USING PigStorage() AS (t, u, v); B = GROUP A BY t; C = FOREACH B GENERATE group, COUNT(A.t) as mycount; D = ORDER C BY mycount; STORE D INTO ‘mysortedcount’ USING PigStorage();
Use the LIMIT Operator
Often you are not interested in the entire output but rather a sample or top results. In such cases, using LIMIT can yield a much better performance as we push the limit as high as possible to minimize the amount of data travelling through the pipeline.
Sample:
A = load 'myfile' as (t, u, v); B = limit A 500;
Top results:
A = load 'myfile' as (t, u, v); B = order A by t; C = limit B 500;
Prefer DISTINCT over GROUP BY/GENERATE
To extract unique values from a column in a relation you can use DISTINCT or GROUP BY/GENERATE. DISTINCT is the preferred method; it is faster and more efficient.
Example using GROUP BY - GENERATE:
A = load 'myfile' as (t, u, v); B = foreach A generate u; C = group B by u; D = foreach C generate group as uniquekey; dump D;
Example using DISTINCT:
A = load 'myfile' as (t, u, v); B = foreach A generate u; C = distinct B; dump C;
Compress the Results of Intermediate Jobs
If your Pig script generates a sequence of MapReduce jobs, you can compress the output of the intermediate jobs using LZO compression. (Use the EXPLAIN operator to determine if your script produces multiple MapReduce Jobs.)
By doing this, you will save HDFS space used to store the intermediate data used by PIG and potentially improve query execution speed. In general, the more intermediate data that is generated, the more benefits in storage and speed that result.
You can set the value for these properties:
- pig.tmpfilecompression - Determines if the temporary files should be compressed or not (set to false by default).
- pig.tmpfilecompression.codec - Specifies which compression codec to use. Currently, Pig accepts "gz" and "lzo" as possible values. However, because LZO is under GPL license (and disabled by default) you will need to configure your cluster to use the LZO codec to take advantage of this feature. For details, see http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ.
On the non-trivial queries (one ran longer than a couple of minutes) we saw significant improvements both in terms of query latency and space usage. For some queries we saw up to 96% disk saving and up to 4x query speed up. Of course, the performance characteristics are very much query and data dependent and testing needs to be done to determine gains. We did not see any slowdown in the tests we peformed which means that you are at least saving on space while using compression.
With gzip we saw a better compression (96-99%) but at a cost of 4% slowdown. Thus, we don't recommend using gzip.
Example
-- launch Pig script using lzo compression java -cp $PIG_HOME/pig.jar -Djava.library.path=<path to the lzo library> -Dpig.tmpfilecompression=true -Dpig.tmpfilecompression.codec=lzo org.apache.pig.Main myscript.pig
Combine Small Input Files
Processing input (either user input or intermediate input) from multiple small files can be inefficient because a separate map has to be created for each file. Pig can now combined small files so that they are processed as a single map.
You can set the values for these properties:
- pig.maxCombinedSplitSize – Specifies the size, in bytes, of data to be processed by a single map. Smaller files are combined untill this size is reached.
- pig.splitCombination – Turns combine split files on or off (set to “true” by default).
This feature works with PigStorage. However, if you are using a custom loader, please note the following:
- If your loader implementation makes use of the PigSplit object passed through the prepareToRead method, then you may need to rebuild the loader since the definition of PigSplit has been modified.
- The loader must be stateless across the invocations to the prepareToRead method. That is, the method should reset any internal states that are not affected by the RecordReader argument.
- If a loader implements IndexableLoadFunc, or implements OrderedLoadFunc and CollectableLoadFunc, its input splits won't be subject to possible combinations.


I am a former music student turned frontend developer, having in the last few years discovered the creative joy of web design and programming. What started as an interest in designing the visual part of websites with HTML and CSS eventually led to a number of university-level courses in both the theoretical and practical aspects of how we use and develop the internet today. At the time of writing I am working on personal projects as well as of course constantly trying to broaden my knowledge in HTML, CSS, JavaScript, and whatever else can be used to create awesome websites.
What I mostly use for frontend development:
If you are interested in recruiting me for a project, or if you have any comments, feel free to email me at<EMAIL_ADDRESS>or use one of the links below.
GitHub


--- layout: post title: rough Storyboards date: '2017-12-20T04:18:35+01:00' categories: - Storyboards picture_url: https://film-Storyboards.tumblr.com/post/168747135716/rough-Storyboards ---


--- layout: archive title: Zulip Chat Archive permalink: /stream/113488-general/topic/evaluating.20proofs.3F.html ---
@Mario Carneiro I think you said that it is possible to evaluate proofs... do you have an example? can we also break proof irrelevance? maybe have an unsound but computable non-classical.some
?
the axiom of choice is definitely nonconstructive
You can evaluate proofs by using #reduce
the axiom of choice is definitely nonconstructive
even unsound?
no...
wait, I don't think I understand what you are asking
it's easy to compute a value if you are allowed to be wrong
so I want a function that reduce
s to 0
when given the input (⟨0, rfl⟩ : ∃ n, n = n)
Not in the VM
#reduce (⟨0, rfl⟩ : ∃ n, n = n) -- Exists.intro 0 (eq.refl 0)
why not?
because that term has no representation in the VM
it is a neutral value, a "unit"
so reduce
is not VM?
no
it is kernel reduction
interesting
{% endraw %}Last updated: Apr 24 2020 at 19:02 UTC


World Map
Map Stats
Reset
Rural Population
Forest Area
Year:
The map dataset was obtained from
Natural Earth
public domain.
Global development data obtained from
The World Bank.


A bite that makes you smile :)MASAH Infused Restaurant
Our Amazing Team
Click to link personal website
Florian Herkert
Member
Ashley
Marketing Leader
Lee Hansol
Technical Leader
Misbahul Hasan
Project Manager
Kim Dongsun
Project Manager
Shen Yichuan
Knowledge Leader
Ma KyungHo
Technical Leader
Lorem ipsum dolor sit amet, consectetur adipisicing elit. Aut eaque, laboriosam veritatis, quos non quis ad perspiciatis, totam corporis ea, alias ut unde.


Bid me discourse, I will enchant thine ear, Or like a fairy trip upon the green, Or, like a nymph, with long dishevell'd hair, Dance on the sands, and yet no footing seen: Love is a spirit all compact of fire, Not gross to sink, but light, and will aspire.


ANNOVA
Phone:8887951971
/
<EMAIL_ADDRESS>Toggle Navigation
Home
Research
Policy Design
Services
About Us
Contact Us
Service Price
Toggle Search


Hi my name is Leon
I am an avid Computer Scientist and Athlete.
I am currently studying a Computer Science and Mathematics Degree at the
University of Bath, UK. However there is not a lot of emphasis on web-development
and so this is the portfolio I have created on the passage of teaching myself
full-stack-development.


This article describes the Thread Pool design pattern. This is a concurrency design pattern, a category of design pattern used by software engineers, when writing computer programs.
Table of Contents
Introduction
The Thread Pool pattern is a design pattern, used in software engineering to organise the processing of a large number of queued tasks through a smaller/limited number of threads. Results can also be queued. When a thread finishes it's task it requests another. If none are available, the thread can wait or terminate.
The size of the thread pool is a tuning parameter that allows fine tuning of resource usage and responsiveness, for best results. Pooling threads helps improve performance by saving time creating the thread, however, too many idle threads can consume resources. The number of threads is based on percentage used of CPU, queued requests, and/or the number of processors in the system.
This pattern is defined as a concurrency design pattern because it relates to thread management.
Benefits
Better performance is the main gain with this pattern. Juggling the various factors that determine performance becomes more important as computers leaverage multiple processor cores.
If the number of tasks is large, creating a thread for each may become impossible or impractical, so the best answer is a thread pool.
Examples of the pattern
A web server increases of decreases the number of request processing threads, depending on load.
- A Method of Worker Thread Pooling (C++)
- Java theory and practice: Thread pools and work queues (Java)
- Multithreaded Programming with Visual Basic .NET (VB.net)
- Practical threaded programming with Python (Python)
- How to: Use a Thread Pool (C#)
See Also
Link to domain parent articles and related articles in TechNet Wiki.
Community Resources
These are the external links, including links to Microsoft and TechNet sites that are non-Wiki- Programming the Thread Pool in the .NET Framework
- .NET Framework Class Library: ThreadPool Class
- .NET Framework Developer's Guide: Threading
- The Managed Thread Pool
- Threads & Threading
References section
Use this section if you pulled source material and ideas from other sites, blogs, or forums. Make sure you have permission from authors to use their material.- [Please contribute]


How to find your Apple Watch's serial number using your Apple Watch or iPhone, Business Insider - Business Insider Singapore
by Jennifer Still, Business Insider US- You can find your Apple Watch serial number in a few different ways.
- Your Apple Watch serial number provides you with information on which version of the Apple Watch you have.
- Visit Business Insider’s homepage for more stories.
While generally speaking, knowing the serial number of your Apple Watch isn’t necessarily something you’ll need, being able to find it is important should the need arise.
After all, the serial number is the definitive way of identifying your watch and letting you know which model you have. It can also be important when cashing in on a warranty or insurance policy you’ve taken out on the device, both of which will likely be tied to the watch’s serial number.
There are several ways you can find out your Apple Watch‘s serial number, both on the device itself and via your iPhone. Each method is simple and straightforward and will get you the info you need in just a few seconds. Here’s how to do it.
Check out the products mentioned in this article:
Apple Watch Series 5 (From $399 at Best Buy)
iPhone 11 (From $699.99 at Best Buy)
How to find your Apple Watch‘s serial number using your Apple Watch
1. Press the Apple Watch’s digital crown to launch the app home screen.
2. Find the Settings app, which looks like a small cog wheel, and tap it to open.
3. Scroll down and tap “General.”
4. Tap “About.”
5. Scroll down to the “Serial Number” section and your device’s serial number will be listed below.
If you don’t have your Apple Watch handy at the moment but still want to know the device’s serial number, you can do so via your paired iPhone.
How to find your Apple Watch’s serial number using your iPhone
1. Tap the Watch app on your iPhone’s home screen to open it.
2. Under the “My Watch” tab, scroll down to find “General,” and tap to open.
3. Tap “About.” The watch’s serial number will be displayed on this screen.
Related coverage from How To Do Everything: Tech:
-
How to share your Apple Watch Activity and send messages to friends during your workout
-
How to change your Apple Watch to military time in 4 steps, using the Watch app on your iPhone
-
How to flip your Apple Watch screen by changing its orientation, in 2 different ways
-
How to turn off Airplane Mode on your Apple Watch in 2 different ways


Fabflix
MAIN
Customer Info
First Name
Last Name
Credit Card No
Credit Card Expiration date
(Eg: YYYY-MM-DD)
Address


Let’s go to your favorite coffee shop or wherever you feel comfortable and let’s be birth nerds for a couple of hours. It is a moment for us to get acquainted and for you to learn more about what I offer as labor and postpartum Doula and see that we are a good fit for each other.
Schedule your Free Doula Care Consultation now, click here.
My offerings to you as your Labor Doula includes the following:
Every birth is unique, and it is a precious moment that you will forever remember. You deserve all my attention and to do just that I accept a limited amount of clients per month and book several months in advance. You may with me at any time to discuss availability around your estimated due date.
You can connect me at<PHONE_NUMBER> or at<EMAIL_ADDRESS>or Schedule your Free Birth Choices Consultation now, click here.
 The first 50% of my fee is due as a retainer to secure my availability for you, cover the prenatal visits, and my on-call commitment to you. The remaining 50% of the fee covers the birth and postpartum visit and is due by week 36 of gestation. Please note that if you hire me at or after 36 weeks, the whole fee is due at the time of the contract signature.Below you will find the different investments you can make towards your postpartum period or a loved one’s.
For more details on my offerings as you postpartum doula, visit my postpartum page,
here.
10 Hours Day Support - $400
With this package, you receive ten hours of daytime postpartum care, set to the schedule that works best for you. If the package is purchased before the baby arrives you will have a 1-hour prenatal visit to connect before the baby arrives and three 3-hour visits to be scheduled anytime in the first three months.
22 Hours Day Support - $880
This Day Support Postpartum Package gives you twenty-one hours of postpartum doula care set to the schedule that works best for you. If the package is purchased before the baby arrives you will have a 1-hour prenatal visit to connect before the baby arrives and 21 hours to be scheduled anytime in the first three months in 3-hours visits.
3 – 8 Hours Night Support $1,200
This Night Support Postpartum Package gives you twenty-four hours of postpartum doula care, spread out over three nights.
6 – 8 Hours Night Support $2,400
This Night Support Postpartum Package gives you forty-eight hours of postpartum doula care, spread out over three nights.
As every birth is unique, so is each family postpartum experience. You deserve all my attention and to do just that I accept a limited amount of client per month and book several months in advance. You may with me at any time to discuss availability around your estimated due date.
You can connect me at<PHONE_NUMBER> or<EMAIL_ADDRESS>or Schedule your Free Care Consultation now, click here.
 Payment plans for postpartum care must be PAID IN FULL before the start of services. If you need a payment plan, please contact me at doulabelle@oulaladoula.com.


/home/tungly/env/projects/l42
/
vendor
/
patchwork
/
utf8
/
class
(Dashboard)
Class Coverage Distribution
Class Complexity
Top Project Risks
Bootup
(3080)
Least Tested Methods
Bootup::filterRequestUri
(0%)
Bootup::anonymous function
(0%)
Bootup::filterRequestInputs
(0%)
Bootup::filterString
(0%)
Bootup::initLocale
(0%)
Bootup::initIntl
(0%)
Bootup::initUtf8Encode
(0%)
Bootup::initMbstring
(0%)
Bootup::initIconv
(0%)
Bootup::initExif
(0%)


Our Bioinformatics Studio is an inclusive environment for bioinformatics at Temple University. Whether you’ve never coded before or you’re a coding whiz, we’ll help you jump into the mix!
SlackGuided hands-on bioinformatics training at every level of expertise in an open studio format. Learn bioinformatics coding and tools by doing it yourself. We’ll meet you where you’re at and get you started in no time!
Programming interview preparation and drills. Build out your Git portfolio with collaborative bioinformatics projects. Take your programming skills to the next level!
Hear and discuss bleeding-edge science with invited speakers from industry and academia.
| Date | Speaker/Tutorial | Time |
|---|---|---|
| 8/30/19 | Welcome Session! Pizza will be supplied! | 5-6PM |
| 9/6/19 | TBD | |
| 9/13/19 | TBD | |
| 9/20/19 | TBD | |
| 9/27/19 | TBD | |
| 10/4/19 | TBD | |
| 10/11/19 | TBD | |
| 10/18/19 | TBD | |
| 10/25/19 | TBD | |
| 11/1/19 | Happy Halloween! | 5-6pm |
| 11/8/19 | TBD | |
| 11/15/19 | TBD | |
| 11/22/19 | TBD | |
| 11/29/19 | No session! Happy Thanksgiving! | |
| 12/6/19 | Last Session! Pizza will be supplied! | 5-6pm |
| Available Tutorials |
|---|
| Intro to Python |
| PANDAS Introduction |
Officer
Officer
Officer
Officer
We're currently looking for undergraduate officers and a webmaster. Contact us if interested!


Applications, at times, may want to show some portion of UI in a different language other than the rest of the application. Applications can be more explicit when loading resources by making a call to the current Windows.ApplicationModel.Resources.Core.ResourceManager with an overridden context. The overridden context then lets the application be more specific about the configuration and type of resource to load. Advanced functionality such as this is typically found in the Windows.ApplicationModel.Resources.Core namespace within the ResourceManager class.
To view a resource that has been loaded inline in a different language, choose the language below and click Show.


Declare click listeners for <Button />
elements of a custom notification layout.
ℹ See Android Custom Notification Layout for setup instructions.
You can declare your own custom <Button />
elements and register click-listeners upon them using the actions parameter:
Customize the notification channel ID.
Defaults to your.package.name.TSLocationManager
NOTE: It is not typically required to change this. Typical use-cases are for users who use an existing Android foreground-service who wish the SDK to share an existing notification and channel.
Configure the name of the plugin's notification-channel used to display the Config.foregroundService notification.
Configure the color of the persistent notification icon in the Notification Bar when running with Config.foregroundService true
Configure the large icon of the persistent notification in the Notification Bar when running with Config.foregroundService true
Specifies the name of your custom Android Layout XML file.
ℹ See Android Custom Notification Layout for setup instructions.
Even if you have no experience with Android Layouts, it doesn't take much to figure out the basics. You'll mostly be adding <TextView />
, <ImageView />
and <Button />
elements. The key thing to be aware of is the android:id
of these elements and how these are referenced from BackgroundGeolocation.notification
configuration: your android:id
must be prefixed with the word notification
(eg: notificationText
). There is one exception: applicationName
, which the plugin will automatically render your Android application name.
When BackgroundGeolocation renders your custom notification layout, it will be querying for the following elements addressed by their android:id
. When found, their content will be updated from the corresponding "Data-source":
Layout element android:id |
Data-source |
|---|---|
applicationName |
Application name from AndroidManifest |
notificationTitle |
title |
notificationText |
text |
notificationSmallIcon |
smallIcon |
notificationLargeIcon |
largeIcon |
When running the service with foregroundService: true, Android requires a persistent notification in the Notification Bar. This will control the priority of that notification as well as the position of the notificaiton-bar icon.
The following notificationPriority
values defined as static constants upon the BackgroundGeolocation object:
| Value | Description |
|---|---|
| BackgroundGeolocation.NOTIFICATION_PRIORITY_DEFAULT | Notification weighted to top of list; notification-bar icon weighted left |
| BackgroundGeolocation.NOTIFICATION_PRIORITY_HIGH | Notification strongly weighted to top of list; notification-bar icon strongly weighted to left |
| BackgroundGeolocation.NOTIFICATION_PRIORITY_LOW | Notification weighted to bottom of list; notification-bar icon weighted right |
| BackgroundGeolocation.NOTIFICATION_PRIORITY_MAX | Same as NOTIFICATION_PRIORITY_HIGH |
| BackgroundGeolocation.NOTIFICATION_PRIORITY_MIN | Notification strongly weighted to bottom of list; notification-bar icon hidden |
Configure the small icon of the persistent notification in the Notification Bar when running with Config.foregroundService true
Configure the Android Foreground Service icon and notification to be displayed always. Defaults to false
.
The default behaviour is for the notification to be shown only while the SDK detects the device to be moving. Some developers desire to provide full-disclosure to their users when the SDK has been enabled.
Custom strings to render into <TextView />
elements of a custom notification layout.
ℹ See Android Custom Notification Layout for setup instructions.
You can declare your own custom <TextView />
elements and render data into them using the notification.strings
parameter.
Configure the text of the persistent notification in the Notification Bar when running with Config.foregroundService true
Configure the title of the persistent notification in the Notification Bar when running with Config.foregroundService true
Generated using TypeDoc
[Android only] The Android operating system requires a persistent notification when running a foreground service.
Custom Notification Layouts
Use the layout option to provide the name of your own Android Layout XML File. See Android Custom Notification Layout for setup instructions.


DATE:
{{myForm.date | date}}
Contract Price/CWT:
{{myForm.pricePerCWT | number:2}}
Exchange Rate:
{{myForm.exchangeRate}}
Contract Quantity (CWT):
{{myForm.qtyCWT | number:2}}
No of Shipments:
{{myForm.noOfShupment}}
FCL’s/shipment:
{{myForm.packedId}}
Brokerage Commission:
{{myForm.brokerageCWT | number:2}}
Brokerage/CWT:
{{myForm.commisionCWT | number:2}}
Ocean freight + $100/FC:
${{myForm.oceanFreightBL | number:2}}
FCL/Shipment:
{{myForm.packedId}}
Ocean frt/CWT:
${{myForm.oceanFreightCWT | number:2}}
Bl/fee:
${{myForm.blFee}}
BL/fee per CWT:
${{myForm.blFeeCWT | number:2}}
Bl/fee:
${{myForm.blFee}}
Total Freight/CWT:
${{myForm.totalBlFeeCWT | number:2}}
Documentation Costing:
${{myForm.documentCosting}}
Document cost/CWT:
${{myForm.documentCostingCWT | number:2}}
LC Costing:
${{myForm.lcCost}}
LC Costing/CWT:
${{myForm.lcCostCWT | number:2}}
Insurance Rate:
${{myForm.insuranceRate}}
Insurance Rate CWT:
${{myForm.insuranceRateCWT | number:2}}
ARI Policy:
${{myForm.ariPolicy}}
ARI Policy CWT:
${{myForm.ariPolicyCWT | number:2}}
Price in USD:
${{myForm.priceUSD | number:2}}
Interest days X Rate:
${{myForm.interestDays}}
Price in CAD:
${{myForm.priceCAD | number:2}}
CA$ Price/CWT:
${{myForm.cadCWT | number:2}}
Interest days X Rate:
{{myForm.interestDays | number:2}}
Interest Rate/CWT:
${{myForm.interestRateCWT | number:2}}
Stuffing:
{{myForm.stuffingType}}
Stuffing/CWT:
${{myForm.stuffingCWT | number:2}}
Inland Freight:
{{myForm.inlineFreight}}
Inland Freight CWT:
${{myForm.inlineFreightCWT | number:2}}
Bag Type:
{{myForm.bagCostType}}
Bag Cost CWT:
${{myForm.bagCostCWT | number:2}}
Certificate of Analysis:
{{myForm.certificateAnalysisName}}
COA Cost CWT:
${{myForm.coaCost | number:2}}
Inland freight stuffing buffer:
{{myForm.stuffingBuffer}}
Misc Cost - {{myForm.missCost1}}:
${{myForm.missCostCWT1 | number:2}}
Inland freight stuffing buffer:
{{myForm.stuffingBuffer}}
Misc Cost - {{myForm.missCost2}}:
${{myForm.missCostCWT2 | number:2}}
Inland freight stuffing buffer:
{{myForm.stuffingBuffer}}
Net FOB(CAD):
${{myForm.netFOBCAD | number:2}}
Inland freight stuffing buffer:
{{myForm.stuffingBuffer}}
Target FOB(CAD):
${{myForm.targetFOBCAD | number:2}}
Inland freight stuffing buffer:
{{myForm.stuffingBuffer}}
Over/Under Target:
${{myForm.underTarget | number:2}}
{{d.shipmentType || '--'}}
{{(d.end_date | date)|| '--'}}
{{ (d.ship | number:2) || '--'}}
{{d.exchangeRate|| '--'}}