Retrieving Data & Results Sets
-
class
Cake\ORM\
Table
While table objects provide an abstraction around a ‘repository’ or collection
of objects, when you query for individual records you get ‘entity’ objects.
While this section discusses the different ways you can find and load entities,
you should read the Entities section for more information on
entities.
Debugging Queries and ResultSets
Since the ORM now returns Collections and Entities, debugging these objects can
be more complicated than in previous CakePHP versions. There are now various
ways to inspect the data returned by the ORM.
debug($query)
Shows the SQL and bound params, does not show results.
debug($query->all())
Shows the ResultSet properties (not the results).
debug($query->toArray())
An easy way to show each of the results.
debug(json_encode($query, JSON_PRETTY_PRINT))
More human readable results.
debug($query->first())
Show the properties of a single entity.
debug((string)$query->first())
Show the properties of a single entity as JSON.
Getting a Single Entity by Primary Key
-
Cake\ORM\Table::
get
($id, $options = [])
It is often convenient to load a single entity from the database when editing or
view entities and their related data. You can do this by using get()
:
// In a controller or table method.
// Get a single article
$article = $articles->get($id);
// Get a single article, and related comments
$article = $articles->get($id, [
'contain' => ['Comments']
]);
If the get operation does not find any results a
Cake\Datasource\Exception\RecordNotFoundException
will be raised. You can
either catch this exception yourself, or allow CakePHP to convert it into a 404
error.
Like find()
, get()
also has caching integrated. You can use the
cache
option when calling get()
to perform read-through caching:
// In a controller or table method.
// Use any cache config or CacheEngine instance & a generated key
$article = $articles->get($id, [
'cache' => 'custom',
]);
// Use any cache config or CacheEngine instance & specific key
$article = $articles->get($id, [
'cache' => 'custom', 'key' => 'mykey'
]);
// Explicitly disable caching
$article = $articles->get($id, [
'cache' => false
]);
Optionally you can get()
an entity using Custom Finder Methods. For
example you may want to get all translations for an entity. You can achieve that
by using the finder
option:
$article = $articles->get($id, [
'finder' => 'translations',
]);
Using Finders to Load Data
-
Cake\ORM\Table::
find
($type, $options = [])
Before you can work with entities, you’ll need to load them. The easiest way to
do this is using the find()
method. The find method provides an easy and
extensible way to find the data you are interested in:
// In a controller or table method.
// Find all the articles
$query = $articles->find('all');
The return value of any find()
method is always
a Cake\ORM\Query
object. The Query class allows you to further
refine a query after creating it. Query objects are evaluated lazily, and do not
execute until you start fetching rows, convert it to an array, or when the
all()
method is called:
// In a controller or table method.
// Find all the articles.
// At this point the query has not run.
$query = $articles->find('all');
// Iteration will execute the query.
foreach ($query as $row) {
}
// Calling all() will execute the query
// and return the result set.
$results = $query->all();
// Once we have a result set we can get all the rows
$data = $results->toArray();
// Converting the query to an array will execute it.
$data = $query->toArray();
Note
Once you’ve started a query you can use the Query Builder
interface to build more complex queries, adding additional conditions,
limits, or include associations using the fluent interface.
// In a controller or table method.
$query = $articles->find('all')
->where(['Articles.created >' => new DateTime('-10 days')])
->contain(['Comments', 'Authors'])
->limit(10);
You can also provide many commonly used options to find()
. This can help
with testing as there are fewer methods to mock:
// In a controller or table method.
$query = $articles->find('all', [
'conditions' => ['Articles.created >' => new DateTime('-10 days')],
'contain' => ['Authors', 'Comments'],
'limit' => 10
]);
The list of options supported by find() are:
conditions
provide conditions for the WHERE clause of your query.
limit
Set the number of rows you want.
offset
Set the page offset you want. You can also use page
to make
the calculation simpler.
contain
define the associations to eager load.
fields
limit the fields loaded into the entity. Only loading some fields
can cause entities to behave incorrectly.
group
add a GROUP BY clause to your query. This is useful when using
aggregating functions.
having
add a HAVING clause to your query.
join
define additional custom joins.
order
order the result set.
Any options that are not in this list will be passed to beforeFind listeners
where they can be used to modify the query object. You can use the
getOptions()
method on a query object to retrieve the options used. While
you can pass query objects to your controllers, we recommend that you package
your queries up as Custom Finder Methods instead. Using custom finder
methods will let you re-use your queries and make testing easier.
By default queries and result sets will return Entities objects. You
can retrieve basic arrays by disabling hydration:
$query->enableHydration(false);
// Prior to 3.4.0
$query->hydrate(false);
// $data is ResultSet that contains array data.
$data = $query->all();
Getting the First Result
The first()
method allows you to fetch only the first row from a query. If
the query has not been executed, a LIMIT 1
clause will be applied:
// In a controller or table method.
$query = $articles->find('all', [
'order' => ['Articles.created' => 'DESC']
]);
$row = $query->first();
This approach replaces find('first')
in previous versions of CakePHP. You
may also want to use the get()
method if you are loading entities by primary
key.
Note
The first()
method will return null
if no results are found.
Getting a Count of Results
Once you have created a query object, you can use the count()
method to get
a result count of that query:
// In a controller or table method.
$query = $articles->find('all', [
'conditions' => ['Articles.title LIKE' => '%Ovens%']
]);
$number = $query->count();
See Returning the Total Count of Records for additional usage of the count()
method.
Finding Key/Value Pairs
It is often useful to generate an associative array of data from your
application’s data. For example, this is very useful when creating <select>
elements. CakePHP provides a simple to use method for generating ‘lists’ of
data:
// In a controller or table method.
$query = $articles->find('list');
$data = $query->toArray();
// Data now looks like
$data = [
1 => 'First post',
2 => 'Second article I wrote',
];
With no additional options the keys of $data
will be the primary key of your
table, while the values will be the ‘displayField’ of the table. You can use the
setDisplayField()
method on a table object to configure the display field of
a table:
class ArticlesTable extends Table
{
public function initialize(array $config)
{
$this->setDisplayField('title');
// Prior to 3.4.0
$this->displayField('title');
}
}
When calling list
you can configure the fields used for the key and value
with the keyField
and valueField
options respectively:
// In a controller or table method.
$query = $articles->find('list', [
'keyField' => 'slug',
'valueField' => 'title'
]);
$data = $query->toArray();
// Data now looks like
$data = [
'first-post' => 'First post',
'second-article-i-wrote' => 'Second article I wrote',
];
Results can be grouped into nested sets. This is useful when you want
bucketed sets, or want to build <optgroup>
elements with FormHelper:
// In a controller or table method.
$query = $articles->find('list', [
'keyField' => 'slug',
'valueField' => 'title',
'groupField' => 'author_id'
]);
$data = $query->toArray();
// Data now looks like
$data = [
1 => [
'first-post' => 'First post',
'second-article-i-wrote' => 'Second article I wrote',
],
2 => [
// More data.
]
];
You can also create list data from associations that can be reached with joins:
$query = $articles->find('list', [
'keyField' => 'id',
'valueField' => 'author.name'
])->contain(['Authors']);
Customize Key-Value Output
Lastly it is possible to use closures to access entity accessor methods in your
list finds.
// In your Authors Entity create a virtual field to be used as the displayField:
protected function _getLabel()
{
return $this->_properties['first_name'] . ' ' . $this->_properties['last_name']
. ' / ' . __('User ID %s', $this->_properties['user_id']);
}
This example shows using the _getLabel()
accessor method from
the Author entity.
// In your finders/controller:
$query = $articles->find('list', [
'keyField' => 'id',
'valueField' => function ($article) {
return $article->author->get('label');
}
]);
You can also fetch the label in the list directly using.
// In AuthorsTable::initialize():
$this->setDisplayField('label'); // Will utilize Author::_getLabel()
// In your finders/controller:
$query = $authors->find('list'); // Will utilize AuthorsTable::getDisplayField()
Finding Threaded Data
The find('threaded')
finder returns nested entities that are threaded
together through a key field. By default this field is parent_id
. This
finder allows you to access data stored in an ‘adjacency list’ style table. All
entities matching a given parent_id
are placed under the children
attribute:
// In a controller or table method.
$query = $comments->find('threaded');
// Expanded default values
$query = $comments->find('threaded', [
'keyField' => $comments->primaryKey(),
'parentField' => 'parent_id'
]);
$results = $query->toArray();
echo count($results[0]->children);
echo $results[0]->children[0]->comment;
The parentField
and keyField
keys can be used to define the fields that
threading will occur on.
Tip
If you need to manage more advanced trees of data, consider using
Tree instead.
Custom Finder Methods
The examples above show how to use the built-in all
and list
finders.
However, it is possible and recommended that you implement your own finder
methods. Finder methods are the ideal way to package up commonly used queries,
allowing you to abstract query details into a simple to use method. Finder
methods are defined by creating methods following the convention of findFoo
where Foo
is the name of the finder you want to create. For example if we
wanted to add a finder to our articles table for finding published articles we
would do the following:
use Cake\ORM\Query;
use Cake\ORM\Table;
class ArticlesTable extends Table
{
public function findOwnedBy(Query $query, array $options)
{
$user = $options['user'];
return $query->where(['author_id' => $user->id]);
}
}
// In a controller or table method.
$articles = TableRegistry::get('Articles');
$query = $articles->find('ownedBy', ['user' => $userEntity]);
Finder methods can modify the query as required, or use the $options
to
customize the finder operation with relevant application logic. You can also
‘stack’ finders, allowing you to express complex queries effortlessly. Assuming
you have both the ‘published’ and ‘recent’ finders, you could do the following:
// In a controller or table method.
$articles = TableRegistry::get('Articles');
$query = $articles->find('published')->find('recent');
While all the examples so far have show finder methods on table classes, finder
methods can also be defined on Behaviors.
If you need to modify the results after they have been fetched you should use
a Modifying Results with Map/Reduce function to modify the results. The map reduce features
replace the ‘afterFind’ callback found in previous versions of CakePHP.
Dynamic Finders
CakePHP’s ORM provides dynamically constructed finder methods which allow you to
express simple queries with no additional code. For example if you wanted to
find a user by username you could do:
// In a controller
// The following two calls are equal.
$query = $this->Users->findByUsername('joebob');
$query = $this->Users->findAllByUsername('joebob');
// In a table method
$users = TableRegistry::get('Users');
// The following two calls are equal.
$query = $users->findByUsername('joebob');
$query = $users->findAllByUsername('joebob');
When using dynamic finders you can constrain on multiple fields:
$query = $users->findAllByUsernameAndApproved('joebob', 1);
You can also create OR
conditions:
$query = $users->findAllByUsernameOrEmail('joebob', 'joe@example.com');
While you can use either OR or AND conditions, you cannot combine the two in a
single dynamic finder. Other query options like contain
are also not
supported with dynamic finders. You should use Custom Finder Methods to
encapsulate more complex queries. Lastly, you can also combine dynamic finders
with custom finders:
$query = $users->findTrollsByUsername('bro');
The above would translate into the following:
$users->find('trolls', [
'conditions' => ['username' => 'bro']
]);
Once you have a query object from a dynamic finder, you’ll need to call
first()
if you want the first result.
Note
While dynamic finders make it simple to express queries, they come with some
additional performance overhead.
Retrieving Associated Data
When you want to grab associated data, or filter based on associated data, there
are two ways:
- use CakePHP ORM query functions like
contain()
and matching()
- use join functions like
innerJoin()
, leftJoin()
, and rightJoin()
You should use contain()
when you want to load the primary model, and its
associated data. While contain()
will let you apply additional conditions to
the loaded associations, you cannot constrain the primary model based on the
associations. For more details on the contain()
, look at
Eager Loading Associations Via Contain.
You should use matching()
when you want to restrict the primary model based
on associations. For example, you want to load all the articles that have
a specific tag on them. For more details on the matching()
, look at
Filtering by Associated Data Via Matching And Joins.
If you prefer to use join functions, you can look at
Adding Joins for more information.
Eager Loading Associations Via Contain
By default CakePHP does not load any associated data when using find()
.
You need to ‘contain’ or eager-load each association you want loaded in your
results.
Eager loading helps avoid many of the potential performance problems
surrounding lazy-loading in an ORM. The queries generated by eager loading can
better leverage joins, allowing more efficient queries to be made. In CakePHP
you define eager loaded associations using the ‘contain’ method:
// In a controller or table method.
// As an option to find()
$query = $articles->find('all', ['contain' => ['Authors', 'Comments']]);
// As a method on the query object
$query = $articles->find('all');
$query->contain(['Authors', 'Comments']);
The above will load the related author and comments for each article in the
result set. You can load nested associations using nested arrays to define the
associations to be loaded:
$query = $articles->find()->contain([
'Authors' => ['Addresses'], 'Comments' => ['Authors']
]);
Alternatively, you can express nested associations using the dot notation:
$query = $articles->find()->contain([
'Authors.Addresses',
'Comments.Authors'
]);
You can eager load associations as deep as you like:
$query = $products->find()->contain([
'Shops.Cities.Countries',
'Shops.Managers'
]);
You can select fields from all associations with multiple easy contain()
statements:
$query = $this->find()->select([
'Realestates.id',
'Realestates.title',
'Realestates.description'
])
->contain([
'RealestateAttributes' => [
'Attributes' => [
'fields' => [
// Aliased fields in contain() must include
// the model prefix to be mapped correctly.
'Attributes__name' => 'attr_name'
]
]
]
])
->contain([
'RealestateAttributes' => [
'fields' => [
'RealestateAttributes.realestate_id',
'RealestateAttributes.value'
]
]
])
->where($condition);
If you need to reset the containments on a query you can set the second argument
to true
:
$query = $articles->find();
$query->contain(['Authors', 'Comments'], true);
Passing Conditions to Contain
When using contain()
you are able to restrict the data returned by the
associations and filter them by conditions:
// In a controller or table method.
$query = $articles->find()->contain('Comments', function ($q) {
return $q
->select(['body', 'author_id'])
->where(['Comments.approved' => true]);
});
This also works for pagination at the Controller level:
$this->paginate['contain'] = [
'Comments' => function (\Cake\ORM\Query $query) {
return $query->select(['body', 'author_id'])
->where(['Comments.approved' => true]);
}
];
Note
When you limit the fields that are fetched from an association, you must
ensure that the foreign key columns are selected. Failing to select foreign
key fields will cause associated data to not be present in the final result.
It is also possible to restrict deeply-nested associations using the dot
notation:
$query = $articles->find()->contain([
'Comments',
'Authors.Profiles' => function ($q) {
return $q->where(['Profiles.is_published' => true]);
}
]);
In the above example, you’ll still get authors even if they don’t have
a published profile. To only get authors with a published profile use
matching(). If you have defined custom
finders in your associations, you can use them inside contain()
:
// Bring all articles, but only bring the comments that are approved and
// popular.
$query = $articles->find()->contain('Comments', function ($q) {
return $q->find('approved')->find('popular');
});
Note
For BelongsTo
and HasOne
associations only the where
and
select
clauses are used when loading the associated records. For the
rest of the association types you can use every clause that the query object
provides.
If you need full control over the query that is generated, you can tell contain()
to not append the foreignKey
constraints to the generated query. In that
case you should use an array passing foreignKey
and queryBuilder
:
$query = $articles->find()->contain([
'Authors' => [
'foreignKey' => false,
'queryBuilder' => function ($q) {
return $q->where(...); // Full conditions for filtering
}
]
]);
If you have limited the fields you are loading with select()
but also want to
load fields off of contained associations, you can pass the association object
to select()
:
// Select id & title from articles, but all fields off of Users.
$query = $articles->find()
->select(['id', 'title'])
->select($articles->Users)
->contain(['Users']);
Alternatively, if you have multiple associations, you can use enableAutoFields()
:
// Select id & title from articles, but all fields off of Users, Comments
// and Tags.
$query->select(['id', 'title'])
->contain(['Comments', 'Tags'])
->enableAutoFields(true) // Prior to 3.4.0 use autoFields(true)
->contain(['Users' => function($q) {
return $q->autoFields(true);
}]);
New in version 3.1: Selecting columns via an association object was added in 3.1
Sorting Contained Associations
When loading HasMany and BelongsToMany associations, you can use the sort
option to sort the data in those associations:
$query->contain([
'Comments' => [
'sort' => ['Comments.created' => 'DESC']
]
]);
Filtering by Associated Data Via Matching And Joins
A fairly common query case with associations is finding records ‘matching’
specific associated data. For example if you have ‘Articles belongsToMany Tags’
you will probably want to find Articles that have the CakePHP tag. This is
extremely simple to do with the ORM in CakePHP:
// In a controller or table method.
$query = $articles->find();
$query->matching('Tags', function ($q) {
return $q->where(['Tags.name' => 'CakePHP']);
});
You can apply this strategy to HasMany associations as well. For example if
‘Authors HasMany Articles’, you could find all the authors with recently
published articles using the following:
$query = $authors->find();
$query->matching('Articles', function ($q) {
return $q->where(['Articles.created >=' => new DateTime('-10 days')]);
});
Filtering by deep associations is surprisingly easy, and the syntax should be
already familiar to you:
// In a controller or table method.
$query = $products->find()->matching(
'Shops.Cities.Countries', function ($q) {
return $q->where(['Countries.name' => 'Japan']);
}
);
// Bring unique articles that were commented by 'markstory' using passed variable
// Dotted matching paths should be used over nested matching() calls
$username = 'markstory';
$query = $articles->find()->matching('Comments.Users', function ($q) use ($username) {
return $q->where(['username' => $username]);
});
Note
As this function will create an INNER JOIN
, you might want to consider
calling distinct
on the find query as you might get duplicate rows if
your conditions don’t exclude them already. This might be the case, for
example, when the same users comments more than once on a single article.
The data from the association that is ‘matched’ will be available on the
_matchingData
property of entities. If you both match and contain the same
association, you can expect to get both the _matchingData
and standard
association properties in your results.
Using innerJoinWith
Using the matching()
function, as we saw already, will create an INNER
JOIN
with the specified association and will also load the fields into the
result set.
There may be cases where you want to use matching()
but are not interested
in loading the fields into the result set. For this purpose, you can use
innerJoinWith()
:
$query = $articles->find();
$query->innerJoinWith('Tags', function ($q) {
return $q->where(['Tags.name' => 'CakePHP']);
});
The innerJoinWith()
method works the same as matching()
, that
means that you can use dot notation to join deeply nested
associations:
$query = $products->find()->innerJoinWith(
'Shops.Cities.Countries', function ($q) {
return $q->where(['Countries.name' => 'Japan']);
}
);
Again, the only difference is that no additional columns will be added to the
result set, and no _matchingData
property will be set.
New in version 3.1: Query::innerJoinWith() was added in 3.1
Using notMatching
The opposite of matching()
is notMatching()
. This function will change
the query so that it filters results that have no relation to the specified
association:
// In a controller or table method.
$query = $articlesTable
->find()
->notMatching('Tags', function ($q) {
return $q->where(['Tags.name' => 'boring']);
});
The above example will find all articles that were not tagged with the word
boring
. You can apply this method to HasMany associations as well. You could,
for example, find all the authors with no published articles in the last 10
days:
$query = $authorsTable
->find()
->notMatching('Articles', function ($q) {
return $q->where(['Articles.created >=' => new \DateTime('-10 days')]);
});
It is also possible to use this method for filtering out records not matching
deep associations. For example, you could find articles that have not been
commented on by a certain user:
$query = $articlesTable
->find()
->notMatching('Comments.Users', function ($q) {
return $q->where(['username' => 'jose']);
});
Since articles with no comments at all also satisfy the condition above, you may
want to combine matching()
and notMatching()
in the same query. The
following example will find articles having at least one comment, but not
commented by a certain user:
$query = $articlesTable
->find()
->notMatching('Comments.Users', function ($q) {
return $q->where(['username' => 'jose']);
})
->matching('Comments');
Note
As notMatching()
will create a LEFT JOIN
, you might want to consider
calling distinct
on the find query as you can get duplicate rows
otherwise.
Keep in mind that contrary to the matching()
function, notMatching()
will not add any data to the _matchingData
property in the results.
New in version 3.1: Query::notMatching() was added in 3.1
Using leftJoinWith
On certain occasions you may want to calculate a result based on an association,
without having to load all the records for it. For example, if you wanted to
load the total number of comments an article has along with all the article
data, you can use the leftJoinWith()
function:
$query = $articlesTable->find();
$query->select(['total_comments' => $query->func()->count('Comments.id')])
->leftJoinWith('Comments')
->group(['Articles.id'])
->enableAutoFields(true); // Prior to 3.4.0 use autoFields(true);
The results for the above query will contain the article data and the
total_comments
property for each of them.
leftJoinWith()
can also be used with deeply nested associations. This is
useful, for example, for bringing the count of articles tagged with a certain
word, per author:
$query = $authorsTable
->find()
->select(['total_articles' => $query->func()->count('Articles.id')])
->leftJoinWith('Articles.Tags', function ($q) {
return $q->where(['Tags.name' => 'awesome']);
})
->group(['Authors.id'])
->enableAutoFields(true); // Prior to 3.4.0 use autoFields(true);
This function will not load any columns from the specified associations into the
result set.
New in version 3.1: Query::leftJoinWith() was added in 3.1
Changing Fetching Strategies
As you may know already, belongsTo
and hasOne
associations are loaded
using a JOIN
in the main finder query. While this improves query and
fetching speed and allows for creating more expressive conditions when
retrieving data, this may be a problem when you want to apply certain clauses to
the finder query for the association, such as order()
or limit()
.
For example, if you wanted to get the first comment of an article as an
association:
$articles->hasOne('FirstComment', [
'className' => 'Comments',
'foreignKey' => 'article_id'
]);
In order to correctly fetch the data from this association, we will need to tell
the query to use the select
strategy, since we want order by a particular
column:
$query = $articles->find()->contain([
'FirstComment' => [
'strategy' => 'select',
'queryBuilder' => function ($q) {
return $q->order(['FirstComment.created' =>'ASC'])->limit(1);
}
]
]);
Dynamically changing the strategy in this way will only apply to a specific
query. If you want to make the strategy change permanent you can do:
$articles->FirstComment->setStrategy('select');
// Prior to 3.4.0
$articles->FirstComment->strategy('select');
Using the select
strategy is also a great way of making associations with
tables in another database, since it would not be possible to fetch records
using joins
.
Fetching With The Subquery Strategy
As your tables grow in size, fetching associations from them can become
slower, especially if you are querying big batches at once. A good way of
optimizing association loading for hasMany
and belongsToMany
associations is by using the subquery
strategy:
$query = $articles->find()->contain([
'Comments' => [
'strategy' => 'subquery',
'queryBuilder' => function ($q) {
return $q->where(['Comments.approved' => true]);
}
]
]);
The result will remain the same as with using the default strategy, but this
can greatly improve the query and fetching time in some databases, in
particular it will allow to fetch big chunks of data at the same time in
databases that limit the amount of bound parameters per query, such as
Microsoft SQL Server.
You can also make the strategy permanent for the association by doing:
$articles->Comments->setStrategy('subquery');
// Prior to 3.4.0
$articles->Comments->strategy('subquery');
Working with Result Sets
Once a query is executed with all()
, you will get an instance of
Cake\ORM\ResultSet
. This object offers powerful ways to manipulate
the resulting data from your queries. Like Query objects, ResultSets are
a Collection and you can use any collection
method on ResultSet objects.
Result set objects will lazily load rows from the underlying prepared statement.
By default results will be buffered in memory allowing you to iterate a result
set multiple times, or cache and iterate the results. If you need work with
a data set that does not fit into memory you can disable buffering on the query
to stream results:
$query->enableBufferResults(false);
// Prior to 3.4.0
$query->bufferResults(false);
Turning buffering off has a few caveats:
- You will not be able to iterate a result set more than once.
- You will also not be able to iterate & cache the results.
- Buffering cannot be disabled for queries that eager load hasMany or
belongsToMany associations, as these association types require eagerly
loading all results so that dependent queries can be generated.
Warning
Streaming results will still allocate memory for the entire results when
using PostgreSQL and SQL Server. This is due to limitations in PDO.
Result sets allow you to cache/serialize or JSON encode results for API
results:
// In a controller or table method.
$results = $query->all();
// Serialized
$serialized = serialize($results);
// Json
$json = json_encode($results);
Both serializing and JSON encoding result sets work as you would expect. The
serialized data can be unserialized into a working result set. Converting to
JSON respects hidden & virtual field settings on all entity objects
within a result set.
In addition to making serialization easy, result sets are a ‘Collection’ object and
support the same methods that collection objects
do. For example, you can extract a list of unique tags on a collection of
articles by running:
// In a controller or table method.
$articles = TableRegistry::get('Articles');
$query = $articles->find()->contain(['Tags']);
$reducer = function ($output, $value) {
if (!in_array($value, $output)) {
$output[] = $value;
}
return $output;
};
$uniqueTags = $query->all()
->extract('tags.name')
->reduce($reducer, []);
Some other examples of the collection methods being used with result sets are:
// Filter the rows by a calculated property
$filtered = $results->filter(function ($row) {
return $row->is_recent;
});
// Create an associative array from result properties
$articles = TableRegistry::get('Articles');
$results = $articles->find()->contain(['Authors'])->all();
$authorList = $results->combine('id', 'author.name');
The Collections chapter has more detail on what can be
done with result sets using the collections features. The Adding Calculated Fields
section show how you can add calculated fields, or replace the result set.
Getting the First & Last Record From a ResultSet
You can use the first()
and last()
methods to get the respective records
from a result set:
$result = $articles->find('all')->all();
// Get the first and/or last result.
$row = $result->first();
$row = $result->last();
Getting an Arbitrary Index From a ResultSet
You can use skip()
and first()
to get an arbitrary record from
a ResultSet:
$result = $articles->find('all')->all();
// Get the 5th record
$row = $result->skip(4)->first();
Checking if a Query or ResultSet is Empty
You can use the isEmpty()
method on a Query or ResultSet object to see if it
has any rows in it. Calling isEmpty()
on a Query object will evaluate the
query:
// Check a query.
$query->isEmpty();
// Check results
$results = $query->all();
$results->isEmpty();
Loading Additional Associations
Once you’ve created a result set, you may need to load
additional associations. This is the perfect time to lazily eager load data. You
can load additional associations using loadInto()
:
$articles = $this->Articles->find()->all();
$withMore = $this->Articles->loadInto($articles, ['Comments', 'Users']);
You can eager load additional data into a single entity, or a collection of
entities.
Modifying Results with Map/Reduce
More often than not, find operations require post-processing the data that is
found in the database. While entities’ getter methods can take care of most of
the virtual property generation or special data formatting, sometimes you
need to change the data structure in a more fundamental way.
For those cases, the Query
object offers the mapReduce()
method, which
is a way of processing results once they are fetched from the database.
A common example of changing the data structure is grouping results together
based on certain conditions. For this task we can use the mapReduce()
function. We need two callable functions the $mapper
and the $reducer
.
The $mapper
callable receives the current result from the database as first
argument, the iteration key as second argument and finally it receives an
instance of the MapReduce
routine it is running:
$mapper = function ($article, $key, $mapReduce) {
$status = 'published';
if ($article->isDraft() || $article->isInReview()) {
$status = 'unpublished';
}
$mapReduce->emitIntermediate($article, $status);
};
In the above example $mapper
is calculating the status of an article, either
published or unpublished, then it calls emitIntermediate()
on the
MapReduce
instance. This method stores the article in the list of articles
labelled as either published or unpublished.
The next step in the map-reduce process is to consolidate the final results. For
each status created in the mapper, the $reducer
function will be called so
you can do any extra processing. This function will receive the list of articles
in a particular “bucket” as the first parameter, the name of the “bucket” it
needs to process as the second parameter, and again, as in the mapper()
function, the instance of the MapReduce
routine as the third parameter. In
our example, we did not have to do any extra processing, so we just emit()
the final results:
$reducer = function ($articles, $status, $mapReduce) {
$mapReduce->emit($articles, $status);
};
Finally, we can put these two functions together to do the grouping:
$articlesByStatus = $articles->find()
->where(['author_id' => 1])
->mapReduce($mapper, $reducer);
foreach ($articlesByStatus as $status => $articles) {
echo sprintf("There are %d %s articles", count($articles), $status);
}
The above will ouput the following lines:
There are 4 published articles
There are 5 unpublished articles
Of course, this is a simplistic example that could actually be solved in another
way without the help of a map-reduce process. Now, let’s take a look at another
example in which the reducer function will be needed to do something more than
just emitting the results.
Calculating the most commonly mentioned words, where the articles contain
information about CakePHP, as usual we need a mapper function:
$mapper = function ($article, $key, $mapReduce) {
if (stripos($article['body'], 'cakephp') === false) {
return;
}
$words = array_map('strtolower', explode(' ', $article['body']));
foreach ($words as $word) {
$mapReduce->emitIntermediate($article['id'], $word);
}
};
It first checks for whether the “cakephp” word is in the article’s body, and
then breaks the body into individual words. Each word will create its own
bucket
where each article id will be stored. Now let’s reduce our results to
only extract the count:
$reducer = function ($occurrences, $word, $mapReduce) {
$mapReduce->emit(count($occurrences), $word);
}
Finally, we put everything together:
$wordCount = $articles->find()
->where(['published' => true])
->andWhere(['published_date >=' => new DateTime('2014-01-01')])
->enableHydrate(false) // Prior to 3.4.0 use hydrate(false)
->mapReduce($mapper, $reducer)
->toArray();
This could return a very large array if we don’t clean stop words, but it could
look something like this:
[
'cakephp' => 100,
'awesome' => 39,
'impressive' => 57,
'outstanding' => 10,
'mind-blowing' => 83
]
One last example and you will be a map-reduce expert. Imagine you have
a friends
table and you want to find “fake friends” in our database, or
better said, people who do not follow each other. Let’s start with our
mapper()
function:
$mapper = function ($rel, $key, $mr) {
$mr->emitIntermediate($rel['target_user_id'], $rel['source_user_id']);
$mr->emitIntermediate(-$rel['source_user_id'], $rel['target_user_id']);
};
The intermediate array will be like the following:
[
1 => [2, 3, 4, 5, -3, -5],
2 => [-1],
3 => [-1, 1, 6],
4 => [-1],
5 => [-1, 1],
6 => [-3],
...
]
Positive numbers mean that a user, indicated with the first-level key, is
following them, and negative numbers mean that the user is followed by them.
Now it’s time to reduce it. For each call to the reducer, it will receive a list
of followers per user:
$reducer = function ($friends, $user, $mr) {
$fakeFriends = [];
foreach ($friends as $friend) {
if ($friend > 0 && !in_array(-$friend, $friends)) {
$fakeFriends[] = $friend;
}
}
if ($fakeFriends) {
$mr->emit($fakeFriends, $user);
}
};
And we supply our functions to a query:
$fakeFriends = $friends->find()
->enableHydrate(false) // Prior to 3.4.0 use hydrate(false)
->mapReduce($mapper, $reducer)
->toArray();
This would return an array similar to this:
[
1 => [2, 4],
3 => [6]
...
]
The resulting array means, for example, that user with id 1
follows users
2
and 4
, but those do not follow 1
back.
Stacking Multiple Operations
Using mapReduce in a query will not execute it immediately. The operation will
be registered to be run as soon as the first result is attempted to be fetched.
This allows you to keep chaining additional methods and filters to the query
even after adding a map-reduce routine:
$query = $articles->find()
->where(['published' => true])
->mapReduce($mapper, $reducer);
// At a later point in your app:
$query->where(['created >=' => new DateTime('1 day ago')]);
This is particularly useful for building custom finder methods as described in the
Custom Finder Methods section:
public function findPublished(Query $query, array $options)
{
return $query->where(['published' => true]);
}
public function findRecent(Query $query, array $options)
{
return $query->where(['created >=' => new DateTime('1 day ago')]);
}
public function findCommonWords(Query $query, array $options)
{
// Same as in the common words example in the previous section
$mapper = ...;
$reducer = ...;
return $query->mapReduce($mapper, $reducer);
}
$commonWords = $articles
->find('commonWords')
->find('published')
->find('recent');
Moreover, it is also possible to stack more than one mapReduce
operation for
a single query. For example, if we wanted to have the most commonly used words
for articles, but then filter it to only return words that were mentioned more
than 20 times across all articles:
$mapper = function ($count, $word, $mr) {
if ($count > 20) {
$mr->emit($count, $word);
}
};
$articles->find('commonWords')->mapReduce($mapper);
Removing All Stacked Map-reduce Operations
Under some circumstances you may want to modify a Query
object so that no
mapReduce
operations are executed at all. This can be done by
calling the method with both parameters as null and the third parameter
(overwrite) as true
:
$query->mapReduce(null, null, true);


amino CoV acid Fig. -nCoV protein spike selection evolutionary OpenUrl SARS-related sites coronavirus branch COVID--related branches convergent shared used model phylogenetic gene CrossRef ancestral convergence evolution sequence SARS-CoV genes identical positive &; RBD high sequences new Table leading parallel receptor bat receptor-binding substitutions using Science View nearly phylogeny results showed studies substitution Darwinian Download adaptive analyses respiratory CoV, acids acute completely distinct molecular novel possible suggesting test Abstract CoV. Open Web based data-hide-link-title=""> data-icon-position="" different figure full-length likelihood related selected shows strains suggest syndrome tab tree used. Biology Li, ORFa analyses, branch) examine human intensity intermediates. observed severe shown ACE FREE Molecular S). SARS-like Text alignment convergence. coronaviruses phenotypic protein, reference shown. significant strong ;<;div Evolution JTT RBD. S), apparently branch-site compared domain genome intensified located sister statistical strains, study value -nCoV, Given ORFa, PAML Poisson RaTG. SARS Virology Wang, analysis approach assumes cell determine focal foreground https://doi.org/./... node previous protein. result signals similar similarity structure supports this, unusually <;em>;S<;/em>; (KY), (bold) (reference (test COVID- National Nature PSGs Parallel Proceedings RBM RBM, RELAX SARS-CoV, Sciences Selection Wu, Zhang, addition annotated basis bioRxiv, branches, capable categories common coronavirus; coronaviruses. follows gene. homologous host humans incidence independent inline likely model. neutrality noted occurred origin popup published receptor, relative relaxed selection. studied subgenus subunit table:


These release notes cover new developer and user-facing incompatibilities, features, and major improvements.
WARNING: No release note provided for this incompatible change.
The trash emptier may no longer be run using "hadoop org.apache.hadoop.fs.Trash". The trash emptier runs on the NameNode (if configured). Old trash checkpoints may be deleted using "hadoop fs -expunge".
distcp skips CRC on 0 byte files.
If fs.trash.interval is configured on the server then the client's value for this configuration is ignored.
FsShell's "mkdir" no longer implicitly creates all non-existent parent directories. The command adopts the posix compliant behavior of requiring the "-p" flag to auto-create parent directories.
Merged the change to branch-2
WARNING: No release note provided for this incompatible change.
WARNING: No release note provided for this incompatible change.
WARNING: No release note provided for this incompatible change.
Resolve sporadic distcp issue due to having two DistCp classes (v1 & v2) in the classpath.
Improved excpetion handling of shutting down web server. (Devaraj K via Eric Yang)
This is an incompatible change: Before this change, if a file is already open for write by one client, and another client calls fs.create() with overwrite=true, an AlreadyBeingCreatedException is thrown. After this change, the file will be deleted and the new file will be created successfully.
The datanode now performs 4MB readahead by default when reading data from its disks, if the native libraries are present. This has been shown to improve performance in many workloads. The feature may be disabled by setting dfs.datanode.readahead.bytes to "0".
WARNING: No release note provided for this incompatible change.
getBlockLocations(), and hence open() for read, will now throw SafeModeException if the NameNode is still in safe mode and there are no replicas reported yet for one of the blocks in the file.
Add a utility method HdfsUtils.isHealthy(uri) for checking if the given HDFS is healthy.
This change adds two new configuration parameters.
Please see hdfs-default.xml for detailed description.
HDFS no longer silently ignores missing or unreadable host files specified by dfs.hosts or dfs.hosts.exclude. In order to specify that no hosts should be included or excluded, administrators should either refrain from setting the relevant config properties, or create an empty file in order to represent an empty list.
WARNING: No release note provided for this incompatible change.
libhdfs is enhanced to read directly into user-supplied buffers when possible, reducing the number of memory copies.
Introduced a new command, "hdfs dfsadmin -rollEdits" which requests that the active NameNode roll its edit log. This can be useful for administrators manually backing up log segments.
libhdfs now uses the server block size configuration rather than the deprecated dfs.block.size client configuration.
This jira removes functionality that has not been used/applicable since release 0.17. The incompatibility introduced by this change will not affect any HDFS users.
Due to the requirement that KSSL use weak encryption types for Kerberos tickets, HTTP authentication to the NameNode will now use SPNEGO by default. This will require users of previous branch-1 releases with security enabled to modify their configurations and create new Kerberos principals in order to use SPNEGO. The old behavior of using KSSL can optionally be enabled by setting the configuration option "hadoop.security.use-weak-http-crypto" to "true".
WARNING: No release note provided for this incompatible change.
WARNING: No release note provided for this incompatible change.
-Djava.library.path in mapred.child.java.opts can cause issues with native libraries. LD_LIBRARY_PATH through mapred.child.env should be used instead.
The Job Summary log may contain commas in values that are escaped by a '' character. This was true before, but is more likely to be exposed now.
ContainerTokens now have an expiry interval so that stale tokens cannot be used for launching containers.
Fixed NodeManagers' decommissioning at RM to accept IP addresses also.
Removes two sets of previously available config properties:
In favor of two new, generically named properties:
Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides.
Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
Note that to apply this you should first run the script - ./MAPREDUCE-3543v3.sh svn, then apply the patch.
If this is merged to more then trunk, the version inside of hadoop-tools/hadoop-gridmix/pom.xml will need to be udpated accordingly.
Fixed a bug in MR client to redirect to JobHistoryServer correctly when RM forgets the app.


{% extends "base.html" %} {% block title %}
Make sure that all words are spelled correctly.Or use different keywords.


Toggle navigation
{{host}}
/
{{pathname}}
Your History
{{item.id}}
{{item.expires | timeRemaining}} left
Nothing to see here. Make a new geobin!
Create a New Geobin


Week by Week
Class Syllabus¶
Prerequisites - Suggested Reading/Viewing¶
Pangeo, COGS and maybe Stac¶
The Future of Data-driven Discovery in the Cloud - Ryan Abernathey (Columbia University)
https://www.youtube.com/watch?v=7kDYfUe0Zhw
The above youtube is worth watching - 14 minutes
COGs and ARD¶
https://medium.com/planet-stories/cloud-native-geospatial-part-2-the-cloud-optimized-geotiff-6b3f15c696ed
https://medium.com/planet-stories/towards-on-demand-analysis-ready-data-f94d6eb226fc
Week by Week Class Summaries¶
WEEK ONE¶
Week One Objectives¶
FEEDBACK - send me direct chats with pithy comments that might improve our mission
ESIP - don’t hold back science - Rich Signell
Discuss the goals for this class - success criteria - move one more science project to the cloud
Comments and viewpoint of Neal Pastick
Demos
I will move some of these to Neal’s interim web servers
Discuss the Student cross section
http://<IP_ADDRESS>:57778/00define_ET.html#students
http://<IP_ADDRESS>:57778/00define_ET.html#student-individual-goals
Discuss two Case Studies - Commonalities and Not
Evapo
permaFrost
Contact Tony Butzer for cloud questions
Discuss notebook love hate meter - any notebook shy/sensitive folks?
its just a tool; “you’re a tool tony”
There will be lots and lots of notebooks in this class
Point to the two computers - field any questions
web http://<IP_ADDRESS>:57778 hweb
mini-pangeo <IP_ADDRESS> harmony
Computer Access Methods
Jupyter via browser and url and
VPN
Terminal inside Jupyter
direct ssh and scp access - public key
ssh<EMAIL_ADDRESS>ssh<EMAIL_ADDRESS>open ports allowed via firewalls
80, 8080 http
443, 8443 https
22 ssh,scp
use of
tmux
for session preservation across network/VPN failures/timeouts - long running jobs
Point to the website - nginx/sphinx driven
Point to the github -
Quick Jupyter Session - jupyter fitting from single jupyter – to self scaling hubs
focus here will be primarily mini-pangeo scope - a group of 13 students
Self-directed study hints
notebook repo - past pangeo training
If you need one on one jupyter training I can arrange an hour with you
github link - needs pruning - notebook repo here:
https://github.com/tonybutzer/notebook/tree/master/00-Tutorial/00-Jupyter-Notebook-Pieschke
Tony can help you find just the right example or create it for you.
We wil use some of these examples in class and the Labs
future todo - feedback based
create very small notebook snippets that illustrate concepts
refine sections of this “paper” and build examples
Project Kickoff and refinement of goals
Is this anything? - trial and error Seinfeld
doing this class is scary must be what its like to do stand-up comedy
this class is
new
sort of a science experiment in itself
PRIOR Experience
linux
docker
AWS
python
jupyter
management/leadership
I want to build a notebook that radar charts the students
Class questionnaire - via email - or maybe a notebook
Communication preference of each student
efficiency and boredom mitigation
perhaps split each 1 hour into - concepts and practice - allow folks to bail at 1/2 way point
perhaps - threads or channels
jupyter thread
linux thread
docker thread
devops thread
python thread
management leadership - steering committee thread for PANGEO - strategy
I want Neal to drive most of the agenda and deep dive talks - at least initially - people can opt out
for example I could write a chapter on costs in AWS -perhaps most would be interested - I know Neal is
Week ONE - DEC 1,2020 Introduction and Basic Cloud Tour
Quick overview of this document
added a glossary
please skim
if you have comments or questions - please use Teams/Chat to contact me
I will likely post some of the converstaions in this document
Neal’s team won a free mini-pangeo
terraform
the littleist jupyter hub - intro
Neal wanted a kanban too/also
SEE DEVOPS WEEKLY
Teams Note¶
Hey Class: - the course website is starting to take shape and is here:
http://<IP_ADDRESS>:57778
we will fill in more color/detail as we go through our CASE Studies.
our companion github repo is here - can’t live in the cloud without github/dockerhub.
https://github.com/tonybutzer/harmony
this is also where our cache of notebooks will live - like this one:
https://github.com/tonybutzer/harmony/blob/main/class/06-Instructor/01-Students.ipynb
https://nbviewer.jupyter.org/github/tonybutzer/harmony/blob/main/class/06-Instructor/01-Students.ipynb
- See you all on tuesday December 1st at 12 NOON
- See you next week for an open lab
- The week after that I will look at Areas of Interest and how they relate to ET modeling
WEEK TWO¶
Preface¶
The cross organization and cross project communication is way more important than the technology lessons¶
working together and sharing talent
moving to the cloud where its natural to share everything
creating libraries and documentation for the common good of the data center
allowing our hard work in science to be broadcasted to the widest possible audience
removing data and people silos wherever and whenever we can
Welcome to the JupyterHub and Pangeo¶
extraordinary week 2
let the fun begin
Outline¶
For Neal’s Project ( Case study one projects journey into the clouds )¶
Acquire a CHS CUSTOM ACCOUNT and a Project VPC
[DONE]
Setup 2 computers - a tiny admin box and jupyterHub micro-pangeo
Project Management steps rapidly developed via a sphinx website - backed up in git
Spin up a jupyter hub for coding and engineering collaboration
layout the data pipline big steps and refine a simple WBS/todo-list to execute.
Spin up a quick Kanboard to track actions and report progress to Neal
Deep Thoughts
by Neal PastickKey takeaways from last week
Motivating words to inspire the troops to forge ahead.
CHS CUSTOM Account status
next steps
Phases for the Data and Processing from:
portal scraping, ftp transfers to buckets
this is mostly a solved problem
this is also easier in the cloud - can be done in parallel – multithreaded
…
SageMaker ML and voila moments
Michelle Anthony - any thoughts or questions about how this disruptive technology might vector your mid-term roadmap?
Matt Rigge - can you talk about your role and your team and your pipeline you are targeting for cloud operation.
Dev Dahal - talk about logging into Jupyter - opening a terminal and cloning a repo and what you hope to use this class for in your real job
Tree Regrowth - When I get my SDSU students (perhaps…)
Anyone else?
We talked about a bunch of stuff in week 1
Anything we need to clarify about the cloud, AWS or CHS?
feedback … or … takeaways?
Example panel based applications
Glaciers
similar to some of our WMS apps with far less backend engineering - scaffolding
Data Viz applets
widgets
graph display libraries
SDSU viz expert team - coming soon
Gradually these classes should become more democratic - with lots of individuals taking the mic.
Review last Week
pangeo is great
cloud is great
public data assets stored in the cloud == game changing paradigm
you MUST start to use cloud, readily available (fingertips), analysis ready remote sensing data assets - or be left behind
L8
Collection02
Sentinel
HLS …bump …bump … YES
Evporation Model Outputs - after they go thru ScienceBase validation - but maybe even raw for evaluation
one computer to rule them all - t3a.2xlarge
8 CPU, 32Gig Memory, 200Gig local-ebs-elastic-block-storage - with S3 access
technology zen equation
science_joy = 1 * ec2 + docker * 1e3
A World of Twos¶
two computers
t2.micro
t3a-2xlarge
two infrastructure elements
ec2
docker
two technology companions
jupyter
python
two science - image processing packages; fundamental building blocks.
xarray
pandas
two people - who are “ALL IN” on the Cloud!
tony butzer
Rich Signell
two barriers toward cloud acceptance
jargon
complexity
Special Recognition for Kelcy (AWS CHS Credentials)¶
implications for data movent to and from buckets directly from servers on-prem at EROS
“Hey Tony; can I have an account on AWS onboard mini-pangeo?”¶
“Yep, Yes, Si, qui, Ja”
हो¶
Hō¶
Student Getting Started Instructions¶
Signing into the jupyterhub-mini-pangeo¶
<IP_ADDRESS>
username is ‘[LASTNAME]’
password is
harmony
git repo clone¶
clone our notebook repo
next week we will do a
git pull
for updates
Instructions¶
open a terminal in jupyter¶
- find Home Page - browser tab - or middle click jupyterhub above - tony to demo
- [new] button - terminal
- pwd ; ls;
mkdir opt
cd opt
git clone http://github.com/tonybutzer/notebook
Don’t feel like cloning sheep¶
Just use group login:¶
http://<IP_ADDRESS>
harmony
harmony
Winner Winner Chicken Dinner¶
that is all I wanted to get accomplished
Everything else this week is a bonus!
One on one sessions and jupyter tours available¶
just open a chat with tony butzer for custom notebook tour relevant to your technology interests
If I can Help you to the cloud - just ask¶
its free free free - I am funded by OpenDataCube
Rules of Behaviour and IT Security Training¶
its a priviledge to work in a sponsored cloud
Be Responsible not malicious
Save the Planet with Science - Please!
oh and if computer or language things break - use the TEAMS-CHAT - to tonybutzer
How to cheat at Markdown¶
Markdown is another zen item in the technology universe.
below are two simple tables in markdown
and a great editor for learning markdown in 10 minutes
and as always all you need is a browser
We don’t need no education
We don’t need no thought control
No dark sarcasm in the classroom
Teachers leave them kids alone… – Pink Floyd –
| sign | poet-line |
| ------:| -----------:|
| > | When the Star Belly Sneetches had frankfurter roasts
| > | Or picnics or parties or marshmallow toasts,
| > | They never invited the Plain-Belly Sneetches
| > | They left them out cold, in the dark of the beaches.
| > | They kept them away.
| > | Never let them come near.
| > | And that’s how they treated them year after year.
| > | -- Dr. Seuss --
| sign | poet-line |
| ------| -----------|
| > | When the Star Belly Sneetches had frankfurter roasts
| > | Or picnics or parties or marshmallow toasts,
| > | They never invited the Plain-Belly Sneetches
| > | They left them out cold, in the dark of the beaches.
| > | They kept them away.
| > | Never let them come near.
| > | And that’s how they treated them year after year.
| > | -- Dr. Seuss --
https://jonschlinkert.github.io/remarkable/demo/ – [Clear] -¶
try the above text in the remarkable wysiwig tool
Science should be done with pandas and xarray¶
XARRAY¶
I’m doing science with
xarray
xarray is like the Chicago airport - its not your final destination - but you can get anywhere from there
everything else in python is just glue
Notebook Drive By¶
Notebooks are:¶
like storyboards for the real movie
teaching vehicle to illustrate code and its outputs
tool for rapid application development and collaboration
the on-ramp to exciting panel applications
a way to quickly teleport to your cloud computer with just a browser - even from the beach
Xarray Basic - hello xarray¶
A word on public data assets¶
public - that sounds nice
planet really pushed this pangeo world - thanks planet
Notebook illustrates Pangeo, Xarray, Public Data Assets and Panda¶
this notebook was featured by tony butzer at AR19
the point is that when we move the Scientist to the data
everyone wins because so much more science and collaboration can be done
Can xarray, pangeo and python help automate change detection pipelines for tree regrowth for Dr. Zhou?
Jupyter Roles/Features/Capabilities¶
text editor
python coding and exploring
teaching/collaborating tool - commands and output in the same place - writes text books as you go
file transfer agent - upload and download files
web app development accelerator - panel package
teleports people with a simple browser right inside their free computer
AWS onboarding enabler - view the cloud - see the grand canyon without getting hurt
terminal and bash access
poor man’s powerpoint tool
and for
Science
- well that is what jup was born for - for those that realize it anyway… and so much more … “A lot of people say this Jupyter thing is going to be huge – its fantastic! – who knew?”
WEEK 3 - Meet Logan¶
Logan’s assignment:¶
Data Wrangling –> Portal Scraping¶
Logan needs to download Neal’s invasive AOI input data
This week he will load rangeland cover v2.0
Band 1 –> annual forb and grass
forb - herbaceous flowering plants
Fortunately Neal only needs Band 1 from the image and a small subset windowed
else its half of CONUS and 136 Gigabytes per file
Also fortunately the web site has a Readme file with an example
gdal_translate
command
Crowd Source¶
http://<IP_ADDRESS>/user/harmony/tree/opt/logan/00-notebooks/00-portal-scraping/00-MRLC-Notebooks/IMG
Logan Writes¶
[10:13 AM] Megard, Logan J Looks like to convert the .img you can either use [gdal_translate -of GTiff Test.img New.tif] or [rio convert Test.img rio.tif]
Matt Rigge is correct unzipping is an extra burden
Also did I mention proprietary data formats are an extra pain
we don’t like vendor lockin strategies in an Open Systems World
Review Prior Week(s)¶
You should work in the cloud¶
A CHS CUSTOM account is useful
Jupyter and Python are nice
Tony’s role is to guide you up the learning curve
and then turn you loose to use science to save the planet
Summary¶
This week we demonstrate how easy it is to work in the cloud
by using geojson to quickly display extents:
Our huge source files that are 1/2 of CONUS
Our semi-big general Area of Interest (AOI)
Our smaller school/class size imagery to be stuffed in an Xarray - what else?
We also explore command line tools for¶
gotta learn to love the cli if you want to be a power user - of just about any tool
GDAL window slurping accross the net
rio for getting the extent bounds into a geojson
aws s3 ls (plus cp, mv, rm, cat) for seeing things in our s3 buckets
Logan notices the
s3 commands
are a direct corollary/map to theLINUX commands
for files
vsicurl and vsis3 and s3://
We will also build on things we learned in week one and week two¶
Finally; we will explore a bucket of portal scraped inputs¶
aws s3 ls s3://dev-et-data/logan/in/raw/
Since this class is about case studies¶
This week lets switch to following our Invasive Machine-Learning (SageMaker) AWS Project
Neal is working with CHS to get his cloud space started up with a very small server
Then Logan will create and destroy on-demand computers at will for Neal and the Team
Quick a word about rio¶
rio is the command line tool - swiss army knife for data wranglers
like gdal for dummies; build on rasterio/gdal
! rio –help
Usage: rio [OPTIONS] COMMAND [ARGS]...
Rasterio command line interface.
Options:
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
--aws-profile TEXT Select a profile from the AWS credentials file
--aws-no-sign-requests Make requests anonymously
--aws-requester-pays Requester pays data transfer costs
--version Show the version and exit.
--gdal-version
--help Show this message and exit.
Commands:
blocks Write dataset blocks as GeoJSON features.
bounds Write bounding boxes to stdout as GeoJSON.
calc Raster data calculator.
clip Clip a raster to given bounds.
convert Copy and convert raster dataset.
edit-info Edit dataset metadata.
env Print information about the Rasterio environment.
gcps Print ground control points as GeoJSON.
info Print information about a data file.
insp Open a data file and start an interpreter.
mask Mask in raster using features.
merge Merge a stack of raster datasets.
overview Construct overviews in an existing dataset.
rasterize Rasterize features.
rm Delete a dataset.
sample Sample a dataset.
shapes Write shapes extracted from bands or masks.
stack Stack a number of bands into a multiband dataset.
transform Transform coordinates.
warp Warp a raster dataset.
Objectives -¶
Answer some of the staffing needs in the cloud.
The pace of technology means skill workers need to constantly tune-up their skills.
This means a steady diet of self training via
Google
Youtube
and classes like this one :-)
Look At Data Wrangling
Current State
Future State
Cool Notebook of the week and WHY its cool!
Logan is a Senior GIS DevOps Dude¶
Logan works on science projects in the cloud
currently invasives and then permafrost
Logan is a full-stack engineer - still honing his skills but he has a high aptitude and a thirst for knowledge
Lets follow Logan with his new AWS account as he sets his project up for success
A day in the life of Logan the data slayer¶
Logan sets up his git repo¶
https://github.com/tonybutzer/logan/tree/main/00-notebooks/00-portal-scraping
logan reviews Neal-Bruce-Dev-Rigge-et-al-half-of-eros-paper-here
Logan’s coolest notebooks¶
Logan creates his new server¶
Selects t3a.xlarge with 400 Gig of disk¶
his t3a
[t3a.xlarge]
machine will cost $141.79 - Logan doubles that for his boss and says $300/monthLogan remembers that the desktop tax they used to charge projects was $4000/year -
Logan also notes this will be for 4-5 science users
Logan wonders if he should bring up reserved instances and decides - “well I’ll have that in my pocket”
Logan ponders on the nuanced complexities of the AWS ec2 service landscape¶
Logan keeps repeating - keep it simple - keep it small - tony’s influence perhaps
Logan thinks - “ok so for a standard-reserved and everything up to on-demand it would be $100 to $111 to $141”
Logan decides it will cost more to meet on all these service model variances than the dollars it saves.
Logan thinks about this decision and again Logan decides …
it will cost more to meet on this than the dollars it saves.
Now that logan has determined his main ec2 computer and calculated the costs its time to spin up the system and bring up some foundation technologies.
Terraform Init & Apply¶
Demo Here?
Picture of Logan bringing up ec2 instances for Invasives
Logan now wants to train his users¶
and the scientists on his team are a tough crowd
logan does not like propietary formats
ERDAS IMAGINE
ESRI Shapefile
logan likes geojson and geotiff files - FOR EVERYTHING
logan can create and download geojson files using this site
Logan borrows some material from USGS Workshops¶
Scientific Software Environment of the Future - Rich Signell¶
USGS:¶
Try: https://pangeo.chs.usgs.gov
Examples: https://code.chs.usgs.gov/pangeo/notebooks
Join: USGS Pangeo Team: GS-Pangeo
Larger Community:¶
Follow: medium.com/pangeo
Discuss: discourse.pangeo.io
Chat: gitter.im/pangeo-data
Logan starts to look at Data Wrangling¶
1st stop portal scraping¶
Logan sighs and wishes everything was just in COGs and used STAC - but you can’t change the science world overnight
Logan shoots Dev Dahal a quick email with links to his favorite public data sets:
GA will compare our Collection02 STAC work against these public data assets:
Then its back to work.
Logan 00-notebooks/00-portal-scraping¶
github¶
Logan notices some of these files are absolutely HUGE
wonders about costs of storing
studies this S3 document - AmazonS3
Logan needs to use these linux commands
ls -lh
du -hd 2 .
df -h .
free -h
lscpu
uptime
top
Logan trys these commands by bringing up a terminal
He thinks his students should play with those commands in the terminal
He also thinks he himself should get familiar with those in case people ask him to explain this Linux stuff
How to display AOI in Jupyter
geojson and shapefile
folium
docker
Why Libraries/packages¶
code reuse is easier
keep notebooks smaller and they flow better for the reader
abstact complex code
installable with
pip
example library
Logan uses Jupyter to “bread-board” his python¶
then he creates python libraries/packages with setuptools and setup.py
Logan wants to show his fellow scientist how this is done
and don’t foget building and orchestrating docker containers - Logan makes a mental note
END OF WEEK 3¶
next week we will look at more xarray code - and Logan’s additional wrangling steps
WEEK 4 - Standing up Your First Instance in the Cloud¶
You really should be doing science in the cloud
¶
LCMAP to move to the cloud and use S3 and Xarray - Jon Morton¶
Data Wrangling Tip of The Week¶
Jennifer Rover and Kelcy Smith
Pixel location –> it varies - especially in proprietary data format headers
know what your pipeline creates - lest you be a propagator of garbage
nudge a pixel if its off center and not alligned with the rest of your data friends
Network tip of the week¶
pushing data to the cloud is faster from EROS OA then thru the VPN - duh!
the <IP_ADDRESS> network is called the Office Automation Network
it has access to the cloud CHS
the <IP_ADDRESS>/16 address space is via the VPN and its considerably slower
Logan pointed out that the speed of then net was faster from his VDI
so we used putty, puttygen, pageant, filezilla and port 22 to xfer files up
after we created another t3a-2xlarge with terraform out of thin air.
Using the AWS Console to Create¶
Introducing Logan Megard by Neal Pastick¶
What’s going on in your project Neal
Only One week with AWS and we already have our instances up
GIS Expert
Emerging DevOps Cloud Engineer
Picture of the AWS Console¶
1. Create ssh-keygen to make an ssh key for your command prompt
Save the ssh key so the key can be added to your instance
so you can log onto it once its added
2. Log onto aws console. Select region (Oregon West),
Go to dashboard and EC2.
create key pairs (Save keypair)
3. Launch instance, free tier only
ubuntu 18
t2 micro
Need a vpc and subnet
storage
tags
Security
review and launch
add key pair (one I already have)
4. connect via .pem location
need to be in file location
launch initial instance
5. set up new keygen
cat authorized_keys or cat ~/.ssh/authorized_keys
vi authorized_key
6. cd /opt
git clone https://github.com/tonybutzer/logan
ls
dir
7. sudo apt update
sudo apt install python3
8. ping <IP_ADDRESS>
history
lscpu
uptime
9. logout
stop instance
terminate instance
Done :)
Logan captures great notes to be used as a cheat sheet¶
Logan heard that Steffi Kagone has some notes as well¶
Names:
ssh -i megard-uswest-keypair.pem<EMAIL_ADDRESS>log into inital instance for Ec2
need the .pem key to log on the first time
cat authorized_keys or cat ~/.ssh/authorized_keys
shows .pem and other ssh keys added
ssh-keygen
creates ssh key for your cmd (keep a copy to add to the server)
ssh-rsa
shows key
vi authorized_key
adds
ssh ubuntu@[ip_address]
log into ip address (Ec2 instance)
uptime
shows time for ec2 instance
lscpu
shows build of ec2 instance (cpu, memory ...)
who
shows who is on the server
w
shows how much cpu and memory is being used and by what
ping <IP_ADDRESS>
shows speed test of server (ctr+d or ctr+c to exit)
logout
leave server (Ec2 instance)
sudo apt update
looks for update for packages
sudo apt install [python3]
installs package
pwd
print working directory
cd
moves into the folder
git clone [http address]
Copies git folder to current directory
ls
shows whats in current directory
dir
shows everything in directory
more [file]
displays a text file stopping at a page of the screen
cat [file]
stands for concatinate and print to screen - same as more but does not use a PAGER
history
prints out full history of commands in session
Custom:
pinstance
- status - shows all ec2 instances in the VPC #eco in this case.
- start - starts an instance - spins it up
- stop - stops an instance - spins it down - #AWS does not charge money for spun down instances
pin - bash alias for pinstance status | grep -A1 megard
PANGEO MISSION STATEMENT¶
Our mission is to cultivate an ecosystem in which the next generation of open-source analysis tools for ocean, atmosphere and climate science can be developed, distributed, and sustained. These tools must be scalable in order to meet the current and future challenges of big data, and these solutions should leverage the existing expertise outside of the geoscience community.
Stuart Udahl¶
Plans to protect air and water, wilderness and wildlife are in fact plans to protect man.
END of WEEK 4 - Happy Holidays¶
Interstitial¶
Examples of common forbs:
poison oak (top left), -
western ragweed (top right),
silverleaf croton (bottom left)
and green briar (bottom right)
WEEK 5 - Data Wrangling is Underway - Understanding S3¶
Happy Holidays¶
looking forward to 2021
thanks for particpating in these cloud activities - this effort is important
Topics covered today - Major Theme - Data in S3¶
Silo Busting
IT Security in the Cloud - strategy/impacts
Pangeo and mini-pangeo when to use
Logan demonstrates - viewing buckets via the AWS Console
Logan’s notebook for examining a COG/nonCOG - xarray
Logan’s methods for scaling/resampling vs. windowing/clipping
Tony - adventures in portal scraping for rapv2 and BITv3 - range-land herbs
Tony - presents notebook of the week - Pandas and Violin Graphs
No silos; No Silos; NO SILOS!¶
lets work together
“many hands make light work”
Good communication with CHS - Eric Driscoll Larson’s team
Thanks Neal
EROS cloud pilot projects are very important toward shaping science
EROS is a WORLD CLASS DATA CENTER - we need to exert our influence
PANGEO, CHS Custom Pilots, CHS Science Data Lake, ESRI Integration
AI/ML
Steffi Kagone - talk about the model we just ran over the weekend
The pontential for using Docker to scale in the cloud
Visualization effort - and outside synergies - Gabe, Darin
Sharing data between Science elements - Terry Sohl, Ward Sanford - Wes Zell - Greg Rouze
how it would be nice to have ESRI tools in the cloud with direct access for s3 bucket data/data lakes
In the Cloud Queue
VegDyn - Trenton, Dinesh
LandFire - Anyone in class from Landfire? - Neal invited as a consultant.
LCMAP
WaterSmart
Logan to train Steffi’s team on VDI meets FileZilla and the cloud EC2 - ssh key managment etc.
Barter system for skill exchange - new staffing models needed to address the pace of change in technology
Understanding COGs and NonCOGs¶
data sizes and overviews
A word about Data Compression¶
Below is a summary of the compression ratios of each method in the worst case:
10-meter Sentinel-2 bands,
internal tiling and overviews.
| method | Compression Ration | Reason to use |
|---|---|---|
| LZW | 1.16:1 | Quick to run and easy to adopt |
| DEFLATE | 1.38:1 | A slower but better performing alternative to LZW |
| ZSTD | 1.45:1 | Promising, CPU intensive but not mature |
Table. Summary of the results of the different compression algorithms
In the cloud you can test all compression techniques¶
and change your mind - its fast and cheap to reprocess to other formats
zarr is an interesting technology as well
Email from my OpenDataCube Tasmanian Friend - Alex Leith¶
Aussie lessons learned will likely influence LCMAP
Hey Tony
And hey Jeff [Briden] and Jon [Morton] of {LCMAP FAME}
Happy to talk about how we do ODC work in the cloud, including doing continental-scale data processing work.
We have some neat new examples using AWS tools like their Simple Queue Service along with Kubernetes to run a whole of Africa pixel quality analysis (so just visiting Sentinel-2’s SCL band, over all pixels over Africa) and we’re able to process that on 5 large EC2 instances in ~20 minutes, which is fun! Costs us $20 USD.
Our land cover folks are doing something similar in Australia with LCCS, and I could bring them along. They're science users, and are happy working with Python and the ODC
along with Kubernetes and job queues to run through that analysis, including iterating quite fast. I’m not sure about the costings of that, but it’s been really successful this year!
So yes, happy to talk some time in the new year. Might need to be late Jan when we’re all back on board!
Cheers,
IT Security and Compliance Part II¶
Logan and Tony - “WE GOT THIS!”¶
IT Security is way easier in the cloud¶
Paul Exter Email
SSM Agent and auto Updates
AmazonSSMManagedInstanceCore IAM Policy attached to ec2 instances
lower server footprint and vulnerability attack surface
reduce user numbers in my Landsat mini-pangeos
migrate users to big pangeo; teach cookbook examples in the Big Pangeo
migrate eco users to the eco VPC mini-pangeo
add encryption where i have to - or decomission less important services
IAM - things you will never, ever, need to know - Whew!¶
but if you do there’s an app for that - I mean a Jupyter Notebook for that!
aws iam list-attached-role-policies –role-name lsds-developer-ec2
Jupyter notebooks simplify every complex concept :-)
Kelcy - where do you author notebooks?¶
Answer: pangeo.cr.usgs.gov
PANGEO manged service - is nice because IT Security is someone elses problem
Where is your S3 bucket?
You are working on a STAC Collection 2 Example - Right?
IT Security Goal¶
Improve IT Security Compliance - with minor impacts to creativity, scaling and science
Keep our eyes on the mission - not the distractions.
The Oregon Pronunciation Conundrum - perhaps we should move to us-east-1¶
because that is in Virginia
I found out the 53% of the folks use the fake-pronunciation for Oregon
I am glad I am in the 12% who actually get it right. :-)
I am thinking about starting a Facebook campaign
organ trail?
or-AGAIN
My friend once told me “You must be the most pedantic person in the entire world.”
“Third most, actually.” – :-)
Logan Data Wrangling, S3 and early validation notebooks¶
Logan demonstrates - viewing buckets via the AWS Console
Logan’s notebook for examining a COG/nonCOG - xarray
Logan’s methods for scaling/resampling vs. windowing/clipping
S3 buckets view from the AWS Console
Quick Sanity Check of NED Data
in0 –> in1 (COGs) Model Ready Data Perhaps
s3://eco-w1/in0/rapv2
; etc ….
Wrangle rapv2 and mrlc_zip¶
next week we will use the cloud to create COGs from these in parallel - via docker orchestration
https://github.com/tonybutzer/logan/blob/main/00-notebooks/00-portal-scraping/00-MRLC-Notebooks/00-data-scraping-grab-zips.ipynb
Featured Notebook in Big Pangeo pangeo.chs.usgs.gov¶
I have an account in pangeo.chs.usgs.gov
this is a nice capability for basic jupyter notebook training
Steve Labahn got a Pangeo account at the click of a button
This is a managed service and there are benefits - since many items are taken care of.
I think you can request a bucket to go with your user ???
You can definitely play with notebooks here
Superheros Notebook¶
http://<IP_ADDRESS>/user/harmony/notebooks/opt/harmony-notebooks/00-pandas/00-demos/02-plotly-express/super_heros_pandas_plotly_espress_demo.ipynb


Win: 0
Loss: 0
This is your heart!! Try to beat like the red heart
<-->
0
- -You will have a heart beats at the start of the game.
- -By clicking on a heart you will add a specific amount of points to your heart.
- -Win the game by matching your heart beats to the given random number.
- -Lose if your number surpasses the given number.
- -The value of each heart is hidden from you until you click on it.
- -Each time the game restarts, the value of each heart will change.


class Illuminate\Redis\Events\CommandExecuted
Construction
Hierarchy
Constants
Properties
-
string $command = null
The Redis command that was executed.
-
array $parameters = null
The array of command parameters.
-
float $time = null
The number of milliseconds it took to execute the command.
-
Illuminate\Redis\Connections\
Connection $connection = nullThe Redis connection instance.
-
string $connectionName = null
The Redis connection name.
Methods
-
__construct ($command, $parameters, $time, $connection) : void
Create a new event instance.
Parameters string $command array $parameters float null $time Illuminate\Redis\Connections\ Connection $connection Return void


|
AceTime
1.0
Date and time classes for Arduino that support timezones from the TZ Database, and a system clock that can synchronize from an NTP server or an RTC chip.
|
The classes provide a thin layer of indirection for accessing the zoneinfo files stored in the zonedb/ and zonedbx/ directories. More...
Go to the source code of this file.
Classes | |
| class | ace_time::basic::ZoneRuleBroker |
| Data broker for accessing ZoneRule. More... | |
| class | ace_time::basic::ZonePolicyBroker |
| Data broker for accessing ZonePolicy. More... | |
| class | ace_time::basic::ZoneEraBroker |
| Data broker for accessing ZoneEra. More... | |
| class | ace_time::basic::ZoneInfoBroker |
| Data broker for accessing ZoneInfo. More... | |
| class | ace_time::basic::ZoneRegistryBroker |
| Data broker for accessing the ZoneRegistry. More... | |
| class | ace_time::extended::ZoneRuleBroker |
| Data broker for accessing ZoneRule. More... | |
| class | ace_time::extended::ZonePolicyBroker |
| Data broker for accessing ZonePolicy. More... | |
| class | ace_time::extended::ZoneEraBroker |
| Data broker for accessing ZoneEra. More... | |
| class | ace_time::extended::ZoneInfoBroker |
| Data broker for accessing ZoneInfo. More... | |
| class | ace_time::extended::ZoneRegistryBroker |
| Data broker for accessing the ZoneRegistry. More... | |
Functions | |
| uint16_t | ace_time::internal::timeCodeToMinutes (uint8_t code, uint8_t modifier) |
| Convert (timeCode, timeModifier) fields in zoneinfo to minutes. More... | |
| uint8_t | ace_time::internal::toSuffix (uint8_t modifier) |
| Extract the 'w', 's' 'u' suffix from the 'modifier' field, so that they can be compared against kSuffixW, kSuffixS and kSuffixU. | |
| int16_t | ace_time::extended::toDeltaMinutes (int8_t deltaCode) |
| int16_t | ace_time::extended::toOffsetMinutes (int8_t offsetCode, int8_t deltaCode) |
The classes provide a thin layer of indirection for accessing the zoneinfo files stored in the zonedb/ and zonedbx/ directories.
When ACE_TIME_USE_PROGMEM or ACE_TIME_USE_PROGMEM are enabled, the zoneinfo files are stored in flash memory (using the PROGMEM keyword), and cannot be accessed directly on microcontrollers using the Harvard architecture (e.g. AVR) where data and program live in 2 different address spaces. The data in flash memory must be accessed using helper routines in <pgmspace.h>. These classes abstract away this difference so that the code BasicZoneProcessor and ExtendedZoneProcessor can be written to be (mostly) agnostic to how the zoneinfo files are stored.
When ACE_TIME_USE_PROGMEM are disabled, the compiler will optimize away this entire abstraction layer, so the resulting machine code is no bigger than (and in most cases, identifical to) accessing the zoneinfo files directly.
The abstraction layer is thin enough that the code in BasicZoneProcessor and ExtendedZoneProcessor did not change very much. It was mostly a mechanical source code replacement of direct zoneinfo access to using these data brokers.
The helper functions live in the internal:: namespace. The classes are somewhat duplicated between the 'basic' and 'extended' namespaces. They used to be identical so that they could be templatized. But supporting one-minute resolution for 'extended' meant that the implementations diverged, so I had to manual duplicate the classes.
Definition in file Brokers.h.


NKN Shell Client
Remember Key Pair
Load Key Pair From Seed
Or
Generate New Key Pair
Secret Seed:
Public Key:
Please make sure this public key is in remote nshd's authorized_pubkeys file!
Clear Key Pair
Remote nshd address
Client identifier (optional)
Remember address and identifier
Connect


About
My interest in mechanical engineering started around the time I learned to walk. When I wasn't at school or in the pool practicing, you could find me in my dad's machine shop building, fixing, and tinkering anything I could get my hands on. Flash forward about 20 years, I continue to do what I love every day as a senior mechanical engineering student at Rose-Hulman Institute of Technology with minors in Computer Science, Mathematics, and Computational Science, further challenging me and guiding me towards a technology driven future.
Throughout my college years, I have enhanced my design and technical skills immensely, specifically working on projects such as a fully functioning 2D Core XY plotter, a fully optimized panel airplane wing design, and a series of safety lifting fixtures for die additions.
From 3 rotations at General Motors working as a Tool and Die Group Leader Intern, Mechanical Engineering Intern, and Facilities Engineering Intern, I have a gained a proficiency in manufacturing processes, root cause analysis, and computer aided engineering skills. I hope to continue to implement these skills to make a lasting impact in the world of engineering. Frequently described as efficient, ambitious, and communicative by my peers, I can be relied upon to help your company achieve its goals.


Headquartered at Mumbai, Maharashtra, India.
Office Address
205, Gulmohar Residency,
Dombivli-E, Maharashtra, India.
Pin code - 421201.
Mail to<EMAIL_ADDRESS>Contact No.: +91 72083 00456
/ +91 83081 44800
CIN: U72900MH2020PTC349767.
GSTIN: 27AAFCI8928R1ZL.
Follow Us at:
FAQ
1. What does ILXS Pvt. Ltd. offer?
Ans. ILXS offer products in the category of Interactive Electronic Technical Manual (IETM), S1000D, CBT and Record Management System(RMS). We are capable of developing software solutions to any customer requirements in the field of Document Management / Product Management / Process Management.
2. What is IETM or Interactive Electronic Technical Manual?
Ans. Interactive Electronic Technical Manual - IETM is categorised into 5 classes depending on the requirement of features. For more details visit products section.
3. What is CBT or Computer Based Training?
Ans. Computer Based Training - all the product description, operation and training are available in application in the form of interactive animated voice-over videos.
4. What is the level of modification available in IETM at customer end?
Ans. Content Modification using word editor and XML or HTML editor which ever form opted with the IETM package. Add, edit, delete options are available for Pages, Links, Images and Videos. Users can manual define n number of hotspot for image.
5. What is login based content access?
Ans. All the products of ILXS feature credential based access to the content.
6. What is the limit for node creation in Index?
Ans. The only thing which limits the amount of data is, the amount of system's capability to process database at normal operation.
7. What is hardware or OS requirement for ILXS products?
Ans. ILXS provides a complete backward compatibility to more than decade old hardware, while maintaining latest technology in application development.
8. What is the level of ease provided in installation?
Ans. All ILXS products are provided in 4 to 8 installation steps, we believe in keeping a simple installation procedure.


This product has been tested on Google Chrome version 74 and Firefox version 66, however it can theoretically be run using any other browser as long as it supports JavaScript. Due to the limited testing this product has undergone, please alert the developers if it does not work on your browser or device so we can improve the product.
For Google Chrome, the minimum hardware and software requirements include running the Windows 7 OS or a later version, the OS X Yosemite version 10.10 or later, and the 64-bit versions of either Ubuntu 14.04 or later, Debian 8 or later, OpenSUSE 13.3 or later, or Fedora Linux 24 or later. Unless you are running a version of the OS X operating system, you will also need to have an Intel Pentium 4 processor or later version (as long as it is SSE2 capable).
No installation is required. This product runs from the web browser from the URL https://adeeb10abbas.github.io/WhereToPoop/
In the event of an error with the website, it is recommended that network settings be checked to confirm an Internet connection and the website restarted by closing the web browser, re-opening the browser, and returning to the website.
In the event an unexpected or undocumented issue occurs, please contact one of the developers below or use one of the "Contact Us" links within the site itself:
Our project, Where to Poop, was created by five Drexel College of Computing and Informatics students to help people find toilets when they need them. Since our tagline is, "Yelp for Toilets," we include ways to rate toilets and to find toilets based on your own preferences and needs. Toilets can easily be added to the map by filling out a simple form, and once your added toilet is ready, we'll alert you that it's on the map.
This project is licensed under the MIT License in order to allow as many people as possible to benefit from our project. That said, the Google Maps API used in this product also has its own terms and conditions which need respecting if you are modifying this project to suit your own needs.
Thank you for using our app and please do not hesitate to reach out.


This is an express project that offers basic RESTful to GET, PUT, POST and DELETE data. resources routing are loaded dynamically. Persistence is done by using fs having file per resource collection
run service by running:
npm run dev
Assuming the service is running on http://localhost:3000 Send a POST request (using an app like Insomnia or Postman) to http://localhost:3000/resources with a body like:
{
"resourceName": "theResourceName",
"identifier": "id",
"idGenType": "uuid"
}
Send a GET request to http://localhost:3000/resources


Familiar user interface similar to the ones you already know.
Graphics powered by three.js with dynamic ligths, shadows, post-processing, etc.
Fully features 3D physics engine powered by cannon.js
Easy to learn language, familiar for web and game developers.
Want 2D? No problem! Switch between 2D and 3D with the click of a button.
Ready for web virtual reality, on mobile and desktop platforms.
Graphical material editor similar to the ones found in 3D design tools.
GPU accelerated particle system powered by SPE.
Drag and drop files into the editor, it just works.
Create 2D animations using Esoteric spine animation tool and import them to nunuStudio.
Audio engine built on top of web audio API with support for positional audio.
nunuStudio was build on top of nwjs it works on windows, linux and OS X.


The SITRA.fi form for Data Sharing federations. https://media.sitra.fi/2020/06/01093003/rulebook-for-a-fair-data-economyversion1-1.pdf
Repo available at - https://github.com/CommonAccord/FI-SITRA-RuleBookForFairData
Form of General Terms available at http://www.commonaccord.org/index.php?action=source&file=G/FI-SITRA-RuleBookForFairData/GeneralTerms/Form/0.md
YouTube video of the conversion process, with special focus on parameterizing Defined Terms, available at https://www.youtube.com/watch?v=n6P-BYnM90s


Your opponent has left the game.
The game will close in {{ decoTimer - timer }} seconds.
You can use any blue tiles.


This example contains a simple second order thermal model of a motor. The periodic power losses are described by table "lossTable":
| time | winding losses | core losses |
| 0 | 100 | 500 |
| 360 | 100 | 500 |
| 360 | 1000 | 500 |
| 600 | 1000 | 500 |
Since constant speed is assumed, the core losses keep constant whereas the winding losses are low for 6 minutes (no-load) and high for 4 minutes (over load).
The winding losses are corrected by (1 + alpha*(T - T_ref)) because the winding's resistance is temperature dependent whereas the core losses are kept constant (alpha = 0).
The power dissipation to the environment is approximated by heat
flow through a thermal conductance between winding and core,
partially storage of the heat in the winding's heat capacity as
well as the core's heat capacity and finally by forced convection
to the environment.
Since constant speed is assumed, the convective conductance keeps
constant.
Using Modelica.Thermal.FluidHeatFlow it would be possible to model
the coolant air flow, too (instead of simple dissipation to a
constant ambient's temperature).
Simulate for 7200 s; plot Twinding.T and Tcore.T.


Peter Staker
Welcome to my page! I'm a technology enthusiast with a background in philosophy. I’ve worked as a Bartender, Event Promoter, Brand Ambassador, Prosecutor Assistant and I’ve helped open two restaurants. Education and lifelong learning are very important to me and over these past few years, I've fully committed to a growth mindset and a never-ending pursuit of knowledge. I completed a web development certificate while I earned my BA from the University of Washington. I was lucky enough to discover my passion for technology, development, and design, which motivated me to take numerous online courses covering a broad range of technologies, tools and concepts. My ultimate goal is to begin my new tech career in an environment conducive to learning, growth, and professional development.


Discussion
Now that all the classes have been created, let's look at a few important key points in the classes and subclasses.
Getters/Setters
You should notice that for every property there is at least a get<Prop> method and some properties also have a set<Prop> method.
This is a good practice as it allows one to add additional needs in the future when manipulating a property. You should never directly set a property, but always use its "getter" or "setter" for better control.
For example, in our Mage class:
... function Mage:constructor(name, isType, health, maxHealth, weapon, mana, maxMana) Mage.super.constructor(self, name, isType, health, maxHealth, weapon) self.mana = mana self.maxMana = maxMana end function Mage:setMana(amount) self.mana = amount end function Mage:getMana() return self.mana end ...
We have a mana
property, but we would always use the getMana
and setMana
methods to manipulate that property. You would never want to to do self.mana = val
or mana = self.mana
in any of your methods.
To refill our mana we use a seperate method called refillMana
:
... function Mage:refillMana(amount) amount = amount or self:getMaxMana() amount = self:getMana() + amount self:setMana(math.min(amount, self:getMaxMana())) end ...
You should notice that we never call the property directly, but use the "getter" and "setter" methods instead.
In the example code above we "clamp" the mana
value to the maxMana
(using math.min
). It's better to do this in a seperate method so we don't pollute the setting of the raw mana
property.
The setMana
method itself will take any value:
... function Mage:setMana(amount) self.mana = amount end ...
This allows us to override the maxMana
if we want to with a "buff" or potion. For example, let's assume we have some "buff" functionality built into our Mage class that adds additional bonus mana when active, we can then easily adjust our refillMana
method:
... function Mage:refillMana(amount) amount = amount or self:getMaxMana() amount = self:getMana() + amount --do we have a buff? if self:hasActiveBuff() then amount = amount + self:getBuffValue() self:setMana(amount) --no clamping here else self:setMana(math.min(amount, self:getMaxMana())) end end ...
You could put the refillMana
functionality directly in the setMana
method, but it is better to abstract the functionality out, keeping the property "getter" and "setter" clean.
Subclass Notes
All subclass are created using the extends
method on the class or subclass we want to extend. We do not use Classy.create
, but we need to pull in the base class using require
so that we can subclass it.
... --The base class (or subclass) local Character = require("classes.Character") --Create a subclass by "extending" local Warrior = Character:extends("Warrior") ...
When creating a subclass, we need to make sure to pass up the required base class arguments by using the constructor
function. This done using the super property:
... function Warrior:constructor(name, isType, health, maxHealth, weapon) --Pass the arguments up the hierarchy using "super" Warrior.super.constructor(self, name, isType, health, maxHealth, weapon) end ...
Pay attention to the signature of the super.constructor
function. You need to make sure you pass a reference of self
to it, along with any additional arguments, and use only dot (.) syntax:
<Class>.super.constructor(self[, args])
This is very important to remember, or your subclasses will not work as expected..
In the case of subclasses that have their own specific properties, we only need to pass the ones the base class needs, and we then store the others on the subclass itself:
... function Mage:constructor(name, isType, health, maxHealth, weapon, mana, maxMana) --Pass the base class arguments up the hierarchy using "super" Mage.super.constructor(self, name, isType, health, maxHealth, weapon) --Store properties specific to this subclass self.mana = mana self.maxMana = maxMana end ...


June 2010 - Santiago, Chile - Research commissioned by Nielsen.
Clorox is a brand inside the Clorox Company with products for cleaning and disinfecting, toilet and bathroom and for doing laundry.
Nielsen was hired by Clorox Chile to understand how the brand is performing in the category stain remover for clothes.
The goal of the research was to gain further understanding and insights on how housewives of various social classes in Chile represent and use cleaning products.
We requested Clorox Chile information about past qualitative reports to gain a better understanding of the brand and better understanding of the business model and customer segments. We performed observation and in-home ethnographic interviews with women using a laundry product (pre-store) as well as accompanying the women in their shopping trip to mini-markets and supermarkets. The idea was to observe the woman in her role of shopper as well as the user of the category.
Copyright Clara Gonzalez Sueyro 2014


Girl Develop It is here to provide affordable and accessible programs to learn software through mentorship and hands-on instruction.
Some "rules"
Jeff Linwood
<EMAIL_ADDRESS>@jefflinwood
www.jefflinwood.com
Major thanks to our Sponsor, GE!
Day 1 12:00pm - 12:30pm - Introduction to the class, students intro, agenda, set up 12:30pm - 1:00pm - Swift and iOS Programming - What goes into building an iPhone app? (Presentation) 1:00pm - 2:00pm - Intro to Swift the Programming Language using Xcode Playgrounds (Hands on Exercise) 2:00pm - 2:15pm - Break 2:15pm - 2:45pm - iPhone App Development Concepts (Presentation) 2:45pm - 3:45pm - Building your first iOS App (Hands on Exercise) 3:45pm - 4:00pm - Wrap Up/Feedback Day 2 12:00pm - 12:30pm - Review of Day 1 12:30pm - 1:00pm - iPhone App User Interface Design (Paper Prototyping) (Team Exercise) 1:00pm - 2:00pm - Designing a User Interface with Storyboard (Hands on Exercise) 2:00pm - 2:15pm - Break 2:15pm - 2:45pm - Developing a Camera App for iOS (Presentation) 2:45pm - 3:45pm - Building a Camera App (Hands on Exercise) 3:45pm - 4:00pm - Wrap Up/Next Steps
Enjoy teaching Intro to iOS Development at UT-Austin
Teach corporate training sessions
Mobile app developer for 6 years, enterprise Java developer before that for 10 years
Does everyone have Xcode downloaded on their Mac?
If not, get it from the App Store
Highly recommend the free App Development with Swift ebook from Apple's iBooks Store
Swift Playgrounds app on newer iPads
What goes into building a mobile app? Lots of different choices when you create an app for iOS or Android. Most of the time, there will be wireframes or mockups, a detailed visual design, a mobile app component for each platform, and a server component. An iOS app is usually written in Swift or Objective-C, and an Android app uses Java or Kotlin. There are also cross-platform options like PhoneGap and React Native.
You can use existing services such as Google's Firebase. Another option is Parse, which Buddy hosts, or you can run your own Parse Server. You could also build your mobile app to work with a server you write in a language like Python, Ruby, or Javascript with Node.js.
When you write code for iOS, you can use one of two languages - Swift or Objective-C. Swift is the newer progamming language we will use in this class. Objective-C is older, so you may find examples of it when you search on Google for solutions, or if you are working with an older project.
Xcode lets you design your application screens in Storyboard, which is a visual editor. To make your screens work on all sizes of iPhones and iPads, you can use Auto Layout tools. These are more complicated than using CSS rules with HTML.
File->New Playground...
Make sure iOS is selected
let name = "Jeff"
let greeting = "Hello, \(name)"
print(greeting)
var isAdult:Bool
if age >= 18 {
isAdult = true
} else {
isAdult = false
}
print(isAdult)
func isAdult(age:Int) -> Bool {
if age >= 18 {
return true
} else {
return false
}
}
@IBAction func pickAPhoto(_ sender: Any) {
if !UIImagePickerController.isSourceTypeAvailable(.photoLibrary) {
return;
}
let picker = UIImagePickerController()
picker.delegate = self
picker.sourceType = .photoLibrary
present(picker, animated: true, completion: nil)
}
func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [String : Any]) {
if let image = info[UIImagePickerControllerOriginalImage] as? UIImage {
photoImageView.image = image
}
picker.dismiss(animated: true, completion: nil)
}
func imagePickerControllerDidCancel(_ picker: UIImagePickerController) {
picker.dismiss(animated: true, completion: nil)
}
@IBAction func shareAPhoto(_ sender: Any) {
guard let image = photoImageView.image else {
return
}
let activityViewController = UIActivityViewController(activityItems: [image],
applicationActivities: nil)
present(activityViewController, animated: true, completion: nil)
}