MOAT IoT Example Applications
Here's two MOAT IoT applications:
- Temp2Twitter: Send temperature to twitter
- PushFromTwitter: Fetch a tweet message and shows onto LCD when a button pushed
Example app: emp2Twitter
Overview
Function
Temp2Twitter is an application that monitors temperature and upload filtered data to Temp2Twitter server app through MOAT IoT platform, Temp2Twitter server app will send these data to twitter, as well as showing them on its web pages.
Temperature is took by an arduino board equiped with a temperature sensor. The board is connected to a notebook through USB port, where Temp2Twitter client app runs on.
Temp2Twitter client app takes temperature every specified seconds, each temperature data will be handled by CEP rules, if passed through, they'll be seen on twitter and web page, or the data will be throw away.
Workflow
Temp2Twitter send request to arduino server for current temperature
- Arduino server send command to arduino board to get temperature
- Arduino board send current temperature to arduino server, arduino server send http response to Temp2Twitter client app
- Temp2Twitter client app send notification to Service-Sync server
- ServiceSync server notify Temp2Twitter server app with http request
- Temp2Twitter server app send the temperature message to twitter
Implementation Detail
NOTE THAT THIS EXAMPLE USES DRAFT VERSION OF APIs
1. First You need defined The Temperature Model.
That is a simple POJO Class. Note that UID is a required.
And The PluginDao defined :
{% highlight java %} package com.yourinventit.iot.temperature.moat; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; import com.yourinventit.dmc.api.moat.PluginDao; import com.yourinventit.iot.temperature.model.TemperatureModel;public class TemperatureModelPluginDao implements PluginDaoFactory defined:
{% highlight java %} package com.yourinventit.iot.temperature.osgi; import java.lang.reflect.Method; import java.util.HashMap; import java.util.Map; import org.osgi.framework.BundleContext; import com.yourinventit.dmc.api.moat.ContextFactory; import com.yourinventit.iot.temperature.moat.TemperatureModelPluginDao; class TemperatureContextFactory implements ContextFactory { private BundleContext bundleContext; private TemperatureModelPluginDao temperatureModelPluginDao; public void setBundleContext(BundleContext bundleContext) { this.bundleContext = bundleContext; } public void setMyTemperatureModelPluginDao( TemperatureModelPluginDao myTempModelPluginDao) { this.temperatureModelPluginDao = myTempModelPluginDao; } public Map createExecutionContext(Object model, Method method) { final Map context = new HashMap(); context.put(BundleContext.class, bundleContext); context.put(TemperatureModelPluginDao.class, temperatureModelPluginDao); return context; } } {% endhighlight %}2. Secode Let's write a BundleActivator Class.
Please note that the comment content.
3. Write a moat js file.
Statement package info at "package.json", rootMoUri is particularly important. It determines the position of your model registration.
Write your app moat js like follow "save-data!1.0.js" ,it will save you temp data to Amazon database.
{% highlight java %} var moat = require('moat'); var context = moat.init(); var session = context.session; var clientRequest = context.clientRequest; var database = context.database; var objects = clientRequest.objects; var size = objects.length; for ( var i = 0; i < size; i++) { var container = objects[i]; if (container['@array']) { var array = container['array']; for ( var j = 0; j < array.length; j++) { save(array[j]); session.log("array["+j+"]:"+array[j]); } } else { save(container); } } function save(entity) { var result = database.insert(entity); session.log('The object@uid:' + result.uid + ' has been saved to the database.'); } {% endhighlight %}If your web app want to receive the real-time temp data, you can send the data by method fetchUrlSync, like this
{% highlight java %} try { var targetUrl = 'http://localhost:8080/IoTDemo/TwitterServlet'; var tokenResponse = session.fetchUrlSync( targetUrl, { method : 'POST', payload : { senserData:array[0] } }); } catch (e) { session.log(e); } {% endhighlight %}4. In your Web app.
Query temperature data. your can use the REST API query all you saved data:
And The PluginDao defined :
{% highlight ruby %} curl -X GET -H "Acept application/json" -d "l=1" 'http://${server_ip}/${server_port}/moat/v1/sys/device/9999e63a3a4a0fd7013a4a65d273014c/ twitter-demo/TemperatureModel?l=500&token=aaaaaaaaaaaaaa' {% endhighlight %}Example app: PushFromTwitter
Overview
Function
PushFromTwitter is an application that fetchs twitter messages and shows it on a LCD when a button clicked. The button to be pushed and LCD used to show tweet message are connected to an arduino board. The pushing action and tweets fetched back will shows on web pages of PushFromTwitter server app.
The arduino board with LCD and buttons is connected to a notebook through USB port, where PushFromTwitter client app runs on.
Workflow
- Button been pushed
- Arduino notify arduino server runs on notebook
- PushFromTwitter client app got a http response from a long polling request to arduino server
- PushFromTwitter client sends notification of push action to ServiceSync server
- ServiceSync server notify PushFromTwitter server app (by http request)
- PushFromTwitter server app search from twitter (with configured keywords) and got the latest message from search results
- PushFromTwitter server app send the twitter message to PushFromTwitter client app through MOAT IoT platform
- PushFromTwitter client app send the twitter message to arduino server
- Arduino server send command to arduino board to show the message on the LCD.
Implementation Detail
NOTE THAT THIS EXAMPLE USES DRAFT VERSION OF APIs
1. First You need defined The LCDControll Model.
That method display(Map context) receive param data from Moat service and send message to Arduino server by http post. Note that UID is a required.
And The PluginDao defined :
{% highlight java %} package com.yourinventit.iot.lcd.moat; import com.yourinventit.dmc.api.moat.PluginDao.SingletonOnMemory; import com.yourinventit.iot.lcd.model.LCDControllModel; public class LCDControllModelPluginDao extends SingletonOnMemoryFactory defined:
{% highlight java %} package com.yourinventit.iot.lcd.osgi; import java.lang.reflect.Method; import java.util.HashMap; import java.util.Map; import org.osgi.framework.BundleContext; import com.yourinventit.dmc.api.moat.ContextFactory; import com.yourinventit.iot.lcd.moat.LCDControllModelPluginDao; class LCDContextFactory implements ContextFactory { private BundleContext bundleContext; private LCDControllModelPluginDao lcdControllModelPluginDao; public void setBundleContext(BundleContext bundleContext) { this.bundleContext = bundleContext; } public void setLcdControllModelPluginDao( LCDControllModelPluginDao lcdControllModelPluginDao) { this.lcdControllModelPluginDao = lcdControllModelPluginDao; } public Map createExecutionContext(Object model, Method method) { final Map context = new HashMap(); context.put(BundleContext.class, bundleContext); context.put(LCDControllModelPluginDao.class, lcdControllModelPluginDao); return context; } } {% endhighlight %}2. Let's write a BundleActivator Class.
Source code, please note that the comment content:
3. Write a moat js file.
Statement package info at "package.json", rootMoUri is particularly important. It determines the position of your model registration.
Write your app moat js like follow "exec-bundle-method!1.0.js" ,it will invoke the method display of LCDControllModel class and send the parameter "args" value to bundle.
{% highlight javascript %} var moat = require('moat'); var context = moat.init(); var session = context.session; // Get dmJob arguments. session.log("\n\n This is exec-bundle-method!1.0.js [START]"); var args = context.clientRequest.dmJob.arguments; session.log("dmJob.arguments:"+args); var token = session.runSingleton('LCDControllModel','display', args); // Commits all preset requests // and a result object is returned. var result = session.commit(); // Gets the execution result // with the query token. var doItResult = result[token]; // Check if the result status is OK. if (doItResult.status == '200') { // do something session.log("execute successful!!!"); {% endhighlight %}4. In your Web app.
Query temperature data. your can use the REST API query all you saved data:
Use curl command post dm job. it'll exec the "exec-bundle-method!1.0.js" code to invoke method runSingleton, send arguments value to display method of LCDControllModel.
{% highlight java %} curl -v -X POST -H "Accept:application/json" -H "Content-Type:application/json" -d "{\"name\":\"ivi-186\",\"jobServiceId\": \"urn:moat:336ad129-5ba5-4912-963e-0edecee52664:Houston-SMD:exec-bundle-method:1.0.0\", \"activatedAt\":\"Wed, 22 Oct 2012 03:15:03 GMT\", \"expiredAt\":\"Wed, 22 Oct 2012 04:30:03 GMT\",\"arguments\": {\"text\":\"TO LCD DISPLAY\"}}" http://${server_ip}/${server_port}/moat/v1/sys/dmjob/?token=aaaaaaaaaaaaaaa {% endhighlight %}


LOGO
Congratulations, you are registered!
Your account information
Login:
{email}
Password:
{password}
Click the link below to confirm your email
Сonfirm
If you have not registered, ignore this email.


PART II. HAWAII STATE OCCUPATIONAL
INFORMATION COORDINATING COMMITTEE--REPEALED
§§373C-11 to 373C-15 REPEALED. L 2001, c 171, §8.


|
cds
1.5.0
|
Type traits for EllenBinTree class. More...
#include <cds/intrusive/details/ellen_bintree_base.h>
Public Types | |
| typedef base_hook | hook |
| Hook used. More... | |
| typedef opt::none | key_extractor |
| Key extracting functor. More... | |
| typedef opt::none | compare |
| Key comparison functor. More... | |
| typedef opt::none | less |
| Specifies binary predicate used for key compare. More... | |
| typedef opt::v::empty_disposer | disposer |
| Disposer. More... | |
| typedef atomicity::empty_item_counter | item_counter |
| Item counter. More... | |
| typedef opt::v::relaxed_ordering | memory_model |
| C++ memory ordering model. More... | |
| typedef CDS_DEFAULT_ALLOCATOR | update_desc_allocator |
| Allocator for update descriptors. More... | |
| typedef CDS_DEFAULT_ALLOCATOR | node_allocator |
| Allocator for internal nodes. More... | |
| typedef empty_stat | stat |
| Internal statistics. More... | |
| typedef cds::opt::v::rcu_throw_deadlock | rcu_check_deadlock |
| RCU deadlock checking policy (only for RCU-based EllenBinTree) More... | |
Type traits for EllenBinTree class.
Key comparison functor.
No default functor is provided. If the option is not specified, the less
is used.
See cds::opt::compare option description for functor interface.
You should provide compare
or less
functor. See predicate requirements.
| typedef opt::v::empty_disposer cds::intrusive::ellen_bintree::type_traits::disposer |
Disposer.
The functor used for dispose removed items. Default is opt::v::empty_disposer.
Hook used.
Possible values are: ellen_bintree::base_hook, ellen_bintree::member_hook, ellen_bintree::traits_hook.
Item counter.
The type for item counting feature (see cds::opt::item_counter). Default is no item counter (atomicity::empty_item_counter)
Key extracting functor.
You should explicit define a valid functor. The functor has the following prototype:
It should initialize dest
key from src
data. The functor is used to initialize internal nodes.
Specifies binary predicate used for key compare.
See cds::opt::less option description for predicate interface.
You should provide compare
or less
functor. See predicate requirements.
C++ memory ordering model.
List of available memory ordering see opt::memory_model
Allocator for internal nodes.
The allocator type is used for internal_node.
| typedef cds::opt::v::rcu_throw_deadlock cds::intrusive::ellen_bintree::type_traits::rcu_check_deadlock |
RCU deadlock checking policy (only for RCU-based EllenBinTree)
List of available options see opt::rcu_check_deadlock
Internal statistics.
Possible types: ellen_bintree::empty_stat (the default), ellen_bintree::stat or any other with interface like stat
.
Allocator for update descriptors.
The allocator type is used for update_desc.
Update descriptor is helping data structure with short lifetime and it is good candidate for pooling. The number of simultaneously existing descriptors is bounded number limited the number of threads working with the tree. Therefore, a bounded lock-free container like cds::container::VyukovMPMCCycleQueue
is good choice for the free-list of update descriptors, see cds::memory::vyukov_queue_pool free-list implementation.
Also notice that size of update descriptor is constant and not dependent on the type of data stored in the tree so single free-list object can be used for several EllenBinTree
object.


{% extends "utils/base.html" %}
{% from 'utils/greeting.html' import greeting %}
{% from 'utils/waste_entry.html' import waste_table, waste_display_script %}
{% block body %}
{{ greeting('Thank you for your request, ' ~ session.get('name') ~ '!',
'Your request is waiting for confirmation from our team. You will receive an SMS notification on ' ~ session.get('phone') ~ ' once a collection has been scheduled') }}
{{ waste_table(waste_entries=waste_entries, description='You submitted the following waste information:') }}


01-data-gathering.Rmd
Steps: - 1) create a vector of all species of interest. - 2) decide if want to download mp3 or not. - 3) set directory for mp3 storage.
# list here all the species.
bird_species <- c("megascops choliba",
"strix hylophila",
"pulsatrix koeniswaldiana",
"megascops atricapilla",
"glaucidium minutissimum")
# Should the MP3 be downloaded to the hard disk?
download <- TRUE
# insert here which folder do you want to store the mp3 files.
dir <- "dados/mp3_originals"
if(!fs::dir_exists(dir)) fs::dir_create(dir)
The warbleR::querxc()
function retrieve mp3 files from xeno-canto.org repository.
The package {wikiaves} is not on CRAN yet. Github install only.
# install.packages("remotes")
# remotes::install_github("Athospd/wikiaves")
library(wikiaves)
# set donwload = TRUE for download the files to your hard disk.
metadata_wa <- bird_species %>%
wa_metadata(download = download, path = dir) %>%
tidyr::nest(mp3_wa = c(-term))


|
VDPAU
|
Macros | |
| #define | VDP_COLOR_TABLE_FORMAT_B8G8R8X8 |
| 8-bit per component packed into 32-bits More... | |
Typedefs | |
| typedef uint32_t | VdpColorTableFormat |
| The set of all known color table formats, for use with VdpOutputSurfacePutBitsIndexed. More... | |
| typedef VdpStatus | VdpOutputSurfaceQueryCapabilities(VdpDevice device, VdpRGBAFormat surface_rgba_format, VdpBool *is_supported, uint32_t *max_width, uint32_t *max_height) |
| Query the implementation's VdpOutputSurface capabilities. More... | |
| typedef VdpStatus | VdpOutputSurfaceQueryGetPutBitsNativeCapabilities(VdpDevice device, VdpRGBAFormat surface_rgba_format, VdpBool *is_supported) |
| Query the implementation's capability to perform a PutBits operation using application data matching the surface's format. More... | |
| typedef VdpStatus | VdpOutputSurfaceQueryPutBitsIndexedCapabilities(VdpDevice device, VdpRGBAFormat surface_rgba_format, VdpIndexedFormat bits_indexed_format, VdpColorTableFormat color_table_format, VdpBool *is_supported) |
| Query the implementation's capability to perform a PutBits operation using application data in a specific indexed format. More... | |
| typedef VdpStatus | VdpOutputSurfaceQueryPutBitsYCbCrCapabilities(VdpDevice device, VdpRGBAFormat surface_rgba_format, VdpYCbCrFormat bits_ycbcr_format, VdpBool *is_supported) |
| Query the implementation's capability to perform a PutBits operation using application data in a specific YCbCr/YUB format. More... | |
| typedef uint32_t | VdpOutputSurface |
| An opaque handle representing a VdpOutputSurface object. More... | |
| typedef VdpStatus | VdpOutputSurfaceCreate(VdpDevice device, VdpRGBAFormat rgba_format, uint32_t width, uint32_t height, VdpOutputSurface *surface) |
| Create a VdpOutputSurface. More... | |
| typedef VdpStatus | VdpOutputSurfaceDestroy(VdpOutputSurface surface) |
| Destroy a VdpOutputSurface. More... | |
| typedef VdpStatus | VdpOutputSurfaceGetParameters(VdpOutputSurface surface, VdpRGBAFormat *rgba_format, uint32_t *width, uint32_t *height) |
| Retrieve the parameters used to create a VdpOutputSurface. More... | |
| typedef VdpStatus | VdpOutputSurfaceGetBitsNative(VdpOutputSurface surface, VdpRect const *source_rect, void *const *destination_data, uint32_t const *destination_pitches) |
| Copy image data from a VdpOutputSurface to application memory in the surface's native format. More... | |
| typedef VdpStatus | VdpOutputSurfacePutBitsNative(VdpOutputSurface surface, void const *const *source_data, uint32_t const *source_pitches, VdpRect const *destination_rect) |
| Copy image data from application memory in the surface's native format to a VdpOutputSurface. More... | |
| typedef VdpStatus | VdpOutputSurfacePutBitsIndexed(VdpOutputSurface surface, VdpIndexedFormat source_indexed_format, void const *const *source_data, uint32_t const *source_pitch, VdpRect const *destination_rect, VdpColorTableFormat color_table_format, void const *color_table) |
| Copy image data from application memory in a specific indexed format to a VdpOutputSurface. More... | |
| typedef VdpStatus | VdpOutputSurfacePutBitsYCbCr(VdpOutputSurface surface, VdpYCbCrFormat source_ycbcr_format, void const *const *source_data, uint32_t const *source_pitches, VdpRect const *destination_rect, VdpCSCMatrix const *csc_matrix) |
| Copy image data from application memory in a specific YCbCr format to a VdpOutputSurface. More... | |
A VdpOutputSurface stores RGBA data in a defined format.
A VdpOutputSurface may be filled with:
VdpOutputSurface content may be accessed by:
VdpVideoSurfaces are directly displayable using a VdpPresentationQueue object.
| #define VDP_COLOR_TABLE_FORMAT_B8G8R8X8 |
8-bit per component packed into 32-bits
This format is an array of packed 32-bit RGB color values. Bits [31:24] are unused, bits [23:16] contain R, bits [15:8] contain G, and bits [7:0] contain B. Note: The format is physically an array of uint32_t values, and should be accessed as such by the application in order to avoid endianness issues.
| typedef uint32_t VdpColorTableFormat |
The set of all known color table formats, for use with VdpOutputSurfacePutBitsIndexed.
| typedef uint32_t VdpOutputSurface |
An opaque handle representing a VdpOutputSurface object.
| typedef VdpStatus VdpOutputSurfaceCreate(VdpDevice device, VdpRGBAFormat rgba_format, uint32_t width, uint32_t height,VdpOutputSurface *surface) |
Create a VdpOutputSurface.
| [in] | device | The device that will contain the surface. |
| [in] | rgba_format | The format of the new surface. |
| [in] | width | The width of the new surface. |
| [in] | height | The height of the new surface. |
| [out] | surface | The new surface's handle. |
The memory backing the surface will be initialized to 0 color and 0 alpha (i.e. black.)
| typedef VdpStatus VdpOutputSurfaceDestroy(VdpOutputSurface surface) |
Destroy a VdpOutputSurface.
| [in] | surface | The surface's handle. |
| typedef VdpStatus VdpOutputSurfaceGetBitsNative(VdpOutputSurface surface, VdpRect const *source_rect, void *const *destination_data, uint32_t const *destination_pitches) |
Copy image data from a VdpOutputSurface to application memory in the surface's native format.
| [in] | surface | The surface's handle. |
| [in] | source_rect | The sub-rectangle of the source surface to copy. If NULL, the entire surface will be retrieved. |
| [in] | destination_data | Pointers to the application data buffers into which the image data will be written. Note that this is an array of pointers, one per plane. The destination_format parameter will define how many planes are required. |
| [in] | destination_pitches | Pointers to the pitch values for the application data buffers. Note that this is an array of pointers, one per plane. The destination_format parameter will define how many planes are required. |
| typedef VdpStatus VdpOutputSurfaceGetParameters(VdpOutputSurface surface,VdpRGBAFormat *rgba_format, uint32_t *width, uint32_t *height) |
Retrieve the parameters used to create a VdpOutputSurface.
| [in] | surface | The surface's handle. |
| [out] | rgba_format | The format of the surface. |
| [out] | width | The width of the surface. |
| [out] | height | The height of the surface. |
| typedef VdpStatus VdpOutputSurfacePutBitsIndexed(VdpOutputSurface surface, VdpIndexedFormat source_indexed_format, void const *const *source_data, uint32_t const *source_pitch, VdpRect const *destination_rect, VdpColorTableFormat color_table_format, void const *color_table) |
Copy image data from application memory in a specific indexed format to a VdpOutputSurface.
| [in] | surface | The surface's handle. |
| [in] | source_indexed_format | The format of the application's data buffers. |
| [in] | source_data | Pointers to the application data buffers from which the image data will be copied. Note that this is an array of pointers, one per plane. The source_indexed_format parameter will define how many planes are required. |
| [in] | source_pitches | Pointers to the pitch values for the application data buffers. Note that this is an array of pointers, one per plane. The source_indexed_format parameter will define how many planes are required. |
| [in] | destination_rect | The sub-rectangle of the surface to fill with application data. If NULL, the entire surface will be updated. |
| [in] | color_table_format | The format of the color_table. |
| [in] | color_table | A table that maps between source index and target color data. See VdpColorTableFormat for details regarding the memory layout. |
| typedef VdpStatus VdpOutputSurfacePutBitsNative(VdpOutputSurface surface, void const *const *source_data, uint32_t const *source_pitches, VdpRect const *destination_rect) |
Copy image data from application memory in the surface's native format to a VdpOutputSurface.
| [in] | surface | The surface's handle. |
| [in] | source_data | Pointers to the application data buffers from which the image data will be copied. Note that this is an array of pointers, one per plane. The source_format parameter will define how many planes are required. |
| [in] | source_pitches | Pointers to the pitch values for the application data buffers. Note that this is an array of pointers, one per plane. The source_format parameter will define how many planes are required. |
| [in] | destination_rect | The sub-rectangle of the surface to fill with application data. If NULL, the entire surface will be updated. |
| typedef VdpStatus VdpOutputSurfacePutBitsYCbCr(VdpOutputSurface surface, VdpYCbCrFormat source_ycbcr_format, void const *const *source_data, uint32_t const *source_pitches, VdpRect const *destination_rect, VdpCSCMatrix const *csc_matrix) |
Copy image data from application memory in a specific YCbCr format to a VdpOutputSurface.
| [in] | surface | The surface's handle. |
| [in] | source_ycbcr_format | The format of the application's data buffers. |
| [in] | source_data | Pointers to the application data buffers from which the image data will be copied. Note that this is an array of pointers, one per plane. The source_ycbcr_format parameter will define how many planes are required. |
| [in] | source_pitches | Pointers to the pitch values for the application data buffers. Note that this is an array of pointers, one per plane. The source_ycbcr_format parameter will define how many planes are required. |
| [in] | destination_rect | The sub-rectangle of the surface to fill with application data. If NULL, the entire surface will be updated. |
| [in] | csc_matrix | The color space conversion matrix used by the copy operation. If NULL, a default matrix will be used internally. Th default matrix is equivalent to ITU-R BT.601 with no procamp changes. |
| typedef VdpStatus VdpOutputSurfaceQueryCapabilities(VdpDevice device, VdpRGBAFormat surface_rgba_format,VdpBool *is_supported, uint32_t *max_width, uint32_t *max_height) |
Query the implementation's VdpOutputSurface capabilities.
| [in] | device | The device to query. |
| [in] | surface_rgba_format | The surface format for which information is requested. |
| [out] | is_supported | Is this surface format supported? |
| [out] | max_width | The maximum supported surface width for this chroma type. |
| [out] | max_height | The maximum supported surface height for this chroma type. |
| typedef VdpStatus VdpOutputSurfaceQueryGetPutBitsNativeCapabilities(VdpDevice device, VdpRGBAFormat surface_rgba_format,VdpBool *is_supported) |
Query the implementation's capability to perform a PutBits operation using application data matching the surface's format.
| [in] | device | The device to query. |
| [in] | surface_rgba_format | The surface format for which information is requested. |
| [out] | is_supported | Is this surface format supported? |
| typedef VdpStatus VdpOutputSurfaceQueryPutBitsIndexedCapabilities(VdpDevice device, VdpRGBAFormat surface_rgba_format, VdpIndexedFormat bits_indexed_format, VdpColorTableFormat color_table_format,VdpBool *is_supported) |
Query the implementation's capability to perform a PutBits operation using application data in a specific indexed format.
| [in] | device | The device to query. |
| [in] | surface_rgba_format | The surface format for which information is requested. |
| [in] | bits_indexed_format | The format of the application data buffer. |
| [in] | color_table_format | The format of the color lookup table. |
| [out] | is_supported | Is this surface format supported? |
| typedef VdpStatus VdpOutputSurfaceQueryPutBitsYCbCrCapabilities(VdpDevice device, VdpRGBAFormat surface_rgba_format, VdpYCbCrFormat bits_ycbcr_format,VdpBool *is_supported) |
Query the implementation's capability to perform a PutBits operation using application data in a specific YCbCr/YUB format.
| [in] | device | The device to query. |
| [in] | surface_rgba_format | The surface format for which information is requested. |
| [in] | bits_ycbcr_format | The format of the application data buffer. |
| [out] | is_supported | Is this surface format supported? |


Chapter 13 Data transformation
13.1 Introduction
Visualisation is an important tool for insight generation, but it is rare that you get the data in exactly the right form you need. Often you’ll need to create some new variables or summaries, or maybe you just want to rename the variables or reorder the observations in order to make the data a little easier to work with.
데이터분석을 하다보면 주어진 데이터가 분석하기 편한 형태로 주어지는 경우는 많지 않다. 대부분의 경우는 분석하기 편한 형태로 가공하기 위해 새로운 변수의 생성 또는 요약, 변수명 재정의, 데이터 순서 조정 등의 전처리가 필요하다. 이번 절에서는 dplyr
패키지를 이용하여 데이터를 변환하는 방법에 대해 알아보자 한다.
In this chapter, you’ll learn how to transform your data using the dplyr package. We’ll illustrate the key ideas using data from the nycflights13 package, and use ggplot2 to help us understand the data.
13.1.1 Prerequisites
library(nycflights13)
library(tidyverse)
13.1.2 dplyr basics
In this chapter you are going to learn the five key dplyr functions that allow you to solve the vast majority of your data manipulation challenges:
이번 절에서는 데이터 전처리 업무시 가장 많이 사용되는 dplyr
패키지의 5가지 동사 명령어에 대해 알아보자
- Pick observations by their values (
filter()
). - Reorder the rows (
arrange()
). - Pick variables by their names (
select()
). - Create new variables with functions of existing variables (
mutate()
). - Collapse many values down to a single summary (
summarise()
).
These can all be used in conjunction with group_by()
which changes the scope of each function from operating on the entire dataset to operating on it group-by-group. These six functions provide the verbs for a language of data manipulation.
All verbs work similarly:
The first argument is a data frame.
The subsequent arguments describe what to do with the data frame, using the variable names (without quotes).
The result is a new data frame.
Together these properties make it easy to chain together multiple simple steps to achieve a complex result.
Let’s dive in and see how these verbs work.
13.2 Filter rows with filter()
filter()
allows you to subset observations based on their values. The first argument is the name of the data frame. The second and subsequent arguments are the expressions that filter the data frame. For example, we can select all flights on January 1st with:
filter()
는 관측치 값을 기준으로 데이터를 subset
filter(flights, month == 1, day == 1)
#> # A tibble: 842 x 19
#> year month day dep_time sched_dep_time dep_delay arr_time
#> <int> <int> <int> <int> <int> <dbl> <int>
#> 1 2013 1 1 517 515 2. 830
#> 2 2013 1 1 533 529 4. 850
#> 3 2013 1 1 542 540 2. 923
#> 4 2013 1 1 544 545 -1. 1004
#> 5 2013 1 1 554 600 -6. 812
#> 6 2013 1 1 554 558 -4. 740
#> # ... with 836 more rows, and 12 more variables: sched_arr_time <int>,
#> # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
#> # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
#> # minute <dbl>, time_hour <dttm>
When you run that line of code, dplyr executes the filtering operation and returns a new data frame. dplyr functions never modify their inputs, so if you want to save the result, you’ll need to use the assignment operator,
<-
:
결과를 저장하기 위해 변수에 assign 필요
jan1 <- filter(flights, month == 1, day == 1)
R either prints out the results, or saves them to a variable. If you want to do both, you can wrap the assignment in parentheses:
출력과 assign을 동시에 하기 위해 ()로 묶는다.
(dec25 <- filter(flights, month == 12, day == 25))
#> # A tibble: 719 x 19
#> year month day dep_time sched_dep_time dep_delay arr_time
#> <int> <int> <int> <int> <int> <dbl> <int>
#> 1 2013 12 25 456 500 -4. 649
#> 2 2013 12 25 524 515 9. 805
#> 3 2013 12 25 542 540 2. 832
#> 4 2013 12 25 546 550 -4. 1022
#> 5 2013 12 25 556 600 -4. 730
#> 6 2013 12 25 557 600 -3. 743
#> # ... with 713 more rows, and 12 more variables: sched_arr_time <int>,
#> # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
#> # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
#> # minute <dbl>, time_hour <dttm>
13.2.1 Comparisons
To use filtering effectively, you have to know how to select the observations that you want using the comparison operators. R provides the standard suite:
>
,>=
,<
,<=
,!=
(not equal), and==
(equal).
filtering을 효과적으로 하기 위해서는 비교구문을 잘 활용할 필요
When you’re starting out with R, the easiest mistake to make is to use
=
instead of==
when testing for equality. When this happens you’ll get an informative error:
=
와==
구분
filter(flights, month = 1)
#> Error: `month` (`month = 1`) must not be named, do you need `==`?
There’s another common problem you might encounter when using
==
: floating point numbers. These results might surprise you!
floating-point number 사용시 approximation에 주의
sqrt(2) ^ 2 == 2
#> [1] FALSE
1/49 * 49 == 1
#> [1] FALSE
Computers use finite precision arithmetic (they obviously can’t store an infinite number of digits!) so remember that every number you see is an approximation. Instead of relying on
==
, usenear()
:
==
보다 near()
사용
near(sqrt(2) ^ 2, 2)
#> [1] TRUE
near(1 / 49 * 49, 1)
#> [1] TRUE
13.2.2 Logical operators
Multiple arguments to filter()
are combined with “and”: every expression must be true in order for a row to be included in the output. For other types of combinations, you’ll need to use Boolean operators yourself: &
is “and”, |
is “or”, and !
is “not”. Figure 13.1 shows the complete set of Boolean operations.
The following code finds all flights that departed in November or December:
filter(flights, month == 11 | month == 12)
The order of operations doesn’t work like English. You can’t write filter(flights, month == 11 | 12)
, which you might literally translate into “finds all flights that departed in November or December”. Instead it finds all months that equal 11 | 12
, an expression that evaluates to TRUE
. In a numeric context (like here), TRUE
becomes one, so this finds all flights in January, not November or December. This is quite confusing!
A useful short-hand for this problem is x %in% y
. This will select every row where x
is one of the values in y
. We could use it to rewrite the code above:
nov_dec <- filter(flights, month %in% c(11, 12))
Sometimes you can simplify complicated subsetting by remembering De Morgan’s law: !(x & y)
is the same as !x | !y
, and !(x | y)
is the same as !x & !y
. For example, if you wanted to find flights that weren’t delayed (on arrival or departure) by more than two hours, you could use either of the following two filters:
filter(flights, !(arr_delay > 120 | dep_delay > 120))
filter(flights, arr_delay <= 120, dep_delay <= 120)
As well as &
and |
, R also has &&
and ||
. Don’t use them here! You’ll learn when you should use them in [conditional execution].
Whenever you start using complicated, multipart expressions in filter()
, consider making them explicit variables instead. That makes it much easier to check your work. You’ll learn how to create new variables shortly.
13.2.3 Missing values
One important feature of R that can make comparison tricky are missing values, or NA
s (“not availables”). NA
represents an unknown value so missing values are “contagious”: almost any operation involving an unknown value will also be unknown.
NA > 5
#> [1] NA
10 == NA
#> [1] NA
NA + 10
#> [1] NA
NA / 2
#> [1] NA
The most confusing result is this one:
NA == NA
#> [1] NA
It’s easiest to understand why this is true with a bit more context:
# Let x be Mary's age. We don't know how old she is.
x <- NA
# Let y be John's age. We don't know how old he is.
y <- NA
# Are John and Mary the same age?
x == y
#> [1] NA
# We don't know!
If you want to determine if a value is missing, use is.na()
:
is.na(x)
#> [1] TRUE
filter()
only includes rows where the condition is TRUE
; it excludes both FALSE
and NA
values. If you want to preserve missing values, ask for them explicitly:
df <- tibble(x = c(1, NA, 3))
filter(df, x > 1)
#> # A tibble: 1 x 1
#> x
#> <dbl>
#> 1 3.
filter(df, is.na(x) | x > 1)
#> # A tibble: 2 x 1
#> x
#> <dbl>
#> 1 NA
#> 2 3.
13.2.4 Exercises
Find all flights that
- Had an arrival delay of two or more hours
- Flew to Houston (
IAH
orHOU
) - Were operated by United, American, or Delta
- Departed in summer (July, August, and September)
- Arrived more than two hours late, but didn’t leave late
- Were delayed by at least an hour, but made up over 30 minutes in flight
- Departed between midnight and 6am (inclusive)
Another useful dplyr filtering helper is
between()
. What does it do? Can you use it to simplify the code needed to answer the previous challenges?How many flights have a missing
dep_time
? What other variables are missing? What might these rows represent?Why is
NA ^ 0
not missing? Why isNA | TRUE
not missing? Why isFALSE & NA
not missing? Can you figure out the general rule? (NA * 0
is a tricky counterexample!)
13.3 Arrange rows with arrange()
arrange()
works similarly to filter()
except that instead of selecting rows, it changes their order. It takes a data frame and a set of column names (or more complicated expressions) to order by. If you provide more than one column name, each additional column will be used to break ties in the values of preceding columns:
arrange(flights, year, month, day)
#> # A tibble: 336,776 x 19
#> year month day dep_time sched_dep_time dep_delay arr_time
#> <int> <int> <int> <int> <int> <dbl> <int>
#> 1 2013 1 1 517 515 2. 830
#> 2 2013 1 1 533 529 4. 850
#> 3 2013 1 1 542 540 2. 923
#> 4 2013 1 1 544 545 -1. 1004
#> 5 2013 1 1 554 600 -6. 812
#> 6 2013 1 1 554 558 -4. 740
#> # ... with 3.368e+05 more rows, and 12 more variables:
#> # sched_arr_time <int>, arr_delay <dbl>, carrier <chr>, flight <int>,
#> # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>,
#> # distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
Use desc()
to re-order by a column in descending order:
arrange(flights, desc(arr_delay))
#> # A tibble: 336,776 x 19
#> year month day dep_time sched_dep_time dep_delay arr_time
#> <int> <int> <int> <int> <int> <dbl> <int>
#> 1 2013 1 9<PHONE_NUMBER>. 1242
#> 2 2013 6 15 1432 1935 1137. 1607
#> 3 2013 1 10 1121 1635 1126. 1239
#> 4 2013 9 20 1139 1845 1014. 1457
#> 5 2013 7 22 845 1600 1005. 1044
#> 6 2013 4 10 1100 1900 960. 1342
#> # ... with 3.368e+05 more rows, and 12 more variables:
#> # sched_arr_time <int>, arr_delay <dbl>, carrier <chr>, flight <int>,
#> # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>,
#> # distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
Missing values are always sorted at the end:
df <- tibble(x = c(5, 2, NA))
arrange(df, x)
#> # A tibble: 3 x 1
#> x
#> <dbl>
#> 1 2.
#> 2 5.
#> 3 NA
arrange(df, desc(x))
#> # A tibble: 3 x 1
#> x
#> <dbl>
#> 1 5.
#> 2 2.
#> 3 NA
13.3.1 Exercises
How could you use
arrange()
to sort all missing values to the start? (Hint: useis.na()
).Sort
flights
to find the most delayed flights. Find the flights that left earliest.Sort
flights
to find the fastest flights.Which flights travelled the longest? Which travelled the shortest?
13.4 Select columns with select()
It’s not uncommon to get datasets with hundreds or even thousands of variables. In this case, the first challenge is often narrowing in on the variables you’re actually interested in. select()
allows you to rapidly zoom in on a useful subset using operations based on the names of the variables.
select()
is not terribly useful with the flights data because we only have 19 variables, but you can still get the general idea:
# Select columns by name
select(flights, year, month, day)
#> # A tibble: 336,776 x 3
#> year month day
#> <int> <int> <int>
#> 1 2013 1 1
#> 2 2013 1 1
#> 3 2013 1 1
#> 4 2013 1 1
#> 5 2013 1 1
#> 6 2013 1 1
#> # ... with 3.368e+05 more rows
# Select all columns between year and day (inclusive)
select(flights, year:day)
#> # A tibble: 336,776 x 3
#> year month day
#> <int> <int> <int>
#> 1 2013 1 1
#> 2 2013 1 1
#> 3 2013 1 1
#> 4 2013 1 1
#> 5 2013 1 1
#> 6 2013 1 1
#> # ... with 3.368e+05 more rows
# Select all columns except those from year to day (inclusive)
select(flights, -(year:day))
#> # A tibble: 336,776 x 16
#> dep_time sched_dep_time dep_delay arr_time sched_arr_time arr_delay
#> <int> <int> <dbl> <int> <int> <dbl>
#> 1 517 515 2. 830 819 11.
#> 2 533 529 4. 850 830 20.
#> 3 542 540 2. 923 850 33.
#> 4 544 545 -1. 1004 1022 -18.
#> 5 554 600 -6. 812 837 -25.
#> 6 554 558 -4. 740 728 12.
#> # ... with 3.368e+05 more rows, and 10 more variables: carrier <chr>,
#> # flight <int>, tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>,
#> # distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
There are a number of helper functions you can use within select()
:
starts_with("abc")
: matches names that begin with “abc”.ends_with("xyz")
: matches names that end with “xyz”.contains("ijk")
: matches names that contain “ijk”.matches("(.)\\1")
: selects variables that match a regular expression. This one matches any variables that contain repeated characters. You’ll learn more about regular expressions in [strings].num_range("x", 1:3)
matchesx1
,x2
andx3
.
See ?select
for more details.
select()
can be used to rename variables, but it’s rarely useful because it drops all of the variables not explicitly mentioned. Instead, use rename()
, which is a variant of select()
that keeps all the variables that aren’t explicitly mentioned:
rename(flights, tail_num = tailnum)
#> # A tibble: 336,776 x 19
#> year month day dep_time sched_dep_time dep_delay arr_time
#> <int> <int> <int> <int> <int> <dbl> <int>
#> 1 2013 1 1 517 515 2. 830
#> 2 2013 1 1 533 529 4. 850
#> 3 2013 1 1 542 540 2. 923
#> 4 2013 1 1 544 545 -1. 1004
#> 5 2013 1 1 554 600 -6. 812
#> 6 2013 1 1 554 558 -4. 740
#> # ... with 3.368e+05 more rows, and 12 more variables:
#> # sched_arr_time <int>, arr_delay <dbl>, carrier <chr>, flight <int>,
#> # tail_num <chr>, origin <chr>, dest <chr>, air_time <dbl>,
#> # distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
Another option is to use select()
in conjunction with the everything()
helper. This is useful if you have a handful of variables you’d like to move to the start of the data frame.
select(flights, time_hour, air_time, everything())
#> # A tibble: 336,776 x 19
#> time_hour air_time year month day dep_time sched_dep_time
#> <dttm> <dbl> <int> <int> <int> <int> <int>
#> 1 2013-01-01 05:00:00 227. 2013 1 1 517 515
#> 2 2013-01-01 05:00:00 227. 2013 1 1 533 529
#> 3 2013-01-01 05:00:00 160. 2013 1 1 542 540
#> 4 2013-01-01 05:00:00 183. 2013 1 1 544 545
#> 5 2013-01-01 06:00:00 116. 2013 1 1 554 600
#> 6 2013-01-01 05:00:00 150. 2013 1 1 554 558
#> # ... with 3.368e+05 more rows, and 12 more variables: dep_delay <dbl>,
#> # arr_time <int>, sched_arr_time <int>, arr_delay <dbl>, carrier <chr>,
#> # flight <int>, tailnum <chr>, origin <chr>, dest <chr>, distance <dbl>,
#> # hour <dbl>, minute <dbl>
13.4.1 Exercises
Brainstorm as many ways as possible to select
dep_time
,dep_delay
,arr_time
, andarr_delay
fromflights
.What happens if you include the name of a variable multiple times in a
select()
call?What does the
one_of()
function do? Why might it be helpful in conjunction with this vector?vars <- c("year", "month", "day", "dep_delay", "arr_delay")
Does the result of running the following code surprise you? How do the select helpers deal with case by default? How can you change that default?
select(flights, contains("TIME"))
13.5 Add new variables with mutate()
Besides selecting sets of existing columns, it’s often useful to add new columns that are functions of existing columns. That’s the job of mutate()
.
mutate()
always adds new columns at the end of your dataset so we’ll start by creating a narrower dataset so we can see the new variables. Remember that when you’re in RStudio, the easiest way to see all the columns is View()
.
flights_sml <- select(flights,
year:day,
ends_with("delay"),
distance,
air_time
)
mutate(flights_sml,
gain = arr_delay - dep_delay,
speed = distance / air_time * 60
)
#> # A tibble: 336,776 x 9
#> year month day dep_delay arr_delay distance air_time gain speed
#> <int> <int> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 2013 1 1 2. 11. 1400. 227. 9. 370.
#> 2 2013 1 1 4. 20. 1416. 227. 16. 374.
#> 3 2013 1 1 2. 33. 1089. 160. 31. 408.
#> 4 2013 1 1 -1. -18. 1576. 183. -17. 517.
#> 5 2013 1 1 -6. -25. 762. 116. -19. 394.
#> 6 2013 1 1 -4. 12. 719. 150. 16. 288.
#> # ... with 3.368e+05 more rows
Note that you can refer to columns that you’ve just created:
mutate(flights_sml,
gain = arr_delay - dep_delay,
hours = air_time / 60,
gain_per_hour = gain / hours
)
#> # A tibble: 336,776 x 10
#> year month day dep_delay arr_delay distance air_time gain hours
#> <int> <int> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 2013 1 1 2. 11. 1400. 227. 9. 3.78
#> 2 2013 1 1 4. 20. 1416. 227. 16. 3.78
#> 3 2013 1 1 2. 33. 1089. 160. 31. 2.67
#> 4 2013 1 1 -1. -18. 1576. 183. -17. 3.05
#> 5 2013 1 1 -6. -25. 762. 116. -19. 1.93
#> 6 2013 1 1 -4. 12. 719. 150. 16. 2.50
#> # ... with 3.368e+05 more rows, and 1 more variable: gain_per_hour <dbl>
If you only want to keep the new variables, use transmute()
:
transmute(flights,
gain = arr_delay - dep_delay,
hours = air_time / 60,
gain_per_hour = gain / hours
)
#> # A tibble: 336,776 x 3
#> gain hours gain_per_hour
#> <dbl> <dbl> <dbl>
#> 1 9. 3.78 2.38
#> 2 16. 3.78 4.23
#> 3 31. 2.67 11.6
#> 4 -17. 3.05 -5.57
#> 5 -19. 1.93 -9.83
#> 6 16. 2.50 6.40
#> # ... with 3.368e+05 more rows
13.5.1 Useful creation functions
There are many functions for creating new variables that you can use with mutate()
. The key property is that the function must be vectorised: it must take a vector of values as input, return a vector with the same number of values as output. There’s no way to list every possible function that you might use, but here’s a selection of functions that are frequently useful:
Arithmetic operators:
+
,-
,*
,/
,^
. These are all vectorised, using the so called “recycling rules”. If one parameter is shorter than the other, it will be automatically extended to be the same length. This is most useful when one of the arguments is a single number:air_time / 60
,hours * 60 + minute
, etc.Arithmetic operators are also useful in conjunction with the aggregate functions you’ll learn about later. For example,
x / sum(x)
calculates the proportion of a total, andy - mean(y)
computes the difference from the mean.Modular arithmetic:
%/%
(integer division) and%%
(remainder), wherex == y * (x %/% y) + (x %% y)
. Modular arithmetic is a handy tool because it allows you to break integers up into pieces. For example, in the flights dataset, you can computehour
andminute
fromdep_time
with:transmute(flights, dep_time, hour = dep_time %/% 100, minute = dep_time %% 100 ) #> # A tibble: 336,776 x 3 #> dep_time hour minute #> <int> <dbl> <dbl> #> 1 517 5. 17. #> 2 533 5. 33. #> 3 542 5. 42. #> 4 544 5. 44. #> 5 554 5. 54. #> 6 554 5. 54. #> # ... with 3.368e+05 more rows
Logs:
log()
,log2()
,log10()
. Logarithms are an incredibly useful transformation for dealing with data that ranges across multiple orders of magnitude. They also convert multiplicative relationships to additive, a feature we’ll come back to in modelling.All else being equal, I recommend using
log2()
because it’s easy to interpret: a difference of 1 on the log scale corresponds to doubling on the original scale and a difference of -1 corresponds to halving.Offsets:
lead()
andlag()
allow you to refer to leading or lagging values. This allows you to compute running differences (e.g.x - lag(x)
) or find when values change (x != lag(x))
. They are most useful in conjunction withgroup_by()
, which you’ll learn about shortly.(x <- 1:10) #> [1] 1 2 3 4 5 6 7 8 9 10 lag(x) #> [1] NA 1 2 3 4 5 6 7 8 9 lead(x) #> [1] 2 3 4 5 6 7 8 9 10 NA
Cumulative and rolling aggregates: R provides functions for running sums, products, mins and maxes:
cumsum()
,cumprod()
,cummin()
,cummax()
; and dplyr providescummean()
for cumulative means. If you need rolling aggregates (i.e. a sum computed over a rolling window), try the RcppRoll package.x #> [1] 1 2 3 4 5 6 7 8 9 10 cumsum(x) #> [1] 1 3 6 10 15 21 28 36 45 55 cummean(x) #> [1] 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5
Logical comparisons,
<
,<=
,>
,>=
,!=
, which you learned about earlier. If you’re doing a complex sequence of logical operations it’s often a good idea to store the interim values in new variables so you can check that each step is working as expected.Ranking: there are a number of ranking functions, but you should start with
min_rank()
. It does the most usual type of ranking (e.g. 1st, 2nd, 2nd, 4th). The default gives smallest values the small ranks; usedesc(x)
to give the largest values the smallest ranks.y <- c(1, 2, 2, NA, 3, 4) min_rank(y) #> [1] 1 2 2 NA 4 5 min_rank(desc(y)) #> [1] 5 3 3 NA 2 1
If
min_rank()
doesn’t do what you need, look at the variantsrow_number()
,dense_rank()
,percent_rank()
,cume_dist()
,ntile()
. See their help pages for more details.row_number(y) #> [1] 1 2 3 NA 4 5 dense_rank(y) #> [1] 1 2 2 NA 3 4 percent_rank(y) #> [1] 0.00 0.25 0.25 NA 0.75 1.00 cume_dist(y) #> [1] 0.2 0.6 0.6 NA 0.8 1.0
13.5.2 Exercises
Currently
dep_time
andsched_dep_time
are convenient to look at, but hard to compute with because they’re not really continuous numbers. Convert them to a more convenient representation of number of minutes since midnight.Compare
air_time
witharr_time - dep_time
. What do you expect to see? What do you see? What do you need to do to fix it?Compare
dep_time
,sched_dep_time
, anddep_delay
. How would you expect those three numbers to be related?Find the 10 most delayed flights using a ranking function. How do you want to handle ties? Carefully read the documentation for
min_rank()
.What does
1:3 + 1:10
return? Why?What trigonometric functions does R provide?
13.6 Grouped summaries with summarise()
The last key verb is summarise()
. It collapses a data frame to a single row:
summarise(flights, delay = mean(dep_delay, na.rm = TRUE))
#> # A tibble: 1 x 1
#> delay
#> <dbl>
#> 1 12.6
(We’ll come back to what that na.rm = TRUE
means very shortly.)
summarise()
is not terribly useful unless we pair it with group_by()
. This changes the unit of analysis from the complete dataset to individual groups. Then, when you use the dplyr verbs on a grouped data frame they’ll be automatically applied “by group”. For example, if we applied exactly the same code to a data frame grouped by date, we get the average delay per date:
by_day <- group_by(flights, year, month, day)
summarise(by_day, delay = mean(dep_delay, na.rm = TRUE))
#> # A tibble: 365 x 4
#> # Groups: year, month [?]
#> year month day delay
#> <int> <int> <int> <dbl>
#> 1 2013 1 1 11.5
#> 2 2013 1 2 13.9
#> 3 2013 1 3 11.0
#> 4 2013 1 4 8.95
#> 5 2013 1 5 5.73
#> 6 2013 1 6 7.15
#> # ... with 359 more rows
Together group_by()
and summarise()
provide one of the tools that you’ll use most commonly when working with dplyr: grouped summaries. But before we go any further with this, we need to introduce a powerful new idea: the pipe.
13.6.1 Combining multiple operations with the pipe
Imagine that we want to explore the relationship between the distance and average delay for each location. Using what you know about dplyr, you might write code like this:
by_dest <- group_by(flights, dest)
delay <- summarise(by_dest,
count = n(),
dist = mean(distance, na.rm = TRUE),
delay = mean(arr_delay, na.rm = TRUE)
)
delay <- filter(delay, count > 20, dest != "HNL")
# It looks like delays increase with distance up to ~750 miles
# and then decrease. Maybe as flights get longer there's more
# ability to make up delays in the air?
ggplot(data = delay, mapping = aes(x = dist, y = delay)) +
geom_point(aes(size = count), alpha = 1/3) +
geom_smooth(se = FALSE)
#> `geom_smooth()` using method = 'loess' and formula 'y ~ x'
There are three steps to prepare this data:
Group flights by destination.
Summarise to compute distance, average delay, and number of flights.
Filter to remove noisy points and Honolulu airport, which is almost twice as far away as the next closest airport.
This code is a little frustrating to write because we have to give each intermediate data frame a name, even though we don’t care about it. Naming things is hard, so this slows down our analysis.
There’s another way to tackle the same problem with the pipe, %>%
:
delays <- flights %>%
group_by(dest) %>%
summarise(
count = n(),
dist = mean(distance, na.rm = TRUE),
delay = mean(arr_delay, na.rm = TRUE)
) %>%
filter(count > 20, dest != "HNL")
This focuses on the transformations, not what’s being transformed, which makes the code easier to read. You can read it as a series of imperative statements: group, then summarise, then filter. As suggested by this reading, a good way to pronounce %>%
when reading code is “then”.
Behind the scenes, x %>% f(y)
turns into f(x, y)
, and x %>% f(y) %>% g(z)
turns into g(f(x, y), z)
and so on. You can use the pipe to rewrite multiple operations in a way that you can read left-to-right, top-to-bottom. We’ll use piping frequently from now on because it considerably improves the readability of code, and we’ll come back to it in more detail in [pipes].
Working with the pipe is one of the key criteria for belonging to the tidyverse. The only exception is ggplot2: it was written before the pipe was discovered. Unfortunately, the next iteration of ggplot2, ggvis, which does use the pipe, isn’t quite ready for prime time yet.
13.6.2 Missing values
You may have wondered about the na.rm
argument we used above. What happens if we don’t set it?
flights %>%
group_by(year, month, day) %>%
summarise(mean = mean(dep_delay))
#> # A tibble: 365 x 4
#> # Groups: year, month [?]
#> year month day mean
#> <int> <int> <int> <dbl>
#> 1 2013 1 1 NA
#> 2 2013 1 2 NA
#> 3 2013 1 3 NA
#> 4 2013 1 4 NA
#> 5 2013 1 5 NA
#> 6 2013 1 6 NA
#> # ... with 359 more rows
We get a lot of missing values! That’s because aggregation functions obey the usual rule of missing values: if there’s any missing value in the input, the output will be a missing value. Fortunately, all aggregation functions have an na.rm
argument which removes the missing values prior to computation:
flights %>%
group_by(year, month, day) %>%
summarise(mean = mean(dep_delay, na.rm = TRUE))
#> # A tibble: 365 x 4
#> # Groups: year, month [?]
#> year month day mean
#> <int> <int> <int> <dbl>
#> 1 2013 1 1 11.5
#> 2 2013 1 2 13.9
#> 3 2013 1 3 11.0
#> 4 2013 1 4 8.95
#> 5 2013 1 5 5.73
#> 6 2013 1 6 7.15
#> # ... with 359 more rows
In this case, where missing values represent cancelled flights, we could also tackle the problem by first removing the cancelled flights. We’ll save this dataset so we can reuse in the next few examples.
not_cancelled <- flights %>%
filter(!is.na(dep_delay), !is.na(arr_delay))
not_cancelled %>%
group_by(year, month, day) %>%
summarise(mean = mean(dep_delay))
#> # A tibble: 365 x 4
#> # Groups: year, month [?]
#> year month day mean
#> <int> <int> <int> <dbl>
#> 1 2013 1 1 11.4
#> 2 2013 1 2 13.7
#> 3 2013 1 3 10.9
#> 4 2013 1 4 8.97
#> 5 2013 1 5 5.73
#> 6 2013 1 6 7.15
#> # ... with 359 more rows
13.6.3 Counts
Whenever you do any aggregation, it’s always a good idea to include either a count (n()
), or a count of non-missing values (sum(!is.na(x))
). That way you can check that you’re not drawing conclusions based on very small amounts of data. For example, let’s look at the planes (identified by their tail number) that have the highest average delays:
delays <- not_cancelled %>%
group_by(tailnum) %>%
summarise(
delay = mean(arr_delay)
)
ggplot(data = delays, mapping = aes(x = delay)) +
geom_freqpoly(binwidth = 10)
Wow, there are some planes that have an average delay of 5 hours (300 minutes)!
The story is actually a little more nuanced. We can get more insight if we draw a scatterplot of number of flights vs. average delay:
delays <- not_cancelled %>%
group_by(tailnum) %>%
summarise(
delay = mean(arr_delay, na.rm = TRUE),
n = n()
)
ggplot(data = delays, mapping = aes(x = n, y = delay)) +
geom_point(alpha = 1/10)
Not surprisingly, there is much greater variation in the average delay when there are few flights. The shape of this plot is very characteristic: whenever you plot a mean (or other summary) vs. group size, you’ll see that the variation decreases as the sample size increases.
When looking at this sort of plot, it’s often useful to filter out the groups with the smallest numbers of observations, so you can see more of the pattern and less of the extreme variation in the smallest groups. This is what the following code does, as well as showing you a handy pattern for integrating ggplot2 into dplyr flows. It’s a bit painful that you have to switch from %>%
to +
, but once you get the hang of it, it’s quite convenient.
delays %>%
filter(n > 25) %>%
ggplot(mapping = aes(x = n, y = delay)) +
geom_point(alpha = 1/10)
RStudio tip: a useful keyboard shortcut is Cmd/Ctrl + Shift + P. This resends the previously sent chunk from the editor to the console. This is very convenient when you’re (e.g.) exploring the value of n
in the example above. You send the whole block once with Cmd/Ctrl + Enter, then you modify the value of n
and press Cmd/Ctrl + Shift + P to resend the complete block.
There’s another common variation of this type of pattern. Let’s look at how the average performance of batters in baseball is related to the number of times they’re at bat. Here I use data from the Lahman package to compute the batting average (number of hits / number of attempts) of every major league baseball player.
When I plot the skill of the batter (measured by the batting average, ba
) against the number of opportunities to hit the ball (measured by at bat, ab
), you see two patterns:
As above, the variation in our aggregate decreases as we get more data points.
There’s a positive correlation between skill (
ba
) and opportunities to hit the ball (ab
). This is because teams control who gets to play, and obviously they’ll pick their best players.
# Convert to a tibble so it prints nicely
batting <- as_tibble(Lahman::Batting)
batters <- batting %>%
group_by(playerID) %>%
summarise(
ba = sum(H, na.rm = TRUE) / sum(AB, na.rm = TRUE),
ab = sum(AB, na.rm = TRUE)
)
batters %>%
filter(ab > 100) %>%
ggplot(mapping = aes(x = ab, y = ba)) +
geom_point() +
geom_smooth(se = FALSE)
#> `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'
This also has important implications for ranking. If you naively sort on desc(ba)
, the people with the best batting averages are clearly lucky, not skilled:
batters %>%
arrange(desc(ba))
#> # A tibble: 18,915 x 3
#> playerID ba ab
#> <chr> <dbl> <int>
#> 1 abramge01 1. 1
#> 2 banisje01 1. 1
#> 3 bartocl01 1. 1
#> 4 bassdo01 1. 1
#> 5 berrijo01 1. 1
#> 6 birasst01 1. 2
#> # ... with 1.891e+04 more rows
You can find a good explanation of this problem at http://varianceexplained.org/r/empirical_bayes_baseball/ and http://www.evanmiller.org/how-not-to-sort-by-average-rating.html.
13.6.4 Useful summary functions
Just using means, counts, and sum can get you a long way, but R provides many other useful summary functions:
Measures of location: we’ve used
mean(x)
, butmedian(x)
is also useful. The mean is the sum divided by the length; the median is a value where 50% ofx
is above it, and 50% is below it.It’s sometimes useful to combine aggregation with logical subsetting. We haven’t talked about this sort of subsetting yet, but you’ll learn more about it in [subsetting].
not_cancelled %>% group_by(year, month, day) %>% summarise( avg_delay1 = mean(arr_delay), avg_delay2 = mean(arr_delay[arr_delay > 0]) # the average positive delay ) #> # A tibble: 365 x 5 #> # Groups: year, month [?] #> year month day avg_delay1 avg_delay2 #> <int> <int> <int> <dbl> <dbl> #> 1 2013 1 1 12.7 32.5 #> 2 2013 1 2 12.7 32.0 #> 3 2013 1 3 5.73 27.7 #> 4 2013 1 4 -1.93 28.3 #> 5 2013 1 5 -1.53 22.6 #> 6 2013 1 6 4.24 24.4 #> # ... with 359 more rows
Measures of spread:
sd(x)
,IQR(x)
,mad(x)
. The mean squared deviation, or standard deviation or sd for short, is the standard measure of spread. The interquartile rangeIQR()
and median absolute deviationmad(x)
are robust equivalents that may be more useful if you have outliers.# Why is distance to some destinations more variable than to others? not_cancelled %>% group_by(dest) %>% summarise(distance_sd = sd(distance)) %>% arrange(desc(distance_sd)) #> # A tibble: 104 x 2 #> dest distance_sd #> <chr> <dbl> #> 1 EGE 10.5 #> 2 SAN 10.4 #> 3 SFO 10.2 #> 4 HNL 10.0 #> 5 SEA 9.98 #> 6 LAS 9.91 #> # ... with 98 more rows
Measures of rank:
min(x)
,quantile(x, 0.25)
,max(x)
. Quantiles are a generalisation of the median. For example,quantile(x, 0.25)
will find a value ofx
that is greater than 25% of the values, and less than the remaining 75%.# When do the first and last flights leave each day? not_cancelled %>% group_by(year, month, day) %>% summarise( first = min(dep_time), last = max(dep_time) ) #> # A tibble: 365 x 5 #> # Groups: year, month [?] #> year month day first last #> <int> <int> <int> <dbl> <dbl> #> 1 2013 1 1 517. 2356. #> 2 2013 1 2 42. 2354. #> 3 2013 1 3 32. 2349. #> 4 2013 1 4 25. 2358. #> 5 2013 1 5 14. 2357. #> 6 2013 1 6 16. 2355. #> # ... with 359 more rows
Measures of position:
first(x)
,nth(x, 2)
,last(x)
. These work similarly tox[1]
,x[2]
, andx[length(x)]
but let you set a default value if that position does not exist (i.e. you’re trying to get the 3rd element from a group that only has two elements). For example, we can find the first and last departure for each day:not_cancelled %>% group_by(year, month, day) %>% summarise( first_dep = first(dep_time), last_dep = last(dep_time) ) #> # A tibble: 365 x 5 #> # Groups: year, month [?] #> year month day first_dep last_dep #> <int> <int> <int> <int> <int> #> 1 2013 1 1 517 2356 #> 2 2013 1 2 42 2354 #> 3 2013 1 3 32 2349 #> 4 2013 1 4 25 2358 #> 5 2013 1 5 14 2357 #> 6 2013 1 6 16 2355 #> # ... with 359 more rows
These functions are complementary to filtering on ranks. Filtering gives you all variables, with each observation in a separate row:
not_cancelled %>% group_by(year, month, day) %>% mutate(r = min_rank(desc(dep_time))) %>% filter(r %in% range(r)) #> # A tibble: 770 x 20 #> # Groups: year, month, day [365] #> year month day dep_time sched_dep_time dep_delay arr_time #> <int> <int> <int> <int> <int> <dbl> <int> #> 1 2013 1 1 517 515 2. 830 #> 2 2013 1 1 2356 2359 -3. 425 #> 3 2013 1 2 42 2359 43. 518 #> 4 2013 1 2 2354 2359 -5. 413 #> 5 2013 1 3 32 2359 33. 504 #> 6 2013 1 3 2349 2359 -10. 434 #> # ... with 764 more rows, and 13 more variables: sched_arr_time <int>, #> # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>, #> # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>, #> # minute <dbl>, time_hour <dttm>, r <int>
Counts: You’ve seen
n()
, which takes no arguments, and returns the size of the current group. To count the number of non-missing values, usesum(!is.na(x))
. To count the number of distinct (unique) values, usen_distinct(x)
.# Which destinations have the most carriers? not_cancelled %>% group_by(dest) %>% summarise(carriers = n_distinct(carrier)) %>% arrange(desc(carriers)) #> # A tibble: 104 x 2 #> dest carriers #> <chr> <int> #> 1 ATL 7 #> 2 BOS 7 #> 3 CLT 7 #> 4 ORD 7 #> 5 TPA 7 #> 6 AUS 6 #> # ... with 98 more rows
Counts are so useful that dplyr provides a simple helper if all you want is a count:
not_cancelled %>% count(dest) #> # A tibble: 104 x 2 #> dest n #> <chr> <int> #> 1 ABQ 254 #> 2 ACK 264 #> 3 ALB 418 #> 4 ANC 8 #> 5 ATL 16837 #> 6 AUS 2411 #> # ... with 98 more rows
You can optionally provide a weight variable. For example, you could use this to “count” (sum) the total number of miles a plane flew:
not_cancelled %>% count(tailnum, wt = distance) #> # A tibble: 4,037 x 2 #> tailnum n #> <chr> <dbl> #> 1 D942DN 3418. #> 2 N0EGMQ 239143. #> 3 N10156 109664. #> 4 N102UW 25722. #> 5 N103US 24619. #> 6 N104UW 24616. #> # ... with 4,031 more rows
Counts and proportions of logical values:
sum(x > 10)
,mean(y == 0)
. When used with numeric functions,TRUE
is converted to 1 andFALSE
to 0. This makessum()
andmean()
very useful:sum(x)
gives the number ofTRUE
s inx
, andmean(x)
gives the proportion.# How many flights left before 5am? (these usually indicate delayed # flights from the previous day) not_cancelled %>% group_by(year, month, day) %>% summarise(n_early = sum(dep_time < 500)) #> # A tibble: 365 x 4 #> # Groups: year, month [?] #> year month day n_early #> <int> <int> <int> <int> #> 1 2013 1 1 0 #> 2 2013 1 2 3 #> 3 2013 1 3 4 #> 4 2013 1 4 3 #> 5 2013 1 5 3 #> 6 2013 1 6 2 #> # ... with 359 more rows # What proportion of flights are delayed by more than an hour? not_cancelled %>% group_by(year, month, day) %>% summarise(hour_perc = mean(arr_delay > 60)) #> # A tibble: 365 x 4 #> # Groups: year, month [?] #> year month day hour_perc #> <int> <int> <int> <dbl> #> 1 2013 1 1 0.0722 #> 2 2013 1 2 0.0851 #> 3 2013 1 3 0.0567 #> 4 2013 1 4 0.0396 #> 5 2013 1 5 0.0349 #> 6 2013 1 6 0.0470 #> # ... with 359 more rows
13.6.5 Grouping by multiple variables
When you group by multiple variables, each summary peels off one level of the grouping. That makes it easy to progressively roll up a dataset:
daily <- group_by(flights, year, month, day)
(per_day <- summarise(daily, flights = n()))
#> # A tibble: 365 x 4
#> # Groups: year, month [?]
#> year month day flights
#> <int> <int> <int> <int>
#> 1 2013 1 1 842
#> 2 2013 1 2 943
#> 3 2013 1 3 914
#> 4 2013 1 4 915
#> 5 2013 1 5 720
#> 6 2013 1 6 832
#> # ... with 359 more rows
(per_month <- summarise(per_day, flights = sum(flights)))
#> # A tibble: 12 x 3
#> # Groups: year [?]
#> year month flights
#> <int> <int> <int>
#> 1 2013 1 27004
#> 2 2013 2 24951
#> 3 2013 3 28834
#> 4 2013 4 28330
#> 5 2013 5 28796
#> 6 2013 6 28243
#> # ... with 6 more rows
(per_year <- summarise(per_month, flights = sum(flights)))
#> # A tibble: 1 x 2
#> year flights
#> <int> <int>
#> 1 2013 336776
Be careful when progressively rolling up summaries: it’s OK for sums and counts, but you need to think about weighting means and variances, and it’s not possible to do it exactly for rank-based statistics like the median. In other words, the sum of groupwise sums is the overall sum, but the median of groupwise medians is not the overall median.
13.6.6 Ungrouping
If you need to remove grouping, and return to operations on ungrouped data, use ungroup()
.
daily %>%
ungroup() %>% # no longer grouped by date
summarise(flights = n()) # all flights
#> # A tibble: 1 x 1
#> flights
#> <int>
#> 1 336776
13.6.7 Exercises
Brainstorm at least 5 different ways to assess the typical delay characteristics of a group of flights. Consider the following scenarios:
A flight is 15 minutes early 50% of the time, and 15 minutes late 50% of the time.
A flight is always 10 minutes late.
A flight is 30 minutes early 50% of the time, and 30 minutes late 50% of the time.
99% of the time a flight is on time. 1% of the time it’s 2 hours late.
Which is more important: arrival delay or departure delay?
Come up with another approach that will give you the same output as
not_cancelled %>% count(dest)
andnot_cancelled %>% count(tailnum, wt = distance)
(without usingcount()
).Our definition of cancelled flights (
is.na(dep_delay) | is.na(arr_delay)
) is slightly suboptimal. Why? Which is the most important column?Look at the number of cancelled flights per day. Is there a pattern? Is the proportion of cancelled flights related to the average delay?
Which carrier has the worst delays? Challenge: can you disentangle the effects of bad airports vs. bad carriers? Why/why not? (Hint: think about
flights %>% group_by(carrier, dest) %>% summarise(n())
)What does the
sort
argument tocount()
do. When might you use it?
13.7 Grouped mutates (and filters)
Grouping is most useful in conjunction with summarise()
, but you can also do convenient operations with mutate()
and filter()
:
Find the worst members of each group:
flights_sml %>% group_by(year, month, day) %>% filter(rank(desc(arr_delay)) < 10) #> # A tibble: 3,306 x 7 #> # Groups: year, month, day [365] #> year month day dep_delay arr_delay distance air_time #> <int> <int> <int> <dbl> <dbl> <dbl> <dbl> #> 1 2013 1 1 853. 851. 184. 41.  46. #> 4 2013 1 1 157. 174. 213. 60.  I generally avoid them except for quick and dirty manipulations: otherwise it’s hard to check that you’ve done the manipulation correctly.
Functions that work most naturally in grouped mutates and filters are known as window functions (vs. the summary functions used for summaries). You can learn more about useful window functions in the corresponding vignette: vignette("window-functions")
.
13.7.1 Exercises
Refer back to the lists of useful mutate and filtering functions. Describe how each operation changes when you combine it with grouping.
Which plane (
tailnum
) has the worst on-time record?What time of day should you fly if you want to avoid delays as much as possible?
For each destination, compute the total minutes of delay. For each, flight, compute the proportion of the total delay for its destination.
Delays are typically temporally correlated: even once the problem that caused the initial delay has been resolved, later flights are delayed to allow earlier flights to leave. Using
lag()
explore how the delay of a flight is related to the delay of the immediately preceding flight.Look at each destination. Can you find flights that are suspiciously fast? (i.e. flights that represent a potential data entry error). Compute the air time a flight relative to the shortest flight to that destination. Which flights were most delayed in the air?
Find all destinations that are flown by at least two carriers. Use that information to rank the carriers.
For each plane, count the number of flights before the first delay of greater than 1 hour.


Temporary -
Static variable in class eu.newsreader.eventcoreference.objects.Sem
Temporaries are properties that only hold during a certain Time, which is
either indicated with the sem:hasTime property to an instance of sem:Time,
or with the sem:hasTimeStamp property to some timestamp(s).
type -
Static variable in class eu.newsreader.eventcoreference.objects.Sem
Type is the super property of the properties that are used to indicate the
type of a Core instance, eventType, actorType, placeType, timeType; and of
roleType.
Type -
Static variable in class eu.newsreader.eventcoreference.objects.Sem
The SEM Type class contains all types of Core instances.


returns all the credentials needed to access this queue for App Engine, it’s all the job data, since we may have to upload the app
# File ../Neptune/lib/task_queues/task_queue_google_app_engine.rb, line 113 def get_creds() return @credentials end
# File ../Neptune/lib/task_queues/task_queue_google_app_engine.rb, line 146 def is_pull_queue_app_running?() host =<EMAIL_ADDRESS>begin NeptuneManager.log("trying a GET on /supportspullqueue") result = GoogleAppEnginePullQueue.get("#{host}/supportspullqueue") NeptuneManager.log("#{host} returned [#{result}]") if result == true NeptuneManager.log("the currently running app supports pull queues") return true else NeptuneManager.log("the currently running app does not support pull queues") return false end rescue NoMethodError # if the host is down or doesn't have the right route NeptuneManager.log("#{host} did not return a 200, so returning false") return false end end
# File ../Neptune/lib/task_queues/task_queue_google_app_engine.rb, line 95 def pop() host =<EMAIL_ADDRESS>begin NeptuneManager.log("trying a GET on /task") task = GoogleAppEnginePullQueue.get("#{host}/task") NeptuneManager.log("#{host} is up and returned a task with payload: #{task}") return task rescue NoMethodError # if the host is down NeptuneManager.log("#{host} is down, returning nil") return nil end end
stores a hash in the queue for later processing, by converting it to JSON and pushing it to our app engine app
# File ../Neptune/lib/task_queues/task_queue_google_app_engine.rb, line 71 def push(item) if item.class != Hash raise BadConfigurationException.new end payload = JSON.dump(item) host =<EMAIL_ADDRESS>begin NeptuneManager.log("trying a PUT on /task") # PUT requests require a Content-Length header in App Engine, so # set an empty body and httparty will add that header in result = GoogleAppEnginePullQueue.put("#{host}/task", :body => '', :query => {:payload => payload}) if result['ok'] NeptuneManager.log("#{host} is up - push succeeded") else NeptuneManager.log("#{host} is up but push did not succeed, " + "returning #{result.inspect}") end rescue NoMethodError # if the host is down NeptuneManager.log("#{host} is down, returning on push") end end
returns the number of messages in the queue Our App Engine app has a special route just for this, with a counter that is associated with the size of the queue
# File ../Neptune/lib/task_queues/task_queue_google_app_engine.rb, line 121 def size() host =<EMAIL_ADDRESS>begin NeptuneManager.log("trying a GET on /size") result = GoogleAppEnginePullQueue.get("#{host}/size") size = result['size'] NeptuneManager.log("#{host} is up and returned a size of #{size}") return size rescue NoMethodError # if the host is down NeptuneManager.log("#{host} is down, returning 0") return 0 end end
returns a string representation of this queue, which is just the queue name and the credentials TODO(cgb): this seems generic enough to move into TaskQueue, so do so
# File ../Neptune/lib/task_queues/task_queue_google_app_engine.rb, line 141 def to_s() return "Queue type: Google App Engine, app id: #{@appid}, " + "appcfg_cookies: #{@appcfg_cookies}" end
# File ../Neptune/lib/task_queues/task_queue_google_app_engine.rb, line 166 def wait_for_app_to_start_serving() host =<EMAIL_ADDRESS>NeptuneManager.log("waiting for app to start serving") loop { if is_pull_queue_app_running? NeptuneManager.log("app is now serving!") break else NeptuneManager.log("app is not yet serving") sleep(10) end } end
creates a new App Engine queue via the gtaskqueue executable
# File ../Neptune/lib/task_queues/task_queue_google_app_engine.rb, line 43 def initialize(credentials) if credentials.class != Hash raise BadConfigurationException.new end @credentials = credentials if<EMAIL_ADDRESS>raise BadConfigurationException.new end @appid = credentials['@appid'] if<EMAIL_ADDRESS>raise BadConfigurationException.new end @appcfg_cookies = credentials['@appcfg_cookies'] # The App Engine pull queue interface we use is an App Engine app, so make # sure that the app is already in App Engine, and if not, upload it. if !is_pull_queue_app_running? app_location = '/root/appscale/AppServer/demos/pullqueueapp' TaskEngineGoogleAppEngine.upload_app(credentials, app_location) wait_for_app_to_start_serving end end


VERY BAD THINGS (Polygram) Starring: Christian Slater, Cameron Diaz, Jon Favreau, Daniel Stern, Jeremy Piven, Jeanne Tripplehorn, Leland Orser. Screenplay: Peter Berg. Producers: Michael Schiffer, Diane Nabatoff and Cindy Cowan. Director: Peter Berg.  Reviewed by Scott Renshaw.
The film's title offers too easy and convenient a bon mot. Expect cadres of critics to be trotting out innumerable variations on "VERY BAD THINGS _is_ one of them," though it's both a cheap analysis and an inaccurate one. VERY BAD THINGS isn't a very bad movie. It's a very good idea approached from entirely the wrong perspective.
The idea involves that most cynically macho of American matrimonial institutions, the bachelor party. The groom-to-be is mild-mannered Los Angeles cubicle-dweller Kyle Fisher (Jon Favreau); his fiancee is Laura Garrety (Cameron Diaz), a woman only slightly obsessed with her nuptial details. The boys' night out is to be an overnight in Las Vegas including Kyle's four best buddies: mile-a-minute real estate hustler Robert Boyd (Christian Slater); family man Adam Berkow (Daniel Stern); Adam's edgy brother Michael (Jeremy Piven); and reticent mechanic Charles Moore (Leland Orser). Though the party begins with the expected drinking, drugs and bare breasts, it ends with the unexpected problem of a dead prostitute on the floor of their hotel room bathroom. That sets off a grotesque chain of events, as the friends' attempts to cover up one death lead to one death after another.
Beneath the macabre, over-the-top situations surrounding the deaths in VERY BAD THINGS is a savage satire of stereotypical gender responses to impending marriage. Kyle watches in a passive daze as Laura orchestrates the ceremony from overture to finale; Laura plows over Kyle's every hesitation with a fiercely muttered "Do you love me?", focusing so completely on _getting_ married she seems oblivious to the idea of _being_ married. The horrific events in Las Vegas become a steroid-injected version of more mundane bachelor party naughtiness, with the bachelor turning the "very bad thing" into both a source of guilt and a potential excuse for getting out of the marriage he fears. To Laura, it's all just one more logistical glitch to be overcome, with Diaz turning in a frighteningly funny vision of a woman whose wedding _will not_ be spoiled by anything.
Diaz nails the proper tone so completely (particularly in the film's final act) that you may suddenly realize what's been so wrong with the rest of the film. While Diaz plays Laura entirely for absurd laughs, the male cast members range from one extreme to another. Slater plays Boyd as too sinister to be funny, Stern and Piven play the brothers Berkow as too genuinely tormented to be funny, and Favreau plays Kyle as too dull to be funny. For much of VERY BAD THINGS, the script by Peter Berg (better known from his role on "Chicago Hope") is simply a plot in motion, pulling the characters along with it. At times there's not even exaggerated humanity in the reactions of the characters, the kind that would have given a more humorous spin to more scenes. For a film that could have and should have been a dark comedy, VERY BAD THINGS is too much like the menacing morality play of SHALLOW GRAVE.
That kind of tone is going to alienate a lot of viewers, particularly as the situations grow increasingly extreme and gruesome. Though VERY BAD THINGS ends with a hilariously horrifying subversion of suburban domestic bliss, it doesn't given you enough reason to stick around for it. Comedy can still work when it's caustic, but at some point a film-maker needs to understand that he has to keep an audience laughing at material this dark to keep them from walking away. VERY BAD THINGS is a great opportunity wasted, and that's the _real_ very bad thing.
On the Renshaw scale of 0 to 10 final flings: 5.
Visit Scott Renshaw's MoviePage http://www.inconnect.com/~renshaw/ *** Subscribe to receive new reviews directly by email! See the MoviePage for details, or reply to this message with subject line "Subscribe".
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


{% assign meeting_date = meeting.date | date: "6:30 PM-8:30 PM %A %-dth %B %Y" | replace: "1th", "1st" | replace: "2th", "2nd" | replace: "3th", "3rd" | replace: "11st", "11th" | replace: "12nd", "12th" | replace: "13rd", "13th" %}


Magnovite brings you a great opportunity to show your creativity. Face painting is an exciting, fun event bringing a splash of colour and a sprinkle of glitter to brighten up the atmosphere. Painting a canvas is tricky, a face even more.
Open Mic invites you to savour your thoughts, express your feelings and bring your beliefs to life with words through the medium of rhyme.
Reduce-Reuse-Recycle your way through this challenging design event to turn one man’s waste into another’s haute couture. Creativity never goes out of style.  Your creation should be a reflection of your own self!
You’ve got the moves and we’ve got the stage. What sets you apart from the others will be your devotion to delivering the best performance. Put on your dancing shoes and come wow us with your moves.
Just-A-Minute is an all round-fun event that is all about the control of the mind over the mouth. Can you make it through sixty seconds of non-stop talking without hesitation, repetition, or deviation?
Music plays an integral role in the lives of Indians. Indian music is magically enchanted, sophisticated with defined melodies that are soothing to the ear. Participants have to express the cultural heritage of India. It is a platform for healthy competition for all those who are passionate about music.
Battle of Bands is an event where you face-off against musical acts from across the country in pursuit of who gets to call themselves “The Best Band”. There is a lot is at stake here with individual awards, a spectacular audience to perform for and the opportunity to turn up the heat and find out who walks away with the bragging rights.
Be it classical or folk, Indian dance forms have never failed to mesmerise us. Dance to express not to impress. Express your feelings via dance on the stage, matching your heartbeat to the rhythm and blow the audience away. The better you express the bigger the awards you win.
Street play is an event to bring out your theatrical side. Organized in an open air venue, participants compete to attract the attention of the audience with their acting skills and flair for the dramatic. Have you got what it takes? Then claim the stage and tell your story.


Contributing¶
We appreciate all kinds of help, so thank you!
Contributing to Qiskit Terra¶
You can contribute in many ways to this project.
Issue reporting¶
This is a good point to start, when you find a problem please add it to the issue tracker. The ideal report should include the steps to reproduce it.
Doubts solving¶
To help less advanced users is another wonderful way to start. You can
help us close some opened issues. This kind of tickets should be
labeled as question
.
Improvement proposal¶
If you have an idea for a new feature please open a ticket labeled as
enhancement
. If you could also add a piece of code with the idea
or a partial implementation it would be awesome.
Contributor License Agreement¶
We’d love to accept your code! Before we can, we have to get a few legal requirements sorted out. By signing a contributor license agreement (CLA), we ensure that the community is free to use your contributions.
When you contribute to the Qiskit Terra project with a new pull request, a bot will evaluate whether you have signed the CLA. If required, the bot will comment on the pull request, including a link to accept the agreement. The individual CLA document is available for review as a PDF.
NOTE: If you work for a company that wants to allow you to contribute your work, then you’ll need to sign a corporate CLA and email it to us at<EMAIL_ADDRESS>Good first contributions¶
You are welcome to contribute wherever in the code you want to, of course, but we recommend taking a look at the “Good first contribution” label into the issues and pick one. We would love to mentor you!
Doc¶
Review the parts of the documentation regarding the new changes and update it if it’s needed.
Pull requests¶
We use GitHub pull requests to accept the contributions.
A friendly reminder! We’d love to have a previous discussion about the best way to implement the feature/bug you are contributing with. This is a good way to improve code quality in our beloved Qiskit!, so remember to file a new Issue before starting to code for a solution.
So after having discussed the best way to land your changes into the codebase, you are ready to start coding (yay!). We have two options here:
- You think your implementation doesn’t introduce a lot of code, right?. Ok, no problem, you are all set to create the PR once you have finished coding. We are waiting for it!
- Your implementation does introduce many things in the codebase. That sounds great! Thanks!. In this case you can start coding and create a PR with the word: [WIP] as a prefix of the description. This means “Work In Progress”, and allow reviewers to make micro reviews from time to time without waiting for the big and final solution… otherwise, it would make reviewing and coming changes pretty difficult to accomplish. The reviewer will remove the [WIP] prefix from the description once the PR is ready to merge.
Pull request checklist¶
When submitting a pull request and you feel it is ready for review, please double check that:
- the code follows the code style of the project. For convenience, you can
execute
make style
andmake lint
locally, which will print potential style warnings and fixes. - the documentation has been updated accordingly. In particular, if a function or class has been modified during the PR, please update the docstring accordingly.
- your contribution passes the existing tests, and if developing a new feature, that you have added new tests that cover those changes.
- you add a new line to the
CHANGELOG.rst
file, in theUNRELEASED
section, with the title of your pull request and its identifier (for example, “Replace OldComponent with FluxCapacitor (#123)
”.
Commit messages¶
Please follow the next rules for the commit messages:
- It should include a reference to the issue ID in the first line of the commit, and a brief description of the issue, so everybody knows what this ID actually refers to without wasting to much time on following the link to the issue.
- It should provide enough information for a reviewer to understand the changes and their relation to the rest of the code.
A good example:
Issue #190: Short summary of the issue
* One of the important changes
* Another important change
Installing Qiskit Terra from source¶
This section include some tips that will help you install and push source code.
Note
We recommend using Python virtual environments to cleanly separate Qiskit from other applications and improve your experience.
Setup with an environment¶
The simplest way to use environments is by using Anaconda
conda create -y -n QiskitDevenv python=3
source activate QiskitDevenv
For the python code, we need some libraries that can be installed in this way:
cd qiskit-terra
pip install -r requirements.txt
pip install -r requirements-dev.txt
To get the examples working try
$ pip install -e .
and then you can run them with
$ python examples/python/using_qiskit_terra_level_0.py
We recommend that after setting up Terra you set up Aer to get more advanced simulators.
Building the legacy simulators¶
Note
These will become obsolete in Terra 0.8
Dependencies¶
Our build system is based on CMake, so we need to have CMake 3.5 or higher installed. As we will deal with languages that build native binaries, we will need to have installed any of the supported CMake build tools.
On Linux and Mac, we recommend installing GNU g++ 6.1 or higher, on Windows we only support MinGW64 at the moment. Note that a prerequiste for the C++ toolchain is that C++14 must be supported.
Building¶
The preferred way CMake is meant to be used, is by setting up an “out of source” build. So in order to build our native code, we have to follow these steps:
Linux and Mac
qiskit-terra$ mkdir out
qiskit-terra$ cd out
qiskit-terra/out$ cmake ..
qiskit-terra/out$ make
Windows
C:\..\> mkdir out
C:\..\> cd out
C:\..\out> cmake -DUSER_LIB_PATH=C:\path\to\mingw64\lib\libpthreads.a -G "MinGW Makefiles" ..
C:\..\out> make
As you can see, the Windows cmake command invocation is slightly different from the Linux and Mac version, this is because we need to provide CMake with some more info about where to find libphreads.a for later building. Furthermore, we are forcing CMake to generate MingGW makefiles, because we don’t support other toolchain at the moment.
Useful CMake flags¶
There are some useful flags that can be set during cmake command invocation and
will help you change some default behavior. To make use of them, you just need to
pass them right after -D
cmake argument. Example:
.. code:
qiskit-terra/out$ cmake -DUSEFUL_FLAG=Value ..
Flags:
- USER_LIB_PATH
- This flag tells CMake to look for libraries that are needed by some of the native
components to be built, but they are not in a common place where CMake could find
it automatically.
Values: An absolute path with file included.
Default: No value.
Example:
cmake -DUSER_LIB_PATH=C:\path\to\mingw64\lib\libpthreads.a ..
- STATIC_LINKING
- Tells the build system whether to create static versions of the programs being built or not.
Notes: On MacOS static linking is not fully working for all versions of GNU G++/Clang
compilers, so enable this flag in this platform could cause errors.
Values: True|False
Default: False
Example:
cmake -DSTATIC_LINKING=True ..
- CMAKE_BUILD_TYPE
- Tells the build system to create executables/libraries for debugging purposes
or highly optimized binaries ready for distribution.
Values: Debug|Release
Default: “Release”
Example:
cmake -DCMAKE_BUILD_TYPE="Debug" ..
- ENABLE_TARGETS_NON_PYTHON
- We can enable or disable non-python code generation by setting this flag to True or False
respectively. This is mostly used in our CI systems so they can launch some fast tests
for the Python code (which is currently a majority).
Values: True|False
Default: True
Example:
cmake -DENABLE_TARGETS_NON_PYTHON=True ..
- ENABLE_TARGETS_QA
- We can enable or disable QA stuff (lintering, styling and testing) by setting this flag to
True or False respectively. This is mostly used in our CI systems so they can run light
stages pretty fast, and fail fast if they found any issues within the code.
Values: True|False
Default: True
Example:
cmake -DENABLE_TARGETS_QA=True ..
- WHEEL_TAG
- This is used to force platform specific tag name generation when creating wheels package
for Pypi.
Values: “-pWhateverTagName”
Default: No value.
Example:
cmake -DWHEEL_TAG="-pmanylinux1_x86_64" ..
Test¶
New features often imply changes in the existent tests or new ones are needed. Once they’re updated/added run this be sure they keep passing.
For executing the tests, a make test
target is available.
The execution of the tests (both via the make target and during manual invocation)
takes into account the LOG_LEVEL
environment variable. If present, a .log
file will be created on the test directory with the output of the log calls, which
will also be printed to stdout. You can adjust the verbosity via the content
of that variable, for example:
Linux and Mac:
$ cd out
out$ LOG_LEVEL="DEBUG" ARGS="-V" make test
Windows:
$ cd out
C:\..\out> set LOG_LEVEL="DEBUG"
C:\..\out> set ARGS="-V"
C:\..\out> make test
For executing a simple python test manually, we don’t need to change the directory
to out
, just run this command:
Linux and Mac:
$ LOG_LEVEL=INFO python -m unittest test/python/circuit/test_circuit_operations.py
Windows:
C:\..\> set LOG_LEVEL="INFO"
C:\..\> python -m unittest test/python/circuit/test_circuit_operations.py
Note many of the tests will not be executed unless you have setup an IBMQ account. To set this up please go to this page and register an account.
By default, and if there is no user credentials available, the tests that
require online access are run with recorded (mocked) information. This is, the
remote requests are replayed from a test/cassettes
and not real HTTP
requests is generated. If user credentials are found, in that cases it use them
to make the network requests.
How and which tests are executed is controlled by a environment variable
QISKIT_TESTS
. The options are (where uc_available = True
if the user
credentials are available, and False
otherwise):
| Option | Description | Default | If True , forces |
|---|---|---|---|
skip_online |
Skips tests that require remote requests (also, no mocked information is used). Does not require user credentials. | False |
rec = False |
mock_online |
It runs the online tests using mocked information. Does not require user credentials. | not uc_available |
skip_online = False |
run_slow |
It runs tests tagged as slow. | False |
|
rec |
It records the remote requests. It requires user credentials. | False |
skip_online = False
run_slow = False |
It is possible to provide more than one option separated with commas.
The order of precedence in the options is right to left. For example,
QISKIT_TESTS=skip_online,rec
will set the options as
skip_online == False
and rec == True
.
Style guide¶
Please submit clean code and please make effort to follow existing conventions in order to keep it as readable as possible. We use Pylint and PEP 8 style guide: to ensure your changes respect the style guidelines, run the next commands:
All platforms:
$> cd out
out$> make lint
out$> make style
Documentation¶
The documentation Qiskit Terra is in the doc
directory. The
documentation is auto-generated from python
docstrings using Sphinx for generating the
documentation. Please follow Google’s Python Style
Guide
for docstrings. A good example of the style can also be found with
sphinx’s napolean converter
documentation.
You can see the rendered documentation for the stable version of Qiskit Terra at
the landing page.
To generate the documentation, we need to invoke CMake first in order to generate all specific files for our current platform.
See the previous Building section for details on how to run CMake. Once CMake is invoked, all configuration files are in place, so we can build the documentation running this command:
All platforms:
$> make doc
Development cycle¶
Our development cycle is straightforward, use the project boards in Github for project management and use milestones for releases. The features that we want to include in these releases will be tagged and discussed in the project boards. Whenever a new release is close to be launched, we’ll announce it and detail what has changed since the latest version in our Release Notes and Changelog. The channels we’ll use to announce new releases are still being discussed, but for now, you can follow us on Twitter!
Branch model¶
There are two main branches in the repository:
master
- This is the development branch.
- Next release is going to be developed here. For example, if the current latest release version is r1.0.3, the master branch version will point to r1.1.0 (or r2.0.0).
- You should expect this branch to be updated very frequently.
- Even though we are always doing our best to not push code that breaks things, is more likely to eventually push code that breaks something… we will fix it ASAP, promise :).
- This should not be considered as a stable branch to use in production environments.
- The API of Qiskit could change without prior notice.
stable
- This is our stable release branch.
- It’s always synchronized with the latest distributed package, as for now, the package you can download from pip.
- The code in this branch is well tested and should be free of errors (unfortunately sometimes it’s not).
- This is a stable branch (as the name suggest), meaning that you can expect stable software ready for production environments.
- All the tags from the release versions are created from this branch.
Release cycle¶
From time to time, we will release brand new versions of Qiskit Terra. These are well-tested versions of the software.
When the time for a new release has come, we will:
- Merge the
master
branch with thestable
branch. - Create a new tag with the version number in the
stable
branch. - Crate and distribute the pip package.
- Change the
master
version to the next release version. - Announce the new version to the world!
The stable
branch should only receive changes in the form of bug fixes, so the
third version number (the maintenance number: [major].[minor].[maintenance])
will increase on every new change.


Mission
Oak Park Township assists:
- Senior citizens, youth and their families, people with disabilities, and financially struggling residents to fully participate in community life by providing information, supportive services, advocacy, and referrals;
- Partner agencies with funding for youth, mental health, and other human services; and
- All residents with property tax information, advocacy and services, voter registration, public transportation assistance, and community information
Governance
An independently elected Township Board - a Supervisor and four Trustees, governs Oak Park Township. Reporting to the Supervisor and working under the Supervisor's delegated authority, a full-time Township Manager administers the day-to-day operation of the Township and is responsible for all staff activities. Board-appointed Senior Services and Youth Services advisory committees assist the Board in its duties. The Board also appoints a Community Mental Health Board (CMHB) that has full control of expenditures from the Mental Health Fund within the Township Board approved budget.
Finances
Each Township Fiscal Year (April 1 - March 31), the Township Board approves a property tax levy (limited by State Tax Cap legislation), passes a budget and approves all payments of Township funds. Three funds are under Board levy, budget and payment control
- The Town Fund for the expenses of general government, Youth and Senior services and Assessor and Clerk offices
- The General Assistance Fund for the administration of the Gemeral Assistance office and the awarding of grants to citizens meeting well-defined need criteria regarding assets and income
- The Mental Health Fund for the administration of the Community Mental Health Board (CMHB) and to fund the CMHB's prograns and grants for mental health providers serving Oak Park.


Applying auto-width
on the <vaadin-grid-column>
will set the width of the column
to automatically fit the content.
The column width will be calculated and set only once based on the contents of the initially rendered cells.
You can manually trigger the auto sizing behavior again by calling grid.recalculateColumnWidths()
.
By default the column might still expand if there is more space available, unless you also set flex-grow="0"
.
Note: If all the columns have flex-grow="0"
there might be empty space after the last column
if the grid is wider than what is needed to fit all the columns. You probably want to leave at least one column
without flex-grow="0"
so that column will take all the remaining space that is left.
Note: When using column renderer functions, those should be set before setting the grid items
or
dataProvider
. Otherwise the initial column widths may be calculated before the renderers have been run and
the widths of the renderer columns might be wrong initially.
Columns can be grouped together with <vaadin-grid-column-group>
elements.
Grouped columns can have a common header and footer cells.
Note: Remember to define explicit bounds for images in order to avoid glitches once an image has fully loaded.
NOTE: You must explicitly import the vaadin-grid-column-group
in order to use it.
Applying frozen
on a column will make it frozen. It can be
also be applied to a column group to freeze all of its child columns.
Note: Columns are frozen in-place, freeze columns from left to right for a consistent outcome.
Applying hidden
on a column or column group will hide it.
Applying column-reordering-allowed
on the grid will make all
columns reorderable. Drag and drop the columns by headers to reorder.
Applying resizable
on a column will make it resizable.
Note: When using column groups, columns can be reordered only within their parent group.


Patrick Touchette
Software Developer
Python
Grahpical User Interfaces
Web Scraping
Data Analysis and Visualisation
This is a simple ERP (Enterprise resource planning) software prototype, built in python and Tkinter. Reportlab is also used to create reports and matplotlib is used to create charts.
Creates an interlinear book from a Russian text. The English translation is displayed above the Russian word in HTML ruby notation. Uses python 3.6
Courses, tutorials, blog posts, etc...
Passionate about code, I do it as a hobby which I hope to turn into a full time job
I enjoy making things and seeing results
Clear and concise