Just read with interest the revised blog post about backing up the databases/folders, and wondered if anyone had put all this together as a php script that could be run, say as a Cron job? So it would halt sql & raspberry pi_run, back up sql, back up the folders, then restart the services again. Paul
»
Re: Backup/migration of emoncms data [SOLVED]
Definitely interested in this. I have had to start from scratch several times after trying to diagnose a faulty HD.
Re: Backup/migration of emoncms data [SOLVED]
You should be able to do this ok with the newly updated backup script, it doesnt require stopping the services on the remote server. It recalculates the feeds npoints metadata from the download size rather than copying meta data that becomes out of sync with the feed data. So just running backup.php from cron should be ok.
Re: Backup/migration of emoncms data [SOLVED]
I've tried running the script on the same Raspberry Pi which hosts my emoncms, but I'm getting errors as per below.
I've created a backup folder with subdirectories, currently chmod to 777, and the script/libraries chmod to 755.
php backup.php
22 Emoncms.org feeds found
PHP Notice: Undefined variable: id in /home/pi/myscripts/emoncms/usefulscripts/backup/backup.php on line 90
PHP Notice: Undefined variable: server in /home/pi/myscripts/emoncms/usefulscripts/backup/backup.php on line 90
PHPTIMESTORE:
PHP Warning: file_get_contents(/feed/getmeta.json?apikey=57ee7xxxxxxmyapikeyxxxxxb5e&id=): failed to open stream: No such file or directory in /home/pi/myscripts/emoncms/usefulscripts/backup/lib/phptimestore.php on line 7
PHP Notice: Trying to get property of non-object in /home/pi/myscripts/emoncms/usefulscripts/backup/lib/phptimestore.php on line 55
PHP Notice: Trying to get property of non-object in /home/pi/myscripts/emoncms/usefulscripts/backup/lib/phptimestore.php on line 55
--layer: 0 Failed to access remote server
PHP Notice: Undefined variable: id in /home/pi/myscripts/emoncms/usefulscripts/backup/backup.php on line 90
PHP Notice: Undefined variable: server in /home/pi/myscripts/emoncms/usefulscripts/backup/backup.php on line 90
+ lots more similar errors
Re: Backup/migration of emoncms data [SOLVED]
Ah sorry Paul, I missed a crucial bit for phptimestore, should be fixed now.
Re: Backup/migration of emoncms data [SOLVED]
Thanks, that has helped, I just need to sort out a corrupt Timestore feed now.
PHPTIMESTORE: 11
PHP Notice: Trying to get property of non-object in /home/pi/myscripts/emoncms/usefulscripts/backup/lib/phptimestore.php on line 55
PHP Notice: Trying to get property of non-object in /home/pi/myscripts/emoncms/usefulscripts/backup/lib/phptimestore.php on line 55
--layer: 0 Timestore feed 11 corrupt
...presumably, the next step is the 'integrity check' and 'data recovery'.
Paul
Re: Backup/migration of emoncms data [SOLVED]
...integrity check shows;
CHECKING INTEGRITY OF phptimestore META DATA:
Feed 11 [npoints meta file does not exist] [07:07:2014 22:50]
Feed 12 [npoints meta file does not exist] [07:07:2014 22:50]
Feed 13 [npoints meta file does not exist] [07:07:2014 22:50]
Feed 17 [npoints meta file does not exist] [07:07:2014 22:03]
Feed 25 [npoints meta file does not exist] [07:07:2014 22:50]
Feed 36 [npoints meta file does not exist] [07:07:2014 22:50]
Feed 70 [npoints meta file does not exist] [07:07:2014 22:49]
Feed 90 [npoints meta file does not exist] [07:07:2014 22:44]
Feed 94 [npoints meta file does not exist] [07:07:2014 22:50]
Feed 97 [npoints meta file does not exist] [07:07:2014 22:50]
Feed 98 [npoints meta file does not exist] [07:07:2014 22:50]
Feed 99 [npoints meta file does not exist] [07:07:2014 22:50]
Feed 101 [npoints meta file does not exist] [07:07:2014 22:50]
Feed 102 [npoints meta file does not exist] [07:07:2014 22:50]
Error count: 14
Number of feeds: 14
Paul
Re: Backup/migration of emoncms data [SOLVED]
Can you download the feeds from scratch? might be easier, might be work making a copy of the one's you have into another folder
Re: Backup/migration of emoncms data [SOLVED]
...but why does all 14 of my Timestore feeds all appear to have a problem?
They all display OK within emoncms...
Paul
Re: Backup/migration of emoncms data [SOLVED]
Apart from the Timestore issue, it does not appear to be backing up either - see screen shot of my PHPFIWA backup folder after running the script;
Both feed's 120 & 121 should be 4mb in size.
Paul
Re: Backup/migration of emoncms data [SOLVED]
I have also been trying to use the backup script.
My set-up is a RasPi running emoncms v8.0.8 but I am trying to backup to Windows 8.1.
I initially planned to link data to a working Wamp emoncms v8.2.8 configuration but ran into an issue with the backup script and redis (could not find versions of php_redis.ddl and php_redis.pdb that work with php v5.5.12).
Trying to download just the data (by setting $link_to_local_emoncms to false in backup.php) results in the same sort of issues as Paul is having, i.e. corrupt Timestore feeds and failing backup.
Bob
Re: Backup/migration of emoncms data [SOLVED]
@Paul, there should be a 3rd file there with a .dat extension holding the data, so something must be going wrong early on.
What I think has failed is the call to $server."/feed/getmeta.json?apikey=$apikey&id=$id" in lib/phptimestore.php
Il have added a couple of lines to the code to catch this error and print what the server is giving us. I wonder if you can get the latest changes and run it again?
Re: Backup/migration of emoncms data [SOLVED]
With latest changes I get:
39 Emoncms.org feeds found
PHPTIMESTORE: 1
Error in fetching remote meta data, received: false
etc.
PHPTIMESTORE: 41
Error in fetching remote meta data, received: false
PHPFIWA: 43
Cannot open local npoints meta data file
etc.
PHPFIWA: 46
Cannot open local npoints meta data file
Same result from Windows 8 and RasPi also integritycheck.php fails with:
CHECKING INTEGRITY OF phptimestore
META DATA:Feed 1 [npoints meta file does not exist] [08:07:2014 12:39]
They all display OK within emoncms...
Re: Backup/migration of emoncms data [SOLVED]
I should have been more clear, it needs your write apikey, I've added a note to the code. The phpfiwa error may be because you need to create a folder in your target directory for each feed engine type: phpfina, phpfiwa, phptimestore, phptimeseries.
Re: Backup/migration of emoncms data [SOLVED]
The write apikey may have been the issue!
I've updated with Git and just tried it with the read/write apikey, and got exactly the same result - no data backed up. BUT then I deleted the previous data that had been backed-up in the phpfiwa, phpfina etc backup folders, so all of the folders were empty, and ran the script again.
This time success! and over 100mb of data successfully recovered.
Thanks
Paul
Re: Backup/migration of emoncms data [SOLVED]
Just to confirm backup.php now works for me ;-)
One of the other errors I had was down to the backup.php script looking for a folder in my target directory named 'timestore' not 'phptimestore' as stated in the backup instructions.
Now if I could just solve the redis issue with my Windows Wamp set-up...
Thanks
Bob
Re: Backup/migration of emoncms data [SOLVED]
Hi
I have set up a new linux server to act as backup.
I am trying to use backup.php
I think I have edited the file as required but I get the following when I run backup.php:-
ian@ian-N250P /var/www/emoncms/usefulscripts/backup $ php backup.php
PHP Notice: Use of undefined constant _ENABLE_PASSWORD_RESET_ - assumed '_ENABLE_PASSWORD_RESET_' in /var/www/emoncms/settings.php on line 55
22 Emoncms.org feeds found
PHP Notice: Undefined index: datadir in /var/www/emoncms/usefulscripts/backup/backup.php on line 103
Notice: Undefined index: datadir in /var/www/emoncms/usefulscripts/backup/backup.php on line 103
PHPFIWA: 22
--layer: 0 PHP Notice: Undefined variable: feed in /var/www/emoncms/usefulscripts/backup/lib/phpfiwa.php on line 68
Notice: Undefined variable: feed in /var/www/emoncms/usefulscripts/backup/lib/phpfiwa.php on line 68
PHP Notice: Trying to get property of non-object in /var/www/emoncms/usefulscripts/backup/lib/phpfiwa.php on line 68
Notice: Trying to get property of non-object in /var/www/emoncms/usefulscripts/backup/lib/phpfiwa.php on line 68
PHPFiwa feed corrupt
ian@ian-N250P /var/www/emoncms/usefulscripts/backup $
One thing that puzzles me.
in backup.php it states:-
// if you are linking to a local emoncms installation, the settings will be fetched from
// your settings.php file instead.
If I look in settings.php it says
// The default data directory is /var/lib/phpfiwa,phpfina,phptimeseries
Then a few lines down you have:-
'phpfiwa'=>array(
//'datadir'=>'/home/username/emoncmsdata/phpfiwa/'
Is this correct?
Regards
Ian
Re: Backup/migration of emoncms data [SOLVED]
Hello Ian.
To get rid of the _ENABLE_PASSWORD_RESET_ error, set $enable_password_reset to true or false in settings.php.
Thanks for highlighting the datadir error,
I've uncommented the default datadir entries so that they override the defaults set within the engines themselves. Thats likely better as at least now they are accessible to scripts that include settings.php
A quick fix would be to replace the engine defenitions with:
'phpfiwa'=>array(
'datadir'=>'/var/lib/phpfiwa/'
),
'phpfina'=>array(
'datadir'=>'/var/lib/phpfina/'
),
'phptimeseries'=>array(
'datadir'=>'/var/lib/phptimeseries/'
)
Re: Backup/migration of emoncms data [SOLVED]
Hi Trystan
Made changes in settings.php
Now get:-
ian@ian-N250P /var/www/emoncms/usefulscripts/backup $ php backup.php
22 Emoncms.org feeds found
PHPFIWA: 22
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPTIMESERIES: 37
Cannot open local data file
PHPFIWA: 38
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPTIMESERIES: 39
Cannot open local data file
PHPFIWA: 40
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 41
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 42
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 43
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 44
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 45
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 46
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 47
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 48
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 49
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 52
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 55
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 56
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPFIWA: 58
Cannot open local meta data file: Does the phpfiwa folder exist?
PHPTIMESERIES: 59
Cannot open local data file
PHPTIMESERIES: 62
Cannot open local data file
PHPTIMESERIES: 64
Cannot open local data file
PHPTIMESERIES: 65
Cannot open local data file
The folders do exist in \var\lib\. How can I tell where I have made a mistake? I Pulled the latest default.settings.php and used that as the template for my settings.php so I am assuming I have an error in backup.php
Could I have an error in folder permissions? If so how do you check folder permissions from a terminal?
Regards
Ian
Re: Backup/migration of emoncms data [SOLVED]
To check permissions;
$ ls -l
To change directory permissions recursively;
$ sudo find emoncmsdata -type d -exec chmod 755 {} \;
Paul


Please wait, the AMP doc will appear here...
Please wait, the AMP doc will appear here...
Please wait, the AMP doc will appear here...
Please wait, the AMP doc will appear here...
Please wait, the AMP doc will appear here...
Please wait, the AMP doc will appear here...
Please wait, the AMP doc will appear here...
Please wait, the AMP doc will appear here...
Please wait, the AMP doc will appear here...


Before Change
if self.default_graph is None:
error_string = "The attack symbolic graph was not generated."
raise Exception(error_string)
if self.nb_calls_generate > 1:
warnings.warn("Attack was generated symbolically multiple "
"times, using graph defined by first call.")
After Change
// Generate this attack"s graph if it hasn"t been done previously
if not hasattr(self, "_x"):
input_shape = list(X.shape)
input_shape[0] = None
self._x = tf.placeholder(tf.float32, shape=input_shape)
self._x_adv = self.generate(self._x)
// Run symbolic graph without or with true labels
if params["Y"] is None:


Trump’s ambassador to the EU says the president ordered him against his will to work with Rudy Giuliani on Ukraine
by Ashley Collman- Gordon Sondland, the United States Ambassador to the European Union, testified before Congress on Wednesday as part of President Donald Trump's impeachment hearings.
- Sondland said he was forced to work with the president's personal attorney, Rudy Giuliani, on Ukraine matters.
- Follow along our live coverage of the hearing here.
- Visit Business Insider's homepage for more stories.
Gordon Sondland, the United States Ambassador to the European Union, told House impeachment investigators on Wednesday that he was forced to work with the president's personal lawyer, Rudy Giuliani, to pressure the government of Ukraine into a politically-motivated investigation.
Sondland testified that President Donald Trump instructed him to work with Giuliani, as the latter advocated Ukrainian leadership to investigate former Vice President Joe Biden's son's business dealings in the country. He said outgoing energy secretary Rick Perry and former Ukraine envoy Rick Volker were also part of the group.
"Secretary Perry, Ambassador Volker and I worked with Rudy Giuliani on Ukraine matters at the express direction of the president of the united states.
"We did not want to work with Mr. Giuliani.
The impeachment hearings center on President Donald Trump's efforts to pressure Ukraine to launch politically motivated investigations while holding up military aid and a White House meeting.
Sondland's testimony follows a newly revealed phone call between the president and the ambassador, casually discussing the matter.
Calling the president from a cell phone in a Kyiv restaurant, Sondland told the president that Ukrainian President Volodymyr Zelenskiy "loves your a--."
"So, he's gonna do the investigation?" Trump asked in return. Sondland said Zelenskiy was "gonna do it," saying he would do "anything you ask him to."
Sondland took a more serious tone during his testimony on Wednesday, saying he was essentially just following orders, and that he knew that what the president was pushing for was a "quid pro quo." He said he kept several top figures in the administration abreast of the plan.
"Everyone was in the loop," Sondland said. "It was no secret."


Thank you!
Your return is now registered.
A confirmation has been sent to<EMAIL_ADDRESS>What's next?
To make your return experience perfect, we invite you to follow the instructions below
1
Download and print your return label and your packing slip
IMPORTANT: Please note your labels will expire in 5 days
2
Prepare your box
Make sure your box is completely adapted to your items and secure
3
Put your items inside the box
Wrap your items in your box and don't forget to insert your return label inside
4
Stick your packing slip to the box
Make sure it's totally visible for our carrier


Greta Thunberg mural in San Francisco stirs controversy over use of spray paints
by Robert GeartyA massive mural of the face of teen climate activist Greta Thunberg in downtown San Francisco makes its debut Tuesday — even as it stirs controversy.
The artist behind the 60-foot-tall by 30-foot-wide creation is Argentine muralist Andres Iglesias, who goes by the name Cobre and is known for his “hyper-realistic” portraits. He agreed to work for free on the project, which is being sponsored by the environmental nonprofit One Atmosphere.
But Cobre was forced to go on the defensive after social media users began noting the irony of using spray paints – which they say harm the environment – to pay homage to the 16-year-old Thunberg, The Art Newspaper reported Sunday.
Cobre responded that the materials he was using were environmentally friendly, the paper reported.
TEEN CLIMATE ACTIVIST GRETA THUNBERG MURAL IN CANADA VANDALIZED
“The thing is, the sprays I use are eco-friendly, and most of the paint is hand-painted with a roller,” he told the paper. “[It is] water-based so there is zero impact.”
One Atmosphere said Sunday in a post on city guide SF Station’s Instagram page that they were using spray cans that don’t contain environmentally damaging chlorofluorocarbons.
“We are using low-pressure cans with a minimal footprint,” the post said. “The cans and the boxes will also be recycled, although many of the cans will first have a second life as part of a separate art project.”
GRETA THUNBERG WON'T MEET WITH TRUMP: 'I DON'T UNDERSTAND WHY I WOULD DO THAT'
The nonprofit went on to say the cans have been picked up and delivered in an electric vehicle, and the artist and organizers have been traveling almost exclusively by way of electric car, electric scooter or on foot to the project site — the side of building on Mason Street, above the music venue August Hall.
“The lift being used is also electric, not diesel.” the post added.
When asked by The Art Newspaper if he thinks Thunberg will like the project, Cobre responded, “I think she won’t care.”
She declined to comment, according to the paper.
At the same time, the paper reported some budding art critics on Twitter claimed the mural looked more like Russian President Vladimir Putin.
“I can’t be the only person that thinks the mural of Greta Thunberg in San Francisco looks like Putin, right?” one person tweeted, according to the paper.
Time magazine reported that Cobre hopes the image of Thunberg will bring attention to the crisis of climate change.
He told the magazine that while most people have responded positively to the mural, some have messaged him negative comments on Instagram.
CLICK HERE TO GET THE FOX NEWS APP
“If 95 percent [of] people are going to like it, then my job is well done,” Cobre said. “Haters gonna hate.”


id |
String
|
The ID of the Content in StreamHub |
event |
Number
|
The ID of the event in StreamHub spacetime |
bodyHtml |
String
|
An HTML snippet for the main textual part of the Content |
authorId |
String
|
The ID of the Author in StreamHub |
author |
Object
|
Hydrated author data, if you have it handy |
attachments |
oEmbed[]
|
Attached media to display with the Content |
createdAt |
Number
|
Unix timestamp of when the Content was originally created |
updatedAt |
Number
|
Unix timestamp of when the Content was updated (perhaps edited) |
replaces |
String
|
ID of the Content that this Content should replace in displays (perhaps because of an edit) |
parentId |
String
|
ID of the parent Content |
source |
Number
|
Enum of the Content source. See `const/sources` |
type |
Number
|
Enum of the sdkData state type. See `const/types` |


Detailed Information of Project Name: {{ projectName }} Description: {{ projectInfo.description }} Admin E-Group: {{ projectInfo.admin_group }}
Project Attribute for this Project
Project Allocations for this Project


Toggle navigation
Engineering Computation
Home
Lectures
Assignments
Projects
Videos
About
Lectures
Lectures
Lectures for Engineering Computation and Data Science. Requirements: Google Chrome; Laptop
Lecture 1 - Introduction
Class Information & Logistics
Syllabus
Schedule
Academic Honesty
Term Project
Template
Piazza
Introduction
Overview
(
PPT
)
Introduction
(
PPT
)
Sample Projects
Web Platform
Lab 1 - Development Environment Setup & Configuration
Get Chrome and VStudio
GitHub Setup
Lecture 2 - JavaScript Basics
Preparation Material - JavaScript Syntax Basics
Console
Variables
If ... Else
Debugging Basics
Recommended JavaScript Style
JavaScript Language Reference
Preparation Material - Arrays
Array Basics
Preparation Material - Objects
Objects - Hello World
Objects: Assigning Objects to Properties
Preparation Material - Browser Basics
Browser Processing a Web Page
Lecture 3 - Functions
Preparation Material - Functions
Function Basics
Functions are Objects
Scope: Basics
Scope: Forgetting Var
Scope: Nested Functions
Functions Provide Scope
Preparation Material - Timers
SetTimeout vs SetInterval
Basic Timers
Repeating Timer
Animation Basics Using Timers
Mini Lecture
Web Document Basics - Class Mini Lecture
Web Document Basics - Slides
Binary - Slides
Web Documents & Shapes - Class Exercises
Video Walkthrough: Web Documents & Shapes
The code is on Github
Lecture 4 - JSON Data Files, Boston City Data
Preparation Material - JSON
JSON Basics
Walking Through JSON Data Structures, Arrays, Objects
JSON Debugging
Students List Exercise
City of Boston Employee Earnings (22,000 records)
Mini Lecture
Mini Lecture 1
Mini Lecture 2
VSCode debugging
Active Learning
ex-bostoncity
Lecture 5 - Callbacks, Filters, Sorting, Map, Reduce
Preparation Material - Videos/Exercises
Callback Functions
(
Exercise
)
forEach Callback Functions
(
Exercise
)
Filter Callback Functions
(
Exercise
)
Sort Callback Functions
(
Exercise
)
Map Callback Functions
(
Exercise
)
Reduce Callback Functions
(
Exercise
)
Mini Lecture
Callbacks
Active Learning
Lecture 6 - REST, Web Server Routes, FireBase Cloud DataStore, Chat App
Preparation Material - Slides,Videos/Exercises
L6 Slides, Node Package Manager (NPM), Building a Web Server (Express), Communicating to FireBase Cloud DataStore
FireBase Setup Video
Express Web Server Exercise, NPM Init, package.json, Hitting Routes, Testing with Curl
)
Exercise Chat App - DataStore Basics, Events and Triggers etc
)
Lecture 7 - Geospatial - Spatial/Temporal Information
Preparation Material - Videos/Exercises
Map - Hello World
Basic Timers
Repeating Timer
Mini Lecture
Geospatial
Active Learning
Visualization, Geospatial, Real-Time Data
Lecture 8 - Linear Regression - Climate Data and Google Charts
Linear Regression
Linear Regression Equation- Linear Algebra Approach
Machine Learning - Regression
Regression Equation
Google Charts - Climate Data
Active Learning
In class exercise- regression
In class exercise- NASA Climate Data
Lecture 9 - Data Stores
Lecture
Slides
Active Learning
Exercises
Data Stores
Guestbook - In memory storage
(
Exercise
)
Faker - In memory storage
(
Exercise
)
Little DB
(
Exercise
)
Browser Local Storage
(
Exercise
)
Browser IndexedDB
(
Exercise
)
Cloud Storage - MongoDB
(
Exercise
)
SQLite
(
Exercise
)
Lecture 10 - K-Nearest Neighbors (kNN)
Mini Lecture
K-Nearest Neighbors
Active Learning
Exercise: Calculate Distance Between Points
Exercise: kNN
Solution
Lecture 11 - Naive Bayes Classification
Testing Frameworks
Jest
,
Jest CLI
Mocha
,
Chai
Reference
Bayes Rule
Bayes: Improving understanding with data
Mini Lecture
Naive Bayes Classification
Active Learning
Exercises
Lecture 12 - K-Means
Mini Lecture
K-Means
Active Learning
Exercise: Calculate Centroid Between Points
Exercise: kMeans
Animation
Solution 1
Solution 2
Lecture 13 - Three Tiers: Client, Server, & Storage
Mini Lecture
Bad Bank
Exercise
Starter Code
Exercise Solution
Three Tiers - Little HTTP Server
Three Tiers - Data Store
Three Tiers - HTTP Server + Data Store
Three Tiers - HTTP Server + Data Store + HTML Client
Dynamic User Interface
Debugging
Debugging 3 Tiers
Lecture 14 - Containers
Mini Lecture
Containers
Why do we care about containers?
Docker Fundamentals
Should everyone care about containers?
Fundamental Building Blocks of Modern Systems
Docker Commands
Useful docker commands
Basics
Docker "Hello World"
Docker Node Mongo
More Docker Examples
Nginx
NodeJS
NodeJS + Mongo
Python + Mongo
Push/Pull
Sample Code For Image
Publish Commands - Cheat Sheet
Create and Publish a Docker Image
Databases
Little DB
(
solution
)
Little API
(
solution
)
MongoDB
(
solution
)
MySQL
(
solution
)
PostgreSQL
(
solution
)
Redis
(
solution
)
How To Install Docker on Ubuntu
Installation Instructions
Docker - NodeJS Hello World


Press n or j to go to the next uncovered block, b, p or k for the previous block.
| File | Statements | Branches | Functions | Lines | |||||
|---|---|---|---|---|---|---|---|---|---|
| src |
|
0% | 0/54 | 0% | 0/32 | 0% | 0/26 | 0% | 0/54 |
| src/components |
|
0% | 0/0 | 0% | 0/0 | 0% | 0/0 | 0% | 0/0 |
| src/components/Page |
|
0% | 0/12 | 0% | 0/4 | 0% | 0/2 | 0% | 0/12 |
| src/components/ScrollReset |
|
0% | 0/5 | 100% | 0/0 | 0% | 0/2 | 0% | 0/5 |
| src/components/StatusBullet |
|
0% | 0/9 | 100% | 0/0 | 0% | 0/2 | 0% | 0/8 |
| src/layouts/Dashboard |
|
0% | 0/7 | 100% | 0/0 | 0% | 0/2 | 0% | 0/6 |
| src/layouts/Dashboard/components |
|
0% | 0/0 | 0% | 0/0 | 0% | 0/0 | 0% | 0/0 |
| src/layouts/Dashboard/components/TopBar |
|
0% | 0/7 | 100% | 0/0 | 0% | 0/2 | 0% | 0/6 |
| src/layouts/Error |
|
0% | 0/7 | 100% | 0/0 | 0% | 0/2 | 0% | 0/6 |
| src/theme |
|
0% | 0/3 | 100% | 0/0 | 100% | 0/0 | 0% | 0/3 |
| src/theme/overrides |
|
0% | 0/0 | 0% | 0/0 | 0% | 0/0 | 0% | 0/0 |
| src/utils |
|
0% | 0/14 | 0% | 0/3 | 0% | 0/4 | 0% | 0/14 |
| src/views/Error401 |
|
0% | 0/7 | 0% | 0/2 | 0% | 0/2 | 0% | 0/6 |
| src/views/Error404 |
|
0% | 0/7 | 0% | 0/2 | 0% | 0/2 | 0% | 0/6 |
| src/views/Error500 |
|
0% | 0/7 | 0% | 0/2 | 0% | 0/2 | 0% | 0/6 |
| src/views/ListUsers |
|
0% | 0/29 | 0% | 0/10 | 0% | 0/11 | 0% | 0/26 |
| src/views/ListUsers/components |
|
0% | 0/0 | 0% | 0/0 | 0% | 0/0 | 0% | 0/0 |
| src/views/ListUsers/components/Header |
|
100% | 10/10 | 100% | 0/0 | 100% | 4/4 | 100% | 9/9 |
| src/views/ListUsers/components/UserCard |
|
42.86% | 3/7 | 100% | 0/0 | 0% | 0/2 | 50% | 3/6 |
| src/views/Profile |
|
0% | 0/17 | 0% | 0/10 | 0% | 0/5 | 0% | 0/15 |
| src/views/Profile/components |
|
0% | 0/0 | 0% | 0/0 | 0% | 0/0 | 0% | 0/0 |
| src/views/Profile/components/Header |
|
0% | 0/7 | 0% | 0/2 | 0% | 0/2 | 0% | 0/6 |
| src/views/Profile/components/Info |
|
0% | 0/11 | 0% | 0/4 | 0% | 0/3 | 0% | 0/10 |
| src/views/Profile/components/Location |
|
0% | 0/8 | 0% | 0/4 | 0% | 0/2 | 0% | 0/8 |
| src/views/Profile/components/Login |
|
0% | 0/11 | 0% | 0/4 | 0% | 0/3 | 0% | 0/10 |


Invitations
Thanh Phan Phu
has invited you to
Thanh Phan Phu
10 Hours Ago
Notifications
Welcome to Stisla template!
Yesterday
Hi,
Account
Settings
Logout
Home
Welcome back,
!
What's your plan for today?
New project...
Recent Projects
Recent activites
Now
Nguyen Ba Nam
Add a new column to Birthday Party projects
Create a new project
×
Project
Project Name
Description
Team
Assign a team
Create new team
Create a new team
×
Team Info
Team Name
Description
Team Members
Add


Chapter 7 Exploring data #2
Download a pdf of the lecture slides covering this topic.
7.1 Simple statistical tests in R
Let’s pull the fatal accident data just for the county that includes Las Vegas, NV. Each US county has a unique identifier (FIPS code), composed of a two-digit state FIPS and a three-digit county FIPS code. The state FIPS for Nevada is 32; the county FIPS for Clark County is 003. Therefore, we can filter down to Clark County data in the FARS data we collected with the following code:
library(readr)
library(dplyr)
clark_co_accidents <- read_csv("data/accident.csv") %>%
filter(STATE == 32 & COUNTY == 3)
We can also check the number of accidents:
## # A tibble: 1 x 1
## n
## <int>
## 1 201
We want to test if the probability, on a Friday or Saturday, of a fatal accident occurring is higher than on other days of the week. Let’s clean the data up a bit as a start:
library(tidyr)
library(lubridate)
clark_co_accidents <- clark_co_accidents %>%
select(DAY, MONTH, YEAR) %>%
unite(date, DAY, MONTH, YEAR, sep = "-") %>%
mutate(date = dmy(date))
Here’s what the data looks like now:
## # A tibble: 5 x 1
## date
## <date>
## 1 2016-01-10
## 2 2016-02-21
## 3 2016-01-06
## 4 2016-01-13
## 5 2016-02-18
Next, let’s get the count of accidents by date:
clark_co_accidents <- clark_co_accidents %>%
group_by(date) %>%
count() %>%
ungroup()
clark_co_accidents %>%
slice(1:3)
## # A tibble: 3 x 2
## date n
## <date> <int>
## 1 2016-01-03 1
## 2 2016-01-06 1
## 3 2016-01-09 3
We’re missing the dates without a fatal crash, so let’s add those. First, create a dataframe with all dates in 2016:
all_dates <- data_frame(date = seq(ymd("2016-01-01"),
ymd("2016-12-31"), by = 1))
all_dates %>%
slice(1:5)
## # A tibble: 5 x 1
## date
## <date>
## 1 2016-01-01
## 2 2016-01-02
## 3 2016-01-03
## 4 2016-01-04
## 5 2016-01-05
Then merge this with the original dataset on Las Vegas fatal crashes and make any day missing from the fatal crashes dataset have a “0” for number of fatal accidents (n
):
clark_co_accidents <- clark_co_accidents %>%
right_join(all_dates, by = "date") %>%
# If `n` is missing, set to 0. Otherwise keep value.
mutate(n = ifelse(is.na(n), 0, n))
clark_co_accidents %>%
slice(1:3)
## # A tibble: 3 x 2
## date n
## <date> <dbl>
## 1 2016-01-01 0
## 2 2016-01-02 0
## 3 2016-01-03 1
Next, let’s add some information about day of week and weekend:
clark_co_accidents <- clark_co_accidents %>%
mutate(weekday = wday(date, label = TRUE),
weekend = weekday %in% c("Fri", "Sat"))
clark_co_accidents %>%
slice(1:3)
## # A tibble: 3 x 4
## date n weekday weekend
## <date> <dbl> <ord> <lgl>
## 1 2016-01-01 0 Fri TRUE
## 2 2016-01-02 0 Sat TRUE
## 3 2016-01-03 1 Sun FALSE
Now let’s calculate the probability that a day has at least one fatal crash, separately for weekends and weekdays:
clark_co_accidents <- clark_co_accidents %>%
mutate(any_crash = n > 0)
crash_prob <- clark_co_accidents %>%
group_by(weekend) %>%
summarize(n_days = n(),
crash_days = sum(any_crash)) %>%
mutate(prob_crash_day = crash_days / n_days)
crash_prob
## # A tibble: 2 x 4
## weekend n_days crash_days prob_crash_day
## <lgl> <int> <int> <dbl>
## 1 FALSE 260 107 0.412
## 2 TRUE 106 43 0.406
In R, you can use prop.test
to test if two proportions are equal. Inputs include the total number of trials in each group (n =
) and the number of “successes”" (x =
):
##
## 2-sample test for equality of proportions with continuity
## correction
##
## data: crash_prob$crash_days out of crash_prob$n_days
## X-squared = 1.5978e-30, df = 1, p-value = 1
## alternative hypothesis: two.sided
## 95 percent confidence interval:
## -0.1109757 0.1227318
## sample estimates:
## prop 1 prop 2
## 0.4115385 0.4056604
I won’t be teaching in this course how to find the correct statistical test. That’s something you’ll hopefully learn in a statistics course. There are also a variety of books that can help you with this, including some that you can access free online through CSU’s library. One servicable introduction is “Statistical Analysis with R for Dummies”.
You can create an object from the output of any statistical test in R. Typically, this will be (at least at some level) in an object class called a “list”:
## [1] TRUE
So far, we’ve mostly worked with two object types in R, dataframes and vectors. In the next subsection we’ll look more at two object classes we haven’t looked at much, matrices and lists. Both have important roles once you start applying more advanced methods to analyze your data.
7.2 Matrices
A matrix is like a data frame, but all the values in all columns must be of the same class (e.g., numeric, character). R uses matrices a lot for its underlying math (e.g., for the linear algebra operations required for fitting regression models). R can do matrix operations quite quickly.
You can create a matrix with the matrix
function. Input a vector with the values to fill the matrix and ncol
to set the number of columns:
## [,1] [,2] [,3] [,4] [,5]
## [1,] 1 3 5 7 9
## [2,] 2 4 6 8 10
By default, the matrix will fill up by column. You can fill it by row with the byrow
function:
## [,1] [,2] [,3] [,4] [,5]
## [1,] 1 2 3 4 5
## [2,] 6 7 8 9 10
In certain situations, you might want to work with a matrix instead of a data frame (for example, in cases where you were concerned about speed – a matrix is more memory efficient than the corresponding data frame). If you want to convert a data frame to a matrix, you can use the as.matrix
function:
foo <- data.frame(col_1 = 1:2, col_2 = 3:4,
col_3 = 5:6, col_4 = 7:8,
col_5 = 9:10)
(foo <- as.matrix(foo))
## col_1 col_2 col_3 col_4 col_5
## [1,] 1 3 5 7 9
## [2,] 2 4 6 8 10
You can index matrices with square brackets, just like data frames:
## col_1 col_2
## 1 3
You cannot, however, use dplyr
functions with matrices:
All elements in a matrix must have the same class.
The matrix will default to make all values the most general class of any of the values, in any column. For example, if we replaced one numeric value with the character “a”, everything would turn into a character:
## col_1 col_2 col_3 col_4 col_5
## [1,] "a" "3" "5" "7" "9"
## [2,] "2" "4" "6" "8" "10"
7.3 Lists
A list has different elements, just like a data frame has different columns. However, the different elements of a list can have different lengths (unlike the columns of a data frame). The different elements can also have different classes.
bar <- list(some_letters = letters[1:3],
some_numbers = 1:5,
some_logical_values = c(TRUE, FALSE))
bar
## $some_letters
## [1] "a" "b" "c"
##
## $some_numbers
## [1] 1 2 3 4 5
##
## $some_logical_values
## [1] TRUE FALSE
To index an element from a list, use double square brackets. You can use bracket indexing either with numbers (which element in the list?) or with names:
## [1] "a" "b" "c"
## [1] 1 2 3 4 5
You can also index lists with the $
operator:
## [1] TRUE FALSE
To access a specific value within a list element we can index the element e.g.:
## [1] "b"
Lists can be used to contain data with an unusual structure and / or lots of different components. For example, the information from fitting a regression is often stored as a list:
## [1] TRUE
The names
function returns the name of each element in the list:
## [1] "coefficients" "residuals" "fitted.values"
## (Intercept) c(1:10)
## -0.32660336 0.08881277
A list can even contain other lists. We can use the str
function to see the structure of a list:
## List of 2
## $ :List of 2
## ..$ : chr "a"
## ..$ : chr "b"
## $ :List of 2
## ..$ : num 1
## ..$ : num 2
Sometimes you’ll see unnecessary lists-of-lists, perhaps when importing data into R created. Or a list with multiple elements that you would like to combine. You can remove a level of hierarchy from a list using the flatten
function from the purrr
package:
## [[1]]
## [[1]][[1]]
## [1] "a"
##
## [[1]][[2]]
## [1] "b"
##
##
## [[2]]
## [[2]][[1]]
## [1] 1
##
## [[2]][[2]]
## [1] 2
## [[1]]
## [1] "a"
##
## [[2]]
## [1] "b"
##
## [[3]]
## [1] 1
##
## [[4]]
## [1] 2
Let’s look at the list object from the statistical test we ran for Las Vegas:
## List of 9
## $ statistic : Named num 1.6e-30
## ..- attr(*, "names")= chr "X-squared"
## $ parameter : Named num 1
## ..- attr(*, "names")= chr "df"
## $ p.value : num 1
## $ estimate : Named num [1:2] 0.412 0.406
## ..- attr(*, "names")= chr [1:2] "prop 1" "prop 2"
## $ null.value : NULL
## $ conf.int : num [1:2] -0.111 0.123
## ..- attr(*, "conf.level")= num 0.95
## $ alternative: chr "two.sided"
## $ method : chr "2-sample test for equality of proportions with continuity correction"
## $ data.name : chr "crash_prob$crash_days out of crash_prob$n_days"
## - attr(*, "class")= chr "htest"
We can pull out an element using the $
notation:
## [1] 1
Or using the [[
notation:
## prop 1 prop 2
## 0.4115385 0.4056604
You may have noticed, though, that this output is not a tidy dataframe.
Ack! That means we can’t use all the tidyverse tricks we’ve learned so far in the course!
Fortunately, David Robinson noticed this problem and came up with a package called broom
that can “tidy up” a lot of these kinds of objects.
The broom
package has three main functions:
glance
: Return a one-row, tidy dataframe from a model or other R objecttidy
: Return a tidy dataframe from a model or other R objectaugment
: “Augment” the dataframe you input to the statistical function
Here is the output for tidy
for the vegas_test
object (augment
won’t work for this type of object, and glance
gives the same thing as tidy
):
## # A tibble: 1 x 9
## estimate1 estimate2 statistic p.value parameter conf.low conf.high method
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <chr>
## 1 0.412 0.406 1.60e-30 1.000 1 -0.111 0.123 2-sam…
## # … with 1 more variable: alternative <chr>
7.4 Regression models
7.4.1 Formula structure
Regression models can be used to estimate how the expected value of a dependent variable changes as independent variables change.
In R, regression formulas take this structure:
Notice that a tilde, ~
, is used to separate the independent and dependent variables and that a plus sign, +
, is used to join independent variables. This format mimics the statistical notation:
\[ Y_i \sim X_1 + X_2 + X_3 \]
You will use this type of structure in R fo a lot of different function calls, including those for linear models (fit with the lm
function) and generalized linear models (fit with the glm
function).
There are some conventions that can be used in R formulas. Common ones include:
| Convention | Meaning |
|---|---|
I() |
evaluate the formula inside I() before fitting (e.g.,
I(x1 + x2) ) |
: |
fit the interaction between x1 and x2 variables |
* |
fit the main effects and interaction for both variables
(e.g., x1*x2 equals x1 + x2 + x1:x2 ) |
. |
include as independent variables all variables other than
the response (e.g., y ~ . ) |
1 |
intercept (e.g., y ~ 1 for an intercept-only model) |
- |
do not include a variable in the data frame as an
independent variables (e.g., y ~ . - x1 ); usually used
in conjunction with . or 1 |
7.4.2 Linear models
To fit a linear model, you can use the function lm()
. This function is part of the stats
package, which comes installed with base R. In this function, you can use the data
option to specify the data frame from which to get the vectors.
## Warning: `as_data_frame()` is deprecated, use `as_tibble()` (but mind the new semantics).
## This warning is displayed once per session.
## Warning: Unknown levels in `f`: 1, 2
This previous call fits the model:
\[ Y_{i} = \beta_{0} + \beta_{1}X_{1,i} + \epsilon_{i} \]
where:
- \(Y_{i}\) : weight of child \(i\)
- \(X_{1,i}\) : height of child \(i\)
If you run the lm
function without saving it as an object, R will fit the regression and print out the function call and the estimated model coefficients:
##
## Call:
## lm(formula = wt ~ ht, data = nepali)
##
## Coefficients:
## (Intercept) ht
## -8.7581 0.2342
However, to be able to use the model later for things like predictions and model assessments, you should save the output of the function as an R object:
This object has a special class, lm
:
## [1] "lm"
This class is a special type of list object. If you use is.list
to check, you can confirm that this object is a list:
## [1] TRUE
There are a number of functions that you can apply to an lm
object. These include:
| Function | Description |
|---|---|
summary |
Get a variety of information on the model, including coefficients and p-values for the coefficients |
coefficients |
Pull out just the coefficients for a model |
fitted |
Get the fitted values from the model (for the data used to fit the model) |
plot |
Create plots to help assess model assumptions |
residuals |
Get the model residuals |
For example, you can get the coefficients from the model by running:
## (Intercept) ht
## -8.7581466 0.2341969
The estimated coefficient for the intercept is always given under the name “(Intercept)”. Estimated coefficients for independent variables are given based on their column names in the original data (“ht” here, for \(\beta_1\), or the estimated increase in expected weight for a one unit increase in height).
You can use the output from a coefficients
call to plot a regression line based on the model fit on top of points showing the original data (Figure 7.1).
mod_coef <- coefficients(mod_a)
ggplot(nepali, aes(x = ht, y = wt)) +
geom_point(size = 0.2) +
xlab("Height (cm)") + ylab("Weight (kg)") +
geom_abline(aes(intercept = mod_coef[1],
slope = mod_coef[2]), col = "blue")
You can also add a linear regression line to a scatterplot by adding the geom geom_smooth
using the argument method = “lm”
.
You can use the function residuals
on an lm
object to pull out the residuals from the model fit:
## 1 2 3 4 6 7
## 0.1993922 -0.4329393 -0.4373953 -0.1355300 -0.6749080 -1.0838199
The result of a residuals
call is a vector with one element for each of the non-missing observations (rows) in the data frame you used to fit the model. Each value gives the different between the model fitted value and the observed value for each of these observations, in the same order the observations show up in the data frame. The residuals are in the same order as the observations in the original data frame.
You can also use the shorter function coef
as an alternative to coefficients
and the shorter function resid
as an alternative to residuals
.
As noted in the subsection on simple statistics functions, the summary
function returns different output depending on the type of object that is input to the function. If you input a regression model object to summary
, the function gives you a lot of information about the model. For example, here is the output returned by running summary
for the linear regression model object we just created:
##
## Call:
## lm(formula = wt ~ ht, data = nepali)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.0077 -0.5479 -0.0293 0.4972 3.3514
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -8.758147 0.211529 -41.40 <2e-16 ***
## ht 0.234197 0.002459 95.23 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.8733 on 875 degrees of freedom
## (123 observations deleted due to missingness)
## Multiple R-squared: 0.912, Adjusted R-squared: 0.9119
## F-statistic: 9068 on 1 and 875 DF, p-value: < 2.2e-16
This output includes a lot of useful elements, including (1) basic summary statistics for the residuals (to meet model assumptions, the median should be around zero and the absolute values fairly similar for the first and third quantiles), (2) coefficient estimates, standard errors, and p-values, and (3) some model summary statistics, including residual standard error, degrees of freedom, number of missing observations, and F-statistic.
The object returned by the summary()
function when it is applied to an lm
object is a list, which you can confirm using the is.list
function:
## [1] TRUE
With any list, you can use the names
function to get the names of all of the different elements of the object:
## [1] "call" "terms" "residuals" "coefficients"
## [5] "aliased" "sigma" "df" "r.squared"
## [9] "adj.r.squared" "fstatistic" "cov.unscaled" "na.action"
You can use the $
operator to pull out any element of the list. For example, to pull out the table with information on the estimated model coefficients, you can run:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -8.7581466 0.211529182 -41.40396 2.411051e-208
## ht 0.2341969 0.002459347 95.22726 0.000000e+00
The plot
function, like the summary
function, will give different output depending on the class of the object that you input. For an lm
object, you can use the plot
function to get a number of useful diagnostic plots that will help you check regression assumptions (Figure 7.2):
You can also use binary variables or factors as independent variables in regression models. For example, in the nepali
dataset, sex
is a factor variable with the levels “Male” and “Female”. You can fit a linear model of weight regressed on sex for this data with the call:
This call fits the model:
\[ Y_{i} = \beta_{0} + \beta_{1}X_{1,i} + \epsilon_{i} \]
where \(X_{1,i}\) : sex of child \(i\), where 0 = male and 1 = female.
Here are the estimated coefficients from fitting this model:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 11.4226374 0.1375427 83.047920 0.00000000
## sexFemale -0.4866184 0.1982810 -2.454185 0.01431429
You’ll notice that, in addition to an estimated intercept ((Intercept)
), the other estimated coefficient is sexFemale
rather than just sex
, although the column name in the data frame input to lm
for this variable is sex
.
This is because, when a factor or binary variable is input as an independent variable in a linear regression model, R will fit an estimated coefficient for all levels of factors except the first factor level. By default, this first factor level is used as the baseline level, and so its estimated mean is given by the estimated intercept, while the other model coefficients give the estimated difference from this baseline.
For example, the model fit above tells us that the estimated mean weight of males is 11.4, while the estimated mean weight of females is 11.4 + -0.5 = 10.9.
7.4.3 Generalized linear models (GLMs)
You can fit a variety of models, including linear models, logistic models, and Poisson models, using generalized linear models (GLMs).
For linear models, the only difference between lm
and glm
are the mechanics of how they estimate the model coefficients (lm
uses least squares while glm
uses maximum likelihood). You will (almost always) get exactly the same estimated coefficients regardless of whether you use glm
or lm
to fit a linear regression.
For example, here is the code to fit a linear regression model for weight regressed on height from the nepali
dataset:
This call fits the same regression model I fit earlier with the lm
function and saved as mod_a
. You can see that the two methods give exactly the same coefficient estimates:
## (Intercept) ht
## -8.7581466 0.2341969
## (Intercept) ht
## -8.7581466 0.2341969
Unlike the lm
function, however, the glm
function also allows you to fit other model types, including logistic and Poisson models. You can specify the model type using the family
argument to the glm
call:
| Model type | family argument |
|---|---|
| Linear | family = gaussian(link = 'identity') |
| Logistic | family = binomial(link = 'logit') |
| Poisson | family = poisson(link = 'log') |
For example, say we wanted to fit a logistic regression for the nepali
data of whether the probability of a child weighing more than 13 kg is associated with the child’s sex.
First, create a binary variable in the nepali
dataset, wt_over_13
, that is TRUE
if a child weighed more than 13 kilograms and FALSE otherwise. You can use the mutate
function from dplyr
to add this new column (which, as a note, is a logical vector):
## id sex wt ht mage lit died alive age wt_over_13
## 1 120011 Male 12.8 91.2 35 0 2 5 41 FALSE
## 2 120011 Male 12.8 93.9 35 0 2 5 45 FALSE
## 3 120011 Male 13.1 95.2 35 0 2 5 49 TRUE
## 4 120011 Male 13.8 96.9 35 0 2 5 53 TRUE
## 5 120011 Male NA NA 35 0 2 5 57 NA
## 6 120012 Female 14.9 103.9 35 0 2 5 57 TRUE
Now you can fit a logistic regression of wt_over_13
regressed on sex, using a logistic model:
Elements of a GLM can be pulled out in the same way that we looked at elements from the linear model fit with lm
. For example, to see a table of estimated model coefficients, you can run:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -0.8736013 0.1028491 -8.494009 1.996282e-17
## sexFemale -0.2697469 0.1533194 -1.759378 7.851325e-02
Because this model was a logistic model, fit with a log link, here the model coefficient estimate for sexFemale
gives an estimate of the log odds of weight higher than 13 kg associated with females versus males. The p-value for this estimate (Pr(>|z|)
= 0.08) isn’t very small, suggesting that the difference between male and female children in the odds of weighing more than 13 kg is not statistically significant.
7.5 Handling model objects
The broom
package contains tools for converting statistical objects into nice tidy data frames. The tools in the broom
package make it easy to process statistical results in R using the tools of the tidyverse
.
7.5.1 broom::tidy
The tidy()
function returns a data frame with information on the fitted model terms. For example, when applied to one of our linear models we get:
| term | estimate | std.error | statistic | p.value |
|---|---|---|---|---|
| (Intercept) | -8.758 | 0.212 | -41.404 | 0 |
| ht | 0.234 | 0.002 | 95.227 | 0 |
## [1] "tbl_df" "tbl" "data.frame"
You can pass arguments to the tidy()
function. For example, include confidence intervals:
| term | estimate | std.error | statistic | p.value | conf.low | conf.high |
|---|---|---|---|---|---|---|
| (Intercept) | -8.758 | 0.212 | -41.404 | 0 | -9.173 | -8.343 |
| ht | 0.234 | 0.002 | 95.227 | 0 | 0.229 | 0.239 |
7.5.2 broom::augment
The augment()
function adds information about a fitted model to the dataset used to fit the model. For example, when applied to one of our linear models we get information on the fitted values and residuals included in the output:
| .rownames | wt | ht | .fitted | .se.fit | .resid | .hat | .sigma | .cooksd | .std.resid |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 12.8 | 91.2 | 12.601 | 0.033 | 0.199 | 0.001 | 0.874 | 0 | 0.228 |
| 2 | 12.8 | 93.9 | 13.233 | 0.036 | -0.433 | 0.002 | 0.874 | 0 | -0.496 |
| 3 | 13.1 | 95.2 | 13.537 | 0.038 | -0.437 | 0.002 | 0.874 | 0 | -0.501 |
7.5.3 broom::glance
The glance()
functions returns a one row summary of a fitted model object: For example:
| r.squared | adj.r.squared | sigma | statistic | p.value | df | logLik | AIC | BIC | deviance | df.residual |
|---|---|---|---|---|---|---|---|---|---|---|
| 0.912 | 0.912 | 0.873 | 9068.23 | 0 | 2 | -1124.615 | 2255.23 | 2269.56 | 667.351 | 875 |
7.5.4 References– statistics in R
One great (and free online for CSU students through our library) book to find out more about using R for basic statistics is:
If you want all the details about fitting linear models and GLMs in R, Julian Faraway’s books are fantastic. He has one on linear models and one on extensions including logistic and Poisson models:
- Linear Models with R (also free online through the CSU library)
- Extending the Linear Model with R
7.6 In-course exercise
7.6.1 Running a simple statistical test
In last week’s in-course exercise, we found out that about 17% of babies born in the United States between 1980 and 1995 had names that started with an “A” or “K” (60,893 babies out of 359,011).
- What is the proportion of people with names that start with an “A” or “K” in our class?
- Use a simple statistical test to test the hypothesis that the class comes from a binomial distribution with the same distribution as babies born in the US between 1980 and 1995, in terms of chance of having a name that starts with “A” or “K”.
- You may get the warning “Chi-squared approximation may be incorrect”. See if you can figure out this warning.
- If you are able to figure out and run a statistical test, go back to your notes from
last week and write a full script, starting from loading the
babynames
data and creating your own vector of the names in this class, all the way through to running the statistical test you picked.
<IP_ADDRESS> Example R code
Here is a vector with names in our class:
library(stringr)
student_list <- data_frame(name = c("Aeriel", "Rebecca", "Grant",
"Amy", "Jessy", "Alyssa",
"Camron", "Anastasia", "Kyle",
"Ana", "Amanda", "Kathleen",
"Kyle", "Ana", "Amanda", "Kathleen",
"Kayla", "Nichole", "Randy", "Katy",
"Devin"))
student_list <- student_list %>%
mutate(first_letter = str_sub(name, 1, 1))
student_list %>%
slice(1:3)
## # A tibble: 3 x 2
## name first_letter
## <chr> <chr>
## 1 Aeriel A
## 2 Rebecca R
## 3 Grant G
Let’s get the total number of students, and then the total number with a name that starts with “A” or “K”:
## # A tibble: 1 x 1
## n
## <int>
## 1 21
a_or_k_students <- student_list %>%
mutate(a_or_k = first_letter %in% c("A", "K")) %>%
group_by(a_or_k) %>%
count()
a_or_k_students
## # A tibble: 2 x 2
## # Groups: a_or_k [2]
## a_or_k n
## <lgl> <int>
## 1 FALSE 7
## 2 TRUE 14
The proportion of students with names starting with “A” or “K” are 14 / 21 = 0.67.
You could run a statistical test comparing these two proportions:
## Warning in prop.test(x = c(60893, 14), n = c(359011, 21)): Chi-squared
## approximation may be incorrect
##
## 2-sample test for equality of proportions with continuity
## correction
##
## data: c(60893, 14) out of c(359011, 21)
## X-squared = 33.386, df = 1, p-value = 7.557e-09
## alternative hypothesis: two.sided
## 95 percent confidence interval:
## -0.7224875 -0.2716195
## sample estimates:
## prop 1 prop 2
## 0.1696132 0.6666667
You could also test whether the proportion in our class is consistent with the null hypothesis that you were drawn from a binomial distribution with a proportion of 0.17 (in-line with the national values):
## Warning in prop.test(x = 14, n = 21, p = 0.17): Chi-squared approximation
## may be incorrect
##
## 1-sample proportions test with continuity correction
##
## data: 14 out of 21, null probability 0.17
## X-squared = 33.278, df = 1, p-value = 7.99e-09
## alternative hypothesis: true p is not equal to 0.17
## 95 percent confidence interval:
## 0.4310506 0.8451865
## sample estimates:
## p
## 0.6666667
Finally, when we run this test, we get the warning that “Chi-squared approximation may be incorrect”. Based on googling ‘r prop.test “Chi-squared approximation may be incorrect”’,
it sounds like we might be getting this error because we have a pretty low number of
people in the class. One recommendation is to instead run a Chi-square contingency table test, which you can do with chisq.test
, while setting simulate.p.values
to TRUE. This requires
a slightly different set-up for the data (which you can figure out by trying out the
examples in the helpfile for chisq.test
):
## [,1] [,2]
## [1,] 14 7
## [2,] 60893 298118
##
## Pearson's Chi-squared test with simulated p-value (based on 2000
## replicates)
##
## data: a_or_k_names
## X-squared = 36.83, df = NA, p-value = 0.0004998
You could also use binom.test
, which will run as an exact binomial test:
##
## Exact binomial test
##
## data: 14 and 21
## number of successes = 14, number of trials = 21, p-value =
## 5.862e-07
## alternative hypothesis: true probability of success is not equal to 0.17
## 95 percent confidence interval:
## 0.4303245 0.8541231
## sample estimates:
## probability of success
## 0.6666667
7.6.2 Using regression models to explore data #1
For this exercise, you will need the following packages. If do not have them already, you will need to install them.
For this part of the exercise, you’ll use a dataset on weather, air pollution, and mortality counts in Chicago, IL. This dataset is called chicagoNMMAPS
and is part of the dlnm
package. Change the name of the data frame to chic
(this object name is shorter and will be easier to work with). Check out the data a bit to see what variables you have, and then perform the following tasks:
- Write out (on paper, not in R) the regression equation for regressing dewpoint temperature on temperature.
- Try fitting a linear regression of dew point temperature (
dptp
) regressed on temperature (temp
). Save this model as the objectmod_1
(i.e., is the dependent variable of dewpoint temperature linearly associated with the independent variable of temperature). - Based on this regression, does there seem to be a relationship between temperature and dewpoint temperature in Chicago? (Hint: Try using
glance
andtidy
from thebroom
package on the model object to get more information about the model you fit.) What is the coefficient for temperature (in other words, for every 1 degree increase in temperature, how much do we expect the dewpoint temperature to change?)? What is the p-value for the coefficient for temperature? - Plot temperature (x-axis) versus dewpoint temperature (y-axis) for Chicago. Add in the regression line from the model you fit by using the results from
augment
. - Use
autoplot
on the model object to generate some model diagnostic plots (make sure you have theggfortify
package loaded and installed). - Try fitting the regression as a GLM, using
glm()
(but still assuming the outcome variable is normally distributed). Are your coefficients different?
<IP_ADDRESS> Example R code:
The regression equation for the model you want to fit, regressing dewpoint temperature on temperature, is:
\[ Y_t \sim \beta_0 + \beta_1 X_t \]
where \(Y_t\) is the dewpoint temperature on day \(t\), \(X_t\) is the temperature on day \(t\), and \(\beta_0\) and \(\beta_1\) are model coefficients.
Install and load the dlnm
package and then load the chicagoNMMAPS
data. Change the name of the data frame to chic
, so it will be shorter to call for the rest of your work.
Fit a linear regression of dptp
on temp
and save as the object mod_1
:
##
## Call:
## lm(formula = dptp ~ temp, data = chic)
##
## Coefficients:
## (Intercept) temp
## 24.025 1.621
Use functions from the broom
package to pull the same information about the model in a “tidy” format. To find out if the evidence for a linear association between temperature and dewpoint temperature, use the tidy
function to get model coefficients in a tidy format:
## # A tibble: 2 x 5
## term estimate std.error statistic p.value
## <chr> <dbl> <dbl> <dbl> <dbl>
## 1 (Intercept) 24.0 0.113 213. 0
## 2 temp 1.62 0.00763 212. 0
There does seem to be an association between temperature and dewpoint temperature: a unit increase in temperature is associated with a 1.6 unit increase in dewpoint temperature. The p-value for the temperature coefficient is <2e-16. This is far below 0.05, which suggests we would be very unlikely to see such a strong association by chance if the null hypothesis, that the two variables are not associated, were true.
You can also check overall model summaries using the glance
function:
## # A tibble: 1 x 11
## r.squared adj.r.squared sigma statistic p.value df logLik AIC
## <dbl> <dbl> <dbl> <dbl> <dbl> <int> <dbl> <dbl>
## 1 0.898 0.898 5.90 45108. 0 2 -16332. 32670.
## # … with 3 more variables: BIC <dbl>, deviance <dbl>, df.residual <int>
To create plots of the observations and the fit model, use the augment
function to add model output (e.g., predictions, residuals) to the original data frame of observed temperatures and dew point temperatures:
## # A tibble: 3 x 9
## dptp temp .fitted .se.fit .resid .hat .sigma .cooksd .std.resid
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 31.5 -0.278 23.6 0.114 7.93 0.000376 5.90 0.000340 1.34
## 2 29.9 0.556 24.9 0.110 4.95 0.000348 5.90 0.000123 0.839
## 3 27.4 0.556 24.9 0.110 2.45 0.000348 5.90 0.0000300 0.415
Plot these two variables and add in the fitted line from the model (note: I’ve used the color
option to make the color of the points gray). Use the output from augment
to create a plot of the original data, with the predicted values used to plot a fitted line.
augment(mod_1) %>%
ggplot(aes(x = temp, y = dptp)) +
geom_point(size = 0.8, alpha = 0.5, col = "gray") +
geom_line(aes(x = temp, y = .fitted), color = "red", size = 2) +
theme_classic()
Plot some plots to check model assumptions for the model you fit using the autoplot
function on your model object:
Try fitting the model using glm()
. Call it mod_1a
. Compare the coefficients for the two models. You can use the tidy
function on an lm
or glm
object to pull out just the model coefficients and associated model results. Here, I’ve used a pipeline of code to create a tidy data frame that merges these “tidy” coefficient outputs (from the two models) into a single data frame):
mod_1a <- glm(dptp ~ temp, data = chic)
tidy(mod_1) %>%
select(term, estimate) %>%
inner_join(mod_1a %>% tidy() %>% select(term, estimate), by = "term") %>%
rename(estimate_lm_mod = estimate.x,
estimate_glm_mod = estimate.y)
## # A tibble: 2 x 3
## term estimate_lm_mod estimate_glm_mod
## <chr> <dbl> <dbl>
## 1 (Intercept) 24.0 24.0
## 2 temp 1.62 1.62
The results from the two models are identical.
As a note, you could have also just run tidy
on each model object, without merging them together into a single data frame:
## # A tibble: 2 x 5
## term estimate std.error statistic p.value
## <chr> <dbl> <dbl> <dbl> <dbl>
## 1 (Intercept) 24.0 0.113 213. 0
## 2 temp 1.62 0.00763 212. 0
## # A tibble: 2 x 5
## term estimate std.error statistic p.value
## <chr> <dbl> <dbl> <dbl> <dbl>
## 1 (Intercept) 24.0 0.113 213. 0
## 2 temp 1.62 0.00763 212. 0
7.6.3 Using regression models to explore data #2
- Does \(PM_{10}\) vary by day of the week? (Hint: The
dow
variable is a factor that gives day of the week. You can do an ANOVA analysis by fitting a linear model using this variable as the independent variable. Some of the overall model summaries will compare this model to an intercept-only model.) What day of the week is PM10 generally highest? (Check the model coefficients to figure this out.) Try to write out (on paper) the regression equation for the model you’re fitting. - Try using
glm()
to run a Poisson regression of respiratory deaths (resp
) on temperature during summer days. Start by creating a subset with just summer days calledsummer
. (Hint: Use themonth
function with the argumentlabel = TRUE
fromlubridate
to do this– just pull out the subset where the month is 6, 7, or 8, for “Jun”, “Jul”, and “Aug”.) Try to write out the regression equation for the model you’re fitting. - The coefficient for the temperature variable in this model is our best estimate (based on this model) of the log relative risk for a one degree Celcius increase in temperature. What is the relative risk associated with a one degree Celsius increase?
<IP_ADDRESS> Example R code:
Fit a model of \(PM_{10}\) regressed on day of week, where day of week is a factor.
## # A tibble: 7 x 5
## term estimate std.error statistic p.value
## <chr> <dbl> <dbl> <dbl> <dbl>
## 1 (Intercept) 27.5 0.730 37.7 7.47e-273
## 2 dowMonday 6.13 1.03 5.93 3.22e- 9
## 3 dowTuesday 6.80 1.03 6.62 4.05e- 11
## 4 dowWednesday 8.48 1.03 8.26 1.85e- 16
## 5 dowThursday 8.80 1.02 8.60 1.08e- 17
## 6 dowFriday 9.48 1.03 9.24 3.61e- 20
## 7 dowSaturday 3.66 1.03 3.56 3.68e- 4
Use glance
to check some of the overall summaries of this model. The statistic
column here is the F statistic from test comparing this model to an intercept-only model.
## # A tibble: 1 x 11
## r.squared adj.r.squared sigma statistic p.value df logLik AIC
## <dbl> <dbl> <dbl> <dbl> <dbl> <int> <dbl> <dbl>
## 1 0.0259 0.0247 19.1 21.5 4.61e-25 7 -21234. 42484.
## # … with 3 more variables: BIC <dbl>, deviance <dbl>, df.residual <int>
As a note, you may have heard in previous statistics classes that you can use the anova()
command to compare this model to a model with only an intercept (i.e., one that only fits a global mean and uses that as the expected value for all of the observations). Note that, in this case, the F value from anova
for this model comparison is the same as the statistic
you got in the overall summary statistics you get with glance
in the previous code.
## Analysis of Variance Table
##
## Response: pm10
## Df Sum Sq Mean Sq F value Pr(>F)
## dow 6 46924 7820.6 21.5 < 2.2e-16 ***
## Residuals 4856 1766407 363.8
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
The overall p-value from anova
for with day-of-week coefficients versus the model that just has an intercept is < 2.2e-16. This is well below 0.05, which suggests that day-of-week is associated with PM10 concentration, as a model that includes day-of-week does a much better job of explaining variation in PM10 than a model without it does.
Use a boxplot to visually compare PM10 by day of week.
Now try the same plot, but try using the ylim =
option to change the limits on the y-axis for the graph, so you can get a better idea of the pattern by day of week (some of the extreme values are very high, which makes it hard to compare by eye when the y-axis extends to include them all).
## Warning: Removed 292 rows containing non-finite values (stat_boxplot).
Create a subset called summer
with just the summer days:
library(lubridate)
summer <- chic %>%
mutate(month = month(date, label = TRUE)) %>%
filter(month %in% c("Jun", "Jul", "Aug"))
summer %>%
slice(1:3)
## date time year month doy dow death cvd resp temp dptp
## 1 1987-06-01 152 1987 Jun 152 Monday 112 60 5 23.61111 68.50
## 2 1987-06-02 153 1987 Jun 153 Tuesday 111 57 7 22.22222 64.75
## 3 1987-06-03 154 1987 Jun 154 Wednesday 120 59 9 20.55556 47.25
## rhum pm10 o3
## 1 71.875 22.95607 34.94623
## 2 95.250 31.31339 18.96620
## 3 47.125 34.95607 23.59270
Use glm()
to fit a Poisson model of respiratory deaths regressed on temperature. Since you want to fit a Poisson model, use the option family = poisson(link = "log")
.
## # A tibble: 1 x 7
## null.deviance df.null logLik AIC BIC deviance df.residual
## <dbl> <int> <dbl> <dbl> <dbl> <dbl> <int>
## 1 1499. 1287 -3211. 6425. 6436. 1494. 1286
## # A tibble: 2 x 5
## term estimate std.error statistic p.value
## <chr> <dbl> <dbl> <dbl> <dbl>
## 1 (Intercept) 1.91 0.0584 32.7 6.60e-235
## 2 temp 0.00614 0.00258 2.38 1.74e- 2
Use the fitted model coefficient to determine the relative risk for a one degree Celsius increase in temperature. First, remember that you can use the tidy()
function to read out the model coefficients. The second of these is the value for the temperature coefficient. That means that you can use indexing ([2]
) to get just that value. That’s the log relative risk; take the exponent to get the relative risk.
## # A tibble: 1 x 6
## term estimate std.error statistic p.value log_rr
## <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 temp 0.00614 0.00258 2.38 0.0174 1.01
As a note, you can use the conf.int
parameter in tidy
to also pull confidence intervals:
## # A tibble: 2 x 7
## term estimate std.error statistic p.value conf.low conf.high
## <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 (Intercept) 1.91 0.0584 32.7 6.60e-235 1.80 2.02
## 2 temp 0.00614 0.00258 2.38 1.74e- 2 0.00108 0.0112
You could use this to get the confidence interval for relative risk (check out the mutate_at
function if you haven’t seen it before):
tidy(mod_3, conf.int = TRUE) %>%
select(term, estimate, conf.low, conf.high) %>%
filter(term == "temp") %>%
mutate_at(vars(estimate:conf.high), funs(exp(.)))
## Warning: funs() is soft deprecated as of dplyr 0.8.0
## Please use a list of either functions or lambdas:
##
## # Simple named list:
## list(mean = mean, median = median)
##
## # Auto named with `tibble::lst()`:
## tibble::lst(mean, median)
##
## # Using lambdas
## list(~ mean(., trim = .2), ~ median(., na.rm = TRUE))
## This warning is displayed once per session.
## # A tibble: 1 x 4
## term estimate conf.low conf.high
## <chr> <dbl> <dbl> <dbl>
## 1 temp 1.01 1.00 1.01
7.6.4 Exploring Fatality Analysis Reporting System (FARS) data
- Visit http://metrocosm.com/10-years-of-traffic-accidents-mapped.html and explore the interactive visualization created by Max Galka using this public dataset on US fatal motor vehicle accidents.
- Go to FARS web page. We want to get the raw data on fatal accidents. Navigate this page to figure out how you can get this raw data for the whole county for 2015 (hint: you’ll need to access the raw data using FTP, and you may have more success with some web browsers that others). Save 2015 “National” data (csv format) to your computer. What is the structure of how this data is saved (e.g., directory structure, file structure)?
- On the FARS web page, find the documentation describing this raw data. (The relevant documentation file is called Fatality Analysis Reporting System (FARS) Analytical User’s Manual 1975-2015) Look through both this documentation and the raw files you downloaded to figure out what information is included in the data.
- Read the
accident.csv
file for 2015 into R (this is one of the files you’ll get if you download the raw data for 2015). Use the documentation to figure out what each column represents. - Discuss what steps you would need to take to create the following plot. To start, don’t write any code, just develop a plan. Talk about what the dataset should look like right before you create the plot and what functions you could use to get the data from its current format to that format. (Hint: Functions from the
lubridate
package will be very helpful, includingyday
andwday
). - Discuss which of the variables in this dataset could be used to merge the dataset with other appropriate data, either other datasets in the FARS raw data, or outside datasets.
- Try to write the code to create the plot below. This will include some code for cleaning the data and some code for plotting. There is an example answer below, but I’d like you to try to figure it out yourselves first.
<IP_ADDRESS> Example R code
Here is example code for the section above:
library(tidyverse)
library(lubridate)
library(ggthemes)
accident <- read_csv("data/accident.csv") %>%
select(DAY:MINUTE) %>%
select(-DAY_WEEK) %>%
unite(date, DAY:MINUTE, sep = "-", remove = FALSE) %>%
mutate(date = dmy_hm(date),
yday = yday(date),
weekday = wday(date, label = TRUE, abbr = FALSE),
weekend = weekday %in% c("Saturday", "Sunday"))
accident %>%
filter(!is.na(yday)) %>%
group_by(yday) %>%
summarize(accidents = n(),
weekend = first(weekend)) %>%
ggplot(aes(x = yday, y = accidents, color = weekend)) +
geom_point(alpha = 0.5) +
xlab("Day of the year in 2015") +
ylab("# of fatal accidents") +
theme_few() +
geom_smooth(se = FALSE)


**Expressive Vector Engine**
Conversion Functions
These functions provide scalar and SIMD version of type conversions including specific, optimized float <==> int conversions and a general type casting facility. All these functions operates on the basis that a type conversion does not modify a [value](../concepts.html#value)'s cardinal, i.e a [scalar](../concepts.html#scalar_value) is converted to a [scalar](../concepts.html#scalar_value) and a [SIMD value](../concepts.html#simd_value) is converted to a [SIMD value](../concepts.html#simd_value) with the same cardinal.
(insert ../../crumbs.html here)
----------------------------------------------------------------------------------------------------
**List of conversion functions:**
[`eve::convert`](conversion/convert.html)
[`eve::tofloat`](conversion/tofloat.html)
[`eve::toint`](conversion/toint.html)
[`eve::touint`](conversion/touint.html)
----------------------------------------------------------------------------------------------------


ET
Home
Team
News
Code Playground
Email address
We'll never share your email with anyone else.
Password
Submit


A look back at Thierry Henry’s notorious handball 10 years on
The Frenchman broke Irish hearts on a controversial night in Paris.
The Republic of Ireland head into a Euro 2020 qualification battle with Denmark on Monday evening on the 10th anniversary of the notorious Thierry Henry handball which shattered their World Cup dreams.
Here, the PA news agency takes a look back at a controversial night which has become part of Ireland’s sporting folklore.
The build-up
Still feeling aggrieved at FIFA’s decision to change its mind over seeding the play-offs, Giovanni Trapattoni’s men had headed into the first game of their two-legged play-off in confident mood. However, they were disappointing in Dublin and went down to Nicolas Anelka’s lone strike to leave the French with the upper hand.
The game
Perhaps sensing that it was now or never, Ireland turned in perhaps their best performance under Trapattoni at the Stade de France and took a deserved 1-0 lead through skipper Robbie Keane’s first-half goal. They had chances to extend their lead but did not take them and, after 90 minutes, the aggregate scores were level and extra time was required.
Le main de Dieu
Thirteen minutes into extra time, Florent Malouda’s deep free-kick eluded defender Paul McShane and Henry controlled before turning the ball across goal for William Gallas to head home. Replays showed that the then Barcelona frontman had handled twice in bringing the ball under control, but crucially out of sight of referee Martin Hansson and his assistants and the Republic’s protests went unanswered.
What they said
Trapattoni: “Everybody saw the game. You know what happened.”
Keane: “They are all probably clapping hands, (Michel) Platini sitting up there on the phone to Sepp Blatter, probably texting each other, delighted with the result.”
Richard Dunne: “He admitted afterwards he handled it, but it doesn’t make me feel any better because we are not going to the World Cup finals.”
France manager Raymond Domenech: “I did not see a hand. It’s you people who are speaking about this.”
The aftermath
The Football Association of Ireland (FAI) lodged an official complaint with FIFA, although their calls for a replay and then to be included as the 33rd team at the finals both fell on deaf ears. Then chief executive John Delaney later revealed that the FAI had received a payment of five million Euros – at the time around £3.6million – initially in the form of an interest-free loan which ultimately did not have to be repaid in return for an agreement not to press their claim through the courts.
Karma?
France travelled to the finals in South Africa determined to make the most of their lucky escape but did not remain there for very long. Domenech’s players went on strike after Anelka was sent home following a row with the coach and the point they gained from an opening 0-0 draw with Uruguay in Cape Town was the only one they mustered, as defeats to Mexico and the hosts ended their participation at the first hurdle.


We're sorry but this site doesn't work properly without JavaScript enabled. Please enable it to continue.


--- title: Layers keywords: fastai sidebar: home_sidebar summary: "Contains basic layers used both in baseline and reformer" description: "Contains basic layers used both in baseline and reformer" nb_path: "nbs/01_layers.ipynb" ---
bs = 4
sl = 128
d = 64
x = torch.randn(bs, sl, d)
ff = Residual(PreNorm(d, FeedForward(d)))
out = ff(x)
assert (bs, sl, d) == out.size()
out.shape
fix_emb = TransformerEmbedding(256, 64, pos_enc='fixed')
abs_emb = TransformerEmbedding(256, 64, pos_enc='absolute')
axl_emb = TransformerEmbedding(256, 64, pos_enc='axial', axial_shape=(32,16))
print('Total number of parameters in embedding layer')
print(f'Fixed: {total_params(fix_emb)[0]}')
print(f'Absolute: {total_params(abs_emb)[0]}')
print(f'Axial: {total_params(axl_emb)[0]}')


Please wait...
ISSUED KEYS RECORD
more_vert
Issue Key
Delete Key Manually
Key Number
Capacity
Current Residue
Owner
Expire Date
Last Updated


Situated 40nm south of Haifa and 10 miles North of Tel Aviv, Herzliya is a modern town with a marina village which includes the Ritz Carlton, shops and restaurants.
This is a pleasent sailing destination and a safe place to leave your boat whilst you travel inland.
Be sure to email the marina<EMAIL_ADDRESS>and the Israeli Navy<EMAIL_ADDRESS>in advance.
The best port at which to clear in if coming from the north.
Herzliya Marina is a check-in port with a customs and immigration office in the marina building. Be careful not to arrive during the Shabat (Friday afternoon through Saturday night), as there is a substantial surcharge.
Last updated July 2014.
Herzliya Marina is a fully developed marina with 800 berths and all facilities. It is the largest marina in Israel, providing full haul out and yard services, yachting chandlers and good restaurants in the marina as well as a shopping mall.


Cisco Webex Download
Free Video Conferencing Online Meetings Cisco Webex
Webex Vdi Available For Download
Webex Help How To Download Webex Youtube
Cisco Webex Business Review Pcmag
Virtual Meetings Cisco Webex Meetings Cisco
Cisco Webex Meeting Center Download With This Program You Can Set Up A Meeting In Your Webex Account
Webex Support Center Create Meeting Instructional Resources Uw Whitewater Spaces
Cisco Webex Meetings Download 2021 Latest For Windows 10 8 7
Amazon Com Cisco Webex Meetings Appstore For Android
Cisco Webex Meeting Center
Cisco Webex Clemson University South Carolina
Video Conferencing Online Meetings Screen Share Cisco Webex
Webex Mac Os X Download Peatix
Administration Guide For Cisco Webex Meetings Virtual Desktop Software Release 40 8 Install The Cisco Webex Meetings Virtual Desktop Software Support Cisco
How To Join A Webex Meeting From Any Device
Join A Webex Online Meeting Ohsu
Satisfy Your Need For Speed With Cisco Webex Meetings Cisco Blogs
How Do I Schedule And Start Webex Meetings On Behalf Of Someone Else
Documentation Suny Cortland
How To Download A Webex Recording Of A Video Meeting
Cisco Webex Meeting Center ダウンロード
1
Webex Meetings の参加時のトラブルシューティング Cisco Community
Cisco Webex Meetings Apps On Google Play
How To Download Webex For Microsoft Outlook
Joining A Webex Meeting
How Can I Record A Webex Meeting
Video Conferencing Use The Cisco Webex Meetings Desktop App As A Guest
Video Conferencing Record Screen Activity Using The Webex Recorder
Download Webex Meeting Youtube


"Snoopy's Dog House, a family owned business is a place where we can take our much spoiled pets and know
that they will be treated with the respect and dignity that they deserve.
They groomed and took care of our "Little Chealsea" for 16 1/2 years. We knew that we could trust them with
her care even in her aging years. Now we have a new puppy "Maggie" and we have already started with her
grooming at Snoopy's. They are very knowledgable in the grooming of all breeds of cat and dogs. As pet owners
that have professionally shown pets themselves, they are familiar with the whole grooming process. We would
recommend Snoopy's to anyone that cherishes their pet."
- Hurshel and Paulette Johnson
"Snoopy's Dog House has taken care of us for the past 13 years. Pretzel is our third dog to bring to Snoopy's and they have always been great! They are always friendly and make your pet feel as comfortable as possible. Snoopy's Dog House will continue to see us for many years to come."
- Carmen Howard
"Nothing but the Best in Customer Service for our 20 plus years. Snoopy's give the highest personal service and caring for their clients that they service."
- D. R. Buzan


Choose Files
Choose a files to convert
Click choose files or drag and drop the files onto this window.
Choose Files


heart_rate_monitor module¶
-
class
heart_rate_monitor.
ECGData
(file_name, threshold_factor=0, user_input=0)¶ Bases:
object
-
beats_times
()¶ Function returns a numpy array of the times at which a beat occurred :returns self.beats: times at which beats occur
-
correlate_wavelet_dataset
()¶ Function correlates wavelet with complete set of filtered data :returns self.correlted_data: numpy 1D array containing correlation data
-
duration
()¶ Function returns time of ECG :returns max(self.time_dataset.values): time duration of ECG
-
find_max_voltage_index
()¶ Function returns index at which the voltage value in the ECG numpy array is maximum :returns self.max_voltage_index: index at maximum voltage value
-
find_peaks
(threshold_factor)¶ Function returns voltage at each beat :returns self.beat_voltage: array with voltages at beat
-
find_peaks_index
()¶ Function returns index for each beat voltage :returns self.peaks_index: numpy 1D array containing correlated and positive data
-
get_wavelet
()¶ Function obtains clipped part of the complete dataset that contains 60 voltage and time values :returns self.wavelet: clipped portion of the filtered, complete dataset containing 60 values
-
make_json
(file_name)¶ Function returns json file containing all required ECG Data :param file_name: name of file for data to be extracted from
-
mean_hr_bpm
(user_input)¶ Function returns average heart rate over user specified input :param user_input: time in minutes :returns self.hr: average heart rate in beats per second
-
moving_average
()¶ Function takes numpy array with raw ECG information and performs a moving average function over it with a window of 10 :returns self.ma_dataset: numpy array after the moving average is performed
-
num_beats
()¶ Function returns the number of beats that occurred :returns len(self.beat_voltage): length of numpy array
-
voltage_extremes
()¶ Function returns maximum and minimum voltage values in ECG :returns self.max_min: tuple containing maximum and minimum voltage values
-


Home
Chose your item
Time
Starts
Now.
Tailored Jeans
$19.99
Some text about the jeans..
Add to Cart
Tailored Jeans
$19.99
Some text about the jeans..
Add to Cart


What is WayOway?
WayOway is a simple way to plan data gathering for your research project. Create a project, add the items you need and the forms you need to complete. Then monitor and update your project as it progresses. We intentionally kept its functionality, design and code to a minimum, so we can focus on building robust foundations for your project. This allows you to build custom features on top. WayOway gives you a flexible structure to build your research project upon and can handle many different types of data. For example, you can collect respones, samples, questionaires, surveys, specimens, locations, images etc in both remote (offline) locations or in the lab. For physical samples, you can print labels and keep track of your samples as they pass through your analysis processes.
WayOway uses the Hoodie javascript framework and implements all its features for you:- Add, update, remove and find data
- Create an account, Login, Sign out, destroy account
- Reset password
- Change Username and Password
Find out more about Hoodie:
Your data is held in a couchdb based database. Allowing access directly via a web api or via a python or R libraries.
WayOway is an open-source project hosted here:
,
but you can contribute much more than just code
Questions? Ideas?
Talk to us :) open an isue: @hoodiehq


|
Introduction
This is an extremely high level tutorial. Before you start, make
sure you have learned basic TorqueScript syntax, T2D Editor Basics, and
behavior concepts. If you are completely new to Torque, specifically
T2D, I recommended you go through the other tutorials found in this
documentation first.
Making Our Fish Move
In this section we are introduced to behaviors in the process of
creating a movement behavior. We set this movement behavior up to take
key input to allow us to configure our movement keys inside the Level
Builder itself. First you want to create a new project. I named mine "MyFishTutorial".
Next, bring up the resources menu (Project->Resources...) and add the "fishArt" resource. Now you should have a fresh project with a set of fish images. We are
ready to begin making our quick, behavior based, fish game.
Fish Controls Behavior
For our fish game we need only three behaviors. Behaviors are
sets of scripts we can create and then assign to various object inside
the Level Builder. Not only can we assign them but we can configure
different values to customize the behavior.
We will need to create three different behaviors for this quick and simple game, as follows:
- Controls, for using the keys to control our player
- Our interactive objects, to allow them to add and subtract life from our player
- Life management, adding/subtracting life from our player as well as slowly draining life
Now that we know what we need, our next steps will be adding the
scripts for each of these and then testing them. We are going to add a
behavior that we want to show up in the Level Builder, so close down
T2D. Create a script file called "fishControls.cs" inside your Project
Directory/game/behaviors folder, using the following code:
if(!isObject(fishControlsBehavior))
{
%template = new BehaviorTemplate(fishControlsBehavior);
%template.friendlyName = "Fish Tutorial Controls";
%template.behaviorType = "Input";
%template.description = "Movement control for our Fish Tutorial";
%template.addBehaviorField(upKey, "Key to bind to upward movement", keybind, "Up");
%template.addBehaviorField(downKey, "Key to bind to downward movement", keybind, "Down");
%template.addBehaviorField(leftKey, "Key to bind to left movement", keybind, "Left");
%template.addBehaviorField(rightKey, "Key to bind to right movement", keybind, "Right");
%template.addBehaviorField(verticalSpeed, "Speed when moving vertically", float, 15.0);
%template.addBehaviorField(horizontalSpeed, "Speed when moving horizontally", float, 30.0);
}
function fishControlsBehavior::onBehaviorAdd(%this)
{
if (!isObject(moveMap))
return;
// bind our keys on the keyboard
moveMap.bindObj(getWord(%this.upKey, 0), getWord(%this.upKey, 1), "moveUp", %this);
moveMap.bindObj(getWord(%this.downKey, 0), getWord(%this.downKey, 1), "moveDown", %this);
moveMap.bindObj(getWord(%this.leftKey, 0), getWord(%this.leftKey, 1), "moveLeft", %this);
moveMap.bindObj(getWord(%this.rightKey, 0), getWord(%this.rightKey, 1), "moveRight", %this);
%this.up = 0; %this.down = 0; %this.left = 0; %this.right = 0;
}
function fishControlsBehavior::moveUp(%this, %val)
{
%this.up = %val;
%this.updateMovement();
}
function fishControlsBehavior::moveDown(%this, %val)
{
%this.down = %val;
%this.updateMovement();
}
function fishControlsBehavior::moveLeft(%this, %val)
{
%this.left = %val;
%this.updateMovement();
}
function fishControlsBehavior::moveRight(%this, %val)
{
%this.right = %val;
%this.updateMovement();
}
function fishControlsBehavior::updateMovement(%this)
{
// figure out what speed in what direction we are moving based on key input
%this.owner.setLinearVelocityX((%this.right - %this.left) * %this.horizontalSpeed);
%this.owner.setLinearVelocityY((%this.down - %this.up) * %this.verticalSpeed);
// check whether or not we need to flip our direction
if((%this.left != %this.right) && (%this.left != %this.owner.getFlipX()))
%this.owner.setFlipX(%this.left);
}
Testing Movement Behavior
The behavior you just created adds the ability to control the
object it is attached to. You will be able to choose which keys move
the object up, down, left, and right. It even allows you to add default
values for the behavior fields. Time to try it out. Re-open T2D and
load up your project again.
Click and drag in a fish you want to use as your player (be
sure to drag in an animation), resize it if necessary. Click the Edit
tab. You should see a behaviors roll out, click the drop-down and
select the "Fish Tutorial Controls" behavior, then click the add button
(the green plus).
Now you should see the selected behavior listed with six
different properties. The first four properties are our various
keyboard settings for movement keys. Click the dropdown for each key
and set the key you want for each direction of movement. I am using the
typical W = up, S = down, A = left, D = right.
The last two properties are the horizontal and vertical
movement speeds. These should be safe at the default values though you
can tweak them however you want for your game. Time to test! Click the
play button and move your fish around. We have implemented our first
behavior for the Fish game, two more to go.
Giving Our Fish Health
Now that we have our fish moving, we need to add some gameplay elements. We are going to add three different elements:
- Health management to our fish, including losing and gaining health as well as slowly losing it over time
- Bubbles that give health
- Mines that damage health
To accomplish this we will create two more behaviors and then assign them in the Level Builder.
Fish Health Management Behavior
We are going to add some more behaviors that we want to show up
in the Level Builder, so close down T2D. Create a script file called
"fishHealthManagement.cs" inside your Project Directory/game/behaviors
folder, using the following code:
if(!isObject(fishHealthManagementBehavior))
{
%template = new BehaviorTemplate(fishHealthManagementBehavior);
%template.friendlyName = "Fish Tutorial Health Management";
%template.behaviorType = "Gameplay";
%template.description = "Health management for our Fish Tutorial";
%template.addBehaviorField(health, "The current health of the fish", float, 100);
%template.addBehaviorField(maxHealth, "The max health of the fish", float, 100);
%template.addBehaviorField(healthLossTime,
"The ammount of seconds the fish should lose health", float, 3.00);
%template.addBehaviorField(healthLossValue,
"The ammount of health the fish loses over time", float, -10);
}
function fishHealthManagementBehavior::onBehaviorAdd(%this)
{
// calculate the min health
%this.minHealth = %this.maxHealth * 0.20;
// set the max width and height to the starting width and height
%this.maxWidth = %this.owner.getWidth();
%this.maxHeight = %this.owner.getHeight();
// if we are to take damage we must receive collision
%this.owner.setCollisionActive(false, true);
// set a flag to signify that we have health
%this.owner.hasHealth = true;
%this.owner.enableUpdateCallback();
%this.lastSceneTime = 0;
}
function fishHealthManagementBehavior::updateHealth(%this, %modifier)
{
// modify our health based on the modifier
%this.health += %modifier;
// lets clamp the values between the min health and the max health
if(%this.health > %this.maxHealth)
{
%this.health = %this.maxHealth;
}
else if(%this.health < %this.minHealth)
{
%this.health = %this.minHealth;
}
// call the update size method
%this.updateSize();
}
function fishHealthManagementBehavior::updateSize(%this)
{
// get a multiplier based on current health out of max health
%healthMultiplier = %this.health / %this.maxHealth;
// calc the new sizes
%newWidth = %this.maxWidth * %healthMultiplier;
%newHeight = %this.maxHeight * %healthMultiplier;
// update the size on the owner of this behavior
%this.owner.setSize(%newWidth, %newHeight);
}
function fishHealthManagementBehavior::onUpdate(%this)
{
// grab the scene time and figure out how long its been since the last update
%sceneTime = %this.owner.scenegraph.getSceneTime();
%diff = %sceneTime - %this.lastSceneTime;
// if its been the specified loss time then let decrease the health
if(%diff >= %this.healthLossTime)
{
// update the Health with the loss value then store this time as the last time
%this.updateHealth(%this.healthLossValue);
%this.lastSceneTime = %sceneTime;
}
}
Modify Health Behavior
Create a script file called "modifyHealth.cs" inside your Project Directory/game/behaviors folder, using the following code:
if(!isObject(fishModifyHealthBehavior))
{
%template = new BehaviorTemplate(fishModifyHealthBehavior);
%template.friendlyName = "Fish Tutorial Modify Health";
%template.behaviorType = "Gameplay";
%template.description = "Modify health control for our Fish Tutorial";
%template.addBehaviorField(healthModifier,
"Value in which to modify the health by", integer, 2);
%template.addBehaviorField(minSpeed, "The min random speed to float down", float, 5);
%template.addBehaviorField(maxSpeed, "The max random speed to float down", float, 15);
}
function fishModifyHealthBehavior::onBehaviorAdd(%this)
{
// enable collision for sending and then enable the callback
%this.owner.collisionActiveSend = true;
%this.owner.collisionActiveReceive = false;
%this.owner.collisionCallback = true;
// disable physics
%this.owner.collisionPhysicsSend = false;
%this.owner.collisionPhysicsReceive = false;
// set the world limit
%this.owner.worldLimitMin = "-50 -45";
%this.owner.worldLimitMax = "50 45";
%this.owner.worldLimitMode = "NULL";
%this.owner.worldLimitCallback = true;
}
function fishModifyHealthBehavior::onAddToScene(%this, %scenegraph)
{
%this.startPositionY = %this.owner.getPositionY();
%this.respawn();
}
function fishModifyHealthBehavior::onCollision(%this, %dstObj, %srcRef, %dstRef, %time,
%normal, %contacts, %points)
{
// lets update the health of the object we are colliding with
if(%dstObj.hasHealth)
{
%dstObj.updateHealth(%this.healthModifier);
%this.respawn();
}
}
function fishModifyHealthBehavior::onWorldLimitt2dSceneObject::onWorldLimit(%this,
%limitMode, %limit)(%this, %mode, %limit)
{
switch$(%limit)
{
case "bottom":
%this.respawn();
}
}
function fishModifyHealthBehavior::respawn(%this)
{
%this.owner.setPositionY(%this.startPositionY);
%this.owner.setLinearVelocityY(getRandom(%this.minSpeed, %this.maxSpeed));
}
Healing the Player
Now that we have our two behaviors, time to use them!
Open up T2D and load up your project again. Drag in a bubble static sprite (should be the last frame in the 14
frame image) into the level just above the camera area (the camera area
is the outer blue box, resize the bubble if necessary.
Then click on the Edit Tab. Now add the "Fish Tutorial Modify Health"
behavior just like you added the fish control behavior (select it from
the drop down and click the plus button). Change the "healthModifier"
value to 10 (or another value of your choosing, keep in mind our fish
will default to 100 health).
Harming the Player
Next, drag a mine static sprite into the level just above the
camera area, and a little to the side of your bubble. Then click on the
Edit Tab.
Add the "Fish Tutorial Modify Health" behavior just like you
added the fish control behavior (select it from the drop down and click
the plus button). Change the "healthModifier" value to -10 (this way it
will subtract life).
Adding Health and Final Testing
Our final step to is to select our player fish, click the Edit
Tab, and add the "Fish Tutorial Health Management" behavior. This
behavior has a few different properties that I will explain:
- health: this is the starting health
- maxHealth: The maximum health of the fish
- healthLossTime: The amount of seconds the fish will lose health
- healthLossValue: The amount of health the fish will lose at the interval of healthLossTime
The default values should work fine, though feel free to tweak
to your delight. That is what T2D and behaviors are all about. When you
are ready, hit the play button. You should be able to swim around and
gather bubbles to increase in health and size and get hit by a mine to
decrease in health and size.
Adding Some Backgrounds
Now that our base gameplay is done we add some background images as
well as making minor changes to customize our fish game. There are
three layers of background images:
- Our water background
- First layer of rocks
- Second layer of rocks
To fully take advantage of these layers we need to layer our
fish and both our bubble and mine sprites properly. If we layer our
fish player properly we will get a great effect of our fish swimming
between the rocks. Keep in mind the lower number layers are in front.
The higher the layer number the further back it is. So layer 0 is all
the way up front, while layer 31 (there are 32 total layers) is the
furthest back.
Player Fish Layer: Set our player object to Layer 15
Bubble/Mine Layer: Set our mine and bubble objects to Layer 14
Background Layer: Add the background image, and set it to Layer 31
Far Rocks Layer: Add the rocksfarimage sprite, and set it to Layer 30
Near Rocks Layer: Add the rocksnearimage sprite, and set it to Layer 12
Now if you play your level you should be able to swim your
fish between the rocks. The bubbles and mines should also layer
properly now as well. Now comes the fun part, you can take what we have
done and customize the game. Here are various ways to customize the
game:
- Create a behavior that spawns multiple bubbles and mines that fall to the bottom
- Resize the backgrounds to fit in the level better, or even much larger to make the world seem bigger
- Change the values of any of the behaviors to change the game play experience, such as:
- Giving the fish much more life
- Making the bubbles give much more life and duplicate the mines so there are lots of mines and very few bubbles
- Change the player speed to make it move around faster or slower
- Increase or decrease the min and max speeds for the bubbles or mines
- Create a new behavior to have other fish swim back and forth
- Decorate the background with various other sprites and/or other effect
Conclusion
The focus of this tutorial was not to complete a fully polished
product, but to show how quickly you can prototype a game using
Behaviors, T2D's Editors, and the Resource Manager. There are other
tutorials, such as the non-behavior Fish Game and Fish Demo, which you
can use to add on to this sample game.
|


Ocean [al41], ocean, in The Inner Sea
Last updated Turn #149
Ocean [al41], ocean, in The Inner Sea
------------------------------------------------------------------------
14: Pleasure Sloop [v5n], ship, 30 hulls, 1 rowing port, 90 sails, arrived
14: from Ocean [al40], owner:
14: Dorothy [v1b], Pen, with one worker, 94 sailors
14: Apollo [b5e], Imperial Empire, with 11 workers, 63 lt. xbowmen,
14: accompanied by:
14: Lutra [a4w], Imperial Empire, with 11 paladins, 19 pikemen
14: Lungor [y4r], Imperial Empire
14: Merbit [e2m], Imperial Empire
14: Pink Blazer [n8e], Imperial Empire
14: Kiki Ghelfond [p2w], Imperial Empire
14: Hera [g3f], Imperial Empire, with three soldiers, 28 pikemen
14: Thor [r4g], Imperial Empire, with 31 pikemen
14: Eos [a2t], Imperial Empire, with ten workers, 48 lt. xbowmen
14: Khaki Blazer [w4j], Imperial Empire, with four soldiers,
14: three pikemen, 24 lt. foot
14: Banian [e7n], Imperial Empire, duke, the High Priest of the
14: Church of Eres [151], with one soldier, ten angels
15: Pleasure Sloop [v5n], ship, 30 hulls, 1 rowing port, 90 sails,
15: departed for Ocean [al42].
Routes leaving Ocean:
North, to Ocean [ak41], 6 days
East, to Ocean [al42], 6 days
South, to Ocean [am41], 6 days
West, to Ocean [al40], 6 days


Gateway
BOT
18-Jul-18 09:48 AM
[telegram] <Crypto> Ah okey
[telegram] <Crypto> I Tried it out, now it says you have no rewards to claim. Shouldn’t it start giving me new rewards right away?
[telegram] <Lennart> After one hour
[telegram] <Lennart> This is the reason why claiming daily is not effective
[telegram] <Lennart> You lose 1 hour of generating interest
[telegram] <Lennart> I claim every two weeks
[telegram] <Crypto> Ah okey Thanks:)
[telegram] <Lennart> Welcome
[telegram] <Crypto> My hyperdex is stuck
[telegram] <Crypto> Cant click on the x
[telegram] <Crypto> I just downloaded it, wanted to test atomic swaps
[telegram] <Acura360> There was a quick bug in the new release (re @Crypto: )
[telegram] <Acura360> The bugfix is now implemented as i know
hyperdex - Desktop App for BarterDEX
[telegram] <Lennart> Is the direct swap Feature already enabled? Wanna test this
[telegram] <Lennart> This might be the gamechanger


Tooltips: Explicit Positioning Content - no offsets
These tooltips test various positioning and alignments. Valid positions are left/right/top/bottom. Valid alignments are left/right/top/bottom/center. Left align means left sides should line up. Right align means right sides should line up. Center align means centers should line up.


ABOUT US
Our website follows the safe harbor provisions of 17 U.S.C. 512 otherwise known as Digital Millennium Copyright Act ("DMCA"). To file a copyright infringement notification with us, you will need to send a written communication that includes substantially the following: A physical or electronic signature of a person authorized to act on behalf of the owner of an exclusive right that is allegedly infringed. Identification of the copyrighted work claimed to have been infringed, or, if multiple copyrighted works at a single online site are covered by a single notification, a representative list of such works at that site. Identification of the material that is claimed to be infringing or to be the subject of infringing activity and that is to be removed or access to which is to be disabled, and information reasonably sufficient to permit the service provider to locate the material. Providing URLs in the body of an email is the best way to help us locate content quickly.