◄ Slovakia ►
Individual results
| Year | Contestant | Rank | Award |
|---|---|---|---|
| 2023 | Adam Džavoronok | 92 | Silver Medal |
| 2023 | Eduard Mrug | 228 | Honourable Mention |
| 2023 | Matej Zigo | 238 | Honourable Mention |
| 2023 | Tomáš Kubrický | 256 | Honourable Mention |
| 2022 | Pavol Pivko | 123 | Bronze Medal |
| 2022 | Csaba Dékány | 139 | Bronze Medal |
| 2022 | Adam Džavoronok | 157 | Bronze Medal |
| 2022 | Matej Zigo | 281 | Honourable Mention |
| 2021 | Jozef Csipes | 75 | Silver Medal |
| 2021 | Pavol Pivko | 114 | Bronze Medal |
| 2021 | Adam Džavoronok | 127 | Bronze Medal |
| 2021 | Pavol Šimkovič | 149 | Bronze Medal |
| 2021 | Štefan Slavkovský | 166 | Bronze Medal |
| 2019 | Ronald Doboš | 79 | Silver Medal |
| 2019 | Jonáš Dujava | 80 | Silver Medal |
| 2019 | Róbert Jurčo | 127 | Bronze Medal |
| 2018 | Okánik Martin | 155 | Bronze Medal |
| 2018 | Švrčková Jana | 158 | Bronze Medal |
| 2018 | Kopunec Matúš | 167 | Bronze Medal |
| 2018 | Róbert Jurčo | 213 | Honourable Mention |
| 2017 | Jozef Lipták | ≥1 | Gold Medal |
| 2017 | Čermaák Filip | ≥137 | Bronze Medal |
| 2017 | Juraj Halabrin | ≥137 | Bronze Medal |
| 2017 | Ivan Grega | ≥137 | Bronze Medal |
| 2017 | Okánik Martin | ≥137 | Bronze Medal |
| 2016 | Martin Gažo | 67 | Silver Medal |
| 2016 | Jozef Lipták | 123 | Bronze Medal |
| 2016 | Juraj Májek | 129 | Bronze Medal |
| 2016 | Juraj Halabrin | 134 | Bronze Medal |
| 2016 | Adam Škrlec | 145 | Bronze Medal |
| 2015 | Martin Gažo | 88 | Silver Medal |
| 2015 | Filip Ayazi | 143 | Bronze Medal |
| 2015 | Jozef Bucko | 168 | Bronze Medal |
| 2015 | David Bugar | 173 | Bronze Medal |
| 2015 | Miroslav Gasparek | 219 | Honourable Mention |
| 2014 | Patrik Turzák | 76 | Silver Medal |
| 2014 | Jakub Kvorka | 117 | Silver Medal |
| 2014 | Filip Ayazi | 162 | Bronze Medal |
| 2014 | Ján Ondráš | 198 | Bronze Medal |
| 2014 | Matej Badin | 205 | Bronze Medal |
| 2013 | Patrik Turzák | 119 | Bronze Medal |
| 2013 | Tomáš Gonda | 150 | Bronze Medal |
| 2013 | Ján Ondráš | 168 | Bronze Medal |
| 2013 | Jakub Kvorka | 180 | Bronze Medal |
| 2013 | Ladislav Ürge | 313 | |
| 2012 | Peter Kosec | 71 | Silver Medal |
| 2012 | Patrik Svancara | 90 | Silver Medal |
| 2012 | Patrik Turzák | 94 | Silver Medal |
| 2012 | Andrej Vlcek | 125 | Bronze Medal |
| 2012 | Samuel Beznák | ≥272 | |
| 2011 | Jakub Safin | 48 | Gold Medal |
| 2011 | Jan Pulmann | 49 | Gold Medal |
| 2011 | Jakub Kocak | 51 | Gold Medal |
| 2011 | Matej Balog | 118 | Silver Medal |
| 2011 | Peter Kosec | 148 | Bronze Medal |
| 2010 | Jakub Kocak | 54 | Silver Medal |
| 2010 | Dusan Nemec | 59 | Silver Medal |
| 2010 | Eugen Hruska | 66 | Silver Medal |
| 2010 | Jan Bogar | 81 | Silver Medal |
| 2010 | Ladislav Baco | 120 | Bronze Medal |
| 2009 | Eugen Hruska | 39 | Gold Medal |
| 2009 | Michal Spisiak | 149 | Bronze Medal |
| 2009 | Stanislav Hrivnak | 150 | Bronze Medal |
| 2009 | Peter Vanya | 151 | Bronze Medal |
| 2009 | Dusan Nemec | 157 | Bronze Medal |
| 2008 | Vladimír Boža | 46 | Gold Medal |
| 2008 | Samuel Hapák | 67 | Silver Medal |
| 2008 | Eugen Hruska | 103 | Bronze Medal |
| 2008 | Filip Kubina | 153 | Bronze Medal |
| 2008 | Pavol Otto | 248 | Honourable Mention |
| 2007 | Tomáš Bzdušek | 7 | Gold Medal |
| 2007 | Vladimír Boža | 44 | Silver Medal |
| 2007 | Samuel Hapák | 89 | Bronze Medal |
| 2007 | József Konczer | 95 | Bronze Medal |
| 2007 | Michal Kovaľ | 153 | Honourable Mention |
| 2006 | Jakub Imriska | 64 | Silver Medal |
| 2006 | Marcela Hrda | 113 | Bronze Medal |
| 2006 | József Konczer | 180 | Honourable Mention |
| 2006 | Tomáš Bzdušek | 182 | Honourable Mention |
| 2006 | Peter Peresini | 186 | Honourable Mention |
| 2005 | Anton Repko | 13 | Gold Medal |
| 2005 | Jakub Zavodny | 60 | Silver Medal |
| 2005 | Daniel Bozik | 142 | Honourable Mention |
| 2005 | Jan Lalinsky | 144 | Honourable Mention |
| 2005 | Matus Tejiscak | 157 | Honourable Mention |
| 2004 | Peter Rakyta | ≥32 | Silver Medal |
| 2004 | Peter Matak | ≥67 | Bronze Medal |
| 2004 | Boris Gazovic | ≥135 | Honourable Mention |
| 2003 | Peter Rakyta | 61 | Bronze Medal |
| 2002 | Tomas Dzetkulic | 85 | Bronze Medal |
| 2002 | Peter Rakyta | 111 | Bronze Medal |
| 2002 | Peter Koltai | 151 | Honourable Mention |
| 2002 | Tomas Skeren | 195 | Honourable Mention |
| 2001 | Branislav Novotny | 118 | Honourable Mention |
| 2001 | Tomas Dzetkulic | 135 | Honourable Mention |
| 2001 | Zoltan Mics | 139 | Honourable Mention |
| 2001 | Stanislav Miklik | 161 | Honourable Mention |
| 2000 | Juraj Suchar | 74 | Honourable Mention |
| 2000 | Keve Kurucz | 88 | Honourable Mention |
| 2000 | Peter Cendula | 96 | Honourable Mention |
| 2000 | Peter Molnar | 121 | Honourable Mention |
| 1999 | Richard Kralovic | 69 | Silver Medal |
| 1999 | Keve Kurucz | 112 | Bronze Medal |
| 1999 | Vaclav Bednar | 148 | Bronze Medal |
| 1999 | Peter Molnar | 180 | Honourable Mention |
| 1999 | Juraj Suchar | 201 | Honourable Mention |
| 1998 | Peter Svrcek | 20 | Silver Medal |
| 1998 | Matus Medo | 50 | Bronze Medal |
| 1998 | Nagaj Daniel | 58 | Bronze Medal |
| 1998 | Tamas Martin | 91 | Honourable Mention |
| 1998 | Cyril Adamuscin | 98 | Honourable Mention |
| 1997 | Peter Svrcek | 4 | Gold Medal |
| 1997 | Matus Medo | 41 | Silver Medal |
| 1997 | Cyril Adamuscin | 69 | Bronze Medal |
| 1997 | Karol Gregor | 78 | Bronze Medal |
| 1996 | Peter Svrcek | 25 | Silver Medal |
| 1996 | Martin Plesch | 62 | Bronze Medal |
| 1996 | Jan Rusz | 82 | Bronze Medal |
| 1996 | Martin Kobyl'an | 108 | Honourable Mention |
| 1996 | Stefan Godis | 136 | Honourable Mention |
| 1995 | Peter Macak | 34 | Silver Medal |
| 1995 | Dalibor Blazek | 81 | Bronze Medal |
| 1995 | Martin Pal | 111 | Honourable Mention |
| 1995 | Martin Plesch | 112 | Honourable Mention |
| 1995 | Tamas Varga | 143 | Honourable Mention |
| 1994 | Frantisek Mizera | 36 | Honourable Mention |
| 1994 | Milos Gaj | 49 | Honourable Mention |
| 1993 | Milos Gaj | 26 | Silver Medal |
| 1993 | Milos Volauf | 42 | Bronze Medal |
| 1993 | Kamil Budinsky | 73 | Honourable Mention |
Results may be incomplete or incorrect.
Please send the relevant information to the webmaster:
<EMAIL_ADDRESS>If you are a participant you can send a personal page URL to be attached to your name.
If you are a participant you can send a personal page URL to be attached to your name.


About and Contact¶
License¶
1D DFT solver is distributed under the Apache License 2.0 license ??
Authors¶
- Ryan Pederson
Ryan Pederson is a PhD Candidate at University of California Irvine working with Kieron Burke’s theoretical chemistry group.
Acknowledgments¶
Mention Funding here


mp3 sample audio + controls + play one/1 time
mp3 sample audio + autoplay hidden (without controls )
mp3 sample audio + preload + controls + autoplay


History
15:45
[05/30/2017] 04:37 AM
[05/30/2017] 07:50 AM
[05/29017] 08:12 PM
MAIN
MAP
NEARBY ( <5km)
Display only habits with location within 5km


--- layout: archive title: Zulip Chat Archive permalink: /stream/113488-general/topic/repo.html ---
where can i find repo of introduction_to_lean? just want to fix a trivial typo ;p.
What link are you looking at?
https://leanprover.github.io/introduction_to_lean/
My best guess is https://github.com/avigad/introduction_to_lean but note that that tutorial is very out of date, so it's unlikely to be updated. You'd be better off looking at the resources at the community-maintained "Lean links" here.
{% endraw %}Last updated: Apr 24 2020 at 19:02 UTC


1.5 hour race on a legendary Monza F1 track
100km, 1h, sprints of 333.33m and 1000m and an up-hill race
Last year's program was 1 h, 3 h, 6 h and 200 m velodrome sprint
Trackcrazy - 24 h endurance event on a flat 2.72 km Alastaro Circuit. Recumbents welcome!
Recumbent summer meeting in Finland
1000 m standing start, 1 h, "course à l'Australienne", rando 100 km
200 m flying start, 1 hour, 6 hours, 12 hours
6 and 24 hours races on a legendary 3.8 km Paul Ricard track. Recumbents are welcome!
Individual and Team (duo) Time Trials, very flat road, see Openrunner ID 6698287


Motivation.
Pleasure, Passion, Purpose
There are three levels of happiness that we can attain in life, but they have differing levels of sustainability.
Pleasure is probably the easiest to attain, and the one that most people think will bring them happiness in life. But this is also the most short-lived form of happiness, which causes people to then have to find the next new thing to provide them pleasure.
The happiness of passion is derived from finding something that you love to do and then doing that in life, and to a degree that is hard enough to be challenging but easy enough to be accomplished. This provides one with a sense of fulfillment which is longer lasting that pleasure.
Finally, there is happiness from purpose. This is the most sustainable and comes from taking part in something bigger than yourself, creating good in something that may not, and probably doesn't, revolve around you.
My core values are...
- Honesty
- Compassion
- Empathy
- Integrity
- Kindness
- Creativity
- Humour
I stand for...
- Embracing fear and vulnerability
- Being there for others
- Common sense
- Wholeheartedness
Resiliency
I'm a big believer in the importance of resiliency and have come across it in several different realms of both personal and professional life.
In becoming a therapist, we learned that we needed to be resilient to the hardships of others and to the stresses of caring for those others, otherwise we wouldn't be able to do as good a job.
Likewise in our personal life, we need to be resilient to the emotional bumps and bruises that are unavoidable in life. Things don't always work out how we want but if we have resiliency then we can still succeed despite these setbacks.
I think a big part of building resiliency comes from being mindful. If we are aware of our emotions, and aware of the antecedents, and we get perspective on everything going on, then often we can take actions to protect ourselves.


|
cds
1.2.0
|
Michael's hash set. More...
#include <cds/intrusive/michael_set.h>
Public Types | |
| typedef OrderedList | ordered_list |
| type of ordered list used as a bucket implementation | |
| typedef ordered_list | bucket_type |
| bucket type | |
| typedef Traits | options |
| Traits template parameters. | |
| typedef ordered_list::value_type | value_type |
| type of value stored in the list | |
| typedef GC | gc |
| Garbage collector. | |
|
typedef ordered_list::key_comparator | key_comparator |
| key comparision functor | |
| typedef ordered_list::disposer | disposer |
| Node disposer functor. | |
|
typedef cds::opt::v::hash_selector < typename options::hash > ::type | hash |
| Hash functor for value_type and all its derivatives that you use. | |
| typedef options::item_counter | item_counter |
| Item counter type. | |
|
typedef cds::details::Allocator < bucket_type, typename options::allocator > | bucket_table_allocator |
| Bucket table allocator. | |
| typedef michael_set::details::iterator < bucket_type, false > | iterator |
| Forward iterator. | |
| typedef michael_set::details::iterator < bucket_type, true > | const_iterator |
| Const forward iterator. |
Public Member Functions | |
| iterator | begin () |
| Returns a forward iterator addressing the first element in a set. | |
| iterator | end () |
| Returns an iterator that addresses the location succeeding the last element in a set. | |
| const_iterator | begin () const |
| Returns a forward const iterator addressing the first element in a set. | |
| const_iterator | end () const |
| Returns an const iterator that addresses the location succeeding the last element in a set. | |
| MichaelHashSet (size_t nMaxItemCount, size_t nLoadFactor) | |
| Initialize hash set. | |
| ~MichaelHashSet () | |
| Clear hash set and destroy it. | |
| bool | insert (value_type &val) |
| Inserts new node. | |
| template<typename Func > | |
| bool | insert (value_type &val, Func f) |
| Inserts new node. | |
| template<typename Func > | |
| std::pair< bool, bool > | ensure (value_type &val, Func func) |
Ensures that the val exists in the set. | |
| bool | unlink (value_type &val) |
Unlink the item val from the set. | |
| template<typename Q > | |
| bool | erase (const Q &val) |
| Delete the item from the set. | |
| template<typename Q , typename Func > | |
| bool | erase (const Q &val, Func f) |
| Delete the item from the set. | |
| template<typename Q , typename Func > | |
| bool | find (Q &val, Func f) |
Find the key val . | |
| template<typename Q > | |
| bool | find (Q const &val) |
Find the key val . | |
| void | clear () |
| Clears the set (non-atomic) | |
| bool | empty () const |
| Checks if the set is empty. | |
| size_t | size () const |
| Returns item count in the set. | |
| size_t | bucket_count () const |
| Returns the size of hash table. |
Protected Member Functions | |
| template<typename Q > | |
| size_t | hash_value (const Q &key) const |
Calculates hash value of key . | |
| template<typename Q > | |
| bucket_type & | bucket (const Q &key) |
Returns the bucket (ordered list) for key . |
Protected Attributes | |
| item_counter | m_ItemCounter |
| Item counter. | |
| hash | m_HashFunctor |
| Hash functor. | |
| bucket_type * | m_Buckets |
| bucket table |
Michael's hash set.
Michael's hash table algorithm is based on lock-free ordered list and it is very simple. The main structure is an array T
of size M
. Each element in T
is basically a pointer to a hash bucket, implemented as a singly linked list. The array of buckets cannot be dynamically expanded. However, each bucket may contain unbounded number of items.
Template parameters are:
GC
- Garbage collector used. Note the GC
must be the same as the GC used for OrderedList
OrderedList
- ordered list implementation used as bucket for hash set, for example, MichaelList, LazyList. The intrusive ordered list implementation specifies the type T
stored in the hash-set, the reclamation schema GC
used by hash-set, the comparison functor for the type T
and other features specific for the ordered list.Traits
- type traits. See michael_set::type_traits for explanation. Instead of defining Traits
struct you can use option-based syntax with michael_set::make_traits metafunction.Hash functor Some member functions of Michael's hash set accept the key parameter of type Q
which differs from value_type
. It is expected that type Q
contains full key of value_type
, and for equal keys of type Q
and value_type
the hash values of these keys must be equal too. The hash functor Traits::hash
should accept parameters of both type:
How to use
First, you should define ordered list type to use in your hash set:
Second, you should declare Michael's hash set container:
Now, you can use Foo_set
in your application.
Like other intrusive containers, you may build several containers on single item structure:
| typedef michael_set::details::iterator< bucket_type, true > cds::intrusive::MichaelHashSet< GC, OrderedList, Traits >::const_iterator |
Const forward iterator.
For iterator's features and requirements see iterator
| typedef michael_set::details::iterator< bucket_type, false > cds::intrusive::MichaelHashSet< GC, OrderedList, Traits >::iterator |
Forward iterator.
The forward iterator for Michael's set is based on OrderedList
forward iterator and has some features:
Therefore, the use of iterators in concurrent environment is not good idea. Use the iterator for the concurrent container for debug purpose only.
|
inline |
Initialize hash set.
The Michael's hash set is non-expandable container. You should point the average count of items nMaxItemCount
when you create an object. nLoadFactor
parameter defines average count of items per bucket and it should be small number between 1 and 10. Remember, since the bucket implementation is an ordered list, searching in the bucket is linear [O(nLoadFactor)
]. Note, that many popular STL hash map implementation uses load factor 1.
The ctor defines hash table size as rounding nMacItemCount / nLoadFactor
up to nearest power of two.
| nMaxItemCount | estimation of max item count in the hash set |
| nLoadFactor | load factor: estimation of max number of items in the bucket. Small integer up to 10, default is 1. |
|
inline |
|
inline |
Returns the size of hash table.
Since MichaelHashSet cannot dynamically extend the hash table size, the value returned is an constant depending on object initialization parameters; see MichaelHashSet::MichaelHashSet for explanation.
|
inline |
Clears the set (non-atomic)
The function unlink all items from the set. The function is not atomic. It cleans up each bucket and then resets the item counter to zero. If there are a thread that performs insertion while clear
is working the result is undefined in general case: empty()
may return true
but the set may contain item(s). Therefore, clear
may be used only for debugging purposes.
For each item the disposer
is called after unlinking.
|
inline |
Checks if the set is empty.
Emptiness is checked by item counting: if item count is zero then the set is empty. Thus, the correct item counting feature is an important part of Michael's set implementation.
|
inline |
|
inline |
Ensures that the val
exists in the set.
The operation performs inserting or changing data with lock-free manner.
If the item val
not found in the set, then val
is inserted into the set. Otherwise, the functor func
is called with item found. The functor signature is:
with arguments:
bNew
- true
if the item has been inserted, false
otherwiseitem
- item of the setval
- argument val
passed into the ensure
function If new item has been inserted (i.e. bNew
is true
) then item
and val
arguments refers to the same thing.The functor may change non-key fields of the item
; however, func
must guarantee that during changing no any other modifications could be made on this item by concurrent threads.
You may pass func
argument by reference using boost::ref
or cds::ref.
Returns std::pair<bool, bool>
where first
is true
if operation is successfull, second
is true
if new item has been added or false
if the item with key
already is in the set.
|
inline |
Delete the item from the set.
The function searches an item with key equal to val
in the set, unlinks it from the set, and returns true
. If the item with key equal to val
is not found the function return false
.
Note the hash functor should accept a parameter of type Q
that can be not the same as value_type
.
|
inline |
Delete the item from the set.
The function searches an item with key equal to val
in the set, call func
functor with item found, unlinks it from the set, and returns it. The Func
interface is
The functor may be passed by reference with boost:ref
If the item with key equal to val
is not found the function return false
.
Note the hash functor should accept a parameter of type Q
that can be not the same as value_type
.
|
inline |
Find the key val
.
The function searches the item with key equal to val
and calls the functor f
for item found. The interface of Func
functor is:
where item
is the item found, val
is the find
function argument.
You can pass f
argument by reference using boost::ref
or cds::ref.
The functor may change non-key fields of item
. Note that the functor is only guarantee that item
cannot be disposed during functor is executing. The functor does not serialize simultaneous access to the set item
. If such access is possible you must provide your own synchronization schema on item level to exclude unsafe item modifications.
The val
argument is non-const since it can be used as f
functor destination i.e., the functor may modify both arguments.
Note the hash functor specified for class Traits
template parameter should accept a parameter of type Q
that can be not the same as value_type
.
The function returns true
if val
is found, false
otherwise.
|
inline |
Find the key val
.
The function searches the item with key equal to val
and returns true
if it is found, and false
otherwise.
Note the hash functor specified for class Traits
template parameter should accept a parameter of type Q
that can be not the same as value_type
.
|
inline |
Inserts new node.
The function inserts val
in the set if it does not contain an item with key equal to val
.
Returns true
if val
is placed into the set, false
otherwise.
|
inline |
Inserts new node.
This function is intended for derived non-intrusive containers.
The function allows to split creating of new item into two part:
f
functor to initialize value-field of val
.The functor signature is:
where val
is the item inserted. User-defined functor f
should guarantee that during changing val
no any other changes could be made on this set's item by concurrent threads. The user-defined functor is called only if the inserting is success and can be passed by reference using boost::ref
|
inline |
Unlink the item val
from the set.
The function searches the item val
in the set and unlink it from it if it is found and is equal to val
.
The function returns true
if success and false
otherwise.


Heading elements implement six levels of document headings,

is the most important and

is the least. A heading element briefly describes the topic of the section it introduces. Heading information may be used by user agents, for example, to construct a table of contents for a document automatically.

Heading level 1

Heading level 2

Heading level 3

Heading level 4

Heading level 5

Heading level 6

Heading level 1 with .h2 style

Heading level 2 with .h1 style

The HTML
Element (or the HTML Anchor Element) defines a hyperlink, the named target destination for a hyperlink, or both.
Here we cube


[gecode-users] Problem compiling
Guido Tack
tack at ps.uni-sb.de
Mon Jan 21 08:47:07 CET 2008
Martin Mann wrote:
> from time to time I encountered some linker problems with g++ due to
> the ordering of the libraries to link. (I am not sure if I did
> static or dynamic linking)
> check: http://www.network-theory.co.uk/docs/gccintro/gccintro_18.html
>
> Maybe that's the case here too. Check your ordering of the gecode
> libs - in general the opposite order of the includes.
> Think e.g. -lgecodekernel has to be the last one not the second.
Order is one aspect (but AFAIK only for static libs). But your
problem is that you didn't link libgecodesupport (which is new in
2.0.0), that should be the library that contains the vtable for
Exception.
Cheers,
Guido
More information about the gecode-users
mailing list


| Home | Libraries | People | FAQ | More |
Copyright © 2005, 2006 Eric Niebler
Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
Table of Contents 目录
“It is better to be approximately right than exactly wrong.”
"近似的正确要好于精确的错误"
-- Old adage 古谚语
Boost.Accumulators is both a library for incremental statistical computation
as well as an extensible framework for incremental calculation in general.
The library deals primarily with the concept of an accumulator,
which is a primitive computational entity that accepts data one sample at a
time and maintains some internal state. These accumulators may offload some
of their computations on other accumulators, on which they depend. Accumulators
are grouped within an accumulator set. Boost.Accumulators
resolves the inter-dependencies between accumulators in a set and ensures that
accumulators are processed in the proper order.
大致上，Boost.Accumulators 既是一个用于增量统计计算的库，也是一个用于增量计算的可扩展框架。起初该库主要涉及 accumulator累加器 概念，它是一个简单的计算实体，每次接受一个数据样本并维护一些内部状态。这些累加器可能将其中一些计算卸给其它所依赖的累加器。多个累加器被组成一个 accumulator set累加器集。Boost.Accumulators
解决了一个累加器集中各累加器间的互相依赖，并确保各个累加器以正确的顺序进行处理。
Last revised: November 14, 2009 at 10:38:39 GMT |


Feature Engineering in Machine Learning
09 Jul 2019For a good time I believed that you can’t really give a long talk on feature engineering which will be relevant across multiple problem spaces. This belief originated from having seen people; calling problem specific special treatment of data; feature engineering . However , over the years I have unconsciously gathered pretty standard ways of treating different kind of features which are applicable irrespective of the kind of problem that you are working with. This of course doesn’t mean that there is no problem specific feature engineering , it simply implies that; you CAN have a good length discussion on generic feature engineering which applies well across different problems .
Sequence of different tips and tricks mentioned here doesn’t hold much meaning . I am simply writing/coding away things as and when I recall them here .
1. How to represent cyclic features
An intuitive example of such data columns are months or weekday numbers. Problem with simply giving them numbers is that the representation doesn’t really make sense . Why ?
Lets have a look :
| Month | Coded_Value |
|---|---|
| January | 1 |
| February | 2 |
| March | 3 |
| April | 4 |
| May | 5 |
| June | 6 |
| July | 7 |
| August | 8 |
| September | 9 |
| October | 10 |
| November | 11 |
| December | 12 |
December isn’t really (12-1=11)
months away from January. The next guy is January, they are just 1
month apart. Their numerical coding which we intend to use as a simple numeric column , doesn’t represent the same information.
A simple trick , in such cases of cyclical data is to use cyclical functions for encoding the values.
In general this will be :
There is still a slight issue, simple plot of transformed values will clarify .
You can see that, now values start to turn up cyclically to have difference among them reducing as months near to end of year. The issue that I was talking about this approach , will become apparent if you look at values of month 2
and 4
. They have same values . The way to counter this is to use two cyclical functions ; both sin
and cos
. Any cyclic numeric column will be encoded as two columns, sin and cos transformations like this :
They look truly cyclic now. All the values in 2 dimensions are different and retain distances among them as per their cyclic nature. :
Here is a quick function for the same :
def code_cyclic_features(data,columns,drop_cols=True):
for col in columns:
max_val=max(data[col])
data[col+'_sin']=np.sin(2*np.pi*data[col]/max_val)
data[col+'_cos']=np.cos(2*np.pi*data[col]/max_val)
if drop_cols:
del data[col]
return data
2. Coding Categorical Features as dummies/flags
Ignoring the discussion where you can convert the categorical feature to some numeric representation without changing the sense of information .
Everybody starts coding categorical columns by creating n-1
dummies for n categories . However we should keep in mind that if any category doesn’t have enough observations/data, we won’t be able to extract consistent/reliable patterns from it. Its a common practice to ignore such categories. You have to decide a frequency cutoff. There isn’t much point in searching the depths of internet to find a good value for this cutoff. Keep the tradeoff in mind , higher cutoff will lead to information loss and lower cutoff will lead to data explosion and overfitting.
Here is a quick python function which will enable you to create dummies with frequency cutoff.
def code_categorical_to_dummies(data,columns,frequency_cutoff=100):
for col in columns:
freq_table=data[col].value_counts(dropna=False)
cats=freq_table.index[freq_table>frequency_cutoff][:-1]
for cat in cats:
data[col+'_'+cat]=(data[col]==cat).astype(int)
del data[col]
return data
3. Categorical Embeddings
Even when you use frequency cutoff , sometimes creating dummies results in data explosion as all of the categories might have good number of observation. Consider a data which has 11 columns where 10 are numeric and 11th column has 50 eligible categories . You’ll end up with 60 columns in the data. This could be a problem if you further decide to build a randomforest
or gbm
family models. You’ll decide the max_features parameter to be some random subset of these 60 columns in your data. By sheer number, the flag variables coming from a single column will be selected a lot more and , 10 numeric columns don’t get enough exposure due to inbuilt nature of the algorithm. We’d like to represent these flag variables at much lower dimensions than creating 50 separate columns in this scenario.
What comes to your rescue is embedding derived from neural networks which work as auto encoders . Here is an example in code which will help you in understanding the concept .
file = r'/Users/lalitsachan/Dropbox/0.0 Data/loans data.csv'
data=pd.read_csv(file)
data['State'].nunique()
47
Column State here has 47 unique values . We can create dummies but that will lead to having 47 one hot encoded columns . Lets see how we can come up with model driven embeddings for this column and bring down the dimension for the same .
dummy_data=pd.get_dummies(data['State'],prefix='State')
dummy_data.shape[1]
47
Lets prepare our response which is predicting interest on the basis of columns in the data. Some preprocessing on the response because it has come as character because of ‘%’ sign in the data.
y=pd.to_numeric(data['Interest.Rate'].str.replace("%",""),errors='coerce')
y.fillna(y.mean(),inplace=True)
from keras.models import Model,Sequential
from keras.layers import Dense,Input
embedding_dim=3
inputs=Input(shape=(dummy_data.shape[1],))
embedded_output=Dense(embedding_dim)(inputs)
outputs=Dense(1)(embedded_output)
model=Model(inputs=inputs,outputs=outputs)
embedder=Model(inputs=inputs,outputs=embedded_output)
We are going to use output of embedder as our transformed data. Embedding is being driven by predicting the response .
model.compile(optimizer='adam',loss='mse')
model.fit(dummy_data,y,epochs=100,batch_size=100)
Once this is done , we can use embedder to get out transformed data, new low dimensional representation of 47 one hot encoded columns
low_dim=pd.DataFrame(embedder.predict(dummy_data),columns=['emb1','emb2','emb3'])
low_dim.head()
We can use the same trained embedder for test/val data as well.
4. Feature Selection using feature importance
Its a common feature of many tree based ensemble method (both bagging and boosting), feature importance , which can be used for quick feature selection . But many sources failed to mention, where to draw the line. What level of feature importance should be set as cutoff , below which you can drop all the columns from further considerations .
Truth is , people don’t mention this because as such there is no fixed cutoff. And meaning of numeric value of feature importance can be data dependent. However a neat little trick can be used every time. You can add a random noise column as one of the features . Any column which gets its feature importance value below that, can be considered doing worse than white noise , which we know is not related to data and hence can be dropped.
Here is a quick example using one of the pre loaded datasets in sklearn.
from sklearn.datasets import load_boston
boston=load_boston()
X=pd.DataFrame(boston.data,columns=boston.feature_names)
y=boston.target
We’ll add random noise to the data
X['white_noise']=np.random.random(size=(X.shape[0]))
Now lets build a quick random Forest model and check feature importance to see which features we can safely drop before further more detailed experiments with the data
from sklearn.ensemble import RandomForestRegressor
rf=RandomForestRegressor(n_estimators=200)
rf.fit(X,y)
imp_data=pd.DataFrame({'imp':rf.feature_importances_,'cols':X.columns})
imp_data.sort_values('imp',ascending=False)
We can safely say that , we can drop columns INDUS
, RAD
, ZN
, CHAS
in context of predicting the target for this problem .
Few notes on the takeaways/followups .
- This is not an exhaustive list by any means. I discussed few things which I see; are not commonly discussed or are generally looked over. I’ll make additions if I recall something else.
- Whenever I discussed one of the tricks, I skimmed over other details. For example , when we were talking about categorical embeddings , we can very well first ignore categories which have low frequencies , create dummies for the rest and then go for categorical embeddings .
- Feel free to let me know if you’d like me to add some discussion as part/addition to the list above.


I've been thinking about how to display my gas usage, gleaned from a pulse-based meter, where 1 pulse represents 1 cubic foot of gas. Currently, on detection of a pulse, the WH of that CuFt of gas can be converted to average Watts over the duration since the previous pulse, and interrupt.ino on the Arduino tracks that duration, and makes the calculation, currently plonking it into emonPi.power2. That gives a single value written into the Feed at the time the value is received by emoncms. Well and good, but that gives me a series of spikes, one per pulse, scattered across a graph over a period of time, often with very long gaps between them.
After sending the value, emonPi.power2 is cleared to zero, as the current value is no longer relevant after the pulse, and no new value can be calculated until the next pulse is detected, and that might be hours in the future. The current value's actual relevance is for the time between the most recent pulse and the previous pulse, and yet nothing is written into any of the datapoints in the Feed for that entire duration except the one coinciding with the pulse, all the rest are zero.
That seems a very artificial way to display that data. I think there might be a better way, a way that spreads the data across more of the graph than current mechanisms for gas/other pulse meters do.
What if we write a suitable value retrospectively into as many datapoints preceding the pulse as far as either the previous pulsetime, or, say, 15 minutes, whichever is shorter.
My reasoning for a limit is that it doesn't make sense to write a very low value (35.3W) into every datapoint for the previous nine hours while my central heating was off overnight, it would make more sense to assume that all of that CuFt was consumed within, say, the last 15 minutes, and that all of the time prior to that limit no gas was consumed at all after the previous pulse. I would therefore calculate the Watts as 1273W, and write that into the last 90 datapoints, assuming 10 second intervals. Specifying a limit allows for long periods when it is clear that no gas is being consumed, and limits the number of writes in the update, and importantly, doesn't contradict the data, it is just a convenience that reinterprets the data in a more useful and arguably more intuitive manner, more closely resembling the electric Watts graphs. It achieves a more granular appearance than single bare pulses can do.
Any ideas as to how to achieve this? Is there a mechanism that exists in processes to do this? Where would I start?
Retrospectively write data to a feed?
Submitted by MikeSims on Thu, 21/04/2016 - 19:10I've been thinking about how to display my gas usage, gleaned from a pulse-based meter, where 1 pulse represents 1 cubic foot of gas. Currently, on detection of a pulse, the WH of that CuFt of gas can be converted to average Watts over the duration since the previous pulse, and interrupt.ino on the Arduino tracks that duration, and makes the calculation, currently plonking it into emonPi.power2. That gives a single value written into the Feed at the time the value is received by emoncms. Well and good, but that gives me a series of spikes, one per pulse, scattered across a graph over a period of time, often with very long gaps between them.
After sending the value, emonPi.power2 is cleared to zero, as the current value is no longer relevant after the pulse, and no new value can be calculated until the next pulse is detected, and that might be hours in the future. The current value's actual relevance is for the time between the most recent pulse and the previous pulse, and yet nothing is written into any of the datapoints in the Feed for that entire duration except the one coinciding with the pulse, all the rest are zero.
That seems a very artificial way to display that data. I think there might be a better way, a way that spreads the data across more of the graph than current mechanisms for gas/other pulse meters do.
What if we write a suitable value retrospectively into as many datapoints preceding the pulse as far as either the previous pulsetime, or, say, 15 minutes, whichever is shorter.
My reasoning for a limit is that it doesn't make sense to write a very low value (35.3W) into every datapoint for the previous nine hours while my central heating was off overnight, it would make more sense to assume that all of that CuFt was consumed within, say, the last 15 minutes, and that all of the time prior to that limit no gas was consumed at all after the previous pulse. I would therefore calculate the Watts as 1273W, and write that into the last 90 datapoints, assuming 10 second intervals. Specifying a limit allows for long periods when it is clear that no gas is being consumed, and limits the number of writes in the update, and importantly, doesn't contradict the data, it is just a convenience that reinterprets the data in a more useful and arguably more intuitive manner, more closely resembling the electric Watts graphs. It achieves a more granular appearance than single bare pulses can do.
Any ideas as to how to achieve this? Is there a mechanism that exists in processes to do this? Where would I start?
Mike


Once you click start, a song will start to play.
You have 40 seconds to guess
who the band is!
Turn your sound on and click Start to get going!
You Score Is:
Guess the name of this band
Guess the name of this band
Guess the name of this band
Guess the name of this band
Guess the name of this band


Meow Data Format
This document describes, in general, the data format Meow uses to store your documents in MongoDB.
Collection naming
The name of the collection is based on the name of the class that is stored in it. CamelCase is converted to snake_case, and the class name is made plural. Some examples (class: collection):
User
:users
Man
:men
CarWheel
:car_wheels
Property naming
By default, property names are converted from camelCase to snake_case. For example:
firstName
:first_name
event
:event
As you can see, this mostly works, but sometimes you might want some more control. For example, for a property named linkedIn
, Meow will by default use linked_in
as the key. To customize this, you can place a sourcery annotation next to the variable:
// sourcery: key = linkedin
var linkedIn: String
The Key enum
Meow will store all property names in an enum named Key
in every model. You can use this enum if you ever need to acccess the raw (database) name of a property.
To access the key name of a property MyProperty
on MyModel
: MyModel.Key.myProperty.keyString
Structs and classes
Meow stores structs and classes as BSON documents. The key enum (see above) is used to define the keys in these documents.
For example, the following model:
class User : Model {
enum Gender {
case male, female
}
struct Details {
var gender: Gender
var firstName: String
var lastName: String
}
var username: String
var details: Details
init(...) { ... }
}
will be serialized to this (in OpenKitten BSON notation):
[
"_id": ObjectId(),
"username": "Bob26",
"details": [
"gender": "malae",
"first_name": "Bob",
"last_name": "Something"
]
]
Supported primitives and MongoKitten types
The following types are supported by BSON and will be stored as-is without conversion.
Date
DateReferences
References are stored in a custom format. For example, a reference to a User
with the id afafafafafafafafafafafaf
looks like this in BSON:
[
"_id": ObjectId("afafafafafafafafafafafaf"),
"_ref": "users"
]
We do not use DBRefs for relations. DBRefs include a $
in their key names which complicates querying.


Marisa is an undergraduate student of Computer Science at UC San Diego with a planned graduation date of June 2016. She
has interned at NASA's launch complex, Kennedy Space Center as a Subject Matter Analyst for Expertise Directories. She has also worked at Lockheed Martin as a software engineer intern and at the Taiwan International Graduate Program as a
Social Networks and Human-Centered Intern. She is completing her second term tutoring an upper division Computer Science Class called
Digital Systems Lab. Current projects include working with on an application called
GameBuilder for mobile phones that will enable the production and publication of games with ease. For the last four years she has served as a core member of the quarterly hosted Beginner Programming Competition hosted by Women in Computing (WIC).


Pipe function in Python Polars. Don't leave the pipe flow!
Are you exploring Polars as an alternative to Pandas? We love it for the pipe flow feeling! Learn in 3 lines how and when to use it
Well, well, hanging out in Python, missing the pipe feeling
?? There are some good news! The pipe function in the Polars Python module allows you to chain operations together by passing the result of one operation as the input to the next operation. For sure, this is not one of the greatest advantages that Polars offers over Pandas…. But it does feel so nice!
Here you can find and example of how to keep the pipe flow even in Python!
import polars as pl
import random
# Create a Polars DataFrame with base columns
df = pl.DataFrame({
'name': ['Alice', 'Bob', 'Charlie'],
'offensive_skill': [5, 30, 85],
'defensive_skill': [92, 30, 10]
})
# Define polars custom functions to apply
def add_position_column(df):
df = df.with_columns(
pl.when(pl.col('defensive_skill') > 50).then('CB')
.when(pl.col('offensive_skill') > 50).then('FW')
.otherwise('bench').alias("position")
)
return df
def add_squad_number_column(df):
df = df.with_columns(
pl.when(pl.col('position') == 'CD').then(pl.lit(random.sample(range(2, 6), 1)[0], dtype=pl.Int8))
.when(pl.col('position') == 'FW').then(pl.lit(random.sample(range(7, 19), 1)[0], dtype=pl.Int8))
.otherwise('-').alias("squad_number")
)
return df
# Chain operations together using the pipe function
(
df
.pipe(add_position_column)
.pipe(add_squad_number_column)
)
| name | offensive_skill | defensive_skill | position | squad_number |
|---|---|---|---|---|
| str | i64 | i64 | str | str |
| "Alice" | 5 | 92 | "CB" | "-" |
| "Bob" | 30 | 30 | "bench" | "-" |
| "Charlie" | 85 | 10 | "FW" | "15" |
Polars pipe and lazy evaluation
An extra trick is to use the lazy evaluation in order to maximize the advantages of query optimization and parallelization. We need a big enough df and complex operations to make it worth.
| name | offensive_skill | defensive_skill | position | squad_number |
|---|---|---|---|---|
| str | i64 | i64 | str | str |
| "Alice" | 5 | 92 | "CB" | "-" |
| "Bob" | 30 | 30 | "bench" | "-" |
| "Charlie" | 85 | 10 | "FW" | "12" |


Functional Neuroimaging offers a Combination of Temporal and Spatial Acuity
- 4D fMRI: 3 spatial dimensions (approx $2mm \times 2mm \times 2mm$) + 1 time dimension (approx $TR=2.5s$)
- view of the dynamic brain at each spatial position
- most other modalities offer only strongly one or the other
Thousands of Processing Options, each offering theoretical desiderata
- Particular pipeline options work well together, and others do not, in a nonconvex fashion
- Best pipelines per dataset might not be globally optimal: necessity for meganalysis
FNGS: A Scalable and Robust Pipelines for fMRI connectomics
- pipeline refined from results of Discriminability optimal pipeline
- Offers local and cloud scaling options, enabling basic meganalysis on clusters or remotes
- Containerized: trivial to deploy
- Offers high-end performance at low computational requirements
Functional Preprocessing removes spatial artifacts
Anatomical Preprocessing enables superior Brain Extraction
Registration Spatially Aligns Scans
Nuisance Correction removes Noise Variables from Signal
Downsampling Yields Computationally Feasible Dimensionality
FNGS is effective on Local or Cloud Infrastructure
- Local
- Single or Multithread depending on how many threads/RAM you have available
- Cloud
- Deployment scripts and web-service to interface with AWS Batch
FNGS Offers Web-based Cloud Deployment
FNGS Offers Superior Computational Performance
| Metric |
FNGS |
CPAC |
fmriprep |
| execution time (s) |
1560 |
3120 |
40000 |
| $RAM_{mx}$ (GB) |
4.3 |
3.5 |
15.1 |
| $RAM_{avg}$ (GB) |
1.0 |
1.7 |
3.6 |
| $CPU_{avg}$ (threads) |
1.15 |
1.33 |
3.81 |
| $Disk$ (GB) |
1.3 |
2.2 |
12.1 |
... Without Sacrificing Robustness
- FNGS: $\bar{d} = .887$, CPAC: $\bar{d} = .889$
- $p=0.87$, not significant
Future Goals
- FNGS is "tuned" for resting-state fMRI
- Is there a difference with what we would do for task fMRI?
- How can we quantify this? modifications of Signal-Subgraphing
- FNIRT is far from perfect
- Are there better registration models out there? Possibility of LDDMM?


Profile Analysis - Parallel Coordinates
Opacity:
20%
Shadows
Dark
Hide Ticks
Total Selected
Group Breakdown
Export
Remove
Keep


the anonymous instant proxy
{{.Name}} allows you to instantly view a webpage without them seeing any trace of you. all access is proxied via the {{.Name}} service.
some features (especially javascript) are therefore disabled. sorry for that.


Donec id elit non mi porta gravida at eget metus. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus. Etiam porta sem malesuada magna mollis euismod. Donec sed odio dui.
Donec id elit non mi porta gravida at eget metus. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus. Etiam porta sem malesuada magna mollis euismod. Donec sed odio dui.
Donec sed odio dui. Cras justo odio, dapibus ac facilisis in, egestas eget quam. Vestibulum id ligula porta felis euismod semper. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.
FAQs
Where do graduating students attend high school?
Our students go on to attend both public and select private schools in our area including The Kinkaid School, Strake Jesuit College Preparatory, St. Agnes Academy, St. John’s School, St. Thomas High School, Episcopal High School, Awty International, Houston Christian High School, and Fort Bend Christian Academy.
Is lunch provided?
It is our goal to provide lunch options that are both delicious and healthy. Parents can order lunches through The Healthy Lunch Box. All meals include an entrée, a side, a dessert and drink. A vegetarian option is also available daily. Parents in ECP can sign up for lunch service for the semester or the year. Parents in K-8th grades can order lunch daily through The Healthy Lunch Box online ordering system (Click Here for more information).
What are the hours of operation?
Where can I purchase my child’s uniform?
We require all students to wear a school uniform unless otherwise directed. Uniform colors and pieces vary based on grade.


Our proposal period has now closed, and we have accepted this amazing list of proposed sessions to build the 2015 program.
Session proposals closed on April 10. We spent a couple weeks reviewing proposals within the SRCCON team, with additional reviews and perspectives from across our community, in order to assemble a final list of sessions.
As we did last year, we made it a priority to build a balanced program that reflects the makeup of our communities, and we actively welcomed session proposals from members of communities underrepresented in journalism and technology, and from non-coastal and small-market news organizations.
Successful sessions often emerge from a single question or problem—if you’ve been struggling with just about any aspect of your work, you can bet other folks have dealt with it, too.
Last year’s sessions dealt with topics including newsroom cross-training, CMSes, skillshares on specific libraries and tools, interactive design specifics, diversity in news technology teams, security, the business of news startups, and many different approaches to data-wrangling. To give you a taste, here’s a small selection of great sessions from 2014.
Session facilitators aren’t expected to be the experts in the room—they design and guide sessions, share what they know, and learn from their peers. You won’t need slides, just the willingness to facilitate an engaged conversation or lead a workshop or skillshare. (Or pitch another session format entirely!)
If your session is accepted, we’ll provide as much help as you’d like figuring out what kind of session format will work best for your topic—we can even match you up with a co-facilitator if you’re pitching solo and want some support.
Please note: Because SRCCON is a collaborative, peer-to-peer event, rather than a series of prepared conference talks, all participants must buy a ticket—this approach also helps us keep our ticket prices low and allows us to offer scholarships to those who need help with travel and lodging expenses. The facilitators of all accepted sessions will be given an opportunity to purchase a ticket before the public on-sale date. SRCCON tickets go on public sale April 29, and we expect them to sell out very quickly.
In addition to the daytime sessions that form the core of SRCCON, we’ll also host a series of informal evening talks on non-work topics, open up space for small-group collaboration, and run a hands-on tea- and coffee-hacking station. Stay tuned for more information about additional program items and related events in Minneapolis-St. Paul.


homelocator.Rmd
Identifying meaningful locations, such as home and/or work locations from mobile technology is an essential step in the field of mobility analysis. The homelocator
library is designed for identifying home locations of user based on the spatio-temporally features contained within the mobility data, such as social media data, mobile phone data, mobile app data and so on.
Although there is a myriad of different approaches to inferring meaningful locations in the last 10 years, a consensus or single best approach has not emerged in the current state-of-the-art. The actual algorithms used are not always discussed in detail in publications and the source codes are seldom released in public, which makes comparing algorithms as well as reproducing work difficult.
So, the main objective for developing this package is to provide a consistent framework and interface for the adoption of different approaches so that researchers are able to write structured, algorithmic ‘recipes’, or use the existing embedded “recipes” to identify meaningful locations according to their research requirements. With this package, users are able to achieve an apples-to-apples comparison across approaches.
We hope that through packages like homelocator
, future work that relies on the inference of meaningful locations becomes less ‘custom’ (with each researcher writing their own algorithm) but instead will use common, comparable algorithms in order to increase transparency and reproducibility in the field.
# Load homelocator library
library(homelocator)
#> Welcome to homelocator package!
To explore the basic data manipulation functions of homelocator
, we’ll use a test_sample
dataset.
# load data
data("test_sample", package = "homelocator")
# show test sample
test_sample %>% head(3)
#> # A tibble: 3 x 3
#> u_id created_at grid_id
#> <int> <dttm> <int>
#> 1 92298565 2016-04-17 22:43:06 1581
#> 2 33908340 2014-10-03 16:29:48 1461
#> 3 92298565 2014-02-07 07:26:15 1136
Note: if you use your own dataset, please make sure that you have converted the timestamps to the appropriate local time zone.
The validate_dataset()
function makes sure your input dataset contains all three necessary variables: user, location and timestamp. There are 4 arguments in this function:
user
: name of column that holds a unique identifier for each user.timestamp
: name of column that holds specific timestamp for each data point. This timestamp should be in POSIXct
format.location
: name of column that holds a unique identifier for each location.keep_other_vars
: option to keep or remove any other variables of the input dataset. The default is FALSE
.
df_validated <- validate_dataset(test_sample, user = "u_id", timestamp = "created_at", location = "grid_id", keep_other_vars = FALSE)
#> 🎉 Congratulations!! Your dataset has passed validation.
#> 👤 There are 100 unique users in your dataset.
#> 🌏 Now start your journey identifying their meaningful location(s)!
#> 👏 Good luck!
#>
# show result
df_validated %>% head(3)
#> # A tibble: 3 x 3
#> u_id grid_id created_at
#> <int> <int> <dttm>
#> 1 92298565 1581 2016-04-17 22:43:06
#> 2 33908340 1461 2014-10-03 16:29:48
#> 3 92298565 1136 2014-02-07 07:26:15
The nest_verbose()
and unnest_verbose()
functions work in the same way as nest and unnest functions in tidyverse but with some additional status information such as the elapsed running time.
df
: a dataframe...
: for nest_verbose()
refers to selected columns to nest and for unnest_verbose()
refers to the list column to unnest.
# nest data
df_nested <- nest_verbose(df_validated, c("created_at", "grid_id"))
#> 🛠 Start nesting...
#> ✅ Finish nesting!
#> ⌛ Nesting time: 0.019 secs
#>
# show result
df_nested %>% head(3)
#> # A tibble: 3 x 2
#> u_id data
#> <int> <list>
#> 1 92298565 <tibble [1,291 × 2]>
#> 2 33908340 <tibble [1,170 × 2]>
#> 3 11616678 <tibble [938 × 2]>
# show result
df_nested$data[[1]] %>% head(3)
#> # A tibble: 3 x 2
#> created_at grid_id
#> <dttm> <int>
#> 1 2016-04-17 22:43:06 1581
#> 2 2014-02-07 07:26:15 1136
#> 3 2012-08-18 19:26:31 1038
# unnest data
df_unnested <- unnest_verbose(df_nested, data)
#> 🛠 Start unnesting...
#> ✅ Finish unnesting!
#> ⌛ Unnesting time: 0.012 secs
#>
# show result
df_unnested %>% head(3)
#> # A tibble: 3 x 3
#> u_id created_at grid_id
#> <int> <dttm> <int>
#> 1 92298565 2016-04-17 22:43:06 1581
#> 2 92298565 2014-02-07 07:26:15 1136
#> 3 92298565 2012-08-18 19:26:31 1038
The nest_double_nest()
and unnest_double_nested()
functions work in a similar way as nest_verbose()
and unnest_verbose()
functions but they apply to an already nested data frame. In other words, they map nest
and unnest
function to each element of a list created a double-nested data frame, or vice versa. This is an essential step in many home location algorithms as they often operate ‘per user, per location’.
df
: a nested dataframe...
: for nest_double_nest()
refers to selected columns to nest and for unnest_double_nested()
refers to list-column to unnest.
# double nest data (e.g., nesting column: created_at)
df_double_nested <- nest_nested(df_nested %>% head(100), c("created_at"))
#> 🛠 Start nesting...
#>
#> ✅ Finish nesting!
#> ⌛ Nesting time: 0.745 secs
#>
# show result
df_double_nested %>% head(3)
#> # A tibble: 3 x 2
#> u_id data
#> <int> <list>
#> 1 92298565 <tibble [200 × 2]>
#> 2 33908340 <tibble [83 × 2]>
#> 3 11616678 <tibble [61 × 2]>
# show result
df_double_nested$data[[1]] %>% head(3)
#> # A tibble: 3 x 2
#> grid_id data
#> <int> <list>
#> 1 1581 <tibble [3 × 1]>
#> 2 1136 <tibble [7 × 1]>
#> 3 1038 <tibble [8 × 1]>
df_double_nested$data[[1]]$data[[1]] %>% head(3)
#> # A tibble: 3 x 1
#> created_at
#> <dttm>
#> 1 2016-04-17 22:43:06
#> 2 2014-12-28 03:26:34
#> 3 2014-12-03 09:26:36
# unnest nested data
df_double_unnested <- df_double_nested %>%
head(100) %>% ## take first 100 rows for example
unnest_double_nested(., data)
#> 🛠 Start unnesting...
#> ✅ Finish unnesting!
#> ⌛ Unnesting time: 0.423 secs
#>
# show result
df_double_unnested %>% head(3)
#> # A tibble: 3 x 3
#> u_id grid_id created_at
#> <int> <int> <dttm>
#> 1 92298565 1581 2016-04-17 22:43:06
#> 2 92298565 1581 2014-12-28 03:26:34
#> 3 92298565 1581 2014-12-03 09:26:36
The enrich_timestamp()
function creates additional variables that are derived from the timestamp column. These include the year, month, day, day of the week and hour of the day. These are often used/needed as intermediate variables in home location algorithms.
df
: a nested dataframetimestamp
: name of column that holds the specific timestamp for each data point. This timestamp should be in POSIXct
format.tz
: a character string containing the time zone to convert to. This should be one of the time zones recognized by R. The default tz
is set to Asia/Singapore
.
#original variables
df_nested[1, ] %>% unnest(cols = c(data)) %>% head(3)
#> # A tibble: 3 x 3
#> u_id created_at grid_id
#> <int> <dttm> <int>
#> 1 92298565 2016-04-17 22:43:06 1581
#> 2 92298565 2014-02-07 07:26:15 1136
#> 3 92298565 2012-08-18 19:26:31 1038
# create new variables from "created_at" timestamp column
df_enriched <- enrich_timestamp(df_nested, timestamp = "created_at", tz = "Asia/Singapore")
#> 🛠 Enriching variables from timestamp...
#>
#> ✅ Finish enriching! New added variables: year, month, day, wday, hour, ymd.
#> ⌛ Enriching time: 0.3 secs
#>
# show result
df_enriched[1, ] %>% unnest(cols = c(data)) %>% head(3)
#> # A tibble: 3 x 9
#> u_id created_at grid_id year month day wday hour ymd
#> <int> <dttm> <int> <dbl> <dbl> <int> <dbl> <int> <date>
#> 1 92298565 2016-04-17 22:43:06 1581 2016 4 17 1 22 2016-04-17
#> 2 92298565 2014-02-07 07:26:15 1136 2014 2 7 6 7 2014-02-07
#> 3 92298565 2012-08-18 19:26:31 1038 2012 8 18 7 19 2012-08-18
The summarise_nested()
function works similar to dplyr’s regular summarise
function but operates within a nested tibble. summarise_double_nested()
conversely operates within a double-nested tibble.
df
: a nested dataframenest_cols
: a selection of columns to nest in existing list-column...
: name-value pairs of summary functions
# summarize in nested dataframe
# e.g., summarise total number of tweets and total number of places per user
df_summarize_nested <- summarise_nested(df_enriched,
n_tweets = n(),
n_locs = n_distinct(grid_id))
#> 🛠 Start summarising values...
#>
#> ✅ Finish summarising! There are 2 new added variables: n_tweets, n_locs
#> ⌛ Summarising time: 0.179 secs
#>
# show result
df_summarize_nested %>% head(3)
#> # A tibble: 3 x 4
#> u_id data n_tweets n_locs
#> <int> <list> <int> <int>
#> 1 92298565 <tibble [1,291 × 8]> 1291 200
#> 2 33908340 <tibble [1,170 × 8]> 1170 83
#> 3 11616678 <tibble [938 × 8]> 938 61
# summarize in double nested dataframe
# take first 100 users for example
# e.g summarise total number of tweets and totla number of distinct days
df_summarize_double_nested <- summarise_double_nested(df_enriched %>% head(100),
nest_cols = c("created_at", "ymd", "year", "month", "day", "wday", "hour"),
n_tweets = n(), n_days = n_distinct(ymd))
#> 🛠 Start summarising values...
#> ✅ Finish summarising! There are 2 new added variables: n_tweets, n_days
#> ⌛ Summarising time: 5.871 secs
#>
# show result
df_summarize_double_nested[1, ]
#> # A tibble: 1 x 2
#> u_id data
#> <int> <list>
#> 1 92298565 <tibble [200 × 4]>
df_summarize_double_nested[1, ]$data[[1]] %>% head(3)
#> # A tibble: 3 x 4
#> grid_id data n_tweets n_days
#> <int> <list> <int> <int>
#> 1 1581 <tibble [3 × 7]> 3 3
#> 2 1136 <tibble [7 × 7]> 7 6
#> 3 1038 <tibble [8 × 7]> 8 3
The remove_top_users()
function allows to remove top N
percent of active users based on the total number of data points per user. Although the majority of users are real people, some accounts are run by algorithms or ‘bots’, whereas others can be considered as spam accounts. Removing a certain top N
percent of active users is an oft-used approach to remove such accounts and reduce the number of such users in the final dataset.
df
: a dataframe with columns of user id, and data point countsuser
: name of column that holds unique identifier for each usercounts
: name of column that holds the data points frequency for each usertopNpct_user
: a decimal number that represent the certain percentage of users to remove
# remove top 1% active users (e.g based on the frequency of tweets sent by users)
df_removed_active_users <- remove_top_users(df_summarize_nested, user = "u_id",
counts = "n_tweets", topNpct_user = 1)
#> 👤 There are 100 users at this moment.
#> 🛠 Start removing top 1% top active users...
#> ✅ Finish removing! Removed 0 active users!
#> 👤 There are 100 users left.
#> ⌛ Removing time: 0.003 secs
#>
# show result
df_removed_active_users %>% head(3)
#> # A tibble: 3 x 4
#> u_id data n_tweets n_locs
#> <int> <list> <int> <int>
#> 1 40153763 <tibble [1,688 × 8]> 1688 233
#> 2 92298565 <tibble [1,291 × 8]> 1291 200
#> 3 31088399 <tibble [1,210 × 8]> 1210 79
The filter_verbose()
function works in the same way as filter function in tidyverse but with additional information about the number of (filtered) users remaining in the dataset. And the filter_nested()
function works in the similar way as filter_verbose()
but is applied within a nested tibble.
df
: a dataframe with columns of user id, and variables that your are going to apply the filter function. If the column not in the dataset, you need to add that column before you apply the filteruser
: name of column that holds unique identifier for each user...
: Logical predicates defined in terms of the variables in df. Only rows match conditions are kept.
# filter users with less than 10 tweets sent at less than 10 places
df_filtered <- filter_verbose(df_removed_active_users, user = "u_id",
n_tweets > 10 & n_locs > 10)
#> 👤 There are 100 users at this moment.
#> 🛠 Start filtering users...
#> ✅ Finish filtering! Filterred 36 users!
#> 👤 There are 64 users left.
#> ⌛ Filtering time: 0.002 secs
#>
# show result
df_filtered %>% head(3)
#> # A tibble: 3 x 4
#> u_id data n_tweets n_locs
#> <int> <list> <int> <int>
#> 1 40153763 <tibble [1,688 × 8]> 1688 233
#> 2 92298565 <tibble [1,291 × 8]> 1291 200
#> 3 31088399 <tibble [1,210 × 8]> 1210 79
# filter tweets that sent on weekends and during daytime (8am to 6pm)
df_filter_nested <- filter_nested(df_filtered, user = "u_id",
!wday %in% c(1, 7), # 1 means Sunday and 7 means Saturday
!hour %in% seq(8, 18, 1))
#> 👤 There are 64 users at this moment.
#> 🛠 Start filtering user...
#>
#> ✅ Finish Filtering! Filterred 0 users!
#> 👤 There are 64 users left!
#> ⌛ Filtering time: 0.078 secs
#>
# show result
df_filter_nested %>% head(3)
#> # A tibble: 3 x 4
#> u_id data n_tweets n_locs
#> <int> <list> <int> <int>
#> 1 40153763 <tibble [495 × 8]> 1688 233
#> 2 92298565 <tibble [424 × 8]> 1291 200
#> 3 31088399 <tibble [463 × 8]> 1210 79
df_filter_nested$data[[1]] %>% head(3)
#> # A tibble: 3 x 8
#> created_at grid_id year month day wday hour ymd
#> <dttm> <int> <dbl> <dbl> <int> <dbl> <int> <date>
#> 1 2016-06-23 00:36:25 759 2016 6 23 5 0 2016-06-23
#> 2 2015-02-17 19:14:10 778 2015 2 17 3 19 2015-02-17
#> 3 2015-03-31 23:21:51 759 2015 3 31 3 23 2015-03-31
The mutate_verbose()
function works in the same way as mutate function in tidyverse but with additional information about the elapsed running time.
df
: a dataframe...
: name-value pairs of expressionsFunction mutate_nested()
works in the similar way as mutate_verbose()
but it adds new variables inside a nested tibble.
df
: a nested dataframe...
: name-value pairs of expressions
## let's use pre-discussed functions to filter some users first
colnmaes_data <- df_filtered$data[[1]] %>% names()
colnmaes_to_nest <- colnmaes_data[-which(colnmaes_data == "grid_id")]
df_cleaned <- df_filtered %>%
summarise_double_nested(., nest_cols = colnmaes_to_nest,
n_tweets_loc = n(), # number of tweets sent at each location
n_hrs_loc = n_distinct(hour), # number of unique hours of sent tweets
n_days_loc = n_distinct(ymd), # number of unique days of sent tweets
period_loc = as.numeric(max(created_at) - min(created_at), "days")) %>% # period of tweeting
unnest_verbose(data) %>%
filter_verbose(., user = "u_id",
n_tweets_loc > 10 & n_hrs_loc > 10 & n_days_loc > 10 & period_loc > 10)
#> 🛠 Start summarising values...
#> ✅ Finish summarising! There are 4 new added variables: n_tweets_loc, n_hrs_loc, n_days_loc, period_loc
#> ⌛ Summarising time: 5.92 secs
#>
#> 🛠 Start unnesting...
#> ✅ Finish unnesting!
#> ⌛ Unnesting time: 0.006 secs
#>
#> 👤 There are 64 users at this moment.
#> 🛠 Start filtering users...
#> ✅ Finish filtering! Filterred 35 users!
#> 👤 There are 29 users left.
#> ⌛ Filtering time: 0.001 secs
#>
# show cleaned dataset
df_cleaned %>% head(3)
#> # A tibble: 3 x 9
#> u_id n_tweets n_locs grid_id data n_tweets_loc n_hrs_loc n_days_loc
#> <int> <int> <int> <int> <lis> <int> <int> <int>
#> 1 4.02e7 1688 233 759 <tib… 467 23 366
#> 2 4.02e7 1688 233 842 <tib… 57 16 51
#> 3 4.02e7 1688 233 1212 <tib… 71 13 59
#> # … with 1 more variable: period_loc <dbl>
# ok, then let's apply the mutate_nested function to add four new variables: wd_or_wk, time_numeric, rest_or_work, wk.am_or_wk.pm
df_expanded <- df_cleaned %>%
mutate_nested(wd_or_wk = if_else(wday %in% c(1,7), "weekend", "weekday"),
time_numeric = lubridate::hour(created_at) + lubridate::minute(created_at) / 60 + lubridate::second(created_at) / 3600,
rest_or_work = if_else(time_numeric >= 9 & time_numeric <= 18, "work", "rest"),
wk.am_or_wk.pm = if_else(time_numeric >= 6 & time_numeric <= 12 & wd_or_wk == "weekend", "weekend_am", "weekend_pm"))
#> 🛠 Start adding variable(s)...
#> ✅ Finish adding! There are 4 new added variables: wd_or_wk, time_numeric, rest_or_work, wk.am_or_wk.pm
#> ⌛ Adding time: 0.236 secs
#>
# show result
df_expanded %>% head(3)
#> # A tibble: 3 x 9
#> u_id n_tweets n_locs grid_id data n_tweets_loc n_hrs_loc n_days_loc
#> <int> <int> <int> <int> <lis> <int> <int> <int>
#> 1 4.02e7 1688 233 759 <tib… 467 23 366
#> 2 4.02e7 1688 233 842 <tib… 57 16 51
#> 3 4.02e7 1688 233 1212 <tib… 71 13 59
#> # … with 1 more variable: period_loc <dbl>
The function prop_factor_nested()
allows you to calculate the proportion of categories for a variable inside the list-column and convert each categories to a new variable adding to the dataframe. For example, inside the list-column, the variable wd_or_wk
has two categories named weekend or weekday, when you call prop_factor_nested()
function, it calculates the proportion of weekend and weekday separately, and the results are then expanded to two new columns called weekday
and weekend
adding to the dataframe.
df
: a nested dataframevar
: name of column to calculate inside the list-column
# categories for a variable inside the list-column: e.g weekend or weekday
df_expanded$data[[1]] %>% head(3)
#> # A tibble: 3 x 11
#> created_at year month day wday hour ymd wd_or_wk
#> <dttm> <dbl> <dbl> <int> <dbl> <int> <date> <chr>
#> 1 2016-06-23 00:36:25 2016 6 23 5 0 2016-06-23 weekday
#> 2 2014-09-03 15:46:45 2014 9 3 4 15 2014-09-03 weekday
#> 3 2015-03-31 23:21:51 2015 3 31 3 23 2015-03-31 weekday
#> # … with 3 more variables: time_numeric <dbl>, rest_or_work <chr>,
#> # wk.am_or_wk.pm <chr>
# calculate proportion of categories for four variables: wd_or_wk, rest_or_work, wk.am_or_wk.pm
df_expanded <- df_expanded %>%
prop_factor_nested(wd_or_wk, rest_or_work, wk.am_or_wk.pm)
#> 🛠 Start calculating proportion...
#>
#> ✅ Finish calculating! There are 6 new calculated variables: rest, weekday, weekend, weekend_am, weekend_pm, work
#> ⌛ Calculating time: 1.314 secs
#>
# show result
df_expanded %>% head(3)
#> # A tibble: 3 x 15
#> u_id n_tweets n_locs grid_id data n_tweets_loc n_hrs_loc n_days_loc
#> <int> <int> <int> <int> <lis> <int> <int> <int>
#> 1 4.02e7 1688 233 759 <tib… 467 23 366
#> 2 4.02e7 1688 233 842 <tib… 57 16 51
#> 3 4.02e7 1688 233 1212 <tib… 71 13 59
#> # … with 7 more variables: period_loc <dbl>, rest <dbl>, weekday <dbl>,
#> # weekend <dbl>, weekend_am <dbl>, weekend_pm <dbl>, work <dbl>
The score_nested()
function allows you to give a weighted value for one or more variables in a nested dataframe.
df
: a nested dataframe by useruser
: name of column that holds unique identifier for each userlocation
: name of column that holds unique identifier for each locationkeep_ori_vars
: option to keep or drop original varialbes...
: name-value pairs of expressionThe score_summary()
function summarises all scored columns and return one single summary score per row.
df
: a dataframeuser
: name of column that holds unique identifier for each userlocation
: name of column that holds unique identifier for each location...
: name of scored columns
## let's add two more variables before we do the scoring
df_expanded <- df_expanded %>%
summarise_nested(n_wdays_loc = n_distinct(wday),
n_months_loc = n_distinct(month))
#> 🛠 Start summarising values...
#>
#> ✅ Finish summarising! There are 2 new added variables: n_wdays_loc, n_months_loc
#> ⌛ Summarising time: 0.184 secs
#>
df_expanded %>% head(3)
#> # A tibble: 3 x 17
#> u_id n_tweets n_locs grid_id data n_tweets_loc n_hrs_loc n_days_loc
#> <int> <int> <int> <int> <lis> <int> <int> <int>
#> 1 4.02e7 1688 233 759 <tib… 467 23 366
#> 2 4.02e7 1688 233 842 <tib… 57 16 51
#> 3 4.02e7 1688 233 1212 <tib… 71 13 59
#> # … with 9 more variables: period_loc <dbl>, rest <dbl>, weekday <dbl>,
#> # weekend <dbl>, weekend_am <dbl>, weekend_pm <dbl>, work <dbl>,
#> # n_wdays_loc <int>, n_months_loc <int>
# when calculating scores, you can give weight to different variables, but the total weight should add up to 1
df_scored <- df_expanded %>%
score_nested(., user = "u_id", location = "grid_id", keep_original_vars = F,
s_n_tweets_loc = 0.1 * (n_tweets_loc/max(n_tweets_loc)),
s_n_hrs_loc = 0.1 * (n_hrs_loc/24),
s_n_days_loc = 0.1 * (n_days_loc/max(n_days_loc)),
s_period_loc = 0.1 * (period_loc/max(period_loc)),
s_n_wdays_loc = 0.1 * (n_wdays_loc/7),
s_n_months_loc = 0.1 * (n_months_loc/12),
s_weekend = 0.1 * (weekend),
s_rest = 0.2 * (rest),
s_wk_am = 0.1 * (weekend_am))
#> 🛠 Start scoring ...
#> ✅ Finish scoring! There are 9 new added variables: s_n_tweets_loc, s_n_hrs_loc, s_n_days_loc, s_period_loc, s_n_wdays_loc, s_n_months_loc, s_weekend, s_rest, s_wk_am
#> ⌛ Scoring time: 0.124 secs
#>
df_scored %>% head(3)
#> # A tibble: 3 x 2
#> u_id data
#> <int> <list>
#> 1 40153763 <tibble [12 × 10]>
#> 2 92298565 <tibble [11 × 10]>
#> 3 31088399 <tibble [4 × 10]>
df_scored$data[[1]]
#> # A tibble: 12 x 10
#> grid_id s_n_tweets_loc s_n_hrs_loc s_n_days_loc s_period_loc s_n_wdays_loc
#> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 759 0.1 0.0958 0.1 0.0932 0.1
#> 2 842 0.0122 0.0667 0.0139 0.1 0.1
#> 3 1212 0.0152 0.0542 0.0161 0.0225 0.1
#> 4 593 0.0133 0.0583 0.0142 0.0863 0.1
#> 5 1125 0.00600 0.0542 0.00656 0.0778 0.1
#> 6 797 0.00835 0.0667 0.0104 0.0851 0.1
#> 7 843 0.0274 0.0583 0.0301 0.0692 0.1
#> 8 862 0.0278 0.0583 0.0295 0.0722 0.1
#> 9 1709 0.00343 0.0542 0.00437 0.0870 0.1
#> 10 1174 0.00835 0.0542 0.0101 0.0752 0.1
#> 11 614 0.00321 0.0542 0.00410 0.0986 0.0714
#> 12 1038 0.00428 0.0458 0.00519 0.0784 0.1
#> # … with 4 more variables: s_n_months_loc <dbl>, s_weekend <dbl>, s_rest <dbl>,
#> # s_wk_am <dbl>
### we can replace the score function by mutate function
df_scored_2 <- df_expanded %>%
nest_verbose(-u_id) %>%
mutate_nested(s_n_tweets_loc = 0.1 * (n_tweets_loc/max(n_tweets_loc)),
s_n_hrs_loc = 0.1 * (n_hrs_loc/24),
s_n_days_loc = 0.1 * (n_days_loc/max(n_days_loc)),
s_period_loc = 0.1 * (period_loc/max(period_loc)),
s_n_wdays_loc = 0.1 * (n_wdays_loc/7),
s_n_months_loc = 0.1 * (n_months_loc/12),
s_weekend = 0.1 * (weekend),
s_rest = 0.2 * (rest),
s_wk_am = 0.1 * (weekend_am))
#> 🛠 Start nesting...
#> ✅ Finish nesting!
#> ⌛ Nesting time: 0.008 secs
#>
#> 🛠 Start adding variable(s)...
#>
#> ✅ Finish adding! There are 9 new added variables: s_n_tweets_loc, s_n_hrs_loc, s_n_days_loc, s_period_loc, s_n_wdays_loc, s_n_months_loc, s_weekend, s_rest, s_wk_am
#> ⌛ Adding time: 0.065 secs
#>
df_scored_2 %>% head(3)
#> # A tibble: 3 x 2
#> u_id data
#> <int> <list>
#> 1 40153763 <tibble [12 × 25]>
#> 2 92298565 <tibble [11 × 25]>
#> 3 31088399 <tibble [4 × 25]>
df_scored_2$data[[1]]
#> # A tibble: 12 x 25
#> n_tweets n_locs grid_id data n_tweets_loc n_hrs_loc n_days_loc period_loc
#> <int> <int> <int> <lis> <int> <int> <int> <dbl>
#> 1 1688 233 759 <tib… 467 23 366 1211.
#> 2 1688 233 842 <tib… 57 16 51 1300.
#> 3 1688 233 1212 <tib… 71 13 59 292.
#> 4 1688 233 593 <tib… 62 14 52 1121.
#> 5 1688 233 1125 <tib… 28 13 24 1011.
#> 6 1688 233 797 <tib… 39 16 38 1106.
#> 7 1688 233 843 <tib… 128 14 110 900.
#> 8 1688 233 862 <tib… 130 14 108 938.
#> 9 1688 233 1709 <tib… 16 13 16 1130.
#> 10 1688 233 1174 <tib… 39 13 37 977.
#> 11 1688 233 614 <tib… 15 13 15 1281.
#> 12 1688 233 1038 <tib… 20 11 19 1018.
#> # … with 17 more variables: rest <dbl>, weekday <dbl>, weekend <dbl>,
#> # weekend_am <dbl>, weekend_pm <dbl>, work <dbl>, n_wdays_loc <int>,
#> # n_months_loc <int>, s_n_tweets_loc <dbl>, s_n_hrs_loc <dbl>,
#> # s_n_days_loc <dbl>, s_period_loc <dbl>, s_n_wdays_loc <dbl>,
#> # s_n_months_loc <dbl>, s_weekend <dbl>, s_rest <dbl>, s_wk_am <dbl>
# sum varialbes score for each location
df_score_summed <- df_scored %>%
score_summary(., user = "u_id", location = "grid_id", starts_with("s_"))
#> 🛠 Start summing scores...
#>
#> ✅ Finish summing! There are 1 new added variables: score
#> ⌛ Summing time: 0.096 secs
#>
df_score_summed %>% head(3)
#> # A tibble: 3 x 2
#> u_id data
#> <int> <list>
#> 1 40153763 <tibble [12 × 11]>
#> 2 92298565 <tibble [11 × 11]>
#> 3 31088399 <tibble [4 × 11]>
df_score_summed$data[[1]]
#> # A tibble: 12 x 11
#> grid_id s_n_tweets_loc s_n_hrs_loc s_n_days_loc s_period_loc s_n_wdays_loc
#> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 759 0.1 0.0958 0.1 0.0932 0.1
#> 2 842 0.0122 0.0667 0.0139 0.1 0.1
#> 3 1212 0.0152 0.0542 0.0161 0.0225 0.1
#> 4 593 0.0133 0.0583 0.0142 0.0863 0.1
#> 5 1125 0.00600 0.0542 0.00656 0.0778 0.1
#> 6 797 0.00835 0.0667 0.0104 0.0851 0.1
#> 7 843 0.0274 0.0583 0.0301 0.0692 0.1
#> 8 862 0.0278 0.0583 0.0295 0.0722 0.1
#> 9 1709 0.00343 0.0542 0.00437 0.0870 0.1
#> 10 1174 0.00835 0.0542 0.0101 0.0752 0.1
#> 11 614 0.00321 0.0542 0.00410 0.0986 0.0714
#> 12 1038 0.00428 0.0458 0.00519 0.0784 0.1
#> # … with 5 more variables: s_n_months_loc <dbl>, s_weekend <dbl>, s_rest <dbl>,
#> # s_wk_am <dbl>, score <dbl>
The extract_location()
function allows you to sort the locations of each each user based on descending value of selected column, and return the top N location(s) for each user. The extracted location(s) then represent the most likely home location of the user.
df
: a nested dataframe by useruser
: name of column that holds unique identifier for each userlocation
: name of column that holds unique identifier for each locationshow_n_loc
: a single number that decides the number of locations to be extractedkeep_score
: option to keep or remove columns with scored value...
: name of column(s) that used to sort the locations
# extract homes for users based on score value (each user return 1 most possible home)
df_home <- df_score_summed %>%
extract_location(., user = "u_id", location = "grid_id", show_n_loc = 1, keep_score = F, score)
#> 🛠 Start extracting homes for users...
#>
#> 🎉 Congratulations!! Your have found 29 users' potential home(s).
#> ⌛ Extracting time: 0.085 secs
#>
df_home %>% head(3)
#> # A tibble: 3 x 2
#> u_id home
#> <int> <chr>
#> 1 40153763 759
#> 2 92298565 1582
#> 3 31088399 1113
# extract homes for users and keep scores of locations
df_home <- df_score_summed %>%
extract_location(., user = "u_id", location = "grid_id", show_n_loc = 1, keep_score = T, score)
#> 🛠 Start extracting homes for users...
#> 🎉 Congratulations!! Your have found 29 users' potential home(s).
#> ⌛ Extracting time: 0.08 secs
#>
df_home %>% head(3)
#> # A tibble: 3 x 3
#> u_id data home
#> <int> <list> <chr>
#> 1 40153763 <tibble [12 × 11]> 759
#> 2 92298565 <tibble [11 × 11]> 1582
#> 3 31088399 <tibble [4 × 11]> 1113
df_home$data[[3]]
#> # A tibble: 4 x 11
#> grid_id s_n_tweets_loc s_n_hrs_loc s_n_days_loc s_period_loc s_n_wdays_loc
#> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 880 0.0774 0.1 0.0577 0.0796 0.1
#> 2 1113 0.1 0.0917 0.1 0.1 0.1
#> 3 862 0.0519 0.1 0.0477 0.0716 0.1
#> 4 1594 0.00501 0.05 0.00846 0.0621 0.0714
#> # … with 5 more variables: s_n_months_loc <dbl>, s_weekend <dbl>, s_rest <dbl>,
#> # s_wk_am <dbl>, score <dbl>
The spread_nested()
function works in the same way as spread function in tidyverse but works inside a nested data frame. It allows to spread a key-value pairs across multiple columns.
df
: a nested dataframevar_key
: column name or positionvar_value
: column name or position
# let's add one timeframe variable first and calculate the number of data points at each timeframe
df_timeframe <- df_enriched %>%
mutate_nested(timeframe = if_else(hour >= 2 & hour < 8, "Rest", if_else(hour >= 8 & hour < 19, "Active", "Leisure")))
#> 🛠 Start adding variable(s)...
#>
#> ✅ Finish adding! There are 1 new added variables: timeframe
#> ⌛ Adding time: 0.177 secs
#>
colnames_nested_data <- df_timeframe$data[[1]] %>% names()
colnames_to_nest <- colnames_nested_data[-which(colnames_nested_data %in% c("grid_id", "timeframe"))]
df_timeframe <- df_timeframe %>%
head(20) %>% # take first 20 users as example
summarise_double_nested(., nest_cols = colnames_to_nest,
n_points_timeframe = n())
#> 🛠 Start summarising values...
#> ✅ Finish summarising! There are 1 new added variables: n_points_timeframe
#> ⌛ Summarising time: 3.522 secs
#>
df_timeframe$data[[2]]
#> # A tibble: 102 x 4
#> grid_id timeframe data n_points_timeframe
#> <int> <chr> <list> <int>
#> 1 1461 Active <tibble [280 × 7]> 280
#> 2 1461 Leisure <tibble [173 × 7]> 173
#> 3 1461 Rest <tibble [418 × 7]> 418
#> 4 1239 Active <tibble [1 × 7]> 1
#> 5 1460 Leisure <tibble [53 × 7]> 53
#> 6 1449 Leisure <tibble [21 × 7]> 21
#> 7 1629 Leisure <tibble [1 × 7]> 1
#> 8 1260 Active <tibble [2 × 7]> 2
#> 9 1399 Rest <tibble [2 × 7]> 2
#> 10 1345 Active <tibble [1 × 7]> 1
#> # … with 92 more rows
# spread timeframe in nested dataframe with key is timeframe and value is n_points_timeframe
df_timeframe_spreaded <- df_timeframe %>%
spread_nested(., key_var = "timeframe", value_var = "n_points_timeframe")
#> 🛠 Start spreading timeframe variable...
#> ✅ Finish spreading! There are 3 new added variables: Active, Leisure, Rest
#> ⌛ Spreading time: 0.252 secs
#>
df_timeframe_spreaded$data[[2]]
#> # A tibble: 102 x 5
#> grid_id data Active Leisure Rest
#> <int> <list> <int> <int> <int>
#> 1 354 <tibble [1 × 7]> 0 1 0
#> 2 355 <tibble [1 × 7]> 1 0 0
#> 3 374 <tibble [1 × 7]> 0 1 0
#> 4 597 <tibble [1 × 7]> 0 1 0
#> 5 611 <tibble [1 × 7]> 1 0 0
#> 6 612 <tibble [1 × 7]> 1 0 0
#> 7 613 <tibble [1 × 7]> 1 0 0
#> 8 614 <tibble [1 × 7]> 1 0 0
#> 9 645 <tibble [3 × 7]> 3 0 0
#> 10 645 <tibble [1 × 7]> 0 0 1
#> # … with 92 more rows
The arrange_nested()
function works in the same way as arrange function in tidyverse but works inside a nested data frame. It allows to sort rows by variables.
df
: a nested dataframe...
: comma separated list of unquoted variable names
df_enriched$data[[3]]
#> # A tibble: 938 x 8
#> created_at grid_id year month day wday hour ymd
#> <dttm> <int> <dbl> <dbl> <int> <dbl> <int> <date>
#> 1 2014-07-18 10:08:21 1375 2014 7 18 6 10 2014-07-18
#> 2 2013-11-24 23:16:24 1375 2013 11 24 1 23 2013-11-24
#> 3 2013-06-12 00:59:42 1375 2013 6 12 4 0 2013-06-12
#> 4 2013-06-07 19:56:56 1375 2013 6 7 6 19 2013-06-07
#> 5 2013-06-12 01:08:29 1375 2013 6 12 4 1 2013-06-12
#> 6 2013-06-02 20:53:09 1375 2013 6 2 1 20 2013-06-02
#> 7 2013-06-10 20:57:17 1375 2013 6 10 2 20 2013-06-10
#> 8 2013-06-04 17:42:06 1375 2013 6 4 3 17 2013-06-04
#> 9 2014-07-10 01:00:45 1375 2014 7 10 5 1 2014-07-10
#> 10 2013-06-13 01:48:27 1375 2013 6 13 5 1 2013-06-13
#> # … with 928 more rows
df_arranged <- df_enriched %>%
arrange_nested(desc(hour)) # arrange the hour in descending order
#> 🛠 Start sorting...
#>
#> ✅ Finish sorting!
#> ⌛ Sorting time: 0.263 secs
#>
df_arranged$data[[3]]
#> # A tibble: 938 x 8
#> created_at grid_id year month day wday hour ymd
#> <dttm> <int> <dbl> <dbl> <int> <dbl> <int> <date>
#> 1 2013-11-24 23:16:24 1375 2013 11 24 1 23 2013-11-24
#> 2 2013-06-08 23:41:19 1375 2013 6 8 7 23 2013-06-08
#> 3 2013-06-02 23:21:04 1375 2013 6 2 1 23 2013-06-02
#> 4 2013-06-05 23:31:59 1375 2013 6 5 4 23 2013-06-05
#> 5 2013-06-02 23:29:43 1375 2013 6 2 1 23 2013-06-02
#> 6 2013-06-06 23:51:50 1375 2013 6 6 5 23 2013-06-06
#> 7 2014-04-27 23:40:59 1375 2014 4 27 1 23 2014-04-27
#> 8 2013-06-05 23:47:27 1375 2013 6 5 4 23 2013-06-05
#> 9 2013-06-02 23:02:04 1375 2013 6 2 1 23 2013-06-02
#> 10 2013-06-06 23:07:49 1375 2013 6 6 5 23 2013-06-06
#> # … with 928 more rows
The arrange_double_nested()
function works in the similar way as arrange_nested()
function, but it applies to a double nested data frame. You need to choose the columns to nest inside a nested dataframe and then sort rows by selected column.
- `df`: a nested dataframe
- `nest_cols`: name of columns to nest in existing list-column
- `...`: comma separated list of unquoted variable names
# get the name of columns to nest
colnames_nested_data <- df_enriched$data[[1]] %>% names()
colnmaes_to_nest <- colnames_nested_data[-which(colnames_nested_data %in% c("grid_id"))]
df_double_arranged <- df_enriched %>%
head(20) %>% # take the first 20 users for example
arrange_double_nested(., nest_cols = colnmaes_to_nest,
desc(created_at)) # sort by time in descending order
#> 🛠 Start sorting...
#> ✅ Finish sorting!
#> ⌛ Sorting time: 2.5 secs
#>
# original third user
df_enriched[3, ]
#> # A tibble: 1 x 2
#> u_id data
#> <int> <list>
#> 1 11616678 <tibble [938 × 8]>
# third user data points
df_enriched$data[[3]]
#> # A tibble: 938 x 8
#> created_at grid_id year month day wday hour ymd
#> <dttm> <int> <dbl> <dbl> <int> <dbl> <int> <date>
#> 1 2014-07-18 10:08:21 1375 2014 7 18 6 10 2014-07-18
#> 2 2013-11-24 23:16:24 1375 2013 11 24 1 23 2013-11-24
#> 3 2013-06-12 00:59:42 1375 2013 6 12 4 0 2013-06-12
#> 4 2013-06-07 19:56:56 1375 2013 6 7 6 19 2013-06-07
#> 5 2013-06-12 01:08:29 1375 2013 6 12 4 1 2013-06-12
#> 6 2013-06-02 20:53:09 1375 2013 6 2 1 20 2013-06-02
#> 7 2013-06-10 20:57:17 1375 2013 6 10 2 20 2013-06-10
#> 8 2013-06-04 17:42:06 1375 2013 6 4 3 17 2013-06-04
#> 9 2014-07-10 01:00:45 1375 2014 7 10 5 1 2014-07-10
#> 10 2013-06-13 01:48:27 1375 2013 6 13 5 1 2013-06-13
#> # … with 928 more rows
# arranged third user
df_double_arranged[3, ]
#> # A tibble: 1 x 2
#> u_id data
#> <int> <list>
#> 1 11616678 <tibble [61 × 2]>
# arranged time
df_double_arranged$data[[3]]$data[[2]]
#> # A tibble: 3 x 7
#> created_at year month day wday hour ymd
#> <dttm> <dbl> <dbl> <int> <dbl> <int> <date>
#> 1 2013-06-07 09:44:12 2013 6 7 6 9 2013-06-07
#> 2 2013-06-04 09:42:57 2013 6 4 3 9 2013-06-04
#> 3 2013-06-02 13:40:22 2013 6 2 1 13 2013-06-02
The top_n_nested()
function works in the same way as top_n function in tidyverse but for nested dataframe. It allows to select the top entries in each group, ordered by wt.
df_enriched$data[[2]]
#> # A tibble: 1,170 x 8
#> created_at grid_id year month day wday hour ymd
#> <dttm> <int> <dbl> <dbl> <int> <dbl> <int> <date>
#> 1 2014-10-03 16:29:48 1461 2014 10 3 6 16 2014-10-03
#> 2 2014-01-29 23:29:27 1461 2014 1 29 4 23 2014-01-29
#> 3 2014-12-07 04:54:01 1461 2014 12 7 1 4 2014-12-07
#> 4 2014-09-24 00:27:25 1461 2014 9 24 4 0 2014-09-24
#> 5 2014-05-05 09:19:41 1239 2014 5 5 2 9 2014-05-05
#> 6 2014-12-17 16:36:28 1461 2014 12 17 4 16 2014-12-17
#> 7 2014-01-29 23:00:14 1461 2014 1 29 4 23 2014-01-29
#> 8 2014-12-15 07:47:08 1461 2014 12 15 2 7 2014-12-15
#> 9 2014-09-28 01:28:52 1461 2014 9 28 1 1 2014-09-28
#> 10 2015-01-20 16:24:24 1461 2015 1 20 3 16 2015-01-20
#> # … with 1,160 more rows
## get the top 1 row based on hour
df_top_1 <- df_enriched %>%
top_n_nested(., n = 1, wt = "hour")
#> 🛠 Start selecting top 1 row(s)...
#>
#> ✅ Finish selecting top 1 row(s)!
#> ⌛ Selecting time: 0.163 secs
#>
df_top_1$data[[2]]
#> # A tibble: 46 x 8
#> created_at grid_id year month day wday hour ymd
#> <dttm> <int> <dbl> <dbl> <int> <dbl> <int> <date>
#> 1 2014-01-29 23:29:27 1461 2014 1 29 4 23 2014-01-29
#> 2 2014-01-29 23:00:14 1461 2014 1 29 4 23 2014-01-29
#> 3 2014-05-07 23:15:11 1460 2014 5 7 4 23 2014-05-07
#> 4 2014-06-20 23:00:19 1461 2014 6 20 6 23 2014-06-20
#> 5 2014-05-07 23:05:08 1425 2014 5 7 4 23 2014-05-07
#> 6 2014-08-15 23:40:17 1460 2014 8 15 6 23 2014-08-15
#> 7 2014-04-28 23:27:11 1461 2014 4 28 2 23 2014-04-28
#> 8 2015-03-06 23:04:42 1471 2015 3 6 6 23 2015-03-06
#> 9 2014-12-21 23:01:58 1461 2014 12 21 1 23 2014-12-21
#> 10 2014-05-07 23:23:21 1460 2014 5 7 4 23 2014-05-07
#> # … with 36 more rows
To use the embedded recipes to identify the home location for users, you can use identify_location()
function.
df
: a dataframe with columns for the user id, location, timestampuser
: name of column that holds unique identifier for each usertimestamp
: name of timestamp column. Should be POSIXctlocation
: name of column that holds unique identifier for each locationtz
: a character string containing the time zone to convert to and it should be recognized in Rrecipe
: embedded algorithms to identify the most possible home locations for usersshow_n_loc
: number of potential homes to extractkeep_score
: option to keep or remove calculated result/score per user per locationCurrent available recipes:
recipe_HMLC
:
recipe_FREQ
recipe_OSNA
: Efstathiades et al.2015
recipe_APDM
: Ahas et al. 2010
# recipe: homelocator -- HMLC
identify_location(test_sample, user = "u_id", timestamp = "created_at",
location = "grid_id", show_n_loc = 1, recipe = "HMLC")
# recipe: Frequency -- FREQ
identify_location(test_sample, user = "u_id", timestamp = "created_at",
location = "grid_id", show_n_loc = 1, recipe = "FREQ")
# recipe: Online Social Network Activity -- OSNA
identify_location(test_sample, user = "u_id", timestamp = "created_at",
location = "grid_id", show_n_loc = 1, recipe = "OSNA")
# recipe: Online Social Network Activity -- APDM
## APDM recipe strictly returns the most likely home location
## It is important to load the neighbors table before you use the recipe!!
## example: st_queen <- function(a, b = a) st_relate(a, b, pattern = "F***T****")
## neighbors <- st_queen(df_sf) ===> convert result to dataframe
data("df_neighbors", package = "homelocator")
identify_location(test_sample, user = "u_id", timestamp = "created_at",
location = "grid_id", recipe = "APDM", keep_score = F)