I'm an English expat living in Avoca Beach, Australia. I'm a proud father of 3 girls and strive to be an active, engaged and responsible member of both my local community and the broader open-source tech community.
I work for Dimension Data, a global services company as their group director of innovation and technical talent.
All that means is I care about our people having the best skills to pursue future careers in the ever-changing world of tech.
I Tweet often, blog frequently and speak seldomly. All of my research and tinkering is Open-Source, I'm an active member of some large open-source projects as well as maintaining my own.


Gateway
BOT
21-May-19 12:36 PM
[telegram] <Bram4> Hello, isnt it possible to run an asset chain in native and kmd in lite simultanious anymore in agama?
[telegram] <kracken> Guys what are the advantages of using komodo to secure a project's network?
[telegram] <kracken> I want to help convince polis to use komodo to improve security
[telegram] <kracken> What arw the Disadvantages of using komodo, in regards to 51% attacks?
[telegram] <Lennart> dpow protected chains are effectively protected by the massive Bitcoin hashrate. that's a huge advantage isnt it?
[telegram] <Lennart> as for disadvantage.. i dont know, takes some steps to setup maybe.. it costs some kmd..
[telegram] <kracken> What is the cost? (re @Lennart: as for disadvantage.. i dont know, takes some steps to setup maybe.. it costs some kmd..)
[telegram] <kracken> Absolutely 100% (re @Lennart: dpow protected chains are effectively protected by the massive Bitcoin hashrate. that's a huge advantage isnt it?)
[telegram] <kracken> I can't figure out why it isn't used by everyone
[telegram] <Lennart> more will come for sure
[telegram] <Lennart> as for cost i think it's best to talk to the team directly as there are different levels of dpow AFAIK
[telegram] <Lennart> around 300 KMD
[telegram] <Lennart> per year
[telegram] <Lennart> if seriously interested, you should join the Komodo Discord channel
[telegram] <wktong> Not many merchant accept Komodo, too sad.
[telegram] <Lennart> not many merchants accept crypto
[telegram] <kracken> 300KMD/year (re @Lennart: around 300 KMD)
[telegram] <kracken> I wonder how much that is in usd and btc terms. I haven't been following the price
[telegram] <Lennart> all i know is that if you'd bought around 7000 KMD the rewards alone will pay for dPoW..


Concepts¶
Why airflow-fs?¶
Although Airflow provides a large set of builtin hooks and operators to work with, these builtin components generally lack a common reusable interface across related components. This limitation is especially glaring in the set of file system hooks provided by Airflow, which require a developer to use entirely different interfaces for working with different file systems. This (among other things) has lead to the widespread development of many a-to-b operators (e.g., S3ToHiveOperator, GcsToS3Operator, etc.), resulting in unnecessary code duplication.
airflow-fs aims to solve this issue by defining a common interface for file system hooks, which is based on a subset of functions from the os and shutil modules in the Python standard library. This interface allows operations to be performed across different file systems using the same code, easing their use for developers. Moreover, the common interface increases the composability of file system hooks, enabling the development of common operators for performing tasks, independent of the underlying file system(s).
File system hooks¶
File system hooks are Airflow hooks that follow a common interface by extending the FSHook base class and providing implementations for several abstract methods. This allows us to work with different file systems using virtually the same code. For example, following this approach makes reading a file from an FTP server or an S3 file system virtually identical:
from airflow_fs.hooks import FtpHook, S3Hook
# Reading a file from FTP.
with FtpHook(conn_id="ftp_default") as ftp_hook:
with ftp_hook.open("some_file.txt") as file_:
content = file_.read()
# Reading a file from S3.
with S3Hook(conn_id="s3_default") as s3_hook:
with s3_hook.open("some_file.txt") as file_:
content = file_.read()
For more details on the methods provided by file system hooks, see the Usage and API sections of this documentation.
Out of the box, airflow-fs provides hooks for a number of frequently used file systems such as FTP, S3, SFTP and HDFS. Support for additional file systems can be added by implementing additional FsHook subclasses, which provide file system-specific implementations for the following methods:
- open - Opens a file for reading writing, similar to the builtin open function.
- exists - Checks if a given file or directory exists, similar to os.path.exists.
- isdir - Checks if a given path points to a directory, similar to os.path.isdir.
- listdir - Lists files and subdirectories in a given directory, similar to os.listdir.
- mkdir - Creates a new directory, similar to os.mkdir.
- rm - Deletes a file, similar to os.unlink.
- rmtree - Deletes a directory tree, similar to shutil.rmtree.
Additional methods for more complex operations such as copying files, etc. are readily provided by the FsHook base class (see the FSHook API for more details). These methods are generally implemented using the base methods above and do not have to implemented for each specific hook.
File system operators¶
Besides hooks, airflow-fs also provides several file system operators for performing common tasks such as copying and deleting files. These operators are built on top of file system hooks to make their implementation independent of the underlying file system.
For example, copying two files between any two file systems can be achieved using the CopyFileOperator:
from airflow_fs.hooks import S3Hook, FtpHook
from airflow_fs.operators import CopyFileOperator
copy_file_task = CopyFileOperator(
src_path="my-bucket/example.txt",
dest_path="example.txt",
src_hook=S3Hook(conn_id="s3_default"),
dest_hook=FtpHook(conn_id="ftp_default"),
task_id="copy_file",
dag=dag)
)
For more details on the different file system operators, see the Usage and API sections.
File system sensors¶
In a fashion similar to the file system operators, airflow-fs also provides file system sensors that can be used to wait for files or directories to show up on a given file system.
For example, the FileSensor class allows us to wait for files matching a given file pattern:
from airflow_fs.hooks import S3Hook
from airflow_fs.sensors import FileSensor
file_sensor = FileSensor(
path="my-bucket/*.txt",
hook=S3Hook(conn_id="s3_default"),
task_id="file_sensor",
dag=dag
)
For more details, see the Usage and API sections.


Problem
Once painstakingly collected, data is most value when actionable steps are taken from it. Methods of safely distributing and communicating results from data are not all created equally. Dashboards are a safe and effective way to communicate results from data.
Actions
We’ll walk through an example of how to create a dashboard in R using flexdashboard, an R framework for creating dashboards with R and Markdown.
Example
For this example we’ll grab data from Google Analytics and use it to make 3 charts on a dashboard.
- A histogram
- A time series chart
- Sortable table
To get this all to work, we’ll need to wrap our code in a flexdashboard template. You can find out how to do that here. We’ll use the row template to get our charts displaying well on the page and allow for vertical scrolling.
---
title: "My Google Analytics Dashboard"
output:
flexdashboard::flex_dashboard:
orientation: rows
vertical_layout: scroll
---
For chart 1 we’ll use DT, a data table library to create a searchable and sortable table for our dashboard. I used data from the in-market segment in Google Analytics. Plotting the chart with default stylings is easy:
Chart 2 will be our histogram. The code for creating a histogram in Highcharter is very simple as well:
hchart(dataset$Sessions, color = "#B71C1C", name = "Sessions") %>%
hc_title(text = "Sessions May 2015 - May 2016")
This should plot a histogram of traffic to my website from May 2015 through May 2016. This chart is also zoomable, which is a nice feature to get for free.
Chart 3 is a time series chart. I will also use Highcharter for this as well:
highchart() %>%
hc_add_series_times_values(as.Date(dataset$Day.Index) , dataset$Sessions, name = "Sessions")
This gives us visits by time over the same range.
From here you can click “Knit” in Rstudio and it will compile this markdown document into HTML. You will have a working, interactive and responsive dashboard. I’ve published a working version to rPubs for reference here.
References
Problem
You love sports. You love data. If you’ve ever gone on an epic journey in search of sports data, you’ve probably resorted to scraping data from sites like ESPN or Baseball Reference and have spent countless hours writing Python code to use the powerful web scraping library BeautifulSoup. Maybe you even have a nice Python client that scrapes stats.nba.com or you wrote an R package to scrape Baseball Reference.
However, we all know that websites get redesigned, formats change and sometimes access to specific stats is revoked (e.g. the NBA did this with player movement tracking but attributes it to ‘technical difficulties’). Suddenly, you are scrambling to update your scraping code in order to account for a few new divs or other elements on the web page that were renamed.
Actions
This is where the Stattleship API comes to the rescue. We’ve built an easy-to-access set of sports data, stats and accomplishments for multiple sports (and expanding). We’ve partnered with Gracenote to provide all of our NFL, NBA, NHL and MLB game data. Our service is designed for creative fans who want to use Stattleship data to build sports apps that scale.
We’ve cleaned, structured, and categorized all of the box score, player stats and game log data for you. We’ve even quantified specific performances and feats so that you can quickly identify whether a stat is a common-place occurrence or a record-breaking achievement.
In this tutorial, we will show you how to use our R wrapper to access live and up-to-date sports data via the Stattleship API. There is also a Ruby gem for all of you Rubyists: https://github.com/stattleship/stattleship-ruby
Example
There are a few main endpoints that we will focus on. They are as follows:
- games - contains game information such as date, attendance, scoreline, etc.
- players- contains player information such as name, position, draft round, school, salary, weight, birthday, etc.
- teams - contains team information such as name, division, colors, hashtags, etc.
- game_logs - contains player-level game log information such as Kevin Durant’s assists from a specific game, total minutes played, etc.
- team_game_logs - contains team-level game log information such as total points, rebounds and turnovers by the Boston Celtics in a specific game
Here’s a basic Entity-Relationship Diagram outlining how these objects relate to one another:
In order to get started, sign up for a free API token from here. Now load up RStudio and start following along! The first few lines of this R script will install the R package from Github to your machine:
(If you don’t have devtools
installed you will have to install that first.)
install.packages("devtools")
devtools::install_github("stattleship/stattleship-r")
## Load the stattleshipR package
library(stattleshipR)
Next you need to set your API token in the R environment:
## Get your free token from www.stattleship.com
set_token("your-API-token-goes-here")
Now we need to specify the sport, league and endpoint we are interested in. In this case we will fetch all MLB regular season game logs for the Red Sox to date. This is as easy as setting three parameters: sport
, league
and ep
for endpoint.
sport <- 'baseball'
league <- 'mlb'
ep <- 'game_logs'
This last parameter, called q_body
is where we can set more granular options such as requesting a specific team, stat, player or season.
q_body <- list(team_id='mlb-bos', status='ended', interval_type='regularseason')
Now that all of our parameters are set, we can use ss_get_result
to send our request to the Stattleship API. Notice the walk=TRUE
option. This ensures that the request will walk through all pages of results and return everything. Results are returned 40 rows per page.
gls <- ss_get_result(sport=sport, league=league, ep=ep, query=q_body, walk=TRUE)
We now have a list in R that has 5 elements (there were 5 pages of results returned).
We want to combine all of the pages of results into one data.frame
:
game_logs<-do.call('rbind', lapply(gls, function(x) x$game_logs))
Let’s check out all of the data we now have access to from yesterday’s games:
Wow, over 80 variables for each player including strikeouts, walks, doubles, runs, and more.
I want to include player information into this data set though so I have more than just player_ids
. Let’s retrieve all Boston Red Sox players by changing the ep
to players
and pass team_id='mlb-bos
into the options list.
sport <- 'baseball'
league <- 'mlb'
ep <- 'players'
q_body <- list(team_id='mlb-bos')
pls <- ss_get_result(sport=sport, league=league, ep=ep, query=q_body, walk=TRUE)
players<-do.call('rbind', lapply(pls, function(x) x$players))
In order to merge the two data.frames we can simply rename the id
column to player_id
so we can use merge
in R like this:
colnames(players)[1] <- 'player_id'
game_logs <- merge(players, game_logs, by='player_id')
Now let’s do some basic player calculations using dplyr
.
install.packages('dplyr')
library(dplyr)
## Only include game_logs where a player actually played the game, calculate mean batting average,
## total runs, total bases and pull in salary info
stats <-
game_logs %>%
filter(game_played == TRUE) %>%
group_by(name) %>%
summarise(totalRuns = sum(runs), meanBA = mean(batting_average), totalBases=sum(total_bases), salary=max(salary))
Good! Now let’s plot it all using ggplot2
.
ggplot(stats, aes(x=totalRuns, y=meanBA, size=totalBases, label=name, color=salary)) + geom_text()
This plot actually enables us to visualize 4 different variables at once. The x and y axes display total runs and mean batting average, the size of the labels indicate how many total bases the player has had, and the color indicates salary. Xander Bogaerts and Travis Shaw are looking like quite a bargain at this point in the season already.
Let us know if you build anything cool with this awesome sports data API! We also welcome contributions to our R wrapper via pull requests in Github. We have a public Slack channel as well where you can join us to talk sports, data, get R or Stattleship API help, and provide feedback.
References / Other Examples
Problem
As data collection and aggregation grows in an organization, presenting and consuming that data can become a real challenge. A good foundation in visualization eases this burden. Using bar charts effectively can help with sw
Actions
We will walk through an example of how to create a useful and beautiful barchart in R.
Example
In R, the library ggplot2 is your best friend when it comes to creatingstatic bar charts. You can create a simple one with the following code:
library(ggplot2)
ggplot(mpg, aes(class, fill = class)) + geom_bar()
This bar chart is simple and effect but we can do better. We can start simple by adding titles.
ggplot(mpg, aes(class, fill = class)) + geom_bar() +
labs(title = "Count of Car Classes MPG Dataset", x = "Car Class", y = "Count")
The background space in this chart isn’t really pleasing to the eyes. We can make a customize the chart and make it all grey for effect:
library(ggplot2)
library(scales)
library(grid)
library(RColorBrewer)
library(extrafont)
library(reshape)
library(ggthemes)
blogData = mpg
palette <- brewer.pal("Greys", n=9)
color.background = palette[2]
ggplot(mpg, aes(class, fill = class)) + geom_bar() +
labs(title = "Count of Car Classes MPG Dataset", x = "Car Class", y = "Count")+
theme_bw(base_size=9)+
theme(panel.background=element_rect(fill=color.background, color=color.background)) +
theme(plot.background=element_rect(fill=color.background, color=color.background)) +
theme(panel.border=element_rect(color=color.background))+
theme(legend.background = element_rect(fill=color.background))
A few more modifications can be made.
- Sort by count
- Flip the axis
- Overlay the count
- Change the font of the chart
- Change the opacity of the bars
- Remove the legend (doesn’t add any value)
- Some other aesthetic fixes
library(ggplot2)
library(scales)
library(grid)
library(RColorBrewer)
library(extrafont)
library(reshape)
library(ggthemes)
library(dplyr)
blogData = mpg
palette <- brewer.pal("Greys", n=9)
color.background = palette[2]
newMPG <- group_by(mpg,class) %>%
summarise(total = n()) %>%
arrange(desc(total))
ggplot(newMPG, aes(x=reorder(class, +total), total)) + coord_flip()+ geom_bar(fill = "#6f00ff", stat="identity") +
labs(title = "Count of Car Classes", x = "Car Class", y = "Count")+
geom_text(aes(y=total - 1, label=total), position = position_dodge(.09), vjust= .5, size = 3, hjust = 1.3, fontface= "bold", family = "Hack")+
my_theme()
my_theme <- function() {
# Generate the colors for the chart procedurally with RColorBrewer
palette <- brewer.pal("Greys", n=9)
color.background = palette[2]
color.grid.major = palette[3]
color.axis.text = palette[6]
color.axis.title = palette[7]
color.title = palette[9]
# Begin construction of chart
theme_bw(base_size=9) +
# Set the entire chart region to a light gray color
theme(panel.background=element_rect(fill=color.background, color=color.background)) +
theme(plot.background=element_rect(fill=color.background, color=color.background)) +
theme(panel.border=element_rect(color=color.background)) +
# Format the grid
theme(panel.grid.major=element_line(color=color.grid.major,size=.25)) +
theme(panel.grid.minor=element_blank()) +
theme(axis.ticks=element_blank()) +
# Format the legend, but hide by default
theme(legend.position="none") +
theme(legend.background = element_rect(fill=color.background)) +
theme(legend.text = element_text(size=7,color=color.axis.title)) +
# Set title and axis labels, and format these and tick marks
theme(plot.title=element_text(color=color.title, size=15, vjust=1.25,face="bold", family="Hack")) +
theme(axis.text.x=element_text(size=10,color=color.axis.text)) +
theme(axis.text.y=element_text(size=10,color=color.axis.text)) +
theme(axis.title.x=element_text(size=10,color=color.axis.title, vjust=0, face="bold", family= "Hack")) +
theme(axis.title.y=element_text(size=10,color=color.axis.title, vjust=1.25, face="bold", family = "Hack")) +
theme(strip.text.y = element_text(size = 10, colour = "black", face="bold", family = "Hack")) +
theme(strip.background = element_rect(fill=color.background, color=color.background, size=1)) +
# Plot margins
theme(plot.margin = unit(c(0.35, 0.2, 0.3, 0.35), "cm"))
}
Now we have a chart worth sharing.
References:
ggplot2
Max Woolf


The purpose of yinarlingi is to provide a set of convenience functions for validating Warlpiri dictionary data. Essentially, it pre-configures various functions from the tidylex and tidyverse packages with parameters specific to the Warlpiri dictionary data, so that those within the Warlpiri dictionary project can quickly run and re-run frequently used routines (e.g. data validation, derived views), and also so that these tests can be run in a Continuous Testing/Continuous Deployment environment such as GitLab pipelines:
Yinarlingi is a Warlpiri word for echidna.
The Warlpiri dictionary consists of a large plain text file of the form (where \me
stands for main entry; see full list of codes):
\me jampaly(pa) (N) (PV): (La,Wi,Y)
\dm spatial: tactile: \edm
\gl sharp, pointed \egl
\rv sharp \erv
\eg
\we Karlangu ka karri jampalypa ngulaju yiri-nyayirni. \[@@]
\et The digging stick is sharp, that is very sharp pointed. \ewe
\eeg
\ant jampilypa, munju \eant
\cf jaarn-karri-mi, lirra jampalypa \ecf
\syn jalkarra, larrilpi, yiri \esyn
\eme
\sse jampaly-pi-nyi (V):
\def xERG cause y to come to be sharp (<jampalypa>) \edef
\gl sharpen, trim to ^point \egl
\rv sharpen \erv
\eg
\we Jampaly-pungulpalu wangkinypa, yangka kuja munju-jarrija. \[@@]
\et They sharpened the stone-axe which had became blunt. \ewe
\we Karlangulu jampalypa-pungu. \[Tiger Jakamarra]
\et They sharpened the points of the digging sticks. \ewe
\eeg
\csl YSL#1336, YSL#313 \ecsl
\syn yiri-ma-ni \esyn
\esse
...
A well-structured Warlpiri entry is defined as a Nearley grammar (see full grammar here), of which a snippet is shown below:
entryBlock -> "org":? "dm":* "def":? "lat":? "gl":? "rv":? "cm":*
(exampleBlock:+ | paradigmExample:+):?
crossRefs
We can see here that a semantic domain line dm
must come before the gloss line gl
. So, the following data (provided in wlp-lexicon_invalid-sequence.txt
) clearly does not satisfy this requirement:
\me jampaly(pa) (N) (PV): (La,Wi,Y)
\gl sharp, pointed \egl
\dm spatial: tactile: \edm
...
The yinarlingi test suite uses the test_code1_ordered()
function to catch such cases (notice code1_ok
is FALSE
on the third line):
library(yinarlingi)
# Get local path to the yinarlingi-provided file, 'wlp-lexicon_invalid-sequence.txt'
invalid_lexicon_path <- system.file("extdata/wlp-lexicon_invalid-sequence.txt", package = "yinarlingi")
test_code1_ordered(invalid_lexicon_path) %>%
knitr::kable() # For pretty display of table
| me_start | line | data | code1 | code1_ok | error |
|---|---|---|---|---|---|
| 1 : jampaly(pa) (N) (PV): (La,Wi,Y) | 1 | jampaly(pa) (N) (PV): (La,Wi,Y) | me | TRUE | NA |
| 2 | sharp, pointed | gl | TRUE | NA | |
| 3 | spatial: tactile: | dm | FALSE | Unexpected code1 | |
| 4 | sharp | rv | NA | NA | |
| 5 | eg | NA | NA | ||
| 6 | Karlangu ka karri jampalypa ngulaju yiri-nyayirni. [@@] | we | NA | NA |