class: center, middle, inverse, title-slide # Nonlinear Models ## Machine Learning in Economics ### Hüseyin Taştan ### Yildiz Technical University --- <style type="text/css"> .remark-slide-content { font-size: 28px; padding: 1em 4em 1em 4em; } .my-large-font { font-size: 40px; } .my-small-font { font-size: 20px; } .my-medium-font { font-size: 25px; } </style> # Lecture Outline - [Linear vs Nonlinear models](#linvsnonlin) - [Polynomial regression](#poly) - [Step functions](#step) - [Basis Functions](#basis) - [Regression Splines](#splines) - [Natural Splines](#natural) - [Smoothing Splines](#smoothing) - [Local regression](#local) - [Generalized Additive Models (GAMs)](#gams) --- class: my-medium-font name: linvsnonlin # Linear vs Nonlinear Models Model: `\(y = f(x) + u\)`, the form of `\(f(x)\)` determines the nature of the relationship .pull-left[ Linear Models - `\(f(x) = \beta_0 + \beta_1 x\)` - Linear in parameters and variables - Starting point, usually good approximations to reality - But may be improved significantly ] .pull-right[ Nonlinear Models - `\(f(x)\)` can be any function of `\(x\)` - Polynomial regression - Step function - Regression splines - Smoothing splines - Local regression - Generalized Additive Models (GAMs) - ... and others (neural nets, support vector machines, trees, etc.) ] --- name: poly # Polynomial Regression - For simplicity assume that there is a single predictor `\(x\)`. A polynomial of order `\(d\)` can be written as `$$y = \beta_0 + \beta_1 x + \beta_2 x^2 + \ldots + \beta_d x^d + \epsilon$$` - We added `\(d\)` powers of a single predictor in an additive manner. - Because the model is still linear in parameters, we can still use OLS. - The order of the polynomial, `\(d\)`, may be thought of as a tuning parameter that controls the complexity of the model. - In practice, `\(d\)` can be chosen by validation methods. --- # Ex.: Degree-4 polynomial regression of wage on age `$$\widehat{wage} = \hat{\beta}_0 + \hat{\beta}_1 ~age + \hat{\beta}_2~age^2 + \hat{\beta}_3~age^3 + \hat{\beta}_4~ age^4$$` .pull-left[ ![](img/nonlin1.PNG) ] .pull-right[ - Solid blue line represents the fitted curve surrounded by 95% confidence interval. - There is a distinct high-earners group (those earning more than $250.000) - How can we model high-earners category? ] --- # Logistic Regression with Polynomial Terms - Similar to the polynomial regression, we can modify the logistic regression with polynomial terms. - For example, for high-earners group we have `$$\operatorname{Pr}\left(y_{i}>250 \mid x_{i}\right)=\frac{\exp \left(\beta_{0}+\beta_{1} x_{i}+\beta_{2} x_{i}^{2}+\ldots+\beta_{d} x_{i}^{d}\right)}{1+\exp \left(\beta_{0}+\beta_{1} x_{i}+\beta_{2} x_{i}^{2}+\ldots+\beta_{d} x_{i}^{d}\right)}$$` where we treated wage as a binary variable by splitting it into two groups: those earning more than 250 thousand dollars, and those earning less. - The next graph shows the predicted probabilities and 95% confidence interval. --- # Example: Logistic Regression with Polynomial Terms ![](img/nonlin2.PNG) There are only 79 high-earners in the data set. Because the sample size is fairly small, the variances of coefficients are large resulting in wide confidence intervals. --- name: step # Step Functions .pull-left[ Indicator function: We create cutpoints `\(c_1,c_2,\ldots,c_k\)` in the range of `\(X\)` and then construct `\(K+1\)` new variables: `$$\begin{array}{ll} C_{0}(X) & =I\left(X<c_{1}\right) \\ C_{1}(X) & =I\left(c_{1} \leq X<c_{2}\right) \\ C_{2}(X) & =I\left(c_{2} \leq X<c_{3}\right) \\ & \vdots \\ C_{K-1}(X) & =I\left(c_{K-1} \leq X<c_{K}\right) \\ C_{K}(X) & =I\left(c_{K} \leq X\right) \end{array}$$` ] .pull-right[ - The indicator function, `\(I(\cdot)\)`, takes the value of 1 if the event inside the parenthesis is true, and 0 otherwise. It creates dummy variables based on the cutoff values `\(c\)`. - E.g. Consider: `\(c_1=35\)`, `\(c_2=50\)`, `\(c_3=65\)` - Thus, `\(I(age<35)=1\)` for those younger than 35 and 0 otherwise. Note that, each observation **must** fall into one of the categories `\(C_0,C_1,…,C_K\)`. ] --- # Step Functions - Notice that for any value of `\(X\)` we have `$$C_0(X) + C_1(X) + \ldots + C_K(X) = 1$$` because `\(X\)` must be in one of `\(K+1\)` groups. - Next, we simply fit an OLS regression using `\(C_1(X), C_2(X),\ldots, C_K(X)\)` as predictors `$$y_{i}=\beta_{0}+\beta_{1} C_{1}\left(x_{i}\right)+\beta_{2} C_{2}\left(x_{i}\right)+\ldots+\beta_{K} C_{K}\left(x_{i}\right)+\epsilon_{i}$$` - `\(\beta_0\)` can be interpreted as the mean value of `\(Y\)` when `\(X<c_1\)` - `\(\beta_j\)` represents average increase in response for `\(X\)` in `\(c_j\leq X<c_{j+1}\)` relative to `\(X<c_1\)`. - The next graph shows regression and logistic regression of wage on step functions of age (using cutpoints 35, 50 and 65) --- class: my-small-font # Example: Regression and Logistic regression ![](img/nonlin3.PNG) (source: ISLR Fig. 7.2, p.269) --- name: basis # Basis Functions - Polynomials and piecewise-constant regression models are special cases of basis functions - We have a family of functions, `\(b_1(X), b_2(X),\ldots,b_K(X)\)` that can applied to each predictor. - Then we simply use the basis function values instead of the original `\(X\)` values: `$$y_{i}=\beta_{0}+\beta_{1} b_{1}\left(x_{i}\right)+\beta_{2} b_{2}\left(x_{i}\right)+\beta_{3} b_{3}\left(x_{i}\right)+\ldots+\beta_{K} b_{K}\left(x_{i}\right)+\epsilon_{i} .$$` - For polynomial regression we have `\(b_j(x_i) = x_i^j\)` - For piece-wise constant functions we have `\(b_j(x_i) =I(c_j\leq x_i <c_{j+1})\)` - Many alternatives exist: wavelets, Fourier series, splines - Next, we will consider regression splines. --- # Piecewise Polynomials - An alternative to fitting a high-degree polynomial over the entire range of `\(X\)` is to fit separate low-degree polynomials over different regions of `\(X\)`. - The range of `\(X\)` values are cut into sub-intervals as in the step function approach. The cut points are called **knots** - For example, we may fit a piece-wise cubic model using the knot `\(c\)` `$$y_{i}=\left\{\begin{array}{ll} \beta_{01}+\beta_{11} x_{i}+\beta_{21} x_{i}^{2}+\beta_{31} x_{i}^{3}+\epsilon_{i} & \text { if } x_{i}<c \\ \beta_{02}+\beta_{12} x_{i}+\beta_{22} x_{i}^{2}+\beta_{32} x_{i}^{3}+\epsilon_{i} & \text { if } x_{i} \geq c \end{array}\right.$$` - It can be fitted using OLS. - We can change the degree of the polynomial or use any number of knots. --- # Example: Unconstrained piecewise cubic polynomial .pull-left[ ![](img/nonlin4.PNG) ] .pull-right[ - Piecewise cubic polynomial regression of wage on age with a single knot at age=50. - Notice that there is a huge discontinuity at age=50. - We can constrain the polynomial such that it is continuous at the knot. - In addition to continuity we may impose the smoothness condition where all the `\(d-1\)` derivatives of the degree `\(d\)` polynomial are continuous as well. ] --- # Example: Constrained piecewise cubic polynomial .pull-left[ ![](img/nonlin5.PNG) ] .pull-right[ - This graph shows the same piecewise cubic polynomial constrained to be continuous at age=50. - Note that there is still very small but visible jump at the knot. - Splines provide a smoother alternative that have the highest amount of continuity. This is shown in the next graph. ] --- name: splines # Example: Splines ![](img/nonlin6.PNG) Cubic splines are constrained to be continuous and have continuous first and second derivatives. <small>(Source: ISLR Fig. 7.3, p.272)<small> --- # Linear Splines - A linear spline with `\(K\)` knots, `\(\xi_k\)`, `\(k=1,2,\ldots,K\)`, can be written as `$$y_{i}=\beta_{0}+\beta_{1} x_{i}+\beta_{2} (x_i-\xi_1)_{+} + \beta_{3} (x_i-\xi_2)_{+} +\ldots + \beta_{1+K} (x_i-\xi_K)_{+} + \epsilon_{i}$$` <!-- where `\(b_k\)` are basis functions --> <!-- $$\begin{aligned} --> <!-- b_{1}\left(x_{i}\right) &=x_{i} \\ --> <!-- b_{k+1}\left(x_{i}\right) &=\left(x_{i}-\xi_{k}\right)_{+}, \quad k=1, \ldots, K --> <!-- \end{aligned}$$ --> where `\((\cdot)_+\)` means the positive part: `$$(x-\xi)_{+} = \left\{\begin{array}{cl} (x-\xi) & \text { if } x>\xi \\ 0 & \text { otherwise } \end{array}\right.$$` - For example, a linear spline regression with knots at age=25, 40 , and 60 can be written as `$$wage = \beta_0 + \beta_1 age + \beta_2 (age-25)_+ + \beta_3 (age-40)_+ + \beta_4 (age-60)_+ + \epsilon$$` There are `\(K=3\)` knots and 5 parameters to estimate. --- # Linear Splines: An illustration .pull-left[ ![](img/nonlin7.PNG) ] .pull-right[ - Global linear function `\(f(x) = \beta_0 + \beta_1 x\)` shown in blue. - Linear spline using a single knot at 0.6: `\(f(x) = \beta_0 + \beta_1 x + \beta_2 (x-0.6)_+\)` - The basis function is `\(b(x) =(x-0.6)_+\)` (shown in orange). ] - Note that the basis function starts at zero and maintains continuity at the knot=0.6. - Global function now changes slope at the knot. --- # Cubic Splines - Spline functions of degree `\(d\)` are piecewise polynomial functions of degree `\(d\)` but have continuous derivatives up to order `\(d-1\)` at knots. - A cubic spline with `\(K\)` knots can be written as `$$y_{i}=\beta_{0}+\beta_{1} b_{1}\left(x_{i}\right)+\beta_{2} b_{2}\left(x_{i}\right)+\cdots+\beta_{K+3} b_{K+3}\left(x_{i}\right)+\epsilon_{i}$$` - Cubic spline basis: start with `\(x,x^2, x^3\)` and add one truncated power basis function per knot `\(\xi\)`: `$$h(x, \xi)=(x-\xi)_{+}^{3}=\left\{\begin{array}{cl} (x-\xi)^{3} & \text { if } x>\xi \\ 0 & \text { otherwise } \end{array}\right.$$` where `\((\cdot)_+\)` means the positive part. --- # Cubic Splines - Let's say there are `\(K\)` knots. In order to fit a cubic spline to this data set we may just fit an OLS regression using `\(X,X^2,X^3\)` and `\(h(x, \xi_1)\)`, `\(h(x, \xi_2)\)`, `\(\ldots\)`, `\(h(x, \xi_K)\)` as predictors. - A cubic spline with `\(K\)` knots will use `\(4+K\)` degrees of freedom (number of parameters to estimate) - For example, for the wage regression on age with knots at 25, 40 , 60 `$$\begin{array}{ll} wage & = & \beta_0 + \beta_1 age + \beta_2 age^2 + \beta_3 age^3 + \\ & & \beta_4 (age-25)_+^3 + \beta_5 (age-40)_+^3 + \beta_6 (age-60)_+^3 + \epsilon \end{array}$$` Since `\(K=3\)` there are 7 parameters to estimate. --- # Cubic Spline: an illustration .pull-left[ ![](img/nonlin8.PNG) ] .pull-right[ - Cubic spline basis (orange) is continuous at the knot=0.6. - Global function now smoothly changes shape at the knot. ] - A cubic spline with a single knot at 0.6 `$$f(x) = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3+\beta_4 (x-0.6)_+^3$$` --- name: natural # Natural Splines .pull-left[ ![](img/nonlin9.PNG) ] .pull-right[ - Splines tend to have high variability at the very small and very large values of X - Note high variations in the figure where confidence bands at the extreme values are very large. - A natural cubic spline extrapolates linearly beyond the boundary knots. ] - Boundary constraints: function is forced to be linear for age<25 and age>60 - This adds 4 = 2x2 extra constraints, and allows us to put more internal knots for the same degrees of freedom as a regular cubic spline. - Note that the confidence band for the natural cubic spline is narrower. --- # Choosing the number and location of knots - In practice we need to answer: How many knots should we use? Where should we place the knots? - We can place more knots where the function changes more rapidly and fewer knots where the function seems to be more stable. - Another option, which is applied widely, is to set the number of knots and then let the software place the knots uniformly using quantiles of X (e.g. 25th, 50th, and 75th percentiles). - We can use cross-validation methods to choose the number of knots. --- # Comparing Polynomial Regression and Splines .pull-left[ ![](img/nonlin10.PNG) ] .pull-right[ - Degree-15 polynomial (blue) vs Natural cubic spline (red) - Polynomial regression tends to have wild boundary behavior - In contrast splines are more stable compared to high degree polynomials. - Splines introduce flexibility by increasing the number of knots but keeping the degree fixed. ] <small>(Source: ISLR Fig. 7.7, p.277)<small> --- name: smoothing # Smoothing Splines - In regression splines, we basically defined a set of basis functions and applied OLS to fit the model. - A related approach tries to find a function `\(g(x)\)` that fits the data well. That is it minimizes `\(RSS = \sum_{i=1}^{n}\left(y_{i}-g\left(x_{i}\right)\right)^{2}\)` - But if don't put some restrictions on this, we may end up with severely over-fitting the data. We certainly don't want that. We want the fitted function to be smooth. - One way to achieve this is to consider a penalty term: `$$\sum_{i=1}^{n}\left(y_{i}-g\left(x_{i}\right)\right)^{2}+\lambda \int g^{\prime \prime}(t)^{2} d t$$` where `\(\lambda\)` is a nonnegative tuning parameter and `\(g^{\prime \prime}(t)\)` is the second derivative of `\(g(x)\)` --- # Smoothing Splines - Notice that the problem has the familiar Loss+Penalty structure: `$$\underbrace{\sum_{i=1}^{n}\left(y_{i}-g\left(x_{i}\right)\right)^{2}}_{Loss}+ \underbrace{\lambda \int g^{\prime \prime}(t)^{2} d t}_{Penalty}$$` - Recall that the first derivative, `\(g^{\prime}(t)\)`, measures the slope of a function at `\(t\)`. - The second derivative, `\(g^{\prime \prime}(t)\)`, shows how much slope changes at `\(t\)`. - The second derivative is a measure of roughness, it is large if the function is changing rapidly near `\(t\)` and it is close to zero otherwise. - By taking the integral of the squared second derivative, `\(\int g^{\prime \prime}(t)^{2} d t\)` measures the total change in the function `\(g^{\prime}(t)\)` over the entire range. --- # Smoothing Splines - Notice that the problem has the familiar Loss+Penalty structure: `$$\underbrace{\sum_{i=1}^{n}\left(y_{i}-g\left(x_{i}\right)\right)^{2}}_{Loss}+ \underbrace{\lambda \int g^{\prime \prime}(t)^{2} d t}_{Penalty}$$` - `\(\int g^{\prime \prime}(t)^{2} d t\)` will take on a small value if `\(g()\)` is very smooth and `\(g^{\prime}(t)\)` will be close to constant. - On the other hand if `\(g()\)` is wiggly and jumpy then `\(g^{\prime}(t)\)` will vary significantly and `\(\int g^{\prime \prime}(t)^{2} d t\)` will take on a large value. - The penalty term, `\(\lambda \int g^{\prime \prime}(t)^{2} d t\)`, pushes the function to be smooth. The larger the `\(\lambda\)` parameter, the smoother the function. - When `\(\lambda=0\)`, the penalty term has no effect so perfect interpolation will result. - When `\(\lambda \rightarrow \infty\)`, `\(g()\)` will be perfectly smooth and linear in the limit. --- # Choosing the smoothing parameter - As `\(\lambda\)` increases, the function becomes linear (i.e., least squares line). (second derivative of a linear function is zero) - For an intermediate value of `\(\lambda\)`, the function `\(g(\cdot)\)` will smoothly approximate the training observations. - The tuning parameter `\(\lambda\)` controls the roughness of the smoothing spline and hence the effective degrees of freedom. - As `\(\lambda\)` increases from 0 to `\(\infty\)` the effective degrees of freedom decreases from `\(n\)` to 2. - Smoothing splines avoid the knot selection issue, instead we must choose `\(\lambda\)` (or the effective degrees of freedom). - We can use Cross-validation or LOOCV in order to choose the tuning parameter. --- # Smoothing spline example ![](img/nonlin11.PNG) <small>(Source: ISLR Fig. 7.8, p.280)<small> --- name: local # Local Regression - Local regression involves computing the fit at a target point `\(x_0\)` using only the nearby training observations. - Using a weight function, we put more weights on the observations around the target point. - The weight function is sometimes called the kernel function: `$$K_{i 0}=\frac{1}{h} k\left(\frac{x_{i}-x_{0}}{h}\right)$$` - Local regression minimizes the following objective function `$$\sum_{i=1}^{n} K_{i 0}\left(y_{i}-\beta_{0}-\beta_{1}\left(x_{i}-x_{0}\right)\right)^{2}$$` --- # Local Regression Illustrated ![](img/nonlin12.PNG) <small>(Orange colored points are local to the target value represented by orange vertical line. Blue curve is the actual function, light orange is the local regression fit. Source: ISLR Fig. 7.9, p.281)<small> --- # Local regression - To perform local regression in practice, we need to decide the fraction of observations to use, say `\(s=k/n\)`, around the target point ( `\(k\)` is the number of obs. within the local window). This is called the span `\(s\)`. - The role of span is similar to the tuning parameter `\(\lambda\)` in smoothing splines. It controls the flexibility of non-linear fit. - The smaller the value of `\(s\)`, the more wiggly the function. - Conversely, a large value of `\(s\)` will result in fitting a regression line using all observations. - The span `\(s\)` can be specified directly or chosen by cross-validation. - The next graph displays Local regression of wage on age using two values for the span, `\(s=0.2\)` and `\(s=0.7\)` - The fit with the higher span is smoother as expected. --- class: my-small-font # Local Regression: Example ![](img/nonlin13.PNG) <small>(Source: ISLR Fig. 7.10, p.283)<small> --- name: gams # Generalized Additive Models (GAMs) - GAMs allow modeling nonlinear relationships in multiple variables in a flexible way while keeping the additive structure of the model: `$$\begin{aligned} y_{i} &=\beta_{0}+\sum_{j=1}^{p} f_{j}\left(x_{i j}\right)+\epsilon_{i} \\ &=\beta_{0}+f_{1}\left(x_{i 1}\right)+f_{2}\left(x_{i 2}\right)+\cdots+f_{p}\left(x_{i p}\right)+\epsilon_{i} \end{aligned}$$` where `\(f_{j}\left(x_{i j}\right)\)` is a smooth non-linear function. - The model is **additive** because we calculate separate nonlinear functions for our predictors and then add them up. For example, `$$\text { wage }=\beta_{0}+f_{1}(\text { year })+f_{2}(\text { age })+f_{3}(\text { education })+\epsilon$$` (see the next graph) --- # GAMs: Example ![](img/nonlin14.PNG) <small>Natural splines in year and age with 4 and 5 degrees of freedom, respectively, and a step function in the qualitative variable education (Source: ISLR Fig. 7.11, p.284)<small> --- # GAMs: Example ![](img/nonlin15.PNG) <small>Smoothing splines in year and age with 4 and 5 degrees of freedom, respectively, and a step function in the qualitative variable education (Source: ISLR Fig. 7.12, p.285)<small> --- class: my-medium-font # GAMS for Classification - GAMs can also be used when the response variable is qualitative. - The logit function is linear in predictors: `$$\log \left(\frac{p(X)}{1-p(X)}\right)=\beta_{0}+\beta_{1} X_{1}+\beta_{2} X_{2}+\cdots+\beta_{p} X_{p}$$` - Using GAM the logit model can be written as `$$\log \left(\frac{p(X)}{1-p(X)}\right)=\beta_{0}+f_{1}\left(X_{1}\right)+f_{2}\left(X_{2}\right)+\cdots+f_{p}\left(X_{p}\right)$$` - For example `$$\log \left(\frac{p(X)}{1-p(X)}\right)=\beta_{0}+\beta_{1} \times \text { year }+f_{2}(\text { age })+f_{3}(\text { education })$$` where `\(p(X)=\operatorname{Pr}(\text { wage }>250 \mid \text { year, age }, \text { education })\)` --- # Classification GAMs: Example I(wage>250) ![](img/nonlin16.PNG) <small> f1 is linear in year, f2 is a smoothing spline with 5 degrees of freedom, and f3 is a step function in education. (Source: ISLR Fig. 7.13, p.287)<small>


THE GIRL ON THE BRIDGE A film review by Steve Rhodes Copyright 2000 Steve Rhodes RATING (0 TO ****): ** 1/2
So you're looking for an erotic knife-throwing movie? I thought so.
But, had you considered one about a knife thrower who is willing to do it blind in order to pump up the crowd? If so, then Patrice Leconte's THE GIRL ON THE BRIDGE (LA FILLE SUR LE PONT) is just the picture for you.
Chief among its delights are Serge Frydman's sharp and witty script and the sexually charged performance by Vanessa Paradis as the film's eponymous lead, Adele. With pixyish good looks and large gaps between her teeth, Paradis appears less like a movie star than every man's wish for his girl next door.
Shot in an oversaturated black and white that manages to obscure the subtitles of some of the best sections of dialog, the movie is a fairy tale for adults. The depressed damsel in distress is about to jump off of a bridge until she is rescued by a mysterious stranger, Gabor (Daniel Auteuil).
Adele explains that she is like flypaper in that she picks up all of the crud.
 No, he is not a pimp. He is a middle-age knife thrower. "Past the age of 40, knife throwing becomes sporadic," he explains to Adele, hence he tries to find women to work with him who have nothing to lose. And being on the verge of suicide is certainly proof of a lack of belief in the future.
Adele admits that she has a weak right eye, which Gabor sees as an attribute. "Eyes don't matter," he tells her. "See less. Fear less."
What she does see is men, all except Gabor that is. "Boys attract me like beautiful clothes," she confesses. "I always want to try them on." Since her first gig is at a circus, this presents a wonderful opportunity to try out sex with none other than a professional contortionist.
The movie is never better than when she's strapped to the board, withering in ecstasy and moaning with every knife strike. As magical, exotic and erotic as these scenes are, they are too few and far between. The movie begins to end fairly predictably and then has the bad sense of timing not to stop but to keep going for yet another, equally predictable ending. THE GIRL ON THE BRIDGE, which is already a reasonably short film, would be improved dramatically by being much shorter still. Fairy tales work best when they don't overstay their welcome.
THE GIRL ON THE BRIDGE runs 1:32. It is in French with English subtitles. It is rated R for some sexuality and would be acceptable for most teenagers.
Email<EMAIL_ADDRESS>Web: http://www.InternetReviews.com
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Every new Datalog interpreter should be tested with a program that contains an atom like p(X, X, Y) where a variable occurs more than once, thus having to match an earlier occurrence of itself.
Programbuys("a", "a", "noodles"). buys("a", "b", "watch"). buys("b", "a", "socks"). buys("b", "c", "gloves"). buys("c", "c", "suit"). selfBuys(X):- buys(X, X, Something). likes(X, Y):- buys(X, Y, Something). selfLikes(X):- likes(X, X).Query
selfLikes(X)
Hint: Hovering over a graph expands it
{{trace_is_here}}


C:\xampp\htdocs\laravel
/
vendor
/
laravel
/
framework
/
src
/
Illuminate
/
Database
/
Query
/
Grammars
(Dashboard)
Class Coverage Distribution
Class Complexity
Top Project Risks
Grammar
(2213)
MySqlGrammar
(68)
Least Tested Methods
Grammar::compileOrders
(0%)
Grammar::compileLimit
(0%)
Grammar::compileOffset
(0%)
Grammar::compileBasicHaving
(0%)
Grammar::compileHaving
(0%)
Grammar::whereRaw
(0%)
Grammar::compileGroups
(0%)
Grammar::compileHavings
(0%)
Grammar::compileUnions
(0%)
Grammar::compileUnion
(0%)


{% extends "server/base.html" %} {% block title %}Your Progress{% endblock %} {% block content %}
{{ player.team.name }}
The game hasn't started yet! Have some patience!


| 047:013 Khan | : |
And many a town, stronger than your town (Makkah) (O Muhammad SAW) which has driven you out We have destroyed. And there was none to help them. |
| 047:013 Maulana | : |
And how many a town, more powerful than thy town which has driven thee out -- We destroyed them, so there was no helper for them. |
| 047:013 Pickthal | : |
And how many a township stronger than thy township (O Muhammad) which hath cast thee out, have We destroyed, and they had no helper! |
| 047:013 Rashad | : |
Many a community was much stronger than the community that evicted you from your town; when we annihilated them, no one could help them. |
| 047:013 Sarwar | : |
(Muhammad), how many towns, much more powerful than the one from which you have been expelled, have We destroyed and left helpless?. |
| 047:013 Shakir | : |
And how many a town which was far more powerful than the town of yours which has driven you out: We destroyed them so there was no helper for them. |
| 047:013 Sherali | : |
And how many a township, mightier than thy town which has driven thee out, have WE destroyed, and they had no helper. |
| 047:013 Yusufali | : |
And how many cities, with more power than thy city which has driven thee out, have We destroyed (for their sins)? and there was none to aid them. |
Concepts/Themes Covered: أركان الإسلام > التوحيد > توحيد الله تعالي > دعوة من لا يقر بالوحدانية إلى الاعتبار بمن سبقهم Pillars of Islam> Islamic> Oneness of Allah>A Call to those who do not Believe in Monotheism to Learn from their Antecedents أركان الإسلام > (محمد (صلى الله عليه وسلم > هجرته ومنزلة المهاجرين Pillars of Islam> The Blessed Muhammad(PBUH)> His Immigration and the Immigrants Rank القصص والتاريخ > العبر التاريخية فى أنباء القرى The Stories and the History> The Historic Examples of Towns' News
This tool uses eight English parallel translations for the Holy Quran imported from Mr. Clay Smith. You can also visit Qurany Concepts for browsing the Quran themes or topics. If you want to search the Holy Quran for keywords, please visit Qurany Keywords.


StatapMY is a showcase of inspiring and ambitious local Malaysian startup companies that have been successfully launched their product to the masses. Whether it is just an app or web service, StatapMY will document them all in this little site. It's built and curated by @nurfaizfoat.
Local startups scene is at its best now. So, we accept all submission as long as it follows the instruction on How to Submit section below.
To submit a site suggestion, open an issue or create a pull request. Pull requests will be given higher priority since they are easier to include. Make sure the screenshot is 1000x800 and please double check that everything looks good before submitting. It’s also a good idea to run the screenshot through an image optimizer like TinyPNG before including it. This will help keep the website fast and the repository small as possible.
This site is hugely inspired by Bootstrap Expo, Built with Ember and Beautiful Open. It's built with Jekyll, developed on GitHub, and yes it is hosted on GitHub Pages.


It will help to have some prior knowledge of HTML, CSS, JavaScript, and writing Arduino sketches. We'll cover building the web page, writing the client and server scripts, and writing and downloading the Arduino sketch in the following sections. You can also download the example from GitHub
To build and run this example you will need the following:
Arduino UNO
Breadboard
USB A-B Cable
5mm LED
220 Ω Resistor
Jumper Wires
We'll cover installing the software in the next section.


User
พนักงานในระบบ
| User | Name | Target | Project | WIN | Lost | Forecast | LastUpdate |
|---|---|---|---|---|---|---|---|
| Herman Beck | 100,000 ฿ |
|
|
|
|
||
| Messsy Adam | 100,000 ฿ |
|
|
|
|
||
| John Richards | 100,000 ฿ |
|
|
|
|
||
| Peter Meggik | 100,000 ฿ |
|
|
|
|
||
| Summary | 100,000,000 ฿ |
|
|
|
|
|
| Column 1 | Column 2 |
|---|---|
| Row 1 Data 1 | Row 1 Data 2 |
| Row 2 Data 1 | Row 2 Data 2 |
Data table
| Order # | Purchased On | Customer | Ship to | Base Price | Purchased Price | Status | Actions |
|---|---|---|---|---|---|---|---|
| 1 | 2012/08/03 | Edinburgh | New York | $1500 | $3200 | ||
| 2 | 2015/04/01 | Doe | Brazil | $4500 | $7500 | ||
| 3 | 2010/11/21 | Sam | Tokyo | $2100 | $6300 | ||
| 4 | 2016/01/12 | Sam | Tokyo | $2100 | $6300 | ||
| 5 | 2017/12/28 | Sam | Tokyo | $2100 | $6300 | ||
| 6 | 2000/10/30 | Sam | Tokyo | $2100 | $6300 | ||
| 7 | 2011/03/11 | Cris | Tokyo | $2100 | $6300 | ||
| 8 | 2015/06/25 | Tim | Italy | $6300 | $2100 | ||
| 9 | 2016/11/12 | John | Tokyo | $2100 | $6300 | ||
| 10 | 2003/12/26 | Tom | Germany | $1100 | $2300 |


What Will Learn?
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry’s standard dummy tn an unknodard dummy text ever since the 1500s, when an unknowwn printer took a galley of type and scrambled it to make a type shas been the industry’s standard dummy text ever since the 1500s, look like readable English when an un loream ipsum.
Requirements
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry’s standard dummy tex an unknown printer took. Lorem Ipsum is simply dummy text of the printing.
Description
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry’s standard dummy text ever since the 1500s, when an un it to make a type specimen book. It has survived not only fivnged.Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry’s standard dummy text ever since the 1500s, when an un it to make a type specimen book. It has survived not only fivnged.
Getting Started with Uniaro
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an un it to make a type specimen book. It has survived not only fivnged.
5.00 average based on 2 ratings
Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature College in Virginia ?
It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum.


Susan Granger's review of "THE SIEGE" (20th Century-Fox)
In this instance, the whole is less than the sum of its parts. Denzel Washington is a superb actor, an Oscar winner for "Glory" and nominee for "Cry Freedom" and "Malcolm X," as is Annette Bening, an Oscar winner for "The Grifters," and Bruce Willis has proven his ability in thrillers such as "The Fifth Element" and "Twelve Monkeys" and faced disaster in the "Die Hard" trilogy. Director Edward Zwick has not only helmed "Courage Under Fire" and "Glory" but he's also the co-creator of TV's "thirtysomething" and "My So-Called Life." With credentials like these, one expects more than a tepid political potboiler. Based on a story by Lawrence Wright, the convoluted plot revolves around an FBI agent (Denzel Washington), a United States Army General (Bruce Willis), and a burnt-out, murky-minded CIA agent (Annette Bening) who team up to find an underground cell of Islamic terrorists who are causing chaos in New York City, blowing up a bus in Brooklyn and threatening children. In a panic, the President imposes martial law and suspends citizens' Constitutional rights, allowing the Army and the National Guard to supplant local law enforcement for the duration of the operation, thereby creating a police state. The implications deepen when all people of Arabian descent are considered suspicious and rounded up, like the Japanese-Americans during World War II, posing questions like: Does one have to become monstrous to fight a monster? How far do you go to protect freedom? While it all reads well, it doesn't play out on the screen. On the Granger Movie Gauge of 1 to 10, "The Siege" is a formulaic, forgettable 4. It's not dramatic enough to deserve merit, not suspenseful enough to attract an action-oriented audience, and so preachy as to be almost ludicrous.
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


B - Arizona Department of Corrections
BACA, LAWRENCE, 011424, 04/23/1939. BACA, LEWIS, 001282 . 022954, 08/ 09/1961. BALLESTEROS, HECTOR, NAVARRO, 025253, 04/08/1964, 022954 .
http://www.azcorrections.gov/Historical/100B.aspx
Treatment for Herniated or Bulging Discs
Gloria M. Navarro - Judgepedia
Jul 21, 2012 . Navarro received her B.A. in 1989 from the University of Nevada, Las Vegas and her law degree from Arizona State University in 1992.
http://judgepedia.org/index.php/Gloria_M._Navarro
City of Tempe, AZ : Calendar of Events & Meetings : Larry Campbell ...
City of Tempe, AZ Home · Home . The 11th Annual Larry Campbell Tree of Lights will sparkle in the museum lobby again this holiday season. Presented by .
http://www.tempe.gov/index.aspx?page=20&recordid=5985
larry navarro in az Herniated or Bulging Discs
Official Football Roster - Arizona State University Official Athletic Site
Larry Porter - Assistant Coach (Running Backs). Paul Randolph - Senior . Arizona State (ASU) Arizona State Womens Football Replica Tee-Maroon. $34.99.
http://www.thesundevils.com/sports/m-footbl/mtt/asu-m-footbl-mtt.html
SLM Certified | Arizona Landscape Contractors Association
Arizona Landscape Contractors' Association . Scottsdale, AZ 85254 . Munguia, Jr, David; Nagy, James; Navarro, Larry; Peralez, Frank; Pioche, Tyrone; Porras, .
http://www.azlca.com/slm-certified
Unlicensed Contractor Violations - Arizona Registrar of Contractors
Arizona Registrar of Contractors Unlicensed Contractor Violations. . 06:48 AM Contact FAQ Site Map. All Gov of AZ ROC . Phoenix, AZ 85007 . MC COID LARRY, Tucson, 06/13/2011 . RENTERIA-NAVARRO JOSE, Peoria, 04/03/2012 .
http://www.azroc.gov/wanted/violations.html
Official Impairment Listing for Spinal Disorders
San Francisco vs. Arizona: 3 Keys to a Cardinals Victory over 49ers ...
Oct 26, 2012 . This week's three keys to an Arizona Cardinals victory are brought to . double- teaming Larry Fitzgerald and leaving the other receivers with one-on-one chances. . Patrick Willis and Navarro Bowman are likely the best inside .
http://bleacherreport.com/articles/1385126-san-francisco-vs-arizona-3-keys-to-a-cardinals-victory-over-49ers-on-mnf
- embassy suite wash dc (narrowing of the spine)
The Story of Jack Frost - iTunes - Apple
Oct 2, 2012 . Get a free sample or buy The Story of Jack Frost by Farrah McDoogle & Larry Navarro on the iTunes Store. You can read this book with iBooks .
https://itunes.apple.com/us/book/the-story-of-jack-frost/id506095303?mt=11Arizona Cardinals: William Powell
Latest News for William Powell, Bio, Stats, Injury Reports, Photos, Video Highlights, and Game Logs for Arizona Cardinals Running Back William Powell.
http://www.azcardinals.com/team/roster/William-Powell/c35d1b65-dad0-4204-8326-071efa0d8a97
Arizona Department of Corrections
CAMARENA, JOHN ALD, AZ, 017231, 04/24/1953 . CAMPER, LAWRENCE, 026557, 11/20/1965 . CARDENAS, JUSUS, NAVARRO, 022475, 01/27/1961 .
http://www.azcorrections.gov/Historical/100C.aspx
CNBC's Kudlow Report: Politics, Business, Market Analysis - CNBC
1 day ago . Larry Kudlow is guessing that stocks are correctly sniffing that there will be no . Jon Kyl, (R-AZ) weighs in on where "fiscal cliff" talks are likely headed. . Peter Navarro, University Of California- Irvine professor, and Ann Lee, .
http://www.cnbc.com/id/15838446/CNBCrsquos_Kudlow_Report_Politics_Business_Market_Analysis__CNBC
Medical-Vocational Allowance for Herniated Disc
The Story of Jack Frost - iTunes - Apple
Get a free sample or buy The Story of Jack Frost by Farrah McDoogle & Larry Navarro on the iTunes Store. You can read this book with iBooks on your iPhone, .
http://itunes.apple.com/us/book/isbn9781442453050
Laci Navarro - Surprise, Centennial High School | Facebook
Laci Navarro studied at Centennial High School, lives in Surprise, Arizona and is from Peoria, Arizona. Join Facebook to connect with Laci Navarro and others .
http://www.facebook.com/Laci48821
Operation Rx Closes Yavapai County Street Pharmacies
Jul 20, 2012 . The Arizona investigation, dubbed Operation Rx, began earlier in 2012 and involved hundreds of hours of teamwork by . Larry Navarro says: .
http://sedonaeye.com/operation-rx-closes-yavapai-county-street-pharmacies
Appealing a Denial of Benefits
Gilbert Navarro II | LinkedIn
Gilbert Navarro II. Experienced in Customer Service and Managerial Processes. Location: San Manuel, Arizona (Phoenix, Arizona Area); Industry: Consumer .
http://www.linkedin.com/pub/gilbert-navarro-ii/36/838/808
MMA Fighters A-Z - UFC fighters and more - MMA - ESPN
Cummings, Zak · Cummins, Patrick · Cummo, Luke · Cunningham, Adam · Cureton, Larry · Curio, Vagner · Curnut, Greg · Curran, Jeff · Curran, Josh · Curran , Pat .
http://espn.go.com/mma/fighters


Overview of getdns
getdns is an implementation of a modern asynchronous DNS API; the specification was originally edited by Paul Hoffman. It is intended to make all types of DNS information easily available to application developers and non-DNS experts.
Why you might want getdns
Traditional access to DNS data from applications has several limitations:
- APIs require applications to have considerable sophistication about DNS data and data types
- Some kinds of data about the response (notably, the resource record set time to live) is not exposed via any API, so applications need to process raw protocol responses to get such data
- APIs are often blocking, meaning asynchronous access is not possible without some work
- Sophisticated uses of the DNS (things like IDNA and DNSSEC validation) require considerable application work, possibly by application developers with little experience with the vagaries of DNS.
getdns also provides an experimental DNS Privacy enabled client called 'stubby' - see below for more details.
Motivation for providing the API
The developers are of the opinion that DNSSEC offers a unique global infrastructure for establishing and enhancing cryptographic trust relations. With the development of this API we intend to offer application developers a modern and flexible interface that enables end-to-end trust in the DNS architecture, and which will inspire application developers to implement innovative security solutions in their applications.
API Documentation
Note that this implementation offers additional functionality to supplement that in the official getdns API. Some additions are convenient utility functions but other functionality is experimental prior to be being recommended for inclusion in the official API. The Doxygen documentation provides the details of the full API for this implementation.
License
This implementation is licensed under the New BSD License (BSD-new).
Obtaining and getting started with getdns
The project home page at getdnsapi.net provides documentation, binary downloads, and news regarding the getdns API implementation. This README file captures the goals and direction of the project and the current state of the implementation.
If you are just getting started with the library take a look at the section below that describes building and handling external dependencies for the library.
Examples
Once it is built you should take a look at src/examples to see how the library is used.
Download
Download the sources from our github repo or from getdnsapi.net and verify the download using the checksums (SHA1 or MD5) or using gpg to verify the signature. Our keys are available from the pgp keyservers
- wille, key id E5F8F8212F77A498 m@nl netla bs.n l
Releases
Release numbering follows the Semantic Versioning approach. The code is currently under active development.
The following requirements were met as conditions for the present release:
- code compiles cleanly on at least the primary target platforms: OSX, RHEL/CentOS Linux, FreeBSD
- examples must compile and run cleanly
- there must be clear documentation of supported and unsupported elements of the API
Building and External Dependencies
If you are installing from packages, you have to install the library and also the library-devel (or -dev) for your package management system to get the the necessary compile time files.
External dependencies are linked outside the getdns API build tree (we rely on configure to find them). We would like to keep the dependency tree short. Please refer to section for building on Windows for separate dependency and build instructions for that platform.
- libunbound from NLnet Labs version 1.4.16 or later.
- libidn from the FSF version 1. (Note that the libidn version means the conversions between A-labels and U-labels may permit conversion of formally invalid labels under IDNA2008.)
- libssl and libcrypto from the OpenSSL Project version 0.9.7 or later. (Note: version 1.0.1 or later is required for TLS support, version 1.0.2 or later is required for TLS hostname authentication)
- Doxygen is used to generate documentation; while this is not technically necessary for the build it makes things a lot more pleasant.
For example, to build on a recent version of Ubuntu, you would need the following packages:
# apt install build-essential libunbound-dev libidn11-dev libssl-dev libtool m4 autoconf
If you are building from git, you need to do the following before building:
# git submodule update --init # libtoolize -ci # (use glibtoolize for OS X, libtool is installed as glibtool to avoid name conflict on OS X) # autoreconf -fi
As well as building the getdns library two other tools may be installed:
- getdns_query: a command line test script wrapper for getdns
- stubby: an experimental DNS Privacy enabled client
Note: If you only want to build stubby, then use the --with-stubby
option when running 'configure'.
Minimizing dependencies
- getdns can be configured for stub resolution mode only with the
--enable-stub-only
option to configure. This removes the dependency onlibunbound
. - Currently getdns only offers two helper functions to deal with IDN:
getdns_convert_ulabel_to_alabel
andgetdns_convert_alabel_to_ulabel
. If you do not need these functions, getdns can be configured to compile without them with the--without-libidn
option to configure. - When both
--enable-stub-only
and--without-libidn
options are used, getdns has only one dependency left, which is OpenSSL.
Extensions and Event loop dependencies
The implementation works with a variety of event loops, each built as a separate shared library. See this Doxygen page and this man page for more details.
- libevent. Note: the examples require this and should work with either libevent 1.x or 2.x. 2.x is preferred.
- libuv
- libev
Stubby
- Stubby is an experimental implementation of a DNS Privacy enabled stub resolver than encrypts DNS queries using TLS. It is currently suitable for advanced/technical users - all feedback is welcome!
- Details on how to use Stubby can be found in the Stubby Reference Guide.
- Also see dnsprivacy.org for more information on DNS Privacy.
Regression Tests
A suite of regression tests are included with the library, if you make changes or just want to sanity check things on your system take a look at src/test. You will need to install libcheck. The check library is also available from many of the package repositories for the more popular operating systems.
DNSSEC dependencies
For the library to be DNSSEC capable, it needs to know the root trust anchor. The library will try to load the root trust anchor from /etc/unbound/getdns-root.key
by default. This file is expected to have one or more DS
or DNSKEY
resource records in presentation (i.e. zone file) format. Note that this is different than the format of BIND.keys.
$ Zero configuration DNSSEC
When the root trust anchor is not installed in the default location and a DNSSEC query is done, getdns will try to use the trust anchors published here: http://data.iana.org/root-anchors/root-anchors.xml . It will validate these anchors with the ICANN Certificate Authority certificate following the procedure described in [RFC7958]. The root-anchors.xml
and root-anchors.p7s
S/MIME signature will be cached in the $HOME/.getdns
directory.
When using trust-anchors from the root-anchors.xml
file, getdns will track the keys in the root DNSKEY rrset and store a copy in $HOME/.getdns/root.key. Only when the KSK DNSKEY's change, a new version of root-anchors.xml
is tried to be retrieved from data.iana.org.
A installed trust-anchor from the default location (/etc/unbound/getdns-root.key
) that fails to validate the root DNSKEY RRset, will also trigger the "Zero configuration DNSSEC" procedure described above.
Support
Mailing lists
We have a getdns users list for this implementation.
The getdns-api mailing list is a good place to engage in discussions regarding the design of the API.
Tickets and Bug Reports
Tickets and bug reports should be reported via the GitHub issues list.
Features of this release
Goals
The goals of this implementation of the getdns API are:
- Provide an open source implementation, in C, of the formally described getdns API by getdns API team at https://getdnsapi.net/spec.html
- Support FreeBSD, OSX, Linux (CentOS/RHEL, Ubuntu) via functional "configure" script
- Support Windows 8.1
- Include examples and tests as part of the build
- Document code using doxygen
- Leverage github as much as possible for project coordination
- Follow the BSD coding style/standards ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/share/misc/style
Non-goals (things we will not be doing at least initially) include:
- implementation of the traditional DNS related routines (gethostbyname, etc.)
Language Bindings
In parallel, the team is actively developing bindings for various languages. For more information, visit this webpage.
Unsupported getDNS Features
The following API calls are documented in getDNS but not supported by the implementation at this time:
- Detecting changes to resolv.conf and hosts
- MDNS, NIS and NetBIOS namespaces (only DNS and LOCALFILES are supported)
Minor omissions
The following minor implementation omissions are noted:
Recursive mode does not support:
- TLS as a transport
- Non-zero connection idle timeouts or query pipelining
- Anything other than query_type and resolution_type in the return_call_reporting extension
Stub mode does not support:
- Non zero idle timeouts for synchronous calls
Known Issues
- None
Supported Platforms
The primary platforms targeted are Linux and FreeBSD, other platform are supported as we get time. The names listed here are intended to help ensure that we catch platform specific breakage, not to limit the work that folks are doing.
- RHEL/CentOS 6.4
- OSX 10.8
- Ubuntu 16.04
- Microsoft Windows 8.1
We intend to add Android and other platforms to future releases as we have time to port it.
Platform Specific Build Reports
FreeBSD
If you're using FreeBSD, you may install getdns via the ports tree by running: cd /usr/ports/dns/getdns && make install clean
If you are using FreeBSD 10 getdns can be intalled via 'pkg install getdns'.
CentOS and RHEL 6.5
We rely on the most excellent package manager fpm to build the linux packages, which means that the packaging platform requires ruby 2.1.0. There are other ways to build the packages; this is simply the one we chose to use.
# cat /etc/redhat-release CentOS release 6.5 (Final) # uname -a Linux host-10-1-1-6 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux # cd getdns-0.2.0rc1 # ./configure --prefix=/home/deploy/build # make; make install # cd /home/deploy/build # mv lib lib64 # . /usr/local/rvm/config/alias # fpm -x "*.la" -a native -s dir -t rpm -n getdns -v 0.2.0rc1 -d "unbound" -d "libevent" -d "libidn" --prefix /usr --vendor "Verisign Inc., NLnet Labs" --license "BSD New" --url "https://getdnsapi.net" --description "Modern asynchronous API to the DNS" .
OSX
# sw_vers ProductName: Mac OS X ProductVersion: 10.8.5 BuildVersion: 12F45 Built using PackageMaker, libevent2. # ./configure --with-libevent --prefix=$HOME/getdnsosx/export # make # make install edit/fix hardcoded paths in lib/*.la to reference /usr/local update getdns.pmdoc to match release info build package using PackageMaker create dmg A self-compiled version of OpenSSL or the version installed via Homebrew is required. Note: If using a self-compiled version, manual configuration of certificates into /usr/local/etc/openssl/certs is required for TLS authentication to work.
Homebrew
If you're using Homebrew, you may run brew install getdns
. By default, this will only build the core library without any 3rd party event loop support.
To install the event loop integration libraries that enable support for libevent, libuv, and libev, run: brew install getdns --with-libevent --with-libuv --with-libev
. All switches are optional.
Note that in order to compile the examples, the --with-libevent
switch is required.
Additionally, the OpenSSL library installed by Homebrew is linked against. Note that the Homebrew OpenSSL installation clones the Keychain certificates to the default OpenSSL location so TLS certificate authentication should work out of the box.
Microsoft Windows 8.1
The build has been tested using the following: 32 bit only Mingw: Mingw(3.21.0) and Msys 1.0 on Windows 8.1 32 bit build on a 64 bit Mingw [Download latest from: http://mingw-w64.org/doku.php/download/mingw-builds and http://msys2.github.io/]. IMPORTANT: Install tested ONLY on the "x86_64" for 64-bit installer of msys2.
Dependencies
The following dependencies are
- openssl-1.0.2j
- libidn
Instructions to build openssl-1.0.2j: Open the mingw32_shell.bat from msys2 in order to build:
If necessary, install the following using pacman:
pacman -S pkg-config libtool automake pacman -S autoconf automake-wrapper tar -xvf openssl-1.0.2j.tar cd openssl-1.0.2j/ ./Configure --prefix=${LOCALDESTDIR} --openssldir=${LOCALDESTDIR}/etc/ssl --libdir=lib shared zlib-dynamic mingw make make install
To configure:
./configure --enable-stub-only --with-trust-anchor="c:\\\MinGW\\\msys\\\1.0\\\etc\\\unbound\\\getdns-root.key" --with-ssl=<location of openssl from above> --with-getdns_query
The trust anchor is also installed by unbound on c:\program Files (X86)\unbound\root.key
and can be referenced from there or anywhere else that the user chooses to configure it.
After configuring, do a make
and make install
to build getdns for Windows.
Example test queries:
./getdns_query.exe -s gmadkat.com A @<IP_ADDRESS> +return_call_reporting (UDP) ./getdns_query.exe -s gmadkat.com A @<IP_ADDRESS> -T +return_call_reporting (TCP) ./getdns_query.exe -s gmadkat.com A -l L @<IP_ADDRESS> +return_call_reporting (TLS without authentication) ./getdns_query.exe -s www.huque.com A +dnssec_return_status +return_call_reporting (DNSSEC)
Contributors
- Claus Assman
- Theogene Bucuti
- Andrew Cathrow, Verisign Labs
- Neil Cook
- Saúl Ibarra Corretgé
- Craig Despeaux, Verisign, Inc.
- John Dickinson, Sinodun
- Sara Dickinson, Sinodun
- Robert Edmonds
- Angelique Finan, Verisign, Inc.
- Simson Garfinkel
- Daniel Kahn Gillmor
- Neel Goyal, Verisign, Inc.
- Bryan Graham, Verisign, Inc.
- Robert Groenenberg
- Jim Hague, Sinodun
- Paul Hoffman
- Scott Hollenbeck, Verising, Inc.
- Christian Huitema
- Shumon Huque, Verisign Labs
- Jelte Janssen
- Guillem Jover
- Shane Kerr
- Anthony Kirby
- Olaf Kolkman, NLnet Labs
- Sanjay Mahurpawar, Verisign, Inc.
- Allison Mankin, Verisign, Inc. - Verisign Labs.
- Sai Mogali, Verisign, Inc.
- Linus Nordberg
- Benno Overeinder, NLnet Labs
- Joel Purra
- Tom Pusateri
- Prithvi Ranganath, Verisign, Inc.
- Hoda Rohani, NLnet Labs
- Rushi Shah, Verisign, Inc.
- Vinay Soni, Verisign, Inc.
- Melinda Shore, No Mountain Software LLC
- Bob Steagall, Verisign, Inc.
- Andrew Sullivan
- Ondřej Surý
- Willem Toorop, NLnet Labs
- Gowri Visweswaran, Verisign Labs
- Wouter Wijngaards, NLnet Labs
- Glen Wiley, Verisign, Inc.
- Paul Wouters
Acknowledgements
The development team explicitly acknowledges Paul Hoffman for his initiative and efforts to develop a consensus based DNS API. We would like to thank the participants of the mailing list for their contributions.


Queen Elizabeth scraps Prince Andrew’s lavish birthday bash
by Laura ItalianoQueen Elizabeth has scrapped plans for a lavish 60th-birthday bash for her son Prince Andrew.
Instead, she’ll mark the Feb. 19 occasion with a small family dinner, The Sunday Times of London reported.
The downsizing comes a day after she stripped Andrew of his royal duties and booted his staff from Buckingham Palace in the wake of his disastrous recent BBC interview on his friendship with pedophile financier Jeffrey Epstein.


BW -- Isolation Layer for Applications
Copyright © 1996-2013, Brian Bray.
Applications need to isolate themselves from their OS environment for long term viability and cross platform development. The BW project is an isolation layer to explore this concept in C++ application code.
The architecture of this effort is based on the concept of reverse dependency. A technology supplier typically defines an API to a library of useful application services. Naive use of this API then binds the application to this service provider by creating a dependency on both the API and the library. Concepts and definitions from the API tend to propagate throughout the application. When multiple APIs for different unrelated services are used, these definition collide and conflict within the application code. Namespaces mitigate this problem but are infrequently used.
For example, even very basic types, like integers, are frequently defined in API's. Application code can end up littered with INT32's, int32's, WINT's, VENDORINT's, SDWORD's etc. all refering to possibly different types and defined with a mix of typedef's and macro's in the same namespace. The problem is intense for complex types that require memory management, like strings.
This dependency is created when application code #includes the library headers that define the API. In a reverse dependency, library code #includes headers from the application. These application headers define the interface expected and used along with a factory interface that lets the library supply object(s) that comply with the interface. If the goal of the application developer is to avoid overdependence on a single vendor, then the APIs are defined using the fundamental types of the language as much as possible, and application significant types where needed.
In theory, each application would have a completely independent set of needs and thus define a totally unique interface. In practice, however, the needs of applications in a single domain are similar, exploit similar specific technologies, and library APIs are designed to make typical application development easy by meeting these common needs. Long established services frequently support standardized APIs and competition causes convergence of service features even if the API's are not identical. So applications want to use APIs from major service providers without becoming dependent on them.
So this creates a pragmatic opportunity for an isolation layer whose implementation(s) depend on major service provider libraries and application interfaces defined as subsets of language centric bindings of common standards. The same isolation layer could interface similar applications in a domain to a range of competing service providers.
BW is an exploration of what happens when this concept is tested.
The details of the C++ language centric bindings to the selected common standards are described in the superset API documentation here. Applications would use a subset of these definitions.
Some ideas for applications in my domain of interest are given here.
Legacy documentation links (i.e. these are obsolete):


Lessons Learned Updating to Swift 3
Recently I have been working on a project for my kids. Of course since it was a new project I decided to do it in the newest version of Swift. I wanted to make some simple games that could help my young ones learn. I love playgrounds and all that they enable us to do. I came across this project that @uraimo which has a running list of Awesome Swift Playgrounds. There is one for spriteKit, which I really wanted to make a game with but of course it was written in Swift 2.2 I was not about to let that hold me back so I went ahead and started trying to convert it to Swift 3. I went through all the changes that Xcode suggested and finally made it to a point where I had no errors. The issue was that the example did not perform the way it did previously. After going back and converting each function 1 by 1. I determined the function that was not being called.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
func didBeginContact(contact: SKPhysicsContact) {
// Executed if the ball makes contact with the block.
if contact.bodyA.categoryBitMask == CategoryBitMask.Ball && contact.bodyB.categoryBitMask == CategoryBitMask.Block {
let block = contact.bodyB.node as! SKSpriteNode
// Turn the block gray if it hasn't been hit yet.
if block.name == "Block" {
block.color = SKColor.darkGray
block.name = "HalfBlock"
}
// Remove the block from the scene if it has already been hit.
else {
block.removeFromParent()
}
}
}
Then the next step was to figure out what I am missing but when I would try and find something it was already outdated. Even Apple’s documentation was for objectiveC
. Then it hit me that this function was not that swifty
which is what allot of the changes for Swift 3 were all about. There is to much redundancy func didBeginContact(contact: SKPhysicsContact) {
and therein lies the solution. See Xcode suggested to add the perimeter contact:
and technically the method is valid so no errors were thrown. But this method is key for detecting collision between two objects and is the method that conforms to SKPhysicsContactDelegate
.
So since swift is aiming to be super readable if you take out the brackets it should read like a sentence of sorts. “did Begin Contact contact SKPhysicsContact” is a super redundant sentence in my book. Take out the first contact and all is well. I don’t know why Xcode did not remove it when it added the parameter label like it does in many other instances but that is neither here nor there.
So I corrected my mistake and everything works just as it should. Corrected Code :
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
func didBegin(_ contact: SKPhysicsContact) {
// Executed if the ball makes contact with the block.
if contact.bodyA.categoryBitMask == CategoryBitMask.Ball && contact.bodyB.categoryBitMask == CategoryBitMask.Block {
let block = contact.bodyB.node as! SKSpriteNode
// Turn the block gray if it hasn't been hit yet.
if block.name == "Block" {
block.color = SKColor.darkGray
block.name = "HalfBlock"
}
else {
block.removeFromParent()
}
}
}
Conclusion: When updating for Swift 3 don’t trust XCode (She will lie to you). Make sure you understand what is going on with the code and check your syntax. Allot changed but I think allot has really improved with the changes.
Now go out there and explore, change variables and learn about SpriteKit
more, if you would like to check out the playground you can get the updated Swift 3 version here


OVA for Flowplayer - Configuring Overlays
- Minimal setup
- Flash display mode only
- HTML 5 display mode only
- Illustrating the supported non-linear types
- Flash display mode (overlay)
- HTML 5 display mode (overlay)
- Positioning Non-Linear Ads
- Restricting Content Selection
- By Count
- By Size
- Declaring size properties once for all Ad Slots via "Ads.Overlays"
- Applying size rules per Ad Slot
- By Ad Type
- By Mime Type
- Restricting content selection based on mime type across all ad slots
- Restricting based on mime type per ad slot
- Restricting based on mime type per ad slot and display mode
- By Size and Ad Type
- Customising the Display Templates
- General HTML 5 Display Options
- Scaling Overlays
- Setting the Preferred Display Mode
- Customising the Close and Minimize Buttons
- Flash Display Mode
- HTML5 Display Mode
- General Settings
These examples use an OpenX installation that we are running at Open Video Ads. To gain more insight
into the zone/banner setup that we have in place for the examples, please read this short guide.
Back to main index
Copyright 2011 Open Video Ads


Sedan |
|||
|
A sedan or saloon is a passenger car in a three-box configuration with A, B and C-pillars and principal volumes articulated in separate compartments for engine, passenger and cargo. The passenger compartment features two rows of seats and adequate passenger space in the rear compartment for adult passengers. The cargo compartment is typically in the rear, with the exception of some rear-engined models, such as the Renault Dauphine, Tatra T613, Volkswagen Type 3 and Chevrolet Corvair. It is one of the most common car body styles. A battery electric sedan such as the Tesla Model S has no engine compartment, but a front cargo compartment and a rear compartment for cargo or optionally for additional passengers. |
|||
Seat Availability4-5 people |
Number of Doors5 |
IIHS Crashworthiness RatingGood |


ServiceComb Pack 0.4.0 Cluster : Implementation
Now you can run multiple Alpha server in High Availability (HA) mode which support to run the event scanning in the master node only. You can also turn off event table scanning for some nodes with parameter alpha.event.scanner.enabled=false
. However, transaction compensation will fail when the node that enabled the event table scan is crashed.
We implemented database-based distributed lock in version 0.4.0 version, event scanning only runs on the master node in the cluster, When the master node is down, other nodes in the cluster will elect a new master node.
Quick Starts
Enable cluster support by parameter alpha.cluster.master.enabled=true
.
 The variable nodeStatus is the node type(Master or Slave). Later, I will introduce how nodeStatus
is initialized.
@Bean
TxConsistentService txConsistentService(
@Value("${alpha.event.pollingInterval:500}") int eventPollingInterval,
@Value("${alpha.event.scanner.enabled:true}") boolean eventScannerEnabled,
ScheduledExecutorService scheduler,
TxEventRepository eventRepository,
CommandRepository commandRepository,
TxTimeoutRepository timeoutRepository,
OmegaCallback omegaCallback) {
if (eventScannerEnabled) {
new EventScanner(scheduler,
eventRepository, commandRepository, timeoutRepository,
omegaCallback, eventPollingInterval, nodeStatus).run();
LOG.info("Starting the EventScanner.");
}
TxConsistentService consistentService = new TxConsistentService(eventRepository);
return consistentService;
}
The pollEvents method of EventScanner.java
for scanning events, If the master node nodeStatus.isMaster()
returns true.
private void pollEvents() {
scheduler.scheduleWithFixedDelay(
() -> {
// only pull the events when working in the master mode
if(nodeStatus.isMaster()){
updateTimeoutStatus();
findTimeoutEvents();
abortTimeoutEvents();
saveUncompensatedEventsToCommands();
compensate();
updateCompensatedCommands();
deleteDuplicateSagaEndedEvents();
updateTransactionStatus();
}
},
0,
eventPollingInterval,
MILLISECONDS);
}
Construct NodeStatus in AlphaConfig.java
by the following to ensure that the event scan will work regardless of whether you have configured the alpha.cluster.master.enabled
parameter, Here you can see the node is a slave just after startup when cluster mode is enabled. Later, I will introduce how to switch to master.
@Bean
NodeStatus nodeStatus (){
if(masterEnabled){
return new NodeStatus(NodeStatus.TypeEnum.SLAVE);
}else{
return new NodeStatus(NodeStatus.TypeEnum.MASTER);
}
}
@Autowired
NodeStatus nodeStatus;
ClusterLockService.java
is in charge of node state switching, It periodically perform lock preemption and set the node as a master after successful preemption.
@Autowired
LockProvider lockProvider;
...
...
@Scheduled(cron = "0/1 * * * * ?")
public void masterCheck() {
if (applicationReady) {
this.locker = lockProvider.lock(this.getMasterLock());
if (this.locker.isPresent()) {
if (!this.locked) {
this.locked = true;
nodeStatus.setTypeEnum(NodeStatus.TypeEnum.MASTER);
LOG.info("Master Node");
}
//Keep locked
} else {
if (this.locked || !lockExecuted) {
locked = false;
nodeStatus.setTypeEnum(NodeStatus.TypeEnum.SLAVE);
LOG.info("Slave Node");
}
}
lockExecuted = true;
}
}
Lock architecture
As you can see from the previous description, in the masterCheck
method of ClusterLockService.java
, get a lock by this.locker = lockProvider.lock(this.getMasterLock());
and determine if the lock was successful.
LockProvider.java
is an interface. Currently, we provide an implementation based on JDBC. The package structure and class dependencies are as follows:
Dependencies are as follows
-
LockProvider.java
The interface defines the lock methods
-
LockProviderPersistence.java
The interface defines the following three persistence methods
- initLock Create a lock, try to lock and return whether the lock was successful
- updateLock Update the lock, lock it again and return it successfully (the interface design of the update lock is designed for non-long connection lock design, for example, for locking according to a fixed period)
- unLock
-
AbstractLockProvider.java
Implemented the
LockProvider.java
interface and called the internalLockProviderPersistence.java
interface for locking operations
JDBC implementation
I like to run with you, the same rhythm, the same heartbeat, I feel very good, when you fall behind, I lead everyone to run together
- JDBC class diagram
-
JdbcLockPersistence.java
Implement
LockProviderPersistence.java
interface operation database -
JdbcLockProvider.java
Inherit the abstract class
AbstractLockProvider.java
, passing theLockProviderPersistence.java
interface implementation classJdbcLockPersistence
in the constructor. -
LockProviderJdbcConfiguration.java
Declare a LockProvider Bean with @Configuration
-
JPA
MasterLockEntityRepository.java、SpringMasterLockRepository.java、org.apache.servicecomb.pack.alpha.server.cluster.master.provider.jdbc.jpa.*
-
Create master_lock Table
CREATE TABLE IF NOT EXISTS master_lock (
serviceName varchar(36) not NULL,
expireTime datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
lockedTime datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
instanceId varchar(255) not NULL,
PRIMARY KEY (serviceName)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
- serviceName Service name, this field takes the value
$ {spring.application.name}
- expireTime Lock expiration time, this field takes value lockedTime + 5s
- lockedTime Last lock time
- instanceId Service instance ID, this field takes the value
$ {alpha.server.host}]:$ {alpha.server.port}
Lock processing
The process of locking/updating a lock is a cyclically repeating action, as follows:
-
ClusterLockService.java
calls the lock method ofLockProvider.java
interface once per second -
The lock method in the abstract class
AbstractLockProvider.java
will attempt to lock the iniLock method ofJdbcLockPersistence.java
, and the locked SQL implementation is defined inMasterLockEntityRepository.java
- If the table is empty then insert a record and return the lock successfully
- Exception caught and lock failed if there are records with the same service name field in the table
@Transactional @Modifying @Query(value = "INSERT INTO master_lock " + "(serviceName, expireTime, lockedTime, instanceId) " + "VALUES " + "(?1, ?2, ?3, ?4)", nativeQuery = true) int initLock( @Param("serviceName") String serviceName, @Param("expireTime") Date expireTime, @Param("lockedTime") Date lockedTime, @Param("instanceId") String instanceId);
-
If the initLock lock fails, try to update the lock by calling the updateLock method in
MasterLockEntityRepository.java
.- If the record instanceId of the same service name is the same as the instance instanceId, the update succeeds and the lock is successfully returned.
- If the record expireTime of the same service name is less than the current lock time, the update is successful and the lock is successful.
@Transactional @Modifying(clearAutomatically = true) @Query("UPDATE org.apache.servicecomb.pack.alpha.server.cluster.master.provider.jdbc.jpa.MasterLock t " + "SET t.expireTime = :expireTime" + ",t.lockedTime = :lockedTime " + ",t.instanceId = :instanceId " + "WHERE t.serviceName = :serviceName AND (t.expireTime <= :lockedTime OR t.instanceId = :instanceId)") int updateLock( @Param("serviceName") String serviceName, @Param("lockedTime") Date lockedTime, @Param("expireTime") Date expireTime, @Param("instanceId") String instanceId);
-
UnLock
Reserved interface for extending other ways of locking
Other lock implementations
Other types of locks can be implemented through the LockProvider.java
and LockProviderPersistence.java
interfaces, such as zookeeper or redis etc.
Note
-
Need to synchronize time across multiple servers
-
Need to configure the correct time zone parameters when using MySQL database
Leave a Comment
Your email address will not be published. Required fields are marked *


class: left, middle, inverse, title-slide # The Life & Times of a Reproducible Clinical Project in R ### Jennifer L Thompson, MPH
Vanderbilt University Medical Center
Department of Biostatistics + Center for Critical Illness, Brain Dysfunction & Survivorship
### R/Medicine 2018
bit.ly/jlt-rmed2018
jent103
--- layout: true
bit.ly/jlt-rmed2018 • @jent103
 --- # Reproducibility, for Scientific Reasons ### Given your data, someone else can reproduce your exact results - Increased trustworthiness - More rigorous, reliable science - Learning more from one another's work *"Science should be ‘show me’, not ‘trust me’; it should be ‘help me if you can’, not ‘catch me if you can’."* -- *Philip B. Stark, [Nature 2018](https://www.nature.com/articles/d41586-018-05256-0)* ??? This is my informal definition for what we mean by computational reproducibility, which is. We all want the work we do to be both meaningful and correct, and want to be able to discern whether other research is trustworthy. Reproducible practices like sharing organized code scripts can help us with that. --- class: center name: personal # Reproducibility, for Personal Reasons "Did I mention that subjects with IDs > 100 are actually kangaroos and should be excluded?"
 I won the lottery 👋; now my coworker is taking over
 Journal reviews back after 8 months! Time for revisions!
 "How does that patient have 15 months of treatment when we only followed people for 12 months?" ??? Our reasons for pursuing reproducibility don't have to be entirely altruistic or philosophical. The same tools that make it easier for *others* to reproduce your analyses also make it easier for you to reproduce them, and have more confidence in your own results. True story: Someone asked me last month for a script I started writing in 2009 Even worse: This script was related to a project with five different data sources and multiple data management scripts Past me was not living her most reproducible life :( - she had not considered this scenario nine years ago when starting this project, and it was a bit of a mess. Current me is a bit more experienced and knows to expect the unexpected. All these tools we'll talk about today benefit from **forethought** - the more we can think ahead, the better off we'll be. --- class: center # Plan Ahead to Live Your Best (Research) Life
 *“It’s not thinking, ‘This is easiest for myself right now.’ It’s thinking, ‘When I’m working on this next week, next month, right before I graduate — **how do I set myself up so that it’s easier later**?’* -- *Julia Stewart Lowndes in ["A toolkit for data transparency takes shape," Nature 20 August 2018](https://www.nature.com/articles/d41586-018-05990-5)* --
 ![](images/rory_darknessdragons.gif) ??? I love this quote from Julie Lowndes - reproducibility is partially about making life easier for our future selves. But sometimes we may feel a bit like Rory Gilmore in her first semester here at Yale, maybe that it's a little overwhelming to think years from now when you have an abstract deadline next week. With all the tools and "best practices" that exist for reproducible research, sometimes it can be overwhelming or feel like an all-or-nothing proposition - if you haven't given someone a full Docker image and every iota of code and data, what's the point? While this is certainly the ideal, it can be intimidating depending on your starting point and the time you have available. (In the case of my hypothetical pulmonary fellow who needs a conference abstract in a week, creating a fully reproducible masterpiece is not my immediate priority.) --- class: inverse, center, middle ## TODAY'S GOAL Demonstrate several R tools & practices which can help us not only improve our scientific rigor, but make our lives more pleasant throughout the course of a project. ??? BUT! We can use these tools incrementally or all together - anything we do will help, and I've learned this from my own experience. My goal today: show you how these tools built for increasing the level or ease of reproducibility have been helpful in my work in clinical research. Your goal: Take what applies to and interests you and mold it to your own life (then share those ideas so we can learn from you!). Some of you may be using some or all of these tools, but hopefully we can all learn at least one new thing! I am kind of assuming that everyone is aware of and using literate programming tools like RMarkdown; if you're not, come talk to me later and I'm happy to try to convince you. :) --- class: inverse, center, middle # Phase I:
Set Yourself Up for Success ??? So the first thing I'll talk about is how to organize your project. This may feel kind of basic, but it has been really key for my sanity. And setting up our workflow well will come back to help us in a later segment, so stay tuned. --- background-image: url(images/filestructure.png) background-size: 32% background-position: 15% 85% # Organized Files, Happier Life ### One project, one directory .pull-right[ - Keep yourself organized - Allow someone else *(inc. your future self!)* to use your code - Reuse/adapt components for other projects **[RStudio Projects](https://support.rstudio.com/hc/en-us/articles/200526207-Using-Projects) + version control can help!** Projects: - Allow >1 project open at once - Keep you in the right place - Encourage 1:1 philosophy - Facilitate [`packrat`](https://cran.r-project.org/package=packrat), version control - "Your life will be better [if you use Projects]" - *Mine Cętinkaya-Rundel, yesterday* ] ??? Here I've shown an example file structure with deidentified components for a real study that we have ongoing. This is pretty typical of what I need for my projects; you may need something different, but the point is to find an organized structure that works for you. The most important aspect is that one project equals one directory. When things are organized well, it's not only calming, but makes things easier: - easier to pass along to someone else, either to reproduce or take over - easier to recognize when you can reuse components for other projects - easier to actually reuse those components, because things are intentionally modular If you're an RStudio user, using Projects can be helpful here, as Mine mentioned yesterday. The first reason I started using them was the ability to have more than one RStudio session open at once with no headaches, and as Mine discussed yesterday, they can help make sure we're in the correct directory. They also encourage a "one project, one directory" mindset, since each analysis gets its own directory, which gets one project file. Even more helpful is that this is also the structure that works best with version control, and Projects have several features that help facilitate getting git or other version control set up. If you haven't yet drunk the version control Kool-Aid, I'll give you some very practical ways that it's helped me: --- # Why You, a `[data person]`, Should Use Version Control
1. Encourages modular thinking and organization 1. Easier to know when and why something changed 1. Documents/timestamps your work 1. Easier to collaborate/work on multiple machines --

  ??? 1. Thinking in terms of git commits has helped me make additions and changes in a more thoughtful, structured way - compare this to my former "go to a meeting, come out with a list of changes and make them all at once" approach that gets confusing and tangled. 1. When you realize something in your results has changed, it's an easier and more systematic process to figure out why. You can roll back to a previous version, see what happens, and hopefully eventually either fix the issue or explain why it was necessary. 1. Using version control gives you a bit of insurance: You can prove that you stuck to your analysis plan (which is hopefully version controlled itself!) and didn't add a bunch of post hoc analyses after you broke a treatment blind. Basically, I use version control because I'm an optimistic cynic: One day, something will go wrong or change, or someone will question me, and this is a really helpful tool for managing those situations. --- background-image: url(images/usethis_hex.png) background-size: 15% background-position: 95% 5% # Use `usethis` for This [`usethis`](http://usethis.r-lib.org/) is particularly helpful for creating R packages, but has
 plenty of functions that will help you set up a general project: - Projects (`create_project()`) - Standardized file structures (`use_r()`, `use_data_raw()`) - git and GitHub (`use_git()`, `use_github()`) - And more! Bonus points: Use `usethis` to turn your project into a fully transportable package! #### **And also read these: Jenny Bryan on** - [Project-oriented workflow](https://www.tidyverse.org/articles/2017/12/workflow-vs-script/) - [Naming things](https://speakerdeck.com/jennybc/how-to-name-files) - Version control: - ["Excuse me, do you have a moment to talk about version control?"](https://peerj.com/preprints/3159/) - [Happy Git with R](http://happygitwithr.com/) *with STAT545 TAs* ??? The `usethis` package is basically built to make setting up all of these details, like git repositories and file structures, a bit easier. Especially with git, getting set up can be a little tricky, for example (I know this from personal experience). And `usethis` is particularly helpful if you build R packages, as it really shines when setting up things like documentation and licensing. And of course, read anything Jenny Bryan has written on these topics. --- class: inverse, center, middle # Phase II:
Access data programmatically ??? Now that our workflow is set up we're ready to actually work with some data. By "accessing data programmatically" I mean keeping data stored in some sort of database that is not CSV files or spreadsheets floating around, and accessing it via code of some kind rather than manually exporting or saving email attachments. This is an ideal, of course, and depending on your situation may or may not be tenable, but I hope to inspire you to pursue it whenever possible! --- .pull-left[ ## Tired **Many files, potentially with different versions, from disparate sources, manually created/exported** - No one *(including future you)* knows for certain what you did - Potential organizational nightmare - Manually exporting from data source is a potential pain point with plenty of room for error

 ###
] .pull-right[ ## Wired **Accessing your data programmatically
*(REDCap API, SQL database...)*** - Track exactly what data you access + all steps taken to manipulate it - Raw data remains in original state/location - Rerun same script(s) throughout data collection -> use most current data to monitor, clean, explore #
] ??? Our group used to use a system that involved me manually exporting about 25 forms as separate CSVs every time I went to update the data. It was, as you might imagine, not the most fun part of my day, and more importantly it was rife with opportunity for error. What if I accidentally skipped one or saved one with the wrong name? It's hard to overcome institutional inertia if you're used to getting data in spreadsheets, and APIs might be a bit intimidating to folks who are used to receiving data in formats that they can open and see. I empathize with that, but after having Excel reformat dates and not being able to remember years later which spreadsheet contained what, I've become an API believer. Writing code to extract data via APIs allows you to track exactly which data points you downloaded. All your original data remains untouched in a safe location, and there's a complete record of what you did to transform or summarize it. So anyone with access to that database can reproduce your work later if needed. --- # Example: REDCap API ```r library(httr) monthly_post <- httr::POST( url = "https://redcap.vanderbilt.edu/api/", body = list( * token = Sys.getenv("MYTOKEN"), ## API token gives you permission content = "record", ## export *records* format = "csv", ## export as *CSV* forms = "monthly_data", ## which form(s)? fields = c("study_id"), ## additional fields events = paste(sprintf("month_%s_arm_1", 1:3), collapse = ","), ## all 3 monthly visit events rawOrLabel = "label" ## export factor *labels* v codes ) ) monthly_df <- read.csv( text = as.character(monthly_post), stringsAsFactors = FALSE, na.strings = "" ) ``` ??? Our group uses REDCap basically exclusively, so here I've shown an example of how I use the `httr` package to extract data via the REDCap API. My toy example here is from a longitudinal study where we had a baseline visit; three monthly visits; and a study completion visit. I'm exporting this data in order to summarize some of the monthly values. If you use a different data capture system your code might look a bit different. My goal is less about the specifics of the code than to demonstrate some key points: - First, this is more secure than passing around even password-protected Excel spreadsheets, since you need an individualized API token to access the database; many of us here work with PHI, so obviously this is a concern. I usually store my token in my .Renviron file so that I can share the extraction code freely as needed without compromising security. --- # Example: REDCap API ```r library(httr) monthly_post <- httr::POST( url = "https://redcap.vanderbilt.edu/api/", body = list( token = Sys.getenv("MYTOKEN"), ## API token gives you permission content = "record", ## export *records* format = "csv", ## export as *CSV* * forms = "monthly_data", ## which form(s)? * fields = c("study_id"), ## additional fields * events = paste(sprintf("month_%s_arm_1", 1:3), collapse = ","), ## all 3 monthly visit events rawOrLabel = "label" ## export factor *labels* v codes ) ) monthly_df <- read.csv( text = as.character(monthly_post), stringsAsFactors = FALSE, na.strings = "" ) ``` ??? - Next, these lines tell me exactly what data I exported - there's a **record** of exactly what we did. Here, I export the monthly data collection form, along with the patient ID that's collected on a separate form, and export only the records related to our monthly data collection. It's clear to anyone coming after me what data I used to perform my next steps. --- # Example: REDCap API ```r library(httr) monthly_post <- httr::POST( url = "https://redcap.vanderbilt.edu/api/", body = list( token = Sys.getenv("MYTOKEN"), ## API token gives you permission content = "record", ## export *records* format = "csv", ## export as *CSV* forms = "monthly_data", ## which form(s)? fields = c("study_id"), ## additional fields events = paste(sprintf("month_%s_arm_1", 1:3), collapse = ","), ## all 3 monthly visit events * rawOrLabel = "label" ## export factor *labels* v codes ) ) monthly_df <- read.csv( text = as.character(monthly_post), stringsAsFactors = FALSE, na.strings = "" ) ``` ??? - Finally, depending on your data capture system, using APIs can give you **flexibility**. This last line in the `httr` call gives an example of an option you can give REDCap using the API - here I'm telling it I want text labels for categorical variables and not the numeric codes, but there may be times when you want the opposite. You can choose based on your needs for any given export. --- # Now We Can... Once the initial script or report is built, we can - **with one line of code*** - do the following, always assured that we're using the most recently updated data: 1. Frequently monitor accuracy & completion during prospective data collection ([tutorial on GitHub](https://github.com/jenniferthompson/DataCleanExample)) 1.  - Enrollment targets, protocol pain points... ### API Resources - General: [`httr`](https://cran.r-project.org/package=httr); [intro to APIs by Lucy D'Agostino McGowan](http://bit.ly/lucystats-jsm2018) - [REDCap](https://projectredcap.org/): [`redcapAPI`](https://cran.r-project.org/package=redcapAPI) (Benjamin Nutter), [`REDCapR`](https://ouhscbbmc.github.io/REDCapR/) (Will Beasley) - [ropensci.org/packages](https://ropensci.org/packages/) for specific web APIs ??? Once we're using the API to export data, the world is our oyster - we don't have to worry about waiting on updated spreadsheets or manually extracting data. I use this capability to do several things, including: - Working with our research coordinators to clean our prospectively collected data every few months. There's a tutorial for our process linked from my web site if you're interested in that. - Generate a lot of repeated analyses and reports throughout the study process - things like NIH progress reports or DSMB reports. We also use the `flexdashboard` format of RMarkdown to monitor our studies, making sure we're hitting enrollment targets and looking for potential pain points with protocol aspects like wearing devices or drawing blood. Again, I can run those reports once a week with one line of code, assured that it's current with the database. If you're new to working with APIs, the `httr` package has some great vignettes; if you do use REDCap, there are at least two packages specifically for working with that API. If you're using other sources of web data, rOpenSci has a collection of packages for working with APIs that you should investigate. --- class: inverse, center, middle # Phase III:
Test Your (Data) Assumptions ??? Once our data is in our hands, we want to make sure it reflects our understanding of it and our needs. If you develop software, you're likely already familiar with the concept of testing; if you do any data wrangling, you've definitely done this informally - checked to see if your calculated age is negative, for example. So what I'll encourage in the next few slides is a *formal* way to incorporate those practices into data exporting and wrangling, so that we're more confident in our final data and therefore our final conclusions. --- # Testing Data with `assertr` .pull-left[ ### **Sinking feeling *(n)***: Going far too long without realizing you've made a major yet sneaky error in your data wrangling. ] .pull-right[  ] -- With [`assertr`](https://github.com/ropensci/assertr) *(author: Tony Fischetti)*, we can - Include assertions in scripts that - Load raw data - *is it what we expect?* - Create analysis datasets - *is our code doing what we expect?* - Discover problems **early** - Be more **confident** in your data wrangling + analysis ??? Hopefully I'm not the only one who's had this experience: You've done your data wrangling, everything looks OK, and then one day someone asks the exact right question that allows you to uncover a mistake you made. The `assertr` package helps us avoid this by providing ways to proactively make **assertions** about our data. We can do this when reading in raw data, to make sure we understand the data we're dealing with and that the data fits the conventions we've established. We can also do this after doing some data management, to make sure our raw data and our code are working together as well as we'd expect. Using these assertions can help us discover special cases or unmet assumptions earlier. --- # Checking Raw Data ```r library(assertr) ## Creatinine must be <=20 monthly_df %>% verify(creat_m <= 20) ``` -- .left-column[ ### Result: 👎 Pipeline stops and prints an informative message ] .right-column[ ``` verification [creat_m <= 20] failed! (1 failure) verb redux_fn predicate column index value 1 verify NA creat_m <= 20 NA 5 NA ``` ```r monthly_df[5, c("study_id", "redcap_event_name", "creat_m")] ``` ``` study_id redcap_event_name creat_m 5 3 Month 2 25 ``` ] ??? In our example study, we believe that any creatinine above 20 must be some kind of data issue. So we use `assertr`'s `verify` function at the beginning of a data management pipeline to "verify" that that assumption is true. If that condition is met, the function will return the original data.frame, and we can continue with our data management. Of course, this being an example, the condition is *not* met. `verify` throws an error and stops our pipeline in its tracks, and prints a message showing us that the fifth row in our data.frame violates this assumption. In this case, my most likely guess is that someone forgot a decimal point. We can work with the PI or the data entry team to resolve the issue. Once it's resolved and we're using the updated data, this same pipeline will run with no issue. One note: If you're not a pipeline user, `assertr` certainly works with base R as well! It's built to work nicely within pipelines but you can use it no matter your preference. `verify(monthly_df, is.na(creat_m) | creat_m <= 20)` --- # Checking Raw Data ```r monthly_df %>% ## Visit date *must* be entered; ## HDL, LDL must be within range assert(not_na, date_visit_m) %>% assert(within_bounds(25, 95), hdl_m) %>% assert(within_bounds(1, 200), ldl_m) ``` -- .pull-left[ ## Result: 👍 Returns our original data frame so we can move on down the pipeline! ] .pull-right[ ![](images/rory_yesgood.gif) ] ??? It's similar to `verify`, but `assertr`'s `assert` function comes with built-in predicate functions which perform common checks; in this case, we want to make sure that every monthly visit has a recorded date, using `not_na()`, and that all our cholesterol values fall within a reasonable range using `within_bounds()`. Here, our assertion discovers no problems, and we can continue with confidence. --- # You Decide ## How Strict Your Assertions Are .left-column[ #### **Q:** What if you want to test assumptions, but keep going no matter what? #### **A:** `error_fun` ] -- .right-column[ ```r ## Maybe it's OK if creatinine is super high newdf <- monthly_df %>% * verify(creat_m <= 20, error_fun = error_append) attr(newdf, "assertr_errors") ``` ``` [[1]] verification [creat_m <= 20] failed! (1 failure) verb redux_fn predicate column index value 1 verify NA creat_m <= 20 NA 5 NA ``` ### Result: 😑 ] ??? Sometimes you might want to know about an issue but still keep the pipeline going. The `error_fun` argument gives you control over how `assertr` behaves when the conditions are met or unmet. You have several options; in this case, I've run our original assertion, but set this argument to `error_append`. Using this option will allow the pipeline to keep going, and will return our original data.frame as though there were no problems, but it will add an `attribute` to that data.frame which includes all our errors. The tradeoff to this approach is that you do need to be paying attention somehow: sink those error attributes messages to a text file or take some other tactic, so you don't defeat the purpose and miss a potential problem. --- # For More: `assertr` [tutorial](https://ropensci.org/tutorials/assertr_tutorial/)  ??? Hopefully these basic examples have intrigued you enough to investigate further. If you're interested, I'd really recommend this `assertr` tutorial that Tony has put together with some great examples. --- class: inverse, center, middle # Phase IV:
Take Advantage of That Workflow ??? Now that we've tested our data, we're finally ready to perform our analysis and report our conclusions. In my experience projects can get complex quickly, and in this case, there are some powerful tools that can help us take advantage of that structure we worked on earlier. --- background-image: url(images/drake_hex.png) background-size: 15% background-position: 95% 5% # `drake` Built to encourage and enable efficient, reproducible workflows .pull-left[ 1. Set up a workflow made of separate components 1. Describe the `plan` that will create/update these `targets` 1. `make()` the `plan` - `drake` knows which components are up to date, which to update - Full reproducibility without unnecessary rebuilding ] .pull-right[ Much more at: - [ropensci.github.io/drake](https://ropensci.github.io/drake/) - [`drake` manual](https://ropenscilabs.github.io/drake-manual/) - Will Landau's R/Pharma talk: [wlandau.github.io/drake-talk](wlandau.github.io/drake-talk) ]  ??? I'm new to `drake` myself, and as soon as I sat down to try it out, I wished I'd had it available two years ago, when I first started work on a project that involves bootstrapping about twelve separate models. The point of `drake` is to know when something needs to be updated and when it needs to be left in peace. You set up your workflow and define a set of components, or "targets"; describe a "plan" that will either create or update these targets as needed; and then "make" the plan. (If you're familiar with the concept of a `makefile`, that terminology will be familiar to you, but here we can do everything right within R.) `drake` knows which components need to be updated due to changes you've made since the last `make`, and which are current and can be left alone. So it takes care of any changes without wasting time or computing power on things that are already perfectly fine. `drake` has some fantastic documentation, including a web site and a full manual. To get started, I'd recommend checking out Will Landau's recent talk at R/Pharma, which was all about `drake`. It's much more powerful than I have time to go into here. So! You may have noticed that several times I've referenced rOpenSci, so... what exactly is rOpenSci? --- class: center, middle #  #   ??? rOpenSci is an organization that creates the **technical** and **social** infrastructure to empower a community of people, of which I'm a part, who are interested in improving scientific research via increased reproducibility and transparency. So, how exactly do we create that infrastructure? --- #  Software .pull-left[ ### Guidelines & resources for developers [`ropensci.github.io/dev_guide`](https://ropensci.github.io/dev_guide/)  ] .pull-right[ ### Discoverability - Blog posts, social media - Searchable database  ] ??? The first piece is through **software**. If you're a package developer, rOpenSci has an excellent manual to guide you through best practices in development and maintenance. This is really useful whether or not you submit your package for review with rOpenSci. If you do successfully submit your package for review, you'll get the benefits of more discoverability: for example, rOpenSci's web site has a searchable list of all its packages, and houses material like the `assertr` tutorial I linked to earlier. --- #  Peer Review #### **Goal:** Bring the best facets of academic peer review to software, in a transparent, non-adversarial process .left-column[ - **Developers:**
Feedback on design; visibility - **Reviewers:**
Improve a package; learn about design & development ] .right-column[   ] ??? That peer review process is the second piece of the rOpenSci puzzle. We review packages that fall within the scope of enabling and encouraging reproducible research and managing the data life cycle. *Developers* get the benefit of feedback on design and usability, as well as additional visibility from rOpenSci blog posts. *Reviewers* get to contribute to the community, help improve a package, and almost certainly learn a lot as they do so. I was one of the reviewers for the `skimr` package, which is really fantastic, by the way, and learned a lot during the process by thinking about things like package API design. --- #  Community Welcoming, diverse community that - Builds capacity of users & developers - Fosters pride in members' work - Advocates for data sharing, reusable software - My experience: This culture and expectation leads to - More & better collaboration - Better products - Stronger research -- > "rOpenSci combines expertise and approachability, and its community inspires people to collaborate as the best versions of themselves." -- *[Will Landau](https://twitter.com/wmlandau/status/1030178221717946371), rOpenSci community member & `drake` developer* ??? Finally, we create the social infrastructure by putting together this fantastic community of people with really diverse social and professional backgrounds, all interested in making their work more reproducible and more open. By facilitating these relationships and creating a welcoming, friendly, approachable culture, we increase what we're able to produce and create more and stronger collaboration. This quote from Will Landau sums up my feelings so well that I asked if I could just quote him: "rOpenSci combines expertise and approachability, and its community inspires people to collaborate as the best versions of themselves." --- # Info/Get Involved .left-column[  ] .right-column[ ### [ropensci.org](https://ropensci.org) ### [discuss.ropensci.org](https://discuss.ropensci.org) ### [ropensci.org/packages](https://ropensci.org/packages) ### Contribute a package via the
[onboarding process](http://onboarding.ropensci.org/)
or [sign up to review](https://ropensci.org/onboarding/)! ###
[@ropensci](https://twitter.com/ropensci) ] --- # Thank you! .pull-left[ ### Slides/contact ####
[bit.ly/jlt-rmed2018](https://bit.ly/jlt-rmed2018) ####
[jenthompson.me](https://jenthompson.me) ### Find rOpenSci: ####
[ropensci.org](https://ropensci.org) ####
[@ropensci](https://twitter.com/ropensci) ] .pull-right[

 ![](images/logan_fun.gif) ] --- class: inverse, middle, center BONUS SLIDES --- background-image: url(images/leek_decisions.png) background-size: 85% background-position: 50% 90% # Jeff Leek Made This [JSM 2018 talk](https://docs.google.com/presentation/d/1oCbf8NHpJSQAe6LoeD--wV-KbKbdGawrLHYJqwzfj_M/edit#slide=id.p) #### Or, Use RMarkdown to Record Your Decisions ??? I love this diagram from Jeff Leek so much: Saying "we did multivariable logistic regression" only tells the reader one of many, many decisions you made in your analysis, which can be affected by all kinds of factors as Jeff points out.