The preprocessing
module¶
This module contains the classes that are used to prepare the data before
being passed to the models. There is one Preprocessor per data mode or
model component: wide
, deeptabular
, deepimage
and deeptext
.
WidePreprocessor ¶
WidePreprocessor(wide_cols, crossed_cols=None)
Bases: BasePreprocessor
Preprocessor to prepare the wide input dataset
This Preprocessor prepares the data for the wide, linear component.
This linear model is implemented via an Embedding layer that is
connected to the output neuron. WidePreprocessor
numerically
encodes all the unique values of all categorical columns wide_cols +
crossed_cols
. See the Example below.
Parameters:
-
wide_cols
(
List[str]
) –List of strings with the name of the columns that will label encoded and passed through the
wide
component -
crossed_cols
(
List[Tuple[str, str]]
) –List of Tuples with the name of the columns that will be
'crossed'
and then label encoded. e.g. [('education', 'occupation'), ...]. For binary features, a cross-product transformation is 1 if and only if the constituent features are all 1, and 0 otherwise.
Attributes:
-
wide_crossed_cols
(
List
) –List with the names of all columns that will be label encoded
-
encoding_dict
(
Dict
) –Dictionary where the keys are the result of pasting
colname + '_' + column value
and the values are the corresponding mapped integer. -
inverse_encoding_dict
(
Dict
) –the inverse encoding dictionary
-
wide_dim
(
int
) –Dimension of the wide model (i.e. dim of the linear layer)
Examples:
>>> import pandas as pd
>>> from pytorch_widedeep.preprocessing import WidePreprocessor
>>> df = pd.DataFrame({'color': ['r', 'b', 'g'], 'size': ['s', 'n', 'l']})
>>> wide_cols = ['color']
>>> crossed_cols = [('color', 'size')]
>>> wide_preprocessor = WidePreprocessor(wide_cols=wide_cols, crossed_cols=crossed_cols)
>>> X_wide = wide_preprocessor.fit_transform(df)
>>> X_wide
array([[1, 4],
[2, 5],
[3, 6]])
>>> wide_preprocessor.encoding_dict
{'color_r': 1, 'color_b': 2, 'color_g': 3, 'color_size_r-s': 4, 'color_size_b-n': 5, 'color_size_g-l': 6}
>>> wide_preprocessor.inverse_transform(X_wide)
color color_size
0 r r-s
1 b b-n
2 g g-l
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py
66 67 68 69 70 71 72 |
|
fit ¶
fit(df)
Fits the Preprocessor and creates required attributes
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
WidePreprocessor
–WidePreprocessor
fitted object
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py
74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 |
|
transform ¶
transform(df)
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
np.ndarray
–transformed input dataframe
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py
99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 |
|
inverse_transform ¶
inverse_transform(encoded)
Takes as input the output from the transform
method and it will
return the original values.
Parameters:
-
encoded
(
np.ndarray
) –numpy array with the encoded values that are the output from the
transform
method
Returns:
-
pd.DataFrame
–Pandas dataframe with the original values
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py
122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 |
|
fit_transform ¶
fit_transform(df)
Combines fit
and transform
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
np.ndarray
–transformed input dataframe
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py
144 145 146 147 148 149 150 151 152 153 154 155 156 157 |
|
TabPreprocessor ¶
TabPreprocessor(
cat_embed_cols=None,
continuous_cols=None,
quantization_setup=None,
cols_to_scale=None,
auto_embed_dim=True,
embedding_rule="fastai_new",
default_embed_dim=16,
with_attention=False,
with_cls_token=False,
shared_embed=False,
verbose=1,
*,
scale=False,
already_standard=None,
**kwargs
)
Bases: BasePreprocessor
Preprocessor to prepare the deeptabular
component input dataset
Parameters:
-
cat_embed_cols
(
Optional[Union[List[str], List[Tuple[str, int]]]]
) –List containing the name of the categorical columns that will be represented by embeddings (e.g. ['education', 'relationship', ...]) or a Tuple with the name and the embedding dimension (e.g.: [ ('education',32), ('relationship',16), ...])
-
continuous_cols
(
Optional[List[str]]
) –List with the name of the continuous cols
-
quantization_setup
(
Optional[Union[int, Dict[str, Union[int, List[float]]]]]
) –Continuous columns can be turned into categorical via
pd.cut
. Ifquantization_setup
is anint
, all continuous columns will be quantized using this value as the number of bins. Alternatively, a dictionary where the keys are the column names to quantize and the values are the either integers indicating the number of bins or a list of scalars indicating the bin edges. -
cols_to_scale
(
Optional[List[str]]
) –List with the names of the columns that will be standarised via sklearn's
StandardScaler
-
scale
(
bool
) –note: this arg will be removed in the next release. Please use
cols_to_scale
instead.
Bool indicating whether or not to scale/standarise continuous cols. It is important to emphasize that all the DL models for tabular data in the library also include the possibility of normalising the input continuous features via aBatchNorm
or aLayerNorm
.
Param alias:scale_cont_cols
. -
already_standard
(
List[str]
) –note: this arg will be removed in the next release. Please use
cols_to_scale
instead.
List with the name of the continuous cols that do not need to be scaled/standarised. -
auto_embed_dim
(
bool
) –Boolean indicating whether the embedding dimensions will be automatically defined via rule of thumb. See
embedding_rule
below. -
embedding_rule
(
Literal[google, fastai_old, fastai_new]
) –If
auto_embed_dim=True
, this is the choice of embedding rule of thumb. Choices are:-
fastai_new: \(min(600, round(1.6 \times n_{cat}^{0.56}))\)
-
fastai_old: \(min(50, (n_{cat}//{2})+1)\)
-
google: \(min(600, round(n_{cat}^{0.24}))\)
-
-
default_embed_dim
(
int
) –Dimension for the embeddings if the embed_dim is not provided in the
cat_embed_cols
parameter andauto_embed_dim
is set toFalse
. -
with_attention
(
bool
) –Boolean indicating whether the preprocessed data will be passed to an attention-based model (more precisely a model where all embeddings must have the same dimensions). If
True
, the paramcat_embed_cols
must just be a list containing just the categorical column names: e.g. ['education', 'relationship', ...]. This is because they will all be encoded using embeddings of the same dim, which will be specified later when the model is defined.
Param alias:for_transformer
-
with_cls_token
(
bool
) –Boolean indicating if a
'[CLS]'
token will be added to the dataset when using attention-based models. The final hidden state corresponding to this token is used as the aggregated representation for classification and regression tasks. If not, the categorical (and continuous embeddings if present) will be concatenated before being passed to the final MLP (if present). -
shared_embed
(
bool
) –Boolean indicating if the embeddings will be "shared" when using attention-based models. The idea behind
shared_embed
is described in the Appendix A in the TabTransformer paper: 'The goal of having column embedding is to enable the model to distinguish the classes in one column from those in the other columns'. In other words, the idea is to let the model learn which column is embedded at the time. See:pytorch_widedeep.models.transformers._layers.SharedEmbeddings
. -
verbose
(
int
) –
Other Parameters:
-
**kwargs
–
pd.cut
andStandardScaler
related args
Attributes:
-
embed_dim
(
Dict
) –Dictionary where keys are the embed cols and values are the embedding dimensions. If
with_attention
is set toTrue
this attribute is not generated during thefit
process -
label_encoder
(
LabelEncoder
) –see
pytorch_widedeep.utils.dense_utils.LabelEncder
-
cat_embed_input
(
List
) –List of Tuples with the column name, number of individual values for that column and, If
with_attention
is set toFalse
, the corresponding embeddings dim, e.g. [('education', 16, 10), ('relationship', 6, 8), ...]. -
standardize_cols
(
List
) –List of the columns that will be standarized
-
scaler
(
StandardScaler
) –an instance of
sklearn.preprocessing.StandardScaler
-
column_idx
(
Dict
) –Dictionary where keys are column names and values are column indexes. This is neccesary to slice tensors
-
quantizer
(
Quantizer
) –an instance of
Quantizer
Examples:
>>> import pandas as pd
>>> import numpy as np
>>> from pytorch_widedeep.preprocessing import TabPreprocessor
>>> df = pd.DataFrame({'color': ['r', 'b', 'g'], 'size': ['s', 'n', 'l'], 'age': [25, 40, 55]})
>>> cat_embed_cols = [('color',5), ('size',5)]
>>> cont_cols = ['age']
>>> deep_preprocessor = TabPreprocessor(cat_embed_cols=cat_embed_cols, continuous_cols=cont_cols)
>>> X_tab = deep_preprocessor.fit_transform(df)
>>> deep_preprocessor.embed_dim
{'color': 5, 'size': 5}
>>> deep_preprocessor.column_idx
{'color': 0, 'size': 1, 'age': 2}
>>>
>>> cont_df = pd.DataFrame({"col1": np.random.rand(10), "col2": np.random.rand(10) + 1})
>>> cont_cols = ["col1", "col2"]
>>> tab_preprocessor1 = TabPreprocessor(continuous_cols=cont_cols, quantization_setup=3)
>>> ft_cont_df = tab_preprocessor1.fit_transform(cont_df)
>>>
>>> # simply to illustrate other types of inputs to the param 'quantization_setup'
>>> tab_preprocessor2 = TabPreprocessor(continuous_cols=cont_cols, quantization_setup=tab_preprocessor1.quantization_setup)
>>> ft_cont_df2 = tab_preprocessor2.fit_transform(cont_df)
>>> np.all(ft_cont_df == ft_cont_df2)
>>> True
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py
241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 |
|
fit ¶
fit(df)
Fits the Preprocessor and creates required attributes
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
TabPreprocessor
–TabPreprocessor
fitted object
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py
299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 |
|
transform ¶
transform(df)
Returns the processed dataframe
as a np.ndarray
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
np.ndarray
–transformed input dataframe
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py
348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 |
|
inverse_transform ¶
inverse_transform(encoded)
Takes as input the output from the transform
method and it will
return the original values.
Parameters:
-
encoded
(
np.ndarray
) –array with the output of the
transform
method
Returns:
-
pd.DataFrame
–Pandas dataframe with the original values
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py
386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 |
|
fit_transform ¶
fit_transform(df)
Combines fit
and transform
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
np.ndarray
–transformed input dataframe
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py
431 432 433 434 435 436 437 438 439 440 441 442 443 444 |
|
Quantizer ¶
Quantizer(quantization_setup, **kwargs)
Helper class to perform the quantization of continuous columns. It is
included in this docs for completion, since depending on the value of the
parameter 'quantization_setup'
of the TabPreprocessor
class, that
class might have an attribute of type Quantizer
. However, this class is
designed to always run internally within the TabPreprocessor
class.
Parameters:
-
quantization_setup
(
Dict[str, Union[int, List[float]]]
) –Dictionary where the keys are the column names to quantize and the values are the either integers indicating the number of bins or a list of scalars indicating the bin edges.
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py
61 62 63 64 65 66 67 68 69 |
|
TextPreprocessor ¶
TextPreprocessor(
text_col,
max_vocab=30000,
min_freq=5,
maxlen=80,
pad_first=True,
pad_idx=1,
already_processed=False,
word_vectors_path=None,
n_cpus=None,
verbose=1,
)
Bases: BasePreprocessor
Preprocessor to prepare the deeptext
input dataset
Parameters:
-
text_col
(
str
) –column in the input dataframe containing the texts
-
max_vocab
(
int
) –Maximum number of tokens in the vocabulary
-
min_freq
(
int
) –Minimum frequency for a token to be part of the vocabulary
-
maxlen
(
int
) –Maximum length of the tokenized sequences
-
pad_first
(
bool
) –Indicates whether the padding index will be added at the beginning or the end of the sequences
-
pad_idx
(
int
) –padding index. Fastai's Tokenizer leaves 0 for the 'unknown' token.
-
already_processed
(
Optional[bool]
) –Boolean indicating if the sequence of elements is already processed or prepared. If this is the case, this Preprocessor will simply tokenize and pad the sequence.
Param aliases:
not_text
.This parameter is thought for those cases where the input sequences are already fully processed or are directly not text (e.g. IDs)
-
word_vectors_path
(
Optional[str]
) –Path to the pretrained word vectors
-
n_cpus
(
Optional[int]
) –number of CPUs to used during the tokenization process
-
verbose
(
int
) –Enable verbose output.
Attributes:
-
vocab
(
Vocab
) –an instance of
pytorch_widedeep.utils.fastai_transforms.Vocab
-
embedding_matrix
(
np.ndarray
) –Array with the pretrained embeddings
-
tokens
(
List
) –List with Lists of str containing the tokenized texts
Examples:
>>> import pandas as pd
>>> from pytorch_widedeep.preprocessing import TextPreprocessor
>>> df_train = pd.DataFrame({'text_column': ["life is like a box of chocolates",
... "You never know what you're gonna get"]})
>>> text_preprocessor = TextPreprocessor(text_col='text_column', max_vocab=25, min_freq=1, maxlen=10)
>>> text_preprocessor.fit_transform(df_train)
The vocabulary contains 24 tokens
array([[ 1, 1, 1, 1, 10, 11, 12, 13, 14, 15],
[ 5, 9, 16, 17, 18, 9, 19, 20, 21, 22]], dtype=int32)
>>> df_te = pd.DataFrame({'text_column': ['you never know what is in the box']})
>>> text_preprocessor.transform(df_te)
array([[ 1, 1, 9, 16, 17, 18, 11, 0, 0, 13]], dtype=int32)
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py
79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 |
|
fit ¶
fit(df)
Builds the vocabulary
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
TextPreprocessor
–TextPreprocessor
fitted object
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py
106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 |
|
transform ¶
transform(df)
Returns the padded, 'numericalised' sequences
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
np.ndarray
–Padded, 'numericalised' sequences
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py
135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 |
|
fit_transform ¶
fit_transform(df)
Combines fit
and transform
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
np.ndarray
–Padded, 'numericalised' sequences
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py
165 166 167 168 169 170 171 172 173 174 175 176 177 178 |
|
inverse_transform ¶
inverse_transform(padded_seq)
Returns the original text plus the added 'special' tokens
Parameters:
-
padded_seq
(
np.ndarray
) –array with the output of the
transform
method
Returns:
-
pd.DataFrame
–Pandas dataframe with the original text plus the added 'special' tokens
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py
180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 |
|
ImagePreprocessor ¶
ImagePreprocessor(
img_col, img_path, width=224, height=224, verbose=1
)
Bases: BasePreprocessor
Preprocessor to prepare the deepimage
input dataset.
The Preprocessing consists simply on resizing according to their aspect ratio
Parameters:
-
img_col
(
str
) –name of the column with the images filenames
-
img_path
(
str
) –path to the dicrectory where the images are stored
-
width
(
int
) –width of the resulting processed image.
-
height
(
int
) –width of the resulting processed image.
-
verbose
(
int
) –Enable verbose output.
Attributes:
-
aap
(
AspectAwarePreprocessor
) –an instance of
pytorch_widedeep.utils.image_utils.AspectAwarePreprocessor
-
spp
(
SimplePreprocessor
) –an instance of
pytorch_widedeep.utils.image_utils.SimplePreprocessor
-
normalise_metrics
(
Dict
) –Dict containing the normalisation metrics of the image dataset, i.e. mean and std for the R, G and B channels
Examples:
>>> import pandas as pd
>>>
>>> from pytorch_widedeep.preprocessing import ImagePreprocessor
>>>
>>> path_to_image1 = 'tests/test_data_utils/images/galaxy1.png'
>>> path_to_image2 = 'tests/test_data_utils/images/galaxy2.png'
>>>
>>> df_train = pd.DataFrame({'images_column': [path_to_image1]})
>>> df_test = pd.DataFrame({'images_column': [path_to_image2]})
>>> img_preprocessor = ImagePreprocessor(img_col='images_column', img_path='.', verbose=0)
>>> resized_images = img_preprocessor.fit_transform(df_train)
>>> new_resized_images = img_preprocessor.transform(df_train)
NOTE:
Normalising metrics will only be computed when the fit_transform
method is run. Running transform
only will not change the computed
metrics and running fit
only simply instantiates the resizing
functions.
Source code in pytorch_widedeep/preprocessing/image_preprocessor.py
67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 |
|
fit ¶
fit(df)
Instantiates the Preprocessors AspectAwarePreprocessor
and
SimplePreprocessor
for image resizing.
Seepytorch_widedeep.utils.image_utils.AspectAwarePreprocessor
and
pytorch_widedeep.utils.image_utils.SimplePreprocessor
.
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
ImagePreprocessor
–ImagePreprocessor
fitted object
Source code in pytorch_widedeep/preprocessing/image_preprocessor.py
83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 |
|
transform ¶
transform(df)
Resizes the images to the input height and width.
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe with the
img_col
Returns:
-
np.ndarray
–Resized images to the input height and width
Source code in pytorch_widedeep/preprocessing/image_preprocessor.py
106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 |
|
fit_transform ¶
fit_transform(df)
Combines fit
and transform
Parameters:
-
df
(
pd.DataFrame
) –Input pandas dataframe
Returns:
-
np.ndarray
–Resized images to the input height and width
Source code in pytorch_widedeep/preprocessing/image_preprocessor.py
173 174 175 176 177 178 179 180 181 182 183 184 185 186 |
|


Dojox SortList test
Simple sortable example
Children of a TabContainer
Child of a BorderContainer
Raw, degradable UL list:
one
two
three
four
five
six
four
five
six
four
five
six
four
five
six
normal ul:
one
one
one
one
one
one
one
one


Angular Upload
Upload files using FormData, fall back to iframe upload when FormData isn't supported
Works in all browsers
Lightweight
No dependency on jQuery
Last upload response
{{responseData | json}}
Simple Example
Fileupload
Advanced Markup Example
Force IFrame upload
Allow Multiple (only works with FormData upload)
Allowed mimetypes. example "image/jpeg,application/pdf" the accept attribute on upload-button also allows for a javascript array of mimetypes
Fileupload
Uploaded files
{{file}}


11/11/14 Lecture 8: Statistics and Dimensionality ReductionIn this lecture, we'll cover a background of basic probability and statistics, then cover hypothesis testing, and finally dimensionality reduction using principle components analysis. Lecture slidesLecture files |


In order to actually send data to Logstash, we need a device that can grab the information that we want. For this project, I used Bro, Filebeat, and Packetbeat. Again,, these commands are for an RPM distribution of Linux.
sudo yum install libpcap
curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.2.2-x86_64.rpm
sudo rpm -vi packetbeat-5.2.2-x86_64.rpm
That's really it, from here you need to change the configuration to your liking. Something that you will definitely need to do is change the interface that your are listening on, and the output to Logstash. If you want a more real time visualization, also change the packetbeat flow reporting period to something small, like 1 second. If you need to know what interfaces you have available, you can use this command:
ip a
These steps can also be found in a repo [here](https://github.com/bah8892/NetworkMonitorVisConfig/blob/master/rpm/InstallBroScript.sh).
Update the system
yum update -y && yum upgrade -y
yum install git vim curl -y
yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel git -y
cd /opt
git clone --recursive git://git.bro.org/bro
cd bro
./configure
make
make install
export PATH=/usr/local/bro/bin:$PATH
broInterface=$(ip a | grep '2:' | grep 'en' | awk '{print $2}' |rev | cut -c 2- | rev)
echo $broInterface
sed -i "s#interface=eth0#interface=$broInterface#g" /usr/local/bro/etc/node.cfg
Enable Bro JSON logging
sed -i 's#const use_json = F#const use_json = T#g' /usr/local/bro/share/bro/base/frameworks/logging/writers/ascii.bro
In order for the bro logs to work with the visualization, we need to change the default delimiter of a “.” to a “_”, this makes them C# friendly. We can do this in file below:
/usr/local/bro/share/bro/base/frameworks/logging/main.bro
List all your possibe interfaces for Bro:
ip a
Change the interface that Bro is monitoring here:
/usr/local/bro/etc/node.cfg
Install Filebeat
Now that we have installed Bro, we can start sending logs to our Logstash server.
sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/elastic.repo << EOF
[elastic-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
sudo yum install filebeat -y
systemctl enable filebeat
service filebeat restart
mkdir /etc/filebeat/conf.d/
cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.bak
cat > /etc/filebeat/filebeat.yml << EOF
filebeat:
registry_file: /var/lib/filebeat/registry
config_dir: /etc/filebeat/conf.d
## Change this to whatever your logstash server is running on
output.logstash:
hosts: ["$1:5044"]
EOF
After you make that basic configuration, go in and change the “hosts” line to whatever logstash host you are using. You can use multiple hosts if you want to, but I do not. Now we have to tell Filebeat what logs we want to forward. So just add this configuration file to the conf.d directory.
cat > /etc/filebeat/conf.d/bro.yml << EOF
filebeat.prospectors:
- input_type: log
paths:
- /usr/local/bro/logs/current/conn.log
- /usr/local/bro/logs/current/dhcp.log
- /usr/local/bro/logs/current/http.log
- /usr/local/bro/logs/current/ssl.log
- /usr/local/bro/logs/current/dns.log
- /usr/local/bro/logs/current/known_services.log
- /usr/local/bro/logs/current/ssh.log
- /usr/local/bro/logs/current/weird.log
fields:
sensorID: $result
sensorType: networksensor
document_type: bro
EOF
systemctl restart filebeat
That will gather all the important bro logs for the visualization, and keep some of the grarbage out that Bro generates. If you DO want all the Bro logs sent to Logstash, just replace all of the file locations with the one below:
- /usr/local/bro/logs/current/*.log
This will forward ALL .log files that bro generates to Logstash.


About
I am currently at 6 River Systems (a Shopify company) working on all things robotics for application in fulfillment. See more about our robots, called Chucks here.
Previously, I was at Voyage Auto building decision making ingredients (behavior planning, predictions and remote discrete assistance) for autonomous driving, and before that, I was at Apple SPG building autonomous systems. Prior to venturing into industry, I was an academic conducting research on normative navigation behaviors for mobile service robots at the University of Freiburg as part of my PhD studies. In the more distant past, I have been fortunate to work in a number of labs including DFKI, the Manipulation lab at RI-CMU, TU-Munich, Idiap research institute among others.
My main research interest is in sequential decision making, and how it is applied to a number of AI tasks in various domains like robotics (e.g. self driving cars, warehouse fulfillment), value alignment, intelligent behavior design, amongst others.
In my free time, I like to learn new things. I am currently learning elixir and rust programming languages.


Sean McDermott
Intro to Programming
Ping Pong will count up to the number from 1 with the following exceptions:


This guide is for all committers and contributors that want to help with reviewing code contributions. Thank you for your effort - good reviews are one of the most important and crucial parts of an open source project. This guide should help the community to make reviews such that:
Every review needs to check the following six aspects. We encourage to check these aspects in order, to avoid spending time on detailed code quality reviews when formal requirements are not met or there is no consensus in the community to accept the change.
Check whether the contribution is sufficiently well-described to support a good review. Trivial changes and fixes do not need a long description. If the implementation is exactly according to a prior discussion on Jira or the development mainling list, only a short reference to that discussion is needed. If the implementation is different from the agreed approach in the consensus discussion, a detailed description of the implementation is required for any further review of the contribution.
Any pull request that changes functionality or behavior needs to describe the big picture of these changes, so that reviews know what to look for (and don’t have to dig through the code to hopefully understand what the change does).
A contribution is well-described if the following questions 2, 3, and 4 can be answered without looking at the code.
This question can be directly answered from the linked Jira issue. For pull requests that are created without prior consensus, a discussion in Jira to seek consensus will be needed.
For [hotfix]
pull requests, consensus needs to be checked in the pull request.
Some changes require attention and approval from specific committers. For example, changes in parts that are either very performance sensitive, or have a critical impact on distributed coordination and fault tolerance need input by a committer that is deeply familiar with the component.
As a rule of thumb, special attention is required when the Pull Request description answers one of the questions in the template section “Does this pull request potentially affect one of the following parts” with ‘yes’.
This question can be answered with
If the pull request needs specific attention, one of the tagged committers/contributors should give the final approval.
In this step, we check if a contribution folllows the agreed upon approach from the previous discussion in Jira or the mailing lists.
This question should be answerable from the Pull Request description (or the linked Jira) as much as possible.
We recommend to check this before diving into the details of commenting on individual parts of the change.
This is the detailed code review of the actual changes, covering:
Code guidelines can be found in the Flink Code Style and Quality Guide.
If the pull request introduces a new feature, the feature should be documented. The Flink community is maintaining both an English and a Chinese documentation. So both documentations should be updated. If you are not familiar with the Chinese language, please open a Jira assigned to the chinese-translation
component for Chinese documentation translation and link it with current Jira issue. If you are familiar with Chinese language, you are encouraged to update both sides in one pull request.
See more about how to contribute documentation.
The Flink community is using a service called @flinkbot to help with the review of the pull requests.
The bot automatically posts a comment tracking the review progress for each new pull request:
### Review Progress
* [ ] 1. The description looks good.
* [ ] 2. There is consensus that the contribution should go into to Flink.
* [ ] 3. [Does not need specific attention | Needs specific attention for X | Has attention for X by Y]
* [ ] 4. The architectural approach is sound.
* [ ] 5. Overall code quality is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) if you have questions about the review process.
Reviewers can instruct the bot to tick off the boxes (in order) to indicate the progress of the review.
For approving the description of the contribution, mention the bot with @flinkbot approve description
. This works similarly with consensus
, architecture
and quality
.
For approving all aspects, put a new comment with @flinkbot approve all
into the pull request.
The syntax for requiring attention is @flinkbot attention @username1 [@username2 ..]
.


Loading...
Connection to Ribs
Your pseudo
Your password
Connection
Forgot your password ?
Forgot your password
Enter your E-mail
Send
Return


Tuple Level
Written Syntax: RelationA - RelationB
Arcade Syntax: RelationA - RelationB
Operation Definition
This binary operation results in a relation that contains all tuples of A that is not in B. Since relations are sets, the result only has unique tuples. This operation requires:
- 1) A – a relation whose attributes in listed order have the same domains as that as B.
- 2) B – a relation whose attributes in listed order have the same domains as that as A
Usefulness:
This is useful to find the tuples unique to a relation relative to another relation. It is equivalent to the difference operation in normal sets.
Example:
Foo (Orginal)
| A |
B |
| 1 |
2 |
| 3 |
4 |
| 4 |
5 |
| 10 |
11 |
Bar (Orginal)
| A |
B |
| 1 |
2 |
| 5 |
6 |
| 4 |
9 |
| 11 |
10 |
Foo - Bar
| A |
B |
| 3 |
4 |
| 4 |
5 |
| 10 |
11 |


|
Ruby |
JavaScript |
| for...in |
number_list = [1, 2, 3, 4]
for number in number_list
puts number
end
>1
>2
>3
>4
for...in in Ruby iterates through each item in an Array. In a hash for...in iterates through each key and value individually.
|
var number_list = [1, 2, 3, 4];
for (index in number_list)
console.log(number_list[index]);
>1
>2
>3
>4
JavaScript's for...in iterates through an array by its index, whereas Ruby's for...in or each method iterates through each item in the array/object directly. for...in in JavaScript iterates through an object by each property.
|
| while |
number = 12
while number<16
puts "Your number is too small"
number+=1
end
>"Your number is too small"
>"Your number is too small"
>"Your number is too small"
>"Your number is too small"
while creates a loop that executes a specified statement as long as the test condition evaluates to true. The condition is evaluated before executing the statement.
|
var number = 12;
while (number<16){
console.log("Your number is too small");
number++
}
>"Your number is too small"
>"Your number is too small"
>"Your number is too small"
>"Your number is too small"
while creates a loop that executes a specified statement as long as the test condition evaluates to true. The condition is evaluated before executing the statement.
|
| do |
number = 0
loop do
p "endless loop until you break"
number+=1
break if number == 5
end
>"endless loop until you break"
>"endless loop until you break"
>"endless loop until you break"
>"endless loop until you break"
>"endless loop until you break"
The loop do function in Ruby creates an endless loop, unless ended with a break statement.
|
var brownies = 5;
do {
brownies -= 1;
console.log("Hurry before they run out!");
} while (brownies > 1);
>"Hurry before they run out!"
>"Hurry before they run out!"
>"Hurry before they run out!"
>"Hurry before they run out!"
do...while in JavaScript creates a loop that executes a specified statement until the test condition evaluates to false. It is similiar to a while loop except that the condition is evaluated after executing the statement, so that the specified statement is exectued at least once.
|
| until |
money = 0
until money==4 do
puts("You are getting richer!")
money+=1
end
>"You are getting richer!"
>"You are getting richer!"
>"You are getting richer!"
>"You are getting richer!"
until creates a loop that executes a specified statement until the test condition evaluates to true.
|
|
| each |
tv_shows = ["Friends","The Office", "Seinfeld"]
tv_shows.each do |show|
puts show
end
>"Friends"
>"The Office"
>"Seinfeld"
each loop goes through each item in a list (like an array or hash) and executes the code inside the block.
|
var each = function(array, iterate) {
for (var i=0; i < array.length; i++) {
iterate(array[i]);
}
};
var desserts = ["ice cream", "brownies", "cookies"];
each(desserts, function(element) { console.log("I want " + element + "!"); });
>I want ice cream!
>I want brownies!
>I want cookies!
each is not a JavaScript built-in method, but can be created as a function that takes an array and goes through each element in the array using a nested function.
|
| for |
|
var holiday_destinations = ["Italy", "Myanmar", "Iran", "Japan"]
for (var i=0; i < holiday_destinations.length; i++)
console.log(holiday_destinations[i]);
>Italy
>Myanmar
>Iran
>Japan
for loop initializes a variable, sets a condition which will evaluate to true or false, and then incremenets the variable each time the loop has been executed
|


Participant
models
Participants
db
participantFk
TestSections
participantId
TestSections
TestSection
participants
Participants
potentialLength
TestSections
TestSection
potentialWpm
TestSections
TestSection
prefix
Routes
RoutesPrefix
pressTime
KeystrokeSubmit
Keystrokes
Keystroke


“Time management” is the process of organizing and planning how to divide your time between specific activities. Good time management enables you to work smarter – not harder – so that you get more done in less time, even when time is tight and pressures are high. Failing to manage your time damages your effectiveness and causes stress. source
So far, I tried pomodoro timebox technique, it works for me and I want to keep using it. I also tried Daily Scrum and spent 15 min planning attack for the daily tasks in the morning. I'm planning to keep using the techniques, work in small iterations and to note time spent on different tasks.
The challenge might be conditioning myself and creating a habit of making breaks off the desk, which is important for my back pain.
#reflection


public class PercentEscaper extends UnicodeEscaper
UnicodeEscaper
that escapes some set of Java characters using the
URI percent encoding scheme. The set of safe characters (those which remain
unescaped) can be specified on construction.
For details on escaping URIs for use in web pages, see section 2.4 of RFC 3986.
In most cases this class should not need to be used directly. If you have no
special requirements for escaping your URIs, you should use either
CharEscapers#uriEscaper()
or CharEscapers#uriEscaper(boolean)
.
When encoding a String, the following rules apply:
plusForSpace
was specified, the space character " " is
converted into a plus sign "+".
RFC 2396 specifies the set of unreserved characters as "-", "_", ".", "!", "~", "*", "'", "(" and ")". It goes on to state:
Unreserved characters can be escaped without changing the semantics of the URI, but this should not be done unless the URI is being used in a context that does not allow the unescaped character to appear.
For performance reasons the only currently supported character encoding of this class is UTF-8.
Note: This escaper produces uppercase hexidecimal sequences. From RFC 3986:
"URI producers and normalizers should use uppercase hexadecimal digits for
all percent-encodings."
| Modifier and Type | Field and Description |
|---|---|
static java.lang.String |
SAFECHARS_URLENCODER
A string of safe characters that mimics the behavior of
URLEncoder . |
static java.lang.String |
SAFEPATHCHARS_URLENCODER
A string of characters that do not need to be encoded when used in URI
path segments, as specified in RFC 3986.
|
static java.lang.String |
SAFEQUERYSTRINGCHARS_URLENCODER
A string of characters that do not need to be encoded when used in URI
query strings, as specified in RFC 3986.
|
| Constructor and Description |
|---|
PercentEscaper(java.lang.String safeChars,
boolean plusForSpace)
Constructs a URI escaper with the specified safe characters and optional
handling of the space character.
|
| Modifier and Type | Method and Description |
|---|---|
protected char[] |
escape(int cp)
Escapes the given Unicode code point in UTF-8.
|
java.lang.String |
escape(java.lang.String s)
Returns the escaped form of a given literal string.
|
protected int |
nextEscapeIndex(java.lang.CharSequence csq,
int index,
int end)
Scans a sub-sequence of characters from a given
CharSequence ,
returning the index of the next character that requires escaping. |
codePointAt, escape, escapeSlow
public static final java.lang.String SAFECHARS_URLENCODER
URLEncoder
.public static final java.lang.String SAFEPATHCHARS_URLENCODER
public static final java.lang.String SAFEQUERYSTRINGCHARS_URLENCODER
public PercentEscaper(java.lang.String safeChars, boolean plusForSpace)
safeChars
- a non null string specifying additional safe characters for
this escaper (the ranges 0..9, a..z and A..Z are always safe
and should not be specified here)plusForSpace
- true if ASCII space should be escaped to +
rather than
%20
java.lang.IllegalArgumentException
- if any of the parameters were invalidprotected int nextEscapeIndex(java.lang.CharSequence csq, int index, int end)
UnicodeEscaper
CharSequence
,
returning the index of the next character that requires escaping.
Note: When implementing an escaper, it is a good idea to override
this method for efficiency. The base class implementation determines
successive Unicode code points and invokes UnicodeEscaper.escape(int)
for each
of them. If the semantics of your escaper are such that code points in
the supplementary range are either all escaped or all unescaped, this
method can be implemented more efficiently using
CharSequence.charAt(int)
.
Note however that if your escaper does not escape characters in the supplementary range, you should either continue to validate the correctness of any surrogate characters encountered or provide a clear warning to users that your escaper does not validate its input.
See PercentEscaper
for an example.
nextEscapeIndex
in class UnicodeEscaper
csq
- a sequence of charactersindex
- the index of the first character to be scannedend
- the index immediately after the last character to be scannedpublic java.lang.String escape(java.lang.String s)
UnicodeEscaper
If you are escaping input in arbitrary successive chunks, then it is not
generally safe to use this method. If an input string ends with an
unmatched high surrogate character, then this method will throw
IllegalArgumentException
. You should either ensure your input is
valid UTF-16 before
calling this method or use an escaped Appendable
(as returned by
UnicodeEscaper.escape(Appendable)
) which can cope with arbitrarily split input.
Note: When implementing an escaper it is a good idea to override
this method for efficiency by inlining the implementation of
UnicodeEscaper.nextEscapeIndex(CharSequence, int, int)
directly. Doing this for
PercentEscaper
more than doubled the performance for unescaped
strings (as measured by CharEscapersBenchmark
).
escape
in interface Escaper
escape
in class UnicodeEscaper
s
- the literal string to be escapedstring
protected char[] escape(int cp)
escape
in class UnicodeEscaper
cp
- the Unicode code point to escape if necessarynull
if no escaping was
needed


TALK RADIO A film review by Shane R. Burridge Copyright 1997 Shane R. Burridge
Talk Radio (1989) 110m.
Oliver Stone's film of Eric Bogosian's stage play (they collaborated on the screenplay) was received favourably by critics but ended up unnoticed by the public amid Stone's bigger-budget successes. It begins and ends with marathon performances by Bogosian in the role of Barry Champlain, a Dallas shock-jock on the verge of being syndicated nationwide. Everything about Champlain is on the edge, including his rocky relationships with producer/girlfriend Leslie Hope, wife Ellen Greene, and boss Alec Baldwin. Isolated in his darkened studio, he rants to unseen callers as if he were addressing his invisible demons. His listening audience is largely hostile, and he is always on the attack. At first we are prepared to accept Champlain as a champion for the underdog; as more and more callers run the gauntlet of his show we grow to mistrust him. The moment we believe he has finally lost his balance is when he turns on one of his admirers and insults her mercilessly, using exactly the same time of confrontational and interruptive tactics he reserves for the bigots, fascists, and junkies that comprise most of his audience.
Champlain is so unpredictable and fast-talking that it is difficult for us to figure out his real agenda. But he doesn't seem to have one. It's the line he snaps to a detractor at a ball game that gives him away, when he tells her the reason she doesn't like his show is because she has no sense of humor. Champlain is no activist, merely an egotist who is running his own private put-on (similarly, TALK RADIO masquerades as an issues piece but is essentially a character study) and is concerned only with his ratings. There are moments he has on air when he looks as if he may let the mask slip - but he has gone too far to turn back now.
Clearly, enjoyment of TALK RADIO will depend on how much you like Bogosian's performance. The scenes with him on the air crackle thanks to Stone's charged direction and tight editing by David Brenner and Joe Hutshing. If you've ever listened to late-night talkback and found some of the callers unsettling, you're bound to relive that feeling again in the paranoid atmosphere of Champlain's studio. Bogosian's meeting with an incoherent teenager (Michael Wincott) is by turns hilarious and chilling. He's just one of several memorable callers, many of whom were voiced by the same group of actors. Inspiration for both film and play was the assassination of Denver DJ Alan Berg by White Supremacists - exactly the kind of audience that would be incensed by Champlain's uncompromising broadcasts.
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


--- title: about title_abbrev: about layout: default ---
genre2 is an online literary + art journal devoted to pushing boundaries, ours and yours. Our mission is simple: publish writers’ best work outside their primary genre. That’s it. We favor no specific style or aesthetic, and we value the brand-newbie and established author equally. If you’ve created something awesome, we want to see it. Surprise us by surprising yourself.
genre2 was conceived on a cold afternoon in December 2017 at an old oak dining table just outside Pittsburgh, PA. Instead of workshopping each other’s fiction per usual, three good friends (all three married, but only two to each other, which is good because we try to keep things legal) channeled their usual caffeine and name-calling into a new idea: creating a space dedicated to the knockout work today’s writers are producing beyond the confines of their primary genres.
Why? Why not? And because sometimes you have eighteen free minutes over Christmas break and come up with an idea that’s going to take hundreds of man hours, and you jump in without really thinking things through!
So we asked around, and, as suspected, we found that our poet friends sometimes write stories, and our novelist friends sometimes draw pictures, and the essayists we half-know from Twitter also do acrobatics (these people haven’t been super helpful for the journal, but they’re still very cool). And not only do they produce this other work, but they’re also siked about the chance to get it out into the world….
Fast forward nine months, and you’ve got genre2 in the flesh. Well, sort of. We launched our first online issue on Sept. 15, 2018, and we couldn’t be prouder of the thirteen poems, stories, pieces of nonfiction, and visual works of art produced by our fourteen talented contributors.
Now the plan looks something like this:
That’s it for now, but stay tuned as this story gets much, much longer!


Proof of concept, experimental, contact form powered by Webtask. No Back End, no deploy, no Dev-Ops, just code.
This will insert the user contact request in a database and send an email notifying about it.


How Hedgehogs Became Instagram’s Most Miserable Celebrities
Notoriously shy and antisocial, pet hedgehogs have developed big followings online
A Male Birth Control Pill Won’t Change Society
Science is close to giving us effective male contraceptive, but it won’t matter unless men are
willing to use it
Online Conspiracy Theories Start With the Young
Middle school kids love watching videos about supposed celebrity secrets. But does this set them
up to be radicalized by more dangerous…
Israel’s Silent Cyberpower Is Reshaping the Middle East
As its cyber-espionage sector continues to grow, Israel is starting to face questions about who
its technology is empowering
What if the End of Game of Thrones Is Terrible? What if the End of Game of Thrones Is Terrible?
Problems in the show’s last season bode ill for its grand
finale
1
1
1


Use your Spotify account to find out when your favorite artist are having their next concerts. Do not miss out on shows near you!
An easy way for major concert goers to find their favorite artists concerts. Just login to your Spotify Account and we will do the rest finding your top artists nearest venues and major concert dates.


Australia storm back as Pakistan wickets clatter
Brisbane, Australia - Josh Hazlewood and Pat Cummins led an Australian fightback as Pakistan lost five wickets for just 19 runs after lunch on the opening day of the first cricket Test at the Gabba on Thursday.
At tea, Pakistan were 125 for five, with Asad Shafiq not out on 18 and wicketkeeper Mohammad Rizwan alongside him on 19.
The visitors had frustrated the Australian pace attack in the first session, going into lunch at 57-0 with openers Azhar Ali and Shan Masood unbeaten on 28 and 21 respectively.
The pair began to open up after the break with some sparkling attacking shots, but Hazlewood and fellow quick Cummins at last found their length and began to turn the screws.
The pressure began to tell and with the score on 75 Masood waved at a Cummins delivery outside off stump and got an edge to Steve Smith at second slip.
Azhar went the next ball when he edged Hazlewood to Joe Burns at first slip, before Haris Sohail (1) and Babar Azam (1) fell in quick succession to poor shots as the visitors crashed to 78-4.
When Nathan Lyon had Iftikhar Ahmed caught at bat-pad for seven, Pakistan were reeling at 94 for five before Shafiq and Rizwan steadied the ship in the half-hour before tea.
Pakistan earlier won the toss and chose to bat on a hot and humid morning in Brisbane.
Azhar and Masood began cautiously against the Australian pace duo of Mitchell Starc (1-35) and Hazlewood (2-30), who bowled quickly but without ever really troubling the Pakistan openers.
The visitors crawled to 33 at the first drinks break and took almost two hours to reach 50, bringing up the half-century opening stand with a Masood boundary through square leg in the penultimate over before lunch.
Pakistan had earlier caused a major surprise when they left out experienced seamer Mohammad Abbas, opting instead for pacemen Shaheen Afridi, Imran Khan and 16-year-old Naseem Shah, who were impressive in their lead-up game against Australia A.
Read also
Read More
Paine says Australia have done homework on young Pakistan
"Pakistan have got a knack of finding these young fast bowlers so it looks like they have another one to add to that rich history of fast bowlers that they seem to produce."


| Name |
|---|
| {{ item.name }} Delete |
| You have no subscribers at this time. Maybe you need to add a national authority manually? Do it here |
| Name |
|---|
| {{ item.name }} Delete |
| You have no subscribers at this time. Maybe you need to add a national authority manually? Do it here |


--- layout: blogpost title: How rankings are calculated date: 2020-01-01 loadMathJax: true ---
The model for the calculation of the Dancehall Battle Rankings was developed over multiple months in the last part of 2019 during which time different formulas were tested and discussed. The main goal was to create formulas that allow ranking different dancers and countries across different events at different times at different locations.
In what follows we will explain six different types of rankings. Three for dancers and three for countries:
The following variables are used for the different formulas discussed later:
\(C\) is the set of all countries.
\(D\) is the set of all dancers.
\(B\) is the set of all battles.
\(c\) is a single country.
\(d\) is a single dancer.
\(b\) is a single battle.
\(b_{date}\) is the date that battle \(b\) took place.
\(D_c\) is the set of all dancers from country \(c\).
\(D_{c,i}\) is the dancer of country \(c\) at index \(i\).
The first index is 1.
\(B_d\) is the set of all battles won by dancer \(d\).
\(B_{d,i}\) is the battle of dancer \(d\) at index \(i\).
The first index is 1.
\(H_d\) is the set of all battles won by dancer \(d\) in their own country.
\(H_{d,i}\) is the battle of dancer \(d\) at index \(i\).
The first index is 1.
\(A_d\) is the set of all battles won by dancer \(d\) in a country that is not their own.
\(A_{d,i}\) is the battle of dancer \(d\) at index \(i\).
The first index is 1.
\(E_c\) is the set of all battles in country \(c\).
\(E_{c,i}\) is the battle of country \(c\) at index \(i\).
The first index is 1.
The dancer rankings award points to every dancer that participated in a battle in the last 12 months. Their points are calculated based on the number of battles they have won, together with how long ago the battles took place.
There are three rankings for individual dancers: "dancer", "dancer (1 vs 1)", and "dancer (2 vs 2)". The dancer ranking takes into account both 1 vs 1 and 2 vs 2 battles. The dancer (1 vs 1) ranking takes only into account 1 vs 1 battles. The dancer (2 vs 2) ranking takes only into account 2 vs 2 battles.
A dancer get points for every win in the past 12 months. Wins that happened more than 12 months ago are not taking into account. The more months that have passed since the battle took place, the lower the amount of points that is earned. For example, a dancer gets less points for a win from 6 months ago then for a win from 2 months ago. The points currently used can be found in the following table:
| # months ago | Points |
|---|---|
| 0 | 18 |
| 1 | 17 |
| 2 | 16 |
| 3 | 13 |
| 4 | 12 |
| 5 | 11 |
| 6 | 8 |
| 7 | 7 |
| 8 | 6 |
| 9 | 3 |
| 10 | 2 |
| 11 | 1 |
| \(\geq\) 12 | 0 |
The formula to calculate the total points \(points_d\) for a dancer \(d\) given a set of battles \(B_d\), which are won by \(d\), is $$points_d = \sum_{i=0}^{|B_d|} \text{pointsOfBattle}(B_{d,i})$$ where $$ \begin{align} \text{pointsOfBattle}(b) &= lookUp(\text{diff}(b_{date}, today)) \\ \text{lookUp}(months) &= \text{look up the points in the above table based on the months} \\ \text{diff}(date1, date2) &= \text{calculate the difference in months between 2 dates} \end{align} $$
Let us use these formulas with an example. Assume that today is 2019-12-15 and we want to calculate the points \(points_{boris}\) of a dancer called Boris who won the following three 1 vs 1 battles:
| Battle | Variable name | Date |
|---|---|---|
| Battle A | \(battle_a\) | 2019-06-24 |
| Battle B | \(battle_b\) | 2019-11-12 |
| Battle C | \(battle_c\) | 2019-12-01 |
The result is 47 points for Boris.
The aforementioned battles are 1 vs 1 battles. In case it are 2 vs 2 battles, then the points are divided in half: half of the points for each of the two dancers that won the battle. In our example, the points are then 6, 8.5, and 9, resulting in 23.5 points total for one dancer.
The country rankings award points to every country (represented by one of its dancers) that participated in a battle in the last 12 months. Their points are calculated based on the number of battles they have won, together with how long ago the battles took place.
There are three rankings for countries: "country", "country (home)", and "country (away)". The country ranking takes into account both home and away battles. The country (home) ranking takes only into account home battles. The country (away) ranking takes only into account away battles.
All three country rankings are based on the dancer rankings: for each country its total points depends on the points acquired by the individual dancers of that country.
The formula to calculate the total points \(points_c\) for country \(c\) is $$points_c = \sum_{i=1}^{|D_c|} \sum_{j=1}^{|B_{D_{c,i}}|} \text{pointsOfBattle}(B_{D_{c,i},j})$$
Let us use this formula with an example. We want to calculate the points of France \(points_{france}\) taking into account the following battles:
| Battle | Variable name | Battle's country | # months ago | Winner | Winner's country |
|---|---|---|---|---|---|
| Battle D | \(battle_d\) | France | 5 | Helen | France |
| Battle E | \(battle_e\) | Spain | 7 | Joseph | France |
| Battle F | \(battle_f\) | Ukraine | 2 | Maria | Italy |
\(D_{france}\) contains the dancers from France: Helen and Joseph. \(B_{helen}\) contains the battles won by Helen: Battle D. \(B_{joseph}\) contains the battles won by Joseph: Battle E.
$$ \begin{align} points_{france} &= \sum_{i=1}^{|D_{france}|} \sum_{j=1}^{|B_{D_{france,i}}|} \text{pointsOfBattle}(B_{D_{france,i},j}) \\ &= \sum_{j=1}^{|B_{helen}|} \text{pointsOfBattle}(B_{helen,j}) + \sum_{j=1}^{|B_{joseph}|} \text{pointsOfBattle}(B_{joseph,j}) \\ &= \text{pointsOfBattle}(battle_d) + \text{pointsOfBattle}(battle_e) \\ &= 11 + 7 \\ &= 18 \end{align} $$
The total points for France is 18, based on the wins by Helen and Joseph.
The country (home) ranking is different from the country ranking because it only takes into account home wins for dancers. In other words, only wins where the country of the battle and the country of the winner are the same are taken into account.
The formula to calculate the total points \(points_c\) for country \(c\) is $$points_c = \sum_{i=1}^{|D_c|} \sum_{j=1}^{|H_{D_{c,i}}|} \text{pointsOfBattle}(H_{D_{c,i},j})$$
Let us use this formula with an example. We want to calculate the points of France \(points_{france}\) taking into account aforementioned battles.
\(D_{france}\) contains the dancers from France: Helen and Joseph. \(H_{helen}\) contains the battles in France won by Helen: Battle D. \(H_{joseph}\) contains the battles in France won by Joseph: it is empty, because Joseph did not win any battles in France.
$$ \begin{align} points_{france} &= \sum_{i=1}^{|D_{france}|} \sum_{j=1}^{|H_{D_{france,i}}|} \text{pointsOfBattle}(H_{D_{france,i},j}) \\ &= \sum_{j=1}^{|H_{helen}|} \text{pointsOfBattle}(H_{helen,j}) + \sum_{j=1}^{|H_{joseph}|} \text{pointsOfBattle}(H_{joseph,j}) \\ &= \text{pointsOfBattle}(battle_d) \\ &= 11 \end{align} $$
The total points for France is 11, based on the wins by Helen (and Joseph).
The country (away) ranking is different from the country ranking because it only takes into account away wins for dancers. In other words, only wins where the country of the battle and the country of the winner are different are taken into account.
The formula to calculate the total points \(points_c\) for country \(c\) is $$ points_c = \text{round} \left( \frac{\sum\limits_{i=1}^{|D_c|} \sum\limits_{j=1}^{|A_{D_{c,i}}|} \text{pointsOfBattle}(A_{D_{c,i},j})} {\sum\limits_{k=1}^{|B|} \text{pointsOfBattle}(B_k) - \sum\limits_{l=1}^{|E_c|} \text{pointsOfBattle}(E_{c,l})} \cdot 100000 \right) $$
Let us use this formula with an example. We want to calculate the points of France \(points_{france}\) taking into account the aforementioned battles.
\(D_{france}\) contains the dancers from France: Helen and Joseph. \(A_{helen}\) contains the battles in other countries won by Helen: it is empty, because Helen did not win any battles outside France. \(A_{joseph}\) contains the battles in other countries won by Joseph: Battle E.
$$ \begin{align} points_{france} &= \text{round} \left( \frac{\sum\limits_{i=1}^{|D_c|} \sum\limits_{j=1}^{|A_{D_{c,i}}|} \text{pointsOfBattle}(A_{D_{c,i},j})} {\sum\limits_{k=1}^{|B|} \text{pointsOfBattle}(B_k) - \sum\limits_{l=1}^{|E_c|} \text{pointsOfBattle}(E_{c,l})} \cdot 100000 \right) \\ &= \text{round} \left( \frac{\sum\limits_{j=1}^{|A_{helen}|} \text{pointsOfBattle}(A_{helen,j}) + \sum\limits_{j=1}^{|A_{joseph}|} \text{pointsOfBattle}(A_{joseph,j})} {\sum\limits_{k=1}^{|B|} \text{pointsOfBattle}(B_k) - \sum\limits_{l=1}^{|E_c|} \text{pointsOfBattle}(E_{c,l})} \cdot 100000 \right) \\ &= \text{round} \left( \frac{\text{pointsOfBattle}(battle_e)} {\sum\limits_{k=1}^{|B|} \text{pointsOfBattle}(B_k) - \sum\limits_{l=1}^{|E_c|} \text{pointsOfBattle}(E_{c,l})} \cdot 100000 \right) \\ &= \text{round} \left( \frac{7} {\sum\limits_{k=1}^{|B|} \text{pointsOfBattle}(B_k) - \sum\limits_{l=1}^{|E_c|} \text{pointsOfBattle}(E_{c,l})} \cdot 100000 \right) \\ &= \text{round} \left( \frac{7} {34 - 11} \cdot 100000 \right) \\ &= \text{round} \left( \frac{7} {23} \cdot 100000 \right) \\ &= 30,435 \end{align} $$
The total points for France is 30,435, based on the wins by Joseph (and Helen). The fraction and the multiplication by 100,000 make it not meaningful to compare these points with points of other rankings. Furthermore, the multiplication is used to avoid points that only differ in the decimals.
In case you have questions or remarks about the rankings you can always contact us via email.


Passion Library
ABOUT US
PICk A BOOK
SUGESSTION
AUTHOR
HOME
Take a Look
Start Reading
Our Abstract Section


Shellshock took terminals by storm years ago. While the majority of devices have been patched, it is still worth testing for.


--- layout: post title: "Smashing the Stack for Fun and Profit" subtitle: "Understanding Buffer Overflow" date: 2021-04-07 21:10:13 -0400 ---


Wickedblocks
Short Header.
All you have to do is choose the section you need, remove the one that you do not need for that project and paste the one you need in that moment. All the section have been given the same left/right padding. Because consistence is king. (Bullshit)
Action


You are here. Portfolio uses my own implementation of a filtering system where items are filtered based on the type of work they entailed.
A previous implementation of the website


Joe Edwards ruled out of Plymouth Argyle's Home Park clash against Morecambe
The 29-year-old has been an ever present for the Pilgrims in League Two this season
by Chris ErringtonJoe Edwards will be suspended for Plymouth Argyle’s League Two game against Morecambe on Saturday.
Edwards will miss out on the return to Home Park of former Pilgrims’ boss Derek Adams after he was booked in the 1-0 defeat at Cambridge United on Saturday.
He picked up his fifth yellow card of the season in the 43rd minute for a foul on Reggie Lambe.
It came in Argyle’s final league fixture before an amnesty on any player being banned for one match for five bookings came into force.
Edwards has played the last five games at right wing-back after being used as a holding midfielder by Ryan Lowe for much of the season.
His suspension could open the way for Peterborough United loan signing George Cooper to return to the starting line-up against Morecambe.
Edwards is one of only two players to have started every league game for Argyle this term, with goalkeeper Alex Palmer being the other one.
Left wing-back Callum McFadzean had also been an ever present until being ruled out of the game at Cambridge by a knee injury.


Priest Franz:
So, we meet at last! I am Franz, it's my sincerest pleasure to meet you, my child.
Quest


pyshgp.gp.selection module¶
The selection
module defines classes to select Individuals from Populations.
-
class
pyshgp.gp.selection.
Elite
[source]¶ Bases:
pyshgp.gp.selection.Selector
Returns the best N individuals by total error.
-
select
(population: pyshgp.gp.population.Population, n: int = 1) → Sequence[pyshgp.gp.individual.Individual][source]¶ Return n individuals from the population.
Parameters: population – A Population of Individuals. Returns: The selected Individuals. Return type: Sequence[Individual]
-
select_one
(population: pyshgp.gp.population.Population) → pyshgp.gp.individual.Individual[source]¶ Return single individual from population.
Parameters: population – A Population of Individuals. Returns: The selected Individual. Return type: Individual
-
-
class
pyshgp.gp.selection.
FitnessProportionate
[source]¶ Bases:
pyshgp.gp.selection.Selector
Fitness proportionate selection, also known as roulette wheel selection.
See: https://en.wikipedia.org/wiki/Fitness_proportionate_selection
-
select
(population: pyshgp.gp.population.Population, n: int = 1) → Sequence[pyshgp.gp.individual.Individual][source]¶ Return n individuals from the population.
Parameters: population – A Population of Individuals. Returns: The selected Individuals. Return type: Sequence[Individual]
-
select_one
(population: pyshgp.gp.population.Population) → pyshgp.gp.individual.Individual[source]¶ Return single individual from population.
Parameters: population – A Population of Individuals. Returns: The selected Individual. Return type: Individual
-
-
class
pyshgp.gp.selection.
Lexicase
(epsilon: Union[bool, float, numpy.ndarray] = False)[source]¶ Bases:
pyshgp.gp.selection.Selector
Lexicase Selection.
All training cases are considered iteratively in a random order. For each training cases, the population is filtered to only contain the Individuals which have an error value within epsilon of the best error value on that case. This filtering is repeated until the population is down to a single Individual or all cases have been used. After the filtering iterations, a random Individual from the remaining set is returned as the selected Individual.
See: https://ieeexplore.ieee.org/document/6920034
-
select_one
(population: pyshgp.gp.population.Population) → pyshgp.gp.individual.Individual[source]¶ Return single individual from population.
Parameters: population – A Population of Individuals. Returns: The selected Individual. Return type: Individual
-
-
class
pyshgp.gp.selection.
Selector
[source]¶ Bases:
abc.ABC
Base class for all selection algorithms.
-
select
(population: pyshgp.gp.population.Population, n: int = 1) → Sequence[pyshgp.gp.individual.Individual][source]¶ Return n individuals from the population.
Parameters: population – A Population of Individuals. Returns: The selected Individuals. Return type: Sequence[Individual]
-
select_one
(population: pyshgp.gp.population.Population) → pyshgp.gp.individual.Individual[source]¶ Return single individual from population.
Parameters: population – A Population of Individuals. Returns: The selected Individual. Return type: Individual
-
-
class
pyshgp.gp.selection.
Tournament
(tournament_size: int = 7)[source]¶ Bases:
pyshgp.gp.selection.Selector
Tournament selection.
See: https://en.wikipedia.org/wiki/Tournament_selection
Parameters: tournament_size (int, optional) – Number of individuals selected uniformly randomly to participate in the tournament. Default is 7. -
tournament_size
¶ Number of individuals selected uniformly randomly to participate in the tournament. Default is 7.
Type: int, optional
-
select_one
(population: pyshgp.gp.population.Population) → pyshgp.gp.individual.Individual[source]¶ Return single individual from population.
Parameters: population – A Population of Individuals. Returns: The selected Individual. Return type: Individual
-
-
pyshgp.gp.selection.
choice
(a, size=None, replace=True, p=None)¶ Generates a random sample from a given 1-D array
New in version 1.7.0.
Parameters: - a (1-D array-like or int) – If an ndarray, a random sample is generated from its elements. If an int, the random sample is generated as if a were np.arange(a)
- size (int or tuple of ints, optional) – Output shape. If the given shape is, e.g.,
(m, n, k)
, thenm * n * k
samples are drawn. Default is None, in which case a single value is returned. - replace (boolean, optional) – Whether the sample is with or without replacement
- p (1-D array-like, optional) – The probabilities associated with each entry in a. If not given the sample assumes a uniform distribution over all entries in a.
Returns: samples – The generated random samples
Return type: single item or ndarray
Raises: ValueError
– If a is an int and less than zero, if a or p are not 1-dimensional, if a is an array-like of size 0, if p is not a vector of probabilities, if a and p have different lengths, or if replace=False and the sample size is greater than the population sizeSee also
randint()
,shuffle()
,permutation()
Examples
Generate a uniform random sample from np.arange(5) of size 3:
>>> np.random.choice(5, 3) array([0, 3, 4]) >>> #This is equivalent to np.random.randint(0,5,3)
Generate a non-uniform random sample from np.arange(5) of size 3:
>>> np.random.choice(5, 3, p=[0.1, 0, 0.3, 0.6, 0]) array([3, 3, 0])
Generate a uniform random sample from np.arange(5) of size 3 without replacement:
>>> np.random.choice(5, 3, replace=False) array([3,1,0]) >>> #This is equivalent to np.random.permutation(np.arange(5))[:3]
Generate a non-uniform random sample from np.arange(5) of size 3 without replacement:
>>> np.random.choice(5, 3, replace=False, p=[0.1, 0, 0.3, 0.6, 0]) array([2, 3, 0])
Any of the above can be repeated with an arbitrary array-like instead of just integers. For instance:
>>> aa_milne_arr = ['pooh', 'rabbit', 'piglet', 'Christopher'] >>> np.random.choice(aa_milne_arr, 5, p=[0.5, 0.1, 0.1, 0.3]) array(['pooh', 'pooh', 'pooh', 'Christopher', 'piglet'], dtype='|S11')
-
pyshgp.gp.selection.
get_selector
(name: str, **kwargs) → pyshgp.gp.selection.Selector[source]¶ Get the selector class with the given name.
-
pyshgp.gp.selection.
median_absolute_deviation
(x: numpy.ndarray) → numpy.float64[source]¶ Return the MAD.
Parameters: a (array-like, shape = (n,)) – Returns: mad Return type: float
-
pyshgp.gp.selection.
random
()¶ random_sample(size=None)
Return random floats in the half-open interval [0.0, 1.0).
Results are from the “continuous uniform” distribution over the stated interval. To sample \(Unif[a, b), b > a\) multiply the output of random_sample by (b-a) and add a:
(b - a) * random_sample() + a
Parameters: size (int or tuple of ints, optional) – Output shape. If the given shape is, e.g., (m, n, k)
, thenm * n * k
samples are drawn. Default is None, in which case a single value is returned.Returns: out – Array of random floats of shape size (unless size=None
, in which case a single float is returned).Return type: float or ndarray of floats Examples
>>> np.random.random_sample() 0.47108547995356098 >>> type(np.random.random_sample()) <type 'float'> >>> np.random.random_sample((5,)) array([ 0.30220482, 0.86820401, 0.1654503 , 0.11659149, 0.54323428])
Three-by-two array of random numbers from [-5, 0):
>>> 5 * np.random.random_sample((3, 2)) - 5 array([[-3.99149989, -0.52338984], [-2.99091858, -0.79479508], [-1.23204345, -1.75224494]])
-
pyshgp.gp.selection.
shuffle
(x)¶ Modify a sequence in-place by shuffling its contents.
This function only shuffles the array along the first axis of a multi-dimensional array. The order of sub-arrays is changed but their contents remains the same.
Parameters: x (array_like) – The array or list to be shuffled. Returns: Return type: None Examples
>>> arr = np.arange(10) >>> np.random.shuffle(arr) >>> arr [1 7 5 2 9 4 3 6 0 8]
Multi-dimensional arrays are only shuffled along the first axis:
>>> arr = np.arange(9).reshape((3, 3)) >>> np.random.shuffle(arr) >>> arr array([[3, 4, 5], [6, 7, 8], [0, 1, 2]])


| CS 107 - Introduction to Scientific Computation Homework #3 |
1. Polynomial: A polynomial is a function of the form c0 * x0 + c1 * x1 + c2 * x2 + ... + cn * xn. Since x0=1, this is equivalent to c0 + c1 * x + c2 * x2 + ... + cn * xn. All polynomial curve can be expressed in this form. The familiar line y = mx + b has n=1 with c1 being slope m and c0 being y-intercept b. A quadratic curve y = ax2 + bx + c has n=2 with c0, c1, and c2 being c, b, and a, respectively.
You will create a polynomial function "polynomial" in file polynomial.m that takes in (1) a scalar x, and (2) a vector coeff of coefficients (e.g. [c0 c1 c2]) and returns the y for that given polynomial. Note: The degree n of the polynomial is implicitly given by the length of the coefficient vector.
Example usage:
>> polynomial(7, [5])
ans =
5
>> polynomial(7, [5 2])
ans =
19
>> polynomial(7, [5 2 3])
ans =
166
2. Numerical Integration: While symbolic integration, the symbolic manipulation we learn in Calculus classes, is important and has valuable applications, there are many occasions where one is given raw data to integrate rather than a function. Given a function y = f(x), integration attempts to find the area underneath the curve over some interval. When one is given raw (x,y) values, a common simple method for integration is, for each adjacent pair of points, (xi, f(xi)) and (xi+1, f(xi+1)) to consider the trapezoid formed by these points and (xi, 0) and (xi+1, 0). A simple, good numerical integration approximation of the integral of a function can be formed by computing and summing the area of these trapezoids. Note: Area below the axis y = 0 is considered negative area.
For this exercise, we'll take a function sin(x) with a known integral function -cos(x), and compare the results of this trapezoidal numerical integration technique with the known correct results.
Begin by downloading the m-file called
trapezoidIntegrate.m
. and follow
the implementation instructions therein.
From 0 to 1.5708 with step 0.1, the approximate area is 0.928488. The actual area is 1, so the error is 0.0715117. From 0 to 1.5708 with step 0.001, the approximate area is 0.999204. The actual area is 1, so the error is 0.00079641. From 0 to 1.5708 with step 1e-005, the approximate area is 0.999994. The actual area is 1, so the error is 6.3268e-006.3. Numerical Differentiation: Differentiation is the process of computing the rate of change, or slope, of a function. In cases where we are given a function that has no simple symbolic differentiation, we can turn to numerical differentiation.
Imagine you are standing at a point on a mountain slope, and you wish to measure the slope where you're standing. A single point altitude doesn't help. However, suppose you take an accurate altitude reading both one small step uphill and one small step downhill. By dividing the rise (change in altitude between the steps) by the run (distance horizontally between the two steps), we gain a good estimate for the slope where you're standing.
This analogy describes the simple three-point estimation method. Given an x value, a function f(x), and a small step h, one can estimate the derivative (i.e. slope) of f at x as (f(x + h) - f(x - h)) / (2 * h).
For this exercise, we'll again take the function sin(x) with a known derivative function cos(x), and compare the results of this numerical differentiation technique with the known correct results along a sequence of points.
Begin by creating a function called simpleDifferentiate.m
.
Function simpleDifferentiate
will take dx and h as parameters (in
that order). It will return the greatest (maximum absolute) error. First, assign values to a set of
variables:
xMin = 0; % minimum x test value xMax = 2 * pi; % maximum x test value x = xMin:dx:xMax; % x test values
Next, compute a vector of y values for sin(x).
Plot x versus y as dots and hold the graph output.
Compute estimates and exact values for the derivative (cos(x)) at each x test
value, plotting each with "x" marks and lines, respectively. An example
plot for the variable values above appears below.
Finally, compute the maximum absolute error in the estimation, and display the
following output (with <placeholders> substituted with values):
From <xMin> to <xMax> with step <dx>, we numerically differentiate with h = <h>. The greatest error is <maximum absolute error> at x = <x value for maximum absolute error>.For example, here is the output of three m-file executions with dx = .1 and h assigned .5, .05, and .005, respectively.
From 0 to 6.28319 with step 0.1, we numerically differentiate with h = 0.5. The greatest error is 0.0411489 at x = 0. From 0 to 6.28319 with step 0.1, we numerically differentiate with h = 0.05. The greatest error is 0.000416615 at x = 0. From 0 to 6.28319 with step 0.1, we numerically differentiate with h = 0.005. The greatest error is 4.16666e-006 at x = 0.


yasuragi
言語交流学園
« Previous
ないで form conjugation
つもり・Planning to do
Grammar
つもり is used when you want to say what you're planning to do.
Verb + つもり
食
た
べるつもり
planning to eat.
見
み
るつもり
planning to see/watch.
帰
かえ
るつもり
planning to go home.
読
よ
むつもり
planning to read.
Need more explanation? Join our server via
discord.gg/yasuragi
Support us on
Patreon.com/yasuragi


Rory McIlroy leads WGC-HSBC Champions heading into final round
by By Press AssociationRory McIlroy claimed the outright lead at the WGC-HSBC Champions in China with a third successive five-under round of 67.
McIlroy closed day two one shot behind Matthew Fitzpatrick but capitalised as the Englishman slipped from the top of the leaderboard on Saturday, finishing with a birdie to head into the final round with a one-stroke advantage over Louis Oosthuizen, who shot 65.
For the second day running McIlroy hit the turn in 33, but he went bogey-free for the first time this week and added two birdies on the back nine.
Oosthuizen opened his round with five birdies in pursuit of a maiden World Golf Championships title, while Fitzpatrick dropped to third, two shots off McIlroy alongside Xander Schauffele of the United States.
England’s Paul Waring is a shot further back in fifth.
Despite holding the lead, McIlroy saw plenty of room for improvement in his round.
READ MORE
McIlroy's closing eagle moves him second at WGC HSBC Champions
“I feel like I weathered a mini-storm at the start,” he said. “After that, the round started to get going and I hit some good shots and held some good putts.
But I gave myself plenty of chances – a lot of chances from between 15 and 20 feet, so you’re not expecting to hole all of them – but I hit a lot of good putts today that didn’t go in.
“I stayed patient, I birdied the holes that I needed to birdie and I built a very gradual round. No bogeys, which was nice, but just kept plugging away trying to give myself chances.
“If I go out tomorrow and replicate what I’ve done the first three days, I think I’ll be pretty hard to beat.”


For african cusine and bevereges visit us or order in through iReserve.
Your order of will cost you Ksh.


[1]
See under opinion section; See under opinion section
[2]
VIDEO QUALITY,STILL AND SOUND IS GREAT TOO,USB A DEFINITE; NOTHING WORTH WYNING ABOUT


Welcome to my page!
I’m Shirin, a biologist turned bioinformatician turned data scientist.
I’m especially interested in machine learning and data visualization. While I am using R most every day at work, I wanted to have an incentive to regularly explore other types of analyses and other types of data that I don’t normally work with. I have also very often benefited from other people’s published code in that it gave me ideas for my own work; and I hope that sharing my own analyses will inspire others as much as I often am by what can be be done with data. It’s amazing to me what can be learned from analyzing and visualizing data!
My tool of choice for data analysis so far has been R. I also organize the MünsteR R-users group on meetup.com.
I love dancing and used to do competitive ballroom and latin dancing. Even though I don’t have time for that anymore, I still enjoy teaching “social dances” once a week with the Hochschulsport (university sports courses).
I created the R package exprAnalysis, designed to streamline my RNA-seq data analysis pipeline. It is available via Github. Instructions for installation and usage can be found here.
This blog will showcase some of the analyses I have been doing with different data sets (all freely available). I will also host teaching materials for students to access in conjunction with R courses I am giving.
Also check out R-bloggers for lots of cool R stuff!


George was just standing there smiling in the doorway to his son’s bedroom. The door was slightly closed and he was just checking up on Stephen. George was just surprised is all because apparently Stephen had a copy of the house’s dictionary in his lap. In front of Stephen was his first computer. Stephen was in the act of teaching his computer English. Stephen had a copy of a Webster’s Dictionary and for a reason that George could not understand Stephen was typing each word from the dictionary into the word processor meticulously one after the other. Now George was just watching. George stood there for a full fifteen minutes just to make certain that what he was watching as he thought for certain that his son was engaged in teaching himself to type and was trying to learn new vocabulary words.
George then knocks on the door.
Stephen looks over and says for George to come in. His dad.George walks over some toys and into Stephen’s room and asks a question that he truly knows he knows the answer to as what else could Stephen be doing other than learning how to type as he chose words from the dictionary to type into Microsoft Word.
So George politely asks.
“Stephen, what are you doing?” asked George.
Stephen replies quite seriously.
“Dad, I am teaching my computer.” said Stephen.
“What are you teaching it?” replied George.
“I am teaching it how to read.” said Stephen.
“Don’t you think your computer already knows how to read?” said George.
“How could it, no one has ever taken the time to teach it.” said Stephen.
“How long have you been doing this?” said George.
“All afternoon.” said Stephen.
“How are you teaching your computer to read?” asked George.
“I am picking out the really hard words that the computer does not know from the dictionary and I am typing them so that when it has to pass its next test it will get an A+.” replied Stephen.

“About forty six.” replied Stephen.
“How do you know the computer is learning?” replied Stephen.
“Well when I used spell check the computer let me know that it did not know all of the words. I want my computer to know all of the words. I want a smart computer so it can pass it’s next test.” said Stephen.
“How did the computer do on it’s last test?” asked George.
“It failed.” said Stephen.
“Who was the teacher that gave the test?” said George.
“Lynda.” said Stephen.
“Who is Lynda.” asked George.
“Lynda is my computer’s teacher.” said Stephen.
“Stephen, how many words are you going to teach your computer?” asked George.
“I am going to teach it all of the hard words and then it should be fine on it’s next test.” said Stephen.
“Do you want some help, I think I know a quick way of teaching it all of the hard words.” replied George.
“NO! There is no quick way to learn all of the hard words. I want my computer to get an A+ and you cannot get A+’s by cheating, this is the only way to do it.” said Stephen.
“That’s true , you are doing it the right way.” said George.“Alright, well Dad. My computer wants to let you know that it wants to get back to studying.” said Stephen.
“Alright. Well I will leave you two alone. Just make sure to clean up for dinner.” said George.
“I will.” said Stephen.
George turns his back to walk away.
“Dad, did you know the computer already knows all of the words?” said Stephen.
“Yes Stephen. I knew that.” said George.
“Well, it might not know one of em.” said Stephen.
“Well if there is a word that that computer does not know, its probably in the dictionary. There are so many words in the dictionary Stephen that I am certain it does not know a few of em.” said George.
“Well, if I can help it learn one word, it will be smarter than the rest of the class won’t it?” said Stephen.
“Yes Stephen, if you can teach the computer just one word it does not already know, it will be smarter than the rest of the computers.” said George.
“Alright, Dad, I have to get back to work. The computer is really good at spelling.” said Stephen.
“It usually is.” said George.
“Well, I am doing a good job aren’t I.” said Stephen.
“Yes you are, I am certain the computer appreciates someone teaching it new words.” said George.
“I hope so.” said Stephen.
“Alright, well I am going to go back downstairs. Let me know how your student does on the test.” said George.
“Its going to get an A+.” said Stephen.
“Yes it will Stephen, yes it will.” said George.
“Thanks Dad.” said Stephen.
“Anytime son.” said George.
With that George walked out of the room , kind of realizing that sometimes things are not quite as they seem on the surface, especially when it comes to kids using the computer.


Chest of Wisdom:
[A voice emanates from the chest]
I will tell you a secret that will bring you great riches. Find the Forbidden Love Scroll and take it to Torai. You will be amply rewarded.


Hashes
Array
)
To create a Hash, we use curly braces {}
hash = {}
# => {}
Hashes always have pairs of keys and values
hash = { key: "value", another_key: "another value" }
# => {:key=>"value", :another_key=>"another value"}
hash = { key: "value", another_key: "another value" }
can also be written like this
hash = { :key => "value", :another_key => "another value" }
It's just a different way to write it,
whereas key: "value"
is more common
movie = {
name: "The Matrix",
released: 1999,
imdb_rating: 8.7
}
# => {:name=>"The Matrix", :released=>1999, :imdb_rating=>8.7}
We can use the key in [square brackets] to read a value
hash[:key]
# => "value"
Reading from movie
movie[:name]
# => "The Matrix"
movie[:released]
# => 1999
We can assign a new value to a key
hash[:key] = "new value"
# => "new value"
Updating a value for movie
movie[:imdb_rating] = 9.3
# => 9.3
movie
# => {:name=>"The Matrix", :released=>1999, :imdb_rating=>9.3}
The same way we update values,
we can add new key/value pairs
hash[:new_key] = "yet another new value"
# => "yet another new value"
Adding a key/value pair to movie
movie[:tomatometer] = "88%"
# => "88%"
movie
# => {:name=>"The Matrix", :released=>1999, :imdb_rating=>9.3,
# :tomatometer=>"88%"}
We use the method .delete(:key)
to delete a pair
hash.delete(:new_key)
# => "yet another new value"
Removing a key/value pair from movie
movie.delete(:tomatometer)
# => "88%"
movie
# => {:name=>"The Matrix", :released=>1999, :imdb_rating=>9.3}
# creating new hash
movie = { name: "WALL·E", released: 2008, imdb_rating: 4.8 }
# updating values
movie[:imdb_rating] = 8.4
# adding new key/value pairs
movie[:tomatometer] = "88%"
# deleting key/value pairs
movie.delete(:tomatometer)
movie
# => {:name=>"WALL·E", :released=>2008, :imdb_rating=>8.4}
We mainly use Hashes for two purposes
person = {
name: 'Harry',
age: 14,
hair_color: 'Black'
}
airports = {
CDG: 'Paris',
JFK: 'New York',
YYZ: 'Toronto'
}
There are some useful methods for hashes
.keys
.values
.key?(key)
.has_value?(value)
movie = { name: "Fargo", released: 1996, imdb_rating: 8.1 }
movie.keys # => [:name, :released, :imdb_rating]
movie.values # => ["Fargo", 1996, 8.1]
movie.key?(:name) # => true
movie.key?(:director) # => false
movie.has_value?(8.1) # => true
movie.has_value?(2019) # => false
What questions do you have?


Look: Dwyane Wade + Gabrielle Union Shows LeBron James’ Son The Most Respect – “We Been Telling Y’All This Whole Time It’s Bigger Than Basketball”
by Team SOHHNBA superstar LeBron James has true love and respect for Dwyane Wade and his wife Gabrielle Union. King James went to Instagram this weekend to share an epic shot of his former Miami Heat teammate and his queen turning up at one of his son’s high school basketball games. Look and comment below!


General discussion about Komodo. For useful links and more info click the pinned messages to the right. DON'T DM @jl777
100 messages
Before 05-Dec-19 12:00 AM
nocturnal07-Jun-19 09:24 AM
what does 'disputed' mean in practical terms?
LocoMB07-Jun-19 09:25 AM
yeah ok that was my question sort of
jl777c07-Jun-19 09:25 AM
more than one person makes a claim for the same address
that triggers the dispute process
Keppie07-Jun-19 09:25 AM
How long was binance/bittrex asked to close kmd deposits ?
nocturnal07-Jun-19 09:25 AM
so why would my address become disputed if not in first batch?
jl777c07-Jun-19 09:25 AM
soon after we found evidence of stolen funds
LocoMB07-Jun-19 09:26 AM
only, the rightful claimant won't know it is disputed- only the unrightful claimant
jl777c07-Jun-19 09:26 AM
there have been scammers trying to get the old privkeys
such scammer would then submit a false claim
Mr.Cryptodude07-Jun-19 09:27 AM
Those with funds under 7777 will get them today?
jl777c07-Jun-19 09:27 AM
so it is VERY important that you submit a claim
that would trigger the dispute process if somebody else made a claim for your funds
I promised the < 7777 will get refunds before the 15th
Mr.Cryptodude07-Jun-19 09:28 AM
Alright
jl777c07-Jun-19 09:28 AM
still have a bunch of custom code to write to process the claims
i repeat, submit a claim if you havent done so already
nocturnal07-Jun-19 09:29 AM
when is cut off for 'first batch'?
sunday ok?
at work with imminent deadlines
jl777c07-Jun-19 09:29 AM
it is up to you
nocturnal07-Jun-19 09:30 AM
when is the cut off?
for first batch
jl777c07-Jun-19 09:30 AM
there is no cutoff
nocturnal07-Jun-19 09:30 AM
so confused
jl777c07-Jun-19 09:30 AM
the next step i have is to send out the special tx to the compromised addresses
that are small and undisputed
once you get that you need to spend it to prove you have the privkey
I will process the very small ones first
nocturnal07-Jun-19 09:32 AM
ok sounds long, will have to wait. best of luck with all
LocoMB07-Jun-19 09:32 AM
do you have timestamps on when the claims were made? as in, the earlier, the more credible?
1CryptoTrader07-Jun-19 09:34 AM
guys, i cant get into my agama wallet?
miles07-Jun-19 09:34 AM
What a week - A massive thank you to @jl777c and the entire KMD team for acting so swiftly to protect the community and their funds, being online everywhere at all hours - truly amazing to see, thank you
4
Kaed07-Jun-19 09:34 AM
@jl777c So we'll need to use the compromised seed, right ?
jl777c07-Jun-19 09:35 AM
yes, you need to use the compromised seed
and yes the initial contact will tell us which is the imposter. since we have the vast majority of funds already claimed anybody that comes in now will immediately be flagged and investigators put on their trail
Kaed07-Jun-19 09:36 AM
Okay. Which way (wallet) is the most easy and fast to do it when i'll need to verify i'm the owner ?
jl777c07-Jun-19 09:37 AM
anything is fine, it is a compromised privkey anyway
you just need to show you have access to it
Kaed07-Jun-19 09:38 AM
Yes but i don't want to mess with my oceanqt wallet
(well, no seed there so nvm :p)
jl777c07-Jun-19 09:39 AM
you could log in to verus wallet with old seed, do the one tx and be done with it
Kaed07-Jun-19 09:39 AM
Ok so i'll get verus wallet then
jl777c07-Jun-19 09:39 AM
verus wallet is very good
dxer66607-Jun-19 09:44 AM
be carefull with scammers . last night someone with simillar id of komodo team, give me all of my data include seed .. email ,tx id and etc...
jl777c07-Jun-19 09:47 AM
do not give out your seed to ANYBODY, there is no need for that in the claim process
tomsmith2607-Jun-19 09:49 AM
I filled out the form of fund return already but I have not yet recieved my funds so when is it available to get back my KMDs
jl777 announced his plan: most small refunds will be done by june 15, larger or contested ones will take longer but not much: "Believe me, i want to complete all the refunds as soon as possible so we can get back to normal" #KMD
Lennart07-Jun-19 09:50 AM
will you process the small balances only of KMD, or are assets like DEX and VRSC also refunded in that time period?
jl777c07-Jun-19 09:50 AM
KMD first
Malti07-Jun-19 09:50 AM
@dxer666 somebody gave you your seed??
jl777c07-Jun-19 09:50 AM
then onto the other chains
Lennart07-Jun-19 09:51 AM
ok thx
Steve1007-Jun-19 09:52 AM
@jl777c my sister needs to file a claim but she asked me what txid did you need on the form? I looked at the form and wasn't sure. Is txid on the form the txid of the transaction of kmd team sweeping her wallet?
jl777c07-Jun-19 09:52 AM
if in doubt, leave it as 0
Steve1007-Jun-19 09:52 AM
Lol ok, I'll tell her thanks
polycryptoblog07-Jun-19 09:53 AM
I will help you out @Steve10 this weekend
Steve1007-Jun-19 09:53 AM
Great job in defending against this by the way
polycryptoblog07-Jun-19 09:53 AM
Or later today I got to sleep first lol
jl777c07-Jun-19 09:53 AM
i did the blockchain analysis, so the txid info is not needed anymore
Steve1007-Jun-19 09:53 AM
@polycryptoblog thank you! I'll have some questions about ARRR too
1
istante07-Jun-19 09:54 AM
@jl777c i feel to say you thanks for all you make in these days... thanks jl777...(edited)
jl777c07-Jun-19 09:55 AM
i am very sorry this all happened and doing my best to get things back to normal
3
dxer66607-Jun-19 09:56 AM
@Malti no one scammer wanted help me to fill form, and her id was like komodo team members, i sent to him all info... and finally he said me that you will receive email....he also asked about my discord email..
Steve1007-Jun-19 09:57 AM
@jl777c unfortunately it happens to even the best of them like kmd and BNB. It's the way a team handles it that makes or breaks them. CZ did it well and you have too.
jabe07-Jun-19 10:03 AM
Hi, just joined - is there much hope that the hackers will get caught, or at least not be able to move stolen funds?
matspb07-Jun-19 10:03 AM
What amount would you describe as 'small' for the purpose of refunding?
polycryptoblog07-Jun-19 10:05 AM
Under 7777
@jabe the situation is still under investigation and all the facts need to be considered to decide the best course of action, when a decision has been made the community will be informed.
Cryptoon07-Jun-19 10:08 AM
Most probably funds will be flagged at minimum and when they touch a CEX the account will be blocked/investigated I assume?
jabe07-Jun-19 10:08 AM
@polycryptoblog Good to know. I'm very impressed with how this situation is being handled, hats off.
1
debaas07-Jun-19 10:10 AM
Microsoft has a bug in RDP, let's people take control of a remote computer. This is the biggest software giant in the world. RDP is used by millions of people. Bugs/hacks/exploits happen to the best. The important thing is how to solve them. And the komodo team is handling this very well.
mrlynch07-Jun-19 10:12 AM
i wouldn't call microsoft the best tho
Cryptoon07-Jun-19 10:12 AM
Given the technical skill needed to pull off this feat in such a short time, I wouldnt be surprised kmd team is capable enough to find out who is responsible... First and foremost is to make the people whole again, burning people responsible at the cross will follow later :p
Warkid07-Jun-19 10:35 AM
To be technical: If the seed was send in one form to a server... how is it possible only the Komodo team had acess to all new entered seeds (the last 2 month) and noone else? So it was send to a Komodo server, right??
jl777c07-Jun-19 10:36 AM
not at all
we decoded the exploit and figured out where the data was, and copied it from the hacker
Warkid07-Jun-19 10:37 AM
So you were faster than the hacker and he (or she?) did not get anything?
lion07-Jun-19 10:37 AM
@jl777c I have already applied! My funds were swept away to your safe wallet! The balance is now zero - do I still need some kind of transaction?
Warkid07-Jun-19 10:38 AM
If so, you got my respect and i apologize for all I criticized
jl777c07-Jun-19 10:39 AM
no problem. i know everybody is stressed out and worried
lion07-Jun-19 10:41 AM
@jl777c I applied for all the instructions, but how can I make sure that it is safely delivered and processed? Will they contact me at discord?
jl777c07-Jun-19 10:44 AM
now that the vast majority of addresses have been claimed, i can reveal the three step process that will be used for processing refunds.
1. a special transaction will be send to your compromised address. it will be for a fraction of a KMD
2. you need to send 0.0001 (or more) from the compromised address to the destination address you specified
3. IF and only IF, your address has no disputes, or contradictory claim info, and the proper destination address receives the special payment, a refund tx will be made. PLEASE do not ask when these refund tx will be made, i am working as fast as i can, so it will happen when it happens.
what do you need to do?
if your claim amount matches the current batches size range check to see if you got the special transaction. Then make sure you only send it to the address you claimed. after that, just wait. i will announce status of each batch.
if you didnt get a special tx, that means your address is under dispute or you filed too many claims or it is the wrong size. PLEASE wait until we process all the problem free cases first. They will be done AFTER the easy cases are done.
Please dont expect any refund payments until june 15, maybe i can get it done sooner, but step 3 is a LOT of custom code that cannot have any errors
5
Warkid07-Jun-19 10:52 AM
Can we assume, the Komodo Ocean QT is a safe wallet? Just asking cause everybody is talking about the Verus.
jl777c07-Jun-19 10:53 AM
oceanQT is good too
Warkid07-Jun-19 10:53 AM
Thank you!
RAE07-Jun-19 10:56 AM
Is this special process for both KMD and BTC claims?
jl777c07-Jun-19 10:56 AM
we are doing KMD first
techblack07-Jun-19 10:57 AM
I choose a Destination adress on broker which say if transaction is lower Khan 0,1 KMD they will be Lost. Do i need to refill Form with another adress?
lion07-Jun-19 10:58 AM
@jl777c I filled it out correctly - and I had less than 7777 funds - but I haven’t received anything at the compromised address so far! But I have both the private key of the compromised wallet and access to the wallet of the exchange from which I sent ALL means of accumulation to agama. - what to do with it - is this additional evidence if fraudsters want to challenge my funds?
Buddz199307-Jun-19 10:59 AM
It prob will be send in the next few days no?
lion07-Jun-19 11:00 AM
@jl777c Does this mean if I did not receive a short-cut transaction from you to a compromised address - but I correctly filled out the application form - does this mean that someone else disputes my address? How to make sure that my application has reached? After all, you do not recommend creating more than one application in order to avoid a long resolution of the issue!
Cryptoon07-Jun-19 11:01 AM
@lion please read... target date to start refunds is june 15th... we are 7th now. Wait a week.
techblack07-Jun-19 11:03 AM
I need to send at lest 0,11KMD from compromised adress will that be possible or do i need to change Destination adress in Form???
John the Cashist07-Jun-19 11:04 AM
@Hollowman9000 Monitor the seed collection point to see if the attacker accidentally uses compromised agama and steal back stolen funds.
That would be brilliant
lion07-Jun-19 11:08 AM
@Cryptoon Thank you - I understood this earlier! it is not clear how to make sure that:
1. My application has reached the right place
2. That my claim is not controversial
3. Where to expect feedback - my discors are indicated in the application - or should I just wait as a positive sign for a mini transaction to my compromised address? and it will happen before the 15th of the month? Thank!
Or can I send a small amount to the old compromised address now and then from that address, for example, send a transaction to the new VERUS to prove ownership of the private key and seeds?
jl777c07-Jun-19 11:09 AM
no
wait for the special transaction. if you try to make a false one, your account will go into the disputed category


The Dawn box might be one of the most recognisable icons in Kenya.
Whichever way you slice it, The Dawn is an industry leader in the quick service pizza game.
The first Slice of Heaven outlet opened its doors in 2007 in Nairobi, and rose quickly in popularity.
Serving hot oven-fresh pizza with its signature all-natural tomato sauce and using only the freshest ingredients, The Dawn has established itself at the top of the industry and is easily one of the first choices in quick service pizza in Kenya.


---
layout: single
author_profile: false
author: mark
title: "Submit Your Bach Bracket Picks"
permalink: /bachelor-submit/
---
Drag the contestants in the order you think they will be told "thanks but no thanks".
- Top of the list = contestant who makes it to the end
- Bottom of the list = contestant you think will go home first
Good luck!
($10 USD / entry. Entries are due no later than Monday May 20 @ 8PM EDT)