{% extends "template.html" %} {% block main %}
Let's Encrypt is a certificate authority that will grant you free SSL certificates if you can prove you own a domain name. The protocol for granting a cert in this way is called the ACME protocol, and with an optional Mu Server add-on you can automate SSL certificate creation and renewal with any ACME-based certificate authority.
Note that unlike many Let's Encrypt integrations, you do not need to install any cert-bots or pre-configure your server in anyway. Instead, just configure DNS and deploy your code.
Assuming you have a domain such as your-domain.example.org
you need to config
the DNS settings of that domain to point to your server.
Note that the ACME integration is built on top of the acme4j library, which uses other dependencies such as Bouncy Castle. The mu-acme dependency brings in all the extra dependencies needed.
<dependency>
<groupId>io.muserver</groupId>
<artifactId>mu-server</artifactId>
<version>{{version}}</version>
</dependency>
<dependency>
<groupId>io.muserver</groupId>
<artifactId>mu-acme</artifactId>
<version>{{acmeversion}}</version>
</dependency>
Note: the mu-acme source code and issue tracker is on Github.
In your startup code, you need to create a cert manager telling it which ACME service to use, where to write the certs and other files, and which domain name you require.
The following example uses the Let's Encrypt staging server and starts a simple Hello World
web server. If this is deployed on a server that the your-domain.example.org
domain resolves to, then a free SSL cert will be acquired and automatically renewed every
three months.
The AcmeCertManager needs to know the address of an ACME server. Convenience methods for the Let's Encrypt staging and production services are predefined, but any ACME service can be used:
AcmeCertManagerBuilder.acmeCertManager()
.withAcmeServerURI(URI.create("ACME server URI"));
The config dir is where mu-acme will write various files. It is recommended that you back up this directory and keep it secure as it contains your ACME user key, domain private key, and the actual server certificate.
With the cert manager built, you can start a server. Note that most ACME providers require you to have an HTTP port open on port 80. You can directly open port 80 like in the example above or use something like IP Tables to redirect port 80 to another port.
The cert manager will provide the SSL context. The first time you start the server, there is
no cert available, and a self-signed cert is temporarily used until one is acquired (which is
typically within a few seconds). On subsequent server restarts, the cert from the configDir
is used.
The first handler you add to your server should be the handler from the cert manager. This is used by the library to prove that you own the domain and will not have any other effect.
Finally, after starting the server, you need to start the cert manager. This will start a periodic check of the cert validity. If the cert is due to expire within 3 days, then a new cert is acquired and Mu Server will start using the new cert. No restart or manual intervention required.
Mu Server provides a handler which can redirect all traffic to HTTPS and set HSTS headers. Just add the following handler AFTER the acme handler when building the server:
.addHandler(certManager.createHandler())
.addHandler(
HttpsRedirectorBuilder.toHttpsPort(443)
.withHSTSExpireTime(365, TimeUnit.DAYS)
.includeSubDomains(true)
)
The AcmeCertManagerBuilder
has a disable(boolean)
method.
If you pass true to this method
when building the manager, then a no-op manager will be returned. You can then use the cert
manager as if it was a real one, however no certs will be requested and a self-signed cert
will be used.
If packaged as an uber jar, you will need to exclude some files.
The following is an example using the maven-shade-plugin
:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.4.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.example.yourapp.App</mainClass>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
{% endblock %}


After each compression run (when obfuscation is enabled) JSCruncher assembles a special map which contains every token substitution it made. They are always displayed in a pane along the bottom of the application called the 'session map pane'.
When you are first crafting map files for your unique project, this pane is invaluable. With it, you can quickly determine what tokens are being obfuscated. If you decide any of the tokens need to remain intact, you can either manually add them to your map file(s), or simply right-click them and select "Add to user defined map" which places them in the user-defined map file.
The session map is also saved within your log files. After compressing, click on View --> Log to access it. The logs also show which files are using which tokens.
The session map can also be exported using the button on the pane or by going to File --> Export session map. This command saves the session map to a .jmap file.
note: care should be taken that the public does not get ahold of your map files.
They could possibly be used to reverse-engineer your obfuscated files.


Authentication for Amfphp.
This plugin can be deactivated if the project doesn't need to protect access to its services.
On a service object, the plugin looks for a method called _getMethodRoles. If the method exists, the plugin will look for a role in the session that matches the role. If the roles don't match, an Exception is thrown. The _getMethodRoles takes a parameter $methodName, and must return an array of strings containing acceptable roles for the method. If the return value is null, it is considered that that particular method is not protected.
For example:
To authenticate a user, the plugin looks for a 'login' method. This method can either be called explicitly, or by setting a header with the name 'Credentials', containing {userid: userid, password: password}, as defined by the AS2 NetConnection.setCredentials method. It is considered good practise to have a 'logout' method, though this is optional The login method returns a role in a 'string'. It takes 2 parameters, the user id and the password. The logout method should call AmfphpAuthentication::clearSessionInfo();
See the AuthenticationService class in the test data for an example of an implementation.
Located in /Amfphp/Plugins/AmfphpAuthentication/AmfphpAuthentication.php (line 45)
clears the session info set by the plugin. Use to logout
constructor.
looks for a match between the user roles and the accepted roles
called when the service object is created, just before the method call.
Tries to authenticate if a credentials header was sent in the packet. Throws an exception if the roles don't match
looks for a 'Credentials' request header. If there is one, uses it to try to authentify the user.
Documentation generated on Tue, 29 May 2012 23:41:57 +0200 by phpDocumentor 1.4.3


--- layout: page title: R Links permalink: /R/ ---
This course provides an introduction to the fundamentals of the R language and its applications to data analysis.
This course provides a brief introduction to the fundamentals of the R language and focuses on its use for data analysis--including exploratory data analysis, linear and logistic regression, variable selection, model diagnostics, and prediction.
A workshop on Bayesian Analysis with R-Stan, report generation with RMarkdown, and building interactive apps using RShiny.


public class RunnableQueue
extends java.lang.Object
implements java.lang.Runnable
| Modifier and Type | Class and Description |
|---|---|
static interface |
RunnableQueue.IdleRunnable
A
Runnable that can also inform the caller how long it should
be until it is run again. |
protected static class |
RunnableQueue.Link
To store a Runnable.
|
protected static class |
RunnableQueue.LockableLink
To store a Runnable with an object waiting for him to be executed.
|
static interface |
RunnableQueue.RunHandler
This interface must be implemented by an object which wants to
be notified of run events.
|
static class |
RunnableQueue.RunHandlerAdapter
This is an adapter class that implements the RunHandler interface.
|
static class |
RunnableQueue.RunnableQueueState
Type-safe enumeration of queue states.
|
| Modifier and Type | Field and Description |
|---|---|
protected int |
preemptCount
Count of preempt entries in queue, so preempt entries
can be kept properly ordered.
|
protected RunnableQueue.RunHandler |
runHandler
The object which handle run events.
|
protected HaltingThread |
runnableQueueThread
The current thread.
|
static RunnableQueue.RunnableQueueState |
RUNNING
The queue is in the process of running tasks.
|
protected RunnableQueue.RunnableQueueState |
state
The Suspension state of this thread.
|
protected java.lang.Object |
stateLock
Object to synchronize/wait/notify for suspension
issues.
|
static RunnableQueue.RunnableQueueState |
SUSPENDED
The queue is no longer running any tasks and will not
run any tasks until resumeExecution is called.
|
static RunnableQueue.RunnableQueueState |
SUSPENDING
The queue may still be running tasks but as soon as possible
will go to SUSPENDED state.
|
protected boolean |
wasResumed
Used to indicate if the queue was resumed while
still running, so a 'resumed' event can be sent.
|
| Constructor and Description |
|---|
RunnableQueue() |
| Modifier and Type | Method and Description |
|---|---|
static RunnableQueue |
createRunnableQueue()
Creates a new RunnableQueue started in a new thread.
|
protected void |
executionResumed()
Called when execution is being resumed.
|
protected void |
executionSuspended()
Called when execution is being suspended.
|
java.lang.Object |
getIteratorLock()
Returns iterator lock to use to work with the iterator
returned by iterator().
|
RunnableQueue.RunnableQueueState |
getQueueState() |
RunnableQueue.RunHandler |
getRunHandler()
Returns the RunHandler or null.
|
HaltingThread |
getThread()
Returns the thread in which the RunnableQueue is currently running.
|
void |
invokeAndWait(java.lang.Runnable r)
Waits until the given Runnable's
run() has returned. |
void |
invokeLater(java.lang.Runnable r)
Schedules the given Runnable object for a later invocation, and
returns.
|
java.util.Iterator |
iterator()
Returns an iterator over the runnables.
|
void |
preemptAndWait(java.lang.Runnable r)
Waits until the given Runnable's
run() has returned. |
void |
preemptLater(java.lang.Runnable r)
Schedules the given Runnable object for a later invocation, and
returns.
|
void |
resumeExecution()
Resumes the execution of this queue.
|
void |
run()
Runs this queue.
|
protected void |
runnableInvoked(java.lang.Runnable rable)
Called when a Runnable completes.
|
protected void |
runnableStart(java.lang.Runnable rable)
Called just prior to executing a Runnable.
|
void |
setIdleRunnable(RunnableQueue.IdleRunnable r)
Sets a Runnable to be run whenever the queue is empty.
|
void |
setRunHandler(RunnableQueue.RunHandler rh)
Sets the RunHandler for this queue.
|
void |
suspendExecution(boolean waitTillSuspended)
Suspends the execution of this queue after the current runnable
completes.
|
public static final RunnableQueue.RunnableQueueState RUNNING
public static final RunnableQueue.RunnableQueueState SUSPENDING
public static final RunnableQueue.RunnableQueueState SUSPENDED
protected volatile RunnableQueue.RunnableQueueState state
protected final java.lang.Object stateLock
protected boolean wasResumed
protected int preemptCount
protected RunnableQueue.RunHandler runHandler
protected volatile HaltingThread runnableQueueThread
public static RunnableQueue createRunnableQueue()
run()
method.public void run()
run
in interface java.lang.Runnable
public HaltingThread getThread()
run()
method.public void invokeLater(java.lang.Runnable r)
java.lang.IllegalStateException
- if getThread() is null.public void invokeAndWait(java.lang.Runnable r) throws java.lang.InterruptedException
run()
has returned.
Note: invokeAndWait()
must not be called from the
current thread (for example from the run()
method of the
argument).java.lang.IllegalStateException
- if getThread() is null or if the
thread returned by getThread() is the current one.java.lang.InterruptedException
public void preemptLater(java.lang.Runnable r)
java.lang.IllegalStateException
- if getThread() is null.public void preemptAndWait(java.lang.Runnable r) throws java.lang.InterruptedException
run()
has returned.
The given runnable preempts any runnable that is not currently
executing (ie the next runnable started will be the one given).
Note: preemptAndWait()
must not be called from the
current thread (for example from the run()
method of the
argument).java.lang.IllegalStateException
- if getThread() is null or if the
thread returned by getThread() is the current one.java.lang.InterruptedException
public RunnableQueue.RunnableQueueState getQueueState()
public void suspendExecution(boolean waitTillSuspended)
waitTillSuspended
- if true this method will not return
until the queue has suspended (no runnable in progress
or about to be in progress). If resumeExecution is
called while waiting will simply return (this really
indicates a race condition in your code). This may
return before an associated RunHandler is notified.java.lang.IllegalStateException
- if getThread() is null.public void resumeExecution()
java.lang.IllegalStateException
- if getThread() is null.public java.lang.Object getIteratorLock()
public java.util.Iterator iterator()
public void setRunHandler(RunnableQueue.RunHandler rh)
public RunnableQueue.RunHandler getRunHandler()
public void setIdleRunnable(RunnableQueue.IdleRunnable r)
protected void executionSuspended()
protected void executionResumed()
protected void runnableStart(java.lang.Runnable rable)
rable
- The runnable that is about to startprotected void runnableInvoked(java.lang.Runnable rable)
rable
- The runnable that just completed.Copyright © 2022 Apache Software Foundation. All Rights Reserved.


Contents |
All homework must be submitted in report format. A series of equations and “solutions” is insufficient. Each report should introduce the problem and any assumptions, detail the methodology, and discuss the significance and limitations of the results and conclusions. Use full sentences, not bullet points. Imagine your audience to be a colleague or supervisor—someone who understands your field but is not necessarily familiar with the details of your particular problem. Your report should not only be technically correct, but should also be written clearly and follow best practices in technical writing.
Each report has a page limit of 10 pages including any figures, tables, references, and appendices. A title page is not necessary, but be sure to write your name on the first page. I recommend using LaTeX to produce your reports (more on this below), but you are also welcome to use Microsoft Word and MS Equation Editor or any other typesetting/word-processing program that you prefer.
Why a report format? You will learn deeper by having to explain in words, it will better prepare you for the type of work you will have in an engineering career, and the hard work you did will be much more accessible to you when referring to it later.
You will also need to submit any source code that you developed. You should not include source code directly in your report, but you may refer to it from your report if necessary. Source code should be commented so that it is easily readable.
Each submission should consist of one PDF file for the report and one zipped directory containing any auxiliary material (e.g., source code). On Learning Suite you can upload multiple items, just upload them one right after the other (nothing is overwritten even though it only shows you your last submitted file). If you are really worried about it, you can upload everything in one zip, but it makes it easier for us to grade if the PDF is separate because then we can use the build-in PDF viewer in Learning Suite.
LaTeX is widely used in scientific writing and if you are doing graduate work you will likely need to learn it at some point (but check with your advisor, I know that a few research groups prefer MS Word). To be clear, there is nothing wrong with using Microsoft Word (or an equivalent) for your report preparation. These are just tools, what’s more important is doing good work.
If you want to use LateX, I recommend you use Overleaf unless you already have a workflow you are comfortable with. Think of it as Google Docs for LaTeX. There isn’t anything to install, and multiple people can work collaboratively (nice for your project). It also provides a lot of starter templates, in fact I’ve created a simple overleaf example for you that illustrates most of the commands you will use. Many LaTeX tutorials/guides exist online if you need to do anything fancier.


Welcome to pspso’s documentation!¶
Overview and Installation¶
Overview¶
pspso is a python library for selecting machine learning algorithms parameters. The first version supports two single algorithms: Multi-Layer Perceptron (MLP) and Support Vector Machine (SVM). It supports two ensembles: Extreme Gradient Boosting (XGBoost) and Gradient Boosting Decision Trees (GBDT).
Two types of machine learning tasks are supported by pspso:
Regression.
Binary classification.
Three scores are supported in the first version of pspso:
Regression :
Root Mean Square Error (RMSE)
Binary Classication :
Area under the Curve (AUC) of the Receiver Operating Characteristic (ROC)
Accuracy
Usage¶
MLP Example (Binary Classification)¶
pspso is used to select the machine learning algorithms parameters. Below is an example for using the pspso to select the parameters of the MLP. pspso handles the MLP random weights intialization issue that may cause losing the best solution in consecutive iterations.
The following example demonstrates the selection process of the MLP parameters. A variable named params was not given by the user. Hence, the default search space of the MLP is loaded. This search space contains five parameters:
params = {"optimizer": ["RMSprop", "adam", "sgd",'adamax','nadam','adadelta'] ,
"learning_rate": [0.1,0.3,2],
"neurons": [1,40,0],
"hiddenactivation": ['relu','sigmoid','tanh'],
"activation":['relu','sigmoid','tanh']}
The task and the score were defined as binary classification and auc respectively. Then, the PSO was used to select the parameters of the MLP. Results are provided back to the user through the print_results() function.
from sklearn.preprocessing import MinMaxScaler
from pspso import pspso
from sklearn import datasets
from sklearn.model_selection import train_test_split
breastcancer = datasets.load_breast_cancer()
data=breastcancer.data#get the breast cancer dataset input features
target=breastcancer.target# target
X_train, X_test, Y_train, Y_test = train_test_split(data, target,test_size=0.1,random_state=42,stratify=target)
normalize = MinMaxScaler(feature_range=(0,1))#normalize input features
X_train=normalize.fit_transform(X_train)
X_test=normalize.transform(X_test)
X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train,test_size=0.15,random_state=42,stratify=Y_train)
p=pspso(estimator='mlp',task='binary classification', score='auc')
pos,cost,duration,model,optimizer=p.fitpspso(X_train,Y_train,X_val,Y_val)
p.print_results()#print the results
testscore=pspso.predict(p.model,p.estimator,p.task,p.score, X_test, Y_test)
print(1-testscore)
In this example, four parameters were examined: optimizer, learning_rate, hiddenactivation, and activation. The number of neurons in the hidden layer was kept as default.
Output:
Estimator: mlp
Task: binary classification
Selection type: PSO
Number of attempts:50
Total number of combinations: 45360
Parameters:
{'optimizer': 'nadam', 'learning_rate': 0.29, 'neurons': 4, 'hiddenactivation': 'sigmoid', 'activation': 'sigmoid'}
Global best position: [3.8997699 0.28725911 4.21218138 1.41200923 0.84643591]
Global best cost: 0.0
Time taken to find the set of parameters:<PHONE_NUMBER>681183
Number of particles: 5
Number of iterations: 10
0.9867724867724867
XGBoost Example (Binary Classification)¶
from sklearn.preprocessing import MinMaxScaler
from pspso import pspso
from sklearn import datasets
from sklearn.model_selection import train_test_split
breastcancer = datasets.load_breast_cancer()
data=breastcancer.data#get the breast cancer dataset input features
target=breastcancer.target# target
X_train, X_test, Y_train, Y_test = train_test_split(data, target,test_size=0.1,random_state=42,stratify=target)
normalize = MinMaxScaler(feature_range=(0,1))#normalize input features
X_train=normalize.fit_transform(X_train)
X_test=normalize.transform(X_test)
X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train,test_size=0.15,random_state=42,stratify=Y_train)
params = {
"learning_rate": [0.01,0.2,2],
"max_depth": [1,10,0],
"n_estimators": [2,200,0],
"subsample": [0.7,1,1]}
p=pspso(estimator='xgboost',params=params,task='binary classification', score='auc')
pos,cost,duration,model,optimizer=p.fitpspso(X_train,Y_train,X_val,Y_val)
p.print_results()#print the results
testscore=pspso.predict(p.model,p.estimator,p.task,p.score, X_test, Y_test)
print(1-testscore)
XGBoost Example (Regression)¶
The XGBoost is an implementation of boosting decision trees. Five parameters were utilized for selection: objective, learning rate, maximum depth, number of estimators, and subsample. Three categorical values were selected for the objective parameter. The learning rate parameter values range between 0.01 and 0.2 with 2 decimal point, maximum depth ranges between 1 and 10 with 0 decimal points (1,2,3,4,5,6,7,8,9,10), etc. The task and score are selected as regression and RMSE respectively. The number of particles and number of iterations can be left as default values if needed. Then, a pspso instance is created. By applying the fitpspso function, the selection process is applied. Finally, results are printed back to the user. The best model, best parameters, score, time, and other details will be saved in the created instance for the user to check.
from sklearn.preprocessing import MinMaxScaler
from pspso import pspso
from sklearn import datasets
from sklearn.model_selection import train_test_split
boston_data = datasets.load_boston()
data=boston_data.data
target=boston_data.target
X_train, X_test, Y_train, Y_test = train_test_split(data, target,test_size=0.1,random_state=42)
normalize = MinMaxScaler(feature_range=(0,1))#normalize input features
normalizetarget = MinMaxScaler(feature_range=(0,1))#normalize target
X_train=normalize.fit_transform(X_train)
X_test=normalize.transform(X_test)
Y_train=normalizetarget.fit_transform(Y_train.reshape(-1,1))
Y_test=normalizetarget.transform(Y_test.reshape(-1,1))
X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train,test_size=0.25,random_state=42)
params = {
"objective":['reg:tweedie',"reg:linear","reg:gamma"],
"learning_rate": [0.01,0.2,2],
"max_depth": [1,10,0],
"n_estimators": [2,200,0],
"subsample": [0.7,1,1]}
p=pspso(estimator='xgboost',params=params,task='regression', score='rmse')
pos,cost,duration,model,optimizer=p.fitpspso(X_train,Y_train,X_val,Y_val)
p.print_results()#print the results
testscore=pspso.predict(p.model,p.estimator,p.task,p.score, X_test, Y_test)
print(testscore)
User Input¶
The user is required to select the type of the algorithm (‘mlp’, ‘svm’, ‘xgboost’, ‘gbdt’); the task type (‘binary classification’,’regression’), score (‘rmse’, ‘acc’, or ‘auc’). The user can keep the parameters variable empty, where a default set of parameters and ranges is loaded for each algorithm.
from pspso import pspso
task='binary classification'
score='auc'
p=pspso.pspso('xgboost',None,task,score)
Pspso allows the user to provide a range of parameters for exploration. The parameters vary between each algorithm. Any parameter supported by the Scikit-Learn API for GBDT and XGBoost can be added to the selection process. A set of parameters that contains five XGBoost parameters is shown below. The parameters are encoded in JSON object that consists of key,value pairs:
params = {"objective":['reg:tweedie',"reg:linear","reg:gamma"],
"learning_rate": [0.01,0.2,2],
"max_depth": [1,10,0],
"n_estimators": [2,200,0],
"subsample": [0.7,1,1]}
The key can be any parameter belonging to to the algorithm under investigation. The value is a list. Pspso will check the type of the first element in the list, which will determine if the values of the parameter are categorical or numerical.
Categorical Parameters
If the parameter values are categorical, string values are expected to be found in the list, as shown in objective parameter. The values in the list will be automatically mapped into a list of integers, where each integer represents a value in the original list. The order of the values inside the list affect the position of the value in the search space.
Numerical Parameters
If the parameter is numerical, a list of three elements [lb,ub, rv] is expected to be found:
lb: repesents the lowest value in the search space
ub: represents the maximum value in the search space
rv: represents the number of decimal points the parameter values are rounded to before being added for training the algorithm
For e.g if you want pspso to select n_estimators, add the following list [2,200,0]. By that, the lowest n_estimators will be 2, the highest to be examined is 200, and each possible value is rounded to an integer value ( 0 decimal points).
Other parameters
The user is given the chance to handle some of the default parameters such as the number of epochs in the MLP. Although this parameter can be optimized, but its not encouraged. The user can modify this by changing a pspso class instance. For e.g., to change the number of epochs from default to 10 in MLP training:
from pspso import pspso
task='binary classification'
score='auc'
p=pspso.pspso('mlp',None,task,score)# in case of empty set of params (None) default search space is loaded
p.defaultparams['epochs']=10
The verbosity can be modified for any algorithm, which allows showing details of the training process:
from pspso import pspso
task='binary classification'
score='auc'
p=pspso.pspso('mlp',None,task,score)
p.verbosity=1
Early stopping rounds can alos be modified, the user can set a value different to the default value:
from pspso import pspso
task='binary classification'
score='auc'
p=pspso.pspso('xgboost',None,task,score)
p.early_stopping=10
Other parameters such that n_jobs in XGBoost can also be modified before the start of the selection process.
Functions¶
ML Algorithms Functions¶
|
Train the GBDT after decoding the parameters in variable particle. |
|
Train the XGBoost after decoding the parameters in variable particle. |
|
Train the SVM after decoding the parameters in variable particle. |
|
Train the MLP after the decoding the parameters in variable particle. |
Selection Functions¶
|
Select the algorithm parameters based on PSO. |
|
Select the algorithm parameters based on Grid search. |
|
Select the algorithm parameters based on radnom search. |
The fitpsrandom() and fitpsgrid() were implmented as two default selection methods. With fit random search, the number of attempts to be tried is added by the user as a variable. In grid search, all the possible combinations are created and investigated by the package. These functions follow the same encoding schema used in fitpspso(), and were basically added for comparison.
Parameters Functions¶
|
Read the parameters provided by the user. |
|
Decodes the parameters of a list into a meaningful set of parameters. |
|
Set the default parameters of the estimator. |
|
Create a dictionary of default parameters if the user didnt provide parameters. |
Other Functions¶
|
Higher-level method to do forward_prop in the whole swarm. |
|
Used to rebuild the model after selecting the parameters. |
Print the results found in the pspso instance. |
|
A function that will generate all the possible combinations in the search space. |
|
|
A function used to release the score of a model. |
Module Summary¶
-
class
pspso.
pspso
(estimator='xgboost', params=None, task='regression', score='rmse')¶ This class searches for algorithm parameters by using the Particle Swarm Optimization (PSO) algorithm.
-
calculatecombinations
()¶ A function that will generate all the possible combinations in the search space. Used mainly with grid search
Returns
- combinations: list
A list that contains all the possible combinations.
-
static
decode_parameters
(particle)¶ Decodes the parameters of a list into a meaningful set of parameters. To decode a particle, we need the following global variables:parameters, defaultparameters, paramdetails, and rounding.
-
static
f
(q, estimator, task, score, X_train, Y_train, X_val, Y_val)¶ Higher-level method to do forward_prop in the whole swarm.
Inputs
- x: numpy.ndarray of shape (n_particles, dimensions)
The swarm that will perform the search
Returns
- numpy.ndarray of shape (n_particles, )
The computed loss for each particle
-
fitpsgrid
(X_train=None, Y_train=None, X_val=None, Y_val=None)¶ Select the algorithm parameters based on Grid search.
Grid search was implemented to match the training process with pspso and for comparison purposes. I have to traverse each value between x_min, x_max. Create a list seperating rounding value.
-
fitpspso
(X_train=None, Y_train=None, X_val=None, Y_val=None, psotype='global', number_of_particles=5, number_of_iterations=10, options={'c1': 1.49618, 'c2': 1.49618, 'w': 0.7298})¶ Select the algorithm parameters based on PSO.
Inputs
- X_train: numpy.ndarray of shape (a,b)
Contains the training input features, a is the number of samples, b is the number of features
- Y_train: numpy.ndarray of shape (a,1)
Contains the training target, a is the number of samples
- X_train: numpy.ndarray of shape (c,b)
Contains the validation input features, c is the number of samples, b is the number of features
- Y_train: numpy.ndarray of shape (c,1)
Contains the training target, c is the number of samples
- number_of_particles: integer
number of particles in the PSO search space.
- number_of_iterations: integer
number of iterations.
- options: dictionary
A key,value dict of PSO parameters c1,c2, and w
Returns
- pos: list
The encoded parameters of the best solution
- cost: float
The score of the best solution
- duration: float
The time taken to conduct random search.
- model:
The best model generated via random search
- combinations: list of lists
The combinations examined during random search
- results: list
The score of each combination in combinations list
-
fitpsrandom
(X_train=None, Y_train=None, X_val=None, Y_val=None, number_of_attempts=20)¶ Select the algorithm parameters based on radnom search.
With Random search, the process is done for number of times specified by a parameter in the function.
Inputs
- X_train: numpy.ndarray of shape (a,b)
Contains the training input features, a is the number of samples, b is the number of features
- Y_train: numpy.ndarray of shape (a,1)
Contains the training target, a is the number of samples
- X_train: numpy.ndarray of shape (c,b)
Contains the validation input features, c is the number of samples, b is the number of features
- Y_train: numpy.ndarray of shape (c,1)
Contains the training target, c is the number of samples
- number_of_attempts: integer
The number of times random search to be tried.
Returns
- pos: list
The encoded parameters of the best solution
- cost: float
The score of the best solution
- duration: float
The time taken to conduct random search.
- model:
The best model generated via random search
- combinations: list of lists
The combinations examined during random search
- results: list
The score of each combination in combinations list
-
static
forward_prop_gbdt
(particle, task, score, X_train, Y_train, X_val, Y_val)¶ Train the GBDT after decoding the parameters in variable particle. The particle is decoded into parameters of the gbdt. Then, The gbdt is trained and the score is sent back to the fitness function.
Inputs
- particle: list of values (n dimensions)
A particle in the swarm
- task: regression, binary classification
the task to be conducted
- score: rmse (regression), auc (binary classification), acc (binary classification)
the type of evaluation
- X_train: numpy.ndarray of shape (m, n)
Training dataset
- Y_train: numpy.ndarray of shape (m,1)
Training target
- X_val: numpy.ndarray of shape (x, y)
Validation dataset
- Y_val: numpy.ndarray of shape (x,1)
Validation target
Returns
- variable, model
the score of the trained algorithm over the validation dataset, trained model
-
static
forward_prop_mlp
(particle, task, score, X_train, Y_train, X_val, Y_val)¶ Train the MLP after the decoding the parameters in variable particle.
-
static
forward_prop_svm
(particle, task, score, X_train, Y_train, X_val, Y_val)¶ Train the SVM after decoding the parameters in variable particle.
-
static
forward_prop_xgboost
(particle, task, score, X_train, Y_train, X_val, Y_val)¶ Train the XGBoost after decoding the parameters in variable particle. The particle is decoded into parameters of the XGBoost. This function is similar to forward_prop_gbdt The gbdt is trained and the score is sent back to the fitness function.
Inputs
- particle: list of values (n dimensions)
A particle in the swarm
- task: regression, binary classification
the task to be conducted
- score: rmse (regression), auc (binary classification), acc (binary classification)
the type of evaluation
- X_train: numpy.ndarray of shape (m, n)
Training dataset
- Y_train: numpy.ndarray of shape (m,1)
Training target
- X_val: numpy.ndarray of shape (x, y)
Validation dataset
- Y_val: numpy.ndarray of shape (x,1)
Validation target
Returns
- variable, model
the score of the trained algorithm over the validation dataset, trained model
-
static
get_default_params
(estimator, task)¶ Set the default parameters of the estimator. This function assigns the default parameters for the user. Each algorithm has a set of parameters. To allow the user to search for some parameters instead of the supported parameters, this function is used to assign a default value for each parameter. In addition, it gets other parameters for each algorithm. For e.g, it returns the number of epochs, batch_size, and loss for the mlp.
Inputs
- estimator: string value
A string value that determines the estimator: ‘mlp’,’xgboost’,’svm’, or ‘gbdt’
- task: string value
A string value that determines the task under consideration: ‘regression’ or ‘binary classification’
Returns
- defaultparams: Dictionary
A dictionary that contains default parameters to be used.
-
static
get_default_search_space
(estimator, task)¶ Create a dictionary of default parameters if the user didnt provide parameters.
Inputs
- estimator: string value
A string value that determines the estimator: ‘mlp’,’xgboost’,’svm’, or ‘gbdt’
- task: string value
A string value that determines the task under consideration: ‘regression’ or ‘binary classification’
Returns
- params: Dictionary
A dictionary that contains default parameters to be used.
-
static
predict
(model, estimator, task, score, X_val, Y_val)¶ A function used to release the score of a model. If the score is rmse, the value is released. If the score is acc (accuracy), 1-acc is returned back since pso applies a minimization task. If the score is auc, 1-auc is returned back since pso applies a minization task
This class is static and can be used to test the model accuracy over the hold-out sample once the selection process is finalized.
Inputs
- model:
A trained model
- estimator: string value
A string value that determines the estimator: ‘mlp’,’xgboost’,’svm’, or ‘gbdt’
- task: string value
A string value that determines the task under consideration: ‘regression’ or ‘binary classification’
- score: string value
Determines the score (‘rmse’,’auc’,’acc’)
- X_val: numpy.ndarray
Input features
- Y_val: numpy.ndarray
Target
Returns
- met: float
Score value of the model
-
print_results
()¶ Print the results found in the pspso instance. Expected to print general details like estimator, task, selection type, number of attempts examined, total number of combinations, position of the best solution, score of the best solution, parameters, details about the pso algorithm.
-
static
read_parameters
(params=None, estimator=None, task=None)¶ Read the parameters provided by the user.
Inputs
- params: dictionary of key,values added by the user
This dictionary determines the parameters and ranges of parameters the user wants to selection values from.
- estimator: string value
A string value that determines the estimator: ‘mlp’,’xgboost’,’svm’, or ‘gbdt’
- task: string value
A string value that determines the task under consideration: ‘regression’ or ‘binary classification’
Returns
- parameters
The parameters selected by the user
- defaultparams
Default parameters
- x_min: list
The lower bounds of the parameters search space
- x_max: list
The upper bounds of the parameters search space
- rounding: list
The rounding value in each dimension of the search space
- bounds: dict
A dictionary of the lower and upper bounds
- dimensions: integer
Dimensions of the search space
- params: Dict
Dict given by the author
-
static
rebuildmodel
(estimator, pos, task, score, X_train, Y_train, X_val, Y_val)¶ Used to rebuild the model after selecting the parameters.
-
Future Work¶
New Algorithms¶
Other machine learning algorithms and packages will be added such as the catboost.
Cross Validation¶
We are working towards adding the cross validation support that will take the training data and number of folds.
Then split the records and train each fold. The average performance of cross-validation will be retuned back to the user.
Multi-Class Classification¶
We are also working on adding multi-class classification and data oversampling techniques.
Steps for adding another machine learning algorithm¶
The main reason behind the development of this package is to facilitate the use of the algorithms with a minimum amount of code required. However, the steps to add an algorithm are followed:
Step 1: Add a condition in the get_default_search_space() function to include the new algorithm default search space parameters with upper/lower bounds
Step 2: Add a default search space based on the algorithm and the task (binary classification or regression) to the get_default_params() function
Step 3: Create a function forward_prop_algorithmname() that accepts parameters (similar to forward_prop_gbdt,forward_prop_svm) and returns two variables: the model and fitness value
Step 4: Add a condition in the function f() to forward the task to the function created in Step 3
Step 5: Add a condition in the function predict() to allow building the model using the function created in Step 3
Contributing¶
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.
We are working towards adding the cross validation support that will take the training data and number of folds, then split the records and train each fold. Finally, the average performance is retuned to the user.
We are also working on adding multi-class classification and data oversampling techniques.
License¶
Copyright (c) [2020] [Ali Haidar]
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


Ike shuts down troll for telling his friend to hook Mercy up with another guy
- A video is currently making the rounds online that captured the moment Ike reiterated his love and commitment to Mercy
- The reality star made his position known during a live video session of his close friend on Instagram,
- Ike said Mercy is not going anywhere and the trolls can keep trying to get between them
Just like ex Big Brother Naija(BBNaija) stars Bambam and Teddy A who are now a couple, it appears two lovers in the 2019 season of the reality show, Mercy and Ike, may be into one another more than people may know.
Recall that the pair had given viewers mixed feeling with their affair back when they were still in the competition. However, doubts of their love for each other has since been dismissed, with the way they dote over one another.
PAY ATTENTION: Read best news on Nigeria's #1 news app
Celebrity OAP Do2dtun shows off his newly completed luxury mansion in Lagos (photo)
Just recently, a video surfaced on social media that captured the moment Ike responded to an internet troll that tried to come in the way of his feelings for his curvy woman.
PAY ATTENTION: Get your daily relationship tips and advice on Africa Love Aid group
Ike had featured on a live video session on a friend’s Instagram account, and among those that watch the video, a particular user had pointed out that guys that are named Ike are known to be cheats, as such, the reality star’s friend should hook Mercy up with someone else.
The friend immediately responded saying, Ike isn’t a cheat and would never think of cheating on Mercy.
This was followed by Ike pointing out that Mercy is not going anywhere because they love each other. He said internet trolls can continue making efforts to come in between them
Watch the video below:
Davido pens touching note as he reminisces on his life in the past 7 years
PAY ATTENTION: NAIJ.com upgrades to Legit.ng: a letter from our Editor-in-Chief Bayo Olupohunda
Meanwhile, it seems not everyone is okay with how supportive Ike has been of Mercy. Legit.ng previously gathered the report of a Twitter user, who came out to say Ike needs to focus on building his brand and stop following Mercy everywhere.
NAIJ.com (naija.ng) -> Legit.ng We have upgraded to serve you better
BBNaija Exclusive: Mercy reveals whether her body helped her win the show | Legit TV


- Develop a Deployment/Transition Strategy:
-
Web Application: When transitioning a web application, it's essential to plan for server provisioning, database setup, security configurations, and handling the migration of user data. Additionally, testing the application on
different browsers and devices should be included in the strategy.
-
Web Service: For web services, focus on defining APIs, authentication mechanisms, and data formats. Ensure compatibility with various client applications and plan for scalability and load testing to accommodate potential spikes
in usage.
- Database: Transitioning a database requires careful planning for data migration, schema changes, and indexing. Backup and recovery procedures must be in place to avoid data loss during the transition process.
- Batch Job: For batch jobs, consider scheduling and automation of job execution. Error handling and logging mechanisms should be established to monitor job statuses effectively.
- ETL (Extract, Transform, Load): In the case of ETL processes, ensure data sources and targets are correctly defined. Plan for data validation and transformation to maintain data integrity during the transition.
- Develop Plans for Transitioning Systems:
-
Web Application: The transition plan for a web application should include strategies for user training and communication to manage user expectations during the deployment. Additionally, consider the rollout strategy for updates
and version control to handle multiple instances of the application in various environments.
- Web Service: Clearly outline the versioning and backward compatibility strategies to avoid disrupting existing clients. Provide clear documentation for developers using the web service in their applications.
- Database: The transition plan for a database should include backup and recovery procedures, data migration strategies, and consideration of data archiving to manage database growth and performance.
- Batch Job: Include scheduling details, dependencies, and notification mechanisms in the transition plan to ensure smooth execution and monitoring of batch jobs.
- ETL (Extract, Transform, Load): Plan for data validation and reconciliation to ensure data accuracy during the transition. Address error handling and logging to identify and resolve issues during ETL execution.
- Installation:
-
Web Application: During installation, verify the web application's dependencies, configurations, and integration with web servers or cloud platforms. Address security concerns and perform penetration testing to ensure a secure
deployment.
- Web Service: Ensure that the web service is deployed on the appropriate server infrastructure with the necessary security measures in place, such as SSL/TLS encryption for secure communication.
- Database: Verify the database installation and configurations, including access controls, authentication mechanisms, and data replication or clustering for high availability.
- Batch Job: Ensure that the required resources, such as servers or virtual machines, are available for executing the batch job. Set up job scheduling and monitoring mechanisms.
- ETL (Extract, Transform, Load): Check the ETL tool installation and configurations, and validate data sources and target connections.
- Integration:
-
Web Application: Integration testing should focus on verifying interactions with external services, APIs, or databases. Additionally, ensure that user authentication and session management work as expected during integration
testing.
- Web Service: Integration testing for web services involves validating the interactions with client applications and handling various error scenarios gracefully.
- Database: Verify the integration of the database with the application or services that use it. Test data retrieval, storage, and update operations to ensure data consistency.
- Batch Job: Integration testing of batch jobs involves checking job dependencies, input/output data, and error handling during job execution.
- ETL (Extract, Transform, Load): During ETL integration testing, verify data extraction from source systems, data transformation, and loading into the target database.
- Verification and Validation (V&V):
-
Web Application: Validate the web application's user interface, functionality, and performance under different loads and usage scenarios. Security testing and penetration testing are crucial to ensure the application's
resilience to potential attacks.
- Web Service: Validate that the web service meets its API specifications and performs well under various usage patterns. Security testing, such as input validation and authentication, should be thoroughly conducted.
- Database: V&V for databases includes verifying data integrity, database consistency, and performance metrics such as query response times and index usage.
- Batch Job: Verification and validation of batch jobs involve ensuring the correct processing of input data, accurate output generation, and proper error handling.
- ETL (Extract, Transform, Load): Validate data correctness and accuracy at each stage of the ETL process, including data extraction, transformation, and loading.
- Evaluate the readiness of the system to transition into operations:
-
Web Application: Evaluate if the web application meets the defined criteria for usability, security, and performance before transitioning to production. User acceptance testing is crucial to gather feedback from stakeholders.
- Web Service: Ensure that the web service satisfies all contractual requirements and has undergone thorough testing to avoid disruptions in the operational environment.
- Database: Assess if the database has been thoroughly tested, and the required backups and recovery procedures are in place before going live.
- Batch Job: Evaluate the batch job's performance and stability during testing to ensure it is ready for deployment in the production environment.
- ETL (Extract, Transform, Load): Evaluate the ETL process to ensure data accuracy and completeness, and validate that the data is ready for operational use.
- Analyze the results of transition activities and take necessary actions:
- Web Application: Analyze user feedback and application performance data to identify areas for improvement and address any reported issues before full deployment.
- Web Service: Monitor the web service's usage and performance after deployment to identify any unexpected bottlenecks or errors that may require remediation.
- Database: Continuously monitor database performance and usage patterns to optimize its configuration and address any emerging issues.
- Batch Job: Regularly review batch job execution logs and performance metrics to identify and resolve any anomalies.
- ETL (Extract, Transform, Load): Monitor data quality and ETL process performance in the operational environment and address any issues that may arise.
Successful transition and deployment of systems require careful consideration of these activities and their specific requirements based on the system type being deployed. Thorough planning, testing, and analysis are key to ensuring
a smooth and reliable transition into operational use.


This extends the Object.prototype and watches for changes to an Object instance, aka yer model!
Note: ensure ones markup is super clean, otherwise rivets can bind more than once in IE8
{ auction.counter }
This auction is ending in { auction.remaining | time }.
Note: IE8 only calls the change event on a checkbox's when it is no longer focus, aka after clicking the checkbox click away to invoke the change


We are in the middle of cultural change in product fabrication. We are moving from age of design heros towards age of maker culture. Time, we spent on reading product reviews is going to be spent more on reading DIY (Do It Yourself) stories. Birth of Fab Labs, maker communities, easy shared instructions and advanced fabrication tools, mean easy access to build your own things to support your personal life individually. This leads into that design education should be aimed for all of us. Goal is to simplify the design process, so it will be fast and easy to adapt by everyone. This workshop is about giving basic tools for polishing your ideas in paper to shine brighter before going to prototype them. The time you spend on paper will pay you back later.
This workshop is part of the European Maker Week. Workshop will be held in English and it will be open for everyone.


An asset in Google Cloud and its temporal metadata, including the time window when it was observed and its status during that window.
Inheritance
System.Object
TemporalAsset
Implements
Google.Protobuf.IMessage<TemporalAsset>
Google.Protobuf.IMessage
System.IEquatable<TemporalAsset>
Google.Protobuf.IDeepCloneable<TemporalAsset>
Inherited Members
System.Object.ToString()
System.Object.GetHashCode()
System.Object.GetType()
System.Object.MemberwiseClone()
Namespace: Google.Cloud.Asset.V1
Assembly: Google.Cloud.Asset.V1.dll
Syntax
public sealed class TemporalAsset : IMessage<TemporalAsset>, IMessage, IEquatable<TemporalAsset>, IDeepCloneable<TemporalAsset>
Constructors
TemporalAsset()
Declaration
public TemporalAsset()
TemporalAsset(TemporalAsset)
Declaration
public TemporalAsset(TemporalAsset other)
Parameters
| Type | Name | Description |
|---|---|---|
| TemporalAsset | other |
Properties
Asset
An asset in Google Cloud.
Declaration
public Asset Asset { get; set; }
Property Value
| Type | Description |
|---|---|
| Asset |
Deleted
Whether the asset has been deleted or not.
Declaration
public bool Deleted { get; set; }
Property Value
| Type | Description |
|---|---|
| System.Boolean |
PriorAsset
Prior copy of the asset. Populated if prior_asset_state is PRESENT. Currently this is only set for responses in Real-Time Feed.
Declaration
public Asset PriorAsset { get; set; }
Property Value
| Type | Description |
|---|---|
| Asset |
PriorAssetState
State of prior_asset.
Declaration
public TemporalAsset.Types.PriorAssetState PriorAssetState { get; set; }
Property Value
| Type | Description |
|---|---|
| TemporalAsset.Types.PriorAssetState |
Window
The time window when the asset data and state was observed.
Declaration
public TimeWindow Window { get; set; }
Property Value
| Type | Description |
|---|---|
| TimeWindow |
Implements
Google.Protobuf.IMessage<T>
Google.Protobuf.IMessage
System.IEquatable<T>
Google.Protobuf.IDeepCloneable<T>


Preparation time
less than 30 mins
Cooking time
1 to 2 hours
Serves
Serves 2
A beloved classic of Belgian cooking; this is a simple, rustic family dish. The Comté cheese can be replaced by any British cheese. And, of course, omit the ham for a vegetarian version.
For the chicory, put it in a medium saucepan and add the lemon juice, sugar, salt, peppercorns and enough water to barely cover. Place on a medium heat and cover with a circle of baking paper and a lid smaller than the diameter of the pan (this is to keep the chicory submerged). Bring to the boil, then simmer for 45–60 minutes. It is important to cook the chicory slowly to remove most of its bitterness.
Lift the chicory onto a wire rack to drain and cool. Once cool, gently press out any extra moisture with a tea towel.
Preheat the oven to 190C/170C Fan/Gas 5.
For the chard, put the butter and 100ml/3½fl oz water in a saucepan on a high heat. Cook the stalks for 20 minutes, adding more water if necessary. Once tender, lift them out and cook the leaves for 3 minutes in the same water. Strain, mix the stalks and leaves and arrange in a small gratin dish.
Meanwhile, make the cheese sauce. In a small saucepan on a medium heat, melt the butter. Add the flour, whisk until smooth and cook to a nutty blonde colour. Take the pan off the heat, whisk in the milk and return to the heat. Cook for at least 4 minutes, stirring constantly with the whisk until the sauce thickens.
Add the cheese and mustard and cook for 3 minutes, stirring, until the cheese has fully melted. Remove from the heat, taste and season if required.
To assemble the gratin, fold the ham slices in half and arrange one chicory head on each slice. Pack tightly in the gratin dish on top of the chard. Spoon over the cheese sauce and sprinkle with the remaining cheese. Bake in the top of the oven for 25 minutes until golden-brown.


0 0 0 1 0 0 0 0 0 0 1 0
0 0 0 1 0 0 0 1 0 -1 0 0
0 1 0 2 0 3
0 0 1
0 0 0 1 0 0 0 1 0 -1 0 0
0 1 0
1 0 0


This is a fork of the awesome Michail Plushnikov's Lombok Plugin, in order to let Intellij recognise Mateu MDD lombok annotations.


Hello! I'm Finn, and this is my homepage. I'm a computer science student at Washington University in St. Louis, with research interests in human-computer interaction and quantum computing. Here are some things I've been working on:
At WashU , I'm currently working as a research assistant in the department of computer science and engineering. I'm working with Dr. Caitlin Kelleher to develop a model of api learning. This has involved building a logging system, designing a study with human subjects, and building an interface for manually labelling logged data.
Vl/Hcc posterIn the summer of 2019, I worked as a research assistant in the Joint Center for Quantum Information and Computer Science at the University of Maryland. There, I worked on implementing, visualizing, and improving Haah's algorithm for quantum signal processing decomposition.
Haah's algorithmIn the summer of 2018, I was a research assistant at the Human-Computer Interaction Institute of Carnegie Mellon University, working with Dr. Brad Myers. I studied api usability, focusing on discoverability of api elements and the involvement of mental models in api design, leading to two publications.
Chi poster Vl/Hcc paperAt WashU, I'm in the joint B.S./ M.S. program, so I'm currently pursuing two degrees:
I expect to complete both of these after four years, in spring 2020.
I am Head Teaching Assistant for cse 231s: Introduction to Parallel and Concurrent Programming. This has involved course organization, designing assignments, and substitute lecturing.
About this class WikiI am on the executive board for Studio Tesla, a student-run WashU volunteering group that sets up after-school stem clubs in local middle schools. As the head of the Enrichment team, I'm in charge of our curriculum development.
WebsiteI led the development of BitBrawl, an ai programming competition for high school and college students. This competition ran from May to September 2018.
WebsiteIn high school, I was head of programming for my robotics team, BadgerBots Frc Team 1306. In my senior year, our robot, Nautilus, made it to semifinals in our division at the world championship. Our code (for which I was the lead developer) is open source.
Source code Video


This document exists to extend the general install guide provide examples on how to archive emails.
Note: If you plan on importing old emails from an archive, please set up the archiver first so as to create an overlap of new emails coming in and old emails being imported. The system is designed to handle this without creating duplicate entries in the archive.
Set up a Pony Mail mail account/alias on a machine. This can be your local mail server, it can be the machine that Pony Mail is on (install sendmail or postfix etc there), or it can be any other machine with access to the ElasticSearch database that Pony Mail uses.
If this is not the machine Pony Mail was installed on, you'll need to copy the
tools/ directory from your Pony Mail installation to this machine and adjust
ponymail.cfg to point to the right place for the database. You will also need
Python 3 and the helper libraries installed
(pip3 install elasticsearch formatflowed netaddr
)
Set up a mail alias for public and private lists in /etc/aliases
or similar method,
and point them at the archiver script in tools/:
# You may need to add "--altheader delivered-to" to these commands, it varies foo-public: "|/usr/bin/python3 /path/to/tools/archiver.py" foo-private: "|/usr/bin/python3 /path/to/tools/archiver.py --private"
Once done, run newaliases
to update your alias DB.
Use the mailman UI or CLI to subscribe foo-public@ to your public lists and foo-private to your private lists. Don't worry, the contents of private lists are hidden by default till the correct AAA scripting is set up.
First, see the general introduction in the MM2 example, as this applies here as well.
Set up a dot-forward file for a public and a private alias:
.qmail-archive-public: "|/usr/bin/python3 /path/to/tools/archiver.py" .qmail-archive-private: "|/usr/bin/python3 /path/to/tools/archiver.py --private"
Use the ezmlm CLI to subscribe your new aliases to the lists:
ezmlm-sub foolist/<EMAIL_ADDRESS>ezmlm-sub secretlist/<EMAIL_ADDRESS>If you have an custom OAuth2 provider and a binary approach to private access
(either/or), you can enable private access to people by having a key/value pair
called isMember
set to true
in your JSON response from the OAuth server,
provided it is set as an authority in config.lua. This will provide anyone
defined as a member via OAuth full access to all private lists.
If you use LDAP, you can modify the LDAP queries in the example AAA file to suit your organization.
Pony Mail relies on ID generators to produce a unique document ID for each email. Depending on the setup of your installation, we recommend using one of two generators:
full
generator. This generator
uses the full message body, including MTA trails (Received headers etc), to generate an ID.cluster
generator. This will use as many unique traits in an email as possible, but
discard the MTA trail (which differs from machine to machine).We do not recommend using the older generators (medium
and short
)
for new installations, as they are theoretically prone to create
duplicate IDs (See issues #176 #177 #178).
Should you need to import HTML-only emails into the archive, you may enable this
with the --html2text
command line arg. This requires that the html2text
Python3 package
is installed beforehand.
If you need to rewrite list IDs on the fly as emails come in, you can use the debug.cropout
setting for this (in ponymail.cfg
).
You can either use it to just crop away something:
[debug] # Remove 'foo' from all list IDs cropout: foo
Or you can use it as a regex substitution:
[debug] #Replace '.bar.tld' with '.blorg.bar.tld' cropout: <([a-z]+).bar.tld> <\1.blorg.bar.tld>
Apache Pony Mail (Incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.


Barbaric Attack on Student Hours to Uhuru's Event
by Collins NyamungaA student from Kibabii University was brutally attacked by an unknown assailant in the wee hours of Friday, November 22 morning ahead of the institution's 4th Graduation ceremony that was presided over by President Uhuru Kenyatta.
The 21-year-old Bachelor of Science (Renewable Energy and Bio-Fuels Technology) student from Kirinyaga County was sexually assaulted and her left arm chopped off in the harrowing incident.
The victim told NTV reporters that she only recalled being in the company of her friends at around 11 p.m. before the attack that saw her comrades manage to flee to safety.
“The lady came in bleeding from an area that was cut, the left upper limb completely amputated. She came in having lost a lot of blood. Her state was semi-conscious although she is able to identify herself and give some history,” Dr.Martin Mbati, a doctor at the Bungoma West Hospital told journalists.
A report by K24 Digital indicated that the assailant chopped off her arm wrapped it in her brassiere and threw it on the roof of a residence near where the student lived at around 5 a.m. with neighbours failing to respond to her cries for help out of fear.
“Her assailant did not only cut her arm but also raped her. We are taking her to the theatre to manage the injuries,” the doctor is quoted by the publication while confirming the unfortunate news that they were not able to reattach the limb.
Her condition is reported to be stable after an urgent blood transfusion.
President Uhuru Kenyatta, who is also the Kibabii University Chancellor, arrived at the institution at around midday on Friday to preside over a graduation ceremony that where a total of 620 degrees and diplomas were conferred.
Uhuru pledged to allocate Ksh100 million to the university for infrastructural upgrades in a meeting help with the university council and its management.


Dockerize that Python app!
Hi! I'm James
twitter: @ignighted
blog: devblog.jamespacileo.com
Organizer Django Meetup London
Web developer at RoyalSunAlliance
Docker is being embraced... , and center!
Google is officially using it!
Youtube, Ebay, Spotify, Ebay, .... use it!
Even finance companies are using it in production!
Yes Docker is hotter than sliced bread!
Ok...
Docker must have been around for a while then!
Right?!
Drum rolls....
First commit... January 18, 2013!
V 1.0 was released... July 2014!
15,000+ stars 2,800+ forks
Summary
- What's Docker?
- How it can be useful for you.
- How to Dockerize a Python app
Docker enables you to run completely isolated applications within one single Linux server.
each DB, Web App, Web Server, Message Queue, ... = 1 Container
Host OS = Any Linux OS Distro that supports LXC
Container OS = Virtually any Linux Distro
Wait a sec... that sounds a lot like VMs!
Containers vs VMs!
1. Containers don't need to virtualize hardware
2. Same performance as HOST!
3. Minimal space consumption
Just a flesh wound?
4. Creating a container is insanely fast!
500ms VS 1-10 minutes
Time to talk about some use cases...
Development
Instantanously spin up, destoy multiple containers
Emulate real world infrastructure!
Jenkins + Docker!
Fast, accurate and repeatable testing!
Time to get our hands dirty!
Step 1.
Let's choose a base image!
You can use a base OS one:
You can use a modified base e.g.:
- Ubuntu 14.04 base with PostgreSQL 9
- CentOS base with RabbitMQ
- Debian with MongoDB
- ...
Docker Hub
Grab your repos!
Step 2.
Fire the image up in a container
Example Dockerfile
FROM python:2.7
RUN echo 'deb http://us.archive.ubuntu.com/ubuntu/ precise universe' >> /etc/apt/sources.list
RUN apt-get -y update
RUN apt-get -y install python python-pip build-essential
RUN pip install -e git+https://github.com/mitsuhiko/flask.git#egg=flask
RUN pip install flask-sqlalchemy
RUN apt-get -y purge build-essential
RUN apt-get -y autoremove
ADD . /srv/app/
WORKDIR /srv/app/
RUN flask --app=flaskr initdb
EXPOSE 5000
CMD ["flask", "--app=flaskr", "run", "--host=<IP_ADDRESS>"]
Thank You!
Questions!


Similar to last week, we will be discussing the results of user testing of hurricanegolf.com, an online discount golf supply business, except we will accessing the site from mobile devices and evaluating the mobile version of the site. The devices range from 360px width smart phones to iPads (768 pixels width in portrait mode)
The script used for these user tests can be found here, which has been modified slightly to account for the lack of functionality of the mobile version of the site
To review, the tasks we'll be asking the testers to perform are
- Home Page impressions: specifically what is the user able to determine about the business from the home page
- To learn about the company's return policy, asked to assess a specific type of return
- To learn how to track an order
- To find a specific product
This week, the users have a mix of experience with golf, some have not played before but when asked to self-assess, stated they felt comfortable with the terminology. All three test subjects are regular online shoppers. All are regular computer users both at work and at home and all use laptops, tablets and smartphones regularly.
Task 1: Home Page Assessment
Upon viewing the home page, users were able to determine that this business sells golf equipment online, and associated it with being a "discount" seller. These results are mostly similar to last week, with the mobile version of the site distilling down the content to just golf products. On the desktop version, users also correctly identified this however listed several other products as part of the core business as well.
Hurricanegolf.com's home page mobile version
As with last week, user testing suggests that the mobile version of the site correctly communicates the mission of hurricanegolf.com.
Task 2: Return Policy
Users were asked to use the website to determine if a fictitious order from hurricanegolf.com could be returned given the conditions. This information is located in the footer of the home page, pictured below:
On the desktop version, that information can be found by clicking the link in the footer of the home page, pictured here
Unfortunately, the mobile version of the site removes this menu and these links, making the information not accessible. No user was able to locate it, nor was the test facilitator. One user decided the best way to achieve this would be selecting "live help" in the mobile footer.
Hurricanegolf.com's home page footer - mobile version (note the lack of returns section vs desktop version pictured above)
Return policies are often useful information for online retail consumers, especially since buyers cannot sample the product in advance. An improvement to the usability would be to provide a button allowing the user to access the desktop version of the site from their mobile device. A second option would simply be to include this informatoin on the mobile version of the site.
Task 3: Track an order
For this task, users were asked to learn how to track an order. On the desktop version, this information is again located in the footer, similar to return policy. And much like returns, shipping and tracking information is not available in the mobile version of the site's footer.
One user used the search function and was unsuccessful. It was proven that the website does have information related to tracking avialable on the desktop version. It is therefore interesting that the search function did not return results.
All users determined that the site did not have this functionality. Much like the return policy task, an improvement that could be made would be to allow users to access the desktop version of the site from their devices, or to include shipping/tracking info on the mobile version.
Task 4: Shop for specific product
Users were asked to locate all left handed irons for sale, which would require navigating to the "irons" page and filtering for left handed products only. For reference, irons are located under the "golf clubs" menu on the home page, pictured below:
All users were able to successfully locate left handed irons. Unlike the desktop version where users did not immediately recognise the top level menu bar, the prominent hamburger-style collapsible menu on the mobile version on the home page was discovered quickly.
Interestingly enough, on the desktop version a consistent finding was that after users found the "irons" page but before they successfully filtered for "left handed", all users interacted with a "sort-by" drop-down menu where the first text option was "position". Once again, this drop -down menu attracted mobile users before they later filtered for left handed products. The sort-by drop-down menu is a unanimous finding across 6 different users!
Summarizing, the hamburger collapsible navigation icon attracted mobile users and directed them down the correct path, however all mobile users and desktop users attempted to use an unrelated drop-down menu prior to filtering. A design improvement would be to choose a word other than position as the first "sort by" choice to appear in the drop down, which I believe is too closely related to "handedness".
Summary
Hurricanegolf.com's mobile site lacks some key functionality compared to its desktop version. The site could be improved in the following ways:
- Add return policy and tracking links to the mobile footer
- Choose a word other than "position" as the default text in the "sort by" drop-down menus on product pages
Return policy or any information related to returns is a critical piece of information that users need when making online purchases and gives confidence to the buyer. However generally speaking users successfully navigated the mobile site.
Some positive aspects of their mobile design is the logical flow of product searches. There is a very direct, 2 to 3 tap path to most product types, allowing the user to quickly shop for specific products with minimum action. Additionally, menu items are delineated and shaded a different color from the body background, indicating that they are buttons for the user to tap
While the site lacks basic functinality, it does achieve its goal of providing a way for a user to shop using a mobile device


var conditions: ArrayList<(T) -> Boolean>
The list of conditions that must be met. Used within InsnQuery.check


JVM Dependency Management
In general, we recommend the 3rdparty idiom for organizing dependencies on code from outside the source tree. This document describes how to make this work for JVM (Java or Scala) code.
Your JVM code can depend on external, third-party libraries. Pants uses Ivy to resolve and retrieve these JAR files. You should know the (Maven/Ivy groupId, artifactId, and version) you want to use.
3rdparty/jvm
If you have a small to medium number of third-party dependencies, you can define
them all in a single 3rdparty/jvm/BUILD
file. If you have a large number, it
may make sense to organize them in multiple subdirectories, say by category or by publisher.
In the appropriate BUILD
file, you create a jar_library
referencing the jar
s you want:
jar_library( jars=[ jar(org='com.google.auto.value', name='auto-value', rev='1.1'), ], # Is used as an annotation processor. scope='compile', )
Here, the jar_library
's name
defines a target address that other build targets can refer to. The
jar
s refer to jars that Ivy can resolve and fetch.
When using Scala with pants, the scala_jar
symbol will add the appropriate Scala version suffix (e.g. _2.12
) to the name
field automatically.
Your Code's BUILD File
To set up your code to import the external jar, you add a dependency to
the appropriate Java target[s] in your BUILD
file and add import
statements in your Java code.
For example, your BUILD
file might have
java_library(name='autovalue-lib', dependencies = [ '3rdparty/jvm/com/google/auto/value', ], )
And your Java code might have:
import com.google.auto.value.AutoValue;
"Round Trip" Dependencies
It is possible for your code to exist as source in the repo but also be published as a binary to an external repository. If you happen to pull in any third party artifacts, they may express a dependency on the published version of the artifact. This means that the classpath will contain both the version in the repo compiled from source and an older version that was previously published. In this case, you want to be sure that when pants always prefers the version built from source.
Fortunately, the remedy for this is simple. If you add a provides=
parameter
that matches the one used to publish the artifact, pants will always prefer the
local target definition to the published jar if it is in the context:
java_library( name='api', sources=['*.java'], provides=artifact(org='org.archie', name='api', repo=myrepo), ) jar_library( name='bin-dep', jars=[ jar(org='org.archie', name='consumer', rev='1.2.3'), ], dependencies=[ # Include the local, source copy of the API to cause it to be used rather than # any versioned binary copy that the `consumer` lib might depend on transitively. ':api', ])
Controlling JAR Dependency Versions
If you notice that one "foreign" dependency pulls in mostly wrong
things, tell Pants not to pull in its dependencies. In your
3rdparty/.../BUILD
file, use jar
's intransitive
argument; then
carefully add hand-picked versions:
jar_library(name="retro-naming-factory", jars=[ jar(org='retro', name='retro-factory', rev='5.0.18', intransitive=True), ], dependencies=[ # Don't use retro's expected (old, incompatible) common-logging # version, yipe; use the same version we use everywhere else: '3rdparty/jvm/common-logging', ] )
If you notice a small number of transitive dependencies to exclude
Rather than mark the jar
intransitive, you can exclude
some
transitive dependencies from JVM targets:
java_library( name = 'loadtest', dependencies = [ '3rdparty/storm:storm', ], sources = ['*.java'], excludes = [ exclude('org.sonatype.sisu.inject', 'cglib') ] )
Managing Transitive Dependencies
The Problem
If you have jars that pull in many transitive dependencies, you probably want to constrain which versions of those transitive dependencies you pull in. This is valuable for:
- Security concerns (you may want to avoid artifacts with known vulnerabilities, or you may only want to use particular jars which you trust).
- Predictable and consistent behavior across all projects in your repository (described below).
- Caching concerns/build times (described below).
Otherwise, you may have some targets that end up being built with
version 1.2.3
of a transitive dependency, and others that get built with
4.5.6
of that dependency. Worse, the same target might be built with
different versions of a transitive dependency depending on what other
targets happen to be part of the same pants invocation. To illustrate
this, consider the diagram below:
Assume foo
and bar
are binary targets. If you build a binary of foo
with ./pants binary foo
, foo
will be packaged with the example
jar
in addition to its transitive dependencies, which will be resolved as the
common
jar, version 1.2.3
.
Likewise, if you run ./pants binary bar
, it will be packaged with demo
,
and the transitive dependencies of demo
, which here is simply the common
jar version 4.5.6
.
However, if you run ./pants binary foo bar
, ivy will only resolve one
version of common-1.2.3
, which most likely means that both foo
and bar
will get common
version 4.5.6
(because it is the more recent version).
This is a problem, because it may be that common-4.5.6
is not compatible
with 3rdparty:example
, which will break the foo
binary at runtime.
More subtly, if you have many intermediate java_library
targets between
your jvm_binaries
and your jar_library
targets (which is normally the
case), simply changing which combination of java_library
targets are in
the same ./pants
invocation may invalidate the cache and force Pants to
recompile them, even if their sources are unchanged. This is because they
may resolve different versions of their transitive jar dependencies than
the previous time they were compiled, which means their classpaths will be
different. Getting a different classpath causes a cache-miss, forcing a
recompile. In general recompiling when the classpath changes is the
correct thing to do, however this means that unstable transitive
dependencies will cause a lot of cache-thrashing. If you have a large
repository with a large amount of code, recompiles get expensive.
Possible Solutions
There are a few ways to avoid or work around these problems. A simple method is to use the strict ivy conflict manager, which will cause the jar resolution to fail with an error if it detects two artifacts with conflicting versions. This has the advantage of forcing a dev to be aware of (and make a decision about) conflicting versions.
You could also disable transitive jar resolution altogether, and explicitly declare every dependency you need. This ensures that you have total control over your external dependencies, but can be difficult to maintain.
The third option is using managed_jar_dependencies
, to pin the versions of
the subset of your transitive dependencies that you care about.
Managed Jar Dependencies
Maven handles this problem with the <dependencyManagement>
stanza,
and Pants has similar functionality via the managed_jar_dependencies
target.
You can set up your 3rdparty/BUILD
file like so:
managed_jar_dependencies(name='management', artifacts=[ ':commons-io', ':jersey-core', ], ) jar_library(name='commons-io', jars=[ jar('commons-io', 'commons-io', '2.5'), ], ) jar_library(name='jersey-core', jars=[ jar('com.sun.jersey', 'jersey-core', '1.19.1'), ], )
And in pants.toml
, add:
[jar-dependency-management] default_target = "3rdparty:management"
This will force all jar_library
targets in your repository to use the
versions of commons-io
and jersey-core
referenced by the management
target. When resolving transitive dependencies, Pants will always choose
the versions "pinned" by the managed dependencies target.
If a jar_library
omits the version for one of its jar()
s, it will use
the version defined in managed_jar_dependencies
. If a jar()
defines a
version that conflicts with the version set in managed_jar_dependencies
,
an error will be raised and the build will fail (though this behavior can
be modified via the conflict_strategy
option).
This is a bit verbose, and entails a bit of duplicate code (you have to
mention jersey-core
3 times in the above example). You can use the
managed_jar_libraries
target factory instead to make your 3rdparty/BUILD
definitions more concise.
This example is equivalent to the one earlier, but using managed_jar_libraries
instead:
managed_jar_libraries(name='management', artifacts=[ jar('commons-io', 'commons-io', '2.5'), jar('com.sun.jersey', 'jersey-core', '1.19.1'), ], )
This automatically generates jar_library
targets for you, and makes a
managed_jar_dependencies
target to reference them. (Note that you still
need to make the same change to pants.ini).
The generated library targets follow the naming convention
org.name.classifier.ext
, where classifier
and ext
are omitted if they
are the default values.
So in the above example will generate two jar_library
targets, called
3rdparty:commons-io.commons-io
and 3rdparty:com.sun.jersey.jersey-core
.
The artifacts list of managed_jar_libraries
can also accept target
addresses to already-existing jar_libraries
, just like a normal
managed_jar_dependences
target. In this case, managed_jar_libraries
will just use the referenced target, rather than generating a new one.
With Pants you can create multiple managed_jar_dependencies
.
If you have more than one, for any particular jar_library
, you can define
which managed_jar_dependencies
it uses explicitly (rather than using the
default defined in pants.ini
):
jar_library(name='org.apache.hadoop-alternate', jars=[ jar('org.apache.hadoop', 'hadoop-common', '2.7.0'), ], managed_dependencies='3rdparty:management-alternate', )
Using a SNAPSHOT JVM Dependency
Sometimes your code depends on a buggy external JVM dependency. You
think you've fixed the external code, but want to test locally before
uploading it to make sure. To do this, in the jar
dependency for the
artifact, specify the url
attribute to point to the local file and
change the rev
. If you are actively making changes to the dependency,
you can also use the mutable
jar attribute to re-import the file each
time pants is run (otherwise, pants will cache it):
jar_library(name='checkstyle', jars = [ jar(org='com.puppycrawl.tools', name='checkstyle', rev='5.5-SNAPSHOT', url='file:///Users/pantsdev/Src/checkstyle/checkstyle.jar', mutable=True), ], )
Strict Dependencies
What is strict_deps?
Strict_deps is a feature of JVM targets which controls the dependencies available on the compile classpath when that target is being compiled.
When compiling a JVM target with strict_deps disabled, all transitive dependencies of that target are included on the compile classpath.
When compiling a JVM target with strict deps enabled, the compile classpath is more restricted. Instead of containing all transitive dependencies, only the direct dependencies and the targets exported by those dependencies are included in the compile classpath.
Enabling strict_deps speeds up builds because it 1) reduces the work the compiler has to do when searching for symbols, and 2) improves cache hit rates by reducing the number of dependencies that contribute to a target's cache keys. This works because most targets don't need their full transitive dependencies available to compile. However, it does require you to be more exact in writing down your dependencies.
Why strict_deps?
Depending on the shape of your build graph, use of strict_deps improves cache hit rates and per-target compilation time (on the order of 10-20%).
Example
With strict_deps enabled, we can identified some gaps between what the compiler expects on a classpath and what is actually available on the classpath as specified in BUILD files.
Let's look at a simple example involving 3 targets A, B and C. B depends on A, and C depends on B. Thus their dependency graph can be represented like this: C->B->A. Their declarations in BUILD file are shown as below:
scala_library( name='A', source=['A.scala'], ) scala_library( name='B', source=['B.scala'], dependencies=[':A'], ) scala_library( name='C', source=['C.scala'], dependencies=[':B'], )
If we compile C with strict_deps turned off, everything works fine, as all transitive dependencies of C will be added to compile time classpaths. However, the compilation may require additional dependencies if strict_deps is turned on. With strict_deps=True, only B is in C's compile time classpath. There are 2 cases for the failure.
-
C directly uses APIs from A. This is obviously a mistake by author of C, and the fix is simple, just add A to C's dependency list.
-
C does not use APIs from A, but compiler still wants A to be in C's classpaths. This is tricky, as author of C may not even be aware of A. It's hard for C's author to get the right dependency list in this case.
People may wonder why the compiler asks for A to compile C in the first place. There are a number of reasons for this, an example of which is that the java compiler needs all transitively implemented interfaces to be on the classpath so that it can resolve default method implementations.
Strict_deps for Library Developers
The compile overhead of a library change for downstream targets when strict_deps is enabled should be minimized to just the targets that depend directly on the library target or depend on a target that exports that target.
Invalidation
This is a change from the current behavior where all downstream targets will have their compile invalidated.
With strict_deps, a change to a library target will only invalidate the compile artifacts of dependent targets if they have
- A direct dependency on the changed library target, or
- A dependency on it via the export graph.
Before, if a target depending on a library had an implicit dependency on a transitive dependency of the library target it would just work. The transitive dependency would end up on the dependent target’s compile classpath and that target would compile happily. Strict deps removes that implicit transitivity, which exposes undeclared dependencies.
This introduces two new sources of errors.
- When adding a new library dependency to a target, if the library dependency requires some of its dependencies to be on the compile classpath, the compile can fail if those dependencies don’t make it on the classpath. These errors are a bit weird and will be confusing at first.
- When a library target adds a new publicly visible dependency, targets that depend on the library target will fail to compile with the same kind of error message as 1.
Errors, Exports, and Missing-deps-suggest
The errors sometimes have different messages, but usually in Scala they look something like this:
[error] <path-to-current-sources>/<Some-file>.scala:71:32: Symbol 'type <Some-type>' is missing from the classpath. [error] This symbol is required by '<Some-other-type>.timeout'. [error] Make sure that type <Some-type>is in your classpath and check for conflicting dependencies with `-Ylog-classpath`. [error] A full rebuild may help if <Some-other-type>.class was compiled against an incompatible version of <none>.<Some-type>. [error] Some-file.code [error]
NOTE: It's likely the source snippet may not be related to the error
There are two possible solutions for this error. One is to identify the dependency that is missing and add it to your target directly.
Missing-deps-suggest
Missing-deps-suggest is a feature that will try to identify the target that failed to compile and suggest a dependency that may be missing.
If this doesn't provide a solution, then the second solution to this error is exporting the missing type's target in the dependency you depend on.
Exports
A target should export any dependencies which provide types or symbols that are exposed in its public API. For instance, if target A has a method returning something of type X, it should export the target which defines the type X.
Exports are transitive. This means that if a target exports one of its dependencies and that exported dependency has exports, those exports will also be end up on the classpath. This allows the export graph to model requirements created by type hierarchies.
How to Solve An Error With Exports
Let's look at an example failure:
[1/1] Compiling 1 zinc source in 1 target (examples/src/scala/org/pantsbuild/example/strictdeps:h). 17:12:58 00:03 [compile] 17:12:58 00:03 [zinc] [error] Symbol 'type z.Z' is missing from the classpath. [error] This symbol is required by 'method z.X.yyy'. [error] Make sure that type Z is in your classpath and check for conflicting dependencies with `-Ylog-classpath`. [error] A full rebuild may help if 'X.class' was compiled against an incompatible version of z. [error] one error found [error] Compile failed at Sep 19, 2017 5:12:59 PM [0.916s] 17:12:59 00:04 [missing-deps-suggest] compile(examples/src/scala/org/pantsbuild/example/strictdeps:h) failed: Zinc compile failed. FAILURE: Compilation failure: Failed jobs: compile(examples/src/scala/org/pantsbuild/example/strictdeps:h)
There are three targets that need to be identified to fix an error like this:
- the target that failed to compile.
- the dependency of 1 that owns the type that needed the missing type
- the dependency of target 2 that owns the missing type.
Target 1 should be the most straight-forward to find. It's the target named in
the failure message as a failed job. It's mentioned both at the end of the
failing compile, in the line starting with FAILURE: Compilation failure
,
and in a line near the error message. In this example, the failure is
compile(examples/src/scala/org/pantsbuild/example/strictdeps:h)
. So, target
1 is examples/src/scala/org/pantsbuild/example/strictdeps:h
.
Finding target 2 involves looking at target 1's dependencies and figuring out which one contains the type from the error message. First, let's find the type. In a message like this, target 2's type is the one that required the missing type. In this message specifically, it's the following line:
[error] This symbol is required by 'method z.X.yyy'.
The type from target 2 is the method yyy
on z.X
. How to we find
the target for that type? One way would be to eyeball it. Let's look at the
BUILD file.
scala_library( name='h', sources=['H.scala'], dependencies=[':x'], strict_deps=True )
In this example, target 1 only has one dependency, x
. That makes finding
target 2 easy. It's :x
.
Now that we know target 1 and target 2, let's find 3. Since we're doing this manually, we'll look at the build files again.
Here's target 2's BUILD file declaration:
scala_library( name='x', sources=['X.scala'], dependencies=[':z'] )
It too has only one dependency, so we know what Target 3 is. It's :z
.
Now that we know what the missing dependency is, we can fix the error.
Z has to be on the classpath in order to use X, so any target depending
on :x
will also need :z
. We could just add a dependency on :z to :h, but
doing so would result in new users of :x
running into this same error. To
prevent that, let's add an export to :x
of :z
.
scala_library( name='x', sources=['X.scala'], dependencies=[':z'], exports=[':z'] )
Now our target compiles successfully.