Note
Click here to download the full example code
Issues when switching to float¶
Most models in scikit-learn do computation with double,
not float. Most models in deep learning use float because
that’s the most common situation with GPU. ONNX was initially
created to facilitate the deployment of deep learning models
and that explains why many converters assume the converted models
should use float. That assumption does not usually harm
the predictions, the conversion to float introduce small
discrepencies compare to double predictions.
That assumption is usually true if the prediction
function is continuous, , then
. We can determine an upper bound
to the discrepencies :
.
dx is the discrepency introduced by a float conversion,
dx = x - numpy.float32(x)
.
However, that’s not the case for every model. A decision tree trained for a regression is not a continuous function. Therefore, even a small dx may introduce a huge discrepency. Let’s look into an example which always produces discrepencies and some ways to overcome this situation.
More into the issue¶
The below example is built to fail.
It contains integer features with different order
of magnitude rounded to integer. A decision tree compares
features to thresholds. In most cases, float and double
comparison gives the same result. We denote
the conversion (or cast)
numpy.float32(x)
.
However, the probability that both comparisons give different results is not null. The following graph shows the discord areas.
from skl2onnx.sklapi import CastRegressor
from mlprodict.onnxrt import OnnxInference
from mlprodict.onnx_conv import to_onnx as to_onnx_extended
from mlprodict.sklapi import OnnxPipeline
from skl2onnx.sklapi import CastTransformer
from skl2onnx import to_onnx
from onnxruntime import InferenceSession
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import make_regression
import numpy
import matplotlib.pyplot as plt
def area_mismatch_rule(N, delta, factor, rule=None):
if rule is None:
rule = lambda t: numpy.float32(t)
xst = []
yst = []
xsf = []
ysf = []
for x in range(-N, N):
for y in range(-N, N):
dx = (1. + x * delta) * factor
dy = (1. + y * delta) * factor
c1 = 1 if numpy.float64(dx) <= numpy.float64(dy) else 0
c2 = 1 if numpy.float32(dx) <= rule(dy) else 0
key = abs(c1 - c2)
if key == 1:
xsf.append(dx)
ysf.append(dy)
else:
xst.append(dx)
yst.append(dy)
return xst, yst, xsf, ysf
delta = 36e-10
factor = 1
xst, yst, xsf, ysf = area_mismatch_rule(100, delta, factor)
fig, ax = plt.subplots(1, 1, figsize=(5, 5))
ax.plot(xst, yst, '.', label="agree")
ax.plot(xsf, ysf, '.', label="disagree")
ax.set_title("Region where x <= y and (float)x <= (float)y agree")
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.plot([min(xst), max(xst)], [min(yst), max(yst)], 'k--')
ax.legend()
Out:
<matplotlib.legend.Legend object at 0x7f5cdc20a7c0>
The pipeline and the data¶
We can now build an example where the learned decision tree does many comparisons in this discord area. This is done by rounding features to integers, a frequent case happening when dealing with categorical features.
X, y = make_regression(10000, 10)
X_train, X_test, y_train, y_test = train_test_split(X, y)
Xi_train, yi_train = X_train.copy(), y_train.copy()
Xi_test, yi_test = X_test.copy(), y_test.copy()
for i in range(X.shape[1]):
Xi_train[:, i] = (Xi_train[:, i] * 2 ** i).astype(numpy.int64)
Xi_test[:, i] = (Xi_test[:, i] * 2 ** i).astype(numpy.int64)
max_depth = 10
model = Pipeline([
('scaler', StandardScaler()),
('dt', DecisionTreeRegressor(max_depth=max_depth))
])
model.fit(Xi_train, yi_train)
Out:
Pipeline(steps=[('scaler', StandardScaler()),
('dt', DecisionTreeRegressor(max_depth=10))])
The discrepencies¶
Let’s reuse the function implemented in the first example Comparison and look into the conversion.
def diff(p1, p2):
p1 = p1.ravel()
p2 = p2.ravel()
d = numpy.abs(p2 - p1)
return d.max(), (d / numpy.abs(p1)).max()
onx = to_onnx(model, Xi_train[:1].astype(numpy.float32))
sess = InferenceSession(onx.SerializeToString(),
providers=['CPUExecutionProvider'])
X32 = Xi_test.astype(numpy.float32)
skl = model.predict(X32)
ort = sess.run(None, {'X': X32})[0]
print(diff(skl, ort))
Out:
(84.52466935398928, 6.980041539088254)
The discrepencies are significant. The ONNX model keeps float at every step.
In scikit-learn:
CastTransformer¶
We could try to use double everywhere. Unfortunately, ONNX ML Operators only allows float coefficients for the operator TreeEnsembleRegressor. We may want to compromise by casting the output of the normalizer into float in the scikit-learn pipeline.
model2 = Pipeline([
('scaler', StandardScaler()),
('cast', CastTransformer()),
('dt', DecisionTreeRegressor(max_depth=max_depth))
])
model2.fit(Xi_train, yi_train)
Out:
Pipeline(steps=[('scaler', StandardScaler()), ('cast', CastTransformer()),
('dt', DecisionTreeRegressor(max_depth=10))])
The discrepencies.
onx2 = to_onnx(model2, Xi_train[:1].astype(numpy.float32))
sess2 = InferenceSession(onx2.SerializeToString(),
providers=['CPUExecutionProvider'])
skl2 = model2.predict(X32)
ort2 = sess2.run(None, {'X': X32})[0]
print(diff(skl2, ort2))
Out:
(70.40040571810144, 6.980041539088254)
That still fails because the normalizer in scikit-learn and in ONNX use different types. The cast still happens and the dx is still here. To remove it, we need to use double in ONNX normalizer.
model3 = Pipeline([
('cast64', CastTransformer(dtype=numpy.float64)),
('scaler', StandardScaler()),
('cast', CastTransformer()),
('dt', DecisionTreeRegressor(max_depth=max_depth))
])
model3.fit(Xi_train, yi_train)
onx3 = to_onnx(model3, Xi_train[:1].astype(numpy.float32),
options={StandardScaler: {'div': 'div_cast'}})
sess3 = InferenceSession(onx3.SerializeToString(),
providers=['CPUExecutionProvider'])
skl3 = model3.predict(X32)
ort3 = sess3.run(None, {'X': X32})[0]
print(diff(skl3, ort3))
Out:
(2.441649940010393e-05, 5.533155989632114e-08)
It works. That also means that it is difficult to change the computation type when a pipeline includes a discontinuous function. It is better to keep the same types all along before using a decision tree.
Sledgehammer¶
The idea here is to always train the next step based on ONNX outputs. That way, every step of the pipeline is trained based on ONNX output.
Trains the first step.
Converts the step into ONNX
Computes ONNX outputs.
Trains the second step on these outputs.
Converts the second step into ONNX.
Merges it with the first step.
Computes ONNX outputs of the merged two first steps.
…
It is implemented in class OnnxPipeline.
model_onx = OnnxPipeline([
('scaler', StandardScaler()),
('dt', DecisionTreeRegressor(max_depth=max_depth))
])
model_onx.fit(Xi_train, yi_train)
Out:
OnnxPipeline(steps=[('scaler',
<EMAIL_ADDRESS>\xff<=c\xd6\x82<=\x00\x1f\x01<=\x13\xc2\x7f;=\x19\xa4\x00;\xa0\x01\x06:\nai.onnx.ml\x12\x1emlprodict_ONNX(StandardScaler)Z\x11\n\x01X\x12\x0c\n\n\x08\x01\x12\x06\n\x00\n\x02\x08\nb\x18\n\x08variable\x12\x0c\n\n\x08\x01\x12\x06\n\x00\n\x02\x08\nB\x0e\n\nai.onnx.ml\x10\x01B\x04\n\x00\x10\x0f')),
('dt', DecisionTreeRegressor(max_depth=10))])
The conversion.
onx4 = to_onnx(model_onx, Xi_train[:1].astype(numpy.float32))
sess4 = InferenceSession(onx4.SerializeToString(),
providers=['CPUExecutionProvider'])
skl4 = model_onx.predict(X32)
ort4 = sess4.run(None, {'X': X32})[0]
print(diff(skl4, ort4))
Out:
(1.5132379814986052e-05, 5.533155968131901e-08)
It works too in a more simple way.
No discrepencies at all?¶
Is it possible to get no error at all? There is one major obstacle: scikit-learn stores the predicted values in every leave with double (_tree.pyx - _get_value_ndarray), ONNX defines the the predicted values as floats: TreeEnsembleRegressor. What can we do to solve it? What if we could extend ONNX specifications to support double instead of floats. We reuse what was developped in example Other way to convert and a custom ONNX node TreeEnsembleRegressorDouble.
tree = DecisionTreeRegressor(max_depth=max_depth)
tree.fit(Xi_train, yi_train)
model_onx = to_onnx_extended(tree, Xi_train[:1].astype(numpy.float64),
rewrite_ops=True)
oinf5 = OnnxInference(model_onx, runtime='python_compiled')
print(oinf5)
Out:
OnnxInference(...)
def compiled_run(dict_inputs, yield_ops=None):
if yield_ops is not None:
raise NotImplementedError('yields_ops should be None.')
# inputs
X = dict_inputs['X']
(variable, ) = n0_treeensembleregressordouble(X)
return {
'variable': variable,
}
Let’s measure the discrepencies.
X64 = Xi_test.astype(numpy.float64)
skl5 = tree.predict(X64)
ort5 = oinf5.run({'X': X64})['variable']
Perfect, no discrepencies at all.
print(diff(skl5, ort5))
Out:
(0.0, 0.0)
CastRegressor¶
The previous example demonstrated the type difference for the predicted values explains the small differences between scikit-learn and onnxruntime. But it does not with the current ONNX. Another option is to cast the the predictions into floats in the scikit-learn pipeline.
ctree = CastRegressor(DecisionTreeRegressor(max_depth=max_depth))
ctree.fit(Xi_train, yi_train)
onx6 = to_onnx(ctree, Xi_train[:1].astype(numpy.float32))
sess6 = InferenceSession(onx6.SerializeToString(),
providers=['CPUExecutionProvider'])
skl6 = ctree.predict(X32)
ort6 = sess6.run(None, {'X': X32})[0]
print(diff(skl6, ort6))
Out:
(0.0, 0.0)
Success!
Total running time of the script: ( 0 minutes 4.856 seconds)


{% extends "machina/forum/index.html" %}
{% load i18n %}
{% block content %}
{% if not debug %}
{% blocktrans %}
This is a demo site showcasing the use of django-machina. The site and its database are resetted every day. Feel free to register yourself and try it out!
{% endblocktrans %}


|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
public interface ManagedRepositoryContent
ManagedRepositoryContent interface for interacting with a managed repository in an abstract way, without the need for processing based on filesystem paths, or working with the database.
| Method Summary | |
|---|---|
void |
deleteArtifact(ArtifactReference artifactReference)
delete a specified artifact from the repository |
void |
deleteGroupId(String groupId)
|
void |
deleteProject(String namespace,
String projectId)
|
void |
deleteVersion(VersionedReference reference)
Delete from the managed repository all files / directories associated with the provided version reference. |
String |
getId()
Convenience method to get the repository id. |
Set<ArtifactReference> |
getRelatedArtifacts(ArtifactReference reference)
Gather up the list of related artifacts to the ArtifactReference provided. |
String |
getRepoRoot()
Convenience method to get the repository (on disk) root directory. |
ManagedRepository |
getRepository()
Get the repository configuration associated with this repository content. |
Set<String> |
getVersions(ProjectReference reference)
Given a specific ProjectReference , return the list of available versions for
that project reference. |
Set<String> |
getVersions(VersionedReference reference)
Given a specific VersionedReference , return the list of available versions for that
versioned reference. |
boolean |
hasContent(ArtifactReference reference)
Determines if the artifact referenced exists in the repository. |
boolean |
hasContent(ProjectReference reference)
Determines if the project referenced exists in the repository. |
boolean |
hasContent(VersionedReference reference)
Determines if the version reference exists in the repository. |
void |
setRepository(ManagedRepository repo)
Set the repository configuration to associate with this repository content. |
ArtifactReference |
toArtifactReference(String path)
Given a repository relative path to a filename, return the VersionedReference object suitable for the path. |
File |
toFile(ArchivaArtifact reference)
Given an ArchivaArtifact , return the file reference to the artifact. |
File |
toFile(ArtifactReference reference)
Given an ArtifactReference , return the file reference to the artifact. |
String |
toMetadataPath(ProjectReference reference)
Given a ProjectReference , return the path to the metadata for
the project. |
String |
toMetadataPath(VersionedReference reference)
Given a VersionedReference , return the path to the metadata for
the specific version of the project. |
String |
toPath(ArchivaArtifact reference)
Given an ArchivaArtifact , return the relative path to the artifact. |
String |
toPath(ArtifactReference reference)
Given an ArtifactReference , return the relative path to the artifact. |
| Method Detail |
|---|
void deleteVersion(VersionedReference reference) throws ContentNotFoundException
reference
- the version reference to delete.
ContentNotFoundException
void deleteArtifact(ArtifactReference artifactReference) throws ContentNotFoundException
artifactReference
-
ContentNotFoundException
void deleteGroupId(String groupId) throws ContentNotFoundException
groupId
-
ContentNotFoundException
void deleteProject(String namespace, String projectId) throws RepositoryException
namespace
- groupId for mavenprojectId
- artifactId for maven
ContentNotFoundException
RepositoryException
String getId()
Convenience method to get the repository id.
Equivalent to calling .getRepository().getId()
Set<ArtifactReference> getRelatedArtifacts(ArtifactReference reference) throws ContentNotFoundException
Gather up the list of related artifacts to the ArtifactReference provided. This typically inclues the pom files, and those things with classifiers (such as doc, source code, test libs, etc...)
NOTE: Some layouts (such as maven 1 "legacy") are not compatible with this query.
reference
- the reference to work off of.
ContentNotFoundException
- if the initial artifact reference does not exist within the repository.
LayoutException
String getRepoRoot()
Convenience method to get the repository (on disk) root directory.
Equivalent to calling .getRepository().getLocation()
ManagedRepository getRepository()
Set<String> getVersions(ProjectReference reference) throws ContentNotFoundException, LayoutException
ProjectReference
, return the list of available versions for
that project reference.
reference
- the project reference to work off of.
ContentNotFoundException
- if the project reference does nto exist within the repository.
LayoutException
Set<String> getVersions(VersionedReference reference) throws ContentNotFoundException
Given a specific VersionedReference
, return the list of available versions for that
versioned reference.
NOTE: This is really only useful when working with SNAPSHOTs.
reference
- the versioned reference to work off of.
ContentNotFoundException
- if the versioned reference does not exist within the repository.
LayoutException
boolean hasContent(ArtifactReference reference)
reference
- the artifact reference to check for.
boolean hasContent(ProjectReference reference)
reference
- the project reference to check for.
boolean hasContent(VersionedReference reference)
reference
- the version reference to check for.
void setRepository(ManagedRepository repo)
repo
- the repository to associate with this repository content.ArtifactReference toArtifactReference(String path) throws LayoutException
VersionedReference
object suitable for the path.
path
- the path relative to the repository base dir for the artifact.
ArtifactReference
representing the path. (or null if path cannot be converted to
a ArtifactReference
)
LayoutException
- if there was a problem converting the path to an artifact.File toFile(ArtifactReference reference)
ArtifactReference
, return the file reference to the artifact.
reference
- the artifact reference to use.
File toFile(ArchivaArtifact reference)
ArchivaArtifact
, return the file reference to the artifact.
reference
- the archiva artifact to use.
String toMetadataPath(ProjectReference reference)
ProjectReference
, return the path to the metadata for
the project.
reference
- the reference to use.
String toMetadataPath(VersionedReference reference)
VersionedReference
, return the path to the metadata for
the specific version of the project.
reference
- the reference to use.
String toPath(ArtifactReference reference)
ArtifactReference
, return the relative path to the artifact.
reference
- the artifact reference to use.
String toPath(ArchivaArtifact reference)
ArchivaArtifact
, return the relative path to the artifact.
reference
- the archiva artifact to use.
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |


The beautiful home that comes with its own award-winning vineyard and apple orchard
Fancy pouring yourself a glass of wine made from your very own Welsh grapes?
by Joanne RidoutImagine you are hosting a party at your home and suddenly realise you are running out of wine and the nearest shop is miles away.
Not a problem if you are lucky enough to be the owner of this lovely mansion near Monmouth as it has its very own vineyard in the garden.
But this is no single vine, amateur operation.
Ancre Hill Estate is a multi award-winning Welsh vineyard that produces wine that is enjoyed across the globe, with the estate being recognised by some of the major international wine competitions.
Arguably the pinnacle of the estate's success so far was in 2012, when its 2008 sparkling wine won the Bollicine del Mondo for the best sparkling wine in the world.
The brand and business is well established, with on-site facilities including a cellar, a converted stable block used for customer wine tasting events and, of course, a state-of-the-art, fully-equipped winery.
Oh, and outside there are grapes. Lots of grapes.
The estate in its entirety, including a complete farmstead, covers about 137.1 acres, 29.5 acres of which is dedicated to the vines on the south-facing slopes.
There is also a substantial cider orchard within the grounds.
The estate can also boast using traditional biodynamic and organic practices following a holistic, ecological and ethical approach.
According to the vineyard's website, the acreage of vines at the estate are inspected every year by Demeter, the largest certification organisation for biodynamic agriculture, to ensure all cultural practices go beyond organic standards in line with the full biodynamic accreditation achieved in 2014.
But at the heart of the business is a beautiful home .
In fact, there are so many properties and outbuildings that make up the estate, it could be described as being its own little hamlet.
There's the main house, which is a three-storey, double fronted mansion that overlooks the lawn garden and vineyard beyond, plus three other cottages.
The attractive red stone facade and painted side-wing of the main period property are just the beginning of the distinctive original features and classic decor of this impressive home.
The stained glass porch that also boasts Minton floor tiles begins the journey through the mansion, which has original fireplaces, large sash windows, sanded floors, picture rails and coving that combine to give the home a feeling of grandeur.
Colours throughout are predominantly neutral, accent colours are rich in tone, patterns are traditional and furniture is predominantly antique or classic in style.
There are plenty of moments of sparkle though, with classic chandeliers and ornate metallic gold mirror frames being the shiny stars of the interiors show.
The kitchen diner is a space where a more contemporary style is in attendance, with the kitchen units a vibrant shade of teal.
Fixtures and fittings are classic and tasteful and as age appropriate to the style of the home as possible, including the roll-top bath and the column radiators.
The interior design scheme in some of the bedrooms has been given a slightly more modern style, predominantly through the use of 'on-trend' colours such as slate grey and heritage blue.
In total, the main house has five reception rooms, including one dedicated to snooker, plus the large kitchen/diner that leads into a sunny conservatory.
On the first floor the master bedroom has an ensuite and walk-in dressing room.
There are three further bedrooms on this level, one with an ensuite, plus two family bathrooms.
On the second floor can be found the final two bedrooms that have feature exposed beams, with the ceiling raised to the roof. Both of these loft bedrooms have an ensuite.
Then there's the three cottages that could all potentially be used as holiday lets if planning consent already exists or can be secured.
There's a cute two bed, a spacious three bed, plus a four bedroom property which is actually the recently refurbished former Newton Court Farm farmhouse, which now makes up part of the overall estate.
And when a new owner is not picking, squashing or bottling grapes, picking apples or greeting holiday guests, there's plenty to do at the estate.
In the grounds there is an outdoor swimming pool surrounded by swaying palm trees that looks like a scene from a posh south of France villa.
When not relaxing by the pool, sipping a glass of wine of course, there's a tennis court for the owner to enjoy and a pretty garden surrounding the mansion to attend to and enjoy wandering around.
Of course, a ride-on tractor is a must so there's a dedicated shed for it to sleep in, plus a range of garages for machinery and equipment.
The estate comes with a separate office so work and life are physically separated, with maybe the shortest and prettiest commute to work in the region.
And if time off from the grapes is required, as the stunning panoramic views suggest, the estate is surrounded by beautiful Welsh countryside enticing an owner out of the gates for country hiking and water based activities.
Ancre Hill Estates is on the market with estate agent Savills with the price only revealed to interested buyers (price on application). Call the Oxford Botley branch on 01865 269000 for further details.


Single selection
Select your favorite color
{{ color.label }}
Multiple selection
Pick toppings for your pizza
{{ topping.label }}
Grouped options
Pick your Pokemon
{{ creature.label }}
Colors
2000
2100
Alaska
Alabama
English
Spanish
Mihiramon
Sandiramon
Water
Coke
Light
Dark


EIP
LLC
Products
Agave ISY Controller
User Agent for Google Chrome
Fly Paper
Easy Draw
Snake Game
About
Contact
Contact
Contact ...


The data were first presented in this publication:
Djomehri, S.I., Gonzalez, M.E., da Veiga Leprevost, F., Tekula, S.R., Chang, H.Y., White, M.J., Cimino-Mathews, A., Burman, B., Basrur, V., Argani, P. and Nesvizhskii, A.I., 2020. Quantitative proteomic landscape of metaplastic breast carcinoma pathological subtypes and their relationship to triple-negative tumors. Nature communications, 11(1), pp.1-15.
A total of 27 breast tissue samples were studied with TMT (Ref. 1) labeling using the newer SPS MS3 method (Ref. 2). The instrument was a Thermo Fusion Tribrid mass spectrometer (Ref. 3). The paper was looking at the proteomic signature of metaplastic breast cancer and how different metaplastic subtypes might differ. Normal tissue and less aggressive triple negative breast cancer samples were also profiled. One of the metaplastic samples was only normal tissue. The final number of samples were 7 normal tissue, 6 triple negative non-metaplastic, and 14 metaplastic samples (in a few categories). The data in the above publication was processed with an MSFragger/Philosopher data analysis pipeline (github.com/Nesvilab/philosopher) using a reporter ion ratio oriented methodology. The use of ratios and log transformations make the analysis less transparent and not easy to visually check.
In this notebook, I will show that high quality analyses can be done without using ratios. The 27 samples in the experiment were split across 3 10-plex experiments (8-fraction separations for each). The 131 channel contained a pooled reference channel. The pooled channel was used for Internal Reference Scaling (IRS) (Ref. 4) so that all of the data could be kept in its original intensity scale. The Comet/PAW pipeline (Ref. 5) was used for highly sensitive PSM identification and robust TMT reporter ion processing.
The PAW pipeline uses accurate mass measurements from modern instruments in a flexible way maximize PSM identifications. Extended parsimony principles maximize the quantiative information content compared to cumbersome razor peptide approaches. The two-peptide rule is employed for accurate protein identification without any reliance on ad hoc protein ranking functions. Very low intensity reporter ion scans are filtered out and any remaining missing reporter ion data is handled in a simple way.
There were 754,130 MS scans from the 24 RAW files. There were 266,723 PSMs at 1% FDR from the PAW processing. The publication reported 251,305 at a similar FDR. The PAW protein inference had 5,127 non-contaminant, non-decoy proteins (proteins, protein groups, or protein families) with at least two distinct peptides per protein per plex. There were 5,087 proteins that had some reporter ion signals in at least one plex.
After the IRS step, there were 4,132 quantifiable proteins seen in each plex. This was 81% of the total number of protein identifications. Loss of 955 proteins might seem concerning at first glance. However, the 4,132 proteins account for 99.55% of the total reporter ion intensity in the 3-plex experiment. The 955 rejected proteins were only 0.45% of the total signal. If we compute the average IRS adjusted reporter ion intensity for each of the 4,132 proteins and sort them by decreasing average intensity, the last protein has an average intensity of 1,032. The smallest non-zero reporter ion signals in individual scans was around 500 (Orbitrap intensities have some small minimum value (we see 350-ish on our Fusion, this data had about 500) or the data is missing). We require two peptides and 2 times 500 is 1,000 and that is right where the cutoff is.
Another important point with TMT data and missing data for multi-plex experiments is that data end up missing by plex rather than randomly by individual data point. This is likely something no missing data imputation algorithm was designed to consider. The 4,132 proteins only have 242 missing intensities out of 111,564 data points (4132 times 27) or 0.2% missing values. The 955 proteins have 25,785 data points with 11,454 missing data points, for 44.4% missing. Practically speaking, nearly all of the missing data (98%) come from those 955 proteins.
Aside: It is human nature to focus on what is missing rather than what is present. Robust quantification of over 4,000 proteins was unheard of just a few years ago. Starting with more sample and using more fractions is the best way to quantify the 955 "lost" proteins. The experiment could also be repeated on a Lumos or Eclipse to get more identifications and more intense reporter ion signals. Performing a second, better experiment is always more productive than over-analyzing data.
The IRS adjusted reporter ion intensities were analyzed with Jupyter notebooks, an R kernel, the Bioconductor package edgeR (Ref. 6), and its TMM normalization method (Ref. 7). This notebook format allows rich data visualization and quality control checks.
Ref. 1. Thompson, A., Schäfer, J., Kuhn, K., Kienle, S., Schwarz, J., Schmidt, G., Neumann, T. and Hamon, C., 2003. Tandem mass tags: a novel quantification strategy for comparative analysis of complex protein mixtures by MS/MS. Analytical chemistry, 75(8), pp.1895-1904.
Ref. 2. McAlister, G.C., Nusinow, D.P., Jedrychowski, M.P., Wühr, M., Huttlin, E.L., Erickson, B.K., Rad, R., Haas, W. and Gygi, S.P., 2014. MultiNotch MS3 enables accurate, sensitive, and multiplexed detection of differential expression across cancer cell line proteomes. Analytical chemistry, 86(14), pp.7150-7158.
Ref. 3. Senko, M.W., Remes, P.M., Canterbury, J.D., Mathur, R., Song, Q., Eliuk, S.M., Mullen, C., Earley, L., Hardman, M., Blethrow, J.D. and Bui, H., 2013. Novel parallelized quadrupole/linear ion trap/Orbitrap tribrid mass spectrometer improving proteome coverage and peptide identification rates. Analytical chemistry, 85(24), pp.11710-11714.
Ref. 4. Plubell, D.L., Wilmarth, P.A., Zhao, Y., Fenton, A.M., Minnier, J., Reddy, A.P., Klimek, J., Yang, X., David, L.L. and Pamir, N., 2017. Extended multiplexing of tandem mass tags (TMT) labeling reveals age and high fat diet specific proteome changes in mouse epididymal adipose tissue. Molecular & Cellular Proteomics, 16(5), pp.873-890.
Ref. 5. Wilmarth, P.A., Riviere, M.A. and David, L.L., 2009. Techniques for accurate protein identification in shotgun proteomic studies of human, mouse, bovine, and chicken lenses. Journal of ocular biology, diseases, and informatics, 2(4), pp.223-234.
Ref. 6. Robinson, M.D., McCarthy, D.J. and Smyth, G.K., 2010. edgeR: a Bioconductor package for differential expression analysis of digital gene expression data. Bioinformatics, 26(1), pp.139-140.
Ref. 7. Robinson, M.D. and Oshlack, A., 2010. A scaling normalization method for differential expression analysis of RNA-seq data. Genome biology, 11(3), p.R25.
# library imports
library(tidyverse)
library(scales)
library(limma)
library(edgeR)
library(psych)
# ============== CV function ===================================================
CV <- function(df) {
# Computes CVs of data frame rows
# df - data frame,
# returns vector of CVs (%)
ave <- rowMeans(df) # compute averages
sd <- apply(df, 1, sd) # compute standard deviations
cv <- 100 * sd / ave # compute CVs in percent (last thing gets returned)
}
# =========== Boxplot with median label ========================================
labeled_boxplot <- function(df, ylim, title) {
# Makes a box plot with the median value labeled
# df - data frame with data to compute CVs of
# ylim - upper limit for y-axis
# title - plot title
cv = CV(df)
boxplot(cv, ylim = c(0, ylim), notch = TRUE, main = title)
text(x = 0.65, y = boxplot.stats(cv)$stats[3],
labels = round(boxplot.stats(cv)$stats[3], 1))
}
# ================== TMM normalization from DGEList object =====================
apply_tmm_factors <- function(y, color = NULL, plot = TRUE) {
# computes the tmm normalized data from the DGEList object
# y - DGEList object
# returns a dataframe with normalized intensities
# compute and print "Sample loading" normalization factors
lib_facs <- mean(y$samples$lib.size) / y$samples$lib.size
cat("\nLibrary size factors:\n",
sprintf("%-5s -> %f\n", colnames(y$counts), lib_facs))
# compute and print TMM normalization factors
tmm_facs <- 1/y$samples$norm.factors
cat("\nTrimmed mean of M-values (TMM) factors:\n",
sprintf("%-5s -> %f\n", colnames(y$counts), tmm_facs))
# compute and print the final correction factors
norm_facs <- lib_facs * tmm_facs
cat("\nCombined (lib size and TMM) normalization factors:\n",
sprintf("%-5s -> %f\n", colnames(y$counts), norm_facs))
# compute the normalized data as a new data frame
tmt_tmm <- as.data.frame(sweep(y$counts, 2, norm_facs, FUN = "*"))
colnames(tmt_tmm) <- str_c(colnames(y$counts), "_tmm")
# visualize results and return data frame
if(plot == TRUE) {
boxplot(log10(tmt_tmm), col = color, notch = TRUE, main = "TMM Normalized data")
}
tmt_tmm
}
# ================= reformat edgeR test results ================================
collect_results <- function(df, tt, x, xlab, y, ylab) {
# Computes new columns and extracts some columns to make results frame
# df - data in data.frame
# tt - top tags table from edgeR test
# x - columns for first condition
# xlab - label for x
# y - columns for second condition
# ylab - label for y
# returns a new dataframe
# condition average vectors
ave_x <- rowMeans(df[x])
ave_y <- rowMeans(df[y])
# FC, direction, candidates
fc <- ifelse(ave_y > ave_x, (ave_y / ave_x), (-1 * ave_x / ave_y))
direction <- ifelse(ave_y > ave_x, "up", "down")
candidate <- cut(tt$FDR, breaks = c(-Inf, 0.01, 0.05, 0.10, 1.0),
labels = c("high", "med", "low", "no"))
# make data frame
temp <- cbind(df[c(x, y)], data.frame(logFC = tt$logFC, FC = fc,
PValue = tt$PValue, FDR = tt$FDR,
ave_x = ave_x, ave_y = ave_y,
direction = direction, candidate = candidate,
Acc = tt$genes))
# fix column headers for averages
names(temp)[names(temp) %in% c("ave_x", "ave_y")] <- str_c("ave_", c(xlab, ylab))
temp # return the data frame
}
# =============== p-value plots ================================================
pvalue_plots <- function(results, ylim, title) {
# Makes p-value distribution plots
# results - results data frame
# ylim - ymax for expanded view
# title - plot title
p_plot <- ggplot(results, aes(PValue)) +
geom_histogram(bins = 100, fill = "white", color = "black") +
geom_hline(yintercept = mean(hist(results$PValue, breaks = 100,
plot = FALSE)$counts[26:100]))
# we will need an expanded plot
p1 <- p_plot + ggtitle(str_c(title, " p-value distribution"))
p2 <- p_plot + coord_cartesian(xlim = c(0, 1.0), ylim = c(0, ylim)) + ggtitle("p-values expanded")
grid.arrange(p1, p2, nrow = 2) # from gridExtra package
}
# ============= log2 fold-change distributions =================================
log2FC_plots <- function(results, range, title) {
# Makes faceted log2FC plots by candidate
# results - results data frame
# range - plus/minus log2 x-axis limits
# title - plot title
ggplot(results, aes(x = logFC, fill = candidate)) +
geom_histogram(binwidth=0.1, color = "black") +
facet_wrap(~candidate) +
ggtitle(title) +
coord_cartesian(xlim = c(-range, range))
}
# ========== Setup for MA and volcano plots ====================================
transform <- function(results, x, y) {
# Make data frame with some transformed columns
# results - results data frame
# x - columns for x condition
# y - columns for y condition
# return new data frame
df <- data.frame(log10((results[x] + results[y])/2),
log2(results[y] / results[x]),
results$candidate,
-log10(results$FDR))
colnames(df) <- c("A", "M", "candidate", "P")
df # return the data frame
}
# ========== MA plots using ggplot =============================================
MA_plots <- function(results, x, y, title) {
# makes MA-plot DE candidate ggplots
# results - data frame with edgeR results and some condition average columns
# x - string for x-axis column
# y - string for y-axis column
# title - title string to use in plots
# returns a list of plots
# uses transformed data
temp <- transform(results, x, y)
# 2-fold change lines
ma_lines <- list(geom_hline(yintercept = 0.0, color = "black"),
geom_hline(yintercept = 1.0, color = "black", linetype = "dotted"),
geom_hline(yintercept = -1.0, color = "black", linetype = "dotted"))
# make main MA plot
ma <- ggplot(temp, aes(x = A, y = M)) +
geom_point(aes(color = candidate, shape = candidate)) +
scale_y_continuous(paste0("logFC (", y, "/", x, ")")) +
scale_x_continuous("Ave_intensity") +
ggtitle(title) +
ma_lines
# make separate MA plots
ma_facet <- ggplot(temp, aes(x = A, y = M)) +
geom_point(aes(color = candidate, shape = candidate)) +
scale_y_continuous(paste0("log2 FC (", y, "/", x, ")")) +
scale_x_continuous("log10 Ave_intensity") +
ma_lines +
facet_wrap(~ candidate) +
ggtitle(str_c(title, " (separated)"))
# make the plots visible
print(ma)
print(ma_facet)
}
# ========== Scatter plots using ggplot ========================================
scatter_plots <- function(results, x, y, title) {
# makes scatter-plot DE candidate ggplots
# results - data frame with edgeR results and some condition average columns
# x - string for x-axis column
# y - string for y-axis column
# title - title string to use in plots
# returns a list of plots
# 2-fold change lines
scatter_lines <- list(geom_abline(intercept = 0.0, slope = 1.0, color = "black"),
geom_abline(intercept = 0.301, slope = 1.0, color = "black", linetype = "dotted"),
geom_abline(intercept = -0.301, slope = 1.0, color = "black", linetype = "dotted"),
scale_y_log10(),
scale_x_log10())
# make main scatter plot
scatter <- ggplot(results, aes_string(x, y)) +
geom_point(aes(color = candidate, shape = candidate)) +
ggtitle(title) +
scatter_lines
# make separate scatter plots
scatter_facet <- ggplot(results, aes_string(x, y)) +
geom_point(aes(color = candidate, shape = candidate)) +
scatter_lines +
facet_wrap(~ candidate) +
ggtitle(str_c(title, " (separated)"))
# make the plots visible
print(scatter)
print(scatter_facet)
}
# ========== Volcano plots using ggplot ========================================
volcano_plot <- function(results, x, y, title) {
# makes a volcano plot
# results - a data frame with edgeR results
# x - string for the x-axis column
# y - string for y-axis column
# title - plot title string
# uses transformed data
temp <- transform(results, x, y)
# build the plot
ggplot(temp, aes(x = M, y = P)) +
geom_point(aes(color = candidate, shape = candidate)) +
xlab("log2 FC") +
ylab("-log10 FDR") +
ggtitle(str_c(title, " Volcano Plot"))
}
# ============== individual protein expression plots ===========================
# function to extract the identifier part of the accesssion
get_identifier <- function(accession) {
identifier <- str_split(accession, "\\|", simplify = TRUE)
identifier[,3]
}
set_plot_dimensions <- function(width_choice, height_choice) {
options(repr.plot.width=width_choice, repr.plot.height=height_choice)
}
plot_top_tags <- function(results, nleft, nright, top_tags) {
# results should have data first, then test results (two condition summary table)
# nleft, nright are number of data points in each condition
# top_tags is number of up and number of down top DE candidates to plot
# get top ipregulated
up <- results %>%
filter(logFC >= 0) %>%
arrange(FDR)
up <- up[1:top_tags, ]
# get top down regulated
down <- results %>%
filter(logFC < 0) %>%
arrange(FDR)
down <- down[1:top_tags, ]
# pack them
proteins <- rbind(up, down)
color = c(rep("red", nleft), rep("blue", nright))
for (row_num in 1:nrow(proteins)) {
row <- proteins[row_num, ]
vec <- as.vector(unlist(row[1:(nleft + nright)]))
names(vec) <- colnames(row[1:(nleft + nright)])
title <- str_c(get_identifier(row$Acc), ", int: ", scientific(mean(vec), 2),
", FDR: ", scientific(row$FDR, digits = 3),
", FC: ", round(row$FC, digits = 1),
", ", row$candidate)
barplot(vec, col = color, main = title,
cex.main = 1.0, cex.names = 0.7, cex.lab = 0.7)
}
}
The pandas Python script that does the IRS normalization arranges the tab-delimited table so that importing into R is relatively straightforward. Note than this experimental design has only a single reference channel per plex instead of two.
We need to drop contaminant and decoy proteins, and proteins with missing sets of reporter ions. We extract the accessions column, the SL and IRS normed data columns, and collect channels by biological condition. We also grab the reference channels to check the IRS procedure.
# load the IRS-normalized data and check the table
data_import <- read_tsv("labeled_grouped_protein_summary_TMT_9_IRS_normalized.txt", guess_max = 5250)
# "Filter" column flags contams and decoys
# "Missing" column flags proteins without reporter ion intensities (full sets missing)
# the table from pandas is sorted so the rows we want come first
data_all <- filter(data_import, is.na(Filter), is.na(Missing))
data_sl <- data_all %>% select(., contains("SLNorm_")) %>%
select(., -contains("_Unused")) %>%
select(., -contains("_Pool"))
data_irs <- data_all %>% select(., contains("IRSNorm_")) %>%
select(., -contains("_Unused")) %>%
select(., -contains("_Pool"))
data_pool <- data_all %>% select(., contains("_Pool_"))
# save a few columns for the results table
all_results <- data_all %>% select(., ProtGroup, Counter, Accession, Description, starts_with("PSMs_Used"))
# save gene names for edgeR so we can double check that results line up
accessions <- data_all$Accession
# see how many rows of data we have
length(accessions)
We will lump all the metaplastic breast cancer (MBC) samples together to contrast to triple negative breast cancer and normal tissue samples. There will be 14 mbc, 6 triple negative, and 7 normal samples.
# all categories of metaplastic breast cancer tissue
mbc_sl <- select(data_sl, contains("_C"), contains("_SP"), contains("_SQ"))
mbc_irs <- select(data_irs, contains("_C"), contains("_SP"), contains("_SQ"))
# triple negative breast cancer tissue
tn_sl <- select(data_sl, contains("_TN"))
tn_irs <- select(data_irs, contains("_TN"))
# Normal tissue
n_sl <- select(data_sl, contains("_N"))
n_irs <- select(data_irs, contains("_N"))
As part of the IRS script, the total intensities per channel across all three plexes are matched (sample-loading normalization). The IRS method corrects for pseudo random scan selection by the instrument using the pooled reference channel(s). We can look at boxplots after SL-normalization and after IRS-normalization.
# collect the biological replicate channels
bio_sl <- cbind(n_sl, tn_sl, mbc_sl)
bio_irs <- cbind(n_irs, tn_irs, mbc_irs)
# set a color vector for plots
color <- c(rep("red", 7), rep("blue", 6), rep("green", 14))
# boxplots
boxplot(log10(bio_sl), col = color, notch = TRUE, main = "SL Normalized data")
boxplot(log10(bio_irs), col = color, notch = TRUE, main = "IRS Normalized data")
The above boxplots did not look too different. Boxplots are not sufficient to verify that normalization worked as expected. Clustering is an orthogonal way to check things. We should have samples cluster by biological conditions if we have successfully removed other sources of measurement variation.
# clustering of the replicates before IRS
plotMDS(log2(bio_sl), col = color, main = "Replicates before IRS")
# clustering after IRS
plotMDS(log2(bio_irs), col = color, main = "Replicates after IRS")
Without IRS, the samples cluster by TMT plex. After IRS we recover the biological groupings. While IRS might look like other "batch correction" methods, it is actually superior. IRS does not rely on balanced study designs and is more reliable with the relatively small numbers of channels in TMT experiments.
We can also look at the scatter plots of samples to each other within each biological group. We will do that before and after IRS.
# the normal tissue samples (includes the one mis-classified sample)
pairs.panels(log10(n_sl), lm = TRUE, main = "Normal, SLNorm")
pairs.panels(log10(n_irs), lm = TRUE, main = "Normal, IRSNorm")
We have correlation coefficients of around 0.95 for samples within the same plex, and correlation coefficients of around 0.85 between plexes before IRS. After IRS, all correlation coefficients are in the mid to upper 0.9s.
# the triple negative samples
pairs.panels(log10(tn_sl), lm = TRUE, main = "Triple Negative, SLNorm")
pairs.panels(log10(tn_irs), lm = TRUE, main = "Triple Negative, IRSNorm")
# the MBC C samples
pairs.panels(log10(mbc_sl[1:4]), lm = TRUE, main = "Metaplastic C, SLNorm")
pairs.panels(log10(mbc_irs[1:4]), lm = TRUE, main = "Metaplastic C, IRSNorm")
# the MBC Sp samples
pairs.panels(log10(mbc_sl[5:10]), lm = TRUE, main = "Metaplastic Sp, SLNorm")
pairs.panels(log10(mbc_irs[5:10]), lm = TRUE, main = "Metaplastic Sp, IRSNorm")
# the MBC Sq samples
pairs.panels(log10(mbc_sl[11:14]), lm = TRUE, main = "Metaplastic Sq, SLNorm")
pairs.panels(log10(mbc_irs[11:14]), lm = TRUE, main = "Metaplastic Sq, IRSNorm")
This is a little artificial because IRS will make the reference channels identical. The before IRS plots (SLNorm) show how the random sampling distorts the data. These samples are chemically/biologically identical and should yield very similar intensities.
# check the pooled standards
pairs.panels(log10(data_pool[1:3]), lm = TRUE, main = "Pools, SLNorm")
pairs.panels(log10(data_pool[4:6]), lm = TRUE, main = "Pools, IRSNorm")
We are defining the groups that will be compared explicitly and using all of the samples for variance estimates (similar to pooled variance methods). We will put the data into a data frame, grouped by condition. We will define some column indexes for each condition, set some colors for plotting, and see how the data cluster by condition.
# put groups together into a single data frame
tmt_sl <- bio_sl
tmt_irs <- bio_irs
# define the positions of the groups
N <- 1:7
TN <- 8:13
MBC <- 14:27
# set some colors by condition
group <- c(rep("N", 7), rep("TN", 6), rep("MBC", 14))
We will load the data into edgeR data structures and call the calcNormFactors
function to perform library size and the trimmed mean of M-values (TMM) normalization. We will double check if the TMM normalization changed the clustering that we had above.
We need to use the edgeR normalization factors to produce the TMM normalized data that the statistical testing will be working with. EdgeR uses the normalization factors in its statistical modeling but does not output the normalized intensities. We compute the normalized intensities with the apply_tmm_factors function
.
# get the biological sample data into a DGEList object
y <- DGEList(counts = tmt_irs, group = group, genes = accessions)
# run TMM normalization (also includes a library size factor)
y <- calcNormFactors(y)
tmt_tmm <- apply_tmm_factors(y, color = color)
# check the clustering
plotMDS(y, col = color, main = "all samples after TMM")
The IRS method already takes care of the library size type of normalizations. We could still have compositional differences between sample groups that the TMM factors would correct for. We do not have any library size or TMM factors that are too much different from 1.0.
The box plots are very similar after TMM. TMM usually gets the medians (box centers) aligned pretty well.
# put CVs in data frames to simplify plots and summaries
cv_sl <- data.frame(N = CV(bio_sl[N]), TN = CV(bio_sl[TN]), MBC = CV(bio_sl[MBC]))
cv_irs <- data.frame(N = CV(bio_irs[N]), TN = CV(bio_irs[TN]), MBC = CV(bio_irs[MBC]))
cv_tmm <- data.frame(N = CV(tmt_tmm[N]), TN = CV(tmt_tmm[TN]), MBC = CV(tmt_tmm[MBC]))
# see what the median CV values are
medians <- apply(cv_sl, 2, FUN = median)
print("SLNorm median CVs by condition (%)")
round(medians, 2)
medians <- apply(cv_irs, 2, FUN = median)
print("IRSNorm median CVs by condition (%)")
round(medians, 2)
medians <- apply(cv_tmm, 2, FUN = median)
print("Final median CVs by condition (%)")
round(medians, 2)
We have about 50% median CVs before IRS. IRS reduces those to around 30%. TMM makes some small additional improvements.
# see what the CV distibutions look like
# need long form for ggplot
long_cv_sl <- gather(cv_sl, key = "group", value = "cv")
# traditional boxplots
ggplot(long_cv_sl, aes(x = group, y = cv, fill = group)) +
geom_boxplot(notch = TRUE) +
ggtitle("SL CV distributions")
# density plots
ggplot(long_cv_sl, aes(x = cv, color = group)) +
geom_density() +
coord_cartesian(xlim = c(0, 200)) +
ggtitle("SL CV distributions")
# need long form for ggplot
long_cv_tmm <- gather(cv_tmm, key = "group", value = "cv")
# traditional boxplots
ggplot(long_cv_tmm, aes(x = group, y = cv, fill = group)) +
geom_boxplot(notch = TRUE) +
ggtitle("Final CV distributions")
# density plots
ggplot(long_cv_tmm, aes(x = cv, color = group)) +
geom_density() +
coord_cartesian(xlim = c(0, 200)) +
ggtitle("Final CV distributions")
One of the most powerful features of edgeR (and limma) is computing variances across larger numbers of genes (proteins) to get more robust variance estimates for small replicate experiments. Here, we have 27 samples across all conditions to use to improve the variance estimates and reduce false positive differential expression (DE) candidates. We have an edgeR estimateDisp
function that does all of this and a visualization function to check the result.
We loaded the IRS data into DGEList
object y
a few cells above and did the normalization step. We need to estimate the dispersion parameters before we can do the actual statistical testing. This only needs to be done once. Each exact test will take two conditions and compare them using the normalization factors and dispersion estimates saved in y
.
# compute dispersions and plot BCV
y <- estimateDisp(y)
plotBCV(y, main = "BCV plot of IRS/TMM normalized data")
We will specify the pair of interest for the exact test in edgeR and use the experiment-wide dispersion. The decideTestsDGE
call will tell us how many up and down regulated candidates we have at an FDR of 0.10. The topTags
call does the Benjamini-Hochberg multiple testing corrections. We save the test results in tt
. We use a handy MA (mean-average) plotting function from limma
to visualize the DE candidates, and then check the p-value distribution.
# the exact test object has columns like fold-change, CPM, and p-values
et <- exactTest(y, pair = c("N", "TN"))
# this counts up, down, and unchanged genes (proteins) at 10% FDR
summary(decideTestsDGE(et, p.value = 0.10))
# the topTags function adds the BH FDR values to an exactTest data frame
# make sure we do not change the row order (the sort.by parameter)!
topTags(et)$table
tt <- topTags(et, n = Inf, sort.by = "none")
# make an MD plot (like MA plot)
plotMD(et, p.value = 0.10)
abline(h = c(-1, 1), col = "black")
# check the p-value distribution
ggplot(tt$table, aes(PValue)) +
geom_histogram(bins = 100, fill = "white", color = "black") +
geom_hline(yintercept = mean(hist(et$table$PValue, breaks = 100,
plot = FALSE)$counts[26:100])) +
ggtitle("N versus TN p-value distribution")
We have about 22% of the proteins with differential expression. The p-value distribution looks good. We have a flat distribution from non-DE proteins and a large spike at low p-values from the true DE candidates.
We will add the statistical testing results (logFC, p-values, and FDR), condition intensity averages, and candidate status to the results
data frame (which has the TMM-normalized data) and also accumulate all of the three comparisons into all_results
.
Even though we do not have many candidates, we will still make the MA plots, scatter plots, and volcano plot using ggplot2. We will also look at the distribution of log2 expression ratios separated by differential expression (DE) category. The Benjamini-Hochberg corrected edgeR p-values are used to categorize the DE candidates: no > 0.10 > low > 0.05 > med > 0.01 > high.
# get the results summary
results <- collect_results(tmt_tmm, tt$table, N, "N", TN, "TN")
# make column names unique by adding comparison (for the accumulated frame)
results_temp <- results
colnames(results_temp) <- str_c(colnames(results), "_N_TN")
# accumulate the testing results
all_results <- cbind(all_results, results_temp)
# see how many candidates by category
results %>% count(candidate)
# plot log2 fold-changes by category
ggplot(results, aes(x = logFC, fill = candidate)) +
geom_histogram(binwidth=0.1, color = "black") +
facet_wrap(~candidate) +
coord_cartesian(xlim = c(-4, 4)) +
ggtitle("N vs TN logFC distributions by candidate")
We have many comparisons to visualize, so we will use functions to generate a series of plots. We will make: an MA plot with candidates highlighted by color, faceted MA plots separated by candidate status, a scatter plot with candidates highlighted by color, faceted scatter plots separated by candidate status, and a volcano plot with candidates highlighted by color. The solid black lines in the MA and scatter plots are the 1-to-1 lines; the dotted lines are 2-fold change lines.
# make MA plots
MA_plots(results, "ave_N", "ave_TN", "N vs TN")
# make scatter plots
scatter_plots(results, "ave_N", "ave_TN", "N vs TN")
# make a volcano plot
volcano_plot(results, "ave_N", "ave_TN", "N vs TN")
We have robust differential expression changes. We have DE candidates that look convincing in several different views. There are many proteins that are down regulated significantly (large negative fold changes) in breast cancer.
We can look at the data for the top 20 (by edgeR BH-corrected p-values) up and down regulated proteins.
# look at the top 20 candidates in each direction (up in TN, then down in TN)
set_plot_dimensions(6, 3.5)
plot_top_tags(results, 7, 6, 20)
set_plot_dimensions(7, 7)
We will do the same testing to compare normal tissue to metaplastic breast cancer samples.
# the exact test object has columns like fold-change, CPM, and p-values
et <- exactTest(y, pair = c("N", "MBC"))
# this counts up, down, and unchanged genes (proteins) at 10% FDR
summary(decideTestsDGE(et, p.value = 0.10))
# the topTags function adds the BH FDR values to an exactTest data frame
# make sure we do not change the row order (the sort.by parameter)!
topTags(et)$table
tt <- topTags(et, n = Inf, sort.by = "none")
# make an MD plot (like MA plot)
plotMD(et, p.value = 0.10)
abline(h = c(-1, 1), col = "black") # 2-fold change lines
# check the p-value distribution
ggplot(tt$table, aes(PValue)) +
geom_histogram(bins = 100, fill = "white", color = "black") +
geom_hline(yintercept = mean(hist(et$table$PValue, breaks = 100,
plot = FALSE)$counts[26:100])) +
ggtitle("N vs MBC p-value distribution")
We had about 900 candidates in the normal versus triple negative. We have around 1,270 here. The p-value distribution looks good.
We will add the statistical testing results (logFC, p-values, and FDR), condition intensity averages, and candidate status to the results
data frame (which has the TMM-normalized data) and also accumulate all of the three comparisons into all_results
.
# get the results summary
results <- collect_results(tmt_tmm, tt$table, N, "N", MBC, "MBC")
# make column names unique by adding comparison
results_temp <- results
colnames(results_temp) <- str_c(colnames(results), "_N_MBC")
# accumulate the testing results
all_results <- cbind(all_results, results_temp)
# see how many candidates by category
results %>% count(candidate)
# plot log2 fold-changes by category
ggplot(results, aes(x = logFC, fill = candidate)) +
geom_histogram(binwidth=0.1, color = "black") +
facet_wrap(~candidate) +
coord_cartesian(xlim = c(-4, 4)) +
ggtitle("N vs MBC logFC distributions by candidate")
# make MA plots
MA_plots(results, "ave_N", "ave_MBC", "N vs MBC")
# make scatter plots
scatter_plots(results, "ave_N", "ave_MBC", "N vs MBC")
# make a volcano plot
volcano_plot(results, "ave_N", "ave_MBC", "N vs MBC")
We have similar more robust down-regulation in breast cancer compared to normal tissue. We may have more DE candidates compared to the first comparison because we have 14 MBC samples compared to 6 for triple negative. Generally, we should have more statistical power with larger replicate numbers (all things being equal).
# look at the top 20 candidates (up in MBC, then down in MBC)
set_plot_dimensions(6, 3.5)
plot_top_tags(results, 7, 14, 20)
set_plot_dimensions(7, 7)
There are some sample to sample variations in line with what one expects for human subjects. However, we do have clear differences in expression levels. Given the moderate numbers of DE candidates, enrichment and pathway analyses are likely to provide a good index into the biology.
We can compare the non-metaplastic breast cancer samples to the metaplastic samples. These samples were not very distinct from each other in the cluster plots.
# the exact test object has columns like fold-change, CPM, and p-values
et <- exactTest(y, pair = c("TN", "MBC"))
# this counts up, down, and unchanged genes (proteins) at 10% FDR
summary(decideTestsDGE(et, p.value = 0.10))
# the topTags function adds the BH FDR values to an exactTest data frame
# make sure we do not change the row order (the sort.by parameter)!
topTags(et)$table
tt <- topTags(et, n = Inf, sort.by = "none")
# make an MD plot (like MA plot)
plotMD(et, p.value = 0.10)
abline(h = c(-1, 1), col = "black")
# check the p-value distribution
ggplot(tt$table, aes(PValue)) +
geom_histogram(bins = 100, fill = "white", color = "black") +
geom_hline(yintercept = mean(hist(et$table$PValue, breaks = 100,
plot = FALSE)$counts[26:100])) +
ggtitle("TN vs MBC p-value distribution")
We have a reasonable p-value distribution that is consistent with the reduced number of DE candidates. The variance in the samples is likely a factor in the smaller number of candidates. We probably have smaller differences in means between these samples compared to normal tissue, so the p-values are smaller.
We will add the statistical testing results (logFC, p-values, and FDR), condition intensity averages, and candidate status to the results
data frame (which has the TMM-normalized data) and also accumulate all of the three comparisons into all_results
.
Even though we have fewer candidates, we will still make the MA plot, scatter plot, and volcano plot.
# get the results summary
results <- collect_results(tmt_tmm, tt$table, TN, "TN", MBC, "MBC")
# make column names unique by adding comparison
results_temp <- results
colnames(results_temp) <- str_c(colnames(results), "_TN_MBC")
# accumulate the testing results
all_results <- cbind(all_results, results_temp)
# see how many candidates by category
results %>% count(candidate)
# plot log2 fold-changes by category
ggplot(results, aes(x = logFC, fill = candidate)) +
geom_histogram(binwidth=0.1, color = "black") +
facet_wrap(~candidate) +
coord_cartesian(xlim = c(-4, 4)) +
ggtitle("TN vs MBC logFC distributions by candidate")
# make MA plots
MA_plots(results, "ave_TN", "ave_MBC", "TN vs MBC")
# make scatter plots
scatter_plots(results, "ave_TN", "ave_MBC", "TN vs MBC")
# make a volcano plot
volcano_plot(results, "ave_TN", "ave_MBC", "TN vs MBC")
# look at the top 10 candidates (up in MBC, then down in MBC)
set_plot_dimensions(6, 3.5)
plot_top_tags(results, 6, 14, 10)
set_plot_dimensions(7, 7)
We seem to have a few individual samples in the groups that drive the mean differences for these top candidates (at least they seem less consistent that in the first two comparisons). One might expect cancer samples to be more divergent compared to normal tissue. Comparing cancer samples to cancer samples might be more difficult.
We had significant differences between normal tissue samples and either set of cancer samples. It might take some careful comparing and contrasting of the DE candidates to see what proteins might be involved to make metaplastic cancer more aggressive than non-metaplastic breast cancer. We will see how the different sub-categories of the metaplastic samples compare in the second notebook.
all_results
frame to TSV file¶# write the results to disk
write.table(all_results, "three_tissues_results.txt", sep = "\t",
row.names = FALSE, na = " ")
# log the session details
sessionInfo()


public interface HGEventManager extends HGGraphHolder
A HGEventManager
is bound to a single HyperGraph instance. It is responsible for holding
all event listeners and it performs event handling via its dispatch
method. Concrete
implementations can be configured with the HGConfiguration
object when the database is
opened.
Event types are simply identified by their Java class objects.
| Modifier and Type | Method and Description |
|---|---|
<T extends HGEvent> |
addListener(java.lang.Class<T> eventType,
HGListener listener)
Register a new listener for a given event type.
|
void |
clear()
Removal all event listeners for all event types.
|
HGListener.Result |
dispatch(HyperGraph graph,
HGEvent event)
Dispatch an event to all listeners registered for its type.
|
<T extends HGEvent> |
removeListener(java.lang.Class<T> eventType,
HGListener listener)
Remove a listener registered for the particular type.
|
setHyperGraph
<T extends HGEvent> void addListener(java.lang.Class<T> eventType, HGListener listener)
T
- eventType
- listener
- <T extends HGEvent> void removeListener(java.lang.Class<T> eventType, HGListener listener)
T
- eventType
- listener
- void clear()
HGListener.Result dispatch(HyperGraph graph, HGEvent event)
Dispatch an event to all listeners registered for its type. All listeners
are invoked in the order in which they were registered. If a listener returns
a HGListener.Result.cancel
event, the dispatch process is interrupted and
the cancellation is passed on to the caller of the dispatch method without invoking
any further listeners.
Because event types can be organized in hierarchy, rooted at HGEvent
, a dispatcher
must call listener registered for the particular event class and also all its superclasses up
to the HGEvent
type itself.
graph
- event
-


 What assumptions did you make, and what tradeoffs did you consider?"
- I chose to built a Single page application.
- Client: Meteor
- Server: Meteor/Node
- DB: MONGODB(local)
- Fast Prototyping
- EASY to install. Just 2 lines of code on any machine!!!!!!!!!!!!!!
- The installation process will ONLY take ~2 mins
- Meteor will setup Mongo for you. Database gets populated automatically
- Isomorphic JavaScript
- You start on / route and need to login
- else Gandalph yells at you
- When you are logged in
- you can discover the awesome features
- This app has still a few bugs
- it is only for promotion purposes
- You can still access the whole collection -Try: Orders.find() or Orders.find().count() in browser console


EDTV Cast: Matthew McConaughey, Jenna Elfman, Ellen DeGeneres, Woody Harrelson, Martin Landau, Sally Kirkland, Rob Reiner, Dennis Hopper, Elizabeth Hurley Director: Ron Howard Screenplay: Lowell Ganz & Babaloo Mandel Reviewed by Luke Buckmaster
On the Buckmaster scale of 0 stars (bomb), to 5 stars (a masterpiece): 2 and a half stars
Even though it's been about twenty years since Ron Howard starred in 'Happy Days,' I still giggle after reading the words "a Ron Howard film." There's just something about picturing Ritchie Cunningham directing a Hollywood blockbuster that you can't help but laugh at. Howard's new movie EDtv will inevitably be compared to Peter Weir's film The Truman Show, which has similar themes. But whereas Truman focused on a character who was oblivious to his world-wide fame as a star of his own TV show, the protagonist in EDtv eagerly volunteers himself for a media experiment, only to realize that perhaps it wasn't such a good idea. Howard has proved himself more than capable of handling heavy material after the successes of Apollo 13 and Ransom, but here he doesn't manage to direct with the kind of cynical detachment that made Weir's piece succeed so well.
After two years worth of bad ratings, cable television channel 'True TV' decides to try something different. Headed by program director Cynthia Topping (Ellen DeGeneres), they plan to create a show that follows, for 24 hours a day, the life of an average Joe.  Thus, 'EDtv' is born. Sure enough, a camera crew follows Ed everywhere he goes, and it doesn't take long before Ed's life becomes a lot more complicated. Trouble arises when he catches his brother Ray (Woody Harrelson) having an affair on national TV, which leaves Ray's girlfriend Shari (Jenna Elfman) predictably furious. Shari and Ed are attracted to each other and carve a relationship of their own, whilst Ed's long-lost father Hank (Dennis Hopper) arrives on the scene and causes domestic problems with Ed's flamboyant mother Janette (Sally Kirkland).
If cameras didn't follow Ed everywhere, I'd swear that this was just another average drama. At least that's the way that Howard approaches his material; instead of focusing on the media's influence on society, he merely gives a twist to a typical domestic story. Most frustrating in his direction is his lack of attention to interesting themes. When Ed experiences his fifteen minutes of fame, it's clear that he is no longer an average citizen, and thus 'True TV' loses the very thing that made it so popular - its ability to follow an ordinary, fame-less citizen. But the 'True TV' executives are just as oblivious to this as Howard is. He obviously wanted to keep the film simplistic and able to succeed on its own entertaining merits, thus much intrigue is lost.
Edtv consistently tries to persuade its audience that Ed is a character who deserves to be barracked for. Matthew McConaughey mouths his dialogue with the kind of Southern American simplicity that quickly becomes irritating, and although there is nothing drastically wrong with his performance, the film's script doesn't have the strength to support him. Ed succeeds as a likeable nice guy, but hardly somebody who will have you cheering in the aisles (unlike Truman, who's portrayal by Jim Carrey made him impossible not to like).
EDtv is entertaining and sometimes quite funny - after all, this is A Ron Howard Film. But the media-induced world in which the film's characters revolve in is not the unrelentingly cold world we know. The only time I was truly engaged by EDtv was near its completion, when Ed realizes the horror of being stripped of privacy. This diminishes, of course, into an expected happy ending in which everybody departs the set smiling, but most cinema-goers will leave only with a taste of what could have been. EDtv is largely an ineffective film that is hard to hate but easy to forget. Not bad for entertainment value, but don't look for anything profound in Howard's light-hearted direction.
Read more of my reviews at In Film Australia http://infilmau.iah.net
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


What is IoT ?
Architecture study
Methodologies study
Conclusion
"A network of physical objects that contain embedded technology to communicate and sense or interact with their internal states or the external environment"
Gartner
"It’s about connecting devices over the internet, letting them talk to us, applications, and each other"
The guardian
"The IoT is connecting many devices over the Internet in order to gather , process and make decisions using its data"
Our definition
Applications
Management system
Gateway + Network
Sensors
Health care
Transportation
Road traffic management
Arduino
Raspberry Pi
Intel Edison
Bluetooth
WiFi
ZigBee
Software development methodology is splitting of software development work into different phases which include several actions intending for better management
Waterfall Model
Prototyping Model
Iterative Model
Spiral Model
RAD Model
RUP Model
SCRUM Model
IoT Model


Ping-Pong
About Ping Pong
Creadted by:
Ohood Obeidat
GitHub Username/URL:
ohood84
Level 2 Test For:
JavaScript .
Let's play!
Rules:
Ping Pong will count up to your number from 1 with the following exceptions:
Numbers divisible by 3 become
"ping"
Numbers divisible by 5 become
"pong"
Numbers divisible by 3 and 5 become
"pingpong"
Enter your number
:
Ping-Pong
previous
next


An AM send queue that provides a Service Instance pattern for formatted packets and calls an underlying AMSend in a round-robin fashion. Used to share L2 bandwidth between different communication clients.


Got some machines you need to get rid of? Please take this opportunity to list the machines that you would be willing to sell to Midwest Copier. That’s right, we need to buy machines to sell machines. Whether you have one machine or a thousand machines, don't hesitate to let us know.
If you have a rather large list and would rather not input it all into this form, please don't hesitate to contact us directly!
Location of Machines
Where are the machines located?
Note: An Asterisk (*) denotes a required field.
Note: You can add machines from multiple locations, if needed. Simply fill out the address form below, and for each machine you add, the address for the given machine will be taken from these fields. You can change the address at any time while adding machines.


Carbon budgets in this document are based primarily on the budgets laid out in Table 2.2 of the AR5 Synthesis report. This table was used to create the divisions in the emissions pathway chart as well as the temperature corresponding to a 33% chance of exceedance. This was done by noting that the carbon budgets for various temperatures are linear (given the same probability of exceeding said temperature). This relationship may not hold at temperatures too far beyond 3 degrees Celsius.
However, it was not possible to use this table to inform probabilities of exceedance for intermediate levels of emissions. To create this, the AR5 scenario database was used. In the database, probabilities of exceedance are given for 1.5, 2, 3, and 4 degrees Celsius across model runs, using the reduced complexity MAGICC6 model. It is possible to compare cumulative carbon dioxide emissions to 2100 and probabilities of exceedance at the same date.
Initially, a smooth fit across the four temperatures was attempted, using LOWESS regression. However, because the data includes a wide range of scenarios at certain stabilization levels, the smoothing behavior tended to be strange (and arguably unphysical) in these areas. To rectify this, an assumption was made that the probability distribution function for the amount of cumulative emissions necessary to exceed a certain temperature threshold was normally distributed, so the data as given could be fit to the integral of a Gaussian. The two fits generally agree with one another pretty well. Perhaps the most important discrepancy is at low levels of exceedance of 4 degrees Celsius, where the Gaussian fit is somewhat larger than the LOWESS fit.
The stats for impact by degree were pulled from a report based on another report from the National Research Council. Beyond the regimes specified in the report, the stat is specified as unknown. The temperature used to generate the stats is the temperature corresponding to a 33% probability of exceedance. An alternative might have been to pick the median temperature, but the 33% temperature was chosen to be more representative of the damages one might expect, given that the second derivative of most damage functions with respect to temperature is positive for temperatures less than six degrees, and for some damage functions (DICE), every temperature. For instance, if one considers a damage function of the form given by Weitzman (2010) and a normally distributed PDF, the temperature representing the average damage is actually a little over a tenth of a degree higher than the mean temperature increase.More qualitatively, it's arguably more important to communicate tail risks than the median case. The 33% chance of exceedance was chosen specifically because it is listed concretely in the AR5 Synthesis Report, and thus comes from a more authoritative source than the analysis done specifically for this interactive.
The 2030 scenarios come from Climate Action Tracker, corrected to go from CO2e to CO2, and Fawcett et. al. The general scenarios correspond to the RCP scenarios (2.6, 4.5, and 6 for low, medium, and high respectively), but only insofar as the budget to 2100 for carbon dioxide is the same; the emission pathway shown differs from the RCP scenarios, particularly for the medium and high scenarios. The committed general scenario corresponds to existing carbon dioxide commitments from existing infrastructure, as documented by Peter Erickson and Kevin Tempest. Historical data comes from the Global Carbon Project.


This guide is designed in order to give you the necessary information in order to start using SMSLib. Various support questions made it obvious that the documentation is lucking in some areas. Hopefully, this document will show you how to setup and use SMSLib in your applications.
Beyond this documentation, remember that there are two sample mini apps available, the SendMessage
which simply sends a message and the ReadMessages
which reads messages from your phone. Be sure to have a look as, sometimes, code speaks better than the documentation.  I prefer to express myself by coding.
By the way, do not forget to view on javadoc documentation pages. My goal and pattern is to document all public methods that are of use to you. So if you find a method that is not documented in javadocs, chances are that you should not use it directly.
You primary interface with SMSLib is the Service
class. The entire (almost) functionality of SMSLib is exposed via the Service
's methods. You should create only one instance of the Service
class in your application!
The Service
itself is useless. Before starting the Service
, you should define your gateways. Gateways refer to the physical devices or services which can send and/or receive messages. SMSLib has the following predefined gateways:
org.smslib.modem.SerialModemGateway
): For modems which are connected via serial ports (or emulated serial ports, like USB etc).org.smslib.modem.IPModemGateway
): for modems which are connected via IP interfacesorg.smslib.http.ClickatellHTTPGateway
): for sending messages via Clickatell.org.smslib.http.BulkSmsHTTPGateway
): for sending messages via BulkSMS.org.smslib.smpp.jsmpp.JSMPPGateway
): for sending / receiving messages via the SMPP protocol.Each gateway has different initialization parameters (look at the javadocs for this). The point is to create one or more gateways (according to your setup) and add them to the Service
using a syntax like this:
(Service).addGateway(new SerialModemGateway("modem.com1", "COM1", 57600, "Nokia", "6310i"));
Add as many gateways you wish. Once added, SMSLib will manage and use all of them via the rest of the Service
class methods
Once you add all the necessary gateways, start the Service
by issuing:
(Service).startService()
There are four (4) types of inbound messages that can be handled with SMSLib. These are:
InboundMessage
).InboundBinaryMessage
).InboundEncryptedMessage
, see Encryption).StatusReportMessage
, see Delivery reports).There are two ways to read messages:
readMessages()
method of the Service
class. SMSLib will iterate all defined gateways, collect messages from all and return them in a collection. Note that you will be blocked until SMSLib returns. There are many forms of the readMessages()
call - choose the one according to your needs.Service
. SMSLib will then automatically call you upon message reception. For more information, see Callback methods.There are again, four (4) types of messages that SMSLib can send. These are:
OutboundMessage
).OutboundBinaryMessage
).OutboundEncryptedMessage
, see Encryption).OutboundWapSIMessage
).Each message class has several properties, so check the javadoc for these. You can set the type, the encoding, the delivery report etc.
There are two ways to send a message:
sendMessage()
method of the Service
class. Note that you will be blocked until SMSLib returns. Upon return, you can examine the message object to see the fate of your message.queueMessage()
family of methods to queue your message. Queueing a message returns control immediately to you. SMSLib keeps this message in internal queues and send its in the background. To learn about the fate of your message, you should setup a IOutboundMessageNotification
callback method (see Callback methods).The asynchronous sending is covered in a bit more detail in the following section.
To send a message asynchronously, you use the queueMessage
family of methods. To send a message at a specific time, you use the queueMessageAt
family of methods.
Starting from v3.5, SMSLib queues can be set to operate in a non-persistent or persistent way. In the first case, queued messages are lost between SMSLib invocations (i.e. between Service
restarts). This is the default operating mode, compatible with previous versions. If you set SMSLib to work in persistent mode, queued and scheduled messages are retained between SMSLib invocations.
The persistency mode is selected by setting the QUEUE_DIRECTORY
parameter. Look at Configurations Parameters for more information on the use of the QUEUE_DIRECTORY
parameter.
This section briefly describes the Routing and Balancing concepts. If you use only one Gateway, you may safely skip this section.
When you have more than one Gateways define, you may wonder from which Gateway an outbound message will be sent. SMSLib provides for both an automatic way and a manual way to define your routing.
In order to define the preferred Gateway according to your wishes, you can:
sendMessage()
or queueMessage()
methods which take the Gateway ID as a parameter. With these methods you can define which is the Gateway that you wish your message to be sent through.OutboundMessage
object's setGatewayId()
method in order to define your preferred Gateway at a message level.There is also the concept of the wildcard gateway. This is noted by the star character. By setting the gateway to the star character, you actually state that you are letting SMSLib decide which gateway to use.
SMSLib uses a Routing and a Balancing logic in order to automatically determine the gateway to use. Routing logic takes into consideration possible routing rules in order to decide which (of all gateways) are the candidate gateways. Then the Balancing logic gets applied in order to select the preferred of the candidate gateways. This specific gateway is then used for the dispatch of the specific message.
If this all sound complicated, don't be alarmed. SMSLib uses a default set of Routing/Balancing rules which works well all the time. The default Routing rule is actually a "use-any-of" rule, where all gateways are treated as candidates. The default Balancing rule is an implementation of the Round-Robin logic. So, when you have more than one Gateways, all of them are used in turns.


The messageformat parser has encountered an internal error. Please file a github issue against the AngularJS project and provide this message text that triggers the bug. Text: “{0}”
You've just hit a bug in the ngMessageFormat module provided by angular-message-format.min.js. Please file a github issue for this and provide the interpolation text that caused you to hit this bug mentioning the exact version of AngularJS used and we will fix it!
For more information about the MessageFormat syntax in interpolation expressions, please refer to MessageFormat extensions section at AngularJS i18n MessageFormat


Located
Walmart Supercenter
1606 S 72nd St, Omaha, USA
Villa Market
,J Avenue, Thonglor Soi 15, Bangkok,Thailand
Asia Global Market
,2400 Yeager Rd, West Lafayette, IN 47906, United States


this automate the work flow of apply for leave and their approvals
Only e-leave help you to get rid off manual leave system
Features are apply for leave, leave status, leave approved and download history report of leaves taken.
24*7 online availability.
User friendly and require least effort to operate.
Secure access of confidential data.
Fast accessing system.
Generate of required reports.


Gum
by chloe vincent
There are a lot of different flavors of gum!
I like mint gum the best!
Whats your favorite flavor?
Do you like bubblegum?


Building a Docker Image¶
This document explains how to quickly prepare a Docker image from the biocontainers base image. The biocontainers base image have a pre-installed conda environments. The container is recommended to contain only the dependent libraries for your codes to run, so that you do not have to build a new image every time you modify your scripts. The BDP mounts your codes to the container on the fly.
Let’s start by explaining how the Dockerfile
is written.
1. Base Image¶
We use the biocontainers as our base image. You may see their Dockerfile.
The following instruction shows that our package image is based on the bioconductors/bioconductions
image.
FROM biocontainers/biocontainers:latest
The biocontainers/biocontainers
image is built on top of ubuntu:16.04
. All modifications can be traced back based on its dockerfile.
It creates an account biodocker and installed several useful tools such as conda for developers to install their dependent packages/libraries.
2. Install required packages/libraries that needs root privileges¶
The following snippet in the dockerfile file instructs the Docker to build your image with the instructions. The RUN instruction shows you to install the latest NodeJS version for demonstration.
You may also install other packages/scripts that require root privileges.
See also
The dockerfile reference: https://docs.docker.com/engine/reference/builder/
USER root
ENV NODE_PATH=/usr/lib/node_modules
RUN apt-get update && \
curl -sL https://deb.nodesource.com/setup_10.x | bash - && \
apt-get install -y nodejs && \
apt-get clean && \
apt-get purge --auto-remove -y curl
Tip
You can see the commands after the RUN instructions as if you are using the Linux operation system.
Instead of using multiple RUN instructions, using && \
to organize multiple commands into multi-lines. This will reduce the layers of cached images.
use '\'
to split one long line of a command into multi-lines.
3. Install required packages/libraries that need NOT root privileges¶
Because the biocontainers image has created a user account biodocker
, you can switch to the biodocker account to install the required packages or libraries.
The following snippet showing how to switch the user to biodocker (from root).
USER biodocker
The followings show four ways to install the required packages:
When the tools are already in conda¶
Default conda channels r
and bioconda
were already set in the biocontainers base image, you can directly use these channels.
Or you may need the -c
argument to specify the channel. (e.g. conda install -c conda-forge nodejs
)
You may also find other conda packages from the anaconda website.
RUN conda install r-base=3.3.2 \
samtools=1.4 \
bwa=0.7.15 \
fastqc=0.11.5 \
cutadapt
Although it is optional, but is best to specify the version of these required packages/libraries.
When tools need to be downloaded from internet¶
You can use curl
or wget
to retrieve contents from internet.
Remember to clean unneeded files that do not need after installation.
RUN mkdir -p /tmp/trim_galore && \
cd /tmp/trim_galore && \
wget http://www.bioinformatics.babraham.ac.uk/projects/trim_galore/trim_galore_v0.4.3.zip && \
unzip /tmp/trim_galore/trim_galore_v0.4.3.zip && \
mv trim_galore /home/biodocker/bin/trim_galore && \
rm -rf /tmp/trim_galore
Build from the source code¶
The following code block demonstrates an example to download the compressed source code. You need to decompress the file(s) and then build the executable from source codes. You may keep only the binary executable files. You might want to set the file to be executable by setting chmod 755 your-executable-file
. Remember to remove all unneeded files.
RUN cd /tmp && wget http://search.cpan.org/CPAN/authors/id/T/TI/TIMB/DBI-1.636.tar.gz && \
tar zxvf DBI-1.636.tar.gz && \
cd /tmp/DBI-1.636 && \
perl Makefile.PL && \
make && \
make test && \
make install && \
rm -rf /tmp/DBI-1.636 && \
rm -f /tmp/DBI-1.636.tar.gz
Copy the scripts or files from other third party library¶
You may arrange files into your own file structures. It is possible to directly copy files to the package container. Putting your files inside /home/biodocker/ directory since this is the default home directory.
COPY --chown=biodocker:biodocker ["./scripts-inside-container", "/home/biodocker/scripts/"]
More information about the COPY instructions can be found.
Install packages via package manager(s) that do not need the root privilege¶
If your scripts require npm
packages and a package.json is provided in the folder /home/biodocker/scripts, you may need to run npm install
inside that folder.
In fact, you may install required packages/libraries as if you are using a linux system. For example, use pip install
to install python packages may be suitable for your needs.
RUN cd /home/biodocker/scripts && \
npm install && \
npm cache clean
4. Building from Dockerfile¶
In the directory where the Dockerfile
locates, call docker build -t 'your-docker-image-name' .
.
(Don’t forget the last .
in your command, that specifies the Dockerfile locations, which is the current working directory .
.
Tip
It is recommended to use gitlab to save your scripts as well as the container registry. It is free for personal use.
Volume mapping from host folders to docker containers¶
The VOLUME mapping is automatically processed in Big Data Processor.
The Docker Adapter mounts the project folder and the package folder to the container with the paths /project
and /pacakge
folders, respectively.
This means your scripts that editing on the BDP can be accessed in the /package/scripts
.
See also
To fully understand how to map volumes between hosts and containers, please read the docker run reference (the -v
part).


{% extends "./layout.html" %} {% block content %} {% with messages = get_flashed_messages() %} {% if messages %}
| # | Date | User | Tweet Content | Language | Place | Retweets | Likes |
|---|
Please specify the Regular Expression below if you want to parse the results.
As an initial step, you have to apply some sort of filtering to eliminate excessive Twitter API usage.
This can be done by searching for specific tweets by using "Generic Search" option, searching tweets that are sent by a specific user or you can look for Trend Tweets.
Note that the Generic Search option is available only to signed users.
While Tweet retrieval limit is 150 for registered users, it is 25 for unregistered users.
After you determine the filtering options, Agile TweetViz will retrieve the tweets that you are looking for for you and will give you an ability to parse them.
You will be able to parse through Regular Expressions. Which we think is a convenient option to select tweets precisely according to the user's needs.
Agile TweetViz will apply the regular expression, if specified at all and then will allow you to sort them by replies, likes and retweets etc.
Agile TweetViz will sort the tweets according to your needs and then show them to you. It will also do some analysis and show the results to you.


@(assetsHost: String, assetsBuild: String, studyName: String, studySupportEmail: String, passwordDescription: String, studyId: String)


In 1907 the invention of Bakelite began a revolution in materials by introducing truly synthetic plastic resins into world commerce. Plastic is a polymeric material—that is, a material whose molecules are very large, often resembling long chains made up of a seemingly endless series of interconnected links. It is versatile, lightweight, flexible, moisture resistant, strong, and relatively inexpensive.
Plastics don't react with materials like alcohol, gasoline, or acid, and thus they are great for storing stuff. Unfortunately, that also means that they don't decay. It instead divides down into smaller and smaller pieces called micro-plastics. These pieces can stick around for centuries and can be deadly to sea life if ingested. Everything from seabirds to whales can be seriously hurt or killed by this plastic. Millions of animals have already been killed as a result of plastic pollution, including many endangered species.
By the end of the 20th century, however, plastics were found to be persistent polluters of several environmental niches, from Mount Everest to the bottom of the sea.


Machine coded genetic algorithm (MCGA) is a fast tool for real-valued optimization problems. It uses the byte representation of variables rather than real-values. It performs the classical crossover operations (uniform) on these byte representations. Mutation operator is also similar to classical mutation operator, which is to say, it changes a randomly selected byte value of a chromosome by +1 or -1 with probability 1/2. In MCGAs there is no need for encoding-decoding process and the classical operators are directly applicable on real-values. It is fast and can handle a wide range of a search space with high precision. Using a 256-unary alphabet is the main disadvantage of this algorithm but a moderate size population is convenient for many problems. Package also includes multi_mcga function for multi objective optimization problems. This function sorts the chromosomes using their ranks calculated from the non-dominated sorting algorithm.
| Version: | 3.0.3 |
| Depends: | GA |
| Imports: | Rcpp (≥ 0.11.4) |
| LinkingTo: | Rcpp |
| Published: | 2018-05-13 |
| Author: | Mehmet Hakan Satman |
| Maintainer: | Mehmet Hakan Satman <mhsatman at istanbul.edu.tr> |
| License: | GPL-2 | GPL-3 [expanded from: GPL (≥ 2)] |
| NeedsCompilation: | yes |
| Citation: | mcga citation info |
| In views: | Optimization |
| CRAN checks: | mcga results |
| Reference manual: | mcga.pdf |
| Package source: | mcga_3.0.3.tar.gz |
| Windows binaries: | r-devel: mcga_3.0.3.zip, r-release: mcga_3.0.3.zip, r-oldrel: mcga_3.0.3.zip |
| OS X binaries: | r-release: mcga_3.0.3.tgz, r-oldrel: mcga_3.0.3.tgz |
| Old sources: | mcga archive |
Please use the canonical form https://CRAN.R-project.org/package=mcga to link to this page.


--- layout: post title: Use 'local' when running on the Workgroup Server categories: [] tags: dotnet-sdk serviceapi webdrawer web-client status: publish type: post published: true meta: {} ---
One tip that is not as well known as it should be is the use of the reserved work 'local' to refer to a local Workgroup Server. Any time a .Net SDK application is talking to a Workgroup Server on the same machine you should use 'local' for the Workgroup Server name, rather than using the machine name. Why is this? Because 'local' will use a named pipes connection to the Workgroup Server rather than going through the network layer.
Of course the Web Client, ServiceaAPI and WebDrawer are .Net SDK applications, so of you are connecting to a WorkGroup Server on the same machine use 'local'. Note the MSI will probably not suggest this to you so you wil have to edit the hptrim.config manually. Make it look something like this:
<workgroupServer port="1137" workPath="C:\HP Records Manager\ServiceAPIWorkpath" name="local" alternateName="MY_ALTERNATE_WGS" alternatePort="1137" />
Hello, World!
If you are writing an application to run on the same machine as the Workgroup Server do something like this:
using (Database database = new Database()) { database.Id = "J1"; database.WorkgroupServerName = "local"; database.Connect(); //do something }


|
|
IFXList< T > Class Template ReferenceFully bidirectional Doubly-Linked List.
More...
Inheritance diagram for IFXList< T >:
List of all members.
Detailed Description
template<class T>
class IFXList< T >
Fully bidirectional Doubly-Linked List.
There is a NULL entry at both beginning and end.
If SearchForContent() is to be used and type T has any data complexity the operator == should probably be defined for type T. Some examples of data complexity are:
- pointer data members
- data members that do not contribute to 'equality'
- data whose equality can be more efficiently determined with something other than the default bytewise comparison.
- fuzzy or soft data
The element stored is actually T*, not T, so templating on a pointer is probably adding an additional level of indirection. Note that this is not true for IFXArray which stores actual instances.
Example:
No contexts will lose place if elements are removed, even if removed with a different context. Reference counting prevents invalid pointers and heirs follow destruction chains to find valid nodes.
By using SetAutoDestruct(true), all elements remaining on the list are deleted when the list is destructed.
WARNING: if these elements were not created with new(), failures will probably occur. Note that SetAutoDestruct(true) is precarious when elements may be in more than one list. In that case, it is very possible for elements to be deleted that are still members of other lists.
Note that much of the functionality of this class is implemented in the non-template class IFXCoreList. This is done to reduce template instantiation size.
Uses IFXUnitAllocator to manage IFXListNode's.
Constructor & Destructor Documentation
|
template<class T> |
| virtual IFXList< T >::~IFXList |
( |
void |
|
) |
[inline, virtual] |
|
Member Function Documentation
|
template<class T> |
| void IFXList< T >::Append |
( |
IFXList< T > & |
list |
) |
[inline] |
|
|
|
Appends a copy of an entire list of identical type.
|
|
template<class T> |
| T** IFXList< T >::Append |
( |
T * |
entry |
) |
[inline] |
|
|
|
Inserts a new element at the end of the list.
Returns the location of the new element or NULL if the internal node allocation failed. |
|
template<class T> |
| BOOL IFXList< T >::Delete |
( |
T * |
entry |
) |
[inline] |
|
|
|
Removes and deletes the first entry that match the argument.
|
|
template<class T> |
| void IFXList< T >::DeleteAll |
( |
void |
|
) |
[inline] |
|
|
|
Removes all nodes and delete their pointers.
|
|
|
Removes and deletes the current node on the given context.
|
|
|
Returns the current element of the context.
|
|
template<class T> |
| T* IFXList< T >::GetElement |
( |
long |
index |
) |
const [inline] |
|
|
|
Gets an element by index.
This is not an array, so this can be an inefficient operation. |
|
template<class T> |
| T* IFXList< T >::GetHead |
( |
void |
|
) |
const [inline] |
|
|
|
Returns the first element of the list (no context required).
|
|
template<class T> |
| T* IFXList< T >::GetTail |
( |
void |
|
) |
const [inline] |
|
|
|
Returns the last element of the list (no context required).
|
|
|
Inserts a new element after the context.
Returns the location of the new element or NULL if the internal node allocation failed. |
|
|
Inserts a new element before the context.
Returns the location of the new element or NULL if the internal node allocation failed. |
|
|
Returns the current element before decrementing to the previous.
|
|
|
Returns the current element before incrementing to the next.
|
|
|
Decrements the context and return the previous element.
|
|
|
Increments the context and return the next element.
|
|
template<class T> |
| T** IFXList< T >::Prepend |
( |
T * |
entry |
) |
[inline] |
|
|
|
Inserts a new element at the beginning of the list.
Returns the location of the new element or NULL if the internal node allocation failed. |
|
|
This Remove() method uses a hint IFXListContext to check current, next, and prev nodes (in that order) before reverting to a head to tail scan.
In most cases this significantly reduces execution time. |
|
template<class T> |
| BOOL IFXList< T >::Remove |
( |
T * |
entry |
) |
[inline] |
|
|
|
This is the straightforward remove implementation.
This Remove() will look for the entry in the list by simply checking each member entry in order. This can be very slow in some cases. See Remove(T *entry, IFXListContext &hint) for a faster alternative. |
|
template<class T> |
| T* IFXList< T >::SearchForContent |
( |
IFXListContext & |
context, |
|
|
const T * |
value |
|
) |
const [inline] |
|
|
|
Returns the first element with the same value.
The elements are compared to the contents of *value. The value argument is passed by pointer instead of value or reference to prevent a requirement that the T be an instantiable class. |
|
template<class T> |
| T* IFXList< T >::SearchForElement |
( |
IFXListContext & |
context, |
|
|
T * |
entry |
|
) |
const [inline] |
|
|
|
Returns the first element matching the given pointer.
Only pointer are compared. Matches are not made by content they point to. |
|
|
Rewinds the context to the beginning of the list and return the first element.
|
|
|
Moves the context to the end of the list and return the last element.
|
© 1999 - 2006 Intel Corporation. All rights reserved.
* Other names and brands may be claimed as the property of others.
|
|