T
- public abstract class Metric<T extends InformationLoss<?>>
extends java.lang.Object
implements java.io.Serializable
| Modifier and Type | Class and Description |
|---|---|
static class |
Metric.AggregateFunction
Pluggable aggregate functions.
|
| Modifier and Type | Method and Description |
|---|---|
static Metric<ILSingleDimensional> |
createAECSMetric()
Creates a new instance of the AECS metric.
|
static Metric<ILSingleDimensional> |
createAECSMetric(double gsFactor)
Creates a new instance of the AECS metric.
|
static Metric<ILSingleDimensional> |
createAmbiguityMetric()
Creates an instance of the ambiguity metric.
|
static Metric<ILSingleDimensional> |
createDiscernabilityMetric()
Creates an instance of the discernability metric.
|
static Metric<ILSingleDimensional> |
createDiscernabilityMetric(boolean monotonic)
Creates an instance of the discernability metric.
|
static Metric<AbstractILMultiDimensional> |
createEntropyMetric()
Creates an instance of the non-monotonic non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createEntropyMetric(boolean monotonic)
Creates an instance of the non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createEntropyMetric(boolean monotonic,
double gsFactor)
Creates an instance of the non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createEntropyMetric(boolean monotonic,
double gsFactor,
Metric.AggregateFunction function)
Creates an instance of the non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createEntropyMetric(boolean monotonic,
Metric.AggregateFunction function)
Creates an instance of the non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createEntropyMetric(double gsFactor)
Creates an instance of the non-monotonic non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createHeightMetric()
Creates an instance of the height metric.
|
static Metric<AbstractILMultiDimensional> |
createHeightMetric(Metric.AggregateFunction function)
Creates an instance of the height metric.
|
static Metric<ILSingleDimensional> |
createKLDivergenceMetric()
Creates an instance of the KL Divergence metric.
|
static Metric<AbstractILMultiDimensional> |
createLossMetric()
Creates an instance of the loss metric which treats generalization and suppression equally.
|
static Metric<AbstractILMultiDimensional> |
createLossMetric(double gsFactor)
Creates an instance of the loss metric with factors for weighting generalization and suppression.
|
static Metric<AbstractILMultiDimensional> |
createLossMetric(double gsFactor,
Metric.AggregateFunction function)
Creates an instance of the loss metric with factors for weighting generalization and suppression.
|
static Metric<AbstractILMultiDimensional> |
createLossMetric(Metric.AggregateFunction function)
Creates an instance of the loss metric which treats generalization and suppression equally.
|
abstract InformationLoss<?> |
createMaxInformationLoss()
Returns an instance of the maximal value.
|
static Metric<?> |
createMetric(Metric<?> metric,
int minLevel,
int maxLevel)
This method supports backwards compatibility.
|
abstract InformationLoss<?> |
createMinInformationLoss()
Returns an instance of the minimal value.
|
static Metric<AbstractILMultiDimensional> |
createNormalizedEntropyMetric()
Creates an instance of the normalized entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createNormalizedEntropyMetric(Metric.AggregateFunction function)
Creates an instance of the normalized entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecisionMetric()
Creates an instance of the non-monotonic precision metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecisionMetric(boolean monotonic)
Creates an instance of the precision metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecisionMetric(boolean monotonic,
double gsFactor)
Creates an instance of the precision metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecisionMetric(boolean monotonic,
double gsFactor,
Metric.AggregateFunction function)
Creates an instance of the precision metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecisionMetric(boolean monotonic,
Metric.AggregateFunction function)
Creates an instance of the precision metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecisionMetric(double gsFactor)
Creates an instance of the non-monotonic precision metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecisionMetric(double gsFactor,
Metric.AggregateFunction function)
Creates an instance of the non-monotonic precision metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecisionMetric(Metric.AggregateFunction function)
Creates an instance of the non-monotonic precision metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedEntropyMetric(double threshold)
Creates a potentially precomputed instance of the non-monotonic non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedEntropyMetric(double threshold,
boolean monotonic)
Creates a potentially precomputed instance of the non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedEntropyMetric(double threshold,
boolean monotonic,
double gsFactor)
Creates a potentially precomputed instance of the non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedEntropyMetric(double threshold,
boolean monotonic,
double gsFactor,
Metric.AggregateFunction function)
Creates a potentially precomputed instance of the non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedEntropyMetric(double threshold,
boolean monotonic,
Metric.AggregateFunction function)
Creates a potentially precomputed instance of the non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedEntropyMetric(double threshold,
double gsFactor)
Creates a potentially precomputed instance of the non-monotonic non-uniform entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedLossMetric(double threshold)
Creates a potentially precomputed instance of the loss metric which treats generalization
and suppression equally.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedLossMetric(double threshold,
double gsFactor)
Creates a potentially precomputed instance of the loss metric with factors for weighting generalization and suppression.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedLossMetric(double threshold,
double gsFactor,
Metric.AggregateFunction function)
Creates a potentially precomputed instance of the loss metric with factors for weighting generalization and suppression.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedLossMetric(double threshold,
Metric.AggregateFunction function)
Creates a potentially precomputed instance of the loss metric which treats generalization and suppression equally.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedNormalizedEntropyMetric(double threshold)
Creates a potentially precomputed instance of the normalized entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createPrecomputedNormalizedEntropyMetric(double threshold,
Metric.AggregateFunction function)
Creates a potentially precomputed instance of the normalized entropy metric.
|
static Metric<AbstractILMultiDimensional> |
createStaticMetric(java.util.Map<java.lang.String,java.util.List<java.lang.Double>> loss)
Creates an instance of a metric with statically defined information loss.
|
static Metric<AbstractILMultiDimensional> |
createStaticMetric(java.util.Map<java.lang.String,java.util.List<java.lang.Double>> loss,
Metric.AggregateFunction function)
Creates an instance of a metric with statically defined information loss.
|
Metric.AggregateFunction |
getAggregateFunction()
Returns the aggregate function of a multi-dimensional metric, null otherwise.
|
MetricConfiguration |
getConfiguration()
Returns the configuration of this metric.
|
MetricDescription |
getDescription()
Returns a description of this metric.
|
double |
getGeneralizationFactor()
Returns the factor used weight generalized values.
|
double |
getGeneralizationSuppressionFactor()
Returns the factor weighting generalization and suppression.
|
InformationLossWithBound<T> |
getInformationLoss(org.deidentifier.arx.framework.lattice.Transformation node,
org.deidentifier.arx.framework.check.groupify.HashGroupify groupify)
Evaluates the metric for the given node.
|
InformationLossWithBound<T> |
getInformationLoss(org.deidentifier.arx.framework.lattice.Transformation node,
org.deidentifier.arx.framework.check.groupify.HashGroupifyEntry entry)
Returns the information loss that would be induced by suppressing the given entry.
|
T |
getLowerBound(org.deidentifier.arx.framework.lattice.Transformation node)
Returns a lower bound for the information loss for the given node.
|
T |
getLowerBound(org.deidentifier.arx.framework.lattice.Transformation node,
org.deidentifier.arx.framework.check.groupify.HashGroupify groupify)
Returns a lower bound for the information loss for the given node.
|
java.lang.String |
getName()
Returns the name of metric.
|
double |
getSuppressionFactor()
Returns the factor used to weight suppressed values.
|
void |
initialize(org.deidentifier.arx.framework.data.DataManager manager,
DataDefinition definition,
org.deidentifier.arx.framework.data.Data input,
org.deidentifier.arx.framework.data.GeneralizationHierarchy[] hierarchies,
ARXConfiguration config)
Initializes the metric.
|
boolean |
isIndependent()
Returns whether this metric requires the transformed data or groups to
determine information loss.
|
boolean |
isMonotonic()
Returns false if the metric is non-monotone using suppression.
|
boolean |
isMultiDimensional()
Returns true if the metric is multi-dimensional.
|
boolean |
isWeighted()
Returns true if the metric is weighted.
|
static java.util.List<MetricDescription> |
list()
Returns a list of all available metrics for information loss.
|
java.lang.String |
toString()
Returns the name of metric.
|
public static Metric<ILSingleDimensional> createAECSMetric()
public static Metric<ILSingleDimensional> createAECSMetric(double gsFactor)
gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.public static Metric<ILSingleDimensional> createAmbiguityMetric()
public static Metric<ILSingleDimensional> createDiscernabilityMetric()
public static Metric<ILSingleDimensional> createDiscernabilityMetric(boolean monotonic)
monotonic
- If set to true, the monotonic variant (DM*) will be createdpublic static Metric<AbstractILMultiDimensional> createEntropyMetric()
public static Metric<AbstractILMultiDimensional> createEntropyMetric(boolean monotonic)
monotonic
- If set to true, the monotonic variant of the metric will be createdpublic static Metric<AbstractILMultiDimensional> createEntropyMetric(boolean monotonic, Metric.AggregateFunction function)
monotonic
- If set to true, the monotonic variant of the metric will be createdfunction
- The aggregate function to be used for comparing resultspublic static Metric<AbstractILMultiDimensional> createEntropyMetric(double gsFactor)
gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.public static Metric<AbstractILMultiDimensional> createEntropyMetric(boolean monotonic, double gsFactor)
gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.monotonic
- If set to true, the monotonic variant of the metric will be createdpublic static Metric<AbstractILMultiDimensional> createEntropyMetric(boolean monotonic, double gsFactor, Metric.AggregateFunction function)
monotonic
- If set to true, the monotonic variant of the metric will be createdgsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.function
- The aggregate function to be used for comparing resultspublic static Metric<AbstractILMultiDimensional> createHeightMetric()
public static Metric<AbstractILMultiDimensional> createHeightMetric(Metric.AggregateFunction function)
function
- The aggregate function to use for comparing resultspublic static Metric<ILSingleDimensional> createKLDivergenceMetric()
public static Metric<AbstractILMultiDimensional> createLossMetric()
public static Metric<AbstractILMultiDimensional> createLossMetric(Metric.AggregateFunction function)
function
- The aggregate function to use for comparing resultspublic static Metric<AbstractILMultiDimensional> createLossMetric(double gsFactor)
gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.public static Metric<AbstractILMultiDimensional> createLossMetric(double gsFactor, Metric.AggregateFunction function)
gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.function
- The aggregate function to use for comparing resultspublic static Metric<?> createMetric(Metric<?> metric, int minLevel, int maxLevel)
metric
- minLevel
- maxLevel
- public static Metric<AbstractILMultiDimensional> createNormalizedEntropyMetric()
public static Metric<AbstractILMultiDimensional> createNormalizedEntropyMetric(Metric.AggregateFunction function)
function
- The aggregate function to use for comparing resultspublic static Metric<AbstractILMultiDimensional> createPrecisionMetric()
public static Metric<AbstractILMultiDimensional> createPrecisionMetric(Metric.AggregateFunction function)
function
- The aggregate function to use for comparing resultspublic static Metric<AbstractILMultiDimensional> createPrecisionMetric(boolean monotonic)
monotonic
- If set to true, the monotonic variant of the metric will be createdpublic static Metric<AbstractILMultiDimensional> createPrecisionMetric(boolean monotonic, Metric.AggregateFunction function)
monotonic
- If set to true, the monotonic variant of the metric will be createdfunction
- public static Metric<AbstractILMultiDimensional> createPrecisionMetric(double gsFactor)
gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.public static Metric<AbstractILMultiDimensional> createPrecisionMetric(double gsFactor, Metric.AggregateFunction function)
function
- The aggregate function to use for comparing resultsgsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.public static Metric<AbstractILMultiDimensional> createPrecisionMetric(boolean monotonic, double gsFactor)
monotonic
- If set to true, the monotonic variant of the metric will be createdgsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.public static Metric<AbstractILMultiDimensional> createPrecisionMetric(boolean monotonic, double gsFactor, Metric.AggregateFunction function)
monotonic
- If set to true, the monotonic variant of the metric will be createdgsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.function
- public static Metric<AbstractILMultiDimensional> createPrecomputedEntropyMetric(double threshold)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.public static Metric<AbstractILMultiDimensional> createPrecomputedEntropyMetric(double threshold, boolean monotonic)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.monotonic
- If set to true, the monotonic variant of the metric will be createdpublic static Metric<AbstractILMultiDimensional> createPrecomputedEntropyMetric(double threshold, boolean monotonic, Metric.AggregateFunction function)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.monotonic
- If set to true, the monotonic variant of the metric will be createdfunction
- The aggregate function to be used for comparing resultspublic static Metric<AbstractILMultiDimensional> createPrecomputedEntropyMetric(double threshold, double gsFactor)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.public static Metric<AbstractILMultiDimensional> createPrecomputedEntropyMetric(double threshold, boolean monotonic, double gsFactor)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.monotonic
- If set to true, the monotonic variant of the metric will be createdpublic static Metric<AbstractILMultiDimensional> createPrecomputedEntropyMetric(double threshold, boolean monotonic, double gsFactor, Metric.AggregateFunction function)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.monotonic
- If set to true, the monotonic variant of the metric will be createdgsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.function
- The aggregate function to be used for comparing resultspublic static Metric<AbstractILMultiDimensional> createPrecomputedLossMetric(double threshold)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.public static Metric<AbstractILMultiDimensional> createPrecomputedLossMetric(double threshold, Metric.AggregateFunction function)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.function
- The aggregate function to use for comparing resultspublic static Metric<AbstractILMultiDimensional> createPrecomputedLossMetric(double threshold, double gsFactor)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.public static Metric<AbstractILMultiDimensional> createPrecomputedLossMetric(double threshold, double gsFactor, Metric.AggregateFunction function)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.gsFactor
- A factor [0,1] weighting generalization and suppression.
The default value is 0.5, which means that generalization
and suppression will be treated equally. A factor of 0
will favor suppression, and a factor of 1 will favor
generalization. The values in between can be used for
balancing both methods.function
- The aggregate function to use for comparing resultspublic static Metric<AbstractILMultiDimensional> createPrecomputedNormalizedEntropyMetric(double threshold)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.public static Metric<AbstractILMultiDimensional> createPrecomputedNormalizedEntropyMetric(double threshold, Metric.AggregateFunction function)
threshold
- The precomputed variant of the metric will be used if
#distinctValues / #rows <= threshold for all quasi-identifiers.function
- The aggregate function to use for comparing resultspublic static Metric<AbstractILMultiDimensional> createStaticMetric(java.util.Map<java.lang.String,java.util.List<java.lang.Double>> loss)
loss
- User defined information loss per attributepublic static Metric<AbstractILMultiDimensional> createStaticMetric(java.util.Map<java.lang.String,java.util.List<java.lang.Double>> loss, Metric.AggregateFunction function)
loss
- User defined information loss per attributefunction
- The aggregate function to use for comparing resultspublic static java.util.List<MetricDescription> list()
public abstract InformationLoss<?> createMaxInformationLoss()
public abstract InformationLoss<?> createMinInformationLoss()
public Metric.AggregateFunction getAggregateFunction()
public MetricConfiguration getConfiguration()
public MetricDescription getDescription()
public double getGeneralizationFactor()
public double getGeneralizationSuppressionFactor()
public final InformationLossWithBound<T> getInformationLoss(org.deidentifier.arx.framework.lattice.Transformation node, org.deidentifier.arx.framework.check.groupify.HashGroupify groupify)
node
- The node for which to compute the information lossgroupify
- The groupify operator of the previous checkpublic final InformationLossWithBound<T> getInformationLoss(org.deidentifier.arx.framework.lattice.Transformation node, org.deidentifier.arx.framework.check.groupify.HashGroupifyEntry entry)
getInformationLoss(node, groupify)
but is guaranteed to be comparable for
different entries from the same groupify operator.entry
- public T getLowerBound(org.deidentifier.arx.framework.lattice.Transformation node)
null
.node
- public T getLowerBound(org.deidentifier.arx.framework.lattice.Transformation node, org.deidentifier.arx.framework.check.groupify.HashGroupify groupify)
null
.node
- groupify
- public java.lang.String getName()
public double getSuppressionFactor()
public final void initialize(org.deidentifier.arx.framework.data.DataManager manager, DataDefinition definition, org.deidentifier.arx.framework.data.Data input, org.deidentifier.arx.framework.data.GeneralizationHierarchy[] hierarchies, ARXConfiguration config)
manager
- definition
- input
- hierarchies
- config
- public boolean isIndependent()
public final boolean isMonotonic()
public final boolean isMultiDimensional()
public final boolean isWeighted()
public java.lang.String toString()
toString
in class java.lang.Object


Number of flood victims in Membakut increases to 90
KOTA KINABALU, Nov 9 — Following intermittent rain over the past two days, the number of flood victims in Membakut sub-district increased to 90 this morning compared to 19 last night.
Sabah Civil Defence Force (APM) in a statement this morning said the 31 families comprising 49 adults, 38 children and three infants were seeking shelter at Dewan Selagon here.
Twelve villages which have been hit by floods in Membakut are Kampung Lumantak, Baitam, Sinoko, Bambangan, Baru, Kayai, Lampuas, Sindok, Siniko, Laut, Jambatan Baru and Kampung Kayai.
“The main road at Jalan Kampung Baru, Bambangan is not passable to vehicles,” the statement added.
Apart from APM, Fire and Rescue Department personnel, the police, Beaufort district office, Health and Social Welfare Offices are also monitoring the flood situation. — Bernama


People are aware of this problem
As an African American, I have a personal interest in seeing more people of color in the tech industry. It is fairly well known that people of color are sparse among developers. I could link some articles drawing attention to this issue, but it seems that has been done before. Similarly, others have tried to diagnose this issue trying to identify its underlying cause and the most efficient solution. I don't have the knowledge or experience to contribute that debate except to reflect on my own experience.
Back in middle school I had strong natural interest and ability in math and science. But, I never had friends or a community of people to explore it with. All the kids my age just wanted to play sports, so I hid my nerdy interest in engineering and tried to fit in. When I was young, there were not any places for young people to develop an interest of this kind, to have fun with it, and to be around peers who enjoyed exploring engineering. There was my first missed opportunity.
Once I reached high school, I started getting a little more serious about my studies and career, which continued into college. At that time, I was chiefly interested in fighting racism and other forms of prejudice, and engaging social justice and law seemed like the obvious place to carry out those aspirations. I sometimes remembered the interest in quantitative sciences from my youth, but I did not see how that work contributed to the social good like law and policy. This was a second missed opportunity.
With that history in mind, I would have needed further encouragement to channel that desire in the right direction. Fortunately, numerous initiatives have begun to better channel those desires in others. The few obstacles that I encountered, a lack of community to enjoy technology with and a failure to see it as a contribution to the public good, have been addressed by various other organizations.
How I can help
I have worked closely with kids for years in neighborhood level community building activities intended to build community and encourage a spirit of service among young people. While I have been in the midst of this career transition, I haven't been able to keep up with those efforts, and I miss it dearly. I have always planned on diving back into it. It is only natural that I weave this interest and experience with my growing career as a developer.
I am especially eager to support projects like Coder Dojo. I don't know a lot about this organization, but it seems to offer a space for young people to nurture their natural interest in science and technology, and to have fun! I would have loved something like that as a kid. Contacting Coder Dojo and other organizations like that is one of my top priorities after I have completed Dev Bootcamp, and to take up similar efforts in high schools and with whatever other young people want to learn along with me.


[35 min] ChIP-seq assays: Probing for epigenetic marks: [PDF]
[20 min] Mapping peaks in ChIP-seq datasets: [PDF]
[15 min] Nextflow: an efficient workflow manager for reproducible research in biological projects: [PDF]
[30 min] R/Bioconductor crash course: comprehensive integration of genomic data: [PDF]
[55min] Processing ChIP-seq datasets: from A to Z with Nextflow workflow manager: [HTML | Rmd (save to open in RStudio) ]
Data for this part of the course was collected from multiple research papers, in which ChIP-seq (or variants of) experiments have been conducted in Yeast, in normal conditions (no specific treatment).
In this practical, we will process multiple datasets from different experiments, using a unified pipeline.
[55min] Motif enrichment analysis with MEME workflow: [HTML | Rmd (save to open in RStudio) ]
Motifs enriched in TF-occupied loci can be identified using motif enrichment tools such as the MEME suite .
Here, we will try to identify putative DNA sequence motifs that may be recognized by each factor studied in this practical.
[55min] Aggregating ChIP-seq results: [HTML | Rmd (save to open in RStudio) ]
Depending on the biological question, ChIP-seq results can be integrated to test hypotheses and uncover biological insights.
Here, we will focus on identifying potential interactions between the different transcription factors analyzed in this course.


{% include 'head.html' %}
{% include 'header.html' %} {% include 'menu.html' %}
Genomic context:
(Drag with mouse to scroll through genomes)
+
-
{% include 'footer.html' %}


Configurations
Update view
Config id
Open
Edit
Delete
Upload configuration
Select a json file (*.json).


Raja mahendra pratap hostel has rooms on twin-sharing / single occupancy (for double bed occupancy) basis with a total capacity of 130 seats for boys and 100 seats for girls. Every room contains adequate furniture exclusively for each student like bed. cupboard,chair and table.
the hostel has a spacious mess having space for boys and girls. nutritious food is served in neat and clean surroundings at subsidized rates. the mess has a saparate section for staff members, guests and residents.
The hostel has common rooms (separately for boys and girls) where residents can read daily newspaper and magazines. Television with cable facilities provided in the common room.
Table tennis, chess and carom board can be played by residents in their space time. however the games room shall remain cosed during exam time.


In this notebook, sections that end with '(IMPLEMENTATION)' in the header indicate that the following blocks of code will require additional functionality which you must provide. Please be sure to read the instructions carefully!
In this notebook, you will build a deep neural network that functions as part of an end-to-end machine translation pipeline. Your completed pipeline will accept English text as input and return the French translation.
We begin by investigating the dataset that will be used to train and evaluate your pipeline. The most common datasets used for machine translation are from WMT. However, that will take a long time to train a neural network on. We'll be using a dataset we created for this project that contains a small vocabulary. You'll be able to train your model in a reasonable time with this dataset.
The data is located in data/small_vocab_en
and data/small_vocab_fr
. The small_vocab_en
file contains English sentences with their French translations in the small_vocab_fr
file. Load the English and French data from these files from running the cell below.
import helper
# Load English data
english_sentences = helper.load_data('data/small_vocab_en')
# Load French data
french_sentences = helper.load_data('data/small_vocab_fr')
print('Dataset Loaded')
Each line in small_vocab_en
contains an English sentence with the respective translation in each line of small_vocab_fr
. View the first two lines from each file.
for sample_i in range(2):
print('small_vocab_en Line {}: {}'.format(sample_i + 1, english_sentences[sample_i]))
print('small_vocab_fr Line {}: {}'.format(sample_i + 1, french_sentences[sample_i]))
From looking at the sentences, you can see they have been preprocessed already. The puncuations have been delimited using spaces. All the text have been converted to lowercase. This should save you some time, but the text requires more preprocessing.
The complexity of the problem is determined by the complexity of the vocabulary. A more complex vocabulary is a more complex problem. Let's look at the complexity of the dataset we'll be working with.
import collections
english_words_counter = collections.Counter([word for sentence in english_sentences for word in sentence.split()])
french_words_counter = collections.Counter([word for sentence in french_sentences for word in sentence.split()])
print('{} English words.'.format(len([word for sentence in english_sentences for word in sentence.split()])))
print('{} unique English words.'.format(len(english_words_counter)))
print('10 Most common words in the English dataset:')
print('"' + '" "'.join(list(zip(*english_words_counter.most_common(10)))[0]) + '"')
print()
print('{} French words.'.format(len([word for sentence in french_sentences for word in sentence.split()])))
print('{} unique French words.'.format(len(french_words_counter)))
print('10 Most common words in the French dataset:')
print('"' + '" "'.join(list(zip(*french_words_counter.most_common(10)))[0]) + '"')
For comparison, Alice's Adventures in Wonderland contains 2,766 unique words of a total of 15,500 words.
For this project, you won't use text data as input to your model. Instead, you'll convert the text into sequences of integers using the following preprocess methods:
Time to start preprocessing the data...
For a neural network to predict on text data, it first has to be turned into data it can understand. Text data like "dog" is a sequence of ASCII character encodings. Since a neural network is a series of multiplication and addition operations, the input data needs to be number(s).
We can turn each character into a number or each word into a number. These are called character and word ids, respectively. Character ids are used for character level models that generate text predictions for each character. A word level model uses word ids that generate text predictions for each word. Word level models tend to learn better, since they are lower in complexity, so we'll use those.
Turn each sentence into a sequence of words ids using Keras's Tokenizer
function. Use this function to tokenize english_sentences
and french_sentences
in the cell below.
Running the cell will run tokenize
on sample data and show output for debugging.
import project_tests as tests
from keras.preprocessing.text import Tokenizer
def tokenize(x):
"""
Tokenize x
:param x: List of sentences/strings to be tokenized
:return: Tuple of (tokenized x data, tokenizer used to tokenize x)
"""
# TODO: Implement -- Done
"""
keras.preprocessing.text.Tokenizer transforms a list of num_samples sequences (lists of scalars)
into a 2D Numpy array of shape (num_samples, num_timesteps).
keras.preprocessing.text.Tokenizer(num_words=None,
filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
lower=True,
split=" ",
char_level=False)
"""
tokenizer = Tokenizer(num_words=355)
#fit_on_texts method vectorizes text, x
tokenizer.fit_on_texts(x)
#return
#1- an array/list of integers for each tokenized sentence in x,
#2- a value-key dictionary for each tokenized sentence in x
return tokenizer.texts_to_sequences(x), tokenizer
tests.test_tokenize(tokenize)
# Tokenize Example output
text_sentences = [
'The quick brown fox jumps over the lazy dog .',
'By Jove , my quick study of lexicography won a prize .',
'This is a short sentence .']
text_tokenized, text_tokenizer = tokenize(text_sentences)
print(text_tokenizer.word_index)
print()
for sample_i, (sent, token_sent) in enumerate(zip(text_sentences, text_tokenized)):
print('Sequence {} in x'.format(sample_i + 1))
print(' Input: {}'.format(sent))
print(' Output: {}'.format(token_sent))
When batching the sequence of word ids together, each sequence needs to be the same length. Since sentences are dynamic in length, we can add padding to the end of the sequences to make them the same length.
Make sure all the English sequences have the same length and all the French sequences have the same length by adding padding to the end of each sequence using Keras's pad_sequences
function.
import numpy as np
from keras.preprocessing.sequence import pad_sequences
def pad(x, length=None):
"""
Pad x
:param x: List of sequences.
:param length: Length to pad the sequence to. If None, use length of longest sequence in x.
:return: Padded numpy array of sequences
"""
# TODO: Implement -- Done
"""
pad_sequences method transforms a list of num_samples sequences (lists of scalars) into a
2D Numpy array of shape (num_samples, num_timesteps).
keras.preprocessing.sequence.pad_sequences(sequences, maxlen=None, dtype='int32',
padding='pre', truncating='pre', value=0.)
truncating = remove values from sequences larger than maxlen either in the
beginning (pre) or in the end (post) of the sequence.
value = value (e.g. 0) to use for padding the sequences to the desired length
"""
return pad_sequences(x, maxlen=length, padding='post', value=0.)
tests.test_pad(pad)
# Pad Tokenized output
test_pad = pad(text_tokenized)
for sample_i, (token_sent, pad_sent) in enumerate(zip(text_tokenized, test_pad)):
print('Sequence {} in x'.format(sample_i + 1))
print(' Input: {}'.format(np.array(token_sent)))
print(' Output: {}'.format(pad_sent))
Your focus for this project is to build neural network architecture, so we won't ask you to create a preprocess pipeline. Instead, we've provided you with the implementation of the preprocess
function.
def preprocess(x, y):
"""
Preprocess x and y
:param x: Feature List of sentences
:param y: Label List of sentences
:return: Tuple of (Preprocessed x, Preprocessed y, x tokenizer, y tokenizer)
"""
preprocess_x, x_tk = tokenize(x)
preprocess_y, y_tk = tokenize(y)
preprocess_x = pad(preprocess_x)
preprocess_y = pad(preprocess_y)
# Keras's sparse_categorical_crossentropy function requires the labels to be in 3 dimensions
preprocess_y = preprocess_y.reshape(*preprocess_y.shape, 1)
return preprocess_x, preprocess_y, x_tk, y_tk
preproc_english_sentences, preproc_french_sentences, english_tokenizer, french_tokenizer =\
preprocess(english_sentences, french_sentences)
print('Data Preprocessed')
In this section, you will experiment with various neural network architectures. You will begin by training four relatively simple architectures.
After experimenting with the four simple architectures, you will construct a deeper architecture that is designed to outperform all four models.
The neural network will be translating the input to words ids, which isn't the final form we want. We want the French translation. The function logits_to_text
will bridge the gab between the logits from the neural network to the French translation. You'll be using this function to better understand the output of the neural network.
def logits_to_text(logits, tokenizer):
"""
Turn logits from a neural network into text using the tokenizer
:param logits: Logits from a neural network
:param tokenizer: Keras Tokenizer fit on the labels
:return: String that represents the text of the logits
"""
index_to_words = {id: word for word, id in tokenizer.word_index.items()}
index_to_words[0] = '<PAD>'
return ' '.join([index_to_words[prediction] for prediction in np.argmax(logits, 1)])
print('`logits_to_text` function loaded.')
A basic RNN model is a good baseline for sequence data. In this model, you'll build a RNN that translates English to French.
from keras.layers import GRU, Input, Dense, TimeDistributed,Activation
from keras.models import Model, Sequential
from keras.optimizers import Adam
from keras.losses import sparse_categorical_crossentropy
import matplotlib.pyplot as plt #need this in order to do my vsualizations
from keras.callbacks import History #need this in order to do my vsualizations
history = History() #need this in order to do my vsualizations
def simple_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
"""
Build and train a basic RNN on x and y
:param input_shape: Tuple of input shape
:param output_sequence_length: Length of output sequence
:param english_vocab_size: Number of unique English words in the dataset
:param french_vocab_size: Number of unique French words in the dataset
:return: Keras model built, but not trained
"""
# TODO: Build the layers -- Done!
"""
Keras components/tools ...
Sequential -- Linear model that we want to add RNN layers to.
GRU -- Gated recurrent unit/layer (used to improve RNN models)
TimeDistributed -- Wrapper for applying timesteps independently to dense layers
Dense -- Basic densely-connected NN layer
Activation -- Applies an activation function to an output (e.g. softmax, relu, sigmoid, tanh and linear)
"""
model = Sequential()
model.add(GRU(output_sequence_length, input_shape=input_shape[1:], return_sequences=True))
model.add(TimeDistributed(Dense(french_vocab_size)))
model.add(Activation('softmax'))
model.compile(loss=sparse_categorical_crossentropy,
optimizer=Adam(lr=0.005),
metrics=['accuracy'])
return model
tests.test_simple_model(simple_model)
# Reshaping the input to work with a basic RNN
tmp_x = pad(preproc_english_sentences, preproc_french_sentences.shape[1])
tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2], 1))
# Train the neural network
simple_rnn_model = simple_model(
tmp_x.shape,
preproc_french_sentences.shape[1],
len(english_tokenizer.word_index)+1,
len(french_tokenizer.word_index)+1) #hmm...Was getting "val_loss: nan" without the "+1"
simple_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=200, validation_split=0.2, verbose=2, callbacks=[history])
#set verbose=2 because the progress bar was causing the GPU to croak
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('simple model loss - 5 epochs')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train loss', 'validation loss'], loc='upper left')
plt.show()
#batch = 64 => 45s/batch Epoch 5/5 -45s - loss: 1.2744 - acc: 0.6514 - val_loss: 1.2601 - val_acc: 0.6528
#batch = 256 => 12s/batch Epoch 5/5 -12s - loss: 1.4551 - acc: 0.6063 - val_loss: 1.4315 - val_acc: 0.6091
#batch = 512 => 8s/batch Epoch 5/5 -7s - loss: 1.5899 - acc: 0.5917 - val_loss: 1.5655 - val_acc: 0.5979
#batch = 1024 => 6s/batch Epoch 5/5 -5s - loss: 1.7518 - acc: 0.5727 - val_loss: 1.7177 - val_acc: 0.5734
#seems the higher the number of batches the quicker the processing and the better the accuracy,
#so for now we'll go with batch = 1024 unless we observe overfitting
# Print prediction(s)
print(english_sentences[0])
print(french_sentences[0])
print("Prediction:")
print(logits_to_text(simple_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))
You've turned the words into ids, but there's a better representation of a word. This is called word embeddings. An embedding is a vector representation of the word that is close to similar words in n-dimensional space, where the n represents the size of the embedding vectors.
In this model, you'll create a RNN model using embedding.
from keras.layers.embeddings import Embedding
history = History() #need this in order to do my vsualizations
def embed_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
"""
Build and train a RNN model using word embedding on x and y
:param input_shape: Tuple of input shape
:param output_sequence_length: Length of output sequence
:param english_vocab_size: Number of unique English words in the dataset
:param french_vocab_size: Number of unique French words in the dataset
:return: Keras model built, but not trained
"""
# TODO: Implement -- Done!
"""
Keras components/tools ...
Embedding -- Turns positive integers (indexes) into dense vectors of fixed size.
eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]
mask_zero: Whether or not the input value 0 is a special
"padding" value that should be masked out.
"""
model = Sequential()
model.add(Embedding(english_vocab_size, french_vocab_size, input_length=output_sequence_length, mask_zero=True))
model.add(GRU(output_sequence_length, return_sequences=True))
model.add(TimeDistributed(Dense(french_vocab_size)))
model.add(Activation('softmax'))
model.compile(loss=sparse_categorical_crossentropy,
optimizer=Adam(),
metrics=['accuracy'])
return model
tests.test_embed_model(embed_model)
# TODO: Reshape the input -- Done!
# borrowed from simple_model
tmp_x = pad(preproc_english_sentences, preproc_french_sentences.shape[1])
tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2]))
# TODO: Train the neural network -- Done!
# borrowed from simple_model
embed_rnn_model = embed_model(
tmp_x.shape,
preproc_french_sentences.shape[1],
len(english_tokenizer.word_index)+1,
len(french_tokenizer.word_index)+1)
embed_rnn_model.summary()
embed_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=200, validation_split=0.2, verbose=2, callbacks=[history])
# Add visualization
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('embedding model loss - 200 epochs')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train loss', 'validation loss'], loc='upper left')
plt.show()
# TODO: Print prediction(s) -- Done!
# borrowed from simple_model
print(english_sentences[0])
print(logits_to_text(embed_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))
#batch = 64 => 91s/batch Epoch 5/5 -91s - loss: 0.7616 - acc: 0.7788 - val_loss: 0.7380 - val_acc: 0.7846
#batch = 1024 => 91s/batch Epoch 5/5 -10s - loss: 1.9122 - acc: 0.5031 - val_loss: 1.8141 - val_acc: 0.5142
One restriction of a RNN is that it can't see the future input, only the past. This is where bidirectional recurrent neural networks come in. They are able to see the future data.
from keras.layers import Bidirectional
history = History() #need this in order to do my vsualizations
def bd_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
"""
Build and train a bidirectional RNN model on x and y
:param input_shape: Tuple of input shape
:param output_sequence_length: Length of output sequence
:param english_vocab_size: Number of unique English words in the dataset
:param french_vocab_size: Number of unique French words in the dataset
:return: Keras model built, but not trained
"""
# TODO: Implement -- Done!
"""
Keras components/tools ...
Biderctional - Bidirectional wrapper for RNNs. Can be applied tto LSTMs and GRUs
"""
model = Sequential()
model.add(Bidirectional(GRU(output_sequence_length, return_sequences=True), input_shape=input_shape[1:]))
model.add(TimeDistributed(Dense(french_vocab_size)))
model.add(Activation('softmax'))
model.compile(loss=sparse_categorical_crossentropy,
optimizer=Adam(),
metrics=['accuracy'])
return model
tests.test_bd_model(bd_model)
# TODO: Train and Print prediction(s) -- Done!
# borrowed from simple_model
# Reshaping the input
tmp_x = pad(preproc_english_sentences, preproc_french_sentences.shape[1])
tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2], 1))
# Train the neural network
bd_rnn_model = bd_model(
tmp_x.shape,
preproc_french_sentences.shape[1],
len(english_tokenizer.word_index)+1,
len(french_tokenizer.word_index)+1)
bd_rnn_model.summary()
bd_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=200, validation_split=0.2, verbose=2, callbacks=[history])
# Add visualization
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('bidirectional model loss - 200 epochs')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train loss', 'validation loss'], loc='upper left')
plt.show()
print(english_sentences[0])
print(logits_to_text(bd_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))
Time to look at encoder-decoder models. This model is made up of an encoder and decoder. The encoder creates a matrix representation of the sentence. The decoder takes this matrix as input and predicts the translation as output.
Create an encoder-decoder model in the cell below.
from keras.layers import RepeatVector
history = History() #need this in order to do my vsualizations
def encdec_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
"""
Build and train an encoder-decoder model on x and y
:param input_shape: Tuple of input shape
:param output_sequence_length: Length of output sequence
:param english_vocab_size: Number of unique English words in the dataset
:param french_vocab_size: Number of unique French words in the dataset
:return: Keras model built, but not trained
"""
# OPTIONAL: Implement -- Done!
"""
RepeatVector -- Repeats the input n times i.e. output_sequence_length times
"""
model = Sequential()
model.add(GRU(output_sequence_length, input_shape=input_shape[1:], return_sequences=False))
model.add(RepeatVector(output_sequence_length))
model.add(GRU(output_sequence_length, return_sequences=True))
model.add(TimeDistributed(Dense(french_vocab_size)))
model.add(Activation('softmax'))
model.compile(loss=sparse_categorical_crossentropy,
optimizer=Adam(),
metrics=['accuracy'])
return model
tests.test_encdec_model(encdec_model)
# OPTIONAL: Train and Print prediction(s) -- Done!
#train
encdec_rnn_model = encdec_model(
tmp_x.shape,
preproc_french_sentences.shape[1],
len(english_tokenizer.word_index)+1,
len(french_tokenizer.word_index)+1)
encdec_rnn_model.summary()
encdec_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=200, validation_split=0.2, verbose=2, callbacks=[history])
# Add visualization
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('encode-decode model loss 5 epochs')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train loss', 'validation loss'], loc='upper left')
plt.show()
#Print
print(english_sentences[0])
print(logits_to_text(encdec_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))
Use everything you learned from the previous models to create a model that incorporates embedding and a bidirectional rnn into one model.
history = History()
def model_final(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
"""
Build and train a model that incorporates embedding, encoder-decoder, and bidirectional RNN on x and y
:param input_shape: Tuple of input shape
:param output_sequence_length: Length of output sequence
:param english_vocab_size: Number of unique English words in the dataset
:param french_vocab_size: Number of unique French words in the dataset
:return: Keras model built, but not trained
"""
# TODO: Implement -- Done!
model = Sequential()
model.add(Embedding(english_vocab_size, french_vocab_size, input_length=input_shape[1], mask_zero=True))
model.add(Bidirectional(GRU(64, return_sequences=False)))
model.add(RepeatVector(output_sequence_length))
model.add(GRU(64, return_sequences=True))
model.add(TimeDistributed(Dense(french_vocab_size)))
model.add(Activation('softmax'))
model.compile(loss=sparse_categorical_crossentropy,
optimizer=Adam(),
metrics=['accuracy'])
return model
tests.test_model_final(model_final)
print('Final Model Loaded')
import numpy as np
from keras.preprocessing.sequence import pad_sequences
def final_predictions(x, y, x_tk, y_tk):
"""
Gets predictions using the final model
:param x: Preprocessed English data
:param y: Preprocessed French data
:param x_tk: English tokenizer
:param y_tk: French tokenizer
"""
# TODO: Train neural network using model_final
model = model_final(x.shape, y.shape[1], len(x_tk.word_index)+1, len(y_tk.word_index)+1)
model.summary()
model.fit(x, y, batch_size=1024, epochs=100, validation_split=0.2, verbose=2, callbacks=[history])
# Add visualization
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('final model loss - 100 epochs')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train loss', 'validation loss'], loc='upper left')
plt.show()
## DON'T EDIT ANYTHING BELOW THIS LINE
y_id_to_word = {value: key for key, value in y_tk.word_index.items()}
y_id_to_word[0] = '<PAD>'
sentence = 'he saw a old yellow truck'
sentence = [x_tk.word_index[word] for word in sentence.split()]
sentence = pad_sequences([sentence], maxlen=x.shape[-1], padding='post')
sentences = np.array([sentence[0], x[0]])
predictions = model.predict(sentences, len(sentences))
print('Sample 1:')
print(' '.join([y_id_to_word[np.argmax(x)] for x in predictions[0]]))
print('Il a vu un vieux camion jaune')
print('Sample 2:')
print(' '.join([y_id_to_word[np.argmax(x)] for x in predictions[1]]))
print(' '.join([y_id_to_word[np.argmax(x)] for x in y[0]]))
final_predictions(preproc_english_sentences, preproc_french_sentences, english_tokenizer, french_tokenizer)
This one is overfitting, so we're trying fewer epochs


I am a student of computer science with major skills in back-end development and data engineering.
I have worked, during my graduation, in projects of general software development, database development and exploration, game development, and so forth.
GPA: 8.9/10
ICMC WebSiteFoG is an extension group of ICMC-USP focused on game development and knowledge dissemination through courses and events open to the public.
Member: Engaged in the group activities since 2017, focused on back-end development of games and assets.
FoG WebSiteA Scientific Research in Procedural Content Generation for platform games. This research aims the generation of two-dimensional (2D) caves, using the algorithms Perlin Noise and Cellular Automata as base.
In this section, only a sample of the major projects developed will be shown.
A fast-paced 2D arena brawler game with two players, where each one choose a character with different abilities, and fight in a battlefield of their choice.
Development: Responsible for the back-end development of scenario hazards, players movement and attacks, and for the integration of the animations.
M.U.S.H. itch.ioDevelopment of a normalized and consistent database and an application to manage buffets of proms and weddings.
Development: Responsible for database management and construction of triggers in Oracle SQL, in addition to the database connection and front-end application in Java.
DB Project github.comAn application that automatically scans the web searching for events in a specific city and display’s it for the user in a website. It also sends a weekly newsletter with content the user is interested in.
Development: Responsible for creating the front-end using React.js.
Methodology: Agile methodology (SCRUM).
Entretenibit github.comA simulation of the flocking behavior of birds or fish, as known as Boids. The final goal was to use evolutionary algorithms to achieve better behaviors for the flocks.
Development: Responsible for the development of the flocking behaviors and for optimizing the code using the new technology of Unity Engine, called DOTS (high-performance multithreaded Data-Oriented Technology Stack).
Flocking Simulation github.comDevelopment of a data warehouse of an university to track students performance. The work consisted of modeling and documentating the problem, including schematisation and ETL (Extract, Transform, Load). In addition to the implementation and loading of the database and some queries.
Development: This work was produced mostly in group. Responsible for the database implementation and load, and queries implementation.
As has been said before, I am a computer science student and my main interests are in the fields of Game Development and Databases. Besides that, during my spare time, I do like to play some games and watch some content around the internet (like YouTube videos, including Anime fights and others). In addition, I like developing games and related stuff.
As simple as it seems what was said, I have done some things during my life until now, such as read some books, English and Japanese classes, Piano classes, and even Swimming ones. At last, but not less important, I like to finish the things that I have started, when possible.


The following method, CoffeesTable.viewTable
outputs the contents of the COFFEES
tables, and demonstrates the use of ResultSet
objects and cursors:
public static void viewTable(Connection con) throws SQLException { Statement stmt = null; String query = "select COF_NAME, SUP_ID, PRICE, SALES, TOTAL from " + dbName + ".COFFEES"; try { stmt = con.createStatement(); ResultSet rs = stmt.executeQuery(query); while (rs.next()) { String coffeeName = rs.getString("COF_NAME"); int supplierID = rs.getInt("SUP_ID"); float price = rs.getFloat("PRICE"); int sales = rs.getInt("SALES"); int total = rs.getInt("TOTAL"); System.out.println(coffeeName + "\t" + supplierID + "\t" + price + "\t" + sales + "\t" + total); } } catch (SQLException e ) { JDBCTutorialUtilities.printSQLException(e); } finally { if (stmt != null) { stmt.close(); } } }
A ResultSet
object is a table of data representing a database result set, which is usually generated by executing a statement that queries the database. For example, the CoffeeTables.viewTable
method creates a ResultSet
, rs
, when it executes the query through the Statement
object, stmt
. Note that a ResultSet
object can be created through any object that implements the Statement
interface, including PreparedStatement
, CallableStatement
, and RowSet
.
You access the data in a ResultSet
object through a cursor. Note that this cursor is not a database cursor. This cursor is a pointer that points to one row of data in the ResultSet
. Initially, the cursor is positioned before the first row. The method ResultSet.next
moves the cursor to the next row. This method returns false
if the cursor is positioned after the last row. This method repeatedly calls the ResultSet.next
method with a while
loop to iterate through all the data in the ResultSet
.
This page covers the following topics:
The ResultSet
interface provides methods for retrieving and manipulating the results of executed queries, and ResultSet
objects can have different functionality and characteristics. These characteristics are type, concurrency, and cursor holdability.
The type of a ResultSet
object determines the level of its functionality in two areas: the ways in which the cursor can be manipulated, and how concurrent changes made to the underlying data source are reflected by the ResultSet
object.
The sensitivity of a ResultSet
object is determined by one of three different ResultSet
types:
TYPE_FORWARD_ONLY
: The result set cannot be scrolled; its cursor moves forward only, from before the first row to after the last row. The rows contained in the result set depend on how the underlying database generates the results. That is, it contains the rows that satisfy the query at either the time the query is executed or as the rows are retrieved.TYPE_SCROLL_INSENSITIVE
: The result can be scrolled; its cursor can move both forward and backward relative to the current position, and it can move to an absolute position. The result set is insensitive to changes made to the underlying data source while it is open. It contains the rows that satisfy the query at either the time the query is executed or as the rows are retrieved.TYPE_SCROLL_SENSITIVE
: The result can be scrolled; its cursor can move both forward and backward relative to the current position, and it can move to an absolute position. The result set reflects changes made to the underlying data source while the result set remains open.The default ResultSet
type is TYPE_FORWARD_ONLY
.
Note: Not all databases and JDBC drivers support all ResultSet
types. The method DatabaseMetaData.supportsResultSetType
returns true
if the specified ResultSet
type is supported and false
otherwise.
The concurrency of a ResultSet
object determines what level of update functionality is supported.
There are two concurrency levels:
CONCUR_READ_ONLY
: The ResultSet
object cannot be updated using the ResultSet
interface.CONCUR_UPDATABLE
: The ResultSet
object can be updated using the ResultSet
interface.The default ResultSet
concurrency is CONCUR_READ_ONLY
.
Note: Not all JDBC drivers and databases support concurrency. The method DatabaseMetaData.supportsResultSetConcurrency
returns true
if the specified concurrency level is supported by the driver and false
otherwise.
The method CoffeesTable.modifyPrices
demonstrates how to use a ResultSet
object whose concurrency level is CONCUR_UPDATABLE
.
Calling the method Connection.commit
can close the ResultSet
objects that
have been created during the current transaction. In some cases, however, this may
not be the desired behavior. The ResultSet
property holdability gives the
application control over whether ResultSet
objects (cursors) are closed when
commit is called.
The following ResultSet
constants may be supplied to the Connection
methods
createStatement
, prepareStatement
, and prepareCall
:
HOLD_CURSORS_OVER_COMMIT
: ResultSet
cursors are not closed; they are holdable: they are held open when the method commit
is called. Holdable cursors might be ideal if your application uses mostly read-only ResultSet
objects.CLOSE_CURSORS_AT_COMMIT
: ResultSet
objects (cursors) are closed when the commit
method is called. Closing cursors when this method is called can result in better performance for some applications.The default cursor holdability varies depending on your DBMS.
Note: Not all JDBC drivers and databases support holdable and non-holdable cursors. The following method, JDBCTutorialUtilities.cursorHoldabilitySupport
, outputs the default cursor holdability of ResultSet
objects and whether HOLD_CURSORS_OVER_COMMIT
and CLOSE_CURSORS_AT_COMMIT
are supported:
public static void cursorHoldabilitySupport(Connection conn) throws SQLException { DatabaseMetaData dbMetaData = conn.getMetaData(); System.out.println("ResultSet.HOLD_CURSORS_OVER_COMMIT = " + ResultSet.HOLD_CURSORS_OVER_COMMIT); System.out.println("ResultSet.CLOSE_CURSORS_AT_COMMIT = " + ResultSet.CLOSE_CURSORS_AT_COMMIT); System.out.println("Default cursor holdability: " + dbMetaData.getResultSetHoldability()); System.out.println("Supports HOLD_CURSORS_OVER_COMMIT? " + dbMetaData.supportsResultSetHoldability( ResultSet.HOLD_CURSORS_OVER_COMMIT)); System.out.println("Supports CLOSE_CURSORS_AT_COMMIT? " + dbMetaData.supportsResultSetHoldability( ResultSet.CLOSE_CURSORS_AT_COMMIT)); }
The ResultSet
interface declares getter methods (for example, getBoolean
and getLong
) for retrieving column values from the current row. You can retrieve values using either the index number of the column or the alias or name of the column. The column index is usually more efficient. Columns are numbered from 1. For maximum portability, result set columns within each row should be read in left-to-right order, and each column should be read only once.
For example, the following method, CoffeesTable.alternateViewTable
, retrieves column values by number:
public static void alternateViewTable(Connection con) throws SQLException { Statement stmt = null; String query = "select COF_NAME, SUP_ID, PRICE, SALES, TOTAL from COFFEES"; try { stmt = con.createStatement(); ResultSet rs = stmt.executeQuery(query); while (rs.next()) { String coffeeName = rs.getString(1); int supplierID = rs.getInt(2); float price = rs.getFloat(3); int sales = rs.getInt(4); int total = rs.getInt(5); System.out.println(coffeeName + "\t" + supplierID + "\t" + price + "\t" + sales + "\t" + total); } } catch (SQLException e ) { JDBCTutorialUtilities.printSQLException(e); } finally { if (stmt != null) { stmt.close(); } } }
Strings used as input to getter methods are case-insensitive. When a getter method is called with a string and more than one column has the same alias or name as the string, the value of the first matching column is returned. The option to use a string as opposed to an integer is designed to be used when column aliases and names are used in the SQL query that generated the result set. For columns that are not explicitly named in the query (for example, select * from COFFEES
) it is best to use column numbers. If column names are used, the developer should guarantee that they uniquely refer to the intended columns by using column aliases. A column alias effectively renames the column of a result set. To specify a column alias, use the SQL AS
clause in the SELECT
statement.
The getter method of the appropriate type retrieves the value in each column. For example, in the method CoffeeTables.viewTable
, the first column in each row of the ResultSet
rs
is COF_NAME
, which stores a value of SQL type VARCHAR
. The method for retrieving a value of SQL type VARCHAR
is getString
. The second column in each row stores a value of SQL type INTEGER
, and the method for retrieving values of that type is getInt
.
Note that although the method getString
is recommended for retrieving the SQL types CHAR
and VARCHAR
, it is possible to retrieve any of the basic SQL types with it. Getting all values with getString
can be very useful, but it also has its limitations. For instance, if it is used to retrieve a numeric type, getString
converts the numeric value to a Java String
object, and the value has to be converted back to a numeric type before it can be operated on as a number. In cases where the value is treated as a string anyway, there is no drawback. Furthermore, if you want an application to retrieve values of any standard SQL type other than SQL3 types, use the getString
method.
As mentioned previously, you access the data in a ResultSet
object through a cursor, which points to one row in the ResultSet
object. However, when a ResultSet
object is first created, the cursor is positioned before the first row. The method CoffeeTables.viewTable
moves the cursor by calling the ResultSet.next
method. There are other methods available to move the cursor:
next
: Moves the cursor forward one row. Returns true
if the cursor is now positioned on a row and false
if the cursor is positioned after the last row.previous
: Moves the cursor backward one row. Returns true
if the cursor is now positioned on a row and false
if the cursor is positioned before the first row.first
: Moves the cursor to the first row in the ResultSet
object. Returns true
if the cursor is now positioned on the first row and false
if the ResultSet
object does not contain any rows.
last:
: Moves the cursor to the last row in the ResultSet
object. Returns true
if the cursor is now positioned on the last row and false
if the ResultSet
object does not contain any rows.beforeFirst
: Positions the cursor at the start of the ResultSet
object, before the first row. If the ResultSet
object does not contain any rows, this method has no effect.afterLast
: Positions the cursor at the end of the ResultSet
object, after the last row. If the ResultSet
object does not contain any rows, this method has no effect.relative(int rows)
: Moves the cursor relative to its current position.absolute(int row)
: Positions the cursor on the row specified by the parameter row
.Note that the default sensitivity of a ResultSet
is TYPE_FORWARD_ONLY
, which means that it cannot be scrolled; you cannot call any of these methods that move the cursor, except next
, if your ResultSet
cannot be scrolled. The method CoffeesTable.modifyPrices
, described in the following section, demonstrates how you can move the cursor of a ResultSet
.
You cannot update a default ResultSet
object, and you can only move its cursor forward. However, you can create ResultSet
objects that can be scrolled (the cursor can move backwards or move to an absolute position) and updated.
The following method, CoffeesTable.modifyPrices
, multiplies the PRICE
column of each row by the argument percentage
:
public void modifyPrices(float percentage) throws SQLException { Statement stmt = null; try { stmt = con.createStatement(); stmt = con.createStatement( ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_UPDATABLE); ResultSet uprs = stmt.executeQuery( "SELECT * FROM " + dbName + ".COFFEES"); while (uprs.next()) { float f = uprs.getFloat("PRICE"); uprs.updateFloat("PRICE", f * percentage); uprs.updateRow(); } } catch (SQLException e ) { JDBCTutorialUtilities.printSQLException(e); } finally { if (stmt != null) { stmt.close(); } } }
The field ResultSet.TYPE_SCROLL_SENSITIVE
creates a ResultSet
object whose cursor can move both forward and backward relative to the current position and to an absolute position. The field ResultSet.CONCUR_UPDATABLE
creates a ResultSet
object that can be updated. See the ResultSet
Javadoc for other fields you can specify to modify the behavior of ResultSet
objects.
The method ResultSet.updateFloat
updates the specified column (in this example, PRICE
with the specified float
value in the row where the cursor is positioned. ResultSet
contains various updater methods that enable you to update column values of various data types. However, none of these updater methods modifies the database; you must call the method ResultSet.updateRow
to update the database.
Statement
, PreparedStatement
and CallableStatement
objects have a list of commands that is associated with them. This list may contain statements for updating, inserting, or deleting a row; and it may also contain DDL statements such as CREATE TABLE
and DROP TABLE
. It cannot, however, contain a statement that would produce a ResultSet
object, such as a SELECT
statement. In other words, the list can contain only statements that produce an update count.
The list, which is associated with a Statement
object at its creation, is initially empty. You can add SQL commands to this list with the method addBatch
and empty it with the method clearBatch
. When you have finished adding statements to the list, call the method executeBatch
to send them all to the database to be executed as a unit, or batch.
For example, the following method CoffeesTable.batchUpdate
adds four rows to the COFFEES
table with a batch update:
public void batchUpdate() throws SQLException { Statement stmt = null; try { this.con.setAutoCommit(false); stmt = this.con.createStatement(); stmt.addBatch("INSERT INTO COFFEES " + "VALUES('Amaretto', 49, 9.99, 0, 0)"); stmt.addBatch("INSERT INTO COFFEES " + "VALUES('Hazelnut', 49, 9.99, 0, 0)"); stmt.addBatch("INSERT INTO COFFEES " + "VALUES('Amaretto_decaf', 49, 10.99, 0, 0)"); stmt.addBatch("INSERT INTO COFFEES " + "VALUES('Hazelnut_decaf', 49, 10.99, 0, 0)"); int [] updateCounts = stmt.executeBatch(); this.con.commit(); } catch(BatchUpdateException b) { JDBCTutorialUtilities.printBatchUpdateException(b); } catch(SQLException ex) { JDBCTutorialUtilities.printSQLException(ex); } finally { if (stmt != null) { stmt.close(); } this.con.setAutoCommit(true); } }
The following line disables auto-commit mode for the Connection
object con so that the transaction will not be automatically committed or rolled back when the method executeBatch
is called.
this.con.setAutoCommit(false);
To allow for correct error handling, you should always disable auto-commit mode before beginning a batch update.
The method Statement.addBatch
adds a command to the list of commands associated with the Statement
object stmt
. In this example, these commands are all INSERT INTO
statements, each one adding a row consisting of five column values. The values for the columns COF_NAME
and PRICE
are the name of the coffee and its price, respectively. The second value in each row is 49 because that is the identification number for the supplier, Superior Coffee. The last two values, the entries for the columns SALES
and TOTAL
, all start out being zero because there have been no sales yet. (SALES
is the number of pounds of this row's coffee sold in the current week; TOTAL
is the total of all the cumulative sales of this coffee.)
The following line sends the four SQL commands that were added to its list of commands to the database to be executed as a batch:
int [] updateCounts = stmt.executeBatch();
Note that stmt
uses the method executeBatch
to send the batch of insertions, not the method executeUpdate
, which sends only one command and returns a single update count. The DBMS executes the commands in the order in which they were added to the list of commands, so it will first add the row of values for Amaretto, then add the row for Hazelnut, then Amaretto decaf, and finally Hazelnut decaf. If all four commands execute successfully, the DBMS will return an update count for each command in the order in which it was executed. The update counts that indicate how many rows were affected by each command are stored in the array updateCounts
.
If all four of the commands in the batch are executed successfully, updateCounts
will contain four values, all of which are 1 because an insertion affects one row. The list of commands associated with stmt
will now be empty because the four commands added previously were sent to the database when stmt
called the method executeBatch
. You can at any time explicitly empty this list of commands with the method clearBatch
.
The Connection.commit
method makes the batch of updates to the COFFEES
table permanent. This method needs to be called explicitly because the auto-commit mode for this connection was disabled previously.
The following line enables auto-commit mode for the current Connection
object.
this.con.setAutoCommit(true);
Now each statement in the example will automatically be committed after it is executed, and it no longer needs to invoke the method commit
.
It is also possible to have a parameterized batch update, as shown in the following code fragment, where con
is a Connection
object:
con.setAutoCommit(false); PreparedStatement pstmt = con.prepareStatement( "INSERT INTO COFFEES VALUES( ?, ?, ?, ?, ?)"); pstmt.setString(1, "Amaretto"); pstmt.setInt(2, 49); pstmt.setFloat(3, 9.99); pstmt.setInt(4, 0); pstmt.setInt(5, 0); pstmt.addBatch(); pstmt.setString(1, "Hazelnut"); pstmt.setInt(2, 49); pstmt.setFloat(3, 9.99); pstmt.setInt(4, 0); pstmt.setInt(5, 0); pstmt.addBatch(); // ... and so on for each new type of coffee int [] updateCounts = pstmt.executeBatch(); con.commit(); con.setAutoCommit(true);
You will get a BatchUpdateException
when you call the method executeBatch
if (1) one of the SQL statements you added to the batch produces a result set (usually a query) or (2) one of the SQL statements in the batch does not execute successfully for some other reason.
You should not add a query (a SELECT
statement) to a batch of SQL commands because the method executeBatch
, which returns an array of update counts, expects an update count from each SQL statement that executes successfully. This means that only commands that return an update count (commands such as INSERT INTO
, UPDATE
, DELETE
) or that return 0 (such as CREATE TABLE
, DROP TABLE
, ALTER TABLE
) can be successfully executed as a batch with the executeBatch
method.
A BatchUpdateException
contains an array of update counts that is similar to the array returned by the method executeBatch
. In both cases, the update counts are in the same order as the commands that produced them. This tells you how many commands in the batch executed successfully and which ones they are. For example, if five commands executed successfully, the array will contain five numbers: the first one being the update count for the first command, the second one being the update count for the second command, and so on.
BatchUpdateException
is derived from SQLException
. This means that you can use all of the methods available to an SQLException
object with it. The following method, JDBCTutorialUtilities.printBatchUpdateException
prints all of the SQLException
information plus the update counts contained in a BatchUpdateException
object. Because BatchUpdateException.getUpdateCounts
returns an array of int
, the code uses a for
loop to print each of the update counts:
public static void printBatchUpdateException(BatchUpdateException b) { System.err.println("----BatchUpdateException----"); System.err.println("SQLState: " + b.getSQLState()); System.err.println("Message: " + b.getMessage()); System.err.println("Vendor: " + b.getErrorCode()); System.err.print("Update counts: "); int [] updateCounts = b.getUpdateCounts(); for (int i = 0; i < updateCounts.length; i++) { System.err.print(updateCounts[i] + " "); } }
Note: Not all JDBC drivers support inserting new rows with the ResultSet
interface. If you attempt to insert a new row and your JDBC driver database does not support this feature, a SQLFeatureNotSupportedException
exception is thrown.
The following method, CoffeesTable.insertRow
, inserts a row into the COFFEES
through a ResultSet
object:
public void insertRow(String coffeeName, int supplierID, float price, int sales, int total) throws SQLException { Statement stmt = null; try { stmt = con.createStatement( ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_UPDATABLE); ResultSet uprs = stmt.executeQuery( "SELECT * FROM " + dbName + ".COFFEES"); uprs.moveToInsertRow(); uprs.updateString("COF_NAME", coffeeName); uprs.updateInt("SUP_ID", supplierID); uprs.updateFloat("PRICE", price); uprs.updateInt("SALES", sales); uprs.updateInt("TOTAL", total); uprs.insertRow(); uprs.beforeFirst(); } catch (SQLException e ) { JDBCTutorialUtilities.printSQLException(e); } finally { if (stmt != null) { stmt.close(); } } }
This example calls the Connection.createStatement
method with two arguments, ResultSet.TYPE_SCROLL_SENSITIVE
and ResultSet.CONCUR_UPDATABLE
. The first value enables the cursor of the ResultSet
object to be moved both forward and backward. The second value, ResultSet.CONCUR_UPDATABLE
, is required if you want to insert rows into a ResultSet
object; it specifies that it can be updatable.
The same stipulations for using strings in getter methods also apply to updater methods.
The method ResultSet.moveToInsertRow
moves the cursor to the insert row. The insert row is a special row associated with an updatable result set. It is essentially a buffer where a new row can be constructed by calling the updater methods prior to inserting the row into the result set. For example, this method calls the method ResultSet.updateString
to update the insert row's COF_NAME
column to Kona
.
The method ResultSet.insertRow
inserts the contents of the insert row into the ResultSet
object and into the database.
Note: After inserting a row with the ResultSet.insertRow
, you should move the cursor to a row other than the insert row. For example, this example moves it to before the first row in the result set with the method ResultSet.beforeFirst
. Unexpected results can occur if another part of your application uses the same result set and the cursor is still pointing to the insert row.


Coffee is one of the world's favorite drinks and is enjoyed on a daily basis by millions of people. Water is the only beverage that's more popular and Thomas Jefferson called coffee, "the favorite drink of the civilized world. While it's unclear exactly where coffee comes from, legend has it that the stimulating effects of coffee berries were accidentally discovered by a goat herder who noticed that his goats were particularly energetic after grazing berries from a specific tree. He passed on his findings to an abbot who made a drink with the berries, which kept him alert through long evening prayers."
Coffee is darkly colored, bitter, slightly acidic and has a stimulating effect in humans, primarily due to its caffeine content. It is one of the most popular drinks in the world, and it can be prepared and presented in a variety of ways (e.g., espresso, French press, caffè latte). It is usually served hot, although iced coffee is a popular alternative.
Clinical studies indicate that moderate coffee consumption is benign or mildly beneficial in healthy adults, with continuing research on whether long-term consumption reduces the risk of some diseases, although those long-term studies are generally of poor quality
Research has linking moderate consumption (about three to four cups per day) with a longer life span, plus a reduced risk of cardiovascular disease, type 2 diabetes and Parkinson's, according to Harvard Health Publishing.


Did you know? Audrey Hepburn's envied slender
figure was a result of malnourishment during her adolescent period as she nearly starved
when the Nazis invaded Holland and cut off food supplies during World War II.


--- layout: post title: The Pinball Number Count post_id: 323 categories: - life date: 2004-11-02 10:15:00.000000000 -07:00 author: Jamis permalink: 2004/11/2/the-pinball-number-count.html ---
Let me just take a moment to wallow in nostalgia.
Wallow. Wallow. Wallow.
I’m sorry that you had to see that, but I just stopped by the blog of whytheluckystiff and discovered that he had uncovered the history of the Pinball Number Count song!
Yes, you heard it correctly the first time. That’s the Pinball Number Count song! He has both the song, and the video, right there on his site.
This is the song (and video) that I remember most vividly from my childhood TV watching days. Watching that glittering silver ball bounce through a freak-show pinball game, with the words “one-two-three-foooour-five, six-se-ven-eight-niiiiine-ten, eleven-twelve!” echoing around me.
Those were the days. Thanks for taking me back, _why.


49th Security Division
What are CTF's?
A Capture the Flag (CTF) competition is a type of computer security competition in which a person or team is given a variety of problems to complete. To receive points, the person or team must submit the proper flag or solution, which is discovered through completing the tasks. We've put up a little CTF for you to try out. If you're having trouble finding the flag, click the hint button to assist you.
Challenge 1: HTML
Challenge 2: Decode
Challenge 3: Log Analysis
For this challenge you'll need this log file
Question 1
Question 2
Question 3
Question 4
Question 5
Challenge 4: Cryptography
More challenges are on the way...


Honor Magic Watch 2 promo images show sleek design and thin bezel
by MichailLast week we came across the design sketch for the upcoming Honor Watch 2 and now we’ve got hold of some promo images of the device as well as a live shot which gives us our first look.
The watch face is about what you’d expect a modern wristwatch. There’s a tachymeter scale design on the outer rim of the casing which adds to the sleekness and the bezel as a whole appears pretty slim. The right hand-side will feature two control dials for which will be used to operate the UI. Last week’s sketches revealed the watch will come in a 46mm version and will feature 2.5D glass.
According to speculations, Honor will release two variants of the Magic Watch 2, one of which is rumored to be a women’s version. Honor is holding an event on November 26 in Beijing for the upcoming V30 and the new Magic Watch 2.


Coos Bay, Oregon's largest bay, is the commercial centre of the state's southern coast, and is a busy commercial port. It is located near some of the finest scenery in the Northwest.
Charleston is an attractive town.
Position 43° 20.8' N, 124° 19.37' W. (Charleston marina)
On entering Coos Bay you must go to the Charleston Marina Visitor's Dock to clear with Customs and Border Protection (CBP). Charleston Marina is located in Charleston which is to the south just after entering the bay. The visitor's dock is the first pier located beside the commercial fishing fleet.
It reduces the waiting time if prior notice of your arrival is given.
Last updated November 2014.
There is a bar at the entrance to Coos Bay. Call the Coast Guard on Tel: (888-3102) for current conditions.
Charleston harbour, just inside Coos Bay, is home to a busy fishing fleet, which supplies a variety of local seafood processors. It's also has a marina. The harbour area includes restaurants, shops, marine supplies and repair operations, and other firms offering a variety of services. All within comfortable walking distance.
The town of Coos Bay is a further 8 miles from the entrance.
Last updated November 2014.


WITNESS LIST State Affairs Committee March 30, 1999-11:00A SB 545 FOR: Andrus, Ken (Self), Carthage SB 846 ON: Williams, Charles E. (State Board of Barber Examiners), San Antonio Registering, but not testifying: On: Brown, Will K. (State Board of Barber Examiners), Austin Lennox, Douglas (State Board of Barber Examiners), Austin SB 872 ON: Herring, Mike (Texas Department of Parks and Wildlife), Austin SB 1047 FOR: Nowonty, Patrick (Texas Economic Development Council), Waco Shields, Chris (Texas Economic Development Council), Austin Registering, but not testifying: For: Woodford, Suzy (Common Cause), Austin On: Ballard, Don (Attorney General's Office-Open Records Division), Austin SB 1656 FOR: Kulvicki, Daniel B. (Self), Dickinson


Mip Map Generation¶
You can generate different MIP map level for the original image by selecting the image (Ruby.bmp in this example) on the project explorer and then click on the “Generate MIP Map” button on the Application Toolbar.
Note: Only one image on the project explorer can be selected for MIP map generation at one time.
A Generate MIP Maps window will appear.
Select the desired Mip-Level for MIP map generation from the drop down list. This drop down list is generated based on the selected original image and the intended targets EncodeWith options (CPU or GPU based). Select the lowest mip-level you want to generate, then press the Generate button.
When MIP map generation completes, will appear in Image View toolbar. Click on the little triangle to expand the drop down list.
You can select the view of each MIP level from the list.
For example, selecting the level 3 will update the current image view as shown below
Image View of Ruby BMP file at Mip Level 3
You can always re-generate MIP map for the same image by repeating the all the steps mentioned in this section.
MIP Map Filter Support Using either GPU or CPU¶
The GUI applications “Generate MIP Map” option can be used on a variety of image formats to produce MIP maps filtered for optimal GPU or CPU use.
Options for using a Compressonator Box filter or DirectX® based filters are provided.
Applying Gamma to MIP Maps¶
Gamma correction can be applied to the image pixels after MIP map generation by changing the default 1.0 value higher (darken the image) or lower (brighten the image).
GPU Based Compression and MIP Map generation¶
An alternative option to generate MIP map levels is provided using the application setting window. A feature to encode textures using just the GPU hardware and its driver components is provided. The encoding process uses no user application shader code to process the textures. Both the GUI and Command-line tools can run BCn GPU extensions provided by most GPU vendors and can be used to evaluate the quality and performance of encoded images using the image views and analysis setting.
Just enable the GUI “Application Settings” options to set the encoding with GPU and optionally set generating GPU based mipmaps as shown:
Once the images are processed users can view the quality of the resulting images with the applications image view PSNR feature,
Cube Maps¶
Cube Maps can be compressed with or without MIP Levels. Only a limited set of texture types (RGBA_8888 and RGBA_F16) are currently supported in DDS and KTX file formats. Compressing, Generating MIP Levels and Viewing Cube Maps uses the same process as image textures. Just place the file onto the Project Explorer and process them as required.
A new notation is used for the cube faces labeled as “Frames” for each cube face. For cube mapped files maximum frames is set to 6. Support for volumetric texture files is been reviewed and the frames limit will be expanded as needed.
When textures are added to the Project Explorer, the properties view will now display the type of texture as either a 2D or Cube Map, The Depth field is used for the frames size, the Depth field will also be used to indicate the z component of a 3D image (These notations may change in future revisions).
View Image Difference¶
To view the difference between a processed image (Ruby_bmp_1) and original image (Ruby_bmp), right mouse button click on Ruby_bmp_1 and select View Image Diff from the context menu or select the View Image Diff Icon on the tool bar
Mouse right mouse button click over Ruby_bmp_1 showing Context menu
You will now see a comparison of the original image with the compressed image
Image Difference view
Note: The windows in this view are not movable.
Analyzing Compressed Images¶
After clicking View Image diff, you can run analysis on the images that show various statistics such as MSE, PSNR and Similarity Indices (SSIM) by selecting on the top right corner of the image diff view.
When analysis process completed, the statistics result will be shown on the Property View


|
lwIP
2.1.3
Lightweight IP stack
|
| #define LWIP_SNMP 0 |
LWIP_SNMP==1: This enables the lwIP SNMP agent. UDP must be available for SNMP transport. If you want to use your own SNMP agent, leave this disabled. To integrate MIB2 of an external agent, you need to enable LWIP_MIB2_CALLBACKS and MIB2_STATS. This will give you the callbacks and statistics counters you need to get MIB2 working.
| #define SNMP_COMMUNITY "public" |
The snmp read-access community. Used for write-access and traps, too unless SNMP_COMMUNITY_WRITE or SNMP_COMMUNITY_TRAP are enabled, respectively.
| #define SNMP_COMMUNITY_TRAP "public" |
The snmp community used for sending traps.
| #define SNMP_COMMUNITY_WRITE "private" |
The snmp write-access community. Set this community to "" in order to disallow any write access.
| #define SNMP_DEBUG LWIP_DBG_OFF |
SNMP_DEBUG: Enable debugging for SNMP messages.
| #define SNMP_DEVICE_ENTERPRISE_OID {1, 3, 6, 1, 4, 1, SNMP_LWIP_ENTERPRISE_OID} |
IANA assigned enterprise ID for lwIP is 26381
If you need to create your own private MIB you'll need to apply for your own enterprise ID with IANA: http://www.iana.org/numbers.html
| #define SNMP_DEVICE_ENTERPRISE_OID_LEN 7 |
Length of SNMP_DEVICE_ENTERPRISE_OID
| #define SNMP_LWIP_ENTERPRISE_OID 26381 |
The OID identifiying the device. This may be the enterprise OID itself or any OID located below it in tree.
| #define SNMP_LWIP_GETBULK_MAX_REPETITIONS 0 |
This value is used to limit the repetitions processed in GetBulk requests (value == 0 means no limitation). This may be useful to limit the load for a single request. According to SNMP RFC 1905 it is allowed to not return all requested variables from a GetBulk request if system load would be too high. so the effect is that the client will do more requests to gather all data. For the stack this could be useful in case that SNMP processing is done in TCP/IP thread. In this situation a request with many repetitions could block the thread for a longer time. Setting limit here will keep the stack more responsive.
| #define SNMP_LWIP_MIB2 LWIP_SNMP |
Indicates if the MIB2 implementation of LWIP SNMP stack is used.
| #define SNMP_LWIP_MIB2_SYSCONTACT "" |
Value return for sysContact field of MIB2. To make sysContact field settable, call snmp_mib2_set_syscontact() to provide the necessary buffers.
| #define SNMP_LWIP_MIB2_SYSDESC "lwIP" |
Value return for sysDesc field of MIB2.
| #define SNMP_LWIP_MIB2_SYSLOCATION "" |
Value return for sysLocation field of MIB2. To make sysLocation field settable, call snmp_mib2_set_syslocation() to provide the necessary buffers.
| #define SNMP_LWIP_MIB2_SYSNAME "FQDN-unk" |
Value return for sysName field of MIB2. To make sysName field settable, call snmp_mib2_set_sysname() to provide the necessary buffers.
| #define SNMP_MAX_COMMUNITY_STR_LEN LWIP_MAX(LWIP_MAX(sizeof(SNMP_COMMUNITY), sizeof(SNMP_COMMUNITY_WRITE)), sizeof(SNMP_COMMUNITY_TRAP)) |
The maximum length of community string. If community names shall be adjusted at runtime via snmp_set_community() calls, enter here the possible maximum length (+1 for terminating null character).
| #define SNMP_MAX_OBJ_ID_LEN 50 |
The maximum number of Sub ID's inside an object identifier. Indirectly this also limits the maximum depth of SNMP tree.
| #define SNMP_MAX_OCTET_STRING_LEN 127 |
The maximum length of strings used.
| #define SNMP_MAX_VALUE_SIZE LWIP_MAX(LWIP_MAX((SNMP_MAX_OCTET_STRING_LEN), sizeof(u32_t)*(SNMP_MAX_OBJ_ID_LEN)), SNMP_MIN_VALUE_SIZE) |
The maximum size of a value.
| #define SNMP_MIB_DEBUG LWIP_DBG_OFF |
SNMP_MIB_DEBUG: Enable debugging for SNMP MIBs.
| #define SNMP_MIN_VALUE_SIZE (2 * sizeof(u32_t*)) /* size required to store the basic types (8 bytes for counter64) */ |
The minimum size of a value.
| #define SNMP_SAFE_REQUESTS 1 |
Only allow SNMP write actions that are 'safe' (e.g. disabling netifs is not a safe action and disabled when SNMP_SAFE_REQUESTS = 1). Unsafe requests are disabled by default!
| #define SNMP_STACK_SIZE DEFAULT_THREAD_STACKSIZE |
SNMP_STACK_SIZE: Stack size of SNMP netconn worker thread
| #define SNMP_THREAD_PRIO DEFAULT_THREAD_PRIO |
SNMP_THREAD_PRIO: SNMP netconn worker thread priority
| #define SNMP_TRAP_DESTINATIONS 1 |
SNMP_TRAP_DESTINATIONS: Number of trap destinations. At least one trap destination is required
| #define SNMP_USE_NETCONN 0 |
SNMP_USE_NETCONN: Use netconn API instead of raw API. Makes SNMP agent run in a worker thread, so blocking operations can be done in MIB calls.
| #define SNMP_USE_RAW 1 |
SNMP_USE_RAW: Use raw API. SNMP agent does not run in a worker thread, so blocking operations should not be done in MIB calls.


Can Wales's dragons roar into Euro 2020 with win over Hungary?
by Derek BiltonA slight profit from the last round of international fixtures after we correctly called a big win for England against Montenegro. The international break just keeps on going, though, with some more crucial fixtures in the next few days, including Wales' must-win clash against Hungary.
Running total: +£26.40
The Banker
It's do-or-die for Wales on Tuesday evening. Slovakia's loss in Croatia means any win for Ryan Giggs' side against Hungary will be enough for them to qualify for Euro 2020. Wales cruised into Tuesday's decisive clash with a mature performance as they beat Azerbaijan 2-0 with minimal fuss in Baku.
Anything other than three points here, though, and the Welsh will be forced to endure the lottery of the playoffs. It really is all to play for too. Hungary can go through with a win or indeed a draw in Cardiff, so the atmosphere should be crackling in the Welsh capital.
Hungary are a decent side but Euro 2016 semifinalists Wales have a formidable recent record on home soil in qualifying (they are unbeaten in 10 matches) and a raucous Cardiff crowd can help drag the 8/11 shots over the line here.
Selection: £5.50 on Wales to beat Hungary at 8/11 with bet365.
The First-Goal Flutter
Republic of Ireland have been usurped by Denmark at the top of Group D so Monday night's clash in Dublin is a big one. The Danes are unbeaten in their last 14 matches and it could be worth a small punt on 'No Goalscorer' at a tidy 13/2. Ireland have only scored six goals in this qualifying campaign, and two of them came at home to Gibraltar. It could be cagey, and if the Irish can keep the shackles on talisman Christian Eriksen there's a good chance this finishes in a stalemate.
Selection: £2 on No Goalscorer at 13/2 with bet365.
The Tasty Treble
Spain have scored 16 goals and conceded just once in four home qualifiers so far and look a punt at 6/4 to cover a two-goal handicap against a very mediocre Romania side.
Elsewhere, we also like Italy to cover a two-goal handicap against Armenia at 21/20. The Italians have recovered from their failure to qualify for Russia 2018 by sauntering to nine wins out of nine in qualifying for Euro 2020. They should make that 10 wins out of 10 and have enough quality to win this by at least a couple.
Finally add in Scotland at 8/15 to beat Kazakhstan. From a betting perspective Scotland are often tough to trust but should be good enough on home soil and will be confident after that win in Cyprus. The Euro 2020 treble comes in at better than 13/2 with bet365 and should go close.
Selection: £2 on the treble at 6.85/1 with bet365.
The Wildcard Tip
Germany are red-hot betting favourites when they host Northern Ireland on Tuesday. A record of six wins from seven matches has left Germany top of Group C heading into their final match, and they can finish with a flourish in Frankfurt.
A 3-0 win for Joachim Low's side is available at 6/1 with bet365 and they should be motivated as anything less than a win opens the door for Netherlands to snatch top spot in the group.
Selection: £1 Germany to win 3-0 at 6/1 with bet365.
18+ http://begambleaware/ Terms and conditions apply.