Available toolkit artifacts include a full distribution zip file as well as binary and source jars in a public Maven 2 repository. The toolkit is documented through a tutorial, API documentation, and several sample test suites.
Follow these instructions to start using JMockit:
jmockit
directory containing the library jars and source, samples, Javadocs
and additional documentation.
If you want to build the JMockit project itself using Ant, then you should also download and unpack (to the
"jmockit" dir) the jmockit-libs.zip
file. To build with Maven, simply use the "jmockit/pom.xml" file
from the full distribution, or from a SVN checkout.
jmockit.jar
to your classpath (in your Java IDE project, Ant
build file, and so on, as required).
-javaagent:jmockit.jar
" (with the proper absolute or relative path to
jmockit.jar
) is passed as an initialization parameter to the JVM when running tests.
This standard parameter causes the JVM to load the "Java agent" that JMockit uses internally for bytecode
instrumentation.
You may have to use it even on a newer JDK 1.6+, if its Attach API implementation is not supported
by JMockit: such is the case with the IBM J9 JDK 1.6, the Mac OS X JDKs, and with JDKs for the Solaris OS.
jmockit.jar
appears before the JUnit jar file in the classpath (this isn't strictly
required, but it will provide the greatest ease of use with JUnit and JDK 1.6+).
jmockit.jar
file.
API documentation in HTML is also available in the jmockit/www/javadoc
directory, as well as
on-line.
For instructions on the use of JMockit from an Ant build script, see the relevant section in the JMockit Tutorial. For Maven, see this section.
The JMockit mocking APIs can be used in tests written with any of the popular testing frameworks:
JUnit 3.8, JUnit 4 (version 4.5 or newer) and TestNG (version 5.14 or newer).
Lets now see how mocking is done with the available JMockit APIs, which are all imported from the
mockit
package.
In a test class, simply declare a mock field of the desired type, annotating it with an appropriate JMockit
annotation such as @Mocked
or @Injectable
.
When the mocked type is a class, applying the latter annotation means that only the instance assigned to the
mock field will have mock behavior; if it is not applied, all instances of the mocked class will be mocked
by default.
import org.junit.*;
import mockit.*;
public class MyFirstJMockitTest
{
// Mocked instances (rather than conventional "mock objects") will be
// automatically created and assigned to annotated mock fields:
@Mocked Collaborator mock1; // all current and future instances will be mocked
@Injectable AnotherDependency anotherMock; // only one instance is a mock
@Test
public void myFirstTestMethod()
{
// Any mock field can be used here or in any other test method of the class.
// They should not be reassigned, however.
}
@Test // a method parameter is by default a "mock parameter" ("@Mocked" is implied)
public void anotherTestMethod(YetAnotherDependency testSpecificMock)
{
...
}
...
}
The template test class above shows something unusual: the second test method actually declares a parameter!
Normally, JUnit (and TestNG) test methods are not allowed to have parameters.
When using JMockit, however, such mock parameters are allowed and quite useful.
In fact, it's best to use mock fields of the test class only when the mocked types are needed by most or all tests in
the class. Otherwise, mock parameters (or local mock fields, which we introduce later) with scope limited to
a single test are preferred.
JMockit will always take care of instantiating the mocked type and either assigning the instance to the mock field
(provided the field is not final
) or passing it as an argument when the test method is invoked by the
test runner.
Although optional, mock parameters can also be annotated with @Mocked
and/or
@Injectable
. (The @Mocked
annotation is always
implied, even when @Injectable
or some other annotation is used.)
Instance fields of the test class which are not annotated, on the other hand, will not be mocked.
Therefore, the use of an annotation for mock fields of the test class is required.
The following template shows the basic structure of a JMockit test method, when using the Expectations & Verifications API for behavior-based testing:
@Test
public void aTestMethod(<any number of mock parameters>)
{
// Record phase: expectations on mocks are recorded; empty if there is nothing to record.
// Replay phase: invocations on mocks are "replayed"; here the code under test is exercised.
// Verify phase: expectations on mocks are verified; empty if there is nothing to verify.
}
Each test method can be divided in three execution phases. First, one or more invocations on mocked types/instances are recorded. (In fact, it is perfectly valid to not record any invocation at all - more on this later.) Second, the production code/class/unit which our test method is supposed to test is exercised, typically with a call to a single tested method. Any invocations to mocked methods/constructors that were previously recorded will now have a chance to be replayed. Third, the test method can explicitly verify that specific invocations to mocked methods/constructors actually happened (or not) during replay. Again, it is perfectly valid for this phase to be empty, with no explicit verifications (but with implicit, automatic, verifications - otherwise no behavior would be tested). Note that we say "mocked methods/constructors". As we will see later, the JMockit mocking API deals with methods (of all kinds) and constructors in the exact same way: you simply invoke them!
In the record phase of a test, we write one or more expectation blocks, inside which invocations to mocked types/instances are recorded.
@Test
public void aTestMethod()
{
new NonStrictExpectations()
{
// A "local" mock field (could also be annotated with @Mocked):
MyCollaborator mock;
{
mock.getData(); result = "my test data";
mock.doSomething(anyInt, "some expected value", anyString); times = 1;
}
};
// In the replay phase, the tested method would somehow call the "getData" and "doSomething"
// methods on a "MyCollaborator" instance.
...
// In the verify phase, we may optionally verify expected invocations to "MyCollaborator"
// objects.
...
}
There are two classes that can be used to define "expectation blocks": Expectations
and NonStrictExpectations
.
The difference is only in the default behavior for the recorded expectations.
In the first case, all expectations associated to mocked types used in the block will be strict by default,
meaning that the invocations are expected to occur in the replay phase in the same order as they were
recorded, and that invocations not recorded are expected to not occur.
In the second case, expectations are always non-strict, meaning that the invocations to mocked types,
whether recorded or not, are allowed to occur in any order and in any number, including not at all.
There is also a @NonStrict
annotation which can be applied to mock fields/parameters,
to force only non-strict expectations on them, even inside an otherwise strict
Expectations
block.
You may be wondering what are those field assignments in the example test above.
This is indeed something that you won't see in any other mocking API. That said, it should feel very intuitive once
the semantics are known:
the result
field takes the desired return value for the preceding invocation, while
times
takes the number of times the preceding invocation is allowed and expected to occur.
(There is also a minTimes
and a maxTimes
field.)
The result
field also accepts a Throwable
instance, which
would cause the corresponding invocation in the replay phase to throw the specified exception or error.
The smart reader probably already figured out why and when we would have a non-empty verification phase in a test. It boils down to the existence or not of non-strict expectations in the test. If all mocked types used in the test have only strict expectations, then all invocations will be implicitly and automatically verified at the end of the test. In this case, there will be no explicit verify phase. If there is at least one mocked type with non-strict expectations, then we can use one or more verification blocks to make sure that the invocations we care about actually occurred during the replay phase.
@Test
public void aTestMethod(@Injectable final MyCollaborator mock)
{
// Non-strict expectations are recorded, or perhaps there are no expectation blocks at all.
// Code under test is exercised:
...
new Verifications()
{{
// If no new instance of the mocked class should have been created with the
// no-args constructor, we can verify it:
new MyCollaborator(); times = 0;
// Here we verify that the "MyCollaborator#doSomething()" method was executed at least once:
mock.doSomething();
// Another verification, which must have occurred no more than three times:
mock.someOtherMethod(anyBoolean, any, withInstanceOf(Xyz.class)); maxTimes = 3;
}};
}
The Verifications API is quite rich and flexible. Besides the Verifications
class
shown above, which allows some invocations to be verified regardless of their order of execution, we have the
VerificationsInOrder
, FullVerifications
, and
FullVerificationsInOrder
subclasses. The "InOrder" suffix simply means that the relative
order of the invocations appearing inside the verification block will have to match the actual order of execution of
corresponding invocations during replay.
The "Full" prefix implies that all invocations that occurred during replay must be accounted for (verified) inside
the verification block (excluding those which are verified implicitly, if any).
Sometimes you are more interested in checking the argument values passed to a mocked collaborator, instead of
checking that the invocation to the collaborator was made from the client/tested method.
Of course, you can check argument values through a recorded or verified expectation, but it becomes more
difficult when those values are complex.
In such cases it may be easier to use the mockit.MockUp<T>
generic class, as the next
test shows.
@Test
public void stateBasedTestMethod()
{
new MockUp<MyCollaborator>()
{
@Mock(invocations = 1) // (the invocation count constraint is optional)
boolean doSomething(int n, String s, ComplexData otherData)
{
assertEquals(1, n);
assertNotNull(otherData);
...
// Return (if non-void) or throw the result we want to produce for the
// invocation to the mocked method:
return otherData.isValid();
}
// Other mock or regular methods...
};
// Exercise the code under test normally; calls to MyCollaborator#doSomething will execute
// the mock method above; if more or less than one such invocation occurs, the test fails.
...
}
The @Mock
annotation marks those methods in the mock-up class which are
supposed to provide mock/stub implementations for the corresponding methods (of the same signature) in the mocked
class.
If no invocation count constraint is specified through attributes of the @Mock
annotation, then any number (including zero) of invocations will be allowed.
The examples above give only a brief overview of the available mocking APIs. There is more.
The full JMockit Tutorial contains an extensive discussion of nearly all methods, fields,
annotations, etc., with many examples.
The API documentation provides a detailed specification for all elements
of the mocking APIs.
Finally, there are hundreds of JUnit sample tests available under the
jmockit/samples
folder, which use nearly everything that is available in the APIs.


Map Playing
Select Maps
Options
All Maps
Favorite Maps
Notifications
Receive notifications when your favorite maps start.
Made with
by
RastaLulz
and reworked by
unixfox
.


class: center, middle
Pure Functional Database Programming
… with Fixpoint Types
Rob Norris • `@tpolecat` • Gemini Observatory 12 Sep 2016 • The Jon Pretty Experience ??? - I'm Rob, thanks Jon for inviting me to speak - I write software for big telescopes (in Scala) and I also work on some open-source projects including a few that we'll talk about today. - So before we get started, if you don't mind admitting it, how many of you have used scalaz or cats? - So today I want to talk about some odds and ends that I think are interesting, and we'll see if you find them interesting too. --- # Goals 1. Gain an intuition about some simple **recursive types**. 2. Learn a little bit about **doobie**, a pure functional JDBC layer for Scala. 3. Gain some confidence pushing on code to see what happens. 4. Do not panic. ??? 1. --- These types seem to be pretty well known in the Haskell community but not so much among Scala programmers, and they're interesting and worth knowing about. 2. --- 3. --- so, this is something I mention from time to time when I give talks and people ask me to say more about it, so I'll try to do that as well. 4. --- Recursion is always mind-blowing at first. Everybody struggles with recursive functions and inductive data types at first, and today we're going to talk about self-referential types, which might be hard to grasp at first. This is ok. I'm going to try to approach it very gradually. If it ends up being too slow for you, I apologize; if it's too fast I also apologize. But please stop me if you have questions. Ok so let's look at a recursive data type. --- # Recursive Data A recursive data type for professors and their Ph.D. students. ```scala case class Prof( name: String, year: Int, students: List[Prof] ) ```
??? - ... - So the kind of invented problem I want to talk about is how to store this in a relational database. So how do I save one of these tree structures to a database and read it back? --- # Recursive Data A recursive data type for professors and their Ph.D. students. ```scala case class Prof( name: String, year: Int, students: List[Prof] ) ``` Natural SQL representation:
```sql CREATE TABLE prof ( id INTEGER IDENTITY, parent INTEGER NULL, name VARCHAR NOT NULL, year INTEGER NOT NULL, FOREIGN KEY(parent) REFERENCES prof(id) ) ``` ??? - Here's how you normally want to do it. - ... - You have the child refer to its parent, so the relationship is inverted. So we need to figure out how to manage that difference. - And you also have this id to deal with, and we need a way to represent the id on the Scala side, so let's talk about that first. - Ok so here's one possibility. --- # Recursive Data A recursive data type for professors and their Ph.D. students. ```scala case class Prof( id : Int, name: String, year: Int, students: List[Prof] ) ``` Natural SQL representation.
```sql CREATE TABLE prof ( id INTEGER IDENTITY, parent INTEGER NULL, name VARCHAR NOT NULL, year INTEGER NOT NULL, FOREIGN KEY(parent) REFERENCES prof(id) ) ``` ??? - ... - Ok so what's wrong with this representatoin for IDs? (new objects) - Also going to get rid of the table definition for now. --- # Recursive Data A recursive data type for professors and their Ph.D. students. ```scala case class Prof( id : Option[Int], name: String, year: Int, students: List[Prof] ) ```
??? - Ok is this any better? - ... - Maybe a little bit, but I think it's irritating because you tend to have three cases: 1. it's new, and we know it's new 2. it's from the database so we know it has an id 3. we're doing some kind of computation that's only looking at the structure and we don't care about database id And this encoding is irritating in all three cases, so I don't like it. So a way to make the id less invasive is to externalize it. --- # Recursive Data A recursive data type for professors and their Ph.D. students. ```scala case class Prof( name: String, year: Int, students: List[Prof] ) type IdProf = (Int, Prof) ```
??? - so ... - but the problem now is that the children don't have ids, right? So we have to do this: --- # Recursive Data A recursive data type for professors and their Ph.D. students. ```scala case class Prof( name: String, year: Int, students: List[(Int, Prof)] ) type IdProf = (Int, Prof) ```
??? - ... Which defeats the purpose of externalizing the ids … the whole point was to not have to talk about them in the model. - Ok so when you have a problem like this where you can't figure out what type to use, sometimes it helps to just make it a parameter. So let's try that. --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala type Prof = ProfF[ProfF] type IdProf = (Int, ProfF[(Int, ProfF)]) ```
??? - ok so the data type isn't recursive anymore - And I'm going to call this `ProfF` just to distinguish it from our old `Prof`, so now we can just say ... - right? no, because ProfF has a type parameter --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala type Prof = ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ProfF[ type IdProf = (Int, ProfF[(Int, ProfF[(Int, ProfF[(Int, ProfF[(Int, ProfF[(Int, ProfF[(Int, ProfF[(Int, ProfF[(Int, ```
??? - so this doesn't work because the types are infinite --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala type Prof = ProfF[Prof] type IdProf = (Int, ProfF[IdProf]) ```
??? - what if we make the type aliases self-referential? - ok? no. doesn't compile - type aliases can't be recursive. but **classes** can ... --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala case class Prof(value: ProfF[Prof]) case class IdProf(id: Int, value: ProfF[IdProf]) ``` ??? - So what if we make these case classes? Does this work? - So this is cool, we have one representation for our data type, `ProfF`, and depending on how we wrap it we can represent just the pure data *or* the data annotated with database ids. - We could claim victory here because this actually does what we want. - But this is functional programming and we like abstraction, so let's push on this a little it and see if we can generalize it a little bit. - Ok, so there's nothing here's that's specific to ProfF, so let's factor that out. --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala case class Prof[F[_]](value: F[Prof]) case class IdProf[F[_]](id: Int, value: F[IdProf]) ``` ??? - But this doesn't compile because `Prof` needs a type argument, which is ... `F` right? --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala case class Prof[F[_]](value: F[Prof[F]]) case class IdProf[F[_]](id: Int, value: F[IdProf[F]]) ``` ??? - We also note that this doesn't depend on the id being an `Int` so let's factor that out too. --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala case class Prof[F[_]](value: F[Prof[F]]) case class IdProf[F[_], A](id: A, value: F[IdProf[F, A]]) ``` ??? - and we had to add the `A` to the type argument here since `IdProf` now has two type parameters. - Ok, so what do we have now? We have two recursive types that are totally generic; the first one can make any `F` recursive, and the second can make any `F` recursive *and* annotate each value with some arbitrary label. - So we have invented something here that's totally generic and potentially really useful, and when this happens I always become suspicious. --- class: center, middle ### tpolecat's law of mistaken invention: ## _"Anything I invent that is sufficiently polymorphic and plausibly useful probably already exists in scalaz."_ ??? - so when this happens take a step back and see if it's already a thing. Because honestly it's usually already a thing. - It turns out that both of these types are well known. --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala case class Fix[F[_]](unfix: F[Fix[F]]) case class Cofree[F[_], A](head: A, tail: F[Cofree[F, A]]) ``` ??? - the first one is called `Fix` and it gives us this fixpoint for any `F`. It lets us write down the infinite type in a finite way. - the second one is called `Cofree` and it does the same thing but associates an arbitrary annotation with each value. This one exists in scalaz and cats. `Fix` doesn't, I don't know. - ... and while we're at it I'll mention another data type that's related. --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala case class Fix[F[_]](unfix: F[Fix[F]]) case class Cofree[F[_], A](head: A, tail: F[Cofree[F, A]]) case class Hmmm[F[_], A](hmmmmmm: A \/ F[Hmmm[F, A]]) ``` ??? - This is just like `Cofree` but instead of an `A` and an `F` it has an `A` *or* an `F`. - Does anyone know what we call this one? --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala case class Fix[F[_]](unfix: F[Fix[F]]) case class Cofree[F[_], A](head: A, tail: F[Cofree[F, A]]) case class Free[F[_], A](resume: A \/ F[Free[F, A]]) ``` ??? - Right, so this is `Free`. In the same way you think of cofree as a structure with labels at each node, free is a structure with labels at the leaves. - `Free` is commonly called the free *monad* because it's a monad if `F` is a functor, add `Cofree` is a *comonad* if `F` is a functor. - It's worth noting that the implementations in cats and scalaz are more complicated than what we have here, because to strictness and stack safety; you want to be able to define free/cofree structures that are very large (or even infinite) so they need to be lazt. And then because they can be arbitrarily big you need a way to traverse them without running out of stack space. So that's the source of complication. In Haskell they're defined just like this. They're both one-liners. Like, half a line. - In any case it's unfortunate because the complexity of the implementation makes them hard to understanc. You have to squint to see it, especially with `Free`. - It's a fun and useful exercise to take that totally trivial free monad (for instance) and build it out util you can write a little DSL and interpreter like everyone likes to do these days. It will be finite and stack limited but it'll work. And you will feel awesome. - Ok, so I want to do a couple more things before we move on. - So the first is that if we have a `Fix` we can always get an `F` out by calling `.unfix`, and we can do that with `Cofree` by calling `.tail`. The general name for this operation is `project` and data types that do this are called *recursive* types. - And note that we can't do it with `Free` because you might not have an `F`; you might have an `A`. But you can always go the *other* way. If you have the `F` you can construct a `Free`. Same with `Fix`. But you can't do this with `Cofree` because you also need an `A`. So this operation (the dual) is called `embed` and data types with ths behavior are *corecursive* types. - So `Cofree` is a recursive type and `Free` is a corecursive type, and `Fix` is both. - And on this basis you can introduce abstractions for recursive and corecursive types that give you tons and tons of operations for free. There's a library called Matryoskha that does this, and it's an amazing library. ... - So some of the things I'm going to derive later in the talk are free if you use Matryoshka. - So, the second thing I want to point out is that `Free` and `Cofree` are also fixpoint types. We can do the same trick we did with `Prof` and factor out the recursion. --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala case class Fix[F[_]](unfix: F[Fix[F]]) case class Cofree[F[_], A](head: A, tail: F[Cofree[F, A]]) case class Free[F[_], A](resume: A \/ F[Free[F, A]]) ``` ```scala case class CofreeF[F[_], A, B](head: A, tail: F[B]) case class FreeF[F[_], A, B](resume: A \/ F[B]) type Cofree[F[_], A] = Fix[CofreeF[F, A, ?]] type Free[F[_], A] = Fix[FreeF[F, A, ?]] ``` ??? - So `Fix` is the primitive structure here. - Alright, enough of that. - Ok so now that we know about the general types we can define `Prof` and `IdProf` trivially. --- # Recursive Data A ~~recursive~~ data type for professors and their Ph.D. students. ```scala case class ProfF[A]( name: String, year: Int, students: List[A] ) ``` ```scala case class Fix[F[_]](unfix: F[Fix[F]]) case class Cofree[F[_], A](head: A, tail: F[Cofree[F, A]]) ``` Annotated and un-annotated `ProfF` trees. ```scala type Prof = Fix[ProfF] type IdProf = Cofree[ProfF, Int] ``` ??? - So by this point I fear I may have lost some people, so just to get back to reality I want to show you how you can construct these things. --- # Recursive Data ```scala case class Prof(name: String, year: Int, students: List[Prof]) val p: Prof = Prof("Hilbert", 1885, List( Prof("Ackermann", 1925, Nil), Prof("Curry", 1930, Nil), Prof("Weyl", 1908, List( Prof("Mac Lane", 1934, List( Prof("Howard", 1956, Nil), Prof("Awodey", 1997, Nil) )) )) )) ```
??? - Here is our original recursive data type and a construction. Nothing fancy here. --- # Recursive Data ```scala case class ProfF[A](name: String, year: Int, students: List[A]) type Prof = Fix[ProfF] val p: Prof = Fix(ProfF("Hilbert", 1885, List( Fix(ProfF("Ackermann", 1925, Nil)), Fix(ProfF("Curry", 1930,Nil)), Fix(ProfF("Weyl", 1908, List( Fix(ProfF("Mac Lane", 1934, List( Fix(ProfF("Howard", 1956, Nil)), Fix(ProfF("Awodey", 1997, Nil)) )))) ))) )) ```
??? - And here's the version with `Fix`. It's identical in structure but there's an extra `Fix` constructor wrapped around the `ProfF` constructor. --- # Recursive Data ```scala case class ProfF[A](name: String, year: Int, students: List[A]) type IdProf = Cofree[ProfF, Int] val p: IdProf = Cofree(1, ProfF("Hilbert", 1885, List( Cofree(2, ProfF("Ackermann", 1925, Nil)), Cofree(3, ProfF("Curry", 1930,Nil)), Cofree(4, ProfF("Weyl", 1908, List( Cofree(5, ProfF("Mac Lane", 1934, List( Cofree(6, ProfF("Howard", 1956, Nil)), Cofree(7, ProfF("Awodey", 1997, Nil)) )))) ))) )) ```
??? - And here's the `Cofree` version, which also has an annotation (an `Int` in this case) associated with each node. - So even though the types here are a little challenging at first and the names are kind of intimidating, but it's just data. You can construct them, you can pattern match on them, and so on. Not scary. -
--- class: center, middle # Mezzanine ??? - Ok, so before we get back to our database problem I want to take a detour and show you an example of programming with these types in a totally pure context, without worrying about IO effects. - So we're going to write some parsers. --- # Programming with Fix and Cofree ```scala case class ProfF[A](name: String, year: Int, students: List[A]) val data = """|Simeon Denis Poisson, 1800 | Gustav Peter Lejeune Dirichlet, 1827 | Rudolf Otto Sigismund Lipschitz, 1853 | C. Felix (Christian) Klein, 1868 | William Edward Story, 1875 | Solomon Lefschetz, 1911 | Albert William Tucker, 1932 | Marvin Lee Minsky, 1954 | Gerald Jay Sussman, 1973 | Guy Lewis Steele, 1980 | Philip Lee Wadler, 1984 | C. L. Ferdinand (Carl Louis) Lindemann, 1873 | David Hilbert, 1885 | Wilhelm Ackermann, 1925 | Haskell Curry, 1930 | Hermann Weyl, 1908 | Saunders Mac Lane, 1934 | Steven Awodey, 1997 | William Howard, 1956 |""".stripMargin ``` ??? So, here's some sample data that we want to parse into our fixpoint types. You may recognize some of these names. --- # Programming with Fix and Cofree ```scala def prof(n: Int): Parser[Fix[ProfF]] = for { _ <- manyN(n, char(' ')) // skip 'n' spaces name <- stringOf(notChar(',')) // read up to a comma _ <- token(char(',')) // skip comma and following space year <- int // read an int _ <- char('\n') // skip a newline ss <- many(prof(n + 2)) // recursively read more at indent + 2 } yield Fix(ProfF(name, year, ss)) ``` ??? - And here's one way to do it. This is written with `atto` which is a parsing library I work on but it would look basically the same with Fastparse or pretty much any monadic parser combinator library. --- # Programming with Fix and Cofree ```scala def prof(n: Int): Parser[Fix[ProfF]] = for { _ <- manyN(n, char(' ')) // skip 'n' spaces name <- stringOf(notChar(',')) // read up to a comma _ <- token(char(',')) // skip comma and following space year <- int // read an int _ <- char('\n') // skip a newline ss <- many(prof(n + 2)) // recursively read more at indent + 2 } yield Fix(ProfF(name, year, ss)) ``` ```scala scala> prof(0).parseOnly(data).option res8: Option[Fix[ProfF]] = Some(Fix(ProfF(Simeon Denis Poisson, 1800, «1»))) ``` ??? - and it works! - but the `toString` for `Fix` isn't that great so I'll call a utility method that prints out the tree. I'm not going to show it to you but it's in the example code. --- # Programming with Fix and Cofree ```scala scala> prof(0).parseOnly(data).option.foreach(drawFix) ProfF(Simeon Denis Poisson, 1800, «1») ProfF(Gustav Peter Lejeune Dirichlet, 1827, «1») ProfF(Rudolf Otto Sigismund Lipschitz, 1853, «1») ProfF(C. Felix (Christian) Klein, 1868, «2») ProfF(William Edward Story, 1875, «1») ProfF(Solomon Lefschetz, 1911, «1») ProfF(Albert William Tucker, 1932, «1») ProfF(Marvin Lee Minsky, 1954, «1») ProfF(Gerald Jay Sussman, 1973, «1») ProfF(Guy Lewis Steele, 1980, «1») ProfF(Philip Lee Wadler, 1984, «0») ProfF(C. L. Ferdinand (Carl Louis) Lindemann, 1873, «1») ProfF(David Hilbert, 1885, «3») ProfF(Wilhelm Ackermann, 1925, «0») ProfF(Haskell Curry, 1930, «0») ProfF(Hermann Weyl, 1908, «1») ProfF(Saunders Mac Lane, 1934, «2») ProfF(Steven Awodey, 1997, «0») ProfF(William Howard, 1956, «0») ``` ??? - right. that looks good. - now let's see what we can do with `Cofree`. --- # Programming with Fix and Cofree ```scala def prof(n: Int): Parser[Fix[ProfF]] = for { _ <- manyN(n, char(' ')) // skip 'n' spaces name <- stringOf(notChar(',')) // read up to a comma _ <- token(char(',')) // skip comma and following space year <- int // read an int _ <- char('\n') // skip a newline ss <- many(prof(n + 2)) // recursively read more at indent + 2 } yield Fix(ProfF(name, year, ss)) ``` ??? - Here's our `Fix` parser ... --- # Programming with Fix and Cofree ```scala def posProf(n: Int): Parser[Cofree[ProfF, (Int, Int)]] = for { p0 <- pos _ <- manyN(n, char(' ')) // skip 'n' spaces name <- stringOf(notChar(',')) // read up to a comma _ <- token(char(',')) // skip comma and following space year <- int // read an int _ <- char('\n') // skip a newline ss <- many(posProf(n + 2)) // recursively read more at indent + 2 p1 <- pos } yield Cofree((p0, p1), ProfF(name, year, ss)) ``` ??? - And here's one with `Cofree`, and what we're doing here is annotating each `ProfF` with the starting and ending parse positions. --- # Programming with Fix and Cofree ```scala def posProf(n: Int): Parser[Cofree[ProfF, (Int, Int)]] = for { p0 <- pos _ <- manyN(n, char(' ')) // skip 'n' spaces name <- stringOf(notChar(',')) // read up to a comma _ <- token(char(',')) // skip comma and following space year <- int // read an int _ <- char('\n') // skip a newline ss <- many(posProf(n + 2)) // recursively read more at indent + 2 p1 <- pos } yield Cofree((p0, p1), ProfF(name, year, ss)) ``` ```scala scala> posProf(0).parseOnly(data).option res10: Option[Cofree[ProfF,(Int, Int)]] = Some(Cofree((0,713),ProfF(Simeon Denis Poisson, 1800, «1»))) ``` ???  - So our original idea for `Cofree` was for database IDs but we can use that slot for anything. Without any changes to our API we now have a representation with source positions. - And you can see how this can be really useful for something like a computer language, because your recursive data type is an AST that you might want to annotate with source positions, types, strictness analysis, any number of things, and by encoding your data types this way you get that ability for free, which is really nice. --- class: center, middle # Fistful of Functors ??? - Ok so before we get back to our database problem I need to explain something that you may have been wondering about. --- # So Why the F? ```scala case class ProfF[A](name: String, year: Int, students: List[A]) ``` `ProfF` is a functor. This means we can map over it. ```scala implicit val ProfFFunctor: Functor[ProfF] = new Functor[ProfF] { def map[A, B](fa: ProfF[A])(f: A => B): ProfF[B] = fa.copy(students = fa.students.map(f)) } ``` Thus. ```scala scala> val ps = ProfF("Bob", 1976, List("foo", "bar")) ps: ProfF[String] = ProfF(Bob,1976,List(foo, bar)) scala> ps.map(s => s.length) res12: ProfF[Int] = ProfF(Bob,1976,List(3, 3)) ``` ??? - ProfF is a functor. This means we can map over it. - And if you follow this design pattern your data type will *always* be a functor, and we call it a *pattern functor*. ProfF is the pattern functor for Prof and IdProf; they differ in the recursive structure that we're applying. - In any case the functor here is straightforward because we can just delegate to `.map` on `List` and we're done. --- # But wait, there's more! ```scala case class ProfF[A](name: String, year: Int, students: List[A]) ``` `ProfF` is a *traversable* functor. This means we can sequence it. ```scala implicit val ProfFTraverse: Traverse[ProfF] = new Traverse[ProfF] { def traverseImpl[G[_]: Applicative, A, B](fa: ProfF[A])(f: A => G[B]): G[ProfF[B]] = fa.students.traverse(f).map(ss => fa.copy(students = ss)) } ``` Thus. ```scala scala> val p = ProfF("Bob", 1975, List(Option(1), Option(2))) p: ProfF[Option[Int]] = ProfF(Bob,1975,List(Some(1), Some(2))) scala> p.sequence res13: Option[ProfF[Int]] = Some(ProfF(Bob,1975,List(1, 2))) ``` ??? - It turns out that `ProfF` is a **traversable** functor, which is even better. - This means we can `sequence` it, which lets us turn a `ProfF[G[A]]` into a `G[ProfF[A]]` if `G` is an applicative functor. We can flip the type constructors, which is what the example illustrates. Look at the types. - If you're not familiar with traversable functors take the time to learn about them; you really can't do monadic programming without them. - The implementation is a little more involved but again we just delegate to `traverse` on `List` for most of the work because `List` is also a traversable functor. - Ok so we defined `traverse` but the operation I showed you is called `sequence`, so what's the relationship. --- # But wait, there's more! Because "map and then sequence" is so common, we define `traverse` which does them in one step. You can define one in terms of the other but they have this relationship: ```scala fa.map(f).sequence <~> fa.traverse(f) // definition of traverse ``` And we can find another identity by equational reasoning. ```scala fa <~> fa.map(identity) // the functor law fa.sequence <~> fa.map(identity).sequence // sequence both sides fa.sequence <~> fa.traverse(identity) // definition of traverse ``` So when you're working with traversable functors it's nice to know both of those substitutions. ??? - and by `<~>` we mean "the same thing" ... you can freely substitute one for the other. - and given that we can find another identity - this is kind of a trivial example but again it's just that's not in your toolchest when you're first learning about functional programming and I always like to give examples when I can. - Ok. --- class: center, middle # Pure Functional Database Programming ??? - Let's talk about doobie. --- # Get your doobie on doobie is a **pure functional** JDBC layer for Scala - No side-effects. doobie programs are **values**. - Built natively for scalaz/scalaz-stream *and* cats/fs2. - It's a Typelevel project! - I have stickers!
??? - I had ten slides about doobie, but I decided that's not enough if you're not familiar with free monads and monadic programming, and too much if you do; so I'm just going to give you the executive summary and I'll link you to a pretty good video at the end of the talk. - In the end I don't think it's going to matter. If I introduce a type you don't understand please stop me and I'll explain it. --- # Get your doobie on Ingredients, the TL;DR version: - Algebras of all operations for all JDBC types. - Free monads thereof; 16 little languages with 1000 operations. - Lifting operation for composing programs written in different languages. - Interpreters into Kleisli arrows for IO-like data types. ??? - Primitives - Composition - Embedding - Semantics Still relatively low-level; this is just a pure functional way to write raw JDBC, so we improve on that with some higher-level constructors. --- # Get your doobie on Ingredients, the TL;DR version: - Algebras of all operations for all JDBC types. - Free monads thereof; 16 little languages with 1000 operations. - Lifting operation for composing programs written in different languages. - Interpreters into Kleisli arrows for IO-like data types. Derived constructors: - Typesafe versions of JDBC operations that use `Int` constants. - Resource-safe constructors for embedded sub-programs. - Composite column vector operations via typeclass-based mapping. - Compile-time checked SQL literals. (New!) ??? These are all built on top of the low-level stuff; there's no ugly plumbing layer and no new types. You can mix high and low-level code all you like. --- class: center, middle # Alrighty. --- # Insert a Tree Recall our `ProfF` type. ```scala case class ProfF[A](name: String, year: Int, students: List[A]) ``` Construct a program to insert a `ProfF`, ignoring children, returning its generated id. ```scala def insertNode(op: Option[Int], p: ProfF[_]): ConnectionIO[Int] = tsql""" INSERT INTO prof (parent, name, year) VALUES ($op, ${p.name}, ${p.year}) RETURNING id """.unique[Int] ``` ??? - Ok so here's a method that constructs a program that inserts a row into the `prof` table and yields the generated id. - (explain) - It disregards the childern entirely, which you know because of the exstential placeholder; the implementation can't do anything with the children because it doesn't even know what type they are. - I'm going to get rid of the `ProfF` definition to make some room. --- # Insert a Tree Construct a program to insert a `ProfF`, ignoring children, returning its generated id. ```scala def insertNode(op: Option[Int], p: ProfF[_]): ConnectionIO[Int] = tsql""" INSERT INTO prof (parent, name, year) VALUES ($op, ${p.name}, ${p.year}) RETURNING id """.unique[Int] ``` ??? --- # Insert a Tree Construct a program to insert a `ProfF`, ignoring children, returning its generated id. ```scala def insertNode(op: Option[Int], p: ProfF[_]): ConnectionIO[Int] = tsql""" INSERT INTO prof (parent, name, year) VALUES ($op, ${p.name}, ${p.year}) RETURNING id """.unique[Int] ``` Construct a program to insert a `Fix[ProfF]` and return an equivalent `Cofree` annotated with generated ids. ```scala def insertTree(fp: Fix[ProfF], op: Option[Int]): ConnectionIO[Cofree[ProfF, Int]] = for { h <- insertNode(op, fp.unfix) // ConnectionIO[Int] t <- fp.unfix // ProfF[Fix[ProfF]] .map(insertTree(_, Some(h))) // ProfF[ConnectionIO[Cofree[ProfF, Int]]] .sequence // ConnectionIO[ProfF[Cofree[ProfF, Int]]] } yield Cofree(h, t) ``` ??? - And here's a metohod that constructs a program to insert a `Fix[ProfF]` into the database, and yield another tree with the same structure but annotated with database ids. Which is what we wanted to do. - So how does this work? - (explain) - ok so we know that `.map` and then `.sequence` is `.traverse` so let's simplify that --- # Insert a Tree Construct a program to insert a `ProfF`, ignoring children, returning its generated id. ```scala def insertNode(op: Option[Int], p: ProfF[_]): ConnectionIO[Int] = tsql""" INSERT INTO prof (parent, name, year) VALUES ($op, ${p.name}, ${p.year}) RETURNING id """.unique[Int] ``` Construct a program to insert a `Fix[ProfF]` and return an equivalent `Cofree` annotated with generated ids. ```scala def insertTree(fp: Fix[ProfF], op: Option[Int]): ConnectionIO[Cofree[ProfF, Int]] = for { h <- insertNode(op, fp.unfix) t <- fp.unfix .traverse(insertTree(_, Some(h))) // same as map + sequence } yield Cofree(h, t) ``` ??? - (next slide) --- # Insert a Tree Construct a program to insert a `ProfF`, ignoring children, returning its generated id. ```scala def insertNode(op: Option[Int], p: ProfF[_]): ConnectionIO[Int] = tsql""" INSERT INTO prof (parent, name, year) VALUES ($op, ${p.name}, ${p.year}) RETURNING id """.unique[Int] ``` Construct a program to insert a `Fix[ProfF]` and return an equivalent `Cofree` annotated with generated ids. ```scala def insertTree(fp: Fix[ProfF], op: Option[Int]): ConnectionIO[Cofree[ProfF, Int]] = for { h <- insertNode(op, fp.unfix) t <- fp.unfix.traverse(insertTree(_, Some(h))) } yield Cofree(h, t) ``` ??? - So, this `insertTree` operation can be generalized but the `Option` type makes it a little messy so we're just going to leave it as-is. --- # Insert a Tree ```scala scala> val p = prof(0).parseOnly(data).option.get // yolo p: Fix[ProfF] = Fix(ProfF(Simeon Denis Poisson, 1800, «1»)) scala> val io = insertTree(p, None).transact(xa).flatMap(c => ioDrawCofree(c)) io: scalaz.effect.IO[Unit] = scalaz.effect.IO$$anon$6@6709fe88 scala> io.unsafePerformIO 1091 :< ProfF(Simeon Denis Poisson, 1800, «1») 1092 :< ProfF(Gustav Peter Lejeune Dirichlet, 1827, «1») 1093 :< ProfF(Rudolf Otto Sigismund Lipschitz, 1853, «1») 1094 :< ProfF(C. Felix (Christian) Klein, 1868, «2») 1095 :< ProfF(William Edward Story, 1875, «1») 1096 :< ProfF(Solomon Lefschetz, 1911, «1») 1097 :< ProfF(Albert William Tucker, 1932, «1») 1098 :< ProfF(Marvin Lee Minsky, 1954, «1») 1099 :< ProfF(Gerald Jay Sussman, 1973, «1») 1100 :< ProfF(Guy Lewis Steele, 1980, «1») 1101 :< ProfF(Philip Lee Wadler, 1984, «0») 1102 :< ProfF(C. L. Ferdinand (Carl Louis) Lindemann, 1873, «1») 1103 :< ProfF(David Hilbert, 1885, «3») 1104 :< ProfF(Wilhelm Ackermann, 1925, «0») 1105 :< ProfF(Haskell Curry, 1930, «0») 1106 :< ProfF(Hermann Weyl, 1908, «1») 1107 :< ProfF(Saunders Mac Lane, 1934, «2») 1108 :< ProfF(Steven Awodey, 1997, «0») 1109 :< ProfF(William Howard, 1956, «0») ``` ??? - so first we parse the value like we did before - then we construct a program to insert it, passing the result to `ioDrawCofree` which is the same printing code, just in `IO` so it doesn't side-effect. - Then we run the program and see that we get back an annotated tree. - I should mention that this whole talk is one long REPL session that has been typechecked and executed, I just haven't been showing output. I use a tool called **tut** to do this. - Ok, so now how do we read trees back? --- # And now read it back. Construct a program to read a `ProfF` whose students are `Int` ... ? ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` ??? - ok so what does this mean? one level and stop - I should acknowledge that this is a very inefficient way to do this; it would be much faster to read the whole thing into memory and assemble the tree locally, but that pushes all the complexity into the SQL which isn't very interesting. But there is a fast version in the sample code. - what if we want two levels? --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Read two layers deep, then stop at ids. ```scala def readNode2(id: Int): ConnectionIO[ProfF[ProfF[Int]]] = readNode(id).flatMap(_.traverse(readNode)) ``` --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Read two layers deep, then stop at ids. ```scala def readNode2(id: Int): ConnectionIO[ProfF[ProfF[Int]]] = readNode(id).flatMap(_.traverse(readNode)) ``` Three, why not? ```scala def readNode3(id: Int): ConnectionIO[ProfF[ProfF[ProfF[Int]]]] = readNode(id).flatMap(_.traverse(readNode2)) ``` --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Read three layers and then stop at a *program* to read the next three layers. ```scala type Three[A] = ConnectionIO[ProfF[ProfF[ProfF[A]]]] def readK(id: Int): Fix[Three] = Fix[Three](readNode3(id).map(_.map(_.map(_.map(readK))))) // urk! ``` ??? - We can write a typeclass that's indexed by a natural number representing the depth, so we're not fixed at two or three or whatever, and that's in the sample code as well. - But what we really want to do here is read the whole thing, so let's get back to that. --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Construct a program to read a `ProfF` tree *annotated* with database ids. ```scala def readTree(id: Int): ConnectionIO[Cofree[ProfF, Int]] = readNode(id).flatMap { pi => pi.map(readTree) // ProfF[ConnectionIO[Cofree[ProfF, Int]]] .sequence // ConnectionIO[ProfF[Cofree[ProfF, Int]]] .map(Cofree(id, _)) // ConnectionIO[Cofree[ProfF, Int]] } ``` ??? - here we go. (explain) - as before we know that `.map` and then `.sequence` is ... --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Construct a program to read a `ProfF` tree *annotated* with database ids. ```scala def readTree(id: Int): ConnectionIO[Cofree[ProfF, Int]] = readNode(id).flatMap { pi => pi.traverse(readTree) .map(Cofree(id, _)) } ``` ??? - (next) --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Construct a program to read a `ProfF` tree *annotated* with database ids. ```scala def readTree(id: Int): ConnectionIO[Cofree[ProfF, Int]] = readNode(id).flatMap(_.traverse(readTree).map(Cofree(id, _))) ``` ??? - Ok so I'd like to take a minute to look at the `readTree` method more closely and see if we can generalize it. - Let's factor out the call to `readNode`. --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Construct a program to read a `ProfF` tree *annotated* with database ids. ```scala def readTree(id: Int)(f: Int => ConnectionIO[ProfF[Int]]): ConnectionIO[Cofree[ProfF, Int]] = f(id).flatMap(_.traverse(readTree(_)(f)).map(Cofree(id, _))) ``` ??? - Ok, so `f` gives us a `ConnectionIO` but all we're doing with it is calling `.flatMap` and then `.map`, so it should work with any monad. Let's factor that out. --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Construct a program to read a `ProfF` tree *annotated* with database ids. ```scala def readTree[M[_]: Monad](id: Int)(f: Int => M[ProfF[Int]]): M[Cofree[ProfF, Int]] = f(id).flatMap(_.traverse(readTree(_)(f)).map(Cofree(id, _))) ``` ??? - So now we get an that `M` of `ProfF[Int]` but the only thing we're doing with the `ProfF` it is calling `.traverse`, so it should work with any traversable functor. Let's factor that out. --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Construct a program to read a `ProfF` tree *annotated* with database ids. ```scala def readTree[M[_]: Monad, F[_]: Traverse](id: Int)(f: Int => M[F[Int]]): M[Cofree[F, Int]] = f(id).flatMap(_.traverse(readTree(_)(f)).map(Cofree(id, _))) ``` ??? - And we're not doing anything with the `Int` but passing it as an argument so let's factor that out too. --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Construct a program to read a `ProfF` tree *annotated* with database ids. ```scala def readTree[M[_]: Monad, F[_]: Traverse, A](a: A)(f: A => M[F[A]]): M[Cofree[F, A]] = f(a).flatMap(_.traverse(readTree(_)(f)).map(Cofree(a, _))) ``` ??? - ok, so now we have something definitely useful and totally generic. --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Construct a program to read a `ProfF` tree *annotated* with database ids. ```scala def unfoldCM[M[_]: Monad, F[_]: Traverse, A](a: A)(f: A => M[F[A]]): M[Cofree[F, A]] = f(a).flatMap(_.traverse(unfoldCM(_)(f)).map(Cofree(a, _))) ``` ??? - This is monadic cofree corecursion; it takes a seed value and feeds it into a function that generates an F in some context, the traverses it to expand the `A`'s in the `F`, forever, or until there are no more `A`s; in this case it means we're at the leaves of the tree. - And what it gives back is a Cofree annotated with the key values. - And does this method exist in scalaz already? NO! The non-monadic version does but not this one. It's an oversight. - and I want to point out it's worth doing this kind of thing **even if you're only calling it once** because the more polymorphic your code is, the fewer ways there are to write it because you have fewer operations available. So that means there are fewer ways to write it wrong. And in fact sometimes if it typechecks it has to be correct because there's only one way to do it. - so in any case, with this method our `readTree` method is trivial. --- # And now read it back. Construct a program to read a `ProfF` whose students are database ids. ```scala def readNode(id: Int): ConnectionIO[ProfF[Int]] = for { d <- tsql"SELECT name, year FROM prof WHERE id = $id".unique[(String, Int)] ss <- tsql"SELECT id FROM prof WHERE parent = $id".as[List[Int]] } yield ProfF(d._1, d._2, ss) ``` Construct a program to read a `ProfF` tree *annotated* with database ids. ```scala def unfoldCM[M[_]: Monad, F[_]: Traverse, A](a: A)(f: A => M[F[A]]): M[Cofree[F, A]] = f(a).flatMap(_.traverse(unfoldCM(_)(f)).map(Cofree(a, _))) def readTree(id: Int): ConnectionIO[Cofree[ProfF, Int]] = unfoldCM(id)(readNode) ``` ??? - But does it work? --- # Does it work? Let's read Hilbert's sub-tree. ```scala val hilbert: ConnectionIO[Int] = tsql""" select max(id) -- sorry from prof where name like '%Hilbert' """.unique[Int] val io: IO[Cofree[ProfF, Int]] = (hilbert flatMap readTree).transact(xa) ``` ```scala scala> io.flatMap(ioDrawCofree(_)).unsafePerformIO 1103 :< ProfF(David Hilbert, 1885, «3») 1104 :< ProfF(Wilhelm Ackermann, 1925, «0») 1105 :< ProfF(Haskell Curry, 1930, «0») 1106 :< ProfF(Hermann Weyl, 1908, «1») 1107 :< ProfF(Saunders Mac Lane, 1934, «2») 1108 :< ProfF(Steven Awodey, 1997, «0») 1109 :< ProfF(William Howard, 1956, «0») ``` ??? - yes! - and I think it's really cool that we're using a totally generic monadic unfold to do database programming. We never once thought "hey we're talking to a database here so we need to be careful, we have a connection and a resultset and iteration" and all that ... we just went where the types took us. - Ok, so what have we seen? --- # Review - We learned about some recursive types. - We used them to do something useful (or at least interesting). - We pushed on our code and did some equational reasoning. - We were not frightened. ??? - `Fix` and `Cofree` popped out of our model naturally just by pushing a little bit; it's not a contrived example, the question of what to do with database IDs is a real thing we think about. - And we used them to represent the same data type in a bunch of different ways: as a simple recursive data type, as an isolated node with unknown child types, as as tree annotated with parse positions, as a node with database references rather than object references, and of course as a tree annotated with database ids. We did all of this with the same definition of `ProfF`. - We did some code pushing. Not much more to say about that, I just wanted to demonstrate the way we think about things when we do FP with expressive types, so hopefully even if you didn't follow all of it you get a sense for how powerful it is. - And hopefully you all survived. And if not I have bad news for you because Greg Pfiel is up next and he's going to be talking about using recursion schemes in a much more general and more powerful way. - sooooo, that's it. --- class: center, middle # Thanks! Rob Norris • @tpolecat I will tweet a link to slides and code (not up yet, sorry). ## Learn You a Recursion Scheme `slamdata/matryoshka` and `sellout/recursion-scheme-talk` on github ## Learn You a Doobie Presentations, Videos, Doc at `tpolecat/doobie` on GitHub Online Exercises at `scala-exercises.org` (!)


Try out our Premium Member services: Virtual Legal Assistant, Query Alert Service and an ad-free experience. Free for one month and pay only if you like it.
1 IN THE HIGH COURT OF KARNATAKA AT BANGALORE DATED THIS THE 10th DAY OF SEPTEMBER 2014 PRFESENT THE HON'BLE MR.D.H. WAGHELA, CHIEF JUSTICE AND THE HON'BLE MR. JUSTICE ASHOK B. HINCHIGERI WRIT PETITION NO.14672/2014 (GM-POL-PIL) BETWEEN: H.R. Harisha Halli, 33 years, S/o.K.B. Rajanna Halli, R/o. Kabbala Village Doddaghatta Post, Basavanapatna Hobli, Channagiri Taluk, Davanagere District-577 014 ... PETITIONER (By Sri. Rudrappa.P., Advocate) AND: 1.The State of Karnataka Ministry of Environment and Forest, (Pollution Control Department), Represented by its Secretary, Vidhana Soudha, Bangalore-560 001. 2.The State of Karnataka, Water Resources Department, Represented by its Secretary, Vidhana Soudha, Bangalore-560 001. 3.The State of Karnataka Education Department, 2 Represented by its Secretary, Vidhana Soudha, Bangalore-560 001. 4.Karnataka State Pollution Control Board, "Parisara Bhavana" 1 to 5th Floor, # 49, Church Street, Bangalore-560 001. Karnataka. Represented by Its Member Secretary. 5.Karnataka State Pollution Control Board, Divisional Office, CA.No.02, 3rd Main, KHB colony, Behind Pragathi Gramin Bank, Near KHB Office, Sadik Nagar Road, Chitradurga-577 501. Represented by Regional Senior Environmental Officer. 6. Karnataka State Pollution Control Board, Regional Office, No.501, 'C' Block, Devaraj Uras Layout, Davanagere-577 006 Represented by its Regional Officer, 7.The Deputy Commissioner Davanagere District, Davanagere-577 002. 8.The Director Pre-University Board, 18th Cross, Malleshwaram, Bangalore-3. 9.The Deputy Director of Public Instructions, Education Department, Davanagere-577 002. 10.The Commissioner Davanagere Mahanagara Palike Corporation Building, P.B. Road, Davanagere-577 002 3 11.The Head Master, Vishwachethana Vidyanikethana Residential School N.H.-4, Bypass Road, Shiramagondanahalli Village Davanagere Taluk, Davanagere District-577 002 12.M/s.Vishwachethana Vidyanikethana Vidha Samste N.H.4, Bypass Road, Shiramagondanahalli Village Davanagere Taluk, Davanagere District-577 002. Represented by its Chairmen. ... RESPONDENTS (By Smt.Shweta Krishnappa, HCGP for R-1 to R-3, 7 and 9, Sri.Gururaj Joshi, Advocate for R-6, Sri.B.S. Hadimani, Advocate for R-11 and R-12, R-5 is served) -0-0-0-0-0- This writ petition is filed under Articles 226 and 227 of the Constitution of India praying to direct respondent Nos.4 to 6 to prevent the Air and Water Pollution by the Respondent Nos.1 and 12 or by any other persons etc. This petition coming on for preliminary hearing this day, Chief Justice made the following:- 4 ORDER
D.H.WAGHELA, CJ(Oral)
1. This petition is moved by an RTI Activist on the basis of the material showing consistent failure on the part of the Karnataka State Pollution Control Board in taking any action against Respondent No.11 and 12, who have admittedly been discharging untreated effluent into an open drain which ultimately joins Bhadra Canal near National Highway-4 at Shiramagondanahalli. Since there is no dispute about the fact of consistent discharge of untreated effluent as also about inaction on the part of the State Pollution Control Board, it is not necessary to further delve into the factual aspect of the matter.
2. However, by filing an affidavit of learned advocate appearing for Respondent No.11 and 12, it is stated that the work of establishing a sewage treatment and recycling plant is already entrusted to M/s.Ecosoft Water Technology, Bangalore, and the work is in progress. It is further declared and undertaken that 5 sewage treatment and recycling plant will be constructed in the premises of Respondent No.12 and will be operational within seven months and latest by 31.3.2015. It is also stated by learned counsel Mr.Gururaj Joshi appearing for the State Pollution Control Board that at least now Respondent No.11 and 12 shall not be permitted to discharge untreated effluent outside the premises of Respondent No.12 and for that purpose the premises shall be inspected by Inspector of Pollution Control Board once every week and in case, if, any violation of the provisions of The Water (Prevention and Control of Pollution) Act, 1974 and Air (Prevention and Control of Pollution) Act, 1981 is found, immediate and energetic action including direction to stop the offending activity and filing of criminal cases will be taken up.
3. Upon the above statements made and recorded, the petition is not pressed for any further relief at this stage. It is however painfully noted during the course of hearing that the State Pollution Control Board appeared to have practically abdicated its duties and functions by 6 not taking any action worth the name for prevention of pollution of the common and important water resource. It was in that context that learned counsel Mr. Gururaj Joshi submitted that the State Pollution Control Board shall take up this matter at the higher level so as to issue necessary standing instructions to all the Officers of the regional offices of the State Pollution Control Board instructing them to take immediate and energetic action in such cases of gross violation of mandatory legal provisions of the aforesaid Acts. He also stated that the instructions proposed to be issued may also include a stipulation that conscious failure of the officers in discharging their duties would be treated as dereliction of duty for the purpose of taking disciplinary action and the damage caused to the environment because of such dereliction of duty may be quantified and ordered to be recovered from the salary of the officers concerned.
4. Accordingly, recording the above statement, petition is disposed with no order as to cost with the expectation that the State Pollution Control Board 7 would take up the matter seriously and file before this Court, within fifteen days, the copies of instructions being circulated in terms of the statements recorded hereinabove.
Sd/-
CHIEF JUSTICE Sd/-
JUDGE.
*alb/-.


Classes
- B
- BNetwork
- BNetwork.PlayerData
- global
- BasicMsg
- MeshWorks
- MeshWorks.Triangle
- UIInvertColor
B
◄ Back to index
Static helper class with very helpful functions
Functions
-
string SerializeString(
string text
)
-
string DeserializeString(
string serialized
)
-
string SerializeDict(
Dictionary<string,string> dictionary
)
-
Dictionary<string,string> DeserializeDict(
string dict
)
-
string SerializeList(
List<string> List
)
-
List<string> Deserializelist(
string list
)
-
Transform[] GetChildren(
Transform target
)
-
void DestroyChildren(
Transform target
)
B.SerializeString
◄ Back to class
string SerializeString(
string text
)
Returns:
- string: Serialized string
Arguments:
- string text: String to serialize
Serializes the provided string to hex ASCII codes.
B.DeserializeString
◄ Back to class
string DeserializeString(
string serialized
)
Returns:
- string: Deserialized string
Arguments:
- string serialized: Previously serialized string
Deserializes strings in a X2 ASCII format
B.SerializeDict
◄ Back to class
string SerializeDict(
Dictionary<string,string> dictionary
)
Returns:
- string: Serialized dictionary
Arguments:
- Dictionary<string,string> dictionary: Dictionary to serialize
A function to serialize a string Dictionary.
B.DeserializeDict
◄ Back to class
Dictionary<string,string> DeserializeDict(
string dict
)
Returns:
- Dictionary<string,string>: Deserialized Dictionary
Arguments:
- string dict: Previously serialized Dictionary
Deserializes a Dictionary in a X2 ASCII format.
B.SerializeList
◄ Back to class
string SerializeList(
List<string> List
)
Returns:
Arguments:
- List<string> List: List to serialize
A function to serialize a string List.
B.Deserializelist
◄ Back to class
List<string> Deserializelist(
string list
)
Returns:
- List<string>: Deserialized List
Arguments:
- string list: Previously serialized List
Deserializes a List in a X2 ASCII format.
B.GetChildren
◄ Back to class
Transform[] GetChildren(
Transform target
)
Returns:
- Transform[]: Children of the provided Transform
Arguments:
- Transform target: The Transform to get children of
Returns all children of the provided Transform
B.DestroyChildren
◄ Back to class
void DestroyChildren(
Transform target
)
Returns:
Arguments:
- Transform target: The Transform to remove children of
Removes all children of the provided Transform
BNetwork
◄ Back to index
Component for establishing lowlevel connections between computers and basic msg exchange.
Methods
-
void Listen(
)
-
void Connect(
string ip
)
-
void StartLocalPlayer(
)
-
void Broadcast(
BasicMsg message
)
-
void BroadcastFromServer(
BasicMsg message
)
Properties
-
NetworkClient client = null
-
BNetwork instance = null
-
int port = 7777
-
bool listening = false
-
BasicMsg connected = null
BNetwork.Listen
◄ Back to class
void Listen(
)
Returns:
Starts the server listening.
BNetwork.Connect
◄ Back to class
void Connect(
string ip
)
Returns:
Arguments:
- string ip: The ip to connect to
Connects the client to a server on the provided ip address.
BNetwork.StartLocalPlayer
◄ Back to class
void StartLocalPlayer(
)
Returns:
Starts a server using Listen and connect to it.
BNetwork.Broadcast
◄ Back to class
void Broadcast(
BasicMsg message
)
Returns:
Arguments:
- BasicMsg message: Message to send
Sends a message to all connected clients.
BNetwork.BroadcastFromServer
◄ Back to class
void BroadcastFromServer(
BasicMsg message
)
Returns:
Arguments:
- BasicMsg message: Message to send
Sends a message to the server.
BNetwork.client
◄ Back to class
NetworkClient client = null
Client used to connect to a server
BNetwork.instance
◄ Back to class
BNetwork instance = null
Singleton instance
BNetwork.port
◄ Back to class
int port = 7777
Port to connect or listen
BNetwork.listening
◄ Back to class
bool listening = false
Is the BNetwork listening?
BNetwork.connected
◄ Back to class
BasicMsg connected = null
Last message recieved
BNetwork.PlayerData
◄ Back to index
global
◄ Back to index
A global object whitch members acessible anywhere.
Properties
global.ip
◄ Back to class
string ip = null
IP address of the player.
BasicMsg
◄ Back to index
Functions
-
new BasicMsg(
string type, string target, string message
)
Properties
-
string type = null
-
string target = null
-
string msg = null
-
int source = null
BasicMsg.BasicMsg
◄ Back to class
new BasicMsg(
string type, string target, string message
)
Returns:
Arguments:
- string type: Message type
- string target: Message target
- string message: Message message
BasicMsg.type
◄ Back to class
string type = null
BasicMsg.target
◄ Back to class
string target = null
BasicMsg.msg
◄ Back to class
string msg = null
BasicMsg.source
◄ Back to class
int source = null
MeshWorks
◄ Back to index
A static class to help with mesh creation.
MeshWorks.Triangle
◄ Back to index
Class describing a single triangle
Functions
-
new MeshWorks.Triangle(
Vector3 a, Vector3 b, Vector3 c
)
Properties
-
Vector3 a = null
-
Vector3 b = null
-
Vector3 c = null
MeshWorks.Triangle.Triangle
◄ Back to class
new MeshWorks.Triangle(
Vector3 a, Vector3 b, Vector3 c
)
Returns:
Arguments:
Creates a new triangle with three points.
MeshWorks.Triangle.a
◄ Back to class
Vector3 a = null
First point
MeshWorks.Triangle.b
◄ Back to class
Vector3 b = null
Second point
MeshWorks.Triangle.c
◄ Back to class
Vector3 c = null
Third point
UIInvertColor
◄ Back to index
UI helper class for dynamic contrast.
Properties
-
Graphic source = null
-
Graphic target = null
UIInvertColor.source
◄ Back to class
Graphic source = null
Graphic with the source color
UIInvertColor.target
◄ Back to class
Graphic target = null
Traget Graphic to set the inverted color


A custom scatterplot with auto-positioned labels to explore the
palmerpenguins dataset made with R
and the
tidyverse
. This blogpost guides you through a highly
customized scatterplot that includes a variety of custom colors,
markers, and fonts. The library ggrepel
is used to
automatically adjust the position of labels in the plots.
This page showcases the work of
Tuo Wang that introduces
packages to make
ggplot2
plots more beautiful. You can find the original code on Tuo’s blog
here.
Thanks to him for accepting sharing his work here! Thanks also to Tomás Capretto who split the original code into this step-by-step guide!
As usual, it is first necessary to load some packages before
building the figure.
ggrepel
provides geoms for ggplot2
to repel overlapping text
labels. Text labels repel away from each other, away from data
points, and away from edges of the plotting area in an automatic
fashion. Also,
randomNames
is used to generate random names that will be the text labels in the
chart.
Note: randomNames
is only available
for R > 4.0.0.
library(ggrepel)
library(palmerpenguins)
library(randomNames)
library(tidyverse)
The palmerpenguins data were collected and made available by
Dr. Kristen Gorman
and the
Palmer Station, Antarctica LTER, a member of the
Long Term Ecological Research Network. This dataset was popularized by
Allison Horst in her R
package
palmerpenguins
with the goal to offer an alternative to the iris dataset for data
exploration and visualization.
data("penguins", package = "palmerpenguins")
First of all, observations with missing values are discarded from the dataset.
<- drop_na(penguins) penguins
Then, each observation is assined a random name. A new variable
highlight
is added to the dataset to indicate which names
are highlighted in the plot. These are the names starting with the
letter "C"
.
## Generate random names
# The results of set.seed may depends on R version.
set.seed(2021+03+27)
<- randomNames(nrow(penguins), which.names = "first")
name_vector
## Create 'highlight' indicator variable
<- penguins %>%
penguins mutate(
name = name_vector,
highlight = case_when(
str_starts(name, "C") ~ name,
TRUE ~ ""
) )
Today’s chart is a scatterplot that shows the association between
the flip length and the bill length of the penguins in the dataset.
Point are colored according to the species to add an extra layer of
information to the vizualisation. The first step is to create a
basic colored scatterplot with ggplot2
. Let’s get
started!
# Note `color = species` and s`hape = species`.
# This means each species will have BOTH a different color and shape.
<- ggplot(penguins, aes(x = flipper_length_mm, y = bill_length_mm)) +
plt geom_point(
aes(color = species, shape = species),
size = 1.5,
alpha = 0.8 # It's nice to add some transparency because there may be overlap.
+
) # Use custom colors
scale_color_manual(
values = c("#386cb0", "#fdb462", "#7fc97f")
)
That’s a pretty good start, but let’s make it better!
What’s truely missing here are labels. It’s very frustrating not knowing which item is hidden under a data point, isn’t it!?
It is pretty challenging to add many labels on a plot since labels
tend to overlap each other, making the figure unreadable.
Fortunately, the ggrepel
package is here to help us. It
provides an algorithm that will automatically place the labels for
us. Let’s do it!
<- plt +
plt geom_text_repel(
aes(label = highlight),
family = "Poppins",
size = 3,
min.segment.length = 0,
seed = 42,
box.padding = 0.5,
max.overlaps = Inf,
arrow = arrow(length = unit(0.010, "npc")),
nudge_x = .15,
nudge_y = .5,
color = "grey50"
)
Isn’t it wonderful how well ggrepel
works?
The chart above is pretty close from being publication ready. What’s needed now is a good title, a legend to make color and shapes more insightful, and some axis customization:
<- plt +
plt # Add axes labels, title, and subtitle
labs(
title = "Palmer Penguins Data Visualization",
subtitle = "Scatter plot of flipper lenth vs bill length",
x = "flip length (mm)",
y = "bill length (mm)"
+
) theme(
# The default font when not explicitly specified
text = element_text(family = "Lobster Two", size = 8, color = "black"),
# Customize legend text, position, and background.
legend.text = element_text(size = 9, family = "Roboto"),
legend.title = element_text(face = "bold", size=12, family = "Roboto"),
legend.position = c(1, 0),
legend.justification = c(1, 0),
legend.background = element_blank(),
# This one removes the background behind each key in the legend
legend.key = element_blank(),
# Customize title and subtitle font/size/color
plot.title = element_text(
family = "Lobster Two",
size = 20,
face = "bold",
color = "#2a475e"
),plot.subtitle = element_text(
family = "Lobster Two",
size = 15,
face = "bold",
color = "#1b2838"
),plot.title.position = "plot",
# Adjust axis parameters such as size and color.
axis.text = element_text(size = 10, color = "black"),
axis.title = element_text(size = 12),
axis.ticks = element_blank(),
# Axis lines are now lighter than default
axis.line = element_line(colour = "grey50"),
# Only keep y-axis major grid lines, with a grey color and dashed type.
panel.grid.minor = element_blank(),
panel.grid.major.x = element_blank(),
panel.grid.major.y = element_line(color = "#b4aea9", linetype ="dashed"),
# Use a light color for the background of the plot and the panel.
panel.background = element_rect(fill = "#fbf9f4", color = "#fbf9f4"),
plot.background = element_rect(fill = "#fbf9f4", color = "#fbf9f4")
)
What a lovely plot!


Contact Us
- PROTECH CONSTRUCTION & DEVELOPMENT CORPORATION
- 9446 Tibig Rd. Kaong, Silang Cavite, Philippines
- +639171596650
- Type of Organization: Corporation
- SEC Registration No.: CS200911053
- Email us at<EMAIL_ADDRESS>PROTECH CONSTRUCTION & DEVELOPMENT CORPORATION shall establish and maintain Quality Management System and continually improve its effectiveness in accordance with ISO 9001:2015 Requirement.
×


tmp_2c78d90
and
tmp_2c78d90
DIAG Set 8 - Annual cycle contour plots of zonal means
Variable
Description
FLNT
TOM net longwave flux
plot
PRECT *
Precipitation rate
plot
PREH2O *
Total precipitable water
plot
SOLIN
Incoming solar radiation
plot
TAUX OCEAN *
Zonal surface stress (ocean)
plot
U 200 *
200 mb zonal wind
plot
CFMIP Observation Simulator Package (COSP) Primary Diagnostics
ISCCP-COSP From D1 Daytime Jul1983-Sep2008
CALIPSO GOCCP 2007-2010
CLDTOT_CAL
Total cloud amount
plot
CLDLOW_CAL
Low-level cloud amount
plot
CLDMED_CAL
Mid-level cloud amount
plot
CLDHGH_CAL
High-level cloud amount
plot
ISCCP-COSP From D1 Daytime Jul1983-Sep2008
CLDTOT_ISCCPCOSP
Total cloud amount (tau > 0.3)
plot
CLDTHICK_ISCCPCOSP
Optically thick cloud amount (tau > 23.0)
plot
MISR L3 CTH-OD V5 Mar2000-Nov2009
CLDTOT_MISR
Total cloud amount (tau > 0.3)
plot
CLDTHICK_MISR
Optically thick cloud amount (tau > 23.0)
plot
MODIS-COSP 2003-2010
CLDTOT_MODIS
Total cloud amount (tau > 0.3)
plot
CLDTHICK_MODIS
Optically thick cloud amount (tau > 23.0)
plot
Additional COSP Diagnostics
CALIPSO GOCCP 2007-2010
CLDTOT_CAL
Total cloud amount
plot
CLDLOW_CAL
Low-level cloud amount
plot
CLDMED_CAL
Mid-level cloud amount
plot
CLDHGH_CAL
High-level cloud amount
plot
CLOUDSAT-COSP From Level 2 GEOPROF Jun2006-Nov2010
CLDTOT_CS2
Total cloud amount
plot
ISCCP-COSP From D1 Daytime Jul1983-Sep2008
CLDTOT_ISCCPCOSP
Total cloud amount (tau > 0.3)
plot
CLDLOW_ISCCPCOSP
Low-topped cloud amount (tau > 0.3)
plot
CLDMED_ISCCPCOSP
Mid-topped cloud amount (tau > 0.3)
plot
CLDHGH_ISCCPCOSP
High-topped cloud amount (tau > 0.3)
plot
CLDTHICK_ISCCPCOSP
Optically thick cloud amount (tau > 23.0)
plot
MEANPTOP_ISCCPCOSP
Mean cloud top pressure
plot
MEANCLDALB_ISCCPCOSP
Mean cloud albedo
plot
MISR L3 CTH-OD V5 Mar2000-Nov2009
CLDTOT_MISR
Total cloud amount (tau > 0.3)
plot
CLDLOW_MISR
Low-topped cloud amount (tau > 0.3)
plot
CLDMED_MISR
Mid-topped cloud amount (tau > 0.3)
plot
CLDHGH_MISR
High-topped cloud amount (tau > 0.3)
plot
CLDTHICK_MISR
Optically thick cloud amount (tau > 23.0)
plot
MODIS-COSP 2003-2010
CLDTOT_MODIS
Total cloud amount (tau > 0.3)
plot
CLDLOW_MODIS
Low-topped cloud amount (tau > 0.3)
plot
CLDMED_MODIS
Mid-topped cloud amount (tau > 0.3)
plot
CLDHGH_MODIS
High-topped cloud amount (tau > 0.3)
plot
CLDTHICK_MODIS
Optically thick cloud amount (tau > 23.0)
plot
CLIMODIS
Total ice cloud amount
plot
CLWMODIS
Total liquid cloud amount
plot
IWPMODIS
Mean ice water path
plot
LWPMODIS
Mean liquid water path
plot
PCTMODIS
Mean cloud top pressure
plot
REFFCLIMODIS
Mean effective radius (ice clouds)
plot
REFFCLWMODIS
Mean effective radius (liquid clouds)
plot
TAUILOGMODIS
Log-weighted mean optical thickness (ice clouds)
plot
TAUWLOGMODIS
Log-weighted mean optical thickness (liquid clouds)
plot
TAUTLOGMODIS
Log-weighted mean optical thickness (all clouds)
plot
TAUIMODIS
Mean optical thickness (ice clouds)
plot
TAUWMODIS
Mean optical thickness (liquid clouds)
plot
TAUTMODIS
Mean optical thickness (all clouds)
plot
*WGNE diagnostic variable


|
Asylo
|
A reference to an enclave held by untrusted code. More...
#include <asylo/platform/primitives/untrusted_primitives.h>
Classes | |
| class | ExitCallProvider |
| An interface to a provider of enclave exit calls. More... | |
| class | ScopedCurrentClient |
| An RAII wrapper that sets thread-local enclave "current client" reference on construction and resets it to the previous value when destroyed. More... | |
Public Member Functions | |
| virtual | ~Client () |
| virtual Status | RegisterExitHandlers () ASYLO_MUST_USE_RESULT |
| An overridable handler registration method. More... | |
| virtual bool | IsClosed () const =0 |
| A predicate for whether the enclave may be entered or will accept messages. More... | |
| virtual Status | Destroy ()=0 |
| virtual absl::string_view | Name () const |
| A getter for the enclave name. More... | |
| void | SetCurrentClient () |
Stores this as the active thread's "current client". More... | |
| Status | EnclaveCall (uint64_t selector, MessageWriter *input, MessageReader *output) ASYLO_MUST_USE_RESULT |
Enters the enclave synchronously at an entry point to trusted code designated by selector . More... | |
| ExitCallProvider * | exit_call_provider () |
| Accessor to the client's exit call provider. More... | |
| virtual void | RegisterMemory (void *mem) |
| Register memory to be freed upon enclave destruction. More... | |
| virtual void | ReleaseMemory () |
| Frees enclave resources registered to the client. More... | |
Static Public Member Functions | |
| static Client * | GetCurrentClient () |
| A static getter for the current client. More... | |
| static PrimitiveStatus | ExitCallback (uint64_t untrusted_selector, MessageReader *in, MessageWriter *out) |
| Enclave exit callback function shared with the enclave. More... | |
Protected Member Functions | |
| Client (const absl::string_view name, std::unique_ptr< ExitCallProvider > exit_call_provider) | |
| Constructs a client, reserved for only backend implementations. More... | |
| virtual Status | EnclaveCallInternal (uint64_t selector, MessageWriter *input, MessageReader *output) ASYLO_MUST_USE_RESULT=0 |
| Provides implementation of EnclaveCall. More... | |
A reference to an enclave held by untrusted code.
This declares the primitive API exposed to untrusted application code by the Asylo runtime. Each Asylo backend is responsible for providing an implementation of this interface. To support multiple implementations, the interface defines a generic "Enclave Backend" concept which every backend must implement. An enclave backend is a structure compatible with:
|
inlinevirtual |
|
inlineprotected |
Constructs a client, reserved for only backend implementations.
| name | The name of the enclave. |
| exit_call_provider | A pointer an ExitCallProvider that the Client takes ownership of. The provider is the source of all ExitHandler s. |
|
pure virtual |
| Status asylo::primitives::Client::EnclaveCall | ( | uint64_t | selector, |
| MessageWriter * | input, | ||
| MessageReader * | output | ||
| ) |
Enters the enclave synchronously at an entry point to trusted code designated by selector
.
Input input
is copied into the enclave, which may occur locally inside the same address space as the caller or remotely via RPC. Conversely, results are copied and returned in 'output'.
| selector | The identification number to select a registered handler in the enclave. |
| input | A pointer to a MessageWriter, into which all call inputs must be pushed. |
| output | A pointer to a MessageReader from which to read outputs from the call. |
|
protectedpure virtual |
Provides implementation of EnclaveCall.
This method is virtual for backends to override. The public EnclaveCall method provides necessary boilerplate around each call to this implementation.
| selector | The identification number to select a registered handler in the enclave. |
| input | A pointer to a MessageWriter, into which all call inputs must be pushed. |
| output | A pointer to a MessageReader from which to read outputs from the call. |
|
inline |
Accessor to the client's exit call provider.
|
static |
Enclave exit callback function shared with the enclave.
| untrusted_selector | The identification number to select a registered handler in the current client. |
| in | A pointer to a MessageReader, from which all inputs are read. |
| out | A pointer to a MessageWriter, into which all call outputs are written. |
|
static |
A static getter for the current client.
|
pure virtual |
A predicate for whether the enclave may be entered or will accept messages.
|
inlinevirtual |
A getter for the enclave name.
Enclave names are used for fetching client instances from the enclave manager.
|
virtual |
An overridable handler registration method.
This allows backend implementations to register special-purpose exit handlers that might only be appropriate to that backend. The default implementation registers nothing and returns Ok.
|
inlinevirtual |
Register memory to be freed upon enclave destruction.
| mem | A pointer to be passed to free on enclave exit. |
|
inlinevirtual |
Frees enclave resources registered to the client.
| void asylo::primitives::Client::SetCurrentClient | ( | ) |
Stores this
as the active thread's "current client".
This should only be called if an enclave entry happens without going through a regular enclave entry point (like a fork from inside the enclave).


The people counting stystem is one of the smart city sollution where it provides both real-time and historical data for big data analytics. Business-people looking to open a new shop or restaurant could be given accurate figures for footfall past their proposed location to help them assess the potential for their new venture and for Monitoring the numbers of people arriving on platforms provides vital information on the use of routes by time of day
We propose an automatic people counting system by using Kaa Internet-of-Things (IoT) platform consisting of a single camera and Raspberry Pi. In this system, we count the number of moving people in bidirection. The background subtraction and morphological operations are used to extract foreground objects from background images. Although the algorithms for the people counting and classification are not novel, our technical contribution is that we have implemented them into our IoT platform. In addition, the images do not need be sent to the server, but all the image processing is done inside the device and only the results are uploaded to the server. This system can be applied to for customer behavior analysis or security.
Read MoreCount the number of people entering, exiting and passing by your store in real time.
Discover your power hours in which your stores generate most traffic, and have biggest sale opportunities.
Optimize staff operation in accordance to the number of visitors and their needs within your stores during power hours
Benchmark high-performing stores and optimize your lowest performing stores to reach their performance.
To effectively serve people it helps if the city authorities know how many people are going where and when. The flow of pedestrians affects the whole city environment. It is important to know not only the numbers of people but also how the numbers change over time. An accurate people metering system provides feedback on changes in people’s behaviour within the city and allows the council to effectively manage the flow of people, vehicles and bicycles around the area. This is also known as smart mobility.


Smile's classification algorithms are in the package
smile.classification
and all algorithms implement the interface
Classifier
that has a method predict
to predict the class label of an instance.
An overloaded version in SoftClassifier
can also calculate the a posteriori
probabilities besides the class label.
For all algorithms, the model can be trained through
the constructor. Meanwhile, each algorithm has a Trainer
companion
class that can hold model hyper-parameters and be applied to multiple training
datasets.
Some algorithms with online learning capability also implement the interface
OnlineClassifier
Online learning is a model of induction that learns one instance at a time.
The method learn
updates the model with a new instance.
The high-level operators are defined in Scala trait
smile.classification.Operators
and also in the package object of smile.classification
.
In what follows, we will discuss each algorithm, their high-level Scala API, and examples.
In machine learning and pattern recognition, classification refers to an algorithmic procedure for assigning a given input object into one of a given number of categories. The input object is formally termed an instance, and the categories are termed classes.
The instance is usually described by a vector of features, which together constitute a description of all known characteristics of the instance. Typically, features are either categorical (also known as nominal, i.e. consisting of one of a set of unordered items, such as a gender of "male" or "female", or a blood type of "A", "B", "AB" or "O"), ordinal (consisting of one of a set of ordered items, e.g. "large", "medium" or "small"), integer-valued (e.g. a count of the number of occurrences of a particular word in an email) or real-valued (e.g. a measurement of blood pressure).
Classification normally refers to a supervised procedure, i.e. a procedure that produces an inferred function to predict the output value of new instances based on a training set of pairs consisting of an input object and a desired output value. The inferred function is called a classifier if the output is discrete or a regression function if the output is continuous.
The inferred function should predict the correct output value for any valid input object. This requires the learning algorithm to generalize from the training data to unseen situations in a "reasonable" way.
A wide range of supervised learning algorithms is available, each with its strengths and weaknesses. There is no single learning algorithm that works best on all supervised learning problems. The most widely used learning algorithms are AdaBoost and gradient boosting, support vector machines, linear regression, linear discriminant analysis, logistic regression, naive Bayes, decision trees, k-nearest neighbor algorithm, and neural networks (multilayer perceptron).
If the feature vectors include features of many different kinds (discrete, discrete ordered, counts, continuous values), some algorithms cannot be easily applied. Many algorithms, including linear regression, logistic regression, neural networks, and nearest neighbor methods, require that the input features be numerical and scaled to similar ranges (e.g., to the [-1,1] interval). Methods that employ a distance function, such as nearest neighbor methods and support vector machines with Gaussian kernels, are particularly sensitive to this. An advantage of decision trees (and boosting algorithms based on decision trees) is that they easily handle heterogeneous data.
If the input features contain redundant information (e.g., highly correlated features), some learning algorithms (e.g., linear regression, logistic regression, and distance based methods) will perform poorly because of numerical instabilities. These problems can often be solved by imposing some form of regularization.
If each of the features makes an independent contribution to the output, then algorithms based on linear functions (e.g., linear regression, logistic regression, linear support vector machines, naive Bayes) generally perform well. However, if there are complex interactions among features, then algorithms such as nonlinear support vector machines, decision trees and neural networks work better. Linear methods can also be applied, but the engineer must manually specify the interactions when using them.
There are several major issues to consider in supervised learning:
The accuracy of the inferred function depends strongly on how the input object is represented. Typically, the input object is transformed into a feature vector, which contains a number of features that are descriptive of the object. The number of features should not be too large, because of the curse of dimensionality; but should contain enough information to accurately predict the output.
There are many algorithms for feature selection that seek to identify the relevant features and discard the irrelevant ones. More generally, dimensionality reduction may seek to map the input data into a lower dimensional space prior to running the supervised learning algorithm.
Overfitting occurs when a statistical model describes random error or noise instead of the underlying relationship. Overfitting generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. A model which has been overfit will generally have poor predictive performance, as it can exaggerate minor fluctuations in the data.
The potential for overfitting depends not only on the number of parameters and data but also the conformability of the model structure with the data shape, and the magnitude of model error compared to the expected level of noise or error in the data.
In order to avoid overfitting, it is necessary to use additional techniques (e.g. cross-validation, regularization, early stopping, pruning, Bayesian priors on parameters or model comparison), that can indicate when further training is not resulting in better generalization. The basis of some techniques is either (1) to explicitly penalize overly complex models, or (2) to test the model's ability to generalize by evaluating its performance on a set of data not used for training, which is assumed to approximate the typical unseen data that a model will encounter.
Regularization involves introducing additional information in order to solve an ill-posed problem or to prevent over-fitting. This information is usually of the form of a penalty for complexity, such as restrictions for smoothness or bounds on the vector space norm. A theoretical justification for regularization is that it attempts to impose Occam's razor on the solution. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters.
Mean squared error (MSE) can be broken down into two components: variance and squared bias, known as the bias-variance decomposition. Thus in order to minimize the MSE, we need to minimize both the bias and the variance. However, this is not trivial. Therefore, there is a tradeoff between bias and variance.
The k-nearest neighbor algorithm (k-NN) is a method for classifying objects by a majority vote of its neighbors, with the object being assigned to the class most common amongst its k nearest neighbors (k is a positive integer, typically small). k-NN is a type of instance-based learning, or lazy learning where the function is only approximated locally and all computation is deferred until classification.
The best choice of k depends upon the data; generally, larger values of k reduce the effect of noise on the classification, but make boundaries between classes less distinct. A good k can be selected by various heuristic techniques, e.g. cross-validation. In binary problems, it is helpful to choose k to be an odd number as this avoids tied votes.
A drawback to the basic majority voting classification is that the classes with the more frequent instances tend to dominate the prediction of the new object, as they tend to come up in the k nearest neighbors when the neighbors are computed due to their large number. One way to overcome this problem is to weight the classification taking into account the distance from the test point to each of its k nearest neighbors.
Often, the classification accuracy of k-NN can be improved significantly if the distance metric is learned with specialized algorithms such as Large Margin Nearest Neighbor or Neighborhood Components Analysis.
Nearest neighbor rules in effect compute the decision boundary in an implicit manner. It is also possible to compute the decision boundary itself explicitly, and to do so in an efficient manner so that the computational complexity is a function of the boundary complexity.
The nearest neighbor algorithm has some strong consistency results. As the amount of data approaches infinity, the algorithm is guaranteed to yield an error rate no worse than twice the Bayes error rate (the minimum achievable error rate given the distribution of the data). k-NN is guaranteed to approach the Bayes error rate, for some value of k (where k increases as a function of the number of data points).


Mortuus
Terra
Navigate
Home
Forum
Server
Want to join the Mortuus Terra development team?! Contact<EMAIL_ADDRESS>if you're interested to join or want more info!


Please complete All relevant areas, and be truthful to all the provided information as we do a background check on all the provided information
First Name:
Second Name:
National ID/Passport
Gender: Single Married other
Email:
Telephone:
Account Type: Savings Account Personal Account Current Account Fixed Account
Currency: RWF USD


Ravenclaw is one of the four Houses of Hogwarts School of Witchcraft and Wizardry, founded by Rowena Ravenclaw. Members of this house are characterised by their wit, learning, and wisdom. Its house colours are blue and bronze, and its symbol is an eagle. The house ghost, who in life was the daughter of the house's founder Rowena Ravenclaw, is the Grey Lady. Ravenclaw corresponds roughly to the element of air; the House colours blue and bronze were chosen to represent the sky and eagle feathers respectively, both having much to do with air.


All of my experience with databases has been with relational databases and "traditional" SQL to interact with MySQL and Microsoft SQL Server. However, I am more than willing to quickly bone-up on a "NO-SQL" database scheme such as MongoDB in a real hurry if the employer needs it.
- Creation of Dynamic SQL Queries in PHP by Programatically Constructing the "WHERE" Clause
- Daily Work with 300-400 Distinct Microsoft SQL Server Database Tables Over the Course of 14 months Working for Company with $97 million Yearly Gross Revenue
- API Development/Maintenance: SQL Queries/Updates for Company Involving Millions of Payroll Records
- MySQL Installation, Configuration and Use in Many Contexts
- PhpMyAdmin: Installation, Configuration, Maintenance
- Custom SQL Solutions in which Specific Rows are "Locked" to a Specific Employee (Avoiding the possibility that two employees attempt to update the same record simultaneously
- JOINS: Left, Right, Inner, Outer Involving 10+ Tables
- .NET Framework ADO Dataobjects, MS SQL Server, and SQL Express
- Crystal Reports for Microsoft: Export PDF Legal Documents for Electronic Court Filings
- Dynamic Reports Creation in PHP from MySQL Database, Export to MS Word, Excel, PDF


We provide you a simple way to rent the vehicle that you would like.Among the park of our cars, everyone can find a car that matches his desire and capabilities!


Larisa Lebedco's Lab 4
First Name
Last Name
Email
Phone
Text field
This is some default message text.
Age
18-35
35-50
50-65
SUBMIT FORM


NOISES OFF
February 16 - 19, 1995
Director - Michael Garner
Assistant Director - Jena Crossley O'Brien
THE CAST
Dotty Otley - Mary Anne Sumrall
Lloyd Dallas - Gene Hudson
Garry LeJeune - Justin Adcock
Brooke Ashton - Alicia Glover Kirkpatrick
Poppy Norton-Taylor - Erin Kate Holston
Frederick Fellowes - Roger Russell
Belinda Blair - Joy Davis
Tim Allgood - John E. Moore
Selsdon Mowbray - James Holston