Getting Started
- Starting Point: SparkSession
- Creating DataFrames
- Untyped Dataset Operations (aka DataFrame Operations)
- Running SQL Queries Programmatically
- Global Temporary View
- Creating Datasets
- Interoperating with RDDs
- Aggregations
Starting Point: SparkSession
The entry point into all functionality in Spark is the SparkSession
class. To create a basic SparkSession
, just use SparkSession.builder()
:
import org.apache.spark.sql.SparkSession
val spark = SparkSession
.builder()
.appName("Spark SQL basic example")
.config("spark.some.config.option", "some-value")
.getOrCreate()
// For implicit conversions like converting RDDs to DataFrames
import spark.implicits._
The entry point into all functionality in Spark is the SparkSession
class. To create a basic SparkSession
, just use SparkSession.builder()
:
import org.apache.spark.sql.SparkSession;
SparkSession spark = SparkSession
.builder()
.appName("Java Spark SQL basic example")
.config("spark.some.config.option", "some-value")
.getOrCreate();
The entry point into all functionality in Spark is the SparkSession
class. To create a basic SparkSession
, just use SparkSession.builder
:
from pyspark.sql import SparkSession
spark = SparkSession \
.builder \
.appName("Python Spark SQL basic example") \
.config("spark.some.config.option", "some-value") \
.getOrCreate()
The entry point into all functionality in Spark is the SparkSession
class. To initialize a basic SparkSession
, just call sparkR.session()
:
sparkR.session(appName = "R Spark SQL basic example", sparkConfig = list(spark.some.config.option = "some-value"))
Note that when invoked for the first time, sparkR.session()
initializes a global SparkSession
singleton instance, and always returns a reference to this instance for successive invocations. In this way, users only need to initialize the SparkSession
once, then SparkR functions like read.df
will be able to access this global instance implicitly, and users don’t need to pass the SparkSession
instance around.
SparkSession
in Spark 2.0 provides builtin support for Hive features including the ability to
write queries using HiveQL, access to Hive UDFs, and the ability to read data from Hive tables.
To use these features, you do not need to have an existing Hive setup.
Creating DataFrames
With a SparkSession
, applications can create DataFrames from an existing RDD
,
from a Hive table, or from Spark data sources.
As an example, the following creates a DataFrame based on the content of a JSON file:
val df = spark.read.json("examples/src/main/resources/people.json")
// Displays the content of the DataFrame to stdout
df.show()
// +----+-------+
// | age| name|
// +----+-------+
// |null|Michael|
// | 30| Andy|
// | 19| Justin|
// +----+-------+
With a SparkSession
, applications can create DataFrames from an existing RDD
,
from a Hive table, or from Spark data sources.
As an example, the following creates a DataFrame based on the content of a JSON file:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
Dataset<Row> df = spark.read().json("examples/src/main/resources/people.json");
// Displays the content of the DataFrame to stdout
df.show();
// +----+-------+
// | age| name|
// +----+-------+
// |null|Michael|
// | 30| Andy|
// | 19| Justin|
// +----+-------+
With a SparkSession
, applications can create DataFrames from an existing RDD
,
from a Hive table, or from Spark data sources.
As an example, the following creates a DataFrame based on the content of a JSON file:
# spark is an existing SparkSession
df = spark.read.json("examples/src/main/resources/people.json")
# Displays the content of the DataFrame to stdout
df.show()
# +----+-------+
# | age| name|
# +----+-------+
# |null|Michael|
# | 30| Andy|
# | 19| Justin|
# +----+-------+
With a SparkSession
, applications can create DataFrames from a local R data.frame,
from a Hive table, or from Spark data sources.
As an example, the following creates a DataFrame based on the content of a JSON file:
df <- read.json("examples/src/main/resources/people.json")
# Displays the content of the DataFrame
head(df)
## age name
## 1 NA Michael
## 2 30 Andy
## 3 19 Justin
# Another method to print the first few rows and optionally truncate the printing of long values
showDF(df)
## +----+-------+
## | age| name|
## +----+-------+
## |null|Michael|
## | 30| Andy|
## | 19| Justin|
## +----+-------+
Untyped Dataset Operations (aka DataFrame Operations)
DataFrames provide a domain-specific language for structured data manipulation in Scala, Java, Python and R.
As mentioned above, in Spark 2.0, DataFrames are just Dataset of Row
s in Scala and Java API. These operations are also referred as “untyped transformations” in contrast to “typed transformations” come with strongly typed Scala/Java Datasets.
Here we include some basic examples of structured data processing using Datasets:
// This import is needed to use the $-notation
import spark.implicits._
// Print the schema in a tree format
df.printSchema()
// root
// |-- age: long (nullable = true)
// |-- name: string (nullable = true)
// Select only the "name" column
df.select("name").show()
// +-------+
// | name|
// +-------+
// |Michael|
// | Andy|
// | Justin|
// +-------+
// Select everybody, but increment the age by 1
df.select($"name", $"age" + 1).show()
// +-------+---------+
// | name|(age + 1)|
// +-------+---------+
// |Michael| null|
// | Andy| 31|
// | Justin| 20|
// +-------+---------+
// Select people older than 21
df.filter($"age" > 21).show()
// +---+----+
// |age|name|
// +---+----+
// | 30|Andy|
// +---+----+
// Count people by age
df.groupBy("age").count().show()
// +----+-----+
// | age|count|
// +----+-----+
// | 19| 1|
// |null| 1|
// | 30| 1|
// +----+-----+
For a complete list of the types of operations that can be performed on a Dataset refer to the API Documentation.
In addition to simple column references and expressions, Datasets also have a rich library of functions including string manipulation, date arithmetic, common math operations and more. The complete list is available in the DataFrame Function Reference.
// col("...") is preferable to df.col("...")
import static org.apache.spark.sql.functions.col;
// Print the schema in a tree format
df.printSchema();
// root
// |-- age: long (nullable = true)
// |-- name: string (nullable = true)
// Select only the "name" column
df.select("name").show();
// +-------+
// | name|
// +-------+
// |Michael|
// | Andy|
// | Justin|
// +-------+
// Select everybody, but increment the age by 1
df.select(col("name"), col("age").plus(1)).show();
// +-------+---------+
// | name|(age + 1)|
// +-------+---------+
// |Michael| null|
// | Andy| 31|
// | Justin| 20|
// +-------+---------+
// Select people older than 21
df.filter(col("age").gt(21)).show();
// +---+----+
// |age|name|
// +---+----+
// | 30|Andy|
// +---+----+
// Count people by age
df.groupBy("age").count().show();
// +----+-----+
// | age|count|
// +----+-----+
// | 19| 1|
// |null| 1|
// | 30| 1|
// +----+-----+
For a complete list of the types of operations that can be performed on a Dataset refer to the API Documentation.
In addition to simple column references and expressions, Datasets also have a rich library of functions including string manipulation, date arithmetic, common math operations and more. The complete list is available in the DataFrame Function Reference.
In Python, it’s possible to access a DataFrame’s columns either by attribute
(df.age
) or by indexing (df['age']
). While the former is convenient for
interactive data exploration, users are highly encouraged to use the
latter form, which is future proof and won’t break with column names that
are also attributes on the DataFrame class.
# spark, df are from the previous example
# Print the schema in a tree format
df.printSchema()
# root
# |-- age: long (nullable = true)
# |-- name: string (nullable = true)
# Select only the "name" column
df.select("name").show()
# +-------+
# | name|
# +-------+
# |Michael|
# | Andy|
# | Justin|
# +-------+
# Select everybody, but increment the age by 1
df.select(df['name'], df['age'] + 1).show()
# +-------+---------+
# | name|(age + 1)|
# +-------+---------+
# |Michael| null|
# | Andy| 31|
# | Justin| 20|
# +-------+---------+
# Select people older than 21
df.filter(df['age'] > 21).show()
# +---+----+
# |age|name|
# +---+----+
# | 30|Andy|
# +---+----+
# Count people by age
df.groupBy("age").count().show()
# +----+-----+
# | age|count|
# +----+-----+
# | 19| 1|
# |null| 1|
# | 30| 1|
# +----+-----+
For a complete list of the types of operations that can be performed on a DataFrame refer to the API Documentation.
In addition to simple column references and expressions, DataFrames also have a rich library of functions including string manipulation, date arithmetic, common math operations and more. The complete list is available in the DataFrame Function Reference.
# Create the DataFrame
df <- read.json("examples/src/main/resources/people.json")
# Show the content of the DataFrame
head(df)
## age name
## 1 NA Michael
## 2 30 Andy
## 3 19 Justin
# Print the schema in a tree format
printSchema(df)
## root
## |-- age: long (nullable = true)
## |-- name: string (nullable = true)
# Select only the "name" column
head(select(df, "name"))
## name
## 1 Michael
## 2 Andy
## 3 Justin
# Select everybody, but increment the age by 1
head(select(df, df$name, df$age + 1))
## name (age + 1.0)
## 1 Michael NA
## 2 Andy 31
## 3 Justin 20
# Select people older than 21
head(where(df, df$age > 21))
## age name
## 1 30 Andy
# Count people by age
head(count(groupBy(df, "age")))
## age count
## 1 19 1
## 2 NA 1
## 3 30 1
For a complete list of the types of operations that can be performed on a DataFrame refer to the API Documentation.
In addition to simple column references and expressions, DataFrames also have a rich library of functions including string manipulation, date arithmetic, common math operations and more. The complete list is available in the DataFrame Function Reference.
Running SQL Queries Programmatically
The sql
function on a SparkSession
enables applications to run SQL queries programmatically and returns the result as a DataFrame
.
// Register the DataFrame as a SQL temporary view
df.createOrReplaceTempView("people")
val sqlDF = spark.sql("SELECT * FROM people")
sqlDF.show()
// +----+-------+
// | age| name|
// +----+-------+
// |null|Michael|
// | 30| Andy|
// | 19| Justin|
// +----+-------+
The sql
function on a SparkSession
enables applications to run SQL queries programmatically and returns the result as a Dataset<Row>
.
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
// Register the DataFrame as a SQL temporary view
df.createOrReplaceTempView("people");
Dataset<Row> sqlDF = spark.sql("SELECT * FROM people");
sqlDF.show();
// +----+-------+
// | age| name|
// +----+-------+
// |null|Michael|
// | 30| Andy|
// | 19| Justin|
// +----+-------+
The sql
function on a SparkSession
enables applications to run SQL queries programmatically and returns the result as a DataFrame
.
# Register the DataFrame as a SQL temporary view
df.createOrReplaceTempView("people")
sqlDF = spark.sql("SELECT * FROM people")
sqlDF.show()
# +----+-------+
# | age| name|
# +----+-------+
# |null|Michael|
# | 30| Andy|
# | 19| Justin|
# +----+-------+
The sql
function enables applications to run SQL queries programmatically and returns the result as a SparkDataFrame
.
df <- sql("SELECT * FROM table")
Global Temporary View
Temporary views in Spark SQL are session-scoped and will disappear if the session that creates it
terminates. If you want to have a temporary view that is shared among all sessions and keep alive
until the Spark application terminates, you can create a global temporary view. Global temporary
view is tied to a system preserved database global_temp
, and we must use the qualified name to
refer it, e.g. SELECT * FROM global_temp.view1
.
// Register the DataFrame as a global temporary view
df.createGlobalTempView("people")
// Global temporary view is tied to a system preserved database `global_temp`
spark.sql("SELECT * FROM global_temp.people").show()
// +----+-------+
// | age| name|
// +----+-------+
// |null|Michael|
// | 30| Andy|
// | 19| Justin|
// +----+-------+
// Global temporary view is cross-session
spark.newSession().sql("SELECT * FROM global_temp.people").show()
// +----+-------+
// | age| name|
// +----+-------+
// |null|Michael|
// | 30| Andy|
// | 19| Justin|
// +----+-------+
// Register the DataFrame as a global temporary view
df.createGlobalTempView("people");
// Global temporary view is tied to a system preserved database `global_temp`
spark.sql("SELECT * FROM global_temp.people").show();
// +----+-------+
// | age| name|
// +----+-------+
// |null|Michael|
// | 30| Andy|
// | 19| Justin|
// +----+-------+
// Global temporary view is cross-session
spark.newSession().sql("SELECT * FROM global_temp.people").show();
// +----+-------+
// | age| name|
// +----+-------+
// |null|Michael|
// | 30| Andy|
// | 19| Justin|
// +----+-------+
# Register the DataFrame as a global temporary view
df.createGlobalTempView("people")
# Global temporary view is tied to a system preserved database `global_temp`
spark.sql("SELECT * FROM global_temp.people").show()
# +----+-------+
# | age| name|
# +----+-------+
# |null|Michael|
# | 30| Andy|
# | 19| Justin|
# +----+-------+
# Global temporary view is cross-session
spark.newSession().sql("SELECT * FROM global_temp.people").show()
# +----+-------+
# | age| name|
# +----+-------+
# |null|Michael|
# | 30| Andy|
# | 19| Justin|
# +----+-------+
Creating Datasets
Datasets are similar to RDDs, however, instead of using Java serialization or Kryo they use a specialized Encoder to serialize the objects for processing or transmitting over the network. While both encoders and standard serialization are responsible for turning an object into bytes, encoders are code generated dynamically and use a format that allows Spark to perform many operations like filtering, sorting and hashing without deserializing the bytes back into an object.
case class Person(name: String, age: Long)
// Encoders are created for case classes
val caseClassDS = Seq(Person("Andy", 32)).toDS()
caseClassDS.show()
// +----+---+
// |name|age|
// +----+---+
// |Andy| 32|
// +----+---+
// Encoders for most common types are automatically provided by importing spark.implicits._
val primitiveDS = Seq(1, 2, 3).toDS()
primitiveDS.map(_ + 1).collect() // Returns: Array(2, 3, 4)
// DataFrames can be converted to a Dataset by providing a class. Mapping will be done by name
val path = "examples/src/main/resources/people.json"
val peopleDS = spark.read.json(path).as[Person]
peopleDS.show()
// +----+-------+
// | age| name|
// +----+-------+
// |null|Michael|
// | 30| Andy|
// | 19| Justin|
// +----+-------+
import java.util.Arrays;
import java.util.Collections;
import java.io.Serializable;
import org.apache.spark.api.java.function.MapFunction;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.Encoder;
import org.apache.spark.sql.Encoders;
public static class Person implements Serializable {
private String name;
private int age;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getAge() {
return age;
}
public void setAge(int age) {
this.age = age;
}
}
// Create an instance of a Bean class
Person person = new Person();
person.setName("Andy");
person.setAge(32);
// Encoders are created for Java beans
Encoder<Person> personEncoder = Encoders.bean(Person.class);
Dataset<Person> javaBeanDS = spark.createDataset(
Collections.singletonList(person),
personEncoder
);
javaBeanDS.show();
// +---+----+
// |age|name|
// +---+----+
// | 32|Andy|
// +---+----+
// Encoders for most common types are provided in class Encoders
Encoder<Integer> integerEncoder = Encoders.INT();
Dataset<Integer> primitiveDS = spark.createDataset(Arrays.asList(1, 2, 3), integerEncoder);
Dataset<Integer> transformedDS = primitiveDS.map(
(MapFunction<Integer, Integer>) value -> value + 1,
integerEncoder);
transformedDS.collect(); // Returns [2, 3, 4]
// DataFrames can be converted to a Dataset by providing a class. Mapping based on name
String path = "examples/src/main/resources/people.json";
Dataset<Person> peopleDS = spark.read().json(path).as(personEncoder);
peopleDS.show();
// +----+-------+
// | age| name|
// +----+-------+
// |null|Michael|
// | 30| Andy|
// | 19| Justin|
// +----+-------+
Interoperating with RDDs
Spark SQL supports two different methods for converting existing RDDs into Datasets. The first method uses reflection to infer the schema of an RDD that contains specific types of objects. This reflection-based approach leads to more concise code and works well when you already know the schema while writing your Spark application.
The second method for creating Datasets is through a programmatic interface that allows you to construct a schema and then apply it to an existing RDD. While this method is more verbose, it allows you to construct Datasets when the columns and their types are not known until runtime.
Inferring the Schema Using Reflection
The Scala interface for Spark SQL supports automatically converting an RDD containing case classes
to a DataFrame. The case class
defines the schema of the table. The names of the arguments to the case class are read using
reflection and become the names of the columns. Case classes can also be nested or contain complex
types such as Seq
s or Array
s. This RDD can be implicitly converted to a DataFrame and then be
registered as a table. Tables can be used in subsequent SQL statements.
// For implicit conversions from RDDs to DataFrames
import spark.implicits._
// Create an RDD of Person objects from a text file, convert it to a Dataframe
val peopleDF = spark.sparkContext
.textFile("examples/src/main/resources/people.txt")
.map(_.split(","))
.map(attributes => Person(attributes(0), attributes(1).trim.toInt))
.toDF()
// Register the DataFrame as a temporary view
peopleDF.createOrReplaceTempView("people")
// SQL statements can be run by using the sql methods provided by Spark
val teenagersDF = spark.sql("SELECT name, age FROM people WHERE age BETWEEN 13 AND 19")
// The columns of a row in the result can be accessed by field index
teenagersDF.map(teenager => "Name: " + teenager(0)).show()
// +------------+
// | value|
// +------------+
// |Name: Justin|
// +------------+
// or by field name
teenagersDF.map(teenager => "Name: " + teenager.getAs[String]("name")).show()
// +------------+
// | value|
// +------------+
// |Name: Justin|
// +------------+
// No pre-defined encoders for Dataset[Map[K,V]], define explicitly
implicit val mapEncoder = org.apache.spark.sql.Encoders.kryo[Map[String, Any]]
// Primitive types and case classes can be also defined as
// implicit val stringIntMapEncoder: Encoder[Map[String, Any]] = ExpressionEncoder()
// row.getValuesMap[T] retrieves multiple columns at once into a Map[String, T]
teenagersDF.map(teenager => teenager.getValuesMap[Any](List("name", "age"))).collect()
// Array(Map("name" -> "Justin", "age" -> 19))
Spark SQL supports automatically converting an RDD of
JavaBeans into a DataFrame.
The BeanInfo
, obtained using reflection, defines the schema of the table. Currently, Spark SQL
does not support JavaBeans that contain Map
field(s). Nested JavaBeans and List
or Array
fields are supported though. You can create a JavaBean by creating a class that implements
Serializable and has getters and setters for all of its fields.
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.MapFunction;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.Encoder;
import org.apache.spark.sql.Encoders;
// Create an RDD of Person objects from a text file
JavaRDD<Person> peopleRDD = spark.read()
.textFile("examples/src/main/resources/people.txt")
.javaRDD()
.map(line -> {
String[] parts = line.split(",");
Person person = new Person();
person.setName(parts[0]);
person.setAge(Integer.parseInt(parts[1].trim()));
return person;
});
// Apply a schema to an RDD of JavaBeans to get a DataFrame
Dataset<Row> peopleDF = spark.createDataFrame(peopleRDD, Person.class);
// Register the DataFrame as a temporary view
peopleDF.createOrReplaceTempView("people");
// SQL statements can be run by using the sql methods provided by spark
Dataset<Row> teenagersDF = spark.sql("SELECT name FROM people WHERE age BETWEEN 13 AND 19");
// The columns of a row in the result can be accessed by field index
Encoder<String> stringEncoder = Encoders.STRING();
Dataset<String> teenagerNamesByIndexDF = teenagersDF.map(
(MapFunction<Row, String>) row -> "Name: " + row.getString(0),
stringEncoder);
teenagerNamesByIndexDF.show();
// +------------+
// | value|
// +------------+
// |Name: Justin|
// +------------+
// or by field name
Dataset<String> teenagerNamesByFieldDF = teenagersDF.map(
(MapFunction<Row, String>) row -> "Name: " + row.<String>getAs("name"),
stringEncoder);
teenagerNamesByFieldDF.show();
// +------------+
// | value|
// +------------+
// |Name: Justin|
// +------------+
Spark SQL can convert an RDD of Row objects to a DataFrame, inferring the datatypes. Rows are constructed by passing a list of key/value pairs as kwargs to the Row class. The keys of this list define the column names of the table, and the types are inferred by sampling the whole dataset, similar to the inference that is performed on JSON files.
from pyspark.sql import Row
sc = spark.sparkContext
# Load a text file and convert each line to a Row.
lines = sc.textFile("examples/src/main/resources/people.txt")
parts = lines.map(lambda l: l.split(","))
people = parts.map(lambda p: Row(name=p[0], age=int(p[1])))
# Infer the schema, and register the DataFrame as a table.
schemaPeople = spark.createDataFrame(people)
schemaPeople.createOrReplaceTempView("people")
# SQL can be run over DataFrames that have been registered as a table.
teenagers = spark.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
# The results of SQL queries are Dataframe objects.
# rdd returns the content as an :class:`pyspark.RDD` of :class:`Row`.
teenNames = teenagers.rdd.map(lambda p: "Name: " + p.name).collect()
for name in teenNames:
print(name)
# Name: Justin
Programmatically Specifying the Schema
When case classes cannot be defined ahead of time (for example,
the structure of records is encoded in a string, or a text dataset will be parsed
and fields will be projected differently for different users),
a DataFrame
can be created programmatically with three steps.
- Create an RDD of
Row
s from the original RDD; - Create the schema represented by a
StructType
matching the structure ofRow
s in the RDD created in Step 1. - Apply the schema to the RDD of
Row
s viacreateDataFrame
method provided bySparkSession
.
For example:
import org.apache.spark.sql.types._
// Create an RDD
val peopleRDD = spark.sparkContext.textFile("examples/src/main/resources/people.txt")
// The schema is encoded in a string
val schemaString = "name age"
// Generate the schema based on the string of schema
val fields = schemaString.split(" ")
.map(fieldName => StructField(fieldName, StringType, nullable = true))
val schema = StructType(fields)
// Convert records of the RDD (people) to Rows
val rowRDD = peopleRDD
.map(_.split(","))
.map(attributes => Row(attributes(0), attributes(1).trim))
// Apply the schema to the RDD
val peopleDF = spark.createDataFrame(rowRDD, schema)
// Creates a temporary view using the DataFrame
peopleDF.createOrReplaceTempView("people")
// SQL can be run over a temporary view created using DataFrames
val results = spark.sql("SELECT name FROM people")
// The results of SQL queries are DataFrames and support all the normal RDD operations
// The columns of a row in the result can be accessed by field index or by field name
results.map(attributes => "Name: " + attributes(0)).show()
// +-------------+
// | value|
// +-------------+
// |Name: Michael|
// | Name: Andy|
// | Name: Justin|
// +-------------+
When JavaBean classes cannot be defined ahead of time (for example,
the structure of records is encoded in a string, or a text dataset will be parsed and
fields will be projected differently for different users),
a Dataset<Row>
can be created programmatically with three steps.
- Create an RDD of
Row
s from the original RDD; - Create the schema represented by a
StructType
matching the structure ofRow
s in the RDD created in Step 1. - Apply the schema to the RDD of
Row
s viacreateDataFrame
method provided bySparkSession
.
For example:
import java.util.ArrayList;
import java.util.List;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
// Create an RDD
JavaRDD<String> peopleRDD = spark.sparkContext()
.textFile("examples/src/main/resources/people.txt", 1)
.toJavaRDD();
// The schema is encoded in a string
String schemaString = "name age";
// Generate the schema based on the string of schema
List<StructField> fields = new ArrayList<>();
for (String fieldName : schemaString.split(" ")) {
StructField field = DataTypes.createStructField(fieldName, DataTypes.StringType, true);
fields.add(field);
}
StructType schema = DataTypes.createStructType(fields);
// Convert records of the RDD (people) to Rows
JavaRDD<Row> rowRDD = peopleRDD.map((Function<String, Row>) record -> {
String[] attributes = record.split(",");
return RowFactory.create(attributes[0], attributes[1].trim());
});
// Apply the schema to the RDD
Dataset<Row> peopleDataFrame = spark.createDataFrame(rowRDD, schema);
// Creates a temporary view using the DataFrame
peopleDataFrame.createOrReplaceTempView("people");
// SQL can be run over a temporary view created using DataFrames
Dataset<Row> results = spark.sql("SELECT name FROM people");
// The results of SQL queries are DataFrames and support all the normal RDD operations
// The columns of a row in the result can be accessed by field index or by field name
Dataset<String> namesDS = results.map(
(MapFunction<Row, String>) row -> "Name: " + row.getString(0),
Encoders.STRING());
namesDS.show();
// +-------------+
// | value|
// +-------------+
// |Name: Michael|
// | Name: Andy|
// | Name: Justin|
// +-------------+
When a dictionary of kwargs cannot be defined ahead of time (for example,
the structure of records is encoded in a string, or a text dataset will be parsed and
fields will be projected differently for different users),
a DataFrame
can be created programmatically with three steps.
- Create an RDD of tuples or lists from the original RDD;
- Create the schema represented by a
StructType
matching the structure of tuples or lists in the RDD created in the step 1. - Apply the schema to the RDD via
createDataFrame
method provided bySparkSession
.
For example:
# Import data types
from pyspark.sql.types import *
sc = spark.sparkContext
# Load a text file and convert each line to a Row.
lines = sc.textFile("examples/src/main/resources/people.txt")
parts = lines.map(lambda l: l.split(","))
# Each line is converted to a tuple.
people = parts.map(lambda p: (p[0], p[1].strip()))
# The schema is encoded in a string.
schemaString = "name age"
fields = [StructField(field_name, StringType(), True) for field_name in schemaString.split()]
schema = StructType(fields)
# Apply the schema to the RDD.
schemaPeople = spark.createDataFrame(people, schema)
# Creates a temporary view using the DataFrame
schemaPeople.createOrReplaceTempView("people")
# SQL can be run over DataFrames that have been registered as a table.
results = spark.sql("SELECT name FROM people")
results.show()
# +-------+
# | name|
# +-------+
# |Michael|
# | Andy|
# | Justin|
# +-------+
Aggregations
The built-in DataFrames functions provide common
aggregations such as count()
, countDistinct()
, avg()
, max()
, min()
, etc.
While those functions are designed for DataFrames, Spark SQL also has type-safe versions for some of them in
Scala and
Java to work with strongly typed Datasets.
Moreover, users are not limited to the predefined aggregate functions and can create their own.
Untyped User-Defined Aggregate Functions
Users have to extend the UserDefinedAggregateFunction abstract class to implement a custom untyped aggregate function. For example, a user-defined average can look like:
import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.expressions.MutableAggregationBuffer
import org.apache.spark.sql.expressions.UserDefinedAggregateFunction
import org.apache.spark.sql.types._
object MyAverage extends UserDefinedAggregateFunction {
// Data types of input arguments of this aggregate function
def inputSchema: StructType = StructType(StructField("inputColumn", LongType) :: Nil)
// Data types of values in the aggregation buffer
def bufferSchema: StructType = {
StructType(StructField("sum", LongType) :: StructField("count", LongType) :: Nil)
}
// The data type of the returned value
def dataType: DataType = DoubleType
// Whether this function always returns the same output on the identical input
def deterministic: Boolean = true
// Initializes the given aggregation buffer. The buffer itself is a `Row` that in addition to
// standard methods like retrieving a value at an index (e.g., get(), getBoolean()), provides
// the opportunity to update its values. Note that arrays and maps inside the buffer are still
// immutable.
def initialize(buffer: MutableAggregationBuffer): Unit = {
buffer(0) = 0L
buffer(1) = 0L
}
// Updates the given aggregation buffer `buffer` with new input data from `input`
def update(buffer: MutableAggregationBuffer, input: Row): Unit = {
if (!input.isNullAt(0)) {
buffer(0) = buffer.getLong(0) + input.getLong(0)
buffer(1) = buffer.getLong(1) + 1
}
}
// Merges two aggregation buffers and stores the updated buffer values back to `buffer1`
def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {
buffer1(0) = buffer1.getLong(0) + buffer2.getLong(0)
buffer1(1) = buffer1.getLong(1) + buffer2.getLong(1)
}
// Calculates the final result
def evaluate(buffer: Row): Double = buffer.getLong(0).toDouble / buffer.getLong(1)
}
// Register the function to access it
spark.udf.register("myAverage", MyAverage)
val df = spark.read.json("examples/src/main/resources/employees.json")
df.createOrReplaceTempView("employees")
df.show()
// +-------+------+
// | name|salary|
// +-------+------+
// |Michael| 3000|
// | Andy| 4500|
// | Justin| 3500|
// | Berta| 4000|
// +-------+------+
val result = spark.sql("SELECT myAverage(salary) as average_salary FROM employees")
result.show()
// +--------------+
// |average_salary|
// +--------------+
// | 3750.0|
// +--------------+
import java.util.ArrayList;
import java.util.List;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.expressions.MutableAggregationBuffer;
import org.apache.spark.sql.expressions.UserDefinedAggregateFunction;
import org.apache.spark.sql.types.DataType;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
public static class MyAverage extends UserDefinedAggregateFunction {
private StructType inputSchema;
private StructType bufferSchema;
public MyAverage() {
List<StructField> inputFields = new ArrayList<>();
inputFields.add(DataTypes.createStructField("inputColumn", DataTypes.LongType, true));
inputSchema = DataTypes.createStructType(inputFields);
List<StructField> bufferFields = new ArrayList<>();
bufferFields.add(DataTypes.createStructField("sum", DataTypes.LongType, true));
bufferFields.add(DataTypes.createStructField("count", DataTypes.LongType, true));
bufferSchema = DataTypes.createStructType(bufferFields);
}
// Data types of input arguments of this aggregate function
public StructType inputSchema() {
return inputSchema;
}
// Data types of values in the aggregation buffer
public StructType bufferSchema() {
return bufferSchema;
}
// The data type of the returned value
public DataType dataType() {
return DataTypes.DoubleType;
}
// Whether this function always returns the same output on the identical input
public boolean deterministic() {
return true;
}
// Initializes the given aggregation buffer. The buffer itself is a `Row` that in addition to
// standard methods like retrieving a value at an index (e.g., get(), getBoolean()), provides
// the opportunity to update its values. Note that arrays and maps inside the buffer are still
// immutable.
public void initialize(MutableAggregationBuffer buffer) {
buffer.update(0, 0L);
buffer.update(1, 0L);
}
// Updates the given aggregation buffer `buffer` with new input data from `input`
public void update(MutableAggregationBuffer buffer, Row input) {
if (!input.isNullAt(0)) {
long updatedSum = buffer.getLong(0) + input.getLong(0);
long updatedCount = buffer.getLong(1) + 1;
buffer.update(0, updatedSum);
buffer.update(1, updatedCount);
}
}
// Merges two aggregation buffers and stores the updated buffer values back to `buffer1`
public void merge(MutableAggregationBuffer buffer1, Row buffer2) {
long mergedSum = buffer1.getLong(0) + buffer2.getLong(0);
long mergedCount = buffer1.getLong(1) + buffer2.getLong(1);
buffer1.update(0, mergedSum);
buffer1.update(1, mergedCount);
}
// Calculates the final result
public Double evaluate(Row buffer) {
return ((double) buffer.getLong(0)) / buffer.getLong(1);
}
}
// Register the function to access it
spark.udf().register("myAverage", new MyAverage());
Dataset<Row> df = spark.read().json("examples/src/main/resources/employees.json");
df.createOrReplaceTempView("employees");
df.show();
// +-------+------+
// | name|salary|
// +-------+------+
// |Michael| 3000|
// | Andy| 4500|
// | Justin| 3500|
// | Berta| 4000|
// +-------+------+
Dataset<Row> result = spark.sql("SELECT myAverage(salary) as average_salary FROM employees");
result.show();
// +--------------+
// |average_salary|
// +--------------+
// | 3750.0|
// +--------------+
Type-Safe User-Defined Aggregate Functions
User-defined aggregations for strongly typed Datasets revolve around the Aggregator abstract class. For example, a type-safe user-defined average can look like:
import org.apache.spark.sql.{Encoder, Encoders, SparkSession}
import org.apache.spark.sql.expressions.Aggregator
case class Employee(name: String, salary: Long)
case class Average(var sum: Long, var count: Long)
object MyAverage extends Aggregator[Employee, Average, Double] {
// A zero value for this aggregation. Should satisfy the property that any b + zero = b
def zero: Average = Average(0L, 0L)
// Combine two values to produce a new value. For performance, the function may modify `buffer`
// and return it instead of constructing a new object
def reduce(buffer: Average, employee: Employee): Average = {
buffer.sum += employee.salary
buffer.count += 1
buffer
}
// Merge two intermediate values
def merge(b1: Average, b2: Average): Average = {
b1.sum += b2.sum
b1.count += b2.count
b1
}
// Transform the output of the reduction
def finish(reduction: Average): Double = reduction.sum.toDouble / reduction.count
// Specifies the Encoder for the intermediate value type
def bufferEncoder: Encoder[Average] = Encoders.product
// Specifies the Encoder for the final output value type
def outputEncoder: Encoder[Double] = Encoders.scalaDouble
}
val ds = spark.read.json("examples/src/main/resources/employees.json").as[Employee]
ds.show()
// +-------+------+
// | name|salary|
// +-------+------+
// |Michael| 3000|
// | Andy| 4500|
// | Justin| 3500|
// | Berta| 4000|
// +-------+------+
// Convert the function to a `TypedColumn` and give it a name
val averageSalary = MyAverage.toColumn.name("average_salary")
val result = ds.select(averageSalary)
result.show()
// +--------------+
// |average_salary|
// +--------------+
// | 3750.0|
// +--------------+
import java.io.Serializable;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Encoder;
import org.apache.spark.sql.Encoders;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.TypedColumn;
import org.apache.spark.sql.expressions.Aggregator;
public static class Employee implements Serializable {
private String name;
private long salary;
// Constructors, getters, setters...
}
public static class Average implements Serializable {
private long sum;
private long count;
// Constructors, getters, setters...
}
public static class MyAverage extends Aggregator<Employee, Average, Double> {
// A zero value for this aggregation. Should satisfy the property that any b + zero = b
public Average zero() {
return new Average(0L, 0L);
}
// Combine two values to produce a new value. For performance, the function may modify `buffer`
// and return it instead of constructing a new object
public Average reduce(Average buffer, Employee employee) {
long newSum = buffer.getSum() + employee.getSalary();
long newCount = buffer.getCount() + 1;
buffer.setSum(newSum);
buffer.setCount(newCount);
return buffer;
}
// Merge two intermediate values
public Average merge(Average b1, Average b2) {
long mergedSum = b1.getSum() + b2.getSum();
long mergedCount = b1.getCount() + b2.getCount();
b1.setSum(mergedSum);
b1.setCount(mergedCount);
return b1;
}
// Transform the output of the reduction
public Double finish(Average reduction) {
return ((double) reduction.getSum()) / reduction.getCount();
}
// Specifies the Encoder for the intermediate value type
public Encoder<Average> bufferEncoder() {
return Encoders.bean(Average.class);
}
// Specifies the Encoder for the final output value type
public Encoder<Double> outputEncoder() {
return Encoders.DOUBLE();
}
}
Encoder<Employee> employeeEncoder = Encoders.bean(Employee.class);
String path = "examples/src/main/resources/employees.json";
Dataset<Employee> ds = spark.read().json(path).as(employeeEncoder);
ds.show();
// +-------+------+
// | name|salary|
// +-------+------+
// |Michael| 3000|
// | Andy| 4500|
// | Justin| 3500|
// | Berta| 4000|
// +-------+------+
MyAverage myAverage = new MyAverage();
// Convert the function to a `TypedColumn` and give it a name
TypedColumn<Employee, Double> averageSalary = myAverage.toColumn().name("average_salary");
Dataset<Double> result = ds.select(averageSalary);
result.show();
// +--------------+
// |average_salary|
// +--------------+
// | 3750.0|
// +--------------+


| TreeNT Unit
Classes Types Const |
| Classes |
| ETreeNTError |
| TCustomTreeNT |
| TTreeNodeList |
| TTreeNT |
| TTreeNTNode |
| TTreeNTNodes |
| Types |
| Global Constants |
| ckCheckChecked |
| ckCheckDisabled |
| ckCheckEmpty |
| ckCheckGrayed |
| ckEmpty |
| ckRadioChecked |
| ckRadioDisabled |
| ckRadioEmpty |
| ckRadioGrayed |
| DefaultOptions |


Class
FaiIndexData structure for access to FAI indices.
Data structure for access to FAI indices.
| Defined in | <seqan/seq_io.h> |
|---|---|
| Signature |
class FaiIndex;
|
Member Function Overview
-
FaiIndex::FaiIndex();
Constructor.
Interface Function Overview
-
bool build(faiIndex, seqFileName[, faiFileName]);
Create a FaiIndex from FASTA file. -
void clear(faiIndex);
Reset a FaiIndex object to the state after default construction. -
bool getIdByName(rID, faiIndex, name);
Return reference ID (numeric index in the file) of a sequence in a FAI file. -
__uint64 numSeqs(faiIndex);
Return the number of sequences known to a FaiIndex. -
bool open(faiIndex, fastaFilename [, faiFileName]);
Open a FaiIndex object. -
void readRecord(faiIndex, fastaFileName[, faiFileName]);
Read a FAI index from file. -
void readRegion(str, faiIndex, rID, beginPos, endPos);, void readRegion(str, faiIndex, region);
Read a region through an FaiIndex. -
void readSequence(str, faiIndex, rID);
Load a whole sequence from a FaiIndex. -
bool save(faiIndex[, faiFileName]);
Save a FaiIndex object. -
__uint64 sequenceLength(faiIndex, rID);
Return length of the sequence with the given id in the FaiIndex. -
CharString sequenceName(faiIndex, rID);
Return the name of the sequence with the given id in the FaiIndex.
Detailed Description
FAI indices allow the rast random access to sequences or parts of sequences in a FASTA file. Originally, they were introduced in the samtools program.
Also see the Indexed FASTA I/O Tutorial.
Example
The following example demonstrates the usage of the FaiIndex class.
#include <seqan/basic.h> #include <seqan/seq_io.h> #include <seqan/sequence.h> using namespace seqan; int main() { CharString path = getAbsolutePath("/demos/dox/seq_io/example.fa"); FaiIndex faiIndex; // Try to read the FAI index. if (!open(faiIndex, toCString(path))) { std::cerr << "Could not read the FAI index. Not fatal, we can just build it.\n"; return 1; } // Try to build the FAI index (in memory) if reading was unsuccessful. If // building into memory succeeded, we try to write it out. if (!build(faiIndex, toCString(path))) { std::cerr << "FATAL: Could not build FAI index.\n"; return 1; } if (!save(faiIndex)) { std::cerr << "FATAL: Could not write out FAI index after building.\n"; return 1; } // Now, read the first 1000 characters of chr1. unsigned idx = 0; if (!getIdByName(idx, faiIndex, "chr")) { std::cerr << "FATAL: chr1 not found in FAI index.\n"; return 1; } CharString seq; readRegion(seq, faiIndex, idx, 0, 100); // Now print the first 100 characters we just read. std::cout << "chr:1-100 = " << seq << "\n"; return 0; }
The output is as follows:
chr:1-100 = CCTATCTAATAATATACCTTATACTGGACTAGTGCCAATATTAAAATGAAGTGGGCGTAGTGTGTAATTTGATTGGGTGGAGGTGTGGCTTTGGCGTGCT
Member Functions Detail
FaiIndex::FaiIndex();
Constructor.
Data Races
Thread safety unknown!
Interface Functions Detail
bool build(faiIndex, seqFileName[, faiFileName]);
Create a FaiIndex from FASTA file.
Parameters
faiIndex
|
The FaiIndex to build into. |
|---|---|
seqFileName
|
Path to the FASTA file to build an index for. Type: char const *. |
faiFileName
|
Path to the FAI file to use as the index file. Type: char const *. Default: "${seqFileName}.fai". |
Returns
bool |
true on success, false otherwise. |
|---|
Data Races
Thread safety unknown!
void clear(faiIndex);
Reset a FaiIndex object to the state after default construction.
Parameters
faiIndex
|
The FaiIndex to clear. |
|---|
Data Races
Thread safety unknown!
bool getIdByName(rID, faiIndex, name);
Return reference ID (numeric index in the file) of a sequence in a FAI file.
Parameters
faiIndex
|
The FaiIndex to query. |
|---|---|
name
|
The name of the sequence to look the id up for. Type: ContainerConcept. |
rID
|
The id of the sequence is written here. |
Returns
bool |
true if a sequence with the given name is known in the index, false otherwise. |
|---|
Data Races
Thread safety unknown!
__uint64 numSeqs(faiIndex);
Return the number of sequences known to a FaiIndex.
Parameters
faiIndex
|
The FaiIndex to query. |
|---|
Returns
__uint64 |
The number of sequences in the index. |
|---|
Data Races
Thread safety unknown!
bool open(faiIndex, fastaFilename [, faiFileName]);
Open a FaiIndex object.
Parameters
faiIndex
|
The FaiIndex to write out. |
|---|---|
fastaFilename
|
Path to the FASTA file to build an index for. Type: char const *. |
faiFileName
|
The name of the FAI file to open. This parameter is optional. By default, the FAI file name is derived from the FASTA file name. Type: char const *. |
Returns
bool |
true on success, false otherwise. |
|---|
Data Races
Thread safety unknown!
void readRecord(faiIndex, fastaFileName[, faiFileName]);
Read a FAI index from file.
Parameters
faiIndex
|
The FaiIndex to read into. |
|---|---|
fastaFileName
|
Path to the FASTA file to read. Type: char const *. |
faiFileName
|
Path to the FAI file to read. Type: char const *. Defaults to "${fastaFileName}.fai". |
Data Races
Thread safety unknown!
void readRegion(str, faiIndex, rID, beginPos, endPos);
void readRegion(str, faiIndex, region);
Read a region through an FaiIndex.
Parameters
str
|
The String to read the sequence into. |
|---|---|
faiIndex
|
The FaiIndex to read from. |
rID
|
The id of the sequence to read (Type: unsigned). |
beginPos
|
The begin position of the region to read (Type: unsigned). |
endPos
|
The end position of the region to read (Type: unsigned). |
region
|
The GenomicRegion to read. |
Data Races
Thread safety unknown!
void readSequence(str, faiIndex, rID);
Load a whole sequence from a FaiIndex.
Parameters
str
|
The String to read into. |
|---|---|
faiIndex
|
The FaiIndex to read from. |
seqID
|
The index of the sequence in the file. |
Data Races
Thread safety unknown!
bool save(faiIndex[, faiFileName]);
Save a FaiIndex object.
Parameters
faiIndex
|
The FaiIndex to write out. |
|---|---|
faiFileName
|
The name of the FAI file to write to. This parameter is optional only if the FAI index knows the FAI file name from a previous build call. By default, the FAI file name from the previous call to build is used. Type: char const *. |
Returns
bool |
true on success, false otherwise. |
|---|
Data Races
Thread safety unknown!
__uint64 sequenceLength(faiIndex, rID);
Return length of the sequence with the given id in the FaiIndex.
Parameters
faiIndex
|
The FaiIndex to query. |
|---|---|
rID
|
The id of the sequence to get the length of. |
Returns
__uint64 |
The length of the sequence with index rID in faiIndex. |
|---|
Data Races
Thread safety unknown!
CharString sequenceName(faiIndex, rID);
Return the name of the sequence with the given id in the FaiIndex.
Parameters
faiIndex
|
The FaiIndex to query. |
|---|---|
rID
|
The index of the sequence. |
Returns
CharString |
The name of the sequence with the given id. |
|---|
Data Races
Thread safety unknown!


An exporter for `glTF` 2.0.
[link:https://www.khronos.org/gltf glTF] (GL Transmission Format) is an
[link:https://github.com/KhronosGroup/glTF/tree/master/specification/2.0 open format specification]
for efficient delivery and loading of 3D content. Assets may be provided either in JSON (.gltf)
or binary (.glb) format. External files store textures (.jpg, .png) and additional binary
data (.bin). A glTF asset may deliver one or more scenes, including meshes, materials,
textures, skins, skeletons, morph targets, animations, lights, and/or cameras.
[name] is an add-on, and must be imported explicitly. See [link:#manual/introduction/Installation Installation / Addons].
import { GLTFExporter } from 'three/addons/exporters/GLTFExporter.js';
GLTFExporter supports the following [link:https://github.com/KhronosGroup/glTF/tree/master/extensions/ glTF 2.0 extensions]:
The following glTF 2.0 extension is supported by an external user plugin
// Instantiate a exporter
const exporter = new GLTFExporter();
// Parse the input and generate the glTF output
exporter.parse(
scene,
// called when the gltf has been generated
function ( gltf ) {
console.log( gltf );
downloadJSON( gltf );
},
// called when there is an error in the generation
function ( error ) {
console.log( 'An error happened' );
},
options
);
[example:misc_exporter_gltf]
Creates a new [name].
[page:Object input] — Scenes or objects to export. Valid options:
exporter.parse( scene1, ... )
exporter.parse( [ scene1, scene2 ], ... )
exporter.parse( object1, ... )
exporter.parse( [ object1, object2 ], ... )
exporter.parse( [ scene1, object1, object2, scene2 ], ... )
Generates a .gltf (JSON) or .glb (binary) output from the input (Scenes or Objects)
[link:https://github.com/mrdoob/three.js/blob/master/examples/jsm/exporters/GLTFExporter.js examples/jsm/exporters/GLTFExporter.js]


Colored list view provides the list that has gradient background color.
You should check if your browser supports WebGL since colored list view uses the WebGL technology.
By default, all

and

elements with the class="ui-colored-list"
or data-role="coloredlistview"
attribute are displayed as colored list view components.
To manually create a colored list view component, use the component constructor from the tau
namespace:
The constructor requires an HTMLElement
parameter to create the component, and you can get it with the document.getElementById()
method. The constructor can also take a second parameter, which is an object defining the configuration options for the component.

Item

Item

Item

| Option | Input type | Default value | Description |
|---|---|---|---|
| data-colored-listview-number | number | 15 | The number of list items to be applied colored list view. |


Home Page
Cycle 2
What I did
For cycle 2, we are doing CAD and Fusion 360. On the first day of class, we learned what all that meant.
On the second day, we played around on Fusion 360 and made some squares and extruded some shapes.
On the third day, we spent the whole class making 6 designs of 50 as a practice.
These were all the designs I made. It took a couple of them to figure out what I was doing,
but I eventually got it and made swift work of the remaining designs. Next task is to make
a piece of furniture, so I am making a drawer with a hidden compartment. It is turning out
well so far.
Making these designs did not warrant many failures. The practice drawings were difficult, but
never had any failures. The only failure I had was being unable to figure out how to download
the file for my drawer to put on this page.


Sahi - Web Automation and Test Tool
Documentation in this directory is not current.
Please visit http://sahipro.com/ for the latest documentation.
Editor Support
Since Sahi script is based on javascript, it is easy to provide text editor/IDE support.Textpad
Download sahi_textpad.zipUnzip the file.
Install sahi.syn file as given here.
Copy sahi.TCL to C:\Documents and Settings\<UserName>\Application Data\TextPad
Restart textpad.
jEdit
Download Sahi edit mode.Installation:
1. Copy sahi.xml to the /modes directory in the jEdit root folder
2. Add the following XML node to the file "catalog" in the same directory
<MODE NAME="sahi" FILE="sahi.xml" FILE_NAME_GLOB="*.sah" />
Thanks to Anders Lund Haheim for this contribution.


We extend dogs!
- visualize killer systems
- syndicate user-centric niches
- evolve mission-critical platforms
- matrix dot-com infomediaries
- incentivize customized content
- integrate 24/7 portals
- synthesize user-centric ROI
- grow revolutionary web-readiness
- leverage leading-edge platforms
- streamline global metrics
- extend clicks-and-mortar initiatives
- utilize integrated eyeballs
- exploit frictionless markets


{% include "header.html" %} Contact Me Have questions? I have answers.
Want to get in touch? Fill out the form below to send me a message and I will get back to you as soon as possible!


#include <alert.h>
|
|
| IMPLEMENT_SERIALIZE (READWRITE(vchMsg);READWRITE(vchSig);) void SetNull() |
| |
|
bool | IsNull () const |
| |
|
uint256 | GetHash () const |
| |
|
bool | IsInEffect () const |
| |
|
bool | Cancels (const CAlert &alert) const |
| |
|
bool | AppliesTo (int nVersion, std::string strSubVerIn) const |
| |
|
bool | AppliesToMe () const |
| |
|
bool | RelayTo (CNode *pnode) const |
| |
|
bool | CheckSignature () const |
| |
|
bool | ProcessAlert (bool fThread=true) |
| |
|
| IMPLEMENT_SERIALIZE (READWRITE(this->nVersion);nVersion=this->nVersion;READWRITE(nRelayUntil);READWRITE(nExpiration);READWRITE(nID);READWRITE(nCancel);READWRITE(setCancel);READWRITE(nMinVer);READWRITE(nMaxVer);READWRITE(setSubVer);READWRITE(nPriority);READWRITE(strComment);READWRITE(strStatusBar);READWRITE(strReserved);) void SetNull() |
| |
|
std::string | ToString () const |
| |
|
void | print () const |
| |
|
|
std::vector< unsigned char > | vchMsg |
| |
|
std::vector< unsigned char > | vchSig |
| |
|
int | nVersion |
| |
|
int64_t | nRelayUntil |
| |
|
int64_t | nExpiration |
| |
|
int | nID |
| |
|
int | nCancel |
| |
|
std::set< int > | setCancel |
| |
|
int | nMinVer |
| |
|
int | nMaxVer |
| |
|
std::set< std::string > | setSubVer |
| |
|
int | nPriority |
| |
|
std::string | strComment |
| |
|
std::string | strStatusBar |
| |
|
std::string | strReserved |
| |
An alert is a combination of a serialized CUnsignedAlert and a signature.
The documentation for this class was generated from the following files:


Shehnaaz Gill blamed for ‘breaking’ Siddharth Shukla’s friendship with Asim Riaz, angry fans vent ire on social media
by JKR StaffAsim Riaz and Siddharth Shukla have become the embodiment of true friendship in this year’s Bigg Boss house even though the two did not know each other before entering the show. Both have stood by each other through thick and thin since the start of the reality show. Their bonding has been a topic of intense social media conversations. But, if the promo broadcast by Bigg Boss on Monday night is anything to go by, their famous friendship has ended, prompting their fans on social media to vent out their frustration.
In the promo, Asim and Siddharth are seen engaged in a heated argument with both raising their voices.
While, it’s not clear what triggered the fight between them, their fans have already begun making a desperate plea to the makers of the reality show to ensure that their friendship remains intact.
What appears to have incensed the fans of Siddharth and Asim was the Monday night’s development, when the former was seen mending fences with Shehnaaz Gill. Many felt that Shehnaaz had finally succeeded in creating a rift between Siddharth and Asim.


New to site? Create Account
or
Already a member ? Log in
©2016 All Rights Reserved. Gentelella Alela! is a Bootstrap 3 template. Privacy and Terms


The inputs below this paragraph programmatically enable/disable the "form elements" Tab. When a Tab becomes disabled, its corresponding tab panel is checked for any/all IDS Enterprise controls, which are also programmatically enabled/disabled.
This functionality is different from the Enabling/Disabling the Entire Control, which is also possible in the Tabs API.
Bricks-and-clicks? Evolve ubiquitous matrix B2B 24/365 vertical 24/365 platforms standards-compliant global leverage dynamic 24/365 intuitive ROI seamless rss-capable. Cutting-edge grow morph web services leverage; ROI, unleash reinvent innovative podcasts citizen-media networking.
Frictionless webservices, killer open-source innovate, best-of-breed, whiteboard interactive back-end optimize capture dynamic front-end. Initiatives ubiquitous 24/7 enhance channels B2B drive frictionless web-readiness generate recontextualize widgets applications. Sexy sticky matrix, user-centred, rich user-centric: peer-to-peer podcasting networking addelivery optimize streamline integrated proactive: granular morph.
Exploit niches; enable A-list web-enabled holistic end-to-end. Exploit experiences value-added tagclouds, open-source cross-platform e-tailers, user-contributed, implement! Convergence, solutions front-end, "synergize markets initiatives integrateAJAX-enabled platforms; wireless, supply-chains reinvent, mindshare, synergies implement, drive evolve!" Post incentivize; rich-clientAPIs customized revolutionize 24/365 killer incentivize integrate intuitive utilize!


Version 1.3
Copyright © 2005 by Jorge Cwik.
http://pasti.fxatari.com
This documented may be freely distributed as long as is not modified in anyway.
Chapter I:†††††† Introduction
Chapter II:†††† Technical
description
Chapter III:††† Prefetch
tables
Most CPUs, including many old 8-bit ones, read from the instruction stream ahead of execution. This mechanism is usually known as prefetch, queue or pipeline. The main goal is to optimize resources and gain in performance thanks to concurrent execution.
Modern processors implement this very aggressively and attempt to fill the prefetch queue as much and as early as possible. A modern CPU can partially process one instruction while other is executing, or even actually execute multiple instructions simultaneously (superscaling or multicore).
The 68000 is not as powerful, it is quite an old CPU. It canít execute two instructions at the same time. But it can read and decode the next instruction while the current one is being executed.
The prefetch queue in the 68000 is a simple FIFO (first in, first out) with space for two words (four bytes). This means that before any instruction starts execution, its opcode word and the next one were already read. The second word, depending on the instruction, could be an extension word for the current instruction, or the opcode for the next one.
A modern CPU usually has more or less independent separate units. One unit processes actual CPU operations such as arithmetic or logic operations, while other unit fills the prefetch queue. This way the prefetch unit can attempt to read from the instruction stream whenever the bus is free and there is still enough space in the queue. And the CPU can keep its internal processing as long as operations are pending and data is available. So one unit normally doesnít need to wait for the other one. If the prefetch unit is delayed because external RAM (or ROM) is not fast enough, this wonít necessary delay internal execution.
It is important to understand that the 68000 prefetch is rather simple, and not as complex as in a modern CPU. All bus accesses, including prefetch ones, are fixed in microcode. Different instructions might perform the prefetch access differently than other instructions. But for a specific opcode, prefetchs are always executed exactly at the same cycles. The actual absolute time depends on the external system. And if any bus access (prefetch or not) is delayed because of wait states, then the whole CPU process is stalled.
The prefetch queue is re-filled during the execution of an instruction. Most instructions perform as many prefetch cycles as the total number of words in the instruction. For example NOP, the simplest instruction, it performs one prefetch at the address two words (four bytes) above the location of the NOP. MOVE $10000,$20000, one possible form of the longest instruction, performs five prefetch cycles.
The exceptions are instructions that modify the instruction stream, such as branches. They sometimes need to perform extra prefetch cycles from the new modified program counter.
The prefetch queue results in faster execution due to optimization of CPU resources. On one side bus utilization is optimized. Many instructions have cycles that donít need to access the external bus. The extreme cases are slow instructions like MUL, DIV and SHIFTS. While the CPU is busy with its internal processing, it can perform prefetch cycles for the next instruction without affecting the current instruction in any way.
But the most important gain is related to concurrent execution. As described earlier the 68000 can decode the next opcode and fetch one further word, all of this concurrently with the execution of the current instruction. Actually, it is the concurrency that allows the bus optimization.
Letís see what would happen without prefetch queue when executing, i.e., a simple NOP:
Before anything else the NOP opcode needs to be fetched. Every 68000 bus access requires a minimum of four machine cycles. So nothing could be done for four full cycles. Then the opcode needs to be decoded. This would require another two cycles. And finally the instruction would be executed. In this case it would be just to update the PC and would take another two cycles (every 68000 internal processing requires at least two machine cycles). Note that no step could be performed before the previous one is finished.
So a NOP would have taken 8 cycles, twice as much as it really requires thanks to the prefetch.
The prefetch has some subtle effects that sometimes are not obvious. In first place it means that the precise time when instructions are fetched is affected. This might have some consequences if hardware I/O access is involved.
In second place it means that code might be fetched beyond what is really intended. Letís suppose you have a routine located at the very end of some memory space (end of RAM, end of ROM, etc). Letís suppose i.e., that this routine ends with an RTS instruction at the last word of that memory block. When the CPU executes the RTS, and before returning to the caller, it performed already a prefetch one word beyond the RTS. What exactly will happen when attempting a fetch outside the memory block depends on the system. Most of the time it will be harmless, but not always. For example, in an Atari ST with 4 MB of RAM, this will trigger a bus error.
But probably the most important effect of the prefetch is on self-modified code Ö
The most important effect of the prefetch is self modified code. This is a term describing code that modifies its own instruction stream. This is usually done for performance reasons, or for protections and anti-hacking techniques.
If the modified code is written far away enough, then the effect of the prefetch is not relevant. Only when code close enough to the current execution is written, then the exact behavior of the prefetch is significant.
Letís assume the following code:
†††† moveq
††††#0,D0
lea ††††††wcode,A0
move.l††† #$4E714E71,(A0)† ; NOP opcodes
wcode:
addq††††† #1,D0††††††††††† ; written here
addq††††† #1,D0††††††††††† ; and here
What is the result in D0, zero, one or two? Obviously it depends on which instructions were executed, the ďoldĒ ADDQ ones or the ďnewĒ NOP ones. And this in turn this depends on the exact behavior of the prefetch.
Weíll return later to this example after a more detailed description of the prefetch queue.
There is a lot of misconception about the structure of the 68000 prefetch. Probably one of the reasons is that different instructions produce different prefetch behaviors on self-modified code. Another reason is that the official literature barely covers the prefetch. And in the few places that it is mentioned, it is misleading.
The 68000 prefetch is exactly the same as the 68010 one. The official appendix on the MC68010 Loop Mode Operation says: ďÖ the MC68010/MC68012 has a two word prefetch queue in addition to a one word instruction decode register, ÖĒ
The above quote is absolutely valid for the 68000 as well. The only difference is that the 68010 can perform a special use of the prefetch queue for loop mode, while the 68000 does not.
There are three internal word registers involved in the prefetch:
The IRC and IR registers are the actual prefetch queue. The IRD is the only register that is somehow seen externally at the programming model. And because of this reason the IRD register is sometimes referred simply as IR. You canít access this register directly, but its content is the one that is stored in the exception frame on a bus or address error.
Any instruction opcode comes from external memory; traverse the prefetch queue, finally reaching the IRD for actual execution. Extension words might enter the queue at IRC but they never reach the decoder.
When the CPU starts processing after reset, the prefetch is initialized with the following steps:
1) Loads IRC from external memory.
2) Transfer IRC to IR.
3) Reload IRC from external memory.
4) Transfer IR to IRD.
Normal instructions donít perform the first step. For example, NOP performs the rest of the steps (step 2 to step 4). Actually all that NOP does is these prefetch steps plus incrementing the PC. All the instructions, disregarding how complex they are, perform the same prefetch processing (steps 2 to 4) towards the end of their execution.
These steps are the ones that enforce the basic prefetch rule: just before the start of any instruction two words (no more and no less) are already fetched. One word will be in IRD and another one in IRC.
For simple opcodes without extension words, the whole code flow simply traverses the queue as described above. The prefetch flow is more complicated when extension words are involved. Letís make a brief analysis of the prefetch flow on complex instructions† (opcodes with at least one extension word).
When a complex instruction starts, its first extension word is already in IRC. During the instruction processing more fetch cycles are performed. Unless the instruction performs a branch, the number of fetch cycles is always exactly the same as the total number of words for the instruction. This is usually done in two separate stages.
During the first stage extra fetch cycles are performed, one for each extension word. Most of the extension words are processed directly by the instruction and not transferred to the prefetch queue. Otherwise extension words would reach the decoder. Only the last word fetched at this stage is transferred to IRC. This last word is actually not part of the current instruction, it is instead the opcode of the next one. Remember that one extension word was already fetched during the previous instruction.
The second stage is the same as in simple opcodes. Steps 2 to 4 are performed. So the last prefetch done at the first stage reaches IRD. And a new word is fetched into IRC.
Letís look at an example:
4E71†††††††††† nop
203C 1234 5678 move.l†† #$12345678,D0
5280†††††††††† addq.l†† #1,D0
5380†††††††††† subq.l†† #1,D0
Instruction border between the NOP and the MOVE:
IRC: $1234
IRD: $203C
Stage 1 of MOVE execution:
Fetch $5678 and uses it internally together with $1234 already fetched.
Fetch $5280, transfer to IRC.
IRC: $5280
IRD: $203C
Stage 2 of MOVE execution:
Transfer IRC to IR.
Fetch $5380 into IRC.
Transfer IR to IRD.
IRC: $5380
IRD: $5280
Starts ADDQ execution.
So why some self-modified code seems to indicate that the prefetch queue is just one word, or even variable? Because the exact effect of the prefetch on self modified code depends, not only on the size of the queue, but also on the exact order of prefetch and write bus access cycles. Different instructions perform the prefetch cycles at different points in relation to the write data cycles.
The MOVE instruction performs a total of 5 bus cycles. Three prefetch cycles plus two write cycles. But in which order? Obviously at least one prefetch cycle must be performed first to complete the immediate long word (the opcode and the high word of the immediate were already prefetched at the previous instruction). But thatís all we can assume. We canít simply deduce the order of the rest of the prefetch cycles in relation to the write cycles. Any order would be correct and it wouldnít really matter if it werenít for special situations like self modified code.
If the MOVE instruction performs all the prefetch cycles before any write, then the ďoldĒ content of the two last instructions will be executed and the result will be two. If two prefetch cycles are performed after both write cycles, then the ďnewĒ NOPs will be executed and the result will be zero. And the writes and the last prefetch could be interleaved in such a way that the result will be one.
So what is the real result? In this specific case the result will be one. But the actual behavior depends and varies on each opcode. If similar code is rewritten using an instruction different than MOVE (say, EOR), then the result will be different. It might also be different depending on the destination addressing mode.
Weíll classify instructions into 3 different types depending on how the prefetch is affected:
Class 0: All prefetch cycles are performed before all the write cycles. Then the CPU will execute the next two following words as they were before the write. We can include in this category, if you want, all instructions that donít write to memory. If the above example would be rewritten with a class 0 instruction, then the result in D0 would be two.
Class 1: The last prefeth cycle is performed after the write cycles. Then the current instruction can modify one word of the prefetch queue before it is read. That word will be two instructions ahead, or one extension word of the next instruction. In the above example the result in D0 would be one.
Class 2: Two prefetch cycles are performed after the write cycles. Then any words overwritten by the current instruction will be prefetched with the new modified content. In other words, the next instruction would execute as if no prefetch feature would exist. This category is the only one that can modify the actual opcode of the next instruction. In the above example the result in D0 would be zero.
The names/numbers were choose because they reflect (figuratively) how they affect the prefetch. Class 0 instructions donít affect the prefetch, class 1 ones affect one word of the prefetch, and class 2 ones affect both words. To be precise, the class number reflects how many words on the instruction stream can be modified before they are prefetched by the current instruction.
So weíll attempt below to describe the exact behavior for all the instructions that write to memory. Weíll show the exact bus access order, and then the effect of the prefetch can be easily predicted. The order is detailed as much as is relevant for the prefetch behavior. There are subtle details that are not covered in this article (for example, the order of reading or writing the two halves of a long word memory access).
Read modify instructions
We use this term to classify instructions that read a memory operand, modify it internally and write it back to the same memory location. It includes:
ADD/SUB/OR/AND/EOR/NEG/NOT/NEGX
plus CLR and MOVE from SR.
Note that both CLR and MOVE from SR (which shouldnít be read/modify instructions) are un-optimized in the 68000. The memory operand is read before it is written. Also donít confuse read/modify instructions with read/modify cycles as performed by the TAS instruction.
Exact bus access execution order:
1) Perform as many prefetch cycles as extension words are in the memory addressing mode. Obviouslly optional if there are none.
2) Read memory operand. One bus cycle if byte or word operand, two if long.
3) Peform the last prefetch cycle. Concurrently the logical/arithmetic operation is done internally.
4) Write bus cycle(s).
So the net effect is these instructions are of class 0.
MOVE instructions. Most variants, except the ones noted below
1) Perform as many prefetch cycles as extension words are in the source operand (optional).
2) Read source operand (optional if source is register).
3) Perform as many prefetch cycles as extension words are in the destination operand (optional).
4) Writes memory operand.
5) Perform last prefetch cycle.
Net effect is that MOVE instructions are of class 1.
MOVE <ea>,-(An)
When the destination addressing mode is pre-decrement, steps 4 and 5 above are inverted. So it behaves like a read modify instruction and it is a class 0 instruction.
Note: The behavior is the same disregarding transfer size (byte, word or long), and disregarding the source addressing mode.
MOVE memory,(xxx).L
When the destination addressing mode is long absolute and the source operand is any memory addr.mode, step 4 is interleaved in the middle of step 3. Step 3 only performs a single prefetch in this case. The other prefetch cycle that is normally performed at that step is deferred after the write cycles.
So, two prefetch cycles are performed after the write ones. It is a class 2 instruction.
Note: The behavior is the same disregarding transfer size (byte, word or long). But if the source operand is a data or address register, or immediate, then the behavior is the same as other MOVE variants (class 1 instruction).
EXCEPTIONS
All exceptions, including interrupts, bus and address errors, traps, etc. We donít discuss here the whole process of writing the exception frame, which is very interesting and will be elaborated in another article. We only cover here the prefetch effect on exceptions
As it might seem natural, the prefetch is reloaded at the end of the exception processing. At that point the whole exception frame was already written. So if the supervisor stack pointer is pointing to the exception handler, two words of the prefetch are affected.
Furthermore, the exception frame is wholly written before the vector itself is fetched. So if the SSP is pointing at the exception table, it might overwrite the exception vector itself before it is being fetched.
1) Write exception frame (detailed in another article)
2) Fetch exception vector
3) Perform two prefetch cycles.
So it is a class 2+ behavior.
BSR
1) Stack is written
2) Two prefetch cycles are performed.
Class 2 instruction.
JSR
1) When addr. mode is absolute long, prefetch the last word of the absolute address. Step not needed for other modes.
2) Perform one prefetch cycle
3) Write stack
4) Perform last prefetch cycle.
Class 1 instruction.
PEA for absolute short and absolute long addr. modes.
1) One (absolute short) or two (absolute long) prefetch cycles.
2) Write cycles.
3) Last prefetch cycle.
Class 1 instruction.
PEA, rest addr. modes.
1) Perform prefetch cycles according to addr. mode.
2) Perform last prefetch cycle.
3) Write cycles.
Class 0 instruction.
LINK
1) One prefetch cycle
2) Write cycles
3) One prefetch cycle
Class 1 instruction.
BCHG/BCLR/BSET
ASL/ASR/LSL/LSR/ROL/ROR/ROXL/ROXR
TAS
1) Prefetch cycles, as many as extension words.
2) Read operand.
3) Last prefetch cycle.
4) Write operand.
Class 0 instruction.
Scc
1) Prefetch cycles, as many as extension words.
2) Last prefetch cycle.
3) Write operand.
Class 0 instruction.
ADDX,SUBX
ABCD,SBCD
1) Read source operand.
2) Read destination operand.
3) Prefetch cycle.
4) Write operand.
Class 0 instruction.
MOVEP
1) One prefetch cycle for the extension word
2) Write operand
3) Last prefetch cycle
Class 1 instruction.
MOVEM
1) Prefetch cycles, as many as extension words.
2) Write registers to memory
3) Perform last prefetch
Class 1 instruction.


2018-11-25: How to batch export from Ulysses to any (Markdown) app (for simple notes without images) This is a small guide to help moving files from...
2018-11-24: Got my photoblog maintainingfocus running on @blot, so far so...
2018-11-23: After seeing quite a few people posting about Blot, I...
2018-11-23: Leadership A good leader show which road to travel, a great...
2018-11-22: The beauty within I like words. I like plain and simple thoughts and...
2018-11-21: ∞ Sorry Mark Zuckerberg, Facebook isn’t a “positive force” In a Tuesday interview with CNN, Mark Zuckerberg defended Facebook’s...
2018-11-20: Dark mode support for the Marfa theme done! You can...
2018-11-20: Working on enabling Safari dark mode support on the Micro.blog...
2018-11-19: ∞ Computers have learned to make us jump through hoops The other day I had to log in to a...
2018-11-18: ∞ To www or not to www For 20 years or so, there has been the debate...
2018-11-18: Micro guide: Getting your hosted Microblog IPv6 ready This is a small guide (hence the category “micro guide”)...
2018-11-17: Had some problems with my two year old iPhone SE,...
2018-11-17: Creative commons licence Redid the footer of my website and am quite happy...
2018-11-11: Finally had some time to dust off my micro.blog hosted...
2018-06-04: Quite the announcements at WWDC from Apple regarding data and...
2018-05-25: Instapaper has temporary shut down for EU residents. I’m not...
2018-05-14: Row Log Update 🚣♂ Calendar Week: 19 Workout: 7,034 meters...
2018-05-12: Row Log Update 🚣♂ Calendar Week: 18 Workout: 6.855 meters...
2018-05-10: Row Log Update 🚣♂ Calendar Week: 18 6,434 m (+...
2018-05-09: Facebook accused of extremists to one another through ‘suggested friends’ feature (The Telegraph) Using sophisticated algorithms, Facebook is designed to connect people who...
2018-05-08: Row Log Update 🚣♂ Calendar Week: 18 Total Meters This...
2018-05-07: For the new indoor rowing season1 I decided to start...
2018-05-07: Row Log Update 🚣♂ Calendar Week: 18 - start of...
2018-05-04: How very social Samsung Smart TV. During the first minute after power-on, the...
2018-05-03: Who’s next? Step 1: get dating apps to build themselves on your...
2018-05-03: Looking in to switching (back) to Linux. It would be...
2018-05-02: Disruptive vs Sustaining innovation and the Tesla case I always enjoy Ben en James talking business strategy in...
2018-05-01: Just finished moving some old photos from instagram to my...
2018-04-30: By the way, here is an important lesson about delegation:...
2018-04-30: Challenge accepted
2018-04-26: The joys of technology: That moment when your TV asks...
2018-04-25: On turning off unnecessary notifications turn off all lock screen notifications (this is better for...
2018-04-25: This will be interesting.. WhatsApp will require users in Europe to be at least...
2018-04-24: 1Blocker will be releasing a new version called 1Blocker X...
2018-04-23: Unread will no longer show “Untitled” or placeholder text from...
2018-04-23: ∞ Collected Goodness Good material. Good art. Good living. Collected Goodness.com by Drew Coffman Very interesting and inspiring stuff,...
2018-04-19: Unlock the Night Owl pin, time to sign off
2018-04-19: Scaling is hard Most of the design tolerances of the Model 3 are...
2018-04-18: Far more than 87 million people may have had their...
2018-04-18: Today I’ve switched RSS service. After Greader, quite some time...
2018-04-17: ∞ It’s all about perspective Mobile apps are still often lacking in efficiency, Apple’s Smart...
2018-04-16: Micro guide: Additional steps and recommendations when using Cloudflare for your DNS Cloudflare users wanting to use a custom domain name for...
2018-04-16: Micro Guides: small articles which can be of great help Yesterday I had the opportunity to help someone from the...
2018-04-15: Micro Guide: Changing the default colors of the Micro.blog Marfa theme I switched my theme for this microblog to Marfa. Marfa...
2018-04-12: Microcasting (short form podcasting) comes to Micro.blog. Welcome Wavelength. What...
2018-04-12: The Irony Mark Zuckerberg left some of his notes on the table...
2018-04-11: Do I really need to switch Text Editors? Questioning if I really need to look in to switching...
2018-04-10: Benedict Evans in his article “The death of the newsfeed”...
2018-04-10: The battery of my Apple Watch(v1) really starts showing it’s...
2018-04-09: ∞ Obscure E-Mail Vulnerability This vulnerability is a result of an interaction between two...
2018-04-09: After writing another post on Facebook and the recent privacy...
2018-04-04: ∞ Facebook Sharply Increases Estimate of How Many Users’ Information Was Harvested By Cambridge Analytica Facebook on Wednesday said the personal information of up to...
2018-04-03: ∞ Lasting appeal of RSS Still, the lasting appeal of RSS remains the parts that...
2018-04-03: “Measure your success by the success of others.” Dave Winer...
2018-04-02: ∞ Cloudflare launches Cloudflare’s mission is to help build a better Internet. We’re...
2017-11-20: ∞ A List Apart's 20th anniversary view on the centralising of the internet in to only a handful of all-powerful networks “In recent years, we’ve seen our rich universe of diverse,...


Hey, I'm Rashel. I've completed Diploma In Computer Engineering from Bhola Polytechnic Institute. My family consists of 6 members including me and my younger brother,Father, Mother, Grandmother, Grandfather. My father is a Farmer and my mother is a housewife.My Short-term goal is to be a Web Developer & Long-term goal is to become a successful Computer Engineer. Now I'm working at AnZa Corporation Ltd as a Assistant Computer Engineer. My strengths are hard-working, self-motivating and dedicated towards my work. My weakness is I feel uncomfortable until I finish my work in the given time. My hobbies are Travelling, listening to music and playing Cricket & Badminton. Thats all about me, thank you.
Bhola Polytechnic Institute
Obtained Result: Fresher
Passing Year: 2020
Aligaon High School, Charfassion.
Obtained Result: 4.33(GPA)
Passing Year: 2016
Some quick example text to build on the card title and make up the bulk of the card's content.Some quick example text to build on the card title and make up the bulk of the card's content.
Some quick example text to build on the card title and make up the bulk of the card's content.Some quick example text to build on the card title and make up the bulk of the card's content.
Some quick example text to build on the card title and make up the bulk of the card's content.Some quick example text to build on the card title and make up the bulk of the card's content.
Some quick example text to build on the card title and make up the bulk of the card's content.
Some quick example text to build on the card title and make up the bulk of the card's content.
Some quick example text to build on the card title and make up the bulk of the card's content.


Basic Auth
By default, an GRIP server allows open access to its API endpoints, but it can be configured to require basic password authentication. To enable this, include users and passwords in your config file:
Server:
BasicAuth:
- User: testuser
Password: abc123
Make sure to properly protect the configuration file so that it’s not readable by everyone:
$ chmod 600 grip.config.yml
To use the password, set the GRIP_USER
and GRIP_PASSWORD
environment variables:
$ export GRIP_USER=testuser
$ export GRIP_PASSWORD=abc123
$ grip list
Using the Python Client
Some GRIP servers may require authorizaiton to access its API endpoints. The client can be configured to pass authorization headers in its requests:
import gripql
# Basic Auth Header - {'Authorization': 'Basic dGVzdDpwYXNzd29yZA=='}
G = gripql.Connection("https://bmeg.io", user="test", password="password").graph("bmeg")
Although GRIP only supports basic password authentication, some servers may be proctected via a nginx or apache server. The python client can be configured to handle these cases as well:
import gripql
# Bearer Token - {'Authorization': 'Bearer iamnotarealtoken'}
G = gripql.Connection("https://bmeg.io", token="iamnotarealtoken").graph("bmeg")
# OAuth2 / Custom - {"OauthEmail"<EMAIL_ADDRESS>"OauthAccessToken": "iamnotarealtoken", "OauthExpires":<PHONE_NUMBER>}
G = gripql.Connection("https://bmeg.io", credential_file="~/.grip_token.json").graph("bmeg")


Contents Design
Media Design
3D Design
Visual Design
UIUX Design
Web publishing
2021-10-20
2021-04-23


Ayanian, Nora
Bekey, George (emeritus)
Bolas, Mark
Gupta, Satyandra
Itti, Laurent
Loeb, Gerald
Matarić, Maja
Perez-Arancibia, Nestor
Requicha, Aristides
Savla, Ketan
Schaal, Stefan
Shen, Wei-Min
Sukhatme, Gaurav
Valero-Cuevas, Francisco
Computer Science
Computer Science
Institute for Creative Technologies
Aerospace and Mechanical Engineering
Computer Science
Biomedical Engineering
Computer Science
Aerospace and Mechanical Engineering
Computer Science
Civil and Environmental Engineering
Computer Science
Computer Science and ISI
Computer Science
Biomedical Engineering
David, Caron
Flashner, Henryk
Khoshnevis, Behrokh "Berok"
Koenig, Sven
Lerman, Kristina
McLaughlin, Margaret
McNitt-Gray, Jill
Narayanan, Shri
Winstein, Carolee
Biology, USC Dornsife College
Aerospace & Mechanical Eng, USC Viterbi School of Engineering
Industrial and Systems Engineering, USC Viterbi School of Engineering
Computer Science Department, USC Viterbi School of Engineering
Information Sciences Institute (ISI) and Computer Science, USC Viterbi School of Engineering
USC Annenberg School of Communication and Journalism
Kinesiology, USC Dornsife College and Biomedical Engineering, USC Viterbi School of Engineering
Ming Hsieh Electrical Engineering, USC Viterbi School of Engineering
Biokinesiology & Physical Therapy


Of the three tracks, Hardware has sent more students to some of the coolest intership seats available at Google, LinkedIn, and Exelon. If you want to go into the field of IT that expands into Networking, Cyber Security, and more, this track is for you.


3DPUG Manchester was started in June 2012. It is as a meeting for 3D printing enthusiasts to exchange knowledge and advice on building, maintaining and operating hobbyist-type 3D printers.
Our meetings normally consist of a show-and-tell segment, where attendees show what they've been working on over the previous month, then a general discussion about issues and advice.
Our venue also happens to contain a fully-stocked workshop, so 3D printer issues can often be troubleshot and solved on the night!
We meet from 7.30pm on the Second Tuesday of every month at Hackspace Manchester.
The exact dates of our upcoming meetings can be found on the HACMan Events Calendar.
Hackspace Manchester
Top Floor,
42 Edge Street,
Manchester,
M4 1HN


Sweet Free Stuff
///////////////////////////////////////////
Free Poise Pad Sample Kit
Posted: 13 Jun 2016 06:00 AM PDT
http://feedproxy.google.com/&edburner&utm_medium=email
The post Free Poise Pad Sample Kit appeared first on SweetFreeStuff.com.
CommentsThanks for the free sample. by MyrnaThanks ‘Poise’ for a nicely
packed sample. by kedgyPoise is the only one i use. thanks poise. by
ceciliabraswellUse them. by Debra doolittleThank You Poise. I can’t wait to
get them. by DebPlus 20 more…
///////////////////////////////////////////
Free Subscription to Parents Magazine
Posted: 13 Jun 2016 02:30 AM PDT
http://feedproxy.google.com/&edburner&utm_medium=email
Free Subscription to Parents Magazine
The post Free Subscription to Parents Magazine appeared first on
SweetFreeStuff.com.
CommentsThank you!!! by Connie FarrellThank you kindly. by minkieThank
you.. by Craig
///////////////////////////////////////////
Free Subscription to Draft Magazine
Posted: 13 Jun 2016 01:40 AM PDT
http://feedproxy.google.com/&edburner&utm_medium=email
Free Subscription to Draft Magazine
The post Free Subscription to Draft Magazine appeared first on
SweetFreeStuff.com.
///////////////////////////////////////////
Free Similac Go & Grow Samples
Posted: 12 Jun 2016 11:32 PM PDT
http://feedproxy.google.com/&edburner&utm_medium=email
Get 1 packet of Food Mix-Ins and 2 packets of Toddler Drink. Also receive
up to $40 in coupons. Child must be 10-24 months old to qualify. Free
Similac Go Grow Samples Coupons
The post Free Similac Go Grow Samples appeared first on SweetFreeStuff.com.
///////////////////////////////////////////
Free Purina One Tender Selects Blend Cat Food Sample
Posted: 12 Jun 2016 10:00 PM PDT
http://feedproxy.google.com/&edburner&utm_medium=email
Click on Sign up for your free sample. Free Purina One Tender Selects Blend
Cat Food Sample
The post Free Purina One Tender Selects Blend Cat Food Sample appeared
first on SweetFreeStuff.com.
///////////////////////////////////////////
Free Spring Samples
Posted: 12 Jun 2016 08:00 PM PDT
http://feedproxy.google.com/&edburner&utm_medium=email
Sign up and receive free Spring product samples! Free Spring Samples
The post Free Spring Samples appeared first on SweetFreeStuff.com.
CommentsThanks by emelsythThanks by YolandaThanks much! by Cheryl
JonesThanks by Brenda Duncan


Preparation time
less than 30 mins
Cooking time
30 mins to 1 hour
Serves
Serves 4
Venison, double marinated in spices, with sweet young root veg, spicy meatballs and a hot chilli sauce.
Preheat the oven to 200C/400F/Gas 6
For the venison, rub the olive oil, salt and chilli powder over the steaks and set aside, covered, for 30 minutes.
Mix together the ingredients for second marinade in a bowl and set aside.
Heat the oil in a large ovenproof frying pan, add the meat and fry over medium heat for 2-3 minutes, until browned all over. Transfer to a roasting tray, spread the second marinade over the top of the meat and roast in the oven for 6-8 minutes, if you like your meat pink. Cook for an additional few minutes if you like it cooked medium to well done.
Remove from the oven and leave the meat to rest for 6-8 minutes in a warm place.
Meanwhile, for the meatballs, combine all the ingredients together and pass them through a mincer or grind them in a food processor to a smooth paste.
Divide into 12 balls, the fry in the venison frying pan for about two minutes over a high heat and remove them from the pan. (You may need to do this in batches.)
For the root vegetables, heat the oil in a heavy based pan and add the whole spices until they crackle and pop.
Add the root vegetables and fry over high heat for 2-3 minutes. Then add salt, sugar and vinegar and mix together, transfer to a pre-heated baking tray and roast for 6-8 minutes in the oven.
For the sauce, mix together the yoghurt, cornmeal and half the stock to a smooth paste and pass through a fine sieve to get rid of any lumps.
Next, heat the oil in a heavy-based frying pan and add the panch phoran. When the spices begin to crackle, add the onion and fry for 2-3 minutes until golden-brown.
Add the turmeric, stir for 30 seconds then whisk in the yoghurt and cornmeal mixture and stir for another 2-3 minutes, taking care that the yoghurt does not split.
Add the other half of the stock, green chillies, and salt, and simmer for about 2-3 minutes until the sauce thickens and gets a glaze. Stir in the jaggery and chopped coriander .
To serve, divide the sauce between four serving plates and place the venison steaks on top. You can serve the meat sliced or as steaks, whatever you prefer. Place the root vegetables and meatballs around the meat and serve hot.


{{ define "main" }}
| Name | Roles | Can access the following listings(limited based on) | Last Login | Status | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Chain Name | Country | State | City | Location Tags | Booking Value | Star Rating | |||||
| Karishma Sinha |<EMAIL_ADDRESS>| Business Manager/Market Manager | All | India | Goa | All | All | Per Night < 2k per night < 5k |
3 star 5 star |
15/9/2017 10:23:15 |
Active
|
| Kiran More |<EMAIL_ADDRESS>| Business Manager/Market Manager | Taj | India | Goa | All | All | All | All | 15/9/2017 10:23:15 |
Active
|
| Irfan Shaikh |<EMAIL_ADDRESS>| Loading Team | All | India | Goa, Karnataka | All | North Goa | Per Night < 2k per night < 5k |
3 star 5 star |
15/9/2017 10:23:15 |
Inactive
|
| Roushni Khan |<EMAIL_ADDRESS>| Finance/Legal | All | India | Goa | All | All | Per Night < 2k per night < 5k |
3 star 5 star |
15/9/2017 10:23:15 |
Active
|
| Keval Tanna |<EMAIL_ADDRESS>| Business Manager/Market Manager | Taj | India | Madhya Pradesh | All | All | All | All | 15/9/2017 10:23:15 |
Inactive
|
| Divya Caculo |<EMAIL_ADDRESS>| Contact Team | All | India | Delhi | All | North Goa | Per Night < 2k per night < 5k |
3 star 5 star |
15/9/2017 10:23:15 |
Inactive
|


Embrace Professional Learning
As Ken Willers’ story illustrates, the value of providing teachers with choice in their PD is strong. Finding time to plan for personalization can feel daunting, but here’s a simple strategy to get you started creating experiences that will resonate with your teachers:
-
Distribute the Technology & Learning Professional Development Report to teacher-leaders and curriculum personnel. Have them think about what the data reveals about the staff’s current strengths and gaps.
-
As a group, select a few targeted learning experiences that address specific gaps, such as a session on using digital cameras as learning tools.
Here's a resource to help.
-
Present the options you created to teachers and allow them to choose the learning experiences that are most relevant to their needs. Because you’ve selected options that address existing gaps, whatever they pick will help.
|


June 2014: Bleeding Edge Web
News from the Bleeding Edge
— Super Abbreviated, But Ultra Fresh
Customize Your Workflow.js
— Corey Butler (@goldglovecb)
Google I/O
- Material Design
- Polymer
- Android L & Beyond
- Android Wear
More Google
- Web Development "Starter Kit"
Material Design
- "Back to Basics"
- Like paper, but expandable
- Consistency across devices
- Has an "elevation" attribute
- Include in Polymer
- Demo
Android L
- Dramatic UI Overhaul
- Universal Syncing
- Simpler User Experience
- Available Today
- Enterprise Support (Data Separation/Samsung Knox)
Android Wear
- Syncs w/ Phone, TV, Car, etc.
Customize Your Workflow.js
Workflow:
A workflow consists of an orchestrated and repeatable pattern of business activity enabled by the systematic organization of resources into processes that transform materials, provide services, or process information.
a.k.a.
Workflow.js:
Use JavaScript to git-r-done... better.
My workflow...
Runs smooth. Stays on track. Best Ever!
Begrudgingly Enlarge Logo
Gets Worse:
- Setup Apache/IIS/NGINX
- Deploy to Test Server (assuming there is one)
- Redeploying changes = PITA
- Testing AJAX
Prototyped in 12hrs instead of 12 days.
In 6-7 Weeks...
- 400+ Tweets
- 500+ Github Stars
- 4700+ Downloads (100-200/day)
Application Components
- Desktop Shell
- Server Infrastructure
- GUI
- Installers
Pour on the JS awesome sauce!
The Good:
- Runs on Windows, OSX, & Linux
- Use HTML5/JS/CSS3 (Chrome)
- Node.js Support
- Supports Icons, System Tray, Native Menus
The Bad:
- Slow Startup (Really, But Not Really)
- No Notifications API (NOT REALLY Chrome)
- Node.js Version Lag
- Security Concerns?
- Permissions
- App Store Issues
Node-Webkit Alternatives
- Macgap
- Brackets-Shell (Adobe)
- Atom-Shell (Github)
- deskShell (formerly AppJS)
- WinJS (Microsoft)
- Qt (Nokia)
- Chrome Extensions | Apps (Google)
- CEF (Not Recommended)
Why Node-Webkit Won
- Community support & velocity (Intel Labs)
- Project maturity
- Pro features (Icons)
- Easier distribution w/ Installers
- Prior experience
- Didn't know about alternatives
Why Linux Didn't (i.e. Windows & OSX Only)
- Hard to maintain "Simple" brand
- Many flavors of Linux
- Smaller community of desktop Linux users
- Fenix does run on Linux
Node Modules
- Web Servers: send
- SSH Tunneling (Sharing): localtunnel
- Request Browser: express
- API: express
Port Detection: portscanner
- Markdown Support: marked
- NOT Using node-windows or node-mac
- ... several more
Fenix Command Line
- Delivered separately as a Node.js CLI utility
- AJAX API, not REST
- API is HTTP based: the universal language
The UI
- HTML5
- CSS3, including Flexbox
- Animate.css, Hover.css
- jQuery
- EcmaScript 5 Constructs
- Custom "Mini-MVC" & Event Bus
- No Framework
Blending GUI and server-side OOP feels strange.
Installers
Installers
- Windows: InnoSetup (Not Trivial)
- Mac: DMG/.app
- Not Required
Why?
- Easier deployment
- Faster setup, no training
- More professional
- Easier updates
Q & A
</presentation>
<BuffaloBilliards>
←
→
/
#