Spark SQL Programming Guide
- Overview
- Getting Started
- Data Sources
- Performance Tuning
- Other SQL Interfaces
- Compatibility with Other Systems
- Writing Language-Integrated Relational Queries
- Spark SQL DataType Reference
Overview
Spark SQL allows relational queries expressed in SQL, HiveQL, or Scala to be executed using Spark. At the core of this component is a new type of RDD, SchemaRDD. SchemaRDDs are composed of Row objects, along with a schema that describes the data types of each column in the row. A SchemaRDD is similar to a table in a traditional relational database. A SchemaRDD can be created from an existing RDD, a Parquet file, a JSON dataset, or by running HiveQL against data stored in Apache Hive.
All of the examples on this page use sample data included in the Spark distribution and can be run in the spark-shell
.
Spark SQL allows relational queries expressed in SQL or HiveQL to be executed using Spark. At the core of this component is a new type of RDD, JavaSchemaRDD. JavaSchemaRDDs are composed of Row objects, along with a schema that describes the data types of each column in the row. A JavaSchemaRDD is similar to a table in a traditional relational database. A JavaSchemaRDD can be created from an existing RDD, a Parquet file, a JSON dataset, or by running HiveQL against data stored in Apache Hive.
Spark SQL allows relational queries expressed in SQL or HiveQL to be executed using Spark. At the core of this component is a new type of RDD, SchemaRDD. SchemaRDDs are composed of Row objects, along with a schema that describes the data types of each column in the row. A SchemaRDD is similar to a table in a traditional relational database. A SchemaRDD can be created from an existing RDD, a Parquet file, a JSON dataset, or by running HiveQL against data stored in Apache Hive.
All of the examples on this page use sample data included in the Spark distribution and can be run in the pyspark
shell.
Spark SQL is currently an alpha component. While we will minimize API changes, some APIs may change in future releases.
Getting Started
The entry point into all relational functionality in Spark is the SQLContext class, or one of its descendants. To create a basic SQLContext, all you need is a SparkContext.
val sc: SparkContext // An existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
// createSchemaRDD is used to implicitly convert an RDD to a SchemaRDD.
import sqlContext.createSchemaRDD
In addition to the basic SQLContext, you can also create a HiveContext, which provides a superset of the functionality provided by the basic SQLContext. Additional features include the ability to write queries using the more complete HiveQL parser, access to HiveUDFs, and the ability to read data from Hive tables. To use a HiveContext, you do not need to have an existing Hive setup, and all of the data sources available to a SQLContext are still available. HiveContext is only packaged separately to avoid including all of Hive’s dependencies in the default Spark build. If these dependencies are not a problem for your application then using HiveContext is recommended for the 1.2 release of Spark. Future releases will focus on bringing SQLContext up to feature parity with a HiveContext.
The entry point into all relational functionality in Spark is the JavaSQLContext class, or one of its descendants. To create a basic JavaSQLContext, all you need is a JavaSparkContext.
JavaSparkContext sc = ...; // An existing JavaSparkContext.
JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext(sc);
In addition to the basic SQLContext, you can also create a HiveContext, which provides a strict super set of the functionality provided by the basic SQLContext. Additional features include the ability to write queries using the more complete HiveQL parser, access to HiveUDFs, and the ability to read data from Hive tables. To use a HiveContext, you do not need to have an existing Hive setup, and all of the data sources available to a SQLContext are still available. HiveContext is only packaged separately to avoid including all of Hive’s dependencies in the default Spark build. If these dependencies are not a problem for your application then using HiveContext is recommended for the 1.2 release of Spark. Future releases will focus on bringing SQLContext up to feature parity with a HiveContext.
The entry point into all relational functionality in Spark is the SQLContext class, or one of its decedents. To create a basic SQLContext, all you need is a SparkContext.
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
In addition to the basic SQLContext, you can also create a HiveContext, which provides a strict super set of the functionality provided by the basic SQLContext. Additional features include the ability to write queries using the more complete HiveQL parser, access to HiveUDFs, and the ability to read data from Hive tables. To use a HiveContext, you do not need to have an existing Hive setup, and all of the data sources available to a SQLContext are still available. HiveContext is only packaged separately to avoid including all of Hive’s dependencies in the default Spark build. If these dependencies are not a problem for your application then using HiveContext is recommended for the 1.2 release of Spark. Future releases will focus on bringing SQLContext up to feature parity with a HiveContext.
The specific variant of SQL that is used to parse queries can also be selected using the
spark.sql.dialect
option. This parameter can be changed using either the setConf
method on
a SQLContext or by using a SET key=value
command in SQL. For a SQLContext, the only dialect
available is “sql” which uses a simple SQL parser provided by Spark SQL. In a HiveContext, the
default is “hiveql”, though “sql” is also available. Since the HiveQL parser is much more complete,
this is recommended for most use cases.
Data Sources
Spark SQL supports operating on a variety of data sources through the SchemaRDD
interface.
A SchemaRDD can be operated on as normal RDDs and can also be registered as a temporary table.
Registering a SchemaRDD as a table allows you to run SQL queries over its data. This section
describes the various methods for loading data into a SchemaRDD.
RDDs
Spark SQL supports two different methods for converting existing RDDs into SchemaRDDs. The first method uses reflection to infer the schema of an RDD that contains specific types of objects. This reflection based approach leads to more concise code and works well when you already know the schema while writing your Spark application.
The second method for creating SchemaRDDs is through a programmatic interface that allows you to construct a schema and then apply it to an existing RDD. While this method is more verbose, it allows you to construct SchemaRDDs when the columns and their types are not known until runtime.
Inferring the Schema Using Reflection
The Scala interaface for Spark SQL supports automatically converting an RDD containing case classes to a SchemaRDD. The case class defines the schema of the table. The names of the arguments to the case class are read using reflection and become the names of the columns. Case classes can also be nested or contain complex types such as Sequences or Arrays. This RDD can be implicitly converted to a SchemaRDD and then be registered as a table. Tables can be used in subsequent SQL statements.
// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
// createSchemaRDD is used to implicitly convert an RDD to a SchemaRDD.
import sqlContext.createSchemaRDD
// Define the schema using a case class.
// Note: Case classes in Scala 2.10 can support only up to 22 fields. To work around this limit,
// you can use custom classes that implement the Product interface.
case class Person(name: String, age: Int)
// Create an RDD of Person objects and register it as a table.
val people = sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt))
people.registerTempTable("people")
// SQL statements can be run by using the sql methods provided by sqlContext.
val teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
// The results of SQL queries are SchemaRDDs and support all the normal RDD operations.
// The columns of a row in the result can be accessed by ordinal.
teenagers.map(t => "Name: " + t(0)).collect().foreach(println)
Spark SQL supports automatically converting an RDD of JavaBeans into a Schema RDD. The BeanInfo, obtained using reflection, defines the schema of the table. Currently, Spark SQL does not support JavaBeans that contain nested or contain complex types such as Lists or Arrays. You can create a JavaBean by creating a class that implements Serializable and has getters and setters for all of its fields.
public static class Person implements Serializable {
private String name;
private int age;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getAge() {
return age;
}
public void setAge(int age) {
this.age = age;
}
}
A schema can be applied to an existing RDD by calling applySchema
and providing the Class object
for the JavaBean.
// sc is an existing JavaSparkContext.
JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext(sc);
// Load a text file and convert each line to a JavaBean.
JavaRDD<Person> people = sc.textFile("examples/src/main/resources/people.txt").map(
new Function<String, Person>() {
public Person call(String line) throws Exception {
String[] parts = line.split(",");
Person person = new Person();
person.setName(parts[0]);
person.setAge(Integer.parseInt(parts[1].trim()));
return person;
}
});
// Apply a schema to an RDD of JavaBeans and register it as a table.
JavaSchemaRDD schemaPeople = sqlContext.applySchema(people, Person.class);
schemaPeople.registerTempTable("people");
// SQL can be run over RDDs that have been registered as tables.
JavaSchemaRDD teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
// The results of SQL queries are SchemaRDDs and support all the normal RDD operations.
// The columns of a row in the result can be accessed by ordinal.
List<String> teenagerNames = teenagers.map(new Function<Row, String>() {
public String call(Row row) {
return "Name: " + row.getString(0);
}
}).collect();
Spark SQL can convert an RDD of Row objects to a SchemaRDD, inferring the datatypes. Rows are constructed by passing a list of key/value pairs as kwargs to the Row class. The keys of this list define the column names of the table, and the types are inferred by looking at the first row. Since we currently only look at the first row, it is important that there is no missing data in the first row of the RDD. In future versions we plan to more completely infer the schema by looking at more data, similar to the inference that is performed on JSON files.
# sc is an existing SparkContext.
from pyspark.sql import SQLContext, Row
sqlContext = SQLContext(sc)
# Load a text file and convert each line to a dictionary.
lines = sc.textFile("examples/src/main/resources/people.txt")
parts = lines.map(lambda l: l.split(","))
people = parts.map(lambda p: Row(name=p[0], age=int(p[1])))
# Infer the schema, and register the SchemaRDD as a table.
schemaPeople = sqlContext.inferSchema(people)
schemaPeople.registerTempTable("people")
# SQL can be run over SchemaRDDs that have been registered as a table.
teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
# The results of SQL queries are RDDs and support all the normal RDD operations.
teenNames = teenagers.map(lambda p: "Name: " + p.name)
for teenName in teenNames.collect():
print teenName
Programmatically Specifying the Schema
When case classes cannot be defined ahead of time (for example,
the structure of records is encoded in a string, or a text dataset will be parsed
and fields will be projected differently for different users),
a SchemaRDD
can be created programmatically with three steps.
- Create an RDD of
Row
s from the original RDD; - Create the schema represented by a
StructType
matching the structure ofRow
s in the RDD created in Step 1. - Apply the schema to the RDD of
Row
s viaapplySchema
method provided bySQLContext
.
For example:
// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
// Create an RDD
val people = sc.textFile("examples/src/main/resources/people.txt")
// The schema is encoded in a string
val schemaString = "name age"
// Import Spark SQL data types and Row.
import org.apache.spark.sql._
// Generate the schema based on the string of schema
val schema =
StructType(
schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)))
// Convert records of the RDD (people) to Rows.
val rowRDD = people.map(_.split(",")).map(p => Row(p(0), p(1).trim))
// Apply the schema to the RDD.
val peopleSchemaRDD = sqlContext.applySchema(rowRDD, schema)
// Register the SchemaRDD as a table.
peopleSchemaRDD.registerTempTable("people")
// SQL statements can be run by using the sql methods provided by sqlContext.
val results = sqlContext.sql("SELECT name FROM people")
// The results of SQL queries are SchemaRDDs and support all the normal RDD operations.
// The columns of a row in the result can be accessed by ordinal.
results.map(t => "Name: " + t(0)).collect().foreach(println)
When JavaBean classes cannot be defined ahead of time (for example,
the structure of records is encoded in a string, or a text dataset will be parsed and
fields will be projected differently for different users),
a SchemaRDD
can be created programmatically with three steps.
- Create an RDD of
Row
s from the original RDD; - Create the schema represented by a
StructType
matching the structure ofRow
s in the RDD created in Step 1. - Apply the schema to the RDD of
Row
s viaapplySchema
method provided byJavaSQLContext
.
For example:
// Import factory methods provided by DataType.
import org.apache.spark.sql.api.java.DataType
// Import StructType and StructField
import org.apache.spark.sql.api.java.StructType
import org.apache.spark.sql.api.java.StructField
// Import Row.
import org.apache.spark.sql.api.java.Row
// sc is an existing JavaSparkContext.
JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext(sc);
// Load a text file and convert each line to a JavaBean.
JavaRDD<String> people = sc.textFile("examples/src/main/resources/people.txt");
// The schema is encoded in a string
String schemaString = "name age";
// Generate the schema based on the string of schema
List<StructField> fields = new ArrayList<StructField>();
for (String fieldName: schemaString.split(" ")) {
fields.add(DataType.createStructField(fieldName, DataType.StringType, true));
}
StructType schema = DataType.createStructType(fields);
// Convert records of the RDD (people) to Rows.
JavaRDD<Row> rowRDD = people.map(
new Function<String, Row>() {
public Row call(String record) throws Exception {
String[] fields = record.split(",");
return Row.create(fields[0], fields[1].trim());
}
});
// Apply the schema to the RDD.
JavaSchemaRDD peopleSchemaRDD = sqlContext.applySchema(rowRDD, schema);
// Register the SchemaRDD as a table.
peopleSchemaRDD.registerTempTable("people");
// SQL can be run over RDDs that have been registered as tables.
JavaSchemaRDD results = sqlContext.sql("SELECT name FROM people");
// The results of SQL queries are SchemaRDDs and support all the normal RDD operations.
// The columns of a row in the result can be accessed by ordinal.
List<String> names = results.map(new Function<Row, String>() {
public String call(Row row) {
return "Name: " + row.getString(0);
}
}).collect();
When a dictionary of kwargs cannot be defined ahead of time (for example,
the structure of records is encoded in a string, or a text dataset will be parsed and
fields will be projected differently for different users),
a SchemaRDD
can be created programmatically with three steps.
- Create an RDD of tuples or lists from the original RDD;
- Create the schema represented by a
StructType
matching the structure of tuples or lists in the RDD created in the step 1. - Apply the schema to the RDD via
applySchema
method provided bySQLContext
.
For example:
# Import SQLContext and data types
from pyspark.sql import *
# sc is an existing SparkContext.
sqlContext = SQLContext(sc)
# Load a text file and convert each line to a tuple.
lines = sc.textFile("examples/src/main/resources/people.txt")
parts = lines.map(lambda l: l.split(","))
people = parts.map(lambda p: (p[0], p[1].strip()))
# The schema is encoded in a string.
schemaString = "name age"
fields = [StructField(field_name, StringType(), True) for field_name in schemaString.split()]
schema = StructType(fields)
# Apply the schema to the RDD.
schemaPeople = sqlContext.applySchema(people, schema)
# Register the SchemaRDD as a table.
schemaPeople.registerTempTable("people")
# SQL can be run over SchemaRDDs that have been registered as a table.
results = sqlContext.sql("SELECT name FROM people")
# The results of SQL queries are RDDs and support all the normal RDD operations.
names = results.map(lambda p: "Name: " + p.name)
for name in names.collect():
print name
Parquet Files
Parquet is a columnar format that is supported by many other data processing systems. Spark SQL provides support for both reading and writing Parquet files that automatically preserves the schema of the original data.
Loading Data Programmatically
Using the data from the above example:
// sqlContext from the previous example is used in this example.
// createSchemaRDD is used to implicitly convert an RDD to a SchemaRDD.
import sqlContext.createSchemaRDD
val people: RDD[Person] = ... // An RDD of case class objects, from the previous example.
// The RDD is implicitly converted to a SchemaRDD by createSchemaRDD, allowing it to be stored using Parquet.
people.saveAsParquetFile("people.parquet")
// Read in the parquet file created above. Parquet files are self-describing so the schema is preserved.
// The result of loading a Parquet file is also a SchemaRDD.
val parquetFile = sqlContext.parquetFile("people.parquet")
//Parquet files can also be registered as tables and then used in SQL statements.
parquetFile.registerTempTable("parquetFile")
val teenagers = sqlContext.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19")
teenagers.map(t => "Name: " + t(0)).collect().foreach(println)
// sqlContext from the previous example is used in this example.
JavaSchemaRDD schemaPeople = ... // The JavaSchemaRDD from the previous example.
// JavaSchemaRDDs can be saved as Parquet files, maintaining the schema information.
schemaPeople.saveAsParquetFile("people.parquet");
// Read in the Parquet file created above. Parquet files are self-describing so the schema is preserved.
// The result of loading a parquet file is also a JavaSchemaRDD.
JavaSchemaRDD parquetFile = sqlContext.parquetFile("people.parquet");
//Parquet files can also be registered as tables and then used in SQL statements.
parquetFile.registerTempTable("parquetFile");
JavaSchemaRDD teenagers = sqlContext.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19");
List<String> teenagerNames = teenagers.map(new Function<Row, String>() {
public String call(Row row) {
return "Name: " + row.getString(0);
}
}).collect();
# sqlContext from the previous example is used in this example.
schemaPeople # The SchemaRDD from the previous example.
# SchemaRDDs can be saved as Parquet files, maintaining the schema information.
schemaPeople.saveAsParquetFile("people.parquet")
# Read in the Parquet file created above. Parquet files are self-describing so the schema is preserved.
# The result of loading a parquet file is also a SchemaRDD.
parquetFile = sqlContext.parquetFile("people.parquet")
# Parquet files can also be registered as tables and then used in SQL statements.
parquetFile.registerTempTable("parquetFile");
teenagers = sqlContext.sql("SELECT name FROM parquetFile WHERE age >= 13 AND age <= 19")
teenNames = teenagers.map(lambda p: "Name: " + p.name)
for teenName in teenNames.collect():
print teenName
Configuration
Configuration of Parquet can be done using the setConf
method on SQLContext or by running
SET key=value
commands using SQL.
| Property Name | Default | Meaning |
|---|---|---|
spark.sql.parquet.binaryAsString |
false | Some other Parquet-producing systems, in particular Impala and older versions of Spark SQL, do not differentiate between binary data and strings when writing out the Parquet schema. This flag tells Spark SQL to interpret binary data as a string to provide compatibility with these systems. |
spark.sql.parquet.cacheMetadata |
false | Turns on caching of Parquet schema metadata. Can speed up querying of static data. |
spark.sql.parquet.compression.codec |
snappy | Sets the compression codec use when writing Parquet files. Acceptable values include: uncompressed, snappy, gzip, lzo. |
JSON Datasets
Spark SQL can automatically infer the schema of a JSON dataset and load it as a SchemaRDD. This conversion can be done using one of two methods in a SQLContext:
jsonFile
- loads data from a directory of JSON files where each line of the files is a JSON object.jsonRdd
- loads data from an existing RDD where each element of the RDD is a string containing a JSON object.
// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
// A JSON dataset is pointed to by path.
// The path can be either a single text file or a directory storing text files.
val path = "examples/src/main/resources/people.json"
// Create a SchemaRDD from the file(s) pointed to by path
val people = sqlContext.jsonFile(path)
// The inferred schema can be visualized using the printSchema() method.
people.printSchema()
// root
// |-- age: IntegerType
// |-- name: StringType
// Register this SchemaRDD as a table.
people.registerTempTable("people")
// SQL statements can be run by using the sql methods provided by sqlContext.
val teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
// Alternatively, a SchemaRDD can be created for a JSON dataset represented by
// an RDD[String] storing one JSON object per string.
val anotherPeopleRDD = sc.parallelize(
"""{"name":"Yin","address":{"city":"Columbus","state":"Ohio"}}""" :: Nil)
val anotherPeople = sqlContext.jsonRDD(anotherPeopleRDD)
Spark SQL can automatically infer the schema of a JSON dataset and load it as a JavaSchemaRDD. This conversion can be done using one of two methods in a JavaSQLContext :
jsonFile
- loads data from a directory of JSON files where each line of the files is a JSON object.jsonRdd
- loads data from an existing RDD where each element of the RDD is a string containing a JSON object.
// sc is an existing JavaSparkContext.
JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext(sc);
// A JSON dataset is pointed to by path.
// The path can be either a single text file or a directory storing text files.
String path = "examples/src/main/resources/people.json";
// Create a JavaSchemaRDD from the file(s) pointed to by path
JavaSchemaRDD people = sqlContext.jsonFile(path);
// The inferred schema can be visualized using the printSchema() method.
people.printSchema();
// root
// |-- age: IntegerType
// |-- name: StringType
// Register this JavaSchemaRDD as a table.
people.registerTempTable("people");
// SQL statements can be run by using the sql methods provided by sqlContext.
JavaSchemaRDD teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19");
// Alternatively, a JavaSchemaRDD can be created for a JSON dataset represented by
// an RDD[String] storing one JSON object per string.
List<String> jsonData = Arrays.asList(
"{\"name\":\"Yin\",\"address\":{\"city\":\"Columbus\",\"state\":\"Ohio\"}}");
JavaRDD<String> anotherPeopleRDD = sc.parallelize(jsonData);
JavaSchemaRDD anotherPeople = sqlContext.jsonRDD(anotherPeopleRDD);
Spark SQL can automatically infer the schema of a JSON dataset and load it as a SchemaRDD. This conversion can be done using one of two methods in a SQLContext:
jsonFile
- loads data from a directory of JSON files where each line of the files is a JSON object.jsonRdd
- loads data from an existing RDD where each element of the RDD is a string containing a JSON object.
# sc is an existing SparkContext.
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
# A JSON dataset is pointed to by path.
# The path can be either a single text file or a directory storing text files.
path = "examples/src/main/resources/people.json"
# Create a SchemaRDD from the file(s) pointed to by path
people = sqlContext.jsonFile(path)
# The inferred schema can be visualized using the printSchema() method.
people.printSchema()
# root
# |-- age: IntegerType
# |-- name: StringType
# Register this SchemaRDD as a table.
people.registerTempTable("people")
# SQL statements can be run by using the sql methods provided by sqlContext.
teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
# Alternatively, a SchemaRDD can be created for a JSON dataset represented by
# an RDD[String] storing one JSON object per string.
anotherPeopleRDD = sc.parallelize([
'{"name":"Yin","address":{"city":"Columbus","state":"Ohio"}}'])
anotherPeople = sqlContext.jsonRDD(anotherPeopleRDD)
Hive Tables
Spark SQL also supports reading and writing data stored in Apache Hive.
However, since Hive has a large number of dependencies, it is not included in the default Spark assembly.
In order to use Hive you must first run “sbt/sbt -Phive assembly/assembly
” (or use -Phive
for maven).
This command builds a new assembly jar that includes Hive. Note that this Hive assembly jar must also be present
on all of the worker nodes, as they will need access to the Hive serialization and deserialization libraries
(SerDes) in order to access data stored in Hive.
Configuration of Hive is done by placing your hive-site.xml
file in conf/
.
When working with Hive one must construct a HiveContext
, which inherits from SQLContext
, and
adds support for finding tables in in the MetaStore and writing queries using HiveQL. Users who do
not have an existing Hive deployment can still create a HiveContext. When not configured by the
hive-site.xml, the context automatically creates metastore_db
and warehouse
in the current
directory.
// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
sqlContext.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src")
// Queries are expressed in HiveQL
sqlContext.sql("FROM src SELECT key, value").collect().foreach(println)
When working with Hive one must construct a JavaHiveContext
, which inherits from JavaSQLContext
, and
adds support for finding tables in in the MetaStore and writing queries using HiveQL. In addition to
the sql
method a JavaHiveContext
also provides an hql
methods, which allows queries to be
expressed in HiveQL.
// sc is an existing JavaSparkContext.
JavaHiveContext sqlContext = new org.apache.spark.sql.hive.api.java.HiveContext(sc);
sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)");
sqlContext.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src");
// Queries are expressed in HiveQL.
Row[] results = sqlContext.sql("FROM src SELECT key, value").collect();
When working with Hive one must construct a HiveContext
, which inherits from SQLContext
, and
adds support for finding tables in in the MetaStore and writing queries using HiveQL. In addition to
the sql
method a HiveContext
also provides an hql
methods, which allows queries to be
expressed in HiveQL.
# sc is an existing SparkContext.
from pyspark.sql import HiveContext
sqlContext = HiveContext(sc)
sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
sqlContext.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src")
# Queries can be expressed in HiveQL.
results = sqlContext.sql("FROM src SELECT key, value").collect()
Performance Tuning
For some workloads it is possible to improve performance by either caching data in memory, or by turning on some experimental options.
Caching Data In Memory
Spark SQL can cache tables using an in-memory columnar format by calling cacheTable("tableName")
.
Then Spark SQL will scan only required columns and will automatically tune compression to minimize
memory usage and GC pressure. You can call uncacheTable("tableName")
to remove the table from memory.
Note that if you call cache
rather than cacheTable
, tables will not be cached using
the in-memory columnar format, and therefore cacheTable
is strongly recommended for this use case.
Configuration of in-memory caching can be done using the setConf
method on SQLContext or by running
SET key=value
commands using SQL.
| Property Name | Default | Meaning |
|---|---|---|
spark.sql.inMemoryColumnarStorage.compressed |
false | When set to true Spark SQL will automatically select a compression codec for each column based on statistics of the data. |
spark.sql.inMemoryColumnarStorage.batchSize |
1000 | Controls the size of batches for columnar caching. Larger batch sizes can improve memory utilization and compression, but risk OOMs when caching data. |
Other Configuration Options
The following options can also be used to tune the performance of query execution. It is possible that these options will be deprecated in future release as more optimizations are performed automatically.
| Property Name | Default | Meaning |
|---|---|---|
spark.sql.autoBroadcastJoinThreshold |
10000 | Configures the maximum size in bytes for a table that will be broadcast to all worker nodes when performing a join. By setting this value to -1 broadcasting can be disabled. Note that currently statistics are only supported for Hive Metastore tables where the command `ANALYZE TABLE <tableName> COMPUTE STATISTICS noscan` has been run. |
spark.sql.codegen |
false | When true, code will be dynamically generated at runtime for expression evaluation in a specific query. For some queries with complicated expression this option can lead to significant speed-ups. However, for simple queries this can actually slow down query execution. |
spark.sql.shuffle.partitions |
200 | Configures the number of partitions to use when shuffling data for joins or aggregations. |
Other SQL Interfaces
Spark SQL also supports interfaces for running SQL queries directly without the need to write any code.
Running the Thrift JDBC server
The Thrift JDBC server implemented here corresponds to the HiveServer2
in Hive 0.12. You can test the JDBC server with the beeline script that comes with either Spark or Hive 0.12.
To start the JDBC server, run the following in the Spark directory:
./sbin/start-thriftserver.sh
This script accepts all bin/spark-submit
command line options, plus a --hiveconf
option to
specify Hive properties. You may run ./sbin/start-thriftserver.sh --help
for a complete list of
all available options. By default, the server listens on localhost:10000. You may override this
bahaviour via either environment variables, i.e.:
export HIVE_SERVER2_THRIFT_PORT=<listening-port>
export HIVE_SERVER2_THRIFT_BIND_HOST=<listening-host>
./sbin/start-thriftserver.sh \
--master <master-uri> \
...
```
or system properties:
./sbin/start-thriftserver.sh \
--hiveconf hive.server2.thrift.port=<listening-port> \
--hiveconf hive.server2.thrift.bind.host=<listening-host> \
--master <master-uri>
...
```
Now you can use beeline to test the Thrift JDBC server:
./bin/beeline
Connect to the JDBC server in beeline with:
beeline> !connect jdbc:hive2://localhost:10000
Beeline will ask you for a username and password. In non-secure mode, simply enter the username on your machine and a blank password. For secure mode, please follow the instructions given in the beeline documentation.
Configuration of Hive is done by placing your hive-site.xml
file in conf/
.
You may also use the beeline script that comes with Hive.
Running the Spark SQL CLI
The Spark SQL CLI is a convenient tool to run the Hive metastore service in local mode and execute queries input from the command line. Note that the Spark SQL CLI cannot talk to the Thrift JDBC server.
To start the Spark SQL CLI, run the following in the Spark directory:
./bin/spark-sql
Configuration of Hive is done by placing your hive-site.xml
file in conf/
.
You may run ./bin/spark-sql --help
for a complete list of all available
options.
Compatibility with Other Systems
Migration Guide for Shark User
Scheduling
To set a Fair Scheduler pool for a JDBC client session,
users can set the spark.sql.thriftserver.scheduler.pool
variable:
SET spark.sql.thriftserver.scheduler.pool=accounting;
Reducer number
In Shark, default reducer number is 1 and is controlled by the property mapred.reduce.tasks
. Spark
SQL deprecates this property in favor of spark.sql.shuffle.partitions
, whose default value
is 200. Users may customize this property via SET
:
SET spark.sql.shuffle.partitions=10;
SELECT page, count(*) c
FROM logs_last_month_cached
GROUP BY page ORDER BY c DESC LIMIT 10;
You may also put this property in hive-site.xml
to override the default value.
For now, the mapred.reduce.tasks
property is still recognized, and is converted to
spark.sql.shuffle.partitions
automatically.
Caching
The shark.cache
table property no longer exists, and tables whose name end with _cached
are no
longer automatically cached. Instead, we provide CACHE TABLE
and UNCACHE TABLE
statements to
let user control table caching explicitly:
CACHE TABLE logs_last_month;
UNCACHE TABLE logs_last_month;
NOTE: CACHE TABLE tbl
is lazy, similar to .cache
on an RDD. This command only marks tbl
to ensure that
partitions are cached when calculated but doesn’t actually cache it until a query that touches tbl
is executed.
To force the table to be cached, you may simply count the table immediately after executing CACHE TABLE
:
CACHE TABLE logs_last_month;
SELECT COUNT(1) FROM logs_last_month;
Several caching related features are not supported yet:
- User defined partition level cache eviction policy
- RDD reloading
- In-memory cache write through policy
Compatibility with Apache Hive
Spark SQL is designed to be compatible with the Hive Metastore, SerDes and UDFs. Currently Spark SQL is based on Hive 0.12.0.
Deploying in Existing Hive Warehouses
The Spark SQL Thrift JDBC server is designed to be “out of the box” compatible with existing Hive installations. You do not need to modify your existing Hive Metastore or change the data placement or partitioning of your tables.
Supported Hive Features
Spark SQL supports the vast majority of Hive features, such as:
- Hive query statements, including:
SELECT
GROUP BY
ORDER BY
CLUSTER BY
SORT BY
- All Hive operators, including:
- Relational operators (
=
,⇔
,==
,<>
,<
,>
,>=
,<=
, etc) - Arithmetic operators (
+
,-
,*
,/
,%
, etc) - Logical operators (
AND
,&&
,OR
,||
, etc) - Complex type constructors
- Mathematical functions (
sign
,ln
,cos
, etc) - String functions (
instr
,length
,printf
, etc)
- Relational operators (
- User defined functions (UDF)
- User defined aggregation functions (UDAF)
- User defined serialization formats (SerDes)
- Joins
JOIN
{LEFT|RIGHT|FULL} OUTER JOIN
LEFT SEMI JOIN
CROSS JOIN
- Unions
- Sub-queries
SELECT col FROM ( SELECT a + b AS col from t1) t2
- Sampling
- Explain
- Partitioned tables
- All Hive DDL Functions, including:
CREATE TABLE
CREATE TABLE AS SELECT
ALTER TABLE
- Most Hive Data types, including:
TINYINT
SMALLINT
INT
BIGINT
BOOLEAN
FLOAT
DOUBLE
STRING
BINARY
TIMESTAMP
ARRAY<>
MAP<>
STRUCT<>
Unsupported Hive Functionality
Below is a list of Hive features that we don’t support yet. Most of these features are rarely used in Hive deployments.
Major Hive Features
- Spark SQL does not currently support inserting to tables using dynamic partitioning.
- Tables with buckets: bucket is the hash partitioning within a Hive table partition. Spark SQL doesn’t support buckets yet.
Esoteric Hive Features
- Tables with partitions using different input formats: In Spark SQL, all table partitions need to have the same input format.
- Non-equi outer join: For the uncommon use case of using outer joins with non-equi join conditions
(e.g. condition “
key < 10
”), Spark SQL will output wrong result for theNULL
tuple. UNION
type andDATE
type- Unique join
- Single query multi insert
- Column statistics collecting: Spark SQL does not piggyback scans to collect column statistics at the moment and only supports populating the sizeInBytes field of the hive metastore.
Hive Input/Output Formats
- File format for CLI: For results showing back to the CLI, Spark SQL only supports TextOutputFormat.
- Hadoop archive
Hive Optimizations
A handful of Hive optimizations are not yet included in Spark. Some of these (such as indexes) are less important due to Spark SQL’s in-memory computational model. Others are slotted for future releases of Spark SQL.
- Block level bitmap indexes and virtual columns (used to build indexes)
- Automatically convert a join to map join: For joining a large table with multiple small tables, Hive automatically converts the join into a map join. We are adding this auto conversion in the next release.
- Automatically determine the number of reducers for joins and groupbys: Currently in Spark SQL, you
need to control the degree of parallelism post-shuffle using “
SET spark.sql.shuffle.partitions=[num_tasks];
”. - Meta-data only query: For queries that can be answered by using only meta data, Spark SQL still launches tasks to compute the result.
- Skew data flag: Spark SQL does not follow the skew data flags in Hive.
STREAMTABLE
hint in join: Spark SQL does not follow theSTREAMTABLE
hint.- Merge multiple small files for query results: if the result output contains multiple small files, Hive can optionally merge the small files into fewer large files to avoid overflowing the HDFS metadata. Spark SQL does not support that.
Writing Language-Integrated Relational Queries
Language-Integrated queries are experimental and currently only supported in Scala.
Spark SQL also supports a domain specific language for writing queries. Once again, using the data from the above examples:
// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
// Importing the SQL context gives access to all the public SQL functions and implicit conversions.
import sqlContext._
val people: RDD[Person] = ... // An RDD of case class objects, from the first example.
// The following is the same as 'SELECT name FROM people WHERE age >= 10 AND age <= 19'
val teenagers = people.where('age >= 10).where('age <= 19).select('name)
teenagers.map(t => "Name: " + t(0)).collect().foreach(println)
The DSL uses Scala symbols to represent columns in the underlying table, which are identifiers
prefixed with a tick ('
). Implicit conversions turn these symbols into expressions that are
evaluated by the SQL execution engine. A full list of the functions supported can be found in the
ScalaDoc.
Spark SQL DataType Reference
- Numeric types
ByteType
: Represents 1-byte signed integer numbers. The range of numbers is from-128
to127
.ShortType
: Represents 2-byte signed integer numbers. The range of numbers is from-32768
to32767
.IntegerType
: Represents 4-byte signed integer numbers. The range of numbers is from-2147483648
to2147483647
.LongType
: Represents 8-byte signed integer numbers. The range of numbers is from-9223372036854775808
to9223372036854775807
.FloatType
: Represents 4-byte single-precision floating point numbers.DoubleType
: Represents 8-byte double-precision floating point numbers.DecimalType
: Represents arbitrary-precision signed decimal numbers. Backed internally byjava.math.BigDecimal
. ABigDecimal
consists of an arbitrary precision integer unscaled value and a 32-bit integer scale.
- String type
StringType
: Represents character string values.
- Binary type
BinaryType
: Represents byte sequence values.
- Boolean type
BooleanType
: Represents boolean values.
- Datetime type
TimestampType
: Represents values comprising values of fields year, month, day, hour, minute, and second.
- Complex types
ArrayType(elementType, containsNull)
: Represents values comprising a sequence of elements with the type ofelementType
.containsNull
is used to indicate if elements in aArrayType
value can havenull
values.MapType(keyType, valueType, valueContainsNull)
: Represents values comprising a set of key-value pairs. The data type of keys are described bykeyType
and the data type of values are described byvalueType
. For aMapType
value, keys are not allowed to havenull
values.valueContainsNull
is used to indicate if values of aMapType
value can havenull
values.StructType(fields)
: Represents values with the structure described by a sequence ofStructField
s (fields
).StructField(name, dataType, nullable)
: Represents a field in aStructType
. The name of a field is indicated byname
. The data type of a field is indicated bydataType
.nullable
is used to indicate if values of this fields can havenull
values.
All data types of Spark SQL are located in the package org.apache.spark.sql
.
You can access them by doing
import org.apache.spark.sql._
| Data type | Value type in Scala | API to access or create a data type |
|---|---|---|
| ByteType | Byte | ByteType |
| ShortType | Short | ShortType |
| IntegerType | Int | IntegerType |
| LongType | Long | LongType |
| FloatType | Float | FloatType |
| DoubleType | Double | DoubleType |
| DecimalType | scala.math.sql.BigDecimal | DecimalType |
| StringType | String | StringType |
| BinaryType | Array[Byte] | BinaryType |
| BooleanType | Boolean | BooleanType |
| TimestampType | java.sql.Timestamp | TimestampType |
| ArrayType | scala.collection.Seq |
ArrayType(elementType, [containsNull]) Note: The default value of containsNull is false. |
| MapType | scala.collection.Map |
MapType(keyType, valueType, [valueContainsNull]) Note: The default value of valueContainsNull is true. |
| StructType | org.apache.spark.sql.Row |
StructType(fields) Note: fields is a Seq of StructFields. Also, two fields with the same name are not allowed. |
| StructField | The value type in Scala of the data type of this field (For example, Int for a StructField with the data type IntegerType) | StructField(name, dataType, nullable) |
All data types of Spark SQL are located in the package of
org.apache.spark.sql.api.java
. To access or create a data type,
please use factory methods provided in
org.apache.spark.sql.api.java.DataType
.
| Data type | Value type in Java | API to access or create a data type |
|---|---|---|
| ByteType | byte or Byte | DataType.ByteType |
| ShortType | short or Short | DataType.ShortType |
| IntegerType | int or Integer | DataType.IntegerType |
| LongType | long or Long | DataType.LongType |
| FloatType | float or Float | DataType.FloatType |
| DoubleType | double or Double | DataType.DoubleType |
| DecimalType | java.math.BigDecimal | DataType.DecimalType |
| StringType | String | DataType.StringType |
| BinaryType | byte[] | DataType.BinaryType |
| BooleanType | boolean or Boolean | DataType.BooleanType |
| TimestampType | java.sql.Timestamp | DataType.TimestampType |
| ArrayType | java.util.List |
DataType.createArrayType(elementType) Note: The value of containsNull will be false DataType.createArrayType(elementType, containsNull). |
| MapType | java.util.Map |
DataType.createMapType(keyType, valueType) Note: The value of valueContainsNull will be true. DataType.createMapType(keyType, valueType, valueContainsNull) |
| StructType | org.apache.spark.sql.api.java |
DataType.createStructType(fields) Note: fields is a List or an array of StructFields. Also, two fields with the same name are not allowed. |
| StructField | The value type in Java of the data type of this field (For example, int for a StructField with the data type IntegerType) | DataType.createStructField(name, dataType, nullable) |
All data types of Spark SQL are located in the package of pyspark.sql
.
You can access them by doing
from pyspark.sql import *
| Data type | Value type in Python | API to access or create a data type |
|---|---|---|
| ByteType |
int or long Note: Numbers will be converted to 1-byte signed integer numbers at runtime. Please make sure that numbers are within the range of -128 to 127. |
ByteType() |
| ShortType |
int or long Note: Numbers will be converted to 2-byte signed integer numbers at runtime. Please make sure that numbers are within the range of -32768 to 32767. |
ShortType() |
| IntegerType | int or long | IntegerType() |
| LongType |
long Note: Numbers will be converted to 8-byte signed integer numbers at runtime. Please make sure that numbers are within the range of -9223372036854775808 to<PHONE_NUMBER>854775807. Otherwise, please convert data to decimal.Decimal and use DecimalType. |
LongType() |
| FloatType |
float Note: Numbers will be converted to 4-byte single-precision floating point numbers at runtime. |
FloatType() |
| DoubleType | float | DoubleType() |
| DecimalType | decimal.Decimal | DecimalType() |
| StringType | string | StringType() |
| BinaryType | bytearray | BinaryType() |
| BooleanType | bool | BooleanType() |
| TimestampType | datetime.datetime | TimestampType() |
| ArrayType | list, tuple, or array |
ArrayType(elementType, [containsNull]) Note: The default value of containsNull is False. |
| MapType | dict |
MapType(keyType, valueType, [valueContainsNull]) Note: The default value of valueContainsNull is True. |
| StructType | list or tuple |
StructType(fields) Note: fields is a Seq of StructFields. Also, two fields with the same name are not allowed. |
| StructField | The value type in Python of the data type of this field (For example, Int for a StructField with the data type IntegerType) | StructField(name, dataType, nullable) |


#include <pthread.h> int pthread_join(pthread_t thread, void **retval);
Compile and link with -pthread.
If retval is not NULL, then pthread_join() copies the exit status of the target thread (i.e., the value that the target thread supplied to pthread_exit(3)) into the location pointed to by *retval. If the target thread was canceled, then PTHREAD_CANCELED is placed in *retval.
If multiple threads simultaneously try to join with the same thread, the results are undefined. If the thread calling pthread_join() is canceled, then the target thread will remain joinable (i.e., it will not be detached).
Joining with a thread that has previously been joined results in undefined behavior.
Failure to join with a thread that is joinable (i.e., one that is not detached), produces a "zombie thread". Avoid doing this, since each zombie thread consumes some system resources, and when enough zombie threads have accumulated, it will no longer be possible to create new threads (or processes).
There is no pthreads analog of waitpid(-1, &status, 0), that is, "join with any terminated thread". If you believe you need this functionality, you probably need to rethink your application design.
All of the threads in a process are peers: any thread can join with any other thread in the process.


AWS::Inspector::AssessmentTarget
The AWS::Inspector::AssessmentTarget
resource creates an Amazon Inspector
assessment target - a resource that contains information about an Amazon Inspector
application.
Topics
Syntax
To declare this entity in your AWS CloudFormation template, use the following syntax:
JSON
{ "Type" : "AWS::Inspector::AssessmentTarget", "Properties" : { "AssessmentTargetName" :
String
, "ResourceGroupArn" :String
} }
YAML
Type: AWS::Inspector::AssessmentTarget Properties: AssessmentTargetName:
String
ResourceGroupArn:String
Properties
AssessmentTargetName
-
The name of the Amazon Inspector assessment target.
Required: No
Type: String
Update requires: Replacement
ResourceGroupArn
-
The ARN that specifies the resource group that is associated with the assessment target. If unspecified, all Amazon EC2 instances in your AWS account in the current region will be included in the assessment target.
Required: No
Type: String
Update requires: No interruption
Return Values
Fn::GetAtt
Fn::GetAtt
returns a value for a specified attribute of this type. The
following are the available attributes and sample return values.
Arn
-
The Amazon Resource Name (ARN) that specifies the assessment target that is created.
For more information about using Fn::GetAtt
, see Fn::GetAtt.
Examples
Declaring an Amazon Inspector Assessment Target Resource
The following example shows how to declare an AWS::Inspector::AssessmentTarget resource to create an Amazon Inspector assessment target.
JSON
"myassessmenttarget": { "Type": "AWS::Inspector::AssessmentTarget", "Properties": { "AssessmentTargetName" : "MyAssessmentTarget", "ResourceGroupArn" : "arn:aws:inspector:us-west-2:123456789012:resourcegroup/0-AB6DMKnv" } }
YAML
myassessmenttarget: Type: AWS::Inspector::AssessmentTarget Properties: AssessmentTargetName : "MyAssessmentTarget" ResourceGroupArn : "arn:aws:inspector:us-west-2:123456789012:resourcegroup/0-AB6DMKnv"


Deuteronomy 8:12
Lest when thou hast eaten and art full, and hast built goodly houses, and dwelt therein;


About the Experiment:
Human Emotion is a big riddle engaging psychologists and neuroscientists to measure and analyze it with contemporary cutting-edge tools. However, still community is struggling to agree on some definition of emotion. In addition, most of the research on emotions is representing the western societies. Although in the past there were some experiments with cross-cultural sampling, the south asian culture(specially India) is not much explored. Largely the research is based on pictures of faces which has low ecological validity. This online survey is an attempt to create a stimulus dataset which is suitable for native Indian population. Once analyzed and made publicly available this stimulus dataset can be used by many researchers nationally as well as internationally(on indian population). This presentation is made possible with the beautiful effort of undergraduate and postgraduate students and excellent mentor and experts.Enter your details


What do you want to search for on Leniva List?
flash_on
Fast craigslist web scraper
Scrape away right here with a beautiful GUI.
camera
Filter your searches
We built in functionality to filter based on pricing.
chrome_reader_mode
A much sexier front-end to Craigslist
We built this with ❤ and sliders


Components Page
Logo
Koala Logo Image
Leaves Motif
Bird Motif
Map Background Location Permission Page
Buttons / Info Architecture


Standard Card
Use fd-card
component with a fd-card-header
and fd-card-content
to create a
simple card
Compact mode
Use fdCompact
option to make the card compact
Card loader
Use [isLoading]="true"
along with fd-card-loader
component to show alternative content
during data loading
Use fd-card-footer
if footer is required to be added to the card. Use
*fdCardFooterActionItem
for action elements.
If footer has not enough place for all your actions, use overflow button to show more actions.
Analytical Card
Use fd-card-kpi-header
component along with [cardType]="analytical"
to display analytical
data in the card header.
Here for illustration we use google charts
library to render a chart within the card content.
Table Card
Use [cardType]="table"
to display table card
Bar Chart List
In order to display bar chart list in a card we can use list styles along with a custom bar component. For this
example in order to display bar chart we created a simple fd-card-bar
component. For card with list it
is not recommended to keep header interactive, or navigable. Such a header can be achieved by adding
[interactive]="false"
to fd-card-header
element.
Object card
The object card can display a single object or a group of objects.
Calendar card
The calendar card displays calendar within card.
Quick view card
Quick view cards are single-object cards. They display the basic details for one object, such as the name, address,
and phone numbers for a contact.
List card
List cards display a set of items or links in a list format.
Link List card
Link list cards allow you to display a collection of links or images that can reference both internal and external
targets.Links and images are handled as two separate variants: list and image.
Card Loading
Here is the examples of various cards in which content is hidden under loading placeholders (skeletons). For more
info about using this technique please see Skeleton docs.


Mali’s military abandons isolated outposts amid attacks
by The Associated PressBAMAKO, Mali — Mali’s military has abandoned some of its most isolated outposts in the desert north while extremist attacks have killed more than 100 soldiers in just six weeks.
The West African nation’s president is facing a decline in military morale, a sentiment that helped spark a coup against his predecessor in 2012 amid an uprising that saw separatists and Islamic fighters take over large swaths of the north.
Some also are angry at a French military presence.
Mali’s president has pledged “improved conditions for our men,” saying troops will take a more offensive role.
The extremists add to their weapons stockpile with each raid. Fighters linked to al-Qaida and the Islamic State group have published images on social media of their spoils of war including automatic weapons, motorcycles and other vehicles.
The Associated Press


We're sorry but Redexc Systems doesn't work properly without JavaScript enabled. Please enable it to continue.


This example demonstrates HMI for Electronic Vehicle Charging Station (EVSE) application. QT QWidget tool is used to draw the screen.
The demo shows four screens
1. Welcome screen
2. User options
3. Charging rate
4. Charging status


SoftBank Corp CEO says WeWork Japan can become profitable 'in near future'
by ReutersTOKYO (Nov 5):SoftBank Corp CEO Ken Miyauchi said on Tuesday WeWork’s Japanese business can become profitable “in the near future”.
WeWork Japan, a joint venture between the office-space sharing startup, telco SoftBank Corp and its parent SoftBank Group Corp, is growing well with high occupancy rates, executives said at a news conference.
SoftBank Group was forced to bail out U.S-based WeWork after its IPO attempt flopped and is leading a restructuring at the money-losing firm.


v0.2¶
-
GET
/api/v0.2/attributes/
(attribute_name)¶ Retrieve available values for a given attribute_name separated by semicolons
Parameters: - attribute_name – uniqueid of an attribute name
Example requests:
# get all values for attribute "Foor finish material" (uniqueid: floorfinishmaterial) GET https://shelter-database.org/api/v0.2/attributes/floorfinishmaterial
-
GET
/api/v0.2/attributes/pictures/
(language_code)¶
-
GET
/api/v0.2/attributes/pictures
¶ Retrieve attribute pictures in a given language. If no language code prameter supplied, the default language is English.
Parameters: - language_code (string) – language code, in lower case two letter format. Example: ‘fr’ for French
Example requests:
# get attribute pictures in default language (English) GET https://shelter-database.org/api/v0.2/attributes/pictures # get attribute pictures in French GET https://shelter-database.org/api/v0.2/attributes/pictures/fr
-
GET
/api/v0.2/attributes/pictures/has/
(uniqueid)¶ Retrieve attribute picture for specific attribute
Parameters: - uniqueid (string) – uniqueid of the attribute
Example requests:
# get pictures for Foundation Type (uniqueid: foundationtype) GET https://shelter-database.org/api/v0.2/attributes/pictures/has/foundationtype
-
GET
/api/v0.2/documentation
¶ Serves this documentation page
-
GET
/api/v0.2/glossary
¶ Retrieve the glossary in JSON format, readable by Glossarizer (https://github.com/PebbleRoad/glossarizer)
-
GET
/api/v0.2/shelters/
(int: shelter_id)¶
-
GET
/api/v0.2/shelters
¶ Retrieves shelters with all of their attributes and pictures.
Parameters: - shelter_id (int) – a unique shelter ID generated by the server
Query Parameters: - format – if set to
prettytext
, attribute names are retrieved as nicely formatted text (Capital letters, special characters and spaces allowed) - attribute – attribute name
- value – attribute value
- q – Full text search. Works in English language only.
 Only retrieves shelters that have pictures. If no count parameter is supplied, the API retrieves the latest shelter.
Parameters: - count (int) – number of latest shelters to return
Example requests:
# get latest shelter GET https://shelter-database.org/api/v0.2/shelters/latest # get the 3 latest shelters GET https://shelter-database.org/api/v0.2/shelters/latest/3


@(user: User) @main("Login Success")(new ArrayList())(new ArrayList()) {
Password successfully changed. Please check your email at @user.email for a confirmation email.


--- layout: article title: Great Migration Begins Reversal description: Return of African Americans, Blacks to the American South ---
Using American Community Survey data from 2005-2010, I worked with Paul Overberg to visualize the movement of African Americans in and out of metros. I relied on D3 primarily for the front-end presenation and used Python to convert the spreadsheet data into a nested json file.
Read the full story and explore the interactive.


| CONTENTS |
Many of the techniques in this chapter aren't nearly as applicable as they once were. At the time of this writing, EIDE disks are about a dollar a gigabyte; even fast-wide SCSI isn't that expensive. Often the solution to running low on disk space is just to buy more.
That said, many of these techniques illustrate useful things to know about Unix. It's common these days to run Unix on an old, spare machine where it's not worth the trouble of upgrading the disks. You may also be dealing with a Unix box at work or school that uses expensive, highly reliable disks with expensive backup procedures in place, where more disk space just isn't an option. It never hurts to know how to eke the last few bytes out of a partition.
This chapter also has a lot of information about compressing and decompressing files, which is fairly common. (These days, you may well compress files to save network bandwidth rather than disk space, but the same principles apply.) So enjoy exploring!
DH
Sometimes you don't want to remove a file completely you just want to empty it:
If an active process has the file open (not uncommon for log files), removing the file and creating a new one will not affect the logging program; those messages will just keep going to the file that's no longer linked. Emptying the file doesn't break the association, and so it clears the file without affecting the logging program.
When you remove a file and create a new one with the same name, the new file will have your default permissions and ownership (Section 50.3). It's better to empty the file now, then add new text later; this won't change the permissions and ownership.
Completely empty files (ones that ls -l says have zero characters) don't take any disk space to store (except the few bytes that the directory entry (Section 10.2) uses).
You can use the empty files as "place markers" to remind you that something was there or belongs there. Some Unix logging programs won't write errors to their log files unless the log files already exist. Empty files work fine for that.
Empty files hold a "timestamp" (just as files with text do) that shows when the file was last modified. I use empty files in some directories to remind me when I've last done something (backups, printouts, etc.). The find -newer (Section 9.8) command can compare other files to a timestamp file.
Well, you get the idea by now.
How can you empty a file? Watch out: when some editors say that a file has "no lines," they may still append a newline character when writing the file. Just one character still takes a block of disk space to store. Here are some better ways to get a properly empty file:
In Bourne-type shells like sh and bash, the most efficient way is to redirect the output of a null command:
$ > afile
If the file already exists, that command will truncate the file without needing a subprocess.
Copy the Unix empty file, /dev/null (Section 43.12), on top of the file:
% cp /dev/null afile
Or just cat it there:
% cat /dev/null > afile
You can also "almost" empty the file, leaving just a few lines, this way:
tail Section 12.8
% tail afile > tmpfile % cat tmpfile > afile % rm tmpfile
That's especially good for log files that you never want to delete completely. Use cat and rm, not mv -- mv will break any other links to the original file (afile) and replace it with the temporary file.
JP
Some Unix programs usually background or daemon programs insist on writing a log file. You might not want the log file itself as much as you want the disk space that the log file takes. Here are a few tips:
Some programs will write to a log file only if the log file exists. If the program isn't running, try removing the log file.
If you remove a log file and the program recreates it, look for command-line options or a configuration-file setup that tells the program not to make the log file.
If you can't get the program to stop writing the log file, try replacing the log file with a symbolic link to /dev/null (Section 43.12):
# rm logfile # ln -s /dev/null logfile
The program won't complain, because it will happily write its log file to /dev/null, which discards everything written to it. (Writing to /dev/null is also known as "throwing it in the bit bucket," since all the bits just go away.) Watch out for programs that run at reboot or those that run from the system crontab (Section 25.2) to truncate and replace the log file. These programs might replace the symbolic link with a small regular file that will start growing again.
Does a system mailbox for a user like bin keep getting mail (Section 1.21) that you want to throw away? You may be able to add a .forward file to the account's home directory with this single line:
/dev/null
Or add an alias in the system mail alias file that does the same thing:
bin: /dev/null
If your system has a command like newaliases to rebuild the alias database, don't forget to use it after you make the change.
JP
You might have copies of the same file in several directories for the following reasons:
Several different users need to read it (a data file, a program setup file, a telephone list, etc.).
It's a program that more than one person wants to use. For some reason, you don't want to keep one central copy and put its directory in your search path (Section 27.6).
The file has a strange name or it's in a directory you don't usually use. You want a name that's easier to type, but you can't use mv.
Instead of running cp, think about ln. There are lots of advantages to links (Section 10.3). One big advantage of hard links is that they don't use any disk space.[1] The bigger the file, the more space you save with a link. A symbolic link always takes some disk space, so a hard link might be better for ekeing the most space out of your disk. Of course, you have to use a symbolic link if you want to link across filesystems, and symbolic links are much more obvious to other people, so a symlink is less likely to confuse people. Generally the clarity is worth the little bit of extra disk space.
JP
Here are techniques to keep you from creating large files (which can happen by accident, such as with runaway programs). Your shell may be able to set process limits. If you're writing a program in C or another language that has access to kernel system calls, you can set these limits yourself. And there's one more trick you can use.
These limits are passed to child processes. So, if your shell sets a limit, all programs you start from that shell will inherit the limit from their parent process.
Many shells have a built-in command that uses system calls to set resource limits. This is usually done from a shell setup file (Section 3.3), but can also be done from the command line at a shell prompt. To set a maximum file size in C-type shells and zsh , use the command limit filesize max-size. In the Korn shell and bash, use ulimit -f max-size. For example, the following csh and ksh commands keep you from creating any files larger than 2 MB:
% limit filesize 2m $ ulimit -f 2000
Similarly, on many systems, you can use limit and ulimit to restrict the size of core dump files. Core dumps are generally large files, and if you are not actively developing or debugging, they are often not interesting or useful. To set a maximum size for core dumps, execute one of these commands:
% limit coredumpsize max-size $ ulimit -c max-size
To eliminate core dumps entirely, use 0 (zero) for max-size. Because core dumps are essential for effective debugging, any users who actively debug programs should know the commands unlimit coredumpsize (which removes this restriction in csh ) and ulimit -c unlimited for bash and ksh.
File size limits only apply to processes that are invoked from a shell where the limit is set. For instance, at and cron jobs might not read the shell setup file (Section 3.3) that sets your limit. One way to fix this is to set the limit explicitly before you start the job. For instance, to keep your cron job named cruncher from core-dumping, make the crontab entry similar to one of these:
47 2 * * * ulimit -c 0; cruncher 47 2 * * * bash -c 'ulimit -c 0; exec cruncher'
If you've written a daemon (Section 1.10) in C that starts as your workstation boots up (so no shell is involved), have your program invoke a system call like ulimit(3) or setrlimit(2).
If the unwanted files are created in a directory where you can deny write permission to the directory itself and the files are not created by a process running as root (filesystem permissions don't apply to root) simply make the directory unwritable. (If the process needs to write temporary files, have it use /tmp. An environment variable such as TMP or TMPDIR may control this.)
You can prevent the files from being created by putting a zero-size unwritable file in the directory where the files are being created. Because the file is zero-length, it doesn't take any disk space to store:
chmod Section 50.5
% touch core % chmod 000 core
If all else fails, try making a symbolic link to /dev/null (Section 43.12).
ML and JP
gzip is a fast and efficient compression program distributed by the GNU project. The basic function of gzip is to take a file filename, compress it, save the compressed version as filename.gz, and remove the original, uncompressed file. The original file is removed only if gzip is successful; it is very difficult to delete a file accidentally in this manner. Of course, being GNU software, gzip has more options than you want to think about, and many aspects of its behavior can be modified using command-line options.
First, let's say that we have a large file named garbage.txt:
rutabaga% ls -l garbage.txt* -rw-r--r-- 1 mdw hack 312996 Nov 17 21:44 garbage.txt
If we compress this file using gzip, it replaces garbage.txt with the compressed file garbage.txt.gz. We end up with the following:
rutabaga% gzip garbage.txt rutabaga% ls -l garbage.txt* -rw-r--r-- 1 mdw hack 103441 Nov 17 21:48 garbage.txt.gz
Note that garbage.txt is removed when gzip completes.
You can give gzip a list of filenames; it compresses each file in the list, storing each with a .gz extension. (Unlike the zip program for Unix and MS-DOS systems, gzip will not, by default, compress several files into a single .gz archive. That's what tar is for; see Section 15.7.)
Go to http://examples.oreilly.com/upt3 for more information on: gzip
How efficiently a file is compressed depends upon its format and contents. For example, many audio and graphics file formats (such as MP3 and JPEG) are already well compressed, and gzip will have little or no effect upon such files. Files that compress well usually include plain-text files and binary files such as executables and libraries. You can get information on a gzip ped file using gzip -l. For example:
rutabaga% gzip -l garbage.txt.gz compressed uncompr. ratio uncompressed_name<PHONE_NUMBER>96 67.0% garbage.txt
To get our original file back from the compressed version, we use gunzip, as in:
rutabaga% gunzip garbage.txt.gz rutabaga% ls -l garbage.txt -rw-r--r-- 1 mdw hack 312996 Nov 17 21:44 garbage.txt
which is identical to the original file. Note that when you gunzip a file, the compressed version is removed once the uncompression is complete.
gzip stores the name of the original, uncompressed file in the compressed version. This allows the name of the compressed file to be irrelevant; when the file is uncompressed it can be restored to its original splendor. To uncompress a file to its original filename, use the -N option with gunzip. To see the value of this option, consider the following sequence of commands:
rutabaga% gzip garbage.txt rutabaga% mv garbage.txt.gz rubbish.txt.gz
If we were to gunzip rubbish.txt.gz at this point, the uncompressed file would be named rubbish.txt, after the new (compressed) filename. However, with the -N option, we get the following:
rutabaga% gunzip -N rubbish.txt.gz rutabaga% ls -l garbage.txt -rw-r--r-- 1 mdw hack 312996 Nov 17 21:44 garbage.txt
gzip and gunzip can also compress or uncompress data from standard input and output. If gzip is given no filenames to compress, it attempts to compress data read from standard input. Likewise, if you use the -c option with gunzip, it writes uncompressed data to standard output. For example, you could pipe the output of a command to gzip to compress the output stream and save it to a file in one step, as in:
rutabaga% ls -laR $HOME | gzip > filelist.gz
This will produce a recursive directory listing of your home directory and save it in the compressed file filelist.gz. You can display the contents of this file with the command:
rutabaga% gunzip -c filelist.gz | less
This will uncompress filelist.gz and pipe the output to the less (Section 12.3) command. When you use gunzip -c, the file on disk remains compressed.
The gzcat command is identical to gunzip -c. You can think of this as a version of cat for compressed files. Some systems, including Linux, even have a version of the pager less for compressed files: zless.
When compressing files, you can use one of the options -1, -2, through -9 to specify the speed and quality of the compression used. -1 (also - -fast) specifies the fastest method, which compresses the files less compactly, while -9 (also - -best) uses the slowest, but best compression method. If you don't specify one of these options, the default is -6. None of these options has any bearing on how you use gunzip; gunzip can uncompress the file no matter what speed option you use.
Go to http://examples.oreilly.com/upt3 for more information on: bzip, bzip2
Another compression/decompression program has emerged to take the lead from gzip. bzip2 is the new kid on the block and sports even better compression (on the average about 10 to 20% better than gzip), at the expense of longer compression times. You cannot use bunzip2 to uncompress files compressed with gzip and vice versa. Since you cannot expect everybody to have bunzip2 installed on their machine, you might want to confine yourself to gzip for the time being if you want to send the compressed file to somebody else (or, as many archives do, provide both gzip- and bzip2-compressed versions of the file). However, it pays to have bzip2 installed, because more and more FTP servers now provide bzip2-compressed packages to conserve disk space and, more importantly these days, bandwidth. You can recognize bzip2-compressed files from their typical .bz2 file name extension.
While the command-line options of bzip2 are not exactly the same as those of gzip, those that have been described in this section are, except for - -best and - -fast, which bzip2 doesn't have. For more information, see the bzip2 manual page.
The bottom line is that you should use gzip/gunzip or bzip2/bunzip2 for your compression needs. If you encounter a file with the extension .Z, it was probably produced by compress, and gunzip can uncompress it for you.
[These days, the only real use for compress if you have gzip and bzip2 is for creating compressed images needed by some embedded hardware, such as older Cisco IOS images. DJPH]
MW, MKD, and LK
In the Unix filesystem, files are stored in blocks. Each nonempty file, no matter how small, takes up at least one block.[2] A directory tree full of little files can fill up a lot of partly empty blocks. A big file is more efficient because it fills all (except possibly the last) of its blocks completely.
The tar (Section 39.2) command can read lots of little files and put them into one big file. Later, when you need one of the little files, you can extract it from the tar archive. Seems like a good space-saving idea, doesn't it? But tar, which was really designed for magnetic tape archives, adds "garbage" characters at the end of each file to make it an even size. So, a big tar archive uses about as many blocks as the separate little files do.
Okay, then why am I writing this article? Because the gzip (Section 15.6) utility can solve the problems. It squeezes files down compressing them to get rid of repeated characters. Compressing a tar archive typically saves 50% or more. The bzip2 (Section 15.6) utility can save even more.
|
Making a compressed archive of a directory and all of its subdirectories is easy: tar copies the whole tree when you give it the top directory name. Just be sure to save the archive in some directory that won't be copied so tar won't try to archive its own archive! I usually put the archive in the parent directory. For example, to archive my directory named project, I'd use the following commands. The .tar.gz extension isn't required, but is just a convention; another common convention is .tgz. I've added the gzip - -best option for more compression but it can be a lot slower, so use it only if you need to squeeze out every last byte. bzip2 is another way to save bytes, so I'll show versions with both gzip and bzip2. No matter what command you use, watch carefully for errors:
.. Section 1.16, -r Section 14.16
% cd project % tar clf - . | gzip --best > ../project.tar.gz % gzcat ../project.tar.gz | tar tvf - Quick verification % tar clf - . | bzip2 --best > ../project.tar.bz2 % bzcat ../project.tar.bz2 | tar tvf - Quick verification % cd .. % rm -r project
Go to http://examples.oreilly.com/upt3 for more information on: tar
If you have GNU tar or another version with the z option, it will run gzip for you. This method doesn't use the gzip - -best option, though so you may want to use the previous method to squeeze out all you can. Newer GNU tars have an I option to run bzip2. Watch out for other tar versions that use -I as an "include file" operator check your manpage or tar help. If you want to be sure that you don't have a problem like this, use the long options ( -- gzip and -- bzip2) because they're guaranteed not to conflict with something else; if your tar doesn't support the particular compression you've asked for, it will fail cleanly rather than do something you don't expect.
Using the short flags to get compression from GNU tar, you'd write the previous tar command lines as follows:
tar czlf ../project.tar.gz . tar cIlf ../project.tar.bz2 .
In any case, the tar l (lowercase letter L) option will print messages if any of the files you're archiving have other hard links (Section 10.4). If a lot of your files have other links, archiving the directory may not save much disk space the other links will keep those files on the disk, even after your rm -r command.
Any time you want a list of the files in the archive, use tar t or tar tv:
less Section 12.3
% gzcat project.tar.gz | tar tvf - | less rw-r--r--239/100 485 Oct 5 19:03 1991 ./Imakefile rw-rw-r--239/100 4703 Oct 5 21:17 1991 ./scalefonts.c rw-rw-r--239/100 3358 Oct 5 21:55 1991 ./xcms.c rw-rw-r--239/100 12385 Oct 5 22:07 1991 ./io/input.c rw-rw-r--239/100 7048 Oct 5 21:59 1991 ./io/output.c ... % bzcat project.tar.bz2 | tar tvf - | less ... % tar tzvf project.tar.gz | less ... % tar tIvf project.tar.bz2 | less ...
To extract all the files from the archive, type one of these tar command lines:
% mkdir project % cd project % gzcat ../project.tar.gz | tar xf - % mkdir project % cd project % bzcat ../project.tar.bz2 | tar xf - % mkdir project % cd project % tar xzf ../project.tar.gz % mkdir project % cd project % tar xIf ../project.tar.bz2
Of course, you don't have to extract the files into a directory named project. You can read the archive file from other directories, move it to other computers, and so on.
You can also extract just a few files or directories from the archive. Be sure to use the exact name shown by the previous tar t command. For instance, to restore the old subdirectory named project/io (and everything that was in it), you'd use one of the previous tar command lines with the filename at the end. For instance:
% mkdir project % cd project % gzcat ../project.tar.gz | tar xf - ./io
JP
Two tools, df and du, report how much disk space is free and how much is used by any given directory. For each filesystem, df tells you the capacity, how much space is in use, and how much is free. By default, it lists both local and remote (i.e., NFS (Section 1.21)) filesystems. Under Linux or BSD Unix, the output from df looks like this:
% df Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/ad0s1a 99183 37480 53769 41% / /dev/ad2s1e 3943876 1873453 1754913 52% /home /dev/ad0s1f 3360437 1763460 1328143 57% /usr /dev/ad0s1e<PHONE_NUMBER>5 450567 4% /var procfs 4 4 0 100% /proc toy:/usr 17383462 15470733 522053 97% /toy ...
This report shows information about four local filesystems, the local procfs filesystem, and one remote filesystem (from the system toy). Note that a normal filesystem that is 100% full really has 5 to 10% free space but only the superuser (Section 1.18) can use this reserved space, and that usually isn't a good idea. The reserved space is primarily for recovering from the disk filling up for some reason; the superuser can still successfully copy files and the like to free up space. Special filesystems often don't do this sort of block reservation; procfs and ISO-9660 (CD-ROM and CD-R) filesystems don't care.
df can be invoked in several other ways:
If you already know that you're interested in a particular filesystem, you can use a command such as df /home or df . (. means "the current directory" (Section 1.16)).
If your system uses NFS and you are interested only in local (non-NFS) filesystems, use the command df -t ufs (most BSDs) or df -t ext2fs (most Linuxes). You should always use this command if remote file servers are down. If you have mounted remote disks that are unavailable, df will be extremely slow or hang completely.
If you are interested in inode usage rather than filesystem data capacity, use the command df -i. This produces a similar report showing inode statistics.
If you are using the older System V filesystem (for example, on Solaris), the report from df will look different. The information it presents, however, is substantially the same. Here is a typical report:
% df / (/dev/root ): 1758 blocks 3165 i-nodes /u (/dev/u ): 108 blocks 13475 i-nodes /usr (/dev/usr ): 15694 blocks 8810 i-nodes
[If you get this sort of output from df, you may be able to get the BSDish display by using df -P or df -k. You may also want to try the GNU df. DH]
There are 1,758 physical blocks (always measured as 512-byte blocks for this sort of df, regardless of the filesystem's logical block size) and 3,165 inodes available on the root filesystem. To find out the filesystem's total capacity, use df -t. The command df -l only reports on your system's local filesystems, omitting filesystems mounted by NFS or RFS.
It is often useful to know how much storage a specific directory requires. This can help you to determine if any users are occupying more than their share of storage. The du utility provides such a report. Generally you want to use the -k to du; by default its reports are in disk blocks and thus somewhat harder to read. -k asks df to report its numbers in kilobytes. Here's a simple report from du:
% du -k 107 ./reports 888 ./stuff 32 ./howard/private 33 ./howard/work 868 ./howard 258 ./project/code 769 ./project 2634 .
du shows that the current directory and all of its subdirectories occupy about 2.5 MB (2,634 KB). The biggest directories in this group are stuff and howard, which have a total of 888 KB and 868 KB, respectively. The total for each directory includes the totals for any subdirectories, as well as files in the directory itself. For instance, the two subdirectories private and work contribute 65 KB to howard; the rest of the 868 KB is from files in howard itself. (So, to get the grand total of 2,634, du adds 107, 888, 868, and 769, plus files in the top-level directory.) du does not show individual files as separate items unless you use its -a option.
The -s option tells du to report the total amount of storage occupied by a directory; it suppresses individual reports for all subdirectories. For example:
% du -s 2634 .
This is essentially the last line of the previous report. du -s is particularly useful for showing only the files in the current directory, rather than showing every directory down the tree:
% cd /home % du -sk * 69264 boots 18236 chaos 1337820 deb ...
ML, from System Performance Tuning (O'Reilly, 2002)
Here's a quick little command that will compress (Section 15.6) files in the current directory and below. It uses find (Section 9.2) to find the files recursively and pick the files it should compress:
-sizeSection 9.14, xargs Section 28.17
% find . ! -perm -0100 -size +1 -type f -print | xargs gzip -v
This command finds all files that are the following:
Not executable (! -perm -0100), so we don't compress shell scripts and other program files.
Bigger than one block, since it won't save any disk space to compress a file that takes one disk block or less. But, depending on your filesystem, the -size +1 may not really match files that are one block long. You may need to use -size +2, -size +1024c, or something else.
Regular files (-type f) and not directories, named pipes, etc.
The -v switch to gzip tells you the names of the files and how much they're being compressed. If your system doesn't have xargs, use the following:
% find . ! -perm -0100 -size +1 -type f -exec gzip -v {} \;
Tune the find expressions to do what you want. Here are some ideas for more, read your system's find manual page:
Skip any file that's already gzipped (filename ends with .gz ).
Only compress files that have no other (hard) links.
Only compress files that belong to you.
Only compress files that haven't been accessed (read, edited, etc.) for more than 60 days.
You might want to put this in a job that's run every month or so by at (Section 25.5) or cron (Section 25.2).
JP
After you compile and debug a program, there's a part of the executable binary that you can delete to save disk space. The strip command does the job. Note that once you strip a file, you can't use a symbolic debugger like dbx or gdb on it!
Here's an example. I'll compile a C program and list it. Then I'll strip it and list it again. How much space you save depends on several factors, but you'll almost always save something.
-s Section 9.14
% cc -o echoerr echoerr.c % ls -ls echoerr 52 -rwxr-xr-x 1 jerry 24706 Nov 18 15:49 echoerr % strip echoerr % ls -ls echoerr 36 -rwxr-xr-x 1 jerry 16656 Nov 18 15:49 echoerr
The GNU strip has a number of options to control what symbols and sections are stripped from the binary file. Check the strip manpage for specific details of the version you have.
If you know that you want a file stripped when you compile it, your compiler probably has a -s option (which is passed to ld after compilation is complete). If you use ld directly say, in a makefile (Section 11.10) use the -s option there.
Here's a shell script named stripper that finds all the unstripped executable files in your bin directory (Section 7.4) and strips them. It's a quick way to save space on your account. (The same script, searching the whole filesystem, will save even more space for system administrators but watch out for unusual filenames):
xargs Section 28.17
#! /bin/sh skipug="! -perm -4000 ! -perm -2000" # SKIP SETUID, SETGID FILES find $HOME/bin -type f \( -perm -0100 $skipug \) -print | xargs file | sed -n '/executable .*not stripped/s/: TAB .*//p' | xargs -rpl strip
The find (Section 9.2) finds all executable files that aren't setuid or setgid and runs file (Section 12.6) to get a description of each. The sed command skips shell scripts and other files that can't be stripped. sed searches for lines from file like the following:
/usr/local/bin/xemacs: TAB xxx... executable xxx... not stripped
with the word "executable" followed by "not stripped." sed removes the colon, tab, and description, then passes the filename to strip.
The final xargs command uses the options -r (to not run strip if sed outputs no names to strip), -p (to be interactive, asking before each strip), and -l (to process one filename at a time). None of those options are required; if you don't want them, you might at least use -t so the script will list the files it's stripping.
JP
No matter how much disk space you have, you will eventually run out. One way the system administrator can force users to clean up after themselves is to impose quotas on disk usage. Many Unixes have quota systems available: check your manual pages with a command like apropos quota.
If you're a user, how do quotas affect you? Sooner or later, you may find that you're over your quota. Quotas are maintained on a per-filesystem basis. They may be placed on disk storage (the number of blocks) and on inodes (the number of files). The quota system maintains the concept of hard and soft limits. When you exceed a soft limit, you'll get a warning (WARNING: disk quota exceeded), but you can continue to accumulate more storage. The warning will be repeated whenever you log in. At some point (i.e., after some number of sessions in which the storage stays above the soft limit), the system loses patience and refuses to allocate any more storage. You'll get a message like OVER DISK QUOTA: NO MORE DISK SPACE. At this point, you must delete files until you're again within the soft limit. Users are never allowed to exceed their hard limit. This design allows you to have large temporary files without penalty, provided that they do not occupy too much disk space long-term.
There may also be a quota on the number of files (i.e., inodes) that you can own per filesystem. It works exactly the same way; you'll get a warning when you exceed the soft limit; if you don't delete some files, the system will eventually refuse to create new files.
The quota command shows a user's quota on each filesystem where quotas have been set. With no option, it displays a line for each system where you're over quota. The -v option shows a line for each system where you have a quota. The output can be a bit confusing on systems with the automounter running, since it mounts things dynamically and uses symlinks to make things appear where you expect them, so the filesystem names may not match the directory names you're accustomed to:
$ quota Over disk quota on /home/jpeek, remove 228K within 4.0 days Over file quota on /home/jpeek, remove 13 files within 4.5 days $ quota -v Filesystem usage quota limit timeleft files quota limit timeleft /export/users 0 8000 9000 0 600 750 /export/home9 8228 8000 9000 4.0 days 613 600 750 4.5 days
In this case, the automounter has clearly mounted my home directory on /export/home9, since that shows the same information that quota showed me in the first command.
ML and JP
[1] The link entry takes a few characters in the directory where you make the link. Unless this makes the directory occupy another disk block, the space available on the disk doesn't change.
[2] Completely empty files (zero characters) don't take a block.
| CONTENTS |


Namespace GoRogue
Classes
AdjacencyRule
Class representing a method for determining which coordinates are adjacent to a given coordinate, and which directions those neighbors are in. Cannot be instantiated -- premade static instances are provided.
AdvancedLayeredSpatialMap<T>
A more complex version of LayeredSpatialMap<T> that does not require the items in it to implement IHasID, instead requiring the specification of a custom System.Collections.Generic.IEqualityComparer<T> to use for hashing and comparison of items.
AdvancedMultiSpatialMap<T>
A more complex version of MultiSpatialMap<T> that does not require the items in it to implement IHasID, instead requiring the specification of a custom System.Collections.Generic.IEqualityComparer<T> to use for hashing and comparison of items.
AdvancedSpatialMap<T>
A more complex version of SpatialMap<T> that does not require the items in it to implement IHasID, instead requiring the specification of a custom System.Collections.Generic.IEqualityComparer<T> to use for hashing and comparison of items.
BoundedRectangle
This class defines a rectanglar area, whose position is automatically "locked" to being inside a rectangular bounding box as it is changed. A typical use might be keeping track of a camera's view area.
ComponentContainer
A class implementing a flexible, type-based system for adding components to objects. To utilize it, you can either give your object that you want to have components a ComponentContainer field, or if you wish to avoid the extra field to access components, you may either inherit from ComponentContainer, or have your object implement IHasComponentsvia a backing field of type ComponentContainer.
Direction
Represents the concept of a "direction" on a grid, and "defines" the coordinate plane GoRogue uses via the YIncreasesUpward flag. Interacts with Coord and other supported library's equivalent types to allow easy translation of positions in a direction, and contains numerous helper functions for retrieving directions in various orders, getting direction closest to a line, etc.
DisjointSet
Basic representation of a disjoint set data structure.
Distance
Class representing methods of calculating distance on a grid. You cannot create instances of this class using a constructor -- instead this class contains static instances representing the various distance calculations.
Effect<TriggerArgs>
Class designed to represent any sort of in-game effect. This could be anything from a simple physical damage effect to a heal effect or permanent effects. These might include AOE effects, damage over time effects, or even potentially a special effect that simply boosts a stat.
EffectArgs
Default argument for any effect. Any class that is used as the template argument for an effect must inherit from this class.
EffectTrigger<TriggerArgs>
Represents an "event" that can automatically trigger and manage one or more Effect<TriggerArgs> instances, and acts as part of the implementation of duration in Effect.
FOV
Implements the capability to calculate a grid-based field of view for a map.
IDGenerator
Class designed as a helper for situations where you need to generate and assign a unique integer to each instance of a class, eg. for a class implementing IHasID.
ItemEventArgs<T>
Event arguments for spatial map events pertaining to an item (ItemAdded, ItemRemoved, etc.)
ItemMovedEventArgs<T>
Event arguments for spatial maps ItemMoved event.
LayeredSpatialMap<T>
ISpatialMap<T> implementation that can be used to efficiently represent multiple "layers" of objects, with each layer represented as an ISpatialMap<T> instance. It provides the regular spatial map functionality, as well as adds layer masking functionality that allow functions to operate on specific layers only.
LayerMasker
Allows convenient interpretation and creation of layer masks (bit-masks) that can be used to interact with LayeredSpatialMap<T> and Map.
Lines
Provides implementations of various (line-drawing) algorithms which are useful for for generating points closest to a line between two points on a grid.
MathHelpers
Static class consisting of mathematical "helper" functions and constants -- things like angle unit conversions, and other helpful functions.
MultiSpatialMap<T>
An implementation of ISpatialMap<T> that allows multiple items to reside at any given position at the same time. If you wish to allow only one item to reside at each location at a time, use SpatialMap<T> instead. For a situation involving different categories or layers of items, you may want to look at LayeredSpatialMap<T>.
Radius
Class representing different shapes that define the concept of a radius on a grid. You cannot create instances of this class using a constructor -- instead, this class contains static instances representing the various radius shapes.
RadiusAreaProvider
Class containing functions capable of returning all unique positions inside of a given radius and (optional) bounds.
SpatialMap<T>
An implementation of ISpatialMap<T> that allows only one item at each position at a time. If you need multiple items to be able to reside at one location at the same time, use MultiSpatialMap<T> or LayeredSpatialMap<T> instead.
Utility
Static class containing extension helper methods for various built-in C# classes, as well as a static helper method for "swapping" references.
Structs
Coord
A structure that represents a standard 2D grid coordinate. Provides numerous static functions, operators, and implicit converstions that enable common grid/position-related math and operations, as well as enable interoperability with position representations from other libraries.
Rectangle
Represents a rectangle in terms of grid squares. Provides numerous static functions that enable creation and common operations involving rectangles, as well as implicit conversion operators that enable interoperability with rectangle representations from other libraries.
Interfaces
IHasComponents
Interface for an object that has components that can be added, removed, checked for, and retrieved by type. Typically, you would implement this via a backing field of type ComponentContainer, which implements the logic for these functions.
IHasID
Interface for a class that has an ID value, typically used for items in a spatial map, or generally for purposes of hashing.
IHasLayer
Interface to be implemented by objects that will be added to a LayeredSpatialMap<T>.
IReadOnlyDisjointSet
Read-only representation of DisjointSet
IReadOnlyFOV
Read-only interface of the FOV class.
IReadOnlyLayeredSpatialMap<T>
Interface implementing only the read-only functions for LayeredSpatialMap<T>/ AdvancedLayeredSpatialMap<T>.
IReadOnlyRadiusAreaProvider
Read-only interface of RadiusAreaProvider.
IReadOnlySpatialMap<T>
Implements the read-only interface of ISpatialMap<T>.
ISpatialMap<T>
Interface common to spatial map implementations provided with GoRogue, which are designed to be a convenient way to store items that reside on a map. TLDR; If you're about to use a List to store all the objects on a map, consider using a (GoRogue-provided) ISpatialMap implementation instead.
ISpatialTuple<T>
Interface specifying return type for item-location pairs in spatial maps.
Enums
AdjacencyRule.Types
Enum representing AdjacencyRule types. Each AdjacencyRule instance has a Type field which contains the corresponding value from this enum. Useful for easy mapping of AdjacencyRule types to a primitive type (for cases like a switch statement).
Direction.Types
Enum representing Direction types. Each Direction instance has a Type field which contains the corresponding value from this enum. Useful for easy mapping of Direction types to a primitive type (for cases like a switch statement).
Distance.Types
Enum representing Distance types. Each Distance instance has a Type field which contains the corresponding value from this enum. Useful for easy mapping of Distance types to a primitive type (for cases like a switch statement).
Lines.Algorithm
Various supported line-drawing algorithms.
Radius.Types
Enum representing Radius types. Each Radius instance has a Type field which contains the corresponding value from this enum. Useful for easy mapping of Radius types to a primitive type (for cases like a switch statement).


im sorry if this tha wrong place to ask for this but i dont know where else to turn....i have a ps3 , i even bought tha game at wal mrt ,but when i sign in to battle log im in pc forum so i cant activate my soldier to check my stats...
BrainFooLong
Administrator
Wrong place. Goto Battlelog Forums or EA Support.
BBLog have nothing to do with this.
Greetz


Low Below Average Average Above Average High
Question 2: Physical: strength, agility and dexterity
Question 3: Thinking: working with new ideas and creative thinking
Question 4: Attention: focus on the problem at hand
Question 5: Judgment: making clear, decisive decisions
Question 6: Communication: listening, speaking and working with others
Question 7: Mechanical: working with tools and equipment
Low Below Average High Above Average High
Question 8: People: interaction with others, ability to train and counsel
Question 9: Management: planning, proper use of time and resources
Question 10: Medical: biology and health care
Question 11: Business: economics, finance and marketing
Question 12: Art: painting, drawing and sculpture
Question 13: Culinary: food and drinks
Question 14: Computers: programming and information technology
Question 15: Law: legal and criminal justice


Victor Hugo
A Hugo boilerplate for creating truly epic websites
This is a boilerplate for using Hugo as a static site generator and Webpack as your asset pipeline. Victor Hugo setup to use PostCSS and Babel for CSS and JavaScript compiling/transpiling. This project is released under the MIT license. Please make sure you understand its implications and guarantees.


Woops..! its Looks Like You are Currently offline. Please Try again as soon as your Network Connection is Available again.