Wakulla, A Story of Adventure in Florida
by Kirk Munroe
"A Queer Christmas Day"
Additional Information
- Year Published: 1886
- Language: English
- Country of Origin: United States of America
- Source: Munroe, K. (1886). Wakulla, A Story of Adventure in Florida. New York: Harper.
-
Readability:
- Flesch–Kincaid Level: 9.0
- Word Count: 2,327
- Genre: Adventure
- Keywords: florida
- ✎ Cite This
Downloads
All night long the Nancy Bell sailed back and forth within sight of the light that marked the mouth of the river. Soon after daylight a pilot-boat was seen approaching her in answer to the signal which was flying from the main rigging. As the boat ran alongside, a colored pilot clambered to the deck and declared it did him good to see a big schooner waiting to come into the St. Mark’s once more.
“Uster be a plenty of ‘em,” said he to “Captain Li,” “but dey’s scurcer’n gole dollars now-adays, an’ I’se proud to see ‘em comin’ ag’in.”
By the time breakfast was over and the Elmers came on deck, they found the schooner running rapidly up a broad river, between wide expanses of low salt-marshes, bounded by distant pine forests, and studded here and there with groups of cabbage palms. The channel was a regular zig-zag, and they ran now to one side and then far over to the other to escape the coral reefs and oyster bars with which it is filled. This occupied much time; but the breeze was fresh, and within an hour they had run eight miles up the river, and were passing the ruins of the old Spanish Fort of St. Mark’s. A few minutes later sails were lowered, and the schooner was moored to one of the rotten old wharves that still remain to tell of St. Mark’s former glory.
“And is this St. Mark’s?” asked Mrs. Elmer, looking with a feeling of keen disappointment at the dozen or so tumble-down frame buildings that, perched on piles above the low, wet land, looked like dilapidated old men with shaky legs, and formed all that was to be seen of the town.
“Yes, miss,” answered the colored pilot, who seemed to consider her question addressed to him. “Dis yere’s St. Mark’s, or what de gales has lef’ of hit. ‘Pears like dey’s been mighty hard on de ole town, sence trade fell off, an’ mos’ of de folkses moved away. Uster be wharves all along yere, an’ cotton-presses, an’ big war’houses, an’ plenty ships in de ribber; but now dey’s all gone. Dem times we uster hab fo’ trains of kyars a day; but now dere’s only one train comes tree times in de week, an’ hit’s only got one kyar. Ole St. Mark’s a-seein’ bad times now, for sho.”
As soon as he could get ashore, Mr. Elmer, accompanied by Mark and the captain, went up into the village to find out what he could regarding their destination and future movements. In about an hour he returned, bringing a package of letters from the post-office, and the information that Uncle Christopher Bangs’s place was at Wakulla, some six miles farther up the river. As the river above St. Mark’s is quite crooked, and bordered on both sides by dense forests, and as no steam-tug could be had, the captain did not care to attempt to carry the schooner any farther up. Mr. Elmer had therefore chartered a large, flat-bottomed lighter, or scow, to carry to Wakulla the cargo of household goods, tools, building material, etc., that they had brought with them.
As “Captain Li” was anxious to proceed on his voyage to Pensacola as quickly as possible, the lighter was at once brought alongside the schooner, and the work of discharging the Elmers’ goods into her was begun.
“By-the-way, Mark,” said Mr. Elmer, as the schooner’s hatches were removed, “I am just reminded that this is Christmas-day, and that there is a present down in the hold for you from your Uncle Christmas. It will be one of the first things taken out, so see if you can recognize it.”
He had hardly spoken before the sailors, who had gone down into the hold, passed carefully up to those on deck a beautiful birch- bark canoe, with the name Ruth painted on its bows.
“That’s it, Father! That’s it! I’m sure it is. Oh! Isn’t she a beauty?” shouted Mark, wild with delight. “Oh! Father, how did he know just exactly what I wanted most?” and the excited boy rushed down into the cabin to beg his mother and Ruth to come on deck and see his Christmas present.
The canoe was followed by two paddles painted a bright vermilion, and as they were placed in her, and she was laid to one side of the deck, she was indeed as pretty a little craft as can be imagined, and one that would delight any boy’s heart.
“I knew we were going to live near a river, my dear,” said Mr. Elmer, in answer to his wife’s anxious expression as she looked at the canoe, “and as Mark is a good swimmer and very careful in boats, I thought a canoe would afford him great pleasure, and probably prove very useful to all of us. So when Uncle Christopher asked me what I thought the boy would like most for a Christmas present, I told him a canoe.”
“Well, I hope it will prove safe,” sighed Mrs. Elmer; “but I wish it were flat-bottomed, and built of thick boards instead of that thin bark.”
“Oh, Mother!” said Mark, “you might as well wish it were a canal- boat at once.”
“Yes, I believe canal-boats are generally considered safer than canoes,” answered his mother with a smile. “By-the-way, Mark”—and she turned to her husband—“one of the letters you brought was from Uncle Christopher, and he says he thinks he forgot to tell us that there is a house on his place, which he hopes we will find in a fit condition to occupy.”
 This material was now being transferred from the schooner to the lighter, and now it seemed almost a pity to have brought it; still they were very glad to learn that they were likely to find a house all ready to move into.
It wanted but two hours of sundown when the last of the Elmers’ goods were stowed in the lighter, and as there was nothing to detain him any longer, “Captain Li” said he should take advantage of the ebb tide that night to drop down the river and get started for Pensacola. As rowing and poling the heavy lighter up the river would at best prove but slow work, and as there was no hotel or place for them to stay in St. Mark’s, Mr. Elmer thought they too had better make a start, and take advantage of the last of the flood tide and what daylight still remained.
So good-byes were exchanged, and feeling very much as though they were leaving home for the second time, the Elmers left the comfortable cabin that had sheltered them for nearly a month. Followed by Jan, they went on board their new craft, and the lines were cast off. The crew of four strong colored men bent over the long sweeps, and followed by a hearty cheer from the crew of the schooner, the scow moved slowly up the river. In a few minutes a bend hid St. Mark’s and the tall masts of the Nancy Bell from sight, and on either side of them appeared nothing but unbroken forest.
The river seemed narrow and dark after the open sea to which the Elmers had been so long accustomed, and from its banks the dense growth of oak, cedar, magnolia, palm, bay, cypress, elm, and sweet gum trees, festooned with moss, and bound together with a net-work of vines, rose like walls, shutting out the sunlight. Strange water-fowl, long-legged and long-billed, flew screaming away as they advanced, and quick splashes in the water ahead of them told of the presence of other animal life.
At sunset they were nearly two miles from St. Mark’s, and opposite a cleared spot on the bank, where was piled a quantity of light- wood or pitch-pine. Here the captain and owner of the lighter, who was a young white man named Oliver Johnson, proposed that they should tie up for the night.
To this Mr. Elmer consented, and as soon as the boat was made fast to the bank, active preparations were begun for cooking supper, and for making everything as snug and comfortable as possible.
A large sail was stretched across some poles, in the form of a tent, over the after-part of the lighter, and beneath this two comfortable beds were made up from the abundant supply of mattresses and blankets belonging to the Elmers. Jan Jansen and Captain Johnson, who, Mark said, must be related, as their names were the same, spread their blankets in the forward end of the boat. On shore the negro crew built for themselves a thatched lean-to of poles and palm-leaves beside the fire, that was already throwing its cheerful light across the dark surface of the river.
While the men were busy arranging the shelters and bedding, Mrs. Elmer and Ruth, assisted by one of the negroes, were cooking supper over a bed of coals that had been raked from the fire. A huge pot of coffee sent forth clouds of fragrant steam, and in two frying-pans some freshly caught fish sizzled and browned in a most gratifying and appetizing manner. In a couple of kettles hung over the fire hominy and sweet potatoes bubbled, boiled, and tried to outdo each other in getting done. Fresh-made bread and a good supply of butter had been brought from the schooner. When the supper was all ready, and spread out on a green table-cloth of palm-leaves, Mark and Ruth declared that this picnic was even jollier than the one on the island of the Florida Reef, and that this was after all one of the very best Christmases they had ever known.
After supper, and when the dishes had all been washed and put away, the Elmers, Captain Johnson, and Jan sought the shelter of the canvas awning from the heavy night-dew which had begun to fall as soon as the sun went down. They lifted the sides, so that they could look out and see the fire around which the crew were gathered. After a while one of these started a plaintive negro melody, which sounded very sweetly through the still air. The others took it up, and they sang for an hour or more, greatly to the delight of the children, to whom such music was new. Many of the words were composed as they sang, and Mark and Ruth could not help laughing at some of them, which, though sung very soberly, sounded funny. One song which they afterwards remembered was:
“Oh, dey put John on de islan’
When de Bridegroom come;
Yes, dey put John on de islan’
When de Bridegroom come;
An’ de rabens come an’ fed him
When de Bridegroom come;
Yes, de rabens come an’ fed him
When de Bridegroom come.
An’ five of dem was wise
When de Bridegroom come;
Yes, five of dem was wise
When de Bridegroom come;
An’ five of dem was foolish
When de Bridegroom come;
Yes, five of dem was foolish
When de Bridegroom come.
Oh, gib us of yo’ ile
When de Bridegroom come;
Oh, gib us of yo’ ile
When de Bridegroom come;
Fo’ you’ll nebber get to heaben
When de Bridegroom come;
No, you’ll nebber get to heaben
When de Bridegroom come;
Aless you’s ile a-plenty
When de Bridegroom come;
Aless you’s ile a-plenty
When de Bridegroom come.”
In the midst of the singing a voice called out from the treetops,
“Who, who, who, who’s there?” or at least so it sounded.
Immediately the singing stopped, and one of the negroes answered,
“Some folkses from de Norf, Marse Owl, an’ Cap’n Johnsin, an’ me, an’ Homer, an’ Virgil, an’ Pete.”
“What does he mean by that?” asked Mr. Elmer of the captain.
“Oh,” answered he, “it’s one of their superstitions that they’ll have bad luck if they don’t answer an owl politely when he asks ‘Who’s there?’ and give the names of all the party, if they know them.”
Soon after this all hands sought their blankets, good-nights were said, the fire died down, and all was quiet in the camp, though several times some sleepy negro roused himself sufficiently to answer the owl’s repeated question of “Who’s there?”
It must have been nearly midnight when the camp was startled by a crash, a series of smothered cries, and a loud splashing in the water. It was evident that something serious had happened, but what it was no one could make out in the darkness.


Smart UI with Microsoft Graph
Step 1 - Select workbook
Select Workbook
{{selectedWorkbook}}
Step 2 - Select user
Search organization people
Step 3 - Send email
{{foo}}
Send Email


Lionel Messi
The greatest football player in the world
- 1987- Born in Rosario, Argentina
- 2003-05- Rise to the first team
- 2005-08- Becoming a starting eleven player
- 2009-11- Sustained success
- 2012- A record-breaking year
- 2013-2014- Messidependencia
- 2014-15- A historic treble
- 2015-16- Domestic success
- 2016-17- Fourth Golden Boot
- 2017-18- Domestic double and a record fifth Golden Boot
- 2018-19- Barcelona captain, tenth La Liga title, and a record sixth Golden Boot
- 2019-20- A record sixth Ballon d'Or
"Money is not a motivating factor…My motivation comes from playing the game I love. If I wasn’t paid to be a professional footballer, I would willingly play for nothing"
-Lionel Messi
More Info


--- layout: example categories: example/v1.0.0 version: v1.0.0 title: Omnivore with a custom layer description: Pass additional data as a third argument in addition to what's loaded with Omnivore. tags: - omnivore ---


The metro Detroit area has five distinct destination districts, perfect for tourism and travel: everything from a lively urban vibe to wide-open scenic spaces.
Here, We Work Hard and Play Hard.


public class DataFrame
extends java.lang.Object
implements scala.Serializable
A DataFrame
is equivalent to a relational table in Spark SQL. The following example creates
a DataFrame
by pointing Spark SQL to a Parquet data set.
val people = sqlContext.read.parquet("...") // in Scala
DataFrame people = sqlContext.read().parquet("...") // in Java
Once created, it can be manipulated using the various domain-specific-language (DSL) functions
defined in: DataFrame
(this class), Column
, and functions
.
To select a column from the data frame, use apply
method in Scala and col
in Java.
val ageCol = people("age") // in Scala
Column ageCol = people.col("age") // in Java
Note that the Column
type can also be manipulated through its various functions.
// The following creates a new column that increases everybody's age by 10.
people("age") + 10 // in Scala
people.col("age").plus(10); // in Java
A more concrete example in Scala:
// To create DataFrame using SQLContext
val people = sqlContext.read.parquet("...")
val department = sqlContext.read.parquet("...")
people.filter("age > 30")
.join(department, people("deptId") === department("id"))
.groupBy(department("name"), "gender")
.agg(avg(people("salary")), max(people("age")))
and in Java:
// To create DataFrame using SQLContext
DataFrame people = sqlContext.read().parquet("...");
DataFrame department = sqlContext.read().parquet("...");
people.filter("age".gt(30))
.join(department, people.col("deptId").equalTo(department("id")))
.groupBy(department.col("name"), "gender")
.agg(avg(people.col("salary")), max(people.col("age")));
| Constructor and Description |
|---|
DataFrame(SQLContext sqlContext,
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan)
A constructor that automatically analyzes the logical plan.
|
| Modifier and Type | Method and Description |
|---|---|
DataFrame |
agg(Column expr,
Column... exprs)
Aggregates on the entire
DataFrame without groups. |
DataFrame |
agg(Column expr,
scala.collection.Seq<Column> exprs)
Aggregates on the entire
DataFrame without groups. |
DataFrame |
agg(scala.collection.immutable.Map<java.lang.String,java.lang.String> exprs)
(Scala-specific) Aggregates on the entire
DataFrame without groups. |
DataFrame |
agg(java.util.Map<java.lang.String,java.lang.String> exprs)
(Java-specific) Aggregates on the entire
DataFrame without groups. |
DataFrame |
agg(scala.Tuple2<java.lang.String,java.lang.String> aggExpr,
scala.collection.Seq<scala.Tuple2<java.lang.String,java.lang.String>> aggExprs)
(Scala-specific) Aggregates on the entire
DataFrame without groups. |
Column |
apply(java.lang.String colName)
Selects column based on the column name and return it as a
Column . |
DataFrame |
as(java.lang.String alias)
Returns a new
DataFrame with an alias set. |
DataFrame |
as(scala.Symbol alias)
(Scala-specific) Returns a new
DataFrame with an alias set. |
DataFrame |
cache() |
DataFrame |
coalesce(int numPartitions)
Returns a new
DataFrame that has exactly numPartitions partitions. |
Column |
col(java.lang.String colName)
Selects column based on the column name and return it as a
Column . |
Row[] |
collect()
|
java.util.List<Row> |
collectAsList()
|
java.lang.String[] |
columns()
Returns all column names as an array.
|
long |
count()
Returns the number of rows in the
DataFrame . |
void |
createJDBCTable(java.lang.String url,
java.lang.String table,
boolean allowExisting)
Deprecated.
As of 1.340, replaced by
write().jdbc() . |
GroupedData |
cube(Column... cols)
Create a multi-dimensional cube for the current
DataFrame using the specified columns,
so we can run aggregation on them. |
GroupedData |
cube(scala.collection.Seq<Column> cols)
Create a multi-dimensional cube for the current
DataFrame using the specified columns,
so we can run aggregation on them. |
GroupedData |
cube(java.lang.String col1,
scala.collection.Seq<java.lang.String> cols)
Create a multi-dimensional cube for the current
DataFrame using the specified columns,
so we can run aggregation on them. |
GroupedData |
cube(java.lang.String col1,
java.lang.String... cols)
Create a multi-dimensional cube for the current
DataFrame using the specified columns,
so we can run aggregation on them. |
DataFrame |
describe(scala.collection.Seq<java.lang.String> cols)
Computes statistics for numeric columns, including count, mean, stddev, min, and max.
|
DataFrame |
describe(java.lang.String... cols)
Computes statistics for numeric columns, including count, mean, stddev, min, and max.
|
DataFrame |
distinct()
|
DataFrame |
drop(Column col)
Returns a new
DataFrame with a column dropped. |
DataFrame |
drop(java.lang.String colName)
Returns a new
DataFrame with a column dropped. |
DataFrame |
dropDuplicates()
|
DataFrame |
dropDuplicates(scala.collection.Seq<java.lang.String> colNames)
(Scala-specific) Returns a new
DataFrame with duplicate rows removed, considering only
the subset of columns. |
DataFrame |
dropDuplicates(java.lang.String[] colNames)
Returns a new
DataFrame with duplicate rows removed, considering only
the subset of columns. |
scala.Tuple2<java.lang.String,java.lang.String>[] |
dtypes()
Returns all column names and their data types as an array.
|
DataFrame |
except(DataFrame other)
Returns a new
DataFrame containing rows in this frame but not in another frame. |
void |
explain()
Only prints the physical plan to the console for debugging purposes.
|
void |
explain(boolean extended)
Prints the plans (logical and physical) to the console for debugging purposes.
|
<A extends scala.Product> |
explode(scala.collection.Seq<Column> input,
scala.Function1<Row,scala.collection.TraversableOnce<A>> f,
scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
(Scala-specific) Returns a new
DataFrame where each row has been expanded to zero or more
rows by the provided function. |
<A,B> DataFrame |
explode(java.lang.String inputColumn,
java.lang.String outputColumn,
scala.Function1<A,scala.collection.TraversableOnce<B>> f,
scala.reflect.api.TypeTags.TypeTag<B> evidence$2)
(Scala-specific) Returns a new
DataFrame where a single column has been expanded to zero
or more rows by the provided function. |
DataFrame |
filter(Column condition)
Filters rows using the given condition.
|
DataFrame |
filter(java.lang.String conditionExpr)
Filters rows using the given SQL expression.
|
Row |
first()
Returns the first row.
|
<R> RDD<R> |
flatMap(scala.Function1<Row,scala.collection.TraversableOnce<R>> f,
scala.reflect.ClassTag<R> evidence$4)
Returns a new RDD by first applying a function to all rows of this
DataFrame ,
and then flattening the results. |
void |
foreach(scala.Function1<Row,scala.runtime.BoxedUnit> f)
Applies a function
f to all rows. |
void |
foreachPartition(scala.Function1<scala.collection.Iterator<Row>,scala.runtime.BoxedUnit> f)
Applies a function f to each partition of this
DataFrame . |
GroupedData |
groupBy(Column... cols)
Groups the
DataFrame using the specified columns, so we can run aggregation on them. |
GroupedData |
groupBy(scala.collection.Seq<Column> cols)
Groups the
DataFrame using the specified columns, so we can run aggregation on them. |
GroupedData |
groupBy(java.lang.String col1,
scala.collection.Seq<java.lang.String> cols)
Groups the
DataFrame using the specified columns, so we can run aggregation on them. |
GroupedData |
groupBy(java.lang.String col1,
java.lang.String... cols)
Groups the
DataFrame using the specified columns, so we can run aggregation on them. |
Row |
head()
Returns the first row.
|
Row[] |
head(int n)
Returns the first
n rows. |
java.lang.String[] |
inputFiles()
Returns a best-effort snapshot of the files that compose this DataFrame.
|
void |
insertInto(java.lang.String tableName)
Deprecated.
As of 1.4.0, replaced by
write().mode(SaveMode.Append).saveAsTable(tableName) . |
void |
insertInto(java.lang.String tableName,
boolean overwrite)
Deprecated.
As of 1.4.0, replaced by
write().mode(SaveMode.Append|SaveMode.Overwrite).saveAsTable(tableName) . |
void |
insertIntoJDBC(java.lang.String url,
java.lang.String table,
boolean overwrite)
Deprecated.
As of 1.4.0, replaced by
write().jdbc() . |
DataFrame |
intersect(DataFrame other)
Returns a new
DataFrame containing rows only in both this frame and another frame. |
boolean |
isLocal()
Returns true if the
collect and take methods can be run locally
(without any Spark executors). |
JavaRDD<Row> |
javaRDD()
|
protected JavaRDD<byte[]> |
javaToPython()
Converts a JavaRDD to a PythonRDD.
|
DataFrame |
join(DataFrame right)
Cartesian join with another
DataFrame . |
DataFrame |
join(DataFrame right,
Column joinExprs)
Inner join with another
DataFrame , using the given join expression. |
DataFrame |
join(DataFrame right,
Column joinExprs,
java.lang.String joinType)
Join with another
DataFrame , using the given join expression. |
DataFrame |
join(DataFrame right,
scala.collection.Seq<java.lang.String> usingColumns)
Inner equi-join with another
DataFrame using the given columns. |
DataFrame |
join(DataFrame right,
java.lang.String usingColumn)
Inner equi-join with another
DataFrame using the given column. |
DataFrame |
limit(int n)
Returns a new
DataFrame by taking the first n rows. |
protected org.apache.spark.sql.catalyst.plans.logical.LogicalPlan |
logicalPlan() |
<R> RDD<R> |
map(scala.Function1<Row,R> f,
scala.reflect.ClassTag<R> evidence$3)
Returns a new RDD by applying a function to all rows of this DataFrame.
|
<R> RDD<R> |
mapPartitions(scala.Function1<scala.collection.Iterator<Row>,scala.collection.Iterator<R>> f,
scala.reflect.ClassTag<R> evidence$5)
Returns a new RDD by applying a function to each partition of this DataFrame.
|
DataFrameNaFunctions |
na()
Returns a
DataFrameNaFunctions for working with missing data. |
protected scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> |
numericColumns() |
DataFrame |
orderBy(Column... sortExprs)
Returns a new
DataFrame sorted by the given expressions. |
DataFrame |
orderBy(scala.collection.Seq<Column> sortExprs)
Returns a new
DataFrame sorted by the given expressions. |
DataFrame |
orderBy(java.lang.String sortCol,
scala.collection.Seq<java.lang.String> sortCols)
Returns a new
DataFrame sorted by the given expressions. |
DataFrame |
orderBy(java.lang.String sortCol,
java.lang.String... sortCols)
Returns a new
DataFrame sorted by the given expressions. |
DataFrame |
persist() |
DataFrame |
persist(StorageLevel newLevel) |
void |
printSchema()
Prints the schema to the console in a nice tree format.
|
SQLContext.QueryExecution |
queryExecution() |
DataFrame[] |
randomSplit(double[] weights)
Randomly splits this
DataFrame with the provided weights. |
DataFrame[] |
randomSplit(double[] weights,
long seed)
Randomly splits this
DataFrame with the provided weights. |
RDD<Row> |
rdd()
|
void |
registerTempTable(java.lang.String tableName)
Registers this
DataFrame as a temporary table using the given name. |
DataFrame |
repartition(int numPartitions)
Returns a new
DataFrame that has exactly numPartitions partitions. |
protected org.apache.spark.sql.catalyst.expressions.NamedExpression |
resolve(java.lang.String colName) |
GroupedData |
rollup(Column... cols)
Create a multi-dimensional rollup for the current
DataFrame using the specified columns,
so we can run aggregation on them. |
GroupedData |
rollup(scala.collection.Seq<Column> cols)
Create a multi-dimensional rollup for the current
DataFrame using the specified columns,
so we can run aggregation on them. |
GroupedData |
rollup(java.lang.String col1,
scala.collection.Seq<java.lang.String> cols)
Create a multi-dimensional rollup for the current
DataFrame using the specified columns,
so we can run aggregation on them. |
GroupedData |
rollup(java.lang.String col1,
java.lang.String... cols)
Create a multi-dimensional rollup for the current
DataFrame using the specified columns,
so we can run aggregation on them. |
DataFrame |
sample(boolean withReplacement,
double fraction)
Returns a new
DataFrame by sampling a fraction of rows, using a random seed. |
DataFrame |
sample(boolean withReplacement,
double fraction,
long seed)
Returns a new
DataFrame by sampling a fraction of rows. |
void |
save(java.lang.String path)
Deprecated.
As of 1.4.0, replaced by
write().save(path) . |
void |
save(java.lang.String path,
SaveMode mode)
Deprecated.
As of 1.4.0, replaced by
write().mode(mode).save(path) . |
void |
save(java.lang.String source,
SaveMode mode,
java.util.Map<java.lang.String,java.lang.String> options)
Deprecated.
As of 1.4.0, replaced by
write().format(source).mode(mode).options(options).save(path) . |
void |
save(java.lang.String source,
SaveMode mode,
scala.collection.immutable.Map<java.lang.String,java.lang.String> options)
Deprecated.
As of 1.4.0, replaced by
write().format(source).mode(mode).options(options).save(path) . |
void |
save(java.lang.String path,
java.lang.String source)
Deprecated.
As of 1.4.0, replaced by
write().format(source).save(path) . |
void |
save(java.lang.String path,
java.lang.String source,
SaveMode mode)
Deprecated.
As of 1.4.0, replaced by
write().format(source).mode(mode).save(path) . |
void |
saveAsParquetFile(java.lang.String path)
Deprecated.
As of 1.4.0, replaced by
write().parquet() . |
void |
saveAsTable(java.lang.String tableName)
Deprecated.
As of 1.4.0, replaced by
write().saveAsTable(tableName) . |
void |
saveAsTable(java.lang.String tableName,
SaveMode mode)
Deprecated.
As of 1.4.0, replaced by
write().mode(mode).saveAsTable(tableName) . |
void |
saveAsTable(java.lang.String tableName,
java.lang.String source)
Deprecated.
As of 1.4.0, replaced by
write().format(source).saveAsTable(tableName) . |
void |
saveAsTable(java.lang.String tableName,
java.lang.String source,
SaveMode mode)
Deprecated.
As of 1.4.0, replaced by
write().mode(mode).saveAsTable(tableName) . |
void |
saveAsTable(java.lang.String tableName,
java.lang.String source,
SaveMode mode,
java.util.Map<java.lang.String,java.lang.String> options)
Deprecated.
As of 1.4.0, replaced by
write().format(source).mode(mode).options(options).saveAsTable(tableName) . |
void |
saveAsTable(java.lang.String tableName,
java.lang.String source,
SaveMode mode,
scala.collection.immutable.Map<java.lang.String,java.lang.String> options)
Deprecated.
As of 1.4.0, replaced by
write().format(source).mode(mode).options(options).saveAsTable(tableName) . |
StructType |
schema()
Returns the schema of this
DataFrame . |
DataFrame |
select(Column... cols)
Selects a set of column based expressions.
|
DataFrame |
select(scala.collection.Seq<Column> cols)
Selects a set of column based expressions.
|
DataFrame |
select(java.lang.String col,
scala.collection.Seq<java.lang.String> cols)
Selects a set of columns.
|
DataFrame |
select(java.lang.String col,
java.lang.String... cols)
Selects a set of columns.
|
DataFrame |
selectExpr(scala.collection.Seq<java.lang.String> exprs)
Selects a set of SQL expressions.
|
DataFrame |
selectExpr(java.lang.String... exprs)
Selects a set of SQL expressions.
|
void |
show()
Displays the top 20 rows of
DataFrame in a tabular form. |
void |
show(boolean truncate)
Displays the top 20 rows of
DataFrame in a tabular form. |
void |
show(int numRows)
Displays the
DataFrame in a tabular form. |
void |
show(int numRows,
boolean truncate)
Displays the
DataFrame in a tabular form. |
DataFrame |
sort(Column... sortExprs)
Returns a new
DataFrame sorted by the given expressions. |
DataFrame |
sort(scala.collection.Seq<Column> sortExprs)
Returns a new
DataFrame sorted by the given expressions. |
DataFrame |
sort(java.lang.String sortCol,
scala.collection.Seq<java.lang.String> sortCols)
Returns a new
DataFrame sorted by the specified column, all in ascending order. |
DataFrame |
sort(java.lang.String sortCol,
java.lang.String... sortCols)
Returns a new
DataFrame sorted by the specified column, all in ascending order. |
SQLContext |
sqlContext() |
DataFrameStatFunctions |
stat()
Returns a
DataFrameStatFunctions for working statistic functions support. |
Row[] |
take(int n)
Returns the first
n rows in the DataFrame . |
DataFrame |
toDF()
Returns the object itself.
|
DataFrame |
toDF(scala.collection.Seq<java.lang.String> colNames)
Returns a new
DataFrame with columns renamed. |
DataFrame |
toDF(java.lang.String... colNames)
Returns a new
DataFrame with columns renamed. |
JavaRDD<Row> |
toJavaRDD()
|
RDD<java.lang.String> |
toJSON()
Returns the content of the
DataFrame as a RDD of JSON strings. |
DataFrame |
toSchemaRDD()
Deprecated.
As of 1.3.0, replaced by
toDF() . |
java.lang.String |
toString() |
DataFrame |
unionAll(DataFrame other)
Returns a new
DataFrame containing union of rows in this frame and another frame. |
DataFrame |
unpersist() |
DataFrame |
unpersist(boolean blocking) |
DataFrame |
where(Column condition)
Filters rows using the given condition.
|
DataFrame |
where(java.lang.String conditionExpr)
Filters rows using the given SQL expression.
|
DataFrame |
withColumn(java.lang.String colName,
Column col)
Returns a new
DataFrame by adding a column or replacing the existing column that has
the same name. |
DataFrame |
withColumnRenamed(java.lang.String existingName,
java.lang.String newName)
Returns a new
DataFrame with a column renamed. |
DataFrameWriter |
write()
:: Experimental ::
Interface for saving the content of the
DataFrame out into external storage. |
public DataFrame(SQLContext sqlContext, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan)
This reports error eagerly as the DataFrame
is constructed, unless
SQLConf.dataFrameEagerAnalysis
is turned off.
sqlContext
- (undocumented)logicalPlan
- (undocumented)public DataFrame toDF(java.lang.String... colNames)
DataFrame
with columns renamed. This can be quite convenient in conversion
from a RDD of tuples into a DataFrame
with meaningful names. For example:
val rdd: RDD[(Int, String)] = ...
rdd.toDF() // this implicit conversion creates a DataFrame with column name _1 and _2
rdd.toDF("id", "name") // this creates a DataFrame with column name "id" and "name"
colNames
- (undocumented)public DataFrame sort(java.lang.String sortCol, java.lang.String... sortCols)
DataFrame
sorted by the specified column, all in ascending order.
// The following 3 are equivalent
df.sort("sortcol")
df.sort($"sortcol")
df.sort($"sortcol".asc)
sortCol
- (undocumented)sortCols
- (undocumented)public DataFrame sort(Column... sortExprs)
DataFrame
sorted by the given expressions. For example:
df.sort($"col1", $"col2".desc)
sortExprs
- (undocumented)public DataFrame orderBy(java.lang.String sortCol, java.lang.String... sortCols)
DataFrame
sorted by the given expressions.
This is an alias of the sort
function.sortCol
- (undocumented)sortCols
- (undocumented)public DataFrame orderBy(Column... sortExprs)
DataFrame
sorted by the given expressions.
This is an alias of the sort
function.sortExprs
- (undocumented)public DataFrame select(Column... cols)
df.select($"colA", $"colB" + 1)
cols
- (undocumented)public DataFrame select(java.lang.String col, java.lang.String... cols)
select
that can only select
existing columns using column names (i.e. cannot construct expressions).
// The following two are equivalent:
df.select("colA", "colB")
df.select($"colA", $"colB")
col
- (undocumented)cols
- (undocumented)public DataFrame selectExpr(java.lang.String... exprs)
select
that accepts
SQL expressions.
df.selectExpr("colA", "colB as newName", "abs(colC)")
exprs
- (undocumented)public GroupedData groupBy(Column... cols)
DataFrame
using the specified columns, so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns grouped by department.
df.groupBy($"department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)public GroupedData rollup(Column... cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns rolluped by department and group.
df.rollup($"department", $"group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)public GroupedData cube(Column... cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns cubed by department and group.
df.cube($"department", $"group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)public GroupedData groupBy(java.lang.String col1, java.lang.String... cols)
DataFrame
using the specified columns, so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of groupBy that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns grouped by department.
df.groupBy("department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)public GroupedData rollup(java.lang.String col1, java.lang.String... cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of rollup that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns rolluped by department and group.
df.rollup("department", "group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)public GroupedData cube(java.lang.String col1, java.lang.String... cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of cube that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns cubed by department and group.
df.cube("department", "group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)public DataFrame agg(Column expr, Column... exprs)
DataFrame
without groups.
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(max($"age"), avg($"salary"))
df.groupBy().agg(max($"age"), avg($"salary"))
expr
- (undocumented)exprs
- (undocumented)public DataFrame describe(java.lang.String... cols)
This function is meant for exploratory data analysis, as we make no guarantee about the
backward compatibility of the schema of the resulting DataFrame
. If you want to
programmatically compute summary statistics, use the agg
function instead.
df.describe("age", "height").show()
// output:
// summary age height
// count 10.0 10.0
// mean 53.3 178.05
// stddev 11.6 15.7
// min 18.0 163.0
// max 92.0 192.0
cols
- (undocumented)public SQLContext sqlContext()
public SQLContext.QueryExecution queryExecution()
protected org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan()
protected org.apache.spark.sql.catalyst.expressions.NamedExpression resolve(java.lang.String colName)
protected scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> numericColumns()
public java.lang.String toString()
toString
in class java.lang.Object
public DataFrame toDF()
public DataFrame toDF(scala.collection.Seq<java.lang.String> colNames)
DataFrame
with columns renamed. This can be quite convenient in conversion
from a RDD of tuples into a DataFrame
with meaningful names. For example:
val rdd: RDD[(Int, String)] = ...
rdd.toDF() // this implicit conversion creates a DataFrame with column name _1 and _2
rdd.toDF("id", "name") // this creates a DataFrame with column name "id" and "name"
colNames
- (undocumented)public StructType schema()
DataFrame
.public scala.Tuple2<java.lang.String,java.lang.String>[] dtypes()
public java.lang.String[] columns()
public void printSchema()
public void explain(boolean extended)
extended
- (undocumented)public void explain()
public boolean isLocal()
collect
and take
methods can be run locally
(without any Spark executors).public void show(int numRows)
DataFrame
in a tabular form. Strings more than 20 characters will be
truncated, and all cells will be aligned right. For example:
year month AVG('Adj Close) MAX('Adj Close)
1980 12 0.503218 0.595103
1981 01 0.523289 0.570307
1982 02 0.436504 0.475256
1983 03 0.410516 0.442194
1984 04 0.450090 0.483521
numRows
- Number of rows to show
public void show()
DataFrame
in a tabular form. Strings more than 20 characters
will be truncated, and all cells will be aligned right.public void show(boolean truncate)
DataFrame
in a tabular form.
truncate
- Whether truncate long strings. If true, strings more than 20 characters will
be truncated and all cells will be aligned right
public void show(int numRows, boolean truncate)
DataFrame
in a tabular form. For example:
year month AVG('Adj Close) MAX('Adj Close)
1980 12 0.503218 0.595103
1981 01 0.523289 0.570307
1982 02 0.436504 0.475256
1983 03 0.410516 0.442194
1984 04 0.450090 0.483521
numRows
- Number of rows to showtruncate
- Whether truncate long strings. If true, strings more than 20 characters will
be truncated and all cells will be aligned right
public DataFrameNaFunctions na()
DataFrameNaFunctions
for working with missing data.
// Dropping rows containing any null values.
df.na.drop()
public DataFrameStatFunctions stat()
DataFrameStatFunctions
for working statistic functions support.
// Finding frequent items in column with name 'a'.
df.stat.freqItems(Seq("a"))
public DataFrame join(DataFrame right)
DataFrame
.
Note that cartesian joins are very expensive without an extra filter that can be pushed down.
right
- Right side of the join operation.public DataFrame join(DataFrame right, java.lang.String usingColumn)
DataFrame
using the given column.
Different from other join functions, the join column will only appear once in the output,
i.e. similar to SQL's JOIN USING
syntax.
// Joining df1 and df2 using the column "user_id"
df1.join(df2, "user_id")
Note that if you perform a self-join using this function without aliasing the input
DataFrame
s, you will NOT be able to reference any columns after the join, since
there is no way to disambiguate which side of the join you would like to reference.
right
- Right side of the join operation.usingColumn
- Name of the column to join on. This column must exist on both sides.public DataFrame join(DataFrame right, scala.collection.Seq<java.lang.String> usingColumns)
DataFrame
using the given columns.
Different from other join functions, the join columns will only appear once in the output,
i.e. similar to SQL's JOIN USING
syntax.
// Joining df1 and df2 using the columns "user_id" and "user_name"
df1.join(df2, Seq("user_id", "user_name"))
Note that if you perform a self-join using this function without aliasing the input
DataFrame
s, you will NOT be able to reference any columns after the join, since
there is no way to disambiguate which side of the join you would like to reference.
right
- Right side of the join operation.usingColumns
- Names of the columns to join on. This columns must exist on both sides.public DataFrame join(DataFrame right, Column joinExprs)
DataFrame
, using the given join expression.
// The following two are equivalent:
df1.join(df2, $"df1Key" === $"df2Key")
df1.join(df2).where($"df1Key" === $"df2Key")
right
- (undocumented)joinExprs
- (undocumented)public DataFrame join(DataFrame right, Column joinExprs, java.lang.String joinType)
DataFrame
, using the given join expression. The following performs
a full outer join between df1
and df2
.
// Scala:
import org.apache.spark.sql.functions._
df1.join(df2, $"df1Key" === $"df2Key", "outer")
// Java:
import static org.apache.spark.sql.functions.*;
df1.join(df2, col("df1Key").equalTo(col("df2Key")), "outer");
right
- Right side of the join.joinExprs
- Join expression.joinType
- One of: inner
, outer
, left_outer
, right_outer
, leftsemi
.public DataFrame sort(java.lang.String sortCol, scala.collection.Seq<java.lang.String> sortCols)
DataFrame
sorted by the specified column, all in ascending order.
// The following 3 are equivalent
df.sort("sortcol")
df.sort($"sortcol")
df.sort($"sortcol".asc)
sortCol
- (undocumented)sortCols
- (undocumented)public DataFrame sort(scala.collection.Seq<Column> sortExprs)
DataFrame
sorted by the given expressions. For example:
df.sort($"col1", $"col2".desc)
sortExprs
- (undocumented)public DataFrame orderBy(java.lang.String sortCol, scala.collection.Seq<java.lang.String> sortCols)
DataFrame
sorted by the given expressions.
This is an alias of the sort
function.sortCol
- (undocumented)sortCols
- (undocumented)public DataFrame orderBy(scala.collection.Seq<Column> sortExprs)
DataFrame
sorted by the given expressions.
This is an alias of the sort
function.sortExprs
- (undocumented)public Column apply(java.lang.String colName)
Column
.
Note that the column name can also reference to a nested column like a.b
.colName
- (undocumented)public Column col(java.lang.String colName)
Column
.
Note that the column name can also reference to a nested column like a.b
.colName
- (undocumented)public DataFrame as(java.lang.String alias)
DataFrame
with an alias set.alias
- (undocumented)public DataFrame as(scala.Symbol alias)
DataFrame
with an alias set.alias
- (undocumented)public DataFrame select(scala.collection.Seq<Column> cols)
df.select($"colA", $"colB" + 1)
cols
- (undocumented)public DataFrame select(java.lang.String col, scala.collection.Seq<java.lang.String> cols)
select
that can only select
existing columns using column names (i.e. cannot construct expressions).
// The following two are equivalent:
df.select("colA", "colB")
df.select($"colA", $"colB")
col
- (undocumented)cols
- (undocumented)public DataFrame selectExpr(scala.collection.Seq<java.lang.String> exprs)
select
that accepts
SQL expressions.
df.selectExpr("colA", "colB as newName", "abs(colC)")
exprs
- (undocumented)public DataFrame filter(Column condition)
// The following are equivalent:
peopleDf.filter($"age" > 15)
peopleDf.where($"age" > 15)
condition
- (undocumented)public DataFrame filter(java.lang.String conditionExpr)
peopleDf.filter("age > 15")
conditionExpr
- (undocumented)public DataFrame where(Column condition)
filter
.
// The following are equivalent:
peopleDf.filter($"age" > 15)
peopleDf.where($"age" > 15)
condition
- (undocumented)public DataFrame where(java.lang.String conditionExpr)
peopleDf.where("age > 15")
conditionExpr
- (undocumented)public GroupedData groupBy(scala.collection.Seq<Column> cols)
DataFrame
using the specified columns, so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns grouped by department.
df.groupBy($"department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)public GroupedData rollup(scala.collection.Seq<Column> cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns rolluped by department and group.
df.rollup($"department", $"group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)public GroupedData cube(scala.collection.Seq<Column> cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns cubed by department and group.
df.cube($"department", $"group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)public GroupedData groupBy(java.lang.String col1, scala.collection.Seq<java.lang.String> cols)
DataFrame
using the specified columns, so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of groupBy that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns grouped by department.
df.groupBy("department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)public GroupedData rollup(java.lang.String col1, scala.collection.Seq<java.lang.String> cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of rollup that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns rolluped by department and group.
df.rollup("department", "group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)public GroupedData cube(java.lang.String col1, scala.collection.Seq<java.lang.String> cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of cube that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns cubed by department and group.
df.cube("department", "group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)public DataFrame agg(scala.Tuple2<java.lang.String,java.lang.String> aggExpr, scala.collection.Seq<scala.Tuple2<java.lang.String,java.lang.String>> aggExprs)
DataFrame
without groups.
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg("age" -> "max", "salary" -> "avg")
df.groupBy().agg("age" -> "max", "salary" -> "avg")
aggExpr
- (undocumented)aggExprs
- (undocumented)public DataFrame agg(scala.collection.immutable.Map<java.lang.String,java.lang.String> exprs)
DataFrame
without groups.
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(Map("age" -> "max", "salary" -> "avg"))
df.groupBy().agg(Map("age" -> "max", "salary" -> "avg"))
exprs
- (undocumented)public DataFrame agg(java.util.Map<java.lang.String,java.lang.String> exprs)
DataFrame
without groups.
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(Map("age" -> "max", "salary" -> "avg"))
df.groupBy().agg(Map("age" -> "max", "salary" -> "avg"))
exprs
- (undocumented)public DataFrame agg(Column expr, scala.collection.Seq<Column> exprs)
DataFrame
without groups.
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(max($"age"), avg($"salary"))
df.groupBy().agg(max($"age"), avg($"salary"))
expr
- (undocumented)exprs
- (undocumented)public DataFrame limit(int n)
DataFrame
by taking the first n
rows. The difference between this function
and head
is that head
returns an array while limit
returns a new DataFrame
.n
- (undocumented)public DataFrame unionAll(DataFrame other)
DataFrame
containing union of rows in this frame and another frame.
This is equivalent to UNION ALL
in SQL.other
- (undocumented)public DataFrame intersect(DataFrame other)
DataFrame
containing rows only in both this frame and another frame.
This is equivalent to INTERSECT
in SQL.other
- (undocumented)public DataFrame except(DataFrame other)
DataFrame
containing rows in this frame but not in another frame.
This is equivalent to EXCEPT
in SQL.other
- (undocumented)public DataFrame sample(boolean withReplacement, double fraction, long seed)
DataFrame
by sampling a fraction of rows.
withReplacement
- Sample with replacement or not.fraction
- Fraction of rows to generate.seed
- Seed for sampling.public DataFrame sample(boolean withReplacement, double fraction)
DataFrame
by sampling a fraction of rows, using a random seed.
withReplacement
- Sample with replacement or not.fraction
- Fraction of rows to generate.public DataFrame[] randomSplit(double[] weights, long seed)
DataFrame
with the provided weights.
weights
- weights for splits, will be normalized if they don't sum to 1.seed
- Seed for sampling.public DataFrame[] randomSplit(double[] weights)
DataFrame
with the provided weights.
weights
- weights for splits, will be normalized if they don't sum to 1.public <A extends scala.Product> DataFrame explode(scala.collection.Seq<Column> input, scala.Function1<Row,scala.collection.TraversableOnce<A>> f, scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
DataFrame
where each row has been expanded to zero or more
rows by the provided function. This is similar to a LATERAL VIEW
in HiveQL. The columns of
the input row are implicitly joined with each row that is output by the function.
The following example uses this function to count the number of books which contain a given word:
case class Book(title: String, words: String)
val df: RDD[Book]
case class Word(word: String)
val allWords = df.explode('words) {
case Row(words: String) => words.split(" ").map(Word(_))
}
val bookCountPerWord = allWords.groupBy("word").agg(countDistinct("title"))
input
- (undocumented)f
- (undocumented)evidence$1
- (undocumented)public <A,B> DataFrame explode(java.lang.String inputColumn, java.lang.String outputColumn, scala.Function1<A,scala.collection.TraversableOnce<B>> f, scala.reflect.api.TypeTags.TypeTag<B> evidence$2)
DataFrame
where a single column has been expanded to zero
or more rows by the provided function. This is similar to a LATERAL VIEW
in HiveQL. All
columns of the input row are implicitly joined with each value that is output by the function.
df.explode("words", "word"){words: String => words.split(" ")}
inputColumn
- (undocumented)outputColumn
- (undocumented)f
- (undocumented)evidence$2
- (undocumented)public DataFrame withColumn(java.lang.String colName, Column col)
DataFrame
by adding a column or replacing the existing column that has
the same name.colName
- (undocumented)col
- (undocumented)public DataFrame withColumnRenamed(java.lang.String existingName, java.lang.String newName)
DataFrame
with a column renamed.
This is a no-op if schema doesn't contain existingName.existingName
- (undocumented)newName
- (undocumented)public DataFrame drop(java.lang.String colName)
DataFrame
with a column dropped.
This is a no-op if schema doesn't contain column name.colName
- (undocumented)public DataFrame drop(Column col)
DataFrame
with a column dropped.
This version of drop accepts a Column rather than a name.
This is a no-op if the DataFrame doesn't have a column
with an equivalent expression.col
- (undocumented)public DataFrame dropDuplicates()
DataFrame
that contains only the unique rows from this DataFrame
.
This is an alias for distinct
.public DataFrame dropDuplicates(scala.collection.Seq<java.lang.String> colNames)
DataFrame
with duplicate rows removed, considering only
the subset of columns.
colNames
- (undocumented)public DataFrame dropDuplicates(java.lang.String[] colNames)
DataFrame
with duplicate rows removed, considering only
the subset of columns.
colNames
- (undocumented)public DataFrame describe(scala.collection.Seq<java.lang.String> cols)
This function is meant for exploratory data analysis, as we make no guarantee about the
backward compatibility of the schema of the resulting DataFrame
. If you want to
programmatically compute summary statistics, use the agg
function instead.
df.describe("age", "height").show()
// output:
// summary age height
// count 10.0 10.0
// mean 53.3 178.05
// stddev 11.6 15.7
// min 18.0 163.0
// max 92.0 192.0
cols
- (undocumented)public Row[] head(int n)
n
rows.n
- (undocumented)public Row head()
public Row first()
public <R> RDD<R> map(scala.Function1<Row,R> f, scala.reflect.ClassTag<R> evidence$3)
f
- (undocumented)evidence$3
- (undocumented)public <R> RDD<R> flatMap(scala.Function1<Row,scala.collection.TraversableOnce<R>> f, scala.reflect.ClassTag<R> evidence$4)
DataFrame
,
and then flattening the results.f
- (undocumented)evidence$4
- (undocumented)public <R> RDD<R> mapPartitions(scala.Function1<scala.collection.Iterator<Row>,scala.collection.Iterator<R>> f, scala.reflect.ClassTag<R> evidence$5)
f
- (undocumented)evidence$5
- (undocumented)public void foreach(scala.Function1<Row,scala.runtime.BoxedUnit> f)
f
to all rows.f
- (undocumented)public void foreachPartition(scala.Function1<scala.collection.Iterator<Row>,scala.runtime.BoxedUnit> f)
DataFrame
.f
- (undocumented)public Row[] take(int n)
n
rows in the DataFrame
.n
- (undocumented)public Row[] collect()
public java.util.List<Row> collectAsList()
public long count()
DataFrame
.public DataFrame repartition(int numPartitions)
DataFrame
that has exactly numPartitions
partitions.numPartitions
- (undocumented)public DataFrame coalesce(int numPartitions)
DataFrame
that has exactly numPartitions
partitions.
Similar to coalesce defined on an RDD
, this operation results in a narrow dependency, e.g.
if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of
the 100 new partitions will claim 10 of the current partitions.numPartitions
- (undocumented)public DataFrame distinct()
DataFrame
that contains only the unique rows from this DataFrame
.
This is an alias for dropDuplicates
.public DataFrame persist()
public DataFrame cache()
public DataFrame persist(StorageLevel newLevel)
newLevel
- (undocumented)public DataFrame unpersist(boolean blocking)
blocking
- (undocumented)public DataFrame unpersist()
public RDD<Row> rdd()
DataFrame
as an RDD
of Row
s. Note that the RDD is
memoized. Once called, it won't change even if you change any query planning related Spark SQL
configurations (e.g. spark.sql.shuffle.partitions
).public void registerTempTable(java.lang.String tableName)
DataFrame
as a temporary table using the given name. The lifetime of this
temporary table is tied to the SQLContext
that was used to create this DataFrame.
tableName
- (undocumented)public DataFrameWriter write()
DataFrame
out into external storage.
public RDD<java.lang.String> toJSON()
DataFrame
as a RDD of JSON strings.public java.lang.String[] inputFiles()
protected JavaRDD<byte[]> javaToPython()
public DataFrame toSchemaRDD()
toDF()
.public void createJDBCTable(java.lang.String url, java.lang.String table, boolean allowExisting)
write().jdbc()
.DataFrame
to a JDBC database at url
under the table name table
.
This will run a CREATE TABLE
and a bunch of INSERT INTO
statements.
If you pass true
for allowExisting
, it will drop any table with the
given name; if you pass false
, it will throw if the table already
exists.url
- (undocumented)table
- (undocumented)allowExisting
- (undocumented)public void insertIntoJDBC(java.lang.String url, java.lang.String table, boolean overwrite)
write().jdbc()
.DataFrame
to a JDBC database at url
under the table name table
.
Assumes the table already exists and has a compatible schema. If you
pass true
for overwrite
, it will TRUNCATE
the table before
performing the INSERT
s.
The table must already exist on the database. It must have a schema
that is compatible with the schema of this RDD; inserting the rows of
the RDD in order via the simple statement
INSERT INTO table VALUES (?, ?, ..., ?)
should not fail.
url
- (undocumented)table
- (undocumented)overwrite
- (undocumented)public void saveAsParquetFile(java.lang.String path)
write().parquet()
.DataFrame
as a parquet file, preserving the schema.
Files that are written out using this method can be read back in as a DataFrame
using the parquetFile
function in SQLContext
.path
- (undocumented)public void saveAsTable(java.lang.String tableName)
write().saveAsTable(tableName)
.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
When the DataFrame is created from a non-partitioned HadoopFsRelation
with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
tableName
- (undocumented)public void saveAsTable(java.lang.String tableName, SaveMode mode)
write().mode(mode).saveAsTable(tableName)
.SaveMode.ErrorIfExists
as the save mode.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
When the DataFrame is created from a non-partitioned HadoopFsRelation
with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
tableName
- (undocumented)mode
- (undocumented)public void saveAsTable(java.lang.String tableName, java.lang.String source)
write().format(source).saveAsTable(tableName)
.SaveMode.ErrorIfExists
as the save mode.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
When the DataFrame is created from a non-partitioned HadoopFsRelation
with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
tableName
- (undocumented)source
- (undocumented)public void saveAsTable(java.lang.String tableName, java.lang.String source, SaveMode mode)
write().mode(mode).saveAsTable(tableName)
.SaveMode
specified by mode, and a set of options.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
When the DataFrame is created from a non-partitioned HadoopFsRelation
with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
tableName
- (undocumented)source
- (undocumented)mode
- (undocumented)public void saveAsTable(java.lang.String tableName, java.lang.String source, SaveMode mode, java.util.Map<java.lang.String,java.lang.String> options)
write().format(source).mode(mode).options(options).saveAsTable(tableName)
.SaveMode
specified by mode, and a set of options.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
When the DataFrame is created from a non-partitioned HadoopFsRelation
with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
tableName
- (undocumented)source
- (undocumented)mode
- (undocumented)options
- (undocumented)public void saveAsTable(java.lang.String tableName, java.lang.String source, SaveMode mode, scala.collection.immutable.Map<java.lang.String,java.lang.String> options)
write().format(source).mode(mode).options(options).saveAsTable(tableName)
.SaveMode
specified by mode, and a set of options.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
When the DataFrame is created from a non-partitioned HadoopFsRelation
with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
tableName
- (undocumented)source
- (undocumented)mode
- (undocumented)options
- (undocumented)public void save(java.lang.String path)
write().save(path)
.SaveMode.ErrorIfExists
as the save mode.path
- (undocumented)public void save(java.lang.String path, SaveMode mode)
write().mode(mode).save(path)
.SaveMode
specified by mode,
using the default data source configured by spark.sql.sources.default.path
- (undocumented)mode
- (undocumented)public void save(java.lang.String path, java.lang.String source)
write().format(source).save(path)
.SaveMode.ErrorIfExists
as the save mode.path
- (undocumented)source
- (undocumented)public void save(java.lang.String path, java.lang.String source, SaveMode mode)
write().format(source).mode(mode).save(path)
.SaveMode
specified by mode.path
- (undocumented)source
- (undocumented)mode
- (undocumented)public void save(java.lang.String source, SaveMode mode, java.util.Map<java.lang.String,java.lang.String> options)
write().format(source).mode(mode).options(options).save(path)
.SaveMode
specified by mode, and a set of options.source
- (undocumented)mode
- (undocumented)options
- (undocumented)public void save(java.lang.String source, SaveMode mode, scala.collection.immutable.Map<java.lang.String,java.lang.String> options)
write().format(source).mode(mode).options(options).save(path)
.SaveMode
specified by mode, and a set of optionssource
- (undocumented)mode
- (undocumented)options
- (undocumented)public void insertInto(java.lang.String tableName, boolean overwrite)
write().mode(SaveMode.Append|SaveMode.Overwrite).saveAsTable(tableName)
.tableName
- (undocumented)overwrite
- (undocumented)public void insertInto(java.lang.String tableName)
write().mode(SaveMode.Append).saveAsTable(tableName)
.tableName
- (undocumented)