{% extends "base.html" %} {% block title %}400 CSRF validation failed{% endblock %} {% block page_content %}
{{ e.description }}
Please reload the page and try again.


I’m debating whether to have the authors name in the blog post as right now I’m the only one writing for the site.
Open to guest posts fyi


Would You Like Hash With That?Bane SrdjevicX/V/MMXIV |
Good logic coding revolves almost entirely around organizing and manipulating data, and two of the most useful tools you'll be using in your pursuit of creating the perfect program are the hash and the array. These are two very common constructs that can be used to organize, store, and sort data, not to mention allow for some pretty neat manipulation. But that's just talk, and you need something you can really sink your teeth into. Fine. Let's get down to hash tacks then.
Arrays
I like to think of arrays as a shelf.  Books, pots, pans, integers, bookends, figurines, other shelves--anything! Breaking metaphor, an array is a space that can be segmented an infinite amount of times and contain absolutely anything you can think of in it, even other arrays! Strings, tuples, integers, floats, methods, classes, hashes--anything!
This is how you make one:
array = [ ]
------ (this is just an empty array, but note the brackets! Parenthesis are tuples and curly braces are hashes)
array=[(1,2),1,'blue',[ (1,2),1,'blue']]
------- (this one is full of all kinds of neat things!)
Then this is how you call on a specific entry in the array (remember that the first item has a place value of 0):
array[1]
------ (this returns the second item in the array, which would be 1 if we continue using the stored information from above)
array[3]
------ (this will return [ (1,2),1,'blue'])
But hold up! Calling array[3] brings up a whole new array, so what if I want to just call a single element from that array and not the entire array?! Luckily, this is one of the few times in coding syntax where common sense actually had a say in how things are done.
To call a single entry in an array that's stored in an array simply type:
array[3][0]
-------(this will return the first entry in the entry stored at position [3] in array: (1,2))
array[0][1]
-------(this will return the second entry in the entry stored at position [0] in array: 2)
But wait, the entry located in array[0] is a tuple and not an array! If you noticed this then you're an attentive cookie. You can use this trick on tuples even! But that's all I'm telling you, so go punch some code into IRB if you want to see what else it works on.
Last trick:
If you want to call an entry in an array from the OTHER direction, meaning the last entry first, simply go negative:
array[-1]
-------- (this will return the last entry in the array: [ (1,2),1,'blue']. It works the same as array[3])
array[-3]
-------- (this will return the third entry from the back: 1)
Hashes
Hashes, on the other hand, are a little more specific than arrays. Essentially, without getting into how they interact with methods, they are the same things as arrays except for how you store data into them and how you call data from them.
To create a hash:
hash{}
--------- (creates an empty hash)
But now you have to store information into the hash. This is what that would look like:
hash{ 'a'=>5,
5=>13,
:steel => 'iron and wine',
'hints'=>{:steel=>13, 'b'=> {'a'=>5, 'b'=> 6, 'c'=>'ab'}, 'c'=>'ab'},
}
Looks a little funky, right? But it's pretty simple. Each entry in a hash has two components: the key and the value. The value is the data being stored, and the key is the "address" it's being stored at. Observe:
hash['a']
------- (This will return the value 5, because that is the value stored at 'a')
Important to note here is that you are not defining 'a' as 5, you are simply saying that there is a person named 5 who is living at house 'a'. That 5 can be replaced with anything you want, even other hashes! Values can be anything, but keys cannot. Keys have to be either strings, integers, floats, or symbols. They cannot be tuples or hashes or arrays.
What does this mean? It means that hashes are a much more specific way to store and retrieve data than arrays are. Arrays make it easier to manipulate data, but it can get sometimes lost in the mish mash if you're not careful. That's why it's better to manipulate data in an array and then store it in a hash afterwards.
With that suggestion, I'll leave you with two thoughts: how would you go through an array to turn it into a hash using nothing but code, and how would you get a specific value from a hash that's stored as a value in another hash? Bonus question: how would you call a value in a hash that's stored in an array?
Enjoy. Oh, and cipher something, too. Hashes are fun for that nonsense.
Water Tribe, Out


nips nips2003 nips2003-129 nips2003-129-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Hsin Chen, Patrice Fleury, Alan F. Murray
Abstract: This paper presents VLSI circuits with continuous-valued probabilistic behaviour realized by injecting noise into each computing unit(neuron). Interconnecting the noisy neurons forms a Continuous Restricted Boltzmann Machine (CRBM), which has shown promising performance in modelling and classifying noisy biomedical data. The Minimising-Contrastive-Divergence learning algorithm for CRBM is also implemented in mixed-mode VLSI, to adapt the noisy neurons’ parameters on-chip. 1
[1] H. Chen and A. Murray, “A continuous restricted boltzmann machine with an implementable training algorithm,” IEE Proc. of Vision, Image and Signal Processing, vol. 150, no. 3, pp. 153–158, 2003.
[2] T. Tang, H. Chen, and A. Murray, “Adaptive Stochastic Classiﬁer for Noisy pH-ISFET Measurements,” in Proceedings of Thirteenth International Conference on Artiﬁcial Neural Networks (ICANN2003), (Istanbul, Turkey), pp. 638–645, Jun. 2003.
[3] G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” Neural Computation, vol. 14, no. 8, pp. 1771–1800, 2002.
[4] B. J. Frey, “Continuous sigmoidal belief networks trained using slice sampling,” Advances in Neural Information Processing Systems, vol. 9, pp. 452–458, 1997.
[5] A. F. Murray, “Novelty detection using products of simple experts-a potential architecture for embedded systems,” Neural Networks, vol. 14, no. 9, pp. 1257–1264, 2001.
[6] H. Chible, “Analog circuit for synapse neural networks vlsi implementation,” The 7th IEEE Int. Conf. on Electronics, Circuits and Systems (ICECS 2000), vol. 2, pp. 1004–1007, 2000.
[7] M. Banu and Y. Tsividis, “Floating voltage-controlled resistors in cmos technology,” Electronics Letters, vol. 18, pp. 678–679, 1982.
[8] E. Vittoz, “Mos transistors operated in the lateral bipolar mode and their application in cmos technology,” IEEE Journal of Solid-State Circuits, vol. sc-18, no. 3, pp. 273–279, 1983.
[9] J. Alspector, J. W. Gannett, S. Haber, M. B. Parker, and R. Chu, “A vlsi-eﬃcient technique for generating multiple uncorrelated noise sources and its application to stochastic neural networks,” IEEE Trans. Circuits and Systems, vol. 38, no. 1, pp. 109–123, 1991.
[10] P. Fleury and A. Murray, “Mixed-signal vlsi implementation of the product of experts’ minimizing contrastive divergence learning scheme,” in IEEE Proc. of the Int. Sym. on Circuits and Systems (ISCAS 2003), vol. 5, (Bangkok, Thailand), pp. 653–656, May 2003.
[11] G. Wegmann and E. Vittoz, “Basic principles of accurate dynamic current mirrors,” IEE Proc. on Circuits, Devices and Systems, vol. 137, pp. 95–100, April 1990.
[12] G. Cauwenberghs, “An analog vlsi recurrent neural network,” IEEE Tran. on Neural Networks, vol. 7, pp. 346–360, Mar. 1996.


Ivan Belyavtsev
22.01.2022
After vertex shader stage we have an NDC coordinates of each vertex of the triangle.
Next steps:
Could be done in homogeneous coordinates or cartesian
\[M_{ss} = \left[\begin{array}{cccc} width & 0 & 0 & 0 \\ 0 & - height & 0 & 0 \\ 0 & 0 & 1 & 0 \\ x & height + y & 0 & 1 \end{array}\right]\]
where \(x\) and \(y\) are offsets from the screen space \((0, 0)\) [1]
Let’s take two points \((X, Y)\) and \((X+dx, Y+dy)\) on an edge
Then, define an edge function:
\[E(x, y) = (x - X)dy - (y - Y)dx\]
[2]
If \(E(x, y)>0\) then \((x, y)\) on the “right” side
If \(E(x, y)<0\) then \((x, y)\) on the “left” side
If \(E(x, y)=0\) then \((x, y)\) on the edge
[2]
[2]
[3]
Lets compare \[E(x, y) = (x - X)dy - (y - Y)dx\]
and \[(u_1, u_2, 0) \times (v_1, v_2, 0) = (0, 0, u_1v_1-u_2v_1)\]
The edge function value is a twice area of the triangle
Let \(u = \frac{S_{CAP}}{S_{ABC}}\), \(v = \frac{S_{BCP}}{S_{ABC}}\), \(w = \frac{S_{ABP}}{S_{ABC}}\)
Therefore, \(u+v+w=1\), and \(P=uA+vB+wC\). The barycentric could be used for attribute interpolation [4]
Input: interpolated vertex attributes
Output: result pixel color
The pixel shader should be provided by graphics programmer [5]
cg::renderer::rasterizer::edge_function
methodpixel_shader
lambda for the instance of cg::renderer::rasterizer
Rasterization
and Pixel shader
stages to draw
method of cg::renderer::rasterizer


--- date: 2010-02-10T00:00:00Z published: true title: Why Web Developers Should Switch to Google Chrome url: /2010/02/10/why-web-developers-should-switch-to-google-chrome/ ---
Some of my guys already have.


Also note: Blue items are directly in the navigation, while Green available from drop down, yellow in navigation on the page.
The links don't currently work, as they're from the perspective of the 1950 folder. If the structure of the webspace stays as it is now replace the "./" with "/1950/".
The above sitemap was created from an HTML unordered list with SlickMap CSS.
The tooltips are just CSS and inspired by a w3schools tutorial.


Ascent Sobriety Tool helps members track their daily mood & thoughts as well as various lengths of sobriety. In a time where more and more people are living sober lifestyles, this app caters to those who are seeking to track continuous time as well as discontinuous time. This application was built with Express, Sequelize, MySQL, CORS, Nodemailer, Passport, Express-Handlebars, Moment, ApexCharts, and Bootstrap.
An 8-Bit styled Burger Restaurant where users can order, eat, and delete burgers of their choosing. Built with Node.js, Express, Express-Handlebars, and MySQL.
For the Password Generator assignment we were asked to create a an application that would provide a random password to a user based on length, letter, number, and special character options that the user needed. The application uses HTML, CSS, Javascript.
In this project we were asked to work as a group to come up with an original application concept based on API's of our own choosing. After a bit of pivoting due to usibility issues with a few API's that we initially chose, we landed on the idea of an application that would allow you to choose a job based on the restaurants located nearby. The application uses HTML, CSS, jQuery, Javascript, Mapquest API, Muse API, Zomato API, and the Materialize CSS framework.


Hoffenheim waste chance to go fourth after Mainz hammering
by BeSoccerHoffenheim missed out on an opportunity to move into the Bundesliga's top four on Sunday, suffering a heavy 5-1 home defeat by a Mainz side playing under newly appointed coach Achim Beierlorzer.
The hosts failed to win for the first time in five games as the visitors moved out of the relegation zone to 13th spot with only their third victory of the campaign.
Beierlorzer, who was sacked by struggling Cologne nine days before taking over at the Opel Arena on November 18, started his reign successfully despite Ridle Baku's first-half red card as Cameroon midfielder Pierre Kunde Malong scored twice to open his account for the season.
Levin Oeztunali claimed the first goal of proceedings on 33 minutes before his midfield partner Baku was dismissed seconds before the break for a studs-up challenge on the home side's Sebastian Rudy.
Mainz continued to attack in the second half, peppering the Hoffenheim box until an own goal from Pavel Kaderabek in the 52nd minute doubled their lead.
Kunde tapped in for his first of the season 10 minutes later before Andrej Kramaric claimed a consolation effort for the home side in the 83rd minute.
Jean-Paul Boetius snuffed out any chance of a comeback in the 90th minute, however, scoring on the break to make it 4-1 and Kunde added his second two minutes later to complete the rout.
Mainz coach Beierlorzer praised sacked former boss Sandro Schwarz, dedicating the victory to him.
"It was an amazing achievement," Â Beierlorzer said. "Sandro deserves a very, very large share of it. It's guaranteed that the team fought for him today and brought home the victory."
- Hertha woes continue -
Earlier on Sunday, goalkeeper Rune Jarstein was sent off as Hertha Berlin continued to stutter, losing 4-0 at Augsburg.
Featured links
- The latest football transfers news and rumours
- Mourinho replaces Pochettino at Tottenham
- Kante explains why he turned down PSG
- What Cristiano said about Messi to his Man Utd teammates
Hertha lost a fifth straight game and slid below Augsburg after Philipp Max, Sergio Cordova, Andre Hahn and Florian Niederlechner scored for the hosts. Hertha are just a point above the relegation zone.
Left-back Max opened the scoring for the Bavarians in the 18th minute with a free-kick but the turning point came 10 minutes later.
Jarstein earned a red card with a studs-up challenge on Niederlechner. Cordova claimed the home side's second in the process.
Substitute Hahn added a third in the 52nd minute with a superb curling strike from the edge of the box. Niederlechner bagged his side's fourth in the 79th minute.
"The disappointment is understandably big, we have to digest that first," said Hertha captain Niklas Stark.
"There were so many things we did which you just can't do while playing away in the Bundesliga.
"We simply have to do better."
Earlier on Sunday, Borussia Dortmund's squad were booed and jeered when they appeared at the side's annual general meeting, with fans screaming "shame on you".
Coach Lucien Favre is under pressure as Dortmund, expected to challenge for the title after an ambitious close-season recruitment drive, languish in sixth after a draw to lowly Paderborn on Friday.
On Saturday, managerless Bayern Munich closed the gap at the top of the table to just a point with a 4-0 romp at Fortuna Duesseldorf as leaders Borussia Moenchengladbach suffered a shock 2-0 loss at Union Berlin.


GVHS Reach
Login
Register
Boards
☰
Login
Register
Boards
Login
Email:
Password:
Submit
Don't have an account?
Forgot your password?


Gateway
BOT
30-Oct-19 01:45 AM
[telegram] <omboti> And nothing happens when oI clicked trade
[telegram] <omboti> I looked at my history
[telegram] <Regnar> Did you notice that your 2 switched to a smaller amount first?
[telegram] <Regnar> History is for completed trades, "orders" is for current trades either open or being executed (re @omboti: I looked at my history)
[telegram] <omboti> Yeah I pick the very top
[telegram] <omboti> Btw 2 swaps was successful but I don't remember the process i select
[telegram] <Regnar> Yes and that should change the values based on the price and the available trade volume at that price. Then you click trade. It will say confirm trade details and you click confirm (re @omboti: Yeah I pick the very top)
[telegram] <omboti> But for sure from the order book
[telegram] <omboti> Yes that what i did (re @Regnar: Yes and that should change the values based on the price and the available trade volume at that price. Then you click trade. It will say confirm trade details and you click confirm)
[telegram] <omboti> So the very top is the best price to swap at the moment?
[telegram] <Regnar> Yes, but may not have the most available volume (re @omboti: So the very top is the best price to swap at the moment?)
[telegram] <Regnar> What do you mean? When you click trade it should say 0/3 and begin the process (re @omboti: And nothing happens when oI clicked trade)
[telegram] <omboti> Yeah i noticed that
[telegram] <omboti> Than will increase to match status? (re @Regnar: What do you mean? When you click trade it should say 0/3 and begin the process)
[telegram] <omboti> Ok so i clue is the order at the top is the best offer
[telegram] <Regnar> When it matches with the other person it will say 1/3. It's peer to peer so when you click someone else's order to fill it, it still has you communicating with them that you are matching their order (re @omboti: Than will increase to match status?)
[telegram] <Regnar> Yes, and it will auto fill the volume if the best price is below your desired swap amount (re @omboti: Ok so i clue is the order at the top is the best offer)
[telegram] <omboti> Yay i have 3 matches transactions lol
[telegram] <Regnar> Nice!! (re @omboti: Yay i have 3 matches transactions lol)
[telegram] <omboti> So there are no partial order match! Either we take the order book or just put our order book
[telegram] <Regnar> Partial fills are supported (re @omboti: So there are no partial order match! Either we take the order book or just put our order book)
[telegram] <omboti> I onlybtrade morty and Rick test now
[telegram] <omboti> Why is transaction in and out in both assets different
[telegram] <omboti> 3 transactions in morty and 6 in rick
[telegram] <omboti> 6 out transactions in rick i mean
Documentation for developers building on the Komodo Platform
[telegram] <Regnar> There are a series of transactions being made, these are part of the Atomic Swap


---
layout: article
title: Podcast.. Finally
date: 2019-12-17 9:39 PM
coverPhoto: images/lake-irene-1679708_640.jpg
---
Our podcast is being submitted to several services via Anchor. This will take at least a week due to the holidays. In the meantime you can listen to our intro podcast.


Configure Docker Assemble
Estimated reading time: 4 minutesAlthough you don’t need to configure anything to build a project using Docker Assemble, you may wish to override the defaults, and in some cases, add fields that weren’t automatically detected from the project file. To support this, Docker Assemble allows you to add a file docker-assemble.yaml
to the root of your project. The settings you provide in the docker-assemble.yaml
file overrides any auto-detection and can themselves be overridden by command-line arguments
The docker-assemble.yaml
file is in YAML syntax and has the following informal schema:
-
version
: (string) mandatory, must contain0.2.0
-
image
: (map) contains options related to the output image.-
platforms
: (list of strings) lists the possible platforms which can be built (for example,linux/amd64
,windows/amd64
). The default is determined automatically from the project type and content. Note that by default Docker Assemble will build only forlinux/amd64
unless--push
is used. See Building Multiplatform images. -
ports
: (list of strings) contains ports to expose from a container running the image. e.g80/tcp
or8080
. Default is to automatically determine the set of ports to expose where possible. To disable this and export no ports specify a list containing precisely one element ofnone
. -
labels
: (map) contains labels to write into the image askey
-value
(string) pairs. -
repository-namespace
: (string) the registry and path component of the desired output image. e.g.docker.io/library
ordocker.io/user
. -
repository-name
: (string) the name of the specific image withinrepository-namespace
. Overrides any name derived from the build system specific configuration. -
tag
: (string) the default tag to use. Overrides and version/tag derived from the build system specific configuration. -
healthcheck
: (map) describes how to check a container running the image is healthy.-
kind
: (string) sets the type of Healthcheck to perform. Valid values arenone
,simple-tcpport-open
andspringboot
. See Health checks. -
interval
: (duration) the time to wait between checks. -
timeout
: (duration) the time to wait before considering the check to have hung. -
start-period
: (duration) period for the container to initialize before the retries starts to count down -
retries
: (integer) number of consecutive failures needed to consider a container as unhealthy.
-
-
-
springboot
: (map) if this is a Spring Boot project then contains related configuration options.-
enabled
: (boolean) true if this is a springboot project. -
java-version
: (string) configures the Java version to use. Valid options are8
and10
. -
build-image
: (string) sets a custom base build image -
runtime-images
(map) sets a custom base runtime image by platform. For valid keys, refer to the Spring Boot section in Custom base images.
-
-
aspnetcore
: (map) if this is an ASP.NET Core project then contains related configuration options.-
enabled
: (boolean) true if this is an ASP.NET Core project. -
version
: (string) configures the ASP.NET Core version to use. Valid options are1.0
,1.1
,2.0
and2.1
. -
build-image
: (string) sets a custom base build image -
runtime-images
(map) sets a custom base runtime image by platform. For valid keys, refer to the ASP.NET Core section in Custom base images.
-
Notes:
The only mandatory field in
docker-assemble.yaml
isversion
. All other parameters are optional.At most one of
dotnet
orspringboot
can be present in the yaml file.Fields of type duration are integers with nanosecond granularity. However the following units of time are supported:
ns
,us
(orµs
),ms
,s
,m
,h
. For example,25s
.
Each setting in the configuration file has a command line equivalent which can be used with the -o/--option
argument, which takes a KEY=VALUE
string where KEY
is constructed by joining each element of the YAML hierarchy with a period (.).
For example, the image → repository-namespace
key in the YAML becomes -o image.repository-namespace=NAME
on the command line and springboot → enabled
becomes -o springboot.enabled=BOOLEAN
.
The following convenience aliases take precedence over the -o/--option
equivalents:
-
--namespace
is an alias forimage.repository-namespace
; -
--name
corresponds toimage.repository-name
; -
--tag
corresponds toimage.tag
; -
--label
corresponds toimage.labels
(can be used multiple times); -
--port
corresponds toimage.ports
(can be used multiple times)


This page contains a list of AsyncAPI tooling. Would you like to add your tool to this list? Let us know!
Please, before you decide to create a new tool, consider contributing to the existing ones. Thanks!
The following is a list of tools that generate code from an AsyncAPI document, and not the other way around.
| Link | Description | Language/Framework |
|---|---|---|
| AsyncAPI Generator | Use your AsyncAPI definition to generate literally anything. Markdown documentation, Node.js code, Java code, HTML documentation, anything! Please, check out the templates directory to get a list of the supported languages/formats. | Node.js/Hermes, Java/Spring, Markdown, HTML, and more. |
The language you’re looking for is not here? You have created a new code generator and want to list it here? Let us know!
The following is a list of tools that generate human-readable documentation from an AsyncAPI document.
| Link | Description | Language/Kind |
|---|---|---|
| AsyncAPI Generator | Use your AsyncAPI definition to generate literally anything. Markdown documentation, Node.js code, HTML documentation, anything! | CLI / Javascript |
| AsyncAPI React | React component for rendering documentation from your specification in real-time in the browser. | Javascript |
| Bump | OpenApi 2 & 3 / AsyncAPI 2 documentation generator, with automatic changelog and visual diff. | SaaS |
| Widdershins | OpenApi 3.0 / Swagger 2.0 / AsyncAPI 1.0 definition to Slate / Shins compatible markdown. Thanks to @PermittedSoc. | CLI / Javascript |
The following is a list of tools that validate AsyncAPI documents.
| Link | Description |
|---|---|
| AsyncAPI Parser | It parses and validates AsyncAPI documents. |
| Check-API | It allows you to validate a local file or remote URL with a single command-line or programmatic invocation. It returns an exitCode of 0 on success and 1 on failure, making it suitable for use in Continuous Integration environments. Thanks to @PermittedSoc. |
| asyncapi-validator | It allows you to validate schema of your messages against your AsyncAPI schema definition. You can use it with Kafka, RabbitMQ or any other messaging/queue. Thanks to @waleedashraf. |
The following is a list of tools that generate AsyncAPI documents from your code.
| Link | Description | Language/Framework |
|---|---|---|
| Go AsyncAPI | It uses reflection to translate Go structures in JSON Schema definitions and arrange them in AsyncAPI schema. Thanks to @vearutop. | Go |
| Saunter | Like Swashbuckle for AsyncAPI. Generates (and hosts) an AsyncAPI schema document from your code. Thanks to @tehmantra. | C#/dotnet |


Select your ethereum account :
No Ethereum account found
Please wait.... Registration will begin soon.
You have ether.
Would you like to register for this vote?
Enter your inscription code:
Waiting for Ethereum to accept your demand.
The Election Authority should have finished registration before
All votes should have been cast before
Do you want to check your vote?
Please enter a password:


[click on any image to enlarge]
[click on any image to enlarge]
Connections can then be edited by clicking on the icon, selecting Network Manager, then Edit to open the settings manager. Also, notice the icon changes its appearance once a connection is established. A common feature that is enabled automatically is to have the connection in question automatically connect as soon as it is detected.
Connection problems can be notoriously difficult to diagnose, especially on a system that is not able to make a connection at all. If these steps fail, the best advice we can give is to get on a system that does make a connection (or use a wired connection if wireless is failing) and join us on IRC or the Bodhi Forums. See the Resources Page.


Rev. Moses How
04 Jan 2016 tagged in diaries, moses how, new bedford free public library, seamen's bethel, thesis thoughtsI’ve been reading through the diaries of Reverend Moses How, who served as chaplain of the Seamen’s Bethel from 1844 to 1859. Reading How’s diaries (there are several in the collection) isn’t as interesting as one may think – they read straightforward, more like a planner than the source of secrets. How visited jails, Sunday schools, Port Society meetings, and traveled around the south coast of Massachusetts to talk to numerous people.
As I first skimmed through the diaries, I saw tallies at the top of each page. Sometimes there were two or three lines – sometimes there were a few dozen. I didn’t think about those tallies until I started reading.
He was keeping track of the number of funerals he had attended.
I found this devastating at first, how depressing it must have been for him. But then again, I couldn’t think much of it – he was a chaplain and reverend, it was regular responsibility in his line of work. How kept records of lots of things in each year of service – how many marriages he had performed, how many births in his congregation, how many sermons he had given and where they took place. But funerals were the only ones that are left with a permanent mark in his diary – a physical remnant of his counting.
I’ve been reading How’s diaries to get a sense of his interactions with the seamen, though he doesn’t go into much detail. He was there for 14 years, but the strongest interactions he records are ones with dead men. It seems he struggled with his duties at the Bethel, trying to work for both the Port Society and to also carry out assistance to the seamen:
I am more and more convinced that any man who will be a true friend to the sailor and look out honestly for his temporal welfare will be unpopular with most of this — community. If I had known as much about the duties and difficulties attending as a pastor at the Seamen’s Bethel, as I do now, I am confident I should have never consented to go there, and now I see the wants of seamen, and their need of a friend they can confide in. I cannot bear the thought of leaving them, and yet, I sometimes think I cannot stay there any longer. I hope my heavenly Father will teach and guide me in the way I should go. (December 8, 1855)
Apparently, How was guided to stay, because he didn’t announce his resignation until late 1858:
I shall retire from my field of labor at the time you may designate, but not without feelings of deep anxiety for the prosperity of the Bethel, the Ladies’ Branch, and the Mariner’s Home, and earnest prayer that the abundance of the seas may be yet converted unto God. (October 21, 1858)
But I’m moved by that first passage I came across – wondering what had happened for How to commit this to paper. Was this an outburst? Or was it a decades’ worth of experiences finally coming out? What are the “wants of seamen” that he was unable to provide?


|
Here's what good ol' Wikipedia has to say about the Echo:
"Amazon Echo is a voice command device from Amazon.com with functions including question answering, playing music and controlling smart devices. The device consists of a 9.25-inch (23.5 cm) tall cylinder speaker with a seven-piece microphone array, speakers that include a woofer/tweeter and a remote control"
What does this mean for everyday users?
- I can tell it to play my music.
I can tell it what to add to my shopping list.
- I ask it questions like "whats the weather?" and "whats traffic like for my commute?"
If I'm in a humorous mood, I say "Alexa tell me a joke"
|
A typical day with me and Alexa:
Me: "Alexa play playlist, Sredna's classical music"
Alexa: "Sredna's classical music from music library" -she then proceeds to play that playlist.
-
Me: "Alexa add Milk to my shopping list"
Alexa:"I've put milk on your shopping list"-milk has been added to my shopping list that I can see on my amazon echo phone app.
-
Me: "Alexa, whats the weather like?"
Alexa: "Right now in Renton its 91 degrees with clear skies and sun, tonight's forecast has clear skies and a low of 64."
-
Me: "Alexa tell me a joke"
Alexa: "How many therapists does it take to change a light bulb? One, but it has to really want to change."
-
Me: "Alex do you love me?"
Alexa: "I can't do that, but I can find Lionel Richie songs if you like."
-
Me: "Alexa, don't let the bed bugs bight"
Alexa: "Thankfully I don't sleep in a bed."
|


obs = subsample( gray_scale(obs), 84, 110 )
obs = obs[26:110, :]
obs = binary_image(obs)
obs = np.reshape(obs, (84, 84, 1))
obs = torch.from_numpy(obs).view(1, -1).float()
return obs
class CartPole:
obs (torch.Tensor): Pre-processed observation.
"""
obs = subsample( gray_scale(obs), 84, 110 )
print (obs.shape)
sys.exit(0)
obs = obs[26:100, :]
obs = binary_image(obs)
//obs = np.reshape(obs, (74, 84, 1))


THE BIG KAHUNA
Reviewed by Harvey Karten Lions Gate Films Director: John Swanbeck Writer: Roger Rueff, from his play "Hospitality Suite" Cast: Kevin Spacey, Danny De Vito, Peter Facinelli
Some actors were born to play scam artists. Alec Baldwin as chief huckster Blake in "Glengarry Glen Ross." Ben Affleck in a similar role as Jim Young in "Boiler Room," Steve Martin as Jimmy Dell in "The Spanish Prisoner." Kevin Spacey as an industrial lubricants salesman, Larry, in "The Big Kahuna." There's one difference, though. In "Kahuna," written by Roger Rueff from his play "Hospitality Suite" and directed by first-time helmer John Swanbeck, Spacey comes across as huckster in the initial part of the story, the comic scenes. His con-man guise gradually fades as Swanbeck moves him into the second part: a deeper, more intense, more human portion of the narrative. By watching this remarkable staged play on the big screen, we leave the theater with more insight about the characters and, I dare say, more intuition about ourselves. Isn't this what good cinema is all about?
"The Big Kahuna" exhibits the typical structure of a theatrical drama--a group of people essentially locked up together in a room to work out emotional difficulties (a good example being Eugene O'Neill's "Long Day's Journey Into Night" in which a single family reveal with brutal honesty the causes of their tragic lives). Playwright Roger Rueff is no O'Neill, but in showing us his trio's long night's journey into day he cuts through their defenses so incisively that he reveals the goodness beneath one seemingly shallow man's bravado and the inanity behind a seemingly principled character's saintliness. An older man is the catalyst for the war between two opposing viewpoints of life, a referee in a 90-minute bout pitting people of contrasting philosophies in the ring, each taking something from the other that makes him a better person.
"The Big Kahuna" is set almost wholly inside a relatively small hospitality suite on the sixteenth floor of a hotel in Witchita, Kansas, to be the hub of a business party taking place in a few hours. Three marketing reps from a company selling industrial lubricants--middle-aged, fast-talking Larry (Kevin Spacey), the older Phil (Danny De Vito) and a nervous kid on his first real job, Bob (Pete Facinelli)--have been sent to the Midwest location to sell the company's product. Their major target is the president of a client corporation whom the trio are eager to snare into a big sale. The big problem is that the big Kahuna seems to be a no-show, but just as Larry and Phil despair, they find out from young Bob that this kid was talking to the man all night. He was not discussing lubricants, though; a practicing Baptist, Bob spent the hours addressing the nature of God with the corporate president.
"The Big Kahuna" turns our expectations inside out. As we observe the cynical Larry deliver quips about beautiful women and complaints about the cheap look of the room and the food; as we watch the newly divorced and troubled Phil reading a copy of Penthouse magazine before the party is to begin; we imagine that the hero of the evening will be young Bob, who is pure at heart, a young man who takes seriously the New Testament concept that if you lust after a woman you are as guilty as the man who actually commits adultery. Having gazed down his priggish nose at the "fallen" men with whom he is teamed, wearing a shining, metaphoric halo about his head because he discussed higher things with the client than plain old lubricants, Bob is jolted, absolutely stunned, to find that of the threesome, he is the one with the least character. How he comes to this cognizance should not be revealed in a review. The point of the movie is to have us indulge the threesome working on one another rather than on their clients and to discover how each of the three--Phil, who is tired and looks to another way of life now that he faces his mortality; Larry, whose repartee hits all the right comic notes; and Bob, who is so full of his upright self that he cannot distinguish character from cottage cheese--leave the hotel in the morning wiser, more attuned, more human. For a script this trenchant, director Swanbeck and the writer, Roger Rueff, could not have assembled a better cast than the extraordinary Kevin Spacey, the remarkable Danny De Vito, and, big surprise, newcomer Pete Facinelli who as an actor must have felt like the salesman he is playing. In a room with the two greats, he must have been most deeply affected by the key line in the story. The cynical Larry asks the kid, "You think you're ready?" to which Bob answers, "There's only one way to find out and that is to throw me into the ocean and see if I can swim." Larry's retort: "We are about to throw you off a cliff and see if you can fly." Bob does indeed fly as does the whole film. Well done all around.
Rated R. Running Time: 91 minutes. (C) 2000 Harvey Karten<EMAIL_ADDRESS>The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


This option allows you to add additional items to the grid's context menu. The option accepts a function which is expected to return an array of objects with the following supported attributes.
divider
(boolean)
- If true, this menu item will be a non-clickable divider line.enabled
(boolean)
- If true, this menu item will be visible.fn
(function)
- The function that will be executed when this menu item is clicked. More info below.icon
(string)
- Optional icon to display at the front of the item. Can be an img tag or a custom bit of HTML code.menu
(array)
- An array of submenu items.name
(string)
- This is the text that will be displayed to the user for this menu item.title
(boolean)
- If true, this menu item will be a non-clickable section title.value
(boolean)
- If true, this menu item will display a checkmark icon.
The function you provide will be supplied with the following arguments:
event
(object)
- The Javascript event object from the click event.grid
(object)
- The current instance of the Doby Grid.args
(object)
- Various information about the area in which the user initiated the menu (with row, cell and column information).
(function)
: Defaultnull
.
{
menuExtensions: function (event, grid, args) {
return [{
icon: '',
name: "Menu Item",
menu: [{
name: "Submenu Item",
fn: function (event) {
// Perform some action here
}
}]
}, {
divider: true
}, {
name: "Toggle Button",
value: your_option,
fn: function (event) {
your_option = !your_option;
}
}]
}
}
Right-click anywhere on the grid.


gdb(1) GNU Tools gdb(1) NAME gdb - The GNU Debugger SYNOPSIS gdb [-help] [-nx] [-q] [-batch] [-cd=dir] [-f] [-b bps] [-tty=dev] [-s symfile] [-e prog] [-se prog] [-c core] [-x cmds] [-d dir] [prog [core|procID]] gdb [options] --args prog [arguments] gdbtui [options] DESCRIPTION The purpose of a debugger such as GDB is to allow you to see what is going on ``inside'' another program while it executes—or what another program was doing at the moment it crashed. GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in the act: · Start your program, specifying anything that might affect its behavior. · Make your program stop on specified conditions. · Examine what has happened, when your program has stopped. · Change things in your program, so you can experiment with correct‐ ing the effects of one bug and go on to learn about another. You can use GDB to debug programs written in C, C++, and Modula-2. For‐ tran support will be added when a GNU Fortran compiler is ready. GDB is invoked with the shell command gdb. Once started, it reads com‐ mands from the terminal until you tell it to exit with the GDB command quit. You can get online help from gdb itself by using the command help. You can run gdb with no arguments or options; but the most usual way to start GDB is with one argument or two, specifying an executable program as the argument: gdb program You can also start with both an executable program and a core file speci‐ fied: gdb program core You can, instead, specify a process ID as a second argument, if you want to debug a running process: gdb program 1234 would attach GDB to process 1234 (unless you also have a file named `1234'; GDB does check for a core file first). Here are some of the most frequently needed GDB commands: break [file:]function Set a breakpoint at function (in file). run [arglist] Start your program (with arglist, if specified). bt Backtrace: display the program stack. print expr Display the value of an expression. c Continue running your program (after stopping, e.g. at a break‐ point). next Execute next program line (after stopping); step over any function calls in the line. edit [file:]function look at the program line where it is presently stopped. list [file:]function type the text of the program in the vicinity of where it is presently stopped. step Execute next program line (after stopping); step into any function calls in the line. help [name] Show information about GDB command name, or general information about using GDB. quit Exit from GDB. For full details on GDB, see Using GDB: A Guide to the GNU Source-Level Debugger, by Richard M. Stallman and Roland H. Pesch. The same text is available online as the gdb entry in the info program. OPTIONS Any arguments other than options specify an executable file and core file (or process ID); that is, the first argument encountered with no associ‐ ated option flag is equivalent to a `-se' option, and the second, if any, is equivalent to a `-c' option if it's the name of a file. Many options have both long and short forms; both are shown here. The long forms are also recognized if you truncate them, so long as enough of the option is present to be unambiguous. (If you prefer, you can flag option arguments with `+' rather than `-', though we illustrate the more usual conven‐ tion.) All the options and command line arguments you give are processed in sequential order. The order makes a difference when the `-x' option is used. -b bps Set the line speed (baud rate or bits per second) of any serial interface used by GDB for remote debugging. -batch Run in batch mode. Exit with status 0 after processing all the command files specified with `-x' (and `.gdbinit', if not inhib‐ ited). Exit with nonzero status if an error occurs in executing the GDB commands in the command files. Batch mode may be useful for running GDB as a filter, for example to download and run a program on another computer; in order to make this more useful, the message Program exited normally. (which is ordinarily issued whenever a program running under GDB control terminates) is not issued when running in batch mode. -c FILE, -core=FILE Use file file as a core dump to examine. -cd=directory Run GDB using directory as its working directory, instead of the current directory. -d DIRECTORY, -directory=DIRECTORY Add directory to the path to search for source files. -e FILE, -exec=FILE Use file file as the executable file to execute when appropriate, and for examining pure data in conjunction with a core dump. -f, -fullname Emacs sets this option when it runs GDB as a subprocess. It tells GDB to output the full file name and line number in a standard, recognizable fashion each time a stack frame is displayed (which includes each time the program stops). This recognizable format looks like two ` 32' characters, followed by the file name, line number and character position separated by colons, and a newline. The Emacs-to-GDB interface program uses the two ` 32' characters as a signal to display the source code for the frame. -h, -help List all options, with brief explanations. -n, -nx Do not execute commands from any `.gdbinit' initialization files. Normally, the commands in these files are executed after all the command options and arguments have been processed. -s FILE, -symbols=FILE Read symbol table from file file. -se=file Read symbol table from file file and use it as the executable file. q, -quiet ``Quiet''. Do not print the introductory and copyright messages. These messages are also suppressed in batch mode. -tty=device Run using device for your program's standard input and output. --args Pass arguments after the program name to the program when it is run. -tui Run GDB using a text (console) user interface. -write Enable writing into executable and core files. -x FILE, -command=FILE Execute GDB commands from file file. SEE ALSO `gdb' entry in info; Using GDB: A Guide to the GNU Source-Level Debugger, Richard M. Stallman and Roland H. Pesch, July 1991. COPYING Copyright (c) 1991, 2010 Free Software Foundation, Inc. Permission is granted to make and distribute verbatim copies of this man‐ ual provided the copyright notice and this permission notice are pre‐ served on all copies. Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permis‐ sion notice identical to this one. Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be included in translations approved by the Free Software Foundation instead of in the original Eng‐ lish. GNU Tools 22may2002 gdb(1)


Razer launches Kraken Ultimate headset with THX Spatial Audio
Razer is staying busy in November
by Shawn KnightIn brief: Razer on Thursday announced a new gaming headset featuring THX Spatial Audio technology for improved positional accuracy. And if $129.99 is outside of your budget, there's a second option on the table that's just $59.99.
The Razer Kraken Ultimate is described as a competitive gaming headset. It packs custom-tuned 50mm drivers beneath over-ear, RGB backlit cups featuring cooling gel for improved comfort and hidden channels for gamers that wear glasses. Razer said it produces the richest, most immersive and lifelike audio in the Kraken line.
The THX Spatial Audio technology reportedly delivers “pinpoint accuracy and greater than 7.1 surround sound” while the active noise cancelling microphone detects and nullifies background noise as you talk. Full headphone and microphone technical specifications are as follows:
Headphones
- Frequency response: 20 Hz – 20 kHz
- Impedance: 32 Ω @ 1 kHz
- Sensitivity (@1 kHz): 109 dB
- Input power: 30 mW (Max)
- Drivers: 50 mm, with Neodymium magnets
- Inner ear cup diameter: 64 mm / 2.5 in.
- Connection type: USB Digital;
- Cable length: 1.3 m / 4.27 ft.
- Approx. weight: 390 g / 0.86 lbs
Microphone
- Frequency response: 100 Hz – 10 kHz
- Impedance: 32 Ω @ 1 kHz
- Signal-to-noise ratio: > 60 dB
- Sensitivity (@1 kHz): -45 ± 3 dB
- Pick-up pattern: Unidirectional ECM boom
- Click-to-Mute Function
It’s been a busy November for Razer. Last week, the gaming brand unveiled a pair of Basilisk gaming mice covering both budget and premium price points. They also announced the Razer Kraken X USB on Thursday, a more affordable option that affords 7.1 surround sound and green earcup lighting in a lightweight design that tips the scales at just 275g.
The Razer Kraken Ultimate retails for $129.99 and is available to purchase directly from Razer and authorized retailers from starting today. The Kraken X USB can be yours for just $59.99.


This fixture is for manual testing purposes, and the patterns used in implementing it should not be used as a model. This is mainly for anyone working on making changes to the `schedule` module.
Calls the callback within the frame when not blocked:
Accepts multiple callbacks and calls within frame when not blocked
Schedules callbacks in correct order when they use scheduleCallback to schedule themselves
Calls timed out callbacks and then any more pending callbacks, defers others if time runs out
When some callbacks throw errors, still calls them all within the same frame
IMPORTANT: Open the console when you run this! Inspect the logs there!
When some callbacks throw errors and some also time out, still calls them all within the same frame
IMPORTANT: Open the console when you run this! Inspect the logs there!
Continues calling callbacks even when user switches away from this tab
Can pause execution, dump scheduled callbacks, and continue where it left off
Can force a specific framerate
IMPORTANT: This test may be flaky if other tests have been run in this js instance. To get a clean test refresh the page before running test 9
Runs scheduled JS work for 99% of the frame time when nothing else is using the thread.
NOTE: Try this test both when nothing else is running and when something is using the compositor thread in another visible tab with video or WebGL content (Shift+Click).
Runs scheduled JS work more than 95% of the frame time when inserting DOM nodes.
NOTE: Try this test both when nothing else is running and when something is using the compositor thread in another visible tab with video or WebGL content (Shift+Click).


Position:Wing Forward
Club:Valley Rovers
Hurling Level:Intermediate
Favourite Hurler:Henry Shefflin
Favourite Hurling Moment:2005 Cork Winning the All Ireland
South East A Hurling-U12,2010
Under 16 A County Football-U16,2016
Minor League Football-2017
Junior A League Football-2017
Junior A League Hurling-2017+2018
South East Junior A Hurling-2018
I started powerlifting when I was in sixth year.When I first started it was because I wanted to become stronger and it was only a hobby as such.
As time went on I grew more fond of the sport and did my first competition,placing third.
I also did a competition last December and won my overall category,Hitting Personal Records too which I was delighted with.
I train 5-6 times a week usually in my gym at home but I also train in the Maradyke Arena since I have many breaks during my college day.


|
|||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
Object org.apache.spark.sql.DataFrame
public class DataFrame
:: Experimental :: A distributed collection of data organized into named columns.
A DataFrame
is equivalent to a relational table in Spark SQL. The following example creates
a DataFrame
by pointing Spark SQL to a Parquet data set.
val people = sqlContext.read.parquet("...") // in Scala
DataFrame people = sqlContext.read().parquet("...") // in Java
Once created, it can be manipulated using the various domain-specific-language (DSL) functions
defined in: DataFrame
(this class), Column
, and functions
.
To select a column from the data frame, use apply
method in Scala and col
in Java.
val ageCol = people("age") // in Scala
Column ageCol = people.col("age") // in Java
Note that the Column
type can also be manipulated through its various functions.
// The following creates a new column that increases everybody's age by 10.
people("age") + 10 // in Scala
people.col("age").plus(10); // in Java
A more concrete example in Scala:
// To create DataFrame using SQLContext
val people = sqlContext.read.parquet("...")
val department = sqlContext.read.parquet("...")
people.filter("age > 30")
.join(department, people("deptId") === department("id"))
.groupBy(department("name"), "gender")
.agg(avg(people("salary")), max(people("age")))
and in Java:
// To create DataFrame using SQLContext
DataFrame people = sqlContext.read().parquet("...");
DataFrame department = sqlContext.read().parquet("...");
people.filter("age".gt(30))
.join(department, people.col("deptId").equalTo(department("id")))
.groupBy(department.col("name"), "gender")
.agg(avg(people.col("salary")), max(people.col("age")));
| Constructor Summary | |
|---|---|
DataFrame(SQLContext sqlContext,
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan)
A constructor that automatically analyzes the logical plan. |
| Method Summary | ||
|---|---|---|
DataFrame |
agg(Column expr,
Column... exprs)
Aggregates on the entire DataFrame without groups. |
|
DataFrame |
agg(Column expr,
scala.collection.Seq<Column> exprs)
Aggregates on the entire DataFrame without groups. |
|
DataFrame |
agg(scala.collection.immutable.Map<String,String> exprs)
(Scala-specific) Aggregates on the entire DataFrame without groups. |
|
DataFrame |
agg(java.util.Map<String,String> exprs)
(Java-specific) Aggregates on the entire DataFrame without groups. |
|
DataFrame |
agg(scala.Tuple2<String,String> aggExpr,
scala.collection.Seq<scala.Tuple2<String,String>> aggExprs)
(Scala-specific) Aggregates on the entire DataFrame without groups. |
|
Column |
apply(String colName)
Selects column based on the column name and return it as a Column . |
|
DataFrame |
as(String alias)
Returns a new DataFrame with an alias set. |
|
DataFrame |
as(scala.Symbol alias)
(Scala-specific) Returns a new DataFrame with an alias set. |
|
DataFrame |
cache()
|
|
DataFrame |
coalesce(int numPartitions)
Returns a new DataFrame that has exactly numPartitions partitions. |
|
Column |
col(String colName)
Selects column based on the column name and return it as a Column . |
|
Row[] |
collect()
Returns an array that contains all of Row s in this DataFrame . |
|
java.util.List<Row> |
collectAsList()
Returns a Java list that contains all of Row s in this DataFrame . |
|
String[] |
columns()
Returns all column names as an array. |
|
long |
count()
Returns the number of rows in the DataFrame . |
|
void |
createJDBCTable(String url,
String table,
boolean allowExisting)
Deprecated. As of 1.340, replaced by write().jdbc() . |
|
GroupedData |
cube(Column... cols)
Create a multi-dimensional cube for the current DataFrame using the specified columns,
so we can run aggregation on them. |
|
GroupedData |
cube(scala.collection.Seq<Column> cols)
Create a multi-dimensional cube for the current DataFrame using the specified columns,
so we can run aggregation on them. |
|
GroupedData |
cube(String col1,
scala.collection.Seq<String> cols)
Create a multi-dimensional cube for the current DataFrame using the specified columns,
so we can run aggregation on them. |
|
GroupedData |
cube(String col1,
String... cols)
Create a multi-dimensional cube for the current DataFrame using the specified columns,
so we can run aggregation on them. |
|
DataFrame |
describe(scala.collection.Seq<String> cols)
Computes statistics for numeric columns, including count, mean, stddev, min, and max. |
|
DataFrame |
describe(String... cols)
Computes statistics for numeric columns, including count, mean, stddev, min, and max. |
|
DataFrame |
distinct()
Returns a new DataFrame that contains only the unique rows from this DataFrame . |
|
DataFrame |
drop(Column col)
Returns a new DataFrame with a column dropped. |
|
DataFrame |
drop(String colName)
Returns a new DataFrame with a column dropped. |
|
DataFrame |
dropDuplicates()
Returns a new DataFrame that contains only the unique rows from this DataFrame . |
|
DataFrame |
dropDuplicates(scala.collection.Seq<String> colNames)
(Scala-specific) Returns a new DataFrame with duplicate rows removed, considering only
the subset of columns. |
|
DataFrame |
dropDuplicates(String[] colNames)
Returns a new DataFrame with duplicate rows removed, considering only
the subset of columns. |
|
scala.Tuple2<String,String>[] |
dtypes()
Returns all column names and their data types as an array. |
|
DataFrame |
except(DataFrame other)
Returns a new DataFrame containing rows in this frame but not in another frame. |
|
void |
explain()
Only prints the physical plan to the console for debugging purposes. |
|
void |
explain(boolean extended)
Prints the plans (logical and physical) to the console for debugging purposes. |
|
|
explode(scala.collection.Seq<Column> input,
scala.Function1<Row,scala.collection.TraversableOnce<A>> f,
scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
(Scala-specific) Returns a new DataFrame where each row has been expanded to zero or more
rows by the provided function. |
|
|
explode(String inputColumn,
String outputColumn,
scala.Function1<A,scala.collection.TraversableOnce<B>> f,
scala.reflect.api.TypeTags.TypeTag<B> evidence$2)
(Scala-specific) Returns a new DataFrame where a single column has been expanded to zero
or more rows by the provided function. |
|
DataFrame |
filter(Column condition)
Filters rows using the given condition. |
|
DataFrame |
filter(String conditionExpr)
Filters rows using the given SQL expression. |
|
Row |
first()
Returns the first row. |
|
|
flatMap(scala.Function1<Row,scala.collection.TraversableOnce<R>> f,
scala.reflect.ClassTag<R> evidence$4)
Returns a new RDD by first applying a function to all rows of this DataFrame ,
and then flattening the results. |
|
void |
foreach(scala.Function1<Row,scala.runtime.BoxedUnit> f)
Applies a function f to all rows. |
|
void |
foreachPartition(scala.Function1<scala.collection.Iterator<Row>,scala.runtime.BoxedUnit> f)
Applies a function f to each partition of this DataFrame . |
|
GroupedData |
groupBy(Column... cols)
Groups the DataFrame using the specified columns, so we can run aggregation on them. |
|
GroupedData |
groupBy(scala.collection.Seq<Column> cols)
Groups the DataFrame using the specified columns, so we can run aggregation on them. |
|
GroupedData |
groupBy(String col1,
scala.collection.Seq<String> cols)
Groups the DataFrame using the specified columns, so we can run aggregation on them. |
|
GroupedData |
groupBy(String col1,
String... cols)
Groups the DataFrame using the specified columns, so we can run aggregation on them. |
|
Row |
head()
Returns the first row. |
|
Row[] |
head(int n)
Returns the first n rows. |
|
void |
insertInto(String tableName)
Deprecated. As of 1.4.0, replaced by write().mode(SaveMode.Append).saveAsTable(tableName) . |
|
void |
insertInto(String tableName,
boolean overwrite)
Deprecated. As of 1.4.0, replaced by write().mode(SaveMode.Append|SaveMode.Overwrite).saveAsTable(tableName) . |
|
void |
insertIntoJDBC(String url,
String table,
boolean overwrite)
Deprecated. As of 1.4.0, replaced by write().jdbc() . |
|
DataFrame |
intersect(DataFrame other)
Returns a new DataFrame containing rows only in both this frame and another frame. |
|
boolean |
isLocal()
Returns true if the collect and take methods can be run locally
(without any Spark executors). |
|
JavaRDD<Row> |
javaRDD()
Returns the content of the DataFrame as a JavaRDD of Row s. |
|
DataFrame |
join(DataFrame right)
Cartesian join with another DataFrame . |
|
DataFrame |
join(DataFrame right,
Column joinExprs)
Inner join with another DataFrame , using the given join expression. |
|
DataFrame |
join(DataFrame right,
Column joinExprs,
String joinType)
Join with another DataFrame , using the given join expression. |
|
DataFrame |
join(DataFrame right,
String usingColumn)
Inner equi-join with another DataFrame using the given column. |
|
DataFrame |
limit(int n)
Returns a new DataFrame by taking the first n rows. |
|
|
map(scala.Function1<Row,R> f,
scala.reflect.ClassTag<R> evidence$3)
Returns a new RDD by applying a function to all rows of this DataFrame. |
|
|
mapPartitions(scala.Function1<scala.collection.Iterator<Row>,scala.collection.Iterator<R>> f,
scala.reflect.ClassTag<R> evidence$5)
Returns a new RDD by applying a function to each partition of this DataFrame. |
|
DataFrameNaFunctions |
na()
Returns a DataFrameNaFunctions for working with missing data. |
|
DataFrame |
orderBy(Column... sortExprs)
Returns a new DataFrame sorted by the given expressions. |
|
DataFrame |
orderBy(scala.collection.Seq<Column> sortExprs)
Returns a new DataFrame sorted by the given expressions. |
|
DataFrame |
orderBy(String sortCol,
scala.collection.Seq<String> sortCols)
Returns a new DataFrame sorted by the given expressions. |
|
DataFrame |
orderBy(String sortCol,
String... sortCols)
Returns a new DataFrame sorted by the given expressions. |
|
DataFrame |
persist()
|
|
DataFrame |
persist(StorageLevel newLevel)
|
|
void |
printSchema()
Prints the schema to the console in a nice tree format. |
|
org.apache.spark.sql.SQLContext.QueryExecution |
queryExecution()
|
|
DataFrame[] |
randomSplit(double[] weights)
Randomly splits this DataFrame with the provided weights. |
|
DataFrame[] |
randomSplit(double[] weights,
long seed)
Randomly splits this DataFrame with the provided weights. |
|
RDD<Row> |
rdd()
Represents the content of the DataFrame as an RDD of Row s. |
|
void |
registerTempTable(String tableName)
Registers this DataFrame as a temporary table using the given name. |
|
DataFrame |
repartition(int numPartitions)
Returns a new DataFrame that has exactly numPartitions partitions. |
|
GroupedData |
rollup(Column... cols)
Create a multi-dimensional rollup for the current DataFrame using the specified columns,
so we can run aggregation on them. |
|
GroupedData |
rollup(scala.collection.Seq<Column> cols)
Create a multi-dimensional rollup for the current DataFrame using the specified columns,
so we can run aggregation on them. |
|
GroupedData |
rollup(String col1,
scala.collection.Seq<String> cols)
Create a multi-dimensional rollup for the current DataFrame using the specified columns,
so we can run aggregation on them. |
|
GroupedData |
rollup(String col1,
String... cols)
Create a multi-dimensional rollup for the current DataFrame using the specified columns,
so we can run aggregation on them. |
|
DataFrame |
sample(boolean withReplacement,
double fraction)
Returns a new DataFrame by sampling a fraction of rows, using a random seed. |
|
DataFrame |
sample(boolean withReplacement,
double fraction,
long seed)
Returns a new DataFrame by sampling a fraction of rows. |
|
void |
save(String path)
Deprecated. As of 1.4.0, replaced by write().save(path) . |
|
void |
save(String path,
SaveMode mode)
Deprecated. As of 1.4.0, replaced by write().mode(mode).save(path) . |
|
void |
save(String source,
SaveMode mode,
java.util.Map<String,String> options)
Deprecated. As of 1.4.0, replaced by write().format(source).mode(mode).options(options).save(path) . |
|
void |
save(String source,
SaveMode mode,
scala.collection.immutable.Map<String,String> options)
Deprecated. As of 1.4.0, replaced by write().format(source).mode(mode).options(options).save(path) . |
|
void |
save(String path,
String source)
Deprecated. As of 1.4.0, replaced by write().format(source).save(path) . |
|
void |
save(String path,
String source,
SaveMode mode)
Deprecated. As of 1.4.0, replaced by write().format(source).mode(mode).save(path) . |
|
void |
saveAsParquetFile(String path)
Deprecated. As of 1.4.0, replaced by write().parquet() . |
|
void |
saveAsTable(String tableName)
Deprecated. As of 1.4.0, replaced by write().saveAsTable(tableName) . |
|
void |
saveAsTable(String tableName,
SaveMode mode)
Deprecated. As of 1.4.0, replaced by write().mode(mode).saveAsTable(tableName) . |
|
void |
saveAsTable(String tableName,
String source)
Deprecated. As of 1.4.0, replaced by write().format(source).saveAsTable(tableName) . |
|
void |
saveAsTable(String tableName,
String source,
SaveMode mode)
Deprecated. As of 1.4.0, replaced by write().mode(mode).saveAsTable(tableName) . |
|
void |
saveAsTable(String tableName,
String source,
SaveMode mode,
java.util.Map<String,String> options)
Deprecated. As of 1.4.0, replaced by write().format(source).mode(mode).options(options).saveAsTable(tableName) . |
|
void |
saveAsTable(String tableName,
String source,
SaveMode mode,
scala.collection.immutable.Map<String,String> options)
Deprecated. As of 1.4.0, replaced by write().format(source).mode(mode).options(options).saveAsTable(tableName) . |
|
StructType |
schema()
Returns the schema of this DataFrame . |
|
DataFrame |
select(Column... cols)
Selects a set of column based expressions. |
|
DataFrame |
select(scala.collection.Seq<Column> cols)
Selects a set of column based expressions. |
|
DataFrame |
select(String col,
scala.collection.Seq<String> cols)
Selects a set of columns. |
|
DataFrame |
select(String col,
String... cols)
Selects a set of columns. |
|
DataFrame |
selectExpr(scala.collection.Seq<String> exprs)
Selects a set of SQL expressions. |
|
DataFrame |
selectExpr(String... exprs)
Selects a set of SQL expressions. |
|
void |
show()
Displays the top 20 rows of DataFrame in a tabular form. |
|
void |
show(int numRows)
Displays the DataFrame in a tabular form. |
|
DataFrame |
sort(Column... sortExprs)
Returns a new DataFrame sorted by the given expressions. |
|
DataFrame |
sort(scala.collection.Seq<Column> sortExprs)
Returns a new DataFrame sorted by the given expressions. |
|
DataFrame |
sort(String sortCol,
scala.collection.Seq<String> sortCols)
Returns a new DataFrame sorted by the specified column, all in ascending order. |
|
DataFrame |
sort(String sortCol,
String... sortCols)
Returns a new DataFrame sorted by the specified column, all in ascending order. |
|
SQLContext |
sqlContext()
|
|
DataFrameStatFunctions |
stat()
Returns a DataFrameStatFunctions for working statistic functions support. |
|
Row[] |
take(int n)
Returns the first n rows in the DataFrame . |
|
DataFrame |
toDF()
Returns the object itself. |
|
DataFrame |
toDF(scala.collection.Seq<String> colNames)
Returns a new DataFrame with columns renamed. |
|
DataFrame |
toDF(String... colNames)
Returns a new DataFrame with columns renamed. |
|
JavaRDD<Row> |
toJavaRDD()
Returns the content of the DataFrame as a JavaRDD of Row s. |
|
RDD<String> |
toJSON()
Returns the content of the DataFrame as a RDD of JSON strings. |
|
DataFrame |
toSchemaRDD()
Deprecated. As of 1.3.0, replaced by toDF() . |
|
String |
toString()
|
|
DataFrame |
unionAll(DataFrame other)
Returns a new DataFrame containing union of rows in this frame and another frame. |
|
DataFrame |
unpersist()
|
|
DataFrame |
unpersist(boolean blocking)
|
|
DataFrame |
where(Column condition)
Filters rows using the given condition. |
|
DataFrame |
withColumn(String colName,
Column col)
Returns a new DataFrame by adding a column. |
|
DataFrame |
withColumnRenamed(String existingName,
String newName)
Returns a new DataFrame with a column renamed. |
|
DataFrameWriter |
write()
:: Experimental :: Interface for saving the content of the DataFrame out into external storage. |
| Methods inherited from class Object |
|---|
equals, getClass, hashCode, notify, notifyAll, wait, wait, wait |
| Constructor Detail |
|---|
public DataFrame(SQLContext sqlContext, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan)
This reports error eagerly as the DataFrame
is constructed, unless
SQLConf.dataFrameEagerAnalysis
is turned off.
sqlContext
- (undocumented)logicalPlan
- (undocumented)| Method Detail |
|---|
public DataFrame toDF(String... colNames)
DataFrame
with columns renamed. This can be quite convenient in conversion
from a RDD of tuples into a DataFrame
with meaningful names. For example:
val rdd: RDD[(Int, String)] = ...
rdd.toDF() // this implicit conversion creates a DataFrame with column name _1 and _2
rdd.toDF("id", "name") // this creates a DataFrame with column name "id" and "name"
colNames
- (undocumented)
public DataFrame sort(String sortCol, String... sortCols)
DataFrame
sorted by the specified column, all in ascending order.
// The following 3 are equivalent
df.sort("sortcol")
df.sort($"sortcol")
df.sort($"sortcol".asc)
sortCol
- (undocumented)sortCols
- (undocumented)
public DataFrame sort(Column... sortExprs)
DataFrame
sorted by the given expressions. For example:
df.sort($"col1", $"col2".desc)
sortExprs
- (undocumented)
public DataFrame orderBy(String sortCol, String... sortCols)
DataFrame
sorted by the given expressions.
This is an alias of the sort
function.
sortCol
- (undocumented)sortCols
- (undocumented)
public DataFrame orderBy(Column... sortExprs)
DataFrame
sorted by the given expressions.
This is an alias of the sort
function.
sortExprs
- (undocumented)
public DataFrame select(Column... cols)
df.select($"colA", $"colB" + 1)
cols
- (undocumented)
public DataFrame select(String col, String... cols)
select
that can only select
existing columns using column names (i.e. cannot construct expressions).
// The following two are equivalent:
df.select("colA", "colB")
df.select($"colA", $"colB")
col
- (undocumented)cols
- (undocumented)
public DataFrame selectExpr(String... exprs)
select
that accepts
SQL expressions.
df.selectExpr("colA", "colB as newName", "abs(colC)")
exprs
- (undocumented)
public GroupedData groupBy(Column... cols)
DataFrame
using the specified columns, so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns grouped by department.
df.groupBy($"department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)
public GroupedData rollup(Column... cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns rolluped by department and group.
df.rollup($"department", $"group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)
public GroupedData cube(Column... cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns cubed by department and group.
df.cube($"department", $"group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)
public GroupedData groupBy(String col1, String... cols)
DataFrame
using the specified columns, so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of groupBy that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns grouped by department.
df.groupBy("department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)
public GroupedData rollup(String col1, String... cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of rollup that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns rolluped by department and group.
df.rollup("department", "group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)
public GroupedData cube(String col1, String... cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of cube that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns cubed by department and group.
df.cube("department", "group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)
public DataFrame agg(Column expr, Column... exprs)
DataFrame
without groups.
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(max($"age"), avg($"salary"))
df.groupBy().agg(max($"age"), avg($"salary"))
expr
- (undocumented)exprs
- (undocumented)
public DataFrame describe(String... cols)
This function is meant for exploratory data analysis, as we make no guarantee about the
backward compatibility of the schema of the resulting DataFrame
. If you want to
programmatically compute summary statistics, use the agg
function instead.
df.describe("age", "height").show()
// output:
// summary age height
// count 10.0 10.0
// mean 53.3 178.05
// stddev 11.6 15.7
// min 18.0 163.0
// max 92.0 192.0
cols
- (undocumented)
public SQLContext sqlContext()
public org.apache.spark.sql.SQLContext.QueryExecution queryExecution()
public String toString()
toString
in class Object
public DataFrame toDF()
public DataFrame toDF(scala.collection.Seq<String> colNames)
DataFrame
with columns renamed. This can be quite convenient in conversion
from a RDD of tuples into a DataFrame
with meaningful names. For example:
val rdd: RDD[(Int, String)] = ...
rdd.toDF() // this implicit conversion creates a DataFrame with column name _1 and _2
rdd.toDF("id", "name") // this creates a DataFrame with column name "id" and "name"
colNames
- (undocumented)
public StructType schema()
DataFrame
.
public scala.Tuple2<String,String>[] dtypes()
public String[] columns()
public void printSchema()
public void explain(boolean extended)
extended
- (undocumented)public void explain()
public boolean isLocal()
collect
and take
methods can be run locally
(without any Spark executors).
public void show(int numRows)
DataFrame
in a tabular form. For example:
year month AVG('Adj Close) MAX('Adj Close)
1980 12 0.503218 0.595103
1981 01 0.523289 0.570307
1982 02 0.436504 0.475256
1983 03 0.410516 0.442194
1984 04 0.450090 0.483521
numRows
- Number of rows to show
public void show()
DataFrame
in a tabular form.
public DataFrameNaFunctions na()
DataFrameNaFunctions
for working with missing data.
// Dropping rows containing any null values.
df.na.drop()
public DataFrameStatFunctions stat()
DataFrameStatFunctions
for working statistic functions support.
// Finding frequent items in column with name 'a'.
df.stat.freqItems(Seq("a"))
public DataFrame join(DataFrame right)
DataFrame
.
Note that cartesian joins are very expensive without an extra filter that can be pushed down.
right
- Right side of the join operation.
public DataFrame join(DataFrame right, String usingColumn)
DataFrame
using the given column.
Different from other join functions, the join column will only appear once in the output,
i.e. similar to SQL's JOIN USING
syntax.
// Joining df1 and df2 using the column "user_id"
df1.join(df2, "user_id")
Note that if you perform a self-join using this function without aliasing the input
DataFrame
s, you will NOT be able to reference any columns after the join, since
there is no way to disambiguate which side of the join you would like to reference.
right
- Right side of the join operation.usingColumn
- Name of the column to join on. This column must exist on both sides.
public DataFrame join(DataFrame right, Column joinExprs)
DataFrame
, using the given join expression.
// The following two are equivalent:
df1.join(df2, $"df1Key" === $"df2Key")
df1.join(df2).where($"df1Key" === $"df2Key")
right
- (undocumented)joinExprs
- (undocumented)
public DataFrame join(DataFrame right, Column joinExprs, String joinType)
DataFrame
, using the given join expression. The following performs
a full outer join between df1
and df2
.
// Scala:
import org.apache.spark.sql.functions._
df1.join(df2, $"df1Key" === $"df2Key", "outer")
// Java:
import static org.apache.spark.sql.functions.*;
df1.join(df2, col("df1Key").equalTo(col("df2Key")), "outer");
right
- Right side of the join.joinExprs
- Join expression.joinType
- One of: inner
, outer
, left_outer
, right_outer
, leftsemi
.
public DataFrame sort(String sortCol, scala.collection.Seq<String> sortCols)
DataFrame
sorted by the specified column, all in ascending order.
// The following 3 are equivalent
df.sort("sortcol")
df.sort($"sortcol")
df.sort($"sortcol".asc)
sortCol
- (undocumented)sortCols
- (undocumented)
public DataFrame sort(scala.collection.Seq<Column> sortExprs)
DataFrame
sorted by the given expressions. For example:
df.sort($"col1", $"col2".desc)
sortExprs
- (undocumented)
public DataFrame orderBy(String sortCol, scala.collection.Seq<String> sortCols)
DataFrame
sorted by the given expressions.
This is an alias of the sort
function.
sortCol
- (undocumented)sortCols
- (undocumented)
public DataFrame orderBy(scala.collection.Seq<Column> sortExprs)
DataFrame
sorted by the given expressions.
This is an alias of the sort
function.
sortExprs
- (undocumented)
public Column apply(String colName)
Column
.
colName
- (undocumented)
public Column col(String colName)
Column
.
colName
- (undocumented)
public DataFrame as(String alias)
DataFrame
with an alias set.
alias
- (undocumented)
public DataFrame as(scala.Symbol alias)
DataFrame
with an alias set.
alias
- (undocumented)
public DataFrame select(scala.collection.Seq<Column> cols)
df.select($"colA", $"colB" + 1)
cols
- (undocumented)
public DataFrame select(String col, scala.collection.Seq<String> cols)
select
that can only select
existing columns using column names (i.e. cannot construct expressions).
// The following two are equivalent:
df.select("colA", "colB")
df.select($"colA", $"colB")
col
- (undocumented)cols
- (undocumented)
public DataFrame selectExpr(scala.collection.Seq<String> exprs)
select
that accepts
SQL expressions.
df.selectExpr("colA", "colB as newName", "abs(colC)")
exprs
- (undocumented)
public DataFrame filter(Column condition)
// The following are equivalent:
peopleDf.filter($"age" > 15)
peopleDf.where($"age" > 15)
condition
- (undocumented)
public DataFrame filter(String conditionExpr)
peopleDf.filter("age > 15")
conditionExpr
- (undocumented)
public DataFrame where(Column condition)
filter
.
// The following are equivalent:
peopleDf.filter($"age" > 15)
peopleDf.where($"age" > 15)
condition
- (undocumented)
public GroupedData groupBy(scala.collection.Seq<Column> cols)
DataFrame
using the specified columns, so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns grouped by department.
df.groupBy($"department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)
public GroupedData rollup(scala.collection.Seq<Column> cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns rolluped by department and group.
df.rollup($"department", $"group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)
public GroupedData cube(scala.collection.Seq<Column> cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
// Compute the average for all numeric columns cubed by department and group.
df.cube($"department", $"group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
cols
- (undocumented)
public GroupedData groupBy(String col1, scala.collection.Seq<String> cols)
DataFrame
using the specified columns, so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of groupBy that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns grouped by department.
df.groupBy("department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)
public GroupedData rollup(String col1, scala.collection.Seq<String> cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of rollup that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns rolluped by department and group.
df.rollup("department", "group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)
public GroupedData cube(String col1, scala.collection.Seq<String> cols)
DataFrame
using the specified columns,
so we can run aggregation on them.
See GroupedData
for all the available aggregate functions.
This is a variant of cube that can only group by existing columns using column names (i.e. cannot construct expressions).
// Compute the average for all numeric columns cubed by department and group.
df.cube("department", "group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
col1
- (undocumented)cols
- (undocumented)
public DataFrame agg(scala.Tuple2<String,String> aggExpr, scala.collection.Seq<scala.Tuple2<String,String>> aggExprs)
DataFrame
without groups.
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg("age" -> "max", "salary" -> "avg")
df.groupBy().agg("age" -> "max", "salary" -> "avg")
aggExpr
- (undocumented)aggExprs
- (undocumented)
public DataFrame agg(scala.collection.immutable.Map<String,String> exprs)
DataFrame
without groups.
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(Map("age" -> "max", "salary" -> "avg"))
df.groupBy().agg(Map("age" -> "max", "salary" -> "avg"))
exprs
- (undocumented)
public DataFrame agg(java.util.Map<String,String> exprs)
DataFrame
without groups.
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(Map("age" -> "max", "salary" -> "avg"))
df.groupBy().agg(Map("age" -> "max", "salary" -> "avg"))
exprs
- (undocumented)
public DataFrame agg(Column expr, scala.collection.Seq<Column> exprs)
DataFrame
without groups.
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(max($"age"), avg($"salary"))
df.groupBy().agg(max($"age"), avg($"salary"))
expr
- (undocumented)exprs
- (undocumented)
public DataFrame limit(int n)
DataFrame
by taking the first n
rows. The difference between this function
and head
is that head
returns an array while limit
returns a new DataFrame
.
n
- (undocumented)
public DataFrame unionAll(DataFrame other)
DataFrame
containing union of rows in this frame and another frame.
This is equivalent to UNION ALL
in SQL.
other
- (undocumented)
public DataFrame intersect(DataFrame other)
DataFrame
containing rows only in both this frame and another frame.
This is equivalent to INTERSECT
in SQL.
other
- (undocumented)
public DataFrame except(DataFrame other)
DataFrame
containing rows in this frame but not in another frame.
This is equivalent to EXCEPT
in SQL.
other
- (undocumented)
public DataFrame sample(boolean withReplacement, double fraction, long seed)
DataFrame
by sampling a fraction of rows.
withReplacement
- Sample with replacement or not.fraction
- Fraction of rows to generate.seed
- Seed for sampling.
public DataFrame sample(boolean withReplacement, double fraction)
DataFrame
by sampling a fraction of rows, using a random seed.
withReplacement
- Sample with replacement or not.fraction
- Fraction of rows to generate.
public DataFrame[] randomSplit(double[] weights, long seed)
DataFrame
with the provided weights.
weights
- weights for splits, will be normalized if they don't sum to 1.seed
- Seed for sampling.
public DataFrame[] randomSplit(double[] weights)
DataFrame
with the provided weights.
weights
- weights for splits, will be normalized if they don't sum to 1.
public <A extends scala.Product> DataFrame explode(scala.collection.Seq<Column> input, scala.Function1<Row,scala.collection.TraversableOnce<A>> f, scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
DataFrame
where each row has been expanded to zero or more
rows by the provided function. This is similar to a LATERAL VIEW
in HiveQL. The columns of
the input row are implicitly joined with each row that is output by the function.
The following example uses this function to count the number of books which contain a given word:
case class Book(title: String, words: String)
val df: RDD[Book]
case class Word(word: String)
val allWords = df.explode('words) {
case Row(words: String) => words.split(" ").map(Word(_))
}
val bookCountPerWord = allWords.groupBy("word").agg(countDistinct("title"))
input
- (undocumented)f
- (undocumented)evidence$1
- (undocumented)
public <A,B> DataFrame explode(String inputColumn, String outputColumn, scala.Function1<A,scala.collection.TraversableOnce<B>> f, scala.reflect.api.TypeTags.TypeTag<B> evidence$2)
DataFrame
where a single column has been expanded to zero
or more rows by the provided function. This is similar to a LATERAL VIEW
in HiveQL. All
columns of the input row are implicitly joined with each value that is output by the function.
df.explode("words", "word"){words: String => words.split(" ")}
inputColumn
- (undocumented)outputColumn
- (undocumented)f
- (undocumented)evidence$2
- (undocumented)
public DataFrame withColumn(String colName, Column col)
DataFrame
by adding a column.
colName
- (undocumented)col
- (undocumented)
public DataFrame withColumnRenamed(String existingName, String newName)
DataFrame
with a column renamed.
This is a no-op if schema doesn't contain existingName.
existingName
- (undocumented)newName
- (undocumented)
public DataFrame drop(String colName)
DataFrame
with a column dropped.
This is a no-op if schema doesn't contain column name.
colName
- (undocumented)
public DataFrame drop(Column col)
DataFrame
with a column dropped.
This version of drop accepts a Column rather than a name.
This is a no-op if the DataFrame doesn't have a column
with an equivalent expression.
col
- (undocumented)
public DataFrame dropDuplicates()
DataFrame
that contains only the unique rows from this DataFrame
.
This is an alias for distinct
.
public DataFrame dropDuplicates(scala.collection.Seq<String> colNames)
DataFrame
with duplicate rows removed, considering only
the subset of columns.
colNames
- (undocumented)
public DataFrame dropDuplicates(String[] colNames)
DataFrame
with duplicate rows removed, considering only
the subset of columns.
colNames
- (undocumented)
public DataFrame describe(scala.collection.Seq<String> cols)
This function is meant for exploratory data analysis, as we make no guarantee about the
backward compatibility of the schema of the resulting DataFrame
. If you want to
programmatically compute summary statistics, use the agg
function instead.
df.describe("age", "height").show()
// output:
// summary age height
// count 10.0 10.0
// mean 53.3 178.05
// stddev 11.6 15.7
// min 18.0 163.0
// max 92.0 192.0
cols
- (undocumented)
public Row[] head(int n)
n
rows.
n
- (undocumented)
public Row head()
public Row first()
public <R> RDD<R> map(scala.Function1<Row,R> f, scala.reflect.ClassTag<R> evidence$3)
f
- (undocumented)evidence$3
- (undocumented)
public <R> RDD<R> flatMap(scala.Function1<Row,scala.collection.TraversableOnce<R>> f, scala.reflect.ClassTag<R> evidence$4)
DataFrame
,
and then flattening the results.
f
- (undocumented)evidence$4
- (undocumented)
public <R> RDD<R> mapPartitions(scala.Function1<scala.collection.Iterator<Row>,scala.collection.Iterator<R>> f, scala.reflect.ClassTag<R> evidence$5)
f
- (undocumented)evidence$5
- (undocumented)
public void foreach(scala.Function1<Row,scala.runtime.BoxedUnit> f)
f
to all rows.
f
- (undocumented)public void foreachPartition(scala.Function1<scala.collection.Iterator<Row>,scala.runtime.BoxedUnit> f)
DataFrame
.
f
- (undocumented)public Row[] take(int n)
n
rows in the DataFrame
.
n
- (undocumented)
public Row[] collect()
Row
s in this DataFrame
.
public java.util.List<Row> collectAsList()
Row
s in this DataFrame
.
public long count()
DataFrame
.
public DataFrame repartition(int numPartitions)
DataFrame
that has exactly numPartitions
partitions.
numPartitions
- (undocumented)
public DataFrame coalesce(int numPartitions)
DataFrame
that has exactly numPartitions
partitions.
Similar to coalesce defined on an RDD
, this operation results in a narrow dependency, e.g.
if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of
the 100 new partitions will claim 10 of the current partitions.
numPartitions
- (undocumented)
public DataFrame distinct()
DataFrame
that contains only the unique rows from this DataFrame
.
This is an alias for dropDuplicates
.
public DataFrame persist()
public DataFrame cache()
public DataFrame persist(StorageLevel newLevel)
newLevel
- (undocumented)
public DataFrame unpersist(boolean blocking)
blocking
- (undocumented)
public DataFrame unpersist()
public RDD<Row> rdd()
DataFrame
as an RDD
of Row
s. Note that the RDD is
memoized. Once called, it won't change even if you change any query planning related Spark SQL
configurations (e.g. spark.sql.shuffle.partitions
).
public JavaRDD<Row> toJavaRDD()
DataFrame
as a JavaRDD
of Row
s.
public JavaRDD<Row> javaRDD()
DataFrame
as a JavaRDD
of Row
s.
public void registerTempTable(String tableName)
DataFrame
as a temporary table using the given name. The lifetime of this
temporary table is tied to the SQLContext
that was used to create this DataFrame.
tableName
- (undocumented)public DataFrameWriter write()
DataFrame
out into external storage.
public RDD<String> toJSON()
DataFrame
as a RDD of JSON strings.
public DataFrame toSchemaRDD()
toDF()
.
public void createJDBCTable(String url, String table, boolean allowExisting)
write().jdbc()
.
DataFrame
to a JDBC database at url
under the table name table
.
This will run a CREATE TABLE
and a bunch of INSERT INTO
statements.
If you pass true
for allowExisting
, it will drop any table with the
given name; if you pass false
, it will throw if the table already
exists.
url
- (undocumented)table
- (undocumented)allowExisting
- (undocumented)public void insertIntoJDBC(String url, String table, boolean overwrite)
write().jdbc()
.
DataFrame
to a JDBC database at url
under the table name table
.
Assumes the table already exists and has a compatible schema. If you
pass true
for overwrite
, it will TRUNCATE
the table before
performing the INSERT
s.
The table must already exist on the database. It must have a schema
that is compatible with the schema of this RDD; inserting the rows of
the RDD in order via the simple statement
INSERT INTO table VALUES (?, ?, ..., ?)
should not fail.
url
- (undocumented)table
- (undocumented)overwrite
- (undocumented)public void saveAsParquetFile(String path)
write().parquet()
.
DataFrame
as a parquet file, preserving the schema.
Files that are written out using this method can be read back in as a DataFrame
using the parquetFile
function in SQLContext
.
path
- (undocumented)public void saveAsTable(String tableName)
write().saveAsTable(tableName)
.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
Also note that while this function can persist the table metadata into Hive's metastore, the table will NOT be accessible from Hive, until SPARK-7550 is resolved.
tableName
- (undocumented)public void saveAsTable(String tableName, SaveMode mode)
write().mode(mode).saveAsTable(tableName)
.
SaveMode.ErrorIfExists
as the save mode.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
Also note that while this function can persist the table metadata into Hive's metastore, the table will NOT be accessible from Hive, until SPARK-7550 is resolved.
tableName
- (undocumented)mode
- (undocumented)public void saveAsTable(String tableName, String source)
write().format(source).saveAsTable(tableName)
.
SaveMode.ErrorIfExists
as the save mode.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
Also note that while this function can persist the table metadata into Hive's metastore, the table will NOT be accessible from Hive, until SPARK-7550 is resolved.
tableName
- (undocumented)source
- (undocumented)public void saveAsTable(String tableName, String source, SaveMode mode)
write().mode(mode).saveAsTable(tableName)
.
SaveMode
specified by mode, and a set of options.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
Also note that while this function can persist the table metadata into Hive's metastore, the table will NOT be accessible from Hive, until SPARK-7550 is resolved.
tableName
- (undocumented)source
- (undocumented)mode
- (undocumented)public void saveAsTable(String tableName, String source, SaveMode mode, java.util.Map<String,String> options)
write().format(source).mode(mode).options(options).saveAsTable(tableName)
.
SaveMode
specified by mode, and a set of options.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
Also note that while this function can persist the table metadata into Hive's metastore, the table will NOT be accessible from Hive, until SPARK-7550 is resolved.
tableName
- (undocumented)source
- (undocumented)mode
- (undocumented)options
- (undocumented)public void saveAsTable(String tableName, String source, SaveMode mode, scala.collection.immutable.Map<String,String> options)
write().format(source).mode(mode).options(options).saveAsTable(tableName)
.
SaveMode
specified by mode, and a set of options.
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an insertInto
.
Also note that while this function can persist the table metadata into Hive's metastore, the table will NOT be accessible from Hive, until SPARK-7550 is resolved.
tableName
- (undocumented)source
- (undocumented)mode
- (undocumented)options
- (undocumented)public void save(String path)
write().save(path)
.
SaveMode.ErrorIfExists
as the save mode.
path
- (undocumented)public void save(String path, SaveMode mode)
write().mode(mode).save(path)
.
SaveMode
specified by mode,
using the default data source configured by spark.sql.sources.default.
path
- (undocumented)mode
- (undocumented)public void save(String path, String source)
write().format(source).save(path)
.
SaveMode.ErrorIfExists
as the save mode.
path
- (undocumented)source
- (undocumented)public void save(String path, String source, SaveMode mode)
write().format(source).mode(mode).save(path)
.
SaveMode
specified by mode.
path
- (undocumented)source
- (undocumented)mode
- (undocumented)public void save(String source, SaveMode mode, java.util.Map<String,String> options)
write().format(source).mode(mode).options(options).save(path)
.
SaveMode
specified by mode, and a set of options.
source
- (undocumented)mode
- (undocumented)options
- (undocumented)public void save(String source, SaveMode mode, scala.collection.immutable.Map<String,String> options)
write().format(source).mode(mode).options(options).save(path)
.
SaveMode
specified by mode, and a set of options
source
- (undocumented)mode
- (undocumented)options
- (undocumented)public void insertInto(String tableName, boolean overwrite)
write().mode(SaveMode.Append|SaveMode.Overwrite).saveAsTable(tableName)
.
tableName
- (undocumented)overwrite
- (undocumented)public void insertInto(String tableName)
write().mode(SaveMode.Append).saveAsTable(tableName)
.
tableName
- (undocumented)
|
|||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |


We examined 383 enzyme superfamilies in CATH/Gene 3D v4.2 that contain well-known (experimentally-validated) catalytic domains, and identified the proportion of functional families that have enzyme annotations and compared them to those that lack any enzyme annotation. These are highly populated superfamilies accounting for 64% of sequences in all CATH enzyme superfamilies and 60% of all sequences in CATH.
A functional family was considered to have enzyme annotations if it has at least one relative that has an EC annotation in UniProtKB and an experimental Gene Ontology (GO) annotation for ‘catalytic activity’. For a third of these enzyme superfamilies, all functional families were annotated as enzymes. However, approximately 252 enzyme superfamilies (two-thirds) had varying proportions of functional families that had no enzyme annotations in the EC classification or GO, suggesting that these are very likely to be pseudoenzymes.
| Superfamily ID | FunFams | Sequences | FunFams With Annotations | FunFams Without Annotations | Uniq EC | Uniq GO | |
|---|---|---|---|---|---|---|---|
| Superfamily ID | FunFams | Sequences | FunFams With Annotations | FunFams Without Annotations | Uniq EC | Uniq GO |