Note: The Nested Teams API is currently available for developers to preview. See the blog post for full details. To access the API, you must provide a custom media type in the Accept
header:
application/vnd.github.hellcat-preview+json
Warning: The API may change without advance notice during the preview period. Preview features are not supported for production use. If you experience any issues, contact your GitHub Enterprise site administrator.
For organization-owned repositories, the list of collaborators includes outside collaborators, organization members that are direct collaborators, organization members with access through team memberships, organization members with access through default organization permissions, and organization owners.
If you pass the hellcat-preview
media type, team members will include the members of child teams.
GET /repos/:owner/:repo/collaborators/:username
Status: 204 No Content
Status: 404 Not Found


Genesis 44:7
And they said unto him, Wherefore saith my lord these words? God forbid that thy servants should do according to this thing:


Uncle Tom's Cabin (Told to the Children)
by Harriet Beecher Stowe
Chapter 13: George Fights For Freedom
Additional Information
- Year Published: 1852
- Language: English
- Country of Origin: United States of America
- Source: Stowe, H. B. (1852). Uncle Tom's Cabin (Told to the Children). H. E. Marshall, (Ed.).
-
Readability:
- Flesch–Kincaid Level: 3.7
- Word Count: 934
- Genre: Historical Fiction
- Keywords: 19th century literature, american literature
- ✎ Cite This
Downloads
The day after George and Eliza met each other once more at the end of so many sad months of parting, was a very happy one in the Quaker house.
The two had much to say to each other. George had to tell how he had escaped from his cruel master, and how he had followed Eliza all the way and at last found her. Then there were plans to make for going on towards Canada. It was arranged that they should start that night at ten o'clock. 'The pursuers are hard after thee, we must not delay,' said Simeon.
Rachel was happy and busy, packing up food and clothes for them to take on the journey.
Late in the afternoon another Quaker, called Phineas, came with the dreadful news that the wicked men, whom Haley had sent to catch Eliza, were only a few miles away.
So George and Eliza decided to start as soon as it was dark.
On and on, all through the dark night they drove. About three o'clock, George heard the click of a horse's hoof coming behind them.
'That's Simeon,' said Phineas, who was driving, as he pulled up the horses to listen.
'Halloa, there, Simeon,' he shouted, 'what news? Are they coming?'
'Yes, right on behind, eight or ten of them.'
'Oh! what shall we do?' groaned Eliza.
But Phineas knew the road well. He lashed the horses till they flew along, the waggon rattling and jumping over the hard road behind them.
On they went till they came to a place where the rocks rose straight up from the road like a wall. It seemed impossible for any one to climb up there. But Phineas knew a way.
He stopped the horses. 'Here, Simeon,' he said, 'take the waggon, and drive on as fast as thou canst, and bring back help. Now follow me,' he said to the others, 'quick, for your lives. Run now, if you you ever did run.'
Quicker than we can say it, they were following him up a tiny narrow path to the top of the rocks, and Simeon was galloping the horses with the empty waggon along the road.
'We are pretty safe here,' said Phineas, when they had reached the top. 'Only one person can come up that path at a time. If any one tries it, shoot him.'
The men who were chasing them had now arrived at the foot of of the rocks.
After some hunting about, they found the path, and, headed by Tom Loker, began to climb up.

For answer, the little man took aim at George, and fired.
Eliza screamed, but the shot did not hurt him. It passed close to his hair, nearly touched her cheek, and struck a tree behind.
Tom Loker came on. George waited until he was near enough, then he fired. The shot hit him in the side. But, though wounded, he would not go back. With a yell like that of a mad bull he came leaping on, and sprang right in among them.
Quakers are not allowed to use guns and pistols, so Phineas had been standing back while George shot. Now he sprang forward. As Tom Loker landed in the middle of them, he gave him a great push, saying, 'Friend, thee isn't wanted here.'
Down fell Tom Loker, down, down the steep side of the rock. He crashed and crackled among trees, bushes, logs, loose stones, till he lay bruised and groaning far below. The fall might have killed him, had it not been broken by his clothes catching on the branches of a large tree.
Cruel people are, very often, cowardly too. When the men saw their leader first wounded, and then thrown down, they all ran away. Mounting their horses, they rode off as fast as they could, leaving Tom Loker lying on the ground wounded and groaning with pain.
As soon as Phineas and the others saw that the wicked men had really ridden away, they climbed down, meaning to walk along the road till they met Simeon.
They had just reached the bottom, when they saw him coming back with the waggon and two other men.
'Now we are safe,' cried Phineas joyfully:
'Well, do stop then,' said Eliza, 'and do something for that poor man. He is groaning dreadfully.'
'It would be no more than Christian,' said George. 'Let us take him with us.'
They lifted the wounded man gently, as if he had been a friend instead of a cruel enemy, and laid him in the waggon. Then they all set out once more.
A drive of about an hour brought them to a neat farm–house. There the tired travellers were kindly received and given a good breakfast.
Tom Loker was put into a comfortable bed, far cleaner and softer than any he had ever slept in before. George and Eliza walked about the garden hand–in–hand, feeling happy together, and almost safe. They were so near Canada now.


@(params: Map[String, Any]) @uk.gov.hmrc.hmrcemailrenderer.templates.helpers.html.template_main(params, params("service").toString + " update") {
Your client has requested you to act as their agent.
You can accept the client by going to your @{params("service").toString} online service and clicking the accept button
From HMRC
}


An exception is a rarely occurring (exceptional!) condition that requires deviation from the program's normal flow. Normally, an exception should not result in total failure, but instead be attended by an exception handler. Exception handling is a built-in construct in many programming languages. ...
-3
votes
2answers
48 views
how to add exception (),? [on hold]
I send two string arrays to compare between them , but i don't need check the parthenses(,) and gomma ? how to add the exception?
public static boolean CheckStatAppearinLeftAndRight1(String[] ...
0
votes
2answers
19 views
New class/file for each Exception?
for a new project I decided to renew my coding practices. The one thing that bothers me right now is the use of Exceptions. In the past I have used a custom class MyException which inherited the ...
0
votes
1answer
36 views
Where is this going wrong? Exception in thread “main” java.lang.ArrayIndexOutOfBoundsException: 5
I am trying to count the number of Space objects in a Space[][], of a particular colour. When I use this method to count the number of objects of a particular colour in a row, it works fine:
public ...
0
votes
2answers
23 views
IOException can not be resolved to a type error
For my final in Java we have a "exceptions" part on the test with try, catch, and finally calls. When I try to put the example code into Eclipse I get errors in the catch and throw new areas. All of ...
0
votes
1answer
25 views
Strange Breeze error when query DOES return data (update: Metadata not loading correctly)
I'm using a ODataController to return data to the breeze dataservice, this is the dataservice
app.dataservice = (function (breeze) {
breeze.config.initializeAdapterInstances({ dataService: ...
1
vote
2answers
38 views
Scala @throws multiple exceptions
I need to call scala code from java, so I need to tell the compiler that a certain method throws certain exceptions. This is easy to do for one exception, but I'm struggling to declare that a method ...
-2
votes
0answers
30 views
java exception when running a jar file
I got a code in jar file and added it to a new project to its build path.Then I tried to run it and got this exception:
Exception in thread "main" java.lang.UnsatisfiedLinkError: Could not load SWT ...
-1
votes
6answers
39 views
android.content.ActivityNotFoundException on calling new Activity in Android
I'm getting exception
12-12 17:31:18.401: E/AndroidRuntime(18177): FATAL EXCEPTION: main
12-12 17:31:18.401: E/AndroidRuntime(18177): android.content.ActivityNotFoundException: Unable to find ...
0
votes
0answers
16 views
Mule File endpoint, move sub-directories structure and have FileNotFoundException
I need to move sub-directories structure to output directory.
I use attribute recursive="true" for file connector.
Everything works fine, but periodically I get FileNotFoundException.
Something like ...
1
vote
7answers
49 views
Exception in thread “main” java.lang.NullPointerException - methods
I'm new to Java. Tried to do a simple library and I've encountered a problem.
"Exception in thread "main" java.lang.NullPointerException
at Book.showData(Book.java:22)
at ...
5
votes
8answers
104 views
What is the order of execution in try,catch and finally
If we give return statement like this in try, what will be the order of execution
try{
--- ----
-----
return a;
}
catch{
}
finally{
}
Here what will be order of execution if there is return in ...
1
vote
4answers
56 views
How to prevent MessageBox on Exception in C#? [on hold]
I'm trying to make a C# (Windows Forms) application which utilizes SQLite. Anyway, my code is this :
try
{
db.Insert("COLUMN_NAME", data);
}
catch (Exception e)
{
...
3
votes
1answer
83 views
Random “int is not subscriptable” behaviour
I'm reading a valid JSON file (nested 5 levels deep), then adding some data to it, and subsequently trying to use that data for some calculations.
I'm getting int is not subscriptable errors in a ...
1
vote
0answers
9 views
How do I get an unhandled exception to be reported in SML/NJ?
I have the following SML program in a file named testexc.sml:
structure TestExc : sig
val main : (string * string list -> int)
end =
struct
exception OhNoes;
...
1
vote
1answer
26 views
Segfault on deleting class member pointer to a stream
I'm trying to write a class for handling exceptions. I thought it would be useful to add a << operator to give me an easy way to throw error messages. Bellow is my solution to the problem so ...


- You need a Google account
- Push options -> open editor
- Make a copy of your own (File -> copy)
- Make it public to read (File -> Share)
- Copy the URL and save it into the form
- A read only view should now appear
- Share the URL in the chat and give edit access to all players, so everyone can participate in editing the map


8 Analytic Hierarchy Process
8.1 Defining weights through expert judgment
- How expert can come up with values for the weight?
- How to reach agreement among expert on the suggested values?
- Is there an alternative to lengthy consensus building?
Often teams of expert suffer from non-aligned goals, power politics, group dynamics and lack of mutual understanding.
How to build consensus on complex decisions in order to raise sufficient confidence in decision outcomes?
8.2 The Analytic Hierarchy Process (AHP)
- Structured technique for organizing and analyzing complex decisions, based on mathematics and psychology
- Comprehensive and rational framework to organize feeling, intuition & logic for structuring group decision making. Rather than prescribing a “correct” decision, the AHP helps decision makers find one that best suits their goal and their understanding of the problem
- Comparison is made to define priorities between 2 alternatives based on decision-maker’s feeling of priority dues to importance, preference and likelihood of influence. This breaks down complex decision into small judgement
- Developed in the 70’s by Thomas Saaty in Wharton Business School. Decision Making for Leaders: The Analytic Hierarchy Process for Decisions in a Complex World
8.3 Limitation of AHP
Assumes a minimum level of judgment consistency among experts
Limited number of criteria to minimize the number of pairwise comparison estimation
Data should be available for all criteria
Assumes that when new alternatives are added to a decision problem, the ranking of the old alternatives is not changing
8.4 How to? A step by step approach
- Define vulnerability criteria
- Build the expert consultation form
- Generate the AHP file from the collected data
- Run AHP algorithm
- Review results
- Apply the formula
8.5 Define vulnerability criteria
List potential criteria that would contribute to vulnerability within the current context. Note that criteria can be grouped togehter using hierarchy.
Establish treehold for criteria so that they can be formulated as simple binary questions
Criteria should be saved in a configuration file using the format here data/criteria.csv
.
8.5.1 Criteria without hierarchy
In this case N*(N-1)/2 pairs to review: 5 criteria -> 10 comparisons
| Criteria-Code | Criteria-Level-1-label | Criteria-Level-2-label |
|---|---|---|
| age | Age of head of household is above 50 | |
| gender | Gender of head of household is female | |
| size | Household size is above 5 | |
| needs | Occurrence of Specific needs | |
| assitance | Do not Receive assistance |
PS: 6 criteria -> 15 comparison, 7 criteria -> 24 comparisons, 8 criteria -> 28 comparisons, 9 criteria -> 36 comparisons
8.5.2 Criteria with hierarchy
In this case 6 pairs to review…
| Criteria-Code | Criteria-Level-1-label | Criteria-Level-2-label |
|---|---|---|
| age | Demography | Age of head of household is above 50 |
| gender | Demography | Gender of head of household is female |
| size | Demography | Household size is above 5 |
| needs | Occurrence of Specific needs | |
| assitance | Do not Receive assistance |
8.6 Build the expert consultation form
A form will allow to collect from expert priorities between criteria by making a series of judgments based on pairwise comparisons:
Collect judgment for all pairwise comparison and each expert
Ranking Scales for Criteria
Use the script 1-Build-xlsform.R
to build a xlsform file based on criteria defined above.
8.7 Form
The form can be used within UNHCR Kobo server. Experts can be humanitarian case workers that are used to assess vulnerability. See an example here
8.8 3. Generate the AHP file from the collected data
Once the selected experts (aka. decision-makers) have filled the online form , data can be exported from UNHCR Kobo server in csv format.
Use the script 2-build-hierarchy.R
to build a xlsform file to create the AHP file.
This create the file that format correctly the pairwise preferences of each decision-makers to run the next step.
8.9 4. Run AHP algorithm
- Calculation done using R statistical language.
- Synthesize these judgments to yield a set of overall priorities
- Check judgments consistency (consistency ratio)
- Compute weights for each expert
- Mathematical calculations to convert these judgments to priorities for each of the four criteria
Note: Some proprietary software options are also available but would require a Data Protection Impact Assessment before using them.
8.10 5. Review results
Knit the 3-final-report.Rmd
to get the report. You can see an example here.
An interactive interface is available to interact with results.
Lack of consistency is often observed
If consistency ratio is above 0.1, then judgement are untrustworthy because they are too close to randomness -> exercise needs to be repeated or abandonned.
8.11 6. Apply the formula
- Calculate mean relative weight
- Apply the vulnerability formula
- Collect data on each criteria for all household
- Apply weight to each record for the different criteria to get the vulnerability level of each household


THE BABY-SITTERS CLUB A film review by James Berardinelli Copyright 1995 James Berardinelli
RATING (0 TO 10): 4.9
United States, 1995 U.S. Availability: wide release 8/18/95 Running Length: 1:35 MPAA Classification: PG (Mature themes) Theatrical Aspect Ratio: 2.35:1
Cast: Schuyler Fisk, Rachael Leigh Cook, Larisa Oleynik, Bre Blair, Tricia Joe, Ellen Burstyn, Brooke Adams, Peter Horton Director: Melanie Mayron Producers: Jane Startz and Peter O. Almond Screenplay: Dalene Young based on the books by Ann M. Martin Cinematography: Wally Kurant Music: David Michael Frank U.S. Distributor: Columbia Pictures
The summer of 1995 has been a good season for so-called family films. A number of features aimed primarily at children have held their own with a more mature crowd. THE BABY-SITTERS CLUB, Melanie Mayron's movie based on the popular books by Ann M. Martin, is not a member of that category. Unlike THE LITTLE PRINCESS, POCAHONTAS, THE INDIAN IN THE CUPBOARD, and BABE, this particular motion picture is unlikely to grab the attention of anyone much past their pre-teen years.
The baby-sitters of Stoneybrook have banded together to form a club. Seven strong, they work like an agency, receiving calls from clients and handing out assignments to their select members. In the words of Kristy (impressive newcomer Schuyler Fisk), the club's founder, "We're more than just a club -- we're friends -- best friends. And nothing will ever change that." Of course, the minute she utters those words, everyone in the audience knows that something's going to happen this particular summer to try.
Character development is virtually non-existent, with the personalities of seven girls, three boys, and a few adults being painted with the broad strokes that define stereotypes. They're all here: the tomboy, the put-upon best friend, the dreamy California girl, and the worldly New York transplant. Each has at least one story all to herself. One's having problems with a boyfriend, another's forging a friendship with an older woman (played by Ellen Burstyn), and a third is trying to cope with the sudden return of her long-lost father (Timothy Horton). All-in-all, the months between June and August are very busy for the inhabitants of this film.
Ultimately, this looks and feels like a made-for-TV movie or an after school special. The direction by THIRTYSOMETHING cast member Mayron is unimaginative, and the script recycles all kinds of cliched situations and facile solutions. For the most part, THE BABY-SITTERS CLUB wants to teach its viewers the value of friendship, but the message comes across with sledgehammer subtlety in the midst of a sickeningly- sweet shower of goodwill. This is too much like a '90s version of THE BRADY BUNCH.
Some will argue that I'm being too hard on THE BABY-SITTERS CLUB. It is, after all, likely to work for the niche it's aimed at. On a more objective basis, though, the film doesn't succeed as a comedy, fantasy, or drama. Fans and would-be fans of Ann M. Martin's books may be enthralled, but, for the rest of us, THE BABY-SITTERS CLUB is something of an endurance contest.
- James Berardinelli<EMAIL_ADDRESS>The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Registration Begins
Opening Ceremony
Hacking Starts
Lunch
Fun Event
Snack Break
Dinner
Breakfast
Evaluation Begins
Hacking Ends
Announcement of Shortlisted Teams
Final Presentation by Shortlisted Teams
Prize Distribution


Scaloni pleased with Argentina progress as unbeaten run continues
by BeSoccerWhile seeking improvement, Lionel Scaloni is happy with Argentina's development after their unbeaten run continued.
Lionel Scaloni is satisfied with Argentina's progress after a friendly draw against Uruguay on Monday.
Argentina came from behind twice to secure a 2-2 draw in Tel Aviv, where Lionel Messi's stoppage-time penalty cancelled out Luis Suarez's goal after Edinson Cavani and Sergio Aguero had struck earlier.
It extended Argentina's unbeaten run to seven matches as they continued preparations for 2022 World Cup qualifying, which begins next year.
Scaloni is pleased with Argentina's development, although the coach still sees room for improvement.
"At the level of results, we are fine. At the level of the game, it can be improved," he said, via Ole.
Featured links
- Ballon d'Or winner already been decided
- The latest football transfers news and rumours
- Possible 'Clasico' in the Spanish Supercup final
- Who has the most free-kicks goals: Messi or Cristiano?
"The team feeling is the best we have. The important thing is that the players who play do it well. Some make a difference and others support."
Scaloni acknowledged the challenge awaiting his team, describing the CONMEBOL qualifiers as the toughest in the world.
"Now we have to think about what we did well and what we can correct. What is coming is difficult," he said.
"I spoke with people from Uruguay. It will be difficult in the qualifiers. Those in South America are the most difficult in the world.
"We have a good base."


Welcome All..
Interpolation
{{name}}
{{name == "BManisha" ? 'Hola' : 'Chao'}}
Directives
{{name}}
{{sampleHtml}}
My Profile
Two-Way Binding
Apple
banana
someX
someY
You chocen : {{choice}}
Conditions
This is True
This is False
I'm here tooo
Loops
{{item}}
{{index}} - {{todo.id}} - {{todo.title}}
Styles
My Profile
This is in Red color
This is in {{myColor}}
This is a {{statustext}} Message
This is an example
This is an example


About
I’m Muomila Rangar
For over 12 years I have been successfully providing business information to poeple.
I’m an IT University student, I like to create simple & standard UI/UX design and development. It’s my profession to create client's projects that make a smile after done.
-
Name: Muomila Rangar
-
Location: CS-124, NY, NewYork, USA
-
Mail<EMAIL_ADDRESS>-
Date of Birth: 1996 & 29 Years
-
Degree: Master of Computer
-
Phone: +997 4638 255


SchoolGuru
Add Assignment
About
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Enter the assignment information:
Assignment:
Due Date:
Estimated Total Time (in hours):
How many days in advance will you start working on this assignment?:
Enter


Perhaps someone could help me here, I am looking for a build that can go toe to toe with a High level PM. I have tired a fighter/Sor/RDD. Also tried a fighter/wm/cot and in PvP, I was destroyed. I have look over the builds posted, as well as check Nwikki, to gain a understanding, but i am lost. Any help would be appreciated.
High lvl Mage, TS+IGMS spamming. Nuff said. _________________ They can't stop us Let them try For Heavy Metal We will die
As Kail said, use a mage, you won't care about AC when you use spells, so a caster is definitely the best, but if you want it the hard way, there might be a posibility.
What happens is his AC is probably amazingly high isn't it? for high ACs you need to go Arcane Archer, if he's a good player he might also use Epic Warding which makes it even worse, you won't deal damage even if you hit... you might be able to hit and deal damage through with a Paladin / Wizard / Arcane Archer, buffing with Divine Might and True Strike. If that doesn't works then just use a caster, there's no way around of it, if AA can't hit him and damage him, probably nothing can. _________________ "My name is Thaxll'ssyllia, but you simians may refer to me merely as "Sir", if you prefer a less... syllable intensive workout."
Edwin Odesseiron - Baldur's Gate II: Shadows of Amn
Edited By Thaxll'ssyllia on 09/29/07 03:22
Touch attacks will do too, all that Natural AC will go down the flush. Implosion is a good solution too.
But to give better advice, what PM heavy build are we talking about exactly?


This page is for modifying credentials for the selected cluster.
Once the changes are submitted, credentials will be modified for all the iLO's in this cluster.


Where Have You Been?
Neumos
Chop Suey
High Line
Barboza
The Black Lodge
Neptune
Cafe Racer
The Kraken Lounge
Fremont Abbey
The High Dive
Nectar
Substation
The Tractor Tavern
The Sunset
Egan's Ballard Jam House
The Josephine
Ballard Homestead
The Crocodile
El Corazon
The Vera Project
The Triple Door
Showbox at the Market
Showbox SODO
Paramount Theatre
The Moore
Your Favorite Neighborhood Is...
You Should Explore...
Neighborhoods
Fremont
Ballard
University District
Capitol Hill
Downtown
Here's a recent show at
Here
is a link to their calendar
Jack Fehrenbacher made this


{% extends "global/BaseMotherhood.html" %} {% load staticfiles otree_tags %} {% block title %} Instructions {% endblock %} {% block content %} {% load otree_tags staticfiles %}
In this part of the experiment you will be presented with a series of randomly selected letters and numbers.
You will have {{ player.task_timer}} seconds to type as many sequences as possible. Note that each character has to be correct.
Once you have entered your summation, click the Next button or press the Enter key.
You will get 1 point for each correct answer. Your score will not decrease if you provide an incorrect answer.
After {{ player.task_timer}} seconds, the task will stop and you will receive a summary of the results.
Please raise your hand and alert an experimenter if you have any questions.
Otherwise, if you are ready, please click "Next" below to start the time and this {{ player.task_timer}} second task.


S3Guard is a feature for the S3A client of the S3 object store, which can use a (consistent) database as the store of metadata about objects in an S3 bucket.
S3Guard
May improve performance on directory listing/scanning operations, including those which take place during the partitioning period of query execution, the process where files are listed and the work divided up amongst processes.
Permits a consistent view of the object store. Without this, changes in objects may not be immediately visible, especially in listing operations.
Offers a platform for future performance improvements for running Hadoop workloads on top of object stores
The basic idea is that, for each operation in the Hadoop S3 client (s3a) that reads or modifies metadata, a shadow copy of that metadata is stored in a separate MetadataStore implementation. Each MetadataStore implementation offers HDFS-like consistency for the metadata, and may also provide faster lookups for things like file status or directory listings.
For links to early design documents and related patches, see HADOOP-13345.
Important
A seemingly recurrent problem with S3Guard is that people think S3Guard is turned on but it isn’t. You can set org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level to avoid this. The property sets what to do when an S3A FS is instantiated without S3Guard. The following values are available:
The default setting is INFORM. The setting is case insensitive. The required level can be set in the core-site.xml.
The latest configuration parameters are defined in core-default.xml. You should consult that file for full information, but a summary is provided here.
A core concept of S3Guard is that the directory listing data of the object store, the metadata is replicated in a higher-performance, consistent, database. In S3Guard, this database is called The Metadata Store
By default, S3Guard is not enabled.
The Metadata Store to use in production is bonded to Amazon’s DynamoDB database service. The following setting will enable this Metadata Store:
<property> <name>fs.s3a.metadatastore.impl</name> <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value> </property>
Note that the NullMetadataStore store can be explicitly requested if desired. This offers no metadata storage, and effectively disables S3Guard.
<property> <name>fs.s3a.metadatastore.impl</name> <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value> </property>
More settings will may be added in the future. Currently the only Metadata Store-independent setting, besides the implementation class above, are the allow authoritative and fail-on-error flags.
The authoritative expression in S3Guard is present in two different layers, for two different reasons:
Authoritative S3Guard
Authoritative directory listings (isAuthoritative bit)
More on Authoritative S3Guard:
This configuration only enables authoritative mode in the client layer. It is recommended that you leave the default setting here:
<property> <name>fs.s3a.metadatastore.authoritative</name> <value>false</value> </property>
. Note that a MetadataStore MAY persist this bit in the directory listings. (Not MUST).
Note that if this is set to true, it may exacerbate or persist existing race conditions around multiple concurrent modifications and listings of a given directory tree.
In particular: If the Metadata Store is declared as authoritative, all interactions with the S3 bucket(s) must be through S3A clients sharing the same Metadata Store
It can be configured how long an entry is valid in the MetadataStore if the authoritative mode is turned off, or the path is not configured to be authoritative. If ((lastUpdated + ttl) <= now) is false for an entry, the entry will be expired, so the S3 bucket will be queried for fresh metadata. The time for expiry of metadata can be set as the following:
<property> <name>fs.s3a.metadatastore.metadata.ttl</name> <value>15m</value> </property>
By default, S3AFileSystem write operations will fail when updates to S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then updates the metadata in S3Guard. If the metadata write fails, MetadataPersistenceException is thrown. The file in S3 is not rolled back.
If the write operation cannot be programmatically retried, the S3Guard metadata for the given file can be corrected with a command like the following:
hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
Programmatic retries of the original operation would require overwrite=true. Suppose the original operation was FileSystem.create(myFile, overwrite=false). If this operation failed with MetadataPersistenceException a repeat of the same operation would result in FileAlreadyExistsException since the original operation successfully created the file in S3 and only failed in writing the metadata to S3Guard.
Metadata update failures can be downgraded to ERROR logging instead of exception by setting the following configuration:
<property> <name>fs.s3a.metadatastore.fail.on.write.error</name> <value>false</value> </property>
Setting this false is dangerous as it could result in the type of issue S3Guard is designed to avoid. For example, a reader may see an inconsistent listing after a recent write since S3Guard may not contain metadata about the recently written file due to a metadata write error.
As with the default setting, the new/updated file is still in S3 and is not rolled back. The S3Guard metadata is likely to be out of sync.
Here are the DynamoDBMetadataStore settings. Other Metadata Store implementations will have their own configuration parameters.
First, choose the name of the table you wish to use for the S3Guard metadata storage in your DynamoDB instance. If you leave it unset/empty, a separate table will be created for each S3 bucket you access, and that bucket’s name will be used for the name of the DynamoDB table. For example, this sets the table name to my-ddb-table-name
<property> <name>fs.s3a.s3guard.ddb.table</name> <value>my-ddb-table-name</value> <description> The DynamoDB table name to operate. Without this property, the respective S3 bucket names will be used. </description> </property>
It is good to share a table across multiple buckets for multiple reasons, especially if you are not using on-demand DynamoDB tables, and instead prepaying for provisioned I/O capacity.
You are billed for the provisioned I/O capacity allocated to the table, even when the table is not used. Sharing capacity can reduce costs.
You can share the “provision burden” across the buckets. That is, rather than allocating for the peak load on a single bucket, you can allocate for the peak load across all the buckets, which is likely to be significantly lower.
It’s easier to measure and tune the load requirements and cost of S3Guard, because there is only one table to review and configure in the AWS management console.
When you don’t grant the permission to create DynamoDB tables to users. A single pre-created table for all buckets avoids the needs for an administrator to create one for every bucket.
When wouldn’t you want to share a table?
When you are using separate billing for specific buckets allocated to specific projects.
When different users/roles have different access rights to different buckets. As S3Guard requires all users to have R/W access to the table, all users will be able to list the metadata in all buckets, even those to which they lack read access.
You may also wish to specify the region to use for DynamoDB. If a region is not configured, S3A will assume that it is in the same region as the S3 bucket. A list of regions for the DynamoDB service can be found in Amazon’s documentation. In this example, to use the US West 2 region:
<property> <name>fs.s3a.s3guard.ddb.region</name> <value>us-west-2</value> </property>
When working with S3Guard-managed buckets from EC2 VMs running in AWS infrastructure, using a local DynamoDB region ensures the lowest latency and highest reliability, as well as avoiding all long-haul network charges. The S3Guard tables, and indeed, the S3 buckets, should all be in the same region as the VMs.
Next, you can choose whether or not the table will be automatically created (if it doesn’t already exist). If you want this feature, set the fs.s3a.s3guard.ddb.table.create option to true.
<property> <name>fs.s3a.s3guard.ddb.table.create</name> <value>true</value> <description> If true, the S3A client will create the table if it does not already exist. </description> </property>
Next, you need to decide whether to use On-Demand DynamoDB and its pay-per-request billing (recommended), or to explicitly request a provisioned IO capacity.
Before AWS offered pay-per-request billing, the sole billing mechanism, was “provisioned capacity”. This mechanism requires you to choose the DynamoDB read and write throughput requirements you expect to need for your expected uses of the S3Guard table. Setting higher values cost you more money -even when the table was idle Note that these settings only affect table creation when fs.s3a.s3guard.ddb.table.create is enabled. To change the throughput for an existing table, use the AWS console or CLI tool.
For more details on DynamoDB capacity units, see the AWS page on Capacity Unit Calculations.
Provisioned IO capacity is billed per hour for the life of the table, even when the table and the underlying S3 buckets are not being used.
There are also charges incurred for data storage and for data I/O outside of the region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB: path names and summary details of objects —the actual data is stored in S3, so billed at S3 rates.
With provisioned I/O capacity, attempting to perform more I/O than the capacity requested throttles the operation and may result in operations failing. Larger I/O capacities cost more.
With the introduction of On-Demand DynamoDB, you can now avoid paying for provisioned capacity by creating an on-demand table. With an on-demand table you are not throttled if your DynamoDB requests exceed any pre-provisioned limit, nor do you pay per hour even when a table is idle.
You do, however, pay more per DynamoDB operation. Even so, the ability to cope with sudden bursts of read or write requests, combined with the elimination of charges for idle tables, suit the use patterns made of S3Guard tables by applications interacting with S3. That is: periods when the table is rarely used, with intermittent high-load operations when directory trees are scanned (query planning and similar), or updated (rename and delete operations).
We recommending using On-Demand DynamoDB for maximum performance in operations such as query planning, and lowest cost when S3 buckets are not being accessed.
This is the default, as configured in the default configuration options.
<property> <name>fs.s3a.s3guard.ddb.table.capacity.read</name> <value>0</value> <description> Provisioned throughput requirements for read operations in terms of capacity units for the DynamoDB table. This config value will only be used when creating a new DynamoDB table. If set to 0 (the default), new tables are created with "per-request" capacity. If a positive integer is provided for this and the write capacity, then a table with "provisioned capacity" will be created. You can change the capacity of an existing provisioned-capacity table through the "s3guard set-capacity" command. </description> </property> <property> <name>fs.s3a.s3guard.ddb.table.capacity.write</name> <value>0</value> <description> Provisioned throughput requirements for write operations in terms of capacity units for the DynamoDB table. If set to 0 (the default), new tables are created with "per-request" capacity. Refer to related configuration option fs.s3a.s3guard.ddb.table.capacity.read </description> </property>
Encryption at rest can help you protect sensitive data in your DynamoDB table. When creating a new table, you can set server side encryption on the table using the default AWS owned customer master key (CMK), AWS managed CMK, or customer managed CMK. S3Guard code accessing the table is all the same whether SSE is enabled or not. For more details on DynamoDB table server side encryption, see the AWS page on Encryption at Rest: How It Works.
These are the default configuration options, as configured in core-default.xml.
<property> <name>fs.s3a.s3guard.ddb.table.sse.enabled</name> <value>false</value> <description> Whether server-side encryption (SSE) is enabled or disabled on the table. By default it's disabled, meaning SSE is set to AWS owned CMK. </description> </property> <property> <name>fs.s3a.s3guard.ddb.table.sse.cmk</name> <value/> <description> The KMS Customer Master Key (CMK) used for the KMS encryption on the table. To specify a CMK, this config value can be its key ID, Amazon Resource Name (ARN), alias name, or alias ARN. Users only need to provide this config if the key is different from the default DynamoDB KMS Master Key, which is alias/aws/dynamodb. </description> </property>
The DynamoDB metadata store takes advantage of the fact that the DynamoDB service uses the same authentication mechanisms as S3. S3Guard gets all its credentials from the S3A client that is using it.
All existing S3 authentication mechanisms can be used.
In production, it is likely only some buckets will have S3Guard enabled; those which are read-only may have disabled, for example. Equally importantly, buckets in different regions should have different tables, each in the relevant region.
 All options with the under fs.s3a.bucket.BUCKETNAME.KEY are propagated to the options fs.s3a.KEY for that bucket only.
As an example, here is a configuration to use different metadata stores and tables for different buckets
First, we define shortcuts for the metadata store classnames:
<property> <name>s3guard.null</name> <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value> </property> <property> <name>s3guard.dynamo</name> <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value> </property>
Next, Amazon’s public landsat database is configured with no metadata store:
<property> <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name> <value>${s3guard.null}</value> <description>The read-only landsat-pds repository isn't managed by S3Guard</description> </property>
Next the ireland-2 and ireland-offline buckets are configured with DynamoDB as the store, and a shared table production-table:
<property> <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name> <value>${s3guard.dynamo}</value> </property> <property> <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name> <value>${s3guard.dynamo}</value> </property> <property> <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name> <value>production-table</value> </property>
The region of this table is automatically set to be that of the buckets, here eu-west-1; the same table name may actually be used in different regions.
Together then, this configuration enables the DynamoDB Metadata Store for two buckets with a shared table, while disabling it for the public bucket.
We call an operation out-of-band (OOB) when a bucket is used by a client with S3Guard, and another client runs a write (e.g delete, move, rename, overwrite) operation on an object in the same bucket without S3Guard.
The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs: * A client with S3Guard * B client without S3Guard (Directly to S3)
OOB OVERWRITE, NOT authoritative mode:
OOB DELETE, authoritative mode:
OOB DELETE, NOT authoritative mode:
Note: authoritative and NOT authoritative mode behaves the same at OOB DELETE case.
The behaviour in case of getting directory listings: * File status in metadata store gets updated during the listing the same way as in getFileStatus.
Note that in some cases an AWS region or s3a:// URI can be provided.
Metadata store URIs include a scheme that designates the backing store. For example (e.g. dynamodb://table_name;). As documented above, the AWS region can be inferred if the URI to an existing bucket is provided.
The S3A URI must also be provided for per-bucket configuration options to be picked up. That is: when an s3a URL is provided on the command line, all its “resolved” per-bucket settings are used to connect to, authenticate with and configure the S3Guard table. If no such URL is provided, then the base settings are picked up.
hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
Creates and initializes an empty metadata store.
A DynamoDB metadata store can be initialized with additional parameters pertaining to capacity.
If these values are both zero, then an on-demand DynamoDB table is created; if positive values then they set the Provisioned Throughput of the table.
[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
Server side encryption (SSE) can be enabled with AWS managed customer master key (CMK), or customer managed CMK. By default the DynamoDB table will be encrypted with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed CMK for DynamoDB “alias/aws/dynamodb” will be used.
[-sse [-cmk KMS_CMK_ID]]
Tag argument can be added with a key=value list of tags. The table for the metadata store will be created with these tags in DynamoDB.
[-tag key=value;]
Example 1
hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0 s3a://ireland-1
Creates an on-demand table “ireland-team”, in the same location as the S3 bucket “ireland-1”.
Example 2
hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0 --write 0
Creates a table “ireland-team” in the region “eu-west-1.amazonaws.com”
Example 3
hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
Creates a table “ireland-team” with tags “first” and “second”. The read and write capacity will be those of the site configuration’s values of fs.s3a.s3guard.ddb.table.capacity.read and fs.s3a.s3guard.ddb.table.capacity.write; if these are both zero then it will be an on-demand table.
Example 4
hadoop s3guard init -meta dynamodb://ireland-team -sse
Creates a table “ireland-team” with server side encryption enabled. The CMK will be using the default AWS managed “alias/aws/dynamodb”.
hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
Pre-populates a metadata store according to the current contents of an S3 bucket/path. If the -meta option is omitted, the binding information is taken from the core-site.xml configuration.
Usage
hadoop s3guard import import [OPTIONS] [s3a://PATH] import metadata from existing S3 data Common options: -authoritative - Mark imported directory data as authoritative. -verbose - Verbose Output. -meta URL - Metadata repository details (implementation-specific) Amazon DynamoDB-specific options: -region REGION - Service region for connections URLs for Amazon DynamoDB are of the form dynamodb://TABLE_NAME. Specifying both the -region option and an S3A path is not supported.
Example
Import all files and directories in a bucket into the S3Guard table.
hadoop s3guard import s3a://ireland-1
Import a directory tree, marking directories as authoritative.
hadoop s3guard import -authoritative -verbose s3a://ireland-1/fork-0008 2020-01-03 12:05:18,321 [main] INFO - Metadata store DynamoDBMetadataStore{region=eu-west-1, tableName=s3guard-metadata, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/s3guard-metadata} is initialized. 2020-01-03 12:05:18,324 [main] INFO - Starting: Importing s3a://ireland-1/fork-0008 2020-01-03 12:05:18,324 [main] INFO - Importing directory s3a://ireland-1/fork-0008 2020-01-03 12:05:18,537 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-false 2020-01-03 12:05:18,630 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-true 2020-01-03 12:05:19,142 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-0 2020-01-03 12:05:19,191 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-1 2020-01-03 12:05:19,240 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-0 2020-01-03 12:05:19,289 [main] INFO - Dir s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-1 2020-01-03 12:05:19,314 [main] INFO - Updated S3Guard with 0 files and 6 directory entries 2020-01-03 12:05:19,315 [main] INFO - Marking directory tree s3a://ireland-1/fork-0008 as authoritative 2020-01-03 12:05:19,342 [main] INFO - Importing s3a://ireland-1/fork-0008: duration 0:01.018s Inserted 6 items into Metadata Store
hadoop s3guard diff [-meta URI] s3a://BUCKET
Lists discrepancies between a metadata store and bucket. Note that depending on how S3Guard is used, certain discrepancies are to be expected.
Example
hadoop s3guard diff s3a://ireland-1
Prints and optionally checks the s3guard and encryption status of a bucket.
hadoop s3guard bucket-info [-guarded] [-unguarded] [-auth] [-nonauth] [-magic] [-encryption ENCRYPTION] s3a://BUCKET
Options
| argument | meaning |
|---|---|
| -guarded | Require S3Guard to be enabled |
| -unguarded | Force S3Guard to be disabled |
| -auth | Require the S3Guard mode to be “authoritative” |
| -nonauth | Require the S3Guard mode to be “non-authoritative” |
| -magic | Require the S3 filesystem to be support the “magic” committer |
| -encryption <type> | Require a specific server-side encryption algorithm |
The server side encryption options are not directly related to S3Guard, but it is often convenient to check them at the same time.
Example
hadoop s3guard bucket-info -guarded -magic s3a://ireland-1
List the details of bucket s3a://ireland-1, mandating that it must have S3Guard enabled (“-guarded”) and that support for the magic S3A committer is enabled (“-magic”)
Filesystem s3a://ireland-1 Location: eu-west-1 Filesystem s3a://ireland-1 is using S3Guard with store DynamoDBMetadataStore{region=eu-west-1, tableName=ireland-1} Authoritative S3Guard: fs.s3a.metadatastore.authoritative=false Metadata Store Diagnostics: ARN=arn:aws:dynamodb:eu-west-1:00000000:table/ireland-1 billing-mode=provisioned description=S3Guard metadata store in DynamoDB name=ireland-1 read-capacity=20 region=eu-west-1 retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS) size=12812 status=ACTIVE table={AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: ireland-1, KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}], TableStatus: ACTIVE, CreationDateTime: Fri Aug 25 19:07:25 BST 2017, ProvisionedThroughput: {LastIncreaseDateTime: Tue Aug 29 11:45:18 BST 2017, LastDecreaseDateTime: Wed Aug 30 15:37:51 BST 2017, NumberOfDecreasesToday: 1, ReadCapacityUnits: 20,WriteCapacityUnits: 20}, TableSizeBytes: 12812,ItemCount: 91, TableArn: arn:aws:dynamodb:eu-west-1:00000000:table/ireland-1,} write-capacity=20 The "magic" committer is supported S3A Client Signing Algorithm: fs.s3a.signing-algorithm=(unset) Endpoint: fs.s3a.endpoint=s3-eu-west-1.amazonaws.com Encryption: fs.s3a.server-side-encryption-algorithm=none Input seek policy: fs.s3a.experimental.input.fadvise=normal Change Detection Source: fs.s3a.change.detection.source=etag Change Detection Mode: fs.s3a.change.detection.mode=server Delegation token support is disabled
This listing includes all the information about the table supplied from
hadoop s3guard bucket-info -unguarded -encryption none s3a://landsat-pds
List the S3Guard status of clients of the public landsat-pds bucket, and verifies that the data is neither tracked with S3Guard nor encrypted.
Filesystem s3a://landsat-pdsLocation: us-west-2 Filesystem s3a://landsat-pds is not using S3Guard Endpoint: fs.s3a.endpoints3.amazonaws.com Encryption: fs.s3a.server-side-encryption-algorithm=none Input seek policy: fs.s3a.experimental.input.fadvise=normal
Note that other clients may have a S3Guard table set up to store metadata on this bucket; the checks are all done from the perspective of the configuration settings of the current client.
hadoop s3guard bucket-info -guarded -auth s3a://landsat-pds
Require the bucket to be using S3Guard in authoritative mode. This will normally fail against this specific bucket.
Lists or deletes all pending (uncompleted) multipart uploads older than given age.
hadoop s3guard uploads (-list | -abort | -expect <num-uploads>) [-verbose] \ [-days <days>] [-hours <hours>] [-minutes <minutes>] [-seconds <seconds>] \ [-force] s3a://bucket/prefix
The command lists or deletes all multipart uploads which are older than the given age, and that match the prefix supplied, if any.
For example, to delete all uncompleted multipart uploads older than two days in the folder at s3a://my-bucket/path/to/stuff, use the following command:
hadoop s3guard uploads -abort -days 2 s3a://my-bucket/path/to/stuff
We recommend running with -list first to confirm the parts shown are those that you wish to delete. Note that the command will prompt you with a “Are you sure?” prompt unless you specify the -force option. This is to safeguard against accidental deletion of data, which is especially risky without a long age parameter as it can affect in-fight uploads.
The -expect option is similar to -list, except it is silent by default, and terminates with a success or failure exit code depending on whether or not the supplied number matches the number of uploads found that match the given options (path, age).
Deletes a metadata store. With DynamoDB as the store, this means the specific DynamoDB table use to store the metadata.
hadoop s3guard destroy [-meta URI] ( -region REGION | s3a://BUCKET )
This does not delete the bucket, only the S3Guard table which it is bound to.
Examples
hadoop s3guard destroy s3a://ireland-1
Deletes the table which the bucket ireland-1 is configured to use as its MetadataStore.
hadoop s3guard destroy -meta dynamodb://ireland-team -region eu-west-1
Delete all file entries in the MetadataStore table whose object “modification time” is older than the specified age.
hadoop s3guard prune [-days DAYS] [-hours HOURS] [-minutes MINUTES] [-seconds SECONDS] [-tombstone] [-meta URI] ( -region REGION | s3a://BUCKET )
A time value of hours, minutes and/or seconds must be supplied.
The -tombstone option instructs the operation to only purge “tombstones”, markers of deleted files. These tombstone markers are only used briefly, to indicate that a recently deleted file should not be found in listings. As a result, there is no adverse consequences in regularly pruning old tombstones.
Example
hadoop s3guard prune -days 7 s3a://ireland-1
Deletes all entries in the S3Guard table for files older than seven days from the table associated with s3a://ireland-1.
hadoop s3guard prune -tombstone -days 7 s3a://ireland-1/path_prefix/
Deletes all entries in the S3Guard table for tombstones older than seven days from the table associated with s3a://ireland-1 and with the prefix path_prefix
hadoop s3guard prune -hours 1 -minutes 30 -meta dynamodb://ireland-team -region eu-west-1
Delete all file entries more than 90 minutes old from the table "ireland-team" in the region eu-west-1.
This recursively checks a S3Guard table to verify that all directories underneath are marked as “authoritative”, and/or that the configuration is set for the S3A client to treat files and directories urnder the path as authoritative.
hadoop s3guard authoritative authoritative [OPTIONS] [s3a://PATH] Audits a DynamoDB S3Guard repository for all the entries being 'authoritative' Options: -required Require directories under the path to be authoritative. -check-config Check the configuration for the path to be authoritative -verbose Verbose Output.
Verify that a path under an object store is declared to be authoritative in the cluster configuration -and therefore that file entries will not be validated against S3, and that directories marked as “authoritative” in the S3Guard table will be treated as complete.
hadoop s3guard authoritative -check-config s3a:///ireland-1/fork-0003/test/ 2020-01-03 11:42:29,147 [main] INFO Metadata store DynamoDBMetadataStore{ region=eu-west-1, tableName=s3guard-metadata, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/s3guard-metadata} is initialized. Path /fork-0003/test is not configured to be authoritative
Scan a store and report which directories are not marked as authoritative.
hadoop s3guard authoritative s3a://ireland-1/ 2020-01-03 11:51:58,416 [main] INFO - Metadata store DynamoDBMetadataStore{region=eu-west-1, tableName=s3guard-metadata, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/s3guard-metadata} is initialized. 2020-01-03 11:51:58,419 [main] INFO - Starting: audit s3a://ireland-1/ 2020-01-03 11:51:58,422 [main] INFO - Root directory s3a://ireland-1/ 2020-01-03 11:51:58,469 [main] INFO - files 4; directories 12 2020-01-03 11:51:58,469 [main] INFO - Directory s3a://ireland-1/Users 2020-01-03 11:51:58,521 [main] INFO - files 0; directories 1 2020-01-03 11:51:58,522 [main] INFO - Directory s3a://ireland-1/fork-0007 2020-01-03 11:51:58,573 [main] INFO - Directory s3a://ireland-1/fork-0001 2020-01-03 11:51:58,626 [main] INFO - files 0; directories 1 2020-01-03 11:51:58,626 [main] INFO - Directory s3a://ireland-1/fork-0006 2020-01-03 11:51:58,676 [main] INFO - Directory s3a://ireland-1/path 2020-01-03 11:51:58,734 [main] INFO - files 0; directories 1 2020-01-03 11:51:58,735 [main] INFO - Directory s3a://ireland-1/fork-0008 2020-01-03 11:51:58,802 [main] INFO - files 0; directories 1 2020-01-03 11:51:58,802 [main] INFO - Directory s3a://ireland-1/fork-0004 2020-01-03 11:51:58,854 [main] INFO - files 0; directories 1 2020-01-03 11:51:58,855 [main] WARN - Directory s3a://ireland-1/fork-0003 is not authoritative 2020-01-03 11:51:58,905 [main] INFO - files 0; directories 1 2020-01-03 11:51:58,906 [main] INFO - Directory s3a://ireland-1/fork-0005 2020-01-03 11:51:58,955 [main] INFO - Directory s3a://ireland-1/customsignerpath2 2020-01-03 11:51:59,006 [main] INFO - Directory s3a://ireland-1/fork-0002 2020-01-03 11:51:59,063 [main] INFO - files 0; directories 1 2020-01-03 11:51:59,064 [main] INFO - Directory s3a://ireland-1/customsignerpath1 2020-01-03 11:51:59,121 [main] INFO - Directory s3a://ireland-1/Users/stevel 2020-01-03 11:51:59,170 [main] INFO - files 0; directories 1 2020-01-03 11:51:59,171 [main] INFO - Directory s3a://ireland-1/fork-0001/test 2020-01-03 11:51:59,233 [main] INFO - Directory s3a://ireland-1/path/style 2020-01-03 11:51:59,282 [main] INFO - files 0; directories 1 2020-01-03 11:51:59,282 [main] INFO - Directory s3a://ireland-1/fork-0008/test 2020-01-03 11:51:59,338 [main] INFO - files 15; directories 10 2020-01-03 11:51:59,339 [main] INFO - Directory s3a://ireland-1/fork-0004/test 2020-01-03 11:51:59,394 [main] WARN - Directory s3a://ireland-1/fork-0003/test is not authoritative 2020-01-03 11:51:59,451 [main] INFO - files 35; directories 1 2020-01-03 11:51:59,451 [main] INFO - Directory s3a://ireland-1/fork-0002/test 2020-01-03 11:51:59,508 [main] INFO - Directory s3a://ireland-1/Users/stevel/Projects 2020-01-03 11:51:59,558 [main] INFO - files 0; directories 1 2020-01-03 11:51:59,559 [main] INFO - Directory s3a://ireland-1/path/style/access 2020-01-03 11:51:59,610 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-0-2-0-false 2020-01-03 11:51:59,660 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-0-2-1-false 2020-01-03 11:51:59,719 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-true 2020-01-03 11:51:59,773 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true 2020-01-03 11:51:59,824 [main] INFO - files 0; directories 2 2020-01-03 11:51:59,824 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-0-2-1-true 2020-01-03 11:51:59,879 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-1-false 2020-01-03 11:51:59,939 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-false 2020-01-03 11:51:59,990 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-0-2-0-true 2020-01-03 11:52:00,042 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false 2020-01-03 11:52:00,094 [main] INFO - files 0; directories 2 2020-01-03 11:52:00,094 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-1-true 2020-01-03 11:52:00,144 [main] WARN - Directory s3a://ireland-1/fork-0003/test/ancestor is not authoritative 2020-01-03 11:52:00,197 [main] INFO - Directory s3a://ireland-1/Users/stevel/Projects/hadoop-trunk 2020-01-03 11:52:00,245 [main] INFO - files 0; directories 1 2020-01-03 11:52:00,245 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-0 2020-01-03 11:52:00,296 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-1 2020-01-03 11:52:00,346 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-0 2020-01-03 11:52:00,397 [main] INFO - Directory s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-1 2020-01-03 11:52:00,479 [main] INFO - Directory s3a://ireland-1/Users/stevel/Projects/hadoop-trunk/hadoop-tools 2020-01-03 11:52:00,530 [main] INFO - files 0; directories 1 2020-01-03 11:52:00,530 [main] INFO - Directory s3a://ireland-1/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws 2020-01-03 11:52:00,582 [main] INFO - files 0; directories 1 2020-01-03 11:52:00,582 [main] INFO - Directory s3a://ireland-1/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target 2020-01-03 11:52:00,636 [main] INFO - files 0; directories 1 2020-01-03 11:52:00,637 [main] INFO - Directory s3a://ireland-1/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir 2020-01-03 11:52:00,691 [main] INFO - files 0; directories 3 2020-01-03 11:52:00,691 [main] INFO - Directory s3a://ireland-1/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/2 2020-01-03 11:52:00,752 [main] INFO - Directory s3a://ireland-1/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/5 2020-01-03 11:52:00,807 [main] INFO - Directory s3a://ireland-1/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/8 2020-01-03 11:52:00,862 [main] INFO - Scanned 45 directories - 3 were not marked as authoritative 2020-01-03 11:52:00,863 [main] INFO - audit s3a://ireland-1/: duration 0:02.444s
Scan the path/bucket and fail if any entry is non-authoritative.
hadoop s3guard authoritative -verbose -required s3a://ireland-1/ 2020-01-03 11:47:40,288 [main] INFO - Metadata store DynamoDBMetadataStore{region=eu-west-1, tableName=s3guard-metadata, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/s3guard-metadata} is initialized. 2020-01-03 11:47:40,291 [main] INFO - Starting: audit s3a://ireland-1/ 2020-01-03 11:47:40,295 [main] INFO - Root directory s3a://ireland-1/ 2020-01-03 11:47:40,336 [main] INFO - files 4; directories 12 2020-01-03 11:47:40,336 [main] INFO - Directory s3a://ireland-1/Users 2020-01-03 11:47:40,386 [main] INFO - files 0; directories 1 2020-01-03 11:47:40,386 [main] INFO - Directory s3a://ireland-1/fork-0007 2020-01-03 11:47:40,435 [main] INFO - files 1; directories 0 2020-01-03 11:47:40,435 [main] INFO - Directory s3a://ireland-1/fork-0001 2020-01-03 11:47:40,486 [main] INFO - files 0; directories 1 2020-01-03 11:47:40,486 [main] INFO - Directory s3a://ireland-1/fork-0006 2020-01-03 11:47:40,534 [main] INFO - files 1; directories 0 2020-01-03 11:47:40,535 [main] INFO - Directory s3a://ireland-1/path 2020-01-03 11:47:40,587 [main] INFO - files 0; directories 1 2020-01-03 11:47:40,588 [main] INFO - Directory s3a://ireland-1/fork-0008 2020-01-03 11:47:40,641 [main] INFO - files 0; directories 1 2020-01-03 11:47:40,642 [main] INFO - Directory s3a://ireland-1/fork-0004 2020-01-03 11:47:40,692 [main] INFO - files 0; directories 1 2020-01-03 11:47:40,693 [main] WARN - Directory s3a://ireland-1/fork-0003 is not authoritative 2020-01-03 11:47:40,693 [main] INFO - audit s3a://ireland-1/: duration 0:00.402s 2020-01-03 11:47:40,698 [main] INFO - Exiting with status 46: `s3a://ireland-1/fork-0003': Directory is not marked as authoritative in the S3Guard store
This command is primarily for testing.
Alter the read and/or write capacity of a s3guard table created with provisioned I/O capacity.
hadoop s3guard set-capacity [--read UNIT] [--write UNIT] ( -region REGION | s3a://BUCKET )
The --read and --write units are those of s3guard init.
It cannot be used to change the I/O capacity of an on demand table (there is no need), and nor can it be used to convert an existing table to being on-demand. For that the AWS console must be used.
Example
hadoop s3guard set-capacity -read 20 -write 20 s3a://ireland-1
Set the capacity of the table used by bucket s3a://ireland-1 to 20 read and 20 write. (This is a low number, incidentally)
2017-08-30 16:21:26,343 [main] INFO s3guard.S3GuardTool (S3GuardTool.java:initMetadataStore(229)) - Metadata store DynamoDBMetadataStore{region=eu-west-1, tableName=ireland-1} is initialized. 2017-08-30 16:21:26,344 [main] INFO s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:updateParameters(1084)) - Current table capacity is read: 25, write: 25 2017-08-30 16:21:26,344 [main] INFO s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:updateParameters(1086)) - Changing capacity of table to read: 20, write: 20 Metadata Store Diagnostics: ARN=arn:aws:dynamodb:eu-west-1:00000000000:table/ireland-1 billing-mode=provisioned description=S3Guard metadata store in DynamoDB name=ireland-1 read-capacity=25 region=eu-west-1 retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS) size=12812 status=UPDATING table={ ... } write-capacity=25
After the update, the table status changes to UPDATING; this is a sign that the capacity has been changed.
Repeating the same command will not change the capacity, as both read and write values match that already in use.
2017-08-30 16:24:35,337 [main] INFO s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:updateParameters(1090)) - Table capacity unchanged at read: 20, write: 20 Metadata Store Diagnostics: ARN=arn:aws:dynamodb:eu-west-1:00000000000:table/ireland-1 billing-mode=provisioned description=S3Guard metadata store in DynamoDB name=ireland-1 read-capacity=20 region=eu-west-1 retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS) size=12812 status=ACTIVE table={ ... } write-capacity=20
Note: There is a limit to how many times in a 24 hour period the capacity of a bucket can be changed, either through this command or the AWS console.
Compares S3 with MetadataStore, and returns a failure status if any rules or invariants are violated. Only works with DynamoDB metadata stores.
hadoop s3guard fsck [-check | -internal] [-fix] (s3a://BUCKET | s3a://PATH_PREFIX)
-check operation checks the metadata store from the S3 perspective, but does not fix any issues. The consistency issues will be logged in ERROR loglevel.
-internal operation checks the internal consistency of the metadata store, but does not fix any issues.
-fix operation fixes consistency issues between the metadatastore and the S3 bucket. This parameter is optional, and can be used together with check or internal parameters, but not alone. The following fix is implemented: - Remove orphan entries from DDB
The errors found will be logged at the ERROR log level.
Note: -check and -internal operations can be used only as separate commands. Running fsck with both will result in an error.
Example
hadoop s3guard fsck -check s3a://ireland-1/path_prefix/
Checks the metadata store while iterating through the S3 bucket. The path_prefix will be used as the root element of the check.
hadoop s3guard fsck -internal s3a://ireland-1/path_prefix/
Checks the metadata store internal consistency. The path_prefix will be used as the root element of the check.
If you run into network connectivity issues, or have a machine failure in the middle of an operation, you may end up with your metadata store having state that differs from S3. The S3Guard CLI commands, covered in the CLI section above, can be used to diagnose and repair these issues.
There are some logs whose log level can be increased to provide more information.
# Log S3Guard classes log4j.logger.org.apache.hadoop.fs.s3a.s3guard=DEBUG # Log all S3A classes log4j.logger.org.apache.hadoop.fs.s3a=DEBUG # Enable debug logging of AWS DynamoDB client log4j.logger.com.amazonaws.services.dynamodbv2.AmazonDynamoDB # Log all HTTP requests made; includes S3 interaction. This may # include sensitive information such as account IDs in HTTP headers. log4j.logger.com.amazonaws.request=DEBUG
If all else fails, S3Guard is designed to allow for easy recovery by deleting the metadata store data. In DynamoDB, this can be accomplished by simply deleting the table, and allowing S3Guard to recreate it from scratch. Note that S3Guard tracks recent changes to file metadata to implement consistency. Deleting the metadata store table will simply result in a period of eventual consistency for any file modifications that were made right before the table was deleted.
When dealing with support calls related to the S3A connector, “is S3Guard on?” is the usual opening question. This can be determined by looking at the application logs for messages about S3Guard starting -the absence of S3Guard can only be inferred by the absence of such messages.
There is a another strategy: have the S3A Connector log whenever S3Guard is not enabled
This can be done in the configuration option fs.s3a.s3guard.disabled.warn.level
<property> <name>fs.s3a.s3guard.disabled.warn.level</name> <value>silent</value> <description> Level to print a message when S3Guard is disabled. Values: "warn": log at WARN level "inform": log at INFO level "silent": log at DEBUG level "fail": raise an exception </description> </property>
The fail option is clearly more than logging; it exists as an extreme debugging tool. Use with care.
Operations which modify metadata will make changes to S3 first. If, and only if, those operations succeed, the equivalent changes will be made to the Metadata Store.
These changes to S3 and Metadata Store are not fully-transactional: If the S3 operations succeed, and the subsequent Metadata Store updates fail, the S3 changes will not be rolled back. In this case, an error message will be logged.
S3Guard tables are created with a version marker entry and table tag. The entry is created with the primary key and child entry of ../VERSION; the use of a relative path guarantees that it will not be resolved. Table tag key is named s3guard_version.
When the table is initialized by S3Guard, the table will be tagged during the creating and the version marker entry will be created in the table. If the table lacks the version marker entry or tag, S3Guard will try to create it according to the following rules:
Note: If the user does not have sufficient rights to tag the table the initialization of S3Guard will not fail, but there will be no version marker tag on the dynamo table.
Versioning policy
Note: this policy does not indicate any intent to upgrade table structures in an incompatible manner. The version marker in tables exists to support such an option if it ever becomes necessary, by ensuring that all S3Guard client can recognise any version mismatch.
All users of the DynamoDB table must have write access to it. This effectively means they must have write access to the entire object store.
There’s not been much testing of using a S3Guard Metadata Store with a read-only S3 Bucket. It should work, provided all users have write access to the DynamoDB table. And, as updates to the Metadata Store are only made after successful file creation, deletion and rename, the store is unlikely to get out of sync, it is still something which merits more testing before it could be considered reliable.
Historically, DynamoDB has been not only billed on use (data and I/O requests) -but on provisioned I/O Capacity.
With Provisioned IO, when an application makes more requests than the allocated capacity permits, the request is rejected; it is up to the calling application to detect when it is being so throttled and react. S3Guard does this, but as a result: when the client is being throttled, operations are slower. This capacity throttling is averaged over a few minutes: a briefly overloaded table will not be throttled, but the rate cannot be sustained.
The load on a table is visible in the AWS console: go to the DynamoDB page for the table and select the “metrics” tab. If the graphs of throttled read or write requests show that a lot of throttling has taken place, then there is not enough allocated capacity for the applications making use of the table.
Similarly, if the capacity graphs show that the read or write loads are low compared to the allocated capacities, then the table may be overprovisioned for the current workload.
The S3Guard connector to DynamoDB can be configured to make multiple attempts to repeat a throttled request, with an exponential backoff between them.
The relevant settings for managing retries in the connector are:
<property> <name>fs.s3a.s3guard.ddb.max.retries</name> <value>9</value> <description> Max retries on throttled/incompleted DynamoDB operations before giving up and throwing an IOException. Each retry is delayed with an exponential backoff timer which starts at 100 milliseconds and approximately doubles each time. The minimum wait before throwing an exception is sum(100, 200, 400, 800, .. 100*2^N-1 ) == 100 * ((2^N)-1) </description> </property> <property> <name>fs.s3a.s3guard.ddb.throttle.retry.interval</name> <value>100ms</value> <description> Initial interval to retry after a request is throttled events; the back-off policy is exponential until the number of retries of fs.s3a.s3guard.ddb.max.retries is reached. </description> </property> <property> <name>fs.s3a.s3guard.ddb.background.sleep</name> <value>25ms</value> <description> Length (in milliseconds) of pause between each batch of deletes when pruning metadata. Prevents prune operations (which can typically be low priority background operations) from overly interfering with other I/O operations. </description> </property>
Having a large value for fs.s3a.s3guard.ddb.max.retries will ensure that clients of an overloaded table will not fail immediately. However queries may be unexpectedly slow.
If operations, especially directory operations, are slow, check the AWS console. It is also possible to set up AWS alerts for capacity limits being exceeded.
Amazon DynamoDB On-Demand removes the need to pre-allocate I/O capacity for S3Guard tables. Instead the caller is only charged per I/O Operation.
When idle, S3Guard tables are only billed for the data stored, not for any unused capacity. For this reason, there is no performance benefit from sharing a single S3Guard table across multiple buckets.
Creating a S3Guard Table with On-Demand Tables
The default settings for S3Guard are to create on-demand tables; this can also be done explicitly in the s3guard init command by setting the read and write capacities to zero.
hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0 s3a://ireland-1
Enabling DynamoDB On-Demand for an existing S3Guard table
You cannot currently convert an existing S3Guard table to being an on-demand table through the s3guard command.
It can be done through the AWS console or the CLI. From the Web console or the command line, switch the billing to pay-per-request.
Once enabled, the read and write capacities of the table listed in the hadoop s3guard bucket-info command become “0”, and the “billing-mode” attribute changes to “per-request”:
> hadoop s3guard bucket-info s3a://example-bucket/ Filesystem s3a://example-bucket Location: eu-west-1 Filesystem s3a://example-bucket is using S3Guard with store DynamoDBMetadataStore{region=eu-west-1, tableName=example-bucket, tableArn=arn:aws:dynamodb:eu-west-1:11111122223333:table/example-bucket} Authoritative S3Guard: fs.s3a.metadatastore.authoritative=false Metadata Store Diagnostics: ARN=arn:aws:dynamodb:eu-west-1:11111122223333:table/example-bucket billing-mode=per-request description=S3Guard metadata store in DynamoDB name=example-bucket persist.authoritative.bit=true read-capacity=0 region=eu-west-1 retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=250 MILLISECONDS) size=66797 status=ACTIVE table={AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}], TableName: example-bucket, KeySchema: [{ AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}], TableStatus: ACTIVE, CreationDateTime: Thu Oct 11 18:51:14 BST 2018, ProvisionedThroughput: { LastIncreaseDateTime: Tue Oct 30 16:48:45 GMT 2018, LastDecreaseDateTime: Tue Oct 30 18:00:03 GMT 2018, NumberOfDecreasesToday: 0, ReadCapacityUnits: 0, WriteCapacityUnits: 0}, TableSizeBytes: 66797, ItemCount: 415, TableArn: arn:aws:dynamodb:eu-west-1:11111122223333:table/example-bucket, TableId: a7b0728a-f008-4260-b2a0-aaaaabbbbb,} write-capacity=0 The "magic" committer is supported
DynamoDB Auto Scaling can automatically increase and decrease the allocated capacity.
Before DynamoDB On-Demand was introduced, autoscaling was the sole form of dynamic scaling.
Experiments with S3Guard and DynamoDB Auto Scaling have shown that any Auto Scaling operation will only take place after callers have been throttled for a period of time. The clients will still need to be configured to retry when overloaded until any extra capacity is allocated. Furthermore, as this retrying will block the threads from performing other operations -including more I/O, the the autoscale may not scale fast enough.
This is why the DynamoDB On-Demand appears is a better option for workloads with Hadoop, Spark, Hive and other applications.
If autoscaling is to be used, we recommend experimenting with the option, based on usage information collected from previous days, and choosing a combination of retry counts and an interval which allow for the clients to cope with some throttling, but not to time-out other applications.
S3Guard provides read-after-overwrite consistency through ETags (default) or object versioning checked either on the server (default) or client. This works such that a reader reading a file after an overwrite either sees the new version of the file or an error. Without S3Guard, new readers may see the original version. Once S3 reaches eventual consistency, new readers will see the new version.
Readers using S3Guard will usually see the new file version, but may in rare cases see RemoteFileChangedException instead. This would occur if an S3 object read cannot provide the version tracked in S3Guard metadata.
S3Guard achieves this behavior by storing ETags and object version IDs in the S3Guard metadata store (e.g. DynamoDB). On opening a file, S3AFileSystem will look in S3 for the version of the file indicated by the ETag or object version ID stored in the metadata store. If that version is unavailable, RemoteFileChangedException is thrown. Whether ETag or version ID and server or client mode is used is determed by the fs.s3a.change.detection configuration options.
When the first S3AFileSystem clients are upgraded to a version of S3AFileSystem that contains these change tracking features, any existing S3Guard metadata will not contain ETags or object version IDs. Reads of files tracked in such S3Guard metadata will access whatever version of the file is available in S3 at the time of read. Only if the file is subsequently updated will S3Guard start tracking ETag and object version ID and as such generating RemoteFileChangedException if an inconsistency is detected.
Similarly, when S3Guard metadata is pruned, S3Guard will no longer be able to detect an inconsistent read. S3Guard metadata should be retained for at least as long as the perceived possible read-after-overwrite temporary inconsistency window. That window is expected to be short, but there are no guarantees so it is at the administrator’s discretion to weigh the risk.
S3 Select does not provide a capability for server-side ETag or object version ID qualification. Whether fs.s3a.change.detection.mode is “client” or “server”, S3Guard will cause a client-side check of the file version before opening the file with S3 Select. If the current version does not match the version tracked in S3Guard, RemoteFileChangedException is thrown.
It is still possible that the S3 Select read will access a different version of the file, if the visible file version changes between the version check and the opening of the file. This can happen due to eventual consistency or an overwrite of the file between the version check and the open of the file.
Rename is implemented via copy in S3. With fs.s3a.change.detection.mode set to “client”, a fully reliable mechansim for ensuring the copied content is the expected content is not possible. This is the case since there isn’t necessarily a way to know the expected ETag or version ID to appear on the object resulting from the copy.
Furthermore, if fs.s3a.change.detection.mode is “server” and a third-party S3 implementation is used that doesn’t honor the provided ETag or version ID, S3AFileSystem and S3Guard cannot detect it.
When fs.s3.change.detection.mode is “client”, a client-side check will be performed before the copy to ensure the current version of the file matches S3Guard metadata. If not, RemoteFileChangedException is thrown. Similar to as discussed with regard to S3 Select, this is not sufficient to guarantee that same version is the version copied.
When fs.s3.change.detection.mode server, the expected version is also specified in the underlying S3 CopyObjectRequest. As long as the server honors it, the copied object will be correct.
All this said, with the defaults of fs.s3.change.detection.mode of “server” and fs.s3.change.detection.source of “etag”, when working with Amazon’s S3, copy should in fact either copy the expected file version or, in the case of an eventual consistency anomaly, generate RemoteFileChangedException. The same should be true when fs.s3.change.detection.source = “versionid”.
The S3Guard version tracking metadata (ETag or object version ID) could become out of sync with the true current object metadata in S3. For example, S3Guard is still tracking v1 of some file after v2 has been written. This could occur for reasons such as a writer writing without utilizing S3Guard and/or S3AFileSystem or simply due to a write with S3AFileSystem and S3Guard that wrote successfully to S3, but failed in communication with S3Guard’s metadata store (e.g. DynamoDB).
If this happens, reads of the affected file(s) will result in RemoteFileChangedException until one of:
The S3Guard metadata for a file can be corrected with the s3guard import command as discussed above. The command can take a file URI instead of a bucket URI to correct the metadata for a single file. For example:
hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
The table which was intended to be used as a S3guard metadata store does not have any version marker indicating that it is a S3Guard table.
It may be that this is not a S3Guard table.
This indicates that the version of S3Guard which created (or possibly updated) the database table is from a different version that that expected by the S3A client.
This error will also include the expected and actual version numbers.
If the expected version is lower than the actual version, then the version of the S3A client library is too old to interact with this S3Guard-managed bucket. Upgrade the application/library.
If the expected version is higher than the actual version, then the table itself will need upgrading.
S3Guard could not find the DynamoDB table for the Metadata Store, and it was not configured to create it. Either the table was missing, or the configuration is preventing S3Guard from finding the table.
org.apache.hadoop.fs.s3a.AWSServiceThrottledException: listFiles on s3a://bucket/10/d1/d2/d3: com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ProvisionedThroughputExceededException;
The I/O load of clients of the (shared) DynamoDB table was exceeded.
The I/O load of clients of the (shared) DynamoDB table was exceeded, and the number of attempts to retry the operation exceeded the configured amount.
org.apache.hadoop.fs.s3a.AWSServiceThrottledException: ProvisionTable on s3guard-example: com.amazonaws.services.dynamodbv2.model.LimitExceededException: Subscriber limit exceeded: Provisioned throughput decreases are limited within a given UTC day. After the first 4 decreases, each subsequent decrease in the same UTC day can be performed at most once every 3600 seconds. Number of decreases today: 6. Last decrease at Wednesday, July 25, 2018 8:48:14 PM UTC. Next decrease can be made at Wednesday, July 25, 2018 9:48:14 PM UTC
There’s are limit on how often you can change the capacity of an DynamoDB table; if you call set-capacity too often, it fails. Wait until the after the time indicated and try again.
java.io.IOException: Invalid region specified "iceland-2": Region can be configured with fs.s3a.s3guard.ddb.region: us-gov-west-1, us-east-1, us-east-2, us-west-1, us-west-2, eu-west-1, eu-west-2, eu-west-3, eu-central-1, ap-south-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, ap-northeast-2, sa-east-1, cn-north-1, cn-northwest-1, ca-central-1 at org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory$DefaultDynamoDBClientFactory.getRegion at org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory$DefaultDynamoDBClientFactory.createDynamoDBClient
The region specified in fs.s3a.s3guard.ddb.region is invalid.
ValidationException; One or more parameter values were invalid: Neither ReadCapacityUnits nor WriteCapacityUnits can be specified when BillingMode is PAY_PER_REQUEST (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException)
On-Demand DynamoDB tables do not have any fixed capacity -it is an error to try to change it with the set-capacity command.
A filesystem write operation failed to persist metadata to S3Guard. The file was successfully written to S3 and now the S3Guard metadata is likely to be out of sync.
See Fail on Error for more detail.
An exception like the following could occur for a couple of reasons:
the S3Guard metadata is out of sync with the true S3 metadata. For example, the S3Guard DynamoDB table is tracking a different ETag than the ETag shown in the exception. This may suggest the object was updated in S3 without involvement from S3Guard or there was a transient failure when S3Guard tried to write to DynamoDB.
S3 is exhibiting read-after-overwrite temporary inconsistency. The S3Guard metadata was updated with a new ETag during a recent write, but the current read is not seeing that ETag due to S3 eventual consistency. This exception prevents the reader from an inconsistent read where the reader sees an older version of the file.
org.apache.hadoop.fs.s3a.RemoteFileChangedException: open 's3a://my-bucket/test/file.txt': Change reported by S3 while reading at position 0. ETag 4e886e26c072fef250cfaf8037675405 was unavailable at org.apache.hadoop.fs.s3a.impl.ChangeTracker.processResponse(ChangeTracker.java:167) at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:207) at org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:355) at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$2(Invoker.java:195) at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109) at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:193) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:215) at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:348) at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:381) at java.io.FilterInputStream.read(FilterInputStream.java:83)
The AWS SDK has an issue where it will throw a relatively generic AmazonClientException caused by NullPointerException when copying a file and specifying a precondition that cannot be met. This can bubble up from S3AFileSystem.rename(). It suggests that the file in S3 is inconsistent with the metadata in S3Guard.
org.apache.hadoop.fs.s3a.AWSClientIOException: copyFile(test/rename-eventually2.dat, test/dest2.dat) on test/rename-eventually2.dat: com.amazonaws.AmazonClientException: Unable to complete transfer: null: Unable to complete transfer: null at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:201) at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111) at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:314) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:406) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:310) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:285) at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3034) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:1258) at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:1119) at org.apache.hadoop.fs.s3a.ITestS3ARemoteFileChanged.lambda$testRenameEventuallyConsistentFile2$6(ITestS3ARemoteFileChanged.java:556) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498) at org.apache.hadoop.fs.s3a.ITestS3ARemoteFileChanged.testRenameEventuallyConsistentFile2(ITestS3ARemoteFileChanged.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: null at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286) at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265) at com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) at org.apache.hadoop.fs.s3a.impl.CopyOutcome.waitForCopy(CopyOutcome.java:72) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$14(S3AFileSystem.java:3047) at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109) ... 25 more Caused by: java.lang.NullPointerException at com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk(CopyCallable.java:154) at com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:134) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:132) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:43) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more
com.amazonaws.services.dynamodbv2.model.ResourceInUseException: Attempt to change a resource which is still in use: Table is being deleted: s3guard.test.testDynamoDBInitDestroy351245027 (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceInUseException;)
You have attempted to call hadoop s3guard destroy on a table which is already being destroyed.
For details on how to test S3Guard, see Testing S3Guard


My name is Nathan Heidt. I'm an inventor at heart and an engineer by trade with a background in machine learning and robotics. I'm also well versed in lower level engineering such as PCB design and Firmware Programming. I have a BSE in Computer Engineering from the University of Michigan and currently live in Oakland California. I originally came out to the bay to work on my first company Lagoon Systems. Lagoon was a water conservation and management company that helped me achieve a life dream of getting on the Forbes 30 under 30 list. Today, I work primarily in machine learning and I'm most interested in the field of cognitive robotics.
I'm passionate about the fields of deep space and deep sea exploration. Through robotics, I believe that we can push human knowledge into places no human could go. In order to achieve this, autonomous systems must be capable of improvising and learning just like humans do. I'm constantly working on my own robotics projects to explore these concepts. Beyond the technical side of things, I'm also an avid sailor and reader. I'm driven to explore the world in many ways.
| Languages | Machine Learning | Robotics | Tools |
|---|---|---|---|
| C | Theano | ROS | *Nix |
| Python | Caffe | OpenCV | Git |
| C++ | scikit-learn | PCL | Solidworks (3D modelling) |
| Java | Keras | SLAM techniques | Altium and Eagle (PCB design) |
| Assembly (Arm, thumb) | Tensorflow | Multiple View Geometry | Machining |
Click the button below to download my resume in PDF form
Download


All Classes
Evaluator
Expression
ExpressionNode
GlobalVar
Intermediary
Mapping
MappingConversions
MappingMan
Operator
PatchPanel
TimerMan
VariableResolver


--- layout: post title: Chrome Web Store - Google I/O 2010 Keynote Day 1, pt. 5 categories: [] tags: - apple - appstore - chrome - google status: publish type: post published: true meta: _original_post_id: '' _original_import_author: John Duprey author: ---
Apple had best move quickly or Google may beat them at their own game on the desktop. Google is creating a platform for application discovery, distribution, and purchase similar to the iPod/iPhone/iPad appstore all for the desktop - where Chrome, Google's super-fast web browser, *is* the platform.
https://chrome.google.com/webstore


When a sentence ends with a capital letter an end-of-sentence space must be used.
\@ Causes an "end-of-sentence" space after punctuation when typed before the punctuation character.
Needed only if the character preceding the punctuation character is not a lowercase letter or a number.


Pack Name
Pack Format
6 (1.16.2-1.16.5)
5 (1.15-1.16.1)
4 (1.13-1.14.4)
3 (1.11-1.12.2)
2 (1.9-1.10.2)
1 (1.6.1-1.8.9)
Location
Change
A new folder will be created inside this folder
Files


{%extends 'index.html'%} {%load static%}
{%block content%}We would be glad to have feedback from you. Drop us a line, whether it is a comment, a question, a work proposition or just a hello. You can use either the form below or the contact details on the right.


{%extends "layout.html"%} {% block head %} {% endblock %} {%block content%}
Disease diagnosis plays a pivotal role in healthcare informatics. It is crucial to diagnose the disease at an early stage. The lack or delay in proper treatment can also lead to the death of patients. Therefore disease diagnosis can prove to be life saving in many cases. Disease diagnose system can assist doctors, physicians and healthcare professionals in effective decision making for disease diagnosis as well.
The disease is diagnosed using the algorithm(Decision Tree Classifier) and the user has to enter the symptoms from the given drop-down menu.
User is provided with resources for preventing common diseases. Insighful information related to yoga asanas are given for preventing diseases.
The web application provides latest health related news from all over the world and provide location of nearby health centers. Have conversation with our medibot.
Machine Learning is an emerging approach that helps in diagnosis, diagnosis of a disease. The accuracy of an algorithm is determined by the performance on the given dataset. Our model is able to diagnose data with 90% average accuracy.
Rating:
However we have not been awarded any awards or certifiacte for the idea yet but we wish that we can enhance this application that it can be of great help in medical industry and is able to save hundreds of lives.
Rating:
In future, we are planning to implement the feature of disease diagnosis through whatsapp. We also aim to develop the feature of seeking online medical help via a calling feature.
Rating:


This component displays what is inside as an option of the dropdown.
This component displays what is inside as an icon. It can be used as a child of the dropdown or any of the options.


Custom form fields
Globally defined custom fields
You could configure application to include custom inputs inside you form definition, they will be abide the same
rules as default ones if they implement
ControlValueAccessor interface.
To configure custom input definition do following
-
Create component which implements
ControlValueAccessor interface
-
Configure it by using with holistic library
-
Include component in app root module declarations section or if this component exported by some module
import one to root module
-
Add this component to
entryComponents of app root
module
-
Configure HlcClrFormModule to use it as following
HlcClrFormModule.forRoot({{ '{' }}myCustomField: CustomFieldComponent{{ '}' }})
-
Define
custom FormLayout type derived from holistic ClrFormLayouts.ClrFormLayout type in order to
typescript getting know this custom component belongs new FormLayout definition.
-
Now use this new FormLayout interface to define form with this custom component included across your
application like any other standard one. Notice
kind of this component will be
myCustomField.
Custom fields defined inside the form
If you want to use some specific component inside of form which is not included in global components definition
layout there is option to do this in-place.