The Project Gutenberg EBook of The Tale of Timmy Tiptoes, by Beatrix Potter This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.net Title: The Tale of Timmy Tiptoes Author: Beatrix Potter Release Date: January 25, 2005 [EBook #14797] Language: English Character set encoding: ASCII *** START OF THIS PROJECT GUTENBERG EBOOK THE TALE OF TIMMY TIPTOES *** Produced by Robert Cicconetti, Emmy and the PG Online Distributed Proofreading Team (http://www.pgdp.net).
Author of "The Tale of Peter Rabbit," etc.
FREDERICK WARNE & CO., INC.  He had a nest thatched with leaves in the top of a tall tree; and he had a little squirrel wife called Goody.
Timmy Tiptoes sat out, enjoying the breeze; he whisked his tail and chuckled—"Little wife Goody, the nuts are ripe; we must lay up a store for winter and spring." Goody Tiptoes was busy pushing moss under the thatch—"The nest is so snug, we shall be sound asleep all winter." "Then we shall wake up all the thinner, when there is nothing to eat in spring-time," replied prudent Timothy.
When Timmy and Goody Tiptoes came to the nut thicket, they found other squirrels were there already.
Timmy took off his jacket and hung it on a twig; they worked away quietly by themselves.
Every day they made several journeys and picked quantities of nuts. They carried them away in bags, and stored them in several hollow stumps near the tree where they had built their nest.
When these stumps were full, they began to empty the bags into a hole high up a tree, that had belonged to a wood-pecker; the nuts rattled down—down—down inside.
"How shall you ever get them out again? It is like a money-box!" said Goody.
"I shall be much thinner before spring-time, my love," said Timmy Tiptoes, peeping into the hole.
They did collect quantities—because they did not lose them! Squirrels who bury their nuts in the ground lose more than half, because they cannot remember the place.
The most forgetful squirrel in the wood was called Silvertail. He began to dig, and he could not remember. And then he dug again and found some nuts that did not belong to him; and there was a fight. And other squirrels began to dig,—the whole wood was in commotion!
Unfortunately, just at this time a flock of little birds flew by, from bush to bush, searching for green caterpillars and spiders. There were several sorts of little birds, twittering different songs.
The first one sang—"Who's bin digging-up my nuts? Who's-been-digging-up my nuts?"
And another sang—"Little bita bread and-no-cheese! Little bit-a-bread an'-no-cheese!"
The squirrels followed and listened. The first little bird flew into the bush where Timmy and Goody Tiptoes were quietly tying up their bags, and it sang—"Who's-bin digging-up my nuts? Who's been digging-up my-nuts?"
Timmy Tiptoes went on with his work without replying; indeed, the little bird did not expect an answer. It was only singing its natural song, and it meant nothing at all.
But when the other squirrels heard that song, they rushed upon Timmy Tiptoes and cuffed and scratched him, and upset his bag of nuts. The innocent little bird which had caused all the mischief, flew away in a fright!
Timmy rolled over and over, and then turned tail and fled towards his nest, followed by a crowd of squirrels shouting—"Who's-been digging-up my-nuts?"
They caught him and dragged him up the very same tree, where there was the little round hole, and they pushed him in. The hole was much too small for Timmy Tiptoes' figure. They squeezed him dreadfully, it was a wonder they did not break his ribs. "We will leave him here till he confesses," said Silvertail Squirrel, and he shouted into the hole—
"Who's-been-digging-up my-nuts?"
Timmy Tiptoes made no reply; he had tumbled down inside the tree, upon half a peck of nuts belonging to himself. He lay quite stunned and still.
Goody Tiptoes picked up the nut bags and went home. She made a cup of tea for Timmy; but he didn't come and didn't come.
Goody Tiptoes passed a lonely and unhappy night. Next morning she ventured back to the nut-bushes to look for him; but the other unkind squirrels drove her away.
She wandered all over the wood, calling—
"Timmy Tiptoes! Timmy Tiptoes! Oh, where is Timmy Tiptoes?"
In the meantime Timmy Tiptoes came to his senses. He found himself tucked up in a little moss bed, very much in the dark, feeling sore; it seemed to be under ground. Timmy coughed and groaned, because his ribs hurted him. There was a chirpy noise, and a small striped Chipmunk appeared with a night light, and hoped he felt better?
It was most kind to Timmy Tiptoes; it lent him its night-cap; and the house was full of provisions.
The Chipmunk explained that it had rained nuts through the top of the tree—"Besides, I found a few buried!" It laughed and chuckled when it heard Timmy's story. While Timmy was confined to bed, it 'ticed him to eat quantities—"But how shall I ever get out through that hole unless I thin myself? My wife will be anxious!" "Just another nut—or two nuts; let me crack them for you," said the Chipmunk. Timmy Tiptoes grew fatter and fatter!
Now Goody Tiptoes had set to work again by herself. She did not put any more nuts into the woodpecker's hole, because she had always doubted how they could be got out again. She hid them under a tree root; they rattled down, down, down. Once when Goody emptied an extra big bagful, there was a decided squeak; and next time Goody brought another bagful, a little striped Chipmunk scrambled out in a hurry.
"It is getting perfectly full-up down-stairs; the sitting-room is full, and they are rolling along the passage; and my husband, Chippy Hackee, has run away and left me. What is the explanation of these showers of nuts?"
"I am sure I beg your pardon; I did not know that anybody lived here," said Mrs. Goody Tiptoes; "but where is Chippy Hackee? My husband, Timmy Tiptoes, has run away too." "I know where Chippy is; a little bird told me," said Mrs. Chippy Hackee.
She led the way to the woodpecker's tree, and they listened at the hole.

"You could squeeze in, through that little round hole," said Goody Tiptoes. "Yes, I could," said the Chipmunk, "but my husband, Chippy Hackee, bites!"
Down below there was a noise of cracking nuts and nibbling; and then the fat squirrel voice and the thin squirrel voice sang—
"For the diddlum day
Day diddle dum di!
Day diddle diddle dum day!"
Then Goody peeped in at the hole, and called down—"Timmy Tiptoes! Oh fie, Timmy Tiptoes!" And Timmy replied, "Is that you, Goody Tiptoes? Why, certainly!"
He came up and kissed Goody through the hole; but he was so fat that he could not get out.
Chippy Hackee was not too fat, but he did not want to come; he stayed down below and chuckled.
And so it went on for a fortnight; till a big wind blew off the top of the tree, and opened up the hole and let in the rain.
Then Timmy Tiptoes came out, and went home with an umbrella.
But Chippy Hackee continued to camp out for another week, although it was uncomfortable.
At last a large bear came walking through the wood. Perhaps he also was looking for nuts; he seemed to be sniffing around.
Chippy Hackee went home in a hurry!
And when Chippy Hackee got home, he found he had caught a cold in his head; and he was more uncomfortable still.
And now Timmy and Goody Tiptoes keep their nut-store fastened up with a little padlock.
And whenever that little bird sees the Chipmunks, he sings—"Who's-been-digging-up my-nuts? Who's been digging-up my-nuts?" But nobody ever answers!
End of Project Gutenberg's The Tale of Timmy Tiptoes, by Beatrix Potter *** END OF THIS PROJECT GUTENBERG EBOOK THE TALE OF TIMMY TIPTOES *** ***** This file should be named 14797-h.htm or 14797-h.zip ***** This and all associated files of various formats will be found in: http://www.gutenberg.net/1/4/7/9/14797/ Produced by Robert Cicconetti, Emmy and the PG Online Distributed Proofreading Team (http://www.pgdp.net). Updated editions will replace the previous one--the old editions will be renamed. Creating the works from public domain print editions means that no one owns a United States copyright in these works, so the Foundation (and you!) can copy and distribute it in the United States without permission and without paying copyright royalties. Special rules, set forth in the General Terms of Use part of this license, apply to copying and distributing Project Gutenberg-tm electronic works to protect the PROJECT GUTENBERG-tm concept and trademark. Project Gutenberg is a registered trademark, and may not be used if you charge for the eBooks, unless you receive specific permission. If you do not charge anything for copies of this eBook, complying with the rules is very easy. You may use this eBook for nearly any purpose such as creation of derivative works, reports, performances and research. They may be modified and printed and given away--you may do practically ANYTHING with public domain eBooks. Redistribution is subject to the trademark license, especially commercial redistribution. *** START: FULL LICENSE *** THE FULL PROJECT GUTENBERG LICENSE PLEASE READ THIS BEFORE YOU DISTRIBUTE OR USE THIS WORK To protect the Project Gutenberg-tm mission of promoting the free distribution of electronic works, by using or distributing this work (or any other work associated in any way with the phrase "Project Gutenberg"), you agree to comply with all the terms of the Full Project Gutenberg-tm License (available with this file or online at http://gutenberg.net/license). Section 1. General Terms of Use and Redistributing Project Gutenberg-tm electronic works 1.A. By reading or using any part of this Project Gutenberg-tm electronic work, you indicate that you have read, understand, agree to and accept all the terms of this license and intellectual property (trademark/copyright) agreement. If you do not agree to abide by all the terms of this agreement, you must cease using and return or destroy all copies of Project Gutenberg-tm electronic works in your possession. If you paid a fee for obtaining a copy of or access to a Project Gutenberg-tm electronic work and you do not agree to be bound by the terms of this agreement, you may obtain a refund from the person or entity to whom you paid the fee as set forth in paragraph 1.E.8. 1.B. "Project Gutenberg" is a registered trademark. It may only be used on or associated in any way with an electronic work by people who agree to be bound by the terms of this agreement. There are a few things that you can do with most Project Gutenberg-tm electronic works even without complying with the full terms of this agreement. See paragraph 1.C below. There are a lot of things you can do with Project Gutenberg-tm electronic works if you follow the terms of this agreement and help preserve free future access to Project Gutenberg-tm electronic works. See paragraph 1.E below. 1.C. The Project Gutenberg Literary Archive Foundation ("the Foundation" or PGLAF), owns a compilation copyright in the collection of Project Gutenberg-tm electronic works. Nearly all the individual works in the collection are in the public domain in the United States. If an individual work is in the public domain in the United States and you are located in the United States, we do not claim a right to prevent you from copying, distributing, performing, displaying or creating derivative works based on the work as long as all references to Project Gutenberg are removed. Of course, we hope that you will support the Project Gutenberg-tm mission of promoting free access to electronic works by freely sharing Project Gutenberg-tm works in compliance with the terms of this agreement for keeping the Project Gutenberg-tm name associated with the work. You can easily comply with the terms of this agreement by keeping this work in the same format with its attached full Project Gutenberg-tm License when you share it without charge with others. 1.D. The copyright laws of the place where you are located also govern what you can do with this work. Copyright laws in most countries are in a constant state of change. If you are outside the United States, check the laws of your country in addition to the terms of this agreement before downloading, copying, displaying, performing, distributing or creating derivative works based on this work or any other Project Gutenberg-tm work. The Foundation makes no representations concerning the copyright status of any work in any country outside the United States. 1.E. Unless you have removed all references to Project Gutenberg: 1.E.1. The following sentence, with active links to, or other immediate access to, the full Project Gutenberg-tm License must appear prominently whenever any copy of a Project Gutenberg-tm work (any work on which the phrase "Project Gutenberg" appears, or with which the phrase "Project Gutenberg" is associated) is accessed, displayed, performed, viewed, copied or distributed: This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.net 1.E.2. If an individual Project Gutenberg-tm electronic work is derived from the public domain (does not contain a notice indicating that it is posted with permission of the copyright holder), the work can be copied and distributed to anyone in the United States without paying any fees or charges. If you are redistributing or providing access to a work with the phrase "Project Gutenberg" associated with or appearing on the work, you must comply either with the requirements of paragraphs 1.E.1 through 1.E.7 or obtain permission for the use of the work and the Project Gutenberg-tm trademark as set forth in paragraphs 1.E.8 or 1.E.9. 1.E.3. If an individual Project Gutenberg-tm electronic work is posted with the permission of the copyright holder, your use and distribution must comply with both paragraphs 1.E.1 through 1.E.7 and any additional terms imposed by the copyright holder. Additional terms will be linked to the Project Gutenberg-tm License for all works posted with the permission of the copyright holder found at the beginning of this work. 1.E.4. Do not unlink or detach or remove the full Project Gutenberg-tm License terms from this work, or any files containing a part of this work or any other work associated with Project Gutenberg-tm. 1.E.5. Do not copy, display, perform, distribute or redistribute this electronic work, or any part of this electronic work, without prominently displaying the sentence set forth in paragraph 1.E.1 with active links or immediate access to the full terms of the Project Gutenberg-tm License. 1.E.6. You may convert to and distribute this work in any binary, compressed, marked up, nonproprietary or proprietary form, including any word processing or hypertext form. However, if you provide access to or distribute copies of a Project Gutenberg-tm work in a format other than "Plain Vanilla ASCII" or other format used in the official version posted on the official Project Gutenberg-tm web site (www.gutenberg.net), you must, at no additional cost, fee or expense to the user, provide a copy, a means of exporting a copy, or a means of obtaining a copy upon request, of the work in its original "Plain Vanilla ASCII" or other form. Any alternate format must include the full Project Gutenberg-tm License as specified in paragraph 1.E.1. 1.E.7. Do not charge a fee for access to, viewing, displaying, performing, copying or distributing any Project Gutenberg-tm works unless you comply with paragraph 1.E.8 or 1.E.9. 1.E.8. You may charge a reasonable fee for copies of or providing access to or distributing Project Gutenberg-tm electronic works provided that - You pay a royalty fee of 20% of the gross profits you derive from the use of Project Gutenberg-tm works calculated using the method you already use to calculate your applicable taxes. The fee is owed to the owner of the Project Gutenberg-tm trademark, but he has agreed to donate royalties under this paragraph to the Project Gutenberg Literary Archive Foundation. Royalty payments must be paid within 60 days following each date on which you prepare (or are legally required to prepare) your periodic tax returns. Royalty payments should be clearly marked as such and sent to the Project Gutenberg Literary Archive Foundation at the address specified in Section 4, "Information about donations to the Project Gutenberg Literary Archive Foundation." - You provide a full refund of any money paid by a user who notifies you in writing (or by e-mail) within 30 days of receipt that s/he does not agree to the terms of the full Project Gutenberg-tm License. You must require such a user to return or destroy all copies of the works possessed in a physical medium and discontinue all use of and all access to other copies of Project Gutenberg-tm works. - You provide, in accordance with paragraph 1.F.3, a full refund of any money paid for a work or a replacement copy, if a defect in the electronic work is discovered and reported to you within 90 days of receipt of the work. - You comply with all other terms of this agreement for free distribution of Project Gutenberg-tm works. 1.E.9. If you wish to charge a fee or distribute a Project Gutenberg-tm electronic work or group of works on different terms than are set forth in this agreement, you must obtain permission in writing from both the Project Gutenberg Literary Archive Foundation and Michael Hart, the owner of the Project Gutenberg-tm trademark. Contact the Foundation as set forth in Section 3 below. 1.F. 1.F.1. Project Gutenberg volunteers and employees expend considerable effort to identify, do copyright research on, transcribe and proofread public domain works in creating the Project Gutenberg-tm collection. Despite these efforts, Project Gutenberg-tm electronic works, and the medium on which they may be stored, may contain "Defects," such as, but not limited to, incomplete, inaccurate or corrupt data, transcription errors, a copyright or other intellectual property infringement, a defective or damaged disk or other medium, a computer virus, or computer codes that damage or cannot be read by your equipment. 1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except for the "Right of Replacement or Refund" described in paragraph 1.F.3, the Project Gutenberg Literary Archive Foundation, the owner of the Project Gutenberg-tm trademark, and any other party distributing a Project Gutenberg-tm electronic work under this agreement, disclaim all liability to you for damages, costs and expenses, including legal fees.   LIMITED RIGHT OF REPLACEMENT OR REFUND - If you discover a defect in this electronic work within 90 days of receiving it, you can receive a refund of the money (if any) you paid for it by sending a written explanation to the person you received the work from. If you received the work on a physical medium, you must return the medium with your written explanation. The person or entity that provided you with the defective work may elect to provide a replacement copy in lieu of a refund. If you received the work electronically, the person or entity providing it to you may choose to give you a second opportunity to receive the work electronically in lieu of a refund. If the second copy is also defective, you may demand a refund in writing without further opportunities to fix the problem. 1.F.4. Except for the limited right of replacement or refund set forth in paragraph 1.F.3, this work is provided to you 'AS-IS' WITH NO OTHER WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTIBILITY OR FITNESS FOR ANY PURPOSE. 1.F.5. Some states do not allow disclaimers of certain implied warranties or the exclusion or limitation of certain types of damages. If any disclaimer or limitation set forth in this agreement violates the law of the state applicable to this agreement, the agreement shall be interpreted to make the maximum disclaimer or limitation permitted by the applicable state law. The invalidity or unenforceability of any provision of this agreement shall not void the remaining provisions. 1.F.6. INDEMNITY - You agree to indemnify and hold the Foundation, the trademark owner, any agent or employee of the Foundation, anyone providing copies of Project Gutenberg-tm electronic works in accordance with this agreement, and any volunteers associated with the production, promotion and distribution of Project Gutenberg-tm electronic works, harmless from all liability, costs and expenses, including legal fees, that arise directly or indirectly from any of the following which you do or cause to occur: (a) distribution of this or any Project Gutenberg-tm work, (b) alteration, modification, or additions or deletions to any Project Gutenberg-tm work, and (c) any Defect you cause. Section 2. Information about the Mission of Project Gutenberg-tm Project Gutenberg-tm is synonymous with the free distribution of electronic works in formats readable by the widest variety of computers including obsolete, old, middle-aged and new computers. It exists because of the efforts of hundreds of volunteers and donations from people in all walks of life. Volunteers and financial support to provide volunteers with the assistance they need, is critical to reaching Project Gutenberg-tm's goals and ensuring that the Project Gutenberg-tm collection will remain freely available for generations to come. In 2001, the Project Gutenberg Literary Archive Foundation was created to provide a secure and permanent future for Project Gutenberg-tm and future generations. To learn more about the Project Gutenberg Literary Archive Foundation and how your efforts and donations can help, see Sections 3 and 4 and the Foundation web page at http://www.pglaf.org. Section 3. Information about the Project Gutenberg Literary Archive Foundation The Project Gutenberg Literary Archive Foundation is a non profit 501(c)(3) educational corporation organized under the laws of the state of Mississippi and granted tax exempt status by the Internal Revenue Service. The Foundation's EIN or federal tax identification number is 64-6221541. Its 501(c)(3) letter is posted at http://pglaf.org/fundraising. Contributions to the Project Gutenberg Literary Archive Foundation are tax deductible to the full extent permitted by U.S. federal laws and your state's laws. The Foundation's principal office is located at 4557 Melan Dr. S. Fairbanks, AK, 99712., but its volunteers and employees are scattered throughout numerous locations. Its business office is located at 809 North 1500 West, Salt Lake City, UT 84116,<PHONE_NUMBER>, email<EMAIL_ADDRESS>Email contact links and up to date contact information can be found at the Foundation's web site and official page at http://pglaf.org For additional contact information: Dr. Gregory B. Newby Chief Executive and Director<EMAIL_ADDRESS>Section 4. Information about Donations to the Project Gutenberg Literary Archive Foundation Project Gutenberg-tm depends upon and cannot survive without wide spread public support and donations to carry out its mission of increasing the number of public domain and licensed works that can be freely distributed in machine readable form accessible by the widest array of equipment including outdated equipment. Many small donations ($1 to $5,000) are particularly important to maintaining tax exempt status with the IRS. The Foundation is committed to complying with the laws regulating charities and charitable donations in all 50 states of the United States. Compliance requirements are not uniform and it takes a considerable effort, much paperwork and many fees to meet and keep up with these requirements. We do not solicit donations in locations where we have not received written confirmation of compliance. To SEND DONATIONS or determine the status of compliance for any particular state visit http://pglaf.org While we cannot and do not solicit contributions from states where we have not met the solicitation requirements, we know of no prohibition against accepting unsolicited donations from donors in such states who approach us with offers to donate. International donations are gratefully accepted, but we cannot make any statements concerning tax treatment of donations received from outside the United States. U.S. laws alone swamp our small staff. Please check the Project Gutenberg Web pages for current donation methods and addresses. Donations are accepted in a number of other ways including including checks, online payments and credit card donations. To donate, please visit: http://pglaf.org/donate Section 5. General Information About Project Gutenberg-tm electronic works. Professor Michael S. Hart is the originator of the Project Gutenberg-tm concept of a library of electronic works that could be freely shared with anyone. For thirty years, he produced and distributed Project Gutenberg-tm eBooks with only a loose network of volunteer support. Project Gutenberg-tm eBooks are often created from several printed editions, all of which are confirmed as Public Domain in the U.S. unless a copyright notice is included. Thus, we do not necessarily keep eBooks in compliance with any particular paper edition. Most people start at our Web site which has the main PG search facility: http://www.gutenberg.net This Web site includes information about Project Gutenberg-tm, including how to make donations to the Project Gutenberg Literary Archive Foundation, how to help produce our new eBooks, and how to subscribe to our email newsletter to hear about new eBooks.


The description of the Solitaire analysis dialog can be obtained here.
For the Solitaire cipher, there is kind of an attack (Ciphertext-only-attack), which makes a automatical or semi-automatical key search. For that it is necessary to know the final deck and the related ciphertext.
Because the algorithm is strong reversible, by reversing the steps of the algorithm the initial deck can be generated and so the symmetric key. With this key the ciphertext can be decrypted. The problem of the analysis is the non-complete reversibility of the algorithm with the effect that not each step can be reversed in an unique manner. If a wrong way is chosen, the further analysis is also wrong.
The analysis has three general steps:
1. Determining the initial deck
The Solitaire algorithm is passed through backwards for each ciphertext letter. That means it starts with step 5, then follows step 4 and so on.
After the algorithm is passed through backwards for each ciphertext letter and for each zero round, the initial deck is generated. Because of the aspect, that step 1 and step 2 cannot reversed in an unique manner, the algorithm is not completely reversible. If one of the jokers is sorted in at the wrong position, the decryption leads to a wrong result.
It follows the same procedure as at the decryption with Solitaire.
2. Generating the keystream
Now, the algorithm is used once on the initial deck for each ciphertext letter and for each zero round ( Solitaire keystream generation ). So the keystream, which has already been used for the encryption of the plaintext is created.
3. Decryption of the ciphertext
The ciphertext will be decrypted with this keystream.


This is an
alpha
. If you find something wrong,
let us know
Definitions Catalogue
ALPHA
Where Government defines its digital data.
Definitions Catalogue
ALPHA
☰ Menu
About Us
Discover APIs
Share an API
Data Definitions
Playbook
Community
Github


The Upload Tool
Upload is the standard starting point for the Lexos workflow. When you
begin a new session or reset your workspace, you will be automatically
redirected to Upload.
Note: Lexos supports only the Chrome and Firefox browsers.
Usage
- Upload Files
Drag your document files into the box labeled "Drag Files Here", or
click the "Browse" button to use the file browser to locate your
files. Most browsers will allow you to shift- or control-click to select
multiple files.
- Accepted File Types
Lexos accepts files as represented with filename (dot) extensions of:
.txt
, .html
, .xml
, .sgml
,
or .lexos
(Lexos workspace files, see below). Make sure that your
filenames use these file extensions.
- Maximum File Size
There are some restrictions on file upload size in order to prevent the
browser from hanging. Upload times may be slow for large files,
particularly if you are working over the internet. The maximum file size
of 250MB is approximately the size of nine Webster's Unabridged
Dictionaries. If you experience a problem, try uploading smaller files,
or, if you are uploading many files, try uploading them in smaller
batches.
- Upload List
Once you have selected your files, they will begin to upload, one at a
time. As each upload is complete, you will see the document names appear
in the "Upload List" at the bottom of the page. The bigger the file the
longer it will take to upload and show up on the page. After uploading is
complete, each file is considered a document by Lexos. You can activate,
de-activate, and re-label, and classify your documents using the
Manage tool.
- Character Encoding
Lexos will automatically convert all files to UTF-8 character encoding.
If you are uploading .html
, .xml
, or .sgml
files that contain special characters, the Scrubber tool
will help you to convert them to UTF-8 characters. If you are using raw
text files (.txt
), we recommend that you "Save-As" a copy of
your files with UTF-8 encoding (and Unix line breaks) prior to uploading to
Lexos (for example, using BBedit on MacOS, or NotePad++ on Windows).
Saving .lexos Workspace files
Lexos does not permanently save your files on our server; however, Lexos
allows you to save and download your workspace (files, options, etc.) for
later use. For example, perhaps you have uploaded 10 files and scrubbed
them with certain parameters. You can save your Workspace by clicking "Save"
in the upper right corner which will download a file with a .lexos
filename extension. Upon returning to Lexos, rather that re-uploading and
scrubbing your files again, you can Upload your .lexos
file.
Your working state will be restored for subsequent work.
The Lexos Web Scraper
At present, your documents must be available as files on your computer.
However, Lexos has a web scraper tool, which allows you to download files
off the internet. This is especially useful when you are using files from
sources such as
Project Gutenberg.
Enter the URL you wish to scrape in the provided field and click the "Scrape",
this will download all the HTML markup in a web page (the HTML tags can be
removed using the Scrubber tool). Upload times may vary, depending on internet
speeds. If the process seems to hang, try uploading fewer URLs. Large-scale
web scraping should not be done in Lexos.


This book contains the entire documentation for GitBook (platform and toolchain).
GitBook is a tool for building beautiful books using Git and Markdown. It can generate your book in multiple formats: PDF, ePub, mobi or as a website.
The GitBook toolchain is open source and completely free, the source code of the tool is available on GitHub.
Issues and question related to the format and the toolchain should be posted at github.com/GitbookIO/gitbook/issues
We're always happy to help out with your books or any other questions you might have. You can ask a question or signal an issue on the following contact form at gitbook.com/contact.
You can contribute to improve this documentation on GitHub.


AWS “IAM Assumed Roles” allows applications to change the AWS role with which to authenticate with AWS services. The assumed roles can have different rights from the main user login.
The S3A connector supports assumed roles for authentication with AWS. A full set of login credentials must be provided, which will be used to obtain the assumed role and refresh it regularly. By using per-filesystem configuration, it is possible to use different assumed roles for different buckets.
IAM Assumed Roles are unlikely to be supported by third-party systems supporting the S3 APIs.
This document assumes you know about IAM Assumed roles, what they are, how to configure their policies, etc.
Trying to learn how IAM Assumed Roles work by debugging stack traces from the S3A client is “suboptimal”.
The S3A connector support IAM Assumed Roles in two ways:
For Delegation Token integration, see (Delegation Tokens)[delegation_tokens.html]
To for Assumed Role authentication, the client must be configured to use the Assumed Role Credential Provider, org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider, in the configuration option fs.s3a.aws.credentials.provider.
This AWS Credential provider will read in the fs.s3a.assumed.role options needed to connect to the Security Token Service Assumed Role API, first authenticating with the full credentials, then assuming the specific role specified. It will then refresh this login at the configured rate of fs.s3a.assumed.role.session.duration
To authenticate with the AWS STS service both for the initial credential retrieval and for background refreshes, a different credential provider must be created, one which uses long-lived credentials (secret keys, environment variables). Short lived credentials (e.g other session tokens, EC2 instance credentials) cannot be used.
A list of providers can be set in s.s3a.assumed.role.credentials.provider; if unset the standard BasicAWSCredentialsProvider credential provider is used, which uses fs.s3a.access.key and fs.s3a.secret.key.
Note: although you can list other AWS credential providers in to the Assumed Role Credential Provider, it can only cause confusion.
To use assumed roles, the S3A client credentials provider must be set to the AssumedRoleCredentialProvider, and fs.s3a.assumed.role.arn to the previously created ARN.
<property> <name>fs.s3a.aws.credentials.provider</name> <value>org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider</value> </property> <property> <name>fs.s3a.assumed.role.arn</name> <value>arn:aws:iam::90066806600238:role/s3-restricted</value> </property>
The STS service itself needs the caller to be authenticated, which can only be done with a set of long-lived credentials. This means the normal fs.s3a.access.key and fs.s3a.secret.key pair, environment variables, or some other supplier of long-lived secrets.
The default is the fs.s3a.access.key and fs.s3a.secret.key pair. If you wish to use a different authentication mechanism, set it in the property fs.s3a.assumed.role.credentials.provider.
<property> <name>fs.s3a.assumed.role.credentials.provider</name> <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value> </property>
Requirements for long-lived credentials notwithstanding, this option takes the same values as fs.s3a.aws.credentials.provider.
The safest way to manage AWS secrets is via Hadoop Credential Providers.
Here are the full set of configuration options.
<property> <name>fs.s3a.assumed.role.arn</name> <value /> <description> AWS ARN for the role to be assumed. Required if the fs.s3a.aws.credentials.provider contains org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider </description> </property> <property> <name>fs.s3a.assumed.role.session.name</name> <value /> <description> Session name for the assumed role, must be valid characters according to the AWS APIs. Only used if AssumedRoleCredentialProvider is the AWS credential provider. If not set, one is generated from the current Hadoop/Kerberos username. </description> </property> <property> <name>fs.s3a.assumed.role.policy</name> <value/> <description> JSON policy to apply to the role. Only used if AssumedRoleCredentialProvider is the AWS credential provider. </description> </property> <property> <name>fs.s3a.assumed.role.session.duration</name> <value>30m</value> <description> Duration of assumed roles before a refresh is attempted. Only used if AssumedRoleCredentialProvider is the AWS credential provider. Range: 15m to 1h </description> </property> <property> <name>fs.s3a.assumed.role.sts.endpoint</name> <value/> <description> AWS Security Token Service Endpoint. If unset, uses the default endpoint. Only used if AssumedRoleCredentialProvider is the AWS credential provider. </description> </property> <property> <name>fs.s3a.assumed.role.sts.endpoint.region</name> <value>us-west-1</value> <description> AWS Security Token Service Endpoint's region; Needed if fs.s3a.assumed.role.sts.endpoint points to an endpoint other than the default one and the v4 signature is used. Only used if AssumedRoleCredentialProvider is the AWS credential provider. </description> </property> <property> <name>fs.s3a.assumed.role.credentials.provider</name> <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, com.amazonaws.auth.EnvironmentVariableCredentialsProvider </value> <description> List of credential providers to authenticate with the STS endpoint and retrieve short-lived role credentials. Used by AssumedRoleCredentialProvider and the S3A Session Delegation Token and S3A Role Delegation Token bindings. </description> </property>
The S3A client needs to be granted specific permissions in order to work with a bucket. Here is a non-normative list of the permissions which must be granted for FileSystem operations to work.
Disclaimer The specific set of actions which the S3A connector needs will change over time.
As more operations are added to the S3A connector, and as the means by which existing operations are implemented change, the AWS actions which are required by the client will change.
These lists represent the minimum actions to which the client’s principal must have in order to work with a bucket.
Permissions which must be granted when reading from a bucket:
s3:Get* s3:ListBucket
When using S3Guard, the client needs the appropriate DynamoDB access permissions
To use SSE-KMS encryption, the client needs the SSE-KMS Permissions to access the KMS key(s).
These permissions must all be granted for write access:
s3:Get* s3:Delete* s3:Put* s3:ListBucket s3:ListBucketMultipartUploads s3:AbortMultipartUpload
When to read data encrypted using SSE-KMS, the client must have kms:Decrypt permission for the specific key a file was encrypted with.
kms:Decrypt
To write data using SSE-KMS, the client must have all the following permissions.
kms:Decrypt kms:GenerateDataKey
This includes renaming: renamed files are encrypted with the encryption key of the current S3A client; it must decrypt the source file first.
If the caller doesn’t have these permissions, the operation will fail with an AccessDeniedException: the S3 Store does not provide the specifics of the cause of the failure.
To use S3Guard, all clients must have a subset of the AWS DynamoDB Permissions.
To work with buckets protected with S3Guard, the client must have all the following rights on the DynamoDB Table used to protect that bucket.
dynamodb:BatchGetItem dynamodb:BatchWriteItem dynamodb:DeleteItem dynamodb:DescribeTable dynamodb:GetItem dynamodb:PutItem dynamodb:Query dynamodb:UpdateItem
This is true, even if the client only has read access to the data.
For the hadoop s3guard table management commands, extra permissions are required:
dynamodb:CreateTable dynamodb:DescribeLimits dynamodb:DeleteTable dynamodb:Scan dynamodb:TagResource dynamodb:UntagResource dynamodb:UpdateTable
Without these permissions, tables cannot be created, destroyed or have their IO capacity changed through the s3guard set-capacity call. The dynamodb:Scan permission is needed for s3guard prune
The dynamodb:CreateTable permission is needed by a client when it tries to create the DynamoDB table on startup, that is fs.s3a.s3guard.ddb.table.create is true and the table does not already exist.
Mixing permissions down the “directory tree” is limited only to the extent of supporting writeable directories under read-only parent paths.
Disclaimer: When a client lacks write access up the entire directory tree, there are no guarantees of consistent filesystem views or operations.
Particular troublespots are “directory markers” and failures of non-atomic operations, particularly rename() and delete().
A directory marker such as /users/ will not be deleted if the user alice creates a directory /users/alice and she only has access to /users/alice.
When a path or directory is deleted, the parent directory may not exist afterwards. In the example above, if alice deletes /users/alice and there are no other entries under /users/alice, then the directory marker /users/ cannot be created. The directory /users will not exist in listings, getFileStatus("/users") or similar.
Rename will fail if it cannot delete the items it has just copied, that is rename(read-only-source, writeable-dest) will fail —but only after performing the COPY of the data. Even though the operation failed, for a single file copy, the destination file will exist. For a directory copy, only a partial copy of the source data may take place before the permission failure is raised.
S3Guard: if S3Guard is used to manage the directory listings, then after partial failures of rename/copy the DynamoDB tables can get out of sync.
This example has the base bucket read only, and a directory underneath, /users/alice/ granted full R/W access.
{ "Version" : "2012-10-17", "Statement" : [ { "Sid" : "4", "Effect" : "Allow", "Action" : [ "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:Get*" ], "Resource" : "arn:aws:s3:::example-bucket/*" }, { "Sid" : "5", "Effect" : "Allow", "Action" : [ "s3:Get*", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts" ], "Resource" : [ "arn:aws:s3:::example-bucket/users/alice/*", "arn:aws:s3:::example-bucket/users/alice", "arn:aws:s3:::example-bucket/users/alice/" ] } ] }
Note how three resources are provided to represent the path /users/alice
| Path | Matches |
|---|---|
| /users/alice | Any file alice created under /users |
| /users/alice/ | The directory marker alice/ created under /users |
| /users/alice/* | All files and directories under the path /users/alice |
Note that the resource arn:aws:s3:::example-bucket/users/alice* cannot be used to refer to all of these paths, because it would also cover adjacent paths like /users/alice2 and /users/alicebob.
The Assumed Role Credential Provider is enabled, but fs.s3a.assumed.role.arn is unset.
java.io.IOException: Unset property fs.s3a.assumed.role.arn at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.<init>(AssumedRoleCredentialProvider.java:76) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520) at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)
This can arise if the role ARN set in fs.s3a.assumed.role.arn is invalid or one to which the caller has no access.
java.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Not authorized to perform sts:AssumeRole (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: aad4e59a-f4b0-11e7-8c78-f36aaa9457f6):AccessDenied at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520) at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
You can’t assume a role with the root account of an AWS account; you need to create a new user and give it the permission to change into the role.
java.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2): No AWS Credentials provided by AssumedRoleCredentialProvider : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520) at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) ... 22 more Caused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
The value of fs.s3a.assumed.role.session.duration is too low.
org.apache.hadoop.fs.s3a.AWSBadRequestException: request role credentials: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 1 validation error detected: Value '20' at 'durationSeconds' failed to satisfy constraint: Member must have value greater than or equal to 900 (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError; Request ID: b9a82403-d0a7-11e8-98ef-596679ee890d)
Fix: increase.
The value of fs.s3a.assumed.role.session.duration is too high.
org.apache.hadoop.fs.s3a.AWSBadRequestException: request role credentials: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: The requested DurationSeconds exceeds the MaxSessionDuration set for this role. (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError; Request ID: 17875165-d0a7-11e8-b85f-d15a599a7f6d)
There are two solutions to this
Irrespective of the maximum duration of a role, the AWS role API only permits callers to request any role for up to 12h; attempting to use a larger number will fail.
Caused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 1 validation error detected: Value '345600' at 'durationSeconds' failed to satisfy constraint: Member must have value less than or equal to 43200 (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError; Request ID: dec1ca6b-d0aa-11e8-ac8c-4119b3ea9f7f)
For full sessions, the duration limit is 129600 seconds: 36h.
org.apache.hadoop.fs.s3a.AWSBadRequestException: request session credentials: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 1 validation error detected: Value '345600' at 'durationSeconds' failed to satisfy constraint: Member must have value less than or equal to 129600 (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError; Request ID: a6e73d44-d0aa-11e8-95ed-c5bba29f0635)
For both these errors, the sole fix is to request a shorter duration in fs.s3a.assumed.role.session.duration.
The policy set in fs.s3a.assumed.role.policy is not valid according to the AWS specification of Role Policies.
org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on : com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException: The policy is not in the valid JSON format. (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c): MalformedPolicyDocument: The policy is not in the valid JSON format. (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520) at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) Caused by: com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException: The policy is not in the valid JSON format. (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)
The policy set in fs.s3a.assumed.role.policy is not valid JSON.
org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on : com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException: Syntax errors in policy. (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45): MalformedPolicyDocument: Syntax errors in policy. (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520) at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055) ... 19 more
You can’t use the Assumed Role Credential Provider as the provider in fs.s3a.assumed.role.credentials.provider.
java.io.IOException: AssumedRoleCredentialProvider cannot be in fs.s3a.assumed.role.credentials.provider at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.<init>(AssumedRoleCredentialProvider.java:86) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520) at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
There’s an space or other typo in the fs.s3a.access.key or fs.s3a.secret.key values used for the inner authentication which is breaking signature creation.
org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign) in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date. (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature; Request ID: c4a8841d-f556-11e7-99f9-af005a829416):IncompleteSignature: 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign) in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date, (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature; at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520) at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) Caused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign) in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date, (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature; at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)
The credentials used to authenticate with the AWS Security Token Service are invalid.
[ERROR] Failures: [ERROR] java.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: The security token included in the request is invalid. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: 74aa7f8a-f557-11e7-850c-33d05b3658d7):InvalidClientTokenId at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520) at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474) Caused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: The security token included in the request is invalid. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: 74aa7f8a-f557-11e7-850c-33d05b3658d7) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055) ... 25 more
The session name, as set in fs.s3a.assumed.role.session.name must match the wildcard [\w+=,.@-]*.
If the property is unset, it is extracted from the current username and then sanitized to match these constraints. If set explicitly, it must be valid.
org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\w+=,.@-]* (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError; Request ID: 7c437acb-f55d-11e7-9ad8-3b5e4f701c20):ValidationError: 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\w+=,.@-]* (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError; at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616) at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520) at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) Caused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\w+=,.@-]* (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError; at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)
If an operation fails with an AccessDeniedException, then the role does not have the permission for the S3 Operation invoked during the call.
java.nio.file.AccessDeniedException: s3a://bucket/readonlyDir: rename(s3a://bucket/readonlyDir, s3a://bucket/renameDest) on s3a://bucket/readonlyDir: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2805F2ABF5246BB1; S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=), S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=:AccessDenied at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:216) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:143) at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:853) ... Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2805F2ABF5246BB1; S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=), S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw= at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)
This is the policy restriction behaving as intended: the caller is trying to perform an action which is forbidden.
If a policy has been set in fs.s3a.assumed.role.policy then it must declare all permissions which the caller is allowed to perform. The existing role policies act as an outer constraint on what the caller can perform, but are not inherited.
If the policy for a bucket is set up with complex rules on different paths, check the path for the operation.
The policy may have omitted one or more actions which are required. Make sure that all the read and write permissions are allowed for any bucket/path to which data is being written to, and read permissions for all buckets read from.
If the bucket is using SSE-KMS to encrypt data:
Without permissions, the request fails and there is no explicit message indicating that this is an encryption-key issue.
This problem is most obvious when you fail when writing data in a “Writing Object” operation.
If the client does have write access to the bucket, verify that the caller has kms:GenerateDataKey permissions for the encryption key in use.
java.nio.file.AccessDeniedException: test/testDTFileSystemClient: Writing Object on test/testDTFileSystemClient: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: E86544FF1D029857) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:243) at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111) at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:314) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:406) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:310) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:285) at org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:150) at org.apache.hadoop.fs.s3a.WriteOperationHelper.putObject(WriteOperationHelper.java:460) at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.lambda$putObject$0(S3ABlockOutputStream.java:438) at org.apache.hadoop.util.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:219) at org.apache.hadoop.util.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:219) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: E86544FF1D029857)
Note: the ability to read encrypted data in the store does not guarantee that the caller can encrypt new data. It is a separate permission.
java.nio.file.AccessDeniedException: bucket1: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::980678866538:assumed-role/s3guard-test-role/test is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-1:980678866538:table/bucket1 (Service: AmazonDynamoDBv2; Status Code: 400;
The caller is trying to access an S3 bucket which uses S3Guard, but the caller lacks the relevant DynamoDB access permissions.
The dynamodb:DescribeTable operation is the first one used in S3Guard to access, the DynamoDB table, so it is often the first to fail. It can be a sign that the role has no permissions at all to access the table named in the exception, or just that this specific permission has been omitted.
If the role policy requested for the assumed role didn’t ask for any DynamoDB permissions, this is where all attempts to work with a S3Guarded bucket will fail. Check the value of fs.s3a.assumed.role.policy
This is a low-level networking error. Possible causes include:
org.apache.hadoop.fs.s3a.AWSClientIOException: request session credentials: com.amazonaws.SdkClientException: Unable to execute HTTP request: null: Unable to execute HTTP request: null at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:88) at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:124) at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:183) at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
This is based on conflict between the values of fs.s3a.assumed.role.sts.endpoint and fs.s3a.assumed.role.sts.endpoint.region Two variants, “not '''”
Variant 1: Credential should be scoped to a valid region, not 'us-west-1' (or other string)
java.nio.file.AccessDeniedException: : request session credentials: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Credential should be scoped to a valid region, not 'us-west-1'. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: d9065cc4-e2b9-11e8-8b7b-f35cb8d7aea4):SignatureDoesNotMatch
One of:
Variant 2: Credential should be scoped to a valid region, not ''
java.nio.file.AccessDeniedException: : request session credentials: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Credential should be scoped to a valid region, not ''. ( Service: AWSSecurityTokenService; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: bd3e5121-e2ac-11e8-a566-c1a4d66b6a16):SignatureDoesNotMatch
This should be intercepted earlier: an endpoint has been specified but not a region.
There’s special handling for the central sts.amazonaws.com region; when that is declared as the value of fs.s3a.assumed.role.sts.endpoint.region then there is no need to declare a region: whatever value it has is ignored.


"I would say if you have a dream, follow it. It doesn't really matter whether you are a woman or from India or from wherever."


Recipe Unlimited Corp. sales and earnings fall in third quarter
by The Canadian PressVAUGHAN, Ont. -- Recipe Unlimited Corp. saw its sales slip in its most recent quarter amid challenging industry conditions.
The Vaughan, Ont.-based company says total system sales fell $10.7 million to $869.1 million for the third-quarter ended Sept. 29, compared with $879.8 million in the same quarter the previous year.
The company formerly known as Cara Operations says same-restaurant sales, a key retail metric, fell 3.1 per cent in the quarter.
Recipe says net earnings decreased to $6.7 million for the quarter, compared with $23.8 million in the third quarter the previous year.
Adjusted net earnings came in at $19.5 million or 31 cents per diluted share, equal to forecasts according to financial markets data firm Refinitiv. That compared with $25.3 million or 39 cents per diluted share for the same quarter in 2018.
Recipe owns 19 brands, including Swiss Chalet, Harvey's, the Keg and New York Fries.
This report by The Canadian Press was first published Nov. 12, 2019.


|
SDLXX
1.0.0
Modern C++17 wrapper for SDL
|
Class that represents SDLXX object that initializes the video API. More...
#include <SDLXX_core.h>
Public Types | |
| enum | Subsystem : uint32_t { Subsystem::TIMER = 0x00000001u, Subsystem::AUDIO = 0x00000010u, Subsystem::VIDEO = 0x00000020u, Subsystem::JOYSTICK = 0x00000200u, Subsystem::HAPTIC = 0x00001000u, Subsystem::GAMECONTROLLER = 0x00002000u, Subsystem::EVENTS = 0x00004000u, Subsystem::SENSOR = 0x00008000u, Subsystem::EVERYTHING } |
| An enumeration of library subsystems. More... | |
| enum | HintPriority { HintPriority::DEFAULT, HintPriority::NORMAL, HintPriority::OVERRIDE } |
| An enumeration of hint priorities. More... | |
Public Member Functions | |
| SDLXX_core (const std::unordered_set< Subsystem > &subsystems={}) | |
| Construct the SDLXX_core object that initializes the specified Subsystems of the library. More... | |
| ~SDLXX_core () | |
| Destroy the SDLXX_core object cleaning up all initialized subsystems. More... | |
| void | initSubsystem (const Subsystem &subsystem) |
| Initialize specific SDL subsystem. More... | |
| void | initSubsystem (const std::unordered_set< Subsystem > &subsystems) |
| Initialize specific SDL subsystems. More... | |
| void | quitSubsystem (const Subsystem &subsystem) |
| Clean up specific SDL subsystem. More... | |
| void | quitSubsystem (const std::unordered_set< Subsystem > &subsystems) |
| Clean up specific SDL subsystems. More... | |
| std::unordered_set< Subsystem > | wasInit (const std::unordered_set< Subsystem > &subsystems={}) const |
| Return a subset of specified subsystems which have previously been initialized, or a set of subsystems, if no subsystems were specified. More... | |
| bool | wasInit (const Subsystem &subsystem) const |
| Return true if a specified subsystem has previously been initialized. More... | |
Static Public Member Functions | |
| static Version | getCompiledSdlVersion () |
| Get the SDL version the library was compiled against. More... | |
| static Version | getLinkedSdlVersion () |
| Get the version of SDL that is linked against the library. More... | |
| static bool | setHint (const std::string &name, const std::string &value, const HintPriority &priority=HintPriority::NORMAL) |
| Set a hint with a specific priority. If priority is not specified, normal priority will be used. More... | |
| static std::optional< std::string > | getHint (const std::string &name) |
| Get a hint value. More... | |
| static std::optional< bool > | getHintBoolean (const std::string &name) |
| Get a hint as a boolean value. More... | |
| static void | clearHints () |
| Clear all hints. More... | |
Class that represents SDLXX object that initializes the video API.
|
strong |
|
strong |
An enumeration of library subsystems.
| Enumerator | |
|---|---|
| TIMER | Timer subsystem. |
| AUDIO | Audio subsystem. |
| VIDEO | Video subsystem; automatically initializes the events subsystem. |
| JOYSTICK | Joystick subsystem; automatically initializes the events subsystem. |
| HAPTIC | Haptic (force feedback) subsystem. |
| GAMECONTROLLER | Controller subsystem; automatically initializes the joystick subsystem. |
| EVENTS | Events subsystem. |
| SENSOR | Sensor subsystem. |
| EVERYTHING | All subsystems. |
|
explicit |
Construct the SDLXX_core object that initializes the specified Subsystems of the library.
This object must be constructed before using most other library functions.
| subsystems | Subsystems that should be initialized |
| sdlxx::core::SDLXX_core::~SDLXX_core | ( | ) |
Destroy the SDLXX_core object cleaning up all initialized subsystems.
|
static |
Clear all hints.
This function is called during destruction of SDLXX core object to free stored hints.
|
static |
Get the SDL version the library was compiled against.
This is determined by what header the compiler used. Note that if you dynamically linked the library, you might have a slightly newer or older version at runtime. That version can be determined with SDLXX_core::getLinkedSdlVersion()
|
static |
Get a hint value.
| name | The name of the hint |
|
static |
Get a hint as a boolean value.
| name | The name of the hint |
|
static |
| void sdlxx::core::SDLXX_core::initSubsystem | ( | const Subsystem & | subsystem | ) |
Initialize specific SDL subsystem.
Subsystem initialization is ref-counted, you must call SDLXX_core::quitSubsystem() for each SDLXX_core::initSubsystem() to correctly shutdown a subsystem manually (or destroy SDLXX object to force shutdown). If a subsystem is already loaded then this call will increase the ref-count and return.
| void sdlxx::core::SDLXX_core::initSubsystem | ( | const std::unordered_set< Subsystem > & | subsystems | ) |
Initialize specific SDL subsystems.
Subsystem initialization is ref-counted, you must call SDLXX_core::quitSubsystem() for each SDLXX_core::initSubsystem() to correctly shutdown a subsystem manually (or destroy SDLXX object to force shutdown). If a subsystem is already loaded then this call will increase the ref-count and return.
| void sdlxx::core::SDLXX_core::quitSubsystem | ( | const Subsystem & | subsystem | ) |
Clean up specific SDL subsystem.
Subsystem initialization is ref-counted, you must call SDLXX_core::quitSubsystem() for each SDLXX_core::initSubsystem() to correctly shutdown a subsystem manually (or destroy SDLXX object to force shutdown). If a subsystem is loaded multiple times then this call will decrease the ref-count and return.
If you start a subsystem using a call to that subsystem's init function instead of constructing SDLXX core object or calling SDLXX_core::initSubsystem(), SDLXX_core::quitSubsystem() and SDLXX_core::wasInit() will not work. You will need to use that subsystem's quit function directly instead.
| void sdlxx::core::SDLXX_core::quitSubsystem | ( | const std::unordered_set< Subsystem > & | subsystems | ) |
Clean up specific SDL subsystems.
Subsystem initialization is ref-counted, you must call SDLXX_core::quitSubsystem() for each SDLXX_core::initSubsystem() to correctly shutdown a subsystem manually (or destroy SDLXX object to force shutdown). If a subsystem is loaded multiple times then this call will decrease the ref-count and return.
If you start a subsystem using a call to that subsystem's init function instead of constructing SDLXX core object or calling SDLXX_core::initSubsystem(), SDLXX_core::quitSubsystem() and SDLXX_core::wasInit() will not work. You will need to use that subsystem's quit function directly instead.
|
static |
Set a hint with a specific priority. If priority is not specified, normal priority will be used.
The priority controls the behavior when setting a hint that already has a value. Hints will replace existing hints of their priority and lower. Environment variables are considered to have override priority.
| name | The name of the hint |
| value | The value of the hint |
| priority | The priority of the hint or SDLXX_core::HintPriority::NORMAL |
| std::unordered_set<Subsystem> sdlxx::core::SDLXX_core::wasInit | ( | const std::unordered_set< Subsystem > & | subsystems = {} | ) | const |
Return a subset of specified subsystems which have previously been initialized, or a set of subsystems, if no subsystems were specified.
If you start a subsystem using a call to that subsystem's init function instead of constructing SDLXX core object or calling SDLXX_core::initSubsystem(), SDLXX_core::quitSubsystem() and SDLXX_core::wasInit() will not work. You will need to use that subsystem's quit function directly instead.
| bool sdlxx::core::SDLXX_core::wasInit | ( | const Subsystem & | subsystem | ) | const |
Return true if a specified subsystem has previously been initialized.
If you start a subsystem using a call to that subsystem's init function instead of constructing SDLXX core object or calling SDLXX_core::initSubsystem(), SDLXX_core::quitSubsystem() and SDLXX_core::wasInit() will not work. You will need to use that subsystem's quit function directly instead.


The Doxygen package contains a documentation system for C++, C, Java, Objective-C, Corba IDL and to some extent PHP, C# and D. It is useful for generating HTML documentation and/or an off-line reference manual from a set of documented source files. There is also support for generating output in RTF, PostScript, hyperlinked PDF, compressed HTML, and Unix man pages. The documentation is extracted directly from the sources, which makes it much easier to keep the documentation consistent with the source code.
You can also configure Doxygen to extract the code structure from undocumented source files. This is very useful to quickly find your way in large source distributions. Used along with Graphviz, you can also visualize the relations between the various elements by means of include dependency graphs, inheritance diagrams, and collaboration diagrams, which are all generated automatically.
This package is known to build and work properly using an LFS-7.4 platform.
Download (HTTP): http://ftp.stack.nl/pub/doxygen/doxygen-1.8.5.src.tar.gz
Download (FTP): ftp://ftp.stack.nl/pub/doxygen/doxygen-1.8.5.src.tar.gz
Download MD5 sum: db51274568755e2c75c2657e30a78a55
Download size: 6.2 MB
Estimated disk space required: 81 MB
Estimated build time: 1.2 SBU (includes building the GUI frontend)
Graphviz-2.34.0, ghostscript-9.10, Python-2.7.5, Qt-4.8.5 (for doxywizard) and texlive-20130530
User Notes: http://wiki.linuxfromscratch.org/blfs/wiki/doxygen
Install Doxygen by running the following commands:
./configure --prefix /usr \ --docdir /usr/share/doc/doxygen-1.8.5 && make
This package does not come with a test suite.
Now, as the root
user:
make install
If you wish to generate and install the package documentation (note
that man pages have already been installed), you must have
Python, TeX Live (for HTML docs) and Ghostscript (for PDF docs) installed, then
issue the following command as the root
user:
make install_docs
--with-doxywizard
: Use this parameter
if Qt is installed and you wish to
build the GUI front-end.
There is no real configuration necessary for the Doxygen package although three additional packages are required if you wish to use extended capabilities. If you need to use the language translation features, you must have Python-2.7.5 installed. If you require formulas to create PDF documentation, then you must have texlive-20130530 installed. If you require formulas to convert PostScript files to bitmaps, then you must have ghostscript-9.10 installed.
Last updated on 2013-08-24 15:00:43 -0700


Figure 1:General process of fractional distillation
Fractional distillation can be used to separate two or more liquids. For example, ethanol (alcohol) can be separated from a mixture of ethanol and water because the two liquids have different boiling points.
Figure 2:Fractional distillation of ethanol and water
Separating two or more liquids
Fractional distillation is used to separate crude oil into useful liquids that have different boiling points. Petrol and diesel are useful fractions of crude oil.
Figure 3:Fractional distillation of crude oil


FIC: Fast Immutable Collections, for Dart
Click here to check out FIC on Github
Click here for Marcelo Glasberg's — the author of most of this package's implementation — article.
Have you ever wondered if there was a way of getting rid of the plague of mutability in OOP and imperative languages? Or maybe restrict it? If you did, you're not alone.
Managing many objects each inside their respective architectural layers can be an even bigger challenge if they keep unexpectedly changing throughout their respective lifetimes. Joshua Bloch even goes as far as to state:
Classes should be immutable unless there’s a very good reason to make them mutable.
Interestingly, if you're a Java programmer, one of the main raisons d'êtres of the language itself is to abstract away the use of pointers, which is a lower level manifestation of mutability. James Gosling, the main developer/creator of Java, deliberately set out to make it very difficult for people to deal with pointers and direct memory allocation because they were the #1 source of bugs in lower level languages like C — that's also the reason for garbage-collection's existence.
If you wanna go down the rabbit hole... this is such a major topic there are those who have dedicated decades of their careers to completely avoiding or bypassing state.
Pure functional programming languages try to respect mathematical functions, which, by definition, need to be deterministic, meaning, if you input something, the output needs to always be the same.
But how can anything be useful in the real world without any state? Well, it's a bit complicated and counter-intuitive, but there's a solution for abstracting state and still remaining pure, namely, Monads.
If you're interested, I suggest checking out Haskell, Coq
, Clojure or Elm, though there's an endless sea of — largely unknown — programming languages in this area.
Dart doesn't feature native immutable collections==
) defaulting to
reference equality (identity
). In the end, they are
actually more or less like sealed pointers to memory locations. This
means that doing something like
Person("Bob") == Person("Bob")
will yield
false
, since those objects are not the same instance or
memory location. However you can override this behavior, despite it
being annoying boilerplate
You can find the whole code used in this article, or, alternatively, inside this gist. You can also test it out live through DartPad, however it uses some packages which are not available on DartPad.
@immutable
class Person {
final String name;
const Person(this.name);
@override
bool operator ==(Object other) =>
identical(this, other) ||
other is Person &&
runtimeType == other.runtimeType &&
name == other.name;
@override
int get hashCode => name.hashCode;
}
Overriding equals (==
) and hashCode
In your next layer of abstraction, it is very likely you might end up
with a collection of these simple objects, like a
set, list or map of Person
.
Once you get there, you might want to deploy the same tactic you used
for layer one, meaning the collections themselves will be immutable.
If you don't, changes to the objects in the collections might happen
without your knowledge, because there will be no programmatic way of
detecting them — static analysis, for instance, won't point it
out it because it is a valid feature.
Now that you've hopefully decided to also make your collection objects
immutable, you're gonna have to solve the problem of internally
dealing with
mutable operations on top of mutable collections, such as
Dart's List
, Set
and Map
.
Eventually, you might come up with the typical pattern of creating a
private constructor to help manage the endless and dangerous
collection copies you're gonna have to manage inside the object's
state:
@immutable
class People {
final List<Person> _people;
People(List<Person> people): _people = List.of(people);
/// Since there are no references to the list once it is inside
/// the object, there's no need for copying.
People._(this._people);
// An example of immutable operaton on a mutable collection
// (List). Now imagine dozens of these operations, each needing
// extra care from you in order to not cause any side effects.
People add(Person person) {
var newPeopleList = List.of(_people)..add(person);
return People._(newPeopleList);
}
// If you want to create value equality for this class, you will
// need to override reference equality. The simplest way is to
// do this is to use the `collection` package.
@override
bool operator ==(Object other) =>
identical(this, other) ||
other is People &&
runtimeType == other.runtimeType &&
const ListEquality().equals(_people, other._people);
@override
int get hashCode => const ListEquality().hash(_people);
@override
String toString() => _people.toString();
// Optionally, you could also write your own list equality.
// Something close to this:
bool _listEquals<T>(List<T> a, List<T> b) {
if (a == null)
return b == null;
if (b == null || a.length != b.length)
return false;
if (identical(a, b))
return true;
// Don't write this recursively, you might end up with
// space leaks. For more info on this, check out
// Tail Call Optimization (TCO).
for (int index = 0; index < a.length; index += 1) {
if (a[index] != b[index]) return false;
}
return true;
}
}
Pay close attention! It's very important that all of the items in the supposedly immutable collection are all immutable. You absolutely cannot have items changing state in a supposedly immutable collection, all the more because static analysis will not typically warn you at any point about any of the changes.
The People._
constructor avoids wasteful recopying of the
_people
list through the public constructor, since you've
already copied it inside the "mutation" method
(People.add
). But this type of pattern is clunky and
needs a lot of boilerplate. Not to mention that, even though, from the
outside, programmers can't change the inner list, whoever is working
inside the object still has a ton of room for mistakes.
This situation is where FIC shines. Its collections are already programmatically sealed for mutable operations. There is no room for the programmer to mutate its collections, and all that while minimizing boilerplate and arguably vastly simplifying the code:
@immutable
class PeopleWithFic {
final IList<Person> people;
PeopleWithFic(this.people);
PeopleWithFic add(Person person) =>
PeopleWithFic(people.add(person));
@override
bool operator ==(Object other) =>
identical(this, other) ||
other is People &&
runtimeType == other.runtimeType &&
people == other.people;
@override
int get hashCode => people.hashCode;
}
FIC uses deep equality by default, which makes dealing with
collections of values much more natural and concise. And now you can
treat the whole inner list as a value,
people == other.people
actually works safely and as most
people would expect. The whole list is immutable and locked so, if you
wish, you can now expose the people
attribute/property
publicly.
The native List
in Dart features a handy
constructor/factoryunmodifiable
which seemingly does exactly
everything you need for basic immutability. So why on Earth would
anyone go on as big a rant as we did?
The 2 main reasons for not being satisfied are the insufficiency and clunkiness of that constructor/factory.
List.unmodifiable
will offer immutability or at least
level 1 immutability, which is actually what FIC offers as well. Take
a brief look at
// 1) This is ok. Immutability does work in this case.
// (This is covered in the `List.unmodifiable` docs.)
print("1) Direct Immutability\n\n");
var list = [1, 2, 3];
var unmodifiableList1 = List.unmodifiable(list);
list.add(4);
print("Original List: $list"); // [1, 2, 3]
print("Unmodifiable List 1: $unmodifiableList1"); // Still [1, 2, 3]
print("\n\n---------------------------------------------------------\n\n");
// 2) This is not ok. Immutability doens't work because the nested elements
// are not themselves immutable.
// (This is also covered in the `List.unmodifiable` docs.)
print("2) Nested Immutability\n\n");
var list1 = [1, 2];
var list2 = [1, 2, 3];
var nestedList = [list1, list2];
var unmodifiableList2 = List.unmodifiable(nestedList);
list1.add(10);
print("Original List 1: $list1"); // [1, 2, 10]
print("Original List 2: $list2"); // [1, 2, 3]
print("UnmodifiableList 2: $unmodifiableList2"); // [[1, 2, 10], [1, 2, 3]]
All of that seems enough for most people but the real problem starts when you actually want to apply changes to the collections. With that unmodifiable interface, you can only resort to copying it on the outside and then, once you're done with the dangerous mutable operations, retransform the result into a new unmodifiable object.
That's very problematic because you're copying stuff, which is very
error prone and slow, and the process itself isn't really as clean as
using an API that's made for this type of operation. Besides all that,
using operations such as add
on the unmodifiable object
will only get flagged during runtime, which will make catching bugs
more difficult.
Take a look at the difference:
// 3) How would you modify, add, remove, etc. this
// `List.unmodifiable` object?
// FIC makes it much simpler and less error prone...
// And copying things to make changes will make `List.unmodifiable`
// unbearably slow.
print("3) Clunky modifications natively | FIC makes it easy!\n\n");
var list3 = [1, 2, 3];
var unmodifiableList3 = List.unmodifiable(list3);
var ilist3 = list3.lock;
var newUnmodifiableList3 =
List.unmodifiable(unmodifiableList3.toList()..add(4));
var newIlist3 = ilist3.add(4);
print("New UnmodifiableList 3: $newUnmodifiableList3");
print("New IList 3: $newIlist3");
Both built_collection and kt.dart are excellent packages, you could use either of them instead of FIC and live happily ever after. Nonetheless, when it comes to the end user, the programmer, FIC offers some valuable contributions/improvements:
- The built_collection package is based on the Builder Design Pattern, which helps in making it very fast, since operations on collections will only complete when you "rebuild" the collection, almost like lazy collections. However, going through the builder pattern as an API leads to somewhat cumbersome code, and you have to remember when to rebuild your collections. FIC offers a much more natural API with near-equal performance, something we will discuss in the next section.
- On the other side of the spectrum, kt.dart brings the easier Kotlin Collections API while making some sacrifices in performance, since it mostly copies the collection on every "mutable" operation. FIC brings in the regular, intuitive OOP collection API and plenty of functional operations, while avoiding copies and preserving performance.
FIC's whole implementation is complicated. There are a lot of details. But one of those details is worth knowing even for those who don't want to have anything to do with developing this package.
The major detail is that FIC doesn't actually copy
everything whenever the user attempts a mutable operation.
For example, when adding elements to a list, FIC will sort of "create
an internal variable" that points to those elements. This is similar
to the structure of data in many functional programming languages: in
the end it becomes something like a (recursive) tree or a
trie
If the data you're trying to add is a single item or an immutable
collection, FIC will create a recursive branch in its structure. So,
in the end, if you're adding a huge immutable IList
to
another IList
, the change will be virtually immediate,
much faster than the native List
s actually. If you add a
regular List
to an IList
, then FIC will need
to make an immutable, safe copy though, that's why we recommend that,
once you decide to go for FIC, we recommend sticking to it throughout
the whole process, algorithm or application.
FIC features two other inner packages: benchmark
and
benchmark_app
— they are now inside the
example
folder. The benchmark app package uses the
benchmark package to run many of the typical collection operations and
compare the different collection packages to each other.
Overall, FIC performs very well. In most cases, it is on par with the
native collections, and sometimes it even beats them, as in the case
of IList.addAll
. But sometimes it is indeed slower than
expected, even though not by that much — some operations on maps
and sets did not end up being so optimal within FIC's design context.
IList
, an immutable listISet
, an immutable setIMap
, an immutable map-
IMapOfSets
, an immutable map of sets (a multimap) -
Lock and unlock extensions, so you can easily transform mutable
collections into immutable ones, and vice-versa. For example:
[1, 2].lock
. - Global and local configurations that alter how your immutable collections behave with respect to equality, sorting, caching, and flushing.
- Optional deep equalities and cached hash codes, which let you treat your collections as value-objects.
- Mixins for you to build your own immutable collections or objects.
- View wrappers, so that you can work with immutable collection as if they were mutable.
-
ListSet
, a very efficient fixed-size mutable collection which is at the same time an orderedSet
and aList
. -
ListMap
is a mutable, fixed-sized, and ordered map which has some very efficientList
methods, likesort
andshuffle
, and lets you efficiently read its information by index. -
General purpose extensions to the native collections:
List
,Set
,Map
. See classesFicListExtension
,FicSetExtension
, andFicMapExtension
. For example:[1, 2, 1].removeDuplicates()
-
Other extensions:
IterableExtension
,IteratorExtension
,MapIteratorExtension
,FicComparatorExtension
,FicComparableExtension
andFicBooleanExtension
. For example:false.compareTo(true)
-
Comparators and related helpers to be used with native or immutable collections, or any sort algorithms, such as
compareObject
,sortBy
,sortLike
,if0
, insort.dart
.
Developing FIC took way longer than both I and
Marcelo Glasberg
expected. Marcelo is a much more experienced and savvy programmer and
the one responsible for 90% of the design and implementation. My work
was about 90% testing and trying to poke holes into his code to find
bugs
So far, since publishing this package on pub.dev, we've found very few problems. And Marcelo and I have been using it internally in his own company, so we believe FIC is ready for whatever its use may be. But you never know... if you ever face any bug, please do us and the community a favor and report them here.
Regardless of each person's attributions, we both had to research and read a ton of articles to complete this project. I took the time of cataloguing all of these resources in the bibliography section of the documentation, but some of the other final sections also feature these resources indirectly.
If we ever come back to redesign this project or create something
similar, we plan on diving deeper into
Hash Array Mapped Tries (HAMT), which is a very important data structure for functional programming
languages' implementations — you can watch Rich Hickey, the
author of Clojure, explain a little bit a about this data structure
here
Follow the discussion on Reddit if you wish.


{% extends "base.html" %} {% block content %} {% include "include_search.html" %}
The most recent changes to users
The most recent changes to aliases


@(from: String, to: String, count: Int)
@import helper._
@* Call main template with title and html to display *@
@main("Transfer Delete Confirmation"){
}{
@if(count==1){
Sure you want to delete transfer from @from to @to?
}else{
Sure you want to delete @count transfers from @from to @to?