3 radical Labour conference policies that got watered down in election manifesto
Labour launched their manifesto in Birmingham unveiling what they called a programme for "real change". The extensive document promises a plan to nationalise energy, water, rail and broadband but some of the radical plans supported by the membership have been watered down.
by Nicola Bartlett, https://www.mirror.co.uk/authors/nicola-bartlett/Labour's conference in Brighton demonstrated the power of the party's members.
A number of well-coordinated campaigns saw the grassroots seizing control of the conference.
Those resolutions were then considered in the clause iv meeting where the manifesto was written by different elements of the party.
The programme for "real change" contains a number of radical policies including nationalised mail, rail, water, energy and broadband, free NHS prescriptions and six years' free adult education in England.
It pledges a £10 minimum wage for all over-16s, an ambition to reach a 32-hour working week over a decade and the end of Universal Credit .
But some of those radical policies supported by the party's membership at its conference have been watered down.
1) Green New Deal
The manifesto does not commit to a net-zero target by 2030 instead promising to work towards that.
One of the big rows at conference in Brighton was over what the grassroots campaign called Labour's Green New Deal with a nod to the green movement in the US.
After a major push by members, Labour committed to work towards a net-zero carbon emissions target by 2030 in Brighton.
The ambitious plan won a vote at the party's annual conference after days of divisive debates over the target.
The "Labour's Socialist Green New Deal" motion called for the party to work towards a net-zero target for carbon emissions by 2030.
The GMB union, one of the three largest who are affiliated to Labour , consistently opposed the 2030 target throughout the conference.
Instead they wanted to see a less ambitious target because of concerns about potential job losses without a plan in place to adapt heavy industry.
Looking at the manifesto today it seems that the union's concerns were listened to.
Instead of a firm commitment to achieve net zero emissions by 2030 the Green New Deal aims to achieve the "substantial majority of our emissions reductions by 2030".
It also says reductions have to happen in a way that "delivers an economy that serves the interests of the many, not the few" in a clear nod to the concerns of the unions over jobs.
Other eye-catching elements on the environment included:
-
A “windfall tax” on oil and gas companies
-
Nationalising the ‘Big Six’ energy companies
-
One million green jobs
-
800,000 climate apprenticeships
- A £250 billion Green Transformation Fund
This is the crucial line in the manifesto: " Our Green New Deal aims to achieve the substantial majority of our emissions reductions by 2030 in a way that is evidence-based, just and that delivers an economy that serves the interests of the many, not the few."
2) Abolish Eton
An "abolish Eton" motion passed overwhelmingly on the conference floor after a campaign by grassroots members which took the party leadership by surprise.
It pushed the party to go further than they had originally planned which involved closing the tax loopholes used by the elite private schools.
The motion resolved for the party to "integrate all private schools into the state sector".
This would include withdrawal of charitable status and "all other public subsidies and tax privileges", including business rate exemption.
The policy, which came from the grassroots, passed overwhemingly on conference floor and was endorsed by the Education Secretary Angela Rayner.
In her speech she said Labour will make “the whole education system fairer through the integration of private schools”.
The manifesto does commit the party to close "tax loopholes" which would but it does not include the crucial pledge for the "endowments, investments and properties held by private schools to be redistributed democratically across the country’s educational institutions”.
It also asks the Social Justice Commission to advise on "integrating private schools and creating a comprehensive education system."
This is both committing the party to eventually scrapping private schools but also an acknowledgement that it is likely to take a long time to do.
It means the party's hierarchy has managed to water down the policy enough to avoid accusations of vindictively attacking Eton while also keeping members on board.
Giving the responsibility to Labour's new Social Justice Commission stresses the link between private education and ineuqlity.
The crucial part of the manifesto says: "We will close the tax loopholes enjoyed by elite private schools and use that money to improve the lives of all children, and we will ask the Social Justice Commission to advise on integrating private schools and creating a comprehensive education system."
3) Freedom of movement
Freedom of movement has been a big battleground within the party over recent months.
There is a member-led campaign to commit to freedom of movement without any caveats.
Instead the manifesto acknowledges that if the UK stays in the EU then freedom of movement will continue.
But if the UK leaves then the commitment is less clear.
The manifesto says that FoM would be "subject to negotiations" but it contains a commitment to "protect those rights".
Committing to Freedom of Movement is somewhat complicated by the party's two-step approach to the Brexit question promising to renegotiate a Brexit deal with the EU and then holding a referendum on whether to support that deal or stay in the EU.
The party have not said which have those options they would campaign for.
The exact wording in the manifesto is: "If we remain in the EU, freedom of movement would continue. If we leave, it will be subject to negotiations, but we recognise the social and economic benefits that free movement has brought both in terms of EU citizens here and UK citizens abroad – and we will seek to protect those rights."


Welcome to Maniac Editor’s documentation!¶
ForceSpin¶
# ForceSpin ## Bio: This entity forces the player into “spin” state, useful for 2 tile high tubes.
## Attributes:
### Direction: Determines the Direction the Player will Roll In. Note that for the player to stop rolling once used, the player must come in contact with another ForceSpin Entity with the opposite direction.
0 - Forwards 1 - Backwards
### Size: Changes the Size in Which the ForceSpin Will Be Applied.
### Angle: Changes the Angle in which the ForceSpin Object will trigger.
GasPlatform¶
## Bio: This entity is found in Oil Ocean Zone (OOZ), and acts very differently from the other types of platforms. Exclusively used in OOZ Officially, this platform will “pop-up” depending on it’s settings
## Attributes:
### Type Sets how the GasPlatform with behave.
0 - Pop Up on set interval; Interval set by the Interval and IntervalOffset attributes. 1 - Pop Up with a added Spring-Like Effect 2 (and Beyond) - Nothing; Solid Dumb Platform
### ChildCount Depending on it’s value, this attribute causes objects to move with the platform. For Example, if you set childCount to 2, and the Entity Slot is 287, Entities with a Slot of 288, and 289 will move with the platform.
### Interval Handles how long between the gas platform coming to a stop, and poping up next time.
### IntervalOffset Sets the Time in the Interval you want to start at when the stage loads
Platform¶
`
TODO: Unfinished Documentation
`
## Bio: Found in Almost Every Zone in Sonic Mania, this object allows Sonic to either cross gaps and get to places he otherwise couldn’t do on his own, or simply get to places without doing anything at all, lazily. However, some aren’t meant to carry sonic, some move objects, some can even hurt Sonic if you set them up correctly. There’s plenty of things you can do with platforms.
## Attributes:
### Type Determines the Platform’s Behavior
0 - Default 1 - Collapsing Platform; Shortly after the player stands on this type, the platform will begin to fall. This type is known to respawn after the matter however once the player goes far away enough. 2 - Normal Up and Down Moving Platforms 3 - Circular Path Moving Platforms 4 - Back and Forth Moving Platforms; Move from side to side. 5 - A Dud Platform? 6 - Player Pushable Platform 7 - OOZ Moving Spike Platforms; They move back and forth when they hit walls. 8 - Waiting Moving Platforms; These Wait for the Player to be on top of them, and when the player is, it will move to its destination. 9 - Waiting Moving Platforms(with Bobbing); These Wait for the Player to be on top of them, and when the player is, it will move to its destination. 10 - Follows Player? 11 - Circular TMZ2 Style Platform Lift Thing? (IDK)
### Amplitude Determines where the platform will move (relative to the place it’s placed) if it is a moving type set by the type variable.
### Speed Determines how fast the platform will move if it’s a type that moves. Must be higher than 0 for the platform to able to move, the larger it is, the faster the platform will move into position.
### HasTension When using Type 3, determines if you want to see the tension bars (the chains that attach the platform the it’s initial position) or not.
### FrameID Determines what Texture/Frame to use. The Frames it uses can be found in the Platform.bin for the Zone in it’s Sprite Folder. Attribute has an upper limit of 127. If set to -1, it will appear Invisible, useful in combination with ChildCount.
### Collision Determines how the platform will behave when an player interacts with it. Your Results may vary depending on what your Type Attribute is set to.
0 - Solid on Top only 1 - Solid from all sides 2 - ??? 3 - Hazardous from all sides 4 - ??? 5 - Hazardous from left and right sides 6 - Hazardous from bottom only 7 - Hazardous from top only 8 - Platform makes Sonic look like he is on a turntable 9 - Sticky from all sides 10 - Sticky from top only 11 - Sticky from left side only 12 - Sticky from right side only 13 - Sticky from bottom side only 14 - Platform makes Sonic look like he is on a barstool
### TileOrigin Usage Unknown
### ChildCount Depending on it’s value, this attribute causes objects to move with the platform. For Example, if you set childCount to 2, and the Entity Slot is 287, Entities with a Slot of 288, and 289 will move with the platform.
### Angle When Using Type 3, sets the starting point for the platform on its circular path.
Player¶
## Bio:
This entity is where the player will spawn when the scene is loaded. You can have multiple Player Entities in a scene, but depending on its attributes, if there is a confliction, it will always prioritize based on a higher Slot ID.
## Attributes:
### CharacterID: Determines who spawns at the object
0 - World? 1 - Sonic 2 - Tails 3 - Sonic and Tails 4 - Knuckles 5 - Sonic and Knuckles 6 - Tails and Knuckles 7 - Team Sonic (Sonic,Tails and Knuckles)
NOTES: Sonic and Tails will simply be handled as Sonic if you used for say Sonic Only. Also, Mighty and Ray are Handled as Sonic, Meaning wherever Sonic can spawn, mighty and ray will and only will spawn if the CharacterID is set to Sonic.
Ring¶
## Bio: Where coins are for Mario, rings are for Sonic. These collectables are everywhere. They are essential for staying alive safely. They are also pretty complex given their abundance.
## Attributes:
### Type: Changes the Type of Ring
0 - Normal (Common) 1 - Large Rings (Wields 5 Rings) 2 - Sparkling Rings (Wields 1 Ring; Twinkles instead of displaying a ring)
### PlaneFilter: Determines the FG Layer the Ring is On; The outcome of the value depends on Layer Setup of your scene.
### Move Type: Determines if and how the the ring will move
0 - Normal (No Movement) 1 - Left/Right/Up/Down Movement 2- Circular Movement <= 3 - No Ring At All? (Needs Checking)
Note: Movement also relies on the following attributes.
### Amplitude: Only works on rings with a movement type higher than 0, for type 1, allows you to set how far the ring will go up and down and or left and right. For type 2, make the circle that the ring will circumfurimate around larger or smaller. The object simply will not do much without this attribute being set to something other than 0.
### Speed: At 0, rings will not move along their movement path defined by their movement type at all. It gets faster the more you increment the number.
### Angle: Determines the Starting Position for the Ring when using Type 2 Movement. As seen in Studiopolis Act 2.
SpinBooster¶
`
TODO: Unfinished Documentation
`
## Bio: Much like the ForceSpin this object is useful when making 2 tile high tube sections. This in specific, gives the player a bit of a “boost” so he can gain more momentum.
## Attributes:
Direction: Determines the Direction in which the “boost” will be applied.
0 - Forwards 1 - Backwards
### Size: Changes the Size in Which the SpinBooster Effect Will Be Applied.
### AutoGrip: IDK
### Bias: IDK
### BoostPower: The Multiplier for the amount of boost a SpinBooster Entity will give off
### Boost Always: Whether the SpinBooster should be allowed to boost regardless of the situation or not.
### ForwardOnly: Whether the Spin Booster should only work in the Forwards Direction or not.
### PlaySound: Whether the Spin Booster Should Play the Spin SFX or not.
### AllowTubeInput: Where the SpinBooster can work with the TransportTube entities or not.
Spring¶
## Bio:
Usable in almost every zone, the spring is essential to building mania stages, allowing you to get over anything with enough momentum. The Red Springs provide more momentum than the other springs
## Attributes:
### Type: This Attribute Controls the Direction of the Spring, as well as the type of spring.
0 - Upwards Facing Yellow Spring 1 - Upwards Facing Red Spring 2 - Sideways Facing Yellow Spring 3 - Sideways Facing Red Spring 4 - Upwards Facing Yellow Spring 5 - Upwards Facing Red Spring
### FlipFlag: This attribute can rotate the spring’s orientation. Note that types 0 and 1 have some funky functions with this attribute.
0 - Disabled; Normal State 1 - Flip Horizontally 2 - Flip Upside-Down 3 - Flip Upside-Down Horizontally
### OnGround: Determines if the Spring is “IN” the Ground, so it can react accordingly.
### PlaneFilter: Determines the FG Layer the Spring is On; The outcome of the value depends on Layer Setup of your scene.
TransportTube¶
TODO: Unfinished Documentation
## Bio: The Heart of Chemical Plant and Plenty of Other Smaller Uses throughout mania, these entities act as a pipeline highway for Sonic and co. They are very complex, but this guide aims to tell you all the possible variations you can make with this object.
## Attributes:
### Types/DirMask: These Variables rely heavily on very specific combinations, they are listed below.
#### Type 0 Dir Masks - The Most Common and Most Potent; Generic Transport Tube Moving
5: North-West 6: South-West 9: North-East 10: South-East 20: North East-West 40: North West-West 68: South East-West 136: South West - East
- #### Type 1 Dir Masks - Entry and Exit Tubes
3 - Up and Down Roll
#### Type 5 Dir Masks - Selector Tubes 15 - All Directions 14 - Left, Right and Down
- #### Type 6 Dir Masks - Sprinting Exits?
12 - East?
#### Type 2/4 Dir Masks - Freely Mid-Transport Paths?
Water¶
## Bio: Sonic’s Greatest Weakness and the Most Deadly Hazard. Comes in Many Different Variations. Some of which have functions that are specifically stage locked.
## Attributes:
### Type: Determines the Type of Water that is being used.
- 0 - Initial Water Height Position Marker;
Uses CPZ/HCZ/AIZ Style Water; Starting Height Based on Entity Position (not the values it has)
1 - Bounded Water Box; GHZ Style Water 2 - Water Bubbles 3 - Bounding Boxes for Adjusting Water Height.
### Size: Used for type 1, for the water box itself, and type 3 for the area in which the player can activate the water height adjustment.
### Height: Used in type 3 for setting the water height. In a CPZ/Normal setup, the water level can go up and down between the x and y values of this attribute. In a HCZ however, it will not bob and stay at the level defined by either x or y.
### Speed: For Type 2, Must be higher than 0 for the water to able to change height, the larger it is, the faster the water will move into position.
### Button Tag: Used in HCZ often, with a type 2, this will make the object un-intractable unless the button tag is triggered with a “Button” with the same “tag”.
### R, G, and B: For type 1 only, sets the color of the water, with RGB being all 0 being black, and RGB being all 255 being transparent.
### Priority: Determines the layer the water is Interactable with; The outcome of the value depends on Layer Setup of your scene.
### Destroy on Trigger: Removes the object once the “trigger” from the “Button” to the “tag” we have set has been sent. NOTES: CPZ/HCZ/AIZ Style Water CAN NOT be given a custom water pallet outside of those stage setups. They are hard locked to those stage setup entities. Attempting to use these in zones without water palettes will cause the water to appear transparent (AKA No palette adjustment).
# Object ## Bio:
## Attributes:


This project came to life as a product of an internal AngularJS introduction course at it-minds.dk.
The source code can be found at Github.


FIRE DOWN BELOW 1997 A film review by Timothy Voon Copyright 1997 Timothy Voon 2 :-( :-( for a vague whiff of smoke
Cast: Steven Seagal, Marg Helgenberger, Harry Dean Stanton, Kris Kristofferson, Stephen Lang, Levon Helm, Brad Hunt Director: Felix Enriquez Alcala Screenplay: Jeb Stuart and Philip Morton
There is something pristine about Segal and his action heroes who remain virtually invincible in harms way. He will occasionally be cut or stabbed, but you'll never see too much blood on that well chiselled face and solid torso. For some reason blood, dirt, sweat and bruises don't hang well on Segal. Punishing injury is for the die hard Willis, physical damage follows the kick boxing Van Damme, and multiple gun shot wounds riddle after the robotic Swharzenegger. As for Segal it's not his style to let the enemy throw too many close punches, let alone get his expensive clothes dirty. For that reason, I like Segal because he has redefined the meaning of a clean fight.
Unfortunately, there isn't anything unique about "Fire Down Below" that hasn't already traversed down the road of previous Segal action thrillers. He's a special agent once again, with a slick appearance and familiar pony tail; well dressed and diet controlled since his last siting with pot belly in the embarrassing "The Glimmer Man". Once again the hero he creates is perfectly balanced, poetically just and an effortless killer who defeats up to twenty men with guns in any given situation.  The real let down for me was how very ordinary this movie turned out.
I do not carry a keen love of pure special effects in movies, but in an action flick, especially in a Segal action flick I would like to see more than a couple of bins explode and objects burn. I would also like to see Segal wearing something outside his usual dark overcoat and leather jackets; especially in a small country town where he is supposed to blend with the natives in order to carry out an investigation. Come on Steven, how about some jeans, a pleat shirt and a cowboy hat? Too risque or just trying to hide weight? You need to dress the part if you want us to believe you have some carpentry skills. Just swinging the hammer in clothing suitable for a "Planet Hollywood" presentation doesn't quite cut the cheese.
On the unusual side of things, Segal has decided to include a love interest for his hero notably absent in his most recent movies. He has strangely chosen a frail, weedy looking creature (Marg Helgenberger) to be the main contender. There is no question that Segal remains the perfect gentleman in his movies and women naturally fall for that sort of a thing. But it becomes hard to swallow when he comes across as a Russian Mafia thug, whilst his love interest is mostly clothed in rags and lives like a hermit. The only thing they have in common is her porch needs fixing, he likes her cooking and they both collect honey. Maybe that's enough for some relationships to work, but there certainly isn't any chemistry under the stove.
So overall I was disappointed by this latest of Segal offerings. I guess in my heart, I still carry that simple wish that he'll get back into whites and rescue another battle cruiser from the claws of terrorists. Somehow rescuing country bumpkins from the jaws of polluting industrialists doesn't carry the same missed heartbeat of thrill and ride.
Timothy Voon e-mail<EMAIL_ADDRESS>The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


http://www.speakers-podium.com/susangranger.
Susan Granger's review of "IN THE COMPANY OF SPIES" (SHOWTIME TV)
On Sunday night (Oct. 24) at 8 PM, Showtime TV presents an original espionage thriller that revolves around the CIA in this contemporary era of increased openness and accountability. Tom Berenger stars as a retired operative who angrily resigned five years ago from his position as head of the East Asian division and has since opened a Thai restaurant in Washington, D.C... He's brought back into action by his former boss, Ron Silver, to save a colleague who has been captured by the North Korean authorities. Korean Internal Security (KIS) knows the suspected agent possesses valuable knowledge of a covert operation but they don't know its nature, nor does the CIA. Berenger's assignment is to find out what the spy knows and to try to save his life. As the story unfolds, a small team of American spies, whose remarkable talents combine the newest technological techniques with old-fashioned infiltration work, reveals that, indeed, something sinister and critical is brewing in North Korea, something that could conceivably threaten the United States. The believable story portrays today's CIA in a realistic light, which helped the producers become the first film-makers to receive true access to the operational world of the Agency, filming some of the scenes at CIA Headquarters in Langley, Virginia. But its really the meticulously developed, character-driven script by Roger Towne ("The Natural"), expertly directed by Tim Matheson, that makes this presentation so compelling. On the Granger Movie Gauge of 1 to 10, "In the Company of Spies" is a compelling, suspenseful 8. This made-for-TV spy saga is as good or better than any you'll see at local movie theaters.
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Tales from the Borderlands is a five part episodic game series from the creators of The Wolf Among Us and The Walking Dead:
A Telltale Games Series. Set on the unforgiving world of Pandora after the events seen in Borderlands 2, this
is a story full of Borderlands’ trademark humor, following two adventurers on their quest for greatness.
You’ll play as Rhys, a Hyperion ‘suit’ with dreams of being the next Handsome Jack, and Fiona, a Pandoran con artist looking
to score her biggest ever swindle. Thrown together as unwilling partners in an adventure to recover cash
they both think is theirs, their journey will take you on a wild ride where gangsters, bandit lords, and
Vault Hunters are just some of the obstacles you’ll encounter, in this new take on the award-winning universe
created by Gearbox Software.


apache_beam.io package¶
Subpackages¶
- apache_beam.io.gcp package
- Subpackages
- apache_beam.io.gcp.datastore package
- Subpackages
- apache_beam.io.gcp.datastore.v1 package
- Submodules
- apache_beam.io.gcp.datastore.v1.datastoreio module
- apache_beam.io.gcp.datastore.v1.datastoreio_test module
- apache_beam.io.gcp.datastore.v1.fake_datastore module
- apache_beam.io.gcp.datastore.v1.helper module
- apache_beam.io.gcp.datastore.v1.helper_test module
- apache_beam.io.gcp.datastore.v1.query_splitter module
- apache_beam.io.gcp.datastore.v1.query_splitter_test module
- Module contents
- apache_beam.io.gcp.datastore.v1 package
- Module contents
- Subpackages
- apache_beam.io.gcp.internal package
- apache_beam.io.gcp.tests package
- apache_beam.io.gcp.datastore package
- Submodules
- apache_beam.io.gcp.bigquery module
- apache_beam.io.gcp.bigquery_test module
- apache_beam.io.gcp.gcsio module
- apache_beam.io.gcp.gcsio_test module
- apache_beam.io.gcp.pubsub module
- apache_beam.io.gcp.pubsub_test module
- Module contents
- Subpackages
Submodules¶
apache_beam.io.avroio module¶
Implements a source for reading Avro files.
-
class
apache_beam.io.avroio.
ReadFromAvro
(file_pattern=None, min_bundle_size=0, validate=True)[source]¶ Bases:
apache_beam.transforms.ptransform.PTransform
A
PTransform
for reading avro files.
-
class
apache_beam.io.avroio.
WriteToAvro
(file_path_prefix, schema, codec='deflate', file_name_suffix='', num_shards=0, shard_name_template=None, mime_type='application/x-avro')[source]¶ Bases:
apache_beam.transforms.ptransform.PTransform
A
PTransform
for writing avro files.
apache_beam.io.avroio_test module¶
-
class
apache_beam.io.avroio_test.
TestAvro
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
-
RECORDS
= [{'name': 'Thomas', 'favorite_number': 1, 'favorite_color': 'blue'}, {'name': 'Henry', 'favorite_number': 3, 'favorite_color': 'green'}, {'name': 'Toby', 'favorite_number': 7, 'favorite_color': 'brown'}, {'name': 'Gordon', 'favorite_number': 4, 'favorite_color': 'blue'}, {'name': 'Emily', 'favorite_number': -1, 'favorite_color': 'Red'}, {'name': 'Percy', 'favorite_number': 6, 'favorite_color': 'Green'}]¶
-
SCHEMA
= <avro.schema.RecordSchema object>¶
-
apache_beam.io.concat_source module¶
Concat Source, which reads the union of several other sources.
-
class
apache_beam.io.concat_source.
ConcatRangeTracker
(start, end, source_bundles)[source]¶ Bases:
apache_beam.io.iobase.RangeTracker
Range tracker for ConcatSource
-
class
apache_beam.io.concat_source.
ConcatSource
(sources)[source]¶ Bases:
apache_beam.io.iobase.BoundedSource
A
BoundedSource
that can group a set ofBoundedSources
.Primarily for internal use, use the
apache_beam.Flatten
transform to create the union of several reads.-
sources
¶
-
apache_beam.io.concat_source_test module¶
Unit tests for the sources framework.
-
class
apache_beam.io.concat_source_test.
ConcatSourceTest
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
apache_beam.io.filebasedsource module¶
A framework for developing sources for new file types.
To create a source for a new file type a sub-class of FileBasedSource
should
be created. Sub-classes of FileBasedSource
must implement the method
FileBasedSource.read_records()
. Please read the documentation of that method
for more details.
For an example implementation of FileBasedSource
see avroio.AvroSource
.
-
class
apache_beam.io.filebasedsource.
FileBasedSource
(file_pattern, min_bundle_size=0, compression_type=_CompressionType(auto), splittable=True, validate=True)[source]¶ Bases:
apache_beam.io.iobase.BoundedSource
A
BoundedSource
for reading a file glob of a given type.-
MIN_FRACTION_OF_FILES_TO_STAT
= 0.01¶
-
MIN_NUMBER_OF_FILES_TO_STAT
= 100¶
-
read_records
(file_name, offset_range_tracker)[source]¶ Returns a generator of records created by reading file ‘file_name’.
Parameters: - file_name – a
string
that gives the name of the file to be read. MethodFileBasedSource.open_file()
must be used to open the file and create a seekable file object. - offset_range_tracker – a object of type
OffsetRangeTracker
. This defines the byte range of the file that should be read. See documentation iniobase.BoundedSource.read()
for more information on reading records while complying to the range defined by a givenRangeTracker
.
Returns: a iterator that gives the records read from the given file.
- file_name – a
-
splittable
¶
-
apache_beam.io.filebasedsource_test module¶
-
class
apache_beam.io.filebasedsource_test.
EOL
[source]¶ Bases:
object
-
CRLF
= 2¶
-
LF
= 1¶
-
LF_WITH_NOTHING_AT_LAST_LINE
= 4¶
-
MIXED
= 3¶
-
-
class
apache_beam.io.filebasedsource_test.
LineSource
(file_pattern, min_bundle_size=0, compression_type=_CompressionType(auto), splittable=True, validate=True)[source]¶
-
class
apache_beam.io.filebasedsource_test.
TestConcatSource
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
-
class
apache_beam.io.filebasedsource_test.
TestFileBasedSource
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
-
class
apache_beam.io.filebasedsource_test.
TestSingleFileSource
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
apache_beam.io.fileio module¶
File-based sources and sinks.
-
class
apache_beam.io.fileio.
ChannelFactory
[source]¶ Bases:
object
-
static
open
(path, mode, mime_type='application/octet-stream', compression_type=_CompressionType(auto))[source]¶
-
static
size_in_bytes
(path)[source]¶ Returns the size of a file in bytes.
Parameters: path – a string that gives the path of a single file.
-
static
size_of_files_in_glob
(path, file_names=None)[source]¶ Returns a map of file names to sizes.
Parameters: - path – a file path pattern that reads the size of all the files
- file_names – List of file names that we need size for, this is added to support eventually consistent sources where two expantions of glob might yield to different files.
-
static
-
class
apache_beam.io.fileio.
CompressionTypes
[source]¶ Bases:
object
Enum-like class representing known compression types.
-
AUTO
= _CompressionType(auto)¶
-
BZIP2
= _CompressionType(bzip2)¶
-
GZIP
= _CompressionType(gzip)¶
-
UNCOMPRESSED
= _CompressionType(uncompressed)¶
-
classmethod
detect_compression_type
(file_path)[source]¶ Returns the compression type of a file (based on its suffix).
-
-
class
apache_beam.io.fileio.
FileSink
(file_path_prefix, coder, file_name_suffix='', num_shards=0, shard_name_template=None, mime_type='application/octet-stream', compression_type=_CompressionType(auto))[source]¶ Bases:
apache_beam.io.iobase.Sink
A sink to a GCS or local files.
To implement a file-based sink, extend this class and override either
write_record()
orwrite_encoded_record()
.If needed, also overwrite
open()
and/orclose()
to customize the file handling or write headers and footers.The output of this write is a PCollection of all written shards.
-
close
(file_handle)[source]¶ Finalize and close the file handle returned from
open()
.Called after all records are written.
By default, calls
file_handle.close()
iff it is not None.
-
open
(temp_path)[source]¶ Opens
temp_path
, returning an opaque file handle object.The returned file handle is passed to
write_[encoded_]record
andclose
.
-
apache_beam.io.fileio_test module¶
Unit tests for local and GCS sources and sinks.
-
class
apache_beam.io.fileio_test.
MyFileSink
(file_path_prefix, coder, file_name_suffix='', num_shards=0, shard_name_template=None, mime_type='application/octet-stream', compression_type=_CompressionType(auto))[source]¶
-
class
apache_beam.io.fileio_test.
TestChannelFactory
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
-
class
apache_beam.io.fileio_test.
TestCompressedFile
(methodName='runTest')[source]¶ Bases:
apache_beam.io.fileio_test._TestCaseWithTempDirCleanUp
apache_beam.io.iobase module¶
Sources and sinks.
A Source manages record-oriented data input from a particular kind of source (e.g. a set of files, a database table, etc.). The reader() method of a source returns a reader object supporting the iterator protocol; iteration yields raw records of unprocessed, serialized data.
A Sink manages record-oriented data output to a particular kind of sink (e.g. a set of files, a database table, etc.). The writer() method of a sink returns a writer object supporting writing records of serialized data to the sink.
-
class
apache_beam.io.iobase.
BoundedSource
[source]¶ Bases:
apache_beam.transforms.display.HasDisplayData
A source that reads a finite amount of input records.
This class defines following operations which can be used to read the source efficiently.
- Size estimation - method
estimate_size()
may return an accurate estimation in bytes for the size of the source. - Splitting into bundles of a given size - method
split()
can be used to split the source into a set of sub-sources (bundles) based on a desired bundle size. - Getting a RangeTracker - method
get_range_tracker()
should return aRangeTracker
object for a given position range for the position type of the records returned by the source. - Reading the data - method
read()
can be used to read data from the source while respecting the boundaries defined by a givenRangeTracker
.
A runner will perform reading the source in two steps.
- Method
get_range_tracker()
will be invoked with start and end positions to obtain aRangeTracker
for the range of positions the runner intends to read. Source must define a default initial start and end position range. These positions must be used if the start and/or end positions passed to the methodget_range_tracker()
areNone
- Method read() will be invoked with the
RangeTracker
obtained in the previous step.
Mutability
A
BoundedSource
object should not be mutated while its methods (for example,read()
) are being invoked by a runner. Runner implementations may invoke methods ofBoundedSource
objects through multi-threaded and/or reentrant execution modes.-
default_output_coder
()[source]¶ Coder that should be used for the records returned by the source.
Should be overridden by sources that produce objects that can be encoded more efficiently than pickling.
-
estimate_size
()[source]¶ Estimates the size of source in bytes.
An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before performing decompression or other processing.
Returns: estimated size of the source if the size can be determined, None
otherwise.
-
get_range_tracker
(start_position, stop_position)[source]¶ Returns a RangeTracker for a given position range.
Framework may invoke
read()
method with the RangeTracker object returned here to read data from the source.Parameters: - start_position – starting position of the range. If ‘None’ default start position of the source must be used.
- stop_position – ending position of the range. If ‘None’ default stop position of the source must be used.
Returns: a
RangeTracker
for the given position range.
-
read
(range_tracker)[source]¶ Returns an iterator that reads data from the source.
The returned set of data must respect the boundaries defined by the given
RangeTracker
object. For example:- Returned set of data must be for the range
[range_tracker.start_position, range_tracker.stop_position)
. Note that a source may decide to return records that start afterrange_tracker.stop_position
. See documentation in classRangeTracker
for more details. Also, note that framework might invokerange_tracker.try_split()
to perform dynamic split operations. range_tracker.stop_position may be updated dynamically due to successful dynamic split operations. - Method
range_tracker.try_split()
must be invoked for every record that starts at a split point. - Method
range_tracker.record_current_position()
may be invoked for records that do not start at split points.
Parameters: range_tracker – a RangeTracker
whose boundaries must be respected when reading data from the source. A runner that reads this source muss pass aRangeTracker
object that is notNone
.Returns: an iterator of data read by the source. - Returned set of data must be for the range
-
split
(desired_bundle_size, start_position=None, stop_position=None)[source]¶ Splits the source into a set of bundles.
Bundles should be approximately of size
desired_bundle_size
bytes.Parameters: - desired_bundle_size – the desired size (in bytes) of the bundles returned.
- start_position – if specified the given position must be used as the starting position of the first bundle.
- stop_position – if specified the given position must be used as the ending position of the last bundle.
Returns: an iterator of objects of type ‘SourceBundle’ that gives information about the generated bundles.
- Size estimation - method
-
class
apache_beam.io.iobase.
RangeTracker
[source]¶ Bases:
object
A thread safe object used by Dataflow source framework.
A Dataflow source is defined using a ‘’BoundedSource’’ and a ‘’RangeTracker’’ pair. A ‘’RangeTracker’’ is used by Dataflow source framework to perform dynamic work rebalancing of position-based sources.
Position-based sources
A position-based source is one where the source can be described by a range of positions of an ordered type and the records returned by the reader can be described by positions of the same type.
In case a record occupies a range of positions in the source, the most important thing about the record is the position where it starts.
Defining the semantics of positions for a source is entirely up to the source class, however the chosen definitions have to obey certain properties in order to make it possible to correctly split the source into parts, including dynamic splitting. Two main aspects need to be defined:
- How to assign starting positions to records.
- Which records should be read by a source with a range ‘[A, B)’.
Moreover, reading a range must be efficient, i.e., the performance of reading a range should not significantly depend on the location of the range. For example, reading the range [A, B) should not require reading all data before ‘A’.
The sections below explain exactly what properties these definitions must satisfy, and how to use a
RangeTracker
with a properly defined source.Properties of position-based sources
The main requirement for position-based sources is associativity: reading records from ‘[A, B)’ and records from ‘[B, C)’ should give the same records as reading from ‘[A, C)’, where ‘A <= B <= C’. This property ensures that no matter how a range of positions is split into arbitrarily many sub-ranges, the total set of records described by them stays the same.
The other important property is how the source’s range relates to positions of records in the source. In many sources each record can be identified by a unique starting position. In this case:
- All records returned by a source ‘[A, B)’ must have starting positions in this range.
- All but the last record should end within this range. The last record may or may not extend past the end of the range.
- Records should not overlap.
Such sources should define “read ‘[A, B)’” as “read from the first record starting at or after ‘A’, up to but not including the first record starting at or after ‘B’”.
Some examples of such sources include reading lines or CSV from a text file, reading keys and values from a BigTable, etc.
The concept of split points allows to extend the definitions for dealing with sources where some records cannot be identified by a unique starting position.
In all cases, all records returned by a source ‘[A, B)’ must start at or after ‘A’.
Split points
Some sources may have records that are not directly addressable. For example, imagine a file format consisting of a sequence of compressed blocks. Each block can be assigned an offset, but records within the block cannot be directly addressed without decompressing the block. Let us refer to this hypothetical format as <i>CBF (Compressed Blocks Format)</i>.
Many such formats can still satisfy the associativity property. For example, in CBF, reading ‘[A, B)’ can mean “read all the records in all blocks whose starting offset is in ‘[A, B)’”.
To support such complex formats, we introduce the notion of split points. We say that a record is a split point if there exists a position ‘A’ such that the record is the first one to be returned when reading the range ‘[A, infinity)’. In CBF, the only split points would be the first records in each block.
Split points allow us to define the meaning of a record’s position and a source’s range in all cases:
- For a record that is at a split point, its position is defined to be the largest ‘A’ such that reading a source with the range ‘[A, infinity)’ returns this record.
- Positions of other records are only required to be non-decreasing.
- Reading the source ‘[A, B)’ must return records starting from the first split point at or after ‘A’, up to but not including the first split point at or after ‘B’. In particular, this means that the first record returned by a source MUST always be a split point.
- Positions of split points must be unique.
As a result, for any decomposition of the full range of the source into position ranges, the total set of records will be the full set of records in the source, and each record will be read exactly once.
Consumed positions
As the source is being read, and records read from it are being passed to the downstream transforms in the pipeline, we say that positions in the source are being consumed. When a reader has read a record (or promised to a caller that a record will be returned), positions up to and including the record’s start position are considered consumed.
Dynamic splitting can happen only at unconsumed positions. If the reader just returned a record at offset 42 in a file, dynamic splitting can happen only at offset 43 or beyond, as otherwise that record could be read twice (by the current reader and by a reader of the task starting at 43).
-
SPLIT_POINTS_UNKNOWN
= <object object>¶
-
fraction_consumed
()[source]¶ Returns the approximate fraction of consumed positions in the source.
** Thread safety **
Methods of the class
RangeTracker
including this method may get invoked by different threads, hence must be made thread-safe, e.g. by using a single lock object.Returns: the approximate fraction of positions that have been consumed by successful ‘try_split()’ and ‘report_current_position()’ calls, or 0.0 if no such calls have happened.
-
position_at_fraction
(fraction)[source]¶ Returns the position at the given fraction.
Given a fraction within the range [0.0, 1.0) this method will return the position at the given fraction compared to the position range [self.start_position, self.stop_position).
** Thread safety **
Methods of the class
RangeTracker
including this method may get invoked by different threads, hence must be made thread-safe, e.g. by using a single lock object.Parameters: fraction – a float value within the range [0.0, 1.0). Returns: a position within the range [self.start_position, self.stop_position).
-
set_current_position
(position)[source]¶ Updates the last-consumed position to the given position.
A source may invoke this method for records that do not start at split points. This may modify the internal state of the
RangeTracker
. If the record starts at a split point, methodtry_claim()
must be invoked instead of this method.Parameters: position – starting position of a record being read by a source.
-
set_split_points_unclaimed_callback
(callback)[source]¶ Sets a callback for determining the unclaimed number of split points.
By invoking this function, a
BoundedSource
can set a callback function that may get invoked by theRangeTracker
to determine the number of unclaimed split points. A split point is unclaimed ifRangeTracker.try_claim()
method has not been successfully invoked for that particular split point. The callback function accepts a single parameter, a stop position for the BoundedSource (stop_position). If the record currently being consumed by theBoundedSource
is at position current_position, callback should return the number of split points within the range (current_position, stop_position). Note that, this should not include the split point that is currently being consumed by the source.This function must be implemented by subclasses before being used.
Parameters: callback – a function that takes a single parameter, a stop position, and returns unclaimed number of split points for the source read operation that is calling this function. Value returned from callback should be either an integer larger than or equal to zero or RangeTracker.SPLIT_POINTS_UNKNOWN
.
-
split_points
()[source]¶ Gives the number of split points consumed and remaining.
For a
RangeTracker
used by aBoundedSource
(within aBoundedSource.read()
invocation) this method produces a 2-tuple that gives the number of split points consumed by theBoundedSource
and the number of split points remaining within the range of theRangeTracker
that has not been consumed by theBoundedSource
.More specifically, given that the position of the current record being read by
BoundedSource
is current_position this method produces a tuple that consists of (1) number of split points in the range [self.start_position(), current_position) without including the split point that is currently being consumed. This represents the total amount of parallelism in the consumed part of the source. (2) number of split points within the range [current_position, self.stop_position()) including the split point that is currently being consumed. This represents the total amount of parallelism in the unconsumed part of the source.Methods of the class
RangeTracker
including this method may get invoked by different threads, hence must be made thread-safe, e.g. by using a single lock object.- ** General information about consumed and remaining number of split
- points returned by this method. **
- Before a source read (
BoundedSource.read()
invocation) claims the first split point, number of consumed split points is 0. This condition holds independent of whether the input is “splittable”. A splittable source is a source that has more than one split point. - Any source read that has only claimed one split point has 0 consumed split points since the first split point is the current split point and is still being processed. This condition holds independent of whether the input is splittable.
- For an empty source read which never invokes
RangeTracker.try_claim()
, the consumed number of split points is 0. This condition holds independent of whether the input is splittable. - For a source read which has invoked
RangeTracker.try_claim()
n times, the consumed number of split points is n -1. - If a
BoundedSource
sets a callback through functionset_split_points_unclaimed_callback()
,RangeTracker
can use that callback when determining remaining number of split points. - Remaining split points should include the split point that is currently being consumed by the source read. Hence if the above callback returns an integer value n, remaining number of split points should be (n + 1).
- After last split point is claimed remaining split points becomes 1, because this unfinished read itself represents an unfinished split point.
- After all records of the source has been consumed, remaining number of split points becomes 0 and consumed number of split points becomes equal to the total number of split points within the range being read by the source. This method does not address this condition and will continue to report number of consumed split points as (“total number of split points” - 1) and number of remaining split points as 1. A runner that performs the reading of the source can detect when all records have been consumed and adjust remaining and consumed number of split points accordingly.
- Before a source read (
** Examples **
A “perfectly splittable” input which can be read in parallel down to the individual records.
Consider a perfectly splittable input that consists of 50 split points.
- Before a source read (
BoundedSource.read()
invocation) claims the first split point, number of consumed split points is 0 number of remaining split points is 50. - After claiming first split point, consumed number of split points is 0 and remaining number of split is 50.
- After claiming split point #30, consumed number of split points is 29 and remaining number of split points is 21.
- After claiming all 50 split points, consumed number of split points is 49 and remaining number of split points is 1.
a “block-compressed” file format such as
avroio
, in which a block of records has to be read as a whole, but different blocks can be read in parallel.Consider a block compressed input that consists of 5 blocks.
- Before a source read (
BoundedSource.read()
invocation) claims the first split point (first block), number of consumed split points is 0 number of remaining split points is 5. - After claiming first split point, consumed number of split points is 0 and remaining number of split is 5.
- After claiming split point #3, consumed number of split points is 2 and remaining number of split points is 3.
- After claiming all 5 split points, consumed number of split points is 4 and remaining number of split points is 1.
an “unsplittable” input such as a cursor in a database or a gzip compressed file.
Such an input is considered to have only a single split point. Number of consumed split points is always 0 and number of remaining split points is always 1.
By default
RangeTracker` returns ``RangeTracker.SPLIT_POINTS_UNKNOWN
for both consumed and remaining number of split points, which indicates that the number of split points consumed and remaining is unknown.Returns: A pair that gives consumed and remaining number of split points. Consumed number of split points should be an integer larger than or equal to zero or RangeTracker.SPLIT_POINTS_UNKNOWN
. Remaining number of split points should be an integer larger than zero orRangeTracker.SPLIT_POINTS_UNKNOWN
.
-
try_claim
(position)[source]¶ Atomically determines if a record at a split point is within the range.
This method should be called if and only if the record is at a split point. This method may modify the internal state of the
RangeTracker
by updating the last-consumed position toposition
.** Thread safety **
Methods of the class
RangeTracker
including this method may get invoked by different threads, hence must be made thread-safe, e.g. by using a single lock object.Parameters: position – starting position of a record being read by a source. Returns: True
, if the given position falls within the current range, returnsFalse
otherwise.
-
try_split
(position)[source]¶ Atomically splits the current range.
Determines a position to split the current range, split_position, based on the given position. In most cases split_position and position will be the same.
Splits the current range ‘[self.start_position, self.stop_position)’ into a “primary” part ‘[self.start_position, split_position)’ and a “residual” part ‘[split_position, self.stop_position)’, assuming the current last-consumed position is within ‘[self.start_position, split_position)’ (i.e., split_position has not been consumed yet).
If successful, updates the current range to be the primary and returns a tuple (split_position, split_fraction). split_fraction should be the fraction of size of range ‘[self.start_position, split_position)’ compared to the original (before split) range ‘[self.start_position, self.stop_position)’.
If the split_position has already been consumed, returns
None
.** Thread safety **
Methods of the class
RangeTracker
including this method may get invoked by different threads, hence must be made thread-safe, e.g. by using a single lock object.Parameters: position – suggested position where the current range should try to be split at. Returns: a tuple containing the split position and split fraction if split is successful. Returns None
otherwise.
-
class
apache_beam.io.iobase.
Read
(source)[source]¶ Bases:
apache_beam.transforms.ptransform.PTransform
A transform that reads a PCollection.
-
class
apache_beam.io.iobase.
Sink
[source]¶ Bases:
apache_beam.transforms.display.HasDisplayData
A resource that can be written to using the
df.io.Write
transform.Here
df
stands for Dataflow Python code imported in following manner.import apache_beam as beam
.A parallel write to an
iobase.Sink
consists of three phases:- A sequential initialization phase (e.g., creating a temporary output directory, etc.)
- A parallel write phase where workers write bundles of records
- A sequential finalization phase (e.g., committing the writes, merging output files, etc.)
For exact definition of a Dataflow bundle please see https://cloud.google.com/dataflow/faq.
Implementing a new sink requires extending two classes.
- iobase.Sink
iobase.Sink
is an immutable logical description of the location/resource to write to. Depending on the type of sink, it may contain fields such as the path to an output directory on a filesystem, a database table name, etc.iobase.Sink
provides methods for performing a write operation to the sink described by it. To this end, implementors of an extension ofiobase.Sink
must implement three methods:initialize_write()
,open_writer()
, andfinalize_write()
.- iobase.Writer
iobase.Writer
is used to write a single bundle of records. Aniobase.Writer
defines two methods:write()
which writes a single record from the bundle andclose()
which is called once at the end of writing a bundle.See also
df.io.fileio.FileSink
which provides a simpler API for writing sinks that produce files.Execution of the Write transform
initialize_write()
andfinalize_write()
are conceptually called once: at the beginning and end of aWrite
transform. However, implementors must ensure that these methods are idempotent, as they may be called multiple times on different machines in the case of failure/retry or for redundancy.initialize_write()
should perform any initialization that needs to be done prior to writing to the sink.initialize_write()
may return a result (let’s call thisinit_result
) that contains any parameters it wants to pass on to its writers about the sink. For example, a sink that writes to a file system may return aninit_result
that contains a dynamically generated unique directory to which data should be written.To perform writing of a bundle of elements, Dataflow execution engine will create an
iobase.Writer
using the implementation ofiobase.Sink.open_writer()
. When invokingopen_writer()
execution engine will provide theinit_result
returned byinitialize_write()
invocation as well as a bundle id (let’s call thisbundle_id
) that is unique for each invocation ofopen_writer()
.Execution engine will then invoke
iobase.Writer.write()
implementation for each element that has to be written. Once all elements of a bundle are written, execution engine will invokeiobase.Writer.close()
implementation which should return a result (let’s call thiswrite_result
) that contains information that encodes the result of the write and, in most cases, some encoding of the unique bundle id. For example, if each bundle is written to a unique temporary file,close()
method may return an object that contains the temporary file name. After writing of all bundles is complete, execution engine will invokefinalize_write()
implementation. As parameters to this invocation execution engine will provideinit_result
as well as an iterable ofwrite_result
.The execution of a write transform can be illustrated using following pseudo code (assume that the outer for loop happens in parallel across many machines):
init_result = sink.initialize_write() write_results = [] for bundle in partition(pcoll): writer = sink.open_writer(init_result, generate_bundle_id()) for elem in bundle: writer.write(elem) write_results.append(writer.close()) sink.finalize_write(init_result, write_results)
init_result
Methods of ‘iobase.Sink’ should agree on the ‘init_result’ type that will be returned when initializing the sink. This type can be a client-defined object or an existing type. The returned type must be picklable using Dataflow coder
coders.PickleCoder
. Returning an init_result is optional.bundle_id
In order to ensure fault-tolerance, a bundle may be executed multiple times (e.g., in the event of failure/retry or for redundancy). However, exactly one of these executions will have its result passed to the
iobase.Sink.finalize_write()
method. Each call toiobase.Sink.open_writer()
is passed a unique bundle id when it is called by theWriteImpl
transform, so even redundant or retried bundles will have a unique way of identifying their output.The bundle id should be used to guarantee that a bundle’s output is unique. This uniqueness guarantee is important; if a bundle is to be output to a file, for example, the name of the file must be unique to avoid conflicts with other writers. The bundle id should be encoded in the writer result returned by the writer and subsequently used by the
finalize_write()
method to identify the results of successful writes.For example, consider the scenario where a Writer writes files containing serialized records and the
finalize_write()
is to merge or rename these output files. In this case, a writer may use its unique id to name its output file (to avoid conflicts) and return the name of the file it wrote as its writer result. Thefinalize_write()
will then receive anIterable
of output file names that it can then merge or rename using some bundle naming scheme.write_result
iobase.Writer.close()
andfinalize_write()
implementations must agree on type of thewrite_result
object returned when invokingiobase.Writer.close()
. This type can be a client-defined object or an existing type. The returned type must be picklable using Dataflow codercoders.PickleCoder
. Returning awrite_result
wheniobase.Writer.close()
is invoked is optional but if uniquewrite_result
objects are not returned, sink should, guarantee idempotency when same bundle is written multiple times due to failure/retry or redundancy.More information
For more information on creating new sinks please refer to the official documentation at
https://cloud.google.com/dataflow/model/custom-io#creating-sinks
.-
finalize_write
(init_result, writer_results)[source]¶ Finalizes the sink after all data is written to it.
Given the result of initialization and an iterable of results from bundle writes, performs finalization after writing and closes the sink. Called after all bundle writes are complete.
The bundle write results that are passed to finalize are those returned by bundles that completed successfully. Although bundles may have been run multiple times (for fault-tolerance), only one writer result will be passed to finalize for each bundle. An implementation of finalize should perform clean up of any failed and successfully retried bundles. Note that these failed bundles will not have their writer result passed to finalize, so finalize should be capable of locating any temporary/partial output written by failed bundles.
If all retries of a bundle fails, the whole pipeline will fail without finalize_write() being invoked.
A best practice is to make finalize atomic. If this is impossible given the semantics of the sink, finalize should be idempotent, as it may be called multiple times in the case of failure/retry or for redundancy.
Note that the iteration order of the writer results is not guaranteed to be consistent if finalize is called multiple times.
Parameters: - init_result – the result of
initialize_write()
invocation. - writer_results – an iterable containing results of
Writer.close()
invocations. This will only contain results of successful writes, and will only contain the result of a single successful write for a given bundle.
- init_result – the result of
-
initialize_write
()[source]¶ Initializes the sink before writing begins.
Invoked before any data is written to the sink.
Please see documentation in
iobase.Sink
for an example.Returns: An object that contains any sink specific state generated by initialization. This object will be passed to open_writer() and finalize_write() methods.
-
open_writer
(init_result, uid)[source]¶ Opens a writer for writing a bundle of elements to the sink.
Parameters: - init_result – the result of initialize_write() invocation.
- uid – a unique identifier generated by the system.
Returns: an
iobase.Writer
that can be used to write a bundle of records to the current sink.
-
class
apache_beam.io.iobase.
SourceBundle
(weight, source, start_position, stop_position)¶ Bases:
tuple
-
source
¶ Alias for field number 1
-
start_position
¶ Alias for field number 2
-
stop_position
¶ Alias for field number 3
-
weight
¶ Alias for field number 0
-
-
class
apache_beam.io.iobase.
Write
(sink)[source]¶ Bases:
apache_beam.transforms.ptransform.PTransform
A
PTransform
that writes to a sink.A sink should inherit
iobase.Sink
. Such implementations are handled using a composite transform that consists of threeParDo``s - (1) a ``ParDo
performing a global initialization (2) aParDo
performing a parallel write and (3) aParDo
performing a global finalization. In the case of an emptyPCollection
, only the global initialization and finalization will be performed. Currently only batch workflows support custom sinks.Example usage:
pcollection | beam.io.Write(MySink())
This returns a
pvalue.PValue
object that represents the end of the Pipeline.The sink argument may also be a full PTransform, in which case it will be applied directly. This allows composite sink-like transforms (e.g. a sink with some pre-processing DoFns) to be used the same as all other sinks.
This transform also supports sinks that inherit
iobase.NativeSink
. These are sinks that are implemented natively by the Dataflow service and hence should not be updated by users. These sinks are processed using a Dataflow native write transform.
-
class
apache_beam.io.iobase.
WriteImpl
(sink)[source]¶ Bases:
apache_beam.transforms.ptransform.PTransform
Implements the writing of custom sinks.
-
class
apache_beam.io.iobase.
Writer
[source]¶ Bases:
object
Writes a bundle of elements from a
PCollection
to a sink.A Writer
iobase.Writer.write()
writes and elements to the sink whileiobase.Writer.close()
is called after all elements in the bundle have been written.See
iobase.Sink
for more detailed documentation about the process of writing to a sink.
apache_beam.io.range_trackers module¶
iobase.RangeTracker implementations provided with Dataflow SDK.
-
class
apache_beam.io.range_trackers.
GroupedShuffleRangeTracker
(decoded_start_pos, decoded_stop_pos)[source]¶ Bases:
apache_beam.io.iobase.RangeTracker
A ‘RangeTracker’ for positions used by’GroupedShuffleReader’.
These positions roughly correspond to hashes of keys. In case of hash collisions, multiple groups can have the same position. In that case, the first group at a particular position is considered a split point (because it is the first to be returned when reading a position range starting at this position), others are not.
-
class
apache_beam.io.range_trackers.
LexicographicKeyRangeTracker
(start_position=None, stop_position=None)[source]¶ Bases:
apache_beam.io.range_trackers.OrderedPositionRangeTracker
A range tracker that tracks progress through a lexicographically ordered keyspace of strings.
-
class
apache_beam.io.range_trackers.
OffsetRangeTracker
(start, end)[source]¶ Bases:
apache_beam.io.iobase.RangeTracker
A ‘RangeTracker’ for non-negative positions of type ‘long’.
-
OFFSET_INFINITY
= inf¶
-
last_record_start
¶
-
-
class
apache_beam.io.range_trackers.
OrderedPositionRangeTracker
(start_position=None, stop_position=None)[source]¶ Bases:
apache_beam.io.iobase.RangeTracker
An abstract base class for range trackers whose positions are comparable.
Subclasses only need to implement the mapping from position ranges to and from the closed interval [0, 1].
-
UNSTARTED
= <object object>¶
-
fraction_to_position
(fraction, start, end)[source]¶ Converts a fraction between 0 and 1 to a position between start and end.
-
-
class
apache_beam.io.range_trackers.
UnsplittableRangeTracker
(range_tracker)[source]¶ Bases:
apache_beam.io.iobase.RangeTracker
A RangeTracker that always ignores split requests.
This can be used to make a given
RangeTracker
object unsplittable by ignoring all calls totry_split()
. All other calls will be delegated to the givenRangeTracker
.
apache_beam.io.range_trackers_test module¶
Unit tests for the range_trackers module.
-
class
apache_beam.io.range_trackers_test.
GroupedShuffleRangeTrackerTest
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
-
class
apache_beam.io.range_trackers_test.
LexicographicKeyRangeTrackerTest
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
Tests of LexicographicKeyRangeTracker.
-
fraction_to_key
(fraction, start=None, end=None)¶ Linearly interpolates a key that is lexicographically fraction of the way between start and end.
-
key_to_fraction
(key, start=None, end=None)¶ Returns the fraction of keys in the range [start, end) that are less than the given key.
-
-
class
apache_beam.io.range_trackers_test.
OffsetRangeTrackerTest
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
-
class
apache_beam.io.range_trackers_test.
OrderedPositionRangeTrackerTest
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
-
class
DoubleRangeTracker
(start_position=None, stop_position=None)[source]¶ Bases:
apache_beam.io.range_trackers.OrderedPositionRangeTracker
-
class
apache_beam.io.source_test_utils module¶
Helper functions and test harnesses for source implementations.
This module contains helper functions and test harnesses for checking
correctness of source (a subclass of iobase.BoundedSource
) and range
tracker (a subclass of``iobase.RangeTracker``) implementations.
Contains a few lightweight utilities (e.g. reading items from a source such as
readFromSource()
, as well as heavyweight property testing and stress
testing harnesses that help getting a large amount of test coverage with few
code.
Most notable ones are:
* assertSourcesEqualReferenceSource()
helps testing that the data read by
the union of sources produced by BoundedSource.split()
is the same as data
read by the original source.
* If your source implements dynamic work rebalancing, use the
assertSplitAtFraction()
family of functions - they test behavior of
RangeTracker.try_split()
, in particular, that various consistency
properties are respected and the total set of data read by the source is
preserved when splits happen. Use assertSplitAtFractionBehavior()
to test
individual cases of RangeTracker.try_split()
and use
assertSplitAtFractionExhaustive()
as a heavy-weight stress test including
concurrency. We strongly recommend to use both.
- For example usages, see the unit tests of modules such as
- apache_beam.io.source_test_utils_test.py
- apache_beam.io.avroio_test.py
-
class
apache_beam.io.source_test_utils.
ExpectedSplitOutcome
[source]¶ Bases:
object
-
MUST_BE_CONSISTENT_IF_SUCCEEDS
= 3¶
-
MUST_FAIL
= 2¶
-
MUST_SUCCEED_AND_BE_CONSISTENT
= 1¶
-
-
class
apache_beam.io.source_test_utils.
SplitAtFractionResult
(num_primary_items, num_residual_items)¶ Bases:
tuple
-
num_primary_items
¶ Alias for field number 0
-
num_residual_items
¶ Alias for field number 1
-
-
class
apache_beam.io.source_test_utils.
SplitFractionStatistics
(successful_fractions, non_trivial_fractions)¶ Bases:
tuple
-
non_trivial_fractions
¶ Alias for field number 1
-
successful_fractions
¶ Alias for field number 0
-
-
apache_beam.io.source_test_utils.
assertReentrantReadsSucceed
(source_info)[source]¶ Tests if a given source can be read in a reentrant manner.
Assume that given source produces the set of values {v1, v2, v3, ... vn}. For i in range [1, n-1] this method performs a reentrant read after reading i elements and verifies that both the original and reentrant read produce the expected set of values.
Parameters: source_info – a three-tuple that gives the reference iobase.BoundedSource
, position to start reading at, and a position to stop reading at.Raises: ValueError
– if source is too trivial or reentrant read result in an incorrect read.
-
apache_beam.io.source_test_utils.
assertSourcesEqualReferenceSource
(reference_source_info, sources_info)[source]¶ Tests if a reference source is equal to a given set of sources.
Given a reference source (a
BoundedSource
and a position range) and a list of sources, assert that the union of the records read from the list of sources is equal to the records read from the reference source.Parameters: - reference_source_info – a three-tuple that gives the reference
iobase.BoundedSource
, position to start reading at, and position to stop reading at. - sources_info – a set of sources. Each source is a three-tuple that is of the same format described above.
Raises: ValueError
– if the set of data produced by the reference source and the given set of sources are not equivalent.- reference_source_info – a three-tuple that gives the reference
-
apache_beam.io.source_test_utils.
assertSplitAtFractionBehavior
(source, num_items_to_read_before_split, split_fraction, expected_outcome)[source]¶ Verifies the behaviour of splitting a source at a given fraction.
Asserts that splitting a
BoundedSource
either fails after readingnum_items_to_read_before_split
items, or succeeds in a way that is consistent according toassertSplitAtFractionSucceedsAndConsistent()
.Parameters: - source – the source to perform dynamic splitting on.
- num_items_to_read_before_split – number of items to read before splitting.
- split_fraction – fraction to split at.
- expected_outcome – a value from ‘ExpectedSplitOutcome’.
Returns: a tuple that gives the number of items produced by reading the two ranges produced after dynamic splitting. If splitting did not occur, the first value of the tuple will represent the full set of records read by the source while the second value of the tuple will be ‘-1’.
-
apache_beam.io.source_test_utils.
assertSplitAtFractionBinary
(source, expected_items, num_items_to_read_before_split, left_fraction, left_result, right_fraction, right_result, stats, start_position=None, stop_position=None)[source]¶ Performs dynamic work rebalancing for fractions within a given range.
Asserts that given a start position, a source can be split at every interesting fraction (halfway between two fractions that differ by at least one item) and the results are consistent if a split succeeds.
Parameters: - source – source to perform dynamic splitting on.
- expected_items – total set of items expected when reading the source.
- num_items_to_read_before_split – number of items to read before splitting.
- left_fraction – left fraction for binary splitting.
- left_result – result received by splitting at left fraction.
- right_fraction – right fraction for binary splitting.
- right_result – result received by splitting at right fraction.
- stats – a
SplitFractionStatistics
for storing results.
-
apache_beam.io.source_test_utils.
assertSplitAtFractionExhaustive
(source, start_position=None, stop_position=None, perform_multi_threaded_test=True)[source]¶ Performs and tests dynamic work rebalancing exhaustively.
Asserts that for each possible start position, a source can be split at every interesting fraction (halfway between two fractions that differ by at least one item) and the results are consistent if a split succeeds. Verifies multi threaded splitting as well.
Parameters: - source – the source to perform dynamic splitting on.
- perform_multi_threaded_test – if true performs a multi-threaded test otherwise this test is skipped.
Raises: ValueError
– if the exhaustive splitting test fails.
-
apache_beam.io.source_test_utils.
assertSplitAtFractionFails
(source, num_items_to_read_before_split, split_fraction)[source]¶ Asserts that dynamic work rebalancing at a given fraction fails.
Asserts that trying to perform dynamic splitting after reading ‘num_items_to_read_before_split’ items from the source fails.
Parameters: - source – source to perform dynamic splitting on.
- num_items_to_read_before_split – number of items to read before splitting.
- split_fraction – fraction to split at.
-
apache_beam.io.source_test_utils.
assertSplitAtFractionSucceedsAndConsistent
(source, num_items_to_read_before_split, split_fraction)[source]¶ Verifies some consistency properties of dynamic work rebalancing.
Equivalent to the following pseudocode::
original_range_tracker = source.getRangeTracker(None, None) original_reader = source.read(original_range_tracker) items_before_split = read N items from original_reader suggested_split_position = original_range_tracker.position_for_fraction( split_fraction) original_stop_position - original_range_tracker.stop_position() split_result = range_tracker.try_split() split_position, split_fraction = split_result primary_range_tracker = source.get_range_tracker( original_range_tracker.start_position(), split_position) residual_range_tracker = source.get_range_tracker(split_position, original_stop_position) assert that: items when reading source.read(primary_range_tracker) == items_before_split + items from continuing to read 'original_reader' assert that: items when reading source.read(original_range_tracker) = items when reading source.read(primary_range_tracker) + items when reading source.read(residual_range_tracker)
Parameters: - source – source to perform dynamic work rebalancing on.
- num_items_to_read_before_split – number of items to read before splitting.
- split_fraction – fraction to split at.
-
apache_beam.io.source_test_utils.
readFromSource
(source, start_position=None, stop_position=None)[source]¶ Reads elements from the given
`BoundedSource`
.Only reads elements within the given position range. :param source:
iobase.BoundedSource
implementation. :param start_position: start position for reading. :param stop_position: stop position for reading.Returns: the set of values read from the sources.
apache_beam.io.source_test_utils_test module¶
apache_beam.io.sources_test module¶
Unit tests for the sources framework.
-
class
apache_beam.io.sources_test.
LineSource
(file_name)[source]¶ Bases:
apache_beam.io.iobase.BoundedSource
A simple source that reads lines from a given file.
-
TEST_BUNDLE_SIZE
= 10¶
-
apache_beam.io.textio module¶
A source and a sink for reading from and writing to text files.
-
class
apache_beam.io.textio.
ReadFromText
(file_pattern=None, min_bundle_size=0, compression_type=_CompressionType(auto), strip_trailing_newlines=True, coder=StrUtf8Coder, validate=True, skip_header_lines=0, **kwargs)[source]¶ Bases:
apache_beam.transforms.ptransform.PTransform
A PTransform for reading text files.
Parses a text file as newline-delimited elements, by default assuming UTF-8 encoding. Supports newline delimiters ‘n’ and ‘rn’.
This implementation only supports reading text encoded using UTF-8 or ASCII. This does not support other encodings such as UTF-16 or UTF-32.
-
class
apache_beam.io.textio.
WriteToText
(file_path_prefix, file_name_suffix='', append_trailing_newlines=True, num_shards=0, shard_name_template=None, coder=ToStringCoder, compression_type=_CompressionType(auto), header=None)[source]¶ Bases:
apache_beam.transforms.ptransform.PTransform
A PTransform for writing to text files.
apache_beam.io.textio_test module¶
Tests for textio module.
-
class
apache_beam.io.textio_test.
TextSinkTest
(methodName='runTest')[source]¶ Bases:
apache_beam.io.textio_test._TestCaseWithTempDirCleanUp
apache_beam.io.tfrecordio module¶
TFRecord sources and sinks.
-
class
apache_beam.io.tfrecordio.
ReadFromTFRecord
(file_pattern, coder=BytesCoder, compression_type=_CompressionType(auto), **kwargs)[source]¶ Bases:
apache_beam.transforms.ptransform.PTransform
Transform for reading TFRecord sources.
-
class
apache_beam.io.tfrecordio.
WriteToTFRecord
(file_path_prefix, coder=BytesCoder, file_name_suffix='', num_shards=0, shard_name_template='-SSSSS-of-NNNNN', compression_type=_CompressionType(auto), **kwargs)[source]¶ Bases:
apache_beam.transforms.ptransform.PTransform
Transform for writing to TFRecord sinks.
apache_beam.io.tfrecordio_test module¶
-
class
apache_beam.io.tfrecordio_test.
TestEnd2EndWriteAndRead
(methodName='runTest')[source]¶ Bases:
apache_beam.io.tfrecordio_test._TestCaseWithTempDirCleanUp
-
class
apache_beam.io.tfrecordio_test.
TestTFRecordSink
(methodName='runTest')[source]¶ Bases:
apache_beam.io.tfrecordio_test._TestCaseWithTempDirCleanUp
-
class
apache_beam.io.tfrecordio_test.
TestTFRecordSource
(methodName='runTest')[source]¶ Bases:
apache_beam.io.tfrecordio_test._TestCaseWithTempDirCleanUp
Module contents¶
A package defining several input sources and output sinks.


- For individual users
- Instant access to solutions
- Ask your tech questions
- Start your 30-day Free Trial
Main Topics
Browse All TopicsHi,
I am trying to run an CRM application on WSAD 5.1 with Websphere Application Server 5.0 test environment and getting this nested exception :
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
com.ibm.ejs.container.Bean
com.ibm.ejs.container.Bean
Does anyone know why and when this exception occur and how to resolve it ?? I found a bug fix on ibm site for Websphere Application Server but i am using WSAD to run locally.
Thanks
SR
This question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
900+ topic areas (and counting)
It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul.
We love the accolades. We love to be seen as the digital white knight who fixed the server, the computer, the email, and anything else that life depends on.
We love it so much, we sometimes make horrible decisions. Sometimes, we work "FOR FREE."
SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispersed nature of the blog posts that deal with some of these issues. This article will help SCCM OSD newbies with some of these issues.
For our deployments, we always install MDT 2008 on our SCCM server and select Configure ConfigMgr Integration from the MDT area on the start menu. This gives us:
Get full access to the help you need.
30 Day free trial. Register in 60 seconds.
Bringing you the latest in technology
| Congressman Proposes Bill to Ban... | 11/28/12 |
| Death to Internet Anonymity: Google... | 11/27/12 |
| TEDx San Luis Obispo | 11/20/12 |
| SharePoint 2010 No-Code Solutions... | 11/19/12 |
| Keeping Your Passwords And Your Win... | 11/15/12 |
| Fun and Games with Hackers: How to... | 11/15/12 |
Listen while you work
Because of our active community of experts, on average new questions receive their first comment in under 30 minutes.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30 Day free trial. Register in 60 seconds.
Business Accounts
Answer for Membership


Org Chart
Manage Employees-
Management
-
Mina Mona
Head Management -
Tia Nia
Product Manager
-
-
Finance
-
Mina Mona
Head Management -
Tia Nia
Product Manager
-
-
Marketing


MST3k Soundboard
It Stinks!
You can stop all sounds pressing
esc
or
enter
Stop All Sounds
Gain
Drive


A widget is, most basically, simply a folder in our repository (linked
above) that contains a widget.js
file. The directory
structure tends to look something like this:
myWidget |-- widget.js |-- chat | |-- index.js | +-- more_supporting_files +-- test |-- one.test.js +-- another.test.js
The widget.js
might look something like this:
module.exports = {
// Package name
name: 'Polls',
// Whether it's enabled by default
default: true,
// Beam username(s) of the author(s).
authors: ['connor4312'],
// Hooks for Beam chat servers/frontend.
hooks: {
chat: require('./chat')
},
// Description to display to users.
description: 'Allow people to create polls and let viewers vote on them!',
// Additional permissions roles should get to use the widget.
permissions: {
User: ['chat:poll_vote'],
Mod: ['chat:poll_vote', 'chat:poll_start'],
Admin: ['chat:poll_vote', 'chat:poll_start'],
Owner: ['chat:poll_vote', 'chat:poll_start']
}
};
It's mostly straightforward. The two parts that might need additional explanation are the permissions and hooks. Hooks is a map of strings to functions to register your widget various "environments" (currently, chat is the only available hook, but this will expand in the future). The process for hooking differs between environments; refer to the sections below for more information.
Permissions are simply an array of permission strings that your widget needs to operate. In the example, you can see that we let Users, Mods, Admins, and Owners have a permission to vote in polls, but only Mods, Admins, and Owners have permission to start polls. You can verify and restrict access based on permissions in individual hooks.
We allow you to hook into the chat server backend, adding middleware-like functions to modify or abort ongoing actions. Backend widgets are written in Node.js, and will be reviewed fairly rigorously as they'll be running in a production environment! In general:
The following modules are installed on the chat server available for you to consume anywhere in your widget. We'd prefer to avoid adding additional modules for widgets; if you think that you absolutely need something, let us know and we'll find a solution.
On the chat server we can hook into both events and methods, which provide two distinct types of functionality. When a user wants to do one specific task, they trigger a method for it, and that's the only time a method can be called. Events, on the other hand, can be triggered in a variety of circumstances, such as a user joining a channel, a chat message being sent, or your own events!
It is important to note that not every user of a chat will be
connected to the same chat server instance. For communication across
chats, we generally publish Redis events using the channel.emit
method, then listen for them later.
Now, let's get started! Be sure to have the JSDoc for the chat server (linked at the top of the page) open for reference, and an existing widget as well.
// Example hooks file `polls/chat/index.js`
module.exports = function (hook) {
hook.method('permission', 'vote:start', ['poll_start', require('./start').hook]);
hook.method('permission', 'vote:choose', ['poll_vote', require('./vote')]);
hook.event('PollStart', require('./events').start);
hook.event('PollEnd', require('./events').end);
};
Method hooks are triggered by a user sending a "message" directly to the chat server. The handler methods can be defined as guarded or unguarded:
.method(String event, Function handler) |
The simplest invocation. Your handler function won't be guarded.
Anyone, including guests, can send the method. Handler will be
called with |
.method(String guard, String event, Any handler) |
You function will be guarded by the guard you specify. Currently we have:
|
Event hooks are a bit simpler. They are currently only triggered by events emitted across Redis, and are called with the data they were broadcast on and arbitrary data. Example:
/**
* Handles a new creation poll event. It attaches an _ongoing_vote vote
* to the channel object for use later, and broadcasts to users.
* @param {Object} channel
* @param {Object} data
*/
module.exports.start = function (channel, data) {
channel._ongoing_vote = data.endsAt;
channel.broadcast('PollStart', data);
};
You can trigger these events yourself as well, by using the
channel.emit
method. For example:
channel.emit('PollStart', {
question: question,
answers: answers,
duration: duration,
endsAt: Date.now() + duration * 1000
});


Class: ActionText::AttachmentGallery
| Relationships & Source Files | |
| Super Chains via Extension / Inclusion / Inheritance | |
|
Class Chain:
|
|
|
Instance Chain:
|
|
| Inherits: | Object |
| Defined in: | actiontext/lib/action_text/attachment_gallery.rb |
Constant Summary
-
TAG_NAME =
private
# File 'actiontext/lib/action_text/attachment_gallery.rb', line 7"div"
Class Method Summary
- .attachment_selector
- .find_attachment_gallery_nodes(content)
- .fragment_by_canonicalizing_attachment_galleries(content)
- .fragment_by_replacing_attachment_gallery_nodes(content)
- .from_node(node)
- .new(node) ⇒ AttachmentGallery constructor
- .selector
::ActiveModel::Callbacks
- self
::ActiveModel::Naming
- self
| param_key | Returns string to use for params names. |
| plural | Returns the plural class name of a record or class. |
| route_key | Returns string to use while generating route names. |
| singular | Returns the singular class name of a record or class. |
| singular_route_key | Returns string to use while generating route names. |
| uncountable? | Identifies whether the class name of a record or class is uncountable. |
| extended, model_name_from_record_or_class |
Instance Attribute Summary
- #node readonly
::ActiveModel::API
- Included
| #_validators, #_validators?, | |
| #persisted? | Indicates if the model is persisted. |
| #validation_context |
::ActiveModel::AttributeAssignment
- Included
Instance Method Summary
::ActiveModel::API
- Included
| #initialize | Initializes a new model with the given |
::ActiveModel::Conversion
- Included
| #to_key | Returns an |
| #to_model | If your object is already designed to implement all of the Active Model you can use the default |
| #to_param | Returns a |
| #to_partial_path | Returns a |
::ActiveModel::Validations::HelperMethods
- Included
| #validates_absence_of | Validates that the specified attributes are blank (as defined by Object#present?). |
| #validates_acceptance_of | Encapsulates the pattern of wanting to validate the acceptance of a terms of service check box (or similar agreement). |
| #validates_comparison_of | Validates the value of a specified attribute fulfills all defined comparisons with another value, proc, or attribute. |
| #validates_confirmation_of | Encapsulates the pattern of wanting to validate a password or email address field with a confirmation. |
| #validates_exclusion_of | Validates that the value of the specified attribute is not in a particular enumerable object. |
| #validates_format_of | Validates whether the value of the specified attribute is of the correct form, going by the regular expression provided. |
| #validates_inclusion_of | Validates whether the value of the specified attribute is available in a particular enumerable object. |
| #validates_length_of | Validates that the specified attributes match the length restrictions supplied. |
| #validates_numericality_of | Validates whether the value of the specified attribute is numeric by trying to convert it to a float with |
| #validates_presence_of | Validates that the specified attributes are not blank (as defined by Object#blank?). |
| #validates_size_of | |
| #_merge_attributes |
::ActiveModel::Validations
- Included
| #errors | Returns the |
| #invalid? | Performs the opposite of |
| #read_attribute_for_validation | Hook method defining how an attribute value should be retrieved. |
| #valid? | Runs all the specified validations and returns |
| #validate | Alias for ActiveModel::Validations#valid?. |
| #validate! | Runs all the validations within the specified context. |
| #validates_with | Passes the record off to the class or classes specified and allows them to add errors based on more complex conditions. |
| #raise_validation_error, #run_validations!, | |
| #initialize_dup | Clean the |
::ActiveModel::AttributeAssignment
- Included
| #assign_attributes | Allows you to set all the attributes by passing in a hash of attributes with keys matching the attribute names. |
| #_assign_attribute, #_assign_attributes |
::ActiveModel::ForbiddenAttributesProtection
- Included
Constructor Details
.new(node) ⇒ AttachmentGallery
Class Method Details
.attachment_selector
[ GitHub ]# File 'actiontext/lib/action_text/attachment_gallery.rb', line 41
def "#{ActionText::Attachment.tag_name}[presentation=gallery]" end
.find_attachment_gallery_nodes(content)
[ GitHub ].fragment_by_canonicalizing_attachment_galleries(content)
[ GitHub ].fragment_by_replacing_attachment_gallery_nodes(content)
[ GitHub ].from_node(node)
[ GitHub ].selector
[ GitHub ]# File 'actiontext/lib/action_text/attachment_gallery.rb', line 45
def selector "#{TAG_NAME}:has(#{} + #{})" end
Instance Attribute Details
#node (readonly)
[ GitHub ]# File 'actiontext/lib/action_text/attachment_gallery.rb', line 50
attr_reader :node
Instance Method Details
#attachments
[ GitHub ]# File 'actiontext/lib/action_text/attachment_gallery.rb', line 56
def @attachments ||= node.css(ActionText::AttachmentGallery. ).map do |node| ActionText::Attachment.from_node(node).with_full_attributes end end
#inspect
[ GitHub ]# File 'actiontext/lib/action_text/attachment_gallery.rb', line 66
def inspect "#<#{self.class.name} size=#{size.inspect}>" end
#size
[ GitHub ]# File 'actiontext/lib/action_text/attachment_gallery.rb', line 62
def size .size end


Every day is an opportunity to achieve with Office 365.
What is Office 365
Office 365 is a cloud-based subscription service that brings together the best tools for the way people work today. By combining best-in-class apps like Excel and Outlook with powerful cloud services like OneDrive and Microsoft Teams, Office 365 lets anyone create and share anywhere on any device.


Apache™ FOP: Metadata
Overview
Document metadata is an important tool for categorizing and finding documents. Various formats support different kinds of metadata representation and to different levels. One of the more popular and flexible means of representing document or object metadata is XMP (eXtensible Metadata Platform, specified by Adobe). PDF 1.4 introduced the use of XMP. The XMP specification lists recommendation for embedding XMP metdata in other document and image formats. Given its flexibility it makes sense to make use this approach in the XSL-FO context. Unfortunately, unlike SVG which also refers to XMP, XSL-FO doesn't recommend a preferred way of specifying document and object metadata. Therefore, there's no portable way to represent metadata in XSL-FO documents. Each implementation does it differently.
Embedding XMP in an XSL-FO document
As noted above, there's no officially recommended way to embed metadata in XSL-FO. Apache FOP supports embedding XMP in XSL-FO. Currently, only support for document-level metadata is implemented. Object-level metadata will be implemented when there's interest.
Document-level metadata can be specified in the fo:declarations element. XMP specification recommends to use x:xmpmeta, rdf:RDF, and rdf:Description elements as shown in example below. Both x:xmpmeta and rdf:RDF elements are recognized as the top-level element introducing an XMP fragment (as per the XMP specification).
Example
[..] </fo:layout-master-set> <fo:declarations> <x:xmpmeta xmlns:x="adobe:ns:meta/"> <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"> <!-- Dublin Core properties go here --> <dc:title>Document title</dc:title> <dc:creator>Document author</dc:creator> <dc:description>Document subject</dc:description> </rdf:Description> <rdf:Description rdf:about="" xmlns:xmp="http://ns.adobe.com/xap/1.0/"> <!-- XMP properties go here --> <xmp:CreatorTool>Tool used to make the PDF</xmp:CreatorTool> </rdf:Description> </rdf:RDF> </x:xmpmeta> </fo:declarations> <fo:page-sequence ... [..]
Implementation in Apache FOP
Currently, XMP support is only available for PDF output.
Originally, you could set some metadata information through FOP's FOUserAgent by using its set*() methods (like setTitle(String) or setAuthor(String). These values are directly used to set value in the PDF Info object. Since PDF 1.4, adding metadata as an XMP document to a PDF is possible. That means that there are now two mechanisms in PDF that hold metadata.
Apache FOP now synchronizes the Info and the Metadata object in PDF, i.e. when you set the title and the author through the FOUserAgent, the two values will end up in the (old) Info object and in the new Metadata object as XMP content. If instead of FOUserAgent, you embed XMP metadata in the XSL-FO document (as shown above), the XMP metadata will be used as-is in the PDF Metadata object and some values from the XMP metadata will be copied to the Info object to maintain backwards-compatibility for PDF readers that don't support XMP metadata.
The mapping between the Info and the Metadata object used by Apache FOP comes from the PDF/A-1 specification. For convenience, here's the mapping table:
| Document information dictionary | XMP | |||
|---|---|---|---|---|
| Entry | PDF type | Property | XMP type | Category |
| Title | text string | dc:title | Text | External |
| Author | text string | dc:creator | seq Text | External |
| Subject | text string | dc:description["x-default"] | Text | External |
| Keywords | text string | pdf:Keywords | Text | External |
| Creator | text string | xmp:CreatorTool | Text | External |
| Producer | text string | pdf:Producer | Text | Internal |
| CreationDate | date | xmp:CreationDate | Date | Internal |
| ModDate | date | xmp:ModifyDate | Date | Internal |
Namespaces
Metadata is made of property sets where each property set uses a different namespace URI.
The following is a listing of namespaces that Apache FOP recognizes and acts upon, mostly to synchronize the XMP metadata with the PDF Info dictionary:
| Set/Schema | Namespace Prefix | Namespace URI |
|---|---|---|
| Dublin Core | dc | http://purl.org/dc/elements/1.1/ |
| XMP Basic | xmp | http://ns.adobe.com/xap/1.0/ |
| Adobe PDF Schema | http://ns.adobe.com/pdf/1.3/ |
Please refer to the XMP Specification for information on other metadata namespaces.
Property sets (Namespaces) not listed here are simply passed through to the final document (if supported). That is useful if you want to specify a custom metadata schema.


PC
Instruction Memory
Instruction Register
RegisterFile
ADR
BDR
ALU
ALUOutDR
SignZero Extend
Data Memory
DBDR
Instruction:


Mango: Easy Web Development
Mango is an easy to use, easy to test, high performance web application framework that stays out of your way and makes your life simple.


AngularJS handles validation beautifully using ng-messages library and allows you to do many of the normal validations right off the bat and still give you the flexibility to do your own validation. Below is many of the normal validations (phone, email) and a username check done through remote validation. The following usernames are taken: "username", "test", "john", "jack" and "peter".
Form API Documentation
NG-Messages API Documentation


Create takes a javascript object, which should have format like this -
{ Title: { Label: string, ShowClose: boolean }, Content: { Label: string, Class: string }, ButtonType: string Buttons: [{ Label: string, Class: string, Value: string }], ExecuteAfter: Function, ExecuteBefore: Function }
| Simple Customize Dialog | |
|---|---|
|
|
| Customize Dialog With Title | |
|---|---|
|
|
| Customize Dialog With Buttons | |
|---|---|
|
|
|
| Note : you can also mention ButtonType: Alert or Prompt to get the already defined buttons. |


Query:
US police shoot 12 year old boy toy gun
This query was designed to re-find the following podcast: Link to MP3 - (manual transcript)
Audio summaries
Link to MP3


Publisher offering to give Alaska newspaper away to new editor
by Ben HooperNov. 21 (UPI) -- The publisher of an Alaska newspaper said he is willing to give the entire operation away to a person or couple willing to relocate to the town and oversee the operation.
Larry Persily, owner of The Skagway News, said the he has been editing the paper remotely ever since his editor resigned, and he is now seeking a new editor to live in Skagway and take ownership of the paper.
"The only way this paper has a long-term future, and anything that I've ever seen that works with small town weeklies or bi-weeklies is where the small town editor owns, lives and are in the community," Persily wrote. "And that's what this needs."
Persily said he would be willing to turn the paper over without charge to a person or couple who either live in Skagway or would be willing to relocate.
The publisher purchased the newspaper in April for an undisclosed amount, but he said it was less than six figures.
Interested potential owner/editors are being asked to email Persily at<EMAIL_ADDRESS>(0) Leave a comment
upi.com/6961096
Latest Headlines
Odd News // 1 hour ago
Chicago woman attempting to hula hoop for 100 hours
Nov. 21 (UPI) -- A Chicago woman is more than two-thirds of the way to her goal of breaking a Guinness World Record by hula-hooping for 100 hours.
Odd News // 1 hour ago
Green Giant cooks up 1,009 pounds of green bean casserole
Nov. 21 (UPI) -- Vegetable company Green Giant announced it broke a Guinness World Record by cooking up 1,009 pounds of green bean casserole in New York.
Odd News // 1 hour ago
Zoo searching for escaped red panda in France
Nov. 21 (UPI) -- A French zoo is asking locals to keep an eye out for a red panda that escaped the facility when snowfall caused branches to fall onto its enclosure.
Odd News // 21 hours ago
Troopers, driver narrowly escape out of control truck
Nov. 20 (UPI) -- An Illinois State Police dashboard camera was recording when two troopers and a stranded driver had a near-miss with a box truck that lost control on an icy highway.
Odd News // 21 hours ago
Maryland woman wins two big Keno jackpots in one day
Nov. 20 (UPI) -- A Maryland woman is celebrating a particularly lucky day after she scored two major Keno jackpots.
Odd News // 22 hours ago
Men use tree saw to separate bucks with antlers tangled together
Nov. 20 (UPI) -- A group of Michigan neighbors used a saw to free two bucks found with their antlers stuck together in a farm field.
Odd News // 23 hours ago
Hotel room costs $1, but it's livestreamed at all times
Nov. 20 (UPI) -- A Japanese hotel offers a room that costs only $1 per night, but there's a catch -- the guest's entire stay is livestreamed on YouTube.
Odd News // 1 day ago
Territorial turkey chases Wisconsin mail carrier every day
Nov. 20 (UPI) -- A Wisconsin mail carrier said a territorial turkey has been chasing him through a neighborhood every day for about a month.
Odd News // 1 day ago
Large python crashes through spa ceiling in China
Nov. 19 (UPI) -- Workers at a spa in China were shocked when a 10-foot python fell through the ceiling -- and experts said it may have been there for up to 10 years.
Odd News // 1 day ago
Police help rescue coyote stuck in vehicle bumper
Nov. 19 (UPI) -- Authorities in Illinois came to the rescue of an unlucky coyote that was struck by a vehicle and ended up stuck in the bumper.