Eclipse 3.4 Debug Platform Build Notes
Summary of API changes in 3.4
Console Encoding
- A new method
ILaunchManager#getEncoding(ILaunchConfiguration)
was added to allow consumers of the DebugPlugin#ATTR_CONSOLE_ENCODING
preference to get the encoding to use when launching the configuration specified
as the parameter to getEncoding(..). The javadoc for DebugPlugin#ATTR_CONSOLE_ENCODING
has been updated to reflect that a launch configuration now inherits its default
encoding from its associated resource(s) when unspecified.
Launching Enhancements
- added
ILaunchShortcut2
. The new interface adds extensions to the standard launch shortcut (ILaunchShortcut
) to specify how selections and editors should be launched.
- added "
public boolean hasAttribute(String attributeName)
"
to ILaunchConfiguration
. The method allows clients to determine
if an attribute is present on a launch configuration.
- added "
public Object removeAttribute(String attributeName)
"
to ILaunchConfigurationWorkingCopy
. The method allows clients
to remove attributes from configurations.
Expression Enhancements
- added
IWatchExpressionFactoryAdapter2
. The new interface allows the
'Create Watch Expression' action to operate on arbitrary objects, instead of just
IVariable
s.
Getting Adapters
- added a new method
DebugPlugin#getAdapter(Object element, Class type)
that
gets an adapter from an object in the correct manner. Duplicate methods in the platform
were changed to use this convenience method.
Debug Model Presentation
- A new optional extension interface (
IDebugModelPresentationExtension
) allows debug model presentations to control whether they are called in the UI thread.
Modules View
- A Modules View has been added to the platform, along with appropriate constants in
IDebugUIConstants
.
ID_MODULE_VIEW
- view identifier
EMPTY_MODULES_GROUP
- menu group identifier
MODULES_GROUP
- menu group identifier
June 12, 2008
Problem Reports Fixed
Bug 236922: external tool builder does not run
June 5, 2008
Problem Reports Fixed
Bug 235824: [doc] Help topics missing
June 2, 2008
Problem Reports Fixed
Bug 234642: [doc] Javadoc of ILaunchesListener should link to ILaunchListener
May 28, 2008
Problem Reports Fixed
Bug 208062: First context menu has mysterious Run As/Debug As/Profile As menu entries
Bug 233730: Run control actions not refreshed after event with both CONTENT and STATE flags for parent context issued
May 28, 2008
Problem Reports Fixed
Bug 233703: Debug plugin and ProcessConsoleManager misses launches
May 21, 2008
Problem Reports Fixed
Bug 232699: TVT34:TCT330: TVT FR - Space missing before colon sign in French
May 20, 2008
Problem Reports Fixed
Bug 232832: TVT34:TCT275: KO: 'Import Launch Configurations' in English
May 14, 2008
Problem Reports Fixed
Bug 225688: DBCS3.4: In expression view, DBCS characters are changed to "?" after restarting eclipse
Bug 228445: 'Create Watch Expression' shows up in every context menu
Bug 224019: Command handler conflict in registers view
May 13, 2008
Problem Reports Fixed
Bug 229356: Run/Debug settings property page sizes to screen width for long name config
May 9, 2008
Problem Reports Fixed
Bug 217127: Regression in performance test PerfContextualLaunchMenu#testContextualLaunchMenu()
May 8, 2008
Problem Reports Fixed
Bug 230969: NPE disposing a model proxy
Bug 223851: Contextual launch not choosing most recent configuration
Bug 209243: Console window with black background
May 7, 2008
Problem Reports Fixed
Bug 229349: Poor wording in Import breakpoint dialog
May 6, 2008
Problem Reports Fixed
Bug 229344: AssertionFailedException during "Enable Breakpoint" with access/modification turned off
Bug 229362: filter count on LCD considers PRIVATE configs in total
Bug 229363: Run/Debug settings page should not be visible for .jpage files
Bug 220613: Platform UI now provides a collapse all command
Bug 229344: AssertionFailedException during "Enable Breakpoint" with access/modification turned off
Bug 227477: IModelProxy.init() and installed() are called inside a synchronized() block in ModelContentProvider
May 5, 2008
Problem Reports Fixed
Bug 222802: NPE if DebugUITools.launch(...) is called on background thread
Bug 225567: [Memory View] Tooltip from rendering stays forever
3.4 M7 - May 2, 2008
Problem Reports Fixed
Bug 228267: Inspect popup dialog disposes presentation context from variables view
Bug 228264: NPE when copying stack trace.
Bug 220611: Add PageSwitcher command to the Console View
Bug 228716: breakpoints view context activated for any part
Bug 220442: adopt identifier attribute for schemas
Bug 217104: NPE when returning null from force return
April 22, 2008
Problem Reports Fixed
Bug 220611: Add PageSwitcher command to the Console View
Bug 226203: Invoking Find action in Registers views with large group "hangs" IDE
Bug 203147: NPE while 'copy variables' in Variables view
Bug 198568: Copy Stack doesn't always cancel
Bug 191625: Undefined context while filtering dialog/window contexts
April 8, 2008
Problem Reports Fixed
Bug 215432: [Memory View] Memory View: real estate & new/remove rendering workflow
Bug 222267: Expressions aren't update correclty (ExpressionEventHandler doesn't preserve event source for CDT change events)
Bug 223992: NPE from "Model Proxy installed notification job"
Bug 225094: Get rid of outdated context Id "org.eclipse.ui.globalScope"
Bug 189786: Invalid thread access stopping Debug UI plugin
Link: enabling breakpoints during debug has poor performance
Bug 225336: [Memory View] MemoryView mistreats small bounded memory blocks
Bug 225366: [Memory View] Rendering view pane is blank when view is re-opened
Bug 225385: [Memory View] Need ability to disable add/remove rendering actions in RenderingViewPane
3.4 M6 - March 28, 2008
Problem Reports Fixed
Bug 188704: Top index not maintained properly in variables / registers view.
Bug 215489: Stackframe shows as running when suspended
Bug 127770: [variables] Expansion lost in variables view when holding down step key
Bug 221905: clean up javadoc tags
Bug 211158: Add a modules view to the set of standard debugger views.
Bug 223791: Compiler warning in org.eclipse.debug.ui
Bug 224128: LCD title not correct creating a new configuration
March 11, 2008
Problem Reports Fixed
Bug 219643: Memory: typo EXTENSION_POINT_MEMORY_RENDERIGNS
Bug 219794: NPE in console MatchJob
Bug 222228: Should delegate to platform adapater manager for adapters
Link: LazyStart from Debug plugins
February 19, 2008
Problem Reports Fixed
Bug 217769: [breakpoints] ToggleBreakpointAction always creates TextSelection with 0 lengh
February 12, 2008
Problem Reports Fixed
Bug 211920: [DND] Add expression by drag and drop does not honor "Create Watch Expression" enablement
Bug 217951: Missing detail formatter for registers group
Bug 217900: [breakpoints] Breakpoint export dialog does not size well on linux
Bug 218080: [breakpoints] No warning when using an inavlid path in breakpoint export dialog
3.4 M5 - February 8, 2008
Problem Reports Fixed
Bug 218086: Filtering statics also filters constants
Bug 218240: [expressions] Missing check for delegate and factory when creating watch expression
Bug 213069: Extend PresentationContext to save and restore properties.
Bug 216509: Memory View: typo, s/isMeomryBlockRemoved/isMemoryBlockRemoved
Bug 208062: First context menu has mysterious Run As/Debug As/Profile As menu entries
January 29, 2008
Problem Reports Fixed
Bug 217279: [launch] Allow ILaunchShortcut2 to provide resource mapping
Bug 216777: [detail panes] widget disposed exception
Bug 209883: Expression view actions use getAdapter() to work with non-standard debug models.
Bug 199915: API Request: DelegatingModelPresentation.isInitialized(...)
January 22, 2008
Problem Reports Fixed
Bug 215242: Debuggers cannot set the selected frame for Suspend events
Bug 215723: Attempt to reveal the call stack when stopped
January 8, 2008
Problem Reports Fixed
Bug 214094: i can't edit the run configuration because of an AssertionFailedException
Bug 214359: [launching] clean up launch config filtering
Bug 213719: NPE when closing the Variables view.
Bug 213609: Need a clarification on usage of IElement*Provider interfaces with update arrays.
December 18, 2007
Problem Reports Fixed
Bug 212638: Always save editors before launch preference should take effect immediately
Bug 213244: VariableAdapterService should also call IAdaptable.getAdapter() for adaptables that implement this method directly.
Bug 213212: ViewerAdapterService should use IAdapterManager
Bug 212314: DebugModelPropertyTester does not use the IDebugModelProvider adapter.
December 14, 3.4M4
Problem Reports Fixed
Bug 208039: Tweaks to context launching
Bug 208797: SelectedResourceManager.getCurrentSelection0() should use IAdapterManager
Bug 212420: Remove workaround for bug 99234
Bug 212145: [launching] Selection cannot be launched and there are no recent launches
Bug 210023: Another NPE in DefaultWatchExpressionModelProxy
Bug 202678: IViewerUpdate should have a method to return viewer input.
Bug 208034: [launching] Unexpected error when opening launch config dialog from resource properties
Bug 208127: breakpoint import does not work
Bug 211928: ExpressionDropAdapter makes implementation to IWatchExpressionFactoryAdapterExtension mandatory
Bug 212442: Failing import breakpoints tests
Bug 212581: [launching] widget disposed exception launching
December 4, 2007
Problem Reports Fixed
Bug 198428: Deadlock deleting launch configuration
Bug 211769: NPE in LaunchManager
Bug 211076: Debug annotations could allow all possible text styles
November 27, 2007
Problem Reports Fixed
Bug 210471: Cannot change the selected detail pane viewer.
November 20, 2007
Problem Reports Fixed
Bug 209920: While loading class SelectedResourceManager, thread timed out waiting
Bug 209668: The field org.eclipse.debug.core.DebugPlugin#INTERNAL_ERROR has changed value between HEAD and 3.3.0
Bug 184233: Expression view does not support alternative content providers
Bug 146825: [console] NPE in process console participant
Bug 205335: ModelContentProvider does not cancel stale updates when switching viewer input.
November 13, 2007
Problem Reports Fixed
Bug 208431: [launching] clicking checkbox twice disables OK button
Bug 208109: touching Environment tab makes launch config dirty
Bug 208707: (Console) Remove binding for show console view with CTRL+ALT+Q
3.4 M3 - November 2, 2007
Problem Reports Fixed
Bug 207600: export configurations page does not update as text typed into export directory
Bug 205980: new artwork for export/import launch configurations
Bug 208084: Error writing to console
Bug 207293: Wrong context help for the builder properties dialog
Bug 85370: [string variables] string_prompt for passwords
Bug 205305: Add perspective extensions for newly moved log view
Bug 206407: AbstractMemoryRendering connect()s the wrong "this"
Bug 207157: Breakpoint Export action disposes shared Label Provider.
October 23, 2007
Problem Reports Fixed
Bug 198896: Actions "Open Debug Dialog..." and "Open Run Dialog..." labeled incorrectly
Bug 207079: NPE running test suites in I20071016-1215
Bug 206822: Debug views should indicate busy status while there are viewer updates in progress.
October 16, 2007
Problem Reports Fixed
Bug 12898: [launching] Import/Export of launch configs
October 9, 2007
Problem Reports Fixed
Bug 205207: [launching] '&' shows in run tooltip if debug plugins not loaded
Bug 204378: An internal error occurred during: "Compute launch button tooltip".
October 2, 2007
Problem Reports Fixed
Bug 90737: "Terminate All" action should be renamed to "Terminate/Disconnect All"
September 24, 2007
Problem Reports Fixed
Bug 203907: Run --> External Tools history menus not updated after organize favorites
Bug 203615: TextConsoleViewer leaks Cursors
3.4 M2 - September 21, 2007
Problem Reports Fixed
Bug 202456: Variable Selection Dialog needs better help
Bug 202798: Breakpoint working set description
Bug 141009: [expressions] Add 'Watch' to the display view context menu
Bug 199125: Improve UI for breakpoint working sets
Bug 203253: Deleting config bring Run and Debug menus out of sync
Bug 202450: [Actions] Empty cascading menus show up as enabled
Bug 173156: [properties] process property page cuts off text
Bug 89178: [launching] Open Properties dialog by clicking "Run As..." with Ctrl key
Bug 188339: [scrapbook] Expressions from scrapbook does not convert to watch expression correctly
Bug 188385: context launch does not work with method in outline selected
Bug 203870: Launch tooltip not in synch with selection
Bug 202564: MessageConsole.clearConsole() flushes output queued up after the clearConsole call
Bug 203850: Expression order not always persisted
Bug 204033: Local configurations auto-deleted when project closed
Bug 204189: Always prompted to save launch configurations
September 11, 2007
Problem Reports Fixed
Bug 202059: Run dialogue: switching between read-only or removed launch configurations.
Bug 39875: [launching] Managing invalid launch configs
Bug 198785: Remove launch from Run/Debug drop down
September 4, 2007
Problem Reports Fixed
Bug 176508: [console] IOConsole Updater error with long output lines
Bug 199294: LaunchManager provokes NPEs when *.launch files are on remote EFS locations
Bug 137769: [launching] Launch history is not updated/Run Last Launched fails due to working set filter
Bug 198896: Actions "Open Debug Dialog..." and "Open Run Dialog..." labeled incorrectly
Bug 200185: Problem highlighting first position of source in debug mode.
August 28, 2007
Problem Reports Fixed
Bug 198545: Debug Core should use EMPTY_STRING constant
Bug 200831: Dragging text to bad location in expressions view does not update cursor
Bug 201239: Javadoc of ILaunchesListener should mention ILaunchesListener2
August 21, 2007
Problem Reports Fixed
Bug 200168: Keybindings on the Mac conflict with reserved OS bindings
Bug 200105: auto expand does not work in the async tree viewer
Bug 199500: Usability Problem in 'Save and Launch' Dialog
Bug 176627: [Registers view] Switching stack frames: Unnecessary scrolling to top and group closing
Bug 168574: Support for IElementMementoProvider on each element in a viewer
Bug 200595: reduce duplication of #getAdapter code for viewers
3.4 M1 - August 10, 2007
Problem Reports Fixed
190873: [launching] Environment tab grows wider on each config selection
198437: EMPTY_STRING constant not used
177554: [messages] remove internal error messages from translations
July 31, 2007
Problem Reports Fixed
198248: NPE at org.eclipse.debug.ui.CommonTab.getDefaultSharedConfigLocation
197998: Workbench refuses to show Run or Debug dialog
197834: Expression Evaluation Failure with Generics
197145: [launching] Changing AutoSelect in LaunchConfigurationViewer from a LaunchConfigurationTab
197645: [launching] selection of encoding does twice as much work
197532: NPE in DefaultWatchExpressionModelProxy
185196: Debug selection dialogs should subclass AbstractDebugSelectionDialog
194598: 44% of time closing projects spent in LaunchManager#resourceChanged
194303: [launching] CME during "Compute launch button tooltip"
197000: [launching] Auto Launch Configuration deletion should be resource specific
174020: Background colour makes variable value unreadable in variables view
197698: [launching] Deleting launch config from LCD does not remove config
191379: Console encoding does not match workspace encoding
191382: JVM file encoding does not match console encoding
July 17, 2007
Problem Reports Fixed
191515: Some run menu actions are missing help context ids
193175: inconsistent action names for "Open Run/Debug/Profile Dialog..."
177554: [messages] remove internal error messages from translations
July 10, 2007
Problem Reports Fixed
175584: [expressions] Allow re-ordering of expressions in the Expressions View
195232: Outdated Run Toolbar tool tip
191723: A lot more classes being loaded than in 3.1
195461: calling dispose on LaunchConfigurationTabGroupWrapper can get a NullPointerException
48197: [Dialogs] Add Folders Selection Dialog should not list closed projects.
191071: Delete Unused Debug Help Context ID Constants
194458: CCE when dragging to Breakpoints View
193595: ProcessConsole is internal API but states that "Clients may instantiate this class." in the javadocs
47931: Copies of serializeDocument(Document)
66983: [evaluation] EmptyStackException from invalid condition
192212: Screenshots out of date in isv doc
193179: Should there be an 'add' button in the expression view toolbar?
191974: Concurrent modification exception
191636: [breakpoints] import breakpoints causes exception
189328: [Accessibility] Run dialog (Common tab) - controls do not provide labels
191388: TVT33:TCT635: pl: truncations in Perspective preferences panel
193432: Clicking link in Ant console should ensure resulting editor is visible
184057: Add drag and drop to expression view.


Introduction
The pathom
library provides a set of functionality for building robust parsers to
process EQL requests.
If you are not familiar with EQL, check its docs for a general intro on EDN Graph Queries
.
In case you are not sure what this library is for, check the rationale to understand the motivations for it.
Pathom 3
Hello, if you are new here, I suggest you have a look at Pathom 3!
Pathom 3 still in alpha, but it’s the library that will get updated ahead, and most of the API is already stable, Pathom 2 (current here) is only updating in case of bug fixes.
What does it look like?
Let’s start with a demo! Use the buttons to load pre-defined queries and feel free to
edit and try around (start typing inside the []
and it will auto-complete to help you explore the API!).
Source code for this demo:
(ns com.wsscode.pathom.book.intro.demo
(:require [com.wsscode.pathom.core :as p]
[com.wsscode.pathom.connect :as pc]))
(pc/defresolver answer [_ _]
{::pc/output [:answer-to-everything]}
{:answer-to-everything 42})
(pc/defresolver answer-plus-one [_ {:keys [answer-to-everything]}]
{::pc/input #{:answer-to-everything}
::pc/output [:answer-plus-one]}
{:answer-plus-one (inc answer-to-everything)})
(def registry
[answer answer-plus-one])
(def parser
(p/parser
{::p/env {::p/reader [p/map-reader
pc/reader2
pc/open-ident-reader
p/env-placeholder-reader]
::p/placeholder-prefixes #{">"}}
::p/mutate pc/mutate
::p/plugins [(pc/connect-plugin {::pc/register registry})
p/error-handler-plugin
p/trace-plugin]}))
(comment
; to call the parser and get some data out of it, run:
(parser {} [:answer-to-everything]))
I hope you had a good time playing around! The intention of this library is to make it as easy as possible to implement API’s like the one you saw you in the demo above.
How it works
The following diagram illustrates the main pieces that make a pathom
parser:
To break down in parts, let’s imagine we are processing the following request:
[:answer-to-everything :answer-plus-one]
Parser
The parser receives the whole query to process and iterates over each attribute calling the reader on each one of them.
In our example, the parser receives the full query
[:answer-to-everything :answer-plus-one]
.
Readers
A reader processes a single entry from the query, the entry can be a property, a join or an ident. It is common to have a chain of readers (as shown in the photo and demo), when it’s a chain, it tries each reader in order. A reader can respond with either a value or signal the engine that it can’t process the key, in which case, it triggers the next reader.
In our example, the reader gets called once for :answer-to-everything
and then again with
:answer-plus-one
.
Resolvers
Pathom is capable of traversing a graph of dependencies to resolve data. This feature is called Connect and each resolver is an edge of this graph traversal. You can find more information about them in connect resolvers docs.
In the Pathom
library
The library includes:
-
A composable reader abstraction.
-
The concept of entity, which works as a base framework for reusable and sharable readers.
-
A plugin system with some built-in plugins:
-
Error handler: Handles errors at an attribute level.
-
Request cache: For caching the results of possibly repetitive parsing which can happen on a single request.
-
Tracer: A plugin to measure and debug each step of the parsing process.
-
-
Connect: A higher-level abstraction that can resolve attribute relationships automatically. For example, automatic traversal of database joins or resolving data through network requests. This enables exploratory capabilities and a much simpler access model when the need arises to do extra work for resolving a single conceptual join.
-
GraphQL integration: Use GraphQL endpoints directly from your query system (in development).
Most people find the most leverage in the Connect features, which allows you to quickly build dynamic query processing systems to satisfy client data requests with ease.
Aliases Used in Code Examples
Throughout the book, our code examples use aliases instead of explicit namespaces. The aliases used, assume that you have the following namespace requires in your environment:
(ns my-namespace
(:require
[com.wsscode.pathom.core :as p]
[com.wsscode.pathom.connect :as pc]
[com.wsscode.pathom.connect.graphql2 :as pcg]
[com.wsscode.pathom.graphql :as pg]
[com.wsscode.pathom.sugar :as ps]
[com.wsscode.pathom.trace :as pt]
; clj only
[com.wsscode.common.async-clj :refer [let-chan go-catch <? <?maybe]
; cljs only
[com.wsscode.common.async-cljs :refer [let-chan <!p go-catch <? <?maybe]]))
So, any time you see the usage of a namespace in a keyword or a function like p/parser
or ::p/reader
you should remember
that these are the namespaces involved.
Contributing
In every page of this documentation, you can find an icon at the top right, click on that icon to edit the current page (it opens the Github link to edit the page source).
How to Use This Library
Most of our current user base is made up of Fulcro
users, however this library is a stand-alone thing
that you can use to fulfill any system using EQL queries. The purpose of this library is to make it
much easier to build code that can process EQL on both the client and server-side.
We expect you to have one or more of the following needs:
-
You want to fulfill a client UI query from some server-side data source(s).
-
You want to build a client-side parser for directly filling local UI data queries from a local data source.
-
You want to build a parser (client or server) that uses async APIs to fulfill different parts of a query. Perhaps gluing together data sources from various micro-services.
-
You want to use a GraphQL API from the client.
-
You want to provide third-party users a GraphQL API (Future Work)
When building most parsers, you’ll want to use Pathom Connect.
To process EQL queries against GraphQL you’ll use the GraphQL Integration.


The Bachelor of Science in Information Technology (BSIT) is a three-year
program that prepares students to be IT professionals well-versed on IT applications, including web, database,
programming, and networking technologies. Students will have a solid IT foundation and will learn how to plan,
install, customize, operate, manage, administer, and maintain Information Technology infrastructures. The
program provides a solid foundation in IT, with a background in both software and hardware management, and
additional emphasis on strong database management skills.
e-Learning or Oline Learning is available across all Higher Education Programs
The Bachelor of Science in Computer Science (BSCS) is a three-year
program that prepares students to be IT professionals and researchers with proficiency in designing and
developing computing solutions. It is a general computer science program which focuses on concepts and theories,
including algorithmic foundations and the implementation and application of information and computing solutions.
The program develops the necessary skills to specify, design, develop, and manage computer-based systems
solutions and provide systems level support and development. It fosters strong research and analytical skills;
which graduates are expected to use as a solid foundation in their IT career.
e-Learning or Online Learning is available across all Higher Education Programs
The Bachelor of Science in Information Systems (BSIS) with Specialization in
Project Management is a three-year program that prepares students to be IT professionals with a solid
background in both Information Technology and business concepts and applications. Students are trained to be
expert in the design and implementation of computing solutions that integrate IT with business processes. This
program is right for students who would want to work in a business environment with technical or management
positions.
e-Learning or Online Learning is available across all Higher Education Programs
The Bachelor of Science in Business Administration (BSBA), major in Digital
Marketing and Web Analytics, is a three-year program which prepares students for positions of
responsibility in marketing, business, or entrepreneurship. It arms prospective executives with the knowledge
and disciplines required to effectively lead, organize, and run a business. The program uses an integrated
approach among the functional areas of business as well as sensitivity to the business environment.
The Marketing Management program prepares the students to be responsive to the total environment by providing
technical skills and competencies in the areas of marketing. Theory is blended with practicum activities to give
the students a broad and enriched base for a career in marketing.
e-Learning or Online Learning is available across all Higher Education Programs
The Bachelor of Science in Entrepreneurship (BSE) with Specialization in
Technopreneurship is a three-year program course is designed to allow students to develop, organize and
manage a business venture along with any of its risks in order to make a profit. Through its combination of
classroom training and experiential learning, the BS Entrepreneurship program will help aspiring entrepreneurs
acquire the skills, values and attitudes that will increase their chances of success.
e-Learning or Online Learning is available across all Higher Education Programs
The Bachelor of Science in Office Administration (BSOA) with Specialization in
Digital Productivity Tools is a three-year program that prepares the students for a career in an
outcome-focused, technology rich, professional environment. This course familiarizes the students with current
techniques in office practice and procedures, developments in office systems and technology, good team-working
and management skills and application of the principles of good human relations and communications to prepare
them to be key players in day-to-day office operations.
e-Learning or Online Learning is available across all Higher Education Programs


This is the API service of Annif. We provide an open REST-style API for automated subject indexing.
See the interactive Swagger documentation. You can also use use the swagger.json file directly with Swagger compliant tools.
Usage of the API is currently unrestricted. API keys are not used. Please be gentle with our server! In future, we may require registration and API keys for heavy use.
The data returned by the API may be used for any purpose without restriction.


Just like a traditional signature, this mark is used by CrossDottr to represent your intent to sign documents.


Regression checklist
This is a checklist, to keep a record of subtle bugs in the AMI that can be hard to spot.
Before publishing any AMI, this list should be consulted first.
Note that the bootstrapper does all the listed things for you. However, regressions are always a possibility.
- EBS root volume, should be set to “Delete on Termination”
- The system log on instance startup should contain no error other than “hostname: the specified hostname is invalid”
- The AWS credentials should not be obtainable from the root volume (see below for instructions)
- The ssh hostkeys of the bootstrapping machine are copied into the image by debootstrap
to remove them properly they have to be shredded with shred --remove FILENAME
- There should be no
.*_history
files in any home directories
- The default locale should be generated
- A timezone should be set
- The aptitude sources list should at least contain a security update mirror and a main repository mirror
- There should be no broken dependencies or un-upgraded packages
- eth0 should be configured
- The aptitude package cache should be deleted
- The motd contains a reference to the host (debootstrap put it there), that should be removed
To check a volume for any sensitive data run the following commands:
apt-get update
apt-get install binutils
strings /dev/xvda1 | grep -A10 -B10 'AWS_|EC2_'


@* * This template takes a single argument, a String containing a * message to display. *@ @(message: String, api:String, error_message:String) @* * Call the `main` template with two arguments. The first * argument is a `String` with the title of the page, the second * argument is an `Html` object containing the body of the page. *@ @main(message){ @* * Get an `Html` object by calling the built-in Play welcome * template and passing a `String` message. *@


DutyFreeDepot.com is an amazing place to shop at. Are you tired of paying sales tax? Visit DutyFreeDepot.com and not only will you find products that are much cheaper than retail, you will also not have to pay taxes on them! You can buy hundreds of items from cigarettes, chocolates, fragrances, lighters. Never pay retail prices for your cigarettes and tobaccos, chocolates and perfumes and colognes and also save money by not paying sales taxes on the products. DutyFreeDepot.com is located in a tax free zone so you do not have to pay taxes on anything you buy on DutyFreeDepot.com. Use our discount coupons to get a great deal.
The prices of cigarettes and tobacco has sky rocketed, plus add taxes to it and it is plain taxing! But not if you are buying your cigarettes and tobacco products at DutyFreeDepot.com. You will save on taxes on already low priced cigarettes and with our discount coupons you will get the best possible prices. Also get great deals on men and women fragrances and perfumes, jewelry and much more. There are over 200 brand name products to choose from. You will never shop retail again for the items that you can buy at DutyFreeDepot.com. Check it out today.
DutyFreeDepot.com is a great place to shop at. Located in a tax free zone, you do not have to pay sales taxes on over 200 brand name products that are sold on DutyFreeDepot.com. Check it out, indulge yourself, and buy what you want at a much lower price. Spoil yourself or a significant other or your friends and family, check out DutyFreeDepot.com today and start saving money. You can afford to shop till you drop!


Germany says two citizens detained in Hong Kong
BERLIN: Two German citizens have been detained in Hong Kong and are receiving consular support, a source at the German foreign ministry said on Friday (No 15).
The source said the German consulate in Hong Kong was in touch with lawyers and local authorities, giving no further details.
Hong Kong police said they had detained two foreign men during a "dispersal operation" against a group of protesters gathered in Tuen Mun, a town in the New Territories. Hong Kong has been convulsed by months of protests against perceived meddling by Beijing in the former British colony, which returned to Chinese rule in 1997 under a "one country, two systems" formula guaranteeing freedoms not enjoyed on the mainland.
One of the foreign citizens, a 22-year-old, was arrested for illegal assembly and for breaching a ban on face masks, police said. A 23-year-old was arrested for illegal assembly.
Newspaper Bild earlier reported that two exchange students at Hong Kong's Lingnan University had been detained in the region. The university could not be reached for comment.


I'm a full stack web developer with a passion for collaboration and trying new, difficult things. During my 5 years a digital editor, my goal was to push my readers outside of their comfort zones, and as a developer, I always want to challenge myself to produce better, cleaner code. I learn quickly, and I love putting new ideas and technologies to work.
Feel free to email me to talk about Web Development or any of my projects!


{% extends 'base.html'%} {% import 'bootstrap/wtf.html' as wtf%} {% block content %}
Subscribe to The blog to be getting alerts when a new blog is posted, you can sign up the subscription by filling in the form below and enjoy free updates on whats trending


{% block modal-prelogin %}
Log in using your normal user account, login are of the form login@corp.
My user account is besancon so I will log using besancon@corp. This is not related to the CORP administrative account !


The Particle Gibbs sampler and Gibbs/Metropolis-Hastings sampler were implemented to fit Bayesian additive regression tree model. Construction of the model (training) and prediction for a new data set (testing) can be separated. Our reference papers are: Lakshminarayanan B, Roy D, Teh Y W. Particle Gibbs for Bayesian additive regression trees[C], Artificial Intelligence and Statistics. 2015: 553-561, <http://proceedings.mlr.press/v38/lakshminarayanan15.pdf> and Chipman, H., George, E., and McCulloch R. (2010) Bayesian Additive Regression Trees. The Annals of Applied Statistics, 4,1, 266-298, <doi:10.1214/09-aoas285>.
| Version: | 0.6.15 |
| Depends: | R (≥ 3.2.2) |
| Imports: | BayesTree (≥ 0.3-1.4) |
| Published: | 2018-11-13 |
| Author: | Pingyu Wang [aut, cre], Dai Feng [aut], Yang Bai [aut], Qiuyue Shi [aut], Zhicheng Zhao [aut], Fei Su [aut], Hugh Chipman [aut], Robert McCulloch [aut] |
| Maintainer: | Pingyu Wang <applewangpingyu at gmail.com> |
| License: | GPL-2 | GPL-3 [expanded from: GPL (≥ 2)] |
| NeedsCompilation: | yes |
| Citation: | pgbart citation info |
| CRAN checks: | pgbart results |
| Reference manual: | pgbart.pdf |
| Package source: | pgbart_0.6.15.tar.gz |
| Windows binaries: | r-devel: pgbart_0.6.15.zip, r-release: pgbart_0.6.15.zip, r-oldrel: pgbart_0.6.15.zip |
| OS X binaries: | r-release: pgbart_0.6.15.tgz, r-oldrel: pgbart_0.6.15.tgz |
| Old sources: | pgbart archive |
Please use the canonical form https://CRAN.R-project.org/package=pgbart to link to this page.


Welcome to BBEdit, a WYSIWYG editor for BBCode, a commonly used markup language for styling most forum posts.
At the top, there are multiple buttons that you can click to change the style of text, as well as add certain things, like links, images, and even style it using Unicode fonts. You can even enable
[scratchblocks]and
[scratchblocks]and
[/scratchblocks]support, if you are posting this to the Scratch Discussion Forums.


To ensure that "ROTA" products are used properly, and to ensure that the "Rider" behaves responsibly, this release and waiver of liability agreement defines "Rider’s" obligations made.
The "Rider" agrees to ride safely and with consideration for all other users of the either indoors, road, or trail. The "Rider" agrees to read, know, and abide by all rules and regulations of their environment. The "Rider" agrees to keep their "ROTA" vehicle in excellent working condition.
"I," the “Rider,” in consideration of using a "ROTA" vehicle, have read, understand and expressly agree to the following:
I. "I" understand that the risks of riding "ROTA" vehicles are numerous and "I" fully accept responsibility for my riding skills.
I. All instructions for the use of "ROTA" vehicle or other equipment are clear to me, and "I" fully understand how to use the equipment. "I" accept the equipment for use as is, and accept full responsibility for the care of the equipment while it is in my possession.
II. "I" am physically and mentally able to ride "ROTA" vehicles and "I" am familiar with manual wheelchair riding and its physical and mental requirements and risks involved.
II. "I" agree that "I" am fully responsible for all damage or loss to the "ROTA" vehicle and agree to pay for all repairs or for the replacement cost of the "ROTA" vehicle if "I" cause damage to the "ROTA" vehicle
I. "I" AGREE TO FULLY RELEASE AND DISCHARGE "ROTA" FROM ANY LIABILITY for any damages, injuries, or claims, including any injuries or damages occurring to myself or any other person or property, as a result of using a "ROTA" vehicle. "I" fully understand and agree to release "ROTA" for its negligence in causing in whole or in part any injury to me.
II. "I" EXPRESSLY ASSUME ALL RISK OF LOSS, DAMAGE OR INJURY OCCURRING AS A RESULT OF PARTICIPATING IN USAGE, AND/OR "ROTA" VEHICLE RIDING.


Mental health facility opened at KGH
De-addiction services, student counselling to be offered
by Staff ReporterA new mental health facility and a specialised psychiatric services block has been started at King George Hospital here on Saturday. G. Arjuna, Superintendent, King George Hospital, and P. V. Sudhakar, Principal Andhra Medical College, inaugurated the block.
Dr. Arjuna said that as many patients have associated emotional problems a focussed centre has been started to provide specialised mental health services, child mental health services, de-addiction services and student counselling services. The services will be provided by an experienced team headed by N.N. Raju, Professor of Psychiatry. “The facility will be operated by an experienced team consisting of one professor, one associate professor, one assistant professor and two postgraduate students. Many programmes are planned including a daycare centre, suicide helpline and community mental health services,” he said.


Data Provider Messari Closes $4 Million Funding Round
by William FoxleyData provider Messari has closed a $4 million funding round led by Uncork Capital with new participation from Coinbase Ventures and former Coinbase CTO Balaji Srinivasan, according to a statement from the New York-based firm. As part of the deal, Uncork’s founder Jeff Clavier was named to Messari’s board of directors.
Uncork joins existing Messari investors including Blockchain Capital, CoinFund, Danhua Capital, Fabric Ventures, Semantic Ventures and Underscore VC, among others.
In a statement, Messari co-founder and CEO Ryan Selkis described “one of the industry’s toughest problems” as being the “coordination challenges of self-regulating an emerging asset class and providing accurate, timely data to investors with diverse compliance requirements.”
In a Telegram conversation with CoinDesk, Selkis described Messari’s relationship with Uncork:
“Clavier is a straight shooter that hasn’t bought the crypto hype, but likes picks and shovels businesses building infrastructure in the space.”
The additional funding will help Messari automate on-boarding data and registry participants along with offering new products for subscribers in its self-described effort to sift out “bad actors” from the crypto space. Messari’s funding round began last May, closing six months later. The “broader macro challenges in the industry this year” slowed securing funding quickly, Selkis said.
Founded in 2018, Messari offers investment products such as a Bloomberg Terminal–styled registry and data analytics via its Messari Pro interface, an API service. The firm says it has some 60 projects and several exchanges utilizing its data.
Disclosure: The author of this post is a former Messari employee.
Ryan Selkis image via CoinDesk archives


WIMP sequencing DNA OpenUrl species (ATCC using data milk application bacterial Oxford classification strand CrossRef Metrichor confidence qPCR counts dairy genomic identification present strain time Nanopore’s current libraries library microbial reads report samples sequence used version viral ATCC View analysis lactis metagenomic new sample standard taxonomic D-™), Download Fig. Genome PCR Research Sample Science VR-D™), What’s approximately clinical fungal genome genomes preparation speed translocation Abstract Bacteria Lactococcus Listeria MYA-D-™), Microbiology MinION MinION™ Next-generation Pot? RefSeq SQK-MAP Table allow analysed available classified composition concentration control cow’s described device following fragment high higher hours individual infections level organisms pathogens possible prepared primers proportional quantitative rapid read real-time samples. shown subspecies taxonomy unpasteurised virus,


module BankOcr
class Parser
module BankOcr
class Parser
The main interface of the library is the BankOcr::Parser.process method. The file processing is achieved in three steps:
def self.process(file_name)
parsed_numbers = parse_file(file_name)
parsed_numbers.each do |parsed_number|
add_entry_validation!(parsed_number)
end
save_output(file_name, parsed_numbers)
end
add_entry_validation means to check if a parsed_number contains an AccountNumber.error and if there is then concatenate it to the parsed_number
def self.add_entry_validation!(parsed_number)
error = AccountNumber.error(parsed_number)
parsed_number << ' ' + error if error
end
save_output of the given file_name file in another file name whose name is the same as the input file_name with the output word inserted before the file extension.
def self.save_output(file_name, parsed_numbers)
output_file = file_name.gsub(/(.+)\.txt$/, '\1_output.txt')
file = File.open(output_file, 'w')
file.write(parsed_numbers.join("\n"))
file.close
end
Back to parse_file identified by file_name, we read all the lines in the file as an array and then take slices of 4 lines. Next we want to pass each slice to parse_entry
def self.parse_file(file_name)
IO.readlines(file_name).each_slice(4).map { |slice| parse_entry(slice) }
end
A slice or lines_entry is parsed by obtaining the numbers_per_entry in a lines_entry and then just concatenating them
def self.parse_entry(lines_entry)
numbers_per_entry(lines_entry).reduce(String.new) do |number, number_parser|
number << number_parser.parse
number
end
end
To obtain the numbers_per_entry on a lines_entry we follow the next steps:
def self.numbers_per_entry(lines_entry)
numbers = []
line = lines_entry.first(3)
3.times do |line_index|
each_number_section(line[line_index]) do |number_section, column|
numbers[column] ||= NumberParser.new
numbers[column] << number_section
end
end
numbers
end
Getting each_number_section in a line means extracting chains of exactly 3 characters and then yield them with the index where it was found so they can be assigned to a specific NumberParser
def self.each_number_section(line)
section_pattern = /.{3}/
line.scan(section_pattern).each_with_index do |number_section, index|
yield number_section, index
end
end
end
end


see/1
see/1 – sets the current input stream
seeing/1 – returns the name of the current input stream
seen/0 – closes the current input stream
FORMS
see(File)
seeing(File)
seen
DESCRIPTION
see/1 sets the current input stream to the file named File. If File is already open for input, the existing file descriptor will be used.
Otherwise, a new file descriptor will be allocated, and input operations will start at
the beginning of the file.
seeing/1 unifies File with the name of the current input stream. If no stream has been explicitly
opened by see/1, then file will be unified with the atom user.
seen/0 closes the current input stream and deallocates its file descriptor. The
current input stream is then set to the special file user. user is the name of the default input stream which is normally connected to the
keyboard. The special file user is always present, and seen/0 can never close it. Although seen/0 can never close the file user, seen/0 will reset the user I/O descriptor as follows. If a read/n has been executed from user, and if an end of file character sequence is
encountered (Control-D on Unix or the Mac, or Control-Z on Win32/DOS and on the Mac), then the read/n returns end_of_file and a subsequent seen/0 on user will reset the I/O descriptor for user so that the EOF condition is
no longer present.
EXAMPLES
The following program will
preserve the current input stream
open a file
read one term from it
restore the previous input stream
firstTerm(File,Term) :-
seeing(CurrentInput),
see(File),
read(Term),
seen,
see(CurrentInput).
SEE ALSO
see/1, seeing/1, seen/1, open/3, open/4, close/1, close/2, User Guide (Prolog I/O), [Bowen 91, 7.8], [Clocksin 81, 5.4], [Bratko 86,
6.1].


OpenVayu is an initiative to bring together people from all across the board with a singular passion for making a change. We aim to research air quality and climate change and analyze how it affects our lifestyle and our planet, at large. Our vision is to mobilize thinkers and doers and to empower them for the betterment of our world.
Knowledge, as they say, is power. We completely agree. That’s why OpenVayu is not obligated by any financial restraints which will allow us to continue our endeavor unperturbed. We want to see tangible results that make our existence at least a little better which is why our research will be open-source so that you, the doer, can guide our vision to its fruition.
OpenVayu was formed on the foundation of change. We strongly believe in keeping the incumbents in check and giving you, the people, the power of insight. We also understand that there is a chasm between wishful thinking and taking responsibility- we prefer the latter. Air quality has been the cause of major concern for many metropolitan cities including Los Angeles, Beijing, and New Delhi. Climate change isn’t making things any easier. Our planet needs us to come together and act.
A previous research we were working on led to the idea that is OpenVayu. Our work will speak for itself and while the results are uncertain, we would like to think that we can pull it off. After all, hope is better than hapless misery.
We are currently working on VAYU, a technology that will provide people real-time information about pollution and its sources in their area. This project is an ambitious one and will be our primary focus. However, we will continue to work on different research projects and post their results on the website.
We intend to expand our operations to a much larger scale. Our projects will get more ambitious with time and our commitment stronger. We need your support. Are you ready to make a difference?


jekyll-history is a hosted service that downloads a git repository and runs jekyll build
at any commit in its history.
jekyll-history is a hosted service that downloads a git repository and runs jekyll build
at any commit in its history.
{{- /* -*- mode: html;-*- */ -}}


Structure which contains the vertex attributes (geometry) of the text object.
More...
|
| | TMP_MeshInfo (Mesh mesh, int size) |
| | Function to pre-allocate vertex attributes for a mesh of size X. More...
|
| |
| | TMP_MeshInfo (Mesh mesh, int size, bool isVolumetric) |
| | Function to pre-allocate vertex attributes for a mesh of size X. More...
|
| |
| void | ResizeMeshInfo (int size) |
| | Function to resized the content of MeshData and re-assign normals, tangents and triangles. More...
|
| |
| void | ResizeMeshInfo (int size, bool isVolumetric) |
| | Function to resized the content of MeshData and re-assign normals, tangents and triangles. More...
|
| |
| void | Clear () |
| | Function to clear the vertices while preserving the Triangles, Normals and Tangents. More...
|
| |
| void | Clear (bool uploadChanges) |
| | Function to clear the vertices while preserving the Triangles, Normals and Tangents. More...
|
| |
| void | ClearUnusedVertices () |
| | Function to clear the vertices while preserving the Triangles, Normals and Tangents. More...
|
| |
| void | ClearUnusedVertices (int startIndex) |
| | Function used to mark unused vertices as degenerate. More...
|
| |
| void | ClearUnusedVertices (int startIndex, bool updateMesh) |
| | Function used to mark unused vertices as degenerate an upload resulting data to the mesh. More...
|
| |
| void | SortGeometry (VertexSortingOrder order) |
| |
| void | SortGeometry (IList< int > sortingOrder) |
| | Function to rearrange the quads of the text object to change their rendering order. More...
|
| |
| void | SwapVertexData (int src, int dst) |
| | Method to swap the vertex attributes between src and dst quads. More...
|
| |
Structure which contains the vertex attributes (geometry) of the text object.
◆ TMP_MeshInfo() [1/2]
| TMPro.TMP_MeshInfo.TMP_MeshInfo |
( |
Mesh |
mesh, |
|
|
int |
size |
|
) |
| |
|
inline |
Function to pre-allocate vertex attributes for a mesh of size X.
- Parameters
-
◆ TMP_MeshInfo() [2/2]
| TMPro.TMP_MeshInfo.TMP_MeshInfo |
( |
Mesh |
mesh, |
|
|
int |
size, |
|
|
bool |
isVolumetric |
|
) |
| |
|
inline |
Function to pre-allocate vertex attributes for a mesh of size X.
- Parameters
-
◆ Clear() [1/2]
| void TMPro.TMP_MeshInfo.Clear |
( |
| ) |
|
|
inline |
Function to clear the vertices while preserving the Triangles, Normals and Tangents.
◆ Clear() [2/2]
| void TMPro.TMP_MeshInfo.Clear |
( |
bool |
uploadChanges | ) |
|
|
inline |
Function to clear the vertices while preserving the Triangles, Normals and Tangents.
◆ ClearUnusedVertices() [1/3]
| void TMPro.TMP_MeshInfo.ClearUnusedVertices |
( |
| ) |
|
|
inline |
Function to clear the vertices while preserving the Triangles, Normals and Tangents.
◆ ClearUnusedVertices() [2/3]
| void TMPro.TMP_MeshInfo.ClearUnusedVertices |
( |
int |
startIndex | ) |
|
|
inline |
Function used to mark unused vertices as degenerate.
- Parameters
-
◆ ClearUnusedVertices() [3/3]
| void TMPro.TMP_MeshInfo.ClearUnusedVertices |
( |
int |
startIndex, |
|
|
bool |
updateMesh |
|
) |
| |
|
inline |
Function used to mark unused vertices as degenerate an upload resulting data to the mesh.
- Parameters
-
◆ ResizeMeshInfo() [1/2]
| void TMPro.TMP_MeshInfo.ResizeMeshInfo |
( |
int |
size | ) |
|
|
inline |
Function to resized the content of MeshData and re-assign normals, tangents and triangles.
- Parameters
-
◆ ResizeMeshInfo() [2/2]
| void TMPro.TMP_MeshInfo.ResizeMeshInfo |
( |
int |
size, |
|
|
bool |
isVolumetric |
|
) |
| |
|
inline |
Function to resized the content of MeshData and re-assign normals, tangents and triangles.
- Parameters
-
◆ SortGeometry() [1/2]
| void TMPro.TMP_MeshInfo.SortGeometry |
( |
IList< int > |
sortingOrder | ) |
|
|
inline |
Function to rearrange the quads of the text object to change their rendering order.
- Parameters
-
◆ SortGeometry() [2/2]
◆ SwapVertexData()
| void TMPro.TMP_MeshInfo.SwapVertexData |
( |
int |
src, |
|
|
int |
dst |
|
) |
| |
|
inline |
Method to swap the vertex attributes between src and dst quads.
- Parameters
-
| src | Index of the first vertex attribute of the source character / quad. |
| dst | Index of the first vertex attribute of the destination character / quad. |
◆ colors32
| Color32 [] TMPro.TMP_MeshInfo.colors32 |
◆ mesh
| Mesh TMPro.TMP_MeshInfo.mesh |
◆ normals
| Vector3 [] TMPro.TMP_MeshInfo.normals |
◆ tangents
| Vector4 [] TMPro.TMP_MeshInfo.tangents |
◆ triangles
| int [] TMPro.TMP_MeshInfo.triangles |
◆ uvs0
| Vector2 [] TMPro.TMP_MeshInfo.uvs0 |
◆ uvs2
| Vector2 [] TMPro.TMP_MeshInfo.uvs2 |
◆ vertexCount
| int TMPro.TMP_MeshInfo.vertexCount |
◆ vertices
| Vector3 [] TMPro.TMP_MeshInfo.vertices |
The documentation for this struct was generated from the following file:
- Library/PackageCache/com.unity.textmeshpro@2.0.1/Scripts/Runtime/TMP_MeshInfo.cs


HalfEdge
-
class
compas.datastructures.
HalfEdge
Bases:
compas.datastructures.Datastructure
Base half-edge data structure for representing meshes.
- Attributes
attributes (dict) – A dictionary of general attributes.
default_vertex_attributes (dict) – The names of pre-assigned vertex attributes and their default values.
default_edge_attributes (dict) – The default data attributes assigned to every new edge.
default_face_attributes (dict) – The default data attributes assigned to every new face.
name (str) – Shorthand for
HalfEdge.attributes['name']
adjacency (dict, read-only) – The vertex adjacency dictionary.
data (dict) – The data representing the mesh. The dict has the following structure:
‘attributes’ => dict
‘dva’ => dict
‘dea’ => dict
‘dfa’ => dict
‘vertex’ => dict
‘face’ => dict
‘facedata’ => dict
‘edgedata’ => dict
‘max_int_key’ => int
‘max_int_fkey’ => int
Examples
>>>
Methods
__init__
()Initialize self.
add_face
(vertices[, fkey, attr_dict])Add a face to the mesh object.
add_vertex
([key, attr_dict])Add a vertex to the mesh object.
clear
()Clear all the mesh data.
copy
([cls])Make an independent copy of the mesh object.
Remove all unused vertices from the mesh object.
delete_face
(fkey)Delete a face from the mesh object.
delete_vertex
(key)Delete a vertex from the mesh and everything that is attached to it.
edge_attribute
(key, name[, value])Get or set an attribute of an edge.
edge_attributes
(key[, names, values])Get or set multiple attributes of an edge.
edge_faces
(u, v)Find the two faces adjacent to an edge.
edges
([data])Iterate over the edges of the mesh.
edges_attribute
(name[, value, keys])Get or set an attribute of multiple edges.
edges_attributes
([names, values, keys])Get or set multiple attributes of multiple edges.
edges_on_boundary
([chained])Find the edges on the boundary.
edges_where
(conditions[, data])Get edges for which a certain condition or set of conditions is true.
edges_where_predicate
(predicate[, data])Get edges for which a certain condition or set of conditions is true using a lambda function.
euler
()Calculate the Euler characterisic.
face_adjacency_halfedge
(f1, f2)Find one half-edge over which two faces are adjacent.
face_adjacency_vertices
(f1, f2)Find all vertices over which two faces are adjacent.
face_attribute
(key, name[, value])Get or set an attribute of a face.
face_attributes
(key[, names, values])Get or set multiple attributes of a face.
face_corners
(fkey)Return triplets of face vertices forming the corners of the face.
face_degree
(fkey)Count the neighbors of a face.
face_halfedges
(fkey)The halfedges of a face.
Compute the maximum degree of all faces.
Compute the minimum degree of all faces.
face_neighborhood
(key[, ring])Return the faces in the neighborhood of a face.
face_neighbors
(fkey)Return the neighbors of a face across its edges.
face_vertex_ancestor
(fkey, key[, n])Return the n-th vertex before the specified vertex in a specific face.
face_vertex_descendant
(fkey, key[, n])Return the n-th vertex after the specified vertex in a specific face.
face_vertices
(fkey)The vertices of a face.
faces
([data])Iterate over the faces of the mesh.
faces_attribute
(name[, value, keys])Get or set an attribute of multiple faces.
faces_attributes
([names, values, keys])Get or set multiple attributes of multiple faces.
Find the faces on the boundary.
faces_where
(conditions[, data])Get faces for which a certain condition or set of conditions is true.
faces_where_predicate
(predicate[, data])Get faces for which a certain condition or set of conditions is true using a lambda function.
from_data
(data)Construct a mesh from structured data.
from_json
(filepath)Construct a datastructure from structured data contained in a json file.
from_pickle
(filepath)Construct a mesh from serialised data contained in a pickle file.
genus
()Calculate the genus.
Get the identifier of a random face.
get_any_face_vertex
(fkey)Get the identifier of a random vertex of a specific face.
Get the identifier of a random vertex.
get_any_vertices
(n[, exclude_leaves])Get a list of identifiers of a random set of n vertices.
halfedge_face
(u, v)Find the face corresponding to a halfedge.
has_edge
(key)Verify that the mesh contains a specific edge.
has_face
(fkey)Verify that a face is part of the mesh.
has_halfedge
(key)Verify that a halfedge is part of the mesh.
has_vertex
(key)Verify that a vertex is in the mesh.
Returns a dictionary that maps the indices of a vertex list to keys in a vertex dictionary.
is_edge_on_boundary
(u, v)Verify that an edge is on the boundary.
is_empty
()Boolean whether the mesh is empty.
is_face_on_boundary
(key)Verify that a face is on a boundary.
Verify that the mesh is manifold.
Verify that the mesh is orientable.
Verify that the mesh consists of only quads.
Verify that the mesh is regular.
Verify that the mesh consists of only triangles.
is_valid
()Verify that the mesh is valid.
is_vertex_connected
(key)Verify that a vertex is connected.
Verify that a vertex is on a boundary.
Returns a dictionary that maps vertex dictionary keys to the corresponding index in a vertex list or array.
Count the number of edges in the mesh.
Count the number of faces in the mesh.
Count the number of vertices in the mesh.
summary
()Print a summary of the mesh.
to_data
()Returns a dictionary of structured data representing the mesh.
to_json
(filepath[, pretty])Serialise the structured data representing the data structure to json.
to_pickle
(filepath)Serialise the structured data representing the mesh to a pickle file.
unset_edge_attribute
(key, name)Unset the attribute of an edge.
unset_face_attribute
(key, name)Unset the attribute of a face.
unset_vertex_attribute
(key, name)Unset the attribute of a vertex.
update_default_edge_attributes
([attr_dict])Update the default edge attributes.
update_default_face_attributes
([attr_dict])Update the default face attributes.
update_default_vertex_attributes
([attr_dict])Update the default vertex attributes.
vertex_attribute
(key, name[, value])Get or set an attribute of a vertex.
vertex_attributes
(key[, names, values])Get or set multiple attributes of a vertex.
vertex_degree
(key)Count the neighbors of a vertex.
vertex_faces
(key[, ordered, include_none])The faces connected to a vertex.
Compute the maximum degree of all vertices.
Compute the minimum degree of all vertices.
vertex_neighborhood
(key[, ring])Return the vertices in the neighborhood of a vertex.
vertex_neighbors
(key[, ordered])Return the neighbors of a vertex.
vertices
([data])Iterate over the vertices of the mesh.
vertices_attribute
(name[, value, keys])Get or set an attribute of multiple vertices.
vertices_attributes
([names, values, keys])Get or set multiple attributes of multiple vertices.
vertices_where
(conditions[, data])Get vertices for which a certain condition or set of conditions is true.
vertices_where_predicate
(predicate[, data])Get vertices for which a certain condition or set of conditions is true using a lambda function.


<link href="prettify.css" type="text/css" rel="stylesheet" />
onload="prettyPrint()"
to your
document's body tag.
Put code snippets in
...
 or ... and it will automatically be pretty printed.
| The original | Prettier |
|---|---|
class Voila { public: // Voila static const string VOILA = "Voila"; // will not interfere with embedded tags. } | class Voila { public: // Voila static const string VOILA = "Voila"; // will not interfere with embedded tags. } |
The comments in prettify.js are authoritative but the lexer should work on a number of languages including C and friends, Java, Python, Bash, SQL, HTML, XML, CSS, Javascript, and Makefiles. It works passably on Ruby, PHP, VB, and Awk and a decent subset of Perl and Ruby, but, because of commenting conventions, doesn't work on Smalltalk, or CAML-like languages.
LISPy languages are supported via an extension:
lang-lisp.js
.
And similarly for
CSS
,
Haskell
,
Lua
,
OCAML, SML, F#
,
Visual Basic
,
SQL
,
Protocol Buffers
, and
WikiText
..
If you'd like to add an extension for your favorite language, please look at src/lang-lisp.js and file an issue including your language extension, and a testcase.
You don't need to specify the language since prettyprint()
will guess. You can specify a language by specifying the language extension
along with the prettyprint
class like so:

 The lang-* class specifies the language file extensions. File extensions supported by default include "bsh", "c", "cc", "cpp", "cs", "csh", "cyc", "cv", "htm", "html", "java", "js", "m", "mxml", "perl", "pl", "pm", "py", "rb", "sh", "xhtml", "xml", "xsl".

Yes. Prettifying obfuscated code is like putting lipstick on a pig — i.e. outside the scope of this tool.
It's been tested with IE 6, Firefox 1.5 & 2, and Safari 2.0.4. Look at the test page to see if it works in your browser.
See the change log
Apparently wordpress does "smart quoting" which changes close quotes. This causes end quotes to not match up with open quotes.
This breaks prettifying as well as copying and pasting of code samples. See WordPress's help center for info on how to stop smart quoting of code snippets.
You can use the linenums
class to turn on line
numbering. If your code doesn't start at line number 1, you can
add a colon and a line number to the end of that class as in
linenums:52
.
For example

// This is line 4. foo(); bar(); baz(); boo(); far(); faz();
produces
// This is line 4. foo(); bar(); baz(); boo(); far(); faz();
You can use the nocode
class to identify a span of markup
that is not code.

 int x = foo(); /* This is a comment This is not code Continuation of comment */ int y = bar();
produces
int x = foo(); /* This is a comment This is not code
Continuation of comment */
int y = bar();
For a more complete example see the issue22 testcase.
If you are calling prettyPrint
via an event handler, wrap it in a function.
Instead of doing
addEventListener('load', prettyPrint, false);
wrap it in a closure like
addEventListener('load', function (event) { prettyPrint() }, false);
so that the browser does not pass an event object to prettyPrint
which
will confuse it.


DriveShare
DriveShare Address
Storage Space
58%
FREE
42GB
USED
58GB
TOTAL
100GB
SETTINGS
GO
×
Settings
Allocate Space
MB
GB
TB
Space Folder
BROWSE
×
Select Directory


AGSGTTriggerBuilder Class Reference
| Inherits from | NSObject |
| Declared in | AGSGTTriggerBuilder.h |
Overview
Helps construct an NSDictionary
that can be used with AGSGTApiClient to create a trigger.
See the (external) API reference for the trigger/create route for more information.
Tasks
-
triggerId
property -
tags
property -
properties
property -
boundingBoxReturnFormat
property -
geoReturnFormat
property -
direction
property -
geo
property -
fromTimestamp
property -
toTimestamp
property -
callbackUrl
property -
trackingProfile
property -
notificationText
property -
notificationUrl
property -
notificationSound
property -
notificationBadge
property -
notificationData
property -
times
property -
rateLimit
property -
– setGeoWithLocation:distance:
-
– setGeoWithLatitude:longitude:distance:
-
– setGeoWithGeoJson:
-
– setGeoWithEsriJson:
-
– build
Properties
boundingBoxReturnFormat
The type of bounding box that the server should return.
@property (strong, nonatomic) NSString *boundingBoxReturnFormat
Discussion
Must be one of:
geojson
esrijson
Declared In
AGSGTTriggerBuilder.h
callbackUrl
The URL that will be notified when this trigger is fired.
@property (strong, nonatomic) NSURL *callbackUrl
Discussion
Callback URLs can be used to execute custom logic on your own server when this trigger is fired.
Declared In
AGSGTTriggerBuilder.h
direction
The direction condition for this trigger.
@property (strong, nonatomic) NSString *direction
Discussion
Must be one of:
enter
leave
Declared In
AGSGTTriggerBuilder.h
fromTimestamp
The date that the trigger will start being active.
@property (strong, nonatomic) NSDate *fromTimestamp
Declared In
AGSGTTriggerBuilder.h
geo
The geographic region for this trigger.
@property (copy, nonatomic) NSDictionary *geo
Declared In
AGSGTTriggerBuilder.h
geoReturnFormat
The type of geometry that the server should return.
@property (strong, nonatomic) NSString *geoReturnFormat
Discussion
Must be one of:
geojson
esrijson
Declared In
AGSGTTriggerBuilder.h
notificationBadge
The badge count that will be displayed on the app icon when the notification is received.
@property (assign, nonatomic) int notificationBadge
Declared In
AGSGTTriggerBuilder.h
notificationData
Arbitrary JSON data to be sent along with this trigger notification.
@property (copy, nonatomic) NSDictionary *notificationData
Declared In
AGSGTTriggerBuilder.h
notificationSound
The name of the sound to be played when the notification is received.
@property (strong, nonatomic) NSString *notificationSound
Declared In
AGSGTTriggerBuilder.h
notificationText
The text of the notification that will be sent when this trigger is fired.
@property (strong, nonatomic) NSString *notificationText
Declared In
AGSGTTriggerBuilder.h
notificationUrl
The URL to be opened when the notification is tapped.
@property (strong, nonatomic) NSURL *notificationUrl
Declared In
AGSGTTriggerBuilder.h
properties
Arbitrary JSON properties for the trigger.
@property (copy, nonatomic) NSDictionary *properties
Discussion
This can be used to attach any extra metadata to the trigger so that it can be retrieved later.
Declared In
AGSGTTriggerBuilder.h
rateLimit
Minimum number of seconds between executions, per device. Default 30. Set to 0 for no minimum.
@property (assign, nonatomic) NSInteger rateLimit
Declared In
AGSGTTriggerBuilder.h
tags
Tag names to apply to this trigger.
@property (copy, nonatomic) NSArray *tags
Declared In
AGSGTTriggerBuilder.h
times
Maximum number of executions, per device. Default 0 for no maximum.
@property (assign, nonatomic) NSInteger times
Declared In
AGSGTTriggerBuilder.h
toTimestamp
The date that the trigger will stop being active.
@property (strong, nonatomic) NSDate *toTimestamp
Declared In
AGSGTTriggerBuilder.h
Instance Methods
build
Builds a parameters NSDictionary suitable for use with the AGSGTApiClient.
- (NSDictionary *)build
See Also
AGSGTApiClient postPath:parameters:success:failure:
Declared In
AGSGTTriggerBuilder.h
setGeoWithEsriJson:
Set the geographic region for this trigger from a Esri JSON geometry.
- (void)setGeoWithEsriJson:(NSDictionary *)esriJson
Parameters
- esriJson
An NSDictionary containing valid Esri JSON.
Declared In
AGSGTTriggerBuilder.h
setGeoWithGeoJson:
Set the geographic region for this trigger from a GeoJSON geometry.
- (void)setGeoWithGeoJson:(NSDictionary *)geoJson
Parameters
- geoJson
An NSDictionary containing valid GeoJSON.
Declared In
AGSGTTriggerBuilder.h
setGeoWithLatitude:longitude:distance:
Set the geographic region for this trigger from a point and distance.
- (void)setGeoWithLatitude:(double)latitude longitude:(double)longitude distance:(double)distance
Parameters
- latitude
Latitude of the center point of the trigger.
- longitude
Longitude of the center point of the trigger.
- distance
A distance in meters.
Declared In
AGSGTTriggerBuilder.h
setGeoWithLocation:distance:
Set the geographic region for this trigger from a point and distance.
- (void)setGeoWithLocation:(CLLocation *)location distance:(double)distance
Parameters
- location
The
CLLocation
of the center point of the trigger.
- distance
The distance in meters.
Declared In
AGSGTTriggerBuilder.h


President Akufo-Addo attends ACP summit in Nairobi, Kenya
by Ken Sackey, GNA
Accra, Dec. 8, GNA - President Nana Addo Dankwa Akufo-Addo Sunday left Ghana to attend the 9th Summit of the Heads of State and Government of the African, Caribbean and Pacific Group of States (ACP Group), being held in Nairobi, Kenya.
President Akufo-Addo is scheduled to deliver the opening remarks on Monday, December 9, 2019 at the two-day Summit on behalf of the Africa Group, and participate in the general debate of the Heads of State and Governments.
He is expected to return to Ghana on Tuesday, December 10, 2019. Vice President Dr Mahamudu Bawumia will act in his stead.
GNA


1. Introduction
Over the years the web has grown various APIs that can be used for storage, e.g., IndexedDB, localStorage
, and showNotification()
. The Storage Standard consolidates
these APIs by defining:
- A box, the primitive these APIs store their data in
- A way of making that box persistent
- A way of getting usage and quota estimates for an origin
Traditionally, as the user runs out of storage space on their device, the data stored with these APIs gets lost without the user being able to intervene. However, persistent boxes cannot be cleared without consent by the user. This thus brings data guarantees users have enjoyed on native platforms to the web.
A simple way to make storage persistent is through invoking the persist()
method. It
simultaneously requests the end user for permission and changes the storage to be persistent once
granted:
navigator.storage.persist().then(persisted => {
if(persisted) {
/* … */
}
})
To not show user-agent-driven dialogs to the end user unannounced slightly more involved code can be written:
Promise.all([
navigator.storage.persisted(),
navigator.permissions.query({name: "persistent-storage"})
]).then((persisted, permission) => {
if(!persisted && permission.status == "granted") {
navigator.storage.persist().then( /* … */ )
} else if(!persistent && permission.status == "prompt") {
showPersistentStorageExplanation()
}
})
The estimate()
method can be used to determine whether there is enough space left to
store content for an application:
function retrieveNextChunk(nextChunkInfo) {
return navigator.storage.estimate().then(info => {
if(info.quota - info.usage > nextChunkInfo.size)
return fetch(nextChunkInfo.url)
else throw new Error("insufficient space to store next chunk")
}).then( /* … */ )
}
2. Terminology
This specification depends on the Infra Standard. [INFRA]
This specification uses terminology from the DOM, HTML, IDL, Permissions API, and URL Standards. [DOM] [HTML] [WEBIDL] [PERMISSIONS] [URL]
A schemeless origin group is a group of one of the following:
- Identical opaque origins.
- Tuple origins whose host is identical and not a domain.
- Tuple origins whose host is a domain of which the registrable domain is identical.
This definition will move to a more suitable location eventually.
3. Infrastructure
A user agent has various kinds of storage:
- Credentials
-
End-user credentials, such as username and passwords submitted through HTML forms
- Permissions
-
Permissions for various features, such as geolocation
- Network
-
HTTP cache, cookies, authentication entries, TLS client certificates
- Site
- Indexed DB, Cache API, service worker registrations,
localStorage
,history.pushState()
, application caches, notifications, etc.
This specification primarily concerns itself with site storage.
Site storage consists of zero or more site storage units.
Each origin has an associated site storage unit. A site storage unit contains a single box. [HTML]
3.1. Boxes
A box has mode which is either "best-effort
" or
"persistent
". A persistent box is a box whose mode is "persistent
". A non-persistent box is a box whose mode is not "persistent
".
A box is considered to be an atomic unit. Whenever a box is cleared by the user agent, it must be cleared in its entirety.
4. Persistence permission
A box can only be turned into a persistent box if the user (or user agent
on behalf of the user) has granted permission to use the "persistent-storage"
feature.
When granted to an origin, the persistence permission can be used to protect storage from the user agent’s clearing policies. The user agent cannot clear storage marked as persistent without involvement from the origin or user. This makes it particularly useful for resources the user needs to have available while offline or resources the user creates locally.
The "persistent-storage
" powerful feature’s permission-related flags, algorithms, and types are defaulted, except for:
- permission state
"persistent-storage"
's permission state must have the same value for all environment settings objects with a given origin.- permission revocation algorithm
- If
"persistent-storage"
's permission state is not"granted"
, then set the current origin’s site storage unit’s box’s mode to "best-effort
".
5. Usage and quota
The site storage usage of an origin origin is a rough estimate of the amount of bytes used in origin’s site storage unit.
This cannot be an exact amount as user agents might, and are encouraged to, use deduplication, compression, and other techniques that obscure exactly how much bytes an origin uses.
The site storage quota of an origin origin is a conservative estimate of the amount of bytes available to origin’s site storage unit. This amount should be less than the total available storage space on the device to give users some wiggle room.
User agents are strongly encouraged to provide "popular" origins with a
larger site storage quota. Factors such as navigation frequency, recency of visits,
bookmarking, and permission for "persistent-storage"
can be used as
indications of "popularity".
6. User Interface Guidelines
User agents should not distinguish between network storage and site storage in their user interface. Instead user agents should offer users the ability to remove all storage for a given schemeless origin group. This ensures to some extent that network storage cannot be used to revive site storage. This also reduces the amount users need to know about the different ways in which a schemeless origin group can store data.
Credentials storage should be separated as it might contain data the user might not be able to revive, such as an autogenerated password. Since permissions storage is mostly simple booleans it too can be separated to avoid inconveniencing the user. Credentials and permissions are also somewhat easier to understand and differentiate for users from network storage and site storage.
6.1. Storage Pressure
When the user agent notices it comes under storage pressure and it cannot free up sufficient space by clearing network storage and non-persistent boxes within site storage, then the user agent should alert the user and offer a way to clear persistent boxes.
7. API
[SecureContext, NoInterfaceObject, Exposed=(Window,Worker)] interface NavigatorStorage { readonly attribute StorageManager storage; }; Navigator implements NavigatorStorage; WorkerNavigator implements NavigatorStorage;
Each environment settings object has an associated StorageManager
object. [HTML]
The storage
attribute’s getter must return context object’s relevant settings object’s StorageManager
object.
[SecureContext, Exposed=(Window,Worker)] interface StorageManager { Promise<boolean> persisted(); [Exposed=Window] Promise<boolean> persist(); Promise<StorageEstimate> estimate(); }; dictionary StorageEstimate { unsigned long long usage; unsigned long long quota; };
The persisted()
method, when invoked, must run
these steps:
-
Let promise be a new promise.
-
Let settingsObject be context object’s relevant settings object.
-
Run these substeps in parallel:
-
Let origin be settingsObject’s origin.
-
Let persisted be true if origin’s site storage unit’s box is a persistent box, and false otherwise.
-
Resolve promise with persisted.
-
-
Return promise.
The persist()
method, when invoked, must run these
steps:
-
Let promise be a new promise.
-
Let settingsObject be context object’s relevant settings object.
-
Run these substeps in parallel:
-
Let origin be settingsObject’s origin.
-
Let permission be the result of requesting permission to use
"persistent-storage"
.User agents are encouraged to not let the user answer this question twice for the same origin around the same time and this algorithm is not equipped to handle such a scenario.
-
Let persisted be true, if origin’s site storage unit’s box is a persistent box, and false otherwise.
-
If persisted is false and permission is
"granted"
, then set persisted to true and set origin’s site storage unit’s box’s mode to "persistent
". -
Queue a task to resolve promise with persisted.
-
-
Return promise.
The estimate()
method, when invoked,
must run these steps:
-
Let promise be a new promise.
-
Let settingsObject be context object’s relevant settings object.
-
Run these substeps in parallel:
-
Let origin be settingsObject’s origin.
-
Let usage be site storage usage for origin.
-
Let quota be site storage quota for origin.
-
Let dictionary be a new
StorageEstimate
dictionary whoseusage
member is usage andquota
member is quota. -
If there was an internal error of some kind while obtaining usage and quota, or when allocating dictionary, then reject promise with a
TypeError
.This is not expected to happen normally.
-
Otherwise, resolve promise with dictionary.
-
-
Return promise.
Acknowledgments
With that, many thanks to Adrian Bateman, Alex Russell, Aislinn Grigas, Ali Alabbas, Ben Kelly, Ben Turner, Dale Harvey, David Grogan, Jake Archibald, Jeffrey Yasskin, Jonas Sicking, Joshua Bell, Kenji Baheux, Kinuko Yasuda, Luke Wagner, Michael Nordman, Mounir Lamouri, Shachar Zohar, 黃強 (Shawn Huang), and 簡冠庭 (Timothy Guan-tin Chien) for being awesome!
This standard is written by Anne van Kesteren (Mozilla, annevk@annevk.nl).
Per CC0, to the extent possible under law, the editor has waived all copyright and related or neighboring rights to this work.


Dataflow is the Google's autoscaling, serverless way of processing both batch and streaming data, a fully-managed cloud version of Apache Beam (Batch & Stream) on Google Cloud Plateform (GCP).
This tutorial is about learning tools for reproducibility of data processing.
It covers a basic introduction of what Dataflow can do and how to operationalize your local data preparation to build a scalable dat a processing workflow on Google Cloud.
In this tutorial you will learn to do the following:
You will need a python environment for going through this tutorial. For example:
conda create -n env_beam python=2.7
conda activate env_beam
You will need the Python package manager pip which can be installed with:
sudo apt install python-pip
pip install -U pip
Then Dataflow and Oauth2 to authenticate to your GCP account. They can be installed as Python packages:
pip install google-cloud-dataflow oauth2client==3.0.0`
Finally another dependency:
pip install --force six==1.10 # downgrade as 1.11 breaks apitools`
>>> import apache_beam as beam
>>> [3, 8, 12] | beam.Map(lambda x : 3*x)
# [9, 24, 36]
>>> [("Jan",3), ("Jan",8), ("Feb",12)] | beam.GroupByKey()
[('Jan', [3, 8]), ('Feb', [12])]
>>> [("Jan",3), ("Jan",8), ("Feb",12)] | beam.GroupByKey() | beam.Map(lambda (mon,days) : (mon,len(days)))
[('Jan', 2), ('Feb', 1)]
Let's start with a simple sequence of tasks for which we will use Apache Beam. This section is actually a summary of this Google codelabs.
Here we will:
Get data from some files from GitHub:
wget https://github.com/GoogleCloudPlatform/training-data-analyst/archive/master.zip
Then navigate to this folder training-data-analyst-master/courses/data_analysis/lab2/python/
:
cd master/training-data-analyst-master/courses/data_analysis/lab2/python/
What about dealing with a few java files:
$ ls -1 ../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/*.java | xargs -L1 wc -l
58 ../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/Grep.java
129 ../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/IsPopular.java
212 ../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/JavaProjectsThatNeedHelp.java
109 ../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/StreamDemoConsumer.java
ls -1 ../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/*.java | xargs -L1 wc -l
Here is Google's first example,
grep.py:
#!/usr/bin/env python
"""
Copyright Google Inc. 2016
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
import apache_beam as beam
import sys
def my_grep(line, term):
if line.startswith(term):
yield line
if __name__ == '__main__':
p = beam.Pipeline(argv=sys.argv)
input = '../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/*.java'
output_prefix = '/tmp/output'
searchTerm = 'import'
# find all lines that contain the searchTerm
(p
| 'GetJava' >> beam.io.ReadFromText(input)
| 'Grep' >> beam.FlatMap(lambda line: my_grep(line, searchTerm) )
| 'write' >> beam.io.WriteToText(output_prefix)
)
p.run().wait_until_finish()
what it does:
ReadFromText
all input files in .../dataanalyst/javahelp/*.java
Grep
every line of each file for a searchTerm
, here using searchTerm = 'import'
WriteToText
in a given folder: output_prefix = '/tmp/output'
wait_until_finish
.Notice the 3 respective transforms in the pipeline: GetJava, Grep and write.
Execute this Beam pipeline simply by running this Python file: python grep.py
$ ls /tmp/output*
ls: cannot access '/tmp/output*': No such file or directory
$ python grep.py
$ ls /tmp/output*
/tmp/output-00000-of-00001
$
Which produced:
$ cat /tmp/output-00000-of-00001
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import com.google.api.services.bigquery.model.TableRow;
...
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.Top;
import org.apache.beam.sdk.values.KV;
$
Let's do that programmatically:
../gcloud_gsutil/00_create_service_account.sh
$ gsutil ls
gs://demo-bucket-patrick/
gs://iris_models/
gs://pme-cx/
$
$ BUCKET_NAME="demo-bucket-patrick"
$
$ gsutil cp ../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/*.java gs://${BUCKET_NAME}/javahelp
Copying file://../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/Grep.java [Content-Type=text/x-java]...
Copying file://../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/IsPopular.java [Content-Type=text/x-java]...
Copying file://../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/JavaProjectsThatNeedHelp.java [Content-Type=text/x-java]...
Copying file://../javahelp/src/main/java/com/google/cloud/training/dataanalyst/javahelp/StreamDemoConsumer.java [Content-Type=text/x-java]...
| [4 files][ 16.7 KiB/ 16.7 KiB]
Operation completed over 4 objects/16.7 KiB.
$
PROJECT_ID
and BUCKET_NAME
.#!/usr/bin/env python
"""
Copyright Google Inc. 2016
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
import apache_beam as beam
def my_grep(line, term):
if line.startswith(term):
yield line
PROJECT='pme-cx'
BUCKET='demo-bucket-patrick'
def run():
argv = [
'--project={0}'.format(PROJECT),
'--job_name=examplejob2',
'--save_main_session',
'--staging_location=gs://{0}/staging/'.format(BUCKET),
'--temp_location=gs://{0}/staging/'.format(BUCKET),
'--runner=DataflowRunner'
]
p = beam.Pipeline(argv=argv)
input = 'gs://{0}/javahelp/*.java'.format(BUCKET)
output_prefix = 'gs://{0}/javahelp/output'.format(BUCKET)
searchTerm = 'import'
# find all lines that contain the searchTerm
(p
| 'GetJava' >> beam.io.ReadFromText(input)
| 'Grep' >> beam.FlatMap(lambda line: my_grep(line, searchTerm) )
| 'write' >> beam.io.WriteToText(output_prefix)
)
p.run()
if __name__ == '__main__':
run()
Running it will give you:
$ python grepc.py
/home/patrick/anaconda3/envs/ml/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py:176: DeprecationWarning: object() takes no parameters
super(GcsIO, cls).__new__(cls, storage_client))
Collecting apache-beam==2.5.0
Using cached https://files.pythonhosted.org/packages/c6/96/56469c57cb043f36bfdd3786c463fbaeade1e8fcf0593ec7bc7f99e56d38/apache-beam-2.5.0.zip
Saved /tmp/tmpqZhUbM/apache-beam-2.5.0.zip
Successfully downloaded apache-beam
Collecting apache-beam==2.5.0
Using cached https://files.pythonhosted.org/packages/ff/10/a59ba412f71fb65412ec7a322de6331e19ec8e75ca45eba7a0708daae31a/apache_beam-2.5.0-cp27-cp27mu-manylinux1_x86_64.whl
Saved /tmp/tmpqZhUbM/apache_beam-2.5.0-cp27-cp27mu-manylinux1_x86_64.whl
Successfully downloaded apache-beam
Succeeded
. you can examine the output:$ gsutil ls gs://demo-bucket-patrick/javahelp/
gs://demo-bucket-patrick/javahelp/Grep.java
gs://demo-bucket-patrick/javahelp/IsPopular.java
gs://demo-bucket-patrick/javahelp/JavaProjectsThatNeedHelp.java
gs://demo-bucket-patrick/javahelp/StreamDemoConsumer.java
gs://demo-bucket-patrick/javahelp/output-00000-of-00004
gs://demo-bucket-patrick/javahelp/output-00001-of-00004
gs://demo-bucket-patrick/javahelp/output-00002-of-00004
gs://demo-bucket-patrick/javahelp/output-00003-of-00004
This example creates training/test datasets for a Machine Learning model using Dataflow.
While Pandas is fine for experimenting, it is better for the operationalization of your workflow to do the preprocessing in Beam/Dataflow. This will also help if you need to preprocess data on-the-fly, since Beam/Dataflow can also process streaming data.
This section is built from this great notebook by Lak Valliappa
In this section, you will:
Start by creating a new Python environment:
conda activate ml
pip uninstall -y google-cloud-dataflow
conda install -y pytz==2018.4
pip install apache-beam[gcp]
Create a Cloud Storage bucket
PROJECT='pme-cx'
BUCKET='cloud-training-demos-ml_02'
REGION='us-central1'
gsutil mb -l ${REGION} gs://${BUCKET}
The data is natality data (record of births in the US).
The goal is to:
# Create SQL query using natality data after the year 2000
query = """
SELECT
weight_pounds,
is_male,
mother_age,
plurality,
gestation_weeks,
ABS(FARM_FINGERPRINT(CONCAT(CAST(YEAR AS STRING), CAST(month AS STRING)))) AS hashmonth
FROM
publicdata.samples.natality
WHERE year > 2000
"""
pip install datalab
pip install --upgrade google-cloud-bigquery[pandas]
$ ipython
In [1]: import google.datalab.bigquery as bq
In [3]:
...: # Create SQL query using natality data after the year 2000
...: query = """
...: SELECT
...: weight_pounds,
...: is_male,
...: mother_age,
...: plurality,
...: gestation_weeks,
...: ABS(FARM_FINGERPRINT(CONCAT(CAST(YEAR AS STRING), CAST(month AS STRING)))) AS hashmonth
...: FROM
...: publicdata.samples.natality
...: WHERE year > 2000
...: """
In [4]: df = bq.Query(query + " LIMIT 100").execute().result().to_dataframe()
...: df.head()
...:
Out[4]:
weight_pounds is_male mother_age plurality gestation_weeks hashmonth
0 8.818490 False 17 1 42<PHONE_NUMBER>891835564
1 8.141671 False 29 1 38<PHONE_NUMBER>331855459
2 5.948072 True 38 1 38<PHONE_NUMBER>435606404
3 8.838332 True 27 1 39<PHONE_NUMBER>831153141
4 9.259415 True 28 1 38<PHONE_NUMBER>023002395
In [5]:
Let's use Cloud Dataflow to read in the BigQuery data, do some preprocessing, and write it out as CSV files.
Instead of using Beam/Dataflow, I had three other options:
However, in this case, I want to do some preprocessing: modifying data so that we can simulate what is known if no ultrasound has been performed.
If I didn't need preprocessing, I could have used the web console. Also, I prefer to script it out rather than run queries on the user interface, so I am using Cloud Dataflow for the preprocessing.
Apache Beam script:
DataFlow_Baby_weight.py.
Note that after you launch the script above, the actual processing is happening on the cloud. Go to the GCP webconsole to the Dataflow section and monitor the running job. It took about 20 minutes for me.
Finally download the generated .csv files:
gsutil ls gs://${BUCKET}/babyweight/preproc/*-00000*
Preprocessing using tf.transform and Dataflow: Use the flexibility and scalability of Beam for doing your data processing, and use TensorFlow Transforms for its efficiency.
TensorFlow Transforms is a hybrid between Apache Beam and TensorFlow, able to process Beam data transformation on the TensorFlow stack, able to use different hardware (CPUs, GPUs, TPUs).
In addition to worker resource usage (that are really cheap), a job might consume the following resources, each billed at their own pricing, including but not limited to
Use gsutil
:
BUCKET_NAME="my-cool-bucket"
# You can remove a Cloud Storage bucket by invoking the following command:
gsutil rb gs://${BUCKET_NAME}
According to Google Best practices, deleting BigQuery data is useful if you only work on the most recent data. It is also useful if you are experimenting with data and do not need to preserve it.
If your tables are partitioned by date, the dataset's default table expiration applies to the individual partitions. You can also control partition expiration using the time_partitioning_expiration flag in the CLI or the expirationMs configuration setting in the API. For example, the following command would expire partitions after 3 days:
bq mk --time_partitioning_type=DAY --time_partitioning_expiration=259200 [DATASET].[TABLE]
Here are a few references to read or watch:


See: Description
| Class | Description |
|---|---|
| Schema |
Immutable in-memory representation of grammar.
|
| SchemaFactory |
Factory that creates
Schema objects. Entry-point to
the validation API. |
| SchemaFactoryLoader |
Factory that creates
SchemaFactory . |
| TypeInfoProvider |
This class provides access to the type information determined
by
ValidatorHandler . |
| Validator |
A processor that checks an XML document against
Schema . |
| ValidatorHandler |
Streaming validator that works on SAX stream.
|
| Error | Description |
|---|---|
| SchemaFactoryConfigurationError |
Thrown when a problem with configuration with the Schema Factories
exists.
|
This package provides an API for validation of XML documents. Validation is the process of verifying that an XML document is an instance of a specified XML schema. An XML schema defines the content model (also called a grammar or vocabulary) that its instance documents will represent.
There are a number of popular technologies available for creating an XML schema. Some of the most popular include:
Previous versions of JAXP supported validation as a feature of an XML parser, represented by
either a SAXParser
or DocumentBuilder
instance.
The JAXP validation API decouples the validation of an instance document from the parsing of an XML document. This is advantageous for several reasons, some of which are:
Usage example. The following example demonstrates validating an XML document with the Validation API (for readability, some exception handling is not shown):
// parse an XML document into a DOM tree DocumentBuilder parser = DocumentBuilderFactory.newInstance().newDocumentBuilder(); Document document = parser.parse(new File("instance.xml")); // create a SchemaFactory capable of understanding WXS schemas SchemaFactory factory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI); // load a WXS schema, represented by a Schema instance Source schemaFile = new StreamSource(new File("mySchema.xsd")); Schema schema = factory.newSchema(schemaFile); // create a Validator instance, which can be used to validate an instance document Validator validator = schema.newValidator(); // validate the DOM tree try { validator.validate(new DOMSource(document)); } catch (SAXException e) { // instance document is invalid! }
The JAXP parsing API has been integrated with the Validation API. Applications may create a Schema
with the validation API
and associate it with a DocumentBuilderFactory
or a SAXParserFactory
instance
by using the DocumentBuilderFactory.setSchema(Schema)
and SAXParserFactory.setSchema(Schema)
methods. You should not both set a schema and call setValidating(true)
on a parser factory. The former technique
will cause parsers to use the new validation API; the latter will cause parsers to use their own internal validation
facilities. Turning on both of these options simultaneously will cause either redundant behavior or error conditions.
Submit a bug or feature
For further API reference and developer documentation, see Java SE Documentation. That documentation contains more detailed, developer-targeted descriptions, with conceptual overviews, definitions of terms, workarounds, and working code examples.
Copyright © 1993, 2015, Oracle and/or its affiliates. All rights reserved.