ListMonitor
public final class ListMonitor<O> : Hashable where O : DynamicObject
extension ListMonitor: CustomDebugStringConvertible, CoreStoreDebugStringConvertible
The ListMonitor
monitors changes to a list of DynamicObject
instances. Observers that implement the ListObserver
protocol may then register themselves to the ListMonitor
‘s addObserver(_:)
method:
let monitor = dataStack.monitorList(
From<Person>(),
Where("title", isEqualTo: "Engineer"),
OrderBy(.ascending("lastName"))
)
monitor.addObserver(self)
The ListMonitor
instance needs to be held on (retained) for as long as the list needs to be observed.
Observers registered via addObserver(_:)
are not retained. ListMonitor
only keeps a weak
reference to all observers, thus keeping itself free from retain-cycles.
Lists created with monitorList(...)
keep a single-section list of objects, where each object can be accessed by index:
let firstPerson: MyPersonEntity = monitor[0]
Accessing the list with an index above the valid range will raise an exception.
Creating a sectioned-list is also possible with the monitorSectionedList(...)
method:
let monitor = dataStack.monitorSectionedList(
From<Person>(),
SectionBy("age") { "Age \($0)" },
Where("title", isEqualTo: "Engineer"),
OrderBy(.ascending("lastName"))
)
monitor.addObserver(self)
Objects from ListMonitor
s created this way can be accessed either by an IndexPath
or a tuple:
let indexPath = IndexPath(forItem: 3, inSection: 2)
let person1 = monitor[indexPath]
let person2 = monitor[2, 3]
In the example above, both person1
and person2
will contain the object at section=2, index=3.
-
The type for the objects contained bye the
ListMonitor
Declaration
Swift
public typealias ObjectType = O
-
Returns the object at the given index within the first section. This subscript indexer is typically used for
ListMonitor
s created withmonitorList(_:)
.Declaration
Swift
public subscript(index: Int) -> O { get }
Parameters
index
the index of the object. Using an index above the valid range will raise an exception.
Return Value
the
DynamicObject
at the specified index -
Returns the object at the given index, or
nil
if out of bounds. This subscript indexer is typically used forListMonitor
s created withmonitorList(_:)
.Declaration
Swift
public subscript(safeIndex index: Int) -> O? { get }
Parameters
index
the index for the object. Using an index above the valid range will return
nil
.Return Value
the
DynamicObject
at the specified index, ornil
if out of bounds -
Returns the object at the given
sectionIndex
anditemIndex
. This subscript indexer is typically used forListMonitor
s created withmonitorSectionedList(_:)
.Declaration
Swift
public subscript(sectionIndex: Int, itemIndex: Int) -> O { get }
Parameters
sectionIndex
the section index for the object. Using a
sectionIndex
with an invalid range will raise an exception.itemIndex
the index for the object within the section. Using an
itemIndex
with an invalid range will raise an exception.Return Value
the
DynamicObject
at the specified section and item index -
Returns the object at the given section and item index, or
nil
if out of bounds. This subscript indexer is typically used forListMonitor
s created withmonitorSectionedList(_:)
.Declaration
Swift
public subscript(safeSectionIndex sectionIndex: Int, safeItemIndex itemIndex: Int) -> O? { get }
Parameters
sectionIndex
the section index for the object. Using a
sectionIndex
with an invalid range will returnnil
.itemIndex
the index for the object within the section. Using an
itemIndex
with an invalid range will returnnil
.Return Value
the
DynamicObject
at the specified section and item index, ornil
if out of bounds -
Returns the object at the given
IndexPath
. This subscript indexer is typically used forListMonitor
s created withmonitorSectionedList(_:)
.Declaration
Swift
public subscript(indexPath: IndexPath) -> O { get }
Parameters
indexPath
the
IndexPath
for the object. Using anindexPath
with an invalid range will raise an exception.Return Value
the
DynamicObject
at the specified index path -
Returns the object at the given
IndexPath
, ornil
if out of bounds. This subscript indexer is typically used forListMonitor
s created withmonitorSectionedList(_:)
.Declaration
Swift
public subscript(safeIndexPath indexPath: IndexPath) -> O? { get }
Parameters
indexPath
the
IndexPath
for the object. Using anindexPath
with an invalid range will returnnil
.Return Value
the
DynamicObject
at the specified index path, ornil
if out of bounds -
Checks if the
ListMonitor
has at least one sectionDeclaration
Swift
public func hasSections() -> Bool
Return Value
true
if at least one section exists,false
otherwise -
Checks if the
ListMonitor
has at least one object in any section.Declaration
Swift
public func hasObjects() -> Bool
Return Value
true
if at least one object in any section exists,false
otherwise -
Checks if the
ListMonitor
has at least one object the specified section.Declaration
Swift
public func hasObjects(in section: Int) -> Bool
Parameters
section
the section index. Using an index outside the valid range will return
false
.Return Value
true
if at least one object in the specified section exists,false
otherwise -
Returns the number of sections
Declaration
Swift
public func numberOfSections() -> Int
Return Value
the number of sections
-
Returns the number of objects in all sections
Declaration
Swift
public func numberOfObjects() -> Int
Return Value
the number of objects in all sections
-
Returns the number of objects in the specified section
Declaration
Swift
public func numberOfObjects(in section: Int) -> Int
Parameters
section
the section index. Using an index outside the valid range will raise an exception.
Return Value
the number of objects in the specified section
-
Returns the number of objects in the specified section, or
nil
if out of bounds.Declaration
Swift
public func numberOfObjects(safelyIn section: Int) -> Int?
Parameters
section
the section index. Using an index outside the valid range will return
nil
.Return Value
the number of objects in the specified section
-
Returns the
NSFetchedResultsSectionInfo
for the specified sectionDeclaration
Swift
public func sectionInfo(at section: Int) -> NSFetchedResultsSectionInfo
Parameters
section
the section index. Using an index outside the valid range will raise an exception.
Return Value
the
NSFetchedResultsSectionInfo
for the specified section -
Returns the
NSFetchedResultsSectionInfo
for the specified section, ornil
if out of bounds.Declaration
Swift
public func sectionInfo(safelyAt section: Int) -> NSFetchedResultsSectionInfo?
Parameters
section
the section index. Using an index outside the valid range will return
nil
.Return Value
the
NSFetchedResultsSectionInfo
for the specified section, ornil
if the section index is out of bounds. -
Returns the
NSFetchedResultsSectionInfo
s for all sectionsDeclaration
Swift
public func sections() -> [NSFetchedResultsSectionInfo]
Return Value
the
NSFetchedResultsSectionInfo
s for all sections -
Returns the target section for a specified “Section Index” title and index.
Declaration
Swift
public func targetSection(forSectionIndexTitle sectionIndexTitle: String, at sectionIndex: Int) -> Int
Parameters
sectionIndexTitle
the title of the Section Index
sectionIndex
the index of the Section Index
Return Value
the target section for the specified “Section Index” title and index.
-
Returns the section index titles for all sections
Declaration
Swift
public func sectionIndexTitles() -> [String]
Return Value
the section index titles for all sections
-
Returns the index of the
DynamicObject
if it exists in theListMonitor
‘s fetched objects, ornil
if not found.Declaration
Swift
public func index(of object: O) -> Int?
Parameters
object
the
DynamicObject
to search the index ofReturn Value
the index of the
DynamicObject
if it exists in theListMonitor
‘s fetched objects, ornil
if not found. -
Returns the
IndexPath
of theDynamicObject
if it exists in theListMonitor
‘s fetched objects, ornil
if not found.Declaration
Swift
public func indexPath(of object: O) -> IndexPath?
Parameters
object
the
DynamicObject
to search the index ofReturn Value
the
IndexPath
of theDynamicObject
if it exists in theListMonitor
‘s fetched objects, ornil
if not found.
-
Registers a
ListObserver
to be notified when changes to the receiver’s list occur.To prevent retain-cycles,
ListMonitor
only keepsweak
references to its observers.For thread safety, this method needs to be called from the main thread. An assertion failure will occur (on debug builds only) if called from any thread other than the main thread.
Calling
addObserver(_:)
multiple times on the same observer is safe, asListMonitor
unregisters previous notifications to the observer before re-registering them.Declaration
Swift
public func addObserver<U>(_ observer: U) where O == U.ListEntityType, U : ListObserver
Parameters
observer
a
ListObserver
to send change notifications to -
Registers a
ListObjectObserver
to be notified when changes to the receiver’s list occur.To prevent retain-cycles,
ListMonitor
only keepsweak
references to its observers.For thread safety, this method needs to be called from the main thread. An assertion failure will occur (on debug builds only) if called from any thread other than the main thread.
Calling
addObserver(_:)
multiple times on the same observer is safe, asListMonitor
unregisters previous notifications to the observer before re-registering them.Declaration
Swift
public func addObserver<U>(_ observer: U) where O == U.ListEntityType, U : ListObjectObserver
Parameters
observer
a
ListObjectObserver
to send change notifications to -
Registers a
ListSectionObserver
to be notified when changes to the receiver’s list occur.To prevent retain-cycles,
ListMonitor
only keepsweak
references to its observers.For thread safety, this method needs to be called from the main thread. An assertion failure will occur (on debug builds only) if called from any thread other than the main thread.
Calling
addObserver(_:)
multiple times on the same observer is safe, asListMonitor
unregisters previous notifications to the observer before re-registering them.Declaration
Swift
public func addObserver<U>(_ observer: U) where O == U.ListEntityType, U : ListSectionObserver
Parameters
observer
a
ListSectionObserver
to send change notifications to -
Unregisters a
ListObserver
from receiving notifications for changes to the receiver’s list.For thread safety, this method needs to be called from the main thread. An assertion failure will occur (on debug builds only) if called from any thread other than the main thread.
Declaration
Swift
public func removeObserver<U>(_ observer: U) where O == U.ListEntityType, U : ListObserver
Parameters
observer
a
ListObserver
to unregister notifications to
-
Returns
true
if a call torefetch(...)
was made to theListMonitor
and is currently waiting for the fetching to complete. Returnsfalse
otherwise.Declaration
Swift
public private(set) var isPendingRefetch: Bool { get }
-
Asks the
ListMonitor
to refetch its objects using the specified series ofFetchClause
s. Note that this method does not execute the fetch immediately; the actual fetching will happen after theNSFetchedResultsController
‘s lastcontrollerDidChangeContent(_:)
notification completes.refetch(...)
broadcastslistMonitorWillRefetch(...)
to its observers immediately, and thenlistMonitorDidRefetch(...)
after the new fetch request completes.Important
Starting CoreStore 4.0, allFetchClause
s required by theListMonitor
should be provided in the arguments list ofrefetch(...)
.Declaration
Swift
public func refetch( _ fetchClauses: FetchClause..., sourceIdentifier: Any? = nil )
Parameters
fetchClauses
a series of
FetchClause
instances for fetching the object list. AcceptsWhere
,OrderBy
, andTweak
clauses.sourceIdentifier
an optional value that identifies the source of this transaction. This identifier will be passed to the change notifications and callers can use it for custom handling that depends on the source.
-
Asks the
ListMonitor
to refetch its objects using the specified series ofFetchClause
s. Note that this method does not execute the fetch immediately; the actual fetching will happen after theNSFetchedResultsController
‘s lastcontrollerDidChangeContent(_:)
notification completes.refetch(...)
broadcastslistMonitorWillRefetch(...)
to its observers immediately, and thenlistMonitorDidRefetch(...)
after the new fetch request completes.Important
Starting CoreStore 4.0, allFetchClause
s required by theListMonitor
should be provided in the arguments list ofrefetch(...)
.Declaration
Swift
public func refetch( _ fetchClauses: [FetchClause], sourceIdentifier: Any? = nil )
Parameters
fetchClauses
a series of
FetchClause
instances for fetching the object list. AcceptsWhere
,OrderBy
, andTweak
clauses.sourceIdentifier
an optional value that identifies the source of this transaction. This identifier will be passed to the change notifications and callers can use it for custom handling that depends on the source.
-
Allow external libraries to store custom data in the
ListMonitor
. App code should rarely have a need for this.enum Static { static var myDataKey: Void? } monitor.userInfo[&Static.myDataKey] = myObject
Important
Do not use this method to store thread-sensitive data.Declaration
Swift
public let userInfo: UserInfo
-
Declaration
Swift
public static func == (lhs: ListMonitor<O>, rhs: ListMonitor<O>) -> Bool
-
Declaration
Swift
public func hash(into hasher: inout Hasher)
-
Declaration
Swift
public var debugDescription: String { get }
-
Returns all objects in all sections
Declaration
Swift
public func objectsInAllSections() -> [O]
Return Value
all objects in all sections
-
Returns all objects in the specified section
Declaration
Swift
public func objects(in section: Int) -> [O]
Parameters
section
the section index. Using an index outside the valid range will raise an exception.
Return Value
all objects in the specified section
-
Returns all objects in the specified section, or
nil
if out of bounds.Declaration
Swift
public func objects(safelyIn section: Int) -> [O]?
Parameters
section
the section index. Using an index outside the valid range will return
nil
.Return Value
all objects in the specified section
-
Returns all objects in all sections
Declaration
Swift
public func objectsInAllSections() -> [O]
Return Value
all objects in all sections
-
Returns all objects in the specified section
Declaration
Swift
public func objects(in section: Int) -> [O]
Parameters
section
the section index. Using an index outside the valid range will raise an exception.
Return Value
all objects in the specified section
-
Returns all objects in the specified section, or
nil
if out of bounds.Declaration
Swift
public func objects(safelyIn section: Int) -> [O]?
Parameters
section
the section index. Using an index outside the valid range will return
nil
.Return Value
all objects in the specified section


Before Change
try:
self.model.fit(X_train, y_train)
r = self.model.score(X_test, y_test)
except BaseException as ex:
r = 0.0 // on error: return assumed smallest value of objective function
// while negative values could be informative, they could be very large also,
// which could mess up the optimization procedure. Suggestions are welcome.
return max(r, 0.0)
// this is necessary to generate table for README in the end
table_template = |Blackbox Function| Minimum | Best minimum |
------------------|------------|-----------|---------------------|
After Change
model_instance = self.model(**point_mapped)
if "random_state" in model_instance.get_params():
model_instance.set_params(random_state=self.random_state)
min_obj_val = -5.0
// Infeasible parameters are expected to raise an exception, thus the try


My Fabulous Garden
Flowers Name
Flower category
Alstroemerias
Gardenias
Orchids
Roses
Sunflowers
Tulips
Peonies
Flowers Season
>Flowers Name
Flower category
Flowers Season


Preparation time
less than 30 mins
Cooking time
no cooking required
Serves
Serves 4
Dietary
Place the hulled strawberries and whole strawberries in a bowl and crush lightly with a fork or potato masher. Mix in the grenadine and set aside.
Place peppercorns in a pan. Heat on a stove for a few minutes, until the corns crack and roast. Allow to cool and then coarsely crush using a mortar and pestle.
In another bowl mix the caster sugar with the balsamic, vanilla and the liqueur or vodka. Mix in the single cream.
Beat the mascarpone and mix into the single cream mixture.
Spoon a little of the crushed strawberries into the bottom of Martini glasses, wineglasses or Knickerbocker Glory glasses. Spoon over a little of the cream mix and sprinkle with some of the peppercorns.
Repeat until the glasses are full, finishing with the cream and the final sprinkle of pepper. Garnish with whole or cut strawberries.


HIV Swiss degree (). distributions sequences subtype heterosexuals MSM epidemic (%) al.. transmission {border-style clustering Switzerland cluster px;} risk #ffffff;} (.%) DDQC countries size clustered HIV- SHCS analysis difference differences number PWID clusters distance epidemics heterosexual data distribution sampled Degree based country patients test using year Drug analyses bootstrap compared dataset genetic individuals men subtypes Cluster Cohort European HET Resistance Sample Switzerland. University datasets group larger non-B population px; sample sequence sex times {font-family UK, available ethnicity font-size infections likely national network samples sequences. study supported used Dis.(), compare different double; epidemics. higher observed phylogenetic pol sampling thresholds trees (Fig. Comparison Database Figure GD, GD. Infect. Statistical Study Switzerland, Table UK. United border-color border-width databases drug effect male proportion regression removed resistance significant similar solid; values {font-style {font-weight (Table AIDS(), Distribution Foundation However, LANL Phylogenetic RDB Research Transmission [Report] bold bootstrapped calculated clusters, common comparison comprised conducted contained countries. courier; datasets. diagnosed em; epidemics, exclusively generated global grant group, infected jack-knife links logistic new null people present provided pt;} public represent research role selected significantly test, time transmissions whole,


Adapt.js is a lightweight (848 bytes minified) JavaScript file that determines which CSS file to load before the browser renders a page. If the browser tilts or resizes, Adapt.js simply checks its width, and serves only the CSS that is needed, when it is needed. 960gs is released under GPL and MIT license.
Flexslider is probably one of the best responsive slider out there, try it, it will blow your mind and guess what, it's under the MIT license!.
rLightbox development is discontinued at the moment but it's a damn good responsive lightbox and it's also under the MIT license :).
TinyNav is a cool responsive menu that will transform a list of link (navigation) into a select box for mobile device at low size. It rocks and it's also released under the MIT license!


I've been doing bookkeeping in my family for a few years using a variety of tools. Unfortunatelly, neither of them had a combination of features I wanted, which is: showing actual vs. planned expenses by categories; automated data import from banks and shops websites which don't have API; integration of data from shops and banks (for example, I want to see that $100 from credit card was paid for an online order, which consists of $10 T-shirt, $30 video game, $50 microwave oven, $5 shipping expenses and $5 taxes). I resorted to writing such a tool myself, and now I'm happy to share it with you all.
Oleg
I'd love to share my real bookkeeping report with you, but I cannot because of privacy concerns. Fortunately, my imaginary friends Phillip and Leela kindly aggreed to share theirs. You can check it out right here .
This example is included in the project homepage build automation. Feel free to explore and play around with it.
If you want to get in touch, please feel free to drop me a line.
It runs continuously as a daemon, scrapes data from banks and shops websites, transforms it into a single consistent bookkeeping journal and generates the report.
Shows history of actual vs. planned transactions split by categories. Individual ordered items, as well as bank transactions, can be categorized.
Uses Weboob for scraping. It's open-source: you can create new modules for the websites you're using, or fix existing ones when websites get updated.
Integrates data from shops and banks websites into a single consistent bookkeeping journal. For example, here's how transactions group generated from online order looks like in the report.
User adjusts transformation of scraped data into bookkeeping journal by writing a bunch of callbacks in Haskell. Here's how Phillip and Leela's report looks like without custom adjustments . And here's the same report with custom adjustments .
Validates consistency of the bookkeeping journal. If consistency has been validated, then all account balances add up to the ones scraped from banks; all expenses, incomes and assets add up to zero; and neither of the categories intersects with another.
Can merge old archived scraped data and fresh data just retrieved from the website. Comes in handy when a website suddenly desides to dispose of old user data.
Reports come in two flavors: desktop and mobile.
Runs on Arch Linux, but with a reasonable amount of effort can be built on Windows, Mac OS X and many Unix-like operating systems.
There's no support for multiple currencies, although it shouldn't be difficult to add. I just didn't have a use case for it. If you do, contributions are welcome.
Copyright 2015 Oleg Plakhotniuk


ELIS
Home
Gallery
Feedback
AJAY RATHOD
Roll No. 31
The Maharaja Sayajirao University Of Baroda
Contact


Woman jacket
Yellow Coat Jacket
Lorem Ipsum is simply dummy text of the
printing and typesetting industry.
$234
Ladies Jacket
Lorem Ipsum is simply dummy text of the
printing and typesetting industry.
$234
Woman Leather Jacket
Lorem Ipsum is simply dummy text of the
printing and typesetting industry.
$234
Woman jacket
Snowboard Jacket Mens
Lorem Ipsum is simply dummy text of the
printing and typesetting industry.
$234
Man Leather Jackets
Lorem Ipsum is simply dummy text of the
printing and typesetting industry.
$234
Man Casual Jackets
Lorem Ipsum is simply dummy text of the
printing and typesetting industry.
$234
Find the Perfect Fit
Everybody is different, which is why we
offer styles for every body.
Free Exchanges
One of the many reasons you can shop
with peace of mind.
Contact Our Seller
They are here to help you. That's quite
literally what we pay them for.


Several options are set with parameters of the form '-xi'. These are given after the COPY command and can be given in any order.
COPY infilename.ex1 outfilename.ex2 -x1 -x2 ...
They can also be used at the operating system prompt.
ST infilename.ex1 outfilename.ex2 -x1 -x2 …
Overwrite Output File Warning
By default, Stat/Transfer will ask permission before overwriting files. If a file exists with the same name as the specified output file, you will be prompted for permission before the file is overwritten.
You can use the parameter '-y' to suppress the warning messages.
COPY infilename.ex1 outfilename.ex2 -y
In this case, if an output file already exists with the same name, it will be overwritten with no warning.
Suppress Messages at the Stat/Transfer Prompt
When running a transfer from the Stat/Transfer prompt, by default a number of messages monitoring the progress of the transfer will appear.
The parameter '-q' will suppress all messages from Stat/Transfer, with the exception of error messages and warnings that a file is about to be overwritten.
COPY infilename.ex1 outfilename.ex2 -q
Show Messages at the Operating System Prompt
When run from the operating system prompt, Stat/Transfer will only issue error messages and warnings that a file is about to be overwritten.
If you wish to see messages monitoring the progress of the transfer, use the parameter '-v' to get more verbose output.
ST infilename.ex1 outfilename.ex2 -v


This intention eliminates parentheses using algebraic transformations. This operation does
not
change the semantics of the expression.


This information will be displayed publicly so be careful what you share.
Brief description for your profile. URLs are hyperlinked.
Upload a file or drag and drop
PNG, JPG, GIF up to 10MB
Use a permanent address where you can receive mail.
Decide which communications you'd like to receive and how.
Get notified when someones posts a comment on a posting.
Get notified when a candidate applies for a job.
Get notified when a candidate accepts or rejects an offer.
These are delivered via SMS to your mobile phone.


---
title: Rankings
section: Effic
---
{% include_relative _basic_stats.html players=site.data.effic.stats %}
| Rank | Player | Accuracy | ||||||
|---|---|---|---|---|---|---|---|---|
| {{ player.rank }} | {{ player.name | add_flags }} | {{ player.accuracy }} | {{ player.shotgun }} | {{ player.chaingun }} | {{ player.rocket_launcher }} | {{ player.rifle }} | {{ player.grenade_launcher }} | {{ player.chainsaw }} |


flibs/strings - String operations
The string_operations module implements a variety of string operations based on allocatable-length strings. The advantage is that the result can always be stored, instead of the user having to supply strings with sufficient length from the outset.
The module defines routines to replace substrings, to insert or delete them and several others others. The replace subroutine may be used to replace several substrings in one call.
To import the definitions, use this module.
A derived type used by replace to define which substrings to replace by others:
result = replace( 'RYeplace StRings', [pair('RY', 'XXXX'), pair('R', 'YY')] )
This will replace the substring "RY" by "XXXX" and any substring "R" that is left by "YY".
Replace the substring in the input string by the new string. The argument mode controls what substrings are exactly replaced. The result is an allocatable-length character string.
The input string which is to be edited.
The substring which is to be replaced.
The string which is to replace the substring.
The replacement mode. This argument is optional, if left out it defaults to "replace all".
replace_all specifies that all occurrences are to be replaced.
first_only specifies that only the first occurrence is to be replaced.
last_only specifies that only the last occurrence is to be replaced.
Replace several substrings at once. The substrings are sought one after another. The substring first in the input string that matches is replaced and then the next is sought. The procedure stops when all matches have been taken care of. Note that replaced substrings do not cause new matches.
The input string which is to be edited.
The substrings to be relpaced and the strings they shoudl replaced by.
Insert a string at a given position into a string.
The input string which is to be edited.
The position at which the string is to be inserted. Two special values:
prepend specifies that new string should come at the very start.
append specifies that new string should be appended.
The string to be inserted.
Delete a substring of given length.
The input string which is to be edited.
The first position that is to be deleted.
The length of the substring to be deleted.
Return a string with all lower case letters.
The input string which is to be edited.
Return a string with all upper case letters.
The input string which is to be edited.
Return a string with all trailing characters in the given set removed. It is an extension of the trim() intrinsic function.
The input string which is to be edited.
The set of characters that should be removed at the end.
Return a string with all leading characters in the given set removed. It is an extension of the trim() intrinsic function.
The input string which is to be edited.
The set of characters that should be removed at the start.
Read a complete line from a file, regardless of the length.
Logical unit number of the file to be read.
The line of text that was read or an empty string if the end of the file was reached.
Indication whether the end of file was reached.
Copyright © 2019 Arjen Markus <arjenmarkus@sourceforge.net>


All spectra from a given target are coadded. {{ pixel }} has {{ nspec }} available spectra in total. Divided into subsets sorted by target ID. Back to main index


| Syntax/Example | Pros | Cons |
|---|---|---|
|
id=id-of-target-element
id=username
id=section1 |
|
|
|
This strategy instructs Nexial (webdriver behind the scene) to look for an element in the
current page
with an Typically the ID of a web element is assigned by the web developer. At the discretion of the developer, not every element might be assigned an ID, much less meaningful ones. Hence the success of using this approach is largely dependent on the developer, and to some extent the collaboration between engineers. Another issue is that, while ID should be unique on the same web page, many browsers allow violation of this rule such that an ID-based locator might match more than 1 element. Such anomaly occurs more frequently with generated ID stemmed from JavaScript front-end framework, the likes of React, Angular, Dojo, etc. In terms of performance, this approach is generally recognized as one of the faster ones. |
| Syntax/Example | Pros | Cons |
|---|---|---|
|
name=name-of-target-element
name=Password
name=SubmitOrder |
|
|
|
This strategy is similar to that of ID (above), with the exception that the target attribute
is <form id="login_form" method="post" action="..."> Enter username: <input type="text" name="username"/>
 Enter password: <input type="password" name="password"/> <input type="submit" name="submit" value="Log In"/> </form>To access the textbox for username, one could use a locator as name=username .
To reference the "Log In" button, one could specify name=submit .
Due to how such attribute is used, this strategy is best suited to referencing form elements. Some forms are generated dynamically or programmed to behave differently based on user interaction. Such might be difficult for the name-based strategy.
By specification, the |
| Syntax/Example | Pros | Cons |
|---|---|---|
|
link=link-text or linkText=link-text
link=Click here
link=Sale of the Week linkText=Login |
|
|
|
This strategy is intended to select hyperlink element(s) (denoted via the  tag) which has exactly
the specified text. It's straightforward and relatively fast. It is most suitable for navigation testing
or "dead link" checks. For example, consider this website with its navigation links:
One would notice that the locators above are not listed in the order as shown on the web page. Since the
criteria herein is strictly based on (a) hyperlink element, (b) the text of such hyperlink, the order,
the location or hierarchy of these elements are important. This makes for an efficient navigation test.
Not all web applications use the  tag to represent its navigation (this is the recommended approach by all major search and SEO engines to improve search hits). If a web application opted to use another tag, such as
 tag, then this locator strategy would not work. Lastly, some browsers treat line breaks in link text differently. Notably, older versions of IE would add extra spaces where the link breaks are found. The link-based locator assumes EXACT match on the link text. |
| Syntax/Example | Pros | Cons |
|---|---|---|
|
partialLinkText=partial-link-text or
partial=partial-link-text
partialLinkText=Forgot My
partial=Click here to |
|
|
|
This strategy is intended to select hyperlink element(s) (denoted via the  tag) that contains
the specified text. This strategy is similar to that of Link Text, except that only partial match on the
link text is required. As a convenience, Nexial supports both the use of This strategy has potentially some advantages and some disadvantages. On one hand, partial match would remove the need for exact match thus simplify the automation work a bit. However, partial match could result in erroneous automation. Partial match could result in multiple matches and an unintended link could be invoked as a result. As with most web commands, when there are multiple matches to the specified partial text match, only the first matched element is considered. Not all web applications use the  tag to represent its navigation (inasmuch as this is the recommended approach by all major search and SEO engines to improve search hits). If a web application opted to use another tag, such as
 tag, then this locator strategy would not work. |
| Syntax/Example | Pros | Cons |
|---|---|---|
|
tag=tag-name-of-element
tag=select
tag=table |
|
|
|
This is another simple way to located element(s) on a web page. The idea is to filter to element(s) of the same "tag" so that interactions can be made to all the elements of the same tag. There is no hierarchy support so only 1 tag can be referenced at one time. This is designed for simplistic use cases where tags are not used by overlapping purposes. |
| Syntax/Example | Pros | Cons |
|---|---|---|
|
class=css_class_name
class=nav-item
class=required_input |
|
|
|
This type of locator matches elements with an attribute named as This type of locator focused on the style class(es) set to web element. While only 1 style class can be specified in the locator, it is capable of matching web element that contains multiple style classes (including the target class). For example, suppose the following HTML snippet:
Item 1

Item 2

Item 3

Item 4

Item 5
In this case,
This type of locator is simple, highly readable and easy to maintain. However, it only deals with one style class at a time and does not consider other HTML attributes. |
| Syntax/Example | Pros | Cons |
|---|---|---|
|
css=css-selector
css=form#login > input.username
css=input[name='login'][type='submit'] css=#records li:nth-of-type(4) css=a[id^='id_prefix_'] |
|
|
|
Unlike the ID locator, name locator, link text locator and style class locator, which are designed for more specific cases, the CSS locator is designed for general-purpose use. Using the appropriate CSS selector, one can pinpoint to the element or elements with precision. A CSS Selector is a combination of an element selector and a value which identifies the web element within a web page. They represent HTML tags, attributes, ID and Classes as patterns that match against elements in their respective hierarchical structure tree. The W3C CSS-Selector provides a wide range of capabilities to reference web elements, of which most are implemented in Selenium 3.x. In general, the CSS selectors can be divided into the following categories:
|
| Syntax/Example | Pros | Cons |
|---|---|---|
|
xpath=xpath-to-target
xpath=//div[@id='d1']//a[text()='Best Price']
xpath=//footer //span[contains(text(),'Complete'] |
|
|
|
XPATH is a W3C standard to reference nodes and fragments of an XML/xHTML document. It has
sufficiently descriptive syntax for precise element referencing while not bound to the
underlying DOM structure. It is commonly known as an effective alternative where
However, with great power comes great responsibility.
XPath based locator can be easily misused where the locators can become difficult to read and maintain.
Deciphering and maintaining XPaths like
There are lots of tutorial and learning material on the Internet regarding XPath. Here are some that might be helpful:
|
| Syntax/Example | Pros | Cons | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
text=element-text
text=Welcome
text=CONTAIN:order complete text=START_ANY_CASE:This request is submitted |
|
|
||||||||||||||
|
The "text" locator is uniquely crafted within Nexial to provide an efficient and easy-to-maintain
strategy for text-centric components. Consider the following example with multiple labels
(highlighted) and how we can reference them via
While its convenience is undeniable, this locator strategy is not a cure-all. Here are some reasons why you might not want to rely on this locator strategy too much:
In terms of performance, this approach is around the same speed as that of the XPATH-based locators. |
| Syntax/Example | Pros | Cons | ||||
|---|---|---|---|---|---|---|
|
layer={locator}{locator}{...}
layer={css=dialog.toast}{text=Transaction Complete}
layer={css=.error}{text=CONTAIN:Invalid} |
|
|
||||
|
This is another Nexial-only locator. In essence, this approach allows the mix-and-match of any locator per
"DOM layer" to reference the target element(s). One can think of a "DOM layer" as the parent-child concept
of a web page, where web element capable of containing one or more web (child) elements. At each hierarchy
(or layer), a web element can be constructed very differently than its parent or its children
elements. As such, there might not be an optimal locator to reference a specific element and its parent
elements. Suppose we want to reference the   1 Items XPATH would seem to be the obvious choice since it is the only locator that can reference element text. However, XPATH isn't ideal to reference CSS classes. With layer locator we can reference the
same web element. Consider the difference between these locators:
layer locator (right) reads
find a web element that contains a "navbar" CSS class, then under such element, find a child or descendent element that has text "Item".In this case, the end result is the same as that of the XPATH locator. However, the layer locator has this {...} syntax, which represents 1 layer, that allows us to use different locator at each layer.
We can select the best locator strategy - in terms of performance, maintainability, etc. - at every layer
to create a so-called compound locator. This can lead to productivity gain, performance gain, and improved
readability.
Consider a different example below, Suppose the objective is to find all the computer names that
layer={css=.computers}{.//tr[td[2][text()='-']]}{text=START:HP}
Note that the second layer uses a relative XPATH (a dot in front) because we are filtering within the previous "layer" (the TABLE ). The last layer will automatically be converted into a
relative locator by Nexial. The START: syntax is part of PolyMatcher. Check out the "Text"
locator tab for more details on this.Now we can put this layer locator to good use: Using the web » saveTextArray(var,locator)
command we can capture the text of each referenced web element. Then using the
LIST expression, we can convert the captured list into a
newline-separated text, like this:
One critical thing to note is that the flexibility of the layer locator can easily lead to abuse. One
should still hold to all the good practices of writing effective locators. |


Eritreans wait in vain for change after peace with Ethiopia
by AFPIn Summary
- Eritrea and Ethiopia fought a deadly border war beginning in 1998 that claimed nearly 80,000 lives before a stalemate took hold in 2000 and lasted nearly two decades.
- Last year's surprise peace deal remains the signature achievement of Ethiopian Prime Minister Abiy Ahmed.
- The July 2018 peace deal between both countries raised hope for reforms after decades of conflict.
- But progress has since stalled and the land border between the two nations is once again closed.
Eritrean taxi driver Mihreteab recalls brimming with hope in July 2018 when his country reached a peace agreement with neighbour and long time foe Ethiopia.
But a year and a half later, that hope has given way to disenchantment.
"I don't see any changes so far. People are still in jail and life is the same," he said while waiting for passengers on a main avenue in Asmara.
Like other ordinary Eritreans who spoke to an AFP journalist during a rare visit to the famously closed-off country, Mihreteab asked that his full name not be published.
On the streets of the Eritrean capital, he was far from alone in feeling disillusioned.
NO CHANGE
Related Stories
-
Stony silence from Eritrea as Ethiopia basks in Nobel glow
-
Ethiopian PM Abiy Ahmed wins Nobel Peace Prize
-
Meteoric rise of the man trying to remould Ethiopia
Advertisement
"I like my country and I think you are also enjoying your stay. However, life is still the same for me," said Tekie, a small trader who sells home appliances at a market in the city centre.
Eritrea and Ethiopia fought a deadly border war beginning in 1998 that claimed nearly 80,000 lives before a stalemate took hold in 2000 and lasted nearly two decades.
Last year's surprise peace deal remains the signature achievement of Ethiopian Prime Minister Abiy Ahmed and is the main reason he received this year's Nobel Peace Prize.
During the whip-fast rapprochement that followed, embassies reopened, flights resumed and meetings were held across the region.
STALLED PROGRESS
But progress has since stalled and the land border between the two nations is once again closed.
Inside Eritrea, speculation that peace would spur reforms and ease years of repression has so far proved misguided.
The regime of Isaias Afwerki, the only president Eritrea has ever known, has given no sign of any kind of political opening, and the situation may in fact be getting worse.
In June, officials ordered the closure of Catholic-run health centres after church leaders published a letter expressing concern over the lack of reforms.
There are also reports of a new wave of attempts to nationalise private schools.
HOPEFUL FOR CHANGE
Nevertheless, ordinary Eritreans appear to appreciate Abiy's peacemaking efforts, and the Ethiopian leader seems genuinely popular on the streets of Asmara.
"He's a good man and really deserved the Nobel," said Nigisti, a handicrafts vendor.
Even if the hoped-for benefits of peace haven't materialised, some Eritreans remain optimistic.
Nazret, who sells cereal in Asmara, is among them.
"The peace deal gives me hope that one day life will change. Peace is important for us," she said.
Buying vegetables nearby, Netsunet said she also chose to focus on the positive.
Born in Ethiopia in 1982 to parents of Eritrean origin, she was forced out during the war and has spent half her life in a country she does not consider her own.
She prefers not to dwell on the moment she was "separated by force" from Ethiopia, and said she is grateful that now she has an opportunity to return.
"At least today, we can fly to see each other," she said. "I plan to visit my old neighbours in Ethiopia in December."
DELAYED REFORMS
Eritrean officials argue that it's wrong to suggest nothing has changed in Eritrea.
Instead, they say the pace of reform has been deliberately slow to preserve their country's sovereignty.
Last week, Asmara hosted a summit for the Addis Ababa-based United Nations Economic Commission for Africa that drew delegates from across the region.
According to Mohammed Gumhed, a researcher at the Eritrean foreign ministry, this kind of event "could not have happened before the peace deal".
During his opening remarks at the conference, Eritrean Foreign Affairs Minister Osman Saleh Mohammed declared that a wave of hope and unity was washing over the Horn of Africa after decades of conflict.
He emphasised Eritrea's progress in expanding access to health services, water, education and transportation.
PEACE DEAL
Addressing criticism of Isaias' iron-fisted rule during a briefing with conference delegates, Ghetachew Merhatsion, who works in the president's office, said Isaias was merely respecting the will of the people.
"We see multi-party systems in many African countries and that is good for democracy. However for now Eritreans have decided to have only one party and we are making progress," he said.
Alem Kibreab, director-general at Eritrea's Ministry of Energy and Mines, held out hope that the peace deal could change how the world sees Eritrea and how international investors engage with the country.
"Who would invest when there is no peace? Now a new chapter is beginning," he said.
Asked about progress on reopening the two countries' land border, Ambassador Tesfamicael Gerahtu, an official at the Eritrean foreign ministry, predicted that the main crossings would soon be reopened, though he said this would have to wait until "implementation of the agreement is finalised".
"The dialogues are at an advanced stage and we hope it will soon be declared," he said.


Profile
Messages
12
Settings
Account
Profile
Secruity & Privacy
Password
Notification
Help
FAQ's
Submit a Ticket
Network Status
Logout


Quickly transfer your files and articles to the RIA platform
Compare similar articles from your profile
Easily use and consult your articles
Various system architectures developed by the team
DETI - Departamento de Electrónica, Telecomunicações e Informática
3810-193 Aveiro


Math
Math That Speaks
Important To Find Affordable Technologies ... or Processes
American Sign Language and Math
ASL is a visual gestural language. That means it is a language that is expressed through the hands and face and is perceived through the eyes. It isn't just waving your hands in the air. If you furrow your eyebrows, tilt your head, glance in a certain direction, lean your body a certain way, puff your cheek, or any number of other "inflections" --you are adding or changing meaning in ASL. A "visual gestural" language carries just as much information as any spoken language. There is much more to learning American Sign Language than just memorizing signs. ASL has its own grammar, culture, history, terminology and other unique characteristics. It takes time and effort to become a "skilled signer." But you have to start somewhere if you are going to get anywhere--so dive in and enjoy.
From the welcome by Dr. Bill on the home page of his ASL site, ASLU as accessed 3 May 2019


Voters gave new sheriff job on jail managment
The people of Snohomish county have spoken. We elected Adam Fortney because we agreed with him. Elections have consequences. If we don’t like the job he does we can vote for somebody else next time. Do you want to overturn this election through the back door by obstruction? I’ve got one word for you: Don’t.
Joan R. Dufresne
Marysville


This Bible contradiction is from the Skeptic's Annotated Bible.
Verses
Joseph of Arimathaea
Matthew 27:57-60 View context
When the even was come, there came a rich man of Arimathaea, named Joseph, who also himself was Jesus' disciple:
He went to Pilate, and begged the body of Jesus. Then Pilate commanded the body to be delivered.
And when Joseph had taken the body, he wrapped it in a clean linen cloth,
And laid it in his own new tomb, which he had hewn out in the rock: and he rolled a great stone to the door of the sepulchre, and departed.
Mark 15:43-46 View context
Joseph of Arimathaea, an honourable counsellor, which also waited for the kingdom of God, came, and went in boldly unto Pilate, and craved the body of Jesus.
And Pilate marvelled if he were already dead: and calling [unto him] the centurion, he asked him whether he had been any while dead.
And when he knew [it] of the centurion, he gave the body to Joseph.
And he bought fine linen, and took him down, and wrapped him in the linen, and laid him in a sepulchre which was hewn out of a rock, and rolled a stone unto the door of the sepulchre.
Luke 23:50-53 View context
And, behold, [there was] a man named Joseph, a counsellor; [and he was] a good man, and a just:
(The same had not consented to the counsel and deed of them;) [he was] of Arimathaea, a city of the Jews: who also himself waited for the kingdom of God.
This [man] went unto Pilate, and begged the body of Jesus.
And he took it down, and wrapped it in linen, and laid it in a sepulchre that was hewn in stone, wherein never man before was laid.
Joseph of Arimathaea and Nicodemus
John 19:38-42 View context
And after this Joseph of Arimathaea, being a disciple of Jesus, but secretly for fear of the Jews, besought Pilate that he might take away the body of Jesus: and Pilate gave [him] leave. He came therefore, and took the body of Jesus.
And there came also Nicodemus, which at the first came to Jesus by night, and brought a mixture of myrrh and aloes, about an hundred pound [weight].
Then took they the body of Jesus, and wound it in linen clothes with the spices, as the manner of the Jews is to bury.
Now in the place where he was crucified there was a garden; and in the garden a new sepulchre, wherein was never man yet laid.
There laid they Jesus therefore because of the Jews' preparation [day]; for the sepulchre was nigh at hand.
The Jews and their rulers
Acts 13:27-29 View context
For they that dwell at Jerusalem, and their rulers, because they knew him not, nor yet the voices of the prophets which are read every sabbath day, they have fulfilled [them] in condemning [him].
And though they found no cause of death [in him], yet desired they Pilate that he should be slain.
And when they had fulfilled all that was written of him, they took [him] down from the tree, and laid [him] in a sepulchre.


SQL is a language that allows prorammers to access data in databases. Most of the time (always?) you pass your queries in form of strings to the database. In online services it is quite common that the programmer formulates a template and the user fills in variables.
Example: IMDb
Take a look at IMDb. Users can search for movies by title:
When you search for "Harry Potter" for example, the following happens:
You obviously interacted with imdb.com in a very dynamic way. The output of the website depends on what you typed in and IMDb has to search in its database for your search terms.
The programmers might have created a query that looks like this
SELECT * FROM `movie` WHERE title=$_GET['q']
Where $_GET['q']
is your query.
Proof of concept
You need:
- Apache Web Server
- PHP
- MySQL
- PhpMyAdmin (for convenience)
When you search for "LAMP" (for Linux users) or for "WAMP" (for Windows users) you find a lot of information how to install this.
Place the following as hack.php
in your web servers directory (might be /var/www
):
<?
$mysqlhost = "localhost";
$mysqluser = "root";
$mysqlpwd = "asdfasdf";
$connection = mysql_connect($mysqlhost, $mysqluser, $mysqlpwd) or die
("Your connection string was wrong");
$db_selected = mysql_select_db('imdb', $connection);
$q = $_GET['q'];
if ($q != "") {
$result = mysql_query("SELECT * FROM `movies` WHERE title='$q'");
if (!$result) {
die('MySQL query error: ' . mysql_error());
}
echo "Found ".mysql_num_rows($result)." movies:
";
while ($row = mysql_fetch_assoc($result)) {
echo $row["id"].": ".$row["title"]."
";
}
}
?>
<form method="get" action="hack.php">
<input type="text" name="q"/>
<input type="submit" />
</form>
Now create a database called imdb
with PHPMyAdmin and execute the following SQL:
CREATE TABLE IF NOT EXISTS `movies` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) AUTO_INCREMENT=4;
INSERT INTO `movies` (`id`, `title`) VALUES
(1, 'Harry Potter'),
(2, 'Lord of the Rings'),
(3, 'Rise of the Silver Surfer');
CREATE TABLE IF NOT EXISTS `users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`username` varchar(255) NOT NULL,
`password` varchar(32) NOT NULL,
`email` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) AUTO_INCREMENT=3 ;
INSERT INTO `users` (`id`, `username`, `password`, `email`) VALUES
(1, 'admin', 'qewrtqwert', 'admin@imdb.com'),
(2, 'user', 'secret', 'mylittlepony@stupid.com');
Now go to http://localhost/hack.php. It should look like this:
When you search for "Harry Potter" it should show you "1: Harry Potter". Note that there could be a lot of information, but I wanted to keep this example as small as possible.
This resulted in the following query:
SELECT * FROM `movies` WHERE title='Harry Potter'
But a Hacker could also enter a string like this: ' OR '1'='1
:
Even worse, the attacker could know that you use MySQL. Then he might know that MySQL uses INFORMATION_SCHEMA tables. He might enter this into the title input element:
' UNION SELECT table_name, table_type FROM information_schema.tables WHERE '1'='1
which results in this query:
SELECT * FROM `movies` WHERE title='' UNION SELECT table_name, table_type FROM information_schema.tables WHERE '1'='1'
which gives:
This way, the attacker gets all table names from all databases on this machine. So he essentially can get everything stored in your database. And, of course, after getting everything he could drop it:
History
Just a few famous examples to show you that this happens all the time:
- 2005: USC admissions page (source)
- 2006: 800,000 datasets of personal information of students of UCLA (source)
- 2008, 2009: Heartland Payment Systems; 130 million credit and debit cards (source 1, source 2, see Albert Gonzalez)
- 2010: Royal Navy Website; Passwords and Usernames stolen (source)
- 2011: Sony:
LulzSec says it accessed the passwords, email addresses, home addresses and dates of birth of one million users. The group says it also stole all admin details of Sony Pictures, including passwords. 75,000 music codes and 3.5 million music coupons were also accessed, according to the press release.
(source) - 2011: Expedia (source)
- 2011: MySql - Usernames and passwords stolen (source 1, source 2)
- 2011: Comodo (source 1, source 2)
Solutions
- Sanitize user input, e.g. with mysql_real_escape_string
- Use prepared statements
- Switch of error reporting (this makes attacks more difficult, but doesn't prevent them)
See also
- Challenge Websites: Try if you can write SQL injections yourself ☺


Demo 1 - Basic Inputs
First Name (only "Dave" allowed)
Last Name (not validated by Ok, anything goes)
Email address (Optional, only ".de" allowed)
Demo 2 - Advanced Inputs
First Name (only starting with 'D' allowed; Only uppercase allowed)
Last Name ("Bar" may not be used)
Foo
Bar
Fizz
Message (Must have an even number of letters)
File (Name must contain he letter 'a')
Checkbox to tick (Has to be ticked. Used to set custom message)
Demo 3 - Message customization
First Name (Only 'Dave' allowed)


IM.js
Quick image comparison pixel by pixel.
Examples:
Test different image files with same pixel info.
Test different image files with, only, one pixel with different pixel info in one of the images.
Test different image files tolerance is set to 80% of matching. Only one different pixel.
Test different image files with different image info and show the diff in one new canvas.
Test different image files tolerance is set to 80% of matching. The different image has 50% of different pixel info.
Other projects:
Hydra.js
Hermes.js
Design Patterns
Cutter.js
CSSplus


Nathaniel Jansen
About
|
Portfolio
|
Contact
Portfolio
Coming Soon
Coming Soon
Coming Soon
Coming Soon
Connect with Me


Maszlee: Nepal plans to introduce Bahasa Melayu as elective subject in schools
KUALA LUMPUR, Nov 22 — Nepal plans to introduce Bahasa Melayu (BM) as an elective subject in its schools to equip students with communication skills in the professional setting in Malaysia.
The matter, however, needs further discussion with Nepalese Cabinet and relevant authorities, said Nepalese Education, Science and Technology Minister Giriraj Mani Pokharel when he held a bilateral meeting with Malaysia’s Education Minister Maszlee Malik recently.
Giriraj also welcomed Maszlee’s proposal to set up a Malaysian language institution in the South Asian nation to teach BM to Nepalese.
Speaking to Bernama when met on the sidelines of the World Innovative Summit For Education 2019, held between November 19 to 21 in Doha, Qatar, Maszlee said:
“There are approximately 357,000 Nepali currently working in Malaysia, primarily in the security, manufacturing and hospitality sectors.
“Teaching BM and setting up institutions would be useful when Nepalese seek employment in Malaysia as they would be able to communicate fluently and boost their career.”
Maszlee said both leaders also discussed on enhancing and improving Technical and Vocational Education and Training (TVET) studies in Nepal.
“Nepalese minister was impressed with our German Dual Training System for TVET and for them, it would be useful.
“This is because Nepal wants to produce skilful Nepalese workers to work abroad with better pay and a TVET certification,” he said.
Maszlee said Nepal was also looking for opportunities to send their students to pursue post-graduate studies in Malaysia, and to collaborate with Malaysian universities.
Giriraj also extended an invitation to Maszlee to visit Nepal in conjunction with the Visit Nepal 2020, and to attend the Artificial Intelligence Conference in Kathmandu next year. — Bernama


The Jaxon Files
About
Contact
SStudies
Amer Govt/Pltcs
US History
Economics
Science
Biology
Chemistry
Physics
ELA
Literature 9
Literature 10
English Language
Math
Alg I, Geo A
Alg II, Geo B
Precalculus
Calculus


Notes
This map shows reported COVID-19 cases per capita as a Local Authority hex cartogram - this gives each Local Authority the same visual weight on the map. Some Local Authorities in England are missing because of inconsistencies. We don't have data for Scotland, Wales, or Northern Ireland. If you know where they publish CSV files of these numbers, please get in touch.
Note: Public Health England provide raw data for "Upper Tier Local Authorities". This map is for Local Authorities so where an Upper Tier Local Authority isn't a Local Authority, each Local Authority in that Upper Tier Local Authority is given the same per-capita value (UK geography is hard). When you click on one of these Local Authorities you will be shown the total for the Upper Tier Local Authority.
Visualisation: ODI Leeds / Data: Public Health England provide COVID-19 cases as a CSV download. The population estimates (2020) for Local Authorities are from the ONS.


HIV Arm (%) ART study incidence participants intervention (.) (.%) HIV-positive data Medicine Health coverage testing communities {border-style University community effect px;} #ffffff;} HPTN (., Arms School population prevalence prevention trial London viral Research estimated treatment Triplet men similar suppression Department Health, [], arms baseline number provided vs. PopART S). South universal (Table Desmond Hygiene PC, Stellenbosch Tropical Tutu UTT enrolled impact observed rate ratio used (PopART) Adjusted CHiPs Centre, Child Faculty Paediatrics Sciences, ZAMBART Zambia aged evidence proportion sex (PC) CI) International National Overall Study analysis compared infection local primary px; results study, women years {font-family (Figure .-., AIDS Africa Center College Data England Imperial Institute January New antiretroviral collab: consent end follow-up font-size included including matched performed seven shown shows triplets Missing < (%CI: Cancer FHI Figure Fred Hutchinson Table Viral arms. did double; effects incidence, intervention, measured partners reduction sample status supplementary using value written (see ./py; Baseline Cape Centre Population Zambart additional adjusted based border-color border-width collected combination communities. data, delivered delivery government greater guidelines higher infections lower medicine;:-. months offered reported rural solid; support table therapy transmission {font-style {font-weight %CI: Africa. Analysis HIV-negative HayesR, Impact Lancet Medical Oxford Statistical Trial UNAIDS age, analyses available bold care community-randomized consistent courier; determined em; enrollment estimates funding high individual individuals informed interaction load mathematical members men: methods occurred participants. person-years pt;} random randomized reduce reduced research restricted review services sexual showed target test triplet uptake


class: center, middle ### W4995 Applied Machine Learning # Preprocessing and Feature Transformations 02/06/19 Andreas C. Müller ??? Today we’ll talk about preprocessing and feature engineering. What we’re talking about today mostly applies to linear models, and not to tree-based models, but it also applies to neural nets and kernel SVMs. FIXME: add quantile transformer FIXME: some slides have screenshots FIXME: target encoder from dirtycat or categorical encoder contrib package? FIXME better example / explanation for target encoding FIXME add more complex grid-search pipeline example, add grid-searching over pipeline steps FIXME add grid-search column-transformer example FIXME add hashing, add Max number of columns FIXME walk through example of frequency encoder better, call target encoder FIXME add ELS example about leaking information with feature selection!!! FIXME add code corresponding to correct and incorrect scaling of test data --- class: center, middle Coming up with features is difficult, time-consuming, requires expert knowledge. "Applied machine learning" is basically feature engineering. .quote_author[Andrew Ng] --- class: middle ![:scale 90%](images/boston_scatter.png) ??? Let’s take a look at the the boston housing dataset. The idea was to predict house prices. Here are the features on the x axis and the response, so price, on the y axis. What are some thing you can notice? (concentrated distributions, skewed distibutions, discrete variable, linear and non-linear effects, different scales) The last thing that we're going to do after we did all the preprocessing is a little bit more about feature engineering and how to add features. Feature engineering is most important if we have simple models like linear models. If you do something like support vector machines or neural networks, you probably don't have to do as much feature engineering the because the model can learn more complex things themselves. --- class: center, middle #Scaling ??? N/A --- class: center, middle .center[ ![:scale 70%](images/img_2.png) ] ??? Let’s start with the different scales. Many model want data that is on the same scale. Linear models: the different scales mean different penalty. L2 is the same for all! We can also see non-gaussian distributions here btw! --- # Scaling and Distances ![:scale 90%](images/knn_scaling.png) ??? Here is an example of the importance of scaling using a distance-based algorithm, K nearest neighbors. My favorite toy dataset with two classes in two dimensions. The scatter plots look identical, but on the left hand side, the two axes have very different scales. The x axis has much larger values than the y axis. On the right hand side, I used standard scaler and so both features have zero mean and unit variance. So what do you think will happen if I use k nearest neighbors here? Let's see --- # Scaling and Distances ![:scale 90%](images/knn_scaling2.png) ??? As you can see, the difference is quite dramatic. Because the X axis has such a larger magnitude on the left-hand side, only distances along the x axis matter. However, the important feature for this task is the y axis. So the important feature gets entirely ignored because of the different scales. And usually the scales don't have any meaning - it could be a matter of changing meters to kilometers. --- class: center # Ways to Scale Data
![:scale 80%](images/img_3.png) ??? Here's an illustration of four of the most common ways. One of the most common ones that we already saw before is the Standard Scaler. StandardScaler subtracts the mean and divides by standard deviation. Making all the features have a zero mean and a standard deviation of one. One thing about this is that it won't guarantee any minimum or maximum values. The range can be arbitrarily large. MinMaxScaler subtracts minimum and divides by range. Scales between 0 and 1. So all the features will have an exact minimum at zero and exact maximum at one. This mostly makes sense if there are actually minimum and maximum values in your data set. If it's actually Gaussian distribution, the scaling might not make a lot of sense, because if you add one more data point, it's very far away and it will cram all the other data points more together. This will make sense if you have a grayscale value between 0 and 255 or something else that has like clearly defined boundaries. Another alternative is the RobustScaler. It’s the robust version of the StandardScaler. It computes the median and the quartiles. This cannot be skewed by outliers. The StandardScaler uses mean and standard deviation. So if you have a point that’s very far away, it can have unlimited influence on the mean. The RobustScaler uses robust statistics, so it’s not skewed be outliers. The final one is the Normalizer. This projects things either on the L1 or L0, meaning it makes sure to vectors have length one either in L1 norm or L2 norm. If you do this for L2 norm, it means you don't care about the length you project on a circle. More commonly used in L1 norm, it projects onto the diamond. What that does is basically it means you make sure the sum of all the entries is one. That's often used if you have histograms or if you have counts of things. If you want to make sure that you have frequency features instead of count features, you can use the L1 normalizer. --- class: spacious # Sparse Data - Data with many zeros – only store non-zero entries. - Subtracting anything will make the data “dense” (no more zeros) and blow the RAM. - Only scale, don’t center (use MaxAbsScaler) ??? FIXME ugly slide There’s another one that's important for sparse data. It’s called the MaxAbsScaler in scikit-learn. So sparse data is the kind of data that happens actually quite frequently in practice where most of the features are zero most of the time. So if you have tens of thousands of features, but they're always nearly zero. For example, you can think about these being particular user actions and at any time, any user makes only very few actions. But there are many, many possibilities. So if you have data like that, you only want to store the non-zero values, you don't want to store all the zeros. If you would store all the zeros, the data wouldn't even fit into the RAM often. So you have this very large sparse data set and you want to scale it, but if you subtract anything from it, if you want to make it zero mean then all the entries will be non-zero because the chance that the mean is actually zero is small. So then if you subtract something from everything, you're going to move all those zeros away from zero and then you can't store the data in sparse format and your RAM will blow up. Basically, you can subtract anything from sparse matrix because that won't be sparse anymore. But we can still scale it because any number of times zero is still zero. So if we scale it, then it will have the same structure again. The MaxAbsScaler sets the maximum absolute values to one. Basically, it looks at the maximum absolute values in the whole dataset, or for each feature and makes sure that this is one, just by scaling by one divided by the maximum absolute value. --- # Standard Scaler Example ```python from sklearn.linear_model import Ridge X, y = boston.data, boston.target X_train, X_test, y_train, y_test = train_test_split( X, y, random_state=0) scaler = StandardScaler() scaler.fit(X_train) X_train_scaled = scaler.transform(X_train) ridge = Ridge().fit(X_train_scaled, y_train) X_test_scaled = scaler.transform(X_test) ridge.score(X_test_scaled, y_test) ``` ``` 0.634 ``` ??? Here’s how you do the scaling with StandardScaler in scikit-learn. Similar interface to models, but“transform” instead of “predict”. “transform” is always used when you want a new representation of the data. Fit on training set, transform training set, fit ridge on scaled data, transform test data, score scaled test data. The fit computes mean and standard deviation on the training set, transform subtracts the mean and the standard deviation. We fit on the training set and apply transform on both the training and the test set. That means the training set mean gets subtracted from the test set, not the test-set mean. That’s quite important. --- class: center, middle .center[ ![:scale 100%](images/img_5.png) ] ??? Here’s an illustration why this is important using the min-max scaler. Left is the original data. Center is what happens when we fit on the training set and then transform the training and test set using this transformer. The data looks exactly the same, but the ticks changed. Now the data has a minimum of zero and a maximum of one on the training set. That’s not true for the test set, though. No particular range is ensured for the test-set. It could even be outside of 0 and 1. But the transformation is consistent with the transformation on the training set, so the data looks the same. On the right you see what happens when you use the test-set minimum and maximum for scaling the test set. That’s what would happen if you’d fit again on the test set. Now the test set also has minimum at 0 and maximum at 1, but the data is totally distorted from what it was before. So don’t do that. --- # Sckit-Learn API Summary ![:scale 90%](images/img_6.png) Efficient shortcuts: ```python est.fit_transform(X) == est.fit(X).transform(X) # mostly est.fit_predict(X) == est.fit(X).predict(X) # mostly ``` ??? Here’s a summary of the scikit-learn methods. All models have a fit method which takes the training data X_train. If the model is supervised, such as our classification and regression models, they also take a y_train parameter. The scalers don’t use a y_train because they don’t use the labels at all – you could say they are unsupervised methods, but arguably they are not really learning methods at all. Models (also known as estimators in scikit-learn) to make a prediction of a target variable, you use the predict method, as in classification and regression. If you want to create a new representation of the data, a new kind of X, then you use the transform method, as we did with scaling. The transform method is also used for preprocessing, feature extraction and feature selection, which we’ll see later. All of these change X into some new form. There’s two important shortcuts. To fit an estimator and immediately transform the training data, you can use fit_transform. That’s often more efficient then using first fit and then transform. The same goes for fit_predict. --- class: smaller, compact ```python from sklearn.model_selection import cross_val_score from sklearn.linear_model import RidgeCV scores = cross_val_score(RidgeCV(), X_train, y_train, cv=10) np.mean(scores), np.std(scores) ``` ``` (0.717, 0.125) ``` ```python scores = cross_val_score(RidgeCV(), X_train_scaled, y_train, cv=10) np.mean(scores), np.std(scores) ``` ``` (0.718, 0.127) ``` ```python from sklearn.neighbors import KNeighborsRegressor scores = cross_val_score(KNeighborsRegressor(), X_train, y_train, cv=10) np.mean(scores), np.std(scores) ``` ``` (0.499, 0.146) ``` ```python from sklearn.neighbors import KNeighborsRegressor scores = cross_val_score(KNeighborsRegressor(), X_train_scaled, y_train, cv=10) np.mean(scores), np.std(scores) ``` ``` (0.750, 0.106) ``` ??? Let’s apply the scaler to the Boston housing data. First I used the StandardScaler to scale the training data. Then I applied ten-fold cross-validation to evaluate the Ridge model on the data with and without scaling. I used RidgeCV which automatically picks alpha for me. With and without scaling we get an R^2 of about .72, so no difference. Often there is a difference for Ridge, but not in this case. If we use KneighborsRegressor instead, we see a big difference. Without scaling R^2 is about .5, and with scaling it’s .75. That makes sense since we saw that for distance calculations basically all features are dominated by the TAX feature. However, there is a bit of a problem with the analysis we did here. Can you see it? --- class: center, middle # A note on preprocessing # (and pipelines) ??? I want to talk a bit more about preprocessing and cross-validation here, and introduce pipelines. --- class: some-space #Leaking Information .left-column[ Information Leak ![:scale 100%](images/img_9.png) ] .right-column[ No Information leakage ![:scale 100%](images/img_10.png)] .reset-column[ Need to include preprocessing in cross-validation ! ] ??? What we did was we trained the scaler on the training data, and then applied cross-validation to the scaled data. Tha’s what’s show on the left. The problem is that we use the information of all of the training data for scaling, so in particular the information in the test fold. This is also known as information leakage. If we apply our model to new data, this data will not have been used to do the scaling, so our cross-validation will give us a biased result that might be too optimistic. On the right you can see how we should do it: we should only use the training part of the data to find the mean and standard deviation, even in cross-validation. That means that for each split in the cross-validation, we need to scale the data a bit differently. This basically means the scaling should happen inside the cross-validation loop, not outside. In practice, estimating mean and standard deviation is quite robust and you will not see a big difference between the two methods. But for other preprocessing steps that we’ll see later, this might make a huge difference. So we should get it right from the start. --- class: smaller ```python from sklearn.linear_model import Ridge X, y = boston.data, boston.target X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0) scaler = StandardScaler() scaler.fit(X_train) X_train_scaled = scaler.transform(X_train) ridge = Ridge().fit(X_train_scaled, y_train) X_test_scaled = scaler.transform(X_test) ridge.score(X_test_scaled, y_test) ``` ``` 0.634 ``` ```python from sklearn.pipeline import make_pipeline pipe = make_pipeline(StandardScaler(), Ridge()) pipe.fit(X_train, y_train) pipe.score(X_test, y_test) ``` ``` 0.634 ``` ??? Now I want to show you how to do preprocessing and crossvalidation right with scikit-learn. At the top here you see the workflow for scaling the data and then applying ridge again. Fit the scaler on the training set, transform on the training set, fit ridge on the training set, transform the test set, and evaluate the model. Because this is such a common pattern, scikit-learn has a tool to make this easier, the pipeline. The pipeline is an estimator that allows you to chain multiple transformations of the data before you apply a final model. You can build a pipeline using the make_pipeline function. Just provide as parameters all the estimators. All but the last one need to have a transform method. Here we only have two steps: the standard scaler and ridge. make_pipeline returns an estimator that does both steps at once. We can call fit on it to fit first the scaler and then ridge on the scaled data, and when we call score, it transforms the data and then evaluates the model. The code below is exactly equivalent to the code above, only shorter. --- class: left, middle .center[ ![:scale 70%](images/img_13.png) ] ??? Let’s dive a bit more into the pipeline. Here is an illustration of what happens with three steps, T1, T2 and Classifier. Imagine T1 to be a scaler and T2 to be any other transformation of the data. If we call fit on this pipeline, it will first call fit on the first step with the input X. Then it will transform the input X to X1, and use X1 to fit the second step, T2. Then it will use T2 to transform the data from X1 to X2. Then it will fit the classifier on X2. If we call predict on some data X’, say the test set, it will call transform on T1, creating X’1. Then it will use T2 to transform X’1 into X’2, and call the predict method of the classifier on X’2. This sounds a bit complicated, but it’s really just doing “the right thing”to apply multiple transformation steps. --- .padding-top[ ```python from sklearn.neighbors import KNeighborsRegressor knn_pipe = make_pipeline(StandardScaler(), KNeighborsRegressor()) scores = cross_val_score(knn_pipe, X_train, y_train, cv=10) np.mean(scores), np.std(scores) ``` ``` (0.745, 0.106) ``` ] ??? How does that help with the cross-validation problem? Because now all steps are contain in pipeline, we can simply pass the whole pipeline to crossvalidation, and all processing will happen inside the cross-validation loop. That solve the data leakage problem. Here you can see how we can build a pipeline using a standard scaler and kneighborsregressor and pass it to cross-validation. --- # Naming Steps ```python from sklearn.pipeline import make_pipeline knn_pipe = make_pipeline(StandardScaler(), KNeighborsRegressor()) print(knn_pipe.steps) ``` ``` [('standardscaler', StandardScaler(with_mean=True, with_std=True)), ('kneighborsregressor', KNeighborsRegressor(algorithm='auto', ...))] ``` ```python from sklearn.pipeline import Pipeline pipe = Pipeline((("scaler", StandardScaler()), ("regressor", KNeighborsRegressor))) ``` ??? But let’s talk a bit more about pipelines, because they are great. The pipeline has an attribute called steps, which --- contains its steps. Steps is a list of tuples, where the first entry is a string and the second is an estimator (model). The string is the “name” that is assigned to this step in the pipeline. You can see here that our first step is called “standardscaler” in all lower case letters, and the second is called kneighborsregressor, also all lower case letters. By default, step names are just lowercased classnames. You can also name the steps yourself using the Pipeline class directly. Then you can specify the steps as tuples of name and estimator. make_pipeline is just a shortcut to generate the names automatically. --- # Pipeline and GridSearchCV .small-padding-top[ ```python from sklearn.model_selection import GridSearchCV knn_pipe = make_pipeline(StandardScaler(), KNeighborsRegressor()) param_grid = {'kneighborsregressor__n_neighbors': range(1, 10)} grid = GridSearchCV(knn_pipe, param_grid, cv=10) grid.fit(X_train, y_train) print(grid.best_params_) print(grid.score(X_test, y_test)) ``` ``` {'kneighborsregressor__n_neighbors': 7} 0.60 ``` ] ??? These names are important for using pipelines with gridsearch. Recall that for using GridSearchCV you need to specify a parameter grid as a dictionary, where the keys are the parameter names. If you are using a pipeline inside GridSearchCV, you need to specify not only the parameter name, but also the step name – because multiple steps could have a parameter with the same name. The way to do this is to use the stepname, then two underscores, and then the parameter name, as the key for the param_grid dictionary. You can see that the best_params_ will have this same format. This way you can tune the parameters of all steps in a pipeline at once! And you don’t have to worry about leaking information, since all transformations are contained in the pipeline. You should always use pipelines for preprocessing. Not only does it make your code shorter, it also makes it less likely that you have bugs. --- class: left, middle # Discrete features --- # Categorical Variables .smaller[ ```python import pandas as pd df = pd.DataFrame( {'boro': ['Manhattan', 'Queens', 'Manhattan', 'Brooklyn', 'Brooklyn', 'Bronx'], 'vegan': ['No', 'No','No','Yes', 'Yes', 'No']}) ``` ]
boro
vegan
0
Manhattan
No
1
Queens
No
2
Manhattan
No
3
Brooklyn
Yes
4
Brooklyn
Yes
5
Bronx
No
??? Before we can apply a machine learning algorithm, we first need to think about how we represent our data. Earlier, I said x \in R^n. That’s not how you usually get data. Often data has units, possibly different units for different sensors, it has a mixture of continuous values and discrete values, and different measurements might be on totally different scales. First, let me explain how to deal with discrete input variables, also known as categorical features. They come up in nearly all applications. Scikit-learn requires you to explicitly handle these, and assumes in general that all your input is continuous numbers. This is different from how many libraries in R do it, which deal with categorical variables implicitly. --- # Ordinal encoding .smaller[ ```python df['boro_ordinal'] = df.boro.astype("category").cat.codes df ``` ] .left-column[
boro
boro_ordinal
vegan
0
Manhattan
2
No
1
Queens
3
No
2
Manhattan
2
No
3
Brooklyn
1
Yes
4
Brooklyn
1
Yes
5
Bronx
0
No
] -- .right-column[ ![:scale 100%](images/boro_ordinal.png) ] ??? If you encode all three values using the same feature, then you are imposing a linear relation between them, and in particular you define an order between the categories. Usually, there is no semantic ordering of the categories, and so we shouldn’t introduce one in our representation of the data. --- # Ordinal encoding .smaller[ ```python df['boro_ordinal'] = df.boro.astype("category").cat.codes df ``` ] .left-column[
boro
boro_ordinal
vegan
0
Manhattan
2
No
1
Queens
3
No
2
Manhattan
2
No
3
Brooklyn
1
Yes
4
Brooklyn
1
Yes
5
Bronx
0
No
] .right-column[ ![:scale 100%](images/boro_ordinal_classification.png) ] --- # One-Hot (Dummy) Encoding .narrow-left-column[
boro
vegan
0
Manhattan
No
1
Queens
No
2
Manhattan
No
3
Brooklyn
Yes
4
Brooklyn
Yes
5
Bronx
No
] .wide-right-column[ .tiny[ ```python pd.get_dummies(df) ```
boro_Bronx
boro_Brooklyn
boro_Manhattan
boro_Queens
vegan_No
vegan_Yes
0
0
0
1
0
1
0
1
0
0
0
1
1
0
2
0
0
1
0
1
0
3
0
1
0
0
0
1
4
0
1
0
0
0
1
5
1
0
0
0
1
0
] ] ??? Instead, we add one new feature for each category, And that feature encodes whether a sample belongs to this category or not. That’s called a one-hot encoding, because only one of the three features in this example is active at a time. You could actually get away with n-1 features, but in machine learning that usually doesn’t matter One way to do is with Pandas. Here I have an example of a data frame where I have the boroughs of New York as a categorical variable and variable saying whether they are vegan. One to get the dummies is to get dummies on this data frame. This will create new columns, it will actually replace borough column by four columns that correspond to the four different values. The get_dummies applies transformation to all columns that have a dtype that's either object or categorical. In this case we didn't actually want to transform the target variable vegan. --- # One-Hot (Dummy) Encoding .narrow-left-column[
boro
vegan
0
Manhattan
No
1
Queens
No
2
Manhattan
No
3
Brooklyn
Yes
4
Brooklyn
Yes
5
Bronx
No
] .wide-right-column[ .tiny[ ```python pd.get_dummies(df, columns=['boro']) ```
vegan
boro_Bronx
boro_Brooklyn
boro_Manhattan
boro_Queens
0
No
0
0
1
0
1
No
0
0
0
1
2
No
0
0
1
0
3
Yes
0
1
0
0
4
Yes
0
1
0
0
5
No
1
0
0
0
] ] ??? We can specify selectively which columns to apply the encoding to. --- # One-Hot (Dummy) Encoding .narrow-left-column[
boro
vegan
0
2
No
1
3
No
2
2
No
3
1
Yes
4
1
Yes
5
0
No
] .wide-right-column[ .tiny[ ```python pd.get_dummies(df_ordinal, columns=['boro']) ```
vegan
boro_0
boro_1
boro_2
boro_3
0
No
0
0
1
0
1
No
0
0
0
1
2
No
0
0
1
0
3
Yes
0
1
0
0
4
Yes
0
1
0
0
5
No
1
0
0
0
] ] ??? This also helps if the variable was already encoded using integers. Sometimes, someone has already encoded the categorical variables to integers like here. So here this is exactly the same information only except instead of strings you have them numbered. If you call the get_dummies on this nothing happens because none of them are object data types or categorical data types. If you want to look at the One Hot Encoding, you can explicitly pass columns equal and this will transform into boro_1, boro_2, boro_3. In this case get_dummies usually wouldn't do anything, but we can tell it which variables are categorical and it will dummy encode those for us. --- class: center, middle .center[![:scale 50%](images/img_27.png)] .left-column[![:scale 30%](images/img_28.png)] .right-column[![:scale 50%](images/img_29.png)] ??? Let's say instead of predicting whether someone is vegan, we want to predict their salary. That would also be an integer. There is no way to a-priory distinguish that one of These should be treated as categorical and one as ordinal or continuous. --- class: center, middle .center[![:scale 70%](images/img_30.png)] ??? If someone else gives you a new data set and in this new data set there is Staten Island, Manhattan, Bronx and Brooklyn. So new dataset doesn't have anyone from Queens. So now you transform this with get_dummies, you get something that has the same shape as the original data but actually, the last column means something completely different. Because now the last column is Staten Island, not Queens. If someone gives you separate training and test data sets, if you call get_dummies, you don't know that the columns correspond actually to the same thing. Unless you take care of the names, unfortunately, scikit-learn completely ignores column names. --- class: smaller #Pandas Categorical Columns .smaller[ ```python import pandas as pd df = pd.DataFrame({'salary': [103, 89, 142, 54, 63, 219], 'boro': ['Manhattan', 'Queens', 'Manhattan', 'Brooklyn', 'Brooklyn', 'Bronx']}) df['boro'] = pd.Categorical(df.boro, categories=['Manhattan', 'Queens', 'Brooklyn', 'Bronx', 'Staten Island']) pd.get_dummies(df) ``` ] .tiny[
salary
boro_Manhattan
boro_Queens
boro_Brooklyn
boro_Bronx
boro_Staten Island
0
103
1
0
0
0
0
1
89
0
1
0
0
0
2
142
1
0
0
0
0
3
54
0
0
1
0
0
4
63
0
0
1
0
0
5
219
0
0
0
1
0
] ??? The way to fix this is by using Pandas categorical types. Since we know what the boroughs of Manhattan are, we can create Pandas categorical dtype, we can create this categorical dtype with the categories Manhattan, Queens, Brooklyn, Bronx, and Staten Island. So now I have my column here and I'm going to convert it to a categorical dtype. So now it will not actually store the strings. It will just internally store zero to four, and it will also store what are the possible values. If a call get_dummies it will use all the possible values and for each of the possible values it will create a column. Even though Staten Island has not appeared in my dataset, it will still make a column for Staten Island. If I fix this categorical dtype I can apply it to the training and test data set and that'll make sure all the columns are always the same no matter what are the values are actually in the data set. --- #OneHotEncoder - Deprecated mode, don't use. ```python from sklearn.preprocessing import OneHotEncoder df = pd.DataFrame({'salary': [103, 89, 142, 54, 63, 219], 'boro': [0, 1, 0, 2, 2, 3]}) ohe = OneHotEncoder(categorical_features=[0]).fit(df) ohe.transform(df).toarray() ``` ``` array([[ 1., 0., 0., 0., 103.], [ 0., 1., 0., 0., 89.], [ 1., 0., 0., 0., 142.], [ 0., 0., 1., 0., 54.], [ 0., 0., 1., 0., 63.], [ 0., 0., 0., 1., 219.]]) ``` ??? You can also do this with scikit-learn. There are two ways to do this scikit-learn. Here's the old way. It’s a transformer, so you can call fit on something and call transform on something else. This fit transform paradigm in scikit-learn prevents you from having these columns matches. What this does is basically, it only considers the ones that are there during training and if there's a new value coming in during testing, it either throws that out or gives you an error depending on what parameters you set. This will actually produce by default, the sparse matrix so it will not actually store all the zeros. an You can also set a configuration option so that it will always return. The annoying thing with this is, this works only for integers. So one hot encoder, if you give it a string, it'll just say no. The one hot encoder assumes that the integers are from zero to number of categories, so if I would want to encode the salary column, it'll create 219 features, which are also not always what you want. The other way to do it is a new way which is not released yet. What this does is it works on strings and integers. But it always does it on all columns and it computes the unique values. Here for salary there six unique value, so add six columns for salary and then there are four values for borough. So this is actually what you want but you only want to apply it to a subset of the columns. --- # OneHotEncoder - New mode: always transforms all columns ```python import pandas as pd df = pd.DataFrame({'salary': [103, 89, 142, 54, 63, 219], 'boro': ['Manhattan', 'Queens', 'Manhattan', 'Brooklyn', 'Brooklyn', 'Bronx']}) ce = OneHotEncoder().fit(df) ce.transform(df).toarray() ``` ``` array([[ 0., 0., 1., 0., 0., 0., 0., 1., 0., 0.], [ 0., 0., 0., 1., 0., 0., 1., 0., 0., 0.], [ 0., 0., 1., 0., 0., 0., 0., 0., 1., 0.], [ 0., 1., 0., 0., 1., 0., 0., 0., 0., 0.], [ 0., 1., 0., 0., 0., 1., 0., 0., 0., 0.], [ 1., 0., 0., 0., 0., 0., 0., 0., 0., 1.]]) ``` ??? - Fit-transform paradigm ensures train and test-set categories correspond. --- # The Future (it's here) ## OneHotEncoderEncoder + ColumnTransformer ```python categorical = df.dtypes == object preprocess = make_column_transformer( (StandardScaler(), ~categorical), (OneHotEncoder(), categorical)) model = make_pipeline(preprocess, LogisticRegression()) ``` ??? New in scikit-learn 0.20 is the column transformer, which allows you to transforms only some of the columns. For example, you can call categorical encoder only on the categorical columns and call StandardScaler on the non-categorical columns, and then use that to preprocess your data. Right now using Pandas, make sure your column names match up, make everything to an integer, or use column transformer and everything is awesome. In contrast to basically all other estimators in sklearn, this uses the column information in pandas and allows you to slice out different columns based on column names, integer indices or boolean masks. In this example I'm constructing a boolean mask --- class:center, middle ![:scale 100%](images/column_transformer_schematic.png) ??? Here's a schematic of the column transformer. Most commonly you might want to separate continuous and categorical columns, but you can select any subsets of columns you like. They can also overlap. Or you can apply multiple transformations to the same set of columns. Let's say I want a scaled version of the data, but I also want to extract principal components. I can use the same column as inputs to multiple transformers, and the results will be concatenated. FIXME add code --- class: some-space # OneHot vs Statisticians - One-hot is redundant (last one is 1 – sum of others) - Can introduce co-linearity - Can drop one - Choice which one matters for penalized models - Keeping all can make the model more interpretable ??? N/A --- class: some-space #Models Supporting Discrete Features - In principle: - All tree-based models, naive Bayes - In scikit-learn: - None - In scikit-learn "soon": - Decision trees, random forests ??? In principle all tree-based models support categorical features, in scikit-learn none of them do, hopefully, soon they will. So what you can do is either you do the One Hot Encoder or you just encode this as integers and treat it as a continuous. If you have very high categorical variables with many levels, maybe it keeping it as an integer might make more sense. --- class: center, middle # More encodings for categorical features: ## http://contrib.scikit-learn.org/categorical-encoding/ --- class: some-space #Target-Based Encoding - For high cardinality categorical features - Example: US states, given low samples - Instead of 50 one-hot variables, one “response encoded” variable. - For regression: - "people in this state have an average response of y” - Binary classification: – “people in this state have likelihood p for class 1” - Multiclass: – One feature per class: probability distribution ??? So there's also another way to encode categorical variables that is often used, I like to call it target-Based Encoding. It's basically for very high cardinality categorical features. For example, if you have categorical feature it's all US states and you don't have a lot of samples or if you have categorical features that's all US zip codes, if you have all different things, you don't want to do One Hot Encoding. So you get 50 new features, which if you don't have a lot of data would be a lot of features. So instead, you can use one single variable, it basically encodes the response. So for regression, it would be people in this state have an average response of that. Obviously you don't want to do this on the test set basically or you want to do this on the whole dataset for each level of the categorical variable, you want to find out what is the mean response and just use this as the future value. So you get one single future. For binary classification, you can just use the fraction of people that are classified as Class One. For multi-class, you usually do the percentage or fraction of people in each of the classes. So in multi-class, you get one new feature per class and you count for each state how many people in this state are classified for each of them. --- class: compact # Example: Adult census, native-country .smallest[ ```python data = pd.read_csv("adult.csv") data.columns ``` ``` Index(['age', 'workclass', 'education', 'education-num', 'marital-status', 'occupation', 'relationship', 'race', 'gender', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'income'], dtype='object') ``` ```python data['native-country'].value_counts() ``` .left-column[ ``` United-States 29170 Mexico 643 ? 583 Philippines 198 Germany 137 Canada 121 Puerto-Rico 114 El-Salvador 106 India 100 Cuba 95 England 90 Jamaica 81 South 80 China 75 Italy 73 Dominican-Republic 70 ``` ] .right-column[ ``` Vietnam 67 Guatemala 64 Japan 62 Poland 60 Columbia 59 Taiwan 51 Haiti 44 Iran 43 Portugal 37 Nicaragua 34 Peru 31 France 29 Greece 29 Ecuador 28 Ireland 24 Hong 20 Trinadad&Tobago 19 Cambodia 19 Thailand 18 Laos 18 Yugoslavia 16 Outlying-US(Guam-USVI-etc) 14 Hungary 13 Honduras 13 Scotland 12 Holand-Netherlands 1 Name: native-country, dtype: int64 ``` ] ] ??? A slightly different example which is the adult census. This is the dataset with stuff about jobs of adults living in the US and the idea is to predict whether their income is smaller or bigger than 50K a year. One of the variables in this is the country of origin. Most of the people in this dataset come from the US but there's a bunch of people from other countries.  FIXME this slide is terribly confusing So the way I do this is for each country that someone can be from, I can look at how many people make less than 50k and more than 50k. People from the Netherlands alway make less than 50k in this dataset. People from Iran doing pretty well prepared, people from Ireland are not doing so well. For each native country, I just added a feature that I call frequency. And if it's in the States, I give the feature value that corresponds to the United States in the table. So everybody from the United States will have the same feature and so on. And so now I just have a single feature that hopefully encodes the information of the country. --- class: left, middle # Feature Distributions ??? Now that we discussed scaling and pipelines, let’s talk about some more preprocessing methods. One important aspect is dealing with different input distributions. --- class: left, middle .center[ ![:scale 85%](images/img_17.png) ] ??? Here is a box plot of the boston housing data after transforming it with the standard scaler. Even though the mean and standard deviation are the same for all features, the distributions are quite different. You can see very concentrated distributions like Crim and B, and very skewed distribuations like RAD and Tax (and also crim and B). Many models, in particular linear models and neural networks, work better if the features are approximately normal distributed. Let’s also check out the histograms of the data to see a bit better what’s going on. --- class: left, middle .center[ ![:scale 90%](images/boston_hist.png) ] ??? Clearly CRIM and ZN and B are very peaked, and LSTAT and DIS and Age are very asymmetric. Sometimes you can use a hack like applying a logarithm to the data to get better behaved values. There is slightly more rigorous technique though. --- # Power Transformations .left-column[ $$bc_{\lambda}(x) = \cases{\frac{x^\lambda - 1}{\lambda} & \text{if } \lambda \neq 0\cr log(x) & \text{if } \lambda = 0 }$$ Only applicable for positive x! ] .right-column[ ![:scale 90%](images/img_19.png) ] .reset-column[ ```python from sklearn.preprocessing import PowerTransformer pt = PowerTransformer(method='box-cox') # for any data: Yeo-Johnson pt.fit(X) ``` ] ??? The Box-Cox transformation is a family of univariate functions to transform your data, parametrized by a parameter lambda. For lamda=1 the function is the identity, for lambda = 2 it is square, lambda =0 is log and there is many other functions in between. For a given dataset, a separate parameter lambda is determined for each feature, by minimizing the skewdness of the data (making skewdness close to zero, not close to -inf), so it is more “gaussian”. The skewdness of a function is a measure of the asymmetry of a function and is 0 for functions that are symmetric around their mean. Unfortunately the Box-Cox transformation is only applicable to positive features. --- .wide-left-column[ ![:scale 85%](images/boston_hist.png) ] .narrow-right-column[Before] .clear-column[] .wide-left-column[![:scale 85%](images/boston_hist_boxcox.png)] .narrow-right-column[After] ??? Here are the histograms of the original data and the transformed data. The title of each subplot shows the estimated lambda. If the lambda is close to 1, the transformation didn’t change much. If it is away from 1, there was a significant transformation. You can clearly see the effect on “CRIM” which was approximately log-transformed, and lstat and nox which were approximately transformed by sqrt. For the binary CHAS the transformation doesn’t make a lot of sense, though. --- .wide-left-column[ ![:scale 85%](images/boston_scatter.png) ] .narrow-right-column[Before] .clear-column[] .wide-left-column[![:scale 85%](images/boston_bc_scaled_scatter.png)] .narrow-right-column[After] ??? Here is a comparison of the feature vs response plot before and after the box-cox transformation. The dis, lstat and crim relationships now look a bit more obvious and linear. ---


« CSS « CSS Reference
The :nth-child
pseudo-class matches an element that has an+b-1
siblings before it in the document tree, for a given positive or zero value for n
, and has a parent element.
This can more clearly be described this way: the matching element is the bth
child of an element after all its children have been split into groups of a
elements each.
The values a
and b
must both be integers, and the index of an element's first child is 1.
Among other things, this allows selectors to match every other row in a table.
element:nth-child(an + b) { style properties }
tr:nth-child(2n+1)
tr:nth-child(odd)
tr:nth-child(2n)
tr:nth-child(even)
span:nth-child(0n+1)
:first-child
selector.span:nth-child(1)
span:nth-child(-n+3)
Usage example
span:nth-child(2n+1) { background-color: lime; }
...where...

 This span is limed! This span is not. :( But this one is! Sadly, this one is not...

... should look like ...
This span is limed! This span is not. :( But this one is! Sadly, this one is not...
| Feature | Chrome | Firefox (Gecko) | Internet Explorer | Opera | Safari |
|---|---|---|---|---|---|
| Basic support | 4.0 | 3.5 (1.9.1) | 9 | 10.10 | 4 |
| Feature | Android | Firefox Mobile (Gecko) | IE Mobile | Opera Mobile | Safari Mobile |
|---|---|---|---|---|---|
| Basic support | ? | ? | ? | ? | ? |