HMDA API Documentation
Introduction
For more information on HMDA, checkout the About HMDA page on the CFPB website.
The HMDA Platform
The new HMDA Platform (“V2”) was totally updated over the past year to utilize a more flexible and resilient approach (Kubernetes Microservices) in order to better serve our customers. This repository contains the code for the entirety of the HMDA platform backend. This platform has been designed to accommodate the needs of the HMDA filing process by financial institutions, as well as the data management and publication needs of the HMDA data asset.
The HMDA Platform uses sbt's multi-project builds, each project representing a specific task. The platform is an Akka Cluster application that can be deployed on a single node or as a distributed application. For more information on how Akka Cluster is used, see the documentation here
The HMDA Platform is composed of the following modules:
Parser (JS/JVM)
Module responsible for reading incoming data and making sure that it conforms to the HMDA File Specification
Data Validation
Module responsible for validating incoming data by executing validation rules as per the Edit Checks documentation
Persistence
Module responsible for persisting information into the system. It becomes the system of record for HMDA data
Cluster
Module responsible for managing the various cluster roles, as well as starting the Hmda Platform
API
This module contains both public APIs for HMDA data for general use by third party clients and web applications, as well as endpoints for receiving data and providing information about the filing process for Financial Institutions
API Model
This module contains objects and JSON protocols for use by the API project
Query
This module is responsible for interacting with the back-end database, as well as conversion between model objects and database objects.
Panel
This module is responsible for parsing and persisting a CSV-format panel file
Model (JS/JVM)
This module is responsible for maintaining the objects used in our platform
Census
This module is responsible for geographic translation (e.g. state number -> state code)
Publication
This module generates Aggregate and Disclosure reports, as required by HMDA statute.
Dependencies
Java 9 SDK
The HMDA Platform runs on the Java Virtual Machine (JVM), and requires the Java 9 JDK to build and run the project. This project is currently being built and tested on Oracle JDK 9. See Oracle's JDK Install Overview for install instructions.
The HMDA Platform should also run on JDK 8.
Scala
The HMDA Platform is written in Scala. To build it, you will need to download and install Scala 2.12.x
In addition, you'll need Scala's interactive build tool sbt. Please refer to sbt's installation instructions to get started.
Project structure
The HMDA Platform is divided into individual projects, each responsible for a subset of the functionality, as follows:
hmda-platform
This is the main filing application, exposing the APIs necessary to upload, validate and store HMDA files.
check-digit
Microservice that exposes functionality to create a check digit from a loan id, and to validate Univeral Loan Identifiers
Building and Running
Running from the SBT prompt
- To run the project from the
SBT
prompt for development purposes, issue the following commands on a terminal:
$ sbt
sbt:root> project hmda-platform
sbt:hmda-platform> reStart
Building and running the .jar
- To build JVM artifacts, from the sbt prompt first choose the project you want to build and use the assembly command:
$ sbt
sbt:root> project check-digit
sbt:check-digit>assembly
This task will create a fat jar
, which can be executed on any JDK9
compliant JVM
:
java -jar target/scala-2.12/check-digit.jar
Building and running the Docker image
- To build a
Docker
image that runs thehmda-platform
as a single node cluster, from the sbt prompt:
$sbt
sbt:root> project hmda-platform
sbt:hmda-platform> docker:publishLocal
This task will create a Docker
image. To run a container with the HMDA Platform
filing application as a single node cluster:
docker run --rm -ti -p 8080:8080 -p 8081:8081 -p 8082:8082 -p 19999:19999 hmda/hmda-platform
The same approach can be followed to build and run Docker containers for the other microservices that form the HMDA Platform.
Certain environment variables can be passed in to set the log level of the micro service
ZOOKEEPER_LOG_LEVEL (Defaulted to WARN)
KAFKA_LOG_LEVEL (Defaulted to INFO)
CASSANDRA_LOG_LEVEL (Defaulted to INFO)
PLATFORM_LOG_LEVEL (Defaulted to WARN)
INSTITUTION_LOG_LEVEL (Defaulted to INFO)
CHECKDIGIT_LOG_LEVEL (Defaulted to DEBUG)
Running the application in clustered mode (mesos)
For a 3 node cluster deployed through the DC/OS CLI, the following command can be used:
dcos marathon app add mesos/hmda-platform-host-mode.json
For more details, please refer to the Marathon Documentation
Resources
API Documentation
- HMDA Platform Public API Documentation
- HMDA Platform ULI API Documentation
- HMDA Platform Filers API Documentation
Development
Data Specifications
Platform Public API
This documentation describes de public HMDA Platform HTTP API
Institutions
Search Option 1
/institutions?domain=<domain>
GET
- Returns a list of institutions filtered by their email domain. If none are found, an HTTP 404 error code (not found) is returned
Example response, with HTTP code 200:
{
"institutions": [
{
"activityYear" : 2019,
"LEI" : "54930084UKLVMY22DS16",
"agency" : 1,
"institutionType" : 17,
"institutionId2017" : "12345",
"taxId" : "99-00000000",
"rssd" : 12345,
"emailDomains" : ["bank0.com"],
"respondent" : {
"name" : "xvavjuitZa",
"state" : "NC",
"city" : "Raleigh"
},
"parent" : {
"idRssd" :<PHONE_NUMBER>,
"name" : "Parent Name"
},
"assets" : 450,
"otherLenderCode" :<PHONE_NUMBER>,
"topHolder" : {
"idRssd" : 442825905,
"name" : "TopHolder Name"
},
"hmdaFiler" : true
}
]
}
Search Option 2
/institutions?domain=<domain>&lei=<lei>&respondentName=<respondentName>&taxId=<taxId>
GET
- Returns a list of institutions filtered by field values. If none are found, an HTTP 404 error code (not found) is returned
Example response, with HTTP code 200:
{
"institutions": [
{
"activityYear" : 2019,
"LEI" : "54930084UKLVMY22DS16",
"agency" : "1",
"institutionType" : "17",
"institutionId2017" : "12345",
"taxId" : "99-00000000",
"rssd" : "Pb",
"emailDomains"<EMAIL_ADDRESS>"respondent" : {
"name" : "xvavjuitZa",
"state" : "NC",
"city" : "Raleigh"
},
"parent" : {
"idRssd" : "1520162208",
"name" : "Parent Name"
},
"assets" : "450",
"otherLenderCode" : "1406639146",
"topHolder" : {
"idRssd" : "442825905",
"name" : "TopHolder Name"
},
"hmdaFiler" : true
}
]
}
Search Option 3
/institutions/<institutionID>
GET
Retrieves the details of an institution. If not found, returns HTTP code 404
Example Response with HTTP code 200, in
JSON
format:{ "activityYear" : 2019, "LEI" : "54930084UKLVMY22DS16", "agency" : 1, "institutionType" : 17, "institutionId2017" : "12345", "taxId" : "99-00000000", "rssd" : 12345, "emailDomains" : ["bank0.com"], "respondent" : { "name" : "xvavjuitZa", "state" : "NC", "city" : "Raleigh" }, "parent" : { "idRssd" :<PHONE_NUMBER>, "name" : "Parent Name" }, "assets" : 450, "otherLenderCode" :<PHONE_NUMBER>, "topHolder" : { "idRssd" : 442825905, "name" : "TopHolder Name" }, "hmdaFiler" : true }
TS Parsing and Validation
Parsing
/ts/parse
POST
Returns aJSON
representation of a TS, or a list of errors if the TS fails to parse
Example body, in
JSON
format:{ "ts" : "1|Bank<EMAIL_ADDRESS>Main St|Washington|DC|20001|9|100|99-999999|10Bx939c5543TqA1144M" }
Example response, in
JSON
format:{ "id": 1, "institutionName": "Bank 0", "year": 2018, "quarter": 4, "contact": { "name": "Jane", "phone": "111-111-1111", "email"<EMAIL_ADDRESS>"address": { "street": "123 Main St", "city": "Washington", "state": "DC", "zipCode": "20001" } }, "agency": 9, "totalLines": 100, "taxId": "99-999999", "LEI": "10Bx939c5543TqA1144M" }
Validation
/ts/validate
POST
- Returns aJSON
representation of a TS, or a list of edits if the TS fails to validate
Example body, in
JSON
format:{ "ts"<EMAIL_ADDRESS>Hocus Potato Way|Testtown|UT|84096|9|1000|02-1234567|BANK1LEIFORTEST12345" }
Example successful response, in
JSON
format: See aboveExample failed response, in
JSON
format:{ "syntactical": { "errors": [ "S300" ] }, "validity": { "errors": [ "V600", "V601" ] }, "quality": { "errors": [] } }
LAR Parsing and Validation
Parsing
/lar/parse
POST
- Returns aJSON
representation of a LAR, or a list of errors if the LAR fails to parse
Example body, in
JSON
format:{ "lar": "2|10Bx939c5543TqA1144M|10Bx939c5543TqA1144M999143X38|20180721|1|1|1|1|1|110500|1|20180721|123 Main St|Beverly Hills|CA|90210|06037|06037264000|1|1|1|1|1||1|1|1|1|1||3|3|5|7|7|7|7||||5|7|7|7|7||||3|3|1|1|3|3|30|30|36|1|0.428|1|1|750|750|1|9|1|9|10|10|10|10||2399.04|NA|NA|NA|NA|4.125|NA|42.95|80.05|360|NA|1|2|1|1|350500|1|1|5|NA|1|1|12345|1|1|1|1|1||1|1|1|1|1||1|1|1" }
Example response, in
JSON
format:{ "larIdentifier": { "id": 2, "LEI": "10Bx939c5543TqA1144M", "NMLSRIdentifier": "12345" }, "loan": { "ULI": "10Bx939c5543TqA1144M999143X38", "applicationDate": "20180721", "loanType": 1, "loanPurpose": 1, "constructionMethod": 1, "occupancy": 1, "amount": 110500, "loanTerm": "360", "rateSpread": "0.428", "interestRate": "4.125", "prepaymentPenaltyTerm": "NA", "debtToIncomeRatio": "42.95", "loanToValueRatio": "80.05", "introductoryRatePeriod": "NA" }, "larAction": { "preapproval": 1, "actionTakenType": 1, "actionTakenDate": 20180721 }, "geography": { "street": "123 Main St", "city": "Beverly Hills", "state": "CA", "zipCode": "90210", "county": "06037", "tract": "06037264000" }, "applicant": { "ethnicity": { "ethnicity1": 1, "ethnicity2": 1, "ethnicity3": 1, "ethnicity4": 1, "ethnicity5": 1, "otherHispanicOrLatino": "", "ethnicityObserved": 3 }, "race": { "race1": 5, "race2": 7, "race3": 7, "race4": 7, "race5": 7, "otherNativeRace": "", "otherAsianRace": "", "otherPacificIslanderRace": "", "raceObserved": 3 }, "sex": { "sex": 1, "sexObserved": 3 }, "age": 30, "creditScore": 750, "creditScoreType": 1, "otherCreditScoreModel": "9" }, "coApplicant": { "ethnicity": { "ethnicity1": 1, "ethnicity2": 1, "ethnicity3": 1, "ethnicity4": 1, "ethnicity5": 1, "otherHispanicOrLatino": "", "ethnicityObserved": 3 }, "race": { "race1": 5, "race2": 7, "race3": 7, "race4": 7, "race5": 7, "otherNativeRace": "", "otherAsianRace": "", "otherPacificIslanderRace": "", "raceObserved": 3 }, "sex": { "sex": 1, "sexObserved": 3 }, "age": 30, "creditScore": 750, "creditScoreType": 1, "otherCreditScoreModel": "9" }, "income": "36", "purchaserType": 1, "hoepaStatus": 1, "lienStatus": 1, "denial": { "denialReason1": 10, "denialReason2": 10, "denialReason3": 10, "denialReason4": 10, "otherDenialReason": "" }, "loanDisclosure": { "totalLoanCosts": "2399.04", "totalPointsAndFees": "NA", "originationCharges": "NA", "discountPoints": "NA", "lenderCredits": "NA" }, "nonAmortizingFeatures": { "balloonPayment": 1, "interestOnlyPayment": 2, "negativeAmortization": 1, "otherNonAmortizingFeatures": 1 }, "property": { "propertyValue": "350500.0", "manufacturedHomeSecuredProperty": 1, "manufacturedHomeLandPropertyInterest": 1, "totalUnits": 5, "multiFamilyAffordableUnits": "NA" }, "applicationSubmission": 1, "payableToInstitution": 2, "AUS": { "aus1": 1, "aus2": 1, "aus3": 1, "aus4": 1, "aus5": 1, "otherAUS": "" }, "ausResult": { "ausResult1": 1, "ausResult2": 1, "ausResult3": 1, "ausResult4": 1, "ausResult5": 1, "otherAusResult": "" }, "reverseMortgage": 1, "lineOfCredit": 1, "businessOrCommercialPurpose": 1 }
Validation
/lar/validate
POST
- Returns aJSON
representation of a LAR, or a list of edits if the LAR fails to validate
Example body, in
JSON
format:{ "lar": "2|10Bx939c5543TqA1144M|10Bx939c5543TqA1144M999143X38|20180721|1|1|1|1|1|110500|1|20180721|123 Main St|Beverly Hills|CA|90210|06037|06037264000|1|1|1|1|1||1|1|1|1|1||3|3|5|7|7|7|7||||5|7|7|7|7||||3|3|1|1|3|3|30|30|36|1|0.428|1|1|750|750|1|9|1|9|10|10|10|10||2399.04|NA|NA|NA|NA|4.125|NA|42.95|80.05|360|NA|1|2|1|1|350500|1|1|5|NA|1|1|12345|1|1|1|1|1||1|1|1|1|1||1|1|1" }
Example successful response, in
JSON
format: See aboveExample failed response, in
JSON
format:{ "syntactical": { "errors": [] }, "validity": { "errors": [ "V619-1", "V619-2", "V619-3", "V676-3", "V676-5", "V677-2" ] }, "quality": { "errors": [ "Q617" ] } }
HMDA File Parsing and Validation
Parsing
/hmda/parse
POST
- Parses a HMDA file. Returns list of errors per line inJSON
format, if found.
Example file:
1|Bank<EMAIL_ADDRESS>Main St|Washington|DC|20001|9|100|99-999999|10Bx939c5543TqA1144M 2|10Bx939c5543TqA1144M|10Bx939c5543TqA1144M999143X38|20180721|A|1|1|1|1|110500|1|20180721|123 Main St|Beverly Hills|CA|90210|06037|06037264000|1|1|1|1|1||1|1|1|1|1||3|3|5|7|7|7|7||||5|7|7|7|7||||3|3|1|1|3|3|30|30|36|1|0.428|1|1|750|750|1|9|1|9|10|10|10|10||2399.04|NA|NA|NA|NA|4.125|NA|42.95|80.05|360|NA|1|2|1|1|350500|1|1|5|NA|1|1|12345|1|1|1|1|1||1|1|1|1|1||1|1|1
Example Response, in
JSON
format:{ "validated": [ { "lineNumber": 2, "errors": "loan type is not numeric" } ] }
/hmda/parse/csv
POST
- Parses a HMDA file. Returns list of errors per line inCSV
format, if found.
Example file:
1|Bank<EMAIL_ADDRESS>Main St|Washington|DC|20001|9|100|99-999999|10Bx939c5543TqA1144M 2|10Bx939c5543TqA1144M|10Bx939c5543TqA1144M999143X38|20180721|A|1|1|1|1|110500|1|20180721|123 Main St|Beverly Hills|CA|90210|06037|06037264000|1|1|1|1|1||1|1|1|1|1||3|3|5|7|7|7|7||||5|7|7|7|7||||3|3|1|1|3|3|30|30|36|1|0.428|1|1|750|750|1|9|1|9|10|10|10|10||2399.04|NA|NA|NA|NA|4.125|NA|42.95|80.05|360|NA|1|2|1|1|350500|1|1|5|NA|1|1|12345|1|1|1|1|1||1|1|1|1|1||1|1|1
Example response, in
CSV
format:lineNumber|errors 2|loan type is not numeric
Check Digit(ULI)
Check digit generation
/uli/checkDigit
POST
- Calculates check digit and full ULI from a loan id.
Example payload, in JSON
format:
{
"loanId": "10Bx939c5543TqA1144M999143X"
}
Example response:
{
"loanId": "10Cx939c5543TqA1144M999143X",
"checkDigit": 10,
"uli": "10Cx939c5543TqA1144M999143X10"
}
A file with a list of Loan Ids can also be uploaded to this endpoint for batch check digit generation.
Example file contents:
10Cx939c5543TqA1144M999143X
10Bx939c5543TqA1144M999143X
Example response in JSON
format:
{
"loanIds": [
{
"loanId": "10Bx939c5543TqA1144M999143X",
"checkDigit": 38,
"uli": "10Bx939c5543TqA1144M999143X38"
},
{
"loanId": "10Cx939c5543TqA1144M999143X",
"checkDigit": 10,
"uli": "10Cx939c5543TqA1144M999143X10"
}
]
}
/uli/checkDigit/csv
POST
- calculates check digits for loan ids submitted as a file
Example file contents:
10Cx939c5543TqA1144M999143X
10Bx939c5543TqA1144M999143X
Example response in CSV
format:
loanId,checkDigit,uli
10Bx939c5543TqA1144M999143X,38,10Bx939c5543TqA1144M999143X38
10Cx939c5543TqA1144M999143X,10,10Cx939c5543TqA1144M999143X10
ULI Validation
/uli/validate
POST
- Validates a ULI (correct check digit)
Example payload, in JSON
format:
{
"uli": "10Bx939c5543TqA1144M999143X38"
}
Example response:
{
"isValid": true
}
A file with a list of ULIs can also be uploaded to this endpoint for batch ULI validation.
Example file contents:
10Cx939c5543TqA1144M999143X10
10Bx939c5543TqA1144M999143X38
10Bx939c5543TqA1144M999133X38
Example response in JSON
format:
{
"ulis": [
{
"uli": "10Cx939c5543TqA1144M999143X10",
"isValid": true
},
{
"uli": "10Bx939c5543TqA1144M999143X38",
"isValid": true
},
{
"uli": "10Bx939c5543TqA1144M999133X38",
"isValid": false
}
]
}
/uli/validate/csv
POST
- Batch validation of ULIs
Example file contents:
10Cx939c5543TqA1144M999143X10
10Bx939c5543TqA1144M999143X38
10Bx939c5543TqA1144M999133X38
Example response in CSV
format:
uli,isValid
10Cx939c5543TqA1144M999143X10,true
10Bx939c5543TqA1144M999143X38,true
10Bx939c5543TqA1144M999133X38,false
Filers API
Filters
/filers/<filing year>
GET
- Returns a list of filers for the filing year defined in the URI.
Note : This endpoint is currently only supported for the 2018 filing year
Example response:
{
"institutions":[
{
"lei":"12345677654321",
"name":"Test Bank 1",
"period":"2018"
},
{
"lei":"5734315621455",
"name":"Test Bank 12",
"period":"2018"
}
]
}
/filers/<filing year>/<lei>/msaMds
GET
- Returns all MsaMds for the specified filer
Example response:
{
"institution":{
"lei":"12345677654321",
"name":"Test Bank 1",
"period":"2018"
},
"msaMds":[
{
"id":"47664",
"name":"FARMINGTON HILLS,MI"
},
{
"id":"19180",
"name":"DANVILLE,IL"
}
]
}


Custom components
Besides navigation, navbar component supports custom content such as: dropdowns with submenu, dropdown menus with custom content, full width dropdowns, language selection, forms, tabs, multi level menus, date pickers etc. Everything is adapted to use in different navbar color schemes and sizes. Examples below demonstrate some of these components, other examples related to mega menu
can be found on this page.
Mixed components
Date range picker
Language selector
Navbar navigation
Basic navigation
Navbar navigation links build on our .nav
options with their own modifier class and require the use of toggler classes for proper responsive styling. Navigation in navbars will also grow to occupy as much horizontal space as possible to keep your navbar contents securely aligned. Dropdown menus require a wrapping element for positioning, so be sure to use separate and nested elements for .nav-item
and .nav-link
.
Navigation icons
Navbar navigation supports icons: text with left and right positioned icons, multiple icons within 1 item, icons only and carets for items with dropdown menu. By default, sidebar control buttons are placed in the left navigation. To add icons, place <i>
element with icon class to the navigation link element. To use with text, depending on the position place icon before or after item text. Use .ml-2
or .mr-2
to add left/right spacer between the icon and nav link text.
Badges
Flexibility of the navbar navigation also allows you to use 2 kinds of badges - inline
and floating
. Both types can have left and right positions. To use inline badge, add .position-static
class to the element, default placement is absolute with top-right position. To use left positioned elements, place it before text in inline version and add .right-auto.left-0
to the absolute positioned element.
Badge marks
Instead badges, which should contain text by default, navbar component supports badge marks - small rounded indicators. These indicators support all available colors and can have 2 different styles - circle and ring. To use ring, add .border-[color]
classes. To use circle, add .bg-[color]
and .border-[color]
classes. Both variations require .badge-mark
class by default. Also these indicators support left/right alignment and static/absolute positioning.
Basic components
Navbar buttons
Various buttons are supported as part of navbar components. This is also a great reminder that vertical alignment utilities can be used to align different sized elements. Button groups and button dropdowns in different colors, sizes and styles are also supported. For multiple buttons, use reponsive spacing
utility classes for proper vertical and horizontal alignment.
Navbar text
Wrap strings of text in an element with .navbar-text
, usually on a <span>
tag for proper leading and color. This class adjusts vertical alignment and horizontal spacing for strings of text. In some cases links within .navbar-text
may need color adjustments, use color
utility classes to style links properly.
Progress bars
Provide up-to-date feedback on the progress of a workflow or action with simple yet flexible progress bars. Progress bars inside navbar support all possible styling options: colors, animations, labels, appearance, sizes etc. Also you can add text and icon labels to display current action, it's also available in both left and right positions.
Form components
Place form content within container with spacing
utility classes for proper vertical alignment and collapsed behavior in narrow viewports. Use the alignment
options to decide where it resides within the navbar content. Navbar supports all form components: checkboxes, radios, default and custom selects, file inputs etc. Some form controls, like input groups, may require width
utility classes to be show up properly within a navbar.
Basic form controls
Default and custom selects
Input group, file select
Input with icons
Select2 select
Multiselect select
Checkboxes - both alignment
Radios - both alignment
Switchery toggles


This page briefly describes our superpixel-based classification framework for sub-image identification and percent cover estimation of benthic biota. The method is able to leverage existing expert annotation efforts. Typically less than 1 − 2 % of the collected images from benthic surveys end up being annotated and processed for science purposes, and usually only a subset of pixels within each image are scored. This results in a tiny fraction of total amount of collected data being utilised, O(0.00001%). The proposed framework uses these sparse, human-annotated point labels to train a superpixel-based automated classification system, which can be used to efficiently extrapolate the classified results to every pixel across all the images of an entire survey. The proposed framework has the potential to broaden the spatial extent and resolution for the identification and percent cover estimation of benthic biota.
LEFT: example of annotated image scored using sparse random points. RIGHT: example of image classified using the system described here.
The following diagram provides an overview of the system:
Flow diagram of the proposed pipeline for sub-image classification of benthic biota. The blue arrows show the ﬂow of unlabelled data and outputs from automated processing steps, and the red arrows show the ﬂow of data that requires manual annotation by a human expert.
Segmentation oﬀers some notable advantages over defining a fixed shaped and sized pixel patch for classification. For example, if a patch is positioned over a boundary between two class types, it may be diﬃcult to determine the class label assignment, which may confound the data used for training and prediction. The figure below shows an illustrative example of this.
Classification of sub-image regions using superpixels vs square patches. (a) shows a sample image with a 100 × 100 pixel bounding box around a chosen region of interest, (b) shows a zoomed in view of the chosen region and (c) shows the class ground truth. (d) shows the classification possible with a superpixel / segmentation based approach and (e), (f) and (g) show the classification possible using non-overlapping square patches of size 100 × 100, 50 × 50 & 25 × 25 pixels, respectively
It is evident that the resolution of the classification results may also be limited by the choice of patch size and the resolution of patch positioning. Large patches may contain multiple classes making it more diﬃcult to assign a single, specific class label and small patches may be diﬃcult to classify as they lack context. These factors aﬀect both the ability to classify and the resolution of the classification, which in turn may confound statistics, such as percent cover, which will be computed from the classification results. Examples of images classified using the superpixel framework are shown below.
Superpixel classification example images. The first row shows the original image examples; the second row shows labels overlaid onto segmented images (with the unlabeled superpixels coloured randomly); and the third row shows the output from the automated classifier.
In the image below, we can see the spatial layout of the classifier estimates vs the manually labeled points. The results show good correspondence between manual label estimates (black circles) and automated estimates (filled circles). In addition, the results appear to make sense scientifically: deeper regions are dominated by sand and the photosynthesising classes tend to be limited to the photic zone ( < 60m).
Spatial layout of percentage cover, estimated by automated superpixel classification for every pixel of all 7733 images in the survey compared to that estimated using 50 point-count of the 75 images that were scored using CPCe.
The classification results can also be used to query unannotated data. The image below show example images that have been extracted from the unlabeled data.
Non-overlapping unscored sample images for each class. Each row shows thumbnails of the 5 images that contain the highest proportion for each class. The figure also presents the range in percent cover across the images that are shown.
For more details, refer to:
A. Friedman, "Automated
interpretation of benthic stereo imagery", PhD Thesis, University of Sydney,
March, 2013


--- layout: default title: Contribute slug: contribute lead: "Get involved!" base_url: "../" ---
If you run into unexpected behavior or run into a bug please report it. All bugs are kept track of on our issue tracker. You can add a bug report there or if you are not sure it's a bug send an email to the developer mailing list.
If you are interested in contributing fixes, code or documentation to SunPy please checkout our developer guide. SunPy uses a distributed version control system called Git and the code is hosted on GitHub. To get started create an account on github and fork SunPy then submit your changes via a pull request. If you are unfamiliar with Git and GitHub our developer documentation provides a sample workflow but you can also find lots of tutorials online. Please read up on our Coding Standards before getting started. You can browse our issue list or our wishlist to find something that you can tackle. Please send a quick email to the dev list or stop by the chat room to let people know what you are planning on working on (see the Help section).
If you have an idea for a major enhancement or a proposed change to SunPy please consider submitting an SEP. This process is modeled after the Python Enhancement Proposals (PEPs) and provides a structured proposal for the SunPy community to evaluate your idea. You can find the template and more information about SEPs in the appropriate SunPy repository. SEPs are submitted as pull requests into that repository.
We welcome the development of affiliated packages (packages that depend on SunPy). Please take a look at this SEP to read about our policies surrounding affiliated packages. Please send an email to the mailing list to start a dialogue.
| No. | Package Name | Stable | Code Repository | Maintainer |
|---|---|---|---|---|
| 1 | IRISPy | Yes | IRISPy | Daniel Ryan |
| 2 | SolarBExploration | Yes | SolarBExploration | Stuart Mumford |
| 3 | sunkit-sst | Yes | sunkit-sst | Nabil Freij |


Last weekend, one of my friends and I found ourselves waiting near the exit of dressing rooms in a busy London clothes shop. In that shop, anyone going in the dressing rooms was given a small plastic panel with a number corresponding to the number of articles of clothing she wanted to test. Each customer was only allowed to bring 6 articles inside. At the exit, an employee takes the small piece of plastic back, along with all unwanted items which will then be put back on shelves and racks inside the store.
As we were standing right next to that lady, we started to notice that very few customers would actually buy anything. We saw a lot of people coming out of the dressing room with 4,5 or even 6 articles of clothing, and give them all back. Being bored, we thought it would be a good idea to actually record this data, and analyse them.
We took notes of how many items were tried buy each person, and how many were kept. In the approximately 20 minutes we spent there, we saw 75 customers come out of dressing rooms. Here is the histogram of the number of items they had brought in (according to the small plastic panel).
We can already see some interesting trends. There are two peaks in this histogram, suggesting two types of customers : Those who bring few (1 or 2) articles, and those who want to try many items on (5 or 6). In comparison, few customers bring 3 or 4 items. But how does this translate into sales? Let’s have a look at the histogram of items kept.</p>
As you can see, the number of articles finally kept does not really look like the number of articles brought into the dressing room. In fact, the number of articles kept can be closely matched by what we call the gaussian distribution. That is the redline you can see on the figure. It is very frequently encountered in nature. So frequently, in fact, that we also call it the normal distribution.
So we know that some people go in with a lot of clothes, and some with very few. We also know that the number of articles kept obeys the normal distribution. But what we would like to know is how does the number of articles brought in correlate with the number of articles brought out. In other words, we want to know if we can predict how many articles of clothing someone will end up buying based on how many the bring into the dressing room.
We can accomplish this by quite easily. In the following graph, you can see the mean number of articles kept, as a function of number of articles brought in the dressing room :
The numbers fit nicely on a line, with a slope of 0.28 (drawn in red). Roughly speaking, for every 100 items tried on in that shop’s dressing room, only 28 are bought. The remaining 72 have to be put back on shelves by the staff. We can also see that the number of clothe brought in can be a pretty good indicator of how many will be bought. The more items brought in, the more items are kept.
But, while we were recording the data, we actually noticed a pretty large number of people who actually did not seem to want to buy anything. They seemed just to be interested in trying clothes with their friends, and giving them all back afterwards. And we wondered, do they somehow skew our data? Is there something to be learnt by eliminating them from our dataset? Have a look at this next graph.
Here you can see the same data than on the last graph, but we have added another curve. In green you can see the mean number of articles bought if we exclude those who do not buy anything. There is something funny about this graph. It seems that people who bring in 4 items actually end up buying less than people who brought in only 3. I do not know if there is any truth behind this observation, or if it is just due to a small number of samples. other than that, the trend of “The more you bring in, the more you buy” seems to hold.


Start At The End
What is it?
Start At The End is a simple exercise to identify a set of assumptions which must be tested in order achieve a long term goal. It’s a rather simple exercise, which asks participants to describe what success looks like for their work, and use this to define the goal of the effort. We default to 12 months, but it can be longer or shorter. Then, the exercise asks participants to imagine all the ways their project can fail, and using the results to enumerate key questions or assumptions that must be examined at the start of the work. The technique is derived from Chapter 4 of The Sprint Book1. At the end of the exercise, you will have:
- A shared understand of your effort’s goal
- A list of key questions or assumptions that need to be examined early in the work effort in order for the team to achieve the goal.
- A prioritization of the key questions
Why use Start At The End?
- Most planning activities revolve around juggling a “shopping list of features,” as Gojko Adzic calls them in Impact Mapping. Even though the features are delivered, often the business objective is not achieved. Start At The End gives us a light weight approach to put the business value at the center of our work and make sure that assumptions that could lead to failure are clearly identified for examination
- Start At The End is really easy and quick to facilitate, but often yields great results. It’s a great technique for those new to facilitating workshops or sessions that are short on time.
- The technique forces participants to think about failure, which many teams never do. It’s surprising how much you can learn about a domain just by getting participants to share their fears.
- It’s easy to communicate the results of the session in slide ware and project rooms. We’d recommend keeping the results up in the project space throughout your effort.
Related Practices
- Impact Mapping is another practice which leads to the same outputs. Compared to Start At The End, Impact Mapping produces a higher fidelity understanding of the domain, but at the cost of increased complexity for facilitation. Generally speaking, Impact Mapping is the better fit when building products or services, and Start At The End is a better fit when discussing organizational change or other generally nebulous efforts.
Who do you need?
- Facilitator
- Product Owner
- Project Sponsor, both technical and business
- Architects
Suggested Time
1-2 hours. The session can be done without any participant prep.
Difficulty
- Facilitator: Easy
- Participants: Easy
Facilitation Materials Needed
None required, but 3x5 sticky notes with markers can be useful if you want participants to jot their notes down independently and then present back.
How does it fit?
- Typically done in the Why area of the Discovery of the Open Practice Library


[deviation_in_coeff norm_coeff mixed_coeff count] = OptiMixedCoeff(coeff, Q, deviation_value) This function exports the mixed coefficients vector due to acceptable deviation value. The mixed vector holds coefficients rounded to SPT and NPT and normal normalized coefficients. How it works: If the deviation of the coeff. is less than or equal the deviation value, the coeff. will be rounded to the value having the smallest deviation, i.e. coeff. = 53 -> SPT = 64, NPT = 48 64 - 53 = 11, while 53 - 48 = 5 so the coeff. will be rounded to 48 not 64
0001 function [deviation_in_coeff norm_coeff mixed_coeff count] = OptiMixedCoeff(coeff, Q, Nb, deviation_value) 0002 0003 % 0004 % [deviation_in_coeff norm_coeff mixed_coeff count] = OptiMixedCoeff(coeff, Q, deviation_value) 0005 % 0006 % This function exports the mixed coefficients vector due to acceptable 0007 % deviation value. The mixed vector holds coefficients rounded to SPT and 0008 % NPT and normal normalized coefficients. How it works: 0009 % If the deviation of the coeff. is less than or equal the deviation value, 0010 % the coeff. will be rounded to the value having the smallest deviation, 0011 % i.e. 0012 % coeff. = 53 -> SPT = 64, NPT = 48 0013 % 64 - 53 = 11, while 53 - 48 = 5 0014 % so the coeff. will be rounded to 48 not 64 0015 % 0016 0017 count = 0; 0018 0019 norm_coeff = NormalizedCoeff(coeff, Q); 0020 spt_coeff = SPT(norm_coeff); 0021 npt_coeff = NPT(norm_coeff, Nb, Q); 0022 0023 dev_spt = abs(norm_coeff - spt_coeff); 0024 dev_npt = abs(norm_coeff - npt_coeff); 0025 0026 for i = 1 : length(coeff), 0027 tmp_dev = [dev_spt(i) dev_npt(i)]; 0028 tmp_coeff = [spt_coeff(i) npt_coeff(i)]; 0029 if (tmp_dev(1) <= deviation_value & tmp_dev(2) <= deviation_value), 0030 if min(tmp_dev), 0031 mixed_coeff(i) = tmp_coeff(IndexOfMinValue(tmp_coeff)); 0032 deviation_in_coeff(i) = min(tmp_dev); 0033 count = count + 1; 0034 else 0035 mixed_coeff(i) = norm_coeff(i); 0036 deviation_in_coeff(i) = 0; 0037 end 0038 else 0039 mixed_coeff(i) = norm_coeff(i); 0040 deviation_in_coeff(i) = 0; 0041 end 0042 end