Remote Endpoints¶
Overview¶
The remote
command group allows users to manage the service endpoints
Singularity will interact with for many common command flows. This includes
managing credentials for image storage services, remote builders, and key
servers used to locate public keys for SIF image verification. Currently,
there are three main types of remote endpoints managed by this command group:
the public Sylabs Cloud (or local Singularity Enterprise installation), OCI
registries and keyservers.
Public Sylabs Cloud¶
Sylabs introduced the online Sylabs Cloud to enable users to Create, Secure, and Share their container images with others.
A fresh, default installation of Singularity is configured to connect
to the public cloud.sylabs.io
services. If you only want to use the public services you just need to
obtain an authentication token, and then singularity remote login
:
Go to: https://cloud.sylabs.io/
Click “Sign In” and follow the sign in steps.
Click on your login id (same and updated button as the Sign in one).
Select “Access Tokens” from the drop down menu.
Enter a name for your new access token, such as “test token”
Click the “Create a New Access Token” button.
Click “Copy token to Clipboard” from the “New API Token” page.
Run
singularity remote login
and paste the access token at the prompt.
Once your token is stored, you can check that you are able to connect
to the services with the status
subcommand:
$ singularity remote status
INFO: Checking status of default remote.
SERVICE STATUS VERSION URI
Builder OK v1.1.14-0-gc7a68c1 https://build.sylabs.io
Consent OK v1.0.2-0-g2a24b4a https://auth.sylabs.io/consent
Keyserver OK v1.13.0-0-g13c778b https://keys.sylabs.io
Library OK v1.0.16-0-gb7eeae4 https://library.sylabs.io
Token OK v1.0.2-0-g2a24b4a https://auth.sylabs.io/token
INFO: Access Token Verified!
Valid authentication token set (logged in).
If you see any errors you may need to check if your system requires
proxy environment variables to be set, or if a firewall is blocking
access to *.sylabs.io
. Talk to your system administrator.
You can interact with the public Sylabs Cloud using various Singularity commands:
pull, push, build –remote, key, search, verify, exec, shell, run, instance
Note
Using docker://
, oras://
and shub://
URIs with these commands
does not interact with the Sylabs Cloud.
Managing Remote Endpoints¶
Users can setup and switch between multiple remote endpoints, which
are stored in their ~/.singularity/remote.yaml
file. Alternatively, remote endpoints can be set system-wide by an
administrator.
Generally, users and administrators should manage remote endpoints
using the singularity remote
command, and avoid editing
remote.yaml
configuration files directly.
Note
The following commands in this section configures Singularity to use and authenticate to the public Sylabs Cloud, a private installation of Singularity Enterprise, or community-developed services that are API compatible.
List and Login to Remotes¶
To list
existing remote endpoints, run this:
$ singularity remote list
Cloud Services Endpoints
========================
NAME URI ACTIVE GLOBAL EXCLUSIVE
SylabsCloud cloud.sylabs.io YES YES NO
Keyservers
==========
URI GLOBAL INSECURE ORDER
https://keys.sylabs.io YES NO 1*
The YES
in the ACTIVE
column for SylabsCloud
shows that this is the
current default remote endpoint.
To login
to a remote, for the first time or if your token expires
or was revoked:
# Login to the default remote endpoint
$ singularity remote login
# Login to another remote endpoint
$ singularity remote login <remote_name>
# example...
$ singularity remote login SylabsCloud
singularity remote login SylabsCloud
INFO: Authenticating with remote: SylabsCloud
Generate an API Key at https://cloud.sylabs.io/auth/tokens, and paste here:
API Key:
INFO: API Key Verified!
If you login
to a remote that you already have a valid token for,
you will be prompted, and the new token will be verified, before it
replaces your existing credential. If you enter an incorrect token
your existing token will not be replaced:
$ singularity remote login
An access token is already set for this remote. Replace it? [N/y]y
Generate an access token at https://cloud.sylabs.io/auth/tokens, and paste it here.
Token entered will be hidden for security.
Access Token:
FATAL: while verifying token: error response from server: Invalid Credentials
# Previous token is still in place
Note
It is important for users to be aware that the login command will store the supplied credentials or tokens unencrypted in your home directory.
Add & Remove Remotes¶
To add
a remote endpoint (for the current user only):
$ singularity remote add <remote_name> <remote_uri>
For example, if you have an installation of Singularity enterprise hosted at enterprise.example.com:
$ singularity remote add myremote https://enterprise.example.com
INFO: Remote "myremote" added.
INFO: Authenticating with remote: myremote
Generate an API Key at https://enterprise.example.com/auth/tokens, and paste here:
API Key:
You will be prompted to setup an API key as the remote is added. The web address needed to do this will always be given.
To add
a global remote endpoint (available to all users on the
system) an administrative user should run:
$ sudo singularity remote add --global <remote_name> <remote_uri>
# example..
$ sudo singularity remote add --global company-remote https://enterprise.example.com
[sudo] password for dave:
INFO: Remote "company-remote" added.
INFO: Global option detected. Will not automatically log into remote.
Note
Global remote configurations can only be modified by the
root user and are stored in the etc/singularity/remote.yaml
file, at the Singularity installation location.
Conversely, to remove
an endpoint:
$ singularity remote remove <remote_name>
Use the --global
option as the root user to remove a global
endpoint:
$ sudo singularity remote remove --global <remote_name>
Set the Default Remote¶
A remote endpoint can be set as the default to use with commands such
as push
, pull
etc. via remote use
:
$ singularity remote use <remote_name>
The default remote shows up with a YES
under the ACTIVE
column in the output of remote list
:
$ singularity remote list
Cloud Services Endpoints
========================
NAME URI ACTIVE GLOBAL EXCLUSIVE
SylabsCloud cloud.sylabs.io YES YES NO
company-remote enterprise.example.com NO YES NO
myremote enterprise.example.com NO NO NO
Keyservers
==========
URI GLOBAL INSECURE ORDER
https://keys.sylabs.io YES NO 1*
* Active cloud services keyserver
$ singularity remote use myremote
INFO: Remote "myremote" now in use.
$ singularity remote list
Cloud Services Endpoints
========================
NAME URI ACTIVE GLOBAL EXCLUSIVE
SylabsCloud cloud.sylabs.io NO YES NO
company-remote enterprise.example.com NO YES NO
myremote enterprise.example.com YES NO NO
Keyservers
==========
URI GLOBAL INSECURE ORDER
https://keys.example.com YES NO 1*
* Active cloud services keyserver
Singularity 3.7 introduces the ability for an administrator to make a remote
the only usable remote for the system by using the --exclusive
flag:
$ sudo singularity remote use --exclusive company-remote
INFO: Remote "company-remote" now in use.
$ singularity remote list
Cloud Services Endpoints
========================
NAME URI ACTIVE GLOBAL EXCLUSIVE
SylabsCloud cloud.sylabs.io NO YES NO
company-remote enterprise.example.com YES YES YES
myremote enterprise.example.com NO NO NO
Keyservers
==========
URI GLOBAL INSECURE ORDER
https://keys.example.com YES NO 1*
* Active cloud services keyserver
This, in turn, prevents users from changing the remote they use:
$ singularity remote use myremote
FATAL: could not use myremote: remote company-remote has been set exclusive by the system administrator
If you do not want to switch remote with remote use
you can:
Make
push
andpull
use an alternative library server with the--library
option.Make
build --remote
use an alternative remote builder with the--builder
option.Make
keys
use an alternative keyserver with the-url
option.
Keyserver Configurations¶
By default, Singularity will use the keyserver correlated to the active cloud
service endpoint. This behavior can be changed or supplemented via the
add-keyserver
and remove-keyserver
commands. These commands allow an
administrator to create a global list of key servers used to verify container
signatures by default, where order 1
is the first in the list. Other
operations performed by Singularity that reach out to a keyserver will only
use the first entry, or order 1
, keyserver.
When we list our default remotes, we can see that the default keyserver is
https://keys.sylabs.io
and the asterisk next to its order indicates that
it is the keyserver associated to the current remote endpoint. We can also see
the INSECURE
column indicating that Singularity will use TLS when
communicating with the keyserver.
$ singularity remote list
Cloud Services Endpoints
========================
NAME URI ACTIVE GLOBAL EXCLUSIVE
SylabsCloud cloud.sylabs.io YES YES NO
Keyservers
==========
URI GLOBAL INSECURE ORDER
https://keys.sylabs.io YES NO 1*
* Active cloud services keyserver
We can add a key server to list of keyservers with:
$ sudo singularity remote add-keyserver https://pgp.example.com
$ singularity remote list
Cloud Services Endpoints
========================
NAME URI ACTIVE GLOBAL EXCLUSIVE
SylabsCloud cloud.sylabs.io YES YES NO
Keyservers
==========
URI GLOBAL INSECURE ORDER
https://keys.sylabs.io YES NO 1*
https://pgp.example.com YES NO 2
* Active cloud services keyserver
Here we can see that the https://pgp.example.com
keyserver was appended to
our list. If we would like to specify the order in the list that this key is
placed, we can use the --order
flag:
$ sudo singularity remote add-keyserver --order 1 https://pgp.example.com
$ singularity remote list
Cloud Services Endpoints
========================
NAME URI ACTIVE GLOBAL EXCLUSIVE
SylabsCloud cloud.sylabs.io YES YES NO
Keyservers
==========
URI GLOBAL INSECURE ORDER
https://pgp.example.com YES NO 1
https://keys.sylabs.io YES NO 2*
* Active cloud services keyserver
Since we specified --order 1
, the https://pgp.example.com
keyserver was
placed as the first entry in the list and the default keyserver was moved to
second in the list. With the keyserver configuration above, all image default
image verification performed by Singularity will first reach out to
https://pgp.example.com
and then to https://keys.sylabs.io
when
searching for public keys.
If a keyserver requires authentication before usage, users can login before using it:
$ singularity remote login --username ian https://pgp.example.com
Password (or token when username is empty):
INFO: Token stored in /home/ian/.singularity/remote.yaml
Now we can see that https://pgp.example.com
is logged in:
$ singularity remote list
Cloud Services Endpoints
========================
NAME URI ACTIVE GLOBAL EXCLUSIVE
SylabsCloud cloud.sylabs.io YES YES NO
Keyservers
==========
URI GLOBAL INSECURE ORDER
https://pgp.example.com YES NO 1
https://keys.sylabs.io YES NO 2*
* Active cloud services keyserver
Authenticated Logins
=================================
URI INSECURE
https://pgp.example.com NO
Note
It is important for users to be aware that the login command will store the supplied credentials or tokens unencrypted in your home directory.
Managing OCI Registries¶
It is common for users of Singularity to use OCI registries as sources for their container images. Some registries require credentials to access certain images or the registry itself. Previously, the only methods in Singularity to supply credentials to registries were to supply credentials for each command or set environment variables for a single registry. See Authentication via Interactive Login and Authentication via Environment Variables
Singularity 3.7 introduces the ability for users to supply credentials on a per
registry basis with the remote
command group.
Users can login to an oci registry with the remote login
command by
specifying a docker://
prefix to the registry hostname:
$ singularity remote login --username ian docker://docker.io
Password (or token when username is empty):
INFO: Token stored in /home/ian/.singularity/remote.yaml
$ singularity remote list
Cloud Services Endpoints
========================
NAME URI ACTIVE GLOBAL EXCLUSIVE
SylabsCloud cloud.sylabs.io YES YES NO
Keyservers
==========
URI GLOBAL INSECURE ORDER
https://keys.sylabs.io YES NO 1*
* Active cloud services keyserver
Authenticated Logins
=================================
URI INSECURE
docker://docker.io NO
Now we can see that docker://docker.io
shows up under
Authenticated Logins
and Singularity will automatically supply the
configured credentials when interacting with DockerHub. We can also see
the INSECURE
column indicating that Singularity will use TLS when
communicating with the registry.
We can login to multiple OCI registries at the same time:
$ singularity remote login --username ian docker://registry.example.com
Password (or token when username is empty):
INFO: Token stored in /home/ian/.singularity/remote.yaml
$ singularity remote list
Cloud Services Endpoints
========================
NAME URI ACTIVE GLOBAL EXCLUSIVE
SylabsCloud cloud.sylabs.io YES YES NO
Keyservers
==========
URI GLOBAL INSECURE ORDER
https://keys.sylabs.io YES NO 1*
* Active cloud services keyserver
Authenticated Logins
=================================
URI INSECURE
docker://docker.io NO
docker://registry.example.com NO
Singularity will supply the correct credentials for the registry based off of
the hostname when using the following commands with a docker://
or
oras://
URI:
pull, push, build, exec, shell, run, instance
Note
It is important for users to be aware that the login command will store the supplied credentials or tokens unencrypted in your home directory.


Blogging is one another hobby of mine.Connect to my Blogspot!
Not a pro still learning
Really a Beginner
Find out some of my works Github workspace
I do take photos as well
How does Google's Humming Bird Search Algorithm work ? Might be the way I tried
Since everyone does it I am also into the trap.
No GUI where there is a need for the implementation in C++ Btb I did it for Data Structures Lab
The Chennai Floods and Cyclones struck me so hard so that I moved onto creating a Map Webframe that could server helping the people.It is slightly an exploitation of Google maps API


| "https://images.unsplash.com/photo-1569098446737-c22abf95abf7?ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&ixlib=rb-1.2.1&auto=format&fit=crop&w=1482&q=80" | The owner is Brendon Takarinda I am a blogger and a web developer. I love music and fashion.I will be in the music industry |
The owner is Brendon Takarinda


TIME
2020/10/20
2020/11/14-11/15
2020/11/20-11/22
VENUE
Shanbei Gymnasium (BLD.8), Xiangshan Campus, China Academy of Art, Hangzhou;
Room 405,408, BLD.4, Nanshan Campus, China Academy of Art;
Nanyuan Auditorium, 1F Building 4, Nanshan Campus, China Academy of Art.
CONVENER
Prof. HUANG Sunquan & LIU Yisi
HOST
Institute of Network Society, CCVS, SIMA, China Academy of Art
CO-ORGANIZER
Dimension Inc. dWeb Shanghai
COMMITTEE
Prof. GAO Shiming，Prof. GUAN Huaibin, Prof. HUANG Sunquan
ENROLL
For ‘The Web of Phronesis’
For Panel 1, also the Panel 6 at the Inter-World-View
While capitalism seems to be generally frustrated with the global spread of the Covid-19, surveillance capitalism has secretly risen with the virus explosion. In the name of security, a centralized digital regime with unchallengeable legitimacy power is now strictly monitoring people, which seems to bring about an information centralized online society.
Aristotle indicates that the ‘phronesis’ (wisdom of praxis) produced by praxis can lead people to ‘aletheuein’ and to achieve the ‘solicitous circumspection’. It means that there should be phronesis to manage the use of techne, so that the techne with major flaws in revealing the alethea (truth) but outside itself could be made the best use of. According to this, we assume that to take the thought (with action) as the starting point of the techne production would construct the future of humans under the Covid-19 state of emergency. We believe that the starting point would be able to trigger discussions that go beyond proliferation and concentration of global technology, therefore forks our futures.
In the previous Annual Conferences “Forces of Reticulation” (2016), “Another Walk with Lefebvre: Critique of Urbanism and Everyday Life in the Algorithmic Age” (2017), “Intelligent Urban Fabric” (2018) and “Neitizen21: Beyond Personal Account” (2019), INS connected worldwide scholars and activists to construct criticism toward current network society and explore a network of practices together. The fifth annual conference, the Web of Phronesis, tries to explore technical practices hindered by social dilemma, and to review the life of infinite branches created by love between we and the world along with imaginations.
PANEL I: RADICAL MARKET
2020/10/20 Tuesday 18:30-21:30
VENUE
Shanbei Gymnasium (BLD.8), Xiangshan Campus, China Academy of Art, Hangzhou
MODERATOR
Huang SunQuan & Liu Yihong
COMMENTARY
Liu yisi & Yan Suji
The global crises of declining productivity, economic stagnation, conflict and inequality are caused by the inherent predatory accumulation, monopoly, and inequitable distribution of capitalism, as well as the poor democratic mechanism, restrictions on labors and immigrants’ mobility. The “radical market” is a thought experiment seeks “radical change”, in order to plot the bottom-up economic, social, and informational technology plans of future, taking into account the allocation efficiency of the free market and investment incentives at the same time, and turning crisis into opportunities for the common good. What are the critical issues of the current crisis? How to strike a balance between a free economy and state economic intervention in long-term risk management? How does modern society deal with the associated risks that are positively correlated with its system complexity?
KEYNOTE SPEAKERS
Vitalik Buterin（Founder of Ethereum）
Glen Weyl（Author of ‘Radical Markets’，Principal Researcher at Microsoft Research New England）
This session is also the ‘Panel 6: Crisis-Change’of the ‘Inter-World-View Hangzhou China 2020’
WEST LAKE MESH NETWORK WORKSHOP
2020/11/14─15 Saturday、Sunday 10：00-17：00
CONVENOR
DWeb Shanghai
VENUE
Room 405,408, BLD.4, Nanshan Campus, China Academy of Art
REFERENCES
https://github.com/DWebShanghai/draft/blob/master/proj_west_lake.org
How could we keep in touch without commercial internets? With technical supports of dWeb Shanghai, attandees of the workshop will try to set up wireless internet in groups at Building 4, Nanshan Campus, China Academy of Art. In this two-day workshop, participants are going to install the equipments on Day1, then design and set up the internet on Day 2. Like an utopia, this wireless network should be able to get accessed by all people in Nanshan campus until the end of the fifth annual conference. It will be the first experiment of setting up Mesh Network around the West Lake, Hangzhou, in China.
The mesh network will be come a platform recording the weather of participants’ locations such as the temperature, the humidity and qualities of the air, with no need to pay for the internet provided by ISP.
The DIY of mesh network all around the world is updating knowledge about the ‘communities’, the ‘sharing’ and the ‘nearby’.
Freifunk in Germany covers over 400 wireless communities; B4RN in Lancasire is now serving UK government with optical fiber. As a community network, the coverage of guifi.net in Europe is probably the widest. Tens of thousands of Spanish people including volunteers, non-profit organizations, public management organizations and media companies are served by this network.
A non-profit organization in the rural area at Mount Olympus in Greece, Sarantaporo, have been established for saving the community from financial crisis in 2010 by local youths since 2013. The wireless mesh network they DIY’ed is now serving over 5000 residents in more than 10 villages. People exchanges views and goods on the applications, which help the organization won the European BroadbandAwards in 2019. FFDN in France is a combination of community networks.
All the mesh networks are distinguished from the others. The networks are covering not only urban areas, but also rural areas, slums and immigrants communities. The motivations to set up the mesh networks are different, whereas all the networks could be defined as decentralized webs. The webs belong to users rather than commercial internet service providers, they liberates individuals from digital oligarchy and empower the autonomy of users.
YOUTH ROUND TABLE
2020/11/20 Friday 10：30-17：00
VENUE
Nanyuan Auditorium, 1F Building 4, Nanshan Campus, China Academy of Art
The Web of Phronesis focuses on not only praxis, but also theories. To architect a better future, we try to review the intelligence and experiences gained by technology using. In addition to a workshop and four panels, we called for papers from young scholars and people with faiths all around the world, holding the Youth Round Table. In this session, there will be ten presentations from the scholars carefully evaluated and selected by the committee. Selectees are going to present on the spot, discussing with each others and expand their practice and thinking from the philosophical, artistic, socialized and technical levels.
PANEL II: INTERNET ARCHIVE
2020/11/20 Friday 18：30-21：30
VENUE
Nanyuan Auditorium, 1F Building 4, Nanshan Campus, China Academy of Art
MODERATOR
Katt Gu
The truths history narrates could always be footnotes to the facts. They are quarantined by the reality of the time. Since the establishment of the Internet Archive in 1996, information generated by the Internet has far exceeded the total amount of human history. Today, digitized files and metadata records have become the main issue of art museums and galleries. Under this circumstance, to preserve, to record and to file human civilization are critical challenges to technicalize and to externalize human history.
The ‘archive’ is accumulation of documents cataloged by authorities. Under this circumstance, it is the method to catalog, operated by archivists, made up the narrative of human history.
The automatic digital archives are now saving the hidden and the deleted to photoelectric media. Under this circumstance, human civilization resists the absence, which means ideologies and bias on cultures can do nothing to question recorded truths, and people have no power are not allowed to correct or even to simply write a history. The shared root of modernology and archaeology found that there is no authorities presently but only searching algorithm and the metadata in our age. It seems like we are living in a time without archivist.
KEYNOTE SPEAKERS
Zhou Pengan（Founder of the first interactive online computer museum in China）
Elaine Lin (Head of Collections at Asia Art Archive)
Sam Williams (CEO & Co-Founder at Arweave)
PANEL III: DWEB
2020/11/21 Saturday 18：30-21：30
VENUE
Nanyuan Auditorium, 1F Building 4, Nanshan Campus, China Academy of Art
MODERATOR
Huang SunQuan
Should Internets owned by oligarchs? How to create Internet in which inclusive love could stand, publicness would be compatible with privacy, and everyone would be welcomed? When decentralizing, people are exploring the relationship between networks and human society rather than simply updating technologies. Decentralizing is not for the sake of decentralization, but for benefiting human societies, to decreased harms and inconveniences that digital orligarchy have made. Decentralized internets should be open to all the people, because that is what Internet was originally made for. In this panel, practitioners of Decentralized Web will be invited to share their experiences on architecting in the atlas of the future has already taken place.
KEYNOTE SPEAKERS
Dominic Tarr（Javascript Cyberhobo and computer wizard，Co-founder at scuttlebutt）
Dylan Reibling（Film director and interactive artist）
Danny O‘Brien（the International Director of the Electronic Frontier Foundation）
Liu Yisi（Convenor of dWeb Shanghai，CTO at Dimension.im）
Mark Nadal（mathematician turned programmer, runs a VC backed Open Source company ERA）
PANEL IV: CRYPTO-IST
2020/11/22 Sunday 18：30-22：00
VENUE
Nanyuan Auditorium, 1F Building 4, Nanshan Campus, China Academy of Art
MODERATOR
Huang SunQuan & Liu Yisi
Although to ensure one’s privacy and security on the internet are basic human rights, the monopoly and the algorithmic improvement of social media are now preventing the netizens from seizing back their lost human rights. To retake online privacy and security needs revolutions on both economy and society.
Gladly we found that there are plenty of technologies prepared for this revolution have progressed substantially in the past decade, including public key encryption, end-to-end encryption, zero-knowledge proof, and plenty of software protocols that are used in interactions, authentications and verification. We are aiming to achieve the ultimate possibilities within the existing social system.
KEYNOTE SPEAKERS
Kin Ko（Founder of Likecoin）
Yan Suji（Founder of Dimension.im ）
Frank Braun & Smuggler（Crypto-anarchists，Founders of Cypherpunk Bitstream）
Harry Halpin（CEO of Nym Technologies, visiting Professor at KU Leuven in Belgium）
Pavol Luptak（cryptoanarchist and voluntaryist focused on technology and society hacking, the founder of Nethemba, the co-founder of Hacktrophy)
Github https://github.com/DimensionDev
Twitter https://twitter.com/realMaskNetwork
Facebook https://www.facebook.com/masknetwork
Discord https://discord.com/invite/cKGW45g
Telegram (English) https://t.me/maskbook_group
Telegram （中文） https://t.me/masknetwork_cn


Datasets
In this tutorial will be using the following datasets:
311 Service Requests - Water Leaks. Download from NYC OpenData. See the downloading and preping section below for more information on how to filter and download this dataset.
American Community Survey - Table S0101 (AGE AND SEX). Download from the U.S. Census Bureau FactFinder site. See Tutorial 02 - Census Data & Joins for more information on how to download, prep, and join census data to polygons shapefiles.
Census Tracts - New York State 2017 census tracts. Download from U.S. Census Bureau - Tiger/Line Shapefiles. Select
2017
andCensus Tracts
, and clickSubmit
. Then, selectNew York
as the state and clickDownload
.Boroughs - New York City boroughs. Download from NYC Planning - Open Data. Choose "Borough Boundaries (Clipped to Shoreline)", under "Borough Boundaries & Community Districts".
Hydrography - New York City hydrography. Download from NYC Open Data. Once you get to the NYC OpenData page, click
Export
and choose theShapefile
format.United States Hydrographic Polygons. Download from the Columbia University Libraries Geodata portal. Download the
Original Shapefile
.United States State Boundaries. Download from the U.S. Department of Transportation. Download the
Shapefile
.
A packaged file with the census and block group data can be found here.
About 311 Data
The main dataset we will be using in this tutorial is based on 311 data. 311 is a service provided by the city of New York where people can call in (dialing 311) and submit complaints or ask questions about living in the city (the service also accepts complaints online). Some of the most popular complaints filed through 311 are about parking, noise, garbage, rodents, dead or damaged trees, air quality, construction permits, graffiti, homelessness, street conditions, taxis and water quality. There are other categories and each one of these has its own subcategories. Furthermore, each entry in the dataset comes with the following fields (amongst others):
Created date
Closed date
Agency
Complaint type
Descriptor
Location type
Incident zip code
Incident address
Street name
City
Landmark
Facility type
Status
Due date
Resolution description
Community board
X and Y coordinates
Latitude and longitude
The dataset is very interesting and a great resource for anyone studying New York. Nevertheless, a word of caution is necessary: many people use this dataset to describe and analyze conditions in New York; however, the 311 data does not describe the city, it describes the complaints people file, it is not about the city, it is about the complaints, and even though the complaints might tell us something about the city, the distinction is crucial. Every dataset has its own biases and the 311 dataset has very strong ones: it collects data ONLY about the people who complain and ONLY about what they choose to complain about; it doesn't say anything about the people who do not complain or the things people don't complain about. There is no 1 to 1 relationship between the 311 complaints and the conditions in the ground. That being said, though, it is still a great resource. You can find out more about the 311 service here.
Filtering, Downloading, and Preping 311 Data
The first step in this tutorial is to select, filter and download the 311 data. The NYC OpenData portal is a great resource for data related to New York City and it provides an easy way of accessing 311 data. In it's search bar type
311
and it should take you to a list of datasets related to 311 data. The one we are looking for is called311 Service Requests from 2010 to Present
. Once you click there, selectView Data
.Once you've accessed the dataset you will see something like this:
Here, we need to filter the database to download only the complaints related to water leaks in the Bronx for the last three months of 2018. To filter the data do the following:
On the right-hand panel, where it says "Filter", create a small query using the drop-down menus. Where it says
Unique Key
, change it toComplaint Type
. Keep theis
and then type inWater Leak
in the space below.Next, click on
Add a New Filter Condition
and create another query that readsCreated Date
is between
10/01/2018 12:00:00 AM
and01/01/2019 12:00:00 AM
.Finally, add another filter condition that reads
City
is
Bronx
.You should see something like this:
Finally, click on the
Export
button at the top right-hand corner of the site and choose theCSV
(comma separated values) format. Your file should start downloading then.Now open your .csv file in Excel. You should have around 3,320 records. If you explore this file you will notice that it contains both X and Y data (X Coordinate, Y Coordinate), as well as Latitude and Longitude. This means that if we wanted we could just import this table, as is, into QGIS and get points from just those fields, without going through the geocoding process. However, since the purpose of this tutorial is to show you how to geocode based on address data, we will ignore those fields and use instead the address data also present in this file. For more information on how to add data with geographic coordinates to QGIS see this tutorial.
In order to geocode we need to have the following fields: address, city, state, and country. Sometimes having the zipcode also helps. Looking at this data you'll notice we already have the address (
Incident Address
) and the city (City
). We are only missing state and country, which is easy to create since we know that all the locations on this file are in the Bronx. Go ahead and create two new columns calledState
andCountry
and fill them withNew York
(for the state) andUnited States
(for the country).Finally, save your file as
Windows Comma Separated (.csv)
.Before moving forward a note of caution: the actual geocoding operation in QGIS takes some time. In our recent tests, it took us almost one hour to geocode a file this size. If you just want to learn how to geocode we strongly encourage you to select only 100 or 200 rows and save them as a separate file to use for geocoding. In the data package for this tutorial we are inlcuding the point shapefile which we produced by geocoding the original large file.
Geocoding
The actual geocoding process is very straight forward. Just take into account that since the geocoding plugin uses an external service (GoogleMaps or OpenStreetMap API) the more points you have, the more time it will take; similarly there are limits as to how many points you can submit for geocoding. If you need to geocode more than 2,000 points it is probably worth it doing it through a Python script that directly queries one of those APIs.
First, you will need to install the
MMQGIS
plugin, which includes a geocoding service.To install this plugin go to
Plugins
Manage and Install Plugins...
and search forMMQGIS
. Once you find it, clickInstall plugin
.Once the plugin is installed, open a new map and you should see a new top-level menu called
MMQGIS
.Next add the
Boroughs
shapefile. This is not necessary but it will help us verify the results of the geocoding.Finally, open the geocoding window by clicking on
MMQGIS
and selectingGeocode
andGeocode CSV with Google/OpenStreetMap
.Select the file we prepared. Remember, if you are just doing this to learn, choose a file with no more than 200 addresses; anything larger than that might take a long time. Set the address field to
Incident Address
, the city filed asCity
, the state field asState
and the country field asCountry
.Set the web service to
OpenStreetMap / Nominatim
. And select the output file location and the location for the csv file with the addresses that were not located.Once you are ready click
OK
. It will take a while because the plugin takes into account the limits of the API and delays the request accordingly. For 200 or so requests it should take between 5 and 10 minutes.Once the geocoding process finishes you should have a new layer in your map with your geocoded points. This layer will still have all the data present in the base file. In addition, you should also have on your computer a file with the records that were the service was unable to geocode.
Additional Geocoding Options
Often you might need to geocode more than two or three hundred records. In these cases you should probably use your own Google Maps Geocoding API key. However, remember that even if you use your own key, Google still has a limit of 50 requests per minute and a total of 2,500 requests per day.
To obtain a key you should use a Google account to access the Google Developer Console. Once you sign in, you should create a
New Project
and for that project go toEnable APIs and Services
.Once there search for "geocoding" and once you find it, enable the
Geocoding API
for this project. Note that Google has changed its fee structure and now charges for geocoding. At the moment of writing this tutorial Google offers $200 montly credit for free, but once you pass that limit you will be charged for geocoding. More information on their pricing can be found here.Once you've enabled the API, click on the main menu of the Google APIs and navigate to
Credentials
on your new project. There, click on theCreate Credentials
button and create your API key. This is the key that you will be able to use in the MMQGIS geocoder with theGoogle Maps Web Service
.
Joining Point Data
Once you have your point data in QGIS, it is a pretty straight forward process to join it to a polygon shapefile. For this tutorial, we will be joining the water leaks data to census tracts polygons with population data. This will allow us to create a map of number of water leaks complaints per person in the Bronx.
First, in addition to your points and the boroughs, add the census data shapefile from the last tutorial.
Next, select
Vector
/Analysis Tools
/Count Points in Polygon...
. Here choose the census data as thePolygons
and your points shapefile as thePoints
.Leave the
Weight
andClass
fields blank and change theCount field name
toCount
.Click
Run
and when it's done, clickClose
. You should have a new layer, and if you open its attribute table you should see theCount
field at the end. Note that this is just a temporary layer. If you close QGIS this layer will dissapear. If you want to preserve it, you should export it as a new shapefile.Now that you have a count of the points in each polygon, you can go ahead an symbolize the layer based on the count, divided by the number of people living in each census tract. For this map it might be better to use complaints per 1,000 residents as the chosen metric.
Finally, add some of the other layers we have been using and compose your final map.


---
type: typography
title: Colors
usage: Use this suite of color utilities for text colors.
stylesheet: '/scss/common/palette.scss'
---
.text-dark-blue
.text-medium-blue
.text-light-blue
.text-dark-grey
.text-darker-grey
.text-medium-grey
.text-light-gray
.text-white
.text-pure-white
.text-orange


|
Table 1
|
Chemical composition of four kinds of purified montmorilloni ... |
|
Table 2
|
Calculation results of layer charge density of four kinds of ... |
|
Table 3
|
Atomic coordinates of montmorillonite. |
|
Table 4
|
The hydrogen bond number and adsorption energy of montmorill ... |


SVG to PNG online converter
File size: 45.98KiB, type: svg
Converted PNG image:
Upload and convert SVG to PNG. Rasterize vector images.
You can set your desired dimensions for the output image in pixels, or let the tool determine it
automatically.
SVG is vector image format, and it can be scaled to any size without losing quality.
This tool currently does not support animated PNG output.


PicturePuzzle Class Reference
[Example scripts (models)]
Inherits Example.
Detailed Description
Example: Picture puzzlesThis example solves picture-puzzles. A picture-puzzle is composed of a matrix of markers. For each row/column there is a specification on how many groups of markers (separated by one or more unmarked spots) and their length. The objective is to find a valid assignment, which incidentally may also produce a pretty picture.
Picture puzzles are also called Nonograms.
Problem 12 at http://www.csplib.org/.
Definition at line 44 of file picture-puzzle.cc.
Picture specifications | |
| A specification is given by a list of integers. The first two integers (w and h) specify the number of columns and rows respectively. Then w + h groups of integers follows. Each group is started by the number of integers it contains (n), followed by n integers specifying the sizes of the stretches of markers in that row/column. | |
| const int * | specs [] |
| List of specifications. | |
| const unsigned | n_examples |
| Number of specifications. | |
Public Member Functions | |
| PicturePuzzle (const Options &o) | |
| Construction of the model. | |
| PicturePuzzle (bool share, PicturePuzzle &s) | |
| Constructor for cloning s. | |
| virtual Space * | copy (bool share) |
| Copy space during cloning. | |
| virtual void | print (void) |
| Print solution. |
Constructor & Destructor Documentation
|
|
Construction of the model.
Definition at line 76 of file picture-puzzle.cc. |
|
|
Constructor for cloning s.
Definition at line 103 of file picture-puzzle.cc. |
Member Function Documentation
|
|
Copy space during cloning.
Implements Gecode::Space. Definition at line 110 of file picture-puzzle.cc. |
|
|
Print solution.
Reimplemented from Example. Definition at line 116 of file picture-puzzle.cc. |
Friends And Related Function Documentation
|
|
List of specifications.
Definition at line 475 of file picture-puzzle.cc. |
|
|
Number of specifications.
Definition at line 478 of file picture-puzzle.cc. |
The documentation for this class was generated from the following file:
- examples/picture-puzzle.cc (Revision: 3517)