Yes. A little late to the pi cluster game. But most of the time you have to actually do something for yourself.
Also, in starting up this project, I didn't see any full-fleshed guides on what you need to get this going, from hardware to walkthroughs. So I'll try and document my steps and missteps throughout this all.
The Hadoop framework was successfully installed and configured on a computing cluster of four Raspberry Pi Zeros as datanodes and one Raspberry Pi 2 (Model B v1.1). Spark was successfully installed and run on the Pi 2, but could not be configured to run on the pi Zeros due to their ARMv6 architecture. Basic test case executions were run on both frameworks to test the runtime and power usage of the cluster. Running times of simple "word count" jobs was surprisngly comparable to a professional Hadoop/Spark provider (Altiscale), on the same order of minutes per test type. The Yarn resource manager was installed and configured to monitor node and job activity. While capable of running Hadoop, the extensive difficulty in configuration outweighed the benefits of running on cheap physical devices. Frankly, I would highly recommend implementing a cluster from scratch, but have to stress to use cheap virtual instances such as those provided commercially such as through Digital Ocean, AWS, or similar academically available resources instead of the Pi.
That's it! The finished cluster. Photography isn't my strong point.
Given the pi’s 1GHz single-core CPU, 512MB of RAM, and the 16GB SD storage which must be split between the OS and actual HDFS data, choosing an appropriate OS was prudent. For example, the base Raspbian Stretch (Debian-based) OS image is around 4GB, with the ‘headless’ version being around 1.2GB [1]. Further, some OSs run from RAM (e.g. Alpine Extended [2]). Finally, the BCM2835 chip on the pi is ARMv6, eliminating all of the Ubuntu ARM (ARMv7) options. Thus, an ideal situation would be low RAM usage during operation and a small disk footprint. The Debian-based DietPi v6.6 (2018-04-01) was chosen for this project due to its low RAM and SD card footprints, as well as its automatable installation option [3].
Each microSD card was manually loaded with the DietPi OS using an Ubuntu system. When it was determined that a Raspberry Pi 2 would be required as the namenode, it was set up similarly, except using the DietPi_ARMv7_Stretch image.
b@bk53e $: lsblk # my card-reader/writer is on /dev/sdf
b@bk53e $: sudo umount /dev/sdf
b@bk53e $: dd if=~/path/to~/DietPi_v6.6_RPi-ARMv6-Stretch.img of=/dev/sdf
OpenSSH, Vim, and Docker do not come ready on the DietPi, but can be enabled, using the DietPi automation file:
b@bk53e $: vim /path/to/sdcard~/boot/dietpi.txt
> AUTO_SETUP_AUTOMATED=1
> CONFIG_BOOT_WAIT_FOR_NETWORK=2
> AUTO_SETUP_GLOBAL_PASSWORD=MapReduce
> AUTO_SETUP_INSTALL_SOFTWARE_ID=0 # OpenSSH
> AUTO_SETUP_INSTALL_SOFTWARE_ID=20 # Vim
> AUTO_SETUP_INSTALL_SOFTWARE_ID=162 # DOCKER
> CONFIG_CHECK_DIETPI_UPDATES=0 # disable updates booting
Installation of each image was verified via SSH connection test to the device on an Ubuntu machine.
b@bk53e $: sudo arp-scan --localnet # to find LAN connection IP addresses of each
> ssh root@<\IPaddress>
Note, this was also tested on a Windows 7 machine, instead using:
arp -a # to find LAN connection IP addresses of each
> # PuTTY manual SSH
Of the five SD cards imaged, one failed to properly resize the rootFS partition and two further systems failed on their autoconfiguration. All three were re-imaged and worked properly afterwards.
For convenience, hostnames were assigned by order of their connection in the ethernet switch, zpm-master, zpm-slave0, zpm-slave1, zpm-slave2, zpm-slave3 (See Appendix 2). Additionally, because the DietPi image used creates a default of MAC address of 00:e0:4c:53:44:58, eth0 MAC addresses were manually modified, to facilitate easier networking. More specifically, on my Ubuntu system, prior to changing, each node was showing up with the same IP address (<IP_ADDRESS>), though interestingly, this didn’t occur on Windows. Thus, the last octet (default: 58) was manually changed to range from 50 to 54, for each device.
(upper left) local IP addresses of the devices, note as well the sequential MAC addresses. (left lower and both right) Three devices pinging each other successfully over the ethernet switch.
For convenience, each address|node-name pair was added to /etc/hosts for all pi’s (ex. Echo “<IP_ADDRESS> zpm-master” >> /etc/hosts). Thus physical devices will be referred to by their hostname throughout this project.
Each device will be required to communicate with any other device, so SSH keys were generated and shared across all nodes:
> root@zpm-master $: ssh-keygen -t rsa -b 4096
> root@zpm-master $: ssh-copy-id root@zpm-slave0
> root@zpm-master $: ssh-copy-id root@zpm-slave1
> …
> root@zpm-slave0 $: ssh-keygen -t rsa -b 4096
> root@zpm-slave0 $: ssh-copy-id root@zpm-master
> root@zpm-slave0 $: ssh-copy-id root@zpm-slave1
...
Finally, a strange omission from the DietPi image was discovered: the device will not reconnect automatically to a dropped ethernet connection. Thus, a network-monitor script has been added and is run as a cron job every 10 minutes, attempting to reconnect if the ethernet link is down (see source code for the project).
Java 8 (1.8.0_65) and scala (2.11.8) were installed on each device.
> root@zpm-master $: apt-get install oracle-java8-jdk scala -y
This is slow on the devices. Across all 5 devices running simultaneously - though sharing a single ethernet connection - installation took a little over 10 minutes on a 15Mbps internet connection.
Hadoop (2.7.5) was downloaded and installed onto only the namenode (master). The slaves will be configured later.
root @zpm-master $: cd /opt
> root@zpm-master $: wget https://mirror.dsrg.utoronto.ca/apache/hadoop/common/hadoop-2.7.5/hadoop-2.7.5.tar.gz
> root@zpm-master $: tar -xvzf hadoop-2.7.5.tar.gz
…
Then environment variables were updated to house Hadoop, adding to ~/.bashrc:
# -- HADOOP ENVIRONMENT VARIABLES START -- #
export HADOOP_HOME=/opt/hadoop-2.7.5
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIV_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
# -- HADOOP ENVIRONMENT VARIABLES END -- #
# -- JAVA AND SCALA HOME VARIABLES -- #
export JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre
And it was checked that Hadoop installed correctly:
root@zpm-master:~# hadoop version
Hadoop 2.7.5
Subversion<EMAIL_ADDRESS>-r 18065c2b6806ed4aa6a3187d77cbe21bb3dba075
Compiled by kshvachk on 2017-12-16T01:06Z
Compiled with protoc 2.5.0
From source with checksum 9f118f95f47043332d51891e37f736e9
This command was run using /opt/hadoop-2.7.5/share/hadoop/common/hadoop-common-2.7.5.jar
Next, the namenode was configured to refer to the hostnames of the master and datanote (slave) devices. Inside of /opt/hadoop-2.7.5/etc/hadoop:
> root@zpm-master $: touch master slaves
> root@zpm-master $: echo “zpm-master” >> master
> root@zpm-master $: echo “zpm-slave0” >> slaves
…
I prepared the HDFS into the namenode, which will be formatted once all datanodes are configured:
> root@zpm-master: mkdir -p /opt/hadoop-2.7.5/hdfs/namenode
> root@zpm-master: chmod 750 /opt/hadoop-2.7.5/hdfs/namenode
And defined the namenode hdfs configuration (/opt/hadoop-2.7.5/etc/hadoop/core-site.xml):
<\configuration>
<\property>
<\name>fs.default.name<\/name>
<\value>hdfs://zpm-master:9000/<\/value>
<\/property>
<\property>
<\name>fs.default.FS<\/name>
<\value>hdfs://zpm-master:9000/<\/value>
<\/property>
<\/configuration>
And the datanode hdfs configurations (/opt/hadoop-2.7.5/etc/hadoop/hdfs-site.xml). Note that replication was set to a sensible factor of 3, though any number up to the amount of datanodes could have been chosen, depending on desire for resiliency:
<\configuration>
<\property>
<\name>dfs.datanode.data.dir<\/name>
<\value>/opt/hadoop-2.7.5/hdfs/datanode<\/value>
<\final>true<\/final>
<\/property>
<\property>
<\name>dfs.namenode.name.dir<\/name>
<\value>/opt/hadoop-2.7.5/hdfs/namenode<\/value>
<\final>true<\/final>
<\/property>
<\property>
<\name>dfs.namenode.http-address<\/name>
<\value>zpm-master:50070<\/value>
<\/property>
<\property>
<\name>dfs.replication<\/name>
<\value>4<\/value>
<\/property>
<\/configuration>
The yarn resource manager ports were similarly configured (/opt/hadoop-2.7.5/etc/hadoop/yarn-site.xml):
<\configuration>
<\property>
<\name>yarn.resourcemanager.resource-tracker.address<\/name>
<\value>zpm-master:8025<\/value>
<\/property>
<\property>
<\name>yarn.resourcemanager.scheduler.address<\/name>
<\value>zpm-master:8035<\/value>
<\/property>
<\property>
<\name>yarn.resourcemanager.address<\/name>
<\value>zpm-master:8050<\/value>
<\/property>
<\property>
<\description>Max available cores data node.<\/description>
<\name>yarn.nodemanager.resource.cpu-vcores<\/name>
<\value>1<\/value>
<\/property>
<\/configuration>
And finally, the mapreduce job tracker (/opt/hadoop-2.7.5/etc/hadoop/mapred-site.xml):
<\configuration>
<\property>
<\name>mapreduce.job.tracker<\/name>
<\value>zpm-master:5431<\/value>
<\/property>
<\property>
<\name>mapred.framework.name<\/name>
<\value>yarn<\/value>
<\/property>
<\/configuration>
Each slave node was then setup copying the configuration of the namenode. While here, it is a good time to also set the java home environment, so Yarn and Hadoop can access it:
> root@zpm-slave0 $: mkdir -p /opt/hadoop-2.7.5/hdfs/datanode
> root@zpm-slave0 $: chmod 750 /opt/hadoop/hdfs/datanode
> root@zpm-slave0 $: rmdir /opt/hadoop/hdfs/namenode
…
> root@zpm-slave0 $: echo “export JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre” >>~/.bashrc
With the master and each slave now configured to recognize the filesystem; so format the namenode.
root@zpm-master: hdfs namenode -format
Some of the output information here is notable.
18/04/12 02:09:52 INFO util.GSet: Computing capacity for map cachedBlocks
18/04/12 02:09:52 INFO util.GSet: VM type = 32-bit
18/04/12 02:09:52 INFO util.GSet: 0.25% max memory 966.8 MB = 2.4 MB
It appears that the system is reading double the amount of memory that the pi zero has (512MB). This was fixed later when tweaking the configuration defaults (see final configuration files).
> root@zpm-master $: start-yarn.sh
> root@zpm-master $: start-dfs.sh
As noted, the pi zero uses an ARMv6 chip, but it was here discovered that the default distributions of Hadoop for ARM chips are compiled for ARMv7 only. Thus, this will cause an error when Yarn attempts to run. The distributed file system, however, was able to run and could be examined via the namenode’s server process (<IP_ADDRESS>:50070). However, since the goal of this process is to set up a usable Hadoop/Spark cluster, and building an ARMv6 binary for yarn is well beyond the scope of this project, it was elected to proceed forward by replacing one of the pi zeros with a Raspberry Pi 3 - which has an ARMv7 CPU. Due to a lack of additional hardware and an insufficient powersource, the initial pi zero namenode was removed from the cluster.
The datanodes, however, could still be run - just only in client mode, not server. More specifically, each instance of “-server” within the datanode options, was replaced with “-client”.
> root@zpm-slave2:/opt/hadoop-2.7.5# grep -r DATANODE_OPTS .
> ./bin/hdfs: HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS"
> ./bin/hdfs: HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"
> ./bin/hdfs.cmd: set HADOOP_OPTS=%HADOOP_OPTS% -server %HADOOP_DATANODE_OPTS%
> root@zpm-slave2:/opt/hadoop-2.7.5# grep -r "\-server" .
> ./yarn: YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"
Thus, the Raspberry Pi 2 was set up as above, with the local IP address of <IP_ADDRESS>, hostname = zpm-master. The 4 datanodes were re-configured with ssh-keys for the new namenode. Once set up, the hdfs was reformatted and Yarn/Hadoop could be run.
Figure: Hadoop server summary page (localhost:<IP_ADDRESS>:50070) showing start time. As no files have been loaded to HDFS just yet, it is correctly showing empty.
YARN server summary page (localhost:<IP_ADDRESS>:8088) showing a summary of the datanodes in the cluster. Presently each device is indicating 8GB of available memory. This is the default set by yarn, and was updated later to appropriately match the pi zero.
Given the 512MB of memory on the pi zeros, configuration of memory was critical to running the cluster. Checking `top` for each of the zeros showed a total memory of 492752 KiB of memory, and about 349000 KiB of free memory with just the base OS processes running. Four specific values were assigned: the yarn nodemanager resource, yarn scheduler, yarn mapreduce resource, and the map reduce memory, each of which can be seen in Figure 5 below. Nodemanager memory is set to the maximum believed that the device and handle, after accounting for the operating system, in this case 320 MB. Following, I set the yarn scheduler maximum and minimum to be 320 MB and 128 MB, respectively. Yarn app mapreduce is set to 64 MB.
Figure: Memory allocation illustration for the nodeManager on each node in the Hadoop cluster (source: www.linode.com). The nodemanager should be the maximum reasonable amount of memory that the device can dedicate after the OS. More specifically, it is the amount reasonable for one container instance. On devices with ample memory, multiple containers can be run simultaneously. Since the pi zero is so limited, it was deemed impossible to run multiple containers at once.
Table 1. Initial Memory allocation variables for each node of the cluster. These were later all heavily modified during testing of the cluster.
| File | Property | Allocation (MB) |
|---|---|---|
| yarn-site.xml | Yarn.nodemanager.resource.memory-mb | 340 |
| yarn-site.xml | Yarn.scheduler.maximum-allocation-mb | 340 |
| yarn-site.xml | Yarn.scheduler.minimum-allocation-mb | 128 |
| yarn-site.xml | Yarn.app.mapreduce.am.resource.mb | 64 |
| mapred-site.xml | Mapreduce.map.memory.mb | 128 |
| mapred-site.xml | Mapreduce.reduce.memory.mb | 128 |
While configuring the $HADOOP_HOME/etc/hadoop/hadoop-env.sh, the java heap maximum was also set. Again This took multiple trials to determine a minimum viable amount:
# The maximum amount of heap to use, in MB. Default is 1000.
export HADOOP_HEAPSIZE=64
Multiple options exist to load data into hdfs. Here, copyFromLocal was used to load Shakespeare.txt. I’ll use it to run the first hadoop job: wordCount.
> root@zpm-master $: cd /home & mkdir localData
> root@zpm-master $: curl https://lintool.github.io/bespin-data/Shakespeare.txt > localData/Shakespeare.txt
> root@zpm-master $: hadoop fs -mkdir -p /home/data
> root@zpm-master $: hadoop fs -copyFromLocal localData/Shakespeare.txt data/Shakespeare.txt
> root@zpm-master $: hadoop fs -ls /home/data
> Found 1 items -rw-r--r-- 3 root supergroup 5328041 2018-04-15 01:57 /home/data/Shakespeare.txt
Figure: Alternatively, files can be located on HDFS via the namenode browser. Note the inefficiency of loading small files to HDFS, where a 5.08 MB text file is allocated a full 128 MB block.
The ‘hello world’ of Hadoop, on this modest cluster. Since Maven is not installed, and would only take up limited resources on the namenode, the compiled jars for all jobs were copied from the local dev laptop to the cluster.
> root@zpm-master $: hadoop jar target/1.0.jar WordCount -input /home/data/Shakespeare.txt -output wc
While running resource usage was checked via terminal `top` as well as the Yarn server. On the first run, the program did not complete, hanging for 60 minutes. Though data appeared to be transferring across the network and each slave could be ssh’d into, it appears that the namenode froze. A SIGTERM command was unable to halt the operation and the namenode could not be ssh’d into. Further, the yarn and hadoop web clients were not accessible. Graceful shutdown was attempted via `stop-yarn.sh` and `stop-dfs.sh` through one of the datanode devices. A hardboot was required on the namenode itself. It appears that the third slave node (zpm-slave2) was the fault here, however the logs on that datanode reveal nothing outstanding. Inside of hdfs-site.xml, blocksize was changed from default 128 MB to 5MB chunks. Since small files will be used, and need to minimize RAM usage, this should limit MapReduce tasks to 5 MB chunks at a time. This requires a reformat of the hdfs as well as reloading of all data to hdfs. Sadly, a second attempt at running the job lead to another seizure of the namenode. This time, yarn was being monitored immediately from job submission, to see if it showed up in the web client. The server froze before the job submission appeared. Again the logs for each node were unenlightening. A third time, instead of submitting via the hadoop command, `yarn` was attempted in place instead (i.e. $: yarn jar target/…), to similar results. Following a post on Reddit [4], I attempted some fairly drastic reconfiguration of the memory allocations for the cluster. This failed similarly. With apologies to the health of the SD card, I enabled a 4G swap partition on the namenode as outlined in a Digital Ocean posting [5].
> root@zpm-master $: fallocate -l 4G /var/swap
> root@zpm-master $: mkwap /var/swap
> root@zpm-master $: swapon /var/swap
> root@zpm-master $: echo “/var/swap none swap sw 0 0” >> /etc/fstab
Finally! The namenode does not freeze. Though, while watching `top` on the namenode during WordCount indicates a minimum of around 26 MB unused, it does indeed show around 100 MB of the swap space being utilized. Sadly however, the job fails: Exception in thread "main" java.io.IOException: No space left on device. This is despite the namenode SD card still having 4.8 GB of available space, and the overall with 45.4 GB available (`hadoop fs -df -h`). Reducing the swap size to 1G did not solve this error.
As /tmp does indeed appear to fill up during WordCount, it was remounted to 2.0G (`mount /tmp -o remount,size=2G`) to see how much gets used. Until it froze, the namenode now used up to 511 MB of space in /tmp. Again, watching `top` on the namenode, the HADOOP_HEAPSIZE was increased from 64 MB to 256 MB, allowing the job submission to get as far as initialization of the JVM metrics, where it froze again. For a final try, sysctl swappiness as set to 25, and the HADOOP_HEAPSIZE to 512MB. Thus far, job progression has not yet made it to running any of the actual data. That is, the namenode is dying simply on the startup of the JVM and its associated objects. Peaking at 644 MB in /tmp, using up the remaining ~512 MB of RAM as well as a peak of 504 MB of swap memory, the job finally finishes in the short time of 174.444 seconds. The output saves to hdfs://user/root/wc.
>root@zpm-master:/home/1.0# hadoop fs -cat /user/root/wc/part* | sort -r -k 2 -n | head
the 27361
and 26028
i 20681
to 19150
of 17463
a 14593
you 13615
my 12481
in 10956
that 10890
In the interest of time and exploration of possibilities of this project, it was here elected to move on to Spark, instead of testing the runtimes of the other simple jobs. One note however was that the job did not show up in the yarn server - likely a configuration oversight somewhere to be solved at a later date.
While installing Spark is actually relatively straightforward (left as an exercise for the reader), it is ultimately impossible on thead little Pi Zeros due to their ARMv6 architecture:
> Error occurred during initialization of VM
> Server VM is only supported on ARMv7+ VFP
Without much left to do, this is where the project ends.
Feel free to contact me with any questions about the build!
This project was submitted in a different form as a CS 651 (W2019, UWaterloo) final project.


HTML Selectors
If you look at the code for this page (right-click and choose "View Source"), you'll see
that each HTML element has a css entry in the <style> section at the top of the document.
You can easily change the appearance of this page by tweaking the CSS values at the top of this document,
saving, and reloading the page.


[Gecode] Discarding propagators
Gustavo Gutierrez
ggutierrez at atlas.puj.edu.co
Fri Mar 18 23:00:04 CET 2005
Hi all,
Few weeks ago i posted a message with subject "design question". There, i
was trying to implement a propagator that uses other propagators to verify
some kind of consistency and then decide to prune or not domain values.
That is, the user-imposed propagator calls auxiliar propagators, and using
the outcome of such propagators, takes certain actions.
In order to do that without fall in space failures i use a copy of
variables and create a new propagator, then call the propagate method
directly. This new propagator needs to be disabled after the consistency
proof (i.e. it will not exist as long as ordinary propagators imposed by
user). In order to "simulate" propagator disabling i added a new method
that contains a flag. When the flag becomes true the propagate method
returns ES_SUBSUMMED. My hope is that the auxiliar propagator will be
removed (in a safe way) from the propagation queue at some time after
this.
I was able of implementing the "main" propagator following this approach.
My problem is that it works fine in some examples, but for others it
raises a segfault. Very likely, I have a misassumption regarding the role
and functioning of the propagation queue. I would apprecciate any comments
and/or suggestions on this issue.
Thanks in advance,
Gustavo Gutierrez
More information about the gecode-users
mailing list


Colin Skipp, who played Tony Archer, dies aged 80
The soap said that Tony Archer ‘was absolutely Colin’s creation’.
Colin Skipp, who played Tony Archer for more than 40 years on The Archers, has died at the age of 80.
The Radio 4 programme said Skipp, who featured in many of Ambridge’s most memorable storylines, “will be greatly missed”.
Skipp played Tony Archer from 1967 to 2013 before bowing out of the BBC soap because of ill health.
Its editor Jeremy Howe said: “Colin had a wonderful voice, which made Tony singular and distinctive: He was absolutely Colin’s creation”.
Pat Gallimore, who plays Tony’s wife Pat in the Radio 4 drama, said: “Colin and I shared many years together as radio spouses before ill health forced him to retire, and I have 40 years of memories to look back on with happiness.
“Colin was a very fine actor who brought truth and integrity to the role of Tony, as well as being a good friend and generous colleague.”
His character faced tragedy over the years, blaming himself when his son John was killed in a tractor accident in 1998.
Tony had a reckless streak in his youth and at one point an exasperated Pat moved out.
The couple had two other children, Helen and Tom.
They created Bridge Farm with its profitable organic unit, which was at the vanguard of radical thinking at the time.
Skipp retired from the programme after having a series of heart attacks and actor David Troughton took over the character.


New Client
Client Login
Add Credit Card
Search Products
Add Product
Buy products
List URLs
List Products
List Categories
List Food
List Drinks
List Desserts
Shopping Cart


{% extends "base.html" %} {% load static %} {% load wagtailcore_tags %} {% block extra_css %} {% endblock extra_css %} {% block content %}
Buffalo Mutual Aid Network is a social action group for peer-to-peer organizing, humanitarian assistance, and reliable information sharing developed in response to the COVID-19 crisis.
Consider volunteering to provide mutual aid during this crisis. This can include delivering groceries or supplies. You might have another skill or resource to contribute, such as providing childcare. Compensation for your transportation, time, or service might be available.
Sign up to provide aidAll money raised will go towards mutual aid, which will include providing food, over the counter medicine and cleaning supplies. We are preparing to include direct support such as childcare, care giving, and running errands. This is not charity, this is solidarity and the truest sense of community. Please donate now to support these efforts!
Donate on GoFundMeThis Solidarity Relief Fund is for workers who live in the Greater Buffalo area and is similar to a microgrant. If your income has been directly affected as a result of COVID-19, you can request emergency financial mutual aid here. Note that our network has a separate form for requesting non-monetary aid.
Request money Donate moneyBuffalo Mutual Aid Network is an evolving community effort managed entirely by volunteers. This website is an ongoing project with Code for Buffalo volunteers.
If you'd like to join us in helping lead this collective effort, welcome!
Roles we need filled:
Roles we need filled:
Photos by Perry Grone, RoseBox رز باکس, Kat Yukawa, Tatiana Byzova on Unsplash
{% endblock photo_credit %}


A type alias for the exception handler function.
Construct a new signal.
The sender which owns the signal.
The sender which owns the signal.
Connect a slot to the signal.
The slot to invoke when the signal is emitted.
The this
context for the slot. If provided,
this must be a non-primitive object.
true
if the connection succeeds, false
otherwise.
Disconnect a slot from the signal.
The slot to disconnect from the signal.
The this
context for the slot. If provided,
this must be a non-primitive object.
true
if the connection is removed, false
otherwise.
Emit the signal and invoke the connected slots.
The args to pass to the connected slots.
Slots are invoked synchronously in connection order.
Exceptions thrown by connected slots will be caught and logged.
Clear all signal data associated with the given object.
The object for which the data should be cleared.
This removes all signal connections and any other signal data associated with the object.
Remove all connections where an object is the sender or receiver.
The object of interest.
If a thisArg
is provided when connecting a signal, that object
is considered the receiver. Otherwise, the slot
is considered
the receiver.
Remove all connections between a sender and receiver.
The sender object of interest.
The receiver object of interest.
If a thisArg
is provided when connecting a signal, that object
is considered the receiver. Otherwise, the slot
is considered
the receiver.
Remove all connections where the given object is the receiver.
The receiver object of interest.
If a thisArg
is provided when connecting a signal, that object
is considered the receiver. Otherwise, the slot
is considered
the receiver.
Remove all connections where the given object is the sender.
The sender object of interest.
Get the signal exception handler.
The current exception handler.
The default exception handler is console.error
.
Set the signal exception handler.
The function to use as the exception handler.
The old exception handler.
The exception handler is invoked when a slot throws an exception.
Generated using TypeDoc
A concrete implementation of
ISignal
. The namespace for theSignal
class statics.Example
import { ISignal, Signal } from '@phosphor/signaling'; class SomeClass { constructor(name: string) { this.name = name; } readonly name: string; get valueChanged: ISignal<this, number> { return this._valueChanged; } get value(): number { return this._value; } set value(value: number) { if (value === this._value) { return; } this._value = value; this._valueChanged.emit(value); } private _value = 0; private _valueChanged = new Signal<this, number>(this); } function logger(sender: SomeClass, value: number): void { console.log(sender.name, value); } let m1 = new SomeClass('foo'); let m2 = new SomeClass('bar'); m1.valueChanged.connect(logger); m2.valueChanged.connect(logger); m1.value = 42; // logs: foo 42 m2.value = 17; // logs: bar 17


Quokka Marketplace
Add a Listing
Name of Item
Main category
Choose...
Mobiles & Tablets
TVs
Appliance
Home & Kitchen
Health & Beauty
Price
Photo
Quantity
Submit


Navigation
Home
Livestock
Another Menu
Profile
Settings
Logout
Overview
Livestock
Employees
Employees
Sheep market prices
2020
2019
6
Livestock
#
Tag number
Breed
Weight
Year on farm
Employees


{% extends "layout-home.html" %} {% block pageTitle %} {{ serviceName }} {% endblock %} {% block content %} {% include "./partials/_actions.html" %} {% include "./partials/_video.html" %} {% include "./partials/_announcements.html" %}


When : $date$
Requested url : $url$
User agent : $agent$
JMVC says : $msg$
The requested page was not found. In ~$secToRedir$ seconds, you will be redirected to the Home page.


Normal text emphasized text secondary text ancillary text What I’m saying is that it’s so, so simple to make sites easier to read.   Click me!
 — Brad Hollande
Normal text emphasized text secondary text ancillary text What I’m saying is that it’s so, so simple to make sites easier to read.   Click me!  — Brad Hollande


Name:
Surname:
Name:
Surname:
User Form
Name:
Surname:
Initial textarea text!
First Item
Second Item
Please select a Country
Argentina
Brasil
Chile
Canada
+ User
User


2021-02-22 - Learn about the history and philosophy behind the Chapeaux project.
2021-02-18 - Chapeaux is a digital supply chain that delivers web assets for developers and stakeholders.


WMS -- Web Map Services
Driver short name
WMS
Build dependencies
libcurl
Accessing several different types of web image services is possible using the WMS format in GDAL.
Driver capabilities
Supports CreateCopy()
This driver supports the GDALDriver::CreateCopy()
operation
Supports Georeferencing
This driver supports georeferencing
Supports VirtualIO
This driver supports virtual I/O operations (/vsimem/, etc.)
XML description file
Services are accessed by creating a local
service description XML file -- there are examples below for each of the
supported image services. It is important that there be no spaces or
other content before the <GDAL_WMS>
element.
<GDAL_WMS> |
|
<Service name="WMS"> |
Define what mini-driver to use, currently supported are: WMS, WorldWind, TileService, TMS, TiledWMS, VirtualEarth or AGS. (required) |
<Version>1.1.1</Version> |
WMS version. (optional, defaults to 1.1.1) |
<ServerUrl>http://host.domain.com/wms.cgi?</ServerUrl> |
WMS server URL. (required) |
<SRS>EPSG:4326</SRS> |
Image projection (optional, defaults to EPSG:4326 in WMS and 102100 in AGS, WMS version 1.1.1 or below only and ArcGIS Server). For ArcGIS Server the spatial reference can be specified as either a well-known ID or as a spatial reference json object |
<CRS>CRS:83</CRS> |
Image projection (optional, defaults to EPSG:4326, WMS version 1.3.0 or above only) |
<ImageFormat>image/jpeg</ImageFormat> |
Format in which to request data. Paletted formats like image/gif will be converted to RGB. (optional, defaults to image/jpeg) |
<Transparent>FALSE</Transparent> |
Set to TRUE to include "transparent=TRUE" in the WMS GetMap request (optional defaults to FALSE). The request format and BandsCount need to support alpha. |
<Layers>modis%2Cglobal_mosaic</Layers> |
A URL encoded, comma separated string of layers (required, except for TiledWMS) |
<TiledGroupName>Clementine</TiledGroupName> |
Comma separated list of layers. (required for TiledWMS) |
<Styles></Styles> |
Comma separated list of styles. (optional) |
<BBoxOrder>xyXY</BBoxOrder> |
Reorder bbox coordinates arbitrarily. May be required for version 1.3 servers. (optional) x - low X coordinate, y - low Y coordinate, X - high X coordinate, Y - high Y coordinate |
</Service> |
|
<DataWindow> |
Define size and extents of the data. (required, except for TiledWMS and VirtualEarth) |
<UpperLeftX>-180.0</UpperLeftX> |
X (longitude) coordinate of upper-left corner. (optional, defaults to -180.0, except for VirtualEarth) |
<UpperLeftY>90.0</UpperLeftY> |
Y (latitude) coordinate of upper-left corner. (optional, defaults to 90.0, except for VirtualEarth) |
<LowerRightX>180.0</LowerRightX> |
X (longitude) coordinate of lower-right corner. (optional, defaults to 180.0, except for VirtualEarth) |
<LowerRightY>-90.0</LowerRightY> |
Y (latitude) coordinate of lower-right corner. (optional, defaults to -90.0, except for VirtualEarth) |
<SizeX>2666666</SizeX> |
Image size in pixels. |
<SizeY>1333333</SizeY> |
Image size in pixels. |
<TileX>0</TileX> |
Added to tile X value at highest resolution. (ignored for WMS, tiled image sources only, optional, defaults to 0) |
<TileY>0</TileY> |
Added to tile Y value at highest resolution. (ignored for WMS, tiled image sources only, optional, defaults to 0) |
<TileLevel>0</TileLevel> |
Tile level at highest resolution. (tiled image sources only, optional, defaults to 0) |
<TileCountX>0</TileCountX> |
Can be used to define image size, SizeX = TileCountX * BlockSizeX * 2TileLevel. (tiled image sources only, optional, defaults to 0) |
<TileCountY>0</TileCountY> |
Can be used to define image size, SizeY = TileCountY * BlockSizeY * 2TileLevel. (tiled image sources only, optional, defaults to 0) |
<YOrigin>top</YOrigin> |
Can be used to define the position of the Y origin with respect to the tile grid. Possible values are 'top', 'bottom', and 'default', where the default behavior is mini-driver-specific. (TMS mini-driver only, optional, defaults to 'bottom' for TMS) |
</DataWindow> |
|
<Projection>EPSG:4326</Projection> |
Image projection (optional, defaults to value reported by mini-driver or EPSG:4326) |
<IdentificationTolerance>2</IdentificationTolerance> |
Identification tolerance (optional, defaults to 2) |
<BandsCount>3</BandsCount> |
Number of bands/channels, 1 for grayscale data, 3 for RGB, 4 for RGBA. (optional, defaults to 3) |
<DataType>Byte</DataType> |
Band data type, one of: Byte, Int16, UInt16, Int32, UInt32, Float32, Float64, etc.. (optional, defaults to Byte) |
<DataValues NoData="0 0 0" min="1 1 1" max="255 255 255" /> |
Define NoData and/or minimum and/or maximum value for bands. nodata_values, min_values, max_values can be one single value, or a value per band, with a space separator between value |
<BlockSizeX>1024</BlockSizeX> |
Block size in pixels. (optional, defaults to 1024, except for VirtualEarth) |
<BlockSizeY>1024</BlockSizeY> |
Block size in pixels. (optional, defaults to 1024, except for VirtualEarth) |
<OverviewCount>10</OverviewCount> |
Count of reduced resolution layers each having 2 times lower resolution. (optional, default is calculated at runtime) |
<Cache> |
Enable local disk cache. Allows for offline operation. (optional but present in the autogenerated service file, cache is disabled when not present or if it is overridden with GDAL_ENABLE_WMS_CACHE=NO) |
<Path>./gdalwmscache</Path> |
Location where to store cache files. It is safe to use same cache path for different data sources. /vsimem/ paths are supported allowing for temporary in-memory cache. (optional, defaults to ./gdalwmscache if GDAL_DEFAULT_WMS_CACHE_PATH configuration option is not specified) |
<Depth>2</Depth> |
Number of directory layers. 2 will result in files being written as cache_path/A/B/ABCDEF... (optional, defaults to 2) |
<Extension>.jpg</Extension> |
Append to cache files. (optional, defaults to none) |
<Type>file</Type> |
Cache type. Now supported only 'file' type. In 'file' cache type files are stored in file system folders. (optional, defaults to 'file') |
<Expires>604800</Expires> |
Time in seconds cached files will stay valid. If cached file expires it is deleted when maximum size of cache is reached. Also expired file can be overwritten by the new one from web. Default value is 7 days (604800s). |
<MaxSize>67108864</MaxSize> |
The cache maximum size in bytes. If cache reached maximum size, expired cached files will be deleted. Default value is 64 Mb (67108864 bytes). |
<CleanTimeout>120</CleanTimeout> |
Clean Thread Run Timeout in seconds. How often to run the clean thread, which finds and deletes expired cached files. Default value is 120s. Use value of 0 to disable the Clean Thread (effectively unlimited cache size). If you intend to use very large cache size you might want to disable the cache clean or to use a much longer timeout as the time that takes to scan the cache files for expired cache files might be long. ("disabled" was the only option for GDAL <= 2.2; "120s" was the only option for 2.3 <= GDAL <= 3.1). |
<Unique>True</Unique> |
If set to true the path will appended with md5 hash of ServerURL. Default value is true. |
</Cache> |
|
<MaxConnections>2</MaxConnections> |
Maximum number of simultaneous connections. (optional, defaults to 2). Can also be set with the |
<Timeout>300</Timeout> |
Connection timeout in seconds. (optional, defaults to 300) |
<OfflineMode>true</OfflineMode> |
Do not download any new images, use only what is in cache. Useful only with cache enabled. (optional, defaults to false) |
<AdviseRead>true</AdviseRead> |
Enable AdviseRead API call - download images into cache. (optional, defaults to false) |
<VerifyAdviseRead>true</VerifyAdviseRead> |
Open each downloaded image and do some basic checks before writing into cache. Disabling can save some CPU cycles if server is trusted to always return correct images. (optional, defaults to true) |
<ClampRequests>false</ClampRequests> |
Should requests, that otherwise would be partially outside of defined data window, be clipped resulting in smaller than block size request. (optional, defaults to true) |
<UserAgent>GDAL WMS driver (http://www.gdal.org/frmt_wms.html)</UserAgent> |
HTTP User-agent string. Some servers might require a well-known user-agent such as "Mozilla/5.0" (optional, defaults to "GDAL WMS driver (http://www.gdal.org/frmt_wms.html)"). When used with some servers, like OpenStreetMap ones, it is highly recommended to put a custom user agent to avoid being blocked if the default user agent had to be blocked. |
<Accept>mimetype>/Accept> |
HTTP Accept header to specify the MIME type of the expected output of the server. Empty by default |
<UserPwd>user:password</UserPwd> |
User and Password for HTTP authentication (optional). |
<UnsafeSSL>true</UnsafeSSL> |
Skip SSL certificate verification. May be needed if server is using a self signed certificate (optional, defaults to false). |
<Referer>http://example.foo/</Referer> |
HTTP Referer string. Some servers might require it (optional). |
<ZeroBlockHttpCodes>204,404</ZeroBlockHttpCodes> |
Comma separated list of HTTP response codes that will be interpreted as a 0 filled image (i.e. black for 3 bands, and transparent for 4 bands) instead of aborting the request. (optional, defaults to 204) |
<ZeroBlockOnServerException>true</ZeroBlockOnServerException> |
Whether to treat a Service Exception returned by the server as a 0 filled image instead of aborting the request. (optional, defaults to false) |
</GDAL_WMS> |
|
Starting with GDAL 2.3, additional HTTP headers can be sent by setting the GDAL_HTTP_HEADER_FILE configuration option to point to a filename of a text file with “key: value” HTTP headers.
Minidrivers
The GDAL WMS driver has support for several internal 'minidrivers', which allow access to different web mapping services. Each of these services may support a different set of options in the Service block.
WMS
Communications with an OGC WMS server. Has support for both tiled and untiled requests.
WMS layers can be queried (through a GetFeatureInfo request) with the gdallocationinfo utility, or with a GetMetadataItem("Pixel_iCol_iLine", "LocationInfo") call on a band object.
gdallocationinfo "WMS:http://demo.opengeo.org/geoserver/gwc/service/wms?SERVICE=WMS&VERSION=1.1.1&
REQUEST=GetMap&LAYERS=og%3Abugsites&SRS=EPSG:900913&
BBOX=-1.15841845090625E7,5479006.186718751,-1.1505912992109375E7,5557277.703671876&
FORMAT=image/png&TILESIZE=256&OVERVIEWCOUNT=25&MINRESOLUTION=0.0046653459640220&TILED=true"
-geoloc -11547071.455 5528616 -xml -b 1
Output:
Report pixel="248595" line="191985">
<BandReport band="1">
<LocationInfo>
<wfs:FeatureCollection xmlns="http://www.opengis.net/wfs"
xmlns:wfs="http://www.opengis.net/wfs"
xmlns:gml="http://www.opengis.net/gml"
xmlns:og="http://opengeo.org"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://opengeo.org http://demo.opengeo.org/geoserver/wfs?service=WFS&version=1.0.0&request=DescribeFeatureType&typeName=og%3Abugsites http://www.opengis.net/wfs http://demo.opengeo.org/geoserver/schemas/wfs/1.0.0/WFS-basic.xsd">
<gml:boundedBy>
<gml:Box srsName="http://www.opengis.net/gml/srs/epsg.xml#26713">
<gml:coordinates xmlns:gml="http://www.opengis.net/gml" decimal="." cs="," ts=" ">601228,4917635 601228,4917635</gml:coordinates>
</gml:Box>
</gml:boundedBy>
<gml:featureMember>
<og:bugsites fid="bugsites.40946">
<gml:boundedBy>
<gml:Box srsName="http://www.opengis.net/gml/srs/epsg.xml#26713">
<gml:coordinates xmlns:gml="http://www.opengis.net/gml" decimal="." cs="," ts=" ">601228,4917635 601228,4917635</gml:coordinates>
</gml:Box>
</gml:boundedBy>
<og:cat>86</og:cat>
<og:str1>Beetle site</og:str1>
<og:the_geom>
<gml:Point srsName="http://www.opengis.net/gml/srs/epsg.xml#26713">
<gml:coordinates xmlns:gml="http://www.opengis.net/gml" decimal="." cs="," ts=" ">601228,4917635</gml:coordinates>
</gml:Point>
</og:the_geom>
</og:bugsites>
</gml:featureMember>
</wfs:FeatureCollection>
</LocationInfo>
<Value>255</Value>
</BandReport>
</Report>
TileService
Service to support talking to a WorldWind TileService. Access is always tile based.
WorldWind
Access to web-based WorldWind tile services. Access is always tile based.
TMS
The TMS Minidriver is designed primarily to support the users of the TMS Specification. This service supports only access by tiles.
Because TMS is similar to many other 'x/y/z' flavored services on the web, this service can also be used to access these services. To use it in this fashion, you can use replacement variables, of the format ${x}, ${y}, etc.
Supported variables (name is case sensitive) are :
${x} -- x position of the tile
${y} -- y position of the tile. This can be either from the top or the bottom of the tileset, based on whether the YOrigin parameter is set to true or false.
${z} -- z position of the tile -- zoom level
${version} -- version parameter, set in the config file. Defaults to 1.0.0.
${format} -- format parameter, set in the config file. Defaults to 'jpg'.
${layer} -- layer parameter, set in the config file. Defaults to nothing.
http://tilecache.osgeo.org/wms-c/Basic.py/${version}/${layer}/${z}/${x}/${y}.${format}
The TMS Service has 3 XML configuration elements that are different from
other services: Format
which defaults to jpg
, Layer
which
has no default, and Version
which defaults to 1.0.0
.
Additionally, the TMS service respects one additional parameter, at the
DataWindow level, which is the YOrigin element. This element should be
one of bottom
(the default in TMS) or top
, which matches
OpenStreetMap and many other popular tile services.
Two examples of usage of the TMS service are included in the examples below.
OnEarth Tiled WMS
The OnEarth Tiled WMS minidriver supports the Tiled WMS specification implemented for the JPL OnEarth driver per the specification at http://web.archive.org/web/20130511182803/http://onearth.jpl.nasa.gov/tiled.html.
Only the ServerUrl and the TiledGroupName are required, most of the required information is automatically fetched from the remote server using the GetTileService method at open time.
A typical OnEarth Tiled WMS configuration file might look like:
<GDAL_WMS>
<Service name="TiledWMS">
<ServerUrl>https://gibs.earthdata.nasa.gov/twms/epsg4326/best/twms.cgi?</ServerUrl>
<TiledGroupName>MODIS Terra CorrectedReflectance TrueColor tileset</TiledGroupName>
<Change key="${time}">2020-02-02</Change>
</Service>
</GDAL_WMS>
The TiledWMS minidriver can use the following open options :
TiledGroupName -- The value is a string that identifies one of the tiled services available on the server
Change -- A <Key>:<Value> pair, which will be passed to the server. The key has to match a change key that the server declares for the respective tiled group. This option can be used multiple times, for different keys. Example: - Change=time:2020-02-02
These open options are only accepted if the corresponding XML element is not present in the configuration file.
VirtualEarth
Access to web-based Virtual Earth tile services. Access is always tile based.
The ${quadkey} variable must be found in the ServerUrl element.
The DataWindow element might be omitted. The default values are :
UpperLeftX = -20037508.34
UpperLeftY = 20037508.34
LowerRightX = 20037508.34
LowerRightY = -20037508.34
TileLevel = 21
OverviewCount = 20
SRS = EPSG:3857
BlockSizeX = 256
BlockSizeY = 256
ArcGIS REST API
Access to ArcGIS REST map service resource (untiled requests).
AGS layers can be queried (through a GetFeatureInfo request) with the gdallocationinfo utility, or with a GetMetadataItem("Pixel_iCol_iLine", "LocationInfo") call on a band object.
gdallocationinfo -wgs84 "<GDAL_WMS><Service name=\"AGS\"><ServerUrl>http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Specialty/ESRI_StateCityHighway_USA/MapServer</ServerUrl><BBoxOrder>xyXY</BBoxOrder><SRS>3857</SRS></Service><DataWindow><UpperLeftX>-20037508.34</UpperLeftX><UpperLeftY>20037508.34</UpperLeftY><LowerRightX>20037508.34</LowerRightX><LowerRightY>-20037508.34</LowerRightY><SizeX>512</SizeX><SizeY>512</SizeY></DataWindow></GDAL_WMS>" -75.704 39.75
Internet Imaging Protocol (IIP) (GDAL 2.1 and later)
Access to images served through IIP protocol. The server must support the JTL (Retrieve a tile as a complete JFIF image) extension of the IIP protocol.
If using the XML syntax, the ServerURL must contain the FIF parameter.
Otherwise it is also possible to use "IIP:http://foo.com/FIF=image_name" syntax as connection string, to retrieve from the server information on the full resolution dimension and the number of resolutions.
The XML definition can then be generated with "gdal_translate IIP:http://foo.com/FIF=image_name out.xml -of WMS"
Configuration options
The following configuration options are available:
GDAL_MAX_CONNECTIONS=<integer>: (GDAL >= 3.2) Defaults to
2
. Set the maximum number of simultaneous connections.
Examples
Warning
The below examples rely on external servers that might go down over time. They are just here to demonstrate how to use the various services.
- onearth_global_mosaic.xml - Landsat mosaic from a OnEarth WMS server
gdal_translate -of JPEG -outsize 500 250 onearth_global_mosaic.xml onearth_global_mosaic.jpg
gdal_translate -of JPEG -projwin -10 55 30 35 -outsize 500 250 onearth_global_mosaic.xml onearth_global_mosaic2.jpg
Note : this particular server does no longer accept regular WMS queries.
metacarta_wmsc.xml - It is possible to configure a WMS Service conforming to a WMS-C cache by specifying a number of overviews and specifying the 'block size' as the tile size of the cache. The following example is a sample set up for a 19-level "Global Profile" WMS-C cache.
- tileservice_bmng.xml - - TileService, Blue Marble NG (January)
- tileservice_nysdop2004.xml - TileService, NYSDOP 2004
- OpenStreetMap TMS Service Example: Connect to OpenStreetMap tile service. Note that this file takes advantage of the tile cache; more information about configuring the tile cache settings is available above. Please also change the <UserAgent>, to avoid the default one being used, and potentially blocked by OSM servers in case a too big usage of it would be seen.
gdal_translate -of PNG -outsize 512 512 frmt_wms_openstreetmap_tms.xml openstreetmap.png
- MetaCarta TMS Layer Example, accessing the default MetaCarta TMS layer.
gdal_translate -of PNG -outsize 512 256 frmt_wms_metacarta_tms.xml metacarta.png
BlueMarble Amazon S3 Example accessed with the TMS minidriver.
Google Maps accessed with the TMS minidriver.
ArcGIS MapServer Tiles accessed with the TMS minidriver.
OnEarth Tiled WMS Clementine, daily, and srtm examples.
VirtualEarth Aerial Layer accessed with the VirtualEarth minidriver.
ArcGIS online sample server layer accessed with the ArcGIS Server REST API minidriver.
IIP online sample server layer accessed with the IIP minidriver.
Open syntax
The WMS driver can open :
a local service description XML file :
gdalinfo description_file.xml
the content of a description XML file provided as filename :
gdalinfo "<GDAL_WMS><Service name=\"TiledWMS\"><ServerUrl>https://gibs.earthdata.nasa.gov/twms/epsg4326/best/twms.cgi?</ServerUrl><TiledGroupName>MODIS Terra CorrectedReflectance Bands367 tileset</TiledGroupName></Service></GDAL_WMS>"
the base URL of a WMS service, prefixed with WMS: :
gdalinfo "WMS:http://wms.geobase.ca/wms-bin/cubeserv.cgi"
A list of subdatasets will be returned, resulting from the parsing of the GetCapabilities request on that server.
a pseudo GetMap request, such as the subdataset name returned by the previous syntax :
gdalinfo "WMS:http://wms.geobase.ca/wms-bin/cubeserv.cgi?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=DNEC_250K%3AELEVATION%2FELEVATION&SRS=EPSG:42304&BBOX=-3000000,-1500000,6000000,4500000"
the base URL of a Tiled WMS service, prefixed with WMS: and with request=GetTileService as GET argument:
gdalinfo "WMS:https://gibs.earthdata.nasa.gov/twms/epsg4326/best/twms.cgi?request=GetTileService"
A list of subdatasets will be returned, resulting from the parsing of the GetTileService request on that server.
the URL of a REST definition for a ArcGIS MapServer:
gdalinfo "http://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer?f=json&pretty=true"
(GDAL >= 2.1.0) the URL of a IIP image:
gdalinfo "IIP:http://merovingio.c2rmf.cnrs.fr/fcgi-bin/iipsrv.fcgi?FIF=globe.256x256.tif"
Generation of WMS service description XML file
The WMS service description XML file can be generated manually, or created as the output of the CreateCopy() operation of the WMS driver, only if the source dataset is itself a WMS dataset. Said otherwise, you can use gdal_translate with as source dataset any of the above syntax mentioned in "Open syntax" and as output an XML file. For example:
gdal_translate "http://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer?f=json" wms.xml -of WMS
The generated file will come with default values that you may need to edit.


--- title: J1149 - Home layout: default bodyClass: page-home ---
J1149 provides documentation and examples for creating software with Project PAI.


Using CPUID to get Processor Information
Chip manufacturers are constantly trying to improve the performance of their products. One way that Intel has historically accomplished this is through the inclusion of datapaths meant for certain common use cases (floating point operations in large matrices for example.) But, in order to take advantage of those datapaths, it must be supported across several layers of abstraction.
At the level of assembly language, the Instruction Set Architecture (ISA) is essentially the Application Programming Interface (API) of the CPU - it details all the functions that are available, their inputs, and outputs. However, the adding new instructions to the ISA poses issues for backward compatibility.
A program that would benefit from the new datapath would not have any way of determining if the computer supports the new instructions or not. Thus, the program would either choose to
- Ignore the performance improvements and implement the task in an older manner. Or,
- Use the new instructions and forgo compatibility with older hardware.
Fortunately, the x86 ISA (and by extension x86-64) has a solution that allows a program to use the new instructions (if applicable) without necessarily crashing on older hardware.
The CPUID Instruction
As the name suggests, the CPUID instruction allows a program to peer into the black box of the CPU and get some information about it and its features. However, using CPUID is a little different from the more common x86 instructions like add
or mov
. The CPUID instruction is a good example why x86(-64) is called a Complex Instruction Set Computer (CISC): CPUID doesn’t take any arguments, what it does is dependent on the content of registers when CPUID runs, and its output changes the values of multiple registers. All in a single instruction!
By using CPUID and carefully following Intel’s documentation (starting on page 3-190) on setting up the input and interpreting the instruction’s output, a program can determine if it should take one code branch that uses newer technology, or if it must use another branch designed for older technology.
Example Usage
Microsoft Visual C++ offers two simple wrapper functions for using CPUID (see below.) Some alternative ways would be to use in-line assembly (although this is dependent on compilers) or write your own function in assembly ;)
1
2
3
4
5
6
7
8
9
10
void __cpuid(
int cpuInfo[4],
int function_id
);
void __cpuidex(
int cpuInfo[4],
int function_id,
int subfunction_id
);
To summarize from Microsoft’s documentation, the values of the registers eax
, ebx
, ecx
, and edx
and returned in the cpuInfo array. The primary input value is passed in eax
as function_id. An additional input value is required to access some CPUID
functionality, which is passed into ecx
as subfunction_id.
By disassembling the code, (I’ve used the 32-bit version of x64dbg here) we can get an example of how the above C++ is implemented in assembly.
1
2
3
4
5
6
7
mov eax, 1
xor ecx, ecx
cpuid
mov dword ptr ds:[esi],eax
mov dword ptr ds:[esi+4],ebx
mov dword ptr ds:[esi+8],ecx
mov dword ptr ds:[esi+C],edx
As expected, the assembly moves the passes the primary input in eax
, sets ecx
to 0 by using xor on itself, and then uses the CPUID
instruction. Lines 4-7 are storing the values of eax
, ebx
, ecx
, and edx
respectively in the output array. The value of esi
a pointer to the first element of the array. Indices 1, 2, and 3 are each offset by 4 bytes from the previous array index.
For a given input, each of the four registers contains 32 bits that can be used as flags to indicate the support of a certain feature. We can mask off that bit and reference the documentation to generate a true/false list of supported features.
This type of information is most important at the level of the compiler, as it is the piece of software that must determine how to map code in high-level languages into an assembly language program that is as efficient as possible.


Page Links: Index Page
| Nick Thomas 25th Nov 2010, 18:00 permalink Post: 774 |
Reading this thread has encouraged me to rewatch my Concorde ITVV DVD. During one of the checklists the term TLA is read out as part of the checklist and the reply was something like "13 degrees set"
Am I right in assuming that TLA stands for throttle lever angle? and if so did setting the throttles to a certain angle then give a known thrust setting? Regards Nick |
|
| ChristiaanJ 25th Nov 2010, 18:29 permalink Post: 775 |
Nick
,
I think it refers to setting a 'bug' for the TLA during noise reduction. Have a google for "Concorde TLA" and poke around a bit, I think you'll find far more far sooner than I can help you with...... CJ |
|
| Bellerophon 25th Nov 2010, 21:00 permalink Post: 776 |
Nick Thomas
... Am I right in assuming that TLA stands for throttle lever angle? ... Yes. TLA was a TLA for throttle lever angle. ... did setting the throttles to a certain angle then give a known thrust setting? ... Yes, the calculated thrust required at the noise abatement point. The TLA was calculated by the NHP (as part of the take-off performance calculations) and later cross-checked by the Captain and F/E. I don\x92t know whether it is audible on the DVD, but the checklist question was \x93Clock and TLA bugs\x94 . If not already set, the Captain and F/O would set the noise abatement time on their respective (countdown) clocks and the F/E would cross-checked them, and, if not already set, the F/E would set the TLA bugs (on an engraved scale on either side of the throttle quadrant) to the noise abatement setting and the Captain and F/O would cross-check him. The checklist response was \x94Set\x94 . Best Regards Bellerophon |
|
| Nick Thomas 25th Nov 2010, 23:17 permalink Post: 780 |
Thanks all for your answers. Bellerophon when I typed my TLA question I could not remember the exact checklist phrase but on reading your answer I am certain that "Clocks and TLA bugs" is clearly said.
It never crossed my mind that there was a connection between Clocks and TLA. Of course if I had given it a little more thought I would have realised that they must be connected on the checklist for a good reason. The other thing I noticed on the DVD was that the FE said that the brake pressure gauge didn't provide a reading untill about FL 450. I just wondered why that was? Regards Nick |
|
| M2dude 26th Nov 2010, 08:58 permalink Post: 782 |
EXWOK
Quote:
The 18\xb0 TLA limitation was set to prevent TOD 'pop' surges, due to the resulting large intake ramp angle causing excessive compressor face distortion (the one side effect of the intake 'thin lip' modification). Best Regards Dude |
|
| Bellerophon 22nd Feb 2014, 03:16 permalink Post: 1789 |
ruddman
...Being that the Concorde looks like a slippery sob, how were the descents planned?... The distance required to decel/descend from M2.0 in cruise/climb down to 3,000 at 250 kts was obtained from a checklist chart. Entering with the (expected) FL at Top of Descent and then correcting for the average wind component expected in the descent and also for the temperature deviation from ISA gave the required track miles. It wasn\x92t used a lot, because generally the more critical descent requirement was to decelerate so as to be (just) below M1.0 at a specified point on the arrival route, for noise reasons, to avoid booming land. There was a second chart, utilised in the same way as the first, which provided this information. Sometimes this distance might need to be increased a little, as, if a subsonic cruise was expected before continuing the approach, the engines were \x93warmed\x94 up at M0.97 and after passing FL410, by the application of power, for one minute, by the Flight Engineer. ...Did you just pull the throttles back to flight idle?... Only if you were willing to run the risk four pop surges from the engines and the near certainty of a clip round the ear from your Flight Engineer. Usually the pilots handled the throttles from \x93Power Up to Gear Up\x94 and from \x93Gear Down to Shut Down\x94. The Flight Engineer generally did all the rest, which, thankfully, left all the tricky drills and procedures as his responsibility. ...Or was there a little more engine management and more gradual handling of the engines and descent?... On a normal decel/descent, the handling pilot would select ALT HOLD and then ask the Flight Engineer to reduce power to 18\xba TLA (Throttle Lever Angle). The speed would decay to 350 kts IAS (Indicated Air Speed) IAS HOLD was engaged and the descent flown at 350 kts IAS. The next power reduction (32\xba TLA) came when, still flying at 350 kts IAS, the Mach number reduced through M1.50. ...And I'm guessing the approach speeds were fairly high so hitting the touchdown zone was pretty important?... In terms of not running off the end of the runway, touching down in the correct spot was as important on Concorde as on other aircraft types. However, due to the geometry of Concorde on landing, the tail, engine pods and reverser buckets were already fairly close to the runway. Add in a \x93firm\x94 touchdown, or if the wings are not completely level, and ground clearance becomes marginal, so a prolonged flare and floated landing, with an increasing aircraft attitude, was not acceptable. The risk of a pod, tail or a reverser bucket scrape on Concorde was greater than on most conventional jet aircraft. ... So if things got out of shape a little, and a G/A was required, how do you handle what looks like 4 rockets on the wings and applying the right amount of power?...
|
|
| GBOAH 22nd Mar 2016, 00:26 permalink Post: 1934 |
Hello all, great thread!
I was just reading a few of my Concorde things and I have seen mention of several other manuals used by the crews other than the flying manual, like the performance manual, cruise control manual and navigation manual. I believe (perhaps wrongly!) that the performance manual contained the take-off data for weight, V speeds, noise time, TLA etc and I'm guessing the cruise manual had all the tables for cruise performances. Am I correct in my guess work? What kind of info was in the navigation manual as I think airfield charts were separate? Any info would be great! |
Page Links: Index Page


GlobalMouseY ( )
Returns the absolute y position of the mouse on the display.
Note: See also
GlobalMouseX
for absolute X position and
GetGlobalMouse
to get both
Example:


 A lot of people are feeling unsure in their current job and may be looking at a new path. We wanted to do something to welcome you into the Microsoft data platform community, gain some knowledge, and expand your network.
On April 29th we will be running 10 hours of content across US time zones. Sessions will begin at 9am Eastern Time, with a new session starting each hour
On April 30th we will be running 10 hours of content European time zones. Sessions will begin at 9am BST (UTC+1) with a new session starting each hour
Each day we’ll be having 10 50-minute sessions, each of which will introduce you to a new technology in the Microsoft Data Platform.
Sound good? Sound interesting? Let’s go!


Returns true if the value is an object and not an array or null.
Install with npm:
$ npm install isobject --save
Use is-plain-object if you want only objects that are created by the Object
constructor.
Install with npm:
$ npm install isobject
Install with bower
$ bower install isobject
var isObject = require('isobject');
True
All of the following return true
:
isObject({});
isObject(Object.create({}));
isObject(Object.create(Object.prototype));
isObject(Object.create(null));
isObject({});
isObject(new Foo);
isObject(/foo/);
False
All of the following return false
:
isObject();
isObject(function () {});
isObject(1);
isObject([]);
isObject(undefined);
isObject(null);
You might also be interested in these projects:
merge-deep: Recursively merge values in a javascript object. | homepage
Object
constructor. | homepagePull requests and stars are always welcome. For bugs and feature requests, please create an issue.
Generate readme and API documentation with verb:
$ npm install verb && npm run docs
Or, if verb is installed globally:
$ verb
Install dev dependencies:
$ npm install -d && npm test
Jon Schlinkert
Copyright © 2016, Jon Schlinkert. Released under the MIT license.
This file was generated by verb, v0.9.0, on April 25, 2016.


define(function(require, exports, module) {
var View = require('samsara/core/View');
var Transform = require('samsara/core/Transform');
var Surface = require('samsara/dom/Surface');
define(function(require, exports, module) {
var View = require('samsara/core/View');
var Transform = require('samsara/core/Transform');
var Surface = require('samsara/dom/Surface');
The Drawer is the side panel that gets exposed by translating the content.
Its initially offset to the left, and slides into place as the content is moved. Its speed is sped up from the content by the threshold parameter, which also defines the cutoff at which the drawer reaches its final position.
var Drawer = View.extend({
defaults : {
length : 250 // length of drawer surface
},
Executed on instantiation. Options are patched by the defaults if unspecified.
initialize : function(options){
Create a list which acts as the side menu
var content = '';
for (var i = 1; i <= 50; i++){
content += '
' + i + '
'
}
Create the drawer
var drawer = new Surface({
size : [options.length, false],
proportions : [false, 1],
origin : [1,0],
content : content,
classes : ['drawer'],
enableScroll: true
});
The drawer rotation and translation animation
var drawerTransform = this.input.map(function (data) {
var angle = Math.min(Math.PI/2 * (data.progress - 1), 0);
return Transform.thenMove(
Transform.rotateY(angle),
[data.value, 0, 0]
);
});
Animate the drawer’s opacity on close and open
var opacity = this.input.pluck('progress');
Create the render subtree
this.add({
transform: drawerTransform,
opacity : opacity
}).add(drawer);
}
});
module.exports = Drawer;
});


--- layout: default title: About bodyid: about cotype: about ---
We believe in working hard and getting it done. We also believe in enjoying life so we build tools that are both powerful and pleasant to use. All of our software is on GitHub and available under the MIT license.
We are available for consulting engagements. We'd love to hear about what you're working on: moc.suoitulos@trats
I started this company in 2006 as a technical consultancy to help clients improve development processes, build production environments, and prepare for launch-day. I've worked hard for small companies like ApproxiMatch and Status.net and large companies like Amdocs and Yellow Pages Group. I had a great time, met a lot of awesome people, and learned a ton.
But there's a blight in the software industry that I'm determined to do something about: overly complex technology that wastes people's time. Technology should reduce overall complexity, yet I have found case after case where the opposite is true. Where's the love?
So in 2008 I started focusing my attention on building great open source tools. I won't claim that our software is perfect, but it's built with care and attention down to even the smallest detail. I'm absolutely committed to ensuring it provides a net benefit to you and your projects. If you have any questions, problems, suggestions, or any other issue with any Solutious products, don't hesitate to contact me.
- Delano (@solutious.com)


An igxTooltip template
You can read more about configuring the igxTooltip component in the README or the official documentation.


Before Change
name = config.get("name", "GPM Desktop Player")
address = config.get("address")
if address is None:
_LOGGER.error("Missing address in config")
return False
add_devices([GPMDP(name, address, create_connection)])
class GPMDP(MediaPlayerDevice):
After Change
def setup_platform(hass, config, add_devices_callback, discovery_info=None):
Setup the GPMDP platform.
codeconfig = _load_config(hass.config.path(GPMDP_CONFIG_FILE))
if len(codeconfig):
code = codeconfig.get("CODE")
elif discovery_info is not None:
if "gpmdp" in _CONFIGURING:
return
code = None
else:
code = None
setup_gpmdp(hass, config, code, add_devices_callback)
class GPMDP(MediaPlayerDevice):


Assgnment On List
- Background skill
- Unix command
- VIM text editor
- HTML
- Minimal pange
- Heading
- Elements
- Lists
- Unordered
- ordered
- Defination
- Nested
- Links
- Absolute
- Relative
- Images
- css
- Anatomy
- Basic Selectors
- Element
- Class
- Group
- The dom
- Advanced Selectors
- Box Model
- Programming
- Python
- Java Script
- Database
- Flat File
- Relational


Thomas leads Appalachian St. to 56-27 win over Georgia St.
ATLANTA (AP) — Zac Thomas passed for four touchdowns and ran for another, Darrynton Evans had 131 yards rushing and a score, and Appalachian State beat Georgia State 56-27 on Saturday night.
Thomas was 19-of-31 passing for 256 yards with an interception and added 52 yards rushing on six carries. Corey Sutton had 173 yards receiving and three touchdowns — both career highs — for Appalachian State (9-1, 5-1 Sun Belt Conference).
Destin Coates scored on a 67-yard run and, less than a minute later, Chris Bacon’s 34-yard pick-6 gave Georgia State (6-4, 3-3) a 21-7 lead with 1:38 left in the first quarter but the Mountaineers scored the next 49 points.
Sutton answered with a 20-yard TD catch 89 seconds later, Thomas scored on a 3-yard run and Shaun Jolly returned an interception 30 yards for a touchdown to give Appalachian State its first lead before a 1-yard touchdown run by Evans made it 35-21 at halftime.
Raykwon Anderson scored on a 5-yard run to give the Mountaineers a 56-21 lead with 11 minutes left in the game.


{{ codemirror.include_codemirror() }}
{{ title }}
Analyze
Profiles
Modules
Profile name:
add
Analysis Profiles
{% for profile in profiles %}
{{profile}}
delete
mode_edit
{% endfor %}
Please Confirm
Are you sure to proceed?


title
contents here
after clicking 'title' bar,
ctrl+b " to split pane vertically
crtl+b % to split pane horizontally
ctrl+b d to close pane


AtomRadioButtonList is inherited from AtomListBox, but by default it displays radio buttons and automatically provides selected value property. You can control layout property by giving it a table layout, which accepts number of columns, cell width and cell height parameters. In addition, you can also control your layout by overriding atom-check-box-list css styles.


Feather
Just breathe slow and steady, in and out.
Spacebar or Left Click for floating
•
M to turn on the sound
Feather is a mini game inspired by Celeste. It's purpose is to visualize the floating feather meditation technique, a technique designed to relax and reduce anxiety.
 When you inhale, the feather rises a few inches, when you exhale, it descends. Keeping a simple count of four for the inhalation, two for a pause and four for the exhalation, you will get into a rythm that will help you when you have trouble breathing, or in the middle of a panic attack.
The game helps you visualize the feather. Whenever you press the spacebar, the feather rises, when you release it, it will descend. Just keep the feather inside the white box and breathe while you play.
Explanation


Mastaba Abatsam
Home
Page
Mastaba
mas, ta, ba
I like.
yes
no
one
two
last
one
two
last
one
two
last
one
two
last
one
two
last


Source code for lingvo.tasks.mt.layers
# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Lingvo MT layers.
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from six.moves import range
import tensorflow as tf
from lingvo.core import base_layer
from lingvo.core import layers
from lingvo.core import layers_with_attention
[docs]class TransformerStack(base_layer.LayerBase):
"""Stacked self- multi-head attention and fully connected layers.
With optional layer normalization applied to the final output.
See 'Attention Is All You Need' https://arxiv.org/abs/1706.03762
for details.
"""
[docs] @classmethod
def Params(cls):
"""Configs for TransformerStack."""
p = super(TransformerStack, cls).Params()
# Transformer related
p.Define('model_dim', 1024, 'Characteristic depth (dimension).')
p.Define('num_transformer_layers', 6, 'Number of transformer layers.')
p.Define('transformer_tpl', layers_with_attention.TransformerLayer.Params(),
'TransformerLayer params tpl.')
p.Define('random_seed', None,
'If set, this decides the random seed to apply in various random'
' ops such that this encoder is deterministic. Set this'
' random_seed only for unittests.')
p.Define('ln_tpl', layers.LayerNorm.Params(), 'Layer norm default params')
p.Define('ln_output', False,
'If set, layer normalization is applied to the final output'
' of the encoder transformer stack.')
p.Define('is_transparent', False,
'If set, outputs a merger of embeddings and layer outputs.')
p.Define('num_transparent_outputs', 6, 'Number of transparent outputs.')
p.Define(
'transparent_merger_tpl',
layers.WeightedSumLayer.Params().Set(add_weight_summaries=True),
'Merger op for layer outputs.')
p.Define('packed_input', False,
'If True, assumes multiple training samples per input.')
p.transformer_tpl.tr_atten_tpl.num_attention_heads = 8
p.transformer_tpl.tr_fflayer_tpl.hidden_dim = 8192
return p
@base_layer.initializer
def __init__(self, params):
super(TransformerStack, self).__init__(params)
p = self.params
with tf.variable_scope(p.name):
# Add transformer layers.
transformer_layer_params = []
for i in range(p.num_transformer_layers):
params = p.transformer_tpl.Copy()
params.name = 'trans_%d' % (i)
params.source_dim = p.model_dim
params.packed_input = p.packed_input
params.random_seed = p.random_seed
transformer_layer_params.append(params)
self.CreateChildren('trans', transformer_layer_params)
# Initialize TransformerStack output layer norm
if p.ln_output:
params = p.ln_tpl.Copy()
# Keeping historic 'enc_out_ln' name for checkpoint compatibility.
params.name = 'enc_out_ln'
params.input_dim = p.model_dim
self.CreateChild('layer_norm_out', params)
if p.is_transparent:
transparent_params = []
if not p.num_transparent_outputs:
raise ValueError('num_transparent_outputs should be greater than 0.')
for i in range(p.num_transparent_outputs):
transparent_param = p.transparent_merger_tpl.Copy()
transparent_param.name = 'transparent_%d' % i
transparent_param.num_sources = 1 + p.num_transformer_layers
transparent_params.append(transparent_param)
self.CreateChildren('transparent_merger', transparent_params)
[docs] def FProp(self, theta, transformer_input, paddings, src_segment_id=None):
"""Transforms source sequence of Tensors with Transformers layers.
Args:
theta: A `.NestedMap` object containing weights' values of this
layer and its children layers.
transformer_input: A sequence of input Tensors of [time, batch, dim]
shape.
paddings: A sequence of 0s and 1s indicating input paddings of
[time, batch] shape.
src_segment_id: A sequence of ints indicating segment ids of
[time, batch] shape.
Returns:
(outputs, out_paddings, segment_ids) tuple. `outputs` is of the shape
[time, batch, depth], and `out_paddings` has shape [time, batch]. If
is_transparent is True, can return a list of num_transformer_layers
tensors of shape [time, batch, depth] if `p.is_eval` is False, and a
[time, batch, depth, num_transparent_outputs] tensor if `p.is_eval` is
True. If packed_input is True, also returns segment_id, otherwise returns
None.
"""
p = self.params
if p.packed_input:
assert src_segment_id is not None, ('Need to specify src_segment_id if '
'packed input is supported.')
outputs_list = [transformer_input]
with tf.name_scope(p.name):
for i, transformer_l in enumerate(self.trans):
# For encoder, keys, values and queries are the same
transformer_output, _ = transformer_l.FProp(
theta.trans[i],
transformer_input,
paddings,
source_segment_id=src_segment_id)
transformer_input = transformer_output
outputs_list.append(transformer_output)
if p.ln_output:
transformer_output = self.layer_norm_out.FProp(theta.layer_norm_out,
transformer_output)
# When is_transparent is set, it outputs a list of tensors during
# training and the stacked tensors otherwise. This dual behavior is meant
# to avoid excessive memory usage during training (which was prohibiting
# training on TPUs), and simplify the beam search interface.
if p.is_transparent:
transformer_output = []
for i in range(p.num_transparent_outputs):
merged_outputs = self.transparent_merger[i].FProp(
theta.transparent_merger[i], outputs_list)
transformer_output.append(merged_outputs)
if p.is_eval:
transformer_output = tf.stack(transformer_output, 3)
return transformer_output, paddings, src_segment_id


--- layout: post title: 'DistNet introducation: A powerful and scalable CNN' date: 2014-02-21 17:24:50.000000000 -08:00 type: post published: true status: publish categories: - Programming tags: [] meta: _edit_last: '53481042' _publicize_pending: '1' author: login: allenbo email<EMAIL_ADDRESS>display_name: allenbo first_name: '' last_name: '' ---
The project, Yury and me were being doing for last semester is a distributed convolutional neural network (DCNN).
We always use a regular convolutional neural network to detect objects in images, since the convolutional layers in the network would reserve the local features of the image and the detection of images are heavily relied on the those features. I will use a paragraph to explain the way that the convolutional layers work and bring the context to this.
A convolutional layer, not like a fully connected layer, is locally connected to the input. For fully connected layer, the computation model would be like this:
We can see the the input are fully connected to the hidden layer, and each value of input will activate the neuron of hidden layer and affect the output of the hidden layer, which is specifically the input of the output layer. However the convolutional layer, doing convolution operation while being activated, use another computation model like this:
Only part of the input will activate the neuron of the convolutional layer, and the computation operation is simply a sum of the output of activating patch.
So people, who are doing the image recognition, are truly using the convolutional neural network and trying to improve the accuracy of the model. However, there are some practical limitation of the research that most people are doing. The bigger the model is, theoretically, the better the result will be. Along with the growth of the model, we need more time and GPU memory to train the model. At some case, however, model is too big to fit on single GPU memory ( lot of GPUs now have 6GB memory space). Since the input and output and gradient for each layer is getting too large when the model is becoming bigger and bigger, we need some distributed convolutional neural network to, not only reduce the size of memory requirement on each GPU, but accelerate the training procedure. And that is what we are doing for the final project.
[Cuda]: NVIDIA GPU programming toolkits
[PyCuda]: a python interface for cuda programming
[NumPy]: a python numeric computation library
[CudaConv]: a very efficient implementation of convolution operation on GPU
[OpenMPI]: a message passing library that follow MPI standard
The cuda API is provided by NVIDIA, giving us a c interface to program with GPU. pycuda is a python wrap of cuda, providing several predefined function to do the heavily used operations. However, pycuda doesn’t fulfill every requirement we need, so we have to do extend the pycuda library to support some critical operations. I will talk about that later. NumPy is a famous numeric python library, we hardly use it in the project. And CudaConv is an efficient implementation of convolution-based operations for GPU platform, developed by Alex Krizhevsky.
Since we have a very good cuda kernel library, we don’t have to recreate the wheel and simple accommodate my modification and functionality we want the distributed model have with the original one. Another trivial thing we have to do is extending PyCuda library for our use. PyCuda library is designed generally for every GPU-based task and is not quite comprehensive for this project. We use the class GPUArray out of PyCuda and extend a lot of operations and functionalities, like creating a brand-new GPUArray while copying noncontiguous memory.
In order to develop a fast distributed network, there are two basic problems that we have to solve. One is developing a highly optimized GPU kernel for all major computational operations and thanks to Alex we had one already. So the only problem left is developing a scheme to distribute the computations and managing the communication between multiple GPUs.
Given Alex’s implementation, we are able to build a single GPU system to train neural networks and that’s one of the features of distnet. Even capable of working on multiple GPUs, distnet still can use one GPU to train a network if you currently don’t have more than one GPU.
The way we distribute the computation is aiming at reducing the communication overhead. There are several ways we can use to distribute the model. They are batch-based data parallelism, image-based data parallelism and model parallelism. Since we are using image-based data parallelism in the distnet, I will briefly reference the other two parallelism and the disadvantages of them.
Batch-based data parallelism
This approach distributes the computation by reducing the batch size of single GPU. Due to heavy communication overhead, in order to get a acceptable speedup, we have to use stochastic SGD to train the model. Since the model doesn’t synchronize every batch, we can expect an equivalent model to the nonparalleled one.
Model parallelism
This approach splits the filters across multiple GPUs in order to distribute the computation. That’s when the first convolution layer has 96 filters, it dispatches the first 48 filters at GPU 0 and another 48 filters at GPU 1, if we happen to have two. Since the input of second convolution layer, which is the output of the first convolution layer, has to be the complete copy of the 96 filters, we have to communicate across GPUs every layer. Again, the communication overhead is unacceptable.
Image-based data parallelism
The way we are using in distnet is image-based data parallelism, which is cutting the image into squares and storing the portion locally on each GPU when dispatching the computation. Since the convolutional operation is computed locally, every GPU doesn’t have to acquire too much data from other GPU, which ensures a limited and acceptable communication overhead. The partition looks like this.
The figure shows the way we partition the image. Assume we have 4 GPUs, the image is partitioned into 4 pieces in square and stored on those GPUs. Each GPU is responsible for the computation of it’s partition and performs independently. But all together 4 GPUs will work on the same and complete minibatch.
Since the input(image) is partitioned into 4 pieces, the output of this layer, basically convolutional layer, will be partitioned into 4 pieces, just like input. Things would be easy if GPUs don’t have to communicate with each other while computing, however, even though, most of the output need the input only stored locally, the boundary output would require data from other GPUs. As figure above shows, the red patch in The figure shows the way we partition the image. Assume we have 4 GPUs, the image is partitioned into 4 pieces in square and stored on those GPUs. Each GPU is responsible for the computation of it’s partition and performs independently. But all together 4 GPUs will work on the same and complete minibatch.
Since the input(image) is partitioned into 4 pieces, the output of this layer, basically convolutional layer, will be partitioned into 4 pieces, just like input. Things would be easy if GPUs don’t have to communicate with each other while computing, however, even though most of the output need the input only stored locally, the boundary output would require data from other GPUs. As figure above shows, the red patch on the bigger cube (input) indicates the communication.
Communication is nontrivial. The amount of data needed from different GPUs vary from GPUs and layers. In order to keep track of the entire arrangement of input (an instance of distributed array), we reserve some metadata in every dist-array on each GPUs. Since we build the communication on the top of MPI library, we will see the same instance in different processor represent the same input and output with different metadata. The metadata we use in dist-array is an object area. This object will claim the ownership of some area of the dist-array. And of course, the combination of all area objects of the same dist-array instance will be the entire area of the instance.
Since we have the area metadata, we could use a hash table to map GPU to area of the dist-array and make it identical at each instance. When we read some region across server GPUs, we could easily know which GPUs we should go to and which part of data stored at corresponding GPU we should fetch.
For example, we partition a 2d 10 x 10 array to 4 GPUs illustrated by the figure.
After that, the metadata on each GPU would be ( inclusive)
|
GPU_ID |
Area |
|
0 |
from(0, 0) to (4, 4) |
|
1 |
from(0, 5) to (4, 9) |
|
2 |
from(5, 0) to (9, 4) |
|
3 |
from(5, 5) to (9, 9) |
When we fetch a region from (0, 0) to (6, 6), we could easily iterate the key-value pair in this hashtable find out that we could get data in the area from(0, 0) to(4, 4) locally. Meanwhile we have to talk to GPU1 to get data in the area from (0, 5) to (4, 6). The requests would be (inclusive)
|
GPU_ID |
Area |
|
0 |
from(0, 0) to (4,4) ( locally) |
|
1 |
from(0, 5) to (4, 6) |
|
2 |
from(5, 0) to (6, 4) |
|
3 |
from(5, 5) to (6, 6) |
Every time we perform a read operation we will have to communicate with other potential GPUs twice. One is to exchange the metadata and the other is to send and receive the real data.
Now given a region we know how to fetch the complete data locally and remotely, we still have to figure out which region we want to read at each operation. Sometimes the width and height of the complete input are not even, we can’t divide evenly into squares. There are some tricks in partitioning, however, the basic idea is push the extra point to the last tile of the squares. Based on this refinement, when we are trying to figure out the area of data for corresponding GPU, we not only calculate with parameters of input, but double check with parameters of output. That’s, after get a potential region with filter size, padding, stride, image size, we check if the output of this region is exactly the same one. If it not, we do some fine-tuning.
Since there are overlappings in read operations, when we write back to that region, we have to reduce server pieces of data to the overlapping region. The is automatically dealt with by the dist-array instance.
In order to provide a uniformed interface of array operations, we redefine the array interfaces in the original convolutional neural network and accommodate the dist-array with the same interface.
In practice, we find some refinements extremely helpful to improve the overall performance.
Memory Cache
The underlying library we use to send data is OpenMPI. MPI is a message passing standard that is well designed for sharing data across hosts. OpenMPI can satisfy most needs, however, it doesn’t support strided buffer when sending messages. The buffer we are trying to send out has to be contiguous and the buffer we prepare to receive the data has to be contiguous. When we are reading a region that covers all GPUs, the other GPUs have to copy the noncontiguous buffer that we need and activate MPI library to send out. We have to prepare some buffers to receive those data and assemble them together to get a contiguous buffer as the input of the computation.
This unexpected overhead takes way many time than we thought to copy data, create a new instance and write back to the dist-array. As we looked deep into this overhead, we found that as long as we know the shape of incoming buffers and outsending buffers at the first place, we can create other buffers as a cache inside the dist-array and never have to recreate them. After we improve the strided copy function, we save some time.
Shared Memory
Although the cached memory helps, it’s tradeoff between memory space and runtime. We waste a bulk of memory space to do that. Those cached memory doesn't consume too many space when the model is not to big. However, with the model size growing up dramatically, we have to bring the unsettled problem into list.
In order to remain high speed of data passing over GPUs, we have to use cache. This is the rule we have to follow. Fortunately those caches are redundant over the time. When first convolutional layer are passing data across GPUs, second layer doesn't have to hold the cache and are allow to free the memory. So we create a shared buffer to hold the shared memory and reduce the usage of GPU memory. It reduce 50% of memory in dist-array.
We test DistNet in several settings and are thrilled to claim that DistNet accelerate the training procedure by a magnitude with multiple GPUs. The model we are using is Alex’s model in the paper ImageNet Classiﬁcation with Deep Convolutional Neural Networks.


--- layout: testimonials ---
At the Ultimate Clean we only have one goal. To provide our clients with the best possible cleaning service.
The most wonderful job done at our place, on this first experience with Brett he rescheduled work to fit in an urgent job, fantastic & instant communications, great care taken with a range of carpets & rugs, some vintage & antique (but willing to recommend other more specialist rug cleaners). A rare time when I have rung a service & had 100% confidence in them from the first phonecall, so much so that trusted them to lock & leave my house whilst I returned to the country. Got back to find everything optimal, carpet & rugs have never looked so good, everything locked back up exactly as per instruction with complete liason at every step. Very happy. Thank you so much Brett! Susan Bibra Lake
Brett Prosser from The Ultimate Clean is the epitome of professionalism, providing utmost care and attention to any task at hand. I have referred to Brett for his expert opionion many times and have always received a highly professional and honest service. Currently, our couch has been restored to its original glory thanks to Brett, we didn't even remember its original colour! Don't use anyone else. Sharee, South Perth
These guys did a great job on our old carpets. I didn't think there would be a lot they could do but they managed to get so many old stains out making it look fresh and lovely :) they were recommended by a family member and i will pass the recommendation on to any one who il looking for a professional clean. Haylee, Riverton
Brett Prosser from The Ultimate Clean is the epitome of professionalism, providing utmost care and attention to any task at hand. I have referred to Brett for his expert opionion many times and have always received a highly professional and honest service. Currently, our couch has been restored to its original glory thanks to Brett, we didn't even remember its original colour! Don't use anyone else. Sharee, South Perth
The Ultimate Clean are professional in every sense of the word. From organising a time that fits in with my busy house to the amazing results Brett produced with our and rugs then to the follow-up service. We highly recommend your business to anyone looking for the best possible results. Great job guys! Nicole, City Beach
Knew our sofa & chairs would be a challenge after a previous clean that was more than a disaster but after Brett came today, have to say his company has the right name. The Ultimate Clean it was - he is so professional & a perfectionist just like me - beyond happy. Gail, Victoria Park


Toggle navigation
About
Maps
Contact
Log In
Sign Up
Clubs
Add Club
Swimming Clubs
Athletic Clubs
Walking Clubs
More Clubs...
Map Route
Home
Map Route
Portlethen Moss
Torry Battery
Newtonhill/Skateraw Walk
North Kincardine Viewpoints
Stonehaven
Routes
North Kincardineshire Viewpoints
Portlethen Moss Walk
North Sea Trail and Creel Walk
Newtonhill/Skateraw Walk
Mounth, Post and Templar Walk
All Walks


Failing private school in Exeter is visited for third time this year by Ofsted
Some windows leading onto the roof were not secured consistently and a hazardous cleaning agent was found unlocked and accessible to pupils
by Anita MerrittAn independent school which costs up to £3,830 per term to attend is being visited by Ofsted inspectors for the third time this year after education and safety concerns were exposed.
Bramdean School in Heavitree was inspected in January and it was rated as requiring improvement.
Inspectors reported there had been a decline in the quality of education since the previous inspection in 2015 when it was rated good overall.
Ofsted carried out an emergency inspection in May following a complaint that raised ‘wider concerns’ about the school. It related to pupils’ safety and well-being on the school grounds and premises.
Inspectors concluded some aspects of the premises or facilities had not been maintained well enough since the last inspection and pupils were being exposed to avoidable risks.
These included some windows leading onto the roof were not secured consistently and a hazardous cleaning agent was found unlocked and accessible to pupils.
The report said: “The danger posed through insecure high-level windows is not acceptable."
Other concerns included:
- Some breaches in safety on the premises have gone unnoticed
- On occasions where leaders have not been quick enough to either identify or respond to health and safety issues, pupils are put at an unnecessarily increased risk of harm.
However, pupils told inspectors they feel safe and well cared for at school, and they had confidence in staff.
Inspectors concluded the school does not meet the independent school standards for the welfare, health and safety of pupils, for premises and accommodation at schools, and quality of leadership in and management.
The concerns highlighted In January included leaders do not identify weaknesses quickly enough, and senior leaders do not have an in-depth view of the quality of education and do not hold teachers stringently to account.
Teachers’ expectations of pupils were said to be not consistently high, and leaders were not providing high-quality books and other literature for pupils to read and enjoy, especially for older pupils.
At Bramdean, which was founded in 1901, Diane Stoneman is the proprietor as well as the headteacher. It does not have a governing body.
At the January inspection the school had 97 pupils aged between three to 18 years old. It was noted the headteacher does not use ‘checks and balances’ to audit or review key aspects of the school’s or her own performance.
Inspectors said: “This contributes towards some inaccuracies in self-evaluation and impedes the rate of improvement in the school."
This week inspectors have been back at the school and the outcome of that visit will then be published at a later date.
A spokesperson for Ofsted said: “We can confirm that we have inspected this school, but we will say more about it when the inspection report is published.”
The school were approached for a comment.
If you have a story to share, please contact Devon Live reporter Anita Merritt on 01392 346755 or email anita.merritt@reachplc.com


Velle
Fill in your trip details to get started!
Itinerary Name
Destination
Date of Departure
Date of Arrival
Number of members
4
6
8


Here's some testing of cascading style sheets.
Also known as creme patissiere, this is the traditional filling for fresh fruit tarts. It is also used to fill choux puffs.
2 cups milk
1 vanilla bean
6 egg yolks
175g castor sugar
50g cornflour
Scald milk with vanilla bean. Beat egg yolks with cornflour until thick. Pour in milk and whisk until quite smooth.
Also known as legacy patissiere, this is the choux puffs.
to contact the author, email me on<EMAIL_ADDRESS>to download all my recipes in acrobat format, click here
for other recipes, see biodynamic recipes from http://www.biodyn.nu