X-Road: External Load Balancer Installation Guide
Version: 1.17 Doc. ID: IG-XLB
| Date | Version | Description | Author |
|---|---|---|---|
| 22.3.2017 | 1.0 | Initial version | Jarkko Hyöty, Olli Lindgren |
| 27.4.2017 | 1.1 | Added slave node user group instructions | Tatu Repo |
| 15.6.2017 | 1.2 | Added health check interface maintenance mode | Tatu Repo |
| 21.6.2017 | 1.3 | Added chapter 7 on upgrading the security server cluster | Olli Lindgren |
| 02.03.2018 | 1.4 | Added uniform terms and conditions reference | Tatu Repo |
| 15.11.2018 | 1.5 | Updates for Ubuntu 18.04 support | Jarkko Hyöty |
| 20.12.2018 | 1.6 | Update upgrade instructions | Jarkko Hyöty |
| 11.09.2019 | 1.7 | Remove Ubuntu 14.04 support | Jarkko Hyöty |
| 08.10.2020 | 1.8 | Added notes about API keys and caching | Janne Mattila |
| 19.10.2020 | 1.9 | Remove xroad-jetty and nginx mentions and add xroad-proxy-ui-api | Caro Hautamäki |
| 19.10.2020 | 1.10 | Added information about management REST API permissions | Petteri Kivimäki |
| 23.12.2020 | 1.11 | Updates for Ubuntu 20.04 support | Jarkko Hyöty |
| 02.07.2021 | 1.12 | Updates for state sync | Jarkko Hyöty |
| 25.08.2021 | 1.13 | Update X-Road references from version 6 to 7 | Caro Hautamäki |
| 17.09.2021 | 1.14 | Add note about the proxy health check now also checking global conf validity | Caro Hautamäki |
| 17.06.2022 | 1.15 | Replace the word "replica" with "secondary" | Petteri Kivimäki |
| 26.09.2022 | 1.16 | Remove Ubuntu 18.04 support | Andres Rosenthal |
| 01.03.2023 | 1.17 | Updates for user groups in secondary nodes | Petteri Kivimäki |
Table of Contents
- License
- 1. Introduction
- 2. Overview
- 3. X-Road Installation and configuration
- 4. Database replication setup
- 5. Configuring data replication with rsync over SSH
- 6. Verifying the setup
- 7. Upgrading a clustered X-Road security server installation
License
This document is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/.
1. Introduction
1.1 Target Audience
The intended audience of this installation guide are the X-Road security server administrators responsible for installing and configuring X-Road security servers to use external load balancing. The document is intended for readers with a good knowledge of Linux server management, computer networks, database administration, clustered environments and the X-Road functioning principles.
1.2 Terms and abbreviations
See X-Road terms and abbreviations documentation [TA-TERMS].
1.3 References
| Document Id | Document |
|---|---|
| [SS-CLUSTER] | Readme: Security server cluster setup with Ansible |
| [IG-SS] | X-Road: Security Server Installation Guide |
| [UG-SS] | X-Road 7 Security Server User Guide |
| [TA-TERMS] | X-Road Terms and Abbreviations |
2. Overview
This document describes the external load balancing support features implemented by X-Road and the steps necessary to configure security servers to run as a cluster where each node has an identical configuration, including their keys and certificates. X-Road security server configuration changes are handled by a single primary server and one or more secondary servers.
Chapter 3. X-Road Installation and configuration describes the installation on a high level and as a whole. The later chapters cover the details of the different parts of the installation and configuration. The last chapter briefly describes how the configuration can be verified.
2.1 Goals and assumptions
The primary goal of the load balancing support is, as the name suggests, load balancing, not fault tolerance. A clustered environment increases fault tolerance but some X-Road messages can still be lost if a security server node fails.
The implementation does not include a load balancer component. It should be possible to use any external load balancer component that supports HTTP-based health checks for the nodes and load balancing at the TCP level (eg. haproxy, nginx, AWS ALB or Classic Load Balancer, or a hardware appliance). A health check service is provided for monitoring a node's status, this is described in more detail in section 3.4 Health check service configuration
The load balancing support is implemented with a few assumptions about the environment that users should be aware of. Carefully consider these assumptions before deciding if the supported features are suitable for your needs.
2.1.1 Basic assumptions about the load balanced environment
- Adding or removing nodes to or from the cluster is infrequent. New nodes need to be added manually and this takes some time.
- Changes to the configuration files are relatively infrequent and some downtime in ability to propagate the changes can be tolerated.
- The cluster uses a primary-secondary model and the configuration primary is not replicated.
2.1.2 Consequences of the selected implementation model
- Changes to the
serverconf
database, authorization and signing keys are applied via the configuration primary, which is a member of the cluster. The replication is one-way from primary to secondaries and the secondaries should treat the configuration as read-only. - The cluster nodes can continue operation if the primary fails but the configuration can not be changed until:
- the primary comes back online, or
- some other node is manually promoted to be the primary.
- If a node fails, the messages being processed by that node are lost.
- It is the responsibility of the load balancer component to detect the failure and route further messages to other nodes. Because there potentially is some delay before the failure is noticed, some messages might be lost due to the delay.
- Recovering lost messages is not supported.
- Configuration updates are asynchronous and the cluster state is eventually consistent.
- If the primary node fails or communication is interrupted during a configuration update, each secondary should have a valid configuration, but the cluster state can be inconsistent (some members might have the old configuration while some might have received all the changes).
2.2 Communication with external servers and services: The cluster from the point of view of a client or service
When external security servers communicate with the cluster, they see only the public IP address of the cluster which is registered to the global configuration as the security server address. From the caller point of view, this case is analogous to making a request to a single security server.
When a security server makes a request to an external server (security server, OCSP, TSA or a central server), the external server sees only the public IP address. Note that depending on the configuration, the public IP address might be different from the one used in the previous scenario. It should also be noted that the security servers will independently make requests to OCSP and TSA services as well as to the central server to fetch the global configuration as needed.
2.3 State replication from the primary to the secondaries
2.3.1 Replicated state
<IP_ADDRESS> serverconf
database replication
| Data | Replication | Replication method |
|---|---|---|
| serverconf database | replication required | PostgreSQL streaming replication (Hot standby) |
The serverconf database replication is done using streaming replication with hot standby. Note that PostgreSQL replication is all-or-nothing: it is not possible to exclude databases from the replication. This is why the replicated serverconf and non-replicated messagelog databases need to be separated to different instances.
<IP_ADDRESS> Key configuration and software token replication from /etc/xroad/signer/*
| Data | Replication | Replication method |
|---|---|---|
| keyconf and the software token | replicated |
rsync+ssh (scheduled) |
Previously, any external modification to /etc/xroad/signer/keyconf.xml
was overwritten by the X-Road signer process if
it was running. Therefore, replicating the signer configuration without service disruptions would have required taking the
cluster members offline one-by-one. The load balancing support adds the possibility for external modifications to the
keyconf.xml to be applied on secondary nodes without service disruptions. The actual state replication is done with a scheduled
rsync over ssh. This might take a few minutes so a slight delay in propagating the changes must be tolerated by the
clustered environment. A small delay should usually cause no problems as new keys and certificates are unlikely to be used
immediately for X-Road messaging. Changes to the configuration are also usually relatively infrequent. These were one of
the basic assumptions about the environment.
Users should make sure this holds true for them.
The secondary nodes use the keyconf.xml
in read-only mode: no changes made from the admin UI are persisted to disk. secondaries
reload the configuration from disk periodically and apply the changes to their running in-memory configuration.
<IP_ADDRESS> Other server configuration parameters from /etc/xroad/*
| Data | Replication | Replication method |
|---|---|---|
| other server configuration parameters | replicated |
rsync+ssh (scheduled) |
The following configurations are excluded from replication:
-
db.properties
(node-specific) -
postgresql/*
(node-specific keys and certs) -
globalconf/
(syncing globalconf could conflict withconfclient
) -
conf.d/node.ini
(specifies node type: primary or secondary)
2.3.2 Non-replicated state
<IP_ADDRESS> messagelog
database
The messagelog database is not replicated. Each node has its own separate messagelog database. However, in order to support PostgreSQL streaming replication (hot standby mode) for the serverconf data, the serverconf and messagelog databases must be separated. This requires modifications to the installation (a separate PostgreSQL instance is needed for the messagelog database) and has some implications on the security server resource requirements as a separate instance uses some memory.
<IP_ADDRESS> OCSP responses from /var/cache/xroad/
The OCSP responses are currently not replicated. Replicating them could make the cluster more fault tolerant but the replication cannot simultaneously create a single point of failure. A distributed cache could be used for the responses.
3. X-Road Installation and configuration
This chapter details the complete installation on a high level, with links to other chapters that go into the details.
You can set up the cluster manually, or use the provided Ansible playbook [SS-CLUSTER] if it suits your purposes.
3.1 Prerequisites
In order to properly set up the data replication, the secondary nodes must be able to connect to:
- the primary server using SSH (tcp port 22), and
- the primary
serverconf
database (e.g. tcp port 5433).
3.2 primary installation
-
Install the X-Road security server packages using the normal installation procedure or use an existing standalone node.
-
Stop the xroad services.
-
Create a separate PostgreSQL instance for the
serverconf
database (see section 4. Database replication setup for details). -
Change
/etc/xroad/db.properties
to point to the separate database instance:-
serverconf.hibernate.connection.url
: Change the url port number from5432
to5433
(or the port you specified)
-
-
If you are using an already configured server as the primary, the existing configuration was replicated to the secondaries in step 3. Otherwise, proceed to configure the primary server: install the configuration anchor, set up basic information, create authentication and signing keys and so on. See the security server installation guide [IG-SS] for help with the basic setup.
-
Set up the configuration file replication, see section 5. Configuring data replication with rsync over SSH
- Additionally,
rssh
shell can be used to restrict secondary access further, but note that it is not available on RHEL.
- Additionally,
-
Configure the node type as
master
in/etc/xroad/conf.d/node.ini
:[node] type=master
Change the owner and group of the file to
xroad:xroad
if it is not already. -
Disable support for client-side pooled connections (HTTP connection persistence) in
/etc/xroad/conf.d/local.ini
- Because the load balancing works at TCP level, disabling persistent HTTP connections is recommended so that the load balancer can evenly distribute the traffic.
[proxy] server-support-clients-pooled-connections=false
- Because the load balancing works at TCP level, disabling persistent HTTP connections is recommended so that the load balancer can evenly distribute the traffic.
-
Start the X-Road services.
3.3 Secondary installation
-
Install security server packages using the normal installation procedure. Alternatively you can also install only the packages required for secondary nodes.
xroad-proxy-ui-api
package can be omitted, but the admin graphical user interface (which is provided by this package) can be handy for diagnostics. It should be noted that changing a secondary's configuration via the admin gui is not possible (except entering token PIN). -
Stop the
xroad
services. -
Create a separate PostgreSQL instance for the serverconf database (see section 4. Database replication setup for details)
-
Change
/etc/xroad/db.properties
to point to the separate database instance and change password to match the one defined in the primary database (the password is part of the data that is replicated to the secondaries).-
serverconf.hibernate.connection.url
: Change the url port number from5432
to5433
(or the port you specified) -
serverconf.hibernate.connection.password
: Change to match the primary db's password (in plaintext).
-
-
Set up SSH between the primary and the secondary (the secondary must be able to access
/etc/xroad
via ssh)- Create an SSH keypair for
xroad
user and copy the public key to authorized keys of the primary node (/home/xroad-slave/.ssh/authorized_keys
)
- Create an SSH keypair for
-
Set up state synchronization using rsync+ssh. See section 5. Configuring data replication with rsync over SSH
- Make the initial synchronization between the primary and the secondary.
rsync -e ssh -avz --delete --exclude db.properties --exclude "/postgresql" --exclude "/conf.d/node.ini" --exclude "/gpghome" xroad-slave@<primary>:/etc/xroad/ /etc/xroad/
Where
<primary>
is the primary server's DNS or IP address. -
Configure the node type as
slave
in/etc/xroad/conf.d/node.ini
.[node] type=slave
Change the owner and group of the file to
xroad:xroad
if it is not already. -
Start the X-Road services.
-
If you wish to use the secondary security server's admin user interface, you need to implement additional user group restrictions. As noted in step 1, changes to the secondary node security server configuration must not be made through its admin user interface, as any such changes would be overwritten by the replication. To disable UI editing privileges for all users, remove the following user groups from the secondary security server:
xroad-security-officer
xroad-registration-officer
xroad-service-administrator
xroad-system-administrator
Note:
xroad-security-officer
should remain, otherwise you will not be able to enter token PIN codes.After removing these groups, the super user created during the security server installation is a member of two UI privilege groups:
xroad-securityserver-observer
andxroad-security-officer
. These groups allow read-only access to the admin user interface and provide a safe way to use the UI for checking the configuration status of the secondary security server. In addition, the groups allow the user to enter the token PIN code. Since admin UI users are UNIX users that are members of specific privilege groups, more users can be added to the groups as necessary. Security server installation scripts detect the node type of existing installations and modify user group creation accordingly. Instead, version upgrades do not overwrite or modify this configuration during security server updates.For more information on user groups and their effect on admin user interface privileges in the security server, see the Security Server User Guide [UG-SS].
Also, the secondary security server's management REST API can be used to read the secondary's configuration. However, modifying the secondary's configuration using the management REST API is blocked. API keys are replicated from the primary to the secondaries, and the keys that are associated with the
xroad-securityserver-observer
role have read-only access to the secondary. In addition, the keys that are associated with thexroad-securityserver-observer
andxroad-security-officer
roles, are able to enter token PIN codes. The keys that are not associated with thexroad-securityserver-observer
role, don't have any access to the secondary. See next item for more details.For more information on the management REST API, see the Security Server User Guide [UG-SS].
-
Note about API keys and caching. If API keys have been created for primary node, those keys are replicated to secondaries, like everything else from
serverconf
database is. The keys that are associated with thexroad-securityserver-observer
role have read-only access to the secondary. Instead, the keys that are not associated with thexroad-securityserver-observer
role, don't have any access to the secondary and API calls will fail. To avoid this, secondary REST API should only be accessed using keys associated with thexroad-securityserver-observer
role, and only for operations that read configuration, not updates.Furthermore, API keys are accessed through a cache that assumes that all updates to keys (e.g. revoking keys, or changing permissions) are done using the same node. If API keys are changed on primary, the changes are not reflected on the secondary caches until the next time
xroad-proxy-ui-api
process is restarted. To address this issue, you should restart secondary nodes'xroad-proxy-ui-api
processes after API keys are modified (and database has been replicated to secondaries), to ensure correct operation.Improvements to API key handling in clustered setups will be included in later releases.
-
It is possible to use the autologin-package with secondary nodes to enable automatic PIN-code insertion, however the autologin-package default implementation stores PIN-codes in plain text and should not be used in production environments. Instructions on how to configure the autologin-package to use a more secure custom PIN-code storing implementation can be found in autologin documentation
The configuration is now complete. If you do not want to set up the health check service, continue to chapter 6 to verify the setup.
3.4 Health check service configuration
The load balancing support includes a health check service that can be used to ping the security server using HTTP to see if it is healthy and likely to be able to send and receive messages. The service is disabled by default but can be enabled via configuration options.
| Proxy service configuration option | Default value | Description |
|---|---|---|
| health-check-interface |
<IP_ADDRESS> (all network interfaces) |
The network interface this service listens to. This should be an address the load balancer component can use to check the server status |
| health-check-port |
0 (disabled) |
The tcp port the service listens to for HTTP requests. The default value 0 disables the service. |
Below is a configuration that can be added to /etc/xroad/conf.d/local.ini
on the primary that would enable the health check
service on all the nodes once the configuration has been replicated. Changes to the settings require restarting the
xroad-proxy
service to take effect. This example enables listening to all available network interfaces (<IP_ADDRESS>
) on
port 5588.
[proxy]
health-check-interface=<IP_ADDRESS>
health-check-port=5588
The service can be accessed using plain HTTP. It will return HTTP 200 OK
if the proxy should be able to process messages
and HTTP 500 Internal Server Error
otherwise. A short message about the failure reason, if available, is added to the
body of the response. The service runs as a part of the xroad-proxy
service.
In addition to implicitly verifying that the xroad-proxy
service is running, the health checks verify that:
- The server authentication key is accessible and that the OCSP response for the certificate is
good
. This requires a runningxroad-signer
service in good condition. - The
serverconf
database is accessible. - The
global configuration
is valid and not expired.
Each of these status checks has a separate timeout of 5 seconds. If the status check fails to produce a response in this
time, it will be considered a health check failure and will cause a HTTP 500
response.
In addition, each status check result will be cached for a short while to avoid excess resource usage. A successful status check result will be cached for 2 seconds before a new verification is triggered. This is to make sure the OK results are as fresh as possible while avoiding per-request verification. In contrast, verification failures will be cached for 30 seconds before a new verification is triggered. This should allow for the security server to get up and running after a failure or possible reboot before the status is queried again.
Security server's health check interface can also be manually switched to a maintenance mode in order to inform the load balancing solution that the security server will be undergoing maintenance and should be removed from active use.
When in maintenance mode the health check interface will only respond with HTTP 503 Service unavailable
and the message
Health check interface is in maintenance mode
and no actual health check diagnostics will be run. Maintenance mode is disabled
by default and will automatically reset to its default when the proxy service is restarted.
Maintenance mode can be enabled or disabled by sending HTTP GET
-request from the target security server to its proxy admin port 5566
.
The intended new state can be defined using the targetState
HTTP-parameter:
| Command | URI |
|---|---|
| Enable maintenance mode | http://localhost:5566/maintenance?targetState=true |
| Disable maintenance mode | http://localhost:5566/maintenance?targetState=false |
Proxy admin port will respond with 200 OK
and a message detailing the actualized maintenance mode state change,
e.g. Maintenance mode set: false => true
. In case the maintenance mode state could not be changed, the returned
message will detail the reason.
3.4.1 Known check result inconsistencies vs. actual state
There is a known but rarely and not naturally occurring issue where the health check will report an OK condition for a limited time but sending some messages might not be possible. This happens when an admin user logs out of the keys.
The health check will detect if the tokens (the key containers) have not been signed into after xroad-signer
startup.
It will however, not detect immediately when the tokens are manually logged out of. The keys are cached by the xroad-proxy
process for a short while. As long as the authentication key is still cached, the health check will return OK, even though
the necessary signing context values for sending a message might no longer be cached. This means messages might fail to be sent
even if the health check returns OK. As the authentication key expires from the cache (after a maximum of 5 minutes), the
health check will start returning failures. This is a feature of caching and not a bug per se. In addition, logging out
of a security server's keys should not occur by accident so it should not be a surprise that the node cannot send messages
after not having access to it's keys.
3.4.2 Health check examples
Before testing with an actual load balancer, you can test the health check service with curl
, for example.
Below is an example response from the Health check service when everything is up and running and messages should go through this node:
$ curl -i localhost:5588
HTTP/1.1 200 OK
Content-Length: 0
Server: Jetty(8.y.z-SNAPSHOT)
And a health check service response on the same node when the service xroad-signer
is not running:
$ curl -i localhost:5588
HTTP/1.1 500 Server Error
Transfer-Encoding: chunked
Server: Jetty(8.y.z-SNAPSHOT)
Fetching health check response timed out for: Authentication key OCSP status
Continue to chapter 6 to verify the setup.
4. Database replication setup
For technical details on the PostgreSQL replication, refer to the official documentation. Note that the versions of PostgreSQL distributed with RHEL and Ubuntu are different. At the time of writing, RHEL 7 distributes PostgreSQL version 9.2, and RHEL 8 version 10; the replication configuration is the same for these versions. On Ubuntu 20.04 using PostgreSQL version 12 and on 22.04 using version 14, the configuration has some differences.
4.1 Setting up TLS certificates for database authentication
This section describes how to create and set up certificate authentication between the secondary and primary database instances.
For further details on the certificate authentication, see the PostgreSQL documentation.
-
Generate the Certificate Authority key and a self-signed certificate for the root-of-trust:
openssl req -new -x509 -days 7300 -nodes -sha256 -out ca.crt -keyout ca.key -subj '/O=cluster/CN=CA'
The subject name does not really matter here. Remember to keep the
ca.key
file in a safe place.Alternatively, an existing internal CA can be used for managing the certificates. A sub-CA should be created as the database cluster root-of-trust and used for issuing the secondary and primary certificates.
-
Generate keys and certificates signed by the CA for each postgresql instance, including the primary. Do not use the CA certificate and key as the database certificate and key.
Generate a key and the Certificate Signing Request for it:
openssl req -new -nodes -days 7300 -keyout server.key -out server.csr -subj "/O=cluster/CN=<nodename>"
Note: The
<nodename>
(the subject common name) will be used for identifying the cluster nodes. For secondary nodes, it needs to match the replication user name that is added to the primary database and the username that the secondary node database uses to connect to the primary. For example, in a system with one primary and two secondaries, the names of the nodes could beprimary
,replica1
andreplica2
. Other parts of the subject name do not matter and can be named as is convenient.For more information on adding the replication user name to the primary database, see chapter 4.3 Configuring the primary instance for replication.
Configuring the username on the secondary nodes is detailed in chapter 4.4 Configuring the secondary instance for replication).
Sign the CSR with the CA, creating a certificate:
openssl x509 -req -in server.csr -CAcreateserial -CA ca.crt -CAkey ca.key -days 7300 -out server.crt
Repeat the above steps for each node.
-
Copy the certificates and keys to the nodes:
First, prepare a directory for them:
sudo mkdir -p -m 0755 /etc/xroad/postgresql sudo chmod o+x /etc/xroad
Then, copy the certificates (ca.crt, and the instance's server.crt and server.key) to
/etc/xroad/postgresql
on each cluster instance.Finally, set the owner and access rights for the key and certificates:
sudo chown postgres /etc/xroad/postgresql/* sudo chmod 400 /etc/xroad/postgresql/*
4.2 Creating a separate PostgreSQL instance for the serverconf
database
4.2.1 on RHEL
Create a new systemctl
service unit for the new database. As root, execute the following command:
cat <<EOF >/etc/systemd/system/postgresql-serverconf.service
.include /lib/systemd/system/postgresql.service
[Service]
Environment=PGPORT=5433
Environment=PGDATA=/var/lib/pgsql/serverconf
EOF
Create the database and configure SELinux:
PGSETUP_INITDB_OPTIONS="--auth-local=peer --auth-host=md5" postgresql-setup initdb postgresql-serverconf
semanage port -a -t postgresql_port_t -p tcp 5433
systemctl enable postgresql-serverconf
4.2.2 on Ubuntu
sudo -u postgres pg_createcluster -p 5433 10 serverconf
In the above command, 10
is the postgresql major version. Use pg_lsclusters
to find out what version(s) are available.
4.3 Configuring the primary instance for replication
Edit postgresql.conf
and set the following options:
On RHEL, PostgreSQL config files are located in the
PGDATA
directory/var/lib/pgql/serverconf
.
Ubuntu keeps the config in/etc/postgresql/<version>/<cluster name>
, e.g./etc/postgresql/10/serverconf
)
ssl = on
ssl_ca_file = '/etc/xroad/postgresql/ca.crt'
ssl_cert_file = '/etc/xroad/postgresql/server.crt'
ssl_key_file = '/etc/xroad/postgresql/server.key'
listen_addresses = '*' # (default is localhost. Alternatively: localhost, <IP of the interface the secondaries connect to>")
# PostgreSQL 9.2 (RHEL 7)
wal_level = hot_standby
# PostgreSQL 10 & 12 (RHEL 8, Ubuntu 20.04)
wal_level = replica
max_wal_senders = 3 # should be ~ number of secondaries plus some small number. Here, we assume there are two secondaries.
wal_keep_segments = 8 # keep some wal segments so that secondaries that are offline can catch up.
# PostgreSQL >=14 (Ubuntu 22.04)
wal_level = replica
max_wal_senders = 3 # should be ~ number of secondaries plus some small number. Here, we assume there are two secondaries.
wal_keep_size = 8 # keep some wal size so that secondaries that are offline can catch up.
For more information about the streaming replication configuration options, see the PostgreSQL documentation.
Edit pg_hba.conf
and enable connections to the replication pseudo database using client certificates. See chapter
4.1 for the authentication setup.
hostssl replication +slavenode samenet cert
Note: The CN field in the certificate subject must match a replication user name in postgresql. See the PostgreSQL documentation for more details.
The samenet
above assumes that the secondaries will be in the same subnet as the primary.
Start the primary instance:
Ubuntu 20.04:
systemctl start postgresql@12-serverconf
Ubuntu 22.04:
systemctl start postgresql@14-serverconf
RHEL:
systemctl start postgresql-serverconf
Create the replication user(s) with password authentication disabled:
sudo -u postgres psql -p 5433 -c "CREATE ROLE slavenode NOLOGIN";
sudo -u postgres psql -p 5433 -c "CREATE USER "<nodename>" REPLICATION PASSWORD NULL IN ROLE slavenode";
Create a user named serverconf
for local serverconf
database access:
sudo -u postgres psql -p 5433 -c "CREATE USER serverconf PASSWORD '<password>'";
Copy the serverconf
database from the default instance to the new instance:
sudo -u postgres pg_dump -C serverconf | sudo -u postgres psql -p 5433 -f -
To avoid confusion, the old serverconf
database on the primary should be renamed, or even deleted.
sudo -u postgres psql -p 5432 -c "ALTER DATABASE serverconf RENAME TO serverconf_old";
4.4 Configuring the secondary instance for replication
Prerequisites:
- A separate postgresql instance has been created.
- TLS keys and certificates have been configured in
/etc/xroad/postgresql
as described in section 4.1 Setting up TLS certificates for database authentication
Go to the postgresql data directory:
- RHEL:
/var/lib/pgsql/serverconf
- Ubuntu:
/var/lib/postgresql/<postgresql major version>/serverconf
Clear the data directory:
rm -rf *
Then, do a base backup with pg_basebackup
:
sudo -u postgres PGSSLMODE=verify-ca PGSSLROOTCERT=/etc/xroad/postgresql/ca.crt PGSSLCERT=/etc/xroad/postgresql/server.crt PGSSLKEY=/etc/xroad/postgresql/server.key pg_basebackup -h <primary> -p 5433 -U <nodename> -D .
Where <primary>
is the DNS or IP address of the primary node and <nodename>
is the node name (the replication user name added to the primary database).
Note: This warning by pg_basebackup
can be ignored:
NOTICE: WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup
On RHEL 7/8 (PostgreSQL <12), add the following recovery.conf
to the data directory. Set the owner of the file to postgres:postgres
, mode 0600
.
standby_mode = 'on'
primary_conninfo = 'host=<primary> port=5433 user=<nodename> sslmode=verify-ca sslcert=/etc/xroad/postgresql/server.crt sslkey=/etc/xroad/postgresql/server.key sslrootcert=/etc/xroad/postgresql/ca.crt'
trigger_file = '/var/lib/xroad/postgresql.trigger'
Where, as above, <primary>
is the DNS or IP address of the primary node and <nodename>
is the node name (the replication user name added to the primary database).
On Ubuntu 20.04 & 22.04 (PostgreSQL >=12), create an empty standby.signal
file in the data directory. Set the owner of the file to postgres:postgres
, mode 0600
.
Next, modify postgresql.conf
:
On RHEL, PostgreSQL config files are located in the
PGDATA
directory/var/lib/pgql/serverconf
.
Ubuntu keeps the config in/etc/postgresql/<version>/<cluster name>
, e.g./etc/postgresql/12/serverconf
)
ssl = on
ssl_ca_file = '/etc/xroad/postgresql/ca.crt'
ssl_cert_file = '/etc/xroad/postgresql/server.crt'
ssl_key_file = '/etc/xroad/postgresql/server.key'
listen_addresses = localhost
# no need to send WAL logs
# wal_level = minimal
# max_wal_senders = 0
# wal_keep_segments = 0
hot_standby = on
hot_standby_feedback = on
On Ubuntu 20.04 & 22.04 (PostgreSQL >=12) only, add the primary_conninfo to postgresql.conf:
primary_conninfo = 'host=<primary> port=5433 user=<nodename> sslmode=verify-ca sslcert=/etc/xroad/postgresql/server.crt sslkey=/etc/xroad/postgresql/server.key sslrootcert=/etc/xroad/postgresql/ca.crt'
Where, as above, <primary>
is the DNS or IP address of the primary node and <nodename>
is the node name (the replication user name added to the primary database).
Notice that on RHEL, during pg_basebackup
the postgresql.conf
was copied from the primary node so the WAL sender
parameters should be disabled. Also check that listen_addresses
is localhost-only.
Finally, start the database instance
RHEL:
systemctl start postgresql-serverconf
Ubuntu 20.04:
systemctl start postgresql@12-serverconf
Ubuntu 22.04:
systemctl start postgresql@14-serverconf
5. Configuring data replication with rsync over SSH
5.1 Set up SSH between secondaries and the primary
On the primary, set up a system user that can read /etc/xroad
. A system user has their password disabled and can not log
in normally.
Ubuntu:
adduser --system --shell /bin/bash --ingroup xroad xroad-slave
RHEL:
useradd -r -m -g xroad xroad-slave
Create an .ssh
folder and the authorized keys file:
sudo mkdir -m 755 -p /home/xroad-slave/.ssh && sudo touch /home/xroad-slave/.ssh/authorized_keys
Warning: The owner of the file should be root
and xroad-slave
should not have write permission to the file.
On the secondary nodes, create an ssh key (ssh-keygen
) without a passphrase for the xroad
user and add the public keys in
the /home/xroad-slave/.ssh/authorized_keys
of the primary node. To finish, from secondary nodes, connect to the primary host
using ssh
and accept the host key.
5.2 Set up periodic configuration synchronization on the secondary nodes
The following configuration, which will be set up on the secondary nodes will synchronize the configuration in /etc/xroad
periodically (once per minute) and before the services are started. That means that during boot, if the primary server is
available, the configuration will be synchronized before the xroad-proxy
service is started. If the primary node is down,
there will be a small delay before the services are started.
Note that only modifications to the signer keyconf will be applied when the system is running. Changes to any other
configuration files, like local.ini
, require restarting the services, which is not automatic.
5.2.1 Use systemd for configuration synchronization
First, add xroad-sync
as a systemd
service.
Create a new file /etc/systemd/system/xroad-sync.service
:
[Unit]
Description=X-Road Sync Task
After=network.target
Before=xroad-proxy.service
Before=xroad-signer.service
Before=xroad-confclient.service
Before=xroad-proxy-ui-api.service
[Service]
User=xroad
Group=xroad
Type=oneshot
Environment=XROAD_USER=xroad-slave
Environment=MASTER=<primary_host>
ExecStartPre=/usr/bin/test ! -f /var/tmp/xroad/sync-disabled
ExecStart=/usr/bin/rsync -e "ssh -o ConnectTimeout=5 " -aqz --timeout=10 --delete-delay --exclude db.properties --exclude "/conf.d/node.ini" --exclude "*.tmp" --exclude "/postgresql" --exclude "/globalconf" --exclude "/gpghome" --delay-updates --log-file=/var/log/xroad/slave-sync.log ${XROAD_USER}@${MASTER}:/etc/xroad/ /etc/xroad/
[Install]
WantedBy=multi-user.target
WantedBy=xroad-proxy.service
Where <primary_host>
is the DNS name or IP address of the primary node.
The service will log rsync
events to /var/log/xroad/slave-sync.log
.
Then, add a timer for periodic updates.
Create a new file /etc/systemd/system/xroad-sync.timer
:
[Unit]
Description=Sync X-Road configuration
[Timer]
OnBootSec=60
OnUnitActiveSec=60
[Install]
WantedBy=timers.target
RHEL only: Configure SELinux to allow rsync
to be run as a systemd
service
setsebool -P rsync_client 1
setsebool -P rsync_full_access 1
Finally, enable the services:
systemctl enable xroad-sync.timer xroad-sync.service
systemctl start xroad-sync.timer
About the
rsync
options
--delay-updates
and--delete-delay
make the sync more atomic by delaying modifications until data has been downloaded. It is not fully atomic, however, since the files will be moved into place one by one. If the synchronization is disrupted, no modifications will be made.- low connect timeout (5 seconds) and receive timeout (10 seconds) ensure that the synchronization won't hang if e.g. a network connection fails.
5.3 Set up log rotation for the sync log on the secondary nodes
The configuration synchronization will log events to /var/log/xroad/slave-sync.log
on the secondary nodes. The following
configuration example rotates those logs daily and keeps them for 7 days which should be enough for troubleshooting.
Create a new file /etc/logrotate.d/xroad-slave-sync
on the secondary nodes:
/var/log/xroad/slave-sync.log {
daily
rotate 7
missingok
compress
su xroad xroad
nocreate
}
6. Verifying the setup
This chapter briefly describes how to check that the replication works. Message delivery is difficult to test without a connection to an X-Road instance test environment.
6.1 Verifying rsync+ssh replication
To test the configuration file replication, a new file can be added to /etc/xroad/
or /etc/xroad/signer/
on the
primary node and verify it has been replicated to the secondary nodes in a few minutes. Make sure the file is owned by
the group xroad
.
Alternatively, check the sync log /var/log/xroad/slave-sync.log
on the secondary nodes and verify it lists successful
transfers. A transfer of an added test file called sync.testfile
to /etc/xroad/signer/
might look like this:
2017/03/10 11:42:01 [10505] receiving file list
2017/03/10 11:42:01 [10507] .d..t...... signer/
2017/03/10 11:42:01 [10507] >f..t...... signer/keyconf.xml
2017/03/10 11:42:01 [10507] >f+++++++++ signer/sync.testfile
2017/03/10 11:42:01 [10505] sent 264 bytes received 1,886 bytes total size 65,346
6.2 Verifying database replication
To see if the database replication is working, connect to the new serverconf
instance on the primary node and verify
that the secondary nodes are listed.
sudo -u postgres psql -p 5433 -c "select * from pg_stat_replication;"
A successful replication with two secondary nodes could look like this:
| pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | state | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1890 | 16719 | hdev-ss3 | walreceiver | <IP_ADDRESS> | 45275 | 2017-03-10 06:30:50.470084+02 | streaming | 0/4058A40 | 0/4058A40 | 0/4058A40 | 0/4058A40 | 0 | async | |
| 1891 | 16718 | hdev-ss2 | walreceiver | <IP_ADDRESS> | 50174 | 2017-03-10 06:30:50.918481+02 | streaming | 0/4058A40 | 0/4058A40 | 0/4058A40 | 0/4058A40 | 0 | async |
For more information on the pg_stat_replication
view, see the PostgreSQL documentation.
6.3 Verifying replication from the admin user interface
Verifying the cluster setup via the admin interface requires the cluster to be part of an existing X-Road instance like
FI-DEV
or FI-TEST
or using a custom, configured X-Road environment with at least a central server and the security
server cluster behind a load balancer.
To test the configuration file replication from the admin user interface, a key can be created in the admin interface of the primary node. In addition, a certificate signing request can be created for the key in the UI, downloaded, signed by an external CA and then uploaded back to the admin UI. For help on these tasks, see the Security Server User Guide [UG-SS].
The keys and certificate changes should be propagated to the secondary nodes in a few minutes.
The serverconf
database replication can also be tested on the admin UI once the basic configuration, as mentioned in
3. X-Road Installation and configuration is done. A new subsystem can be added
to the primary node. A registration request can be sent to the central server, but it is not required. The added subsystem
should appear on the secondary nodes immediately.
7. Upgrading a clustered X-Road security server installation
This chapter briefly discusses ways of upgrading the X-Road software in a clustered environment. The offline option will disrupt message delivery while the online option should allow upgrades with minimal disruption.
7.1 Offline upgrade
If the X-Road security server cluster can be shut down for an offline upgrade, the procedure remains fairly simple:
- Stop the X-Road services (
xroad-proxy
,xroad-signer
,xroad-confclient
,xroad-proxy-ui-api
andxroad-monitor
) on all the nodes. You can read more about the services in the Security Server User Guide [UG-SS] chapter on System services. - Upgrade the packages on the primary node to the new software version.
- Let any database and configuration changes propagate to the cluster members.
- Upgrade the packages on the secondary nodes.
- Start the X-Road services.
7.2 Online rolling upgrade
It is possible to upgrade the software in a cluster to a new version with minimal service disruption.
The steps are in more detail below, but in short, the procedure is:
- Pause the database and configuration synchronization on the secondary nodes. Pausing the synchronization ensures that potentially incompatible changes are not propagated to secondaries before they are upgraded.
- Set the primary node to maintenance mode or manually disable it from the external load balancer, upgrade the software, then resume operation.
- One by one, set a secondary node to maintenance mode or manually disable it from the external load balancer, re-enable synchronization, upgrade it, then resume operation.
7.2.1 Pausing the database and configuration synchronization
-
Pause the database synchronization. Assuming that the
serverconf
database is running in port5433
, issue the following command:# PostgreSQL version < 10 sudo -u postgres psql -p 5433 -c 'select pg_xlog_replay_pause();'
# PostgreSQL version >= 10 sudo -u postgres psql -p 5433 -c 'select pg_wal_replay_pause();'
-
Disable the configuration synchronization on the secondary nodes:
sudo -u xroad touch /var/tmp/xroad/sync-disabled
Note: Check that the synchronization service is configured to honor the
sync-disabled
flag. See the chapter on Setting up periodic configuration synchronization on the secondary nodes for more details.
7.2.2 Upgrading the primary
-
Either use the health check maintenance mode or manually disable the primary node from your external load balancer. A disabled node on the load balancer should be handled gracefully so that in-flight requests are allowed to finish while new requests are routed to other nodes (connection draining).
You can read more about the health check maintenance mode in the chapter about the health check service configuration.
In short, to enable the maintenance mode, on the primary node, call the proxy admin port (default port
5566
) with:curl http://localhost:5566/maintenance?targetState=true
The admin port should respond with:
Maintenance mode set: false => true
-
Check that the primary is no longer processing requests and stop the X-Road services (
xroad-proxy
,xroad-signer
,xroad-confclient
,xroad-monitor
,xroad-proxy-ui-api
) on the primary node. You can read more about the services in the Security Server User Guide [UG-SS] chapter on System services.To ensure that the node is no longer processing requests, you can monitor
/var/log/xroad/proxy.log
to verify that no more requests are arriving or check that there are no connections to the port 5500 with:watch -n1 ss -tn state established sport = :5500 or dport = :5500
-
Upgrade the packages on the primary node to the new software version.
-
Start the X-Road services and wait until the primary node is healthy.
-
a) If the maintenance mode was enabled, the maintenance status from the health check port was cleared on startup of the
xroad-proxy
service. The health check should start returning a200 OK
status as soon as security server can process messages.b) If the primary node was disabled manually from the external load balancer, verify that the primary node is working and enable it from the load balancer. To check if a node is healthy, you can use the health check service:
curl -i http://localhost:<health-check-port>
See 3.4 Health check service configuration for more details.
7.2.3 Upgrade a single secondary node
Repeat this process for each secondary node, one by one.
-
Gracefully disable the secondary node from the load balancer, either manually or using the health check maintenance mode. See step 1 from the primary update instructions for more details.
-
Stop the X-Road services once the secondary has stopped processing requests. See step 2 from the primary update instructions for more details.
-
Enable database synchronization on the secondary:
#PostgreSQL version < 10 sudo -u postgres psql -p 5433 -c 'select pg_xlog_replay_resume()'
#PostgreSQL version >= 10 sudo -u postgres psql -p 5433 -c 'select pg_wal_replay_resume()'
Note that the above command assumes that the
serverconf
database is running in port5433
.Note: Before proceeding, make sure that the database is up to date. The following should return
t
:#PostgreSQL < 10 sudo -u postgres psql -p 5433 -c 'select pg_last_xlog_replay_location() = pg_last_xlog_receive_location()'
#PostgreSQL >= 10 sudo -u postgres psql -p 5433 -c 'select pg_last_wal_replay_lsn() = pg_last_wal_receive_lsn()'
-
Upgrade the packages on the secondary node to the new software version.
-
Enable the shared configuration synchronization on the secondary node:
sudo rm /var/tmp/xroad/sync-disabled
-
Wait for the primary node configuration changes to propagate to the secondary node.
The configuration synchronization can be forced, if necessary.
service xroad-sync start
-
Restart the X-Road services and wait until the secondary node is healthy.
-
After the node is healthy, enable the secondary node in the load balancer if you manually disabled it. If using the maintenance mode, it was cleared on
xroad-proxy
service restart. See step 5 from the primary update instructions for more details.


Calculates landscape metrics for categorical landscape patterns in a tidy workflow. 'landscapemetrics' reimplements the most common metrics from 'FRAGSTATS' (<https://www.umass.edu/landeco/research/fragstats/fragstats.html>) and new ones from the current literature on landscape metrics. This package supports 'raster' spatial objects and takes RasterLayer, RasterStacks, RasterBricks or lists of RasterLayer from the 'raster' package as input arguments. It further provides utility functions to visualize patches, select metrics and building blocks to develop new metrics.
| Version: | 0.3 |
| Depends: | R (≥ 3.1) |
| Imports: | cli, crayon, dplyr, ggplot2, methods, raster, Rcpp, sp, tibble |
| LinkingTo: | Rcpp, RcppArmadillo |
| Suggests: | covr, testthat, knitr, rmarkdown |
| Enhances: | stars, sf |
| Published: | 2018-12-08 |
| Author: | Maximillian H.K. Hesselbarth [aut, cre], Marco Sciaini [aut], Jakub Nowosad [aut], Sebastian Hanss [aut], Laura J. Graham [ctb] (Input on package structure), Jeffrey Hollister [ctb] (Input on package structure), Kimberly A. With [ctb] (Input on package structure), Florian Privé [ctb] (Original author of underlying C++ code for get_nearestneighbour() function), Jeremy VanDerWal [ctb] (Original author of underlying C code for get_patches() function), Matt Strimas-Mackey [ctb] (Bugfix in sample_metrics()) |
| Maintainer: | Maximillian H.K. Hesselbarth <maximilian.hesselbarth at uni-goettingen.de> |
| BugReports: | https://github.com/r-spatialecology/landscapemetrics/issues |
| License: | GPL-3 |
| URL: | https://r-spatialecology.github.io/landscapemetrics/ |
| NeedsCompilation: | yes |
| SystemRequirements: | C++11 |
| Materials: | NEWS |
| CRAN checks: | landscapemetrics results |
| Reference manual: | landscapemetrics.pdf |
| Vignettes: |
Get Started |
| Package source: | landscapemetrics_0.3.tar.gz |
| Windows binaries: | r-devel: landscapemetrics_0.3.zip, r-release: landscapemetrics_0.3.zip, r-oldrel: landscapemetrics_0.3.zip |
| OS X binaries: | r-release: landscapemetrics_0.3.tgz, r-oldrel: landscapemetrics_0.3.tgz |
| Old sources: | landscapemetrics archive |
Please use the canonical form https://CRAN.R-project.org/package=landscapemetrics to link to this page.


I have the tendency to be a little bit of a jaded New York about, well, just about everything. So it goes without saying that even though I have done the research about DBC, heard amazing things about what’s in store, and wrote a check for 12.2 g’s, there is that part of me that’s like “how great can it be”? And, at the risk of having my “true” New Yorker’s status revoked, homeboy’s fireside chat just got me pumped about DBC.
This is what I’m all about. I’m not worried at all anymore about this program. And I have the tendency to worry.  Plain and simple.
 Well done, sir. Well done.


{% endif %} {% unless pageVersion == site.latest %}This is not a stable version!
You are currently viewing the documentation for a development version. It is not guaranteed that this documentation is up to date. Things will likely change without announcement or versioning incrementation. If there is no other documentation available, there are likely no releases available for this repository. The content is, therefore, likely still in development and not production ready. Use at your own risk!
{% endunless %} {% endif %} {{ content }}Please be aware that the page you are currently viewing is not for the latest available version!


View the right hand panel.
You can see from the view in the right that the views are successfully created, despite the warnings that were issued. The views have not been validated yet. This is because we have the AUTO_REVAL database configuration value set to DEFERRED. If the AUTO_REVAL database configuration value was set to IMMEDIATE then the revalidation would occur as soon as any changes to the view or dependent objects are made.
You should see a list of two views in the top right panel.


WITNESS LIST Education Committee April 29, 2005 -12:00A HB 2 ON COMMITTEE SUBSTITUTE (NEW CSHB 2 w/Amendments Anderson, David General Counsel (Texas Education Agency), Austin, TX Cloudt, Criss Associate Commissioner (Texas Education Agency), Austin, TX Parks, Ursula Manager, Public Education Team (Legislative Budget Board), Austin, TX


5 Montana hospitals settle employee insurance lawsuit
BOZEMAN, Mont. (AP) - Five Montana hospitals are settling a lawsuit that claimed they made a deal with an insurer to inflate employee health coverage prices in return for $26 million.
The hospitals, which deny wrongdoing, agreed to pay $6.9 million to employees to resolve the case, The Bozeman Daily Chronicle reported Saturday. A U.S. District Court judge in Great Falls preliminarily approved the settlement Oct. 17, and notices were sent last week to thousands of hospital employees who could be entitled to the money.
The complaint says the hospitals agreed in 2012 to exclusively buy employee health insurance plans from Blue Cross Blue Shield of Montana for six years in exchange for about $26 million and two seats on the insurer’s board of directors.
The hospitals involved include Bozeman Health Deaconess Hospital, Billings Clinic, St. Peter’s Health in Helena, Community Medical Center in Missoula and Northern Montana Hospital in Havre.
In individual statements Friday, hospitals said they took the money to benefit their employees and that the settlement avoids an expensive lawsuit.
Bozeman Health officials disagreed that the hospital should have invested the money from Blue Cross in its employees immediately, saying it used the funds as a reserve for its health plans to keep employees’ premiums low.
“Bozeman Health continues to prioritize the health, safety, and wellness of all Bozeman Health employees and patients, and we’re pleased to have settled this matter,” the statement said.
John Doran with Blue Cross Blue Shield of Montana said the insurer can’t comment on the case because it’s not a defendant.
The complaint was filed in 2017 on behalf of eight hospital workers who claimed the hospitals “attempted to - and did - profit at the expense of the plans and the participants.”
A hearing on the final approval of the settlement is scheduled for Dec. 11. If approved, the hospitals will be protected from any further litigation related to the claims.
Information from: Bozeman Daily Chronicle, http://www.bozemandailychronicle.com
Sign up for Daily Newsletters
Copyright © 2019 The Washington Times, LLC.
Please read our comment policy before commenting.


(PECL getdns >= 0.11.0)
php_getdns_dict_get_int — Get an integer value from a dictionary.
$dict
, string $name
, int &$value
)This function gets an integer value from a dictionary. The dictionary type is used to manage name/value pairs in which the names are strings and the data types of the values are heterogeneous and include bindata, dictionaries, lists, and integers.
dict
The previously created dictionary that is to be used with this request.
name
The name whose associated value is to be returned.
value
The unsigned 32-bit integer value to be returned. Note that this argument is passed by reference and any existing value will be overwritten.
This function returns an integer response code. GETDNS_RETURN_GOOD indicates successful completion.


We make it easy to do your homework. Just tell us what science subject you're working on and we'll match you with a qualified tutor. We offer science help for all grades including.


DEPENDENCIES: Globus toolkit (2.4.3 or higher) (GT) should be installed. There are two different ways to install GT:
1) goto the globus toolkit web page www.globus.org and follow the download and compilation instructions;
2) goto the Virtual Data Toolkit web page http://vdt.cs.wisc.edu/ and follow the instructions (STRONGLY SUGGESTED);
Once a globus client has been installed in your system with a specific flavour it is possible to compile and install the CFITSIO libraries. Specific configuration flags must be used:
1) -with-gsiftp[[=PATH]] Enable Globus Toolkit gsiftp protocol support PATH=GLOBUS_LOCATION i.e. the location of your globus installation
2) -with-gsiftp-flavour[[=PATH] defines the specific Globus flavour ex. gcc32
Both the flags must be used and it is mandatory to set both the PATH and the flavour.
USAGE: To access files on a gridftp server it is necessary to use a gsiftp prefix:
example: gsiftp://remote_server_fqhn/directory/filename
The gridftp driver uses a local buffer on a temporary file the file is located in the /tmp directory. If you have special permissions on /tmp or you do not have a /tmp directory, it is possible to force another location setting the GSIFTP_TMPFILE environment variable (ex. export GSIFTP_TMPFILE=/your/location/yourtmpfile).
Grid FTP supports multi channel transfer. By default a single channel transmission is available. However, it is possible to modify this behavior setting the GSIFTP_STREAMS environment variable (ex. export GSIFTP_STREAMS=8).


Notes on 1bit SGD
Recently I studied how CNTK implemented the 1-Bit SGD in sync mode. Here is a note of the implementation.
Most of the implementation is in files, 1BitSGD\AllReduceDistGradAggregator.h
& 1BitSGD\MatrixQuantizer.h
1. Background
The Gradients of each layer in CNTK is stored as a Maxtrix of MxN, e.g. M inputs, N outputs. And the quantization of gradients is based on each column, e.g. each output.
There are several steps during distributed aggregation,
- Encoding local sub-gradients of each Mini-Batch using 1 Bit by quantization.
- Send local quantized sub-gradients to other computation node in striped mode to get aggregated.
- Receive a strip of quantized sub-gradients from other nodes.
- Decoding each received strip of quantized sub-gradients by unquantization, then add the unquantized results to aggregated gradients strip.
- Encoding the strip of aggregated gradients using 1 bits by quantization.
- Broadcast the quantized aggregated gradients strip to other nodes.
- Receiving the quantized aggregated gradients strips from other nodes, and merge them to local quantized gradient matrix.
- Decoding the aggregated local quantized gradient matrix as an approximation of the global gradients.
2. Quantization&UnQuantization
The quantization is based on each column. E.g.
Q = 0 if V>=0
Q = 1 if V<0
The max and min values of that column are stored together with the quantized results. See, QuantizedColumn
The unquantization will use the max and min values of that column, e.g. (See ValueQuantizer
)
3. Strip
A strip is a group of columns in the gradient matrix, which is decided by the number of distributed computation nodes. For example, suppose there are 4 nodes, and 8 columns, then each nodes will have a strip of 2 columns.
4. Residual
The residual of each column is calculated during quantization and used for the quantization of next mini-batch by adding it to the local sub-gradients of that column.
5. An example of message exchange
Here is an example picture of the message exchange during aggregation.
Node0 will collect quantized local-sub-gradients of Strip0 from all other 3 nodes. Then unquantizes each of them and add results to aggregated results of strip0. Then quantize the aggregated results of Strip0, e.g. AggStrip0 and broadcast it to other 3 nodes.


FScenario
FScenario is a .NET project to write integration tests in a more safe and fun way by exposing common arrangements, disposable fixtures and reliable assertions.
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: |
|
The library comes with comprehensible documentation about the major parts of the project and the complete API reference of the project:
Polling Targets contains a further explanation of the polling functionality to have Open-Minded Assertions
Undoable IO contains a further explanation of the IO operations and their undoable counterparts to have a Zero-Waste Environment
Logging contains a further explanation of how you can centrilize and add more logs for a No-Stress Defect Localization
-
API Reference contains automatically generated documentation for all types, modules and functions in the library. This includes additional brief samples on using most of the functions.
See also other F#/C# testing libraries
-
FSec is a testing library to simplifying Security Tests for .NET programs. The library exposes all kinds of different generators that the developer can use to discover possible security issues with the application.
-
FsCheck is a tool for testing .NET programs automatically. The programmer provides a specification of the program, in the form of properties which functions, methods or objects should satisfy, and FsCheck then tests that the properties hold in a large number of randomly generated cases.
Contributing and copyright
The project is hosted on GitHub where you can report issues, fork the project and submit pull requests. If you're adding a new public API, please also consider adding samples that can be turned into a documentation.
The library is available under Public Domain license, which allows modification and redistribution for both commercial and non-commercial purposes. For more information see the License file in the GitHub repository.
Icons made by Vectors Market from www.flaticon.com is licensed by CC 3.0.
from Index
from FScenario
static member AppendAllLines : path:string * contents:IEnumerable<string> -> unit + 1 overload
static member AppendAllText : path:string * contents:string -> unit + 1 overload
static member AppendText : path:string -> StreamWriter
static member Copy : sourceFileName:string * destFileName:string -> unit + 1 overload
static member Create : path:string -> FileStream + 2 overloads
static member CreateText : path:string -> StreamWriter
static member Decrypt : path:string -> unit
static member Delete : path:string -> unit
static member Encrypt : path:string -> unit
static member Exists : path:string -> bool
...
File.WriteAllText(path: string, contents: string) : unit
File.WriteAllText(path: string, contents: string, encoding: Text.Encoding) : unit
--------------------
File.WriteAllText(path: string, contents: string) : unit
File.WriteAllText(path: string, contents: string, encoding: Text.Encoding) : unit
from FScenario
File.Exists(path: string) : bool
--------------------
File.Exists(path: string) : bool


|
Eastern Palace Melancholic Sky, by Corey Aing
|
Controls:
Z - Shoot
X - Bomb (You're invincible during the animation), use this if you're in a pinch!
SHIFT (Hold) - Focus fire, move slower, and show your hitbox
To play, click the Start Game button
Note: Game may load a little bit longer than usual the first time due to preloading the assets
|


| 137 | (A) | aye by two and by two tiłł thay ware twelue makede |
| 132 | (C) | Ay by two & by two . tyłł alle were disceuered |
| 133 | (D) | Euere by two and by two tul þei were ałł disseueryd |
| 131 | (E) | euer be two & be two . tyl alle departyd . |
| (Ex) | ---- no text ---- | |
| 136 | (L) | ay by two & by two til hy were a-twynne |
| 130 | (P) | Ay be two and be two his wordes to schewen |
| 132 | (U) | Ay by two and by two / til alle were disseuered |
| (V) | ---- no text ---- |


A lazy approach to demonstrate what your product can do for your users, the Laravel way
Written by Regis Freyd and published on August 21, 2020
When users end up on your SAAS marketing site, they want to see as quickly as possible what this product can do for them and how it will help. They want to see what the product can do for real. Screenshots are not enough. Videos are slightly better. Asking users to set up their accounts to see what it will look like is a no go, and the worst idea of all is to set up a button requesting a demo from a salesman.
The best way to demonstrate the capabilities and features of your product is to fill an account with a set of fake data, ideally without any lorem ipsum but actual words. How would you do that?
Several strategies let us achieve exactly that. We’ll discuss the pros and cons of each, from a marketing perspective and more importantly from a developer perspective.
- Upon account creation, fill the account with fake data for the user. This is what I call the Basecamp way.
- Ask the user to use a button to populate the account with fake data.
- Seed the database at the console level.
First option: the Basecamp way
When you create an account in Basecamp, they automatically populate your account with 3 sample projects. Each one of them contains data that seem extremely real, alongside real photos and real comments and interactions. It’s a smart idea as the user can immediately jump in and see, for real, what the product does. However, users will have to find a way to dismiss this information and bring their account back to the original state. Granted, it’s not hard to find in Basecamp, but still.
From a Laravel developer perspective, here is what this approach means:
- Every time an account is created, you have to run a job (or a command that triggers a bunch of jobs) to fill the account with fake data. I suggest that you use queues for that, which Laravel makes it super easy to use.
- This job will actually block the account creation process, as you will need to make sure the job is done before displaying the welcome state to your users.
- That means having a mechanism in your views that will wait for the process to end. You can use several techniques for that, the best way being WebSockets with something like Pusher or your own Redis.
- If your site gains traction and you suddenly attract lots of visitors, the number of jobs will accumulate, the queue will get bigger and the signup process will take significantly more time.
- To prevent this, you can use more workers to work on the queue jobs, but it’s very likely that you still have only one master database so it won’t be much faster even if you add more workers.
- Another drawback, which is kind of inevitable, is the increase of the database’s size as every new user will have his account filled with fake data. Depending on the project’s size, this can accumulate quickly. Unless you provide a mechanism to remove this data, which is an entirely new feature in itself (more on that later).
This approach is a lot of work. It’s also a solution that we can throw money at, even though that’s a solution that most indie hackers don’t like: if you want to reduce the time jobs spend in the queues, you simply need a more powerful database server and your data will be processed faster.
Second option: populate the account on demand
The second option is a bit simpler, but not that much. After having created the account, the user sees a message asking him if he wants to populate the account with a set of data so he can see what the product will look like.
What I like with this approach is the fact that the user will be in control. He will either choose to browse the product with all the possible blank states in the app, or browse it with a lot of fake data. However, it comes with the drawback of having one hard choice to make when you first land on the page: should you fill the account with all these data?
Technically, this option is slightly less complex than the first option. You won’t need to block the UI upon account creation - but you will still need some kind of mechanisms to show the user that the account is currently being populated.
Third option: seed the account at the console level
This is the solution I’ve chosen for my upcoming application, OfficeLife.
OfficeLife will be available in several flavors: as a SAAS on officelife.io, and as a self-hosted option, available for free on GitHub, DockerHub, etc… When you provide an open-source version of your software, you need to be very careful with all the options you provide to you and your future users. Every decision you make will affect everyone.
I’ve decided to provide a way to populate a dummy account, but only when the administrator of the instance will choose to. The solution is to provide a custom Laravel command that will populate the account. You need to connect to a terminal and run the command. This will wipe the database, run all the migrations, create an account, and populate all the data.
The command is php artisan setup:dummyaccount
. And the script is as follow (it has been greatly, greatly shortened as the original file is gigantic):
The advantages compared to previous solutions:
- The administrator is in complete control of when this process occurs.
- You won’t block any signup process as it’ll be executed behind the scenes before any user gets on the platform.
- Technically, it’s not complex per se.
However, with this solution, how would your users test your SAAS? It’s simple: create a separate instance from your main one, something like demo.officelife.io
. This link would be available from your marketing site, probably on the header, and would be separate from the page where your users would signup for real.
This instance would have a cron that would run every 5 minutes, which would trigger the setup:dummyaccount
command. On the login page, simply indicate the login and password that the users need to enter to be logged, and here we go - users have a complete account to play with, with no drawbacks.
Why would the script run every 5 minutes?
- To prevent users from entering too much garbage. If you don’t reset the database at a regular interval, users would create too much fake data and the quality of the data, that all test users would see, would drastically be degraded.
The technical complexity of this solution is limited. The command is pretty simple to write. There are two things to set up in your app:
- Setup the cron that triggers the command every 5 minutes. Laravel makes it dead easy and has a good documentation for this.
- Put in place a message in the header of your app, that will warn users the app is reset every 5 minutes.
Removing fake data from the database
Removing fake data means having the ability to know which data is fake. What does it mean?
In my apps, I mainly use services to do anything that talks to the database.  The name doesn’t matter.
In every service of my app, one of the parameters needed by the service is a boolean that indicates whether the data is dummy data or not. Take a look at how it’s being used in the snippet below (not working code):
In every table of the
Regis Freyd is a full-stack developer and product manager living in Montreal. He’s the author of the popular open source personal CRM Monica. You should follow him on Twitter.
Hi from Canada 🇨🇦
You are not being tracked. Analytics of this site are provided by Fathom, a privacy-friendly analytics tool.


Library: XML
Package: DOM
Header: Poco/DOM/NamedNodeMap.h
Description
Objects implementing the NamedNodeMap interface are used to represent collections of nodes that can be accessed by name. Note that NamedNodeMap does not inherit from NodeList; NamedNodeMaps are not maintained in any particular order. Objects contained in an object implementing NamedNodeMap may also be accessed by an ordinal index, but this is simply to allow convenient enumeration of the contents of a NamedNodeMap, and does not imply that the DOM specifies an order to these Nodes.
NamedNodeMap objects in the DOM are live.
A NamedNodeMap returned from a method must be released with a call to release() when no longer needed.
Inheritance
Direct Base Classes: DOMObject
All Base Classes: DOMObject
Known Derived Classes: DTDMap, AttrMap
Member Summary
Member Functions: getNamedItem, getNamedItemNS, item, length, removeNamedItem, removeNamedItemNS, setNamedItem, setNamedItemNS
Inherited Functions: autoRelease, duplicate, release
Destructor
~NamedNodeMap
virtual ~NamedNodeMap();
Member Functions
getNamedItem
virtual Node * getNamedItem(
const XMLString & name
) const = 0;
Retrieves a node specified by name.
getNamedItemNS
virtual Node * getNamedItemNS(
const XMLString & namespaceURI,
const XMLString & localName
) const = 0;
Retrieves a node specified by name.
item
virtual Node * item(
unsigned long index
) const = 0;
Returns the index'th item in the map. If index is greater than or equal to the number of nodes in the map, this returns null.
length
virtual unsigned long length() const = 0;
Returns the number of nodes in the map. The range of valid child node indices is 0 to length - 1 inclusive.
removeNamedItem
virtual Node * removeNamedItem(
const XMLString & name
) = 0;
Removes a node specified by name. When this map contains the attributes attached to an element, if the removed attribute is known to have a default value, an attribute immediately appears containing the default value.
removeNamedItemNS
virtual Node * removeNamedItemNS(
const XMLString & namespaceURI,
const XMLString & localName
) = 0;
Removes a node specified by name.
setNamedItem
virtual Node * setNamedItem(
Node * arg
) = 0;
Adds a node using its nodeName attribute. If a node with that name is already present in this map, it is replaced by the new one. As the nodeName attribute is used to derive the name which the node must be stored under, multiple nodes of certain types (those that have a "special" string value) cannot be stored as the names would clash. This is seen as preferable to allowing nodes to be aliased.
setNamedItemNS
virtual Node * setNamedItemNS(
Node * arg
) = 0;
Adds a node using its nodeName attribute. If a node with that namespace URI and that local name is already present in this map, it is replaced by the new one.


OUR REF#:CM2334/U DATE:22ND/08/2006 Dear Sir/Madam, I am Mr. Leu Cheng,we are a group of business men who deal on import and export of raw materials into the MEXICO;CANADA;USA AND EUROPE. We are searching for representatives who can help us establish a medium of getting to our customers in the Mexico,USA AND CANADA as well as making payments through you to us. Please if you are interested in transacting business with us we will be very glad. Please contact us for more information. Subject to your satisfaction you will be given the opportunity to negotiate your mode of payments,which we will pay for your services as our representative in MEXICO,USA AND CANADA regions. Note that, as our representative, you will receive 10% of whatever amount you clear for the company and the balance will be paid into an account,we will avail to you. Please, to facilitate the conclusion of this transaction.If accepted,Do provide to me promptly the below validated DATAS to our e-mailing address below: (1)YOUR FULL NAME ............. (2)CONTACT ADDRESS....... (3)TELEPHONE#.................. (4)OCCUPATION(S).............. (5)AGE/SEX............. (6)MARITAL STATUS.............. We do sincerely hope to hear from you soon. Thank you. Best Regards, Mr.Leu Cheng, MANAGER ALT EM@IL ME<EMAIL_ADDRESS>======================================================================= ================== DISCLAIMER NOTE AND CONFIDENTIALITY: leucheng and associates want to inform that the total content of this e-mail is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this message you are hereby notified that any use,review,retransmission,dissemination,distribution, reproduction whether orally or through any other media and/or any action taken upon this message is prohibited. If you received this in error, please contact the sender and delete the material from any computer.


In Java, there is a class named String, which implements strings of characters, such as names and addresses. We have seen in previous sections the way to declare and create an object:
String address = new String("ABC");
One way to create two strings, s1 and s2, which contain the values ABC and DEF, is shown in the following code fragment.
String s1 = new String("ABC"); String s2 = new String("DEF");
But we do not always have to use the new operator with a String.
One more thing to note is the difference between Strings and chars. A char is a primitive and exists in the Java language itself. It is a single data value (a single char) of fixed size. In Java, when a char is used as a literal, it is enclosed in single quotes. For example:
the literal 'A' is a char
A String is an object that is made up of one or more chars. Even if the String has only one char in it, the String is still an object and has the behavior of objects that will be defined in the next section. In Java, when a String is used as a literal, it is enclosed in double quotes. For example:
the literal "A" is a String
In II-D, we discussed object methods. As with class methods, it is often necessary to state explicitly which object the method is to use. Once again, the . (dot) operator is used to specify the particular object that is to be used for the method. Just as with class methods, the left-hand side of the . (dot) operator is the object to use, and the right-hand side is the method to be called. The following code fragment illustrates this point:
String sb1 = new String("ABC"); String sb2 = new String("DEFGHI"); System.out.println("The third character of sb1 is " + sb1.charAt(2)); System.out.println("The third character of sb2 is " + sb2.charAt(2));
In the code above, the charAt method of the String class is run twice, and in each case the third character is taken from the String. (Note that the index for the String ranges from 0 to length() – 1. The first char in the String is at index 0, the next at index 1, and so on.)
In the first println statement, the method charAt, which is on the right-hand side of the . (dot) operator, retrieves the third character from the String sb1, which is on the left-hand side of the . (dot) operator. Because sb1 contains the String ABC, the third character, C, is printed.
In the second println statement, the method charAt, which is on the right-hand side of the . (dot) operator, retrieves the third character from the String sb2, which is on the left-hand side of the . (dot) operator. Because sb2 contains the String ABC, the third character, F, is printed.
To use an instance method, it is necessary to first allocate an object to provide a context for that method. For example, consider the following code fragment:
String sb1; System.out.println("The third character of sb1 is " + sb1.charAt(3)); // Invalid
This code fragment does not make sense because there is no String to query for the third character. This situation is very different from trying to access a character outside of the range of the String. Attempting to call a method on an object that has not been initialized never makes sense and will always result in a null reference exception. Because exceptions will not be covered in this course, you may consider an exception to be an error message.
Some examples of using String methods are presented in the following program. Move your mouse over OUTPUT to see what is printed.
/* File: UMUC_ShowString.java * Author: InstructorX * Date: dd-mm-yyyy * Purpose: Show some string functions */ public class UMUC_ShowString { public static void main(String args[]) { String s1 = new String("ABCDEF"); String s2 = new String("UVWXYZ"); // c1 is set to D, the character in // s1 at position 3 char c1 = s1.charAt(3); System.out.println("c = " + c1); // OUTPUT // c1 is set to X, the character in // s2 at position 3 c1 = s2.charAt(3); System.out.println("c = " + c1); // OUTPUT // s3 is set to CDE, a substring of s1 from // positions 2 to 5 String s3 = s1.substring(2,5); System.out.println(s3); // OUTPUT } // end method } // end class
A complete catalog of all String methods can be found in the Java API located at java.sun.com.
We will now discuss one other important method in the String class—the equals method. Recall that the == operator is used in Java to compare two objects to see if they are the same. This comparison is called equality.
There are times, however, when what is to be checked is not that the objects are the same, but that the values contained in the objects are the same. If the values in the two objects are the same, it is said that the objects are equivalent. For example, consider the following program:
/* File: UMUC_TestString.java * Author: InstructorX * Date: dd-mm-yyyy * Purpose: Test two strings for equality and equivalence */ public class UMUC_TestString { public static void main(String[] args) { String s1 = new String("ABC"); String s2 = s1; String s3 = new String("ABC"); if (s1 == s2) System.out.println("s1 and s2 are equal"); // OUTPUT if (s1.equals(s2)) System.out.println("s1 and s2 are equivalent"); // OUTPUT if (s1 == s3) System.out.println("s1 and s3 are equal"); if (s1.equals(s3)) System.out.println("s1 and s3 are equivalent"); // OUTPUT } // end method } // end class
In this program, Strings s1 and s2 are equal (they refer to the same object). Because they refer to the same object, that object must have the same value, so they are also equivalent.
Strings s1 and s3 refer to different objects, so they are not equal. The Strings s1 and s3 have the same data, however, so these Strings are equivalent.
In summation:
If two variables represent the same object, they are equal and equivalent.
If two variables represent different objects that have the same values, they are not equal but are equivalent.
If two variables represent different objects that have different values, they are neither equal nor equivalent.
In I-D, we introduced wrapper classes found in the java.lang package. Although the two examples applied to only doubles and integers, there are wrapper classes for all of the primitive data types. The Character class has several methods that allow the manipulation of characters. For example, we can compare two characters and determine if they are equal.
Character char1 = new Character('a'); Character char2 = new Character('b'); char1.compareTo(char2); // returns 1 because they are 1 unit apart char1.equals(char2); // returns false because they are not equal
Other methods will allow us to determine if the character is a letter or a digit, or if it is uppercase or lowercase. Two methods will return either a lowercase or an uppercase character. If the Character class is used with the String class, it is possible to conduct several operations on the contents of the string.


Author: Wolf-Dieter Fink
Level: Advanced
Technologies: Asynchronous EJB
Summary: The ejb-asynchronous
quickstart demonstrates the behavior of asynchronous EJB invocations by a deployed EJB and a remote client and how to handle errors.
Target Product: JBoss EAP
Source: https://github.com/jboss-developer/jboss-eap-quickstarts/
The ejb-asynchronous
quickstart demonstrates the behavior of asynchronous EJB invocations in Red Hat JBoss Enterprise Application Platform. The methods are invoked by both an EJB in the deployment and by a remote client. The quickstart also shows error handling if the asynchronous method invocation fails.
The example is composed of 2 Maven modules, each with a shared parent. The modules are as follows:
ejb
: This module contains the EJB's and will be deployed to the serverclient
: This module contains a remote EJB clientThe root pom.xml
builds each of the submodules in the above order and deploys the archive to the server.
The application this project produces is designed to be run on Red Hat JBoss Enterprise Application Platform 7.1 or later.
All you need to build this project is Java 8.0 (Java SDK 1.8) or later and Maven 3.3.1 or later. See Configure Maven for JBoss EAP 7.1 to make sure you are configured correctly for testing the quickstarts.
In the following instructions, replace EAP7_HOME
with the actual path to your JBoss EAP installation. The installation path is described in detail here: Use of EAP7_HOME and JBOSS_HOME Variables.
For Linux: EAP7_HOME/bin/standalone.sh
For Windows: EAP7_HOME\bin\standalone.bat
mvn clean install wildfly:deploy
ejb/target/ejb-asynchronous-ejb.jar
to the running instance of the server.Check whether the application is deployed successfully.
cd client
mvn exec:exec
Note: This quickstart requires quickstart-parent
artifact to be installed in your local Maven repository. To install it, navigate to quickstarts project root directory and run the following command:
mvn clean install
Check the client output
INFO: The server log should contain a message at (about) <date>, indicating that the call to the asynchronous bean completed.
INFO: Got the async result as expected => returning at <date>, duration was 200ms
INFO: Got the async result as expected after wait => returning at <date>, duration was 1500ms
INFO: Catch the expected Exception of the asynchronous execution!
INFO: Results of the parallel (server) processing : [returning at <date> duration was 5000ms, returning at <date>, duration was 3000ms]
Check the server log.
There should be two INFO log messages for the fireAndForget
invocation:
'fireAndForget' Will wait for 15000ms
and 15sec later (the client should be finished at this time)
action 'fireAndForget' finished
mvn wildfly:undeploy
You can also start the server and deploy the quickstarts or run the Arquillian tests from Eclipse using JBoss tools. For general information about how to import a quickstart, add a JBoss EAP server, and build and deploy a quickstart, see Use JBoss Developer Studio or Eclipse to Run the Quickstarts.
This quickstart consists of multiple projects, so it deploys and runs differently in JBoss Developer Studio than the other quickstarts.
ejb-asynchronous-ejb
project and choose Run As
--> Maven Install
.ejb-asynchronous-ejb
project and choose Run As
--> Run on Server
.ejb-asynchronous-client
project and choose Run As
--> Java Application
.Select Java Application
window, choose AsynchronousClient - org.jboss.as.quickstarts.ejb.asynchronous.client
and click OK
.Console
window.ejb-asynchronous-ejb
project and choose Run As
--> Maven build
. Enter wildfly:undeploy
for the Goals
and click Run
.If you want to debug the source code of any library in the project, run the following command to pull the source into your local repository. The IDE should then detect it.
mvn dependency:sources


Subsection of a single module, can be composed of multiple items in a course.
| Name | Description |
|---|---|
| course_branch_id | Each session is associated with a single branch. Learners enrolled in a session see the course contents associated with this branch. |
| course_lesson_id | 5 character alphanumeric string identifying an individual lesson within a course. course_lesson_id need not be unique, but a course_lesson_id/course_id pair is unique. |
| course_module_id | 5 character alphanumeric string identifying an individual module within a course. course_module_id need not be unique, but a course_module_id/course_id pair is unique. |
| course_branch_lesson_order | The order of a lesson within a particular module, with "0" indicating the first lesson in a module. |
| course_branch_lesson_name | The name of a lesson, as seen in the learner view of the course. |
CREATE TABLE course_branch_lessons ( course_branch_id VARCHAR(50) ,course_lesson_id VARCHAR(50) ,course_module_id VARCHAR(50) ,course_branch_lesson_order INT4 ,course_branch_lesson_name VARCHAR(10000) ,PRIMARY KEY (course_branch_id, course_lesson_id) ,FOREIGN KEY (course_branch_id, course_module_id) REFERENCES course_branch_modules(course_branch_id, course_module_id) ,FOREIGN KEY (course_branch_id) REFERENCES course_branches(course_branch_id) );


The Pick and Scroll
Archive
Learning React: A Visual Timeline
| 30 Dec 2017
Just Enough HTML & CSS to Create a Simple Blog Template
| 18 Nov 2017
An Interactive Visualization of the 2018 Cap Situation
| 14 Sep 2017
The Elements of Winning Basketball, Part 2
| 27 Aug 2017
The Elements of Winning Basketball
| 15 Jun 2017
The Lost Coast: A Trip Journal
| 21 Feb 2016