IMPORTANT Enabling this feature and running Docker containers in your cluster has security implications. Given Docker’s integration with many powerful kernel features, it is imperative that administrators understand Docker security before enabling this feature.
Docker combines an easy-to-use interface to Linux containers with easy-to-construct image files for those containers. In short, Docker enables users to bundle an application together with its preferred execution environment to be executed on a target machine. For more information about Docker, see their documentation.
The Linux Container Executor (LCE) allows the YARN NodeManager to launch YARN containers to run either directly on the host machine or inside Docker containers. The application requesting the resources can specify for each container how it should be executed. The LCE also provides enhanced security and is required when deploying a secure cluster. When the LCE launches a YARN container to execute in a Docker container, the application can specify the Docker image to be used.
Docker containers provide a custom execution environment in which the application’s code runs, isolated from the execution environment of the NodeManager and other applications. These containers can include special libraries needed by the application, and they can have different versions of native tools and libraries including Perl, Python, and Java. Docker containers can even run a different flavor of Linux than what is running on the NodeManager.
Docker for YARN provides both consistency (all YARN containers will have the same software environment) and isolation (no interference with whatever is installed on the physical machine).
The LCE requires that container-executor binary be owned by root:hadoop and have 6050 permissions. In order to launch Docker containers, the Docker daemon must be running on all NodeManager hosts where Docker containers will be launched. The Docker client must also be installed on all NodeManager hosts where Docker containers will be launched and able to start Docker containers.
To prevent timeouts while starting jobs, any large Docker images to be used by an application should already be loaded in the Docker daemon’s cache on the NodeManager hosts. A simple way to load an image is by issuing a Docker pull request. For example:
sudo docker pull library/openjdk:8
The following properties should be set in yarn-site.xml:
<configuration> <property> <name>yarn.nodemanager.container-executor.class</name> <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value> <description> This is the container executor setting that ensures that all applications are started with the LinuxContainerExecutor. </description> </property> <property> <name>yarn.nodemanager.linux-container-executor.group</name> <value>hadoop</value> <description> The POSIX group of the NodeManager. It should match the setting in "container-executor.cfg". This configuration is required for validating the secure access of the container-executor binary. </description> </property> <property> <name>yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users</name> <value>false</value> <description> Whether all applications should be run as the NodeManager process' owner. When false, applications are launched instead as the application owner. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.allowed-runtimes</name> <value>default,docker</value> <description> Comma separated list of runtimes that are allowed when using LinuxContainerExecutor. The allowed values are default, docker, and javasandbox. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.type</name> <value></value> <description> Optional. Sets the default container runtime to use. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.image-name</name> <value></value> <description> Optional. Default docker image to be used when the docker runtime is selected. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.image-update</name> <value>false</value> <description> Optional. Default option to decide whether to pull the latest image or not. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.allowed-container-networks</name> <value>host,none,bridge</value> <description> Optional. A comma-separated set of networks allowed when launching containers. Valid values are determined by Docker networks available from `docker network ls` </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.default-container-network</name> <value>host</value> <description> The network used when launching Docker containers when no network is specified in the request. This network must be one of the (configurable) set of allowed container networks. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.host-pid-namespace.allowed</name> <value>false</value> <description> Optional. Whether containers are allowed to use the host PID namespace. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed</name> <value>false</value> <description> Optional. Whether applications are allowed to run in privileged containers. Privileged containers are granted the complete set of capabilities and are not subject to the limitations imposed by the device cgroup controller. In other words, privileged containers can do almost everything that the host can do. Use with extreme care. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.delayed-removal.allowed</name> <value>false</value> <description> Optional. Whether or not users are allowed to request that Docker containers honor the debug deletion delay. This is useful for troubleshooting Docker container related launch failures. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.stop.grace-period</name> <value>10</value> <description> Optional. A configurable value to pass to the Docker Stop command. This value defines the number of seconds between the docker stop command sending a SIGTERM and a SIGKILL. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.privileged-containers.acl</name> <value></value> <description> Optional. A comma-separated list of users who are allowed to request privileged containers if privileged containers are allowed. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.capabilities</name> <value>CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE</value> <description> Optional. This configuration setting determines the capabilities assigned to docker containers when they are launched. While these may not be case-sensitive from a docker perspective, it is best to keep these uppercase. To run without any capabilites, set this value to "none" or "NONE" </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.enable-userremapping.allowed</name> <value>true</value> <description> Optional. Whether docker containers are run with the UID and GID of the calling user. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.userremapping-uid-threshold</name> <value>1</value> <description> Optional. The minimum acceptable UID for a remapped user. Users with UIDs lower than this value will not be allowed to launch containers when user remapping is enabled. </description> </property> <property> <name>yarn.nodemanager.runtime.linux.docker.userremapping-gid-threshold</name> <value>1</value> <description> Optional. The minimum acceptable GID for a remapped user. Users belonging to any group with a GID lower than this value will not be allowed to launch containers when user remapping is enabled. </description> </property> </configuration>
In addition, a container-executor.cfg file must exist and contain settings for the container executor. The file must be owned by root with permissions 0400. The format of the file is the standard Java properties file format, for example
`key=value`
The following properties are required to enable Docker support:
| Configuration Name | Description |
|---|---|
yarn.nodemanager.linux-container-executor.group |
The Unix group of the NodeManager. It should match the yarn.nodemanager.linux-container-executor.group in the yarn-site.xml file. |
The container-executor.cfg must contain a section to determine the capabilities that containers are allowed. It contains the following properties:
| Configuration Name | Description |
|---|---|
module.enabled |
Must be “true” or “false” to enable or disable launching Docker containers respectively. Default value is 0. |
docker.binary |
The binary used to launch Docker containers. /usr/bin/docker by default. |
docker.allowed.capabilities |
Comma separated capabilities that containers are allowed to add. By default no capabilities are allowed to be added. |
docker.allowed.devices |
Comma separated devices that containers are allowed to mount. By default no devices are allowed to be added. |
docker.allowed.networks |
Comma separated networks that containers are allowed to use. If no network is specified when launching the container, the default Docker network will be used. |
docker.allowed.ro-mounts |
Comma separated directories that containers are allowed to mount in read-only mode. By default, no directories are allowed to mounted. |
docker.allowed.rw-mounts |
Comma separated directories that containers are allowed to mount in read-write mode. By default, no directories are allowed to mounted. |
docker.allowed.volume-drivers |
Comma separated list of volume drivers which are allowed to be used. By default, no volume drivers are allowed. |
docker.host-pid-namespace.enabled |
Set to “true” or “false” to enable or disable using the host’s PID namespace. Default value is “false”. |
docker.privileged-containers.enabled |
Set to “true” or “false” to enable or disable launching privileged containers. Default value is “false”. |
docker.privileged-containers.registries |
Comma separated list of privileged docker registries for running privileged docker containers. By default, no registries are defined. |
docker.trusted.registries |
Comma separated list of trusted docker registries for running trusted privileged docker containers. By default, no registries are defined. |
docker.inspect.max.retries |
Integer value to check docker container readiness. Each inspection is set with 3 seconds delay. Default value of 10 will wait 30 seconds for docker container to become ready before marked as container failed. |
docker.no-new-privileges.enabled |
Enable/disable the no-new-privileges flag for docker run. Set to “true” to enable, disabled by default. |
docker.allowed.runtimes |
Comma separated runtimes that containers are allowed to use. By default no runtimes are allowed to be added. |
docker.service-mode.enabled |
Set to “true” or “false” to enable or disable docker container service mode. Default value is “false”. |
Please note that if you wish to run Docker containers that require access to the YARN local directories, you must add them to the docker.allowed.rw-mounts list.
In addition, containers are not permitted to mount any parent of the container-executor.cfg directory in read-write mode.
The following properties are optional:
| Configuration Name | Description |
|---|---|
min.user.id |
The minimum UID that is allowed to launch applications. The default is no minimum |
banned.users |
A comma-separated list of usernames who should not be allowed to launch applications. The default setting is: yarn, mapred, hdfs, and bin. |
allowed.system.users |
A comma-separated list of usernames who should be allowed to launch applications even if their UIDs are below the configured minimum. If a user appears in allowed.system.users and banned.users, the user will be considered banned. |
feature.tc.enabled |
Must be “true” or “false”. “false” means traffic control commands are disabled. “true” means traffic control commands are allowed. |
feature.yarn.sysfs.enabled |
Must be “true” or “false”. See YARN sysfs support for detail. The default setting is disabled. |
Part of a container-executor.cfg which allows Docker containers to be launched is below:
yarn.nodemanager.linux-container-executor.group=yarn [docker] module.enabled=true docker.privileged-containers.enabled=true docker.privileged-containers.registries=local docker.trusted.registries=centos docker.allowed.capabilities=SYS_CHROOT,MKNOD,SETFCAP,SETPCAP,FSETID,CHOWN,AUDIT_WRITE,SETGID,NET_RAW,FOWNER,SETUID,DAC_OVERRIDE,KILL,NET_BIND_SERVICE docker.allowed.networks=bridge,host,none docker.allowed.ro-mounts=/sys/fs/cgroup docker.allowed.rw-mounts=/var/hadoop/yarn/local-dir,/var/hadoop/yarn/log-dir
In order to work with YARN, there are two requirements for Docker images.
First, the Docker container will be explicitly launched with the application owner as the container user. If the application owner is not a valid user in the Docker image, the application will fail. The container user is specified by the user’s UID. If the user’s UID is different between the NodeManager host and the Docker image, the container may be launched as the wrong user or may fail to launch because the UID does not exist. See User Management in Docker Container section for more details.
Second, the Docker image must have whatever is expected by the application in order to execute. In the case of Hadoop (MapReduce or Spark), the Docker image must contain the JRE and Hadoop libraries and have the necessary environment variables set: JAVA_HOME, HADOOP_COMMON_PATH, HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME, HADOOP_YARN_HOME, and HADOOP_CONF_DIR. Note that the Java and Hadoop component versions available in the Docker image must be compatible with what’s installed on the cluster and in any other Docker images being used for other tasks of the same job. Otherwise the Hadoop components started in the Docker container may be unable to communicate with external Hadoop components.
If a Docker image has a command set, the behavior will depend on whether the YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE
is set to true. If so, the command will be overridden when LCE launches the image with YARN’s container launch script.
If a Docker image has an entry point set and YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE is set to true, launch_command will be passed to ENTRYPOINT program as CMD parameters in Docker. The format of launch_command looks like: param1,param2 and this translates to CMD [ “param1”,“param2” ] in Docker.
If an application requests a Docker image that has not already been loaded by the Docker daemon on the host where it is to execute, the Docker daemon will implicitly perform a Docker pull command. Both MapReduce and Spark assume that tasks which take more that 10 minutes to report progress have stalled, so specifying a large Docker image may cause the application to fail.
The Docker plugin utilizes cgroups to limit resource usage of individual containers. Since launched containers belong to YARN, the command line option --cgroup-parent
is used to define the appropriate control group.
Docker supports two different cgroups driver: cgroupfs
and systemd
. Note that only cgroupfs
is supported - attempt to launch a Docker container with systemd
results in the following, similar error message:
Container id: container_1561638268473_0006_01_000002 Exit code: 7 Exception message: Launch container failed Shell error output: /usr/bin/docker-current: Error response from daemon: cgroup-parent for systemd cgroup should be a valid slice named as "xxx.slice". See '/usr/bin/docker-current run --help'. Shell output: main : command provided 4
This means you have to reconfigure the Docker deamon on each host where systemd
driver is used.
Depending on what OS Hadoop is running on, reconfiguration might require different steps. However, if systemd
was chosen for cgroups driver, it is likely that the systemctl
command is available on the system.
Check the ExecStart
property of the Docker daemon:
~$ systemctl show --no-pager --property=ExecStart docker.service ExecStart={ path=/usr/bin/dockerd-current ; argv[]=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
This example shows that the native.cgroupdriver
is systemd
. You have to modify that in the unit file of the daemon.
~$ sudo systemctl edit --full docker.service
This brings up the whole configuration for editing. Just replace the systemd
string to cgroupfs
. Save the changes and restart both the systemd and Docker daemon:
~$ sudo systemctl daemon-reload ~$ sudo systemctl restart docker.service
Before attempting to launch a Docker container, make sure that the LCE configuration is working for applications requesting regular YARN containers. If after enabling the LCE one or more NodeManagers fail to start, the cause is most likely that the ownership and/or permissions on the container-executor binary are incorrect. Check the logs to confirm.
In order to run an application in a Docker container, set the following environment variables in the application’s environment:
| Environment Variable Name | Description |
|---|---|
YARN_CONTAINER_RUNTIME_TYPE |
Determines whether an application will be launched in a Docker container. If the value is “docker”, the application will be launched in a Docker container. Otherwise a regular process tree container will be used. |
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE |
Names which image will be used to launch the Docker container. Any image name that could be passed to the Docker client’s run command may be used. The image name may include a repo prefix. |
YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE |
Controls whether the Docker container’s default command is overridden. When set to true, the Docker container’s command will be “bash path_to_launch_script”. When unset or set to false, the Docker container’s default command is used. |
YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK |
Sets the network type to be used by the Docker container. It must be a valid value as determined by the yarn.nodemanager.runtime.linux.docker.allowed-container-networks property. |
YARN_CONTAINER_RUNTIME_DOCKER_PORTS_MAPPING |
Allows a user to specify ports mapping for the bridge network Docker container. The value of the environment variable should be a comma-separated list of ports mapping. It’s the same to “-p” option for the Docker run command. If the value is empty, “-P” will be added. |
YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_PID_NAMESPACE |
Controls which PID namespace will be used by the Docker container. By default, each Docker container has its own PID namespace. To share the namespace of the host, the yarn.nodemanager.runtime.linux.docker.host-pid-namespace.allowed property must be set to true. If the host PID namespace is allowed and this environment variable is set to host, the Docker container will share the host’s PID namespace. No other value is allowed. |
YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER |
Controls whether the Docker container is a privileged container. In order to use privileged containers, the yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed property must be set to true, and the application owner must appear in the value of the yarn.nodemanager.runtime.linux.docker.privileged-containers.acl property. If this environment variable is set to true, a privileged Docker container will be used if allowed. No other value is allowed, so the environment variable should be left unset rather than setting it to false. |
YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS |
Adds additional volume mounts to the Docker container. The value of the environment variable should be a comma-separated list of mounts. All such mounts must be given as source:dest[:mode] and the mode must be “ro” (read-only) or “rw” (read-write) to specify the type of access being requested. If neither is specified, read-write will be assumed. The mode may include a bind propagation option. In that case, the mode should either be of the form [option] , rw+[option] , or ro+[option] . Valid bind propagation options are shared, rshared, slave, rslave, private, and rprivate. The requested mounts will be validated by container-executor based on the values set in container-executor.cfg for docker.allowed.ro-mounts and docker.allowed.rw-mounts . |
YARN_CONTAINER_RUNTIME_DOCKER_TMPFS_MOUNTS |
Adds additional tmpfs mounts to the Docker container. The value of the environment variable should be a comma-separated list of absolute mount points within the container. |
YARN_CONTAINER_RUNTIME_DOCKER_DELAYED_REMOVAL |
Allows a user to request delayed deletion of the Docker container on a per container basis. If true, Docker containers will not be removed until the duration defined by yarn.nodemanager.delete.debug-delay-sec has elapsed. Administrators can disable this feature through the yarn-site property yarn.nodemanager.runtime.linux.docker.delayed-removal.allowed. This feature is disabled by default. When this feature is disabled or set to false, the container will be removed as soon as it exits. |
YARN_CONTAINER_RUNTIME_YARN_SYSFS_ENABLE |
Enable mounting of container working directory sysfs sub-directory into Docker container /hadoop/yarn/sysfs. This is useful for populating cluster information into container. |
YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE |
Enable Service Mode which runs the docker container as defined by the image but does not set the user (–user and –group-add). |
The first two are required. The remainder can be set as needed. While controlling the container type through environment variables is somewhat less than ideal, it allows applications with no awareness of YARN’s Docker support (such as MapReduce and Spark) to nonetheless take advantage of it through their support for configuring the application environment.
Once an application has been submitted to be launched in a Docker container, the application will behave exactly as any other YARN application. Logs will be aggregated and stored in the relevant history server. The application life cycle will be the same as for a non-Docker application.
WARNING Care should be taken when enabling this feature. Enabling access to directories such as, but not limited to, /, /etc, /run, or /home is not advisable and can result in containers negatively impacting the host or leaking sensitive information. WARNING
Files and directories from the host are commonly needed within the Docker containers, which Docker provides through volumes. Examples include localized resources, Apache Hadoop binaries, and sockets. To facilitate this need, YARN-6623 added the ability for administrators to set a whitelist of host directories that are allowed to be bind mounted as volumes into containers. YARN-5534 added the ability for users to supply a list of mounts that will be mounted into the containers, if allowed by the administrative whitelist.
In order to make use of this feature, the following must be configured.
docker.allowed.ro-mounts
and docker.allowed.rw-mounts
to the list of parent directories that are allowed to be mounted.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS
environment variable.The administrator supplied whitelist is defined as a comma separated list of directories that are allowed to be mounted into containers. The source directory supplied by the user must either match or be a child of the specified directory.
The user supplied mount list is defined as a comma separated list in the form source:destination or source:destination:mode. The source is the file or directory on the host. The destination is the path within the container where the source will be bind mounted. The mode defines the mode the user expects for the mount, which can be ro (read-only) or rw (read-write). If not specified, rw is assumed. The mode may also include a bind propagation option (shared, rshared, slave, rslave, private, or rprivate). In that case, the mode should be of the form option, rw+option, or ro+option.
The following example outlines how to use this feature to mount the commonly needed /sys/fs/cgroup directory into the container running on YARN.
The administrator sets docker.allowed.ro-mounts in container-executor.cfg to “/sys/fs/cgroup”. Applications can now request that “/sys/fs/cgroup” be mounted from the host into the container in read-only mode.
At application submission time, the YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS environment variable can then be set to request this mount. In this example, the environment variable would be set to “/sys/fs/cgroup:/sys/fs/cgroup:ro”. The destination path is not restricted, “/sys/fs/cgroup:/cgroup:ro” would also be valid given the example admin whitelist.
YARN’s Docker container support launches container processes using the uid:gid identity of the user, as defined on the NodeManager host. User and group name mismatches between the NodeManager host and container can lead to permission issues, failed container launches, or even security holes. Centralizing user and group management for both hosts and containers greatly reduces these risks. When running containerized applications on YARN, it is necessary to understand which uid:gid pair will be used to launch the container’s process.
As an example of what is meant by uid:gid pair, consider the following. By default, in non-secure mode, YARN will launch processes as the user nobody
(see the table at the bottom of Using CGroups with YARN for how the run as user is determined in non-secure mode). On CentOS based systems, the nobody
user’s uid is 99
and the nobody
group is 99
. As a result, YARN will call docker run
with --user 99:99
. If the nobody
user does not have the uid 99
in the container, the launch may fail or have unexpected results.
One exception to this rule is the use of Privileged Docker containers. Privileged containers will not set the uid:gid pair when launching the container and will honor the USER or GROUP entries in the Dockerfile. This allows running privileged containers as any user which has security implications. Please understand these implications before enabling Privileged Docker containers.
There are many ways to address user and group management. Docker, by default, will authenticate users against /etc/passwd
(and /etc/shadow
) within the container. Using the default /etc/passwd
supplied in the Docker image is unlikely to contain the appropriate user entries and will result in launch failures. It is highly recommended to centralize user and group management. Several approaches to user and group management are outlined below.
The most basic approach to managing user and groups is to modify the user and group within the Docker image. This approach is only viable in non-secure mode where all container processes will be launched as a single known user, for instance nobody
. In this case, the only requirement is that the uid:gid pair of the nobody user and group must match between the host and container. On a CentOS based system, this means that the nobody user in the container needs the UID 99
and the nobody group in the container needs GID 99
.
One approach to change the UID and GID is by leveraging usermod
and groupmod
. The following sets the correct UID and GID for the nobody user/group.
usermod -u 99 nobody groupmod -g 99 nobody
This approach is not recommended beyond testing given the inflexibility to add users.
When organizations already have automation in place to create local users on each system, it may be appropriate to bind mount /etc/passwd and /etc/group into the container as an alternative to modifying the container image directly. To enable the ability to bind mount /etc/passwd and /etc/group, update docker.allowed.ro-mounts
in container-executor.cfg
to include those paths. When submitting the application, YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS
will need to include /etc/passwd:/etc/passwd:ro
and /etc/group:/etc/group:ro
.
There are several challenges with this bind mount approach that need to be considered.
This approach is not recommended beyond testing given the inflexibility to modify running containers.
An alternative approach that allows for centrally managing users and groups is SSSD. System Security Services Daemon (SSSD) provides access to different identity and authentication providers, such as LDAP or Active Directory.
The traditional schema for Linux authentication is as follows:
application -> libpam -> pam_authenticate -> pam_unix.so -> /etc/passwd
If we use SSSD for user lookup, it becomes:
application -> libpam -> pam_authenticate -> pam_sss.so -> SSSD -> pam_unix.so -> /etc/passwd
We can bind-mount the UNIX sockets SSSD communicates over into the container. This will allow the SSSD client side libraries to authenticate against the SSSD running on the host. As a result, user information does not need to exist in /etc/passwd of the docker image and will instead be serviced by SSSD.
Step by step configuration for host and container:
# yum -y install sssd-common sssd-proxy
# cat /etc/pam.d/sss_proxy auth required pam_unix.so account required pam_unix.so password required pam_unix.so session required pam_unix.so
# cat /etc/sssd/sssd/conf [sssd] services = nss,pam config_file_version = 2 domains = proxy [nss] [pam] [domain/proxy] id_provider = proxy proxy_lib_name = files proxy_pam_target = sss_proxy
# systemctl start sssd
# getent passwd -s sss localuser
It’s important to bind-mount the /var/lib/sss/pipes directory from the host to the container since SSSD UNIX sockets are located there.
-v /var/lib/sss/pipes:/var/lib/sss/pipes:rw
All the steps below should be executed on the container itself.
Install only the sss client libraries
# yum -y install sssd-client
make sure sss is configured for passwd and group databases in
/etc/nsswitch.conf
configure the PAM service that the application uses to call into SSSD
# cat /etc/pam.d/system-auth #%PAM-1.0 # This file is auto-generated. # User changes will be destroyed the next time authconfig is run. auth required pam_env.so auth sufficient pam_unix.so try_first_pass nullok auth sufficient pam_sss.so forward_pass auth required pam_deny.so account required pam_unix.so account [default=bad success=ok user_unknown=ignore] pam_sss.so account required pam_permit.so password requisite pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type= password sufficient pam_unix.so try_first_pass use_authtok nullok sha512 shadow password sufficient pam_sss.so use_authtok password required pam_deny.so session optional pam_keyinit.so revoke session required pam_limits.so -session optional pam_systemd.so session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so
Save the docker image and use the docker image as base image for your applications.
test the docker image launched in YARN environment.
$ id uid=5000(localuser) gid=5000(localuser) groups=5000(localuser),1337(hadoop)
Privileged docker container can interact with host system devices. This can cause harm to host operating system without proper care. In order to mitigate risk of allowing privileged container to run on Hadoop cluster, we implemented a controlled process to sandbox unauthorized privileged docker images.
The default behavior disallows any privileged docker containers. Privileged docker is only allowed with ENTRYPOINT enabled docker image, and docker.privileged-containers.enabled
is set to enabled. Docker image can run with root privileges in the docker container, but access to host level devices are disabled. This allows developer and tester to run docker images from internet with some restrictions to prevent harm to host operating system.
When docker images have been certified by developers and testers to be trustworthy. The trusted image can be promoted to trusted docker registry. System administrator can define docker.trusted.registries
, and setup private docker registry server to promote trusted images. System administrator may choose to allow official docker images from Docker Hub to be part of trusted registries. “library” is the name to use for trusting official docker images. Container-executor.cfg example:
[docker] docker.privileged-containers.enabled=true docker.trusted.registries=library
Fine grained access control can also be defined using docker.privileged-containers.registries
to allow only a subset of Docker images to run as privileged containers. If docker.privileged-containers.registries
is not defined, YARN will fall back to use docker.trusted.registries
as access control for privileged Docker images. Fine grained access control example:
[docker] docker.privileged-containers.enabled=true docker.privileged-containers.registries=local/centos:latest docker.trusted.registries=library
In development environment, local images can be tagged with a repository name prefix to enable trust. The recommendation of choosing a repository name is using a local hostname and port number to prevent accidentially pulling docker images from Docker Hub or use reserved Docker Hub keyword: “local”. Docker run will look for docker images on Docker Hub, if the image does not exist locally. Using a local hostname and port in image name can prevent accidental pulling of canonical images from docker hub. Example of tagging image with localhost:5000 as trusted registry:
docker tag centos:latest localhost:5000/centos:latest
Let’s say you have an Ubuntu-based image with some changes in the local repository and you wish to use it. The following example tags the local_ubuntu
image:
docker tag local_ubuntu local/ubuntu:latest
Next, you have to add local
to docker.trusted.registries
. The image can be referenced by using local/ubuntu
.
Trusted images are allowed to mount external devices such as HDFS via NFS gateway, or host level Hadoop configuration. If system administrators allow writing to external volumes using docker.allow.rw-mounts directive
, privileged docker container can have full control of host level files in the predefined volumes.
For YARN Service HTTPD example, container-executor.cfg must define centos docker registry to be trusted for the example to run.
On restart, the NodeManager, as part of the NodeManager’s recovery process, will validate that a container is still running by checking for the existence of the container’s PID directory in the /proc filesystem. For security purposes, operating system administrator may enable the hidepid mount option for the /proc filesystem. If the hidepid option is enabled, the yarn user’s primary group must be whitelisted by setting the gid mount flag similar to below. Without the yarn user’s primary group whitelisted, container reacquisition will fail and the container will be killed on NodeManager restart.
proc /proc proc nosuid,nodev,noexec,hidepid=2,gid=yarn 0 0
The Docker client command will draw its configuration from the default location, which is $HOME/.docker/config.json on the NodeManager host. The Docker configuration is where secure repository credentials are stored, so use of the LCE with secure Docker repos is discouraged using this method.
YARN-5428 added support to Distributed Shell for securely supplying the Docker client configuration. See the Distributed Shell help for usage. Support for additional frameworks is planned.
As a work-around, you may manually log the Docker daemon on every NodeManager host into the secure repo using the Docker login command:
docker login [OPTIONS] [SERVER] Register or log in to a Docker registry server, if no server is specified "https://index.docker.io/v1/" is the default. -e, --email="" Email -p, --password="" Password -u, --username="" Username
Note that this approach means that all users will have access to the secure repo.
Hadoop integrates with Docker Trusted Registry via YARN service API. Docker registry can store Docker images on HDFS, S3 or external storage using CSI driver.
NFS Gateway provides capability to mount HDFS as NFS mount point. Docker Registry can configure to write to HDFS mount point using standard file system API.
In hdfs-site.xml, configure NFS configuration:
<property> <name>nfs.exports.allowed.hosts</name> <value>* rw</value> </property> <property> <name>nfs.file.dump.dir</name> <value>/tmp/.hdfs-nfs</value> </property> <property> <name>nfs.kerberos.principal</name><EMAIL_ADDRESS></property> <property> <name>nfs.keytab.file</name> <value>/etc/security/keytabs/nfs.service.keytab</value> </property>
Run NFS Gateway on all datanodes as hdfs user using:
$ $HADOOP_HOME/bin/hdfs --daemon start nfs3
On each datanode, nfs mount point is exposed to /hdfs, using:
# mount -t nfs -o vers=3,proto=tcp,nolock,noacl,sync $DN_IP:/ /hdfs
Where DN_IP is the IP address of the datanode.
Container-executor.cfg is configured to allow trusted Docker images from library.
[docker] docker.privileged-containers.enabled=true docker.trusted.registries=library,registry.docker-registry.registry.example.com:5000 docker.allowed.rw-mounts=/tmp,/usr/local/hadoop/logs,/hdfs
Docker Registry can be started using YARN service: registry.json
{ "name": "docker-registry", "version": "1.0", "kerberos_principal" : { "principal_name"<EMAIL_ADDRESS>"keytab" : "file:///etc/security/keytabs/registry.service.keytab" }, "components" : [ { "name": "registry", "number_of_containers": 1, "artifact": { "id": "registry:latest", "type": "DOCKER" }, "resource": { "cpus": 1, "memory": "256" }, "run_privileged_container": true, "configuration": { "env": { "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE":"true", "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS":"/hdfs/apps/docker/registry:/var/lib/registry" }, "properties": { "docker.network": "host" } } } ] }
YARN service configures docker mounts from /hdfs/apps/docker/registry to /var/lib/registry inside docker container.
yarn app -launch docker-registry /tmp/registry.json
Docker trusted registry is deployed in YARN framework, and the URL to access the registry following Hadoop Registry DNS format:
registry.docker-registry.$USER.$DOMAIN:5000
When docker-registry application reaches STABLE state in YARN, user can push or pull docker images to Docker Trusted Registry by prefix image name with registry.docker-registry.registry.example.com:5000/.
Docker Registry provides its own S3 driver and YAML configuration. YARN service configuration can generate YAML template, and enable direct Docker Registry to S3 storage. This option is the top choice for deploying Docker Trusted Registry on AWS. Configuring Docker registry storage driver to S3 requires mounting /etc/docker/registry/config.yml file (through YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS), which needs to configure an S3 bucket with its corresponding accesskey and secretKey.
Sample config.yml
version: 0.1 log: fields: service: registry http: addr: :5000 storage: cache: blobdescriptor: inmemory s3: accesskey: #AWS_KEY# secretkey: #AWS_SECRET# region: #AWS_REGION# bucket: #AWS_BUCKET# encrypt: #ENCRYPT# secure: #SECURE# chunksize: 5242880 multipartcopychunksize: 33554432 multipartcopymaxconcurrency: 100 multipartcopythresholdsize: 33554432 rootdirectory: #STORAGE_PATH#
Docker Registry can be started using YARN service: registry.json
{ "name": "docker-registry", "version": "1.0", "kerberos_principal" : { "principal_name"<EMAIL_ADDRESS>"keytab" : "file:///etc/security/keytabs/registry.service.keytab" }, "components" : [ { "name": "registry", "number_of_containers": 1, "artifact": { "id": "registry:latest", "type": "DOCKER" }, "resource": { "cpus": 1, "memory": "256" }, "run_privileged_container": true, "configuration": { "env": { "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE":"true", "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS":"<path to config.yml>:/etc/docker/registry/config.yml", }, "properties": { "docker.network": "host" } } } ] }
For further details and parameters that could be configured in the S3 storage driver, please refer https://docs.docker.com/registry/storage-drivers/s3/.
This example assumes that Hadoop is installed to /usr/local/hadoop
.
Additionally, docker.allowed.ro-mounts
in container-executor.cfg
has been updated to include the directories: /usr/local/hadoop,/etc/passwd,/etc/group
.
To submit the pi job to run in Docker containers, run the following commands:
HADOOP_HOME=/usr/local/hadoop YARN_EXAMPLES_JAR=$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar MOUNTS="$HADOOP_HOME:$HADOOP_HOME:ro,/etc/passwd:/etc/passwd:ro,/etc/group:/etc/group:ro" IMAGE_ID="library/openjdk:8" export YARN_CONTAINER_RUNTIME_TYPE=docker export YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$IMAGE_ID export YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=$MOUNTS yarn jar $YARN_EXAMPLES_JAR pi \ -Dmapreduce.map.env.YARN_CONTAINER_RUNTIME_TYPE=docker \ -Dmapreduce.map.env.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=$MOUNTS \ -Dmapreduce.map.env.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$IMAGE_ID \ -Dmapreduce.reduce.env.YARN_CONTAINER_RUNTIME_TYPE=docker \ -Dmapreduce.reduce.env.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=$MOUNTS \ -Dmapreduce.reduce.env.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$IMAGE_ID \ 1 40000
Note that the application master, map tasks, and reduce tasks are configured independently. In this example, we are using the openjdk:8
image for all three.
This example assumes that Hadoop is installed to /usr/local/hadoop
and Spark is installed to /usr/local/spark
.
Additionally, docker.allowed.ro-mounts
in container-executor.cfg
has been updated to include the directories: /usr/local/hadoop,/etc/passwd,/etc/group
.
 In this example, we are using the openjdk:8
image for both.
When Docker support was introduced to Hadoop 2.x, the platform was designed to run existing Hadoop programs inside Docker container. Log redirection and environment setup are integrated with Node Manager. In Hadoop 3.x, Hadoop Docker support extends beyond running Hadoop workload, and support Docker container in Docker native form using ENTRYPOINT from dockerfile. Application can decide to support YARN mode as default or Docker mode as default by defining YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE environment variable. System administrator can also set as default setting for the cluster to make ENTRY_POINT as default mode of operation.
In yarn-site.xml, add YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE to node manager environment white list:
<property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME,YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE</value> </property>
In yarn-env.sh, define:
export YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE=true
There are two requirements when ENTRYPOINT is not used:
/bin/bash
must be available inside the image. This is generally true, however, tiny Docker images (eg. ones which use busybox for shell commands) might not have bash installed. In this case, the following error is displayed:
Container id: container_1561638268473_0015_01_000002 Exit code: 7 Exception message: Launch container failed Shell error output: /usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:235: starting container process caused "exec: \"bash\": executable file not found in $PATH". Shell output: main : command provided 4
find
command must also be available inside the image. Not having find
causes this error:
Container exited with a non-zero exit code 127. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : /tmp/hadoop-systest/nm-local-dir/usercache/hadoopuser/appcache/application_1561638268473_0017/container_1561638268473_0017_01_000002/launch_container.sh: line 44: find: command not found
YARN SysFS is a pseudo file system provided by the YARN framework that exports information about clustering information to Docker container. Cluster information is exported to /hadoop/yarn/sysfs path. This API allows application developer to obtain clustering information without external service dependencies. Custom application master can populate cluster information by calling node manager REST API. YARN service framework automatically populates cluster information to /hadoop/yarn/sysfs/app.json. For more information about YARN service, see: YARN Service.
Docker Container Service Mode runs the container as defined by the image but does not set the user (–user and –group-add). This mode is disabled by default. The administrator sets docker.service-mode.enabled to true in container-executor.cfg under docker section to enable.
Part of a container-executor.cfg which allows docker service mode is below:
yarn.nodemanager.linux-container-executor.group=yarn [docker] module.enabled=true docker.privileged-containers.enabled=true docker.service-mode.enabled=true
Application User can enable or disable service mode at job level by exporting environment variable YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE in the application’s environment with value true or false respectively.


The only thing standing between you and your goal is the story you keep telling yourself as to why you can't achieve it.


7 Logistic Regression - Ex: Maternal Risk Factor for Low Birth Weight Delivery
library(tidyverse)
library(haven) # read in SPSS dataset
library(furniture) # nice table1() descriptives
library(stargazer) # display nice tables: summary & regression
library(texreg) # Convert Regression Output to LaTeX or HTML Tables
library(psych) # contains some useful functions, like headTail
library(car) # Companion to Applied Regression
library(sjPlot) # Quick plots and tables for models
library(pscl) # psudo R-squared function
library(glue) # Interpreted String Literals
7.1 Background
More complex example demonstrating modeling decisions
Another set of data from a study investigating predictors of low birth weight
id
infant’s unique identification number
Dependent variable (DV) or outcome
low
Low birth weight (outcome) (0 = birth weight >2500 g (normal), 1 = birth weight < 2500 g (low))bwt
actual infant birth weight in grams
Indepdentend variables (IV) or predictors
age
Age of mother, in yearslwt
Mother’s weight at last menstrual period, in poundsrace
Race: 1 = White, 2 = Black, 3 = Othersmoke
Smoking status during pregnancy:1 = Yes, 0 = Noptl
History of premature labor: 0 = None, 1 = One, 2 = two, 3 = threeht
History of hypertension: 1 = Yes, 0 = Noui
Uterine irritability: 1 = Yes, 0 = Noftv
Number of physician visits in 1st trimester: 0 = None, 1 = One, … 6 = six
7.1.1 Raw Dataset
The data is saved in a text file (.txt) without any labels.
lowbwt_raw <- read.table("https://raw.githubusercontent.com/CEHS-research/data/master/Regression/lowbwt.txt",
header = TRUE,
sep = "",
na.strings = "NA",
dec = ".",
strip.white = TRUE)
tibble::glimpse(lowbwt_raw)
Observations: 189
Variables: 11
$ id <int> 85, 86, 87, 88, 89, 91, 92, 93, 94, 95, 96, 97, 98, 99, ...
$ low <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
$ age <int> 19, 33, 20, 21, 18, 21, 22, 17, 29, 26, 19, 19, 22, 30, ...
$ lwt <int> 182, 155, 105, 108, 107, 124, 118, 103, 123, 113, 95, 15...
$ race <int> 2, 3, 1, 1, 1, 3, 1, 3, 1, 1, 3, 3, 3, 3, 1, 1, 2, 1, 3,...
$ smoke <int> 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0,...
$ ptl <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,...
$ ht <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,...
$ ui <int> 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1,...
$ ftv <int> 0, 3, 1, 2, 0, 0, 1, 1, 1, 0, 0, 1, 0, 2, 0, 0, 0, 3, 0,...
$ bwt <int> 2523, 2551, 2557, 2594, 2600, 2622, 2637, 2637, 2663, 26...
7.1.2 Declare Factors
lowbwt_clean <- lowbwt_raw %>%
dplyr::mutate(id = factor(id)) %>%
dplyr::mutate(low = factor(low,
levels = c(0, 1),
labels = c("birth weight >2500 g (normal)",
"birth weight < 2500 g (low)"))) %>%
dplyr::mutate(race = factor(race,
levels = 1:3,
labels = c("White",
"Black",
"Other"))) %>%
dplyr::mutate(ptl_any = as.numeric(ptl > 0)) %>% # colapse into 0 = none vs. 1 = at least one
dplyr::mutate(ptl = factor(ptl)) %>% # declare the number of pre-term labors to be a factor: 0, 1, 2, 3
dplyr::mutate_at(vars(smoke, ht, ui, ptl_any), # declare all there variables to be factors with the same two levels
factor,
levels = 0:1,
labels = c("No", "Yes"))
Display the structure of the ‘clean’ version of the dataset
'data.frame': 189 obs. of 12 variables:
$ id : Factor w/ 189 levels "4","10","11",..: 60 61 62 63 64 65 66 67 68 69 ...
$ low : Factor w/ 2 levels "birth weight >2500 g (normal)",..: 1 1 1 1 1 1 1 1 1 1 ...
$ age : int 19 33 20 21 18 21 22 17 29 26 ...
$ lwt : int 182 155 105 108 107 124 118 103 123 113 ...
$ race : Factor w/ 3 levels "White","Black",..: 2 3 1 1 1 3 1 3 1 1 ...
$ smoke : Factor w/ 2 levels "No","Yes": 1 1 2 2 2 1 1 1 2 2 ...
$ ptl : Factor w/ 4 levels "0","1","2","3": 1 1 1 1 1 1 1 1 1 1 ...
$ ht : Factor w/ 2 levels "No","Yes": 1 1 1 1 1 1 1 1 1 1 ...
$ ui : Factor w/ 2 levels "No","Yes": 2 1 1 2 2 1 1 1 1 1 ...
$ ftv : int 0 3 1 2 0 0 1 1 1 0 ...
$ bwt : int 2523 2551 2557 2594 2600 2622 2637 2637 2663 2665 ...
$ ptl_any: Factor w/ 2 levels "No","Yes": 1 1 1 1 1 1 1 1 1 1 ...
7.2 Exploratory Data Analysis
# Knit to Website: output = "html"
# Knit to PDF: output = "latex2"
# View on Screen: output = ""text", or "markdown", "html"
lowbwt_clean %>%
furniture::table1("Age, years" = age,
"Weight, pounds" = lwt,
"Race" = race,
"Smoking During pregnancy" = smoke,
"History of Premature Labor, any" = ptl_any,
"History of Premature Labor, number" = ptl,
"History of Hypertension" = ht,
"Uterince Irritability" = ui,
"1st Tri Dr Visits" = ftv,
splitby = ~ low,
test = TRUE,
output = "html")
| birth weight >2500 g (normal) | birth weight < 2500 g (low) | P-Value | |
|---|---|---|---|
| n = 130 | n = 59 | ||
| Age, years | 0.103 | ||
| 23.7 (5.6) | 22.3 (4.5) | ||
| Weight, pounds | 0.02 | ||
| 133.3 (31.7) | 122.1 (26.6) | ||
| Race | 0.082 | ||
| White | 73 (56.2%) | 23 (39%) | |
| Black | 15 (11.5%) | 11 (18.6%) | |
| Other | 42 (32.3%) | 25 (42.4%) | |
| Smoking During pregnancy | 0.04 | ||
| No | 86 (66.2%) | 29 (49.2%) | |
| Yes | 44 (33.8%) | 30 (50.8%) | |
| History of Premature Labor, any | <.001 | ||
| No | 118 (90.8%) | 41 (69.5%) | |
| Yes | 12 (9.2%) | 18 (30.5%) | |
| History of Premature Labor, number | <.001 | ||
| 0 | 118 (90.8%) | 41 (69.5%) | |
| 1 | 8 (6.2%) | 16 (27.1%) | |
| 2 | 3 (2.3%) | 2 (3.4%) | |
| 3 | 1 (0.8%) | 0 (0%) | |
| History of Hypertension | 0.076 | ||
| No | 125 (96.2%) | 52 (88.1%) | |
| Yes | 5 (3.8%) | 7 (11.9%) | |
| Uterince Irritability | 0.035 | ||
| No | 116 (89.2%) | 45 (76.3%) | |
| Yes | 14 (10.8%) | 14 (23.7%) | |
| 1st Tri Dr Visits | 0.389 | ||
| 0.8 (1.1) | 0.7 (1.0) |
7.3 Logistic Regression - Simple, un-adjusted
low1.age <- glm(low ~ age, family=binomial(logit), data=lowbwt_clean)
low1.lwt <- glm(low ~ lwt, family=binomial(logit), data=lowbwt_clean)
low1.race <- glm(low ~ race, family=binomial(logit), data=lowbwt_clean)
low1.smoke <- glm(low ~ smoke, family=binomial(logit), data=lowbwt_clean)
low1.ptl <- glm(low ~ ptl_any, family=binomial(logit), data=lowbwt_clean)
low1.ht <- glm(low ~ ht, family=binomial(logit), data=lowbwt_clean)
low1.ui <- glm(low ~ ui, family=binomial(logit), data=lowbwt_clean)
low1.ftv <- glm(low ~ ftv, family=binomial(logit), data=lowbwt_clean)
Note: the parameter estimates here are for the LOGIT scale, not the odds ration (OR) or even the probability.
# Knit to Website: texreg::htmlreg()
# Knit to PDF: texreg::texreg()
# View on Screen: texreg::screenreg()
texreg::htmlreg(list(low1.age, low1.lwt, low1.race, low1.smoke),
custom.model.names = c("Age", "Weight", "Race", "Smoker"),
caption = "Simple, Unadjusted Logistic Regression: Models 1-4",
caption.above = TRUE)
| Age | Weight | Race | Smoker | ||
|---|---|---|---|---|---|
| (Intercept) | 0.38 | 1.00 | -1.15*** | -1.09*** | |
| (0.73) | (0.79) | (0.24) | (0.21) | ||
| age | -0.05 | ||||
| (0.03) | |||||
| lwt | -0.01* | ||||
| (0.01) | |||||
| raceBlack | 0.84 | ||||
| (0.46) | |||||
| raceOther | 0.64 | ||||
| (0.35) | |||||
| smokeYes | 0.70* | ||||
| (0.32) | |||||
| AIC | 235.91 | 232.69 | 235.66 | 233.80 | |
| BIC | 242.40 | 239.17 | 245.39 | 240.29 | |
| Log Likelihood | -115.96 | -114.35 | -114.83 | -114.90 | |
| Deviance | 231.91 | 228.69 | 229.66 | 229.80 | |
| Num. obs. | 189 | 189 | 189 | 189 | |
| p < 0.001, p < 0.01, p < 0.05 |
# Knit to Website: texreg::htmlreg()
# Knit to PDF: texreg::texreg()
# View on Screen: texreg::screenreg()
texreg::htmlreg(list(low1.ptl, low1.ht, low1.ui, low1.ftv),
custom.model.names = c("Pre-Labor", "Hypertension", "Uterine", "Visits"),
caption = "Simple, Unadjusted Logistic Regression: Models 5-8",
caption.above = TRUE)
| Pre-Labor | Hypertension | Uterine | Visits | ||
|---|---|---|---|---|---|
| (Intercept) | -1.06*** | -0.88*** | -0.95*** | -0.69*** | |
| (0.18) | (0.17) | (0.18) | (0.19) | ||
| ptl_anyYes | 1.46*** | ||||
| (0.41) | |||||
| htYes | 1.21* | ||||
| (0.61) | |||||
| uiYes | 0.95* | ||||
| (0.42) | |||||
| ftv | -0.14 | ||||
| (0.16) | |||||
| AIC | 225.90 | 234.65 | 233.60 | 237.90 | |
| BIC | 232.38 | 241.13 | 240.08 | 244.38 | |
| Log Likelihood | -110.95 | -115.32 | -114.80 | -116.95 | |
| Deviance | 221.90 | 230.65 | 229.60 | 233.90 | |
| Num. obs. | 189 | 189 | 189 | 189 | |
| p < 0.001, p < 0.01, p < 0.05 |
7.4 Logistic Regression - Multivariate, with Main Effects Only
Main-effects multiple logistic regression model
low1_1 <- glm(low ~ age + lwt + race + smoke + ptl_any + ht + ui,
family = binomial(logit),
data = lowbwt_clean)
summary(low1_1)
Call:
glm(formula = low ~ age + lwt + race + smoke + ptl_any + ht +
ui, family = binomial(logit), data = lowbwt_clean)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.6459 -0.7992 -0.5103 0.9388 2.2018
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 0.63691 1.23028 0.518 0.60467
age -0.03775 0.03781 -0.998 0.31808
lwt -0.01491 0.00704 -2.118 0.03419 *
raceBlack 1.21274 0.53248 2.278 0.02275 *
raceOther 0.80412 0.44843 1.793 0.07294 .
smokeYes 0.84640 0.40806 2.074 0.03806 *
ptl_anyYes 1.22175 0.46301 2.639 0.00832 **
htYes 1.83869 0.70324 2.615 0.00893 **
uiYes 0.71113 0.46311 1.536 0.12465
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 234.67 on 188 degrees of freedom
Residual deviance: 196.83 on 180 degrees of freedom
AIC: 214.83
Number of Fisher Scoring iterations: 4
7.5 Logistic Regression - Multivariate, with Interactions
Before removing non-significant main effects, test plausible interactions
Try interactions between age and lwt, age and smoke, lwt and smoke, 1 at a time
7.5.1 Age and Weight
low1_2 <- glm(low ~ age + lwt + race + smoke + ptl_any + ht + ui + age:lwt,
family = binomial(logit),
data = lowbwt_clean)
summary(low1_2)
Call:
glm(formula = low ~ age + lwt + race + smoke + ptl_any + ht +
ui + age:lwt, family = binomial(logit), data = lowbwt_clean)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.6426 -0.8004 -0.5163 0.9400 2.1989
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 1.1396293 4.0443041 0.282 0.77811
age -0.0597273 0.1724339 -0.346 0.72906
lwt -0.0188377 0.0309225 -0.609 0.54240
raceBlack 1.2071359 0.5341716 2.260 0.02383 *
raceOther 0.7977750 0.4505381 1.771 0.07661 .
smokeYes 0.8433655 0.4083953 2.065 0.03892 *
ptl_anyYes 1.2260463 0.4642795 2.641 0.00827 **
htYes 1.8389418 0.7034162 2.614 0.00894 **
uiYes 0.7142688 0.4642468 1.539 0.12391
age:lwt 0.0001718 0.0013146 0.131 0.89600
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 234.67 on 188 degrees of freedom
Residual deviance: 196.82 on 179 degrees of freedom
AIC: 216.82
Number of Fisher Scoring iterations: 5
<IP_ADDRESS> Compare Model Fits vs. Likelihood Ratio Test
# A tibble: 2 x 5
`Resid. Df` `Resid. Dev` Df Deviance `Pr(>Chi)`
<dbl> <dbl> <dbl> <dbl> <dbl>
1 180 197. NA NA NA
2 179 197. 1 0.0170 0.896
<IP_ADDRESS> Type II Analysis of Deviance Table
# A tibble: 8 x 3
`LR Chisq` Df `Pr(>Chisq)`
<dbl> <dbl> <dbl>
1 1.02 1 0.313
2 5.00 1 0.0254
3 6.27 2 0.0436
4 4.38 1 0.0364
5 7.13 1 0.00758
6 7.18 1 0.00738
7 2.33 1 0.127
8 0.0170 1 0.896
<IP_ADDRESS> Type III Analysis of Deviance Table
# A tibble: 8 x 3
`LR Chisq` Df `Pr(>Chisq)`
<dbl> <dbl> <dbl>
1 0.119 1 0.730
2 0.372 1 0.542
3 6.27 2 0.0436
4 4.38 1 0.0364
5 7.13 1 0.00758
6 7.18 1 0.00738
7 2.33 1 0.127
8 0.0170 1 0.896
7.5.2 Age and Smoking
low1_3 <- glm(low ~ age + lwt + race + smoke + ptl_any + ht + ui + age:smoke,
family = binomial(logit),
data = lowbwt_clean)
summary(low1_3)
Call:
glm(formula = low ~ age + lwt + race + smoke + ptl_any + ht +
ui + age:smoke, family = binomial(logit), data = lowbwt_clean)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.7020 -0.8016 -0.4970 0.9055 2.2124
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 1.311029 1.483078 0.884 0.37670
age -0.068293 0.053865 -1.268 0.20485
lwt -0.014701 0.007001 -2.100 0.03575 *
raceBlack 1.126482 0.543684 2.072 0.03827 *
raceOther 0.768241 0.451199 1.703 0.08863 .
smokeYes -0.601286 1.782234 -0.337 0.73583
ptl_anyYes 1.197295 0.460992 2.597 0.00940 **
htYes 1.860851 0.705311 2.638 0.00833 **
uiYes 0.783999 0.472451 1.659 0.09703 .
age:smokeYes 0.063744 0.076634 0.832 0.40552
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 234.67 on 188 degrees of freedom
Residual deviance: 196.14 on 179 degrees of freedom
AIC: 216.14
Number of Fisher Scoring iterations: 5
<IP_ADDRESS> Compare Model Fits vs. Likelihood Ratio Test
# A tibble: 2 x 5
`Resid. Df` `Resid. Dev` Df Deviance `Pr(>Chi)`
<dbl> <dbl> <dbl> <dbl> <dbl>
1 180 197. NA NA NA
2 179 196. 1 0.699 0.403
7.5.3 Weight and Smoking
low1_4 <- glm(low ~ age + lwt + race + smoke + ptl_any + ht + ui + lwt:smoke,
family = binomial(logit),
data = lowbwt_clean)
summary(low1_4)
Call:
glm(formula = low ~ age + lwt + race + smoke + ptl_any + ht +
ui + lwt:smoke, family = binomial(logit), data = lowbwt_clean)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.6816 -0.7874 -0.5251 0.8876 2.2365
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 1.74840 1.59439 1.097 0.27282
age -0.03768 0.03809 -0.989 0.32259
lwt -0.02362 0.01077 -2.193 0.02828 *
raceBlack 1.23110 0.53358 2.307 0.02104 *
raceOther 0.71646 0.45183 1.586 0.11281
smokeYes -1.13566 1.75557 -0.647 0.51770
ptl_anyYes 1.26472 0.46488 2.721 0.00652 **
htYes 1.74326 0.70738 2.464 0.01373 *
uiYes 0.80121 0.47044 1.703 0.08855 .
lwt:smokeYes 0.01555 0.01344 1.157 0.24740
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 234.67 on 188 degrees of freedom
Residual deviance: 195.44 on 179 degrees of freedom
AIC: 215.44
Number of Fisher Scoring iterations: 5
<IP_ADDRESS> Compare Model Fits vs. Likelihood Ratio Test
# A tibble: 2 x 5
`Resid. Df` `Resid. Dev` Df Deviance `Pr(>Chi)`
<dbl> <dbl> <dbl> <dbl> <dbl>
1 180 197. NA NA NA
2 179 195. 1 1.39 0.238
7.6 Logistic Regression - Multivariate, Simplify
No interactions are significant Remove non-significant main effects
7.6.1 Remove the least significant perdictor: ui
low1_5 <- glm(low ~ age + lwt + race + smoke + ptl_any + ht,
family = binomial(logit),
data = lowbwt_clean)
summary(low1_5)
Call:
glm(formula = low ~ age + lwt + race + smoke + ptl_any + ht,
family = binomial(logit), data = lowbwt_clean)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.6533 -0.8202 -0.5299 0.9709 2.1982
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 0.924910 1.202693 0.769 0.44187
age -0.042784 0.037567 -1.139 0.25475
lwt -0.015436 0.007044 -2.191 0.02843 *
raceBlack 1.168452 0.532577 2.194 0.02824 *
raceOther 0.814620 0.442740 1.840 0.06578 .
smokeYes 0.858332 0.404787 2.120 0.03397 *
ptl_anyYes 1.333970 0.457573 2.915 0.00355 **
htYes 1.740511 0.703104 2.475 0.01331 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 234.67 on 188 degrees of freedom
Residual deviance: 199.15 on 181 degrees of freedom
AIC: 215.15
Number of Fisher Scoring iterations: 4
7.7 Logistic Regression - Multivariate, Final Model
Since the mother’s age is theoretically a meaningful variable, it should probably be retained.
Revise so that age is interpreted in 5-year and lwt in 20 lb increments and the intercept has meaning.
low1_6 <- glm(low ~ I((age - 20)/5) + I((lwt - 125)/20) + race + smoke + ptl_any + ht,
family = binomial(logit),
data = lowbwt_clean)
summary(low1_6)
Call:
glm(formula = low ~ I((age - 20)/5) + I((lwt - 125)/20) + race +
smoke + ptl_any + ht, family = binomial(logit), data = lowbwt_clean)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.6533 -0.8202 -0.5299 0.9709 2.1982
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.8603 0.4092 -4.546 5.48e-06 ***
I((age - 20)/5) -0.2139 0.1878 -1.139 0.25475
I((lwt - 125)/20) -0.3087 0.1409 -2.191 0.02843 *
raceBlack 1.1685 0.5326 2.194 0.02824 *
raceOther 0.8146 0.4427 1.840 0.06578 .
smokeYes 0.8583 0.4048 2.120 0.03397 *
ptl_anyYes 1.3340 0.4576 2.915 0.00355 **
htYes 1.7405 0.7031 2.475 0.01331 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 234.67 on 188 degrees of freedom
Residual deviance: 199.15 on 181 degrees of freedom
AIC: 215.15
Number of Fisher Scoring iterations: 4
7.7.1 Several \(R^2\) measures with the pscl::pR2()
function
llh llhNull G2 McFadden r2ML
-99.5757045 -117.3359981 35.5205871 0.1513627 0.1713353
r2CU
0.2409463
7.7.2 Parameter Estiamtes Table
<IP_ADDRESS> Using texreg::screenreg()
Default: parameters are in terms of the ‘logit’ or log odds ratio
# Knit to Website: texreg::htmlreg()
# Knit to PDF: texreg::texreg()
# View on Screen: texreg::screenreg()
texreg::htmlreg(low1_6)
| Model 1 | ||
|---|---|---|
| (Intercept) | -1.86*** | |
| (0.41) | ||
| I((age - 20)/5) | -0.21 | |
| (0.19) | ||
| I((lwt - 125)/20) | -0.31* | |
| (0.14) | ||
| raceBlack | 1.17* | |
| (0.53) | ||
| raceOther | 0.81 | |
| (0.44) | ||
| smokeYes | 0.86* | |
| (0.40) | ||
| ptl_anyYes | 1.33** | |
| (0.46) | ||
| htYes | 1.74* | |
| (0.70) | ||
| AIC | 215.15 | |
| BIC | 241.09 | |
| Log Likelihood | -99.58 | |
| Deviance | 199.15 | |
| Num. obs. | 189 | |
| p < 0.001, p < 0.01, p < 0.05 |
The texreg
package uses an intermediate function called extract()
to extract information for the model and then put it in the right places in the table. We can invervene by writing our own extract_exp()
function to use instead.
extract_exp <- function(fit_glm){
beta = coef(fit_glm)
betaci = confint(fit_glm)
fit_glm_exp = texreg::extract(fit_glm)
fit_glm_exp@coef = exp(beta)
<EMAIL_ADDRESS>= exp(betaci[, 1])
<EMAIL_ADDRESS>= exp(betaci[, 2])
return(fit_glm_exp)
}
# Knit to Website: texreg::htmlreg()
# Knit to PDF: texreg::texreg()
# View on Screen: texreg::screenreg()
texreg::htmlreg(extract_exp(low1_6),
custom.coef.names = c("BL: 125 lb, 20 yr old White Mother",
"Additional 5 years older",
"Additional 20 lbs pre-pregnancy",
"Race: Black vs. White",
"Race: Other vs. White",
"Smoking During pregnancy",
"History of Any Premature Labor",
"History of Hypertension"),
custom.model.names = "OR, Low Birth Weight",
single.row = TRUE,
custom.note = "* The value of '1' is outside the confidence interval for the OR")
| OR, Low Birth Weight | ||
|---|---|---|
| BL: 125 lb, 20 yr old White Mother | 0.16 [0.07; 0.33]* | |
| Additional 5 years older | 0.81 [0.55; 1.16]* | |
| Additional 20 lbs pre-pregnancy | 0.73 [0.55; 0.95]* | |
| Race: Black vs. White | 3.22 [1.13; 9.31]* | |
| Race: Other vs. White | 2.26 [0.96; 5.50]* | |
| Smoking During pregnancy | 2.36 [1.08; 5.32]* | |
| History of Any Premature Labor | 3.80 [1.57; 9.53]* | |
| History of Hypertension | 5.70 [1.49; 24.76]* | |
| AIC | 215.15 | |
| BIC | 241.09 | |
| Log Likelihood | -99.58 | |
| Deviance | 199.15 | |
| Num. obs. | 189 | |
| * The value of ‘1’ is outside the confidence interval for the OR |
<IP_ADDRESS> Using sjPlot::tab_model()
Parameters Exponentiated:
sjPlot::tab_model(low1_6,
emph.p = TRUE,
pred.labels = c("BL: 125 lb, 20 yr old White Mother",
"Additional 5 years old",
"Additional 20 lbs pre-pregnancy",
"Race: Black vs. White",
"Race: Other vs. White",
"Smoking During pregnancy",
"History of Any Premature Labor",
"History of Hypertension"))
| low | |||
|---|---|---|---|
| Predictors | Odds Ratios | CI | p |
| BL: 125 lb, 20 yr old White Mother | 0.16 | 0.07 – 0.35 | <0.001 |
| Additional 5 years old | 0.81 | 0.56 – 1.17 | 0.255 |
| Additional 20 lbs pre-pregnancy | 0.73 | 0.56 – 0.97 | 0.028 |
| Race: Black vs. White | 3.22 | 1.13 – 9.14 | 0.028 |
| Race: Other vs. White | 2.26 | 0.95 – 5.38 | 0.066 |
| Smoking During pregnancy | 2.36 | 1.07 – 5.22 | 0.034 |
| History of Any Premature Labor | 3.80 | 1.55 – 9.31 | 0.004 |
| History of Hypertension | 5.70 | 1.44 – 22.61 | 0.013 |
| Observations | 189 | ||
| R2 Tjur | 0.180 |
7.7.3 Marginal Model Plot
<IP_ADDRESS> Focus on: Mother’s Age, weight, and race
effects::Effect(focal.predictors = c("age", "lwt", "race"),
mod = low1_6,
xlevels = list(age = c(20, 30, 40),
lwt = seq(from = 80, to = 250, by = 5))) %>%
data.frame() %>%
dplyr::mutate(age_labels = glue("Mother Age: {age}")) %>%
ggplot(aes(x = lwt,
y = fit)) +
geom_line(aes(color = race,
linetype = race),
size = 1) +
theme_bw() +
facet_grid(.~ age_labels) +
labs(title = "Risk of Low Birth Weight",
subtitle = "Illustates risk given mother is a non-smoker, without a history of pre-term labor or hypertension",
x = "Mother's Weight Pre-Pregnancy, pounds",
y = "Predicted Probability\nBaby has Low Birth Weight (< 2500 grams)",
color = "Mother's Race",
linetype = "Mother's Race") +
theme(legend.position = c(1, 1),
legend.justification = c(1.1, 1.1),
legend.background = element_rect(color = "black"),
legend.key.width = unit(2, "cm")) +
scale_linetype_manual(values = c("longdash", "dotted", "solid")) +
scale_color_manual(values = c( "coral2", "dodger blue", "gray50"))
<IP_ADDRESS> Focus on: Mother’s weight and smothing status during pregnancy, as well as history of any per-term labor and hypertension
effects::Effect(focal.predictors = c("lwt", "smoke", "ptl_any", "ht"),
fixed.predictors = list(age = 20),
mod = low1_6,
xlevels = list(lwt = seq(from = 80, to = 250, by = 5))) %>%
data.frame() %>%
dplyr::mutate(smoke = forcats::fct_rev(smoke)) %>%
dplyr::mutate(ptl_any_labels = glue("History of Preterm Labor: {ptl_any}")) %>%
dplyr::mutate(ht_labels = glue("History of Hypertension: {ht}") %>% forcats::fct_rev()) %>%
ggplot(aes(x = lwt,
y = fit)) +
geom_line(aes(color = smoke,
linetype = smoke),
size = 1) +
theme_bw() +
facet_grid(ht_labels ~ ptl_any_labels) +
labs(title = "Risk of Low Birth Weight",
subtitle = "Illustates risk given the mother is 20 years old and white",
x = "Mother's Weight Pre-Pregnancy, pounds",
y = "Predicted Probability\nBaby has Low Birth Weight (< 2500 grams)",
color = "Mother Smoked",
linetype = "Mother Smoked") +
theme(legend.position = c(1, .5),
legend.justification = c(1.1, 1.15),
legend.background = element_rect(color = "black"),
legend.key.width = unit(1.5, "cm")) +
scale_linetype_manual(values = c("longdash", "solid")) +
scale_color_manual(values = c( "coral2", "dodger blue"))


Crystal Collector
How to Play:
Click on any one of the four crystals to generate a number
The numbers will be added together, and any crystal may be selected multiple times
The goal is to match the Goal Number without going over
If the player reaches the Goal Number without going over, the player wins
To start a new round, click "New Game"
Going over the Goal Number will result in "Game Over" - Good Luck!
Goal Number:
Current Number:
Wins:
Losses:
New Game


{{ define "converting" }} {{ $title := "Converting Book" }} {{ template "header" $title }} {{ template "searchform" }}
Your file is being converted. Please wait.
This page will periodically reload to check progress. Once the conversion is complete, the download will start.
Time till next reload:
{{ template "footer" }} {{ end }}


Toggle navigation
RiceBook
Main Page
Ricebook User Profile
User Profile Picture
Upload a new image
User Current Info
Cool Guy
<EMAIL_ADDRESS>123-123-1234
77005
Update Info
Display Name :
Email Address :
Phone Number :
Zipcode :


If you think a movie has exceptional cinematography, submit it and it may appear in The Cinematic Photo
Submit


Advanced Method for Using DSRM Credentials (Windows 2012 R2)
What’s really interesting about this account is that since it’s a valid local administrator account, it can be used to authenticate over the network to the DC (if the DsrmAdminLogonBehavior regkey is set to 2) . Furthermore, the attacker doesn’t need to know the actual password, all that’s required is the password hash. This means that once an attacker has the password hash for the DSRM account, it can be “passed” to the DC for valid admin access to the DC across the network using Pass-the-Hash. This was tested successfully in limited lab testing on a Windows Server 2008 R2 & 2012 R2 Domain Controllers.
Mimikatz “privilege::debug” “sekurlsa::pth /domain:ADSDC03 /user:Administrator /ntlm:7c08d63a2f48f045971bc2236ed3f3ac” exit


s6
Software
skarnet.org
Why another supervision suite ?
Supervision suites are becoming quite common. Today, we already have:
- Good (?) old System V init, which can be made to supervise services if you perform /etc/inittab voodoo.
BSD init can also be used the same way with the /etc/ttys file, but for some reason, nobody among BSD
developers is using /etc/ttys to this purpose, so I won't consider BSD init here.
- daemontools, the pioneer
- daemontools-encore, Bruce Guenter's upgrade to daemontools
- runit, Gerrit Pape's suite, well-integrated with Debian
- perp, Wayne Marshall's take on supervision
- Integrated init systems providing a lot of features, process supervision being one of them.
For instance, Upstart, MacOS X's
launchd,
and Fedora's systemd.
Why is s6 needed ? What does it do differently ? Here are the criteria I used.
Supervision suites should not wake up unless notified.
- System V init fails the test: it wakes up every 5 seconds, for the reason that
/dev/initctl might have changed.
m(
- daemontools fails the test: it wakes up every 5 seconds to check for new services.
- daemontools-encore does the same.
- the current version of runit fails the test: it wakes up every 14 seconds. But this is a workaround for a bug in some Linux kernels;
there is no design flaw in runit that prevents it from passing the test.
- perp works.
- Upstart works. I have no idea what other integrated init systems do: it's much too difficult to strace them
to see exactly where they're spending their time, and when it is possible, the trace output is so big that it's
hard to extract any valuable information from it.
- s6 works. The -t option to s6-svscan makes it check its services
with a configurable timeout; by default, this timeout is infinite, i.e. it
never wakes up unless it receives a command via s6-svscanctl.
Supervision suites should provide a program that can run as process 1.
- System V init is process 1, so no problem here.
- Integrated init systems, by definition, provide a process 1.
- daemontools was not designed to take over init, although
it can be made to work with
enough hacking skills. Same thing with daemontools-encore.
- runit provides an init functionality, but the mechanism is
separate from the supervision itself; the runit process, not the
runsvdir process, runs as process 1. This lengthens the supervision
chain.
- perp was not designed to run as process 1. It probably could be made to work too
without too much trouble.
- s6-svscan was designed from the start to be run as process 1, although it
does not have to.
Supervision suites should be bug-free, lightweight and easy to understand.
- daemontools, daemontools-encore, runit and perp all qualify. All of this is excellent quality
code, unsurprisingly.
- System V init is understandable, and reasonably lightweight; but it is still
too big for what it does - poorly. The /etc/inittab file needs to be parsed;
that parser has to be in process 1. There is support in process 1 for the whole
"runlevel" concept, which is a primitive form of service management. The same
executable handles all 3 stages of the machine's lifetime and does not separate
them properly. All in all, System V init does its job, but is showing its age
and nowadays we know much better designs.
- This is where integrated init systems fail, hard. By wanting to organize
the way a the machine is operated - so, machine state management - in the
same package as the init and process supervision system, they add
incredible complexity where it does not belong.
- Upstart uses ptrace to watch its children fork(), and links
process 1 against libdbus. This is insane.
Process 1 should be absolutely stable, it should be guaranteed
to never crash, so the whole of its source code should be under control. At
Upstart's level of complexity, those goals are outright impossible to achieve,
so this approach is flawed by design. It is a shame, because the concepts
and ideas behind Upstart are good and sound; it's the
implementation choices that are its downfall.
- launchd suffers from the same kind of problems. Example: Services
running under launchd must be configured
using
XML; the launchctl process interprets the XML, converts it into a
key-value store (which is strictly
less powerful than XML, so why do they even use XML in the first place?)
and sends it to launchd via a Mach-specific IPC. Process 1 needs to be
linked against the library that handles the Mach IPC, it needs to decode
the key-value store, and use it to run and supervise a daemon. And it
needs to keep everything in memory. This is
a lot more complex and resource-consuming than it needs to be.
- systemd is much, much worse than the other ones, and a real danger
for the future of GNU/Linux. I have a special page
dedicated to it.
What those systems fail to recognize is that process supervision, rooted in
process 1, is a good thing, and machine management is also a good thing, but
those are two different functions, and a good init system
needs, and should, only provide process supervision, in
order to keep such a crucial piece of code as easy to maintain as possible.
Machine management can be added on top of a process supervision
suite, in a different package, and it has nothing to do with process 1.
- s6, which has been designed with embedded environments in mind, tries
harder than anyone to pass this. It tries so hard that s6-svscan
and s6-supervise, the two long-running programs that make the
supervision chain, do not even allocate heap memory, and their main
program source files are less than 500 lines long.
Supervision suites should provide a basis for high-level service management.
- Neither System V init, daemontools, runit or perp
provides any hooks to wait for a service to go up or down. runit provides a
waiting mechanism, but it's based on polling, and the ./check script
has to be manually written for every service.
- daemontools-encore qualifies: the notify script can be used for
inter-service communication. But it's just a hook: all the real notification
work has to be done by the notify script itself, no notification framework is
provided.
- Integrated init systems provide high-level service management
themselves. Again, this is not good design: service management has nothing
to do with init or process supervision, and should be implemented on top
of it, not as a part of it.
- s6 comes with an event notification
library, and command-line tools based on this library, thus providing a simple
API for future service management tools to build upon.
Artistic considerations
- s6-svscan and s6-supervise are entirely asynchronous.
Even during trouble (full process table, for instance), they'll remain reactive
and instantly respond to commands they may receive. s6-supervise has
even been implemented as a full deterministic finite automaton, to ensure it
always does the right thing under any circumstance. Other supervision suites
do not achieve that for now.
- daemontools' svscan
maintains an open pipe between a daemon and its logger, so even if the daemon,
the logger, and both
supervise processes
die, the pipe is still the same so no logs are lost, ever, unless
svscan itself dies.
- runit has only one supervisor, runsv,
for both a daemon and its logger. The pipe is maintained by runsv.
If the runsv process dies, the pipe disappears and logs are lost.
So, runit does not offer as strong a guarantee as daemontools.
- perp has only one process, perpd,
acting both as a "daemon and logger supervisor" (like runsv) and as a
"service directory scanner" (like runsvdir). It maintains the pipes
between the daemons and their respective loggers. If perpd dies, everything
is lost. Since perpd cannot be run as process 1, this is a possible SPOF for
a perp installation; however, perpd is well-written and has virtually no risk of
dying, especially compared to process 1 behemoths provided by integrated
init systems.
- Besides, the runsv model, which has to handle both a daemon
and its logger, is more complex than the supervise model (which
only has to handle a daemon). Consequently, the runsvdir model is
simpler than the svscan model, but there is only one svscan
instance when there are several runsvs and supervises.
The perpd model is obviously the most complex; while very understandable,
perpd is unarguably harder to maintain than the other two.
- So, to achieve maximum simplicity and code reuse, and minimal memory
footprint, s6's design is close to daemontools' one.
And when s6-svscan is run as process 1,
pipes between daemons and loggers are never lost.
Conclusion
All in all, I believe that s6 offers the best overall implementation of a
supervision suite as it should be designed. At worst, it's just another
take on daemontools with a reliable
base library and a few nifty features.


Here is the information you entered:
To enter another email address, click on the Back button in your browser or the Return button shown below.


|
Groovy 2.1.7 | |||||||
| PREV NEXT | FRAMES NO FRAMES |
| Interface Summary | |
|---|---|
| AntlrASTProcessor | An interface for processing antlr AST objects |
| SourceInfo |
| Class Summary | |
|---|---|
| AntlrASTProcessSnippets | Process to decorate antlr AST with ending line/col info, and if possible the snipppet of source from the start/end line/col for each node. |
| AntlrParserPlugin | A parser plugin which adapts the JSR Antlr Parser to the Groovy runtime |
| AntlrParserPluginFactory | @version $Revision$ |
| EnumHelper | |
| GroovySourceAST | We have an AST subclass so we can track source information. |
| GroovySourceToken | This is a Token sub class to track line information |
| LexerFrame | @author Santhosh Kumar T |
| LineColumn | An object representing a line and column position |
| SourceBuffer | A simple buffer that provides line/col access to chunks of source code held within itself. |
| UnicodeEscapingReader | Translates GLS-defined unicode escapes into characters. |
| UnicodeLexerSharedInputState | GRECLIPSE-805 Support for unicode escape sequences |
| Exception Summary | |
|---|---|
| ASTParserException | Thrown when trying to parse the AST |
| ASTRuntimeException | @author James Strachan |
Copyright © 2003-2013 The Codehaus. All rights reserved.


Enter how many passwords you want and how long you want them to be.
set of passwords: 8 this prints 8 passwords to your screen
length of the passwords for example 25
type /checkmy/"Password" at the end of the url for example: "riseofrice.de:8080/checkmy/12345678Ab!"


Preparation time
less than 30 mins
Cooking time
10 to 30 mins
Serves
Serves 4
Give meatballs the Indian treatment in this easy curry recipe. Serve with plenty of veggies for a healthy dinner.
Each serving provides 220kcal, 29g protein, 8g carbohydrate (of which 6g sugars), 8g fat (of which 3g saturates), 1.9g fibre and 0.3g salt.
Mix the beef mince, garlic, half of the onion, pepper, garam masala and egg together in a bowl. Using your hands, shape the meat mixture into 24 balls.
Preheat the grill to hot. Line the bottom of a grill tray with aluminium foil and place the meatballs on the grill tray. (This will allow the fat to drain off into the lined tray, saving on washing up.) Grill the meatballs under a medium heat for 10-15 minutes, turning occasionally, or until nicely browned.
Heat the rapeseed oil in a frying pan, and add the cumin seeds and the remaining onion. Cook for 4-5 minutes, or until softened, then stir in the ginger.
Add the chopped tomatoes, tomato purée and turmeric and cook for 10 minutes or until the sauce has thickened.
Add the cooked meatballs, reduce the heat and simmer for 20 minutes. Add 60-90ml/2-3fl oz water if the curry is a bit dry. Serve with rice.


The virtualization extension enables row virtualization for the ko-grid. Virtualization is a common optimization when displaying many records, only a few of which fit on the screen at one time. Only those which will be visible in the view port are added to the DOM. This increases performance while reducing memory consumption. Also, when a remote data source is used, it reduces the records which need to be requested from the server.


We extend cats!
- recontextualize seamless e-services
- leverage front-end niches
- syndicate next-generation content
- synthesize holistic web services
- facilitate next-generation solutions
- harness turn-key markets
- reintermediate robust relationships
- envisioneer proactive initiatives


{% extends "header.html" %} {% block body %}
Chances are, if you're viewing this page, you're brand new to Python.
You might even be new to Programming all-together. Either way, you have come to the right place, and chosen the right language!
Why Python?
Python is very beginner-friendly. The syntax (words and structure) is extremely simple to read and follow, most of which can be understand even if you do not know any programming. Let me show you:
Garage = "Ferrari", "Honda", "Porsche", "Toyota" for each_car in Garage: print(each_car)
"print()" is a built-in Python function that will output some text to the console.
When someone refers to "the console," they are referring to where information from your program is ouput. You will see an example of "output to console" below. If you want this message to go away, just click again on the "console" button that you originally clicked on.
Whenever you see a word encased in blue like "console" is above, you can click on it for more information, and click it again to make the information disappear again. Any words or phrases that may not be known by everyone will have this functionality.
Looking at the code about cars in the garage, can you guess what will happen? You probably have a general idea. For each_car in the garage, we're going to do something. What are we doing? We are printing each car.
Since "printing" outputs some text to the "console," you can probably figure out that the console will say something like "Ferrari, Honda, Porsche, Toyota."
But hey, let's not take my word for it, let's try it! Here's an editor and console for you. Feel free to run the pre-existing code, and then just delete all of the code there.
Next, copy and paste the garage code above into the editor (left side) so it is the only code in the editor ... then click run and see if the output is as expected.:
 Now there's no turning back.
What can Python do?Python is a fully-functional programming language that can do anything almost any other language can do, at comprable speeds.
Python is capable of threading and GPU processing just like any other language. Most of the data processing modules are actually just Python wrappers around C/C++ code.
"Modules" are pre-written Python code that you "import" in your Python program. Since there are many tasks that people commonly do, we have modules that people have written that do these taks for you, and they usually do them in the cleanest and most efficient method possible. Sometimes you will see people refer to "DRY." This stands for Don't Repeat Yourself, which often also translates into "Don't Repeat Someone Else."
The phrase "wrapper" means that someone has placed, like a wrapper, Python code over another language. So, when you have a Python wrapper around C++ code, what someone has done is written some Python code that interacts with the C++ language. This allows you to make use of various aspects of the language being wrapped, in this case C++, without actually needing to know or understand that language.
Thus, Python can be used to make games, do data analysis, control robot and hardware, create GUIs, or even to create websites.
"GUI" stands for Graphical User Interface, and is used to describe a program that incorporates graphics to make the program more interactive for the user.
99% of the code that brings this very page to you is actually Python code!
If you're curious to see some examples of what you can do with Python code, browse around the topics in the Dashboard.
The Dashboard is a collection of topics and tutorials offered here on PythonProgramming.net, as well as progress-tracking and suggestions for logged-in users. Guests can still navigate the Dashboard.
Congratulations, you have finished the first of many tutorials on the topic of Python. I recommend you
before proceeding so that this page is marked as complete. We do not share your information, nor do we spam. We don't even require you to confirm you email! (You may want to use a real one, however, just in case you lose your password).Using the button to get to the next tutorial will successfully mark this one as "complete" in your dashboard, if you are logged in.
Many of the tutorials within this basics series can be done using the embedded Python console that will be included, but some will not. To install Python, you will need to head to Python.org, or view the Python installation tutorial video:
You can either head on to the
, or head to the for other topics.


Clients may want to receive push notifications when events are received at the homeserver. This is managed by a distinct entity called the Push Gateway.
Table of Contents
This version of the specification is generated from matrix-doc as of Git commit push_gateway/release-r0.1.1,40e6f82c.
For the full historical changelog, see https://github.com/matrix-org/matrix-doc/blob/main/changelogs/legacy/push_gateway.rst
The following other versions are also available, in reverse chronological order:
A client's homeserver forwards information about received events to the push gateway.  If the endpoint returns an HTTP error code, the homeserver SHOULD retry for a reasonable amount of time using exponential backoff.
When pushing notifications for events, the homeserver is expected to include all of the event-related fields in the /notify request. When the homeserver is performing a push where the format is "event_id_only", only the event_id, room_id, counts, and devices are required to be populated.
This endpoint is invoked by HTTP pushers to notify a push gateway about an event or update the number of unread notifications a user has. In the former case it will contain selected information about the event. In either case it may contain numeric counts of the number of unread events of different types the user has. The counts may be sent along with a notification about an event or by themselves.
Notifications about a particular event will normally cause the user to be alerted in some way. It is therefore necessary to perform duplicate suppression for such notifications using the event_id field to avoid retries of this HTTP API causing duplicate alerts. The operation of updating counts of unread notifications should be idempotent and therefore do not require duplicate suppression.
Notifications are sent to the URL configured when the pusher is created. This means that the HTTP path may be different depending on the push gateway.
| Rate-limited: | No. |
|---|---|
| Requires auth: | No. |
Request format:
| Parameter | Type | Description |
|---|---|---|
| JSON body parameters | ||
| notification | Notification | Required. Information about the push notification |
| Parameter | Type | Description |
|---|---|---|
| event_id | string | The Matrix event ID of the event being notified about. This is required if the notification is about a particular Matrix event. It may be omitted for notifications that only contain updated badge counts. This ID can and should be used to detect duplicate notification requests. |
| room_id | string | The ID of the room in which this event occurred. Required if the notification relates to a specific Matrix event. |
| type | string | The type of the event as in the event's type field. |
| sender | string | The sender of the event as in the corresponding event field. |
| sender_display_name | string | The current display name of the sender in the room in which the event occurred. |
| room_name | string | The name of the room in which the event occurred. |
| room_alias | string | An alias to display for the room in which the event occurred. |
| user_is_target | boolean | This is true if the user receiving the notification is the subject of a member event (i.e. the state_key of the member event is equal to the user's Matrix ID). |
| prio | enum | The priority of the notification. If omitted, high is assumed. This may be used by push gateways to deliver less time-sensitive notifications in a way that will preserve battery power on mobile devices. One of: ["high", "low"] |
| content | EventContent | The content field from the event, if present. The pusher may omit this if the event had no content or for any other reason. |
| counts | Counts | This is a dictionary of the current number of unacknowledged communications for the recipient user. Counts whose value is zero should be omitted. |
| devices | [Device] | Required. This is an array of devices that the notification should be sent to. |
| Parameter | Type | Description |
|---|---|---|
| unread | integer | The number of unread messages a user has across all of the rooms they are a member of. |
| missed_calls | integer | The number of unacknowledged missed calls a user has across all rooms of which they are a member. |
| Parameter | Type | Description |
|---|---|---|
| app_id | string | Required. The app_id given when the pusher was created. |
| pushkey | string | Required. The pushkey given when the pusher was created. |
| pushkey_ts | integer | The unix timestamp (in seconds) when the pushkey was last updated. |
| data | PusherData | A dictionary of additional pusher-specific data. For 'http' pushers, this is the data dictionary passed in at pusher creation minus the url key. |
| tweaks | Tweaks | A dictionary of customisations made to the way this notification is to be presented. These are added by push rules. |
Response format:
| Parameter | Type | Description |
|---|---|---|
| rejected | [string] | Required. A list of all pushkeys given in the notification request that are not valid. These could have been rejected by an upstream gateway because they have expired or have never been valid. Homeservers must cease sending notification requests for these pushkeys and remove the associated pushers. It may not necessarily be the notification in the request that failed: it could be that a previous notification to the same pushkey failed. May be empty. |
Example request:
POST /_matrix/push/v1/notify HTTP/1.1 Content-Type: application/json { "notification": { "event_id": "$3957tyerfgewrf384", "room_id": "!slw48wfj34rtnrf:example.com", "type": "m.room.message", "sender": "@exampleuser:matrix.org", "sender_display_name": "Major Tom", "room_name": "Mission Control", "room_alias": "#exampleroom:matrix.org", "prio": "high", "content": { "msgtype": "m.text", "body": "I'm floating in a most peculiar way." }, "counts": { "unread": 2, "missed_calls": 1 }, "devices": [ { "app_id": "org.matrix.matrixConsole.ios", "pushkey": "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/", "pushkey_ts": 12345678, "data": {}, "tweaks": { "sound": "bing" } } ] } }
Response:
Status code 200:
A list of rejected push keys.
Example
{ "rejected": [ "V2h5IG9uIGVhcnRoIGRpZCB5b3UgZGVjb2RlIHRoaXM/" ] }


Load appSettingsClass.ClassName
Save appSettingsClass.ClassName
From
https://github.com/ignatandrei/appSettingsEditor
with jsonEditor help
https://github.com/json-editor/json-editor


As in the standard nonsymmetric eigenvalue problem (section <IP_ADDRESS>), two preprocessing steps may be performed on the input matrix pair (A, B). The first one is a permutation, reordering the rows and columns to attempt to make A and B block upper triangular, and therefore to reduce the order of the eigenvalue problems to be solved: we let (A',B') = P1 (A, B) P2, where P1 and P2 are permutation matrices. The second one is a scaling by two-sided diagonal transformation D1 and D2 to make the elements of A''= D1 A' D2 and B'' = D1 B' D2 have magnitudes as close to unity as possible, so as to reduce the effect of the roundoff error made by the later algorithm [100]. We refer to these two operations as balancing.
Balancing is performed by driver xGGEVX, which calls computational routine xGGBAL. The user may choose to optionally permute, scale, do both or do either; this is specified by the input parameter BALANC when xGGEVX is called. Permuting has no effect on the condition numbers or their interpretation as described in the previous subsections. Scaling does, however, change their interpretation, as we now describe.
The output parameters of xGGEVX - ILO(integer), IHI(integer), LSCALE(real array of length N), RSCALE(real array of length N), ABNRM(real) and BBNRM(real) - describe the result of balancing the matrix pair (A, B) to (A'',B''), where N is the dimension of (A,B). The matrix pair (A'',B'') has block upper triangular structure, with at most three blocks: from 1 to ILO-1, from ILO to IHI, and from IHI+1 to N (see section 2.4.8). The first and last blocks are upper triangular, and so already in generalized Schur form. These blocks are not scaled; only the block from ILO to IHI is scaled. Details of the left permutations (P1) and scaling (D1) and the right permutations (P2) and scaling (D2) are described in LSCALE and RSCALE, respectively. (See the specification of xGGEVX or xGGBAL for more information). The one-norms of A'' and B'' are returned in ABNRM and BBNRM, respectively.
The condition numbers
described in earlier subsections are computed
for the balanced matrix pair (A'',B'') in xGGEVX, and so
some interpretation is needed to apply them to the eigenvalues
and eigenvectors of the original matrix pair (A, B).
To use the bounds for eigenvalues in Tables 4.7 and 4.8,
we must replace
by
.
To use the bounds for eigenvectors, we also need to take
into account that bounds on rotation of the right and left eigenvectors
are for the right and left eigenvectors x'' and y'' of
A'' and B'', respectively, which are related to
the right and left eigenvectors x and y by
x'' = D-12 PT2 x and
y'' = D-11 P1 y,
or
x = P2 D2 x'' and
y = PT1 D1 x'' respectively.
Let
be the bound on the rotation of x''
from Table 4.7 and Table 4.8 and
let
be the desired bound on the rotation of x. Let
The numerical example in section 4.11 does no scaling, just permutation.