Istio Reference Architecture
Introduction
A lot of enterprises are embarking on a journey to adopt cloud native technologies to meet their technology strategy goals and address internal and external challenges. One of the key components of the technology stack is the Platform as a Service (PaaS) which will be the target platform for a number of migrated and new applications. These applications will be required to adopt latest standards in security, performance, scalability and resilience in a manner which is agile and efficient.
In order to adopt these standards, enterprises are looking towards open source Istio Service Mesh to manage the ever-increasing mesh of microservices and to help simplify key operational tasks.
Istio implementation brings it’s own challenges, for example:
- Installation and upgrade process using istio-cni.
- Automation of certificate management at the edge (ingress/egress) of mesh.
- Different resource definitions (gateway/VirtualService/DestinationRule/ServiceEntry) created while communicating at the edges (ingress/egress) of mesh for different use cases.
- How to share responsibility across cluster-admin/Mesh Admin/Application Developer.
- Operational aspects (monitoring/logging/tracing).
This document helps you to understand the concept and details the reference architecture implementation.
Some salient features of the implementation are as follows.
Assumption:
- Each Kubernetes cluster has only one service mesh control plane and data plane.
- All the communication within the service mesh takes place via mutual TLS.
- All external communication across two service meshes running in separate clusters takes place via mutual TLS through ingress/egress gateways.
- In order to implement mutual TLS across service meshes, all TLS certificates are provisioned via centralized vault.
- To automate the provisioning of certificate via centralized vault, a cert-manager is used. All clusters have their own cert-manager installed and connected to the centralized vault.
Installation
Istio installation refers to Istio control plane installation, which consists of following component (pods).
- istio-cni plugin (running as DaemonSet) in the kube-system namespace
- ingress gateway in the istio-system namespace
- egress gateway in the istio-system namespace
- istiod in the istio-system namespace
Prerequisite
In order to install Istio, following are the prerequisites.
- cluster-admin permission is required.
- To get istio-cni working, create the required pod security policy and it’s corresponding role and role binding.
- It is recommended security practice to configure third party service account tokens.
- To install Istio in an offline/airgap environment, a container registry is required.
Istio CNI Plugin
Istio injects initContainer (istio-init) in any pod which is part Istio mesh. It programs all the iptables rules required for intercepting all incoming and outgoing request to application pod. These rules are programmed into the pod’s network namespace. That requires some elevated privileges. Service accounts deploying pods to the mesh need to have sufficient Kubernetes RBAC permissions to deploy containers with the NET_ADMIN and NET_RAW capabilities.
Running production workloads in most enterprises with such elevated privileges is a not advisable. Here comes the Istio CNI plugin to rescue. This plugin is a replacement for the istio-init container that performs the same networking functionality but without requiring Istio users to enable elevated privileges.
The istio-cni is installed as a DaemonSet in the kube-system namespace. The rationale behind using kube-system namespace over istio-system is that istio-cni does not support multitenancy and in the event that multiple control planes are running (this is not possible at the moment) in a single Kubernetes cluster, there is no confusion as to which Istio control namespace, istio-cni is running. The istio-cni namespace is configurable using components.cni.namespace parameter during Istio installation.
To get istio-cni running, the corresponding pod security policy, role and role binding are required in the kube-system namespace.
The following Pod security policy needs to be applied. Only important snippets are shown here.
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: istio-control-plane
spec:
runAsUser:
rule: "RunAsAny"
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/opt/cni/bin"
The YAML config above defines a restricted PSP that enables Istio CNI components to be running in a Kubernetes cluster. In particular, this PSP provides access to the host network and host path volumes.
Third Party JWT Token
To authenticate with the Istio control plane, the Istio proxy will use a Service Account token. Kubernetes supports two forms of these tokens:
- Third party tokens, which have a scoped audience and expiration.
- First party tokens, which have no expiration and are mounted into all pods.
To determine if your cluster supports third party tokens, look for the TokenRequest API.
$ kubectl get --raw /api/v1 | jq '.resources[] | select(.name | index("serviceaccounts/token"))'
{
"name": "serviceaccounts/token",
"singularName": "",
"namespaced": true,
"group": "authentication.k8s.io",
"version": "v1",
"kind": "TokenRequest",
"verbs": [
"create"
]
}
In the config above, the kind TokenRequest
indicates third party tokens are
supported by the Kubernetes cluster.
Where third party support is needed, one can add the following flags to the kube-apiserver
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
- --service-account-issuer=kubernetes.default.svc
In order to verify the installation of Istio create a pod which has istio-proxy injected. Verify that the istio-proxy definition in the pod has the istio-token mounted.
volumeMounts:
- mountPath: /var/run/secrets/tokens
name: istio-token
Verify that token is generated at (/var/run/secrets/tokens/istio-token) in
istio-proxy container. Check it has an expiration date exp
and audience aud
defined as istio-ca
.
Container Registry
In order to install Istio in an offline/airgapped environment, a container registry is required which contains all the requires container images needed for installation.
The following images are required for installing Istio 1.7.0
- docker.io/istio/pilot:1.7.0
- docker.io/istio/proxyv2:1.7.0
- docker.io/istio/install-cni:1.7.0
An example of pulling and pushing images to registry:
docker pull docker.io/istio/install-cni:1.7.0
docker tag istio/install-cni:1.7.0 {{ REGISTRY_HOSTNAME }}/istio/istio/install-cni:1.7.0
docker push {{ REGISTRY_HOSTNAME }}/istio/istio/install-cni:1.7.0
All the worker nodes of the Kubernetes cluster need to be able to access this registry and be able to pull required images from this registry.
Installation Method
Istio uses the IstioOperator custom resource definition (CRD) for Istio installation. An example of one such CRD is as follows.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: installed-state
spec:
profile: default
Istio can be installed in two different ways.
-
istioctl command: Providing the full configuration in an IstioOperator CR is considered an Istio best practice for production environments.
-
Istio operator: One needs to consider security implications when using the operator pattern in Kubernetes. With the istioctl install command, the operation will run in the admin user’s security context, whereas with an operator, an in-cluster pod will run the operation in its security context.
NOTE: All the following steps use the istioctl command method.
This command is just for reference to get installation quickly and highlighting the important parameters.
istioctl manifest install --set profile=demo --set components.cni.enabled=true --set components.cni.namespace=kube-system --set values.global.istioNamespace=istio-system --set values.global.hub={{ REGISTRY_HOSTNAME }}/istio --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY --set values.global.controlPlaneSecurityEnabled=true --set meshConfig.enablePrometheusMerge=true --set revision=1-7-0
For a production-grade installation, istiooperator-airgap-1-7-0.yaml is used.
istioctl install -f istiooperator-airgap-1-7-0.yaml
Installation Validation
The istioctl command creates an IstioOperator custom resource called
installed-state-1-7-0
.
$ kubectl get IstioOperator -n istio-system
NAME REVISION STATUS AGE
installed-state-1-7-0 1-7-0 143m
To verify the istio-cni DaemonSet is deployed successfully:
$ kubectl get daemonset -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
istio-cni-node 4 4 4 4 4
To verify complete Istio installation:
istioctl verify-install -f istiooperator-airgap-1-7-0.yaml
If successful, the above command results in following output:
ConfigMap: istio.istio-system checked successfully
cont**
cont**
IstioOperator: installed-state.istio-system checked successfully
Checked 21 custom resource definitions
Checked 2 Istio Deployments
Istio is installed successfully
To verify all images are coming from the centralized container registry:
kubectl get deployments istio-ingressgateway -n istio-system -o yaml|grep -i image:
image: {{ REGISTRY_HOSTNAME }}/istio/proxyv2:1.7.0
Uninstall Method
Uninstall control plane (istiod) only:
istioctl x uninstall -f istiooperator-airgap-1-7-0.yaml
- Uninstall Istio completely (including CNI):
istioctl x uninstall -f istiooperator-airgap-1-7-0.yaml --purge
Integration with CI/CD
Complete installation process can be integrated with existing CI/CD system. The following considerations are required.
- Integration of istioctl with the pipeline.Pipeline running it required cluster-admin permission.
- Integration of some security vulnerability tooling with the pipeline, which ensures that there is no security vulnerability before installing Istio.
Conformant with Security Vulnerability
All images are existing in centralized container registry.
This centralized container registry has policy defined regarding content trust and vulnerability scanning.
For example, Harbor registry interrogation services provides two (Trivy and Clair) vulnerability scanners.
Any image pushed to this registry passes through the vulnerability scanning.
The following is an example of one result of Harbor registry vulnerability scanning with Trivy.
Vulnerability scanners ensure that any new/existing vulnerability is tagged and that no new pod can be created/scaled in cases where images have not passed vulnerability criteria defined by the security team.
Following are examples of how vulnerability scanners prevent any new pods being created.
Warning Failed 9s kubelet, ip-10-0-1-75.us-east-2.compute.internal Failed to pull image "harbor-k8s.xxxx.com/istio/install-cni:1.7.0": rpc error: code = Unknown desc = failed to pull and unpack image "harb│
│or-k8s.xxxx.com/istio/install-cni:1.7.0": failed to copy: httpReaderSeeker: failed open: unexpected status code https://harbor-k8s.xxxx.com/v2/istio/install-cni/manifests/sha256:09ac572eabda6b242aa5140f6bee0cbf809a4eb13b668d047│
│23cc7e8a1c272c6: 412 Precondition Failed - Server message: unknown: The image is not signed in Notary.
Warning Failed 6s kubelet, ip-10-0-1-75.us-east-2.compute.internal Failed to pull image "harbor-k8s.xxxx.com/istio/install-cni:1.7.0": rpc error: code = Unknown desc = failed to pull and unpack image "harb│
│or-k8s.xxxx.com/istio/install-cni:1.7.0": failed to copy: httpReaderSeeker: failed open: unexpected status code https://harbor-k8s.xxxx.com/v2/istio/install-cni/manifests/sha256:09ac572eabda6b242aa5140f6bee0cbf809a4eb13b668d047│
│23cc7e8a1c272c6: 412 Precondition Failed - Server message: unknown: current image with 268 vulnerabilities cannot be pulled due to configured policy in 'Prevent images with vulnerability severity of "High" or higher from runnin│
│g.' To continue with pull, please contact your project administrator to exempt matched vulnerabilities through configuring the CVE allowlist. │
│ Warning Failed 6s kubelet, ip-10-0-1-75.us-east-2.compute.internal Error: ErrImagePull
For any pod which is already running and a new security vulnerability is detected, int is continuously monitored by security team and it is for them to decide whether to kill the pod or keep it running depending upon the level and nature of vulnerability.
Sidecar Injection
There are different ways of injecting the Istio sidecar into a pod.
-
Manual sidecar injection: Manual injection directly modifies configuration, like deployments, and injects the proxy configuration into it.
-
Automatic sidecar injection: Sidecars are automatically added to Kubernetes pods using a mutating webhook admission controller provided by Istio.
- The default configuration injects the sidecar into pods in any namespace
with the
istio.io/rev=1-7-3
label. - Disable the injection policy in configmap and add
sidecar.istio.io/inject
annotation with valuetrue
to the pod template spec to override the default and enable injection.
- The default configuration injects the sidecar into pods in any namespace
with the
Application teams organize the workloads participating in a mesh in single or multiple namespaces depending upon the architecture (microservices bounded context/non microservices). Our general recommendation is to keep the mesh and non mesh related workloads in different namespaces. Automatic sidecar injection with default configuration is recommended as it is easier to maintain during the upgrade process. In this case, the Mesh Admin has clear visibility which workload is running with which version and can take action accordingly.
$ kubectl -n istio-system get configmap istio-sidecar-injector-1-7-0 -o jsonpath='{.data.config}' | grep policy:
policy: enabled
$ kubectl get mutatingwebhookconfiguration istio-sidecar-injector-1-7-0 -o yaml | grep "namespaceSelector:" -A7
namespaceSelector:
matchExpressions:
- key: istio-injection
operator: DoesNotExist
- key: istio.io/rev
operator: In
values:
- 1-7-0
$ kubectl get ns -l istio.io/rev=1-7-0
NAME STATUS AGE
bookinfo Active 15h
Upgrade Process/Patch Management
Upgrading Istio can be done by first running a canary deployment of the new control plane, allowing you to monitor the effect of the upgrade with a small percentage of the workloads, before migrating all of the traffic to the new version. This is much safer than doing an in-place upgrade and is the recommended upgrade method.
Note: Upgrading across more than one minor version (e.g., 1.5.x to 1.7.x) in one step is not officially tested or recommended.
To illustrate the upgrade process please see below (sample bookinfo
example
and an upgrade from 1.7.0 to 1.7.3 is considered as an example).
The above picture shows the important labels and component versions in current installed state (1.7.0).
In order to upgrade Istio to 1.7.3, set revision: 1-7-3
in IstioOperator
definition file as shown in sample
istiooperator-airgap-1-7-3.yaml and
install it using istioctl:
istioctl install -f istiooperator-airgap-1-7-3.yaml
NOTE: istioctl version is the target version (e.g. 1.7.3).
After running the command, you will have two control plane deployments and services and mutating webhook running side-by-side:
kubectl get pods -n istio-system -l app=istiod
NAME READY STATUS RESTARTS AGE
istiod-1-7-0-57f6b5cc56-cn7tr 1/1 Running 0 33m
istiod-1-7-3-57c6c6d9d7-99w68 1/1 Running 0 7m9s
kubectl get svc -n istio-system -l app=istiod
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istiod-1-7-0 ClusterIP <IP_ADDRESS> <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP 34m
istiod-1-7-3 ClusterIP <IP_ADDRESS> <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP 11m
kubectl get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
istio-sidecar-injector-1-7-0 1 34m
istio-sidecar-injector-1-7-3 1 11m
Istio CNI plugin is also upgraded to targeted version.
kubectl get ds istio-cni-node -n kube-system -o yaml|grep image:
image: {{ hostname }}/istio/install-cni:1.7.3
Unlike istiod, Istio gateways do not run revision-specific instances, but are instead in-place upgraded to use the new control plane revision.
istioctl proxy-config endpoints $(kubectl -n istio-system get pod -l app=istio-ingressgateway -o jsonpath='{.items..metadata.name}').istio-system --cluster xds-grpc -ojson | grep hostname
"hostname": "istiod-1-7-3.istio-system.svc"
This in-place upgrade makes gateways unavailable for fraction of time. Thorough testing is required to see how much time it is down. Depending upon the result minReplicas can be adjusted.
hpaSpec
in IstioOperator resource:
ingressGateways:
- enabled: true
k8s:
hpaSpec:
maxReplicas: 5
minReplicas: 2
Number of replicas
in istio-ingressgateway deployment:
spec:
replicas: 2
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
type: RollingUpdate
Using multiple replicas can help in ensuring availability of ingress gateway during upgrade. This need to be tested thoroughly.
Simply installing the new revision has no impact on the existing sidecar proxies. All the workloads are still running with the old version of Istio proxy, e.g. proxyv2:1.7.0, and pointing to the old control plane.
istioctl proxy-config endpoints $(kubectl -n bookinfo get pod -l app=productpage -o jsonpath='{.items..metadata.name}').bookinfo --cluster xds-grpc -ojson | grep hostname
"hostname": "istiod-1-7-0.istio-system.svc"
To upgrade these, one must configure them to point to the new istiod-1-7-3 control plane. This is controlled during sidecar injection based on the namespace label istio.io/rev
.
Figure: Data plane upgrade
To upgrade the namespace bookinfo, remove the istio.io/rev=1-7-0 label
, and
add istio.io/rev=1-7-3 revision
:
kubectl label namespace bookinfo istio.io/rev=1-7-3
Restart the deployment in bookinfo namespace:
kubectl rollout restart deployment -n bookinfo
Verify that all pods are running with new labels:
kubectl get pods -n bookinfo -l istio.io/rev=1-7-3
Verify that the workloads are running new version of Istio proxy, e.g. proxyv2:1.7.3, and pointing to the new control plane:
istioctl proxy-config endpoints $(kubectl -n bookinfo get pod -l app=productpage -o jsonpath='{.items..metadata.name}').bookinfo --cluster xds-grpc -o json | grep hostname
"hostname": "istiod-1-7-3.istio-system.svc"
Grading both the control plane and data plane, you can uninstall the old control plane:
istioctl x uninstall -f istiooperator-airgap-1-7-0.yaml
Confirm that the old control plane has been removed and only the new one still exists in the cluster:
kubectl get pods -n istio-system -l app=istiod
NAME READY STATUS RESTARTS AGE
istiod-1-7-3-57c6c6d9d7-99w68 1/1 Running 0 68m
In case new version is not working properly and want to revert back to previous version:
istioctl install -f istiooperator-airgap-1-7-0.yaml
kubectl label namespace bookinfo istio.io/rev=1-7-0
kubectl rollout restart deployment -n bookinfo
istioctl x uninstall -f istiooperator-airgap-1-7-3.yaml
Build & Release Cadence
Istio produces new builds of Istio for each commit. Around once a quarter or so, Istio build a Long Term Support (LTS) release, and run through a bunch more tests and release qualifications. Finally, if they find something wrong with an LTS release, they issue patches.
The different types represent different product quality levels and different levels of support from the Istio team. In this context, support means that they will produce patch releases for critical issues and offer technical assistance.
Network isolation
To further tighten the security of your applications, the following network policy are created.
Deny by Default
To ensure no inter-pod communication is possible, this policy can be applied globally. The idea here is to ensure that if an application developer wants to have a pod communicating with other pods, they need to create another network policy which is specific to their pod’s needs.
Cluster Admin creates this as default network policy in all the namespaces:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-by-default
namespace: bookinfo
spec:
podSelector: {}
policyTypes:
- Ingress
Allow from Istio Namespace
In order to access workloads running in the service mesh from outside of the service mesh, all requests needs to come from Istio ingress gateway. The general pattern is to expose one service which receives communication from outside (ingress gateway) and interacts with other services internally within the namespace.
Mesh Admin creates a label on the namespace where Istio ingress gateway is running (istio-system).
kubectl label namespace istio-system istio-control-namespace=true
Application Developer creates a network policy.
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-from-istio-namespace
namespace: bookinfo
spec:
podSelector:
matchLabels:
app: productpage
ingress:
- from:
- namespaceSelector:
matchLabels:
istio-control-namespace: "true"
Allow from Same Namespace
This is suggested policy whereby all the pods in the same namespace needs to interact without any policy restriction and because of global network policy (deny-by-default), cannot interact. Please check with the cluster-admin before applying this policy.
Application Developer creates a network policy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-same-namespace
namespace: bookinfo
spec:
podSelector: {}
ingress:
- from:
- podSelector: {}
Ingress Controller in Kubernetes with Istio
Cloud Consideration
Once Istio is installed on any cloud, e.g. AWS, Azure, a Service of Type LoadBalancer provisions an external LoadBalancer. For example, an Elastic Load Balancer is created by default in AWS.
kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
istio-ingressgateway LoadBalancer <IP_ADDRESS> xxxx.us-east-2.elb.amazonaws.com 15021:31413/TCP,80:32673/TCP,443:31820/TCP,31400:30472/TCP,15443:31916/TCP
A workload from outside service mesh is accessed using the ELB EXTERNAL-IP.
curl http://xxxx.us-east-2.elb.amazonaws.com/productpage
In order to use multiple hostnames, a CNAME type Record is required to be configured. In AWS, a Route 53 entry is required.
With the above DNS setting, and after creating the required Gateway Definition and Virtual Service Definition, workloads are accessed with different hostnames.
Hostnames can be defined using following convention:
curl http://productpage.bookinfo.appk8s.com
Istio by default creates an Elastic Load Balancer when installed on AWS. A network load balancer (NLB) can also be created by specifying following annotation during annotation.
gateways:
istio-ingressgateway:
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
NOTE: AWS ELBs and NLBs both support layer 4 load balancing. There has not been much adoption of NLB with open source Istio. But we don’t have any strong opinion when it comes to comparing these two. When it comes to Istio, Istio’s Gateway resource just lets you configure layer 4-6 load balancing properties such as ports to expose TLS settings and so on. Application-layer traffic routing (L7) are configured in virtual service which is bound to Istio gateway.
Existing Ingress Control
If an ingress controller such as HAProxy, Contour or NGINX is in use and you do not wish to replace it with Istio’s ingress gateway, they can be used in conjunction. In this case, the ingress controller will direct traffic intended for Istio service mesh by creating an Ingress object and using istio-ingressgateway as backend.
$ kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway ClusterIP <IP_ADDRESS> <none> 15021/TCP,80/TCP,443/TCP,31400/TCP,15443/TCP 70s
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: istio
namespace: istio-system
labels:
app: istio
spec:
rules:
- host: productpage.bookinfo.appk8s.com
http:
paths:
- backend:
serviceName: istio-ingressgateway
servicePort: 80
NOTE: If using a HTTPS host for the Ingress, create with passthrough mode so that mutual authentication can take place at Istio ingress gateway.
NOTE: The above Ingress definition is just for reference. In this document, only Istio ingress controller using Gateway Definition and Virtual Service Definition is used.
With the above setting, and after creating required Gateway Definition and Virtual Service Definition, workloads are accessed with different hostnames.
curl http://productpage.bookinfo.appk8s.com
If routing traffic to the Istio ingress gateway through an ingress controller, you will put an additional hop on the request path. The following picture illustrates on the left side cluster the additional hop to reach to Istio ingress gateway as compared to without it on right side.
Istio Secure Ingress Gateway with mTLS
By default the TLS protocol only proves the identity of the server to the client using X.509 certificate and the authentication of the client to the server is left to the application layer. TLS also offers client-to-server authentication using client-side X.509 authentication.This two-way authentication, when two parties authenticating each other at the same time is also called Mutual TLS authentication (mTLS).
Following is an example depicting the difference between two (TLS vs mTLS) certificate exchanges. Notice how the client certificate is requested and verified depicted by yellow line in mTLS traffic flow.
As an mTLS service provider, the following certificates are required:
- Server certificate and key
- CA certificate to verify client
These certificates need to be provided as a secret in the namespace where ingress gateway pod is running. In this example it is running in istio-system namespace.
The service provider creates the following gateway definition in the application namespace:
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: bookinfo-gateway
namespace: bookinfo
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- productpage.bookinfo.appk8s.com
port:
name: https
number: 443
protocol: HTTPS
tls:
credentialName: bookinfo-ingressgateway-certs
mode: MUTUAL
In order to automate server certificate and CA certificate and their associated secret, the following integrations are required.
- Integration with vault
- Integration with cert-manager
Integration with Vault
Enterprise Root CA exists in centralize vault. This Root CA is required to sign the following:
- server certificate provided by the service provider of the application/microservice.
- client certificate which is used to access the above microservice.
In order to access vault from Istio control plane namespace (istio-system), the following needs to be configured:
Configure allowed domain (appk8s.com):
vault-0 -- vault write pki/roles/appk8s-dot-com allowed_domains=appk8s.com allow_subdomains=true max_ttl=72h
Configure Kubernetes authentication to vault:
vault exec vault-0 -- vault auth enable kubernetes
Configure Kubernetes authentication with service account:
vault-0 -- vault write auth/kubernetes/config token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"<EMAIL_ADDRESS>A role needs to be created that specify which ServiceAccount in which namespace can access Vault:
vault exec vault-0 -- vault write auth/kubernetes/role/issuer bound_service_account_names=issuer bound_service_account_namespaces={{ controlPlaneNamespace }} policies=pki ttl=20m
Integration with Cert-Manager
Helm base installation is required. A cluster-admin who has permission to create CRDs, should be able to install it.
Cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources like vault.
The diagram above describes interaction among different namespaces and vault:
- Mesh admin creates an Issuer in istio-system namespace.
- Once Issuer is created, cert-manager auto discovers new certificate-issuer .
- cert-manager connects certificate-issuer to centralized Vault PKI engine.
- Application Developer creates Istio gateway definition in dev namespace with credentialName required for mutual TLS.
- Application Developer creates certificate in istio-system namespace with the required dnsNames and commonName.
- cert-manager, which is watching istio-system namespace for any new certificate created, picks up the certificate and creates corresponding CertificateRequests in istio-system namespace.
- cert-manager sends Certificate Sign Request (CSR) to centralized Vault PKI engine to sign certificate. Vault signs the certificate. CertificateRequests and Certificate turns into Ready state.
- cert-manager generates Kubernetes secret in istio-system namespace.
Mesh Admin Responsibility
Mesh Admin needs to perform following steps:
- create ServiceAccount, which is configured above in Vault.
kubectl -n istio-system create serviceaccount issuer
- get the secret from above ServiceAccount.
kubectl -n istio-system get serviceaccount issuer -o json | jq -r .secrets[].name
issuer-token-6qr6j
- Configure following Issuer in Istio control plane namespace.
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: vault-issuer
spec:
vault:
server: http://vault.vault
path: pki/sign/appk8s-dot-com
auth:
kubernetes:
mountPath: /v1/auth/kubernetes
role: issuer
secretRef:
name: issuer-token-6qr6j
key: token
Application Developer Responsibility
Application Developer needs to perform following steps.
- Create gateway definition.
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: bookinfo-gateway
namespace: bookinfo
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- productpage.bookinfo.appk8s.com
port:
name: https
number: 443
protocol: HTTPS
tls:
credentialName: bookinfo-ingressgateway-certs
mode: MUTUAL
- Create the Certificate required for the gateway definition.
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: bookinfo-ingressgateway-certs
namespace: istio-system
spec:
secretName: bookinfo-ingressgateway-certs
issuerRef:
name: vault-issuer
commonName: productpage.bookinfo.appk8s.com
dnsNames:
- productpage.bookinfo.appk8s.com
- Verify the CertificateRequests is created successfully.
kubectl get certificaterequests -n istio-system
NAME READY AGE
bookinfo-ingressgateway-certs-1409582587 True 5h25m
- Verify the Certificate is created successfully.
kubectl get certificate bookinfo-ingressgateway-certs -n istio-system
NAME READY SECRET AGE
bookinfo-ingressgateway-certs True bookinfo-ingressgateway-certs 174m
- Verify the corresponding secret is created.
kubectl get secret bookinfo-ingressgateway-certs -n istio-system
NAME TYPE DATA AGE
bookinfo-ingressgateway-certs kubernetes.io/tls 3 174m
Secret Structure
Get the detail of secret data.
kubectl -n istio-system get secret appk8s-dot-com -o=jsonpath='{$.data}'
The following describes the contents of the secrets created (ca.crt, tls.crt, tls.key). What goes where depends upon the integration with cert-manager and what Root CA or Intermediate CA is configured in vault.
ca.crt:
- In the case that vault is configured as Intermediate CA cert, it will consist of Intermediate CA cert only.
- In case vault is configured as Root CA cert, it will consist Root CA cert only.
- The thing to notice here is that it does not consist of the full chain of authority.
tls.crt:
- Cert Issued by Root CA in vault.
- Issuing CA certificate.
- In case vault is configured as Intermediate CA cert, it will consist of Intermediate CA cert and complete chain of authority.
tls.key:
- TLS key generated
NOTE:
- Full automation can be achieved in case that the Root CA is configured in vault.
- Full automation can not be achieved in the case that an intermediate CA certificate in vault.
- In the case that the intermediate certificate is configured in vault, it must get the full chain of CA certificate in tls.crt field but not in ca.crt field of the certificate generated.
- ca.crt contains the CA certificate which has issued the certificate not the full chain.
- In order to get mTLS working, ca.crt field should have the complete chain. https://github.com/jetstack/cert-manager/issues/2358
- This issue can partially be resolved by creating bookinfo-ingressgateway-certs-ca manually. This will bring challenges in day 2 operations. You will need to write a utility which keeps track of CA cert in vault and the moment it changes, recreate all the secrets (bookinfo-ingressgateway-certs-ca) in istio-system automatically.
Access Privilege Required to Create a Certificate
- An Application team can have following namespaces segregated by network policy.
- istio-system
- cert-manager
- Dev
- Test
- Prod
- Depending upon their role in the application team, different namespace admins may be appropriate.
- Admins of Dev, Test and Prod do not have complete access on istio-system and cert-manager.
- App developers and pipelines for Dev, Test and Prod do require permission to create certificate in istio-system namespace.
- Create a ClusterRole which has privilege to create certificate in istio-system.
- Create a RoleBinding for different tenants which have privilege to create certificate in istio-system.
High Availability for Gateway
Depending upon load, the number of gateway increases/decreases when the horizontal pod autoscaler is enabled.
ingressGateways:
- enabled: true
k8s:
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-ingressgateway
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 40Mi
Istio Ingress Gateway with JWT
Request Authentication
Request auth is used for end-user authentication to verify the credential attached to the request. Istio enables request-level authentication with JSON Web Token (JWT) validation and a streamlined developer experience using a custom authentication provider or any OpenID Connect providers.
curl -H "Authorization: Bearer <JWT_TOKEN>" https://productpage.bookinfo.appk8s.com/productpage
As a MeshAdmin, RequestAuthentication definition can be applied globally (across all the namespaces) in istio-system namespace.
```yaml
apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
name: "reqauth-istio-system"
namespace: istio-system
spec:
jwtRules:
- issuer: http://<OIDC_PROVIDER>/auth/realms/istio
jwksUri: http://<OIDC_PROVIDER>/auth/realms/istio/protocol/openid-connect/certs
forwardOriginalToken: true
```
issuer
: JSON Web Token (JWT) issuer
jwksUri
: The JSON Web Key Set (JWKS) is a set of keys containing the public keys used to verify any JSON Web Token (JWT).
forwardOriginalToken
: By default, JWT request tokens are not passed from one
service to another service. It is the responsibility of Application Developer
to pass these tokens from one service to another. This can be set to true to
pass the JWT Request token to next service. This forward is valid only for first
hop.
Istio checks the presented token, if presented against the rules in the request authentication policy, it rejects requests with invalid tokens. When requests carry no token, they are accepted by default. To reject requests without tokens, provide authorization rules that specify the restrictions for specific operations.
As MeshAdmin an AuthorizationPolicy definition can be applied in istio-system namespace so that Istio ingress gateway does not let any request through without passing valid JWT.
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: "frontend-ingress"
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
action: DENY
rules:
- from:
- source:
notRequestPrincipals: ["*"]
The above AuthorizationPolicy says that any request coming to ingress gateway
with label istio: ingressgateway
is denied if it does not have a JWT in
request header.
A token can be obtained as follows. This is an example of one of the OIDC providers: keycloak. This can vary depending upon the provider.
curl -X POST 'http://<OIDC_PROVIDER>/auth/realms/istio/protocol/openid-connect/token' -H "Content-Type: application/x-www-form-urlencoded" -d 'username=prod001&password=< PASSWORD >&grant_type=password&client_id=servicemesh' |jq -r .access_token
NOTE: In above example username is prod001
. This will be used to illustrate
some examples in the next section.
One can access the service by passing the token obtained in the above step to the <JWT_TOKEN> placeholder.
curl -H "Authorization: Bearer <JWT_TOKEN>" https://productpage.bookinfo.appk8s.com/productpage
As an Application developer, one can specify their application-specific AuthorizationPolicy based on the token attribute. In following example only a specific user(prod001) can access the service.
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: policy
namespace: bookinfo
spec:
selector:
matchLabels:
app: productpage
action: ALLOW
rules:
- from:
- source:
namespaces: ["istio-system"]
when:
- key: request.auth.claims[preferred_username]
values: ["prod001"]
Note: Using JWT alone does not encrypt the traffic. One should use mTLS and JWT together.
curl -H --cacert ca.crt --cert tls.crt --key /tls.key -H "Authorization: Bearer <JWT_TOKEN>" https://productpage.bookinfo.appk8s.com/productpage
Service Mesh Communication
Istio Identity
Identity is a fundamental concept of any security infrastructure. At the beginning of a service-to-service communication, the two parties must exchange credentials with their identity information for mutual authentication. In Kubernetes, Service Accounts are treated as service Identities. Istio uses X.509 certificates to carry identities in SPIFFE format. Istio provisions identities through the secret discovery service (SDS).
Istio CA
By default, Istio’s CA generates a self-signed root certificate and key, and uses them to sign the workload certificates. By default this certificate is valid for 10 years. It’s mounted in istiod pod.
kubectl get secret istio-ca-secret -o=jsonpath='{$.data.ca-cert\.pem}' -n istio-system | base64 --decode |openssl x509 -noout -text |grep -A4 -i Issuer
Issuer: O=cluster.local
Validity
Not Before: Oct 28 17:17:26 2020 GMT
Not After : Oct 26 17:17:26 2030 GMT
One can plug in their own root/intermediate certificate by creating a secret
named cacerts
in istio-system namespace.
kubectl create secret generic cacerts -n istio-system \
--from-file=ca-cert.pem \
--from-file=ca-key.pem \
--from-file=root-cert.pem \
--from-file=cert-chain.pem
mTLS Mode
Mesh Admin can ensure that all the services inside the mesh are communicating via mTLS by applying following PeerAuthentication definition:
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT
The above Global (in istio-system namespace) PeerAuthentication policy enforces Application Developer always specify traffic policy as ISTIO_MUTUAL as follows:
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: productpage
spec:
host: productpage
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
subsets:
- name: v1
labels:
version: v1
With the DestinationRule TLS mode policy set to ISTIO_MUTUAL, the Istio CA generates all the required client and server certificates automatically.
The certificate can be verified if they are created using the SPIFFE format based on service account.
kubectl -n bookinfo exec $(kubectl -n bookinfo get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- openssl s_client -showcerts -connect details:9080 | openssl x509 -noout -text | grep -B1 -A2 spiffe
X509v3 Subject Alternative Name: critical
URI:spiffe://cluster.local/ns/bookinfo/sa/bookinfo-details
The issuer of the certificate can be verified by executing the following command. By default, the workload certificates are provisioned for 24 hours.
kubectl -n bookinfo exec $(kubectl -n bookinfo get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- openssl s_client -showcerts -connect details:9080 | openssl x509 -noout -text | grep -A3 Issuer
Issuer: O=cluster.local
Validity
Not Before: Oct 29 07:26:19 2020 GMT
Not After : Oct 30 07:26:19 2020 GMT
Egress Gateway in Istio
Istio Secure Egress Gateway mTLS Origination
Within Cluster
Use case: An app running in a namespace sleep
which is part of service
mesh is connecting to a Kubernetes service, e.g.
my-nginx.mesh-external.svc.cluster.local
, residing in the same Kubernetes
cluster but not in the service mesh.
The diagram above describes the interaction between different namespaces and vault.
Within Cluster Steps
- Mesh admin creates an Issuer in istio-system namespace.
- Once Issuer is created, cert-manager auto discovers new certificate-issuer.
- cert-manager connects certificate-issuer to Centralized Vault PKI engine.
- Application Developer creates the following in
sleep
namespace:- Istio egress gateway definition and destination rule (for egress service).
- virtual service which routes
- traffic coming from app container on port 80 to egress service on port 443.
- traffic coming to egress service on port 443 to external service.
- Application Developer creates Certificate in
istio-system
namespace with the required dnsNames and commonName. - cert-manager, which is watching
istio-system
namespace for any new Certificate created, picks up the Certificate and creates corresponding CertificateRequests inistio-system
namespace. - cert-manager sends Certificate Sign Request (CSR) to Centralized Vault PKI engine to sign certificate. Vault signs the certificate. CertificateRequests and Certificate turns into Ready state.
- cert-manager generates Kubernetes secret in
istio-system
namespace. - Application Developer creates DestinationRule (for external service) with credentialName required for mutual TLS.
Mesh Admin Responsibility
Mesh Admin needs to perform following steps.
- Create ServiceAccount, which is configured above in vault.
kubectl -n istio-system create serviceaccount issuer
- Get the secret from above ServiceAccount.
kubectl -n istio-system get serviceaccount issuer -o json | jq -r .secrets[].name
issuer-token-6qr6j
- Configure the following Issuer in Istio control plane namespace.
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: vault-issuer
spec:
vault:
server: http://vault.vault
path: pki/sign/appk8s-dot-com
auth:
kubernetes:
mountPath: /v1/auth/kubernetes
role: issuer
secretRef:
name: issuer-token-6qr6j
key: token
Application Developer Responsibility
NOTE: EXTERNAL_SERVICE_HOST=my-nginx-client.mesh-external.svc.cluster.local
- Create egress gateway definition in
sleep
namespace as all traffic external to mesh goes via this Egress Gateway.
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: istio-egressgateway
spec:
selector:
istio: egressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- { { EXTERNAL_SERVICE_HOST } }
tls:
mode: ISTIO_MUTUAL
- Create destination rule in
sleep
namespace. It is required as it determines how ISTIO_MUTUAL service residing in service mesh communicates with egress gateway.
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: egressgateway-for-nginx
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
subsets:
- name: nginx
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: ISTIO_MUTUAL
sni: { { EXTERNAL_SERVICE_HOST } }
- Create virtual service in
sleep
namespace. This virtual service has two functions:- HTTP (port:80) traffic originating from app container is redirected to istio-egressgateway.istio-system.svc.cluster.local on port 443
- Any traffic coming to istio-egressgateway.istio-system.svc.cluster.local on port 443 is redirected to external service (EXTERNAL_SERVICE_HOST)
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: direct-nginx-through-egress-gateway
spec:
hosts:
- { { EXTERNAL_SERVICE_HOST } }
gateways:
- istio-egressgateway
- mesh
http:
- match:
- gateways:
- mesh
port: 80
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: nginx
port:
number: 443
weight: 100
- match:
- gateways:
- istio-egressgateway
port: 443
route:
- destination:
host: { { EXTERNAL_SERVICE_HOST } }
port:
number: 443
weight: 100
- Create Certificate in
istio-system
namespace. It is required because a client certificate is required to communicate to the external service with mutual TLS. This Certificate definition creates a secret (client-credential) mentioned in the destination rule below.
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: client-credential
namespace: istio-system
spec:
secretName: client-credential
issuerRef:
name: vault-issuer
commonName: { { EXTERNAL_SERVICE_HOST } }
dnsNames:
- { { EXTERNAL_SERVICE_HOST } }
- Create DestinationRule in
istio-system
namespace. It is required as it determines how MUTUAL external service communicates with egress gateway.
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: originate-tls-for-nginx
spec:
host: { { EXTERNAL_SERVICE_HOST } }
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: MUTUAL
credentialName: client-credential
sni: { { EXTERNAL_SERVICE_HOST } }
Inter Cluster
Use case: An app running in a namespace sleep
which is part of service
mesh is connecting to an external Kubernetes service, e.g.
my-nginx.mesh-external.svc.cluster.local
, residing in different Kubernetes
cluster.
The diagram above describes the interaction between different namespaces and Vault.
Please see the Within cluster steps section for all different steps involved. The only difference here is in step 4 where an additional service entry is required in order to access the service residing in different Kubernetes cluster. This is required to make sure external service is available in Istio’s internal service registry.
Mesh Admin Responsibility
Mesh Admin needs to perform following steps.
- Create a ServiceAccount which is configured above in vault.
kubectl -n istio-system create serviceaccount issuer
- Get the secret from the above ServiceAccount.
kubectl -n istio-system get serviceaccount issuer -o json | jq -r .secrets[].name
issuer-token-6qr6j
- Configure the following Issuer in Istio control plane namespace.
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: vault-issuer
spec:
vault:
server: http://vault.vault
path: pki/sign/appk8s-dot-com
auth:
kubernetes:
mountPath: /v1/auth/kubernetes
role: issuer
secretRef:
name: issuer-token-6qr6j
key: token
Application Developer Responsibility
NOTE: EXTERNAL_SERVICE_HOST=my-nginx-client.mesh-external.appk8s.com
Please see Within cluster section for all required responsibilities. Note that an additional service entry is required. This is required to make sure external service is available in Istio’s internal service registry.
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: nginx-k8s
spec:
hosts:
- { { EXTERNAL_SERVICE_HOST } }
ports:
- number: 443
name: https
protocol: HTTPS
resolution: DNS
Access Privilege Required to Create a Certificate and Gateway
- All application teams are responsible for their service mesh.
- An application team can have the following namespaces segregated by network policy.
- istio-system
- cert-manager
- Dev
- Test
- Prod
- Depending upon their role in application teams, different namespace admins may be appropriate.
- Admins of Dev, Test and Prod do not have complete access on istio-system and cert-manager.
- App developers and pipelines for Dev, Test and Prod do require permission to create certificate in istio-system namespace.
- Create a ClusterRole which has privileges to create Certificate and Gateway in
istio-system
. - Create a RoleBinding for different tenants which has permission to create Certificate in ‘istio-system’.
High Availability for Gateway
Depending upon load, the number of gateway increases/decreases when the horizontal pod autoscaler is enabled.
egressGateways:
- enabled: true
k8s:
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-egressgateway
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 40Mi
Observability
Telemetry & Prometheus
Istio generates metrics for all service traffic into, out from, and within an Istio service mesh. These metrics provide information on behaviors such as the overall volume of traffic, the error rates within the traffic, and the response times for requests. By default, Istio configures Envoy to record minimal statistics.
kubectl -n bookinfo exec -it {{ SOURCE_POD_NAME}} -c istio-proxy -- pilot-agent request GET stats | more
cluster_manager.cds.version_text: "2020-10-21T10:31:01Z/6"
listener_manager.lds.version_text: "2020-10-21T10:31:01Z/6"
cluster.xds-grpc.assignment_stale: 0
cluster.xds-grpc.assignment_timeout_received:
|
|
Istio control plane components export metrics on their own internal behaviors to provide insight on the health and function of the mesh control plane.
Prometheus is an open-source metrics and alerting toolkit. Prometheus works on pull based mechanism. That means our applications or the Istio service proxy expose an endpoint with the latest metrics from which Prometheus can the pull/scrape the metrics.
Proxy-Level Metrics
Each proxy generates a rich set of metrics about all traffic passing through the proxy (both inbound and outbound). The proxies also provide detailed statistics about the administrative functions of the proxy itself, including configuration and health information.
Some examples are as follows.
envoy_cluster_upstream_rq_completed{cluster_name="xds-grpc"} 7164
envoy_cluster_ssl_connection_error{cluster_name="xds-grpc"} 0
Service-Level Metrics
Istio provides a set of service-oriented metrics for monitoring service communications.
Control Plane Metrics
These metrics allow monitoring of the behavior of Istio itself.
Prometheus Architecture
In an Istio mesh, each component exposes an endpoint that emits metrics. To gather metrics for the entire mesh, configure Prometheus to scrape:
- The control plane (istiod deployment)
- Ingress and Egress gateways
- The Envoy sidecar
- The user applications (if they expose Prometheus metrics)
The recommended approach for production-scale monitoring of Istio with Prometheus is to use hierarchical federation in combination with a collection of recording rules.
The Istio pods in the control and data planes declare they are ready to have metrics scraped by Prometheus by way of annotations in pod definition.
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/path: /stats/prometheus
prometheus.io/port: "15090"
prometheus.io/scrape: "true"
One can verify that the pod metrics are ready to be scraped by checking targets in Prometheus.
Prometheus instances that are deployed locally to each cluster for Istio act as
initial collectors. A local Prometheus scrapes the metrics from different
services. Metrics fill up underlying storage very fast over a period of days or
weeks.The recommended approach is to keep a short retention window on this local
cluster. If you deploy this Prometheus server to the istio-system
namespace,
you do not need to worry about additional network policy.
Modify the configuration of your production-ready deployment of Prometheus to scrape the federation endpoint of the Istio Prometheus. It is not required to scrape all the metrics. Only production relevant metrics are scraped.
Grafana
Prometheus is a time-series database and collection toolkit. Once metrics are collected, Prometheus brings a simple expression browser to help explore trends and metrics, Grafana is a popular and powerful open-source graph and visualization tool that works well with Prometheus.
Istio comes with some pre built dashboards which consist of following sections.
Mesh Summary View: provides Global Summary view of the Mesh which shows HTTP/gRPC and TCP workloads in the Mesh.
Individual Services View: provides metrics about requests and responses for each individual service within the mesh (HTTP/gRPC and TCP).
Individual Workloads View: provides metrics about requests and responses for each individual workload within the mesh (HTTP/gRPC and TCP).
Installing Prometheus and Grafana
Prometheus Installation
Istio comes with built-in Prometheus installation, which contains all the required scraping configurations.
To use an existing Prometheus instance, add the scraping configurations from prometheus/configmap.yaml to your configuration.
In both the above cases, access to metrics are provided using Grafana with RBAC.
Grafana Installation
Istio provides a basic sample installation to quickly get Grafana up and running. It is bundled with all of the Istio dashboards already installed. One can remotely access Grafana securely.
If using an existing Grafana instance, the following dashboards can be imported into its dashboard:
- Mesh Dashboard provides an overview of all services in the mesh.
- Service Dashboard provides a detailed breakdown of metrics for a service.
- Workload Dashboard provides a detailed breakdown of metrics for a workload.
- Performance Dashboard monitors the resource usage of the mesh.
- Control Plane Dashboard monitors the health and performance of the control plane.
Prometheus Annotations
By passing –set meshConfig.enablePrometheusMerge=true during installation, appropriate prometheus.io annotations will be added to all data plane pods to expose metrics for scraping. With this option, the Envoy sidecar will merge Istio’s metrics with the application metrics. The merged metrics will be scraped from /stats/prometheus:15020. This option exposes all the metrics in plain text.
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/path: /stats/prometheus
prometheus.io/port: "15020"
prometheus.io/scrape: "true"
In the case that the application is also exposing custom app metrics in Prometheus format, Application Developer needs to provide application specific Prometheus annotation in pod definition.
Following is an example of a Spring Boot application pod definition exposing JVM specific metrics:
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/path: /manage/prometheus
prometheus.io/port: "8383"
prometheus.io/scrape: "true"
Once the above pod definition is applied in a namespace that is part of the service mesh, istio-proxy is injected and the above annotations are merged as shown in the following pod definition.
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/path: /stats/prometheus
prometheus.io/port: "15020"
prometheus.io/scrape: "true"
At the same time ISTIO_PROMETHEUS_ANNOTATIONS environment variable is updated to the application specific Prometheus annotation.
env:
- name: ISTIO_PROMETHEUS_ANNOTATIONS
value: '{"scrape":"true","path":"/manage/prometheus","port":"8383"}
name: istio-proxy
Application specific metrics can be queried from Prometheus as follows.
If you pass –set meshConfig.enablePrometheusMerge=false during installation, Application Developer will need to provide appropriate prometheus.io annotations inside the pod/deployment.
Logging
Istio uses meshConfig.accessLogFile
parameter during installation to specify
where the logs should be redirected.
spec:
meshConfig:
accessLogFile: /dev/stdout
It’s recommended practice for containerized applications to write logs to the standard output and error streams. These streams are redirected by container engine to its configured location (usually /var/log/containers).
Log Aggregation
Logs are produced by containers at the configured location on their respective nodes, where they are running.
An example of Istio control plane logs are as follows.
ls -al /var/log/containers/
istiod-1-7-3-_istio-system_discovery-xx.log -> /var/log/pods/istio-system_istiod-1-7-3-xx/discovery/0.log
istio-egressgateway_istio-system_istio-proxy-xx.log -> /var/log/pods/istio-system_istio-egressgateway_xx/istio-proxy/0.log
istio-ingressgateway_istio-system_istio-proxy-xx.log -> /var/log/pods/istio-system_istio-ingressgateway_xx/istio-proxy/0.log
An example of Istio data plane logs are as follows.
ls -al /var/log/containers/
productpage-v1_bookinfo_istio-proxy-xx.log -> /var/log/pods/bookinfo_productpage-v1-xx/istio-proxy/0.log
productpage-v1_bookinfo_istio-validation-xx.log -> /var/log/pods/bookinfo_productpage-v1-xx/istio-validation/0.log
productpage-v1_bookinfo_productpage-xx.log -> /var/log/pods/bookinfo_productpage-v1-xx/productpage/0.log
An example of Istio CNI logs are as follows.
ls -al /var/log/containers/
istio-cni-node-cfxml_kube-system_install-cni-xx.log -> /var/log/pods/kube-system_istio-cni-node-xx/install-cni/0.log
istio-cni-node-cfxml_kube-system_repair-cni-xx.log -> /var/log/pods/kube-system_istio-cni-node-xx/repair-cni/0.log
These logs are created on all the nodes where pods are running. A logging agent such as Fluentd, FluentBit or Logstash is required to be running on all the nodes and have access to log directories. It collects logs from those directories and forwards it to a back end log storage such as ElasticSearch, Splunk or Kafka. Most of the logging agents provided by different vendors are implemented as a DaemonSet.
An example of a FluentBit integration with Splunk is as follows. It runs as DaemonSet using a ConfigMap to provide a configuration file. This configuration file has two sections: input and output.
This example is provided to explain the concept. Complete details are omitted.
Input Config
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Things to note:
- Path: From where logs need to be collected.
- Tag: An internal string used to filter logs for Output phase.
Output Config
output-splunk.conf: |
[OUTPUT]
Name splunk
Match kube.*
Host <SPLUNK_HOST_NAME>
Port 8088
Splunk_Token 98706938-a99d-459f-9255-ca7e192d05a9
TLS On
TLS.Verify Off
Things to note:
- Name: shows output sent to Splunk
- Tag: Which tag is used to filter logs for Output phase.
- Splunk_Token: Splunk index specific token for HTTP Event Collector.
Once a FluentBit DaemonSet is created with the above configuration, the FluentBit agent starts sending logs to Splunk.
Log Display
In Splunk, one can view the logs by entering the appropriate query. Some screenshots follow.
Known Bugs
- Istio control plane only installs in istio-system namespace https://github.com/istio/istio/issues/24037
Appendix
Gateway Definition
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: bookinfo-gateway
namespace: bookinfo
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- productpage.bookinfo.appk8s.com
port:
name: http
number: 80
protocol: HTTP
Virtual Service Definition
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: bookinfo
namespace: bookinfo
spec:
gateways:
- bookinfo-gateway
hosts:
- productpage.bookinfo.appk8s.com
http:
- match:
- uri:
exact: /productpage
- uri:
prefix: /static
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
route:
- destination:
host: productpage
port:
number: 9080
PodSecurityPolicy Definition for Istio CNI
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: istio-control-plane
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default,runtime/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
spec:
allowPrivilegeEscalation: false
hostNetwork: true
volumes:
- configMap
- emptyDir
- projected
- secret
- downwardAPI
- persistentVolumeClaim
- hostPath
runAsUser:
rule: "RunAsAny"
seLinux:
rule: "RunAsAny"
supplementalGroups:
ranges:
- max: 65535
min: 1
rule: "RunAsAny"
fsGroup:
ranges:
- max: 65535
min: 1
rule: MustRunAs
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/opt/cni/bin"
Following Role is required in kube-system namespace
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: istio-control-plane
rules:
- apiGroups:
- extensions
resources:
- podsecuritypolicies
resourceNames:
- istio-control-plane
verbs:
- use
Following RoleBinding is required in kube-system namespace
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: istio-control-plane
subjects:
- kind: ServiceAccount
name: istio-cni
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: istio-control-plane
Production-Grade IstioOperator v1.7.0
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: installed-state
namespace: istio-system
spec:
addonComponents:
istiocoredns:
enabled: false
components:
base:
enabled: true
cni:
enabled: true
namespace: kube-system
egressGateways:
- enabled: true
k8s:
env:
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-egressgateway
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 40Mi
service:
ports:
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tls
port: 15443
targetPort: 15443
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
name: istio-egressgateway
ingressGateways:
- enabled: true
k8s:
env:
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-ingressgateway
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 40Mi
service:
ports:
- name: status-port
port: 15021
targetPort: 15021
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tcp
port: 31400
targetPort: 31400
- name: tls
port: 15443
targetPort: 15443
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
name: istio-ingressgateway
istiodRemote:
enabled: false
pilot:
enabled: true
k8s:
env:
- name: PILOT_TRACE_SAMPLING
value: "100"
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 1
periodSeconds: 3
timeoutSeconds: 5
resources:
requests:
cpu: 10m
memory: 100Mi
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
policy:
enabled: false
k8s:
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-policy
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
telemetry:
enabled: false
k8s:
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: GOMAXPROCS
value: "6"
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-telemetry
replicaCount: 1
resources:
limits:
cpu: 4800m
memory: 4G
requests:
cpu: 1000m
memory: 1G
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
hub: docker.io/istio
meshConfig:
accessLogFile: /dev/stdout
defaultConfig:
proxyMetadata: {}
enablePrometheusMerge: true
outboundTrafficPolicy:
mode: REGISTRY_ONLY
profile: demo
revision: 1-7-0
values:
base:
enableCRDTemplates: false
validationURL: ""
clusterResources: true
gateways:
istio-egressgateway:
autoscaleEnabled: false
env: {}
name: istio-egressgateway
secretVolumes:
- mountPath: /etc/istio/egressgateway-certs
name: egressgateway-certs
secretName: istio-egressgateway-certs
- mountPath: /etc/istio/egressgateway-ca-certs
name: egressgateway-ca-certs
secretName: istio-egressgateway-ca-certs
type: ClusterIP
zvpn: {}
istio-ingressgateway:
applicationPorts: ""
autoscaleEnabled: false
debug: info
domain: ""
env: {}
meshExpansionPorts:
- name: tcp-istiod
port: 15012
targetPort: 15012
- name: tcp-dns-tls
port: 853
targetPort: 8853
name: istio-ingressgateway
secretVolumes:
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
secretName: istio-ingressgateway-certs
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
secretName: istio-ingressgateway-ca-certs
type: LoadBalancer
zvpn: {}
global:
arch:
amd64: 2
ppc64le: 2
s390x: 2
configValidation: true
controlPlaneSecurityEnabled: true
defaultNodeSelector: {}
defaultPodDisruptionBudget:
enabled: true
defaultResources:
requests:
cpu: 10m
enableHelmTest: false
hub: harbor-k8s.shk8s.de/istio
imagePullPolicy: ""
imagePullSecrets: []
istioNamespace: istio-system
istiod:
enableAnalysis: false
jwtPolicy: third-party-jwt
logAsJson: false
logging:
level: default:info
meshExpansion:
enabled: false
useILB: false
meshNetworks: {}
mountMtlsCerts: false
multiCluster:
clusterName: ""
enabled: false
network: ""
omitSidecarInjectorConfigMap: false
oneNamespace: false
operatorManageWebhooks: false
pilotCertProvider: istiod
priorityClassName: ""
proxy:
autoInject: enabled
clusterDomain: cluster.local
componentLogLevel: misc:error
enableCoreDump: false
excludeIPRanges: ""
excludeInboundPorts: ""
excludeOutboundPorts: ""
image: proxyv2
includeIPRanges: "*"
logLevel: warning
privileged: false
readinessFailureThreshold: 30
readinessInitialDelaySeconds: 1
readinessPeriodSeconds: 2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 40Mi
statusPort: 15020
tracer: zipkin
proxy_init:
image: proxyv2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 10Mi
sds:
token:
aud: istio-ca
sts:
servicePort: 0
tracer:
datadog:
address: $(HOST_IP):8126
lightstep:
accessToken: ""
address: ""
stackdriver:
debug: false
maxNumberOfAnnotations: 200
maxNumberOfAttributes: 200
maxNumberOfMessageEvents: 200
zipkin:
address: ""
trustDomain: cluster.local
useMCP: false
grafana:
accessMode: ReadWriteMany
contextPath: /grafana
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- disableDeletion: false
folder: istio
name: istio
options:
path: /var/lib/grafana/dashboards/istio
orgId: 1
type: file
datasources:
datasources.yaml:
apiVersion: 1
env: {}
envSecrets: {}
image:
repository: grafana/grafana
tag: 7.0.5
nodeSelector: {}
persist: false
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
security:
enabled: false
passphraseKey: passphrase
secretName: grafana
usernameKey: username
service:
annotations: {}
externalPort: 3000
name: http
type: ClusterIP
storageClassName: ""
tolerations: []
istiocoredns:
coreDNSImage: coredns/coredns
coreDNSPluginImage: istio/coredns-plugin:0.2-istio-1.1
coreDNSTag: 1.6.2
istiodRemote:
injectionURL: ""
mixer:
adapters:
kubernetesenv:
enabled: true
prometheus:
enabled: true
metricsExpiryDuration: 10m
stackdriver:
auth:
apiKey: ""
appCredentials: false
serviceAccountPath: ""
enabled: false
tracer:
enabled: false
sampleProbability: 1
stdio:
enabled: false
outputAsJson: false
useAdapterCRDs: false
policy:
adapters:
kubernetesenv:
enabled: true
useAdapterCRDs: false
autoscaleEnabled: true
image: mixer
sessionAffinityEnabled: false
telemetry:
autoscaleEnabled: true
env:
GOMAXPROCS: "6"
image: mixer
loadshedding:
latencyThreshold: 100ms
mode: enforce
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
replicaCount: 1
sessionAffinityEnabled: false
tolerations: []
pilot:
appNamespaces: []
autoscaleEnabled: false
autoscaleMax: 5
autoscaleMin: 1
configMap: true
configNamespace: istio-config
cpu:
targetAverageUtilization: 80
enableProtocolSniffingForInbound: true
enableProtocolSniffingForOutbound: true
env: {}
image: pilot
keepaliveMaxServerConnectionAge: 30m
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
policy:
enabled: false
replicaCount: 1
tolerations: []
traceSampling: 1
prometheus:
contextPath: /prometheus
hub: docker.io/prom
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
provisionPrometheusCert: true
retention: 6h
scrapeInterval: 15s
security:
enabled: true
tag: v2.19.2
tolerations: []
sidecarInjectorWebhook:
enableNamespacesByDefault: false
injectLabel: istio-injection
objectSelector:
autoInject: true
enabled: false
rewriteAppHTTPProbe: true
telemetry:
enabled: true
v1:
enabled: false
v2:
enabled: true
metadataExchange:
wasmEnabled: false
prometheus:
enabled: true
wasmEnabled: false
stackdriver:
configOverride: {}
enabled: false
logging: false
monitoring: false
topology: false
tracing:
jaeger:
accessMode: ReadWriteMany
hub: docker.io/jaegertracing
memory:
max_traces: 50000
persist: false
spanStorageType: badger
storageClassName: ""
tag: "1.18"
nodeSelector: {}
opencensus:
exporters:
stackdriver:
enable_tracing: true
hub: docker.io/omnition
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 200m
memory: 400Mi
tag: 0.1.9
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
provider: jaeger
service:
annotations: {}
externalPort: 9411
name: http-query
type: ClusterIP
zipkin:
hub: docker.io/openzipkin
javaOptsHeap: 700
maxSpans: 500000
node:
cpus: 2
probeStartupDelay: 10
queryPort: 9411
resources:
limits:
cpu: 1000m
memory: 2048Mi
requests:
cpu: 150m
memory: 900Mi
tag: 2.20.0
version: ""
Production-Grade IstioOperator v1.7.3
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: installed-state
namespace: istio-system
spec:
addonComponents:
istiocoredns:
enabled: false
components:
base:
enabled: true
cni:
enabled: true
namespace: kube-system
egressGateways:
- enabled: true
k8s:
env:
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-egressgateway
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 40Mi
service:
ports:
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tls
port: 15443
targetPort: 15443
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
name: istio-egressgateway
ingressGateways:
- enabled: true
k8s:
env:
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-ingressgateway
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 40Mi
service:
ports:
- name: status-port
port: 15021
targetPort: 15021
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tcp
port: 31400
targetPort: 31400
- name: tls
port: 15443
targetPort: 15443
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
name: istio-ingressgateway
istiodRemote:
enabled: false
pilot:
enabled: true
k8s:
env:
- name: PILOT_TRACE_SAMPLING
value: "100"
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 1
periodSeconds: 3
timeoutSeconds: 5
resources:
requests:
cpu: 10m
memory: 100Mi
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
policy:
enabled: false
k8s:
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-policy
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
telemetry:
enabled: false
k8s:
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: GOMAXPROCS
value: "6"
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-telemetry
replicaCount: 1
resources:
limits:
cpu: 4800m
memory: 4G
requests:
cpu: 1000m
memory: 1G
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
hub: docker.io/istio
meshConfig:
accessLogFile: /dev/stdout
defaultConfig:
proxyMetadata: {}
enablePrometheusMerge: true
outboundTrafficPolicy:
mode: REGISTRY_ONLY
profile: demo
revision: 1-7-3
values:
base:
enableCRDTemplates: false
validationURL: ""
clusterResources: true
gateways:
istio-egressgateway:
autoscaleEnabled: false
env: {}
name: istio-egressgateway
secretVolumes:
- mountPath: /etc/istio/egressgateway-certs
name: egressgateway-certs
secretName: istio-egressgateway-certs
- mountPath: /etc/istio/egressgateway-ca-certs
name: egressgateway-ca-certs
secretName: istio-egressgateway-ca-certs
type: ClusterIP
zvpn: {}
istio-ingressgateway:
applicationPorts: ""
autoscaleEnabled: false
debug: info
domain: ""
env: {}
meshExpansionPorts:
- name: tcp-istiod
port: 15012
targetPort: 15012
- name: tcp-dns-tls
port: 853
targetPort: 8853
name: istio-ingressgateway
secretVolumes:
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
secretName: istio-ingressgateway-certs
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
secretName: istio-ingressgateway-ca-certs
type: LoadBalancer
zvpn: {}
global:
arch:
amd64: 2
ppc64le: 2
s390x: 2
configValidation: true
controlPlaneSecurityEnabled: true
defaultNodeSelector: {}
defaultPodDisruptionBudget:
enabled: true
defaultResources:
requests:
cpu: 10m
enableHelmTest: false
hub: harbor-k8s.shk8s.de/istio
imagePullPolicy: ""
imagePullSecrets: []
istioNamespace: istio-system
istiod:
enableAnalysis: false
jwtPolicy: third-party-jwt
logAsJson: false
logging:
level: default:info
meshExpansion:
enabled: false
useILB: false
meshNetworks: {}
mountMtlsCerts: false
multiCluster:
clusterName: ""
enabled: false
network: ""
omitSidecarInjectorConfigMap: false
oneNamespace: false
operatorManageWebhooks: false
pilotCertProvider: istiod
priorityClassName: ""
proxy:
autoInject: enabled
clusterDomain: cluster.local
componentLogLevel: misc:error
enableCoreDump: false
excludeIPRanges: ""
excludeInboundPorts: ""
excludeOutboundPorts: ""
image: proxyv2
includeIPRanges: "*"
logLevel: warning
privileged: false
readinessFailureThreshold: 30
readinessInitialDelaySeconds: 1
readinessPeriodSeconds: 2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 40Mi
statusPort: 15020
tracer: zipkin
proxy_init:
image: proxyv2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 10Mi
sds:
token:
aud: istio-ca
sts:
servicePort: 0
tracer:
datadog:
address: $(HOST_IP):8126
lightstep:
accessToken: ""
address: ""
stackdriver:
debug: false
maxNumberOfAnnotations: 200
maxNumberOfAttributes: 200
maxNumberOfMessageEvents: 200
zipkin:
address: ""
trustDomain: cluster.local
useMCP: false
grafana:
accessMode: ReadWriteMany
contextPath: /grafana
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- disableDeletion: false
folder: istio
name: istio
options:
path: /var/lib/grafana/dashboards/istio
orgId: 1
type: file
datasources:
datasources.yaml:
apiVersion: 1
env: {}
envSecrets: {}
image:
repository: grafana/grafana
tag: 7.0.5
nodeSelector: {}
persist: false
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
security:
enabled: false
passphraseKey: passphrase
secretName: grafana
usernameKey: username
service:
annotations: {}
externalPort: 3000
name: http
type: ClusterIP
storageClassName: ""
tolerations: []
istiocoredns:
coreDNSImage: coredns/coredns
coreDNSPluginImage: istio/coredns-plugin:0.2-istio-1.1
coreDNSTag: 1.6.2
istiodRemote:
injectionURL: ""
mixer:
adapters:
kubernetesenv:
enabled: true
prometheus:
enabled: true
metricsExpiryDuration: 10m
stackdriver:
auth:
apiKey: ""
appCredentials: false
serviceAccountPath: ""
enabled: false
tracer:
enabled: false
sampleProbability: 1
stdio:
enabled: false
outputAsJson: false
useAdapterCRDs: false
policy:
adapters:
kubernetesenv:
enabled: true
useAdapterCRDs: false
autoscaleEnabled: true
image: mixer
sessionAffinityEnabled: false
telemetry:
autoscaleEnabled: true
env:
GOMAXPROCS: "6"
image: mixer
loadshedding:
latencyThreshold: 100ms
mode: enforce
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
replicaCount: 1
sessionAffinityEnabled: false
tolerations: []
pilot:
appNamespaces: []
autoscaleEnabled: false
autoscaleMax: 5
autoscaleMin: 1
configMap: true
configNamespace: istio-config
cpu:
targetAverageUtilization: 80
enableProtocolSniffingForInbound: true
enableProtocolSniffingForOutbound: true
env: {}
image: pilot
keepaliveMaxServerConnectionAge: 30m
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
policy:
enabled: false
replicaCount: 1
tolerations: []
traceSampling: 1
prometheus:
contextPath: /prometheus
hub: docker.io/prom
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
provisionPrometheusCert: true
retention: 6h
scrapeInterval: 15s
security:
enabled: true
tag: v2.19.2
tolerations: []
sidecarInjectorWebhook:
enableNamespacesByDefault: false
injectLabel: istio-injection
objectSelector:
autoInject: true
enabled: false
rewriteAppHTTPProbe: true
telemetry:
enabled: true
v1:
enabled: false
v2:
enabled: true
metadataExchange:
wasmEnabled: false
prometheus:
enabled: true
wasmEnabled: false
stackdriver:
configOverride: {}
enabled: false
logging: false
monitoring: false
topology: false
tracing:
jaeger:
accessMode: ReadWriteMany
hub: docker.io/jaegertracing
memory:
max_traces: 50000
persist: false
spanStorageType: badger
storageClassName: ""
tag: "1.18"
nodeSelector: {}
opencensus:
exporters:
stackdriver:
enable_tracing: true
hub: docker.io/omnition
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 200m
memory: 400Mi
tag: 0.1.9
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
provider: jaeger
service:
annotations: {}
externalPort: 9411
name: http-query
type: ClusterIP
zipkin:
hub: docker.io/openzipkin
javaOptsHeap: 700
maxSpans: 500000
node:
cpus: 2
probeStartupDelay: 10
queryPort: 9411
resources:
limits:
cpu: 1000m
memory: 2048Mi
requests:
cpu: 150m
memory: 900Mi
tag: 2.20.0
version: ""


Power button is a two-state type of toggle button with an indicator.
Used as indicator.
Raised Power button
A power button can control the loading process.


Toggle navigation
Yoga Garden
ABOUT
CLASSES
RESOURCES
CONTACT
404
OOPS, PAGE NOT FOUND!
BACK HOME
Powered by
Hugo


type runtime.mSpanList
13 uses
runtime (current package)
mgc.go#L1018: free mSpanList
mgc.go#L1021: busy mSpanList
mheap.go#L376: type mSpanList struct {
mheap.go#L385: list *mSpanList // For debugging. TODO: Remove.
mheap.go#L1550: func (list *mSpanList) init() {
mheap.go#L1555: func (list *mSpanList) remove(span *mspan) {
mheap.go#L1576: func (list *mSpanList) isEmpty() bool {
mheap.go#L1580: func (list *mSpanList) insert(span *mspan) {
mheap.go#L1598: func (list *mSpanList) insertBack(span *mspan) {
mheap.go#L1617: func (list *mSpanList) takeAll(other *mSpanList) {
stack.go#L149: span mSpanList
stack.go#L155: free [heapAddrBits - pageShift]mSpanList // free lists by log_2(s.npages)
The pages are generated with Golds v0.2.7-preview. (GOOS=darwin GOARCH=amd64)


Blake Shelton, Ashley McBryde are early CMA Awards winners
14 Nov 2019, 02:16 by Karen Butler
Left to right, Carrie Underwood, Dolly Parton and Reba McEntire are the hosts of the CMAs. Photo by John Angelillo/UPI | License Photo
Jennifer Nettles. Photo by John Angelillo/UPI | License Photo
Keith Urban (R) and his wife, Nicole Kidman. Photo by John Angelillo/UPI | License Photo
Dolly Parton (L) and Zach Williams. Photo by John Angelillo/UPI | License Photo
Martina McBride. Photo by John Angelillo/UPI | License Photo
Happily Grey. Photo by John Angelillo/UPI | License Photo
The Stereotypes. Photo by John Angelillo/UPI | License Photo
Joe Diffie (L) and his wife, Tara. Photo by John Angelillo/UPI | License Photo
Pam Tillis. Photo by John Angelillo/UPI | License Photo
Madison Marlow (L) and Taylor Dye of the group Maddie and Tae. Photo by John Angelillo/UPI | License Photo
Nettles wears a dress reading "Equal Play." Photo by John Angelillo/UPI | License Photo
Joel and Luke Smallbone of King and Country. Photo by John Angelillo/UPI | License Photo
Chase Rice. Photo by John Angelillo/UPI | License Photo
The Highwomen. Photo by John Angelillo/UPI | License Photo
Scotty McCreery. Photo by John Angelillo/UPI | License Photo
Janie Fricke. Photo by John Angelillo/UPI | License Photo
Mickey Guyton. Photo by John Angelillo/UPI | License Photo
A Thousand Horses. Photo by John Angelillo/UPI | License Photo
Justin Moore. Photo by John Angelillo/UPI | License Photo
Danielle Bradbery. Photo by John Angelillo/UPI | License Photo
Riley Green. Photo by John Angelillo/UPI | License Photo
Buddy Quaid (L) and Tanya Tucker. Photo by John Angelillo/UPI | License Photo
Hannah Brown. Photo by John Angelillo/UPI | License Photo
Ingrid Andress. Photo by John Angelillo/UPI | License Photo
Gone West. Photo by John Angelillo/UPI | License Photo
Home Free. Photo by John Angelillo/UPI | License Photo
Maren Morris (L) and Ryan Hurd. Photo by John Angelillo/UPI | License Photo
Carrie Underwood. Photo by John Angelillo/UPI | License Photo
Parton. Photo by John Angelillo/UPI | License Photo
McEntire. Photo by John Angelillo/UPI | License Photo
The Highwomen pose with spouses and children. Photo by John Angelillo/UPI | License Photo
Kidman. Photo by John Angelillo/UPI | License Photo
Recording artist Blake Shelton won the Single of the Year CMA Award on Wednesday for "God's Country." File Photo by Jim Ruymen/UPI | License Photo
Singer Ashley McBryde, winner of the award for New Female Artist of the Year, appears backstage at the 54th annual Academy of Country Music Awards held in Las Vegas on April 7. File Photo by Jim Ruymen/UPI | License Photo
Nov. 13 (UPI) -- Blake Shelton won the Single of the Year prize for his song "God's Country" at the Country Music Association Awards show in Nashville on Wednesday.
Ashley McBryde was named New Artist of the Year and the Song of the Year honor went to Luke Combs, Wyatt B. Durrette III and Robert Williford for "Beautiful Crazy."
Host Carrie Underwood and about a dozen other female artists, including Reba McEntire , Tanya Tucker , Jennifer Nettles and Dolly Parton opened the ABC broadcast with a medley of country hits.
Moments from the red carpet of 2019 CMAs
Read More


Previous: Sharpsign, Up: Standard Macro Characters
Note that the Lisp reader will generally signal an error of type reader-error when reading an expression_2 that has been abbreviated because of length or level limits (see *print-level*, *print-length*, and *print-lines*) due to restrictions on "..", "...", "#" followed by whitespace_1, and "#)".


Benefits of Using this Webapp
Simple
No knowledge of encrypting, decrypting, or the terminal is needed
Easy to Use
No need to download external clients or interact with package managers
Platform Independent
Works anywhere in any web browser
Offline Usage
This webapp does not need internet to work
Secure
Uses the PGP standard which has gone through decades of improvements and testing
Encrypting and Decrypting on the Terminal
First, you must install a PGP client on your computer. GNU's implementation, GPG, is quite popular and available on most common platforms.
Here is how to encrypt files on the terminal
$ gpg --symmetric file.txt
You are then asked to enter the encryption passphrase.
Decrypting is similar; just pass the encrypted file to gpg
$ gpg file.txt.pgp
And enter in the passphrase
Disclaimer
This project is not affiliated with the IOTA Foundation. You use this software at your own risk. I, Peter Ryszkiewicz, am not responsible for any misuse or abuse of this software. See the MIT License below for more details.
MIT License
Copyright (c) 2017 Peter Ryszkiewicz
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


--- title: "Keynote: Christian Bizer" updated: 2016-06-30 01:00:00 layout: default group: program menu: sub ordered: 4 status: published ---
Christian Bizer is a professor at the University of Mannheim where he is part of the Data and Web Science Group. He explores technical and empirical questions concerning the development of global, decentralized information environments. Christian Bizer initialized the W3C Linking Open Data community effort. He co-founded the DBpedia project which derives a comprehensive knowledge base from Wikipedia. He also initialized the WebDataCommons project which monitors the deployment of RDFa, Microdata, and Microformat data on the Web by analysing large Web crawls. Further results of his work include the Named Graphs data model which was adopted into the RDF and SPARQL recommendations; the D2RQ mapping language for publishing relational databases on the Web; the Silk - Identity Resolution Framework, the Berlin SPARQL Benchmark, and the Mannheim Search Join Engine.
Is the Semantic Web what we expected? Adoption Patterns and Content-driven Challenges
Semantic Web technologies, such as Linked Data and Schema.org, are used by a significant number of websites to support the automated processing of their content. In the talk, I will contrast the original vision of the Semantic Web with empirical findings about the adoption of Semantic Web technologies on the Web. The analysis will show areas in which data providers behave as envisioned by the Semantic Web community but will also reveal areas in which real-world adoption patterns strongly deviate. Afterwards, I will discuss the challenges that result from the current adoption situation. To address these challenges, I will exemplify entity reconciliation, vocabulary matching, and data quality assessment techniques which exploit all semantic clues that are provided while being tolerant to noise and lazy data providers.


The following table shows containers status for the running cluster
| Id | Host IP | Bridge Name | Bridge IP | Container Id | Container IP | Interface | Host Name |
|---|---|---|---|---|---|---|---|
| 1 | Credit Card | 04/07/2014 | Call in to confirm | Credit Card | 04/07/2014 | Call in to confirm | Call in to confirm |
| 2 | Water | 01/07/2014 | Paid | Credit Card | 04/07/2014 | Call in to confirm | Call in to confirm |
| 3 | Internet | 05/07/2014 | Change plan | Credit Card | 04/07/2014 | Call in to confirm | Call in to confirm |
| 4 | Electricity | 03/07/2014 | Pending | Credit Card | 04/07/2014 | Call in to confirm | Call in to confirm |
| 5 | Telephone | 06/07/2014 | Due | Credit Card | 04/07/2014 | Call in to confirm | Call in to confirm |


--- ---
Delphi's TToolBar control manages tool buttons, TToolButton controls. Typically, the tool buttons correspond to items in an application's menu and give the user more direct access to the application's commands.
Depending on the Style property of the button placed on a tool bar, a button, a drop down button, a text button or a divider will be displayed to the user.
When tool button's Style is set to tbsDropDown, the button displays a down-pointing arrow (suitable for accessing a drop-down menu). When its DropDownMenu property points to an instance of a TPopupMenu, the button when clicked will display the associated popup menu. The actual popup menu will get displayed ONLY when you click the down-pointing arrow.
If a user clicks on the button itself and not on the down-pointing arrow the popup menu will NOT get displayed.
To make sure that the drop down menu gets displayed when you click on the button, as opposed to clicking on the down-pointing arrow, you need to handle button's OnClick event and force the display of the associated popup menu. Fortunately, there's a CheckMenuDropdown method of the tool button that you can call to manually display a button’s pop-up menu. It returns true and displays the menu if the tool button has an associated menu. It returns false if there is no menu to display.
//ensure DropDownMenu Popup to pop up when button is clicked. procedure TForm.dropDownMenuButtonClick(Sender: TObject) ; begin //dropDownMenuButton is the name of the tool button //with Style set to tbsDropDown and assigned DropDownMenu dropDownMenuButton.CheckMenuDropdown; end;


What to Stream on Hulu This Weekend
by Genevieve ScaranoHulu has a lot to offer, however, deciding what to watch on the platform isn’t always easy. If you need help putting together your binge-watching lineup, you’ve come to the right place: We’ve picked the top documentaries, TV series, and movies to stream on Hulu from November 15 to November 17. Gory zombies, boxing matches, and girl squads will probably sabotage your weekend plans, because you’ll want to stream Hulu all day and night.
Anna and the Apocalypse
It’s Christmas and the undead want to celebrate in Anna and the Apocalypse.
Blindspotting
When a young ex-con goes out with his best friend for a night, things turn for the worst in Blindspotting. He witnesses a shooting and struggles to report it or stay silent until his probation is up.
Veronica Mars
Our favorite detective is back in Veronica Mars on Hulu: In the series, the maven is investigating a string of attacks in Neptune, a popular spring break destination for college students. Is someone random behind the gruesome operation? Or is someone in her inner circle to blame?
Creed II
The showdown of a lifetime takes place in Creed II and it forces each character to confront their past. The son of a boxer who was mortally wounded and a trainer work together for another major match, and this time, it’s against the killer’s son, who has a good shot at winning.
Dollface
Breakups are difficult to heal from, as shown in Dollface. But, having a supportive squad helps get through the bad times. After she’s dumped by her boyfriend, Jules (Kat Dennings) uses her imagination and girlfriends to move past her old flame.
Mom and Dad
Mom and Dad has been described as a messed-up version of Home Alone. Reports of parents killing their kids are coming in, and a sister and brother have to escape their house ASAP.
More on Geek.com:


| Variables |
Controls |
Non-stage tailored |
Stage tailored |
| n |
414 |
416 |
413 |
| Alcohol-related variables |
|
|
|
| Number of drinks past week (M, SD) |
12.7, 12.8 |
12.9, 14.0 |
12.6, 12.9 |
| AUDIT score (M, SD) |
8.0, 3.8 |
8.4, 3.8 |
7.6, 3.3 |
| AUDIT-C score (M, SD) |
5.7, 1.5 |
5.8, 1.5 |
5.6, 1.4 |
| Number of AUD symptoms (M, SD) |
1.1, 1.4 |
1.2, 1.3 |
1.0, 1.2 |
| Socio-demographic variables |
|
|
|
| Male (n, %) |
265, 64.0 |
262, 63.0 |
266, 64.4 |
| Age in years (M, SD) |
30.1, 10.9 |
30.6, 11.7 |
29.5, 10.7 |
| Living in a partnership (n, %) |
233, 56.3 |
233 56.0 |
241 58.4 |
| School education (n, %) |
|
|
|
| <10 years |
83, 20.0 |
86, 20.7 |
91, 22.0 |
| 10– 11 years |
242, 58.5 |
217, 52.1 |
209, 50.6 |
| >11 years^{a} |
89, 21.5 |
113, 27.2 |
113, 27.4 |
| Unemployment-related variables |
|
|
|
| Recruitment site (n, %) |
|
|
|
| Site 1 |
131, 31.6 |
141, 33.9 |
156, 37.8 |
| Site 2 |
128, 30.9 |
115, 27.6 |
97, 23.5 |
| Site 3 |
155, 37.4 |
160, 38.5 |
160, 38.7 |
| Current unemployment in months (M,SD) |
5.0, 16.9 |
4.5, 16.2 |
3.9, 10.8 |
| Total unemployment in months (M, SD) |
25.6, 41.7 |
24.3, 36.3 |
20.0, 32.3 |
| Motivation to change (n, %) |
|
|
|
| Pre-contemplation |
308, 74.4 |
318, 76.4 |
321, 77.7 |
| Contemplation |
47, 11.4 |
44, 10.6 |
41, 9.9 |
| Preparation |
8, 1.9 |
12, 2.9 |
7, 1.7 |
| Action |
51, 12.3 |
42, 10.1 |
44, 10.7 |
corpus-all-uclii-01/10.1111.add.12677/tables/table2/table.svg.html
|


About Cashcow
For many years you know Cashcow from your own experience or heard about it. It cost us many years to perfection but it still cannot be challenged.


dvc diff
Show changes in DVC tracked file and directories.


Hi I'm Tyrone Walton, and I am the founder of SL Personal Training. We focuses on making optimal health a priority in your life. Sessions are lead by fitness coaches with extensive education and experience in the craft of conditioning the mind & body. We specialize in:
SL Personal Training specialists creatively make your workouts more exciting & effective. We provide the best quality of training to transform your body & meet your fitness goals.
Meet the team of professionals
Feel free to leave us a comment on the Grayscale template overview page on Start Bootstrap to give some feedback about this theme!


Heading
h1. Heading 1
There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected humour, or randomised words which don’t look even slightly.
h2. Heading 2
There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected humour, or randomised words which don’t look even.
h3. Heading 3
There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected humour, or randomised words which don’t look.
h4. Heading 4
There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected humour, or randomised words which don’t.
h5. Heading 5
There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected humour, or randomised words which.
h6. Heading 6
There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected humour, or randomised words.
Paragraph
Lorem ipsum dolor sit amet, a mark here adipisicing elit. Atque, iusto, minus sequi natus nesciunt rerum tenetur corrupti autem officiis fugiat expedita laudantium ea aspernatur
Consectetur adipisicing elit. Corrupti, aliquam, voluptates, nulla, blanditiis totam voluptatem voluptatum quod ipsa debitis non ab odio natus.
Blockquote
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer posuere erat a ante.
Someone famous in Source Title
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer posuere erat a ante.
Someone famous in Source Title


{% extends 'base_test.html' %} {% load i18n %} {% block additional-headers %} {% endblock %} {% block content %}
You are about to delete a Menu item, this procedure is irreversible.
Do you want to proceed?


{% load i18n static %} {% load fontawesome_5 %}
MrMap - Registration
{% fontawesome_5_static %}
{% include 'includes/navbar-sign-out.html' %} {% include 'includes/messages.html' %}
{% include 'includes/user_activation.html' %}
{% include 'includes/footer.html' %}


--- title: Plot training history type: docs repo: https://github.com/rstudio/keras menu: main: name: Plot training history - keras parent: keras-reference aliases: - /reference/keras/plot.keras_training_history.html - /keras/reference/plot.keras_training_history.html - /guide/keras/reference/plot.keras_training_history.html - /tools/tools/keras/reference/plot.keras_training_history.html - /installation/keras/reference/plot.keras_training_history.html - /tutorials/keras/reference/plot.keras_training_history.html - /guide/tools/keras/reference/plot.keras_training_history.html - /deploy/keras/reference/plot.keras_training_history.html - /tools/keras/reference/plot.keras_training_history.html - /tutorials/tools/keras/reference/plot.keras_training_history.html ---
Plots metrics recorded during training.
# S3 method for keras_training_history
plot(
x,
y,
metrics = NULL,
method = c("auto", "ggplot2", "base"),
smooth = getOption("keras.plot.history.smooth", TRUE),
theme_bw = getOption("keras.plot.history.theme_bw", FALSE),
...
)
| x | Training history object returned from
|
| y | Unused. |
| metrics | One or more metrics to plot (e.g. |
| method | Method to use for plotting. The default "auto" will use ggplot2 if available, and otherwise will use base graphics. |
| smooth | Whether a loess smooth should be added to the plot, only
available for the |
| theme_bw | Use |
| ... | Additional parameters to pass to the |


--- layout: page title: Radio ---
Use ARROW keys to change radio option, ENTER key submits form.
Please read the documentation to understand how these examples were created.
Updated: Sep 18th, 2020


The Test Scores App
Test-1:
Test-2:
Test-3:
Total Points:
Final Grade:
Thanks for using the Test Scores Application!


10. Using Expressions
10.5 In-Memory Evaluation
In additions to providing a database-independent WHERE clause for SQL queries,
expressions support in-memory evaluation. An expressions can be evaluated with any type
of objects that follow Java Beans method naming convention. Of course this includes
DataObjects. The following API is used for expressions evaluation:
- public Object evaluate(Object object)
Evaluates expression with object, returning the result.
- public boolean match(Object object)
Returns true if an object "matches" expression criteria.
- public java.util.List filterObjects(java.util.List objects)
Returns a list of objects from the original list that match expression criteria.
Limitation of In-Memory Expressions
Current limitation of in-memory expressions is that no collections are permitted in the object property
path. In case of DataObjects that means that path containing to-many relationships may not work for
in-memory evaluation.
|
Here is an example of evaluating expression with a single object:
public class User extends CayenneDataObject {
public String getName() {
...
}
}
...
public class NonPersistentUser extends Object {
protected String name;
public String getName() {
return name;
}
...
}
...
Expression exp = ExpressionFactory.inExp("name", new Object[] {"John", "Bob"});
User persistentObject;
NonPersistentUser nonPersistentBean;
...
// evaluate with DataObject
if(exp.match(persistentObject)) {
// do something
}
if(exp.match(nonPersistentBean)) {
// do something else
}
|
Another example - using expression to filter a list objects:
Expression exp = ExpressionFactory.likeExp("artistName", "A%");
List startWithA = exp.filterObjects(artists);
|
|


Study
Muren 2015
Skoldenberg 2011
Yamasaki 2007
Kumar 2011
Kinov 2006
Overall (I-squared = 0.0%, p = 0.812)
P=0,000
NOTE: Weights are from random effects analysis
Risedronate Control
oT
WMD (95% Cl)
2.00 (—4.79, 8.79)
7.00 (2.17, 11.83)
7.00 (0.46, 13.54)
6.00 (2.44, 9.56)
6.00 (1.92, 10.08)
5.91 (3.81, 8.01)
Weight
9.56
18.90
10.30
34.79
26.45
100.00
-13.5
0) 135