Deploy in Kubernetes - OpenShift + with CRI-O Runtime

This section assumes you have the sensor image: Obtain the Container Sensor Image

CRI-O runtime is supported by the General (host) sensor and Registry sensor. It is not supported by Build CI/CD sensor.

Modify the cssensor-openshift-crio-ds.yml file

Below is the Kubernetes DaemonSet deployment template that can be used to deploy the Qualys Container Sensor. For customers' convenience this template is available in the QualysContainerSensor.tar.xz. Note that you can download the yml file directly from https://github.com/Qualys/cs_sensor

The field alignment in the yml file is very important. Please make sure to honor the formatting provided in the template.

kind: List

apiVersion: v1

items:

# Create custom namespace qualys

- kind: Namespace

apiVersion: v1

metadata:

# Service Account

- kind: ServiceAccount

apiVersion: v1

metadata:

namespace: qualys

# Role for all permission to qualys namespace

- kind: Role

# if k8s version is 1.17 and earlier then change apiVersion to "rbac.authorization.k8s.io/v1beta1"

apiVersion: rbac.authorization.k8s.io/v1

metadata:

namespace: qualys

rules:

- apiGroups: [""]

resources: ["pods"]

verbs: ["create", "delete", "deletecollection"]

- apiGroups: ["batch"]

resources: ["jobs"]

verbs: ["get","create", "delete", "deletecollection"]

- apiGroups: [""]

resources: ["pods/attach"]

verbs: ["create"]

# ClusterRole for read permission to whole cluster

- kind: ClusterRole

# if k8s version is 1.17 and earlier then change apiVersion to "rbac.authorization.k8s.io/v1beta1"

apiVersion: rbac.authorization.k8s.io/v1

metadata:

rules:

- apiGroups: [""]

resources: ["nodes", "pods/status", "replicationcontrollers/status", "nodes/status"]

verbs: ["get"]

- apiGroups: [""]

resources: ["pods"]

verbs: ["get","list","watch"]

- apiGroups: [""]

resources: ["pods/exec"]

verbs: ["create"]

- apiGroups: ["apps"]

resources: ["replicasets/status", "daemonsets/status", "deployments/status", "statefulsets/status"]

verbs: ["get"]

- apiGroups: ["batch"]

resources: ["jobs/status", "cronjobs/status"]

verbs: ["get"]

# RoleBinding to assign permissions in qualys-reader-role to qualys-service-account

- kind: RoleBinding

# if k8s version is 1.17 and earlier then change apiVersion to "rbac.authorization.k8s.io/v1beta1"

apiVersion: rbac.authorization.k8s.io/v1

metadata:

namespace: qualys

subjects:

- kind: ServiceAccount

namespace: qualys

roleRef:

kind: Role

apiGroup: rbac.authorization.k8s.io

# ClusterRoleBinding to assign permissions in qualys-cluster-reader-role to qualys-service-account

- kind: ClusterRoleBinding

# if k8s version is 1.17 and earlier then change apiVersion to "rbac.authorization.k8s.io/v1beta1"

apiVersion: rbac.authorization.k8s.io/v1

metadata:

subjects:

- kind: ServiceAccount

namespace: qualys

roleRef:

kind: ClusterRole

apiGroup: rbac.authorization.k8s.io

- kind: SecurityContextConstraints

apiVersion: security.openshift.io/v1

metadata:

allowHostDirVolumePlugin: true

allowHostNetwork: true

allowHostIPC: false

allowHostPID: false

allowHostPorts: false

allowPrivilegedContainer: false

readOnlyRootFilesystem: false

runAsUser:

type: RunAsAny

seLinuxContext:

type: RunAsAny

users:

- system:serviceaccount:qualys:qualys-service-account

# Qualys Container Sensor pod with

- apiVersion: apps/v1

kind: DaemonSet

metadata:

namespace: qualys

labels:

k8s-app: qualys-cs-sensor

spec:

selector:

matchLabels:

updateStrategy:

type: RollingUpdate

template:

metadata:

labels:

spec:

#tolerations:

# this toleration is to have the daemonset runnable on master nodes

# remove it if want your masters to run sensor pod

#- key: node-role.kubernetes.io/master

# effect: NoSchedule

serviceAccountName: qualys-service-account

containers:

- name: qualys-container-sensor

image: qualys/qcs-sensor:latest

imagePullPolicy : IfNotPresent

resources:

limits:

cpu: "0.5" # Default CPU usage limit on each node for sensor.

args: ["--k8s-mode", "--container-runtime", "cri-o"]

env:

- name: CUSTOMERID

value: __customerId

- name: ACTIVATIONID

value: __activationId

- name: POD_URL

value:

- name: QUALYS_SCANNING_CONTAINER_LAUNCH_TIMEOUT

value: "10"

# uncomment(and indent properly) below section if proxy is required to connect Qualys Cloud

#- name: qualys_https_proxy

# value: <proxy FQDN or Ip address>:<port#>

- name: QUALYS_POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

- name: QUALYS_POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

volumeMounts:

- mountPath: /var/run/crio/crio.sock

readOnly: true

- mountPath: /usr/local/qualys/qpa/data

- mountPath: /usr/local/qualys/qpa/data/conf/agent-data

# uncomment(and indent properly) below section if proxy(with CA cert) required to connect Qualys Cloud

#- mountPath: /etc/qualys/qpa/cert/custom-ca.crt

# name: proxy-cert-path

securityContext:

allowPrivilegeEscalation: false

volumes:

- name: socket-volume

hostPath:

path: /var/run/crio/crio.sock

type: Socket

- name: persistent-volume

hostPath:

path: /usr/local/qualys/sensor/data

type: DirectoryOrCreate

- name: agent-volume

hostPath:

path: /etc/qualys

type: DirectoryOrCreate

# uncomment(and indent properly) below section if proxy(with CA cert) required to connect Qualys Cloud

#- name: proxy-cert-path

# hostPath:

# path: <proxy certificate path>

# type: File

hostNetwork: true

Qualys Container Sensor DaemonSet should be deployed in 'qualys' namespace as a part of ServiceAccount with adequate permission to communicate with Kubernetes API Server. The Role, ClusterRole, RoleBinding, and ClusterRoleBindings are used to assign the necessary permissions to the ServiceAccount.

You’ll need these permissions:

get, list, watch - to monitor the resources to be scanned for vulnerabilities across the cluster
create, delete, deletecollection - to spawn containers for image vulnerability assessment in 'qualys' namespace, scan pods across the cluster and then clean up after itself

Ensure that the Container Sensor has read and write access to the persistent storage and the cri-o daemon socket.

Modify parameters in the yaml file

Copy the cssensor-openshift-crio-ds.yml file to Openshift cluster's master node then modify it by providing values for the following parameters. In order for the yaml file to work properly, ensure you only update the parameters/sections specified below. Note that you can download the yml file directly from https://github.com/Qualys/cs_sensor

General Sensor is assumed here. As mentioned earlier, CRI-O is supported by the General (host) sensor and Registry sensor. It is not supported by Build CI/CD sensor.

Uncomment the tolerations section under spec if you want Sensor daemonset to be deployed on master nodes

spec:

#tolerations:

# this toleration is to have the daemonset runnable on master nodes

# remove it if want your masters to run sensor pod

#- key: node-role.kubernetes.io/master

# effect: NoSchedule

If you want to prioritize Qualys Sensor PODs:

Locate and Uncomment the below lines of code in the .yml file. And change the PriorityClass value mentioned under kind: PriorityClass as per your requirement.

#- kind: PriorityClass

# apiVersion: scheduling.k8s.io/v1

# metadata:

# name: qualys-priority-class

# value: 0

# preemptionPolicy: PreemptLowerPriority

# description: Priority class for daemonset

Also, locate the PriorityClass name present below and Uncomment it.

#priorityClassName: qualys-priority-class

containers:

- name: qualys-container-sensor

image: <CS Sensor image name in the private/docker hub registry>

args: ["--k8s-mode", "--container-runtime", "cri-o"]

If you want to deploy a Registry Sensor provide the args value as:

args: ["--k8s-mode","--container-runtime", "cri-o", "--registry-sensor"]

If you want to change the log level, provide "--log-level", "<a number between 0 and 5>" as an additional value in args, e.g if you want logs in trace provide

args: ["--k8s-mode", "--container-runtime", "cri-o", "--log-level", "5"]

If you want to launch the sensor with scan thread value other than default 4, provide
"--scan-thread-pool-size", "<number of threads>" as an additional value in args.

args: ["--k8s-mode", "--container-runtime", "cri-o", "--scan-thread-pool-size", "6"]

If you want to define the number of archived qpa.log files to be generated and size per log file, provide "--log-filesize", "<digit><K/M/>" where "K" means kilobyte and "M" means megabyte, and "--log-filepurgecount", "<digit>" as an additional value in args. Default is
"--log-filesize": "10M" and "--log-filepurgecount": "5" applied via config.

"--log-filesize": can be used to define the maximum size per log file. For example, "10K" (kilobytes), "10M" (megabytes) or "10" (bytes).

"--log-filepurgecount": can be used to define the number of archived log files to be generated, please note that there will always be current qpa.log file in log/ directory.

args: ["--k8s-mode", "--container-runtime", "cri-o", "--log-filesize", "5M", "--log-filepurgecount", "4"]

If you want to mask environment variables for images and containers in sensor logs and in the Container Security UI, add the "--mask-env-variable" parameter to args:

args: ["--k8s-mode", "--container-runtime", "cri-o", "--mask-env-variable"]

If you want to disable image scans for General Sensor, add the "--disableImageScan" parameter to args:

args: ["--k8s-mode", "--container-runtime", "cri-o", "--disableImageScan"]

If you want to disable container scans, add the "--disableContainerScan" parameter to args:

args: ["--k8s-mode", "--container-runtime", "cri-o", "--disableContainerScan"]

If you want to disable log4j vulnerability scanning on your container images, add the
"--disable-log4j-scanning" parameter to args:

args: ["--k8s-mode", "--container-runtime", "cri-o", "--disable-log4j-scanning"]

If you want to optimize image scans for General Sensor, add the "--optimize-image-scans" parameter to args:

args: ["--k8s-mode", "--container-runtime", "cri-o", "--optimize-image-scans"]

If you want to enable secret detection for container images, add the "--perform-secret-detection" parameter to args. Note that secret detection is supported only on CICD and registry sensors.

args: ["--k8s-mode", “--container-runtime”, “containerd”, "--perform-secret-detection"]

If you want to enable malware detection for container images, add the "--perform-malware-detection" parameter to args. Note that malware detection is supported only on registry sensor.

args: ["--k8s-mode", "--registry-sensor", "--perform-malware-detection"]

If you want to limit the memory usage of sensor container, add the "--limit-resource-usage" parameter to args.

args: ["--k8s-mode", "--limit-resource-usage"]

If you want to assign tags to a sensor or sensor profile,
args: ["--k8s-mode", "--tag-sensor-profile"]

For more information, refer to Important Points Related to Sensor Tagging.

Under resources specify the following:

resources:

limits:

cpu: "0.5" # Default CPU usage limit on each node for sensor.

For example, to limit the CPU usage to 5%, set resources:limits:cpu: "0.05". This limits the CPU usage to 5% of one core on the host.

If there are multiple processors on a node, setting the resources:limits:cpu value applies the CPU limit to one core only. For example, if you have 4 CPUs on the system and you want to set CPU limit as 20% of overall CPU capacity, then the CPU limit should be set to 0.8 i.e., 80% of one core only which becomes 20% of total CPU capacity.

To disable any CPU usage limit, set resources:limits:cpu value to 0.

Optionally, if you want to specify the memory resources for Container Sensor, you can specify it under resources. Recommended values for the Container Sensor’s memory requests and memory limits are:

resources:

limits:

cpu: "0.5" # Default CPU usage limit on each node for sensor

memory: "500Mi"

requests:

memory: "300Mi"

If you want to specify the memory resources for SCA Scan, or Secret Scan, or Malware Scan you can specify it under resources. It is recommended to use "--limit-resource-usage" argument while installing the sensor.
Recommended values for the Container Sensor’s memory requests and memory limits for above mentioned scan types are:

resources:

limits:

cpu: "0.5" # Default CPU usage limit on each node for sensor

memory: "3.5Gi"

requests:

memory: "1.5Gi"

The above recommendation is based on the average image size of 1.5GB, if the image sizes are more, please increase the memory limit accordingly.

When either of the memory resource values (limits or requests) is specified for Container Sensor, we automatically apply both memory requests and memory limits to image scanning containers. Default values are 200Mi and 700Mi, respectively.

Additionally, you could overwrite one or both values by specifying the following variables under env. In this example, the values were changed to 300Mi and 800Mi.

- name: QUALYS_SCANNING_CONTAINER_MEMORYREQUESTMB

value: "300Mi"

- name: QUALYS_SCANNING_CONTAINER_MEMORYLIMITMB

value: "800Mi"

To change the default CPU limit for image scanning PODs, specify the following variables under env.
In this example, the values were changed to 300m and 150m. The default value for 'QUALYS_SCANNING_CONTAINER_CPULIMIT' is '200m' and QUALYS_SCANNING_CONTAINER_CPUREQUEST is '100m'.

- name: QUALYS_SCANNING_CONTAINER_CPULIMIT
value: "300m"

- name: QUALYS_SCANNING_CONTAINER_CPUREQUEST
value: "150m"

Under env specify the following:

Activation ID (Required)

- name: ACTIVATIONID

value: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

Customer ID (Required)

- name: CUSTOMERID

value: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

Specify POD_URL when using docker hub image. Otherwise, remove it.

- name: POD_URL

value: <Specify POD URL>

Specify the scanning container launch timeout in minutes. If this env variable is not present, then 10 minutes is the default.

- name: QUALYS_SCANNING_CONTAINER_LAUNCH_TIMEOUT

value: "10"

With each scan, we check the node status to see if the node is schedulable or not, and launch the scan only if the node is schedulable. If the node status indicates that the node is unschedulable, then we retry the scan after a default interval of 15 minutes. You can increase or decrease the time the sensor waits before retrying the scan by specifying a different scan retry interval in minutes.

- name: UNSCHEDULABLE_NODE_SCAN_RETRY_INTERVAL

value: "30"

Specify proxy information, or remove if not required:

- name: qualys_https_proxy

value: <PROXY_URL>

Uncomment and indent properly the proxy information, or keep it as is if not required:

- For the communication between the sensor and the backend (Container Management Service):

#- name: qualys_https_proxy

# value: <proxy FQDN or Ip address>:<port#>

- For a registry sensor version 1.21.0 or later used for a public registry:

#- name: https_proxy

# value: <proxy FQDN or Ip address>:<port#>

Uncomment proxy-cert-path under volumes, or keep it as is if not required:

#- name: proxy-cert-path

# hostPath:

# path: /root/cert/proxy-certificate.crt

# type: File

Activation ID and Customer ID are required. Use the Activation ID and Customer ID from your subscription. To get the Activation ID and Customer ID, login to the Container Security UI, go to Configurations > Sensors, click Download, and then click any sensor type. The installation command on the Installation Instructions screen contains your Activation ID and Customer ID. Activation ID is like a password, do not share it.

If you are using an https proxy, ensure that all Kubernetes nodes have a valid certificate file for the sensor to communicate with the Container Management Server.

If you are not using a proxy and you have kept the above-mentioned parts commented, you can keep the following part commented from volumeMounts as well:

#- mountPath: /etc/qualys/qpa/cert/custom-ca.crt

# name: proxy-cert-path

How to deploy the Container Sensor DaemonSet

Once you have modified the cssensor-openshift-crio-ds.yml file, run the following command on OpenShift master to create a DaemonSet:

oc create -f cssensor-openshift-crio-ds.yml

How to remove the Container Sensor DaemonSet

To uninstall Qualys Container Sensor, run the following command on Openshift master:

oc delete -f cssensor-openshift-crio-ds.yml

The persistent storage will need to be removed manually on each worker node.

Launch sensor without persistent storage

You can run the sensor without using persistent storage on host. In this case data is not stored on host but stored at the /usr/local/qualys/qpa/data folder relative to the Sensor.

To launch sensor without persistent storage, modify the cssensor-openshift-crio-ds.yml file and provide "--sensor-without-persistent-storage" as an additional value in args.

args: ["--k8s-mode", "--container-runtime", "cri-o", "--sensor-without-persistent-storage"]

It is recommended to use the "--enable-console-logs" option along with "--sensor-without-persistent-storage" to preserve the logs.

Under volumeMounts remove/comment the persistent-volume section.

volumeMounts:

- mountPath: /usr/local/qualys/qpa/data

Under volumes remove/comment the persistent-volume section.

volumes:

- name: persistent-volume

hostPath:

path: /usr/local/qualys/sensor/data

type: DirectoryOrCreate