Deploy in Kubernetes with TKGI - Containerd Runtime

This section assumes you have the sensor image: Obtain the Container Sensor Image

Perform the following steps for creating a DaemonSet for the Qualys sensor to be deployed in Kubernetes.

Ensure that the Container Sensor has read and write access to the persistent storage and the docker daemon socket.

Modify the cssensor-containerd-ds.yml file

Below is the Kubernetes DaemonSet deployment template that can be used to deploy the Qualys Container Sensor. For customers' convenience this template is available in the QualysContainerSensor.tar.xz. Note that you can download the yml file directly from https://github.com/Qualys/cs_sensor

The field alignment in the yml file is very important. Please make sure to honor the formatting provided in the template.

kind: List

apiVersion: v1

items:

# Create custom namespace qualys

- kind: Namespace

apiVersion: v1

metadata:

# Service Account

- kind: ServiceAccount

apiVersion: v1

metadata:

namespace: qualys

# Role for all permission to qualys namespace

- kind: Role

# if k8s version is 1.17 and earlier then change apiVersion to "rbac.authorization.k8s.io/v1beta1"

apiVersion: rbac.authorization.k8s.io/v1

metadata:

namespace: qualys

rules:

- apiGroups: ["","batch"]

resources: ["pods","jobs"]

verbs: ["get", "list", "watch","create", "delete", "deletecollection"]

- apiGroups: [""]

resources: ["pods/attach", "pods/exec"]

verbs: ["create"]

# ClusterRole for read permission to whole cluster

- kind: ClusterRole

# if k8s version is 1.17 and earlier then change apiVersion to "rbac.authorization.k8s.io/v1beta1"

apiVersion: rbac.authorization.k8s.io/v1

metadata:

rules:

- apiGroups: [""]

resources: ["pods/exec"]

verbs: ["create"]

- apiGroups: [""]

resources: ["nodes", "pods", "pods/status", "replicationcontrollers/status", "nodes/status"]

verbs: ["get"]

- apiGroups: ["apps"]

resources: ["replicasets/status", "daemonsets/status", "deployments/status", "statefulsets/status"]

verbs: ["get"]

- apiGroups: ["batch"]

resources: ["jobs/status", "cronjobs/status"]

verbs: ["get"]

# RoleBinding to assign permissions in qualys-reader-role to qualys-service-account

- kind: RoleBinding

# if k8s version is 1.17 and earlier then change apiVersion to "rbac.authorization.k8s.io/v1beta1"

apiVersion: rbac.authorization.k8s.io/v1

metadata:

namespace: qualys

subjects:

- kind: ServiceAccount

namespace: qualys

roleRef:

kind: Role

apiGroup: rbac.authorization.k8s.io

# ClusterRoleBinding to assign permissions in qualys-cluster-reader-role to qualys-service-account

- kind: ClusterRoleBinding

# if k8s version is 1.17 and earlier then change apiVersion to "rbac.authorization.k8s.io/v1beta1"

apiVersion: rbac.authorization.k8s.io/v1

metadata:

subjects:

- kind: ServiceAccount

namespace: qualys

roleRef:

kind: ClusterRole

apiGroup: rbac.authorization.k8s.io

# Qualys Container Sensor pod with

- apiVersion: apps/v1

kind: DaemonSet

metadata:

namespace: qualys

labels:

k8s-app: qualys-cs-sensor

spec:

selector:

matchLabels:

updateStrategy:

type: RollingUpdate

template:

metadata:

labels:

spec:

#tolerations:

# this toleration is to have the daemonset runnable on master nodes

# remove it if want your masters to run sensor pod

#- key: node-role.kubernetes.io/master

# effect: NoSchedule

serviceAccountName: qualys-service-account

containers:

- name: qualys-container-sensor

image: qualys/qcs-sensor:latest

imagePullPolicy : IfNotPresent

resources:

limits:

cpu: "0.5" # Default CPU usage limit on each node for sensor.

args: ["--k8s-mode", "--container-runtime", "containerd"]

env:

- name: CUSTOMERID

value: __customerId

- name: ACTIVATIONID

value: __activationId

- name: POD_URL

value:

- name: QUALYS_SCANNING_CONTAINER_LAUNCH_TIMEOUT

value: "10"

# uncomment(and indent properly) below section if proxy is required to connect Qualys Cloud

#- name: qualys_https_proxy

# value: <proxy FQDN or Ip address>:<port#>

- name: QUALYS_POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

- name: QUALYS_POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

volumeMounts:

- mountPath: /var/run/containerd/containerd.sock

readOnly: true

- mountPath: /usr/local/qualys/qpa/data

- mountPath: /usr/local/qualys/qpa/data/conf/agent-data

# uncomment(and indent properly) below section if proxy(with CA cert) required to connect Qualys Cloud

#- mountPath: /etc/qualys/qpa/cert/custom-ca.crt

# name: proxy-cert-path

securityContext:

allowPrivilegeEscalation: false

volumes:

- name: socket-volume

hostPath:

path: /var/run/containerd/containerd.sock

type: Socket

- name: persistent-volume

hostPath:

path: /usr/local/qualys/sensor/data

type: DirectoryOrCreate

- name: agent-volume

hostPath:

path: /etc/qualys

type: DirectoryOrCreate

# uncomment(and indent properly) below section if proxy(with CA cert) required to connect Qualys Cloud

#- name: proxy-cert-path

# hostPath:

# path: <proxy certificate path>

# type: File

hostNetwork: true

Qualys Container Sensor DaemonSet should be deployed in 'qualys' namespace as part of ServiceAccount with adequate permission to communicate with Kubernetes API Server. The Role, ClusterRole, RoleBinding and ClusterRoleBinding are used to assign the necessary permissions to ServiceAccount. If you already have Qualys Container Sensor running in a different namespace other than 'qualys', you’ll need to uninstall Qualys Container Sensor from the other namespace and deploy it fresh in the 'qualys' namespace.

You’ll need these permissions:

get, list, watch - to monitor the resources to be scanned for vulnerabilities across the cluster
create, delete, deletecollection - to spawn containers for image vulnerability assessment in 'qualys' namespace, scan pods across the cluster and then clean up after itself

Ensure that the Container Sensor has read and write access to the persistent storage and the containerd daemon socket.

Modify parameters in the yaml file

Copy the cssensor-containerd-ds.yml file to Kubernetes cluster's master node then modify it by providing values for the following parameters. In order for the yaml file to work properly, ensure you only update the parameters/sections specified below. Note that you can download the yml file directly from https://github.com/Qualys/cs_sensor

General Sensor is assumed here. Uncomment the tolerations section under spec if you want Sensor daemonset to be deployed on master nodes.

spec:

#tolerations:

# this toleration is to have the daemonset runnable on master nodes

# remove it if want your masters to run sensor pod

#- key: node-role.kubernetes.io/master

# effect: NoSchedule

If you want to prioritize Qualys Sensor PODs:

Locate and Uncomment the below lines of code in the .yml file. And change the PriorityClass value mentioned under kind: PriorityClass as per your requirement.

#- kind: PriorityClass

# apiVersion: scheduling.k8s.io/v1

# metadata:

# name: qualys-priority-class

# value: 0

# preemptionPolicy: PreemptLowerPriority

# description: Priority class for daemonset

Also, locate the PriorityClass name present below and Uncomment it.

#priorityClassName: qualys-priority-class

containers:

- name: qualys-container-sensor

image: <CS Sensor image name in the private/docker hub registry>

args: ["--k8s-mode", "--container-runtime", "containerd"]

If you want to deploy a Registry Sensor provide the args value as:

args: ["--k8s-mode","--container-runtime", "containerd", "--registry-sensor"]

If you want to deploy a CICD Sensor, provide the args value as:

args: ["--k8s-mode","--container-runtime", "containerd", "--cicd-deployed-sensor"]

If you want to change the log level, provide "--log-level", "<a number between 0 and 5>" as an additional value in args, e.g if you want logs in trace provide:

args: ["--k8s-mode", "--container-runtime", "containerd", "--log-level", "5"]

If you want to launch the sensor with scan thread value other than default 4, provide
"--scan-thread-pool-size", "<number of threads>" as an additional value in args.

args: ["--k8s-mode", "--container-runtime", "containerd", "--scan-thread-pool-size", "6"]

If you want to define the number of archived qpa.log files to be generated and size per log file, provide "--log-filesize", "<digit><K/M/>" where "K" means kilobyte and "M" means megabyte, and "--log-filepurgecount", "<digit>" as an additional value in args. Default is
"--log-filesize": "10M" and "--log-filepurgecount": "5" applied via config.

"--log-filesize": can be used to define the maximum size per log file. For example, "10K" (kilobytes), "10M" (megabytes) or "10" (bytes).

"--log-filepurgecount": can be used to define the number of archived log files to be generated, please note that there will always be current qpa.log file in log/ directory.

args: ["--k8s-mode", "--container-runtime", "containerd", "--log-filesize", "5M", "--log-filepurgecount", "4"]

If you want to mask environment variables for images and containers in sensor logs and in the Container Security UI, add the "--mask-env-variable" parameter to args:

args: ["--k8s-mode", "--container-runtime", "containerd", "--mask-env-variable"]

If you want to disable image scans for General Sensor, add the "--disableImageScan" parameter to args:

args: ["--k8s-mode", "--container-runtime", "containerd", "--disableImageScan"]

If you want to disable container scans, add the "--disableContainerScan" parameter to args:

args: ["--k8s-mode", "--container-runtime", "containerd", "--disableContainerScan"]

If you want to specify a scanning policy, add the "--scanning-policy" parameter to args. The available values for the scanning policy are: “DynamicScanningOnly”, “StaticScanningOnly”, and “DynamicWithStaticScanningAsFallback”.

args: ["--k8s-mode", "--scanning-policy", “StaticScanningOnly”]

If you want to disable log4j vulnerability scanning on your container images, add the
"--disable-log4j-scanning" parameter to args:

args: ["--k8s-mode", "--container-runtime", "containerd", "--disable-log4j-scanning"]

If you want to disable log4j static detection for dynamic/static image scans, add the
"--disable-log4j-static-detection" parameter to args:

args: ["--k8s-mode", "--container-runtime", "containerd", "--disable-log4j-static-detection"]

If you want to optimize image scans for General Sensor, add the "--optimize-image-scans" parameter to args:

args: ["--k8s-mode", "--container-runtime", "containerd", "--optimize-image-scans"]

If you want to enable secret detection for container images, add the "--perform-secret-detection" parameter to args. Note that secret detection is supported only on CICD and registry sensors

args: ["--k8s-mode", “--container-runtime”, “containerd”, "--perform-secret-detection"]

If you want to enable malware detection for container images, add the "--perform-malware-detection" parameter to args. Note that malware detection is supported only on registry sensor.

args: ["--k8s-mode", “--container-runtime”, “containerd”, "--registry-sensor", "--perform-malware-detection"]

If you want to assign tags to a sensor or sensor profile,
args: ["--k8s-mode", “--container-runtime”, “containerd”, "----tag-sensor-profile"]

For more information, refer to Important Points Related to Sensor Tagging.

If you want to scan the secured private registry (HTTPS) having a self-signed certificate, you can either provide the certificate through the yaml file or use the "--insecure-registry" argument to ignore the certificate check.
In case you provide both - the certificate through the yaml file and you use "--insecure-registry" argument - you will be able to login to the secured private registry.

args: ["--k8s-mode", “--container-runtime”, “containerd”, "--registry-sensor", "--insecure-registry"]

To provide self-signed certificate through the Yaml file, follow the steps mentioned below.

1. Open the .yaml file, and search "volumeMounts:".

2. Uncomment the following lines from the code.

# name: registry-cert-volume

# subPath: ca.crt

# readOnly: true

3. Similarly, uncomment the following lines and save the file.

# - name: registry-cert-volume

# secret:

# secretName: cert-config

Under resources specify the following:

resources:

limits:

cpu: 0.5 # Default CPU usage limit (20% of one core on the host).

For example, for limiting the cpu usage to 5%, set resources:limits:cpu: "0.05". This limits the cpu usage to 5% of one core on the host.

If there are multiple processors on a node, setting the resources:limits:cpu value applies the CPU limit to one core only. For example, if you have 4 CPUs on the system and you want to set CPU limit as 20% of overall CPU capacity, then the CPU limit should be set to 0.8 i.e., 80% of one core only which becomes 20% of total CPU capacity.

To disable any CPU usage limit, set resources:limits:cpu value to 0.

Optionally, if you want to specify the memory resources for Container Sensor, you can specify it under resources. Recommended values for the Container Sensor’s memory requests and memory limits are:

resources:

limits:

cpu: "0.5" # Default CPU usage limit on each node for sensor

memory: "500Mi"

requests:

memory: "300Mi"

To change the default CPU limit for image scanning PODs, specify the following variables under env.
In this example, the values were changed to 300m and 150m. The default value for 'QUALYS_SCANNING_CONTAINER_CPULIMIT' is '200m' and QUALYS_SCANNING_CONTAINER_CPUREQUEST is '100m'.

- name: QUALYS_SCANNING_CONTAINER_CPULIMIT
value: "300m"

- name: QUALYS_SCANNING_CONTAINER_CPUREQUEST
value: "150m"

If you want to specify the memory resources for SCA Scan, or Secret Scan, or Malware Scan you can specify it under resources. It is recommended to use "--limit-resource-usage" argument while installing the sensor.
Recommended values for the Container Sensor’s memory requests and memory limits for above mentioned scan types are:

resources:

limits:

cpu: "0.5" # Default CPU usage limit on each node for sensor

memory: "3.5Gi"

requests:

memory: "1.5Gi"

The above recommendation is based on the average image size of 1.5GB, if the image sizes are more, please increase the memory limit accordingly.

When either of the memory resource values (limits or requests) is specified for Container Sensor, we automatically apply both memory requests and memory limits to image scanning containers. Default values are 200Mi and 700Mi, respectively.

Additionally, you could overwrite one or both values by specifying the following variables under env. In this example, the values were changed to 300Mi and 800Mi.

- name: QUALYS_SCANNING_CONTAINER_MEMORYREQUESTMB

value: "300Mi"

- name: QUALYS_SCANNING_CONTAINER_MEMORYLIMITMB

value: "800Mi"

Under env specify the following:

Activation ID (Required)

- name: ACTIVATIONID

value: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

Customer ID (Required)

- name: CUSTOMERID

value: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

Specify POD_URL when using docker hub image. Otherwise, remove it.

- name: POD_URL

value: <Specify POD URL>

Specify the scanning container launch timeout in minutes. If this env variable is not present, then 10 minutes is the default.

- name: QUALYS_SCANNING_CONTAINER_LAUNCH_TIMEOUT

value: "10"

With each scan, we check the node status to see if the node is schedulable or not, and launch the scan only if the node is schedulable. If the node status indicates that the node is unschedulable, then we retry the scan after a default interval of 15 minutes. You can increase or decrease the time the sensor waits before retrying the scan by specifying a different scan retry interval in minutes.

- name: UNSCHEDULABLE_NODE_SCAN_RETRY_INTERVAL

value: "30"

Uncomment and indent properly the proxy information, or keep it as is if not required:

- For the communication between the sensor and the backend (Container Management Service):

#- name: qualys_https_proxy

# value: <proxy FQDN or Ip address>:<port#>

- For a registry sensor version 1.21.0 or later used for a public registry:

#- name: https_proxy

# value: <proxy FQDN or Ip address>:<port#>

Uncomment proxy-cert-path under volumes, or keep it as is if not required:

#- name: proxy-cert-path

# hostPath:

# path: /root/cert/proxy-certificate.crt

# type: File

Activation ID and Customer ID are required. Use the Activation ID and Customer ID from your subscription. To get the Activation ID and Customer ID, login to the Container Security UI, go to Configurations > Sensors, click Download, and then click any sensor type. The installation command on the Installation Instructions screen contains your Activation ID and Customer ID. Activation ID is like a password, do not share it.

If you are using an https proxy, ensure that all Kubernetes nodes have a valid certificate file for the sensor to communicate with the Container Management Server.

If you are not using a proxy and you have kept the above-mentioned parts commented, you can keep the following part commented from volumeMounts as well:

#- mountPath: /etc/qualys/qpa/cert/custom-ca.crt

# name: proxy-cert-path

How to deploy the Container Sensor DaemonSet

Once you have modified the cssensor-containerd-ds.yml file, run the following command on Kubernetes master to create a DaemonSet:

kubectl create -f cssensor-containerd-ds.yml

How to remove the Container Sensor DaemonSet

To uninstall Qualys Container Sensor, run the following command on Kubernetes master:

kubectl delete -f cssensor-containerd-ds.yml

The persistent storage will need to be removed manually on each worker node.

Launch sensor without persistent storage

You can run the sensor without using persistent storage on host. In this case data is not stored on host but stored at the /usr/local/qualys/qpa/data folder relative to the Sensor.

To launch sensor without persistent storage, modify the cssensor-containerd-ds.yml file and provide "--sensor-without-persistent-storage" as an additional value in args.

args: ["--k8s-mode", "--container-runtime", "containerd", "--sensor-without-persistent-storage"]

It is recommended to use the "--enable-console-logs" option along with "--sensor-without-persistent-storage" to preserve the logs.

Under volumeMounts remove/comment the persistent-volume section.

volumeMounts:

- mountPath: /usr/local/qualys/qpa/data

Under volumes remove/comment the persistent-volume section.

volumes:

- name: persistent-volume

hostPath:

path: /usr/local/qualys/sensor/data

type: DirectoryOrCreate

How to Tag Target Images for the CICD Sensor

To tag CICD images with qualys_scan_target:<tag> to mark them for scanning, use the nerdctl tool. Use <image-repo:tag> for tagging an image with qualys_scan_target:<any-tag>.

nerdctl -n k8s.io tag <image-repo:tag> qualys_scan_target:<any-tag>

For example,

./nerdctl -n k8s.io tag docker.io/library/known:latest qualys_scan_target:known

The nerdctl binary must be available on the host. You must create target images in the k8s.io namespace only.