Container Security Troubleshooting
Check sensor logs
The sensor log file is located at (by default):
/usr/local/qualys/sensor/data/logs/qpa.log
Diagnostic script
Qualys provides a script to collect diagnostic information about the sensor. You must run the script on the host on which you want to collect the diagnostic information from.
The diagnostic script is present in the QualysContainerSensor.tar.xz that you downloaded for installing the sensor.
The script is called Sensor_Diagnostic_Script.py. You must have Python installed on the host in order to run the script.
The script collects the following information from the host and puts it in a tar file called SensorDiagnostic.tar. You can send that file to Qualys Support for further assistance.
The SensorDiagnostic.tar includes 'ScanInfo.json', 'qpa.log' of qualys-container-sensor from given persistent storage, docker logs of qualys-container-sensor, and all information described below in the 'SensorDiagnostic.log’ file. If ‘ScanInfo.json’ and Sensor logs are not available on the Docker host then this script creates empty ‘ScanInfo.json’ and qpa.log files, and appends “File not found” to them.
- Operating System Information (Type of OS i.e. Linux or Mac and other details)
- Proxy Configuration (Type of proxy set e.g. system, docker, cloud-agent proxy)
- CPU Architecture (Details about model, CPUs, cores, etc)
- RAM Usage (Memory allocation and utilization on host)
- Docker Version (Docker version installed on host)
- Socket Configuration (Docker socket configuration on host e.g. TCP/unix domain)
- Number of docker images (Count of all docker images and their details)
- Number of docker containers (Count of all docker containers and their details)
- CPU and Memory usage of running containers (First result of all resource usage statistics)
Sensor crashes during upgrade
Use installsensor.sh to reinstall Qualys container sensor keeping the "Storage" value as it was for earlier Sensor. This will ensure that the new sensor will not be marked as another Sensor and will simply upgrade the existing one.
For help on install command, see Installing Sensors.
At any given point in time, DO NOT delete the persistent storage. Else, the sensor deployed thereafter will be marked as a new sensor.
What if sensor restarts?
The Sensor is designed to handle restart scenarios and will continue functioning normally after restart. No customer intervention is needed until the sensor crashes. Sensor will restart according to the sensor restart policy.
Sensor restart policy
Exceptions will be handled gracefully and the sensor will restart as per its restart policy for recoverable and irrecoverable errors, as described below.
Recoverable errors
The sensor will return a recoverable error code 24 in cases like the sensor has crashed or the sensor caught an exception. In these cases, the sensor will recover on its own, and will keep on restarting. There is no max limit set on the number of restarts, but the time between two restarts will increase with the number of restarts needed, up to 16 minutes.
For example, the time between two restarts could be 1 minute to start, then 2 minutes, then 4 minutes, then 8 minutes, then 16 minutes. Once 16 minutes is reached, the time between restarts will remain at 16 minutes. No core dump file will be created.
Irrecoverable errors
If the sensor returns an irrecoverable error code, it means the sensor will not recover on its own and the sensor will exit. For standalone deployments, the sensor will exit upon receiving the irrecoverable error code. For DaemonSet deployments, when the sensor exits with an irrecoverable error code, the Kubernetes Pod restart policy will restart the exited container. Irrecoverable error codes must be resolved by making changes to the deployment files and deployment arguments.
Duplicate Kubernetes containers
While searching for containers you may see duplicates of containers orchestrated by Kubernetes. This is because Kubernetes spins up a monitoring container for every service container it brings up. Qualys container sensor sees them as two different containers and reports and scans both of the containers.
To see results without duplicate containers add the following string to queries used for searching Kubernetes containers.
not label.key:POD
For example, use this query to find running containers in Kubernetes:
state:"RUNNING" and not label.key:POD
No Sensor Activity in "Asset Details" page
Sensor activity is not recorded in Asset Details page for sensors without persistent storage.
This is normal behavior for sensors without persistent storage. To see sensor activity details, you need to install sensor with persistent storage, or you can install Qualys Cloud Agent followed by mounting the /etc/qualys while installing the sensor without persistent storage.
Below is the sensor behavior with and without persistent storage.
| Sensor and its storage | /etc/qualys | Sensor Activity |
|---|---|---|
| Sensor without storage | /etc/qualys is not mounted by default | No |
| Sensor without storage | Qualys Cloud Agent is installed and then the sensor is installed explicitly by mounting /etc/qualys | Yes |
| Sensor with storage | /etc/qualys is accessed by default | Yes |