Troubleshooting Issues on FIM on Linux - Agent Binary (v4.1 or later)

From Linux Agent binary version 4.1 onwards, there is a common EDR process that takes care of both EDR and FIM events. Now there is no 'fimc' process.

Verifying FIM Process Status on Your Asset

How to check if the process for FIM is running on the asset?

When UseAuditDispatcher=1, it is mandatory to have auditd service is in running state and EDR plugin process could be seen registered with auditd service

[root@135302-T490 ~]# ps -elf | grep edr 0 R root 20804 20785 0 80 0 - 28162 - 03:42 pts/1 00:00:00 grep --color=auto edr 0 S root 28469 28467 0 76 -4 - 44598 futex_ Nov14 ? 00:01:23 /usr/local/qualys/cloud-agent/bin/edr-plugin /usr/local/qualys/cloud-agent/sock/audit_disp.sock 5 S root 28489 1 0 80 0 - 149099 futex_ Nov14 ? 00:23:11 /usr/local/qualys/cloud-agent/bin/qualys-edr

How EDR plugin process get registered with auditd service?

[root@135302-T490 ~]# systemctl status auditd auditd.service - Security Auditing Service Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2021-11-14 18:03:40 IST; 2 weeks 4 days ago Docs: man:auditd(8) https://github.com/linux-audit/audit-documentation Process: 28473 ExecStartPost=/sbin/augenrules --load (code=exited, status=0/SUCCESS) Process: 28464 ExecStart=/sbin/auditd (code=exited, status=0/SUCCESS) Main PID: 28465 (auditd) CGroup: /system.slice/auditd.service ├─28465 /sbin/auditd ├─28467 /sbin/audispd └─28469 /usr/local/qualys/cloud-agent/bin/edr-plugin /usr/local/qualys/cloud-agent/sock/audit_disp.sock

When UseAuditDispatcher=0, it is mandatory to have auditd service is in stopped state

[root@135302-T490 ~]# ps -elf | grep edr root 3288 1 2 10:59 ? 00:02:06 /usr/local/qualys/cloud-agent/bin/qualys-edr

Verifying FIM Profile Association with Assets

Q. How do we check if the FIM Profile (manifest) is associated with the asset?

FIM manifest (.json) can be located under following location:

'/usr/local/qualys/cloud-agent/fim/manifests/'

[root@135302-T490 ~]# ls -l /usr/local/qualys/cloud-agent/fim/manifests/ total 12 -rw------- 1 root root 9588 Sep 16 08:00 22e653bf-6d51-4933-8f01-71e4279578a8.json

Location of FIM logs

Log file for FIM has been renamed from fimc.log to edr.log and can be seen under same location: '/var/log/qualys/'

[root@135302-T490 qualys]# ls -l /var/log/qualys/ | grep edr.log -rw------- 1 root root 4558524 Dec 3 03:54 edr.log -rw------- 1 root root 10485883 Dec 3 03:45 edr.log.0 -rw------- 1 root root 10485838 Dec 3 01:59 edr.log.1 -rw------- 1 root root 10485797 Dec 2 22:56 edr.log.2 -rw------- 1 root root 10485781 Dec 2 21:55 edr.log.3 -rw------- 1 root root 10485814 Dec 2 18:45 edr.log.4

Prerequisites for FIM

Following are the prerequisites for Qualys FIM to function successfully in the auditd-compatible mode:

The auditd package must be present on the endpoint.
sudo systemctl status auditd
If SELinux is enabled, the following commands and packages should be present on system.
On RHEL based systems, using the commands, users can ensure that even if SELinux is enabled, audispd will be allowed to receive events from the auditd daemon. This allows the FIM module to receive audit events.

Commands

$ which checkmodule /usr/bin/checkmodule

# SELinux policy module package
$ which semodule_package /usr/bin/semodule_package

# SELinux policy module manager
$ which semodule /usr/sbin/semodule

$ getenforce Enforcing/permissive

Packages

In case of missing commands or utilities, you can install following packages using yum utility.

yum install policycoreutils-python

yum install policycoreutils

yum install libselinux-utils
Audit Configuration i.e. '/etc/audit/auditd.conf' must have the line “dispatcher = /sbin/audispd” uncommented.
Dispatcher daemon has been deprecated in RHEL 8.

In RHEL 8, the Audit dispatcher daemon (audisp) functionality is integrated in the Audit daemon (auditd). Configuration files of plugins for the interaction of real-time analytical programs with Audit events are located in the /etc/audit/plugins.d/ directory by default. Refer to the Auditing section of the Red Hat Documentation.

Hence, this prerequsiteis not required RHEL 8 onwards.
If the auditd is stopped externally, the FIM module does not receive any events.
Qualys Agent must be running as root user only.
The agent itself can be running as a non-root user with sudo permissions and the qualys-edr process will always run as root
Audit should not be in immutable mode.
Use the following command to check the output of 'auditctl -s' command for flag 'enabled':

[root@135302-T490 ~]# auditctl -s enabled 1 -> this is unlocked state failure 1 pid 28465 rate_limit 0 backlog_limit 64 lost 83 backlog 6 loginuid_immutable 0 unlocked

The enabled 2 means that the audit configuration is immutable.

If audit configuration is in immutable mode, it does not allow Qualys FIM module to add/edit any FIM rules. To recover from audit immutable mode, Linux-Ubuntu distribution requires a system reboot. For more information, refer to auditctl(8) — Linux manual page from Linux document.

Remediation

Remove '-e 2' rule (if present) from the following audit configuration files and reboot the machine.
1. /etc/audit/audit.rules
2. /etc/audit/rules.d/audit.rules
  
  This configuration must be remediated for Qualys FIM to function successfully
  
  To proceed with this prerequisite, the host needs to be rebooted. How should I schedule this?
  
  Once the audit configuration set i.e. '-e 2' option is removed, the host machines need to go under planned reboots. Such reboots can take place during regular maintenance cycle for activities such as scheduled system updates, patch application, kernel updates or when glibc is updated.
  
  These reboots can be scheduled and planned for batches and not all servers at a time.
The '-a never,task ' rule must not be present in the audit configuration files

Check files '/etc/audit/audit.rules' and '/etc/audit/rules.d/audit.rules' for the presence of this rule.
No other events are generated, if the "-a never,task" rule is present in the audit system in output of (auditctl -l). As per RedHat's official documentation, a "never" rule means that no audit records will be generated. This can be used to suppress event generation
Remediation
Remove the rule ' -a never,task' (if present) from the following audit configuration files and restart Qualys Agent service
1. /etc/audit/audit.rules
2. /etc/audit/rules.d/audit.rules
  
  Use the following command to restart Qualys Agent service
  
  service qualys-cloud-agent restart
  
  This issue must be remediated for Qualys FIM to function successfully.

Impact of Deploying FIM Rules in Immutable Mode on Asset Stability

If the audit system is in immutable mode and FIM Rules are pushed onto it via FIM UI (Qualys platform), can it cause an auto-reboot of the asset on which qualys-cloud-agent is deployed?

No, Qualys Cloud Agent is not designed to cause auto-reboot of the host machine.

However, auto reboot can happen if some specific set of configurations are set on the asset. For example: setting up 'kernel panic' mode in audit system.

auditctl -f [0..2] Set failure mode 0=silent 1=printk 2=panic.

This option lets you determine how you want the kernel to handle critical errors.

'-f 2' option 'panic' causes the kernel to panic to prevent use of the machine.

Even in this case, when the 'kernel panic' mode is set in the audit system, which is immutable (i.e., '-e2' is also set), auto-reboot won't happen when FIM rules are applied.

Ensuring Audit System Configuration for Kernel Panic Mode

How to check if audit system is set for 'kernel panic' mode?
Check the output of command 'auditctl -s'

[root@135302-T490 ~]# auditctl -s enabled 1 failure 2 -> this is panic mode pid 28465 rate_limit 0 backlog_limit 64 lost 83 backlog 6 loginuid_immutable 0 unlocked

Audit Compliance

Q. The prerequisite states 'Audit should not be in immutable mode,' yet CIS recommends 'Ensure the audit configuration is immutable.' Could using Qualys FIM result in non-compliance?

No, the system remains PCI Compliant as the FIM application itself is a compliance requirement. Refer to PCI-DSS Requirement no 11.5. Moreover, exceptions can be made for CIS Level 2 Profiles.

Immutable Mode in Audit system
This setting is configured to alert administrators for any modifications in the audit rules on the Linux system.

If the audit configuration is set to immutable mode, it prevents the Qualys FIM module from adding or editing any FIM rules. Therefore, it must be unlocked.

How does Qualys ensure compliance and Security when the Immutable Mode is removed?

Issue: When the immutable mode is removed, the audit rules can be modified.

Solution: Qualys FIM, with its out-of-the-box monitoring profiles, provides rules to alert for any modifications on critical audit system files in real-time, ensuring that nothing goes unnoticed.
Qualys FIM rule to monitor ‘/etc/audit/audit.rules’ file in real-time:

Qualys FIM rule to monitor ‘/etc/audit/auditd.conf’ file in real-time.

Qualys FIM rule to monitor ‘/etc/audit/rules.d/audit.rules’ file in real-time.

These rules will trigger change events for any modification, providing exhaustive event details such as the user and process responsible for the change, exact timestamp, and asset details on which the activity was performed.

We are partners and contributors with CIS and we had this situation reviewed by CIS.

Summary: The Level 2 profile serves as defense-in-depth measures that could potentially hinder the effectiveness or negatively impact the technology's performance. In such cases, we have the option to modify or revoke the CIS recommendation.

Following are some common examples where customers typically revoke the guidelines to ensure smooth operation in the production environment.

The following requirements must be met:

CIS states: 5.2.8 - Ensure SSH root login is disabled (Scored).
DISA guideline states:
- The system must disable SSH.
- Group ID (Vulid): V-72247 - The Red Hat Enterprise Linux operating system must not permit direct logons to the root account using remote access via SSH.

If we strictly follow these guidelines as per the benchmarks, we may end up disrupting many critical day-to-day functions in the production environment.

Advantages of Using auditd over Kernel Approach

Let us discuss why Qualys FIM does not employ Kernel injection. In ideal scenarios, Kernel Injection is not recommended, as modifying the kernel driver with a third-party tool may pose a significant security risk. This could also lead to performance issues, necessitating manual intervention with the endpoints. Additionally, with every version update, users would need to make changes to the kernel, which may not be a feasible option for customers. As a result, the auditd channel emerges as a more secure and comprehensive method for collecting data.

On Linux platforms, Qualys FIM operates in compatible mode with auditd and retrieves audit event notifications from the audit daemon. The Qualys FIM plugin process is initiated as a child of the audit dispatcher, which gathers the audit event notifications from the auditd daemon and disseminates them to all child plugin processes, imposing negligible load on the target.