Asset De-duplication in Network Passive Sensor
Asset De-duplication With Managed Assets
Qualys backend services consume asset information from multiple sources such as Qualys Cloud agent (CA), Qualys scanner, Cloud Agent Passive Sensing (CAPS), and Network Passive Sensors (NPS), among others.
Qualys backend services consider an asset as managed when it is reported by CA or scanned by Qualys scanner. The asset is considered unmanaged if reported by CAPS or NPS but not yet seen or reported through managed sources. The services rely on an internal asset feed to keep track of all managed assets.
When NPS or CAPS reports assets, the services cross-check the attributes for matching ones in the managed asset information it has tracked via the feed. The de-duplication process is initiated only when NPS or CAPS reports asset updates. NPS de-duplicates the passively sensed asset with the managed asset using a criterion such as the Cloud Agent correlation ID, MAC address, hostname, or IP address (Refer to the following section for IP-based de-duplication criteria). However, various factors may affect the time it takes for the de-duplication to trigger. A de-duplicated asset is always listed in the managed inventory. For more information, refer to Time to trigger for de-duplication.
De-duplication uses the following criteria in the order of descending priority:
- De-duplication based on correlation ID works if the passive sensor discovers the correlation ID, and the same is available for a managed asset. The way NPS learns the correlation ID is if it gets a copy of the scan traffic when the Qualys scanner runs a VM scan of the asset containing a cloud agent, and the scan profile has the QID that queries the asset for its correlation ID.
The agent correlation ID is generated by agents installed on Windows and Linux hosts. VM scan also has an option to use the agent correlation ID to de-duplicate agent-collected data with the results of authenticated or unauthenticated scans.
- De-duplication uses MAC if the asset is not de-duplicated using the correlation ID.
- If MAC is also not known, then an exact match is necessary for hostname-based de-duplication. If the passive sensor senses "johndoe" as the hostname and the managed asset’s hostname is reported with a domain name such as "johndoe.somedomain.org" or vice versa, the assets will not duplicate. In such cases, the user can add domain names such as “somedomain.org” as input to NPS, for it to de-duplicate them with the managed asset.
To add a domain, go to the Sensors tab and click View Details from the Quick Action menu of a sensor. Alternatively, you can click the sensor and go to the sensor details page > General Settings > Add domain.
To learn more about configuring the domains, refer to the Manage Sensors. - Finally, if neither of the above conditions is met, then NPS uses only the IP address to de-duplicate, provided both managed and unmanaged IPs are in the same network.
For details on the need for the Network feature, refer to Appendix D: Extending the Network Feature.
If the user does not have “Network” in the subscription, NPS defaults to treating all IPs as a part of one global default network.
IP-only de-duplication uses additional IP configuration: dynamic (DHCP) and static IPs. An unmanaged asset with a static IP is immediately de-duplicated with the managed asset of the same IP. If the unmanaged asset has a dynamic IP assigned from the DHCP pool, the de-duplication with the managed asset of the same IP happens, provided the timestamp of the managed scan and the timestamp of the asset seen by the passive sensor are close to each other, i.e., within a stipulated time period. This time period is the asset's DHCP lease period, identified by PS, or the IP inactivity time period if PS is unable to discover the DHCP lease period in cases where the DHCP flow is not seen on the sniffing interface or the appliance is overloaded and drops the DHCP packet. The user can configure an IP subnet/range as static or dynamic at the time of configuring the internal asset configuration for the appliance.
Time to trigger for de-duplication
The time takes for the de-duplication to trigger depends on the following factors:
- It is possible that the CA or Qualys Scanner has reported the managed asset, but the passive sensor has not yet. This is because the passive sensor is not deployed in the access network and cannot detect attributes such as the asset's MAC address.
- The Qualys scanner detected the asset through an unauthenticated scan, which only provided the IP address and no MAC or hostname information. The passive sensor also detected the asset using only the IP address, but the time difference between the reports from both sources is greater than the IP inactivity period.
- Ideally, de-duplication should happen in real-time. However, even though the Qualys platform is designed to handle multiple tenants and can scale up or down based on the load, there may be temporary delays due to various factors such as batch processing of workloads, scheduled maintenance downtime, or temporary surges in workload. This can result in a backlog of data that needs to be processed, which could lead to assets being deduplicated at a later time.
The asset reported by the passive sensor is listed in the unmanaged inventory provided:
- Asset is not detected by an active scan
- Asset is detected by active scan but not yet de-duplicated
De-duplication Within Unmanaged Assets
NPS de-duplicates assets within the unmanaged inventory based on MAC address, hostname, or IP.
De-duplication uses the following criteria:
- Hostname-based de-duplication happens only for non-mobile devices or for hostnames not added to the exclusion list.
For more details on adding hostnames to the exclusion list and why it is needed, refer to the General Settings section. - NPS de-duplicates assets within the unmanaged inventory based on IP, when the same IP is reported by multiple passive sensors, provided all sensors are part of the same network (Network subscription and configuration are needed only when the enterprise network has overlapping/same IP addresses). As a result, if more than one passive sensors are deployed in a network and are configured with the same IP addresses in the internal asset group, then NPS de-duplicates assets reported by different passive sensors if the IP address is the same and provided that the MAC address is not learnt. IPs configured as static qualify for immediate de-duplication, whereas for IPs configured as DHCP, assets will be de-duplicated only if their timestamps are close enough – i.e., within IP inactivity time.
If the user configures Network Passive Sensor (NPS) with an IP as static in one internal asset group and the same IP as DHCP in a different internal asset group, then NPS treats the IP as static.
An enterprise having two subnets that have the same/overlapping IP addresses would need:
- to deploy 2 passive sensors, one for each network.
- would want to inventory assets uniquely within each subnet and not have the same IP (seen without MAC or hostname) reported by the 2 passive sensors, de-duplicate into one asset.
To do this, the user must associate each sensor with its own network and also create network range tags that are a combination of network and IP ranges. A tag can be created in VMDR > Assets > Asset Groups > New Asset Group OR CSAM > Tags > New Tag > Create tag. Create a dynamic tag with the rule "IP Address in Range(s) + Network(s)".
To elaborate on the above with an example, take the case of an enterprise with two branch offices in locations L1 and L2. Both locations have the same subnet allocated to the Wi-Fi network, say IP range IPn. Passive sensors NPS1 and NPS2 are two sensors deployed to sense subnet IPn in locations L1 and L2, respectively.
User must,
- Create 2 networks N1 and N2 for locations L1 and L2
- Configure NPS1 with network N1 and NPS2 with network N2
- Create tags T1 and T1 of type IP2+Network such that T1 = N1, IPn and T2= N2, IPn
The above configuration ensures that when same IP is reported by NPS1 and NPS2, 2 separate assets one in each network is created.
The following table summarizes the asset de-duplication criteria used in NPS:
| Qualys Cloud Agent CorrelationID | Macs | IPs | Hostnames | Network | De-duplicate? |
|---|---|---|---|---|---|
| Same | Does not matter | Does not matter | Does not matter | Does not matter | Yes |
| N/A | Same | Does not matter | Does not matter | Does not matter | Yes |
| N/A | Different | Does not matter | Same | Does not matter | Yes |
| N/A | Different | Same | Different | Same | Yes, within IP inactivity |
| N/A | Different | Same | Different | Different | No |
| N/A | N/A | Same | N/A | Same | Yes, within IP inactivity |
| N/A | N/A | Same | N/A | Different | No |
Exclude Hostname from Asset de-duplication
User has an option to specify hostnames that NPS should exclude while de-duplicating assets using hostnames. NPS de-duplicates managed and unmanaged assets based on multiple conditions, of which hostname is one.
To explain why a user may have to exclude some hostnames from being considered in de-duplication criteria, let us take one example, which is a common scenario in Operational Technology (OT) network topologies. Consider an OT network having many industrial access level switches where each switch is connected to assets such as PLCs (Programmable Logic Controller), HMIs (Human Machine Interface), RTUs (Remote Terminal Unit), motion control drives, and so on. It is common in OT networks to only minimally configure the devices and leave the other configurable parameters unchanged. So, all the industrial switches could be minimally configured to have different IPs, but their hostnames are unchanged and remain the same as the default configuration. Similarly, for HMI devices as well. As a result, one common hostname is used by all switches, and another common one is used by all HMI devices. To track each such device as an independent asset in the inventory, the user can add the common switch hostname and the common HMI hostname in the exclusion list provided by this feature.