AWS S3 Connector

Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance. You can use Amazon S3 to store and retrieve any amount of data at any time from anywhere.

What is the AWS S3 Connector?

The AWS S3 CSV connector establishes a connection from Qualys ETM to your AWS S3 Buckets to retrieve CSV files. You can then execute the CloudFormation template Qualys provides to run scheduled upload of the CSV files and fetch the latest findings from your S3 Buckets.

Prerequisites

Before creating the connector, you must create a Role in your AWS environment with the necessary authentication policies. The ARN and External ID used for this policy must be stored for later use.

1. Configure Permissions

Let's create a policy with the ARN of the required S3 Bucket resource.

  1. Navigate to the Add permissions page:
  2. Click Create policy
  3. In the JSON editor, paste the following policy:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Sid": "VisualEditor0",

            "Effect": "Allow",

            "Action": [

                "s3:GetObject"

            ],

            "Resource": "arn:aws:s3:::minimal-policy-bucket/*"

        }

    ]

}

  1. Replace minimal-policy-bucket with your actual S3 bucket name.
  2. Click Review policy
  3. Provide a name and description for the policy.

  4. Click Create policy

2. Create an IAM Role

Let's proceed by creating a new IAM Role.

  1. Log in to the AWS Management Console
  2. Navigate to IAM (Identity and Access Management)
  3. In the left navigation pane, select Roles
  4. Click Create role
  5. Under Select trusted entity:
    1. Choose AWS account
    2. Select Another AWS account
    3. Enter the Qualys AWS account ID
    4. Check the Require external ID checkbox
    5. Provide the External ID provided by Qualys.

3. Attach the Policy

  1. Select the policy you just created and click Next.
  2. Enter a role name (e.g., "QualysETMConnectorRole")
  3. Click Create role

4. Store the Role ARN

  1. Navigate to the newly created role
  2. On the role summary page, locate the Role ARN
  3. Copy the Role ARN value (it should look like: arn:aws:iam::123456789012:role/RoleName)

Create a New CSV Connector

Review and Confirm

Review the configurations provided in the previous steps. Ensure all details are correct and complete. Confirm the setup to finalize the configuration of the AWS S3 connector.

This streamlined process allows for efficient data integration, ensuring accuracy and consistency across systems.

Saving and Running the Connector

When saving and running the connector, a popup appears with the following input fields:

Upload CSV File: Upload the CSV file containing the data to be processed. This file should match the structure defined in the selected data model and will undergo the transformations specified in the chosen Transform Map.

After selecting the Transform Map and uploading the CSV file, the connector will process the data accordingly, transforming and importing it as per the configurations set.

Schedule Event-based Connector Run

The S3 connector provides a CloudFormation Tempate (CFT) that you can configure and execute in your AWS account. The CFT triggers a lambda function that listens for events in your S3 Buckets and runs the connector to discover new findings.

To configure the Event-based Connector Run, you must first download the CFT from your Connector.

  1. Navigate to the Connector app.
  2. Select your connector.
  3. From the Quick Actions menu, click Download Cloud Template.

Now, let's go to your AWS account.

  1. Navigate to CloudFormation > Stacks > Create stack > With new resources (standard)
  2. Upload the YAML template.
  3. Provide the username and password of your Qualys account. The other stack parameters are predefined. 
  4. Click Next and save the changes.
  5. Now, navigate to Events.

  6. Click Create Event notification and assign a name to the notification  

  7. Select the following configurations.

  8. Select Choose from your Lambda functions for the trigger notification and select the Lambda function created with the CloudFormation stack.

    The function only processes newly uploaded files and will not retrieve or process files that existed in the bucket before the function was configured.

  9. Click Save changes. Once, the event notification is created we can see it as below.

Now we are done with all the configurations required to run the S3 connection created on the Qualys platform. From now on, for any upload made to the configured S3 bucket, the connection is notified to fetch the file data and send it further to ETM. 

The S3 connection is only made for CSV files its is not going to parse the data for any non-csv file uploads.

AWS Lambda Concurrency Limitations with S3 File Uploads

When large numbers of files are uploaded to S3 buckets, there's an important limitation to consider with Lambda triggers:

Challenge: Lambda can handle up to 1000 concurrent executions, but runs into issues when thousands of files are uploaded simultaneously.

The Problem

When you upload thousands of files to an S3 bucket together:

  • Each upload triggers a Lambda function (via event notification)
  • Lambda immediately attempts to fetch the file using GetObject
  • If uploads happen very rapidly (within seconds), this leads to:
    • Lambda throttling (beyond 1000/second limit)
    • Dropped events (S3 won't retry failed notifications)
    • Potential S3 400/500 errors as files may still be finalizing during fetch
Recommended Solution

For users uploading files via AWS CLI or scripts:

  1. Add delays between uploads
  2. Process files in batches (~100 files per batch)

This approach allows 100 files to upload at once, then waits for the specified delay before uploading the next batch. This method:

  • Uses Lambda functions efficiently
  • Ensures the connector receives notifications for all uploads
  • Has been tested and proven effective in production environments

 

 

 

 

 

© 2025 Qualys , Inc. All rights reserved. Privacy Policy.