Troubleshoot Issues with CloudTrail Log Collection

The following article describes common issues encountered by an Alert Logic® customer during their Amazon Web Services (AWS) CloudTrail log collection setup. Symptoms of CloudTrail log collection not working properly include a lack of log collection from CloudTrail sources and a “Forbidden” error message in the status history of a CloudTrail log source.

When you have successfully configured a CloudTrail source within the Alert Logic console, the source status should switch from “New” to “OK”, and you should begin to see the message volume for the source change from zero to the number matching the volume of messages being generated by AWS CloudTrail. This process normally takes no longer than 15 minutes but may take up to an hour depending on the volume of messages being generated by the target trail. 

Solution

There are two occurrences within the Alert Logic console that often mean that a log source has not been configured correctly. A log source can remain in a “New” state and not show any messages being collected or the error “'Forbidden' AWS service is unavailable” can appear. Utilize the steps outlined below for the error that you are experiencing.

Log Source Remains “New” & Shows No Messages Collected

A log source will remain in a “New” state and will not show messages being collected under the source’s Stats page when it has not been configured correctly. Access the Stats page by clicking on the log source, which opens a right-hand panel.

Because there are multiple infrastructure components within your AWS account that allow for collection of these messages to occur, there are two different things to check for when troubleshooting:

Confirm that the CloudTrail source you are trying to collect from has logging enabled

1. Navigate to the CloudTrail section of the AWS Console.
   
   
2. Select the name of the trail from the Trails page and look at the Logging setting in the top right corner.
   
   
3. Enable the CloudTrail Logging. This will cause the trail to begin actively monitoring activity within your account. New log messages will now be generated to be collected by Alert Logic.
   
   Note: Because Alert Logic only collects log messages that have been generated after the log source has been configured, even if there are other messages that were previously generated by the CloudTrail service, they will not be collected.

Confirm that the configured AWS SQS queue has been subscribed to the SNS topic to which the CloudTrail service is publishing

1. Click on the log source within the Alert Logic console and locate the name of the SQS queue that you configured for collection.
   
   
2. Once you have noted the SQS queue name, navigate to the CloudTrail section of the AWS Console.
   
   
3. Select the name of the CloudTrail source you are trying to collect from and note the name of the SNS topic that the trail is publishing notifications to.
   
   
4. With the SNS topic and SQS queue names, navigate to the SNS section of the AWS Console and select Topics from the left navigation bar.
   
   
5. Click on the ARN of the SNS topic configured for that trail and confirm that the SQS queue you have configured for collection is listed under the topic’s subscriptions.
   
   
6. If the SQS queue you configured for collection is not listed under the topic’s subscriptions, create the subscription by selecting Create Subscription.

Error “’Forbidden’ AWS Service is Unavailable”

Confirm that the appropriate IAM policy document has been used for the source’s configured credentials

1. From the Log Sources page of the Alert Logic console, click on the pencil icon next to the CloudTrail source.
   
   Note: The Log Sources page can be found within the navigation menu (), if applicable, at Configure (or Configuration) > Deployments > select a deployment > Configure Log Sources.
   
   
2. Note the name of the IAM role configured on the log source.
   
   
3. Access the Log Management Credentials page by opening the navigation menu (), if applicable, at Configure (or Configuration) > Log Management > Credentials.
   
   
4. Click the Pencil icon next to the name of the credentials used by the source. A sidebar displays that includes a Role ID that corresponds to the ARN of the IAM Role within your AWS Console that Alert Logic is using to collect messages.
   
   
5. From the IAM section of the AWS console, select the Roles page and click on the name of the role that corresponds to the one configured within the Alert Logic credentials.
   
   
6. Under Permissions, click on the name of the IAM policy linked to the role to view its JSON template. Confirm that the policy document matches the one located within the Alert Logic Amazon Web Services Log Manager CloudTrail documentation.
   
   
7. If the policy document is correct and matches the same template provided in the documentation, confirm that the “/*” characters are located next to the resource for the s3-GetObject permission. For example, if your bucket name is “al-cloudtrail-bucket”, the resource line should look like this:
   
   “Resource”: “arn:aws:s3:::al-cloudtrail-bucket/*”
   
   
8. If the “/*” characters are not present or if there is a specific prefix other than an asterisk, this is likely the cause of the issue. Add the appropriate “/*” to the end of the resource line and save the IAM policy.

If the IAM policy is correct, the issue may lie with the S3 bucket’s Access Control List or Bucket Policy.

1. From the S3 section of the AWS Console, click on the name of the target S3 bucket and select the Permissions. Here, you can view both configuration options.
   
   
2. If you have a custom configuration and believe that changes need to be made to its settings, Alert Logic recommends that you consult with AWS Support, as editing customer configuration settings can result in bucket security issues. You can also reach out to Alert Logic Support to confirm this.