Enabling Log Streaming

Overview

Log streaming captures performance logs at the edge in near real-time. You have access to either view all the available metrics and their related values, or choose which request-response cycle metrics you want to collect.  Currently, we are only supporting Amazon S3 for the log streaming end point and is only available for CDN and WAF. This solution is in the Limited Availability phase, which means, current features and capabilities are subject to change and new features and fields will be added. Consequently, as we make changes and add new features, we will update this page. 

You can use this document to learn how to configure your AWS account to enable StackPath to stream logs to your S3 bucket.

At a high level, you must:

• Review and configure fields
• Select and configure log destinations
  • S3: Create an IAM role in AWS

Step 1: Review and Configure Fields

The following fields are enabled by default and cannot be changed or removed.

Review the available field options below and work with your Account or Sales manager to enable the log streaming fields that are right for your site. 

Logs are sent to your repository in the format shown below. The fields are comma delimited and structured in a "Field Key : Content" format, for example, "cs-method":"post".

Note, not all of the available fields are represented in this sample.

{"account-hierarchy":["s2s4e9p8","i9j2f5d8","m5e3w3b2"],"accounthash":"m5e3w3b2","attempted-bytes":2617,"box":"gfs-cds289.fr8.hwcdn.net","c-ip":"161.97.143.54","c-port":60492,"cache-hit":"MISS","country":"DE","cs-bytes":754,"cs-method":"POST","cs-origin":"","cs-referrer":"","cs-uri-query":"","cs-uri-stem":"/xmlrpc.php","cs-user-agent":"Nuclei - Open-source project (github.com/projectdiscovery/nuclei)","cs-version":"http","date":"2022-01-27","filesize":1728,"host-header":"blog.stackpath.com:80","obfuscated-c-ip":"161.97.143.0","op-id":"05c83050-45b8-40ae-b024-ad16a9d131a3","pop-location":"fr8","pop-region":"eu","request-id":"f2d743e7-e2a8-4664-a3a3-97643a57a6e5","response-content-length":0,"response-content-range":"","response-content-type":"text/html; charset=UTF-8","response-fbt":13,"response-last-modified":"","s-ip":"151.139.86.128","sc-body-bytes-sent":1728,"sc-bytes":2617,"server-port":80,"sitehash":"k2e9a3p5","time":"00:00:01","time-taken-ms":14,"unixtime":1643241601,"vendor-id":"3rdstackpath_eu","via-info":"","x-byte-range":"","x-forwarded-for":"","x-hw":"1643241601.cds004.fr8.h2,1643241601.cds289.fr8.sc,1643241601.cdn2-wafbe01-fra1.stackpath.systems.-.w,1643241601.cds289.fr8.p"}


Step 2: Select and Configure Log Destination

Logs can be sent to your S3 bucket. Follow the steps below for the sink you would like logs sent to.

AWS S3 Bucket

StackPath needs the following information to write to your S3 bucket:

• Bucket name
• Bucket region
• Role ARN for an AWS IAM role (instructions below)
  • This role must be associated with the StackPath AWS Account ID 517500695256
  • This role must have a policy attached with write and list access to the bucket

Step 1: Create an AWS Policy for the S3 Bucket

1. Log in to your AWS console. 
2. Under Security, Identity, and Compliance, click IAM. 
3. In the left-side navigation, click Policies.
4. Click Create Policy.
5. Click JSON.

6. Paste the JSON file provided by StackPath. This policy will give StackPath access to write logs to your Amazon S3 bucket.

   "Version": "2012-10-17",   "Statement": [     {       "Effect": "Allow",       "Action": [         "s3:PutObject",         "s3:ListBucket"       ],       "Resource": [         "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",         "arn:aws:s3:::DOC-EXAMPLE-BUCKET"       ]     }   ] }

7. Replace DOC-EXAMPLE-BUCKET with the name of your S3 bucket.
8. Click Next: Tags. 
9. (Optional) Add tags to this policy.
         • To learn more, please review this AWS article.
10. Click Next: Review.
11. In Name, enter a descriptive name for the policy.
    • StackPath recommends that you enter StackPath or SP in the name.
    • Copy the policy name for later use.
12. Click Create policy.

Step 2: Create an AWS IAM role for StackPath 

1. Log in to your AWS console. 
2. Under Security, Identity, and Compliance, click IAM. 
3. In the left-side navigation, click Roles. 
4. Click Create role. 
5. Select Another AWS account. 
6. For Account ID, enter the StackPath ID: 517500695256
7. Click Next: Permissions.
8. Select the policy you created in "Step 1: Create an AWS Policy for the S3 Bucket".
9. Click Next: Tags.
10. (Optional) Add tags to this role. 
11. Click Next: Review.
12. In Role name, enter a descriptive name for the role. 
    • StackPath recommends that you enter StackPath or SP in the name. 
13. Click Create role. 

Step 3: Obtain the Newly Created Role's ARN

To stream your logs, StackPath needs the ARN of the newly created role.

1. In the AWS console, in the left-side navigation, click Roles. 
2. Select the newly created the IAM role.
3. Copy the value for Role ARN.
4. Share the following information with StackPath:
   1. Role ARN
   2. Bucket name
   3. Bucket region