Enabling CDN Log Streaming

Overview

Log streaming captures performance logs at the Edge in near real-time. You have access to either view all the available metrics and their related values, or choose which request-response cycle metrics you want to collect.  We currently only support Amazon S3 for the CDN log streaming end point.

This solution is in the Limited Availability phase, which means, current features and capabilities are subject to change and new features and fields may be added. Consequently, as we continue to make changes and add new features, this page is subject to updates.

You can use this document to learn how to configure your AWS account to enable StackPath to stream logs to your S3 bucket.

At a high level, you will:

• Review and select available fields (optional).
• Create and configure an AWS S3 bucket, enabling it to receive logs from StackPath.
  
  • Create AWS IAM policy
  • Create AWS IAM role
    
    • Obtain the role's ARN

Support

Your Account or Sales representative can assist you in enabling any of the fields listed in the Optional Fields section below, or you may reach out to Support at hi@stackpath.com for assistance with this as well.

Support requires that you provide the following information when requesting assistance with enabling logs or additional fields:

• Site name
• Desired fields
• AWS bucket name
• AWS bucket region
• Role ARN for an AWS IAM role
  • This role must be associated with the StackPath AWS Account ID 517500695256
  • This role must have a policy attached with write and list access to the bucket

CDN Log Fields

Default Fields

The following fields are enabled by default and cannot be changed or removed:

Available Fields

The table below contains a list of all of the fields that are available for CDN log streaming.

If you do not provide StackPath with a list of specified fields, then we will enable all of them for you by default.

Log Format

Logs are sent to your repository in the format shown below. The fields are comma delimited and structured in a "Field Key : Content" format, for example, "cs-method":"post".

Note, not all of the available fields are represented in this sample:

{"account-hierarchy":["s2s4e9p8","i9j2f5d8","m5e3w3b2"],"accounthash":"m5e3w3b2","attempted-bytes":2617,"box":"gfs-cds289.fr8.hwcdn.net","c-ip":"161.97.143.54","c-port":60492,"cache-hit":"MISS","country":"DE","cs-bytes":754,"cs-method":"POST","cs-origin":"","cs-referrer":"","cs-uri-query":"","cs-uri-stem":"/xmlrpc.php","cs-user-agent":"Nuclei - Open-source project (github.com/projectdiscovery/nuclei)","cs-version":"http","date":"2022-01-27","filesize":1728,"host-header":"blog.stackpath.com:80","obfuscated-c-ip":"161.97.143.0","op-id":"05c83050-45b8-40ae-b024-ad16a9d131a3","pop-location":"fr8","pop-region":"eu","request-id":"f2d743e7-e2a8-4664-a3a3-97643a57a6e5","response-content-length":0,"response-content-range":"","response-content-type":"text/html; charset=UTF-8","response-fbt":13,"response-last-modified":"","s-ip":"151.139.86.128","sc-body-bytes-sent":1728,"sc-bytes":2617,"server-port":80,"sitehash":"k2e9a3p5","time":"00:00:01","time-taken-ms":14,"unixtime":1643241601,"vendor-id":"3rdstackpath_eu","via-info":"","x-byte-range":"","x-forwarded-for":"","x-hw":"1643241601.cds004.fr8.h2,1643241601.cds289.fr8.sc,1643241601.cdn2-wafbe01-fra1.stackpath.systems.-.w,1643241601.cds289.fr8.p"}


AWS Configuration

Logs can be sent to your S3 bucket. Follow the steps below to configure the sink you would like logs sent to.

Step 1: Create an AWS Policy for the S3 Bucket

1. Log in to your AWS console. 
2. Under Security, Identity, and Compliance, click IAM. 
3. In the left-side navigation, click Policies.
4. Click Create Policy.
5. Click JSON.

6. Paste the JSON file provided by StackPath. This policy will give StackPath access to write logs to your Amazon S3 bucket.

   "Version": "2012-10-17",   "Statement": [     {       "Effect": "Allow",       "Action": [         "s3:PutObject",         "s3:ListBucket"       ],       "Resource": [         "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",         "arn:aws:s3:::DOC-EXAMPLE-BUCKET"       ]     }   ] }

7. Replace DOC-EXAMPLE-BUCKET with the name of your S3 bucket.
8. Click Next: Tags. 
9. (Optional) Add tags to this policy.
         • To learn more, please review this AWS article.
10. Click Next: Review.
11. In Name, enter a descriptive name for the policy.
    • StackPath recommends that you enter StackPath or SP in the name.
    • Copy the policy name for later use.
12. Click Create policy.

Step 2: Create an AWS IAM role for StackPath 

1. Log in to your AWS console. 
2. Under Security, Identity, and Compliance, click IAM. 
3. In the left-side navigation, click Roles. 
4. Click Create role. 
5. Select Another AWS account. 
6. For Account ID, enter the StackPath ID: 517500695256
7. Click Next: Permissions.
8. Select the policy you created in the "Create an AWS Policy for the S3 Bucket" section above.
9. Click Next: Tags.
10. (Optional) Add tags to this role. 
11. Click Next: Review.
12. In Role name, enter a descriptive name for the role. 
    • StackPath recommends that you enter StackPath or SP in the name. 
13. Click Create role. 

Step 3: Obtain the Newly Created Role's ARN

To stream your logs, StackPath needs the ARN of the newly created role.

1. In the AWS console, in the left-side navigation, click Roles. 
2. Select the newly created the IAM role.
3. Copy the value for Role ARN.
4. Share the following information with StackPath:
   1. Role ARN
      1. This role must be associated with the StackPath AWS Account ID 517500695256
      2. This role must have a policy attached with write and list access to the bucket
   2. Bucket name
   3. Bucket region

Streaming CDN and WAF Logs

It is possible to use the same bucket for a site that has both CDN and WAF logging enabled, however, if you select this method, we do recommend configuring the bucket to use a separate subdirectory/file prefix to prevent the mixing of both log types.

We recommend configuring the subdirectories for both your CDN and WAF logs as follows:

bucketname/stackpath/cdn-access/{{sitehash}}
bucketname/stackpath/waf-access/{{sitename}}