Overview
Log streaming captures performance logs at the Edge in near real-time. You have access to either view all the available metrics and their related values, or choose which request-response cycle metrics you want to collect. We currently only support Amazon S3 for the CDN log streaming end point.
This solution is in the Limited Availability phase, which means, current features and capabilities are subject to change and new features and fields may be added. Consequently, as we continue to make changes and add new features, this page is subject to updates.
You can use this document to learn how to configure your AWS account to enable StackPath to stream logs to your S3 bucket.
At a high level, you will:
- Review and select available fields (optional).
- Create and configure an AWS S3 bucket, enabling it to receive logs from StackPath.
- Create AWS IAM policy
- Create AWS IAM role
- Obtain the role's ARN
Support
Your Account or Sales representative can assist you in enabling any of the fields listed in the Optional Fields section below, or you may reach out to Support at hi@stackpath.com for assistance with this as well.
Support requires that you provide the following information when requesting assistance with enabling logs or additional fields:
- Site name
- Desired fields
- AWS bucket name
- AWS bucket region
- Role ARN for an AWS IAM role
- This role must be associated with the StackPath AWS Account ID 517500695256
- This role must have a policy attached with write and list access to the bucket
CDN Log Fields
Default Fields
The following fields are enabled by default and cannot be changed or removed:
Index |
Field |
Field Key |
Type |
Format |
Description |
-1 |
Site Hash |
sitehash |
string |
|
In json, this field will be at the top level and not part of "m1" |
-2 |
Box |
box |
string |
|
The box and service identifier |
-3 |
Request ID |
request_id |
string |
|
UUID request number |
-4 |
Sequence Number |
sequence number |
integer |
|
Incrementing sequence number. This number does not persist will begin at 1 on restart |
-5 |
Account Hash |
accounthash |
string |
|
The immediate parent account hash for the site |
-6 |
Account Hierarchy |
account-hierarchy |
json array |
|
The full list of account hierarchy |
Available Fields
The table below contains a list of all of the fields that are available for CDN log streaming.
If you do not provide StackPath with a list of specified fields, then we will enable all of them for you by default.
Index |
Field |
Field Key |
Type |
Format |
Description |
1 |
Request Date |
date |
string |
YYYY-MM-DD |
The date the request was made to the CDN. |
2 |
Request Time |
time |
string |
HH:MM:SS |
The time represented in GMT when the request was made to the CDN. |
3 |
HTTP Method |
cs-method |
string |
string |
Request Method: The HTTP method used for the request (HEAD, GET). |
4 |
Client IP |
obfuscated-c-ip |
string |
a.b.c.d |
The Obfuscated IP address of the user agent making the request. |
5 |
HTTP Version |
cs-version |
string |
string |
Request Protocol: This should either be HTTP or HTTPS depending on how the content was requested on the CDN. |
6 |
Referrer |
cs-referrer |
string |
URL |
The referrer header sent by the user agent. A referrer is not a required header so this field may not always have a URL. |
7 |
User-Agent String |
cs-user-agent |
string |
string |
The string sent by the user agent to the identity itself. |
8 |
File Size |
filesize |
integer |
bytes |
Size of the asset being delivered. |
9 |
Request Size |
cs-bytes |
integer |
bytes |
The total size of the request header. Request Bytes (Header + Body) |
10 |
Response Bytes Sent |
sc-bytes |
integer |
bytes |
Bytes Sent: Total bytes in the response to the client. |
11 |
Web Server IP |
s-ip |
string |
a.b.c.d [port] |
Server IP: The IP address of the edge server - we default to the Anycast IP. |
12 |
Response Seconds |
time-taken |
float |
floating-point number with 3 decimal places |
The number of seconds (accurate to the millisecond) taken to deliver the asset. |
13 |
HTTP Status Code |
sc-status |
integer |
number |
Response Status: The HTTP status code returned to the user agent. This number should be a valid HTTP status code. |
14 |
Query String |
cs-uri-query |
string |
string |
The query string parameters provided in the request. |
15 |
Path |
cs-uri-stem |
string |
URI |
The full URI of the request. |
16 |
Range Bytes |
x-byte-range |
string |
range |
Request Range Header: For range request, you'll see the range requested and delivered to the client in the form bytes=<range requested>. |
17 |
Host Header |
host-header |
string |
The domain name, for example, example.com. |
|
19 |
Debug Header |
x-hw |
string |
|
X-HW Header |
20 |
Cache Hit Information |
cache-hit |
string |
HIT/MISS |
Indicates whether a request is hit. Valid values: HIT and MISS. |
21 |
Origin Pull Request ID |
op-id |
string |
|
Similar to Request ID but for Origin Pull request |
22 |
Client IP Address |
c-ip |
string |
|
The IP address of a client, for example, 34.26.127.44/240e:ff:a008:30:3::12. |
23 |
Client Port |
c-port |
integer |
|
The TCP port between a client and CDN, for example, port 43245. |
24 |
Node Port |
server-port |
integer |
|
The server port used to provide external services, for example, port 80. |
25 |
Request Time |
unixtime |
integer |
|
The UNIX time in seconds, for example, 1562142849. |
26 |
Request Time in Milliseconds |
time-taken-ms |
integer |
|
The interval between the time when a request is received and the time when all responses are returned, in milliseconds. |
27 |
Response Bytes without Header |
sc-body-bytes-sent |
integer |
|
The number of bytes in an actual response body, excluding the HTTP header. |
28 |
Server Country |
country |
string |
XX |
The 2-alphabet code of the country where the request is served in upper caseases conforming to ISO 3166 standard https://www.iban.com/country-codes |
29 |
Time to First Byte |
response-fbt |
integer |
|
Time from processing to us putting bytes into the buffer |
30 |
Client Request XFF |
x-forwarded-for |
string |
|
Value of the request header X-Forwarded-For header |
31 |
Response Content Type |
response-content-type |
string |
|
The content of Header Content-Type in an HTTP response. Example: image/webp. If no value is available, this field is empty. |
32 |
Response Last Modified |
response-last-modified |
string |
|
The content of Header Last-Modified in an HTTP response, for example, Mon, 23 Jan 2017 13:27:36 GMT. If no value is available, this field is empty. |
33 |
Response Content Length |
response-content-length |
integer |
|
The value of the Header Content-Length field in a response. It denotes the exact byte length of the HTTP body. |
34 |
Response Content Range |
response-content-range |
string |
|
The range information (created by the origin site) in the response header, for example, bytes=0-99/200. When server supports range request, this field denotes where in the full resource this partial message belongs. |
35 |
Response Header Via-Info |
via-info |
string |
|
The content of the Via header in an HTTP response. If no value is available, this field is empty. |
36 |
POP Location |
pop-location |
string |
|
POP location |
37 |
POP Region |
pop-region |
string |
|
Region Location of the POP |
39 |
Origin Header |
origin-header |
string |
|
Origin Header from Request Header |
Log Format
Logs are sent to your repository in the format shown below. The fields are comma delimited and structured in a "Field Key : Content" format, for example, "cs-method":"post"
.
Note, not all of the available fields are represented in this sample:
{"account-hierarchy":["s2s4e9p8","i9j2f5d8","m5e3w3b2"],"accounthash":"m5e3w3b2","attempted-bytes":2617,"box":"gfs-cds289.fr8.hwcdn.net","c-ip":"161.97.143.54","c-port":60492,"cache-hit":"MISS","country":"DE","cs-bytes":754,"cs-method":"POST","cs-origin":"","cs-referrer":"","cs-uri-query":"","cs-uri-stem":"/xmlrpc.php","cs-user-agent":"Nuclei
- Open-source project (github.com/projectdiscovery/nuclei)","cs-version":"http","date":"2022-01-27","filesize":1728,"host-header":"blog.stackpath.com:80","obfuscated-c-ip":"161.97.143.0","op-id":"05c83050-45b8-40ae-b024-ad16a9d131a3","pop-location":"fr8","pop-region":"eu","request-id":"f2d743e7-e2a8-4664-a3a3-97643a57a6e5","response-content-length":0,"response-content-range":"","response-content-type":"text/html;
charset=UTF-8","response-fbt":13,"response-last-modified":"","s-ip":"151.139.86.128","sc-body-bytes-sent":1728,"sc-bytes":2617,"server-port":80,"sitehash":"k2e9a3p5","time":"00:00:01","time-taken-ms":14,"unixtime":1643241601,"vendor-id":"3rdstackpath_eu","via-info":"","x-byte-range":"","x-forwarded-for":"","x-hw":"1643241601.cds004.fr8.h2,1643241601.cds289.fr8.sc,1643241601.cdn2-wafbe01-fra1.stackpath.systems.-.w,1643241601.cds289.fr8.p"}
AWS Configuration
Logs can be sent to your S3 bucket. Follow the steps below to configure the sink you would like logs sent to.
Step 1: Create an AWS Policy for the S3 Bucket
- Log in to your AWS console.
- Under Security, Identity, and Compliance, click IAM.
- In the left-side navigation, click Policies.
- Click Create Policy.
- Click JSON.
-
Paste the JSON file provided by StackPath. This policy will give StackPath access to write logs to your Amazon S3 bucket.
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",
"arn:aws:s3:::DOC-EXAMPLE-BUCKET"
]
}
]
} - Replace DOC-EXAMPLE-BUCKET with the name of your S3 bucket.
- Click Next: Tags.
- (Optional) Add tags to this policy.
• To learn more, please review this AWS article. - Click Next: Review.
- In Name, enter a descriptive name for the policy.
- StackPath recommends that you enter StackPath or SP in the name.
- Copy the policy name for later use.
- Click Create policy.
Step 2: Create an AWS IAM role for StackPath
- Log in to your AWS console.
- Under Security, Identity, and Compliance, click IAM.
- In the left-side navigation, click Roles.
- Click Create role.
- Select Another AWS account.
- For Account ID, enter the StackPath ID: 517500695256
- Click Next: Permissions.
- Select the policy you created in the "Create an AWS Policy for the S3 Bucket" section above.
- Click Next: Tags.
- (Optional) Add tags to this role.
- Click Next: Review.
- In Role name, enter a descriptive name for the role.
- StackPath recommends that you enter StackPath or SP in the name.
- Click Create role.
Step 3: Obtain the Newly Created Role's ARN
To stream your logs, StackPath needs the ARN of the newly created role.
- In the AWS console, in the left-side navigation, click Roles.
- Select the newly created the IAM role.
- Copy the value for Role ARN.
- Share the following information with StackPath:
- Role ARN
- This role must be associated with the StackPath AWS Account ID 517500695256
- This role must have a policy attached with write and list access to the bucket
- Bucket name
- Bucket region
- Role ARN
Streaming CDN and WAF Logs
It is possible to use the same bucket for a site that has both CDN and WAF logging enabled, however, if you select this method, we do recommend configuring the bucket to use a separate subdirectory/file prefix to prevent the mixing of both log types.
We recommend configuring the subdirectories for both your CDN and WAF logs as follows:
bucketname/stackpath/cdn-access/{{sitehash}}
bucketname/stackpath/waf-access/{{sitename}}