Overview
Log streaming captures performance logs at the edge in near real-time. You have access to either view all the available metrics and their related values, or choose which request-response cycle metrics you want to collect. Currently, we are only supporting Amazon S3 for the log streaming end point and is only available for CDN and WAF. This solution is in the Limited Availability phase, which means, current features and capabilities are subject to change and new features and fields will be added. Consequently, as we make changes and add new features, we will update this page.
You can use this document to learn how to configure your AWS account to enable StackPath to stream logs to your S3 bucket.
At a high level, you must:
- Review and configure fields
- Select and configure log destinations
- S3: Create an IAM role in AWS
Step 1: Review and Configure Fields
The following fields are enabled by default and cannot be changed or removed.
Index |
Field |
Field Key |
Type |
Format |
Description |
-1 |
Site Hash |
sitehash |
string |
|
In json, this field will be at the top level and not part of "m1" |
-2 |
Box |
box |
string |
|
The box and service identifier |
-3 |
Request ID |
request_id |
string |
|
UUID request number |
-4 |
Sequence Number |
sequence number |
integer |
|
Incrementing sequence number. This number does not persist will begin at 1 on restart |
-5 |
Account Hash |
accounthash |
string |
|
The immediate parent account hash for the site |
-6 |
Account Hierarchy |
account-hierarchy |
json array |
|
The full list of account hierarchy |
Review the available field options below and work with your Account or Sales manager to enable the log streaming fields that are right for your site.
Click here to view all available Log Streaming Fields
Index |
Field |
Field Key |
Type |
Format |
Description |
1 |
Request Date |
date |
string |
YYYY-MM-DD |
The date the request was made to the CDN. |
2 |
Request Time |
time |
string |
HH:MM:SS |
The time represented in GMT when the request was made to the CDN. |
3 |
HTTP Method |
cs-method |
string |
string |
Request Method: The HTTP method used for the request (HEAD, GET). |
4 |
Client IP |
obfuscated-c-ip |
string |
a.b.c.d |
The Obfuscated IP address of the user agent making the request. |
5 |
HTTP Version |
cs-version |
string |
string |
Request Protocol: This should either be HTTP or HTTPS depending on how the content was requested on the CDN. |
6 |
Referrer |
cs-referrer |
string |
URL |
The referrer header sent by the user agent. A referrer is not a required header so this field may not always have a URL. |
7 |
User-Agent String |
cs-user-agent |
string |
string |
The string sent by the user agent to the identity itself. |
8 |
File Size |
filesize |
integer |
bytes |
Size of the asset being delivered. |
9 |
Request Size |
cs-bytes |
integer |
bytes |
The total size of the request header. Request Bytes (Header + Body) |
10 |
Response Bytes Sent |
sc-bytes |
integer |
bytes |
Bytes Sent: Total bytes in the response to the client. |
11 |
Web Server IP |
s-ip |
string |
a.b.c.d [port] |
Server IP: The IP address of the edge server - we default to the Anycast IP. |
12 |
Response Seconds |
time-taken |
float |
floating-point number with 3 decimal places |
The number of seconds (accurate to the millisecond) taken to deliver the asset. |
13 |
HTTP Status Code |
sc-status |
integer |
number |
Response Status: The HTTP status code returned to the user agent. This number should be a valid HTTP status code. |
14 |
Query String |
cs-uri-query |
string |
string |
The query string parameters provided in the request. |
15 |
Path |
cs-uri-stem |
string |
URI |
The full URI of the request. |
16 |
Range Bytes |
x-byte-range |
string |
range |
Request Range Header: For range request, you'll see the range requested and delivered to the client in the form bytes=<range requested>. |
17 |
Host Header |
host-header |
string |
The domain name, for example, example.com. |
|
19 |
Debug Header |
x-hw |
string |
|
X-HW Header |
20 |
Cache Hit Information |
cache-hit |
string |
HIT/MISS |
Indicates whether a request is hit. Valid values: HIT and MISS. |
21 |
Origin Pull Request ID |
op-id |
string |
|
Similar to Request ID but for Origin Pull request |
22 |
Client IP Address |
c-ip |
string |
|
The IP address of a client, for example, 34.26.127.44/240e:ff:a008:30:3::12. |
23 |
Client Port |
c-port |
integer |
|
The TCP port between a client and CDN, for example, port 43245. |
24 |
Node Port |
server-port |
integer |
|
The server port used to provide external services, for example, port 80. |
25 |
Request Time |
unixtime |
integer |
|
The UNIX time in seconds, for example, 1562142849. |
26 |
Request Time in Milliseconds |
time-taken-ms |
integer |
|
The interval between the time when a request is received and the time when all responses are returned, in milliseconds. |
27 |
Response Bytes without Header |
sc-body-bytes-sent |
integer |
|
The number of bytes in an actual response body, excluding the HTTP header. |
28 |
Server Country |
country |
string |
XX |
The 2-alphabet code of the country where the request is served in upper caseases conforming to ISO 3166 standard https://www.iban.com/country-codes |
29 |
Time to First Byte |
response-fbt |
integer |
|
Time from processing to us putting bytes into the buffer |
30 |
Client Request XFF |
x-forwarded-for |
string |
|
Value of the request header X-Forwarded-For header |
31 |
Response Content Type |
response-content-type |
string |
|
The content of Header Content-Type in an HTTP response. Example: image/webp. If no value is available, this field is empty. |
32 |
Response Last Modified |
response-last-modified |
string |
|
The content of Header Last-Modified in an HTTP response, for example, Mon, 23 Jan 2017 13:27:36 GMT. If no value is available, this field is empty. |
33 |
Response Content Length |
response-content-length |
integer |
|
The value of the Header Content-Length field in a response. It denotes the exact byte length of the HTTP body. |
34 |
Response Content Range |
response-content-range |
string |
|
The range information (created by the origin site) in the response header, for example, bytes=0-99/200. When server supports range request, this field denotes where in the full resource this partial message belongs. |
35 |
Response Header Via-Info |
via-info |
string |
|
The content of the Via header in an HTTP response. If no value is available, this field is empty. |
36 |
POP Location |
pop-location |
string |
|
POP location |
37 |
POP Region |
pop-region |
string |
|
Region Location of the POP |
39 |
Origin Header |
origin-header |
string |
|
Origin Header from Request Header |
Logs are sent to your repository in the format shown below. The fields are comma delimited and structured in a "Field Key : Content" format, for example, "cs-method":"post"
.
Note, not all of the available fields are represented in this sample.
{"account-hierarchy":["s2s4e9p8","i9j2f5d8","m5e3w3b2"],"accounthash":"m5e3w3b2","attempted-bytes":2617,"box":"gfs-cds289.fr8.hwcdn.net","c-ip":"161.97.143.54","c-port":60492,"cache-hit":"MISS","country":"DE","cs-bytes":754,"cs-method":"POST","cs-origin":"","cs-referrer":"","cs-uri-query":"","cs-uri-stem":"/xmlrpc.php","cs-user-agent":"Nuclei
- Open-source project (github.com/projectdiscovery/nuclei)","cs-version":"http","date":"2022-01-27","filesize":1728,"host-header":"blog.stackpath.com:80","obfuscated-c-ip":"161.97.143.0","op-id":"05c83050-45b8-40ae-b024-ad16a9d131a3","pop-location":"fr8","pop-region":"eu","request-id":"f2d743e7-e2a8-4664-a3a3-97643a57a6e5","response-content-length":0,"response-content-range":"","response-content-type":"text/html;
charset=UTF-8","response-fbt":13,"response-last-modified":"","s-ip":"151.139.86.128","sc-body-bytes-sent":1728,"sc-bytes":2617,"server-port":80,"sitehash":"k2e9a3p5","time":"00:00:01","time-taken-ms":14,"unixtime":1643241601,"vendor-id":"3rdstackpath_eu","via-info":"","x-byte-range":"","x-forwarded-for":"","x-hw":"1643241601.cds004.fr8.h2,1643241601.cds289.fr8.sc,1643241601.cdn2-wafbe01-fra1.stackpath.systems.-.w,1643241601.cds289.fr8.p"}
Step 2: Select and Configure Log Destination
Logs can be sent to your S3 bucket. Follow the steps below for the sink you would like logs sent to.
AWS S3 Bucket
StackPath needs the following information to write to your S3 bucket:
- Bucket name
- Bucket region
- Role ARN for an AWS IAM role (instructions below)
- This role must be associated with the StackPath AWS Account ID 517500695256
- This role must have a policy attached with write and list access to the bucket
Step 1: Create an AWS Policy for the S3 Bucket
- Log in to your AWS console.
- Under Security, Identity, and Compliance, click IAM.
- In the left-side navigation, click Policies.
- Click Create Policy.
- Click JSON.
-
Paste the JSON file provided by StackPath. This policy will give StackPath access to write logs to your Amazon S3 bucket.
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",
"arn:aws:s3:::DOC-EXAMPLE-BUCKET"
]
}
]
} - Replace DOC-EXAMPLE-BUCKET with the name of your S3 bucket.
- Click Next: Tags.
- (Optional) Add tags to this policy.
• To learn more, please review this AWS article. - Click Next: Review.
- In Name, enter a descriptive name for the policy.
- StackPath recommends that you enter StackPath or SP in the name.
- Copy the policy name for later use.
- Click Create policy.
Step 2: Create an AWS IAM role for StackPath
- Log in to your AWS console.
- Under Security, Identity, and Compliance, click IAM.
- In the left-side navigation, click Roles.
- Click Create role.
- Select Another AWS account.
- For Account ID, enter the StackPath ID: 517500695256
- Click Next: Permissions.
- Select the policy you created in "Step 1: Create an AWS Policy for the S3 Bucket".
- Click Next: Tags.
- (Optional) Add tags to this role.
- Click Next: Review.
- In Role name, enter a descriptive name for the role.
- StackPath recommends that you enter StackPath or SP in the name.
- Click Create role.
Step 3: Obtain the Newly Created Role's ARN
To stream your logs, StackPath needs the ARN of the newly created role.
- In the AWS console, in the left-side navigation, click Roles.
- Select the newly created the IAM role.
- Copy the value for Role ARN.
- Share the following information with StackPath:
- Role ARN
- Bucket name
- Bucket region