StackPath Support

Edge Rules Recipe: Allow Blank Referrers for Social Networks

This article shows how to allow blank referrers for social network crawlers when HTTP referrer protection is enabled. Social networks use different user agents to access content, so we will allow blank referrer for these user agents only. Any other user agents attempting to access CDN assets will receive a 403 Forbidden status code, as well as requests with non-listed referrers.

The HTTP referer (originally a misspelling of referrer) is an HTTP header field that identifies the address of the webpage (i.e. the URI or IRI) that linked to the resource being requested. By checking the referrer, the new webpage can see where the request originated. -Wikipedia

Skill Level: Intermediate 

Time Required: 10 minutes

Tools Required:

  • Edge Rules 

Edge Rule Setup

  1. Log into the StackPath Control Panel
  2. Select Sites and then select Manage next to the site you wish to create the edge rule with
  3. Select CDN and choose EdgeRules
  4. Select Add New Rule and then + New Rule
  5. Enter in the following directives: 
      • Add directive WHITELISTED REFERRERS and set it to www.yourdomain.com
      • Add directive SET VARIABLE  and set it to $flag 1
      • Add directive SET VARIABLE and set it to $flag 0 with the following condition: 
      • $http_user_agent as RegEx(Case Insensitive)  to value of relevant social network user agent
      • Add directive SET VARIABLE and set it to $invalid_referer $flag with the following  condition: 
      • $http_referer as RegEx(Case Insensitive) yo ^$
      • Add directive STATUS CODE and set it to 403 with the following condition: 
      • $invalid_referer as (Case Insensitive) to 1

Testing

To test this Edge Rule please ues the following two Curl examples. The results should match the examples below. 

Curl example without referrer:

curl -I http://site.company.stackpathdns.com/image.png
HTTP/1.1 403 Forbidden
Date: Tue, 24 Jan 2017 15:55:08 GMT
Content-Type: text/html
Content-Length: 162
Server: NetDNA-cache/2.2
X-Cache: MISS
Connection: keep-alive

Curl example with valid Google user agent: 

curl -I http://site.company.stackpathdns.com/image.png --referer http://www.b92.net -A "User-Agent: Google"
HTTP/1.1 200 OK
Date: Tue, 24 Jan 2017 15:55:45 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 27556
Set-Cookie: SPSI=6d9b4bed11b5ba69b8f7f8510c23612b; Path=/
Set-Cookie: adOtr=obsvl; Expires=Sat, 21-Jan-17 15:55:45 GMT; Path=/
Set-Cookie: UTGv2=D-h4e259c9224bd7c18e3f1735ce58aef01378; Expires=Wed, 24-Jan-18 15:55:45 GMT; Path=/
cache-control: no-store, no-cache, max-age=0, must-revalidate, private, max-stale=0, post-check=0, pre-check=0
Server: NetDNA-cache/2.2
X-Cache: MISS
Connection: keep-alive

 

 

Return to top
Powered by Zendesk