A robots.txt file improves SEO by instructing search engine crawlers to which pages they can crawl and index. This EdgeRule will redirect any request to the StackPath CDN for a robots.txt to the file that is stored on the origin server.
If you have not already, please check this guide to get started.
Creating the Rule
- Navigate to Sites and select the CDN Site you would like to create the rule for
- Navigate to your EdgeRules tab
- Select Add Delivery Rule
- In the IF section, select the URL Option, with the Matches modifier and paste in
*://*/robots.txt - In the THEN section select the Redirect rule and add in
https://%client.request.host%/robots.txt. Please include the full path to the robots.txt file.
The initial IF statement declares that any request for "robots.txt" will trigger this rule. The %client.request.host% in the DO section is an accepted variable that will use the information in the HOST request header sent by the end-user. This can be replaced with the domain, if you would prefer, or if these would be separate values. Remember to include the appropriate file path for your robots.txt file!
Verifying the Rule
This rule adds a public response header to the CDN Response. Verify this EdgeRule by checking the headers from CDN assets using the browser inspect tools or cURL requests.
For browser tools, right-click anywhere on your website and select inspect or inspect element and navigate to the network tab. If you do not see any requests, refresh the page with this tab open to view them as they come in.
The initial request for your robots.txt should result with a 301 Permanent redirection, and lead to the new URL specified by the rule.
When performing a cURL test, the option -IL is preferred, to follow any redirection links and to see the headers appropriately. Below is an example of a curl, with the 301 response and location of the new robots.txt file.
If any questions arise or any assistance can be further provided, feel free to drop us a line at hi@stackpath.com, we'll be happy to help you out.