To prevent a page from being presented in Google Search Results you can include a 'noindex' header in HTTP Response headers. EdgeRules are a perfect way to implement this into a StackPath Site.
Important note about the x-robots-tag from Google: For thenoindexdirective to be effective, the page must not be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see thenoindexdirective, and the page can still appear in search results, for example, if other pages link to it.
If you have not already, please check this guide to get started with EdgeRules.
Creating the Rule
- Select the Site this EdgeRule will be impacting
- Navigate to your EdgeRules tab
- Select Add Delivery Rule
- Create the following EdgeRule:
IF URL Matches Regular Expression /\.(?.pdf|htm?l|jpe?g|gif)$/
THEN Add Response Header X-Robots-Tag noindex
This rule can be applied to any number of file types by prefixing them with a ? to indicate a wildcard location, and | to separate them. In this example, the regex pattern /\.(?.pdf|htm?l|jpe?g|gif)$/ will apply this rule to .pdf, .html, .jpg, .jpeg, and .gif file types.
Verifying the Rule
This EdgeRule adds a public response header to the CDN Response. Verify it by checking the headers from CDN assets using browser inspect tools or cURL requests.
For browser tools, right-click anywhere on your website and select inspect or inspect element and navigate to the network tab. If you do not see any requests, you can refresh the page with this tab open to view them as they come in.
Select an asset coming from the CDN from this tab to reveal the headers associated with the request. Our x-robots-tag header should appear here.
When performing a cURL test, the option -I is preferred, to show the headers appropriately. Below is an example of a curl, with the x-robots-tag header highlighted.
If any questions arise or any assistance can be further provided, feel free to drop us a line at hi@stackpath.com or open up a chat with our 24/7 live support, we'll be happy to help you out.