To prevent a page from being presented in Google Search Results you can include a 'noindex' header in HTTP Response headers. EdgeRules are a perfect way to implement this into a StackPath CDN site.
Important note about the x-robots-tag from Google: For thenoindexdirective to be effective, the page must not be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see thenoindexdirective, and the page can still appear in search results, for example, if other pages link to it.
If you have not already, please check this guide to get started with EdgeRules.
Creating the Rule
- Select the CDN Site this EdgeRule will be impacting
- Navigate to your EdgeRules tab
- Select Add Custom Rule
- In the IF section, select the URL Option and the RegEx Matches modifier. It is possible to apply this rule to any number of file types by prefixing them with a ? to indicate a wildcard location, and | to separate them. For example, the rule
/\.(?.pdf|htm?l|jpe?g|gif)$/will apply to pdf, html, jpg, jpeg, and gif files. - In the THEN section, apply a new custom response header that can tag these files with
X-Robots-Tag: noindex.
This rule can be applied to any number of file types by prefixing them with a ? to indicate a wildcard location, and | to separate them. For example, the rule /\.(?.pdf|htm?l|jpe?g|gif)$/ will apply to pdf, html, jpg, jpeg, and gif files.
Verifying the Rule
This EdgeRule adds a public response header to the CDN Response. Verify it by checking the headers from CDN assets using browser inspect tools or cURL requests.
For browser tools, right-click anywhere on your website and select inspect or inspect element and navigate to the network tab. If you do not see any requests, you can refresh the page with this tab open to view them as they come in.
Select an asset coming from the CDN from this tab to reveal the headers associated with the request. Our x-robots-tag header should appear here.
When performing a cURL test, the option -I is preferred, to show the headers appropriately. Below is an example of a curl, with the x-robots-tag header highlighted.
If you have any questions or would like any assistance setting up this rule, feel free to open up a chat with our 24/7 live support, we'll be happy to help you out.