Info
Content

Crawling in login areas, checkout processes and other restricted pages

Per default the crawler will only visit URLs on which the CMP-Code is present. This can lead to problems when it is not possible for the crawler to reach the (same) content as a normal user would reach (e.g. because the user logged into a login area, added products to his shopping cart or performed other tasks in order to get a different output from your website).

Using custom cookies

In the CMP settings you can set cookie authentication (CMPs > Edit > Show Crawler settings > Cookie-Authentication). This is a mechanism in order to tell the crawler to send cookies to the server (although these cookies might never exist elsewhere). Once the crawler visits the website, the cookies will be present and the website can read these. The website can then react in a different way than how it would if the cookies were not present (e.g. by allowing access to the crawler to a restricted area or showing certain content to the crawler that otherwise requieres other steps to be performed prior to the visit).

In order to setup cookie authentication, please insert one item per line where an item consists of domain:cookiename:cookievalue. Example:

mywebsite.com:mycookie:123
myotherwebsite.com:othercookie:let_me_in
a-third-website.com:authentication:crawler
a-third-website.com:token:bfe926da3fc1
Back to top