Introduction: Monitors
What is a Monitor?
A Monitor is an automated process that will observe one or multiple website(s) via our integrated Crawler. The crawler will be sent to the pages found in a website, the crawler will analyze the content of the page and report it back to our system where you can see the results.
What can a monitor be used for?
The monitoring feature can be used for various things, here are some examples:
-
Compliance Monitoring
- Which cookies are set on my website?
- Which vendors are setting cookies?
- Which non-essential cookies are set without consent?
- Which vendors are transferring data outside of the EU?
- ...
-
SEO Monitoring
- How many images are loaded on my website?
- How many CSS, JS and other files are loaded on my website?
- How high is the filesize of my website?
- How quick or slow does my website load?
- ...
-
Monetization Monitoring
- Which advertising vendors are present on my website?
- Are all advertisers listed in my CMP?
- Are all advertisers listed in my ads.txt file?
- Is PrebidJS active on all my pages?
- ...
-
Analytics Monitoring
- Is my tagmanager present on all my pages?
- Is my analytics vendor present on all my pages?
- Is Google Consent Mode or Microsoft Consent Mode activated on all my pages?
- ...
Setup a Monitor for a CMP
The system will automatically setup one monitor for each CMP that is present in your account. The settings for the monitor can be found under Menu > CMPs > Crawler settings.
Setup a Monitor for a Website
In order to create a new monitor, please navigate to Menu > Monitors and click on Create Monitor.
Start a manual crawl
As an alternative to using a monitor, you can also manually start a single crawl. This way no monitor or CMP needs to be created.
How will the monitor find pages to be crawled?
For CMP-Monitors, the system will collect the URLs on which the CMP code is installed. The crawler will randomly take URLs from the list and crawl these URLs. By default the crawler will not follow links within the page.
Note: For each CMP you can set the crawling mode under Menu > CMPs > Crawler settings. Here you can also enable to follow links. If this is enabled, the crawler will first follow all links found before resuming to crawl random URLs from the CMP code installation.
For a Monitor that is not a CMP-Monitor, the system will require one or more URLs as a starting point. The system will use these URLs as level 0 and follow all links from this URLs that are withing the same domain. By default the system will crawl a maximum of 500 pages per monitor and follow a maximum of 5 levels deep.
Monitor settings
You can access the monitor settings via the "..." menu under Menu > Monitors:
For each monitor you will find the following settings:
| Setting | Description |
| Name | Name of your Monitor |
| Active | Status for the crawler. If paused, the crawler will no longer add new results. |
| URLs | List of URLs as a start for the crawler. Multiple URLs can be given line by line. |
| Max crawl depth | Maximum level of clicks a page can be away from the start URL |
| Max crawled pages | Maximum amount of pages that should be crawled per month |
| Max crawls per day | Maximum amount of crawls that should be performed per day (0=unlimited) |
| Crawl regulation ratio | Ratio of crawls to be done from within the EU or the US. |
| Crawl acceptance ratio | Ratio of crawls to be done with acceptance or rejection |
| Excluded / Included domains or paths | List of domains or paths that shall be excluded or included from the crawling. |
| Crawler HTTP Auth | Authentication parameters. See also here. |
| Cookie Authentication | Cookies to be give to the crawler. |
| Tag / Notes | Internal notes for you |
| List of tagmanagers / analytics vendors | List of vendors that the system considers a tagmanager or analytics vendor. This is used for the Analytics Reporting. If non-empty, the listed vendors will be used instead of the default vendors set in the system. |
| Google Account IDs | List of Google account IDs (e.g. "AW-....", "UA-....") to be used for id verification. If non-empty, the list of IDs will be used, otherwise the list set in the account settings will be used. |
