Q&A: What is the International IAB/ABC Spiders and Bots list?

What is the International IAB/ABC Spiders and Robots list?

The International IAB US/ABC UK Spiders & Robots List is a key resource for digital media owners to minimise non-human traffic being counted in their web analytics. Established and launched in 2006 by ABC and the IAB US, it is a list of known robotic user agents that is updated and shared with subscribers each month. Media owners can apply the list to their digital analytics, and this ensures that these known robotic user agents can be recognised and separated for reporting.

Why do robotic user agents need to be separated?

Robotic user agents explore the internet and index content so relevant sites can be easily found for search purposes or for ad placement. They are entirely legitimate and provide a useful and effective function. However, as a side effect of exploring the internet, they can significantly impact ad impression and site traffic counts if included in a company’s analytics. Traffic consisting of robotic user agents is non-human rather than fraudulent; the technology exists to be helpful to internet users and has not been ‘disguised’ as a human user. This means that they can be easily detected and filtered by implementing the Spiders & Robots list.

Why is it useful for digital media owners?

Implementation of the Spiders & Robots list is a fundamental step for companies who want to promote transparency and trust in digital ad trading by minimising non-human traffic counted in their web analytics. Implementation of the list is required in the U.K. via the JICWEBS Web Traffic Standards, and in the U.S. via the IAB’s Ad Impression Measurement Guidelines.

Why is it useful for digital media buyers?

Media agencies and advertisers can trade with greater confidence with media owners who demonstrate a commitment to transparent trading, as it provides reassurance that digital campaigns are reaching their intended audience.

How does the list fit into the work being done on Anti Ad-fraud?

It is important to filter out known non-human activity as a basic first measure to reduce any invalid traffic being reported in web analytics. Fraudulent activity from unknown sources is now being addressed by industry representatives through JICWEBS and with ABC in their ongoing work to reduce exposure to ad fraud.

—

To stop spiders affecting your data download the registration form below and send it to the IAB Europe Team: communication@iabeurope.eu.

Note: IAB Europe members get a special discount on the subscription rate.