In recent months, we have seen a significant increase in illegitimate automated accesses to our platform. In order to protect our users and improve the resilience of our services, we recently deployed Anubis, a solution aligned with our values of openness and transparency.
A Recent Phenomenon, Intensified by AI
With the development of artificial intelligence (AI) tools, a growing number of robots (bots) are browsing the web, often simulating human behavior. These robots, called “crawlers,” are used to collect large amounts of data, feed large language models (LLMs), and improve the responses of AI assistants.
The activity of these bots can generate high volumes of traffic, sometimes affecting site performance and rendering viewing statistics less representative of actual human usage. Open platforms such as Érudit, which disseminate high-quality scientific content, are particularly targeted because they provide a valuable source of high-quality texts for model training.
It should be noted that well-identified robots (e.g., those used by traditional search engines to index content) generally respect the access restrictions put in place by platforms. They therefore do not pose a risk to the stability of services.
This phenomenon now affects all digital research and knowledge dissemination infrastructures. Several digital libraries and institutional repositories, such as Wikimedia, OAPEN, DOAB, and the collections of the universities of North Carolina, Duke, and Tufts, report frequent interactions with aggressive bots. A survey by the Confederation of Open Access Repositories (April 2025) indicates that 90% of respondents encounter these bots at least once a week, sometimes more, which can cause slowdowns or temporary service interruptions.
Preserving Open Access Without Intrusive Monitoring
A massive influx of these automated requests has several undesirable effects:
- Performance degradation: network and server overload, slowdowns, and even outages.
- Biased statistics: bots can account for the majority of traffic, masking actual usage and distorting the analysis of reading behavior.
- Risks to equitable access: some human users may sometimes find themselves displaced by bots.
Guaranteeing access to scholarly publications is the very cornerstone of Érudit’s mission. That is why, in the summer of 2025, we undertook an in-depth analysis of technical solutions capable of maintaining a balance between performance, confidentiality, and openness for the use of our platform. After several tests, we selected Anubis, a solution designed specifically to address the challenges associated with AI bots. The goal is simple: to block illegitimate access while maintaining a smooth experience for our users.
Concrete Benefits: More Reliable Statistics and a More Stable Platform
Before Anubis was put in place, our IT team had been manually filtering the increasing illegitimate accesses since the fall of 2024, leading to an apparent drop in usage statistics. This drop should be interpreted positively: it corresponds to a cleaning up of traffic and provides a more accurate picture of actual usage. Since the filter was implemented in September 2025, we have seen a significant reduction in undesirable automated access and increased system stability.
The deployment of Anubis is an important step in our strategy to protect knowledge dissemination infrastructures. However, this is not a one-off intervention:
- our IT team will continue to fine-tune the Anubis settings;
- our scientific development team will analyze changes in bot behavior;
- Erudit will continue to actively participate in the international community’s work on regulating AI bots and preserving open access.
We would like to thank Xe Iaso, creator of Anubis, as well as the members of our IT team (especially Béranger, David C., and David M.) for their decisive contribution to the deployment of this solution.
References
Aery, S. (2025). Anubis Pilot Project Report—June 2025. Duke University Libraries. https://hdl.handle.net/10161/32990
Almas, B. (2025, April 11th). Aggressive AI Harvesting of Digital Resources—Community Groups—LYRASIS Wiki. https://wiki.lyrasis.org/pages/viewpage.action?pageId=364743621
Liu, E., Luo, E., Shan, S., Voelker, G. M., Zhao, B. Y., & Savage, S. (2025). Somesite I Used To Crawl : Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. arXiv. https://doi.org/10.1145/3730567.3732913
Mueller, B., Danis, C., Lavagetto, G., & Wikimedia Foundation. (2025, April 1st). How crawlers impact the operations of the Wikimedia projects. Diff. https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/
Panitch, J. (2025, June 9th). Library IT vs. The AI bots. UNC University Libraries. https://library.unc.edu/news/library-it-vs-the-ai-bots/
Sarah. (2025, February 26th). In case you didn’t know this about me, I’m part of the team for Perseus Digital Library. Since last week, we’ve been experiencing continuing… [Post]. Bluesky. https://bsky.app/profile/lepidopterane.bsky.social/post/3lj3wtoa56s2w
Shearer, K., & Walk, P. (2025). The impact of AI bots and crawlers on open repositories : Results of a COAR survey, April 2025. Confederation of Open Access Repositories. https://coar-repositories.org/wp-content/uploads/2025/06/Report-of-the-COAR-Survey-on-AI-Bots-June-2025-1.pdf
Snijder, R. (2025a, April 11th). Computers gung ho? Responding to the OAPEN Library and DOAB’s response time [Billet]. OAPEN – Online Library of Open Access Books. https://doi.org/10.58079/13qrk
Snijder, R. (2025b, May 27th). Funnelling locusts – further reflections on the OAPEN Library and DOAB’s response time [Billet]. OAPEN – Online Library of Open Access Books. https://doi.org/10.58079/140ni