The bots we are having a problem with are AI bots which repeatedly try to hit every page on the site in order to scrape English-language content for AI training purposes.
When a human clicks through to a page on DU, some data is transferred from our servers to the person's browser, and this costs us a tiny amount of money with our web host. But when AI scrapers visit DU, they attempt to visit millions of pages in a very short space of time, grab the content as quickly as possible, and leave. They're not interacting with the pages, just trying to scrape the content.
Given that DU is a very old site, there are a lot of pages to scrape. This can cause our database and server infrastructure to overload as excessive requests back up in the system. During this period, it can be impossible to access DU pages as the request queue clears out. This is why we sometimes switch the site to "members only" mode, because it prevents bots accessing the site -- but that has to be done manually, and so if this problem happens in the middle of the night, there's not much that can be done.
The constant running battle to stop these bots is now plaguing pretty much everyone trying to run websites that are independent of social media platforms. Their activity significantly increases our data transfer costs, and our server costs have essentially doubled over the past year. Extra downtime and extra costs are the price we are unfortunately all paying for these AI pirates to steal our content.
= new reply since forum marked as read
