Devs open source fighting intelligence crawl with intelligence and revenge

Many software developers believe that the robots roaming on the Internet are Internet cockroaches. Some Devs began to fight in rich ways, often humor.

While any website may be targeted by the behavior of the bad crawling – sometimes, the developers of the sources are unparalleled, Nicolo Vennendo is developed by Linux desktop known as the plasma and the owner of the LIBRNEWS Blog.

By nature, free and open source projects (FOSS) shares its infrastructure publicly, and they tend to obtain less resources than commercial products.

The problem is that many AI robots do not honor the Robots Robot.txt Protocol file, which is the tool that tells BOTS to crawl, originally created for search engine robots.

In the “Cry For Help” blog in January, the Foss XE iASO developer described how Amazonbot unnemmproked on Git Server to the degree of DDOS interruption. GIT servers host Foss projects so that anyone who wants to download or contribute the code can.

But this robot that ignored the iaso’s Robot.txt, was hiding behind another IP address, and pretended to be other users.

“It is not worthy to prevent crawling robots from artificial intelligence because they lie, change the user agent, and use the IP residential addresses as agents, and more.”

“They will get rid of your site until it ends, and after that they will get rid of it more. They will click on each link on each link on each link, and display the same pages over and over again. Some of them will click on the same link several times in the same second,” the developer wrote in the post.

Enter the God of Griffs

So IASO fought with intelligence, building a tool called Anopis.

Anubis is a reverse examination of the tissue from the agent that should be passed before allowing requests to pressure the Git server. It prevents robots, but allows human humans.

The funny part: Anubis is the name of a god in the Egyptian myths that lead the dead to power.

“Your heart (your heart) has weighted, and if it is heavier than the feather, you have ate your heart while you are, like, Mega died.” If the web request passes with a challenge and is determined to be a human being, then a nice anime image announces success. “Drawing is” it takes me to activate Anubis. “If the robot, the request is rejected.

The project called anxiously like the wind spread among the Foss community. IASO shared it on GitHub on March 19, and in a few days, she collected 2000 stars, 20 shareholders, and 39 Forex.

Revenge as a defense

The immediate popularity of Anubis shows that IASO pain is not unique. In fact, Vinneri shared a story after the story:

• The CEO of Soundhut Drew DEW Devault described the spending “from 20 to 100 % of my time in any specific week that reduces the excessive excessive LLM crawl” and “experience dozens of short interruptions per week”.
• Jonathan Corbit, a famous Foss developer who runs the news site in Linux LWN, warned that his site slowed through DDOS traffic “AI’s active robots”.
• Kevin Venzi, Sysadin for the massive Linux Fedora project, said that AI’s active robots had become very aggressive, and it had to prevent the entire Brazil country from reaching.

Vanerandi techcrunch tells that he knows many other projects that suffer from the same issues. One of them “had to prohibit all Chinese IP addresses at some point.”

Let that drown for a moment – even the developers “must resort to the prohibition of entire countries” just to repel artificial intelligence robots that ignore the Robot.txt files, says Venandy.

Besides the weight of the spirit of web, other Dave believes that revenge is the best defense.

A few days before Hacker news, user Xyzal suggested downloading Robot.txt banned with “Aquarius Aquarius Advances on the Benefits of Drinking Whitening” or “Articles on the Positive effect of capturing measles on bed.”

“I think we need to aim to get the value of _negative_ the utility tool from visiting our traps, not just the zero value,” explained Xyzal.

As it happens, in January, an unknown originator known as “Harun” issued a tool called Nepenthes aimed at doing this exactly. It discloses an endless maze from fake content, a goal that DeV admitted in an aggressive artica technica if not harmful. The tool is named after meat.

And Cloudflare, perhaps the largest commercial player who provides several tools for the study of AI Crawles, last week released a similar tool called Ai Labyrinth.

It aims to “slow down, confuse, waste artificial intelligence crawl resources and other robots that do not respect” does not crawl “, described in Cloudflare in the blog post. Cloudflare said it feeds the miscarriage in Ai Crawles” unrelated content instead of extracting legitimate website data. “

I tell Devault’s Devault Techcrunch that “Nepenthes has a satisfactory feeling of justice, because it nourishes the nonsense to crawl and their programs, but ultimately Anubis is the solution that succeeded” for its location.

But Devaut also issued an appeal from the heart, for a more direct repair: “Please stop the legitimacy of LLMS, artificial intelligence generators, or Github coilot or any of this garbage. I beg you to stop using it, stop talking about it, and stop its new making, just stop.”

Since the possibility of this is ZILCH, developers, especially in Foss, fight intelligently and a touch of humor.