scraperbot protection - Patchwork and Bunsen behind Anubis

scraperbot protection - Patchwork and Bunsen behind Anubis

[Mark Wielaard]

Hi hackers,

TLDR; When using https://patchwork.sourceware.org or Bunsen
https://builder.sourceware.org/testruns/ you might now have to enable
javascript. This should not impact any scripts, just browsers (or bots
pretending to be browsers). If it does cause trouble, please let us
know. If this works out we might also "protect" bugzilla, gitweb,
cgit, and the wikis this way.

We don't like to hav to do this, but as some of you might have noticed
Sourceware has been fighting the new AI scraperbots since start of the
year. We are not alone in this.

https://lwn.net/Articles/1008897/
https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/

We have tried to isolate services more and block various ip-blocks
that were abusing the servers. But that has helped only so much.
Unfortunately the scraper bots are using lots of ip addresses
(probably by installing "free" VPN services that use normal user
connections as exit point) and pretending to be common
browsers/agents.  We seem to have to make access to some services
depend on solving a javascript challenge.

So we have installed Anubis https://anubis.techaro.lol/ in front of
patchwork and bunsen. This means that if you are using a browser that
identifies as Mozilla or Opera in their User-Agent you will get a
brief page showing the happy anime girl that requires javascript to
solve a challenge and get a cookie to get through. Scripts and search
engines should get through without. Also removing Mozilla and/or Opera
from your User-Agent will get you through without javascript.

We want to thanks Xe Iaso who has helped us set this up and worked
with use over the Easter weekend solving some of our problems/typos.
Please check out if you want to be one of their patrons as thank you.
https://xeiaso.net/notes/2025/anubis-works/
https://xeiaso.net/patrons/

Cheers,

Mark

Re: scraperbot protection - Patchwork and Bunsen behind Anubis

[Guinevere Larsen via Gdb]

On 4/21/25 12:59 PM, Mark Wielaard wrote:
> Hi hackers,
>
> TLDR; When using https://patchwork.sourceware.org or Bunsen
> https://builder.sourceware.org/testruns/ you might now have to enable
> javascript. This should not impact any scripts, just browsers (or bots
> pretending to be browsers). If it does cause trouble, please let us
> know. If this works out we might also "protect" bugzilla, gitweb,
> cgit, and the wikis this way.
>
> We don't like to hav to do this, but as some of you might have noticed
> Sourceware has been fighting the new AI scraperbots since start of the
> year. We are not alone in this.
>
> https://lwn.net/Articles/1008897/
> https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/
>
> We have tried to isolate services more and block various ip-blocks
> that were abusing the servers. But that has helped only so much.
> Unfortunately the scraper bots are using lots of ip addresses
> (probably by installing "free" VPN services that use normal user
> connections as exit point) and pretending to be common
> browsers/agents.  We seem to have to make access to some services
> depend on solving a javascript challenge.

Jan Wildeboer, on the fediverse, has a pretty interesting lead on how AI 
scrapers might be doing this: 
https://social.wildeboer.net/@jwildeboer/114360486804175788 (this is the 
last post in the thread because it was hard to actually follow the 
thread given the number of replies, please go all the way up and read 
all 8 posts).

Essentially, there's a library developer that pays developers to just 
"include this library and a few more lines in your TOS". This library 
then allows the app to sell the end-user's bandwidth to clients of the 
library developer, allowing them to make requests. This is how big 
companies are managing to have so many IP addresses, so many of those 
being residential IP addresses, and it also means that by blocking those 
IP addresses we will be - necessarily - blocking real user traffic to 
our platforms.

I'm happy to see that the sourceware is moving to a more comprehensive 
solution, and if this is successful, I'd suggest that we also try to do 
that to the forgejo instance, and remove the IPs blocked because of this 
scraping.

>
> So we have installed Anubis https://anubis.techaro.lol/ in front of
> patchwork and bunsen. This means that if you are using a browser that
> identifies as Mozilla or Opera in their User-Agent you will get a
> brief page showing the happy anime girl that requires javascript to
> solve a challenge and get a cookie to get through. Scripts and search
> engines should get through without. Also removing Mozilla and/or Opera
> from your User-Agent will get you through without javascript.
>
> We want to thanks Xe Iaso who has helped us set this up and worked
> with use over the Easter weekend solving some of our problems/typos.
> Please check out if you want to be one of their patrons as thank you.
> https://xeiaso.net/notes/2025/anubis-works/
> https://xeiaso.net/patrons/
>
> Cheers,
>
> Mark
>

-- 
Cheers,
Guinevere Larsen
She/Her/Hers

Re: scraperbot protection - Patchwork and Bunsen behind Anubis

[Jonathan Wakely via Gdb]

On Tue, 22 Apr 2025 at 13:36, Guinevere Larsen via Gcc <gcc@gcc.gnu.org> wrote:
>
> On 4/21/25 12:59 PM, Mark Wielaard wrote:
> > Hi hackers,
> >
> > TLDR; When using https://patchwork.sourceware.org or Bunsen
> > https://builder.sourceware.org/testruns/ you might now have to enable
> > javascript. This should not impact any scripts, just browsers (or bots
> > pretending to be browsers). If it does cause trouble, please let us
> > know. If this works out we might also "protect" bugzilla, gitweb,
> > cgit, and the wikis this way.
> >
> > We don't like to hav to do this, but as some of you might have noticed
> > Sourceware has been fighting the new AI scraperbots since start of the
> > year. We are not alone in this.
> >
> > https://lwn.net/Articles/1008897/
> > https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/
> >
> > We have tried to isolate services more and block various ip-blocks
> > that were abusing the servers. But that has helped only so much.
> > Unfortunately the scraper bots are using lots of ip addresses
> > (probably by installing "free" VPN services that use normal user
> > connections as exit point) and pretending to be common
> > browsers/agents.  We seem to have to make access to some services
> > depend on solving a javascript challenge.
>
> Jan Wildeboer, on the fediverse, has a pretty interesting lead on how AI
> scrapers might be doing this:
> https://social.wildeboer.net/@jwildeboer/114360486804175788 (this is the
> last post in the thread because it was hard to actually follow the
> thread given the number of replies, please go all the way up and read
> all 8 posts).
>
> Essentially, there's a library developer that pays developers to just
> "include this library and a few more lines in your TOS". This library
> then allows the app to sell the end-user's bandwidth to clients of the
> library developer, allowing them to make requests. This is how big
> companies are managing to have so many IP addresses, so many of those
> being residential IP addresses, and it also means that by blocking those
> IP addresses we will be - necessarily - blocking real user traffic to
> our platforms.

It seems to me that blocking real users *who are running these shady
apps* is perfectly reasonable.

They might not realise it, but those users are part of the problem. If
we block them, maybe they'll be incentivised to stop using the shady
apps. And if users stop using those apps, maybe those app developers
will stop bundling the libraries that piggyback on users' bandwidth.

>
> I'm happy to see that the sourceware is moving to a more comprehensive
> solution, and if this is successful, I'd suggest that we also try to do
> that to the forgejo instance, and remove the IPs blocked because of this
> scraping.

For now, maybe. This thread already explained how to get around Anubis
by changing the UserAgent string - how long will it be until these
peer-to-business network libraries figure that out?

Re: scraperbot protection - Patchwork and Bunsen behind Anubis

[Guinevere Larsen via Gdb]

On 4/22/25 10:06 AM, Jonathan Wakely wrote:
> On Tue, 22 Apr 2025 at 13:36, Guinevere Larsen via Gcc <gcc@gcc.gnu.org> wrote:
>> On 4/21/25 12:59 PM, Mark Wielaard wrote:
>>> Hi hackers,
>>>
>>> TLDR; When using https://patchwork.sourceware.org or Bunsen
>>> https://builder.sourceware.org/testruns/ you might now have to enable
>>> javascript. This should not impact any scripts, just browsers (or bots
>>> pretending to be browsers). If it does cause trouble, please let us
>>> know. If this works out we might also "protect" bugzilla, gitweb,
>>> cgit, and the wikis this way.
>>>
>>> We don't like to hav to do this, but as some of you might have noticed
>>> Sourceware has been fighting the new AI scraperbots since start of the
>>> year. We are not alone in this.
>>>
>>> https://lwn.net/Articles/1008897/
>>> https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/
>>>
>>> We have tried to isolate services more and block various ip-blocks
>>> that were abusing the servers. But that has helped only so much.
>>> Unfortunately the scraper bots are using lots of ip addresses
>>> (probably by installing "free" VPN services that use normal user
>>> connections as exit point) and pretending to be common
>>> browsers/agents.  We seem to have to make access to some services
>>> depend on solving a javascript challenge.
>> Jan Wildeboer, on the fediverse, has a pretty interesting lead on how AI
>> scrapers might be doing this:
>> https://social.wildeboer.net/@jwildeboer/114360486804175788 (this is the
>> last post in the thread because it was hard to actually follow the
>> thread given the number of replies, please go all the way up and read
>> all 8 posts).
>>
>> Essentially, there's a library developer that pays developers to just
>> "include this library and a few more lines in your TOS". This library
>> then allows the app to sell the end-user's bandwidth to clients of the
>> library developer, allowing them to make requests. This is how big
>> companies are managing to have so many IP addresses, so many of those
>> being residential IP addresses, and it also means that by blocking those
>> IP addresses we will be - necessarily - blocking real user traffic to
>> our platforms.
> It seems to me that blocking real users *who are running these shady
> apps* is perfectly reasonable.
>
> They might not realise it, but those users are part of the problem. If
> we block them, maybe they'll be incentivised to stop using the shady
> apps. And if users stop using those apps, maybe those app developers
> will stop bundling the libraries that piggyback on users' bandwidth.

If an IP mapped perfectly to one user, maybe. But I can't control what 
other users of the same ISP in the same area as me are doing, and we're 
sharing an IP. And worse, if I still lived with my family, no way would 
I be able to veto what my parents are using their phone for, so because 
they have a shady app I wouldn't be able to access systems? that doesn't 
seem fair at all

Not to mention the fact that "read and understand the entirety of the 
TOS of every single app" assumes a pretty decent amount of free time for 
users that they may not have, and we wouldn't want to make open source 
even more hostile for people who are overwhelmed or overworked already. 
Of course people should, but having that as a requirement excludes 
people like... well, myself to be quite honest.

>
>> I'm happy to see that the sourceware is moving to a more comprehensive
>> solution, and if this is successful, I'd suggest that we also try to do
>> that to the forgejo instance, and remove the IPs blocked because of this
>> scraping.
> For now, maybe. This thread already explained how to get around Anubis
> by changing the UserAgent string - how long will it be until these
> peer-to-business network libraries figure that out?
>
hopefully longer than the bubble lasts

-- 
Cheers,
Guinevere Larsen
She/Her/Hers

Re: scraperbot protection - Patchwork and Bunsen behind Anubis

[Jonathan Wakely via Gdb]

On Tue, 22 Apr 2025, 14:17 Guinevere Larsen, <guinevere@redhat.com> wrote:
>
> On 4/22/25 10:06 AM, Jonathan Wakely wrote:
> > On Tue, 22 Apr 2025 at 13:36, Guinevere Larsen via Gcc <gcc@gcc.gnu.org> wrote:
> >> On 4/21/25 12:59 PM, Mark Wielaard wrote:
> >>> Hi hackers,
> >>>
> >>> TLDR; When using https://patchwork.sourceware.org or Bunsen
> >>> https://builder.sourceware.org/testruns/ you might now have to enable
> >>> javascript. This should not impact any scripts, just browsers (or bots
> >>> pretending to be browsers). If it does cause trouble, please let us
> >>> know. If this works out we might also "protect" bugzilla, gitweb,
> >>> cgit, and the wikis this way.
> >>>
> >>> We don't like to hav to do this, but as some of you might have noticed
> >>> Sourceware has been fighting the new AI scraperbots since start of the
> >>> year. We are not alone in this.
> >>>
> >>> https://lwn.net/Articles/1008897/
> >>> https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/
> >>>
> >>> We have tried to isolate services more and block various ip-blocks
> >>> that were abusing the servers. But that has helped only so much.
> >>> Unfortunately the scraper bots are using lots of ip addresses
> >>> (probably by installing "free" VPN services that use normal user
> >>> connections as exit point) and pretending to be common
> >>> browsers/agents.  We seem to have to make access to some services
> >>> depend on solving a javascript challenge.
> >> Jan Wildeboer, on the fediverse, has a pretty interesting lead on how AI
> >> scrapers might be doing this:
> >> https://social.wildeboer.net/@jwildeboer/114360486804175788 (this is the
> >> last post in the thread because it was hard to actually follow the
> >> thread given the number of replies, please go all the way up and read
> >> all 8 posts).
> >>
> >> Essentially, there's a library developer that pays developers to just
> >> "include this library and a few more lines in your TOS". This library
> >> then allows the app to sell the end-user's bandwidth to clients of the
> >> library developer, allowing them to make requests. This is how big
> >> companies are managing to have so many IP addresses, so many of those
> >> being residential IP addresses, and it also means that by blocking those
> >> IP addresses we will be - necessarily - blocking real user traffic to
> >> our platforms.
> > It seems to me that blocking real users *who are running these shady
> > apps* is perfectly reasonable.
> >
> > They might not realise it, but those users are part of the problem. If
> > we block them, maybe they'll be incentivised to stop using the shady
> > apps. And if users stop using those apps, maybe those app developers
> > will stop bundling the libraries that piggyback on users' bandwidth.
>
> If an IP mapped perfectly to one user, maybe. But I can't control what
> other users of the same ISP in the same area as me are doing, and we're
> sharing an IP. And worse, if I still lived with my family, no way would
> I be able to veto what my parents are using their phone for, so because
> they have a shady app I wouldn't be able to access systems?

Yeah, maybe.

> that doesn't seem fair at all

What's fair about those users of shady apps being able to prevent all
of us from accessing the systems we need to use?

The apps and ISPs and cloud providers that allow this traffic should
be excluded from the public net, and let market forces push people to
choose better services. Why reward the app developers who are
profiting this way, when we could discourage it?

If users are currently oblivious to the problems caused by their bad
choice of apps, they won't stop using the malware that creates a
botnet (just a semi-legal form because nobody had to get hacked to
install the malware). App stores and ISPs should certainly be
protecting users from being exploited this way, but that's not
happening.


> Not to mention the fact that "read and understand the entirety of the
> TOS of every single app" assumes a pretty decent amount of free time for
> users that they may not have, and we wouldn't want to make open source
> even more hostile for people who are overwhelmed or overworked already.

I don't think we should have to tolerate bad actors (in this case,
that means the malware apps, but by extension the users of those
malware apps).

> Of course people should, but having that as a requirement excludes
> people like... well, myself to be quite honest.
>
> >
> >> I'm happy to see that the sourceware is moving to a more comprehensive
> >> solution, and if this is successful, I'd suggest that we also try to do
> >> that to the forgejo instance, and remove the IPs blocked because of this
> >> scraping.
> > For now, maybe. This thread already explained how to get around Anubis
> > by changing the UserAgent string - how long will it be until these
> > peer-to-business network libraries figure that out?
> >
> hopefully longer than the bubble lasts


I expect them to figure it out in a matter of weeks, and then updated
apps will start using newer versions of the libraries.

Re: scraperbot protection - Patchwork and Bunsen behind Anubis

[Aurelien Jarno via Gdb]

On 2025-04-22 14:06, Jonathan Wakely wrote:
> On Tue, 22 Apr 2025 at 13:36, Guinevere Larsen via Gcc <gcc@gcc.gnu.org> wrote:
> >
> > On 4/21/25 12:59 PM, Mark Wielaard wrote:
> > > Hi hackers,
> > >
> > > TLDR; When using https://patchwork.sourceware.org or Bunsen
> > > https://builder.sourceware.org/testruns/ you might now have to enable
> > > javascript. This should not impact any scripts, just browsers (or bots
> > > pretending to be browsers). If it does cause trouble, please let us
> > > know. If this works out we might also "protect" bugzilla, gitweb,
> > > cgit, and the wikis this way.
> > >
> > > We don't like to hav to do this, but as some of you might have noticed
> > > Sourceware has been fighting the new AI scraperbots since start of the
> > > year. We are not alone in this.
> > >
> > > https://lwn.net/Articles/1008897/
> > > https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/
> > >
> > > We have tried to isolate services more and block various ip-blocks
> > > that were abusing the servers. But that has helped only so much.
> > > Unfortunately the scraper bots are using lots of ip addresses
> > > (probably by installing "free" VPN services that use normal user
> > > connections as exit point) and pretending to be common
> > > browsers/agents.  We seem to have to make access to some services
> > > depend on solving a javascript challenge.
> >
> > Jan Wildeboer, on the fediverse, has a pretty interesting lead on how AI
> > scrapers might be doing this:
> > https://social.wildeboer.net/@jwildeboer/114360486804175788 (this is the
> > last post in the thread because it was hard to actually follow the
> > thread given the number of replies, please go all the way up and read
> > all 8 posts).
> >
> > Essentially, there's a library developer that pays developers to just
> > "include this library and a few more lines in your TOS". This library
> > then allows the app to sell the end-user's bandwidth to clients of the
> > library developer, allowing them to make requests. This is how big
> > companies are managing to have so many IP addresses, so many of those
> > being residential IP addresses, and it also means that by blocking those
> > IP addresses we will be - necessarily - blocking real user traffic to
> > our platforms.
> 
> It seems to me that blocking real users *who are running these shady
> apps* is perfectly reasonable.

How do you detect them? From my experience at other hosting places, 
those IPs, just make a few request per hours or per day, with a standard
User Agent. As such it's difficult to differentiate them from normal 
users.

The problem is that you suddenly have hundreds of thousands of requests 
per hours from just a slightly lower number of IPs. And in the middle 
you also have legit users using IPs from the same net block.

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                     http://aurel32.net

Re: scraperbot protection - Patchwork and Bunsen behind Anubis

[Chris Packham via Gdb]

Hi Mark,

On Tue, 22 Apr 2025, 4:00 am Mark Wielaard, <mark@klomp.org> wrote:

> Hi hackers,
>
> TLDR; When using https://patchwork.sourceware.org or Bunsen
> https://builder.sourceware.org/testruns/ you might now have to enable
> javascript. This should not impact any scripts, just browsers (or bots
> pretending to be browsers). If it does cause trouble, please let us
> know. If this works out we might also "protect" bugzilla, gitweb,
> cgit, and the wikis this way.
>
> We don't like to hav to do this, but as some of you might have noticed
> Sourceware has been fighting the new AI scraperbots since start of the
> year. We are not alone in this.
>
> https://lwn.net/Articles/1008897/
>
> https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/
>
> We have tried to isolate services more and block various ip-blocks
> that were abusing the servers. But that has helped only so much.
> Unfortunately the scraper bots are using lots of ip addresses
> (probably by installing "free" VPN services that use normal user
> connections as exit point) and pretending to be common
> browsers/agents.  We seem to have to make access to some services
> depend on solving a javascript challenge.
>
> So we have installed Anubis https://anubis.techaro.lol/ in front of
> patchwork and bunsen. This means that if you are using a browser that
> identifies as Mozilla or Opera in their User-Agent you will get a
> brief page showing the happy anime girl that requires javascript to
> solve a challenge and get a cookie to get through. Scripts and search
> engines should get through without. Also removing Mozilla and/or Opera
> from your User-Agent will get you through without javascript.
>
> We want to thanks Xe Iaso who has helped us set this up and worked
> with use over the Easter weekend solving some of our problems/typos.
> Please check out if you want to be one of their patrons as thank you.
> https://xeiaso.net/notes/2025/anubis-works/
> https://xeiaso.net/patrons/


Ah that might explain a few things. We've seen sporadic failures in the
crosstool-ng CI builds (run via a github action) where a download of the
newlib snapshot failed (but worked fine when I tried the download manually).

The good news is that this finally prompted me to look at why we were
downloading something that should have been cached. I've fixed that now so
whatever extra load our builds were contributing should stop soon.

We might still get caught up in the bot detection when a package hosted on
sourceware.org is updated. I'm not sure if there is anything we can do
about that. I totally understand why this is necessary (AI scraper bots
have taken the crosstool-ng website down twice).

Re: scraperbot protection - Patchwork and Bunsen behind Anubis

[Christophe Lyon via Gdb]

Hi!

Thanks for all the hard work maintaining all this fundamental infrastructure.

On Mon, 21 Apr 2025 at 18:00, Mark Wielaard <mark@klomp.org> wrote:
>
> Hi hackers,
>
> TLDR; When using https://patchwork.sourceware.org or Bunsen
> https://builder.sourceware.org/testruns/ you might now have to enable
> javascript. This should not impact any scripts, just browsers (or bots
> pretending to be browsers). If it does cause trouble, please let us
> know. If this works out we might also "protect" bugzilla, gitweb,
> cgit, and the wikis this way.
>
> We don't like to hav to do this, but as some of you might have noticed
> Sourceware has been fighting the new AI scraperbots since start of the
> year. We are not alone in this.
>
> https://lwn.net/Articles/1008897/
> https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/
>
> We have tried to isolate services more and block various ip-blocks
> that were abusing the servers. But that has helped only so much.

In terms of isolation, since I have no idea how / which services are
currently isolated, I may ask obvious questions..

We were wondering if it would be possible / suitable to have https
requests served by one container,
and ssh ones by another? Maybe that's already the case though...

Speaking with CI in mind, the Linaro CI is currently severely impacted by these
scraperbots too:
- directly because our git servers are also overloaded, so our build
process often fail to checkout build scripts & infra scripts
- indirectly because when the above succeed we may fail to connect to sourceware

so maybe it would help if we switched to ssh access for our CI user
when cloning GCC / binuitls / etc sources?

If only the containers serving https requests are impacted, ssh access
could still work well?
(that would mean creating a Linaro-CI user on sourceware, I don't know
what the policy is?)

Thanks,

Christophe

> Unfortunately the scraper bots are using lots of ip addresses
> (probably by installing "free" VPN services that use normal user
> connections as exit point) and pretending to be common
> browsers/agents.  We seem to have to make access to some services
> depend on solving a javascript challenge.
>
> So we have installed Anubis https://anubis.techaro.lol/ in front of
> patchwork and bunsen. This means that if you are using a browser that
> identifies as Mozilla or Opera in their User-Agent you will get a
> brief page showing the happy anime girl that requires javascript to
> solve a challenge and get a cookie to get through. Scripts and search
> engines should get through without. Also removing Mozilla and/or Opera
> from your User-Agent will get you through without javascript.
>
> We want to thanks Xe Iaso who has helped us set this up and worked
> with use over the Easter weekend solving some of our problems/typos.
> Please check out if you want to be one of their patrons as thank you.
> https://xeiaso.net/notes/2025/anubis-works/
> https://xeiaso.net/patrons/
>
> Cheers,
>
> Mark

Re: scraperbot protection - Patchwork and Bunsen behind Anubis

[Frank Ch. Eigler via Gdb]

Hi -

> We were wondering if it would be possible / suitable to have https
> requests served by one container,
> and ssh ones by another? Maybe that's already the case though...

SSH stuff is already (un)contained.

> [...]
> so maybe it would help if we switched to ssh access for our CI user
> when cloning GCC / binuitls / etc sources?

What you'd need is a ssh identity with logon access for the bots.
That's not something we normally do (sourceware accounts are personal
and usually have read-write access to -some- git repos.)  But we could
support read-only ssh accounts for bots.  (read-only for safety, i.e.,
not member of any particular git unix-group/project.)

- FChE