From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id QhfbNRhkCGgsyAAAWB0awg (envelope-from ) for ; Tue, 22 Apr 2025 23:52:56 -0400 Received: by simark.ca (Postfix, from userid 112) id CD6681E0C3; Tue, 22 Apr 2025 23:52:56 -0400 (EDT) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-5.3 required=5.0 tests=ARC_SIGNED,ARC_VALID,BAYES_00, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham autolearn_force=no version=4.0.1 Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id AE3381E05C for ; Tue, 22 Apr 2025 23:52:55 -0400 (EDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 589773857C7F for ; Wed, 23 Apr 2025 03:52:55 +0000 (GMT) Received: from mail-yw1-x1131.google.com (mail-yw1-x1131.google.com [IPv6:2607:f8b0:4864:20::1131]) by sourceware.org (Postfix) with ESMTPS id 9E61A3858D38; Wed, 23 Apr 2025 03:52:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 9E61A3858D38 ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 9E61A3858D38 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1745380338; cv=none; b=pzGSDucT8yr1lAUqg4Z4uA2lvLEdLOCb1IwkG/smo4MvYWOB6feqixZJZWYDLaAV/Nss/oUZHQwTFDNOPeKAz1sJXT8xd6oGX2nx+cxoNqN/sfq2weMZkpJpGf4FOGOn587RUu8n1hA6sWTKwdC5pnlBqD405NbXWiW//kE0lCg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1745380338; c=relaxed/simple; bh=X22pAUJLJ095Q55X4ul1G3j1biq7vktnF1NGi4iEypI=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=WHFCTA4vnOl301VF0nvRolwBGp1uufBYu902WZm1NOu7j14YzoPa2ACVpc0dPznGIXOSSpYVPCxZmmbKFmlmuhrHRP7kvXZXkkYdAxGaV9uK2yl9PNmlZ4XYgF3QM7A/H1N8ntyIeOMJfg+VJuMAe4Pmtrhf4wO3Sp1p3LFLy7o= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9E61A3858D38 Received: by mail-yw1-x1131.google.com with SMTP id 00721157ae682-6ef60e500d7so49161977b3.0; Tue, 22 Apr 2025 20:52:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745380338; x=1745985138; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lzYuKtD6/ZeOvl1BnMUAIbc7Zs+9sHF0mwQOjSJAwLQ=; b=El1gqVBbIhT0VDECcCE8Gv+3CDHXEQSQ92uGvt46pmkn8q5CtuY3VU/oXALxtHH0W7 Tg2ft2LJa1PnozB7BQ16bcAFKEGxvja8e7k8xqP0n1amZhsnNnHMMS+CtMSZke5lPmly rImxWvhSIv76Prj6L9R2WX85pjkk9tov6pityLZ308CWz+3eJwAza//QHv/YAPuXn2+u z/guS9w+zP7KUs3ERzA52QNgkCLW33XDPzINsvxW+bp63HP73RxPq0UvEJA20gz+CXG0 kTMprDf/RwzeP3TP2BaUjBTaw9ZbeCwDZZGJnAYqqp/6o2Gp1Blzetnz2lukLkIqmos3 SShA== X-Forwarded-Encrypted: i=1; AJvYcCUvDzqPRcP5hkFOjQH1wGxGGv7YxlsDJqko/NijJdEva1h0Teik45H28yugRwMwYUjMl+zF@sourceware.org, AJvYcCVXJ4AOvkOvVw/6IBp23UmVtq8j0uYjQdwKLsqB4VnOUOw3IOSPQl7T6awNZB+XZ7TJjJJELl+Rq5u5ig==@sourceware.org, AJvYcCVw0wgYsLUp4MQMdJR5nGLsFgudBQ+8lmeFP44xxGrsXeg3ROYS3KgaC0r4Q3XvFfRJqUu6h5yg3+n/@sourceware.org, AJvYcCWIcBir6sytRj6Krl1Yw2hHKerWt9fKFY8M/5KGMtP0v3U0NVDCum9AtpKTN1urkDbk6xSnlKkB@sourceware.org, AJvYcCWZcwtmGjM5BCzJTGz2duhbTlbBYpebkyTUIxUeYduhdJZGoUe+tg6wrPBnhrVQNwXlj8GOMSCpNtDhxk4VVA==@sourceware.org, AJvYcCXJriHfMWGT2BKA9Jk6zZ8jNJZznqcvnEKndRy3InHn8bf8WHzVDVJvzlMiKN8XccnpGKQ3Q9EtswQ95w==@sourceware.org X-Gm-Message-State: AOJu0Yzw85yJ7IkhWK1aOy0K9Qal3DznLi0jV501jr3hZcaniTJw5F8B 1z3G2h2v7YcWnv3Jr432RwSHWOtMhqmKQICO20/hGx/cFi5luCYleeL39QZOcm3HQ5Ia8uN3Ur2 FQbC0OXcVmrRMjS5SoTqJsr81gnyaQ5Sd X-Gm-Gg: ASbGncsaGW/EnAg70wyUY+tosRm7YdPeeyJjIoEODksx+K4IqFu3gCmCwehdkyYtbEE ZduZ//7IbgIx8ytIsu3aLXTyZ3NLUEjjxM/WQnbgM2TksgREe/El0mvgRQEJmkSlC2pj6AfOwdN vuJ89qi4nPUxe/ZNS+0qY05tc= X-Google-Smtp-Source: AGHT+IFrKpycjf0BO7meaKhAGpJaeHBmacPwvMU+v8LDhouYj4nM99rLMXx4Poh9giq/E7EFWb0u2GZ7u/5WoWZfFDc= X-Received: by 2002:a05:690c:d0e:b0:706:ae3b:cca1 with SMTP id 00721157ae682-706ccdda0b8mr259998787b3.29.1745380337764; Tue, 22 Apr 2025 20:52:17 -0700 (PDT) MIME-Version: 1.0 References: <20250421155940.GE2323@gnu.wildebeest.org> In-Reply-To: <20250421155940.GE2323@gnu.wildebeest.org> Date: Wed, 23 Apr 2025 15:52:05 +1200 X-Gm-Features: ATxdqUFxXy_0x5E6_i9jielb6EYs88ilp8sz8q5GZTqDzF3VF6oHpK79_SgrGf0 Message-ID: Subject: Re: scraperbot protection - Patchwork and Bunsen behind Anubis To: Mark Wielaard Cc: binutils@sourceware.org, elfutils-devel@sourceware.org, gcc@gcc.gnu.org, GDB Mailing list , libc-alpha@sourceware.org, libabigail@sourceware.org, Newlib , overseers@sourceware.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.30 X-BeenThere: gdb@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gdb mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Chris Packham via Gdb Reply-To: Chris Packham Errors-To: gdb-bounces~public-inbox=simark.ca@sourceware.org Sender: "Gdb" Hi Mark, On Tue, 22 Apr 2025, 4:00=E2=80=AFam Mark Wielaard, wrote: > Hi hackers, > > TLDR; When using https://patchwork.sourceware.org or Bunsen > https://builder.sourceware.org/testruns/ you might now have to enable > javascript. This should not impact any scripts, just browsers (or bots > pretending to be browsers). If it does cause trouble, please let us > know. If this works out we might also "protect" bugzilla, gitweb, > cgit, and the wikis this way. > > We don't like to hav to do this, but as some of you might have noticed > Sourceware has been fighting the new AI scraperbots since start of the > year. We are not alone in this. > > https://lwn.net/Articles/1008897/ > > https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-= forcing-blocks-on-entire-countries/ > > We have tried to isolate services more and block various ip-blocks > that were abusing the servers. But that has helped only so much. > Unfortunately the scraper bots are using lots of ip addresses > (probably by installing "free" VPN services that use normal user > connections as exit point) and pretending to be common > browsers/agents. We seem to have to make access to some services > depend on solving a javascript challenge. > > So we have installed Anubis https://anubis.techaro.lol/ in front of > patchwork and bunsen. This means that if you are using a browser that > identifies as Mozilla or Opera in their User-Agent you will get a > brief page showing the happy anime girl that requires javascript to > solve a challenge and get a cookie to get through. Scripts and search > engines should get through without. Also removing Mozilla and/or Opera > from your User-Agent will get you through without javascript. > > We want to thanks Xe Iaso who has helped us set this up and worked > with use over the Easter weekend solving some of our problems/typos. > Please check out if you want to be one of their patrons as thank you. > https://xeiaso.net/notes/2025/anubis-works/ > https://xeiaso.net/patrons/ Ah that might explain a few things. We've seen sporadic failures in the crosstool-ng CI builds (run via a github action) where a download of the newlib snapshot failed (but worked fine when I tried the download manually)= . The good news is that this finally prompted me to look at why we were downloading something that should have been cached. I've fixed that now so whatever extra load our builds were contributing should stop soon. We might still get caught up in the bot detection when a package hosted on sourceware.org is updated. I'm not sure if there is anything we can do about that. I totally understand why this is necessary (AI scraper bots have taken the crosstool-ng website down twice).