From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id VMwDAUde72ohzjAAWB0awg (envelope-from ) for ; Sat, 07 Nov 2026 12:22:15 -0500 Authentication-Results: simark.ca; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=C4TP0tfZ; dkim-atps=neutral Received: by simark.ca (Postfix, from userid 112) id CC02B1E09A; Sat, 07 Nov 2026 12:22:14 -0500 (EST) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=ARC_SIGNED,ARC_VALID,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,RCVD_IN_VALIDITY_CERTIFIED_BLOCKED, RCVD_IN_VALIDITY_RPBL_BLOCKED,RCVD_IN_VALIDITY_SAFE_BLOCKED autolearn=ham autolearn_force=no version=4.0.1 Received: from vm01.sourceware.org (vm01.sourceware.org [38.145.34.32]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id AF9BB1E023 for ; Sat, 07 Nov 2026 12:22:11 -0500 (EST) Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id DC4E84BA23D6 for ; Sat, 7 Nov 2026 17:22:04 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org DC4E84BA23D6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1794072124; bh=PrelQ/Id2OvtLiWxEADa769C4ege5h7ZqsxKMI6+vJk=; h=Date:To:Subject:Cc:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From:Reply-To:From; b=C4TP0tfZ6ibNhGwYL8FMhq8HWpS1RKVDjfPyhZmW/toHSqKuy1PuShfY2NJXMgrnJ F1xzjy+k5hLbjm2eZECbHVNe+7WpMd5HGXy/ttkASv69TdlTP6sMa9KSFLLrZ998r8 w/XH7281scmBteXY37O9eG3Z75ZKJ36NZKt6p1aI= Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id 969DA4BA2E1A for ; Sat, 7 Nov 2026 17:21:26 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 969DA4BA2E1A ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 969DA4BA2E1A ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1794072086; cv=none; b=TvSWtqgQp2GzZbIK2tQD0FaZ7uFwt4J3gZDsX2RbmCbM20kwX98/uh+Po8VPeRBpx8QUpWZKV5ZuNE81NwfP1z3UtYy0NQVD+aRToMR2pbQRYdTM1fzoQ2juAHKBTNkbfWGu8GWB2/zQfsGddwmexNierXhxEoLOK88NtKOLFIk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1794072086; c=relaxed/simple; bh=nBCVQpkBoZMCZXfug4yX0OpbAwSf02h07zCKDTsvoiI=; h=DKIM-Signature:Message-ID:Date:MIME-Version:To:From:Subject; b=W2hJDfJmooALhCa9Z5n5Ac2L/+SNU0GfNIyRO7lH+lVX3rPt393z3f884ZZ2LUNOz7qCZgjPrkBH0QzeQAWmCZwrv2jA5IA2vm5rVxItL33Z+2ANY+W/SQmKymNntsldJocfycPCv+jDfPdsaQpLJDL+3T7yqc8iudEQwacGwHI= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Jgiy/MYE DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 969DA4BA2E1A Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-55-cXmdRPG5PAqBTo2mL4Fh4A-1; Sat, 07 Nov 2026 12:21:24 -0500 X-MC-Unique: cXmdRPG5PAqBTo2mL4Fh4A-1 X-Mimecast-MFC-AGG-ID: cXmdRPG5PAqBTo2mL4Fh4A_1794072084 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-91eb05c8241so31219016d6.1 for ; Sat, 07 Nov 2026 09:21:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794072084; x=1794676884; h=content-transfer-encoding:content-type:cc:organization:autocrypt :subject:from:to:content-language:user-agent:mime-version:date :message-id:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=PrelQ/Id2OvtLiWxEADa769C4ege5h7ZqsxKMI6+vJk=; b=huQ/OMNY+fsTyhnNpCVI2Pbeza3chShDhC5aA3m3cFZycVuwpffMsflTkZ5BJjMYxZ 7+Su6AbA4IMbwbqVspyAEidTw6e5lx7baSJH7/nOGdVd1wFhiEOQ9+Tre/pJa7E68gP0 LlwaTx38kwGIPHRvQrBMRt6XLF1PCB7pz6D3CTevbw9+5M6DeT/yCE/DTC0k1pHOnMgS UsT9ET89/5LmA1qHmerhKbbCDNpZALmX6QCHzM7V1YB37CJidxMWCVVmu1Ly+NhVR7kA ufMynZPPqX7MK7ggIwAuP7nFmbXhQx55QiBZU+QGs9vnl+IO89spkds61bTJjSnb36Kg 2foA== X-Gm-Message-State: APuU4Z+rRx07AHr573u4VQABj5sNBqpxZ1wW+XQWWpRsuBXpLULTozh0 Kc6ASc3nUGCkOvmMNKDpE+DYul/xx9oxZ76S+AvId1X6fe2Ucu9lnYZx/NqMrVUO5QtPiWxqgx3 4bt/F04diu7xM9tgdIxroLY2nPqk1zWrjCaRsGmxLHRTBYrtl29if X-Gm-Gg: AYBFou2oQ1NphBHQRpho/EBFKyWVadxwUdsxwMvnrlK/5OUb/euzqgDVwuCixO11f9Q uNWe8g73a51CyGkCcgHW5uZvvqEnflOhzpVBaq1cdF5haMKnwPDktOlnanXP86c2arVZIFrYnnk CEmHUpSA/ElNJbMwLeDdV5sh49fKYrEW3Zq7Jh85w183Y7EUwevXxpYQGVsRRPBXgxF4dTaurCe Lvc0SA3H5fsPPu/mqI6am31Eq8BI2rz6eaTgRa8mYh5daAbyfve8HWR1YTWtg5O5agtJZJGSEMv fB+tmK3lKchguWoGmIeVMZL7QluyPDUV9u/sS7UXpYXhhQsuBQmJPRFN8BDKYDMykTWZ+i5oCWq HsrgDdHCkV0sCy9E9tr+tjfzCce8GiKlgkPtUlaKRQ3o+8DLPkxH0yZ74kO1VP7j3+He3DQ+dAp a3gAon7qmSQakhB66BYWhSDSRNly6RbCab9Kkuiu4zE3ybnIw= X-Received: by 2002:a05:620a:199b:b0:945:b967:475f with SMTP id af79cd13be357-945f349b214mr988995385a.16.1794072083520; Sat, 07 Nov 2026 09:21:23 -0800 (PST) X-Received: by 2002:a05:620a:199b:b0:945:b967:475f with SMTP id af79cd13be357-945f349b214mr988987285a.16.1794072082136; Sat, 07 Nov 2026 09:21:22 -0800 (PST) Received: from [192.168.0.116] ([198.48.244.52]) by smtp.gmail.com with ESMTPSA id af79cd13be357-945f5a97aa1sm398484585a.8.2026.11.07.09.21.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 07 Nov 2026 09:21:20 -0800 (PST) Message-ID: <272e308a-a703-4824-b823-cd8e5266f00b@redhat.com> Date: Sat, 7 Nov 2026 12:21:19 -0500 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: glibc developers Subject: [RFC] Provide horizontally scaled git pull mirrors for the GNU Toolchain, starting with glibc. Autocrypt: addr=carlos@redhat.com; keydata= xsFNBFef5BoBEACvJ15QMMZh4stKHbz0rs78XsOdxuug37dumTx6ngrDCwZ61k7nHQ+uxLuo QvLSc6YJGBEfiNFbs1hvhRFNR7xJbzRYmin7kJZZ/06fH2cgTkQhN0mRBP8KsKKT+7SvvBL7 85ZfAhArWf5m5Tl0CktZ8yoG8g9dM4SgdvdSdzZUaWBVHc6TjdAb9YEQ1/jpyfHsQp+PWLuQ ZI8nZUm+I3IBDLkbbuJVQklKzpT1b8yxVSsHCyIPFRqDDUjPL5G4WnUVy529OzfrciBvHdxG sYYDV8FX7fv6V/S3eL6qmZbObivIbLD2NbeDqw6vNpr+aehEwgwNbMVuVfH1PVHJV8Qkgxg4 PqPgQC7GbIhxxYroGbLJCQ41j25M+oqCO/XW/FUu/9x0vY5w0RsZFhlmSP5lBDcaiy3SUgp3 MSTePGuxpPlLVMePxKvabSS7EErLKlrAEmDgnUYYdPqGCefA+5N9Rn2JPfP7SoQEp2pHhEyM 6Xg9x7TJ+JNuDowQCgwussmeDt2ZUeMl3s1f6/XePfTd3l8c8Yn5Fc8reRa28dFANU6oXiZf 7/h3iQXPg81BsLMJK3aA/nyajRrNxL8dHIx7BjKX0/gxpOozlUHZHl73KhAvrBRaqLrr2tIP LkKrf3d7wdz4llg4NAGIU4ERdTTne1QAwS6x2tNa9GO9tXGPawARAQABzSpDYXJsb3MgTydE b25lbGwgKFdvcmspIDxjYXJsb3NAcmVkaGF0LmNvbT7CwZUEEwEIAD8CGwMGCwkIBwMCBhUI AgkKCwQWAgMBAh4BAheAFiEEcnNUKzmWLfeymZMUFnkrTqJTQPgFAmod99EFCRRfRy8ACgkQ FnkrTqJTQPiF+hAAhstxT3ao2pBuNQNh7/yieJxsQ0YrI0vZeIMM/P5kxzr0X/+CNCFx/bbk ogjOVgVMUD1WzqW6zY8lsj3/RPiBsowZBJYXIsWfdvYq33WwgtiXA/1AUZ60/lfWIQdPQJDL 8/FXzRy8P4B/DoSpMOU11S5Kk8j5R9mBk/NSrNreO/nB1e4qNFF5GluBUsxMjKrGkkOxUZPQ uGNjcgdxRibNVD8n+lp9UU9+rsFKNOK7F0l9/ltR6j8kVagMKikXsaoUFU6WUbGHQ+ZBPdY8 o8izXbQLoW/kD1qxbMsviBPaszaRZTyKd4ZEz54dx4up3vLorBoLI6YcG2oTqIdx8YZSXdfK WVPhQsJKG/QgiPztGYzT93KvhHlQ3+D9GZBU0ovAWc4YkBjxLPsw4DLhgFGLgIGhFHuMY3jy qlzez8KpoehH7mEgc5yGBtbnejnhuo7N6q6SUEGcSWlyLsdHMPZOg2UhpJ4pS9u7vG9Cz7+b yxItTMJ0XhRjQIkdm+38lH1yaOqVU9jvTgpPf3P9vT3Xo5vCa8PM5U7oL/Be2O6JH4hj9835 h0AJfKsDJD50OgS9zI8e51X+vSTir1Kk+tMJJgZLYqYsSKOwUm+vVLby1pK171lIuu1qjx6I Rqz9CxmD5iRWIeIaOGGI+gce4eeMBH1IbFS6EjHT2hu+AfoURjjOwU0EV5/kGgEQAKvTJke+ QSjATmz11ALKle/SSEpUwL5QOpt3xomEATcYAamww0HADfGTKdUR+aWgOK3vqu6Sicr1zbuZ jHCs2GaIgRoqh1HKVgCmaJYjizvidHluqrox6qqc9PG0bWb0f5xGQw+X2z+bEinzv4qaep1G 1OuYgvG49OpHTgZMiJq9ncHCxkD2VEJKgMywGJ4Agdl+NWVn0T7w6J+/5QmBIE8hh4NzpYfr xzWCJ9iZ3skG4zBGB4YEacc3+oeEoybc10h6tqhQNrtIiSRJH+SUJvOiNH8oMXPLAjfFVy3d 4BOgyxJhE0UhmQIQHMJxCBw81fQD10d0dcru0rAIEldEpt2UXqOr0rOALDievMF/2BKQiOA7 PbMC3/dwuNHDlClQzdjil8O7UsIgf3IMFaIbQoUEvjlgf5cm9a94gWABcfI1xadAq9vcIB5v +9fM71xDgdELnZThTd8LByrG99ExVMcG2PZYXJllVDQDZqYA1PjD9e0yHq5whJi3BrZgwDaL 5vYZEb1EMyH+BQLO3Zw/Caj8W6mooGHgNveRQ1g9FYn3NUp7UvS22Zt/KW4pCpbgkQZefxup KO6QVNwwggV44cTQ37z5onGbNPD8+2k2mmC0OEtGBkj+VH39tRk+uLOcuXlGNSVk3xOyxni0 Nk9M0GvTvPKoah9gkvL/+AofN/31ABEBAAHCwXwEGAEIACYCGwwWIQRyc1QrOZYt97KZkxQW eStOolNA+AUCah337wUJFF9HVQAKCRAWeStOolNA+FSnD/sHeLb9vaXHABC/1eyrScwSDfZs MFsdjzZZqGbnbdj4uP4oyt3uztsxJASgHRQ/DhaPsms3Ai+kWhf5EHOBC9cPSn0aHZL52nT+ MdbTu2VvvVRfECSm+lo/NkTGoXkF+n6TFd7tQGmSjdudvUhiKfSzep3GYseknmb9+7/DGluv QjuDb8quUVglMifp3qRUH4yPo7BS8ak59t4FUc8m2ggglgcvzKCRgqgEtrP2vM6EFzUiRJ1Y 0et5kedtiAI1e9ehVeyytFELputja7EzAxWv+pQ2m/jhfoAHx/FTqw4PdrtH2zY9SmCheWDE wIgA+eyAHTG/X5DFq2P45agY6K+cvlDvr6V2NDsoOYI91A0VKahKkNIDAsMJ7sqMys/qw9kp dJVZaDqoNdY9NtDayLWXPXyJZEAy0Xsvw0vdoswthtjQmeZWhx2fMSCwG8mQlAya+nnfalPi iuPL6BUkk7naKk9wHpEZDMlyp6CLefLe1ioVFhnN2nL5SseY9nnmDo05lidf11U4VVWsVHJ+ ViM9CQLD+NCazBNavYUPBEpFuT7HX4zWfpTNU9GXbEYcvrUb6Oo59/9ns7e9stTuskyxxxyY ryfF9FLmLQkM9soiwxGIt6k7ObkNsBRYzuM5q1lmDV6MA1llxUPmf9e5b3OFJ7gqXeRZ5niK hsfHTTi7JQ== Organization: Red Hat, LLC. Cc: gdb developers , binutils developers , gcc developers X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 0j5how5SJnSa6TePj08AYpavCc9K8A9dSikigF_Qa9A_1794072084 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: gdb@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gdb mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Carlos O'Donell via Gdb Reply-To: Carlos O'Donell Errors-To: gdb-bounces~public-inbox=simark.ca@sourceware.org Sender: "Gdb" Situation: We currently serve all unauthenticated git read traffic through the following URL: https://sourceware.org/git/glibc.git Similarly for binutils/gdb: https://sourceware.org/git/binutils-gdb.git This includes all git clone and fetch traffic. We also handle all write traffic e.g. `git push` to the same host. This has resulted in a bottleneck at the host, that at times resulted in developers being unable to access sources due to IP deny lists, or the disabling of HTTPS [1] for secure access. Background: Unauthenticated, or anonymous traffic, usually takes many forms including developers doing git pull (in aggregate), CI/CD systems in downstream testing, downstream mirrors (like the gitlab/github ones we have), and LLMs pulling the latest sources. In such a git pull operation the majority of the CPU and memory time can be dominated by the CPU and IO required by the git-upload-pack process. There are some things that can be done on the server side to minimize the cost, likewise on the client side e.g. shallow clones, maintenance and tuning etc. However, the read traffic is anonymous and untrusted, but has a right to access the sources of the project; we don't discriminate against use [2]. We also want that reader to have confidence they are getting sources that they can trust e.g. HTTPS connected pull (following the glibc SSDLC [3]). A single host can be scaled "vertically" e.g. more CPUs, more memory etc. It is not possible to scale the single host read side in a horizontal fashion e.g. geographically close. A vertically scaled host absorbs read traffic spikes up to the limit of the current network which might not be possible to scale before hitting physical limits e.g. data center limits, cage network limits etc. Assessment: It should be possible to safely serve anonymous reads from a mirror. The requirements of this reader are likely sufficiently served by the eventual consistency that a replica can provide. A read-only mirror is a replica of the canonical repository and serves only anonymous git pulls. The server never accepts a git push and has no credentials with which to push. A compromise in the read-only mirror can be detected by comparison to the canonical read-write server. We can therefore scale horizontally by adding N mirror nodes behind a single hostname and distribute with geodns, round-robin, or anycast. Nodes can be added, routed around, replaced, all without impacting the read-write host, and can be operated by the same or different FOSS service providers. There are many robust FOSS solutions that exist to provide such services, but they require scaling horizontally and paying the cost for that scale in admin and hardware. If a read-only mirror is compromised it has no access to any other system and cannot alter history. The damage is limited (but serious) to serving stale or incorrect read data to anonymous clients, which while not done today is detectable via signed content and ref verification (which can be done continuously as a separated and distinct process). The best solution remains to get all commits signed, along with a transparency log, but a defense-in-depth approach can start slowly. The canonical repository remains the single source of truth and the only thing that accepts writes. Note that for now cgit/gitweb reads are not addressed in this RFC but remain a relevant amount of load and traffic that can be discussed in another RFC. Recommendation: The goal is to scale horizontally and ensure pulls succeed from anywhere. The recommended mechanism after discussion between the CTI TAC and LF IT, who deploys similar infrastructure, is grokmirror: https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git/ This is the same tooling kernel.org uses to run its mirror network, and is similar to the 4-node mirror network the GNU Project is standing up on savannah, referenced in the prior DNS end points RFC[4]: https://lists.gnu.org/archive/html/savannah-hackers-public/2026-05/msg00034.html The recommendation is to stand up a read-only git mirror network, replicated from the canonical host with grokmirror, under a project service specific end point: git.glibc.coretoolchain.dev -> read-only git mirror network As in the DNS RFC [4], this dovetails into a conversation with the GNU Project and FSF over matching DNS service endpoints which we can document. This introduces the requirement to trust the mirror network operator, but provides horizontal scaling for service robustness. The best ways to mitigate trust risks are to make progress on signing all commits, setting up a transparency log, and generally adopting a defense-in-depth approach. Keep all authenticated push and write traffic on the canonical write end point from the DNS end points RFC: git.glibc.sourceware.org -> read/write git access This works to collapse all anonymous reads to a fixed set of trusted mirrors. While we don't control where the anonymous reads come from or where they go, providing for the availability of moving traffic to N trusted readers versus all anonymous traffic should be measurable. The mirror nodes would run on distinct VM systems and split across US, Canada, and the EU to provide local mirrors for developers. There will need to be some anti-abuse control at the nodes, but the scaling can be reviewed against cost to alleviate that during operation. This has a secure supply chain consequence in that grokmirror must write a manifest on update of the write side of git server in order to support optimal remote mirror pull updates e.g. fingerprint change. This is a much smaller attack surface than the AdaCore git hooks. Only "grok-manifest" has to run in the trusted path, uses only local data (configurable), and arguably manipulates data that should not block it e.g. sha1 (git show-ref output). There is going to be replication lag, and if a developer needs strong read-after-write consistency they should pull from the canonical server. We will monitor replication lag to determine how small we want to make that lag since it has costs. The CTI services are currently provided by LF IT. The service provider commits to GNU Ethical Repository hosting criteria Grade B for services. DNS end points can be moved in the future if service providers change. This should be scaled up to handle gcc, and binutils and gdb too. -- Cheers, Carlos. Notes: - It is a hard requirement that fetch_objstore be disabled in grokmirror. - coretoolchain.dev, being a .dev domain is HSTS-preloaded, no HTTP. [1] https://inbox.sourceware.org/overseers/20251229005935.GJ30914@gnu.wildebeest.org/ [2] https://www.gnu.org/software/repo-criteria.en.html "Does not discriminate against classes of users" [3] https://sourceware.org/glibc/wiki/SSDLC/Policy/glibc [4] https://inbox.sourceware.org/libc-alpha/8e167d49-86c5-45ba-841d-f9f74151d596@redhat.com/