Re: Sourceware Open Office, Fri 16:00 UTC

Mark Wielaard
2026-03-16T01:13:47Z

For those that couldn't attend. It was a longer than usual
meeting. But we will continue the discussions on list of course.

- New hardware and how forgejo action runners can use it
- Servers, VMs and services
- 2026 infrastructure goals and budget plans for paid staff

On Fri, Mar 13, 2026 at 02:30:10AM +0100, Mark Wielaard wrote:
> Last month we finalized our hardware refresh cycle and moved all
> services off bare metal servers onto VMs (for x86_64 hardware) which
> greatly simplifies backups, redundancy and moving services around. For
> hardware we should be good for the next couple of years now and thanks
> to your generous donations our hardware refresh fund is back up
> strenght so we are able to replace any server in case they might
> break. We now also have two extra forgejo action runners (sw3runner1:
> 40 vcpus, 256GB mem, 500GB disk and sw3runner2: 16 vcpus, 96GB mem,
> 420GB disk) which can be used in forge workflows.

= New hardware and how forgejo action runners can use it

The new forge action runners sw3runner1 and sw3runner2 seem to pick up
existing workflows. But that isn't really taking advantage of the new
40/16 vcpu setups.

It doesn't look like we have global stats, but this is for gcc:
https://forge.sourceware.org/gcc/gcc-TEST/actions

There's workflows and there are actions. Workflows use pre-made
images. Actions can make them on the fly it seems. We are currently
using workflows and it isn't clear how in forgejo one can create
images on the fly (from container files as the buildbot does).

For elfutils the workflow uses an image/action
https://forge.sourceware.org/elfutils/elfutils/src/branch/main/.forgejo/workflows

Our runners are in a isolated VM
https://forgejo.org/docs/latest/admin/actions/docker-access/#forgejo-runner-in-a-vm
which spawn containers for actions/workflows. When you use
sourceware-debian-runner it will be debian stable, when using
sourceware-fedora-runner it will be fedora latest. The runner adds
labels: https://forgejo.org/docs/latest/admin/actions/#choosing-labels

There is an increase in forge spam signups but a) they seem to never
activate their account. And b) if they do they are restricted, so that
is fine. We don't have a good workflow for "real" signups. Those
people should be added to a Contributor group for a project to be able
to participate. Those people are still restricted though, so they can
only contribute to one specific project.

It might be nice if we could white-list certain domains so that folk
signing up through those had fewer hoops to jump through. It could be
automated with the api.
https://forge.sourceware.org/api/swagger/#/admin/adminEditUser

= Servers, VMs and services

Discussion about all the different servers we have now and which
services, backups and redundant services run where. There is the
osuosl openstack cluster where snapshots runs in a VM. There is
sourceware-builder3 which has 4 VMs. sw3runner1, sw3runner2, sw3bb1
and sw3bb2. There is server1 which has 8 VMs, currently "sourceware",
forge-stage, debuginfod, and forge (the new one) with slots for inbox,
builder, patchwork and bunsen. There is the RH OSPO openstack cluster
where the current forge still runs (already synced with the new
one). Then there are server2 which is setup for running VMs but
doesn't yet. And server3 which should have a sourceware hot backup VM
(or vice versa server2/server3; server2 has more ram)

Mark promises an ascii art drawing.

= 2026 infrastructure goals and budget plans for paid staff

We are setup pretty nice for hardware and hosting now. And our
hardware refresh fund is also back up to about $8000+. We should be
good for the next couple of years now.

So for this year we would like to get a budget setup to help some of
the volunteer admins to get paid staff help to finalize some of our
(security) plans
https://sourceware.org/sourceware-security-vision.html#plans

There are two questions here. First is if the plans are complete are
we missing a broad category? Second what it would take to hire paid
staff/contractors to do some of the work specified (we could of course
do everything ourselves, but that might take much longer)

Discussion about how at Fosdem we started the process to estimate how
many hours each task would take, what level of expertise someone would
need and how much they cost. e.g. different levels of contractors or
(junior) adminstrators. The hours/days it would take the volunteer
admins, and factor for someone not yet familiar with setups.

Next up we'll do a community survey, like we did last year, to see
which services are most important and what priority which plans
have. And setup a meeting with the SFC staff to finalize the budget,
publish it and get a grant or corporate donation to execute.

> Keep Sourceware worry-free, friendly and independent by donating
> https://sourceware.org/donate.html support our fiscal sponser SFC
> https://sfconservancy.org/sustainer and/or support OSUOSL for
> hosting Free Software projects https://osuosl.org/donate/
>
> Do you or your company want to sponsor Sourceware plans financially
> https://sourceware.org/sourceware-security-vision.html#plans
> donate hardware or services then contact us at sponsor@sourceware.org