Core Toolchain Infrastructure - October 2024 update

Core Toolchain Infrastructure - October 2024 update

[Carlos O'Donell via Gdb]

Core Toolchain Infrastructure - October 2024 update

The Core Toolchain Infrastructure (CTI) Project’s mission is to support
the GNU Toolchain community with secure infrastructure and state of the
art services required to support the community’s development efforts to
be a trusted foundation in a secure supply chain.

We want to keep the GNU Toolchain development community updated with the
latest information on the CTI project and how it can support the GNU
Toolchain.

Since 2022 [1][2] the CTI project has been working to carry out a
detailed enumeration of the services required by the projects and how
those services may be provided in a secure and robust fashion.

We have completed a detailed service enumeration for the GNU Toolchain:
https://cti.coretoolchain.dev/projects/enum.html

In 2024 we published updated project documentation:
https://cti.coretoolchain.dev

Also in 2024 we've looked at the details of the services we provide, why
we provide them, and how we might provide them with with increased
security and robustness, including putting together a statement of work
with the Linux Foundation Core IT team (who provides similar services to
the Linux Kernel) [3].

The CTI Technical Advisory Committee (TAC) is made up of members of the
development community of the projects, and is working on behalf of the
GNU Toolchain to improve security and robustness of the infrastructure
used by the community. The CTI TAC meets monthly and the meetings are
open for anyone to attend.

There is still a lot of work ahead of us to find a way forward to state
of the art sustainable secure and robust services for the GNU Toolchain
projects. Recent discussions on the glibc mailing list make it clear
that we need to expand and discuss more about our "why" along with
the "what" and "how" of these changes.

We will be sending out one of these updates every 3 months to all the
projects to keep you updated on our progress and discussions with the
GNU Toolchain projects and the wider 

Sincerely,
Core Toolchain Infrastructure Technical Advisory Committee

[1] https://inbox.sourceware.org/overseers/ef140a6b-c72d-bd63-b94c-bceeb365b32a@redhat.com/
[2] https://inbox.sourceware.org/overseers/2513b668-9ebd-9e78-7263-dc24f4a9558a@redhat.com/
[3] https://inbox.sourceware.org/libc-alpha/eb0d5e7b-0280-4c5a-aff6-3418a4210a39@redhat.com/

Re: Core Toolchain Infrastructure - October 2024 update

[Mark Wielaard]

Hi Carlos,

On Tue, Oct 29, 2024 at 06:02:03PM -0400, Carlos O'Donell via Gcc wrote:
> Recent discussions on the glibc mailing list make it clear
> that we need to expand and discuss more about our "why" along with
> the "what" and "how" of these changes.

Zoe wrote a good summary of that discussion back in July:
https://inbox.sourceware.org/f20ce996-e9c6-4b6c-856d-eec6e14af26e@fsf.org/
Has anything changed since then to address the issues raised by her
and others?

I don't believe the community is helped by trying to set up yet
another, corporate controlled, organization or doing some highly
disruptive move of some parts of the services our projects are using.

A couple of years ago we spend the time and energy to setup Sourceware
as an organization that provides free infrastructure for our free
software projects. And made sure to work out our relationship with our
fiscal sponsor: https://sourceware.org/Conservancy-Sourceware-FSA.pdf
https://sfconservancy.org/projects/policies/conflict-of-interest-policy.html

We have a generic mission to provide core toolchain and developer
tools project with free software infrastructure:
https://sourceware.org/mission.html

The history and general roadmap for the next 25 years are descriped at
https://sourceware.org/sourceware-25-roadmap.html

And a specific roadmap and projects for state of the art sustainable
secure and robust services, the Secure Sourceware Project Goals, are
described in https://sourceware.org/sourceware-security-vision.html

I noticed you attended the Infrastructure BoF at the Cauldron and seem
to be experimenting with the new Forge we setup. I hope you will be
happy to work with the existing community and the existing
organizations that support the GNU toolchain and the Sourceware
infrastructure, instead of trying to setup yet another organization
that would split our efforts.

Cheers,

Mark

Re: Core Toolchain Infrastructure - October 2024 update

[Carlos O'Donell via Gdb]

On 10/30/24 6:39 AM, Mark Wielaard wrote:
> Hi Carlos,
> 
> On Tue, Oct 29, 2024 at 06:02:03PM -0400, Carlos O'Donell via Gcc wrote:
>> Recent discussions on the glibc mailing list make it clear
>> that we need to expand and discuss more about our "why" along with
>> the "what" and "how" of these changes.
> 
> Zoe wrote a good summary of that discussion back in July:
> https://inbox.sourceware.org/f20ce996-e9c6-4b6c-856d-eec6e14af26e@fsf.org/
> Has anything changed since then to address the issues raised by her
> and others?

Yes, that the CTI TAC needs to expand the discussion of the "why" to the broader
list of the project, and that starts by writing up (something I'm in the progress
of doing) the detailed notes for glibc, particularly why we would want to meet
any of the requirements (and which specific ones) for a secure software development
framework. I'm writing these notes up for the community to continue our discussion.

Then once we have the full "why" written down, list the pros and the cons of an
LF IT-based solution and alternatives, including Sourceware, and again "why" the
TAC recommends one solution over the other.

I can get down to specific requirements and possible solutions for them, including
things like securing logins with 2FA etc. Which *could* be solved by Sourceware
today possibly using Nitrokeys (open hardware and FOSS), for example.

Having all the details spelled out would allow Sourceware to make progress on the
same issues raised, and I can even file infrastructure bugs if that helps.

> I don't believe the community is helped by trying to set up yet
> another, corporate controlled, organization or doing some highly
> disruptive move of some parts of the services our projects are using.

My position here is that the costs of running secure and robust infrastructure are
quite high, and engaging directly with corporate sponsors like we have done before
is the simplest way to pay for FOSS infrastructure. CTI is exactly the same model
we have today, but with broader corporate involvement, instead of just IBM paying
for the current services. This engagement happens in a place where the larger
contributors are already engaged at the Linux Foundation.

Have you discussed with IBM and other larger sponsors to pay Sourceware PLC to 
fund expanding the current services?

My deepest concerns here is that Sourceware PLC cannot convince larger sponsors
to provide the funding to do what needs to be done to scale out and improve our
services.
 
> I noticed you attended the Infrastructure BoF at the Cauldron and seem
> to be experimenting with the new Forge we setup. I hope you will be
> happy to work with the existing community and the existing
> organizations that support the GNU toolchain and the Sourceware
> infrastructure, instead of trying to setup yet another organization
> that would split our efforts.

I'm excited that the GNU Toolchain community is looking at different workflows and
solutions, but if I'm honest the same question of funding and service/workload
isolation applies.

I'm *more* excited to pay Codeberg directly to support the GNU Toolchain to support
the development of Forgejo, particularly given that larger groups like Fedora are
considering Forgejo.

Thanks for your feedback. We can continue the discussion once I post more to the
overseers list.

-- 
Cheers,
Carlos.

Re: Core Toolchain Infrastructure - October 2024 update

[Mark Wielaard]

Hi Carlos,

On Wed, 2024-10-30 at 08:32 -0400, Carlos O'Donell wrote:
> I can get down to specific requirements and possible solutions for them, including
> things like securing logins with 2FA etc. Which *could* be solved by Sourceware
> today possibly using Nitrokeys (open hardware and FOSS), for example.

Yes, a nitrokey distribution scheme is part of the Secure Sourceware
Project Goals: https://sourceware.org/sourceware-security-vision.html

We discussed this with OpenSSF and submitted a funding request to
OpenSSF Alpha Omega for this particular part. OpenSSF initially was
supportive to funding these kinds of security plans, but they have been
silent for the last couple of months. If you have contacts to get this
going forward again that would be great.

> Having all the details spelled out would allow Sourceware to make progress on the
> same issues raised, and I can even file infrastructure bugs if that helps.

Yes, please file bugzilla reports against the Sourceware Infrastructure
project:
https://sourceware.org/bugzilla/buglist.cgi?product=sourceware&component=Infrastructure
Or bring it up on the overseers list or during the Sourceware open
office hours. https://sourceware.org/mission.html#organization

> My deepest concerns here is that Sourceware PLC cannot convince larger sponsors
> to provide the funding to do what needs to be done to scale out and improve our
> services.

Thanks for your concern. The whole idea of setting up Sourceware as an
organization with Conservancy as a fiscal sponsor is precisely to make
these kind of sponsorships easy. And to expand funding to be able to
accept community donations and grants:
https://sourceware.org/donate.html


> I'm excited that the GNU Toolchain community is looking at different workflows and
> solutions, but if I'm honest the same question of funding and service/workload
> isolation applies.
> 
> I'm *more* excited to pay Codeberg directly to support the GNU Toolchain to support
> the development of Forgejo, particularly given that larger groups like Fedora are
> considering Forgejo.

Yes, we did already discuss this. But it is too early for that. Richard
setup a wiki page for the Forge Experiment that includes a list of
various bugs/issues in Forgejo that we would like to see resolved
before we can call the experiment an success.
https://gcc.gnu.org/wiki/ForgeExperiment
When we are a bit further into the experiment to know which ones are
real blockers, we could fund the work to get those done.

Cheers,

Mark

Re: Core Toolchain Infrastructure - October 2024 update

[Karen M. Sandler via Gdb]

On 2024-10-30 11:45, Mark Wielaard wrote:
> Hi Carlos,
> 
> On Wed, 2024-10-30 at 08:32 -0400, Carlos O'Donell wrote:
>> I can get down to specific requirements and possible solutions for 
>> them, including
>> things like securing logins with 2FA etc. Which *could* be solved by 
>> Sourceware
>> today possibly using Nitrokeys (open hardware and FOSS), for example.
> 
> Yes, a nitrokey distribution scheme is part of the Secure Sourceware
> Project Goals: https://sourceware.org/sourceware-security-vision.html
> 
> We discussed this with OpenSSF and submitted a funding request to
> OpenSSF Alpha Omega for this particular part. OpenSSF initially was
> supportive to funding these kinds of security plans, but they have been
> silent for the last couple of months. If you have contacts to get this
> going forward again that would be great.
> 
>> Having all the details spelled out would allow Sourceware to make 
>> progress on the
>> same issues raised, and I can even file infrastructure bugs if that 
>> helps.
> 
> Yes, please file bugzilla reports against the Sourceware Infrastructure
> project:
> https://sourceware.org/bugzilla/buglist.cgi?product=sourceware&component=Infrastructure
> Or bring it up on the overseers list or during the Sourceware open
> office hours. https://sourceware.org/mission.html#organization
> 
>> My deepest concerns here is that Sourceware PLC cannot convince larger 
>> sponsors
>> to provide the funding to do what needs to be done to scale out and 
>> improve our
>> services.
> 
> Thanks for your concern. The whole idea of setting up Sourceware as an
> organization with Conservancy as a fiscal sponsor is precisely to make
> these kind of sponsorships easy. And to expand funding to be able to
> accept community donations and grants:
> https://sourceware.org/donate.html

Yes, SFC is already set up to receive donations from most of the large 
companies that are consistent funders in this space (we're registered in 
their vendor systems). Similarly, we regularly have fundraising meetings 
with them across our member projects. If you have a particular lead or 
suggestion for Sourceware, please let me/us know and we'll follow up!

karen



> 
> 
>> I'm excited that the GNU Toolchain community is looking at different 
>> workflows and
>> solutions, but if I'm honest the same question of funding and 
>> service/workload
>> isolation applies.
>> 
>> I'm *more* excited to pay Codeberg directly to support the GNU 
>> Toolchain to support
>> the development of Forgejo, particularly given that larger groups like 
>> Fedora are
>> considering Forgejo.
> 
> Yes, we did already discuss this. But it is too early for that. Richard
> setup a wiki page for the Forge Experiment that includes a list of
> various bugs/issues in Forgejo that we would like to see resolved
> before we can call the experiment an success.
> https://gcc.gnu.org/wiki/ForgeExperiment
> When we are a bit further into the experiment to know which ones are
> real blockers, we could fund the work to get those done.
> 
> Cheers,
> 
> Mark

Re: Core Toolchain Infrastructure - October 2024 update

[Joseph Myers via Gdb]

On Wed, 30 Oct 2024, Mark Wielaard wrote:

> Yes, we did already discuss this. But it is too early for that. Richard
> setup a wiki page for the Forge Experiment that includes a list of
> various bugs/issues in Forgejo that we would like to see resolved
> before we can call the experiment an success.
> https://gcc.gnu.org/wiki/ForgeExperiment
> When we are a bit further into the experiment to know which ones are
> real blockers, we could fund the work to get those done.

There is also my long list of considerations at 
https://gcc.gnu.org/pipermail/gcc/2024-September/244806.html for 
evaluation - it's not just a few specific issues, there are lots of things 
to evaluate.  Some may be supported well already, some may need more work 
on the ForgeJo side, some may involve significant local scripting to 
combine with existing or new APIs on the ForgeJo side.

I don't think there's any real evaluation been done yet of what email 
notifications can be set up to go to mailing lists and what those 
notifications look like (for pull requests, commits, etc.) and how much is 
configurable there or needs custom scripting, or of what configuration is 
available for checks on commits that are allowed to enter either the 
repository at all or specific branches thereof (both commits pushed 
directly and commits merged via PRs) - but such checks are certainly 
important to avoid various subsequent automation (e.g. the nightly cron 
jobs) falling over.  (I expect many of these things to involve some kind 
of configuration hooks that link into appropriate APIs / scripting we'd 
need to provide, rather than ForgeJo having configuration options that 
directly match what we want.)  Likewise, for API support sufficient for 
people to build their own efficient workflows for reviewing lots of 
patches, where they currently have such workflows based on email.

As discussed in the previous thread, such evaluation would probably take a 
narrative form discussing how desired features relate to what ForgeJo 
provides, not checkbox yes/no for each item, and so provide a basis for 
further consideration of what we achieve through appropriate hooks / 
scripting, what we seek to get added as ForgeJo features (possibly paying 
for features to be implemented) and what we might end up deciding is less 
important or has underlying goals that can be achieved in a different way.

Each toolchain project may reach its own conclusions about what's 
important for possibly moving to a forge - we don't have any group that 
makes decisions for the toolchain as a whole, and each project has its own 
existing hooks and other scripting / automation that would need adapting.

-- 
Joseph S. Myers
josmyers@redhat.com

Re: Core Toolchain Infrastructure - October 2024 update

[Carlos O'Donell via Gdb]

On 10/30/24 11:45 AM, Mark Wielaard wrote:
> Hi Carlos,
> 
> On Wed, 2024-10-30 at 08:32 -0400, Carlos O'Donell wrote:
>> I can get down to specific requirements and possible solutions for them, including
>> things like securing logins with 2FA etc. Which *could* be solved by Sourceware
>> today possibly using Nitrokeys (open hardware and FOSS), for example.
> 
> Yes, a nitrokey distribution scheme is part of the Secure Sourceware
> Project Goals: https://sourceware.org/sourceware-security-vision.html

Have you broken down those project goals into actionable steps that could be taken?

For example filing Sourceware Infrastructure bugs for each service that needs to be
migrated into a VM and isolated (with a top level tracker for "Increased isolation")?

If you're going to ask for funding, having a list of concrete goals the funding
will solve, broken down to the level at which you can write an SOW, is very very
beneficial.
 
> We discussed this with OpenSSF and submitted a funding request to
> OpenSSF Alpha Omega for this particular part. OpenSSF initially was
> supportive to funding these kinds of security plans, but they have been
> silent for the last couple of months. If you have contacts to get this
> going forward again that would be great.

I do have contacts at the OpenSSF and I'd be glad to help. We just met with one of
their team members today as part of the CTI TAC meeting.

Do you have your funding request anywhere that I can read it?

>> Having all the details spelled out would allow Sourceware to make progress on the
>> same issues raised, and I can even file infrastructure bugs if that helps.
> 
> Yes, please file bugzilla reports against the Sourceware Infrastructure
> project:
> https://sourceware.org/bugzilla/buglist.cgi?product=sourceware&component=Infrastructure
> Or bring it up on the overseers list or during the Sourceware open
> office hours. https://sourceware.org/mission.html#organization

For tracking purposes I'll file them as Sourceware Infrastructure bugs and
we can go from there.

>> My deepest concerns here is that Sourceware PLC cannot convince larger sponsors
>> to provide the funding to do what needs to be done to scale out and improve our
>> services.
> 
> Thanks for your concern. The whole idea of setting up Sourceware as an
> organization with Conservancy as a fiscal sponsor is precisely to make
> these kind of sponsorships easy. And to expand funding to be able to
> accept community donations and grants:
> https://sourceware.org/donate.html

What you have done is make it *possible* for an organization to place money at the
fiscal sponsor for the mission you've set out, and while this is a measure of ease,
the hardest step is still to come. You need to convince sponsors to donate.

David, Joel and I have been the trustees of the GNU Toolchain Fund since we worked
with the FSF to set it up in 2017. Since then the hardest step is getting larger
sponsors to support.

How have your fund raising activities been going for the Sourceware fund at the SFC?

Have you allocated and spent any of that funding to move the project goals forward?
 
>> I'm excited that the GNU Toolchain community is looking at different workflows and
>> solutions, but if I'm honest the same question of funding and service/workload
>> isolation applies.
>>
>> I'm *more* excited to pay Codeberg directly to support the GNU Toolchain to support
>> the development of Forgejo, particularly given that larger groups like Fedora are
>> considering Forgejo.
> 
> Yes, we did already discuss this. But it is too early for that. Richard
> setup a wiki page for the Forge Experiment that includes a list of
> various bugs/issues in Forgejo that we would like to see resolved
> before we can call the experiment an success.
> https://gcc.gnu.org/wiki/ForgeExperiment
> When we are a bit further into the experiment to know which ones are
> real blockers, we could fund the work to get those done.

Yes, I agree we're too early.

Fedora has commented publicly that Codeberg's informal position was that they
probably did not have the capacity to host a project of Fedora's size.

https://discussion.fedoraproject.org/t/a-vote-in-favor-of-forgejo/112059/5

-- 
Cheers,
Carlos.

Re: Core Toolchain Infrastructure - October 2024 update

[Joseph Myers via Gdb]

On Wed, 30 Oct 2024, Carlos O'Donell via Gcc wrote:

> Have you broken down those project goals into actionable steps that 
> could be taken?
> 
> For example filing Sourceware Infrastructure bugs for each service that 
> needs to be migrated into a VM and isolated (with a top level tracker 
> for "Increased isolation")?

Note that we have the CTI service enumeration which effectively identifies 
the various services that should be isolated for various GNU toolchain 
projects.

There may be other services on Sourceware that aren't used by any of the 
GNU toolchain projects with CTI service enumerations.  Those can still be 
of security relevance to the toolchain, if a compromise of one of those 
services could be leveraged to compromise other services or projects - 
isolation matters both inward and outward, for both current and future 
services and projects.

-- 
Joseph S. Myers
josmyers@redhat.com

Re: Core Toolchain Infrastructure - October 2024 update

[Mark Wielaard]

Hi Carlos,

On Wed, Oct 30, 2024 at 12:52:13PM -0400, Carlos O'Donell wrote:
> > We discussed this with OpenSSF and submitted a funding request to
> > OpenSSF Alpha Omega for this particular part. OpenSSF initially was
> > supportive to funding these kinds of security plans, but they have been
> > silent for the last couple of months. If you have contacts to get this
> > going forward again that would be great.
> 
> I do have contacts at the OpenSSF and I'd be glad to help. We just
> met with one of their team members today as part of the CTI TAC
> meeting.

Thanks, I see the OpenSSF General Manager and the Technical Program
Managers have gotten different positions or moved on from OpenSSF. I
added the new contacts to reach out to.

> > Yes, please file bugzilla reports against the Sourceware
> > Infrastructure project:
> > https://sourceware.org/bugzilla/buglist.cgi?product=sourceware&component=Infrastructure
> > Or bring it up on the overseers list or during the Sourceware open
> > office hours. https://sourceware.org/mission.html#organization
> 
> For tracking purposes I'll file them as Sourceware Infrastructure
> bugs and we can go from there.

Thanks, that would be useful input.

> >> My deepest concerns here is that Sourceware PLC cannot convince
> >> larger sponsors to provide the funding to do what needs to be
> >> done to scale out and improve our services.
> > 
> > Thanks for your concern. The whole idea of setting up Sourceware as an
> > organization with Conservancy as a fiscal sponsor is precisely to make
> > these kind of sponsorships easy. And to expand funding to be able to
> > accept community donations and grants:
> > https://sourceware.org/donate.html
> 
> What you have done is make it *possible* for an organization to
> place money at the fiscal sponsor for the mission you've set out,
> and while this is a measure of ease, the hardest step is still to
> come. You need to convince sponsors to donate.

The hardest step and what cost most of the energy was setting up the
organization, the PLC, working out our relationship with our fiscal
sponsor, making sure to get the governance right. And setting rules
for making sure to preserve software freedom and diversify income
sources.

Large monetary donations from corporations are certainly nice, but you
have to make sure the community keeps in control. Having large
corporations dominate the funding is risky, so we are also explicitly
looking at individual donations and grants.

Our largest sponsors provide hardware and services directly instead of
exchanging money. https://sourceware.org/mission.html#sponsors
They are valued partners with who we can discuss community and
services goals. For example about cyber security regulations.

> How have your fund raising activities been going for the Sourceware
> fund at the SFC?

Very well, thanks. See our last yearly report:
https://inbox.sourceware.org/20240529190215.GA26515@gnu.wildebeest.org/
We have been getting more hardware and assistence from our sponsors to
expand our services and are pulling in ~$250,- dollars a month from
individual donations and small grants. We are currently just spending
~5% of that to make sure we are building up enough reserve to be able
to replace any hardeware and services in case one of our regular
sponsors might have to drop out.

Cheers,

Mark