GDB CVS ok; was CVS outage

GDB CVS ok; was CVS outage

[Andrew Cagney]

FYI,

GDB's CVS repository looks ok, thanks!

Andrew

---- Original "back up" e-mail posted to overseers@ ----

From: Chris Faylor

We're back online.

As most of you know, we suffered a pretty severe hardware outage on
Thursday.  The best theory right now is that a bad hard drive in the
RAID array indirectly caused a data corruption problem.  The problem was
really due to a problem with old firmware (which mgalgoci has since
updated) which we think caused corruption when the replacement disk was
brought online.
We've experimented to make sure that the new firmware does not duplicate
this problem and, as far as we can tell, we are ok from now on, so this
problem should not reoccur.
Restoral information:

The CVS repository was restored to its state about ~2 hours before the
system was brought down on Thursday at around 1PM EST (18:00 GMT).  The
other volumes were restored from backups that were less than 24 hours
old.
After the CVS volume was restored, Ian Taylor added any missing checkins
to the gcc repository.  Other repositories reflect the last backup state.
So, it is possible that some repositories may be in an odd state now
with the data on a user's disk appearing to be newer than what is in
CVS.
So we did lose some data.  It may be noticeable in the web pages or it
is possible that we lost some subscription information so that someone
who subscribed to a mailing list may have to resubscribe.  If something
was transferred to ftp, it may have to be transferred again.
htdig is down and may be down and out.  There is an ominous internal
error now if you attempt to search.  I'll fix that tomorrow (unless
someone beats me to it).  The fate of htdig is still in question,
however.  It hasn't been running right lately, no one wants to
maintain it, and it may not be the best search solution.
I hope that Angela and Ian will respond to this message with any
information that I missed.
Kudos:

Matt Galgoci was the man onsite who got everything working after the
hard drive and subsequent RAID firmware problems.  We'd be totally dead
in the water if we didn't have someone like him available to help out.
The free software community owes a huge debt of thanks to Angela Thomas
for 1) backing up the system so regularly and so reliably and 2)
spending countless hours in the last several days transferring the
backups, commiserating on the best way to get the system up and running,
and generally doing whatever it took to get the system up.
Ian Taylor also provided his usual services, making sure that
qmail was working ok and providing general guiding advice.
And a BIG thanks to Daniel Berlin.  His knowledge of RAID, LVM,
mysql, and just general technical expertise were invaluable.  He
stopped us from panicking when the system came back up with
what appeared to be missing logical volume information by
providing us with the right commands to do to restore things
to a sane state.
And now I'm going to sleep.

cgf

GDB CVS ok; was CVS outage

[Andrew Cagney]

FYI,

GDB's CVS repository looks ok, thanks!

Andrew

---- Original "back up" e-mail posted to overseers@ ----

From: Chris Faylor

We're back online.

As most of you know, we suffered a pretty severe hardware outage on
Thursday.  The best theory right now is that a bad hard drive in the
RAID array indirectly caused a data corruption problem.  The problem was
really due to a problem with old firmware (which mgalgoci has since
updated) which we think caused corruption when the replacement disk was
brought online.

We've experimented to make sure that the new firmware does not duplicate
this problem and, as far as we can tell, we are ok from now on, so this
problem should not reoccur.

Restoral information:

The CVS repository was restored to its state about ~2 hours before the
system was brought down on Thursday at around 1PM EST (18:00 GMT).  The
other volumes were restored from backups that were less than 24 hours
old.

After the CVS volume was restored, Ian Taylor added any missing checkins
to the gcc repository.  Other repositories reflect the last backup state.
So, it is possible that some repositories may be in an odd state now
with the data on a user's disk appearing to be newer than what is in
CVS.

So we did lose some data.  It may be noticeable in the web pages or it
is possible that we lost some subscription information so that someone
who subscribed to a mailing list may have to resubscribe.  If something
was transferred to ftp, it may have to be transferred again.

htdig is down and may be down and out.  There is an ominous internal
error now if you attempt to search.  I'll fix that tomorrow (unless
someone beats me to it).  The fate of htdig is still in question,
however.  It hasn't been running right lately, no one wants to
maintain it, and it may not be the best search solution.

I hope that Angela and Ian will respond to this message with any
information that I missed.

Kudos:

Matt Galgoci was the man onsite who got everything working after the
hard drive and subsequent RAID firmware problems.  We'd be totally dead
in the water if we didn't have someone like him available to help out.

The free software community owes a huge debt of thanks to Angela Thomas
for 1) backing up the system so regularly and so reliably and 2)
spending countless hours in the last several days transferring the
backups, commiserating on the best way to get the system up and running,
and generally doing whatever it took to get the system up.

Ian Taylor also provided his usual services, making sure that
qmail was working ok and providing general guiding advice.

And a BIG thanks to Daniel Berlin.  His knowledge of RAID, LVM,
mysql, and just general technical expertise were invaluable.  He
stopped us from panicking when the system came back up with
what appeared to be missing logical volume information by
providing us with the right commands to do to restore things
to a sane state.

And now I'm going to sleep.

cgf

Re: GDB CVS ok; was CVS outage

[Christopher Faylor]

On Sun, Feb 06, 2005 at 11:18:58AM -0500, Andrew Cagney wrote:
>FYI,
>
>GDB's CVS repository looks ok, thanks!

Thanks.  I should have mentioned that we were pretty sure that gdb's cvs
was ok.  I track the gdb-cvs mailing list so we could tell that it was
restored properly.

But getting independent verification is a relief.

cgf