From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cagney To: gdb-patches@sources.redhat.com Cc: overseers@sources.redhat.com Subject: GDB CVS ok; was CVS outage Date: Sun, 06 Feb 2005 16:20:00 -0000 Message-id: <42064372.3090708@gnu.org> X-SW-Source: 2005-02/msg00011.html FYI, GDB's CVS repository looks ok, thanks! Andrew ---- Original "back up" e-mail posted to overseers@ ---- From: Chris Faylor We're back online. As most of you know, we suffered a pretty severe hardware outage on Thursday. The best theory right now is that a bad hard drive in the RAID array indirectly caused a data corruption problem. The problem was really due to a problem with old firmware (which mgalgoci has since updated) which we think caused corruption when the replacement disk was brought online. We've experimented to make sure that the new firmware does not duplicate this problem and, as far as we can tell, we are ok from now on, so this problem should not reoccur. Restoral information: The CVS repository was restored to its state about ~2 hours before the system was brought down on Thursday at around 1PM EST (18:00 GMT). The other volumes were restored from backups that were less than 24 hours old. After the CVS volume was restored, Ian Taylor added any missing checkins to the gcc repository. Other repositories reflect the last backup state. So, it is possible that some repositories may be in an odd state now with the data on a user's disk appearing to be newer than what is in CVS. So we did lose some data. It may be noticeable in the web pages or it is possible that we lost some subscription information so that someone who subscribed to a mailing list may have to resubscribe. If something was transferred to ftp, it may have to be transferred again. htdig is down and may be down and out. There is an ominous internal error now if you attempt to search. I'll fix that tomorrow (unless someone beats me to it). The fate of htdig is still in question, however. It hasn't been running right lately, no one wants to maintain it, and it may not be the best search solution. I hope that Angela and Ian will respond to this message with any information that I missed. Kudos: Matt Galgoci was the man onsite who got everything working after the hard drive and subsequent RAID firmware problems. We'd be totally dead in the water if we didn't have someone like him available to help out. The free software community owes a huge debt of thanks to Angela Thomas for 1) backing up the system so regularly and so reliably and 2) spending countless hours in the last several days transferring the backups, commiserating on the best way to get the system up and running, and generally doing whatever it took to get the system up. Ian Taylor also provided his usual services, making sure that qmail was working ok and providing general guiding advice. And a BIG thanks to Daniel Berlin. His knowledge of RAID, LVM, mysql, and just general technical expertise were invaluable. He stopped us from panicking when the system came back up with what appeared to be missing logical volume information by providing us with the right commands to do to restore things to a sane state. And now I'm going to sleep. cgf From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 4536 invoked by alias); 6 Feb 2005 16:20:00 -0000 Mailing-List: contact gdb-patches-help@sources.redhat.com; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sources.redhat.com Received: (qmail 3769 invoked from network); 6 Feb 2005 16:19:39 -0000 Received: from unknown (HELO mx1.redhat.com) (66.187.233.31) by sourceware.org with SMTP; 6 Feb 2005 16:19:39 -0000 Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by mx1.redhat.com (8.12.11/8.12.11) with ESMTP id j16GJduR003225; Sun, 6 Feb 2005 11:19:39 -0500 Received: from localhost.redhat.com (vpn50-74.rdu.redhat.com [172.16.50.74]) by int-mx1.corp.redhat.com (8.11.6/8.11.6) with ESMTP id j16GJcO24211; Sun, 6 Feb 2005 11:19:38 -0500 Received: from [127.0.0.1] (localhost.localdomain [127.0.0.1]) by localhost.redhat.com (Postfix) with ESMTP id EDBA07D79; Sun, 6 Feb 2005 11:19:00 -0500 (EST) Message-ID: <42064372.3090708@gnu.org> Date: Mon, 07 Feb 2005 00:12:00 -0000 From: Andrew Cagney User-Agent: Mozilla Thunderbird 0.8 (X11/20041020) MIME-Version: 1.0 To: gdb-patches@sources.redhat.com Cc: overseers@sources.redhat.com Subject: GDB CVS ok; was CVS outage Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-SW-Source: 2005-02/txt/msg00014.txt.bz2 Message-ID: <20050207001200.6yhTxW1xIsAOu921zScRjldEyfCiKJvn9DOBpMv0M-I@z> FYI, GDB's CVS repository looks ok, thanks! Andrew ---- Original "back up" e-mail posted to overseers@ ---- From: Chris Faylor We're back online. As most of you know, we suffered a pretty severe hardware outage on Thursday. The best theory right now is that a bad hard drive in the RAID array indirectly caused a data corruption problem. The problem was really due to a problem with old firmware (which mgalgoci has since updated) which we think caused corruption when the replacement disk was brought online. We've experimented to make sure that the new firmware does not duplicate this problem and, as far as we can tell, we are ok from now on, so this problem should not reoccur. Restoral information: The CVS repository was restored to its state about ~2 hours before the system was brought down on Thursday at around 1PM EST (18:00 GMT). The other volumes were restored from backups that were less than 24 hours old. After the CVS volume was restored, Ian Taylor added any missing checkins to the gcc repository. Other repositories reflect the last backup state. So, it is possible that some repositories may be in an odd state now with the data on a user's disk appearing to be newer than what is in CVS. So we did lose some data. It may be noticeable in the web pages or it is possible that we lost some subscription information so that someone who subscribed to a mailing list may have to resubscribe. If something was transferred to ftp, it may have to be transferred again. htdig is down and may be down and out. There is an ominous internal error now if you attempt to search. I'll fix that tomorrow (unless someone beats me to it). The fate of htdig is still in question, however. It hasn't been running right lately, no one wants to maintain it, and it may not be the best search solution. I hope that Angela and Ian will respond to this message with any information that I missed. Kudos: Matt Galgoci was the man onsite who got everything working after the hard drive and subsequent RAID firmware problems. We'd be totally dead in the water if we didn't have someone like him available to help out. The free software community owes a huge debt of thanks to Angela Thomas for 1) backing up the system so regularly and so reliably and 2) spending countless hours in the last several days transferring the backups, commiserating on the best way to get the system up and running, and generally doing whatever it took to get the system up. Ian Taylor also provided his usual services, making sure that qmail was working ok and providing general guiding advice. And a BIG thanks to Daniel Berlin. His knowledge of RAID, LVM, mysql, and just general technical expertise were invaluable. He stopped us from panicking when the system came back up with what appeared to be missing logical volume information by providing us with the right commands to do to restore things to a sane state. And now I'm going to sleep. cgf