Well, this is a bit of a long story. You’ll just have to bear with me.
On Friday, March 25, 2016, in the evening, a hard drive failed in the hosting company’s server. This took down TFP, TFP Forums, and many other sites which were hosted on that hard drive, in that server. When they attempted to replace the bad hard drive, the new drive was DOA and did severe damage to the RAID array causing a lot of websites to get corrupted, including part of mine. Unfortunately, the backup server also failed at the same time and its data was lost. We had no up to date external backups because that system failed silently and nobody knew about it. We will no longer be using that system.
So, this left me with a non-functional website and a damaged (due to partial data loss) database. Over the past week, I worked with the hosting company and the TFP website team and we did our best to recover as much data as we could and resurrect this site on a new server. We were mostly successful. A few records went corrupt and we had an entire table go bad.
Now that we’re back up, here is what to keep in mind:
-The unread posts table got totally corrupted, so all posts will be marked as unread. You can go ahead and click on “Mark forums as read”.
-The users table was damaged, we lost over a thousand users so many may need to re-register. Please try to log in first! Sorry for the inconvenience if you were affected.
-There may be some missing posts here and there, or some missing topics. Remember this was a partially corrupt database we recovered from.
Thank you everyone for being patient as we got through the past week.
Based solely on the users who recently posted in the Fire Alarm General Discussion, it appears Lambda, Firefly, electronicwiz101, 7002t, Rob Goebel, firealarmgeek, and NewEnglandElevators at least are missing.
I would also be interested in a list of missing users, if it is possible and Admin is willing to disclose the information.
I would as well. Congratulations for getting the board back online!
It’s a real shame TFP has had to go through crap like this, but I’m glad we didn’t lose the entire board.
Hey everybody. It’s great to see the forums back in action. I couldn’t log in under my old user credentials, so I went ahead and re-registered. My old created topics came back with SQL errors, and I was unsuccessful in finding any of my other postings. (The “Introduce Yourself” topic is one example, when I originally announced my return to the forums.)
He was the first one I noticed missing. His expertise is a major asset to the forum, as well as that of the other techs and experienced collectors. Hope he reregisters!
We really tried our best to recover as much as possible but some things were so damaged there was really nothing more we could do. Again sorry for the inconvenience. If you remember how many posts you were at before the system went down, I might be able to modify the database to look like you have a lot of posts even though you do not.
No, it’s fine (although I remember having 3 1/2 - 4 stars). I’m just upset (who isn’t?) that any of this happened in the first place. Sigh, at least we recovered as much as we did, so we’re really lucky in that sense.
Also, off topic, the typo “dont” is driving my “ADHD-induced OCD” crazy. I know it’s crazy, but that’s me.
Most of my information survived. My registration was still there and works. A quick search revealed that my posts seem to be there. The only anomaly I see is in PMs. My inbox shows 54% full but there are no messages. Same with sent messages. I searched for this problem and found not much help. People who had full (or nearly) message boxes may need to have their message limit increased. That was the workaround that showed up.
Oh yeah, congratulations on recovering all that could be.
Retired STR-SG, I’ve increased your private message limit. If anyone else is having this problem, just let me know and I’ll do the same.
Also, if anyone who re-registers was a technician or retired staff, let me know so I can change your rank. I’ve been watching the list of newly-registered users, but feel free to PM me if I missed you.
What a mess, good job at getting what’s left back up though. With all of the errors, would it be a good idea to just archive the current forum at some point? Or convert from phpBB to another software and maybe the issues will get ironed out in the process?
just throwing out some possibilities, not trying to create even more work for you guys! good luck getting all of this sorted, looks like there’s plenty left to be done.
From the description of what happened I don’t see this as a phpBB failure. Probably any other forum system software would not have survived any better. This looks like a series of hardware failures that are “not suppose to happen.” The description did not say what RAID system it is. If it is a RAID 5 the redundancy can rebuild the replacement of a single failed drive. There was a bad out of the box replacement drive, that should not happen, which corrupted the system. Then what are the chances of a separate backup server failing? Obviously that Murphy guy was at work. I run a web site that has used a phpBB forum since 2005. Our host has had failures but we have never had any data loss. I can only hope that continues.
Yea but there were database errors leading up to it to the hardware failure (hindsight being 20/20, probably indications of the drive failing), and now it seems to be a spaghetti mess. not sure how you’d fix it, but having a database full of errors seems like asking for trouble. not sure to fix it, but putting it into archive mode and starting over (pull over what’s left of the user database) might be a decent route to take.
