Website and Forum Outages Explained

[Phil Coyle]

Over the past few weeks you have likely noticed intermittent outages of our website and forums. It is important to BVA Administrators that you, as members, are aware of why these outages have occurred and what we are doing to prevent something similar from happening again.

 

For those of you on a short-haul flight, the summary version is:

• We were informed by our ISP that the way the back-end of our website was structured was going to result in a substantial cost increase

• Despite several attempts, we were unable to negotiate to reduce the cost increase (we were, however, able to negotiate an extension of the date on which this cost increase would occur)

• Despite being very happy with the service and reliability of our current ISP, we recognized the importance of cost to our community and attempted to migrate our service to another ISP

• In the process, and despite a substantial amount of research, we determined that the new ISP simply wouldn’t be able to meet the high standards we have come to expect

• We migrated our website back to our original host and re-designed our website’s back-end to comply with our original ISP’s standards

• The website and forums are now back online and operating normally

• We did our best to keep the community informed of all changes, being sensitive to the need for confidentiality while we worked with multiple ISPs

• This entire effort cost the community $0.01

 

We appreciate the many messages of support we received over the past few weeks as well as emails letting us know when the site was offline. While we often had intentionally brought the site offline to work on it, there were times where a user notified us of an unexpected issue that we were then able to solve.

 

Above all, we appreciate the patience and understanding you have shown during the disruptions to our website and forums.

 

If you’re mid-way through a long-haul flight, below are a few more details. As always, feel free to respond to this post or e-mail us with and questions, comments, or suggestions.

 

----------------------------------------

 

As you know, BVA is a community of volunteers supported by donations. None of our administrators, including those who were involved in the work we’ve been doing on the website, receive any monetary compensation for their efforts.

BVA treats the money that you as members donate to the community more carefully than if it was our own. We determined that paying more money to our ISP for no corresponding increase in service when there were other alternatives available was not an acceptable option.

 

That’s why we decided that—despite the effort involved—migrating the website to a different ISP would be the best course of action. We carefully evaluated our alternatives and in our research found out that BVA’s website needs are overly complex for the options offered by most ISPs. While there are hundreds of ISP choices, our community’s needs limit those options to just a few providers.

 

BVA’s website and forums run on different database-driven application systems. We regularly email our community (now larger than 3,500 members) with event reminders and updates. We run several complex applications from the site. We have many administrators who have cascading levels of access throughout our website. Normally, constraints like these would force BVA to pay for a “dedicated server” to run our website. (This is the server type we use to host FSX, and it costs us about $2,200 per year. By comparison, we pay our web ISP less than $100/year for the hosting we receive now.) There are just a few ISPs whose policies align with our needs.

 

We also considered hosting our website on the same server that we use to host FSX. We decided that was not the best option from the perspective of security (better not to have everything stored/hosted in one place), efficiency (we want to save the computing power and bandwidth of our FSX server for FSX), and availability (we restart the FSX server almost daily in order to ensure our users have a stable connection and the website would be unavailable during these restarts).

The alternative ISP we found appeared to allow us to do everything we needed to, with the exception of sending mass emails—that would require an add-on module (for a one-time cost of about $100). We researched policies and conducted a few calls to confirm that everything appeared to be in order. Based on our research, the A-Team determined that migrating to the alternative ISP was the most cost efficient decision. We also determined it would minimize the amount of disruption to our services because we could do most of the work “behind the scenes”.

 

While the migration itself was smooth and involved minimal downtime, our work with the alternative ISP gave us a new perspective on how lucky we are to receive the excellent support and customer service we do from our current provider. It quickly became apparent that, despite being one of the largest hosting companies (hosting over 9 million domains), support with this alternative provider was poor. For example, the ISP wouldn’t make an exception in its security policy for the add-on module we needed to install to send emails to our community—nor provide us with any other solution. In another instance, we asked support to make a simple change (to enable custom “404 error” pages to work) because the control panel they provided to us wasn’t working as it should have. In working on the issue, the ISP caused our website to go down, did not get it back online for over 12 hours, didn’t solve the original problem, and tried to tack $5/month on to our hosting plans for a service we weren’t actually using.

 

This, and several other incidents, led us to believe that the alternative ISP simply wouldn’t be able accommodate our needs without consistent outages and a lot of follow-up effort on our part. I’m sure that company is capable of doing excellent work but, as I mentioned, BVA is a special case. We need to fit our complicated needs into a simple (and cheap) hosting structure and not everyone can do that.

 

Consequently, we made the determination that the only way to move forward was to re-design our website’s back-end functionality to meet the requirements of our current ISP. Once again, we did a lot of the work behind the scenes and at what I’ll call “non-peak periods” (read: not a lot of sleep was had) to minimize disruption to service. We also intentionally moved slowly, making sure to read the “fine print” on anything we purchased to ensure that we could recover all of our costs if we needed to reverse course. That’s why the entire affair—which taught us a lot about our website, allowed us to perform some needed changes to the website’s functionality, and reminded us of the value of our current ISP—cost the community just $0.01 (already reflected in our budget).

 

Our website, forums, and other applications should all be online and working properly. If you click on a link and see an image of an FMS that says “this page’s position can’t be found”, please let us know—it means the link is broken and we need to fix it. Similarly, if you have any bookmarks to old website pages with a “dnn6” in them, please update those bookmarks to the new pages.

 

404Error.png[/attachment:galnumk0]

As you know, a team of BVA’s administrators has been working hard over the past few weeks to make all this happen. Their efforts and dedication of time should not be overlooked. Bill, Brandon, Zac, and others have all been involved over this period. We as members owe them our thanks for their tireless efforts to ensure that we have the best possible service at the lowest possible cost. As Bill has said, it’s probably the most work any group of people have done for a penny.

[Jeremy Valentine]

Thanks again you guys! The work you all accomplish behind the scenes is consistently above and beyond all expectations.