DDO unavailable: Saturday March 30th

[Solarpower]

This image was (allegedly) smuggled from the datacenter.

This one is more accurate. Guess we all know what've happened here...

 

[Justfungus]

ok, that is different, it is at least trying to update now ... it fails and times out ... but it is new at least.

 

[Krell]

If they've outsourced their DC then a DR plan would be expected of the supplier. Or somebody hasn't procured right.

Yeah but what recovery SLA did they pay for

Ahh, here it is again... the "poor, poor little indie studio SSG" card. That sympathy card has some serious, unwarranted miles on it.

Pro-tip - They're a multi-million dollar company (not even counting the LoTRO revenue) who's literal entire business is selling 1s and 0s.
If they haven't "justified" having a proper offsite failover plan then they have essentially "decided" they don't want to stay in business if/when they encounter a true DR scenario.


IT history is littered with the corpses of hosting companies who suddenly shutdown / failed virtually overnight.

Trusting your hosting provider to also be your DR site/planner is putting all of your eggs in one basket. Not a good idea.

Redundancy is one of the core tenets of DR planning. Diversity (in your service providers) is one of the core tenets of redundancy.

multi-million profit margins? I doubt it. It doesn’t matter what your gross income is. Can they afford high availability infrastructure? We don’t know. Clearly they don’t have it now. This isn’t the first multi-day outage. They seem willing to accept the risk. Next time they have data center issues, it’s likely we’ll be down for a couple days again.

 

[Xaerxiessia]

non valid credentials with auto-login script (skipping updates)

 

[Hipparan]

I have to laugh at all of you who are all in on the "a good data center is foolproof! They must have cheaped out!" Two real world examples, both Microsoft Azure: One time they were doing cleanup and had a script to delete old SQL VM instance backups. Unfortunately the script was written wrong on instead of deleting "old backups" it deleted the live customer VMs. Every SQL instance running on the whole East Coast node went poof. Oopsie. That was a three hour outage for my company. Another real case, they somehow had a whole rack of management servers that was totally outside all the fault tolerance mechanisms. No backups, no secondary power supply, not even scripted into the VM management systems. And then they had a power failure, and all the actual customer machines were fine, they failed over fine, but this rack of network routers, VM managers, etc went down hard. And when it got back on power and rebooted, there was nothing to tell the master VM manager that these servers were supposed to be core infrastructure, and it just saw a bunch of high power servers and started spinning up high price tier customer VMs on it, overwriting all the critical VMs that weren't backed up. That one didn't hit my company, but it was a 24+ outage for a lot of Azure customers.

My company hosts 24/7 services and we had a nice local data center we were using. Well the data center got bought out by a larger company and they moved our servers over to their "new high tech" data center. One weekend suddenly all of our servers went down. Data center had 0 communication and we were down for 8 hours. Turns out that the OS on their switches had an update that introduced a really nasty bug that brought everything down. The data center didn't tell us until after it happened a second time the next weekend. Even then it was a couple of weeks of silence before we got any word as to what happened. Needless to say we aren't using them anymore.

 

[Frantik]

I tried logging to see if anything had changed... it tries, and for the moment fails, to connect to "patch server". Maybe a little light at the end of the tunnel? We shall see...

 

[Solarpower]

Maybe a little light at the end of the tunnel?

That's just a train. Don't have you hopes up and, please, get off the rails !

 

[Frantik]

That's just a train. Don't have you hopes up and, please, get off the rails !

No trains in my tunnel! No rails. But perhaps my eyes playing tricks on me?

 

[Grimstad]

It is now attempting to connect, but getting a failure error

I tried logging to see if anything had changed... it tries, and for the moment fails, to connect to "patch server". Maybe a little light at the end of the tunnel? We shall see...

Same here

 

[Kimbere]

Yeah but what recovery SLA did they pay for

multi-million profit margins? I doubt it. It doesn’t matter what your gross income is. Can they afford high availability infrastructure? We don’t know. Clearly they don’t have it now. This isn’t the first multi-day outage. They seem willing to accept the risk. Next time they have data center issues, it’s likely we’ll be down for a couple days again.

Nobody said multi-million dollar margins. It does matter what your gross revenue is though. It matters a lot.

For example, if proper backups, DR planning, redundancy, etc. cost $250k/year and, as a company, your gross revenue is only $500k/year, DR would be 50% of your gross revenue. In that scenario, you likely can't afford it.

If however, if your company's gross revenue is say $10m/year and proper DR is $250k/year, that's 2.5% of your gross revenue. That's a no-brainer, especially in an industry where your product is entirely digital.

If you can't devote 2-3% of your gross revenue to a DR plan to ensure business continuity in the event of a serious disaster, you've got serious management/executive board level issues.

 

[Dandonk]

To sleep, perchance to dream.... that DDO will be up tomorrow...

 

[Justfungus]

Elf Artificer - goes into a 6 hour lecture on possible causes of the games failure.
Half Orc Barbarian - "It Broken."

 

[Jeeboheebo]

*runs into the thread* THE SERVERS ARE DOWN!! *runs a bit around* "THE SERVERS ARE DOWN"

 

[SylvanPhoenix]

If you can't devote 2-3% of your gross revenue to a DR plan to ensure business continuity in the event of a serious disaster, you've got serious management/executive board level issues.

What's interesting is in this cross-post, Cordo mentions despite their redundancies they were able to get it working. That suggests they did indeed have a DR plan but perhaps it hadn't been gameday'd any time recently and ended up not working out.

 

[Frantik]

Chill time!

 

[Dementedleo1974]

This was reason I was debating on paying for the VIP. I love the game, but if I cant play it and all, why should I continue to pay for it? This is a travisty to recruit people to pay a sub fee and excpect good servers at least . Just imagine if WOW , FFXIV, or GW2 went down this long with no timeline or anything.......

 

[Myosin]

When your companies entirely livelihood depends on the games being operational

Most companies of significant size are run by people who depend on their diverse stock portfolios - not on the company performance. Only wage/salary workers depend on the company staying afloat.

 

[Urklore]

ooo Folks are flaming DDO Twitter and Facebook because a pre-programmed social media add just got posted to each channel about dragon lord being free to VIP and this week's bonus weekend but the game has been down for almost two days.

 

[Solarpower]

Just imagine if WOW , FFXIV, or GW2 went down this long with no timeline or anything.......

"Imagine" he said...
Why do we have to imagine it if we literally have to wait 2-3 days after the launch to play any new Blizzard game/wow-expansion ? Every time that small indie Blizzard company didn't expect that much players who have crushed the servers just with their numbers.

 

[GrizzlyOso]

OK, I've been severely over productive this weekend at this point.