Hopefully it's as easy as that. Otherwise I begin to wonder how long EG7 will let it go on.
Cormyr is an epic fail as far as lag goes :(
- Thread starter Pom
- Start date
Blaster
Well-known member
In their 2023 presentation EG7 stated their intention to shift their business model away from non-first-party live service games like DDO to franchising first party games like Everquest, Mech Warrior, My Singing Monsters, the recently acquired Palia, and whatever Cold Iron Studios is working on next. (In the presentation they used the Call of Duty franchise model as example of what they want to start doing.)
EG7 likes that their live service games generate predictable income, and that includes DDO. As long as it sits in that pocket of predictability and doesn't sway too far below expectations I imagine EG7 will continue to do what they've stated in the past they will do: invest enough money to add new content (which usually spikes revenue upwards) and keep the games running until the first-party franchise model takes over as their biggest revenue stream.
EG7 likes that their live service games generate predictable income, and that includes DDO. As long as it sits in that pocket of predictability and doesn't sway too far below expectations I imagine EG7 will continue to do what they've stated in the past they will do: invest enough money to add new content (which usually spikes revenue upwards) and keep the games running until the first-party franchise model takes over as their biggest revenue stream.
The lag went up by minutes when myth drannor was released. I like that expansion, but it is obvious that serious performance issues were introduced at that moment.
64 bit systems can contain a lot of memory in comparison with 32 bit. These servers should have worked on 64 bit 20 years ago. But what it doesn’t solve is performance issues in code. This is what we are seeing now on every server … bad / slow code … and better hardware can’t solve this.
Since this seems hard for non-tech people, these should be major priorities besides what marketing tells you…if you want to survive 2030 as a company:
1) Fix the performance bugs and the showstoppers and get a descent QA, its killing your company on bugs/user experience
2) Get on with 64bit and get rid of
stone-aged hardware, from there on you can start thinking forward (tech, not content). If you don’t, you WILL kill your company on tech debt. It is obvious you have a huge amount already and it stops you from doing anything constructive.
3) Split the system in microservices, seperate the database, automatically spin up more resources when needed and spin down when less is necessary. Its economically better and user experience will be better.
4) graphics engine could use some love
Signed
A Software architect with 24 years of experience who is also responsible for security in a major software company
64 bit systems can contain a lot of memory in comparison with 32 bit. These servers should have worked on 64 bit 20 years ago. But what it doesn’t solve is performance issues in code. This is what we are seeing now on every server … bad / slow code … and better hardware can’t solve this.
Since this seems hard for non-tech people, these should be major priorities besides what marketing tells you…if you want to survive 2030 as a company:
1) Fix the performance bugs and the showstoppers and get a descent QA, its killing your company on bugs/user experience
2) Get on with 64bit and get rid of
stone-aged hardware, from there on you can start thinking forward (tech, not content). If you don’t, you WILL kill your company on tech debt. It is obvious you have a huge amount already and it stops you from doing anything constructive.
3) Split the system in microservices, seperate the database, automatically spin up more resources when needed and spin down when less is necessary. Its economically better and user experience will be better.
4) graphics engine could use some love
Signed
A Software architect with 24 years of experience who is also responsible for security in a major software company
Rusty_helmet
Teh_troll’s Fluffer
Yeah actually listening to the playerbase would be a great start. But they won't do that.
Rusty_helmet
Teh_troll’s Fluffer
Optimization is hard when you don't know what you are doing on a home grown engine, so you can't exactly go get someone that immediately knows what they are doing.
Bitss
VIP since 2006
How many more have they introduced? I have lag wiped 6-8 years ago, so this is nothing new.
Expansions are what players want, if you saw the heat posters get when suggesting otherwise. That's what makes money, the saying goes.
If they're bleeding players so badly, seems to me fiscally responsible would be to start selling FIXPAKS.
In old software with a lot of functionality and a ‘severely customized’ framework, the speedup time of a new dev can take up to 6 months until they are worth what you pay them. Or even up to a year until they can try the harder parts.
But I suppose not everyone started working there in 2024, and they still know what they changed last spring which pushed ddo into unplayability.
And then there is the part of technical debt, you can get away some time ignoring it, but it is obviously neglected for way too long. This tends to make change harder as it is accumulated until the point where it is nearly impossible to do something without breaking something else.
Guess where they are if they are now figuring out how to implement 64bit which can permit caching (not yet, they just talked about it)
That software runs obviously on 1 physical server together with its database and everything it needs.
This is called a monolith, and the larger it gets, the more complex, the harder it becomes to maintain and to change something.
Another symptom is the weekly reboot that takes hours.
This looks in my book as serious memory leaks, and these are very hard to find in a huge monolith.
Even worse, what happens in that time besides rebooting? A physical server boots up in less than 5 minutes if it runs a lot of software.
What goes wrong if they need another extention of yet another few hours?
I would start to suspect them from backupping the database, re-installing that server (maybe from a template or an image), restoring the database, and running some basic tests.
This would take hours. This sounds ridiculous but realistic from what we see and hear. And is also the perfect example how technical debt slowly kills software until handled properly.
With microservices running on a few virtual servers (add and remove when necessary), there is only downtime when an upgrade is rolled out and even then it is only a few minutes. And should you have a memoryleak, it is in that part of your system that keeps growing in memory usage, that is a start to find it.
This means splitting up the monolith in small specialized processes that are loosely linked to eachother and are not necessary on the same virtual server.
I vaguely remember that the Cold Iron Studios project was something EQ related. That would be consistent with building on IP they own, but it wasn't a solid "we are doing this" statement. It was more of a "we're exploring the idea" statement.
Last edited:
Eoin-1
Apply the North Wind visual effect...
??? Myth Drannor released in the middle of Q3 (July, August, and September). The pre-release sale started in the middle of June... so like 2 weeks of sales to people who buy before even knowing what the quests would be like (June 26th was when we got to see the 1st 3 quests).
Note: On the the 12th EG7 will be talking about July, August, and September. So those numbers won't reflect the damage caused by the crippling lag or Cormyr or the lantern event or the additional whales harpooned (new coffer). Q4 (October, November, and December) will be discussed on February the 11th.
MD released in June and didn't help their performance for Q2 according to their own investor relations report.
"Putting aside the impact from the revenue recognition timing, the performance for the quarter still
came in short of expectations with DCUO, DDO, and PS2 underperforming. DCUO’s new Episode 47:
Brainiac Returns did not reach the anticipated level of player engagement. DDO saw an uptick with
the new expansion in June, but not enough to offset the slower start for the period."
GrizzlyOso
Well-known member
Let’s see what else happened… a vip program that was widely lambasted a year before it came out and actually CAUSED people to STOP vip instead of the stated cause of keeping vip ?
Maybe they should acknowledge what actually happened instead of “players didn’t buy MD, it’s probably the players”.
Eoin-1
Apply the North Wind visual effect...
So 18 days of pre-order didn't make up for the other 73 days of poor performance... Color me surprised. Seeing an uptick at the end of the quarter because of the sale of a race, an iconic, an archetype, racial AP, and cosmetics means Myth Drannor was a failure... 44 days of pre-order and 48 of actually being able to play it Q3... nah, not as important.
Not saying saying the Q3 numbers will be pretty, just seems odd to me that people put so much emphasis in MD being Q2 when the pre-order was only live for a little under 20% of the time. Heck, if I didn't have racial completionist and wanted to keep it, I wouldn't have bought the cheapest option until it went live in the middle of August.
Rusty_helmet
Teh_troll’s Fluffer
Easy to say very hard to do. Most systems just start from the ground up to do that. They have neither the manpower nor the knowhow to do that, so here we are, hoping that a 64 bit server will help this game limp along for another 4-5 years.
Rusty_helmet
Teh_troll’s Fluffer
Their best bet would be to put out a bounty for a third party to come in and analyze their code and find the leak(s). Because as this game uses more and more memory with each update, the functional drop in playability continues to increase. Many of the leaks are very old, and I think they hoped that the 64 bit server would shed light on a lot of them. Clearly it has not been effective enough.
Kinda like a death spiral. DDO under preforms so gets less attention, so it under preforms more so, so again less attention. Maybe if they put more time and money to DDO trying to fix the problems, they'd get more people to stay and maybe even pay for it.
JustHavingFunBro
Well-known member
They're too busy getting rid of devs trying to save a buck. Its going to cost them the entire game.
Where did I ever say this was easy?
I did say however that they don’t have a choice. And that this is a huge work. And yes we (our company) did such things before, we are talking years, not weeks.
But once software has that much functionality it is never easier or faster to recreate it from scratch.
In the contrary, because you need that much just to launch (people dont accept that you can do only 5% of what you could do before), that ship will be entered and sunk long before you reach critical mass.
The only way to get rid of tech debt is starting to work on it
Good thing is that new content is not hampered by this job, its an entirely other specialty
And as a last hint, if you want to find memoryleaks in a huge monolith. 1) Start using valgrind. 2) Create a wrapper class that requests and returns memory, extra identifier from the caller is mandatory. Find and replace all with that wrapper class (again huge work, but there is no free lunch)
Make a start & stop control that checks everything coming in and out. Start ,,, do some actions, close down again… Stop … result should be ‘no difference’
If it isn’t, you can pinpoint it on the id.