Hardware and Hosting - What every online game developer should know
George Dolbier is a Senior Technology Architect currently working for IBM. He has been working in the interactive entertainment industry for nearly 10 years, primarily as a software engineer or in engineering management.
Most organizations embarking on a new online game tend to focus on minimizing the upfront capital cost of the equipment. But, keep this in mind; the first year's hosting costs alone could be double your initial capital outlay. The architecture of your online engine and the technology it runs on can and will significantly impact hosting costs. You could find yourself in the situation of saving a few bucks on the servers only to find out that their power consumption, or heat dissipation, for example, result in increased hosting costs.
The design team does a beautiful job
Online games, whether they are simple 'casual' games, or massively multiplayer persistent worlds, are based around three types of technology: Servers, Storage and Networking. Most companies can hire a team of people with experience architecting and implementing an online game. That team can spend weeks analyzing the latest offerings from a pantheon of vendors. The result will be a system architecture and design for your title. Your team will analyze every detail: bandwidth and latency required to support 'x' number of clients, and the number of nodes required will be debated repeatedly. Finally your completed system architecture is a masterpiece, it's going to rock. Everything is set - all you need to do is hook it up to the Internet, right?
Now you have to find a place to house your masterpiece. Lots of people do that right? Broadband is cheap these days, every local ISP and Telco provider seems to offer hosting services. Power is cheap, and all you really need is a room, how hard can it be?
But beauty hides something ugly
So, you do a Google search, find some local hosting providers and ask them for un-official quotes. You give them your bandwidth requirements; tell them how many systems make up your server farm. All at once, reality gives you a dope slap upside the head. The budgetary quotes make you see flickering lights, and the vultures, circling the soon-to be corpse that was your title, cause the flicker. Your title's operational budget has suffered a blow out even Firestone would admit to. You need to find a place to house your masterpiece, but you had no idea that the bandwidth cost alone would be more than your entire operational budget - and the Telco wants you to sign a 3 year contract!
A brief introduction to hosting economics
Why is hosting an MMOG so expensive? (hey, it's only one of the most complex technological endeavors undertaken by human kind, what's the deal?)
So when you called Joe Telco the rep was real friendly and you discussed the details at le espensive swankve. After a nice dinner, your rep picks up the check, and hands you an itemized "unofficial, back of the napkin" quote…. I hope you had seafood.
Stuffing a bunch of computers into a closet and wiring them up isn't that expensive right? WRONG. The technology in a datacenter is admittedly pretty simple, but the problem is that a datacenter is where computing, real world physics and economics all come together.
The facility itself is a highly secure building, housing highly specialized rooms that require immense electrical and cooling capacity, as well as custom fire suppression equipment and a whole host of other features. Without going into too much detail a co-location provider's costs break down basically as follows: Rent, Power, Network Cooling and Miscellaneous.
Rent
Why the big deal about rent? As you can expect the room that is going to house your online masterpiece is special. Machine rooms often have raised floors, overhead cable treys, plumbing for water cooling, and the chillers, chemical fire suppression system, high security physical security centralized environmental monitoring and management.
All of this, like any bit of real-estate, is purchased on a long term loan and the terms of the loan are divided by the number square feet in the facility.
Power
While some locations charge for electrical power by the KWH, many co-location providers see power supply as an opportunity to add margin to their business. Your monthly power fee will vary by the type of power connection, rather than by the variable KWH. On the face of things this might look like a good deal to the customer, as a nice consistent, predictable cost. Unfortunately the ability to reap any benefit from modern, power efficient system design is taken away.
Cabling PDUs
What is a PDU? It's a glorified power strip. Most states require licensed bonded electrical installation professionals to install anything more capable than the lowly power strip. Let the voice of experience be your guide, yes the installation fees are expensive but this is one task better left to the expert. Wrestling high amperage capable cables inside a machine room is no fun.
UPS
Oh yeah, so what if the power goes out? Think batteries, huge flywheels, and diesel generators. Huge hunks of equipment, the design of which hasn't changed much since the Carter administration.
Networking
Network bandwidth is likely the most costly component of hosting an online game. The bandwidth required for a typical game is far in excess that of even your largest web sites, and as everyone knows in the game industry, it's not necessarily the size of the pipe, it's the average ping time that makes or breaks an online game experience. Rent is just a fact of life, and is often bizarrely linked to ping time, that is, facilities within close proximity to an internet backbone, tend to have higher rent rates than those without good ping times.
Cooling
Cooling a room full of modern, high density systems is becoming more and more of an issue. Most datacenters today were built at the height of the 'dot com bubble', a time when a CPU called the Pentium was the new kid on the block. Roofing was cheap then. Many analysts thought the 1GHz CPU was a physically impossibility, let alone practical for all but the most esoteric of situations. Fast forward ten to fifteen years or so, and we commonly find 42 x dual processor systems running at 4GHz in a single rack, pumping out enough BTUs to blow-dry a full sized poodle from sopping to fluffy kinky curl in fifteen seconds.
The alert reader will be aware of a nagging question: what was that comment about roofing? Before we get to that, have you ever wondered why HVAC contractors typically spec systems in TONS? It takes literally tons of cooling equipment to cool even a modest machine room. The more heat you have to get rid of, the more cooling tonnage you need. Well here's the deal, you typically put major components of your HVAC system on top of the roof. Now you get the picture TONS of equipment on the roof, For an added twist let's set our time machine to somewhere circa 1990, and imagine that most datacenter roofs where designed with the notion that single processor 2u computers would come down the road eventually.
Back to our story
After three artery hardening weeks of searching and negotiation, technical adjustments and compromises you finally go to install those 3 racks full of 42 (or 84 +) servers each, and the co-location facility tells you something like "you're special, you'll need to leave an empty rack space between all your racks", or "Oh, you need the special HgZBfx connector, that will cost you 12x the quoted rate, and there'll be an extra 25% installation fee, Do you want UPS with that?"
THE VERY NEXT DAY your publisher calls, and tells you the launch date in Germany and Korea has been moved up, competition is fierce from Korea, and if the title doesn't launch ASAP, you're going to get killed in the market. Since you went through the pain of figuring everything out for the initial launch you know most of the questions to ask, so it should be a piece of cake.
It's only a few weeks before Alpha and you're about to ship equipment to Germany, you are conversing with your selected hosting facility, and they ask you to confirm the number of 220 circuits. Huh, 220? What?
Just to be safe you call your server vendor. They've got you a screaming deal on rack mount systems, and you think they'll be ecstatic to hear that you are moving up the purchase date for the European and Korean operations. When your client rep finally returns your phone call you find out that their European VAR is actually based in Connecticut, and don't actually have systems installed on the European continent, but they do have a roadmap that will lead to a sales and service partnership in Taiwan by the end of 2006! And since another hundred systems was not part of the original order, the 'per system' price has changed. Oh, and the delivery schedule will be 6 weeks out minimum: "That ok with you?" This bit of lovely news initiates your third coronary, and you switch carriers to somewhere less stressful like a killer bee keeper, or a high-tension power-line service technician.
Conclusion
So what's the moral of this story? Understand that the cost of hosting your online game will likely outweigh the initial capital cost of the equipment, but there are technical decisions you can make to keep these hosting costs down. Selecting technology that minimizes floor space, cooling, and power requirements will pay off in the long term, AND will likely reduce your initial hosting bill. Successful online operations get deployed in multiple hosting facilities in many different regions. Select equipment vendors that have direct operations or at least good VAR operations in all your target regions, make sure you get the names of your 7x24x365 service providers in every region before buying anything. Selecting a hosting provider experienced in online game operation seems like a no-brainer, but nothing is as easy as it seems at first.
(c) George Dolbier, 2005