Rough Sea Games » Server Administration

Multiple Game Instances Management

Thomas — Mon, 08 Nov 2010 09:55:05 +0000

In most MMOGs you have more than one game instance, because its much easier to implement instead of a huge game universe for all players. Because the game instances do not interact, the players can be allowed to play on multiple game instances at the same time.

The problem is that most Admintools are just a special interface to the game. They run in the same environment as the game and therefore there are multiple Admintool instances in a multi game instance game too. This is very annoying to the Community Management, because they have to use X Admintools to manage X game instances. In our planning phase of our Admintool we of course talked to the people that should use it. The Community Managers told us that they’ve coded scripts that execute queries on several Admintools to make their work more comfortable. Yeah! Nice guys, they optimize their workflow on themself. But wait a second… The Community Management is not supposed to write code, especially if its a workaround for a bad architecture. There is no reason to have an Admintool for every game instance, but there are much reasons to have only one Admintool for every game instance:

You are able to query multiple game instances with a single mouse click. (e.g. search for an user with a specific email or IP)
There is a central role and user management. No need to do the setup again and again.
Its possible to merge statistics from several game instances to an overall statistic.
The Community Management do not have dozens of Admintool windows. There is only one Admintool to rule them all.

I hope i’ve convinced you. If you know reasons that speak against a single Admintool feel free to write a comment.

Ok, enough of the “why we do what we do” thingy. Lets talk about the how. If an Admintool is not just an interface to the game instance, it has to be an application on its own. That means you do not only have game servers, you have an Admintool server too. Additionally the Admintool needs a way to communicate to the game servers and vice versa. A direct remote SQL connection is out of the question for security and other reasons. We decided do use ActiveMq for the communication. The main reason is the guaranteed delivery feature. If the Admintool goes down, all messages targeting the Admintool are stored on the senders hard disc. They will be sent once the Admintool goes up again. This way no tracking data from the game will be lost.

The Admintool needs a way to know of all the game instances floating in the universe. You don’t have to worry. There is no list of game servers and there connection properties stored at the admintool. Instead we’ve decided to make the Admintool the sun of our game universe with many game planets surrounding it. This makes it much easier to maintain all the connections. The game servers will connect themself to the Admintool at their first startup. The Admintool sends the game server a unique internal server id which is used for subsequent connections (e.g. after a update and server restart). This way the Admintool can manage a list of all game instances in the world.

Thats all for now. Next time I write about multiple server queries and how to filter, sort and page them.

Share/Bookmark

Tools for MMOG Administration

Thomas — Mon, 13 Sep 2010 15:28:44 +0000

Hey there,

this time I tell you about the project the R&D department worked on for the last months. We developed a tool that helps us and the Community Management department to administrate a Massive Multiplayer Online Game. We just call it Admintool. A tool that allows us read/write access to the game database. This is required, because not everyone follows the rules of your game. There are abusive ingame messages, players with multi accounts or bots. The evil is everywhere out there. You may think: But I have phpMyAdmin installed on the game server. This way I’m able to access every part of the game database.

Of course you can. But it’s not you who does the Community Management. It’s to easy to do harm to your database using direct database access especially for people without technical skills. A good Admintool let the users see what ther’re supposed to see and let them only do changes to things that does not have any impact on your game mechanics.

But this is not the only reason why an Admintool is a must have. Just imagine you want to ban a player from your game. It’s much easier to fill out a form with a reason and ban duration than manually do all the logic behind this. If changes are only possible through the Admintool it’s much easier to find abusive behaviour from the Admintool users too. You just have to log every change on the game database automatically.

Last but not least: Tracking. In a MMOG you should track nearly everything. You should track when a player register his account and which IP he used and you should track when a player spents money in your game. Of course you don’t require an Admintool to do the tracking. But you can use the Admintool to show the results of the tracking. Statistics and graphs that tell you how well your last marketing campaign worked or which is the most important feature in your game.

I hope I have convinced you to use an admintool for MMOG Administration. In the upcoming posts I will tell you about the features our Admintool has and how they are implemented. Next time I will show how our Admintool is able to manage dozens of game instances with only one Admintool.

Share/Bookmark

InnoDB the “case-sensitive monster”

Ole — Fri, 04 Jun 2010 11:51:35 +0000

Hello,

this posting is about InnoDB’s limitation of lower / upper cases and the resulting problems of
InnoDB database migrations to different operating systems.

THE TASK:

Import an InnoDB database from Mac OS MySQL 5.x to Linux MySQL 5.x (in our case Rhel 5 ).

No big deal, I thought.

Made a mysql dump and imported the dump to Linux. 5 minutes work, just the way admins love it !

THE CHALLENGE:

Imported the dump. I am done! That’s what I thought. Just let’s check the app that accesses the freshly imported DB.

Running the app brought me exceptions. Some tables couldn’t be found… URGH! Checked the tables… everything was fine. The dump was fine… the tables were present. WTF!!!!!

THE SOLUTION:

After 10 hours of pain, we found the solution.

MySQL and especially the storage engine InnoDB depends on the case sensitivity of the operating system. Windows and Mac OS (by default) are not case-sensitive. So if you are importing a Mac or Win DB-dump to a case-sensitive operating system (like Linux), the DB will be handled case sensitively, although the dump is not a case-sensitive one. I think this problem should be fixed. I know fixing this problem is nearly impossible because the OS has to be capable of doing this. But giving some sort of warning while importing a non-case-sensitive DB to a case-sensitive DB-Server… that can’t be so difficult !

How to fix our specific problem:

Add the following line to your MySQL Server Config:

lower_case_table_names=1

By default Windows MySQL uses lower_case_table_names=0 and MacOS MySQL lower_case_table_names=2

The MySQL documentation advises setting lower_case_table_names=1 for InnoDB. For more information, please read the MySQL documentation.

Share/Bookmark

Effortless Text Localization

Thomas — Tue, 29 Sep 2009 14:32:24 +0000

Automatization of repeated work is one of the keys to productive development. Another is abstraction of common problems to allow concentration on project specific work. Localization is one of the problems you have in nearly every project, especially in iPhone projects. The iPhone SDK brings interesting solutions that abstract many parts of the localization process.

The tool ibtool, for example, extracts strings from an interface automatically. The strings are placed in a .strings file, a textfile with key/value pairs. To localize an interface, you have to translate .strings and merge the strings back into the interface again. Because you have individual interfaces for every localization, it’s possible to adjust widgets individually for each one.

genstrings is another tool inside the iPhone SDK. It extracts textIDs from the source code and write them into a .strings file. You may ask how the tool knows which texts need localization and which do not. The solution is the macro NSLocalizedString, which will be replaced by a .strings file lookup method by the preprocessor, but also searched for by the genstrings tool to create the files.

Both tools help you to create a localized application without paying much attention to localization itself. But you cannot expect the localization department to search for .strings files inside your project and create localized versions of them. Of course this would be possible, but not very convenient, because you have to migrate the translated texts back into the interfaces using ibtool. Another reason for us at Rough Sea is that we use a localization interface from our publisher. This interface is well known to the localization department and the content is placed in a centralized database on a server.

So we have the great tools from Apple that help us to separate texts from the project and we have the great tool from our publisher that handles the whole translation and reviewing process. Now we need something to tie those tools together, because we do not want to insert new texts from the .strings file into the publisher’s localization tool manually or vice versa. This glue tool has to execute the Apple tools, extract the texts from the .strings files and insert them into the publisher’s loca tool. On the other hand, it has to check for new localized texts from the publisher’s loca tool, build the required .strings files from the results and merge them back into the interfaces. Sounds quite easy, but of course there are some obstacles to get there. You have to handle other things, like the deletion of a text entry or changes to an already translated interface. So you have to know what has changed since the last update and stuff like that.

It turns out that you only have to integrate this glue tool into the build process of your build server. The tool will update the localization database whenever the code or the interface changes and it will update the localized versions when the database changes. As a coder you only need to remember to use the text macro around your text id. You don’t have to add this text id in a file or anything else. As you commit your changes, the build server will do this for you. As an interface designer it’s the same: just create your interfaces in the primary language and commit it. After the localization department finishes localizing those texts, they will be inserted into the localized versions automatically. Of course you have to make adjustments to the interface if there are loca bugs like labels that are to small to hold the translated text.

As you can see those localization tools are a big black box for coders, interface designers and translators. The coders only have to write code, the interface designers design interfaces and the translators translate texts. At the end there will be a localized product.

Share/Bookmark

Secure your Webserver (part 2- A)

Ole — Mon, 20 Jul 2009 08:12:59 +0000

Hello everybody,

Finally you are able to read Part 2 of “Secure your Webserver”. Part 2 will be about Linux, Webserver and the other important services.

As I mentioned in Part 1, I will not write about Windows, MacOS or other operating systems, because the most common one for a webserver is Unix / Linux. Part 2 will have a part A and B. Today I am going to write about distributions, user managament, user rights, secure shell and their enormous importance for security.

Before you start, choose the right distribution of Linux for your aims. There are a lot of Linux versions (distributions) on the market. Most of them are free or have a free community edition. The most common free distributions are CentOS, Debian, Fedora, Gentoo, OpenSuse, Mandriva and Ubuntu. Of course there are also commercial destributions like Red Hat Linux Enterprises (RHEL) or Suse Linux Enterprises.

In general all these distributions are fine for a webserver. Their differences are minor and more a personal choice than a real technical question. A NEWBIE should maybe use OpenSUSE, Ubuntu or Fedora, as the support of the community seems to be bigger for them. Commercial products are usually superior to free distributions in their support system. They grant better support via phone, etc.

After you have chosen the right distrubtion for your purpose and installed it on your server machine, it is time to think about security.

1. User-Management and Shell-Access. (Secure Shell Daemon)

Linux security is mostly based on user rights. User rights are essential to the security concept of Unix and Linux systems. Usually all distributions are very strict and separate services (daemons), users, administrators, essential services like a webserver, MTA & MDA (Mail Transfer Agent & Mail Delivery Agent) into different groups and users.

All these groups and users have different write and read accesses. Usually groups are created to give a bunch of users the same rights. Therefore groups can make your administrator’s life far easier.

The most important “user” is called root. Root is the highest ranked user on a system. The “root user” has full read and write access. In the Windows world it would be the “Administrator”. This is the reason why it is not smart to use the root user in your everyday work. The root user should only be used for system critical and important parts. In all other cases it is wise to use a “normal” user, which you have created.
It is also possible to run root commands via your user account. Important commands for this purpose are “sudo” and “su”. Sudo runs a command line with a special user. The command su makes it possible to log in as another user via your own user shell. Of course you need the right password of the user to perform these actions.

It is common for Linux to allow remote login via secure shell (SSH), especially if your webserver is not reachable for you in person, e.g. a dedicated server in a data centre of your webhost. All connections via SSH are encrypted. It is nearly impossible to decrypt the data via your client and your server (Maybe the NSA or the CIA are able to decrypt this – who knows?). You should take care to choose the ssh version 2 protocol. This is safer, as recently some weaknesses were discovered in protocoll version 1. A common SSH-client for Windows is “Putty”.

It is possible to permit or forbid login via SSH for specific users or user groups. Maybe it is not wise to allow direct root login via SSH. It is also possible to login via a certificate. You create an public and private key. Upload the public part to your server (usually: /homedir/.ssh/authorized) and log in without password. Of course it is wiser to secure your private key with a passphrase. In this case you need to type the phassphrase to decrypt the private key to log in.

Next part will be about firewalls, virusscanners and how to avoid spam problems.

Share/Bookmark

Secure your Webserver (part 1)

Ole — Mon, 05 Jan 2009 10:44:37 +0000

Hello ,

today I am going to write about server security. I decided to split the post into 3 parts. The future parts will be published during the next weeks.

Part 1 deals with server security in general and the conception of your personal strategy to avoid security problems.

Security has become more and more important in the last 10 years , as the numbers of internet users and server services are growing year by year. Especially the web 2.0 revolution brings new security problems. Nowadays users with a bad understanding of the technical background are setting up blogs, forums, websites and other services.

Therefore I wrote a small introduction for those newcomers!

In general there are 3 main areas you have to keep an eye on.

1. Network infrastructure:

I will not deal with this, because usually only professionals can influence the network infrastructure or your provider does this for you with routing , firewalls and filters.

2. Operating System:

A big part of security solutions and problems rely on the chosen operating system. All common operating systems (e.g. Linux, Windows, Unix) have advantages and disadvantages. If you expect me to write down which is the best one, I WILL NOT ! Nobody can tell you. It depends on so many factors like the services you want to run, your personal knownledge about the operating system, etc… . Maybe you do not even have the chance to choose your OS as your provider pre installed already one for you.

Unfortunately, I will have to focus on one operating system in the second part of my post. In my point of view the most common one is Linux. Although I am aware of the fact that other operating systems are great. So do not bug me with comments like: “You hate Windows ! Why not using FreeBSD ? Solaris is the best one !”

3. Applications & Daemons

Daemons or services are the core of your security solutions and also the source of most security issues. Before offering several services, e.g. a Web-Server , Ftp-Server or an Email-Server, think about which services you really need and if it is really smart to offer all services on a single machine. Every application can be corrupted or compromised. Avoiding services and daemons is always a clever strategy to minimize the risks. Moreover security tools hinder security issues as well, for example virus scanner, firewall, spam filter, a handy user rights management, etc… .

Please think about all these facts before you run a public server.

The second part will be about the practical parts of server management to build a secure server. We are going to leave the boring theory, promised!

Share/Bookmark

Use Mumble as a virtual conference room

Ole — Wed, 03 Dec 2008 16:23:04 +0000

Hello folks,

I am sure you know that communication is one key for successful development or project work. Companies usually have a conference room, or you meet somebody in the hall or in the elevator that you need to talk to. But nowadays people work from home in different cities, countries or even continents. Therefore you need a virtual place to meet.

That’s why I am going to write a little about Murmur / Mumble for those who haven’t heard about it. Mumble is a voice over IP service, like Battlecom, Teamspeak or Ventrillo. The advantages of Mumble are an excellent quality of speech, database support, server & client support for all major operating systems, text chat, and encrypted traffic — all for free!

First of all, we should differentiate between the server (Murmur) and the client (Mumble). Murmur defines and handles all Mumble requests: a typical client / server system.
But enough of this chitchat. We have to set up a Murmur!

Installation of Murmur:

Therefore we should start with installing Murmur. Installing Murmur on Windows shouldn’t be a problem, but what about Linux? All common distributions offer a binary packet of Murmur that you can install via your package management tool like yast, yum install “packetname” , apt-get install “packetname”, etc.

I am going to describe another way and will use the
“Static Linux Server stable packet” which you can download here.

Download murmur-static_x86-X.XX.tar.bz2.

Now untar the compressed file.

tar xfvz murmur-static_x86-X.XX.tar.bz2

Go to the directory which has been untared or move it for example to /home/murmur. Of course you can also run Murmur in the current directory.

Now it is time to decide whether to use SQLite or a “real” database like MySQL. SQlite has the advantage of running with Murmur right out of the box. The disadvantage is command-line-based user management, like adding new users or administrators. With MySQL you could use phpMyadmin (if installed) to do the management, so future administrators wouldn’t need shell access to add new users.

So if you want to use SQLite, ignore the MySQL part of the murmur.ini

Create a database or use an existing database of MySQL!

Edit the murmur.ini with MySQL support:


database=putyourdatabasenameinhere

dbDriver=QMYSQL
dbUsername=database username
dbPassword=Ilove-securePasswords
dbHost=localhost #localhost or the ip of the remote database
dbPort=3306 #usually the default port of mysql is 3306
dbPrefix=murmur_ #prefix name of your tables within the database

### SQlite and MySQL has to edit the lines below ###

host= the ip or domain # if you want to bind the murmur server to a default address

port=64738 #port you use for your murmur server

serverpassword=NicePassword # password for non-registered users. Keep it empty for no password login

Save the murmur.ini and exit it.

Now set up a superuser with the following command:

./murmur.x86 -ini murmur.ini -supw

Under Windows, of course, use cmd and type murmur.exe instead of murmur.x86

Now the superuser is saved in the database.

That’s it!

Murmur is running. Now you can login as “superuser” with your Mumble client and configure the rooms with a helpful GUI. Click on “channel” and then “add”. Now you can add a channel/room. This shouldn’t be a problem.

Enjoy your virtual conference room!

Documentation is available at mumble.sourceforge.net

Share/Bookmark

Why do I feel like a Swiss army knife?

Ole — Sun, 09 Nov 2008 16:19:53 +0000

Hi, let me introduce myself with my first blog post . My name is Ole and my job could be briefly described as a server administrator – and yes, I feel like a Swiss army knife. Oh, please do not misunderstand me. This feeling isn`t frustrating or negative. I like my job. You need a wide range of knowledge to do this job, as admins work with systems and not only with few applications or “just” software.

My job is to deal with the hardware, software, server services, system security and especially user`s wishes.
(Hey, just in a professional way, of course !) Anyway, sometimes I experienced that administrators ignore users and their needs. Although their job is to serve the users, but I guess, this topic would be worth of another blog post.

So perhaps you are wondering: “Why does he like to be the swiss army knife of Rough Sea Games and work with game developers ? ” Short answer: “I LIKE GAMES and I LIKE COMPUTER SYSTEMS .” A perfect fusion of my personal interests. This is great !

Anyway, my time is running short, so I promise to keep in touch with you and write new posts about my work. In the upcoming posts I will write down some interesting hints about server administration and maybe help other Swiss army knifes to keep their systems alive with those blog posts. Feel free to post some comments.

PS: WE ROCK THE BOAT!

Share/Bookmark