Designing the storage for a very large game world

Question

I'm starting up game programming again. 10 years ago I was making games in qbasic and I havn't done any game programming since, so I am quite rusty. I have been programming all the time though, I am web developer/DBA/admin now. I have several questions, but I'm going to limit it to one per post.

The game I am working on is going to be large, very large world. It is going to be somewhat like URW, but a even larger world and more like an 'RPG'.

What I have been trying to decide, is what is the best way layout the map, save it, and access it. I thought up the idea of using sqlite to store the data. I could then even use the sqlite db as the save file for the game, nice and easy.

Anyone have any tips about how I should go about this or ideas for other storage methods?

Here are the requirements for my game:

• I need full random access to spot in the game world (the NPC's, monsters, animals will all be active all the time).
• I'm using Stackless Python 3.1, options are quite limited unless I do a lot of work.
• Needs to be able to handle a very large world.
• Concurrency support would be a plus, but I don't think I will need it.

Answer 1

Don't mess with relational databases unless you're forced to use them by external factors.

Look at Python's pickle, shelve.

Shelve is fast and scales well. It eliminates messy conversion between Python and non-Python representation.

---

Edit.

More important advice. Do not get bogged down in technology choices. Get the locations, items, characters, rules, etc. to work. In Python. As simply and correctly as possible.

Do not burn a single brain calorie on anything but core model, correctness, and a basic feature set to prove things work.

Once you have a model that actually works, and you can exercise with some sophisticated unit tests, then you can make technology choices.

Once you have a model, you can meaningfully scale it up to millions of locations and see what kind of storage is required. The model can't change -- it's the essence of the application. Only the access layer and persistence layer can change to adjust performance.

Answer 2

It sounds like what you're asking for is a type of spacial index. For a very large 2d game I'd recommend using a quadtree. Quadtree works well when you have a large area and activity tends to happen in localized regions of the area, which is the case for most RPG-type games. It will keep your storage requirements low and hopefully speed up collision detection as well.

As for saving the game, things like player and monster stats can go into a database, if you're worried about those changing often. For the actual level layout I'd recommend using a binary file format specific to your game. There's not many database-type queries you usually need to perform on the level layout and you can make great optimizations using your own format. I wouldn't know how to begin storing a quadtree-like format in a database (although I'm sure its possible).

Answer 3

I am using non-relationnal database to store big amounts of datas. If you can work on a 64 bits hardware, MongoDB with its Python driver is really very good. I do not know if this is ok with Stackless, but it is a possiblity.