I am planning on writing a 3D game that will be using VBOs for rendering. Let's say, for example, that the terrain is a set of tiles and their vertices are all in the same VBO. The player should be able to scroll through the tiles, and at all times would see only a part of them.
I would like to perform frustum culling on those tiles. I already have found some sources on the maths part of frustum culling, but I am not sure how I would go about implementing this with a VBO - do people do that somehow in the vertex shader, or do they just call the rendering function to draw a subset of the VBO.
Given that your camera acts like in Diablo (wherever Isometric or with Perspective):
If you have a fixed map size, you can use 1 VBO for the base geometry of your map, Assuming you will use a heightmap based solution. The Quads not visible will be discarded by your graphics card after the vertex shader, not affecting your pixel fillrate. They are not worth the overhead of culling on your side. Details like Rocks, Houses etc will have their own VBO anyways.
If you aim for a streaming content engine with a huge seamless world, create chunks, the size of a chunk depends on your game. Divide your terrain into those chunks and test the camera frustum against their bounding boxes before drawing.
About drawing chunks:
The simplest way, which is enough for most games, is to make each chunk its own geometry, VBO, and so on. But you can optimize later and your terrain implementation should not drive your engine API designs (you will have to implement many different ways to draw things in your engine, for instance particles, post processing effects, etc..).
One way you can optimize is with only one VBO for the geometry and the usage of instanced drawing, just like in particle systems you then take some other source for some of your data, like your global transformation, the height of each vertex and so on.
But keep in mind, most games dont really need that much optimization in just the terrain. There will come other systems across your path more worthy of optimizations.