Finding location of Files (Ceph)

Question

Using a filesystem it is possible to find the physical location of files using filefrag, hdparm or Fibmap.

What about Ceph? Is it possible to find out where a file is stored ? (which OSD, which Disk, where physically on the disk).

Lets say i have a folder with 5 files in it. I would like to create a script that outputs a file that stores the following information: 1.How many blocks is each file split into, 2.Each block --> OSD, 3.Each block --> which Disk, 4.Where (sector) on the disk is each block stored.

I would appreciate any input.

Answer 1

I am not sure if this is what you're looking for, but you can find out SOME information about the location of an object in a Ceph Object Storage Cluster:

http://docs.ceph.com/docs/jewel/rados/operations/monitoring-osd-pg/

For example, if you have a pool "pool-1" and want to find out where your object "testobject-1" is stored, you can get that information with this command (from your mon-node):

 sudo ceph osd map pool-1 testobject-1    

Which will give you a result like this:

osdmap e58 pool 'pool-1' (7) object 'testobject-1' -> pg 7.74dc35e2 (7.62) -> up
 ([1,0,2], p1) acting ([1,0,2], p1)

This tells you that your object is in placement group 7.62 on osd-nodes 1, 0 and 2 / primary partition (p1).

Now you can drill down a bit further, e.g. on your osd.0 node, switch to your root user and cd into /var/lib/ceph/osd/ceph-0 --> this is the location where the actual blocks are stored. If you do a "ls" you will see this:

root@osd-1:/var/lib/ceph/osd/ceph-0# ls
block ceph_fsid fsid keyring ready type whoami

This is all I could find out, and I don't know how you can actually "read" the contents of the block.

I hope this could help you at least a little bit....