Multiple Inodes used to reference the same socket

218 views Asked by At

I have a collection of applications that all bind to the same Multicast UDP port on a single Linux machine. (in this case 23003). The application is multi-threaded so, in some cases, it might bind to the same port multiple times in the same application in different threads.

That said, when I perform an lsof, I see that there are references from a single pid to multiple inodes that all refer to the same UDP port. That's fine; however, when I look across multiple applications, sometimes that inode number is unique, which I'd expect, and some times it's NOT unique; meaning multiple applications are referring to the same socket inode.

I would expect one or the other, but not both. See output below. In the first output, PID 18774 and others have two different inodes (one 28267669, the other unique-per-application) both referring to port 20003. In the second command you can see all of the applications that are all looking at inode 28267669. Can anyone explain? Why I'm seeing both unique AND non-unique inode references?

[root@v20z5 log]# grep ChangeLogLevelMsg foo.log | sort | uniq -c
      1 15:55:24: PNUFR::pre: PID 18771, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18774, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18774, INODE 28267759, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18776, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18776, INODE 28267747, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18781, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18781, INODE 28267769, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18782, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18782, INODE 28267915, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18784, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18784, INODE 28267907, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18786, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18786, INODE 28267997, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18789, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18789, INODE 28267935, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18790, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18790, INODE 28267996, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18792, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18792, INODE 28267952, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18793, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18793, INODE 28268096, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18795, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18795, INODE 28268057, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18798, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18798, INODE 28267964, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18803, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18803, INODE 28268173, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18804, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18804, INODE 28268071, PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18831, INODE , PORT 23003 => TAG 3 ChangeLogLevelMsg
      1 15:55:24: PNUFR::pre: PID 18831, INODE 28268071, PORT 23003 => TAG 3 ChangeLogLevelMsg

[root@v20z5 log]# grep ChangeLogLevelMsg foo.log | grep 669       
15:55:24: PNUFR::pre: PID 18771, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18774, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18776, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18781, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18782, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18784, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18786, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18789, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18790, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18792, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18793, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18795, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18798, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18803, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18804, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18831, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg

[root@v20z5 log]# grep ChangeLogLevelMsg foo.log 
15:55:24: PNUFR::pre: PID 18771, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18774, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18774, INODE 28267759, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18776, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18776, INODE 28267747, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18781, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18781, INODE 28267769, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18782, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18782, INODE 28267915, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18784, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18784, INODE 28267907, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18786, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18786, INODE 28267997, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18789, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18789, INODE 28267935, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18790, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18790, INODE 28267996, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18792, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18792, INODE 28267952, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18793, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18793, INODE 28268096, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18795, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18795, INODE 28268057, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18798, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18798, INODE 28267964, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18803, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18803, INODE 28268173, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18804, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18804, INODE 28268071, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18831, INODE 28267669, PORT 23003 => TAG 3 ChangeLogLevelMsg
15:55:24: PNUFR::pre: PID 18831, INODE 28268071, PORT 23003 => TAG 3 ChangeLogLevelMsg
0

There are 0 answers