I am making an antiCheating program for java game. I want to monitor the loading of all classes and find out which jar file it is loaded from, or is dynamically generated, so that I can calaulate hash and confirm whether the jar package is legal.
The legality cannot be confirmed by verifying the signature of the jar package because most jar packages are not signed. I tried to use the callback function of jvmti, but I couldn't find a callback method that can track the jar package.
I also try to find url from URLClassloader but I found many classes are loaded by custom classloader. Custom classloader are too many to adapt.
Client side anticheats are most likely not what you should do, just because they can easily be bypassed or disabled entirely, since they run on the client's pc - which the client has full control over. Especially languages like java are very easy to reverse engineer, and while making the checker in c++ would make it harder, it's still possible and easy to circumvent that.
That being said, you can hook the native method that defines classes, ClassLoader.defineClass0, .defineClass1 and .defineClass2, and check the data there.