I'm trying to make a Java project that pinpoints the place on a image of a map, when given coordinates (taken from Google Maps).
I've tried using the top-left corner of the image (place that has highest latitude, and the lowest longitude), as an some kind of an reference point, which would be (0,0) point on the map image, and than I've tried to calculate every place on the map based on that reference point. However, this method proved inaccurate, probably because of the curvature of the Earth (mind that the map I'm working with (Serbia) covers area of 4° latitude, and 4° longitude).
I've seen couple of answers talking about converting into Mercator projection, but they are not very clear to me, because they are not covering a case similar to mine, and are not written in Java.
What can I do to pinpoint those points more accurately (±3km would be accurate enough)?
As comments have pointed oit correctly, in order to precisely convert between geographic coordinates and map position, you have to know the method of projection used for the map, and a sufficient number of parameters so that tuning the remaining parameters using a suitable set of reference points becomes feasible.
So if you assume a conic projection, then read the document David pointed out, and this referenced follow-up as well. As you can see, within the family of conic projections, there are a few alternatives to choose from. Each of them is described by a few parameters (i.e. standard parallels, cone constant, aspect ratio, …). You'd make guesses for these and then use some numerical optimization to obtain a best fit. Then you take the best parameter fit for each kind of projection and see which of them has the best overall fit. Quite a bit of work. If you don't want to implement the code for all these projections you can use proj.4 either on the command line or as a native library. To do the numeric optimization, you could possibly try to adapt one of the COIN-OR projects to your application.
In any case, the first step would be creating a suitable set of reference points which you can use to evaluate the fit. So pick a few prominent points on your map and find Google Earth coordinates for these. I'd say you should have at least a dozen points, to account for the fact that you know so little about your map. Otherwise there is a great risk that you will tune the large number of parameters to exactly fit your points while the rest of the map is still completely off. Even with this number of reference points, since the area of Serbia is not that big (compared to maps spanning whole continents), the errors of a wrong guess or a bad fit might be very small. So it might be hard to actually decide which projection has been used.
With all that I said above, and even with external libraries taking care of the projection and the numerical optimization, it might easily take you half a year just to set up the tools to work out the projection. So decide whether that's worth the effort. If not, there are several alternatives. One would be to take a different map, one where you know the projection. Or contact the author of your map and obtain the projection. Or ask someone working in geodesics in Serbia, because they might have enough experience to recognize the projection at a glance, I don't know.
One other option is by combining the fact that you need reference points with the fact that you might not be able to work out the exact projection in any case. Simply combine these in the following way: choose a suitably dense set of reference points, evenly distributed over the map. Then interpolate between them, picewise linearily or with higher degree or using some weighted interpolation scheme or whatever. You know there is a projection behind all this, but you give up on working out the projection, and simply mitigate the symptom: by having enough reference points, each data item is close enough to a reference point to keep the error smaller than your threshold.