efficient way to geocode huge amount of twitter user location

Question

I'd like to geocode around 2M strings representing cities or states or countries of the whole world every day. Theses strings are not very clean, (you have to make distinction between "Paris,France" and "Paris,TX" for instance, but you have you also have to geocode "Paris" with the french city !) that's why I'm looking to use GMaps API or OSM Nominatim API.

The brute force solution would be to dump OSM data on my computer and process it locally, but I hope I can find an easier way to do it.

Obviously, I can reduce the amount of strings to be geocoded, store the results not to have to query twice a string, but I would still have hundred thousands strings to geocode...

Thanks !

Answer 1

To comply with the usage policy when using OSM's Nominatim instance you have to wait 1 second between each request. Geocoding 100.000 requests would take around 28 hours which could be feasible for you. Alternatively you can use one of the other Nominatim instances (after checking their usage policy!) or try to install it locally.