I am building a Flask application where the ultimate goal is to calculate the midpoint of two or more origin latitude and longitude coordinates. I am hoping to use the OSMNX python package to produce a street network from one of the origin points. Then I hope to use the street network to retrieve the nearest node for each origin coordinate. Then, I will use the Networkx package to find the shortest path between the two nodes. Finally, I will query for the middle node in the path and set that as my midpoint.
However, I am failing to do this and I am not sure why. Everytime my code hits osmnx.graph_from_point(origin, 3200, 'network', 'walk'), it creates an infinite loop within my Flask application that re-runs the function under execution.
For instance, in the application, I have two files, meeting_service.py and osmnx_service.py. The Flask route being used is /midpoint. This route leads to a function called midpoint() in meeting_service. The function midpoint() checks if there are two or more origin points and calls either two function midpoint_two() or midpoint_more() both of which rest in osmnx_service.py.
For this example, I am concerned with midpoint_two(). The osmnx.graph_from_point function is within the midpoint_two function that lies within the large osmnx_service.py file. However, when the osmnx.graph_from_point function is called an infinite loop occurs and returns to midpoint() from meeting_service.py, from the first line of midpoint() to the line before osmnx.graph_from_point function in osmnx_service.py.
Because the code is stuck in an infinite loop. A Timeout Exception is thrown. The error can be seen below:
127.0.0.1 - - [16/Mar/2024 11:00:31] "POST /getMidpoint HTTP/1.1" 500 -
Traceback (most recent call last):
File "/Users/moisesherrera/Library/Python/3.9/lib/python/site-packages/flask/app.py", line 1478, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/moisesherrera/Library/Python/3.9/lib/python/site-packages/flask/app.py", line 1458, in wsgi_app
response = self.handle_exception(e)
File "/Users/moisesherrera/Library/Python/3.9/lib/python/site-packages/flask_cors/extension.py", line 176, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/Users/moisesherrera/Library/Python/3.9/lib/python/site-packages/flask/app.py", line 1455, in wsgi_app
response = self.full_dispatch_request()
File "/Users/moisesherrera/Library/Python/3.9/lib/python/site-packages/flask/app.py", line 869, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/moisesherrera/Library/Python/3.9/lib/python/site-packages/flask_cors/extension.py", line 176, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/Users/moisesherrera/Library/Python/3.9/lib/python/site-packages/flask/app.py", line 867, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/moisesherrera/Library/Python/3.9/lib/python/site-packages/flask/app.py", line 852, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/Users/moisesherrera/Desktop/LocisNet_Software_Dev/gistwebapp/geoprocessing_engine/src/controllers/meeting_controllers.py", line 5, in get_midpoint
return midpoint()
File "/Users/moisesherrera/Library/Python/3.9/lib/python/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
return current_app.ensure_sync(fn)(*args, **kwargs)
File "/Users/moisesherrera/Desktop/LocisNet_Software_Dev/gistwebapp/geoprocessing_engine/src/services/meeting_service.py", line 36, in midpoint
account = Account.query.filter_by(username=username).first()
File "/Library/Python/3.9/site-packages/sqlalchemy/orm/query.py", line 2748, in first
return self.limit(1)._iter().first() # type: ignore
File "/Library/Python/3.9/site-packages/sqlalchemy/orm/query.py", line 2847, in _iter
result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
File "/Library/Python/3.9/site-packages/sqlalchemy/orm/session.py", line 2308, in execute
return self._execute_internal(
File "/Library/Python/3.9/site-packages/sqlalchemy/orm/session.py", line 2180, in _execute_internal
conn = self._connection_for_bind(bind)
File "/Library/Python/3.9/site-packages/sqlalchemy/orm/session.py", line 2047, in _connection_for_bind
return trans._connection_for_bind(engine, execution_options)
File "<string>", line 2, in _connection_for_bind
File "/Library/Python/3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
ret_value = fn(self, *arg, **kw)
File "/Library/Python/3.9/site-packages/sqlalchemy/orm/session.py", line 1143, in _connection_for_bind
conn = bind.connect()
File "/Library/Python/3.9/site-packages/sqlalchemy/engine/base.py", line 3269, in connect
return self._connection_cls(self)
File "/Library/Python/3.9/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
self._dbapi_connection = engine.raw_connection()
File "/Library/Python/3.9/site-packages/sqlalchemy/engine/base.py", line 3293, in raw_connection
return self.pool.connect()
File "/Library/Python/3.9/site-packages/sqlalchemy/pool/base.py", line 452, in connect
return _ConnectionFairy._checkout(self)
File "/Library/Python/3.9/site-packages/sqlalchemy/pool/base.py", line 1269, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/Library/Python/3.9/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
rec = pool._do_get()
File "/Library/Python/3.9/site-packages/sqlalchemy/pool/impl.py", line 158, in _do_get
raise exc.TimeoutError(
sqlalchemy.exc.TimeoutError: QueuePool limit of size 20 overflow 30 reached, connection timed out, timeout 60.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
This error does not identify the cause of the infinite loop, only that there was a Timeout Error. but it does seem to confirm that the error occurs during execution of the midpoint() function in meeting_service.py.
Before I post the code in osmnx_service.py, it is important to state that the origin is a tuple with latitude first, longitude second. I have revised the proper usage of the osmnx function graph_from_point and I am fairly positive I am using it correctly. I even wrote a script on how I would use it before building the Flask Application to increase the viability of my project. Nevertheless, the environments which I tested the algorithm and where I am now using it for a full-stack application are very different. For testing I used a single one file flask application and a single file jupyter notebook whereas I am now dealing with a Flask application of at least 30 files.
I thought that the problem was multithreading. I thought that because osmnx is making API calls to OpenStreetMap then that could be one of the sources of the issue. Therefore, for each call to an osmnx function I wrote a Custom Thread to ensure that every call was within a single thread that pauses the main Flask thread until the Custom Thread is complete. However, the result was exactly the same, an infinite loop that returns to midpoint() in meeting_service. Below are the functions used in meeting_service.py followed by those in osmnx_service. Note the the osmnx.graph_from_point function is in the customer thread named GraphThread. I had to use custom threads to allow for return variables.
def check_meetings(active_meeting, account, locations):
## create origin node
origin_point = shape.to_shape(account.location)
origin_lon = origin_point.x
origin_lat = origin_point.y
origin_tuple = (origin_lat, origin_lon)
for meeting in active_meeting:
print(len(active_meeting))
if len(active_meeting) != 2:
print("More than two participants")
locations.append(meeting.location)
else:
if meeting.location != account.location:
dest_point = shape.to_shape(meeting.location)
dest_lon = dest_point.x
dest_lat = dest_point.y
destination = (dest_lat, dest_lon)
print("just two participants")
return osmnx_service.midpoint_two(origin_tuple, destination)
return osmnx_service.midpoint_more(locations)
@jwt_required(optional=False)
def midpoint():
username = get_jwt_identity()
account = Account.query.filter_by(username=username).first()
print(account.location)
print(type(account.location))
request_form = request.form.to_dict()
print(request_form)
request_id = request_form['meeting_request_id']
active_meeting = ActiveMeeting.query.filter_by(meeting_request_id=request_id).all()
print(active_meeting)
print(type(active_meeting))
locations = []
print(locations)
midpoint_tuple = **check_meetings**(active_meeting, account, locations)
return jsonify({
"userMessage": "Midpoint Identified: " + str(midpoint_tuple[1]) + ", " + str(midpoint_tuple[0]) + ". Walk to "
"the"
"designated "
"location"
"for "
"your "
"meeting",
"my_location": account.location,
"midpoint_info": midpoint_tuple
})
midpoint() calls check_meetings() which determines whether to run midpoint_two or midpoint_more in my osmnx_service.py.
The multithread code for osmnx_service.py is as follows:
def midpoint_two(origin, destination):
print(threading.active_count())
print(origin)
print(destination)
print(threading.enumerate())
ourGraphThread = GraphThread(origin=origin)
**ourGraphThread.start()** where code enters infinte loop
ourGraphThread.join()
user_graph = ourGraphThread.graph
print(user_graph)
destinationNodeThread = NearestNodeThread(graph=user_graph, node=destination)
destinationNodeThread.start()
destinationNodeThread.join()
dest_node = destinationNodeThread.nearest_node
print(dest_node)
## Origin node
originNodeThread = NearestNodeThread(graph=user_graph, node=origin)
originNodeThread.start()
originNodeThread.join()
origin_node = originNodeThread.nearest_node
print(origin_node)
## Route
ourRouteThread = RouteThread(graph=user_graph, origin_node=origin_node, destination_node=dest_node)
ourRouteThread.start()
ourRouteThread.join()
route = ourRouteThread.route
midpoint_val = math.trunc(len(route) / 2 - 1)
midpoint_node = route[midpoint_val]
print("midpoint node")
print(midpoint_node)
midpoint_lat = user_graph.nodes[midpoint_node]['y']
midpoint_long = user_graph.nodes[midpoint_node]['x']
print(midpoint_lat)
print(type(midpoint_lat))
print(midpoint_long)
midpoint_tuple = (midpoint_long, midpoint_lat)
return midpoint_tuple
CUSTOM THREADS
class GraphThread(Thread):
# constructor
def __init__(self, origin):
# execute the base constructor
Thread.__init__(self)
# set a default value
self.graph = None
self.origin = origin
# function executed in a new thread
def run(self):
# block for a moment
sleep(1)
# store data in an instance variable
self.graph = osmnx.graph_from_point(self.origin, 3200, 'network', 'walk')
class NearestNodeThread(Thread):
# constructor
def __init__(self, graph, node):
# execute the base constructor
Thread.__init__(self)
# set a default value
self.graph = graph
self.nearest_node = None
self.start_node = node
# function executed in a new thread
def run(self):
# block for a moment
sleep(1)
# store data in an instance variable
self.nearest_node = osmnx.distance.nearest_nodes(self.graph, self.start_node[1], self.start_node[0])
class RouteThread(Thread):
# constructor
def __init__(self, graph, origin_node, destination_node):
# execute the base constructor
Thread.__init__(self)
# set a default value
self.graph = graph
self.origin_node = origin_node
self.destination_node = destination_node
self.route = None
# function executed in a new thread
def run(self):
# block for a moment
sleep(1)
# store data in an instance variable
self.route = networkx.shortest_path(self.graph, self.origin_node, self.destination_node)
As mentioned above, I have tried multithreading but I fear that this is not the issue. However, I believe that it may optimize my code in the end so I am hesitant to revert to my previous version, because in either case, it was malfunctioning with the infinite loop. The stack trace gives me no more information then I have provided above with TimeoutError.
However, I will provide proof of the infinite loop. When I run the Flask Application, on IntelliJ IDEA, the built in terminal presents the following information repeatedly leading up to the TimeoutError shown above.
<class 'list'>
[]
2
just two participants
108
01010000XXXXXXXXXXXX30925dc03009bc3896094140
<class 'geoalchemy2.elements.WKBElement'>
{'meeting_request_id': '1'}
2024-03-16 10:59:18,260 INFO sqlalchemy.engine.Engine SELECT active_meeting.id AS active_meeting_id, active_meeting.created AS active_meeting_created, active_meeting.updated AS active_meeting_updated, active_meeting.firstname AS active_meeting_firstname, active_meeting.lastname AS active_meeting_lastname, ST_AsEWKB(active_meeting.location) AS active_meeting_location, active_meeting.meeting_request_id AS active_meeting_meeting_request_id
FROM active_meeting
WHERE active_meeting.meeting_request_id = %(meeting_request_id_1)s
2024-03-16 10:59:18,260 INFO sqlalchemy.engine.Engine [cached since 25.2s ago] {'meeting_request_id_1': '1'}
(34.XXXXX69, -118.XXXXX076)
(34.XXXXX03, -118.XXXXX094)
[<_MainThread(MainThread, started 4299277696)>, <FSEventsObserver(Thread-1, started daemon 6206697472)>, <FSEventsEmitter(Thread-3, started daemon 6223523840)>, <FSEventsEmitter(Thread-4, started daemon 6240350208)>, <FSEventsEmitter(Thread-5, started daemon 6257176576)>, <FSEventsEmitter(Thread-6, started daemon 6274002944)>, <FSEventsEmitter(Thread-7, started daemon 6290829312)>, <FSEventsEmitter(Thread-8, started daemon 6307655680)>, <FSEventsEmitter(Thread-9, started daemon 10754224128)>, <Thread(Thread-2, started daemon 10771050496)>, <Thread(Thread-37, started daemon 10820562944)>, <Thread(Thread-38, started daemon 10837389312)>, <Thread(Thread-39, started daemon 10854215680)>, <Thread(Thread-40, started daemon 10871042048)>, <GraphThread(Thread-41, started daemon 10887868416)>, <GraphThread(Thread-42, started daemon 10904694784)>, <GraphThread(Thread-44, started daemon 10938347520)>, <GraphThread(Thread-43, started daemon 10921521152)>, <Thread(Thread-45, started daemon 10955173888)>, <Thread(Thread-46, started daemon 10972000256)>, <GraphThread(Thread-48, started daemon 11005652992)>, <GraphThread(Thread-47, started daemon 10988826624)>, <Thread(Thread-49, started daemon 11022479360)>, <Thread(Thread-50, started daemon 11039305728)>, <Thread(Thread-51, started daemon 11056132096)>, <Thread(Thread-52, started daemon 11072958464)>, <GraphThread(Thread-53, started daemon 11089784832)>, <GraphThread(Thread-55, started daemon 11123437568)>, <GraphThread(Thread-56, started daemon 11140263936)>, <GraphThread(Thread-54, started daemon 11106611200)>, <Thread(Thread-57, started daemon 11247202304)>, <Thread(Thread-58, started daemon 11265077248)>, <GraphThread(Thread-59, started daemon 11308642304)>, <GraphThread(Thread-60, started daemon 11325468672)>, <Thread(Thread-61, started daemon 11412729856)>, <Thread(Thread-62, started daemon 11429556224)>, <GraphThread(Thread-63, started daemon 11461636096)>, <GraphThread(Thread-64, started daemon 11479773184)>, <Thread(Thread-65, started daemon 11563724800)>, <Thread(Thread-66, started daemon 11585531904)>, <GraphThread(Thread-67, started daemon 11623067648)>, <GraphThread(Thread-68, started daemon 11649331200)>, <Thread(Thread-69, started daemon 11793625088)>, <Thread(Thread-70, started daemon 11810451456)>, <GraphThread(Thread-71, started daemon 11827277824)>, <GraphThread(Thread-72, started daemon 11844104192)>, <Thread(Thread-73, started daemon 11883212800)>, <Thread(Thread-74, started daemon 11900039168)>, <GraphThread(Thread-75, started daemon 11916865536)>, <GraphThread(Thread-76, started daemon 11933691904)>, <Thread(Thread-77, started daemon 11950518272)>, <Thread(Thread-78, started daemon 11967344640)>, <GraphThread(Thread-79, started daemon 11984171008)>, <GraphThread(Thread-80, started daemon 12000997376)>, <Thread(Thread-81, started daemon 12208140288)>, <Thread(Thread-82, started daemon 12226801664)>, <GraphThread(Thread-83, started daemon 12284784640)>, <GraphThread(Thread-84, started daemon 12301611008)>, <Thread(Thread-85, started daemon 12420673536)>, <Thread(Thread-86, started daemon 12450344960)>, <GraphThread(Thread-87, started daemon 12494958592)>, <GraphThread(Thread-88, started daemon 12562493440)>, <Thread(Thread-89, started daemon 12639825920)>, <Thread(Thread-90, started daemon 12662943744)>, <GraphThread(Thread-91, started daemon 12720402432)>, <Thread(Thread-92, started daemon 11213385728)>, <GraphThread(Thread-93, started daemon 11383894016)>, <Thread(Thread-94, started daemon 12766851072)>, <GraphThread(Thread-95, started daemon 12783677440)>, <GraphThread(Thread-96, started daemon 12800503808)>, <Thread(Thread-97, started daemon 12817330176)>, <Thread(Thread-98, started daemon 12834156544)>, <GraphThread(Thread-99, started daemon 12850982912)>, <GraphThread(Thread-100, started daemon 12867809280)>, <Thread(Thread-101, started daemon 12884635648)>, <Thread(Thread-102, started daemon 12901462016)>, <GraphThread(Thread-103, started daemon 12918288384)>, <GraphThread(Thread-104, started daemon 12935114752)>, <Thread(Thread-105, started daemon 12951941120)>, <Thread(Thread-106, started daemon 12968767488)>, <GraphThread(Thread-107, started daemon 12985593856)>, <GraphThread(Thread-108, started daemon 13002420224)>, <Thread(Thread-109, started daemon 13072461824)>, <Thread(Thread-110, started daemon 13089288192)>, <GraphThread(Thread-112, started daemon 13136048128)>, <GraphThread(Thread-111, started daemon 13116862464)>, <Thread(Thread-113, started daemon 13274247168)>, <Thread(Thread-114, started daemon 13293170688)>, <GraphThread(Thread-115, started daemon 13317074944)>, <GraphThread(Thread-116, started daemon 13337047040)>, <Thread(Thread-117, started daemon 13434646528)>, <Thread(Thread-118, started daemon 13458812928)>, <GraphThread(Thread-119, started daemon 13499232256)>, <GraphThread(Thread-120, started daemon 13524185088)>, <Thread(Thread-121, started daemon 13568864256)>, <Thread(Thread-122, started daemon 13585690624)>, <GraphThread(Thread-123, started daemon 13602516992)>, <GraphThread(Thread-124, started daemon 13619343360)>, <Thread(Thread-125, started daemon 13683355648)>, <Thread(Thread-126, started daemon 13709357056)>, <GraphThread(Thread-127, started daemon 13735620608)>, <GraphThread(Thread-128, started daemon 13787754496)>, <Thread(Thread-129, started daemon 13962391552)>, <Thread(Thread-130, started daemon 13988917248)>, <GraphThread(Thread-131, started daemon 14020423680)>, <GraphThread(Thread-132, started daemon 14047801344)>, <Thread(Thread-133, started daemon 14064627712)>, <Thread(Thread-134, started daemon 14081454080)>]
[<ActiveMeeting 1>, <ActiveMeeting 2>]
<class 'list'>
[]
2
just two participants
109
(34.XXXXXXX9, -118.2XXXX6)
(34.XXXXXXX03, -118.XXXXX94)
[<_MainThread(MainThread, started 4299277696)>, <FSEventsObserver(Thread-1, started daemon 6206697472)>, <FSEventsEmitter(Thread-3, started daemon 6223523840)>, <FSEventsEmitter(Thread-4, started daemon 624 .
....
If you notice there is repeating content suggesting that the code returns to midpoint(). If you also notice, because it is stuck in an infinite loop, there are many threads that open, but none of them finish, which indicate that the osmnx.graph_from_point function is not terminating, but the reason for that is unclear. I am only querying for a 2 mile search area of a point. Therefore, I would not expect this problem to be related to the size of my query. Furthermore, I have successfully conducted this exact query in JupyterNotebook within a second.
My guess this has something to do with Flask and/or the structure of my code.
If you have any suggestions please let me know.
Moises
I solved the problem and made it work so the midpoint is returned and the function only run once. The problem itself I believe was multi-level, but the source of the error, I believe was none of the above suggestions. The infinite loop stemmed from the front-end system.
The flask application is connected to a react front end system which has a timer function. On timeout, the midpoint function in the flask application runs.
However, the timer function on the react front-end was sending a new request to /get_midpopint each second; because of that, midpoint was not able to finish and before finishing, it would be called again.
There was no infinite loop just a non ending continuous calls to the route which overloaded the system. I fixed the issue on the front-end and there was no more "infinite loop"
However, the problem itself provided an opportunity to optimize the application. Multiple threads were open and unclosed by the time midpoint was called at least 30 threads had been created. I don't think this is good practice. I found a solution online to not allow more than one thread to be active at any one time, and it can be implemented in a flask application very easily. Below is a start flask file. All the needs to be done is place "threading=False" as an argument to the flask app initiation.
As a result of the parameter, I was able to reduce the threads to 2 main threads, and 10 other processes in the background. The issue was not with flask-sqlalchemy. And i hope I am left with a more optimal application..
Thank you.