Taking this example from the manual,
import pandas as pd
us_cities = pd.read_csv("https://raw.githubusercontent.com/plotly/datasets/master/us-cities-top-1k.csv")
import plotly.express as px
fig = px.scatter_mapbox(us_cities, lat="lat", lon="lon", hover_name="City", hover_data=["State", "Population"],
color_discrete_sequence=["fuchsia"], zoom=3, height=300)
fig.update_layout(mapbox_style="open-street-map")
fig.update_layout(margin={"r":0,"t":0,"l":0,"b":0})
fig.write_image('/tmp/so.pdf')
one can see that the map (including the plotted dots) is in fact actually just an embedded raster graphic:
How do I export OpenStreetMap plots into full vector graphics?

The background map you're using originates from a webmap tile service that serves png's and not vector graphics...
From the docs: (https://plotly.com/python/mapbox-layers/))
To access vector-data from openstreetmap for free, i suggest to have a look at osmnx, download the (vector) data you need locally and then create map from that.