matplotlib setting axis limits is not working

Question

I am trying to specify figure limits in matplotlib, but they are not being enforced. I am setting the limits in my subplot, and I saw the figures were clearly at different extents, so I have printed out the specified and actual limits to confirm that they are different.

How can I force the figure limits, regardless of data extents?

import matplotlib.pyplot as plt
import contextily as ctx

def mymap(df, xlimits, ylimits):
    mycolorsdict = {'A':'green', 'B':'orange', 'C':'blue'}
    
    fig, ax = plt.subplots(figsize=(10,10))
    
    for key, group in df.groupby('category'):
        group.plot(ax=ax, kind='scatter', 
                   x='long', y='lat', 
                   color=mycolorsdict[key]
                  )
    ctx.add_basemap(ax, crs='epsg:4326', 
                    source=ctx.providers.Esri.WorldShadedRelief
                   )
    ax.set_xlim = xlimits
    ax.set_ylim = ylimits
    ax.set_aspect('equal')
    ax.ticklabel_format(useOffset=False, style='plain')
    ax.tick_params(axis='both', labelsize=8)
    print(ylimits, ax.get_ylim())
    print(xlimits, ax.get_xlim())

And when I run this function for a couple of dataframes, the maps are not at the same limits nor at the limits I specified.

xl = [-116.423639, -116.120922]
yl = [34.288848, 34.779539]

for cat in df.category.unique():
        df2 = df[df.category == cat].copy()
        mymap(df2, xl, yl)

It returns limits different that what I specified. What I specified is on the left, and the actual are on the right.

[34.288848, 34.779539] (34.26499695, 34.80380005000001)
[-116.423639, -116.120922] (-116.43791284999999, -116.10754215000003)
[34.288848, 34.779539] (34.327880050000005, 34.71235095)
[-116.423639, -116.120922] (-116.38095349999999, -116.1129605)