Issue adjusting figsize with matplotlib barh subplot

Question

I've tried specifying in a few ways but have not been able to get this figure any bigger than what's shown.

category_names = ['Database', 'Frontend', 'QA', 'ML', 'Fullstack']
labels = list(final_df.index)
data = np.array(final_df.iloc[:, 1:])
data_cum = data.cumsum(axis=1)
category_colors = plt.get_cmap('RdYlGn')(np.linspace(0, 1000, data.shape[1]))

fig, ax = plt.subplots(figsize=(100,75))
ax.invert_yaxis()
# ax.xaxis.set_visible(False)
ax.set_xlim(0, 200)

for i, (colname, color) in enumerate(zip(category_names, category_colors)):
    widths = data[:, i]
    starts = data_cum[:, i] - widths
    ax.barh(labels, widths, left=starts, height=0.5,
            label=colname, color=color)
    xcenters = starts + widths / 2

    r, g, b, _ = color
    text_color = 'white' if r * g * b < 0.5 else 'darkgrey'
    for y, (x, c) in enumerate(zip(xcenters, widths)):
        ax.text(x, y, str(int(c)), ha='center', va='center',
                color=text_color, fontsize=15)

If I make the figsize any bigger, the kernel dies and I've tried adjusting height and np.linspace params, as well as trying to set size with fig.set_size_inches. Any ideas on what's going on here?