Preventing scaling text while scaling parent div

Question

Now am working on project of a floor plan map that contains a certain amount of rooms – each room is being represented as a Raphael.js shape, and in the middle of the shape there’s a number of tasks. Font size No of tasks needs to be fixed for each shape.

For zooming in:

(this._zoom <= 6 && this._zoom > 0) ?
(this.count) ? this._textPaperSet.forEach(function (e) {
e.transform("...s" + (1 / defaultZoomStep));
}) : ''
                : '';

And zooming out similarly:

(this._zoom >= -0.16) ?
(this.count) ? this._textPaperSet.forEach(function (e) {
e.transform("...s" + (1 * defaultZoomStep));
}) : ''
                : '';

And that works fine.

Clicking ‘Fit to screen’ button fits maps to screen:

var zoom = (Math.min(this._inner.width() / this._mapWidth,
this._inner.height() / this._mapHeight) * mapZoomFudgeFactor);
this.zoomTo(zoom);

However font gets scaled too. Is there any way stopping it?