Error: RGB Terrain rendering in MapLibreGL base shifted vertically -10000

Question

I am developing a 3D terrain using Maplibre GL, i want to use my own DEM data to visualize the terrain. I have converted DEM with EPSG:3857 into RGB terrain by utilizing water-gis tools. The tools comply with the MaplibreGL equation: height = -10000 + ((R * 256 * 256 + G * 256 + B) * 0.1). Nonetheless, when i tried to visualize the raster-tiles into the raster-dem there is a vertical distortion occurred, it is shifted vertically, the picture as follows: enter image description here

To debug this error, i have tried to change the base of the equation from (-10000) to (0) but it turns out that evoke other error to the terrain. The zoom level can only enable until 12, after that the map returns blank white screen. The base still shifted vertically but not as significant when the base is (-10000), picture as follows: enter image description here

Other debugging effort is by customizing the encoding of maplibre, the code as follows:

 // Use a different source for terrain and hillshade layers, to improve render quality
              terrainSource: {
                type: 'raster-dem',
                // url: 'https://demotiles.maplibre.org/terrain-tiles/tiles.json',
                tiles: ['http://localhost:5173/BDGtiles/{z}/{x}/{y}.png'],
                tileSize: 256,
                encoding: "custom",
                baseShift: 10000,
                redFactor: 0,
                greenFactor:1,
                blueFactor:1
}

The terrain touches the ground with this method but the terrain surface getting rougher and abstract. image as follows: enter image description here

How can i repair the base shifting issue from -10000 to 0 without causing any other bug?

Thank You