Android Custom Keyboard view is displaced to top when dismissing

Question

I am facing an issue with custom keyboard (built with InputMethodService). Below are the steps to reproduce the issue

1. Lanuch keyboard(while focusing on Edit text)
2. Click Home button/other launcher icon
3. Keyboard is displaced to top before dismissing view

I have tried with android:softInputMode="adjustNothing" and android:softInputMode="adjustPan" but the issue happens to exist.

If you have a solution for this, please add the solution as a comment

The below is the manifest structure

 <application
        android:allowBackup="true"
        android:fullBackupOnly="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:roundIcon="@mipmap/ic_launcher_round"
        android:supportsRtl="true"
        android:theme="@style/Theme.SampleKeyboard">
        <service android:name=".service.KeyboardService"
            android:exported="true"
            android:windowSoftInputMode="adjustNothing"
            android:permission="android.permission.BIND_INPUT_METHOD">
            <intent-filter>
                <action android:name="android.view.InputMethod" />
            </intent-filter>
            <meta-data
                android:name="android.view.im"
                android:resource="@xml/method" />
            <meta-data
                android:name="android:windowSoftInputMode"
                android:value="adjustNothing" />
        </service>
    </application>