Layout not shifting up when keyboard is open

Question

In my app, I have a few TextViews above an EditText. When the EditText is focused and the keyboard is open, I want all of the views to shift upwards so they are all still visible. I added android:windowSoftInputMode="adjustPan" to the Manifest file, and the views do shift up the first time I the EditText is focused. However, any subsequent time, all of the views stay in the same place, making the EditText covered by the keyboard. How do I make this work indefinitely?

Here is the XML for the EditText:

<EditText
    android:id="@+id/answer"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:ems="10"
    android:gravity="center_horizontal"
    android:textColor="#ffffff"
    android:layout_marginBottom="113dp"
    android:layout_alignParentBottom="true"
    android:layout_alignParentLeft="true"
    android:layout_alignParentStart="true"
    android:imeOptions="actionDone"
    android:singleLine="true"
    />