Android differentiate between navigate with argument and navigateUp

Question

I am using the Android Navigation component moving from fragment to fragment in my app

So, I have a fragment defined something like this

<fragment
    android:id="@+id/my_fragment_dest"
    android:name="com.mycompany.myapp.view.fragment.main.MyFragment"
    tools:layout="@layout/my_fragment" >
    <argument android:name="doSpecialThing"
        app:argType="boolean"
        app:nullable="false"
        android:defaultValue="false"/>
</fragment>

and a start fragment like this

<fragment
    android:id="@+id/home_dest"
    android:name="com.mycompany.myapp.view.fragment.main.StartFragment"
    tools:layout="@layout/start_fragment">
    <action
        android:id="@+id/action_home_dest_to_my_fragment_dest"
        app:launchSingleTop="true"
        app:destination="@id/my_fragment_dest" />
</fragment>

and then in onViewCreated of MyFragment I check the value:

override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
    super.onViewCreated(view, savedInstanceState)
    doSpecialThing =
        arguments?.let { MyFragmentArgs.fromBundle(it).doSpecialThing} ?: false
}

This fragment can be called from the "start" screen, in which case I want to do the special thing, so I call

    val bundle = bundleOf(
        "doSpecialThing" to true
    )
    findNavController().navigate(
        R.id.action_home_dest_to_my_fragment_dest,
        bundle, animationThing()
    )

but it also calls other fragments, and when they are finished they go back to it calling

    findNavController().navigateUp()

But for some reason, although navigateUp doesn't include any bundle, and the default in the navigation xml file is "false" I still get "doSpecialThing" set to "true"

I would prefer not to explicitly call the navigate with a bundle set to false for all the "back" cases - is there a way to make it clear tha tI got to this fragment via "back" or "navigateUp"

Thanks