Dagger 2 - Bind into map of a dependent Component

Question

I would like to know if or how it is possible to contribute into a map of a parent component.

This is my setup of the parent component and its graph:

@Singleton
@Component(modules = AppModule.class)
public interface AppComponent {
    @Component.Builder
    interface Builder {
        @BindsInstance
        Builder application(Application application);

        AppComponent build();
    }

    ViewModelProvider.Factory plusViewModelFactory();
}

@Module
public abstract class AppModule {
    @Multibinds
    abstract Map<Class<? extends ViewModel>, ViewModel> bindViewModels();

    @Binds
    abstract ViewModelProvider.Factory bindViewModelFactory(MyViewModelFactory myViewModelFactory);
}

@Singleton
public class MyViewModelFactory implements ViewModelProvider.Factory {
    private final Map<Class<? extends ViewModel>, Provider<ViewModel>> creators;

    @Inject
    MyViewModelFactory(Map<Class<? extends ViewModel>, Provider<ViewModel>> creators) {
        this.creators = creators;
    }

    @Override
    public <T extends ViewModel> T create(Class<T> modelClass) {
        // ... Irrelevant code
    }
}

Second I want each Activity to have a Component that depends on the AppComponent:

public class StartActivity extends Activity {
    @Module
    static abstract class StartModule {
        @Binds
        @IntoMap
        @ViewModelKey(StartViewModel.class)
        abstract ViewModel bindStartViewModel(StartViewModel startViewModel);
    }

    @StartScope
    @Component(dependencies = AppComponent.class, modules = StartModule.class)
    interface StartComponent {
        void inject(StartActivity startActivity);
    }

    @Inject
    ViewModelProvider.Factory viewModelFactory;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        StartComponent component = DaggerStartActivity_StartComponent
                .builder()
                .appComponent(((MyApplication) getApplication()).getAppComponent())
                .build();
        component.inject(this);    
        super.onCreate(savedInstanceState);
        System.out.println(viewModelFactory.toString());
        // ... Irrelevant code
    }
}

The bound ViewModelProvider.Factory of the AppComponent is correctly injected into this StartActivity but obviously empty, as the AppModule simply doesn't bind any providers of any ViewModel itself. What I would like to happen though is that providers of ViewModels bound later by each Activity are put into the same Map that the factory uses. Either I'm not looking at this problem from the right angle or there is a feature missing in Dagger 2 for this purpose.

Optionally I could have gone with declaring my Activity's component as a subcomponent but then I lose the modularity.