How to protect routes from one provider, but allow another in Symfony 6?

Question

My system has two user providers. One is an AdminUser and the other is a regular User. I can't use roles to check access.Access should be determined by the user's instance.

I defined two different firewalls and settings. This helps to demarcate the entrance. But now an ordinary authorized user can open the main page in /admin or /admin/login

How can I do this without using roles. I tried to write custom_authenticators. But I'm stumped and not sure if this is right for me. My security settings now look like this.

providers:
    app_user_provider:
        entity:
            class: App\Entity\User
            property: email
    admin_user_provider:
        entity:
            class: App\Entity\AdminUser
            property: email
firewalls:
    dev:
        pattern: ^/(_(profiler|wdt)|css|images|js)/
        security: false
    admin:
        lazy: true
        pattern: ^/admin
        provider: admin_user_provider
        form_login:
            login_path: admin_app_login
            check_path: admin_app_login
            username_parameter: _email
            password_parameter: _password
            # where to redirect after success login
            default_target_path: admin_home
        logout:
            path: admin_app_logout
            # where to redirect after logout
            target: admin_app_login
    main:
        lazy: true
        provider: app_user_provider
        form_login:
            login_path: user_login
            check_path: user_login
            username_parameter: _email
        logout:
            path: user_logout
            target: user_login