We used Firebase A/B a lot and we noticed that sometimes users may not get their remote config values due to various reasons(eg:network error, slow network),If we upload key event that we want to use in A/B testing at this time, Will the event include in A/B analytics? Which group will the event belong?
We hope the event can be include in the right group only when the user get the right A/B remote config value.
An example scene:
1.I Create a A/B Test named "Login UI" and use remote config "new_login_ui" to test a new login UI.
new_login_ui: default value is "":old UI, GROUP A:"false":old UI, GROUP B:"true":new UI
2.I use an event "login_success" as the test goal metric.
3.A user entered my app but get "new_login_ui"'s default value "", so app give the user old UI, after user login success, I upload the "login_success" event.
4.Next time user entered my app, I found user get "new_login_ui" value "true", so the user is grouped to GROUP B.
Then the confuse occur, will the event "login_success" with old UI behavior included in GROUP B which is the new UI group?
If it's true, then the A/B result is ruined because the login_success is belong to old UI group.
If the app receives the default parameter value, probably, the user is not yet included in the experiment. If the user satisfies all the targeting conditions and it calls fetch > activate or fetchAndActivate method on Remote Config, you should see a conditional user property (e.g. firebase_exp_id, id being the experiment number) in the analytics verbose logs. To validate your implementation, you may enable the verbose logging and debug mode in your app, then use the DebugView console.
Once the conditional user property is set, the SDK will start to track the events that will be logged by the user.
You may use the Remote Config loading strategy 2 if you're loading Remote Config values from experiment. This will give A/B Testing additional time as the user is placed into an experiment and the experimental values are applied.