Adding email address to account with phone number login attribute

I have a user pool that allows both email and phone number for creating an account, but not a username. My account creation flow in the app only allows users to choose one or the other to create their account. If a user has created an account with the email address (or phone number) and wants to switch to using the other, how can I allow that? I've tried manually adding and verifying a phone number to an account that already logs in with an email address. But even after adding that phone number I can't login with it.