When I answer a call with my phone it still defaults to the watch speaker.

I have a Nexus 6p phone and the Huawii watch. Love them both but when I answer a call with the phone it still defaults to the speaker on the watch. How can I set it up so when I answer on the watch it uses the watch speaker and with the phone the phone speaker?
Thanks