Ive owned a Nexus 6 since release ,,would going to a 5x be much of an upgrade?

Just looking to get some opinions ..my phone plan is soon to be up ...Ive had the Nexus 6 for 2 years and it's been a good phone for the time being ..But I believe its time to move on and I am looking for something smaller ( would prefer to stay with a stock android device aswell) not really looking to blow so much on a pixel..any suggestions or opinions? thanks!