It’s very easy to install any version of Python using pyenv. I would be careful not to change the system Python in 16.04 because it will probably mess up apt.
As soon as you start needing C libraries, you start getting minor issues here and there. Numpy modules in particular seem prone to issues -- even on a bog-standard x86 desktop. It's got to be much worse on the very niche Nvidia SoC.
