Can't. It's a fixed 30 year mortgage at a 3.375% rate. Refinancing it will increase that by at least a point. Plus, I originally did get the loan through a local credit union, who then sold it to a consolidator who sold it to a big national bank (which happened to be Wells Fargo). Local credit unions generally don't hold onto mortgages. They don't have the economies of scale to service them efficiently.
My 'local' credit union does. They dont sell mortgages/loans to anyone and keep them 100% in house. Its why even after moving to Germany I keep that bank for US assets. They just dont seem as criminally motivated as other banks.