That’s a clever way of doing it, yeah.
From UI side, I’d probably not autofill, but add autocomplete (e.g. using <datalist>¹). This way you can still have browser autofill the user’s address if saved (do also set the autocomplete² props correctly!), and suggest one or multiple variants you have in your database, and let the user fill it out manually still, without having to find and click “My address isn’t here”.
¹<datalist>: https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/...
(quick note re: compat; Firefox does support it for the text fields, check the table at the bottom)
²autocomplete: https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/...