Yes, there is the "speaking" part inferred, but the grammar still does not work like that. As a supposedly educated English native speaker, you should know better.
If I said: "As an idiot, you're wrong", I could not possibly claim that I called me an idiot. Why suddenly the subject would change in the very same sentence structure?