In my personal opinion, while the flexibility of the old XUL addons was amazing, the two big issues are compatibility and performance.

Compatibility: these addons could be broken very easily because they could depend on almost anything, and with the monthly release cycle, it is very difficult for mod authors to keep up. For instance, some addons would work by taking a core browser function written in JS, convert it to a string, run a regular expression to edit the string, then use eval to create a new function to replace the old one. In some release, the syntax of the "convert a function to a string" output changed slightly and it broke these addons, because it broke the regexp they were using.

Performance: XUL addons could do all sorts of things that are horrible for performance, and there was no real way for a user to tell what was causing it, because the addon wasn't isolated in any way. I ran into somebody who was having severe performance issues because the browser was generating colossal amounts of garbage for no reason. It eventually turned out that on a whim they'd installed a "LaTeX the World" addon, which would look for LaTeX typesetting instructions on pages and replace it with the nice looking output. The problem was, the way it worked was that every 10 seconds or so it would convert the entire contents of every single tab you had open into a zillion strings, search those strings, then throw them out.

The problem isn't security per se, it's compatibility. Exposing all the browser internals to extensions means that all the internals are part of the platform's public API and it's almost impossible to change anything. A lot of HN users will be like "that's fine, software should be finished, I don't want any more features", but things like performance and especially security require ongoing maintenance. The particular thing that killed off Firefox's old extension model was that it blocked migration to a multi-process architecture, which was clearly necessary even at the time and became even moreso when Spectre showed up a couple years later. "Warning cones and blood red messages" do not solve this because a vulnerable architecture exposes all users to exploitation, not just those who choose to use sketchy extensions.

(Also we know from long experience that "warning cones and blood red messages" don't in practice suffice to prevent end users from being exploited, but that's a separate issue.)

For me after they started limiting what extensions can do, firefox became a lot more stable. Before that I was using chrome because of firefox instability with extensions I wanted to use. After they got in isolated process for tabs and limiting extension permissions it actually improved quite a lot and became stable.