You interact with websites using physical means, so it's an important aspect of how users experience them. If a website has tiny buttons and icons, it might be hard to use with a phone. On a large screen computer with a precision pointing tool it might be preferable that they are small. If the information density is ten words per 2000 vertical pixels, it's going to be a pain in the ass with a scroll wheel, not so much on a tablet where scrolling has almost friction-less inertia. Low contrast might be bad if you are using a phone, great if you're in a dark room.

If it's your job to improve the user experience, you're doing a bad job if ergonomics is not a consideration.