I think you are right to some extent, the Irish were definitely "racialised" in 19th century britain and considered essentially subhuman.
There was a potato famine in the Highlands of Scotland as well as Ireland, so it did hit Britain:
https://en.wikipedia.org/wiki/Highland_Potato_Famine
And we can see that "prompt and major charitable efforts by the rest of the United Kingdom ensured that there was relatively little starvation" - but that similar to in Ireland, "The terms on which charitable relief was given, however, led to destitution and malnutrition amongst its recipients".
I guess in the mainland UK the scale was far smaller, but it didn't stop an extremely laissez faire and heartless response from government authorities.