That's a falsehood pushed by the south to deny their support of slavery. This short video about it from prager University is helpful: https://www.youtube.com/watch?v=pcy7qV-BGF4
Here you can read South Carolina's actual declaration of Succession, where they make a point of spending most of the 13 pages talking about how their "right" of slavery was being infringed upon: http://www.teachingushistory.org/pdfs/DecImmCauses.pdf