At this point the cost of handling cash is way higher than handling cards and as no one in Sweden ever uses cash its no longer relevant at all anyway. Now many (maybe even most?) dont accept cash to avoid the cost of handling cash instead.
And you're mistaken if you think cash is cheaper for most stores. You risk theft (so need to pay for measures against that), you have straight losses from mistakes, have to spend time handling and counting cash, spend time depositing, spend time buying change, etc etc
Regarding cash, does your bank not charge a fee for depositing cash?
And you didn’t answer why you shouldn’t pay that fee.
Managing cash has costs too, they're just harder to quantify: you have to ferry it to a bank, you have increased risk of theft, fraud, and robberies, you need extra time to actually check the register etc.
And then you risk losing business if you don't offer card payments because it's just more convenient for most customers (you may like wise lose some if you don't take cash, but that's a vanishing market).