It's implied term that Bankers have the right to charge fees.
So in this sense, it is considered your mistake of did not pay attention to the fees and charge when you sign agreement to apply for the credit card.
The charge because they provide you the 'credit facilities', when you need the money, you can sign up to the amount limit you have, so this facility comes with a price, not considering whether you use or not... what's matter is you HAVE it, in the case when you NEED it.
Well, your credit rating is come down as we speak (imagine you're oweing bank money for how many years?) your record will hinder you from getting loans for car/home next time, or you have to pay much higher interest rate than 'clean borrower', because you already have the record of not paying back money, no matter what's the amount. So, if you have to pay higher interest, in the long run you're losing thousands instead of hundreds. So really, it's up to you whether you want to pay back or not...