The couple of dictionary definitions I've looked at describe a fine as a fee, penalty or punishment for doing something that you shouldn't have or breaking a rule (which may or may not include breaking a law). They certainly don't mention that it has to be a court of law imposing it.
So I think it's perfectly acceptable to say, for example, an imposition of a financial penalty for not returning one's library book on time is a fine, regardless of whether a law has been broken.
I spent a lot of time in court when I was younger. In the Magistrates' Court, it was generally referred to as a fine, whereas in Crown Court the term was often "financial penalty". I believe the concept was brought to us by William the Conqueror with the notion of a "finished" payment to end a dispute, from
finis, so it could be done without a court case, I suppose. I can't see a library book law, so can't rule out that it is merely a fee for breaching the terms of an agreement that has become a "fine" through common parlance.
………….they do everything by the book.
... which they will throw at you if need be.