Reddit mentions of LSC A Book of Abstract Algebra

Chartrand's Introductory Graph Theory is the best intro I know of, good for self-study. (BTW, if you're interested in modern algebra, I recommend Pinter's intro. If it's not affordable or findable through a library, you'll probably be able to find it online.)

To answer your broader question, The Elements of Computing Systems: Building a Modern Computer from First Principles should give you a lot of confidence on how computers work.

If you're reading Structure and Interpretation of Computer Programs, feel free to skip any math-y examples. It's written for MIT students whose prereqs are still fresh in their minds. Here's an interesting recent interview with one of the authors, where he discusses the foundations of computer science and what a programmer might want to know. (The notion of abstraction, etc.)

I love Pinter's abstract algebra. I haven't read Stewart's book, so I don't know how it compares, but I highly recommend Pinter.