Reddit mentions of Groups and Symmetry (Undergraduate Texts in Mathematics)

Weyl's symmetry is what you're looking for. The next step up from this would require some group theory, since mathematicians interested in symmetry usually study symmetry groups of objects or spaces. I have not read it but this book looks like a good next read, at least the first four(ish) chapters. Another possibility is Armstrong's book, though I'm not familiar with this book either.

This falls into the branch of mathematics called Group Theory. If you think about it, your question has nothing to do with matrices, but instead is just a question about permutations.

You can ignore the matrix structure of your 2x2 matrix and just think about permutations of four elements (abcd). There are 24 permutations and the collection of permutations forms a Group (called S4, or the Symmetric Group on 4 elements).

S4 (in fact any symmetric group) can be generated by just two elements -- an element that permutes exactly two items, and an element that cycles all 4 items

So, for instance, from the permutation that sends (abcd) to (bacd) and the permutation that sends (abcd) to (bcda) you can generate all permutations. Note that you have to use a cycle here (something like a->b->c->d->a, or a->c->d->b->a)

Group Theory is a huge topic, but was originally developed to study questions about permutations.

You can read more about Symmetric Groups here http://mathworld.wolfram.com/SymmetricGroup.html or here https://en.wikipedia.org/wiki/Symmetric_group but if you've never been exposed to Group Theory at all, then those references may be a bit dense. If you're interested, I'd recommend you start with with something like https://www.amazon.com/Groups-Symmetry-Undergraduate-Texts-Mathematics/dp/0387966757 It starts out with (almost) your exact example -- the symmetries of the tetrahedron. Note that a tetrahedron has 4 sides, so every time you rotate (or flip inside-out) a tetrahedron, you're just permuting the 4 sides. It's not quite the same as your example because you can't get all 24 permutations of the 4 sides by rotating and flipping -- you only get 12 of them. The group of those 12 permutations is called A4 (or the Alternating Group on 4 elements). It is a sub-group of S4 (the one you are interested in).