Reddit mentions of The Electromagnetic Field (Dover Books on Physics)

It seems to me that introductory electromagnetism is, physically, very simple.

If the subject is difficult, I suspect it has more to do with the math than the physics. Unlike introductory mechanics, most problems in E/M rely heavily on vectors and vector calculus (and for many students E/M is also a first introduction to other more sophisticated mathematical ideas, like Laplace's equation and coordinate transformations).

As far as introductory level books go, though, I think Griffiths handles the added mathematical rigour of E/M quite well. Griffiths explains his math in great detail throughout the text, and chapter 1 is a thorough and complete, but straightforward and simple, treatment of vector calculus; I recommend that you study it in great detail (and work many problems) before continuing to the physics. Preparation in linear algebra is probably also helpful as well.

Also, keep in mind that there are several approaches to electromagnetism. As I recall, Griffiths develops the theory more or less historically, and only makes the connection with special relativity in the final chapters. If you want to look at the ideas from another angle, you could try a book like Purcell or Schwartz, which use special relativity to derive magnetism as a theoretical, rather than experimental, result. Personally, I find this approach more elegant, interesting, and even a little easier; nonetheless, understanding both approaches is important in the long run.

Edit: By the way, another book to consider is Shadowitz (I have only read the first 5 chapters, and I still recommend it on that basis alone). Shadowitz develops the basic theory very logically and consistently: chapters 2 through 5 cover the divergence and curl of E and B (one chapter each). At times the explanations are lengthy, but this might be useful for a struggling student.