Reddit mentions of OSPF: Anatomy of an Internet Routing Protocol

/u/LordBiff has the answer.

Discontiguous masks (that's the term for what you're asking about) are a thing. They used to actually work, just like the term mask (as opposed to length) implies. I tested this ages ago on a network with SunOS 4.1 servers and routers (running gated). It worked just like you'd expect.

John Moy discussed it in OSPF Anatomy of an Internet Routing Protocol

Subnet numbers usually were assigned to immediately follow the network prefix.
If there was a gap between the network prefix and the subnet number, the subnet
mask was termed discontiguous. An example of a discontiguous subnet mask is
using the fourth byte of a Class B network to indicate the subnet number,
resulting in a subnet mask of 255.255.0.255. The combination of VLSMs and
discontiguous subnet masks was a bad one, for two reasons. First, certain
assignments of discontiguous subnet masks could result in multiple subnets
matching the same number of bits, making the concept of best match ambiguous!
Second, common routing table lookup algorithms, such as Patricia (see
Section 2.1), could not handle discontiguous masks efficiently. With
discontiguous subnet masks already discouraged by RFC 922, the introduction of
VLSMs made them virtually unsupported. Discontiguous subnet masks are now
prohibited by the latest router-requirements RFC [12].

That last bit is a reference to RFC1812 10.2.2:

It is possible using arbitrary address masks to create situations
in which routing is ambiguous (i.e., two routes with different but
equally specific subnet masks match a particular destination
address). This is one of the strongest arguments for the use of
network prefixes, and the reason the use of discontiguous subnet
masks is not permitted.

how deep in the weeds do you want to get into OSPF? do you want to understand enough just to be able to troubleshoot and bring up a new router, or [re]design the entire network?
John Moy's book should still be the standard; he wrote the RFC.

If you want to actually design a network, I still love Russ White's Cisco Press book on Optimal Routing Design.

If you just want an overview, the Cisco OSPF design guide can give you the nomenclature. Though the examples are IOS, the principles carry over.

Along with /u/totallygeek recommendations, if you're going to deploy OSPF onto a network, I would add:

• Figure out what you're trying to gain from using OSPF that you currently don't have in your current network. Redundancy? Faster convergence? Building out a WAN?
  
• Layout the IP addressing FIRST. You're designing an IP network... worry first about the IP addressing before speeds and feeds.
  
• OSPF, IM(strong)O, should be used modularly. Hand in hand with your IP addressing, you really should take advantage of building different areas. Don't go overboard and create multiple areas just for the heck of it, but don't get lazy and put everything into area 0 either.
  
• Decide how you will split up your network. Will it be based along functional business units (i.e., financing, warehouse, engineering), location based (floors, buildings, cities, geographic regions) or in some other way.
  
• Be stringent with what you advertise inter-area, either using access-lists/routing filters as suggested, or better yet, with the more flexible route-maps.
  
  Personally, I would stay away from virtual links as your abstracting what should be physical links onto harder-to-troubleshoot virtual links. I would also keep the area IDs the same as the top level network. For instance, if I was using 172.16.0.0/16 as the supernet for a building, the OSPF area ID would also be 172.16.0.0/16, but that's just me. There is more than 1 way to build a good network and as long as you are consistent on a logical design, that's what matters.

I would also look at the vendor agnostic books on networking subjects. This book on OSPF and this one on BGP actually helped me understand things a lot. They might be older books but the protocols themselves haven't changed too much in that time.

A few classics:

Radia Perlman: Interconnections

Mike Padlipsky: The Elements of Networking Style

John Moy: OSPF: Anatomy of an Internet Routing Protocol

Interconnections in particular is a real classic that will help to give you a grasp on why things are the way they are now, and to recognize changes in our industry and how they relate to past iteratations (à la RFC 1925 Rule 11.

The OSPF book gives you insight into why certain design decisions and tradeoffs were made at the time (late 1980s) with regards to hardware and resource availability.

Mike's book is hilarious and scathing and reminds us that it is possible to overanalyze the technical details of protocol design and implementation.

Yes, that is a good one. I would also not waste a ton of time on routing TCP/IP vol I & II but they are probably worth using as a reference as needed. I would add Moy's routing book to the list. It is as good as Halabi's and helped me out quite a bit on my first CCIE.

As others have mentioned, CCNA will get your started with the basics. After that it's going to depend on what your job's focus is. These are my top 3 recommended readings for anyone getting into networking.

• TCP/IP - TCP/IP Illustrated Vol 1.
  
• OSPF - OSPF: Anatomy of an Internet Routing Protocol
  
• BGP - Internet Routing Architectures