Reddit mentions of Understanding Linux Network Internals: Guided Tour to Networking on Linux

You're asking a very, very very large question here. I'll try and directly answer this but I may go off on some detail. Also these details will be summarized so anyone reading this, you can pick it apart. It isn't meant to be exact.

> What exactly is a packet?

Its just a chunk of data. This chunk of data contains all types of things it will need to get where it is going safely. Source. Destination. Checksums. Settings. And of course the data within. In TCP this chunk of data can be up to 64 kilobytes. However before this chunk of data hits the physical wire, it has to be split up into its Maximum Transmissible Unit. Which is just a fancy way of saying "this is the agreed upon maximum chunk of data size for this network". Typically the MTU is around 1500 bytes. It is also known as a "frame" since an MTU of 9000 is also called "jumbo frames"

Think of an MTU size like the gears in a car. A lower MTU size is like a lower gear. It gets you going faster, since once you have 1500 bytes of data you can fire that packet off, instead of sitting around waiting for 9000 bytes of data to fill up.

The downside to this is that each packet sent requires a series of interrupts. And interrupts can be costly (especially back in the day!). When an interrupt fires, it goes to the kernel and says "HEY! I NEED SOMETHING DONE!". The kernel says "alright little buddy, lets get you what you need." and then the kernel assigns a core to handle the interrupt. When the interrupt is being handled, nothing else on the core gets done. It has a full lock on that core until the job is done or the interrupt itself (depending on the type of interrupt) is interrupted itself. And yes before you ask, certain types of interrupts cannot be interrupted.

So you can imagine back in the day of single core systems, this could really suck. Because if you have a single core servicing interrupts for 1500 MTU sizes instead of 9000 MTU sizes, you're servicing 6* as many interrupts. Which means your processor literally stops what it is doing to handle the interrupt, and then once done goes back to what it was doing. I could go into kernel preemption and interrupting interrupts but I won't. Too long.

These days interrupts aren't so bad as we have many many cores to service them. If you've ever run into the irqbalance service, it is responsible for basically round-robining interrupts across all cores (cat /proc/interrupts to see).

> how do different linux OSes manage them

All linux distros use some version of the Linux kernel. How the network is managed can vary little to a lot from version to version. There are also local settings for the kernel. Things in sysctl like packet buffer sizes. You also have hardware buffers like the NIC ring buffer/NIC buffer/send/receive queue (I've seen it referred to as all of those) which is set by tools like ethtool. Then you have deeper level settings like congestion algorithms, congestion windows, etc etc. There are entire books written on this. Two of my favorites are Understanding the Linux Kernel and Understanding Linux Network Internals. But to crudely summarize the answer here, the kernel manages all of this for the most part.

> how this packet of data is physically sent over the internet

Basically, buffers and interrupts (hardware/software). These chunks of data sit in a buffer somewhere until either the buffer fills up, and the owner of the buffer sends an interrupt to the kernel saying "hey I need attention, I'm running out of buffer space, can you pick up some packets for me and free up some space?" to software just repeatedly polling a memory address to check for new packets in said buffer. Things like kernel bypass allow userspace programs to read the memory directly on the NIC. But usually a packet goes from the NIC, to the kernel, to the application.

The internet is really just a big conga line of buffers and interrupts. There are tons of places a packet can be dropped. Throughout the process a packet moves through the "7 layers of the OSI model" moving from the physical medium all the way up to the end user application.

The details of how this works are endless. I've literally spent 5 hours in interviews talking about all this and how it works. So like I said your question is pretty huge :)

I found the O'Reily Linux Networking book to be very informative.

https://www.amazon.com/Understanding-Linux-Network-Internals-Networking/dp/0596002556/ref=sr_1_3?s=books&ie=UTF8&qid=1543153401&sr=1-3&keywords=linux+networking

Also here
https://www.amazon.com/Practice-Cloud-System-Administration-Practices/dp/032194318X/ref=sr_1_1?s=books&ie=UTF8&qid=1543153468&sr=1-1&keywords=cloud+administration

When I started, the most memorable three resources I know I had were the following:

• Bjarne's book - I don't think any C++ developer can truly call themselves even intermediate until they have absorbed at least half the content in the book. I started off with his 3rd edition, which is way less intimidating and shorter, but I subsequently ran through this entire book after it came out. There are no shortcuts on here - you need to read it.
  
• Effective C++ 3rd Edition - I would almost require this one as it prevents any new C++ developer from getting caught in C++ gotchas. You should ideally follow this book up with his 4th edition afterwords. The reason why I recommended the 3rd first is because that book is much more newbie friendly. The 4th edition is targeted towards experienced C++ developers that already have had their feet wet with C++11/14 and want to figure out some best practices. Note the 3rd edition is for C++98 while the 4th is for C++11/14.
  
  After you're done with the two above required books, here are some useful readings:
  
  
• What Every Programmer Should Know About Memory - This is an absolutely essential reading whether or not you've taken any systems courses. It's the foundation of what you will be programming towards (optimizing CPU cache usage).
  
• 1024cores - I believe this guy works/worked at Google at one point, but his site is essential to understanding multi-threaded programming (which is essential in the field). Browse through his site and learn what you can.
  
• Linux Kernel Development - Robert Love (who also works at Google) has probably written the most concise and understandable book on the Linux kernel I've ever read, and I've run through the Daniel Bovet's book and Michael Kirrisk's. For one thing, they're 1,000 and 1,500+ pages, respectively. Secondly, all I've found in those two books that I didn't find in Robert Love's is the implementation details in some areas as well as the details on the scheduler. Robert Love's incredible descriptions on the bottom-half/tasklets were already more than effective for normal understanding. I believe the latter books were more detailed in the networking areas as well, but in that case, you're better off with Understanding Linux Network Internals.
  
  The above readings will probably be a solid 6-12 months to read and absorb assuming you spend a couple hours a day, but I think it'll be well worth it in the long run since this type of stuff sticks with you for a long time. I read the above around 2013, and I can still talk about the CFS/other schedulers, software interrupts, and how the CPU cache works. It'll also make converting to other languages even more of a breeze because you'll know how everything works underneath the hood.