Reddit mentions of Understanding the Linux Kernel, Third Edition

You're asking a very, very very large question here. I'll try and directly answer this but I may go off on some detail. Also these details will be summarized so anyone reading this, you can pick it apart. It isn't meant to be exact.

> What exactly is a packet?

Its just a chunk of data. This chunk of data contains all types of things it will need to get where it is going safely. Source. Destination. Checksums. Settings. And of course the data within. In TCP this chunk of data can be up to 64 kilobytes. However before this chunk of data hits the physical wire, it has to be split up into its Maximum Transmissible Unit. Which is just a fancy way of saying "this is the agreed upon maximum chunk of data size for this network". Typically the MTU is around 1500 bytes. It is also known as a "frame" since an MTU of 9000 is also called "jumbo frames"

Think of an MTU size like the gears in a car. A lower MTU size is like a lower gear. It gets you going faster, since once you have 1500 bytes of data you can fire that packet off, instead of sitting around waiting for 9000 bytes of data to fill up.

The downside to this is that each packet sent requires a series of interrupts. And interrupts can be costly (especially back in the day!). When an interrupt fires, it goes to the kernel and says "HEY! I NEED SOMETHING DONE!". The kernel says "alright little buddy, lets get you what you need." and then the kernel assigns a core to handle the interrupt. When the interrupt is being handled, nothing else on the core gets done. It has a full lock on that core until the job is done or the interrupt itself (depending on the type of interrupt) is interrupted itself. And yes before you ask, certain types of interrupts cannot be interrupted.

So you can imagine back in the day of single core systems, this could really suck. Because if you have a single core servicing interrupts for 1500 MTU sizes instead of 9000 MTU sizes, you're servicing 6* as many interrupts. Which means your processor literally stops what it is doing to handle the interrupt, and then once done goes back to what it was doing. I could go into kernel preemption and interrupting interrupts but I won't. Too long.

These days interrupts aren't so bad as we have many many cores to service them. If you've ever run into the irqbalance service, it is responsible for basically round-robining interrupts across all cores (cat /proc/interrupts to see).

> how do different linux OSes manage them

All linux distros use some version of the Linux kernel. How the network is managed can vary little to a lot from version to version. There are also local settings for the kernel. Things in sysctl like packet buffer sizes. You also have hardware buffers like the NIC ring buffer/NIC buffer/send/receive queue (I've seen it referred to as all of those) which is set by tools like ethtool. Then you have deeper level settings like congestion algorithms, congestion windows, etc etc. There are entire books written on this. Two of my favorites are Understanding the Linux Kernel and Understanding Linux Network Internals. But to crudely summarize the answer here, the kernel manages all of this for the most part.

> how this packet of data is physically sent over the internet

Basically, buffers and interrupts (hardware/software). These chunks of data sit in a buffer somewhere until either the buffer fills up, and the owner of the buffer sends an interrupt to the kernel saying "hey I need attention, I'm running out of buffer space, can you pick up some packets for me and free up some space?" to software just repeatedly polling a memory address to check for new packets in said buffer. Things like kernel bypass allow userspace programs to read the memory directly on the NIC. But usually a packet goes from the NIC, to the kernel, to the application.

The internet is really just a big conga line of buffers and interrupts. There are tons of places a packet can be dropped. Throughout the process a packet moves through the "7 layers of the OSI model" moving from the physical medium all the way up to the end user application.

The details of how this works are endless. I've literally spent 5 hours in interviews talking about all this and how it works. So like I said your question is pretty huge :)

Yeah, I feel much the same way. I just look at the portions that interest me. I am not smart enough to understand all of it, so I just study and try to learn small parts individually. I bought Understanding the Linux Kernel. It is pretty good book. Some of it is over my head, but it covers the 2.6 kernel in depth. I'd recommend it.

I got a bookmark folder filled with resources just for the day I choose not being a lazy slob. Here's some:

https://www.kernel.org/doc/html/latest/ - Official docs

http://www.dit.upm.es/~jmseyas/linux/kernel/hackers-docs.html - Resource list including books and webpages

Some books i got from libgen:

Professional Kernel Architecture

Understanding the Linux Kernel, Third Edition

Linux Kernel Development (3rd Edition)

1. Understanding the Linux Kernel
   
2. http://www.tldp.org/LDP/lkmpg/2.6/html/
   
3. also relevant: http://eudyptula-challenge.org/

To learn about the user space APIs
A good intro to the inner workings
A fairly detailed reference of the inner workings

Understanding the Linux Kernel is the book that they recommend you read throughout the course, and it actually seemed to be really helpful to the people who read it. You could brush up on C and basic data structures since you'll be coding a lot in C; you also use x86 but you'll learn that during the class.

Besides that there's not much you could do since most of the material, if not all of it, will probably be new to you unless you've learned about system software/operating systems before. However, if you want a head start then you could start reading Understanding the Linux Kernel. Good luck, and may the Linux gods be on your side during MP3.

http://www.amazon.com/Design-Implementation-FreeBSD-Operating-System/dp/0321968972/ref=sr_1_1?ie=UTF8&qid=1418830375&sr=8-1&keywords=BSD

http://www.amazon.com/Understanding-Linux-Kernel-Third-Daniel/dp/0596005652/ref=sr_1_1?ie=UTF8&qid=1418830402&sr=8-1&keywords=kernel

http://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468/ref=sr_1_2?ie=UTF8&qid=1418830420&sr=8-2&keywords=kernel

http://www.amazon.com/Art-Linux-Kernel-Design-Implementation-ebook/dp/B00L2EBELM/ref=sr_1_3?ie=UTF8&qid=1418830446&sr=8-3&keywords=kernel

http://www.amazon.com/Professional-Kernel-Architecture-Wolfgang-Mauerer/dp/0470343435/ref=sr_1_5?ie=UTF8&qid=1418830461&sr=8-5&keywords=kernel

Most books are still made out of paper.

Try these : UTLK
and Linux Programming Interface

Understanding the Linux Kernel, Third Edition Writing my own kernel this semester for a class, this is the course textbook and has been an invaluable reference!

Three esxi servers, each with more than one NIC, and one separate vcenter server is a great starting point. This allows you to tune DRS, HA, and even fault tolerance. Once you get that down, you'll want to be able to tune VMs properly to run most effectively in a virtual environment.

I enjoyed reading these books, though some are "dated" now, the contents are still very relevant. They won't get you anywhere in particular by themselves, but when you combine them with the self-teaching nature of sysadmins I've previously described, these will generously add to your toolset.

HA and DRS deepdive
Sed & Awk

Mastering Regular Expressions. I use rubular.com often.

Pro Puppet

Anything by Bruce Schneier is usually worth your time.

Though I no longer administer a large number of Windows machines, I am a huge fan of Mark Minasi. The Server 2003 book was super helpful in building and maintaining Windows Domains.

I have an old edition of the DNS and Bind book kicking around somewhere.

Understanding the Linux Kernel has largely been useful to me when doing anything "close to the kernel". Not a good beginner's book.

I've never used an apache book, but I enjoyed the Varnish book. This definitely helped me.

Of course, these books don't cover everything, and those listed are relevant to my interests so your mileage may vary. You'll never go wrong teaching yourself new skills though!

EDIT: I forgot about the latest book I've read. I used tmux for a little over a year before purchasing a book on it, and it has improved my use of the program.

The best one for general OS design & implementation:

http://www.amazon.com/Operating-Systems-Design-Implementation-3rd/dp/0131429388/ref=sr_1_1?s=books&ie=UTF8&qid=1411043393&sr=1-1&keywords=operating+system+design+and+implementation

Linux-specific:

http://www.amazon.com/Understanding-Linux-Kernel-Third-Edition/dp/0596005652/ref=sr_1_1?ie=UTF8&qid=1411043351&sr=8-1&keywords=understanding+the+linux+kernel

http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/dp/0672329468/ref=pd_bxgy_b_text_z

It's really very easy. I've only written one linux driver for a new hardware serial port and a few other kernel modules and have written a couple dozen drivers for other operating systems.

This is my favourite linux book although I'm sure that have others, it's the only one I can name off the top of my head.

I would start off by adding a new API. That was my favourite exercize and only required a few lines of code.

edit: An Important thing to be able to do is to control the hardware without an OS. When you can do that, squeezing it into whatever is usually a simple task. Arduino is a nice learning platform for hardware.

No, they are not. For example, semaphores are often not kernel objects. And in any case, the kernel is written in a programming language. Anyway, I've always found https://www.amazon.co.uk/Understanding-Linux-Kernel-Daniel-Plerre/dp/0596005652 to be a good introduction to OS programming in general. In C.

har har on the boots...

And I'm not talking about how to use these things, I'm talking about how they work on a very low level. I know how to use grub. I now want to know how grub works (and rEFIt and bcd stuff) and what they are doing through every step. Think this, but for everything up until you load the kernel, or run ntoskrnl.exe, or whatever.

There is an OReilly book

http://www.amazon.com/Understanding-Linux-Kernel-Third-Edition/dp/0596005652

The Linux Device Drivers book can be downloaded for free online. I find it quite useful while going through device driver code. Also, the book Understanding the Linux Kernel by Daniel P. Bovet and Marco Cesati is pretty good.

The tutorials linked at the top-level README of the repo are great. If you want more detail, I'd recommend the books:

• Modern operating systems
  
• Understanding the Linux kernel
  
  Those are the ones we used in college and they're pretty good, though more dense and thorough.
  
  Good luck!

Understanding the Linux Kernel is great and, obviously, very specific to Linux.

​

Linus has cited that he read Bach's book about the design and implementation of UNIX as inspiration for the development of Linux.

​

Read both and really enjoyed them! I hope you like!

How much time do you have?
How much do you know related to low-level hardware and how things work close to the “metal”?

You can get a decent understanding by googling/looking at blogs that try to approach this subject. There are also a couple of book that are free to read on github.

If you want to go really deep I recommend a book https://www.amazon.com/Understanding-Linux-Kernel-Third-Daniel/dp/0596005652
Although it’s pretty old and refers to 2.6 kernel it’s one of the best books I’ve ever seen if you’re interested in the Linux Kernel (booting included).
After you learn/come across some of the fundamental notions explained in this book it’s going to be easier to grasp and understand other more advanced material or even look at the kernel source code yourself.

I strongly recommend Understanding the Linux Kernel, 3rd Edition: http://www.amazon.com/Understanding-Linux-Kernel-Third-Edition/dp/0596005652

Although it's based on version 2.6 of Linux, I think its level of detail is unmatched.