See also note on:

  • Concurrency


Beej’s Guide to Network Programming Linux Kernel Development build the smallest possible linux system

Serial port programing termios

ioctl fcntl

kernel newbies labs download kernel source. wow it compresses well

Building the kernel.

make menuconfig Can git diff to see changes? Poking around General setup

currnt config cat /boot/config-$(uname -r) Rbuilding a kernel for ubuntu fakeroot?

hmm. I needed to turn off debian certs

process management memory management virtual file system device drivers arch

dmesg man dmesg dmesg -n 5 st level dmesg --level=emerg,alert,crit,err,warn

smbios smbus

man sysfs.5

procfs sysfs debugfs

sudo ls /sys/kernel/debug
sudo cat /sys/kernel/debug/dell_laptop/rfkill
ls /sys/kernel

htop. K toggles kernel threads. Kill signals. cpu affinity. l open file, x file lock, s strace, tree view, space tag process, c tag and children, name search, name filter, strace is very fun. needs sudo

Linux Boot

bios real mode

bootloader grub

acpi Advanced Configuration and Power Interface spec. hmm special registers. AML machine language bytecode APM is old deprecated method. Many info tables. spec

kernel parameters. `sysct

sysctl -a
ls /boot # some intertestying styuff in here
# kernel symbol table
# vmlinuz is the kernel
# initrd is iniital ram disk
# efi folder. This is where efi partition is mounted
# grub folder. grub.cfg has the grub options in it. x86_64-efi has grub modules

initramfs vmlinux startup_32 start_kernel generic initialization hmm. <asm.h/..> stuff is found here

/sbin/init, pid 1 - often systemd.


APIC advanced programmable interrupt controller Interrupt Request

cat /proc/interrupts
#watch -n0.1 --no-title cat /proc/interrupts

# IR-IO-APIC   17-fasteoi   idma64.1, i2c_designware.1
# IR-PCI-MSI 32768-edge      i915


Scheduling even older scheduler old scheduler. the SCHED_NORMAL default scheduler kernel/sched folder




available vs free. free is free, available is free + cache swap. vm.swappiness

File System


df # "disk free"
df -a # see everything
df -i /dev/nvme0n1p7 # inodes
ls -i /bin/true # inode of file
# debugfs
du # "disk usage"
ncdu # optionally installable eaiser to read netcurses disk usage

fsck - file system check


inode + filesystm idntifies. inode is per filesystem ext4

virtual file system - an abstraction so you can plug in different actual file systems sysfs devfs debugfs tmpfs - /tmp may be backed by ram journal. physical journal. write entire copy of data. expensive. logical journalling just journal metadata File systems: - extents are contiguous ranges of blocks. 4 can be stored in inode. delayed allocation. badblacks blkid dumpe2fs e2fsck e4defrag debugfs

copy on write filesystems fsync

rsync - remote sync


block superblock simplefs - a simple file system for Linux

MBR bootsector code. 4 partition entries in classic. CHS addressing. anyiquated addressing GPT. logical block addressing table header. glonal identifier for partition type. Ok. A pretty simple system


logical interface AHCI is a form factor

what is in / /var /run /boot /sys /proc /etc /dev /home /root /mnt /media /lost+found /bin /sbin /srv /swapfile /usr /tmp /lib



systemd units. service and others. mounts systemctl enable disable start stop conf files. You can override fields. udev

journalctl -f. Lotso f filtering options journald


insmod barebones command line to insert a module modprobe - does depndency lookup modinfo lsmod see what’s running

modprobe -c
ls /etc/modprobe.d/ # blacklist.conf
ls /lib/modules can lookup lots of lsmod stuff in here

sudo cat /proc/modules
ls /lib/modules/$(uname -r)

Hmm. I guess you really do have to buy into the kernel makefile. Fine.

mkdir /tmp/hellomod
cd /tmp/hellomod
echo "
 * hello-1.c - The simplest kernel module. 
#include <linux/module.h> /* Needed by all modules */ 
#include <linux/printk.h> /* Needed for pr_info() */ 
int init_module(void) 
    pr_info(\"Hello world 1.\n\"); 
    /* A non 0 return means init_module failed; module can't be loaded. */ 
    return 0; 
void cleanup_module(void) 
    pr_info(\"Goodbye world 1.\n\"); 
" > hello-1.c

echo "
obj-m += hello-1.o
PWD := \$(CURDIR) 
 make -C /lib/modules/$(uname -r)/build M=\$(PWD) modules

 make -C /lib/modules/$(uname -r)/build M=\$(PWD) clean
" > Makefile
cat Makefile
cd /tmp/hellomod
modinfo hello-1.ko
sudo insmod hello-1.ko
sudo rmmod hello_1
sudo dmesg


/proc/kallsyms # all symbols kernel knows about inside modules and elsewhere.

System Calls transitioning syscall code

The Definitive Guide to Linux System Calls some nice info on how syscalls happens. Interrupt x80, syscall instruction etc. VDSO - v

Porting OpenBSD pledge() to Linux


Linux Kernel Labs

kernel documentation


  • pthreads



no libc small pre-processed fixed size stack ~4kb no floating point?

processes are tracked processes own resources pids. ps



LSM linux security module selinux and apparmor are based on lsm

selinux is externally sandboxing a process. landlock is program developer voluntarily giving up access


firejail? cool links

landlock restrict ambient rights, global file system access discussion about landlock, incudes ocmparsion of some other features

openbsd pledge, unveil

security things in Linux v…

kmsan KernelMemorySanitizer, a detector of uses of uninitialized memory in the Linux kernel



Where should this section go? debugging ebpf - syscall errors to talk to bpf program Cannoli, e9patch C++ debugging tracing ptrace perf strace

PIN hardware counters

ebpf There’s a doucmentary? what huh a computerphile. the hype machine is insane

Seccomp filters


ubpf_test ubpf_plugiin

echo "
static int idouble(int a) {
    int temp = 0;
    while(a > 0){
        temp +=  a;
        return temp;

int bpf_prog(void *ctx) {
        int a = 3;
        a = idouble(a);

        return (a);
" > /tmp/hello.c
clang -O2 -target bpf -c /tmp/hello.c -o /tmp/hello.o
/home/philip/Downloads/ubpf/build/bin/ubpf_test /tmp/hello.o

#sudo opensnoop-bpfcc # see every open
#sudo execsnoop-bpfcc # see every exec
sudo bitesize-bpfcc #
sudo stackcount-bpfcc # see every stack trace. hmm.

execsnoop, opensnoop, ext4slower (or btrfs, xfs, zfs*), biolatency, biosnoop, cachestat, tcpconnect, tcpaccept, tcpretrans, runqlat, and profil bcc

from bcc import BPF

BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print() uprobes

cillium go library read modify and loadc



libuv libev libevent

Alternate shells

There were those beautiful pictures on that one shell. charmbracelet

warp - for mac, linux comin oh my zsh fish

Command Line Tools

awk jq Searching through stuff “similar to ack but faster” supercharged grep

grep -C 10

gnu parallel

diffoscope recursively diff?


Using Landlock to Sandbox GNU Make Limitting what make can access? Only should be allowed to access files it depends on explicitly in make rules pledge and unvil system calls


Maybe git deserves it’s own file git-bisect





Registry WSL powershell



microkernels mirage os


Microkernel Functional correctness But also binary level verification. Uses gcc but disassemblers result to verify





Booting is like a whole thing.

BIOS basic input output system - loads first sector and runs it. 16 bit code

UEFI There’s a python for uefi? /boot/efi/ ‘System Volume Information’ - windows stuff EFI/Boot bootx64.efi fbx64.efi mmx64.efi EFI/windows EFI/dell EFI/ubuntu BOOTX64.CSV grub.cfg grubx64.efi mmx64.efi - mokmanager. signing bootloader shimx64.efi some funky business on secure boot

ACPI - iasl. intel acpi dcompil;er/decompiler


GRUB grand unified bootloader

POST - power on self test

MBR master boot record. 512 bytes. See sector lisp, sector forth, sector games

bootloader stages - more an more complex systems

TPM secure boot So like malware can really fuck you by manipulating the boot process? I could see that.

Formal methods applied to booting

Formal Verification of a Modern Boot Loader 2018 - SABLE. Isabelle Towards a verified first-stage bootloader in Coq - 2020 - phd dissertation

SPIN 2009 Verified functional programming of an IoT operating system’s bootloader - 2021 Low* Riot Formally Verifying Security Properties for OpenTitan Boot Code with Uppaal - 2021

Model checking boot code from AWS data centers- 2020 - CBMC

File System

See also databases

Disk sectors. Disk rotation speed Disk Seek time

RAID - Redundant array of inexpensive disks. Copy data to multiple disks, or use error correction. RAID0 just interleaves disks for parallelism striping - put subsequent blocks on different disks RAID 1 - mirroring. Just rwwrite the same thing to multiple disks complete operating system from scratch

Hypervisors - like OS for OSes hypervisor from scratch 5 Days To Virtualization: A Series On Hypervisor Development

How do programs start?


Memory management

Scheduling - interrupts

Stuff proc documentation

cat /proc/self/mems
cat /proc/self/status query OS info as sqlite virtual table