syzkaller: Kernel Fuzzing

Automated syscall fuzzing to find kernel bugs

What is syzkaller?

syzkaller is an automated kernel fuzzer developed by Google. It generates random sequences of syscalls and executes them in a VM, monitoring for crashes, hangs, memory errors (via KASAN/KCSAN), and other anomalies.

Since 2015, syzkaller has found thousands of kernel bugs reported on syzbot.

Architecture

syz-manager (host)
├── Manages VM pool (QEMU/GCE/VMware)
├── Generates corpus of interesting syscall sequences
├── Mutates programs: add/remove/modify syscall arguments
├── Tracks coverage via kcov (kernel coverage instrumentation)
└── Reports crashes via email/dashboard

syz-fuzzer (in VM)
├── Executes programs (syz-executor subprocess)
├── Measures kcov coverage
└── Reports new coverage / crashes to manager

syz-executor (in VM)
├── Runs as root
├── Executes actual syscalls (one program at a time)
└── Monitors for crashes (KASAN, UBSAN, KCSAN, general panics)

kcov: kernel coverage instrumentation

syzkaller uses kcov to guide fuzzing toward new code paths:

/* CONFIG_KCOV=y: instrument every basic block */
/* Each kernel function gets instrumented: */
void my_function(void)
{
    __sanitizer_cov_trace_pc();   /* kcov inserted this */
    /* ... kernel code ... */
    if (condition) {
        __sanitizer_cov_trace_pc();   /* another edge */
        /* ... */
    }
}

/* Userspace (syz-executor) reads coverage via: */
int kcov_fd = open("/sys/kernel/debug/kcov", O_RDWR);
ioctl(kcov_fd, KCOV_INIT_TRACE, COVER_SIZE);
void *cover = mmap(NULL, COVER_SIZE * sizeof(unsigned long), ...);
ioctl(kcov_fd, KCOV_ENABLE, KCOV_TRACE_PC);

/* Execute syscall */
syscall(...);

ioctl(kcov_fd, KCOV_DISABLE, 0);
unsigned long n_pcs = cover[0];
/* cover[1..n_pcs] = PC addresses of executed kernel code */

Syscall descriptions (syzlang)

syzkaller uses a description language (syzlang) to model syscall arguments:

# sys/linux/socket.txt (excerpt):
socket(domain flags[socket_domain], type flags[socket_type], proto int32) fd[sock]

connect(fd fd[sock], addr ptr[in, sockaddr], addrlen bytesize[addr])

# Flags:
socket_domain = AF_UNIX, AF_INET, AF_INET6, AF_NETLINK, ...
socket_type = SOCK_STREAM, SOCK_DGRAM, SOCK_RAW, SOCK_SEQPACKET

# Structures:
sockaddr_in {
    family  const[AF_INET, int16]
    port    proc[20000, 4, int16be]  # port in range [20000, 20004]
    addr    ipv4_addr                # special IPv4 address type
    pad     array[const[0, int8], 8]
}

This allows syzkaller to generate valid-ish syscall sequences rather than completely random bytes.

Running syzkaller locally

# Requirements: Go, QEMU, Linux kernel with kcov

# 1. Build a fuzzing kernel:
cat >> .config << 'EOF'
CONFIG_KCOV=y
CONFIG_DEBUG_INFO=y
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_UBSAN=y
CONFIG_UBSAN_SANITIZE_ALL=y
CONFIG_FAULT_INJECTION=y
CONFIG_FAULT_INJECTION_DEBUG_FS=y
CONFIG_FAILSLAB=y
CONFIG_FAIL_PAGE_ALLOC=y
CONFIG_FAIL_MAKE_REQUEST=y
CONFIG_LOCKDEP=y
CONFIG_PROVE_LOCKING=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_FORTIFY_SOURCE=y
CONFIG_HARDENED_USERCOPY=y
CONFIG_RANDOMIZE_BASE=y
EOF
make -j$(nproc)

# 2. Build syzkaller:
git clone https://github.com/google/syzkaller
cd syzkaller
make

# 3. Create config (syzkaller.cfg):
cat > syzkaller.cfg << 'EOF'
{
    "target": "linux/amd64",
    "http": "127.0.0.1:56741",
    "workdir": "./workdir",
    "kernel_obj": "/path/to/kernel",
    "image": "/path/to/vm.img",
    "sshkey": "/path/to/id_rsa",
    "syzkaller": ".",
    "procs": 8,
    "type": "qemu",
    "vm": {
        "count": 4,
        "kernel": "/path/to/bzImage",
        "cpu": 2,
        "mem": 2048
    }
}
EOF

# 4. Run:
./bin/syz-manager -config syzkaller.cfg

# 5. Open browser: http://127.0.0.1:56741
# Shows: corpus size, crashes found, coverage map

Reproducing crashes

When syzkaller finds a crash, it generates a reproducer:

/* C reproducer (generated by syz-repro tool): */
#define _GNU_SOURCE
#include <unistd.h>
#include <sys/syscall.h>
#include <sys/socket.h>
/* ... */

int main(void)
{
    /* Sequence of syscalls that triggers the bug: */
    int sock = socket(AF_INET, SOCK_RAW, IPPROTO_ICMP);
    /* ... more syscalls ... */
    return 0;
}

# Compile and run reproducer:
gcc -o repro repro.c && sudo ./repro
# Should trigger the same kernel crash

# Or use the syzlang program directly:
./bin/syz-execprog -executor ./bin/syz-executor -repeat=10 crash.log

syzbot: continuous fuzzing

syzbot runs syzkaller 24/7 on upstream kernels:

# Search for bugs affecting your code:
# https://syzkaller.appspot.com/upstream

# Each bug report includes:
# - Title: e.g., "KASAN: use-after-free in sock_destroy"
# - Kernel version and commit
# - C reproducer
# - syzlang reproducer
# - Stack trace

# To report that you've fixed a bug:
# Reply to the syzbot email with: #syz fix: <commit subject line>
# syzbot will verify and close the bug

Triage and analysis

# Decode a crash address:
./scripts/faddr2line vmlinux sk_destruct+0x123/0x456
# → net/core/sock.c:1857

# Find the guilty commit (bisect):
git bisect start
git bisect bad HEAD
git bisect good v5.15
git bisect run ./scripts/faddr2line vmlinux <addr>

# Analyze a core dump with crash:
crash vmlinux core.dump
crash> bt      # backtrace of crashed process
crash> dis -l sock_destroy  # disassemble with source lines