Living Systems_

The chroot Technique - a Swiss army multitool for Linux systems

Live USB stick

Ever ran into a Linux box that just won’t boot, even after making sure that BIOS settings are OK and that no major hardware errors are at hand?

Then you need to know about the chroot technique, which can be a real life saver.

For example, I managed to repair a Nanopore GridION device this way the other week, after the official method of re-installing it via an .iso file failed, which reminded me to make the effort of documenting these steps.

I came across the technique only after having used Linux as my daily driver for more than a decade (thanks Matt !), which means I think it is worthy of much more attention, given its usefulness. Thus I’m hoping to spread the word a bit through this post.

TLDR

In short, the idea is that if you can access the hard drive of a broken or non-bootable Linux machine (this could be done e.g. by booting from a Live USB-stick, or plugging in the hard-drive as an extra drive in another Linux machine), you will mount this hard drive in a way that you trick your current Linux session that this is the harddrive of the currently running system, which it is not of course.

The trick to get this to work is to create a file tree based on two things:

  1. The hard drive partition of the broken system that was mounted as the / folder on the host operating system.
  2. A set of special system folders from the currently running, temporary, operating system (the live stick or replacement computer), which are not normal directories on the disk, but rather are containing system information needed by the running linux system. These include /sys, /proc and a few others, which we will go into details about below.

Then what you do is that you run the chroot command on this newly assembled folder structure, meaning that your currently running linux session will replace its current file system with this new blend of folders, mostly from the old, broken, hard drive.

It is a bit like hooking up a malfunctioning car to externally provided power source so that you can at least access its instrumentation, check the mileage, any error codes on the display etc.

Anyways, what this lets you do is then to run various commands that could hopefully repair or diagnose your system, such as apt upgrade, dpkg-reconfigure for broken packages on Debian based systems, or something else, depending on the specifics of your system.

The process

It is good to have the full process for setting up the assembled folder structure and doing the chroot, in one place, so that you can easily find it when you need it. Optimally you would even print out a small cheatsheet with these commands, or have them stored in a text file on your “repair toolkit” USB stick, or similar.

Anyways, below you will find all the steps of the process:

  1. Boot up your replacement OS (A live USB stick, or other computer with the broken computer’s hard drive attached to it).
  2. Figure out which partition on the broken system’s hard drive holds the root file system, and which one holds the /boot folder.
    • I typically use gparted for that, which is available on most Ubuntu/Linux Mint based live disks.
    • You can also use command-line based tools like sudo fdisk -l
    • I typically try to mount each of the broken system’s partitions into the file navigator (they typically show up as mountable drives in Thunar i Linux Mint XFCE) to really make sure I select the right partition.
    • If you are not able to do that, you will have to do some guessing based on filesystem size and type, whether they are already available in /etc/fstab etc.
    • In this example, we will assume that the root folder is in partition: 
      /dev/nvme0n1p5
      … and that the /boot partition is in: 
      /dev/nvme0n1p3
      (These happen to be the case for the Nanopore GridION device I mentioned that I repaired this way recently).
  3. Create a folder somewhere in your currently running system’s file system, that will act as the new root folder, as well as the /boot folder inside that: 
    sudo mkdir /rescue
sudo mkdir /rescue/boot
  4. Mount the broken system’s main and boot partitions into these folders, e.g: 
    sudo mount /dev/nvme0n1p5 /rescue
sudo mount /dev/nvme0n1p3 /rescue/boot
  5. Then, mount in the required special folders from your currently running system, into the appropriate places in this new file structure (the mount command is on the form mount -t <file-system-type> -o <options> <device> <mount-point>) 
    sudo mount -t proc proc /rescue/proc
sudo mount -t sysfs sys /rescue/sys
sudo mount -o bind /dev /rescue/dev
sudo mount -t devpts pts /rescue/dev/pts
  6. Now we are ready to chroot, or “change rootfolder” into this newly created folder structure: 
    chroot /rescue /bin/bash -i
  7. And we’re done!

Now you can look around in the file system to try to figure out what is causing the system to not work, as well as use whatever file commands you need to try to repair the broken system,

In our case with the broken GridION device, we found some broken symlinks and empty initrd.img-* files in /boot, which indicated that a Linux Kernel update had probably got interrupted mid-way through during a package update or similar.

In this case, this was fixed by running sudo apt update && sudo apt upgrade, which complained about some broken packages, and suggested us to run sudo dpkg-reconfigure which finally fixed the problem.

Read more

Updates