Fixing a broken Ubuntu upgrade

This article describes my experience upgrading from Ubuntu 22.04 to 24.04. It is not really a guide, but more a collection of problems that I encountered and which steps were needed to resolve them. Mainly it is for me to refer to if I run into this issue again, but it may be useful for you as well.

I was experiencing occasional freezes on Ubuntu 22.04. These would freeze up my desktop and also prevent me from going into TTY (using ctrl+alt+F3). Looking at /var/log/syslog, I saw errors related to DRM (direct rendering manager) in the kernel. One of my colleagues recommend that I upgrade to a new Ubuntu version to get a new kernel to resolve this.

The mistake

During the install process, it asked me whether I wanted to update my PAM (plugable authentication modules) rules. I knew that I had changes in /etc/pam.d/common-auth, so I thought it would be safer to return to the installation defaults and make my custom changes again after successful installation. As I got up for a tea break, I locked my computer (part of my muscle memory). Then when I got back, it would no longer accept my password. Now I had this issue where I knew that I had a distribution upgrade running, but I could no longer access my computer.

I waited a while to hopefully install as much as it could until the next user prompt, but eventually I had to force power down my laptop. After rebooting, my Ubuntu install would not even get to the point where it asked for my drive's encryption password, it would just state a kernel panic. So not recovery shell was accessible to me.

Live USB

I had a live USB stick lying around with Ventoy on it and a Ubuntu 24.04 ISO, which gave me a path to a root shell on my laptop. Because my disk is encrypted, I needed to mount and decrypt my drive. To find out which disk was mine, I used the lsblk command. I knew the size of my drive and knew that it probably had a boot partition or two. Once I found the name of my drive I could use the following command to decrypt it:

cryptsetup luksOpen /dev/name_of_drive_from_lsblk my_volume

I could then open the drive in Files and use the "Open in Terminal" option under the right-click menu. This is also where I was able to backup all my data.

Fixing PAM

I have no idea why the install made the specific changes it did to my PAM but there was a difference between the common-auth on the live USB and the one on my install, so I just replaced it:

cp etc/pam.d/common-auth etc/pam.d/common-auth.backup cp /etc/pam.d/common-auth etc/pam.d

Changing root

Since I could not boot into my own Ubuntu, I had to piggy back off of the kernel in my live USB to make changes to my install. Luckily Linux has this concept of changing root, so I could change the root to my laptop's drive. I needed to do a bit of setup before I was able to get apt working:

# Assuming you're the folder you want to change root to. mount --bind /dev dev
mount --bind /dev/pts dev/pts
mount --bind /proc proc
mount --bind /sys sys
chroot .

Fixing the upgrade

Inside the changed root, you can hopefully use the following commands to finish the distribution upgrade:

apt update -m
dpkg --configure -a
apt dist-upgrade
apt --fix-broken install

Fixing boot

After this, I still got a kernel panic saying that VFS was unable to mount the root file-system. I used these commands in the live USB to help with this:

update-initramfs -u -k 6.8.0-44-generic
update-grub

I also ran boot repair in the live USB using the following commands and following the GUI instructions:

add-apt-repository ppa:yannubuntu/boot-repair
apt update
apt install boot-repair
boot-repair

Kernel searching

After this, Grub was allowing me to select kernels running locally on my machine again. Some kernels were booting successfully and others weren't. I repeated the steps in "Fixing the upgrade" and "Fixing boot" a few times until all the kernels were working again.

I now have a working Ubuntu again and I'm happy I didn't have to do a clean install. If you are not able to find your fixes here or anywhere else online, you can always make a full back-up of your drive using the live USB and then do a clean install.