I know that some blog posts on this topic have already been published, but nevertheless I have decided to share my real-life experience of bisecting regressions in Fedora Silverblue.

My work laptop, Dell XPS 13 Plus, has an Intel IPU6 webcam. It still lacks upstream drivers, so I have to use RPMFusion packages containing Intel’s software stack to ensure the webcam’s functionality.

However, on Monday, I discovered that the webcam was not working. Just last week, it was functioning, and I had no idea what could have broken it. I don’t pay much attention to updates; they’re staged automatically and applied with the next boot. Fortunately, I use Fedora Silverblue, where problems can be identified using the bisection method.

Silverblue utilizes OSTree, essentially described as Git for the operating system. Each Fedora version is a branch, and individual snapshots are commits. You update the system by moving to newer commits and upgrade by switching to the branch with the new version. Silverblue creates daily snapshots, and since OSTree allows you to revert to any commit in the repository, you can roll back the system to any previous date.

I decided to revert to previous states and determine when the regression occurred. I downloaded commit metadata for the last 7 days:

sudo ostree pull --commit-metadata-only --depth 7 fedora fedora/39/x86_64/silverblue

Then, I listed the commits to get their labels and hashes:

sudo ostree log fedora:fedora/39/x86_64/silverblue

Subsequently, I returned the system to the beginning of the previous week:

rpm-ostree deploy 39.20240205.0

After rebooting, I found that the webcam was working, indicating that something had broken it last week. I decided to halve the interval and return to Wednesday:

rpm-ostree deploy 39.20240207.0

In the Wednesday snapshot, the webcam was no longer functioning. Now, I only needed to determine whether it was broken by Tuesday’s or Wednesday’s update. I deployed the Tuesday snapshot:

rpm-ostree deploy 39.20240206.0

I found out that the webcam was still working in this snapshot. It was broken by the Wednesday’s update, so I needed to identify which packages had changed. I used the hashes from one of the outputs above:

rpm-ostree db diff ec2ea04c87913e2a69e71afbbf091ca774bd085530bd4103296e4621a98fc835 fc6cf46319451122df856b59cab82ea4650e9d32ea4bd2fc5d1028107c7ab912

ostree diff commit from: ec2ea04c87913e2a69e71afbbf091ca774bd085530bd4103296e4621a98fc835
ostree diff commit to: fc6cf46319451122df856b59cab82ea4650e9d32ea4bd2fc5d1028107c7ab912
Upgraded:
kernel 6.6.14-200.fc39 -> 6.7.3-200.fc39
kernel-core 6.6.14-200.fc39 -> 6.7.3-200.fc39
kernel-modules 6.6.14-200.fc39 -> 6.7.3-200.fc39
kernel-modules-core 6.6.14-200.fc39 -> 6.7.3-200.fc39
kernel-modules-extra 6.6.14-200.fc39 -> 6.7.3-200.fc39
llvm-libs 17.0.6-2.fc39 -> 17.0.6-3.fc39
mesa-dri-drivers 23.3.4-1.fc39 -> 23.3.5-1.fc39
mesa-filesystem 23.3.4-1.fc39 -> 23.3.5-1.fc39
mesa-libEGL 23.3.4-1.fc39 -> 23.3.5-1.fc39
mesa-libGL 23.3.4-1.fc39 -> 23.3.5-1.fc39
mesa-libgbm 23.3.4-1.fc39 -> 23.3.5-1.fc39
mesa-libglapi 23.3.4-1.fc39 -> 23.3.5-1.fc39
mesa-libxatracker 23.3.4-1.fc39 -> 23.3.5-1.fc39
mesa-va-drivers 23.3.4-1.fc39 -> 23.3.5-1.fc39
mesa-vulkan-drivers 23.3.4-1.fc39 -> 23.3.5-1.fc39
qadwaitadecorations-qt5 0.1.3-5.fc39 -> 0.1.4-1.fc39
uresourced 0.5.3-5.fc39 -> 0.5.4-1.fc39

The kernel upgrade from 6.6 to 6.7 was the logical suspect. I informed a colleague, who maintains the RPMFusion packages with the IPU6 webcams support, that the 6.7 kernel broke the webcam support for me. He asked for the model and informed me that it had an Intel VSC chip, which has a newly added driver in the 6.7 kernel. However, it conflicts with Intel’s software stack in RPMFusion packages. So he asked the Fedora kernel maintainer to temporarily disable the Intel VSC driver. This change will come with the next kernel update.

Until the update arrives, I decided to stay on the last functional snapshot from the previous Tuesday. To achieve this, I pinned it in OSTree. Because the system was currently running on that snapshot, all I had to do was:

sudo ostree admin pin 0

And that’s it. Just a small real-life example of how to identify when and what caused a regression in Silverblue and how to easily revert to a version before the regression and stay on it until a fix arrives.