XFS online filesystem check and repair [LWN.net]

Darrick Wong has been doing work on XFS online
repair for a number of years
and things are getting to the point where most of the filesystem-internal work
has been completed and is under review. The work remaining mostly concerns
the user-space side
to set up a periodic scan and repair cycle, so he wanted to discuss what
user space needs from this kind of feature in a filesystem session at the
2023 Linux Storage, Filesystem,
Memory-Management and BPF Summit that he led remotely. The session may
not have gone quite as he hoped, as it got somewhat derailed by topics that
spilled over from the earlier session on
unprivileged image mounts.

His current patch set for XFS online repair is “out for review on Dave
Chinner’s laptop right now”, so it is time to start talking about the
missing pieces. That means that he will be talking more about user space
than he would normally; there is a user-space driver program that controls
how often the online fsck mechanism runs. There is nothing yet
for notifying user space of problems that were found by an online fsck
pass, nor is there a daemon monitoring for notifications to do anything
about them, such as to issue repair requests. There is no good
infrastructure in the kernel for handling and dispatching such things, he
said.

He said that the earlier discussion in the unprivileged-mounts session
on using fsck to decide that an image was sound enough to mount
made him think that it was a good time to discuss these kinds of issues.

As he noted, there is a command-line program, xfs_scrub,
which opens the block device and root directory, then starts issuing the right
ioctl() commands, but the real use case is not for running a tool
in that fashion. Instead, the idea is that it would do a background check
and repair periodically from
a systemd service; he is struggling a bit with setting that up, but has
something working. It is not, however, much different from the age-old
periodic cron job that reports its results to the system log and hopes an
administrator is paying attention.

He would like to create a notification
system that would allow the system to respond dynamically to the events
that get reported by the periodic scrubbing. He would also like there to
be a way for programs to initiate scrubbing for various reasons, such as a
container manager that notices relatively low activity so it kicks off
scrubbing on the mounted filesystems. Maybe that could mesh with the
unprivileged-mounting use case in some fashion as well, Wong said.

So he wondered if any user-space developers had thoughts on how they might
want to use this facility. He could continue developing “with my
kernel colored glasses on”, but he fears that may not produce the best
results. There was an effort made to scare up Lennart Poettering, who might
have some thoughts on the matter, but who had not made it back to the
filesystem room after the coffee break.

Josef Bacik said that he generally relied on people from Fedora and other
distributions to give him feedback on features of this sort. The
distribution developers often have different ideas on how these things will
be used. So, for thoughts on policies that might be applied to the online
scrubber, he
recommended seeking out people from Linux distributions.

Ted Ts’o replayed some of the earlier discussion around using (offline)
fsck
to check image files before mounting them.
In order to be sure that image files are not modified by user
space while the fsck was being done, Ts’o had said that they
would need to be copied somewhere inaccessible to user space beforehand.
One difference
with the in-kernel fsck equivalent that XFS is planning to add
might be that the copy/snapshot step would be
unnecessary. A
kernel-level fsck might not have that requirement, he suggested,
but that does not really change whether using fsck in that manner
is sufficient.

By that point, Poettering had returned so Wong repeated some of what he had
said earlier. He said that the work on the online scrubber had quite
recently become
more urgent because “a lot more distros than the zero I thought there were
will actually let you mount XFS filesystems without privilege”. There have
also been recent efforts in XFS to flag strange problems (“weird-looking
metadata or outright bad metadata”) that it sees, but that is not connected
with fsnotify events (as ext4 is) to notify user space of these
kinds of corruption. XFS generally knows exactly what the problem was,
which could be encoded in the notification somehow in the hopes that
someone is listening and can take appropriate action. For some filesystems
that might be to unmount and fsck the filesystem, while XFS could
use the online repair facility.

Poettering said that the current practice of having desktops mount
removable media automatically is
“stupid”; the approach that Chrome OS takes with only mounting certain specific
filesystem types (e.g. VFAT), and only through a user-space driver, is much
better and one that other desktops should adopt. The desktop use case is
generally for USB sticks, and people do not normally put XFS on that kind
of media, he said, so those should not be automatically mounted at all.

For mounting filesystem images in containers, though, he thinks
trust should come from dm-verity as described in his earlier talk. Ts’o had
said that fsck might be sufficient for establishing that an ext4
image would not compromise the kernel, so Poettering wondered if Wong would
say the same for XFS. There is a difficult answer to that, Wong said; “as
soon as I say ‘yes’, everybody in the world will watch their fuzzer rigs in
order to try to find all of the things that fsck doesn’t catch”.
That said, he generally agrees with Ts’o that fsck, either online
or offline, should be
robust enough to catch any bad filesystems, but it is not an absolute
guarantee since bugs happen.

Poettering noted that the online checking for XFS was not usable for
establishing trust, since the filesystem would need to be mounted first.
Wong agreed, but wondered about images that had been signed by the
distributor. Poettering and Christian Brauner said that signed images are
fully trustable or, at least, that it is a user-space problem if they are not.
Kent Overstreet said that fsck could not be used to establish
trust in any case because a malicious device could change the data out from
underneath the check. While that is true for, say, USB devices, the
snapshot/copy requirement for a local image file that is getting mounted in
a container removes that possibility, Ts’o said.

Overstreet argued that
requiring the copy was onerous and unenforceable for users.
Instead, he thinks “the responsible thing for us to be doing as
filesystem implementers is to start taking it a little bit more seriously to
just hardening our code at run time”. He said that XFS does
a lot of read- and write-time verification of metadata along with fuzzing,
as does
Overstreet’s bcachefs, so
“we might not be in as bad a shape as we assume”.

Brauner wanted to clarify that the copy and fsck being discussed
was not something that is under the user’s control, but would be handled
by a mount daemon. Overstreet was adamant that it would still be
unacceptable to do the copy and “people are going to want to be able to
mount images in the
cloud untrusted very soon”.

Bacik said that the session was “getting off the rails” at that point.
He said that Wong is interested in what kinds of notifications would be of
interest to
user space and how to handle the policy questions around those; Wong agreed
with that. Poettering
said that he is “not a storage guy” so he does not know what kinds of
policies they might want, but he thinks that simply shutting down the affected
services when a filesystem it relies on has errors is the safest approach.
If systemd were to get a notification of that sort, it could easily be set
up to shut down affected services.

Ts’o said that those who are running these kinds of services should be
consulted about how to handle the events. For example: what do the
Kubernetes people actually want? They may want to shut down affected
services, but give the services a short time frame to send a “goodbye cruel
world”
message or similar. The ext4 notifications that Wong mentioned were
specifically added for the
internal Google Kubernetes-like container manager Borg; the people
maintaining those systems wanted to be able to shut down services in the
face of filesystem corruption.

Wong said things are a little different working for a large database vendor
(Oracle); most of the use of XFS, beyond root filesystems, is for “really
large data partitions where we would like to be able to perform at least
simple
repairs on the 100TB data partition to try to keep the VM running”. At any
given time, the workload running in the VM or container is probably not
accessing the whole 100TB, so there is an opportunity to fix things before
the application even notices. “We would at least like to try to grow new
engines on the plane while it’s flying in order to avoid having to do an
emergency landing.” Restoring 100TB (or even more) can take a long time,
which is best avoided.

Poettering wondered if a mount option that simply instructed XFS to run its
online scrubber whenever it detected an anomaly might be a reasonable
approach. “Why involve user space to trigger the online filesystem check?”
User space is better for performing actions on other parts of the system,
such as shutting down relevant services, so it does not really make sense
for XFS to notify of a problem and have user space say “go fix yourself”.
Wong said that he was willing to write an XFS daemon that would receive
notifications and schedule scrubbing if need be.

He wrapped up by describing some of the fuzzing that is done for XFS, which
uses the XFS
debugger to “walk every single field of every metadata object in the entire
filesystem and fuzz them”. That is part of why the XFS QA test suite takes
almost a week to run; it spends a lot of time fuzzing and checking to see
that the repair tool notices the problems and can fix them, both online
and offline. He thinks he added some fuzzing of ext4 metadata blocks to
fstests along the way, but not to the same level of precision of the XFS fuzz
testing.