Regression: 3.12.y kernels are prone to filesystem dirty page count accounting failures

[P33M]

Following on from an email conversation where the issue was first found:

On 3.12.y kernels, it is possible to break the filesystem dirty page accounting (making it wrap around to 2^32-1 kB of dirty pages) which has the following effects:

• Writes to the filesystem slow to a crawl
• The ACT LED is almost permanently illuminated during IO
• All processes waiting on IO spend a lot of the time sleeping, waiting for IO to complete.

The reason that this occurs is that an interrupt- or preempt-unsafe double decrement on the global number of dirty filesystem pages causes the counter to wrap to an extremely large value, above the value where the kernel will start to flush dirtied filesystem pages out to disk. In effect, every file write operation will immediately trigger a write of the affected page to the physical media on which the filesystem is based: SD card or USB device. For certain workloads this has much of the same symptoms as "swap hell".

A bisect on the git tree between 3.11 and 3.12.21 produced two commits (yet to be bisected):

commit 36bc08cc01709b4a9bb563b35aa530241ddc63e3
Author: Gu Zheng <[email protected]>
Date:   Tue Jul 16 17:56:16 2013 +0800

    fs/aio: Add support to aio ring pages migration

    As the aio job will pin the ring pages, that will lead to mem migrated
    failed. In order to fix this problem we use an anon inode to manage the aio
    pages, and  setup the migratepage callback in the anon inode's address space
    that when mem migrating the aio ring pages will be moved to other mem node s

    Signed-off-by: Gu Zheng <[email protected]>
    Signed-off-by: Benjamin LaHaise <[email protected]>

commit 55708698c5f153f4e390175cdfc395333b2eafbd
Author: Gu Zheng <[email protected]>
Date:   Tue Jul 16 17:56:12 2013 +0800

    fs/anon_inode: Introduce a new lib function anon_inode_getfile_private()

    Introduce a new lib function anon_inode_getfile_private(), it creates a new
    instance by hooking it up to an anonymous inode, and a dentry that describe
    "class" of the file, similar to anon_inode_getfile(), but each file holds a
    single inode. Furthermore, anyone who wants to create a private anon file wi
    benefit from this change.

    Signed-off-by: Gu Zheng <[email protected]>
    Signed-off-by: Benjamin LaHaise <[email protected]>

Either of these could be culprit for allowing the page counter to wrap. Testing required on each in turn.

The canonical test is to stop and restart the mysqld service. If nr_dirty in /proc/meminfo turns into ~4 billion kB instead of a few hundred, then the bug exists.

[P33M]

The first bad commit is 36bc08c.

The bug still exists in rpi-3.15.y branch.

[popcornmix]

So, what is your recommendation? Revert that commit? Fix it?

[P33M]

Investigating. There have been some changes to the code surrounding the patch between 3.12 and 3.15 - one fixing a race condition, which is suspicious.