Consider making TSM MADV_WILLNEED a configuration option

[e-dard]

Proposal:

In 1.5.1 we introduced a feature that used the mmap hint MADV_WILLNEED when loading TSM files: #9380. The general idea was that because we need to scan TSM indexes on startup, using MADV_WILLNEED would let the kernel know it could read ahead pages, or at least prep them in the disk cache. Of course, we can't really control what the kernel does, just hint.

Due to #9534 (comment) we ended up having to remove the MADV_WILLNEED hint in 1.5.2. See #9614

Current behavior:

MADV_WILLNEED is not used as a hint to the kernel that we plan to scan the TSM index.

Desired behavior:

We have found that in some cases, users have moved from 1.5.1 (where we used MADV_WILLNEED) to >= 1.5.2 (where we don't), and have experienced significant IO Wait on >= 1.5.2. It's possible this is down to slow disks, and that in their case, on 1.5.1 their kernel is paging in TSM files in a way that is helping their IO performance.

We could consider making the MADV_WILLNEED hint a configuration option.