Windows Vista: Cached Reading of a File Opened with No Cache Enabled

Below are my observation regarding the behavior of randomly reading a file without disk cache.

Question 1: why ReadyBoost cache hit rate raises when reading a file (opened with no cache) randomly?

Answer: from what I observed, it is because ReadyBoost does cache the file, even if it is opened with a flag saying no cache is needed.

Question 2: why the file is opened with no cache enabled, but is still read without much disk activity when it is read the second time?

Answer: from what I observed, it is because not only ReadyBoost, but also RAM is used to cache the file, even if it is opened with a flag saying no cache is needed. One evidence is that at first the ReadyBoost hit read bytes becomes higher during the read, and later I ran it more times the ReadyBoost hit read bytes doesn’t become high any more; instead it is finished quickly without much disk activity.

Question 3: does Superfetch work for files that are opened with no cache enabled?

Answer: I think yes. Otherwise there would be no reason that ReadyBoost can cache the file–ReadyBoost needs Superfetch to populate the data.

Question 4: is Windows Vista different from previous versions of Windows?

Answer: As I compared with Windows 2000 (although running inside VPC, but I can observe the virtual HDD light), Windows 2000 doesn’t do any caching to the file if it is opened with the no cache flag.

Question 5: can it be the hard disk drive built-in cache that caches the data for the process reading a file with the no cache flag?

Answer: no. Because I also tried randomly reading 400MB data, which is signifcantly larger than the HDD built-in cache, but it is still cached. So I think the answer is no.

See also my test programs randreadtest.cs, randreadtest_ca.cs and randreadtest_big.cs.

using System;
using System.IO;
using System.Runtime.InteropServices;
namespace RobbieTests
{
    public class NativeMethods
    {
        [DllImport("Kernel32.dll", SetLastError = true)]
        public static extern IntPtr
            CreateFile(String lpFileName, UInt32 dwDesiredAccess, UInt32 dwShareMode, IntPtr lpSecurityAttributes, UInt32 dwCreationDisposition, UInt32 dwFlagsAndAttributes, IntPtr hTemplateFile);
        [DllImport("kernel32.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        public static extern bool ReadFile(IntPtr hFile, [Out] byte[] lpBuffer,
           UInt32 nNumberOfBytesToRead, out UInt32 lpNumberOfBytesRead, IntPtr lpOverlapped);
        [DllImport("kernel32.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        public static extern bool CloseHandle(IntPtr hObject);
        [DllImport("Kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
        public static extern UInt32 SetFilePointer(
            [In] /*SafeFileHandle*/ IntPtr hFile,
            [In] Int32 lDistanceToMove,
            [In, Out] ref Int32 lpDistanceToMoveHigh,
            [In] UInt32 dwMoveMethod);
        [DllImport("Kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
        public static extern UInt32 SetFilePointer(
            [In] /*SafeFileHandle*/ IntPtr hFile,
            [In] Int32 lDistanceToMove,
            [In] IntPtr lpDistanceToMoveHigh,
            [In] UInt32 dwMoveMethod);
        public static IntPtr INVALID_HANDLE_VALUE = (IntPtr)(-1);
        public static UInt32 GENERIC_READ = 0×80000000;
        public static UInt32 FILE_SHARE_READ = 0×00000001;
        public static UInt32 FILE_SHARE_WRITE = 0×00000002;
        public static UInt32 OPEN_EXISTING = 0×00000003;
        public static UInt32 FILE_ATTRIBUTE_NORMAL = 0×00000080;
        public static UInt32 FILE_FLAG_NO_BUFFERING = 0×20000000;
        public static UInt32 FILE_BEGIN = 0×00000000;
        public static UInt32 INVALID_SET_FILE_POINTER = 0xFFFFFFFF;
    }
    public class RandReadTest /* random uncached read test */
    {
        public static void Main(string[] args)
        {
            const int BUFSIZE = 4096;
            const int MAXBLOCKCNT = 10240;
            IntPtr hFile = NativeMethods.INVALID_HANDLE_VALUE;
            Random rand = new Random();
            byte[] buffer = new byte[BUFSIZE];
            UInt32 sizeread;
            int i;
            try {
                hFile =
                    NativeMethods.CreateFile(args[0], NativeMethods.GENERIC_READ, NativeMethods.FILE_SHARE_READ | NativeMethods.FILE_SHARE_WRITE,
                    IntPtr.Zero, NativeMethods.OPEN_EXISTING,
                    NativeMethods.FILE_ATTRIBUTE_NORMAL | NativeMethods.FILE_FLAG_NO_BUFFERING, IntPtr.Zero);
                if (hFile == NativeMethods.INVALID_HANDLE_VALUE) {
                    throw new FormatException(“Unable to open the file.”);
                }
                for (i = 0; i < MAXBLOCKCNT; i++) {
                    if (NativeMethods.SetFilePointer(hFile, rand.Next(MAXBLOCKCNT) * BUFSIZE, IntPtr.Zero, NativeMethods.FILE_BEGIN) == NativeMethods.INVALID_SET_FILE_POINTER) {
                        throw new FormatException(“Unable to seek in the file.”);
                    }
                    if (!NativeMethods.ReadFile(hFile, buffer, BUFSIZE, out sizeread, IntPtr.Zero)) {
                        throw new FormatException(“Unable to read the file.”);
                    }
                }
            } catch (Exception ex) {
                Console.WriteLine(ex);
                Console.WriteLine(ex.StackTrace);
            } finally {
                if (hFile != NativeMethods.INVALID_HANDLE_VALUE) {
                    NativeMethods.CloseHandle(hFile);
                }
            }
        }
    }
}