Cheetah spot by spot: DIRECT_IO

Back to another article about the new features of IBM IDS 11.10 (code named Cheetah).
This will be a short article about a new feature that can end an old discussion. I'd bet that at least once, every Informix DBA had an argument about the advantages and disadvantages of using raw devices vs cooked files.
Raw devices are hard disk partitions or at least represent a physical device somewhere (either a local storage disk or a NAS pseudo-device). They are not used by any OS filesystem. The access is as direct and fast as possible. There is no filesystem layer or cache involved. The Informix instance requests the I/O operations directly to the underlying OS. Their advantages can be resumed as:

• They're fast
• You don't waste memory in FS cache (normally useless since IDS uses it's own cache)
• They're the most reliable way to store your data. Since there is no FS layer, it's not possible to have FS corruption
• They use Kernel Asynchronous I/O (KAIO) by default. This allows the I/O operations to be executed in the Kernel context. Simple AIO is asynchronous, but requires an user context that needs to be set up. IDS takes another advantage of KAIO because it is done on CPUVP threads, so you avoid process context switching (from CPU VP to AIO VP)
  

Cooked files are files you create on a filesystem and you use as Informix chunks. Their advantages are more or less the disadvantages of the raw devices:

• They're easy to use. Just "touch chunk;chown informix:informix chunk;chmod 660 chunk" and you're ready
• You can't accidentally create a filesystem over one of your raw devices
• Nowadays several filesystems have options to don't use memory cache

These arguments have been more or less persistent along the time, so the decision alway s depended in great part of your own feelings. If you could live without top of the top performance you might consider cooked files. If you needed the maximum performance then you'd probably choose raw devices.

Once again, the R&D engineers have been alert... Th lastest features available on the newest filesystems and OS lead them to make IDS use a new feature. It's called DIRECT_IO and this is also the name of a new $ONCONFIG parameter. If you set it to 1, IDS will attempt to use the feature if it's available on the OS and the filesystem you're using, for the pagesize you choose.

Whenever this is possible, it means you'll get the following on your chunks I/O:

• Buffer bypassing: Read/writes go directly to disk
• KAIO is used: You'll get a serious performance boost
  
• Concurrent I/O: Bypassing FS inode locking, gives you another performance boost
  

If you try to use it, be alert to any online.log messages you may get. I've tried this in a recently installed RHEL 4 system and it worked as expected. No messages in online log and I could see KAIO threads in the onstat -g ath output-
In a virtual machine running Fedora I got:

19:45:39 libaio.so.1: cannot open shared object file: No such file or directory
19:45:39 Please install libaio package. See machine notes.
19:45:39 Warning: Kernel Asynchronous I/O is disabled.

After a bit of yum'ing I installed libaio and tried again. This time with no errors and an onstat -g ath shows:

cheetah@PacMan.domus.online.pt:informix-> onstat -g ath
Your evaluation license will expire on 2007-11-24 00:00:00

IBM Informix Dynamic Server Version 11.10.UB7TL -- On-Line -- Up 00:00:59 -- 88872 Kbytes

Threads:
tid tcb rstcb prty status vp-class name
*2 47e1dd98 0 1 IO Idle 3lio lio vp 0
*3 47e32970 0 1 IO Idle 4pio pio vp 0
*4 47e47970 0 1 IO Idle 5aio aio vp 0
*5 47e5c970 0 1 IO Idle 6msc msc vp 0
6 47e89bd8 478b2018 3 sleeping secs: 1 1cpu main_loop()
*7 47e5cad0 0 1 running 1cpu soctcppoll
*8 47f76550 0 2 sleeping forever 1cpu soctcplst
9 47f769e8 478b25e0 1 sleeping secs: 1 1cpu flush_sub(0)
*10 47f76b48 0 3 IO Idle 1cpu kaio
11 4807d080 478b2ba8 2 sleeping secs: 1 1cpu aslogflush
12 4807d828 478b3170 1 sleeping secs: 4 1cpu btscanner_0
*28 4864b018 478b42c8 3 sleeping secs: 1 1cpu onmode_mon
*33 485c6080 478b3d00 1 sleeping secs: 14 1cpu dbScheduler
*34 485c58a0 478b4890 1 sleeping forever 1cpu dbWorker1
*35 482be7d0 478b4e58 1 sleeping forever 1cpu dbWorker2

So, this is good... What are the catches? There are a few, but most of them aren't related to IDS itself. It's more to do with OS and filesystem:

• This only works on Unix/Linux (Windows has intrinsic differences)
• The FS must support this funcionality
• It may depend on the page size you choose (some may not be supported in the FS)
  
• It can't be used for temporary dbspaces

If you decide to use it, you should check for any error messages and also you should look for KAIO threads. If the system starts to use KAIO for your normal chunks you'll probably want to reduce your AIO virtual processors. You may also want to use AUTO_AIOVPS (an ONCONFIG parameter that allows IDS to auto-configure the number of AIO virtual processors if it finds the number you define are not keeping up with the I/O needs)

So, this feature can really make a different in the old argument of cooked vs raw chunks. You may decide that it's possible to take the best of both worlds: Performance and manageability.

References:

• DIRECT_IO parameter description in Administrator Reference
• Notes about DIRECT_IO in Performance Guide