Raw flash vs. FTL devices， ubi

Raw flash vs. FTL devices

FTL stands for "Flash Translation Layer" and it is software which emulates a block device on top of flash hardware. At early days FTL ran on the host computer. For example, old PCMCIA flash devices were essentially raw flash devices, and the PCMCIA standard defined the media storage format for them. So the host computer had to run the FTL software driver which implemented PCMCIA FTL. However, nowadays FTL is usually firmware, and it is run by the controller which is built into the storage device. For example, if you look inside an USB flash drive, you'll find there a NAND chip (or several of them), and a micro-controller, which runs FTL firmware. Some USB flash drives are known to have quite powerful ARM processors inside. Similarly, MMC, eMMC, SD, SSD, and other FTL devices have a built-in controller which runs FTL firmware.

All FTL devices have an interface which provides block I/O access. Well, the interfaces are different and they are defined by different specifications, e.g., MMC, eMMC, SD, USB mass storage, ATA, and so on. But all of them provide block-based access to the device. By block-based access we mean that whole device is represented as an linear array of (usually 512-byte) blocks. Each block may be read or written.

Linux has an abstraction of a block device. For example hard drives are block devices. Linux has many file systems and the block I/O subsystem, which includes elevators and so on which have been created to work with block devices (historically - hard drives). So the idea is that the same software may be used with FTL devices. For example, you may use FAT file system on your MMC card, or ext3 file system or your SSD.

Although most flashes on the commodity hardware have FTL, there are systems which have bare flashes and do not use FTL. Those are mostly various handheld devices and embedded systems. Raw flash devices are very different to block devices. They have different work model, they have tighter constraints and more issues than block devices. In case of FTL devices these constraints and issues are hidden, but in case of raw flash the software has to deal with them. Please refer to this table for some more details about the difference between block devices and raw flashes.

UBIFS file system has been designed for raw flash. It doesn't work with block devices and it assumes the raw flash device model. In other words, it assumes the device has eraseblocks, which may be written to, read from, or erased. UBIFS takes care of writing all data out-of-place, doing garbage-collection and so on. UBIFS utilizes UBI, which is doing stuff like wear-leveling and bad eraseblock handling. All these things are not normally needed for block devices.

Very often people ask questions like "why would one need to use raw flash and why not just use eMMC, or something like this?". Well, there is no simple answer, and the following is what UBIFS developers think. Please, take into account the date of this writing (3 May 2009). The answer is given in form of a list of non-structured items, and the reader should structure it in a way which is appropriate for his system. And because mass storage systems mostly use NAND flash (modern FTL devices also have NAND flash arrays inside), we talk specifically about NAND flashes. Also, we'd like to emphasize that we do not give general recommendations and everything depends on system requirements.

• Bare NAND chips are cheaper and simpler, which is very important for small system. However, it seems like the industry pushes FTL devices forward and the situation is not that simple and obvious anymore. Indeed, an FTL device is more complex than a raw NAND of similar size, because an FTL device has an additional controller inside, and so on. But since the industry tends to produce a lot of FTL devices, and actually sell a lot of them, the price is going down.
• If you need a flash storage where you are going to use FAT file system, then in most cases you should stick with an FTL device (eMMC, MMC, SD or whatever). Just make sure the FTL device is doing proper wear-leveling.
• The other situation is when you are going to use your FTL device for system storage (e.g. for rootfs) and use a more robust file system like ext3. In this situation you should take into account various system requirements like tolerance to sudden power cuts. The following items are mostly related to system storage situations.
• FTL devices are "black boxes". FTL algorithms are normally vendor secrets. But we know that NAND flash has issues like wear-leveling, bad blocks handling, read-disturb and so on. And it is important to get them right, especially in case of MLC NAND flash, which may have very short eraseblock life-time (e.g., only 1000 erase-cycles). But because FTL algorithms are closed, it is difficult to be sure whether a specific FTL device gets everything right or not.
• If you start thinking about how FTL could be implemented, you realize that it must do things like garbage collection (sometimes referred to as "reclaim process"). And flash hardware pretty much requires most writes to be out-of-place. But how does FTL behave in case of sudden power-cuts? What if a power-cut happens while it is in the middle of doing garbage collection? Does the FTL device guarantee that the data which was on the flash media before the power cut happens will not disappear or become corrupted?
• The power-cut tolerance may be tested, while it is quite difficult to test stuff like wear-leveling or read-disturb handling, because it may require too much time.
• We have heard reports that some USB flash drives wear out very quickly, i.e., they start reporting I/O errors after few weeks of intensive use. This means that FTL does not do proper wear-leveling. This does not mean that all USB flash drives are bad - just that you should be careful.
• We have heard reports that MMC, eMMC, and SD cards corrupt and lose data if power is cut during writing. Even the data which were there long time before may corrupt or disappear. This means that they have bad FTL which does not do things properly. But again, this does not have to be true for all MMCs/eMMCs and SDs - there are many different vendors. Be careful, though.
• In general, if you glance back into the history, many FTL devices were mostly used with FAT file system for storing stuff like photo and video. FAT file system is not reliable by definition, which suggests that FTL devices may also be not very reliable, just because historically this was not really required. Indeed, it is not a big deal to lose a couple of photos. However, it is crucial to make sure that system libraries do not corrupt because of power-cuts.
• Good FTL must be a rather complex piece of software, especially if it deals with MLC NAND (which is used in most modern mass storage devices). Implementing it in firmware might be a difficult task, and running it might require a powerful controller. Instead, we suspect that vendors may user various tricks or compromises to keep their devices "good enough" and cheap. For example, it is known that some vendors optimize their FTL devices for FAT, and if you start using ext3 on top of one, you might face some unexpected problems or the device may become worse than you would imagine. Of course, it is often difficult to verify this with closed FTL.
• SSD drives are probably very different to eMMC, MMC/SD etc. We have not worked with SSD drives. They are expensive and probably have powerful CPUs that run complex firmware which probably gets things right.
• FTL devices are becoming more popular and better, although it is not easy to distinguish between good and bad FTL devices (of course vendors would assure you their device is perfect). Generally, there is nothing wrong in using an FTL device as long as you trust it, you have tested it, or it simply fits your system requirements.
• In case of raw flash we know exactly what we are doing. UBI/UBIFS handles all aspects of NAND flash like bad erase-blocks and wear-leveling. It guarantees power-cut tolerance. It is open and available, so you may always validate, test, and fix it. There is no lie about what it can and cannot do. On the other hand, with FTL devices you do not have much visibility to what is going on inside, vendors may lie about how good their FTL device is. If you find a bug in the firmware, vendors do not usually provide you a fast and easy way to update it, and so on.
• Theoretically, UBIFS may do a better job because it knows much more information about the files than FTL. For example, UBIFS knows about deleted files, while FTL does not, so FTL may do unneeded work trying to preserve the sectors belonging to deleted files. However, some FTL devices support "discard" requests and may benefit from the file system hints about unused sectors. Nevertheless, in general, UBIFS should do a better job on a bare NAND than a traditional FS on an FTL device with a similar NAND chip. On the other hand, FTL devices may include multiple NAND chips, highly parallelize things and provide fast I/O. SSDs are probably good examples.
• Obviously, the advantage of FTL devices is that you use old and trusted software on top of them. Be careful, though, as sometimes this may not be 100% true. UBIFS authors once tested a good brand eMMC with respect to the power cut tolerance. Some severe problems were found. Also, it was found that ext3 was not really usable with that eMMC either. What happened was that power cuts sometimes left some eMMC sectors not readable - the read operation returned ECC errors. But for ext3 read errors are fatal - it is not designed to handle them. The fsck.ext3 tool also refused to repair a file system which had unreadable sectors.

So it is indeed difficult to give an answer. Just think about cons and pros, take into account your system requirements and decide. Nonetheless, raw flashes are used, mostly in the embedded world, and this is why UBIFS has been developed.