The USB composite framework

http://lwn.net/Articles/395712/

Linux is widely used in mobile devices, which should not come as a surprise. It is a powerful and versatile system, and one of its strengths is its support for USB devices of all kinds. That includes "gadgets" — devices that act as USB slaves — like USB flash drives (i.e. pendrives). The USB composite framework makes writing drivers for these kinds of devices relatively easy.

As users keep more data on their mobile devices, the demand for interoperability with desktop computers increases. No one wants to buy a special cable or a "docking station" just to copy a few photos. What users want is to connect the device via a USB cable and get it working out of the box. Linux can give that to them.

Have you ever wondered how this actually works? What happens behind the scenes when a USB connection is established? Better yet, have you wondered how to write a USB gadget for your new and shiny embedded evaluation board?

In this article, I will try to shed some light on that topic.

USB overview

The Universal Serial Bus (or USB) standard defines a master-slave communication protocol. This means that there is one control entity (a master or a host), which decides who can transmit data through the wire. The other entities (slaves, devices, or gadgets) must obey and respond to the host's requests. Slaves do not communicate with each other. A host is usually a desktop computer, while the gadgets are devices such as mice, keyboards, phones, printers, etc.

People are used to seeing Linux systems in the master or host role on a USB bus. But the Linux USB stack also provides support for the slave or gadget role — the device at the other end of the wire. For example, when one connects a pendrive to a Linux host, it handles it with a usb-storage driver. However, if we had a Linux machine with a USB Device Controller (or UDC), we could run Alan Stern's File Storage Gadget (or FSG). FSG is, as its name implies, a gadget driver which implements the USB mass storage device class (or UMC). That would allow the machine in question to act as a USB drive (aka pendrive).

When a device is connected, an enumeration process begins. During this process, the device is assigned a unique 7-bit identifier used for addressing. As a consequence, up to 127 slaves (including hubs) can be connected to a single host.

Communication is based on logical pipes which join the master with one of a slave's endpoints (or EPs). There can be up to 16 endpoints (numbered from 0 to 15) on a device. Endpoint zero (or EP0) is reserved for setup requests (eg. a query for descriptors, request to set a particular configuration, etc.).

Pipes are unidirectional (one-way) and data can go to (via an IN endpoint) or from (via an OUT endpoint) the host. (It is important to remember that from a slave's point of view, an IN endpoint is the one it writes to and an OUT endpoint is the one it reads from.) There are also four transfer modes: bulk, isochronous, interrupt, and control.

Endpoints are grouped into interfaces which are then grouped intoconfigurations. Different configurations may contain different interfaces, as well as have different power demands. All that information is saved in various descriptors requested by the host during enumeration. One can see them using the lsusb tool. Here is the stripped-down (and annotated) output for a Kingston pendrive:

    Bus 001 Device 004: ID 0951:1614 Kingston Technology
    Device Descriptor:
      idVendor           0x0951 Kingston Technology
      idProduct          0x1614
      bNumConfigurations      1     [only one configuration]
      Configuration Descriptor:     [the first and only config]
        bNumInterfaces          1    [only one interface]
        MaxPower              200mA
        Interface Descriptor:        [the first and only intf.]
          bNumEndpoints           2   [two endpoints]
          bInterfaceClass         8 Mass Storage
          bInterfaceSubClass      6 SCSI
          bInterfaceProtocol     80 Bulk (Zip)
          Endpoint Descriptor:        [the first endpoint]
            bEndpointAddress     0x81  EP 1 IN
            bmAttributes            2
              Transfer Type            Bulk
              Usage Type               Data
          Endpoint Descriptor:        [the second endpoint]
            bEndpointAddress     0x02  EP 2 OUT
            bmAttributes            2
              Transfer Type            Bulk
              Usage Type               Data

After the host receives the descriptors and learns what kind of a gadget has been connected, it can choose a configuration to be used and start communicating. At most one configuration can be active at a time.

Linux USB composite framework

There is, however, another module that implements UMC: my Mass Storage Gadget (or MSG). The obvious question is, why there are two drivers that seem to do the very same thing. This has something to do with the Linux USB composite framework.

The "old way" of creating gadgets is to get the specification and implement everything as a single, monolithic module. Gadget Zero, File Storage Gadget, and GadgetFS are examples of such gadgets.

This approach has two rather big disadvantages:

• many of the common USB functionalities (core device setup requests on EP0) have to be implemented in each and every module; and
• it can be tricky to combine the code from several gadgets into a new gadget with combined functionality.

For those reasons, David Brownell came up with the composite framework which has two advantages over the old approach:

• all of the core USB requests are implemented by the framework; and
• a single functionality or a USB composite function is developed separately from other functions as well as from the USB bus logic that is not directly related to this function. Later, such functions are combined using the composite function to form a composite gadget.

From a composite gadget's perspective, a device has some functions grouped into configurations. One function may be present in any number of configurations. Each function may have several interfaces and other descriptors but that is transparent to the kernel module.

Put on top of the "raw" USB descriptors structure, a USB composite function can be regarded as an abstraction for a group of interfaces.

That is another excellent property of the framework — most implementation details are hidden "under the hood" and one does not need to think about them when developing a gadget. Instead of thinking about endpoints and interfaces, one thinks about functions. Therefore, FSG is a gadget developed in the "old way", whereas MSG is a composite gadget which uses only one composite function — the Mass Storage Function (or MSF). As a matter of fact, MSF has been created from FSG to allow for the creation of more complicated drivers that would have UMC as part of their functionality.

Overall driver structure

In this article, I will try to explain how to create a mass storage composite gadget. It is in the kernel already, but let's forget that FSG and MSG exist for a moment.

What is great about Linux, is that a lot has already been done and one can get results with relatively little effort. As such, I will show how to create a working driver using MSF and some "composite glue".

I will start with the structure of the module, while skipping the details of the Mass Storage Function. The first step is to define a device descriptor. It stores some basic information about the gadget:

    static struct usb_device_descriptor msg_dev_desc = {
    	.bLength =		sizeof msg_dev_desc,
    	.bDescriptorType =	USB_DT_DEVICE,
    	.bcdUSB =		cpu_to_le16(0x0200),
    	.idVendor =		cpu_to_le16(FSG_VENDOR_ID),
    	.idProduct =		cpu_to_le16(FSG_PRODUCT_ID),
    };

The usb_device_descriptor structure has some more fields but they are not required or not important for our module. What has been set is:

bLength and  bDescriptorType

A standard fields each descriptor has.

bsdUSB

The version of USB specification the device supports encoded in BCD (so  0x200 means  2.00).

idVendor and  idProduct

Each device must have a unique vendor and product identifier pair. To avoid collisions, companies (vendors) can buy a vendor ID which gives them a namespace of 65536 product IDs to use. NetChip has donated some product IDs to the Linux community. Later, the Linux Foundation got the whole vendor ID for use with Linux.  FSG_VENDOR_ID is actually NetChip's vendor ID and, along with  FSG_PRODUCT_ID, that is what FSG uses.

The next step is to define an USB configuration which will be provided by the driver. It is described by a usb_configuration structure which, among other things, points to a bind callback function. Its purpose is to bind all USB composite functions to the configuration. Usually, it is a simple function, as most of the job is done prior to its invocation.

Put together it looks as follows:

    static struct usb_configuration msg_config = {
    	.label			= "Linux Mass Storage",
    	.bind			= msg_do_config,
    	.bConfigurationValue	= 1,
    	.bmAttributes		= USB_CONFIG_ATT_SELFPOWER,
    };

    static int __ref msg_do_config(struct usb_configuration *c)
    {
    	return fsg_add(c->cdev, c, &msg_fsg_common);
    }

The msg_config object specifies a label (used for debug messages), the bind callback, configuration's number (each configuration must have a unique, non-zero number), and indicates that the device is self powered. All that the msg_bind does is bind the MSF to the configuration.

That definition is then used by the msg_bind() function, which is a callback to set up composite functions, prepare descriptors, add all configurations supported by the device, etc.:

    static int __ref msg_bind(struct usb_composite_dev *cdev)
    {
    	int ret;

    	ret = msg_fsg_init(cdev);
    	if (ret < 0)
    		return ret;

    	ret = usb_add_config(cdev, &msg_config);
    	if (ret >= 0)
    		set_bit(0, &msg_registered);
    	fsg_common_put(&msg_fsg_common);
    	return ret;
    }

The  msg_bind() function does the following: initializes the Mass Storage Function, adds the previously defined configuration to the USB device, and (at the end)  puts the  msg_fsg_common object. . If everything succeeds, it sets the  msg_registered flag so it is recorded that the gadget has been registered and initialized.

With all of the above, a composite device can be defined. For this purpose, the usb_composite_driverstructure is used. Besides specifying the name, it points to the device descriptors and the bind callback:

    static struct usb_composite_driver msg_device = {
    	.name		= "g_my_mass_storage",
    	.dev		= &msg_dev_desc,
    	.bind		= msg_bind,
    };

At this point, all that is left are the init and exit module functions:

    static int __init msg_init(void)
    {
    	return usb_composite_register(&msg_device);
    }

    static void msg_exit(void)
    {
    	if (test_and_clear_bit(0, &msg_registered))
    		usb_composite_unregister(&msg_device);
    }

They use the usb_composite_register() and usb_composite_unregister() functions to register and unregister the device. The msg_registered variable is used to ensure the device is unregistered only once.

To sum things up:

• A composite device (msg_device) is registered when in msg_init() when the module loads.
• It has a device bind callback (msg_bind()) that initializes MSF and adds configuration to the gadget.
• The configuration (msg_config) has its own bind callback (msg_do_config()), which binds MSF to the configuration.
• The really hard work is done inside the MSF.

Mass Storage Function

With the big picture in mind, lets get into the finer details: the inner workings of the Mass Storage Function. There are a couple of things to watch out for when dealing with it.

First of all, because MSF can be bound to several configurations, it needs to share some data between the instances and at the same time store information specific for each configuration. The fsg_common structureis used for shared data. An instance of this structure needs to be initialized prior to binding MSF.

Because the common object is used by several MSF instances, it has no single owner thus a reference counter is needed to decide when it can be destroyed. That's the reason for the fsg_common_put() call at the end of msg_bind() function.

Closely connected with the fsg_common structure is a worker thread which MSF uses to handle all the host's requests. When a fsg_common object is created, a thread is started as well. It terminates either when thefsg_common object is destroyed or when it is killed with an INT, TERM, or KILL signal. In the latter case, the fsg_common object may still exist even after worker's death. Whatever reason, when thread exits athread_exits callback is invoked.

It is important to note that a signal may terminate the worker thread, but why would one want to do that? The reason is simple. As long as MSF is holding any open files, the filesystems which those files belong to cannot be unmounted. That is bad news for a shutdown script.

What Alan Stern came up with in FSG, is to close all backing files when the worker thread receives an INT, TERM, or KILL signal. Because MSF is to be used with various composite gadgets, rather than hardcoding that behavior a callback has been introduced.

The last thing to note is that MSF is customizable. The UMC specification allows for a single device to have several logical units (sometimes called LUNs, which is strictly speaking incorrect since LUN stands for Logical Unit Number). Each logical unit may be read-only or read-write, may emulate a CD-ROM or disk drive, and may be removable or not.

All of this configuration must be specified when the fsg_common structure is initialized. The fsg_configstructure is used for exactly that purpose. In most cases, a module author does not want to fill it themselves, but rather let a user of the module decide the settings.

To make it as easy as possible, an fsg_module_parameters structure and an FSG_MODULE_PARAMETERS() macro are provided by the MSF. The former stores user-supplied arguments, whereas the latter defines several module parameters.

Having an fsg_module_parameters object, one may use fsg_config_from_params() followed by fsg_common_init()to create an fsg_common object. Alternatively, fsg_common_from_params() can be used which merges the call to the other two functions.

Here is how it all works when put together:

    static struct fsg_module_parameters msg_mod_data = { .stall = 1 };
    FSG_MODULE_PARAMETERS(/* no prefix */, msg_mod_data);

    static struct fsg_common msg_fsg_common;

    static int msg_thread_exits(struct fsg_common *common)
    {
    	msg_exit();
    	return 0;
    }

    static int msg_fsg_init(struct usb_composite_dev *cdev)
    {
    	struct fsg_config config;
    	struct fsg_common *retp;

    	fsg_config_from_params(&config, &msg_mod_data);

    	config.thread_exits = msg_thread_exits;

    	retp = fsg_common_init(&msg_fsg_common, cdev, &config);
    	return IS_ERR(retp) ? PTR_ERR(retp) : 0;
    }

The msg_exit() function has been chosen as MSF's thread_exits callback. Since MSF is nonoperational after the thread has exited, there is no need to keep the composite device registered, instead the gadget is unregistered.

At this point, it should become obvious why the msg_registered flag is being used. Sinceusb_composite_unregister() can be called from two different places, a mechanism to guarantee that it will be called only once is needed — atomic bit operations are perfect for such tasks.

And that would be it. We are done. One can grab the full source code and start playing with it.

The beauty of the composite framework is that all the really hard stuff has been already written. One can write devices and experiment with different configurations without deep knowledge of the USB specification or the Linux gadget API. At the same time, it is a perfect introduction to some more serious USB programming.

Running

To use the gadget, one needs to provide a disk image that will act as a real USB device to the USB host. Using dd on the device is perfect for creating one:

    # dd if=/dev/zero of=disk.img bs=1M count=64

With disk image in place, the module can be loaded:

    # insmod g_my_mass.ko file=$PWD/disk.img

Connecting the device to the host should produce several messages in the host system log, among others:

    usb 1-4.4: new high speed USB device using ehci_hcd and address 8
    usb 1-4.4: New USB device found, idVendor=0525, idProduct=a4a5
    usb-storage: device scan complete
    sd 6:0:0:0: [sdb] Attached SCSI removable disk
    sd 6:0:0:0: [sdb] 131072 512-byte logical blocks: (67.1 MB/64.0 MiB)
     sdb: unknown partition table

All that is left is creating a partition with a filesystem and starting using the pendrive:

    # fdisk /dev/sdb
    ...
    # dmesg -c
    sd 6:0:0:0: [sdb] Assuming drive cache: write through
     sdb: sdb1

    # mkfs.vfat /dev/sdb1
    mkfs.vfat 3.0.9 (31 Jan 2010)
    # mount /dev/sdb1 /mnt/
    # touch /mnt/foo
    # umount /mnt

As has been shown, the gadget works like a charm.

Conclusion

The Linux USB composite framework provides a way to add USB devices in a fairly straightforward way. Before the composite framework came along, developers needed to implement all USB requests for each gadget they wanted to add to the system. The framework handles basic USB requests and separates each USB composite function, which allows gadget authors to think in terms of functions rather than low-level interfaces and communication handling.

As one might guess, this article just scratches the surface of what the composite framework can do. The driver that was shown is a single-configuration, single-function gadget, so the advantages over non-composite gadgets is not readily apparent. A future article may look at drivers for more powerful gadgets using the composite framework.

The USB composite framework

Posted Jul 15, 2010 9:31 UTC (Thu) by linusw (subscriber, #40300) [Link]

The first part of the article looks very inspired by the Wikipedia article on USB so why not link it:
http://en.wikipedia.org/wiki/Universal_Serial_Bus

The USB composite framework

Posted Jul 15, 2010 16:16 UTC (Thu) by jimparis (subscriber, #38647) [Link]

I'm finding this article very confusing. 
When you say things like "The really hard work is done inside the MSF", it sounds like you're talking about the fsg_* functions, which are separate (but mostly copied) from Alan Stern's FSG? So a function like fsg_main_thread in f_mass_storage.c is part of MSF, but the function
fsg_main_thread in file_storage.c is part of FSG?

Having 5 ".c" files #included directly in "the full source code" doesn't help clarify things. Doesn't that also cause symbol conflicts if we wanted to write two modules and install them both?

I guess this article mostly is focused on how to use the composite framework, but the actual implementation of the device (MSF/FSG/whatever) seems to be the more interesting part from a USB driver writer point of view. Does the composite framework make that part any easier?

I'd really look forward to a future article on "more powerful gadgets" as you mention. Particularly interesting would be how such a composite gadget is treated and works on typical operating systems (can each function usually be bound to a driver independently?) and how to deal with hardware limitations (as I understand, most hardware has various limitations on numbers of endpoints or restricts their sizes, etc).

The USB composite framework

Posted Jul 15, 2010 18:38 UTC (Thu) by iabervon (subscriber, #722) [Link]

I guess this article mostly is focused on how to use the composite framework, but the actual implementation of the device (MSF/FSG/whatever) seems to be the more interesting part from a USB driver writer point of view. Does the composite framework make that part any easier?

My impression is that the framework is not intended to make the interesting part any easier, but rather to make the uninteresting part trivial. Having written a USB gadget (not on Linux), I've found that handling EP0 is a big chunk of uninteresting code that needs to be correct and involves fiddly interactions with your drivers, in that you have to make sure that your driver, your descriptor as sent on EP0, and your hardware registers all agree on how the endpoints are configured, and weird things happen if they don't.

The USB composite framework

Posted Jul 25, 2010 11:49 UTC (Sun) by mina86 (subscriber, #68442) [Link]

As I've mentioned, Mass Storage Function is merely a conversion of the File-Storage Gadget so most of the code in MSF is copied with modifications from FSG. As such, your understanding is correct, by MSF I mean all the functions in f_mass_storage.c as well as storage_common.c (the latter is code that is identical (or nearly identical) in both MSF and FSG).

The prefix of the function names may be confusing. At one point I even wanted to use fsg_ for FSG stuff, msf_ for MSF stuff and stor_ for commons but Alan convinced my that it's too much hustle and modification not worth the effort. Hence, the fsg_ prefix stayed.

As far as including “.c” files... to be honest, I'm not entirely convinced why this is done this way but that's the way it is done in other composite gadget drivers as well. The comment you may find in those is:

> Kbuild is not very cooperative with respect to linking separately compiled library objects into one module. So for now we won't use separate compilation ... ensuring init/exit sections work to shrink the runtime footprint, and giving us at least some parts of what a "gcc --combine ... part1.c part2.c part3.c ... " build would.

Going further in your comment, the situation with composite is just like iabervon described. It makes uninteresting things trivial. I would also add that it makes the more interesting part a bit harder.

This is because composite function has to worry about registering interface and string IDs and never knows in how many configurations it will be present or ever how many configurations gadget has.

One of the most complicated thing about converting File-Storage Gadget to Mass Storage Function was to allow for it to be added to several USB configurations. To achieve that I have created the fsg_common structure with reference counting and some such.

As of next articles I can only thank you for interest and get back to work. ;) Nonetheless, the my next article probably won't cover most of your questions but should be valuable addition to this one.

I may only quickly point out that you deal with hardware limitations by obeying them. :P If your UDC has limited number of endpoints you won't be able to stash many composite functions in a single configuration in your gadget. Mass storage is rather simple since in bulk-only mode (the one implemented by MSF) it uses only one IN and one OUT endpoint but generally functions may need more endpoints.

The USB composite framework

Posted Jul 16, 2010 4:38 UTC (Fri) by pabs (subscriber, #43278) [Link]

It would be great to be able to modprobe a mass-storage gadget, a serial gadget and a network gadget and a userland gadget and instantly get a composite device that supports all four.

The USB composite framework

Posted Jul 25, 2010 11:53 UTC (Sun) by mina86 (subscriber, #68442) [Link]

Yep! It would.

As a matter of fact, I've been thinking about something like that for quite some time now (ever since I've started working on USB gadgets I think). I also have some ideas regarding such functionality.

I was thinking more along the lines that you modprobe a “stub” gadget which merely describes what configurations it want to support and then modprobe all the functions. In essence that's what you've described.

I hope to find time for actually implementing it so maybe it'll be even easier then before to write gadgets. :)

The USB composite framework

Posted Mar 9, 2011 21:35 UTC (Wed) by spectrum (guest, #73494) [Link]

I would like to create a USB composite driver that has functionalities of both file storage gadget and RNDIS gadget. 

Is this possible?