Writing Solaris Device Driver: Device Context Management

Device Context Management

Some device drivers, such as drivers for graphics hardware, provide user processes with direct access to the device. These devices often require that only one process at a time accesses the device.

This chapter describes the set of interfaces that enable device drivers to manage access to such devices. The chapter provides information on the following subjects:

• Introduction to Device Context

• Context Management Model

• Context Management Operation

Introduction to Device Context

This section introduces device context and the context management model.

What Is a Device Context?

The context of a device is the current state of the device hardware. The device driver manages the device context for a process on behalf of the process. The driver must maintain a separate device context for each process that accesses the device. The device driver has the responsibility to restore the correct device context when a process accesses the device.

Context Management Model

Frame buffers provide a good example of device context management. An accelerated frame buffer enables user processes to directly manipulate the control registers of the device through memory-mapped access. Because these processes do not use traditional system calls, a process that accesses the device need not call the device driver. However, the device driver must be notified when a process is about to access a device. The driver needs to restore the correct device context and needs to provide any necessary synchronization.

To resolve this problem, the device context management interfaces enable a device driver to be notified when a user process accesses memory-mapped regions of the device, and to control accesses to the device's hardware. Synchronization and management of the various device contexts are the responsibility of the device driver. When a user process accesses a mapping, the device driver must restore the correct device context for that process.

A device driver is notified whenever a user process performs any of the following actions:

• Accesses a mapping

• Duplicates a mapping

• Frees a mapping

• Creates a mapping

The following figure shows multiple user processes that have memory-mapped a device. The driver has granted process B access to the device, and process B no longer notifies the driver of accesses. However, the driver is still notified if either process A or process C accesses the device.

Figure 11–1 Device Context Management

At some point in the future, process A accesses the device. The device driver is notified and blocks future access to the device by process B. The driver then saves the device context for process B. The driver restores the device context of process A. The driver then grants access to process A, as illustrated in the following figure. At this point, the device driver is notified if either process B or process C accesses the device.

Figure 11–2 Device Context Switched to User Process A

On a multiprocessor machine, multiple processes could attempt to access the device at the same time. This situation can cause thrashing. Some devices require a longer time to restore a device context. To prevent more CPU time from being used to restore a device context than to actually use that device context, the minimum time that a process needs to have access to the device can be set using devmap_set_ctx_timeout(9F).

The kernel guarantees that once a device driver has granted access to a process, no other process is allowed to request access to the same device for the time interval specified by devmap_set_ctx_timeout(9F).

Context Management Operation

The general steps for performing device context management are as follows:

1. Define a devmap_callback_ctl(9S) structure.

2. Allocate space to save device context if necessary.

3. Set up user mappings to the device and driver notifications with devmap_devmem_setup(9F).

4. Manage user access to the device with devmap_load(9F) and devmap_unload(9F).

5. Free the device context structure, if needed.

devmap_callback_ctl Structure

The device driver must allocate and initialize a devmap_callback_ctl(9S) structure to inform the system about the entry point routines for device context management.

This structure uses the following syntax:

struct devmap_callback_ctl {    
    int devmap_rev;
    int (*devmap_map)(devmap_cookie_t dhp, dev_t dev,
    uint_t flags, offset_t off, size_t len, void **pvtp);
    int (*devmap_access)(devmap_cookie_t dhp, void *pvtp,
    offset_t off, size_t len, uint_t type, uint_t rw);
    int (*devmap_dup)(devmap_cookie_t dhp, void *pvtp,
    devmap_cookie_t new_dhp, void **new_pvtp);
    void (*devmap_unmap)(devmap_cookie_t dhp, void *pvtp,
    offset_t off, size_t len, devmap_cookie_t new_dhp1,
    void **new_pvtp1, devmap_cookie_t new_dhp2,
    void **new_pvtp2);
};

devmap_rev

The version number of the devmap_callback_ctl structure. The version number must be set to DEVMAP_OPS_REV.

devmap_map

Must be set to the address of the driver's devmap_map(9E) entry point.

devmap_access

Must be set to the address of the driver's devmap_access(9E) entry point.

devmap_dup

Must be set to the address of the driver's devmap_dup(9E) entry point.

devmap_unmap

Must be set to the address of the driver's devmap_unmap(9E) entry point.

Entry Points for Device Context Management

The following entry points are used to manage device context:

• devmap(9E)

• devmap_access(9E)

• devmap_contextmgt(9E)

• devmap_dup(9E)

• devmap_unmap(9E)

devmap_map() Entry Point

The syntax for devmap(9E) is as follows:

int xxdevmap_map(devmap_cookie_t handle, dev_t dev, uint_t flags,
    offset_t offset, size_t len, void **new-devprivate);

The devmap_map() entry point is called after the driver returns from its devmap() entry point and the system has established the user mapping to the device memory. The devmap() entry point enables a driver to perform additional processing or to allocate mapping specific private data. For example, in order to support context switching, the driver has to allocate a context structure. The driver must then associate the context structure with the mapping.

The system expects the driver to return a pointer to the allocated private data in *new-devprivate. The driver must store offset and len, which define the range of the mapping, in its private data. Later, when the system calls devmap_unmap(9E), the driver uses this information to determine how much of the mapping is being unmapped.

flags indicates whether the driver should allocate a private context for the mapping. For example, a driver can allocate a memory region to store the device context if flags is set to MAP_PRIVATE. If MAP_SHARED is set, the driver returns a pointer to a shared region.

The following example shows a devmap() entry point. The driver allocates a new context structure. The driver then saves relevant parameters passed in by the entry point. Next, the mapping is assigned a new context either through allocation or by attaching the mapping to an already existing shared context. The minimum time interval that the mapping should have access to the device is set to one millisecond.

---

Example 11–1 Using the devmap() Routine

static int
int xxdevmap_map(devmap_cookie_t handle, dev_t dev, uint_t flags,
    offset_t offset, size_t len, void **new_devprivate)
{
    struct xxstate *xsp = ddi_get_soft_state(statep,
                  getminor(dev));
    struct xxctx *newctx;

    /* create a new context structure */
    newctx = kmem_alloc(sizeof (struct xxctx), KM_SLEEP);
    newctx->xsp = xsp;
    newctx->handle = handle;
    newctx->offset = offset;
    newctx->flags = flags;
    newctx->len = len;
    mutex_enter(&xsp->ctx_lock);
    if (flags & MAP_PRIVATE) {
        /* allocate a private context and initialize it */
        newctx->context = kmem_alloc(XXCTX_SIZE, KM_SLEEP);
        xxctxinit(newctx);
    } else {
        /* set a pointer to the shared context */
        newctx->context = xsp->ctx_shared;
    }
    mutex_exit(&xsp->ctx_lock);
    /* give at least 1 ms access before context switching */
    devmap_set_ctx_timeout(handle, drv_usectohz(1000));
    /* return the context structure */
    *new_devprivate = newctx;
    return(0);
}

---

devmap_access() Entry Point

The devmap_access(9E) entry point is called when an access is made to a mapping whose translations are invalid. Mapping translations are invalidated when the mapping is created with devmap_devmem_setup(9F) in response to mmap(2), duplicated by fork(2), or explicitly invalidated by a call to devmap_unload(9F).

The syntax for devmap_access() is as follows:

int xxdevmap_access(devmap_cookie_t handle, void *devprivate,
    offset_t offset, size_t len, uint_t type, uint_t rw);

where:

handle

Mapping handle of the mapping that was accessed by a user process.

devprivate

Pointer to the driver private data associated with the mapping.

offset

Offset within the mapping that was accessed.

len

Length in bytes of the memory being accessed.

type

Type of access operation.

rw

Specifies the direction of access.

The system expects devmap_access(9E) to call either devmap_do_ctxmgt(9F) or devmap_default_access(9F) to load the memory address translations before devmap_access() returns. For mappings that support context switching, the device driver should call devmap_do_ctxmgt(). This routine is passed all parameters from devmap_access(9E), as well as a pointer to the driver entry point devmap_contextmgt(9E), which handles the context switching. For mappings that do not support context switching, the driver should call devmap_default_access(9F). The purpose of devmap_default_access() is to call devmap_load(9F) to load the user translation.

The following example shows a devmap_access(9E) entry point. The mapping is divided into two regions. The region that starts at offset OFF_CTXMG with a length of CTXMGT_SIZE bytes supports context management. The rest of the mapping supports default access.

---

Example 11–2 Using the devmap_access() Routine

#define OFF_CTXMG      0
#define CTXMGT_SIZE    0x20000    
static int
xxdevmap_access(devmap_cookie_t handle, void *devprivate,
    offset_t off, size_t len, uint_t type, uint_t rw)
{
    offset_t diff;
    int    error;

    if ((diff = off - OFF_CTXMG) >= 0 && diff < CTXMGT_SIZE) {
        error = devmap_do_ctxmgt(handle, devprivate, off,
            len, type, rw, xxdevmap_contextmgt);
    } else {
        error = devmap_default_access(handle, devprivate,
            off, len, type, rw);
    }
    return (error);
}

---

devmap_contextmgt() Entry Point

The syntax for devmap_contextmgt(9E) is as follows:

int xxdevmap_contextmgt(devmap_cookie_t handle, void *devprivate,
    offset_t offset, size_t len, uint_t type, uint_t rw);

devmap_contextmgt() should call devmap_unload(9F) with the handle of the mapping that currently has access to the device. This approach invalidates the translations for that mapping. The approach ensures that a call to devmap_access(9E) occurs for the current mapping the next time the mapping is accessed. The mapping translations for the mapping that caused the access event to occur need to be validated. Accordingly, the driver must restore the device context for the process requesting access. Furthermore, the driver must call devmap_load(9F) on the handle of the mapping that generated the call to this entry point.

Accesses to portions of mappings that have had their mapping translations validated by a call to devmap_load() do not generate a call to devmap_access(). A subsequent call to devmap_unload() invalidates the mapping translations. This call enables devmap_access() to be called again.

If either devmap_load() or devmap_unload() returns an error, devmap_contextmgt() should immediately return that error. If the device driver encounters a hardware failure while restoring a device context, a -1 should be returned. Otherwise, after successfully handling the access request, devmap_contextmgt() should return zero. A return of other than zero from devmap_contextmgt() causes a SIGBUS or SIGSEGV to be sent to the process.

The following example shows how to manage a one-page device context.

---

Note –

xxctxsave() and xxctxrestore() are device-dependent context save and restore functions. xxctxsave() reads data from the registers and saves the data in the soft state structure. xxctxrestore() takes data that is saved in the soft state structure and writes the data to device registers. Note that the read, write, and save are all performed with the DDI/DKI data access routines.

---

---

Example 11–3 Using the devmap_contextmgt() Routine

static int
xxdevmap_contextmgt(devmap_cookie_t handle, void *devprivate,
    offset_t off, size_t len, uint_t type, uint_t rw)
{
    int    error;
    struct xxctx *ctxp = devprivate;
    struct xxstate *xsp = ctxp->xsp;
    mutex_enter(&xsp->ctx_lock);
    /* unload mapping for current context */
    if (xsp->current_ctx != NULL) {
        if ((error = devmap_unload(xsp->current_ctx->handle,
            off, len)) != 0) {
            xsp->current_ctx = NULL;
            mutex_exit(&xsp->ctx_lock);
            return (error);
        }
    }
    /* Switch device context - device dependent */
    if (xxctxsave(xsp->current_ctx, off, len) < 0) {
        xsp->current_ctx = NULL;
        mutex_exit(&xsp->ctx_lock);
        return (-1);
    }
    if (xxctxrestore(ctxp, off, len) < 0){
        xsp->current_ctx = NULL;
        mutex_exit(&xsp->ctx_lock);
        return (-1);
    }
    xsp->current_ctx = ctxp;
    /* establish mapping for new context and return */
    error = devmap_load(handle, off, len, type, rw);
    if (error)
        xsp->current_ctx = NULL;
    mutex_exit(&xsp->ctx_lock);
    return (error);
}

---

devmap_dup() Entry Point

The devmap_dup(9E) entry point is called when a device mapping is duplicated, for example, by a user process that calls fork(2). The driver is expected to generate new driver private data for the new mapping.

The syntax fordevmap_dup() is as follows:

int xxdevmap_dup(devmap_cookie_t handle, void *devprivate,
    devmap_cookie_t new-handle, void **new-devprivate);

where:

handle

Mapping handle of the mapping being duplicated.

new-handle

Mapping handle of the mapping that was duplicated.

devprivate

Pointer to the driver private data associated with the mapping being duplicated.

* new-devprivate

Should be set to point to the new driver private data for the new mapping.

Mappings that have been created with devmap_dup() by default have their mapping translations invalidated. Invalid mapping translations force a call to the devmap_access(9E) entry point the first time the mapping is accessed.

The following example shows a typical devmap_dup() routine.

---

Example 11–4 Using the devmap_dup() Routine

static int
xxdevmap_dup(devmap_cookie_t handle, void *devprivate,
    devmap_cookie_t new_handle, void **new_devprivate)
{
    struct xxctx *ctxp = devprivate;
    struct xxstate *xsp = ctxp->xsp;
    struct xxctx *newctx;
    /* Create a new context for the duplicated mapping */
    newctx = kmem_alloc(sizeof (struct xxctx), KM_SLEEP);
    newctx->xsp = xsp;
    newctx->handle = new_handle;
    newctx->offset = ctxp->offset;
    newctx->flags = ctxp->flags;
    newctx->len = ctxp->len;
    mutex_enter(&xsp->ctx_lock);
    if (ctxp->flags & MAP_PRIVATE) {
        newctx->context = kmem_alloc(XXCTX_SIZE, KM_SLEEP);
        bcopy(ctxp->context, newctx->context, XXCTX_SIZE);
    } else {
        newctx->context = xsp->ctx_shared;
    }
    mutex_exit(&xsp->ctx_lock);
    *new_devprivate = newctx;
    return(0);
}

---

devmap_unmap() Entry Point

The devmap_unmap(9E) entry point is called when a mapping is unmapped. Unmapping can be caused by a user process exiting or by calling the munmap(2) system call.

The syntax for devmap_unmap() is as follows:

void xxdevmap_unmap(devmap_cookie_t handle, void *devprivate,
    offset_t off, size_t len, devmap_cookie_t new-handle1,
    void **new-devprivate1, devmap_cookie_t new-handle2,
    void **new-devprivate2);

where:

handle

Mapping handle of the mapping being freed.

devprivate

Pointer to the driver private data associated with the mapping.

off

Offset within the logical device memory at which the unmapping begins.

len

Length in bytes of the memory being unmapped.

new-handle1

Handle that the system uses to describe the new region that ends at off - 1. new-handle1 can be NULL.

new-devprivate1

Pointer to be filled in by the driver with the private driver mapping data for the new region that ends at off -1. new-devprivate1 is ignored if new-handle1 is NULL.

new-handle2

Handle that the system uses to describe the new region that begins at off + len. new-handle2 can be NULL.

new-devprivate2

Pointer to be filled in by the driver with the driver private mapping data for the new region that begins at off + len. new-devprivate2 is ignored if new-handle2 is NULL.

The devmap_unmap() routine is expected to free any driver private resources that were allocated when this mapping was created, either by devmap_map(9E) or by devmap_dup(9E). If the mapping is only partially unmapped, the driver must allocate new private data for the remaining mapping before freeing the old private data. Calling devmap_unload(9F) on the handle of the freed mapping is not necessary, even if this handle points to the mapping with the valid translations. However, to prevent future devmap_access(9E) problems, the device driver should make sure the current mapping representation is set to “no current mapping”.

The following example shows a typical devmap_unmap() routine.

---

Example 11–5 Using the devmap_unmap() Routine

static void
xxdevmap_unmap(devmap_cookie_t handle, void *devprivate,
    offset_t off, size_t len, devmap_cookie_t new_handle1,
    void **new_devprivate1, devmap_cookie_t new_handle2,
    void **new_devprivate2)
{
    struct xxctx *ctxp = devprivate;
    struct xxstate *xsp = ctxp->xsp;
    mutex_enter(&xsp->ctx_lock);

    /*
     * If new_handle1 is not NULL, we are unmapping
     * at the end of the mapping.
     */
    if (new_handle1 != NULL) {
        /* Create a new context structure for the mapping */
        newctx = kmem_alloc(sizeof (struct xxctx), KM_SLEEP);
        newctx->xsp = xsp;
           if (ctxp->flags & MAP_PRIVATE) {
               /* allocate memory for the private context
               /* and copy it */
            newctx->context = kmem_alloc(XXCTX_SIZE, KM_SLEEP);
            bcopy(ctxp->context, newctx->context, XXCTX_SIZE);
        } else {
            /* point to the shared context */
            newctx->context = xsp->ctx_shared;
        }
        newctx->handle = new_handle1;
        newctx->offset = ctxp->offset;
        newctx->len = off - ctxp->offset;
        *new_devprivate1 = newctx;
    }
    /*
     * If new_handle2 is not NULL, we are unmapping
     * at the beginning of the mapping.
     */
    if (new_handle2 != NULL) {
        /* Create a new context for the mapping */
        newctx = kmem_alloc(sizeof (struct xxctx), KM_SLEEP);
        newctx->xsp = xsp;
        if (ctxp->flags & MAP_PRIVATE) {
            newctx->context = kmem_alloc(XXCTX_SIZE, KM_SLEEP);
            bcopy(ctxp->context, newctx->context, XXCTX_SIZE);
        } else {
            newctx->context = xsp->ctx_shared;
        }
        newctx->handle = new_handle2;
        newctx->offset = off + len;
        newctx->flags = ctxp->flags;
        newctx->len = ctxp->len - (off + len - ctxp->off);
        *new_devprivate2 = newctx;
    }
    if (xsp->current_ctx == ctxp)
        xsp->current_ctx = NULL;
    mutex_exit(&xsp->ctx_lock);
    if (ctxp->flags & MAP_PRIVATE)
        kmem_free(ctxp->context, XXCTX_SIZE);
    kmem_free(ctxp, sizeof (struct xxctx));
}

---

Associating User Mappings With Driver Notifications

When a user process requests a mapping to a device with mmap(2), the driver`s segmap(9E) entry point is called. The driver must use ddi_devmap_segmap(9F) or devmap_setup(9F) when setting up the memory mapping if the driver needs to manage device contexts. Both functions call the driver's devmap(9E) entry point, which uses devmap_devmem_setup(9F) to associate the device memory with the user mapping. See Chapter 10, Mapping Device and Kernel Memory for details on how to map device memory.

The driver must inform the system of the devmap_callback_ctl(9S) entry points to get notifications of accesses to the user mapping. The driver informs the system by providing a pointer to a devmap_callback_ctl(9S) structure to devmap_devmem_setup(9F). A devmap_callback_ctl(9S) structure describes a set of entry points for context management. These entry points are called by the system to notify a device driver to manage events on the device mappings.

The system associates each mapping with a mapping handle. This handle is passed to each of the entry points for context management. The mapping handle can be used to invalidate and validate the mapping translations. If the driver invalidates the mapping translations, the driver will be notified of any future access to the mapping. If the driver validates the mapping translations, the driver will no longer be notified of accesses to the mapping. Mappings are always created with the mapping translations invalidated so that the driver will be notified on first access to the mapping.

The following example shows how to set up a mapping using the device context management interfaces.

---

Example 11–6 devmap(9E) Entry Point With Context Management Support

static struct devmap_callback_ctl xx_callback_ctl = {
    DEVMAP_OPS_REV, xxdevmap_map, xxdevmap_access,
    xxdevmap_dup, xxdevmap_unmap
};

static int
xxdevmap(dev_t dev, devmap_cookie_t handle, offset_t off,
    size_t len, size_t *maplen, uint_t model)
{
    struct xxstate *xsp;
    uint_t rnumber;
    int    error;
    
    /* Setup data access attribute structure */
    struct ddi_device_acc_attr xx_acc_attr = {
        DDI_DEVICE_ATTR_V0,
        DDI_NEVERSWAP_ACC,
        DDI_STRICTORDER_ACC
    };
    xsp = ddi_get_soft_state(statep, getminor(dev));
    if (xsp == NULL)
        return (ENXIO);
    len = ptob(btopr(len));
    rnumber = 0;
    /* Set up the device mapping */
    error = devmap_devmem_setup(handle, xsp->dip, &xx_callback_ctl,
        rnumber, off, len, PROT_ALL, 0, &xx_acc_attr);
    *maplen = len;
    return (error);
}

---

Managing Mapping Accesses

The device driver is notified when a user process accesses an address in the memory-mapped region that does not have valid mapping translations. When the access event occurs, the mapping translations of the process that currently has access to the device must be invalidated. The device context of the process that requested access to the device must be restored. Furthermore, the translations of the mapping of the process requesting access must be validated.

The functions devmap_load(9F) and devmap_unload(9F) are used to validate and invalidate mapping translations.

devmap_load() Entry Point

The syntax for devmap_load(9F) is as follows:

int devmap_load(devmap_cookie_t handle, offset_t offset,
    size_t len, uint_t type, uint_t rw);

devmap_load() validates the mapping translations for the pages of the mapping specified by handle,offset, and len. By validating the mapping translations for these pages, the driver is telling the system not to intercept accesses to these pages of the mapping. Furthermore, the system must not allow accesses to proceed without notifying the device driver.

devmap_load() must be called with the offset and the handle of the mapping that generated the access event for the access to complete. If devmap_load(9F) is not called on this handle, the mapping translations are not validated, and the process receives a SIGBUS.

devmap_unload() Entry Point

The syntax for devmap_unload(9F) is as follows:

    int devmap_unload(devmap_cookie_t handle, offset_t offset,
     size_t len);

devmap_unload() invalidates the mapping translations for the pages of the mapping specified by handle, offset, and len. By invalidating the mapping translations for these pages, the device driver is telling the system to intercept accesses to these pages of the mapping. Furthermore, the system must notify the device driver the next time that these mapping pages are accessed by calling the devmap_access(9E) entry point.

For both functions, requests affect the entire page that contains the offset and all pages up to and including the entire page that contains the last byte, as indicated by offset + len. The device driver must ensure that for each page of device memory being mapped, only one process has valid translations at any one time.

Both functions return zero if successful. If, however, an error occurred in validating or invalidating the mapping translations, that error is returned to the device driver. The device driver must return this error to the system.