Allocating physically contiguous memory in Solaris

Solaris doesn't easily expose an API to allocate on-demand physically contiguous, custom aligned memory. It has no calls like BSD's contigmalloc() or Darwin's IOMallocContiguous(). Here's a code snippet on how to do it:

struct ddi_dma_attr g_SolarisX86PhysMemLimits = 
{

    DMA_ATTR_V0,            /* Version Number */
    (uint64_t)0,            /* lower limit */
    (uint64_t)0xffffffff,   /* high limit (32-bit PA, 4G) */

    (uint64_t)0xffffffff,   /* counter limit */
    (uint64_t)MMU_PAGESIZE, /* alignment */

    (uint64_t)MMU_PAGESIZE, /* burst size */
    (uint64_t)MMU_PAGESIZE, /* effective DMA size */

    (uint64_t)0xffffffff,   /* max DMA xfer size */
    (uint64_t)0xffffffff,   /* segment boundary */

    1,                      /* scatter-gather list length */
    1,                      /* device granularity */
    0                       /* bus-specific flags */

};

caddr_t kernVirtAddr;
int rc = i_ddi_mem_alloc(NULL, &g_SolarisX86PhysMemLimits, sizeInBytes,
                         1, 0, NULL, &kernVirtAddr, NULL, NULL);

The prototype of i_ddi_mem_alloc is:

i_ddi_mem_alloc(dev_info_t *dip, ddi_dma_attr_t *attr,
        size_t length, int cansleep, int flags,
        ddi_device_acc_attr_t *accattrp, caddr_t *kaddrp,
        size_t *real_length, ddi_acc_hdl_t *ap)

So obviously, Sun intended this to be called only from within drivers, but we can safely pass NULL to those arguments we don't have as we have done in our above example. Don't forget to pass a non-zero value for cansleep unless you have a need for not allowing the allocation to be a non-waiting one.

The above code allocates physically contiguous, page-aligned memory. This can be used for DMA transfers or for other specific tasks that has these requirements.

The key to getting this to be contiguous is the scatter gather list length member of the structure. Setting this to 1 forces the kernel to allocate 1 physically contiguous block of memory!

The high and low limit in my example maxes out at 4 GB (0xffffffff). If you're building on 64-bit platforms or for Blade servers etc., you can safely go beyond this.

I thought I'd document this little 'trick' if you will as none of this as far as I know is really documented publically, you must dig deep into the bowels of the OpenSolaris kernel to find them. Even the allocation call isn't a documented one, but one that isn't likely to change in the kernel.

For freeing memory allocated you use would use:

i_ddi_mem_free(kernVirtAddr, NULL)

Warning:
For almost all your needs you can probably go through the well document DDI functions of Solaris. Don't use what I've suggested unless you really know what you are doing, and you are in a situation where the exposed DDI functions are not sufficient.

Note:
1. Changes subject to http://the-teknomancer.blogspot.com/2007/09/allocating-physically-contiguous-memory.html?showComment=1194010440000#c2179786440051591653
2. Intel IOMMU code in onnv-gate gives a good sample about converting virtual address to physical ones and vice versa.