Solaris Source Insight: PCI bus driver moduls - npe Part 1

 Mon Nov  9 16:17:38 CST 2009

We have seen how PCI bus probe enuerate and configure the PCI system. Let's go ahead with the details of PCI bus drivers. The PCI leaf device driver is function related, we will never touch the details of the fucntionality of a specific device.

[root@blu-nhm-ep:~]modinfo | grep PCI
 15 fffffffffba46ce0   bfb0   -   1  pci_autoconfig (PCI BIOS interface)
 37 fffffffffbacd3f0   ce28 183   1  npe (Host to PCIe nexus driver)
 38 fffffffffbad95c8   5f50   -   1  pcihp (PCI nexus hotplug support)
 40 fffffffffbae14f0   bb00   -   1  pcie (PCIE: PCI framework)
 89 fffffffff7bff000   4c68 184   1  pcieb (PCIe to PCI nexus driver)
 90 fffffffff7999000   1d68  84   1  pci_pci (PCI to PCI bridge nexus driver)


npe - Host to PCIe nexus driver
===============================

It's a bus driver registered with the following data.

[i86pc/io/pciex/npe.c]
 149 struct dev_ops npe_ops = {
 150 |_______DEVO_REV,|______|_______/* devo_rev */
 151 |_______0,|_____|_______|_______/* refcnt  */
 152 |_______npe_info,|______|_______/* info */
 153 |_______nulldev,|_______|_______/* identify */
 154 |_______nulldev,|_______|_______/* probe */
 155 |_______npe_attach,|____|_______/* attach */
 156 |_______npe_detach,|____|_______/* detach */
 157 |_______nulldev,|_______|_______/* reset */
 158 |_______&npe_cb_ops,|___|_______/* driver operations */
 159 |_______&npe_bus_ops,|__|_______/* bus operations */
 160 |_______NULL,|__|_______|_______/* power */
 161 |_______ddi_quiesce_not_needed,||_______/* quiesce */
 162 };

Look at attach function firstly.

---- Question: ----
How to access configure space of a PCI device in drivers?

---- Answer: ----
You can use the standard DDI interfaces to access the configure space. The interfaces include at least the following functions.

ddi_regs_map_setup()
ddi_regs_map_free()
ddi_getX()

Here is a sample:

[i86pc/io/pciex/npe_misc.c]
330 void
331 npe_enable_htmsi_children(dev_info_t *dip)
332 {
333 |_______dev_info_t *cdip = ddi_get_child(dip);
334 |_______ddi_acc_handle_t cfg_hdl;
335
336 |_______for (; cdip != NULL; cdip = ddi_get_next_sibling(cdip)) {
337 |_______|_______if (pci_config_setup(cdip, &cfg_hdl) != DDI_SUCCESS) {
338 |_______|_______|_______cmn_err(CE_NOTE, "!npe_enable_htmsi_children: "
339 |_______|_______|_______    "pci_config_setup failed for %s",
340 |_______|_______|_______    ddi_node_name(cdip));
341 |_______|_______}
342
343 |_______|_______(void) npe_enable_htmsi(cfg_hdl);
344 |_______|_______pci_config_teardown(&cfg_hdl);
345 |_______}
346 }

In this function, pci_config_setup() is called firstly to set up the pci configure space mapping.

323 |_______reg = pci_config_get16(cfg_hdl, ptr + PCI_CAP_ID_REGS_OFF);
324 |_______reg |= PCI_HTCAP_MSIMAP_ENABLE;
325
326 |_______pci_config_put16(cfg_hdl, ptr + PCI_CAP_ID_REGS_OFF, reg);

in npe_enable_htmsi(), pci configure space access functions, such as pci_config_get16() are able to be called.

For more deepness, how does the code work?

[common/os/sunpci.c]
  35 int
  36 pci_config_setup(dev_info_t *dip, ddi_acc_handle_t *handle)
  37 {
  38 |_______caddr_t|cfgaddr;
  39 |_______ddi_device_acc_attr_t attr;
  40
  41 |_______attr.devacc_attr_version = DDI_DEVICE_ATTR_V0;
  42 |_______attr.devacc_attr_endian_flags = DDI_STRUCTURE_LE_ACC;
  43 |_______attr.devacc_attr_dataorder = DDI_STRICTORDER_ACC;
  44
  45 |_______/* Check for fault management capabilities */
  46 |_______if (DDI_FM_ACC_ERR_CAP(ddi_fm_capable(dip))) {
  47 |_______|_______attr.devacc_attr_version = DDI_DEVICE_ATTR_V1;
  48 |_______|_______attr.devacc_attr_access = DDI_FLAGERR_ACC;
  49 |_______}
  50
  51 |_______return (ddi_regs_map_setup(dip, 0, &cfgaddr, 0, 0, &attr, handle));
  52 }

In pci_config_setup(), ddi_regs_map_setup() is called to set up a mapping for a register address space. If you read the man page for this fucntion, you will find the parameter description.

     int ddi_regs_map_setup(dev_info_t *dip, uint_t rnumber, caddr_t *addrp,
          offset_t offset, offset_t len, ddi_device_acc_attr_t *accattrp,
          ddi_acc_handle_t *handlep);

PARAMETERS
     dip         Pointer to the device's dev_info structure.

     rnumber     Index number to the register address space set.

     addrp       A platform-dependent value that, when  added  to
                 an  offset that is less than or equal to the len
                 parameter (see below), is used for the  dev_addr
                 argument   to   the  ddi_get,  ddi_mem_get,  and
                 ddi_io_get/put routines.

     offset      Offset into the register address space.

     len         Length to be mapped. If both len and  offset are 0, the
                 entire space is mapped.

     accattrp    Pointer to a device access  attribute  structure
                 of this mapping (see ddi_device_acc_attr(9S)).

     handlep     Pointer to a data access handle.

So line 51 is to set up mapping of the whole first register address space with the "attr" attribution. If you check add_reg_props() which was called during the first pass of pci enumeration,

[intel/io/pci/pci_boot.c]
2297 /*
2298  * Add the "reg" and "assigned-addresses" property
2299  */
2300 static int
2301 add_reg_props(dev_info_t *dip, uchar_t bus, uchar_t dev, uchar_t func,
2302     int config_op, int pciide)

You will find the first register set was encoded with bus, device and function numbers.

[intel/io/pci/pci_boot.c]
2327 |_______devloc = (uint_t)bus << 16 | (uint_t)dev << 11 | (uint_t)func << 8;
2328 |_______regs[0].pci_phys_hi = devloc;
2329 |_______nreg = 1;|______/* rest of regs[0] is all zero */

So, how ddi_regs_map_setup() manages to setup the mapping? It's simple as just call the bus driver's bus operation.

[common/os/sunddi.c]
7612 |_______result = ddi_map(dip, &mr, offset, len, addrp);

[common/os/sunddi.c]
 134 int
 135 ddi_map(dev_info_t *dp, ddi_map_req_t *mp, off_t offset,
 136     off_t len, caddr_t *addrp)
 137 {
 138 |_______dev_info_t *pdip;
 139
 140 |_______ASSERT(dp);
 141 |_______pdip = (dev_info_t *)DEVI(dp)->devi_parent;
 142 |_______return ((DEVI(pdip)->devi_ops->devo_bus_ops->bus_map)(pdip,
 143 |_______    dp, mp, offset, len, addrp));
 144 }

Oh, PCI bus operations are defined in the nexus driver such as npe. Let's check how npe handles it.

In summary, npe bus map function support three sorts of mappings: IO, Mem and configure space. If the request is for IO or memory, npe_bus_map() relies on its parent's (rootnex) bus map operation. If it's a configure space request, it depends on whether the device support MMIO of configure space.

[i86pc/io/pciex/npe_misc.c]
285 /*
286  * Checks to see if MMCFG is supported.
287  * Returns: TRUE if MMCFG is supported, FALSE if not.
288  *
289  * If a device is attached to a parent whose "dev_type" is "pciex",
290  * the device will support MMCFG access.  Otherwise, use legacy IOCFG access.
291  *
292  * Enable Legacy PCI config space access for AMD K8 north bridges.
293  *|_____Host bridge: AMD HyperTransport Technology Configuration
294  *|_____Host bridge: AMD Address Map
295  *|_____Host bridge: AMD DRAM Controller
296  *|_____Host bridge: AMD Miscellaneous Control
297  * These devices do not support MMCFG access.
298  */
299 boolean_t
300 npe_is_mmcfg_supported(dev_info_t *dip)
301 {
302 |_______int vendor_id, device_id;
303
304 |_______vendor_id = ddi_prop_get_int(DDI_DEV_T_ANY, dip, DDI_PROP_DONTPASS,
305 |_______    "vendor-id", -1);
306 |_______device_id = ddi_prop_get_int(DDI_DEV_T_ANY, dip, DDI_PROP_DONTPASS,
307 |_______    "device-id", -1);
308
309 |_______return !(npe_child_is_pci(dip) ||
310 |_______    IS_BAD_AMD_NTBRIDGE(vendor_id, device_id));
311 }

If it has MMIO supported configure space, it will be treated as a memory mapping request. Otherwise, the lagacy methods are used.

[i86pc/io/pciex/npe.c]
 386 /*
 387  * Configure the access handle for standard configuration space
 388  * access (see pci_fm_acc_setup for code that initializes the
 389  * access-function pointers).
 390  */
 391 static int
 392 npe_setup_std_pcicfg_acc(dev_info_t *rdip, ddi_map_req_t *mp,
 393     ddi_acc_hdl_t *hp, off_t offset, off_t len)
 394 {
 395 |_______int ret;
 396
 397 |_______if ((ret = pci_fm_acc_setup(hp, offset, len)) ==
 398 |_______    DDI_SUCCESS) {
 399 |_______|_______if (DDI_FM_ACC_ERR_CAP(ddi_fm_capable(rdip)) &&
 400 |_______|_______    mp->map_handlep->ah_acc.devacc_attr_access
 401 |_______|_______    != DDI_DEFAULT_ACC) {
 402 |_______|_______|_______ndi_fmc_insert(rdip, ACC_HANDLE,
 403 |_______|_______|_______    (void *)mp->map_handlep, NULL);
 404 |_______|_______}
 405 |_______}
 406 |_______return (ret);
 407 }
 
in pci_fm_acc_setup(), the get/put function pointers are assigned. It always set cautious mechanism for config space gets, while set those for put according to the attributes specified in devacc_attr_access of ddi_device_acc_attr structure.

[man ddi_device_acc_attr]
152      The values defined for devacc_attr_access are:
153
154      DDI_DEFAULT_ACC     If an I/O fault occurs, the system  will
155                          take  the default action, which might be
156                          to panic.
157
158
159      DDI_FLAGERR_ACC     Using  this  value  indicates  that  the
160                          driver  is  hardened:  able to cope with
161                          the incorrect results of I/O  operations
162                          that might result from an I/O fault. The
163                          value also  indicates  that  the  driver
164                          will use ddi_fm_acc_err_get(9F) to check
165                          access handles for faults on  a  regular
166                          basis.
167
168                          If possible, the system should not panic
169                          on such an I/O fault, but should instead
170                          mark the I/O handle  through  which  the
171                          access was made as having faulted.
172
173                          This value is  advisory:  it  tells  the
174                          system  that  the driver can continue in
175                          the face of I/O faults. The  value  does
176                          not  guarantee  that the system will not
177                          panic, as that depends on the nature  of
178                          the  fault  and  the capabilities of the
179                          system. It is quite  legitimate  for  an
180                          implementation  to  ignore this flag and
181                          panic anyway.
182
183
184      DDI_CAUTIOUS_ACC    This value indicates that an  I/O  fault
185                          is  anticipated and should be handled as
186                          gracefully as possible. For example, the
187                          framework  should  not  print  a console
188                          message.
 

[i86pc/io/pci/pci_common.c]
1151 |_______/*
1152 |_______ * always use cautious mechanism for config space gets
1153 |_______ */
1154 |_______ap->ahi_get8 = i_ddi_caut_get8;
1155 |_______ap->ahi_get16 = i_ddi_caut_get16;
1156 |_______ap->ahi_get32 = i_ddi_caut_get32;
1157 |_______ap->ahi_get64 = i_ddi_caut_get64;
1158 |_______ap->ahi_rep_get8 = i_ddi_caut_rep_get8;
1159 |_______ap->ahi_rep_get16 = i_ddi_caut_rep_get16;
1160 |_______ap->ahi_rep_get32 = i_ddi_caut_rep_get32;
1161 |_______ap->ahi_rep_get64 = i_ddi_caut_rep_get64;
1162 |_______if (hp->ah_acc.devacc_attr_access == DDI_CAUTIOUS_ACC) {
1163 |_______|_______ap->ahi_put8 = i_ddi_caut_put8;
1164 |_______|_______ap->ahi_put16 = i_ddi_caut_put16;
1165 |_______|_______ap->ahi_put32 = i_ddi_caut_put32;
1166 |_______|_______ap->ahi_put64 = i_ddi_caut_put64;
1167 |_______|_______ap->ahi_rep_put8 = i_ddi_caut_rep_put8;
1168 |_______|_______ap->ahi_rep_put16 = i_ddi_caut_rep_put16;
1169 |_______|_______ap->ahi_rep_put32 = i_ddi_caut_rep_put32;
1170 |_______|_______ap->ahi_rep_put64 = i_ddi_caut_rep_put64;
1171 |_______} else {
1172 |_______|_______ap->ahi_put8 = pci_config_wr8;
1173 |_______|_______ap->ahi_put16 = pci_config_wr16;
1174 |_______|_______ap->ahi_put32 = pci_config_wr32;
1175 |_______|_______ap->ahi_put64 = pci_config_wr64;
1176 |_______|_______ap->ahi_rep_put8 = pci_config_rep_wr8;
1177 |_______|_______ap->ahi_rep_put16 = pci_config_rep_wr16;
1178 |_______|_______ap->ahi_rep_put32 = pci_config_rep_wr32;
1179 |_______|_______ap->ahi_rep_put64 = pci_config_rep_wr64;
1180 |_______}

For cautious mechanism, the configure space access request will be handled as a bus_ctl request.

[common/os/sunddi.c]
 682 /*
 683  * Request bus_ctl parent to handle a bus_ctl request
 684  *
 685  * (The sparc version is in sparc_ddi.s)
 686  */
 687 int
 688 ddi_ctlops(dev_info_t *d, dev_info_t *r, ddi_ctl_enum_t op, void *a, void *v)
 689 {
 690 |_______int (*fp)();
 691
 692 |_______if (!d || !r)
 693 |_______|_______return (DDI_FAILURE);
 694
 695 |_______if ((d = (dev_info_t *)DEVI(d)->devi_bus_ctl) == NULL)
 696 |_______|_______return (DDI_FAILURE);
 697
 698 |_______fp = DEVI(d)->devi_ops->devo_bus_ops->bus_ctl;
 699 |_______return ((*fp)(d, r, op, a, v));
 700 }

In npe_bus_ctl(), the folowing path deals with pci configure space put/get.

[i86pc/io/pciex/npe.c]
 775 |_______case DDI_CTLOPS_PEEK:
 776 |_______case DDI_CTLOPS_POKE:
 777 |_______|_______return (pci_common_peekpoke(dip, rdip, ctlop, arg, result));

At last, the request is handled with one of the following functions.

pci_getb_func()
pci_getw_func()
pci_getl_func()
pci_putb_func()
pci_putw_func()
pci_putl_func()

These interfaces are defined in ./i86pc/os/pci_cfgspace.c.

[./i86pc/os/pci_cfgspace.c]
 56 /*
 57  * These function pointers lead to the actual implementation routines
 58  * for configuration space access.  Normally they lead to either the
 59  * pci_mech1_* or pci_mech2_* routines, but they can also lead to
 60  * routines that work around chipset bugs.
 61  */
 62 uint8_t (*pci_getb_func)(int bus, int dev, int func, int reg);
 63 uint16_t (*pci_getw_func)(int bus, int dev, int func, int reg);
 64 uint32_t (*pci_getl_func)(int bus, int dev, int func, int reg);
 65 void (*pci_putb_func)(int bus, int dev, int func, int reg, uint8_t val);
 66 void (*pci_putw_func)(int bus, int dev, int func, int reg, uint16_t val);
 67 void (*pci_putl_func)(int bus, int dev, int func, int reg, uint32_t val);

These gobal pointers were initialized at pci_check().

[./i86pc/os/pci_cfgspace.c]
 98 /*
 99  * This code determines if this system supports PCI and which
100  * type of configuration access method is used
101  */
102
103 static int
104 pci_check(void)
105 {

which is called in mlsetup() at boot time. mlsetup() contains routines called right before main(). Interposing this function before main() allows us to call it in a machine-independent fashion.

Okay, back to the topic. For non-cautious mechanism, it will directly call pci configure space access methods.

pci_getb_func()
pci_getw_func()
pci_getl_func()
pci_putb_func()
pci_putw_func()
pci_putl_func()

That means, in current kernel there is no difference between the different configure access attribute in "devacc_attr_access".

---- End of Q/A ----

To be continued ... ....