实验环境:LC480T加速卡
开发环境:windows11+vivado2020
运行环境:ubuntu22.04
硬件电路:LC480T加速卡(xc7k480tffg1156-2)
vivado工程文件下载:
https://download.youkuaiyun.com/download/xiaolangyangyang/91349686
https://download.youkuaiyun.com/download/xiaolangyangyang/91388174
驱动及应用代码下载:https://download.youkuaiyun.com/download/xiaolangyangyang/91349692
LC480T引脚说明下载:https://download.youkuaiyun.com/download/xiaolangyangyang/91350582
xapp1052_xbmd寄存器说明下载:https://download.youkuaiyun.com/download/xiaolangyangyang/91350584
7 Series FPGAs Integrated Block for PCI Express spec:AMD Technical Information Portal
一、说明
xilinx官网xapp1052例程是基于Kintex-7 KC705 Evaluation Platform (xc7k325tffg900-2)开发板,使用7 Series FPGAs Integrated Block for PCI Express ip核,如果直接将项目修改成device为xc7k480tffg1156-2生成bit文件过程会报错,所以重新构建工程,将官网xapp1052例程的xbmd文件夹拷贝到工程,修改顶层文件及约束文件,生成bit文件。
xilinx官网xapp1052例程软件基于Fedora-10操作系统,内核很旧,代码在ubuntu22.04上编译不过,所以考虑从新编写驱动及应用代码。
二、构建vivado项目并生成bin文件
打开工程,生成bit即可,约束文件如下:
##-----------------------------------------------------------------------------
##
## (c) Copyright 2010-2011 Xilinx, Inc. All rights reserved.
##
## This file contains confidential and proprietary information
## of Xilinx, Inc. and is protected under U.S. and
## international copyright and other intellectual property
## laws.
##
## DISCLAIMER
## This disclaimer is not a license and does not grant any
## rights to the materials distributed herewith. Except as
## otherwise provided in a valid license issued to you by
## Xilinx, and to the maximum extent permitted by applicable
## law: (1) THESE MATERIALS ARE MADE AVAILABLE "AS IS" AND
## WITH ALL FAULTS, AND XILINX HEREBY DISCLAIMS ALL WARRANTIES
## AND CONDITIONS, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING
## BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, NON-
## INFRINGEMENT, OR FITNESS FOR ANY PARTICULAR PURPOSE; and
## (2) Xilinx shall not be liable (whether in contract or tort,
## including negligence, or under any other theory of
## liability) for any loss or damage of any kind or nature
## related to, arising under or in connection with these
## materials, including for any direct, or any indirect,
## special, incidental, or consequential loss or damage
## (including loss of data, profits, goodwill, or any type of
## loss or damage suffered as a result of any action brought
## by a third party) even if such damage or loss was
## reasonably foreseeable or Xilinx had been advised of the
## possibility of the same.
##
## CRITICAL APPLICATIONS
## Xilinx products are not designed or intended to be fail-
## safe, or for use in any application requiring fail-safe
## performance, such as life-support or safety devices or
## systems, Class III medical devices, nuclear facilities,
## applications related to the deployment of airbags, or any
## other applications that could lead to death, personal
## injury, or severe property or environmental damage
## (individually and collectively, "Critical
## Applications"). Customer assumes the sole risk and
## liability of any use of Xilinx products in Critical
## Applications, subject only to applicable laws and
## regulations governing limitations on product liability.
##
## THIS COPYRIGHT NOTICE AND DISCLAIMER MUST BE RETAINED AS
## PART OF THIS FILE AT ALL TIMES.
##
##-----------------------------------------------------------------------------
## Project : Series-7 Integrated Block for PCI Express
## File : xilinx_pcie_7x_ep_x8g2.xdc
## Version : 3.3
#
###############################################################################
# User Configuration
# Link Width - x8
# Link Speed - gen2
# Family - kintex7
# Part - xc7k480t
# Package - ffg1156
# Speed grade - -2
# PCIe Block - X0Y0
###############################################################################
#
###############################################################################
# User Time Names / User Time Groups / Time Specs
###############################################################################
###############################################################################
# User Physical Constraints
###############################################################################
###############################################################################
# Pinout and Related I/O Constraints
###############################################################################
#
# SYS reset (input) signal. The sys_reset_n signal should be
# obtained from the PCI Express interface if possible. For
# slot based form factors, a system reset signal is usually
# present on the connector. For cable based form factors, a
# system reset signal may not be available. In this case, the
# system reset signal must be generated locally by some form of
# supervisory circuit. You may change the IOSTANDARD and LOC
# to suit your requirements and VCCO voltage banking rules.
# Some 7 series devices do not have 3.3 V I/Os available.
# Therefore the appropriate level shift is required to operate
# with these devices that contain only 1.8 V banks.
#
set_property PULLUP true [get_ports sys_rst_n]
###############################################################################
# Physical Constraints
###############################################################################
#
# SYS clock 100 MHz (input) signal. The sys_clk_p and sys_clk_n
# signals are the PCI Express reference clock. Virtex-7 GT
# Transceiver architecture requires the use of a dedicated clock
# resources (FPGA input pins) associated with each GT Transceiver.
# To use these pins an IBUFDS primitive (refclk_ibuf) is
# instantiated in user's design.
# Please refer to the Virtex-7 GT Transceiver User Guide
# (UG) for guidelines regarding clock resource selection.
#
set_property LOC IBUFDS_GTE2_X0Y9 [get_cells refclk_ibuf]
###############################################################################
# Timing Constraints
###############################################################################
#
create_clock -name sys_clk -period 10 [get_ports sys_clk_p]
#
#
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
#
#
create_generated_clock -name clk_125mhz_x0y0 [get_pins pcie_7x_0_support_i/pipe_clock_i/mmcm_i/CLKOUT0]
create_generated_clock -name clk_250mhz_x0y0 [get_pins pcie_7x_0_support_i/pipe_clock_i/mmcm_i/CLKOUT1]
create_generated_clock -name clk_125mhz_mux_x0y0 \
-source [get_pins pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/I0] \
-divide_by 1 \
[get_pins pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/O]
#
create_generated_clock -name clk_250mhz_mux_x0y0 \
-source [get_pins pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/I1] \
-divide_by 1 -add -master_clock [get_clocks -of [get_pins pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/I1]] \
[get_pins pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/O]
#
set_clock_groups -name pcieclkmux -physically_exclusive -group clk_125mhz_mux_x0y0 -group clk_250mhz_mux_x0y0
#
# Timing ignoring the below pins to avoid CDC analysis, but care has been taken in RTL to sync properly to other clock domain.
#
#
##############################################################################
# Tandem Configuration Constraints
###############################################################################
set_false_path -from [get_ports sys_rst_n]
set_property PACKAGE_PIN J8 [get_ports sys_clk_p]
set_property IOSTANDARD LVCMOS33 [get_ports sys_rst_n]
set_property PACKAGE_PIN Y26 [get_ports sys_rst_n]
set_property LOC GTXE2_CHANNEL_X0Y23 [get_cells {pcie_7x_1_support_i/pcie_7x_1_i/inst/inst/gt_top_i/pipe_wrapper_i/pipe_lane[0].gt_wrapper_i/gtx_channel.gtxe2_channel_i}]
set_property PACKAGE_PIN F2 [get_ports {pci_exp_txp[0]}]
set_property LOC GTXE2_CHANNEL_X0Y22 [get_cells {pcie_7x_1_support_i/pcie_7x_1_i/inst/inst/gt_top_i/pipe_wrapper_i/pipe_lane[1].gt_wrapper_i/gtx_channel.gtxe2_channel_i}]
set_property PACKAGE_PIN H2 [get_ports {pci_exp_txp[1]}]
set_property LOC GTXE2_CHANNEL_X0Y21 [get_cells {pcie_7x_1_support_i/pcie_7x_1_i/inst/inst/gt_top_i/pipe_wrapper_i/pipe_lane[2].gt_wrapper_i/gtx_channel.gtxe2_channel_i}]
set_property PACKAGE_PIN K2 [get_ports {pci_exp_txp[2]}]
set_property LOC GTXE2_CHANNEL_X0Y20 [get_cells {pcie_7x_1_support_i/pcie_7x_1_i/inst/inst/gt_top_i/pipe_wrapper_i/pipe_lane[3].gt_wrapper_i/gtx_channel.gtxe2_channel_i}]
set_property PACKAGE_PIN M2 [get_ports {pci_exp_txp[3]}]
set_property LOC GTXE2_CHANNEL_X0Y19 [get_cells {pcie_7x_1_support_i/pcie_7x_1_i/inst/inst/gt_top_i/pipe_wrapper_i/pipe_lane[4].gt_wrapper_i/gtx_channel.gtxe2_channel_i}]
set_property PACKAGE_PIN N4 [get_ports {pci_exp_txp[4]}]
set_property LOC GTXE2_CHANNEL_X0Y18 [get_cells {pcie_7x_1_support_i/pcie_7x_1_i/inst/inst/gt_top_i/pipe_wrapper_i/pipe_lane[5].gt_wrapper_i/gtx_channel.gtxe2_channel_i}]
set_property PACKAGE_PIN P2 [get_ports {pci_exp_txp[5]}]
set_property LOC GTXE2_CHANNEL_X0Y17 [get_cells {pcie_7x_1_support_i/pcie_7x_1_i/inst/inst/gt_top_i/pipe_wrapper_i/pipe_lane[6].gt_wrapper_i/gtx_channel.gtxe2_channel_i}]
set_property PACKAGE_PIN T2 [get_ports {pci_exp_txp[6]}]
set_property LOC GTXE2_CHANNEL_X0Y16 [get_cells {pcie_7x_1_support_i/pcie_7x_1_i/inst/inst/gt_top_i/pipe_wrapper_i/pipe_lane[7].gt_wrapper_i/gtx_channel.gtxe2_channel_i}]
set_property PACKAGE_PIN U4 [get_ports {pci_exp_txp[7]}]
###############################################################################
# End
###############################################################################
三、代码列表
头文件:
//--------------------------------------------------------------------------------
//-- Filename: xbmd.h
//--
//-- Description: Main header file for kernel driver
//--
//-- XBMD is an example Red Hat device driver which exercises XBMD design
//-- Device driver has been tested on Red Hat Fedora FC9 2.6.15.
//--------------------------------------------------------------------------------
// Define Result values
#define SUCCESS 0
#define CRIT_ERR -1
// Debug - define will output more info
#define Verbose 1
// Max DMA Buffer Size
#define BUF_SIZE (4096 * 2)
enum {
INITCARD,
INITRST,
DISPREGS,
RDDCSR,
RDDDMACR,
RDWDMATLPA,
RDWDMATLPS,
RDWDMATLPC,
RDWDMATLPP,
RDRDMATLPP,
RDRDMATLPA,
RDRDMATLPS,
RDRDMATLPC,
RDWDMAPERF,
RDRDMAPERF,
RDRDMASTAT,
RDNRDCOMP,
RDRCOMPDSIZE,
RDDLWSTAT,
RDDLTRSSTAT,
RDDMISCCONT,
RDDMISCONT,
RDDLNKC,
DFCCTL,
DFCPINFO,
DFCNPINFO,
DFCINFO,
WRDDMACR,
WRWDMATLPS,
WRWDMATLPC,
WRWDMATLPP,
WRRDMATLPS,
WRRDMATLPC,
WRRDMATLPP,
WRDMISCCONT,
WRDDLNKC,
NUMCOMMANDS
};
linux驱动代码:
#include <linux/module.h>
#include <linux/pci.h>
#include <linux/kernel.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/interrupt.h>
#include <linux/device.h>
#include "xbmd.h"
#define CLASS_NAME "xbmd"
#define DEVICE_NAME "xbmd"
static int major_number = 530;
static struct cdev *xbmd_cdev;
void __iomem *regs;
struct pci_dev *gDev = NULL;
static struct class *xbmd_class = NULL;
static struct device *xbmd_device = NULL;
char *gReadBuffer = NULL;
char *gWriteBuffer = NULL;
dma_addr_t gReadHWAddr;
dma_addr_t gWriteHWAddr;
int gIrq;
static const struct pci_device_id xbmd_ids[] = {
{ PCI_DEVICE(0x10ee, 0x7028) },
{ 0, }
};
dev_t devno = MKDEV(295, 0);
MODULE_DEVICE_TABLE(pci, xbmd_ids);
u32 XPCIe_ReadReg (u32 dw_offset)
{
u32 ret = 0;
ret = ioread32(regs + (4 * dw_offset));
printk("XPCIe_ReadReg @0x%x = 0x%x", regs + (4 * dw_offset), ret);
return ret;
}
void XPCIe_WriteReg (u32 dw_offset, u32 val)
{
printk("XPCIe_WriteReg @0x%x = 0x%x", regs + (4 * dw_offset), val);
iowrite32(val, (regs + (4 * dw_offset)));
}
u32 XPCIe_ReadCfgReg (u32 byte)
{
u32 pciReg;
if (pci_read_config_dword(gDev, byte, &pciReg) < 0) {
printk("%s: XPCIe_ReadCfgReg: Reading PCI interface failed.", DEVICE_NAME);
return (-1);
}
return (pciReg);
}
u32 XPCIe_WriteCfgReg (u32 byte, u32 val)
{
if (pci_write_config_dword(gDev, byte, val) < 0) {
printk("%s: XPCIe_Read Device Control: Reading PCI interface failed.", DEVICE_NAME);
return (-1);
}
return 1;
}
void XPCIe_InitCard(void)
{
XPCIe_WriteReg(0, 1); // Write: DCSR (offset 0) with value of 1 (Reset Device)
XPCIe_WriteReg(0, 0); // Write: DCSR (offset 0) with value of 0 (Make Active)
XPCIe_WriteReg(1, 0);
XPCIe_WriteReg(2, gWriteHWAddr); // Write: Write DMA TLP Address register with starting address
XPCIe_WriteReg(3, 0x20); // Write: Write DMA TLP Size register with default value (32dwords)
XPCIe_WriteReg(4, 0x2000); // Write: Write DMA TLP Count register with default value (2000)
XPCIe_WriteReg(5, 0x00000000); // Write: Write DMA TLP Pattern register with default value (0x0)
XPCIe_WriteReg(6, 0xfeedbeef); // Write: Read DMA Expected Data Pattern with default value (feedbeef)
XPCIe_WriteReg(7, gReadHWAddr); // Write: Read DMA TLP Address register with starting address.
XPCIe_WriteReg(8, 0x20); // Write: Read DMA TLP Size register with default value (32dwords)
XPCIe_WriteReg(9, 0x2000); // Write: Read DMA TLP Count register with default value (2000)
printk("XPCIe_InitCard ok");
}
void XPCIe_InitiatorReset(void)
{
XPCIe_WriteReg(0, 1); // Write: DCSR (offset 0) with value of 1 (Reset Device)
XPCIe_WriteReg(0, 0); // Write: DCSR (offset 0) with value of 0 (Make Active)
printk("XPCIe_InitiatorReset ok");
}
static int xbmd_open(struct inode *inode, struct file *file)
{
printk(KERN_INFO "Device opened\n");
return 0;
}
static int xbmd_release(struct inode *inode, struct file *file)
{
printk(KERN_INFO "Device released\n");
return 0;
}
static ssize_t xbmd_read(struct file *file, char __user *buffer, size_t length, loff_t *offset)
{
copy_to_user(buffer, gWriteBuffer, length);
printk(KERN_INFO "%s: XPCIe_Read: %d bytes have been read...\n", DEVICE_NAME, length);
return 0;
}
static ssize_t xbmd_write(struct file *file, const char __user *buffer, size_t length, loff_t *offset)
{
int ret = SUCCESS;
copy_from_user(gReadBuffer, buffer, length);
printk(KERN_INFO "%s: XPCIe_Write: %d bytes have been written...\n", DEVICE_NAME, length);
return ret;
}
long xbmd_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
{
u32 regx;
long ret = SUCCESS;
switch (cmd) {
case INITCARD: // Initailizes XBMD application
XPCIe_InitCard();
memset(gReadBuffer, 0x0, sizeof(gReadBuffer));
memset(gWriteBuffer, 0x0, sizeof(gWriteBuffer));
break;
case INITRST: // Resets XBMD applications
XPCIe_InitiatorReset();
break;
case DISPREGS:
break;
case RDDCSR: // Read: Device Control Status Register
regx = XPCIe_ReadReg(0);
__put_user(regx, (int *)arg);
break;
case RDDDMACR: // Read: DMA Control Status Register
regx = XPCIe_ReadReg(1);
__put_user(regx, (int *)arg);
break;
case RDWDMATLPA: // Read: Write DMA TLP Address Register
regx = XPCIe_ReadReg(2);
__put_user(regx, (int *)arg);
break;
case RDWDMATLPS: // Read: Write DMA TLP Size Register
regx = XPCIe_ReadReg(3);
__put_user(regx, (int *)arg);
break;
case RDWDMATLPC: // Read: Write DMA TLP Count Register
regx = XPCIe_ReadReg(4);
__put_user(regx, (int *)arg);
break;
case RDWDMATLPP: // Read: Write DMA TLP Pattern Register
regx = XPCIe_ReadReg(5);
__put_user(regx, (int *)arg);
break;
case RDRDMATLPP: // Read: Read DMA TLP Pattern Register
regx = XPCIe_ReadReg(6);
__put_user(regx, (int *)arg);
break;
case RDRDMATLPA: // Read: Read DMA TLP Address Register
regx = XPCIe_ReadReg(7);
__put_user(regx, (int *)arg);
break;
case RDRDMATLPS: // Read: Read DMA TLP Size Register
regx = XPCIe_ReadReg(8);
__put_user(regx, (int *)arg);
break;
case RDRDMATLPC: // Read: Read DMA TLP Count Register
regx = XPCIe_ReadReg(9);
__put_user(regx, (int *)arg);
break;
case RDWDMAPERF: // Read: Write DMA Performance Register
regx = XPCIe_ReadReg(10);
__put_user(regx, (int *)arg);
break;
case RDRDMAPERF: // Read: Read DMA Performance Register
regx = XPCIe_ReadReg(11);
__put_user(regx, (int *)arg);
break;
case RDRDMASTAT: // Read: Read DMA Status Register
regx = XPCIe_ReadReg(12);
__put_user(regx, (int *)arg);
break;
case RDNRDCOMP: // Read: Number of Read Completion w/ Data Register
regx = XPCIe_ReadReg(13);
__put_user(regx, (int *)arg);
break;
case RDRCOMPDSIZE: // Read: Read Completion Size Register
regx = XPCIe_ReadReg(14);
__put_user(regx, (int *)arg);
break;
case RDDLWSTAT: // Read: Device Link Width Status Register
regx = XPCIe_ReadReg(15);
__put_user(regx, (int *)arg);
break;
case RDDLTRSSTAT: // Read: Device Link Transaction Size Status Register
regx = XPCIe_ReadReg(16);
__put_user(regx, (int *)arg);
break;
case RDDMISCCONT: // Read: Device Miscellaneous Control Register
regx = XPCIe_ReadReg(17);
__put_user(regx, (int *)arg);
break;
case RDDMISCONT: // Read: Device MSI Control
regx = XPCIe_ReadReg(18);
__put_user(regx, (int *)arg);
break;
case RDDLNKC: // Read: Device Directed Link Change Register
regx = XPCIe_ReadReg(19);
__put_user(regx, (int *)arg);
break;
case DFCCTL: // Read: Device FC Control Register
regx = XPCIe_ReadReg(20);
__put_user(regx, (int *)arg);
break;
case DFCPINFO: // Read: Device FC Posted Information
regx = XPCIe_ReadReg(21);
__put_user(regx, (int *)arg);
break;
case DFCNPINFO: // Read: Device FC Non Posted Information
regx = XPCIe_ReadReg(22);
__put_user(regx, (int *)arg);
break;
case DFCINFO: // Read: Device FC Completion Information
regx = XPCIe_ReadReg(23);
__put_user(regx, (int *)arg);
break;
case WRDDMACR: // Write: DMA Control Status Register
__get_user(regx, (int *)arg);
XPCIe_WriteReg(1, regx);
break;
case WRWDMATLPS: // Write: Write DMA TLP Size Register
__get_user(regx, (int *)arg);
XPCIe_WriteReg(3, regx);
break;
case WRWDMATLPC: // Write: Write DMA TLP Count Register
__get_user(regx, (int *)arg);
XPCIe_WriteReg(4, regx);
break;
case WRWDMATLPP: // Write: Write DMA TLP Pattern Register
__get_user(regx, (int *)arg);
XPCIe_WriteReg(5, regx);
break;
case WRRDMATLPS: // Write: Read DMA TLP Size Register
__get_user(regx, (int *)arg);
XPCIe_WriteReg(8, regx);
break;
case WRRDMATLPC: // Write: Read DMA TLP Count Register
__get_user(regx, (int *)arg);
XPCIe_WriteReg(9, regx);
break;
case WRRDMATLPP: // Write: Read DMA TLP Pattern Register
__get_user(regx, (int *)arg);
XPCIe_WriteReg(6, regx);
break;
case WRDMISCCONT: // Write: Device Miscellaneous Control Register
__get_user(regx, (int *)arg);
XPCIe_WriteReg(18, regx);
break;
case WRDDLNKC: // Write: Device Directed Link Change Register
__get_user(regx, (int *)arg);
XPCIe_WriteReg(19, regx);
break;
default:
break;
}
return ret;
}
static struct file_operations xbmd_fops = {
.owner = THIS_MODULE,
.open = xbmd_open,
.release = xbmd_release,
.read = xbmd_read,
.write = xbmd_write,
.unlocked_ioctl = xbmd_ioctl,
};
irqreturn_t XPCIe_IRQHandler(int irq, void *dev_id)
{
u32 i, regx;
printk(KERN_WARNING"%s: Interrupt Handler Start ..", DEVICE_NAME);
for (i = 0; i < 32; i++) {
regx = XPCIe_ReadReg(i);
printk(KERN_WARNING"%s : REG<%d> : 0x%X\n", DEVICE_NAME, i, regx);
}
printk(KERN_WARNING"%s Interrupt Handler End ..\n", DEVICE_NAME);
return IRQ_NONE;
}
static int xbmd_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
int ret;
gDev = pdev;
ret = pci_enable_device(pdev);
if (ret) {
printk(KERN_ERR "Failed to enable PCI device\n");
return ret;
}
ret = pci_request_regions(pdev, "xbmd_driver");
if (ret) {
printk(KERN_ERR "Failed to request regions\n");
goto err_disable;
}
regs = pci_ioremap_bar(pdev, 0); // BAR0
if (!regs) {
printk(KERN_ERR "Failed to map BAR0\n");
ret = -ENOMEM;
goto err_release;
}
//u32 value = ioread32(regs);
//printk(KERN_INFO "Register @0x%x value: 0x%x\n", regs, value);
gIrq = gDev->irq;
printk(KERN_INFO"%s: Init: Device IRQ: %d\n",DEVICE_NAME, gIrq);
printk(KERN_INFO"%s: ISR Setup..\n", DEVICE_NAME);
if (0 > request_irq(gIrq, XPCIe_IRQHandler, IRQF_SHARED, DEVICE_NAME, gDev)) {
printk(KERN_INFO"%s: Init: Unable to allocate IRQ",DEVICE_NAME);
return (CRIT_ERR);
}
gReadBuffer = dma_alloc_coherent(&pdev->dev, BUF_SIZE, &gReadHWAddr, GFP_KERNEL);
if (NULL == gReadBuffer) {
printk(KERN_INFO"%s: Init: Unable to allocate gBuffer.\n", DEVICE_NAME);
return (CRIT_ERR);
}
printk(KERN_INFO"%s: Read Buffer Allocation: %X->%X\n", DEVICE_NAME, (u32)gReadBuffer, (u32)gReadHWAddr);
gWriteBuffer = dma_alloc_coherent(&pdev->dev, BUF_SIZE, &gWriteHWAddr, GFP_KERNEL);
if (NULL == gWriteBuffer) {
printk(KERN_INFO"%s: Init: Unable to allocate gBuffer.\n", DEVICE_NAME);
return (CRIT_ERR);
}
printk(KERN_INFO"%s: Write Buffer Allocation: %X->%X\n", DEVICE_NAME, (u32)gWriteBuffer, (u32)gWriteHWAddr);
XPCIe_InitCard();
printk("xbmd_probe ok\n");
return 0;
err_release:
pci_release_regions(pdev);
err_disable:
pci_disable_device(pdev);
return ret;
}
static void xbmd_remove(struct pci_dev *pdev)
{
free_irq(gIrq, gDev);
pci_release_regions(pdev);
pci_disable_device(pdev);
dma_free_coherent(&gDev->dev, BUF_SIZE, gReadBuffer, gReadHWAddr);
dma_free_coherent(&gDev->dev, BUF_SIZE, gWriteBuffer, gWriteHWAddr);
if (NULL != gReadBuffer)
(void) kfree(gReadBuffer);
if (NULL != gWriteBuffer)
(void) kfree(gWriteBuffer);
gReadBuffer = NULL;
gWriteBuffer = NULL;
if (regs) {
printk("Iounmap\n");
iounmap(regs);
}
printk("xbmd_remove ok\n");
}
static struct pci_driver xbmd_driver = {
.name = "xbmd_driver",
.id_table = xbmd_ids,
.probe = xbmd_probe,
.remove = xbmd_remove,
};
#if 1
static int __init xbmd_init(void)
{
int ret;
ret = pci_register_driver(&xbmd_driver);
if (ret < 0)
return ret;
major_number = register_chrdev(0, DEVICE_NAME, &xbmd_fops);
if (major_number < 0) {
printk(KERN_ALERT "Registering char device failed with %d\n", major_number);
goto out_pci_unregister_driver;
}
printk(KERN_INFO "Char device registered with major number %d\n", major_number);
xbmd_class = class_create(CLASS_NAME);
if (IS_ERR(xbmd_class)) {
ret = PTR_ERR(xbmd_class);
goto out_unregister_chedev;
}
xbmd_device = device_create(xbmd_class, NULL, MKDEV(major_number, 0), NULL, DEVICE_NAME);
if (IS_ERR(xbmd_device)) {
ret = PTR_ERR(xbmd_device);
goto out_device_destroy;
}
printk("xbmd_init ok\n");
return 0;
out_device_destroy:
class_destroy(xbmd_class);
out_unregister_chedev:
unregister_chrdev(major_number, DEVICE_NAME);
out_pci_unregister_driver:
pci_unregister_driver(&xbmd_driver);
exit:
return ret;
}
#else
static int __init xbmd_init(void)
{
int ret;
ret = pci_register_driver(&xbmd_driver);
if (ret < 0)
return ret;
#if 0
ret = alloc_chrdev_region(&devno, 0, 1, DEVICE_NAME);
if (ret < 0)
goto out_pci_unregister_driver;
#else
ret = register_chrdev_region(devno, 1, DEVICE_NAME);
if (ret < 0)
goto out_pci_unregister_driver;
#endif
xbmd_cdev = cdev_alloc();
if (IS_ERR(xbmd_cdev)) {
ret = PTR_ERR(xbmd_cdev);
goto out_unregister_dev;
}
cdev_init(xbmd_cdev, &xbmd_fops);
xbmd_cdev->owner = THIS_MODULE;
ret = cdev_add(xbmd_cdev, devno, 1);
if (ret)
goto out_free_cdev;
xbmd_class = class_create(CLASS_NAME);
if (IS_ERR(xbmd_class)) {
ret = PTR_ERR(xbmd_class);
goto out_unregister_cdev;
}
xbmd_device = device_create(xbmd_class, NULL, devno, NULL, DEVICE_NAME);
if (IS_ERR(xbmd_device)) {
ret = PTR_ERR(xbmd_device);
goto out_del_class;
}
return 0;
out_del_class:
class_destroy(xbmd_class);
out_unregister_cdev:
cdev_del(xbmd_cdev);
out_free_cdev:
kfree(xbmd_cdev);
out_unregister_dev:
unregister_chrdev_region(devno, 1);
out_pci_unregister_driver:
pci_unregister_driver(&xbmd_driver);
return ret;
}
#endif
#if 1
static void __exit xbmd_exit(void)
{
if (xbmd_class) {
printk("Destroy device and class\n");
device_destroy(xbmd_class, MKDEV(major_number, 0));
class_destroy(xbmd_class);
}
unregister_chrdev(major_number, DEVICE_NAME);
pci_unregister_driver(&xbmd_driver);
printk("xbmd_exit ok\n");
}
#else
static void __exit xbmd_exit(void)
{
if (xbmd_class) {
printk("Destroy device and class\n");
device_destroy(xbmd_class, devno);
class_destroy(xbmd_class);
}
cdev_del(xbmd_cdev);
kfree(xbmd_cdev);
unregister_chrdev_region(devno, 1);
pci_unregister_driver(&xbmd_driver);
printk("xbmd_exit ok\n");
}
#endif
module_init(xbmd_init);
module_exit(xbmd_exit);
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Charley Zhang");
MODULE_DESCRIPTION("Xilinx xbmd");
linux应用层代码
#include <sys/ioctl.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include "xbmd.h"
unsigned int rxBuffer[1024];
unsigned int txBuffer[1024];
int main(int)
{
int ret = 0;
int devFd = 0;
int i;
for (i = 0; i < 1024; i++) {
rxBuffer[i] = 0x0;
txBuffer[i] = 0xfeedbeef;
}
devFd = open("/dev/xbmd", O_RDWR);
if ( devFd < 0 ) {
printf("Open /dev/xbmd failed!\n");
return -1;
}
// initcard
ioctl(devFd, INITCARD, &ret);
write(devFd, txBuffer, sizeof(txBuffer));
// enable irq & start dma
ret = 0x00810081;
ioctl(devFd, WRDDMACR, &ret);
// wait dma complete
usleep(1000);
ioctl(devFd, RDDDMACR, &ret);
printf("RDDDMACR = 0x%x\n", ret);
// check data
ret = read(devFd, rxBuffer, sizeof(rxBuffer));
printf("rxBuffer[0] = 0x%x\n", rxBuffer[0]);
printf("rxBuffer[1] = 0x%x\n", rxBuffer[1]);
return 0;
}
Makefile
obj-m += xbmd.o
all:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
clean:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
rm -rf app
测试脚本
dmesg -c
make clean
make
gcc-12 -o app app.c
rmmod xbmd.ko
dmesg -c
insmod xbmd.ko
dmesg -c
./app
dmesg -c
四、测试
# ./run_bmd.csh
五、测试结果
实际运行dma没有启动,原因待分析。
六、固化bit文件
使用BPI Flash,待研究。
七、疑问
7 Series FPGAs Integrated Block for PCI Express 实现了CfgRd0 CfgRd1 CfgWr0 CfgWr1等配置TLP吗?还是xbmd代码实现的,如下的BMD_CFG_CTRL.v文件有什么作用?
- EP配置空间由IP实现,通过CFG端口将部分配置寄存器映射到信号上,同时CFG端口提供cfg_mgmt_dwaddr、cfg_mgmt_di、cfg_mgmt_do信号按地址偏移读取配置寄存器,中断由CFG端口interrupt信号线触发,所以无需在AXI总线上回复配置相关TLP包,也无需通过AXI总线写MWr的TLP包去触发中断;
- BMD_CFG_CTRL.v文件主要实现EP对配置寄存器的读写;
- MEM是如何绑定到BAR0:EP端的MEM地址为0x0,PC读写EP端MEM时,TLP包中的地址为PC端分配的PCIe域地址,EP端解析地址时取了地址的bit[12:2],最终带入MEM的地址取了其中的bit[8:2],由于BAR0配置为0xFFFFF800,BAR0长度为2K,所以这个过程已经减去了BAR0的基地址,屏蔽了[1:0],所以是4byte对其,地址线到MEM模块就直接是寄存器编号了。
八、7 Series FPGAs Integrated Block for PCI Express 信号
//----------------------------------------------------------------------------------------------------------------//
// PCI Express (pci_exp) Interface //
//----------------------------------------------------------------------------------------------------------------//
// Tx
output [3:0]pci_exp_txn; ( pci_exp_txn ),
output [3:0]pci_exp_txp; ( pci_exp_txp ),
// Rx
input [3:0]pci_exp_rxn; ( pci_exp_rxn ),
input [3:0]pci_exp_rxp; ( pci_exp_rxp ),
//----------------------------------------------------------------------------------------------------------------//
// AXI-S Interface //
//----------------------------------------------------------------------------------------------------------------//
// Common
output user_clk_out; ( user_clk ),
output user_reset_out; ( user_reset ),
output user_lnk_up; ( user_lnk_up ),
output user_app_rdy; ( ),
// TX
output s_axis_tx_tready; ( s_axis_tx_tready ), // connect to pcie_app_7xconnect
input [63:0]s_axis_tx_tdata; ( s_axis_tx_tdata ), // connect to pcie_app_7xconnect
input [7:0]s_axis_tx_tkeep; ( s_axis_tx_tkeep ), // connect to pcie_app_7xconnect
input [3:0]s_axis_tx_tuser; ( s_axis_tx_tuser ), // connect to pcie_app_7xconnect
input s_axis_tx_tlast; ( s_axis_tx_tlast ), // connect to pcie_app_7xconnect
input s_axis_tx_tvalid; ( s_axis_tx_tvalid ), // connect to pcie_app_7xconnect
// Rx
output [63:0]m_axis_rx_tdata; ( m_axis_rx_tdata ), // connect to pcie_app_7xconnect
output [7:0]m_axis_rx_tkeep; ( m_axis_rx_tkeep ), // connect to pcie_app_7xconnect
output m_axis_rx_tlast; ( m_axis_rx_tlast ), // connect to pcie_app_7xconnect
output m_axis_rx_tvalid; ( m_axis_rx_tvalid ), // connect to pcie_app_7xconnect
input m_axis_rx_tready; ( m_axis_rx_tready ), // connect to pcie_app_7xconnect
output [21:0]m_axis_rx_tuser; ( m_axis_rx_tuser ), // connect to pcie_app_7xconnect
input tx_cfg_gnt; ( tx_cfg_gnt ), // connect to pcie_app_7xconnect
input rx_np_ok; ( rx_np_ok ), // connect to pcie_app_7xconnect
input rx_np_req; ( rx_np_req ), // connect to pcie_app_7xconnect
input cfg_trn_pending; ( cfg_trn_pending ), // connect to pcie_app_7xconnect
input cfg_pm_halt_aspm_l0s; ( cfg_pm_halt_aspm_l0s ),
input cfg_pm_halt_aspm_l1; ( cfg_pm_halt_aspm_l1 ),
input cfg_pm_force_state_en; ( cfg_pm_force_state_en ),
input [1:0]cfg_pm_force_state; ( cfg_pm_force_state ),
input [63:0]cfg_dsn; ( cfg_dsn ), // connect to pcie_app_7xconnect
input cfg_turnoff_ok; ( cfg_turnoff_ok ), // connect to pcie_app_7xconnect
input cfg_pm_wake; ( cfg_pm_wake ), // connect to pcie_app_7xconnect
input cfg_pm_send_pme_to; ( 1'b0 ),
input [7:0]cfg_ds_bus_number; ( 8'b0 ),
input [4:0]cfg_ds_device_number; ( 5'b0 ),
input [2:0]cfg_ds_function_number; ( 3'b0 ),
//----------------------------------------------------------------------------------------------------------------//
// Flow Control Interface //
//----------------------------------------------------------------------------------------------------------------//
output [11:0]fc_cpld; ( ),
output [7:0]fc_cplh; ( ),
output [11:0]fc_npd; ( ),
output [7:0]fc_nph; ( ),
output [11:0]fc_pd; ( ),
output [7:0]fc_ph; ( ),
input [2:0]fc_sel; ( fc_sel ),
//----------------------------------------------------------------------------------------------------------------//
// Configuration (CFG) Interface //
//----------------------------------------------------------------------------------------------------------------//
output [7:0]cfg_bus_number; ( cfg_bus_number ), // connect to pcie_app_7xconnect
output [4:0]cfg_device_number; ( cfg_device_number ), // connect to pcie_app_7xconnect
output [2:0]cfg_function_number; ( cfg_function_number ), // connect to pcie_app_7xconnect
output [15:0]cfg_status; ( cfg_status ), // connect to pcie_app_7xconnect
output [15:0]cfg_command; ( cfg_command ), // connect to pcie_app_7xconnect
output [15:0]cfg_dstatus; ( cfg_dstatus ), // connect to pcie_app_7xconnect
output [15:0]cfg_dcommand; ( cfg_dcommand ), // connect to pcie_app_7xconnect
output [15:0]cfg_dcommand2; ( cfg_dcommand2 ), // connect to pcie_app_7xconnect
output [15:0]cfg_lstatus; ( cfg_lstatus ), // connect to pcie_app_7xconnect
output [15:0]cfg_lcommand; ( cfg_lcommand ), // connect to pcie_app_7xconnect
output cfg_pmcsr_pme_status; ( ),
output cfg_to_turnoff; ( cfg_to_turnoff ), // connect to pcie_app_7xconnect
output cfg_received_func_lvl_rst; ( ),
output [2:0]cfg_pcie_link_state; ( cfg_pcie_link_state ), // connect to pcie_app_7xconnect
output cfg_pmcsr_pme_en; ( ),
output [1:0]cfg_pmcsr_powerstate; ( ),
output [5:0]tx_buf_av; ( tx_buf_av ), // connect to pcie_app_7xconnect
output tx_err_drop; ( tx_err_drop ), // connect to pcie_app_7xconnect
output tx_cfg_req; ( tx_cfg_req ), // connect to pcie_app_7xconnect
//------------------------------------------------//
// RP Only //
//------------------------------------------------//
output cfg_bridge_serr_en; ( ),
output cfg_slot_control_electromech_il_ctl_pulse; ( ),
output cfg_root_control_syserr_corr_err_en; ( ),
output cfg_root_control_syserr_non_fatal_err_en; ( ),
output cfg_root_control_syserr_fatal_err_en; ( ),
output cfg_root_control_pme_int_en; ( ),
output cfg_aer_rooterr_corr_err_reporting_en; ( ),
output cfg_aer_rooterr_non_fatal_err_reporting_en; ( ),
output cfg_aer_rooterr_fatal_err_reporting_en; ( ),
output cfg_aer_rooterr_corr_err_received; ( ),
output cfg_aer_rooterr_non_fatal_err_received; ( ),
output cfg_aer_rooterr_fatal_err_received; ( ),
//----------------------------------------------------------------------------------------------------------------//
// VC interface //
//----------------------------------------------------------------------------------------------------------------//
output [6:0]cfg_vc_tcvc_map; ( ),
// Management Interface
input [31:0]cfg_mgmt_di; ( cfg_mgmt_di ), // connect to pcie_app_7xconnect
input [3:0]cfg_mgmt_byte_en; ( cfg_mgmt_byte_en ), // connect to pcie_app_7xconnect
input [9:0]cfg_mgmt_dwaddr; ( cfg_mgmt_dwaddr ), // connect to pcie_app_7xconnect
input cfg_mgmt_wr_en; ( cfg_mgmt_wr_en ), // connect to pcie_app_7xconnect
input cfg_mgmt_rd_en; ( cfg_mgmt_rd_en ), // connect to pcie_app_7xconnect
input cfg_mgmt_wr_readonly; ( cfg_mgmt_wr_readonly ),
input cfg_mgmt_wr_rw1c_as_rw; ( 1'b0 ),
//------------------------------------------------//
// EP and RP //
//------------------------------------------------//
output [31:0]cfg_mgmt_do; ( cfg_do ), // connect to pcie_app_7xconnect
output cfg_mgmt_rd_wr_done; ( cfg_rd_wr_done), // connect to pcie_app_7xconnect
// Error Reporting Interface
input cfg_err_ecrc; ( cfg_err_ecrc ), // connect to pcie_app_7xconnect
input cfg_err_ur; ( cfg_err_ur ), // connect to pcie_app_7xconnect
input cfg_err_cpl_timeout; ( cfg_err_cpl_timeout ), // connect to pcie_app_7xconnect
input cfg_err_cpl_unexpect; ( cfg_err_cpl_unexpect ), // connect to pcie_app_7xconnect
input cfg_err_cpl_abort; ( cfg_err_cpl_abort ), // connect to pcie_app_7xconnect
input cfg_err_posted; ( cfg_err_posted ), // connect to pcie_app_7xconnect
input cfg_err_cor; ( cfg_err_cor ), // connect to pcie_app_7xconnect
input cfg_err_atomic_egress_blocked; ( 1'b0 ),
input cfg_err_internal_cor; ( 1'b0 ),
input cfg_err_malformed; ( 1'b0 ),
input cfg_err_mc_blocked; ( 1'b0 ),
input cfg_err_poisoned; ( 1'b0 ),
input cfg_err_norecovery; ( 1'b0 ),
input [47:0]cfg_err_tlp_cpl_header; ( cfg_err_tlp_cpl_header ), // connect to pcie_app_7xconnect
output cfg_err_cpl_rdy; ( cfg_err_cpl_rdy ), // connect to pcie_app_7xconnect
input cfg_err_locked; ( cfg_err_locked ), // connect to pcie_app_7xconnect
input cfg_err_acs; ( 1'b0 ),
input cfg_err_internal_uncor; ( 1'b0 ),
//----------------------------------------------------------------------------------------------------------------//
// AER Interface //
//----------------------------------------------------------------------------------------------------------------//
input [127:0]cfg_err_aer_headerlog; ( cfg_err_aer_headerlog ),
input [4:0]cfg_aer_interrupt_msgnum; ( cfg_aer_interrupt_msgnum ),
output cfg_err_aer_headerlog_set; ( ),
output cfg_aer_ecrc_check_en; ( ),
output cfg_aer_ecrc_gen_en; ( ),
//------------------------------------------------//
// EP Only //
//------------------------------------------------//
input cfg_interrupt; ( cfg_interrupt ), // connect to pcie_app_7xconnect
output cfg_interrupt_rdy; ( cfg_interrupt_rdy ), // connect to pcie_app_7xconnect
input cfg_interrupt_assert; ( cfg_interrupt_assert ), // connect to pcie_app_7xconnect
input [7:0]cfg_interrupt_di; ( cfg_interrupt_di ), // connect to pcie_app_7xconnect
output [7:0]cfg_interrupt_do; ( cfg_interrupt_do ), // connect to pcie_app_7xconnect
output [2:0]cfg_interrupt_mmenable; ( cfg_interrupt_mmenable ), // connect to pcie_app_7xconnect
output cfg_interrupt_msienable; ( cfg_interrupt_msienable ), // connect to pcie_app_7xconnect
output cfg_interrupt_msixenable; ( cfg_interrupt_msixenable ), // connect to pcie_app_7xconnect
output cfg_interrupt_msixfm; ( cfg_interrupt_msixfm ), // connect to pcie_app_7xconnect
input cfg_interrupt_stat; ( cfg_interrupt_stat ),
input [4:0]cfg_pciecap_interrupt_msgnum; ( cfg_pciecap_interrupt_msgnum ),
output cfg_msg_received_err_cor; ( ),
output cfg_msg_received_err_non_fatal; ( ),
output cfg_msg_received_err_fatal; ( ),
output cfg_msg_received_pm_as_nak; ( ),
output cfg_msg_received_pme_to_ack; ( ),
output cfg_msg_received_assert_int_a; ( ),
output cfg_msg_received_assert_int_b; ( ),
output cfg_msg_received_assert_int_c; ( ),
output cfg_msg_received_assert_int_d; ( ),
output cfg_msg_received_deassert_int_a; ( ),
output cfg_msg_received_deassert_int_b; ( ),
output cfg_msg_received_deassert_int_c; ( ),
output cfg_msg_received_deassert_int_d; ( ),
output cfg_msg_received_pm_pme; ( ),
output cfg_msg_received_setslotpowerlimit; ( ),
output cfg_msg_received; ( ),
output [15:0]cfg_msg_data; ( ),
//----------------------------------------------------------------------------------------------------------------//
// Physical Layer Control and Status (PL) Interface //
//----------------------------------------------------------------------------------------------------------------//
input [1:0]pl_directed_link_change; ( pl_directed_link_change ), // connect to pcie_app_7xconnect
input [1:0]pl_directed_link_width; ( pl_directed_link_width ), // connect to pcie_app_7xconnect
input pl_directed_link_speed; ( pl_directed_link_speed ), // connect to pcie_app_7xconnect
input pl_directed_link_auton; ( pl_directed_link_auton ), // connect to pcie_app_7xconnect
input pl_upstream_prefer_deemph; ( pl_upstream_prefer_deemph ), // connect to pcie_app_7xconnect
output pl_sel_lnk_rate; ( pl_sel_link_rate ), // connect to pcie_app_7xconnect
output [1:0]pl_sel_lnk_width; ( pl_sel_link_width ), // connect to pcie_app_7xconnect
output [5:0]pl_ltssm_state; ( pl_ltssm_state ), // connect to pcie_app_7xconnect
output [1:0]pl_lane_reversal_mode; ( pl_lane_reversal_mode ), // connect to pcie_app_7xconnect
output pl_phy_lnk_up; ( ),
output [2:0]pl_tx_pm_state; ( ),
output [1:0]pl_rx_pm_state; ( ),
output pl_link_upcfg_cap; ( pl_link_upcfg_capable ), // connect to pcie_app_7xconnect
output pl_link_gen2_cap; ( pl_link_gen2_capable ), // connect to pcie_app_7xconnect
output pl_link_partner_gen2_supported; ( pl_link_partner_gen2_supported ), // connect to pcie_app_7xconnect
output [2:0]pl_initial_link_width; ( pl_initial_link_width ), // connect to pcie_app_7xconnect
output pl_directed_change_done; ( ),
//------------------------------------------------//
// EP Only //
//------------------------------------------------//
output pl_received_hot_rst; ( pl_received_hot_rst ), // connect to pcie_app_7xconnect
//------------------------------------------------//
// RP Only //
//------------------------------------------------//
input pl_transmit_hot_rst; ( 1'b0 ),
input pl_downstream_deemph_source; ( 1'b0 ),
//----------------------------------------------------------------------------------------------------------------//
// System (SYS) Interface //
//----------------------------------------------------------------------------------------------------------------//
input sys_clk; ( sys_clk ),
input sys_rst_n; ( sys_rst_n_c )
FPGA再入门-1-480T板卡试用
YPCB-00338-1P1 FPGA 板卡 逆向 – TiferKing的学习笔记
PCIe应用实战
一步一步开始FPGA逻辑设计 - 高速接口之PCIe
1. 如何写Linux PCI驱动
怎么编写linux PCI驱动
一、如何编写Linux PCI驱动程序
GPU到底是如何工作的?这篇AI Infra入门全部告诉你
了解 CUDA GPU 计算的汇编语言 PTX
从PTX到SASS:编译器视角