hypervisor display显卡节点card0生成过程

原创已于 2023-11-22 14:53:06 修改 · 5.5k 阅读

28 ·

CC 4.0 BY-SA版权

文章标签：

#linux

于 2023-06-21 13:06:50 首次发布

高通hypervisor display 专栏收录该内容

2 篇文章

订阅专栏

文章主要探讨了在QNXhypervisor环境下，基于高通基线1.2.1和内核版本5.4的Android系统中，dev/dri/card0和dev/dri/renderD128两个设备节点的生成过程。通过分析/sys/class/drm/card0目录下的信息，确定了驱动的关键字qcom,sde_kms_hyp，并在$KERNEL_ROOT/techpack/display/目录下找到相关源码。进一步，文章指出驱动加载时msm-cfg模块先加载，然后加载msm-hyp-legacy模块，这两个模块一起创建了所需的drm设备。虽然驱动仅实现了有限的文件操作，但上层应用仍会频繁使用相关接口。最后，文章提到了驱动加载顺序、dts配置和节点使用者的情况。

开始

我们知道android显示模块底层是由drm（Direct Rending Manger）提供的，drm驱动加载完成的标志是显卡设备节点生成:

130|msmnile_gvmq:/sys/class/drm/card0/device/driver # ls /dev/dri/
card0  renderD128
msmnile_gvmq:/sys/class/drm/card0/device/driver #

这两个节点是：

/dev/dri/card0

/dev/dri/renderD128

现在我们想探究一下，在QNX hypervisor下（高通基线1.2.1，内核版本5.4），android侧的这两个文件节点是如何生成的.

定位hyp display源码路径

这两个节点是有驱动模块生成的，所以要了解生成的细节，就必须要分析其驱动源码。为了达成我们这篇文章的目的：hypervisor display显卡节点card0生成过程,我们需要下面几个工作：

收集信息-》定位驱动源码模块-》分析代码-》信息汇总总结

但是目前摆在我们眼前的第一个问题就是需要搞清楚这两个节点是由哪些驱动源码生成的。.

从/dev/和/sys开始

一般/dev/下的文件节点的名字都具有特殊性，通常由驱动模块源码或者内核中相应的子系统模块分配指定。这里我们尝试直接搜索card/render相关的字符串，但是搜索结果太多，并不能很快的定位源码位置。

这里我们可以从/sys目录入手，进入/sys/class/drm/card0目录，这里可以获取到部分信息：

msmnile_gvmq:/sys/class/drm/card0/device # ls
driver  driver_override  drm  modalias  of_node  power  subsystem  uevent
msmnile_gvmq:/sys/class/drm/card0/device # ls
driver  driver_override  drm  modalias  of_node  power  subsystem  uevent
msmnile_gvmq:/sys/class/drm/card0/device # cat of_node/name
qcom,sde_kms_hypmsmnile_gvmq:/sys/class/drm/card0/device # cat dri
driver/           driver_override
msmnile_gvmq:/sys/class/drm/card0/device # cat driver/
bind                                                           soc:qcom,sde-cfg:qcom,sde-sub-cfg@1:qcom,sde_kms_hyp@ae00000/  uevent                                                         unbind
msmnile_gvmq:/sys/class/drm/card0/device # cat driver/

从获取到的信息中，我们知道驱动在适配dts时候的关键字：qcom,sde_kms_hyp，以及部分dts配置信息：soc:qcom,sde-cfg:qcom,sde-sub-cfg@1:qcom,sde_kms_hyp@ae00000

定位驱动源码路径

现在driver目录下搜索：qcom,sde-kms-hyp，但是好像并没有搜索到结果。

紧接着在内核根目录下搜索：qcom,sde-kms-hyp，发现其源码位于：$KERNEL_ROOT/techpack/display/目录下.

需要注意的是，techpack/display目录下的编译配置并不由kernel config文件控制（arch/arm64/config目录下的板级配置文件），而是自己有单独的配置文件，打开文件：techpack/display/Makefile 我们可以看到：

techpack/display目录的源码其实是包含了LA/LV基线以及hypervisor display基线的源码的，这里我们的基线是HYP display基线，所以具体的内核配置记录在文件techpack/display/config/gvmgen3disp.conf 中。

这些配置在编译完成之后不会体现在.config中，也不会体现在/proc/config.gz中.

所以我们需要分析的代码大致有几个目录：

小结

1，我们从/sys目录下的node信息得到在dts中与display相关的关键字

2，通过搜索这些关键字定位驱动目录位于$KERNEL_ROOT/techpack/display/下

3，分析代码发现，display目录下代码包含了单LA/LV基线及Q+A基线的显示驱动代码

4，确认需要我们分析的目录为：

msm-cfg msm-hyp msm-hyp-legacy msm-lease

代码分析

忽略代码分析的过程，这里直接上结论：

1，前面提到的需要分析的代码中，其实只有红框部分代码有用（msm-cfg&msm-hyp-legacy），绿框中的代码在内核驱动加载的时候只会运行register，而根本不会执行probe函数。

2，绿框中的代码完全不编译也不影响系统运行，这些代码感觉像是历史遗留代码

3，驱动加载时，msm-cfg模块会先加载，然后带起msm-hyp-legacy模块，本质上他们是一个模块，目的是创建android系统需要的drm设备card0 和renderD128，驱动实现中，这两个节点只提供非常有限的文件操作功能，比如open/close，ioctl则几乎不实现。这点在目录下NOTES也说的很明白：

NOTES about msm drm/kms hyp driver:

This driver registers with drm framework for the purpose of creating the 
/dev/dri/card0 path, which User Space DRM masters rely on. 
Furthermore, per-CRTC VBLANK and PAGE_FLIP events are queued to the device
path to notify User Space components listeners.

No other IOCTL or HW support is provided through this driver.

但是，不代表上层不会使用这些接口，相反上层依旧会频繁的使用这些接口

[     0.968418],4,527,-;for debug: now we are msm_drm_register techpack/display/msm-hyp/msm_drv_hyp.c +2212
[     7.729034],4,808,-;for debug: now we are msm_open techpack/display/msm-hyp-legacy/msm_drv_hyp.c +56, proess is:android.hardwar pid 426
[     7.731890],4,810,-;for debug: now we are msm_drm_ioctl techpack/display/msm-hyp-legacy/msm_drv_hyp.c +347, proess is:android.hardwar pid 426,
[     7.732684],4,811,-;for debug: now we are msm_ioctl_query_client_id techpack/display/msm-hyp-legacy/msm_drv_hyp.c +333 
[     7.733158],4,812,-;for debug: now we are _msm_parse_dt techpack/display/msm-hyp-legacy/msm_drv_hyp.c +304 
[     7.764510],6,823,-;for debug: now we are msm_open techpack/display/msm-hyp-legacy/msm_drv_hyp.c +56, proess is:android.hardwar pid 426
[     7.764528],6,824,-;for debug: now we are msm_open techpack/display/msm-hyp-legacy/msm_drv_hyp.c +56, proess is:android.hardwar pid 426
[     7.764542],6,825,-;for debug: now we are msm_open techpack/display/msm-hyp-legacy/msm_drv_hyp.c +56, proess is:android.hardwar pid 426
[     8.054259],6,888,-;for debug: now we are msm_open techpack/display/msm-hyp-legacy/msm_drv_hyp.c +56, proess is:composer@2.4-se pid 426
[     8.054283],6,889,-;for debug: now we are msm_open techpack/display/msm-hyp-legacy/msm_drv_hyp.c +56, proess is:composer@2.4-se pid 426
[     8.054292],6,890,-;for debug: now we are msm_open techpack/display/msm-hyp-legacy/msm_drv_hyp.c +56, proess is:composer@2.4-se pid 426
[     8.054306],6,891,-;for debug: now we are msm_open techpack/display/msm-hyp-legacy/msm_drv_hyp.c +56, proess is:composer@2.4-se pid 426
[    12.488209],6,1234,-;for debug: now we are msm_drm_ioctl techpack/display/msm-hyp-legacy/msm_drv_hyp.c +347, proess is:composer@2.4-se pid 426
[    12.488214],6,1235,-;for debug: now we are msm_ioctl_gem_get techpack/display/msm-hyp-legacy/msm_drv_hyp.c +205, proess is:composer@2.4-se pid
[    12.525305],6,1248,-;for debug: now we are msm_drm_ioctl techpack/display/msm-hyp-legacy/msm_drv_hyp.c +347, proess is:composer@2.4-se pid 426
[    12.525311],6,1249,-;for debug: now we are msm_ioctl_gem_get techpack/display/msm-hyp-legacy/msm_drv_hyp.c +205, proess is:composer@2.4-se pid
[    12.542423],6,1259,-;for debug: now we are msm_drm_write techpack/display/msm-hyp-legacy/msm_drv_hyp.c +160, proess is:DRMFE_EventList pid 516
[    12.542427],6,1260,-;for debug: now we are create_vblank_event techpack/display/msm-hyp-legacy/msm_drv_hyp.c +99, proess is:DRMFE_EventList pi
[    12.775925],6,1268,-;for debug: now we are msm_drm_write techpack/display/msm-hyp-legacy/msm_drv_hyp.c +160, proess is:DRMFE_EventList pid 516
[    12.775929],6,1269,-;for debug: now we are create_vblank_event techpack/display/msm-hyp-legacy/msm_drv_hyp.c +99, proess is:DRMFE_EventList pi
[    12.792664],6,1270,-;for debug: now we are msm_drm_write techpack/display/msm-hyp-legacy/msm_drv_hyp.c +160, proess is:DRMFE_EventList pid 516
[    12.792668],6,1271,-;for debug: now we are create_vblank_event techpack/display/msm-hyp-legacy/msm_drv_hyp.c +99, proess is:DRMFE_EventList pi
[    12.809313],6,1272,-;for debug: now we are msm_drm_write techpack/display/msm-hyp-legacy/msm_drv_hyp.c +160, proess is:DRMFE_EventList pid 516
[    12.809316],6,1273,-;for debug: now we are create_vblank_event techpack/display/msm-hyp-legacy/msm_drv_hyp.c +99, proess is:DRMFE_EventList pi
[    12.825947],6,1274,-;for debug: now we are msm_drm_write techpack/display/msm-hyp-legacy/msm_drv_hyp.c +160, proess is:DRMFE_EventList pid 516
[    12.825951],6,1275,-;for debug: now we are create_vblank_event techpack/display/msm-hyp-legacy/msm_drv_hyp.c +99, proess is:DRMFE_EventList pi
[    12.842704],6,1278,-;for debug: now we are msm_drm_write techpack/display/msm-hyp-legacy/msm_drv_hyp.c +160, proess is:DRMFE_EventList pid 516
[    12.842708],6,1279,-;for debug: now we are create_vblank_event techpack/display/msm-hyp-legacy/msm_drv_hyp.c +99, proess is:DRMFE_EventList pi
[    12.859308],6,1284,-;for debug: now we are msm_drm_write techpack/display/msm-hyp-legacy/msm_drv_hyp.c +160, proess is:DRMFE_EventList pid 516

4, msm-hyp-legacy模块中生成card0和renderD128的过程也很简单，在msm-hyp-legacy模块中的msm_pdev_probe函数中调用drm_dev_alloc来申请drm设备，然后调用drm_dev_register注册drm设备，进而生成card0&render128节点

static int msm_pdev_probe(struct platform_device *pdev)
{
    struct device *dev = &pdev->dev;
    struct drm_device *ddev;
    struct msm_drm_private *priv;
    int ret;

    ddev = drm_dev_alloc(&msm_driver, dev);
    if (!ddev) {
        dev_err(dev, "failed to allocate drm_device\n");
        return -ENOMEM;
    }   

    platform_set_drvdata(pdev, ddev);

    priv = kzalloc(sizeof(*priv), GFP_KERNEL);
    if (!priv) {
        ret = -ENOMEM;
        goto priv_alloc_fail;
    }   

    ddev->dev_private = priv;
    priv->dev = ddev;

    ret = drm_dev_register(ddev, 0); 
    if (ret) {
        dev_err(dev, "failed to register drm device\n");
        goto fail;
    }   

    return 0;

fail:

priv_alloc_fail:
    drm_dev_put(ddev);
    return ret;
}

5，在drm_dev_register函数中会生成2个节点，即render128和card0，通过调用drm_minor_register时传递DRM_MINOR_RENDER和DRM_MINOR_PRIMARY两种参数来生成不同的节点.其中DRM_MINOR_RENDER负责生成render128，而DRM_MINOR_PRIMARY则是生成card0节点.

int drm_dev_register(struct drm_device *dev, unsigned long flags)
{
    struct drm_driver *driver = dev->driver;
    int ret;

    mutex_lock(&drm_global_mutex);

    ret = drm_minor_register(dev, DRM_MINOR_RENDER);
    if (ret)
        goto err_minors;

    ret = drm_minor_register(dev, DRM_MINOR_PRIMARY);
    if (ret)
        goto err_minors;

    ret = create_compat_control_link(dev);
    if (ret)
        goto err_minors;

    dev->registered = true;

    if (dev->driver->load) {
        ret = dev->driver->load(dev, flags);
        if (ret)
            goto err_minors;
    }

    if (drm_core_check_feature(dev, DRIVER_MODESET))
        drm_modeset_register_all(dev);

    ret = 0;

    DRM_INFO("Initialized %s %d.%d.%d %s for %s on minor %d\n",
         driver->name, driver->major, driver->minor,
         driver->patchlevel, driver->date,
         dev->dev ? dev_name(dev->dev) : "virtual device",
         dev->primary->index);

    goto out_unlock;

err_minors:
    remove_compat_control_link(dev);
    drm_minor_unregister(dev, DRM_MINOR_PRIMARY);
    drm_minor_unregister(dev, DRM_MINOR_RENDER);
out_unlock:
    mutex_unlock(&drm_global_mutex);
    return ret;
}

6，如果我们对LA的drm驱动做完整的drm测试，他能通过吗？

从分析来看，Q+A的drm驱动只是一个空的架子，没有实质的内容，所以无法进行测试.

7，对于同一个level的驱动，内核如何保证其加载顺序？比如现在我们有两个同是__initcall level的驱动msm_drm_register

内核控制驱动加载顺序的方法有很多种，其中一种就是通过Makefile中的先后顺序进行控制，比如当前我们分析的这个模块：

msm-cfg的位置要先于msm-hyp,从system.map看msm-cfg模块会优先于msm-hyp模块加载.

8，DTS如何与驱动产生联动的？

display相关的dts配置在文件：arch/arm64/boot/dts/vendor/qcom/display/quin-vm-display.dtsi，内容也不多：

&soc {
    sde_cfg: qcom,sde-cfg {
        compatible = "qcom,sde-cfg";

        qcom,sde-sub-cfg@0 {
            reg = <0>;
            wfd_kms: qcom,wfd_kms@0 {
                compatible = "qcom,wfd-kms";
                qcom,client-id = "7815";
            };

            qcom,sde_kms_hyp@ae00000 {
                compatible = "qcom,sde-kms-hyp";
                qcom,kms = <&wfd_kms>;
            };
        };

        qcom,sde-sub-cfg@1 {
            reg = <1>;
            qcom,sde_kms_hyp@ae00000 {
                compatible = "qcom,sde-kms-hyp-legacy";
                qcom,client-id = "7815";
            };
        };
    };  
};

这个文件会被include到文件arch/arm64/boot/dts/vendor/qcom/sa8155-vm-la.dtsi 中。驱动加载的时候，会先通过qcom,sde-cfg匹配到msm-cfg驱动模块，在由这个模块带起剩下的legacy和lease模块.

9,都有谁会使用这个节点：

msmnile_gvmq:/ # lsof | grep renderD128
1|msmnile_gvmq:/ #
1|msmnile_gvmq:/ #
1|msmnile_gvmq:/ #
1|msmnile_gvmq:/ # lsof | grep card0
composer@2.4-se   406     system    7u      CHR              226,0       0t0      16645 /dev/dri/card0
composer@2.4-se   406     system   15u      CHR              226,0       0t0      16645 /dev/dri/card0
composer@2.4-se   406     system   16u      CHR              226,0       0t0      16645 /dev/dri/card0
composer@2.4-se   406     system   17u      CHR              226,0       0t0      16645 /dev/dri/card0
composer@2.4-se   406     system   19u      CHR              226,0       0t0      16645 /dev/dri/card0
composer@2.4-se   406     system   21u      CHR              226,0       0t0      16645 /dev/dri/card0
composer@2.4-se   406     system   22u      CHR              226,0       0t0      16645 /dev/dri/card0
composer@2.4-se   406     system   23u      CHR              226,0       0t0      16645 /dev/dri/card0
msmnile_gvmq:/ #
msmnile_gvmq:/ #
msmnile_gvmq:/ # lsof | grep hgsl
hgsl-release-wq   169       root  cwd       DIR              252,6      4096          2 /
hgsl-release-wq   169       root  rtd       DIR              252,6      4096          2 /
hgsl-release-wq   169       root  txt   unknown                                         /proc/169/exe (readlink: No such file or directory)
provider@2.4-se   399 cameraserv    7u      CHR              241,0       0t0       4542 /dev/hgsl
provider@2.4-se   399 cameraserv    8u      CHR              241,0       0t0       4542 /dev/hgsl
composer@2.4-se   406     system   10u      CHR              241,0       0t0       4542 /dev/hgsl
composer@2.4-se   406     system   11u      CHR              241,0       0t0       4542 /dev/hgsl
composer@2.4-se   406     system   12u      CHR              241,0       0t0       4542 /dev/hgsl
composer@2.4-se   406     system   13u      CHR              241,0       0t0       4542 /dev/hgsl
cameraprovider@   420       root    4u      CHR              241,0       0t0       4542 /dev/hgsl
cameraprovider@   420       root    7u      CHR              241,0       0t0       4542 /dev/hgsl
surfaceflinger   472     system   14u      CHR              241,0       0t0       4542 /dev/hgsl
surfaceflinger   472     system   15u      CHR              241,0       0t0       4542 /dev/hgsl
surfaceflinger   472     system   16u      CHR              241,0       0t0       4542 /dev/hgsl
surfaceflinger   472     system   17u      CHR              241,0       0t0       4542 /dev/hgsl
Binder:1095_3  1095     system  320u      CHR              241,0       0t0       4542 /dev/hgsl
Binder:1095_3  1095     system  330u      CHR              241,0       0t0       4542 /dev/hgsl
Binder:1095_3  1095     system  334u      CHR              241,0       0t0       4542 /dev/hgsl
ndroid.systemui  2167     system   71u      CHR              241,0       0t0       4542 /dev/hgsl
ndroid.systemui  2167     system   72u      CHR              241,0       0t0       4542 /dev/hgsl
ndroid.systemui  2167     system   74u      CHR              241,0       0t0       4542 /dev/hgsl

10, Display模块中被应用层调用的ioctl操作是如何传递到QNX侧的？

事实上，Display模块中的ioctl最终会以signal的方式通知hgsl模块，由hgsl模块通过hab通知到QNX