Posted-Interrupt

最新推荐文章于 2023-01-06 17:02:33 发布

hx_op

最新推荐文章于 2023-01-06 17:02:33 发布

阅读量3k

点赞数

分类专栏：虚拟化 ARCH

本文链接：https://blog.youkuaiyun.com/hx_op/article/details/104029844

版权

ARCH 同时被 2 个专栏收录

11 篇文章

订阅专栏

虚拟化

7 篇文章

订阅专栏

文章目录

1 virtual APIC

这里只讨论利用硬件实现virtual APIC的情况，在硬件实现过程中，会分配一个物理页虚拟APIC。对与APIC的一些寄存器访问可以配置是否产生vm exit，详见《64-ia-32-architectures-software-developer-manual-325462.pdf 》章节29.2.2-29.2.3。硬件对于虚拟化的支持一方面体现在减少软件复杂度，另一方面就是提高效率(比如减少vm exit次数)。

2 Posted-Interrupt原理

这里说的posted-interrupt指的是CPU对于这种方式的硬件支持，在文档《vt-directed-io-spec.pdf》中，会有Interrupt Posting，这种方式首先需要CPU支持posted-interrupt，还需要RC上支持Interrupt Remapping。对于一个透传给虚机的设备，如果RC上支持Interrupt Remapping，CPU支持posted-interrupt，那样的话在处理中断时VMM可以不借助。目前使用的cloudkernel没有开启对RC上Interrupt Remapping的支持。

在正式介绍Posted-Interrupt的实现前，先介绍下这个功能的意义。对于一个中断，肯定是响应延迟越低越好，最理想的状态就是一旦出发中断，虚机可以立马响应。根据发生中断时虚机所处的状态，分下面两种情况：

虚机处于运行状态。当虚机正在运行时，来个一个外部中断，那么理想的方式就是不用vm exit就可以响应中断，而Posted-Interrupt就是干这个的，针对这种情况，后面会具体阐述。如果没有Posted-Interrupt，那么对于一个正在运行的虚机，通过IPI使其vm exit->中断注入->vm entry响应中断

虚机不在运行。对于一个这样的虚机，只需要中断注入->vm entry响应中断，所以vm entry越早(即尽快使vCPU投入运行)，中断响应越实时。Interrupt Remapping时通过配置urgent选项就是为了提高响应速度

还有一点需要注意，当vm entry时可以响应中断，还有哪些情况可以响应呢？虚机对中断的响应，一方面需要VMM去注入，另一方面对于中断也是有响应时机的。参考手册《64-ia-32-architectures-software-developer-manual-325462.pdf 》章节29.2.1，有如下内容：

The following actions cause the evaluation of pending virtual interrupts: VM entry; TPR virtualization; EOI virtual-ization; self-IPI virtualization; and posted-interrupt processing.

这里面就提到了posted-interrupt，也就是说通过posted-interrupt就可以使虚机去检测并响应中断。

对于Posted-Interrupt的支持，手册介绍如下

If the “external-interrupt exiting” VM-execution control is 1, any unmasked external interrupt causes a VM exit (see Section 25.2). If the “process posted interrupts” VM-execution control is also 1, this behavior is changed and the processor handles an external interrupt as follows:
一般情况下，当vCPU在运行，然后触发了外部中断，虚机会vm exit。如果开启了对posted interrupts的支持，这个行为会有所改变

1. The local APIC is acknowledged; this provides the processor core with an interrupt vector, called here the physical vector.
2. If the physical vector equals the posted-interrupt notification vector, the logical processor continues to the next step. Otherwise, a VM exit occurs as it would normally due to an external interrupt; the vector is saved in the VM-exit interruption-information field.
这里面提到了posted-interrupt notification vector，那么当前正在运行的虚机怎么知道这个号是多少呢？实际上这个值是装在了VMCS中，这样的话虚机在运行时就心里有数了

3. The processor clears the outstanding-notification bit in the posted-interrupt descriptor. This is done atomically so as to leave the remainder of the descriptor unmodified (e.g., with a locked AND operation).
这里我有个疑问，上面提到了虚机不需要vm exit，继续运行，那样的话，物理CPU还怎么执行上述操作啊，比如清楚bit位，还有下面的写EOI寄存器操作。在entry_64.S中，也确实定义了POSTED_INTR_VECTOR中断向量
 `apicinterrupt3 POSTED_INTR_VECTOR \
	kvm_posted_intr_ipi smp_kvm_posted_intr_ipi`
而且`smp_kvm_posted_intr_ipi`处理函数干的事就是这里描述的，所以此时硬件到底咋实现的还不清楚，VCPU不用vm exit，然后物理CPU还执行了程序。通过查看smp_kvm_posted_intr_ipi函数commit信息，整个posted-interrup过程对vmm是透明的，smp_kvm_posted_intr_ipi函数就是当vcpu挂起的时候用的，通过这些信息说明3、4两个操作是硬件自己做的。

4. The processor writes zero to the EOI register in the local APIC; this dismisses the interrupt with the posted- interrupt notification vector from the local APIC.

5. The logical processor performs a logical-OR of PIR into VIRR and clears PIR. No other agent can read or write a PIR bit (or group of bits) between the time it is read (to determine what to OR into VIRR) and when it is cleared.
这块操作都是硬件自动的操作，主要是为了将PIR的状态转到virtual APIC中去

6. The logical processor sets RVI to be the maximum of the old value of RVI and the highest index of all bits that were set in PIR; if no bit was set in PIR, RVI is left unmodified.
这里是为了屏蔽其它中断

7. The logical processor evaluates pending virtual interrupts as described in Section 29.2.1.
然后就去执行中断服务了

3 Posted-Interrupt实现

arch/x86/kvm/vmx.c中的vmx_x86_ops，.deliver_posted_interrupt = vmx_deliver_posted_interrupt，这里就是发送Posted-Interrupt的，具体实现如下：

/*
 * Send interrupt to vcpu via posted interrupt way.
 * 1. If target vcpu is running(non-root mode), send posted interrupt
 * notification to vcpu and hardware will sync PIR to vIRR atomically.
 * 2. If target vcpu isn't running(root mode), kick it to pick up the
 * interrupt from PIR in next vmentry.
 */
static void vmx_deliver_posted_interrupt(struct kvm_vcpu *vcpu, int vector)
{
	struct vcpu_vmx *vmx = to_vmx(vcpu);
	int r;

	r = vmx_deliver_nested_posted_interrupt(vcpu, vector);
	if (!r)
		return;

	if (pi_test_and_set_pir(vector, &vmx->pi_desc))  /* 手动置中断状态位 */
		return;

	r = pi_test_and_set_on(&vmx->pi_desc);
	kvm_make_request(KVM_REQ_EVENT, vcpu);
	if (r || !kvm_vcpu_trigger_posted_interrupt(vcpu))
		kvm_vcpu_kick(vcpu);
}

static inline bool kvm_vcpu_trigger_posted_interrupt(struct kvm_vcpu *vcpu)
{
#ifdef CONFIG_SMP
	if (vcpu->mode == IN_GUEST_MODE) {
		struct vcpu_vmx *vmx = to_vmx(vcpu);

		/*
		 * Currently, we don't support urgent interrupt,
		 * all interrupts are recognized as non-urgent
		 * interrupt, so we cannot post interrupts when
		 * 'SN' is set.
		 *
		 * If the vcpu is in guest mode, it means it is
		 * running instead of being scheduled out and
		 * waiting in the run queue, and that's the only
		 * case when 'SN' is set currently, warning if
		 * 'SN' is set.
		 */
		WARN_ON_ONCE(pi_test_sn(&vmx->pi_desc));

		apic->send_IPI_mask(get_cpu_mask(vcpu->cpu),    /* 发送IPI，向量号为POSTED_INTR_VECTOR  */
				POSTED_INTR_VECTOR);
		return true;
	}
#endif
	return false;
}

问题记录

当vCPU正在运行时，有posted-interrupt notification vector产生，此时虚机不需要vm exit，那么对于3、4两个操作是硬件自己做的，还是通过物理机上的posted-interrupt notification vector服务程序做的。通过查看smp_kvm_posted_intr_ipi函数commit信息，整个posted-interrup过程对vmm是透明的，smp_kvm_posted_intr_ipi函数就是当vcpu挂起的时候用的，通过这些信息说明3、4两个操作是硬件自己做的。commit信息如下：

commit d78f2664832f8d70e36422af9a10e44276dced48
Author: Yang Zhang <yang.z.zhang@Intel.com>
Date:   Thu Apr 11 19:25:11 2013 +0800

    KVM: VMX: Register a new IPI for posted interrupt

    Posted Interrupt feature requires a special IPI to deliver posted interrupt
    to guest. And it should has a high priority so the interrupt will not be
    blocked by others.
    Normally, the posted interrupt will be consumed by vcpu if target vcpu is
    running and transparent to OS. But in some cases, the interrupt will arrive
    when target vcpu is scheduled out. And host will see it. So we need to
    register a dump handler to handle it.

    Signed-off-by: Yang Zhang <yang.z.zhang@Intel.com>
    Acked-by: Ingo Molnar <mingo@kernel.org>
    Reviewed-by: Gleb Natapov <gleb@redhat.com>
    Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>