Interrupt Binding

Increasing performance by doing a binding interrupts ?

This is a new concept for me, so i did a quick google and found something usefull.

 

 

---

Interrupt Affinity
Interrupt affinity means binding of interrupts from a specific device to
specific processor(s) in a multiprocessor server. This enforces running the
ISR and DPC routines on the said processor(s). Because network connections
and file server sessions all stay on the same network adapter, binding
interrupts from the network adapter to a processor allows for processing of
incoming packets (server message block (SMB) requests, data) on a specific
set of processors, improving locality and scalability . You cannot configure
affinity on single-processor computers.
The Interrupt-Affinity Filter (IntFiltr) tool allows you to change the
CPU-affinity of the interrupts in a system.
Using this utility, you can direct the interrupts of any device to a
specific processor or set of processors (as opposed to always sending
interrupts to any CPU in the system). Note that different devices can have
different interrupt-affinity settings.

 

 

Interrupt and Process Binding

 

Realtime environments need to minimize or eliminate latency when responding to various events. Ideally, interrupts (IRQs) and user processes can be isolated from one another on different dedicated CPUs.

Interrupts are generally shared evenly between CPUs. This can delay interrupt processing through having to write new data and instruction caches, and often creates conflicts with other processing occurring on the CPU. In order to overcome this problem, time-critical interrupts and processes can be dedicated to a CPU (or a range of CPUs). In this way, the code and data structures needed to process this interrupt will have the highest possible likelihood to be in the processor data and instruction caches. The dedicated process can then run as quickly as possible, while all other non-time-critical processes run on the remainder of the CPUs. This can be particularly important in cases where the speeds involved are in the limits of memory and peripheral bus bandwidth available. Here, any wait for memory to be fetched into processor caches will have a noticeable impact in overall processing time and determinism.

In practice we have found that optimal performance is entirely application specific. For example, in tuning applications for different companies which perform similar functions, the optimal performance tunings were completely different. For one firm, isolating 2 out of 4 CPUs for operating system functions and interrupt handling and dedicating the remaining 2 CPUs purely for application handling was optimal. For another firm, binding the network related application processes onto a CPU which was handling the network device driver interrupt yielded optimal determinism . Ultimately, tuning is often accomplished by trying a variety of settings to discover what works best for your organization.

 

Important

For many of the processes described here, you will need to know the CPU mask for a given CPU or range of CPUs. The CPU mask is typically represented as a 32-bit bitmask (on 32-bit machines), but can also be expressed as a decimal or hexadecimal number. For example: The CPU mask for CPU 0 only is 00000000000000000000000000000001 as a bitmask, 1 as a decimal, and 0x00000001 as a hexadecimal. The CPU mask for both CPU 0 and 1 is 00000000000000000000000000000011 as a bitmask, 3 as a decimal, and 0x00000003 as a hexadecimal.

 

 

Disabling the irqbalance daemon

This daemon is enabled by default and periodically forces interrupts to be handled by CPUs in an even, fair manner. However in realtime deployments, applications are typically dedicated and bound to specific CPUs, so the irqbalance daemon is not required.

1. Check the status of the irqbalance daemon

   # service irqbalance status
   
   irqbalance (pid PID
   
   ) is running...
   
   

2. If the irqbalance daemon is running, stop it using the service command.

   # service irqbalance stop
   
   Stopping irqbalance:             [  OK  ]
   
   

3. Use chkconfig to ensure that irqbalance does not restart on boot.

   # chkconfig irqbalance off

Partially Disabling the irqbalance daemon

An alternative approach to is to disable irqbalance only on those CPUs that have dedicated functions, and enable it on all other CPUs. This can be done by editing the /etc/sysconfig/irqbalance file.

1. Open /etc/sysconfig/irqbalance in your preferred text editor and find the section of the file titled “FOLLOW_ISOLCPUS ”.

   ...[output truncated]...
   
   # FOLLOW_ISOLCPUS
   
   #       Boolean value.  When set to yes, any setting of IRQ_AFFINITY_MASK above
   
   #       is overridden, and instead computed to be the same mask that is defined
   
   #       by the isolcpu kernel command line option.
   
   #
   
   #FOLLOW_ISOLCPUS=no
   
   

2. Enable FOLLOW_ISOLCPUS by removing the # character from the beginning of the line and changing the value to yes .

   ...[output truncated]...
   
   # FOLLOW_ISOLCPUS
   
   #       Boolean value.  When set to yes, any setting of IRQ_AFFINITY_MASK above
   
   #       is overridden, and instead computed to be the same mask that is defined
   
   #       by the isolcpu kernel command line option.
   
   #
   
   FOLLOW_ISOLCPUS=yes
   
   

3. This will make irqbalance operate only on the CPUs not specifically isolated. This is most effective for machines with more than two processors, but works just as well on a dual-core machine.

Manually Assigning CPU Affinity to Individual IRQs

1. You can see which IRQ your devices are on by viewing the /proc/interrupts file.

   # cat /proc/interrupts
   
   

   This file contains a list of IRQs. Each line shows the IRQ number, the number of interrupts that happened in each CPU, followed by the IRQ type and a description.

   CPU0             CPU1
   
   0:   26575949         11         IO-APIC-edge  timer
   
   1:         14          7         IO-APIC-edge  i8042
   
   ...[output truncated]...
   
   

2. To instruct an IRQ to run on only one processor , echo the CPU mask (as a decimal number) to /proc/interrupts . In this example, we are instructing the interrupt with IRQ number 142 to run on CPU 0 only.

   # echo 1 > /proc/irq/142/smp_affinity
   
   

3. This change will only take effect once an interrupt has occurred. To test the settings, generate some disk activity, then check the /proc/interrupts file for changes. Assuming that you have caused an interrupt to occur, you should see that the number of interrupts on the chosen CPU have risen, while the numbers on the other CPUs have not changed.

Binding Processes to CPUs using the taskset utility

The taskset utility uses the process ID (PID) of a task to view or set the affinity, or can be used to launch a command with a chosen CPU affinity. In order to set the affinity, taskset requires the CPU mask expressed as either a decimal or hexadecimal number.

1. To set the affinity of a process that is not currently running, use taskset and specify the CPU mask and the process. In this example, my_embedded_process is being instructed to use only CPU 4 (using the decimal version of the CPU mask).

   # taskset 8 /usr/local/bin/my_embedded_process
   
   

2. It is also possible to set the CPU affinity for processes that are already running by using the -p (--pid ) option with the CPU mask and the PID of the process you wish to change. In this example, the process with a PID of 7013 is being instructed to run only on CPU 0.

   # taskset -p 1 7013
   
   

 

 

Here some code snippet to set cpu affinity:

FILE *fp = fopen("/proc/interrupts", "r"); if (!fp) { err("Cannot open /proc/interrupts"); return -ENOENT; } memset(buff, 0, sizeof(buff)); while (fgets(buff, sizeof(buff), fp) != NULL) { buff[sizeof(buff) - 1] = 0; if(strstr(buff, dev) == NULL) continue; buffp = buff; while(*buffp != ':') { buffp++; } *buffp = 0; ret = atoi(buff); } fclose(fp); return ret;

 

sprintf(file, "/proc/irq/%d/smp_affinity", intr); FILE *fp = fopen(file, "w"); if (!fp) { err("Cannot open %s", file); return -ENOENT; } sprintf(buff, "%d", cpu); ret = fwrite(buff, sizeof(buff), 1, fp); fclose(fp);

 

 

BTW, http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Realtime_Tuning_Guide/index.html

offer detailed tuning realtime linux os knowledge.