multi-process & cpu with multi-cores

本文探讨了多进程环境下,特别是多核心CPU系统中进程间对于共享资源的竞争问题。通过对PHP-FPM实例的分析,揭示了大量进程争夺资源导致系统性能下降的原因,并深入讨论了与之相关的CPU调度机制。

多进程任务,由于CPU core数的增加,会大大的增加其对于cache、总线等共享资源的争夺,从而造成machines with CPUs consisting of multiple cores的表现不如预期。

比如类似PHP-FPM这样启动200个进程(PHP-CGI)的程序,出现了大量争夺共享资源而造成低性能的表现。


系统状态的表现是,压力一旦达到一定程度,CPU sys态会飙高,idle会降为0。


spin_lock占用大部分CPU时间:



__any_online_cpu占用:



strace php-cgi,大量lstate系统调用,根源是PHP函数require_once调用lstate:



关于CPU sys高,表明操作系统将大部分时间花在了kernel mode上。lstate这个系统调用,在kernel mode调用了path_init、path_walk、atomic_dec_and_lock、__d_lookup等函数

系统运行过程中,存在CPU时间片切换。当一个进程能够在其得到的一个时间片内完成所有工作,就不会产生锁的争夺。但当系统压力变大,一个进程在时间片内未完成工作,而其公共资源又被spin_lock锁住,造成大量进程的等待堆积和切换,致使CPU sys高。

当前代码需要满足题意:**Problem Description**: In this task, you will implement a multi-process dataset generator. We have provided you with the `process_labels` function, which generates samples based on labels and font files. Follow the requirements and hints below to complete the code. **Task Requirements**: 1. **Font File Reading**(1 Pt): - Define the variable `font_directory` as the font file storage path `./Font`. Read all `.ttf` and `.ttc` files from this directory and store their paths in the `font_files` list. 2. **Calculate Total Sample Count**(1 Pt): - Set the variable `count` to specify the number of samples to generate per label, with a generation quantity of 100 samples per label. - The `labels` list, which contains all labels, has been previously defined. Calculate the total number of samples, `total_samples`. 3. **Shared Progress Counter and Lock**(2 Pts): - Create a multi-process shared variable `progress` (using `Value`) to track the total number of generated samples. - Create a `Lock` object to manage access to shared resources. 4. **Label Splitting**(1 Pt): - Define the variable `num_processes` to indicate the number of processes. Set it to the number of cores on your processor. - Split `labels` into `num_processes` chunks so that each process handles one chunk of labels. 5. **Multi-process Implementation**(2 Pts): - Create a `Process` instance for each chunk of labels and call the `process_labels` function to handle that chunk. Pass `font_files`, `count`, `progress`, and `lock` as arguments to `process_labels`. - Start each process so they run in parallel. 6. **Progress Bar Implementation**(1 Pt): - Use `tqdm` to display the total progress of sample generation, updating once per second. - In a loop, check the status of each process and use the `progress` variable to update the `tqdm` progress bar until all processes finish. 7. **Wait for Processes to Finish**(1 Pt): - Use `join` to ensure that all processes complete before the code continues executing. 8. **Completion Message**(1 Pt): - After all processes are complete, print “All samples generated”. **Hints**: - Ensure that you pass the correct parameters to `process_labels` in each process. - Use `lock` to ensure thread-safe progress updates. - Try debugging and running the code to ensure that the progress bar and parallel processes display and execute correctly.请你检查一下是否有没完成题意的地方,中文告诉我。另外,当前代码输出为Generating Samples: 100%|██████████| 359300/359300 [50:19<00:00, 119.01it/s]All samples generated,上面多了一行0%,请你把这个多一行的问题解决掉,其他的不要修改,告诉我是否有不满足题目要求的部分即可
最新发布
11-25
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值