文章探讨容器内存QOS，指出容器使用内存受自身内存和节点内存限制。自身内存接近上限会触发内核内存回收，影响应用性能；节点内存不足会触发全局内存回收，甚至致整机异常。还通过多轮测试验证，当free page处于low和min之间会发生swap。
引申文章的结论容器内存QOS
容器在使用内存时主要有以下两个方面的约束：
自身内存限制：当容器自身的内存（含Page Cache）接近容器上限时，会触发内核的内存回收子系统，这个过程会影响容器内应用的内存申请和释放的性能。
节点内存限制：当容器内存超卖（Memory Limit>Request）导致整机内存不足，会触发内核的全局内存回收，这个过程对性能影响较大，极端情况甚至导致整机异常。
文章目录
第一部分：容器自身内存
limit_in_bytes有deployment决定，usage_in_bytes是容器不断申请的内存
当usage_in_bytes接近limit_in_bytes的时候，就会发生SWAP
第二部分：宿主机内存不足
验证结论：当free page处于low和min之间发生swap
Low WaterMark的值由谁决定？
vm.min_free_kbytes
This is used to force the Linux VM to keep a minimum numberof kilobytes free. The VM uses this number to compute a
watermark[WMARK_MIN] value for each lowmem zone in the system.Each lowmem zone gets a number of reserved free pages based
proportionally on its size.
Some minimal amount of memory is needed to satisfy PF_MEMALLOC allocations; if you set this to lower than 1024KB, your system will
become subtly broken, and prone to deadlock under high loads.
Setting this too high will OOM your machine instantly.
环境说明
master版本：kubernetes-1.20.2.2
node版本：kubelet-1.20.3.2-1
测试说明
NUMA架构下，进程使用的内存到达什么程度（本文中watermark），会出现SWAP现象
第一轮测试（interleave）
结论：free page总数为52448，low为28156时，发生SWAP(low: node0+node1+DMA32+DMA=28156)，总数不是触发swap的原因
2023-08-01 13:58:07 # 发生swap
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu-------- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
1 0 0 13383900 17036 1917516 0 0 0 0 5109 4988 1 4 96 0 0 2023-08-01 13:57:12
1 0 0 12201976 17044 1917540 0 0 0 40 6252 5057 0 4 96 0 0 2023-08-01 13:57:13
1 0 0 11014868 17044 1917580 0 0 0 0 5941 5077 1 4 96 0 0 2023-08-01 13:57:14
1 0 0 9826628 17044 1917580 0 0 0 4 4624 5038 1 4 96 0 0 2023-08-01 13:57:15
1 0 0 8639280 17044 1917608 0 0 0 0 4448 4999 0 4 96 0 0 2023-08-01 13:57:16
1 0 0 7452688 17044 1917632 0 0 0 0 4731 5085 1 4 96 0 0 2023-08-01 13:57:17
1 0 0 6263668 17052 1917656 0 0 0 16 4591 4995 0 4 96 0 0 2023-08-01 13:57:18
1 0 0 5075872 17052 1917688 0 0 0 0 4757 5156 0 4 96 0 0 2023-08-01 13:57:19
1 0 0 3882124 17052 1917736 0 0 0 24 5198 5186 0 4 96 0 0 2023-08-01 13:57:20
1 0 0 2690316 17052 1917760 0 0 0 0 5868 5139 0 4 96 0 0 2023-08-01 13:57:21
6 0 0 1491108 17076 1917772 0 0 0 132 5128 6177 1 4 95 0 0 2023-08-01 13:57:22
0 0 0 449544 17076 1917800 0 0 0 0 4790 4765 0 4 96 0 0 2023-08-01 13:57:23
0 0 0 442132 17076 1917800 0 0 0 0 2931 4833 0 0 100 0 0 2023-08-01 13:57:24
0 0 0 435732 17076 1917692 0 0 0 0 3010 4945 0 0 100 0 0 2023-08-01 13:57:25
0 0 0 429568 17076 1917692 0 0 0 0 2794 4827 0 0 100 0 0 2023-08-01 13:57:26
0 0 0 422872 17084 1917688 0 0 0 40 2931 4844 0 0 100 0 0 2023-08-01 13:57:27
0 0 0 416448 17084 1917688 0 0 0 0 2880 4789 0 0 100 0 0 2023-08-01 13:57:28
0 0 0 409988 17084 1917688 0 0 0 0 2926 4867 0 0 100 0 0 2023-08-01 13:57:29
0 0 0 404164 17084 1917576 0 0 0 0 2892 4793 0 0 100 0 0 2023-08-01 13:57:30
0 0 0 397724 17084 1917576 0 0 0 0 2797 4771 0 0 100 0 0 2023-08-01 13:57:31
0 0 0 390216 17084 1917576 0 0 0 0 2939 4853 0 0 100 0 0 2023-08-01 13:57:32
0 0 0 384532 17092 1917576 0 0 0 16 2852 4892 0 0 100 0 0 2023-08-01 13:57:33
0 0 0 378520 17092 1917580 0 0 0 0 3294 4829 0 0 100 0 0 2023-08-01 13:57:34
0 0 0 372896 17092 1917580 0 0 0 0 2945 4825 0 0 100 0 0 2023-08-01 13:57:35
0 0 0 367016 17092 1917628 0 0 0 0 2925 4815 0 0 100 0 0 2023-08-01 13:57:36
0 0 0 361992 17092 1917680 0 0 0 0 3443 4875 0 0 100 0 0 2023-08-01 13:57:37
0 0 0 354880 17092 1917672 0 0 0 0 2869 4840 0 0 100 0 0 2023-08-01 13:57:38
0 0 0 349452 17092 1917704 0 0 0 0 2839 4856 0 0 100 0 0 2023-08-01 13:57:39
0 0 0 344152 17100 1917456 0 0 0 44 2811 4858 0 0 100 0 0 2023-08-01 13:57:40
0 0 0 337628 17100 1917456 0 0 0 0 2848 4849 0 0 100 0 0 2023-08-01 13:57:41
0 0 0 331500 17100 1917456 0 0 0 0 2903 4816 0 0 100 0 0 2023-08-01 13:57:42
0 0 0 325144 17100 1917460 0 0 0 0 2975 4889 0 0 100 0 0 2023-08-01 13:57:43
0 0 0 318320 17100 1917364 0 0 0 0 2886 4809 0 0 100 0 0 2023-08-01 13:57:44
0 0 0 312924 17100 1917368 0 0 0 84 2878 4903 0 0 100 0 0 2023-08-01 13:57:45
0 0 0 306752 17108 1917368 0 0 0 12 2890 4830 0 0 100 0 0 2023-08-01 13:57:46
0 0 0 300328 17108 1917368 0 0 0 0 2905 4863 0 0 100 0 0 2023-08-01 13:57:47
0 0 0 294508 17108 1917376 0 0 0 0 2858 4860 0 0 100 0 0 2023-08-01 13:57:48
0 0 0 288516 17108 1917364 0 0 0 0 2891 4973 0 0 100 0 0 2023-08-01 13:57:49
0 0 0 282684 17108 1917368 0 0 0 8 2810 4836 0 0 100 0 0 2023-08-01 13:57:50
0 0 0 276884 17108 1917368 0 0 0 0 2905 4839 0 0 100 0 0 2023-08-01 13:57:51
0 0 0 270796 17116 1917392 0 0 0 28 2846 4832 0 0 100 0 0 2023-08-01 13:57:52
0 0 0 264736 17116 1917444 0 0 0 0 2935 4873 0 0 100 0 0 2023-08-01 13:57:53
0 0 0 259024 17116 1917444 0 0 0 0 2923 4780 0 0 100 0 0 2023-08-01 13:57:54
0 0 0 252824 17116 1917380 0 0 0 0 3339 4849 0 0 100 0 0 2023-08-01 13:57:55
0 0 0 246836 17116 1917388 0 0 0 0 2918 4804 0 0 100 0 0 2023-08-01 13:57:56
1 0 0 241476 17124 1917380 0 0 0 12 2827 4847 0 0 100 0 0 2023-08-01 13:57:57
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu-------- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
0 0 0 236292 17124 1917292 0 0 0 0 2765 4823 0 0 100 0 0 2023-08-01 13:57:58
0 0 0 229896 17124 1917292 0 0 0 0 2873 4810 0 0 100 0 0 2023-08-01 13:57:59
0 0 0 222036 17124 1917292 0 0 0 0 2905 4795 0 0 100 0 0 2023-08-01 13:58:00
0 0 0 217004 17124 1917292 0 0 0 0 2935 4829 0 0 100 0 0 2023-08-01 13:58:01
0 0 0 207496 17124 1917204 0 0 0 16 5409 7852 1 1 99 0 0 2023-08-01 13:58:02
0 0 0 201400 17132 1917180 0 0 0 60 3004 4933 0 0 100 0 0 2023-08-01 13:58:03
0 0 0 195388 17132 1917256 0 0 0 0 3260 4834 0 0 100 0 0 2023-08-01 13:58:04
0 0 0 192644 17132 1917168 0 0 0 16 4280 6426 0 0 99 0 0 2023-08-01 13:58:05
0 0 0 188004 17132 1917280 0 0 0 0 4151 6277 0 0 100 0 0 2023-08-01 13:58:06
0 0 27400 215820 13964 1885852 0 26900 20 26900 3201 4885 0 0 100 0 0 2023-08-01 13:58:07 # 发生swap
0 0 27400 209136 13964 1885852 0 0 0 4 2863 4814 0 0 100 0 0 2023-08-01 13:58:08
0 0 27400 202204 13964 1885900 0 0 0 0 2797 4850 0 0 100 0 0 2023-08-01 13:58:09
0 0 27400 197848 13972 1885964 0 0 0 232 2883 4820 0 0 100 0 0 2023-08-01 13:58:10
0 0 27400 189532 13972 1885980 0 0 0 0 2941 4781 0 0 100 0 0 2023-08-01 13:58:11
0 0 50176 211328 12576 1860264 0 23028 0 23028 3139 4912 0 0 100 0 0 2023-08-01 13:58:12
0 0 50176 206004 12576 1859976 0 0 0 0 2967 4876 0 0 100 0 0 2023-08-01 13:58:13
0 0 50176 200144 12576 1859696 0 0 0 0 2777 4813 0 0 100 0 0 2023-08-01 13:58:14
0 0 50176 194332 12584 1859764 0 0 0 68 2887 4931 0 0 100 0 0 2023-08-01 13:58:15
0 0 50176 189380 12584 1859828 0 0 0 64 2852 4840 0 0 100 0 0 2023-08-01 13:58:16
0 0 74752 212952 11928 1831328 0 24252 0 24252 3122 4911 0 0 100 0 0 2023-08-01 13:58:17
0 0 74752 207316 11928 1831392 0 0 0 0 2771 4802 0 0 100 0 0 2023-08-01 13:58:18
0 0 74752 202036 11928 1831288 0 0 0 0 3062 5030 0 0 100 0 0 2023-08-01 13:58:19
1 0 74752 196112 11928 1831168 0 0 0 0 3284 4873 0 0 100 0 0 2023-08-01 13:58:20
0 0 74752 189160 11936 1830984 0 0 0 16 3317 4891 0 0 100 0 0 2023-08-01 13:58:21
0 0 99848 212816 10760 1803260 0 25160 628 25160 3235 4946 0 0 99 0 0 2023-08-01 13:58:22
0 0 99848 590144 10760 1803384 0 0 116 0 3030 4893 0 0 99 0 0 2023-08-01 13:58:23
0 0 99848 591800 10760 1803384 0 0 0 0 2875 4833 0 0 100 0 0 2023-08-01 13:58:24
0 0 99848 592112 10760 1803416 0 0 0 0 2870 4886 0 0 100 0 0 2023-08-01 13:58:25
0 0 99848 592808 10760 1803416 0 0 0 0 2782 4787 0 0 100 0 0 2023-08-01 13:58:26
0 0 99848 592640 10768 1803424 0 0 0 20 2874 4787 0 0 100 0 0 2023-08-01 13:58:27
0 0 99848 592664 10768 1803424 0 0 0 0 2853 4822 0 0 100 0 0 2023-08-01 13:58:28
0 0 99848 592460 10768 1803436 0 0 0 0 2817 4858 0 0 100 0 0 2023-08-01 13:58:29
0 0 99848 593112 10768 1803504 0 0 0 0 2576 4763 0 0 100 0 0 2023-08-01 13:58:30
1 0 99848 594024 10768 1803512 0 0 0 0 2638 4438 0 0 100 0 0 2023-08-01 13:58:31
0 0 99848 593624 10768 1803544 0 0 0 0 3201 4755 0 0 100 0 0 2023-08-01 13:58:32
0 0 99848 595256 10776 1803564 0 0 0 12 2171 4325 0 0 100 0 0 2023-08-01 13:58:33
0 0 99848 595372 10776 1803572 0 0 0 0 2119 4239 0 0 100 0 0 2023-08-01 13:58:34
/proc/zoneinfo中node0 node1 DMA32 DMA空闲的页数
Tue Aug 1 13:57:53 CST 2023
65964
Tue Aug 1 13:57:54 CST 2023
64999
Tue Aug 1 13:57:55 CST 2023
61712
Tue Aug 1 13:57:56 CST 2023
60052
Tue Aug 1 13:57:57 CST 2023
58580
Tue Aug 1 13:57:58 CST 2023
57764
Tue Aug 1 13:57:59 CST 2023
55598
Tue Aug 1 13:58:00 CST 2023
53909
Tue Aug 1 13:58:01 CST 2023
51740
Tue Aug 1 13:58:02 CST 2023
50056
Tue Aug 1 13:58:03 CST 2023
48785
Tue Aug 1 13:58:04 CST 2023
47993
Tue Aug 1 13:58:05 CST 2023
46949
Tue Aug 1 13:58:06 CST 2023
53898
Tue Aug 1 13:58:07 CST 2023 # 发生SWAP
52448
Tue Aug 1 13:58:08 CST 2023
50937
Tue Aug 1 13:58:09 CST 2023
49264
Tue Aug 1 13:58:10 CST 2023
47729
Tue Aug 1 13:58:11 CST 2023
52992
Tue Aug 1 13:58:12 CST 2023
51267
Tue Aug 1 13:58:13 CST 2023
50188
Tue Aug 1 13:58:14 CST 2023
48618
Tue Aug 1 13:58:15 CST 2023
47499
Tue Aug 1 13:58:16 CST 2023
53599
Tue Aug 1 13:58:17 CST 2023
51806
Tue Aug 1 13:58:18 CST 2023
50648
Tue Aug 1 13:58:19 CST 2023
49179
Tue Aug 1 13:58:21 CST 2023
47701
Tue Aug 1 13:58:22 CST 2023
53519
Tue Aug 1 13:58:23 CST 2023
147874
Tue Aug 1 13:58:24 CST 2023
148143
Tue Aug 1 13:58:25 CST 2023
148130
第二次测试（interleave）
结论：free page数(node0 + node1)是触发swap的原因
node0 + node 1 watermark
- high 33788
- low 28156
- min 22526
红色代表介于low和min之间的数值，黄色代表发生“换出”的时刻
2023-08-01 15:54:51 # 发生SWAP
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu-------- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
1 0 0 270696 13356 1911668 0 0 0 60 2889 4766 0 0 100 0 0 2023-08-01 15:54:36
1 0 0 264680 13356 1911724 0 0 0 0 2811 4696 0 0 100 0 0 2023-08-01 15:54:37
0 0 0 258588 13356 1911696 0 0 0 0 2802 4701 0 0 100 0 0 2023-08-01 15:54:38
0 0 0 252884 13356 1911696 0 0 0 0 2814 4633 0 0 100 0 0 2023-08-01 15:54:39
0 0 0 246424 13356 1911696 0 0 0 0 2881 4697 0 0 100 0 0 2023-08-01 15:54:40
0 0 0 239936 13364 1911688 0 0 0 48 2814 4780 0 0 100 0 0 2023-08-01 15:54:41
0 0 0 234540 13364 1911832 0 0 0 0 2800 4722 0 0 100 0 0 2023-08-01 15:54:42
0 0 0 229276 13364 1911832 0 0 0 0 2806 4643 0 0 100 0 0 2023-08-01 15:54:43
0 0 0 222704 13364 1911916 0 0 0 0 2916 4723 0 0 100 0 0 2023-08-01 15:54:44
0 0 0 216220 13364 1911920 0 0 0 0 2873 4726 0 0 100 0 0 2023-08-01 15:54:45
0 0 0 210748 13364 1911996 0 0 0 48 2775 4731 0 0 100 0 0 2023-08-01 15:54:46
0 0 0 203924 13372 1912040 0 0 44 16 3394 5640 0 0 100 0 0 2023-08-01 15:54:47
0 0 0 198244 13372 1912084 0 0 0 0 2831 4690 0 0 100 0 0 2023-08-01 15:54:48
0 0 0 192572 13372 1912128 0 0 0 0 2837 4796 0 0 100 0 0 2023-08-01 15:54:49
0 0 0 186160 13372 1912172 0 0 0 0 2783 4718 0 0 100 0 0 2023-08-01 15:54:50
0 0 26632 209544 11664 1883588 0 26532 0 26548 3305 4756 0 0 100 0 0 2023-08-01 15:54:51 # 发生SWAP
0 0 26632 201864 11664 1883792 0 0 0 0 2795 4653 0 0 100 0 0 2023-08-01 15:54:52
1 0 26632 198032 11672 1883824 0 0 0 32 2894 4792 0 0 100 0 0 2023-08-01 15:54:53
0 0 26632 191552 11672 1883816 0 0 0 0 2959 4754 0 0 100 0 0 2023-08-01 15:54:54
0 0 26632 184904 11672 1883716 0 0 0 0 2791 4775 0 0 100 0 0 2023-08-01 15:54:55
0 0 54272 209980 10708 1853732 0 27472 0 27472 3092 4685 0 0 100 0 0 2023-08-01 15:54:56
0 0 54272 203704 10708 1853424 0 0 0 0 2818 4751 0 0 100 0 0 2023-08-01 15:54:57
0 0 54272 198848 10716 1853432 0 0 0 40 2962 4751 0 0 100 0 0 2023-08-01 15:54:58
0 0 54272 193192 10716 1853248 0 0 0 0 2874 4684 0 0 100 0 0 2023-08-01 15:54:59
0 0 54272 187484 10716 1853208 0 0 0 0 2797 4705 0 0 100 0 0 2023-08-01 15:55:00
0 0 83464 213928 9848 1821620 0 28948 0 28948 3185 4839 0 0 100 0 0 2023-08-01 15:55:01
0 0 83464 206944 9848 1821620 0 0 0 0 2928 4665 0 0 100 0 0 2023-08-01 15:55:02
0 0 83464 201428 9856 1821556 0 0 0 24 2882 4750 0 0 100 0 0 2023-08-01 15:55:03
0 0 83464 197520 9856 1821628 0 0 0 0 2744 4730 0 0 100 0 0 2023-08-01 15:55:04
0 0 83464 192652 9856 1821628 0 0 0 0 2780 4795 0 0 100 0 0 2023-08-01 15:55:05
0 0 83464 185548 9856 1821628 0 0 0 0 3308 4775 0 0 100 0 0 2023-08-01 15:55:06
1 0 113160 211432 9056 1789504 0 29696 0 29696 3340 4746 0 0 99 0 0 2023-08-01 15:55:07
1 0 113160 205060 9064 1789560 0 0 0 28 2781 4610 0 0 100 0 0 2023-08-01 15:55:08
0 0 113160 199376 9064 1789596 0 0 0 0 2893 4783 0 0 100 0 0 2023-08-01 15:55:09
0 0 113160 193324 9064 1789532 0 0 0 0 2913 4715 0 0 100 0 0 2023-08-01 15:55:10
0 0 113160 188020 9064 1789568 0 0 0 0 3182 4693 0 0 100 0 0 2023-08-01 15:55:11
1 0 113160 4757668 9064 1789624 0 0 0 0 13473 5128 0 4 96 0 0 2023-08-01 15:55:12
1 0 113160 10269964 9064 1789732 0 0 0 0 25454 5233 0 5 95 0 0 2023-08-01 15:55:13
第三次测试（membind & cpunodebind）
绑定node0执行程序
numactl --cpunodebind=0 --membind=0 <消耗内存>
node0 watermark
- min 9916
- low 12395
- high 14874
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu-------- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
0 0 0 14351512 25104 1962528 0 0 0 0 2904 4970 0 0 100 0 0 2023-08-03 09:55:30
0 0 0 14346912 25104 1962548 0 0 0 0 2884 5065 0 0 100 0 0 2023-08-03 09:55:31
0 0 0 14342216 25104 1962472 0 0 0 0 2837 5064 0 0 100 0 0 2023-08-03 09:55:32
0 0 0 14334776 25112 1962464 0 0 0 16 2873 4965 0 0 100 0 0 2023-08-03 09:55:33
0 0 6912 14342732 23108 1954964 148 6492 3500 6496 3307 5472 0 0 99 1 0 2023-08-03 09:55:34
0 0 6912 14334836 23108 1954924 108 0 776 0 2982 5114 0 0 100 0 0 2023-08-03 09:55:35
0 0 27648 14349636 23108 1954556 400 20736 400 20736 3332 5278 0 0 100 0 0 2023-08-03 09:55:36
0 0 27648 14332524 23108 1954452 4 0 4 8 2823 5060 0 0 100 0 0 2023-08-03 09:55:37
0 0 42240 14341864 23108 1953920 8 14536 8 14540 3002 5070 0 0 100 0 0 2023-08-03 09:55:38
0 0 42240 14335856 23116 1953940 8 0 8 28 3007 5145 0 0 100 0 0 2023-08-03 09:55:39
0 0 54784 14343760 23112 1952520 72 12524 72 12524 3110 5192 0 0 100 0 0 2023-08-03 09:55:40
0 0 54784 14338808 23112 1952548 52 0 52 0 2982 5150 0 0 100 0 0 2023-08-03 09:55:41
0 0 54784 14331508 23112 1952220 12 0 12 0 2939 5054 0 0 100 0 0 2023-08-03 09:55:42
0 0 80384 14350312 23112 1951768 144 25632 144 25632 3146 5127 0 0 100 0 0 2023-08-03 09:55:43
0 0 80384 14345308 23116 1951868 72 0 72 16 3058 5174 0 0 100 0 0 2023-08-03 09:55:44
0 0 80384 14338244 23116 1951888 68 0 68 0 2887 5107 0 0 100 0 0 2023-08-03 09:55:45
1 0 80384 14408156 23116 1951836 8 0 16 0 2922 5116 0 0 100 0 0 2023-08-03 09:55:46
第四次测试（membind & cpunodebind）
绑定node1执行程序
numactl --cpunodebind=1 --membind=1 <消耗内存>
node1 watermark
- min 11347
- low 14183
- high 17020
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu-------- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
0 0 0 14937376 3992 1882076 0 0 0 0 2870 5005 0 0 100 0 0 2023-08-03 10:14:20
0 0 0 14930568 3992 1882096 0 0 0 0 2862 4951 0 0 100 0 0 2023-08-03 10:14:21
0 0 0 14924452 3992 1882120 0 0 0 0 2968 4933 0 0 100 0 0 2023-08-03 10:14:22
0 0 0 14919700 3992 1882128 0 0 0 8 2862 4914 0 0 100 0 0 2023-08-03 10:14:23
0 0 0 14913292 3992 1882108 0 0 88 0 2925 4979 0 0 100 0 0 2023-08-03 10:14:24
0 0 0 14906976 4000 1882208 0 0 0 32 2959 4913 0 0 100 0 0 2023-08-03 10:14:25
3 0 776 14902496 4000 1881796 0 828 0 828 2815 4905 0 0 100 0 0 2023-08-03 10:14:26
0 0 22024 14921620 3932 1860672 0 21084 0 21084 2964 4936 0 0 100 0 0 2023-08-03 10:14:27
0 0 22024 14915864 3936 1860668 0 0 4 12 2810 4909 0 0 100 0 0 2023-08-03 10:14:28
0 0 22024 14909660 3936 1860668 0 0 0 0 2834 4863 0 0 100 0 0 2023-08-03 10:14:29
0 0 22024 14903400 3944 1860664 0 0 0 28 2938 4928 0 0 100 0 0 2023-08-03 10:14:30
0 0 45056 14919788 3944 1837744 0 23024 0 23024 3054 4898 0 0 100 0 0 2023-08-03 10:14:31
0 0 45056 14913600 3944 1838232 0 0 0 0 2920 4966 0 0 100 0 0 2023-08-03 10:14:32
0 0 45056 14906076 3948 1838228 0 0 4 28 2783 4847 0 0 100 0 0 2023-08-03 10:14:33
0 0 68608 14924196 3948 1815208 0 23560 0 23560 3062 4971 0 0 100 0 0 2023-08-03 10:14:34
1 0 68608 14916216 3948 1815208 0 0 0 0 2833 4869 0 0 100 0 0 2023-08-03 10:14:35
0 0 68608 14911940 3956 1815208 0 0 0 32 3265 5021 0 0 100 0 0 2023-08-03 10:14:36
扫一扫
854
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
