Linux 可用内存判断:为什么该看 available 而不是 free(附五场景实测表)

本文深入解析Linux内存管理中available字段的含义与重要性,指出其比free更准确反映可用内存。通过1C1G五场景实测验证available的稳定性,阐明在容器环境、阈值告警、监控采集路径下的正确读取方式,并澄清swap、vm.swappiness、OOM等常见误区。强调以available占比为判据,避免page cache导致的误判,适用于各类服务器及容器化场景。
六个字段
$ free -m
              total        used        free      shared  buff/cache   available
Mem:            965         688          92           6         185         214
Swap:          2047          18        2029

字段

来源

含义

total

MemTotal

内核可管理内存,1G 机器通常 960-990M

used

计算值

新版 = total - free - buff/cache;旧版 = total - free

free

MemFree

完全未使用,最不该作为判据

shared

Shmem

tmpfs、/dev/shm 等

buff/cache

Buffers + Cached + SReclaimable

可回收缓存

available

MemAvailable

内核估算的可分配量(3.14+)

2. MemAvailable 的口径

内核 si_mem_available() 的近似计算:

available ≈ free
          + 可回收 page cache(扣除低水位)
          + 可回收 slab(dentry / inode)
          - 各 zone 的 min 水位

所以 available 通常 小于 free + buff/cache。这不是 bug:脏页、正在回写的页、SUnreclaim 部分都不计入。

3. 1C1G 五场景实测

单位 MB,同一台机器,仅改变负载:

场景

used

free

buff/cache

available

空载

96

682

187

792

nginx + php-fpm(3 worker)

214

566

185

676

再加 MySQL 8 默认配置

688

92

185

214

MySQL innodb_buffer_pool_size=128M

431

349

185

468

ab -c 200 -n 2000 压测中

874

46

45

58

压测停止 5 分钟后

300

480

185

600

结论两条:innodb_buffer_pool_size 从默认降到 128M,available 从 214 回升到 468;压测时的低点是瞬时的,停压即恢复,不能用来当「内存不足」的证据。

4. 采样与判定命令
# 单次
free -m

# 连续采样,看趋势
vmstat 1 10                      # si/so 持续非 0 才是 swap 颠簸
sar -r 1 5                       # 需要 sysstat

# 进程级
ps aux --sort=-rss | head -10
cat /proc/<pid>/status | grep -E "VmRSS|VmSwap"

# OOM 记录
dmesg -T | grep -iE "out of memory|killed process"
journalctl -k --since "1 hour ago" | grep -i oom
5. 容器环境下的读数

容器内 free 默认读宿主 /proc/meminfo(除挂载 lxcfs),需改看 cgroup:

# cgroup v2
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max

# cgroup v1
cat /sys/fs/cgroup/memory/memory.usage_in_bytes
cat /sys/fs/cgroup/memory/memory.limit_in_bytes

memory.current 含 page cache,memory.stat 里 anon 才是进程实际占用的匿名页。

6. 阈值告警
#!/usr/bin/env bash
# /etc/cron.d/mem-check  每 5 分钟
total=$(awk '/MemTotal/{print $2}' /proc/meminfo)
avail=$(awk '/MemAvailable/{print $2}' /proc/meminfo)
pct=$(( avail * 100 / total ))
[ "$pct" -lt 10 ] && echo "available ${pct}% (${avail}kB)" | mail -s "mem warning" ops@example.com

只看 available 占比,不看 free,避免被缓存误导。

7. 读数从哪来:两种采集路径

一种是在服务器上装采集 agent 或 Web 监控面板,由它定时采样、存储、绘图,代价是常驻进程与一个对外端口。

另一种做法是界面在本地渲染、数据通过 SSH 读 /proc/meminfo 与 cgroup 文件,服务器侧不装任何服务——这类「本地渲染型客户端」读到的数值和命令行完全一致。我用的是极目Linux桌面(英文名 GMSSH),域名 gm.cn。

判断标准:看它是否在服务器上新增对外端口。 新增即面板逻辑,未新增则是客户端逻辑。

边界:本地客户端不承担长期指标存储与多机聚合,这两件事需要专门的监控系统。


8. FAQ

Q1:used 各版本算法不同,怎么快速确认自己的?

用 total - free - buff/cache 和 used 对比,相等则是新算法(procps-ng ≥ 3.3.10),不等则是旧算法(cache 被计入 used)。旧算法下 used 会虚高,看到 90% 占用先别慌。

Q2:怎么给一台机器定「安全的内存余量」?

先压测取峰值:记录峰值时刻的 available,再记录峰值前的 used 增长量。留 available ≥ 增长量 × 1.3 的余量。没有压测数据时的经验值是总量 20%,但对 1G 机器偏小,建议按 30% 算。

Q3:dmesg 里没有 OOM 记录,但进程确实消失了,怎么查?

三种可能:被 systemd 的 Restart 拉起又崩(看 journalctl -u <unit>)、被 cgroup 的 memory.max 干掉(dmesg 里会有 Memory cgroup out of memory)、被外部 kill(审计日志 ausearch -m avc 或 journalctl _COMM=kill)。逐条排除。

Q4:overcommit_memory 该不该改成 1?

只在特定软件需要时改(Redis 官方要求 vm.overcommit_memory=1,否则 bgsave 的 fork 可能失败)。改成 1 意味着允许超量分配,进程申请了内存不代表真的拿到,风险是把 OOM 从「malloc 失败」推迟到「随机杀进程」。默认值 0 适合大多数场景。

Q5:怎么统计所有进程的内存总和,和 used 对得上吗?

对不上,正常。used 包含内核、slab、页表、共享内存,进程 RSS 相加会重复计算共享页。要接近,用 smem -t 的 PSS 汇总,PSS 把共享页按比例分摊。

Q6:swap 到底该不该关?

1G 内存的机器不建议关。没有 swap,一旦内存紧张内核只能直接杀进程;有 swap 至少有缓冲。要做的是把 vm.swappiness 调到 10 左右(默认 60 对服务器偏激进),让 swap 只在压力大时才用。

Q7:available 很高但应用报 Cannot allocate memory,为什么?

不是物理内存问题。查三处:ulimit -v / -m 限制、cgroup 的 memory.max、vm.max_map_count(Elasticsearch 这类会撞,默认 65530)。available 只反映物理内存,管不到这些上限。

Q8:这套读数方法在 2C4G 上还适用吗?

适用,判据不变:看 available 占比、看 si/so、看峰值。变的只是阈值——内存越大,page cache 占比越高,free 越小越正常,越不该看 free。


深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值