GNU/Linux - /proc/sys/vm/overcommit_memory

/proc/sys/vm/overcommit_memory "是一个 Linux 内核参数,用于控制系统处理内存分配请求的方式。该参数对决定进程请求内存时内核的行为至关重要。让我们来详细了解一下它的含义和影响:
The "/proc/sys/vm/overcommit_memory" is a Linux kernel parameter that controls how the system handles memory allocation requests. This parameter is crucial in determining the kernel's behavior when processes request memory. Let's explore its meaning and implications in detail:
Overview
overcommit_memory 参数决定内核的内存超量分配策略。内存超额分配允许系统为进程分配的虚拟内存超过可用的物理内存(包括交换)。
The overcommit_memory parameter determines the kernel's policy for memory overcommitment. Memory overcommitment allows the system to allocate more virtual memory to processes than the amount of physical memory (including swap) available.
Available Modes
overcommit_memory 有三种可能的值,分别代表不同的超量提交处理模式:
There are three possible values for overcommit_memory, each representing a different overcommit handling mode:
    1. 模式 0(默认): 启发式超承诺处理
   - 这是默认设置。
   - 内核使用启发式方法决定是否允许或拒绝内存分配请求。
   - 内核通常允许轻微的超量分配,但拒绝明显的地址空间超量分配。
   - 该模式旨在防止疯狂分配,同时减少交换使用。
   - 在此模式下,允许根用户分配稍多的内存。
    1. Mode 0 (Default): Heuristic overcommit handling
   - This is the default setting.
   - The kernel uses a heuristic approach to decide whether to allow or refuse memory allocation requests.
   - It typically allows slight overcommitment but refuses obvious overcommits of address space.
   - This mode aims to prevent wild allocations while reducing swap usage.
   - Root users are allowed to allocate slightly more memory in this mode.
  2. 模式 1:总是超量分配
   - 在这种模式下,无论当前内存使用情况如何,内核总是允许内存分配请求。
   - 这种模式适用于某些科学应用程序,尤其是那些使用稀疏阵列(主要由零页组成)的应用程序。
   - 这种模式可能会有风险,因为它可能会更频繁地导致内存不足(OOM)的情况。
    2. Mode 1: Always overcommit
   - In this mode, the kernel always allows memory allocation requests, regardless of the current memory usage.
   - It's appropriate for some scientific applications, particularly those using sparse arrays that consist mostly of zero pages.
   - This mode can be risky as it may lead to out-of-memory (OOM) situations more frequently.
    3. 模式 2:不过度承诺
   - 这是最保守的模式。
   - 系统提交的总地址空间不允许超过 swap 加上物理 RAM 的可配置百分比(默认为 50%)。
   - 当达到限制时,进程将在内存分配时收到错误信息,而不是在访问页面时被杀死。
   - 该模式适用于需要保证其内存分配在未来可用的应用程序。
    3. Mode 2: Don't overcommit
   - This is the most conservative mode.
   - The total address space commit for the system is not permitted to exceed swap plus a configurable percentage (default 50%) of physical RAM.
   - Processes will receive errors on memory allocation when the limit is reached, rather than being killed while accessing pages.
   - This mode is useful for applications that need to guarantee their memory allocations will be available in the future.
Setting and Viewing the Value
    您可以使用以下命令查看 overcommit_memory 的当前值:
    You can view the current value of overcommit_memory using the following command:
cat /proc/sys/vm/overcommit_memory
    要临时更改数值,可以使用
    To change the value temporarily, you can use:
echo 1 | sudo tee /proc/sys/vm/overcommit_memory
    如需永久更改,请在 /etc/sysctl.conf 中添加以下一行:
    For a permanent change, add the following line to /etc/sysctl.conf:
vm.overcommit_memory=x
其中,“x ”为所需值(0、1 或 2)。
Where 'x' is the desired value (0, 1, or 2).
Related Parameters
  •  vm.overcommit_ratio: 该参数设置模式 2 时允许超量提交的物理 RAM 百分比。
  •  vm.overcommit_kbytes: overcommit_ratio 的替代参数,指定以千字节为单位的超量分配限制。
  • vm.overcommit_ratio: This parameter sets the percentage of physical RAM to allow for overcommitment when in mode 2.
  • vm.overcommit_kbytes: An alternative to overcommit_ratio, specifying the overcommit limit in kilobytes.
在我的Ubuntu里看了下,ratio是50,kbytes是0。
Practical Implications
    1. 模式 0(默认): 兼顾大多数系统的性能和稳定性。
    2. 模式 1:适用于需要大量虚拟内存但不一定全部使用虚拟内存的应用程序。不过,如果物理内存耗尽,可能会导致系统不稳定。
    3. 模式 2:提供最高级别的内存分配保证,但可能会限制某些应用程序的性能。
    1. Mode 0 (Default): Balances performance and stability for most systems.
    2. Mode 1: Useful for applications requiring large amounts of virtual memory but not necessarily using it all. However, it can lead to system instability if physical memory is exhausted.
    3. Mode 2: Provides the highest level of memory allocation guarantee but may limit performance for some applications.
Choosing the Right Mode
超量提交模式的选择取决于具体的使用情况:
- 对于通用系统,默认模式 0 通常比较合适。
- 科学应用或使用稀疏数据结构的应用可能会从模式 1 中受益。
- 需要严格内存分配保证的系统应考虑使用模式 2。
The choice of overcommit mode depends on your specific use case:
- For general-purpose systems, the default mode 0 is usually suitable.
- Scientific applications or those using sparse data structures might benefit from mode 1.
- Systems requiring strict memory allocation guarantees should consider mode 2.
了解并正确配置 overcommit_memory 参数对于优化系统性能和稳定性至关重要,尤其是在有特定内存要求或限制的环境中。
Understanding and properly configuring the overcommit_memory parameter is crucial for optimizing system performance and stability, especially in environments with specific memory requirements or constraints.
资料来源:
[2] Overcommit Accounting - The Linux Kernel Archives, https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
[3] Linux Overcommit Modes | Baeldung on Linux, https://www.baeldung.com/linux/overcommit-modes
[4] Linux memory overcommit details - Stack Overflow, https://stackoverflow.com/questions/19148296/linux-memory-overcommit-details
(rasa) unitree@ubuntu:~/rasa_ws$ rasa run --enable-api /home/unitree/rasa/lib/python3.8/site-packages/rasa/core/tracker_store.py:1044: MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9) Base: DeclarativeMeta = declarative_base() /home/unitree/rasa/lib/python3.8/site-packages/rasa/shared/utils/validation.py:134: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources /home/unitree/rasa/lib/python3.8/site-packages/pkg_resources/__init__.py:3117: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /home/unitree/rasa/lib/python3.8/site-packages/pkg_resources/__init__.py:3117: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /home/unitree/rasa/lib/python3.8/site-packages/sanic_cors/extension.py:39: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. SANIC_VERSION = LooseVersion(sanic_version) Traceback (most recent call last): File "/home/unitree/rasa/lib/python3.8/site-packages/sklearn/__check_build/__init__.py", line 48, in <module> from ._check_build import check_build # noqa ImportError: /home/unitree/rasa/lib/python3.8/site-packages/sklearn/__check_build/../../scikit_learn.libs/libgomp-d22c30c5.so.1.0.0: cannot allocate memory in static TLS block During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/unitree/rasa/bin/rasa", line 8, in <module> sys.exit(main()) File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/__main__.py", line 133, in main cmdline_arguments.func(cmdline_arguments) File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/cli/run.py", line 93, in run rasa.run(**vars(args)) File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/api.py", line 36, in run import rasa.core.run File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/core/run.py", line 25, in <module> from rasa import server, telemetry File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/server.py", line 59, in <module> from rasa.core.agent import Agent File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/core/agent.py", line 22, in <module> from rasa.core.policies.policy import PolicyPrediction File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/core/policies/policy.py", line 26, in <module> from rasa.core.featurizers.tracker_featurizers import TrackerFeaturizer File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/core/featurizers/tracker_featurizers.py", line 31, in <module> from rasa.core.featurizers.single_state_featurizer import SingleStateFeaturizer File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/core/featurizers/single_state_featurizer.py", line 8, in <module> from rasa.nlu.extractors.extractor import EntityTagSpec File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/nlu/extractors/extractor.py", line 30, in <module> import rasa.utils.train_utils File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/utils/train_utils.py", line 32, in <module> from rasa.utils.tensorflow.data_generator import RasaBatchDataGenerator File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/utils/tensorflow/data_generator.py", line 10, in <module> from rasa.utils.tensorflow.model_data import RasaModelData, Data, FeatureArray File "/home/unitree/rasa/lib/python3.8/site-packages/rasa/utils/tensorflow/model_data.py", line 21, in <module> from sklearn.model_selection import train_test_split File "/home/unitree/rasa/lib/python3.8/site-packages/sklearn/__init__.py", line 81, in <module> from . import __check_build # noqa: F401 File "/home/unitree/rasa/lib/python3.8/site-packages/sklearn/__check_build/__init__.py", line 50, in <module> raise_build_error(e) File "/home/unitree/rasa/lib/python3.8/site-packages/sklearn/__check_build/__init__.py", line 31, in raise_build_error raise ImportError( ImportError: /home/unitree/rasa/lib/python3.8/site-packages/sklearn/__check_build/../../scikit_learn.libs/libgomp-d22c30c5.so.1.0.0: cannot allocate memory in static TLS block ___________________________________________________________________________ Contents of /home/unitree/rasa/lib/python3.8/site-packages/sklearn/__check_build: __pycache__ setup.py __init__.py _check_build.cpython-38-aarch64-linux-gnu.so ___________________________________________________________________________ It seems that scikit-learn has not been built correctly. If you have installed scikit-learn from source, please do not forget to build the package before using it: run `python setup.py install` or `make` in the source directory. If you have used an installer, please check that it is suited for your Python version, your operating system and your platform. 这是我出现的错误
07-14
一、单选题(每题3分,共计30分) 1、Centos 中修改文件或目录的访问权限( B )命令 A.chown B.passwd C.chmod D.clear 2、在文件属性中,文件的权限用第( A )列字符表示 A.1-5 B.2-10 C.3-9 D.1-8 3、Hadoop 安装在一台计算机上,需修改相应的配置文件,用一台计算机模拟多台主机的集群是( B )模式 A.全分布模式 B.伪分布模式 C.单机模式 D.全分布 HA 模式 4、配置 Hadoop 环境变量修改(B )文件 A.vi /etc/profile B.vi /etc/profiles C.vi /etc/hosts D.vi ~/input/data 5、配置 SSH 服务需配置(B )文件 A.vi /etc/ssh/sshd B.vi /etc/ssh/sshd_config C.vi /etc/sysconfig/network-scripts/ifcfg-eth0 D.vi ~/.bash_profile 6、hadoop2.0 与 hadoop1.0 区别( D ) A.增加 MapReduce2 B.增加 YARN C.增加 HDFS2 D.增加容错机制 7、在 hadoop 配置中 core-site.xml 的配置是( B)参数 A.集群全局参数 B.HDFS 参数 C.Mapreduce 参数 D.集群资源管理系统参数 8、在 hadoop 配置中 yarn-site.xml 作用是(B) A.用于定义系统级别的参数 B.用于名称节点和数据节点的存放位置 C.用于配置 JobHistory Server 和应用程序参数 D.配置 ResourceManager,NodeManager 的通信端口 9、如没有配置 hadoop.tmp.dir 参数,此时系统默认的临时目录为(D ) A./tmp/hadoop-hadoop B./usr/hadoop-hadoop C./usr/local/hadoop-hadoop D./usr/local/src/hadoop-hadoop 10、以下哪个文件是能够修改 HDFS 的副本保存数量参数(D) A.修改 mapred-site.xml 配置文件 B.修改 core-site.xml 配置文件 C.修改 yarn-site.xml 配置文件 D.修改 hdfs-site.xml 配置文件 得分 二、多选题(每题4分,共20分) 1、以下哪些是 Linux 的特点 A.开放源代码的程序软件,可自由修改 B.与 Unix 系统兼容,具备几乎所有 Unix 的优秀特性 C.可自由传播,收费使用,无任何商业化版权制约 D.适合 Intel 等 x86 CPU 系列架构的计算机 2、以下哪些是 Linux 操作系统组成部分 A.Linux 内核 B.GNU 软件及系统 C.必要的应用程序 D.CPU 3、以下哪些组件属于 Hadoop 生态圈 A.MySQL B.Hive C.HDFS D.Sqoop 4、Hadoop 的优点有哪些 A.扩容能力强 B.成本低 C.高效率 D.高可靠性 5、以下选项中哪些是 SSH 涉及到的两个验证方式 A.主机验证 B.用户身份验证 C.文件系统验证 D.操作系统验证
06-29
你运行了类似 `dpkg -L libcrypt-dev` 或 `find /usr -name "*libcrypt*"`,得到了如下输出: ```text /usr/share/doc/libcryptsetup12 /usr/share/doc/libcrypt-dev /usr/share/doc/libcrypt1 /usr/share/bug/libcryptsetup12 /usr/lib/x86_64-linux-gnu/libcrypto.so /usr/lib/x86_64-linux-gnu/pkgconfig/libcrypt.pc /usr/lib/x86_64-linux-gnu/pkgconfig/libcrypto.pc /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 /usr/lib/x86_64-linux-gnu/libcrypt.so /usr/lib/x86_64-linux-gnu/libcryptsetup.so.12 /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0 /usr/lib/x86_64-linux-gnu/libcrypt.so.1 /usr/lib/x86_64-linux-gnu/libcrypt.a /usr/lib/x86_64-linux-gnu/libcryptsetup.so.12.5.0 /usr/lib/x86_64-linux-gnu/libcrypto.a ``` --- ## ✅ 分析:这些文件说明了什么? ### 🔹 1. `libcrypt` 相关文件(核心 C 加密库) | 文件 | 含义 | |------|------| | `/usr/lib/x86_64-linux-gnu/libcrypt.so` | 动态链接库符号链接(指向最新版本) | | `/usr/lib/x86_64-linux-gnu/libcrypt.so.1` | 主版本符号链接 | | `/usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0` | 实际的动态库文件(实现 `crypt()` 函数) | | `/usr/lib/x86_64-linux-gnu/libcrypt.a` | 静态库文件(用于静态链接) | | `/usr/share/doc/libcrypt-dev/` | 开发文档 | | `/usr/lib/x86_64-linux-gnu/pkgconfig/libcrypt.pc` | pkg-config 配置文件,帮助编译器自动找到头文件和库路径 | ✅ **结论**: - 你的系统已经安装了 `libcrypt-dev` 和 `libcrypt1`。 - 并且是 **x86_64 架构**下的版本(即主机本地编译用的)。 --- ### 🔹 2. `libcrypto.so.*` 是 OpenSSL 的库 ❌ ⚠️ 注意!这不是 `libcrypt`! | 文件 | 所属包 | |------|-------| | `libcrypto.so`, `libcrypto.a`, `libcrypto.pc` | 来自 **OpenSSL**,提供 AES、SHA、RSA 等加密功能 | | `libcrypt.so` | 来自 **glibc or libcrypt**,仅提供传统的密码哈希函数如 `crypt()` | 📌 常见混淆点: - `libcrypt` ≠ `libcrypto` - `crypt()` 函数来自 `libcrypt` - `EVP_encrypt`, `HMAC` 等来自 `libcrypto`(OpenSSL) --- ### 🔹 3. `libcryptsetup.so.*` 是 LUKS/dm-crypt 工具库 - 提供对磁盘加密(如 LUKS)的支持 - 属于 `libcryptsetup` 包 - 与 `crypt()` 函数无关 --- ## 🚫 关键问题:你在做交叉编译 aarch64,但只有 x86_64 的库! 虽然你有: ``` /usr/lib/x86_64-linux-gnu/libcrypt.a ``` 但这只能用于编译 **x86_64 目标程序**。 而你使用的是: ```bash aarch64-linux-gnu-gcc ``` 所以链接器会去寻找: ``` /usr/lib/aarch64-linux-gnu/libcrypt.a ← 缺失! ``` 如果你没有这个路径下的库文件,就会出现: ``` undefined reference to `crypt' ``` 即使你加了 `-lcrypt`,也无法链接成功。 --- ## ✅ 如何验证是否缺少 aarch64 版本的库? 运行: ```bash ls /usr/lib/aarch64-linux-gnu/libcrypt* ``` 预期结果: - 如果存在:输出类似 `libcrypt.so`, `libcrypt.a` - 如果不存在:`No such file or directory` → 表示缺失目标架构库 --- ## ✅ 解决方案:安装 aarch64 架构的 libcrypt 开发包 ### Step 1: 确保已启用 arm64 架构 ```bash sudo dpkg --add-architecture arm64 ``` ### Step 2: 更换为支持 arm64 的 APT 源(如前文所述) 编辑 `/etc/apt/sources.list`,使用 `ports.ubuntu.com` 或阿里云镜像: ```bash sudo nano /etc/apt/sources.list ``` 内容替换为: ```text deb http://ports.ubuntu.com/ubuntu-ports/ focal main universe deb http://ports.ubuntu.com/ubuntu-ports/ focal-updates main universe deb http://ports.ubuntu.com/ubuntu-ports/ focal-security main universe deb http://ports.ubuntu.com/ubuntu-ports/ focal-backports main universe ``` ### Step 3: 更新并安装交叉开发库 ```bash sudo apt update sudo apt install libcrypt-dev:arm64 ``` > ✅ 这个命令会自动安装 aarch64 架构所需的头文件和库(位于 `/usr/lib/aarch64-linux-gnu/`) --- ## ✅ 验证安装成功 ```bash ls /usr/lib/aarch64-linux-gnu/libcrypt* ``` 你应该看到: ``` /usr/lib/aarch64-linux-gnu/libcrypt.so /usr/lib/aarch64-linux-gnu/libcrypt.a /usr/lib/aarch64-linux-gnu/libcrypt.so.1 /usr/lib/aarch64-linux-gnu/libcrypt.so.1.1.0 ``` 同时检查: ```bash pkg-config --cross-prefix=aarch64-linux-gnu- --libs libcrypt ``` 输出应为: ``` -lcrypt ``` 表示可以正确识别。 --- ## ✅ 回到 Nginx 编译 现在你可以重新配置并编译: ```bash cd nginx-1.18.0 make clean rm -rf objs/ ./configure \ --crossbuild=linux:aarch64 \ --with-cc=aarch64-linux-gnu-gcc \ --with-ld-opt="-lcrypt" \ --prefix=/home/thn/nginx-aarch64 \ ... # 其他选项 make -j$(nproc) ``` ✔️ 此时应该不再报 `undefined reference to 'crypt'` 错误。 ---
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

夜流冰

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值