當 mysqdump 遇到 Out of memory

MySQL 的資料量太大, 在使用 mysqldump的時後遇到 Out Of Memory 的時後該怎 辦?

有以下兩種做法:

1. 設定 my.cnf, 設定以下參數, max_allowed_packet=16M 可隨機再調大.

[mysqldump] quick max_allowed_packet = 16M

2. 使用 -q (--quick) 的參數, 使用此參數就不會使用到 Memory.

使用範例: mysqldump -q -u root DB_NAME > DB_NAME.sql

原廠的說明是如下:

--quick, -q

This option is useful for dumping large tables. It forces mysqldump to retrieve rows for a table from the server a row at a time rather than retrieving the entire row set and buffering it in memory before writing it out.

### Causes of TCP Socket OutOfMemory Errors TCP socket-related `OutOfMemory` errors typically occur due to improper management of network resources, excessive connection creation, or misconfigured system limits. These issues can manifest in various ways, such as an overwhelming number of open sockets, inefficient memory allocation for each connection, or failure to release resources after use. One key factor is the maximum memory limit set for TCP sockets within the kernel. When this threshold is reached, new packets may be dropped until memory usage decreases[^1]. This behavior suggests that the system cannot dynamically scale its memory allocation for network operations beyond a certain point without manual intervention. Another potential cause involves the accumulation of `TIME-WAIT` connections. While these connections do not consume significant memory individually, they can accumulate rapidly under high load, especially when thousands of short-lived connections are established per second. Although their overhead is minimal, it's still worth monitoring if the system lacks sufficient memory to handle concurrent data transfers efficiently[^2]. In environments where virtual machines (VMs) are involved, resource contention between services running on the same host—such as message brokers like RabbitMQ and compute nodes like Nova-Compute—can lead to unexpected outages or performance degradation. A VM consuming 100% CPU or large amounts of memory can destabilize critical services, indirectly causing network stack failures and memory exhaustion[^3]. --- ### Solutions and Best Practices To mitigate `OutOfMemory` errors related to TCP sockets, consider implementing the following strategies: #### 1. Adjust Kernel Memory Limits Ensure that the system allows enough memory for TCP sockets by tuning parameters such as `net.ipv4.tcp_rmem` and `net.ipv4.tcp_wmem`. These control receive and send buffer sizes respectively. Increasing them can help accommodate more simultaneous connections without exhausting memory. Example: ```bash sysctl -w net.ipv4.tcp_rmem="4096 87380 6291456" sysctl -w net.ipv4.tcp_wmem="4096 87380 6291456" ``` Persistent settings should be added to `/etc/sysctl.conf`. #### 2. Optimize Connection Handling Reduce the impact of `TIME-WAIT` states by enabling `SO_REUSEADDR` and `SO_LINGER`, which allow faster reuse of ports and graceful closure of connections. Additionally, using keep-alive mechanisms can reduce the frequency of connection setup/teardown cycles. #### 3. Monitor Resource Utilization Use tools like `mpstat`, `vmstat`, and `sar` to monitor CPU and memory usage accurately, especially in multi-core systems. Avoid placing heavy workloads on the same physical machine as critical messaging or coordination services to prevent resource starvation[^3]. #### 4. Tune TCP Stack Parameters Adjusting parameters like `net.ipv4.tcp_max_tw_buckets` can help control the number of `TIME-WAIT` sockets kept in memory. Setting reasonable timeouts with `net.ipv4.tcp_fin_timeout` ensures that connections transition out of `TIME-WAIT` state faster. Example: ```bash sysctl -w net.ipv4.tcp_max_tw_buckets=200000 sysctl -w net.ipv4.tcp_fin_timeout=15 ``` #### 5. Use Efficient Data Types and Buffer Management When handling network data, ensure that buffers are properly sized and released after use. Using appropriate numeric types (e.g., `int`, `float`, or `double`) based on the required precision and performance needs helps avoid unnecessary memory consumption[^4]. #### 6. Implement Backpressure and Throttling For applications handling high volumes of incoming or outgoing traffic, implement backpressure mechanisms to slow down data ingestion when system resources approach critical levels. This prevents sudden spikes from overwhelming the available memory. #### 7. Employ Load Balancing and Horizontal Scaling Distribute incoming connections across multiple instances or nodes to avoid overloading any single machine. This reduces the risk of hitting per-node memory limits and improves overall reliability. ---
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值