trick---减少存储稀疏矩阵所需内存

最新推荐文章于 2025-05-24 13:28:00 发布

原创最新推荐文章于 2025-05-24 13:28:00 发布 · 815 阅读

0 ·

CC 4.0 BY-SA版权

机器学习专栏收录该内容

6 篇文章

订阅专栏

本文介绍了一种有效减少内存使用的方法，通过将数据集转换为稀疏格式，显著降低了内存占用，适用于内存资源有限的情况。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import numpy as np
import pandas as pd

# This trick will enable those participants with less than 16GB of RAM to effectively
# iterate on Santander competition. The code below shows how to save around 2GB of RAM
# for each copy of test data in the ipython notebook.

test_df = pd.read_csv('../input/test.csv', index_col='ID')
test_size_mb = test_df.memory_usage().sum() / 1024 / 1024
print("Test memory size: %.2f MB" % test_size_mb)
# Test memory size: 1879.24 MB

test_df_sparse = test_df.replace(0, np.nan).to_sparse()
test_sparse_size_mb = test_df_sparse.memory_usage().sum() / 1024 / 1024
print("Test sparse memory size: %.2f MB" % test_sparse_size_mb)
# Test sparse memory size: 26.78 MB