<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[LLMUZI123456789的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/LLMUZI123456789</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; LLMUZI123456789]]></copyright><item><title><![CDATA[spark本地模式基础配置流程]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/158736518</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/158736518</guid><author>LLMUZI123456789</author><pubDate>Fri, 06 Mar 2026 14:41:45 +0800</pubDate><description><![CDATA[3、创建专门的环境变量文件，配置文件。1、创建新用户，授予sudo权限。2、创建文件目录，上传解压文件。5、将文件目录授权给指定用户。6、检查主要组件是否安装完成。]]></description><category></category></item><item><title><![CDATA[sklearn中fit、transform、fit_transform用法详解]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/156762916</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/156762916</guid><author>LLMUZI123456789</author><pubDate>Fri, 09 Jan 2026 14:27:13 +0800</pubDate><description><![CDATA[训练阶段（训练集）# 测试/预测阶段（测试集/新数据）# 如果有新数据需要预测记忆技巧fit = 学习规则（只在训练集做一次）transform = 应用规则（训练集、测试集都要做）fit_transform = fit + transform（训练集的快捷方式）记住这个原则：训练集可以了解数据，测试集只能应用从训练集学到的规则。]]></description><category></category></item><item><title><![CDATA[Spark Executor 与 Driver 在三种模式下的区别]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/156196554</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/156196554</guid><author>LLMUZI123456789</author><pubDate>Tue, 23 Dec 2025 15:02:33 +0800</pubDate><description><![CDATA[【代码】Spark Executor 与 Driver 在三种模式下的区别。]]></description><category></category></item><item><title><![CDATA[spark集群文件分发问题]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/155574506</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/155574506</guid><author>LLMUZI123456789</author><pubDate>Thu, 04 Dec 2025 16:46:29 +0800</pubDate><description><![CDATA[将所有节点挂载同一个网络存储，代码中指向挂载点（例如 /mnt/data/train.csv）有一份csv数据在当前节点的本地路径上，本地模式跑可以通过：file://+本地路径正常读取。这是生产环境最标准的方式，将数据上传到HDFS上，让所有节点通过同一地址访问得到该数据；2、无权限上传hdfs时，使用 --files + SparkFiles.get()1、使用HDFS路径，将数据上传到hdfs。脚本使用：SparkFiles.get()3、使用共享文件系统（如NFS）路径。为何不使用广播变量方式？]]></description><category></category></item><item><title><![CDATA[conda打包环境上传spark集群]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/155060458</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/155060458</guid><author>LLMUZI123456789</author><pubDate>Thu, 20 Nov 2025 11:31:45 +0800</pubDate><description><![CDATA[当本地训练的python环境与 Spark 集群其他节点环境不一致时，核心解决方案是：将本地 Python 虚拟环境打包为压缩包，通过 Spark 的–archives参数分发到所有 Executor 节点，强制所有节点使用统一的环境（Python 解释器 + 依赖包）环境打包：将本地虚拟环境（含 Python 解释器、所有依赖包如scikit-learn、pyarrow、自定义类脚本）压缩为归档文件（.zip或.tar.gz）。-n sklearn-spark-env：指定要打包的 conda 环境名；]]></description><category></category></item><item><title><![CDATA[conda环境打包好迁移]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/154944453</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/154944453</guid><author>LLMUZI123456789</author><pubDate>Mon, 17 Nov 2025 12:33:22 +0800</pubDate><description><![CDATA[方法一：使用 conda-pack（推荐，适合离线 / 复杂环境）]]></description><category></category></item><item><title><![CDATA[每日shell脚本day1]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/154683022</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/154683022</guid><author>LLMUZI123456789</author><pubDate>Tue, 11 Nov 2025 10:24:32 +0800</pubDate><description><![CDATA[【代码】每日shell脚本。]]></description><category></category></item><item><title><![CDATA[git 提交代码命令]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/152039916</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/152039916</guid><author>LLMUZI123456789</author><pubDate>Wed, 24 Sep 2025 14:43:51 +0800</pubDate><description><![CDATA[Git提交代码的基本流程包括：1) git status查看修改状态；2) git add暂存修改文件；3) git commit -m提交到本地仓库；4) git push推送至远程仓库。首次推送需关联远程分支，后续可直接推送。建议先拉取最新代码避免冲突，提交说明应简洁清晰。]]></description><category></category></item><item><title><![CDATA[pyspark自定义udf函数]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/151757830</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/151757830</guid><author>LLMUZI123456789</author><pubDate>Tue, 16 Sep 2025 15:37:24 +0800</pubDate><description><![CDATA[在 PySpark 中，UDF（User-Defined Function，用户自定义函数） 是扩展 Spark 功能的核心工具，用于处理内置函数（如pyspark.sql.functions中的函数）无法覆盖的自定义逻辑（如复杂字符串处理、自定义数值计算、多列联动计算等）。仅当内置函数无法满足需求时才用 UDF。Pandas UDF（Vectorized UDF）：基于Apache Arrow批量处理数据，直接操作 Pandas Series/DataFrame，性能比普通 UDF 提升 5-10 倍。]]></description><category></category></item><item><title><![CDATA[win系统配置定时任务]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/151362202</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/151362202</guid><author>LLMUZI123456789</author><pubDate>Tue, 09 Sep 2025 15:18:55 +0800</pubDate><description><![CDATA[1、按下 Win + R 键，输入 taskschd.msc，然后按回车键。]]></description><category></category></item><item><title><![CDATA[win11家庭版配置远程桌面]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/151361741</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/151361741</guid><author>LLMUZI123456789</author><pubDate>Tue, 09 Sep 2025 15:06:40 +0800</pubDate><description><![CDATA[将文件复制到：C:\Program Files\RDP Wrapper，覆盖原来的文件，重启电脑即可对电脑进行远程桌面控制。1、下载安装下面的两个压缩包。]]></description><category></category></item><item><title><![CDATA[spark连接mongodb]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/151357583</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/151357583</guid><author>LLMUZI123456789</author><pubDate>Tue, 09 Sep 2025 11:54:16 +0800</pubDate><description><![CDATA[【代码】spark连接mongodb。]]></description><category></category></item><item><title><![CDATA[python并发任务 executor.map和submit]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/148947852</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/148947852</guid><author>LLMUZI123456789</author><pubDate>Fri, 27 Jun 2025 10:52:28 +0800</pubDate><description><![CDATA[本文对比了Python中ThreadPoolExecutor的submit和map两种并发任务执行方法。submit方法是非阻塞的，通过Future对象异步获取结果，适合复杂场景；map方法按提交顺序返回结果，操作更简单。文章通过代码示例展示了两种方法的基本用法，包括单参数和多参数任务的处理方式，并演示了如何用as_completed异步获取submit结果。两者主要区别在于：submit更灵活但需手动管理Future，map操作简单但结果顺序固定。根据任务复杂度选择合适的方法可以提高并发编程效率。]]></description><category></category></item><item><title><![CDATA[sklearn自定义pipeline的数据处理]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/147816238</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/147816238</guid><author>LLMUZI123456789</author><pubDate>Fri, 09 May 2025 11:02:56 +0800</pubDate><description><![CDATA[【代码】sklearn自定义pipeline的数据处理。]]></description><category></category></item><item><title><![CDATA[pandas写入、读取mysql数据]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/141751886</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/141751886</guid><author>LLMUZI123456789</author><pubDate>Sat, 31 Aug 2024 14:03:53 +0800</pubDate><description><![CDATA[2、从mysql读取数据。]]></description><category></category></item><item><title><![CDATA[Linux快速安装MySQL8]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/141503783</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/141503783</guid><author>LLMUZI123456789</author><pubDate>Sat, 24 Aug 2024 19:45:07 +0800</pubDate><description><![CDATA[简单记录下如何快速安装MySQL8。1、检查是否安装过其他版本。2、更新密钥、安装、配置。]]></description><category></category></item><item><title><![CDATA[LLM大模型本地部署与预训练微调]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/140013884</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/140013884</guid><author>LLMUZI123456789</author><pubDate>Thu, 27 Jun 2024 16:04:29 +0800</pubDate><description><![CDATA[模型微调过程会使用CPU，且占用内存大概22G左右；参照官方文档：https://github.com/QwenLM/Qwen/blob/main/README_CN.md。创建环境，建议单独创建一个新环境，避免与现有环境冲突，创建后要先安装pytorch，再安装项目依赖模块。提前安装好git跟git lfs，否则可能拉取不到模型文件，git lfs主要用于大文件拉取。项目文件以及环境安装好后，继续拉取模型文件，由于内存有限，拉取1.8B即可。1、环境条件：Linux 24G内存左右。]]></description><category></category></item><item><title><![CDATA[python前端streamlit模型部署]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/139329202</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/139329202</guid><author>LLMUZI123456789</author><pubDate>Thu, 30 May 2024 17:06:01 +0800</pubDate><description><![CDATA[【代码】python前端streamlit模型部署。]]></description><category></category></item><item><title><![CDATA[Pipeline快速构建机器学习模型框架]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/139325834</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/139325834</guid><author>LLMUZI123456789</author><pubDate>Thu, 30 May 2024 15:39:53 +0800</pubDate><description><![CDATA[构建机器学习模型过程中，往往会涉及很多步骤：数据处理、特征构造、特征筛选、算法选取等等；]]></description><category></category></item><item><title><![CDATA[Linux（centos7）部署spark]]></title><link>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/137334188</link><guid>https://blog.youkuaiyun.com/LLMUZI123456789/article/details/137334188</guid><author>LLMUZI123456789</author><pubDate>Wed, 03 Apr 2024 14:32:47 +0800</pubDate><description><![CDATA[Spark部署模式主要有4种：Local模式（单机模式）、Standalone模式（使用Spark自带的简单集群管理器）、Spark On Yarn模式（使用YARN作为集群管理器）和Spark On Mesos模式（使用Mesos作为集群管理器）。下面介绍Local模式（单机模式）、跟Spark On Yarn模式（使用YARN作为集群管理器）的简单部署。spark on local模式以及spark on yarn模式部署完成。选择想要安装的版本下载，然后通过Linux客户端Xshell等上传到主机。]]></description><category></category></item></channel></rss>