<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[fitzgerald0的博客]]></title><description><![CDATA[统计，机器学习，算法]]></description><link>https://blog.youkuaiyun.com/fitzgerald0</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; fitzgerald0]]></copyright><item><title><![CDATA[Multi-task LightGBM安装]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/127831002</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/127831002</guid><author>fitzgerald0</author><pubDate>Sun, 13 Nov 2022 13:32:16 +0800</pubDate><description><![CDATA[Multi-task LightGBM安装]]></description><category></category></item><item><title><![CDATA[（时间序列）回归模型融合策略]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/123057793</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/123057793</guid><author>fitzgerald0</author><pubDate>Mon, 21 Feb 2022 22:42:51 +0800</pubDate><description><![CDATA[机器学习回归预测中，存在多个模型预测结果，我们可以选择最优的模型结果 ,也可以组合多个不同模型结果，研究表明,在许多情况下，只需对不同预测方法的预测结果进行平均就可以显著提高预测精度。同时达到最优模型的效果。]]></description><category></category></item><item><title><![CDATA[PySpark与DataFrame简介]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/116455182</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/116455182</guid><author>fitzgerald0</author><pubDate>Thu, 06 May 2021 15:34:54 +0800</pubDate><description><![CDATA[PySpark与DataFrame简介]]></description><category></category></item><item><title><![CDATA[PySpark.ml时间序列特征工程]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/116453472</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/116453472</guid><author>fitzgerald0</author><pubDate>Thu, 06 May 2021 14:43:07 +0800</pubDate><description><![CDATA[PySpark.ml时间序列特征工程]]></description><category></category></item><item><title><![CDATA[PySpark机器学习调参方法]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/116452338</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/116452338</guid><author>fitzgerald0</author><pubDate>Thu, 06 May 2021 13:54:05 +0800</pubDate><description><![CDATA[PySpark.ML调参方法(CrossValidator与TrainValidationSplit)]]></description><category></category></item><item><title><![CDATA[PySpark线性回归与广义线性模型]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/116451185</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/116451185</guid><author>fitzgerald0</author><pubDate>Thu, 06 May 2021 12:31:01 +0800</pubDate><description><![CDATA[本节从原理和代码上讲解销量预测任务中使用到的Spark.ML内置线性回归模型和广义线性模型。]]></description><category></category></item><item><title><![CDATA[Spark.SQL时间序列缺失值填充与异常值处理]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/116306783</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/116306783</guid><author>fitzgerald0</author><pubDate>Fri, 30 Apr 2021 17:14:03 +0800</pubDate><description><![CDATA[PySpark时间序列缺失值填充与异常值修复]]></description><category></category></item><item><title><![CDATA[时序距离度量方式DTW(python)]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/116233369</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/116233369</guid><author>fitzgerald0</author><pubDate>Wed, 28 Apr 2021 14:41:55 +0800</pubDate><description><![CDATA[时序距离度量方式Dynamic Time Warping (DTW) Python代码]]></description><category></category></item><item><title><![CDATA[PySpark时间序列数据统计描述，分布特性与内部特性]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/116090223</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/116090223</guid><author>fitzgerald0</author><pubDate>Sat, 24 Apr 2021 14:29:58 +0800</pubDate><description><![CDATA[为更好的洞察和处理大规模时序数据的特性，本文针对大规模时序数据，从基本统计特性，分布，序列内部检测三方面，提供Spark和借助numpy，scipy，statsmodels封装的成UDF函数脚本与理论讲解]]></description><category></category></item><item><title><![CDATA[销量预测简单模型]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/115918790</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/115918790</guid><author>fitzgerald0</author><pubDate>Tue, 20 Apr 2021 20:58:02 +0800</pubDate><description><![CDATA[销量预测常用简单模型总结，基于python和Spark.SQL]]></description><category></category></item><item><title><![CDATA[PySpark机器学习特征选择]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/115876867</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/115876867</guid><author>fitzgerald0</author><pubDate>Mon, 19 Apr 2021 23:01:04 +0800</pubDate><description><![CDATA[本文基于SPARK.SQL和SPARK.ML实现常见的4种结构化数据特征选择方法，并给出基于树模型的特征选择代码。]]></description><category></category></item><item><title><![CDATA[销量预测建模中常用的损失函数与模型评估指标]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/115471489</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/115471489</guid><author>fitzgerald0</author><pubDate>Tue, 06 Apr 2021 21:49:43 +0800</pubDate><description><![CDATA[本文详细的梳理了销量预测领域常见的损失函数和模型评估指标。]]></description><category></category></item><item><title><![CDATA[PySpark外部包lightgbm的jar依赖文件]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/113838965</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/113838965</guid><author>fitzgerald0</author><pubDate>Wed, 17 Feb 2021 22:36:46 +0800</pubDate><description><![CDATA[PySpark外部包lightgbm的jar依赖文件]]></description><category></category></item><item><title><![CDATA[时间序列可预测性度量]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/108995724</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/108995724</guid><author>fitzgerald0</author><pubDate>Sat, 10 Oct 2020 14:04:31 +0800</pubDate><description><![CDATA[时间序列可预测性度量,讲解从序列长度到平稳性，排序熵等指标]]></description><category></category></item><item><title><![CDATA[k-shape时间序列聚类(tslearn)]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/108188588</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/108188588</guid><author>fitzgerald0</author><pubDate>Sun, 23 Aug 2020 21:15:27 +0800</pubDate><description><![CDATA[时间序列聚类概述


时间序列数据挖掘从技术角度来讲，一般有四种时间序列预测，时间序列分类，时间序列聚类，时间序列异常检测基本上包含了机器学习的几大领域由于时序数据的特殊性，所以每一个方面都与截面数据挖掘有所区别。本文主要讲讲时序聚类中的k-shape算法，和以往的风格一样，重在算法实现。时间序列数据的聚类，关键在于如何定义相似度。
比如基于时间序列特征（len，max,min,std,lag）等可以使用 KMeans算法进行聚类。除了使用欧式距离以外，还可以使用dtw等方法，以及动态规划原理，对时间序列.]]></description><category></category></item><item><title><![CDATA[基于PySpark的销量预测]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/106885568</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/106885568</guid><author>fitzgerald0</author><pubDate>Sun, 21 Jun 2020 15:07:02 +0800</pubDate><description><![CDATA[“ 本文阐述基于Pyspark的sql数据读取、特征处理、寻找最优参数、使用最优参数预测未来销量的全过程，重在预测流程和Pyspark相关知识点的讲解，展示可供企业级开发落地的demo。”]]></description><category></category></item><item><title><![CDATA[PySpark-prophet预测]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/106157008</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/106157008</guid><author>fitzgerald0</author><pubDate>Sat, 16 May 2020 12:01:50 +0800</pubDate><description><![CDATA[简介
Prophet是facebook开源时间序列预测工具,使用时间序列分解与机器学习拟合的方法进行建模预测
。关于prophet模型优点我不打算说，网络上的文章非常多，各种可视化，和参数的解释与demo演示。
但是在正在用到工业上大规模的可供学习的中文材料并不多。
本文打算使用pyspark进行多序列预测建模，会给出一个比较详细的脚本，供交流学习，重点在于使用hive数据/分布式，以及中间数据预处理，以及pandas_udf对多条序列进行循环执行。背景说明，在十万级别的sku序列上使用prophet预测每]]></description><category></category></item><item><title><![CDATA[pyarrow.lib.ArrowInvalid: ‘utf-32-le‘ codec can‘t decode]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/106106663</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/106106663</guid><author>fitzgerald0</author><pubDate>Wed, 13 May 2020 21:12:02 +0800</pubDate><description><![CDATA[使用pyspark的时候定义数据类型
通过@pandas_udf的形式进行装饰的时候发现这个错误
schema = StructType([
    StructField("store_sku", StringType()),
    StructField("ds", StringType()),
    StructField("pro_pred", DoubleType())
])

@pandas_udf(schema, functionType=PandasUDFType.GROUPED_MAP]]></description><category></category></item><item><title><![CDATA[时间序列树模型特征工程汇总]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/104029842</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/104029842</guid><author>fitzgerald0</author><pubDate>Sat, 18 Jan 2020 14:10:02 +0800</pubDate><description><![CDATA[时间序列特征汇总特征一、时间特征特征二、类别mean encoding特征特征三、统计特征特征四：滞后历史特征特征五：高阶特征特征六：外部特征
稍微总结一下，时间序列中的特征，主要针对机器学习树模型，因为是时序数据，所以和寻常的机器学习特征略有不同，比如关注时间特征，滞后特征，滑窗特征等。
特征一、时间特征
import datetime
import pandas as pd
df['ds']=...]]></description><category></category></item><item><title><![CDATA[多元线性模型中共线性产生的原因解析]]></title><link>https://blog.youkuaiyun.com/fitzgerald0/article/details/79381379</link><guid>https://blog.youkuaiyun.com/fitzgerald0/article/details/79381379</guid><author>fitzgerald0</author><pubDate>Mon, 02 Dec 2019 20:05:28 +0800</pubDate><description><![CDATA[在很多书籍中谈到了共线性问题，作为多元统计中基本假设之一，却经常被违背，影响模型稳定性和统计power。在不少的论文中谈到了，国内的很多书籍往往一笔带过。因为，属于统计计算领域内容，非专业人士，也鲜有能明白其中内涵，现依据手上的一些资料和自己的理解，写就一些。

一.共线性概念

共线性问题(collinearity)，是指两个或者更多的自变量高度相关，当只有两个高度相...]]></description><category></category></item></channel></rss>