<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[m0_37754282的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/m0_37754282</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; m0_37754282]]></copyright><item><title><![CDATA[Big dataTechnology foundation 概念选择题]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/111177638</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/111177638</guid><author>m0_37754282</author><pubDate>Mon, 14 Dec 2020 20:28:52 +0800</pubDate><description><![CDATA[I
Note: There may be more than one correct answer.
(1) The characteristics of big data include:
(1) A B C
A. Volume
B. Variety
C. Velocity D. Simplicity
(2) The characteristic of cloud computing solutions include: A. Dynamic provisioning
(2) A B C
B. Scala]]></description><category></category></item><item><title><![CDATA[RDD用法与实例（十四）：closure和accumulators的区别和实例]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/111143826</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/111143826</guid><author>m0_37754282</author><pubDate>Sun, 13 Dec 2020 19:52:25 +0800</pubDate><description><![CDATA[1.RDD的特性：
1.persistent
2.lazy transformation
2.Cluster mode集群模式
Only one master/worker can run on the same machine, but a machine can be both a master and a worker

3.where to run
Most run on drivers
transformations run on executors
actions - executors an.]]></description><category></category></item><item><title><![CDATA[RDD用法与实例（十三）：spark与MapReduce的对比]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/111143583</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/111143583</guid><author>m0_37754282</author><pubDate>Sun, 13 Dec 2020 19:05:31 +0800</pubDate><description><![CDATA[Hadoop MapReduce：
 只用磁盘，只用MapReduce方法，batch模型，只支持Java。
spark
内存或磁盘，很多方法，BIS（batch，interactive，streaming），支持多种语言

]]></description><category></category></item><item><title><![CDATA[spark大数据基础复习：一些概念]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/111071567</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/111071567</guid><author>m0_37754282</author><pubDate>Sun, 13 Dec 2020 16:31:57 +0800</pubDate><description><![CDATA[1.大数据的定义

“Big Data” is data whose scale, complexity, and speed require new architecture, techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it…

————————————————————————————————————
2.大数据的特征Characteristics of Bi]]></description><category></category></item><item><title><![CDATA[图像处理：Lossless compression/Huffman coding/霍夫曼编码计算题]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/111069419</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/111069419</guid><author>m0_37754282</author><pubDate>Sat, 12 Dec 2020 16:25:07 +0800</pubDate><description><![CDATA[注意：编码时候挑概率大的分支设为1
计算过程：



]]></description><category></category></item><item><title><![CDATA[图像处理之Adaboost计算实例]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/111068366</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/111068366</guid><author>m0_37754282</author><pubDate>Sat, 12 Dec 2020 16:15:15 +0800</pubDate><description><![CDATA[想起来的时候只保存了这一题。。不过也没有答案

]]></description><category></category></item><item><title><![CDATA[第六章：图像处理之图像分割Image segmentation]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/111027514</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/111027514</guid><author>m0_37754282</author><pubDate>Fri, 11 Dec 2020 14:35:59 +0800</pubDate><description><![CDATA[outline
– Point, line and edge detection (Prewitt, Sobel and Laplacian of Gaussian)
– Hough transform, e.g., straight line, circle, etc
– Thresholding (Global and Adaptive)
– Statistical mixture model
– Expectation-maximisation method
– Morphological Water]]></description><category></category></item><item><title><![CDATA[第四章：图像处理之图像恢复与滤波Image Restoration and Filtering]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/110953073</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/110953073</guid><author>m0_37754282</author><pubDate>Fri, 11 Dec 2020 01:43:50 +0800</pubDate><description><![CDATA[Image Restoration and Filtering
• Image restoration and filtering
– Random noise distributions and parameter estimation
– Periodic noise
– Arithmetic and Geometric mean filters
– Harmonic and Contraharmonic mean filters
– Order-statistics filters
• Median,]]></description><category></category></item><item><title><![CDATA[第三章：图像处理之频域图像增强Frequency domain image enhancement]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/110948192</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/110948192</guid><author>m0_37754282</author><pubDate>Thu, 10 Dec 2020 14:17:37 +0800</pubDate><description><![CDATA[• Frequency domain image enhancement
– Discrete Fourier transform and inverse
– Notch filter
– Low-pass filtering, e.g., Ideal, Butterworth
and Gaussian
– Image power
– High-pass filtering, e.g., Ideal, Butterworth and Gaussian
————————————————————————————]]></description><category></category></item><item><title><![CDATA[第二章：图像处理之空域图像增强Spatial domain image enhancement]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/110940214</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/110940214</guid><author>m0_37754282</author><pubDate>Wed, 09 Dec 2020 21:11:03 +0800</pubDate><description><![CDATA[outline
• Spatial domain image enhancement
– Intensity transformation functions • Inverse or Image Negatives
• Brightening, e.g., log
• Darkening, e.g., nth power
• Power-law transformation • Intensity normalization
– Histogram Equalization • Contrast stre]]></description><category></category></item><item><title><![CDATA[第一章：图像处理之图像表示Image Representation]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/110940071</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/110940071</guid><author>m0_37754282</author><pubDate>Wed, 09 Dec 2020 20:34:42 +0800</pubDate><description><![CDATA[outline
• Image representation
– Pixel = picture element
– Sampling
– Quantization
1.Pixel

Individual elements are called:
image elements, picture elements (pixels), or image points.
2.sampling
强度值的采样率是图像分辨率。Sampling rate on intensity values is the image ]]></description><category></category></item><item><title><![CDATA[第八章：图像处理之图像压缩ImageCompression]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/110489544</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/110489544</guid><author>m0_37754282</author><pubDate>Fri, 04 Dec 2020 15:33:40 +0800</pubDate><description><![CDATA[Outline
• Introduction
• Fundamentals
• Coding, interpixel and psychovisual redundancies • Fidelity criteria
• Image compression model
• Source encoder and decoder
• Channel encoder and decoder
• Lossless compression
• Huffman encoding
• Lossless predictiv]]></description><category></category></item><item><title><![CDATA[pyspark报错 java.lang.ClassNotFoundException: org.graphframes.GraphFramePythonAPI]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/110086095</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/110086095</guid><author>m0_37754282</author><pubDate>Tue, 24 Nov 2020 16:38:56 +0800</pubDate><description><![CDATA[在这里我用的是spark3.0 + scala2.12版本
首先在cmd上启动pyspark
这里有一个小度量，第一次使用参数启动pyspark，以便它下载所有graphframe的jar依赖项，很多教程启动的时候并没有指定依赖包，这可能会发生错误： （根据你的spark版本去graphframe官网找到对应的下载命令）
官网链接：graphframes
比如我下载对应的0.8.0-spark3.0-s_2.12 后，将它放入spark启动时对应的文件目录下
在终端输入
pyspark --packages]]></description><category></category></item><item><title><![CDATA[使用DGL环境报错：Setting the default backend to “pytorch“,BACKEND can be chosen from mxnet, pytorch, tensor]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/109431809</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/109431809</guid><author>m0_37754282</author><pubDate>Sun, 01 Nov 2020 17:46:55 +0800</pubDate><description><![CDATA[在用python处理图神经网络的时候，常用到DGL，由于改之前忘记截图保存整个错误描述，只记得标题的关键词大意：要将一个dgl文件夹下的json文件的backend默认值设置成pytorch，打开看了之后它默认值就是pytorch很疑惑。
随便试了一下环境配置文件中最后加了一行
export GLBACKEND="pytorch"

保存即可成功。
另外附一下mac系统改环境变量的方法：
在终端输入


记得后缀选择.bash_profile

...]]></description><category></category></item><item><title><![CDATA[RDD用法与实例（十二）：实现PageRank]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/109291861</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/109291861</guid><author>m0_37754282</author><pubDate>Mon, 26 Oct 2020 22:41:19 +0800</pubDate><description><![CDATA[import re
from operator import add

def computeContribs(urls, rank):
    # Calculates URL contributions to the rank of other URLs.
    num_urls = len(urls)
    for url in urls:
        yield (url, rank / num_urls)

def parseNeighbors(urls):
    # Parses a ]]></description><category></category></item><item><title><![CDATA[sparksql学习笔记（一）：基本操作]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/109258054</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/109258054</guid><author>m0_37754282</author><pubDate>Mon, 26 Oct 2020 16:03:24 +0800</pubDate><description><![CDATA[1.1 读入数据
df = spark.read.csv('building.csv', header=True, inferSchema=True)

1.2 展示数据
# show the content of the dataframe
df.show()


1.3 展示数据类型
df.printSchema()


2.1 dataframe创建rdd型数据库
dfrdd = df.rdd
dfrdd.take(3)  #展示前三组，看看是什么样



...]]></description><category></category></item><item><title><![CDATA[machine learning学习笔记（四）逻辑回归]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/108903222</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/108903222</guid><author>m0_37754282</author><pubDate>Mon, 05 Oct 2020 15:40:43 +0800</pubDate><description><![CDATA[在前面已经学习了线性回归，但是线性回归的应用场景往往是，比如：知道你的身高，你的收入，你的颜值分，你的家底，问你未来能交到的对象的总体评分是多少。
我们用离散的数据构造了一个连续的模型，并以此把重心放在“预测数值”上。但是当我们遇到：已知西瓜的颜色，重量，花纹等信息，问是不是好瓜？
答案必然是：yes / no.
所以逻辑回归就引入了这样的问题，虽然叫做回归，但实际却是在解决分类问题。
可是我们拟合出来的图像都是一些线条可怎么进行逻辑性回答呢？
y=1,if x&gt;0
y=0.5 if x=0
y=0 ]]></description><category></category></item><item><title><![CDATA[machine learning学习笔记（三）正则化]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/108902854</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/108902854</guid><author>m0_37754282</author><pubDate>Fri, 02 Oct 2020 17:05:44 +0800</pubDate><description><![CDATA[为什么要正则化？
就是为了解决过拟合问题。
为啥过拟合？
有部分原因就是x1,x2,x3…一大堆特征太多了（一个x代表一个特征）
假设有一个只有两个特征x1,x2的模型算出来的非线性方程是↓

把模型得到的每个项的参数写成一个矩阵w

w0不计入，一个常数，写在外面就行。
没有正则化调整之前的误差方程是:

引入之后：

λ≥0是提前选择的控制强度的超参数。

说人话就是，减重。
给谁减重？
给那些次幂很大，其实没那么重要（甚至导致过拟合）的项加上一个惩罚项来中和它的影响。
假设一组数据集算出来的模型有9次]]></description><category></category></item><item><title><![CDATA[machine learning学习笔记（二）多元线性回归]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/108901702</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/108901702</guid><author>m0_37754282</author><pubDate>Fri, 02 Oct 2020 16:32:36 +0800</pubDate><description><![CDATA[多元线性回归 Linear Regression
线性回归，就是给在平面图上给出一组数据，我想找到一条穿过他们中间的直线，使得每个点到这条直线的距离的和都能最小。
如果是多元线性回归，那么意味着将不再是普通的y=wx+b问题，而是y=w1x1+w2x2+…+b的问题。

但是，拟合必然有损失，我们定义以下损失函数↓
均方误差，顾名思义又算方差，又算平均值。


展开来写：

最后一步实际上是求偏导。要求这个式子的最小值，也就是

他的最小值：

我们在高中学过，求最大最小值，就要求导数=0.
所以对w进行偏]]></description><category></category></item><item><title><![CDATA[machine learning学习笔记（一）：信息熵，条件熵，交叉熵，KL散度，互信息]]></title><link>https://blog.youkuaiyun.com/m0_37754282/article/details/108710306</link><guid>https://blog.youkuaiyun.com/m0_37754282/article/details/108710306</guid><author>m0_37754282</author><pubDate>Fri, 02 Oct 2020 13:58:36 +0800</pubDate><description><![CDATA[琴生不等式 Jensen

由数学归纳法证明
对损失函数 Logarithmic function

entropy 信息熵

log底数一般为2.信息熵代表着X不确定程度。

]]></description><category></category></item></channel></rss>