<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[stupidTomA的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/stupidTomA</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; stupidTomA]]></copyright><item><title><![CDATA[ps参数解析]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/109372869</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/109372869</guid><author>stupidTomA</author><pubDate>Thu, 29 Oct 2020 22:37:56 +0800</pubDate><description><![CDATA[1.查看ps的帮助文档

man ps 命令即可

2.for example



USER: 进程拥有者

PID： pid

%CPU： 占用的cpu百分比

%MEM： 占用的内存使用率

VSZ： 占用的虚拟内存大小

RSS： 占用的内存大小

TTY： 运行的中段号吗

STAT： 进程状态

 D：不可中断 R: 运行中 S： 休眠中 T：暂停中 Z：僵尸进程 W：没有足够的内存可分配

 &lt;：...]]></description><category></category></item><item><title><![CDATA[top参数解析]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/109371522</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/109371522</guid><author>stupidTomA</author><pubDate>Thu, 29 Oct 2020 22:21:14 +0800</pubDate><description><![CDATA[top命令看到的是实时的进程信息



1.第一行 表示 top的刷新使用时间是21：47：46，系统启动时间是3min，当前系统是 2个用户，最近1分钟，五分钟，十五分钟内的平均负载是 0.45， 0.67， 0.32

2.第二行表示 当前系统有177个进程，1个在运行，176个睡眠状态，0个停止，0个僵尸进程

3.第三行是cpu信息，

us 表示当前系统占用的cpu百分比，sy表示内核占用的cpu百分比，ni表示改变过优先级的进程占用的cpu百分比，id表示空闲cpu百分比，wa表示I/O等待百分]]></description><category></category></item><item><title><![CDATA[java.lang.ClassNotFoundException: Class mapreduce.WordcountMapper not found  hadoop]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/108549449</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/108549449</guid><author>stupidTomA</author><pubDate>Sat, 12 Sep 2020 15:04:12 +0800</pubDate><description><![CDATA[环境： hadoop2.7

问题：idea远程连接hadoop，运行mapreduce的程序的时候出错

分析：

可以看到 日志里面打了一行jar没有提交什么的


2020-09-12 14:40:38,391 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_1599887687347_0009
2020-09-12 14:40:38,481 INFO [org.apache.hadoop]]></description><category></category></item><item><title><![CDATA[java   FileInputStream流的一些方法]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/108295544</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/108295544</guid><author>stupidTomA</author><pubDate>Sat, 29 Aug 2020 16:19:57 +0800</pubDate><description><![CDATA[源码里面是使用装饰模式，封装流，然后把每次读取的字节放入一个byte数组中，通过管理该数组实现下面的方法的

测试文件 1


12345678
q
123
iuyweqgahsd

1. 第一个方法 read（）


public static void main(String[] args) throws Exception{
        File file  = new File("E:\\workspace\\flink_demo\\data\\1");
        FileInputStr]]></description><category></category></item><item><title><![CDATA[hadoop安装hive步骤]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/106972846</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/106972846</guid><author>stupidTomA</author><pubDate>Fri, 26 Jun 2020 18:22:06 +0800</pubDate><description><![CDATA[hadoop2.7

hive2.3
1.配置hive-env.sh


# Set HADOOP_HOME to point to a specific hadoop install directory
export HADOOP_HOME=/home/kz/env/hadoop2.7

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/home/kz/env/hive2.3/conf

# Folder ]]></description><category></category></item><item><title><![CDATA[Linux单机hadoop]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/106817277</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/106817277</guid><author>stupidTomA</author><pubDate>Wed, 17 Jun 2020 20:12:25 +0800</pubDate><description><![CDATA[1.工具

jdk1.8

hadoop2.7

2.配置/etc/profile

export JAVA_HOME=/home/kz/env/jdk1.8
export PATH=$PATH:${JAVA_HOME}/bin

export HADOOP_HOME=/home/kz/env/hadoop2.7
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

3.配置hadoop配置文件

3.1hadoop-env.sh

export JAV]]></description><category></category></item><item><title><![CDATA[ubuntu]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/106772930</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/106772930</guid><author>stupidTomA</author><pubDate>Mon, 15 Jun 2020 22:44:25 +0800</pubDate><description><![CDATA[1.ubuntu修改键盘的类别

sudo dpkg-reconfigure keyboard-configuration

2.放大 缩小控制端

ctrl - ctrl +
]]></description><category></category></item><item><title><![CDATA[查看linux文件大小]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/105498242</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/105498242</guid><author>stupidTomA</author><pubDate>Mon, 13 Apr 2020 21:41:13 +0800</pubDate><description><![CDATA[1. df -h /root
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 50G 8.1G 39G 18% /

可以看到root文件夹挂载到/dev/vda1这个磁盘上

2. 查看文件的大小

du -sh * 查看每个文件大小

* 号也可以替换成文件夹或者文件


[root@hadoo...]]></description><category></category></item><item><title><![CDATA[查看linux内存]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/105497039</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/105497039</guid><author>stupidTomA</author><pubDate>Mon, 13 Apr 2020 21:32:32 +0800</pubDate><description><![CDATA[1. ps -aux | sort -k 4n | tail -n 10 对使用内存排序

或者

ps -aux | sort -k 3n | tail -n 10 对cpu进行排序

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2323 0.0 0.1...]]></description><category></category></item><item><title><![CDATA[SparkSQL自定义聚合函数]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/105233132</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/105233132</guid><author>stupidTomA</author><pubDate>Tue, 31 Mar 2020 21:55:56 +0800</pubDate><description><![CDATA[官网上介绍的自定义聚合例子

http://spark.apache.org/docs/latest/sql-getting-started.html#aggregations

SparkSQL的聚合函数支持count(), countDistinct(), avg(), max(), min()等，但是肯定不满足，所以需要用户自定义函数

第一种 弱类型定义 extend UserDefi...]]></description><category></category></item><item><title><![CDATA[SparkSql  查询]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/105232857</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/105232857</guid><author>stupidTomA</author><pubDate>Tue, 31 Mar 2020 21:36:37 +0800</pubDate><description><![CDATA[import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession

object SparkSqlDemo {
  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setAppName("demo").setM...]]></description><category></category></item><item><title><![CDATA[RDD DataFrame DataSet  相互转化的方法]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/105208748</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/105208748</guid><author>stupidTomA</author><pubDate>Mon, 30 Mar 2020 21:12:08 +0800</pubDate><description><![CDATA[import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession

object Transform {

  case class People(name:String,age:Int)
  def main(args: Array[String]): Unit = {

    val sparkConf =...]]></description><category></category></item><item><title><![CDATA[idea中使用sparkSQL]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/105160682</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/105160682</guid><author>stupidTomA</author><pubDate>Sun, 29 Mar 2020 15:22:21 +0800</pubDate><description><![CDATA[1.配置jdk 1.8

2.配置scala scala-2.12.11

3.下载spark， 要和上面的Scala相对应，不然idea里面的醒目启动的时候会报错

4.下载hadoop 2.7.7 https://archive.apache.org/dist/hadoop/common/

，配置HADOOP_HOME，并下载hadoop版本对应的winutils.exe 覆...]]></description><category></category></item><item><title><![CDATA[Spark Yarn 提交流程]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/103349242</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/103349242</guid><author>stupidTomA</author><pubDate>Fri, 14 Feb 2020 11:27:26 +0800</pubDate><description><![CDATA[spark-submit --class wordcount

           --master spark://aml168:7077

           --driver-memory 800M

           --executor-memory 600M 

           -...]]></description><category></category></item><item><title><![CDATA[非root用户之间免密登录]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/103767225</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/103767225</guid><author>stupidTomA</author><pubDate>Mon, 30 Dec 2019 15:52:40 +0800</pubDate><description><![CDATA[1.A机器获取没有设置密码的ssh-keygen

ssh-keygen -t rsa -P '' -f /root/.ssh/id_rsa

2.将A机器的公钥id_rsa.pub发送到B机器的用户test上去，文件位置是/home/test/.ssh，如果.ssh不存在，需要创建,然后将接收到的文件追加到authorized_keys文件里面去

可以使用expect 进行机器命令交互

3....]]></description><category></category></item><item><title><![CDATA[shell   expect]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/103712375</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/103712375</guid><author>stupidTomA</author><pubDate>Thu, 26 Dec 2019 11:41:04 +0800</pubDate><description><![CDATA[第一种方式 复制文件到远程


#!/bin/bash
   expect -c "
   set timeout 10;  
   spawn scp  -o StrictHostKeyChecking=no userhome/.ssh/id_rsa.pub ipaddr$userhome/.ssh/id_rsa.ipaddr; 
   expect { ...]]></description><category></category></item><item><title><![CDATA[记录学习spark遇到的问题]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/103256733</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/103256733</guid><author>stupidTomA</author><pubDate>Tue, 26 Nov 2019 15:12:56 +0800</pubDate><description><![CDATA[1.搭建好StandarOne之后，写了一个例子，将jar包上传到节点上去，运行任务

19/11/26 14:52:50 ERROR TaskSetManager: Task 1 in stage 0.0 failed 4 times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job abo...]]></description><category></category></item><item><title><![CDATA[FAQ]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/102786789</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/102786789</guid><author>stupidTomA</author><pubDate>Mon, 28 Oct 2019 19:13:51 +0800</pubDate><description><![CDATA[1.spark 环境变量要设置JAVA_HOME变量

否则运行spark-shell 会提示找不到路径或者 没有java环境
]]></description><category></category></item><item><title><![CDATA[无法ping通github]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/102670893</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/102670893</guid><author>stupidTomA</author><pubDate>Mon, 21 Oct 2019 20:14:29 +0800</pubDate><description><![CDATA[问题：可以上网访问GitHub，但是在git clone的时候拉不下来项目,然后去ping github.com的时候，请求连接超时

解决方法：

1.配置hosts

2.关闭电脑的防火墙



参考了下面的这篇，但是丢包率有50%，最后关闭了防火墙之后才没有丢包

https://blog.youkuaiyun.com/u010377383/article/details/79365049
...]]></description><category></category></item><item><title><![CDATA[mvn Error: JAVA_HOME not found in your environment.]]></title><link>https://blog.youkuaiyun.com/stupidTomA/article/details/102611707</link><guid>https://blog.youkuaiyun.com/stupidTomA/article/details/102611707</guid><author>stupidTomA</author><pubDate>Thu, 17 Oct 2019 17:41:55 +0800</pubDate><description><![CDATA[问题：

执行maven 编译的时候错误，提示这个，但是执行 java javac命令都有

原因：

当前用户下没有JAVA_HOME 变量

解决方法：

在环境变量里面的用户变量添加JAVA_HOME 变量，在这个用户下添加变量环境就好了

有的答案是说以root权限执行编译就OK，反正就是一个用户权限问题，普通用户找不到，然后root用户就可以了


...]]></description><category></category></item></channel></rss>