<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[SuperDuDu的博客]]></title><description><![CDATA[当能力满足不了自己的野心，那就好好学习吧！]]></description><link>https://blog.youkuaiyun.com/u010741721</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; u010741721]]></copyright><item><title><![CDATA[nutch采集部署]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/105268846</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/105268846</guid><author>u010741721</author><pubDate>Thu, 02 Apr 2020 14:37:46 +0800</pubDate><description><![CDATA[nutch的限制：
不能采集js动态加载页面内容和分页，对非动态js加载的分页，需要根据分页的情况，来进行设置爬取深度（分页与爬取深度存在关系）。这就限制了nutch全爬取存在很大的局限性，nutch本身要遵守robot协议，如果网址本身不允许被爬取，那么将不会被爬取。

nutch2.2.1部署
环境要求：
系统：Centos7.5
运行环境：java1.8
数据库：mysql5.7
编译环境...]]></description><category></category></item><item><title><![CDATA[网盘调研 ]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/105116967</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/105116967</guid><author>u010741721</author><pubDate>Thu, 26 Mar 2020 13:38:03 +0800</pubDate><description><![CDATA[调研网址

github/gitee，知乎，企业网站

调研产品

kiftd（开源软件）:

网址：https://gitee.com/kohgylw/kiftd?_from=gitee_search

实现文件上传，但是不能实现，数据秒传
	可以时间文件的组内共享。
	可以生成下载接口地址，提供在线下载
	平台内部集成图片、pdf和视频的插件，实现图片和视频在线查看。
	本地文件系统存储
亿方...]]></description><category></category></item><item><title><![CDATA[DS开发环境搭载]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/104658798</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/104658798</guid><author>u010741721</author><pubDate>Wed, 04 Mar 2020 17:55:52 +0800</pubDate><description><![CDATA[设计特点：一个分布式易扩展的可视化DAG工作流任务调度系统。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用。 其主要目标如下：

以DAG图的方式将Task按照任务的依赖关系关联起来，可实时可视化监控任务的运行状态
支持丰富的任务类型：Shell、MR、Spark、SQL(mysql、postgresql、hive、sparksql),Python,Sub_Pro...]]></description><category></category></item><item><title><![CDATA[socket查看zookeeper情况（4字命令）]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/104547272</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/104547272</guid><author>u010741721</author><pubDate>Thu, 27 Feb 2020 22:59:38 +0800</pubDate><description><![CDATA[public class FourLetterWordMain {

    private static final int DEFAULT_SOCKET_TIMEOUT = 5000;
    protected static final Logger LOG = LoggerFactory.getLogger(FourLetterWordMain.class);

    /**
     ...]]></description><category></category></item><item><title><![CDATA[大数据产品最新测试基准看哪家（TPC-H or TPC-DS）？]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/104194879</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/104194879</guid><author>u010741721</author><pubDate>Thu, 06 Feb 2020 12:59:35 +0800</pubDate><description><![CDATA[前言
Hadoop的出现让人们尝到了大数据技术的甜头，它的批处理能力已经被工业界充分认可，但是它的延迟性也一直为大家所诟病。随着各行各业的发展，越来越多的业务要求大数据系统既可以处理历史数据，又可以进行实时计算。比如电商推荐系统，当你在京东浏览商品时，京东会根据你的浏览、加车、收藏、删除等行为，实时为你推荐商品。要实现这个功能，推荐引擎首先需要根据历史数据预先离线计算推荐模型，然后从消息队列中实时...]]></description><category></category></item><item><title><![CDATA[前后端非对称加密,解决http明文传输]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103991305</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103991305</guid><author>u010741721</author><pubDate>Wed, 15 Jan 2020 16:12:18 +0800</pubDate><description><![CDATA[后台加密解密的工具类
package com.sdyy.cas.utils;

import org.apache.commons.codec.binary.Base64;

import javax.crypto.Cipher;
import java.math.BigInteger;
import java.security.*;
import java.security.interfaces...]]></description><category></category></item><item><title><![CDATA[防止Xss脚本攻击（XssFilter）]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103991036</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103991036</guid><author>u010741721</author><pubDate>Wed, 15 Jan 2020 16:05:25 +0800</pubDate><description><![CDATA[配置文件
package com.sdyy.cas.config;

import com.google.common.collect.Maps;
import com.sdyy.cas.filter.XssFilter;
import org.springframework.boot.web.servlet.FilterRegistrationBean;
import org.springfra...]]></description><category></category></item><item><title><![CDATA[Java调用ElasticSearch自定义的评分插件]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/102723943</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/102723943</guid><author>u010741721</author><pubDate>Wed, 15 Jan 2020 15:53:17 +0800</pubDate><description><![CDATA[之前，使用phash做了一个基于ElasticSearch的图片相似性的项目，需要自己写一个评分插件，在kibana中用es的语句查询，没有问题，现在要集成到java项目中

kibana中查询语句
GET test_index/_search
{ 
  "from": 0,
  "size": 10, 
  "min_score":80,
  "query": {
    "function_...]]></description><category></category></item><item><title><![CDATA[文件监听同步（watchservice）]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103990786</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103990786</guid><author>u010741721</author><pubDate>Wed, 15 Jan 2020 15:52:17 +0800</pubDate><description><![CDATA[watchservice
private void startCatalogWatcher(File catalogConfigurationDir) throws IOException, InterruptedException
    {
        WatchService watchService = FileSystems.getDefault().newWatchService(...]]></description><category></category></item><item><title><![CDATA[文件监听同步（canal 搭建）]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103985311</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103985311</guid><author>u010741721</author><pubDate>Wed, 15 Jan 2020 14:00:12 +0800</pubDate><description><![CDATA[参考文档
https://github.com/alibaba/canal/wiki/QuickStart

mysql开启bin-log日志
log_bin = /var/lib/mysql/bin-log
skip-name-resolve
binlog-format=ROW

创建canal对象
CREATE USER canal IDENTIFIED BY 'canal'; 
GRANT...]]></description><category></category></item><item><title><![CDATA[文件监听同步（lsyncd）]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103984663</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103984663</guid><author>u010741721</author><pubDate>Wed, 15 Jan 2020 10:50:05 +0800</pubDate><description><![CDATA[使用lsyncd实现多台机器的文件同步
多台机器的ssh互信
ssh-copy-id 对方的机器IP或者主机名

下载安装lsyncd
yum install http://rpmfind.net/linux/epel/7/x86_64/Packages/l/lsyncd-2.2.2-1.el7.x86_64.rpm

配置lsyncd.conf文件
vi /etc/lsyncd.conf #修改...]]></description><category></category></item><item><title><![CDATA[mysql集群双活+高可用]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103704472</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103704472</guid><author>u010741721</author><pubDate>Wed, 25 Dec 2019 19:30:21 +0800</pubDate><description><![CDATA[mysql双主部署+keepalived
  安装mysql（此处使用二进制，还可以用 rpm包安装 或 mysql编译安装）
https://blog.youkuaiyun.com/u012562411/article/details/85220783
 
0.卸载原来版本mysql或mariadb
rpm -e mariadb-libs  --nodeps
1.进入/usr/local，下载tar包
w...]]></description><category></category></item><item><title><![CDATA[Redis4集群搭建操作记录]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103704430</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103704430</guid><author>u010741721</author><pubDate>Wed, 25 Dec 2019 19:28:17 +0800</pubDate><description><![CDATA[cp -r /etc/yum.repos.d /etc/yum.repos.d.bak
rm -rf /etc/yum.repos.d/*
cd /etc/yum.repos.d/ &amp;&amp; vi CentOS-Base.repo
*************************
# CentOS-Base.repo
#
# The mirror system uses the co...]]></description><category></category></item><item><title><![CDATA[presto(三) presto的MPP架构]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103008872</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103008872</guid><author>u010741721</author><pubDate>Tue, 03 Dec 2019 13:23:10 +0800</pubDate><description><![CDATA[]]></description><category></category></item><item><title><![CDATA[git文件上传]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103363121</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103363121</guid><author>u010741721</author><pubDate>Tue, 03 Dec 2019 13:14:48 +0800</pubDate><description><![CDATA[注册git账号
略

生成ssh-key
ssh-keygen -t rsa -C "xxxx@xx"

配置文件
git config --user.name ="xx"
git config --user.email ="xxx@xx"

上传git文件
进入 文件夹
cd xxx
创建初始化文件
git init
创建README文件
touch README.md 
添加README文件
...]]></description><category></category></item><item><title><![CDATA[java 列表或数组按指定大小分组]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103182088</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103182088</guid><author>u010741721</author><pubDate>Thu, 21 Nov 2019 14:34:02 +0800</pubDate><description><![CDATA[原文链接：https://blog.youkuaiyun.com/abc5232033/article/details/75453170
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
 
/**
 * 列表或数组按指定大小分组,用于批量取一部分数据循环处理
 *
 */
public class Arra...]]></description><category></category></item><item><title><![CDATA[基于云打码的验证码识别]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103156937</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103156937</guid><author>u010741721</author><pubDate>Wed, 20 Nov 2019 10:03:07 +0800</pubDate><description><![CDATA[class YDMHttp(object):
    apiurl = 'http://api.yundama.com/api.php'
    username = '*************1'
    password = '**************'
    appid = 9079
    appkey = '********************'

    def __ini...]]></description><category></category></item><item><title><![CDATA[基于Tesseract-OCR的验证码识别]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103082097</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103082097</guid><author>u010741721</author><pubDate>Fri, 15 Nov 2019 11:26:03 +0800</pubDate><description><![CDATA[先上代码

必须现在电脑上按照Tesseract-OCR，下载地址：
链接：https://pan.baidu.com/s/1FBA2lXKdQeYGn3gWC3vBoQ
提取码：zaop

from PIL import Image
import pytesseract
from PIL import ImageEnhance

pytesseract.pytesseract.tesseract...]]></description><category></category></item><item><title><![CDATA[presto(二) presto部署]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/103008587</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/103008587</guid><author>u010741721</author><pubDate>Mon, 11 Nov 2019 13:28:19 +0800</pubDate><description><![CDATA[Presto介绍

presto查询环境搭建
presto组件下载
下载（server和cli）：
https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.212/presto-server-0.212.tar.gz
https://repo1.maven.org/maven2/com/facebook/presto/p...]]></description><category></category></item><item><title><![CDATA[presto(一) presto概述]]></title><link>https://blog.youkuaiyun.com/u010741721/article/details/102922230</link><guid>https://blog.youkuaiyun.com/u010741721/article/details/102922230</guid><author>u010741721</author><pubDate>Tue, 05 Nov 2019 19:03:45 +0800</pubDate><description><![CDATA[presto背景及发展
长话短说，大数据时代，hadoop的解决方案，解决了完美的大数据存储和很好的解决了计算的问题，但是由于hadoop采用的是一套MR的解决方案，这就导致hadoop可以解决大批量的离线计算问题但是不能很好的解决AC-hoc的问题，faceBook开源了presto

presto的特点
多数据源
支持sql
支持混合查询
拓展性
混合计算
高性能
内存计算，流水线

pres...]]></description><category></category></item></channel></rss>