- 博客(64)
- 收藏
- 关注
RSS订阅原创 python spark streaming 从 kafka获取nginx日志，秒级统计url pv，并写入mysql
mysql建库建表>create database nginx;>use nginx> create table url_access (id int NOT NULL AUTO_INCREMENT primary key, timestamp varchar(256), url varchar(256), pv long); #!/usr/bin/env ...
2019-01-09 23:44:03
553
原创 python kafka消息传入spark streaming 实时计算打印
实现了从kafka取nginx日志，并在本地打印出来 #!/usr/bin/env python# -*- coding: utf-8 -*-# @Time : 2019/01/09 18:00# @Author : xuanda# @Site : # @File : kafka_to_sparkstreaming.pyfrom pyspark import...
2019-01-09 18:12:34
800
原创 python spark streaming单机测试，streaming源为本地，统计word count
网上有很多pyspark streaming的测试代码，不过大多都是需要结合kafka做消息来源由于懒得搭kafka，所以想本地生成随机数据作为streaming源，测试spark streaming google查了一些文章，其实spark github中就有类似代码，只不过文件名叫 hdfs_wordcount.py，是针对hdfs的examplehttps://github....
2019-01-03 17:08:28
673
原创 python连接standalone版spark
昨天已经完成了standalone的spark搭建，当然今天在实际使用python连接时，发现其实是有问题的。master和slave其实并没有成功连接，原因是启动slave时指定的master ip地址有误，需要通过netstat查看7077在哪个ip上侦听，不可直接写127.0.0.1 今天成功用python连接了昨天搭建的spark集群做简单计算 环境：python3...
2019-01-02 16:24:21
526
原创 spark单机模式快速搭建
运维做了8年，带了好几年团队，但是逐渐发现自己技术上飘飘然，前几年还写写代码，做做运维，现在只能探探规划，看看方向。什么spark、docker、k8s、hadoop、open-falcon、devops、aiops都带团队做过，但自己真没具体做过。今天开始在运维、开发领域做个新手，重新从0开始学习，看看自己是不是既能上天又能入地 Spark介绍：Spark程序在集群中独立运行，通过S...
2019-01-01 23:29:11
3755
转载 7.5. Packet Filtering with ipchains
7.5. Packet Filtering with ipchainsthe three builtin chains, input, output, forward policy per chain, see targets jumping from chain to chain, -j $TARGET; wher TARGET=chain the big
2010-01-01 19:31:00
482
转载 7.6. Protecting a Host
7.6. Protecting a HostHost protection in the past was typically performed with application layer checks on the originating IP or hostname. This was (and still is) frequently accomplished
2010-01-01 19:31:00
493
转载 7.8. Further Resources
7.8. Further ResourcesThe use of linux packet filtering features is mature and well-documented in many places throughout the Internet. One of the most thorough introductions to the use o
2010-01-01 19:31:00
858
转载 7.1. Rationale for and Introduction to Packet Filtering
7.1. Rationale for and Introduction to Packet FilteringPacket filtering refers to the technique of conditionally allowing or denying packets entering or exiting a network or host based o
2010-01-01 19:30:00
521
转载 7.2. Limits and Weaknesses of Packet Filtering
7.2. Limits and Weaknesses of Packet FilteringAlthough the functionality offered by linux kernels for protecting network resources with packet filtering allows tremendously specific netw
2010-01-01 19:30:00
506
转载 7.3. General Packet Filter Requirements
7.3. General Packet Filter Requirementsminimum ICMP required to meet the networking needs; xref PMTU discussion source quench parameter problem inbound destination unreachable ou
2010-01-01 19:30:00
502
转载 7.4. The Netfilter Architecture
7.4. The Netfilter Architecturepacket filtering engine in kernel 2.2 (skip history, adequately documented elsewhere) packet filtering engine as part of netfilter in kernel 2.4, backwar
2010-01-01 19:30:00
438
转载 6.1. Concepts of Source NAT
6.1. Concepts of Source NAT 6.1.1. Differences Between SNAT and MasqueradingThough SNAT and masquerading perform the same fundamental function, mapping one address spac
2010-01-01 19:29:00
406
转载 Chapter 7. Packet Filtering
Chapter 7. Packet FilteringTable of Contents7.1. Rationale for and Introduction to Packet Filtering7.1.1. History of Linux Packet Filter Support7.2. Limits and Weaknesses of Packet F
2010-01-01 19:29:00
731
转载 Chapter 6. Masquerading and Source Network Address Translation
Chapter 6. Masquerading and Source Network Address TranslationTable of Contents6.1. Concepts of Source NAT6.1.1. Differences Between SNAT and Masquerading6.1.2. Double SNAT/Masquerad
2010-01-01 19:28:00
530
转载 5.4. Stateless NAT and Packet Filtering
5.4. Stateless NAT and Packet FilteringBecause NAT rewrites the packet as it passes through the IP stack, packet filtering can become complex. With attentiveness to the addressing of the
2009-12-31 18:43:00
832
转载 5.5. Destination NAT with netfilter (DNAT)
5.5. Destination NAT with netfilter (DNAT)Destination NAT with netfilter is commonly used to publish a service from an internal RFC 1918 network to a publicly accessible IP. To enable DN
2009-12-31 18:43:00
391
转载 5.6. Port Address Translation (PAT) from Userspace
5.6. Port Address Translation (PAT) from UserspacePort address translation (hereafter PAT) provides a similar functionality to NAT, but is a more specific tool. PAT forwards requests for
2009-12-31 18:43:00
641
转载 5.2. Application Layer Protocols with Embedded Network Information
5.2. Application Layer Protocols with Embedded Network InformationNetwork address translation is beautifully invisible when it works, but has adverse effects on some protocols. Some netw
2009-12-31 18:42:00
429
转载 5.3. Stateless NAT with iproute2
5.3. Stateless NAT with iproute2Stateless NAT, occasionally maligned as dumb NAT [31], is the simplest form of NAT. It involves rewriting addresses passing through a routing device: inbo
2009-12-31 18:42:00
836
转载 5.1. Rationale for and Introduction to NAT
5.1. Rationale for and Introduction to NATNetwork address translation (NAT) is a technique of transparently mapping an IP address or range to another IP address or range. Any routing dev
2009-12-31 18:41:00
590
转载 Chapter 5. Network Address Translation (NAT)
Chapter 5. Network Address Translation (NAT)Table of Contents5.1. Rationale for and Introduction to NAT5.2. Application Layer Protocols with Embedded Network Information5.3. Stateles
2009-12-31 18:40:00
680
转载 4.9. Routing Policy Database (RPDB)
4.9. Routing Policy Database (RPDB) The routing policy database (RPDB) controls the order in which the kernel searches through the routing tables. Each rule has a priority, and rules a
2009-12-30 20:53:00
917
转载 4.10. ICMP and Routing
4.10. ICMP and RoutingICMP is a very important part of the communication between hosts on IP networks. Used by routers and endpoints (clients and servers) ICMP communicates error conditi
2009-12-30 20:53:00
534
转载 4.8. Routing Tables
4.8. Routing TablesLinux kernel 2.2 and 2.4 support multiple routing tables [22]. Beyond the two commonly used routing tables (the local and main routing tables), the kernel supports u
2009-12-30 20:52:00
1272
转载 4.6. Source Address Selection
4.6. Source Address Selection The selection of the correct source address is key to correct communication between hosts with multiple IP addresses. If a host chooses an address from a
2009-12-30 20:51:00
779
转载 4.7. Routing Cache
4.7. Routing CacheThe routing cache is also known as the forwarding information base (FIB). This term may be familiar to users of other routing systems. The routing cache stores rece
2009-12-30 20:51:00
481
转载 4.3. Sending Packets Through a Gateway
4.3. Sending Packets Through a Gateway By comparison to the total number of publicly accessible hosts on the Internet there is an almost insignificant number of hosts inside any locall
2009-12-30 20:50:00
413
转载 4.4. Operating as a Router
4.4. Operating as a RouterOperating as a router allows a linux machine to accept packets on one interface and transmit them on another. This is the nature of a router. The process of a
2009-12-30 20:50:00
333
转载 4.5. Route Selection
4.5. Route Selection Crucial to the proper ability of hosts to exchange IP packets is the correct selection of a route to the destination. The rules for the selection of route path are
2009-12-30 20:50:00
660
转载 4.2. Routing to Locally Connected Networks
4.2. Routing to Locally Connected Networks Any IP network is defined by two sets of numbers: network address and netmask. By convention, there are two ways to represent these two numbe
2009-12-30 20:49:00
549
转载 4.1. Introduction to Linux Routing
4.1. Introduction to Linux RoutingThe design of IP routing allows for very simple route definitions for small networks, while not hindering the flexibility of routing in complex environm
2009-12-30 20:47:00
493
转载 Chapter 4. IP Routing
Chapter 4. IP RoutingTable of Contents4.1. Introduction to Linux Routing4.2. Routing to Locally Connected Networks4.3. Sending Packets Through a Gateway4.4. Operating as a Router4.5.
2009-12-30 20:46:00
405
转载 Chapter 3. Bridging
Chapter 3. BridgingTable of Contents3.1. Concepts of Bridging3.2. Bridging and Spanning Tree Protocol3.3. Bridging and Packet Filtering3.4. Traffic Control with a Bridge3.5. ebtables
2009-12-30 20:45:00
396
转载 2.3. ARP filtering
2.3. ARP filteringThis section should be part of the "ghetto" which will include documentation on ip arp. Theres nothing more to add here at the moment (low priority). #
2009-12-30 20:44:00
444
转载 2.4. Connecting to an Ethernet 802.1q VLAN
2.4. Connecting to an Ethernet 802.1q VLAN Virtual LANs are a way to take a single switch and subdivide it into logical media segments. A single switch port in a VLAN-capable switch ca
2009-12-30 20:44:00
451
转载 2.5. Link Aggregation and High Availability with Bonding
2.5. Link Aggregation and High Availability with Bonding Networking vendors have long offered a functionality for aggregating bandwidth across multiple physical links to a switch. This
2009-12-30 20:44:00
583
转载 2.2. Proxy ARP
2.2. Proxy ARPOccasionally, an IP network must be split into separate segments. Proxy ARP can be used for increased control over packets exchanged between two hosts or to limit exposur
2009-12-30 20:43:00
426
转载 2.1. Address Resolution Protocol (ARP)
2.1. Address Resolution Protocol (ARP)Address Resolution Protocol (ARP) hovers in the shadows of most networks. Because of its simplicity, by comparison to higher layer protocols, ARP ra
2009-12-30 19:35:00
759
转载 Chapter 2. Ethernet Table of Contents
Chapter 2. EthernetTable of Contents2.1. Address Resolution Protocol (ARP)2.1.1. Overview of Address Resolution Protocol2.1.2. The ARP cache2.1.3. ARP Suppression2.1.4. The ARP Flux Prob
2009-12-30 19:33:00
438
空空如也
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人