Introduction of internet P2P technology

Internet P2P download technology

• bittorrent
• metalink

bittorrent

A P2P download protocal:
A communications protocol of peer-to-peer file sharing (“P2P”) which is used to distribute data and electronic files over the Internet. BitTorrent is one of the most common protocols for transferring large files.

1. 通过服务器查询到拥有目标资源的节点
2. 跟节点直接建立连接，下载节点拥有的资源片段。在自己完整下载一部分资源之后，为别人有下载需求节点传输资源。

其中拥有完整资源的节点叫 seed，拥有部分资源的节点叫 peer，更多定义在后文中介绍。

通过 .torrent 了解资源分布并下载资源。

Bram Cohen invent BitTorrent protocal with python in 2003.

Grow Up

The implementation of DHT(distributed hash table) eliminates the need for trackers.

Something about torrents.

• Peer:
  

  someone who does not currently have the completed file.

When a peer is connected, he downloads the pieces he does not have and uploads the pieces he does have. You are a peer if you do not have a complete copy of the file you’re trying to get.

• Seed:
  

  A Seed is someone who already has the complete file but is still sharing. If there are no Seeds, the only way to get a complete file is if all the pieces of the file can be found amongst the peers that are connected. In most cases, when there is no Seed, you probably won’t get the whole file.

Note that the term leech used to be common in bulletin boards and usenet groups. We used it to describe someone who downloads things but never uploads. With torrents, as soon as you get your first piece, you’re sharing. So we call everyone a peer. Of course, I still use leech when I talk about about people who never Seed a file. As a rule of thumb, you should always try to Seed a torrent before for at least 1 full copy. You can see this in the Ratio column. 1.000 or higher means you have seeded at least 1 full copy of the file.

• Tracker:
  

  The tracker is a server that has all the info about the people that are down- and uploading the file.

The tracker itself does not have a copy of the file, it only tracks the people who have the file (seeds) and the people who have part of the file (peers). Torrents can be tied to a specific tracker, but most clients now support trackerless torrents, making it less likely that you will be hurt if you can’t find the
original tracker.

• Scrape:
  

  When your BitTorrent client asks for info from the “tracker”, we call this scraping.

The data you get from scraping tells you how many Seeds and Peers there are for each torrent. This is not limited to just the active Seeds and Peers. It could also include Seeds and Peers that are not currently connected.

• Swarm:
  

  Together, all the Seeds and Peers who are using the same torrent on the same tracker with you.

For example, six Peers and two Seeds on the same tracker make a swarm of eight. So your Swarm is NOT the users you are connected to. It’s perfectly normal NOT to connect to ALL seeds and peers in a swarm. In a minute, we’ll even see how the opposite is true.

First, when you look at the numbers in the Seeds and Peers columns, we see 2 numbers in each column: x(y).

• SEEDS:

  x = the number of seeds from which your client is currently downloading pieces.

  y = the total number of seeds in the swarm

So, if you see 5 (14) under Seeds, you are connected to 5 out of 14 seeds. The tracker knows about 9 more seeds to which you are NOT connected. This could be because these seeds only allow a limited number of connections or there could be other reasons.

Once you get the complete file, you will no longer connect to Seeds because, as a Seed
yourself, you don’t exchange any data with other Seeds. Your client still shows you the Seeds in your swarm so you see something like 0(14).

If you see something like 12 (4), it usually means the tracker only knows about 4 seeds in your swarm, but thanks to a feature called DHT you were able to connect to seeds outside your swarm.

• DHT:
  stands for Distributed hash tables. You don’t need to understand DHT. Just know that DHT makes allows trackers to share the burden of tracking swarms for torrents. If your client only sees the swarm attached to your tracker, if your client supports DHT, it might connect to Seeds and Peers that are connected to another tracker.

• PEERS

  x = the number of peers with which you are currently sharing pieces (downloading or uploading)
  y = the total number of peers in the swarm

This works very much like Seeds.

5 (12) means you are connected to 5 peers but the tracker knows about 7 more peers to which you are NOT connected. 0 (12) means your client knows about 12 peers but you are not connected to any of them. If your file is not complete, this might mean that none of the Peers needs any of the pieces you have. 12 (4) means the tracker knows about 4 peers in your swarm, but DHT is helping you connect to seeds outside your swarm.

One last tidbit:

Trackers are Not Web Sites.
A tracker helps clients to connect to each other and collects data about your swarm. It is basically a “dumb pc” that only knows how to connect your BitTorrent client to other BitTorrent clients that are downloading the same torrent. An indexer is a website that hosts torrent files for download. So Piratebay is an indexer, but not a tracker.

• Referrence

• source code of bittorrent at cvs.sourceforge

Production:

1. 迅雷，搭建有自己的中心资源服务器，所以对国内资源的下载是最好的。
2. BitTorrent：批量下载处理，很好用的。
3. μTorrent
4. aria2: 可以下载包括 bittorrent, megnet, FTP, HTTP等等各种形式链接资源的软件，命令行运行，高度可定制，非常适合在生产环境中搭建自动化下载器。

Metalink

An Internet standard.

“Metalink is an extensible metadata file format that describes one or more computer files available for download. ”
“Besides FTP and HTTP mirror locations and rsync, it also supports listing the P2P methods BitTorrent, ed2k, magnet link or any other that uses a URI.”

Reading Materials:

• metalink Home Page
• The Metalink Download Description Format RFC 5854

发展中的技术

• project-maelstrom
  可以把 Project Maelstrom 的一个网站看作一个Torrent，去中心化了，大大削弱服务器这个概念，把传统P2P只能传文件的概念进行了扩展，将网站作为文件进行传输，使用者越多读取的速度越快。