分布式队列_分布式队列虚拟指南

分布式队列

by Zhia Hwa Chong

志华化

分布式队列虚拟指南 (A Dummy’s Guide to Distributed Queues)

If you’ve ever wondered what Kafka, Heron, real-time streaming, SQS or RabbitMQ are all about, then this article is for you. I’ll discuss in detail why we need a queue for today’s modern software architecture, what are some common technologies used, and how queues are commonly used in the industry. If you enjoy this article, I have a course on scaling distributed systems where I discuss these topics in more detail.

如果您想知道Kafka，Heron，实时流，SQS或RabbitMQ是什么，那么这篇文章适合您。 我将详细讨论为什么今天的现代软件体系结构需要队列，使用了哪些常用技术以及行业中如何使用队列。 如果您喜欢本文，那么我将开设一门有关扩展分布式系统的课程，在其中我将详细讨论这些主题。

OK, let’s get into it!

好的，让我们开始吧！

首先，为什么您需要队列/消息代理？ (First and foremost, why do you need a queue/message broker?)

排队如何挽救柠檬水的故事 (The story of how a queue saved the lemonade stands)

Imagine that you are running a lemonade ? stand, and you built out a nifty little web-app that keeps track of how often your clients return to your lemonade stand.

想象一下，您正在喝柠檬水吗？ 您建立了一个漂亮的小型网络应用程序， 可跟踪您的客户多久返回一次柠檬水摊位。

Your web-app has an endpoint, say yourlemonade.com/traffic, and every time you click on a button, the traffic count goes up by 1. Beautiful.

您的Web应用程序有一个终结点，例如yourlemonade.com/traffic，每当您单击一个按钮时， 流量计数就会增加1.美丽。

As the traffic to your lemonade stand increases, you click the button more and more. Well, since you’re living in a relatively small neighborhood, you only get 10–20 people per day. Your sales proceed as usual, the web-app handles the traffic just fine, and everything is fine and dandy. Perfect.

随着通往柠檬水摊位的流量增加，您越来越多地单击按钮。 好吧，由于您生活在一个相对较小的社区中，因此每天您只会得到10至20个人。 您的销售照常进行，网络应用程序可以很好地处理流量，一切都很好。 完善。

生意兴隆的噩梦 (The nightmare of a booming business)

Now that your lemonade stand has made a name for itself, people from across the city are flocking in to get a taste of your famous lemonade. And on a beautiful Sunday morning, the local news decided to promote your stand, and the traffic EXPLODES.

如今，您的柠檬水摊位已广为人知，全市各地的人们纷纷蜂拥而至，品尝您著名的柠檬水。 在一个美丽的星期日早晨，当地新闻决定宣传您的展位，并且交通量激增 。

As you can imagine, the traffic to your lemonade stand increases from 10–20 people per day to 10,000 per day. You’re tapping the traffic button furiously, which in turns triggers a call to yourlemonade.com/traffic, and your web app keeps incrementing the amount of traffic.

您可以想象，柠檬水摊位的流量从每天10–20人增加到每天10,000人。 您正在疯狂地点击流量按钮，这又触发了对yourlemonade.com/traffic的呼叫，并且您的Web应用程序不断增加流量。

Unfortunately, your web-app is hosted on an 8-bit, 128MB RAM server in your house garage. With the booming business and increased traffic, your web-app cannot handle the scale of traffic any longer.

不幸的是，您的Web应用程序托管在房屋车库中的8位128MB RAM服务器上。 随着业务的蓬勃发展和流量的增加，您的网络应用无法再处理流量规模了。

Eventually, your server dies. ☠️

最终，服务器死了。 ☠️

With that, your entire web app is brought down. You can’t keep track of traffic anymore. People are rushing in, orders are piling up, yet your web-app is down and you can’t handle any transactions until you can start logging the traffic again.

这样，您的整个Web应用程序将被关闭。 您无法再跟踪流量了。 人们涌入，订单不断堆积，但您的网络应用已关闭，您无法处理任何交易，直到可以再次开始记录流量。

What do you do?

你是做什么？

排队救援 (Queue to the rescue)

A moment of brilliance strikes you, what if I place a box in front of the counter where each client can just drop a note saying they were there?

一时的光彩照耀着您， 如果我在柜台前放一个盒子，让每个客户都可以放一个便条说他们在那里，那该怎么办？

Every time a client walks through the door and places an order, you ask them politely to drop their order sheets in a small box placed in front of the payment counter. Outstanding! You have essentially introduced a mechanism to keep track of arrivals while still allowing your business to function as usual.

每次客户走进门并下订单时，您要礼貌地要求他们将订单单放在付款柜台前的小盒子中。 优秀！ 实质上，您已经引入了一种机制来跟踪到达的同时仍允许您的业务照常运行。

This is what we call asynchronous processing, and, welcome to the world of queues. ?

这就是我们所谓的异步处理 ， 欢迎来到队列世界 。 ？

When you start out building software, much like the lemonade stand I mentioned above, it is common for a task to

当您开始构建软件时，就像我上面提到的柠檬水摊位一样，完成任务通常是

1. call a service, then
   致电服务，然后
2. wait for the service to finish, and then
   等待服务完成，然后
3. move on to the next task.
   继续下一个任务。

This is what’s called synchronous processing. Asynchronous processing, on the other hand, allows a task to call a service, and move on to the next task while the service processes the request at its own pace. That’s why a queue is a beautiful, elegant way to unblock your systems because it puts a layer in front of your services and allows them to tackle the tasks at their own pace.

这就是所谓的同步处理。 另一方面， 异步处理允许任务调用服务，并在服务以自己的步调处理请求时移至下一个任务 。 这就是为什么队列是解除系统阻塞的一种优美，优雅的方法，因为它在服务之前放置了一层，并允许他们按自己的步调处理任务。

如果队列是如此强大，那么为什么不把它放在一切前面呢？ (If a queue is so powerful, why don’t we just put it in front of everything?)

As anyone who has dabbled in distributed systems can attest to, scaling a distributed system is extremely tricky and complicated. There are a few things to know about queues that might make a queue an unattractive proposition for your system.

涉足分布式系统的任何人都可以证明，扩展分布式系统非常棘手和复杂。 关于队列，有几件事需要了解，这可能会使队列对您的系统没有吸引力。

Some questions I’d ask before deciding if a queue is the right solution for you:

在确定队列是否适合您之前，我会问一些问题：

• Is your service having issues due to high traffic? If it’s not, maybe you should look into what the bottleneck is before jumping into queues. As Donald Knuth famously said, premature optimization is the root of all evil.

  您的服务是否因流量过大而出现问题？ 如果不是，也许您应该在进入队列之前先研究一下瓶颈。 正如Donald Knuth所说的那样， 过早的优化是万恶之源。

• Do you have in-house expertise in managing a queue? Or do you need to potentially hire a team to do it for you? Maintenance costs, like scaling the queue, can skyrocket if you’re not careful. There are services like Amazon SQS (Simple Queueing Service) which offer a managed solution (i.e. you don’t need to maintain anything on your own).

  您是否拥有管理队列的内部专业知识？ 还是您可能需要雇用一个团队来为您做？ 如果您不小心，维护成本(例如扩展队列)可能会飞涨。 诸如Amazon SQS (简单队列服务)之类的服务提供了托管解决方案(即，您不需要自己维护任何东西)。

• Is it possible to have duplicate entries in the queue? If so, is that acceptable?
  队列中是否可能有重复的条目？ 如果可以，那可以接受吗？
• Do you need to keep a record of all transactions, in case a queue goes down?
  万一队列中断，您是否需要记录所有交易？
• In the case that a queue goes down, does the queue need to be able to replay all the entries? What are your backup options?
  如果队列发生故障，该队列是否需要能够重播所有条目？ 您有哪些备份选项？

There are many more concerns that might be specific to your use case, but hopefully, I’ve made my point that adding a queue isn’t as easy as snapping your fingers.

还有更多可能与您的用例有关的问题，但希望我已经指出，添加队列并不像按手指一样容易。

如何在现代建筑中使用队列 (How Queues Are Used in Modern Architecture)

Queues are ubiquitous in today’s modern distributed systems architecture — adopted across various industries for different use cases, and there are more novel use cases every day.

队列在当今的现代分布式系统体系结构中无处不在-在不同行业中为不同的用例所采用，并且每天都有更多新颖的用例。

Here are some of the real-world use cases for queues:

以下是一些实际的队列用例：

即时串流 (Real-time streaming)

When MapReduce came around, it was a huge phenomenon in the industry because it allowed mere mortals to process petabytes of data in a reasonable amount of time, anywhere from days to hours. This might seem absurd today when data’s available in almost seconds, but pre-MapReduce, it wasn’t easy to extract usable data from extremely large data sets.

当MapReduce出现时，这在业界是一个巨大的现象，因为它允许凡人在合理的时间内(从几天到几小时不等)处理PB的数据。 今天，当数据几乎在几秒钟内可用时，这似乎是荒谬的，但是在MapReduce之前，要从超大型数据集中提取可用数据并不容易。

The appetite for data analytics has grown, and we’re now looking at processing data within hours, and sometimes, milliseconds.

数据分析的需求在增长，我们现在正在考虑在数小时(有时是毫秒)内处理数据。

To achieve low-latency analytics and performance in a continuous fashion, the concept of real-time streaming was conceived.

为了以连续方式实现低延迟分析和性能，构想了实时流传输的概念。

A useful example here is to think of ads: ads on Twitter, for example, are shown to millions of people per day. Yet in order to make sure users don’t see the same ads multiple times within a set period of time, Twitter needs to somehow know the last time a user was exposed to a certain ad.

这里一个有用的例子是考虑广告：例如，Twitter上的广告每天向数百万人展示。 但是，为了确保用户在设定的时间内不会多次看到相同的广告，Twitter需要以某种方式知道用户最后一次接触到某个广告的时间。

If we had relied on MapReduce to perform this action, it would not even be considered a solution because it’ll take upwards of hours to process all that data. Instead, real-time streaming allows us to process ad impressions as they arrive. This is all made possible because of queues that allow data to be continuously streamed and processed in real time.

如果我们依靠MapReduce来执行此操作，那么它甚至都不会被视为解决方案，因为处理所有这些数据将花费数小时的时间。 取而代之的是，实时流媒体使我们能够在广告展示到达时对其进行处理。 由于队列使数据能够连续不断地实时流传输和处理，因此一切皆有可能。

Some technologies you’ll often hear about in real-time streaming use cases are Kafka, Kafka streams, Redis, Spark Streaming (which is different from Spark) and so on.

在实时流使用案例中，您经常会听到的一些技术包括Kafka，Kafka流，Redis，Spark Streaming(与Spark不同)等。

事件驱动架构 (Event-driven architecture)

Queues are used as a critical component of an event-driven architecture, or colloquially known as Pub(lisher)-Sub(scriber). Event-driven architecture is, according to Wikipedia:

队列被用作的一个重要组件事件驱动的架构 ，或俗称为酒吧 (lisher) - 子 (划线)。 根据维基百科，事件驱动架构是：

Event-driven architecture (EDA), is a software architecture pattern promoting the production, detection, consumption of, and reaction to events.

事件驱动体系结构(EDA)是一种软件体系结构模式，可促进事件的产生，检测，使用和React。

I’d like to think of this as subscribing to a newsletter: as a producer of a newsletter, you know who’s subscribed to your newsletter and who’s not. You write the content, and then you send it to your subscribers.

我想将其视为订阅新闻通讯：作为新闻通讯的制作人，您知道谁订阅了您的新闻通讯，而谁没有订阅。 您编写内容，然后将其发送给您的订阅者。

On the other hand, as a subscriber, you might be subscribed to multiple newsletters, but you don’t know who the other subscribers are. But you don’t really care about that. This is a really nice feature because you can now write software that listens to a bunch of events and only responds to the ones you’re interested in.

另一方面，作为订阅者，您可能已订阅了多个新闻通讯，但您不知道其他订阅者是谁。 但是您并不真正在乎。 这是一个非常不错的功能，因为您现在可以编写侦听一堆事件并仅响应您感兴趣的事件的软件。

RabbitMQ and Amazon SQS (Simple Queuing Service) are some of the technologies often used for these types of use cases.

RabbitMQ和Amazon SQS(简单排队服务)是这些类型的用例中经常使用的一些技术。

分布式，容错，可扩展的基础架构 (Distributed, fault-tolerant, scalable infrastructure)

Distributed systems are prone to errors, and a queue is one of several ways to increase resiliency in the architecture. In a microservice architecture (or service-oriented architecture), multiple microservices communicate with each other through queues as shared interfaces.

分布式系统容易出错，队列是提高体系结构弹性的几种方法之一。 在微服务架构(或面向服务的架构 )中，多个微服务通过队列作为共享接口彼此通信。

When a microservice fails unexpectedly, a queue is still able to accept messages. This essentially provides a buffer for our microservice to recover. Once the microservice comes back online, it can pick up the messages from the queue and process them again.

当微服务意外失败时，队列仍然能够接受消息。 这实质上为我们的微服务提供了缓冲 。 一旦微服务重新联机，它可以从队列中提取消息并再次处理它们。

Think of it as your mailbox. While you’re out on vacation in Hawaii, the mailperson will still deliver your mail into the mailbox. Once you return from vacation, you can pick up the mail and process them at your leisure.

将其视为您的邮箱。 当您在夏威夷度假时，邮递员仍会将您的邮件传递到邮箱中。 度假归来后，您可以在闲暇时领取邮件并进行处理。

Thank you for reading! I hope you’ve learned a thing or two about distributed queues from my article. If you enjoyed reading this, please leave a clap, and feel free to join my newsletter here where I write about software and technical interviews!

感谢您的阅读！ 希望您从我的文章中学到了一两个关于分布式队列的知识。 如果您喜欢阅读本文，请鼓掌，并随时加入我的新闻通讯， 在其中撰写有关软件和技术面试的文章！

我推荐的资源 (Resources I Recommend)

To further your understanding of queues and various topics mentioned above, I would highly recommend these resources below. Or join my course on scaling distributed systems to learn more about queues :)

为了进一步了解上面提到的队列和各种主题，我强烈建议在下面使用这些资源。 或者参加我的扩展分布式系统课程 ，以了解有关队列的更多信息:)

• Designing Data-Intensive Applications: Awesome book for learning about scaling distributed systems! Highly recommended.

  设计数据密集型应用程序 ：很棒的书，用于学习有关扩展分布式系统的知识！ 强烈推荐。

• Kafka the Guide: I used this book as a reference guide, and enjoyed it for the high-level description.

  Kafka指南 ：我将本书用作参考指南，并从高层次的介绍中受益。

• Kafka Streams: This is an informative article from Confluent that talks in some high-level detail about Kafka’s implementation of stream processing.

  Kafka Streams ：这是Confluent的内容丰富的文章，详细介绍了Kafka的流处理实现。

• Elements of Programming Interviews: Great for solving coding problems.

  编程面试的要素 ：非常适合解决编码问题。

• Cracking The Coding Interview: Great for covering foundational CS coding problems.

  破解编码面试 ：非常适合解决基础CS编码问题。

• Daily Coding Problem.com: This is a free-to-try website that offers free daily coding problems. You can sign up for interesting daily coding challenges, and you can pay for solutions if you’d like. If you use my referral link (dailycodingproblem.com/zhiachong), you get $10 off!

  Daily Coding Problem.com ：这是一个免费试用的网站，提供免费的每日编码问题。 您可以报名参加有趣的每日编码挑战，也可以为解决方案付费。 如果您使用我的推荐链接( dailycodingproblem.com/zhiachong )，您将获得10美元的优惠！

(FYI, I share more resources on my website: zhiachong.com where I’ve personally tried and tested, and recommend for software engineers of all levels.)

(仅供参考，我在自己的网站zhiachong.com上共享了更多资源，我在该网站上进行了亲自尝试和测试，并向所有级别的软件工程师推荐。)

Cheers!

干杯!

翻译自: https://www.freecodecamp.org/news/a-dummys-guide-to-distributed-queues-2cd358d83780/

分布式队列