Why Map/Reduce?

本文回顾了MapReduce的发展历程,从Google首次提出到各大机构对其的研究与应用。介绍了MapReduce作为一种并行计算模型的价值所在,并列举了多个基于多核乃至GPU的研究案例。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

从Google在OSDI'04上发表了Map/Reduce这篇Paper;到Yahoo!在06年雇佣Doug Cotting,
投入对Hadoop最大的关注,以期提高其计算能力对抗Google;再到CMU CS的系主任发表Tec
h report提出DISC(Data-Intensive Super Computing,在最近一次的Hadoop summit上,
他演讲的slides将Super改成了Scalable)的概念,以及Intel、IBM与Stanford、CMU合作
基于Hadoop搞云计算。Map/Reduce引入的并行计算模型得到了前所未有的关注和应用。

究竟什么是Map/Reduce,为什么需要Map/Reduce,孟岩(此孟岩非彼孟岩,此孟岩是财帮
子的创始人之一)的这篇文章:Map Reduce - the Free Lunch is not over?阐述得很清
楚。(http://www.mengyan.org/blog/archives/2006/11/15/138.html)

随着du-core,multi-core甚至Stanford在做的1024核的出现,我们的Programming Model
会慢慢从OOP转到COP(concurrency-oriented programming)上来。而如何抽象这种progr
amming model的转化,让程序员更加方便的编写并行化程序利用多核甚至分布式系统的计
算能力,成为了很多研究的热点。Map/Reduce就是在这方面做出了不错的努力,这是它的
价值!

于是,在Map/Reduce发表没多久中,很多基于multi-core甚至GPU上的研究也在火热的展开


Phoenix: from Stanford

"Evaluating MapReduce for Multi-core and Multiprocessor Systems" —
paper by Colby Ranger, Ramanan Raghuraman, Arun Penmetsa, Gary Bradski, and
Christos Kozyrakis; from Stanford University

很有趣的是,这篇在当年会议上被推荐为best paper的论文居然出自于Stanford开的一门
课”Advanced Processor Design”上的project。(强大的stanford学生啊!)

Mars:from Hongkong University Science and Technology
(GPGPU很猖獗啊~)

# "Mars: A MapReduce Framework on Graphics Processors" — paper by Bingsheng
He, Wenbin Fang, Qiong Luo, Naga K. Govindaraju, Tuyong Wang; from Hong Kong
University of Science and Technology

Cell B.E. Architecture: from University of Wisconsin-Madison(不愧是CS
Architecture No 1的学校啊!)

# "MapReduce for the Cell B.E. Architecture" — paper by Marc de Kruijf and
Karthikeyan Sankaralingam; from University of Wisconsin-Madison

此外,还有一些项目也在应用Map/Reduce的思想。

QtConcurrent :http://labs.trolltech.com/page/Projects/Threads/QtCo
ncurrent
Open Source C++ MapReduce (non-distributed) implementation from Trolltech

Skynet:http://skynet.rubyforge.org/
Ruby Map/Reduce Framework

最牛的不是语言,不是代码,而是思想!还有Simplified的哲学!
学术界的力量也是不可低估的,很多很神奇很美妙的东西都来自于学术界。大家要好好珍
惜还在学习、做研究的机会。

Thanks to map/Reduce!
Thanks to lisp&other fp!
Thanks to lambda calculus!
Thanks to Alonzo Church!
PS D:\zhWork\202501gsykj\control_map> npm run serve > app@0.1.0 serve D:\zhWork\202501gsykj\control_map > vue-cli-service serve Browserslist: caniuse-lite is outdated. Please run: npx update-browserslist-db@latest Why you should do it regularly: https://github.com/browserslist/update-db#readme INFO Starting development server... [24%] building (3/381 modules) [24%] building (19/426 modules) [BABEL] Note: The code generator has deoptimised the styling of D:\zhWork\202501gsykj\control_map\node_modules\element-ui\lib\element-ui.common.js as it exceeds the max of[38%] building (486/800 modules) [BABEL] Note: The code generator has deoptimised the styling of D:\zhWork\202501gsykj\control_map\node_modules\lodash\lodash.js as it exceeds the max of 500KB. <--- Last few GCs ---> [4820:000001E7C4609C20] 54577 ms: Mark-sweep (reduce) 4088.9 (4101.7) -> 4088.5 (4102.7) MB, 495.5 / 0.0 ms (+ 0.1 ms in 323 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 655 ms) (average mu = 0.590, current mu = 0[4820:000001E7C4609C20] 55203 ms: Mark-sweep (reduce) 4089.5 (4104.7) -> 4089.0 (4105.2) MB, 620.5 / 0.0 ms (average mu = 0.379, current mu = 0.008) allocation failure scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory 1: 00007FF62825412F napi_wrap+133311 2: 00007FF6281EDD06 SSL_get_quiet_shutdown+63062 3: 00007FF6281EEB9D node::OnFatalError+301 4: 00007FF628AD19CE v8::Isolate::ReportExternalAllocationLimitReached+94 5: 00007FF628AB67BD v8::SharedArrayBuffer::Externalize+781 6: 00007FF62895FFCC v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1516 7: 00007FF62894AA4B v8::internal::NativeContextInferrer::Infer+59739 8: 00007FF62892FD1F v8::internal::MarkingWorklists::SwitchToContextSlow+56975 9: 00007FF628943A0B v8::internal::NativeContextInferrer::Infer+31003 10: 00007FF62893AAED v8::internal::MarkCompactCollector::EnsureSweepingCompleted+6285 11: 00007FF628942C5E v8::internal::NativeContextInferrer::Infer+27502 12: 00007FF628946CAB v8::internal::NativeContextInferrer::Infer+43963 13: 00007FF6289506A2 v8::internal::ItemParallelJob::Task::RunInternal+18 14: 00007FF628950631 v8::internal::ItemParallelJob::Run+641 15: 00007FF628923C93 v8::internal::MarkingWorklists::SwitchToContextSlow+7683 16: 00007FF62893AF9C v8::internal::MarkCompactCollector::EnsureSweepingCompleted+7484 17: 00007FF6289397D4 v8::internal::MarkCompactCollector::EnsureSweepingCompleted+1396 18: 00007FF628937348 v8::internal::MarkingWorklists::SwitchToContextSlow+87224 19: 00007FF628965DB1 v8::internal::Heap::LeftTrimFixedArray+929 20: 00007FF628967EA5 v8::internal::Heap::PageFlagsAreConsistent+789 21: 00007FF62895D0C1 v8::internal::Heap::CollectGarbage+2049 22: 00007FF62895B2C5 v8::internal::Heap::AllocateExternalBackingStore+1349 23: 00007FF628975155 v8::internal::GCIdleTimeHandler::ShouldDoContextDisposalMarkCompact+1029 24: 00007FF6289755A5 v8::internal::Factory::AllocateRaw+37 25: 00007FF628986DA6 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray+38 26: 00007FF628986EBF v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawFixedArray+31 27: 00007FF6289769E1 v8::internal::Factory::CopyFixedArrayAndGrow+65 28: 00007FF62879F7E4 v8::internal::WeakArrayList::AddToEnd+3300 29: 00007FF6289ACC2A v8::internal::Isolate::CaptureSimpleStackTrace+1146 30: 00007FF6289AC971 v8::internal::Isolate::CaptureSimpleStackTrace+449 31: 00007FF6289AC0BD v8::internal::Isolate::CaptureAndSetSimpleStackTrace+45 32: 00007FF6289A28A0 v8::internal::MessageLocation::MessageLocation+3664 33: 00007FF6289A63BC v8::internal::MessageFormatter::Format+13100 34: 00007FF62897ADB6 v8::internal::Factory::NewError+134 35: 00007FF628984AF2 v8::internal::Factory::NewTypeError+162 36: 00007FF6289A6561 v8::internal::MessageHandler::MakeMessageObject+337 37: 00007FF6286ABF7C v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+9596 38: 00007FF628B5CDFD v8::internal::SetupIsolateDelegate::SetupHeap+474253 39: 00007FF628AED1C9 v8::internal::SetupIsolateDelegate::SetupHeap+16473 40: 00007FF628AF2FC2 v8::internal::SetupIsolateDelegate::SetupHeap+40530 41: 0000010772004690 npm ERR! code ELIFECYCLE npm ERR! errno 134 npm ERR! app@0.1.0 serve: `vue-cli-service serve` npm ERR! Exit status 134 npm ERR! npm ERR! Failed at the app@0.1.0 serve script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm ERR! A complete log of this run can be found in: npm ERR! D:\softwareInstall\nvmInstall\nodejs\node_cache\_logs\2025-07-19T15_03_28_775Z-debug.log
07-20
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值