Create Local Cloudera Parcels Repo to Save Your ASS

最新推荐文章于 2023-07-10 08:16:56 发布

puffsun

最新推荐文章于 2023-07-10 08:16:56 发布

阅读量2.7w

点赞数

CC 4.0 BY-SA版权

分类专栏： Hadoop 文章标签：大数据 json

本文链接：https://blog.youkuaiyun.com/puffsun/article/details/84455958

Hadoop 专栏收录该内容

57 篇文章

订阅专栏

在中国通过Cloudera Manager安装Hadoop集群时，由于网络问题，下载Cloudera Parcels非常慢。为了解决这个问题，你需要创建本地Parcel仓库。按照Cloudera的指导，下载相应操作系统的parcel文件，修改Cloudera Manager的Local Parcel Repository Path，将parcel文件和对应的manifest.json及SHA校验文件放入仓库中。完成这些步骤后，Cloudera Manager会检测到本地仓库并显示在Parcels页面上。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

If you want to install Hadoop cluster via Cloudera Manager in CHINA, the network to cloudera parels is toooooo slow. You must create your local parcel repo to save your ass, thanks to the DAMN GFW!!

Follow this page from cloudera:

Download parcels:

Link to CDH4： http://archive.cloudera.com/cdh4/parcels/
Link to IMPALA： http://archive.cloudera.com/impala/parcels/

Verify the location of the local parcel repository on your Cloudera Manager server: Go to the Administration page -> Properties tab -> Parcels category. You can change the local repository path in the Local Parcel Repository Path property. By default it is /opt/cloudera/parcel-repo.
Download the .parcel file for your operating system: (el5 or el6 for Red Hat 5 or 6, lucid or precise for Ubuntu and so on) and place it into the local parcel repository on your Cloudera Manager server. In my case it's CDH-4.3.0-1.cdh4.3.0.p0.22-el6.parcel and IMPALA-1.0.1-1.p0.431-el6.parcel

Open the manifest.json file in the same directory as the .parcel file you just copied. Find the section of the manifest that corresponds to the parcel you downloaded: In my case, I'm running RHEL 6 and copied the parcel file CDH-4.2.0-1.cdh4.2.0.p0.10-el6.parcel, then look for the section:

  {
      "parcelName": "CDH-4.3.0-1.cdh4.3.0.p0.22-el6.parcel",
      "components": [
        { "name":     "flume-ng",
          "version":  "1.3.0-cdh4.3.0",
          "pkg_version":  "1.3.0+159" 
        }
        ,{ "name":     "hadoop-0.20-mapreduce",
          "version":  "2.0.0-cdh4.3.0",
          "pkg_version":  "2.0.0+1357" 
        }
        ,{ "name":     "hadoop-hdfs",
          "version":  "2.0.0-cdh4.3.0",
          "pkg_version":  "2.0.0+1357" 
        }
        ,{ "name":     "hadoop-httpfs",
          "version":  "2.0.0-cdh4.3.0",
          "pkg_version":  "2.0.0+1357" 
        }
        ,{ "name":     "hadoop-mapreduce",
          "version":  "2.0.0-cdh4.3.0",
          "pkg_version":  "2.0.0+1357" 
        }
        ,{ "name":     "hadoop-yarn",
          "version":  "2.0.0-cdh4.3.0",
          "pkg_version":  "2.0.0+1357" 
        }
        ,{ "name":     "hadoop",
          "version":  "2.0.0-cdh4.3.0",
          "pkg_version":  "2.0.0+1357" 
        }
        ,{ "name":     "hbase",
          "version":  "0.94.6-cdh4.3.0",
          "pkg_version":  "0.94.6+96" 
        }
        ,{ "name":     "hcatalog",
          "version":  "0.5.0-cdh4.3.0",
          "pkg_version":  "0.5.0+9" 
        }
        ,{ "name":     "hive",
          "version":  "0.10.0-cdh4.3.0",
          "pkg_version":  "0.10.0+121" 
        }
        ,{ "name":     "mahout",
          "version":  "0.7-cdh4.3.0",
          "pkg_version":  "0.7+16" 
        }
        ,{ "name":     "oozie",
          "version":  "3.3.2-cdh4.3.0",
          "pkg_version":  "3.3.2+49" 
        }
        ,{ "name":     "pig",
          "version":  "0.11.0-cdh4.3.0",
          "pkg_version":  "0.11.0+28" 
        }
        ,{ "name":     "sqoop",
          "version":  "1.4.3-cdh4.3.0",
          "pkg_version":  "1.4.3+34" 
        }
        ,{ "name":     "sqoop2",
          "version":  "1.99.1-cdh4.3.0",
          "pkg_version":  "1.99.1+115" 
        }
        ,{ "name":     "whirr",
          "version":  "0.8.2-cdh4.3.0",
          "pkg_version":  "0.8.2+10" 
        }
        ,{ "name":     "zookeeper",
          "version":  "3.4.5-cdh4.3.0",
          "pkg_version":  "3.4.5+19" 
        }
        ,{ "name":     "hue",
          "version":  "2.3.0-cdh4.3.0",
          "pkg_version":  "2.3.0+136" 
        }
      ],
      "hash": "df5cc61b2d257aaf625341f709a4f8e09754038a"
    }

  {
      "parcelName": "IMPALA-1.0.1-1.p0.431-el6.parcel",
      "components": [
        { "name":     "impala",
          "version":  "1.0.1-SNAPSHOT",
          "pkg_version":  "1.0.1"
        }
      ],
      "hash": "992467f2e54bd394cbdd3f4ed97b6e9bead60ff0"
    }

Create a text file whose name is <parcel file name> .sha (e.g. CDH-4.2.0-1.cdh4.2.0.p0.10-el6.parcel.sha) and copy the hash code into it. In my case, I created file CDH-4.3.0-1.cdh4.3.0.p0.22-el6.parcel.sha and IMPALA-1.0.1-1.p0.431-el6.parcel.sha, then I copied the hash value into them respectively:
```
# cat > CDH-4.3.0-1.cdh4.3.0.p0.22-el6.parcel.sha
df5cc61b2d257aaf625341f709a4f8e09754038a
^C
```
```
# cat > IMPALA-1.0.1-1.p0.431-el6.parcel.sha
992467f2e54bd394cbdd3f4ed97b6e9bead60ff0
^C
```
Place both of files into your local parcel repository, /opt/cloudera/parcel-repo by default, depending you settings in the above step.
Once these files are in place, Cloudera Manager will pick up the parcel and it will appear on the Hosts > Parcels page. Note that how quickly this occurs depends on the Parcel Update Frequency setting, set by default to 1 hour. You can change this on the Administration page -> Properties tab -> Parcels category.