AWS & ASP.NET

最新推荐文章于 2024-09-26 07:22:44 发布

weixin_34408624

最新推荐文章于 2024-09-26 07:22:44 发布

阅读量202

点赞数

文章标签： java 大数据数据库

本文详细介绍使用亚马逊云服务构建大数据消息传递及聚合系统的架构方案。涵盖了Kinesis数据流处理、S3对象存储、DynamoDB键值存储、Elastic MapReduce数据处理框架以及RedShift数据仓库等关键技术组件。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

https://dotnetcodr.com/amazon-cloud/

Amazon cloud

Big Data overall architecture

Architecture of a Big Data messaging and aggregation system using Amazon Web Services part 1
Architecture of a Big Data messaging and aggregation system using Amazon Web Services part 2
Architecture of a Big Data messaging and aggregation system using Amazon Web Services part 3
Architecture of a Big Data messaging and aggregation system using Amazon Web Services part 4
Architecture of a Big Data messaging and aggregation system using Amazon Web Services part 5

Amazon Big Data components and code

The message handler: Kinesis

Big Data: using Amazon Kinesis with the AWS.NET API Part 1: introduction
Using Amazon Kinesis with the AWS.NET API Part 2: stream, NET SDK and domain setup
Big Data: using Amazon Kinesis with the AWS.NET API Part 3: sending to the stream
Big Data: using Amazon Kinesis with the AWS.NET API Part 4: reading from the stream
Using Amazon Kinesis with the AWS.NET API Part 5: validation
Using Amazon Kinesis with the AWS.NET API Part 6: storage

The raw data storage: S3

Data storage alternative: DynamoDb

Using Amazon DynamoDb with the AWS.NET API Part 1: introduction
Using Amazon DynamoDb with the AWS.NET API Part 2: code beginnings
Using Amazon DynamoDb with the AWS.NET API Part 3: table operations
Using Amazon DynamoDb with the AWS .NET API Part 4: record insertion
Using Amazon DynamoDb with the AWS .NET API Part 5: updating and deleting records
Using Amazon DynamoDb with the AWS .NET API Part 6: queries
Using Amazon DynamoDb with the AWS .NET API Part 7: its place in Big Data

Data mining and analysis tool: Elastic MapReduce

Using Amazon Elastic MapReduce with the AWS.NET API Part 1: introduction
Using Amazon Elastic MapReduce with the AWS.NET API Part 2: the cluster startup GUI
Using Amazon Elastic MapReduce with the AWS.NET API Part 3: starting and logging into a cluster
Using Amazon Elastic MapReduce with the AWS.NET API Part 4: Hive basics with Hadoop
Using Amazon Elastic MapReduce with the AWS .NET API Part 5: cluster-related code
Using Amazon Elastic MapReduce with the AWS .NET API Part 6: Hive with Amazon S3 and DynamoDb
Using Amazon Elastic MapReduce with the AWS .NET API Part 7: indirect Hive with .NET
Using Amazon Elastic MapReduce with the AWS .NET API Part 8: connection to our Big Data demo

Data mining and analysis tool: RedShift

Using Amazon RedShift with the AWS .NET API Part 1: introduction
Using Amazon RedShift with the AWS .NET API Part 2: MPP definition and first cluster
Using Amazon RedShift with the AWS .NET API Part 3: connecting to the master node
Using Amazon RedShift with the AWS .NET API Part 4: code beginnings
Using Amazon RedShift with the AWS .NET API Part 5: connecting to master node using ODBC
Using Amazon RedShift with the AWS .NET API Part 6: Postgresql to master node using ODBC
Using Amazon RedShift with the AWS .NET API Part 7: data warehousing and the star schema
Using Amazon RedShift with the AWS .NET API Part 8: data warehousing and the star schema 2
Using Amazon RedShift with the AWS .NET API Part 9: data warehousing and the star schema 3
Using Amazon RedShift with the AWS .NET API Part 10: RedShift in Big Data

AWS Big Data summary

Amazon CodePipeline

Introduction to Amazon Code Pipeline with Java part 1: basics of CI/CD
Introduction to Amazon Code Pipeline with Java part 2: setup
Introduction to Amazon Code Pipeline with Java part 3: adding custom job runners
Introduction to Amazon Code Pipeline with Java part 4: comparison with TeamCity and Jenkins
Introduction to Amazon Code Pipeline with Java part 5: architecture key terms
Introduction to Amazon Code Pipeline with Java part 6: third party action overview
Introduction to Amazon Code Pipeline with Java part 7: the third party action user signup process
Introduction to Amazon Code Pipeline with Java part 8: the job agent communication process
Introduction to Amazon Code Pipeline with Java part 9: the job agent continuation token
Introduction to Amazon Code Pipeline with Java part 10: the client web pages
Introduction to Amazon Code Pipeline with Java part 11: starting with the job agent
Introduction to Amazon Code Pipeline with Java part 12: the job agent entry point in code
Introduction to Amazon Code Pipeline with Java part 13: the client token lookup service
Introduction to Amazon Code Pipeline with Java part 14: the loadtest executor service
Introduction to Amazon Code Pipeline with Java part 15: the job processor interface and related objects

Geo-spatial services

Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 1: introduction and goals
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 2: MaxMind source files
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 3: IPv4 range strategy
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 4: lng/lat range strategy
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 5: creating the IPv4 source file for DynamoDb
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 6: uploading IPv4 range to DynamoDb
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 7: querying the IPv4 range table
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 8: creating the lng/lat coordinates source file for DynamoDb
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 9: uploading the co-ordinate range to DynamoDb
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 10: querying the coordinate range table
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 11: uploading the geolocation range to DynamoDb
Using Amazon DynamoDb for IP and co-ordinate based geo-location services part 12: querying the geolocation range to DynamoDb

AMIs

How to manage Amazon Machine Images with the .NET Amazon SDK Part 1: starting an image instance
How to manage Amazon Machine Images with the .NET Amazon SDK Part 2: monitoring and terminating AMI instances, managing Security Groups

Data Pipeline