基于 Databend 和腾讯云 COS 打造新型云数仓

开源、云原生且实时分析型的现代数据仓库DataBend的介绍,及其与其它开源文件存储的结合使用实例_databend功能 Databend 是一个开源、云原生且实时分析型的现代数据仓库,旨在提供高效的数据存储和处理能力。它采用 Rust 语言开发,并支持 Apache Arrow 格式以实现高性能列式存储与查询处理。主要特点:云原生设计:Databend 构建在 Kubernetes 之上,具备弹性伸缩、水平扩展的能力,可轻松部署在公有云或私有云环境中。SQL 支持:Databend 具备完善的 SQL 查询功能,兼容 MySQL 协议,使得用户可以利用熟悉的 SQL 语法进行数据查询和处理。性能优化。 阅读详情

图片

本篇文章向大家演示如何使用 Databend 基于腾讯云 COS 构建新式数仓及其计算能力。如果你也在找一个低成本、高性能、支持弹性的数仓,Databend 可以为大家提供一个基于对象存储的云原生数仓解决方案。目前 Databend 支持数据的 stream load , copy into from stage , insert 等方式的数据写入,部署上支持单机和集群模式。需要更多支持添加微信: 82565387 。 文章较长,建议收藏 PC 端阅读。

Databend 介绍

Databend 是一款使用 Rust 研发、开源、完全面向对象存储架构的新式数仓,提供极速的弹性扩展能力,致力于打造按需、按量的 Data Cloud 产品体验。具备以下特点:

•Vectorized Execution 和 Pull&Push-Based Processor Model

•真正的存储、计算分离架构,高性能、低成本,按需按量使用

•完整的数据库支持,兼容 MySQL ,Clickhouse 协议, SQL Over http

•完善的事务性,支持 Data Time Travel, Database Zero Clone 等功能

•支持基于同一份数据的多租户读写、共享操作

github repo: https://github.com/datafuselabs/databend

Docs : https://databend.rs

关于 Databend 架构图,参考:https://databend.rs/doc/

腾讯云 COS

对象存储(Cloud Object Storage,COS)是由腾讯云推出的无目录层次结构、无数据格式限制,可容纳海量数据且支持 HTTP/HTTPS 协议访问的分布式存储服务。腾讯云 COS 的存储桶空间无容量上限,无需分区管理,适用于 CDN 数据分发、数据万象处理或大数据计算与分析的数据湖等多种场景。

官网:https://cloud.tencent.com/product/cos

测试环境介绍

北京区: CVM SA2.8XLARGE64 & COS(ap-beijing)

操作系统: ubuntu-20

Databend : 使用进二制发布版本 v0.6.99-nightly

下载地址:https://repo.databend.rs/databend/v0.6.99-nightly/databend-v0.6.99-nightly-x86_64-unknown-linux-gnu.tar.gz

本次测试安装部署方式参考:https://databend.rs/doc/deploy/cos

集群部署模式参考:https://databend.rs/doc/deploy/cluster_minio

测试数据

wget --no-check-certificate --continue https://transtats.bts.gov/PREZIP/\
On_Time_Reporting_Carrier_On_Time_Performance_1987_present_{1987..2021}_{1..12}.zip

表结构参考:cat create_ontime.sql

CREATE TABLE ontime
(
    Year                            UInt16 NOT NULL,
    Quarter                         UInt8 NOT NULL,
    Month                           UInt8 NOT NULL,
    DayofMonth                      UInt8 NOT NULL,
    DayOfWeek                       UInt8 NOT NULL,
    FlightDate                      Date NOT NULL,
    Reporting_Airline               String NOT NULL,
    DOT_ID_Reporting_Airline        Int32 NOT NULL,
    IATA_CODE_Reporting_Airline     String NOT NULL,
    Tail_Number                     String NOT NULL,
    Flight_Number_Reporting_Airline String NOT NULL,
    OriginAirportID                 Int32 NOT NULL,
    OriginAirportSeqID              Int32 NOT NULL,
    OriginCityMarketID              Int32 NOT NULL,
    Origin                          String NOT NULL,
    OriginCityName                  String NOT NULL,
    OriginState                     String NOT NULL,
    OriginStateFips                 String NOT NULL,
    OriginStateName                 String NOT NULL,
    OriginWac                       Int32 NOT NULL,
    DestAirportID                   Int32 NOT NULL,
    DestAirportSeqID                Int32 NOT NULL,
    DestCityMarketID                Int32 NOT NULL,
    Dest                            String NOT NULL,
    DestCityName                    String NOT NULL,
    DestState                       String NOT NULL,
    DestStateFips                   String NOT NULL,
    DestStateName                   String NOT NULL,
    DestWac                         Int32 NOT NULL,
    CRSDepTime                      Int32 NOT NULL,
    DepTime                         Int32 NOT NULL,
    DepDelay                        Int32 NOT NULL,
    DepDelayMinutes                 Int32 NOT NULL,
    DepDel15                        Int32 NOT NULL,
    DepartureDelayGroups            String NOT NULL,
    DepTimeBlk                      String NOT NULL,
    TaxiOut                         Int32 NOT NULL,
    WheelsOff                       Int32 NOT NULL,
    WheelsOn                        Int32 NOT NULL,
    TaxiIn                          Int32 NOT NULL,
    CRSArrTime                      Int32 NOT NULL,
    ArrTime                         Int32 NOT NULL,
    ArrDelay                        Int32 NOT NULL,
    ArrDelayMinutes                 Int32 NOT NULL,
    ArrDel15                        Int32 NOT NULL,
    ArrivalDelayGroups              Int32 NOT NULL,
    ArrTimeBlk                      String NOT NULL,
    Cancelled                       UInt8 NOT NULL,
    CancellationCode                String NOT NULL,
    Diverted                        UInt8 NOT NULL,
    CRSElapsedTime                  Int32 NOT NULL,
    ActualElapsedTime               Int32 NOT NULL,
    AirTime                         Int32 NOT NULL,
    Flights                         Int32 NOT NULL,
    Distance                        Int32 NOT NULL,
    DistanceGroup                   UInt8 NOT NULL,
    CarrierDelay                    Int32 NOT NULL,
    WeatherDelay                    Int32 NOT NULL,
    NASDelay                        Int32 NOT NULL,
    SecurityDelay                   Int32 NOT NULL,
    LateAircraftDelay               Int32 NOT NULL,
    FirstDepTime                    String NOT NULL,
    TotalAddGTime                   String NOT NULL,
    LongestAddGTime                 String NOT NULL,
    DivAirportLandings              String NOT NULL,
    DivReachedDest                  String NOT NULL,
    DivActualElapsedTime            String NOT NULL,
    DivArrDelay                     String NOT NULL,
    DivDistance                     String NOT NULL,
    Div1Airport                     String NOT NULL,
    Div1AirportID                   Int32 NOT NULL,
    Div1AirportSeqID                Int32 NOT NULL,
    Div1WheelsOn                    String NOT NULL,
    Div1TotalGTime                  String NOT NULL,
    Div1LongestGTime                String NOT NULL,
    Div1WheelsOff                   String NOT NULL,
    Div1TailNum                     String NOT NULL,
    Div2Airport                     String NOT NULL,
    Div2AirportID                   Int32 NOT NULL,
    Div2AirportSeqID                Int32 NOT NULL,
    Div2WheelsOn                    String NOT NULL,
    Div2TotalGTime                  String NOT NULL,
    Div2LongestGTime                String NOT NULL,
    Div2WheelsOff                   String NOT NULL,
    Div2TailNum                     String NOT NULL,
    Div3Airport                     String NOT NULL,
    Div3AirportID                   Int32 NOT NULL,
    Div3AirportSeqID                Int32 NOT NULL,
    Div3WheelsOn                    String NOT NULL,
    Div3TotalGTime                  String NOT NULL,
    Div3LongestGTime                String NOT NULL,
    Div3WheelsOff                   String NOT NULL,
    Div3TailNum                     String NOT NULL,
    Div4Airport                     String NOT NULL,
    Div4AirportID                   Int32 NOT NULL,
    Div4AirportSeqID                Int32 NOT NULL,
    Div4WheelsOn                    String NOT NULL,
    Div4TotalGTime                  String NOT NULL,
    Div4LongestGTime                String NOT NULL,
    Div4WheelsOff                   String NOT NULL,
    Div4TailNum                     String NOT NULL,
    Div5Airport                     String NOT NULL,
    Div5AirportID                   Int32 NOT NULL,
    Div5AirportSeqID                Int32 NOT NULL,
    Div5WheelsOn                    String NOT NULL,
    Div5TotalGTime                  String NOT NULL,
    Div5LongestGTime                String NOT NULL,
    Div5WheelsOff                   String NOT NULL,
    Div5TailNum                     String NOT NULL
);

加载表结构:

cat create_ontime.sql | mysql -h127.0.0.1 -P3307 -uroot

数据加载

cat load_ontime.sh

echo "unzip ontime ,input your ontime zip dir: ./load_ontime.sh zip_dir"

ls $1/*.zip |xargs -I{} -P 4 bash -c "echo {}; unzip -q {} '*.csv' -d ./dataset"

if [ $? -eq  0 ];
then
    echo "unzip success"
else
    echo "unzip was wrong!!!"
    exit 1
fi

cat create_ontime.sql |mysql -h127.0.0.1 -P3307 -uroot
if [ $? -eq  0 ];
then
    echo "Ontime table create success"
else
    echo "Ontime table create was wrong!!!"
    exit 1
fi


time ls ./dataset/*.csv|xargs -P 8 -I{} curl -H "insert_sql:insert into ontime format CSV" -H "skip_header:1" -F "upload=@{}" -XPUT http://localhost:8081/v1/streaming_load

使用方法

./load_ontime.sh ZIP文件目录

基于 Ontime 测试 SQL 展示

Q1 查询2000年到2008年每天的总的航班总

(0.494 sec., 143.75 million rows/sec., 431.25 MB/sec)

mysql> SELECT DayOfWeek, count(*) AS c FROM ontime WHERE Year >= 2000 AND Year <= 2008 GROUP BY DayOfWeek ORDER BY c DESC;
+-----------+---------+
| DayOfWeek | c       |
+-----------+---------+
|         5 | 8732422 |
|         1 | 8730614 |
|         4 | 8710843 |
|         3 | 8685626 |
|         2 | 8639632 |
|         7 | 8274367 |
|         6 | 7514194 |
+-----------+---------+
7 rows in set (0.50 sec)
Read 71000000 rows, 213 MB in 0.494 sec., 143.75 million rows/sec., 431.25 MB/sec.

mysql> explain SELECT DayOfWeek, count(*) AS c FROM ontime WHERE Year >= 2000 AND Year <= 2008 GROUP BY DayOfWeek ORDER BY c DESC;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                                                                           |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection: DayOfWeek:UInt8, count() as c:UInt64                                                                                                                                                                                                                  |
|   Sort: count():UInt64                                                                                                                                                                                                                                            |
|     AggregatorFinal: groupBy=[[DayOfWeek]], aggr=[[count()]]                                                                                                                                                                                                      |
|       AggregatorPartial: groupBy=[[DayOfWeek]], aggr=[[count()]]                                                                                                                                                                                                  |
|         Filter: ((Year >= 2000) and (Year <= 2008))                                                                                                                                                                                                               |
|           ReadDataSource: scan schema: [Year:UInt16, DayOfWeek:UInt8], statistics: [read_rows: 71000000, read_bytes: 213000000, partitions_scanned: 71, partitions_total: 207], push_downs: [projections: [0, 4], filters: [((Year >= 2000) AND (Year <= 2008))]] |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)

Q2 查询 2000 年到 2008 年延迟超过 10 分钟,每天总的延迟发生情况

( 0.543 sec., 130.71 million rows/sec., 914.95 GB/sec.)

mysql> SELECT DayOfWeek, count(*) AS c FROM ontime WHERE DepDelay>10 AND Year >= 2000 AND Year <= 2008 GROUP BY DayOfWeek ORDER BY c DESC;
+-----------+---------+
| DayOfWeek | c       |
+-----------+---------+
|         5 | 2175733 |
|         4 | 2012848 |
|         1 | 1898879 |
|         7 | 1880896 |
|         3 | 1757508 |
|         2 | 1665303 |
|         6 | 1510894 |
+-----------+---------+
7 rows in set (0.54 sec)
Read 71000000 rows, 497 MB in 0.543 sec., 130.71 million rows/sec., 914.95 MB/sec.

mysql> explain SELECT DayOfWeek, count(*) AS c FROM ontime WHERE DepDelay>10 AND Year >= 2000 AND Year <= 2008 GROUP BY DayOfWeek ORDER BY c DESC;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                                                                                                                     |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection: DayOfWeek:UInt8, count() as c:UInt64                                                                                                                                                                                                                                                            |
|   Sort: count():UInt64                                                                                                                                                                                                                                                                                      |
|     AggregatorFinal: groupBy=[[DayOfWeek]], aggr=[[count()]]                                                                                                                                                                                                                                                |
|       AggregatorPartial: groupBy=[[DayOfWeek]], aggr=[[count()]]                                                                                                                                                                                                                                            |
|         Filter: (((DepDelay > 10) and (Year >= 2000)) and (Year <= 2008))                                                                                                                                                                                                                                   |
|           ReadDataSource: scan schema: [Year:UInt16, DayOfWeek:UInt8, DepDelay:Int32], statistics: [read_rows: 71000000, read_bytes: 497000000, partitions_scanned: 71, partitions_total: 207], push_downs: [projections: [0, 4, 31], filters: [(((DepDelay > 10) AND (Year >= 2000)) AND (Year <= 2008))]] |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q3 2000-2008年机场的延误次数,显示最高的10条

(0.679 sec., 104.59 million rows/sec., 1.78 GB/sec.)

mysql> SELECT Origin, count(*) AS c FROM ontime WHERE DepDelay>10 AND Year >= 2000 AND Year <= 2008 GROUP BY Origin ORDER BY c DESC LIMIT 10;
+--------+--------+
| Origin | c      |
+--------+--------+
| ORD    | 860911 |
| ATL    | 831822 |
| DFW    | 614403 |
| LAX    | 402671 |
| PHX    | 400475 |
| LAS    | 362026 |
| DEN    | 352893 |
| EWR    | 302267 |
| DTW    | 296832 |
| IAH    | 290729 |
+--------+--------+
10 rows in set (0.69 sec)
Read 71000000 rows, 1.21 GB in 0.679 sec., 104.59 million rows/sec., 1.78 GB/sec.

mysql> explain SELECT Origin, count(*) AS c FROM ontime WHERE DepDelay>10 AND Year >= 2000 AND Year <= 2008 GROUP BY Origin ORDER BY c DESC LIMIT 10;
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                                                                                                                       |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Limit: 10                                                                                                                                                                                                                                                                                                     |
|   Projection: Origin:String, count() as c:UInt64                                                                                                                                                                                                                                                              |
|     Sort: count():UInt64                                                                                                                                                                                                                                                                                      |
|       AggregatorFinal: groupBy=[[Origin]], aggr=[[count()]]                                                                                                                                                                                                                                                   |
|         AggregatorPartial: groupBy=[[Origin]], aggr=[[count()]]                                                                                                                                                                                                                                               |
|           Filter: (((DepDelay > 10) and (Year >= 2000)) and (Year <= 2008))                                                                                                                                                                                                                                   |
|             ReadDataSource: scan schema: [Year:UInt16, Origin:String, DepDelay:Int32], statistics: [read_rows: 71000000, read_bytes: 1271665856, partitions_scanned: 71, partitions_total: 207], push_downs: [projections: [0, 14, 31], filters: [(((DepDelay > 10) AND (Year >= 2000)) AND (Year <= 2008))]] |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
7 rows in set (0.00 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q4 2007年各航空公司延误的次数

(0.188 sec., 79.77 million rows/sec., 1.28 GB/sec.)

mysql> SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
+---------+---------+
| Carrier | count() |
+---------+---------+
| WN      |  296451 |
| AA      |  179769 |
| MQ      |  152293 |
| OO      |  147019 |
| US      |  140199 |
| UA      |  135061 |
| XE      |  108571 |
| EV      |  104055 |
| NW      |  102206 |
| DL      |   98427 |
| CO      |   81039 |
| YV      |   79553 |
| FL      |   64583 |
| OH      |   60532 |
| AS      |   54326 |
| B6      |   53716 |
| 9E      |   48578 |
| F9      |   24100 |
| AQ      |    6764 |
| HA      |    4059 |
+---------+---------+
20 rows in set (0.19 sec)
Read 15000000 rows, 240 MB in 0.188 sec., 79.77 million rows/sec., 1.28 GB/sec.

mysql> explain SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                                                                                                                  |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection: IATA_CODE_Reporting_Airline as Carrier:String, count():UInt64                                                                                                                                                                                                                                |
|   Sort: count():UInt64                                                                                                                                                                                                                                                                                   |
|     AggregatorFinal: groupBy=[[IATA_CODE_Reporting_Airline]], aggr=[[count()]]                                                                                                                                                                                                                           |
|       AggregatorPartial: groupBy=[[IATA_CODE_Reporting_Airline]], aggr=[[count()]]                                                                                                                                                                                                                       |
|         Filter: ((DepDelay > 10) and (Year = 2007))                                                                                                                                                                                                                                                      |
|           ReadDataSource: scan schema: [Year:UInt16, IATA_CODE_Reporting_Airline:String, DepDelay:Int32], statistics: [read_rows: 15000000, read_bytes: 250239306, partitions_scanned: 15, partitions_total: 207], push_downs: [projections: [0, 8, 31], filters: [((DepDelay > 10) AND (Year = 2007))]] |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.00 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q5 2007年各航空公司延误的千分比

(0.265 sec., 56.58 million rows/sec., 905.28 MB/sec.)

mysql> SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(cast(DepDelay>10 as Int8))*1000 AS c3 FROM ontime WHERE Year=2007 GROUP BY Carrier ORDER BY c3 DESC;
+---------+--------------------+
| Carrier | c3                 |
+---------+--------------------+
| EV      | 363.53123668047823 |
| AS      |  339.1453631738303 |
| US      |  288.8039271022377 |
| AA      |  283.6112877194699 |
| MQ      |  281.7663100792978 |
| B6      |  280.5745625489684 |
| UA      | 275.63356884257615 |
| YV      | 270.25567158804466 |
| OH      |  256.4567516268981 |
| WN      | 253.62165713752844 |
| CO      | 250.77750030171651 |
| XE      | 249.71881878589517 |
| NW      | 246.56113247419944 |
| F9      | 246.52209492635023 |
| OO      | 245.90051515354253 |
| FL      |  245.4143692596491 |
| DL      | 206.82764258051773 |
| 9E      | 187.66780889391967 |
| AQ      |  145.9016393442623 |
| HA      |  72.25634178905207 |
+---------+--------------------+
20 rows in set (0.27 sec)
Read 15000000 rows, 240 MB in 0.265 sec., 56.58 million rows/sec., 905.28 MB/sec.

mysql> explain SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(cast(DepDelay>10 as Int8))*1000 AS c3 FROM ontime WHERE Year=2007 GROUP BY Carrier ORDER BY c3 DESC;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                                                                                                |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection: IATA_CODE_Reporting_Airline as Carrier:String, (avg(cast((DepDelay > 10) as Int8)) * 1000) as c3:Float64                                                                                                                                                                   |
|   Sort: (avg(cast((DepDelay > 10) as Int8)) * 1000):Float64                                                                                                                                                                                                                            |
|     Expression: IATA_CODE_Reporting_Airline:String, (avg(cast((DepDelay > 10) as Int8)) * 1000):Float64 (Before OrderBy)                                                                                                                                                               |
|       AggregatorFinal: groupBy=[[IATA_CODE_Reporting_Airline]], aggr=[[avg(cast((DepDelay > 10) as Int8))]]                                                                                                                                                                            |
|         AggregatorPartial: groupBy=[[IATA_CODE_Reporting_Airline]], aggr=[[avg(cast((DepDelay > 10) as Int8))]]                                                                                                                                                                        |
|           Expression: IATA_CODE_Reporting_Airline:String, cast((DepDelay > 10) as Int8):Int8 (Before GroupBy)                                                                                                                                                                          |
|             Filter: (Year = 2007)                                                                                                                                                                                                                                                      |
|               ReadDataSource: scan schema: [Year:UInt16, IATA_CODE_Reporting_Airline:String, DepDelay:Int32], statistics: [read_rows: 15000000, read_bytes: 250239306, partitions_scanned: 15, partitions_total: 207], push_downs: [projections: [0, 8, 31], filters: [(Year = 2007)]] |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
8 rows in set (0.00 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q6 2000-2008年各航空公司延误的千分比

(0.935 sec., 75.95 million rows/sec., 1.22 GB/sec.)

mysql> SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(cast(DepDelay>10 as Int8))*1000 AS c3 FROM ontime WHERE Year>=2000 AND Year <=2008 GROUP BY Carrier ORDER BY c3 DESC;
+---------+--------------------+
| Carrier | c3                 |
+---------+--------------------+
| AS      | 293.05649076611434 |
| EV      |  282.0709981074399 |
| YV      |  270.3897636688929 |
| B6      | 257.40594891667007 |
| FL      | 249.28742951361826 |
| XE      | 246.59005902424192 |
| MQ      |  245.3695989400477 |
| WN      | 233.38127235928863 |
| DH      | 227.11013827345042 |
| F9      | 226.08455653226812 |
| UA      | 224.42824657703645 |
| OH      | 215.52882835147614 |
| AA      | 211.97122176454556 |
| US      | 206.60330294168244 |
| HP      | 205.31690167066455 |
| OO      |  202.4243177198239 |
| NW      |  191.7393936377831 |
| TW      |  188.6912623180138 |
| DL      | 187.84162871590732 |
| CO      | 187.71301306878976 |
| 9E      |  181.6396991511518 |
| RU      | 181.46244295416398 |
| TZ      |  176.8928125899626 |
| AQ      | 145.65911608293766 |
| HA      |  79.38672451825789 |
+---------+--------------------+
25 rows in set (0.94 sec)
Read 71000000 rows, 1.14 GB in 0.935 sec., 75.95 million rows/sec., 1.22 GB/sec.

mysql> explain SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(cast(DepDelay>10 as Int8))*1000 AS c3 FROM ontime WHERE Year>=2000 AND Year <=2008 GROUP BY Carrier ORDER BY c3 DESC;
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                                                                                                                       |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection: IATA_CODE_Reporting_Airline as Carrier:String, (avg(cast((DepDelay > 10) as Int8)) * 1000) as c3:Float64                                                                                                                                                                                          |
|   Sort: (avg(cast((DepDelay > 10) as Int8)) * 1000):Float64                                                                                                                                                                                                                                                   |
|     Expression: IATA_CODE_Reporting_Airline:String, (avg(cast((DepDelay > 10) as Int8)) * 1000):Float64 (Before OrderBy)                                                                                                                                                                                      |
|       AggregatorFinal: groupBy=[[IATA_CODE_Reporting_Airline]], aggr=[[avg(cast((DepDelay > 10) as Int8))]]                                                                                                                                                                                                   |
|         AggregatorPartial: groupBy=[[IATA_CODE_Reporting_Airline]], aggr=[[avg(cast((DepDelay > 10) as Int8))]]                                                                                                                                                                                               |
|           Expression: IATA_CODE_Reporting_Airline:String, cast((DepDelay > 10) as Int8):Int8 (Before GroupBy)                                                                                                                                                                                                 |
|             Filter: ((Year >= 2000) and (Year <= 2008))                                                                                                                                                                                                                                                       |
|               ReadDataSource: scan schema: [Year:UInt16, IATA_CODE_Reporting_Airline:String, DepDelay:Int32], statistics: [read_rows: 71000000, read_bytes: 1179110760, partitions_scanned: 71, partitions_total: 207], push_downs: [projections: [0, 8, 31], filters: [((Year >= 2000) AND (Year <= 2008))]] |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
8 rows in set (0.01 sec)
Read 0 rows, 0 B in 0.003 sec., 0 rows/sec., 0 B/sec.

Q7 2000-2008年各航空公司平均延误时间

(0.935 sec., 75.95 million rows/sec., 1.22 GB/sec.)

mysql> SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(DepDelay) * 1000 AS c3 FROM ontime WHERE Year >= 2000 AND Year <= 2008 GROUP BY Carrier;
+---------+--------------------+
| Carrier | c3                 |
+---------+--------------------+
| B6      | 16789.739456036365 |
| NW      | 11717.623092632819 |
| F9      | 11232.889558936127 |
| XE      | 17092.548853057146 |
| YV      |  17971.53933699898 |
| US      |   11868.7097884053 |
| RU      | 12556.249210602802 |
| AS      | 14735.545887755581 |
| HA      |  6851.555976883671 |
| OH      | 12655.103820799075 |
| UA      | 14594.243159716054 |
| TZ      | 12618.760195758565 |
| EV      | 16374.703330010156 |
| HP      | 11625.682112859839 |
| DH      | 15311.949983190174 |
| DL      | 10943.456441165357 |
| 9E      | 13091.087573576122 |
| FL      | 15192.451732538268 |
| MQ      | 14125.201554023559 |
| AQ      |  7323.278123603293 |
| OO      | 11600.594852741107 |
| AA      |  13508.78515494305 |
| TW      | 10842.722114986364 |
| WN      | 10484.932610056378 |
| CO      | 12671.595978518368 |
+---------+--------------------+
25 rows in set (0.74 sec)
Read 71000000 rows, 1.14 GB in 0.727 sec., 97.6 million rows/sec., 1.56 GB/sec.

mysql> explain SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(DepDelay) * 1000 AS c3 FROM ontime WHERE Year >= 2000 AND Year <= 2008 GROUP BY Carrier;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                                                                                                                   |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection: IATA_CODE_Reporting_Airline as Carrier:String, (avg(DepDelay) * 1000) as c3:Float64                                                                                                                                                                                                           |
|   Expression: IATA_CODE_Reporting_Airline:String, (avg(DepDelay) * 1000):Float64 (Before Projection)                                                                                                                                                                                                      |
|     AggregatorFinal: groupBy=[[IATA_CODE_Reporting_Airline]], aggr=[[avg(DepDelay)]]                                                                                                                                                                                                                      |
|       AggregatorPartial: groupBy=[[IATA_CODE_Reporting_Airline]], aggr=[[avg(DepDelay)]]                                                                                                                                                                                                                  |
|         Filter: ((Year >= 2000) and (Year <= 2008))                                                                                                                                                                                                                                                       |
|           ReadDataSource: scan schema: [Year:UInt16, IATA_CODE_Reporting_Airline:String, DepDelay:Int32], statistics: [read_rows: 71000000, read_bytes: 1179110760, partitions_scanned: 71, partitions_total: 207], push_downs: [projections: [0, 8, 31], filters: [((Year >= 2000) AND (Year <= 2008))]] |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q8 每年航班延误平均时间

(0.935 sec., 75.95 million rows/sec., 1.22 GB/sec.)

mysql> SELECT Year, avg(DepDelay) FROM ontime GROUP BY Year;
+------+--------------------+
| Year | avg(DepDelay)      |
+------+--------------------+
| 1987 | 12.380385692195556 |
| 1988 |  7.345867511864449 |
| 1989 |   8.81845473300008 |
| 1990 |  7.966702606180775 |
| 1991 |  6.940411174086677 |
| 1992 |  6.687364706154975 |
| 1993 |  7.207721091071671 |
| 1994 |  7.758752042452116 |
| 1995 |  9.328649903752932 |
| 1996 |  11.14468468976826 |
| 1997 |  9.919225483813925 |
| 1998 | 10.884314711941435 |
| 1999 | 11.567390524113748 |
| 2000 | 13.456897681824556 |
| 2001 | 10.895474364001354 |
| 2002 |   9.97856700710386 |
| 2003 |  9.778465263372038 |
| 2004 | 11.936799840656898 |
| 2005 |  12.60167890747495 |
| 2006 | 14.237297887039372 |
| 2007 | 15.431738868356579 |
| 2008 | 14.654588068064287 |
| 2009 | 13.168984006133062 |
| 2010 | 13.202976628175891 |
| 2011 | 13.496191548097778 |
| 2012 | 13.155971481255131 |
| 2013 | 14.901210490900201 |
| 2014 | 15.513697266113969 |
| 2015 | 14.638336410280733 |
| 2016 | 14.643883269504837 |
| 2017 |  15.70225324299191 |
| 2018 |  16.16188254545747 |
| 2019 | 16.983263489524507 |
| 2020 | 10.624498278073712 |
| 2021 | 15.289615417399649 |
+------+--------------------+
35 rows in set (1.04 sec)
Read 201816232 rows, 1.21 GB in 1.030 sec., 195.93 million rows/sec., 1.18 GB/sec.

mysql> explain SELECT Year, avg(DepDelay) FROM ontime GROUP BY Year;
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                          |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection: Year:UInt16, avg(DepDelay):Float64                                                                                                                                                                   |
|   AggregatorFinal: groupBy=[[Year]], aggr=[[avg(DepDelay)]]                                                                                                                                                      |
|     AggregatorPartial: groupBy=[[Year]], aggr=[[avg(DepDelay)]]                                                                                                                                                  |
|       ReadDataSource: scan schema: [Year:UInt16, DepDelay:Int32], statistics: [read_rows: 201816232, read_bytes: 1210897392, partitions_scanned: 207, partitions_total: 207], push_downs: [projections: [0, 31]] |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
4 rows in set (0.01 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q9 每年有多少航班

(0.509 sec., 396.54 million rows/sec., 793.08 MB/sec.)

mysql> SELECT Year, count(*) as c1 FROM ontime GROUP BY Year;
+------+---------+
| Year | c1      |
+------+---------+
| 1987 |  440403 |
| 1988 | 5202096 |
| 1989 | 5041200 |
| 1990 | 5270893 |
| 1991 | 5076925 |
| 1992 | 5092157 |
| 1993 | 5070501 |
| 1994 | 5180048 |
| 1995 | 5327435 |
| 1996 | 5351983 |
| 1997 | 5411843 |
| 1998 | 5384721 |
| 1999 | 5527884 |
| 2000 | 5683047 |
| 2001 | 5967780 |
| 2002 | 5271359 |
| 2003 | 6488540 |
| 2004 | 7129270 |
| 2005 | 7140596 |
| 2006 | 7141922 |
| 2007 | 7455458 |
| 2008 | 7009726 |
| 2009 | 6450285 |
| 2010 | 6450117 |
| 2011 | 6085281 |
| 2012 | 6096762 |
| 2013 | 6369482 |
| 2014 | 5819811 |
| 2015 | 5819079 |
| 2016 | 5617658 |
| 2017 | 5674621 |
| 2018 | 7213446 |
| 2019 | 7422037 |
| 2020 | 4688354 |
| 2021 | 5443512 |
+------+---------+
35 rows in set (0.52 sec)
Read 201816232 rows, 403.63 MB in 0.509 sec., 396.54 million rows/sec., 793.08 MB/sec.

mysql> explain SELECT Year, count(*) as c1 FROM ontime GROUP BY Year;
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                     |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection: Year:UInt16, count() as c1:UInt64                                                                                                                                               |
|   AggregatorFinal: groupBy=[[Year]], aggr=[[count()]]                                                                                                                                       |
|     AggregatorPartial: groupBy=[[Year]], aggr=[[count()]]                                                                                                                                   |
|       ReadDataSource: scan schema: [Year:UInt16], statistics: [read_rows: 201816232, read_bytes: 403632464, partitions_scanned: 207, partitions_total: 207], push_downs: [projections: [0]] |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
4 rows in set (0.01 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q10 计算每月延迟15分钟的航班平均数

(0.891 sec., 226.44 million rows/sec., 1.59 GB/sec.)

mysql> SELECT avg(cnt) FROM (SELECT Year,Month,count(*) AS cnt FROM ontime WHERE DepDel15=1 GROUP BY Year,Month) a;
+-------------------+
| avg(cnt)          |
+-------------------+
| 81474.99019607843 |
+-------------------+
1 row in set (0.90 sec)
Read 201816232 rows, 1.41 GB in 0.891 sec., 226.44 million rows/sec., 1.59 GB/sec.

mysql> explain SELECT avg(cnt) FROM (SELECT Year,Month,count(*) AS cnt FROM ontime WHERE DepDel15=1 GROUP BY Year,Month) a;
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                                                                             |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection: avg(cnt):Float64                                                                                                                                                                                                                                        |
|   AggregatorFinal: groupBy=[[]], aggr=[[avg(cnt)]]                                                                                                                                                                                                                  |
|     AggregatorPartial: groupBy=[[]], aggr=[[avg(cnt)]]                                                                                                                                                                                                              |
|       Projection: Year:UInt16, Month:UInt8, count() as cnt:UInt64                                                                                                                                                                                                   |
|         AggregatorFinal: groupBy=[[Year, Month]], aggr=[[count()]]                                                                                                                                                                                                  |
|           AggregatorPartial: groupBy=[[Year, Month]], aggr=[[count()]]                                                                                                                                                                                              |
|             Filter: (DepDel15 = 1)                                                                                                                                                                                                                                  |
|               ReadDataSource: scan schema: [Year:UInt16, Month:UInt8, DepDel15:Int32], statistics: [read_rows: 201816232, read_bytes: 1412713624, partitions_scanned: 207, partitions_total: 207], push_downs: [projections: [0, 2, 33], filters: [(DepDel15 = 1)]] |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
8 rows in set (0.01 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q11 计算每月航班平均数

(0.561 sec., 359.58 million rows/sec., 1.08 GB/sec.)

mysql> SELECT avg(c1) FROM (SELECT Year,Month,count(*) AS c1 FROM ontime GROUP BY Year,Month) a;
+-------------------+
| avg(c1)           |
+-------------------+
| 494647.6274509804 |
+-------------------+
1 row in set (0.57 sec)
Read 201816232 rows, 605.45 MB in 0.561 sec., 359.58 million rows/sec., 1.08 GB/sec.

mysql> explain SELECT avg(c1) FROM (SELECT Year,Month,count(*) AS c1 FROM ontime GROUP BY Year,Month) a;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                           |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection: avg(c1):Float64                                                                                                                                                                                       |
|   AggregatorFinal: groupBy=[[]], aggr=[[avg(c1)]]                                                                                                                                                                 |
|     AggregatorPartial: groupBy=[[]], aggr=[[avg(c1)]]                                                                                                                                                             |
|       Projection: Year:UInt16, Month:UInt8, count() as c1:UInt64                                                                                                                                                  |
|         AggregatorFinal: groupBy=[[Year, Month]], aggr=[[count()]]                                                                                                                                                |
|           AggregatorPartial: groupBy=[[Year, Month]], aggr=[[count()]]                                                                                                                                            |
|             ReadDataSource: scan schema: [Year:UInt16, Month:UInt8], statistics: [read_rows: 201816232, read_bytes: 605448696, partitions_scanned: 207, partitions_total: 207], push_downs: [projections: [0, 2]] |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
7 rows in set (0.02 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q12 显示10个两个城市直飞线航班最多的前10个

(2.930 sec., 68.87 million rows/sec., 2.91 GB/sec.)

mysql> SELECT OriginCityName, DestCityName, count(*) AS c FROM ontime GROUP BY OriginCityName, DestCityName ORDER BY c DESC LIMIT 10;
+-------------------+-------------------+--------+
| OriginCityName    | DestCityName      | c      |
+-------------------+-------------------+--------+
| San Francisco, CA | Los Angeles, CA   | 514878 |
| Los Angeles, CA   | San Francisco, CA | 512147 |
| New York, NY      | Chicago, IL       | 456042 |
| Chicago, IL       | New York, NY      | 448756 |
| Chicago, IL       | Minneapolis, MN   | 437913 |
| Minneapolis, MN   | Chicago, IL       | 433688 |
| Los Angeles, CA   | Las Vegas, NV     | 428942 |
| Las Vegas, NV     | Los Angeles, CA   | 422825 |
| New York, NY      | Boston, MA        | 419405 |
| Boston, MA        | New York, NY      | 416324 |
+-------------------+-------------------+--------+
10 rows in set (2.94 sec)
Read 201816232 rows, 8.54 GB in 2.930 sec., 68.87 million rows/sec., 2.91 GB/sec.

mysql> explain SELECT OriginCityName, DestCityName, count(*) AS c FROM ontime GROUP BY OriginCityName, DestCityName ORDER BY c DESC LIMIT 10;
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                                              |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Limit: 10                                                                                                                                                                                                                            |
|   Projection: OriginCityName:String, DestCityName:String, count() as c:UInt64                                                                                                                                                        |
|     Sort: count():UInt64                                                                                                                                                                                                             |
|       AggregatorFinal: groupBy=[[OriginCityName, DestCityName]], aggr=[[count()]]                                                                                                                                                    |
|         AggregatorPartial: groupBy=[[OriginCityName, DestCityName]], aggr=[[count()]]                                                                                                                                                |
|           ReadDataSource: scan schema: [OriginCityName:String, DestCityName:String], statistics: [read_rows: 201816232, read_bytes: 9829664815, partitions_scanned: 207, partitions_total: 207], push_downs: [projections: [15, 24]] |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.00 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q13 显示飞机最多航班的10个城市

(1.223 sec., 165.05 million rows/sec., 3.49 GB/sec.)

mysql> SELECT OriginCityName, count(*) AS c FROM ontime GROUP BY OriginCityName ORDER BY c DESC LIMIT 10;
+-----------------------+----------+
| OriginCityName        | c        |
+-----------------------+----------+
| Chicago, IL           | 12545243 |
| Atlanta, GA           | 10900284 |
| Dallas/Fort Worth, TX |  9011081 |
| Houston, TX           |  6844476 |
| Los Angeles, CA       |  6695628 |
| New York, NY          |  6309911 |
| Denver, CO            |  6283055 |
| Phoenix, AZ           |  5658884 |
| Washington, DC        |  4998047 |
| San Francisco, CA     |  4673365 |
+-----------------------+----------+
10 rows in set (1.23 sec)
Read 201816232 rows, 4.27 GB in 1.223 sec., 165.05 million rows/sec., 3.49 GB/sec.

mysql> explain SELECT OriginCityName, count(*) AS c FROM ontime GROUP BY OriginCityName ORDER BY c DESC LIMIT 10;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                     |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Limit: 10                                                                                                                                                                                                   |
|   Projection: OriginCityName:String, count() as c:UInt64                                                                                                                                                    |
|     Sort: count():UInt64                                                                                                                                                                                    |
|       AggregatorFinal: groupBy=[[OriginCityName]], aggr=[[count()]]                                                                                                                                         |
|         AggregatorPartial: groupBy=[[OriginCityName]], aggr=[[count()]]                                                                                                                                     |
|           ReadDataSource: scan schema: [OriginCityName:String], statistics: [read_rows: 201816232, read_bytes: 4914707403, partitions_scanned: 207, partitions_total: 207], push_downs: [projections: [15]] |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.01 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

Q14 查询 ontime 表总共有多少行

(0.002 sec., 443.51 rows/sec., 443.51 B/sec.)

mysql> SELECT count(*) FROM ontime;
+-----------+
| count()   |
+-----------+
| 201816232 |
+-----------+
1 row in set (0.01 sec)
Read 1 rows, 1 B in 0.002 sec., 443.51 rows/sec., 443.51 B/sec.

mysql> explain SELECT count(*) FROM ontime;
+-----------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                 |
+-----------------------------------------------------------------------------------------------------------------------------------------+
| Projection: count():UInt64                                                                                                              |
|   Projection: 201816232 as count():UInt64                                                                                               |
|     Expression: 201816232:UInt64 (Exact Statistics)                                                                                     |
|       ReadDataSource: scan schema: [dummy:UInt8], statistics: [read_rows: 1, read_bytes: 1, partitions_scanned: 1, partitions_total: 1] |
+-----------------------------------------------------------------------------------------------------------------------------------------+
4 rows in set (0.01 sec)
Read 0 rows, 0 B in 0.002 sec., 0 rows/sec., 0 B/sec.

更多性能测试

Databend On Amazon S3 Performance

https://databend.rs/doc/performance/ec2-s3-performance

Databend On Alibaba Cloud ECS OSS Performance

https://databend.rs/doc/performance/ecs-oss-performance

Databend On Wasabi Performance

https://databend.rs/doc/performance/ec2-wasabi-performance

需要支持请添加微信: 82565387 获取更多帮助。

微信:Databend

文档:https://databend.rs

Databend:用SQL编排AI智能体的数据操作系统架构解析 在数据仓库与AI应用融合的背景下,传统架构常面临数据孤岛与计算分离的挑战。其核心原理在于通过统一的数据处理平面,将结构化分析、向量搜索与AI逻辑执行深度集成。这一架构的技术价值在于实现了数据与智能体计算的安全、高效融合,大幅简化了复杂AI工作流的开发和运维。具体到应用场景,它使得开发者能够直接使用SQL语句来编排和调用内嵌的AI智能体,例如进行实时的情感分析、智能推荐或自动化决策,从而在电商、客服、内容分析等多个领域提升数据驱动的智能化水平。本文以Databend为例,深入探讨了其通过控制平面、执行平面与计 阅读详情

相关推荐

RK3568 解决MIPI屏幕开机只显示uboot logo的问题

之前写过一篇文章介绍了RK3568驱动MIPI屏幕的方法,这次产品迭代,硬件改版,选型的新的屏幕了。所以再次开启调试屏幕的过程。本以为会和上次一样会轻车熟路,事情总是事与愿违,同样的路,遇到新的坑。

fhqlongteng的博客 5427

利用 Addax 异构迁移数据到 Databend

Addax 使用中对于具体的数据库,建议多阅读官方的 reader 或是 writer 的说明,根据实际情况决定合理的配置。mysql2databend.json 配置参考:databend-workshop/mysql2databend.json at main · wubx/databend-workshop (https://github.com/wubx/databend-workshop/blob/main/addax_mysql2databend/mysql2databend.json)

Databend的博客 793

AI Agent在智能投研中的实战:多智能体信息整合与策略生成

本文将带你从0到1完整搭建一套基于LangChain和AutoGen的多智能体AI投研系统首先,我们会一起梳理智能投研的核心痛点、金融数据的独特性,以及为什么传统AI Agent,尤其是多智能体协同架构,是破局的唯一最优解;RAG检索增强生成(但这次是金融领域定制化的,包括向量检索+时序数据检索+知识图谱检索的“三元检索增强”)、AutoGen多智能体协同框架。

软件工程实践的博客 382

Databend 集群部署 | 新手篇(2)

,其中 databend-query 是计算节点,无状态模式,这种架构也有直接用于生产的。实际上生产环境推荐至少部署三个 databend-meta 组成集群,其中 databend-query 可以是一个节点,而且可做到不使用就关闭。如果你使用的是公有云上对象存储,就不用担心对象存储的高可用。下面我们描述一下 Databend 的集群环境搭建。在 Databend 集群中分为两个角色:databend-meta 集群和 databend-query 集群。

Databend的博客 823

AI搜索优化(GEO)实战指南:抢占智能对话流量入口

AI SEO,是针对生成式AI平台(如DeepSeek、豆包、Kimi等)的优化策略,旨在通过适配AI算法对内容的理解和引用逻辑,提升品牌在AI生成答案中的可见性与权威性,从而在对话式搜索场景中抢占流量入口。:强化内容经验性(Experience)、专业性(Expertise)、权威性(Authoritativeness)、可信度(Trustworthiness)内容需避免夸大描述,提供可验证的细节(如数据、用户评价)。

Baixue5209的博客 6795

解密QtIFW高级配置:用XML+JS打造品牌化安装界面(含动态水印/多语言技巧)

本文深入探讨了如何利用Qt Installer Framework(QtIFW)的高级配置功能,通过XML和JavaScript打造专业、品牌化的软件安装界面。文章详细解析了企业级项目目录结构设计、模块化配置文件编写、CSS样式表深度定制、动态水印实现以及智能多语言许可证管理等核心技巧,旨在帮助开发者超越基础打包,为Windows和Linux平台构建具有卓越用户体验的安装程序。

read5的博客 531

Databend 存储架构总览

通过本篇文章带大家理解一下 Databend 的存储结构。Databend 内置的 Table 引擎为 Fuse table engine ,也是接下来要花重点篇幅要讲的。

Databend的博客 900

波分复用-Zblock仿真

Z-BLOCK是一种用于光通信模块的光学元件,其核心结构是在玻璃块一侧贴有4个特定波长滤光片,另一侧镀有反射膜和增透膜。光信号在内部呈现Z字形反射路径,实现波长分离。文章以可见光LED为例,在ZEMAX中演示了非序列模式下的建模过程,重点介绍了滤光片和膜层的参数设置方法。需要注意的是,入射角度需根据玻璃折射率、通道高度和厚度精确计算,否则可能导致光泄漏。该建模方法同样适用于光通信中的光纤激光器系统。

Zhaxibamao的博客 1769

十大 Feature:腾讯云数据仓库TCHouse-D 2.0内核引擎全新升级

在 2.0 内核版本中,腾讯云 TCHouse-D 在查询写入性能、产品易用性、产品安全及降本、日志场景支持等方面都进行了大幅加强。下文将对这些 2.0 内核的新 Feature 进行详细介绍。

cloudbigdata的博客 556

Python和Spyder:一个完美的编程组合

而Spyder作为一个专门为科学计算和数据分析而设计的IDE,提供了高效的开发环境和丰富的工具。无论是初学者还是专业开发者,Python和Spyder都能满足他们的需求,并为他们的工作带来便利和效率。本文将探讨Python和Spyder之间的关系,以及它们如何相互配合,提供一流的编程体验。每个工具都有其特定的优势和适用场景,开发者可以根据自己的需求选择合适的工具。而Spyder作为一个专门为科学计算而设计的IDE,提供了编写、调试和执行Python代码的环境。

CodeHeroicX的博客 1125

Roboflow 与 YOLOv11 数据集全流程:从获取到训练,让你数据无忧

上传图像:点击“Upload”按钮,批量上传本地图像(支持 JPG、PNG 格式,单张最大 10MB)。智能标注:利用 Roboflow 的“Auto-Annotate”功能,基于预训练模型自动生成标注,再手动微调,标注效率提升 70%。格式导出:标注完成后,同样选择“YOLOv11”格式导出,流程与公开数据集一致。

m0_71589036的博客 273

使用 Databend 助力 MySQL 的数据分析

Databend 是一个非常先进的基于对象存储云原生数仓1可以提强大的计算分析及存储能力。让 MySQL DBA 非常眼馋。今天想把 MySQL 的 wubx 库从 MySQL 全量迁移到 Databend 中。 借助工具: dumpling2Dumpling3 是支持以 SQL 文本或者 CSV 格式将 MySQL/TiDB 数据导出的工具。设计初衷是为了替代Mydumper4, 所以基本用法可以参考 Mydumper, 当然在实现中没有完全照搬 Mydumper, 因此存在与 Mydumper 不同的用

Databend的博客 326

汽车行业OTD业务模式入门学习

本文是在学习彭俊松老师的《汽车行业整车订单交付系统》后,对于OTD理解的一些摘录,供诸位参考学习。

qq_40664711的博客 1万+

腾讯云对象存储联合DataBend云数仓打通数据湖和数据仓库

伴随多场景的成功落地,腾讯云对象存储打造基于云技术的湖仓一体解决方案。将数据仓库构建在数据湖上,打通数据仓库和数据湖两套体系,构建以数据湖为中心,融合数据仓库、大数据、AI等技术的生态体系,既有数据湖的灵活性和可扩展性,又有数据仓库的数据管理功能。

云存储小天使的博客 7034

大中型超市收银系统(可直接运行)

光步前台收银系统系统功能:商品销售、商品退货、挂单、收银员权限、会员销售、单品取消、整单取消、商品查询、脱机销售、脱机数据上传,基本资料下载,开钱箱、小票打印、收款结算,支持多币种付款方式(现金、储值卡、礼券、微信等),支持多种促销。运行环境:.net 4.0登陆信息用户名:1001 密码:123456内置商品信息,商品编码可输入00001000 - 00010000,测试效果相关功能键F5 结算 F9挂单 F11商品查询 F8退货 Insert单品取消 Del整单取消 F12开钱箱F2 会员 Ctrl+Z退出系统 Home 锁屏 相关源码待公开

VB摄像头监控

该VB模块可对摄像头捕获的画面进行录像等,可在创建的窗口进行控制

利用云原生数仓 Databend 构建 MySQL 的归档分析服务

MySQL 常用 OLTP 业务环境,一般会使用比较好的硬件资源来提供对外服务。现在 MySQL 数据对外提供的数据动不动好几个 T 也是正常的。在很多业务中,数据有较强的生命周期,在线一段时间后,可能就是失去业务意义,如:这类工作是 DBA 提出归档方案,由开发人员提出对哪些数据可以归档,规范后可以借助于自动化的执行完成。现在常见的归档方式一般分成两大类:MySQL & MariaDB,核心工具是:pt-archive 或是解析 binlog 获取归档的数据。这类方案中,一般是通过购买 PC 机,通常是大

Databend的博客 513

锂电池充电器电路设计 里面有电路

锂电池充电器电路设计 里面有电路 3.7V升压到5V电路设计。

利用 Graviton 和 Spot 实例打造 Databend 高性能数据平台

Databend是一款基于Rust开发的云原生数据仓库解决方案,提供Databend Cloud和私有化部署企业版两条产品线。其创新架构采用元数据节点+无状态计算节点的设计,支持PB级数据处理,实现存算分离和弹性伸缩。通过S3对象存储和Spot实例等技术,显著降低了存储和计算成本(如数据压缩率达30-100倍)。典型应用场景包括实时数据分析、AI训练数据共享、游戏实时统计等,已在国内金融、运营商及海外市场成功落地,帮助客户实现成本降低50%以上、分析效率提升数倍的显著效果。Databend以其开源、高性能和

qq_45804378的博客 754

蓝牙模块HM-10手册

蓝牙模块HM-10手册,该手册对蓝牙模块HM-10进行了详细的介绍和使用方法以及一些AT指令。

Databend Cloud 如何助力 AIGC 初创公司成本下降百倍

Databend Cloud 是基于开源云原生数仓项目 Databend 打造的一款易用、低成本、高性能的新一代大数据分析平台,提供一站式 SaaS 服务,免运维、开箱即用。为了更好了解用户习惯,进一步改善用户体验、优化网站和应用程序,平台需要通过埋点的形式采集用户行为日志并进行分析,最终绘制图表形成看板。在平台上,要想生成满意的动漫角色,涉及模型选择、Prompt 编写、作品调优等多个交互环节,对于特定的需求,还可能涉及图片上传及编辑等功能。

Databend的博客 191

Databend 企业级 AI Agent 数据仓库:Sandbox UDF、向量检索与统一分析架构深度指南

本文以 Databend 仓库根目录 [README.md](https://link.gitcode.com/i/c0e6fe32f4be49a06fe5f9ab4bc9442d) 为核心,系统讲解这款基于 Rust 构建的开源企业级数据仓库如何将大规模 SQL 分析、向量检索、全文检索与自动 Schema 演进统一到单一引擎中,并重点剖析其面向 AI Agent 的「Sandbox UDF +

gitblog_00107的博客 418

Databend Cloud 正式上线腾讯云市场,加速挖掘数据创新价值

近日,新一代大数据分析平台 Databend Cloud 正式上线腾讯云千帆云市场,成为腾讯云 SaaS 生态圈中的重要一员。依托腾讯云平台的技术支持和市场资源, Databend Cloud 将充分发挥数据价值的挖掘能力,助力更多企业实现数据驱动的决策运营。Databend Cloud 是一款基于开源云原生数仓项目 Databend 打造的易用、低成本、高性能的新一代大数据分析平台,提供一站式 SaaS 服务,具有免运维、开箱即用等特点。

Databend的博客 1349

Databend 产品月报(2024年9月)

很高兴为您带来 Databend 2024 年 9 月的最新更新、新功能和改进!我们希望这些增强功能对您有所帮助,并期待您的反馈。

Databend的博客 790

释放无限潜力:Databend 存算分离架构如何让企业数据管理焕然一新?

存算分离架构的出现为云原生数据库带来了新的可能性。通过计算与存储资源的解耦,Databend 实现了高扩展性、弹性使用和更高的资源利用率,解决了传统存算一体架构中存在的扩展性和成本问题。作为一款云原生数据仓库,Databend 的设计思路从一开始就围绕存算分离架构展开,最终为用户提供了一种高效灵活的数据处理解决方案。Databend 存算分离架构的设计与实现不仅是当前技术发展的体现,更为未来数据库系统的发展指明了方向。

Databend的博客 1300

Databend:下一代云数据仓库解决方案

Databend是一个用Rust语言开发的开源云数据仓库,旨在为大规模数据分析提供高性能、低成本的解决方案。作为Snowflake的替代品,Databend专注于快速查询执行和数据摄取,适用于复杂的大规模数据集分析。云原生架构: 与AWS S3、Azure Blob、Google Cloud等云存储服务无缝集成高性能: 基于Rust构建,采用先进的向量化执行技术成本效益: 设计用于可扩展的存储和计算,在提高性能的同时降低成本AI驱动分析: 通过AI函数支持高级分析数据简化。

2401_87458778的博客 1770

Databend 2025:海量数据 × AI 一体化底座,v1.3 即将发布

Databend 正式从单一数仓进化为 Analytics + Search + AI 一体化数据库,v1.3 版本即将发布

Databend的博客 402

Databend Meta-Service 架构概述

Databend 的 meta-service 采用基于 Raft 的分布式架构,实现了高可用性和数据一致性。其核心组件包括分布式日志和状态机,通过精心设计的写入和读取流程,确保了数据操作的可靠性。使用 Raft 日志中的时间戳来提供一致的时钟,解决了分布式系统中的时间同步问题。支持灵活的节点管理和数据更新操作,包括单条数据更新(upsert)和多条数据更新(transaction)。采用全局序列号机制,为实现可靠的比较并交换(CAS)操作提供了基础。

Databend的博客 1646

Databend 设计概述 | 白皮书

Databend 是一个开源的、完全面向云架构的新式数仓,它提供快速的弹性扩展能力,并结合云的弹性、简单性和低成本,使 Data Cloud 构建变得更加容易。 Databend 把数据存储在像 AWS S3 ,Azure Blob 这些云上的存储系统,可以使不同的计算节点挂载同一份数据,从而做到较高的弹性,实现对资源的精细化控制。 Databend 在设计上专注以下能力: 弹性 在 Databend 中,存储和计算资源可以按需、按量弹性扩展。 安全 Databend 中数据文件和网络传输都是端到端加密,

Databend的博客 1189

Databend 借助对象存储帮你实现降本增效

对象存储是一种可以非结构化存储和管理数据的技术。可以简单理解为 NoSQL 接口方式存储和访问数据,对象存储最有名的产品是: AWS S3。这样使对象存储原生具备了: 持久性,可扩展性,数据可用性,对于对象存储比较核心的也是元数据管理。数据归档备份和恢复数据湖ML富媒体但对象存储也有一些限制,这个限制各个厂家没有统一,使用上需要注意各厂家的指标。QPS 限制: 有的厂家限制到 Bucket,有的是限制到用户, 甚至有单文件下载热点控制。带宽,现在一般都能达到 10Gbit/s 以上。

Databend的博客 586
上一篇: 「Datafuse Labs」今天一岁啦
下一篇: 如何阅读一份源代码?
Databend
博客等级 码龄5年 1458粉丝 363原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值