starrocks采用streamload导入数据

最新推荐文章于 2024-06-05 16:55:24 发布

Sylarjyd

最新推荐文章于 2024-06-05 16:55:24 发布

阅读量875

点赞数

CC 4.0 BY-SA版权

文章标签： java

本文链接：https://blog.youkuaiyun.com/Sylarjyd/article/details/132021000

本文档介绍了如何通过Java配置和代码实现对StarRocks的Streamload数据导入，帮助读者理解并掌握这一过程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1、依赖
<dependency>
    <groupId>com.starrocks</groupId>
    <artifactId>flink-connector-starrocks</artifactId>
    <version>1.2.1_flink-1.14_2.12</version>
</dependency>

2、配置

@Configuration
@ConfigurationProperties(prefix = "star")
@Data
@Slf4j
public class StarRocksConfiguration {
    private String url;
    private String port;
    private String format;
    private String user;
    private String pass;
    private String database;
    private String labelPrefix = "streamLoad";
    private String rowSep = "&";
    private Integer retryLimit;
    private Integer interval;
}

3、代码

@Override
public Map<String, Object> write(String tableName, List<String> records) throws Exception {
    StarRocksSinkBuffer bufferEntity = new StarRocksSinkBuffer(starRocksConfiguration.getDatabase(), tableName, starRocksConfiguration.getLabelPrefix());
    for (String record : records) {
        byte[] bts = record.getBytes(StandardCharsets.UTF_8);
        bufferEntity.addToBuffer(bts);
    }
    byte[] data = joinRows(bufferEntity.getBuffer(), (int) bufferEntity.getBatchSize());
    String loadUrl = new StringBuilder(starRocksConfiguration.getUrl())
            .append(":")
            .append(starRocksConfiguration.getPort())
            .