分布式ID

我叫果冻

已于 2023-08-22 22:57:12 修改

阅读量684

点赞数

分类专栏：微服务解决方案文章标签：分布式 java

于 2023-08-20 00:04:45 首次发布

本文链接：https://blog.youkuaiyun.com/GZ946/article/details/132385933

版权

微服务解决方案专栏收录该内容

10 篇文章

订阅专栏

分布式ID

背景
Snowflake（雪花算法）
uid-generator实战
美团leaf实战
代码地址

背景

分布式系统，用什么做为主键呢？

uuid
太长（MySQL官方有明确的建议主键要尽量越短越好[4]，36个字符长度的UUID不符合要求。）、
无规律（在InnoDB引擎下，UUID的无序性可能会引起数据位置频繁变动，严重影响性能。）
Snowflake
Leaf
https://tech.meituan.com/2017/04/21/mt-leaf.html
UidGenerator
https://github.com/baidu/uid-generator/blob/master/README.zh_cn.md

Snowflake（雪花算法）

简介

第一位占用1bit，其值始终是0，没有实际作用。
2.时间戳占用41bit，精确到毫秒，总共可以容纳约69年的时间。
3.工作机器id 占用10bit，其中高位5bit是数据中心ID，低位5bit是工作节点ID，做多可以容纳1024个节点。
4.序列号占用12bit，每个节点每毫秒0开始不断累加，最多可以累加到4095，一共可以产生4096个ID。
SnowFlake算法在同一毫秒内最多可以生成多少个全局唯一ID呢：：同一毫秒的ID数量 = 1024 X 4096 = 4194304

代码实现

public class SnowflakeIdWorker {
    /**
     * 开始时间截 (2015-01-01)
     */
    private final long twepoch = 1420041600000L;
    /**
     * 机器id所占的位数
     */
    private final long workerIdBits = 5L;
    /**
     * 数据标识id所占的位数
     */
    private final long datacenterIdBits = 5L;
    /**
     * 支持的最大机器id，结果是31 (这个移位算法可以很快的计算出几位二进制数所能表示的最大十进制数)
     */
    private final long maxWorkerId = -1L ^ (-1L << workerIdBits);
    /**
     * 支持的最大数据标识id，结果是31
     */
    private final long maxDatacenterId = -1L ^ (-1L << datacenterIdBits);
    /**
     * 序列在id中占的位数
     */
    private final long sequenceBits = 12L;
    /**
     * 机器ID向左移12位
     */
    private final long workerIdShift = sequenceBits;
    /**
     * 数据标识id向左移17位(12+5)
     */
    private final long datacenterIdShift = sequenceBits + workerIdBits;
    /**
     * 时间截向左移22位(5+5+12)
     */
    private final long timestampLeftShift = sequenceBits + workerIdBits + datacenterIdBits;
    /**
     * 生成序列的掩码，这里为4095 (0b111111111111=0xfff=4095)
     */
    private final long sequenceMask = -1L ^ (-1L << sequenceBits);
    /**
     * 工作机器ID(0~31)
     */
    private long workerId;
    /**
     * 数据中心ID(0~31)
     */
    private long datacenterId;
    /**
     * 毫秒内序列(0~4095)
     */
    private long sequence = 0L;
    /**
     * 上次生成ID的时间截
     */
    private long lastTimestamp = -1L;
    /**
     * 构造函数
     * @param workerId     工作ID (0~31)
     * @param datacenterId 数据中心ID (0~31)
     */
    public SnowflakeIdWorker(long workerId, long datacenterId) {
        if (workerId > maxWorkerId || workerId < 0) {
            throw new IllegalArgumentException(String.format("worker Id can't be greater than %d or less than 0", maxWorkerId));
        }
        if (datacenterId > maxDatacenterId || datacenterId < 0) {
            throw new IllegalArgumentException(String.format("datacenter Id can't be greater than %d or less than 0", maxDatacenterId));
        }
        this.workerId = workerId;
        this.datacenterId = datacenterId;
    }
    /**
     * 获得下一个ID (该方法是线程安全的)
     * @return SnowflakeId
     */
    public synchronized long nextId() {
        long timestamp = timeGen();
        // 如果当前时间小于上一次ID生成的时间戳，说明系统时钟回退过这个时候应当抛出异常
        if (timestamp < lastTimestamp) {
            throw new RuntimeException(
                    String.format("Clock moved backwards.  Refusing to generate id for %d milliseconds", lastTimestamp - timestamp));
        }
        // 如果是同一时间生成的，则进行毫秒内序列
        if (lastTimestamp == timestamp) {
            sequence = (sequence + 1) & sequenceMask;
            // 毫秒内序列溢出
            if (sequence == 0) {
                //阻塞到下一个毫秒,获得新的时间戳
                timestamp = tilNextMillis(lastTimestamp);
            }
        }
        // 时间戳改变，毫秒内序列重置
        else {
            sequence = 0L;
        }
        // 上次生成ID的时间截
        lastTimestamp = timestamp;
        // 移位并通过或运算拼到一起组成64位的ID
        return ((timestamp - twepoch) << timestampLeftShift) //
                | (datacenterId << datacenterIdShift) //
                | (workerId << workerIdShift) //
                | sequence;
    }
    /**
     * 阻塞到下一个毫秒，直到获得新的时间戳
     * @param lastTimestamp 上次生成ID的时间截
     * @return 当前时间戳
     */
    protected long tilNextMillis(long lastTimestamp) {
        long timestamp = timeGen();
        while (timestamp <= lastTimestamp) {
            timestamp = timeGen();
        }
        return timestamp;
    }
    /**
     * 返回以毫秒为单位的当前时间
     * @return 当前时间(毫秒)
     */
    protected long timeGen() {
        return System.currentTimeMillis();
    }

    public static void main(String[] args) throws InterruptedException {
        SnowflakeIdWorker idWorker = new SnowflakeIdWorker(0, 0);
        for (int i = 0; i < 10; i++) {
            long id = idWorker.nextId();
            Thread.sleep(1);
            System.out.println(id);
        }
    }
}

缺点

1.会有时钟问题
2.前端long类型只接受19位以内

这个对雪花算法进行改造，其实我们一般系统，要不那么高的并发，可以减少位数，这样就能达到前端19位以内的条件

uid-generator实战

简介
UidGenerator是百度开源的一款分布式高性能的唯一ID生成器，是基于snowflake模型的一种ID生成器
官方有中文文档说明，非常详细，我这里就不详细介绍

1.解决了时钟回调的问题
2.使用ringbuffer,无锁进行id的生产与消费,速度非常快
3.适用于多线程,不会有单线程瓶颈

整合springboot

整合springboot进行了详细说明，因为官方没有给出具体详细说明

1.pom.xml

 <!--for Mysql-->
        <dependency>
            <groupId>org.mybatis.spring.boot</groupId>
            <artifactId>mybatis-spring-boot-starter</artifactId>
            <version>${mybatis.version}</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>druid-spring-boot-starter</artifactId>
            <version>${druid-version}</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
        </dependency>
        <!--必须放在最后,这个包需要要从github下载源码包进行本地打包引入，否则找不到该依赖包-->
        <dependency>
            <groupId>com.baidu.fsg</groupId>
            <artifactId>uid-generator</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>

2.application.yml

spring:
  datasource:
    driver-class-name: com.mysql.cj.jdbc.Driver
    url: jdbc:mysql://192.168.10.10:3306/test?useUnicode=true&characterEncoding=utf-8&userSSL=false&serverTimezone=GMT%2B8
    username: root
    password: 123456
    type: com.alibaba.druid.pool.DruidDataSource
    druid:
      initial-size: 5
      min-idle: 5
      max-active: 20
      max-wait: 60000
      time-between-eviction-runs-millis: 60000
      min-evictable-idle-time-millis: 300000
mybatis:
  mapper-locations: classpath:com/gz/distributed/id/mapper/*.xml
  configuration:
    log-impl: org.apache.ibatis.logging.stdout.StdOutImpl
server:
  port: 8085

3.IdGeneratorConfiguration

package com.gz.distributed.id.config;

import com.baidu.fsg.uid.impl.CachedUidGenerator;
import com.gz.distributed.id.assigner.CustomDisposableWorkerIdAssigner;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class IdGeneratorConfiguration {


    /*
        该类是在默认的基础上修改的
    */
    @Bean
    public CustomDisposableWorkerIdAssigner disposableWorkerIdAssigner() {
        return new CustomDisposableWorkerIdAssigner();
    }
    //默认注入的id生成器,使用时只需从容器取即可
    @Bean
    public CachedUidGenerator cachedUidGenerator() {
        CachedUidGenerator cachedUidGenerator = new CachedUidGenerator();
        cachedUidGenerator.setWorkerIdAssigner(disposableWorkerIdAssigner());
        return cachedUidGenerator;
    }


}

4.CustomDisposableWorkerIdAssigner

package com.gz.distributed.id.assigner;

import com.baidu.fsg.uid.utils.DockerUtils;
import com.baidu.fsg.uid.utils.NetUtils;
import com.baidu.fsg.uid.worker.WorkerIdAssigner;
import com.baidu.fsg.uid.worker.WorkerNodeType;
import com.baidu.fsg.uid.worker.entity.WorkerNodeEntity;
import com.gz.distributed.id.mapper.WorkerNodeMapper;
import org.apache.commons.lang.math.RandomUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.transaction.annotation.Transactional;
public class CustomDisposableWorkerIdAssigner implements WorkerIdAssigner {
    private static final Logger LOGGER = LoggerFactory.getLogger(CustomDisposableWorkerIdAssigner.class);

    @Autowired
    private  WorkerNodeMapper workerNodeMapper;

    /**
     * Assign worker id base on database.<p>
     * If there is host name & port in the environment, we considered that the node runs in Docker container<br>
     * Otherwise, the node runs on an actual machine.
     *
     * @return assigned worker id
     */
    @Override
    @Transactional
    public long assignWorkerId() {
        // build worker node entity
        WorkerNodeEntity workerNodeEntity = buildWorkerNode();

        // add worker node for new (ignore the same IP + PORT)
        workerNodeMapper.addWorkerNode(workerNodeEntity);
        LOGGER.info("Add worker node:" + workerNodeEntity);

        return workerNodeEntity.getId();
    }

    /**
     * Build worker node entity by IP and PORT
     */
    private WorkerNodeEntity buildWorkerNode() {
        WorkerNodeEntity workerNodeEntity = new WorkerNodeEntity();
        if (DockerUtils.isDocker()) {
            workerNodeEntity.setType(WorkerNodeType.CONTAINER.value());
            workerNodeEntity.setHostName(DockerUtils.getDockerHost());
            workerNodeEntity.setPort(DockerUtils.getDockerPort());

        } else {
            workerNodeEntity.setType(WorkerNodeType.ACTUAL.value());
            workerNodeEntity.setHostName(NetUtils.getLocalAddress());
            workerNodeEntity.setPort(System.currentTimeMillis() + "-" + RandomUtils.nextInt(100000));
        }

        return workerNodeEntity;
    }


}

5.mapper

package com.gz.distributed.id.mapper;

import com.baidu.fsg.uid.worker.entity.WorkerNodeEntity;
import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
@Mapper
public interface WorkerNodeMapper {
    /**
     * Get {@link WorkerNodeEntity} by node host
     *
     * @param host
     * @param port
     * @return
     */
    WorkerNodeEntity getWorkerNodeByHostPort(@Param("host") String host, @Param("port") String port);

    /**
     * Add {@link WorkerNodeEntity}
     *
     * @param workerNodeEntity
     */
    void addWorkerNode(WorkerNodeEntity workerNodeEntity);

}

6.controller

package com.gz.distributed.id.controller;

import com.baidu.fsg.uid.UidGenerator;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import javax.annotation.Resource;

@RequiredArgsConstructor
@RestController
public class IdController {

    @Resource
    private UidGenerator uidGenerator;
    @GetMapping("/get-uuid")
    public String getUid() {
        return String.valueOf(uidGenerator.getUID());
    }

}

7.mapper.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.gz.distributed.id.mapper.WorkerNodeMapper">
    <resultMap id="workerNodeRes"
               type="com.baidu.fsg.uid.worker.entity.WorkerNodeEntity">
        <id column="ID" jdbcType="BIGINT" property="id"/>
        <result column="HOST_NAME" jdbcType="VARCHAR" property="hostName"/>
        <result column="PORT" jdbcType="VARCHAR" property="port"/>
        <result column="TYPE" jdbcType="INTEGER" property="type"/>
        <result column="LAUNCH_DATE" jdbcType="DATE" property="launchDate"/>
        <result column="MODIFIED" jdbcType="TIMESTAMP" property="modified"/>
        <result column="CREATED" jdbcType="TIMESTAMP" property="created"/>
    </resultMap>

    <insert id="addWorkerNode" useGeneratedKeys="true" keyProperty="id"
            parameterType="com.baidu.fsg.uid.worker.entity.WorkerNodeEntity">
        INSERT INTO WORKER_NODE
        (HOST_NAME,
         PORT,
         TYPE,
         LAUNCH_DATE,
         MODIFIED,
         CREATED)
        VALUES (
                   #{hostName},
                   #{port},
                   #{type},
                   #{launchDate},
                   NOW(),
                   NOW())
    </insert>

    <select id="getWorkerNodeByHostPort" resultMap="workerNodeRes">
        SELECT
            ID,
            HOST_NAME,
            PORT,
            TYPE,
            LAUNCH_DATE,
            MODIFIED,
            CREATED
        FROM
            WORKER_NODE
        WHERE
            HOST_NAME = #{host} AND PORT = #{port}
    </select>
</mapper>

8.sql

DROP TABLE IF EXISTS WORKER_NODE;
CREATE TABLE WORKER_NODE
(
ID BIGINT NOT NULL AUTO_INCREMENT COMMENT 'auto increment id',
HOST_NAME VARCHAR(64) NOT NULL COMMENT 'host name',
PORT VARCHAR(64) NOT NULL COMMENT 'port',
TYPE INT NOT NULL COMMENT 'node type: ACTUAL or CONTAINER',
LAUNCH_DATE DATE NOT NULL COMMENT 'launch date',
MODIFIED TIMESTAMP NOT NULL COMMENT 'modified time',
CREATED TIMESTAMP NOT NULL COMMENT 'created time',
PRIMARY KEY(ID)
)
 CO

在这里插入图片描述

测试

美团leaf实战

世界上没有两片完全相同的树叶。
— 莱布尼茨

简介
Leaf 最早期需求是各个业务线的订单ID生成需求。在美团早期，有的业务直接通过DB自增的方式生成ID，有的业务通过redis缓存来生成ID，也有的业务直接用UUID这种方式来生成ID。以上的方式各自有各自的问题，因此我们决定实现一套分布式ID生成服务来满足需求
目前Leaf覆盖了美团点评公司内部金融、餐饮、外卖、酒店旅游、猫眼电影等众多业务线。在4C8G VM基础上，通过公司RPC方式调用，QPS压测结果近5w/s，TP999 1ms。

解决了时钟问题

实战与springboot整合
1.pom.xml

<dependency>
            <groupId>com.sankuai.inf.leaf</groupId>
            <artifactId>leaf-boot-starter</artifactId>
            <version>1.0.1-RELEASE</version>
        </dependency>

        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-recipes</artifactId>
            <version>2.6.0</version>
            <exclusions>
                <exclusion>
                    <artifactId>log4j</artifactId>
                    <groupId>log4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>

2.leaf.properties

leaf.name=com.gz.distributed.id
leaf.segment.enable=false
leaf.snowflake.enable=true
leaf.snowflake.address=192.168.10.10:2181
leaf.snowflake.port=9000

3.controller

package com.gz.distributed.id.controller;

import com.baidu.fsg.uid.UidGenerator;
import com.sankuai.inf.leaf.service.SnowflakeService;
import lombok.RequiredArgsConstructor;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import javax.annotation.Resource;

@RestController
public class IdController {

 
    @Resource
    private SnowflakeService snowflakeService;


    @GetMapping("/get-leaf")
    public String getLeaf() {
        return String.valueOf(snowflakeService.getId("id"));
    }

}