System.out.println performance

本文探讨了使用System.out.println进行日志记录时可能遇到的性能问题,特别是当多线程环境中频繁调用此方法时可能导致系统性能下降。文章通过分析PrintStream.java中的实现细节,解释了同步操作如何成为性能瓶颈,并提供了性能测试图表来展示不同条件下性能退化的程度。

Any system that logs vast amounts of information, needs to think about performance. The activity of logging cannot be a synchronous blocking call that returns only when the message has been logged to a persistence store. Enterprise logging systems usually make use of a message bus to carry messages asynchronously to their target persistence store. Be it a database or a file.

Talking about logging brings us to System.out.println() (Lets call is SOP for short). It is a surprisingly commonly method to “log” messages. SOP is not meant to be used as a logging system, but unfortunately there is no dearth of projects that have these statements scattered around the code base. The adverse effects that this statement can bring on the performance of the system is often not recognized as well as it should be.

Why is SOP a bottleneck for performance ? This is why…

Code excerpt from PrintStream.java:

 private void write(String s)
    {
        try
        {
            synchronized (this)
            {
                ensureOpen();
                textOut.write(s);
                textOut.flushBuffer();
                charOut.flushBuffer();
                if (autoFlush && (s.indexOf('\n') >= 0))
                    out.flush();
            }
        }
        catch (InterruptedIOException x)
        {
            Thread.currentThread().interrupt();
        }
        catch (IOException x)
        {
            trouble = true;
        }
    }


All SOP calls on a String result in a corresponding call to the private write(String s) method in PrintStream.java. The synchronized block ensures that every thread has to wait for this call to end before it can proceed. Whats more, the calls to flushBuffer() on the BufferedWriter that is textOut result in the execution of more synchronized blocks. These blocks are expensive to execute.

Here is a chart that shows how performance degrades when a program logs 100,000 messages through various threads. The rate of degradation also depends on the number of characters that are passing through the stream.

System.out.println performance:
System.out.println performance

The degradation experienced in ‘Web application X / Y’ may vary, but it cannot be discounted. Avoid using SOPs to log messages in your app. Even one or two that are left over can harm performance under the right conditions. Let us also not forget about e.printStackTrace() and other forms of writing to console output that follow the same synchronized pattern.

package com.dnf.main; import java.awt.image.BufferedImage; import java.awt.image.DataBufferByte; import java.nio.ByteBuffer; import java.util.concurrent.ArrayBlockingQueue; import java.util.concurrent.BlockingQueue; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; import javax.swing.SwingUtilities; import org.opencv.core.Mat; import org.opencv.imgproc.Imgproc; import org.opencv.videoio.VideoCapture; import org.opencv.videoio.Videoio; import org.opencv.videoio.VideoWriter; import javafx.application.Platform; import javafx.embed.swing.JFXPanel; import javafx.scene.Scene; import javafx.scene.image.ImageView; import javafx.scene.image.PixelFormat; import javafx.scene.image.WritableImage; import javafx.scene.layout.Background; import javafx.scene.layout.BackgroundFill; import javafx.scene.layout.CornerRadii; import javafx.scene.layout.StackPane; import javafx.scene.paint.Color; public class MS2130FXCapture { // 核心资源 private static ImageView imageView; private static VideoCapture videoCapture; // 状态控制 private static final AtomicBoolean isRunning = new AtomicBoolean(false); private static final AtomicBoolean isFXInitialized = new AtomicBoolean(false); // 线程管理 private static Thread captureThread; private static ExecutorService frameProcessingPool; // 资源池控制 private static final int MAT_POOL_SIZE = 4; private static final BlockingQueue<Mat> matPool = new ArrayBlockingQueue<>(MAT_POOL_SIZE); // 显示资源 private static final AtomicReference<BufferedImage> bufImgRef = new AtomicReference<>(); private static final AtomicReference<WritableImage> fxImgRef = new AtomicReference<>(); private static volatile Mat displayMat; // 帧率控制 private static final int TARGET_FPS = 20; private static final long FRAME_INTERVAL_MS = 1000 / TARGET_FPS; private static long lastFrameTime = System.currentTimeMillis(); // 设备参数 - 固定为1920x1080 private static final int DEVICE_WIDTH = 1920; private static final int DEVICE_HEIGHT = 1080; // 性能监控 private static final AtomicInteger frameCounter = new AtomicInteger(0); private static final long PERFORMANCE_CHECK_INTERVAL = 5000; // 5秒检查一次性能 private static long lastPerformanceCheck = System.currentTimeMillis(); // CPU使用率监控 private static long lastCpuTime = System.nanoTime(); private static long lastCpuCheckTime = System.currentTimeMillis(); private static final long CPU_CHECK_INTERVAL = 3000; // 3秒检查一次CPU private static final double MAX_CPU_USAGE = 2.0; // 最大允许CPU使用率 // 内存监控 private static final long MAX_MEMORY_MB = 300; // 最大允许内存300MB private static final long MEMORY_CHECK_INTERVAL = 3000; // 3秒检查一次内存 /** * 启动视频采集 */ public static void startCapture(JFXPanel fxPanel) { System.out.println("[自动采集] 开始初始化采集"); // 确保停止之前的采集 stopCapture(); // 重置状态 isRunning.set(true); System.out.println("[状态] isRunning 设置为 true"); int deviceIndex = findValidDeviceIndex(); if (deviceIndex == -1) { System.err.println("[错误] 未找到MS2130设备"); return; } // 初始化Mat池并预热 initMatPool(); // 初始化JavaFX环境 if (!isFXInitialized.get()) { System.out.println("[初始化] 首次初始化JavaFX"); SwingUtilities.invokeLater(() -> { Platform.runLater(() -> { System.out.println("[JavaFX] 在JavaFX线程中初始化"); initFXCapture(fxPanel, deviceIndex); isFXInitialized.set(true); }); }); } else { System.out.println("[初始化] JavaFX已初始化,直接启动采集"); Platform.runLater(() -> initFXCapture(fxPanel, deviceIndex)); } // 初始化线程池 - 使用单线程处理帧 frameProcessingPool = Executors.newSingleThreadExecutor(); System.out.println("[线程池] 帧处理线程池已创建"); } /** * 初始化Mat对象池 */ private static void initMatPool() { synchronized (matPool) { matPool.clear(); for (int i = 0; i < MAT_POOL_SIZE; i++) { matPool.offer(new Mat()); } System.out.println("[资源池] Mat对象池已初始化,大小: " + MAT_POOL_SIZE); } } /** * 初始化JavaFX UI组件 */ private static void initFXCapture(JFXPanel fxPanel, int deviceIndex) { System.out.println("[UI] 初始化JavaFX UI组件"); imageView = new ImageView(); imageView.setPreserveRatio(true); // 保持宽高比 imageView.setSmooth(false); // 平滑缩放 imageView.setCache(false); // 禁用缓存避免内存问题 // 设置默认显示尺寸为800x450 (16:9) imageView.setFitWidth(640); imageView.setFitHeight(360); StackPane root = new StackPane(imageView); root.setBackground( new Background(new BackgroundFill(Color.BLACK, CornerRadii.EMPTY, javafx.geometry.Insets.EMPTY))); fxPanel.setScene(new Scene(root)); System.out.println("[UI] JavaFX场景已设置"); openDeviceAndStartCapture(deviceIndex); } /** * 枚举有效设备 */ private static int findValidDeviceIndex() { System.out.println("[设备检测] 开始检测可用设备..."); for (int i = 0; i < 4; i++) { VideoCapture testCapture = null; try { System.out.println("[设备检测] 尝试索引: " + i); // 尝试使用DSHOW驱动 testCapture = new VideoCapture(i, Videoio.CAP_DSHOW); if (!testCapture.isOpened()) { System.out.println("[设备检测] DSHOW驱动打开失败,尝试MSMF"); testCapture.release(); testCapture = new VideoCapture(i, Videoio.CAP_MSMF); } if (!testCapture.isOpened()) { System.out.println("[设备检测] 设备 " + i + " 未打开"); continue; } Mat testFrame = new Mat(); if (testCapture.read(testFrame) && !testFrame.empty()) { testFrame.release(); testCapture.release(); return i; } else { System.out.println("[设备检测] 设备 " + i + " 读取帧失败"); } testFrame.release(); } catch (Exception e) { System.err.println("设备检测错误: " + e.getMessage()); } finally { if (testCapture != null && testCapture.isOpened()) { testCapture.release(); } } } System.out.println("[设备检测] 未找到可用设备"); return -1; } /** * 打开设备并配置 */ private static void openDeviceAndStartCapture(int deviceIndex) { System.out.println("[设备] 尝试打开设备: " + deviceIndex); // 尝试多种驱动 videoCapture = new VideoCapture(deviceIndex, Videoio.CAP_DSHOW); if (!videoCapture.isOpened()) { System.out.println("[设备] DSHOW驱动打开失败,尝试MSMF"); videoCapture.open(deviceIndex, Videoio.CAP_MSMF); } if (!videoCapture.isOpened()) { System.err.println("[错误] 无法打开设备"); return; } // 设置设备分辨率为1920x1080 videoCapture.set(Videoio.CAP_PROP_FRAME_WIDTH, DEVICE_WIDTH); videoCapture.set(Videoio.CAP_PROP_FRAME_HEIGHT, DEVICE_HEIGHT); videoCapture.set(Videoio.CAP_PROP_FOURCC, VideoWriter.fourcc('M', 'J', 'P', 'G')); videoCapture.set(Videoio.CAP_PROP_BUFFERSIZE, 2); videoCapture.set(Videoio.CAP_PROP_FPS, TARGET_FPS); // 检查实际设置的分辨率 double actualWidth = videoCapture.get(Videoio.CAP_PROP_FRAME_WIDTH); double actualHeight = videoCapture.get(Videoio.CAP_PROP_FRAME_HEIGHT); System.out.printf("[设备] 请求分辨率1920x1080,实际分辨率: %.0fx%.0f%n", actualWidth, actualHeight); if (actualWidth != DEVICE_WIDTH || actualHeight != DEVICE_HEIGHT) { System.err.println("[警告] 设备分辨率未设置为1920x1080,实际处理将以" + actualWidth + "x" + actualHeight + "进行"); } // 创建显示用的Mat displayMat = new Mat(DEVICE_HEIGHT, DEVICE_WIDTH, org.opencv.core.CvType.CV_8UC3); System.out.println("[资源] 显示Mat已创建"); // 启动采集线程 captureThread = new Thread(() -> captureLoop()); captureThread.setName("VideoCapture-Thread"); captureThread.setDaemon(true); captureThread.setPriority(Thread.MAX_PRIORITY - 1); // 适度优先级 captureThread.start(); System.out.println("[线程] 采集线程已启动"); } /** * 优化的采集循环 */ private static void captureLoop() { System.out.println("[采集线程] 开始运行"); System.out.println("[状态] isRunning = " + isRunning.get()); int consecutiveFailures = 0; long lastFpsTime = System.currentTimeMillis(); // 上次输出FPS的时间 int framesSinceLastReport = 0; // 上次报告后的帧数 while (isRunning.get()) { try { long currentTime = System.currentTimeMillis(); // 性能监控 - 每5秒检查一次 if (currentTime - lastPerformanceCheck > PERFORMANCE_CHECK_INTERVAL) { checkPerformance(); lastPerformanceCheck = currentTime; } // 帧率控制 long elapsed = currentTime - lastFrameTime; if (elapsed < FRAME_INTERVAL_MS) { Thread.sleep(1); continue; } // 获取空闲Mat Mat frameMat = matPool.poll(); if (frameMat == null) { System.out.println("[资源] Mat池为空,创建新Mat"); frameMat = new Mat(); } // 读取帧 boolean readSuccess = videoCapture.read(frameMat); if (!readSuccess || frameMat.empty()) { System.out.println("[警告] 读取帧失败: " + (frameMat.empty() ? "空帧" : "读取失败")); consecutiveFailures++; if (consecutiveFailures > 10) { System.err.println("[错误] 连续10次读取帧失败,停止采集"); break; } // 归还Mat并等待 if (frameMat != null && frameMat.cols() > 0) { matPool.offer(frameMat); } Thread.sleep(10); continue; } consecutiveFailures = 0; // 重置失败计数 // 更新计时器 lastFrameTime = currentTime; framesSinceLastReport++; // 提交处理任务 final Mat finalFrameMat = frameMat; frameProcessingPool.execute(() -> processFrame(finalFrameMat)); // 每秒输出一次FPS if (currentTime - lastFpsTime >= 1000) { double fps = framesSinceLastReport / ((currentTime - lastFpsTime) / 1000.0); System.out.printf("[性能] FPS: %.1f%n", fps); framesSinceLastReport = 0; lastFpsTime = currentTime; } } catch (InterruptedException e) { System.out.println("[线程] 采集线程被中断"); Thread.currentThread().interrupt(); break; } catch (Exception e) { System.err.println("采集线程错误: " + e.getMessage()); e.printStackTrace(); } } System.out.println("[采集线程] 安全退出"); } /** * 优化的帧处理 */ private static void processFrame(Mat frameMat) { try { // 转换颜色空间 if (displayMat == null) { displayMat = new Mat(DEVICE_HEIGHT, DEVICE_WIDTH, frameMat.type()); } Imgproc.cvtColor(frameMat, displayMat, Imgproc.COLOR_BGR2RGB); // 获取帧参数 final int width = displayMat.cols(); final int height = displayMat.rows(); // 初始化或复用BufferedImage BufferedImage bufImg = bufImgRef.get(); if (bufImg == null || bufImg.getWidth() != width || bufImg.getHeight() != height) { bufImg = new BufferedImage(width, height, BufferedImage.TYPE_3BYTE_BGR); bufImgRef.set(bufImg); } // 直接像素复制 byte[] imgPixels = ((DataBufferByte) bufImg.getRaster().getDataBuffer()).getData(); displayMat.get(0, 0, imgPixels); // JavaFX线程安全更新 Platform.runLater(() -> { if (!isRunning.get() || imageView == null) return; // 获取或创建JavaFX图像 WritableImage fxImg = fxImgRef.get(); if (fxImg == null || (int) fxImg.getWidth() != width || (int) fxImg.getHeight() != height) { fxImg = new WritableImage(width, height); fxImgRef.set(fxImg); } // 直接写入像素数据 PixelFormat<ByteBuffer> pixelFormat = PixelFormat.getByteRgbInstance(); fxImg.getPixelWriter().setPixels(0, 0, width, height, pixelFormat, imgPixels, 0, width * 3); // 设置图像 imageView.setImage(fxImg); }); } catch (Exception e) { System.err.println("帧处理错误: " + e.getMessage()); } finally { // 释放资源 if (frameMat != null) { frameMat.release(); } // 补充新的Mat到池中 if (matPool.remainingCapacity() > 0) { matPool.offer(new Mat()); } } } /** * 检查性能并释放资源 */ private static void checkPerformance() { // 检查内存使用 long usedMemoryMB = (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()) / (1024 * 1024); System.out.printf("[性能监控] 内存使用: %d MB%n", usedMemoryMB); if (usedMemoryMB > MAX_MEMORY_MB) { System.err.printf("[性能警告] 内存使用超过阈值 (%d > %d MB),尝试释放资源%n", usedMemoryMB, MAX_MEMORY_MB); releaseUnusedResources(); } // 检查CPU使用率 double cpuUsage = calculateCpuUsage(); System.out.printf("[性能监控] CPU使用率: %.1f%%%n", cpuUsage); if (cpuUsage > MAX_CPU_USAGE) { System.err.printf("[性能警告] CPU使用率超过阈值 (%.1f > %.1f%%),尝试优化%n", cpuUsage, MAX_CPU_USAGE); optimizeForCpu(); } } /** * 计算CPU使用率 */ private static double calculateCpuUsage() { long now = System.nanoTime(); long currentTime = System.currentTimeMillis(); if (currentTime - lastCpuCheckTime < CPU_CHECK_INTERVAL) { return 0.0; } long elapsedTime = currentTime - lastCpuCheckTime; lastCpuCheckTime = currentTime; long elapsedCpu = now - lastCpuTime; lastCpuTime = now; // 简单估算CPU使用率 double cpuUsage = (elapsedCpu / 1e6) / elapsedTime * 100.0; // 限制在合理范围内 return Math.min(100.0, Math.max(0.0, cpuUsage)); } /** * 释放未使用的资源 */ private static void releaseUnusedResources() { System.out.println("[资源优化] 释放未使用资源"); // 释放Mat池中多余资源 synchronized (matPool) { int excess = matPool.size() - MAT_POOL_SIZE; if (excess > 0) { for (int i = 0; i < excess; i++) { Mat mat = matPool.poll(); if (mat != null) { mat.release(); } } System.out.printf("[资源优化] 释放了 %d 个Mat对象%n", excess); } } // 释放BufferedImage和WritableImage if (bufImgRef.get() != null) { bufImgRef.set(null); System.out.println("[资源优化] 释放BufferedImage"); } if (fxImgRef.get() != null) { fxImgRef.set(null); System.out.println("[资源优化] 释放WritableImage"); } // 强制垃圾回收 System.gc(); } /** * CPU优化策略 */ private static void optimizeForCpu() { System.out.println("[CPU优化] 降低处理负载"); // 降低帧率目标 // 这里可以根据实际情况调整帧率 // 但需要确保不影响基本功能 // 跳过一些帧处理 frameCounter.incrementAndGet(); if (frameCounter.get() > 1) { frameCounter.set(0); System.out.println("[CPU优化] 跳过一帧处理"); } } /** * 全局资源清理 */ private static synchronized void cleanupResources() { System.out.println("[资源清理] 开始清理资源"); isRunning.set(false); System.out.println("[状态] isRunning 设置为 false"); // 关闭视频捕获 if (videoCapture != null) { if (videoCapture.isOpened()) { videoCapture.release(); System.out.println("[资源] VideoCapture已释放"); } videoCapture = null; } // 释放Mat对象池 synchronized (matPool) { Mat mat; while ((mat = matPool.poll()) != null) { if (!mat.empty()) { mat.release(); } } System.out.println("[资源] Mat对象池已清空"); } // 释放显示Mat if (displayMat != null) { displayMat.release(); displayMat = null; System.out.println("[资源] displayMat已释放"); } // 关闭线程池 if (frameProcessingPool != null) { try { frameProcessingPool.shutdown(); if (!frameProcessingPool.awaitTermination(500, TimeUnit.MILLISECONDS)) { frameProcessingPool.shutdownNow(); } System.out.println("[线程池] 帧处理线程池已关闭"); } catch (InterruptedException e) { frameProcessingPool.shutdownNow(); Thread.currentThread().interrupt(); } } // 清空图像引用 Platform.runLater(() -> { if (imageView != null) { imageView.setImage(null); } }); bufImgRef.set(null); fxImgRef.set(null); System.out.println("[资源清理] 资源清理完成"); } /** * 停止采集 */ public static synchronized void stopCapture() { if (!isRunning.get()) { System.out.println("[停止采集] 采集未运行,无需停止"); return; } System.out.println("[停止采集] 请求停止..."); isRunning.set(false); if (captureThread != null && captureThread.isAlive()) { captureThread.interrupt(); try { captureThread.join(1000); System.out.println("[线程] 采集线程已停止"); } catch (InterruptedException e) { System.out.println("[线程] 停止采集线程时被中断"); Thread.currentThread().interrupt(); } } cleanupResources(); System.out.println("[停止采集] 资源已释放"); } } 目前cpu的使用率还是很高,给出优化后的完整代码
10-20
package com.PowerDesigner.com; import java.util.ArrayList; import java.util.List; interface Observer{ void update(); } interface ResponseDevice{ void performance(); } class Adapter implements Observer{ ResponseDevice device; void Adapter(ResponseDevice device){ this.device = device; } @Override public void update() { device.performance(); } } class CautionLight implements ResponseDevice{ @Override public void performance() { System.out.println("警示灯闪烁"); } } class Annunciator implements ResponseDevice{ @Override public void performance() { System.out.println("报警器报警"); } } class SecurityDoor implements ResponseDevice{ @Override public void performance() { System.out.println("安全逃生门自动开启"); } } class InsulatedDoor implements ResponseDevice{ @Override public void performance() { System.out.println("隔热门自动关闭"); } } class Thermonsensor{ private double temperature; private java.util.ArrayList<Observer> observers = new ArrayList<>(); void settemperature(double temperature){ this.temperature = temperature; if(temperature>38.0)//设置温度阈值 notifyobserver(); } void registerobserver(Observer observer){ observers.add(observer); } void removeobserver(Observer observer){ observers.remove(observer); } void notifyobserver(){ for(Observer observer:observers) observer.update(); } } public class Observer_Adapter { public static void main(String[] args) { Thermonsensor thermonsensor = new Thermonsensor(); Observer cautionLightAdapter = new Adapter(new CautionLight()); Observer annunciatorAdapter = new Adapter(new Annunciator()); Observer securityDoorAdapter = new Adapter(new SecurityDoor()); Observer insulatedDoorAdapter = new Adapter(new InsulatedDoor()); } } 为什么 Observer cautionLightAdapter = new Adapter(new CautionLight()); Observer annunciatorAdapter = new Adapter(new Annunciator()); Observer securityDoorAdapter = new Adapter(new SecurityDoor()); Observer insulatedDoorAdapter = new Adapter(new InsulatedDoor());会报错
最新发布
12-06
代码是这样处理的:package com.bms.sync.data.service; import com.bms.sync.data.util.TimeUtils; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.beans.factory.annotation.Value; import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.scheduling.annotation.Async; import org.springframework.stereotype.Service; import javax.sql.DataSource; import java.sql.Timestamp; import java.time.LocalDateTime; import java.time.format.DateTimeFormatter; import java.util.List; import java.util.Map; import java.util.Arrays; @Service public class DataMigrationService { private final JdbcTemplate mysqlJdbcTemplate; private final JdbcTemplate tdengineJdbcTemplate; @Value("${table.tag.columns}") private String tagColumnsConfig; @Value("${table.timestamp.field}") private String timestampField; // TDengine时间戳字段名 @Value("${table.tdengine.timestamp.field}") private String tdengineTimestampField; @Value("${performance.batch-size.historical}") private int historicalBatchSize; @Value("${performance.batch-size.incremental}") private int incrementalBatchSize; // 重试配置 @Value("${retry.max-attempts}") private int maxRetries; @Value("${retry.interval}") private long retryInterval; @Autowired private PerformanceMonitorService performanceMonitorService; @Autowired private TableAutoCreateService tableAutoCreateService; @Autowired private RedisQueueService redisQueueService; @Autowired public DataMigrationService(JdbcTemplate mysqlJdbcTemplate, JdbcTemplate tdengineJdbcTemplate, PerformanceMonitorService performanceMonitorService, TableAutoCreateService tableAutoCreateService, RedisQueueService redisQueueService) { this.mysqlJdbcTemplate = mysqlJdbcTemplate; this.tdengineJdbcTemplate = tdengineJdbcTemplate; this.performanceMonitorService = performanceMonitorService; this.tableAutoCreateService = tableAutoCreateService; this.redisQueueService = redisQueueService; } /** * 迁移数据(支持批量处理) * @param mysqlTableName MySQL表名 * @param superTableName TDengine超级表名 */ public void migrateData(String mysqlTableName, String superTableName) { long startTime = System.currentTimeMillis(); try { System.out.println("开始数据迁移: " + mysqlTableName + " -> " + superTableName); // 确保表结构已创建 System.out.println("步骤1: 确保表结构已创建"); ensureTableStructureExists(mysqlTableName, superTableName); // 1. 获取总记录数 System.out.println("步骤2: 获取总记录数"); int totalRecords = getTotalRecordCount(mysqlTableName); if (totalRecords == 0) { System.out.println("没有数据需要迁移"); return; } System.out.println("总记录数: " + totalRecords); // 2. 分批迁移数据(增加批处理大小以提高效率) int batchSize = historicalBatchSize; // 每批处理5000条记录 int totalBatches = (int) Math.ceil((double) totalRecords / batchSize); System.out.println("总批次数: " + totalBatches + ",批处理大小: " + batchSize); // 使用Redis队列处理历史数据 System.out.println("步骤3: 使用Redis队列处理历史数据"); processHistoricalDataWithQueue(mysqlTableName, superTableName, totalRecords, batchSize); System.out.println("数据迁移完成"); } catch (Exception e) { System.err.println("数据迁移失败: " + e.getMessage()); e.printStackTrace(); } finally { long endTime = System.currentTimeMillis(); performanceMonitorService.recordTime("data.migration", endTime - startTime); performanceMonitorService.incrementCounter("data.migration.totalRecords", getTotalRecordCount(mysqlTableName)); System.out.println("数据迁移总耗时: " + (endTime - startTime) + "ms"); } } /** * 确保表结构已创建 * @param mysqlTableName MySQL表名 * @param superTableName TDengine超级表名 */ public void ensureTableStructureExists(String mysqlTableName, String superTableName) { try { System.out.println("确保表结构存在: " + mysqlTableName + " -> " + superTableName); // 从配置中获取标签列 String[] tagColumnsArray = tagColumnsConfig.split(","); List<String> tagColumns = Arrays.asList(tagColumnsArray); // 自动创建表结构 tableAutoCreateService.autoCreateTableStructure(mysqlTableName, superTableName, tagColumns); System.out.println("表结构创建完成"); } catch (Exception e) { System.err.println("确保表结构存在时失败: " + e.getMessage()); e.printStackTrace(); } } /** * 使用Redis队列处理历史数据 * @param mysqlTableName MySQL表名 * @param superTableName TDengine超级表名 * @param totalRecords 总记录数 * @param batchSize 批处理大小 */ private void processHistoricalDataWithQueue(String mysqlTableName, String superTableName, int totalRecords, int batchSize) { try { String queueName = mysqlTableName + "_to_" + superTableName; // 清空之前的队列数据 redisQueueService.clearQueue(queueName); // 使用基于时间戳的分页方式替代LIMIT/OFFSET,避免数据遗漏 String lastProcessedTime = null; String currentTime = java.time.LocalDateTime.now().format(java.time.format.DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")); long startTime = System.currentTimeMillis(); int totalProcessed = 0; boolean hasMoreData = true; while (hasMoreData) { // 构建查询SQL,使用时间戳分页 StringBuilder querySql = new StringBuilder(); querySql.append("SELECT * FROM ").append(mysqlTableName); // 添加时间范围条件,确保时间字段不为空 querySql.append(" WHERE ").append(timestampField).append(" IS NOT NULL"); // 确保时间字段不为空 if (lastProcessedTime != null && !lastProcessedTime.isEmpty()) { querySql.append(" AND ").append(timestampField).append(" > '").append(lastProcessedTime).append("'"); } querySql.append(" ORDER BY ").append(timestampField).append(" ASC"); querySql.append(" LIMIT ").append(batchSize); List<Map<String, Object>> records = mysqlJdbcTemplate.queryForList(querySql.toString()); if (!records.isEmpty()) { // 将数据放入Redis队列 redisQueueService.pushHistoricalDataToQueue(queueName, records); // 更新最大时间戳 Object lastTimestamp = records.get(records.size() - 1).get(timestampField); if (lastTimestamp != null) { lastProcessedTime = lastTimestamp.toString(); } totalProcessed += records.size(); System.out.println("已将 " + records.size() + " 条记录放入Redis队列,总计已处理: " + totalProcessed); } else { hasMoreData = false; } } long endTime = System.currentTimeMillis(); System.out.println("历史数据已全部放入Redis队列,队列大小: " + redisQueueService.getQueueSize(queueName) + ",耗时: " + (endTime - startTime) + "ms"); // 从Redis队列中取出数据并同步到TDengine int successCount = syncDataFromQueue(queueName, superTableName); System.out.println("历史数据同步完成,总计成功同步记录数: " + successCount); } catch (Exception e) { System.err.println("使用Redis队列处理历史数据失败: " + e.getMessage()); e.printStackTrace(); } } /** * 从Redis队列中取出数据并同步到TDengine * @param queueName 队列名称 * @param superTableName TDengine超级表名 * @return 成功同步的记录数 */ private int syncDataFromQueue(String queueName, String superTableName) { int totalSynced = 0; int batchSize = incrementalBatchSize; // 从队列中每次取出100条记录进行同步 long startTime = System.currentTimeMillis(); System.out.println("开始从Redis队列同步数据到TDengine,批处理大小: " + batchSize); int batchCount = 0; while (redisQueueService.getQueueSize(queueName) > 0) { // 先查看但不移除队列中的数据 List<Object> records = redisQueueService.peekBatchFromQueue(queueName, batchSize); if (records != null && !records.isEmpty()) { // 转换为正确的类型 List<Map<String, Object>> convertedRecords = (List<Map<String, Object>>) (List<?>) records; // 批量插入到TDengine并统计成功插入的记录数 int successCount = batchInsertIntoTDengine(superTableName, convertedRecords); totalSynced += successCount; batchCount++; // 只有在数据成功同步后才从队列中移除 if (successCount > 0) { // 从队列中移除已成功处理的数据 redisQueueService.removeBatchFromQueue(queueName, successCount); } // 每处理10批打印一次进度信息 if (batchCount % 10 == 0) { System.out.println("已同步 " + batchCount + " 批次数据,当前批次成功记录数: " + successCount + ",总计已同步: " + totalSynced); } } } long endTime = System.currentTimeMillis(); // 只有在有数据成功同步时才打印完成信息 if (totalSynced > 0) { System.out.println("从Redis队列同步数据完成,总计同步: " + totalSynced + " 条记录,总批次数: " + batchCount + ",耗时: " + (endTime - startTime) + "ms"); } else { System.out.println("Redis队列中没有数据需要同步"); } return totalSynced; } /** * 获取记录数 * @param mysqlTableName MySQL表名 * @return 记录数 */ public int getRecordCount(String mysqlTableName) { // 直接从数据库获取,不使用缓存 String countSql = "SELECT COUNT(*) FROM " + mysqlTableName; Integer count = mysqlJdbcTemplate.queryForObject(countSql, Integer.class); return count != null ? count : 0; } /** * 获取总记录数 * @param tableName 表名 * @return 总记录数 */ public int getTotalRecordCount(String tableName) { // 直接从数据库获取,不使用缓存 String countSql = "SELECT COUNT(*) FROM " + tableName; Integer count = mysqlJdbcTemplate.queryForObject(countSql, Integer.class); return count != null ? count : 0; } /** * 批量插入数据到TDengine * @param superTableName TDengine超级表名 * @param records 记录列表 * @return 成功插入的记录数 */ private int batchInsertIntoTDengine(String superTableName, List<Map<String, Object>> records) { if (records.isEmpty()) { return 0; } int successCount = 0; try { // 一次性检查并创建所有需要的子表 ensureAllChildTablesExist(superTableName, records); // 然后批量插入数据 for (Map<String, Object> record : records) { try { insertIntoTDengine(superTableName, record); successCount++; } catch (Exception e) { System.err.println("单条记录插入失败: " + e.getMessage()); } } } catch (Exception e) { System.err.println("批量数据插入失败: " + e.getMessage()); // 不打印完整堆栈,减少日志输出 } return successCount; } /** * 一次性检查并创建所有需要的子表 * @param superTableName 超级表名 * @param records 记录列表 */ private void ensureAllChildTablesExist(String superTableName, List<Map<String, Object>> records) { try { // 首先确保超级表存在 // 不使用缓存检查超级表是否存在 // 直接确保超级表存在 ensureSuperTableExists(superTableName); // 不使用缓存存储超级表存在信息 // 收集所有需要的子表名 java.util.Set<String> childTableNames = new java.util.HashSet<>(); java.util.Map<String, String> childTableToTagMap = new java.util.HashMap<>(); String tagColumn = tagColumnsConfig; for (Map<String, Object> record : records) { Object tagValueObj = record.get(tagColumn); String tagValue = (tagValueObj != null) ? tagValueObj.toString() : "null"; String childTableName = superTableName + "_" + tagValue; // 仅处理缓存中不存在的表 String cacheKey = "tdengine:table:" + childTableName; // 不使用缓存,直接检查表是否存在 if (!childTableNames.contains(childTableName)) { childTableNames.add(childTableName); childTableToTagMap.put(childTableName, tagValue); } } // 一次性查询所有子表是否存在 if (!childTableNames.isEmpty()) { // 使用更高效的方式检查表是否存在 for (String childTableName : childTableNames) { if (!isTableExists(childTableName)) { String tagValue = childTableToTagMap.get(childTableName); createChildTable(superTableName, childTableName, tagValue); } else { // 缓存已存在的表信息 // 不使用缓存存储表存在信息 } } } } catch (Exception e) { System.err.println("批量检查子表存在时失败: " + e.getMessage()); // 不打印完整堆栈,减少日志输出 } } /** * 检查TDengine表是否存在 * @param tableName 表名 * @return 是否存在 */ private boolean isTableExists(String tableName) { try { String checkSql = "SELECT table_name FROM information_schema.ins_tables WHERE table_name = ?"; List<Map<String, Object>> result = tdengineJdbcTemplate.queryForList(checkSql, tableName); return !result.isEmpty(); } catch (Exception e) { System.err.println("检查表是否存在时出错: " + e.getMessage()); return false; } } /** * 创建子表 * @param superTableName 超级表名 * @param childTableName 子表名 * @param tagValue 标签值 */ private void createChildTable(String superTableName, String childTableName, String tagValue) { try { String createChildTableSql = "CREATE TABLE IF NOT EXISTS " + childTableName + " USING " + superTableName + " TAGS ('" + tagValue + "')"; tdengineJdbcTemplate.execute(createChildTableSql); // 缓存表存在信息 // 不使用缓存存储表存在信息 System.out.println("创建子表: " + childTableName); } catch (Exception e) { System.err.println("创建子表失败: " + e.getMessage()); } } /** * 插入数据到TDengine * @param superTableName TDengine超级表名 * @param record 记录 */ public void insertIntoTDengine(String superTableName, Map<String, Object> record) { long startTime = System.currentTimeMillis(); int retryCount = 0; while (retryCount < maxRetries) { try { // 获取标签列名(从配置中获取) String tagColumn = tagColumnsConfig; // 获取标签值 Object tagValueObj = record.get(tagColumn); String tagValue = (tagValueObj != null) ? tagValueObj.toString() : "null"; // 生成子表名(使用标签值) String childTableName = superTableName + "_" + tagValue; // 优化:仅在缓存中不存在时检查子表 String cacheKey = "tdengine:table:" + childTableName; // 不使用缓存检查表是否存在 // 直接检查表是否存在 if (!isTableExists(childTableName)) { // 确保子表存在 ensureChildTableExists(superTableName, childTableName, tagValue); // 缓存表存在信息,避免频繁检查 // 不使用缓存存储表存在信息 } // 构建SQL StringBuilder sql = new StringBuilder(); sql.append("INSERT INTO ").append(childTableName).append(" USING ").append(superTableName) .append(" TAGS ('").append(tagValue).append("') (").append(tdengineTimestampField).append(", "); // 添加列名 boolean first = true; for (String columnName : record.keySet()) { // 跳过标签列和时间戳字段 if (tagColumn.equals(columnName) || tdengineTimestampField.equals(columnName)) { continue; } if (!first) { sql.append(", "); } // 处理关键字冲突 sql.append(escapeKeyword(columnName)); first = false; } // 获取时间戳字段的值 Object timestampValue = record.get(timestampField); if (timestampValue == null) { // 如果时间戳为空,跳过这条记录 System.err.println("警告: 记录的时间戳字段为空,跳过该记录"); return; } String formattedTimestamp; if (timestampValue instanceof Timestamp) { // 格式化时间戳 Timestamp ts = (Timestamp) timestampValue; formattedTimestamp = "'" + TimeUtils.formatWithMillis(TimeUtils.from(ts)) + "'"; } else if (timestampValue instanceof LocalDateTime) { // 格式化LocalDateTime LocalDateTime ldt = (LocalDateTime) timestampValue; formattedTimestamp = "'" + TimeUtils.formatWithMillis(ldt) + "'"; } else { // 其他情况使用字符串值 formattedTimestamp = "'" + timestampValue.toString() + "'"; } sql.append(") VALUES (").append(formattedTimestamp).append(", "); // 添加值 first = true; for (Map.Entry<String, Object> entry : record.entrySet()) { String columnName = entry.getKey(); Object value = entry.getValue(); // 跳过标签列和时间戳字段 if (tagColumn.equals(columnName) || tdengineTimestampField.equals(columnName)) { continue; } if (!first) { sql.append(", "); } // 处理NULL值 if (value == null) { sql.append("NULL"); } else if (value instanceof String || value instanceof Timestamp || value instanceof LocalDateTime) { // 对字符串和时间戳类型进行特殊处理 if (value instanceof Timestamp) { // 格式化时间戳 Timestamp ts = (Timestamp) value; sql.append("'").append(TimeUtils.formatWithMillis(TimeUtils.from(ts))).append("'"); } else if (value instanceof LocalDateTime) { // 格式化LocalDateTime LocalDateTime ldt = (LocalDateTime) value; sql.append("'").append(TimeUtils.formatWithMillis(ldt)).append("'"); } else { // 字符串值需要添加引号,并处理NULL字符串 String strValue = value.toString(); if ("NULL".equalsIgnoreCase(strValue)) { sql.append("NULL"); } else { sql.append("'").append(strValue.replace("'", "''")).append("'"); } } } else { // 其他类型直接添加 sql.append(value.toString()); } first = false; } sql.append(")"); // System.out.println("SQL语句: " + sql.toString()); tdengineJdbcTemplate.execute(sql.toString()); // 更新性能监控 performanceMonitorService.incrementCounter("data.migration.insert.success", 1); break; // 成功执行,跳出重试循环 } catch (Exception e) { retryCount++; System.err.println("数据插入失败,正在进行第" + retryCount + "次重试: " + e.getMessage()); if (retryCount >= maxRetries) { System.err.println("数据插入最终失败: " + e.getMessage()); performanceMonitorService.incrementCounter("data.migration.insert.failure", 1); throw new RuntimeException("数据插入失败", e); } else { try { Thread.sleep(retryInterval * retryCount); // 使用配置的重试间隔和指数退避 } catch (InterruptedException ie) { Thread.currentThread().interrupt(); throw new RuntimeException("重试被中断", ie); } } } finally { long endTime = System.currentTimeMillis(); performanceMonitorService.recordTime("data.migration.insert", endTime - startTime); } } } /** * 确保子表存在 * @param superTableName 超级表名 * @param childTableName 子表名 * @param tagValue 标签值 */ private void ensureChildTableExists(String superTableName, String childTableName, String tagValue) { // 使用类级锁来确保在多线程环境下不会重复创建表 synchronized (DataMigrationService.class) { try { // 首先确保超级表存在(使用缓存优化) String superTableCacheKey = "tdengine:stable:" + superTableName; // 不使用缓存检查超级表是否存在 // 直接确保超级表存在 ensureSuperTableExists(superTableName); // 不使用缓存存储超级表存在信息 // 检查子表是否存在 if (!isTableExists(childTableName)) { createChildTable(superTableName, childTableName, tagValue); } } catch (Exception e) { System.err.println("确保子表存在时失败: " + e.getMessage()); // 不打印完整堆栈,减少日志输出 } } } /** * 确保超级表存在 * @param superTableName 超级表名 */ private void ensureSuperTableExists(String superTableName) { try { // 检查超级表是否存在 if (!isTableExists(superTableName)) { // 超级表应该已经由TableAutoCreateService创建,这里只是确保缓存存在 // 不再创建默认的超级表结构 System.out.println("警告: 超级表 " + superTableName + " 不存在,需要先分析MySQL表结构并创建"); } } catch (Exception e) { System.err.println("确保超级表存在时失败: " + e.getMessage()); // 不打印完整堆栈,减少日志输出 } } /** * 处理SQL关键字冲突,为字段名添加反引号 * @param columnName 字段名 * @return 处理后的字段名 */ private String escapeKeyword(String columnName) { // TDengine中的关键字列表(部分) String[] keywords = {"query", "order", "group", "limit", "offset", "select", "from", "where", "insert", "update", "delete"}; for (String keyword : keywords) { if (keyword.equalsIgnoreCase(columnName)) { return "`" + columnName + "`"; } } return columnName; } }
11-15
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值