呼叫Java_使用Java处理HTTP呼叫的大文件

小编典典

我想并行化调用，但是不确定是否应该将整个文件读入内存

您应该使用ExecutorService与bounded的BlockingQueue。在读入百万行时，您将作业提交到线程池，直到作业BlockingQueue满为止。这样，您将能够同时运行100个(或最佳数量)的HTTP请求，而无需事先读取文件的所有行。

RejectedExecutionHandler如果队列已满，您将需要设置一个阻止的对象。这比调用方运行处理程序更好。

BlockingQueue queue = new ArrayBlockingQueue(100);

// NOTE: you want the min and max thread numbers here to be the same value

ThreadPoolExecutor threadPool =

new ThreadPoolExecutor(nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, queue);

// we need our RejectedExecutionHandler to block if the queue is full

threadPool.setRejectedExecutionHandler(new RejectedExecutionHandler() {

@Override

public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {

try {

// this will block the producer until there's room in the queue

executor.getQueue().put(r);

} catch (InterruptedException e) {

throw new RejectedExecutionException(

"Unexpected InterruptedException", e);

}

}

});

// now read in the urls

while ((String url = urlReader.readLine()) != null) {

// submit them to the thread-pool. this may block.

threadPool.submit(new DownloadUrlRunnable(url));

}

// after we submit we have to shutdown the pool

threadPool.shutdown();

// wait for them to complete

threadPool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);

...

private class DownloadUrlRunnable implements Runnable {

private final String url;

public DownloadUrlRunnable(String url) {

this.url = url;

}

public void run() {

// download the URL

}

}

2020-09-23