英文:
Multi Thread (ExecutorService) Java on GCP
问题
目前我正在与GCP上的演示项目一起工作。我从一个存储桶中读取了一个巨大的文件，进行一些处理，然后将结果写入另一个存储桶。
在读取文件后，我使用ExecutorService将文件的每一行发送到一个线程中。我测试了使用50、25和10个线程的处理方式（Executors.newFixedThreadPool(10);）。在本地环境中（使用10和25个线程），处理效果非常好。然而，一旦我在GCP Kubernetes上部署项目，它在某个点上就会崩溃（没有日志，没有错误，Pod只是终止并重新启动）。
你有任何想法为什么会发生这种情况吗？
配置信息：1个集群（版本1.14.10-gke.36），共3个vCPU，7.5GB的RAM。每个节点有1个vCPU和2.77GB的RAM，扩展到单个Pod上也是相同的配置。
代码的一部分：
void someMethod(List<String> strings) {
ExecutorService executor = Executors.newFixedThreadPool(10);
strings.forEach(string -> {
executor.execute(() -> {
//一些代码...
toProcess(string);
//一些代码...
});
});
executor.shutdown();
//等待直到所有线程完成
while (!executor.isTerminated()) {}
//一些其他任务的完成
}
public void toProcess(String string) {
Processable[] toDo = {someVisitors};
for (Processable process : toDo) {
if (process.accept(address)) {
break;
}
}
}
（注意：这里只返回翻译的部分，不包含问题的回答。）
英文:
At the moment I'm working with a demo on GCP I read a huge File from a bucket do some process and then write on another bucket the result.
Once I read the file I send each line of the file to a Thread using ExecutorService, the thing is I've tested the process with 50, 25 and 10 Threads (Executors.newFixedThreadPool(10);) and locally work pretty well (with 10 and 25 threads), however, once I deploy the project on the GCP Kubernetes it dies at some point (no log no error the pod just die and it restarts).
Any idea why this could happen?
The config; 1 Cluster (1.14.10-gke.36) 3vCPUs, 7.5gb ram from which I have 3 nodes each one with 1 vCPU and 2.77gb RAM by extension the same to a single pod.
Part of the code;
void someMethod (List<String> strings){
ExecutorService executor = Executors.newFixedThreadPool(10);
strings.forEach(string -> {
executor.execute(()-> {
//Some code...
toProcess(string);
//Some code...
});
});
executor.shutdown();
// Wait until all threads are finished
while (!executor.isTerminated()) {}
//Some more things to finish
}
public void toProcess(String string){
Processable[] toDo = {someVisitors};
for (Processable process : toDo) {
if(process.accept(address)) {
break;
}
}
}
答案1
得分: 0
这是由于Pod的内存不足引起的。该过程使它们的内存耗尽，导致它们重新启动，通过为它们分配更多内存来解决了这个问题。
英文:
This was due to lack of memory of the Pods. The process was making them run out of memory which caused them to restart, this was solved by assigning more memory to them.
专注分享java语言的经验与见解，让所有开发者获益！
评论