OutOfMemoryError: 请求的数组大小超过了虚拟机限制，位于Apache Camel内。

huangapple
未分类评论49阅读模式
英文:

OutOfMemoryError: Requested array size exceeds VM limit in apache camel

问题

以下是翻译好的内容：

我的应用程序使用Apache Camel处理一个txt文件，应用程序使用交换对象将整个文件读入字符串中，然后对每一行执行分隔符添加操作，并使用Apache Camel的交换对象将整个字符串写入文件。我遇到了以下的OutOfMemoryError。我观察到这个异常是在交换对象完成文件处理后，即将文件移动到另一个文件夹（终端点）时发生的。

我的应用程序有16GB的堆空间支持。文件在处理之前的大小为685MB，添加分隔符后可能会增加额外的200MB。有人可以帮助我吗？提前谢谢。

from("file:<filepath>/processed/INPUT?filter=#fileFilter&delay=120000&exclusiveReadLockStrategy=#fileReadLockStrategy&delete=true&autoCreate=true")
    .threads(10, 10)
    .process(fileProcessor).log("Came out of file processor")
    .recipientList(header(DESTIONATION_PATH));

这是一个简单的路由，从INPUT文件夹中读取文件，fileProcessor处理文件内容，添加分隔符。DESTIONATION_PATH是文件将被放置的路径。我可以看到它打印了"log Came out of file processor"，在文件即将发送到目标文件夹时，会出现以下异常。

错误已发生：
org.apache.camel.CamelExecutionException：在交换上执行时发生异常：Exchange[ID-x01btadmdts3a-vsi-uat-dbs-com-1586519346908-0-4]
	在 org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1842) ~[camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:385) ~[camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:64) ~[camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.RecipientList.process(RecipientList.java:133) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157) [camel-core-2.24.1.jar!/:2.24.1]
	在 org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87) [camel-core-2.24.1.jar!/:2.24.1]
	在 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]
	在 java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
	在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
	在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
	在 java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
	被压制：org.apache.camel.CamelExecutionException：在交换上执行时发生异常：Exchange[ID-x01btadmdts3a-vsi-uat-dbs-com-1586519346908-0-4]
		... 26 common frames omitted
	***导致：java.lang.OutOfMemoryError：请求的数组大小超过了VM限制***
		在 java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
		在 java.lang.StringCoding.encode(StringCoding.java:344)
		在 java.lang.String.getBytes(String.java:918)
		在 org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:198)
		在 org.apache.camel.converter.IOConverterOptimised.convertTo(IOConverterOptimised.java:55)
		在 org.apache.camel.impl.converter.OptimisedTypeConverter.convertTo(OptimisedTypeConverter.java:63)
		在 org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:299)
		在 org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:188)
		在 org.apache

<details>
<summary>英文:</summary>

In my application a txt file is being processed using apache camel, application reads whole file into string using exchange object &amp; performs delimeter adding on each line &amp; writes into to the file as a whole string using exchange object of apache camel. I am getting below OutOfMemoryError. ***I observed this exception is occuring after file processing is completed by exchange object &amp; when it is supposed to move the file to another folder(endpoint)* **.

*My application has 16GB heap space support*. File size before processing is 685MB &amp; after adding delimiter it may increase 200 MB extra.
Can anyone help me here? Thank you in advance.

```code
from(&quot;file:&lt;filepath&gt;/processed/INPUT?filter=#fileFilter&amp;delay=120000&amp;exclusiveReadLockStrategy=#fileReadLockStrategy&amp;delete=true&amp;autoCreate=true&quot;)
					.threads(10, 10)
					.process(fileProcessor).log(&quot;Came out of file processor&quot;)
					.recipientList(header(DESTIONATION_PATH));

Its a simple route which read file from INPUT folder & fileProcessor processes the file contents by adding delimter in it. DESTIONATION_PATH is the path where the file will be placed. I can see that it is printing log "Came out of file processor" , below Exception is coming when file is about to send to the destination folder.

Error has occurred: 
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-x01btadmdts3a-vsi-uat-dbs-com-1586519346908-0-4]
	at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1842) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:385) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:64) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.RecipientList.process(RecipientList.java:133) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87) [camel-core-2.24.1.jar!/:2.24.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
	Suppressed: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-x01btadmdts3a-vsi-uat-dbs-com-1586519346908-0-4]
		... 26 common frames omitted
	***Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit***
		at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
		at java.lang.StringCoding.encode(StringCoding.java:344)
		at java.lang.String.getBytes(String.java:918)
		at org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:198)
		at org.apache.camel.converter.IOConverterOptimised.convertTo(IOConverterOptimised.java:55)
		at org.apache.camel.impl.converter.OptimisedTypeConverter.convertTo(OptimisedTypeConverter.java:63)
		at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:299)
		at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:188)
		at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:115)
		at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:333)
		at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305)
		at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169)
		at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80)
		at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
		at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
		at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
		at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715)
		at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638)
		at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
		at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173)
		at org.apache.camel.processor.RecipientList.process(RecipientList.java:133)
		at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
		at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
		at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76)
		at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
		at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
		at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
		at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
		at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
		at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43)
		at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157)
		at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87)
Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300) ~[na:1.8.0_171]
	at java.lang.StringCoding.encode(StringCoding.java:344) ~[na:1.8.0_171]
	at java.lang.String.getBytes(String.java:918) ~[na:1.8.0_171]
	at org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:198) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.converter.IOConverterOptimised.convertTo(IOConverterOptimised.java:55) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.impl.converter.OptimisedTypeConverter.convertTo(OptimisedTypeConverter.java:63) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:299) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:188) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:115) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:333) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) ~[camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.RecipientList.process(RecipientList.java:133) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157) [camel-core-2.24.1.jar!/:2.24.1]
	at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87) [camel-core-2.24.1.jar!/:2.24.1]

答案1

得分: 0

抱歉晚回复。我已经成功解决了上述问题。
当我仔细查看日志时，发现问题出在标准的Java类 "StringCoding.java" 中。

问题出在我的代码的以下两行：

String content = exchange.getIn().getBody(String.class);
//在每行内容字符串中添加分隔符，并将整个字符串写回文件。
//从原始内容中派生新的内容字符串
exchange.getOut().setBody(content);// Camel将内容存储在字符串中，并通过使用字符数组通过路由发送它。

分析:

Java将字符串存储为字符数组，这是在幕后进行的。根据逻辑，我们在每一行中添加了一些分隔符。
我观察到（原始文件中的字符数 + 添加的分隔符数）大于 Integer.MAX_VALUE。
所以，这里Java无法将整个字符串存储到字符数组中，而且Camel无法将文件放在目标文件夹路径中。
这是因为修改后的文件中的字符数超过了最大整数大小。

解决方案:

File contentFile = exchange.getIn().getBody(File.class);
//逐行处理 contentFile 并将记录写回文件。
exchange.getOut().setBody(contentFile);// Camel将内容存储在文件中，并通过使用InputStream通过路由发送它。

我保留了上述代码，并且它起作用了，我们没有看到内存溢出的问题，因为Camel在通过路由发送文件时没有使用数组。

谢谢阅读。愉快编码...！！！

英文:

Sorry for answering late here. I was able to resolve above issue.
When I looked at logs carefully it was failing in the standard java class "StringCoding.java".

Culprit in my code was below 2 lines:

String content = exchange.getIn().getBody(String.class); 
//Add deleimters in each line content string &amp; write back the whole string in file.
//derive new content string from original content
exchange.getOut().setBody(content);// Camel stores content in String &amp; sends it via route using character array.

Analysis:

Java stores string as a character array in behind the scene. As per logic we were adding some delimeters in each line.
I observed that (number of characters in original file + number of delimiters added) was > Integer.MAX_VALUE.
So, here the java was failing to store the whole string in to character array & camel was not able put the file
in the destination folder path. This happened because the number of characters in modified file went beyond maximum integer size.

Solution:

File contentFile = exchange.getIn().getBody(File.class);
//Process the contentFile as file line by line &amp; write the records back in file.
exchange.getOut().setBody(contentFile);// Camel stores content in File &amp; sends it via route using InputStream.

I kept above code & it worked we didn't see OutofMemory issue as no where array is used by camel while sending file via route.

Thank you for reading. Happy Coding...!!!

huangapple
  • 本文由 发表于 2020年4月11日 00:18:36
  • 转载请务必保留本文链接：https://java.coder-hub.com/61144326.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定