When trying to create metadata by "extract from XLS(x)" (one of the default metadata creation options), I am getting a "Java Heap Space" error when the file is larger than ~5MB. Since this is not a memory issue during the graph running, I don't think the properties file is of any use here. Any ideas how to solve this (aside from truncating file)?
I tried increasing the Xmx setting in CloverETLDesigner.ini, but for some reason it won't let me go above 1136M. When I do the INI file pops up when trying to start the designer and it never starts. At 1024M I still get the heap space error. I also tried a 64bit JVM (with -vm setting in the eclipse.ini file) and this made no difference. My machine has 4GB RAM. When watching the designer's memory usage during its attempt to read the XLSX file for metadata in quickly goes from 157M to 1200M. More thoughts?
I tried increasing the Xmx setting in CloverETLDesigner.ini, but for some reason it won't let me go above 1136M. When I do the INI file pops up when trying to start the designer and it never starts. At 1024M I still get the heap space error. I also tried a 64bit JVM (with -vm setting in the eclipse.ini file) and this made no difference. My machine has 4GB RAM. When watching the designer's memory usage during its attempt to read the XLSX file for metadata in quickly goes from 157M to 1200M. More thoughts?
-
Hello, the XLSX reading and metadata extraction process need to load whole XLSX document into memory and this process can reach the Java memory limitations. This is actually one of the improvements planned for CloverETL 3.2. As a workaround, the best way to extract metadata from a large XLSX file is to make a separate XLSX file with just a few records and use this file instead. In order to read such big XLSX files, I recommend to convert them to XLS format and process those.
The maximum java heap space size around 1.1 GB (as you are describing in your post) is in correspondence with Oracle's FAQ (Frequently Asked Questions About the Java HotSpot VM):The maximum theoretical heap limit for the 32-bit JVM is 4G. Due to various additional constraints such as available swap, kernel address space usage, memory fragmentation, and VM overhead, in practice the limit can be much lower. On most modern 32-bit Windows systems the maximum heap size will range from 1.4G to 1.6G. On 32-bit Solaris kernels the address space is limited to 2G. On 64-bit operating systems running the 32-bit VM, the max heap size can be higher, approaching 4G on many Solaris systems.
Using 64 bit JVM with 64 bit Eclipse and CloverETL you should be able to increase java heap size significantly. This however, requires 64 bit version of operating system to be installed as well. -
I have related problem - our graphs require a lot of memory to sort data, and only 1536M setting works on both 8G and 16G machines @Win x64
2048M setting do not ever worked
How can i correctly turn my Clover desktop into 64-bit one?
And how to make this setting common for any graph running? -
Hello golomeen, when you sort lots of data always use ExtSort instead of FastSort. ExtSort needs significantly less memory and should work on large data.
If you install 64bit JRE you can set CloverETL to use it as default in Window | Preferences | Java | Installed JREs (you will probably need to set it up using the Add button).
Please sign in to leave a comment.
Comments 3