Customer Portal

metadata from XLSX heap error

Comments 3

  • Avatar
    ikulman
    0
    Comment actions Permalink
    Hello, the XLSX reading and metadata extraction process need to load whole XLSX document into memory and this process can reach the Java memory limitations. This is actually one of the improvements planned for CloverETL 3.2. As a workaround, the best way to extract metadata from a large XLSX file is to make a separate XLSX file with just a few records and use this file instead. In order to read such big XLSX files, I recommend to convert them to XLS format and process those.

    The maximum java heap space size around 1.1 GB (as you are describing in your post) is in correspondence with Oracle's FAQ (Frequently Asked Questions About the Java HotSpot VM):
    The maximum theoretical heap limit for the 32-bit JVM is 4G. Due to various additional constraints such as available swap, kernel address space usage, memory fragmentation, and VM overhead, in practice the limit can be much lower. On most modern 32-bit Windows systems the maximum heap size will range from 1.4G to 1.6G. On 32-bit Solaris kernels the address space is limited to 2G. On 64-bit operating systems running the 32-bit VM, the max heap size can be higher, approaching 4G on many Solaris systems.

    Using 64 bit JVM with 64 bit Eclipse and CloverETL you should be able to increase java heap size significantly. This however, requires 64 bit version of operating system to be installed as well.
  • Avatar
    golomeen
    0
    Comment actions Permalink
    I have related problem - our graphs require a lot of memory to sort data, and only 1536M setting works on both 8G and 16G machines @Win x64

    2048M setting do not ever worked

    How can i correctly turn my Clover desktop into 64-bit one?
    And how to make this setting common for any graph running?
  • Avatar
    ikulman
    0
    Comment actions Permalink
    Hello golomeen, when you sort lots of data always use ExtSort instead of FastSort. ExtSort needs significantly less memory and should work on large data.

    If you install 64bit JRE you can set CloverETL to use it as default in Window | Preferences | Java | Installed JREs (you will probably need to set it up using the Add button).

Please sign in to leave a comment.