I need to export ~150K records * 27 fields into xlsx and face problems at ~130K with heap of 10G size.
I've partitioned output but still have problem because all writes are at the same phase, it's was not the problem of single file as I supposed.
But if partitions are sorted and compact, is it possible to close completed file and free memory? It is and advice for implementation.
And can I solve my problem in any other way? Clover is 3.1.2. Xlsx format is necessary because there is a pivot on the other sheet which cleans up unless I use xlsx.
I've partitioned output but still have problem because all writes are at the same phase, it's was not the problem of single file as I supposed.
But if partitions are sorted and compact, is it possible to close completed file and free memory? It is and advice for implementation.
And can I solve my problem in any other way? Clover is 3.1.2. Xlsx format is necessary because there is a pivot on the other sheet which cleans up unless I use xlsx.
-
Hi golomeen,
in 3.3M1+ versions is new component http://doc.cloveretl.com/documentation/ ... riter.html which is able to work in stream mode. Than decreases memory requirements a lot.
If upgrade is no option for you, you can try 2 things:
1] If I understood you correctly, you are appending to existing file, right? Check whether this problem persist when you will write into new file, not append into existing. Maybe file you are appending to is too big? It must be whole in memory in form of DOM tree (many times bigger space consumption than on drive).
2] Try to simplify your graph to DataGenerator+XlsDataWriter. Does this problem persist? Can you send us such simplified graph? If you are appending to existing xlsx, please send also that file.
Please sign in to leave a comment.
Comments 1