Is it possible to free memory from previous phases of a graph?
I've got a graph that has multiple phases, the last of which does a ExtSort and then Rollup. This usually works fine, but if the input file is unusually large the graph dies with a "java.lang.OutOfMemoryError: Java heap space". error from the ExtSort. Some of the earlier phases perform ExtSort's with no problems.
(I'm using the -Xmx512 setting)
I'm wondering if there's a way to free some of the earlier phase components from memory?
I've got a graph that has multiple phases, the last of which does a ExtSort and then Rollup. This usually works fine, but if the input file is unusually large the graph dies with a "java.lang.OutOfMemoryError: Java heap space". error from the ExtSort. Some of the earlier phases perform ExtSort's with no problems.
(I'm using the -Xmx512 setting)
I'm wondering if there's a way to free some of the earlier phase components from memory?
-
Hi CodeMonkee,
Clover does free memory after a phase ends. So, it might be a bug in your case. But we need to see your graph to be sure about this.
Anyway, ExtSort is easily adaptable to actual memory consumption by increasing the Buffer capacity (10.485.760 records by default) and the Sorter initial capacity (8.000 by default) parameters. The total capacity is approximately 69,91 * Sorter initial capacity. Try to adjust these parameters according to your graph needs.
Please sign in to leave a comment.
Comments 1