I have a graph that takes some data from a database, converts it into XML, and pumps the XML to a Solr server via HTTP. The strange thing is that each time this graph runs, it seems to consume more memory than the last time. The first run after a restart, it fluctuates between 500 MB and 1.5 GB. The second run fluctuates between 500 MB and 2.25 GB (with the lower bound going up during the duration of the graph). The third run fluctuates between 500 MB and 4 GB. Subsequent runs fluctuate between 500 MB and 7+ GB.
Are there components I should be looking at as potential troublemakers for this? I use HTTP_Connectors, an XMLWriter, a few DBInputTables, some DataGenerators, a SimpleGather, and a small DataWriter to write a log file at the end.
Are there components I should be looking at as potential troublemakers for this? I use HTTP_Connectors, an XMLWriter, a few DBInputTables, some DataGenerators, a SimpleGather, and a small DataWriter to write a log file at the end.
-
Hello, DTaylor,
I have a few additional questions:
1. Which version of CloverETL Designer and Server do you have? Is it local or Server project?
2. Do you use some custom components or java code as transformations?
3. How much records do you process and how do metadata look like?
4. Do you use default java from Sun or some other from IBM or someone else? Which version of java do you have?
5. Did you use jvisualvm for your measures? Can you use it to post also some information about Classes loaded and Threads used?
6. Can you post also your graph? Without any credentials and other possibly confidential things, of course.
Thank you in advance for your answers.
Best regards, -
I have a few additional questions:
1. Which version of CloverETL Designer and Server do you have? Is it local or Server project?
Server is running 3.3.0.21, Designer is 3.3.1.011P. The graph is running in a sandbox on the server.2. Do you use some custom components or java code as transformations?
No custom components or java transformations.3. How much records do you process and how do metadata look like?
The largest channel is a little over 6 million records. There are two additional channels with a little over 800K records and nearly a million records. These are all combined into about 130 XML documents before being transferred to solr. The metadata for the biggest set of records is only four fields (a long, two strings, and a decimal).4. Do you use default java from Sun or some other from IBM or someone else? Which version of java do you have?
We use Java 1.6 from Sun. For connecting to the SQL Server database, we use JTDS 1.2.5. For other graphs that we run that connect to an AS400 system, we use JT400, but I'm not sure what version.5. Did you use jvisualvm for your measures? Can you use it to post also some information about Classes loaded and Threads used?
I use jconsole to do my monitoring. Classes loaded is roughly in the neighborhood of 13,000, but the Classes Loaded does seem to go up over time.6. Can you post also your graph? Without any credentials and other possibly confidential things, of course.
I don't know. I'll see if I can figure out how to scrub the connection information without messing up the graph itself.
Thank you in advance for your answers.
Best regards,[/quote] -
Hello again,
First of all, I recommend you to use always the same version of Designer and Server. Many issues may be caused by different versions but this is not one of them. I have consulted your issue with our developers and they consider this behavior to be ok. This situation is probably caused by Garbage Collector which clears the memory based on some time frame and not based on reaching some memory limit. Important is that it always drops back to 500 MB.
The previous paragraph was about Heap Space memory but I think there may be some other issue with PermGen memory. If the number of Classes loaded goes up, we should find why is this happening. To do that, we need your graph as I requested before. I you do not want to post it here, you can send it to our support e-mail address support@cloveretl.com and we will consider it to be confidential, of course.
Best regards, -
I don't know that sending you this particular graph would help you diagnose a PermGen issue. We run quite a few graphs and the classes take time to go up. They don't go up quickly, so it's likely not this particular graph causing it by itself. We managed to counteract our slowness issue by pulling back the max heap space to give the operating system more room to breath.
I wasn't aware that the designer and the server kept to the same exact versioning scheme outside of the major versions (3.2,3.3,3.4, etc.). I'll keep an eye on that more in the future. Is there a way to get an e-mail or something when the server gets any sort of new version? -
No problem, you have to know better whether it is an issue for you or not.
Regarding the versions, Designer and Server in different versions may work together on some basic level but it is very difficult to troubleshoot anything this way. Cooperation of our products is tested in the same versions and even if they may work in different versions, we can not support that. We would never be sure whether some issue is in bad settings, in our code or in versions incompatibility.
And regarding the new version, I have some good news for you! CloverETL Designer and Server are going to be released in their production version 3.4.0 probably this week.
Best regards,
Please sign in to leave a comment.
Comments 5