I realize this may be a more complex question, with a complex answer, but I'm curious if there is a quick answer for where to at least start troubleshooting.
We are running database ETL jobs on a Linux server, running Tomcat, with 12 CPUs (1 core per socket), and our CPU% isn't budging past 0. Our heap memory has usage (~20-50%), as does our system memory, but our CPU load has never jumped past 0. Is there some configuration we need to look at? I feel like we are not utilizing our CPU like we should be.
Thanks
We are running database ETL jobs on a Linux server, running Tomcat, with 12 CPUs (1 core per socket), and our CPU% isn't budging past 0. Our heap memory has usage (~20-50%), as does our system memory, but our CPU load has never jumped past 0. Is there some configuration we need to look at? I feel like we are not utilizing our CPU like we should be.
Thanks
-
Hi Hewills,
as you rightly anticipated, the answer to your question would be rather extent in its scope. First of all, let me clarify that there is no simple direct or indirect proportion between CPU and memory usage. In other words, you cannot substitute the usage of memory by the usage of CPU as both are utilized for different purposes. Also, having low levels of CPU usage is generally a good thing and typical CPU related issues that we (as CloverCARE Support) encounter are dealing with the exact opposite problem. Nevertheless, let me list a couple of general good practice rules of how you could optimize the CPU usage by various settings in Clover:- The more connections are made to a database from CloverETL, the higher the usage of the CPU is. Worth checking is whether your DB connection definitions are set up with the ‘ThreadSafe Connection’ checkbox set to ‘false’ (navigate to CloverETL Designer > Outline pane > Connections > edit a connection > Advanced tab). Such setup would mean that a single connection instance is used by multiple components in the graph. Unless there is a specific need for such setup, it is generally recommended to set ‘ThreadSafe Connection’ to ‘true’. Generally, this would also lead to the higher CPU usage.
- Another practice that could lead to a more intense CPU utilization is placing more DBInputTable components (in one graph) instead of a single one. This scenario is useful namely when the loaded amount of data is huge. The idea is to load only some part of the table content in each of the DBInputTable components (parallel loading). When the loading is done, you can merge the results, for example, by the SimpleGather component.
- Worth reviewing is also the design of your graphs in terms of the number of threads running concurrently. In other words, if there is no need for running different parts of a graph in different phases, you can increase the CPU utilization by assigning them to the same phase so that they run at the same time.
- There is another component in CloverETL Designer that could significantly affect the usage of CPU or memory and that is FastSort. The setting of this component is often difficult to tune up and if it happens to be inadequate it will cause CPU or memory related issues. Generally, we discourage Clover users from using FastSort (if there is not a specific reason for that) and recommend using ExtSort instead.
Kind regards, -
Quick update...
I've upgraded our CloverETL 4.5.1.3 version to 4.8.1, and now the CPU monitoring is showing activity. For some reason the Clover monitor wasn't showing CPU load before, which was confusing to me, as I could see activity when monitoring the server itself. Now at least it makes sense. Thanks -
Hello Hewills,
thank you for the heads up. After a closer look at this topic, it appears that the issues you were experiencing were most likely caused by this bug in the 4.5 version of CloverETL Server UI. The fact that you no longer have this issue in the 4.8 version coincides with the fact that the bug was resolved with the 4.6 version. I'm sorry for the inconvenience.
Kind regards,
Please sign in to leave a comment.
Comments 3