Hello there!
I am new to CloverETL. I have been playing around with a simple graph to execute an Oracle SQL script using DBInputTable and output the extract into an excel file. Considering that my sql script is a long running query which uses a UNION and is fetching data from various high volume tables, I was thinking if the following is possible.
1. Use two DBInputTable to split the main query into two and run it separately in the same graph.
2. Combine the output from both DBInputTable before extracting into a single excel file.
Main concern here is to improve the performance and reduce the time of execution. Any tip on how to achieve this would be highly appreciated.
Regards.
I am new to CloverETL. I have been playing around with a simple graph to execute an Oracle SQL script using DBInputTable and output the extract into an excel file. Considering that my sql script is a long running query which uses a UNION and is fetching data from various high volume tables, I was thinking if the following is possible.
1. Use two DBInputTable to split the main query into two and run it separately in the same graph.
2. Combine the output from both DBInputTable before extracting into a single excel file.
Main concern here is to improve the performance and reduce the time of execution. Any tip on how to achieve this would be highly appreciated.
Regards.
-
Hello iamgillu - 1. Of course you can use two DBInputTables but you have to edit the SQL query of each input table, so each of them will fetch one half of the results. You don't want them to fetch exactly same data, as it would be redundant. However the slowdown could be caused by the speed of the internet connection (if the database runs on a remote machine) and in this case, this solution would not make any difference.
2. In this case you should be able to add them together by SimpleGather component (since you have the same metadata in both data streams).
In order to speed up the execution you can try to set different values in Fetch Size property of the DBInputTable. There is not any particular value I can advise as this attribute has individual impact on different systems.
Please sign in to leave a comment.
Comments 1