File listener is used to trigger a job flow. It takes a while to process each file (typically 15 minutes in our examples). There are lots of files, arriving at random time.
In case of dozens (even hundreds) of such files arriving in a short time period, many instances of the job flow are fired up, potentially causing problems (out-of-memory etc., which actually happened in real environment).
Tried some ways to avoid the issue using external info (such as files/directories etc.).
For example, each instance moves its file(s) to own work directory (named with RUN_ID) before processing. The job flow counts such work directories first, and starts processing only if the count is under certain limit.
It does not seem to work well, as every instance fired up simultaneously does not see each other (and their work directories). The counting (first operation) in each instance seems to be done before the other operations of all instances fired up simultaneously.
Does CloverETL have any way to detect total number of running instances of a particular job flow (such as one triggered by a file listener)?
Or is there any way to limit the access of a job flow (such as singleton)?
Or is there any semaphore accessible from different instances of the same job flow?
The CloverETL used is version 4.3 (all-in-one bundle: Tomcat 8.0.30).
Thanks.
In case of dozens (even hundreds) of such files arriving in a short time period, many instances of the job flow are fired up, potentially causing problems (out-of-memory etc., which actually happened in real environment).
Tried some ways to avoid the issue using external info (such as files/directories etc.).
For example, each instance moves its file(s) to own work directory (named with RUN_ID) before processing. The job flow counts such work directories first, and starts processing only if the count is under certain limit.
It does not seem to work well, as every instance fired up simultaneously does not see each other (and their work directories). The counting (first operation) in each instance seems to be done before the other operations of all instances fired up simultaneously.
Does CloverETL have any way to detect total number of running instances of a particular job flow (such as one triggered by a file listener)?
Or is there any way to limit the access of a job flow (such as singleton)?
Or is there any semaphore accessible from different instances of the same job flow?
The CloverETL used is version 4.3 (all-in-one bundle: Tomcat 8.0.30).
Thanks.
-
Hi jzhu,
CloverETL does have a way to limit the total number of running instances. You can achieve this via Job Config Properties on the Sandbox (Server Console - Sandboxes - CloverETLProject - Config Properties). The two properties you want to set are max_running_concurrently = <the amount of Jobflow you want to run at the same time> and enqueue_executions = true
Best Regards,
Please sign in to leave a comment.
Comments 2