Split large data set when extracting data from source

marcosnels

• February 01, 2016 00:00

• Answered

Follow

Hi,

We have a folder with a lot of files. We cannot run a graph and extract all the files at once, because our temp folder is not that big. Is it in the UniversalDataReader possible to split the incoming data in sets? So that the temp folder is deleted after every set.

Thnx.

Marco

• 

  slechtaj

  • February 03, 2016 19:00

  0

  Comment actions Permalink

  Hi Marco,
  
  You should use Jobflows (comes with CloverETL Corporate Server) and File operations components (ListFiles) for this.
  ListFiles component can produce the list of files available for data transformation and then using ExecuteGraph, you can execute the graph for every single of the files from the list - this way you will get always one graph processed at a time.

Please sign in to leave a comment.