Hello -
I am using UniversalDataWriter to write the output using 'Records per file' feature. However, the first column of the input data looks looks like the following:
Row number: 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 4, 5, 6, 6, 6, 6, 7, 7, 8, 8....... 400,000
Is there a way to tell Clover that, when breaking out the files, don't break records with same row number into different files. For example, if I choose 5 for the values for 'records per file', '4, 5, 6, 6, 6' will be in one file, then '6, 7, 7, 8, 8' will be written to a separate file. But I want all the records with row 6 in one file. In other words, I want Clover to dynamically adjust output records based on a row value here.
Is it possible?
Thanks,
I am using UniversalDataWriter to write the output using 'Records per file' feature. However, the first column of the input data looks looks like the following:
Row number: 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 4, 5, 6, 6, 6, 6, 7, 7, 8, 8....... 400,000
Is there a way to tell Clover that, when breaking out the files, don't break records with same row number into different files. For example, if I choose 5 for the values for 'records per file', '4, 5, 6, 6, 6' will be in one file, then '6, 7, 7, 8, 8' will be written to a separate file. But I want all the records with row 6 in one file. In other words, I want Clover to dynamically adjust output records based on a row value here.
Is it possible?
Thanks,
-
Hi,
In order to do this, you might take into consideration using "Partition key" attribute. Partitioning simply splits one data flow into multiple different output files based on input field value. There is an article in our documentation that might help you better understand partitioning: http://doc.cloveretl.com/documentation/ ... files.html
Hope this helps.
Please sign in to leave a comment.
Comments 1