Hi ALL,
We are using System Execute component to copy some .csv files from the clover box (Windows box) to our application server (Linux box).
We are using the following system command to perform the copy operation,
echo y | pscp -r -pw ${password} "${rootPath}\${dataOutDirectoryPath}\PRODUCT\*.csv" ${userName}@${destinationPath}/${destinationPathDataImportFolder}/product/
This works fine and copies all the files in data-out directory to destination.
The problem is when we have some file which is bigger than 600 MB. The log shows the progress of copying of this big file till 100%. But at the end of the process we can see this file and the subsequent files (generated by subsequent graphs in the same job flow) have '0' size.
We tried minimizing the file size (by reducing the time period to generate the data) and it works fine and dumps files with proper contents and size.
Please let us know if we have to do some settings change or are there any limitations in Clover handling large volume of data
Thanks,
Sreehareesh
We are using System Execute component to copy some .csv files from the clover box (Windows box) to our application server (Linux box).
We are using the following system command to perform the copy operation,
echo y | pscp -r -pw ${password} "${rootPath}\${dataOutDirectoryPath}\PRODUCT\*.csv" ${userName}@${destinationPath}/${destinationPathDataImportFolder}/product/
This works fine and copies all the files in data-out directory to destination.
The problem is when we have some file which is bigger than 600 MB. The log shows the progress of copying of this big file till 100%. But at the end of the process we can see this file and the subsequent files (generated by subsequent graphs in the same job flow) have '0' size.
We tried minimizing the file size (by reducing the time period to generate the data) and it works fine and dumps files with proper contents and size.
Please let us know if we have to do some settings change or are there any limitations in Clover handling large volume of data
Thanks,
Sreehareesh
-
Hi Sreehareesh,
CloverETL is actually not in charge of such file operations. Script set in SystemExecute is just passed to your OS interpreter. The interpreter does all the work and then passes the possible output back to CloverETL. It seems unlikely that CloverETL somehow corrupts files it does not even read, especially if the small files work ok. It is probably matter of the interpreter itself.
The other possible explanation is that the SystemExecute component is in the wrong phase. It may start processing before the files are complete and ready to transfer.
By the way, you mentioned you are using jobflows. Can I just ask why do you prefer SystemExecute over FileOperations components developed for this exact purpose? Are they not suitable for your task? I have found that my colleague Jaroslav recommended using FileOperations component to you in this forum post viewtopic.php?f=4&t=6817#p10977
Regards,
Please sign in to leave a comment.
Comments 3