Hi,
I am using http connector to make a API call and receive the response file in tmp directory. Since its a big dataset, more than 200k records but I see Clover is limiting it to 100,000 records. I am unable to find the property which limits this dataset. Where can I change this property to pull full dataset.
I am using http connector to make a API call and receive the response file in tmp directory. Since its a big dataset, more than 200k records but I see Clover is limiting it to 100,000 records. I am unable to find the property which limits this dataset. Where can I change this property to pull full dataset.
-
Hi,
Off the top of my head, I am not aware of any parameter or functionality that would limit the number of records or size of response content while using especially HTTPConnector. As I might have not fully understood your use case, would it be possible to provide us with an example of your graph or more detailed explanation what you would like to achieve? (Please internalize your metadata and/or other dependencies to the graph and feel free to remove any sensitive data from your graph.)
Generally, CloverETL allows limiting the number of records in Reader components. Therefore if you process the HTTP response content in a FlatFileReader, for instance, check whether you have any value in the "Max number of records" Property in the reader. (In such a case the graph would look like as in the following forum post https://forum.cloveretl.com/viewtopic.php?t=7407).
In addition, you can control also the max size of each record and other useful parameters by changing so-called Engine properties. For more information see the following documentation page:
http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.gui.docs/docs/cloveretl-default-settings.html
Have a nice day, Eva
Please sign in to leave a comment.
Comments 1