Number of Records

oldforum

• July 16, 2007 00:00

• Answered

Follow

Hello ,

Is there a way to get the number of records in the Reformat Class. I am trying to do some custom transformations and I need to know the total number of records ahead of time before the processing is complete.

Please let me know.

• 

  oldforum

  • July 16, 2007 08:23

  0

  Comment actions Permalink

  Hi !
  
  The question is, what do you mean by "ahead of time". There is currently no way to get information how many records are in source data (being it text file, DB table or any other data source).
  
  The reason is that Clover works on flow of data without imposing restriction that it needs to know how many records it is about to process. You can determine how many records have been processed so far, but it is not easy to predict how many records remain in the source.
  
  I admit that it could be sometimes useful to know ahead of time some approximation - for example when building look-up tables, but relying on exact number would require deep changes in the concept.
  
  Problem is that for some data source it can be really difficult to know the number of records in it. Even in DB - if SQL (JDBC) is used, you don't know the number of records particular query returns till you read the last record. So it would require first go through all the records just to count them and then start again and process them.
  
  What you can do in Reformat transformation is somehow connect to your source data (JDBC or text file) and simply estimate the number of record. Or you can count them precisely using SQL statement (in case of JDBC) or use Clover's data parser and just keep calling get() method till you reach the end, counting the records as you go.
  
  Sorry, but there is currently no better way.
  
  Sometimes you problem can be solved by using different approach. If you could describe why do you need to know the record count, then I could think of other approach.
  
  David.

Please sign in to leave a comment.