Generate something like a batch key for records

petervo85

• July 02, 2014 00:00

• Answered

Follow

Hi there,

Is there a way to do the following using transformation in CloverETL (Eg. Reformat, etc..)?

- Create a new field for each record based on their "rownum", i.e. generate a new field with something like a "batch_key" so I can tag the first 5K records to be batch_key=1, 2nd 5K records to be batch_key=2,etc.. and so on. Would like to do this in CloverETL and not in the input source(s).
- The goal here is to be able to use the batch_key field as the key to a denormalizer component to group/denormalize records into buckets/sets.

Thanks!
Peter

• 

  admin

  • July 03, 2014 06:57

  0

  Comment actions Permalink

  Hi Peter,
  
  you can do following within Reformat:
  

  
  //#CTL2
  
  //current bucket nr
  integer bucket = 0;
  
  //passed records since last bucket nr increment
  integer passed = 0;
  
  function integer transform() {
  	//test bulk size
  	if (passed>10) {
  
  		//reset counter
  		passed = 0;
  		
  		//new bucket
  		++bucket;
  	}
  	
  	//copy input record to output
  	$out.0.* = $in.0.*;
  	
  	//enrich by bucket nr
  	$out.0.bucket = bucket;
  
  	++passed;
  
  	return ALL;
  }
  

  
  
  I hope this helps.

Please sign in to leave a comment.