Append the number of records in a partitioned file

martind

• March 23, 2010 00:00

• Answered

Follow

Hi all,

I am using a StructuredDataWriter with a limit of 1000 records per file. I would like to append the number of records contained in the partitioned file as a footer (ie, 1000 for the first files and 'n' for the last one).
Is there a simple way to obtain this result?

• 

  avackova

  • March 24, 2010 11:36

  0

  Comment actions Permalink

  Hello,
  I haven't found an easy way to do that. My solution is as follows:
  

  • in a transformation component that is before the Writer (it can be a component you have just in your graph or put a Reformat before the Writer) store the number of files you will need and the number of records in the last file:
    

    int fileNumber = 1;
    int counter = 0;
    
    // Transforms input record into output record.
    function transform() {
            .
            .
            .
    	if (counter == ${full_file}){
    		counter = 1;
    		fileNumber++;
    	}else{
    		counter++;
    	}
            .
            .
            .
    }
    
    // Called after the component finishes.
    function finished() {
    	dict_put_str('num_files',to_string(fileNumber));
    	dict_put_str('num_rec',to_string(counter));
    }
    

    where full_file parameter stores number of records for partition (1000)
  
  
  • add a new phase to your graph after the records (without any footer) are written to the files:
    DataGenerator --> StructureWriter
    DataGenerator code:

    boolean initialized = false;
    int index = 0;
    int num_files;
    
    function initInternal(){
    	num_files = str2num(dict_get_str('num_files'));
    }
    
    function setOutput(lastFile) {
    	$number := iif(lastFile, str2num(dict_get_str('num_rec')), ${full_file});
    }
    
    function generate(){
    	if (!initialized){
    		initInternal();
    		initialized = true;
    	}
    	if (index < num_files) {
    		index++;
    		setOutput(index == num_files);
    		return ALL;
    	}
    	return SKIP;
    }

    
    The output from DataGenerator contains the number of records you need to append to the files created before.
    Now configure StructerWriter:[list:153gja1z][*:153gja1z]fileURL - like in the Writer, that stores data records
  
  
  • append=true
  
  
  • recordsPerFile=1

  [/*:m:153gja1z][/list:u:153gja1z]

Please sign in to leave a comment.