Hi,
I want to count how many files are on HDFS.
If there is no file, send an email notifying that the file was not received. Execute a job flow if there is a file.
If HDFS have files, I can count the number of files found by the ListFile component.
However, if there are no files on HDFS, the components behind the ListFile component do not seem to execute.
The code for the Denormalizer component is as follows.
Thanks,
Davis Wang
I want to count how many files are on HDFS.
If there is no file, send an email notifying that the file was not received. Execute a job flow if there is a file.
If HDFS have files, I can count the number of files found by the ListFile component.
However, if there are no files on HDFS, the components behind the ListFile component do not seem to execute.
The code for the Denormalizer component is as follows.
//#CTL2
integer fileCount;
// This transformation defines the way in which multiple input records
// (with the same key) are denormalized into one output record.
// This function is called for each input record from a group of records
// with the same key.
function integer append() {
if($in.0.URL != "" && $in.0.isFile && !$in.0.isDirectory && $in.0.size > 0){
fileCount += 1;
}
return OK;
}
// This function is called once after the append() function was called for all records
// of a group of input records defined by the key.
// It creates a single output record for the whole group.
function integer transform() {
$out.0.count = fileCount;
return OK;
}
// Called during component initialization.
// function boolean init() {}
function boolean init() {
fileCount = 0;
return true;
}
Thanks,
Davis Wang
Comments (2)