Heya,
The Clover wiki for the Concatenate node states that:
I have a need to concatenate a bunch of files together, but I want to make sure the rows from each file stay together. For example, if File1 is hooked to CONCAT:0 and File 2 is hooked to CONCAT:1 of a Concatentate node, will the rows always come out in the order:
All File 1 rows
All File 2 rows
That "waiting/blocked" statement makes me wonder if, for example, CONCAT:0 is blocked because it wating for a Sort node to complete, will it pull rows from CONCAT:1 if they are available? Or will it wait for CONCAT:0, process those rows until EOF, then process CONCAT:1?
If it does process CONCAT:1 first in the example above, the "When reading from one port is done (EOF status)" statement has me wondering what happens if CONCAT:1 becomes blocked (say, waiting on some sort of I/O, like a lookup). Does the Concatenate node keep waiting on CONCAT:1 or will it go check CONCAT:0 to see if it is unblocked?
I know I can put a Sort Node after the concatenate and use some sort of field to get it back in File1 row, File2 row order, but that can be expensive in terms of performance. So I thought I'd ask to see if it is neccessary...
Thanks,
Anna
The Clover wiki for the Concatenate node states that:
All records from all input ports are copied onto output port. It goes port by port (waiting/blocked) if there is currently no data on port. When reading from one port is done (EOF status), continues with the next."
I have a need to concatenate a bunch of files together, but I want to make sure the rows from each file stay together. For example, if File1 is hooked to CONCAT:0 and File 2 is hooked to CONCAT:1 of a Concatentate node, will the rows always come out in the order:
All File 1 rows
All File 2 rows
That "waiting/blocked" statement makes me wonder if, for example, CONCAT:0 is blocked because it wating for a Sort node to complete, will it pull rows from CONCAT:1 if they are available? Or will it wait for CONCAT:0, process those rows until EOF, then process CONCAT:1?
If it does process CONCAT:1 first in the example above, the "When reading from one port is done (EOF status)" statement has me wondering what happens if CONCAT:1 becomes blocked (say, waiting on some sort of I/O, like a lookup). Does the Concatenate node keep waiting on CONCAT:1 or will it go check CONCAT:0 to see if it is unblocked?
I know I can put a Sort Node after the concatenate and use some sort of field to get it back in File1 row, File2 row order, but that can be expensive in terms of performance. So I thought I'd ask to see if it is neccessary...
Thanks,
Anna
-
Dear Ann,
As regards the Concatenate component, it starts reading the records from any port only after all records from all previous ports (those with lower numbers) have been read successfully, without any error.
Thus, this component does not start reading any record through the port 1 if there still remains at least one record that should have been read through the port 0 (and which is still to come and have not been read).
The order on the output is this:
All File 1 rows
All File 2 rows
All File 3 rows
...
All File N-1 rows
All File N rows
Best regards, -
You can also consult Users Guide:
http://www.cloveretl.com/documentation/UserGuide/topic/com.cloveretl.gui.docs/docs/concatenate.html
Regards,
Please sign in to leave a comment.
Comments 2