Hi
Problem: DelimitedDataReaderNIO errors out with java.nio.BufferOverflowException. Does any one know why? Has anyone faced the same problem?
Graph: I have 2 (DelimitedDataReaderNIO ) connected to a Merge with key. The Merge is followed by a Reformat and then a writer. Note there's no sort between reader and merge (since the data is already sorted). Of the 2 (DelimitedDataReaderNIO) the first one is fed with a file with ~8K records (11 columns), the second is fed almost no records. When I run this graph I get java.nio.BufferOverfException on the first reader.
Observations:
1. This graph runs fine for smaller files.
2. The reformat transform has logger.debug statements, which when turned off, the problem goes away.
3. If I put an ext_sort between first reader and merge, the problem goes away. Since the data is already sorted I don't want to incur this cost, could it be that the data if not sorted shall cause this problem?
I think there's a problem of buffering, the reader is too fast and reformat transform too slow, and there's not enough room to buffer in between. How do these components sync speed?
Also, if I replace DelimitedDataReaderNIO with DelimitedDataReader then the graph always fails, and none of the above work arounds work.
Any help is appreciated.
Thanks in advance.
Akhil
Problem: DelimitedDataReaderNIO errors out with java.nio.BufferOverflowException. Does any one know why? Has anyone faced the same problem?
Graph: I have 2 (DelimitedDataReaderNIO ) connected to a Merge with key. The Merge is followed by a Reformat and then a writer. Note there's no sort between reader and merge (since the data is already sorted). Of the 2 (DelimitedDataReaderNIO) the first one is fed with a file with ~8K records (11 columns), the second is fed almost no records. When I run this graph I get java.nio.BufferOverfException on the first reader.
Observations:
1. This graph runs fine for smaller files.
2. The reformat transform has logger.debug statements, which when turned off, the problem goes away.
3. If I put an ext_sort between first reader and merge, the problem goes away. Since the data is already sorted I don't want to incur this cost, could it be that the data if not sorted shall cause this problem?
I think there's a problem of buffering, the reader is too fast and reformat transform too slow, and there's not enough room to buffer in between. How do these components sync speed?
Also, if I replace DelimitedDataReaderNIO with DelimitedDataReader then the graph always fails, and none of the above work arounds work.
Any help is appreciated.
Thanks in advance.
Akhil
-
Hi Agupta!
On the basis your report is hard to say where could be problem. Sad to say that you aren't first with this issue. If your data are not too privat, could you send me it, your graph and metadata and i try debug this issue.
OtaSanek (martin.zatopek@javlinconsulting.cz)
Please sign in to leave a comment.
Comments 1