Hello David,
I changed the Value to 4096 and works Great!!. If you do not mind can you please post the various restrictions on record lengths and the column length values.
In fact this fix solved my delimited file issue. It was manifesting as an error because of the column lengths.
Thanks
Subbu
I changed the Value to 4096 and works Great!!. If you do not mind can you please post the various restrictions on record lengths and the column length values.
In fact this fix solved my delimited file issue. It was manifesting as an error because of the column lengths.
Thanks
Subbu
-
Hello !
Here are some CloverETL related constants which limit the maximum
sizes of different data objects.
All can be found in org.jetel.data.Defaults class, where are several inner
classes (just for distinguishing what is what)
MAX_RECORD_SIZE = 8192
This limits the maximum size of data record in binary form.
The binary for is the form used by Clover when manipulating data.
Parser are here to convert text representation or database representation
of data records to Clover.
Some data can have larger representation in text form - dates, numbers and
some shorter - strings (java stores strings in unicode - 16bits per character)
If you start getting buffer overflow or similar errors, increase this value.
The limit is theoretically 2^31 , but you should keep it under 64K.
Clover at runtime allocates several internal buffers of MAX_RECORD_SIZE, thus
increasing this value increases also memory utilization.
FIELD_BUFFER_LENGTH = 512
This is constant for textual data parser (and also formatters). It determines
what is the maximum size of one particular data field in text format. If you
have data containing long text strings, increase this value. Could be 1024,2048 or
even higher.
The impact on memory utilization is low as each parser your graph uses allocates only
one such buffer.
DEFAULT_INTERNAL_IO_BUFFER_SIZE = 32768
This constant determines the internal buffer clover components allocate for I/O
operations. Again, increasing this value does not have big impact on overall memory
utilization as only few such buffers are used at runtime.
There is no sence in increasing this value to speed up something. It has been tested that
the performance improvement is negligible. However, if you increase the size of
MAX_RECORD_SIZE , make sure this value is minimally 2*MAX_RECORD_SIZE.
Size limits for data types:
----------------------------
String - maximum 32768 characters - 64k bytes. Can NOT be easily changed, but it is doable.
Integer - 4 bytes
Date - 8 bytes
Number - 8 bytes
binary - maximum 32768 - similar to String
Please sign in to leave a comment.
Comments 1