Max field size to small for text

twoolworth

• November 18, 2011 00:00

• Answered

Follow

I'm extracting metadata from my MySQL tables and having issues with columns that are set to a type of TEXT. In MySQL that max size a TEXT column can be is 65535 which comes back properly in the metadata. However CloverETL will not let me create the metadata as it says that 65535 is not a number. I figured it was probably the size of the number that it isn't recognizing rather then say the type which appears to be correct.

The largest value a Java Primitive type of short can be is 32,767 which if I then fill that value in, the error goes away; switching it to 32,768 makes the warning come back. However this really isn't a solution I can except as this table has over 160+ fields in them and at least 60+ of them are set to a type of TEXT. Keep in mind this is only one table and there will be more to come...

Does anyone have any ideas on how to get around this limitation?

• 

  julrych

  • November 18, 2011 20:27

  0

  Comment actions Permalink

  After the extraction of the metadata from database, you can clear size attribute value for the string and byte data types. That will cause CloverETL to allocate the field dynamically to accommodate data. The size property is used for data input (from flat file) or output formatting (to a flat file) only anyway, so no harm leaving it empty. I have raised this issue with CloverETL development team as bug CLD-3022.
  
  When processing records large than 64 kB, you may also run into runtime error saying the the buffer has not enough size to accommodate the record:
  

  
  Node INPUT1 finished with status: ERROR caused by: The size of data buffer is only 12288. Set appropriate parameter in defautProperties file.
  

  
  You can increase the internal buffer sizes by overriding CloverETL Engine default properties as described at Changing Default CloverETL Settings. In fact, CloverETL 3.2, which is going to be released in December 2011, will use dynamically allocated internal buffers to work with large records transparently and optimize memory usage.
• 

  twoolworth

  • November 18, 2011 21:47

  0

  Comment actions Permalink

  Thank you, I hadn't realized I could actually just clear the field out and it wouldn't apply a size limit to the field. This worked just fine, especially since I can select all the fields at once and clear them in one shot within CloverETL.

Please sign in to leave a comment.