strange issue with DBInputTable

mhgrove

• January 15, 2010 00:00

• Answered

Follow

I'm trying to convert some data from a database and I'm running into an odd problem. There are about 111k results I expect from the query I'm passing to the DBInputTable node, however, only 49 records were ever getting processed. After a bit of fiddling, I realized it was only pulling down the number of results as specified in the fetch size (minus one); if I set the fetch size to 1000, I was getting 999 records, if the fetch size is 123, I'd get 122 records.

Is this the expected behavior? If so, is there anything I can change so that it will iterate over the entire result set?

Running the query against the database outside of clover yields the number of results I expect, so it's not a problem with my query.

I'm using a SQLite db, with the driver from Zentus, I don't know if that makes a difference. I've done one other conversion with data from a database in the past, but that was with an Oracle db, and an older version of clover (2.6.3 I think). This is on Mac OSX 10.6.2 w/ Java 1.6.0_15 (64 bit VM)

• 

  avackova

  • January 19, 2010 10:03

  0

  Comment actions Permalink

  Hello mhgrove,
  this is really bug in CloverETL and will be fixed in 2.8.2 version, which is to release in few days. To get to work it properly (in new version) you have to use SQLITE jdbc specific in your connection configuration.
• 

  mhgrove

  • January 19, 2010 12:56

  0

  Comment actions Permalink

  How do I specify the SQLite jdbc driver in the connection configuration?
• 

  mhgrove

  • January 19, 2010 13:25

  0

  Comment actions Permalink

  Hello mhgrove,
  this is really bug in CloverETL and will be fixed in 2.8.2 version, which is to release in few days. To get to work it properly (in new version) you have to use SQLITE jdbc specific in your connection configuration.

  
  
  How do I set this?
• 

  avackova

  • January 19, 2010 13:45

  0

  Comment actions Permalink

  This is attribute of jdbc connection:

  jdbcSpecific="SQLITE"

  In CloverDesigner you can choose proper jdbc specific from combo box (see Creating Internal Database Connections)
• 

  mhgrove

  • January 19, 2010 13:47

  0

  Comment actions Permalink

  actually, i eventually found this in the source code, the attribute "jdbcSpecific" seems to control this, and "SQLITE" should be the value according to the plugin.xml file.
  
  However, when I include this parameter, I get this error message:
  
  

  JDBC specific 'SQLite' does not exist.).

  
  
  So I guess that means it is not properly loading the plugin.xml file? How do I control how those get loaded? Do they need to be in the classpath? Do I point the transformation graph at these somehow?
  
  Thanks.
• 

  mhgrove

  • January 19, 2010 15:26

  0

  Comment actions Permalink

  I found the code to get the plugin.xml files to get loaded
  
  

  	
  EngineInitializer.initEngine("lib/clover/plugins", null, null);
  EngineInitializer.forceActivateAllPlugins();
  

  
  
  So this got rid of the error that Clover did not know what the "SQLITE" value for jdbcSpecific was.
  
  Now things are back to running, but I'm still getting the original behavior of only the fetchSize is being converted, it's not actually iterating through the rest of the result set.
  
  Either I set up the SQLite jdbcSpecific stuff wrong and it's not getting used, or that's not a fix/workaround for the bug.
  
  Is there anyway to tell if the SQLlite jdbcSpecific bits are being used?
• 

  avackova

  • January 20, 2010 12:42

  0

  Comment actions Permalink

  Hello,
  the bug with SQLlite will be fixed in 2.8.2 version, which is releasing today or tomorrow. With older versions you have to set fetch size grater than the number of records in the table or create your own plugin with fixed version of SQLiteSpecific.

Please sign in to leave a comment.