Hi Support,
When querying our LDAP server using the LDAP_READER component, I’m getting the following error message: “LDAP: error code 4 - Sizelimit Exceeded” in the CloverETL logs. I believe that the LDAP limit is set to 50000 records but there are apparently more records to retrieve.
Any idea how I can retrieve the remaining bit of the LDAP. Or can I retrieve the list in two separate batches?
I’m running CloverETL Designer version 3.4.1.010P to create the graph in our test environment and use the CloverETL commandline configuration (clover.bat) to run the graph file it on our production server.
Regards,
JohnD
When querying our LDAP server using the LDAP_READER component, I’m getting the following error message: “LDAP: error code 4 - Sizelimit Exceeded” in the CloverETL logs. I believe that the LDAP limit is set to 50000 records but there are apparently more records to retrieve.
Any idea how I can retrieve the remaining bit of the LDAP. Or can I retrieve the list in two separate batches?
I’m running CloverETL Designer version 3.4.1.010P to create the graph in our test environment and use the CloverETL commandline configuration (clover.bat) to run the graph file it on our production server.
Regards,
JohnD
-
Hi,
Batching/paging is available in CloverETL since version 4.0 - there is an option "pageSize".
Giving your version of Clover, you might try to further increase the maximum number of results returned on LDAP server side. However it seems that you have already quite a high number. You may also try to modify your search query to return less results (do the batching via several smaller read requests). -
Hi David,
Thanks for replying. I looked at the LDAPReader component details (http://doc.cloveretl.com/documentation/ ... table.html) but I can’t seem to find the option called PageSize?
Also, upgrading the designer package to the latest Clover version seems to be straight forward but will that also be the case on my production server where I have the Clover.bat running? Do I need to upgrade any other CloverETL projects running on that server that were created with my current version of CloverETL?
Regards,
John. -
Hi John,
The documentation is unfortunately not up-to-date in this case. CloverETL Designer v 4.0 has that option for LDAPReader and you can specify the size of "batches" in which the LDAP server will communicate with the LDAPReader - provided the server supports that.
I am not sure what do you mean by Clover.bat - that is "obsolete" way of running Clover transformations, which may work with open-source Clover engine, but that has been removed from CloverETL Designer some time ago.
But anyway, upgrading to CloverETL version 4 should not break anything in your older transformations/jobs. -
Hi David,
I’ve checked with the LDAP admins and ‘page size’ is a supported by the LDAP server we’re running.
The way of describing the CloverETL version with ‘CloverETL.bat’ doesn’t help that much. I’ve checked the CloverETL directory on our production server and it contains three directories: bin, docs, libs and plugins. In the root there is a readme.txt file (dated 2012) with line “CloverETL Engine version 3.3.0”. Hopefully that can help with identifying the version/product I’m using?
Regards,
John. -
Hi John,
I just verified that the version where we introduced paging support is 4.0. So in order to get the paging support, you would need to upgrade to CloverETL version 4.
Are you using full commercial Clover package or just the open-source engine ? Upgrading to commercial CloverETL 4 should be relatively simple. With the OSS engine, you might try to "borrow" the LDAP component from version 4 and put it to your version - it is a plugin, so you may just try to replace one .jar with the newer version.
Look for plugins/org.jetel.thirdparty/cloveretl.thirdparty.jar
But be careful, CloverETL 4 requires Java 7 whereas your older version is probably running with Java 6. -
Hi David,
Many thanks for your detailed information. In meantime, I took a more detailed peek on the versions that we are using. Sorry I know it’s bit confusing but this is something I inherited from a colleague that already left the company. We using the Designer-only version to design the graphs/projects and indeed for what is seems to be the Open Source edition to run the graphs on the production server. I downloaded the OS version from SourceFource and noticed the same directory tree setup.
Am I correct with the following upgrade scenario?
- On the development server, remove the current version of the Designer
- On the development server, install the Designer 4.0.4, including the separate Java Development Kit with CloverETL (later JAVA version)
- On the production server remove/backup the ..\CloverETL directory with the Open Source version.
- On the production server install the 4.0.4 Open Source version.
Oh, one other question; do I need to get a new license key as my current one only seems to support CloverETL Designer (3.4)? The software did not accept my license key when trying to register the 4.0.4 Designer Trial version.
Regards,
John. -
Hi John,
Yes, your approach should work. Don't forget that you need Java 7 on your production server. Also don't forget, that running transformation graphs developed in commercial Designer on open source engine is not officially supported. It may work in certain scenarios and may fail in others - due to lack of certain functionality in OSS engine.
As for upgrading your Designer - your old license key won't work with the new Designer. If you are paying the maintenance fee for your designer, then just log into cloveretl.com portal, user section (using the email and password sent to you when purchased the Designer) and download the newer version with the corresponding license key. Otherwise, you may contact our sales@cloveretl.com to ask about upgrade options.
David.
Please sign in to leave a comment.
Comments 7