I'm trying to set up a connection to an SFTP server with a private key, but I can't figure out how to set up the connection. I know what I want to do should be possible per the CloverETL documentation, but there are no instructions in the documentation on what that URL should look like. We're currently on CloverETL 3.5.0.
-
Hi DTaylor,
Detailed information about supported File URL formats, please refer to http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.gui.docs/docs/examples-of-file-url-in-readers.html.
Information about SFTP certificate-based authentication is described in the following paragraph:If certificate-based authentication is used, certificates are placed in ${PROJECT}/ssh-keys/ directory and each private key file name has suffix .key. Only certificates without password are currently supported.
Since the passphrases are not supported yet, you might use third party software (PSFTP, PSCP) that would do the job for you instead. You can call these statements directly from SystemExecute component. For more information about SystemExecute, refer to http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.gui.docs/docs/sysexecute.html. -
Hi DTaylor,
Detailed information about supported File URL formats, please refer to http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.gui.docs/docs/examples-of-file-url-in-readers.html.
Information about SFTP certificate-based authentication is described in the following paragraph:If certificate-based authentication is used, certificates are placed in ${PROJECT}/ssh-keys/ directory and each private key file name has suffix .key. Only certificates without password are currently supported.
Since the passphrases are not supported yet, you might use third party software (PSFTP, PSCP) that would do the job for you instead. You can call these statements directly from SystemExecute component. For more information about SystemExecute, refer to http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.gui.docs/docs/sysexecute.html.
The URL documentation only tells me that connection with a certificate is possible. It doesn't tell me what the URL should look like when I'm connecting with a certificate. I have the key placed in the appropriate directory with the .key suffix, but I have so far been unable to put together a URL that uses it. -
Okay. I'm trying to use a key generated by PuTTY that is in the following format: PuTTY-User-Key-File-2: ssh-rsa
Encryption: none
Comment: <comment>
Public-Lines: <#>
<Public Lines here>
Private-Lines: <#>
<Private Lines here>
Private-MAC: <MAC here>
I've never worked with certificate auth on SFTP before and this was the connection key I was given. It works in FileZilla, but when I try to use it for Clover, I get a java.lang.NullPointerException. I've tried modifying this format in various ways, but I always get the same response. -
Hi DTaylor,
You should see stack trace in "Error log" window - you can open it via "Window-> Show view -> Other -> General -> Error log". Alternatively try to run graph - stack trace should be visible in run log.
Also I don't get "I get this error when trying to validate the connection during connection setup" - could you please provide us with screeshot of where this exception happened?
Thank you. -
Here's the image of the error. I'm still working on trying to get a stack trace. When I attempt to run the reader with the connection, all I get is the URL isn't valid. -
Hi DTaylor,
thank you for the screenshot. Jaroslav has mentioned the way you can get the error log. I have also attached the screenshot of the error log for you reference. On the screenshot, you can also see the icon that you can click to open the log.
Thanks. -
Hi,
Just a short note - aforementioned stack-trace is printed in the log (which is available from the Error Log by selecting the error and clicking the icon that you can see on the screenshot in my previous response). So it would be helpful if you sent us the content of the log.
Thank you, -
I displayed the error log and it is *not* in the error log. There is no entry in the error log for that error. I am now, however, getting the error in my production environment trying to make the connection. Here's the stack trace: 2014-04-21 09:05:20,732 ERROR 9434717 [WatchDog_9434717] Error details:
org.jetel.exception.JetelRuntimeException: Component [SpreadsheetDataWriter:SPREADSHEET_WRITER0] finished with status ERROR. (In0: 0 recs)
at org.jetel.graph.Node.createNodeException(Node.java:566)
at org.jetel.graph.Node.run(Node.java:542)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: Element [SPREADSHEET_WRITER0:SpreadsheetDataWriter]-Component pre-execute initialization failed.
at org.jetel.graph.Node.run(Node.java:497)
... 3 more
Caused by: java.io.IOException: java.lang.NullPointerException
at org.jetel.util.MultiFileWriter.prepareTargets(MultiFileWriter.java:225)
at org.jetel.util.MultiFileWriter.init(MultiFileWriter.java:159)
at com.opensys.cloveretl.component.SpreadsheetWriter.preExecute(Unknown Source)
at org.jetel.graph.Node.run(Node.java:495)
... 3 more
Caused by: java.io.IOException: java.lang.NullPointerException
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:257)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:206)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.connect(PooledSFTPConnection.java:212)
at org.jetel.component.fileoperation.pool.SFTPConnectionFactory.makeObject(SFTPConnectionFactory.java:30)
at org.jetel.component.fileoperation.pool.SFTPConnectionFactory.makeObject(SFTPConnectionFactory.java:21)
at org.jetel.component.fileoperation.pool.DefaultConnectionFactory.makeObject(DefaultConnectionFactory.java:38)
at org.jetel.component.fileoperation.pool.DefaultConnectionFactory.makeObject(DefaultConnectionFactory.java:26)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.jetel.component.fileoperation.pool.ConnectionPool.borrowObject(ConnectionPool.java:91)
at org.jetel.util.protocols.sftp.SFTPConnection.connect(SFTPConnection.java:107)
at org.jetel.util.protocols.sftp.SFTPConnection.getOutputStream(SFTPConnection.java:172)
at org.jetel.util.file.FileUtils.getOutputStream(FileUtils.java:1464)
at org.jetel.util.TargetFile.setOutput(TargetFile.java:622)
at org.jetel.util.TargetFile.setNextOutput(TargetFile.java:369)
at org.jetel.util.TargetFile.initOutput(TargetFile.java:546)
at org.jetel.util.TargetFile.init(TargetFile.java:170)
at org.jetel.util.MultiFileWriter.prepareTargets(MultiFileWriter.java:222)
... 6 more
Caused by: java.lang.NullPointerException
at com.jcraft.jsch.Buffer.putString(Buffer.java:59)
at com.jcraft.jsch.UserAuthKeyboardInteractive.start(UserAuthKeyboardInteractive.java:183)
at com.jcraft.jsch.Session.connect(Session.java:463)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:242)
... 22 more -
My apologies, I missed your post about the additional error log icon. Here's the stack trace from trying to use the URL Dialog to access the SFTP site: java.lang.reflect.InvocationTargetException: Initializing failed (java.lang.NullPointerException)
at com.cloveretl.gui.dialog.url.resource.model.InitializeBrowserOperation.run(InitializeBrowserOperation.java:103)
at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:121)
Caused by: java.io.IOException: java.lang.NullPointerException
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:257)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:206)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.connect(PooledSFTPConnection.java:212)
at org.jetel.component.fileoperation.pool.SFTPConnectionFactory.makeObject(SFTPConnectionFactory.java:30)
at org.jetel.component.fileoperation.pool.SFTPConnectionFactory.makeObject(SFTPConnectionFactory.java:21)
at org.jetel.component.fileoperation.pool.DefaultConnectionFactory.makeObject(DefaultConnectionFactory.java:38)
at org.jetel.component.fileoperation.pool.DefaultConnectionFactory.makeObject(DefaultConnectionFactory.java:26)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.jetel.component.fileoperation.pool.ConnectionPool.borrowObject(ConnectionPool.java:91)
at org.jetel.util.protocols.sftp.SFTPConnection.connect(SFTPConnection.java:107)
at org.jetel.util.protocols.sftp.SFTPConnection.stat(SFTPConnection.java:235)
at com.cloveretl.gui.dialog.url.resource.model.URLSFTPModel.validateConnection(URLSFTPModel.java:311)
at com.cloveretl.gui.dialog.url.resource.handler.a.createResourceModel(RemoteResourceHandler.java:68)
at com.cloveretl.gui.dialog.url.resource.model.VirtualBrowser.setInitialData(VirtualBrowser.java:108)
at com.cloveretl.gui.dialog.url.resource.model.InitializeBrowserOperation$1.run(InitializeBrowserOperation.java:69)
Caused by: java.lang.NullPointerException
at com.jcraft.jsch.Buffer.putString(Buffer.java:59)
at com.jcraft.jsch.UserAuthKeyboardInteractive.start(UserAuthKeyboardInteractive.java:183)
at com.jcraft.jsch.Session.connect(Session.java:463)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:242)
... 14 more
Root exception:
java.io.IOException: java.lang.NullPointerException
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:257)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:206)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.connect(PooledSFTPConnection.java:212)
at org.jetel.component.fileoperation.pool.SFTPConnectionFactory.makeObject(SFTPConnectionFactory.java:30)
at org.jetel.component.fileoperation.pool.SFTPConnectionFactory.makeObject(SFTPConnectionFactory.java:21)
at org.jetel.component.fileoperation.pool.DefaultConnectionFactory.makeObject(DefaultConnectionFactory.java:38)
at org.jetel.component.fileoperation.pool.DefaultConnectionFactory.makeObject(DefaultConnectionFactory.java:26)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.jetel.component.fileoperation.pool.ConnectionPool.borrowObject(ConnectionPool.java:91)
at org.jetel.util.protocols.sftp.SFTPConnection.connect(SFTPConnection.java:107)
at org.jetel.util.protocols.sftp.SFTPConnection.stat(SFTPConnection.java:235)
at com.cloveretl.gui.dialog.url.resource.model.URLSFTPModel.validateConnection(URLSFTPModel.java:311)
at com.cloveretl.gui.dialog.url.resource.handler.a.createResourceModel(RemoteResourceHandler.java:68)
at com.cloveretl.gui.dialog.url.resource.model.VirtualBrowser.setInitialData(VirtualBrowser.java:108)
at com.cloveretl.gui.dialog.url.resource.model.InitializeBrowserOperation$1.run(InitializeBrowserOperation.java:69)
Caused by: java.lang.NullPointerException
at com.jcraft.jsch.Buffer.putString(Buffer.java:59)
at com.jcraft.jsch.UserAuthKeyboardInteractive.start(UserAuthKeyboardInteractive.java:183)
at com.jcraft.jsch.Session.connect(Session.java:463)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:242)
... 14 more -
Hi DTaylor,
thank you for the information. I have created a new ticket in our system for this issue.
You may keep yourself updated about the latest status here: http://bug.javlin.eu/browse/CLO-3577 -
Thanks. The first stack trace though is from running the graph on the server in my production environment. I can run the graph (manually entering the URL in the attribute) on my local machine in the designer without an error, but when I try to run it on the server, I get the first stack trace I provided. -
Hi DTaylor,
I have tried to reproduce the issue on the Server but so far I have not been successful. Can you please share with us your full log with all debug information, and describe how you trigger the graph when it fails?
In order to print debug information to the log, set Log level attribute to DEBUG (under Run -> Run Configurations…).
In the log, you will find what keys have been found. The row will look like this:SFTP connecting to <HOST> using the following private keys:
Thanks. -
To execute this, I connect to the server project from the designer, open the graph, and run the graph. I receive the same error when I run the graph via the server's Scheduling as well.
I had to remove some of the SFTP information from the log like the URL, but I put obvious placeholders in their place. I also had to cut down the log so it would fit in the character allowance. I tried to post it as a text file, but it wouldn't let me. So here it is:2014-04-24 09:07:02,347 DEBUG 9475431 [http-80-2] resources for Java compiler:<Java compiler info removed>
2014-04-24 09:07:02,347 DEBUG 9475431 [http-80-2] Executor configuration; using job config properties:{verbose_mode=false, enqueue_executions=false, max_running_concurrently=1, log_level=WARN}
2014-04-24 09:07:02,347 DEBUG 9475431 [http-80-2] Executor configuration; using job passed properties:{}
2014-04-24 09:07:02,347 DEBUG 9475431 [http-80-2] Temp dir E:\sgi-dev\tomcat_tmp\clover_temp\9475431\clover-dict created for execution #9475431
2014-04-24 09:07:02,347 DEBUG 9475431 [http-80-2] Temp dir E:\sgi-dev\tomcat_tmp\clover_temp\9475431\clover-debug created for execution #9475431
2014-04-24 09:07:02,347 DEBUG 9475431 [http-80-2] resources for Java compiler:<Java compiler info removed>
2014-04-24 09:07:02,363 INFO 9475431 [http-80-2] Initializing job id:1359392341736 hashCode:1131390437
2014-04-24 09:07:02,363 INFO 9475431 [http-80-2] Checking graph configuration...
2014-04-24 09:07:02,363 DEBUG 9475431 [http-80-2] Component 'DENORMALIZER0' runs in INTERPRETED mode
2014-04-24 09:07:02,363 DEBUG 9475431 [http-80-2] Component 'EXT_MERGE_JOIN0' runs in INTERPRETED mode
2014-04-24 09:07:02,379 WARN 9475431 [http-80-2] Issue in component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET]
Input edge metadata field "FirstShipDate" (date) format "yyyy-MM-dd HH:mm:ss" does not have "excel:" prefix, it will be ignored
2014-04-24 09:07:02,379 WARN 9475431 [http-80-2] Issue in component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET]
Input edge metadata field "TheDate" (date) format "yyyy-MM-dd HH:mm:ss" does not have "excel:" prefix, it will be ignored
2014-04-24 09:07:02,379 WARN 9475431 [http-80-2] Issue in component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET1]
Input edge metadata field "FirstShipDate" (date) format "yyyy-MM-dd HH:mm:ss" does not have "excel:" prefix, it will be ignored
2014-04-24 09:07:02,379 WARN 9475431 [http-80-2] Issue in component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET1]
Input edge metadata field "TheDate" (date) format "yyyy-MM-dd HH:mm:ss" does not have "excel:" prefix, it will be ignored
2014-04-24 09:07:02,379 INFO 9475431 [http-80-2] Graph configuration is valid.
2014-04-24 09:07:02,379 INFO 9475431 [http-80-2] Graph initialization (<Graph Name>)
2014-04-24 09:07:02,379 INFO 9475431 [http-80-2] Initializing connection:
2014-04-24 09:07:02,379 INFO 9475431 [http-80-2] DBConnection driver[null]:jndi[java:comp/env/jdbc/AS400]:url[null]:user[null] ... OK
2014-04-24 09:07:02,379 INFO 9475431 [http-80-2] Initializing connection:
2014-04-24 09:07:02,379 INFO 9475431 [http-80-2] DBConnection driver[null]:jndi[java:comp/env/jdbc/ddv4Sql]:url[]:user[] ... OK
2014-04-24 09:07:02,379 INFO 9475431 [http-80-2] Initializing phase 0
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] Initializing edges
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] All edges initialized successfully.
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] Initializing nodes
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] Component 'DENORMALIZER0' runs in INTERPRETED mode
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] DENORMALIZER0 ...OK
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] Component 'EXT_MERGE_JOIN0' runs in INTERPRETED mode
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] EXT_MERGE_JOIN0 ...OK
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] EXT_SORT0 ...OK
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] EXT_SORT1 ...OK
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] EXT_SORT2 ...OK
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] READ_SOURCE_DATA ...OK
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] READ_TACTIC_INFO ...OK
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] SIMPLE_COPY ...OK
2014-04-24 09:07:02,379 DEBUG 9475431 [http-80-2] WRITE_UTILIZATION_SPREADSHEET ...OK
2014-04-24 09:07:02,394 DEBUG 9475431 [http-80-2] WRITE_UTILIZATION_SPREADSHEET1 ...OK
2014-04-24 09:07:02,394 INFO 9475431 [http-80-2] Phase 0 initialized successfully.
2014-04-24 09:07:02,394 INFO 9475431 [http-80-2] Initializing watchdog
2014-04-24 09:07:02,394 DEBUG 9475431 [http-80-2] register MBean with name:org.jetel.graph.runtime:type=CLOVERJMX_1359392341736_9475431
2014-04-24 09:07:02,394 DEBUG 9475431 [http-80-2] updating RunRecord 9475431
2014-04-24 09:07:02,394 INFO 9475431 [WatchDog_9475431] Job parameters: <parameters removed>
<Graph context listing removed>
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EdgeType [Edge3] : DIRECT
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EdgeType [Edge4] : DIRECT
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EdgeType [Edge2] : DIRECT
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EdgeType [Edge5] : DIRECT
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EdgeType [Edge6] : DIRECT
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EdgeType [Edge0] : DIRECT
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EdgeType [Edge1] : DIRECT
2014-04-24 09:07:02,394 DEBUG 9475431 [http-80-2] updated RunRecord 9475431
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EdgeType [Edge7] : DIRECT
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EdgeType [Edge8] : DIRECT
2014-04-24 09:07:02,394 INFO 9475431 [WatchDog_9475431] Pre-execute initialization of connection:
2014-04-24 09:07:02,394 INFO 9475431 [WatchDog_9475431] DBConnection driver[null]:jndi[java:comp/env/jdbc/AS400]:url[null]:user[null] ... OK
2014-04-24 09:07:02,394 INFO 9475431 [WatchDog_9475431] Pre-execute initialization of connection:
2014-04-24 09:07:02,394 INFO 9475431 [WatchDog_9475431] DBConnection driver[null]:jndi[java:comp/env/jdbc/ddv4Sql]:url[]:user[] ... OK
2014-04-24 09:07:02,394 INFO 9475431 [WatchDog_9475431] Starting up all nodes in phase [0]
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] DENORMALIZER0 ... starting
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EXT_MERGE_JOIN0 ... starting
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EXT_SORT0 ... starting
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EXT_SORT1 ... starting
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] EXT_SORT2 ... starting
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] READ_SOURCE_DATA ... starting
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] READ_TACTIC_INFO ... starting
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] SIMPLE_COPY ... starting
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] WRITE_UTILIZATION_SPREADSHEET ... starting
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] WRITE_UTILIZATION_SPREADSHEET1 ... starting
2014-04-24 09:07:02,394 DEBUG 9475431 [WatchDog_9475431] All components are ready to start.
2014-04-24 09:07:02,394 INFO 9475431 [WatchDog_9475431] Successfully started all nodes in phase!
2014-04-24 09:07:02,410 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET1_9475431] java.lang.IllegalArgumentException
2014-04-24 09:07:02,410 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] SFTP connecting to <removed SFTP address> using the following private keys: [<removed key file name/path>]
2014-04-24 09:07:02,410 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] Adding new identity from <removed key file name/path>
2014-04-24 09:07:02,410 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] Connecting to <removed SFTP address> with password authentication
2014-04-24 09:07:02,410 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET1_9475431] SFTP connecting to sftp2.mysgi.com using the following private keys: [<removed key file name/path>]
2014-04-24 09:07:02,410 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET1_9475431] Adding new identity from <removed key file name/path>
2014-04-24 09:07:02,410 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET1_9475431] Connecting to sftp2.mysgi.com with password authentication
2014-04-24 09:07:02,503 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET1_9475431] IOException occured for URL - host: 'sftp2.mysgi.com', path: '/REPORTS/Qlik Reports/Test/Utilization_Source_04-21-2014.xlsx' (user info not shown)
2014-04-24 09:07:03,113 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] Connection pool status: 1 idle, 0 active
2014-04-24 09:07:03,113 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] Connecting to <removed SFTP address> with keyboard-interactive authentication
2014-04-24 09:07:03,972 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] Connection pool status: 1 idle, 0 active
2014-04-24 09:07:03,972 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] SFTP connecting to <removed SFTP address> using the following private keys: [<removed key file name/path>]
2014-04-24 09:07:03,972 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] Adding new identity from <removed key file name/path>
2014-04-24 09:07:03,972 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] Connecting to <removed SFTP address> with password authentication
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] ---------------------** Start of tracking Log for phase [0] **-------------------
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Time: 24/04/14 09:07:04
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Node ID Port #Records #KB Rec/s KB/s
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] ---------------------------------------------------------------------------------
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Read Tactic Info READ_TACTIC_INFO RUNNING
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] %cpu:0 Out:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] ExtSort EXT_SORT0 RUNNING
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] %cpu:0 In:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Out:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Denormalizer DENORMALIZER0 RUNNING
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] %cpu:0 In:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Out:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Read Source Data READ_SOURCE_DATA RUNNING
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] %cpu:0 Out:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] ExtSort EXT_SORT1 RUNNING
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] %cpu:0 In:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Out:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] ExtMergeJoin EXT_MERGE_JOIN0 RUNNING
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] %cpu:0 In:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] In:1 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Out:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] ExtSort EXT_SORT2 RUNNING
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] %cpu:0 In:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Out:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] SimpleCopy SIMPLE_COPY RUNNING
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] %cpu:0 In:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Out:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Out:1 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Write Utilization SpreaWRITE_UTILIZATION_SPREADSHEET RUNNING
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] %cpu:0 In:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] Write Utilization SpreaWRITE_UTILIZATION_SPREADSHEET1 RUNNING
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] %cpu:0 In:0 0 0 0 0
2014-04-24 09:07:04,425 INFO 9475431 [WatchDog_9475431] ---------------------------------** End of Log **--------------------------------
2014-04-24 09:07:04,659 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] Connection pool status: 1 idle, 0 active
2014-04-24 09:07:04,659 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] Connecting to <removed SFTP address> with keyboard-interactive authentication
2014-04-24 09:07:05,487 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] Connection pool status: 1 idle, 0 active
2014-04-24 09:07:05,487 DEBUG 9475431 [WRITE_UTILIZATION_SPREADSHEET_9475431] IOException occured for URL - host: '<removed SFTP address>', path: '<removed file path>' (user info not shown)
2014-04-24 09:07:05,487 ERROR 9475431 [WatchDog_9475431] Component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET] finished with status ERROR. (In0: 0 recs)
Component pre-execute initialization failed.
Unexpected null value.
2014-04-24 09:07:05,487 ERROR 9475431 [WatchDog_9475431] Error details:
org.jetel.exception.JetelRuntimeException: Component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET] finished with status ERROR. (In0: 0 recs)
at org.jetel.graph.Node.createNodeException(Node.java:566)
at org.jetel.graph.Node.run(Node.java:542)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: Element [WRITE_UTILIZATION_SPREADSHEET:Write Utilization Spreadsheet]-Component pre-execute initialization failed.
at org.jetel.graph.Node.run(Node.java:497)
... 3 more
Caused by: java.io.IOException: java.lang.NullPointerException
at org.jetel.util.MultiFileWriter.prepareTargets(MultiFileWriter.java:225)
at org.jetel.util.MultiFileWriter.init(MultiFileWriter.java:159)
at com.opensys.cloveretl.component.SpreadsheetWriter.preExecute(Unknown Source)
at org.jetel.graph.Node.run(Node.java:495)
... 3 more
Caused by: java.io.IOException: java.lang.NullPointerException
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:257)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:206)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.connect(PooledSFTPConnection.java:212)
at org.jetel.component.fileoperation.pool.SFTPConnectionFactory.makeObject(SFTPConnectionFactory.java:30)
at org.jetel.component.fileoperation.pool.SFTPConnectionFactory.makeObject(SFTPConnectionFactory.java:21)
at org.jetel.component.fileoperation.pool.DefaultConnectionFactory.makeObject(DefaultConnectionFactory.java:38)
at org.jetel.component.fileoperation.pool.DefaultConnectionFactory.makeObject(DefaultConnectionFactory.java:26)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.jetel.component.fileoperation.pool.ConnectionPool.borrowObject(ConnectionPool.java:91)
at org.jetel.util.protocols.sftp.SFTPConnection.connect(SFTPConnection.java:107)
at org.jetel.util.protocols.sftp.SFTPConnection.getOutputStream(SFTPConnection.java:172)
at org.jetel.util.file.FileUtils.getOutputStream(FileUtils.java:1464)
at org.jetel.util.TargetFile.setOutput(TargetFile.java:622)
at org.jetel.util.TargetFile.setNextOutput(TargetFile.java:369)
at org.jetel.util.TargetFile.initOutput(TargetFile.java:546)
at org.jetel.util.TargetFile.init(TargetFile.java:170)
at org.jetel.util.MultiFileWriter.prepareTargets(MultiFileWriter.java:222)
... 6 more
Caused by: java.lang.NullPointerException
at com.jcraft.jsch.Buffer.putString(Buffer.java:59)
at com.jcraft.jsch.UserAuthKeyboardInteractive.start(UserAuthKeyboardInteractive.java:183)
at com.jcraft.jsch.Session.connect(Session.java:463)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getSession(PooledSFTPConnection.java:242)
... 22 more
2014-04-24 09:07:05,487 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_DENORMALIZER0,1,main]
2014-04-24 09:07:05,487 WARN 9475431 [READ_SOURCE_DATA_9475431] Optimizing connection failed: The CLOSE_CURSORS_AT_COMMIT option is not currently supported by the setHoldability method.
2014-04-24 09:07:05,487 WARN 9475431 [READ_SOURCE_DATA_9475431] Try to use another jdbc specific
2014-04-24 09:07:05,487 WARN 9475431 [READ_SOURCE_DATA_9475431] java.sql.SQLException: The CLOSE_CURSORS_AT_COMMIT option is not currently supported by the setHoldability method.
2014-04-24 09:07:05,487 DEBUG 9475431 [READ_SOURCE_DATA_9475431] READ_SOURCE_DATA: Sending query "<removed query>"
2014-04-24 09:07:05,487 DEBUG 9475431 [READ_TACTIC_INFO_9475431] READ_TACTIC_INFO: Sending query "<removed query>"
2014-04-24 09:07:05,502 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_DENORMALIZER0,1,main]
2014-04-24 09:07:05,518 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_EXT_MERGE_JOIN0,1,main]
2014-04-24 09:07:05,534 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_EXT_SORT0,1,main]
2014-04-24 09:07:05,549 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_EXT_SORT1,1,main]
2014-04-24 09:07:05,565 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_EXT_SORT2,1,main]
2014-04-24 09:07:05,580 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_READ_SOURCE_DATA,1,main]
2014-04-24 09:07:05,596 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_READ_SOURCE_DATA,1,main]
2014-04-24 09:07:05,612 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_READ_SOURCE_DATA,1,main]
2014-04-24 09:07:05,627 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_READ_SOURCE_DATA,1,main]
2014-04-24 09:07:05,643 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_READ_SOURCE_DATA,1,main]
2014-04-24 09:07:05,659 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_READ_SOURCE_DATA,1,main]
2014-04-24 09:07:05,659 DEBUG 9475431 [exNode_9475431_1359392341736_READ_SOURCE_DATA] READ_SOURCE_DATA: Query execution time: 00:00:00.172
2014-04-24 09:07:05,674 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_READ_TACTIC_INFO,1,main]
2014-04-24 09:07:05,690 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_READ_TACTIC_INFO,1,main]
2014-04-24 09:07:05,690 WARN 9475431 [exNode_9475431_1359392341736_READ_TACTIC_INFO] SQLException when closing statement
java.sql.SQLException: Internal driver error.(class java.lang.InterruptedException)
at com.ibm.as400.access.JDError.throwSQLException(JDError.java:480)
at com.ibm.as400.access.AS400JDBCConnection.sendAndReceive(AS400JDBCConnection.java:2608)
at com.ibm.as400.access.AS400JDBCConnection.sendAndReceive(AS400JDBCConnection.java:2516)
at com.ibm.as400.access.JDTransactionManager.commit(JDTransactionManager.java:190)
at com.ibm.as400.access.AS400JDBCConnection.commit(AS400JDBCConnection.java:615)
at org.apache.tomcat.dbcp.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)
at org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.commit(PoolingDataSource.java:211)
at org.jetel.connection.jdbc.specific.conn.BasicSqlConnection.commit(BasicSqlConnection.java:229)
at org.jetel.connection.jdbc.SQLDataParser.close(SQLDataParser.java:376)
at org.jetel.component.DBInputTable.processSqlQuery(DBInputTable.java:368)
at org.jetel.component.DBInputTable.execute(DBInputTable.java:301)
at org.jetel.graph.Node.run(Node.java:505)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2014-04-24 09:07:05,705 DEBUG 9475431 [WatchDog_9475431] trying to interrupt thread Thread[exNode_9475431_1359392341736_SIMPLE_COPY,1,main]
2014-04-24 09:07:05,721 DEBUG 9475431 [WatchDog_9475431] Phase 0 post-execution
2014-04-24 09:07:05,721 DEBUG 9475431 [WatchDog_9475431] Edges post-execution
2014-04-24 09:07:05,721 DEBUG 9475431 [WatchDog_9475431] Edges post-execution success.
2014-04-24 09:07:05,721 DEBUG 9475431 [WatchDog_9475431] Components post-execution
2014-04-24 09:07:05,721 DEBUG 9475431 [WatchDog_9475431] DENORMALIZER0 ...OK
2014-04-24 09:07:05,721 DEBUG 9475431 [WatchDog_9475431] EXT_MERGE_JOIN0 ...OK
2014-04-24 09:07:05,721 DEBUG 9475431 [WatchDog_9475431] EXT_SORT0 ...OK
2014-04-24 09:07:05,721 DEBUG 9475431 [WatchDog_9475431] EXT_SORT1 ...OK
2014-04-24 09:07:05,721 DEBUG 9475431 [WatchDog_9475431] EXT_SORT2 ...OK
2014-04-24 09:07:05,721 DEBUG 9475431 [WatchDog_9475431] READ_SOURCE_DATA ...OK
2014-04-24 09:07:05,924 DEBUG 9475431 [WatchDog_9475431] READ_TACTIC_INFO ...OK
2014-04-24 09:07:05,924 DEBUG 9475431 [WatchDog_9475431] SIMPLE_COPY ...OK
2014-04-24 09:07:05,924 DEBUG 9475431 [WatchDog_9475431] WRITE_UTILIZATION_SPREADSHEET ...OK
2014-04-24 09:07:05,924 DEBUG 9475431 [WatchDog_9475431] WRITE_UTILIZATION_SPREADSHEET1 ...OK
2014-04-24 09:07:05,924 DEBUG 9475431 [WatchDog_9475431] Phase 0 post-execution succeeded.
2014-04-24 09:07:05,924 INFO 9475431 [WatchDog_9475431] Execution of phase [0] finished with error - elapsed time(sec): 3
2014-04-24 09:07:05,924 ERROR 9475431 [WatchDog_9475431] !!! Phase finished with error - stopping graph run !!!
2014-04-24 09:07:05,924 INFO 9475431 [WatchDog_9475431] Post-execute finalization of connection:
2014-04-24 09:07:05,924 INFO 9475431 [WatchDog_9475431] DBConnection driver[null]:jndi[java:comp/env/jdbc/AS400]:url[null]:user[null] ... OK
2014-04-24 09:07:05,924 INFO 9475431 [WatchDog_9475431] Post-execute finalization of connection:
2014-04-24 09:07:05,924 INFO 9475431 [WatchDog_9475431] DBConnection driver[null]:jndi[java:comp/env/jdbc/ddv4Sql]:url[]:user[] ... OK
2014-04-24 09:07:05,940 INFO 9475431 [WatchDog_9475431] -----------------------** Summary of Phases execution **---------------------
2014-04-24 09:07:05,940 INFO 9475431 [WatchDog_9475431] Phase# Finished Status RunTime(sec) MemoryAllocation(KB)
2014-04-24 09:07:05,940 INFO 9475431 [JobFinalizer_9475431] Finalisation
2014-04-24 09:07:05,940 INFO 9475431 [WatchDog_9475431] 0 ERROR 3 5576902
2014-04-24 09:07:05,940 INFO 9475431 [WatchDog_9475431] ------------------------------** End of Summary **---------------------------
2014-04-24 09:07:05,940 DEBUG 9475431 [JobFinalizer_9475431] releasing runtime env 9475431
2014-04-24 09:07:05,940 INFO 9475431 [WatchDog_9475431] WatchDog thread finished - total execution time: 3 (sec)
2014-04-24 09:07:05,940 DEBUG 9475431 [JobFinalizer_9475431] updating RunRecord 9475431
2014-04-24 09:07:05,940 DEBUG 9475431 [JobFinalizer_9475431] updated RunRecord 9475431
2014-04-24 09:07:05,940 INFO 9475431 [JobFinalizer_9475431] RunTime: 39 s
2014-04-24 09:07:05,940 ERROR 9475431 [JobFinalizer_9475431]
------------------------------------------------- Error details ---------------------------------------------------
Component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET] finished with status ERROR. (In0: 0 recs)
Component pre-execute initialization failed.
Unexpected null value.
-------------------------------------------------------------------------------------------------------------------
2014-04-24 09:07:05,940 INFO 9475431 [JobFinalizer_9475431] Finished Status: ERROR
2014-04-24 09:07:05,940 DEBUG 9475431 [JobFinalizer_9475431] last execution ended with ERROR - free graph; runId=9475431
2014-04-24 09:07:05,940 DEBUG 9475431 [JobFinalizer_9475431] Releasing job logger -
Hi,
From the log it appears that connecting using the key was not successful. So can you try to connect from the server machine without the key (just using username and password), if possible? And if that is successful, can you verify the connection with the key again from the same server?
Thanks. -
The server connection is username and key only. I don't have a username-password combo to connect with. If I open the graph in the designer with my local project and run the graph, it works fine. And I'm able to connect with the key via FileZilla, so I know the key is good.
Here's the stack trace from a fully in-designer run of the same graph:INFO [main] - *** CloverETL framework/transformation graph, (c) 2002-2014 Javlin a.s. ***
INFO [main] - Running with CloverETL library version 3.5.0 build#058 compiled 03/02/2014 14:20:28
INFO [main] - Running on 4 CPU(s), OS Windows 7, architecture amd64, Java version 1.6.0_21, max available memory for JVM 1365376 KB
INFO [main] - Loading default properties from: defaultProperties
DEBUG [main] - Plugin org.jetel.bulkloader loaded.
id - org.jetel.bulkloader
version - 3.5.0.
provider-name - null
component { className = org.jetel.component.OracleDataWriter; type = ORACLE_DATA_WRITER; }
component { className = org.jetel.component.DB2DataWriter; type = DB2_DATA_WRITER; }
component { className = org.jetel.component.InformixDataWriter; type = INFORMIX_DATA_WRITER; }
component { className = org.jetel.component.MsSqlDataWriter; type = MS_SQL_DATA_WRITER; }
component { className = org.jetel.component.MysqlDataWriter; type = MYSQL_DATA_WRITER; }
component { className = org.jetel.component.PostgreSqlDataWriter; type = POSTGRESQL_DATA_WRITER; }
component { className = org.jetel.component.InfobrightDataWriter; type = INFOBRIGHT_DATA_WRITER; }
DEBUG [main] - Plugin org.jetel.component loaded.
id - org.jetel.component
version - 3.5.0.
provider-name - Javlin a.s.
component { className = org.jetel.component.DataReader; type = DATA_READER; }
component { className = org.jetel.component.DataWriter; type = DATA_WRITER; }
component { className = org.jetel.component.DelimitedDataReader; type = DELIMITED_DATA_READER; }
component { className = org.jetel.component.DelimitedDataWriter; type = DELIMITED_DATA_WRITER; }
component { className = org.jetel.component.SimpleCopy; type = SIMPLE_COPY; }
component { className = org.jetel.component.Concatenate; type = CONCATENATE; }
component { className = org.jetel.component.SimpleGather; type = SIMPLE_GATHER; }
component { className = org.jetel.component.Reformat; type = REFORMAT; }
component { className = org.jetel.component.DBInputTable; type = DB_INPUT_TABLE; }
component { className = org.jetel.component.Sort; type = SORT; }
component { className = org.jetel.component.DBOutputTable; type = DB_OUTPUT_TABLE; }
component { className = org.jetel.component.FixLenDataWriter; type = FIXLEN_DATA_WRITER; }
component { className = org.jetel.component.Dedup; type = DEDUP; }
component { className = org.jetel.component.FixLenDataReader; type = FIXLEN_DATA_READER; }
component { className = org.jetel.component.Merge; type = MERGE; }
component { className = org.jetel.component.MergeJoin; type = MERGE_JOIN; }
component { className = org.jetel.component.MergeJoin; type = EXT_MERGE_JOIN; }
component { className = org.jetel.component.MergeJoin; type = SORTED_JOIN; }
component { className = org.jetel.component.Trash; type = TRASH; }
component { className = org.jetel.component.DBExecute; type = DB_EXECUTE; }
component { className = org.jetel.component.HashJoin; type = HASH_JOIN; }
component { className = org.jetel.component.HashJoin; type = EXT_HASH_JOIN; }
component { className = org.jetel.component.CheckForeignKey; type = CHECK_FOREIGN_KEY; }
component { className = org.jetel.component.DBFDataReader; type = DBF_DATA_READER; }
component { className = org.jetel.component.DBFDataWriter; type = DBF_DATA_WRITER; }
component { className = org.jetel.component.ExtFilter; type = EXT_FILTER; }
component { className = org.jetel.component.ExtSort; type = EXT_SORT; }
component { className = org.jetel.component.SortWithinGroups; type = SORT_WITHIN_GROUPS; }
component { className = org.jetel.component.Partition; type = PARTITION; }
component { className = org.jetel.component.DataIntersection; type = DATA_INTERSECTION; }
component { className = org.jetel.component.Aggregate; type = AGGREGATE; }
component { className = org.jetel.component.SystemExecute; type = SYS_EXECUTE; }
component { className = org.jetel.component.RunGraph; type = RUN_GRAPH; }
component { className = org.jetel.component.HttpConnector; type = HTTP_CONNECTOR; }
component { className = org.jetel.component.KeyGenerator; type = KEY_GEN; }
component { className = org.jetel.component.AproxMergeJoin; type = APROX_MERGE_JOIN; }
component { className = org.jetel.component.DBJoin; type = DBJOIN; }
component { className = org.jetel.component.XLSReader; type = XLS_READER; }
component { className = org.jetel.component.XLSWriter; type = XLS_WRITER; }
component { className = org.jetel.component.CloverDataWriter; type = CLOVER_WRITER; }
component { className = org.jetel.component.CloverDataReader; type = CLOVER_READER; }
component { className = org.jetel.component.StructureWriter; type = STRUCTURE_WRITER; }
component { className = org.jetel.component.Normalizer; type = NORMALIZER; }
component { className = org.jetel.component.Denormalizer; type = DENORMALIZER; }
component { className = org.jetel.component.Rollup; type = ROLLUP; }
component { className = org.jetel.component.JmsReader; type = JMS_READER; }
component { className = org.jetel.component.JmsWriter; type = JMS_WRITER; }
component { className = org.jetel.component.LookupJoin; type = LOOKUP_JOIN; }
component { className = org.jetel.component.LookupTableReaderWriter; type = LOOKUP_TABLE_READER_WRITER; }
component { className = org.jetel.component.DataGenerator; type = DATA_GENERATOR; }
component { className = org.jetel.component.SequenceChecker; type = SEQUENCE_CHECKER; }
component { className = org.jetel.component.TextTableWriter; type = TEXT_TABLE_WRITER; }
component { className = org.jetel.component.XmlXPathReader; type = XML_XPATH_READER; }
component { className = org.jetel.component.XMLExtract; type = XML_EXTRACT; }
component { className = org.jetel.component.JsonExtract; type = JSON_EXTRACT; }
component { className = org.jetel.component.XmlReader; type = XML_READER; }
component { className = org.jetel.component.SpeedLimiter; type = SPEED_LIMITER; }
component { className = org.jetel.component.SpeedLimiter; type = SLEEP; }
component { className = org.jetel.component.JavaExecute; type = JAVA_EXECUTE; }
component { className = org.jetel.component.XmlWriter; type = XML_WRITER; }
component { className = org.jetel.component.ExtXmlWriter; type = EXT_XML_WRITER; }
component { className = org.jetel.component.XSLDataTransformer; type = XSL_TRANSFORMER; }
DEBUG [main] - Plugin org.jetel.hadoop loaded.
id - org.jetel.hadoop
version - 3.5.0.
provider-name - Javlin a.s.
connection { className = org.jetel.hadoop.connection.HadoopConnection; type = HADOOP; }
component { className = org.jetel.hadoop.component.HadoopWriter; type = HADOOP_WRITER; }
component { className = org.jetel.hadoop.component.HadoopReader; type = HADOOP_READER; }
urlResolver { className = org.jetel.hadoop.service.filesystem.HadoopPathResolver; type = HADOOP_HDFS; }
DEBUG [main] - Plugin org.jetel.connection loaded.
id - org.jetel.connection
version - 3.5.0.
provider-name - Javlin a.s.
connection { className = org.jetel.connection.jdbc.DBConnectionImpl; type = JDBC; }
connection { className = org.jetel.connection.jms.JmsConnection; type = JMS; }
DEBUG [main] - Plugin org.jetel.ctlfunction loaded.
id - org.jetel.ctlfunction
version - 3.5.0.
provider-name - Javlin a.s.
ctlfunction { libraryName = math; className = org.jetel.ctl.extensions.MathLib; }
ctlfunction { libraryName = date; className = org.jetel.ctl.extensions.DateLib; }
ctlfunction { libraryName = string; className = org.jetel.ctl.extensions.StringLib; }
ctlfunction { libraryName = convert; className = org.jetel.ctl.extensions.ConvertLib; }
ctlfunction { libraryName = container; className = org.jetel.ctl.extensions.ContainerLib; }
ctlfunction { libraryName = random; className = org.jetel.ctl.extensions.RandomLib; }
ctlfunction { libraryName = util; className = org.jetel.ctl.extensions.UtilLib; }
ctlfunction { libraryName = dynamic; className = org.jetel.ctl.extensions.DynamicLib; }
DEBUG [main] - Plugin org.jetel.engine loaded.
id - org.jetel.engine
version - 3.5.0.
provider-name - Javlin a.s.
dictionaryType { className = org.jetel.graph.dictionary.StringDictionaryType; type = string; }
dictionaryType { className = org.jetel.graph.dictionary.ObjectDictionaryType; type = object; }
dictionaryType { className = org.jetel.graph.dictionary.ReadableChannelDictionaryType; type = readable.channel; }
dictionaryType { className = org.jetel.graph.dictionary.WritableChannelDictionaryType; type = writable.channel; }
dictionaryType { className = org.jetel.graph.dictionary.IntegerDictionaryType; type = integer; }
dictionaryType { className = org.jetel.graph.dictionary.LongDictionaryType; type = long; }
dictionaryType { className = org.jetel.graph.dictionary.DecimalDictionaryType; type = decimal; }
dictionaryType { className = org.jetel.graph.dictionary.NumberDictionaryType; type = number; }
dictionaryType { className = org.jetel.graph.dictionary.DateDictionaryType; type = date; }
dictionaryType { className = org.jetel.graph.dictionary.BooleanDictionaryType; type = boolean; }
dictionaryType { className = org.jetel.graph.dictionary.ByteDictionaryType; type = byte; }
dictionaryType { className = org.jetel.graph.dictionary.ListDictionaryType; type = list; }
dictionaryType { className = org.jetel.graph.dictionary.MapDictionaryType; type = map; }
tlCompiler { className = org.jetel.ctl.TLCompiler; type = simple.compiler; }
ctlfunction { libraryName = record; className = org.jetel.ctl.extensions.IntegralLib; }
urlResolver { className = org.jetel.util.file.SMBPathResolver; type = SMB; }
authorityProxy { priority = 0; className = org.jetel.graph.runtime.PrimitiveAuthorityProxy; type = PrimitiveAuthorityProxy; }
DEBUG [main] - Plugin org.jetel.jdbc loaded.
id - org.jetel.jdbc
version - 3.5.0.
provider-name - Javlin a.s.
jdbcSpecific { name = Generic; class = org.jetel.connection.jdbc.specific.impl.DefaultJdbcSpecific; database = GENERIC; }
jdbcSpecific { name = DB2; class = org.jetel.connection.jdbc.specific.impl.DB2Specific; database = DB2; }
jdbcSpecific { name = Derby; class = org.jetel.connection.jdbc.specific.impl.DerbySpecific; productName = Apache Derby; database = DERBY; }
jdbcSpecific { name = EXASolution; class = org.jetel.connection.jdbc.specific.impl.ExasolSpecific; database = EXASOLUTION; }
jdbcSpecific { name = Firebird; class = org.jetel.connection.jdbc.specific.impl.FirebirdSpecific; database = FIREBIRD; }
jdbcSpecific { name = Generic ODBC; class = org.jetel.connection.jdbc.specific.impl.GenericODBCSpecific; database = GENERICODBC; }
jdbcSpecific { name = Hive; class = org.jetel.connection.jdbc.specific.impl.HiveSpecific; database = HIVE; }
jdbcSpecific { name = Informix; class = org.jetel.connection.jdbc.specific.impl.InformixSpecific; database = INFORMIX; }
jdbcSpecific { name = Microsoft Access; class = org.jetel.connection.jdbc.specific.impl.MSAccessSpecific; database = MSACCESS; }
jdbcSpecific { name = MS SQL Server 2008-2012; class = org.jetel.connection.jdbc.specific.impl.MSSQLSpecific; productName = Microsoft SQL Server; database = MSSQL; }
jdbcSpecific { name = MS SQL Server 2000-2005; class = org.jetel.connection.jdbc.specific.impl.MSSQLSpecific2005; database = MSSQL2005; }
jdbcSpecific { name = MySQL; class = org.jetel.connection.jdbc.specific.impl.MySQLSpecific; productName = MySQL; database = MYSQL; }
jdbcSpecific { name = Oracle; class = org.jetel.connection.jdbc.specific.impl.OracleSpecific; productName = Oracle; database = ORACLE; }
jdbcSpecific { name = Pervasive; class = org.jetel.connection.jdbc.specific.impl.PervasiveSpecific; database = PERVASIVE; }
jdbcSpecific { name = PostgreSQL; class = org.jetel.connection.jdbc.specific.impl.PostgreSpecific; database = POSTGRE; }
jdbcSpecific { name = Sybase; class = org.jetel.connection.jdbc.specific.impl.SybaseSpecific; database = SYBASE; }
jdbcSpecific { name = SQLite; class = org.jetel.connection.jdbc.specific.impl.SQLiteSpecific; database = SQLITE; }
jdbcSpecific { name = Vertica; class = org.jetel.connection.jdbc.specific.impl.VerticaSpecific; database = VERTICA; }
jdbcDriver { driverLibrary = lib/mysql/mysql-connector-java-5.1.26-bin.jar; jdbcSpecific = MYSQL; name = MySQL; dbDriver = org.gjt.mm.mysql.Driver; jdbc.zeroDateTimeBehavior = convertToNull; urlHint = jdbc:mysql://hostname:3306/database; database = MYSQL; }
jdbcDriver { driverLibrary = lib/oracle/ojdbc6.jar; jdbcSpecific = ORACLE; name = Oracle; dbDriver = oracle.jdbc.OracleDriver; urlHint = jdbc:oracle:thin:@host:1521:SID,jdbc:oracle:thin:@cluster-alias:1521/service_name,jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=host1)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=host2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=service_name))),jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=cluster_alias)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=service_name))),jdbc:oracle:thin:@host:1521/service_name; database = ORACLE; }
jdbcDriver { driverLibrary = lib/postgre/postgresql-9.2-1002.jdbc4.jar; jdbcSpecific = POSTGRE; name = PostgreSQL; dbDriver = org.postgresql.Driver; urlHint = jdbc:postgresql://hostname/database; database = POSTGRE; }
jdbcDriver { driverLibrary = lib/jtds/jtds-1.2.8.jar; jdbcSpecific = MSSQL; name = Microsoft SQL Server; dbDriver = net.sourceforge.jtds.jdbc.Driver; urlHint = jdbc:jtds:sqlserver://hostname:1433/database;instance=instanceName,jdbc:jtds:sqlserver://hostname:1433/database; database = MSSQL; }
jdbcDriver { driverLibrary = sun.jdbc.odbc.JdbcOdbcDriver; jdbcSpecific = GENERICODBC; name = Generic ODBC; dbDriver = sun.jdbc.odbc.JdbcOdbcDriver; urlHint = jdbc:odbc:dsn_source;; database = GENERICODBC; }
jdbcDriver { driverLibrary = sun.jdbc.odbc.JdbcOdbcDriver; jdbcSpecific = MSACCESS; name = Microsoft Access; dbDriver = sun.jdbc.odbc.JdbcOdbcDriver; urlHint = jdbc:odbc:driver={Microsoft Access Driver (*.mdb)};DBQ=database_file,jdbc:odbc:driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=database_file,jdbc:odbc:odbc_source_name; database = MSACCESS; }
jdbcDriver { driverLibrary = lib/jtds/jtds-1.2.8.jar; jdbcSpecific = SYBASE; name = Sybase; dbDriver = net.sourceforge.jtds.jdbc.Driver; urlHint = jdbc:jtds:sybase://hostname:7100/database; database = SYBASE; }
jdbcDriver { driverLibrary = lib/sqlite/sqlite-jdbc-3.6.20.1.jar; jdbcSpecific = SQLITE; name = SQLite; dbDriver = org.sqlite.JDBC; urlHint = jdbc:sqlite:C:/database; database = SQLITE; }
jdbcDriver { driverLibrary = lib/derby/derbyclient.jar; jdbcSpecific = DERBY; name = Derby; dbDriver = org.apache.derby.jdbc.ClientDriver; urlHint = jdbc:derby://hostname:1527/derby.db; database = DERBY; }
jdbcDriver { driverLibrary = lib/firebird/jaybird-full-2.2.1.jar; jdbcSpecific = FIREBIRD; name = Firebird; dbDriver = org.firebirdsql.jdbc.FBDriver; urlHint = jdbc:firebirdsql:localhost/3050:/firebird/test.gdb; database = FIREBIRD; }
DEBUG [main] - Plugin org.jetel.license loaded.
id - org.jetel.license
version - 3.5.0.
provider-name - Javlin a.s.
DEBUG [main] - Plugin org.jetel.lookup loaded.
id - org.jetel.lookup
version - 3.5.0.
provider-name - Javlin a.s.
lookup { className = org.jetel.lookup.SimpleLookupTable; type = simpleLookup; }
lookup { className = org.jetel.lookup.DBLookupTable; type = dbLookup; }
lookup { className = org.jetel.lookup.RangeLookupTable; type = rangeLookup; }
DEBUG [main] - Plugin org.jetel.sequence loaded.
id - org.jetel.sequence
version - 3.5.0.
provider-name - Javlin a.s.
sequence { className = org.jetel.sequence.SimpleSequence; type = SIMPLE_SEQUENCE; }
sequence { className = org.jetel.sequence.PrimitiveSequence; type = PRIMITIVE_SEQUENCE; }
DEBUG [main] - Plugin org.jetel.thirdparty loaded.
id - org.jetel.thirdparty
version - 3.5.0.
provider-name - Javlin a.s.
component { className = org.jetel.component.Filter; type = FILTER; }
component { className = com.linagora.component.LdapReader; type = LDAP_READER; }
component { className = com.linagora.component.LdapWriter; type = LDAP_WRITER; }
DEBUG [main] - Plugin org.jetel.tlfunction loaded.
id - org.jetel.tlfunction
version - 3.5.0.
provider-name - Javlin a.s.
tlfunction { libraryName = math; className = org.jetel.interpreter.extensions.MathLib; function = sqrt,log,log10,exp,round,pow,pi,e,random,random_gaussian,random_boolean,random_int,random_long,abs,bit_and,bit_or,bit_xor,bit_set,bit_invert,bit_is_set,bit_lshift,bit_rshift; }
tlfunction { libraryName = date; className = org.jetel.interpreter.extensions.DateLib; function = today,dateadd,datediff,trunc,trunc_date,random_date; }
tlfunction { libraryName = string; className = org.jetel.interpreter.extensions.StringLib; function = concat,uppercase,lowercase,substring,left,right,trim,length,replace,split,char_at,is_blank,is_ascii,is_number,is_integer,is_long,is_date,remove_diacritic,remove_blank_space,get_alphanumeric_chars,translate,join,index_of,count_char,chop,remove_nonprintable,remove_nonascii,find,cut,random_string; }
tlfunction { libraryName = convert; className = org.jetel.interpreter.extensions.ConvertLib; function = num2str,date2str,str2date,date2num,str2num,try_convert,base64byte,byte2base64,bits2str,str2bits,hex2byte,byte2hex,num2num,num2bool,bool2num,str2bool,long2date,date2long,to_string,md5,sha,long2pacdecimal,pacdecimal2long,get_field_name,get_field_type; }
tlfunction { libraryName = container; className = org.jetel.interpreter.extensions.ContainerLib; function = remove_all,push,pop,poll,remove,insert,sort,copy,reverse,dict_put_str,dict_get_str; }
DEBUG [main] - Plugin com.opensys.cloveretl.addressdoctor loaded.
id - com.opensys.cloveretl.addressdoctor
version - 3.5.0.058
provider-name - Javlin a.s.
component { className = com.opensys.cloveretl.addressdoctor.component.AddressDoctorI5; type = ADDRESS_DOCTOR_I5; }
DEBUG [main] - Plugin com.opensys.cloveretl.compiler loaded.
id - com.opensys.cloveretl.compiler
version - 3.5.0.058
provider-name - Javlin a.s.
tlCompiler { priority = 1; className = com.opensys.cloveretl.ctl.BytecodeCompiler; type = bytecode.compiler; }
DEBUG [main] - Plugin com.opensys.clover.component loaded.
id - com.opensys.clover.component
version - 3.5.0.058
provider-name - Javlin a.s.
component { className = com.opensys.cloveretl.component.EmailFilter; type = EMAIL_FILTER; }
component { className = com.opensys.cloveretl.component.EmailSender; type = EMAIL_SENDER; }
component { className = com.opensys.cloveretl.component.EmailReader; type = EMAIL_READER; }
component { className = com.opensys.cloveretl.component.FastSort; type = FAST_SORT; }
component { className = com.opensys.cloveretl.component.MetaPivot; type = META_PIVOT; }
component { className = com.opensys.cloveretl.component.MultiLevelReader; type = MULTI_LEVEL_READER; }
component { className = com.opensys.cloveretl.component.RelationalJoin; type = RELATIONAL_JOIN; }
component { className = com.opensys.cloveretl.component.ParallelReader; type = PARALLEL_READER; }
component { className = com.opensys.cloveretl.component.ClusterPartition; type = CLUSTER_PARTITION; }
component { className = com.opensys.cloveretl.component.ClusterSimpleGather; type = CLUSTER_SIMPLE_GATHER; }
component { className = com.opensys.cloveretl.component.ClusterMerge; type = CLUSTER_MERGE; }
component { className = com.opensys.cloveretl.component.ClusterSimpleCopy; type = CLUSTER_SIMPLE_COPY; }
component { className = com.opensys.cloveretl.component.WebServiceClient; type = WEB_SERVICE_CLIENT; }
component { className = com.opensys.cloveretl.component.complexdatareader.ComplexDataReader; type = COMPLEX_DATA_READER; }
component { className = com.opensys.cloveretl.component.DataSampler; type = DATA_SAMPLER; }
component { className = com.opensys.cloveretl.component.Pivot; type = PIVOT; }
component { className = com.opensys.cloveretl.component.SpreadsheetReader; type = SPREADSHEET_READER; }
component { className = com.opensys.cloveretl.component.SpreadsheetWriter; type = SPREADSHEET_WRITER; }
component { className = com.opensys.cloveretl.component.BeanReader; type = BEAN_READER; }
component { className = com.opensys.cloveretl.component.BeanWriter; type = BEAN_WRITER; }
component { className = com.opensys.cloveretl.component.MapWriter; type = MAP_WRITER; }
component { className = com.opensys.cloveretl.component.JsonReader; type = JSON_READER; }
component { className = com.opensys.cloveretl.component.JsonWriter; type = JSON_WRITER; }
component { className = com.opensys.cloveretl.component.ExecuteGraph; type = EXECUTE_GRAPH; }
component { className = com.opensys.cloveretl.component.ExecuteJobflow; type = EXECUTE_JOBFLOW; }
component { className = com.opensys.cloveretl.component.MonitorGraph; type = MONITOR_GRAPH; }
component { className = com.opensys.cloveretl.component.MonitorJobflow; type = MONITOR_JOBFLOW; }
component { className = com.opensys.cloveretl.component.ExecuteScript; type = EXECUTE_SCRIPT; }
component { className = com.opensys.cloveretl.component.KillGraph; type = KILL_GRAPH; }
component { className = com.opensys.cloveretl.component.KillJobflow; type = KILL_JOBFLOW; }
component { className = com.opensys.cloveretl.component.Fail; type = FAIL; }
component { className = com.opensys.cloveretl.component.SetJobOutput; type = SET_JOB_OUTPUT; }
component { className = com.opensys.cloveretl.component.GetJobInput; type = GET_JOB_INPUT; }
component { className = com.opensys.cloveretl.component.TokenGather; type = TOKEN_GATHER; }
component { className = com.opensys.cloveretl.component.Barrier; type = BARRIER; }
component { className = com.opensys.cloveretl.component.Condition; type = CONDITION; }
component { className = com.opensys.cloveretl.component.Success; type = SUCCESS; }
component { className = com.opensys.cloveretl.component.Combine; type = COMBINE; }
component { className = com.opensys.cloveretl.component.ListFiles; type = LIST_FILES; }
component { className = com.opensys.cloveretl.component.CreateFiles; type = CREATE_FILES; }
component { className = com.opensys.cloveretl.component.DeleteFiles; type = DELETE_FILES; }
component { className = com.opensys.cloveretl.component.CopyFiles; type = COPY_FILES; }
component { className = com.opensys.cloveretl.component.MoveFiles; type = MOVE_FILES; }
component { className = com.opensys.cloveretl.component.LoadBalancingPartition; type = LOAD_BALANCING_PARTITION; }
component { className = com.opensys.cloveretl.component.ClusterLoadBalancingPartition; type = CLUSTER_LOAD_BALANCING_PARTITION; }
component { className = com.opensys.cloveretl.component.ClusterRepartition; type = CLUSTER_REPARTITION; }
component { className = com.opensys.cloveretl.component.ClusterRegather; type = CLUSTER_REGATHER; }
component { className = com.opensys.cloveretl.component.ExecuteMapReduce; type = EXECUTE_MAPREDUCE; }
component { className = com.opensys.cloveretl.component.Loop; type = LOOP; }
DEBUG [main] - Plugin com.opensys.cloveretl.ctlfunction loaded.
id - com.opensys.cloveretl.ctlfunction
version - 3.5.0.058
provider-name - Javlin a.s.
ctlfunction { libraryName = string; className = com.opensys.cloveretl.ctl.extensions.StringLib; }
DEBUG [main] - Plugin com.opensys.clover.lookup loaded.
id - com.opensys.clover.lookup
version - 3.5.0.058
provider-name - Javlin a.s.
lookup { className = com.opensys.cloveretl.lookup.PersistentLookupTable; type = persistentLookup; }
lookup { className = com.opensys.cloveretl.lookup.AspellLookupTable; type = aspellLookup; }
DEBUG [main] - Plugin com.opensys.cloveretl.tlfunction loaded.
id - com.opensys.cloveretl.tlfunction
version - 3.5.0.058
provider-name - Javlin a.s.
tlfunction { libraryName = string; className = com.opensys.cloveretl.interpreter.extensions.StringLib; function = soundex,edit_distance,metaphone,NYSIIS; }
DEBUG [main] - Plugin com.opensys.clover.lotusdomino loaded.
id - com.opensys.clover.lotusdomino
version - 3.5.0.058
provider-name - Javlin a.s.
connection { className = com.opensys.cloveretl.connection.domino.DominoConnection; type = DOMINO; }
component { className = com.opensys.cloveretl.component.DominoReader; type = DOMINO_READER; }
component { className = com.opensys.cloveretl.component.DominoWriter; type = DOMINO_WRITER; }
DEBUG [main] - Plugin com.opensys.cloveretl.quickbase loaded.
id - com.opensys.cloveretl.quickbase
version - 3.5.0.058
provider-name - Javlin a.s.
connection { className = com.opensys.cloveretl.quickbase.connection.QuickBaseConnection; type = QUICKBASE; }
component { className = com.opensys.cloveretl.quickbase.component.QuickBaseRecordReader; type = QUICKBASE_RECORD_READER; }
component { className = com.opensys.cloveretl.quickbase.component.QuickBaseQueryReader; type = QUICKBASE_QUERY_READER; }
component { className = com.opensys.cloveretl.quickbase.component.QuickBaseRecordWriter; type = QUICKBASE_RECORD_WRITER; }
component { className = com.opensys.cloveretl.quickbase.component.QuickBaseImportCSV; type = QUICKBASE_IMPORT_CSV; }
INFO [main] - Graph definition file: graph/<Graph name removed>
DEBUG [main] - License location: C:\Program Files (x86)\CloverETL Designer\configuration\com.cloveretl.gui
File license.dat is used as license.
DEBUG [main] - License location: ./licenses
INFO [main] - Graph revision: 1.46 Modified by:
INFO [main] - Checking graph configuration...
DEBUG [main] - Component 'DENORMALIZER0' runs in INTERPRETED mode
DEBUG [main] - Component 'EXT_MERGE_JOIN0' runs in INTERPRETED mode
WARN [main] - Issue in component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET]
Input edge metadata field "FirstShipDate" (date) format "yyyy-MM-dd HH:mm:ss" does not have "excel:" prefix, it will be ignored
WARN [main] - Issue in component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET]
Input edge metadata field "TheDate" (date) format "yyyy-MM-dd HH:mm:ss" does not have "excel:" prefix, it will be ignored
WARN [main] - Issue in component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET1]
Input edge metadata field "FirstShipDate" (date) format "yyyy-MM-dd HH:mm:ss" does not have "excel:" prefix, it will be ignored
WARN [main] - Issue in component [Write Utilization Spreadsheet:WRITE_UTILIZATION_SPREADSHEET1]
Input edge metadata field "TheDate" (date) format "yyyy-MM-dd HH:mm:ss" does not have "excel:" prefix, it will be ignored
INFO [main] - Graph configuration is valid.
INFO [main] - Graph initialization (GenentechUtilizationSource)
INFO [main] - Initializing connection:
INFO [main] - DBConnection driver[org.jetel.connection.jdbc.driver.JdbcDriverImpl@3202a2cc]:jndi[null]:url<Connection Info Removed>
INFO [main] - Initializing connection:
INFO [main] - DBConnection driver[org.jetel.connection.jdbc.driver.JdbcDriverImpl@7c6c2896]:jndi[]:url<Connection Info Removed>
INFO [main] - Initializing phase 0
DEBUG [main] - Initializing edges
DEBUG [main] - All edges initialized successfully.
DEBUG [main] - Initializing nodes
DEBUG [main] - Component 'DENORMALIZER0' runs in INTERPRETED mode
DEBUG [main] - DENORMALIZER0 ...OK
DEBUG [main] - Component 'EXT_MERGE_JOIN0' runs in INTERPRETED mode
DEBUG [main] - EXT_MERGE_JOIN0 ...OK
DEBUG [main] - EXT_SORT0 ...OK
DEBUG [main] - EXT_SORT1 ...OK
DEBUG [main] - EXT_SORT2 ...OK
DEBUG [main] - READ_SOURCE_DATA ...OK
DEBUG [main] - READ_TACTIC_INFO ...OK
DEBUG [main] - SIMPLE_COPY ...OK
DEBUG [main] - WRITE_UTILIZATION_SPREADSHEET ...OK
DEBUG [main] - WRITE_UTILIZATION_SPREADSHEET1 ...OK
INFO [main] - Phase 0 initialized successfully.
DEBUG [main] - register MBean with name:org.jetel.graph.runtime:type=CLOVERJMX_1359392341736_0
INFO [WatchDog_0] - Job parameters: <Parameter Info Removed>
DEBUG [WatchDog_0] - Graph runtime context: <Context Info removed>
DEBUG [WatchDog_0] - EdgeType [Edge3] : DIRECT
DEBUG [WatchDog_0] - EdgeType [Edge4] : DIRECT
DEBUG [WatchDog_0] - EdgeType [Edge2] : DIRECT
DEBUG [WatchDog_0] - EdgeType [Edge5] : DIRECT
DEBUG [WatchDog_0] - EdgeType [Edge6] : DIRECT
DEBUG [WatchDog_0] - EdgeType [Edge0] : DIRECT
DEBUG [WatchDog_0] - EdgeType [Edge1] : DIRECT
DEBUG [WatchDog_0] - EdgeType [Edge7] : DIRECT
DEBUG [WatchDog_0] - EdgeType [Edge8] : DIRECT
INFO [WatchDog_0] - Pre-execute initialization of connection:
INFO [WatchDog_0] - DBConnection driver[org.jetel.connection.jdbc.driver.JdbcDriverImpl@3202a2cc]:jndi[null]:url<Connection Info Removed> ... OK
INFO [WatchDog_0] - Pre-execute initialization of connection:
INFO [WatchDog_0] - DBConnection driver[org.jetel.connection.jdbc.driver.JdbcDriverImpl@7c6c2896]:jndi[]:url<Connection Info Removed> ... OK
INFO [WatchDog_0] - Starting up all nodes in phase [0]
DEBUG [WatchDog_0] - DENORMALIZER0 ... starting
DEBUG [WatchDog_0] - EXT_MERGE_JOIN0 ... starting
DEBUG [WatchDog_0] - EXT_SORT0 ... starting
DEBUG [WatchDog_0] - EXT_SORT1 ... starting
DEBUG [WatchDog_0] - EXT_SORT2 ... starting
DEBUG [WatchDog_0] - READ_SOURCE_DATA ... starting
DEBUG [WatchDog_0] - READ_TACTIC_INFO ... starting
DEBUG [WatchDog_0] - SIMPLE_COPY ... starting
DEBUG [WatchDog_0] - WRITE_UTILIZATION_SPREADSHEET ... starting
DEBUG [WatchDog_0] - WRITE_UTILIZATION_SPREADSHEET1 ... starting
DEBUG [WatchDog_0] - All components are ready to start.
INFO [WatchDog_0] - Successfully started all nodes in phase!
DEBUG [WRITE_UTILIZATION_SPREADSHEET_0] - SFTP connecting to <SFTP Address Removed> using the following private keys: [<Key Path Removed>]
DEBUG [WRITE_UTILIZATION_SPREADSHEET_0] - Adding new identity from <Key Path Removed>
DEBUG [WRITE_UTILIZATION_SPREADSHEET_0] - Connecting to b2bexit-rwc.gene.com with password authentication
INFO [WatchDog_0] - ---------------------** Start of tracking Log for phase [0] **-------------------
INFO [WatchDog_0] - Time: 24/04/14 12:38:17
INFO [WatchDog_0] - Node ID Port #Records #KB Rec/s KB/s
INFO [WatchDog_0] - ---------------------------------------------------------------------------------
INFO [WatchDog_0] - Read Tactic Info READ_TACTIC_INFO RUNNING
INFO [WatchDog_0] - %cpu:0 Out:0 0 0 0 0
INFO [WatchDog_0] - ExtSort EXT_SORT0 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - Denormalizer DENORMALIZER0 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - Read Source Data READ_SOURCE_DATA RUNNING
INFO [WatchDog_0] - %cpu:0 Out:0 0 0 0 0
INFO [WatchDog_0] - ExtSort EXT_SORT1 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - ExtMergeJoin EXT_MERGE_JOIN0 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - In:1 0 0 0 0
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - ExtSort EXT_SORT2 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - SimpleCopy SIMPLE_COPY RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - Out:1 0 0 0 0
INFO [WatchDog_0] - Write Utilization SpreaWRITE_UTILIZATION_SPREADSHEET RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - Write Utilization SpreaWRITE_UTILIZATION_SPREADSHEET1 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - ---------------------------------** End of Log **--------------------------------
DEBUG [READ_SOURCE_DATA_0] - READ_SOURCE_DATA: Sending query <Query Removed>
DEBUG [READ_TACTIC_INFO_0] - READ_TACTIC_INFO: Sending query <Query Removed>
DEBUG [READ_TACTIC_INFO_0] - READ_TACTIC_INFO: Query execution time: 00:00:00.051
DEBUG [READ_SOURCE_DATA_0] - READ_SOURCE_DATA: Query execution time: 00:00:00.087
INFO [WatchDog_0] - ---------------------** Start of tracking Log for phase [0] **-------------------
INFO [WatchDog_0] - Time: 24/04/14 12:38:22
INFO [WatchDog_0] - Node ID Port #Records #KB Rec/s KB/s
INFO [WatchDog_0] - ---------------------------------------------------------------------------------
INFO [WatchDog_0] - Read Tactic Info READ_TACTIC_INFO RUNNING
INFO [WatchDog_0] - %cpu:1 Out:0 1310 130 258 25
INFO [WatchDog_0] - ExtSort EXT_SORT0 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 130 0 25
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - Denormalizer DENORMALIZER0 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - Read Source Data READ_SOURCE_DATA RUNNING
INFO [WatchDog_0] - %cpu:5 Out:0 2731 875 539 172
INFO [WatchDog_0] - ExtSort EXT_SORT1 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 1584 875 312 172
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - ExtMergeJoin EXT_MERGE_JOIN0 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - In:1 0 0 0 0
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - ExtSort EXT_SORT2 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - SimpleCopy SIMPLE_COPY RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - Out:0 0 0 0 0
INFO [WatchDog_0] - Out:1 0 0 0 0
INFO [WatchDog_0] - Write Utilization SpreaWRITE_UTILIZATION_SPREADSHEET RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - Write Utilization SpreaWRITE_UTILIZATION_SPREADSHEET1 RUNNING
INFO [WatchDog_0] - %cpu:0 In:0 0 0 0 0
INFO [WatchDog_0] - ---------------------------------** End of Log **--------------------------------
INFO [WatchDog_0] - ---------------------** Start of tracking Log for phase [0] **-------------------
INFO [WatchDog_0] - Time: 24/04/14 12:38:30
INFO [WatchDog_0] - Node ID Port #Records #KB Rec/s KB/s
INFO [WatchDog_0] - ---------------------------------------------------------------------------------
INFO [WatchDog_0] - Read Tactic Info READ_TACTIC_INFO FINISHED_OK
INFO [WatchDog_0] - %cpu:0 Out:0 4208 419 354 35
INFO [WatchDog_0] - ExtSort EXT_SORT0 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 4208 419 514 35
INFO [WatchDog_0] - Out:0 4208 419 514 51
INFO [WatchDog_0] - Denormalizer DENORMALIZER0 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 4208 419 514 51
INFO [WatchDog_0] - Out:0 4191 162 512 19
INFO [WatchDog_0] - Read Source Data READ_SOURCE_DATA FINISHED_OK
INFO [WatchDog_0] - %cpu:2 Out:0 3367 1077 77 24
INFO [WatchDog_0] - ExtSort EXT_SORT1 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 3367 1077 218 24
INFO [WatchDog_0] - Out:0 3367 1077 411 131
INFO [WatchDog_0] - ExtMergeJoin EXT_MERGE_JOIN0 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 3367 1077 411 131
INFO [WatchDog_0] - In:1 4191 162 512 19
INFO [WatchDog_0] - Out:0 3367 1114 411 136
INFO [WatchDog_0] - ExtSort EXT_SORT2 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 3367 1114 411 136
INFO [WatchDog_0] - Out:0 3367 1114 411 136
INFO [WatchDog_0] - SimpleCopy SIMPLE_COPY FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 3367 1114 411 136
INFO [WatchDog_0] - Out:0 3367 1114 411 136
INFO [WatchDog_0] - Out:1 3367 1114 411 136
INFO [WatchDog_0] - Write Utilization SpreaWRITE_UTILIZATION_SPREADSHEET RUNNING
INFO [WatchDog_0] - %cpu:10 In:0 3367 1114 411 136
INFO [WatchDog_0] - Write Utilization SpreaWRITE_UTILIZATION_SPREADSHEET1 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 3367 1114 411 136
INFO [WatchDog_0] - ---------------------------------** End of Log **--------------------------------
DEBUG [WatchDog_0] - Phase 0 post-execution
DEBUG [WatchDog_0] - Edges post-execution
DEBUG [WatchDog_0] - Edges post-execution success.
DEBUG [WatchDog_0] - Components post-execution
DEBUG [WatchDog_0] - DENORMALIZER0 ...OK
DEBUG [WatchDog_0] - EXT_MERGE_JOIN0 ...OK
DEBUG [WatchDog_0] - EXT_SORT0 ...OK
DEBUG [WatchDog_0] - EXT_SORT1 ...OK
DEBUG [WatchDog_0] - EXT_SORT2 ...OK
DEBUG [WatchDog_0] - READ_SOURCE_DATA ...OK
DEBUG [WatchDog_0] - READ_TACTIC_INFO ...OK
DEBUG [WatchDog_0] - SIMPLE_COPY ...OK
DEBUG [WatchDog_0] - WRITE_UTILIZATION_SPREADSHEET ...OK
DEBUG [WatchDog_0] - WRITE_UTILIZATION_SPREADSHEET1 ...OK
DEBUG [WatchDog_0] - Phase 0 post-execution succeeded.
INFO [WatchDog_0] - ----------------------** Final tracking Log for phase [0] **---------------------
INFO [WatchDog_0] - Time: 24/04/14 12:38:31
INFO [WatchDog_0] - Node ID Port #Records #KB aRec/s aKB/s
INFO [WatchDog_0] - ---------------------------------------------------------------------------------
INFO [WatchDog_0] - Read Tactic Info READ_TACTIC_INFO FINISHED_OK
INFO [WatchDog_0] - %cpu:1 Out:0 4208 419 226 22
INFO [WatchDog_0] - ExtSort EXT_SORT0 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 4208 419 226 22
INFO [WatchDog_0] - Out:0 4208 419 226 22
INFO [WatchDog_0] - Denormalizer DENORMALIZER0 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 4208 419 226 22
INFO [WatchDog_0] - Out:0 4191 162 225 8
INFO [WatchDog_0] - Read Source Data READ_SOURCE_DATA FINISHED_OK
INFO [WatchDog_0] - %cpu:5 Out:0 3367 1077 181 57
INFO [WatchDog_0] - ExtSort EXT_SORT1 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 3367 1077 181 57
INFO [WatchDog_0] - Out:0 3367 1077 181 57
INFO [WatchDog_0] - ExtMergeJoin EXT_MERGE_JOIN0 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 3367 1077 181 57
INFO [WatchDog_0] - In:1 4191 162 225 8
INFO [WatchDog_0] - Out:0 3367 1114 181 59
INFO [WatchDog_0] - ExtSort EXT_SORT2 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 3367 1114 181 59
INFO [WatchDog_0] - Out:0 3367 1114 181 59
INFO [WatchDog_0] - SimpleCopy SIMPLE_COPY FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 3367 1114 181 59
INFO [WatchDog_0] - Out:0 3367 1114 181 59
INFO [WatchDog_0] - Out:1 3367 1114 181 59
INFO [WatchDog_0] - Write Utilization SpreaWRITE_UTILIZATION_SPREADSHEET FINISHED_OK
INFO [WatchDog_0] - %cpu:10 In:0 3367 1114 181 59
INFO [WatchDog_0] - Write Utilization SpreaWRITE_UTILIZATION_SPREADSHEET1 FINISHED_OK
INFO [WatchDog_0] - %cpu:0 In:0 3367 1114 181 59
INFO [WatchDog_0] - ---------------------------------** End of Log **--------------------------------
INFO [WatchDog_0] - Execution of phase [0] successfully finished - elapsed time(sec): 18
INFO [WatchDog_0] - Post-execute finalization of connection:
INFO [WatchDog_0] - DBConnection driver[org.jetel.connection.jdbc.driver.JdbcDriverImpl@3202a2cc]:jndi[null]:url<Connection Info Removed>
INFO [WatchDog_0] - Post-execute finalization of connection:
INFO [WatchDog_0] - DBConnection driver[org.jetel.connection.jdbc.driver.JdbcDriverImpl@7c6c2896]:jndi[]:url<Connection Info Removed>
INFO [WatchDog_0] - -----------------------** Summary of Phases execution **---------------------
INFO [WatchDog_0] - Phase# Finished Status RunTime(sec) MemoryAllocation(KB)
INFO [WatchDog_0] - 0 FINISHED_OK 18 57683
INFO [WatchDog_0] - ------------------------------** End of Summary **---------------------------
INFO [WatchDog_0] - WatchDog thread finished - total execution time: 18 (sec)
INFO [main] - Freeing graph resources.
INFO [main] - Execution of graph successful ! -
Hello DTaylor,
Thank you for all your testing. I have another one for you if I may:
1) Download the file (graph) attached to this post.EnableJSchLogging.grf
2) Copy the graph into your CloverETL Server sandbox.
3) Run it. The graph enables logging of jsch library which handles sftp connections.
4) If it finishes successfully, please run your original graph with log level set to DEBUG.
5) Send me the whole log file of the original graph run with confidential information omitted as before.
The logging is enabled until CloverETL Server is restarted. If you wanted to disable it without restart, please let me know and we will prepare similar graph for this task.
By the way, if you have troubles with upload of some file types, please zip them into one package and upload the zip.
Thanks for your cooperation. -
Okay, here's the log file. -
Thanks. The log file unfortunately does not contain any explanation of the unsuccessful login. Key authentication is tried as the first option but it fails without any error message.
Could you please send an information about:
1) Operating system of the machine with CloverETL Server
2) Version of CloverETL Server and Designer
3) Version of Java used on the machine with CloverETL Server
4) Version of Java used on the machine with CloverETL Designer
And could you please try to install CloverETL Designer on the same machine as CloverETL Server is installed? Just for the purpose of this test, then you can delete it. Is this new Designer instance able to connect to SFTP? -
1) Windows Server 2008 R2 Enterprise
2) Clover Server: 3.5.0.57/57, Designer: 3.5.0.58
3) Server Java Version: Java 6 (64-bit)
4) Designer Java Version: Java 7 (64-bit)
I can try to install the designer, but that will take a while as I will have to set up the database connections and the Clover projects to run the graph and I've already burned quite a bit of time trying to figure this issue out, so I'm dealing with other priorities now. I've connected to SFTP from this server before, just never with certificate authentication. It's always been username and password with our other SFTP connections. I'll get back to you later after I've been able to install the designer. -
That would be great, thanks.
Ad 3) and 4) The different versions of Java may be the cause this issue. If there were some changes or improvements between versions 6 and 7 in this matter, it would explain why you are able to connect using Designer with Java 7 and unable to connect with Server using Java 6. If it is possible, try to run Server with exactly the same version of Java as your Designer uses.
Regards, -
In this case you should shut down your CloverETL Server. You should also make sure that Server uses the new Java – that means JAVA_HOME property should point to a directory where the new Java is installed. This property can be either set as environmental variable, or directly in your Tomcat Server configuration. -
I finally got more time to work more on this. I installed Java 7 on the Clover Server and pointed Tomcat at it. I still received the same issue.
I installed CloverETL Designer 3.5.0 on the server and set the graph up in a test project with its connections and parameter files. I was able to run and connect to the SFTP without a problem through the designer. So it appears to be something with CloverETL Server itself. -
Are you sure CloverETL Server really accepted your setting and used the correct version of Java? Could you please copy+paste and post the value you see in Configuration -> System Info -> System properties -> java.home?
And the other important thing, is your keystore set correctly in CloverETL Server configuration? Could you please post (without passwords) also your certificate setting for both CloverETL Designer and CloverETL Server and the value of Configuration -> System Info -> JVM -> Input arguments passed to the Java virtual machine?
Thanks,
Please sign in to leave a comment.
Comments 35