Customer Portal

UniversalDataReader SFTP issue

  1. CloverDX Customer Portal
  2. Forum
  3. UniversalDataReader SFTP issue
Avatar
gch
  • Answered
Follow
We are trying to read a file from a SFTP server (hosted by Amazon). From our linux server we can login to sftp and upload/download files. However from clover when we are trying to read a file it says file is unreachable. In fact we can reach the file just fine from sftp prompt

File is unreachable: sftp://[username]:***@[servername]/home/oraclesftp/StatusLink.csv
Cannot obtain connection input stream for URL 'sftp://[username]:***@[servername]/home/oraclesftp/StatusLink.csv'. Make sure the URL is valid.
Session.connect: java.io.IOException: End of IO Stream Read
17:55:49,247 ERROR FusionConnectWatchDog_0 - 83423 - Error details:
org.jetel.exception.JetelRuntimeException: Component [ReadStatusURLLink:READ_STATUS_URLLINK] finished with status ERROR. (Out0: 0 recs)
at org.jetel.graph.Node.createNodeException(Node.java:636)
at org.jetel.graph.Node.run(Node.java:603)
at java.lang.Thread.run(Thread.java:744)
Caused by: Element [READ_STATUS_URLLINK:ReadStatusURLLink]-Component pre-execute initialization failed.
at org.jetel.graph.Node.run(Node.java:555)
... 1 more

Comments 8

Sort by Date Votes
  • Avatar
    Vladimir Barton
    0
    Comment actions Permalink
    Hello gch,
    in order to troubleshoot this issue, I would suggest focusing on the following items first before we dig any deeper:

    • Try adding the proper port number to the URL:

      sftp://[username]:***@[servername]:[port]/home/oraclesftp/StatusLink.csv


    • Try checking the URL for any special character that might need escaping. For example, let’s say you are connecting to the server with a user named ‘user#001’ and password ‘pA22w0rd@’. This URL will not work properly:

      sftp://user#001:pA22w0rd@@LinuxServer:2222/home/oraclesftp/StatusLink.csv


      As some of the special characters (e.g. # or @) will get misinterpreted, they need to be escaped so the URL needs to be changed to:

      sftp://user%23001:pA22w0rd%40@LinuxServer:2222/home/oraclesftp/StatusLink.csv



    If this did not help, please get back to us with more details, namely:

    • You mentioned that the SFTP server is hosted by Amazon. Is it an EC2 instance we are talking about?

    • Could you provide us with more details about this SFTP server software information (name, version)?

    • Can you give us some specs about your Linux server as well (sort of the Linux distribution, version)?

    • What is the size of the StatusLink.csv file that you are trying to transfer?

    • Which version of CloverETL Designer and Server are you using?


    Regards,
  • Avatar
    gch
    0
    Comment actions Permalink
    1. Port is default 22.
    2. Password doesn't contain any special character.

    3. You mentioned that the SFTP server is hosted by Amazon. Is it an EC2 instance we are talking about?
    Yes

    4. Could you provide us with more details about this SFTP server software information (name, version)?
    Ubuntu 16.04.3 LTS
    Release 16.04
    Codename: xenial

    Open SSH
    1:7.2p2-4ubuntu2.2

    5. Can you give us some specs about your Linux server as well (sort of the Linux distribution, version)?
    2014 x86_64 x86_64 x86_64 GNU/Linux

    6. What is the size of the StatusLink.csv file that you are trying to transfer?
    5 KB

    7. Which version of CloverETL Designer and Server are you using?
    4.3 community edition

    We are seeing this error in the SFTP server log when Clover calls it

    Nov 13 22:31:50 ip-172-31-28-244 sshd[11209]: Accepted password for oraclesftp from 35.160.99.136 port 47261 ssh2
    Nov 13 22:31:50 ip-172-31-28-244 sshd[11209]: pam_unix(sshd:session): session opened for user oraclesftp by (uid=0)
    Nov 13 22:31:50 ip-172-31-28-244 systemd-logind[1141]: New session 8 of user oraclesftp.
    Nov 13 22:32:33 ip-172-31-28-244 sshd[11243]: fatal: mm_answer_moduli: bad parameters: 2048 2048 1024
    Nov 13 22:32:33 ip-172-31-28-244 sshd[11244]: fatal: mm_answer_moduli: bad parameters: 2048 2048 1024
    Nov 13 22:32:33 ip-172-31-28-244 sshd[11248]: fatal: mm_answer_moduli: bad parameters: 2048 2048 1024
    Nov 13 22:32:33 ip-172-31-28-244 sshd[11247]: fatal: mm_answer_moduli: bad parameters: 2048 2048 1024
    Nov 13 22:32:33 ip-172-31-28-244 sshd[11251]: fatal: mm_answer_moduli: bad parameters: 2048 2048 1024

    When we call SFTP from linux command line it is working fine.
  • Avatar
    rickymartin06
    0
    Comment actions Permalink
    Are you receiveng the path thru metadata? if so try to put your file directly in your DataReader FileURL, if it works you are having problems with your data string remember if you are receiveing path from meta data you need to use something like this port:$0.URL:source.

    Also double check your password doesnt contain an @@@@. i spent like 2 days on an error like this.
  • Avatar
    gch
    0
    Comment actions Permalink
    Metadata is not the problem. If we use a different SFTP server with the same path, it works fine and I have simplied the password like "abcd" and still it doesn't work.

    The issue with openssh new version on the SFTP server software. Based on the error message it seems they changed something in new version of SSH. Was wondering if clover can set some parameter to alleviate the problem
  • Avatar
    gch
    0
    Comment actions Permalink
    Found the root cause of the problem. OpenSSH has disabled Key Exchange Algorithm diffie-hellman-group1-sha1. It is important to notice that this algorithm is not considered safe anymore, but Open SSH supports it.

    Clover somehow can't handle the new key exchange patterns of the OpenSSH. Once we disable all other key exchange patterns and kept the diffie-hellman-group1-sha1 active, clover can read the file. However this is a security hole and sys admins are not ready to keep this pattern available.

    Is there a fix available quickly?
  • Avatar
    Pedro Vazquez Rosario
    0
    Comment actions Permalink
    Hi gch,

    I spoke with our developers and they strongly feel they can find a solution to this problem, but they need more details first. Could you please answer the following questions:

    • Java version

    • CloverETL version
  • Avatar
    gch
    0
    Comment actions Permalink
    First thank you for looking into this. Version for java is JDK7 and Clover version is 4.3
  • Avatar
    Pedro Vazquez Rosario
    0
    Comment actions Permalink
    I'm attaching a graph with CustomJavaComponent, which will print out the supported key exchange schemes of the SFTP. When the graph runs, it will enable verbose SFTP logging at DEBUG log level on the Server, until restarted. Afterwards, you should execute your graph that reads from SFTP with DEBUG log level enabled and send us the log of the execution. To enable DEBUG log level on Designer, please go to Run - Run Configurations - Log Level - DEBUG.

    SFTP_EnableLogging.grf

Please sign in to leave a comment.

Didn't find what you were looking for?

New post