While I am trying to extract data from Linkedin portal I am getting OAUTH error where as my Keys are working perfectly when I test the same on the Linkedin Portal Testing Tools.
Could not connect to api.linkedin.com
Reason:
server returned HTTP response code: 401 for URL:
https://api.linkedin.com/v1/people/~/network/updates
Please help me on extracting data from the Linkedin.
Could not connect to api.linkedin.com
Reason:
server returned HTTP response code: 401 for URL:
https://api.linkedin.com/v1/people/~/network/updates
Please help me on extracting data from the Linkedin.
-
Hi krsna007,
Did you check response content when 401 is received? Maybe there may be explanation of problem. I that does not help, could you please share your graph with us? Please remove all credentials.
Also, there is blog post with sample attached. Did you check it?
I hope this helps. -
Hi kubosj,
Thanks for your reply. I've taken the example graph which you have mentioned but same issue. I feel some configuration I am missing. Please help me on this.
-----Graph Source ------------
<?xml version="1.0" encoding="UTF-8"?><Graph author="157482" created="Tue Dec 16 20:40:24 IST 2014" guiVersion="3.4.1.P" id="1418742663705" licenseType="Commercial" modified="Wed Dec 17 19:19:54 IST 2014" modifiedBy="157482" name="LinkedInGraph" revision="1.16" showComponentDetails="true">
<Global>
<Metadata id="HTTPConnector_Result">
<Record fieldDelimiter="|" name="HTTP_connector_Result" recordDelimiter="\n" type="delimited">
<Field name="content" type="string"/>
<Field name="outputFilePath" type="string"/>
<Field name="statusCode" type="integer"/>
<Field containerType="map" name="header" type="string"/>
<Field containerType="list" name="rawHeaders" type="string"/>
<Field name="errorMessage" type="string"/>
</Record>
</Metadata>
<Metadata id="Metadata0">
<Record fieldDelimiter="|" name="LinkedInResponse" recordDelimiter="\n" type="delimited">
<Field name="content" type="string"/>
</Record>
</Metadata>
<Metadata id="Metadata1" previewAttachmentCharset="ISO-8859-1">
<Record fieldDelimiter="|" name="Updates" previewAttachmentCharset="ISO-8859-1" recordDelimiter="\n" type="delimited">
<Field name="timestamp" type="string"/>
<Field label="update_key" name="update_key" type="string"/>
<Field label="update_type" name="update_type" type="string"/>
<Field label="is_commentable" name="is_commentable" type="string"/>
<Field label="is_likable" name="is_likable" type="string"/>
<Field label="first_name" name="first_name" type="string"/>
<Field label="last_name" name="last_name" type="string"/>
<Field label="headline" name="headline" type="string"/>
<Field label="picture_url" name="picture_url" type="string"/>
<Field name="update_content" type="string"/>
</Record>
</Metadata>
<Property fileURL="workspace.prm" id="GraphParameter0"/>
<Dictionary/>
</Global>
<Phase number="0">
<Node addInputFieldsAsParametersTo="BODY" consumerKey="78tnom9vvahxws" consumerSecret="K9kYvhLpX9MCTNf4" enabled="enabled" guiName="HTTP connector" guiX="169" guiY="120" id="HTTP_CONNECTOR" oAuthAccessTokenSecret="f60bcc48-f260-43a3-89ed-69e478a8b1c6" requestMethod="GET" responseAsFileName="true" type="HTTP_CONNECTOR" url="http://api.linkedin.com/v1/people/~/network/updates"/>
<Node enabled="enabled" guiName="Parse XML with updates" guiX="367" guiY="120" id="PARSE_XML_WITH_UPDATES" schema="${META_DIR}/updates.xsd" sourceUri="port:$0.content:stream" type="XML_EXTRACT">
<attr name="mapping"><![CDATA[<Mappings>
<Mapping element="updates">
<Mapping element="update" outPort="0"
xmlFields="{}is-commentable;{}is-likable;{}timestamp;{}update-key;{}update-type"
cloverFields="is_commentable;is_likable;timestamp;update_key;update_type">
<Mapping element="update-content" useParentRecord="true"
xmlFields="-"
cloverFields="update_content">
<Mapping element="person" useParentRecord="true"
xmlFields="{}first-name;{}headline;{}last-name;{}picture-url"
cloverFields="first_name;headline;last_name;picture_url">
</Mapping>
</Mapping>
</Mapping>
</Mapping>
</Mappings>
]]></attr>
</Node>
<Node enabled="enabled" fileURL="C:/Users/157482/workspace/LinkedIn/data-out/outpu.csv" guiName="UniversalDataWriter" guiX="600" guiY="120" id="UNIVERSAL_DATA_WRITER" type="DATA_WRITER"/>
<Edge debugMode="true" fromNode="HTTP_CONNECTOR:0" guiBendpoints="" guiRouter="Manhattan" id="Edge2" inPort="Port 0 (input)" metadata="Metadata0" outPort="Port 0 (out)" toNode="PARSE_XML_WITH_UPDATES:0"/>
<Edge fromNode="PARSE_XML_WITH_UPDATES:0" guiBendpoints="" guiRouter="Manhattan" id="Edge3" inPort="Port 0 (in)" metadata="Metadata1" outPort="Port 0 (out)" toNode="UNIVERSAL_DATA_WRITER:0"/>
</Phase>
</Graph> -
Hi krsna007,
Thanks for the graph. There are a few points in your graph that I’d like to go through:- The graph from the blog article is designed in version 3.5, and will not work in version 3.4 due to missing attributes on HTTPConnector component (for more information, see ticket CLO-868). If you do want to use earlier version anyway, you should use a workaround described in mentioned ticket.
- Attribute „Store HTTP response to file“ is turned on in your graph, but if you want to consume this file by XMLExtract, you should use a different Processing type in „File URL“ attribute‘s dialog. Basically you have two options. First, turn off „Store HTTP response to file“ attribute, or use „source“ processing type.
- Regarding the processing type, you shouldn’t use „stream“ processing type, since contcatenated input records wouldn’t form a valid XML file. Instead, you should use „discrete“ type that takes each record/field as a separate source. For more information about processing type, please refer to our article Input port reading in our documentation.
Hope this helps! -
Hi,
I've installed 3.5.3 version tool & try to run the graph from clover examples. Still it is showing up the same connectivity issue.
1) Is this something related to proxy setting in my organization? When I try to run "http://api.linkedin.com/v1/people/~/network/updates" this from URL I am getting message "<message>Unknown authentication scheme</message>".
2) Linkedin is providing OAuth 1.0a token key - Is this compatible with Clover 3.5 or do we need to get OAUTH 2.0 token keys.
I am copying the code here ....please help me out resolve the connectivity issue..
---------------Source-------------------------
<?xml version="1.0" encoding="UTF-8"?>
<Graph author="sedlacek" created="Sat Jan 18 13:35:42 CET 2014" guiVersion="3.5.3" id="1390049580722" licenseType="Evaluation" modified="Tue Dec 23 11:32:38 IST 2014" modifiedBy="157482" name="LinkedIn" revision="1.27" showComponentDetails="true">
<Global>
<Metadata id="Metadata0">
<Record fieldDelimiter="|" name="LinkedInResponse" recordDelimiter="\n" type="delimited">
<Field name="content" type="string"/>
</Record>
</Metadata>
<Metadata id="Metadata1" previewAttachmentCharset="ISO-8859-1">
<Record fieldDelimiter="|" name="Updates" previewAttachmentCharset="ISO-8859-1" recordDelimiter="\n" type="delimited">
<Field name="timestamp" type="string"/>
<Field label="update_key" name="update_key" type="string"/>
<Field label="update_type" name="update_type" type="string"/>
<Field label="is_commentable" name="is_commentable" type="string"/>
<Field label="is_likable" name="is_likable" type="string"/>
<Field label="first_name" name="first_name" type="string"/>
<Field label="last_name" name="last_name" type="string"/>
<Field label="headline" name="headline" type="string"/>
<Field label="picture_url" name="picture_url" type="string"/>
<Field name="update_content" type="string"/>
</Record>
</Metadata>
<GraphParameters>
<GraphParameterFile fileURL="workspace.prm"/>
</GraphParameters>
<Note alignment="1" backgroundColorB="225" backgroundColorG="255" backgroundColorR="255" enabled="true" folded="false" height="48" id="Note0" textColorB="0" textColorG="0" textColorR="0" textFontSize="12" title="LinkedIn REST API usage example" titleColorB="0" titleColorG="0" titleColorR="0" titleFontSize="14" width="599" x="17" y="51"/>
<Dictionary/>
</Global>
<Phase number="0">
<Node consumerKey="78tnom9vvahxws" consumerSecret="K9kYvhLpX9MCTNf4" enabled="enabled" guiName="Get updates from LinkedIn" guiX="17" guiY="109" id="GET_UPDATES_FROM_LINKED_IN" oAuthAccessToken="b2ff8768-2e88-4a3b-9afc-2f7f5af72a62" oAuthAccessTokenSecret="6f51209c-0f11-4bc9-af39-5e4ecd5fc285" type="HTTP_CONNECTOR" url="http://api.linkedin.com/v1/people/~/network/updates"/>
<Node enabled="enabled" guiName="Parse XML with updates" guiX="250" guiY="109" id="PARSE_XML_WITH_UPDATES" schema="${META_DIR}/updates.xsd" sourceUri="port:$0.content:stream" type="XML_EXTRACT">
<attr name="mapping"><![CDATA[<Mappings>
<Mapping element="updates">
<Mapping element="update" outPort="0"
xmlFields="{}is-commentable;{}is-likable;{}timestamp;{}update-key;{}update-type"
cloverFields="is_commentable;is_likable;timestamp;update_key;update_type">
<Mapping element="update-content" useParentRecord="true"
xmlFields="-"
cloverFields="update_content">
<Mapping element="person" useParentRecord="true"
xmlFields="{}first-name;{}headline;{}last-name;{}picture-url"
cloverFields="first_name;headline;last_name;picture_url">
</Mapping>
</Mapping>
</Mapping>
</Mapping>
</Mappings>
]]></attr>
</Node>
<Node enabled="enabled" fileURL="${DATAOUT_DIR}/updates.csv" guiName="Write updates to CSV" guiX="467" guiY="109" id="WRITE_UPDATES_TO_CSV" type="DATA_WRITER"/>
<Edge debugMode="false" fromNode="GET_UPDATES_FROM_LINKED_IN:0" guiBendpoints="" guiRouter="Manhattan" id="Edge1" inPort="Port 0 (input)" metadata="Metadata0" outPort="Port 0 (out)" toNode="PARSE_XML_WITH_UPDATES:0"/>
<Edge debugMode="false" fromNode="PARSE_XML_WITH_UPDATES:0" guiBendpoints="" guiRouter="Manhattan" id="Edge0" inPort="Port 0 (in)" metadata="Metadata1" outPort="Port 0 (out)" toNode="WRITE_UPDATES_TO_CSV:0"/>
</Phase>
</Graph>
--------------------------------------LOG----------------------
INFO [main] - *** CloverETL framework/transformation graph, (c) 2002-2014 Javlin a.s. ***
INFO [main] - Running with CloverETL library version 3.5.3 build#014 compiled 09/12/2014 03:52:48
INFO [main] - Running on 4 CPU(s), OS Windows 7, architecture amd64, Java version 1.7.0_07, max available memory for JVM 1845952 KB
INFO [main] - Loading default properties from: defaultProperties
INFO [main] - Engine plug-ins loaded: com.opensys.clover.lotusdomino, com.opensys.cloveretl.ctlfunction, com.opensys.clover.lookup, org.jetel.lookup, com.opensys.clover.component, org.jetel.bulkloader, org.jetel.license, org.jetel.tlfunction, com.opensys.cloveretl.compiler, com.opensys.clover.validator, org.jetel.sequence, org.jetel.thirdparty, com.opensys.cloveretl.addressdoctor, org.jetel.jdbc, org.jetel.component, org.jetel.connection, org.jetel.profiler, com.opensys.cloveretl.tlfunction, org.jetel.hadoop, org.jetel.engine, com.opensys.cloveretl.quickbase, com.opensys.clover.mongodb, org.jetel.ctlfunction
INFO [main] - License No. : CLP1DOWNXX10475024EX
INFO [main] - Graph definition file: graph/LinkedIn.grf
INFO [main] - Graph revision: 1.27 Modified by: 157482 Modified: Tue Dec 23 11:32:38 IST 2014
INFO [main] - Checking graph configuration...
INFO [main] - Graph configuration is valid.
INFO [main] - Graph initialization (LinkedIn)
INFO [main] - Initializing phase 0
INFO [main] - Phase 0 initialized successfully.
INFO [WatchDog_0] - Job parameters: PROJECT=., CONN_DIR=${PROJECT}/conn, DATAIN_DIR=${PROJECT}/data-in, DATAOUT_DIR=${PROJECT}/data-out, DATATMP_DIR=${PROJECT}/data-tmp, GRAPH_DIR=${PROJECT}/graph, JOBFLOW_DIR=${PROJECT}/jobflow, LOOKUP_DIR=${PROJECT}/lookup, META_DIR=${PROJECT}/meta, PROFILE_DIR=${PROJECT}/profile, SEQ_DIR=${PROJECT}/seq, TRANS_DIR=${PROJECT}/trans
INFO [WatchDog_0] - Starting up all nodes in phase [0]
INFO [WatchDog_0] - Successfully started all nodes in phase!
ERROR [WatchDog_0] - Component [Get updates from LinkedIn:GET_UPDATES_FROM_LINKED_IN] finished with status ERROR. (Out0: 0 recs)
Connection to http://api.linkedin.com refused
Connection refused: connect
ERROR [WatchDog_0] - Error details:
org.jetel.exception.JetelRuntimeException: Component [Get updates from LinkedIn:GET_UPDATES_FROM_LINKED_IN] finished with status ERROR. (Out0: 0 recs)
at org.jetel.graph.Node.createNodeException(Node.java:566)
at org.jetel.graph.Node.run(Node.java:542)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://api.linkedin.com refused
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.jetel.component.HttpConnector.buildAndSendRequest(HttpConnector.java:1908)
at org.jetel.component.HttpConnector.process(HttpConnector.java:1875)
at org.jetel.component.HttpConnector.executeForRecord(HttpConnector.java:2019)
at org.jetel.component.HttpConnector.execute(HttpConnector.java:1986)
at org.jetel.graph.Node.run(Node.java:505)
... 1 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:69)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
... 13 more
INFO [WatchDog_0] - Execution of phase [0] finished with error - elapsed time(sec): 1
ERROR [WatchDog_0] - !!! Phase finished with error - stopping graph run !!!
INFO [WatchDog_0] - -----------------------** Summary of Phases execution **---------------------
INFO [WatchDog_0] - Phase# Finished Status RunTime(sec) MemoryAllocation(KB)
INFO [WatchDog_0] - 0 ERROR 1 29481
INFO [WatchDog_0] - ------------------------------** End of Summary **---------------------------
INFO [WatchDog_0] - WatchDog thread finished - total execution time: 1 (sec)
INFO [main] - Freeing graph resources.
ERROR [main] -
---------------------------------------------- Error details ------------------------------------------------
Component [Get updates from LinkedIn:GET_UPDATES_FROM_LINKED_IN] finished with status ERROR. (Out0: 0 recs)
Connection to http://api.linkedin.com refused
Connection refused: connect
-------------------------------------------------------------------------------------------------------------
ERROR [main] - Execution of graph failed ! -
Hi,
I have just tried to run the graph you posted and it works just fine. I did not modify the graph in any way.
It seems that you created the graph ok, it is working. If it does not work on your side, I assume you have some kind of network connectivity issues. (firewall, proxy, security policy in your company, network delays, ...) If you resolve them, you should be able to connect. Are you able to run the graph from a different network to confirm my theory?
Regarding your second question, the four properties you are setting are everything you need to use OAuth 1.0a. They were created exactly for this purpose. You could use OAuth 2.0 too if LinkedIn offers it but settings are a bit more complex in this case and the four properties alone are not enough to create the connection. If you want to see an example of OAuth 2.0 usage, see this Google Analytics blog post.
Regards,
Please sign in to leave a comment.
Comments 5