Hi,
Working on Clover ETL is kind of interesting experience, But few problems are haunting me for long now..
Here is the description of what I am done till now and what issue I am facing.
1) I have created a graph which process Twitter data using REST API.
2) Depending on a variable (twitter user name) in the data-generator component, I call the REST API of twitter.
Everything is working properly fine here, now here is problem domain.
1) I am trying to convert this to a jobflow like looping same graph again and again for variable (which is user name) and I would be getting this variable list either from database or from some text file.
2) I am not able to solve the problem as I am not able to understand, how to use the variable in job flow. Though there are mentioned in the user mannual (data dictionary and getinputvariable components) but couldn't find way to use them to solve my problem, I feel I am looking for some example of jobflow.
Please help me on this...
Regards
Sandeep
Working on Clover ETL is kind of interesting experience, But few problems are haunting me for long now..
Here is the description of what I am done till now and what issue I am facing.
1) I have created a graph which process Twitter data using REST API.
2) Depending on a variable (twitter user name) in the data-generator component, I call the REST API of twitter.
Everything is working properly fine here, now here is problem domain.
1) I am trying to convert this to a jobflow like looping same graph again and again for variable (which is user name) and I would be getting this variable list either from database or from some text file.
2) I am not able to solve the problem as I am not able to understand, how to use the variable in job flow. Though there are mentioned in the user mannual (data dictionary and getinputvariable components) but couldn't find way to use them to solve my problem, I feel I am looking for some example of jobflow.
Please help me on this...
Regards
Sandeep
-
Hi Sundeep,
I guess you mean dictionaries when you talk about variables. As you can see in our documentation, dictionary entries can be retrieved using CTL expressions. In GetJobInput component, you may find these values either in the left pane or on Dictionary tab in the middle bottom pane of the dialog – similar approach (Dictionary tab) is used in other components with graphical transform editor.
If there is a dictionary entry in a Graph you execute from JobFlow, you may simply map these values from source file’s dictionary to target file’s dictionary in Transform editor of ExecuteGraph‘s Input mapping attribute.
ExecuteGraph_InputMapping.png
Another use is also in File URL attributes of some components (such as readers and writers). More information about this is described in our documentation in chapter Using the Dictionary in a Graph.
Hope this helps. -
Hi,
Thanks,
I feel I am really dumb.. still can't figure out the solution... I am attaching a image to explain you whats going in my mind.. :(
Hope this will provide you information about my problem..
Regards
Sandeep -
Hi Sandeep,
You can do this within one simple graph. It may look like as follows:
DBHttpConnectorDB.png
It can be summed up into the following steps:- Create DB connection
- Add DBInputTable into the graph. In the component’s dialog, select recently created connection and add the SQL query into appropriate attribute. The output metadata has to be set accordingly.
- Add HTTPConnector component. For more information about quering Twitter in CloverETL, see the article on our blog: http://www.cloveretl.com/blog/querying- ... cloveretl/
- Data that you have retrieved from Twitter will be sent to the output port, so that you can insert them into the DB using DBOutputTable.
For more information about aforementioned components, please refer to our documentation.- DBInputTable: http://doc.cloveretl.com/documentation/ ... table.html
- DBOutputTable: http://doc.cloveretl.com/documentation/ ... table.html
- HTTPConnector: http://doc.cloveretl.com/documentation/ ... ector.html
Hope this helps. -
Hi,
Thanks so much!!
Actually you have explained very well, and I have actually as mentioned in your reply post.
But my problem is about how to supply tweet-ids one by one to http conector? Because SQL query brings all data at once in the resultset.
If I have a DB table Tweeter with two columns Tweet-screename and id, and table have got 5 values like
tommy,1
robbie,2
bobby,3
sandy,4
jackie,5
so DBInputtable executes select query it's like "select * from Tweeter". which fetches all the data.
what I need is fetching Tommy at first and then supplying to http-connector and retrieving tweet data and then robbie and so on.
How to fetch these values one by one and providing the values to httpconnector as a variable is we need to find..
I hope I explained it further properly..
Regards
Sandeep -
Hi Sandeep,
Jan already answered your question in this post: viewtopic.php?f=4&t=6866#p11113
JobFlows are available on CloverETL Server+, but you probably have only Designer trial. In that case you should use solution from post viewtopic.php?f=4&t=6866#p11170
Thank you for understanding.
Please sign in to leave a comment.
Comments 5