Ok here is what I'm trying to do. I'm new at this so some direction would be appreciated.
I'm using a DBInputTable to fetch the MAX(UPDATE_DATE) from a table in my reporting database where I want to insert new records.
I then want to use that value to filter another DBInputTable in another database such that I only select rows that are newer than the last update in my reporting table.
In other words I want to set up a "pump" where every time it runs it inserts new or updated records in one database into another. The insert part is easy but I have no idea how to filter based on a variable.
I imagine this should be possible but have no idea how to go about it. The interface gives no clues.
I'm using a DBInputTable to fetch the MAX(UPDATE_DATE) from a table in my reporting database where I want to insert new records.
I then want to use that value to filter another DBInputTable in another database such that I only select rows that are newer than the last update in my reporting table.
In other words I want to set up a "pump" where every time it runs it inserts new or updated records in one database into another. The insert part is easy but I have no idea how to filter based on a variable.
I imagine this should be possible but have no idea how to go about it. The interface gives no clues.
-
Dear Neptun,
I think "Incremental reading" feature could solve your problem. See http://doc.cloveretl.com/documentation/ ... table.html especially "Incremental file" and "Incremental key" properties.
Basically you can remember maximum value of last run (timestamp in your case) and then use it in next query.
I hope this helps.
Please sign in to leave a comment.
Comments 1