I have a flat file that where each record has a timestamp. I need to process only the records in the flat file that that have a timestamp greater than the last time I processed the file. I can determine when I last processed the file by running a SQL statement.
select max(timestamp) from mytable;
How do I use the max timestamp value in a FILTER node?
Is there a better way?
select max(timestamp) from mytable;
How do I use the max timestamp value in a FILTER node?
Is there a better way?
-
You can't set property value while the graph is running - well, you could but that would require some Java code to be executed. Nonetheless, the property value is resolved upon graph startup, so changing its value during graph run yeilds nothing.
Two options how you can do it:
i) split the graph into two, prepare some property file in first - using DB_INPUT_TABLE, REFORMAT,WRITER and then let second graph read it from property file
ii) use lookup table to which you can store the value under some key in phase 1 (use LOOKUP_READER_WRITER) and then in phase 2 use the lookup just to get this value into filter.
There are probably some other options , but I would personally use the first one.
David. -
The twist is that I don't know the MY_TIME_PARAMETER when I start to run the graph, so I can't put it on the command line or in a file.
However, I can read it from the DB during the execution of the graph.
If I used a DB_INPUT_TABLE node to read the timestamp value from the DB, and pass the output to a REFORMAT node, can I programmatically set a Property value that can be subsequently used in a later phase? -
Hello !
You may use EXT_FILTER and parameter. Parameter can be read from file or defined on command line. Starting with Clover version 2.2 parameters can be also obtained from environment parameters.
Example ext_filter condition:
$timestamp_field>${MY_TIME_PARAMETER}.
Parameter should be normal string (not quoted) in following format YYYY-MM-DD hh:mm:ss. Or you may use conversion function -> str2date() if your date valu has different format.
David.
Please sign in to leave a comment.
Comments 3