We have recently been tasked with the following high level requirements:
1.) Need system to monitor flat files, know when they have changed.
2.) When a file being monitored has changed, parse the file, looking for specified strings.
3.) Don't re-parse a file, meaning if a file being monitored is appended to, then only parse the new lines.
4.) Write the parsed strings to a database.
We were going to build this in-house, however we thought about looking around at some of the ETL projects to see if we could leverage an existing tool.
Can anyone tell me if Clover.ETL has this capability?
Thanks!!
1.) Need system to monitor flat files, know when they have changed.
2.) When a file being monitored has changed, parse the file, looking for specified strings.
3.) Don't re-parse a file, meaning if a file being monitored is appended to, then only parse the new lines.
4.) Write the parsed strings to a database.
We were going to build this in-house, however we thought about looking around at some of the ETL projects to see if we could leverage an existing tool.
Can anyone tell me if Clover.ETL has this capability?
Thanks!!
-
Hi,
Clover does not monitor changes in files, that would have to be implemented externally (i.e. cron & bash). Clover will support "incremental reading" in the next version (engine 2.5) which will be released soon. This incremental reading allows you to continue reading a file where you finished the previous time. The files can be parsed etc based on a wide range of logic, custom transformations etc. Clover supports output to databases.
Hope this helps,
Jaro
Please sign in to leave a comment.
Comments 1