I have some event file listeners that trigger on new files added to a folder. If the graph runs into an issue, files won't be processed. What is the best way to go back and process those missed files once the graph has been fixed? Is there a way to reset the listener to go back and check for new files going back a certain number of days or hours?
-
Hi Brandon,
There is no option on File event listener which would cause it to go "back". You have two options:
- change the timestamp of those files you want to re-process. For example, using UNIX touch command. That would "draw" listener's attention and it would trigger the processing.
- create a job flow which would pick your files and send them for processing to your standard file transformation graph - very much like the event listener is doing, but you would control which files get picked
Example of simple jobflow picking files & sending them to be process by transformation graph (which would be the graph/jobflow called by your listener):
The filter can have condition like this to ensure you only pick certain file types created/last modified withing time interval:
$in.0.lastModified >= 2024-05-20 && $in.0.lastModified <= 2024-05-23 &&
!$in.0.isDirectory && getFileExtension($in.0.name)=="xlsx"Those "hardcoded" dates can be replace by parameters - thus this becoming unversal "re-run" for your files if processing by event listener fails.
Please sign in to leave a comment.
Comments 1