Is it possible to have a UniversalDataReader (or any Reader for that matter), point to a file folder and read in all the contained files? I know you can highlight multiple files, but can it be set up for a folder whose contents are periodically updated?
Additionally, can you do incremental reading where the reader only looks at files that have been added since the last run of the graph? Here I assume that there is a timestamp in either the filename or a data field, which can be saved to a separate "key" file.
Thanks in advance for the help!
Additionally, can you do incremental reading where the reader only looks at files that have been added since the last run of the graph? Here I assume that there is a timestamp in either the filename or a data field, which can be saved to a separate "key" file.
Thanks in advance for the help!
-
Hello,
to point a Reader to a group of files, use ? and * wild-cards. So for reading all the files from the selected folder, your fileURL attribute should look like fileURL="my_folder/*".
For reading only the files, that have not been read yet use incremental reading feature: set incrementalKey and incrementalFile attributes, eg.:incrementalFile="${DATATMP_DIR}/inc_my_grf.prop"
incrementalKey="my_reader"
After the graph execution, the file specified in incrementalFile attribute contains the names of the files with numbers of records, that have been read. So in the next graph execution there are read only new files or the records, that has been added to existing files since the last execution. But such reading is, of course, slower then reading without any checking, so if it is possible rather remove read files (eg. with SystemExecute component), than use incremental reading.
Please sign in to leave a comment.
Comments 2