Dear support,
I have an incoming file (40 columns) where I read it using file reader, I will store the file with with probably updated values and new rows in an existing file what is the best way to achieve this.
For example, the file came first time which i am reading, all the records are new, so I stored them with my required columns in a csv file with a date column as Today's date.
New file came in, it has 10 records, when I match I already have 5 records matched, now I wanted to store all of these 10 records with the previous ones.
It is better to append the csv file or shall I create a whole new file with previous and new records and then save it?
Also, does CloverDX provides any database sort of support like Alteryx has yxdb format which is the most efficient file type for reading and writing in Alteryx because it has no size limit, is compressed for maximum speed, and includes additional metadata that references the source of the data and how the data was created.
Kind regard
-
Hi,
Let me clarify what you are trying to achieve. Based on the information provided, here is an example:
You have File A stored and you receive File B, which you want to merge with File A. Could you specify the desired output result you are aiming for?
Appending the data to File A using the FlatFileWriter seems more suitable in this situation, assuming our understanding is correct. Both approaches are possible, but keep in mind that creating a new file with all previous and current data will impact system memory usage.
Regarding your third point, CloverDX has a native data format that includes metadata and stores data in binary. Here is a link to our documentation about the CloverDataWriter, which contains useful information about the format, examples, and use cases for appending and creating new files.
Best regards, Tom.
Please sign in to leave a comment.
Comments 1