I'm trying to replicate this process with CloverETL:
My question is: do I need to create multiple paths for each file or can they be transformed together without explicitly referencing individual files?
I have successfully been able to download all of the zipped files from the website by reading in a CSV of filenames and mapping it through the HTTPConnector, but from there, I don't know how to handle the different metadata for each file without working with each file separately.
I'm hesitant to work with them separately because there are 20-30 files and new ones get generated each year so it would require yearly maintenance that I may not be here for and it would get unwieldy quickly.
I was hoping that http://forum.cloveretl.com/viewtopic.php?t=3622 would help me, but it was written in CLT1, and I haven't been able to replicate it successfully.
Thank you for the help!
- download multiple zip files from a website
- unzip the text files within; each has the same name as the zip file they're in
- the text files get converted into tables within a Postgres database
- then some tables get joined together
My question is: do I need to create multiple paths for each file or can they be transformed together without explicitly referencing individual files?
I have successfully been able to download all of the zipped files from the website by reading in a CSV of filenames and mapping it through the HTTPConnector, but from there, I don't know how to handle the different metadata for each file without working with each file separately.
I'm hesitant to work with them separately because there are 20-30 files and new ones get generated each year so it would require yearly maintenance that I may not be here for and it would get unwieldy quickly.
I was hoping that http://forum.cloveretl.com/viewtopic.php?t=3622 would help me, but it was written in CLT1, and I haven't been able to replicate it successfully.
Thank you for the help!
-
I created an example of how you can achieve your use case. I used a combination of Jobflows and Graphs to generate the metadata, and the database tables dynamically based on the data in each file and then inserted the data based on the table name.
Please sign in to leave a comment.
Comments 3