I have a file that I need to access through a universal data reader where the file is updated every Thursday and I would like to be able to access the file on any other day without the need to specify the specific date for "last thursday" within a parameter (which I'm currently doing).
So, I've set up a data generator which figures out what the date is for last thursday and sends it through the output port to the data reader.
Here is the data generator code:
This is generating the correct filename that works perfectly within the 'File URL' box of the data reader if I enter the generated filename in the box manually, but when passed through the input port it will not download the file and process its data.
The generated filename is: zip:(ftp://username:password@DOMAIN.COM/FOLD ... 222013.ZIP)#FILE08222013.CSV
I've tried setting the Port and field to all of the following and it doesn't work:
port:$0.last_thursday_filename:discrete
port:$0.last_thursday_filename:source
port:$0.last_thursday_filename:stream
I get the following which might mean something?
ERROR [WatchDog] - Node NODENAME finished with status: Component has finished and input port 0 still contains some unread records.
How do I get the dynamically generated file-name to pass into the data reader and have it actually download and process the file?
FYI - I've tried doing this through a dictionary but I couldn't get that to work either.
FYI - I wish CloverETL had the ability to generate a date based on something like the following: str2date('last thursday',"MMddyyyy"); PHP has this functionality and I use it all the time. :-) With this function I can simply use that within the file-name like: zip:(ftp://username:password@DOMAIN.COM/FOLDER/FILE`str2date('last thursday',"MMddyyyy")`.ZIP)#FILE`str2date('last thursday',"MMddyyyy")`.CSV
So, I've set up a data generator which figures out what the date is for last thursday and sends it through the output port to the data reader.
Here is the data generator code:
//#CTL2
string last_thursday;
integer day_of_week;
integer days_from_thurs;
// Generates output record.
function integer generate() {
day_of_week = str2integer(date2str(today(), "joda:e"));
if(day_of_week > 4) days_from_thurs = day_of_week - 4;
else days_from_thurs = day_of_week + 4;
last_thursday = date2str(dateAdd(today(), -days_from_thurs, day),"MMddyyyy");
$out.0.last_thursday_filename = "zip:(ftp://${FTP_USERNAME}:${FTP_PASSWORD}@DOMAIN.COM/FOLDER/FILE"+last_thursday+".ZIP)#FILE"+last_thursday+'.CSV';
return OK;
}This is generating the correct filename that works perfectly within the 'File URL' box of the data reader if I enter the generated filename in the box manually, but when passed through the input port it will not download the file and process its data.
The generated filename is: zip:(ftp://username:password@DOMAIN.COM/FOLD ... 222013.ZIP)#FILE08222013.CSV
I've tried setting the Port and field to all of the following and it doesn't work:
port:$0.last_thursday_filename:discrete
port:$0.last_thursday_filename:source
port:$0.last_thursday_filename:stream
I get the following which might mean something?
ERROR [WatchDog] - Node NODENAME finished with status: Component has finished and input port 0 still contains some unread records.
How do I get the dynamically generated file-name to pass into the data reader and have it actually download and process the file?
FYI - I've tried doing this through a dictionary but I couldn't get that to work either.
FYI - I wish CloverETL had the ability to generate a date based on something like the following: str2date('last thursday',"MMddyyyy"); PHP has this functionality and I use it all the time. :-) With this function I can simply use that within the file-name like: zip:(ftp://username:password@DOMAIN.COM/FOLDER/FILE`str2date('last thursday',"MMddyyyy")`.ZIP)#FILE`str2date('last thursday',"MMddyyyy")`.CSV
-
Hi twhitney,
The URL like the following works ok on my side.zip:(ftp://account:password@server/path/file.zip)#file.csv
In UniversalDataReader, you just have to set File URL to port:$0.last_thursday_filename:source
Please, be careful about case sensitivity of URL (file.CSV is not the same file as file.csv) and if the aforesaid does not help, please send also your graph. (without any credentials, of course)
Regarding your idea about "last thursday" dates, I agree it would be a nice feature. Thank you for your feedback. I have started an issue in our bug tracking system, https://bug.javlin.eu/browse/CLO-1727
Best regards, -
Hi,
Thank you for your response. The problem is that this is exactly how I have it set up and it doesn't want to work...
Here is the debug log:ERROR [WatchDog] - Graph execution finished with error
ERROR [WatchDog] - Node BTTITLE finished with status: Component has finished and input port 0 still contains some unread records.
ERROR [WatchDog] - Node BTTITLE error details:
INFO [WatchDog] - [Clover] Post-execute phase finalization: 0
INFO [WatchDog] - [Clover] phase: 0 post-execute finalization successfully.
INFO [WatchDog] - Execution of phase [0] finished with error - elapsed time(sec): 6
ERROR [WatchDog] - !!! Phase finished with error - stopping graph run !!!
INFO [WatchDog] - -----------------------** Summary of Phases execution **---------------------
INFO [WatchDog] - Phase# Finished Status RunTime(sec) MemoryAllocation(KB)
INFO [WatchDog] - 0 ERROR 6 48887
INFO [WatchDog] - 1 N/A 0 0
INFO [WatchDog] - 2 N/A 0 0
INFO [WatchDog] - 3 N/A 0 0
INFO [WatchDog] - 4 N/A 0 0
INFO [WatchDog] - 5 N/A 0 0
INFO [WatchDog] - ------------------------------** End of Summary **---------------------------
INFO [WatchDog] - WatchDog thread finished - total execution time: 6 (sec)
INFO [main] - Freeing graph resources.
ERROR [main] - Execution of graph failed ! -
Hello again,
I am sorry but I can not solve this issue from the log you sent. I would like to ask you for your graph or at least the whole run log with log level set to DEBUG. You can get it by clicking on Run -> Run Configurations... and setting log level from default INFO to DEBUG. Then run your graph again please and send the log to me.
Best regards, -
I have attached a simplified version of the graph for you. -
I appear to have found a work-around for this issue.
As you mentioned, the graph looked fine. The problem, I think, had to do with how many components were running in the same phase. In my graph I am pulling data from about 10 different FTP locations and joining them all together and for some reason the component I was having problems with would not pull in any data. However, if I ran the graph separately without anything else running it would work fine so I changed the phases so that this particular component would run first, then it would pull in all of the others and now it seems to be working properly.
Please sign in to leave a comment.
Comments 6