Sorry if this has been covered, but I searched the forum for a bit prior to posting.
I am evaluating Clover and very new to using it. I am trying to create a jobflow that is a series of graphs, but I am unable to share outputs between them.
I don't want to use temp files, as I feel it will slow down the processing. ( I have not evaluated this statement yet though. ) I have tried to SetJobOutput and dump the contents to a dictionary, but I only get the last record in the next step.
Question being, is it possible to share the output of records from GraphA as the input of GraphB, without writing to disk? If not, can the output of GraphA be used in additional steps in the jobflow, without writing to disk?
I am evaluating Clover and very new to using it. I am trying to create a jobflow that is a series of graphs, but I am unable to share outputs between them.
I don't want to use temp files, as I feel it will slow down the processing. ( I have not evaluated this statement yet though. ) I have tried to SetJobOutput and dump the contents to a dictionary, but I only get the last record in the next step.
Question being, is it possible to share the output of records from GraphA as the input of GraphB, without writing to disk? If not, can the output of GraphA be used in additional steps in the jobflow, without writing to disk?
-
Hi, kevinzona,
Right now, I would suggest you to use temp files. But we are aware of this use case and we would like to implement the solution for it in the version 3.5.
There are a few ways, how to solve it now - using dictionary (in the other way you used it) or named pipe on unix based OS but they are quite complicated to use and not designed for this purpose. On the other hand, using temp files has also a few advantages. (better troubleshooting, ability to work with big data sets, ...)
Best regards,
Please sign in to leave a comment.
Comments 1