Hi,
I want to dynamically modifiy the metadata associated with edges.
I hoped it could be done in DataRecordTransform init() method, but :
- I can modify the metadata associated with the component (via the metadata field)
- the modifications I make are not really taken into account, as the DataRecord itself it created before that based on a metadata object. Specifically, it holds the fields in an array that has the size of the number of field BEFORE I changed the metadata
SO, here is my question :
Is there a hook where I could dynamically change the metadatas in the graph before any datarecord is created ?
(does it seem like a reasonnable thing to do ?)
Franck
I want to dynamically modifiy the metadata associated with edges.
I hoped it could be done in DataRecordTransform init() method, but :
- I can modify the metadata associated with the component (via the metadata field)
- the modifications I make are not really taken into account, as the DataRecord itself it created before that based on a metadata object. Specifically, it holds the fields in an array that has the size of the number of field BEFORE I changed the metadata
SO, here is my question :
Is there a hook where I could dynamically change the metadatas in the graph before any datarecord is created ?
(does it seem like a reasonnable thing to do ?)
Franck
-
Hi Martin,
well, actually modifying Metadata in DataRecordTransform.init() IS possible, as you have access to the targetMetadata and surceMetadata fields, which are not private (maybe they should ?).
But it is true that these modifications are useless, since DataRecord itself is created before that...
Now let me explain what I am trying to achieve... :
- I have many graph where I read a few tables (10 or so) from different databases, join them together, and finally insert into a datamart, choosing some information from each input table.
- Our first try was to do it "on the fly", that is take the fields that interest us on each join in the transformation class. But this proves to be quite difficult to document and to maintain.
- So the idea is this one : create a generic appender class that will :
1) append the metadata of joined table (port 1) to the main stream (port 0) dinamically
2) simply copy each joined field to the output record, 1 to 1, without any real transformation
We can then do the real work in a single transformation class, at the end of the process. This way joiners simply do what they are meant to, and a final transformer does the real work.
Right now I have a SimpleInOutAppender that copies from input ports to one output record, but I have to create the "master" metadata (the one that has all fields from all joined tables) by hand. I would like to make this automated.
... don't hesitate to ask more information ,as I am not sure this is very clear :)
Franck -
Hello Franck!
Unfortunately dynamically modifying of metadata in DataRecordTransform.init() is not possible. The parameter of this method is only input information for your custom initialization of the transformation. Actually for now I don't see any sense of these changes. Can you explain me what is your intention?
Only chance to change metadata dynamically is in initialization time of whole graph (before graph running - see runGraph class for more information or contact me again with more specific description of your issue).
Best regards, Martin.
Please sign in to leave a comment.
Comments 2