Can metadata be concatenated or dynamically altered?

codemonkee

• September 22, 2010 00:00

• Answered

Follow

Our graph reads from a user supplied input file. This file may have a different number of fields for each graph execution. We modify the Data_Reader edge to handle these different files. Each record from the file is processed and new data is added to the original. Can this be accomplished without having to also modify the metadata that receives the input data and process data?

i.e.
Can something like edge3 := edge1 + edge2 be done, where edge3 is of a variable length?

• 

  avackova

  • September 24, 2010 12:30

  0

  Comment actions Permalink

  Hello,
  this is not possible. The only way to have variable metadata on the one edge is to use dynamic metadata. Such metadata is created dynamically according to supplied sql query. It is also possible to create a new CloverETL plugin, with the type that implements org.jetel.database.IConnection. Then you would have your own dynamic metadata.
  
  Another workaround is to create the graph, that creates the new metadata file from the input metadata and the additional metadata. Something like that:

  
  <?xml version="1.0" encoding="UTF-8"?>
  <Graph author="avackova" created="Fri Sep 24 11:15:57 CEST 2010"  id="1285328540248"  modified="Fri Sep 24 14:02:09 CEST 2010" modifiedBy="avackova" name="metadata_creator" revision="1.8">
  <Global>
  <Metadata id="Metadata0" >
  <Record fieldDelimiter="|" name="row" recordDelimiter="\n" type="delimited">
  <Field name="row" type="string"/>
  </Record>
  </Metadata>
  <Property fileURL="workspace.prm" id="GraphParameter0"/>
  </Global>
  <Phase number="0">
  <Node  id="CONCATENATE0" type="CONCATENATE"/>
  <Node  fileURL="${META_DIR}/meta1fmt" id="DATA_READER0" type="DATA_READER"/>
  <Node  fileURL="${META_DIR}/meta2.fmt" id="DATA_READER1" type="DATA_READER"/>
  <Node enabled="enabled"  id="EXT_FILTER0" type="EXT_FILTER">
  <attr name="filterExpression"><![CDATA[$0.row ~= "<Record.*>"]]></attr>
  </Node>
  <Node id="EXT_FILTER1" type="EXT_FILTER">
  <attr name="filterExpression"><![CDATA[$0.row ~= "<Record.*>"]]></attr>
  </Node>
  <Node fileURL="${META_DIR}/meta.fmt" footer="&lt;/Record&gt;"  header="&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;&#10;&lt;Record fieldDelimiter=&quot;;&quot; name=&quot;mta_name&quot; recordDelimiter=&quot;\n&quot; skipSourceRows=&quot;0&quot; type=&quot;delimited&quot;&gt;&#10;" id="STRUCTURE_WRITER0" mask="$row" type="STRUCTURE_WRITER"/>
  <Edge fromNode="CONCATENATE0:0"  id="Edge5" inPort="Port 0 (Body port)" metadata="Metadata0" outPort="Port 0 (out)" toNode="STRUCTURE_WRITER0:0"/>
  <Edge fromNode="DATA_READER0:0" id="Edge1" inPort="Port 0 (in)" metadata="Metadata0" outPort="Port 0 (output)" toNode="EXT_FILTER0:0"/>
  <Edge fromNode="DATA_READER1:0" id="Edge2" inPort="Port 0 (in)" metadata="Metadata0" outPort="Port 0 (output)" toNode="EXT_FILTER1:0"/>
  <Edge fromNode="EXT_FILTER0:0" id="Edge3" inPort="Port 0 (in)" metadata="Metadata0" outPort="Port 0 (accepted)" toNode="CONCATENATE0:0"/>
  <Edge fromNode="EXT_FILTER1:0" id="Edge4" inPort="Port 1 (in)" metadata="Metadata0" outPort="Port 0 (accepted)" toNode="CONCATENATE0:1"/>
  </Phase>
  </Graph>
  

• 

  codemonkee

  • October 13, 2010 17:58

  0

  Comment actions Permalink

  Awesome. The code sample workaround method does the job. I am creating/writing the metadata in phase 0 of the graph and in phase 1 am using it for outputting the combined rows. It is a little bit of a kludge, but at the same time a clever solution. Thanks!

Please sign in to leave a comment.