XMLextract

oldforum

• July 16, 2007 00:00

• Answered

Follow

Hello
I am evaluating clover.etl as a replacement aggregation engine for an application. Many of the data input sources are XML streams. I understand that XMLextract is still under development. I have some ideas about how to handle XML streams and how to break them up into flat representations. However, it appears that the process of accepting the XML mappings as children of Node elements is not implemented as described in the XMLextract javadoc. <p>If the maping elements are entered through the "mapping" property of the GUI, they result as attributes of the NODE element having been processed by Component XMLAttributes.
<pre>
<?xml version="1.0" encoding="UTF-8"?>
<Graph name="cposgin">
<Global>
<Metadata id="Metadata0">
<Record name="Applicant" type="delimited">
<Field name="Field1" type="string" delimiter=":" locale="en.US" nullable="true"/>
</Record>
</Metadata>
<Metadata id="Metadata2">
<Record name="FullName" type="delimited">
<Field name="MiddleName" type="string" delimiter=":" locale="en.US" nullable="true"/>
<Field name="LastName" type="string" delimiter=":" locale="en.US" nullable="true"/>
<Field name="Generation" type="string" delimiter=":\n" locale="en.US" nullable="true"/>
<Field name="FirstName" type="string" delimiter=":" locale="en.US" nullable="true"/>
</Record>
</Metadata>
<Metadata id="Metadata1">
<Record name="FullNameList" type="delimited">
<Field name="Field1" type="string" delimiter=":" locale="en.US" nullable="true"/>
</Record>
</Metadata>
</Global>
<Phase number="0">
<Node id="TRASH0" type="TRASH" enabled="enabled" guiX="412" guiY="51" guiWidth="0" guiHeight="0" guiName="Trash"/>
<Node id="TRASH1" type="TRASH" enabled="enabled" guiX="362" guiY="208" guiWidth="0" guiHeight="0" guiName="Trash"/>
<Node id="TRASH2" type="TRASH" enabled="enabled" guiX="203" guiY="283" guiWidth="0" guiHeight="0" guiName="Names"/>
<Node id="XML_EXTRACT0" type="XML_EXTRACT" enabled="enabled" sourceUri="C:/usr/dropbox/HME/processed/thisapp.xml" guiX="34" guiY="45" guiWidth="0" guiHeight="0" guiName="CPOSGIN">
<attr name="mapping"><mapping element="Applicant" OutPort="0">
<mapping element="FullNameList" OutPort="1">
<mapping element="FullName" OutPort="2"/>
</mapping>
</mapping></attr>
</Node>
<Edge id="Edge0" fromNode="XML_EXTRACT0:0" toNode="TRASH0:0" outPort="0 (CPOSGIN)" inPort="0 (Trash)" guiBendpoints="" metadata="Metadata0"/>
<Edge id="Edge1" fromNode="XML_EXTRACT0:1" toNode="TRASH1:0" outPort="1 (CPOSGIN)" inPort="0 (Trash)" guiBendpoints="" metadata="Metadata1"/>
<Edge id="Edge2" fromNode="XML_EXTRACT0:2" toNode="TRASH2:0" outPort="2 (CPOSGIN)" inPort="0 (Names)" guiBendpoints="" metadata="Metadata2"/>
</Phase>
</Graph>
</pre>
<p>When input directly into the graph description, they are ignored.
<pre>
<?xml version="1.0" encoding="UTF-8"?>
<Graph name="cposgin">
<Global>
<Metadata id="Metadata0">
<Record name="Applicant" type="delimited">
<Field name="Field1" type="string" delimiter=":" locale="en.US" nullable="true"/>
</Record>
</Metadata>
<Metadata id="Metadata2">
<Record name="FullName" type="delimited">
<Field name="MiddleName" type="string" delimiter=":" locale="en.US" nullable="true"/>
<Field name="LastName" type="string" delimiter=":" locale="en.US" nullable="true"/>
<Field name="Generation" type="string" delimiter=":\n" locale="en.US" nullable="true"/>
<Field name="FirstName" type="string" delimiter=":" locale="en.US" nullable="true"/>
</Record>
</Metadata>
<Metadata id="Metadata1">
<Record name="FullNameList" type="delimited">
<Field name="Field1" type="string" delimiter=":" locale="en.US" nullable="true"/>
</Record>
</Metadata>
</Global>
<Phase number="0">
<Node id="TRASH0" type="TRASH" enabled="enabled" guiX="412" guiY="51" guiWidth="0" guiHeight="0" guiName="Trash"/>
<Node id="TRASH1" type="TRASH" enabled="enabled" guiX="362" guiY="208" guiWidth="0" guiHeight="0" guiName="Trash"/>
<Node id="TRASH2" type="TRASH" enabled="enabled" guiX="203" guiY="283" guiWidth="0" guiHeight="0" guiName="Names"/>
<Node id="XML_EXTRACT0" type="XML_EXTRACT" enabled="enabled" sourceUri="C:/usr/dropbox/HME/processed/thisapp.xml" guiX="34" guiY="45" guiWidth="0" guiHeight="0" guiName="CPOSGIN">
<mapping element="Applicant" OutPort="0">
<mapping element="FullNameList" OutPort="1">
<mapping element="FullName" OutPort="2"/>
</mapping>
</mapping>
</Node>
<Edge id="Edge0" fromNode="XML_EXTRACT0:0" toNode="TRASH0:0" outPort="0 (CPOSGIN)" inPort="0 (Trash)" guiBendpoints="" metadata="Metadata0"/>
<Edge id="Edge1" fromNode="XML_EXTRACT0:1" toNode="TRASH1:0" outPort="1 (CPOSGIN)" inPort="0 (Trash)" guiBendpoints="" metadata="Metadata1"/>
<Edge id="Edge2" fromNode="XML_EXTRACT0:2" toNode="TRASH2:0" outPort="2 (CPOSGIN)" inPort="0 (Names)" guiBendpoints="" metadata="Metadata2"/>
</Phase>
</Graph>
</pre>
<p>Am I just misinterpreting the DTD or is this something that is planned for future releases. I see that there is an internal calss in XMLextract for handling "Mapping".
<p>Mike

• 

  oldforum

  • July 16, 2007 08:23

  0

  Comment actions Permalink

  Thanks
  I did figure that out.
  I have rewritten XMLExtract to do some other interesting things like inserting the context of the extracted data segment and some synchronization information when repeating elements terminate. I will submit my results when I am satisfied with the results.
  Mike
• 

  oldforum

  • July 16, 2007 08:23

  0

  Comment actions Permalink

  I added some extension to the XMLExtract myself.
  I had problems with namespaces like
  <prop type="xml:lang">
  Clover's metadata only accepts letters and numbers for field names, so I made something similar to dbFields-cloverFields mapping
  
  here is example:
  xmlFields="xml:lang" cloverFields="lang"
  
  If you find this useful I could send you the patch.
  
  BTW, What are the future plans for XMLExtract component?
• 

  oldforum

  • July 16, 2007 08:23

  0

  Comment actions Permalink

  Hello Mike.
  
  XMLExtract has mapping attribute, which is inner xml node. Now it is one and only component, which need this functionality. In the future we want support this case too. But for now you must write xml code for this component by hand.
  
  OtaSanek
• 

  oldforum

  • July 16, 2007 08:23

  0

  Comment actions Permalink

  HI !
  
  We are definitely interested in your changes. Please, send the changes to david.pavlis <at> centrum.cz
  
  As for the future of the component - it has been created by Kuan Kou who does not seem to be maintaining it any more.
  We have been thinking about recoding this component, but this is currently somewhere in the future.
  
  David.

Please sign in to leave a comment.