Writing complex XML output

CloverDX Customer Portal
Forum
Writing complex XML output

jcuendet

April 09, 2010 00:00

Answered

Dear Javlin team,

Currently, I'm trying to write a complex output XML file. For that, I first used XMLwriter to produce a basic XML file and then post-process it with XSL transformation in order to create the final XML output. It works fine for 10'000 records but then I encounter memory issues. I tried to increase JVM parameters but it's not sufficient as I need to process millions of records.

Therefore, I'm looking for another solution, and I would like to be able to directly generate the correct XML file throught the Clover XMLwriter element (or another one).
I absolutely need the XML mapping functionality as I join two input elements in one and I also need to use different sources.

For example, I would like to do something like:

<Mapping inPort="0" element="contractA">
<Mapping element="specifications" inPort="3" key="id" parentKey="id" fieldsIgnore="id"/>
</Mapping>
<Mapping inPort="1" element="contractB">
<Mapping element="specifications" inPort="3" key="id" parentKey="id" fieldsIgnore="id"/>
</Mapping>
<Mapping inPort="2" element="contractC">
<Mapping element="specifications" inPort="3" key="id" parentKey="id" fieldsIgnore="id"/>
</Mapping>

With 4 different sources (no chance to merge these 3 sources). But this is not a valid mapping and I don't see a clean solution.

Any help is highly welcomed

Jerome Cuendet

Comments 6

avackova

April 09, 2010 12:40
0

Comment actions Permalink
Hello Jerome,
see OutOfMemoryError with XMLWriter topic. It could help you to solve the OutOfMemory as well as mapping problem.
jcuendet

April 09, 2010 12:52
0

Comment actions Permalink
Sounds a good solution as I really would like to have a sequential writing of the XML to avoid any memory issue and ensure scalability.
Thanks a lot
jcuendet

April 09, 2010 13:30
0

Comment actions Permalink
In fact, it is absolutely fine to use hashJoin component but my two input elements have a relation 1 to n.
Meaning that I don't see a solution to write a transform function that produce an output such as:
```
<contract>
  <id>1</id>
  <name>toto</name>
  <specs>
    <spec>
       <idspec>123</idspec>
       <element>abc</element>
    </spec>
    <spec>
       <idspec>456</idspec>
       <element>def</element>
    </spec>
   </specs>
</contract>
```
Where contract elements (id, name) are coming from one input (1 record) and specs elements (idspec, element) are coming from another input (2 records).
Do you see a solution for that?
Thanks
jcuendet

April 09, 2010 14:20
0

Comment actions Permalink
I still encounter issue as I need to produce an XML output such as:
```
<contract>
 <id>1</id>
 <name>toto</name>
 <specs>
  <spec>
   <idspec>1234</idspec>
   <namespec>abcd</namespec>
  </spec>
  <spec>
   <idspec>5678</idspec>
   <namespec>efgh</namespec>
  </spec>
 </specs>
</contract>
```
From two different inputs:
- contract with one record (id, name)
- specs with two row (idspecs, namespec and id as the foreign key)
How can I manage that with hashJoin or other components?
Thanks

avackova

April 12, 2010 11:12

Hello,
you have to con-cat the attributes with the same id before joining by Denormalize component:

<?xml version="1.0" encoding="UTF-8"?>
<Graph author="avackova" created="Thu Jun 11 13:46:15 CEST 2009" guiVersion="0.0.0.devel" id="1244721135669" licenseType="Evaluation Devel" modified="Mon Apr 12 13:06:15 CEST 2010" modifiedBy="avackova" name="xmlJoin" revision="1.48">
<Global>
<Metadata id="Metadata2">
<Record fieldDelimiter="|" name="attributes" recordDelimiter="\n" type="delimited">
<Field name="id" type="integer"/>
<Field name="idspec" type="integer"/>
<Field name="namespec" type="string"/>
</Record>
</Metadata>
<Metadata id="Metadata0">
<Record fieldDelimiter="|" name="master" recordDelimiter="\n" type="delimited">
<Field name="id" type="integer"/>
<Field name="name" type="string"/>
</Record>
</Metadata>
<Metadata id="Metadata3">
<Record fieldDelimiter="|" name="slave" recordDelimiter="\n" type="delimited">
<Field name="id" type="integer"/>
<Field name="output" type="string"/>
</Record>
</Metadata>
<Metadata id="Metadata1">
<Record fieldDelimiter="|" name="xml" recordDelimiter="\n" type="delimited">
<Field name="field1" type="string"/>
</Record>
</Metadata>
<Property fileURL="workspace.prm" id="GraphParameter0"/>
</Global>
<Phase number="0">
<Node id="DATA_GENERATOR0" recordsNumber="1" type="DATA_GENERATOR">
<attr name="generate"><![CDATA[//#TL

// Generates output record.
function generate() {
	$0.id := 1;
	$0.name := 'toto';
}

// Called during component initialization.
// function init() {}

// Called after the component finishes.
// function finished() {}
]]></attr>
</Node>
<Node  id="DATA_GENERATOR1" recordsNumber="2" type="DATA_GENERATOR">
<attr name="generate"><![CDATA[//#TL
int counter = -1;
// Generates output record.
function generate() {
	counter++;
	$0.id := '1';
	$0.idspec := iif(counter == 0, 1234, 5678);
	$0.namespec := iif(counter == 0, 'abcd', 'efgh');
}

// Called during component initialization.
// function init() {}

// Called after the component finishes.
// function finished() {}
]]></attr>
</Node>
<Node id="DENORMALIZER0" key="id" type="DENORMALIZER">
<attr name="denormalize"><![CDATA[//#TL
string output = "";
// This transformation defines the way in which multiple input records 
// (with the same key) are denormalized into one output record. 
// This function is called for each input record from a group of records
// with the same key.
function append() {
	output = output + "\t<spec>\n\t<idspec>" + $idspec + "</idspec>\n\t<namespec>" + $namespec + "</namespec>\n\t</spec>\n";
	
}

// This function is called once after the append() function was called for all records
// of a group of input records defined by the key.
// It creates a single output record for the whole group.
function transform() {
	$0.id := $0.id;
	$0.output := output;
}

// Called after transform() to return the resources that have been used to their initial state
// so that next group of records with different key may be parsed.
function clean() {
	output = "";
}

// Called to return a user-defined error message when an error occurs.
// function getMessage() {}

// Called during component initialization.
// function init() {}

// Called after the component finishes.
// function finished() {}
]]></attr>
</Node>
<Node id="EXT_HASH_JOIN0" joinKey="$id=$id" slaveDuplicates="true" type="EXT_HASH_JOIN">
<attr name="transform"><![CDATA[//#TL

// Transforms input record into output record.
function transform() {
	$0.field1 := "<contract>\\n<id>" + $id + "</id>\\n<name>" + $name + "</name>" + $1.output + "\\n</specs>\\n</contract>";
}

// Called during component initialization.
// function init() {}

// Called after the component finishes.
// function finished() {}
]]></attr>
</Node>
<Node id="TRASH0" type="TRASH"/>
<Edge fromNode="DATA_GENERATOR0:0"  id="Edge0" inPort="Port 0 (driver)" metadata="Metadata0" outPort="Port 0 (out)" toNode="EXT_HASH_JOIN0:0"/>
<Edge fromNode="DATA_GENERATOR1:0"  id="Edge1" inPort="Port 0 (in)" metadata="Metadata2" outPort="Port 0 (out)" toNode="DENORMALIZER0:0"/>
<Edge fromNode="DENORMALIZER0:0"  id="Edge4" inPort="Port 1 (slave)" metadata="Metadata3" outPort="Port 0 (out)" toNode="EXT_HASH_JOIN0:1"/>
<Edge debugMode="true" fromNode="EXT_HASH_JOIN0:0"  id="Edge3" inPort="Port 0 (in)" metadata="Metadata1" outPort="Port 0 (out)" toNode="TRASH0:0"/>
</Phase>
</Graph>

jcuendet

April 14, 2010 08:39
0

Comment actions Permalink
Fantastic :-)
It works well, I used two Denormalizers components (as my structure is a bit more complex than the example I posted), one hashJoin and a structuredDataWriter for final XML.
Thanks a lot for your help

Please sign in to leave a comment.

Quick links

Access my products

SUPPORT & SERVICES

Community

RESOURCES