Hello everyone,
Don't know if this is the right place to post my question but here it goes.
I'm using OEID 3.1 .
I want to update my records in my Data Domain to replace the values in a new column that I've created previously in Endeca.
The query gets my elements and order them in 1-3 minutes top (with 3.2 M records) but when it reaches the modify records components, it takes hours and hours to modify the records.
I noticed that it gets stuck on specific values for a lot of time and then it moves on, happening again in other records. Why is this happening ?
The number of records are the same and I've tried a lot of different ways but this is happening every time.
Here's the code if you want to check it out.
Many thanks everyone.
Regards and have a good week.
<?xml version="1.0" encoding="UTF-8"?><Graph author="oracle" created="Wed May 28 11:05:33 WEST 2014" guiVersion="3.4.1.P" id="1401271876519" licenseType="Commercial" modified="Wed Jul 02 15:54:40 WEST 2014" modifiedBy="oracle" name="LoadInitialTablesToDataDomain" revision="1.403" showComponentDetails="true">
<Global>
<Metadata id="Metadata0" previewAttachmentCharset="ISO-8859-1">
<Record fieldDelimiter=";" name="CC" previewAttachmentCharset="ISO-8859-1" recordDelimiter="\n" recordSize="-1" type="delimited">
<Field eofAsDelimiter="false" length="10" name="PCODIGO" nullable="true" scale="0" shift="0" size="0" type="number"/>
<Field eofAsDelimiter="false" name="CC_VALIDA" nullable="true" shift="0" size="4000" type="string"/>
<Field eofAsDelimiter="false" name="SUGESTAO_PBI" nullable="true" shift="0" size="4000" type="string"/>
</Record>
</Metadata>
<Connection dbConfig="${CONN_DIR}/connection.cfg" id="JDBC0" type="JDBC"/>
<Property fileURL="workspace.prm" id="GraphParameter0"/>
<Dictionary/>
</Global>
<Phase number="0">
<Node enabled="enabled" guiName="ExtSort" guiX="309" guiY="41" id="EXT_SORT" sortKey="PCODIGO(a)" type="EXT_SORT"/>
<Node batchSize="1000000" dataDomainName="pessoas2" enabled="enabled" endecaServerContextRoot="/endeca-server" endecaServerHost="lisobiee01" endecaServerPort="21000" guiName="Modify Records" guiX="599" guiY="46" id="MODIFY_RECORDS" multiAssignBehavior="MATCHES_EXACTLY" multiAssignDelimiter="" operation="REPLACE_ASSIGNMENTS" specAttributes="PCODIGO" type="ENDECA_MODIFY_RECORDS"/>
<Node dbConnection="JDBC0" enabled="enabled" guiName="Select * from Pessoas" guiX="19" guiY="41" id="SELECT_FROM_PESSOAS2" type="DB_INPUT_TABLE">
<attr name="sqlQuery"><![CDATA[select pcodigo, stg_val_cc(pbi, codigo_cartao_cidadao) as CC_VALIDA, stg_sugestao_cc(pbi, codigo_cartao_cidadao) as SUGESTAO_PBI from pessoas where nova_pessoa=0]]></attr>
</Node>
<Edge fromNode="EXT_SORT:0" guiBendpoints="" guiRouter="Manhattan" id="Edge1" inPort="Port 0 (in)" metadata="Metadata0" outPort="Port 0 (out)" toNode="MODIFY_RECORDS:0"/>
<Edge debugMode="true" fromNode="SELECT_FROM_PESSOAS2:0" guiBendpoints="" guiRouter="Manhattan" id="Edge0" inPort="Port 0 (in)" metadata="Metadata0" outPort="Port 0 (out)" toNode="EXT_SORT:0"/>
</Phase>
</Graph>
Don't know if this is the right place to post my question but here it goes.
I'm using OEID 3.1 .
I want to update my records in my Data Domain to replace the values in a new column that I've created previously in Endeca.
The query gets my elements and order them in 1-3 minutes top (with 3.2 M records) but when it reaches the modify records components, it takes hours and hours to modify the records.
I noticed that it gets stuck on specific values for a lot of time and then it moves on, happening again in other records. Why is this happening ?
The number of records are the same and I've tried a lot of different ways but this is happening every time.
Here's the code if you want to check it out.
Many thanks everyone.
Regards and have a good week.
<?xml version="1.0" encoding="UTF-8"?><Graph author="oracle" created="Wed May 28 11:05:33 WEST 2014" guiVersion="3.4.1.P" id="1401271876519" licenseType="Commercial" modified="Wed Jul 02 15:54:40 WEST 2014" modifiedBy="oracle" name="LoadInitialTablesToDataDomain" revision="1.403" showComponentDetails="true">
<Global>
<Metadata id="Metadata0" previewAttachmentCharset="ISO-8859-1">
<Record fieldDelimiter=";" name="CC" previewAttachmentCharset="ISO-8859-1" recordDelimiter="\n" recordSize="-1" type="delimited">
<Field eofAsDelimiter="false" length="10" name="PCODIGO" nullable="true" scale="0" shift="0" size="0" type="number"/>
<Field eofAsDelimiter="false" name="CC_VALIDA" nullable="true" shift="0" size="4000" type="string"/>
<Field eofAsDelimiter="false" name="SUGESTAO_PBI" nullable="true" shift="0" size="4000" type="string"/>
</Record>
</Metadata>
<Connection dbConfig="${CONN_DIR}/connection.cfg" id="JDBC0" type="JDBC"/>
<Property fileURL="workspace.prm" id="GraphParameter0"/>
<Dictionary/>
</Global>
<Phase number="0">
<Node enabled="enabled" guiName="ExtSort" guiX="309" guiY="41" id="EXT_SORT" sortKey="PCODIGO(a)" type="EXT_SORT"/>
<Node batchSize="1000000" dataDomainName="pessoas2" enabled="enabled" endecaServerContextRoot="/endeca-server" endecaServerHost="lisobiee01" endecaServerPort="21000" guiName="Modify Records" guiX="599" guiY="46" id="MODIFY_RECORDS" multiAssignBehavior="MATCHES_EXACTLY" multiAssignDelimiter="" operation="REPLACE_ASSIGNMENTS" specAttributes="PCODIGO" type="ENDECA_MODIFY_RECORDS"/>
<Node dbConnection="JDBC0" enabled="enabled" guiName="Select * from Pessoas" guiX="19" guiY="41" id="SELECT_FROM_PESSOAS2" type="DB_INPUT_TABLE">
<attr name="sqlQuery"><![CDATA[select pcodigo, stg_val_cc(pbi, codigo_cartao_cidadao) as CC_VALIDA, stg_sugestao_cc(pbi, codigo_cartao_cidadao) as SUGESTAO_PBI from pessoas where nova_pessoa=0]]></attr>
</Node>
<Edge fromNode="EXT_SORT:0" guiBendpoints="" guiRouter="Manhattan" id="Edge1" inPort="Port 0 (in)" metadata="Metadata0" outPort="Port 0 (out)" toNode="MODIFY_RECORDS:0"/>
<Edge debugMode="true" fromNode="SELECT_FROM_PESSOAS2:0" guiBendpoints="" guiRouter="Manhattan" id="Edge0" inPort="Port 0 (in)" metadata="Metadata0" outPort="Port 0 (out)" toNode="EXT_SORT:0"/>
</Phase>
</Graph>
-
Dear aiaimanel,
As this issue relates to Endeca Data Integrator please do contact Oracle Endeca support in this matter directly. They need to be informed about this issue and will work with us directly if needed. Thank you for your cooperation!
I can give you some general tips how to find performance bottleneck:
* try to put each component into separate phase so you will see in run log which component is the slow one
* use FastSort instead of ExtSort, or tune parameters (http://www.cloveretl.com/blog/sorting-d ... -fastsort/)
I hope this helps. -
I don't know Endeca much, but judging from your graph - the modification happens inside Endeca specific component - "ENDECA_MODIFY_RECORDS" which has to connect to Endeca server over network. So I would be looking into the performance bottleneck inside Endeca Server or check the network connection.
But as stated before - this is Endeca specific, the component has been created by Endeca team. The CloverETL team does not have visibility into this.
The suggestion about splitting into phases is valid though - it should help verify that the problem lies in the Endeca specific component.
Please sign in to leave a comment.
Comments 2