Customer Portal

The CloverETL MateDate can not meta the oracle number type

Comments 14

  • Avatar
    avackova
    0
    Comment actions Permalink
    Hello,
    try to set format="###" for field LOG_ID (type="decimal")
  • Avatar
    c392715524
    0
    Comment actions Permalink
    Thanks avackova,
    It is working with the change from yours.

    But there is a new error appearing.


    WARN optimizeConnection, Optimazing connection failed: 仅 READ_COMMITTED 和 SERIALIZABLE 是有效的事务处理级
    WARN optimizeConnection, Try to use another jdbc specific
    WARN runInNormalMode, Exeption thrown by: insert into 20081125_logmnrcontents values(?,?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?). Message: Incorrect string value: '\xE6\x9C\x88-08...' for column 'SQL_REDO' at row 1
    ERROR watch, Graph execution finished with error
    ERROR watch, Node OUTPUT finished with status: ERROR caused by: Maximum # of errors exceeded when inserting record. Exeption thrown by: insert into 20081125_logmnrcontents values(?,?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?). Message: Incorrect string value: '\xE6\x9C\x88-08...' for column 'SQL_REDO' at row 1 caused by: java.sql.SQLException: Incorrect string value: '\xE6\x9C\x88-08...' for column 'SQL_REDO' at row 1
    ERROR call, !!! Phase finished with error - stopping graph run !!!
    ERROR runGraph, Execution of graph failed !


    "\xE6\x9C\x88-08..." I think this is some Chinese word,
    By the way ,I hava execute the 'insert into ... values(...,"\xE6\x9C\x88-08..." ,...) ' in mysql database , and it is working well.
    The character set of mysql is utf8/gdb (there is the same error when i change the character set).
    So, please help me.
    What can i do ? DO i change the cloverETL's code, make this "\xE6\x9C\x88-08..." to "\\xE6\\x9C\\x88-08..." ?
    THANK YOU VERY MUCH
  • Avatar
    c392715524
    0
    Comment actions Permalink
    I am sorry that record SQL_REDO is not chinese word only .
    Just now , I have a test for it.
    The SQL_REDO is maybe any word ,there is must same configure is wrong.
    Mygraph is following:

    <Field name="SQL_REDO" type="string" nullable="yes" delimiter="," />
    <Field name="SQL_UNDO" type="string" nullable="yes" delimiter="," />
  • Avatar
    avackova
    0
    Comment actions Permalink
    How do you get values for the table? Are they from a flat file? If yes try to change encoding on your data reader.
  • Avatar
    c392715524
    0
    Comment actions Permalink
    I read data from oracle database and then insert into mysql table
    There is my graph:

    <?xml version="1.0" encoding="UTF-8"?>
    <Graph author="caotong" created="Mon Nor 8 13:10 2008" description="A DATA TRANSFORM FROM ORACLE TO MYSQL." id="0001"
    name="DBTRANSFORM" revision="1.0">
    <Global>
    <Metadata id="Metadata0">
    <Record name="logmnr_contents" type="delimited">
    <Field name="SCN" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="CSCN" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="TIMESTAMP" type="date" nullable="yes" delimiter="," />
    <Field name="COMMIT_TIMESTAMP" type="date" nullable="yes" delimiter="," />
    <Field name="THREAD" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="LOG_ID" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="XIDUSN" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="XIDSLT" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="XIDSQN" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="PXIDUSN" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="PXIDSLT" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="PXIDSQN" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="RBASQN" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="RBABLK" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="RBABYTE" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="UBAFIL" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="UBABLK" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="UBAREC" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="UBASQN" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="ABS_FILE" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="REL_FILE" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="DATA_BLK" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="DATA_OBJ" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="DATA_OBJD" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="SEG_OWNER" type="string" nullable="yes" delimiter="," />
    <Field name="SEG_NAME" type="string" nullable="yes" delimiter="," />
    <Field name="TABLE_NAME" type="string" nullable="yes" delimiter="," />
    <Field name="SEG_TYPE" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="SEG_TYPE_NAME" type="string" nullable="yes" delimiter="," />
    <Field name="TABLE_SPACE" type="string" nullable="yes" delimiter="," />
    <Field name="ROW_ID" type="string" nullable="yes" delimiter="," />
    <Field name="SESSION" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="SERIAL" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="USERNAME" type="string" nullable="yes" delimiter="," />
    <Field name="SESSION_INFO" type="string" nullable="yes" delimiter="," />
    <Field name="TX_NAME" type="string" nullable="yes" delimiter="," />
    <Field name="ROLLBACK" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="OPERATION" type="string" nullable="yes" delimiter="," />
    <Field name="OPERATION_CODE" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="SQL_REDO" type="string" nullable="yes" delimiter="\0" />
    <Field name="SQL_UNDO" type="string" nullable="yes" delimiter="\0" />
    <Field name="RS_ID" type="string" nullable="yes" delimiter="," />
    <Field name="SEQUENCE" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="SSN" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="CSF" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="INFO" type="string" nullable="yes" delimiter="," />
    <Field name="STATUS" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="REDO_VALUE" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="UNDO_VALUE" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="SQL_COLUMN_TYPE" type="string" nullable="yes" delimiter="," />
    <Field name="SQL_COLUMN_NAME" type="string" nullable="yes" delimiter="," />
    <Field name="REDO_LENGTH" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="REDO_OFFSET" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="UNDO_LENGTH" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="UNDO_OFFSET" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="DATA_OBJV" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="SAFE_RESUME_SCN" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
    <Field name="XID" type="byte" nullable="yes" delimiter="," />
    <Field name="PXID" type="byte" nullable="yes" delimiter="," />
    <Field name="AUDIT_SESSIONID" type="numeric" nullable="yes" delimiter="\n" format="#############.#######"/>
    </Record>
    </Metadata>
    <Connection dbConfig="${CONN_DIR}/oracle.cfg" id="Connection0" type="JDBC"/>
    <Connection dbConfig="${CONN_DIR}/mysql.cfg" id="Connection1" type="JDBC"/>
    <Property fileURL="workspace.prm" id="GraphParameter0"/>
    </Global>
    <Phase number="0">
    <Node dbConnection="Connection0" id="INPUT" sqlQuery="select * from v$logmnr_contents " type="DB_INPUT_TABLE"/>
    <Node dbConnection="Connection1" dbTable="logmnr_contents" id="OUTPUT" sqlQuery="insert into 20081125_logmnrcontents values($SCN,$CSCN,
    $TIMESTAMP,
    $COMMIT_TIMESTAMP,
    $THREAD,
    $LOG_ID,
    $XIDUSN,
    $XIDSLT,
    $XIDSQN,
    $PXIDUSN,
    $PXIDSLT,
    $PXIDSQN,
    $RBASQN,
    $RBABLK,
    $RBABYTE,
    $UBAFIL,
    $UBABLK,
    $UBAREC,
    $UBASQN,
    $ABS_FILE,
    $REL_FILE,
    $DATA_BLK,
    $DATA_OBJ,
    $DATA_OBJD,
    $SEG_OWNER,
    $SEG_NAME,
    $TABLE_NAME,
    $SEG_TYPE,
    $SEG_TYPE_NAME,
    $TABLE_SPACE,
    $ROW_ID,
    $SESSION,
    $SERIAL,
    $USERNAME,
    $SESSION_INFO,
    $TX_NAME,
    $ROLLBACK,
    $OPERATION,
    $OPERATION_CODE,
    $SQL_REDO,
    $SQL_UNDO,
    $RS_ID,
    $SEQUENCE,
    $SSN,
    $CSF,
    $INFO,
    $STATUS,
    $REDO_VALUE,
    $UNDO_VALUE,
    $SQL_COLUMN_TYPE,
    $SQL_COLUMN_NAME,
    $REDO_LENGTH,
    $REDO_OFFSET,
    $UNDO_LENGTH,
    $UNDO_OFFSET,
    $DATA_OBJV,
    $SAFE_RESUME_SCN,
    $XID,
    $PXID,
    $AUDIT_SESSIONID);" type="DB_OUTPUT_TABLE"/>
    <Edge fromNode="INPUT:0" id="inedge1" metadata="Metadata0" toNode="OUTPUT:0"/>
    </Phase>

    </Graph>
  • Avatar
    avackova
    0
    Comment actions Permalink
    I can't reproduce your problem - my graph with DBOutputTable to mysql database works properly. Probably problem is with your mysql jdbc driver. Pls check your connection config against the following:
    user=test
    password=
    passwordEncrypted=false
    threadSafeConnection=true
    database=MYSQL
    dbURL=jdbc\:mysql\://hostaname/database
  • Avatar
    c392715524
    0
    Comment actions Permalink
    My mysql configure is ok , and the first error of 'number' is appear again.
    So,if anyone is the developer of this cloveretl. please tell me if the cloveretl can work with data like '\xC6\xB8......' if it is not ,can i have a way to fix that,

    By the way the error of 'number' is not disappear, when my graph have bean wrote like:

    <Field name="LOG_ID" type="decimal" nullable="yes" delimiter="," format="#############.#######"/>

    or

    <Field name="LOG_ID" type="numeric" nullable="yes" delimiter="," format="#############.#######"/>
  • Avatar
    c392715524
    0
    Comment actions Permalink
    IT is hava that error again, help please!

    <Metadata id="Metadata0">
    <Record name="logmnr_contents" type="delimited">
    <Field name="SCN" type="decimal" nullable="yes" delimiter="," format="########.####"/>
    <Field name="CSCN" type="decimal" nullable="yes" delimiter="," format="########.####"/>
    <Field name="TIMESTAMP" type="date" nullable="yes" delimiter="," />
    <Field name="COMMIT_TIMESTAMP" type="date" nullable="yes" delimiter="," />
    <Field name="THREAD" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="LOG_ID" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="XIDUSN" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="XIDSLT" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="XIDSQN" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="PXIDUSN" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="PXIDSLT" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="PXIDSQN" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="RBASQN" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="RBABLK" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="RBABYTE" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="UBAFIL" type="decimal" nullable="yes" delimiter="," format="########.####"/>
    <Field name="UBABLK" type="decimal" nullable="yes" delimiter="," format="########.####"/>
    <Field name="UBAREC" type="decimal" nullable="yes" delimiter="," format="########.####"/>
    <Field name="UBASQN" type="decimal" nullable="yes" delimiter="," format="########.####"/>
    <Field name="ABS_FILE" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="REL_FILE" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="DATA_BLK" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="DATA_OBJ" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="DATA_OBJD" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="SEG_OWNER" type="string" nullable="yes" delimiter="," />
    <Field name="SEG_NAME" type="string" nullable="yes" delimiter="," />
    <Field name="TABLE_NAME" type="string" nullable="yes" delimiter="," />
    <Field name="SEG_TYPE" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="SEG_TYPE_NAME" type="string" nullable="yes" delimiter="," />
    <Field name="TABLE_SPACE" type="string" nullable="yes" delimiter="," />
    <Field name="ROW_ID" type="string" nullable="yes" delimiter="," />
    <Field name="SESSION" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="SERIAL" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="USERNAME" type="string" nullable="yes" delimiter="," />
    <Field name="SESSION_INFO" type="string" nullable="yes" delimiter="," />
    <Field name="TX_NAME" type="string" nullable="yes" delimiter="," />
    <Field name="ROLLBACK" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="OPERATION" type="string" nullable="yes" delimiter="," />
    <Field name="OPERATION_CODE" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="SQL_REDO" type="string" nullable="yes" delimiter=";" />
    <Field name="SQL_UNDO" type="string" nullable="yes" delimiter=";" />
    <Field name="RS_ID" type="string" nullable="yes" delimiter="," />
    <Field name="SEQUENCE" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="SSN" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="CSF" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="INFO" type="string" nullable="yes" delimiter="," />
    <Field name="STATUS" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="REDO_VALUE" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="UNDO_VALUE" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="SQL_COLUMN_TYPE" type="string" nullable="yes" delimiter="," />
    <Field name="SQL_COLUMN_NAME" type="string" nullable="yes" delimiter="," />
    <Field name="REDO_LENGTH" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="REDO_OFFSET" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="UNDO_LENGTH" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="UNDO_OFFSET" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="DATA_OBJV" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="SAFE_RESUME_SCN" type="decimal" nullable="yes" delimiter="," format="########.####" />
    <Field name="XID" type="byte" nullable="yes" delimiter="," />
    <Field name="PXID" type="byte" nullable="yes" delimiter="," />
    <Field name="AUDIT_SESSIONID" type="decimal" nullable="yes" delimiter="\n" format="########.####" />
    </Record>


    DEBUG [INPUT] - Number is out of available precision [8,2]. ()
    java.lang.NumberFormatException: Number is out of available precision [8,2]. ()
    at org.jetel.data.primitive.IntegerDecimal.setValue(IntegerDecimal.java:143)
    at org.jetel.data.primitive.IntegerDecimal.setValue(IntegerDecimal.java:209)
    at org.jetel.data.DecimalDataField.setValue(DecimalDataField.java:282)
    at org.jetel.connection.jdbc.CopySQLData$CopyDecimal.setJetel(CopySQLData.java:671)
    at org.jetel.connection.jdbc.CopySQLData.sql2jetel(CopySQLData.java:145)
    at org.jetel.connection.jdbc.SQLDataParser.populateField(SQLDataParser.java:240)
    at org.jetel.connection.jdbc.SQLDataParser.parseNext(SQLDataParser.java:212)
    at org.jetel.connection.jdbc.SQLDataParser.getNext(SQLDataParser.java:138)
    at org.jetel.connection.jdbc.SQLDataParser.getNext(SQLDataParser.java:182)
    at org.jetel.component.DBInputTable.execute(DBInputTable.java:240)
    at org.jetel.graph.Node.run(Node.java:379)
    at java.lang.Thread.run(Unknown Source)
    ERROR [WatchDog] - Graph execution finished with error
    ERROR [WatchDog] - Node INPUT finished with status: ERROR caused by: Number is out of available precision [8,2]. ()
    DEBUG [WatchDog] - Node INPUT error details:
    java.lang.RuntimeException: Number is out of available precision [8,2]. ()
    at org.jetel.connection.jdbc.SQLDataParser.populateField(SQLDataParser.java:252)
    at org.jetel.connection.jdbc.SQLDataParser.parseNext(SQLDataParser.java:212)
    at org.jetel.connection.jdbc.SQLDataParser.getNext(SQLDataParser.java:138)
    at org.jetel.connection.jdbc.SQLDataParser.getNext(SQLDataParser.java:182)
    at org.jetel.component.DBInputTable.execute(DBInputTable.java:240)
    at org.jetel.graph.Node.run(Node.java:379)
    at java.lang.Thread.run(Unknown Source)
    Caused by: java.lang.NumberFormatException: Number is out of available precision [8,2]. ()
    at org.jetel.data.primitive.IntegerDecimal.setValue(IntegerDecimal.java:143)
    at org.jetel.data.primitive.IntegerDecimal.setValue(IntegerDecimal.java:209)
    at org.jetel.data.DecimalDataField.setValue(DecimalDataField.java:282)
    at org.jetel.connection.jdbc.CopySQLData$CopyDecimal.setJetel(CopySQLData.java:671)
    at org.jetel.connection.jdbc.CopySQLData.sql2jetel(CopySQLData.java:145)
    at org.jetel.connection.jdbc.SQLDataParser.populateField(SQLDataParser.java:240)
    ... 6 more
    INFO [exNode_0_0001_OUTPUT] - Number of commited records: 0
  • Avatar
    c392715524
    0
    Comment actions Permalink
    I think that error appeared again because that the oracle number type is an changed type everytime when you execute the "select".
    AND my oracle.V$logmnr_contents is not a static table but a view which is born everytime when I called the "execute DBMS_LOGMNR.start......",and destroy when I called "execute DBMS_LOGMNR.end....."
    SO could the CloverETL can fix it?
  • Avatar
    avackova
    0
    Comment actions Permalink
    This seems that in database there is an invalid number, that can't be handle by Clover. Try to set data policy to "controlled" on DBInputTable; then numbers, which can't be handled will be skipped.
  • Avatar
    c392715524
    0
    Comment actions Permalink
    Yes, I will try.
    But how can i set data policy to "controlled" on DBInputTable?
    Please give me an example ,
    Thanks
  • Avatar
    c392715524
    0
    Comment actions Permalink
    I will send you an e-mail, please check thanks.
  • Avatar
    avackova
    0
    Comment actions Permalink
    see attribute dataPolicy
  • Avatar
    c392715524
    0
    Comment actions Permalink
    I am sorry it does not work but I am always thank you very much.

Please sign in to leave a comment.