Customer Portal

Using try_convert in embedded clover

Comments 8

  • Avatar
    avackova
    0
    Comment actions Permalink
    Hello,
    it seems, that the function is not loaded during the CloverETL Engine initialization. There is list of the loaded plug-ins on the begging on the graph log. It should contain something like:
    DEBUG [main] - Plugin org.jetel.tlfunction loaded.
    id - org.jetel.tlfunction
    version - 3.1.0.devel20110506143324avackova
    provider-name - null
    tlfunction { libraryName = math; className = org.jetel.interpreter.extensions.MathLib; function = sqrt,log,log10,exp,round,pow,pi,e,random,random_gaussian,random_boolean,random_int,random_long,abs,bit_and,bit_or,bit_xor,bit_set,bit_invert,bit_is_set,bit_lshift,bit_rshift; }
    tlfunction { libraryName = date; className = org.jetel.interpreter.extensions.DateLib; function = today,dateadd,datediff,trunc,trunc_date,random_date; }
    tlfunction { libraryName = string; className = org.jetel.interpreter.extensions.StringLib; function = concat,uppercase,lowercase,substring,left,right,trim,length,replace,split,char_at,is_blank,is_ascii,is_number,is_integer,is_long,is_date,remove_diacritic,remove_blank_space,get_alphanumeric_chars,translate,join,index_of,count_char,chop,remove_nonprintable,remove_nonascii,find,cut,random_string; }
    tlfunction { libraryName = convert; className = org.jetel.interpreter.extensions.ConvertLib; function = num2str,date2str,str2date,date2num,str2num,try_convert,base64byte,byte2base64,bits2str,str2bits,hex2byte,byte2hex,num2num,num2bool,bool2num,str2bool,long2date,date2long,to_string,md5,sha,long2pacdecimal,pacdecimal2long,get_field_name,get_field_type; }
    tlfunction { libraryName = container; className = org.jetel.interpreter.extensions.ContainerLib; function = remove_all,push,pop,poll,remove,insert,sort,copy,reverse,dict_put_str,dict_get_str; }

    If you don't have the similar fragment in your log (note the DEBUG log level) no function will work.

    By the way, if you need to convert something to the string, you would rather use to_string or proper type2str (num2str, date2str, ...) function instead of try_convert. If you want to assign string input field to the string output field use simple mapping:

    $0.QUALIFIED_URL := t$0.QUALIFIED_URL;
    $0.URL_PROTOCOL := $0.URL_PROTOCOL;
    $0.URL_USER := $0.URL_USER;
    $0.URL_PASSWORD := $0.URL_PASSWORD;
    .
    .

    Or, if the names and types of input and output fields are the same, you can use a wild card mapping:
    $0.* := $0.*;

    The above statement assigns the values from input port to the output port for all fields with same names and types. Then you need to assign only the fields, which can't be guessed.
  • Avatar
    fschnack
    0
    Comment actions Permalink
    Hello there! Thanks for the great tip, I really wasn't loading the appropriate plugin.xml. I've added it to my engine initialization code and that error is gone.

    Regarding the string conversion you mention, that sample TL code I sent before wasn't very well written. In most cases, I want to convert a String to another type, mainly number. But is that code's syntax correct? For some reason, now I get a NPE error, but I can't understand why. Here is the CTL I am using now:


    //#TL
    $0.QUALIFIED_URL := $0.QUALIFIED_URL;
    $0.URL_PROTOCOL := $0.URL_PROTOCOL;
    $0.URL_USER := $0.URL_USER;
    $0.URL_PASSWORD := $0.URL_PASSWORD;
    $0.URL_SUBDOMAIN := $0.URL_SUBDOMAIN;
    $0.URL_DOMAIN := $0.URL_DOMAIN;
    $0.URL_PORT := $0.URL_PORT;
    $0.URL_PATH := $0.URL_PATH;
    $0.URL_QUERY := $0.URL_QUERY;
    $0.URL_FRAGMENT := $0.URL_FRAGMENT;
    $0.URL_DOMAIN_COUNTRY := $0.URL_DOMAIN_COUNTRY;
    $0.URL_DOMAIN_CATEGORY := $0.URL_DOMAIN_CATEGORY;
    $0.URL_IP_ADDRESS := $0.URL_IP_ADDRESS;
    $0.URL_QUALIFICATION_FLAG_BRIEF := $0.URL_QUALIFICATION_FLAG_BRIEF;
    $0.URL_QUALIFICATION_TEXT_BRIEF := $0.URL_QUALIFICATION_TEXT_BRIEF;
    $0.URL_QUALIFICATION_FLAG := $0.URL_QUALIFICATION_FLAG;
    $0.URL_QUALIFICATION_TEXT := $0.URL_QUALIFICATION_TEXT;
    $0.QUALIFIED_EMAIL := $0.QUALIFIED_EMAIL;
    $0.EMAIL_NAME := $0.EMAIL_NAME;
    $0.EMAIL_USERNAME := $0.EMAIL_USERNAME;
    $0.EMAIL_DOMAIN := $0.EMAIL_DOMAIN;
    $0.EMAIL_DOMAIN_COUNTRY := $0.EMAIL_DOMAIN_COUNTRY;
    $0.EMAIL_DOMAIN_CATEGORY := $0.EMAIL_DOMAIN_CATEGORY;
    $0.EMAIL_QUALIFICATION_FLAG_BRIEF := $0.EMAIL_QUALIFICATION_FLAG_BRIEF;
    $0.EMAIL_QUALIFICATION_TEXT_BRIEF := $0.EMAIL_QUALIFICATION_TEXT_BRIEF;
    $0.EMAIL_QUALIFICATION_FLAG := $0.EMAIL_QUALIFICATION_FLAG;
    $0.EMAIL_QUALIFICATION_TEXT := $0.EMAIL_QUALIFICATION_TEXT;
    $0.CPF := $0.CPF;
    $0.NOME := $0.NOME;
    $0.ENDERECO := $0.ENDERECO;
    $0.BAIRRO := $0.BAIRRO;
    $0.CEP := $0.CEP;
    $0.CIDADE := $0.CIDADE;
    $0.UF := $0.UF;
    $0.DDD := try_convert( $0.DDD, decimal);
    $0.TELEFONE := $0.TELEFONE;
    $0.DDD1 := $0.DDD1;
    $0.TELEFONE1 := $0.TELEFONE1;
    $0.DDD2 := $0.DDD2;
    $0.TELEFONE2 := $0.TELEFONE2;
    $0.DDD3 := $0.DDD3;
    $0.TELEFONE3 := $0.TELEFONE3;
    $0.DATANASCIMENTO := $0.DATANASCIMENTO;
    $0.SEXO := $0.SEXO;
    $0.FXRENDA := $0.FXRENDA;
    $0.MAE := $0.MAE;
    $0.Q_TIPO_LOGRADOURO := $0.Q_TIPO_LOGRADOURO;
    $0.Q_NOME_LOGRADOURO := $0.Q_NOME_LOGRADOURO;
    $0.Q_TIPO_LOC_HORIZONTAL := $0.Q_TIPO_LOC_HORIZONTAL;
    $0.Q_LOC_HORIZONTAL := $0.Q_LOC_HORIZONTAL;
    $0.Q_TIPO_LOC_VERTICAL := $0.Q_TIPO_LOC_VERTICAL;
    $0.Q_LOC_VERTICAL := $0.Q_LOC_VERTICAL;
    $0.Q_COMPLEMENTO := $0.Q_COMPLEMENTO;
    $0.Q_BAIRRO_INFERIDO := $0.Q_BAIRRO_INFERIDO;
    $0.Q_UF := $0.Q_UF;
    $0.Q_LOCALIDADE := $0.Q_LOCALIDADE;
    $0.Q_CEP := $0.Q_CEP;
    $0.Q_PREPOSICAO := $0.Q_PREPOSICAO;
    $0.Q_TITULO_LOGRADOURO := $0.Q_TITULO_LOGRADOURO;
    $0.Q_ENDERECO_FLAGS := $0.Q_ENDERECO_FLAGS;
    $0.Q_TIPO_LOCALIDADE := $0.Q_TIPO_LOCALIDADE;
    $0.Q_MUNICIPIO := $0.Q_MUNICIPIO;
    $0.Q_END_COMPLETO := $0.Q_END_COMPLETO;
    $0.Q_LONGITUDE := $0.Q_LONGITUDE;
    $0.Q_LATITUDE := $0.Q_LATITUDE;
    $0.Q_ZONA_CENS_COD := $0.Q_ZONA_CENS_COD;
    $0.Q_DENSIDADE_DEMOGRAFICA := $0.Q_DENSIDADE_DEMOGRAFICA;
    $0.Q_GEO_PRECISAO := $0.Q_GEO_PRECISAO;
    $0.MATCH_ENDERECO := $0.MATCH_ENDERECO;
    $0.MATCH_PF_CPF_ENDERECO := $0.MATCH_PF_CPF_ENDERECO;
    $0.EXISTE_LB07 := $0.EXISTE_LB07;
    $0.DT_NASC := $0.DT_NASC;
    $0.QTD1 := $0.QTD1;
    $0.ID := $0.ID;


    And the error log:


    [2011-05-09 10:52:42,781] [ERROR] [Reformat] [java.lang.NullPointerException]
    [2011-05-09 10:52:42,782] [ERROR] [WatchDog] [Phase initialization failed with reason: TratamentoInternetEmail2_refmt_dbwriter ...FAILED !
    Reason: null]
    TratamentoInternetEmail2_refmt_dbwriter ...FAILED !
    Reason: null
    at org.jetel.graph.Phase.init(Phase.java:160)
    at org.jetel.graph.runtime.WatchDog.executePhase(WatchDog.java:449)
    at org.jetel.graph.runtime.WatchDog.call(WatchDog.java:184)
    at org.jetel.graph.runtime.WatchDog.call(WatchDog.java:62)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: null
    at org.jetel.component.WrapperTL.init(WrapperTL.java:198)
    at org.jetel.component.RecordTransformTL.init(RecordTransformTL.java:64)
    at org.jetel.component.RecordTransformFactory.createTransform(RecordTransformFactory.java:256)
    at org.jetel.component.Reformat.init(Reformat.java:332)
    at org.jetel.graph.Phase.init(Phase.java:155)
    ... 6 more


    Thanks a lot!!!
  • Avatar
    avackova
    0
    Comment actions Permalink
    Is it a full stack trace?
    I'm afraid I have too little information to find the reason. Can you show the graph?
  • Avatar
    fschnack
    0
    Comment actions Permalink
    Unfortunately that's the full stack. Here is my full log:



    [2011-05-09 12:03:40,240] [INFO ] [Defaults] [Loading default properties from: defaultProperties]
    [2011-05-09 12:03:40,256] [INFO ] [Defaults] [Loading default properties from: defaultProperties]
    [2011-05-09 12:03:40,456] [WARN ] [JdbcSpecificFactory] [Some of the plugin tried to register already registered JDBC specific under same database name: 'DB2'.]
    [2011-05-09 12:03:40,456] [WARN ] [JdbcSpecificFactory] [Some of the plugin tried to register already registered JDBC specific under same database name: 'INFORMIX'.]
    [2011-05-09 12:03:40,456] [WARN ] [JdbcSpecificFactory] [Some of the plugin tried to register already registered JDBC specific under same database name: 'MSSQL'.]
    [2011-05-09 12:03:40,456] [WARN ] [JdbcSpecificFactory] [Some of the plugin tried to register already registered JDBC specific under same database name: 'MSSQL2005'.]
    [2011-05-09 12:03:40,456] [WARN ] [JdbcSpecificFactory] [Some of the plugin tried to register already registered JDBC specific under same database name: 'MYSQL'.]
    [2011-05-09 12:03:40,456] [WARN ] [JdbcSpecificFactory] [Some of the plugin tried to register already registered JDBC specific under same database name: 'ORACLE'.]
    [2011-05-09 12:03:40,456] [WARN ] [JdbcSpecificFactory] [Some of the plugin tried to register already registered JDBC specific under same database name: 'POSTGRE'.]
    [2011-05-09 12:03:40,456] [WARN ] [JdbcSpecificFactory] [Some of the plugin tried to register already registered JDBC specific under same database name: 'SYBASE'.]
    [2011-05-09 12:03:47,054] [ERROR] [Reformat] [java.lang.NullPointerException]
    [2011-05-09 12:03:47,082] [INFO ] [EngineInitializer] [Checking graph configuration...]
    [2011-05-09 12:03:47,115] [ERROR] [Reformat] [java.lang.NullPointerException]
    [2011-05-09 12:03:47,116] [INFO ] [EngineInitializer] [Graph configuration is valid.]
    [2011-05-09 12:03:47,117] [WARN ] [ConfigurationStatus] [[EscritaBancoDados3] - Unsupported feature]
    [2011-05-09 12:03:47,117] [WARN ] [ConfigurationStatus] [[TratamentoInternetEmail2_refmt_dbwriter].transform - null]
    [2011-05-09 12:03:47,117] [INFO ] [EngineInitializer] [Graph initialization (FLOW_2213107)]
    [2011-05-09 12:03:47,121] [INFO ] [TransformationGraph] [Initializing connection: ]
    [2011-05-09 12:03:47,122] [INFO ] [TransformationGraph] [DBConnection driver[org.jetel.connection.jdbc.driver.JdbcDriver@19a145d1]:jndi[null]:url[jdbc:oracle:thin:@172.16.6.202:1521:lisa]:user[consulta] ... OK]
    [2011-05-09 12:03:47,231] [INFO ] [WatchDog] [register MBean with name:org.jetel.graph.runtime:type=CLOVERJMX_2229829_0]
    [2011-05-09 12:03:47,244] [INFO ] [TransformationGraph] [*** TRANSFORMATION GRAPH CONFIGURATION ***
    ]
    [2011-05-09 12:03:47,276] [INFO ] [TransformationGraph] [--- Phase [0] ---]
    [2011-05-09 12:03:47,276] [INFO ] [TransformationGraph] [ ... nodes ...]
    [2011-05-09 12:03:47,276] [INFO ] [TransformationGraph] [ EscritaBancoDados3 : phase: 0]
    [2011-05-09 12:03:47,276] [INFO ] [TransformationGraph] [ LeituraArquivoGenerico1 : phase: 0]
    [2011-05-09 12:03:47,277] [INFO ] [TransformationGraph] [ TratamentoInternetEmail2 : phase: 0]
    [2011-05-09 12:03:47,277] [INFO ] [TransformationGraph] [ TratamentoInternetEmail2_refmt_dbwriter : phase: 0]
    [2011-05-09 12:03:47,277] [INFO ] [TransformationGraph] [ ... edges ...]
    [2011-05-09 12:03:47,277] [INFO ] [TransformationGraph] [ TratamentoInternetEmail2_0_EscritaBancoDados3_0_dbwriter type: direct]
    [2011-05-09 12:03:47,277] [INFO ] [TransformationGraph] [ TratamentoInternetEmail2_0_EscritaBancoDados3_0 type: direct]
    [2011-05-09 12:03:47,277] [INFO ] [TransformationGraph] [ LeituraArquivoGenerico1_0_TratamentoInternetEmail2_0 type: direct]
    [2011-05-09 12:03:47,277] [INFO ] [TransformationGraph] [--- end phase ---]
    [2011-05-09 12:03:47,277] [INFO ] [TransformationGraph] [*** END OF GRAPH LIST ***]
    [2011-05-09 12:03:47,288] [INFO ] [GraphExecutionListener] [Received notification from JMX.]
    [2011-05-09 12:03:47,288] [INFO ] [GraphExecutionListener] [Received notification from JMX.]
    [2011-05-09 12:03:47,288] [INFO ] [Phase] [[Clover] Initializing phase: 0]
    [2011-05-09 12:03:47,443] [INFO ] [RemoteBeanFactory] [EJB lookup for jnp://sgq01:1099/quality-ear/InternetValidatorBean/remote returning jboss.j2ee:ear=quality-ear.ear,jar=quality-ejb-1.0.13.jar,name=InternetValidatorBean,service=EJB3.]
    [2011-05-09 12:03:47,466] [ERROR] [Reformat] [java.lang.NullPointerException]
    [2011-05-09 12:03:47,473] [ERROR] [WatchDog] [Phase initialization failed with reason: TratamentoInternetEmail2_refmt_dbwriter ...FAILED !
    Reason: null]
    TratamentoInternetEmail2_refmt_dbwriter ...FAILED !
    Reason: null
    at org.jetel.graph.Phase.init(Phase.java:160)
    at org.jetel.graph.runtime.WatchDog.executePhase(WatchDog.java:449)
    at org.jetel.graph.runtime.WatchDog.call(WatchDog.java:184)
    at org.jetel.graph.runtime.WatchDog.call(WatchDog.java:62)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: null
    at org.jetel.component.WrapperTL.init(WrapperTL.java:198)
    at org.jetel.component.RecordTransformTL.init(RecordTransformTL.java:64)
    at org.jetel.component.RecordTransformFactory.createTransform(RecordTransformFactory.java:256)
    at org.jetel.component.Reformat.init(Reformat.java:332)
    at org.jetel.graph.Phase.init(Phase.java:155)
    ... 6 more
    [2011-05-09 12:03:47,498] [ERROR] [WatchDog] [!!! Phase finished with error - stopping graph run !!!]
    [2011-05-09 12:03:47,499] [INFO ] [GraphExecutionListener] [Received notification from JMX.]
  • Avatar
    avackova
    0
    Comment actions Permalink
    Unfortunately, the error in log is thrown from the catch statement. So we know, that this is NullPointerException (probably), but we don't know where the null occurs exactly.
    Try to reduce the transformation to find what is causing the problem:
    • Does the graph with only one direct mapping fail?

    • Does the graph with only one mapping with try_convert function fail?
  • Avatar
    fschnack
    0
    Comment actions Permalink
    It failed in the same way by using this TL expression:


    //#TL
    $0.QUALIFIED_URL := $0.QUALIFIED_URL;
  • Avatar
    fschnack
    0
    Comment actions Permalink
    I found the problem. The problem was that I wasn't declaring the transform() function to put the TL code inside of it. Is that mandatory, right?

    I guess clover's validation didn't get this issue.

    Thanks a lot!
  • Avatar
    avackova
    0
    Comment actions Permalink
    Yes, transform function must be defined and other functions might be defined.

    This NPE is known problem, when using input/output records in global scope and, I believe, is resolved in CloverETL 3.0

Please sign in to leave a comment.