Customer Portal

Error on subsequent graph executions in embedded mode

Comments 2

  • Avatar
    admin
    0
    Comment actions Permalink
    Dear Konrad Ciborowski,

    I am afraid that behavior you reported is a bug in component reset. I created https://bug.javlin.eu/browse/CL-2276

    There is no workaround for this. You need to switch off graph reuse for now. If it is performance issue for you, you can try to do some kind of bulk processing. So graphs will be executed less often but on bigger datasets. Then graph load latency will not be that big issue.
  • Avatar
    cibor
    0
    Comment actions Permalink
    After two days of debugging of the CloverETL engine I was able to pinpoint the problem. The ReadableChannelPortIterator class contains a record field which is initialized to


    record = new DataRecord(inputPort.getMetadata());


    and then, populated by records one by one as they are being read. When the last record is read the record field is set to null and the reset() methods don't re-initialize it. So during the next run the following line in the getNextData() method

    inputPort.readRecord(record);


    takes the execution path to the DirectEdgeclass and the readRecord() method. Ultimately the following line

    record.deserialize(readBuffer);


    throws a NullPointerException (record is null).

    A solution would be to re-initialize all components but it is not that simple as various graph elements contain initialized and firstRun variables which are private (on top of that PhaseConnectionEdge contains a wasInitialized field which is also private. After a lot of trial and error I was able to implement the following workaround. It is a very, very ugly hack (modifying values of private fields by reflection is one of the worst programming practices) but short of modifying the source of the engine I was unable to find anything else. I'm not even 100% certain that my fix doesn't break something elsewhere.


    private void resetGraphPhases(TransformationGraph graph) throws NoSuchFieldException, IllegalArgumentException, IllegalAccessException, NoSuchMethodException, InvocationTargetException, ComponentNotReadyException {

    Phase phases[] = graph.getPhases();
    for (int i = 0; i < phases.length; i++) {
    Phase phase = phases[i];
    resetToInitialState(phase);
    for (Node node : phase.getNodes().values()) {
    if (node.getClass().getName().equals("org.jetel.component.DBInputTable")) {
    resetToInitialState(node);
    }
    }
    for (Edge edge : phase.getEdges().values()) {
    resetToInitialState(edge);
    Field edgeBaseField = edge.getClass().getDeclaredField("edge");
    edgeBaseField.setAccessible(true);
    EdgeBase edgeBase = (EdgeBase)edgeBaseField.get(edge);
    if (edgeBase.getClass().getName().equals("org.jetel.graph.PhaseConnectionEdge")) {
    Field wasInitialized = edgeBase.getClass().getDeclaredField("wasInitialized");
    wasInitialized.setAccessible(true);
    wasInitialized.setBoolean(edgeBase, false);
    }
    }
    phase.init();
    }
    }

    private void resetToInitialState(GraphElement graphElement) throws SecurityException, IllegalAccessException, IllegalArgumentException {
    Class clazz = graphElement.getClass();
    graphElement.free();
    while (!clazz.getName().equals("org.jetel.graph.GraphElement")) {
    clazz = clazz.getSuperclass();
    }
    Field declaredFields[] = clazz.getDeclaredFields();

    for (int i = 0; i < declaredFields.length; i++) {
    if (declaredFields[i].getName().equals("initialized")) {
    declaredFields[i].setAccessible(true);
    declaredFields[i].setBoolean(graphElement, false);
    }
    if (declaredFields[i].getName().equals("firstRun")) {
    declaredFields[i].setAccessible(true);
    declaredFields[i].setBoolean(graphElement, true);
    }
    }
    }

Please sign in to leave a comment.