I have a need to share a map of dynamically created key-value pairs between two nodes that are not connected in my graph and which operate on different record types (different metadata). The dictionary seems like a structure built for this purpose, but with CTL2 I must predefine the keys which does not allow it to be used for my purpose, since my keys are not known until runtime when they are calculated. I explored creating a Lookup or some other read/write-able intermittent store - this is not only extra processing steps, but also I cannot figure out how to get the Reformat node to send the keys from my map to a different output port than the port I am sending records to (transform method only allows one return value per record processed). So I could attach the relevant keys to each record and have a later node parse them out, but again this seems like unnecessary complexity.
Effectively what I want is the ability to create a global variable and reference it in any node. I would also like to be able to persist the data in the node to ensure normalization between runs as well. How would one achieve this?
Thanks!
Effectively what I want is the ability to create a global variable and reference it in any node. I would also like to be able to persist the data in the node to ensure normalization between runs as well. How would one achieve this?
Thanks!
-
There is no way to share variables across components in CTL. And that is made on purpose.
Dictionary is meant rather for an interface definition (for launch services) to define graph inputs, outputs and parametrization.
What you are looking for is lookup table. Reading from lookup table can be done either using LookupJoin component or directly in CTL code using the following syntax:
LookupTableRecordMetadata lookupMatch = lookup(MyLookupTable).get($0.LookupKey);
if(lookupMatch != null) {
// match found
$0.outputField = lookupMatch.lookupValue;
} else {
// match not found
}
Writing into lookup table can be done using LookupTableReaderWriter to which you send the lookup table record.
Reformat allows sending data to multiple outputs as well. the return value from transform() function can be:- SKIP - no record returned.
- >=0 - record generated on specified output port.
- ALL - records generated on all output ports.
The last option is what you need.
If there is Reformat with three output edges with the following CTL code:
function integer transform() {
$0.Field1 = "a";
$1.Field2 = "b";
// the following will not return any record on any output port.
return SKIP;
// the following will return record on output port 0 only.
return 0;
// the following will return record on output port 1 only.
return 1;
// the following will return record on output port 2 only (filled with default values).
return 2;
// the following will return records on all output ports.
return ALL;
}
So, you can use reformat to branch records into several feeds; e.g. send data to LookupTableReaderWriter (you may want to use ExtFilter component in between Reformat and LookupTableReaderWriter to filter out empty records) and to regular output processing. -
I believe you may not understand what I am attempting to do, and why the LookupTable is not effective, I will clarify below. Also, why would you not want to have the ability to pass data between nodes without exporting it in some way? That seems inefficient, although I can understand how that could cause hard-to-debug synch issues between nodes in the same phase.
My reformater cannot send my dynamically-calculated "key-value pair" records to a different port because they are being produced from a record which needs to go to port 0. Since the transform method only has 1 return value, I can't send the record to port 0 and the discovered key-value pairs to port 1. This is further complicated because, I can't send the discovered key-value pairs to port 1 until I have processed all the incoming records (and hence emitted them to port 0).
If you could help me with a way to "create" new records in the Reformatter this might work - i.e. I process all the incoming records and emit them to port 0, then emit "made up" records for each key-value pair (that don't correlate to a record from an input port) to port 1. I guess I could just tag the very last record emitted on port 0 with all the key-value pairs, then siphon them off with a copy/filter/reformatter to send to LookupTableReaderWriter, but how does one detect that the current record calling the transform method of the Reformatter is the last record from the input port? -
Can you share more details about the logic which is used to decide which records should be sent to port 1? From what you are saying, I do not think that reformat is the right component to implement this kind of processing. It actually may be more efficient to implement this functionality using set of components in batch-manner (which can be faster) rather that procedural-manner.
Anyway, if you are still going with the procedural approach, you should be able to get it working using RollUp component. You can store the records in memory (array of records) and once all the records on input are processed and sent to output port 0, you can generate records on output port 1. Just be aware of memory needed for storing all the records. -
Here is example code for what I am doing in my Reformatter:
// This is the map of "normalized" soundex values found in an attribute across all records.
// Needs to be emitted as key-value pair records for other nodes to reference.
map[string, string] sounds;
function integer transform() {
// Other mappings here...
if($0.Position_Title!=null) {
$0.Position_Title = $0.Position_Title.trim();
string[] parts = $0.Position_Title.split("[^A-Za-z]+");
string delimited_soundex = "";
foreach(string part : parts) {
if(!part.isBlank()) {
string sound = part.soundex();
if(sounds[sound]==null || sounds[sound].isBlank())
sounds[sound] = sound + " " + part;
delimited_soundex = delimited_soundex + sounds[sound] + '${MULTIASSIGN_DELIMITER}';
}
}
$0.Position_Title_Sound = delimited_soundex.replace('${MULTIASSIGN_DELIMITER}$', "");
}
return 0;
}
I hadn't thought of trying the Rollup component, I will look into that. -
I now see what you are trying to achieve. At the moment there are two ways to do that: - The way you do it currently, i.e. store sounds in memory. Once this Reformat is done, output data to LookupTableReaderWriter. In other components, you would need to access the lookup table instead of sounds array. If the other components, need to write to the lookup table, it needs to be done in a similar way as described previously.
- Implement transformation in Java where you actually can directly write records into a lookup table. Sample code for Reformat ( I am using metadata with two string fields field1 and field2; field1 is used as a key):
import org.jetel.component.DataRecordTransform;
import org.jetel.data.DataRecord;
import org.jetel.data.GetVal;
import org.jetel.data.RecordKey;
import org.jetel.data.SetVal;
import org.jetel.data.lookup.Lookup;
import org.jetel.data.lookup.LookupTable;
import org.jetel.exception.ComponentNotReadyException;
import org.jetel.exception.TransformException;
public class LookupTableJava extends DataRecordTransform {
LookupTable lookupTable;
Lookup lookup;
DataRecord lookupRecord;
@Override
public boolean init() throws ComponentNotReadyException {
// get and initialise all structures needed for lookups
lookupTable = getGraph().getLookupTable("LookupTable0");
if (!lookupTable.isInitialized()) lookupTable.init();
lookupRecord = new DataRecord(lookupTable.getKeyMetadata());
lookupRecord.init();
lookup = lookupTable.createLookup(new RecordKey(new String[] {"field1"}, lookupTable.getKeyMetadata()), lookupRecord);
return super.init();
}
@Override
public int transform(DataRecord[] input, DataRecord[] output) throws TransformException {
// perform lookup
SetVal.setString(lookupRecord, "field1", GetVal.getString(input[0], "field1"));
lookup.seek();
if(lookup.hasNext()) {
// when match found in lookup table - generate output
SetVal.setString(output[0], "field2", GetVal.getString(lookup.next() , "field2"));
} else {
// when match not found in lookup table - generate output
SetVal.setString(output[0], "field2", "not found");
// insert new record into lookup table
DataRecord newLookupRecord = new DataRecord(lookupTable.getMetadata());
newLookupRecord.init();
SetVal.setString(newLookupRecord, "field1", GetVal.getString(input[0], "field1"));
SetVal.setString(newLookupRecord, "field2", GetVal.getString(input[0], "field2"));
// insert new record into lookup table - this is a operation not supported in CTL2
lookupTable.put(newLookupRecord);
}
// generate output
SetVal.setString(output[0], "field1", GetVal.getString(input[0], "field1"));
return 0;
}
}
By the way, I have raised a feature request CL-2016 for a CTL2 function allowing writing into lookup table.
Please sign in to leave a comment.
Comments 5