Generate Dynamic Metadata from files

derrickwan

• April 23, 2013 00:00

• Answered

Follow

Hi all,

I am appreciate your time to read this. Here is my case.

I want to use CloverETL to process a file which upstream stream produces. Normally, upstream will let me know if they want to added new values into the file.

However, if the upstream forgot to tell me and they make the changes to the file. It will break my system.

Is any way we can dynamic generate the metadata and use it in CloverETL?

thanks,

derrick

• 

  imriskal

  • April 24, 2013 08:08

  0

  Comment actions Permalink

  Hello, Derrick,
  
  Metadata are just XML as the rest of the graph. You can externalize it into some .fmt file and open the file with text editor to see it's internal structure. You can create this file dynamically with CloverETL graph and use it in some other graph, using parameters. Let me know if you want to know more about this topic.
  
  If you use database as your input of data, you have also the second option. See http://doc.cloveretl.com/documentation/ ... adata.html
  
  Best regards,
• 

  calder

  • September 23, 2014 18:17

  0

  Comment actions Permalink

  @imriskal: Please do elaborate on the solutions you suggest. Thanks!!
• 

  novakj

  • September 24, 2014 15:13

  0

  Comment actions Permalink

  Hello,
  
  Example of simple metadata structure you can see here:
  

  <?xml version="1.0" encoding="UTF-8"?>
  <Record fieldDelimiter="|" name="myMetadata" recordDelimiter="\r\n" type="delimited">
  	<Field name="firstname" type="string"/>
  	<Field name="lastname" type="string"/>
  	<Field name="age" type="integer"/>
  </Record>
  

  
  You can generate in your graph this .fmt XML file using XMLWriter component, the metadata files are typically stored in the "meta" directory. The structure of your metadata depends on your specific needs.
  
  If you need more detailed answer, please tell us your desired usage of dynamic metadata.
  
  Best regards,
• 

  jd_surat

  • August 30, 2016 05:15

  0

  Comment actions Permalink

  Hello, Derrick,
  
  Metadata are just XML as the rest of the graph. You can externalize it into some .fmt file and open the file with text editor to see it's internal structure. You can create this file dynamically with CloverETL graph and use it in some other graph, using parameters. Let me know if you want to know more about this topic.
  
  If you use database as your input of data, you have also the second option. See http://doc.cloveretl.com/documentation/ ... adata.html
  
  Best regards,

  
  
  Hello,
  
  Can you please provide more information on how to perform following?
  
  "You can create this file dynamically with CloverETL graph and use it in some other graph, using parameters"
  
  Regards,
  Jigar
• 

  Lukas Cholasta

  • August 30, 2016 12:42

  0

  Comment actions Permalink

  Hi Jigar,
  
  Please see the attached graph. It is using a Normalizer component with some CTL2 functions to create the metadata from the input record. After the metadata is created it is converted into a XML format and written to the HDD by the XMLWriter component. Using this file is as easy as using any other externalized metadata. Once it's linked to your project metadata, it will be up to date as long as your creation graph keeps overwriting the same metadata file.
  
  If you need to use various metadata, you can pass its path in a parameter via e.g. RunGraph component.
  
  Best regards,

  • testing_data.csv
  • create_metadata.grf

Please sign in to leave a comment.