multiplying output

schabluk

• August 22, 2008 00:00

• Answered

Follow

In every Delimiter Data Reader that I'm using for reading flat files i'm getting endless loop. For ex. I have flat file with 30 rows, but on output it gives milions of rows! What's wrong with my env?

INFO [main] - Checking graph configuration...
DEBUG [main] - Opening input file D:/ETL/validator/data-in/file.DAT
DEBUG [main] - Reading input file D:/ETL/validator/data-in/file.DAT
INFO [WatchDog] - Thread started.
INFO [WatchDog] - Running on 2 CPU(s) max available memory for JVM 65088 KB
INFO [WatchDog] - [Clover] Initializing phase: 0
DEBUG [WatchDog] - initializing edges:
DEBUG [WatchDog] - all edges initialized successfully...
DEBUG [WatchDog] - initializing nodes:
DEBUG [WatchDog] - Opening input file D:/ETL/validator/data-in/file.DAT
DEBUG [WatchDog] - Reading input file D:/ETL/validator/data-in/file.DAT
DEBUG [WatchDog] - DELIMITED_DATA_READER0 ...OK
DEBUG [WatchDog] - TRASH0 ...OK
INFO [WatchDog] - [Clover] phase: 0 initialized successfully.
INFO [WatchDog] - Starting up all nodes in phase [0]
DEBUG [WatchDog] - DELIMITED_DATA_READER0 ... started
DEBUG [WatchDog] - TRASH0 ... started
INFO [WatchDog] - Sucessfully started all nodes in phase!
INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **-------------------
INFO [WatchDog] - Time: 22/08/08 10:35:09
INFO [WatchDog] - Node Status Port #Records #KB Rec/s KB/s
INFO [WatchDog] - ----------------------------------------------------------------------------------
INFO [WatchDog] - DELIMITED_DATA_READER0 RUNNING
INFO [WatchDog] - %cpu:0.98 Out:0 1738759 237720 -4 -1
INFO [WatchDog] - TRASH0 RUNNING
INFO [WatchDog] - %cpu:0.83 In:0 1738759 237720 -4 -1
INFO [WatchDog] - ---------------------------------** End of Log **--------------------------------
INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **-------------------
INFO [WatchDog] - Time: 22/08/08 10:35:14
INFO [WatchDog] - Node Status Port #Records #KB Rec/s KB/s
INFO [WatchDog] - ----------------------------------------------------------------------------------
INFO [WatchDog] - DELIMITED_DATA_READER0 RUNNING
INFO [WatchDog] - %cpu:0.97 Out:0 3521796 481495 363440 49689
INFO [WatchDog] - TRASH0 RUNNING
INFO [WatchDog] - %cpu:0.82 In:0 3521801 481496 363441 49689
INFO [WatchDog] - ---------------------------------** End of Log **--------------------------------

• 

  dpavlis

  • August 22, 2008 09:20

  0

  Comment actions Permalink

  Try to use UniversalDataReader - that is a successor to DelimitedDataReader which is able to handle more problems in imput data.
  
  Could you send/post also the .grf file which is causing the problem ?
  
  Nonetheless, your problem is quite odd. We have not seen this behavior before with this component.
• 

  schabluk

  • August 22, 2008 10:04

  0

  Comment actions Permalink

  Same story with Universal Data Reader.
  
  Graph: http://www.sendspace.com/file/lcfs8w
• 

  schabluk

  • August 22, 2008 10:25

  0

  Comment actions Permalink

  This is for sure environment problem (probably eclipse). 1/10 times graph works fine.
• 

  schabluk

  • August 23, 2008 13:24

  0

  Comment actions Permalink

  Furthermore, the output file includes only those values, which are Autofilling. Non of the "in file" data is proceeded to output.
• 

  jausperger

  • September 04, 2008 07:43

  0

  Comment actions Permalink

  I checked up your graph file and I didn't find a metadata file. I suppose you have autofilling for all metadata. Ie:
  
  <Record fieldDelimiter="|" name="recordName1" recordDelimiter="\n" type="delimited">
  <Field auto_filling="source_name" name="field1" type="string"/>
  </Record>
  
  Then you are right. The clover should throw an exception because there isn't a metadata description how to read the data.
• 

  jausperger

  • September 04, 2008 08:05

  0

  Comment actions Permalink

  If you want to get only file name I suppose to use SystemExecute component. Something like dir -... > fileNames.txt and then use a reader.

Please sign in to leave a comment.