I have 2 questions: (Apologize for asking trivial questions -- not a java exp!)
(1) Aggregate component does not work for dataType=long for functions like avg and stddev. Is this a bug or a limitation? If limitation, will this be supported in future?
(2) For functions like stddev, is there a limitation on how many records it can process before running out of memory -- hope not. Say for example, will it work if I pass 20mil records to do stddev?
Thanks
Akhil
(1) Aggregate component does not work for dataType=long for functions like avg and stddev. Is this a bug or a limitation? If limitation, will this be supported in future?
(2) For functions like stddev, is there a limitation on how many records it can process before running out of memory -- hope not. Say for example, will it work if I pass 20mil records to do stddev?
Thanks
Akhil
-
Hello !
To your questions:
(1) this is a current limitation, may/will be fixed in future versions. The problem with long is that for AVG or STDDEV the precision of the output can be bigger than Clover's Numeric (java.lang.Double) can accomodate. With version 1.8 there is new Decimal data type introduced which will enable to implement AVG and STDDEV even for Long
(2) If your data is sorted (meaning Aggregator does not need to keep internal hash table for counting) there is no limit even for STDDEV - the algorithm used allows incremental calculation of STDDEV.
David.
Please sign in to leave a comment.
Comments 1