[jira] [Commented] (AVRO-2042) Can't reliably write decimal/BigDecimal values to file

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[jira] [Commented] (AVRO-2042) Can't reliably write decimal/BigDecimal values to file

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/AVRO-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050746#comment-16050746 ]

Ryan Blue commented on AVRO-2042:
---------------------------------

The problem is that you're converting floats, which causes the BigDecimal code to guess what the intended scale is. Instead, you should use the String constructor. You can see the tests for an example: https://github.com/apache/avro/blob/master/lang/java/avro/src/test/java/org/apache/avro/generic/TestGenericLogicalTypes.java#L140

Another option is to set the scale using [setScale|https://docs.oracle.com/javase/7/docs/api/java/math/BigDecimal.html#setScale(int,%20java.math.RoundingMode)] to normalize incoming doubles or floats.

> Can't reliably write decimal/BigDecimal  values to file
> -------------------------------------------------------
>
>                 Key: AVRO-2042
>                 URL: https://issues.apache.org/jira/browse/AVRO-2042
>             Project: Avro
>          Issue Type: Bug
>    Affects Versions: 1.8.2
>            Reporter: Fred Cohen
>         Attachments: AvroDecimal.avsc, AvroDecimalTest.java
>
>
> Attempting to write some decimal values fails.
> Here's the schema I created for this test:
> {
>   "namespace":"org.test",
>   "type": "record",
>   "name": "DecimalTest",
>   "fields": [
>     {
>       "name": "decimalVal",
>       "type": {"type": "bytes", "logicalType":"decimal","precision": 6, "scale": 4}
>        }
>   ]
> }
> The problem is that to my knowledge I can't control the scale used by BigDecimal.
> While the first value uses a scale of 4, the second one is scale 0.
>         decimals.add(new BigDecimal(12.01F, mathContext));
>         decimals.add(new BigDecimal(12.00F, mathContext));
> It leads to these errors during the write operation:
> org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroTypeException: Cannot encode decimal with scale 0 as scale 4
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> at AvroDecimalTest.writeDecimalTest(AvroDecimalTest.java:67)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: org.apache.avro.AvroTypeException: Cannot encode decimal with scale 0 as scale 4
> at org.apache.avro.Conversions$DecimalConversion.toBytes(Conversions.java:92)
> at org.apache.avro.Conversions$DecimalConversion.toBytes(Conversions.java:62)
> at org.apache.avro.Conversions.convertToRawType(Conversions.java:218)
> at org.apache.avro.generic.GenericDatumWriter.convert(GenericDatumWriter.java:95)
> at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:84)
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:62)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> ... 23 more



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Loading...