Quantcast

Re: java.math.BigDecimal to Avro .avdl file help please

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: java.math.BigDecimal to Avro .avdl file help please

Nathan Fisher
Hi Steve,

I'm seeing the same issue. I think you'll need to drop down to JSON. The
IDL appears to provide a subset of the Avro specification.

I've started mapping the gaps with the following record type:

enum YesNo { YES, NO }
fixed TWO(2);

record IdlHarness {
    // primitives
    boolean bool = true;
    int i = 0;
    int iN = null;
    long l = 0;
    long lN = null;
    float f = 0.1;
    float fN = 0.1;
    double d = 0.1;
    double dN = null;
    bytes b = "banana";
    bytes bN = null;
    string s = "s";
    string sN = null;

    // complex
    YesNo en = "YES";
    YesNo enN = null;
    array<int> a = [1];
    array<int> aN = null;
    map<int> m = { "hello": 1 };
    map<int> mN = null;
    union {null, string} un = "hello";
    union {null, string} unN = null;
    TWO fix = "12";
    TWO fixN = null;

    // logical
    // decimal dec; // error
    date dt = "2017-01-01";
    date dtN = null;
    // time-millis ms; // fails
    // time-micros us; // fails
    // timestamp-millis tsms; // fails
    // timestamp-micros tsus; // fails
    // duration du; // fails
}

I've put my results into the Google sheet below. It's still a work in
progress and I'll try to turn it into a JUnit test scenario.
https://docs.google.com/spreadsheets/d/1dQAyWbrD5PmHzMJ7sVovHBoZug6jlt_x_Htc7ixdbP8/edit?usp=sharing

Kind regards,
Nathan

On Wed, 1 Feb 2017 at 05:12 Steve Sun <[hidden email]> wrote:

Hi Avro users mailing list,

I'm having trouble writing an Avro schema for java.math.BigDecimal type, I
tried the following:

1. Based on Avro official doc:
https://avro.apache.org/docs/1.8.1/spec.html#Decimal, I know I need to
define Logical Types myself to support BigDecimal, but that link gives
example only in avsc, I'm trying to figure it out in avdl.
2. Based on Avro dochttps://
avro.apache.org/docs/1.7.6/api/java/org/apache/avro/reflect/package-summary.html
and this example:
https://github.com/apache/avro/blob/master/share/test/schemas/stringables.avdl,
I wrote below avdl:



    @namespace("test")
    protocol My_Protocol_v1 {

      record BigDecimal {
            @java-class("java.math.BigDecimal") string value;
      }



But it's not working:
This IDL schema compiles fine and can generate a Java class called
BigDecimal, but I cannot really use the generated BigDecimal as
java.math.BigDecimal, what's wrong? or How should I do it?

Thanks a lot
Steve

--
- from my thumbs to your eyes
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: java.math.BigDecimal to Avro .avdl file help please

Doug Cutting-2
I believe this has already been implemented but not yet released.

It was implemented in:

https://issues.apache.org/jira/browse/AVRO-1847

This is slated to be included in the 1.8.2 release, which should soon be out.

Doug

On Wed, Feb 15, 2017 at 6:50 PM, Steve Sun <[hidden email]> wrote:

> Thanks Nathan for your info and effort.
>
> I really appreciate it.
>
> Also, I posted the same question on Stackoverflow:
> http://stackoverflow.com/questions/41969300/java-math-bigdecimal-to-avro-avdl-file,
> looks like we're not alone.
>
> Feel free to share your updates below that question as well to benefit more
> developers.
>
> Steve
>
>
>
> On Wed, Feb 15, 2017 at 8:14 AM, Nathan Fisher <[hidden email]>
> wrote:
>>
>> Hi Steve,
>>
>> I'm seeing the same issue. I think you'll need to drop down to JSON. The
>> IDL appears to provide a subset of the Avro specification.
>>
>> I've started mapping the gaps with the following record type:
>>
>> enum YesNo { YES, NO }
>> fixed TWO(2);
>>
>> record IdlHarness {
>>     // primitives
>>     boolean bool = true;
>>     int i = 0;
>>     int iN = null;
>>     long l = 0;
>>     long lN = null;
>>     float f = 0.1;
>>     float fN = 0.1;
>>     double d = 0.1;
>>     double dN = null;
>>     bytes b = "banana";
>>     bytes bN = null;
>>     string s = "s";
>>     string sN = null;
>>
>>     // complex
>>     YesNo en = "YES";
>>     YesNo enN = null;
>>     array<int> a = [1];
>>     array<int> aN = null;
>>     map<int> m = { "hello": 1 };
>>     map<int> mN = null;
>>     union {null, string} un = "hello";
>>     union {null, string} unN = null;
>>     TWO fix = "12";
>>     TWO fixN = null;
>>
>>     // logical
>>     // decimal dec; // error
>>     date dt = "2017-01-01";
>>     date dtN = null;
>>     // time-millis ms; // fails
>>     // time-micros us; // fails
>>     // timestamp-millis tsms; // fails
>>     // timestamp-micros tsus; // fails
>>     // duration du; // fails
>> }
>>
>> I've put my results into the Google sheet below. It's still a work in
>> progress and I'll try to turn it into a JUnit test scenario.
>>
>> https://docs.google.com/spreadsheets/d/1dQAyWbrD5PmHzMJ7sVovHBoZug6jlt_x_Htc7ixdbP8/edit?usp=sharing
>>
>> Kind regards,
>> Nathan
>>
>> On Wed, 1 Feb 2017 at 05:12 Steve Sun <[hidden email]> wrote:
>>>
>>> Hi Avro users mailing list,
>>>
>>> I'm having trouble writing an Avro schema for java.math.BigDecimal type,
>>> I tried the following:
>>>
>>> 1. Based on Avro official doc:
>>> https://avro.apache.org/docs/1.8.1/spec.html#Decimal, I know I need to
>>> define Logical Types myself to support BigDecimal, but that link gives
>>> example only in avsc, I'm trying to figure it out in avdl.
>>> 2. Based on Avro
>>> dochttps://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/reflect/package-summary.html
>>> and this
>>> example:https://github.com/apache/avro/blob/master/share/test/schemas/stringables.avdl,
>>> I wrote below avdl:
>>>
>>>
>>>
>>>     @namespace("test")
>>>     protocol My_Protocol_v1 {
>>>
>>>       record BigDecimal {
>>>             @java-class("java.math.BigDecimal") string value;
>>>       }
>>>
>>>
>>>
>>> But it's not working:
>>> This IDL schema compiles fine and can generate a Java class called
>>> BigDecimal, but I cannot really use the generated BigDecimal as
>>> java.math.BigDecimal, what's wrong? or How should I do it?
>>>
>>> Thanks a lot
>>> Steve
>>
>> --
>> - from my thumbs to your eyes
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: java.math.BigDecimal to Avro .avdl file help please

Steve Sun
Great! Thanks so much Doug for pointing it out!

On Thu, Feb 16, 2017 at 9:37 AM, Doug Cutting <[hidden email]> wrote:

> I believe this has already been implemented but not yet released.
>
> It was implemented in:
>
> https://issues.apache.org/jira/browse/AVRO-1847
>
> This is slated to be included in the 1.8.2 release, which should soon be
> out.
>
> Doug
>
> On Wed, Feb 15, 2017 at 6:50 PM, Steve Sun <[hidden email]> wrote:
> > Thanks Nathan for your info and effort.
> >
> > I really appreciate it.
> >
> > Also, I posted the same question on Stackoverflow:
> > http://stackoverflow.com/questions/41969300/java-math-
> bigdecimal-to-avro-avdl-file,
> > looks like we're not alone.
> >
> > Feel free to share your updates below that question as well to benefit
> more
> > developers.
> >
> > Steve
> >
> >
> >
> > On Wed, Feb 15, 2017 at 8:14 AM, Nathan Fisher <[hidden email]>
> > wrote:
> >>
> >> Hi Steve,
> >>
> >> I'm seeing the same issue. I think you'll need to drop down to JSON. The
> >> IDL appears to provide a subset of the Avro specification.
> >>
> >> I've started mapping the gaps with the following record type:
> >>
> >> enum YesNo { YES, NO }
> >> fixed TWO(2);
> >>
> >> record IdlHarness {
> >>     // primitives
> >>     boolean bool = true;
> >>     int i = 0;
> >>     int iN = null;
> >>     long l = 0;
> >>     long lN = null;
> >>     float f = 0.1;
> >>     float fN = 0.1;
> >>     double d = 0.1;
> >>     double dN = null;
> >>     bytes b = "banana";
> >>     bytes bN = null;
> >>     string s = "s";
> >>     string sN = null;
> >>
> >>     // complex
> >>     YesNo en = "YES";
> >>     YesNo enN = null;
> >>     array<int> a = [1];
> >>     array<int> aN = null;
> >>     map<int> m = { "hello": 1 };
> >>     map<int> mN = null;
> >>     union {null, string} un = "hello";
> >>     union {null, string} unN = null;
> >>     TWO fix = "12";
> >>     TWO fixN = null;
> >>
> >>     // logical
> >>     // decimal dec; // error
> >>     date dt = "2017-01-01";
> >>     date dtN = null;
> >>     // time-millis ms; // fails
> >>     // time-micros us; // fails
> >>     // timestamp-millis tsms; // fails
> >>     // timestamp-micros tsus; // fails
> >>     // duration du; // fails
> >> }
> >>
> >> I've put my results into the Google sheet below. It's still a work in
> >> progress and I'll try to turn it into a JUnit test scenario.
> >>
> >> https://docs.google.com/spreadsheets/d/1dQAyWbrD5PmHzMJ7sVovHBoZug6jl
> t_x_Htc7ixdbP8/edit?usp=sharing
> >>
> >> Kind regards,
> >> Nathan
> >>
> >> On Wed, 1 Feb 2017 at 05:12 Steve Sun <[hidden email]> wrote:
> >>>
> >>> Hi Avro users mailing list,
> >>>
> >>> I'm having trouble writing an Avro schema for java.math.BigDecimal
> type,
> >>> I tried the following:
> >>>
> >>> 1. Based on Avro official doc:
> >>> https://avro.apache.org/docs/1.8.1/spec.html#Decimal, I know I need to
> >>> define Logical Types myself to support BigDecimal, but that link gives
> >>> example only in avsc, I'm trying to figure it out in avdl.
> >>> 2. Based on Avro
> >>> dochttps://avro.apache.org/docs/1.7.6/api/java/org/
> apache/avro/reflect/package-summary.html
> >>> and this
> >>> example:https://github.com/apache/avro/blob/master/share/
> test/schemas/stringables.avdl,
> >>> I wrote below avdl:
> >>>
> >>>
> >>>
> >>>     @namespace("test")
> >>>     protocol My_Protocol_v1 {
> >>>
> >>>       record BigDecimal {
> >>>             @java-class("java.math.BigDecimal") string value;
> >>>       }
> >>>
> >>>
> >>>
> >>> But it's not working:
> >>> This IDL schema compiles fine and can generate a Java class called
> >>> BigDecimal, but I cannot really use the generated BigDecimal as
> >>> java.math.BigDecimal, what's wrong? or How should I do it?
> >>>
> >>> Thanks a lot
> >>> Steve
> >>
> >> --
> >> - from my thumbs to your eyes
> >
> >
>
Loading...