Quantcast

Incorrect Json schema generated for @java_class

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Incorrect Json schema generated for @java_class

avronewbie
For the IDL field:

@java_class("java.util.ArrayList") array<int> arrayOfInts;

the JSON schema generated by the avro-maven-plugin is

{\"name\":\"arrayOfInts\",\"type\":{\"type\":\"array\",\"items\":\"int\",\"java_class\":\"java.util.ArrayList\"}}

Shouldn't it be generated in JSON as java-class (with hypen instead of underscore) instead of java_class? Because of this reading from avro file is not working.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Incorrect Json schema generated for @java_class

Doug Cutting
Annotation names in IDL are copied directly to schema properties with
the same names.

The bug is in the IDL documentation, which uses the annotation
@java_class in an example, which is meaningless.  The Java
implementation requires the "java-class" schema property, so the
annotation used should be @java-class.

I filed an issue in Jira for this.

https://issues.apache.org/jira/browse/AVRO-1264

Doug

On Wed, Feb 27, 2013 at 7:18 PM, avronewbie <[hidden email]> wrote:

> For the IDL field:
>
> @java_class("java.util.ArrayList") array<int> arrayOfInts;
>
> the JSON schema generated by the avro-maven-plugin is
>
> {\"name\":\"arrayOfInts\",\"type\":{\"type\":\"array\",\"items\":\"int\",\"java_class\":\"java.util.ArrayList\"}}
>
> Shouldn't it be generated in JSON as java-class (with hypen instead of
> underscore) instead of java_class? Because of this reading from avro file is
> not working.
>
>
>
>
> --
> View this message in context: http://apache-avro.679487.n3.nabble.com/Incorrect-Json-schema-generated-for-java-class-tp4026430.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Incorrect Json schema generated for @java_class

Alexandre Normand
Hey Doug,
I was also wondering about something related. In reflection, we can use the java-class property to define the java class for stringable types. Could we find a way to bring some of this to the Specific generated classes so that a stringable type (I'm specifically thinking of BigDecimal and BigInteger) could be exposed as the java-class type instead of Strings? I started looking into this and there seem to be two possibilities, none of which I'm terribly excited about:

1. Use some reflection to invoke the string constructor for the java-class…but that seems dirty as the Specific API probably shouldn't do reflection, right?
2. Handle a very specific list of java-class types (such as BigDecimal and BigInteger) and have code that deals specifically with these in Reader/Writer.

Thoughts?
-- 
Alex

On Thursday, February 28, 2013 at 9:19 AM, Doug Cutting wrote:

Annotation names in IDL are copied directly to schema properties with
the same names.

The bug is in the IDL documentation, which uses the annotation
@java_class in an example, which is meaningless. The Java
implementation requires the "java-class" schema property, so the
annotation used should be @java-class.

I filed an issue in Jira for this.


Doug

On Wed, Feb 27, 2013 at 7:18 PM, avronewbie <[hidden email]> wrote:
For the IDL field:

@java_class("java.util.ArrayList") array<int> arrayOfInts;

the JSON schema generated by the avro-maven-plugin is

{\"name\":\"arrayOfInts\",\"type\":{\"type\":\"array\",\"items\":\"int\",\"java_class\":\"java.util.ArrayList\"}}

Shouldn't it be generated in JSON as java-class (with hypen instead of
underscore) instead of java_class? Because of this reading from avro file is
not working.




--
Sent from the Avro - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Incorrect Json schema generated for @java_class

Doug Cutting
You could get the specific compiler to emit the desired type by
modifying the method SpecificCompiler#getStringType() to take the
Schema as a parameter and, if it has a java-class attribute, return
that.  Then things should work so long as you use ReflectDatumReader
and ReflectDatumWriter.  Would that suffice?

Doug

On Thu, Feb 28, 2013 at 9:25 AM, Alexandre Normand
<[hidden email]> wrote:

> Hey Doug,
> I was also wondering about something related. In reflection, we can use the
> java-class property to define the java class for stringable types. Could we
> find a way to bring some of this to the Specific generated classes so that a
> stringable type (I'm specifically thinking of BigDecimal and BigInteger)
> could be exposed as the java-class type instead of Strings? I started
> looking into this and there seem to be two possibilities, none of which I'm
> terribly excited about:
>
> 1. Use some reflection to invoke the string constructor for the
> java-class…but that seems dirty as the Specific API probably shouldn't do
> reflection, right?
> 2. Handle a very specific list of java-class types (such as BigDecimal and
> BigInteger) and have code that deals specifically with these in
> Reader/Writer.
>
> Thoughts?
> --
> Alex
>
> On Thursday, February 28, 2013 at 9:19 AM, Doug Cutting wrote:
>
> Annotation names in IDL are copied directly to schema properties with
> the same names.
>
> The bug is in the IDL documentation, which uses the annotation
> @java_class in an example, which is meaningless. The Java
> implementation requires the "java-class" schema property, so the
> annotation used should be @java-class.
>
> I filed an issue in Jira for this.
>
> https://issues.apache.org/jira/browse/AVRO-1264
>
> Doug
>
> On Wed, Feb 27, 2013 at 7:18 PM, avronewbie <[hidden email]> wrote:
>
> For the IDL field:
>
> @java_class("java.util.ArrayList") array<int> arrayOfInts;
>
> the JSON schema generated by the avro-maven-plugin is
>
> {\"name\":\"arrayOfInts\",\"type\":{\"type\":\"array\",\"items\":\"int\",\"java_class\":\"java.util.ArrayList\"}}
>
> Shouldn't it be generated in JSON as java-class (with hypen instead of
> underscore) instead of java_class? Because of this reading from avro file is
> not working.
>
>
>
>
> --
> View this message in context:
> http://apache-avro.679487.n3.nabble.com/Incorrect-Json-schema-generated-for-java-class-tp4026430.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Incorrect Json schema generated for @java_class

Alexandre Normand
I agree with the SpecificCompiler#getStringType() change. I actually have an early draft patch that has this change. As for using the ReflectDatum Reader/Writer, I guess that might be sufficient for some use cases but, unfortunately, it isn't for mine ;). We're using kiji (https://github.com/kijiproject) and I don't have control over the Reader/Writer implementation being used. 

I don't want to dirty the Generic/Specific reader/writers but it might be some desirable feature to support JDK classes like the BigDecimal/BigInteger? 

-- 
Alex

On Thursday, February 28, 2013 at 10:04 AM, Doug Cutting wrote:

You could get the specific compiler to emit the desired type by
modifying the method SpecificCompiler#getStringType() to take the
Schema as a parameter and, if it has a java-class attribute, return
that. Then things should work so long as you use ReflectDatumReader
and ReflectDatumWriter. Would that suffice?

Doug

On Thu, Feb 28, 2013 at 9:25 AM, Alexandre Normand
Hey Doug,
I was also wondering about something related. In reflection, we can use the
java-class property to define the java class for stringable types. Could we
find a way to bring some of this to the Specific generated classes so that a
stringable type (I'm specifically thinking of BigDecimal and BigInteger)
could be exposed as the java-class type instead of Strings? I started
looking into this and there seem to be two possibilities, none of which I'm
terribly excited about:

1. Use some reflection to invoke the string constructor for the
java-class…but that seems dirty as the Specific API probably shouldn't do
reflection, right?
2. Handle a very specific list of java-class types (such as BigDecimal and
BigInteger) and have code that deals specifically with these in
Reader/Writer.

Thoughts?
--
Alex

On Thursday, February 28, 2013 at 9:19 AM, Doug Cutting wrote:

Annotation names in IDL are copied directly to schema properties with
the same names.

The bug is in the IDL documentation, which uses the annotation
@java_class in an example, which is meaningless. The Java
implementation requires the "java-class" schema property, so the
annotation used should be @java-class.

I filed an issue in Jira for this.


Doug

On Wed, Feb 27, 2013 at 7:18 PM, avronewbie <[hidden email]> wrote:

For the IDL field:

@java_class("java.util.ArrayList") array<int> arrayOfInts;

the JSON schema generated by the avro-maven-plugin is

{\"name\":\"arrayOfInts\",\"type\":{\"type\":\"array\",\"items\":\"int\",\"java_class\":\"java.util.ArrayList\"}}

Shouldn't it be generated in JSON as java-class (with hypen instead of
underscore) instead of java_class? Because of this reading from avro file is
not working.




--
View this message in context:
Sent from the Avro - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Incorrect Json schema generated for @java_class

avronewbie
In reply to this post by Doug Cutting
Thanks Doug for the reply.

I tried with the annotation @java-class but the compiler throws this exception:

org.apache.avro.compiler.idl.ParseException: Encountered " <STUFF_TO_IGNORE> "- ""
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Incorrect Json schema generated for @java_class

Doug Cutting
Oops, you're right.  Dashes are not currently allowed in IDL property
names.  I think it would be better to permit dashes there than to
automatically convert underscores to dashes or to change the name of
the property that's used by ReflectDatumReader.

I filed a Jira issue for this:

https://issues.apache.org/jira/browse/AVRO-1267

Doug

On Thu, Feb 28, 2013 at 12:10 PM, avronewbie <[hidden email]> wrote:

> Thanks Doug for the reply.
>
> I tried with the annotation @java-class but the compiler throws this
> exception:
>
> org.apache.avro.compiler.idl.ParseException: Encountered " <STUFF_TO_IGNORE>
> "- ""
>
>
>
> --
> View this message in context: http://apache-avro.679487.n3.nabble.com/Incorrect-Json-schema-generated-for-java-class-tp4026430p4026449.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Incorrect Json schema generated for @java_class

Doug Cutting
In reply to this post by Alexandre Normand
If you cannot use ReflectDatumReader and ReflectDatumWriter then
perhaps we could move the readString() and writeString()
implementations from these to SpecificData.  One can imagine also
wishing to, e.g., specify the collection class used for arrays and
maps via the java-class schema property with specific data.  The
reflect package is more about automatically inferring schemas through
introspection.  That's not the case here, rather this is just
permitting more control over what classes are used to implement one's
schemas, which seems reasonably within the scope of specific.

Doug

On Thu, Feb 28, 2013 at 10:36 AM, Alexandre Normand
<[hidden email]> wrote:

> I agree with the SpecificCompiler#getStringType() change. I actually have an
> early draft patch that has this change. As for using the ReflectDatum
> Reader/Writer, I guess that might be sufficient for some use cases but,
> unfortunately, it isn't for mine ;). We're using kiji
> (https://github.com/kijiproject) and I don't have control over the
> Reader/Writer implementation being used.
>
> I don't want to dirty the Generic/Specific reader/writers but it might be
> some desirable feature to support JDK classes like the
> BigDecimal/BigInteger?
>
> --
> Alex
>
> On Thursday, February 28, 2013 at 10:04 AM, Doug Cutting wrote:
>
> You could get the specific compiler to emit the desired type by
> modifying the method SpecificCompiler#getStringType() to take the
> Schema as a parameter and, if it has a java-class attribute, return
> that. Then things should work so long as you use ReflectDatumReader
> and ReflectDatumWriter. Would that suffice?
>
> Doug
>
> On Thu, Feb 28, 2013 at 9:25 AM, Alexandre Normand
> <[hidden email]> wrote:
>
> Hey Doug,
> I was also wondering about something related. In reflection, we can use the
> java-class property to define the java class for stringable types. Could we
> find a way to bring some of this to the Specific generated classes so that a
> stringable type (I'm specifically thinking of BigDecimal and BigInteger)
> could be exposed as the java-class type instead of Strings? I started
> looking into this and there seem to be two possibilities, none of which I'm
> terribly excited about:
>
> 1. Use some reflection to invoke the string constructor for the
> java-class…but that seems dirty as the Specific API probably shouldn't do
> reflection, right?
> 2. Handle a very specific list of java-class types (such as BigDecimal and
> BigInteger) and have code that deals specifically with these in
> Reader/Writer.
>
> Thoughts?
> --
> Alex
>
> On Thursday, February 28, 2013 at 9:19 AM, Doug Cutting wrote:
>
> Annotation names in IDL are copied directly to schema properties with
> the same names.
>
> The bug is in the IDL documentation, which uses the annotation
> @java_class in an example, which is meaningless. The Java
> implementation requires the "java-class" schema property, so the
> annotation used should be @java-class.
>
> I filed an issue in Jira for this.
>
> https://issues.apache.org/jira/browse/AVRO-1264
>
> Doug
>
> On Wed, Feb 27, 2013 at 7:18 PM, avronewbie <[hidden email]> wrote:
>
> For the IDL field:
>
> @java_class("java.util.ArrayList") array<int> arrayOfInts;
>
> the JSON schema generated by the avro-maven-plugin is
>
> {\"name\":\"arrayOfInts\",\"type\":{\"type\":\"array\",\"items\":\"int\",\"java_class\":\"java.util.ArrayList\"}}
>
> Shouldn't it be generated in JSON as java-class (with hypen instead of
> underscore) instead of java_class? Because of this reading from avro file is
> not working.
>
>
>
>
> --
> View this message in context:
> http://apache-avro.679487.n3.nabble.com/Incorrect-Json-schema-generated-for-java-class-tp4026430.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Incorrect Json schema generated for @java_class

Alexandre Normand
Cool. Thanks. I had a patch in progress for this that I haven't completed but I'll resume work. I'll create a ticket now so it's being tracked. 

-- 
Alex

On Thursday, February 28, 2013 at 1:58 PM, Doug Cutting wrote:

If you cannot use ReflectDatumReader and ReflectDatumWriter then
perhaps we could move the readString() and writeString()
implementations from these to SpecificData. One can imagine also
wishing to, e.g., specify the collection class used for arrays and
maps via the java-class schema property with specific data. The
reflect package is more about automatically inferring schemas through
introspection. That's not the case here, rather this is just
permitting more control over what classes are used to implement one's
schemas, which seems reasonably within the scope of specific.

Doug

On Thu, Feb 28, 2013 at 10:36 AM, Alexandre Normand
I agree with the SpecificCompiler#getStringType() change. I actually have an
early draft patch that has this change. As for using the ReflectDatum
Reader/Writer, I guess that might be sufficient for some use cases but,
unfortunately, it isn't for mine ;). We're using kiji
(https://github.com/kijiproject) and I don't have control over the
Reader/Writer implementation being used.

I don't want to dirty the Generic/Specific reader/writers but it might be
some desirable feature to support JDK classes like the
BigDecimal/BigInteger?

--
Alex

On Thursday, February 28, 2013 at 10:04 AM, Doug Cutting wrote:

You could get the specific compiler to emit the desired type by
modifying the method SpecificCompiler#getStringType() to take the
Schema as a parameter and, if it has a java-class attribute, return
that. Then things should work so long as you use ReflectDatumReader
and ReflectDatumWriter. Would that suffice?

Doug

On Thu, Feb 28, 2013 at 9:25 AM, Alexandre Normand

Hey Doug,
I was also wondering about something related. In reflection, we can use the
java-class property to define the java class for stringable types. Could we
find a way to bring some of this to the Specific generated classes so that a
stringable type (I'm specifically thinking of BigDecimal and BigInteger)
could be exposed as the java-class type instead of Strings? I started
looking into this and there seem to be two possibilities, none of which I'm
terribly excited about:

1. Use some reflection to invoke the string constructor for the
java-class…but that seems dirty as the Specific API probably shouldn't do
reflection, right?
2. Handle a very specific list of java-class types (such as BigDecimal and
BigInteger) and have code that deals specifically with these in
Reader/Writer.

Thoughts?
--
Alex

On Thursday, February 28, 2013 at 9:19 AM, Doug Cutting wrote:

Annotation names in IDL are copied directly to schema properties with
the same names.

The bug is in the IDL documentation, which uses the annotation
@java_class in an example, which is meaningless. The Java
implementation requires the "java-class" schema property, so the
annotation used should be @java-class.

I filed an issue in Jira for this.


Doug

On Wed, Feb 27, 2013 at 7:18 PM, avronewbie <[hidden email]> wrote:

For the IDL field:

@java_class("java.util.ArrayList") array<int> arrayOfInts;

the JSON schema generated by the avro-maven-plugin is

{\"name\":\"arrayOfInts\",\"type\":{\"type\":\"array\",\"items\":\"int\",\"java_class\":\"java.util.ArrayList\"}}

Shouldn't it be generated in JSON as java-class (with hypen instead of
underscore) instead of java_class? Because of this reading from avro file is
not working.




--
View this message in context:
Sent from the Avro - Users mailing list archive at Nabble.com.

Loading...