avro 1.7.0 with CDH4

[Alan Miller]

Thanks for the 1.7.0 release.
I was waiting for AVRO-593 so I can use the new mapreduce API with Avro.

Previously, I was using avro-1.6.3 and odiago-avro-1.0.7.jar
and everything was working. My driver class writes Avro files on HDFS
and triggers an AvroJob to process them.

I recompiled my code with avro-1.7.0 (and removed odiago-avro) but now I get
the error below. I know this list is about Avro and not odiago-avro integration but
is AvroKeyInputFormat from avro-1.7.0-tools compatible with odiago's?


2012-07-05 02:21:10,068 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
        at org.apache.avro.mapreduce.AvroKeyInputFormat.createRecordReader(AvroKeyInputFormat.java:46)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:617)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)

[Alan Miller]

I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job so that:

 1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar file in ./lib/
    (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
 2.  I have also set: mapreduce.task.classpath.first=true (in mapred-site.xml)
      to ensure the 1.7.0 jars from the lib dir in my.jar are used first (before avro-1.5.4).

But I still get the same error.

Any ideas?

Alan

[Jacob Metcalf]

Avro 17 is compiled against Hadoop 1 i.e. CDH3.

I have it running aginst CDH4 but I had to patch and recompile Avro, then replace the Avro 153 in the Hadoop lib directory with 17. I have attached my patch against JIRA AVRO-1103 for consideration.

-----Original Message-----

From: Alan Miller
Sent: 5 Jul 2012 11:03:06 GMT
To: [hidden email]
Subject: Re: avro 1.7.0 with CDH4


I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job so
that:

 1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar file
in ./lib/
    (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
 2.  I have also set: mapreduce.task.classpath.first=true (in
mapred-site.xml)
      to ensure the 1.7.0 jars from the lib dir in my.jar are used first
(before avro-1.5.4).

But I still get the same error.

Any ideas?

Alan

--
View this message in context: http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149p4025150.html
Sent from the Avro - Users mailing list archive at Nabble.com.

[Garrett Wu]

What is the correct solution for making avro-1.7 work with both Hadoop 1 (CDH3) and Hadoop 2 (CDH4)? Should we add build profiles for each so we can release two separate avro-mapred jars?

On Thu, Jul 5, 2012 at 5:12 AM, Jacob Metcalf <[hidden email]> wrote:

Avro 17 is compiled against Hadoop 1 i.e. CDH3.

I have it running aginst CDH4 but I had to patch and recompile Avro, then replace the Avro 153 in the Hadoop lib directory with 17. I have attached my patch against JIRA AVRO-1103 for consideration.

-----Original Message-----

From: Alan Miller
Sent: 5 Jul 2012 11:03:06 GMT
To: [hidden email]
Subject: Re: avro 1.7.0 with CDH4

I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job so
that:

 1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar file
in ./lib/
    (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
 2.  I have also set: mapreduce.task.classpath.first=true (in
mapred-site.xml)
      to ensure the 1.7.0 jars from the lib dir in my.jar are used first
(before avro-1.5.4).

But I still get the same error.

Any ideas?

Alan

--
View this message in context: http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149p4025150.html
Sent from the Avro - Users mailing list archive at Nabble.com.

[Jacob Metcalf]

That would be ideal. With MRUnit for example both builds are in the repo and you use <classifier>hadoop2</classifier> to get the right one.



-----Original Message-----

From: Garrett Wu
Sent: 16 Jul 2012 16:56:54 GMT
To: [hidden email]
Cc: [hidden email]
Subject: Re: avro 1.7.0 with CDH4

What is the correct solution for making avro-1.7 work with both Hadoop 1 (CDH3) and Hadoop 2 (CDH4)? Should we add build profiles for each so we can release two separate avro-mapred jars?

On Thu, Jul 5, 2012 at 5:12 AM, Jacob Metcalf <[hidden email]> wrote:

Avro 17 is compiled against Hadoop 1 i.e. CDH3.

I have it running aginst CDH4 but I had to patch and recompile Avro, then replace the Avro 153 in the Hadoop lib directory with 17. I have attached my patch against JIRA AVRO-1103 for consideration.

-----Original Message-----

From: Alan Miller
Sent: 5 Jul 2012 11:03:06 GMT
To: [hidden email]
Subject: Re: avro 1.7.0 with CDH4

I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job so
that:

 1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar file
in ./lib/
    (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
 2.  I have also set: mapreduce.task.classpath.first=true (in
mapred-site.xml)
      to ensure the 1.7.0 jars from the lib dir in my.jar are used first
(before avro-1.5.4).

But I still get the same error.

Any ideas?

Alan

--
View this message in context: http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149p4025150.html
Sent from the Avro - Users mailing list archive at Nabble.com.