Quantcast

avro 1.7.0 with CDH4

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

avro 1.7.0 with CDH4

Alan Miller
Thanks for the 1.7.0 release.
I was waiting for AVRO-593 so I can use the new mapreduce API with Avro.

Previously, I was using avro-1.6.3 and odiago-avro-1.0.7.jar
and everything was working. My driver class writes Avro files on HDFS
and triggers an AvroJob to process them.

I recompiled my code with avro-1.7.0 (and removed odiago-avro) but now I get
the error below. I know this list is about Avro and not odiago-avro integration but
is AvroKeyInputFormat from avro-1.7.0-tools compatible with odiago's?


2012-07-05 02:21:10,068 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
        at org.apache.avro.mapreduce.AvroKeyInputFormat.createRecordReader(AvroKeyInputFormat.java:46)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:617)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: avro 1.7.0 with CDH4

Alan Miller

I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job so that:

 1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar file in ./lib/
    (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
 2.  I have also set: mapreduce.task.classpath.first=true (in mapred-site.xml)
      to ensure the 1.7.0 jars from the lib dir in my.jar are used first (before avro-1.5.4).

But I still get the same error.

Any ideas?

Alan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: avro 1.7.0 with CDH4

Jacob Metcalf
In reply to this post by Alan Miller
Avro 17 is compiled against Hadoop 1 i.e. CDH3.

I have it running aginst CDH4 but I had to patch and recompile Avro, then replace the Avro 153 in the Hadoop lib directory with 17. I have attached my patch against JIRA AVRO-1103 for consideration.

-----Original Message-----

From: Alan Miller
Sent: 5 Jul 2012 11:03:06 GMT
To: [hidden email]
Subject: Re: avro 1.7.0 with CDH4


I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job so
that:

 1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar file
in ./lib/
    (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
 2.  I have also set: mapreduce.task.classpath.first=true (in
mapred-site.xml)
      to ensure the 1.7.0 jars from the lib dir in my.jar are used first
(before avro-1.5.4).

But I still get the same error.

Any ideas?

Alan

--
View this message in context: http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149p4025150.html
Sent from the Avro - Users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: avro 1.7.0 with CDH4

Garrett Wu
What is the correct solution for making avro-1.7 work with both Hadoop 1 (CDH3) and Hadoop 2 (CDH4)? Should we add build profiles for each so we can release two separate avro-mapred jars?

On Thu, Jul 5, 2012 at 5:12 AM, Jacob Metcalf <[hidden email]> wrote:
Avro 17 is compiled against Hadoop 1 i.e. CDH3.

I have it running aginst CDH4 but I had to patch and recompile Avro, then replace the Avro 153 in the Hadoop lib directory with 17. I have attached my patch against JIRA AVRO-1103 for consideration.

-----Original Message-----

From: Alan Miller
Sent: 5 Jul 2012 11:03:06 GMT
To: [hidden email]
Subject: Re: avro 1.7.0 with CDH4


I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job so
that:

 1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar file
in ./lib/
    (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
 2.  I have also set: mapreduce.task.classpath.first=true (in
mapred-site.xml)
      to ensure the 1.7.0 jars from the lib dir in my.jar are used first
(before avro-1.5.4).

But I still get the same error.

Any ideas?

Alan

--
View this message in context: http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149p4025150.html
Sent from the Avro - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: avro 1.7.0 with CDH4

Jacob Metcalf
In reply to this post by Alan Miller
That would be ideal. With MRUnit for example both builds are in the repo and you use <classifier>hadoop2</classifier> to get the right one.



-----Original Message-----

From: Garrett Wu
Sent: 16 Jul 2012 16:56:54 GMT
To: [hidden email]
Cc: [hidden email]
Subject: Re: avro 1.7.0 with CDH4

What is the correct solution for making avro-1.7 work with both Hadoop 1 (CDH3) and Hadoop 2 (CDH4)? Should we add build profiles for each so we can release two separate avro-mapred jars?

On Thu, Jul 5, 2012 at 5:12 AM, Jacob Metcalf <[hidden email]> wrote:
Avro 17 is compiled against Hadoop 1 i.e. CDH3.

I have it running aginst CDH4 but I had to patch and recompile Avro, then replace the Avro 153 in the Hadoop lib directory with 17. I have attached my patch against JIRA AVRO-1103 for consideration.

-----Original Message-----

From: Alan Miller
Sent: 5 Jul 2012 11:03:06 GMT
To: [hidden email]
Subject: Re: avro 1.7.0 with CDH4


I noticed that CDH4 has avro-1.5.4.jar in hadoop/lib so I modified my job so
that:

 1.  avro-1.7.0.jar and avro-tools-1.7.0.jar is contained within my jar file
in ./lib/
    (note: I submit my job via hadoop jar myjar.jar myDriver <myopts ....>)
 2.  I have also set: mapreduce.task.classpath.first=true (in
mapred-site.xml)
      to ensure the 1.7.0 jars from the lib dir in my.jar are used first
(before avro-1.5.4).

But I still get the same error.

Any ideas?

Alan

--
View this message in context: http://apache-avro.679487.n3.nabble.com/avro-1-7-0-with-CDH4-tp4025149p4025150.html
Sent from the Avro - Users mailing list archive at Nabble.com.

Loading...