Quantcast

Deadlock using Avro 1.5.0

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Deadlock using Avro 1.5.0

Shalin Shekhar Mangar
Hello,

I'm seeing a deadlock condition using avro 1.5.0. See the relevant thread dump:

2011-05-09 20:24:50
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.0-b11 mixed mode):

"pool-1-thread-2" prio=10 tid=0x0000000040c66800 nid=0x1e38 in Object.wait() [0x00007f4746257000]
   java.lang.Thread.State: RUNNABLE
at org.apache.avro.io.parsing.Symbol.<clinit>(Symbol.java:542)
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:79)
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50)
at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:82)
at org.apache.avro.io.ResolvingDecoder.<init>(ResolvingDecoder.java:46)
at org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307)
at org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:112)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:127)
at com.aol.service.provider.ServiceContainer$JobExecutor.run(ServiceContainer.java:152)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1" prio=10 tid=0x0000000040c65800 nid=0x1e32 in Object.wait() [0x00007f4746359000]
   java.lang.Thread.State: RUNNABLE
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:237)
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118)
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50)
at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:82)
at org.apache.avro.io.ResolvingDecoder.<init>(ResolvingDecoder.java:46)
at org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307)
at org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:112)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:127)
at com.aol.service.provider.ServiceContainer$JobExecutor.run(ServiceContainer.java:152)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

I was able to workaround this condition by adding the following code at server startup:
Class.forName(Symbol.class.getName());

Is this a known issue?

--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deadlock using Avro 1.5.0

Philip Zeyliger
Shalin,

For what it's worth, I've run into the same issue recently.  I'm trying the same workaround as you are.

This is what it looks like in 'pstack':

    .  Thread 81 (Thread 0x7f7849757700 (LWP 13920)):
    .  #0  0x00007f7874ef43dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    .  #1  0x00007f78744491fe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
    .  #2  0x00007f787443a27a in ObjectMonitor::wait(long, bool, Thread*) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
    .  #3  0x00007f787451f3e0 in ObjectSynchronizer::waitUninterruptibly(Handle, long, Thread*) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libj
    .  #5  0x00007f78741e57ca in instanceKlass::initialize(Thread*) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
    .  #6  0x00007f787420ccac in InterpreterRuntime::_new(JavaThread*, constantPoolOopDesc*, int) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/li
    .  #7  0x00007f787025d5cf in ?? ()
    .  Thread 80 (Thread 0x7f7849656700 (LWP 13921)):
    .  #0  0x00007f7874ef43dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    .  #1  0x00007f78744491fe in os::PlatformEvent::park() () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
    .  #2  0x00007f787443a27a in ObjectMonitor::wait(long, bool, Thread*) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
    .  #3  0x00007f787451f3e0 in ObjectSynchronizer::waitUninterruptibly(Handle, long, Thread*) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libj
    .  #4  0x00007f78741e61ab in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm
    .  #5  0x00007f78741e57ca in instanceKlass::initialize(Thread*) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
    .  #6  0x00007f787420ccac in InterpreterRuntime::_new(JavaThread*, constantPoolOopDesc*, int) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/li
    .  #7  0x00007f787025d5cf in ?? ()



-- Philip

On Mon, May 9, 2011 at 8:11 AM, Shalin Shekhar Mangar <[hidden email]> wrote:
Hello,

I'm seeing a deadlock condition using avro 1.5.0. See the relevant thread dump:

2011-05-09 20:24:50
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.0-b11 mixed mode):

"pool-1-thread-2" prio=10 tid=0x0000000040c66800 nid=0x1e38 in Object.wait() [0x00007f4746257000]
   java.lang.Thread.State: RUNNABLE
at org.apache.avro.io.parsing.Symbol.<clinit>(Symbol.java:542)
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:79)
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50)
at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:82)
at org.apache.avro.io.ResolvingDecoder.<init>(ResolvingDecoder.java:46)
at org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307)
at org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:112)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:127)
at com.aol.service.provider.ServiceContainer$JobExecutor.run(ServiceContainer.java:152)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1" prio=10 tid=0x0000000040c65800 nid=0x1e32 in Object.wait() [0x00007f4746359000]
   java.lang.Thread.State: RUNNABLE
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:237)
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118)
at org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50)
at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:82)
at org.apache.avro.io.ResolvingDecoder.<init>(ResolvingDecoder.java:46)
at org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307)
at org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:112)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:127)
at com.aol.service.provider.ServiceContainer$JobExecutor.run(ServiceContainer.java:152)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

I was able to workaround this condition by adding the following code at server startup:
Class.forName(Symbol.class.getName());

Is this a known issue?

--
Regards,
Shalin Shekhar Mangar.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deadlock using Avro 1.5.0

slandelle
Hi,

I have the same problem with 1.7.3, and sadly the workaround doesn't work for me.

Cheers,

Stéphane
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deadlock using Avro 1.5.0

slandelle
I have a sample to reproduce the problem. Should I fill an issue in JIRA?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deadlock using Avro 1.5.0

slandelle
I think the problem comes from a race condition in the Symbol class initialization.

I extracted the constants (NULL, BOOLEAN, etc) in a dedicated class and I can't reproduce the problem with my test case.

Stéphane
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deadlock using Avro 1.5.0

Scott Carey-2
Please file a JIRA ticket.  Thanks!

On 12/11/12 2:56 AM, "slandelle" <[hidden email]> wrote:

>I think the problem comes from a race condition in the Symbol class
>initialization.
>
>I extracted the constants (NULL, BOOLEAN, etc) in a dedicated class and I
>can't reproduce the problem with my test case.
>
>Stéphane
>
>
>
>--
>View this message in context:
>http://apache-avro.679487.n3.nabble.com/Deadlock-using-Avro-1-5-0-tp291908
>7p4025778.html
>Sent from the Avro - Users mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deadlock using Avro 1.5.0

slandelle
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deadlock using Avro 1.5.0

panpan2523
This post has NOT been accepted by the mailing list yet.
In reply to this post by Shalin Shekhar Mangar
Hollywood stars are crazy about red sole Christian Louboutin shoes the amount of which is cool expensive. However, it is said that the females who abreast accept added than 500 pairs of Christian Louboutin shoes are already added than 3000. It can be said that the account of admirers of air jordan 10 retro shoes includes every body. Which not alone includes Victoria Beckham, Nicole Richie, Kim Kardashian ect. abounding of acclaimed stars, it aswell accomplish the adolescent stars who are actual accepted not alone for their adolescence such as Leighton Meester, Blake Lively, Miley Cyrus, all of whom are aswell crazy about Christian Louboutin red sole top heels.
Loading...