Deep polymorphic type support

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Deep polymorphic type support

Greg Zoller
Hello,

I'm trying to learn if Avro currently supports deep polymorphic types like outlined here:  https://stackoverflow.com/questions/46120578/how-might-i-represent-a-scala-trait-in-avro

If not, could the standard be extended?

The use case is this... I want to send an AlarmMessage, serialized as a Message[PayloadKind].  The receiving side would have enough type data to reconstruct the AlarmMessage (Alarm is of type Event, which is of type PayloadKind).  In other words I'd "listen" for Message[PayloadKind] and be handed an AlarmMessage.

Of course we'd need to have schema entries for any kind of Event (or PayloadKind child for that matter), but there'd also need to be a way to represent these intermediate marker traits.

Thoughts?
Greg

(A trivial json example of what I mean can be found here: https://github.com/gzoller/ScalaJack/blob/master/doc/classesAndTraits.md )
Reply | Threaded
Open this post in threaded view
|

Re: Deep polymorphic type support

Dan Schmitt
Sounds like a job for a Union for payload.

And if the polymorphic payload is avro serialized you don't
have to know the schema ahead of time, you just need a
way to merge the new payload schema into the union as
you get it (so you can push the schema responsibility to
the client code giving you the event.)

I've been futzing with something similar on the C++ avro side
and the provided API needs some tweaks to make it go (I'm
looking at moving the fileReader/Wrtier up to work with
streams, then you can do memory array with the array stream
adapter; and then you need a way to get to addField for
the UnionSchema (which on the C++ side only has default
constructor, no way to access a Union somewhere in a schema
and have addField add the schema from the new data
without knowing a lot about the internals, so you either have
to save it when first creating the schema, and can't do it from
an existing schema, or need more code.)

     Dan S.







On Fri, Sep 8, 2017 at 2:35 PM, Greg Zoller <[hidden email]> wrote:

> Hello,
>
> I'm trying to learn if Avro currently supports deep polymorphic types like
> outlined here:
> https://stackoverflow.com/questions/46120578/how-might-i-represent-a-scala-trait-in-avro
>
> If not, could the standard be extended?
>
> The use case is this... I want to send an AlarmMessage, serialized as a
> Message[PayloadKind].  The receiving side would have enough type data to
> reconstruct the AlarmMessage (Alarm is of type Event, which is of type
> PayloadKind).  In other words I'd "listen" for Message[PayloadKind] and be
> handed an AlarmMessage.
>
> Of course we'd need to have schema entries for any kind of Event (or
> PayloadKind child for that matter), but there'd also need to be a way to
> represent these intermediate marker traits.
>
> Thoughts?
> Greg
>
> (A trivial json example of what I mean can be found here:
> https://github.com/gzoller/ScalaJack/blob/master/doc/classesAndTraits.md )
Reply | Threaded
Open this post in threaded view
|

Re: Deep polymorphic type support

Greg Zoller
Thanks, Dan.  I'm not 100% clear how Union would work for Scala marker traits.  I could see how I'd use them for Message[Alarm], where Alarm is a first-class object.  I've got 2 layers of marker traits in the middle tho:  Message[PayloadKind], where PayloadKind is a trait having no fields, Event is a trait having no fields that extends PayloadKind, and finally Alarm (concrete class) extends Event.

I guess for pure Avro you could flatten things to just Message[Alarm], but this weakens the domain model.

Can you think of a way to use Union types to represent marker traits?  My understanding is Unions can't immediately contain other Unions (i.e. like Event -> PayloadKind).

Greg

On Fri, Sep 8, 2017 at 1:56 PM, Dan Schmitt <[hidden email]> wrote:
Sounds like a job for a Union for payload.

And if the polymorphic payload is avro serialized you don't
have to know the schema ahead of time, you just need a
way to merge the new payload schema into the union as
you get it (so you can push the schema responsibility to
the client code giving you the event.)

I've been futzing with something similar on the C++ avro side
and the provided API needs some tweaks to make it go (I'm
looking at moving the fileReader/Wrtier up to work with
streams, then you can do memory array with the array stream
adapter; and then you need a way to get to addField for
the UnionSchema (which on the C++ side only has default
constructor, no way to access a Union somewhere in a schema
and have addField add the schema from the new data
without knowing a lot about the internals, so you either have
to save it when first creating the schema, and can't do it from
an existing schema, or need more code.)

     Dan S.







On Fri, Sep 8, 2017 at 2:35 PM, Greg Zoller <[hidden email]> wrote:
> Hello,
>
> I'm trying to learn if Avro currently supports deep polymorphic types like
> outlined here:
> https://stackoverflow.com/questions/46120578/how-might-i-represent-a-scala-trait-in-avro
>
> If not, could the standard be extended?
>
> The use case is this... I want to send an AlarmMessage, serialized as a
> Message[PayloadKind].  The receiving side would have enough type data to
> reconstruct the AlarmMessage (Alarm is of type Event, which is of type
> PayloadKind).  In other words I'd "listen" for Message[PayloadKind] and be
> handed an AlarmMessage.
>
> Of course we'd need to have schema entries for any kind of Event (or
> PayloadKind child for that matter), but there'd also need to be a way to
> represent these intermediate marker traits.
>
> Thoughts?
> Greg
>
> (A trivial json example of what I mean can be found here:
> https://github.com/gzoller/ScalaJack/blob/master/doc/classesAndTraits.md )

Reply | Threaded
Open this post in threaded view
|

Re: Deep polymorphic type support

Dan Schmitt
I was thinking:

record of:
    string: id
    Union: Payload

Assuming you have some render comparable function that turns a trait
Payload into an avro chunk
(which will have a schema) it's just a matter of adding the schema to
the Payload union.  You may
have to write that part yourself (I'm playing from the C++ side, not
the scala side, and while the
binary file format is flexible, you do end up writing more code to map
it to various language features..;
I don't know what you'd do to get it to de-serialize to a trait
(probably some extra tag in the payload
schema and code somewhere in the scala API to serialize/de-serialize it.)


On Fri, Sep 8, 2017 at 4:09 PM, Greg Zoller <[hidden email]> wrote:

> Thanks, Dan.  I'm not 100% clear how Union would work for Scala marker
> traits.  I could see how I'd use them for Message[Alarm], where Alarm is a
> first-class object.  I've got 2 layers of marker traits in the middle tho:
> Message[PayloadKind], where PayloadKind is a trait having no fields, Event
> is a trait having no fields that extends PayloadKind, and finally Alarm
> (concrete class) extends Event.
>
> I guess for pure Avro you could flatten things to just Message[Alarm], but
> this weakens the domain model.
>
> Can you think of a way to use Union types to represent marker traits?  My
> understanding is Unions can't immediately contain other Unions (i.e. like
> Event -> PayloadKind).
>
> Greg
>
> On Fri, Sep 8, 2017 at 1:56 PM, Dan Schmitt <[hidden email]> wrote:
>>
>> Sounds like a job for a Union for payload.
>>
>> And if the polymorphic payload is avro serialized you don't
>> have to know the schema ahead of time, you just need a
>> way to merge the new payload schema into the union as
>> you get it (so you can push the schema responsibility to
>> the client code giving you the event.)
>>
>> I've been futzing with something similar on the C++ avro side
>> and the provided API needs some tweaks to make it go (I'm
>> looking at moving the fileReader/Wrtier up to work with
>> streams, then you can do memory array with the array stream
>> adapter; and then you need a way to get to addField for
>> the UnionSchema (which on the C++ side only has default
>> constructor, no way to access a Union somewhere in a schema
>> and have addField add the schema from the new data
>> without knowing a lot about the internals, so you either have
>> to save it when first creating the schema, and can't do it from
>> an existing schema, or need more code.)
>>
>>      Dan S.
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Sep 8, 2017 at 2:35 PM, Greg Zoller <[hidden email]> wrote:
>> > Hello,
>> >
>> > I'm trying to learn if Avro currently supports deep polymorphic types
>> > like
>> > outlined here:
>> >
>> > https://stackoverflow.com/questions/46120578/how-might-i-represent-a-scala-trait-in-avro
>> >
>> > If not, could the standard be extended?
>> >
>> > The use case is this... I want to send an AlarmMessage, serialized as a
>> > Message[PayloadKind].  The receiving side would have enough type data to
>> > reconstruct the AlarmMessage (Alarm is of type Event, which is of type
>> > PayloadKind).  In other words I'd "listen" for Message[PayloadKind] and
>> > be
>> > handed an AlarmMessage.
>> >
>> > Of course we'd need to have schema entries for any kind of Event (or
>> > PayloadKind child for that matter), but there'd also need to be a way to
>> > represent these intermediate marker traits.
>> >
>> > Thoughts?
>> > Greg
>> >
>> > (A trivial json example of what I mean can be found here:
>> > https://github.com/gzoller/ScalaJack/blob/master/doc/classesAndTraits.md
>> > )
>
>