Quantcast

How to use Apache Avro to encode the data in avro binary encoded form?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to use Apache Avro to encode the data in avro binary encoded form?

raihan26
I am trying to use Apache Avro to serialize one of our Attributes data and I want to use binary encoding version for that not the JSON version. In this case, I don't want to serialize to disk or some avro data files. I am planning to write Avro binary encoded-data in Cassandra column family.

Below is the code so far I have- I am not sure how to start encoding into binary version.

    Schema schema = new Parser().parse((AvroExample.class.getResourceAsStream("/user.avsc")));
    GenericRecord record = new GenericData.Record(schema);
    record.put("vbs", "99,E,0");
    record.put("eby", "0,E,,,,,,,15");
    record.put("ste", "77;2,0;1,16;3");
  // what I am supposed to do here to serialize into binary encoded version of Avro?
   // then write to Cassandra here.
Can anyone provide a simple example basis on my code how to do that? And then I can use that to write into Cassandra column family.

Thanks.




Raihan Jamal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to use Apache Avro to encode the data in avro binary encoded form?

raihan26
Can anyone help me on this? @Doug, Any idea what can be done in this case?
A simple example basis on my code will help me to understand better...





Raihan Jamal


On Wed, Sep 18, 2013 at 6:04 PM, Raihan Jamal <[hidden email]> wrote:
I am trying to use Apache Avro to serialize one of our Attributes data and I want to use binary encoding version for that not the JSON version. In this case, I don't want to serialize to disk or some avro data files. I am planning to write Avro binary encoded-data in Cassandra column family.

Below is the code so far I have- I am not sure how to start encoding into binary version.

    Schema schema = new Parser().parse((AvroExample.class.getResourceAsStream("/user.avsc")));
    GenericRecord record = new GenericData.Record(schema);
    record.put("vbs", "99,E,0");
    record.put("eby", "0,E,,,,,,,15");
    record.put("ste", "77;2,0;1,16;3");
  // what I am supposed to do here to serialize into binary encoded version of Avro?
   // then write to Cassandra here.
Can anyone provide a simple example basis on my code how to do that? And then I can use that to write into Cassandra column family.

Thanks.




Raihan Jamal

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to use Apache Avro to encode the data in avro binary encoded form?

Mika Ristimaki
In reply to this post by raihan26
Hi Raihan,

You can try somthing like this, see below.

On Sep 19, 2013, at 4:04 AM, Raihan Jamal <[hidden email]> wrote:

> I am trying to use Apache Avro to serialize one of our Attributes data and I want to use binary encoding version for that not the JSON version. In this case, I don't want to serialize to disk or some avro data files. I am planning to write Avro binary encoded-data in Cassandra column family.
>
> Below is the code so far I have- I am not sure how to start encoding into binary version.
>
>     Schema schema = new Parser().parse((AvroExample.class.getResourceAsStream("/user.avsc")));
>
>     GenericRecord record = new GenericData.Record(schema);
>     record.put("vbs", "99,E,0");
>     record.put("eby", "0,E,,,,,,,15");
>     record.put("ste", "77;2,0;1,16;3");
>
>
>     // what I am supposed to do here to serialize into binary encoded version of Avro?
>
        GenericDatumWriter<GenericRecord> writer = new GenericDatumWriter<GenericRecord>(schema);
        ByteArrayOutputStream os = new ByteArrayOutputStream();
        Encoder e = EncoderFactory.get().binaryEncoder(os, null);
        writer.write(record, e);
        e.flush();
        byte[] byteData = os.toByteArray();
        os.close();
>
>    // then write to Cassandra here.
>

I don't know anything about cassandra, but I'm pretty sure you can write binary blobs there.

-Mika

>
> Can anyone provide a simple example basis on my code how to do that? And then I can use that to write into Cassandra column family.
>
> Thanks.
>
>
>
>
> Raihan Jamal

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to use Apache Avro to encode the data in avro binary encoded form?

raihan26
Thanks Mika. That worked fine

GenericDatumWriter<GenericRecord> writer = new GenericDatumWriter<GenericRecord>(schema); 
        ByteArrayOutputStream os = new ByteArrayOutputStream(); 
        Encoder e = EncoderFactory.get().binaryEncoder(os, null); 
        writer.write(record, e); 
        e.flush(); 
        byte[] byteData = os.toByteArray(); 
        os.close();


That means, I can write byteData as the value for one of my columns. Correct? I will figure out whether we can store binary blobs or not in Cassandra. But just to make sure, I need to store byteData as it is as the value for one of my columns.. Right?

And also how can I deserialize the above data after I retrieve from Cassandra for that particular column? A simple example will make me understand this as well. Thanks.








Raihan Jamal


On Thu, Sep 19, 2013 at 1:53 AM, Mika Ristimaki [via Apache Avro] <[hidden email]> wrote:
Hi Raihan,

You can try somthing like this, see below.

On Sep 19, 2013, at 4:04 AM, Raihan Jamal <[hidden email]> wrote:

> I am trying to use Apache Avro to serialize one of our Attributes data and I want to use binary encoding version for that not the JSON version. In this case, I don't want to serialize to disk or some avro data files. I am planning to write Avro binary encoded-data in Cassandra column family.
>
> Below is the code so far I have- I am not sure how to start encoding into binary version.
>
>     Schema schema = new Parser().parse((AvroExample.class.getResourceAsStream("/user.avsc")));
>
>     GenericRecord record = new GenericData.Record(schema);
>     record.put("vbs", "99,E,0");
>     record.put("eby", "0,E,,,,,,,15");
>     record.put("ste", "77;2,0;1,16;3");
>
>
>     // what I am supposed to do here to serialize into binary encoded version of Avro?
>
        GenericDatumWriter<GenericRecord> writer = new GenericDatumWriter<GenericRecord>(schema);
        ByteArrayOutputStream os = new ByteArrayOutputStream();
        Encoder e = EncoderFactory.get().binaryEncoder(os, null);
        writer.write(record, e);
        e.flush();
        byte[] byteData = os.toByteArray();
        os.close();
>
>    // then write to Cassandra here.
>

I don't know anything about cassandra, but I'm pretty sure you can write binary blobs there.

-Mika

>
> Can anyone provide a simple example basis on my code how to do that? And then I can use that to write into Cassandra column family.
>
> Thanks.
>
>
>
>
> Raihan Jamal




To start a new topic under Avro - Users, email [hidden email]
To unsubscribe from Apache Avro, click here.
NAML

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to use Apache Avro to encode the data in avro binary encoded form?

raihan26
Hello Mika/Doug,

I am thinking to write byteData as it is in one of my columns in Cassandra Column Family so is this right? I have figured it out how can I store it in Cassandra already..
I just wanted to confirm whether I can write byteData as it is or not?





Raihan Jamal


On Thu, Sep 19, 2013 at 10:45 AM, raihan26 <[hidden email]> wrote:
Thanks Mika. That worked fine

GenericDatumWriter<GenericRecord> writer = new GenericDatumWriter<GenericRecord>(schema); 
        ByteArrayOutputStream os = new ByteArrayOutputStream(); 
        Encoder e = EncoderFactory.get().binaryEncoder(os, null); 
        writer.write(record, e); 
        e.flush(); 
        byte[] byteData = os.toByteArray(); 
        os.close();


That means, I can write byteData as the value for one of my columns. Correct? I will figure out whether we can store binary blobs or not in Cassandra. But just to make sure, I need to store byteData as it is as the value for one of my columns.. Right?

And also how can I deserialize the above data after I retrieve from Cassandra for that particular column? A simple example will make me understand this as well. Thanks.








Raihan Jamal


On Thu, Sep 19, 2013 at 1:53 AM, Mika Ristimaki [via Apache Avro] <[hidden email]> wrote:
Hi Raihan,

You can try somthing like this, see below.

On Sep 19, 2013, at 4:04 AM, Raihan Jamal <[hidden email]> wrote:

> I am trying to use Apache Avro to serialize one of our Attributes data and I want to use binary encoding version for that not the JSON version. In this case, I don't want to serialize to disk or some avro data files. I am planning to write Avro binary encoded-data in Cassandra column family.
>
> Below is the code so far I have- I am not sure how to start encoding into binary version.
>
>     Schema schema = new Parser().parse((AvroExample.class.getResourceAsStream("/user.avsc")));
>
>     GenericRecord record = new GenericData.Record(schema);
>     record.put("vbs", "99,E,0");
>     record.put("eby", "0,E,,,,,,,15");
>     record.put("ste", "77;2,0;1,16;3");
>
>
>     // what I am supposed to do here to serialize into binary encoded version of Avro?
>
        GenericDatumWriter<GenericRecord> writer = new GenericDatumWriter<GenericRecord>(schema);
        ByteArrayOutputStream os = new ByteArrayOutputStream();
        Encoder e = EncoderFactory.get().binaryEncoder(os, null);
        writer.write(record, e);
        e.flush();
        byte[] byteData = os.toByteArray();
        os.close();
>
>    // then write to Cassandra here.
>

I don't know anything about cassandra, but I'm pretty sure you can write binary blobs there.

-Mika

>
> Can anyone provide a simple example basis on my code how to do that? And then I can use that to write into Cassandra column family.
>
> Thanks.
>
>
>
>
> Raihan Jamal



To start a new topic under Avro - Users, email [hidden email]
To unsubscribe from Apache Avro, click here.
NAML



View this message in context: Re: How to use Apache Avro to encode the data in avro binary encoded form?
Sent from the Avro - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to use Apache Avro to encode the data in avro binary encoded form?

Doug Cutting
As long as you keep track of the schema somewhere.  You cannot read
the data without the schema used to write it.  So you could store the
schema once for the entire column and directly store the binary
encoding in the cells, but you'll then not be able to alter the schema
without re-writing all of the values.  If you want to be able to alter
the schema then you might prefix each binary-encoded cell value with
an integer schema id, then have a separate table that maps schema ids
to schemas.

Doug

On Fri, Sep 20, 2013 at 2:44 PM, Raihan Jamal <[hidden email]> wrote:

> Hello Mika/Doug,
>
> I am thinking to write byteData as it is in one of my columns in Cassandra
> Column Family so is this right? I have figured it out how can I store it in
> Cassandra already..
> I just wanted to confirm whether I can write byteData as it is or not?
>
>
>
>
>
> Raihan Jamal
>
>
> On Thu, Sep 19, 2013 at 10:45 AM, raihan26 <[hidden email]> wrote:
>>
>> Thanks Mika. That worked fine
>>
>> GenericDatumWriter<GenericRecord> writer = new
>> GenericDatumWriter<GenericRecord>(schema);
>>         ByteArrayOutputStream os = new ByteArrayOutputStream();
>>         Encoder e = EncoderFactory.get().binaryEncoder(os, null);
>>         writer.write(record, e);
>>         e.flush();
>>         byte[] byteData = os.toByteArray();
>>         os.close();
>>
>>
>> That means, I can write byteData as the value for one of my columns.
>> Correct? I will figure out whether we can store binary blobs or not in
>> Cassandra. But just to make sure, I need to store byteData as it is as the
>> value for one of my columns.. Right?
>>
>> And also how can I deserialize the above data after I retrieve from
>> Cassandra for that particular column? A simple example will make me
>> understand this as well. Thanks.
>>
>>
>>
>>
>>
>>
>>
>>
>> Raihan Jamal
>>
>>
>> On Thu, Sep 19, 2013 at 1:53 AM, Mika Ristimaki [via Apache Avro] <[hidden
>> email]> wrote:
>>>
>>> Hi Raihan,
>>>
>>> You can try somthing like this, see below.
>>>
>>> On Sep 19, 2013, at 4:04 AM, Raihan Jamal <[hidden email]> wrote:
>>>
>>> > I am trying to use Apache Avro to serialize one of our Attributes data
>>> > and I want to use binary encoding version for that not the JSON version. In
>>> > this case, I don't want to serialize to disk or some avro data files. I am
>>> > planning to write Avro binary encoded-data in Cassandra column family.
>>> >
>>> > Below is the code so far I have- I am not sure how to start encoding
>>> > into binary version.
>>> >
>>> >     Schema schema = new
>>> > Parser().parse((AvroExample.class.getResourceAsStream("/user.avsc")));
>>> >
>>> >     GenericRecord record = new GenericData.Record(schema);
>>> >     record.put("vbs", "99,E,0");
>>> >     record.put("eby", "0,E,,,,,,,15");
>>> >     record.put("ste", "77;2,0;1,16;3");
>>> >
>>> >
>>> >     // what I am supposed to do here to serialize into binary encoded
>>> > version of Avro?
>>> >
>>>         GenericDatumWriter<GenericRecord> writer = new
>>> GenericDatumWriter<GenericRecord>(schema);
>>>         ByteArrayOutputStream os = new ByteArrayOutputStream();
>>>         Encoder e = EncoderFactory.get().binaryEncoder(os, null);
>>>         writer.write(record, e);
>>>         e.flush();
>>>         byte[] byteData = os.toByteArray();
>>>         os.close();
>>> >
>>> >    // then write to Cassandra here.
>>> >
>>>
>>> I don't know anything about cassandra, but I'm pretty sure you can write
>>> binary blobs there.
>>>
>>> -Mika
>>>
>>> >
>>> > Can anyone provide a simple example basis on my code how to do that?
>>> > And then I can use that to write into Cassandra column family.
>>> >
>>> > Thanks.
>>> >
>>> >
>>> >
>>> >
>>> > Raihan Jamal
>>>
>>>
>>>
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>>
>>> http://apache-avro.679487.n3.nabble.com/How-to-use-Apache-Avro-to-encode-the-data-in-avro-binary-encoded-form-tp4028217p4028221.html
>>> To start a new topic under Avro - Users, email [hidden email]
>>> To unsubscribe from Apache Avro, click here.
>>> NAML
>>
>>
>>
>> ________________________________
>> View this message in context: Re: How to use Apache Avro to encode the
>> data in avro binary encoded form?
>> Sent from the Avro - Users mailing list archive at Nabble.com.
>
>
Loading...