correct way of saving data to file in C++, 1.8.2 api

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

correct way of saving data to file in C++, 1.8.2 api

scarlehoff
Hi,
I'm trying to modify the first example in the c++ tutorial:
http://avro.apache.org/docs/1.8.1/api/cpp/html/index.html (note: docs for c++,1.8.2 are not working for me)
I'm trying to write to a file. For this I modify the example from

    std::auto_ptr<avro::OutputStream> out = avro::memoryOutputStream();
    avro::EncoderPtr e = avro::binaryEncoder();
    e->init(*out);
    c::cpx c1;
    c1.re = 1.0;
    c1.im = 2.13;
    avro::encode(*e, c1);

to
     std::auto_ptr<avro::OutputStream> outFile = avro::fileOutputStream("filetest.db");
     avro::EncoderPtr e = avro::binaryEncoder();
     e->init(*outFile);
     c::cpx c1;
     c1.re = 1.0;
     c1.im = 2.13;
     avro::encode(*e, c1);
     outFile->flush();

I get a file "filetest.db" which I can then read using C++.
However, I cannot read the resulting file using python because the following code fails:
    from avro.datafile import DataFileReader
    from avro.io import DatumReader
    reader = DataFileReader(open("filetest.db", "rb"), DatumReader())
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8f in position 2: invalid start byte

My guess is my C++ code is wrong, but I'm can't find any (recent) documentation on what is the correct way of saving things to a file when using C++.
Thanks,
s
Reply | Threaded
Open this post in threaded view
|

Re: correct way of saving data to file in C++, 1.8.2 api

iampete
Did you ever work this out?  I am having the same problem.
Thanks




--
Sent from: http://apache-avro.679487.n3.nabble.com/Avro-Users-f679479.html
Reply | Threaded
Open this post in threaded view
|

Re: correct way of saving data to file in C++, 1.8.2 api

Tim Upthegrove
I have not tested this, but it is possible that you all want the avro::DataFileWriter<T> templated class.  I believe that class will ensure the appropriate header information is in the Avro data file, and it has a write method that takes in data of type T and serializes it, appending that to the end of the file.  I am not sure what the avro::fileOutputStream does, but perhaps it does not include the header information that is expected in the Avro data file format.  The python class referenced in the original email definitely does expect the header to be in place.  As I mentioned before, I have not tested this, but I figured I'd throw this out as a thread to pull on in case anyone was still stick.

Good luck,
Tim Upthegrove

On Tue, Oct 31, 2017 at 5:33 PM, iampete <[hidden email]> wrote:
Did you ever work this out?  I am having the same problem.
Thanks




--
Sent from: http://apache-avro.679487.n3.nabble.com/Avro-Users-f679479.html