Quantcast

Does Avro Serde support schema evolution?

classic Classic list List threaded Threaded
3 messages Options
kva
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Does Avro Serde support schema evolution?

kva
Hi,

Assume v0 schema is:

{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}

And v1 schema is (just added an optional field named age):

{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]},
     {"name": "age",  "type": ["int", "null"]}
 ]
}

I created a table using v1 user schema on an Avro file written with v0 of the user schema. When running a select * on the table I get an error

Failed with exception java.io.IOException:org.apache.avro.AvroTypeException: Found {...}

Isn't this a scenario that Avro and by extension the AvroSerde should support?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does Avro Serde support schema evolution?

kulkarni.swarnim@gmail.com
You would need to provide a default value for the newly added field in the v1 schema.


On Thu, Oct 17, 2013 at 3:37 PM, kva <[hidden email]> wrote:
Hi,

Assume v0 schema is:

{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}

And v1 schema is (just added an optional field named age):

{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]},
     {"name": "age",  "type": ["int", "null"]}
 ]
}

I created a table using v1 user schema on an Avro file written with v0 of
the user schema. When running a select * on the table I get an error

Failed with exception java.io.IOException:org.apache.avro.AvroTypeException:
Found {...}

Isn't this a scenario that Avro and by extension the AvroSerde should
support?




--
View this message in context: http://apache-avro.679487.n3.nabble.com/Does-Avro-Serde-support-schema-evolution-tp4028398.html
Sent from the Avro - Users mailing list archive at Nabble.com.



--
Swarnim
kva
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does Avro Serde support schema evolution?

kva
Right, thanks. I thought the optionality was sufficient.
Loading...