I’m implementing an application which reads in structured data with an Avro schema applies some dynamically configurable transformations and outputs the data with Avro again.
The problem I have is that for some transformations I need to modify the Avro schema. One transform could be for example that I read a value from a field apply some function to the value and write it back to a new field. In this scenario I need to add the
new field to the output schema. I haven’t found a really good way to do this with Avro. What I’m doing right now is reading all the fields from the old schema, create a new schema and copy all the fields over to this new schema: