[jira] Created: (AVRO-498) Clarify spec for object container files

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (AVRO-498) Clarify spec for object container files

ASF subversion and git services (Jira)
Clarify spec for object container files
---------------------------------------

                 Key: AVRO-498
                 URL: https://issues.apache.org/jira/browse/AVRO-498
             Project: Avro
          Issue Type: Improvement
          Components: spec
            Reporter: Scott Banachowski


I have some proposed changes for the spec in the Object Container Files section.

The fixes are:
1) call out explicitly that when it says long, it means avro-encoded long.
2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).

Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (AVRO-498) Clarify spec for object container files

ASF subversion and git services (Jira)

     [ https://issues.apache.org/jira/browse/AVRO-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Banachowski updated AVRO-498:
-----------------------------------

    Status: Patch Available  (was: Open)

> Clarify spec for object container files
> ---------------------------------------
>
>                 Key: AVRO-498
>                 URL: https://issues.apache.org/jira/browse/AVRO-498
>             Project: Avro
>          Issue Type: Improvement
>          Components: spec
>            Reporter: Scott Banachowski
>         Attachments: AVRO-498.patch
>
>
> I have some proposed changes for the spec in the Object Container Files section.
> The fixes are:
> 1) call out explicitly that when it says long, it means avro-encoded long.
> 2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
> 3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
> Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (AVRO-498) Clarify spec for object container files

ASF subversion and git services (Jira)
In reply to this post by ASF subversion and git services (Jira)

     [ https://issues.apache.org/jira/browse/AVRO-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Banachowski updated AVRO-498:
-----------------------------------

    Attachment: AVRO-498.patch

patch to spec.xml


> Clarify spec for object container files
> ---------------------------------------
>
>                 Key: AVRO-498
>                 URL: https://issues.apache.org/jira/browse/AVRO-498
>             Project: Avro
>          Issue Type: Improvement
>          Components: spec
>            Reporter: Scott Banachowski
>         Attachments: AVRO-498.patch
>
>
> I have some proposed changes for the spec in the Object Container Files section.
> The fixes are:
> 1) call out explicitly that when it says long, it means avro-encoded long.
> 2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
> 3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
> Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (AVRO-498) Clarify spec for object container files

ASF subversion and git services (Jira)
In reply to this post by ASF subversion and git services (Jira)

    [ https://issues.apache.org/jira/browse/AVRO-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852907#action_12852907 ]

Doug Cutting commented on AVRO-498:
-----------------------------------

Whenever the term 'long' is used in the spec it's intended to refer to an Avro-encode long.  Anything else would be called something like "eight-byte, big-endian value" or somesuch.  Perhaps that's what need's clarifying?

As for the metadata, I don't think this is incompatible, since I think every impl reads it as a map, and that this is a spec bug.  Perhaps we should instead change the spec to say that metadata is written as if with the schema {"type":"map", "values": "bytes"}?

> Clarify spec for object container files
> ---------------------------------------
>
>                 Key: AVRO-498
>                 URL: https://issues.apache.org/jira/browse/AVRO-498
>             Project: Avro
>          Issue Type: Improvement
>          Components: spec
>            Reporter: Scott Banachowski
>         Attachments: AVRO-498.patch
>
>
> I have some proposed changes for the spec in the Object Container Files section.
> The fixes are:
> 1) call out explicitly that when it says long, it means avro-encoded long.
> 2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
> 3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
> Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (AVRO-498) Clarify spec for object container files

ASF subversion and git services (Jira)
In reply to this post by ASF subversion and git services (Jira)

    [ https://issues.apache.org/jira/browse/AVRO-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852926#action_12852926 ]

Scott Carey commented on AVRO-498:
----------------------------------

bq. Perhaps we should instead change the spec to say that metadata is written as if with the schema {"type":"map", "values": "bytes"}?

+1
IMO that is the clearest way to describe it.

> Clarify spec for object container files
> ---------------------------------------
>
>                 Key: AVRO-498
>                 URL: https://issues.apache.org/jira/browse/AVRO-498
>             Project: Avro
>          Issue Type: Improvement
>          Components: spec
>            Reporter: Scott Banachowski
>         Attachments: AVRO-498.patch
>
>
> I have some proposed changes for the spec in the Object Container Files section.
> The fixes are:
> 1) call out explicitly that when it says long, it means avro-encoded long.
> 2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
> 3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
> Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (AVRO-498) Clarify spec for object container files

ASF subversion and git services (Jira)
In reply to this post by ASF subversion and git services (Jira)

    [ https://issues.apache.org/jira/browse/AVRO-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852951#action_12852951 ]

Scott Banachowski commented on AVRO-498:
----------------------------------------

Yes, in both cases (longs and metadata), I assumed these were the case from the context, but feel the spec could be more explicit about it.

> Clarify spec for object container files
> ---------------------------------------
>
>                 Key: AVRO-498
>                 URL: https://issues.apache.org/jira/browse/AVRO-498
>             Project: Avro
>          Issue Type: Improvement
>          Components: spec
>            Reporter: Scott Banachowski
>         Attachments: AVRO-498.patch
>
>
> I have some proposed changes for the spec in the Object Container Files section.
> The fixes are:
> 1) call out explicitly that when it says long, it means avro-encoded long.
> 2) since the implementations are serializing the metadata as an avro map, this map should be followed by a block size of 0 (consistent with avro map encoding)
> 3) fix the example header schema so it passes JSONLint (it had an extra comma after the last array value).
> Number 2 above is something I'm not sure about, because I don't know if it's a compatible change... I hope it does not break any implementations.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.