Include YAML as well as JSON

[Zarquan]

As part of the group looking at updating our standards to move away from XML I propose we adopt the following code of practice for all our standards.

Wherever we propose a change to handle JSON we MUST also include an equivalent change to handle YAML.

For example in our discussion about xtype="json"(#33), we MUST also include the same functionality to handle xtype="yaml".

This doesn't add much more complexity, but it forces us think about making our standards serialization agnostic, rather than just moving from one fixed serialization to another.

[Zarquan]

(a) nobody's asking us for that,

I am, precisely because it forces us to think outside the box.

(b) there's no database column type named "yaml" (though admittedly
you could serialise the content of jsonb columns as yaml -- but
why would you do that?)

Is there a hard link between xtype and the database column type ?

Are we suggesting that if a client creates a VOTable containing a column with xtype="json", a database MUST store this as a native JSON column or reject a TAP upload ?

If we allow databases that don't have support for native JSON to store xtype="json" as VARCHAR(), then we should also allow the same for xtype="yaml".

(d) yaml support, while widespread, still is a lot more limited
that json support (e.g., no module in the Python standard
library), and

There is no standard library for JSON in Java. Implementations have to import an external library for it, but that hasn't stopped people claiming JSON is a must have.

Most Java developers use the Jackson libraries, which include support for several serializations including JSON, XML and YAML.

(e) yaml uses whitespace for grammar. In theory, VOTable preserves
whitespace (by virtue of TD being xs:string even in TABLEDATA).
In practice, given XML prettyprinters and all that, I'd rather
not bet on it.

Claiming we can't handle YAML because our standard is tied to an old XML schema that doesn't preserve spaces is
exactly the kind of thing this issue is intended to make us think about.

We should see this as a reason for to update the VOTable standard to preserve spaces, not a reason to avoid new things because we are stuck with an old XML schema.

[Zarquan]

We should see this as a reason for to update the VOTable standard to preserve spaces

See VOTable#54 and VOTable#55

[Zarquan]

(e) yaml uses whitespace for grammar. In theory, VOTable preserves
whitespace (by virtue of TD being xs:string even in TABLEDATA).
In practice, given XML prettyprinters and all that, I'd rather
not bet on it.

You are right, this could happen - I didn't read your comment carefully enough.

[msdemlei]

On Fri, Dec 15, 2023 at 07:20:16AM -0800, Zarquan wrote: > (a) nobody's asking us for that, I am, _precisely because_ it forces us to think outside the box.

Well, as long as we don't forget our client implementors (they're the ones that count) and don't *entirely* discard the all-important question "what does it buy us?", I'm fine with that.

> (b) there's no database column type named "yaml" (though admittedly > you could serialise the content of jsonb columns as yaml -- but > why would you do that?) Is there a hard link between xtype and the database column type ?

No, but having a soft one certainly helps a lot to produce predictable services. If we had essentially equivalent xtypes yaml and json, would I be allowed to upload a yaml FIELD into a jsonb column? And when returning that column, would I be allowed to return it in a json FIELD? You see, these kinds of things come with a rather hefty price tag in terms of implementation and standardisation. So, let's not try it *just* to see whether we can.

Are we suggesting that if a client creates a VOTable containing a column with `xtype="json"`, a database MUST store this as a native JSON column or reject a TAP upload ?

That's a question we've been somewhat weaseling around with geometries, where I've long been requesting that I'm allowed to return any geometry as a MOC. That's no longer as dramatic as it was since we've defused multipolygon, but there are interesting pros and cons to type mapping uploads. My pragmatic short answer is: As long as ADQL doesn't have any operators that would do magic on such colums, it's probably reasonable to pull the JSON through as strings; I don't think we require that the xtype is being passed back to the uploader, though. We probably should. Once there are operators for such columns (and I'd expect these to come for timestamps before they'd come for json), it's becoming a question of when to raise an error, and by gut feeling I'd lean towards failing early (although that would then fail even queries that don't use any such operators and hence would work if we looked the other was... hm). Figuring *this* problem out is a matter for a different bug -- and probably for asking users for what they'd consider the best compromise between non-surprising and non-annoying in the various scenarios there are.