new xtype="json"

[mbtaylor]

I have heard a few people recently talking about putting JSON into VOTable fields. I can't immediately think of a reason why client code would need to know that the string data it's getting from a table is in JSON format, but it might do, and xtype would seem like the most appropriate place to include this information.

[Zarquan]

Would this be able to handle UTF8 strings embedded in the JSON ?
https://www.json.org/json-en.html

"A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes. "

[mbtaylor]

Unicode is a bit of a mess in VOTable anyway, but it's no worse for JSON than for other text content (all of the JSON syntax characters are representable in 7-bit ASCII, i.e. look the same in UTF8 and ASCII). So you could put a JSON string in either a char or unicodeChar column.

[msdemlei]

On Tue, Nov 14, 2023 at 09:03:18AM -0800, Zarquan wrote: Would this be able to handle UTF8 strings embedded in the JSON ? https://www.json.org/json-en.html > "A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes. "

I'm not sure whether I'd prefer that to finally allowing utf-8 in VOTable char, but sure, why not?

[pdowler]

Generally, the presence of an xtype means "you can parse this primitive value into an ???" where ??? is the xtype value, e.g. point. Here, that comes across as "you can parse this string into a json (object)" which isn't really saying anything about the kind of content one has to grok or not.

Consider xtype="moc" which in the details says "using ascii moc serialisation from document ...". That could have been and maybe someday will be "using json moc format from document ...".

json is a serialisation format, not an extended datatype in it's own right, so I'm against this. It doesn't say enough about the type of the value. I am not against serialising with json if that's the most appropriate way to deliver the structure efficiently.

[msdemlei]

On Mon, Nov 27, 2023 at 10:08:06AM -0800, Patrick Dowler wrote: point. Here, that comes across as "you can parse this string into a json (object)" which isn't really saying anything about the kind of content one has to grok or not.

Well, it is saying "you can turn this string into something *more* meaningful by passing it to a json parser". I could argue it's not *much* different from "turn this array into a circle" without saying whether that circle really is an error circle or an aperture or a region of interest. Turning the string into a parsed structure, in turn, could let people directly say col[3]["measurements"][20] in a client -- of course, presuming they know what to expect in that column. I'd hence argue that a json xtype has a fairly clear operational meaning, which is what I like about it. I readily admit that I, too, dislike much else about it, in particular the temptation for data providers to skip metadata declaration and proper relational design.

json is a serialisation format, not an extended datatype in it's own right, so I'm against this. It doesn't say enough about the type of the value. I am not against serialising with json if that's

I, too, would like it better if there were some machine-readable and verifiable way to declare the structure of such a json column. I haven't looked as JSON-schema, but assuming it's not dramatically less powerful than XML schema -- would you be less opposed to this if we required people to attach a JSON schema to a json column? Full disclosure: The reason I'm after this is that DaCHS users asked for ways to pull json/jsonb-valued columns into TAP responses, and I'm already producing xtype="json" for those, under the reasoning that it's better than betraying nothing about the inner structure of these strings. I can't say I like it, but there *are* use cases for that that I cannot readily reject.

[molinaro-m]

-- would you be less opposed to this if we required people to attach a JSON schema to a json column? Full disclosure:

Is there a way to do that? I'd really like if we find a way to allow for a schema attached.
I might have an additional constraint, that is JSON schema attached to column AND row.
I know, it gets too complicated, but if it can be kept in mind while evolving this...

[mbtaylor]

@pdowler wrote:

Generally, the presence of an xtype means "you can parse this primitive value into an ???" where ??? is the xtype value, e.g. point. Here, that comes across as "you can parse this string into a json (object)" which isn't really saying anything about the kind of content one has to grok or not.

Consider xtype="moc" which in the details says "using ascii moc serialisation from document ...". That could have been and maybe someday will be "using json moc format from document ...".

json is a serialisation format, not an extended datatype in it's own right, so I'm against this. It doesn't say enough about the type of the value. I am not against serialising with json if that's the most appropriate way to deliver the structure efficiently.

I think you could apply all those arguments to URI and UUID, which have been introduced as xtypes in this draft. A URI doesn't say what it's the URI of. Just as you might one day want to represent a MOC in JSON, you might want to represent a MOC using its URI. So do we chuck out xtype="uri" as well?

I would say that @msdemlei 's "you can turn this [string] into something more meaningful by [using its xtype]" is the most useful criterion here.

[pdowler]

My point is that parsing json into an programmatically accessible object is an intermediate representation, much like a DOM with xml, and then the intermediate representation gets pulled apart to create some structured object. That object has a type that would be part of a data model and the dm implementation would likely have a class corresponding to that type. eg xtype='uri" -> java.net.URI in my code. Sure, xtype="uri" allows for many different things (ivorn and url would be more restricted), but it's common in a DM to have a field of type "uri".

Yes, you can parse a string into a JSONObject or a DOM... I just don't see that as "more meaningful" - it is still opaque unless you know what structure it is trying to convey.

[msdemlei]

On Thu, Nov 30, 2023 at 09:28:11AM -0800, Patrick Dowler wrote: Yes, you can parse a string into a JSONObject or a DOM... I just don't see that as "more meaningful" - it is still opaque unless you

Well, being able to write col[3]["fluxes"][-1] in your code rather than json.loads(col[3])["fluxes"][-1] very arguably is at least a convenience. Whether that's already something in the way of "meaning" is... a question of what the meaning of meaning is, and I'd like to leave that question open. More pragmatically, I have these people with jsonb columns in their database, and they publish these things through TAP (or, let me hedge here: at least they want to). So, what should I do? If I just write char[*], I feel I'm concealing a *very* relevent fact about the column -- that it's parseable -- from the client. Should I say "this is json" using a UCD? That feels even more wrong than an xtype. Should I tell them to just not push out the jsonb stuff and unroll it through a view? That's an option, but I think in this case that's rather unattractive because none of what is in the JSON should be available for SQL constraints and operators, and of course I'm a coward and would like to avoid an ugly discussion when you have a four-level nested structure in this column. Should I do a <LINK content-type="application/json" href="."> in the FIELD? Uh, I can already see the pitchforks in front of my office... So... what would you do when someone came to you with a table with jsonb columns?

[molinaro-m]

I copy @msdemlei last question. How do I make a little more interoperable JSONB objects stores in a DB field?

[pdowler]

OK, that's fair. char and * are even less useful. Let me ask a leading question...

Right now we have xtype="moc" and the serialization is defined as "ascii moc from REC-MOC-2.0". Let's say a future version of the MOC standard defines "json moc" serialization and people want to put that into a VOTable. How would we handle that? xtype="jsonmoc"? xtype="jmoc"? (shudder) ... xtype="json:moc"? There are two pieces of info to convey and I kind of like the latter one.

I kind of think of xtype as a URN anyway and if we thought about having something like xtype="json:moc" in future, we could then also say that xtype="json" is just a less specific base... not super expressive but more expressive than nothing. We could still encourage people to try to say more with xtype="json:foo". We would also be able to handle different serializations of types without having to make up arbitrary compound words. I wouldn't want this to turn into UCD with rules about primary and qualifiers and such... still just strings but with a style that trivially conveys multiple bits of info. In that context, to me "json:moc" looks better than "moc:json" because that's how I expect people to say/read it.

thoughts?

[msdemlei]

On Fri, Dec 01, 2023 at 09:05:44AM -0800, Patrick Dowler wrote: Right now we have xtype="moc" and the serialization is defined as "ascii moc from REC-MOC-2.0". Let's say a future version of the MOC standard defines "json moc" serialization and people want to put that into a VOTable. How would we handle that? xtype="jsonmoc"?

I'd first say let's not invent more ways to say the same thing.

encourage people to try to say more with xtype="json:foo". We would also be able to handle different serializations of types without

Oh, please, let's just not have (more) things with more than one serialisation in VOTable *at all*; if there's one way to say it, that's enough. I think there is never (+/- epsilon) a benefit to supporting multiple serialisations that outweighs the pains in interoperability and implementability (tabledata vs. binary2 *may* be in that epsilon). As I said above, I do see a benefit in being able to somehow inform a client about the inner structure of a json-serialised field (perhaps JSON-schema, perhaps MIVOT, perhaps still something else) -- but I'd argue very strongly against trying to shoehorn this into an xtype string. These, I'd say, should be simple, atomic and hence unparseable.

[pdowler]

I'm not saying it's a good idea to have mutliple serializations (ascii and json, for example), just that xtype="json" by itself is not saying what kind of structure is there, just the syntax of the serialization. If we want to also use xtype to convey syntax then it opens up that can or worms... I'm against conveying syntax because to me it's very low (admittedly non-zero) value.

If someone wants to convey a some json, they surely have a specific structure in mind. They should use an experimental xtype that denotes the structure, e.g. xtype="proto:foo" and document (tap_schema.columns.description) what the syntax is. That's what I would do in that case.

[msdemlei]

On Tue, Dec 05, 2023 at 09:11:30AM -0800, Patrick Dowler wrote: (ascii and json, for example), just that xtype="json" by itself is not saying what kind of structure is there, just the syntax of the

I suppose our dissent basically is whether knowing something is JSON is already useful enough by itself. I'm not sure what a good way to settle that dissent is... can I argue that there is a media type for json, too, and that doesn't have a defined way to convey the structure, either?

If someone wants to convey a some json, they surely have a specific structure in mind. They should use an experimental xtype that denotes the structure, e.g. xtype="proto:foo" and document (tap_schema.columns.description) what the syntax is. That's what I would do in that case.

Oh, that may be another point of dissent: Frankly, I believe that the overwhelming majority of what people will stuff into jsonb columns is going to be fairly adhoc (which is why I'm not happy about the concept either -- but since it's there, I think we need to do *something* about it). This means that having two different json columns with the same structure/schema will be the huge exception, and hence we'll basically have an xtype string per json column. If that assessment is about right, the plan to define the structure in the xtype string in some way is a non-starter. No client will ever bother to learn such one-shot xtypes. *If* we really want a structure definition for json columns, it'll have to be machine-readable and sit outside of the xtype. Do we *need* such a mechanism to make a json xtype useful? Well, that's back to dissent 1.

[pdowler]

You could be right. I feel like I only know half of a use case, but maybe
it really is ad-hoc and that is the whole use case.