3di | Docs toolchain modifications to support information design

[3di-techx]

Description

Update the way ROS and related documentation is created and compliant, to enable the 3di content strategy and information architecture. Including the following improvements to the documentation:

• Better site search to help with findability. This would be achieved by providing search result filters, and enabling search across multiple sites
• Include lists of related packages to better integrate the Package Index and ROS docs and help readers find the packages they need
• Included lists of related articles to help readers find related content
• Add custom metadata and short descriptions to articles to support SEO, and AIO

Did you use Generative AI?

Yes - a mixture of models from Cursor IDE; including Gemini 3.1 Pro, Sonnet 4.6, and Composer 2.1

Additional Information

Updates from 3di Information Solutions, agreed with Geoff and Tully, and discussed in call 26th May. Consultant for 3di George Lewis has delivered a document of recommendations to accompany this PR.

[kscottz]

I think that will fix the make file so CI runs.

@fujitatomoya this seems a bit more in your wheel house, I would value your input.

@3di-techx this is a lot to digest. You gave us four bullet points for 44 file changes. Can you give a bit more detailed summary of what's going on here? Screenshots of what the final outcome looks like?

[fujitatomoya]

as far as i understand whta this is trying to do, i really suggest to break it down to separate PRs for reviewability. pagefind search, related-packages, related-articles, and the AI enhancement tooling are independently reviewable and independently revertable. i am not inclined to take this PR as this is, that is hard to maintain.

besides, this hard-requires OpenAI API key to use it. is this also accepted by TGC? i think we are allowed to use AI tools on own responsibility, but this PR integrate OpenAI API in this repository. it depends on the user preference, some people use gemini or claude, it does not work for those users. it's wired to one commercial vendor and a paid account, i do not think this is good approach.

besides a couple of workflows are failing, those need to pass 1st of all.

[kscottz]

besides, this hard-requires OpenAI API key to use it. is this also accepted by TGC? i think we are allowed to use AI tools on own responsibility, but this PR integrate OpenAI API in this repository. it depends on the user preference, some people use gemini or claude, it does not work for those users. it's wired to one commercial vendor and a paid account, i do not think this is good approach.

@fujitatomoya I saw that when I did quick read through. Quite frankly, of all of the models services out there OpenAI would be my last pick. My personal preference would be to use an open model like Qwen so we can have some bounds on our costs. I don't know what the status is for us on acquiring free-access to the proprietary models but we should probably default to that. The reality is that we probably want a layer of indirection so we're not beholden to any one provider.

Tracking through the code a bit, and I still don't fully grok what's going on, I am a bit concerned that this is an all or nothing utility (basically you call the makefile command and it updates the entire tree). What's the mechanism for caching the AI output and only generating results for newly added entries? I am also not sure if it is wise to frequently update page-level SEO. Is there functionality to more specifically target just an SEO updates or search updates?

@3di-techx can you speak to this a bit? Like @fujitatomoya said, this would be a whole lot easier to both grok and review as a series of incremental pull requests.