Database can fall out of sync with published posts if publication configuration changes

[paulrobertlloyd]

Lots of things can change over the course of a post’s lifetime:

• Publication:
  • publication.me
  • publication.slugSeparator
  • publication.postTypes
• Post
  • path
  • properties.published
  • properties.slug
  • properties.url

TLDR: Changing any post property will update a post’s record in the database, but changing any publication value will not. As publication configuration can affect post paths and URLs, a post’s path or properties.url value is not authoritative.

Currently Indiekit looks for posts based on a given URL. After all, that’s the information provided in a Micropub update or delete action. However, there are a few scenarios where this causes problems.

Syndicating posts

The syndication endpoint can be asked to syndicate a given URL. If a post with that URL is found in the database, and that post can be syndicated, it will be syndicated.

The syndication endpoint also allows for the last post awaiting syndication to be found.

As part of the update process, besides adding the new syndication URLs, all post properties are updated - including path and properties.url. Should any publication value have changed, when Indiekit comes to look for a post in the database with the new URL/path, it won’t be found.

Given the following scenario, with syndication plugins for Twitter and Mastodon installed, and the Netlify deploy hook configured, a site will be continually republished:

1. A post is published.
2. In the interim, some aspect of a post or the publication’s configuration is altered.
3. The syndication is configured, with a post deploy notification hook in Netlify.
4. Syndication is requested for the most recent un-syndicated post. The most recent post in the database is found. A new path/URL is generated as part of the update operation, using the updated publication values, and a file with the updated properties is published.
5. The database is then queried for the post to update. As a post with the new URL can’t be found, the database is not updated. There is now a discrepancy between the data in the published file in a content store, and the post in the database.
6. Netlify fires the post deploy notification.
7. Syndication is required for the most recent un-syndicated post. The same post is found, and syndication is repeated.
8. Twitter rejects the status update as it matches a recent status, but Mastodon publishes a new post. The post file is updated, this time without the original syndicated Twitter URL, and the previous Mastodon URL is replaced by the newly syndicated URL on Mastodon.
9. Netlify fires the post deploy notification, go to 7.

(See: #569)

Changing a post slug

I came across a similar issue when changing the post slug on a published post. I suspect the above issue was partly to blame.

Potential options to resolve

• Update database before content store #583
• Allow posts to be updated by querying either a URL (given an external Micropub request) or _id (internal Indiekit operation)
• Perform any update operation directly in MongoDB, on the database object itself, using Mongo’s find and replace operations. That means the database is always the single point of truth.
• Don’t keep a permanent record of posts, delete after 30 days?

This is a fairly consequential (existential!) issue with the design of Indiekit. If you don’t change anything in your publication settings, all will remain well, yet in the early stages of experimenting with the product, lots of settings can and will be changed, meaning this issue is more likely to appear, and thus reducing trust in the robustness of this software.

[paulrobertlloyd]

Option 1 seems like a good first step, and would be a fairly simple to change to make.

[sentience]

Your issue description mentions #570, but I think you meant to link to #569.