DMI-TCAT extension for 4CAT

This is a 4CAT extension that packages the integration with DMI-TCAT (the Digital Methods Initiative's Twitter Capture and Analysis Toolkit).

DMI-TCAT is no longer actively maintained, so its integration has been moved out of the 4CAT core codebase and into this separate extension. Installations that still work with TCAT query bins can add this extension to keep that functionality; installations that do not are no longer carrying code they never use.

What this extension provides

Component	Type	What it does
DMI-TCAT Search (HTTP)	data source (dmi-tcat)	Creates datasets from a TCAT query bin through the TCAT web front-end's tweet export endpoint.
DMI-TCAT Search (MySQL)	data source (dmi-tcatv2)	Creates datasets by querying a TCAT MariaDB/MySQL database directly, including free-form queries.
Upload to DMI-TCAT	processor (preset-upload-tcat)	Preset that converts a Twitter/X dataset to TCAT's import format and uploads it to a configured TCAT server.
Convert NDJSON to TCAT JSON	processor (convert-ndjson-for-tcat)	Converts a Twitter/X (twitterv2) dataset into the JSON format TCAT can import.
Upload to TCAT server	processor (tcat-auto-upload)	Uploads a converted dataset to a TCAT server and defines the tcat-auto-upload.* settings.

What stayed in 4CAT core

The Twitter statistics processors (processors/twitter/* — tweet stats, hashtag stats, aggregated stats, mentions export, and so on) are not part of this extension. They also work on the Twitter/X API data source (twitterv2), which remains in core, so they stay there. They already declare compatibility with dmi-tcat-search, so they keep working on TCAT datasets whenever this extension is installed — matching happens by data source name, so no change is needed on either side.

Requirements

• A 4CAT installation at version 1.56 or newer (the processors rely on common.lib.compatibility, added around that release).
• The core twitterv2 data source must remain available: the HTTP data source reuses its tweet-mapping code (from datasources.twitterv2.search_twitter import SearchWithTwitterAPIv2).
• PyMySQL for the MySQL data source (see requirements.txt). On a standard 4CAT install this is already present, because core's 4chan data source uses the same backend.lib.database_mysql helper.

Installation

Through the 4CAT web interface (recommended)

1. In 4CAT, go to the Settings → Extensions page (admin only).
2. Add the repository URL and install. 4CAT clones the repository, reads metadata.json, enables the extension, and restarts to apply migrations and install requirements.txt.

The installer renames the cloned folder to the manifest id (tcat), so it ends up at config/extensions/tcat/ regardless of the repository name.

Configuration

Once installed, configure the data sources and upload target under 4CAT Settings:

• DMI-TCAT Search (HTTP) — the TCAT instance(s) to connect to. See datasources/dmi-tcat/README.md.
• DMI-TCAT Search (MySQL) — the TCAT database connection details. Use a read-only database user. See datasources/dmi-tcatv2/README.md.
• Upload to DMI-TCAT — the tcat-auto-upload.* settings (server URL, username, password, token) used by the upload processors.

All of these settings are defined by the extension's own code, so they appear in the settings interface only while the extension is installed.

Layout

tcat/
├─ metadata.json
├─ requirements.txt
├─ processors/
│  ├─ upload-to-dmi-tcat.py
│  ├─ twitter_ndjson_to_tcat_json.py
│  └─ tcat_auto_upload.py
└─ datasources/
   ├─ dmi-tcat/
   │  ├─ __init__.py
   │  ├─ search_tcat.py
   │  ├─ README.md
   │  └─ dmi-tcat-explorer.css
   └─ dmi-tcatv2/
      ├─ __init__.py
      ├─ search_tcat_v2.py
      ├─ README.md
      └─ dmi-tcat-explorer.css

Note on existing datasets

Datasets that were created with the TCAT data sources while they were part of core have the types dmi-tcat-search or dmi-tcatv2-search. To keep viewing and reprocessing those datasets after the code is removed from core, this extension must be installed.