This is a small experiment to find out how we can use LLMs for big code analysis tasks, where just dropping all code into the LLM and calling a (huge) number of manual prompts is not possible because of context size limits and context quality issues with much data.
The general assumption is that switching from a big prompt an all data in to an iterative approach with small consecutive prompts should be more successful.
The application implements the following strategies to handle the above issues:
- Automate prompt calling using a scriptable execution engine
- Enforce JSON response in all cases, using custom managed result object JSON schemas as input to the LLM.
- Explicitly build an "over all" state/context in form of a JSON structure (in memory and on disk).
- Use a template engine to enrich prompts with dynamic content from this state, thus building minimal prompts as small and selective as possible.
- Use template inclusion to centralize common facts for multiple prompts.
- Allow looping over substructures of the context state, thus allow iterations over findings (like the list of identified source modules).
- Implement dependency handling between prompt calls to allow the engine to always call prompts at times, where depending on information was successfully gathered.
- Store execution state to be able to recover from execution failures.
- Use a template engine to postprocess the gathered state and generate a summarizing documentation again.
- Make use of embedded MCP tools, to modularize analysis capabilities and make the engine extendable.
The current code implements prompts, result objects and documentation templates for executing a simple software architecture analysis.
However, the engine could be used for other purposes, too
The software and the prompts are licensed under GPL.
We use Mise for a definition of used developer tools versions. See the mise.toml for details.
For coding agents and developer conventions, see AGENTS.md and the files in guidelines/.
For simpler test setup we use the repository itself for testing: some tests parse the aggregate SBOM that the build generates under target. That SBOM is created in the process-classes phase, before the tests run, so a single command is enough on a clean checkout:
mvn clean package
With the artefact smoke test included (see below), use:
mvn verify
The build also produces a self-contained artefact, target/LLMAnalysisJinni-jar-with-dependencies.jar, which can be started with java -jar. The verify phase runs an artefact smoke test against it, because the unit tests run on the full classpath and cannot see that a class only resolved at runtime is missing from the packaged jar. That test checks that the jar writes diagnostics to the console and that XML parsers can be created from it. A run started from the jar that produces no output at all is therefore a packaging defect, not a silent success: check the build, not the model configuration.
The terminal UI needs restricted native access for its terminal implementation. The executable jar declares Enable-Native-Access: ALL-UNNAMED in its manifest, which the JDK honours for java -jar. When starting the application another way, for example with mvn exec:java, pass --enable-native-access=ALL-UNNAMED yourself (for example through MAVEN_OPTS), otherwise the JVM prints native access warnings and a future JDK will block the terminal implementation.
Create an empty workspace directory:
mkdir workspaces/test
Call the tool to initialize the workspace and store the required parameter in the configuration file:
workspace init --modelProvider=OLLAMA--modelUrl http://<ollama-url> --model "gpt-oss:20b" -j=true -- <project diretory> analysis/software-architecture <workspace directory>
This creates a config.json in the workspace directory
A typical call to analyze a given project would be:
analyse <workspace directory>
Note, that you only need to pass the workspace directory, because all other required configuration is already stored there.
Tasks whose dependencies are satisfied execute in parallel (configurable via --task-parallelism or config.json). The analysis creates an analysis.json (containing the analysis result) and a state.json (containing tasks execution information) in the workspace directory. Chat interactions are logged per task in logs/<taskId>[_<loopIndex>].log. While the TUI is the display mode, the engine's own log records go to logs/engine.log (overwritten per run) instead of the terminal, and the latest warning or error is also shown inside the frame.
Command line options only override the workspace configuration when they are actually passed:
- A value passed on the command line (for example
--log-response true) wins. - Otherwise the value stored in the workspace
config.jsonapplies. - Otherwise the built-in default applies.
Leaving an option out therefore never resets a setting that was configured in the workspace. The effective value of every such setting and the source it came from are reported in the run log after the configuration is loaded.
The default column below is the built-in default, which only applies when neither the option nor config.json supplies a value.
| Option | Description | Default |
|---|---|---|
--task-parallelism |
Number of parallel DAG tasks to execute concurrently | 2 (built-in) |
--single-step |
Stop execution after one task (including its loop indices) | false |
--execution-trace |
Show verbose SLF4J chat execution trace on console (disables TUI) | false |
--execution-trace-system |
Show system messages in console execution trace | false |
--log-request |
Activate langchain4j low-level log of chat requests | false |
--log-response |
Activate langchain4j low-level log of chat responses | false |
-o / --executeOnly |
Task ID to execute exclusively. Repeat the option (-o A -o B) or use a comma-separated list (-o A,B); ids are not accepted as separate arguments. The workspace directory may be given before or after the option. |
(all active) |
The tool selects the display mode automatically:
- TUI mode (default): Live-updating terminal UI showing task list, status icons, elapsed time, per-worker interaction timelines, loop progress, and token usage. Used when stdout is a terminal and the execution trace is not active.
- Simple mode (non-TTY): Sequential status lines when output is piped, when stdout is not a terminal, or when no terminal can be created.
- Execution-trace mode: Verbose SLF4J console output with full chat message history. Each log line is tagged with the originating task ID via SLF4J MDC. Enabled by
--execution-trace=trueor by the execution trace entry inconfig.json, and mutually exclusive with the TUI.
Whenever the TUI is not used, the run reports the display mode it uses instead and the reason the TUI was not started, so a fall back to plain output is always visible.
If a loop task fails on some indices, the task is marked as failed and will be re-executed on the next run. Already successful indices are preserved and skipped on retry.
For generating a documentation of the analysis results, a possible command line could be:
document workspaces/spring-petclinic
Again, all other information is loaded from the configuration file in the workspace directory.
The documentation is generated as an Asciidoc file (Documentation.adoc by default).
The output format can be changed with the --document-postfix option:
document --document-postfix .md workspaces/spring-petclinic
For Java source analysis, LLMAnalysisJinni needs access to all compiled dependency JARs to resolve types and class references. Set the property "javaTool.jarDependenciesDirectory" in config.json to the path containing all required *.jar files.
For Maven projects, this directory can be created by running:
mvn dependency:copy-dependencies -DoutputDirectory=libs
Then point the config to this directory:
{
"properties": {
"javaTool.jarDependenciesDirectory": "/path/to/project/libs"
}
}The config.json file has the following attributes:
| Attribute | Desription | Default |
|---|---|---|
| modelProvider | Name of the model provider, currently 'OLLAMA' or 'OPENAI' | |
| modelURL | The full connection URL to the ollama instance | |
| modelName | The name of the model, e.g. "gpt-oss:20b" | |
| chatWindowSize | Number of messages in context during one task execution | 50 |
| requestTimeout | Request timeout in minutes | 120 |
| maximumTokens | The maximum number of tokens | 65536 |
| nativeJSON | Use the native JSON support | false |
| logRequests | Log HTTP request | false |
| logResponses | Log TTP responses | false |
| taskParallelism | Number of parallel DAG tasks to execute concurrently | 2 |
| loopParallelism | Number of parallel workers for loop tasks | 1 |
| mcpServers | An array of MCP Servers | |
| projectDirectory | Path to the project directory | |
| analysisDirectory | Path to the analysis files | |
| javaTool.jarDependenciesDirectory | Path to directory containing all dependency JARs for Java analysis |
For the MCP Server configuration:
| Attribute | Desription | Default |
|---|---|---|
| name | Free form name of the MCP Server | |
| type | Type of connection, currently 'http' or 'stdio' | |
| url | Connection URL for type 'http' | |
| command | Array of command line arguments for type 'stdio' | |
| environment | Map of evironment variables names and values for 'stdio' | |
| logEvents | Log events for 'stdio' | false |
The implementation makes use of the following frameworks and libraries:
- Langchain4j for accessing models in chat mode
- Handlebars for templating of prompts and result document
- Jackson for (de)serializing JSON and YAML
- Picocli for CLI parsing
- JavParser for parsing Java Files
- FastCSV for writing CSV files
- networknt/json-schema-validator for validating JSON schemas and data against JSON schemas
- JLine 3 for terminal handling (TUI display, ANSI colours, cursor control)