REL-1277692: Custom JSON Config Supporting RabbitMQ File Log Receiver#61
REL-1277692: Custom JSON Config Supporting RabbitMQ File Log Receiver#61
Conversation
amithshet
changed the title
…e files accordingly and inserted few screenshots.
There was a problem hiding this comment.
Pull request overview
Adds documentation and example configuration to support a RabbitMQ file log receiver via custom JSON (OpenTelemetry overrides) in the Environment Watch setup docs.
Changes:
- Adds
openTelemetryOverrides.logSources(RabbitMQ) to the sampleenvironment-watch-configuration.json. - Extends the troubleshooting guide with a File Log Receiver section and validation steps.
- Introduces a new “File Log Receiver Configuration” document and links it from the main Custom JSON configuration guide.
Reviewed changes
Copilot reviewed 4 out of 7 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| resources/custom-json-troubleshooting-images/environment-watch-configuration.json | Updates the sample JSON to include RabbitMQ file log receiver overrides. |
| elastic-stack-setup/troubleshooting/custom-json-troubleshooting.md | Adds troubleshooting workflow for RabbitMQ file log receiver via custom JSON. |
| elastic-stack-setup/elastic-stack-setup-02-environment-watch/ew-03-extensibility-configuration/ew-extensibility-configuration-06-json-configuration/ew-json-configuration-07-file-log-receiver.md | New dedicated guide for file log receiver configuration and Kibana verification. |
| elastic-stack-setup/elastic-stack-setup-02-environment-watch/ew-03-extensibility-configuration/ew-extensibility-configuration-06-json-configuration/ew-extensibility-configuration-06-json-configuration.md | Updates the overview doc to mention file log receivers and link to the new guide. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
amithshet
requested review from
anujavinash,
dinesh1010101 and
satishdayala-relativity
There was a problem hiding this comment.
@amithshet Reviewed the content and approved. Provided very minor comments, please address them before merging.
| # Custom JSON Configuration | ||
|
|
||
| This document provides an overview of the custom JSON configuration file used by Environment Watch. The configuration allows users to centrally define and customize monitoring for Windows services, certificates, SQL servers, and scrapers, as well as configure Slack notifications for alerting. | ||
| This document provides an overview of the custom JSON configuration file used by Environment Watch. The configuration allows users to centrally define and customize monitoring for Windows services, certificates, SQL servers, file log receivers, and scrapers, as well as configure Slack notifications for alerting. |
There was a problem hiding this comment.
Since JSON has been extended to support multiple components, this could be easier to read if it were broken into bullet points instead of a single sentence. This would make the documentation quicker to scan.
Suggested rewrite:
This document provides an overview of the custom JSON configuration file used by Environment Watch. The configuration allows users to centrally define and customize monitoring for:
Windows services
Certificates
SQL servers
File log receivers
Scrapers
It also includes configuration options for Slack notifications used for alerting.
There was a problem hiding this comment.
Yes, it is updated.
| ### Assumptions | ||
|
|
||
| - The log file is expected to be in **plaintext** format, not JSON. | ||
| - The log file path must be accessible from the host where the Environment Watch Windows service is running. |
There was a problem hiding this comment.
Clarify that the log file must be readable by the Windows account running the Environment Watch Windows service.
This helps avoid confusion where admins assume their own user permissions are sufficient, when the service may be running under a different account. For example: The log file path must be accessible from the host where the Environment Watch Windows service is running. The file must be readable by the Windows account under which the Environment Watch service runs.
There was a problem hiding this comment.
Added the following point:
The log file must be readable by the Windows account running the Environment Watch Windows service.
There was a problem hiding this comment.
can we combine these 2-bullet point as they are in example? as second line is more of a clarification.
|
|
||
| ### Default Behavior | ||
|
|
||
| If a log source is not explicitly configured in the `logSources` array, it will use its default settings from the source code. For example, if the RabbitMQ log source is not defined in the JSON configuration, it will not be enabled by default since the default `enabled` value for the RabbitMQ log source is `false` in the source code. To enable it, you must explicitly define it in the `logSources` array with `"enabled": true`. |
There was a problem hiding this comment.
Consider adding a short opening sentence to clarify that defaults apply at two different levels: log source enablement and individual configuration properties.
Updated:
Defaults apply at two levels: whether a log source is enabled, and the individual configuration properties used to parse logs.
If a log source is not explicitly configured in the logSources array, it will use its default settings from the source code. For example, if the RabbitMQ log source is not defined in the JSON configuration, it will not be enabled by default since the default enabled value for the RabbitMQ log source is false. To enable it, you must explicitly define it in the logSources array with "enabled": true.
At the property level, if a property is not set or is set to an empty or null value, it falls back to the default value defined in the source code. This applies to the following properties:
There was a problem hiding this comment.
Updated the suggested content.
|
|
||
| ## Overview | ||
|
|
||
| The file log receiver allows Environment Watch to collect and parse log files from custom sources using the OpenTelemetry Collector. By defining log sources in the `openTelemetryOverrides` section, users can extend monitoring to include application-specific log files (e.g., RabbitMQ logs) with custom parsing rules for log file path, multiline handling, regex extraction, and timestamp parsing. |
There was a problem hiding this comment.
Clarify who this documentation is intended for to set expectations early.
For example: Overview
The file log receiver allows Environment Watch to collect and parse log files from custom sources using the OpenTelemetry Collector. This configuration is intended for Relativity administrators managing Environment Watch on self‑hosted environments who want to extend monitoring to include application‑specific log files (for example, RabbitMQ logs).
By defining log sources in the openTelemetryOverrides section of the custom JSON configuration file, administrators can configure log file paths, multiline handling, regex‑based field extraction, and timestamp parsing.
There was a problem hiding this comment.
Revised the suggested content.
|
|
||
| --- | ||
|
|
||
| ## Configure File Log Receiver |
There was a problem hiding this comment.
From a Relativity administrator perspective, this section correctly emphasizes that regex configuration and named capture groups are critical to successful log ingestion. Because regex behavior is highly sensitive and varies by log format, it may help to add a small amount of expectation‑setting language without prescribing specific regex implementations. Specifically, consider reinforcing that:
The regex patterns shown are examples aligned to known RabbitMQ log formats, and should be validated against the admin’s actual log output before being used in production.
Timestamp parsing and field extraction are common failure points, and malformed regex can result in logs being dropped or the collector failing to start.
This would help admins understand the operational risk and validation expectation, while still keeping the documentation flexible and not locking in specific regex patterns.
There was a problem hiding this comment.
In here, updated the suggested points under 'Important' banner to convey the operational risk and validation expectation associated with this configuration.
The regex patterns shown are examples aligned to known RabbitMQ log formats, and should be validated against the admin’s actual log output before being used in production.
Timestamp parsing and field extraction are common failure points, and malformed regex can result in logs being dropped or the collector failing to start.
KarunaDhawan
left a comment
KarunaDhawan
left a comment
There was a problem hiding this comment.
Just left a comment beside that everything looks correct format wise.
| ### Assumptions | ||
|
|
||
| - The log file is expected to be in **plaintext** format, not JSON. | ||
| - The log file path must be accessible from the host where the Environment Watch Windows service is running. |
There was a problem hiding this comment.
can we combine these 2-bullet point as they are in example? as second line is more of a clarification.
Custom JSON Config Supporting RabbitMQ File Log Receiver
Please find the SQE-verified evidence attached.
REL-1277692 Custom JSON Config Supporting RabbitMQ File Log Receiver Test Results latest.docx