diff --git a/migration/mongosync_insights/CONFIGURATION.md b/migration/mongosync_insights/CONFIGURATION.md index 5c9b6425..5b209652 100644 --- a/migration/mongosync_insights/CONFIGURATION.md +++ b/migration/mongosync_insights/CONFIGURATION.md @@ -4,7 +4,7 @@ This document explains the configuration management system for Mongosync Insight ## Prerequisites -**Python 3.11+** and **libmagic** system library are required to run Mongosync Insights. See [README.md](README.md) for complete installation instructions including system dependencies. +**Python 3.11+** is required to run Mongosync Insights. See [README.md](README.md) for complete installation instructions. ## Configuration Overview @@ -34,7 +34,7 @@ All configuration can be set using `export` commands before running the applicat |----------|---------|-------------| | `MI_CONNECTION_STRING` | _(empty)_ | MongoDB connection string (optional, can be provided via UI) | | `MI_VERIFIER_CONNECTION_STRING` | _(falls back to `MI_CONNECTION_STRING`)_ | MongoDB connection string for the migration verifier database. When omitted, the value of `MI_CONNECTION_STRING` is used. Set this when the verifier database lives on a different cluster. | -| `MI_INTERNAL_DB_NAME` | `mongosync_reserved_for_internal_use` | MongoDB internal database name | +| `MI_INTERNAL_DB_NAME` | _(auto-detected)_ | MongoDB internal database name. When not set, the app auto-detects between `__mdb_internal_mongosync` (new) and `mongosync_reserved_for_internal_use` (legacy). Set this variable to override auto-detection. | | `MI_POOL_SIZE` | `10` | MongoDB connection pool size | | `MI_TIMEOUT_MS` | `5000` | MongoDB connection timeout in milliseconds | @@ -55,7 +55,7 @@ All configuration can be set using `export` commands before running the applicat | Variable | Default | Description | |----------|---------|-------------| -| `MI_ERROR_PATTERNS_FILE` | `error_patterns.json` _(same directory as the application)_ | Path to a custom error patterns JSON file used during log analysis to detect common errors (e.g., oplog rollover, timeouts, verifier mismatches) | +| `MI_ERROR_PATTERNS_FILE` | `lib/error_patterns.json` _(auto-detected)_ | Path to a custom error patterns JSON file used during log analysis to detect common errors (e.g., oplog rollover, timeouts, verifier mismatches) | ### UI Customization @@ -63,6 +63,16 @@ All configuration can be set using `export` commands before running the applicat |----------|---------|-------------| | `MI_MAX_PARTITIONS_DISPLAY` | `10` | Maximum partitions to display in UI | +### Log Viewer & Snapshot Settings + +| Variable | Default | Description | +|----------|---------|-------------| +| `MI_LOG_VIEWER_MAX_LINES` | `2000` | Maximum number of recent log lines shown in the Log Viewer tail view | +| `MI_LOG_STORE_DIR` | System temp directory | Directory for SQLite log stores and analysis snapshot files | +| `MI_LOG_STORE_MAX_AGE_HOURS` | `24` | TTL in hours for log store and snapshot files (based on last-access mtime) | + +> **Note**: By default, log store databases and snapshot files are saved to the OS temp directory (e.g., `/tmp` on Linux/macOS), which may be cleared on system reboot. Set `MI_LOG_STORE_DIR` to a persistent path (e.g., `/data/mongosync-insights/store`) to retain snapshots across restarts. Files are cleaned up automatically on app startup, on logout, and lazily on access when they exceed the configured TTL. Loading a saved snapshot resets its TTL by touching the file's modification time. + ### Security Settings | Variable | Default | Description | @@ -84,7 +94,7 @@ All configuration can be set using `export` commands before running the applicat --- -## 🚀 Usage Examples +## Usage Examples ### Example 1: Basic Local Development @@ -220,6 +230,24 @@ python3 mongosync_insights.py **Note**: When `MI_VERIFIER_CONNECTION_STRING` is not set, it falls back to `MI_CONNECTION_STRING`. Set it explicitly when the migration-verifier writes to a different cluster. +### Example 8: Persistent Snapshots and Custom Log Viewer + +Configure snapshot storage location, retention period, and log viewer buffer size: + +```bash +# Store snapshots in a persistent directory +export MI_LOG_STORE_DIR=/data/mongosync-insights/store + +# Keep snapshots for 48 hours instead of the default 24 +export MI_LOG_STORE_MAX_AGE_HOURS=48 + +# Show up to 5000 recent log lines in the Log Viewer tail view +export MI_LOG_VIEWER_MAX_LINES=5000 + +# Run the application +python3 mongosync_insights.py +``` + --- ## Troubleshooting diff --git a/migration/mongosync_insights/README.md b/migration/mongosync_insights/README.md index 8d628bd5..ae306726 100644 --- a/migration/mongosync_insights/README.md +++ b/migration/mongosync_insights/README.md @@ -4,12 +4,13 @@ This tool can parse **mongosync** logs and metrics files, read the **mongosync** ## What Does This Tool Do? -Mongosync Insights provides four main capabilities: +Mongosync Insights provides five main capabilities: 1. **Log File Analysis**: Upload and parse mongosync log files to visualize migration progress, data transfer rates, performance metrics, configuration options, and detected errors 2. **Mongosync Metrics Analysis**: Upload and parse `mongosync_metrics.log` files to visualize 40+ mongosync metrics across Collection Copy, CEA, Indexes, Verifier, and more 3. **Live Monitoring**: Connect directly to the **mongosync** internal database or to the **mongosync** progress endpoint for real-time monitoring of ongoing migrations with auto-refreshing dashboards -4. **Migration Verifier Monitoring**: Connect to the database where the [migration-verifier](https://github.com/mongodb-labs/migration-verifier) tool stores its metadata to track verification progress, generation history, and mismatch details +4. **Combined Monitoring**: Provide both a MongoDB connection string and a progress endpoint URL to get a comprehensive view that merges metadata insights with real-time progress data +5. **Migration Verifier Monitoring**: Connect to the database where the [migration-verifier](https://github.com/mongodb-labs/migration-verifier) tool stores its metadata to track verification progress, generation history, and mismatch details ## Prerequisites @@ -64,7 +65,7 @@ python3 mongosync_insights.py The application will start and display: ``` -Starting Mongosync Insights v0.8.0.18 +Starting Mongosync Insights v0.8.1.14 Server: 127.0.0.1:3030 ``` @@ -79,6 +80,15 @@ http://localhost:3030 ## Using Mongosync Insights +### Sidebar Navigation + +Results pages include a left sidebar with quick-access buttons: + +- **Upload** — opens a dialog listing saved analyses with **Load** and **Delete** actions, plus an **"Upload New File"** button to parse a new log file +- **Settings** — configure the live monitoring refresh interval, theme (Light, Dark, or System), and color scheme (MongoDB Green, Blue, Slate, Ocean) +- **Logout** — clears the current session and returns to the home page +- **Credits** — displays developer credits + ### Option 1: Parsing Mongosync Log Files 1. Click the **"Browse"** or **"Choose File"** button @@ -86,6 +96,8 @@ http://localhost:3030 3. Click **"Open"** or **"Upload"** 4. The application will process the file and display results across multiple tabs +**Duplicate Upload Detection:** If you upload a file with the same name as an existing saved analysis, a dialog will appear offering three options: **Load Previous** (open the saved session without re-parsing), **Replace** (delete the saved session and parse the file again), or **Cancel**. + **Supported File Formats:** - Plain text: `.log`, `.json`, `.out` - Compressed: `.gz`, `.zip`, `.bz2`, `.tar.gz`, `.tgz`, `.tar.bz2` @@ -111,12 +123,24 @@ After upload, the results are organized into tabs: | **Options** | Mongosync configuration options extracted from the logs (with **Copy as Markdown** for easy sharing) | | **Collections** | Collection-level progress details (with **Copy as Markdown** for easy sharing) | | **Errors** | Detected error patterns such as oplog rollover, timeouts, verifier mismatches, and write conflicts during cutover | +| **Log Viewer** | Browse recent log lines with severity filtering, semantic focus, multiple view modes (Highlighted, Raw, Pretty JSON, Summary), and full-text search across the entire log file | ![Mongosync Logs Tab](images/mongosync_logs_logs.png) ![Mongosync Metrics Tab](images/mongosync_logs_metrics.png) ![Mongosync Options Tab](images/mongosync_logs_options.png) ![Mongosync Collections and Partitions Tab](images/mongosync_logs_collections_partitions.png) ![Mongosync Errors and Warnings Tab](images/mongosync_logs_errors.png) +![Mongosync Log Viewer Tab](images/mongosync_logs_logviewer.png) + +#### Analysis Snapshot Persistence + +After parsing a log file, the analysis is automatically saved as a **snapshot** to disk. This allows you to reload a previous analysis instantly without re-parsing the original file. + +- The home page displays a **"Previous Analyses"** section below the upload form, listing all saved snapshots with their filename, date, file size, and age +- Click **"Load"** to reopen a saved analysis — all tabs (plots, tables, log viewer) are restored immediately +- Click the **delete** button to remove a snapshot you no longer need +- Snapshots expire automatically after **24 hours** of inactivity; each time you load a snapshot, the TTL resets for another 24 hours +- By default, snapshots are stored in the system's temp directory. Use the `MI_LOG_STORE_DIR` environment variable to set a persistent storage location. See [CONFIGURATION.md](CONFIGURATION.md) for details ### Option 2: Live Monitoring (Metadata) @@ -173,6 +197,18 @@ This combined approach provides: - Full metadata insights from the destination cluster (partitions, collection progress, configuration) - Real-time progress data from the mongosync endpoint (state, lag time, verification status) +#### About the Embedded Verifier + +The [Embedded Verifier](https://www.mongodb.com/docs/cluster-to-cluster-sync/current/reference/verification/embedded/) is mongosync's built-in verification mechanism, available since mongosync v1.9 and enabled by default. It performs document hashing on both source and destination clusters to confirm data was transferred correctly, without requiring any external tools. + +**Embedded Verifier field (Status tab — Option 2):** The "Embedded Verifier" field displays the `verificationmode` value from mongosync's internal metadata. Possible values: `Enabled` (default — verification is active) or `Disabled` (verification was turned off at start). + +**Can Write signal (Endpoint tab — Option 3):** `Can Write: True` is the definitive signal that the embedded verifier has completed successfully and found no mismatches. Until verification passes, `Can Write` remains `False`. This is the key field to watch for confirming migration correctness. + +**Verification phases (Endpoint tab — Option 3):** The "Embedded Verifier Status" table shows a `Phase` field for both source and destination independently. Key phases include `stream hashing` (actively hashing documents from change streams) and `idle` (not yet started or between operations). + +**Verifier Lag Time (Endpoint tab and uploaded metrics):** The `Lag Time Seconds` field in the verification table (and `Verifier Lag Time` in uploaded `mongosync_metrics.log` files) shows how far behind the verifier is in checking documents. High lag means verification will take longer to complete after commit. Persistently high lag may indicate the verifier cannot keep up with the write load. + ### Option 5: Migration Verifier Monitoring 1. Enter the MongoDB **connection string** to the cluster where the [migration-verifier](https://github.com/mongodb-labs/migration-verifier) tool writes its metadata (typically the destination cluster) @@ -187,6 +223,31 @@ This combined approach provides: ![Migration Verifier Dashboard](images/migration_verifier_dashboard.png) +#### Important: Embedded Verifier + +> If verifying a migration done via mongosync, please check if the [Embedded Verifier](https://www.mongodb.com/docs/cluster-to-cluster-sync/current/reference/verification/embedded/) can be used, as it is the preferred approach for verification. + +#### About Migration Verifier + +The [migration-verifier](https://github.com/mongodb-labs/migration-verifier) is a standalone tool that validates migration correctness by comparing documents between source and destination clusters. It stores its state in a MongoDB database (default: `migration_verification_metadata`). + +**How it works:** The verifier operates in two phases. First, an initial check (generation 0) partitions the source data into chunks and compares documents byte-by-byte between source and destination. Then, iterative rechecks (generation 1, 2, ...) re-verify any documents that changed or failed during previous rounds. Only the **last generation's failures** are significant — earlier failures may be transient due to ongoing writes. + +**Key terms:** + +| Term | Description | +|------|-------------| +| **Generation** | A round of verification. Generation 0 is the initial full check; subsequent generations are rechecks of changed/failed documents. | +| **FINAL** | Label shown on the dashboard for the last generation — only its failures indicate real mismatches. | +| **Task statuses** | `added` (unstarted), `processing` (in-progress), `completed` (no issues), `failed` (document mismatch), `mismatch` (collection metadata mismatch). | + +**Metadata collections:** + +| Collection | Purpose | +|------------|---------| +| `verification_tasks` | Tracks each verification task with a generation number, status, and type (`verify` for documents, `verifyCollection` for metadata). | +| `mismatches` | Records document-level mismatches found during verification. | + **Note**: The `MI_VERIFIER_CONNECTION_STRING` environment variable can be used to pre-configure the connection string. When omitted, it falls back to `MI_CONNECTION_STRING`. See **[CONFIGURATION.md](CONFIGURATION.md)** for details. ## Advanced Configuration diff --git a/migration/mongosync_insights/images/migration_verifier_dashboard.png b/migration/mongosync_insights/images/migration_verifier_dashboard.png index 45ee9086..349d5f79 100644 Binary files a/migration/mongosync_insights/images/migration_verifier_dashboard.png and b/migration/mongosync_insights/images/migration_verifier_dashboard.png differ diff --git a/migration/mongosync_insights/images/mongosync_insights_home.png b/migration/mongosync_insights/images/mongosync_insights_home.png index 862c48c5..906bce02 100644 Binary files a/migration/mongosync_insights/images/mongosync_insights_home.png and b/migration/mongosync_insights/images/mongosync_insights_home.png differ diff --git a/migration/mongosync_insights/images/mongosync_logs_logviewer.png b/migration/mongosync_insights/images/mongosync_logs_logviewer.png new file mode 100644 index 00000000..d7cf0a53 Binary files /dev/null and b/migration/mongosync_insights/images/mongosync_logs_logviewer.png differ diff --git a/migration/mongosync_insights/lib/__init__.py b/migration/mongosync_insights/lib/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/migration/mongosync_insights/app_config.py b/migration/mongosync_insights/lib/app_config.py similarity index 87% rename from migration/mongosync_insights/app_config.py rename to migration/mongosync_insights/lib/app_config.py index 5c73f799..71e545b6 100644 --- a/migration/mongosync_insights/app_config.py +++ b/migration/mongosync_insights/lib/app_config.py @@ -7,6 +7,7 @@ import logging import uuid import time +import tempfile import threading from pathlib import Path from functools import lru_cache @@ -22,7 +23,17 @@ # Application constants APP_NAME = "Mongosync Insights" -APP_VERSION = "0.8.0.18" +APP_VERSION = "0.8.1.14" + +DEVELOPER_CREDITS = { + "copyright": "\u00a9 MongoDB Inc.", + "year": "2025 - 2026", + "team_name": "Migration Factory TS Team", + "contributors": [ + {"name": "Marcio Ribeiro", "role": "Development"}, + {"name": "Krishna Kattumadam", "role": "Development"}, + ], +} # File upload settings MAX_FILE_SIZE = int(os.getenv('MI_MAX_FILE_SIZE', str(10 * 1024 * 1024 * 1024))) # 10GB default @@ -38,6 +49,11 @@ 'application/octet-stream' # Generic binary (often used for compressed files) ] +# Log Viewer settings +LOG_VIEWER_MAX_LINES = int(os.getenv('MI_LOG_VIEWER_MAX_LINES', '2000')) +LOG_STORE_DIR = os.getenv('MI_LOG_STORE_DIR', tempfile.gettempdir()) +LOG_STORE_MAX_AGE_HOURS = int(os.getenv('MI_LOG_STORE_MAX_AGE_HOURS', '24')) + # Compressed file MIME types (subset of ALLOWED_MIME_TYPES) COMPRESSED_MIME_TYPES = { 'application/gzip', 'application/x-gzip', @@ -118,6 +134,9 @@ def classify_file_type(filename: str) -> str: # MongoDB settings INTERNAL_DB_NAME = os.getenv('MI_INTERNAL_DB_NAME', "mongosync_reserved_for_internal_use") +INTERNAL_DB_NAME_NEW = "__mdb_internal_mongosync" +VERIFIER_SRC_NAMESPACE = "__mdb_internal_mongosync_verifier_src" +VERIFIER_DST_NAMESPACE = "__mdb_internal_mongosync_verifier_dst" # UI settings MAX_PARTITIONS_DISPLAY = int(os.getenv('MI_MAX_PARTITIONS_DISPLAY', '10')) @@ -284,6 +303,46 @@ def get_database(connection_string, database_name): client = get_mongo_client(connection_string) return client[database_name] +_resolved_internal_db_cache = {} +_resolved_internal_db_lock = threading.Lock() + +def resolve_internal_db_name(connection_string): + """ + Auto-detect which mongosync internal database name exists on the cluster. + + Checks for the new name first (__mdb_internal_mongosync), then falls back + to the legacy name (mongosync_reserved_for_internal_use). Results are cached + per connection string. The MI_INTERNAL_DB_NAME env var acts as a hard override. + + Args: + connection_string (str): MongoDB connection string + + Returns: + str: The resolved internal database name + """ + if os.getenv('MI_INTERNAL_DB_NAME'): + return INTERNAL_DB_NAME + + with _resolved_internal_db_lock: + if connection_string in _resolved_internal_db_cache: + return _resolved_internal_db_cache[connection_string] + + logger = logging.getLogger(__name__) + try: + client = get_mongo_client(connection_string) + db_names = client.list_database_names() + if INTERNAL_DB_NAME_NEW in db_names: + resolved = INTERNAL_DB_NAME_NEW + else: + resolved = INTERNAL_DB_NAME + with _resolved_internal_db_lock: + _resolved_internal_db_cache[connection_string] = resolved + logger.info(f"Resolved internal DB name: {resolved}") + return resolved + except Exception as e: + logger.warning(f"Could not auto-detect internal DB name, using default: {e}") + return INTERNAL_DB_NAME + def validate_connection(connection_string): """ Validate a MongoDB connection string and test connectivity. diff --git a/migration/mongosync_insights/connection_validator.py b/migration/mongosync_insights/lib/connection_validator.py similarity index 100% rename from migration/mongosync_insights/connection_validator.py rename to migration/mongosync_insights/lib/connection_validator.py diff --git a/migration/mongosync_insights/error_patterns.json b/migration/mongosync_insights/lib/error_patterns.json similarity index 88% rename from migration/mongosync_insights/error_patterns.json rename to migration/mongosync_insights/lib/error_patterns.json index e559dc1a..5dbf9aa4 100644 --- a/migration/mongosync_insights/error_patterns.json +++ b/migration/mongosync_insights/lib/error_patterns.json @@ -141,5 +141,19 @@ "pattern": "refetched db-spec for recreating dst coll for natural scan", "friendly_name": "Restart Collection Copy in natural order", "full_error_message": "refetched db-spec for recreating dst coll for natural scan, collSpec: {COLLECTION DETAILS UUID {}}, isSrcDropped: XXX" + }, + { + "pattern": "Change Event Application failed", + "friendly_name": "CEA failed undefined reason" + }, + { + "pattern": "Failed to apply batch #1 for CRUD event application on the destination. Giving up on batch CRUD event application.", + "friendly_name": "Timeout on destination", + "full_error_message": "Failed to apply batch #1 for CRUD event application on the destination. Giving up on batch CRUD event application." + }, + { + "pattern": "Got a fatal error running Mongosync", + "friendly_name": "Fatal error running Mongosync", + "full_error_message": "Got a fatal error running Mongosync" } ] diff --git a/migration/mongosync_insights/file_decompressor.py b/migration/mongosync_insights/lib/file_decompressor.py similarity index 98% rename from migration/mongosync_insights/file_decompressor.py rename to migration/mongosync_insights/lib/file_decompressor.py index d6d83eda..a2882833 100644 --- a/migration/mongosync_insights/file_decompressor.py +++ b/migration/mongosync_insights/lib/file_decompressor.py @@ -190,7 +190,7 @@ def decompress_file(file_obj: BinaryIO, mime_type: str, filename: str = None) -> Raises: ValueError: If the MIME type is not a supported compressed format """ - from app_config import EXTENSION_TO_COMPRESSION + from .app_config import EXTENSION_TO_COMPRESSION logger.info(f"Decompressing file with MIME type: {mime_type}, filename: {filename}") @@ -244,7 +244,7 @@ def is_compressed_mime_type(mime_type: str) -> bool: Returns: True if the MIME type is a supported compressed format """ - from app_config import COMPRESSED_MIME_TYPES + from .app_config import COMPRESSED_MIME_TYPES return mime_type in COMPRESSED_MIME_TYPES @@ -263,7 +263,7 @@ def decompress_gzip_classified(file_obj: BinaryIO, filename: str) -> Iterator[Tu Yields: Tuples of (decompressed line as bytes, file_type string or None) """ - from app_config import classify_file_type + from .app_config import classify_file_type file_type = classify_file_type(filename) if filename else None logger.info(f"Gzip file classified as: {file_type} (filename: {filename})") @@ -285,7 +285,7 @@ def decompress_bzip2_classified(file_obj: BinaryIO, filename: str) -> Iterator[T Yields: Tuples of (decompressed line as bytes, file_type string or None) """ - from app_config import classify_file_type + from .app_config import classify_file_type file_type = classify_file_type(filename) if filename else None logger.info(f"Bzip2 file classified as: {file_type} (filename: {filename})") @@ -323,7 +323,7 @@ def decompress_zip_classified(file_obj: BinaryIO) -> Iterator[Tuple[bytes, Optio Yields: Tuples of (decompressed line as bytes, file_type string or None) """ - from app_config import classify_file_type + from .app_config import classify_file_type file_obj.seek(0) with zipfile.ZipFile(file_obj, 'r') as zf: @@ -375,7 +375,7 @@ def decompress_tar_classified(file_obj: BinaryIO, compression: str = 'gz') -> It Yields: Tuples of (decompressed line as bytes, file_type string or None) """ - from app_config import classify_file_type + from .app_config import classify_file_type file_obj.seek(0) mode = f'r:{compression}' @@ -435,7 +435,7 @@ def decompress_file_classified(file_obj: BinaryIO, mime_type: str, filename: str Raises: ValueError: If the MIME type is not a supported compressed format """ - from app_config import EXTENSION_TO_COMPRESSION + from .app_config import EXTENSION_TO_COMPRESSION logger.info(f"Decompressing file (classified) with MIME type: {mime_type}, filename: {filename}") diff --git a/migration/mongosync_insights/mongosync_plot_metadata.py b/migration/mongosync_insights/lib/live_migration_metrics.py similarity index 81% rename from migration/mongosync_insights/mongosync_plot_metadata.py rename to migration/mongosync_insights/lib/live_migration_metrics.py index 6189dfd3..7bd5b4cc 100644 --- a/migration/mongosync_insights/mongosync_plot_metadata.py +++ b/migration/mongosync_insights/lib/live_migration_metrics.py @@ -4,12 +4,13 @@ from flask import render_template import json import logging +import re import textwrap import requests from datetime import datetime, timezone from bson import Timestamp from pymongo.errors import PyMongoError -from mongosync_plot_utils import format_byte_size, convert_bytes +from .utils import format_byte_size, convert_bytes def get_phase_timestamp(phase_transitions, phase_name): @@ -30,10 +31,10 @@ def gatherMetrics(connection_string): logger = logging.getLogger(__name__) # Import and use the centralized configuration - from app_config import INTERNAL_DB_NAME, get_database + from .app_config import resolve_internal_db_name, get_database TARGET_MONGO_URI = connection_string - internalDb = INTERNAL_DB_NAME + internalDb = resolve_internal_db_name(connection_string) # Connect to MongoDB cluster using connection pooling try: @@ -344,19 +345,19 @@ def format_namespace_filter(filter_data, filter_type="inclusion"): # Update layout fig.update_layout(height=800, width=1550, autosize=True, title_text="Mongosync Status - Timezone info: UTC", showlegend=False, plot_bgcolor="white") - # Convert the figure to JSON + # Convert the figure to JSON-serializable dict (same shape as jsonify expects) plot_json = json.dumps(fig, cls=PlotlyJSONEncoder) - return plot_json + return json.loads(plot_json) def gatherPartitionsMetrics(connection_string): """Generate progress view with partitions, data copy, phases, and collection progress.""" logger = logging.getLogger(__name__) - from app_config import INTERNAL_DB_NAME, MAX_PARTITIONS_DISPLAY, get_database + from .app_config import resolve_internal_db_name, MAX_PARTITIONS_DISPLAY, get_database TARGET_MONGO_URI = connection_string - internalDb = INTERNAL_DB_NAME + internalDb = resolve_internal_db_name(connection_string) try: internalDbDst = get_database(TARGET_MONGO_URI, internalDb) @@ -522,7 +523,7 @@ def gatherPartitionsMetrics(connection_string): ) plot_json = json.dumps(fig, cls=PlotlyJSONEncoder) - return plot_json + return json.loads(plot_json) def gatherEndpointMetrics(endpoint_url): @@ -531,25 +532,29 @@ def gatherEndpointMetrics(endpoint_url): # Create a figure for displaying endpoint data fig = make_subplots( - rows=4, + rows=5, cols=4, - row_heights=[0.25, 0.25, 0.25, 0.25], + row_heights=[0.20, 0.20, 0.20, 0.20, 0.20], subplot_titles=( - "State", "Lag Time", "Can Commit", "Can Write", - "Info", "Mongosync ID", "Coordinator ID", "Collection Copy", - "Direction Mapping", "Source", "Destination", "Events Applied", - "Embedded Verifier Status", "Verifier Document Count" + "State", "Phase", "Can Commit", "Can Write", + "Lag Time", "Est. CEA Catchup", "Collection Copy", + "Est. Oplog Time Remaining", "Collections Index Progress", "Index Building Progress", + "Embedded Verifier Status", "Verifier Document Count", + "Direction Mapping", "Ping Latency", "ID", "Events Applied" ), specs=[ [{}, {}, {}, {}], - [{}, {}, {}, {"type": "pie"}], - [{"type": "table"}, {"type": "table"}, {"type": "table"}, {}], - [{"type": "table", "colspan": 3}, None, None, {"type": "pie"}] + [{}, {}, {"colspan": 2}, None], + [{}, {}, {"colspan": 2}, None], + [{"type": "table", "colspan": 3}, None, None, {}], + [{"type": "table"}, {"type": "table"}, {"type": "table"}, {}] ], horizontal_spacing=0.08, vertical_spacing=0.12 ) + warnings_list = [] + try: # Make HTTP GET request to the endpoint url = f"http://{endpoint_url}" @@ -560,6 +565,7 @@ def gatherEndpointMetrics(endpoint_url): # Extract progress data progress = data.get("progress", {}) + warnings_list = progress.get("warnings", []) # Helper function to format values for display def format_value(value): @@ -614,14 +620,15 @@ def format_lag_time(seconds): except (ValueError, TypeError): return "No Data" - # Row 1: State, Lag Time, Can Commit, Can Write + # Row 1: State, Phase, Can Commit, Can Write state = progress.get("state", "N/A") fig.add_trace(go.Scatter(x=[0], y=[0], text=[format_value(state)], mode='text', textfont=dict(size=20, color=get_color("state", state))), row=1, col=1) - lagTime = progress.get("lagTimeSeconds") - fig.add_trace(go.Scatter(x=[0], y=[0], text=[format_lag_time(lagTime)], mode='text', - textfont=dict(size=20, color="black")), row=1, col=2) + info = progress.get("info") + infoText = "No Data" if info is None or str(info).strip() == "" else str(info).upper() + fig.add_trace(go.Scatter(x=[0], y=[0], text=[infoText], mode='text', + textfont=dict(size=16, color="black")), row=1, col=2) canCommit = progress.get("canCommit", False) fig.add_trace(go.Scatter(x=[0], y=[0], text=[format_value(canCommit)], mode='text', @@ -631,66 +638,53 @@ def format_lag_time(seconds): fig.add_trace(go.Scatter(x=[0], y=[0], text=[format_value(canWrite)], mode='text', textfont=dict(size=20, color=get_color("canWrite", canWrite))), row=1, col=4) - # Row 2: Info, Mongosync ID, Coordinator ID, Collection Copy (pie chart) - info = progress.get("info") - infoText = "No Data" if info is None or str(info).strip() == "" else str(info).upper() - fig.add_trace(go.Scatter(x=[0], y=[0], text=[infoText], mode='text', - textfont=dict(size=16, color="black")), row=2, col=1) - - mongosyncID = progress.get("mongosyncID", "N/A") - fig.add_trace(go.Scatter(x=[0], y=[0], text=[format_value(mongosyncID)], mode='text', - textfont=dict(size=16, color="black")), row=2, col=2) + # Row 2: Lag Time, Est. CEA Catchup, Collection Copy (horizontal bar) + lagTime = progress.get("lagTimeSeconds") + fig.add_trace(go.Scatter(x=[0], y=[0], text=[format_lag_time(lagTime)], mode='text', + textfont=dict(size=20, color="black")), row=2, col=1) - coordinatorID = progress.get("coordinatorID", "N/A") - coordText = format_value(coordinatorID) if coordinatorID else "No Data" - fig.add_trace(go.Scatter(x=[0], y=[0], text=[coordText], mode='text', - textfont=dict(size=16, color="black")), row=2, col=3) + cea_catchup_seconds = progress.get("estimatedSecondsToCEACatchup") + fig.add_trace(go.Scatter( + x=[0], y=[0], text=[format_lag_time(cea_catchup_seconds)], mode='text', + textfont=dict(size=16, color="black") + ), row=2, col=2) - # Collection Copy (pie chart) - Row 2, Col 4 + # Collection Copy (horizontal bar) - Row 2, Cols 3-4 collectionCopy = progress.get("collectionCopy", {}) if collectionCopy and isinstance(collectionCopy, dict): estimatedTotalBytes = collectionCopy.get("estimatedTotalBytes", 0) or 0 estimatedCopiedBytes = collectionCopy.get("estimatedCopiedBytes", 0) or 0 - remainingBytes = max(0, estimatedTotalBytes - estimatedCopiedBytes) if estimatedTotalBytes > 0: - # Format bytes to human-readable format + copiedPct = (estimatedCopiedBytes / estimatedTotalBytes) * 100 + remainingPct = 100 - copiedPct copiedValue, copiedUnit = format_byte_size(estimatedCopiedBytes) - remainingValue, remainingUnit = format_byte_size(remainingBytes) - - # Create labels with formatted byte sizes - copiedLabel = f"Copied ({copiedValue:.2f} {copiedUnit})" - remainingLabel = f"Remaining ({remainingValue:.2f} {remainingUnit})" + totalValue, totalUnit = format_byte_size(estimatedTotalBytes) - # Create pie chart with copied vs remaining bytes - fig.add_trace(go.Pie( - labels=[copiedLabel, remainingLabel], - values=[estimatedCopiedBytes, remainingBytes], - marker=dict(colors=["green", "lightgray"]), - textinfo="percent", - textposition="outside", - textfont=dict(size=12), - hole=0.3, - showlegend=True - ), row=2, col=4) + fig.add_trace(go.Bar( + y=["Progress"], x=[copiedPct], + name=f"Copied ({copiedValue:.2f} {copiedUnit})", + orientation='h', + marker=dict(color="green"), + text=[f"{copiedPct:.1f}%"], textposition="inside" + ), row=2, col=3) + fig.add_trace(go.Bar( + y=["Progress"], x=[remainingPct], + name=f"Total ({totalValue:.2f} {totalUnit})", + orientation='h', + marker=dict(color="lightgray"), + text=[f"{remainingPct:.1f}%"], textposition="inside" + ), row=2, col=3) else: - fig.add_trace(go.Pie( - labels=["No Data"], - values=[1], - marker=dict(colors=["lightgray"]), - textinfo="label", - textfont=dict(size=14), - showlegend=False - ), row=2, col=4) + fig.add_trace(go.Scatter( + x=[0], y=[0], text=["No Data"], mode='text', + textfont=dict(size=14, color="black") + ), row=2, col=3) else: - fig.add_trace(go.Pie( - labels=["No Data"], - values=[1], - marker=dict(colors=["lightgray"]), - textinfo="label", - textfont=dict(size=14), - showlegend=False - ), row=2, col=4) + fig.add_trace(go.Scatter( + x=[0], y=[0], text=["No Data"], mode='text', + textfont=dict(size=14, color="black") + ), row=2, col=3) # Helper function to create table data from dict def dict_to_table(data): @@ -709,41 +703,92 @@ def dict_to_table(data): values.append(val_str) return keys, values - # Row 3: Direction Mapping, Source, Destination, Events Applied - directionMapping = progress.get("directionMapping") - dm_keys, dm_values = dict_to_table(directionMapping) - fig.add_trace(go.Table( - header=dict(values=["Key", "Value"], font=dict(size=12, color='black')), - cells=dict(values=[dm_keys, dm_values], align=['left'], font=dict(size=10, color='darkblue')), - columnwidth=[0.75, 2.5] + # Row 3: Est. Oplog Time Remaining, Collections Index Progress, Index Building Progress + oplog_time_remaining = progress.get("estimatedOplogTimeRemaining", "") + + def parse_oplog_minutes(value): + if not value or value == "not yet checked": + return None + if value == "more than 72 hours": + return 4320 + if value == "less than 15 minutes": + return 15 + m = re.match(r"(\d+)\s+minutes?", value) + if m: + return int(m.group(1)) + m = re.match(r"(\d+)\s+hours?", value) + if m: + return int(m.group(1)) * 60 + return None + + def get_oplog_color(minutes): + if minutes is None: + return "red" + if minutes > 1440: + return "green" + if minutes >= 360: + return "orange" + return "red" + + oplog_minutes = parse_oplog_minutes(oplog_time_remaining) + oplog_color = get_oplog_color(oplog_minutes) + oplog_display = oplog_time_remaining.capitalize() if oplog_time_remaining else "No Data" + fig.add_trace(go.Scatter( + x=[0], y=[0], text=[oplog_display], mode='text', + textfont=dict(size=16, color=oplog_color) ), row=3, col=1) - source = progress.get("source") - src_keys, src_values = dict_to_table(source) - fig.add_trace(go.Table( - header=dict(values=["Key", "Value"], font=dict(size=12, color='black')), - cells=dict(values=[src_keys, src_values], align=['left'], font=dict(size=10, color='darkblue')), - columnwidth=[0.75, 2.5] - ), row=3, col=2) + index_building = progress.get("indexBuilding", {}) + indexes_built = index_building.get("indexesBuilt", 0) or 0 + total_indexes = index_building.get("totalIndexesToBuild", 0) or 0 + collections_finished = index_building.get("collectionsFinished", 0) or 0 + collections_total = index_building.get("collectionsTotal", 0) or 0 - destination = progress.get("destination") - dst_keys, dst_values = dict_to_table(destination) - fig.add_trace(go.Table( - header=dict(values=["Key", "Value"], font=dict(size=12, color='black')), - cells=dict(values=[dst_keys, dst_values], align=['left'], font=dict(size=10, color='darkblue')), - columnwidth=[0.75, 2.5] - ), row=3, col=3) + remaining_colls = max(0, collections_total - collections_finished) + if collections_total > 0: + fig.add_trace(go.Bar( + y=["Collections"], x=[collections_finished], + name="Finished", orientation='h', + marker=dict(color="blue"), + text=[f"{collections_finished}"], textposition="inside" + ), row=3, col=2) + fig.add_trace(go.Bar( + y=["Collections"], x=[remaining_colls], + name="Remaining", orientation='h', + marker=dict(color="lightgray"), + text=[f"{remaining_colls}"], textposition="inside" + ), row=3, col=2) + else: + fig.add_trace(go.Scatter( + x=[0], y=[0], text=["No Data"], mode='text', + textfont=dict(size=14, color="black") + ), row=3, col=2) - totalEventsApplied = progress.get("totalEventsApplied") - fig.add_trace(go.Scatter(x=[0], y=[0], text=[format_value(totalEventsApplied)], mode='text', - textfont=dict(size=14, color="black")), row=3, col=4) + remaining_indexes = max(0, total_indexes - indexes_built) + if total_indexes > 0: + fig.add_trace(go.Bar( + y=["Indexes"], x=[indexes_built], + name="Built", orientation='h', + marker=dict(color="green"), + text=[f"{indexes_built}"], textposition="inside" + ), row=3, col=3) + fig.add_trace(go.Bar( + y=["Indexes"], x=[remaining_indexes], + name="Remaining", orientation='h', + marker=dict(color="lightgray"), + text=[f"{remaining_indexes}"], textposition="inside" + ), row=3, col=3) + else: + fig.add_trace(go.Scatter( + x=[0], y=[0], text=["No Data"], mode='text', + textfont=dict(size=14, color="black") + ), row=3, col=3) - # Row 4: Verification comparison table (source vs destination) + # Row 4: Embedded Verifier Status, Verifier Document Count verification = progress.get("verification", {}) verif_source = verification.get("source", {}) if verification else {} verif_dest = verification.get("destination", {}) if verification else {} - # Define the fields to compare verif_fields = [ ("phase", "Phase"), ("lagTimeSeconds", "Lag Time Seconds"), @@ -753,23 +798,17 @@ def dict_to_table(data): ("estimatedDocumentCount", "Estimated Document Count") ] - # Build table columns field_names = [] source_values = [] dest_values = [] for field_key, field_label in verif_fields: field_names.append(field_label) - - # Get source value src_val = verif_source.get(field_key) if verif_source else None source_values.append(str(src_val) if src_val is not None else "No Data") - - # Get destination value dst_val = verif_dest.get(field_key) if verif_dest else None dest_values.append(str(dst_val) if dst_val is not None else "No Data") - # Create verification comparison table if verification: fig.add_trace(go.Table( header=dict(values=["Field", "Source", "Destination"], font=dict(size=12, color='black')), @@ -783,36 +822,71 @@ def dict_to_table(data): columnwidth=[1.5, 1, 1] ), row=4, col=1) - # Verifier Document Count pie chart (Verified vs Remaining) src_estimated_docs = verif_source.get("estimatedDocumentCount", 0) or 0 if verif_source else 0 dst_estimated_docs = verif_dest.get("estimatedDocumentCount", 0) or 0 if verif_dest else 0 - - # Verified Documents = src_estimated_docs (documents already verified) - # Remaining Documents = dst_estimated_docs - src_estimated_docs (documents left to verify) verified_docs = dst_estimated_docs remaining_docs = max(0, src_estimated_docs - dst_estimated_docs) if verified_docs > 0 or remaining_docs > 0: - fig.add_trace(go.Pie( - labels=[f"Verified ({verified_docs:,})", f"Remaining ({remaining_docs:,})"], - values=[verified_docs, remaining_docs], - marker=dict(colors=["green", "lightgray"]), - textinfo="percent", - textposition="outside", - textfont=dict(size=12), - hole=0.3, - showlegend=True + fig.add_trace(go.Bar( + y=["Documents"], x=[verified_docs], + name="Verified", orientation='h', + marker=dict(color="green"), + text=[f"{verified_docs:,}"], textposition="inside" + ), row=4, col=4) + fig.add_trace(go.Bar( + y=["Documents"], x=[remaining_docs], + name="Remaining", orientation='h', + marker=dict(color="lightgray"), + text=[f"{remaining_docs:,}"], textposition="inside" ), row=4, col=4) else: - fig.add_trace(go.Pie( - labels=["No Data"], - values=[1], - marker=dict(colors=["lightgray"]), - textinfo="label", - textfont=dict(size=14), - showlegend=False + fig.add_trace(go.Scatter( + x=[0], y=[0], text=["No Data"], mode='text', + textfont=dict(size=14, color="black") ), row=4, col=4) + # Row 5: Direction Mapping, Ping Latency, ID, Events Applied + directionMapping = progress.get("directionMapping") + dm_keys, dm_values = dict_to_table(directionMapping) + fig.add_trace(go.Table( + header=dict(values=["Key", "Value"], font=dict(size=12, color='black')), + cells=dict(values=[dm_keys, dm_values], align=['left'], font=dict(size=10, color='darkblue')), + columnwidth=[0.75, 2.5] + ), row=5, col=1) + + source = progress.get("source") + destination = progress.get("destination") + src_ping = source.get("pingLatencyMs") if source and isinstance(source, dict) else None + dst_ping = destination.get("pingLatencyMs") if destination and isinstance(destination, dict) else None + sd_keys = ["Source", "Destination"] + sd_values = [ + f"{src_ping} ms" if src_ping is not None else "No Data", + f"{dst_ping} ms" if dst_ping is not None else "No Data" + ] + fig.add_trace(go.Table( + header=dict(values=["Key", "Value"], font=dict(size=12, color='black')), + cells=dict(values=[sd_keys, sd_values], align=['left'], font=dict(size=10, color='darkblue')), + columnwidth=[0.75, 2.5] + ), row=5, col=2) + + mongosyncID = progress.get("mongosyncID", "N/A") + coordinatorID = progress.get("coordinatorID", "N/A") + id_keys = ["Mongosync", "Coordinator"] + id_values = [ + format_value(mongosyncID), + format_value(coordinatorID) if coordinatorID else "No Data" + ] + fig.add_trace(go.Table( + header=dict(values=["Key", "Value"], font=dict(size=12, color='black')), + cells=dict(values=[id_keys, id_values], align=['left'], font=dict(size=10, color='darkblue')), + columnwidth=[0.75, 2.5] + ), row=5, col=3) + + totalEventsApplied = progress.get("totalEventsApplied") + fig.add_trace(go.Scatter(x=[0], y=[0], text=[format_value(totalEventsApplied)], mode='text', + textfont=dict(size=14, color="black")), row=5, col=4) + except requests.exceptions.Timeout: logger.error(f"Timeout connecting to endpoint: {endpoint_url}") fig.add_trace(go.Scatter(x=[0], y=[0], text=["TIMEOUT - Could not reach endpoint"], mode='text', @@ -834,7 +908,7 @@ def dict_to_table(data): fig.add_trace(go.Scatter(x=[0], y=[0], text=[f"ERROR: {str(e)[:50]}"], mode='text', textfont=dict(size=16, color="red")), row=1, col=1) - # Hide all axes (4 rows x 4 cols = 16 potential axes) + # Hide all axes for non-chart cells for i in range(1, 17): fig.update_layout(**{ f'xaxis{i}': dict(showgrid=False, zeroline=False, showticklabels=False), @@ -843,16 +917,20 @@ def dict_to_table(data): # Update layout fig.update_layout( - height=800, + height=1000, width=1550, autosize=True, title_text=f"Mongosync Endpoint Data - {endpoint_url}", showlegend=False, - plot_bgcolor="white" + plot_bgcolor="white", + barmode="stack" ) plot_json = json.dumps(fig, cls=PlotlyJSONEncoder) - return plot_json + return { + "plot": json.loads(plot_json), + "warnings": warnings_list, + } def plotMetrics(has_connection_string=True, has_endpoint_url=False): @@ -863,7 +941,7 @@ def plotMetrics(has_connection_string=True, has_endpoint_url=False): they are never passed to the client-side JavaScript. """ # Use the centralized configuration - from app_config import REFRESH_TIME + from .app_config import REFRESH_TIME refreshTime = REFRESH_TIME refreshTimeMs = str(int(refreshTime) * 1000) diff --git a/migration/mongosync_insights/lib/log_store.py b/migration/mongosync_insights/lib/log_store.py new file mode 100644 index 00000000..6e8f7f49 --- /dev/null +++ b/migration/mongosync_insights/lib/log_store.py @@ -0,0 +1,289 @@ +""" +SQLite-backed log document store with FTS5 full-text search. + +Stores raw JSON log lines as documents and provides a MongoDB-like +query API for searching across the full log file. Uses SQLite's +JSON functions for field extraction and FTS5 for full-text search +on the message field. +""" +import json +import logging +import os +import sqlite3 +import time +import glob +from typing import Any, Optional + +logger = logging.getLogger(__name__) + + +class LogStore: + """Document store for mongosync log lines backed by SQLite + FTS5.""" + + BATCH_SIZE = 5000 + + def __init__(self, db_path: str): + self.db_path = db_path + self._conn: Optional[sqlite3.Connection] = None + self._pending: list[tuple] = [] + self._total_inserted = 0 + self._open() + + def _open(self): + self._conn = sqlite3.connect(self.db_path, check_same_thread=False) + self._conn.execute("PRAGMA journal_mode=WAL") + self._conn.execute("PRAGMA busy_timeout=5000") + self._conn.execute("PRAGMA synchronous=OFF") + self._conn.execute("PRAGMA cache_size=-64000") # 64 MB cache + self._conn.execute(""" + CREATE TABLE IF NOT EXISTS log_lines ( + rowid INTEGER PRIMARY KEY, + timestamp TEXT, + level TEXT, + message TEXT, + doc TEXT + ) + """) + self._conn.commit() + + def insert_many(self, documents: list[dict]): + """ + Batch-insert raw JSON documents. + + Each document should be a parsed JSON dict from a log line. + The raw JSON string is stored in the `doc` column. + """ + if not documents: + return + rows = [] + for doc in documents: + rows.append(( + doc.get('time', ''), + doc.get('level', ''), + doc.get('message', ''), + json.dumps(doc, separators=(',', ':')) + )) + self._conn.executemany( + "INSERT INTO log_lines(timestamp, level, message, doc) VALUES (?,?,?,?)", + rows + ) + self._conn.commit() + self._total_inserted += len(rows) + + def insert_line(self, line: str, parsed: Optional[dict] = None): + """ + Buffer a single log line for batched insertion. + + Call flush() after the parsing loop to write remaining buffered rows. + If `parsed` is provided it is used to extract fields; otherwise + the raw line string is stored with empty metadata. + """ + if parsed is not None: + self._pending.append(( + parsed.get('time', ''), + parsed.get('level', ''), + parsed.get('message', ''), + line + )) + else: + self._pending.append(('', '', '', line)) + + if len(self._pending) >= self.BATCH_SIZE: + self._flush_pending() + + def _flush_pending(self): + if not self._pending: + return + self._conn.executemany( + "INSERT INTO log_lines(timestamp, level, message, doc) VALUES (?,?,?,?)", + self._pending + ) + self._conn.commit() + self._total_inserted += len(self._pending) + self._pending.clear() + + def flush(self): + """Flush any remaining buffered rows to the database.""" + self._flush_pending() + + def build_fts_index(self): + """ + Build the FTS5 full-text index on the message column. + + Call this once after all inserts are complete for best performance. + """ + self.flush() + logger.info(f"Building FTS5 index over {self._total_inserted} log lines...") + t0 = time.time() + self._conn.execute(""" + CREATE VIRTUAL TABLE IF NOT EXISTS log_fts + USING fts5(message, content=log_lines, content_rowid=rowid) + """) + self._conn.execute(""" + INSERT INTO log_fts(log_fts) VALUES('rebuild') + """) + self._conn.execute("CREATE INDEX IF NOT EXISTS idx_level ON log_lines(level)") + self._conn.execute("CREATE INDEX IF NOT EXISTS idx_timestamp ON log_lines(timestamp)") + self._conn.commit() + elapsed = time.time() - t0 + logger.info(f"FTS5 index built in {elapsed:.2f}s for {self._total_inserted} documents") + + def find(self, query: Optional[dict] = None, skip: int = 0, limit: int = 50) -> dict: + """ + Query log documents with optional filters. + + Args: + query: MongoDB-style query dict. Supported keys: + - "level": exact match (str) or {"$in": [...]} for multiple levels + - "$text": FTS5 full-text search on message field + - "timestamp_gte": lines at or after this timestamp + - "timestamp_lte": lines at or before this timestamp + skip: number of results to skip (for pagination) + limit: max results to return (capped at 200) + + Returns: + dict with keys: results (list of dicts), total (int), skip, limit + """ + if query is None: + query = {} + limit = min(limit, 200) + + conditions = [] + params: list[Any] = [] + use_fts = False + fts_term = '' + + level_filter = query.get('level') + if level_filter: + if isinstance(level_filter, dict) and '$in' in level_filter: + placeholders = ','.join('?' for _ in level_filter['$in']) + conditions.append(f"l.level IN ({placeholders})") + params.extend(level_filter['$in']) + elif isinstance(level_filter, str) and level_filter: + conditions.append("l.level = ?") + params.append(level_filter) + + text_query = query.get('$text', '').strip() + if text_query: + use_fts = True + fts_term = text_query + + ts_gte = query.get('timestamp_gte') + if ts_gte: + conditions.append("l.timestamp >= ?") + params.append(ts_gte) + + ts_lte = query.get('timestamp_lte') + if ts_lte: + conditions.append("l.timestamp <= ?") + params.append(ts_lte) + + where_clause = (" AND ".join(conditions)) if conditions else "1=1" + + if use_fts: + count_sql = f""" + SELECT COUNT(*) FROM log_lines l + JOIN log_fts f ON l.rowid = f.rowid + WHERE log_fts MATCH ? AND {where_clause} + """ + data_sql = f""" + SELECT l.rowid, l.timestamp, l.level, l.message, l.doc + FROM log_lines l + JOIN log_fts f ON l.rowid = f.rowid + WHERE log_fts MATCH ? AND {where_clause} + ORDER BY l.rowid + LIMIT ? OFFSET ? + """ + count_params = [fts_term] + params + data_params = [fts_term] + params + [limit, skip] + else: + count_sql = f"SELECT COUNT(*) FROM log_lines l WHERE {where_clause}" + data_sql = f""" + SELECT l.rowid, l.timestamp, l.level, l.message, l.doc + FROM log_lines l WHERE {where_clause} + ORDER BY l.rowid + LIMIT ? OFFSET ? + """ + count_params = params + data_params = params + [limit, skip] + + total = self._conn.execute(count_sql, count_params).fetchone()[0] + rows = self._conn.execute(data_sql, data_params).fetchall() + + results = [] + for row in rows: + results.append({ + 'line': row[0], + 'timestamp': row[1], + 'level': row[2], + 'message': row[3], + 'raw': row[4] + }) + + return { + 'results': results, + 'total': total, + 'skip': skip, + 'limit': limit + } + + def count(self, query: Optional[dict] = None) -> int: + """Return the count of matching documents.""" + result = self.find(query, skip=0, limit=1) + return result['total'] + + @property + def total_documents(self) -> int: + """Total number of documents in the store.""" + row = self._conn.execute("SELECT COUNT(*) FROM log_lines").fetchone() + return row[0] if row else 0 + + def close(self): + """Close the database connection.""" + if self._conn: + self.flush() + self._conn.close() + self._conn = None + + def delete(self): + """Close connection and delete the database file.""" + self.close() + try: + if os.path.exists(self.db_path): + os.remove(self.db_path) + logger.info(f"Deleted log store: {self.db_path}") + wal = self.db_path + '-wal' + shm = self.db_path + '-shm' + for f in (wal, shm): + if os.path.exists(f): + os.remove(f) + except OSError as e: + logger.warning(f"Failed to delete log store {self.db_path}: {e}") + + @staticmethod + def cleanup_old_stores(store_dir: str, max_age_hours: int = 24): + """ + Delete log store DB files older than max_age_hours. + + Scans the given directory for files matching mi_logstore_*.db, + removing those that exceed the age threshold. Snapshot JSON cleanup + is handled separately by cleanup_old_snapshots() in snapshot_store. + """ + cutoff = time.time() - (max_age_hours * 3600) + removed = 0 + pattern = os.path.join(store_dir, 'mi_logstore_*.db') + + for filepath in glob.glob(pattern): + try: + if os.path.getmtime(filepath) < cutoff: + os.remove(filepath) + for suffix in ('-wal', '-shm'): + extra = filepath + suffix + if os.path.exists(extra): + os.remove(extra) + removed += 1 + except OSError as e: + logger.warning(f"Failed to clean up {filepath}: {e}") + + if removed: + logger.info(f"Cleaned up {removed} expired log store file(s) from {store_dir}") diff --git a/migration/mongosync_insights/lib/log_store_registry.py b/migration/mongosync_insights/lib/log_store_registry.py new file mode 100644 index 00000000..dc34384e --- /dev/null +++ b/migration/mongosync_insights/lib/log_store_registry.py @@ -0,0 +1,127 @@ +""" +Global registry mapping log store IDs to their on-disk SQLite paths. + +Thread-safe singleton that allows the search endpoint to locate the +correct LogStore database for a given store_id returned to the client. +""" +import logging +import os +import threading +import time +from typing import Optional + +from .log_store import LogStore + +logger = logging.getLogger(__name__) + + +class LogStoreRegistry: + """Thread-safe registry of active LogStore instances.""" + + def __init__(self, default_ttl: int = 86400): + """ + Args: + default_ttl: seconds before an entry is considered expired (default 24h) + """ + self._entries: dict[str, dict] = {} + self._lock = threading.Lock() + self._ttl = default_ttl + + def register(self, store_id: str, db_path: str): + """Register a store_id -> db_path mapping.""" + with self._lock: + self._entries[store_id] = { + 'db_path': db_path, + 'created_at': time.time(), + 'store': None, + } + logger.debug(f"Registered log store {store_id[:8]}... -> {db_path}") + + def get_path(self, store_id: str) -> Optional[str]: + """Get the db_path for a store_id, or None if not found/expired.""" + if not store_id: + return None + with self._lock: + entry = self._entries.get(store_id) + if not entry: + return None + if time.time() - entry['created_at'] > self._ttl: + self._remove_entry(store_id) + return None + return entry['db_path'] + + def open_store(self, store_id: str) -> Optional[LogStore]: + """ + Return a cached LogStore connection for the given store_id. + + Returns None if the store_id is not registered or the DB file + no longer exists. The returned LogStore is owned by the registry; + callers must NOT close it. + """ + with self._lock: + entry = self._entries.get(store_id) + if not entry: + return None + if time.time() - entry['created_at'] > self._ttl: + self._remove_entry(store_id) + return None + db_path = entry['db_path'] + if not os.path.exists(db_path): + return None + if entry['store'] is None: + entry['store'] = LogStore(db_path) + return entry['store'] + + def remove(self, store_id: str): + """Remove a store entry and delete the DB file from disk.""" + with self._lock: + self._remove_entry(store_id) + + def _remove_entry(self, store_id: str): + """Internal: remove entry and delete DB file. Caller must hold lock.""" + entry = self._entries.pop(store_id, None) + if entry: + cached_store = entry.get('store') + if cached_store is not None: + try: + cached_store.close() + except Exception: + pass + db_path = entry['db_path'] + try: + if os.path.exists(db_path): + os.remove(db_path) + for suffix in ('-wal', '-shm'): + extra = db_path + suffix + if os.path.exists(extra): + os.remove(extra) + logger.debug(f"Removed log store {store_id[:8]}... ({db_path})") + except OSError as e: + logger.warning(f"Failed to delete log store file {db_path}: {e}") + + def cleanup_expired(self): + """Remove all entries that have exceeded their TTL.""" + now = time.time() + with self._lock: + expired = [ + sid for sid, entry in self._entries.items() + if now - entry['created_at'] > self._ttl + ] + for sid in expired: + self._remove_entry(sid) + if expired: + logger.info(f"Cleaned up {len(expired)} expired log store(s)") + + def remove_all(self): + """Remove all registered stores (used during shutdown).""" + with self._lock: + for sid in list(self._entries.keys()): + self._remove_entry(sid) + + def count(self) -> int: + """Number of currently registered stores.""" + with self._lock: + return len(self._entries) + + +log_store_registry = LogStoreRegistry() diff --git a/migration/mongosync_insights/mongosync_plot_logs.py b/migration/mongosync_insights/lib/logs_metrics.py similarity index 78% rename from migration/mongosync_insights/mongosync_plot_logs.py rename to migration/mongosync_insights/lib/logs_metrics.py index 506d44ff..a3113bb4 100644 --- a/migration/mongosync_insights/mongosync_plot_logs.py +++ b/migration/mongosync_insights/lib/logs_metrics.py @@ -4,6 +4,8 @@ from tqdm import tqdm from flask import request, render_template import json +import uuid as uuid_mod +from collections import deque from datetime import datetime, timezone from dateutil import parser import re @@ -11,10 +13,17 @@ import os import mimetypes from werkzeug.utils import secure_filename -from mongosync_plot_utils import format_byte_size, convert_bytes -from app_config import MAX_FILE_SIZE, ALLOWED_EXTENSIONS, ALLOWED_MIME_TYPES, load_error_patterns, classify_file_type -from file_decompressor import decompress_file_classified, is_compressed_mime_type -from mongosync_plot_prometheus_metrics import MetricsCollector, create_metrics_plots +from .utils import format_byte_size, convert_bytes +from .app_config import ( + MAX_FILE_SIZE, ALLOWED_EXTENSIONS, ALLOWED_MIME_TYPES, + load_error_patterns, classify_file_type, + LOG_VIEWER_MAX_LINES, LOG_STORE_DIR, +) +from .file_decompressor import decompress_file_classified, is_compressed_mime_type +from .otel_metrics import MetricsCollector, create_metrics_plots +from .log_store import LogStore +from .log_store_registry import log_store_registry +from .snapshot_store import save_snapshot def detect_mime_type(file_sample: bytes, filename: str) -> str: @@ -22,7 +31,6 @@ def detect_mime_type(file_sample: bytes, filename: str) -> str: Detect MIME type using magic bytes and file extension. Pure-Python replacement for python-magic — no system libmagic needed. """ - # Check magic bytes from the file header if file_sample[:2] == b'\x1f\x8b': return 'application/gzip' if file_sample[:4] == b'PK\x03\x04': @@ -32,19 +40,16 @@ def detect_mime_type(file_sample: bytes, filename: str) -> str: if len(file_sample) >= 262 and file_sample[257:262] == b'ustar': return 'application/x-tar' - # Fall back to extension-based detection mime_type, _ = mimetypes.guess_type(filename) if mime_type: return mime_type - # If content looks like text, report it as text/plain try: file_sample.decode('utf-8') return 'text/plain' except UnicodeDecodeError: return 'application/octet-stream' - def upload_file(): # Use the centralized logging configuration logger = logging.getLogger(__name__) @@ -112,7 +117,7 @@ def upload_file(): logger.info(f"File validation passed: {filename} ({file_size} bytes, {file_ext}, MIME: {file_mime_type})") # Optimized single-pass log parsing with streaming approach - logging.info("Starting optimized log parsing - single pass through file") + logger.info("Starting optimized log parsing - single pass through file") # Pre-compile all regex patterns once patterns = { @@ -160,10 +165,18 @@ def upload_file(): partition_multi_created = [] partition_sampling_info = [] partition_persisted_after_sampling = [] + verifier_dst_lag_items = [] + verifier_src_lag_items = [] # Initialize metrics collector for prometheus metrics metrics_collector = MetricsCollector() + # Initialize log viewer: tail buffer + SQLite store for full-text search + raw_log_tail = deque(maxlen=LOG_VIEWER_MAX_LINES) + store_id = str(uuid_mod.uuid4()) + db_path = os.path.join(LOG_STORE_DIR, f'mi_logstore_{store_id}.db') + log_store = LogStore(db_path) + # Single pass through the file with streaming line_count = 0 logs_line_count = 0 @@ -182,6 +195,8 @@ def upload_file(): else: # For non-compressed files, classify by filename file_type = classify_file_type(filename) + if file_type is None: + file_type = 'logs' logger.info(f"Non-compressed file classified as: {file_type}") file_iterator = file use_classified = False @@ -224,6 +239,10 @@ def upload_file(): json_obj = json.loads(line) message = json_obj.get('message', '') + # Collect for log viewer: tail buffer + SQLite store + raw_log_tail.append(line) + log_store.insert_line(line, parsed=json_obj) + # Apply all filters to the same parsed object if patterns['replication_progress'].search(message): data.append(json_obj) @@ -282,6 +301,12 @@ def upload_file(): if patterns['partition_persisted_after_sampling'].search(message): partition_persisted_after_sampling.append(json_obj) + if json_obj.get('verifierDstLagTimeSeconds') is not None and 'time' in json_obj: + verifier_dst_lag_items.append(json_obj) + + if json_obj.get('verifierSrcLagTimeSeconds') is not None and 'time' in json_obj: + verifier_src_lag_items.append(json_obj) + # Check for common error patterns for ep in error_patterns: if ep['pattern'].search(message): @@ -297,24 +322,46 @@ def upload_file(): except json.JSONDecodeError as e: invalid_json_count += 1 if invalid_json_count <= 5: # Log first 5 errors to avoid spam - logging.warning(f"Invalid JSON on line {line_count}: {e}") + logger.warning(f"Invalid JSON on line {line_count}: {e}") # Only treat as fatal error if this is the first error AND we haven't processed any valid lines if invalid_json_count == 1 and logs_line_count == 0 and metrics_line_count == 0: - logging.error(f"File appears to contain invalid JSON. First error on line {line_count}: {e}") + logger.error(f"File appears to contain invalid JSON. First error on line {line_count}: {e}") return render_template('error.html', error_title="Invalid File Format", error_message=f"The uploaded file does not contain valid JSON format. Error on line {line_count}: {str(e)}. Please ensure you're uploading a valid mongosync log file in NDJSON format.") - logging.info(f"Processed {line_count} total lines ({logs_line_count} logs, {metrics_line_count} metrics), found {invalid_json_count} invalid JSON lines") - logging.info(f"Found: {len(data)} replication progress, {len(version_info_list)} version info, " + # Finalize log store: flush remaining buffered rows and build FTS index + log_store.flush() + if log_store.total_documents > 0: + log_store.build_fts_index() + log_store_registry.register(store_id, db_path) + logger.info(f"Log store ready: {log_store.total_documents} documents, store_id={store_id[:8]}...") + else: + log_store.delete() + store_id = '' + + logger.info(f"Processed {line_count} total lines ({logs_line_count} logs, {metrics_line_count} metrics), found {invalid_json_count} invalid JSON lines") + logger.info(f"Found: {len(data)} replication progress, {len(version_info_list)} version info, " f"{len(mongosync_ops_stats)} operation stats, {len(mongosync_sent_response)} sent responses, " f"{len(phase_transitions_json)} phase transitions, {len(mongosync_opts_list)} options, " f"{len(mongosync_hiddenflags)} hidden flags, {len(mongosync_crud_rate)} CRUD rate entries, " f"{len(mongosync_partition_progress)} partition progress entries, " f"{len(natural_order_collections)} natural order collections, " f"{len(matched_errors)} common errors") - logging.info(f"Metrics collector: {metrics_collector.metrics_count} metric points from {metrics_collector.line_count} lines") + logger.info(f"Metrics collector: {metrics_collector.metrics_count} metric points from {metrics_collector.line_count} lines") + has_any_log_data = (len(data) > 0 or len(version_info_list) > 0 or len(mongosync_ops_stats) > 0 or + len(mongosync_sent_response) > 0 or len(phase_transitions_json) > 0 or + len(mongosync_partition_progress) > 0 or len(mongosync_crud_rate) > 0) + has_any_metrics_data = metrics_collector.metrics_count > 0 + if not has_any_log_data and not has_any_metrics_data: + logger.warning(f"No recognizable mongosync data found in {filename} ({line_count} lines processed)") + return render_template('error.html', + error_title="No Mongosync Data Found", + error_message=f"The file '{filename}' was processed ({line_count:,} lines) but no recognizable " + f"mongosync log entries or metrics were found. Please ensure you are uploading a " + f"valid mongosync log file (NDJSON format with standard mongosync log messages).") + # Sort log data by timestamp to ensure correct chronological plot ordering # (archives may contain rotated log files in non-chronological order) data.sort(key=lambda x: x.get('time', '')) @@ -322,6 +369,8 @@ def upload_file(): mongosync_crud_rate.sort(key=lambda x: x.get('time', '')) mongosync_partition_progress.sort(key=lambda x: x.get('time', '')) mongosync_sent_response.sort(key=lambda x: x.get('time', '')) + verifier_dst_lag_items.sort(key=lambda x: x.get('time', '')) + verifier_src_lag_items.sort(key=lambda x: x.get('time', '')) # Aggregate partition initialization data per collection partition_init_data = [] @@ -408,7 +457,7 @@ def upload_file(): 'init_ended': ended[:26] if ended else '', 'duration_sec': duration_sec, }) - logging.info(f"Aggregated partition init data for {len(partition_init_data)} collections") + logger.info(f"Aggregated partition init data for {len(partition_init_data)} collections") # Build partition init progress time series (in-progress and completed per collection over time) partition_init_progress_times = [] @@ -442,11 +491,9 @@ def upload_file(): partition_init_progress_times.append(ts) partition_init_progress_in_progress.append(in_prog) partition_init_progress_completed.append(done) - logging.info(f"Built partition init progress time series with {len(init_events)} events") + logger.info(f"Built partition init progress time series with {len(init_events)} events") - # The 'body' field is also a JSON string, so parse that as well - #mongosync_sent_response_body = json.loads(mongosync_sent_response.get('body')) - mongosync_sent_response_body = None + mongosync_sent_response_body = None for response in mongosync_sent_response: try: parsed_body = json.loads(response['body']) @@ -455,7 +502,7 @@ def upload_file(): mongosync_sent_response_body = parsed_body except (json.JSONDecodeError, TypeError): mongosync_sent_response_body = None # If parse fails, use None - logging.warning(f"No message 'sent response' found in the logs") + logger.warning(f"No message 'sent response' found in the logs") # Create a string with all the version information if version_info_list and isinstance(version_info_list[0], dict): @@ -465,17 +512,17 @@ def upload_file(): version_text = f"MongoSync Version: {version}, OS: {os_name}, Arch: {arch}" else: version_text = f"MongoSync Version is not available" - logging.error(version_text) + logger.error(version_text) - logging.info(f"Extracting data") + logger.info(f"Extracting data") # Log if options data is empty if not mongosync_hiddenflags: - logging.info("mongosync_hiddenflags is empty") + logger.info("mongosync_hiddenflags is empty") if not mongosync_opts_list: - logging.info("mongosync_opts_list is empty") + logger.info("mongosync_opts_list is empty") #Getting the Timezone try: @@ -607,6 +654,12 @@ def _parse_oplog_time_remaining_minutes(value): eventRatePerSecond.append(float(rate)) eventRatePerSecond_times.append(datetime.strptime(item['time'][:26], "%Y-%m-%dT%H:%M:%S.%f")) + dst_lag_times = [datetime.strptime(item['time'][:26], "%Y-%m-%dT%H:%M:%S.%f") for item in verifier_dst_lag_items if 'time' in item] + verifierDstLagTimeSeconds = [item['verifierDstLagTimeSeconds'] for item in verifier_dst_lag_items if 'verifierDstLagTimeSeconds' in item] + + src_lag_times = [datetime.strptime(item['time'][:26], "%Y-%m-%dT%H:%M:%S.%f") for item in verifier_src_lag_items if 'time' in item] + verifierSrcLagTimeSeconds = [item['verifierSrcLagTimeSeconds'] for item in verifier_src_lag_items if 'verifierSrcLagTimeSeconds' in item] + # Calculate global date range from all time sources for X-axis synchronization all_times = [] if times: @@ -619,6 +672,10 @@ def _parse_oplog_time_remaining_minutes(value): all_times.extend(estimatedCopiedBytes_times) if index_built_times: all_times.extend(index_built_times) + if dst_lag_times: + all_times.extend(dst_lag_times) + if src_lag_times: + all_times.extend(src_lag_times) if all_times: global_min_date = min(all_times) @@ -642,11 +699,11 @@ def _parse_oplog_time_remaining_minutes(value): # Try get Phase Transitions from the sent response body if it is Live Migrate phase_transitions = mongosync_sent_response_body['progress']['atlasLiveMigrateMetrics']['PhaseTransitions'] except KeyError as e: - logging.error(f"Key not found: {e}") + logger.error(f"Key not found: {e}") phase_transitions = [] else: - logging.warning(f"Key 'progress' not found in mongosync_sent_response_body") + logger.warning(f"Key 'progress' not found in mongosync_sent_response_body") # If phase_transitions is not empty, plot the phase transitions as it is Live Migrate if phase_transitions: @@ -677,48 +734,49 @@ def _parse_oplog_time_remaining_minutes(value): estimated_copied_bytes = convert_bytes(estimated_copied_bytes, estimated_total_bytes_unit) estimatedCopiedBytes_converted = [convert_bytes(b, estimated_total_bytes_unit) for b in estimatedCopiedBytes_series] - logging.info(f"Plotting") + logger.info(f"Plotting") # Create a subplot for the scatter plots (tables are now in a separate tab) - fig = make_subplots(rows=12, cols=2, subplot_titles=("Mongosync Phases", "Mongosync Phases Table", + fig = make_subplots(rows=13, cols=2, subplot_titles=("Mongosync Phases", "Mongosync Phases Table", + "Lag Time (seconds)", "Estimated Source Oplog Time Remaining (minutes)", + "Ping Latency (ms)", "Average Source CRUD Event Rate (Events/sec)", "Partition Init Progress", "Partition Init Summary", "Data Copied (" + estimated_total_bytes_unit + ")", "Estimated Total and Copied " + estimated_total_bytes_unit, "Partitions Copied", "Total and Copied Partitions", - "Lag Time (seconds)", "Estimated Source Oplog Time Remaining (minutes)", - "Change Events Applied", "Events Rate per Second", - "Index Built", "Total and Index Built", - "Ping Latency (ms)", "Average Source CRUD Event Rate (Events/sec)", "Collection Copy - Avg and Max Read time (ms)", "Collection Copy Source Reads", "Collection Copy - Avg and Max Write time (ms)", "Collection Copy Destination Writes", + "Change Events Applied", "Events Rate per Second", "CEA Source - Avg and Max Read time (ms)", "CEA Source Reads", - "CEA Destination - Avg and Max Write time (ms)", "CEA Destination Writes"), - specs=[ [{}, {"type": "table"}], #Mongosync Phases and Phases Table - [{}, {"type": "table"}], #Partition Init Progress and Summary - [{}, {}], #Data Copied Over Time + Estimated Total and Copied - [{}, {}], #Partitions Copied and Completion % - [{}, {}], #Lag Time and Estimated Source Oplog Time Remaining - [{}, {}], #Change Events Applied and Events Rate per Second - [{}, {}], #Index Built and Total and Index Built - [{}, {}], #Ping Latency and CRUD Event Rate - [{}, {}], #Collection Copy Source - [{}, {}], #Collection Copy Destination - [{}, {}], #CEA Source - [{}, {}] ]) #CEA Destination + "CEA Destination - Avg and Max Write time (ms)", "CEA Destination Writes", + "Index Built", "Total and Index Built", + "Source Verifier Lag Time (seconds)", "Destination Verifier Lag Time (seconds)"), + specs=[ [{}, {"type": "table"}], #Row 1: Mongosync Phases and Phases Table + [{}, {}], #Row 2: Lag Time and Estimated Source Oplog Time Remaining + [{}, {}], #Row 3: Ping Latency and CRUD Event Rate + [{}, {"type": "table"}], #Row 4: Partition Init Progress and Summary + [{}, {}], #Row 5: Data Copied Over Time + Estimated Total and Copied + [{}, {}], #Row 6: Partitions Copied and Completion % + [{}, {}], #Row 7: Collection Copy Source + [{}, {}], #Row 8: Collection Copy Destination + [{}, {}], #Row 9: Change Events Applied and Events Rate per Second + [{}, {}], #Row 10: CEA Source + [{}, {}], #Row 11: CEA Destination + [{}, {}], #Row 12: Index Built and Total and Index Built + [{}, {}] ]) #Row 13: Verifier Lag # Add traces - # Mongosync Phases + # Row 1: Mongosync Phases if phase_transitions: fig.add_trace(go.Scatter(x=ts_t_list_formatted, y=phase_list, mode='markers+text',marker=dict(color='green')), row=1, col=1) fig.update_yaxes(showticklabels=False, row=1, col=1) else: fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Mongosync Phases',textfont=dict(size=30, color="black")), row=1, col=1) - fig.update_yaxes(range=[-1, 1], row=1, col=1) # Center the text vertically - fig.update_xaxes(range=[-1, 1], row=1, col=1) # Also center horizontally + fig.update_yaxes(range=[-1, 1], row=1, col=1) + fig.update_xaxes(range=[-1, 1], row=1, col=1) - # Mongosync Phases Table + # Row 1: Mongosync Phases Table if phase_transitions: - # Pair and sort by datetime phase_table_data = sorted(zip(ts_t_list_formatted, phase_list), key=lambda x: x[0]) table_dates = [row[0] for row in phase_table_data] table_phases = [row[1] for row in phase_table_data] @@ -732,32 +790,65 @@ def _parse_oplog_time_remaining_minutes(value): cells=dict(values=[[], []]) ), row=1, col=2) - # Partition Init Progress (Row 2, Col 1) - collections initializing vs completed over time + # Row 2: Lag Time + if lagTimeSeconds: + fig.add_trace(go.Scattergl(x=times, y=lagTimeSeconds, mode='lines', name='Seconds', legendgroup="groupEventsAndLags"), row=2, col=1) + else: + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Lag Time',textfont=dict(size=30, color="black")), row=2, col=1) + fig.update_yaxes(range=[-1, 1], row=2, col=1) + fig.update_xaxes(range=[-1, 1], row=2, col=1) + + # Row 2: Estimated Source Oplog Time Remaining (minutes) + if oplog_remaining_minutes: + fig.add_trace(go.Scattergl(x=oplog_remaining_times, y=oplog_remaining_minutes, mode='lines', name='Minutes Remaining', legendgroup="groupEventsAndLags"), row=2, col=2) + else: + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Oplog Time Remaining',textfont=dict(size=30, color="black")), row=2, col=2) + fig.update_yaxes(range=[-1, 1], row=2, col=2) + fig.update_xaxes(range=[-1, 1], row=2, col=2) + + # Row 3: Ping Latency + if sourcePingLatencyMs or destinationPingLatencyMs: + fig.add_trace(go.Scattergl(x=times, y=sourcePingLatencyMs, mode='lines', name='Source Ping (ms)', legendgroup="groupPingLatency"), row=3, col=1) + fig.add_trace(go.Scattergl(x=times, y=destinationPingLatencyMs, mode='lines', name='Destination Ping (ms)', legendgroup="groupPingLatency"), row=3, col=1) + else: + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Ping Latency', textfont=dict(size=30, color="black")), row=3, col=1) + fig.update_yaxes(range=[-1, 1], row=3, col=1) + fig.update_xaxes(range=[-1, 1], row=3, col=1) + + # Row 3: Average Source CRUD Event Rate + if srcCRUDEventsPerSec: + fig.add_trace(go.Scattergl(x=crud_rate_times, y=srcCRUDEventsPerSec, mode='lines', name='Events/sec', legendgroup="groupCRUDRate"), row=3, col=2) + else: + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='CRUD Event Rate', textfont=dict(size=30, color="black")), row=3, col=2) + fig.update_yaxes(range=[-1, 1], row=3, col=2) + fig.update_xaxes(range=[-1, 1], row=3, col=2) + + # Row 4: Partition Init Progress - collections initializing vs completed over time if partition_init_progress_times: total_collections = len(partition_init_data) if partition_init_data else 0 fig.add_trace(go.Scattergl( x=partition_init_progress_times, y=partition_init_progress_in_progress, mode='lines', name='In Progress', line=dict(color='#2196F3'), legendgroup="groupPartitionInitProgress" - ), row=2, col=1) + ), row=4, col=1) fig.add_trace(go.Scattergl( x=partition_init_progress_times, y=partition_init_progress_completed, mode='lines', name='Completed', line=dict(color='#4CAF50'), legendgroup="groupPartitionInitProgress" - ), row=2, col=1) + ), row=4, col=1) if total_collections > 0: fig.add_trace(go.Scattergl( x=[partition_init_progress_times[0], partition_init_progress_times[-1]], y=[total_collections, total_collections], mode='lines', name='Total Collections', line=dict(color='gray', dash='dash'), legendgroup="groupPartitionInitProgress" - ), row=2, col=1) + ), row=4, col=1) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Partition Init Progress', textfont=dict(size=30, color="black")), row=2, col=1) - fig.update_yaxes(range=[-1, 1], row=2, col=1) - fig.update_xaxes(range=[-1, 1], row=2, col=1) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Partition Init Progress', textfont=dict(size=30, color="black")), row=4, col=1) + fig.update_yaxes(range=[-1, 1], row=4, col=1) + fig.update_xaxes(range=[-1, 1], row=4, col=1) - # Partition Init Summary Table (Row 2, Col 2) + # Row 4: Partition Init Summary Table if partition_init_data: fig.add_trace(go.Table( header=dict(values=["Collection", "Type", "Partitions", "Doc Count", "Duration (s)"]), @@ -768,198 +859,209 @@ def _parse_oplog_time_remaining_minutes(value): [f"{d['doc_count']:,}" if d['doc_count'] else 'N/A' for d in partition_init_data], [d['duration_sec'] if d['duration_sec'] is not None else 'N/A' for d in partition_init_data], ]) - ), row=2, col=2) + ), row=4, col=2) else: fig.add_trace(go.Table( header=dict(values=["Collection", "Type", "Partitions", "Doc Count", "Duration (s)"]), cells=dict(values=[[], [], [], [], []]) - ), row=2, col=2) + ), row=4, col=2) - # Data Copied Over Time + # Row 5: Data Copied Over Time if estimatedCopiedBytes_converted: - fig.add_trace(go.Scattergl(x=estimatedCopiedBytes_times, y=estimatedCopiedBytes_converted, mode='lines', name='Copied ' + estimated_total_bytes_unit, legendgroup="groupTotalCopied"), row=3, col=1) + fig.add_trace(go.Scattergl(x=estimatedCopiedBytes_times, y=estimatedCopiedBytes_converted, mode='lines', name='Copied ' + estimated_total_bytes_unit, legendgroup="groupTotalCopied"), row=5, col=1) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Data Copied Over Time',textfont=dict(size=30, color="black")), row=3, col=1) - fig.update_yaxes(range=[-1, 1], row=3, col=1) # Center the text vertically - fig.update_xaxes(range=[-1, 1], row=3, col=1) # Also center horizontally + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Data Copied Over Time',textfont=dict(size=30, color="black")), row=5, col=1) + fig.update_yaxes(range=[-1, 1], row=5, col=1) + fig.update_xaxes(range=[-1, 1], row=5, col=1) - # Estimated Total and Copied + # Row 5: Estimated Total and Copied if estimated_total_bytes > 0 or estimated_copied_bytes > 0: - fig.add_trace( go.Bar( name='Estimated ' + estimated_total_bytes_unit + ' to be Copied', x=[estimated_total_bytes_unit], y=[estimated_total_bytes], legendgroup="groupTotalCopied" ), row=3, col=2) - fig.add_trace( go.Bar( name='Estimated Copied ' + estimated_total_bytes_unit, x=[estimated_total_bytes_unit], y=[estimated_copied_bytes], legendgroup="groupTotalCopied"), row=3, col=2) + fig.add_trace( go.Bar( name='Estimated ' + estimated_total_bytes_unit + ' to be Copied', x=[estimated_total_bytes_unit], y=[estimated_total_bytes], legendgroup="groupTotalCopied" ), row=5, col=2) + fig.add_trace( go.Bar( name='Estimated Copied ' + estimated_total_bytes_unit, x=[estimated_total_bytes_unit], y=[estimated_copied_bytes], legendgroup="groupTotalCopied"), row=5, col=2) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Estimated Total and Copied',textfont=dict(size=30, color="black")), row=3, col=2) - fig.update_yaxes(range=[-1, 1], row=3, col=2) # Center the text vertically - fig.update_xaxes(range=[-1, 1], row=3, col=2) # Also center horizontally - - # Partitions Copied Over Time - if partition_times: - fig.add_trace(go.Scattergl(x=partition_times, y=partitions_copied, mode='lines', name='Partitions Copied', legendgroup="groupPartitions"), row=4, col=1) - fig.add_trace(go.Scattergl(x=partition_times, y=partitions_total, mode='lines', name='Total Partitions', legendgroup="groupPartitions"), row=4, col=1) - else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Partitions Copied', textfont=dict(size=30, color="black")), row=4, col=1) - fig.update_yaxes(range=[-1, 1], row=4, col=1) # Center the text vertically - fig.update_xaxes(range=[-1, 1], row=4, col=1) # Also center horizontally - - # Total and Copied Partitions - if partition_times: - last_copied = partitions_copied[-1] - last_total = partitions_total[-1] - fig.add_trace(go.Bar(name='Total Partitions', x=['Partitions'], y=[last_total], legendgroup="groupPartitions"), row=4, col=2) - fig.add_trace(go.Bar(name='Copied Partitions', x=['Partitions'], y=[last_copied], legendgroup="groupPartitions"), row=4, col=2) - else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Total and Copied Partitions', textfont=dict(size=30, color="black")), row=4, col=2) - fig.update_yaxes(range=[-1, 1], row=4, col=2) - fig.update_xaxes(range=[-1, 1], row=4, col=2) - - # Lag Time - if lagTimeSeconds: - fig.add_trace(go.Scattergl(x=times, y=lagTimeSeconds, mode='lines', name='Seconds', legendgroup="groupEventsAndLags"), row=5, col=1) - else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Lag Time',textfont=dict(size=30, color="black")), row=5, col=1) - fig.update_yaxes(range=[-1, 1], row=5, col=1) # Center the text vertically - fig.update_xaxes(range=[-1, 1], row=5, col=1) # Also center horizontally - #fig.update_yaxes(title_text="Lag Time (seconds)", row=5, col=1) - - # Estimated Source Oplog Time Remaining (minutes) - if oplog_remaining_minutes: - fig.add_trace(go.Scattergl(x=oplog_remaining_times, y=oplog_remaining_minutes, mode='lines', name='Minutes Remaining', legendgroup="groupEventsAndLags"), row=5, col=2) - else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Oplog Time Remaining',textfont=dict(size=30, color="black")), row=5, col=2) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Estimated Total and Copied',textfont=dict(size=30, color="black")), row=5, col=2) fig.update_yaxes(range=[-1, 1], row=5, col=2) fig.update_xaxes(range=[-1, 1], row=5, col=2) - # Total Events Applied - if totalEventsApplied: - fig.add_trace(go.Scattergl(x=times, y=totalEventsApplied, mode='lines', name='Events', legendgroup="groupEventsAndLags"), row=6, col=1) + # Row 6: Partitions Copied Over Time + if partition_times: + fig.add_trace(go.Scattergl(x=partition_times, y=partitions_copied, mode='lines', name='Partitions Copied', legendgroup="groupPartitions"), row=6, col=1) + fig.add_trace(go.Scattergl(x=partition_times, y=partitions_total, mode='lines', name='Total Partitions', legendgroup="groupPartitions"), row=6, col=1) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Change Events Applied',textfont=dict(size=30, color="black")), row=6, col=1) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Partitions Copied', textfont=dict(size=30, color="black")), row=6, col=1) fig.update_yaxes(range=[-1, 1], row=6, col=1) fig.update_xaxes(range=[-1, 1], row=6, col=1) - # Events Rate per Second - if eventRatePerSecond: - fig.add_trace(go.Scattergl(x=eventRatePerSecond_times, y=eventRatePerSecond, mode='lines', name='Events/sec', legendgroup="groupEventsAndLags"), row=6, col=2) + # Row 6: Total and Copied Partitions + if partition_times: + last_copied = partitions_copied[-1] + last_total = partitions_total[-1] + fig.add_trace(go.Bar(name='Total Partitions', x=['Partitions'], y=[last_total], legendgroup="groupPartitions"), row=6, col=2) + fig.add_trace(go.Bar(name='Copied Partitions', x=['Partitions'], y=[last_copied], legendgroup="groupPartitions"), row=6, col=2) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Events Rate per Second',textfont=dict(size=30, color="black")), row=6, col=2) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Total and Copied Partitions', textfont=dict(size=30, color="black")), row=6, col=2) fig.update_yaxes(range=[-1, 1], row=6, col=2) fig.update_xaxes(range=[-1, 1], row=6, col=2) - # Index Built Over Time - if index_built_times: - fig.add_trace(go.Scattergl(x=index_built_times, y=indexes_built, mode='lines', name='Indexes Built', legendgroup="groupIndexBuilt"), row=7, col=1) + # Row 7: Collection Copy Source Read + if CollectionCopySourceRead or CollectionCopySourceRead_maximum: + fig.add_trace(go.Scattergl(x=times, y=CollectionCopySourceRead, mode='lines', name='Average time (ms)', legendgroup="groupCCSourceRead"), row=7, col=1) + fig.add_trace(go.Scattergl(x=times, y=CollectionCopySourceRead_maximum, mode='lines', name='Maximum time (ms)', legendgroup="groupCCSourceRead"), row=7, col=1) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Index Built', textfont=dict(size=30, color="black")), row=7, col=1) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Collection Copy Source Read',textfont=dict(size=30, color="black")), row=7, col=1) fig.update_yaxes(range=[-1, 1], row=7, col=1) fig.update_xaxes(range=[-1, 1], row=7, col=1) - # Total and Index Built - if index_built_times: - last_built = indexes_built[-1] - last_total = indexes_total[-1] - fig.add_trace(go.Bar(name='Total Indexes', x=['Indexes'], y=[last_total], legendgroup="groupIndexBuilt"), row=7, col=2) - fig.add_trace(go.Bar(name='Indexes Built', x=['Indexes'], y=[last_built], legendgroup="groupIndexBuilt"), row=7, col=2) + # Row 7: Collection Copy Source Reads (numOperations) + if CollectionCopySourceRead_numOperations: + fig.add_trace(go.Scattergl(x=times, y=CollectionCopySourceRead_numOperations, mode='lines', name='Reads', legendgroup="groupCCSourceRead"), row=7, col=2) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Total and Index Built', textfont=dict(size=30, color="black")), row=7, col=2) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Collection Copy Source Reads',textfont=dict(size=30, color="black")), row=7, col=2) fig.update_yaxes(range=[-1, 1], row=7, col=2) fig.update_xaxes(range=[-1, 1], row=7, col=2) - # Ping Latency - if sourcePingLatencyMs or destinationPingLatencyMs: - fig.add_trace(go.Scattergl(x=times, y=sourcePingLatencyMs, mode='lines', name='Source Ping (ms)', legendgroup="groupPingLatency"), row=8, col=1) - fig.add_trace(go.Scattergl(x=times, y=destinationPingLatencyMs, mode='lines', name='Destination Ping (ms)', legendgroup="groupPingLatency"), row=8, col=1) + # Row 8: Collection Copy Destination Write + if CollectionCopyDestinationWrite or CollectionCopyDestinationWrite_maximum: + fig.add_trace(go.Scattergl(x=times, y=CollectionCopyDestinationWrite, mode='lines', name='Average time (ms)', legendgroup="groupCCDestinationWrite"), row=8, col=1) + fig.add_trace(go.Scattergl(x=times, y=CollectionCopyDestinationWrite_maximum, mode='lines', name='Maximum time (ms)', legendgroup="groupCCDestinationWrite"), row=8, col=1) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Ping Latency', textfont=dict(size=30, color="black")), row=8, col=1) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Collection Copy Destination Write',textfont=dict(size=30, color="black")), row=8, col=1) fig.update_yaxes(range=[-1, 1], row=8, col=1) fig.update_xaxes(range=[-1, 1], row=8, col=1) - # Average Source CRUD Event Rate - if srcCRUDEventsPerSec: - fig.add_trace(go.Scattergl(x=crud_rate_times, y=srcCRUDEventsPerSec, mode='lines', name='Events/sec', legendgroup="groupCRUDRate"), row=8, col=2) + # Row 8: Collection Copy Destination Writes (numOperations) + if CollectionCopyDestinationWrite_numOperations: + fig.add_trace(go.Scattergl(x=times, y=CollectionCopyDestinationWrite_numOperations, mode='lines', name='Writes', legendgroup="groupCCDestinationWrite"), row=8, col=2) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='CRUD Event Rate', textfont=dict(size=30, color="black")), row=8, col=2) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Collection Copy Destination Writes',textfont=dict(size=30, color="black")), row=8, col=2) fig.update_yaxes(range=[-1, 1], row=8, col=2) fig.update_xaxes(range=[-1, 1], row=8, col=2) - # Collection Copy Source Read - if CollectionCopySourceRead or CollectionCopySourceRead_maximum: - fig.add_trace(go.Scattergl(x=times, y=CollectionCopySourceRead, mode='lines', name='Average time (ms)', legendgroup="groupCCSourceRead"), row=9, col=1) - fig.add_trace(go.Scattergl(x=times, y=CollectionCopySourceRead_maximum, mode='lines', name='Maximum time (ms)', legendgroup="groupCCSourceRead"), row=9, col=1) + # Row 9: Total Events Applied + if totalEventsApplied: + fig.add_trace(go.Scattergl(x=times, y=totalEventsApplied, mode='lines', name='Events', legendgroup="groupEventsAndLags"), row=9, col=1) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Collection Copy Source Read',textfont=dict(size=30, color="black")), row=9, col=1) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Change Events Applied',textfont=dict(size=30, color="black")), row=9, col=1) fig.update_yaxes(range=[-1, 1], row=9, col=1) fig.update_xaxes(range=[-1, 1], row=9, col=1) - if CollectionCopySourceRead_numOperations: - fig.add_trace(go.Scattergl(x=times, y=CollectionCopySourceRead_numOperations, mode='lines', name='Reads', legendgroup="groupCCSourceRead"), row=9, col=2) + # Row 9: Events Rate per Second + if eventRatePerSecond: + fig.add_trace(go.Scattergl(x=eventRatePerSecond_times, y=eventRatePerSecond, mode='lines', name='Events/sec', legendgroup="groupEventsAndLags"), row=9, col=2) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Collection Copy Source Reads',textfont=dict(size=30, color="black")), row=9, col=2) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Events Rate per Second',textfont=dict(size=30, color="black")), row=9, col=2) fig.update_yaxes(range=[-1, 1], row=9, col=2) fig.update_xaxes(range=[-1, 1], row=9, col=2) - #Collection Copy Destination - if CollectionCopyDestinationWrite or CollectionCopyDestinationWrite_maximum: - fig.add_trace(go.Scattergl(x=times, y=CollectionCopyDestinationWrite, mode='lines', name='Average time (ms)', legendgroup="groupCCDestinationWrite"), row=10, col=1) - fig.add_trace(go.Scattergl(x=times, y=CollectionCopyDestinationWrite_maximum, mode='lines', name='Maximum time (ms)', legendgroup="groupCCDestinationWrite"), row=10, col=1) + # Row 10: CEA Source Read + if CEASourceRead or CEASourceRead_maximum: + fig.add_trace(go.Scattergl(x=times, y=CEASourceRead, mode='lines', name='Average time (ms)', legendgroup="groupCEASourceRead"), row=10, col=1) + fig.add_trace(go.Scattergl(x=times, y=CEASourceRead_maximum, mode='lines', name='Maximum time (ms)', legendgroup="groupCEASourceRead"), row=10, col=1) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Collection Copy Destination Write',textfont=dict(size=30, color="black")), row=10, col=1) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='CEA Source Read',textfont=dict(size=30, color="black")), row=10, col=1) fig.update_yaxes(range=[-1, 1], row=10, col=1) fig.update_xaxes(range=[-1, 1], row=10, col=1) - if CollectionCopyDestinationWrite_numOperations: - fig.add_trace(go.Scattergl(x=times, y=CollectionCopyDestinationWrite_numOperations, mode='lines', name='Writes', legendgroup="groupCCDestinationWrite"), row=10, col=2) + # Row 10: CEA Source Reads (numOperations) + if CEASourceRead_numOperations: + fig.add_trace(go.Scattergl(x=times, y=CEASourceRead_numOperations, mode='lines', name='Reads', legendgroup="groupCEASourceRead"), row=10, col=2) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Collection Copy Destination Writes',textfont=dict(size=30, color="black")), row=10, col=2) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='CEA Source Reads',textfont=dict(size=30, color="black")), row=10, col=2) fig.update_yaxes(range=[-1, 1], row=10, col=2) fig.update_xaxes(range=[-1, 1], row=10, col=2) - #CEA Source - if CEASourceRead or CEASourceRead_maximum: - fig.add_trace(go.Scattergl(x=times, y=CEASourceRead, mode='lines', name='Average time (ms)', legendgroup="groupCEASourceRead"), row=11, col=1) - fig.add_trace(go.Scattergl(x=times, y=CEASourceRead_maximum, mode='lines', name='Maximum time (ms)', legendgroup="groupCEASourceRead"), row=11, col=1) + # Row 11: CEA Destination Write + if CEADestinationWrite or CEADestinationWrite_maximum: + fig.add_trace(go.Scattergl(x=times, y=CEADestinationWrite, mode='lines', name='Average time (ms)', legendgroup="groupCEADestinationWrite"), row=11, col=1) + fig.add_trace(go.Scattergl(x=times, y=CEADestinationWrite_maximum, mode='lines', name='Maximum time (ms)', legendgroup="groupCEADestinationWrite"), row=11, col=1) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='CEA Source Read',textfont=dict(size=30, color="black")), row=11, col=1) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='CEA Destination Write',textfont=dict(size=30, color="black")), row=11, col=1) fig.update_yaxes(range=[-1, 1], row=11, col=1) fig.update_xaxes(range=[-1, 1], row=11, col=1) - if CEASourceRead_numOperations: - fig.add_trace(go.Scattergl(x=times, y=CEASourceRead_numOperations, mode='lines', name='Reads', legendgroup="groupCEASourceRead"), row=11, col=2) + # Row 11: CEA Destination Writes (numOperations) + if CEADestinationWrite_numOperations: + fig.add_trace(go.Scattergl(x=times, y=CEADestinationWrite_numOperations, mode='lines', name='Writes during CEA', legendgroup="groupCEADestinationWrite"), row=11, col=2) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='CEA Source Reads',textfont=dict(size=30, color="black")), row=11, col=2) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='CEA Destination Writes',textfont=dict(size=30, color="black")), row=11, col=2) fig.update_yaxes(range=[-1, 1], row=11, col=2) fig.update_xaxes(range=[-1, 1], row=11, col=2) - #CEA Destination - if CEADestinationWrite or CEADestinationWrite_maximum: - fig.add_trace(go.Scattergl(x=times, y=CEADestinationWrite, mode='lines', name='Average time (ms)', legendgroup="groupCEADestinationWrite"), row=12, col=1) - fig.add_trace(go.Scattergl(x=times, y=CEADestinationWrite_maximum, mode='lines', name='Maximum time (ms)', legendgroup="groupCEADestinationWrite"), row=12, col=1) + # Row 12: Index Built Over Time + if index_built_times: + fig.add_trace(go.Scattergl(x=index_built_times, y=indexes_built, mode='lines', name='Indexes Built', legendgroup="groupIndexBuilt"), row=12, col=1) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='CEA Destination Write',textfont=dict(size=30, color="black")), row=12, col=1) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Index Built', textfont=dict(size=30, color="black")), row=12, col=1) fig.update_yaxes(range=[-1, 1], row=12, col=1) fig.update_xaxes(range=[-1, 1], row=12, col=1) - if CEADestinationWrite_numOperations: - fig.add_trace(go.Scattergl(x=times, y=CEADestinationWrite_numOperations, mode='lines', name='Writes during CEA', legendgroup="groupCEADestinationWrite"), row=12, col=2) + # Row 12: Total and Index Built + if index_built_times: + last_built = indexes_built[-1] + last_total = indexes_total[-1] + fig.add_trace(go.Bar(name='Total Indexes', x=['Indexes'], y=[last_total], legendgroup="groupIndexBuilt"), row=12, col=2) + fig.add_trace(go.Bar(name='Indexes Built', x=['Indexes'], y=[last_built], legendgroup="groupIndexBuilt"), row=12, col=2) else: - fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='CEA Destination Writes',textfont=dict(size=30, color="black")), row=12, col=2) + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Total and Index Built', textfont=dict(size=30, color="black")), row=12, col=2) fig.update_yaxes(range=[-1, 1], row=12, col=2) fig.update_xaxes(range=[-1, 1], row=12, col=2) + # Row 13: Source Verifier Lag Time + if verifierSrcLagTimeSeconds: + fig.add_trace(go.Scattergl(x=src_lag_times, y=verifierSrcLagTimeSeconds, mode='lines', name='Source Verifier Lag Time (seconds)', legendgroup="groupVerifierLag"), row=13, col=1) + else: + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Source Verifier Lag Time', textfont=dict(size=30, color="black")), row=13, col=1) + fig.update_yaxes(range=[-1, 1], row=13, col=1) + fig.update_xaxes(range=[-1, 1], row=13, col=1) + + # Row 13: Destination Verifier Lag Time + if verifierDstLagTimeSeconds: + fig.add_trace(go.Scattergl(x=dst_lag_times, y=verifierDstLagTimeSeconds, mode='lines', name='Destination Verifier Lag Time (seconds)', legendgroup="groupVerifierLag"), row=13, col=2) + else: + fig.add_trace(go.Scatter(x=[0], y=[0], text="NO DATA", mode='text', name='Destination Verifier Lag Time', textfont=dict(size=30, color="black")), row=13, col=2) + fig.update_yaxes(range=[-1, 1], row=13, col=2) + fig.update_xaxes(range=[-1, 1], row=13, col=2) + # Update layout - # 225 per plot (12 rows = 2700) - fig.update_layout(height=2700, width=1450, title_text="Mongosync Replication Progress - " + version_text + " - Timezone info: " + timeZoneInfo, legend_tracegroupgap=190, showlegend=False) + # 225 per plot (13 rows = 2925) + fig.update_layout(height=2925, width=1450, title_text="Mongosync Replication Progress - " + version_text + " - Timezone info: " + timeZoneInfo, legend_tracegroupgap=190, showlegend=False) # Force all y-axes to start at 0 for better visual comparison fig.update_yaxes(rangemode='tozero') + # Add section label annotations above each section group + section_labels = [ + ("Global Migration Metrics", 'yaxis'), # row 1 + ("Collection Copy Metrics", 'yaxis6'), # row 4 + ("CEA Metrics", 'yaxis15'), # row 9 + ("Indexes Metrics", 'yaxis21'), # row 12 + ("Verifier Metrics", 'yaxis23'), # row 13 + ] + for section_name, yaxis_key in section_labels: + domain = fig.layout[yaxis_key].domain + if domain: + y_pos = domain[1] + 0.012 + fig.add_annotation( + x=0.5, y=y_pos, xref='paper', yref='paper', + text=f'{section_name}', + showarrow=False, + font=dict(size=11, color='#1A3C4A'), + bgcolor='rgba(1, 107, 248, 0.12)', + bordercolor='#016BF8', + borderwidth=1, + borderpad=4 + ) + # Synchronize X-axis date range across all date-based plots + # Tables at row 1 col 2 and row 4 col 2 are excluded (no date axis) if global_min_date and global_max_date: - # Sync Mongosync Phases scatter plot (row 1, col 1) fig.update_xaxes(range=[global_min_date, global_max_date], row=1, col=1) - # Sync Partition Init Progress plot (row 2, col 1) - fig.update_xaxes(range=[global_min_date, global_max_date], row=2, col=1) - for row in range(3, 13): # rows 3 through 12 - for col in range(1, 3): # columns 1 and 2 + for row in range(2, 4): # rows 2-3 (both cols are charts) + for col in range(1, 3): + fig.update_xaxes(range=[global_min_date, global_max_date], row=row, col=col) + fig.update_xaxes(range=[global_min_date, global_max_date], row=4, col=1) + for row in range(5, 14): # rows 5-13 (both cols are charts) + for col in range(1, 3): fig.update_xaxes(range=[global_min_date, global_max_date], row=row, col=col) fig.update_layout( @@ -971,12 +1073,12 @@ def _parse_oplog_time_remaining_minutes(value): # Convert the figure to JSON plot_json = json.dumps(fig, cls=PlotlyJSONEncoder) if logs_line_count > 0 else "" - logging.info(f"Render the plot in the browser") + logger.info(f"Render the plot in the browser") # Generate metrics plot if we have metrics data metrics_plot_json = "" if metrics_collector.metrics_count > 0: - logging.info(f"Creating Prometheus metrics plots") + logger.info(f"Creating Prometheus metrics plots") metrics_plot_json = create_metrics_plots(metrics_collector) # Prepare mongosync options data for HTML table @@ -1018,15 +1120,25 @@ def _parse_oplog_time_remaining_minutes(value): has_logs_data = logs_line_count > 0 and len(data) > 0 has_metrics_data = metrics_collector.metrics_count > 0 - # Render the plot in the browser - return render_template('upload_results.html', - plot_json=plot_json, - metrics_plot_json=metrics_plot_json, - options_data=options_data, - hidden_options_data=hidden_options_data, - start_options_data=start_options_data, - natural_order_data=natural_order_data, - errors_data=matched_errors, - partition_init_data=partition_init_data, - has_logs_data=has_logs_data, - has_metrics_data=has_metrics_data) + template_data = { + 'plot_json': plot_json, + 'metrics_plot_json': metrics_plot_json, + 'options_data': options_data, + 'hidden_options_data': hidden_options_data, + 'start_options_data': start_options_data, + 'natural_order_data': natural_order_data, + 'errors_data': matched_errors, + 'partition_init_data': partition_init_data, + 'has_logs_data': has_logs_data, + 'has_metrics_data': has_metrics_data, + 'log_viewer_lines': list(raw_log_tail), + 'log_store_id': store_id, + } + + snapshot_id = str(uuid_mod.uuid4()) + try: + save_snapshot(snapshot_id, filename, file_size, line_count, store_id, template_data) + except Exception as e: + logger.warning(f"Failed to save snapshot: {e}") + + return render_template('upload_results.html', **template_data) diff --git a/migration/mongosync_insights/migration_verifier.py b/migration/mongosync_insights/lib/migration_verifier.py similarity index 98% rename from migration/mongosync_insights/migration_verifier.py rename to migration/mongosync_insights/lib/migration_verifier.py index 3a285b22..12d6c6f0 100644 --- a/migration/mongosync_insights/migration_verifier.py +++ b/migration/mongosync_insights/lib/migration_verifier.py @@ -142,17 +142,17 @@ def get_generation_name(gen_num): def gatherVerifierMetrics(connection_string, db_name="migration_verification_metadata"): """Gather all verifier metrics and create Plotly figure.""" - from app_config import get_database + from .app_config import get_database try: db = get_database(connection_string, db_name) logger.info(f"Connected to verifier database: {db_name}") except PyMongoError as e: logger.error(f"Failed to connect to verifier database: {e}") - return json.dumps({"error": str(e)}) + return {"error": str(e)} except Exception as e: logger.error(f"Unexpected error connecting to verifier database: {e}") - return json.dumps({"error": f"Connection error: {str(e)}"}) + return {"error": f"Connection error: {str(e)}"} try: # Get latest generations history (limited for performance) @@ -643,18 +643,19 @@ def get_dest_ns(t): fig.update_xaxes(title_text="Tasks", row=ns_row, col=1, title_font=dict(size=11)) fig.update_yaxes(tickfont=dict(size=11), row=ns_row, col=1) - return json.dumps(fig, cls=PlotlyJSONEncoder) + plot_json = json.dumps(fig, cls=PlotlyJSONEncoder) + return json.loads(plot_json) except Exception as e: logger.error(f"Error gathering verifier metrics: {e}") import traceback logger.error(traceback.format_exc()) - return json.dumps({"error": f"Error gathering metrics: {str(e)}"}) + return {"error": f"Error gathering metrics: {str(e)}"} def plotVerifierMetrics(db_name="migration_verification_metadata"): """Render the verifier metrics template.""" - from app_config import REFRESH_TIME + from .app_config import REFRESH_TIME refresh_time = REFRESH_TIME refresh_time_ms = str(int(refresh_time) * 1000) diff --git a/migration/mongosync_insights/mongosync_metrics.json b/migration/mongosync_insights/lib/mongosync_metrics.json similarity index 100% rename from migration/mongosync_insights/mongosync_metrics.json rename to migration/mongosync_insights/lib/mongosync_metrics.json diff --git a/migration/mongosync_insights/mongosync_plot_prometheus_metrics.py b/migration/mongosync_insights/lib/otel_metrics.py similarity index 93% rename from migration/mongosync_insights/mongosync_plot_prometheus_metrics.py rename to migration/mongosync_insights/lib/otel_metrics.py index d61fcd7e..63ae9be1 100644 --- a/migration/mongosync_insights/mongosync_plot_prometheus_metrics.py +++ b/migration/mongosync_insights/lib/otel_metrics.py @@ -465,12 +465,17 @@ def create_metrics_plots(collector: MetricsCollector, config_path: Path = None) # Build flat list with None placeholders for sections with odd counts # This keeps each section visually separated in the 2-column grid flat_items = [] # List of metric dicts or None (for empty cells) + section_boundaries = [] # [(section_name, start_row), ...] + current_cell = 0 for section_name, metrics in sections.items(): + start_row = (current_cell // 2) + 1 + section_boundaries.append((section_name, start_row)) for m in metrics: flat_items.append(m) # Add placeholder if section has odd count (fills remaining cell in row) if len(metrics) % 2 == 1: flat_items.append(None) + current_cell += len(metrics) + (1 if len(metrics) % 2 == 1 else 0) total_cells = len(flat_items) @@ -527,6 +532,30 @@ def create_metrics_plots(collector: MetricsCollector, config_path: Path = None) # Force all y-axes to start at 0 fig.update_yaxes(rangemode='tozero') + # Add section label annotations above each section group + for i, (section_name, start_row) in enumerate(section_boundaries): + axis_idx = (start_row - 1) * 2 + 1 + yaxis_key = 'yaxis' if axis_idx == 1 else f'yaxis{axis_idx}' + domain = fig.layout[yaxis_key].domain + if domain: + y_pos = domain[1] + 0.012 + fig.add_annotation( + x=0.5, y=y_pos, xref='paper', yref='paper', + text=f'{section_name}', + showarrow=False, + font=dict(size=11, color='#1A3C4A'), + bgcolor='rgba(1, 107, 248, 0.12)', + bordercolor='#016BF8', + borderwidth=1, + borderpad=4 + ) + # Hide x-axis tick labels on the last row before each section boundary + if i > 0: + prev_last_row = start_row - 1 + if prev_last_row >= 1: + fig.update_xaxes(showticklabels=False, row=prev_last_row, col=1) + fig.update_xaxes(showticklabels=False, row=prev_last_row, col=2) + # Get global date range for X-axis synchronization all_times = [] for metric_data in collector.time_series.values(): @@ -546,32 +575,3 @@ def create_metrics_plots(collector: MetricsCollector, config_path: Path = None) return json.dumps(fig, cls=PlotlyJSONEncoder) -def process_metrics_lines(lines_iterator) -> str: - """ - Process an iterator of metrics log lines and create plots. - - Args: - lines_iterator: Iterator yielding log lines (bytes or str) - - Returns: - JSON string of the Plotly figure, or empty string if no data - """ - collector = MetricsCollector() - - for line in lines_iterator: - # Handle both bytes and string input - if isinstance(line, bytes): - line = line.decode('utf-8', errors='replace') - line = line.strip() - - if not line or not line.startswith('{'): - continue - - collector.process_line(line) - - logger.info(f"Processed {collector.line_count} metrics lines, extracted {collector.metrics_count} metric points") - - if collector.metrics_count == 0: - return "" - - return create_metrics_plots(collector) diff --git a/migration/mongosync_insights/lib/snapshot_store.py b/migration/mongosync_insights/lib/snapshot_store.py new file mode 100644 index 00000000..4ea8f2ae --- /dev/null +++ b/migration/mongosync_insights/lib/snapshot_store.py @@ -0,0 +1,288 @@ +""" +Snapshot persistence for parsed log analysis results. + +Saves all template data (Plotly figures, tables, log viewer lines) as a +JSON file on disk so that a previous analysis can be reloaded instantly +without re-parsing the original log file. Each snapshot references its +companion SQLite log store DB for full-text search. +""" +import glob +import json +import logging +import os +import time +from datetime import datetime, timezone +from typing import Any, Optional + +from .app_config import LOG_STORE_DIR, LOG_STORE_MAX_AGE_HOURS + +logger = logging.getLogger(__name__) + +SNAPSHOT_VERSION = 1 +_SNAPSHOT_PREFIX = 'mi_snapshot_' +_LOGSTORE_PREFIX = 'mi_logstore_' + + +def _snapshot_path(snapshot_id: str) -> str: + return os.path.join(LOG_STORE_DIR, f'{_SNAPSHOT_PREFIX}{snapshot_id}.json') + + +def _snapshot_meta_path(snapshot_id: str) -> str: + return os.path.join(LOG_STORE_DIR, f'{_SNAPSHOT_PREFIX}{snapshot_id}.meta.json') + + +def _is_main_snapshot_basename(basename: str) -> bool: + """True for mi_snapshot_.json but not mi_snapshot_.meta.json.""" + return ( + basename.startswith(_SNAPSHOT_PREFIX) + and basename.endswith('.json') + and not basename.endswith('.meta.json') + ) + + +def logstore_path(store_id: str) -> str: + return os.path.join(LOG_STORE_DIR, f'{_LOGSTORE_PREFIX}{store_id}.db') + + +def save_snapshot( + snapshot_id: str, + source_filename: str, + source_size: int, + line_count: int, + log_store_id: str, + template_data: dict[str, Any], +) -> str: + """ + Save all parsed analysis data to a JSON file on disk. + + Returns the file path of the saved snapshot. + """ + path = _snapshot_path(snapshot_id) + payload = { + 'version': SNAPSHOT_VERSION, + 'snapshot_id': snapshot_id, + 'created_at': datetime.now(timezone.utc).isoformat(), + 'source_filename': source_filename, + 'source_size_bytes': source_size, + 'line_count': line_count, + 'log_store_id': log_store_id, + 'template_data': template_data, + } + with open(path, 'w', encoding='utf-8') as f: + json.dump(payload, f, separators=(',', ':')) + meta_path = _snapshot_meta_path(snapshot_id) + meta_payload = {k: v for k, v in payload.items() if k != 'template_data'} + with open(meta_path, 'w', encoding='utf-8') as f: + json.dump(meta_payload, f, separators=(',', ':')) + logger.info(f"Saved snapshot {snapshot_id[:8]}... for '{source_filename}' ({line_count} lines)") + return path + + +def load_snapshot(snapshot_id: str) -> Optional[dict]: + """ + Load a snapshot from disk and refresh its TTL. + + Touches the mtime of both the snapshot JSON and its companion SQLite + DB so that age-based cleanup is postponed by another TTL cycle. + + Returns the full snapshot dict (including template_data) or None. + """ + path = _snapshot_path(snapshot_id) + if not os.path.exists(path): + logger.warning(f"Snapshot not found: {path}") + return None + + try: + with open(path, 'r', encoding='utf-8') as f: + data = json.load(f) + except (json.JSONDecodeError, OSError) as e: + logger.error(f"Failed to load snapshot {snapshot_id[:8]}...: {e}") + return None + + # Refresh mtime on snapshot file + try: + os.utime(path, None) + except OSError: + pass + + meta_path = _snapshot_meta_path(snapshot_id) + if os.path.exists(meta_path): + try: + os.utime(meta_path, None) + except OSError: + pass + + # Refresh mtime on companion SQLite DB if it exists + store_id = data.get('log_store_id', '') + if store_id: + db_path = logstore_path(store_id) + try: + if os.path.exists(db_path): + os.utime(db_path, None) + except OSError: + pass + + logger.info(f"Loaded snapshot {snapshot_id[:8]}... ('{data.get('source_filename', '?')}')") + return data + + +def _append_snapshot_row( + results: list[dict], + seen_ids: set[str], + mtime: float, + data: dict, + sid_from_file: str, +) -> None: + age_hours = (time.time() - mtime) / 3600 + if age_hours > LOG_STORE_MAX_AGE_HOURS: + return + snapshot_id = data.get('snapshot_id', sid_from_file) + if not snapshot_id or snapshot_id in seen_ids: + return + seen_ids.add(snapshot_id) + results.append({ + 'snapshot_id': snapshot_id, + 'source_filename': data.get('source_filename', 'Unknown'), + 'created_at': data.get('created_at', ''), + 'source_size_bytes': data.get('source_size_bytes', 0), + 'line_count': data.get('line_count', 0), + 'log_store_id': data.get('log_store_id', ''), + 'age_hours': round(age_hours, 1), + 'mtime': mtime, + }) + + +def list_snapshots() -> list[dict]: + """ + Scan LOG_STORE_DIR for snapshot metadata sidecars and return listing fields. + + Reads small ``mi_snapshot_.meta.json`` files (no ``template_data``). + Legacy snapshots with only the main ``.json`` file are listed by parsing + the full file once. + + Returns a list sorted by mtime descending (most recent first). + """ + results: list[dict] = [] + seen_ids: set[str] = set() + meta_pattern = os.path.join(LOG_STORE_DIR, f'{_SNAPSHOT_PREFIX}*.meta.json') + + for filepath in glob.glob(meta_pattern): + try: + mtime = os.path.getmtime(filepath) + with open(filepath, 'r', encoding='utf-8') as f: + data = json.load(f) + basename = os.path.basename(filepath) + suffix = basename[len(_SNAPSHOT_PREFIX):-len('.meta.json')] + _append_snapshot_row(results, seen_ids, mtime, data, suffix) + except (json.JSONDecodeError, OSError) as e: + logger.warning(f"Skipping unreadable snapshot meta {filepath}: {e}") + continue + + main_pattern = os.path.join(LOG_STORE_DIR, f'{_SNAPSHOT_PREFIX}*.json') + for filepath in glob.glob(main_pattern): + basename = os.path.basename(filepath) + if not _is_main_snapshot_basename(basename): + continue + sid = basename[len(_SNAPSHOT_PREFIX):-len('.json')] + if os.path.exists(_snapshot_meta_path(sid)): + continue + try: + mtime = os.path.getmtime(filepath) + with open(filepath, 'r', encoding='utf-8') as f: + data = json.load(f) + _append_snapshot_row(results, seen_ids, mtime, data, sid) + except (json.JSONDecodeError, OSError) as e: + logger.warning(f"Skipping unreadable legacy snapshot {filepath}: {e}") + continue + + results.sort(key=lambda x: x.get('mtime', 0), reverse=True) + for r in results: + r.pop('mtime', None) + return results + + +def delete_snapshot(snapshot_id: str) -> tuple[bool, str]: + """ + Delete a snapshot JSON file, its metadata sidecar, and its SQLite DB. + + Returns (deleted, log_store_id) where deleted is True if the + snapshot file was found and removed. + """ + path = _snapshot_path(snapshot_id) + meta_path = _snapshot_meta_path(snapshot_id) + deleted = False + store_id = '' + + if os.path.exists(meta_path): + try: + with open(meta_path, 'r', encoding='utf-8') as f: + meta = json.load(f) + store_id = meta.get('log_store_id', '') or store_id + except (json.JSONDecodeError, OSError): + pass + + if os.path.exists(path): + if not store_id: + try: + with open(path, 'r', encoding='utf-8') as f: + data = json.load(f) + store_id = data.get('log_store_id', '') + except (json.JSONDecodeError, OSError): + pass + + try: + os.remove(path) + deleted = True + logger.info(f"Deleted snapshot {snapshot_id[:8]}...") + except OSError as e: + logger.warning(f"Failed to delete snapshot {path}: {e}") + + if os.path.exists(meta_path): + try: + os.remove(meta_path) + except OSError as e: + logger.warning(f"Failed to delete snapshot meta {meta_path}: {e}") + + if store_id: + db_path = logstore_path(store_id) + for fpath in (db_path, db_path + '-wal', db_path + '-shm'): + try: + if os.path.exists(fpath): + os.remove(fpath) + except OSError: + pass + + return deleted, store_id + + +def cleanup_old_snapshots(store_dir: str, max_age_hours: int = 24): + """ + Delete snapshot JSON files older than max_age_hours by mtime. + + Skips ``*.meta.json`` (the glob ``mi_snapshot_*.json`` would otherwise + match those). Removes the sibling ``.meta.json`` when deleting a main file. + + Parallels LogStore.cleanup_old_stores for snapshot files. + """ + cutoff = time.time() - (max_age_hours * 3600) + pattern = os.path.join(store_dir, f'{_SNAPSHOT_PREFIX}*.json') + removed = 0 + for filepath in glob.glob(pattern): + basename = os.path.basename(filepath) + if not _is_main_snapshot_basename(basename): + continue + try: + if os.path.getmtime(filepath) < cutoff: + sid = basename[len(_SNAPSHOT_PREFIX):-len('.json')] + meta_file = os.path.join(store_dir, f'{_SNAPSHOT_PREFIX}{sid}.meta.json') + os.remove(filepath) + removed += 1 + if os.path.exists(meta_file): + try: + os.remove(meta_file) + except OSError as e: + logger.warning(f"Failed to clean up snapshot meta {meta_file}: {e}") + except OSError as e: + logger.warning(f"Failed to clean up snapshot {filepath}: {e}") + if removed: + logger.info(f"Cleaned up {removed} expired snapshot(s) from {store_dir}") diff --git a/migration/mongosync_insights/lib/utils.py b/migration/mongosync_insights/lib/utils.py new file mode 100644 index 00000000..3de32f19 --- /dev/null +++ b/migration/mongosync_insights/lib/utils.py @@ -0,0 +1,38 @@ +def format_byte_size(size_bytes): + kilobyte = 1024 + megabyte = kilobyte * 1024 + gigabyte = megabyte * 1024 + terabyte = gigabyte * 1024 + if size_bytes >= terabyte: + value = size_bytes / terabyte + unit = 'TeraBytes' + elif size_bytes >= gigabyte: + value = size_bytes / gigabyte + unit = 'GigaBytes' + elif size_bytes >= megabyte: + value = size_bytes / megabyte + unit = 'MegaBytes' + elif size_bytes >= kilobyte: + value = size_bytes / kilobyte + unit = 'KiloBytes' + else: + value = size_bytes + unit = 'Bytes' + return round(value, 4), unit + +def convert_bytes(size_bytes, target_unit): + kilobyte = 1024 + megabyte = kilobyte * 1024 + gigabyte = megabyte * 1024 + terabyte = gigabyte * 1024 + if target_unit == 'KiloBytes': + value = size_bytes / kilobyte + elif target_unit == 'MegaBytes': + value = size_bytes / megabyte + elif target_unit == 'GigaBytes': + value = size_bytes / gigabyte + elif target_unit == 'TeraBytes': + value = size_bytes / terabyte + else: + value = size_bytes + return round(value, 4) diff --git a/migration/mongosync_insights/mongosync_insights.py b/migration/mongosync_insights/mongosync_insights.py index 7494cc45..5abf82dd 100644 --- a/migration/mongosync_insights/mongosync_insights.py +++ b/migration/mongosync_insights/mongosync_insights.py @@ -1,18 +1,25 @@ import logging import sys import os -from flask import Flask, render_template, request, make_response -from mongosync_plot_logs import upload_file -from mongosync_plot_metadata import plotMetrics, gatherMetrics, gatherPartitionsMetrics, gatherEndpointMetrics -from migration_verifier import plotVerifierMetrics, gatherVerifierMetrics +from flask import Flask, render_template, request, make_response, jsonify, send_from_directory +from lib.logs_metrics import upload_file +from lib.live_migration_metrics import plotMetrics, gatherMetrics, gatherPartitionsMetrics, gatherEndpointMetrics +from lib.migration_verifier import plotVerifierMetrics, gatherVerifierMetrics from pymongo.errors import InvalidURI, PyMongoError -from app_config import ( +from lib.app_config import ( setup_logging, validate_config, get_app_info, HOST, PORT, MAX_FILE_SIZE, - REFRESH_TIME, APP_VERSION, validate_connection, clear_connection_cache, + REFRESH_TIME, APP_VERSION, DEVELOPER_CREDITS, validate_connection, clear_connection_cache, SECURE_COOKIES, CONNECTION_STRING, VERIFIER_CONNECTION_STRING, - PROGRESS_ENDPOINT_URL, validate_progress_endpoint_url, session_store, SESSION_TIMEOUT + PROGRESS_ENDPOINT_URL, validate_progress_endpoint_url, session_store, SESSION_TIMEOUT, + LOG_STORE_DIR, LOG_STORE_MAX_AGE_HOURS, +) +from lib.connection_validator import sanitize_for_display +from lib.log_store import LogStore +from lib.log_store_registry import log_store_registry +from lib.snapshot_store import ( + load_snapshot, list_snapshots as get_snapshot_list, + delete_snapshot as remove_snapshot, cleanup_old_snapshots, ) -from connection_validator import sanitize_for_display # Cookie name for session ID SESSION_COOKIE_NAME = 'mi_session_id' @@ -49,6 +56,13 @@ def _store_session_data(new_data): # Configure Flask for file uploads app.config['MAX_CONTENT_LENGTH'] = MAX_FILE_SIZE + +@app.route('/static/js/') +def mi_static_js(filename): + """Serve shared JS (static_folder is reserved for /images).""" + return send_from_directory(os.path.join(_base_path, 'static', 'js'), filename) + + # Add security headers to all responses @app.after_request def add_security_headers(response): @@ -86,7 +100,7 @@ def add_security_headers(response): # Make app version available to all templates @app.context_processor def inject_app_version(): - return dict(app_version=APP_VERSION) + return dict(app_version=APP_VERSION, developer_credits=DEVELOPER_CREDITS) # Handle file too large error @app.errorhandler(413) @@ -133,12 +147,103 @@ def home_page(): max_file_size_gb=max_file_size_gb) -@app.route('/upload', methods=['POST']) +@app.route('/logout', methods=['POST']) +def logout(): + session_id = request.cookies.get(SESSION_COOKIE_NAME) + if session_id: + session_store.delete_session(session_id) + log_store_registry.cleanup_expired() + cleanup_old_snapshots(LOG_STORE_DIR, LOG_STORE_MAX_AGE_HOURS) + response = make_response('', 200) + response.delete_cookie(SESSION_COOKIE_NAME) + return response + + +@app.route('/uploadLogs', methods=['POST']) def uploadLogs(): return upload_file() -@app.route('/renderMetrics', methods=['POST']) -def renderMetrics(): +@app.route('/search_logs') +def search_logs(): + """Full-text search across the uploaded log file via SQLite FTS5.""" + store_id = request.args.get('store_id', '').strip() + if not store_id: + return jsonify({'error': 'Missing store_id parameter'}), 400 + + q = request.args.get('q', '').strip() + level = request.args.get('level', '').strip() + try: + page = max(1, int(request.args.get('page', 1))) + except (ValueError, TypeError): + page = 1 + try: + per_page = min(max(1, int(request.args.get('per_page', 50))), 200) + except (ValueError, TypeError): + per_page = 50 + + store = log_store_registry.open_store(store_id) + if store is None: + return jsonify({'error': 'Log store not found or expired'}), 404 + + try: + query = {} + if level: + query['level'] = level + if q: + query['$text'] = q + + result = store.find(query, skip=(page - 1) * per_page, limit=per_page) + result['page'] = page + result['per_page'] = per_page + return jsonify(result) + except Exception as e: + logger.error(f"Log search error: {e}") + return jsonify({'error': 'Search failed', 'detail': str(e)}), 500 + +@app.route('/list_snapshots') +def list_snapshots_route(): + """Return JSON list of saved analysis snapshots.""" + try: + snapshots = get_snapshot_list() + return jsonify(snapshots) + except Exception as e: + logger.error(f"Error listing snapshots: {e}") + return jsonify([]) + +@app.route('/load_snapshot/') +def load_snapshot_route(snapshot_id): + """Load a saved analysis snapshot and render the results page.""" + data = load_snapshot(snapshot_id) + if data is None: + return render_template('error.html', + error_title="Snapshot Not Found", + error_message="The requested analysis snapshot was not found or has expired. " + "Please upload and parse the log file again.") + + store_id = data.get('log_store_id', '') + if store_id: + from lib.snapshot_store import logstore_path + db_path = logstore_path(store_id) + if os.path.exists(db_path): + log_store_registry.register(store_id, db_path) + + template_data = data.get('template_data', {}) + return render_template('upload_results.html', **template_data) + +@app.route('/delete_snapshot/', methods=['DELETE']) +def delete_snapshot_route(snapshot_id): + """Delete a saved analysis snapshot.""" + deleted, store_id = remove_snapshot(snapshot_id) + if store_id: + log_store_registry.remove(store_id) + if deleted: + return jsonify({'status': 'ok'}) + return jsonify({'error': 'Snapshot not found'}), 404 + + + +@app.route('/liveMonitoring', methods=['POST']) +def liveMonitoring(): # Get connection string from env var or form (no caching) if CONNECTION_STRING: TARGET_MONGO_URI = CONNECTION_STRING @@ -224,8 +329,8 @@ def renderMetrics(): return response -@app.route('/get_metrics_data', methods=['POST']) -def getMetrics(): +@app.route('/getLiveMonitoring', methods=['POST']) +def getLiveMonitoring(): # Get connection string from env var or in-memory session store if CONNECTION_STRING: connection_string = CONNECTION_STRING @@ -236,11 +341,11 @@ def getMetrics(): if not connection_string: logger.error("No connection string available for metrics refresh") - return {"error": "No connection string available. Please refresh the page and re-enter your credentials."}, 400 + return jsonify({"error": "No connection string available. Please refresh the page and re-enter your credentials."}), 400 - return gatherMetrics(connection_string) + return jsonify(gatherMetrics(connection_string)) -@app.route('/get_partitions_data', methods=['POST']) +@app.route('/getPartitionsData', methods=['POST']) def getPartitionsData(): # Get connection string from env var or in-memory session store if CONNECTION_STRING: @@ -252,11 +357,11 @@ def getPartitionsData(): if not connection_string: logger.error("No connection string available for partitions data refresh") - return {"error": "No connection string available. Please refresh the page and re-enter your credentials."}, 400 + return jsonify({"error": "No connection string available. Please refresh the page and re-enter your credentials."}), 400 - return gatherPartitionsMetrics(connection_string) + return jsonify(gatherPartitionsMetrics(connection_string)) -@app.route('/get_endpoint_data', methods=['POST']) +@app.route('/getEndpointData', methods=['POST']) def getEndpointData(): # Get endpoint URL from env var or in-memory session store if PROGRESS_ENDPOINT_URL: @@ -268,12 +373,12 @@ def getEndpointData(): if not endpoint_url: logger.error("No progress endpoint URL available for endpoint data refresh") - return {"error": "No progress endpoint URL available. Please refresh the page and re-enter your credentials."}, 400 + return jsonify({"error": "No progress endpoint URL available. Please refresh the page and re-enter your credentials."}), 400 - return gatherEndpointMetrics(endpoint_url) + return jsonify(gatherEndpointMetrics(endpoint_url)) -@app.route('/renderVerifier', methods=['POST']) -def renderVerifier(): +@app.route('/Verifier', methods=['POST']) +def Verifier(): """Render the migration verifier monitoring page.""" # Get connection string from env var or form if VERIFIER_CONNECTION_STRING: @@ -338,7 +443,7 @@ def renderVerifier(): return response -@app.route('/get_verifier_data', methods=['POST']) +@app.route('/getVerifierData', methods=['POST']) def getVerifierData(): """Get migration verifier metrics data for AJAX refresh.""" session_id = request.cookies.get(SESSION_COOKIE_NAME) @@ -351,11 +456,11 @@ def getVerifierData(): if not connection_string: logger.error("No connection string available for verifier metrics refresh") - return {"error": "No connection string available. Please refresh the page and re-enter your credentials."}, 400 + return jsonify({"error": "No connection string available. Please refresh the page and re-enter your credentials."}), 400 db_name = session_data.get('verifier_db_name', 'migration_verification_metadata') - return gatherVerifierMetrics(connection_string, db_name) + return jsonify(gatherVerifierMetrics(connection_string, db_name)) if __name__ == '__main__': # Log startup information @@ -364,14 +469,17 @@ def getVerifierData(): logger.info(f"Log file: {app_info['log_file']}") logger.info(f"Server: {app_info['host']}:{app_info['port']}") + # Clean up expired log store DB files and snapshots from previous runs + LogStore.cleanup_old_stores(LOG_STORE_DIR, LOG_STORE_MAX_AGE_HOURS) + cleanup_old_snapshots(LOG_STORE_DIR, LOG_STORE_MAX_AGE_HOURS) + # Import SSL config - from app_config import SSL_ENABLED, SSL_CERT_PATH, SSL_KEY_PATH + from lib.app_config import SSL_ENABLED, SSL_CERT_PATH, SSL_KEY_PATH # Run the Flask app with or without SSL if SSL_ENABLED: import ssl - import os - + # Verify certificate files exist if not os.path.exists(SSL_CERT_PATH): logger.error(f"SSL certificate not found: {SSL_CERT_PATH}") diff --git a/migration/mongosync_insights/mongosync_insights.spec b/migration/mongosync_insights/mongosync_insights.spec index f94fb870..16336380 100644 --- a/migration/mongosync_insights/mongosync_insights.spec +++ b/migration/mongosync_insights/mongosync_insights.spec @@ -24,21 +24,26 @@ a = Analysis( ('templates', 'templates'), # Static images served by Flask ('images', 'images'), - # Runtime JSON configuration files - ('error_patterns.json', '.'), - ('mongosync_metrics.json', '.'), + # Shared JS (served via /static/js/ route) + ('static', 'static'), + # Runtime JSON next to frozen lib/ (app_config + otel_metrics resolve via __file__) + ('lib/error_patterns.json', 'lib'), + ('lib/mongosync_metrics.json', 'lib'), # certifi CA bundle for pymongo TLS (certifi_path, 'certifi'), ], hiddenimports=[ - 'mongosync_plot_logs', - 'mongosync_plot_metadata', - 'mongosync_plot_prometheus_metrics', - 'mongosync_plot_utils', - 'migration_verifier', - 'file_decompressor', - 'app_config', - 'connection_validator', + 'lib.logs_metrics', + 'lib.live_migration_metrics', + 'lib.migration_verifier', + 'lib.otel_metrics', + 'lib.snapshot_store', + 'lib.log_store', + 'lib.log_store_registry', + 'lib.file_decompressor', + 'lib.utils', + 'lib.app_config', + 'lib.connection_validator', 'plotly', 'engineio.async_drivers.threading', 'dns.resolver', diff --git a/migration/mongosync_insights/mongosync_plot_utils.py b/migration/mongosync_insights/mongosync_plot_utils.py deleted file mode 100644 index 7e33477f..00000000 --- a/migration/mongosync_insights/mongosync_plot_utils.py +++ /dev/null @@ -1,44 +0,0 @@ -def format_byte_size(bytes): - # Define the conversion factors - kilobyte = 1024 - megabyte = kilobyte * 1024 - gigabyte = megabyte * 1024 - terabyte = gigabyte * 1024 - # Determine the appropriate unit and calculate the value - if bytes >= terabyte: - value = bytes / terabyte - unit = 'TeraBytes' - elif bytes >= gigabyte: - value = bytes / gigabyte - unit = 'GigaBytes' - elif bytes >= megabyte: - value = bytes / megabyte - unit = 'MegaBytes' - elif bytes >= kilobyte: - value = bytes / kilobyte - unit = 'KiloBytes' - else: - value = bytes - unit = 'Bytes' - # Return the value rounded to two decimal places and the unit separately - return round(value, 4), unit - -def convert_bytes(bytes, target_unit): - # Define conversion factors - kilobyte = 1024 - megabyte = kilobyte * 1024 - gigabyte = megabyte * 1024 - terabyte = gigabyte * 1024 - # Perform conversion based on target unit - if target_unit == 'KiloBytes': - value = bytes / kilobyte - elif target_unit == 'MegaBytes': - value = bytes / megabyte - elif target_unit == 'GigaBytes': - value = bytes / gigabyte - elif target_unit == 'TeraBytes': - value = bytes / terabyte - else: - value = bytes - # Return the converted value rounded to two decimal places and the unit - return round(value, 4) \ No newline at end of file diff --git a/migration/mongosync_insights/requirements.txt b/migration/mongosync_insights/requirements.txt index a65e9cb2..baa8705a 100644 --- a/migration/mongosync_insights/requirements.txt +++ b/migration/mongosync_insights/requirements.txt @@ -24,5 +24,8 @@ six==1.16.0 # HTTP Requests requests==2.32.3 +# TLS Certificate Bundle +certifi==2026.2.25 + # Progress Bars tqdm==4.66.3 diff --git a/migration/mongosync_insights/static/js/mi-snapshots.js b/migration/mongosync_insights/static/js/mi-snapshots.js new file mode 100644 index 00000000..74bc1899 --- /dev/null +++ b/migration/mongosync_insights/static/js/mi-snapshots.js @@ -0,0 +1,334 @@ +/** + * Shared snapshot list + duplicate-upload flow for home and results (sidebar) layouts. + */ +(function () { + 'use strict'; + + function formatFileSize(bytes) { + if (!bytes || bytes === 0) return ''; + if (bytes >= 1073741824) return (bytes / 1073741824).toFixed(1) + ' GB'; + if (bytes >= 1048576) return (bytes / 1048576).toFixed(1) + ' MB'; + if (bytes >= 1024) return (bytes / 1024).toFixed(1) + ' KB'; + return bytes + ' B'; + } + + function formatAge(hours) { + if (hours < 1) return 'Just now'; + if (hours < 24) return Math.round(hours) + 'h ago'; + return Math.round(hours / 24) + 'd ago'; + } + + function formatDate(isoStr) { + if (!isoStr) return ''; + try { + var d = new Date(isoStr); + return d.toLocaleDateString(undefined, { month: 'short', day: 'numeric' }) + ', ' + + d.toLocaleTimeString(undefined, { hour: 'numeric', minute: '2-digit' }); + } catch (e) { + return ''; + } + } + + function fetchSnapshotsJson() { + return fetch('/list_snapshots').then(function (r) { + return r.json(); + }); + } + + function renderSnapshotsHome(container, snapshots) { + if (!snapshots || snapshots.length === 0) { + container.innerHTML = 'No saved analyses'; + return; + } + + var list = document.createElement('ul'); + list.className = 'prev-analyses-list'; + + snapshots.forEach(function (s) { + var size = formatFileSize(s.source_size_bytes); + var date = formatDate(s.created_at); + var age = formatAge(s.age_hours); + var meta = [date, size, age].filter(Boolean).join(' \u00b7 '); + + var item = document.createElement('li'); + item.className = 'prev-analysis-item'; + + var info = document.createElement('div'); + info.className = 'prev-analysis-info'; + + var name = document.createElement('div'); + name.className = 'prev-analysis-name'; + name.title = s.source_filename || ''; + name.textContent = s.source_filename || 'Unknown'; + + var metaDiv = document.createElement('div'); + metaDiv.className = 'prev-analysis-meta'; + metaDiv.textContent = meta; + + info.appendChild(name); + info.appendChild(metaDiv); + + var actions = document.createElement('div'); + actions.className = 'prev-analysis-actions'; + + var loadLink = document.createElement('a'); + loadLink.className = 'prev-analysis-load'; + loadLink.setAttribute('href', '/load_snapshot/' + encodeURIComponent(s.snapshot_id)); + loadLink.textContent = 'Load'; + + var deleteButton = document.createElement('button'); + deleteButton.className = 'prev-analysis-delete'; + deleteButton.title = 'Delete'; + deleteButton.type = 'button'; + deleteButton.textContent = '\u2716'; + deleteButton.addEventListener('click', function () { + deleteSnapshot(s.snapshot_id); + }); + + actions.appendChild(loadLink); + actions.appendChild(deleteButton); + + item.appendChild(info); + item.appendChild(actions); + list.appendChild(item); + }); + + container.innerHTML = ''; + container.appendChild(list); + } + + function renderSnapshotsUploadDialog(container, snapshots) { + if (!snapshots || snapshots.length === 0) { + container.innerHTML = 'No saved analyses'; + return; + } + + container.innerHTML = ''; + snapshots.forEach(function (s) { + var size = formatFileSize(s.source_size_bytes); + var age = formatAge(s.age_hours); + var meta = [size, age].filter(Boolean).join(' \u00b7 '); + var filename = s.source_filename || 'Unknown'; + var snapshotId = String(s.snapshot_id || ''); + + var item = document.createElement('div'); + item.className = 'upload-dialog-item'; + + var info = document.createElement('div'); + info.className = 'upload-dialog-item-info'; + + var name = document.createElement('div'); + name.className = 'upload-dialog-item-name'; + name.title = s.source_filename || ''; + name.textContent = filename; + + var metaDiv = document.createElement('div'); + metaDiv.className = 'upload-dialog-item-meta'; + metaDiv.textContent = meta; + + info.appendChild(name); + info.appendChild(metaDiv); + + var actions = document.createElement('div'); + actions.className = 'upload-dialog-item-actions'; + + var loadLink = document.createElement('a'); + loadLink.className = 'upload-dialog-load-btn'; + loadLink.setAttribute('href', '/load_snapshot/' + encodeURIComponent(snapshotId)); + loadLink.textContent = 'Load'; + + var deleteButton = document.createElement('button'); + deleteButton.className = 'upload-dialog-del-btn'; + deleteButton.title = 'Delete'; + deleteButton.textContent = '\u2716'; + deleteButton.addEventListener('click', function () { + udDeleteSnapshot(snapshotId); + }); + + actions.appendChild(loadLink); + actions.appendChild(deleteButton); + + item.appendChild(info); + item.appendChild(actions); + container.appendChild(item); + }); + } + + function miRefreshUploadDialogSnapshots() { + var container = document.getElementById('uploadDialogSnapshots'); + if (!container) return; + + container.innerHTML = 'Loading...'; + fetchSnapshotsJson() + .then(function (snapshots) { + renderSnapshotsUploadDialog(container, snapshots); + }) + .catch(function () { + container.innerHTML = 'Failed to load'; + }); + } + + window.loadPreviousAnalyses = function () { + var container = document.getElementById('prevAnalysesContent'); + if (!container) return; + + fetchSnapshotsJson() + .then(function (snapshots) { + renderSnapshotsHome(container, snapshots); + }) + .catch(function () { + container.innerHTML = 'Failed to load saved analyses'; + }); + }; + + window.deleteSnapshot = function (id) { + if (!confirm('Delete this saved analysis?')) return; + fetch('/delete_snapshot/' + encodeURIComponent(id), { method: 'DELETE' }) + .then(function () { + loadPreviousAnalyses(); + }) + .catch(function () { + loadPreviousAnalyses(); + }); + }; + + window.openUploadDialog = function () { + var overlay = document.getElementById('uploadDialogOverlay'); + if (!overlay) return; + overlay.classList.add('active'); + miRefreshUploadDialogSnapshots(); + }; + + window.closeUploadDialog = function () { + var overlay = document.getElementById('uploadDialogOverlay'); + if (overlay) overlay.classList.remove('active'); + }; + + window.triggerNewUpload = function () { + closeUploadDialog(); + var input = document.getElementById('sidebarFileInput'); + if (input) input.click(); + }; + + window.udDeleteSnapshot = function (id) { + if (!confirm('Delete this saved analysis?')) return; + fetch('/delete_snapshot/' + encodeURIComponent(id), { method: 'DELETE' }) + .then(function () { + openUploadDialog(); + }) + .catch(function () { + openUploadDialog(); + }); + }; + + var _dupState = { matches: [], form: null, fileInput: null }; + + window.checkDuplicateAndUpload = function (form, fileInput) { + if (!fileInput || !fileInput.files || !fileInput.files.length) return; + var selectedName = fileInput.files[0].name; + _dupState.form = form; + _dupState.fileInput = fileInput; + _dupState.matches = []; + + fetchSnapshotsJson() + .then(function (snapshots) { + var matches = (snapshots || []).filter(function (s) { + return s.source_filename === selectedName; + }); + if (matches.length === 0) { + _dupProceedUpload(); + return; + } + _dupState.matches = matches; + var age = formatAge(matches[0].age_hours); + var duplicateCheckMsg = document.getElementById('duplicateCheckMsg'); + if (!duplicateCheckMsg) return; + + var fileNameStrong = document.createElement('strong'); + var loadPreviousStrong = document.createElement('strong'); + var replaceStrong = document.createElement('strong'); + + duplicateCheckMsg.textContent = ''; + duplicateCheckMsg.appendChild(document.createTextNode('A saved analysis for ')); + fileNameStrong.textContent = '"' + selectedName + '"'; + duplicateCheckMsg.appendChild(fileNameStrong); + duplicateCheckMsg.appendChild(document.createTextNode(' already exists (' + age + ').')); + duplicateCheckMsg.appendChild(document.createElement('br')); + duplicateCheckMsg.appendChild(document.createElement('br')); + loadPreviousStrong.textContent = 'Load Previous'; + duplicateCheckMsg.appendChild(loadPreviousStrong); + duplicateCheckMsg.appendChild(document.createTextNode(' opens the saved session.')); + duplicateCheckMsg.appendChild(document.createElement('br')); + replaceStrong.textContent = 'Replace'; + duplicateCheckMsg.appendChild(replaceStrong); + duplicateCheckMsg.appendChild(document.createTextNode(' deletes the saved session and uploads the file again.')); + + var dupOverlay = document.getElementById('duplicateCheckOverlay'); + if (dupOverlay) dupOverlay.classList.add('active'); + }) + .catch(function () { + _dupProceedUpload(); + }); + }; + + function _dupProceedUpload() { + var dupOverlay = document.getElementById('duplicateCheckOverlay'); + if (dupOverlay) dupOverlay.classList.remove('active'); + var loading = document.getElementById('uploadLoadingOverlay'); + if (loading) loading.classList.add('active'); + if (_dupState.form) _dupState.form.submit(); + } + + window.duplicateLoadPrevious = function () { + var dupOverlay = document.getElementById('duplicateCheckOverlay'); + if (dupOverlay) dupOverlay.classList.remove('active'); + if (_dupState.matches.length > 0) { + var loading = document.getElementById('uploadLoadingOverlay'); + if (loading) loading.classList.add('active'); + window.location.href = '/load_snapshot/' + encodeURIComponent(_dupState.matches[0].snapshot_id); + } + }; + + window.duplicateReplace = function () { + var dupOverlay = document.getElementById('duplicateCheckOverlay'); + if (dupOverlay) dupOverlay.classList.remove('active'); + var loading = document.getElementById('uploadLoadingOverlay'); + if (loading) loading.classList.add('active'); + var delPromises = _dupState.matches.map(function (s) { + return fetch('/delete_snapshot/' + encodeURIComponent(s.snapshot_id), { method: 'DELETE' }).catch(function () {}); + }); + Promise.all(delPromises).then(function () { + if (_dupState.form) _dupState.form.submit(); + }); + }; + + window.duplicateCancel = function () { + var dupOverlay = document.getElementById('duplicateCheckOverlay'); + if (dupOverlay) dupOverlay.classList.remove('active'); + if (_dupState.fileInput) { + _dupState.fileInput.value = ''; + } + _dupState.matches = []; + _dupState.form = null; + _dupState.fileInput = null; + }; + + function wireSidebarUploadIfPresent() { + var sidebarInput = document.getElementById('sidebarFileInput'); + if (!sidebarInput) return; + sidebarInput.addEventListener('change', function () { + if (this.files.length > 0) { + var form = document.getElementById('sidebarUploadForm'); + if (form) { + checkDuplicateAndUpload(form, this); + } + } + }); + } + + if (document.readyState === 'loading') { + document.addEventListener('DOMContentLoaded', wireSidebarUploadIfPresent); + } else { + wireSidebarUploadIfPresent(); + } +})(); diff --git a/migration/mongosync_insights/templates/_theme_vars.html b/migration/mongosync_insights/templates/_theme_vars.html new file mode 100644 index 00000000..734b3d55 --- /dev/null +++ b/migration/mongosync_insights/templates/_theme_vars.html @@ -0,0 +1,179 @@ + + + + diff --git a/migration/mongosync_insights/templates/base.html b/migration/mongosync_insights/templates/base.html index 5abb91e6..4f40258e 100644 --- a/migration/mongosync_insights/templates/base.html +++ b/migration/mongosync_insights/templates/base.html @@ -1,42 +1,573 @@ - + {% block title %}Mongosync Insights{% endblock %} + {% include '_theme_vars.html' %} + + + + + +

{% block header_title %}Mongosync Insights{% endblock %}

@@ -49,6 +580,177 @@

{% block header_title %}Mongosync Insights{% endblock %}

{% block footer %}{% endblock %} + +
+
+
+

Mongosync Insights Settings

+ +
+
+
+ + +
+
+ + +
+
+ + +
+
+
+ + +
+
+
+ + +
+
+

Logout

+

Are you sure you want to logout? Your session will be cleared.

+
+ + +
+
+
+ + +
+
+
+ Developer Credits + +
+
+ {% if developer_credits %} + {{ developer_credits.get('copyright', '') }} {{ developer_credits.get('year', '') }} + {{ developer_credits.get('team_name', '') }} + {% for c in developer_credits.get('contributors', []) %} + {{ c.get('name', '') }}{% if c.get('role') %} — {{ c.get('role') }}{% endif %} + {% endfor %} + {% endif %} +
+
+
+ + +
+
+
+

Upload Log File

+ +
+
+
Saved Analyses
+
+ Loading... +
+
+ +
+
+
+ + +
+
+

Previous Analysis Found

+

+
+ + + +
+
+
+ + +
+ +

Processing Files...

+

Parsing and analyzing data.

+
+ + + + {% block scripts %}{% endblock %} diff --git a/migration/mongosync_insights/templates/base_home.html b/migration/mongosync_insights/templates/base_home.html index 2f851793..bdde6f0d 100644 --- a/migration/mongosync_insights/templates/base_home.html +++ b/migration/mongosync_insights/templates/base_home.html @@ -1,21 +1,65 @@ - + {% block title %}Mongosync Insights{% endblock %} + {% include '_theme_vars.html' %} @@ -28,6 +72,14 @@ {% block footer %}{% endblock %} + +
+ +

Processing Files...

+

Parsing and analyzing data.

+
+ + {% block scripts %}{% endblock %} diff --git a/migration/mongosync_insights/templates/home.html b/migration/mongosync_insights/templates/home.html index 974e7759..b18489f2 100644 --- a/migration/mongosync_insights/templates/home.html +++ b/migration/mongosync_insights/templates/home.html @@ -5,7 +5,6 @@ {% block header_title %}Mongosync Insights - Logs{% endblock %} {% block styles %} - /* Create a container for the forms */ .form-container { display: flex; gap: 20px; @@ -13,47 +12,222 @@ align-items: stretch; } - /* Style individual forms */ form { width: 350px; - border: 1px solid #ccc; + border: 1px solid var(--mi-frame-border); padding: 20px; - border-radius: 8px; - background-color: #f9f9f9; - box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); + border-radius: var(--mi-radius-sm); + background-color: var(--mi-surface-muted); + box-shadow: var(--mi-shadow-sm); + overflow: hidden; } - /* Style the body of the page */ body { - font-family: Arial, sans-serif; - margin: 0; - padding: 0; - background-color: #ffffff; + background-color: var(--mi-frame-bg); + } + + h1, h2 { color: var(--mi-text-primary); } + p, label, small { color: var(--mi-text-secondary); } + strong { color: var(--mi-text-primary); } + a { color: var(--mi-accent-info); } + + input[type="text"], input[type="file"], select { + background: var(--mi-surface-muted); + color: var(--mi-text-primary); + border: 1px solid var(--mi-frame-border); + border-radius: 4px; + padding: 4px 6px; + max-width: 100%; + width: 100%; + box-sizing: border-box; + } + input[type="submit"] { + background: var(--mi-sidebar-bg); + color: #fff; + border: 1px solid var(--mi-sidebar-bg); + border-radius: 4px; + padding: 6px 16px; + cursor: pointer; + font-size: 14px; } + input[type="submit"]:hover { opacity: 0.85; } .version-info { text-align: center; padding-top: 20px; } + + .prev-analyses-divider { + border: none; + border-top: 1px solid var(--mi-frame-border); + margin: 16px 0 12px; + } + .prev-analyses-title { + font-size: 0.95em; + font-weight: 600; + color: var(--mi-text-secondary); + margin: 0 0 10px; + } + .prev-analyses-empty { + font-size: 0.85em; + color: var(--mi-text-secondary); + font-style: italic; + } + .prev-analyses-list { + list-style: none; + padding: 0; + margin: 0; + } + .prev-analysis-item { + display: flex; + align-items: center; + justify-content: space-between; + padding: 8px 0; + border-bottom: 1px solid var(--mi-frame-border); + } + .prev-analysis-item:last-child { border-bottom: none; } + .prev-analysis-info { + flex: 1; + min-width: 0; + overflow: hidden; + } + .prev-analysis-name { + font-size: 0.88em; + font-weight: 600; + color: var(--mi-text-primary); + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; + } + .prev-analysis-meta { + font-size: 0.78em; + color: var(--mi-text-secondary); + margin-top: 2px; + } + .prev-analysis-actions { + display: flex; + gap: 6px; + align-items: center; + flex-shrink: 0; + margin-left: 8px; + } + .prev-analysis-load { + font-size: 0.8em; + padding: 3px 10px; + border: 1px solid var(--mi-sidebar-bg); + border-radius: 4px; + background: var(--mi-sidebar-bg); + color: #fff; + cursor: pointer; + text-decoration: none; + } + .prev-analysis-load:hover { opacity: 0.85; } + .prev-analysis-delete { + font-size: 0.85em; + padding: 2px 6px; + border: none; + background: transparent; + color: var(--mi-accent-danger); + cursor: pointer; + border-radius: 3px; + } + .prev-analysis-delete:hover { background: rgba(204,68,68,0.1); color: #a00; } + + .dup-overlay { + position: fixed; + top: 0; left: 0; right: 0; bottom: 0; + background: rgba(0, 0, 0, 0.45); + display: none; + align-items: center; + justify-content: center; + z-index: 9000; + } + .dup-overlay.active { display: flex; } + .dup-dialog { + background: var(--mi-dialog-bg); + border: 1px solid var(--mi-frame-border); + border-radius: var(--mi-radius-sm); + box-shadow: var(--mi-shadow-md); + width: 420px; + max-width: 90vw; + padding: 24px; + text-align: center; + } + .dup-dialog h3 { + margin: 0 0 10px 0; + font-size: 16px; + color: var(--mi-text-primary); + } + .dup-dialog p { + margin: 0 0 20px 0; + font-size: 13px; + color: var(--mi-text-secondary); + text-align: left; + line-height: 1.5; + } + .dup-actions { + display: flex; + justify-content: center; + gap: 10px; + } + .dup-btn { + padding: 7px 16px; + border-radius: 4px; + cursor: pointer; + font-size: 13px; + border: 1px solid var(--mi-frame-border); + } + .dup-btn-primary { + background: var(--mi-sidebar-bg); + color: #fff; + border-color: var(--mi-sidebar-bg); + } + .dup-btn-danger { + background: transparent; + color: var(--mi-accent-danger); + border-color: var(--mi-accent-danger); + } + .dup-btn-cancel { + background: transparent; + color: var(--mi-text-secondary); + } + + .verifier-note { + background: rgba(255, 192, 16, 0.15); + border: 1px solid var(--mongodb-yellow-base); + border-radius: 4px; + padding: 8px 12px; + font-size: 0.85em; + margin-top: 12px; + margin-bottom: 0; + } {% endblock %} {% block content %}
-

Mongosync Insights v{{ app_version }}

+

Mongosync Insights v{{ app_version }}

-
+

Parse Mongosync Log File



Click the "Upload" button after selecting your Mongosync log file to generate migration progress plots.

Supported formats: .log, .json, .out (uncompressed) or .gz, .zip, .bz2, .tgz, .tar.gz, .tar.bz2 (compressed)
Maximum size: {{ max_file_size_gb | round(1) }} GB

+ +
+
+

Previous Analyses

+
+ Loading... +
+
-
+

Live Migration Monitoring

{{ connection_string_form | safe }} {{ progress_endpoint_form | safe }} @@ -62,7 +236,7 @@

Live Migration Monitoring

-
+

Migration Verifier

{{ verifier_connection_string_form | safe }} @@ -71,8 +245,26 @@

Migration Verifier

Click the "Monitor Verifier" button to monitor the migration-verifier tool progress.

Note: Connect to the MongoDB cluster where the migration-verifier writes its metadata (typically the destination cluster).

+

+ Note: If verifying a migration done via mongosync, check if the + Embedded Verifier + can be used, as it is the preferred approach for verification. +

+ + +
+
+

Previous Analysis Found

+

+
+ + + +
+
+
{% endblock %} {% block scripts %} @@ -118,7 +310,25 @@

Migration Verifier

return true; } +/* loadPreviousAnalyses, checkDuplicateAndUpload, duplicate*, deleteSnapshot — from mi-snapshots.js */ + document.addEventListener('DOMContentLoaded', function() { + loadPreviousAnalyses(); + + var uploadLogsForm = document.getElementById('uploadLogsForm'); + if (uploadLogsForm) { + uploadLogsForm.addEventListener('submit', function(e) { + var fileInput = this.querySelector('input[type="file"]'); + if (!fileInput || !fileInput.files.length) { + e.preventDefault(); + alert('Please select a file to upload.'); + return; + } + e.preventDefault(); + checkDuplicateAndUpload(this, fileInput); + }); + } + var metadataForm = document.getElementById('metadataForm'); if (metadataForm) { metadataForm.addEventListener('submit', function(e) { diff --git a/migration/mongosync_insights/templates/metrics.html b/migration/mongosync_insights/templates/metrics.html index 3013f9c9..2ca0266e 100644 --- a/migration/mongosync_insights/templates/metrics.html +++ b/migration/mongosync_insights/templates/metrics.html @@ -72,6 +72,32 @@ padding: 50px; } + .endpoint-warnings { + display: none; + margin: 0; + padding: 12px 16px; + background-color: #fff3cd; + border: 1px solid #ffc107; + border-bottom: none; + border-radius: 0; + color: #856404; + font-size: 14px; + font-weight: 500; + } + + .endpoint-warnings ul { + margin: 4px 0 0 0; + padding-left: 20px; + } + + .endpoint-warnings li { + margin-bottom: 4px; + } + + .endpoint-warnings strong { + color: #664d03; + } + @media (max-width: 768px) { .plot-container { width: 95%; @@ -114,6 +140,7 @@ {% if has_endpoint_url %}
+
Loading endpoint data...
@@ -157,7 +184,7 @@ async function fetchOverviewData() { try { // No credentials sent - server reads from secure session - const response = await fetch("/get_metrics_data", { + const response = await fetch("/getLiveMonitoring", { method: 'POST', credentials: 'same-origin' // Include session cookie }); @@ -179,7 +206,7 @@ async function fetchPartitionsData() { try { // No credentials sent - server reads from secure session - const response = await fetch("/get_partitions_data", { + const response = await fetch("/getPartitionsData", { method: 'POST', credentials: 'same-origin' // Include session cookie }); @@ -200,16 +227,39 @@ async function fetchEndpointData() { try { - // No credentials sent - server reads from secure session - const response = await fetch("/get_endpoint_data", { + const response = await fetch("/getEndpointData", { method: 'POST', - credentials: 'same-origin' // Include session cookie + credentials: 'same-origin' }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || 'Request failed'); } - const plotData = await response.json(); + const responseData = await response.json(); + const plotData = responseData.plot; + const warnings = responseData.warnings || []; + + const warningsDiv = document.getElementById("endpoint-warnings"); + if (warnings.length > 0) { + warningsDiv.replaceChildren(); + + const warningLabel = document.createElement("strong"); + warningLabel.textContent = "Warning:"; + warningsDiv.appendChild(warningLabel); + + const warningsList = document.createElement("ul"); + warnings.forEach(w => { + const warningItem = document.createElement("li"); + warningItem.textContent = w; + warningsList.appendChild(warningItem); + }); + warningsDiv.appendChild(warningsList); + warningsDiv.style.display = "block"; + } else { + warningsDiv.style.display = "none"; + warningsDiv.replaceChildren(); + } + document.getElementById("loading3").style.display = "none"; document.getElementById("plot3").style.display = "block"; Plotly.react('plot3', plotData.data, plotData.layout); @@ -237,11 +287,29 @@ fetchEndpointData(); } -// Refresh current tab periodically -setInterval(refreshCurrentTab, {{ refresh_time_ms }}); +// Refresh current tab periodically (allow sessionStorage override) +var _refreshMs = parseInt(sessionStorage.getItem('mi_refresh_time'), 10); +_refreshMs = (_refreshMs > 0 ? _refreshMs * 1000 : {{ refresh_time_ms }}); +var _refreshIntervalId = setInterval(refreshCurrentTab, _refreshMs); + +function _updateFooterRefresh(ms) { + var footerP = document.querySelector('footer p'); + if (footerP) footerP.textContent = footerP.textContent.replace(/Refreshing every \d+ seconds/, 'Refreshing every ' + (ms / 1000) + ' seconds'); +} +_updateFooterRefresh(_refreshMs); + +window.addEventListener('mi-refresh-changed', function(e) { + var newMs = e.detail.refreshSec * 1000; + if (newMs > 0 && newMs !== _refreshMs) { + clearInterval(_refreshIntervalId); + _refreshMs = newMs; + _refreshIntervalId = setInterval(refreshCurrentTab, _refreshMs); + _updateFooterRefresh(_refreshMs); + } +}); {% endblock %} {% block footer %} -

Refreshing every {{ refresh_time }} seconds - Version {{ app_version }}

+

Refreshing every {{ refresh_time }} seconds

{% endblock %} diff --git a/migration/mongosync_insights/templates/upload_results.html b/migration/mongosync_insights/templates/upload_results.html index 30e2838b..e2525235 100644 --- a/migration/mongosync_insights/templates/upload_results.html +++ b/migration/mongosync_insights/templates/upload_results.html @@ -64,9 +64,81 @@ min-height: 400px; } - #plot { + #plot, + #metrics-plot { margin: 0 auto; max-width: 1450px; + position: relative; + } + + .chart-info-overlay { + position: absolute; + top: 0; + left: 0; + width: 100%; + height: 100%; + pointer-events: none; + z-index: 20; + } + + .chart-info-badge { + position: absolute; + width: 18px; + height: 18px; + border-radius: 999px; + border: 1px solid #ccc; + background: #fff; + color: #666; + font-size: 11px; + font-weight: 700; + line-height: 16px; + text-align: center; + cursor: pointer; + pointer-events: auto; + box-shadow: 0 1px 3px rgba(0, 0, 0, 0.12); + padding: 0; + } + + .chart-info-badge:hover { + border-color: #00684A; + color: #00684A; + } + + .chart-info-tooltip { + position: absolute; + min-width: 180px; + max-width: 280px; + background: #fff; + border: 1px solid #ccc; + border-radius: 10px; + box-shadow: 0 2px 8px rgba(0, 0, 0, 0.15); + padding: 8px 10px; + font-size: 12px; + color: #555; + pointer-events: auto; + z-index: 21; + } + + .chart-info-tooltip-title { + margin: 0 0 6px 0; + font-weight: 700; + font-size: 12px; + color: #00684A; + } + + .chart-info-item { + display: flex; + align-items: center; + gap: 6px; + margin: 2px 0; + } + + .chart-info-swatch { + width: 11px; + height: 11px; + border-radius: 2px; + border: 1px solid rgba(0, 0, 0, 0.15); + flex-shrink: 0; } .options-container { @@ -105,6 +177,15 @@ font-weight: 600; } + .options-table th.sortable-th { + cursor: pointer; + user-select: none; + } + + .options-table th.sortable-th:hover { + background-color: rgba(0, 104, 74, 0.08); + } + .options-table td { padding: 10px 15px; border-bottom: 1px solid #e0e0e0; @@ -236,6 +317,36 @@ border-left: 4px solid #00684A; } + .error-detail-content { + display: flex; + flex-direction: column; + gap: 12px; + } + + .error-detail-actions { + display: flex; + gap: 10px; + flex-wrap: wrap; + } + + .error-detail-actions .copy-btn { + padding: 8px 14px; + font-size: 12px; + border: 1px solid #00684A; + background-color: #fff; + color: #00684A; + border-radius: 4px; + cursor: pointer; + transition: all 0.2s ease; + } + + .error-detail-actions .copy-btn:hover { + background-color: #00684A; + color: #fff; + transform: translateY(-2px); + box-shadow: 0 2px 8px rgba(0, 104, 74, 0.2); + } + .error-detail-cell pre { white-space: pre-wrap; word-break: break-word; @@ -243,6 +354,12 @@ font-size: 12px; font-family: 'Courier New', monospace; color: #1a5276; + background-color: #fff; + padding: 12px; + border-radius: 4px; + border: 1px solid #ddd; + max-height: 400px; + overflow-y: auto; } @media (max-width: 768px) { @@ -263,6 +380,196 @@ min-width: 100%; } } + + /* ===== Log Viewer Panel ===== */ + .lv-panel { + display: flex; + flex-direction: column; + height: 75vh; + min-height: 400px; + border: 1px solid #3a3a3a; + border-radius: 6px; + overflow: hidden; + background: #1e1e1e; + } + + .lv-header { + display: flex; + justify-content: space-between; + align-items: center; + flex-wrap: wrap; + gap: 8px; + padding: 8px 12px; + background: #2a2a2a; + border-bottom: 1px solid #3a3a3a; + } + + .lv-header-left { + display: flex; + align-items: center; + flex-wrap: wrap; + gap: 10px; + } + + .lv-header-right { + display: flex; + align-items: center; + gap: 8px; + } + + .lv-title { + font-weight: 600; + font-size: 14px; + color: #e0e0e0; + white-space: nowrap; + } + + .lv-label { + font-size: 11px; + color: #888; + white-space: nowrap; + } + + .lv-severity-filter, .lv-tools { display: flex; align-items: center; gap: 6px; } + .lv-severity-buttons, .lv-view-mode, .lv-semantic-filter { display: flex; gap: 3px; align-items: center; } + + .lv-sev-btn, .lv-mode-btn, .lv-focus-btn { + padding: 2px 8px; + font-size: 11px; + border: 1px solid #555; + border-radius: 4px; + background: transparent; + color: #aaa; + cursor: pointer; + transition: all 0.15s; + } + .lv-sev-btn:hover, .lv-mode-btn:hover, .lv-focus-btn:hover { background: #3a3a3a; color: #ddd; } + .lv-sev-btn.active { background: #444; color: #fff; border-color: #888; } + .lv-mode-btn.active { background: #00684A; color: #fff; border-color: #00684A; } + .lv-focus-btn.active { background: #c9820a; color: #fff; border-color: #c9820a; } + + .lv-sev-btn.info.active { border-color: #4a9; } + .lv-sev-btn.debug.active { border-color: #6a8; } + .lv-sev-btn.trace.active { border-color: #888; } + .lv-sev-btn.warn.active { border-color: #e8a735; } + .lv-sev-btn.error.active { border-color: #e84040; } + .lv-sev-btn.fatal.active { border-color: #d42; background: #5a1a1a; } + .lv-sev-btn.panic.active { border-color: #f0f; background: #5a1a3a; } + + .lv-search { + padding: 4px 8px; + font-size: 12px; + border: 1px solid #555; + border-radius: 4px; + background: #1e1e1e; + color: #e0e0e0; + width: 180px; + } + + .lv-line-count { + font-size: 11px; + color: #888; + white-space: nowrap; + } + + .lv-action-btn { + display: flex; + align-items: center; + justify-content: center; + width: 28px; + height: 28px; + background: transparent; + border: 1px solid #555; + border-radius: 4px; + color: #aaa; + cursor: pointer; + } + .lv-action-btn:hover { background: #3a3a3a; color: #ddd; } + + .lv-viewer { + flex: 1; + overflow-y: auto; + font-family: 'Source Code Pro', Menlo, Consolas, monospace; + font-size: 12px; + line-height: 1.5; + padding: 10px; + background: #1e1e1e; + color: #e0e0e0; + } + + .lv-placeholder { + display: flex; + flex-direction: column; + align-items: center; + justify-content: center; + height: 100%; + color: #888; + gap: 12px; + } + + .lv-log-line { + white-space: pre-wrap; + word-break: break-all; + margin: 0; + padding: 2px 0; + } + .lv-log-line.pretty { white-space: pre; word-break: normal; } + .lv-log-line.summary { white-space: normal; word-break: break-word; font-family: inherit; font-size: 12px; } + + .lv-log-line.error-line { background: rgba(232,64,64,0.12); } + .lv-log-line.warn-line { background: rgba(232,167,53,0.10); } + .lv-log-line.phase-line { background: rgba(0,104,74,0.12); } + + /* Syntax highlighting */ + .lv-log-line .timestamp { color: #7ec8e3; } + .lv-log-line .phase { color: #4ade80; font-weight: 600; } + .lv-log-line .level-error { color: #ef4444; font-weight: 600; } + .lv-log-line .level-warn { color: #eab308; font-weight: 600; } + .lv-log-line .level-info { color: #38bdf8; } + .lv-log-line .level-debug { color: #a78bfa; } + .lv-log-line .level-trace { color: #888; } + .lv-log-line .json-bracket { color: #888; } + .lv-log-line .json-comma { color: #888; } + .lv-log-line .json-colon { color: #888; } + .lv-log-line .json-key { color: #7dd3fc; } + .lv-log-line .json-string { color: #86efac; } + .lv-log-line .json-number { color: #93c5fd; } + .lv-log-line .boolean-true { color: #4ade80; } + .lv-log-line .boolean-false { color: #fb923c; } + .lv-log-line .db-name { color: #67e8f9; } + .lv-log-line .metric-value { color: #c084fc; } + .lv-log-line .duration-avg { color: #38bdf8; } + .lv-log-line .duration-max { color: #fb923c; } + .lv-log-line .operation-count { color: #4ade80; } + .lv-log-line .field-state { color: #f472b6; } + .lv-log-line .field-cancommit { color: #2dd4bf; } + .lv-log-line .field-lagtime { color: #facc15; } + .lv-log-line .field-estimated-total { color: #86efac; } + .lv-log-line .field-estimated-copied { color: #6ee7b7; } + .lv-log-line .invalid-json-badge { display: inline-block; background: #7f1d1d; color: #fca5a5; padding: 0 4px; border-radius: 3px; font-size: 10px; margin-right: 4px; } + + /* Pagination */ + .lv-pagination { + display: flex; + align-items: center; + gap: 10px; + padding: 8px 12px; + background: #2a2a2a; + border-top: 1px solid #3a3a3a; + } + .lv-page-btn { + padding: 4px 12px; + font-size: 12px; + border: 1px solid #555; + border-radius: 4px; + background: transparent; + color: #ccc; + cursor: pointer; + } + .lv-page-btn:hover:not(:disabled) { background: #3a3a3a; } + .lv-page-btn:disabled { opacity: 0.4; cursor: default; } + .lv-page-btn.lv-back-to-tail { border-color: #00684A; color: #4ade80; } + .lv-page-info, .lv-page-total { font-size: 12px; color: #888; } {% endblock %} {% block content %} @@ -278,6 +585,7 @@ + {% endif %}
@@ -447,16 +755,16 @@

Partition Initialization Details

- - - - - - - - - - + + + + + + + + + + @@ -504,10 +812,10 @@

Common Errors Found in Log

CollectionTypeReasonPartitionsDoc CountExpected Partition SizeSamplerIDs SampledInit StartedInit Duration (s)CollectionTypeReasonPartitionsDoc CountExpected Partition SizeSamplerIDs SampledInit StartedInit Duration (s)
- - - - + + + + @@ -518,9 +826,15 @@

Common Errors Found in Log

- + {% endfor %} @@ -534,17 +848,221 @@

Common Errors Found in Log

+ +
+
+
+
+ Log Viewer +
+ Filter: +
+ + + + + + + +
+
+
+
+ View: + + + + +
+
+ Focus: + + + + +
+
+
+
+ + 0 lines + + +
+
+
+
+ + + +

Loading log lines...

+
+
+ +
+
{% endif %} {% endblock %} {% block scripts %} {% endblock %} {% block footer %} -

Version {{ app_version }}

{% endblock %} diff --git a/migration/mongosync_insights/templates/verifier_metrics.html b/migration/mongosync_insights/templates/verifier_metrics.html index c4383ee2..4e6d1364 100644 --- a/migration/mongosync_insights/templates/verifier_metrics.html +++ b/migration/mongosync_insights/templates/verifier_metrics.html @@ -116,9 +116,29 @@ {% block scripts %} {% endblock %} {% block footer %} -

Refreshing every {{ refresh_time }} seconds - - Migration Verifier Docs - - Version {{ app_version }}

+

Refreshing every {{ refresh_time }} seconds - + Migration Verifier Docs

{% endblock %}
Friendly NameTimeLevelMessageFriendly NameTimeLevelMessage
{{ error.level }} {{ error.message }}