Feature/local embed bge remme rebuild - #8
Open
riteshverma wants to merge 5 commits into
Open
riteshverma wants to merge 5 commits into
riteshverma wants to merge 5 commits into
Conversation
…gnore - Deleted unused cache files, including mcp_cache.json and various database snapshots. - Removed large binary files related to FAISS indexing and document metadata. - Updated .gitignore to include new local cache and runtime files for better tracking and management.
…rrect run status - routers/runs.py: thread tenant_context + storage_namespace through process_run; derive final_result status from plan_graph (mirror process_resume) so cancelled or crashed runs no longer report "completed"; offload get_embedding via asyncio.to_thread with fail-fast breaker when Ollama is unavailable; add runs.index.json path for fast listing. - core/loop.py, memory/context.py: accept storage_namespace and use it for per-tenant session file paths; async-safe session saves via asyncio.Lock + asyncio.to_thread. - integrations/tenancy.py: add storage_namespace_for_tenant helper. - integrations/registry.py: real tenant-tier adapter hook (instead of a silent no-op); explicit integration_id from client still wins. - core/supabase_auth.py: long-lived PyJWKClient per JWKS URL (TTL wrapper was invalidating its own cache every 5 minutes); verify_supabase_jwt offloaded via asyncio.to_thread; safer bearer extraction. - core/supabase_config.py (new): single source for Supabase URL/keys/flags. - core/embedding.py (new), remme/utils.py: unified Ollama embedding helper that raises on failure instead of returning a zero vector. - tests/integrations/test_runs_router_contract.py: coverage for AST-parsed RAG snippets and explicit tenant_context propagation. Made-with: Cursor
Builds on the Tier 1 commit (tenancy plumbing, JWT handling, unified embeddings, correct run status). Tier 2 - performance and reliability - mcp_servers/multi_mcp.py: asyncio.wait_for around call_tool using settings.mcp.tool_timeout_seconds. - core/event_bus.py: per type:source drop counters, sampled warning logs, optional slow-consumer disconnect, drop_stats accessor. - mcp_servers/server_rag.py: use shared embedding helper with per-item fallback and post-fetch normalization. Tier 3 - scheduler and supabase logging - core/scheduler.py: pytz-based timezone from settings; save_jobs offloaded to a thread; lazy CanonicalRunRequest construction in job_wrapper. - core/supabase_logging.py: shared httpx.AsyncClient; structured warning logs; retry with backoff on 5xx. Tier 4 - integration contracts - integrations/contracts.py: CanonicalRunRequest.query min_length=1; new CanonicalRunResponse envelope. - integrations/adapters/default.py: reject empty queries and validate responses through CanonicalRunResponse. - integrations/profiles.py: warn + Prometheus counter on profile parse errors. Tier 5 - tests, config, docs - tests/core/: event bus, scheduler, and persistence unit tests. - tests/integrations/test_contracts.py: empty-query validation test. - config/settings.defaults.json, config/settings.json, config/settings_loader.py: mcp.tool_timeout_seconds and scheduler.timezone, with MCP_TOOL_TIMEOUT_SECONDS and SCHEDULER_TIMEZONE env overrides. - README.md: configuration override-policy section. - api.py: use get_supabase_config in health_auth. - docs/EAGV2_S18_FLOW_REHEARSAL.md: S18 flow rehearsal notes. Made-with: Cursor
- Optional local-embed extra with sentence-transformers; core/embedding.py routes by models.embedding_provider (ollama vs sentence_transformers) with BGE v1.5 prompts. - settings_loader.get_embedding_provider; defaults include embedding_provider. - models.json: qwen2.5:7b-instruct and bge-base-en-v1.5 metadata. - RemmeStore.rebuild_embeddings_from_texts and scripts/rebuild_remme_embeddings.py. - Lockfile updated for local-embed optional dependency. Made-with: Cursor
Made-with: Cursor
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| for embed_url, payload in _build_embed_candidates(full_text): | ||
| try: | ||
| response = requests.post(embed_url, json=payload, timeout=timeout or OLLAMA_TIMEOUT) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.