[lukaszsamson]

Adds the P2 Spark 4.2 Connect APIs: per-left-row nearest-by joins and seven SQL vector functions. Expands live coverage for tuple/KLL sketches, geospatial overloads, and nested Arrow schemas.

Adds opt-in plan compression negotiated from server settings, with uncompressed fallback on older servers and OTP runtimes. Optional caller-stack metadata preserves the Elixir caller across the Session process and lazy stream execution.

Fixes empty-row local DataFrames and zero-column collection so cardinality survives conversion. Explorer returns an explicit error when its batch concatenation cannot preserve zero-column rows. Rejects empty lambda variable names and covers existing artifact-path and bucket validation behavior.

Validation:

• Independent adversarial review completed; findings fixed and reviewed again.
• tools/check.sh: 2,692 tests passed; format, warnings-as-errors compile, Credo, and Dialyzer passed.
• tools/check.sh integration on Spark 4.2 with provider fixtures: 3,406 passed, 2 skipped, 3 excluded.
• Spark 4.1 integration: 682 passed, 2 skipped, 30 version/provider exclusions.
• tools/run_checklist.sh on Spark 4.1: 216 scripts, no failures.
• Dedicated low-limit Spark 4.2 server confirms compressed plans still enforce maximum uncompressed plan size.
• All 25 CI checks passed, covering Elixir 1.15–1.20, OTP 26–29, and Spark 3.5–4.2 (including every integration and checklist job).

[Copilot]

🟡 Changes recommended

Zero-column row decoding can still silently accept/dropping rows when ArrowBatch.row_count is 0 but the IPC payload contains records, and should validate n_rows vs row_count for zero-column schemas.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR extends SparkEx’s Spark Connect support for Spark 4.2 “P2” features and hardens client-side behavior around plan encoding/transport and zero-column result handling.

Changes:

• Added Spark 4.2 nearest-by join support (per-left-row top‑K) and server-only SQL vector functions, including FLOAT degree literal handling.
• Introduced opt-in negotiated plan compression (ZSTD via OTP :zstd) and opt-in client call-site diagnostics carried across the Session boundary and lazy execution.
• Fixed/covered edge cases around zero-column Arrow/Explorer materialization and local empty-row cardinality preservation; expanded unit/integration regression coverage and documentation.

File summaries

File	Description
test/unit/spark42_small_fixes_test.exs	Adds unit coverage for lambda var validation, bucket arg errors, and artifact path handling.
test/unit/spark42_nearest_by_join_test.exs	Unit coverage for nearest-by join plan construction/encoding and option validation.
test/unit/spark_42_functions_test.exs	Extends unit tests to validate function registry inventory and vector function argument handling.
test/unit/plan_compression_test.exs	Adds unit coverage for plan compression behavior and negotiation caching/fallback paths.
test/unit/call_site_test.exs	Adds unit tests for call-site trace capture/scoping across Session boundary and lazy streams.
test/support/plan_compression_limit.exs	Adds a manual script to validate server maxPlanSize enforcement with compressed plans.
test/integration/spark42_zero_column_result_test.exs	Integration tests for zero-column Arrow/Explorer behavior and hostile metadata validation.
test/integration/spark42_nearest_by_join_test.exs	Integration tests for nearest-by join semantics across modes/join types and correlated expressions.
test/integration/spark42_local_input_test.exs	Integration tests for local empty-row cardinality and round-tripping via Explorer.
test/integration/spark42_functions_test.exs	Expands integration coverage for sketches, geospatial overloads, and vector SQL functions.
test/integration/plan_compression_test.exs	Integration coverage for opt-in compression negotiation and command/analyze behavior.
test/integration/p2_schema_collection_edges_test.exs	Integration tests for nested schemas and collection edge cases.
README.md	Documents optional plan compression and caller diagnostics.
lib/spark_ex/session.ex	Adds plan compression state, call-site propagation wrapper, and zero-column local relation handling.
lib/spark_ex/macros/function_registry.ex	Registers Spark 4.2 server-only vector SQL functions.
lib/spark_ex/macros/function_gen.ex	Generates vector degree functions with FLOAT literal casting/coercion.
lib/spark_ex/functions.ex	Adds helper to cast float literals to Spark FLOAT for vector degrees.
lib/spark_ex/data_frame.ex	Introduces nearest_by_join/5 API with option normalization/validation.
lib/spark_ex/connect/result_decoder.ex	Hardens Arrow batch decoding (row_count validation) and improves zero-column Explorer/row decoding.
lib/spark_ex/connect/plan_encoder.ex	Encodes nearest-by join relation and enforces non-empty lambda variable names.
lib/spark_ex/connect/plan_compression.ex	Implements negotiated plan compression and analyze request compression.
lib/spark_ex/connect/client.ex	Plumbs plan compression + call-site extensions into Execute/Analyze/Config requests.
lib/spark_ex/call_site.ex	Implements opt-in call-site capture and scoped propagation utilities.
lib/spark_ex.ex	Documents :plan_compression connect option.
docs/spark_42.md	Documents Spark 4.2 behaviors (zero-column semantics, vector functions, nearest-by joins, call-site diagnostics).
CHANGELOG.md	Adds an Unreleased entry describing the Spark 4.2 P2 APIs and client fixes.

Review details

• Files reviewed: 26/26 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.