FIX: PR #744 review — arch-correctness gates, osx-arm64 honesty, and bloat cleanup

Correctness / arch:
- audit_bundled_binaries.py: add an ELF e_machine arch gate (linux-64==x86_64,
  linux-aarch64==aarch64) -- the Linux twin of assert_pe_machine.py; +3 tests.
- consolidate-conda-artifacts-job.yml: reconcile the win-arm64 accounting (25->28).
- build-conda-packages.sh + conda-build-pipeline.yml: correct the false osx-arm64
  "static arm64-slice audit (lipo/otool)" claims (no such check exists) to admit the
  arm64 slice is trusted from the universal2 wheel tag; real guards are PE + ELF.

Hygiene:
- rename test_026_driver_load_probe.py -> test_033 (dup with main's test_026).
- build-conda-packages.sh: idempotent conda_builder env create (set -e safe).
- tls_connect_probe.py: MS-ODBCSTR }} brace escaping in _split_top_level.

Bloat removal (ponytail):
- drop the YAGNI _DRIVER_LOAD_FAILURE_MARKERS / _OPENSSL_UNREACHABLE_MARKERS tables
  (describe() decoration only; classifiers use separate positive-marker lists).
- remove the dead targetArch param + always-true ne(...,'arm64') condition + args.
- remove the vestigial -Package/-DriverCondaDir params + odbc no-op branches from
  build-conda-packages.ps1.

Probe/audit unit tests pass; black clean. Validated on a NonOfficial ADO build run.

Author: jahnvi480

OneBranchPipelines/conda-build-pipeline.yml

@@ -150,7 +150,6 @@ extends:
                 parameters:
                   pythonVersion: '${{ parameters.pythonVersions }}'
                   condaSubdir: 'win-64'
-                  targetArch: 'x64'
                   odbcWheelDir: '$(Pipeline.Workspace)/odbc_wheels'
                   odbcWheelFilter: 'mssql_python_odbc-*win_amd64.whl'
               # win-arm64 conda build: CROSS-built on THIS x64 agent via CONDA_SUBDIR, in the
@@ -169,7 +168,6 @@ extends:
                   pythonVersion: '3.12,3.13,3.14'
                   condaSubdir: 'win-arm64'
                   condaTargetSubdir: 'win-arm64'
-                  targetArch: 'x64'
                   odbcWheelDir: '$(Pipeline.Workspace)/odbc_wheels'
                   odbcWheelFilter: 'mssql_python_odbc-*win_arm64.whl'
                   outputDir: '$(Agent.TempDirectory)/conda-bld-winarm64'
@@ -222,7 +220,8 @@ extends:
                   artifactName: 'drop_ConsolidateOdbc_ConsolidateArtifacts'
                   targetPath: '$(Pipeline.Workspace)/odbc_wheels'
               # osx-arm64: CROSS-built on the Intel agent (BEST-EFFORT -- the runtime import
-              # auto-skips; the static arm64-slice audit stands in).
+              # auto-skips). NOTE: osx-arm64 arch is NOT independently verified here (no
+              # Mach-O arch check); it is trusted from the universal2 wheel tag, like PyPI.
               - template: /OneBranchPipelines/steps/conda-build-validate-step-posix.yml@self
                 parameters:
                   condaSubdir: 'osx-arm64'

OneBranchPipelines/jobs/consolidate-conda-artifacts-job.yml

@@ -13,16 +13,19 @@
 # package (which vendors the ODBC payload) is emitted per-Python by each build leg;
 # there is NO separate companion package:
 #   win-64        : 5 py x mssql-python                                       = 5
+#   win-arm64     : 3 py x mssql-python  (py3.12-3.14, cross-built on x64)    = 3
 #   osx-64        : 5 py x mssql-python  (Intel Mac, cross-built via Rosetta) = 5
 #   osx-arm64     : 5 py x mssql-python  (Apple Silicon, native)              = 5
 #   linux-64      : 5 py x mssql-python  (glibc x86_64 host, native)          = 5
 #   linux-aarch64 : 5 py x mssql-python  (x86_64 host + QEMU, best-effort)    = 5
 #   ------------------------------------------------------------------------------
-#   TOTAL (PyPI parity minus win-arm64 + musllinux)                          = 25
-# win-arm64 (no import-validation host on x64) and musllinux (no conda musl subdir)
-# are intentionally NOT conda-built. This job is BEST-EFFORT and never hard-fails on
-# a short set; the release pipeline's conda-release-step enforces the hard gate
-# (required subdirs present + complete Python matrix) before anything is published.
+#   TOTAL (PyPI parity minus musllinux)                                      = 28
+# win-arm64 is cross-built for py3.12-3.14 only (py3.10/3.11 have no win-arm64 deps
+# on Anaconda defaults); its runtime import is skipped on the x64 host, so its arch
+# is enforced by the PE-machine assert (assert_pe_machine.py). musllinux (no conda
+# musl subdir) is intentionally NOT conda-built. This job is BEST-EFFORT and never
+# hard-fails on a short set; the release pipeline's conda-release-step enforces the
+# hard gate (required subdirs present + complete Python matrix) before publish.
 parameters:
   - name: oneBranchType
     type: string

OneBranchPipelines/scripts/build-conda-packages.ps1

@@ -11,7 +11,8 @@
 # binfmt for linux-aarch64, and as a CROSS-build for osx-arm64 on the Intel macOS
 # agent (there is no reverse Rosetta, so the arm64 Python is never executed --
 # conda/*/build.sh extract the universal2 wheel without Python and the section-7
-# runtime import is skipped; the pipeline's static arm64-slice audit stands in).
+# runtime import is skipped -- osx-arm64 arch is NOT independently verified here
+# (trusted from the universal2 wheel tag, like the PyPI wheel; no Mach-O arch check)).
 #
 # Args:
 #   $1 WheelsDir           find-links dir holding the mssql-python + mssql-python-odbc wheels
@@ -27,7 +28,9 @@
 #                          target's Python -- true natively, under Rosetta 2 (osx-64
 #                          on Apple Silicon) and under QEMU binfmt (linux-aarch64 on
 #                          x86_64). For osx-arm64 on the Intel agent it is NOT, so
-#                          that leg auto-skips the import (static arch audit stands in).
+#                          that leg auto-skips the import. NOTE: osx-arm64 arch is not
+#                          independently verified here -- trusted from the universal2
+#                          wheel tag; there is no Mach-O arch check in this pipeline.
 set -euo pipefail
 
 WheelsDir="${1:?WheelsDir required}"
@@ -139,6 +142,10 @@ echo "Using conda: $conda"
 # rides along so the RUNPATH audit reads .conda metadata from this same env.
 condaBuildEnv="conda_builder"
 echo "=== creating dedicated conda-build env ($condaBuildEnv: conda-build<26) ==="
+# Idempotent: a reused agent/workdir may already have this env, and a pre-existing
+# env makes `conda create` fail under `set -e`. Remove it first (best-effort, like
+# the verify envs below) so a rerun recreates cleanly.
+"$conda" env remove -y -n "$condaBuildEnv" 2>/dev/null || true
 "$conda" create -y -n "$condaBuildEnv" -c conda-forge --override-channels "conda-build<26" zstandard
 
 # ---------------------------------------------------------------------------
@@ -290,14 +297,16 @@ for py in $pyvers; do
   "$conda" run -n "$envName" python -c "import sys" >/dev/null 2>&1 || target_runnable=0
   if [ "$target_runnable" = "0" ]; then
     # The ONLY leg allowed to skip the runtime proof is the osx-arm64 cross-build on
-    # an Intel agent (no reverse Rosetta): the arm64 Python genuinely cannot run here,
-    # and the pipeline's static arm64-slice audit (lipo/otool/file on the arm64 Mach-O
-    # payload) stands in -- the same assurance as the shipping PyPI universal2 arm64
-    # slice. Every OTHER target (linux-64/osx-64 native, linux-aarch64 under QEMU
-    # binfmt) MUST run its own import; a leg that cannot is a real failure, never a
-    # silent pass -- otherwise a broken linux-aarch64 package ships unvalidated.
+    # an Intel agent (no reverse Rosetta): the arm64 Python genuinely cannot run here.
+    # CAVEAT: osx-arm64 arch is NOT independently verified in this pipeline. Unlike
+    # Windows (assert_pe_machine.py PE check) and Linux (audit_bundled_binaries.py ELF
+    # e_machine check), there is NO Mach-O arch audit, so the arm64 slice is trusted
+    # from the universal2 wheel tag -- exactly like the shipping PyPI universal2 wheel.
+    # Every OTHER target (linux-64/osx-64 native, linux-aarch64 under QEMU binfmt) MUST
+    # run its own import; a leg that cannot is a real failure, never a silent pass --
+    # otherwise a broken linux-aarch64 package ships unvalidated.
     if [ "${CONDA_SUBDIR:-}" = "osx-arm64" ] && [ "$(uname -s)" = "Darwin" ]; then
-      echo "=== [py $py] osx-arm64 cross on Intel: target Python not executable; skipping runtime import (static arm64-slice audit covers this leg). ==="
+      echo "=== [py $py] osx-arm64 cross on Intel: target Python not executable; skipping runtime import (osx-arm64 arch NOT independently verified -- trusted from the universal2 wheel tag). ==="
       continue
     fi
     echo "ERROR: [py $py] target Python for CONDA_SUBDIR=${CONDA_SUBDIR:-native} is not executable on $(uname -s)/$(uname -m), and this is NOT the osx-arm64 cross-build. Refusing to silently skip validation (linux-aarch64 requires QEMU binfmt to be registered on this leg)." >&2

OneBranchPipelines/scripts/build-conda-packages.sh

@@ -40,11 +40,6 @@ parameters:
   - name: condaTargetSubdir
     type: string
     default: ''
-  # Target architecture of the wheel build; the step is skipped unless 'x64'
-  # (or a native arch) so cross-compiled legs don't attempt a conda build.
-  - name: targetArch
-    type: string
-    default: 'x64'
   # Directory holding the freshly built mssql-python wheel (setup.py bdist_wheel).
   - name: mssqlWheelDir
     type: string
@@ -147,5 +142,4 @@ steps:
         Write-Host "  staged $subdir/$($p.Name)"
       }
     displayName: 'Conda build + validate (${{ parameters.condaSubdir }} py${{ parameters.pythonVersion }})'
-    condition: ne('${{ parameters.targetArch }}', 'arm64')
     continueOnError: ${{ parameters.continueOnError }}

OneBranchPipelines/steps/conda-build-validate-step.yml

@@ -61,24 +61,6 @@
     "certificate",
 )
 
-# Negative signals: the native driver did NOT load / link / resolve. Listed only
-# to produce a clearer FAIL message -- classification is allowlist-based, so an
-# unrecognized exception still fails closed even if it matches nothing here.
-_DRIVER_LOAD_FAILURE_MARKERS = (
-    "failed to load the driver",
-    "failed to load library",
-    "failed to load required function pointers",
-    "odbc driver not found",
-    "mssql-auth.dll",
-    "mssql-python-odbc",
-    "cannot open shared object",  # linux dlopen failure
-    "image not found",  # macOS dlopen failure
-    "no such file or directory",  # driver binary absent
-    "can't open lib",  # unixODBC could not open the driver
-    "unsupported architecture",
-    "unsupported platform",
-)
-
 
 def driver_loaded(exc):
     """FAIL-CLOSED classifier for the connect outcome.
@@ -99,12 +81,7 @@ def describe(exc):
     """Short, human-readable reason string for the probe's stdout / exit line."""
     if exc is None:
         return "clean connect"
-    msg = str(exc)
-    low = msg.lower()
-    for marker in _DRIVER_LOAD_FAILURE_MARKERS:
-        if marker in low:
-            return "driver load failure -> " + msg[:300]
-    return msg[:300]
+    return str(exc)[:300]
 
 
 def main():

conda/driver_load_probe.py

@@ -20,9 +20,9 @@
   These are the only PASS outcomes (see ``_TLS_COMPLETED_MARKERS``).
 * Every other outcome fails closed (non-zero exit). In particular an OpenSSL that
   could not be loaded surfaces BEFORE login as an ``SSL Provider`` /
-  ``libssl``/``libcrypto`` / ``cannot open shared object`` error -- classified
-  here as ``OPENSSL BACKEND UNREACHABLE`` (see ``_OPENSSL_UNREACHABLE_MARKERS``),
-  which is exactly the conda RUNPATH bug this gate is meant to catch.
+  ``libssl``/``libcrypto`` / ``cannot open shared object`` error -- it is not in
+  ``_TLS_COMPLETED_MARKERS`` so it fails closed, which is exactly the conda RUNPATH
+  bug this gate is meant to catch.
 
 IMPORTANT -- masking caveat: this gate is only CONCLUSIVE on a minimal base with
 NO system OpenSSL on the default loader path. On a full agent (or any host with a
@@ -59,22 +59,6 @@
     "password did not match",
 )
 
-# Markers that mean the crypto backend could NOT be loaded / the handshake never
-# ran. Listed for a crisp FAIL message -- classification is allowlist-based, so an
-# unrecognized outcome fails closed even if it matches nothing here.
-_OPENSSL_UNREACHABLE_MARKERS = (
-    "libssl",
-    "libcrypto",
-    "cannot open shared object",  # linux dlopen failure of the crypto backend
-    "image not found",  # macOS dlopen failure
-    "openssl",
-    "ssl provider",  # an SSL Provider error before login = crypto/handshake fail
-    "ssl routines",
-    "encryption not supported",
-    "unable to load",
-    "cannot load",
-)
-
 
 def tls_completed(exc):
     """FAIL-CLOSED classifier: True only when the TLS handshake provably completed.
@@ -101,36 +85,40 @@ def describe(exc):
     """Short, human-readable reason string for the gate's stdout / exit line."""
     if exc is None:
         return "clean connect (TLS handshake completed)"
-    msg = str(exc)
-    low = msg.lower()
-    for marker in _OPENSSL_UNREACHABLE_MARKERS:
-        if marker in low:
-            return "OpenSSL backend unreachable -> " + msg[:300]
-    return msg[:300]
+    return str(exc)[:300]
 
 
 def _split_top_level(conn):
     """Split an ODBC connection string on TOP-LEVEL ``;`` only.
 
     An ODBC value wrapped in ``{...}`` may itself contain ``;`` (MS-ODBCSTR), so a
     naive ``split(';')`` would shred braced values. Track brace depth and break only
-    at depth 0.
+    at depth 0. Inside a braced value ``}}`` is an escaped literal ``}`` (MS-ODBCSTR),
+    NOT a close -- consume both and keep the depth so the value is not split early.
     """
     segments = []
     buf = ""
     depth = 0
-    for ch in conn.strip():
+    s = conn.strip()
+    i = 0
+    while i < len(s):
+        ch = s[i]
         if ch == "{":
             depth += 1
             buf += ch
         elif ch == "}":
+            if depth > 0 and i + 1 < len(s) and s[i + 1] == "}":
+                buf += "}}"  # escaped literal '}' inside a braced value; not a close
+                i += 2
+                continue
             depth = max(0, depth - 1)
             buf += ch
         elif ch == ";" and depth == 0:
             segments.append(buf)
             buf = ""
         else:
             buf += ch
+        i += 1
     segments.append(buf)
     return segments
 
@@ -279,7 +267,7 @@ def main():
     if tls_completed(outcome):
         print("TLS_OK (OpenSSL backend reachable; " + describe(outcome) + ")")
         return
-    sys.exit("TLS/OPENSSL BACKEND UNREACHABLE: " + describe(outcome))
+    sys.exit("TLS HANDSHAKE DID NOT COMPLETE: " + describe(outcome))
 
 
 if __name__ == "__main__":

conda/tls_connect_probe.py

@@ -9,6 +9,9 @@
 out of each built ``.conda`` payload -- via the ``PT_DYNAMIC`` program header the
 *loader itself* uses -- and asserts, statically and exactly:
 
+  * each driver/manager ELF's ``e_machine`` MATCHES the package's conda subdir
+    (``linux-64`` == x86_64, ``linux-aarch64`` == aarch64), so a wrong-arch or
+    mislabeled ``.so`` is caught statically (the Linux twin of ``assert_pe_machine.py``);
   * ``libmsodbcsql*`` and ``libodbcinst.so.2`` carry the EXACT relative ``$ORIGIN``
     climb that lands on the package-root ``lib`` (== ``$PREFIX/lib``), computed from
     each binary's own location -- not a substring, not "any ``..``". A too-short,
@@ -72,6 +75,15 @@
 _DRIVER_NEEDED = ("libkrb5", "libgssapi_krb5", "libodbcinst")
 _ODBCINST_NEEDED = ("libltdl",)
 
+# ELF e_machine architecture ids (ELF header offset 0x12). The conda subdir is the
+# authority: every vendored driver/manager ELF must match it, so an x86_64 .so
+# mislabeled under a linux-aarch64 package is caught statically (the emulated aarch64
+# leg's runtime probe is best-effort and would not). Linux twin of assert_pe_machine.py.
+_EM_X86_64 = 62
+_EM_AARCH64 = 183
+_SUBDIR_MACHINE = {"linux-64": _EM_X86_64, "linux-aarch64": _EM_AARCH64}
+_MACHINE_NAME = {_EM_X86_64: "x86_64", _EM_AARCH64: "aarch64"}
+
 
 def _zstd_decompress(raw: bytes) -> bytes:
     """Decompress a zstandard blob, preferring the 3.14+ stdlib backend."""
@@ -90,6 +102,17 @@ def _is_elf(data: bytes) -> bool:
     return len(data) >= 64 and data[:4] == b"\x7fELF"
 
 
+def elf_machine(data: bytes):
+    """Return the ELF ``e_machine`` architecture id (header offset 0x12), or None.
+
+    Endianness comes from ``e_ident[5]`` (the shipped drivers are ELF64-LE).
+    """
+    if not _is_elf(data):
+        return None
+    en = "<" if data[5] == 1 else ">"
+    return struct.unpack_from(en + "H", data, 0x12)[0]
+
+
 def elf_dynamic(data: bytes) -> dict:
     """Return ``{'runpath': str|None, 'rpath': str|None, 'needed': [str]}``.
 
@@ -289,6 +312,9 @@ def audit_package(path: str) -> list[str]:
         print(f"  SKIP (no Linux ELF payload): {base_name} [subdir={subdir or '?'}]")
         return []
 
+    # The conda subdir is the arch authority; every vendored ELF must match it.
+    expected_machine = _SUBDIR_MACHINE.get(subdir)
+
     errors: list[str] = []
 
     # N2a: the run deps that SERVICE the driver's krb5/openssl/libltdl must be declared.
@@ -342,6 +368,18 @@ def audit_package(path: str) -> list[str]:
             errors.append(f"{name}: expected an ELF binary but the header is not ELF.")
             continue
 
+        # Architecture gate: the ELF machine MUST match the package's conda subdir, so
+        # an x86_64 driver mislabeled under a linux-aarch64 package (which the emulated
+        # leg's best-effort runtime probe would not catch) fails here.
+        if expected_machine is not None:
+            mach = elf_machine(data)
+            if mach != expected_machine:
+                errors.append(
+                    f"{name}: ELF machine {mach} ({_MACHINE_NAME.get(mach, 'unknown')}) does "
+                    f"not match the '{subdir}' package arch {expected_machine} "
+                    f"({_MACHINE_NAME[expected_machine]}) -- wrong-arch/mislabeled driver."
+                )
+
         dyn = elf_dynamic(data)
         entries = _entries(effective_runpath(dyn))
         needed = dyn["needed"]
@@ -480,9 +518,9 @@ def main(argv: list | None = None) -> int:
         print("\nOK: no Linux packages present; nothing to audit (win/osx have no ELF payload).")
     else:
         print(
-            f"\nOK: all {linux_checked} Linux package(s) carry the EXACT $ORIGIN climb, keep "
-            f"their krb5/gssapi/libltdl NEEDEDs, declare krb5/libtool/openssl, and vendor no "
-            f"crypto (conda services them)."
+            f"\nOK: all {linux_checked} Linux package(s) match their subdir arch, carry the "
+            f"EXACT $ORIGIN climb, keep their krb5/gssapi/libltdl NEEDEDs, declare "
+            f"krb5/libtool/openssl, and vendor no crypto (conda services them)."
         )
     return 0