Skip to content

Commit 8cf8759

Browse files
gargsaumyaCopilot
andauthored
FEAT: Add opt-in/opt-out ODBC provider selection (msodbcsql18 / mssql-odbc) (#730)
Linked work item: [AB#47445](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/47445) ### Summary This pull request introduces opt-in/opt-out support for selecting which native ODBC provider is loaded by `mssql-python`, allowing users to choose between the default Microsoft ODBC Driver 18 (`msodbcsql18`) and the Rust-based driver (`mssql-odbc`), if the latter is installed. The selection can be made via a module property or environment variable before the first connection, with diagnostics and warnings for precedence and immutability. The implementation includes a new provider manager, updates to documentation, and improvements to driver loading and shutdown safety. **Native ODBC provider selection and diagnostics:** * Added support for selecting the native ODBC provider via the new `mssql_python.native_provider` module property or the `MSSQL_PYTHON_NATIVE_PROVIDER` environment variable (the env var takes precedence). The provider selection is resolved and frozen at the first connection, with warnings for conflicting or late assignments. The default remains `"msodbcsql18"`, but opt-in to `"mssql-odbc"` is supported if the `mssql-python-rs` package is installed. [[1]](diffhunk://#diff-d95f3a67986de29f30453416b1b4c34e6a43207e9a33e2b1b80ef0c378b0a538R604-R618) [[2]](diffhunk://#diff-b073381cfee6e39fdb7cd5eee52930c6dd1eafa4135f394b634791d9694b76bbR1-R250) [[3]](diffhunk://#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4edR46-R57) [[4]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L21-R26) [[5]](diffhunk://#diff-8b251a7d56f9bb22b686d2b101b0420b1d6cd6934b29edcd408c1f579f7d9e84R33) * Introduced `get_native_provider_info()` for diagnostics, reporting the selected provider, source, package, version, driver path, and whether the selection is frozen. [[1]](diffhunk://#diff-d95f3a67986de29f30453416b1b4c34e6a43207e9a33e2b1b80ef0c378b0a538R79-R92) [[2]](diffhunk://#diff-8b251a7d56f9bb22b686d2b101b0420b1d6cd6934b29edcd408c1f579f7d9e84R47) **Driver loading and connection logic:** * Updated connection logic so the ODBC provider is resolved and frozen only after all Python-side validation succeeds and just before the native driver loads, preventing premature provider locking on failed connection attempts. [[1]](diffhunk://#diff-29bb94de45aae51c23a6426d40133c28e4161e68769e08d046059c7186264e90R31) [[2]](diffhunk://#diff-29bb94de45aae51c23a6426d40133c28e4161e68769e08d046059c7186264e90R734-R743) * Ensured that enabling connection pooling does not prematurely resolve or freeze the ODBC provider, so explicit pooling configuration does not lock in the default provider before selection. **Documentation updates:** * Updated `README.md` and `CHANGELOG.md` to document the new provider selection mechanism, its usage, precedence rules, and diagnostic interface. [[1]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L21-R26) [[2]](diffhunk://#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4edR46-R57) **Type hints and interface improvements:** * Updated type hints in `mssql_python.pyi` to reflect the new `native_provider` property and `get_native_provider_info()` function. [[1]](diffhunk://#diff-8b251a7d56f9bb22b686d2b101b0420b1d6cd6934b29edcd408c1f579f7d9e84R33) [[2]](diffhunk://#diff-8b251a7d56f9bb22b686d2b101b0420b1d6cd6934b29edcd408c1f579f7d9e84R47) * Minor import and typing improvements in `mssql_python/__init__.py`. **Native extension (C++ backend) safety:** * Improved Python shutdown/finalization detection in the C++ extension to use thread-safe, GIL-free APIs (`Py_IsFinalizing` or `_Py_IsFinalizing`), preventing crashes during interpreter shutdown from foreign threads. [[1]](diffhunk://#diff-dde2297345718ec449a14e7dff91b7bb2342b008ecc071f562233646d71144a1R17) [[2]](diffhunk://#diff-dde2297345718ec449a14e7dff91b7bb2342b008ecc071f562233646d71144a1R953-R980) --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: gargsaumya <192222169+gargsaumya@users.noreply.github.com>
1 parent ff5897f commit 8cf8759

12 files changed

Lines changed: 1062 additions & 85 deletions

‎CHANGELOG.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
4343
This is a non-breaking step toward decoupling driver-binary updates from
4444
mssql-python releases; a future major version will make the dependency
4545
explicit and drop the bundled binaries.
46+
- **Opt-in/opt-out native provider selection:** a new `mssql_python.native_provider`
47+
module property (and `MSSQL_PYTHON_NATIVE_PROVIDER` environment variable, which
48+
takes precedence) lets a caller select which native ODBC provider is loaded:
49+
the default `"msodbcsql18"` (Microsoft ODBC Driver 18, unchanged behavior) or
50+
the opt-in `"mssql-odbc"` (a Rust-based driver, shipped inside the
51+
`mssql-python-rs` package alongside the Rust TDS core). The selection must be made before the first
52+
connection; it resolves and freezes then, and a later change is ignored with
53+
a `RuntimeWarning`; a conflicting property assignment also warns when the
54+
environment variable takes precedence. Call `mssql_python.get_native_provider_info()`
55+
for diagnostics (selected id, package, version, driver path, source, and
56+
whether it's frozen). This PR
57+
does not change the default provider or ship any Rust driver binaries.
4658

4759
### Changed
4860
- Connection strings and string connection parameters that contain a NUL

‎README.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,12 @@ The driver is compatible with all the Python versions >= 3.10
1818
> - Import name: `mssql_python_odbc`
1919
> - Current version: **18.6.2.1**
2020
>
21-
> `mssql-python` depends on `mssql-python-odbc==18.6.2.1` and loads the ODBC driver binaries from it at import time. `pip install mssql-python` transparently pulls the companion package alongside it — no separate install step is required.
21+
> `mssql-python` depends on `mssql-python-odbc==18.6.2.1`. The ODBC driver is loaded lazily when the first connection is created. `pip install mssql-python` transparently pulls the companion package alongside it — no separate install step is required.
2222
>
23-
> Starting with v1.13.0, the bundled `libs/` fallback that shipped in v1.12.0 has been removed. `mssql-python` will fail to import if `mssql-python-odbc` is not installed. If you install `mssql-python` from a private index or with `--no-deps`, make sure `mssql-python-odbc==18.6.2.1` is installed alongside it.
23+
> Starting with v1.13.0, the bundled `libs/` fallback that shipped in v1.12.0 has been removed. Creating a connection will fail if `mssql-python-odbc` is not installed. If you install `mssql-python` from a private index or with `--no-deps`, make sure `mssql-python-odbc==18.6.2.1` is installed alongside it.
24+
>
25+
> ### ODBC Provider Selection (opt-in)
26+
> `mssql-python` also supports selecting an alternate native ODBC provider before the first connection, via the `mssql_python.native_provider` module property or the `MSSQL_PYTHON_NATIVE_PROVIDER` environment variable (which takes precedence). A conflicting property assignment emits a `RuntimeWarning`. The default, `"msodbcsql18"`, is unchanged; opting into `"mssql-odbc"` requires the `mssql-python-rs` package (which bundles the Rust ODBC driver alongside the Rust TDS core). Call `mssql_python.get_native_provider_info()` to check the selected provider, source, package version, and resolved driver path.
2427
2528
## Installation
2629

‎mssql_python/__init__.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
import threading
1010
import types
1111
import weakref
12+
from typing import Optional
1213

1314
# Import settings from helpers module
1415
from .helpers import Settings, get_settings, _settings, _settings_lock
@@ -75,6 +76,20 @@
7576
# Pooling
7677
from .pooling import PoolingManager
7778

79+
# ODBC provider selection
80+
from .odbc_provider import ProviderManager
81+
82+
83+
def get_native_provider_info() -> dict:
84+
"""Return the selected native provider for diagnostics.
85+
86+
Reports the provider ``id``, package ``version``, resolved ``driver_path``,
87+
the ``package`` that ships its native binaries, selection ``source``, and
88+
whether the choice is ``frozen`` (loaded and no longer changeable).
89+
"""
90+
return ProviderManager.get_info()
91+
92+
7893
# Global registry for tracking active connections (using weak references)
7994
_active_connections = weakref.WeakSet()
8095
_connections_lock = threading.Lock()
@@ -510,6 +525,9 @@ def _cleanup_connections():
510525
# Module properties
511526
"lowercase",
512527
"native_uuid",
528+
"native_provider",
529+
# Native provider diagnostics
530+
"get_native_provider_info",
513531
]
514532

515533

@@ -583,6 +601,21 @@ def native_uuid(self, value: bool) -> None:
583601
with _settings_lock:
584602
_settings.native_uuid = value
585603

604+
@property
605+
def native_provider(self) -> str:
606+
"""Get the native ODBC provider that will be (or was) loaded.
607+
608+
Honored only when set before the first connection; a later change is
609+
ignored with a warning. The ``MSSQL_PYTHON_NATIVE_PROVIDER`` environment
610+
variable takes precedence over this property.
611+
"""
612+
return ProviderManager.effective()
613+
614+
@native_provider.setter
615+
def native_provider(self, value: Optional[str]) -> None:
616+
"""Set the native ODBC provider selection (or None to clear)."""
617+
ProviderManager.set_property(value)
618+
586619

587620
# Replace the current module with our custom module class
588621
old_module: types.ModuleType = sys.modules[__name__]

‎mssql_python/connection.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
from mssql_python.logging import logger
2929
from mssql_python import ddbc_bindings
3030
from mssql_python.pooling import PoolingManager
31+
from mssql_python.odbc_provider import ProviderManager
3132
from mssql_python.exceptions import (
3233
Warning, # pylint: disable=redefined-builtin
3334
Error,
@@ -730,6 +731,16 @@ def _token_factory():
730731
PoolingManager.enable()
731732
self._pooling = PoolingManager.is_enabled()
732733

734+
# Resolve and freeze the ODBC provider, then hand the selection to the
735+
# native loader so it imports the matching provider package. Done here —
736+
# after every Python-side validation and token acquisition has succeeded
737+
# and immediately before the native driver loads — so a call that fails
738+
# earlier never freezes the selection as a side effect; a later, corrected
739+
# connection can then still choose a different provider without a process
740+
# restart.
741+
_provider = ProviderManager.ensure_available()
742+
ddbc_bindings._set_odbc_provider(_provider)
743+
733744
try:
734745
self._conn = ddbc_bindings.Connection(
735746
self.connection_str,

‎mssql_python/mssql_python.pyi‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ threadsafety: int # 1
3030
# Module Settings - Properties that can be get/set at module level
3131
lowercase: bool # Controls column name case behavior
3232
native_uuid: bool # Controls UUID type handling
33+
native_provider: Optional[str] # Selects the native ODBC provider ('msodbcsql18' or 'mssql-odbc')
3334

3435
# Settings Class
3536
class Settings:
@@ -43,6 +44,7 @@ def get_settings() -> Settings: ...
4344
def setDecimalSeparator(separator: str) -> None: ...
4445
def getDecimalSeparator() -> str: ...
4546
def pooling(max_size: int = 100, idle_timeout: int = 600, enabled: bool = True) -> None: ...
47+
def get_native_provider_info() -> Dict[str, object]: ...
4648
def get_info_constants() -> Dict[str, int]: ...
4749

4850
# Logging Functions

‎mssql_python/odbc_provider.py‎

Lines changed: 251 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,251 @@
1+
"""
2+
Copyright (c) Microsoft Corporation.
3+
Licensed under the MIT license.
4+
Selects which ODBC provider (native driver package) mssql-python loads.
5+
6+
Two providers are supported: ``msodbcsql18`` (the Microsoft ODBC Driver 18,
7+
shipped by ``mssql_python_odbc``) and ``mssql-odbc`` (the Rust driver, shipped
8+
inside ``mssql_py_core`` / the ``mssql-python-rs`` wheel). Selection is process-wide and resolved exactly
9+
once, before the native driver loads, from — in precedence order — the
10+
``MSSQL_PYTHON_NATIVE_PROVIDER`` environment variable, the ``mssql_python.native_provider``
11+
module property, then the release default. An unknown value fails closed rather
12+
than falling back.
13+
"""
14+
15+
import os
16+
import threading
17+
import warnings
18+
import importlib
19+
from typing import Dict, Optional, Tuple
20+
21+
from mssql_python.logging import logger
22+
23+
NATIVE_PROVIDER_ENV_VAR = "MSSQL_PYTHON_NATIVE_PROVIDER"
24+
25+
# Customer-facing provider identifiers.
26+
PROVIDER_MSODBCSQL18 = "msodbcsql18"
27+
PROVIDER_MSSQL_ODBC = "mssql-odbc"
28+
29+
# Phase 1 default. Phase 2 flips this to PROVIDER_MSSQL_ODBC via a documented release.
30+
_DEFAULT_PROVIDER = PROVIDER_MSODBCSQL18
31+
32+
# Provider -> import package that ships its native binaries.
33+
# Keep in sync with ddbc_bindings.cpp's ProviderPackageForId / ProviderDistForId.
34+
_PACKAGE_BY_PROVIDER: Dict[str, str] = {
35+
PROVIDER_MSODBCSQL18: "mssql_python_odbc",
36+
PROVIDER_MSSQL_ODBC: "mssql_py_core",
37+
}
38+
39+
# Provider -> the pip distribution that installs its package (for error hints).
40+
_DIST_BY_PROVIDER: Dict[str, str] = {
41+
PROVIDER_MSODBCSQL18: "mssql-python-odbc",
42+
PROVIDER_MSSQL_ODBC: "mssql-python-rs",
43+
}
44+
45+
46+
def _normalize(value: str) -> str:
47+
"""Return the canonical provider id for ``value`` or raise ``ValueError``.
48+
49+
An unrecognized selection is rejected so a typo fails closed instead of
50+
silently loading the default provider.
51+
"""
52+
canonical = value.strip().lower()
53+
if canonical not in _PACKAGE_BY_PROVIDER:
54+
valid = ", ".join(sorted(_PACKAGE_BY_PROVIDER))
55+
raise ValueError(f"Unknown ODBC provider {value!r}. Valid providers are: {valid}.")
56+
return canonical
57+
58+
59+
class ProviderManager:
60+
"""Process-wide, resolve-once selector for the ODBC provider.
61+
62+
The selection freezes when :meth:`resolve` first runs (at native driver
63+
load). A later change to the module property is ignored with a warning,
64+
mirroring the connection-pool configuration model.
65+
"""
66+
67+
_lock: threading.Lock = threading.Lock()
68+
_property_value: Optional[str] = None
69+
_resolved: Optional[str] = None
70+
_source: Optional[str] = None
71+
72+
@classmethod
73+
def _compute(cls) -> Tuple[str, str]:
74+
"""Apply precedence env var -> module property -> default (lock-free)."""
75+
env_value = os.environ.get(NATIVE_PROVIDER_ENV_VAR)
76+
if env_value and env_value.strip():
77+
return _normalize(env_value), "environment"
78+
if cls._property_value is not None:
79+
return cls._property_value, "property"
80+
return _DEFAULT_PROVIDER, "default"
81+
82+
@classmethod
83+
def set_property(cls, value: Optional[str]) -> None:
84+
"""Set the module-property selection.
85+
86+
Accepts a provider id or ``None`` to clear. A change after the provider
87+
has been resolved is ignored with a warning; the env var still takes
88+
precedence over this value when both are set.
89+
"""
90+
with cls._lock:
91+
canonical = _normalize(value) if value is not None else None
92+
if cls._resolved is not None:
93+
if canonical != cls._resolved:
94+
cls._warn_frozen()
95+
return
96+
cls._property_value = canonical
97+
env_value = os.environ.get(NATIVE_PROVIDER_ENV_VAR)
98+
if canonical is not None and env_value and env_value.strip():
99+
try:
100+
env_provider = _normalize(env_value)
101+
except ValueError:
102+
# Preserve the existing fail-closed error at connection time.
103+
return
104+
if canonical != env_provider:
105+
cls._warn_env_override(canonical, env_provider)
106+
107+
@classmethod
108+
def resolve(cls) -> str:
109+
"""Resolve and freeze the provider, returning its canonical id."""
110+
with cls._lock:
111+
if cls._resolved is None:
112+
cls._resolved, cls._source = cls._compute()
113+
logger.info(
114+
"ODBC provider resolved to '%s' (source=%s)",
115+
cls._resolved,
116+
cls._source,
117+
)
118+
return cls._resolved
119+
120+
@classmethod
121+
def effective(cls) -> str:
122+
"""Return the provider that would be used, without freezing it.
123+
124+
Reports the release default for an invalid selection (e.g. a mistyped
125+
env var) rather than raising - this backs the public getter and
126+
diagnostics, which must stay safe to read at any time. The hard
127+
failure for a bad selection surfaces at :meth:`resolve`/
128+
:meth:`ensure_available` instead.
129+
"""
130+
with cls._lock:
131+
if cls._resolved is not None:
132+
return cls._resolved
133+
try:
134+
provider, _ = cls._compute()
135+
except ValueError:
136+
return _DEFAULT_PROVIDER
137+
return provider
138+
139+
@classmethod
140+
def package_name(cls, provider: Optional[str] = None) -> str:
141+
"""Return the import package that ships ``provider``'s native binaries."""
142+
provider = provider or cls.effective()
143+
return _PACKAGE_BY_PROVIDER[provider]
144+
145+
@classmethod
146+
def ensure_available(cls) -> str:
147+
"""Verify the selected provider's package is installed, then freeze it.
148+
149+
Called before the native driver loads. Fails closed with an actionable
150+
error if the package is missing. The selection is only frozen (via
151+
:meth:`resolve`) once the package has been confirmed importable, so a
152+
failed check here does not permanently lock in a provider that never
153+
actually loaded - a later call can still select a different, installed
154+
provider instead of requiring a process restart.
155+
"""
156+
provider = cls.effective()
157+
package = _PACKAGE_BY_PROVIDER[provider]
158+
try:
159+
importlib.import_module(package)
160+
except ModuleNotFoundError as exc:
161+
if exc.name != package:
162+
# A transitive dependency of an installed package is missing,
163+
# or the package is broken - don't mask it as "not installed".
164+
raise
165+
dist = _DIST_BY_PROVIDER[provider]
166+
raise ImportError(
167+
f"The '{provider}' ODBC provider is selected but its package "
168+
f"'{package}' is not installed. Install it with: pip install {dist}"
169+
) from exc
170+
return cls.resolve()
171+
172+
@classmethod
173+
def is_frozen(cls) -> bool:
174+
"""Whether the provider has been resolved and can no longer change."""
175+
return cls._resolved is not None
176+
177+
@classmethod
178+
def get_info(cls) -> Dict[str, object]:
179+
"""Report the selected provider for diagnostics.
180+
181+
Never raises: an invalid selection is reported via the ``error`` key
182+
(with ``id`` falling back to the default) instead of propagating, so
183+
this stays safe to call at any time, including before a provider is
184+
chosen or resolvable.
185+
"""
186+
with cls._lock:
187+
if cls._resolved is not None:
188+
provider, source, error = cls._resolved, cls._source, None
189+
else:
190+
try:
191+
provider, source = cls._compute()
192+
error = None
193+
except ValueError as exc:
194+
provider, source, error = _DEFAULT_PROVIDER, None, str(exc)
195+
frozen = cls._resolved is not None
196+
197+
version = None
198+
driver_path = None
199+
package = _PACKAGE_BY_PROVIDER[provider]
200+
try:
201+
provider_module = importlib.import_module(package)
202+
version = getattr(provider_module, "__version__", None)
203+
module_file = getattr(provider_module, "__file__", None)
204+
if module_file:
205+
from mssql_python import ddbc_bindings
206+
207+
driver_path = ddbc_bindings._get_odbc_driver_path(
208+
os.path.dirname(os.path.abspath(module_file)), provider
209+
)
210+
except Exception: # pylint: disable=broad-exception-caught
211+
# Diagnostics must remain safe even for a broken provider package.
212+
pass
213+
214+
info: Dict[str, object] = {
215+
"id": provider,
216+
"package": package,
217+
"version": version,
218+
"driver_path": driver_path,
219+
"source": source,
220+
"frozen": frozen,
221+
}
222+
if error is not None:
223+
info["error"] = error
224+
return info
225+
226+
@classmethod
227+
def _warn_env_override(cls, requested: str, effective: str) -> None:
228+
message = (
229+
f"ODBC provider property was set to '{requested}', but "
230+
f"{NATIVE_PROVIDER_ENV_VAR} selects '{effective}' and takes precedence."
231+
)
232+
logger.warning(message)
233+
warnings.warn(message, RuntimeWarning, stacklevel=3)
234+
235+
@classmethod
236+
def _warn_frozen(cls) -> None:
237+
message = (
238+
f"ODBC provider is already loaded as '{cls._resolved}'; ignoring the "
239+
f"change. Select a provider before the first connection, or set the "
240+
f"{NATIVE_PROVIDER_ENV_VAR} environment variable."
241+
)
242+
logger.warning(message)
243+
warnings.warn(message, RuntimeWarning, stacklevel=3)
244+
245+
@classmethod
246+
def _reset_for_testing(cls) -> None:
247+
"""Reset selection state - for testing purposes only."""
248+
with cls._lock:
249+
cls._property_value = None
250+
cls._resolved = None
251+
cls._source = None

‎mssql_python/pooling.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,10 @@ def enable(cls, max_size: int = 100, idle_timeout: int = 600) -> None:
6262
max_size,
6363
idle_timeout,
6464
)
65+
# enable_pooling() only configures the connection-pool manager; it
66+
# does not load the native driver, so it must not resolve/freeze
67+
# the ODBC provider (an explicit pooling() before selecting a
68+
# provider would otherwise lock in the default prematurely).
6569
ddbc_bindings.enable_pooling(max_size, idle_timeout)
6670
cls._config["max_size"] = max_size
6771
cls._config["idle_timeout"] = idle_timeout

0 commit comments

Comments
 (0)