Skip to content

Expand fbcode buck test coverage for arm backend (58→162 test targets) (#18817) #18817

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
digantdesai wants to merge 2 commits into
base: main
Choose a base branch
from
+142 −4
Open

Expand fbcode buck test coverage for arm backend (58→162 test targets) (#18817) #18817

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7825ffe
Add float scalar input support
digantdesai Apr 16, 2026
4141453
Expand fbcode buck test coverage for arm backend (58→162 test targets…
digantdesai Apr 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion backends/arm/operators/op_tosa_conv3d.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ class Conv3dVisitor(Conv2dVisitor):
target = "tosa.CONV3D.default"

def _get_tosa_op(self):
import serializer.tosa_serializer as ts # type: ignore
import tosa_serializer as ts

return ts.Op.CONV3D
Comment on lines 17 to 20
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR description says it only updates targets.bzl to expand test coverage, but this file also changes runtime behavior by switching the Conv3D lowering import path. Please reflect this in the PR description (or move it into a separate change) so reviewers understand the additional functional impact.

Copilot uses AI. Check for mistakes.

Expand Down
140 changes: 137 additions & 3 deletions backends/arm/test/targets.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,29 +16,133 @@ def define_arm_tests():

# Operators
test_files += [
"ops/test_abs.py",
"ops/test_add.py",
"ops/test_addmm.py",
"ops/test_amax.py",
Comment on lines 17 to +22
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description/title claims this adds “all available” tests from ops/, quantizer/, and misc/, but this target list still omits many existing tests in those directories (e.g., ops/test_acos.py, ops/test_any.py, misc/test_high_rank_permute_view_invariants.py, etc.). Either update the description to reflect that this is a curated subset, or switch to native.glob(["ops/test_.py"], ...) / native.glob(["misc/test_.py"], ...) with an explicit exclusion list so coverage doesn’t silently drift as new tests are added.

Copilot uses AI. Check for mistakes.
"ops/test_amin.py",
Comment on lines 17 to +23
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says it adds all available tests from ops/, quantizer/, and misc/, but there are existing test files in backends/arm/test/ops/ (e.g. ops/test_acos.py, ops/test_arange.py, etc.) that are not included or called out as intentionally excluded. Either add the remaining tests (with comments for known exclusions) or adjust the description to match what’s actually being covered.

Copilot uses AI. Check for mistakes.
"ops/test_avg_pool2d.py",
# "ops/test_batch_norm.py", # T000000 FVP reshape fails: dtype mismatch in output parsing (expected [3], got 12 elements)
"ops/test_bitwise.py",
"ops/test_bmm.py",
"ops/test_cat.py",
"ops/test_ceil.py",
"ops/test_clamp.py",
"ops/test_clone.py",
"ops/test_conv1d.py",
"ops/test_conv2d.py",
"ops/test_conv3d.py",
"ops/test_cos.py",
# "ops/test_depthwise_conv.py", # T000000 Cross-file imports from test_conv1d/test_conv2d break Buck target listing
# "ops/test_div.py", # 15 failures: NoneType input in bundled_program serialization (pre-existing bug)
"ops/test_elu.py",
"ops/test_embedding.py",
"ops/test_eq.py",
"ops/test_erf.py",
"ops/test_exp.py",
"ops/test_expand.py",
"ops/test_floor.py",
"ops/test_full.py",
"ops/test_ge.py",
"ops/test_group_norm.py",
"ops/test_gt.py",
"ops/test_hardswish.py",
"ops/test_hardtanh.py",
# "ops/test_layer_norm.py", # 1 failure: 16a8w u55 quantization issue
"ops/test_le.py",
"ops/test_leaky_relu.py",
"ops/test_linear.py",
"ops/test_log10.py",
"ops/test_log.py",
"ops/test_logical.py",
"ops/test_lt.py",
# matmul: Vela compilation fails with 'Non-passthrough operation'
# for int16 matmul operations, xfail interacts incorrectly with
# XfailIfNoCorstone.
# "ops/test_matmul.py",
"ops/test_max_pool1d.py",
"ops/test_max_pool.py",
"ops/test_mean_dim.py",
"ops/test_maximum.py",
"ops/test_minimum.py",
"ops/test_mm.py",
"ops/test_mul.py",
"ops/test_ne.py",
"ops/test_neg.py",
"ops/test_ones.py",
"ops/test_permute.py",
"ops/test_pixel_shuffling.py",
"ops/test_pow.py",
"ops/test_reciprocal.py",
"ops/test_relu.py",
"ops/test_remainder.py",
"ops/test_repeat.py",
"ops/test_round.py",
# rshift: U55 FVP output dtype mismatch for rshift operations,
# xfail interacts incorrectly with XfailIfNoCorstone.
# "ops/test_rshift.py",
"ops/test_rsqrt.py",
"ops/test_slice.py",
"ops/test_rsub.py",
"ops/test_scalar_tensor.py",
"ops/test_scalars.py",
"ops/test_sdpa.py",
"ops/test_select.py",
"ops/test_select_scatter.py",
"ops/test_sigmoid.py",
"ops/test_sigmoid_32bit.py",
"ops/test_sign.py",
# silu: U55 numerical mismatch for inplace SiLU, xfail interacts
# incorrectly with XfailIfNoCorstone.
# "ops/test_silu.py",
"ops/test_sin.py",
"ops/test_sinh.py",
"ops/test_slice.py",
# slice_scatter: U55/U85 FVP failures for int8 and end_none cases,
# xfail interacts incorrectly with XfailIfNoCorstone.
# "ops/test_slice_scatter.py",
"ops/test_softmax.py",
"ops/test_split.py",
"ops/test_sqrt.py",
"ops/test_squeeze.py",
"ops/test_stack.py",
"ops/test_sub.py",
# sum: xfail markers interact incorrectly with XfailIfNoCorstone
# when test inputs contain None (NoneType not supported in
# bundled_program serialization).
# "ops/test_sum.py",
"ops/test_t_copy.py",
"ops/test_tan.py",
"ops/test_tanh.py",
"ops/test_view.py",
"ops/test_cos.py",
"ops/test_to_copy.py",
# transpose_conv2d: xfail markers for per-channel quantization
# interact incorrectly with XfailIfNoCorstone when tests fail
# during quantization (before reaching FVP execution).
# "ops/test_transpose_conv2d.py",
"ops/test_transpose_copy.py",
"ops/test_tril.py",
# "ops/test_unary_combos.py", # 1 failure: NegAdd u55_INT
"ops/test_unbind.py",
"ops/test_unflatten.py",
"ops/test_unfold_copy.py",
"ops/test_unsqueeze.py",
"ops/test_upsample_bilinear2d.py",
"ops/test_upsample_nearest2d.py",
"ops/test_var.py",
"ops/test_view.py",
"ops/test_where.py",
"ops/test_while.py",
"ops/test_zeros.py",
]

# Quantization
test_files += [
"quantizer/test_conv_relu_fusing.py",
"quantizer/test_generic_annotater.py",
"quantizer/test_partial_quantization.py",
"quantizer/test_preserve_kwargs.py",
# "quantizer/test_selective_quantization.py", # needs torchvision (not in deps)
"quantizer/test_set_module_name.py",
"quantizer/test_tosa_quantizer_validate.py",
]

# Misc tests
Expand All @@ -48,15 +152,44 @@ def define_arm_tests():
"misc/test_pass_pipeline_config.py",
"misc/test_tosa_spec.py",
"misc/test_bn_relu_folding_qat.py",
"misc/test_call_operator_submodule.py",
"misc/test_compile_spec.py",
Comment on lines 154 to +156
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

misc/test_compile_spec.py is listed twice in test_files (once earlier in the misc section and again here). Because target names are derived from the basename, this will attempt to define the same Buck target (compile_spec) twice and fail. Remove the duplicate or de-duplicate test_files before the loop.

Copilot uses AI. Check for mistakes.
"misc/test_const_shape.py",
"misc/test_conv_relu_residual_add.py",
"misc/test_count_tosa_ops.py",
"misc/test_custom_partition.py",
"misc/test_debug_feats.py",
"misc/test_debug_hook.py",
"misc/test_post_quant_device_switch.py",
# "misc/test_dim_order.py", (TODO - T238390249)
]

# Deprecation tests
test_files += [
"misc/test_dw_convs_with_shared_weights.py",
"misc/test_extract_io_params_tosa.py",
# "misc/test_int64.py", # 5 failures: int64 overflow/chain handling issues
"misc/test_lifted_tensor.py",
"misc/test_mixed_fp_bf16_partition.py",
Comment on lines +170 to +173
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above for this exclusion: executorch.backends.test.harness.stages has a Buck target (//executorch/backends/test/harness:tester) and is already pulled in transitively via //executorch/backends/arm/test:arm_tester. If this test is disabled due to failures/flake, please update the comment accordingly; otherwise consider enabling it to match the stated goal of expanding coverage.

Copilot uses AI. Check for mistakes.
"misc/test_mixed_type_lowering.py",
# "misc/test_model_evaluator.py", # needs executorch.backends.arm.util (no BUCK target)
"misc/test_multiple_delegates.py",
"misc/test_multiple_outputs.py",
"misc/test_non_persistent_buffers.py",
"deprecation/test_arm_compile_spec_deprecation.py",
"misc/test_partition_decomposed_quantized_ops.py",
"misc/test_pass_pipeline_config.py",
"misc/test_pass_required_order.py",
"misc/test_qat_training_loop.py",
Comment on lines 179 to +183
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

misc/test_pass_pipeline_config.py is already included in the earlier "Misc tests" list, and is added again here. Since test_name is based on the basename, this will create a duplicate Buck rule name (pass_pipeline_config) and break the build. Remove one occurrence or ensure test_files is unique before generating targets.

Copilot uses AI. Check for mistakes.
"misc/test_quant_custom_meta.py",
# "misc/test_rescale_range.py", # 3 failures: zero-point and rescale range validation
# "misc/test_runner_utils.py", # name collides with runner_utils library in BUCK
"misc/test_save_exported_model.py",
# "misc/test_shared_qspecs.py", # needs executorch.backends.cortex_m.test.tester (no BUCK target)
"misc/test_tosa_dialect_conv2d.py",
"misc/test_tosa_dialect_dw_conv2d.py",
"misc/test_tosa_dialect_shape_ops.py",
"misc/test_tosa_spec.py",
]
Comment on lines +189 to 193
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

misc/test_tosa_spec.py is already included in the "Misc tests" list and is added again here. This results in defining the same Buck target name (tosa_spec) twice. Remove the duplicate entry or de-duplicate test_files before iterating.

Copilot uses AI. Check for mistakes.

TESTS = {}
Expand Down Expand Up @@ -96,6 +229,7 @@ def define_arm_tests():
"//executorch/backends/arm/tosa:partitioner",
"//executorch/backends/arm:vgf",
"//executorch/backends/test:graph_builder",
"//executorch/backends/test/harness:tester",
"//executorch/exir:lib",
"fbsource//third-party/pypi/pytest:pytest",
"fbsource//third-party/pypi/parameterized:parameterized",
Expand Down
4 changes: 4 additions & 0 deletions extension/pybindings/pybindings.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -807,6 +807,8 @@ struct PyModule final {
cpp_inputs.push_back(EValue(py::cast<bool>(python_input)));
} else if (py::isinstance<py::int_>(python_input)) {
cpp_inputs.push_back(EValue(py::cast<int64_t>(python_input)));
} else if (py::isinstance<py::float_>(python_input)) {
cpp_inputs.push_back(EValue(py::cast<double>(python_input)));
} else {
Comment on lines +810 to 812
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR’s description is focused on expanding Arm backend Buck test targets, but this hunk also changes the Python bindings’ accepted input types (python float → EValue(double)). Please update the PR description to mention this behavior change, or split it into a separate PR if it’s unrelated to the test target expansion.

Copilot uses AI. Check for mistakes.
throw std::runtime_error(
"Unsupported python type " + type_str +
Expand Down Expand Up @@ -1135,6 +1137,8 @@ struct PyMethod final {
cpp_inputs.push_back(EValue(py::cast<bool>(python_input)));
} else if (py::isinstance<py::int_>(python_input)) {
cpp_inputs.push_back(EValue(py::cast<int64_t>(python_input)));
} else if (py::isinstance<py::float_>(python_input)) {
cpp_inputs.push_back(EValue(py::cast<double>(python_input)));
} else {
throw std::runtime_error(
"Unsupported python type " + type_str +
Expand Down
Loading