[WS2][Attention] H100 validation and reproduction guide for #235 PR1-PR8

[inaniloquentee]

Status

Strict operator-level H100 acceptance passed at commit 8309b77. Training and rollout use rlkernel.attention.deterministic_core.v1; 2-, 4-, and 8-GPU runs produced bitwise-equal Out, LSE, dQ, dK, and dV with every max-abs value equal to 0.0.

This issue is the reproducible GPU-validation record for the eight Attention PRs in #235. It covers the Qwen3-style GQA shape (Hq=32, Hkv=8, D=128), BF16 final outputs, FP32 accumulation/merge state, CP=2, and TP=1/2. It does not claim an end-to-end Megatron/vLLM checkpoint-level active-token dlogp result.

H100 result

Numerical scope

PR stack

All PRs target test. Record the exact fetched head for every rerun; do not copy stale SHAs from this issue.

Order	PR	Owned validation boundary
1	#236	Attention contract, TP/CP ownership, RoPE/cache metadata, strict dispatch
2	#253	Single-GPU full/chunked/paged-KV attribution
3	#238	Deterministic CP reference and fixed logical (Out, LSE) merge
4	#284	Prefill/chunked-prefill backward reference (dQ/dK/dV)
5	#260	Decode KV-cache replay and cache/position/RoPE identity
6	#285	Rank-aware aggregate gate and machine-readable artifacts
7	#263	TP=2/CP=2 engine binding and provenance; not an engine execution claim
8	#279	Shared deterministic core, P2P NCCL reference, self-owned AG/RS, paged-KV adapter

Logical dependencies remain: #236/#238 before #263; #238 before #284; #284 before #285; #253 before #260; #253/#260 before #279.

Clean H100 setup

Use a CUDA 12.4 devel image. The validated image family is:

runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04

Keep the checkout, Conda environment, caches, and artifacts under a user-owned ljj directory. Do not install into (base) or another user's checkout.

export RLK_ROOT="${RLK_ROOT:-$HOME/workspace/ljj}"
export RLK_REPO="${RLK_REPO:-$RLK_ROOT/RL-Kernel}"
export RLK_ENV="${RLK_ENV:-$RLK_ROOT/.conda/envs/rlk-attention-py311}"
export RLK_ARTIFACTS="${RLK_ARTIFACTS:-$RLK_ROOT/results/issue294}"
mkdir -p "$RLK_ROOT" "$RLK_ARTIFACTS"

nvidia-smi -L
nvcc --version

if [ ! -d "$RLK_REPO/.git" ]; then
  git clone https://github.com/RL-Align/RL-Kernel.git "$RLK_REPO"
fi
cd "$RLK_REPO"
[ "$(stat -c %U .)" = "$(id -un)" ] || { echo "checkout is not user-owned"; exit 1; }

source "$(conda info --base)/etc/profile.d/conda.sh"
if [ ! -x "$RLK_ENV/bin/python" ]; then
  conda create -p "$RLK_ENV" python=3.11 -y
fi
conda activate "$RLK_ENV"

NVCC_PATH="$(readlink -f "$(command -v nvcc)")"
export CUDA_HOME="$(dirname "$(dirname "$NVCC_PATH")")"
export TORCH_CUDA_ARCH_LIST="9.0+PTX"
export FORCE_CUDA=1
export MAX_JOBS=8

nvidia-smi -L must show eight H100s and nvcc --version must show CUDA 12.4. If the provider image is runtime-only or resolves a different compiler, recreate the instance with the devel image instead of changing a shared host.

Install the pinned core stack

cd "$RLK_REPO"
python -m pip install --upgrade pip setuptools wheel ninja packaging
python -m pip install --no-cache-dir torch==2.4.1 --index-url https://download.pytorch.org/whl/cu124
python -m pip install --no-cache-dir numpy tabulate pytest

for pr in 236 253 238 284 260 285 263 279; do
  git fetch origin "pull/${pr}/head:refs/remotes/origin/pr/${pr}"
done
git switch --detach 8309b77

export TORCH_LIB="$(python -c 'import os,torch; print(os.path.join(os.path.dirname(torch.__file__), "lib"))')"
export LD_LIBRARY_PATH="$TORCH_LIB:$CUDA_HOME/lib64:$CUDA_HOME/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
export LIBRARY_PATH="$CUDA_HOME/lib64:$CUDA_HOME/lib:$CUDA_HOME/lib64/stubs${LIBRARY_PATH:+:$LIBRARY_PATH}"
python -m pip install --no-build-isolation --no-deps -e .
python -m pip check

python - <<'PY'
import torch
from rl_engine import _C

assert torch.__version__.startswith("2.4.1")
assert torch.version.cuda == "12.4"
assert torch.cuda.is_available() and torch.cuda.device_count() >= 8
assert torch.distributed.is_nccl_available()
assert all(torch.cuda.get_device_capability(i) == (9, 0) for i in range(8))
print("environment OK")
PY

FlashInfer, Transformer Engine, Megatron, vLLM, and a Qwen3 checkpoint are optional engine-integration dependencies, not prerequisites for this strict operator gate. Keep them in a separate compatible engine environment if tested; never upgrade the pinned Torch stack silently.

Test order

1. Focused regression

cd "$RLK_REPO"
git switch --detach 8309b77
python -m pytest tests/test_flashinfer_pr7_attention.py -q
# expected: 54 passed, 2 skipped

git switch --detach refs/remotes/origin/pr/285
python -m pytest tests/test_ws2_attention_gpu_acceptance.py \
  tests/test_ws2_cp_attention_drift_benchmark.py -q
# expected: 22 passed

git switch --detach 8309b77

2. Retained P2P NCCL reference

CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc-per-node=2 \
  scripts/ws2_p2p_nccl_attention_reference_check.py \
  --transport p2p_nccl_reference --repeats 3 \
  --output "$RLK_ARTIFACTS/p2p-nccl-reference-2rank.json"

This checks transport semantics and fixed global_block_index merge order. It is not the production strict arithmetic path.

3. Strict shared-core AG/RS matrix

CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc-per-node=2 \
  scripts/ws2_p2p_nccl_attention_reference_check.py \
  --transport cuda_ag_rs --strict-shared-core --repeats 3 \
  --output "$RLK_ARTIFACTS/strict-shared-core-agrs-2rank-8309b77.json"

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --standalone --nproc-per-node=4 \
  scripts/ws2_p2p_nccl_attention_reference_check.py \
  --transport cuda_ag_rs --strict-shared-core --repeats 3 \
  --output "$RLK_ARTIFACTS/strict-shared-core-agrs-tp2-cp2-4rank-8309b77.json"

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc-per-node=8 \
  scripts/ws2_p2p_nccl_attention_reference_check.py \
  --transport cuda_ag_rs --strict-shared-core --repeats 3 \
  --output "$RLK_ARTIFACTS/strict-shared-core-agrs-tp2-cp2-replica2-8rank-8309b77.json"

Acceptance

Each required JSON must report:

• the exact commit, world size, TP/CP topology, replica index, dtype, and logical block manifest;
• core_id=rlkernel.attention.deterministic_core.v1;
• Split-KV disabled, native_attention_arithmetic=false, and fallback=false;
• communication_autograd=true and global_failure_count=0;
• bitwise-equal Out, attention-domain LSE, dQ, dK, and dV with max abs 0.0;
• repeat-run Out/LSE bitwise equality on every rank.

Artifact	Topology	Result
strict-shared-core-agrs-2rank-8309b77.json	TP=1, CP=2	Passed
strict-shared-core-agrs-tp2-cp2-4rank-8309b77.json	TP=2, CP=2	Passed
strict-shared-core-agrs-tp2-cp2-replica2-8rank-8309b77.json	TP=2, CP=2, two replicas	Passed

Boundary

The strict operator-level Attention gate is complete for the tested shapes and topologies. The following remain separate claims and must not be inferred from these artifacts:

• end-to-end Megatron/vLLM Qwen3 checkpoint replay and active-token model-level dlogp;
• native TE/FlashInfer Attention arithmetic or auto/fixed Split-K execution;
• performance acceptance for communication/computation overlap;
• decode backward or a production fused backward claim.

Related: #235, #108.