feat: add Gateway API foundation (GEP-1748) for Zero-Public-Exposure Ingress by rchincha · Pull Request #400 · Azure/fleet-networking

Ramkumar Chinchani (rchincha) · 2026-08-18T02:13:11Z

What type of PR is this?

feature

What this PR does / why we need it:

Implements the GEP-1748 using Azure AFD as the L7 ingress to achieve Zero-Public-Exposure Ingress solution for highly regulated customer environments.

Which issue(s) this PR fixes:

Fixes #

Requirements:

run make reviewable for basic local test
Read and followed fleet-networking's Code of conduct.
uses conventional commit messages
includes documentation
adds unit tests

How has this code been tested

Special notes for your reviewer

Replace the removed Gateway API generated specification URL with the version-pinned v1.2.1 type definitions used by this implementation. This fixes the markdown-link-check failure in PR Azure#400 while keeping the documentation reference aligned with the module version in go.mod. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d8af5d6-9aba-40a2-8efe-9bf09675667b

codecov · 2026-08-18T02:42:16Z

Codecov Report

❌ Patch coverage is 86.60674% with 298 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.98%. Comparing base (76f455e) to head (05ea818).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/controllers/hub/gatewayresolver/route.go	70.83%	35 Missing and 14 partials ⚠️
pkg/controllers/hub/httproute/controller.go	65.09%	28 Missing and 9 partials ⚠️
cmd/member-net-controller-manager/main.go	32.43%	22 Missing and 3 partials ⚠️
pkg/providers/azure/frontdoor/backend_provision.go	87.56%	12 Missing and 12 partials ⚠️
pkg/controllers/hub/gateway/controller.go	89.84%	12 Missing and 8 partials ⚠️
pkg/controllers/hub/gateway/http_provisioning.go	82.10%	9 Missing and 8 partials ⚠️
pkg/controllers/hub/gatewayclass/controller.go	62.79%	12 Missing and 4 partials ⚠️
cmd/hub-gateway-controller-manager/main.go	85.98%	11 Missing and 4 partials ⚠️
pkg/controllers/hub/gateway/ownership.go	83.33%	6 Missing and 6 partials ⚠️
...kg/controllers/hub/gateway/reference_withdrawal.go	89.21%	6 Missing and 5 partials ⚠️
... and 15 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #400      +/-   ##
==========================================
+ Coverage   67.06%   76.98%   +9.92%     
==========================================
  Files          34       73      +39     
  Lines        3859     7165    +3306     
==========================================
+ Hits         2588     5516    +2928     
- Misses       1090     1301     +211     
- Partials      181      348     +167

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Define a fresh multi-cluster global ingress architecture that uses GatewayClass, Gateway, and HTTPRoute while preserving the existing networking.fleet.azure.com/v1alpha1 ServiceImport API. Align the Fleet hub configuration-cluster model with GKE multi-cluster Gateway guidance while retaining Fleet-specific ServiceImport support. Document environment prerequisites, ownership, status, fail-static behavior, repository placement, feasibility, and the phased tests-first implementation plan. Define optional annotations for Azure Front Door SKU, WAF attachment, health probes, host headers, and public or Private Link connectivity. Clarify that unknown or invalid explicit annotations must fail rather than silently falling back to defaults. Add separate operational guides for public LoadBalancer backends and AFD Premium Private Link to internal LoadBalancer and PLS backends. Identify the actor and placement for every hub, member-cluster, Azure, WAF, PLS approval, DNS, traffic-weight, and cleanup action. Document that the private AFD plus optional WAF plus PLS topology is intended to support SFI-NS253, while public member LoadBalancer origins are not that topology. Do not claim compliance until Private Link reconciliation, enforcement, and end-to-end evidence are complete. Pin references to the Gateway API v1.2.1 type definitions used by the implementation and keep the public documentation free of internal URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d8af5d6-9aba-40a2-8efe-9bf09675667b

Pin Gateway API v1.2.1 to the repository's Kubernetes v0.31.1 baseline and register Gateway, HTTPRoute, ReferenceGrant, and the unchanged Fleet ServiceImport types in a separate hub Gateway manager. Add controller-wide Azure configuration validation, feature gating, health and readiness endpoints, leader election, structured klog events, and repository build targets. The foundation intentionally registers no Azure reconcilers yet. Add typed annotation parsing for AFD SKU, optional WAF attachment, origin connectivity, health probes, and host headers. Reject unknown reserved keys, malformed values, unsupported resource-group overrides, and incompatible Private Link settings. Add a provider-neutral normalized Gateway model with deterministic ordering and separate validation for Gateway API backend weights and Fleet ServiceExport per-cluster weights. Focused tests cover manager startup and failure paths, annotations, and the normalized model. Local race testing is deferred to CI because this Windows host does not have gcc installed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d8af5d6-9aba-40a2-8efe-9bf09675667b

Install the selected Gateway API CRDs in the E2E hub and verify Gateway, HTTPRoute, ReferenceGrant, and Fleet ServiceImport interoperability. Document the managed-hub prerequisite and upgrade path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Separate baseline networking, Traffic Manager, and current Gateway API contract coverage. Document the additional wiring required for future Azure Front Door reconciliation tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ramkumar Chinchani (rchincha) · 2026-10-24T04:06:29Z

https://github.com/rchincha/fleet-networking/blob/rchinchani/gep-1748-gateway-api/docs/howtos/gateway-api-afd-configuration.md

Controllers for the AFD private HTTP feature

Cluster	Controller manager	Relevant controllers / responsibility
Hub	`hub-gateway-controller-manager` (new, separate deployment)	`GatewayClass`, `Gateway` and `HTTPRoute` reconcilers. Manages AFD provisioning, approval observation, readiness, withdrawal and cleanup.
Hub	`hub-net-controller-manager` (existing)	Existing `InternalServiceExport` / `ServiceImport` processing supporting backend discovery and transport.
Each member	`member-net-controller-manager` (existing, extended)	Its `serviceexport-controller` discovers internal load balancer / Private Link Service metadata and publishes it through `InternalServiceExport`. No new member controller deployment.

Both feature gates default to off:

Hub: --enable-afd=true
Member: --enable-private-link-origin-discovery=true

These are implementation/component names—not confirmation that they are currently installed in your clusters.

Category	Signed Commit ID	Files Changed
Docs	`f71c000`	12
Controllers, tests and telemetry	`962c2d9`	93
Build artifacts, packaging and CI	`765444e`	16
Live-test preparation	`05ea818`	11
Total	4 commits	132

Wei Weng (weng271190436) · 2026-10-28T16:00:55Z


 ```bash
 export AZURE_RESOURCE_GROUP=<YOUR-RESOURCE-GROUP-NAME>
 export AZURE_SUBSCRIPTION_ID=<YOUR-SUBSCRIPTION-ID>
-# Available values for AZURE_NETWORK_SETTING are shared-vnet, dynamic-ip-allocation and peered-vnet,
-# and detailed explanations for each network setting are provided in the scripts under folder "test/scripts".
 export AZURE_NETWORK_SETTING=shared-vnet
 export ENABLE_TRAFFIC_MANAGER=true


Can the new AFD tests be added in a way that you just do

export AZURE_SUBSCRIPTION_ID=""
export AZURE_RESOURCE_GROUP=""
export AZURE_NETWORK_SETTING=shared-vnet
export ENABLE_TRAFFIC_MANAGER=false
export ENABLE_AZURE_FRONT_DOOR=true

and then

make e2e-setup
make e2e-tests
make e2e-cleanup

will test the existing scenarios and the new AFD scenario end to end?

This will help people validating that their new changes don't break the new AFD scenario

Wei Weng (weng271190436) · 2026-10-28T19:00:37Z

+	switch state := ptr.Deref(properties.DeploymentStatus, ""); state {
+	case armcdn.DeploymentStatusInProgress, armcdn.DeploymentStatusNotStarted:
+		return "", false, nil
+	case armcdn.DeploymentStatusSucceeded:


I tried to run the test but the disabled endpoint never became DeploymentStatusSucceeded so everything after profile and endpoint are not getting created, such as route, origin groups, origin etc

Did this work for you?

Also wondering if Deployment Status Succeeded is a necessary gate. Someone on Microsoft Learn said no: https://learn.microsoft.com/en-us/answers/questions/5642906/azure-front-door-endpoint-deployment-stuck-at-nots

Gateway private HTTP proof [It] proves controller-owned private delivery, observed probes, withdrawal and cleanup [afd-private-http-proof] /home/weiweng/fleet-networking/test/e2e/gateway_private_http_test.go:45 [FAILED] Timed out after 600.000s. Expected success, but got an error: <*errors.errorString | 0x230d65eb4530>: origin fleet-origin-e03c32ee1716ecbdb5a30cda2e9fecb3 is Missing, not an observed manual approval request { s: "origin fleet-origin-e03c32ee1716ecbdb5a30cda2e9fecb3 is Missing, not an observed manual approval request", } In [It] at: /home/weiweng/fleet-networking/test/e2e/gateway_private_http_test.go:116 @ 10/28/26 18:22:08.582

Also the Gateway status can be more specific to give more visibility to the users

For above issue, the condition is Programmed: False, and the message is "AFD private HTTP graph: Pending"

It seems that throughout the whole process, the message is either "AFD private HTTP graph: Pending" or "AFD private HTTP graph: Staging"

I wonder if the message can indicate that endpoint deployment status is "Not started"/"in progress" so we are waiting?

Consolidate feature design, implementation plans, decisions, operator guidance, chart documentation and test runbooks. Document asynchronous Private Link approval, the ATM comparison and deferred live verification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19936d92-7c14-4bee-b1fd-70f2b4bc340d

Implement private-origin discovery and transport, Gateway API resolution and authorization, owned AFD lifecycle, full-graph readiness, withdrawal and cleanup. Include required API schemas and SDK dependencies, unit and local API-server tests, events and metrics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19936d92-7c14-4bee-b1fd-70f2b4bc340d

Add the separate Gateway image, default-disabled Helm packaging, member configuration, build/publication targets and Helm contracts. Verify real native amd64/arm64 images in CI, restricted nonroot entrypoints, default-disabled HTTP probes, loopback-only registry export/pull and both Linux architectures in the combined MCR manifest. Current amd64 image build/runtime/local-registry proof, full repository-pinned lint, controller regressions, Helm contracts and workflow/ShellCheck validation pass locally. Native arm64 hosted execution and production publication remain unrun; no Azure, cluster or production-registry operations were performed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19936d92-7c14-4bee-b1fd-70f2b4bc340d

Extend the existing Gateway E2E framework and sample workload for opt-in installed-controller private HTTP proof. Follow ATM intent/status/actual-ARM/cleanup patterns while adding private-only frontend, asynchronous approval, fresh HTTP identity/Host, probe-log and withdrawal assertions. Live execution remains deferred; existing ATM tests and E2E workflow are unchanged. Allow 45-minute AFD convergence and cloud-cleanup waits with separate enclosing spec/cleanup deadlines and slower ARM polling. Preserve all success criteria and short controller request budgets; add offline timing-budget regression coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19936d92-7c14-4bee-b1fd-70f2b4bc340d

Wei Weng (weng271190436) · 2026-11-02T21:30:05Z

+// +kubebuilder:rbac:groups="",resources=events,verbs=create;patch
+
+// Reconcile refreshes generation-aware Gateway and listener status.
+func (r *Reconciler) Reconcile(ctx context.Context, request ctrl.Request) (ctrl.Result, error) {


Can you add more progress logs throughout the reconciliation process? Right now the program condition message gives good information, but container logs are important because they will be sent to log systems and will be searchable for debugging without logging onto the k8s cluster

Wei Weng (weng271190436) · 2026-11-02T21:43:25Z

+		return
+	}
+	reason := "AFDDisabledGraphStagingFailed"
+	message := "AFD disabled-graph staging call failed; inspect controller logs. No new ingress readiness is established."


Can the event messages give a concise cause instead of directing users back to logs?

Wei Weng (weng271190436) · 2026-11-02T21:47:13Z

+				return err
+			}
+			if err := verifyInventoryDeployment(route.Properties.ProvisioningState, route.Properties.DeploymentStatus, bootstrap); err != nil {
+				return fmt.Errorf("listed AFD route is not converged: %w", err)


Can we distinguish transient non-convergence from actual provisioning failures? A route can still be converging but it logs an ERROR with a stack trace. Should it be an INFO progress log?

2026-11-02T21:26:28Z ERROR Reconciler error {"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"private-http","namespace":"my-ns-vmsqu"}, "namespace": "my-ns-vmsqu", "name": "private-http", "reconcileID": "7fa50b5c-5566-4b04-8d13-f566236ff968", "error": "listed AFD route is not converged: AFD provisioning changed during inventory verification\nlisted AFD route is not converged: AFD provisioning changed during inventory verification"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:224

Wei Weng (weng271190436) · 2026-11-03T14:23:10Z

+    steps:
+      - uses: actions/checkout@v7
+
+      - name: Verify existing image delivery contracts


this test looks brittle. I am thinking if it is better not to add this.

Wei Weng (weng271190436) · 2026-11-03T15:28:32Z

+done
+
+for target in "${targets[@]}"; do
+  az role assignment create --subscription "$AZURE_SUBSCRIPTION_ID" --role "$reader" \


For E2E, could we prepare hub Reader access on the verified member node resource groups during setup instead of assigning Reader to each newly created PLS during the test? This would allow the existing post-setup propagation window ("Wait for role assignments to be effective" step in CI) to apply and simplify repeated runs. And we don't need afd-pls-reader.sh anymore

or we can use an annotation on the v1.Service to specify the PLS resource group, it can be the same as the resource group where clusters, vnet, AFD are in, such that we just need to role assign Reader on that RG.

Wei Weng (weng271190436) · 2026-11-03T15:40:26Z

+kind: Deployment
+metadata:
+  name: {{ include "hub-gateway-controller-manager.fullname" . }}
+  namespace: {{ .Release.Namespace }}


use fleetSystemNamespace like the existing charts?

Wei Weng (weng271190436) · 2026-11-03T15:44:02Z

@@ -0,0 +1,6 @@
+apiVersion: v2
+name: hub-gateway-controller-manager
+description: Independent Fleet Gateway observer and owned AFD cleanup controller


This controller owns the entire AFD resources lifecycle?

Wei Weng (weng271190436) · 2026-11-03T15:50:18Z

+
+// ValidateListener checks the initial private HTTP-preview listener subset.
+func ValidateListener(listener gatewayv1.Listener) error {
+	if listener.Protocol != gatewayv1.HTTPProtocolType || listener.Port != 80 {


I assume the requirement is to support HTTPS on AFD endpoint? That will be handled later?

Wei Weng (weng271190436) · 2026-11-03T15:51:40Z

+			},
+			PatternsToMatch:     []*string{ptr.To(httpPreviewPattern)},
+			SupportedProtocols:  []*armcdn.AFDEndpointProtocols{ptr.To(armcdn.AFDEndpointProtocolsHTTP)},
+			ForwardingProtocol:  ptr.To(armcdn.ForwardingProtocolHTTPOnly),


Does this need to be HTTPS as well for production? Not necessarily for this PR though.

Wei Weng (weng271190436) · 2026-11-03T16:47:13Z

+			return validatePrivateHTTPBlocked(reply, service.Namespace)
+		}
+		Eventually(ctx, blocked, privateHTTPConvergenceTimeout, privateHTTPPoll).Should(Succeed())
+		Consistently(ctx, blocked, time.Minute, privateHTTPPoll).Should(Succeed(), "Network errors are not affirmative withdrawal evidence")


Gateway private HTTP proof [It] proves controller-owned private delivery, observed probes, withdrawal and cleanup [afd-private-http-proof] /home/weiweng/fleet-networking/test/e2e/gateway_private_http_test.go:55 [FAILED] Failed after 5.115s. Network errors are not affirmative withdrawal evidence Expected success, but got an error: <*errors.errorString | 0x3571cf1325e0>: no affirmative AFD withdrawal response (HTTP 200) { s: "no affirmative AFD withdrawal response (HTTP 200)", } In [It] at: /home/weiweng/fleet-networking/test/e2e/gateway_private_http_test.go:365 @ 11/03/26 16:05:00.934

I saw this when I ran it. Could it be that after private HTTP blocked one time, AFD can still return 200 afterwards because the removal was not propagated to all edges?

maybe this can require a sustained blocked interval rather than requiring consistently blocked after first blocked?

Ramkumar Chinchani (rchincha) changed the title ~~Rchinchani/gep 1748 gateway api~~ feat: add Gateway API controller foundation (GEP-1748) Aug 18, 2026

Ramkumar Chinchani (rchincha) force-pushed the rchinchani/gep-1748-gateway-api branch from 02ad067 to 7bf9918 Compare August 18, 2026 02:29

Wei Weng (weng271190436) reviewed Oct 20, 2026

View reviewed changes

Comment thread pkg/annotations/afd.go Resolved

Comment thread pkg/annotations/afd.go Resolved

Comment thread pkg/annotations/afd.go Resolved

Ramkumar Chinchani (rchincha) and others added 6 commits October 20, 2026 12:42

docs: add GEP-1748 scale and capacity analysis

6fc48db

docs: add scale analysis diagrams

cd2bf54

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ramkumar Chinchani (rchincha) changed the title ~~feat: add Gateway API controller foundation (GEP-1748)~~ feat: add Gateway API foundation (GEP-1748) for Zero-Public-Exposure Ingress Oct 20, 2026

Ramkumar Chinchani (rchincha) force-pushed the rchinchani/gep-1748-gateway-api branch from a7c5ff7 to 264f5cd Compare October 21, 2026 05:04

Ramkumar Chinchani (rchincha) force-pushed the rchinchani/gep-1748-gateway-api branch 3 times, most recently from 8368905 to 05ea818 Compare October 24, 2026 04:18

Wei Weng (weng271190436) reviewed Oct 28, 2026

View reviewed changes

Ramkumar Chinchani (rchincha) force-pushed the rchinchani/gep-1748-gateway-api branch 12 times, most recently from 8d4e77e to 4b52ff6 Compare November 1, 2026 01:09

Ramkumar Chinchani (rchincha) force-pushed the rchinchani/gep-1748-gateway-api branch 4 times, most recently from 9bb4ff3 to 391a154 Compare November 2, 2026 04:15

Ramkumar Chinchani (rchincha) and others added 4 commits November 1, 2026 22:47

Ramkumar Chinchani (rchincha) force-pushed the rchinchani/gep-1748-gateway-api branch from 391a154 to 52ca7dd Compare November 2, 2026 06:48

Wei Weng (weng271190436) reviewed Nov 3, 2026

View reviewed changes

Conversation

Ramkumar Chinchani (rchincha) commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How has this code been tested

Special notes for your reviewer

Uh oh!

codecov Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Ramkumar Chinchani (rchincha) commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Controllers for the AFD private HTTP feature

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Ramkumar Chinchani (rchincha) commented Aug 18, 2026 •

edited

Loading

codecov Bot commented Aug 18, 2026 •

edited

Loading

Ramkumar Chinchani (rchincha) commented Oct 24, 2026 •

edited

Loading