GH-49676: [Python][Packaging] Fix gRPC docker image layer being too big for hcsshim on Windows#49678
GH-49676: [Python][Packaging] Fix gRPC docker image layer being too big for hcsshim on Windows#49678raulcd merged 15 commits intoapache:mainfrom
Conversation
…by not pulling bases
|
@github-actions crossbow submit wheel-windows-* |
|
|
|
Revision: 663e4a6 Submitted crossbow builds: ursacomputing/crossbow @ actions-2d3c582858
|
… reduce layer size
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: a0edd40 Submitted crossbow builds: ursacomputing/crossbow @ actions-a1756da700
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: b45c287 Submitted crossbow builds: ursacomputing/crossbow @ actions-946cdff5a8
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 4b9fd9c Submitted crossbow builds: ursacomputing/crossbow @ actions-e9e5587b8a
|
… of layer size" This reverts commit 4b9fd9c.
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 5479a8d Submitted crossbow builds: ursacomputing/crossbow @ actions-310307e9a4
|
|
@kou any idea what might be the issue here? |
|
I'll take a look at this in a few hours! |
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: a7c0613 Submitted crossbow builds: ursacomputing/crossbow @ actions-66876a4e34
|
|
Sorry. I couldn't take a look at this in a few hours... I'll take a look at this in 24 hours... |
|
As per 🤖 we should implement the change below: "this adds a step that makes the job explicitly check for the docker service, start it if needed, and wait until docker version succeeds before any archery docker pull/build/run commands execute. So this turns an intermittent windows-2022 runner issue into a handled setup step, which should reduce spurious CI failures." --- a/dev/tasks/python-wheels/github.windows.yml
+++ b/dev/tasks/python-wheels/github.windows.yml
@@ -43,6 +43,32 @@
{{ macros.github_checkout_arrow()|indent }}
{{ macros.github_login_ghcr()|indent }}
{{ macros.github_install_archery()|indent }}
+
+ - name: Wait for Docker
+ shell: pwsh
+ run: |
+ $dockerService = Get-Service -Name docker -ErrorAction SilentlyContinue
+ if ($null -eq $dockerService) {
+ Write-Error "Docker service not found."
+ exit 1
+ }
+
+ if ($dockerService.Status -ne "Running") {
+ Write-Host "Starting Docker service..."
+ Start-Service -Name docker
+ }
+
+ for ($i = 1; $i -le 60; $i++) {
+ docker version *> $null
+ if ($LASTEXITCODE -eq 0) {
+ Write-Host "Docker is ready."
+ exit 0
+ }
+ Start-Sleep -Seconds 2
+ }
+
+ Write-Error "Docker failed to become ready."
+ exit 1
- name: Prepare
shell: bash |
|
I think that could solve the "intermittent failure" around docker being unable but the big issue is not that one. The real problem is when docker is running at building the flight dependencies from vcpkg step. If I isolate the different vcpkg features being built the image builds fine until the flight feature step. Flight pulls grpc/abseil/protobuf. The |
|
Docker seemed to have been upgraded on the runner images from 27.x/28.x to 29.1 which might be the issue: |
…lling gRPC with vcpkg
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 14c89a4 Submitted crossbow builds: ursacomputing/crossbow @ actions-d3af6272a6
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 96931be Submitted crossbow builds: ursacomputing/crossbow @ actions-5806a993b9
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: a033258 Submitted crossbow builds: ursacomputing/crossbow @ actions-8d2d071202
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
Revision: 14f1b27 Submitted crossbow builds: ursacomputing/crossbow @ actions-733eebe387
|
|
It makes sense! Let's do it! |
|
@github-actions crossbow submit wheel-windows-cp313-cp313t-amd64 |
|
@github-actions crossbow submit wheel-windows-* |
raulcd
changed the title
|
Revision: a38282c Submitted crossbow builds: ursacomputing/crossbow @ actions-42eaca7fbf
|
|
Once CI finishes, if successful, I'll merge and cherry-pick on the release branch |
|
Revision: a38282c Submitted crossbow builds: ursacomputing/crossbow @ actions-6473f96f72
|
…ig for hcsshim on Windows (#49678) ### Rationale for this change The docker base docker image which installs all the vcpkg dependencies to build Windows wheels was failing to commit the layer for gRPC due to size. ### What changes are included in this PR? Two separate fixes that independently fix the issue: - Patch vcpkg for Windows to remove unconditional `/Z7` - Use D:/ drive instead of C:/ as it has bigger size for Docker storage ### Are these changes tested? Yes via archery ### Are there any user-facing changes? No * GitHub Issue: #49676 Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Co-authored-by: Bryce Mecum <petridish@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
|
After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 5617e8d. There was 1 benchmark result with an error:
There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them. |
4 participants
Rationale for this change
The docker base docker image which installs all the vcpkg dependencies to build Windows wheels was failing to commit the layer for gRPC due to size.
What changes are included in this PR?
Two separate fixes that independently fix the issue:
/Z7Are these changes tested?
Yes via archery
Are there any user-facing changes?
No