Error: Couldn't scale frame: Invalid argument

[Vally11Byte]

Is there an existing issue for this?

• I have searched the existing issues

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the latest beta/pre-release?

I'm too lazy to test

Describe the Bug

I’m experiencing an issue with Sunshine on Wayland (Arch Linux + Hyprland). When using NVENC, encoding fails and Sunshine falls back through Vulkan (not implemented for wlroots), VAAPI, and finally software encoding (libx264), which works correctly.

Error: Couldn't scale frame: Invalid argument

Sunshine version:
2026.516.143833-1

Expected Behavior

When starting a stream from another PC, the host should use NVENC for video encoding.

Additional Context

No response

Host Operating System

Linux

Operating System Version

Omarchy ( arch + hyprland )

Architecture

amd64/x86_64

Package

other (not listed)

GPU Type

NVIDIA

GPU Model

RTX 4070 TI SUPER

GPU Driver/Mesa Version

NVIDIA 595.71.05 CUDA Version: 13.2

Capture Method

wlroots (FreeBSD/Linux)

Apps

Log output

Online logs

https://gist.githubusercontent.com/Vally11Byte/72208afa238d04bf27137340e0a1a2c4/raw/7176d25d77e48456a8cb809b8d9a46c6df938507/gistfile1.txt

[letsbeanti]

Same issue for me on MSI SuprimX 3070ti. Cuda 13.3. Nvidia 610.43.02

[Unayung]

Root cause on my system: the distro package was built without CUDA support. The Couldn't scale frame: Invalid argument message is a red herring, it has nothing to do with scaling.

Setup: Arch (Omarchy), Hyprland 0.56.2, RTX 3080, driver 610.57.04, sunshine 2026.906.222525 from the Omarchy repo, which is a rebuild of the AUR sunshine PKGBUILD.

The chain

nvenc maps to mem_type_e::cuda, so wl_display() returns wlr_vram_t. Its make_avcodec_encode_device() keeps the CUDA branch behind #ifdef SUNSHINE_BUILD_CUDA:

// src/platform/linux/wlgrab.cpp
#ifdef SUNSHINE_BUILD_CUDA
if (mem_type == platf::mem_type_e::cuda) {
  return cuda::make_avcodec_gl_encode_device(width, height, 0, 0);
}
#endif
return std::make_unique<platf::avcodec_encode_device_t>();   // <- data == nullptr

With CUDA compiled out it falls through and returns an empty device. video.cpp then takes this branch:

// src/video.cpp
if (!encode_device->data) {
  auto software_encode_device = std::make_unique<avcodec_software_encode_device_t>();
  ...
}

so the software convert path ends up paired with a VRAM display. Encoder validation calls wlr_vram_t::dummy_img(), which by design does not touch the image:

int dummy_img(platf::img_t *img) override {
  // Empty images are recognized as dummies by the zero sequence number
  return 0;
}

avcodec_software_encode_device_t::convert() then assigns img.data (nullptr) to sws_input_frame->data[0] and calls sws_scale_frame(), which returns EINVAL. The probe fails and Sunshine falls back to libx264.

How to check without reinstalling anything

cuda.cpp contains 27 distinct log strings, 25 of which are exclusive to that file. Grep the binary for one of them:

strings -a /usr/bin/sunshine | grep -c "Couldn't open DRM FD for CUDA device"

0 means the build has no CUDA support. Same-commit comparison on my machine, both cb72dff:

binary	cuda.cpp-exclusive strings present
omarchy/sunshine 2026.906.222525-1.1	0 / 25
official sunshine-bin 2026.906.222525-1	25 / 25

cuInit, cuGraphicsGLRegisterImage and friends are not usable as evidence here. ffmpeg's own dynlink loader supplies them, so they are present in both binaries. That sent me down the wrong path for a while.

Fix

Install a build with CUDA enabled. The AUR sunshine PKGBUILD uses _use_cuda=detect, which resolves to -DSUNSHINE_ENABLE_CUDA=OFF when the cuda package is absent on the build host, so any redistributed rebuild of it is a candidate. Switching to sunshine-bin (the official release binary) gave me hevc_nvenc immediately, with the Sunshine process at 4.5% CPU while streaming 3440x1440.

Suggested upstream change

Two small things would have made this self-diagnosing:

1. Warn when an encoder requiring a hardware memory type is probed but the matching backend was compiled out.
2. Have wlr_vram_t::make_avcodec_encode_device() return nullptr on the fall-through rather than an empty avcodec_encode_device_t. As it stands, a VRAM display is silently paired with the software convert path and the only symptom surfaces several layers away as an EINVAL from swscale.

Ruled out along the way

In case it saves someone else the time: capture = kms (the NVIDIA proprietary driver exposes no framebuffers, the KMS monitor list comes back empty), aspect-ratio and odd-height scaling (3440x1440 to 1920x803 is odd, but a 1920x1080 headless output fails identically), and VA-API (nvidia's is NVDEC-only, No usable encoding entrypoint found for profile VAProfileH264High).

[ReenigneArcher]

Setup: Arch (Omarchy), Hyprland 0.56.2, RTX 3080, driver 610.57.04, sunshine 2026.906.222525 from the Omarchy repo, which is a rebuild of the AUR sunshine PKGBUILD.

Why? We have our own pacman-repo which pre-compiles Sunshine correctly with CUDA. With AUR you have to compile yourself and we're not going to force everyone to install CUDA because it's a huge package.

[Unayung]

Thanks, that makes sense, and I agree CUDA shouldn't be a hard dependency.

For context, I didn't pick the AUR build on purpose. Omarchy carries its own sunshine package, and that's what pacman -S sunshine gave me there. I've switched to your build and NVENC is working well now. I'll let the Omarchy folks know about their package.

Just an idea in case it's useful: a short log line when nvenc is probed on a build without CUDA (something like "this build has no CUDA support") might save the next person some digging, since the current Couldn't scale frame error looks a lot like a Wayland/NVIDIA problem. Completely understand if that's not a priority.

Thanks for maintaining Sunshine!

[ReenigneArcher]

I did put up a PR at omacom/omarchy-pkgs#383 but at this time I don't think anyone is reviewing/approving these. If it gets accepted, I'll throw up another PR there to have it compile with CUDA.