Use ARCamera as input for MediaPipe?

[patrick508]

Hi,

So for a project I'm working on I got the mediapipe sample to work with the webcam as done in the Sample project as well. However in my project I use the ARCameraManager from Unity to render the camera image to the screen. I need this camera because I am also trying to get the depth from this camera.

This is currently giving me issues as MediaPipe tries to start and access the webcam, while ARCamera is already using the camera. I tried to make the MediaPipe sample to work with the ARCamera but failed. It's tightly coupled to the webcam as for now. Is there any input or help I could get regarding this issue? Perhaps someone has already managed to get it to work with the ARFoundation ARCameraManager?

In short what I'm trying to achieve: Give MediaPipe the texture2D I get from ARCameraManager(I managed to get this texture already) and get the pose from that source.

[homuler]

In the following, I assume that you are using PoseTrackingSolution.

It's tightly coupled to the webcam as for now.

Not really.
Currently, texture data is read as follows.

MediaPipeUnityPlugin/Assets/Mediapipe/Samples/Scenes/Pose Tracking/PoseTrackingSolution.cs

Lines 134 to 135 in 2994e44

	// Copy current image to TextureFrame
	ReadFromImageSource(imageSource, textureFrame);

MediaPipeUnityPlugin/Assets/Mediapipe/Samples/Common/Scripts/Solution.cs

Lines 88 to 108 in 2994e44

	protected static void ReadFromImageSource(ImageSource imageSource, TextureFrame textureFrame)
	{
	var sourceTexture = imageSource.GetCurrentTexture();
	// For some reason, when the image is coiped on GPU, latency tends to be high.
	// So even when OpenGL ES is available, use CPU to copy images.
	var textureType = sourceTexture.GetType();
	if (textureType == typeof(WebCamTexture))
	{
	textureFrame.ReadTextureFromOnCPU((WebCamTexture)sourceTexture);
	}
	else if (textureType == typeof(Texture2D))
	{
	textureFrame.ReadTextureFromOnCPU((Texture2D)sourceTexture);
	}
	else
	{
	textureFrame.ReadTextureFromOnCPU(sourceTexture);
	}
	}

So if you have Texture2D at hand, you can read it like this.

// ReadFromImageSource(imageSource, textureFrame); 

// Texture2D texture2d;
textureFrame.ReadTextureFromOnCPU(texture2d);

If you want to do it the right way, you will have to implement an ImageSource class that supports ARCamera.
See ImageSource for more details.

[ROBYER1]

I would also suggest trying out the XRCPUImage with AR Foundation, let me know if you get any further with this
https://docs.unity3d.com/Packages/com.unity.xr.arsubsystems@4.1/api/UnityEngine.XR.ARSubsystems.XRCpuImage.html

[patrick508]

@homuler Thanks for the help! Sorry for the late reaction. I managed to find a solution that is kind of hacky but it works for now. Planning on cleaning it up soon and trying to find a more low level fix.

For those interested:
`
public class ARPoseProvider: BasePoseProvider, IPoseProvider
{
public IEnumerator Start()
{
AssetLoader.Provide(new StreamingAssetsResourceManager());

        StartCoroutine(StartARCameraImagePoseTracking());
        
        yield return StartCoroutine(InitializeInferenceMode());
        
        initialized = true;
    }
    
    private IEnumerator StartARCameraImagePoseTracking()
    {
        yield return StartCoroutine(InitializeARCameraSources());
    
        yield return StartCoroutine(InitializeARCameraPoseGraphRoutine());
        
        arCameraManager.frameReceived += OnARCameraFrameReceived;
    }
    
    private void OnARCameraFrameReceived(ARCameraFrameEventArgs eventArgs)
    {
        arTextureToUpdate = GetCurrentColorTexture();

        UpdateARPose(arTextureToUpdate);
    }
    
    private void UpdateARPose(Texture2D texture)
    {
        arTextureFrame ??= new TextureFrame(texture.width, texture.height);
        
        arTextureFrame.SetPixels32(texture.GetPixels32());

        arCameraPoseTrackingGraph.AddTextureFrameToInputStream(arTextureFrame).AssertOk();
        currentARCameraPose = arCameraPoseTrackingGraph.FetchNextValue();
    }

    unsafe Texture2D GetCurrentColorTexture()
    {
        if (!arCameraManager.TryAcquireLatestCpuImage(out XRCpuImage image))
            return null;

        XRCpuImage.ConversionParams conversionParams = new XRCpuImage.ConversionParams
        {
            inputRect = new RectInt(0, 0, image.width, image.height),
            outputDimensions = new Vector2Int(image.width, image.height),
            outputFormat = TextureFormat.RGBA32,
            transformation = XRCpuImage.Transformation.MirrorX
        };

        int size = image.GetConvertedDataSize(conversionParams);
        NativeArray<byte> buffer = new NativeArray<byte>(size, Allocator.Temp);

        image.Convert(conversionParams, new IntPtr(buffer.GetUnsafePtr()), buffer.Length);
        image.Dispose();

        if (arTexture == null)
        {
            arTexture = new Texture2D(conversionParams.outputDimensions.x, conversionParams.outputDimensions.y, conversionParams.outputFormat, false);
        }

        arTexture.LoadRawTextureData(buffer);
        arTexture.Apply();
        buffer.Dispose();

        return arTexture;
    }
}`

[homuler]

I also have written a minimal code to run the Face Detection solution for those who've gotten to this issue.
Please also refer to it.

// Copyright (c) 2021 homuler
//
// Use of this source code is governed by an MIT-style
// license that can be found in the LICENSE file or at
// https://opensource.org/licenses/MIT.

using Mediapipe;
using Mediapipe.Unity;

using System;
using System.Collections;

using Unity.Collections;
using Unity.Collections.LowLevel.Unsafe;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

using Stopwatch = System.Diagnostics.Stopwatch;

public class ARCameraManagerTest : MonoBehaviour
{
  [SerializeField] private ARCameraManager _cameraManager;
  [SerializeField] private TextAsset _configText; // attach `face_detection_gpu.txt`

  private CalculatorGraph _calculatorGraph;
  private NativeArray<byte> _buffer;
  private Stopwatch _stopwatch;
  private ResourceManager _resourceManager;
  private GpuResources _gpuResources;

  private IEnumerator Start()
  {
    _cameraManager.frameReceived += OnCameraFrameReceived;
    _stopwatch = new Stopwatch();

    _resourceManager = new StreamingAssetsResourceManager();
    yield return _resourceManager.PrepareAssetAsync("face_detection_short_range.bytes");
    yield return _resourceManager.PrepareAssetAsync("face_detection_full_range_sparse.bytes");

    _gpuResources = GpuResources.Create().Value();
    _calculatorGraph = new CalculatorGraph(_configText.text);
    _calculatorGraph.SetGpuResources(_gpuResources).AssertOk();

    _calculatorGraph.ObserveOutputStream("face_detections", 0, OutputCallback, true).AssertOk();

    var sidePacket = new SidePacket();
    sidePacket.Emplace("input_rotation", new IntPacket(0));
    sidePacket.Emplace("input_horizontally_flipped", new BoolPacket(false));
    sidePacket.Emplace("input_vertically_flipped", new BoolPacket(true));
    sidePacket.Emplace("model_type", new IntPacket(0));

    _calculatorGraph.StartRun(sidePacket).AssertOk();
    _stopwatch.Start();
  }

  private void OnDestroy()
  {
    _cameraManager.frameReceived -= OnCameraFrameReceived;

    var status = _calculatorGraph.CloseAllPacketSources();
    if (!status.Ok())
    {
      Debug.Log($"Failed to close packet sources: {status}");
    }

    status = _calculatorGraph.WaitUntilDone();
    if (!status.Ok())
    {
      Debug.Log(status);
    }

    _calculatorGraph.Dispose();
    _gpuResources.Dispose();
    _buffer.Dispose();
  }

  private unsafe void OnCameraFrameReceived(ARCameraFrameEventArgs eventArgs)
  {
    if (_cameraManager.TryAcquireLatestCpuImage(out var image))
    {
      InitBuffer(image);

      var conversionParams = new XRCpuImage.ConversionParams(image, TextureFormat.RGBA32);
      var ptr = (IntPtr)NativeArrayUnsafeUtility.GetUnsafePtr(_buffer);
      image.Convert(conversionParams, ptr, _buffer.Length);
      image.Dispose();

      var imageFrame = new ImageFrame(ImageFormat.Types.Format.Srgba, image.width, image.height, 4 * image.width, _buffer);
      var currentTimestamp = _stopwatch.ElapsedTicks / (TimeSpan.TicksPerMillisecond / 1000);
      var imageFramePacket = new ImageFramePacket(imageFrame, new Timestamp(currentTimestamp));

      _calculatorGraph.AddPacketToInputStream("input_video", imageFramePacket).AssertOk();
    }
  }

  private void InitBuffer(XRCpuImage image)
  {
    var length = image.width * image.height * 4;
    if (_buffer == null || _buffer.Length != length)
    {
      _buffer = new NativeArray<byte>(length, Allocator.Persistent, NativeArrayOptions.UninitializedMemory);
    }
  }

  [AOT.MonoPInvokeCallback(typeof(CalculatorGraph.NativePacketCallback))]
  private static IntPtr OutputCallback(IntPtr graphPtr, int steramId, IntPtr packetPtr)
  {
    try
    {
      using (var packet = new DetectionVectorPacket(packetPtr, false))
      {
        var value = packet.IsEmpty() ? null : packet.Get();

        if (value != null && value.Count > 0)
        {
          foreach (var detection in value)
          {
            Debug.Log(detection);
          }
        }
      }
      return Status.Ok().mpPtr;
    }
    catch (Exception e)
    {
      return Status.FailedPrecondition(e.ToString()).mpPtr;
    }
  }
}