Skip to main content

AR pet demo

For this tutorial, we will create a simple AR pet that will respond to hand gestures and interact with our hand. We'll use the Ur module for hand tracking.

Before you begin​

Before getting started, set up your development environment by following the steps in the Quickstart.

It also helps to have gone through the Create a simple shared AR experience and Handtracker in shared AR experience tutorials, but it's not strictly necessary as this tutorial follows many of the same steps.

Package versions used in this demo​

  • ConjureKit v0.6.31
  • Manna module v0.6.58
  • Ur module v0.6.5
  • AR Foundation v4.2.8
  • ARKit XR Plugin v4.2.8

Initialize ConjureKit and spawn a digital pet asset​

  1. Install the Ur package, import the sample from Ur, and open the Main scene.

  2. Import a digital pet prefab asset of your choice, ideally one that already comes with an animator and a controller.

  3. In the sample's Main.cs script, add the following imports:

using Auki.ConjureKit;
using Auki.ConjureKit.Manna;
using Auki.Util;
using UnityEngine.UI;
using UnityEngine.XR.ARSubsystems;
  1. Create these variables:
[SerializeField] private Text sessionState;
[SerializeField] private Text sessionID;
    
[SerializeField] private GameObject raccoon;
[SerializeField] private Button spawnButton;
    
private bool qrCodeBool;
[SerializeField] Button qrCodeButton;
    
private IConjureKit _conjureKit;
private Manna _manna;
    
private ARCameraManager arCameraManager;
private Texture2D _videoTexture;
  1. In the Start() function, declare the arCameraManager variable, initialize ConjureKit (remember to add YOUR_APP_KEY and YOUR_APP_SECRET), and connect to a Hagall server.
arCameraManager = arCamera.GetComponent<ARCameraManager>();
        
_conjureKit = new ConjureKit(
    arCamera.transform,
    "YOUR_APP_KEY",
    "YOUR_APP_SECRET");
    
_manna = new Manna(_conjureKit);
        
_conjureKit.OnStateChanged += state =>
{
    sessionState.text = state.ToString();
    ToggleControlsState(state == State.Calibrated);
};
    
_conjureKit.OnJoined += session =>
{
    sessionID.text = session.Id.ToString();
};
    
_conjureKit.OnLeft += session =>
{
    sessionID.text = "";
};
    
_conjureKit.OnEntityAdded += CreateRaccoon;
_conjureKit.Connect();
  1. Feed Manna with AR camera video frames acquired from ARCameraManager in order to recognize QR codes and perform instant calibration.
private void Update()
{
    _handTracker.Update();
    FeedMannaWithVideoFrames();
}

private void FeedMannaWithVideoFrames()
{
    var imageAcquired = arCameraManager.TryAcquireLatestCpuImage(out var cpuImage);
    if (!imageAcquired)
    {
        AukiDebug.LogInfo("Couldn't acquire CPU image");
        return;
    }

    if (_videoTexture == null) _videoTexture = new Texture2D(cpuImage.width, cpuImage.height, TextureFormat.R8, false);

    var conversionParams = new XRCpuImage.ConversionParams(cpuImage, TextureFormat.R8);
    cpuImage.ConvertAsync(
        conversionParams,
        (status, @params, buffer) =>
        {
            _videoTexture.SetPixelData(buffer, 0, 0);
            _videoTexture.Apply();
            cpuImage.Dispose();

            _manna.ProcessVideoFrameTexture(
                _videoTexture,
                arCamera.projectionMatrix,
                arCamera.worldToCameraMatrix
            );
        }
    );
}

  1. Delete the ToggleHandLandmarks() function from the sample code as well as the call to it.

  2. Add the following functions: ToggleControlsState(), ToggleLighthouse(), CreateRaccoon(), and CreateRaccoonEntity():

private void ToggleControlsState(bool interactable)
{
    if (spawnButton) spawnButton.interactable = interactable;
    if (qrCodeButton) qrCodeButton.interactable = interactable;
}

public void ToggleLighthouse()
{
    qrCodeBool = !qrCodeBool;
    _manna.SetLighthouseVisible(qrCodeBool);
}

public void CreateRaccoonEntity()
{
    if (_conjureKit.GetState() != State.Calibrated)
        return;
    
    Vector3 position = arCamera.transform.position + arCamera.transform.forward * 0.5f;
    Quaternion rotation = Quaternion.Euler(0, arCamera.transform.eulerAngles.y, 0);
    
    Pose entityPos = new Pose(position, rotation);
    
    _conjureKit.GetSession().AddEntity(
        entityPos,
        onComplete: entity => CreateRaccoon(entity),
        onError: error => Debug.Log(error));
}

private void CreateRaccoon(Entity entity)
{
    if (entity.Flag == EntityFlag.EntityFlagParticipantEntity) return;

    var pose = _conjureKit.GetSession().GetEntityPose(entity);
    Instantiate(raccoon, pose.position, pose.rotation);
}

  1. In Unity, create two legacy text objects called SessionState and SessionID, as well as two buttons called SpawnButton and QR. Position these on the canvas.

  2. Add an On Click callback for SpawnButton to CreateRaccoonEntity(); for the QR button it should be ToggleLighthouse().

  3. Drag the two text objects and the two buttons to their corresponding fields in the Main GameObject. Text objects and buttons dragged to Main GameObject

  4. Drag the digital pet prefab into its field in the Main GameObject as well and edit or create its animator controller. The default sequence of animations should start at "Entry," transition to the "Default State" and any other animations, and end at "Exit." Digital pet animator controller

  5. Hit the Play button and spawn your pet. If it's facing the wrong direction, adjust its Y axis rotation:

Quaternion rotation = Quaternion.Euler(0, 180, 0);

  1. In order to spawn the pet on a surface, find the AR Plane Manager component inside the AR Session Origin GameObject and change its Detection Mode to "Horizontal".

  2. In the CreateRaccoonEntity() function, instead of positioning the pet relative to the AR Camera, perform a raycast to the detected plane and spawn the pet where the ray hits the plane:

public void CreateRaccoonEntity()
{
    if (_conjureKit.GetState() != State.Calibrated)
        return;
    
    Ray ray = arCamera.ScreenPointToRay(new Vector3(Screen.width / 2, Screen.height / 2));
    List<ARRaycastHit> hits = new List<ARRaycastHit>();
    
    if (arRaycastManager.Raycast(ray, hits, TrackableType.PlaneWithinPolygon))
    {
        Quaternion rotation = Quaternion.Euler(0, 180, 0);
        Pose hitPose = new Pose(hits[0].pose.position, rotation);

        _conjureKit.GetSession().AddEntity(
            hitPose,
            onComplete: entity => CreateRaccoon(entity),
            onError: error => Debug.Log(error));
    }
}

If you build and run the project now (remember to add open scenes before building), the app should connect to ConjureKit and the pet should spawn on the floor. Pet spawns on floor

Hand gesture detection​

  1. In the Animator Controller, create a new Trigger parameter and name it after the action you want the pet to do when it detects a specific hand gesture.

  2. Add this animation and make a transition to it from "Any State." In this transition, add a condition that checks if the Trigger parameter is true. Add Trigger parameter to Animator Controller

  3. In Main.cs, add these variables:

public bool hasPlayedDead = false;
private GameObject raccoonObject;
private Animator raccoonAnimator;
  1. Add a new function that finds the spawned pet and triggers the animation:
public void PlayDead()
{
    hasPlayedDead = true;
    raccoonObject = GameObject.Find("Raccoon Cub PA(Clone)");
    raccoonAnimator = raccoonObject.GetComponent<Animator>();
    raccoonAnimator.SetTrigger("PlayDead");
}
  1. Create a new Vector3 called handLandmarksPositions:
private Vector3[] handLandmarksPositions;
  1. Define it in the Start() function:
handLandmarksPositions = new Vector3[HandTracker.LandmarksCount];
  1. In _handTracker.OnUpdate, replace var landMarkPosition and landMarkPosition with handLandmarksPositions[l]:
for (int l = 0; l < HandTracker.LandmarksCount; ++l)
{
    handLandmarksPositions[l] = new Vector3(
        landmarks[handLandmarkIndex + (l * 3) + 0],
        landmarks[handLandmarkIndex + (l * 3) + 1],
        landmarks[handLandmarkIndex + (l * 3) + 2]);

    // Update the landmarks position 
    _handLandmarks[l].transform.localPosition = handPosition + handLandmarksPositions[l];
}
  1. In order to detect a gesture, measure the distances between relevant hand landmarks and call the function that triggers the animation when conditions for that hand gesture are met:
var indexPalmDistance = Vector3.Distance(handLandmarksPositions[8], handLandmarksPositions[0]);
var middlePalmDistance = Vector3.Distance(handLandmarksPositions[12], handLandmarksPositions[0]);
var ringPalmDistance = Vector3.Distance(handLandmarksPositions[16], handLandmarksPositions[0]);
var pinkyPalmDistance = Vector3.Distance(handLandmarksPositions[20], handLandmarksPositions[0]);

if (indexPalmDistance > 0.1f && middlePalmDistance < 0.08f && ringPalmDistance < 0.08f && pinkyPalmDistance < 0.08f)
{
    if (!hasPlayedDead)
    {
        PlayDead();
    }
}

Now when you build and run the project, the animation should be triggered when the correct hand gesture is detected. Pet plays dead when triggered by hand gesture

Hand interaction​

  1. To have the pet perform a different animation when the hand interacts with it, add a second Trigger to the Animator Controller and name it after the action.

  2. Like before, add this animation and make a transition to it from "Any State." In this transition, add a condition that checks if the second Trigger parameter is true.

  3. In Main.cs, create a Renderer called FingertipLandmark. This will be used to detect when the hand touches the pet.

[SerializeField] private Renderer fingertipLandmark;
  1. In _handTracker.OnUpdate, set fingertipLandmark’s position to be at the tip of the index finger:
fingertipLandmark.transform.localPosition = handPosition + handLandmarksPositions[8];
  1. Below _handTracker.OnUpdate, set fingertipLandmark’s parent to be arCamera.transform:
fingertipLandmark.transform.parent = arCamera.transform;
  1. Now create a second function that triggers the second animation:
public void GetUp()
{
    hasPlayedDead = false;
    raccoonObject = GameObject.Find("Raccoon Cub PA(Clone)");
    raccoonAnimator = raccoonObject.GetComponent<Animator>();
    raccoonAnimator.SetTrigger("GetUp");
}

  1. In Unity, create a sphere GameObject which will be the fingertip landmark. Scale it down to 5cm and check “Is Trigger” in the collider component.

  2. Drag this fingertip landmark sphere into its field in the Main GameObject.

  3. In the digital pet prefab, add a Rigidbody component so it can get triggers from other colliders, and check “Is Kinematic”.

  4. Create a new C# script to handle trigger events on the digital pet. Give it a name, for example TouchableByHand.cs.

  5. Make sure it’s in the same namespace as Main.cs and delete the Start() and Update() functions.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

namespace AukiHandTrackerSample
{
    public class TouchableByHand : MonoBehaviour
    {
        
    }
}
  1. Create the following variables:
public Main mainScript;
private GameObject raccoonObject;
private Animator raccoonAnimator;
  1. In the Awake() function, find object of type Main and assign it to mainScript:
private void Awake()
{
    mainScript = FindObjectOfType<Main>();
}
  1. Define the OnTriggerEnter() function which will trigger the second animation when the fingertip landmark touches the pet:
private void OnTriggerEnter(Collider other)
{
    if (mainScript == null)
    {
        mainScript = FindObjectOfType<Main>();
        if (mainScript == null)
        {
            Debug.LogError("Main script not found");
            return;
        }
    }

    if (mainScript.hasPlayedDead)
    {
        mainScript.GetUp();
    }
}
  1. In Unity, add TouchableByHand.cs to the digital pet prefab.

Now when you build and run the project, the second animation should be triggered when the fingertip landmark touches the pet. Pet gets up when triggered by hand interaction

Finishing touches​

  1. Hide the hand landmarks by opening the hand landmark prefab and unchecking it.

  2. Similarly, hide the sphere on the index finger by unchecking its mesh renderer.

  3. To add occlusion culling, go to the AR Camera and add an AR Occlusion Manager component. Set both of the Human Segmentation settings to "Fastest."

  4. Remove the spawn button after the pet has spawned by adding this line at the end of the CreateRaccoon() function:

GameObject.Find("SpawnButton").SetActive(false);

That’s it! Now you have an AR pet that responds to hand gestures and hand interactions.

Complete code​

Main.cs:

using System.Collections.Generic;
using Auki.ConjureKit;
using Auki.ConjureKit.Manna;
using Auki.Ur;
using Auki.Util;
using UnityEngine;
using UnityEngine.UI;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

namespace AukiHandTrackerSample
{
    /// <summary>
    /// This sample shows how to use Ur hand tracker for the most common cases:
    /// - Showing hand calibration state.
    /// - Calibrating hand tracker for specific hand size.
    /// - Visualizing the hand landmarks
    /// </summary>
    public class Main : MonoBehaviour
    {
        private const int NumberOfTrackedHands = 1;
        
        /// References needed to initialize the hand tracker
        [SerializeField] private Camera arCamera;
        [SerializeField] private ARSession arSession;
        [SerializeField] private ARRaycastManager arRaycastManager;
        
        [SerializeField] private Text sessionState;
        [SerializeField] private Text sessionID;
    
        [SerializeField] private GameObject raccoon;
        [SerializeField] private Button spawnButton;
    
        private bool qrCodeBool;
        [SerializeField] Button qrCodeButton;
    
        private IConjureKit _conjureKit;
        private Manna _manna;
    
        private ARCameraManager arCameraManager;
        private Texture2D _videoTexture;
        
        [SerializeField] private HandLandmark handLandmarkPrefab;
        
        private HandTracker _handTracker;
        private List<HandLandmark> _handLandmarks;

        private Vector3[] handLandmarksPositions;
        [SerializeField] private Renderer fingertipLandmark;

        public bool hasPlayedDead = false;
        private GameObject raccoonObject;
        private Animator raccoonAnimator;

        private void Start()
        {
            arCameraManager = arCamera.GetComponent<ARCameraManager>();
        
            _conjureKit = new ConjureKit(
                arCamera.transform,
                "YOUR_APP_KEY",
                "YOUR_APP_SECRET");
    
            _manna = new Manna(_conjureKit);
        
            _conjureKit.OnStateChanged += state =>
            {
                sessionState.text = state.ToString();
                ToggleControlsState(state == State.Calibrated);
            };
    
            _conjureKit.OnJoined += session =>
            {
                sessionID.text = session.Id.ToString();
            };
    
            _conjureKit.OnLeft += session =>
            {
                sessionID.text = "";
            };
    
            _conjureKit.OnEntityAdded += CreateRaccoon;
            _conjureKit.Connect();
            
            
            _handTracker = HandTracker.GetInstance();

            // Initialize the hand tracker
            _handTracker.SetARSystem(arSession, arCamera, arRaycastManager);

            // Initialize a list of HandLandmarks to display landmark index and position
            _handLandmarks = new List<HandLandmark>(NumberOfTrackedHands * HandTracker.LandmarksCount);
            for (int i = 0; i < HandTracker.LandmarksCount; i++)
            {
                _handLandmarks.Add(Instantiate(handLandmarkPrefab));
                _handLandmarks[i].SetText(i.ToString());
                _handLandmarks[i].transform.SetParent(arCamera.transform);
            }

            handLandmarksPositions = new Vector3[HandTracker.LandmarksCount];

            _handTracker.OnUpdate += (landmarks, translations, isRightHand, score) =>
            {
                for (int h = 0; h < NumberOfTrackedHands; ++h)
                {
                    if (score[h] > 0)
                    {
                        var handPosition = new Vector3(
                            translations[h * 3 + 0], 
                            translations[h * 3 + 1],
                            translations[h * 3 + 2]);

                        var handLandmarkIndex = h * HandTracker.LandmarksCount * 3;
                        fingertipLandmark.transform.localPosition = handPosition + handLandmarksPositions[8];
                        
                        for (int l = 0; l < HandTracker.LandmarksCount; ++l)
                        {
                            handLandmarksPositions[l] = new Vector3(
                                landmarks[handLandmarkIndex + (l * 3) + 0],
                                landmarks[handLandmarkIndex + (l * 3) + 1],
                                landmarks[handLandmarkIndex + (l * 3) + 2]);
                    
                            // Update the landmarks position 
                            _handLandmarks[l].transform.localPosition = handPosition + handLandmarksPositions[l];
                        }
                        
                        var indexPalmDistance = Vector3.Distance(handLandmarksPositions[8], handLandmarksPositions[0]);
                        var middlePalmDistance = Vector3.Distance(handLandmarksPositions[12], handLandmarksPositions[0]);
                        var ringPalmDistance = Vector3.Distance(handLandmarksPositions[16], handLandmarksPositions[0]);
                        var pinkyPalmDistance = Vector3.Distance(handLandmarksPositions[20], handLandmarksPositions[0]);
                        
                        if (indexPalmDistance > 0.1f && middlePalmDistance < 0.08f && ringPalmDistance < 0.08f && pinkyPalmDistance < 0.08f)
                        {
                            if (!hasPlayedDead)
                            {
                                PlayDead();
                            }
                        }
                    }
                }
            };

            fingertipLandmark.transform.parent = arCamera.transform;
            _handTracker.Start(NumberOfTrackedHands);
        }

        private void Update()
        {
            _handTracker.Update();
            FeedMannaWithVideoFrames();
        }
        
        private void FeedMannaWithVideoFrames()
        {
            var imageAcquired = arCameraManager.TryAcquireLatestCpuImage(out var cpuImage);
            if (!imageAcquired)
            {
                AukiDebug.LogInfo("Couldn't acquire CPU image");
                return;
            }

            if (_videoTexture == null) _videoTexture = new Texture2D(cpuImage.width, cpuImage.height, TextureFormat.R8, false);

            var conversionParams = new XRCpuImage.ConversionParams(cpuImage, TextureFormat.R8);
            cpuImage.ConvertAsync(
                conversionParams,
                (status, @params, buffer) =>
                {
                    _videoTexture.SetPixelData(buffer, 0, 0);
                    _videoTexture.Apply();
                    cpuImage.Dispose();

                    _manna.ProcessVideoFrameTexture(
                        _videoTexture,
                        arCamera.projectionMatrix,
                        arCamera.worldToCameraMatrix
                    );
                }
            );
        }
        
        private void ToggleControlsState(bool interactable)
        {
            if (spawnButton) spawnButton.interactable = interactable;
            if (qrCodeButton) qrCodeButton.interactable = interactable;
        }

        public void ToggleLighthouse()
        {
            qrCodeBool = !qrCodeBool;
            _manna.SetLighthouseVisible(qrCodeBool);
        }

        public void CreateRaccoonEntity()
        {
            if (_conjureKit.GetState() != State.Calibrated)
                return;
            
            Ray ray = arCamera.ScreenPointToRay(new Vector3(Screen.width / 2, Screen.height / 2));
            List<ARRaycastHit> hits = new List<ARRaycastHit>();
            
            if (arRaycastManager.Raycast(ray, hits, TrackableType.PlaneWithinPolygon))
            {
                Quaternion rotation = Quaternion.Euler(0, 180, 0);
                Pose hitPose = new Pose(hits[0].pose.position, rotation);

                _conjureKit.GetSession().AddEntity(
                    hitPose,
                    onComplete: entity => CreateRaccoon(entity),
                    onError: error => Debug.Log(error));
            }
        }

        private void CreateRaccoon(Entity entity)
        {
            if (entity.Flag == EntityFlag.EntityFlagParticipantEntity) return;

            var pose = _conjureKit.GetSession().GetEntityPose(entity);
            Instantiate(raccoon, pose.position, pose.rotation);
            GameObject.Find("SpawnButton").SetActive(false); // Remove spawn button
        }

        public void PlayDead()
        {
            hasPlayedDead = true;
            raccoonObject = GameObject.Find("Raccoon Cub PA(Clone)");
            raccoonAnimator = raccoonObject.GetComponent<Animator>();
            raccoonAnimator.SetTrigger("PlayDead");
        }
        
        public void GetUp()
        {
            hasPlayedDead = false;
            raccoonObject = GameObject.Find("Raccoon Cub PA(Clone)");
            raccoonAnimator = raccoonObject.GetComponent<Animator>();
            raccoonAnimator.SetTrigger("GetUp");
        }
    }
}

TouchableByHand.cs:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

namespace AukiHandTrackerSample
{
    public class TouchableByHand : MonoBehaviour
    {
        public Main mainScript;
        private GameObject raccoonObject;
        private Animator raccoonAnimator;

        private void Awake()
        {
            mainScript = FindObjectOfType<Main>();
        }

        private void OnTriggerEnter(Collider other)
        {
            if (mainScript == null)
            {
                mainScript = FindObjectOfType<Main>();
                if (mainScript == null)
                {
                    Debug.LogError("Main script not found");
                    return;
                }
            }

            if (mainScript.hasPlayedDead)
            {
                mainScript.GetUp();
            }
        }
    }
}

The full project can be found on GitHub.