(FIX FOUND) EasyAR 3.0.1 Unity iOS camera feed upside down using HDR

+5 votes
asked Aug 28, 2019 by spaceagent (1,290 points)
edited Nov 11, 2019 by spaceagent

The camera feed is upside down on iOS devices using Metal graphics API and with HDR enabled in Graphics settings in Unity. It looks like it is a shader problem related to platform-specific rendering differences.

UPDATE: We managed to find the solution and another related Unity bug.

First the problem is with the EasyAR shaders as initially tought is related to platform-specific rendering differences, second it is a problem with how Unity render its camera and how HDR changes that, as we mentioned in this update.

To fix this we modified all EasyAR shaders to flip the texture uv when #if UNITY_UV_STARTS_AT_TOP is true and a script that set forceIntoRenderTexture to true on the Unity camera, that is doing the same thing that enabling HDR do, except the actual HDR effect.

Here is a link with the modified shaders, just enable forceIntoRenderTexture on the Unity camera with a script and it should work.

8 Answers

0 votes
answered Aug 28, 2019 by nigelkp9 (220 points)
Was your video feed rotated 180 degrees, or was it mirrored? In the forum post you linked, they are experiencing a mirrored video feed, I am just wondering as ours is rotated 180 degrees. I implemented your fix and the video is rotated correctly, but the augmented content is still rotated 180 degrees
commented Aug 29, 2019 by spaceagent (1,290 points)
Yes, the video feed was rotated 180 degrees, the forum post is not related directly with this problem, but it shares the same cause. Also in our case the augmented content was always showing the correct way, we had problems only with the camera feed.
Check a sample scene to see if it is a problem from your scene or from EasyAR.
commented Aug 29, 2019 by nigelkp9 (220 points)
Thank you for the suggestion, I will continue on. Seems like I have two separate issues then
0 votes
answered Nov 10, 2019 by rahnaaqmalina (140 points)
the link is not working, can u reupload again the modified shaders, please..
commented Nov 11, 2019 by spaceagent (1,290 points)
I've updated the link, but to let you know, in the end we didn't use this solution because it is still with bugs. For iOS we switched to OpenGLES graphics API, which is fully supported by EasyAR
0 votes
answered Apr 15, 2020 by marcin77 (230 points)
Hello,

I used attached, updated shaders and also changed Camera.forceIntoRenderTexture setting (in CameraImageRenderer class) and now video I'm recording using NatCorder is recorded correctly (not rotated 180deg).

I didn't wanted to change iOS graphics API to GLES3 but I wanted to stick with Metal so I applied described fix and it works for me.

Best regards
0 votes
answered Apr 24, 2020 by jinnin0105 (150 points)

This problem is very simple

 - CameraImageRenderer.cs:140

targetCamera.projectionMatrix = projection * e.ImageRotationMatrixGlobal.inverse;

to

targetCamera.projectionMatrix = projection;

and you can add bool option

if (InverseMatrix)

    targetCamera.projectionMatrix = projection * e.ImageRotationMatrixGlobal.inverse;

else

    targetCamera.projectionMatrix = projection;

this perfectly works  no need shaders edit

and screen rotation is works

0 votes
answered Nov 6, 2020 by kenn (18,750 points)

We have made a fix (or workaround) for this issue; it will be released in the coming update. The changes are described bellow in case you need it in an older release. (There are some differences in the plugin code between versions, so it may look different.)


1. Replace the whole imageProjection calculation code in CameraImageRenderer.cs,

it may look like

bool cameraFront = ...
var imageProjection = cameraParameters.imageProjection(...
if (renderImageHFlip)
{
    ...
    imageProjection = ...;
}

material.SetMatrix("_TextureRotation", imageProjection);

replace it with

var imageProjection = ImageProjection(cameraParameters, controller.TargetCamera.aspect, arSession.Assembly.Display.Rotation, renderImageHFlip);
material.SetMatrix("_TextureRotation", imageProjection);

ImageProjection method is from internal CameraParameters.imageProjection implementation, with modifications for UV conversion to replace original vertex conversion.

private static Matrix4x4 ImageProjection(CameraParameters parameters, float viewportAspectRatio, int screenRotation, bool manualHorizontalFlip)
{
    var size = parameters.size();
    float imageAspectRatio = (float)size.data_0 / size.data_1;
 
    var imageRotationDegree = parameters.imageOrientation(screenRotation);
    var imageRotation = imageRotationDegree * Mathf.Deg2Rad;
 
    var requiredImageAspectRatio = (imageRotationDegree % 180) == 0 ? viewportAspectRatio : 1.0f / viewportAspectRatio;
 
    Matrix4x4 scale = Matrix4x4.identity;
    if (imageAspectRatio < requiredImageAspectRatio)
    {
        scale.m11 = imageAspectRatio / requiredImageAspectRatio;
    }
    else
    {
        scale.m00 = requiredImageAspectRatio / imageAspectRatio;
    }
 
    Matrix4x4 rotation = Matrix4x4.identity;
    rotation.m00 = Mathf.Cos(-imageRotation);
    rotation.m01 = -Mathf.Sin(-imageRotation);
    rotation.m10 = Mathf.Sin(-imageRotation);
    rotation.m11 = Mathf.Cos(-imageRotation);
 
    var flip = Matrix4x4.identity;
    if (manualHorizontalFlip)
    {
        flip.m00 = -1;
    }
 
    var imageProjection = scale * rotation * flip;
    return imageProjection;
}

2. Replace one line in all CameraImage_* shaders

replace

o.vertex = mul(_TextureRotation, o.vertex);

with

o.uv = MultiplyUV(_TextureRotation, o.uv - float2(0.5, 0.5)) + float2(0.5, 0.5);


There are some other partial workarounds in previous released samples, when you use the above solution, those workarounds should be removed. Search for metal in the whole project including shaders and scripts and remove those code paths. 

commented Jan 12, 2021 by ikaro88 (280 points)
give me the error:
arsession does not exist in the correct context
commented Jan 12, 2021 by kenn (18,750 points)
store a class private reference of session from OnAssemble with type ARSession
commented Jan 13, 2021 by ikaro88 (280 points)
I tried but it tell me:
'ARAssembly does not contain a definition for 'Display' and no extension metod 'Display' accepting a first argument

I am working with easyar 4 and no 3
0 votes
answered Jan 12, 2021 by ikaro88 (280 points)
I have to put the shades to some gameobjects in some way?
0 votes
answered Jun 29, 2021 by kenn (18,750 points)
IMPORTANT NOTICE: This issue has been fixed in EasyAR Sense Unity Plugin 4.2 and later.

If you come into this thread from external links, please download and use the latest releases.
0 votes
answered Sep 8, 2022 by ericstine (180 points)

Thanks for providing such kind of platform where users can see whatever they want and share their awesome content in one place. Hindi Question answer

Welcome to EasyAR SDK Q&A, where you can ask questions and receive answers from other members of the community.
...