(FIX FOUND) EasyAR 3.0.1 Unity iOS camera feed upside down using HDR

+5 votes
asked Aug 28, 2019 by spaceagent (1,230 points)
edited Nov 11, 2019 by spaceagent

The camera feed is upside down on iOS devices using Metal graphics API and with HDR enabled in Graphics settings in Unity. It looks like it is a shader problem related to platform-specific rendering differences.

UPDATE: We managed to find the solution and another related Unity bug.

First the problem is with the EasyAR shaders as initially tought is related to platform-specific rendering differences, second it is a problem with how Unity render its camera and how HDR changes that, as we mentioned in this update.

To fix this we modified all EasyAR shaders to flip the texture uv when #if UNITY_UV_STARTS_AT_TOP is true and a script that set forceIntoRenderTexture to true on the Unity camera, that is doing the same thing that enabling HDR do, except the actual HDR effect.

Here is a link with the modified shaders, just enable forceIntoRenderTexture on the Unity camera with a script and it should work.

5 Answers

0 votes
answered Aug 28, 2019 by nigelkp9 (220 points)
Was your video feed rotated 180 degrees, or was it mirrored? In the forum post you linked, they are experiencing a mirrored video feed, I am just wondering as ours is rotated 180 degrees. I implemented your fix and the video is rotated correctly, but the augmented content is still rotated 180 degrees
commented Aug 29, 2019 by spaceagent (1,230 points)
Yes, the video feed was rotated 180 degrees, the forum post is not related directly with this problem, but it shares the same cause. Also in our case the augmented content was always showing the correct way, we had problems only with the camera feed.
Check a sample scene to see if it is a problem from your scene or from EasyAR.
commented Aug 29, 2019 by nigelkp9 (220 points)
Thank you for the suggestion, I will continue on. Seems like I have two separate issues then
0 votes
answered Nov 10, 2019 by rahnaaqmalina (140 points)
the link is not working, can u reupload again the modified shaders, please..
commented Nov 11, 2019 by spaceagent (1,230 points)
I've updated the link, but to let you know, in the end we didn't use this solution because it is still with bugs. For iOS we switched to OpenGLES graphics API, which is fully supported by EasyAR
0 votes
answered Apr 15 by marcin77 (140 points)
Hello,

I used attached, updated shaders and also changed Camera.forceIntoRenderTexture setting (in CameraImageRenderer class) and now video I'm recording using NatCorder is recorded correctly (not rotated 180deg).

I didn't wanted to change iOS graphics API to GLES3 but I wanted to stick with Metal so I applied described fix and it works for me.

Best regards
0 votes
answered Apr 24 by jinnin0105 (150 points)

This problem is very simple

 - CameraImageRenderer.cs:140

targetCamera.projectionMatrix = projection * e.ImageRotationMatrixGlobal.inverse;

to

targetCamera.projectionMatrix = projection;

and you can add bool option

if (InverseMatrix)

    targetCamera.projectionMatrix = projection * e.ImageRotationMatrixGlobal.inverse;

else

    targetCamera.projectionMatrix = projection;

this perfectly works  no need shaders edit

and screen rotation is works

0 votes
answered Nov 6 by kenn (8,210 points)

We have made a fix (or workaround) for this issue; it will be released in the coming update. The changes are described bellow in case you need it in an older release. (There are some differences in the plugin code between versions, so it may look different.)


1. Replace the whole imageProjection calculation code in CameraImageRenderer.cs,

it may look like

bool cameraFront = ...
var imageProjection = cameraParameters.imageProjection(...
if (renderImageHFlip)
{
    ...
    imageProjection = ...;
}

material.SetMatrix("_TextureRotation", imageProjection);

replace it with

var imageProjection = ImageProjection(cameraParameters, controller.TargetCamera.aspect, arSession.Assembly.Display.Rotation, renderImageHFlip);
material.SetMatrix("_TextureRotation", imageProjection);

ImageProjection method is from internal CameraParameters.imageProjection implementation, with modifications for UV conversion to replace original vertex conversion.

private static Matrix4x4 ImageProjection(CameraParameters parameters, float viewportAspectRatio, int screenRotation, bool manualHorizontalFlip)
{
    var size = parameters.size();
    float imageAspectRatio = (float)size.data_0 / size.data_1;
 
    var imageRotationDegree = parameters.imageOrientation(screenRotation);
    var imageRotation = imageRotationDegree * Mathf.Deg2Rad;
 
    var requiredImageAspectRatio = (imageRotationDegree % 180) == 0 ? viewportAspectRatio : 1.0f / viewportAspectRatio;
 
    Matrix4x4 scale = Matrix4x4.identity;
    if (imageAspectRatio < requiredImageAspectRatio)
    {
        scale.m11 = imageAspectRatio / requiredImageAspectRatio;
    }
    else
    {
        scale.m00 = requiredImageAspectRatio / imageAspectRatio;
    }
 
    Matrix4x4 rotation = Matrix4x4.identity;
    rotation.m00 = Mathf.Cos(-imageRotation);
    rotation.m01 = -Mathf.Sin(-imageRotation);
    rotation.m10 = Mathf.Sin(-imageRotation);
    rotation.m11 = Mathf.Cos(-imageRotation);
 
    var flip = Matrix4x4.identity;
    if (manualHorizontalFlip)
    {
        flip.m00 = -1;
    }
 
    var imageProjection = scale * rotation * flip;
    return imageProjection;
}

2. Replace one line in all CameraImage_* shaders

replace

o.vertex = mul(_TextureRotation, o.vertex);

with

o.uv = MultiplyUV(_TextureRotation, o.uv - float2(0.5, 0.5)) + float2(0.5, 0.5);


There are some other partial workarounds in previous released samples, when you use the above solution, those workarounds should be removed. Search for metal in the whole project including shaders and scripts and remove those code paths. 

Welcome to EasyAR SDK Q&A, where you can ask questions and receive answers from other members of the community.
...