Easy AR 3.0 Unity Doesn't work on Android nor on Windows

Question

It doesn't work on Motorola G6/G6Plus/One with last Android, nor on ASUS 7" Tablet with 4.4 Kitkat, EasyAR 2.3 used to work fine on all of these devices, the only thing it needed was x64 Android Support, pretty sad... because we must move to an alternative, pretty sad, because 2.3 worked really fine, but it is unusable right now.

It is also not working on Windows, web cams are not recognized, they even doesn't turn on, and black screens is the only view.

PD: the message at the beginning with the EasyAR version is mandatory?, it breaks all the UX design

Answer 1

Same problem here!

SDK 2.3 was amazing, I needed too x64 android support too. Now the camera can't focus, troubles with webcam, UI is hiding, it takes a long time to recognize targets... and of course that message of EasyAR version.

Comments

you can find 'GUIPopup.AddShowMessage' in EasyARBehaviour.cs , and remove it. mabay the ui will not work

---

Which UI is hidde and what is your phone model? Can you tell me what sample did you run?  ImageTracker is created using ImageTracker.create, ImageTracker.createWithMode(ImageTrackerMode.PreferQuality) or ImageTracker.createWithMode(ImageTrackerMode.PreferPerformance)？

---

it all can work, If your phone performs well,use ImageTrackerMode.PreferQuality. if not use ImageTrackerMode.PreferPerformance

---

I've tried simple ImageTarget sample. I built it for Galaxy Note 8 and iPhone 6S. I had some UI for the camera in 3 different scenes, that UI was "extended", it's like the canvas expands about to 200%, border images are hidden and central images are visible. And now the camera can't focus, I've tried whole 4 options of focus and it still cannot focus... is there an exact way to make the camera focus?

---

it is a bug ,You should set focus affter you turn on camera, you can find the code in ARsession.cs on line 94

Answer 2

UPDATE!

Webcams on Windows still doesn't work

I was able to create a project from ground, with 13 image targets, duplicating the sample scene for multi_target, I wasn't able to start from the ground yet, some comments:

• For image tracking you must ensure that Easyar Camera Prefer is set to "Prefer Object Sensing", if you use other setting here and you are trying to track images, you are going to view a black screen
• Each image marker is 512x512x72dpi
• None of the Camera Focus Mode Options worked, editing ARsession.cs, moving the line easyarCamera.setFocusMode(CameraFocusMode); from line 96 after easyarCamera.start(); on line 106 worked, I think that the camera must be started before can be focused, on some old devices doesn't work this solution despite their cameras has autofocus functionality
• The markers are not well detected if are projected on a screen, if are printed work much better

For now its work now on Moto G6 Plus latest Android, and on Asus HD7 Android 4.2.2

I'm going to test on Apple iPad Mini 2/4, iPad AIR, iPhone SE, iPhone 6s, when finish this testing I am going to feedback here

Comments

PreferObjectSensing will use Android camera1, and other will use Android camera2 in most cases depending on the device and Android OS version. We experienced many compatibility problems regarding camera2, mostly on old devices, but camera2 gives timestamps for camera images, which are very important for SurfaceTracking and other new functionalities in the coming EasyAR 4.0. We recommend to use PreferObjectSensing for best compatibility if you don't need the new functionalities.
The focus mode problem is a bug in the Unity plugin and can be fixed with what you described. We have experienced compatibility problems with autofocus with camera2 but not camera1 with proper hardware. Could you tell me the names of the devices?
For the Microsoft Webcam, LifeCAM HD-3000 720p camera you mentioned on another thread, could you try commenting out ARSession.cs: 104 "easyarCamera.setSize(new Vec2I((int)CameraSize.x, (int)CameraSize.y));" and check if it works? We found a possible problem with CameraDevice.setSize, but the camera model you mentioned is not sold in China and we can not confirm if that's the cause of your problem. We have changed the camera implementation on Windows for package size reduction.

---

Thanks for your answer, it is very important this feedback so we can still be using this fantastic technology and work for its democratization...

Regarding to the cameras: The webcam problem, is in windows desktop while I am in the Unity editor, it is a faster production pipeline when changes can be tested without publishing, I used 2 webcams LifeCAM HD-3000 that worked pretty well with 2.3 SDK, and another generic USB 2.0 webcam for testing this situation, that it has a label "Made in China" ;), I think it sold in America as StarTec brand, and doesn't work.

Commenting out the code you mention, does exactly the same, it's more a missing hardware activation or something, the hardware it self is not detected, this is the message related to it:

[EasyAR] open camera failed
UnityEngine.Debug:Log(Object)
easyar.ARSession:Init() (at Assets/EasyAR/Scripts/ARSession.cs:100)
easyar.ARSession:<Awake>b__23_0(PermissionStatus, String) (at Assets/EasyAR/Scripts/ARSession.cs:73)
easyar.Detail:FunctorOfVoidFromPermissionStatusAndString_func(IntPtr, PermissionStatus, IntPtr, IntPtr&) (at Assets/EasyAR/Scripts/csapi.cs:2497)
easyar.Detail:easyar_DelayedCallbackScheduler_runOne(IntPtr)
easyar.DelayedCallbackScheduler:runOne() (at Assets/EasyAR/Scripts/csapi.cs:5332)
easyar.EasyARBehaviour:Update() (at Assets/EasyAR/Scripts/EasyARBehaviour.cs:115)

I can confirm that EasyAR is working well on Moto G6 plus right now, x32 and x64, one time freezed cell phone with x64 version, the performance is very good, I have a scene with 13 imageTargets and one Tracker, with only 1 at a time, less than a second to change between targets.

I can confirm it works on ASUS HD7 7 inch tablet with Android 4.2.2, it's not as fluid as on G6, but it's stable, with this old device the autofocus doesn't work at all.

I can confirm that works pretty smooth on iPad Mini 4, iPhone SE, iPhone 6s, a little bit less fluid in iPad AIR 1 and iPad Mini 2, I assume that on iPhones above 6s it will work pretty well.

I can confirm that if the xcode project is generated on windows, then move it to a mac for compile it, it fails, but if you open the project on Unity macOS and generate the xcode project, it compiles well, with some warnings. It would be pretty usefull if the xcode project generated on windows compiled well, this will accelerate a little bit the working pipeline.

I don't wanna remove the popup at the beginning with the EasyAR version, you deserve that, but it is breaking the UX, because is a little bit intrusive, so I modified the EasyAR default GUISkin to be more discreet, but can be remove it at all if one want, commenting line 103 on EasyARBehaviour.cs ;), but I think, if you are developing this amazing technology, we must support it.

This is the Lifecam I use: https://www.microsoft.com/accessories/en-us/products/webcams/lifecam-hd-3000/t3h-00011

---

Is there a way to obtain an EasyAR 3.0 Unity API documentation?

---

For the webcam problem, could you run the diagnostic program and see what happens?
https://github.com/IanusInferus/typemake/files/3388510/MFCaptureD3D.zip
If it runs normally, a camera view will be shown. If there is any error, a dialog box will be shown with error code and error message. You can copy the message by activating the dialog box and press Ctrl+C.

For the ASUS HD7 7 inch tablet autofocus problem, you can try
https://github.com/google/cameraview or https://github.com/IanusInferus/typemake/files/3388539/demo-debug.zip
to check if the Google official sample works correctly on the device.

For the Unity XCode project on Windows, could you give your Unity version for us to investigate?

For the EasyAR version shown at startup, it is intended for test purpose only and we will remove it in the next version.

For the Unity API documentation, a unified API documentation is under translation to English and will be online soon. you may check for type declarations temporarily at https://www.easyar.cn/doc/EasyAR%20SDK/API%20Reference/3.0/3.0.html

---

Thanks a lot for your answer!!

The diagnostic program fails with message:

"Cannot create a video capture device (hr=0xC00D36B9)
An object ran out of media types to suggest therefore the requested chain of streaming objects cannot be completed."

The webcams are detected by the program because are shown in the select device dialogue-box, but after select any of them it fails with the above message

The cameras are fully functional on the camera utility from Windows 10

--

The demo for HD7 works well, a little bit laggy, but it works fine.

--

The Unity version with what I am testing right now for iOS from Unity Windows is 2019.1.8f1, the same version on macOS generated good Xcode projects that compiled as expected, but the generated from Unity Windows still faililg. I am using Xcode 10.2.1 on macOS Mojave

--

Don't remove the SDK version at all, let it as something optional, but without the consequences I'm experiencing right now, a delay of 3 to 4 seconds before the camera is activated and detection begins

--

Thanks a lot for the documentation, although my current Chinese level is below zero ;), I have a very good translator.

-- UPDATE

I successfully tested EasyAR 3 on Huawei P9 Lite (Nuguet) with pretty good performance despite is a middle range cell phone.

---

I think the media types(YUY2, NV12, etc) in your camera are not supported.
Please try this tool and see what the dialog MediaTypes shows.
https://github.com/IanusInferus/typemake/files/3389005/MFCaptureD3D_2.zip

-

Is HD7's autofocus functions correctly with the demo?

-

We will investigate the Unity-XCode problem.

---

What's your Windows version? Is it Windows 7?

---

It is Windows 10 x64, Version 1903 (OS Build 18362.239)

A short video showing what is happening with EasyAR 3:
https://www.dropbox.com/s/fq41p2y93xdfuf2/Desktop%202019.07.14%20-%2010.38.36.01.mp4?dl=0

This is exactly the same system with the same camera but using EasyAR 2.3:
https://www.dropbox.com/s/uw2r10l4n50ebng/Desktop%202019.07.14%20-%2010.47.49.02.mp4?dl=0

---

Please try this tool and see what the dialog MediaTypes shows.
https://github.com/IanusInferus/typemake/files/3389005/MFCaptureD3D_2.zip

I have tested it on a Logitech Webcam C170 on Windows 10 x64 1903, which shows it supports the following media types.
---------------------------
MediaTypes
---------------------------
32595559 YUY2
3231564e NV12
47504a4d MJPG

EasyAR 3.0 now supports NV12 but not YUY2. I suspect that your camera lacks the support of NV12.
I have also found C170 lacks NV12 on Windows 7 in a virtual machine.
EasyAR 2.3 uses a library module which depends on the deprecated DirectShow. We dropped that library module and reimplemented the functionality based on Media Foundation, which is the recommended choice by Microsoft. But it that seems we have not covered enough devices and Windows versions, we have mostly done tests on Windows 10 and Logitech cameras.

---

It works pretty well, and you're totally right, it only supports 32595559 YUY2 and lacks on the support for NV12, if I use any webcam that supports NV12 its going to work fine?

---

Well, I was able to make it work, the issue is caused by the latest windows 10 driver, I have rolled back to the previous driver for the LifeCAM HD-3000 ver 10.0.18362.1, and now this camera is compatible with NV12, YUY2 and MJPG, now I can test on desktop. So you can recommend to users with similar webcams to do that as a possible solution. The difference is that with the newest driver the webcam is detected as an Imagine Device, now is detected as a Camera Device.

---

I have the same webcam and rolling back the driver did the trick! :)

I have another issue now, unfortunately. Any idea why EasyAR fails to initiliaze on Android? Using latest Windows, latest Unity (2019.1.10f1), latest EasyAR on a OnePlus 3 with latest Android.

The exact error message is: "EasyAR initialization failed". And, yes, I do have the key setup correctly.

---

If you are tracking images, ensure that in ARSession in the option "Easyar Camera Prefer" you have selected "Prefer Object Sensing", using this option I've stop experimenting issues on Android.

---

The key's package name need to match the one in the app.
The key's type (Basic or Pro) need to match the SDK's type.
Please refer to https://www.easyar.com/doc/EasyAR%20SDK/Getting%20Started/Getting-Started-with-EasyAR.html#get-a-key

---

Thanks,  juvelez. We will add YUY2 support in a patch version, which will be released recently.

---

Made a separate, clean project. Loaded up HelloAR_ImageTarget.

Verified everything from Project Settings, to Inspector, to the key and bundle name... you name it. It works on Windows without any issues. It does not work on Android (gives the previous error message).

---

We don't check any bundle name on Windows.
That error message means either the key is invalid, or there is something wrong with the unity activity, which is very unlikely.
Do you get any error log from logcat?

---

Just checked.

1) Unable to find libEasyAR.
2) AndroidJavaException: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/app/org.buzauland.testarapp-[redacted]/lib/arm64/libEasyAR.so" not found
 java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/app/org.buzauland.testarapp-[redacted]/lib/arm64/libEasyAR.so" not found

Inside Unity, I can confirm the folders are correct and that all plugins have the correct settings applied.

---

Looks like the gradle project created by Unity only includes the .jar, but not the .so.
Not sure how to fix this.

---

Please follow Android's official guide at https://developer.android.com/distribute/best-practices/develop/64-bit
In the "Unity developers" section, check your Unity version, change your scripting backend to IL2CPP, select ARM64 target architecture.
And select Assets\Plugins\Android\libs\arm64-v8a\libEasyAR.so, in the inspector, make sure that the platform is set to Android and CPU is set to ARM64.

---

Thanks for the reply.
I can confirm that everything is in order in regards to the project settings/setup.
If I didn't specify it earlier, I am using the official HelloAR_ImageTarget scene with Unity 2019.1.10f1. Nothing has been modified.

---

Updated to the latest 3.0.1

Logcat reports the same error: /lib/arm64/libEasyAR.so is nowhere to be found. Only the message inside the app has changed to "Invalid key".

This is using the official package HelloAR_ImageTarget. Project settings are correct. Libs under the Plugins folder are setup properly.

Why isn't the lib included?

---

Something has flashed through my memory.
Do you, by any chance, place your Unity/gradle project in a deep directory that a temporary file path may be longer than 260 characters, which is what Android Studio/gradle permits on Windows?

---

No, the path to the Unity project is 24 characters long.

---

Have you checked whether libEasyAR.so is inside your apk file? (You can rename it to a zip file and view its content.) It shall be located at lib\arm64-v8a\libEasyAR.so inside your apk.  The path /lib/arm64/libEasyAR.so is strange as it is called arm64-v8a on Android. "Invalid key" is also strange as it is not in C# code but in C++ code in libEasyAR.so, which is not loaded according to your description.

---

Oh, we're on to something now. I also forgot to specify that I'm making a bundle.
Anyway, opened the .aab and found it in "base\lib\arm64-v8a\", but NOT in "base\lib\x86\".

So, somewhere, something expects to find the lib in a different location.

---

Could you try disabling "Build App Bundle" and check if it works?

---

That was it, thank you!!

I disabled the bundle, but left ARM64 and x86 checked. When I opened the apk, the lib folder is identical. No EasyARlib.so in x86. Is that normal?

---

It's normal as EasyAR doesn't support Android x86 and there is no libEasyAR.so for x86. It is OK if you don't need to support x86 devices. If you need to support x86 devices, you may need to find a way to remove the x86 lib directory entirely. There are two way to support Android x86 devices: 1) provide complete so libraries for x86; 2) provide no so libraries for x86, and rely on libhoudini to compile arm-v7a libraries to x86. Anyway, they are very old devices and you may not bother to support them.
We will investigate the Android app bundle issue. Thank you.

---

Update!

I've tested Easy AR3.0.1 on iPad Mini2/4 iPad AIR and iPhone6s, and I have to say that the performance is pretty poor, on iPad Mini 2 and iPad AIR, it's lagy I could say 15 maybe 20 fps max, on iPad Mini 4 and iPhone 6s, much less lag but is not smooth at all.

Any advise on how to improve performance on iOS?, I get better performance on a old Android, Moto G3 with android 6 than on iPad Mini 2/AIR

Playing around with the Graphics APIs, I have tested Metal over OpenGLES2 and OpenGLES3 over OpenGL2, and even Auto, and the graphics performance had an evident improvement, but the image detection get worst, with openGLES2 the detection is much more stable, but the graphics performance it's pretty poor.

Another test I did was to scale graphics, adjusting the Fixed DPI to 150, an evident performance gain can be obtained with that, but still feeling lagy, and the image detection is not stable. And all the graphic markers get 5 stars on the testing tool.

I really thought that on iOS will run better than on Android, but it seems to be the contrary.

---

You can try passing ImageTrackerMode.PreferPerformance to ImageTracker.createWithMode for old devices.
https://www.easyar.com/doc/EasyAR%20SDK/API%20Reference/3.0/ImageTracker.html#createwithmode

---

I've tried that too, but the tracking become really unstable, I am using the Unity SDK,

- Is there any performance impact changing the size and resolution of the markers?, I've tried 1024x1024, and 512x512, I thing the recognition is faster using smaller markers, but I'm not really sure if this have something to do with the performance

- The Camera size on ARSession has any performance impact?, I tested all kind of size combinations

- What are Fork Output Num and Join Num for?, If a number different than 2 is used, the camera is black

- ARShader needs all of those shaders at once to work?

---

The pixel size of markers don't have much impact on the performance.
Camera size may contributes to the performance, but there is a known drawback on the current iOS camera implementation in EasyAR that it only supports 640x480, 1280x720 and 1920x1080, which will be fixed in EasyAR 4.
I have tested on an iPad Mini 2 and found that increasing the frame rate can make it somewhat smoother. https://www.easyar.com/doc/EasyAR%20SDK/API%20Reference/3.0/CameraDevice.html#setframeraterange
Fork and Join are dataflow utility nodes used for branching and merging parallel flows. The numbers are used to change the number of output or input ports. You need to connect these ports, or the flow may stuck at Join. https://www.easyar.com/doc/EasyAR%20SDK/API%20Reference/3.0/Overview.html#dataflow
Different operating systems and camera hardwares require different pixel formats. In the current implementation, only YUV_NV12 is needed on iOS.

---

Where is the right place to increase the frameRateRange? I placed this line easyarCamera.setFrameRateRange(3); just after easyarCamera.setFocusMode(CameraFocusMode); in ARSession.cs, is it right? which are the valid FrameRateRange Values?, But I can't see any performance improvement in my iPad mini 2, What index did you use on iPad Mini 2 to improve?

I am using 256x256 markers and I fill the recognition is done a little bit faster than with 512x512, but after recognition phase, all seems to be the same

Is it safe to use Metal as a Graphics API? or Auto?

Which are the recommended values for Fork Output Num and Join Num?

The performance on the iPad mini 4 is quite better than mini 2, but why? the hardware difference are not significant: PowerVR G6450 vs PowerVR G6430, Dualcore 1.3 vs Dualcore 1.5, are you sure you are not doing anything?, because profiling the apps, I get exactly the half FPS on iPad mini 2, ~15fps, while on iPad mini 4 is ~30fps

---

I think it's maybe the difference of default frame rate of camera image capturing.
You can add the following code in ARSession.cs after easyarCamera.setSize.

var n = easyarCamera.supportedFrameRateRangeCount();
for (int k = 0; k < n; k += 1)
{
    var lower = easyarCamera.supportedFrameRateRangeLower(k);
    var upper = easyarCamera.supportedFrameRateRangeUpper(k);
    if ((lower == 30) && (upper == 30))
    {
        easyarCamera.setFrameRateRange(k);
        UnityEngine.Debug.Log("setFrameRateRange: " + lower.ToString() + " " + upper.ToString());
    }
}

---

Well I did exactly what you said and I get exactly 12fps:
https://www.dropbox.com/s/xtcbcvvbl2824bs/Screenshot%202019-08-16%2011.38.01.png?dl=0

The only way I achieve a performance improvement is using Metal or Auto in Graphics API, if I use OpenGLES2 or 3 I get 12fps, but the problem with metal is the recognition became unstable.

---

Due to the information on your instructions to setup EasyAR on Unity where you indicate that OpenGLES2 must be included on the Graphics API are the doubts about using the non opengl APIs.

It would be pretty useful for us to be sure that any non-opengl graphics api can be used, and to solve issues presented when those apis are used.

In my case what happens is that the image recognition become unstable when the camera is approached toward the marker, somehow framerate drops from 30fps to less than 5, and some time later it recovers when camera is far again, or losing the target then found it again helps sometimes for this recovery.