Change the byte array content in a frame, or create a new frame with another byte array

Question

I'm trying to get image processing going using opencv, and I managed to extract the data array from each frame and pass that to opencv Mat, then convert it to GrayScale, but now I want to render the new "frame", how can I construct a new frame object in c++ given the data array? is that's even possible ?

any help is appreciated, thanks.

Answer 1

It is impossible using current EasyAR API.

You can take image data out from EasyAR, but there are no interfaces to input image data into EasyAR, both for algorithms and rendering.

We have designed such features and we have some API compatibilities for this. But those features are not ready yet. We still need some time to polish those interfaces (Image, Frame, FrameStreamer, and Buffer which will be introduced in 2.1), especially for all language bindings including Unity.

Back to your question, if you only need to render a new image you created, you do not need EasyAR. EasyAR rendering feature is extremely thin and you can write your own rendering code, for example, if you are using GLES for rendering, it is as simple as drawing two rectangles with a texture image.

Comments

thank you for your answer, I will be waiting for the new version,
can you provide a sample code (c++) for rendering a byte array ( I'm editing HelloAR sample code), so that instead of using videobg_renderer->render(frame, viewport);
I want to implement something like render(byte array, viewport);
could you please guide me though this, I would like to use GLES but no idea how.

---

I have to say 2.1 is near release but still user input is not supported. Hope we won't make you wait too long.

For the rendering, try read and understand the code of rendering a color box in the sample, then change the C++ code and the shader code a little bit, basically you need to,
1. build a shader to draw using texture ,
vertex shader:
attribute vec4 coord;
attribute vec2 texCoord;
varying vec2 texc;

void main(void)
{
    gl_Position = coord;
    texc = texCoord;
}

fragment shader:
uniform sampler2D texture;
varying vec2 texc;

void main(void)
{
    gl_FragColor = texture2D(texture, texc);
}

2. some code to build gl program and setup textures
3. upload texture using glTexImage2D
4. use glDrawArrays to draw two triangles

Sorry not have enough time to write all the code, it is only some hints, and also you can search on google for how to draw cv::Mat using GLES, there are some related urls, like this
https://stackoverflow.com/questions/16809833/opencv-image-loading-for-opengl-texture

---

thank you so much for your help