From 723b79a3c1378d669ed90484ffd545451c80dd50 Mon Sep 17 00:00:00 2001 From: jinyu Date: Tue, 15 Oct 2024 17:57:04 +0800 Subject: [PATCH 01/19] liveshow --- dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita | 1 + dita/RTC-NG/API/enum_videoapplicationscenariotype.dita | 7 +++++++ dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap | 7 +++++++ dita/RTC-NG/config/keys-rtc-ng-api-ios.ditamap | 7 +++++++ dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap | 7 +++++++ dita/RTC-NG/config/keys-rtc-ng-api-macos.ditamap | 7 +++++++ dita/RTC-NG/config/keys-rtc-ng-links.ditamap | 7 +++++++ 7 files changed, 43 insertions(+) diff --git a/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita b/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita index 5142f5585d6..daa5908075f 100644 --- a/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita +++ b/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita @@ -78,6 +78,7 @@

(2) 适用于视频 1v1 通话场景。针对该场景低延迟、高画质的体验要求，SDK 进行了策略调优，提升了画质、首帧出图、中低端机延迟及弱网流畅度等性能表现。

+

(3) 适用于场景。针对该场景对画质清晰度和首帧出图时间的高要求，SDK 进行了策略调优，重点提升了画质表现和首帧出图体验，同时增强了在弱网环境和低端设备上的画质和流畅度表现。

diff --git a/dita/RTC-NG/API/enum_videoapplicationscenariotype.dita b/dita/RTC-NG/API/enum_videoapplicationscenariotype.dita index a2092fd8af5..6d3aa5285a9 100644 --- a/dita/RTC-NG/API/enum_videoapplicationscenariotype.dita +++ b/dita/RTC-NG/API/enum_videoapplicationscenariotype.dita @@ -33,6 +33,13 @@

+ + + +

3: 秀场直播

+

+ + diff --git a/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap b/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap index bf4073b8d0e..47598a914a3 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap @@ -11807,6 +11807,13 @@ + + + + APPLICATION_SCENARIO_LIVESHOW + + + diff --git a/dita/RTC-NG/config/keys-rtc-ng-api-ios.ditamap b/dita/RTC-NG/config/keys-rtc-ng-api-ios.ditamap index 1bdc38893e8..fac4aadfa53 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-api-ios.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-api-ios.ditamap @@ -10912,6 +10912,13 @@ + + + + AgoraApplicationLiveShowScenario + + + diff --git a/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap b/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap index dd3da6e473e..f909d3ac6bc 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap @@ -10672,6 +10672,13 @@ + + + + APPLICATION_SCENARIO_LIVESHOW + + + diff --git a/dita/RTC-NG/config/keys-rtc-ng-api-macos.ditamap b/dita/RTC-NG/config/keys-rtc-ng-api-macos.ditamap index 1b4ef97a989..251e3519f5a 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-api-macos.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-api-macos.ditamap @@ -10357,6 +10357,13 @@ + + + + AgoraApplicationLiveShowScenario + + + diff --git a/dita/RTC-NG/config/keys-rtc-ng-links.ditamap b/dita/RTC-NG/config/keys-rtc-ng-links.ditamap index 5611004f6b7..4ef860372ad 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-links.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-links.ditamap @@ -90,6 +90,13 @@ + + + + 秀场直播 + + + From 306aaf767eecdd60c3f85a3a574eccc8f923b2c6 Mon Sep 17 00:00:00 2001 From: jinyu Date: Tue, 15 Oct 2024 18:03:38 +0800 Subject: [PATCH 02/19] resolve comments --- dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita b/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita index daa5908075f..8928324f7c3 100644 --- a/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita +++ b/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita @@ -78,7 +78,7 @@

(2) 适用于视频 1v1 通话场景。针对该场景低延迟、高画质的体验要求，SDK 进行了策略调优，提升了画质、首帧出图、中低端机延迟及弱网流畅度等性能表现。

-

(3) 适用于场景。针对该场景对画质清晰度和首帧出图时间的高要求，SDK 进行了策略调优，重点提升了画质表现和首帧出图体验，同时增强了在弱网环境和低端设备上的画质和流畅度表现。

+

(3) 适用于场景。针对该场景对首帧出图时间和画质清晰度的高要求，SDK 进行了策略调优，重点提升了首帧出图体验和画质表现，同时增强了在弱网环境和低端设备上的画质和流畅度表现。

From d3e640ac8e6d4246515d78348767af8596f84396 Mon Sep 17 00:00:00 2001 From: jinyu Date: Wed, 16 Oct 2024 15:13:13 +0800 Subject: [PATCH 03/19] eglcontext --- ...rtcengine_setexternalremoteeglcontext.dita | 65 +++++++++++++++++++ dita/RTC-NG/API/rtc_api_overview.dita | 4 ++ dita/RTC-NG/RTC_NG_API_Android.ditamap | 1 + dita/RTC-NG/RTC_NG_API_CPP.ditamap | 1 + .../RTC-NG/config/keys-rtc-ng-api-cpp.ditamap | 7 ++ .../config/keys-rtc-ng-api-java.ditamap | 7 ++ .../config/relations-rtc-ng-api.ditamap | 2 + 7 files changed, 87 insertions(+) create mode 100644 dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita diff --git a/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita b/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita new file mode 100644 index 00000000000..4dce1069d21 --- /dev/null +++ b/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita @@ -0,0 +1,65 @@ + + + + <ph keyref="setExternalRemoteEglContext"/> + 设置远端视频流渲染的 EGL 环境上下文。 + + + + + + + + +
+

+ public abstract int setExternalRemoteEglContext(Object eglContext); + + + virtual int setExternalRemoteEglContext(void* eglContext) = 0; + + + + +

+
+
+
+ +
自从
+
v4.5.0
+ +
+

通过设置该方法，开发者可以替换 SDK 内部默认的远端 EGL 环境上下文，便于实现统一的 EGL 上下文管理。

+

引擎销毁时，SDK 会自动释放 EGL 环境上下文。

+ 该方法仅适用于 Android。 +
+
+ 适用场景 +

该方法适用于远端视频流输出为纹理格式，且需要进行后处理的场景，例如：硬件解码、自定义远端视频渲染。

+
+
+ 调用时机 +

该方法需要在加入频道前调用。

+
+
+ 调用限制 +

无。

+
+
+ 参数 + + + eglContext + 用于远端视频流渲染的 EGL 环境上下文对象。 + +
+
+ <ph keyref="return-section-title"/> +

方法成功调用时，无返回值；方法调用失败时，会抛出 异常，你需要捕获异常并进行处理。详见了解详情和解决建议。

+
    +
  • 0: 方法调用成功。
    • +
  • < 0: 方法调用失败。详见了解详情和解决建议。
    • +
+ + \ No newline at end of file diff --git a/dita/RTC-NG/API/rtc_api_overview.dita b/dita/RTC-NG/API/rtc_api_overview.dita index 4774af050a7..2432cba24ad 100644 --- a/dita/RTC-NG/API/rtc_api_overview.dita +++ b/dita/RTC-NG/API/rtc_api_overview.dita @@ -1313,6 +1313,10 @@ + + + （仅适用于 Android） + diff --git a/dita/RTC-NG/RTC_NG_API_Android.ditamap b/dita/RTC-NG/RTC_NG_API_Android.ditamap index 5a823f671f4..66a5d3e3884 100644 --- a/dita/RTC-NG/RTC_NG_API_Android.ditamap +++ b/dita/RTC-NG/RTC_NG_API_Android.ditamap @@ -392,6 +392,7 @@ + diff --git a/dita/RTC-NG/RTC_NG_API_CPP.ditamap b/dita/RTC-NG/RTC_NG_API_CPP.ditamap index e7f27c551b8..046d9cfa99b 100644 --- a/dita/RTC-NG/RTC_NG_API_CPP.ditamap +++ b/dita/RTC-NG/RTC_NG_API_CPP.ditamap @@ -399,6 +399,7 @@ + diff --git a/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap b/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap index 47598a914a3..4500cb04a5b 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap @@ -2611,6 +2611,13 @@ + + + + setExternalRemoteEglContext + + + diff --git a/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap b/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap index f909d3ac6bc..c8ba05b81db 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap @@ -2715,6 +2715,13 @@ + + + + setExternalRemoteEglContext + + + diff --git a/dita/RTC-NG/config/relations-rtc-ng-api.ditamap b/dita/RTC-NG/config/relations-rtc-ng-api.ditamap index d189da546bc..edfc05448dd 100644 --- a/dita/RTC-NG/config/relations-rtc-ng-api.ditamap +++ b/dita/RTC-NG/config/relations-rtc-ng-api.ditamap @@ -2425,6 +2425,8 @@ + + From ab134d5ba7e252fa18b246b057c6001ddb5e5d12 Mon Sep 17 00:00:00 2001 From: jinyu Date: Wed, 16 Oct 2024 16:28:23 +0800 Subject: [PATCH 04/19] resolve comments --- dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita b/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita index 4dce1069d21..ec09c09163f 100644 --- a/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita +++ b/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita @@ -36,7 +36,7 @@
适用场景 -

该方法适用于远端视频流输出为纹理格式，且需要进行后处理的场景，例如：硬件解码、自定义远端视频渲染。

+

该方法适用于下行视频后处理，且视频数据为纹理格式的场景，例如：硬件解码、自定义远端视频渲染。

调用时机 From a483386c74f090d2f0dad1e7f47c3c615d536f2a Mon Sep 17 00:00:00 2001 From: jinyu Date: Fri, 18 Oct 2024 17:01:24 +0800 Subject: [PATCH 05/19] colorspace --- dita/RTC-NG/API/class_externalvideoframe.dita | 129 +++++++-- dita/RTC-NG/API/class_videoframe.dita | 253 ++++++++++++++---- 2 files changed, 302 insertions(+), 80 deletions(-) diff --git a/dita/RTC-NG/API/class_externalvideoframe.dita b/dita/RTC-NG/API/class_externalvideoframe.dita index 505c28c9634..53d5a2e5209 100644 --- a/dita/RTC-NG/API/class_externalvideoframe.dita +++ b/dita/RTC-NG/API/class_externalvideoframe.dita @@ -7,20 +7,32 @@

public class AgoraVideoFrame { + public static final int FORMAT_NONE = -1; + public static final int FORMAT_TEXTURE_2D = 10; + public static final int FORMAT_TEXTURE_OES = 11; + public static final int FORMAT_I420 = 1; + public static final int FORMAT_BGRA = 2; + public static final int FORMAT_NV21 = 3; + public static final int FORMAT_RGBA = 4; + public static final int FORMAT_I422 = 16; + public static final int BUFFER_TYPE_NONE = -1; + public static final int BUFFER_TYPE_BUFFER = 1; + public static final int BUFFER_TYPE_ARRAY = 2; + public static final int BUFFER_TYPE_TEXTURE = 3; public AgoraVideoFrame() { - format = 10; + format = 10; timeStamp = 0; stride = 0; height = 0; @@ -37,53 +49,78 @@ rotation = 0; alphaStitchMode = 0; } + public int format; + public long timeStamp; + public int stride; + public int height; + public int textureID; + public boolean syncMode; + public float[] transform; + public javax.microedition.khronos.egl.EGLContext eglContext10; - public android.opengl.EGLContext eglContext14; + public android.opengl.EGLContext eglContext14; + public byte[] buf; + public int cropLeft; + public int cropTop; + public int cropRight; + public int cropBottom; + public int rotation; + public int alphaStitchMode; @Override public String toString() { - return "AgoraVideoFrame{" - + "format=" + format + ", timeStamp=" + timeStamp + ", stride=" + stride - + ", height=" + height + ", textureID=" + textureID - + ", buf.length=" + (buf != null ? buf.length : 0) + ", cropLeft=" + cropLeft - + ", cropTop=" + cropTop + ", cropRight=" + cropRight + ", cropBottom=" + cropBottom - + ", rotation=" + rotation + ", alphaStitchMode=" + alphaStitchMode + '}'; + return "AgoraVideoFrame{" + + "format=" + format + ", timeStamp=" + timeStamp + ", stride=" + stride + + ", height=" + height + ", textureID=" + textureID + + ", buf.length=" + (buf != null ? buf.length : 0) + ", cropLeft=" + cropLeft + + ", cropTop=" + cropTop + ", cropRight=" + cropRight + ", cropBottom=" + cropBottom + + ", rotation=" + rotation + ", alphaStitchMode=" + alphaStitchMode + '}'; } + } - __attribute__((visibility("default"))) @interface AgoraVideoFrame : NSObject + +__attribute__((visibility("default"))) @interface AgoraVideoFrame : NSObject @property(assign, nonatomic) NSInteger format; -@property(assign, nonatomic) CMTime time; -@property(assign, nonatomic) int stride DEPRECATED_MSG_ATTRIBUTE("use strideInPixels instead"); -@property(assign, nonatomic) int strideInPixels; -@property(assign, nonatomic) int height; + +@property(assign, nonatomic) CMTime time; +@property(assign, nonatomic) int stride DEPRECATED_MSG_ATTRIBUTE("use strideInPixels instead"); + +@property(assign, nonatomic) int strideInPixels; +@property(assign, nonatomic) int height; @property(assign, nonatomic) CVPixelBufferRef _Nullable textureBuf; + @property(strong, nonatomic) IMAGE_CLASS * _Nullable image; -@property(strong, nonatomic) NSData *_Nullable dataBuf; + +@property(strong, nonatomic) NSData *_Nullable dataBuf; @property(strong, nonatomic) NSData *_Nullable alphaBuf; @property(assign, nonatomic) AgoraAlphaStitchMode alphaStitchMode; -@property(assign, nonatomic) int cropLeft; -@property(assign, nonatomic) int cropTop; -@property(assign, nonatomic) int cropRight; -@property(assign, nonatomic) int cropBottom; -@property(assign, nonatomic) int rotation; + +@property(assign, nonatomic) int cropLeft; +@property(assign, nonatomic) int cropTop; +@property(assign, nonatomic) int cropRight; +@property(assign, nonatomic) int cropBottom; +@property(assign, nonatomic) int rotation; +@property(strong, nonatomic) AgoraColorSpace *_Nullable colorSpace; + - (void)fillAlphaData; @end - struct ExternalVideoFrame { + +struct ExternalVideoFrame { ExternalVideoFrame() : type(VIDEO_BUFFER_RAW_DATA), format(VIDEO_PIXEL_DEFAULT), @@ -99,44 +136,78 @@ eglContext(NULL), eglType(EGL_CONTEXT10), textureId(0), + fenceObject(0), metadataBuffer(NULL), metadataSize(0), alphaBuffer(NULL), fillAlphaBuffer(false), - alphaStitchMode(0), + alphaStitchMode(NO_ALPHA_STITCH), d3d11Texture2d(NULL), textureSliceIndex(0){} + enum EGL_CONTEXT_TYPE { + EGL_CONTEXT10 = 0, + EGL_CONTEXT14 = 1, }; + enum VIDEO_BUFFER_TYPE { + VIDEO_BUFFER_RAW_DATA = 1, + VIDEO_BUFFER_ARRAY = 2, + VIDEO_BUFFER_TEXTURE = 3, }; + VIDEO_BUFFER_TYPE type; + VIDEO_PIXEL_FORMAT format; + void* buffer; + int stride; + int height; + int cropLeft; + int cropTop; + int cropRight; + int cropBottom; + int rotation; + long long timestamp; - void *eglContext; + + void* eglContext; + EGL_CONTEXT_TYPE eglType; + int textureId; + + long long fenceObject; + float matrix[16]; + uint8_t* metadataBuffer; + int metadataSize; + uint8_t* alphaBuffer; + bool fillAlphaBuffer; - int alphaStitchMode; + + ALPHA_STITCH_MODE alphaStitchMode; + void *d3d11Texture2d; + int textureSliceIndex; + + ColorSpace colorSpace; }; export class ExternalVideoFrame { type?: VideoBufferType; @@ -403,8 +474,8 @@ class ExternalVideoFrame { Texture 帧额外的转换。该参数仅适用于 Texture 格式的视频数据。 - eglContext11 - EGLContext11。该参数仅适用于 Texture 格式的视频数据。 + eglContext10 + EGLContext10。该参数仅适用于 Texture 格式的视频数据。 eglContext14 @@ -414,8 +485,8 @@ class ExternalVideoFrame { eglContext 该参数仅适用于 Texture 格式的视频数据。

    -
  • 当使用 Khronos 定义的 OpenGL 接口 (javax.microedition.khronos.egl.*)时，需要将 eglContext 设置给这个字段。
    • -
  • 当使用 Android 定义的 OpenGL 接口 (android.opengl.*)时，需要将 eglContext 设置给这个字段。
    • +
  • 当使用 Khronos 定义的 OpenGL 接口 (javax.microedition.khronos.egl.*) 时，需要将 eglContext 设置给这个字段。
    • +
  • 当使用 Android 定义的 OpenGL 接口 (android.opengl.*) 时，需要将 eglContext 设置给这个字段。
@@ -506,6 +577,10 @@ class ExternalVideoFrame { time 传入的视频帧的时间戳，以毫秒为单位。不正确的时间戳会导致丢帧或者音视频不同步。 + + + +
diff --git a/dita/RTC-NG/API/class_videoframe.dita b/dita/RTC-NG/API/class_videoframe.dita index 0e2de8f9661..6583607bdf2 100644 --- a/dita/RTC-NG/API/class_videoframe.dita +++ b/dita/RTC-NG/API/class_videoframe.dita @@ -7,42 +7,61 @@

public class VideoFrame implements RefCounted { + public interface Buffer extends RefCounted { - @CalledByNative("Buffer") int getWidth(); - @CalledByNative("Buffer") int getHeight(); - @CalledByNative("Buffer") I420Buffer toI420(); - @Override @CalledByNative("Buffer") void release(); - @Override @CalledByNative("Buffer") void retain(); - @CalledByNative("Buffer") + + @CalledByNative("Buffer") int getWidth(); + + @CalledByNative("Buffer") int getHeight(); + + @CalledByNative("Buffer") I420Buffer toI420(); + + @Override @CalledByNative("Buffer") void release(); + + @Override @CalledByNative("Buffer") void retain(); + + @CalledByNative("Buffer") Buffer cropAndScale( int cropX, int cropY, int cropWidth, int cropHeight, int scaleWidth, int scaleHeight); - @CalledByNative("Buffer") @Nullable Buffer mirror(int frameRotation); - @CalledByNative("Buffer") @Nullable Buffer rotate(int frameRotation); - @CalledByNative("Buffer") + + @CalledByNative("Buffer") @Nullable Buffer mirror(int frameRotation); + + @CalledByNative("Buffer") @Nullable Buffer rotate(int frameRotation); + + @CalledByNative("Buffer") @Nullable Buffer transform(int cropX, int cropY, int cropWidth, int cropHeight, int scaleWidth, int scaleHeight, int frameRotation); } + public interface I420Buffer extends Buffer { - @CalledByNative("I420Buffer") ByteBuffer getDataY(); - @CalledByNative("I420Buffer") ByteBuffer getDataU(); - @CalledByNative("I420Buffer") ByteBuffer getDataV(); - @CalledByNative("I420Buffer") int getStrideY(); - @CalledByNative("I420Buffer") int getStrideU(); - @CalledByNative("I420Buffer") int getStrideV(); + + @CalledByNative("I420Buffer") ByteBuffer getDataY(); + + @CalledByNative("I420Buffer") ByteBuffer getDataU(); + + @CalledByNative("I420Buffer") ByteBuffer getDataV(); + @CalledByNative("I420Buffer") int getStrideY(); + @CalledByNative("I420Buffer") int getStrideU(); + @CalledByNative("I420Buffer") int getStrideV(); } + public interface I422Buffer extends Buffer { - @CalledByNative("I422Buffer") ByteBuffer getDataY(); - @CalledByNative("I422Buffer") ByteBuffer getDataU(); - @CalledByNative("I422Buffer") ByteBuffer getDataV(); - @CalledByNative("I422Buffer") int getStrideY(); - @CalledByNative("I422Buffer") int getStrideU(); - @CalledByNative("I422Buffer") int getStrideV(); - } - public interface RgbaBuffer extends Buffer { @CalledByNative("RgbaBuffer") ByteBuffer getData(); } + @CalledByNative("I422Buffer") ByteBuffer getDataY(); + @CalledByNative("I422Buffer") ByteBuffer getDataU(); + @CalledByNative("I422Buffer") ByteBuffer getDataV(); + @CalledByNative("I422Buffer") int getStrideY(); + @CalledByNative("I422Buffer") int getStrideU(); + @CalledByNative("I422Buffer") int getStrideV(); + } + public interface RgbaBuffer extends Buffer { @CalledByNative("RgbaBuffer") ByteBuffer getData(); } + public interface TextureBuffer extends Buffer { + enum Type { + OES(GLES11Ext.GL_TEXTURE_EXTERNAL_OES), + RGB(GLES20.GL_TEXTURE_2D); private final int glTarget; private Type(final int glTarget) { @@ -57,22 +76,26 @@ EGL_CONTEXT_14; } Type getType(); - @CalledByNative("TextureBuffer") int getTextureId(); + + @CalledByNative("TextureBuffer") int getTextureId(); + Matrix getTransformMatrix(); - @CalledByNative("TextureBuffer") EglBase.Context getEglBaseContext(); - @CalledByNative("TextureBuffer") Object getSourceTexturePool(); - @CalledByNative("TextureBuffer") long getNativeEglContext(); - @CalledByNative("TextureBuffer") int getEglContextType(); - @CalledByNative("TextureBuffer") float[] getTransformMatrixArray(); - @CalledByNative("TextureBuffer") int getSequence(); - @CalledByNative("TextureBuffer") boolean is10BitTexture(); + + @CalledByNative("TextureBuffer") EglBase.Context getEglBaseContext(); + @CalledByNative("TextureBuffer") Object getSourceTexturePool(); + @CalledByNative("TextureBuffer") long getNativeEglContext(); + @CalledByNative("TextureBuffer") int getEglContextType(); + @CalledByNative("TextureBuffer") float[] getTransformMatrixArray(); + + @CalledByNative("TextureBuffer") int getSequence(); + @CalledByNative("TextureBuffer") long getFenceObject(); + @CalledByNative("TextureBuffer") boolean is10BitTexture(); } public interface ColorSpace { enum Range { Invalid(0), Limited(1), - Full(2), - Derived(3); + Full(2); private final int range; private Range(int range) { this.range = range; @@ -161,32 +184,55 @@ kBackCamera, kUnspecified, } + public enum AlphaStitchMode { + ALPHA_NO_STITCH(0), + ALPHA_STITCH_UP(1), + ALPHA_STITCH_BELOW(2), + ALPHA_STITCH_LEFT(3), + ALPHA_STITCH_RIGHT(4); + private final int stitchMode; + private AlphaStitchMode(int stitchMode) { + this.stitchMode = stitchMode; + } + public int value() { + return stitchMode; + } + } + private Buffer buffer; + private int rotation; + private long timestampNs; private ColorSpace colorSpace; private SourceType sourceType; private float sampleAspectRatio; + + private AlphaStitchMode alphaStitchMode = AlphaStitchMode.ALPHA_NO_STITCH; private VideoFrameMetaInfo metaInfo = new VideoFrameMetaInfo(); + private @Nullable ByteBuffer alphaBuffer; - private int alphaStitchMode; + private long nativeAlphaBuffer; + public VideoFrame(Buffer buffer, int rotation, long timestampNs) { - this(buffer, rotation, timestampNs, new WrappedNativeColorSpace(), null, 0L, 1.0f, SourceType.kUnspecified.ordinal()); + this(buffer, rotation, timestampNs, new WrappedNativeColorSpace(), null, 0L, 1.0f, + SourceType.kUnspecified.ordinal()); } @CalledByNative public VideoFrame(Buffer buffer, int rotation, long timestampNs, ColorSpace colorSpace, - ByteBuffer alphaBuffer, float sampleAspectRatio, int sourceType) { + ByteBuffer alphaBuffer, long nativeAlphaBuffer, float sampleAspectRatio, int sourceType) { if (buffer == null) { - throw new IllegalArgumentException("buffer not allowed to be null"); + throw new IllegalArgumentException("buffer not allowed to be null"); } if (rotation % 90 != 0) { - throw new IllegalArgumentException("rotation must be a multiple of 90"); + throw new IllegalArgumentException("rotation must be a multiple of 90"); } this.buffer = buffer; this.rotation = rotation; this.timestampNs = timestampNs; this.colorSpace = colorSpace; this.alphaBuffer = alphaBuffer; + this.nativeAlphaBuffer = nativeAlphaBuffer; this.sampleAspectRatio = sampleAspectRatio; this.sourceType = SourceType.values()[sourceType]; } @@ -197,18 +243,25 @@ public float getSampleAspectRatio() { return sampleAspectRatio; } + @CalledByNative public Buffer getBuffer() { return buffer; } + @CalledByNative public int getRotation() { return rotation; } @CalledByNative public int getAlphaStitchMode() { - return alphaStitchMode; + return alphaStitchMode.value(); } + @CalledByNative + public void setAlphaStitchMode(int stitchMode) { + alphaStitchMode = AlphaStitchMode.values()[stitchMode]; + } + @CalledByNative public long getTimestampNs() { return timestampNs; @@ -217,41 +270,93 @@ public VideoFrameMetaInfo getMetaInfo() { return metaInfo; } + public int getRotatedWidth() { if (rotation % 180 == 0) { - return buffer.getWidth(); + return (alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_LEFT + || alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_RIGHT) + ? buffer.getWidth() / 2 + : buffer.getWidth(); } - return buffer.getHeight(); + return (alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_UP + || alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_BELOW) + ? buffer.getHeight() / 2 + : buffer.getHeight(); } + public int getRotatedHeight() { if (rotation % 180 == 0) { - return buffer.getHeight(); + return (alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_UP + || alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_BELOW) + ? buffer.getHeight() / 2 + : buffer.getHeight(); } - return buffer.getWidth(); + return (alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_LEFT + || alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_RIGHT) + ? buffer.getWidth() / 2 + : buffer.getWidth(); } + public void replaceBuffer(Buffer buffer, int rotation, long timestampNs) { release(); this.buffer = buffer; this.rotation = rotation; this.timestampNs = timestampNs; } + @CalledByNative public ColorSpace getColorSpace() { return colorSpace; } + public void setColorSpace(ColorSpace colorSpace) { + this.colorSpace = colorSpace; + } + @CalledByNative + private int getColorSpaceRange() { + if (colorSpace == null) { + return ColorSpace.Range.Invalid.getRange(); + } + return colorSpace.getRange().getRange(); + } + @CalledByNative + private int getColorSpaceMatrix() { + if (colorSpace == null) { + return ColorSpace.Matrix.Unspecified.getMatrix(); + } + return colorSpace.getMatrix().getMatrix(); + } + @CalledByNative + private int getColorSpaceTransfer() { + if (colorSpace == null) { + return ColorSpace.Transfer.Unspecified.getTransfer(); + } + return colorSpace.getTransfer().getTransfer(); + } + @CalledByNative + private int getColorSpacePrimary() { + if (colorSpace == null) { + return ColorSpace.Primary.Unspecified.getPrimary(); + } + return colorSpace.getPrimary().getPrimary(); + } @CalledByNative public ByteBuffer getAlphaBuffer() { return alphaBuffer; } + public void retainAlphaBuffer() { + JniCommon.nativeAddRef(nativeAlphaBuffer); + } + public void releaseAlphaBuffer() { + JniCommon.nativeReleaseRef(nativeAlphaBuffer); + } public void fillAlphaData(ByteBuffer buffer) { alphaBuffer = buffer; } - public void setAlphaStitchMode(int mode) { - this.alphaStitchMode = mode; - } + @Override public void retain() { buffer.retain(); } + @Override @CalledByNative public void release() { @@ -278,7 +383,8 @@ this.format = format; } } - __attribute__((visibility("default"))) @interface AgoraOutputVideoFrame : NSObject + __attribute__((visibility("default"))) @interface AgoraOutputVideoFrame : NSObject + @property (nonatomic, assign) NSInteger type; @property (nonatomic, assign) int width; @property (nonatomic, assign) int height; @@ -291,12 +397,16 @@ @property (nonatomic, assign) int rotation; @property (nonatomic, assign) int64_t renderTimeMs; @property (nonatomic, assign) int avSyncType; + @property(assign, nonatomic) CVPixelBufferRef _Nullable pixelBuffer; @property (nonatomic, assign) uint8_t* _Nullable alphaBuffer; @property (nonatomic, assign) AgoraAlphaStitchMode alphaStitchMode; + @property(nonatomic, strong) NSDictionary *_Nonnull metaInfo; +@property(nonatomic, strong) AgoraColorSpace* _Nullable colorSpace; @end - struct VideoFrame { + +struct VideoFrame { VideoFrame(): type(VIDEO_PIXEL_DEFAULT), width(0), @@ -316,33 +426,57 @@ textureId(0), d3d11Texture2d(NULL), alphaBuffer(NULL), - alphaStitchMode(0), + alphaStitchMode(NO_ALPHA_STITCH), pixelBuffer(NULL), metaInfo(NULL){ memset(matrix, 0, sizeof(matrix)); } + VIDEO_PIXEL_FORMAT type; + int width; + int height; + int yStride; + int uStride; + int vStride; + uint8_t* yBuffer; + uint8_t* uBuffer; + uint8_t* vBuffer; + int rotation; + int64_t renderTimeMs; + int avsync_type; + uint8_t* metadata_buffer; + int metadata_size; + void* sharedContext; + int textureId; + void* d3d11Texture2d; + float matrix[16]; + uint8_t* alphaBuffer; - int alphaStitchMode; + + ALPHA_STITCH_MODE alphaStitchMode; + void* pixelBuffer; + IVideoFrameMetaInfo* metaInfo; + + ColorSpace colorSpace; }; USTRUCT(BlueprintType) struct FVideoFrame { @@ -608,6 +742,19 @@ class VideoFrame {

缓冲区给出的是指向指针的指针，该接口不能修改缓冲区的指针，只能修改缓冲区的内容。

+
+ 方法 + + + getColorSpace + 获取视频帧的色彩空间属性。 + + + setColorSpace + 设置视频帧的色彩空间属性。 + + +
<text conref="../conref/conref_api_metadata.dita#conref_api_metadata/property"/> @@ -727,9 +874,9 @@ class VideoFrame { pixelBuffer 将数据填充到 CVPixelBuffer。 - - colorSpace - 表示视频帧的色彩空间。详见 。 + + colorSpace + 视频帧的色彩空间属性，默认情况下会应用 Full Range 和 BT.709 标准配置。你可以根据自采集、自渲染的业务需求进行自定义设置，详见 。 sourceType From 0452b39abfb33367dc10aa3c4368f3a61d6f8a4b Mon Sep 17 00:00:00 2001 From: jinyu Date: Fri, 18 Oct 2024 17:03:17 +0800 Subject: [PATCH 06/19] encoding -> capturing --- dita/RTC-NG/API/api_irtcengine_iscameraexposuresupported.dita | 2 +- dita/RTC-NG/API/api_irtcengine_iscamerazoomsupported.dita | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dita/RTC-NG/API/api_irtcengine_iscameraexposuresupported.dita b/dita/RTC-NG/API/api_irtcengine_iscameraexposuresupported.dita index c49d159269e..f9fdf8f143e 100644 --- a/dita/RTC-NG/API/api_irtcengine_iscameraexposuresupported.dita +++ b/dita/RTC-NG/API/api_irtcengine_iscameraexposuresupported.dita @@ -36,7 +36,7 @@
  • 该方法仅适用于 Android 和 iOS。
    • -
  • 该方法必须在 SDK 触发 回调，返回本地视频状态为 (2) 之后调用。
    • +
  • 该方法必须在 SDK 触发 回调，返回本地视频状态为 (1) 之后调用。
  • 建议你在调用 调节曝光系数前，先调用该方法查询当前摄像头是否支持曝光调节。
  • 当你调用该方法时，查询的是当前正在使用的摄像头是否支持曝光调节，即调用 时指定的摄像头。
 diff --git a/dita/RTC-NG/API/api_irtcengine_iscamerazoomsupported.dita b/dita/RTC-NG/API/api_irtcengine_iscamerazoomsupported.dita index 68e714401b7..e4625c76f02 100644 --- a/dita/RTC-NG/API/api_irtcengine_iscamerazoomsupported.dita +++ b/dita/RTC-NG/API/api_irtcengine_iscamerazoomsupported.dita @@ -30,7 +30,7 @@
调用时机 -

该方法必须在 SDK 触发 回调，返回本地视频状态为 (2) 之后调用。

+

该方法必须在 SDK 触发 回调，返回本地视频状态为 (1) 之后调用。

调用限制 From ad7bbc2e06ded6ed0f52df5c0be5a23e7b0b25f1 Mon Sep 17 00:00:00 2001 From: jinyu Date: Mon, 21 Oct 2024 11:54:09 +0800 Subject: [PATCH 07/19] resolve comments --- dita/RTC-NG/config/keys-rtc-ng-links-android.ditamap | 7 ------- dita/RTC-NG/config/keys-rtc-ng-links-harmony.ditamap | 7 ------- dita/RTC-NG/config/keys-rtc-ng-links.ditamap | 7 +++++++ 3 files changed, 7 insertions(+), 14 deletions(-) diff --git a/dita/RTC-NG/config/keys-rtc-ng-links-android.ditamap b/dita/RTC-NG/config/keys-rtc-ng-links-android.ditamap index 38c5a447f70..6cb00aa1ebc 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-links-android.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-links-android.ditamap @@ -125,13 +125,6 @@ - - - - VideoColorSpace - - - diff --git a/dita/RTC-NG/config/keys-rtc-ng-links-harmony.ditamap b/dita/RTC-NG/config/keys-rtc-ng-links-harmony.ditamap index 7306c00e8b4..c9366a30a9f 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-links-harmony.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-links-harmony.ditamap @@ -104,13 +104,6 @@ - - - - VideoColorSpace - - - diff --git a/dita/RTC-NG/config/keys-rtc-ng-links.ditamap b/dita/RTC-NG/config/keys-rtc-ng-links.ditamap index 4ef860372ad..dd5a74c036e 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-links.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-links.ditamap @@ -98,6 +98,13 @@ + + + + VideoColorSpace + + + From 8cf53af0a0b1a0b3f4d507eabe83cdfe3e5f3073 Mon Sep 17 00:00:00 2001 From: jinyu Date: Mon, 21 Oct 2024 15:13:48 +0800 Subject: [PATCH 08/19] audiotrack --- dita/RTC-NG/API/class_audiotrackconfig.dita | 36 +++++++++++++++------ 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/dita/RTC-NG/API/class_audiotrackconfig.dita b/dita/RTC-NG/API/class_audiotrackconfig.dita index 95e46f8b937..abf31b9133c 100644 --- a/dita/RTC-NG/API/class_audiotrackconfig.dita +++ b/dita/RTC-NG/API/class_audiotrackconfig.dita @@ -7,27 +7,35 @@

public class AudioTrackConfig { + public boolean enableLocalPlayback; - - + + public boolean enableAudioProcessing; public AudioTrackConfig() { this.enableLocalPlayback = true; + this.enableAudioProcessing = false; + } + @Override + public String toString() { + return "AudioTrackConfig{" + + "enableLocalPlayback=" + enableLocalPlayback + "enableAudioProcessing" + + enableAudioProcessing + '}'; } - } +} export class AudioTrackConfig { public enableLocalPlayback: boolean = true; } - NS_SWIFT_NAME(AgoraAudioTrackConfig) __attribute__((visibility("default"))) @interface AgoraAudioTrackConfig : NSObject + NS_SWIFT_NAME(AgoraAudioTrackConfig) __attribute__((visibility("default"))) @interface AgoraAudioTrackConfig : NSObject @property (assign, nonatomic) BOOL enableLocalPlayback NS_SWIFT_NAME(enableLocalPlayback); - +@property (assign, nonatomic) BOOL enableAudioProcessing NS_SWIFT_NAME(enableAudioProcessing); @end struct AudioTrackConfig { + bool enableLocalPlayback; - - - AudioTrackConfig() - : enableLocalPlayback(true) {} + + bool enableAudioProcessing; + AudioTrackConfig() : enableLocalPlayback(true),enableAudioProcessing(false) {} }; USTRUCT(BlueprintType) struct FAudioTrackConfig @@ -84,6 +92,16 @@ class AudioTrackConfig { + + enableAudioProcessing + 是否启用音频处理模块： +

    +
  • ：启用音频处理模块，应用回声消除 (AEC)、降噪 (ANS) 和自动增益控制 (AGC)。
    • +
  • ：（默认）不启用音频处理模块。
    • +
+ 该参数设置仅对 类型的自定义音频采集轨道生效。 + +
\ No newline at end of file From 5450d788b408b57c9166d8dbc6667f6176e14b5c Mon Sep 17 00:00:00 2001 From: jinyu Date: Tue, 22 Oct 2024 11:15:27 +0800 Subject: [PATCH 09/19] minor fix --- dita/RTC-NG/API/rtc_api_overview.dita | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dita/RTC-NG/API/rtc_api_overview.dita b/dita/RTC-NG/API/rtc_api_overview.dita index 2432cba24ad..439aa1d3aee 100644 --- a/dita/RTC-NG/API/rtc_api_overview.dita +++ b/dita/RTC-NG/API/rtc_api_overview.dita @@ -1315,7 +1315,7 @@ - （仅适用于 Android） + （仅适用于 Android） From 92e8efbb3cfd473f56431685e2dcb43d89b29519 Mon Sep 17 00:00:00 2001 From: jinyu Date: Mon, 28 Oct 2024 11:43:17 +0800 Subject: [PATCH 10/19] resolve comments --- dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita b/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita index ec09c09163f..5a01a581576 100644 --- a/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita +++ b/dita/RTC-NG/API/api_irtcengine_setexternalremoteeglcontext.dita @@ -36,7 +36,7 @@
适用场景 -

该方法适用于下行视频后处理，且视频数据为纹理格式的场景，例如：硬件解码、自定义远端视频渲染。

+

该方法适用于使用 Texture 格式的视频数据进行远端视频自渲染的场景。

调用时机 From e1a1cfdae368554eaf9ecf5fbad80c595972ffd5 Mon Sep 17 00:00:00 2001 From: jinyu Date: Mon, 28 Oct 2024 15:31:19 +0800 Subject: [PATCH 11/19] =?UTF-8?q?=E5=90=88=E5=9B=BE=20&=20=E5=8F=8C?= =?UTF-8?q?=E6=B5=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- dita/RTC-NG/RTC_NG_API_Android.ditamap | 42 +++++++++++------------- dita/RTC-NG/RTC_NG_API_Blueprint.ditamap | 36 +++++++++----------- dita/RTC-NG/RTC_NG_API_Flutter.ditamap | 36 +++++++++----------- dita/RTC-NG/RTC_NG_API_Unreal.ditamap | 36 +++++++++----------- 4 files changed, 67 insertions(+), 83 deletions(-) diff --git a/dita/RTC-NG/RTC_NG_API_Android.ditamap b/dita/RTC-NG/RTC_NG_API_Android.ditamap index 66a5d3e3884..5d2ad3088ac 100644 --- a/dita/RTC-NG/RTC_NG_API_Android.ditamap +++ b/dita/RTC-NG/RTC_NG_API_Android.ditamap @@ -91,6 +91,10 @@ + + + + @@ -103,6 +107,15 @@ + + + + + + + + + @@ -297,6 +310,12 @@ + + + + + + @@ -326,29 +345,6 @@ - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dita/RTC-NG/RTC_NG_API_Blueprint.ditamap b/dita/RTC-NG/RTC_NG_API_Blueprint.ditamap index ee09bb5749b..b17e8eae2d6 100644 --- a/dita/RTC-NG/RTC_NG_API_Blueprint.ditamap +++ b/dita/RTC-NG/RTC_NG_API_Blueprint.ditamap @@ -97,6 +97,10 @@ + + + + @@ -109,6 +113,12 @@ + + + + + + @@ -317,6 +327,12 @@ + + + + + + @@ -346,26 +362,6 @@ - - - - - - - - - - - - - - - - - - - - diff --git a/dita/RTC-NG/RTC_NG_API_Flutter.ditamap b/dita/RTC-NG/RTC_NG_API_Flutter.ditamap index 45aa0e30ca7..78a07c2fbcc 100644 --- a/dita/RTC-NG/RTC_NG_API_Flutter.ditamap +++ b/dita/RTC-NG/RTC_NG_API_Flutter.ditamap @@ -97,6 +97,10 @@ + + + + @@ -109,6 +113,12 @@ + + + + + + @@ -305,6 +315,12 @@ + + + + + + @@ -334,26 +350,6 @@ - - - - - - - - - - - - - - - - - - - - diff --git a/dita/RTC-NG/RTC_NG_API_Unreal.ditamap b/dita/RTC-NG/RTC_NG_API_Unreal.ditamap index af24969b303..4c2072f2e07 100644 --- a/dita/RTC-NG/RTC_NG_API_Unreal.ditamap +++ b/dita/RTC-NG/RTC_NG_API_Unreal.ditamap @@ -93,6 +93,10 @@ + + + + @@ -105,6 +109,12 @@ + + + + + + @@ -313,6 +323,12 @@ + + + + + + @@ -342,26 +358,6 @@ - - - - - - - - - - - - - - - - - - - - From 4a64c30b3dc21bfe1eb426ae40538b02a6b427f3 Mon Sep 17 00:00:00 2001 From: jinyu Date: Mon, 28 Oct 2024 15:44:19 +0800 Subject: [PATCH 12/19] =?UTF-8?q?=E5=AA=92=E4=BD=93=E6=92=AD=E6=94=BE?= =?UTF-8?q?=E5=99=A8=E7=BC=93=E5=AD=98?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- dita/RTC-NG/RTC_NG_API_Android.ditamap | 28 ++++++++++++------------ dita/RTC-NG/RTC_NG_API_Blueprint.ditamap | 28 ++++++++++++------------ dita/RTC-NG/RTC_NG_API_Flutter.ditamap | 28 ++++++++++++------------ dita/RTC-NG/RTC_NG_API_Unreal.ditamap | 28 ++++++++++++------------ 4 files changed, 56 insertions(+), 56 deletions(-) diff --git a/dita/RTC-NG/RTC_NG_API_Android.ditamap b/dita/RTC-NG/RTC_NG_API_Android.ditamap index 5d2ad3088ac..90062c23bb9 100644 --- a/dita/RTC-NG/RTC_NG_API_Android.ditamap +++ b/dita/RTC-NG/RTC_NG_API_Android.ditamap @@ -510,20 +510,20 @@ - - - - - - - - - - - - - - + + + + + + + + + + + + + + diff --git a/dita/RTC-NG/RTC_NG_API_Blueprint.ditamap b/dita/RTC-NG/RTC_NG_API_Blueprint.ditamap index b17e8eae2d6..9885a2e2e6d 100644 --- a/dita/RTC-NG/RTC_NG_API_Blueprint.ditamap +++ b/dita/RTC-NG/RTC_NG_API_Blueprint.ditamap @@ -527,20 +527,20 @@ - - - - - - - - - - - - - - + + + + + + + + + + + + + + diff --git a/dita/RTC-NG/RTC_NG_API_Flutter.ditamap b/dita/RTC-NG/RTC_NG_API_Flutter.ditamap index 78a07c2fbcc..6a7db2377bf 100644 --- a/dita/RTC-NG/RTC_NG_API_Flutter.ditamap +++ b/dita/RTC-NG/RTC_NG_API_Flutter.ditamap @@ -502,20 +502,20 @@ - - - - - - - - - - - - - - + + + + + + + + + + + + + + diff --git a/dita/RTC-NG/RTC_NG_API_Unreal.ditamap b/dita/RTC-NG/RTC_NG_API_Unreal.ditamap index 4c2072f2e07..2f4cb93f10e 100644 --- a/dita/RTC-NG/RTC_NG_API_Unreal.ditamap +++ b/dita/RTC-NG/RTC_NG_API_Unreal.ditamap @@ -521,20 +521,20 @@ - - - - - - - - - - - - - - + + + + + + + + + + + + + + From 11f4fb75f30e681c0f62554612c133d63e262ce3 Mon Sep 17 00:00:00 2001 From: jinyu Date: Mon, 28 Oct 2024 16:38:08 +0800 Subject: [PATCH 13/19] check --- dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita | 2 +- dita/RTC-NG/API/class_audiotrackconfig.dita | 2 +- dita/RTC-NG/config/keys-rtc-ng-links.ditamap | 7 +++++++ 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita b/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita index 8928324f7c3..d929855a308 100644 --- a/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita +++ b/dita/RTC-NG/API/api_irtcengine_setvideoscenario.dita @@ -77,7 +77,7 @@

-

(2) 适用于视频 1v1 通话场景。针对该场景低延迟、高画质的体验要求，SDK 进行了策略调优，提升了画质、首帧出图、中低端机延迟及弱网流畅度等性能表现。

+

(2) 适用于场景。针对该场景低延迟、高画质的体验要求，SDK 进行了策略调优，提升了画质、首帧出图、中低端机延迟及弱网流畅度等性能表现。

(3) 适用于场景。针对该场景对首帧出图时间和画质清晰度的高要求，SDK 进行了策略调优，重点提升了首帧出图体验和画质表现，同时增强了在弱网环境和低端设备上的画质和流畅度表现。

diff --git a/dita/RTC-NG/API/class_audiotrackconfig.dita b/dita/RTC-NG/API/class_audiotrackconfig.dita index abf31b9133c..7c425f5025d 100644 --- a/dita/RTC-NG/API/class_audiotrackconfig.dita +++ b/dita/RTC-NG/API/class_audiotrackconfig.dita @@ -96,7 +96,7 @@ class AudioTrackConfig { enableAudioProcessing 是否启用音频处理模块：
    -
  • ：启用音频处理模块，应用回声消除 (AEC)、降噪 (ANS) 和自动增益控制 (AGC)。
    • +
  • ：启用音频处理模块，应用回声消除 (AEC)、降噪 (ANS) 和自动增益控制 (AGC) 效果。
  • ：（默认）不启用音频处理模块。
该参数设置仅对 类型的自定义音频采集轨道生效。 diff --git a/dita/RTC-NG/config/keys-rtc-ng-links.ditamap b/dita/RTC-NG/config/keys-rtc-ng-links.ditamap index dd5a74c036e..5af8acfb662 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-links.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-links.ditamap @@ -90,6 +90,13 @@ + + + + 视频 1v1 通话 + + + From 9ae1a381408df337628227795e0bbf3790bcafa8 Mon Sep 17 00:00:00 2001 From: jinyu Date: Mon, 28 Oct 2024 17:36:33 +0800 Subject: [PATCH 14/19] 1 --- dita/RTC-NG/API/rtc_api_data_type.dita | 2 + dita/RTC-NG/RTC_NG_API_Android.ditamap | 8 ++ .../config/keys-rtc-ng-api-java.ditamap | 128 ++++++++++++++++++ 3 files changed, 138 insertions(+) diff --git a/dita/RTC-NG/API/rtc_api_data_type.dita b/dita/RTC-NG/API/rtc_api_data_type.dita index d52c798d28e..a9e804ce1bb 100644 --- a/dita/RTC-NG/API/rtc_api_data_type.dita +++ b/dita/RTC-NG/API/rtc_api_data_type.dita @@ -206,6 +206,7 @@
    • +
    • @@ -216,6 +217,7 @@
    • +
    • diff --git a/dita/RTC-NG/RTC_NG_API_Android.ditamap b/dita/RTC-NG/RTC_NG_API_Android.ditamap index 90062c23bb9..3bf9d6636ff 100644 --- a/dita/RTC-NG/RTC_NG_API_Android.ditamap +++ b/dita/RTC-NG/RTC_NG_API_Android.ditamap @@ -208,6 +208,11 @@ + + + + + @@ -291,6 +296,7 @@ + @@ -349,9 +355,11 @@ + + diff --git a/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap b/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap index c8ba05b81db..ebc1f750a67 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap @@ -1057,6 +1057,20 @@ setRemoteRenderMode     +     + + + setLocalRenderTargetFps + + + + + + + setRemoteRenderTargetFps + + @@ -1556,6 +1570,13 @@ + + + + setExternalMediaProjection + + + @@ -2507,6 +2528,29 @@ + + + + + startLocalAudioMixer + + + + + + + stopLocalAudioMixer + + + + + + + updateLocalAudioMixerConfiguration + + + + @@ -4831,6 +4875,69 @@ + + + + AudioSourceType + + + + + + + AUDIO_SOURCE_MICROPHONE + + + + + + + AUDIO_SOURCE_CUSTOM + + + + + + + AUDIO_SOURCE_MEDIA_PLAYER + + + + + + + AUDIO_SOURCE_LOOPBACK_RECORDING + + + + + + + AUDIO_SOURCE_MIXED_STREAM + + + + + + + AUDIO_SOURCE_REMOTE_USER + + + + + + + AUDIO_SOURCE_REMOTE_CHANNEL + + + + + + + AUDIO_SOURCE_UNKNOWN + + + @@ -5097,6 +5204,13 @@ + + + + LocalAudioMixerConfiguration + + + @@ -5160,6 +5274,13 @@ + + + + MixedAudioStream + + + @@ -6023,6 +6144,13 @@ + + + + PREFER_COMPRESSION_AUTO + + + From 47c6f767479eec94760ac9488ec96468d3520686 Mon Sep 17 00:00:00 2001 From: jinyu Date: Tue, 29 Oct 2024 10:38:52 +0800 Subject: [PATCH 15/19] fix --- dita/RTC-NG/API/rtc_api_overview.dita | 2 +- dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap | 2 +- dita/RTC-NG/config/keys-rtc-ng-links.ditamap | 14 +++++++------- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/dita/RTC-NG/API/rtc_api_overview.dita b/dita/RTC-NG/API/rtc_api_overview.dita index 439aa1d3aee..eed098a3334 100644 --- a/dita/RTC-NG/API/rtc_api_overview.dita +++ b/dita/RTC-NG/API/rtc_api_overview.dita @@ -1315,7 +1315,7 @@ - （仅适用于 Android） + （仅适用于 Android） diff --git a/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap b/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap index ebc1f750a67..7354e4a2325 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap @@ -1057,7 +1057,7 @@ setRemoteRenderMode - diff --git a/dita/RTC-NG/config/keys-rtc-ng-links.ditamap b/dita/RTC-NG/config/keys-rtc-ng-links.ditamap index 5af8acfb662..71fa82634d0 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-links.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-links.ditamap @@ -104,14 +104,14 @@ + + + + VideoColorSpace + + + - - - - VideoColorSpace - - - From d69bfa18ebc202aa2c4c2ab608831ba67c23672a Mon Sep 17 00:00:00 2001 From: Cilla-luodan <85477033+Cilla-luodan@users.noreply.github.com> Date: Wed, 30 Oct 2024 11:35:50 +0800 Subject: [PATCH 16/19] =?UTF-8?q?=E6=90=AD=E8=BD=A6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- dita/RTC-NG/API/class_externalvideoframe.dita | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dita/RTC-NG/API/class_externalvideoframe.dita b/dita/RTC-NG/API/class_externalvideoframe.dita index 53d5a2e5209..4d84556d96d 100644 --- a/dita/RTC-NG/API/class_externalvideoframe.dita +++ b/dita/RTC-NG/API/class_externalvideoframe.dita @@ -473,7 +473,7 @@ class ExternalVideoFrame { transform Texture 帧额外的转换。该参数仅适用于 Texture 格式的视频数据。 - + eglContext10 EGLContext10。该参数仅适用于 Texture 格式的视频数据。 From 01e8bf5aa4f844a03a4a6ee65688764a6516c27b Mon Sep 17 00:00:00 2001 From: jinyu Date: Wed, 30 Oct 2024 16:05:35 +0800 Subject: [PATCH 17/19] resolve comments --- dita/RTC-NG/API/class_externalvideoframe.dita | 56 ------------------- dita/RTC-NG/API/class_videoframe.dita | 2 +- 2 files changed, 1 insertion(+), 57 deletions(-) diff --git a/dita/RTC-NG/API/class_externalvideoframe.dita b/dita/RTC-NG/API/class_externalvideoframe.dita index 4d84556d96d..b2de1d3b9ae 100644 --- a/dita/RTC-NG/API/class_externalvideoframe.dita +++ b/dita/RTC-NG/API/class_externalvideoframe.dita @@ -7,29 +7,17 @@

    public class AgoraVideoFrame { - public static final int FORMAT_NONE = -1; - public static final int FORMAT_TEXTURE_2D = 10; - public static final int FORMAT_TEXTURE_OES = 11; - public static final int FORMAT_I420 = 1; - public static final int FORMAT_BGRA = 2; - public static final int FORMAT_NV21 = 3; - public static final int FORMAT_RGBA = 4; - public static final int FORMAT_I422 = 16; - public static final int BUFFER_TYPE_NONE = -1; - public static final int BUFFER_TYPE_BUFFER = 1; - public static final int BUFFER_TYPE_ARRAY = 2; - public static final int BUFFER_TYPE_TEXTURE = 3; public AgoraVideoFrame() { format = 10; @@ -51,35 +39,20 @@ } public int format; - public long timeStamp; - public int stride; - public int height; - public int textureID; - public boolean syncMode; - public float[] transform; - public javax.microedition.khronos.egl.EGLContext eglContext10; - public android.opengl.EGLContext eglContext14; - public byte[] buf; - public int cropLeft; - public int cropTop; - public int cropRight; - public int cropBottom; - public int rotation; - public int alphaStitchMode; @Override public String toString() { @@ -90,7 +63,6 @@ + ", cropTop=" + cropTop + ", cropRight=" + cropRight + ", cropBottom=" + cropBottom + ", rotation=" + rotation + ", alphaStitchMode=" + alphaStitchMode + '}'; } - } @@ -146,67 +118,39 @@ struct ExternalVideoFrame { textureSliceIndex(0){} enum EGL_CONTEXT_TYPE { - EGL_CONTEXT10 = 0, - EGL_CONTEXT14 = 1, }; enum VIDEO_BUFFER_TYPE { - VIDEO_BUFFER_RAW_DATA = 1, - VIDEO_BUFFER_ARRAY = 2, - VIDEO_BUFFER_TEXTURE = 3, }; VIDEO_BUFFER_TYPE type; - VIDEO_PIXEL_FORMAT format; - void* buffer; - int stride; - int height; - int cropLeft; - int cropTop; - int cropRight; - int cropBottom; - int rotation; - long long timestamp; - void* eglContext; - EGL_CONTEXT_TYPE eglType; - int textureId; - long long fenceObject; - float matrix[16]; - uint8_t* metadataBuffer; - int metadataSize; - uint8_t* alphaBuffer; - bool fillAlphaBuffer; - ALPHA_STITCH_MODE alphaStitchMode; - void *d3d11Texture2d; - int textureSliceIndex; - ColorSpace colorSpace; }; export class ExternalVideoFrame { diff --git a/dita/RTC-NG/API/class_videoframe.dita b/dita/RTC-NG/API/class_videoframe.dita index 6583607bdf2..d2aca9e6441 100644 --- a/dita/RTC-NG/API/class_videoframe.dita +++ b/dita/RTC-NG/API/class_videoframe.dita @@ -876,7 +876,7 @@ class VideoFrame { colorSpace - 视频帧的色彩空间属性，默认情况下会应用 Full Range 和 BT.709 标准配置。你可以根据自采集、自渲染的业务需求进行自定义设置，详见 。 + 视频帧的色彩空间属性，默认情况下会应用 Full Range 和 BT.709 标准配置。你可以根据自定义采集、自定义渲染的业务需求进行自定义设置，详见 。 sourceType From 52d7c551dac51a58694aff1429c5beeb009a3a1a Mon Sep 17 00:00:00 2001 From: jinyu Date: Wed, 30 Oct 2024 16:54:16 +0800 Subject: [PATCH 18/19] 1 --- dita/RTC-NG/API/rtc_api_data_type.dita | 1 + dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap | 17 ++++++++++++----- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/dita/RTC-NG/API/rtc_api_data_type.dita b/dita/RTC-NG/API/rtc_api_data_type.dita index a9e804ce1bb..6fe27c9c5d9 100644 --- a/dita/RTC-NG/API/rtc_api_data_type.dita +++ b/dita/RTC-NG/API/rtc_api_data_type.dita @@ -1278,6 +1278,7 @@

      • +
      • diff --git a/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap b/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap index 7354e4a2325..03c6f2e7eeb 100644 --- a/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap +++ b/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap @@ -5822,38 +5822,45 @@ + + + + AlphaStitchMode + + + - VIDEO_NO_ALPHA_STITCH + ALPHA_NO_STITCH - VIDEO_ALPHA_STITCH_UP + ALPHA_STITCH_UP - VIDEO_ALPHA_STITCH_BELOW + ALPHA_STITCH_BELOW - VIDEO_ALPHA_STITCH_LEFT + ALPHA_STITCH_LEFT - VIDEO_ALPHA_STITCH_RIGHT + ALPHA_STITCH_RIGHT From d471cca113854bb4179701331f236d5f70646387 Mon Sep 17 00:00:00 2001 From: jinyuagora Date: Wed, 30 Oct 2024 08:55:34 +0000 Subject: [PATCH 19/19] Sync cn prototype to en prototype --- .../RTC-NG/API/class_audiotrackconfig.dita | 26 +- .../RTC-NG/API/class_externalvideoframe.dita | 61 +++-- en-US/dita/RTC-NG/API/class_videoframe.dita | 234 ++++++++++++++---- .../RTC-NG/config/keys-rtc-ng-api-cpp.ditamap | 14 ++ .../RTC-NG/config/keys-rtc-ng-api-ios.ditamap | 7 + .../config/keys-rtc-ng-api-java.ditamap | 159 +++++++++++- .../config/keys-rtc-ng-api-macos.ditamap | 7 + 7 files changed, 421 insertions(+), 87 deletions(-) diff --git a/en-US/dita/RTC-NG/API/class_audiotrackconfig.dita b/en-US/dita/RTC-NG/API/class_audiotrackconfig.dita index 165bd8139fd..b7591ff0e36 100644 --- a/en-US/dita/RTC-NG/API/class_audiotrackconfig.dita +++ b/en-US/dita/RTC-NG/API/class_audiotrackconfig.dita @@ -7,27 +7,35 @@

      public class AudioTrackConfig { + public boolean enableLocalPlayback; - - + + public boolean enableAudioProcessing; public AudioTrackConfig() { this.enableLocalPlayback = true; + this.enableAudioProcessing = false; + } + @Override + public String toString() { + return "AudioTrackConfig{" + + "enableLocalPlayback=" + enableLocalPlayback + "enableAudioProcessing" + + enableAudioProcessing + '}'; } - } +} export class AudioTrackConfig { public enableLocalPlayback: boolean = true; } - NS_SWIFT_NAME(AgoraAudioTrackConfig) __attribute__((visibility("default"))) @interface AgoraAudioTrackConfig : NSObject + NS_SWIFT_NAME(AgoraAudioTrackConfig) __attribute__((visibility("default"))) @interface AgoraAudioTrackConfig : NSObject @property (assign, nonatomic) BOOL enableLocalPlayback NS_SWIFT_NAME(enableLocalPlayback); - +@property (assign, nonatomic) BOOL enableAudioProcessing NS_SWIFT_NAME(enableAudioProcessing); @end struct AudioTrackConfig { + bool enableLocalPlayback; - - - AudioTrackConfig() - : enableLocalPlayback(true) {} + + bool enableAudioProcessing; + AudioTrackConfig() : enableLocalPlayback(true),enableAudioProcessing(false) {} }; USTRUCT(BlueprintType) struct FAudioTrackConfig diff --git a/en-US/dita/RTC-NG/API/class_externalvideoframe.dita b/en-US/dita/RTC-NG/API/class_externalvideoframe.dita index cede51769e3..541d031a3a5 100644 --- a/en-US/dita/RTC-NG/API/class_externalvideoframe.dita +++ b/en-US/dita/RTC-NG/API/class_externalvideoframe.dita @@ -20,7 +20,7 @@ public static final int BUFFER_TYPE_ARRAY = 2; public static final int BUFFER_TYPE_TEXTURE = 3; public AgoraVideoFrame() { - format = 10; + format = 10; timeStamp = 0; stride = 0; height = 0; @@ -37,6 +37,7 @@ rotation = 0; alphaStitchMode = 0; } + public int format; public long timeStamp; public int stride; @@ -46,7 +47,6 @@ public float[] transform; public javax.microedition.khronos.egl.EGLContext eglContext10; public android.opengl.EGLContext eglContext14; - public byte[] buf; public int cropLeft; public int cropTop; @@ -56,34 +56,43 @@ public int alphaStitchMode; @Override public String toString() { - return "AgoraVideoFrame{" - + "format=" + format + ", timeStamp=" + timeStamp + ", stride=" + stride - + ", height=" + height + ", textureID=" + textureID - + ", buf.length=" + (buf != null ? buf.length : 0) + ", cropLeft=" + cropLeft - + ", cropTop=" + cropTop + ", cropRight=" + cropRight + ", cropBottom=" + cropBottom - + ", rotation=" + rotation + ", alphaStitchMode=" + alphaStitchMode + '}'; + return "AgoraVideoFrame{" + + "format=" + format + ", timeStamp=" + timeStamp + ", stride=" + stride + + ", height=" + height + ", textureID=" + textureID + + ", buf.length=" + (buf != null ? buf.length : 0) + ", cropLeft=" + cropLeft + + ", cropTop=" + cropTop + ", cropRight=" + cropRight + ", cropBottom=" + cropBottom + + ", rotation=" + rotation + ", alphaStitchMode=" + alphaStitchMode + '}'; } } - __attribute__((visibility("default"))) @interface AgoraVideoFrame : NSObject + +__attribute__((visibility("default"))) @interface AgoraVideoFrame : NSObject @property(assign, nonatomic) NSInteger format; -@property(assign, nonatomic) CMTime time; -@property(assign, nonatomic) int stride DEPRECATED_MSG_ATTRIBUTE("use strideInPixels instead"); -@property(assign, nonatomic) int strideInPixels; -@property(assign, nonatomic) int height; + +@property(assign, nonatomic) CMTime time; +@property(assign, nonatomic) int stride DEPRECATED_MSG_ATTRIBUTE("use strideInPixels instead"); + +@property(assign, nonatomic) int strideInPixels; +@property(assign, nonatomic) int height; @property(assign, nonatomic) CVPixelBufferRef _Nullable textureBuf; + @property(strong, nonatomic) IMAGE_CLASS * _Nullable image; -@property(strong, nonatomic) NSData *_Nullable dataBuf; + +@property(strong, nonatomic) NSData *_Nullable dataBuf; @property(strong, nonatomic) NSData *_Nullable alphaBuf; @property(assign, nonatomic) AgoraAlphaStitchMode alphaStitchMode; -@property(assign, nonatomic) int cropLeft; -@property(assign, nonatomic) int cropTop; -@property(assign, nonatomic) int cropRight; -@property(assign, nonatomic) int cropBottom; -@property(assign, nonatomic) int rotation; + +@property(assign, nonatomic) int cropLeft; +@property(assign, nonatomic) int cropTop; +@property(assign, nonatomic) int cropRight; +@property(assign, nonatomic) int cropBottom; +@property(assign, nonatomic) int rotation; +@property(strong, nonatomic) AgoraColorSpace *_Nullable colorSpace; + - (void)fillAlphaData; @end - struct ExternalVideoFrame { + +struct ExternalVideoFrame { ExternalVideoFrame() : type(VIDEO_BUFFER_RAW_DATA), format(VIDEO_PIXEL_DEFAULT), @@ -99,22 +108,26 @@ eglContext(NULL), eglType(EGL_CONTEXT10), textureId(0), + fenceObject(0), metadataBuffer(NULL), metadataSize(0), alphaBuffer(NULL), fillAlphaBuffer(false), - alphaStitchMode(0), + alphaStitchMode(NO_ALPHA_STITCH), d3d11Texture2d(NULL), textureSliceIndex(0){} + enum EGL_CONTEXT_TYPE { EGL_CONTEXT10 = 0, EGL_CONTEXT14 = 1, }; + enum VIDEO_BUFFER_TYPE { VIDEO_BUFFER_RAW_DATA = 1, VIDEO_BUFFER_ARRAY = 2, VIDEO_BUFFER_TEXTURE = 3, }; + VIDEO_BUFFER_TYPE type; VIDEO_PIXEL_FORMAT format; void* buffer; @@ -126,17 +139,19 @@ int cropBottom; int rotation; long long timestamp; - void *eglContext; + void* eglContext; EGL_CONTEXT_TYPE eglType; int textureId; + long long fenceObject; float matrix[16]; uint8_t* metadataBuffer; int metadataSize; uint8_t* alphaBuffer; bool fillAlphaBuffer; - int alphaStitchMode; + ALPHA_STITCH_MODE alphaStitchMode; void *d3d11Texture2d; int textureSliceIndex; + ColorSpace colorSpace; }; export class ExternalVideoFrame { type?: VideoBufferType; diff --git a/en-US/dita/RTC-NG/API/class_videoframe.dita b/en-US/dita/RTC-NG/API/class_videoframe.dita index a478bc371d9..62f94f3605c 100644 --- a/en-US/dita/RTC-NG/API/class_videoframe.dita +++ b/en-US/dita/RTC-NG/API/class_videoframe.dita @@ -7,42 +7,61 @@

      public class VideoFrame implements RefCounted { + public interface Buffer extends RefCounted { - @CalledByNative("Buffer") int getWidth(); - @CalledByNative("Buffer") int getHeight(); - @CalledByNative("Buffer") I420Buffer toI420(); - @Override @CalledByNative("Buffer") void release(); - @Override @CalledByNative("Buffer") void retain(); - @CalledByNative("Buffer") + + @CalledByNative("Buffer") int getWidth(); + + @CalledByNative("Buffer") int getHeight(); + + @CalledByNative("Buffer") I420Buffer toI420(); + + @Override @CalledByNative("Buffer") void release(); + + @Override @CalledByNative("Buffer") void retain(); + + @CalledByNative("Buffer") Buffer cropAndScale( int cropX, int cropY, int cropWidth, int cropHeight, int scaleWidth, int scaleHeight); - @CalledByNative("Buffer") @Nullable Buffer mirror(int frameRotation); - @CalledByNative("Buffer") @Nullable Buffer rotate(int frameRotation); - @CalledByNative("Buffer") + + @CalledByNative("Buffer") @Nullable Buffer mirror(int frameRotation); + + @CalledByNative("Buffer") @Nullable Buffer rotate(int frameRotation); + + @CalledByNative("Buffer") @Nullable Buffer transform(int cropX, int cropY, int cropWidth, int cropHeight, int scaleWidth, int scaleHeight, int frameRotation); } + public interface I420Buffer extends Buffer { - @CalledByNative("I420Buffer") ByteBuffer getDataY(); - @CalledByNative("I420Buffer") ByteBuffer getDataU(); - @CalledByNative("I420Buffer") ByteBuffer getDataV(); - @CalledByNative("I420Buffer") int getStrideY(); - @CalledByNative("I420Buffer") int getStrideU(); - @CalledByNative("I420Buffer") int getStrideV(); + + @CalledByNative("I420Buffer") ByteBuffer getDataY(); + + @CalledByNative("I420Buffer") ByteBuffer getDataU(); + + @CalledByNative("I420Buffer") ByteBuffer getDataV(); + @CalledByNative("I420Buffer") int getStrideY(); + @CalledByNative("I420Buffer") int getStrideU(); + @CalledByNative("I420Buffer") int getStrideV(); } + public interface I422Buffer extends Buffer { - @CalledByNative("I422Buffer") ByteBuffer getDataY(); - @CalledByNative("I422Buffer") ByteBuffer getDataU(); - @CalledByNative("I422Buffer") ByteBuffer getDataV(); - @CalledByNative("I422Buffer") int getStrideY(); - @CalledByNative("I422Buffer") int getStrideU(); - @CalledByNative("I422Buffer") int getStrideV(); - } - public interface RgbaBuffer extends Buffer { @CalledByNative("RgbaBuffer") ByteBuffer getData(); } + @CalledByNative("I422Buffer") ByteBuffer getDataY(); + @CalledByNative("I422Buffer") ByteBuffer getDataU(); + @CalledByNative("I422Buffer") ByteBuffer getDataV(); + @CalledByNative("I422Buffer") int getStrideY(); + @CalledByNative("I422Buffer") int getStrideU(); + @CalledByNative("I422Buffer") int getStrideV(); + } + public interface RgbaBuffer extends Buffer { @CalledByNative("RgbaBuffer") ByteBuffer getData(); } + public interface TextureBuffer extends Buffer { + enum Type { + OES(GLES11Ext.GL_TEXTURE_EXTERNAL_OES), + RGB(GLES20.GL_TEXTURE_2D); private final int glTarget; private Type(final int glTarget) { @@ -57,22 +76,26 @@ EGL_CONTEXT_14; } Type getType(); - @CalledByNative("TextureBuffer") int getTextureId(); + + @CalledByNative("TextureBuffer") int getTextureId(); + Matrix getTransformMatrix(); - @CalledByNative("TextureBuffer") EglBase.Context getEglBaseContext(); - @CalledByNative("TextureBuffer") Object getSourceTexturePool(); - @CalledByNative("TextureBuffer") long getNativeEglContext(); - @CalledByNative("TextureBuffer") int getEglContextType(); - @CalledByNative("TextureBuffer") float[] getTransformMatrixArray(); - @CalledByNative("TextureBuffer") int getSequence(); - @CalledByNative("TextureBuffer") boolean is10BitTexture(); + + @CalledByNative("TextureBuffer") EglBase.Context getEglBaseContext(); + @CalledByNative("TextureBuffer") Object getSourceTexturePool(); + @CalledByNative("TextureBuffer") long getNativeEglContext(); + @CalledByNative("TextureBuffer") int getEglContextType(); + @CalledByNative("TextureBuffer") float[] getTransformMatrixArray(); + + @CalledByNative("TextureBuffer") int getSequence(); + @CalledByNative("TextureBuffer") long getFenceObject(); + @CalledByNative("TextureBuffer") boolean is10BitTexture(); } public interface ColorSpace { enum Range { Invalid(0), Limited(1), - Full(2), - Derived(3); + Full(2); private final int range; private Range(int range) { this.range = range; @@ -161,32 +184,55 @@ kBackCamera, kUnspecified, } + public enum AlphaStitchMode { + ALPHA_NO_STITCH(0), + ALPHA_STITCH_UP(1), + ALPHA_STITCH_BELOW(2), + ALPHA_STITCH_LEFT(3), + ALPHA_STITCH_RIGHT(4); + private final int stitchMode; + private AlphaStitchMode(int stitchMode) { + this.stitchMode = stitchMode; + } + public int value() { + return stitchMode; + } + } + private Buffer buffer; + private int rotation; + private long timestampNs; private ColorSpace colorSpace; private SourceType sourceType; private float sampleAspectRatio; + + private AlphaStitchMode alphaStitchMode = AlphaStitchMode.ALPHA_NO_STITCH; private VideoFrameMetaInfo metaInfo = new VideoFrameMetaInfo(); + private @Nullable ByteBuffer alphaBuffer; - private int alphaStitchMode; + private long nativeAlphaBuffer; + public VideoFrame(Buffer buffer, int rotation, long timestampNs) { - this(buffer, rotation, timestampNs, new WrappedNativeColorSpace(), null, 0L, 1.0f, SourceType.kUnspecified.ordinal()); + this(buffer, rotation, timestampNs, new WrappedNativeColorSpace(), null, 0L, 1.0f, + SourceType.kUnspecified.ordinal()); } @CalledByNative public VideoFrame(Buffer buffer, int rotation, long timestampNs, ColorSpace colorSpace, - ByteBuffer alphaBuffer, float sampleAspectRatio, int sourceType) { + ByteBuffer alphaBuffer, long nativeAlphaBuffer, float sampleAspectRatio, int sourceType) { if (buffer == null) { - throw new IllegalArgumentException("buffer not allowed to be null"); + throw new IllegalArgumentException("buffer not allowed to be null"); } if (rotation % 90 != 0) { - throw new IllegalArgumentException("rotation must be a multiple of 90"); + throw new IllegalArgumentException("rotation must be a multiple of 90"); } this.buffer = buffer; this.rotation = rotation; this.timestampNs = timestampNs; this.colorSpace = colorSpace; this.alphaBuffer = alphaBuffer; + this.nativeAlphaBuffer = nativeAlphaBuffer; this.sampleAspectRatio = sampleAspectRatio; this.sourceType = SourceType.values()[sourceType]; } @@ -197,18 +243,25 @@ public float getSampleAspectRatio() { return sampleAspectRatio; } + @CalledByNative public Buffer getBuffer() { return buffer; } + @CalledByNative public int getRotation() { return rotation; } @CalledByNative public int getAlphaStitchMode() { - return alphaStitchMode; + return alphaStitchMode.value(); } + @CalledByNative + public void setAlphaStitchMode(int stitchMode) { + alphaStitchMode = AlphaStitchMode.values()[stitchMode]; + } + @CalledByNative public long getTimestampNs() { return timestampNs; @@ -217,41 +270,93 @@ public VideoFrameMetaInfo getMetaInfo() { return metaInfo; } + public int getRotatedWidth() { if (rotation % 180 == 0) { - return buffer.getWidth(); + return (alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_LEFT + || alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_RIGHT) + ? buffer.getWidth() / 2 + : buffer.getWidth(); } - return buffer.getHeight(); + return (alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_UP + || alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_BELOW) + ? buffer.getHeight() / 2 + : buffer.getHeight(); } + public int getRotatedHeight() { if (rotation % 180 == 0) { - return buffer.getHeight(); + return (alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_UP + || alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_BELOW) + ? buffer.getHeight() / 2 + : buffer.getHeight(); } - return buffer.getWidth(); + return (alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_LEFT + || alphaStitchMode == AlphaStitchMode.ALPHA_STITCH_RIGHT) + ? buffer.getWidth() / 2 + : buffer.getWidth(); } + public void replaceBuffer(Buffer buffer, int rotation, long timestampNs) { release(); this.buffer = buffer; this.rotation = rotation; this.timestampNs = timestampNs; } + @CalledByNative public ColorSpace getColorSpace() { return colorSpace; } + public void setColorSpace(ColorSpace colorSpace) { + this.colorSpace = colorSpace; + } + @CalledByNative + private int getColorSpaceRange() { + if (colorSpace == null) { + return ColorSpace.Range.Invalid.getRange(); + } + return colorSpace.getRange().getRange(); + } + @CalledByNative + private int getColorSpaceMatrix() { + if (colorSpace == null) { + return ColorSpace.Matrix.Unspecified.getMatrix(); + } + return colorSpace.getMatrix().getMatrix(); + } + @CalledByNative + private int getColorSpaceTransfer() { + if (colorSpace == null) { + return ColorSpace.Transfer.Unspecified.getTransfer(); + } + return colorSpace.getTransfer().getTransfer(); + } + @CalledByNative + private int getColorSpacePrimary() { + if (colorSpace == null) { + return ColorSpace.Primary.Unspecified.getPrimary(); + } + return colorSpace.getPrimary().getPrimary(); + } @CalledByNative public ByteBuffer getAlphaBuffer() { return alphaBuffer; } + public void retainAlphaBuffer() { + JniCommon.nativeAddRef(nativeAlphaBuffer); + } + public void releaseAlphaBuffer() { + JniCommon.nativeReleaseRef(nativeAlphaBuffer); + } public void fillAlphaData(ByteBuffer buffer) { alphaBuffer = buffer; } - public void setAlphaStitchMode(int mode) { - this.alphaStitchMode = mode; - } + @Override public void retain() { buffer.retain(); } + @Override @CalledByNative public void release() { @@ -278,7 +383,8 @@ this.format = format; } } - __attribute__((visibility("default"))) @interface AgoraOutputVideoFrame : NSObject + __attribute__((visibility("default"))) @interface AgoraOutputVideoFrame : NSObject + @property (nonatomic, assign) NSInteger type; @property (nonatomic, assign) int width; @property (nonatomic, assign) int height; @@ -291,12 +397,16 @@ @property (nonatomic, assign) int rotation; @property (nonatomic, assign) int64_t renderTimeMs; @property (nonatomic, assign) int avSyncType; + @property(assign, nonatomic) CVPixelBufferRef _Nullable pixelBuffer; @property (nonatomic, assign) uint8_t* _Nullable alphaBuffer; @property (nonatomic, assign) AgoraAlphaStitchMode alphaStitchMode; + @property(nonatomic, strong) NSDictionary *_Nonnull metaInfo; +@property(nonatomic, strong) AgoraColorSpace* _Nullable colorSpace; @end - struct VideoFrame { + +struct VideoFrame { VideoFrame(): type(VIDEO_PIXEL_DEFAULT), width(0), @@ -316,33 +426,57 @@ textureId(0), d3d11Texture2d(NULL), alphaBuffer(NULL), - alphaStitchMode(0), + alphaStitchMode(NO_ALPHA_STITCH), pixelBuffer(NULL), metaInfo(NULL){ memset(matrix, 0, sizeof(matrix)); } + VIDEO_PIXEL_FORMAT type; + int width; + int height; + int yStride; + int uStride; + int vStride; + uint8_t* yBuffer; + uint8_t* uBuffer; + uint8_t* vBuffer; + int rotation; + int64_t renderTimeMs; + int avsync_type; + uint8_t* metadata_buffer; + int metadata_size; + void* sharedContext; + int textureId; + void* d3d11Texture2d; + float matrix[16]; + uint8_t* alphaBuffer; - int alphaStitchMode; + + ALPHA_STITCH_MODE alphaStitchMode; + void* pixelBuffer; + IVideoFrameMetaInfo* metaInfo; + + ColorSpace colorSpace; }; USTRUCT(BlueprintType) struct FVideoFrame { diff --git a/en-US/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap b/en-US/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap index bf4073b8d0e..4500cb04a5b 100644 --- a/en-US/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap +++ b/en-US/dita/RTC-NG/config/keys-rtc-ng-api-cpp.ditamap @@ -2611,6 +2611,13 @@ + + + + setExternalRemoteEglContext + + + @@ -11807,6 +11814,13 @@ + + + + APPLICATION_SCENARIO_LIVESHOW + + + diff --git a/en-US/dita/RTC-NG/config/keys-rtc-ng-api-ios.ditamap b/en-US/dita/RTC-NG/config/keys-rtc-ng-api-ios.ditamap index 1bdc38893e8..fac4aadfa53 100644 --- a/en-US/dita/RTC-NG/config/keys-rtc-ng-api-ios.ditamap +++ b/en-US/dita/RTC-NG/config/keys-rtc-ng-api-ios.ditamap @@ -10912,6 +10912,13 @@ + + + + AgoraApplicationLiveShowScenario + + + diff --git a/en-US/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap b/en-US/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap index dd3da6e473e..03c6f2e7eeb 100644 --- a/en-US/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap +++ b/en-US/dita/RTC-NG/config/keys-rtc-ng-api-java.ditamap @@ -1058,6 +1058,20 @@ + + + + setLocalRenderTargetFps + + + + + + + setRemoteRenderTargetFps + + + @@ -1556,6 +1570,13 @@ + + + + setExternalMediaProjection + + + @@ -2507,6 +2528,29 @@ + + + + + startLocalAudioMixer + + + + + + + stopLocalAudioMixer + + + + + + + updateLocalAudioMixerConfiguration + + + + @@ -2715,6 +2759,13 @@ + + + + setExternalRemoteEglContext + + + @@ -4824,6 +4875,69 @@ + + + + AudioSourceType + + + + + + + AUDIO_SOURCE_MICROPHONE + + + + + + + AUDIO_SOURCE_CUSTOM + + + + + + + AUDIO_SOURCE_MEDIA_PLAYER + + + + + + + AUDIO_SOURCE_LOOPBACK_RECORDING + + + + + + + AUDIO_SOURCE_MIXED_STREAM + + + + + + + AUDIO_SOURCE_REMOTE_USER + + + + + + + AUDIO_SOURCE_REMOTE_CHANNEL + + + + + + + AUDIO_SOURCE_UNKNOWN + + + @@ -5090,6 +5204,13 @@ + + + + LocalAudioMixerConfiguration + + + @@ -5153,6 +5274,13 @@ + + + + MixedAudioStream + + + @@ -5694,38 +5822,45 @@ + + + + AlphaStitchMode + + + - VIDEO_NO_ALPHA_STITCH + ALPHA_NO_STITCH - VIDEO_ALPHA_STITCH_UP + ALPHA_STITCH_UP - VIDEO_ALPHA_STITCH_BELOW + ALPHA_STITCH_BELOW - VIDEO_ALPHA_STITCH_LEFT + ALPHA_STITCH_LEFT - VIDEO_ALPHA_STITCH_RIGHT + ALPHA_STITCH_RIGHT @@ -6016,6 +6151,13 @@ + + + + PREFER_COMPRESSION_AUTO + + + @@ -10672,6 +10814,13 @@ + + + + APPLICATION_SCENARIO_LIVESHOW + + + diff --git a/en-US/dita/RTC-NG/config/keys-rtc-ng-api-macos.ditamap b/en-US/dita/RTC-NG/config/keys-rtc-ng-api-macos.ditamap index 1b4ef97a989..251e3519f5a 100644 --- a/en-US/dita/RTC-NG/config/keys-rtc-ng-api-macos.ditamap +++ b/en-US/dita/RTC-NG/config/keys-rtc-ng-api-macos.ditamap @@ -10357,6 +10357,13 @@ + + + + AgoraApplicationLiveShowScenario + + +