integrate ggml's excellent and amazing whisper.cpp to kantv ---- step7

various experiment and invesigation for asr performance on Xiaomi 14

asr performance from 21 secs to 2 secs on Xiaomi 14 by build optimization

refine regular codes and prepare for coding work of implment real-time
English subtile for online English TV

README.md

@@ -12,7 +12,7 @@ KanTV("Kan", aka Chinese PinYin "Kan" or Chinese HanZi "看" or English "watch/l
 
 - Watch encrypted live stream(Google Widevine, Huawei WisePlay, ChinaDRM......)
 
-- Real-time English subtitle by excellent&amazing <a href="https://github.com/ggerganov/whisper.cpp">whisper.cpp</a> for unencrypted/clear online TV  (not finished)  
+- Real-time English subtitle for online TV by excellent&amazing <a href="https://github.com/ggerganov/whisper.cpp">whisper.cpp</a>  (not finished)  
 
 - UI refactor
 
@@ -104,13 +104,13 @@ bazel is **NOT** used currently but put it here for further usage in the future.
   [Android Studio 4.2.1](https://developer.android.google.cn/studio)
 
 
-  [Android NDK-r21e](https://developer.android.com/ndk/downloads)
+  [Android NDK-r26c](https://developer.android.com/ndk/downloads)
 
 
-  then put Android NDK-r21e into /opt/kantv-toolchain accordingly
+  then put Android NDK-r26c into /opt/kantv-toolchain accordingly
 
   ```
-  ls /opt/kantv-toolchain/android-ndk-r21e
+  ls /opt/kantv-toolchain/android-ndk-r26c
   
   ```
 
@@ -176,7 +176,7 @@ time ./build-all.sh
 
 - Build APK from source code by Android Studio IDE manually
 
-  Please attention ![some source codes in ASRFragment.java](https://github.com/cdeos/kantv/blob/master/cdeosplayer/kantv/src/main/java/com/cdeos/kantv/ui/fragment/ASRFragment.java#L131) which affect the running of the ASR demo and the size of the generated APK.
+  Please attention ![some source codes in ASRFragment.java](https://github.com/cdeos/kantv/blob/kantv-poc-with-whispercpp/cdeosplayer/kantv/src/main/java/com/cdeos/kantv/ui/fragment/ASRFragment.java#L155) which affect the running of the ASR demo and the size of the generated APK.
 
 - Latest prebuit APK could be found here [![Github](https://user-images.githubusercontent.com/6889919/122489234-c13db400-d011-11eb-9d8c-8e4b2555dabe.png)](https://github.com/cdeos/kantv/raw/master/release/kantv-latest.apk)(the size of the prebuilt APK is about 90M because it contains **dependent model file of DeepSpeech** for purpose of make ASR demo happy).
 
@@ -192,11 +192,10 @@ The following is some English snapshots.
 
 ![Screenshot_20240301_000503_com cdeos kantv](https://github.com/zhouwg/kantv/assets/6889919/07653f3d-1e7a-4208-a3d8-90b3aecc30b4)
 ![Screenshot_20240301_000509_com cdeos kantv](https://github.com/zhouwg/kantv/assets/6889919/28d549ba-2fd5-434f-bf7a-b66d82d6dde3)
-![990238413](https://github.com/zhouwg/kantv/assets/6889919/44054d57-0149-4d45-8762-46ec80682c66)
+![1210108450](https://github.com/cdeos/kantv/assets/6889919/9f82c290-2ed6-444c-98d4-ef840cdd9083)
 
 
 
-![Screenshot_20240301_114059_com cdeos kantv](https://github.com/zhouwg/kantv/assets/6889919/b0171435-44a5-48bf-9b59-a4b5fbcaa39f)
 ![Screenshot_20240301_114116_com cdeos kantv](https://github.com/zhouwg/kantv/assets/6889919/10224799-cdf8-46f7-acd0-6df64f0fc674)
 ![Screenshot_2024_0304_131033](https://github.com/zhouwg/kantv/assets/6889919/6c5bd531-5577-4570-bc87-aa3a87822d6b)
 
@@ -216,9 +215,9 @@ Changelog could be found <a href="https://github.com/cdeos/kantv/blob/master/rel
 
 ### Roadmap
 
-- real-time English subtitle for online English TV on Xiaomi 14 by excellent and amazing ![whisper.cpp](https://github.com/ggerganov/whisper.cpp)
+- real-time English subtitle for online English TV on Xiaomi 14(because it contains a very powerful mobile SoC) by excellent and amazing ![whisper.cpp](https://github.com/ggerganov/whisper.cpp)
 
-- real-time Chinese subtitle for online English TV on Xiaomi 14 by excellent and amazing ![whisper.cpp](https://github.com/ggerganov/whisper.cpp)
+- real-time Chinese subtitle for online English TV on Xiaomi 14(because it contains a very powerful mobile SoC) by excellent and amazing ![whisper.cpp](https://github.com/ggerganov/whisper.cpp)
 
 - integrate ![gstreamer](https://github.com/cdeos/gstreamer) to project KanTV(<a href="https://www.videolan.org/vlc/" target="_blank">VLC</a> is also excellent and gstreamer is more complicated than VLC but gstreamer was supported by many semiconductor companies. anyway, they are both born in/come from EU)
 
@@ -310,15 +309,16 @@ Report issue in various Android-based phone or even submit PR to this project is
 
 ### Acknowledgement
 
-Many/sincerely thanks to all contributors of the great open source community, especially all original authors and all contributors of the great Linux & Android & FFmpeg and other excellent projects. 
+Many/sincerely thanks to all contributors of the great open source community, especially all original authors and all contributors of the great Linux & Android & FFmpeg && whisper.cpp and other excellent projects. 
 
 Project KanTV has used/tried following open-source projects(list in here is not incomplete):
 <ul>
+  <li><a href="https://github.com/torvalds/linux"     target="_blank">Linux</a></li>
   <li><a href="https://blog.google/products/android/" target="_blank" rel="noopener">Android</a></li>
  	<li><a href="http://ffmpeg.org/" target="_blank" rel="noopener">FFmpeg</a></li>
- 	<li><a href="https://github.com/bilibili/ijkplayer" target="_blank" rel="noopener">ijkplayer</a></li>
- 	<li><a href="https://github.com/google/ExoPlayer" target="_blank" rel="noopener">ExoPlayer</a></li>
   <li><a href="https://github.com/ggerganov/whisper.cpp" target="_blank" rel="noopener">whisper.cpp</a></li>
+  <li><a href="https://github.com/bilibili/ijkplayer" target="_blank" rel="noopener">ijkplayer</a></li>
+ 	<li><a href="https://github.com/google/ExoPlayer" target="_blank" rel="noopener">ExoPlayer</a></li>
  	<li><a href="https://www.videolan.org/vlc/" target="_blank" rel="noopener">libx264/libx265</a></li>
  	<li><a href="https://github.com/mozilla/DeepSpeech" target="_blank" rel="noopener">DeepSpeech</a></li>
  	<li><a href="https://www.intel.com/content/www/us/en/developer/articles/technical/scalable-video-technology.html" target="_blank" rel="noopener">SVT-AV1</a></li>

build/envsetup.sh

@@ -46,7 +46,25 @@ export LOCAL_WHISPERCPP_PATH=${PROJECT_ROOT_PATH}/external/whispercpp
 #modify following lines to adapt to local dev envs
 #export KANTV_TOOLCHAIN_PATH=${PROJECT_ROOT_PATH}/toolchain
 export KANTV_TOOLCHAIN_PATH=/opt/kantv-toolchain
-export ANDROID_NDK=${KANTV_TOOLCHAIN_PATH}/android-ndk-r21e
+#API21：Android 5.0 (Android L)Lollipop
+#API22：Android 5.1 (Android L)Lollipop
+#API23：Android 6.0 (Android M)Marshmallow
+#API24: Android 7.0 (Android N)Nougat
+#API25: Android 7.1 (Android N)Nougat
+#API26：Android 8.0 (Android O)Oreo
+#API27：Android 8.1 (Android O)Oreo
+#API28：Android 9.0 (Android P)Pie
+#API29：Android 10
+#API30：Android 11
+#API31：Android 12
+#API31：Android 12L
+#API33：Android 13
+#API34：Android 14
+export ANDROID_PLATFORM=android-34
+#export ANDROID_NDK=${KANTV_TOOLCHAIN_PATH}/android-ndk-r18b
+#export ANDROID_NDK=${KANTV_TOOLCHAIN_PATH}/android-ndk-r21e
+#export ANDROID_NDK=${KANTV_TOOLCHAIN_PATH}/android-ndk-r24
+export ANDROID_NDK=${KANTV_TOOLCHAIN_PATH}/android-ndk-r26c
 export LOCAL_BAZEL_PATH=${HOME_PATH}/.cache/bazel/_bazel_${BUILD_USER}/d483cd2a2d9204cb5bb4d870c2729238
 export UPSTREAM_WHISPERCPP_PATH=~/cdeos/whisper.cpp
 

build/public.sh

@@ -216,14 +216,14 @@ function dump_global_envs()
     echo -e "PROJECT_HOME_PATH:    ${PROJECT_HOME_PATH}"
     echo -e "PROJECT_ROOT_PATH:    ${PROJECT_ROOT_PATH}"
     echo -e "KANTV_TOOLCHAIN_PATH: ${KANTV_TOOLCHAIN_PATH}"
-
     echo -e "host cpu counts:      ${HOST_CPU_COUNTS}"
 
     if [ "${BUILD_TARGET}" = "android" ]; then
     echo -e "\n"
-    echo -e "ANDROID_NDK:               ${ANDROID_NDK}"
-    echo -e "LOCAL_WHISPERCPP_PATH:     ${LOCAL_WHISPERCPP_PATH}"
-    echo -e "UPSTREAM_WHISPERCPP_PATH:  ${UPSTREAM_WHISPERCPP_PATH}"
+    echo -e "${TEXT_RED}ANDROID_PLATFORM:         ${ANDROID_PLATFORM}${TEXT_RESET}"
+    echo -e "${TEXT_RED}ANDROID_NDK:              ${ANDROID_NDK}${TEXT_RESET}"
+    echo -e "LOCAL_WHISPERCPP_PATH:        ${LOCAL_WHISPERCPP_PATH}"
+    echo -e "UPSTREAM_WHISPERCPP_PATH:     ${UPSTREAM_WHISPERCPP_PATH}"
     echo -e "\n"
     fi
 

cdeosplayer/build.gradle

@@ -8,7 +8,7 @@ buildscript {
         google()
     }
     dependencies {
-        classpath 'com.android.tools.build:gradle:7.0.2'
+        classpath 'com.android.tools.build:gradle:7.0.3'
         classpath 'com.github.dcendents:android-maven-gradle-plugin:1.4.1'
         classpath 'com.jfrog.bintray.gradle:gradle-bintray-plugin:1.7'
     }
@@ -30,30 +30,5 @@ task clean(type: Delete) {
 }
 
 wrapper {
-    gradleVersion = '7.0.2'
+    gradleVersion = '7.0.3'
 }
-
-
-
-/*
-API14: Android 4.0-4.0.2 – Ice Cream Sandwich, released October 2011. Build version
-API15：Android 4.0.3 - 4.0.4 Ice Cream Sandwich
-API16：Android 4.1 Jelly Bean
-API17：Android 4.2 Jelly Bean
-API18：Android 4.3 Jelly Bean
-API19：Android 4.4 KitKat
-API20：Android 4.4W Kitkat Watch
-API21：Android 5.0 (Android L)Lollipop
-API22：Android 5.1 (Android L)Lollipop
-API23：Android 6.0 (Android M)Marshmallow
-API24: Android 7.0 (Android N)Nougat
-API25: Android 7.1 (Android N)Nougat
-API26：Android 8.0 (Android O)Oreo
-API27：Android 8.1 (Android O)Oreo
-API28：Android 9.0 (Android P)Pie
-API29：Android 10.0 (Android Q)
-API30：Android 11.0 (Android R)
-API31：Android 12.0
-API32：Android 13.0
-API33：Android 14.0
-*/

cdeosplayer/cdeosplayer-lib/src/main/java/cdeos/media/player/FFmpegMediaPlayer.java

@@ -211,13 +211,13 @@ public static void loadLibrariesOnce(IjkLibLoader libLoader) {
                 if (libLoader == null)
                     libLoader = sLocalLibLoader;
 
-                CDELog.j(TAG, "load library kantv-ffmpeg-jni");
-                libLoader.loadLibrary("kantv-ffmpeg-jni");
-                CDELog.j(TAG, "after load library kantv-ffmpeg-jni");
+                CDELog.j(TAG, "load library kantv-play-jni");
+                libLoader.loadLibrary("kantv-play-jni");
+                CDELog.j(TAG, "after load library kantv-play-jni");
 
-                CDELog.j(TAG, "load library kantv-ffmpeg");
-                libLoader.loadLibrary("kantv-ffmpeg");
-                CDELog.j(TAG, "after load library kantv-ffmpeg");
+                CDELog.j(TAG, "load library kantv-play");
+                libLoader.loadLibrary("kantv-play");
+                CDELog.j(TAG, "after load library kantv-play");
                 mIsLibLoaded = true;
             }
         }

cdeosplayer/cdeosplayer-lib/src/main/java/cdeos/media/player/KANTVVersion.java

@@ -19,7 +19,7 @@
 
 final class KANTVVersion
 {
-    public static final String KANTV_VERSION="KANTV-v1.3.0";
+    public static final String KANTV_VERSION="KANTV-v1.3.1";
 
     public static native int kantv_anti_tamper();
 }

cdeosplayer/cdeosplayer-lib/src/main/java/org/ggml/whispercpp/whispercpp.java

@@ -8,13 +8,15 @@ public class whispercpp {
 
   public static native String get_systeminfo();
 
+  public static native int    get_cpu_core_counts();
+
   public static native void   set_benchmark_status(int bExitBenchmark);
 
   /**
    *
    * @param modelPath
    * @param audioFile
-   * @param nBenchType  0: memcpy 1: mulmat 2: asr 3: full/whisper_encoder
+   * @param nBenchType  0: asr 1: memcpy 2: mulmat  3: full/whisper_encode
    * @param nThreadCounts
    * @return
    */

cdeosplayer/constants.gradle

@@ -1,4 +1,4 @@
-// Copyright 2017 The Android Open Source Project
+// Copyright 2024 The Android Open Source Project
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -11,18 +11,40 @@
 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 // See the License for the specific language governing permissions and
 // limitations under the License.
+/*
+API14: Android 4.0-4.0.2 – Ice Cream Sandwich, released October 2011. Build version
+API15：Android 4.0.3 - 4.0.4 Ice Cream Sandwich
+API16：Android 4.1 Jelly Bean
+API17：Android 4.2 Jelly Bean
+API18：Android 4.3 Jelly Bean
+API19：Android 4.4 KitKat
+API20：Android 4.4W Kitkat Watch
+API21：Android 5.0 (Android L)Lollipop
+API22：Android 5.1 (Android L)Lollipop
+API23：Android 6.0 (Android M)Marshmallow
+API24: Android 7.0 (Android N)Nougat
+API25: Android 7.1 (Android N)Nougat
+API26：Android 8.0 (Android O)Oreo
+API27：Android 8.1 (Android O)Oreo
+API28：Android 9.0 (Android P)Pie
+API29：Android 10
+API30：Android 11
+API31：Android 12
+API32: Android 12L
+API33：Android 13
+API34：Android 14
+*/
 project.ext {
-    compileSdkVersion = 30
-    buildToolsVersion = "30.0.3"
-    targetSdkVersion  = 28
-    minSdkVersion     = 19
-    //minSdkVersion   = 16
+    //weiguo:2024-03-10, attention here,only validated ok on Xiaomi 14
+    compileSdkVersion = 34
+    buildToolsVersion = "34.0.0"
+    targetSdkVersion  = 34
+    minSdkVersion     = 34
     appTargetSdkVersion = 29
-    ndkVersion = "21.4.7075529"
 
     //version and version code of KanTV
-    releaseVersion     = '1.3.0'
-    releaseVersionCode = 1000300
+    releaseVersion     = '1.3.1'
+    releaseVersionCode = 1000301
 
     //version and version code of internal Exoplayer2 playEngine
     releaseVersionExoplayer     = '2.15.1'
@@ -44,4 +66,4 @@ project.ext {
     if (gradle.ext.has('exoplayerModulePrefix')) {
         modulePrefix += gradle.ext.exoplayerModulePrefix
     }
-}
+}

cdeosplayer/kantv/build.gradle

@@ -44,7 +44,9 @@ android {
         }
     }
 
-    ndkVersion '21.1.6352462'
+    //weiguo:2024-03-10, attention here
+    //ndkVersion '21.1.6352462'
+    ndkVersion  '26.2.11394342'
 
     sourceSets {
         main {

cdeosplayer/kantv/src/main/assets/config.json

@@ -1,6 +1,6 @@
 {
   "kantvServer": "www.cde-os.com",
-  "releaseMode": "0",
-  "apkVersion": "1.3.0",
+  "releaseMode": "1",
+  "apkVersion": "1.3.1",
   "apkForTV": "0"
 }

cdeosplayer/kantv/src/main/java/com/cdeos/kantv/ui/activities/MainActivity.java

@@ -267,7 +267,9 @@ protected void onCreate(@Nullable Bundle savedInstanceState) {
 
         CDEUtils.dumpDeviceInfo();
 
-        if (CDEUtils.isEmulator(this)) {
+        //if (CDEUtils.isEmulator(this))
+        if (false)
+        {
             showWarningDialog(this, "can't running this app in emulator");
             navigationView.getMenu().findItem(R.id.navigation_play).setEnabled(false);
             navigationView.getMenu().findItem(R.id.navigation_home).setEnabled(false);
@@ -320,6 +322,15 @@ private void switchFragment(Class clazz) {
             getFragmentTransaction().hide(previousFragment).commit();
         }
 
+        if (previousFragment != null) {
+            String fragmentName = previousFragment.getClass().getName();
+            if (fragmentName.contains("ASRFragment")) {
+                CDELog.j(TAG, "release ASR resource");
+                asrFragment.release();
+            }
+        }
+
+
         if (clazz == TVGridFragment.class) {
             if (homeFragment == null) {
                 homeFragment = TVGridFragment.newInstance();
@@ -385,4 +396,4 @@ public void onClick(DialogInterface dialog, int which) {
     }
 
     public static native int kantv_anti_tamper();
-}
+}