Skip to content

Commit

Permalink
refine codes according to comments of commit 93397b7
Browse files Browse the repository at this point in the history
  • Loading branch information
@zhouwg
zhouwg committed Mar 9, 2024
1 parent c0604a9 commit 58dedba
Show file tree
Hide file tree
Showing 25 changed files with 37,042 additions and 258 deletions.
18 changes: 11 additions & 7 deletions cdeosplayer/cdeosplayer-lib/src/main/java/org/ggml/whispercpp/whispercpp.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,17 @@
public class whispercpp {
private static final String TAG = whispercpp.class.getName();

public static native String getSystemInfo();
public static native String get_systeminfo();

public static native String benchMemcpy(int nThreadCounts);
public static native void set_benchmark_status(int bExitBenchmark);

public static native String benchMulMat(int nThreadCounts);

public static native void set_mulmat_benchmark_status(int bExitBenchmark);

public static native String transcribe_from_file(String modelPath, String audioFile, int nThreadCounts);
/**
*
* @param modelPath
* @param audioFile
* @param nBenchType 0: memcpy 1: mulmat 2: asr 3: full/whisper_encoder
* @param nThreadCounts
* @return
*/
public static native String bench(String modelPath, String audioFile, int nBenchType, int nThreadCounts);
}
1 change: 1 addition & 0 deletions cdeosplayer/kantv/build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ android {
packagingOptions {
pickFirst 'lib/arm64-v8a/libkantv-core.so'
pickFirst 'lib/arm64-v8a/libwhispercpp.so'
pickFirst 'lib/arm64-v8a/libopenblas.so'
}

versionName project.ext.releaseVersion
Expand Down
2 changes: 1 addition & 1 deletion cdeosplayer/kantv/src/main/assets/config.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"kantvServer": "www.cde-os.com",
"releaseMode": "1",
"releaseMode": "0",
"apkVersion": "1.3.0",
"apkForTV": "0"
}
293 changes: 160 additions & 133 deletions cdeosplayer/kantv/src/main/java/com/cdeos/kantv/ui/fragment/ASRFragment.java
View file

Large diffs are not rendered by default.

Binary file added cdeosplayer/kantv/src/main/jniLibs/arm64-v8a/libopenblas.so
View file
Binary file not shown.
112 changes: 69 additions & 43 deletions cdeosplayer/kantv/src/main/res/layout/fragment_asr.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,56 +39,82 @@
android:layout_height="60dp"
android:layout_marginTop="5dp"
android:text=" " />
</LinearLayout>

<LinearLayout
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="horizontal">

<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="Bench type:" />

<Spinner
android:id="@+id/spinnerBenchType"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:dropDownSelector="#63D81B60"
android:spinnerMode="dropdown" />

<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="Threads:" />

<Spinner
android:id="@+id/spinnerThreadCounts"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:dropDownSelector="#63D81B60"
android:spinnerMode="dropdown" />


</LinearLayout>


<LinearLayout
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="horizontal">

<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="Model:" />

<Spinner
android:id="@+id/spinnerModelName"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:dropDownSelector="#63D81B60"
android:spinnerMode="dropdown" />

<Button
android:id="@+id/btnBenchmark"
android:layout_width="wrap_content"
android:layout_height="30dp"
android:layout_marginLeft="30dp"
android:background="@drawable/button_drawable"
android:text="Benchmark"
android:textAllCaps="false"
android:textColor="#ffffff" />
</LinearLayout>

<LinearLayout
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical">

<TextView
android:id="@+id/asrInfo"
android:layout_width="match_parent"
android:layout_height="150dp"
android:layout_marginTop="5dp"
/>
android:layout_height="600dp"
android:layout_marginTop="5dp" />

<LinearLayout
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="horizontal">

<Button
android:id="@+id/btnBenchmarkMemcpy"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="left"
android:layout_marginLeft="5dp"
android:background="@drawable/button_drawable"
android:text="MemcpyBenchmark"
android:textAllCaps="false"
android:textColor="#ffffff" />

<Button
android:id="@+id/btnBenchmarkMulmat"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="center_horizontal"
android:layout_marginLeft="5dp"
android:layout_marginRight="5dp"
android:background="@drawable/button_drawable"
android:text="MulmatBenchmark"
android:textAllCaps="false"
android:textColor="#ffffff" />

<Button
android:id="@+id/btnBenchmarkInference"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="right"
android:background="@drawable/button_drawable"
android:text="Inference"
android:textAllCaps="false"
android:textColor="#ffffff" />

</LinearLayout>
</LinearLayout>


</LinearLayout>

</ScrollView>
2 changes: 1 addition & 1 deletion cdeosplayer/kantv/src/main/res/values-zh/strings.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@
<string name="EPG">自定义节目</string>
<string name="custom_playlist">自定义节目</string>
<string name="onlinetv">在线电视(严选外语节目)</string>
<string name="ASR">语音研究</string>
<string name="ASR">ASR</string>
<string name="ME">我</string>
<string name="personal_center">个人中心</string>

Expand Down
30 changes: 30 additions & 0 deletions cdeosplayer/kantv/src/main/res/values/arrays.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,4 +71,34 @@
<item>存</item>
<item>不存</item>
</string-array>

<string-array name="benchType">
<item>mempcpy</item>
<item>mulmat</item>
<item>asr</item>
<item>whisper encoder</item>
</string-array>

<string-array name="threadCounts">
<item>1</item>
<item>2</item>
<item>3</item>
<item>4</item>
<item>5</item>
<item>6</item>
<item>7</item>
<item>8</item>
</string-array>

<string-array name="modelName">
<item>tiny</item>
<item>tiny.en</item>
<item>base</item>
<item>base.en</item>
<item>small</item>
<item>small.en</item>
<item>medium</item>
<item>medium.en</item>
<item>large</item>
</string-array>
</resources>
22 changes: 21 additions & 1 deletion external/whispercpp/CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,16 @@ include_directories(${PREBUIT_INC_PATH}/)

add_definitions(-DTARGET_ANDROID)
add_definitions(-D__ARM_NEON)
#add_definitions(-DGGML_USE_CLBLAST)
add_definitions(-DGGML_USE_OPENBLAS)

IF(CMAKE_BUILD_TYPE MATCHES "Debug")
add_definitions(-O0 -g)
ELSE()
add_definitions(-DNDEBUG)
add_definitions(-O3)
#add_definitions(-fvisibility=hidden -fvisibility-inlines-hidden)
ENDIF()


find_library(LOG_LIB log)
Expand All @@ -50,14 +60,24 @@ set_target_properties(kantvcore
IMPORTED_LOCATION
${PREBUIT_LIB_PATH}/libkantv-core.so)


add_library(openblas
SHARED
IMPORTED)

set_target_properties(openblas
PROPERTIES
IMPORTED_LOCATION
${PREBUIT_LIB_PATH}/libopenblas.so)

function(build_library target_name)
add_library(
${target_name}
SHARED
${SOURCE_FILES}
)

target_link_libraries(${target_name} ${LOG_LIB} kantvcore android)
target_link_libraries(${target_name} ${LOG_LIB} kantvcore openblas android)

endfunction()

Expand Down
17 changes: 15 additions & 2 deletions external/whispercpp/bench/CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,27 @@ project(whisperbench)
set(CMAKE_VERBOSE_MAKEFILE on )

set(TARGET ${TARGET_NAME})

set(PREBUIT_INC_PATH ${CMAKE_SOURCE_DIR}/../../../prebuilts/include/)
set(PREBUIT_LIB_PATH ${CMAKE_SOURCE_DIR}/../../../cdeosplayer/kantv/src/main/jniLibs/${ANDROID_ABI}/)

message("project root path: " ${PROJECT_ROOT_PATH})
message("whispercpp path : " ${WHISPERCPP_PATH})
message("PREBUIT_INC_PATH : ${PREBUIT_INC_PATH}")

include_directories(${PREBUIT_INC_PATH}/)

add_library(kantvcore
SHARED
IMPORTED)

set_target_properties(kantvcore
PROPERTIES
IMPORTED_LOCATION
${PREBUIT_LIB_PATH}/libkantv-core.so)

IF (${BUILD_TARGET} MATCHES "android")
add_definitions(-D__ARM_NEON)
link_libraries(log)
link_libraries(log kantvcore android)
ENDIF()


Expand Down
9 changes: 6 additions & 3 deletions external/whispercpp/bench/build.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# Copyright (c) 2024- KanTV Authors. All Rights Reserved.

# Description: build whispercpp's bench tool for target x86-linux and target Android
# Description: build whispercpp's bench tool for target Android(this project only focus on Android-based device)
#

set -e
Expand Down Expand Up @@ -50,6 +50,8 @@ ls -l ${TARGET}
ls -lah ${TARGET}
/bin/cp -fv ${TARGET} ../../${TARGET}_arm64
cd -
echo -e `pwd`
ls -l ${TARGET}_arm64
}


Expand All @@ -63,6 +65,7 @@ ls -l ${TARGET}
ls -lah ${TARGET}
/bin/cp -fv ${TARGET} ../../${TARGET}_armv7a
cd -
ls -l ${TARGET}_armv7a
}


Expand All @@ -79,5 +82,5 @@ cd -
}

build_arm64
build_armv7a
build_x86
#build_armv7a
#build_x86
6 changes: 3 additions & 3 deletions external/whispercpp/build.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ make

ls -l lib${TARGET}.so
ls -lah lib${TARGET}.so
#don't do this because IDE is preferred
#don't do this because IDE is preferred to this open source project for purpose of more easily/friendly
#/bin/cp -fv lib${TARGET}.so ${PROJECT_ROOT_PATH}/cdeosplayer/kantv/src/main/jniLibs/arm64-v8a/
cd -
}
Expand All @@ -62,11 +62,11 @@ make

ls -l lib${TARGET}.so
ls -lah lib${TARGET}.so
#don't do this because IDE is preferred
#don't do this because IDE is preferred to this open source project for purpose of more easily/friendly
#/bin/cp -fv lib${TARGET}.so ${PROJECT_ROOT_PATH}/cdeosplayer/kantv/src/main/jniLibs/armeabi-v7a/
cd -
}


build_arm64
build_armv7a
#build_armv7a
Loading

2 comments on commit 58dedba

@zhouwg
Copy link
Owner Author

@zhouwg zhouwg commented on 58dedba Mar 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ASR/transcription performance on Xiaomi 14 is about 5-10x better then other Android phone, but it's still not enough for real-time subtitle with online TV.

@zhouwg
Copy link
Owner Author

@zhouwg zhouwg commented on 58dedba Mar 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

transcription performance can be improved about 1-3 seconds when enable openblas.

performance of mulmat benchmark seems be improved a lot / significantly when enable openblas.

so I guess Apple's dedicated machine learning library mightbe very important for performance on iOS/Mac.just like GG said before.

and I guess Xiaomi's R&D team has received a lot help from Qualcomm and Qualcomm's dedicated machine learning acceleration library also be used in XiaoAI because Xiaomi is the most important customer of Qualcomm in China. otherwise I can't explain why XiaoAI's English transcription performance is so good(some minor issues with XiaoAI's English transcription while performance of XiaoAI's Chinese transcription is bad and lack of accuracy heavily).

Please sign in to comment.