Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](fe) Fix slf4j contains multiple SLF4J bindings #45945

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

[fix](fe) Fix slf4j contains multiple SLF4J bindings #45945

wants to merge 2 commits into from

Conversation

SWJTU-ZhangLei
Copy link
Contributor

@SWJTU-ZhangLei SWJTU-ZhangLei commented Dec 25, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #45932

Related PR: #

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 25, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@SWJTU-ZhangLei
Copy link
Contributor Author

SWJTU-ZhangLei commented Dec 25, 2024 •
edited
Loading

[ThreadedStreamConsumer] [INFO] Running org.apache.doris.load.loadv2.etl.SparkEtlJobTest java.lang.NoSuchMethodError: org.apache.logging.slf4j.Log4jLoggerFactory: method 'void <init>()' not found at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:53) at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:41) at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150) at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124) at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273) at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:186) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467)

after the fix, fe can print aws sdk debug log, but fe ut failed with new problem above

@SWJTU-ZhangLei SWJTU-ZhangLei marked this pull request as draft December 25, 2024 11:32
@doris-robot
Copy link

TPC-H: Total hot run time: 32663 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 932ce6a1c5fd30e681ddd84d0498a48366668260, data reload: false

------ Round 1 ----------------------------------
q1	17609	6138	6025	6025
q2	2050	303	177	177
q3	10405	1248	714	714
q4	10215	859	425	425
q5	7522	2141	2002	2002
q6	208	185	147	147
q7	895	752	609	609
q8	9259	1390	1172	1172
q9	5362	4991	5008	4991
q10	6767	2292	1860	1860
q11	472	294	265	265
q12	343	356	222	222
q13	17791	3636	2986	2986
q14	234	236	215	215
q15	567	519	508	508
q16	630	621	596	596
q17	548	844	326	326
q18	7056	6568	6425	6425
q19	1234	954	544	544
q20	317	308	192	192
q21	2810	2133	1956	1956
q22	369	330	306	306
Total cold run time: 102663 ms
Total hot run time: 32663 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6194	6161	6196	6161
q2	233	339	237	237
q3	2299	2610	2292	2292
q4	1420	1847	1406	1406
q5	4361	4747	4644	4644
q6	176	171	140	140
q7	1995	1860	1734	1734
q8	2506	2669	2605	2605
q9	6976	6889	6856	6856
q10	2973	3217	2685	2685
q11	570	526	493	493
q12	638	692	575	575
q13	3169	3632	3097	3097
q14	271	292	263	263
q15	569	500	494	494
q16	643	676	636	636
q17	1154	1690	1216	1216
q18	7222	7260	6985	6985
q19	792	985	1104	985
q20	1882	1947	1767	1767
q21	5476	5110	4776	4776
q22	596	615	583	583
Total cold run time: 52115 ms
Total hot run time: 50630 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190480 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 932ce6a1c5fd30e681ddd84d0498a48366668260, data reload: false

query1	969	392	364	364
query2	6519	2306	2324	2306
query3	6721	220	204	204
query4	34202	23745	23350	23350
query5	4395	637	471	471
query6	279	195	181	181
query7	4613	490	306	306
query8	290	244	235	235
query9	9557	2773	2759	2759
query10	469	319	266	266
query11	18235	15406	15027	15027
query12	164	108	104	104
query13	1699	550	422	422
query14	10571	7299	7312	7299
query15	280	204	183	183
query16	8242	620	461	461
query17	1579	742	584	584
query18	2132	399	315	315
query19	233	193	164	164
query20	123	123	123	123
query21	220	121	106	106
query22	4136	4387	4270	4270
query23	34243	33012	33356	33012
query24	6445	2246	2290	2246
query25	483	452	377	377
query26	1197	270	150	150
query27	2019	451	326	326
query28	5454	2507	2459	2459
query29	726	533	396	396
query30	228	175	152	152
query31	983	899	820	820
query32	89	63	61	61
query33	508	347	287	287
query34	746	837	511	511
query35	784	826	749	749
query36	1022	1063	958	958
query37	117	96	77	77
query38	4309	4118	4421	4118
query39	1514	1426	1484	1426
query40	199	116	100	100
query41	46	42	46	42
query42	116	109	107	107
query43	521	528	498	498
query44	1325	807	811	807
query45	179	175	173	173
query46	860	1041	646	646
query47	1907	1937	1876	1876
query48	374	401	337	337
query49	767	489	393	393
query50	636	631	397	397
query51	7037	7262	7242	7242
query52	101	102	96	96
query53	228	253	188	188
query54	479	479	401	401
query55	77	77	83	77
query56	251	270	250	250
query57	1226	1166	1127	1127
query58	237	231	233	231
query59	2976	3135	2972	2972
query60	280	262	245	245
query61	111	114	113	113
query62	874	816	740	740
query63	223	195	195	195
query64	4465	997	640	640
query65	3292	3179	3269	3179
query66	1051	418	316	316
query67	15874	15842	15586	15586
query68	9178	759	570	570
query69	455	281	252	252
query70	1202	1142	1131	1131
query71	425	272	255	255
query72	5793	3861	3821	3821
query73	658	743	389	389
query74	9561	9186	9241	9186
query75	4586	3157	2663	2663
query76	4891	1169	808	808
query77	986	363	270	270
query78	10028	10371	9384	9384
query79	4891	880	590	590
query80	705	524	440	440
query81	481	261	232	232
query82	408	144	128	128
query83	195	177	161	161
query84	292	90	74	74
query85	732	366	296	296
query86	353	337	312	312
query87	4638	4629	4312	4312
query88	3754	2272	2219	2219
query89	426	334	297	297
query90	2073	199	188	188
query91	131	138	106	106
query92	68	56	53	53
query93	3017	884	534	534
query94	655	402	290	290
query95	339	251	255	251
query96	492	604	281	281
query97	2770	2831	2715	2715
query98	231	203	205	203
query99	1599	1587	1470	1470
Total cold run time: 298220 ms
Total hot run time: 190480 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 932ce6a1c5fd30e681ddd84d0498a48366668260, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.08	0.07
query4	1.60	0.10	0.10
query5	0.44	0.42	0.40
query6	1.16	0.65	0.68
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.49	0.52
query10	0.58	0.55	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.84	2.73	2.79
query15	0.90	0.82	0.82
query16	0.38	0.38	0.37
query17	1.06	1.08	1.05
query18	0.24	0.22	0.20
query19	1.94	1.86	2.06
query20	0.02	0.00	0.02
query21	15.36	0.92	0.59
query22	0.75	0.76	0.80
query23	15.24	1.41	0.61
query24	3.00	1.39	1.25
query25	0.16	0.31	0.06
query26	0.21	0.16	0.14
query27	0.06	0.04	0.05
query28	13.60	1.51	1.05
query29	12.56	4.00	3.26
query30	0.25	0.09	0.06
query31	2.83	0.62	0.38
query32	3.22	0.55	0.47
query33	3.11	3.14	3.03
query34	16.70	5.15	4.54
query35	4.58	4.47	4.56
query36	0.67	0.49	0.49
query37	0.09	0.06	0.05
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.8 s
Total hot run time: 31.71 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 1, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 1, 2025

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 1, 2025

PR approved by anyone and no changes requested.

CalvinKirs
CalvinKirs approved these changes Jan 2, 2025
View reviewed changes
Copy link
Member

@CalvinKirs CalvinKirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you provide the latest be.out startup logs? The issue might affect fe/be-java-extensions, causing BE to fail to start. The rollback for #44310 was due to BE failing to start in the performance pipeline.

CalvinKirs
CalvinKirs requested changes Jan 2, 2025
View reviewed changes
Copy link
Member

@CalvinKirs CalvinKirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned above.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 2, 2025
@CalvinKirs
Copy link
Member

[ThreadedStreamConsumer] [INFO] Running org.apache.doris.load.loadv2.etl.SparkEtlJobTest java.lang.NoSuchMethodError: org.apache.logging.slf4j.Log4jLoggerFactory: method 'void <init>()' not found at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:53) at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:41) at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150) at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124) at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273) at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:186) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467)

after the fix, fe can print aws sdk debug log, but fe ut failed with new problem above

This is because this case is located in the spark-dpp module, which does not include any logging implementation. Adding the appropriate slf4j-log4j-impl dependency will resolve the issue.

@SWJTU-ZhangLei
Copy link
Contributor Author

SWJTU-ZhangLei commented Jan 3, 2025 •
edited
Loading

[ThreadedStreamConsumer] [INFO] Running org.apache.doris.load.loadv2.etl.SparkEtlJobTest java.lang.NoSuchMethodError: org.apache.logging.slf4j.Log4jLoggerFactory: method 'void <init>()' not found at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:53) at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:41) at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150) at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124) at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273) at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:186) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467)
after the fix, fe can print aws sdk debug log, but fe ut failed with new problem above

This is because this case is located in the spark-dpp module, which does not include any logging implementation. Adding the appropriate slf4j-log4j-impl dependency will resolve the issue.

thanks for you comments, i will have a try

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs requested changes

@gavinchou gavinchou gavinchou approved these changes

@morningman morningman Awaiting requested review from morningman morningman is a code owner

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] fe log SLF4J: Class path contains multiple SLF4J bindings
5 participants
@SWJTU-ZhangLei @Thearas @doris-robot @CalvinKirs @gavinchou