Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

【2.3.5 我主要将mysql的字符串类型转换到日期类型，这样写到SR做对应里，发现普通的转换都报错，请问什么原因？ 还有官网能否增加转换的例子】org.apache.seatunnel.transform.exception.TransformException: ErrorCode:[COMMON-05], ErrorDescription:[Unsupported operation] - SQL validate failed: SELECT id, order_code, status, order_total_price, order_pay_mode, old_order_code, actual_money, operate_flag, order_commodity_weight, cabinet_name, cabinet_code FROM market_new2.t_electronic_back_order , cause: Unsupported schema syntax #8428

Open
3 tasks done
andyuan1000 opened this issue Jan 2, 2025 · 1 comment
Open
3 tasks done

【2.3.5 我主要将mysql的字符串类型转换到日期类型，这样写到SR做对应里，发现普通的转换都报错，请问什么原因？ 还有官网能否增加转换的例子】org.apache.seatunnel.transform.exception.TransformException: ErrorCode:[COMMON-05], ErrorDescription:[Unsupported operation] - SQL validate failed: SELECT id, order_code, status, order_total_price, order_pay_mode, old_order_code, actual_money, operate_flag, order_commodity_weight, cabinet_name, cabinet_code FROM market_new2.t_electronic_back_order , cause: Unsupported schema syntax #8428

andyuan1000 opened this issue Jan 2, 2025 · 1 comment
Labels
bug

Comments

@andyuan1000
Copy link

andyuan1000 commented Jan 2, 2025 •
edited
Loading

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

我主要将mysql的字符串类型转换到日期类型，这样写到SR做对应里，发现后台一直报：
2025-01-02 18:51:35,129 WARN [i.d.c.m.MySqlValueConverters ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:51:35,129 WARN [i.d.c.m.MySqlValueConverters ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:51:35,129 WARN [i.d.c.m.MySqlValueConverters ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]？
还有官网能否增加转换的例子

SeaTunnel Version

2.3.5

SeaTunnel Config

#定义一些作业的运行参数,具体可以参考 https://seatunnel.apache.org/docs/2.3.1/concept/JobEnvConfig
env {
	parallelism=1 # 并行度，这里是启动5个Source Task来并行的读取数据
    job.mode="STREAMING"  #作业的运行模式，BATCH=离线批同步，STREAMING=实时同步
    job.name="SeaTunnel_market_new_121_163_sr_STREAMING_Job"
    checkpoint.interval=20000 #每10000ms进行一次checkpoint，后面会详细介绍checkpoint对JDBC Source和StarRocks Sink这两个连接器的影响
}
source {
	 MySQL-CDC {
		result_table_name="Table250102"
	 	base-url = "jdbc:mysql://121.46.249.147:33306/market_new2?useUnicode=true&characterEncoding=utf-8&autoReconnect=true&zeroDateTimeBehavior=convertToNull&useSSL=false"
		username = "root"
		password = "Psjt@2020"
		table-names = ["market_new2.t_electronic_back_order"]
		startup.mode="earliest"
		table-names-config = [
		  {
			table = "market_new2.t_electronic_back_order"
			primaryKeys = ["id"]
		  }
		]
	}
}

transform {
  Sql {
    source_table_name = "Table250102"
    result_table_name = "Table25010201"
    query = "SELECT id,PARSEDATETIME(create_date,'yyyy-MM-dd HH:mm:ss'),PARSEDATETIME(record_time,'yyyy-MM-dd') order_code, status, order_total_price, order_pay_mode, old_order_code, actual_money, operate_flag, order_commodity_weight, cabinet_name, cabinet_code FROM Table250102 "
  }
}

sink {
	StarRocks {
        batch_max_rows=10240 # 
        source_table_name="Table25010201"
        table="t_electronic_back_order_st"
        database="market_new_dw"
        base-url="jdbc:mysql://121.46.249.147:9030/"
		username="root"
        password="Hdic@2024"
        nodeUrls=[ "121.46.249.147:8030"]
		enable_upsert_delete = true
    }
}

Running Command

./bin/seatunnel.sh --config ./config/mysql_to_sr_stream.config -m local

Error Exception

, file=mysql-bin.000288, pos=441677047, row=1, server_id=1, event=2}} ConnectRecord{topic='mysql_binlog_source.market_new2.t_electronic_back_order', kafkaPartition=null, key=Struct{id=34336595}, keySchema=Schema{mysql_binlog_source.market_new2.t_electronic_back_order.Key:STRUCT}, value=Struct{after=Struct{id=34336595,order_code=0001010066202409250001,user_name=1001,status=1,scale_code=0066,order_total_price=8.5,cabinet_id=37,order_status=0,old_order_code=,mo_zero=0.0,actual_money=8.5,operate_flag=0,order_commodity_weight=1.0,cabinet_name=江南山水菜柜,cabinet_code=001},source=Struct{version=1.6.4.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1727212510000,db=market_new2,table=t_electronic_back_order,server_id=1,file=mysql-bin.000288,pos=441677229,row=0},op=c,ts_ms=1735813735962}, valueSchema=Schema{mysql_binlog_source.market_new2.t_electronic_back_order.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)})
2025-01-02 18:28:56,445 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:56,445 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:28:56,445 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:56,446 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:56,446 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:28:56,446 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:56,446 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:56,446 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:28:56,446 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:56,695 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:56,695 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:28:56,695 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:56,969 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:56,969 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:28:56,969 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:57,419 INFO  [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-18] - log event: MessageDelayedEvent(createdTime=1735813737419, jobId=927145976831934465, eventType=READER_MESSAGE_DELAYED, delayTime=8600735419, record=SourceRecord{sourcePartition={server=mysql_binlog_source}, sourceOffset={transaction_id=null, ts_sec=1727213002, file=mysql-bin.000288, pos=446146294, row=1, server_id=1, event=2}} ConnectRecord{topic='mysql_binlog_source.market_new2.t_electronic_back_order', kafkaPartition=null, key=Struct{id=34336600}, keySchema=Schema{mysql_binlog_source.market_new2.t_electronic_back_order.Key:STRUCT}, value=Struct{after=Struct{id=34336600,order_code=0143010155202409250001,user_name=1001,status=1,scale_code=0155,order_total_price=3.7,cabinet_id=263,order_status=0,old_order_code=,mo_zero=0.0,actual_money=3.7,operate_flag=0,order_commodity_weight=1.0,cabinet_name=航城菜柜,cabinet_code=143},source=Struct{version=1.6.4.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1727213002000,db=market_new2,table=t_electronic_back_order,server_id=1,file=mysql-bin.000288,pos=446146476,row=0},op=c,ts_ms=1735813736969}, valueSchema=Schema{mysql_binlog_source.market_new2.t_electronic_back_order.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)})
2025-01-02 18:28:58,890 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:58,890 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:28:58,890 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:58,892 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:58,892 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:28:58,892 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:59,419 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:59,419 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:28:59,419 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:59,422 INFO  [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-17] - log event: MessageDelayedEvent(createdTime=1735813739421, jobId=927145976831934465, eventType=READER_MESSAGE_DELAYED, delayTime=8599948421, record=SourceRecord{sourcePartition={server=mysql_binlog_source}, sourceOffset={transaction_id=null, ts_sec=1727213791, file=mysql-bin.000288, pos=453616452, row=1, server_id=1, event=2}} ConnectRecord{topic='mysql_binlog_source.market_new2.t_electronic_back_order', kafkaPartition=null, key=Struct{id=34336601}, keySchema=Schema{mysql_binlog_source.market_new2.t_electronic_back_order.Key:STRUCT}, value=Struct{after=Struct{id=34336601,order_code=3244-001-1240925000889,user_name=1325,status=1,scale_code=pscg148,order_total_price=7.57,cabinet_id=275,order_status=0,old_order_code=,mo_zero=0.0,actual_money=7.57,operate_flag=0,order_commodity_weight=0.76,cabinet_name=浦商生鲜紫薇店,cabinet_code=148},source=Struct{version=1.6.4.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1727213791000,db=market_new2,table=t_electronic_back_order,server_id=1,file=mysql-bin.000288,pos=453616634,row=0},op=c,ts_ms=1735813738890}, valueSchema=Schema{mysql_binlog_source.market_new2.t_electronic_back_order.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)})
2025-01-02 18:28:59,424 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:59,424 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:28:59,424 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:59,426 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:28:59,426 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:28:59,426 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:29:00,143 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:29:00,143 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:29:00,143 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:29:00,153 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:29:00,153 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:29:00,153 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:29:00,879 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 93 and column create_date DATETIME DEFAULT VALUE NULL: class=byte[]
2025-01-02 18:29:00,879 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected JDBC DATE value for field record_time with schema Schema{io.debezium.time.Date:INT32}: class=class [B, value=[50, 48, 50, 52, 45, 48, 57, 45, 50, 53]
2025-01-02 18:29:00,879 WARN  [i.d.c.m.MySqlValueConverters  ] [blc-121.46.249.147:33306] - Unexpected value for JDBC type 91 and column record_time DATE DEFAULT VALUE NULL: class=byte[]
^C2025-01-02 18:29:01,047 INFO  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel-826460] [5.1] Running shutdown hook... Current state: ACTIVE
2025-01-02 18:29:01,047 INFO  [c.h.c.LifecycleService        ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel-826460] [5.1] [localhost]:5801 is SHUTTING_DOWN

Zeta or Flink or Spark Version

无

Java or Scala Version

11

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@andyuan1000 andyuan1000 added the bug label Jan 2, 2025
@andyuan1000 andyuan1000 reopened this Jan 2, 2025
@andyuan1000
Copy link
Author

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@andyuan1000