虎牙基于 Apache Iceberg + Amoro 在实时场景下的实践 #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
||
| # 摘要 | ||
|
|
||
| 本文主要介绍了虎牙基于 Apache Iceberg 在实时数据湖构建方面的实践经验。 |
|
|
||
| 本文主要介绍了虎牙基于 Apache Iceberg 在实时数据湖构建方面的实践经验。 | ||
|
|
||
| [[一段话介绍虎牙情况]] |
There was a problem hiding this comment.
现状需要突出用 Iceberg 遇到问题即抛出使用 Amoro 的动机？
| 这样 Trino 引擎就可以根据需要选择访问 Hadoop 或 Alluxio 集群访问数据文件了。 | ||
|
|
||
|
|
||
| 4. **引入 Amoro 对 Iceberg 表进行持续优化** |
There was a problem hiding this comment.
正文过三分之一，终于见着 Amoro 啦 -_-（建议有的放矢）
| 通过以上3种方式对 Iceberg 表查询的进一步优化，几乎得到了和 OLAP 引擎一样的查询体验。 | ||
|
|
||
|
|
||
| # 近实时数据处理场景 |
There was a problem hiding this comment.
建议标明场景一、二，衔接有点脱，或者就某一场景完整叙述，或者在摘要里点一下。
| 当然 Iceberg 社区也有一个 project 在跟进这块 ，有兴趣也可以关注一下： <https://github.com/apache/iceberg/projects/27> | ||
|
|
||
|
|
||
| **2. 基于 Amoro 管理并监控 Iceberg 表** |
There was a problem hiding this comment.
-
场景二内容也是正文过半，才有了 Amoro；
-
整篇偏向于从虎牙角度讲述自己的完整实践过程，Amoro 为辅；作为社区用户案例最好突出
为什么用 Amoro。
|
|
||
| # 未来规划 | ||
|
|
||
| 目前虎牙和 Amoro 社区保持着紧密联系，积极参与到 Amoro 社区的多个核心功能开发中。目前正在推进 Amoro 监控告警功能的开发中，会将目前公司内的实践贡献给社区。另外在参与 Amoro 对 Iceberg 表 Tag & Branch 相关功能的推进。 |
| 通过对 `flink RowData` 做一层包装, 在数据中加入自定义的分区号： | ||
|
|
||
| ```java | ||
| public class RowDataWrap implements RowData { |
There was a problem hiding this comment.
感觉这个代码没啥必要（不是关键代码），哈哈哈，简单说一下思路就好
|
|
||
| **2. 基于 Watermark 的 Iceberg 近实时调度** | ||
|
|
||
| 上述问题的核心还是实效性，虎牙通过将 ODS/DWD 层从 Hive 表替换为 Iceberg 表以解决上述问题。在替换为 Iceberg 表以后，虎牙将任务调度的周期从1小时一次改为了10分钟一次。 |
There was a problem hiding this comment.
10分钟一次也是因为我们用watermark来进行判定了，感觉watermark可以提到前面来讲
|
|
||
| 在 Iceberg 原生提供的能力中，快照的过期需要通过 Spark 任务触发。而 Amoro 集成了快照管理的能力，通过对表上设置 properties 即可自动化的过期历史快照。 | ||
|
|
||
| 2. Iceberg 表孤儿文件治理 |
There was a problem hiding this comment.
这里其实iceberg自己也有孤儿文件处理的action（spark）, 不过之前我们用离线的方式在每次进行合并之后进行调用都会扫描一次，这个成本有点高，用amoro其实是降低了成本的
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.