Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

怎么在hadoop上跑paddle预估产出embedding #70315

Open
pipipiapia opened this issue Dec 18, 2024 · 2 comments
Open

怎么在hadoop上跑paddle预估产出embedding #70315

pipipiapia opened this issue Dec 18, 2024 · 2 comments
Assignees
@From00
Labels
status/new-issue 新建 type/question 用户提问

Comments

@pipipiapia
Copy link

pipipiapia commented Dec 18, 2024 •
edited
Loading

请提出你的问题 Please ask your question

我用paddlerec训练了dssm动态模型，在paddlecloud gpu上进行用户端的embedding产出，运行非常慢，想改到hadoop上infer，打包的python环境在hadoop上会报动态错误，我的问题是：
1.是不是需要改静态模型训练，静态预估，才能上hadoop；
2.有没有可以用的python包上传hadoop，实在是搞不懂paddle的环境
@danleifeng
Copy link
Contributor

上hadoop是什么意思，hadoop不是指存储吗

@pipipiapia
Copy link
Author

pipipiapia commented Dec 19, 2024 •
edited
Loading

上hadoop是什么意思，hadoop不是指存储吗

hadoop可以做分布式计算，双塔的用户塔数据量很大，gpu跑不起来，想用300多的hadoop cpu集群进行预估；之前pslib等有做，不知道原生态的paddle咋搞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@From00 From00

Labels
status/new-issue 新建 type/question 用户提问
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pipipiapia @From00 @danleifeng