Skip to content

Add Gloo communication to turobmind #3362

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Add Gloo communication to turobmind #3362

wants to merge 10 commits into from
+1,172 −54

Conversation

irexyc
Copy link
Collaborator

@irexyc irexyc commented Mar 28, 2025 •
edited
Loading

Usage

A two node example with tp=4, dp=4 and device_num=4

proxy_server
lmdeploy serve proxy --server-name 10.140.24.141

node 0

LMDEPLOY_DP_MASTER_ADDR=10.140.24.141 \
LMDEPLOY_DP_MASTER_PORT=6900 \
lmdeploy serve api_server \
    Qwen/Qwen2.5-7B-Instruct \
    --server-port 29200 \
    --tp 4 \
    --dp 4 \
    --nnodes 2 \
    --node-rank 0 \
    --ngpus-per-node 2 \
    --proxy-url http://10.140.24.141:8000

node 1

LMDEPLOY_DP_MASTER_ADDR=10.140.24.141 \
LMDEPLOY_DP_MASTER_PORT=6900 \
lmdeploy serve api_server \
    Qwen/Qwen2.5-7B-Instruct \
    --server-port 29202 \
    --tp 4 \
    --dp 4 \
    --nnodes 2 \
    --node-rank 1 \
    --ngpus-per-node 2 \
    --proxy-url http://10.140.24.141:8000

@irexyc irexyc added the WIP label Mar 28, 2025
@irexyc irexyc changed the title [WIP] Add Gloo communication to turobmind Add Gloo communication to turobmind Apr 10, 2025
@irexyc irexyc removed the WIP label Apr 10, 2025
@irexyc
Copy link
Collaborator Author

irexyc commented Apr 10, 2025

oc evaluate diff.csv

@lvhan028 lvhan028 added the enhancement New feature or request label Apr 10, 2025
@lvhan028 lvhan028 requested a review from lzhangzz April 24, 2025 03:45
lvhan028
lvhan028 reviewed May 12, 2025
View reviewed changes
src/turbomind/comm/gloo/CMakeLists.txt
set_property(TARGET gloo_comm PROPERTY POSITION_INDEPENDENT_CODE ON)
target_link_libraries(gloo_comm PUBLIC gloo logger)

if(USE_REDIS)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this block be removed?

lvhan028
lvhan028 reviewed May 12, 2025
View reviewed changes
src/turbomind/comm/gloo/gloo_comm.cc
#include <gloo/math.h>
#include <gloo/rendezvous/context.h>
#include <gloo/rendezvous/prefix_store.h>
#if GLOO_USE_REDIS
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove this

lvhan028
lvhan028 reviewed May 12, 2025
View reviewed changes
lmdeploy/turbomind/turbomind.py
@@ -164,9 +191,8 @@ def _create_weight(self, model_comm):
"""Allocate weight buffer, load params if from_workspace."""

# TODO: support mpi
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove "#TODO: support mpi", can't we?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lvhan028 lvhan028 lvhan028 left review comments

@lzhangzz lzhangzz Awaiting requested review from lzhangzz

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@irexyc @lvhan028