03-大模型部署之基于Kubernetes+vLLM+Ray多机多卡集群部署与容器调度方案
·
03-大模型部署之基于Kubernetes+vLLM+Ray多机多卡集群部署与容器调度方案
1. 架构概述
1.1 为什么需要Ray集成
Ray是一个分布式计算框架,当与vLLM结合时,可以提供以下优势:
- 智能资源调度:Ray能够智能分配和调度GPU资源
- 弹性伸缩:根据工作负载动态调整集群规模
- 容错机制:自动处理节点故障和任务重试
- 多任务并行:支持同时运行多个推理任务
- 资源隔离:为不同用户提供资源隔离
1.2 整体架构设计
┌─────────────────────────────────────────────────────────────────────────────┐
│ Kubernetes集群 │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Ray集群 │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Ray Head │ │Ray Worker 1 │ │Ray Worker 2 │ │Ray Worker N │ │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ │ vLLM Driver │ │ vLLM Worker │ │ vLLM Worker │ │ vLLM Worker │ │ │
│ │ │ │ │ (GPU 0,1) │ │ (GPU 2,3) │ │ (GPU N,N+1) │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 服务层 │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │API Gateway │ │ 负载均衡器 │ │ 监控系统 │ │ 日志收集 │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
2. 环境准备与依赖
2.1 硬件要求
最小配置
- Kubernetes Master节点:4 CPU,8GB内存
- Ray Head节点:8 CPU,32GB内存,1张GPU
- Ray Worker节点:16 CPU,64GB内存,2-4张GPU
- 网络:节点间万兆网络
推荐配置
- 存储:高速SSD存储用于模型缓存
- 内存:每个节点至少128GB内存
- GPU:NVIDIA A100或V100,每节点4-8张
- 网络:InfiniBand或高速以太网
2.2 软件依赖
# Kubernetes组件
kubectl version >= 1.24
helm version >= 3.8
# NVIDIA组件
nvidia-driver >= 470
cuda-toolkit >= 11.7
nvidia-docker2
# Ray组件
ray[default] >= 2.0
ray-dashboard
# vLLM组件
vllm >= 0.2.5
torch >= 2.0.0
3. Ray集群部署
3.1 Ray Head节点配置
# ray-head-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ray-head
namespace: ray-vllm
spec:
replicas: 1
selector:
matchLabels:
app: ray-head
component: head
template:
metadata:
labels:
app: ray-head
component: head
spec:
containers:
- name: ray-head
image: rayproject/ray:2.8.0-py38-gpu
ports:
- containerPort: 6379
name: gcs-server
- containerPort: 8265
name: dashboard
- containerPort: 10001
name: client
command: ["/bin/bash", "-c", "--"]
args:
- |
ray start --head --port=6379 --dashboard-host=0.0.0.0 --dashboard-port=8265 \
--num-cpus=8 --num-gpus=1 --block
env:
- name: RAY_BACKEND_LOG_LEVEL
value: "debug"
resources:
requests:
cpu: 4
memory: "16Gi"
nvidia.com/gpu: 1
limits:
cpu: 8
memory: "32Gi"
nvidia.com/gpu: 1
volumeMounts:
- name: ray-logs
mountPath: /tmp/ray
livenessProbe:
exec:
command:
- ray
- status
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
exec:
command:
- ray
- status
initialDelaySeconds: 10
periodSeconds: 5
volumes:
- name: ray-logs
emptyDir: {}
3.2 Ray Worker节点配置
# ray-worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ray-worker
namespace: ray-vllm
spec:
replicas: 3
selector:
matchLabels:
app: ray-worker
component: worker
template:
metadata:
labels:
app: ray-worker
component: worker
spec:
containers:
- name: ray-worker
image: rayproject/ray:2.8.0-py38-gpu
command: ["/bin/bash", "-c", "--"]
args:
- |
ray start --address=ray-head-svc:6379 \
--num-cpus=16 --num-gpus=2 --block
env:
- name: RAY_BACKEND_LOG_LEVEL
value: "info"
resources:
requests:
cpu: 8
memory: "32Gi"
nvidia.com/gpu: 2
limits:
cpu: 16
memory: "64Gi"
nvidia.com/gpu: 2
volumeMounts:
- name: ray-logs
mountPath: /tmp/ray
- name: model-cache
mountPath: /models
volumes:
- name: ray-logs
emptyDir: {}
- name: model-cache
persistentVolumeClaim:
claimName: model-cache-pvc
3.3 Ray服务配置
# ray-head-service.yaml
apiVersion: v1
kind: Service
metadata:
name: ray-head-svc
namespace: ray-vllm
spec:
selector:
app: ray-head
component: head
ports:
- name: gcs-server
port: 6379
targetPort: 6379
- name: dashboard
port: 8265
targetPort: 8265
- name: client
port: 10001
targetPort: 10001
type: ClusterIP
3.4 Ray集群自动扩缩容
# ray-cluster-autoscaler.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: ray-cluster-config
namespace: ray-vllm
data:
cluster.yaml: |
cluster_name: vllm-cluster
max_workers: 20
upscaling_speed: 2.0
idle_timeout_minutes: 10
provider:
type: kubernetes
namespace: ray-vllm
use_internal_ips: true
head_node:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ray-head
spec:
replicas: 1
template:
spec:
containers:
- name: ray-head
image: rayproject/ray:2.8.0-py38-gpu
resources:
requests:
cpu: 4
memory: "16Gi"
nvidia.com/gpu: 1
limits:
cpu: 8
memory: "32Gi"
nvidia.com/gpu: 1
worker_nodes:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ray-worker
spec:
replicas: 3
template:
spec:
containers:
- name: ray-worker
image: rayproject/ray:2.8.0-py38-gpu
resources:
requests:
cpu: 8
memory: "32Gi"
nvidia.com/gpu: 2
limits:
cpu: 16
memory: "64Gi"
nvidia.com/gpu: 2
4. vLLM与Ray集成
4.1 vLLM Ray部署器
# vllm_ray_deployer.py
import ray
from ray import serve
from vllm import LLM, SamplingParams
@serve.deployment(
name="vllm-deployment",
num_replicas=3,
ray_actor_options={"num_gpus": 1}
)
class VLLMDeployment:
def __init__(self, model_name: str):
self.llm = LLM(
model=model_name,
tensor_parallel_size=1,
gpu_memory_utilization=0.9
)
self.sampling_params = SamplingParams(
temperature=0.8,
top_p=0.95,
max_tokens=1000
)
async def generate(self, prompt: str):
outputs = self.llm.generate([prompt], self.sampling_params)
return {"generated_text": outputs[0].outputs[0].text}
async def generate_batch(self, prompts: list):
outputs = self.llm.generate(prompts, self.sampling_params)
return [
{"prompt": output.prompt, "generated_text": output.outputs[0].text}
for output in outputs
]
# 启动服务
def start_vllm_service(model_name: str = "meta-llama/Llama-2-7b-chat-hf"):
ray.init(address="auto") # 连接到Ray集群
serve.start(http_options={"host": "0.0.0.0", "port": 8000})
# 部署vLLM服务
VLLMDeployment.options(name=model_name.split("/")[-1]).deploy(model_name)
print(f"vLLM服务已启动,模型: {model_name}")
if __name__ == "__main__":
start_vllm_service()
4.2 Ray Serve配置文件
# ray-serve-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: ray-serve-config
namespace: ray-vllm
data:
config.yaml: |
serve:
deployments:
- name: vllm-llama2-7b
import_path: vllm_ray_deployer:VLLMDeployment
init_args:
model_name: "meta-llama/Llama-2-7b-chat-hf"
autoscaling_config:
min_replicas: 1
max_replicas: 10
target_num_ongoing_requests_per_replica: 2
ray_actor_options:
num_gpus: 1
num_cpus: 4
- name: vllm-llama2-13b
import_path: vllm_ray_deployer:VLLMDeployment
init_args:
model_name: "meta-llama/Llama-2-13b-chat-hf"
autoscaling_config:
min_replicas: 1
max_replicas: 5
target_num_ongoing_requests_per_replica: 1
ray_actor_options:
num_gpus: 2
num_cpus: 8
4.3 多模型管理
# multi_model_manager.py
import ray
from ray import serve
from typing import Dict, List
import asyncio
class MultiModelManager:
def __init__(self):
self.models = {}
self.model_configs = {
"llama2-7b": {
"model_name": "meta-llama/Llama-2-7b-chat-hf",
"tensor_parallel_size": 1,
"max_tokens": 1000
},
"llama2-13b": {
"model_name": "meta-llama/Llama-2-13b-chat-hf",
"tensor_parallel_size": 2,
"max_tokens": 1500
},
"llama2-70b": {
"model_name": "meta-llama/Llama-2-70b-chat-hf",
"tensor_parallel_size": 4,
"max_tokens": 2000
}
}
async def deploy_model(self, model_id: str):
if model_id not in self.model_configs:
raise ValueError(f"未知模型ID: {model_id}")
config = self.model_configs[model_id]
@serve.deployment(
name=f"vllm-{model_id}",
num_replicas=1,
ray_actor_options={"num_gpus": config["tensor_parallel_size"]}
)
class ModelDeployment:
def __init__(self):
from vllm import LLM, SamplingParams
self.llm = LLM(
model=config["model_name"],
tensor_parallel_size=config["tensor_parallel_size"]
)
self.sampling_params = SamplingParams(
max_tokens=config["max_tokens"]
)
async def generate(self, prompt: str):
outputs = self.llm.generate([prompt], self.sampling_params)
return outputs[0].outputs[0].text
# 部署模型
ModelDeployment.deploy()
self.models[model_id] = ModelDeployment.get_handle()
return f"模型 {model_id} 部署成功"
async def generate(self, model_id: str, prompt: str):
if model_id not in self.models:
await self.deploy_model(model_id)
handle = self.models[model_id]
return await handle.generate.remote(prompt)
async def generate_batch(self, model_id: str, prompts: List[str]):
if model_id not in self.models:
await self.deploy_model(model_id)
handle = self.models[model_id]
tasks = [handle.generate.remote(prompt) for prompt in prompts]
return await asyncio.gather(*tasks)
# 启动多模型管理器
def start_multi_model_service():
ray.init(address="auto")
serve.start(http_options={"host": "0.0.0.0", "port": 8000})
manager = MultiModelManager()
@serve.deployment(name="model-manager")
class ManagerDeployment:
def __init__(self):
self.manager = manager
async def __call__(self, request):
import json
data = await request.json()
if data["action"] == "generate":
result = await self.manager.generate(
data["model_id"],
data["prompt"]
)
return {"result": result}
elif data["action"] == "generate_batch":
results = await self.manager.generate_batch(
data["model_id"],
data["prompts"]
)
return {"results": results}
elif data["action"] == "deploy":
result = await self.manager.deploy_model(data["model_id"])
return {"message": result}
ManagerDeployment.deploy()
print("多模型管理服务已启动")
if __name__ == "__main__":
start_multi_model_service()
5. 资源调度与负载均衡
5.1 自定义调度器
# custom_scheduler.py
import ray
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy
from ray.util.placement_group import placement_group
class VLLMScheduler:
def __init__(self):
self.placement_groups = {}
self.model_requirements = {
"llama2-7b": {"num_gpus": 1, "num_cpus": 4},
"llama2-13b": {"num_gpus": 2, "num_cpus": 8},
"llama2-70b": {"num_gpus": 4, "num_cpus": 16}
}
def create_placement_group(self, model_id: str, num_replicas: int = 1):
"""为模型创建放置组"""
if model_id not in self.model_requirements:
raise ValueError(f"未知模型: {model_id}")
reqs = self.model_requirements[model_id]
# 创建捆绑包
bundles = [
{"GPU": reqs["num_gpus"], "CPU": reqs["num_cpus"]}
for _ in range(num_replicas)
]
# 创建放置组
pg = placement_group(bundles, strategy="STRICT_SPREAD")
ray.get(pg.ready())
self.placement_groups[model_id] = pg
return pg
def get_scheduling_strategy(self, model_id: str):
"""获取调度策略"""
if model_id not in self.placement_groups:
self.create_placement_group(model_id)
pg = self.placement_groups[model_id]
return PlacementGroupSchedulingStrategy(
placement_group=pg,
placement_group_capture_child_tasks=True
)
def release_placement_group(self, model_id: str):
"""释放放置组"""
if model_id in self.placement_groups:
ray.util.remove_placement_group(self.placement_groups[model_id])
del self.placement_groups[model_id]
# 使用自定义调度器
scheduler = VLLMScheduler()
@ray.remote
def deploy_vllm_with_scheduler(model_id: str):
"""使用自定义调度器部署vLLM"""
from vllm import LLM
strategy = scheduler.get_scheduling_strategy(model_id)
with strategy:
llm = LLM(model=f"meta-llama/{model_id}-chat-hf")
return llm
5.2 负载均衡策略
# load_balancer.py
import ray
from typing import List, Dict
import time
import random
class VLLMLoadBalancer:
def __init__(self):
self.model_instances = {} # model_id -> list of instance handles
self.instance_loads = {} # instance_handle -> current_load
self.last_update = {} # instance_handle -> last_update_time
def register_instance(self, model_id: str, instance_handle):
"""注册vLLM实例"""
if model_id not in self.model_instances:
self.model_instances[model_id] = []
self.model_instances[model_id].append(instance_handle)
self.instance_loads[instance_handle] = 0
self.last_update[instance_handle] = time.time()
def select_instance(self, model_id: str, strategy: str = "least_loaded"):
"""选择实例处理请求"""
if model_id not in self.model_instances:
raise ValueError(f"模型 {model_id} 没有可用实例")
instances = self.model_instances[model_id]
if strategy == "least_loaded":
# 选择负载最少的实例
return min(instances, key=lambda x: self.instance_loads[x])
elif strategy == "round_robin":
# 轮询选择
return instances[random.randint(0, len(instances)-1)]
elif strategy == "random":
# 随机选择
return random.choice(instances)
else:
raise ValueError(f"未知策略: {strategy}")
async def generate(self, model_id: str, prompt: str, strategy: str = "least_loaded"):
"""使用负载均衡生成文本"""
instance = self.select_instance(model_id, strategy)
# 增加负载计数
self.instance_loads[instance] += 1
try:
# 调用实例生成文本
result = await instance.generate.remote(prompt)
return result
finally:
# 减少负载计数
self.instance_loads[instance] -= 1
self.last_update[instance] = time.time()
def get_load_stats(self):
"""获取负载统计"""
stats = {}
for model_id, instances in self.model_instances.items():
stats[model_id] = {
"total_instances": len(instances),
"total_load": sum(self.instance_loads[i] for i in instances),
"avg_load": sum(self.instance_loads[i] for i in instances) / len(instances)
}
return stats
# 全局负载均衡器
load_balancer = VLLMLoadBalancer()
6. 监控与运维
6.1 Ray Dashboard集成
# ray-dashboard-service.yaml
apiVersion: v1
kind: Service
metadata:
name: ray-dashboard
namespace: ray-vllm
spec:
selector:
app: ray-head
component: head
ports:
- port: 8265
targetPort: 8265
type: LoadBalancer
6.2 自定义监控指标
# monitoring.py
import ray
import time
import psutil
from prometheus_client import Counter, Histogram, Gauge, start_http_server
# Prometheus指标
REQUEST_COUNT = Counter('vllm_requests_total', 'Total vLLM requests', ['model_id'])
REQUEST_LATENCY = Histogram('vllm_request_duration_seconds', 'vLLM request latency')
GPU_MEMORY_USAGE = Gauge('vllm_gpu_memory_usage_bytes', 'GPU memory usage', ['gpu_id'])
ACTIVE_INSTANCES = Gauge('vllm_active_instances', 'Number of active vLLM instances', ['model_id'])
class VLLMMonitor:
def __init__(self):
self.start_time = time.time()
start_http_server(8001) # Prometheus metrics端口
@ray.remote
def collect_metrics(self):
"""收集指标"""
import torch
# GPU内存使用情况
if torch.cuda.is_available():
for i in range(torch.cuda.device_count()):
memory_used = torch.cuda.memory_allocated(i)
GPU_MEMORY_USAGE.labels(gpu_id=str(i)).set(memory_used)
# 系统资源使用情况
cpu_percent = psutil.cpu_percent()
memory = psutil.virtual_memory()
return {
"timestamp": time.time(),
"cpu_percent": cpu_percent,
"memory_percent": memory.percent,
"memory_used_gb": memory.used / (1024**3)
}
def record_request(self, model_id: str, duration: float):
"""记录请求指标"""
REQUEST_COUNT.labels(model_id=model_id).inc()
REQUEST_LATENCY.observe(duration)
def update_active_instances(self, model_id: str, count: int):
"""更新活跃实例数"""
ACTIVE_INSTANCES.labels(model_id=model_id).set(count)
# 全局监控实例
monitor = VLLMMonitor()
6.3 告警配置
# prometheus-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: vllm-alerts
namespace: ray-vllm
spec:
groups:
- name: vllm.rules
rules:
- alert: VLLMHighLatency
expr: histogram_quantile(0.95, rate(vllm_request_duration_seconds_bucket[5m])) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "vLLM高延迟告警"
description: "vLLM请求延迟超过10秒"
- alert: VLLMGPUMemoryHigh
expr: vllm_gpu_memory_usage_bytes / (1024^3) > 30
for: 5m
labels:
severity: critical
annotations:
summary: "vLLM GPU内存使用率过高"
description: "GPU内存使用超过30GB"
- alert: VLLMNoActiveInstances
expr: vllm_active_instances == 0
for: 1m
labels:
severity: critical
annotations:
summary: "vLLM没有活跃实例"
description: "所有vLLM实例都已停止"
7. 故障处理与恢复
7.1 自动故障恢复
# fault_tolerance.py
import ray
import time
from typing import Dict, List
class VLLMFailoverManager:
def __init__(self):
self.health_check_interval = 30 # 30秒检查一次
self.max_retries = 3
self.retry_delay = 5
self.unhealthy_instances = set()
@ray.remote
def health_check(self, instance_handle):
"""检查实例健康状态"""
try:
# 发送健康检查请求
result = ray.get(instance_handle.health_check.remote(), timeout=10)
return result.get("status") == "healthy"
except Exception as e:
print(f"健康检查失败: {e}")
return False
async def monitor_instances(self):
"""监控所有实例"""
while True:
# 获取所有实例
all_instances = ray.state.actors()
for actor_id, actor_info in all_instances.items():
if "vllm" in actor_info["name"]:
instance_handle = ray.get_actor(actor_id.split("-")[-1])
# 执行健康检查
is_healthy = await self.health_check.remote(instance_handle)
if not is_healthy:
if actor_id not in self.unhealthy_instances:
print(f"实例 {actor_id} 不健康")
self.unhealthy_instances.add(actor_id)
await self.restart_instance(actor_id)
else:
if actor_id in self.unhealthy_instances:
print(f"实例 {actor_id} 恢复健康")
self.unhealthy_instances.remove(actor_id)
await asyncio.sleep(self.health_check_interval)
async def restart_instance(self, actor_id: str):
"""重启不健康的实例"""
try:
# 获取实例配置
actor_info = ray.state.actors()[actor_id]
model_name = actor_info.get("model_name", "default")
# 杀死旧实例
ray.kill(actor_id)
# 等待一段时间
await asyncio.sleep(self.retry_delay)
# 重启实例
new_actor = VLLMDeployment.options(name=actor_id).remote(model_name)
print(f"实例 {actor_id} 已重启")
except Exception as e:
print(f"重启实例 {actor_id} 失败: {e}")
# 启动故障恢复管理器
failover_manager = VLLMFailoverManager()
7.2 数据备份与恢复
# backup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: vllm-backup
namespace: ray-vllm
spec:
schedule: "0 2 * * *" # 每天凌晨2点执行
jobTemplate:
spec:
template:
spec:
containers:
- name: backup
image: your-registry/vllm-backup:latest
command:
- /bin/bash
- -c
- |
# 备份模型缓存
tar -czf /backup/model-cache-$(date +%Y%m%d).tar.gz /models
# 备份配置文件
kubectl get configmap -n ray-vllm -o yaml > /backup/config-$(date +%Y%m%d).yaml
# 清理旧备份(保留7天)
find /backup -name "*.tar.gz" -mtime +7 -delete
find /backup -name "*.yaml" -mtime +7 -delete
volumeMounts:
- name: model-cache
mountPath: /models
- name: backup-storage
mountPath: /backup
volumes:
- name: model-cache
persistentVolumeClaim:
claimName: model-cache-pvc
- name: backup-storage
persistentVolumeClaim:
claimName: backup-pvc
restartPolicy: OnFailure
8. 最佳实践
8.1 性能优化
- 资源分配:根据模型大小合理分配GPU资源
- 批处理:启用连续批处理提高吞吐量
- 缓存策略:使用PVC缓存模型文件
- 网络优化:使用高速网络减少通信延迟
8.2 安全考虑
- 网络隔离:使用NetworkPolicy限制Pod间通信
- 密钥管理:使用Kubernetes Secret管理API密钥
- 镜像安全:定期扫描镜像漏洞
- 访问控制:实施RBAC权限控制
8.3 运维建议
- 版本管理:使用Git管理配置文件
- 渐进式部署:使用金丝雀发布策略
- 监控告警:建立完善的监控告警体系
- 文档维护:保持运维文档的及时更新
通过以上方案,可以构建一个高可用、可扩展的基于Kubernetes+vLLM+Ray的多机多卡集群部署系统,满足生产环境中的大语言模型推理需求。
更多推荐
所有评论(0)