KVCache 卸载#
注意
如果您还不熟悉 KVCache L1 和 L2 的概念，请参考 AIBrix KVCache Offloading 文档：AIBrix KVCache 卸载框架。
注意
AIBrix KVCache 卸载框架可以作为一个独立的组件使用，无需安装整个 AIBrix 堆栈。
注意
从 v0.4.0 版本开始，支持 vLLM V0 和 V1 连接器。
注意
从 v0.5.0 版本开始，支持 vLLM 和 SGLang。
警告
目前，仅支持 FlashAttention 和 XFormers。
L1 缓存示例#
注意
我们使用与 AIBrix 卸载连接器集成的自定义版 vLLM 来演示其用法。
在部署推理引擎之前，请使用 kubectl get pods -n aibrix-system 和 kubectl get pods -n envoy-gateway-system 确保 envoy-gateway 和 aibrix-gateway 正在运行。其他组件是可选的。
$ kubectl get pods -n aibrix-system
NAME READY STATUS RESTARTS AGE
aibrix-controller-manager-586dd9f868-465dz 1/1 Running 0 16h
aibrix-gateway-plugins-5fcbcbfc84-h7qc8 1/1 Running 0 16h
aibrix-gpu-optimizer-66f49fd947-gfncm 1/1 Running 0 4d23h
aibrix-kuberay-operator-55f4d4d666-bd7hj 1/1 Running 0 4d23h
aibrix-metadata-service-6d5cc8ddd6-444mb 1/1 Running 0 4d23h
aibrix-redis-master-c9b4967c5-pdnkg 1/1 Running 0 16h
$ kubectl get pods -n envoy-gateway-system
NAME READY STATUS RESTARTS AGE
envoy-aibrix-system-aibrix-eg-903790dc-fd69b467d-6fg2z 2/2 Running 0 16h
envoy-gateway-5d48549b5c-6r4cd 1/1 Running 0 16h
现在我们使用以下 yaml 文件创建一个引擎部署
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: deepseek-r1-distill-llama-8b
5 labels:
6 model.aibrix.ai/name: deepseek-r1-distill-llama-8b
7 model.aibrix.ai/port: "8000"
8spec:
9 replicas: 1
10 strategy:
11 rollingUpdate:
12 maxSurge: 1
13 maxUnavailable: 1
14 type: RollingUpdate
15 selector:
16 matchLabels:
17 model.aibrix.ai/name: deepseek-r1-distill-llama-8b
18 template:
19 metadata:
20 labels:
21 model.aibrix.ai/name: deepseek-r1-distill-llama-8b
22 annotations:
23 prometheus.io/path: "/metrics"
24 prometheus.io/port: "8000"
25 prometheus.io/scrape: "true"
26 spec:
27 initContainers:
28 - command:
29 - aibrix_download
30 - --model-uri
31 - tos://aibrix-artifact-testing/models/DeepSeek-R1-Distill-Llama-8B/
32 - --local-dir
33 - /models/
34 env:
35 - name: DOWNLOADER_NUM_CONNECTIONS
36 value: "16"
37 - name: DOWNLOADER_NUM_THREADS
38 value: "16"
39 - name: DOWNLOADER_ALLOW_FILE_SUFFIX
40 value: json, safetensors
41 - name: TOS_ACCESS_KEY
42 valueFrom:
43 secretKeyRef:
44 key: TOS_ACCESS_KEY
45 name: tos-credential
46 - name: TOS_SECRET_KEY
47 valueFrom:
48 secretKeyRef:
49 key: TOS_SECRET_KEY
50 name: tos-credential
51 - name: TOS_ENDPOINT
52 value: https://tos-s3-cn-beijing.ivolces.com
53 - name: TOS_REGION
54 value: cn-beijing
55 image: aibrix-cn-beijing.cr.volces.com/aibrix/runtime:v0.3.0
56 name: init-model
57 volumeMounts:
58 - mountPath: /models
59 name: model-hostpath
60 containers:
61 - name: vllm-openai
62 image: aibrix-container-registry-cn-beijing.cr.volces.com/aibrix/vllm-openai-aibrix-kvcache:v0.10.2-20251022
63 imagePullPolicy: Always
64 command:
65 - python3
66 - -m
67 - vllm.entrypoints.openai.api_server
68 - --port
69 - "8000"
70 - --uvicorn-log-level
71 - warning
72 - --model
73 - /models/DeepSeek-R1-Distill-Llama-8B/
74 - --trust-remote-code
75 - --served-model-name
76 - deepseek-r1-distill-llama-8b
77 - --max-model-len
78 - "32000" # please modify this field if your gpu has more room
79 # - --enable-chunked-prefill
80 - --disable-log-requests
81 - --disable-fastapi-docs
82 - --swap-space
83 - "0"
84 - --api-key
85 - "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"
86 - --no-enable-chunked-prefill
87 - --kv-transfer-config
88 - '{"kv_connector":"AIBrixOffloadingConnectorV1Type3", "kv_role":"kv_both"}'
89 env:
90 - name: VLLM_USE_V1
91 value: "1"
92 - name: AIBRIX_KV_CACHE_OL_L1_CACHE_ENABLED
93 value: "1"
94 # specify the eviction policy, default is S3FIFO
95 - name: AIBRIX_KV_CACHE_OL_L1_CACHE_EVICTION_POLICY
96 value: "S3FIFO"
97 # specify the capacity of L1 cache, default is 10GB
98 - name: AIBRIX_KV_CACHE_OL_L1_CACHE_CAPACITY_GB
99 value: "80"
100 - name: VLLM_RPC_TIMEOUT
101 value: "1000000"
102 volumeMounts:
103 - mountPath: /models
104 name: model-hostpath
105 resources:
106 limits:
107 nvidia.com/gpu: "1"
108 cpu: "10"
109 memory: "120G"
110 requests:
111 nvidia.com/gpu: "1"
112 cpu: "10"
113 memory: "120G"
114 volumes:
115 - name: model-hostpath
116 hostPath:
117 path: /root/models
118 type: DirectoryOrCreate
119
120---
121
122apiVersion: v1
123kind: Service
124metadata:
125 labels:
126 model.aibrix.ai/name: deepseek-r1-distill-llama-8b
127 prometheus-discovery: "true"
128 annotations:
129 prometheus.io/scrape: "true"
130 prometheus.io/port: "8080"
131 name: deepseek-r1-distill-llama-8b # Note: The Service name must match the label value `model.aibrix.ai/name` in the Deployment
132 namespace: default
133spec:
134 ports:
135 - name: serve
136 port: 8000
137 protocol: TCP
138 targetPort: 8000
139 - name: http
140 port: 8080
141 protocol: TCP
142 targetPort: 8080
143 selector:
144 model.aibrix.ai/name: deepseek-r1-distill-llama-8b
145 type: ClusterIP
$ kubectl apply -f samples/kvcache/l1cache/vllm.yaml
deployment.apps/deepseek-r1-distill-llama-8b created
service/deepseek-r1-distill-llama-8b created
注意
目前，vLLM v0.10.2 推荐的连接器是
AIBrixOffloadingConnectorV1Type3。如果需要，您可以通过指定--kv-transfer-config的kv_connector参数切换到其他 AIBrix 连接器。如果您倾向于使用 vLLM V0，请将
VLLM_USE_V1设置为0，并将--kv-transfer-config的值从'{"kv_connector":"AIBrixOffloadingConnectorV1Type3", "kv_role":"kv_both"}'更改为'{"kv_connector":"AIBrixOffloadingConnector", "kv_role":"kv_both"}'AIBRIX_KV_CACHE_OL_L1_CACHE_CAPACITY_GB需要根据 Pod 内存资源需求选择一个合适的值。例如，如果 Pod 内存资源需求为PGB，推理引擎的估计内存消耗为EGB，我们可以将AIBRIX_KV_CACHE_OL_L1_CACHE_CAPACITY_GB设置为P / tensor-parallel-size - E。
现在我们使用 kubectl get pods 命令确保推理服务正在运行
$ kubectl get pods -w
NAME READY STATUS RESTARTS AGE
deepseek-r1-distill-llama-8b-6bb7c97459-lhh77 0/1 PodInitializing 0 87s
deepseek-r1-distill-llama-8b-6bb7c97459-lhh77 1/1 Running 0 4m44s
一旦推理服务运行起来，我们设置端口转发，以便我们可以在本地测试服务
运行
kubectl get svc -n envoy-gateway-system以获取 Envoy Gateway 服务的名称。
$ kubectl get svc -n envoy-gateway-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy-aibrix-system-aibrix-eg-903790dc LoadBalancer 10.97.198.203 115.190.25.67 80:32269/TCP 5d3h
envoy-gateway ClusterIP 10.97.57.193 <none> 18000/TCP,18001/TCP,18002/TCP,19001/TCP 5d3h
运行
kubectl -n envoy-gateway-system port-forward svc/envoy-aibrix-system-aibrix-eg-903790dc 8888:80 &设置端口转发
$ kubectl -n envoy-gateway-system port-forward svc/envoy-aibrix-system-aibrix-eg-903790dc 8888:80 &
Forwarding from 127.0.0.1:8888 -> 10080
Forwarding from [::1]:8888 -> 10080
现在，我们测试服务
curl -v "https://:8888/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer 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" \
-d '{
"model": "deepseek-r1-distill-llama-8b",
"messages": [{"role": "user", "content": "Created container vllm-openai"}],
"temperature": 0.7
}'
其输出将是
* Trying [::1]:8888...
* Connected to localhost (::1) port 8888
> POST /v1/chat/completions HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/8.4.0
> Accept: */*
> Content-Type: application/json
> Authorization: Bearer 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
> Content-Length: 173
>
Handling connection for 8888
< HTTP/1.1 200 OK
< x-went-into-req-headers: true
< date: Wed, 21 May 2025 00:52:06 GMT
< server: uvicorn
< content-type: application/json
< target-pod: 192.168.3.22:8000
< request-id: 34a19ba1-88f2-4aa0-b914-5a28609d6b0a
< transfer-encoding: chunked
<
{"id":"chatcmpl-4ae8be13-5bbf-4bc0-92b6-6e8814296c57","object":"chat.completion","created":1747788726,"model":"deepseek-r1-distill-llama-8b","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"Okay, so I need to create a container called \"vllm-openai\" using Docker. I'm a bit new to this, so I'll have to figure it out step by step. Let me start by understanding what a Docker container is. From what I know, a container is a lightweight virtualization layer that allows me to package and run applications in isolated environments called containers. Docker makes this process easier by managing the containers and their images.\n\nI want to create a container that's specifically for running OpenAI's VLLM (Voss-LSTM), which is an open-source implementation of the original VLLM model by OpenAI. So, the container should have everything necessary to run this model, including the required dependencies and the model itself.\n\nFirst, I'll need to get the OpenAI VLLM code. I think it's available on GitHub, so I'll clone the repository. Let me check the URL: it's probably something like https://github.com/openai/vllm-cpp. Once I have the code, I need to build it. The instructions likely mention using CMake for the build process. I'll have to make sure I have CMake installed on my system. If not, I'll need to install it using my package manager.\n\nAfter cloning and building the code, I need to create a Docker image. The Dockerfile will specify the base image, which should be something like Ubuntu 20.04 LTS since it's a common and supported version. I'll need to set the working directory and copy the built VLLM files into the container. Also, I should install any system dependencies that the VLLM might need, like libraries or tools.\n\nI remember that VLLM requires certain Python packages, so I'll need to install those inside the container. The requirements.txt file probably lists all the necessary packages. Using pip to install them within the container makes sense. Additionally, since the model is quite large, the container might need more memory and CPU resources. I'll set up a non-root user for better security practices and ensure the permissions are set correctly so that the container can run the model without issues.\n\nI also need to expose the necessary ports. The VLLM server might run on port 8080, so I'll map that port in the Docker setup. For testing, I can use curl or a web interface to send requests to this port and see if the model responds correctly.\n\nLet me outline the steps I'll take:\n\n1. Clone the VLLM repository.\n2. Build the VLLM using CMake.\n3. Create a Dockerfile that includes the base OS, build tools, system dependencies, and copies the built VLLM files.\n4. Install the required Python packages using pip.\n5. Set up the container with proper user permissions and resource limits.\n6. Build and run the container.\n7. Test the container by sending requests to the exposed port.\n\nI'm a bit unsure about some parts. For example, how to handle the build process in the Dockerfile? I think the Dockerfile will need to have the necessary CMake commands and possibly install build dependencies like build-essential. Also, I need to make sure that the container has enough memory allocated to run the VLLM model, which can be quite resource-intensive.\n\nAnother thing I'm not sure about is the user setup. Why do I need a non-root user? Isn't it easier to run everything as root? Well, running as a non-root user is more secure, especially since Docker containers have root privileges by default. So, I should create a user and switch to it before running the model.\n\nI should also think about how the container handles persistence. Since the VLLM model is built outside the container, the container will only have the necessary files. If I need to persist the model, I'll have to copy it into the container during the build process. Otherwise, each container restart will require rebuilding the model, which might be time-consuming.\n\nLet me think about the Dockerfile structure. It should start with a FROM instruction based on an Ubuntu image. Then, set the working directory, install build-essential and cmake, clone the repository, build it, and then copy the built files into the container. After that, I'll switch to a non-root user and install the Python dependencies.\n\nWait, but the VLLM requires certain libraries like TensorFlow? Or is it self-contained? I think the VLLM is a standalone model, so maybe it doesn't rely on external libraries beyond what's already in the build. But I should check the requirements to be sure.\n\nAlso, the model is quite large, so the container might take up a lot of disk space. I should consider using a larger disk or use a persistent volume if I need to keep the model data.\n\nI should also document the container, maybe add some notes on how to use it, like the exposed ports and any required environment variables. For example, the VLLM might need an API key or specific configurations to run.\n\nTesting is important. After building the container, I can run it and use curl to send a request to the exposed port. If the response is as expected, the container is working. If not, I'll have to troubleshoot, maybe checking the logs or ensuring all dependencies are correctly installed.\n\nI'm a bit worried about performance. The VLLM model is designed for research purposes, so it's going to be computationally heavy. I should set resource limits in the Docker run command to prevent it from using too much of the host system's resources.\n\nIn summary, the process involves setting up the build environment, compiling the VLLM code into a Docker image, installing necessary dependencies, and ensuring the container runs securely and efficiently.\n</think>\n\nTo create a Docker container for OpenAI's VLLM, follow these organized steps:\n\n### Step-by-Step Guide\n\n1. **Clone the VLLM Repository**\n - Clone the VLLM repository from GitHub:\n ```bash\n git clone https://github.com/openai/vllm-cpp.git\n ```\n - Navigate to the cloned directory:\n ```bash\n cd vllm-cpp\n ```\n\n2. **Build the VLLM**\n - Ensure you have CMake installed. If not, install it using your package manager.\n - Build the VLLM using CMake:\n ```bash\n mkdir build\n cd build\n cmake ..\n make\n ```\n\n3. **Create the Dockerfile**\n - Open a new file named `Dockerfile` and insert the following content:\n ```dockerfile\n FROM ubuntu:20.04\n\n WORKDIR /app\n\n # Install build tools\n RUN apt-get update && apt-get install -y build-essential cmake\n # Install system dependencies\n RUN apt-get install -y libboost-system-dev libboost-filesystem-dev \\\n libboost-chrono-dev libboost-serialization-dev libboost-headers\n # Copy the built VLLM files\n COPY build/vllm-cpp .\n # Install Python dependencies\n RUN useradd -m vllmuser && chown -R vllmuser:vllmuser .\n RUN pip install -r requirements.txt\n # Switch to non-root user\n USER vllmuser\n ```\n - **Note:** Replace `requirements.txt` with your actual file path or content if you haven't created one yet.\n\n4. **Build and Run the Container**\n - Build the Docker image:\n ```bash\n docker build -t vllm-openai .\n ```\n - Run the container, allocating enough resources (e.g., 4GB RAM and 4 CPUs):\n ```bash\n docker run -d --name vllm-openai \\\n -e \"HTTP_PROXY=http://proxy.example.com:8080\" \\\n -e \"HTTPS_PROXY=http://proxy.example.com:8080\" \\\n --ulimits cgroup:1 --cpu-shares 1 --memory 4g \\\n vllm-openai\n ```\n - Replace `proxy.example.com` with your actual proxy server if needed.\n\n5. **Test the Container**\n - Check if the container is running:\n ```bash\n docker ps\n ```\n - Use `curl` to test the API:\n ```bash\n curl https://:8080\n ```\n - If the response is as expected, the container is functioning correctly.\n\n### Notes\n\n- **User Permissions:** The container uses a non-root user (`vllmuser`) for security reasons.\n- **Dependencies:** Ensure all system a* Connection #0 to host localhost left intact
nd Python dependencies are correctly installed as per the VLLM requirements.\n- **Resources:** Adjust CPU and memory allocations based on your system's capacity to handle the VLLM's computational demands.\n- **Volumes:** Consider using a persistent volume to store the VLLM model for longer-term use.\n\nBy following these steps, you'll have a containerized version of OpenAI's VLLM ready to run, ensuring security, efficiency, and ease of use.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":12,"total_tokens":1887,"completion_tokens":1875,"prompt_tokens_details":null},"prompt_logprobs":null}
L2 缓存示例#
让我们使用以下 yaml 配置部署分布式 KV 缓存集群
1apiVersion: orchestration.aibrix.ai/v1alpha1
2kind: KVCache
3metadata:
4 name: kvcache-cluster
5 namespace: default
6 annotations:
7 kvcache.orchestration.aibrix.ai/backend: infinistore
8 infinistore.kvcache.orchestration.aibrix.ai/link-type: "Ethernet"
9 infinistore.kvcache.orchestration.aibrix.ai/hint-gid-index: "7"
10spec:
11 metadata:
12 redis:
13 runtime:
14 image: aibrix-cn-beijing.cr.volces.com/aibrix/redis:7.4.2
15 replicas: 1
16 resources:
17 requests:
18 cpu: 1000m
19 memory: 1Gi
20 limits:
21 cpu: 1000m
22 memory: 1Gi
23 service:
24 type: ClusterIP
25 ports:
26 - name: service
27 port: 12345
28 targetPort: 12345
29 protocol: TCP
30 - name: admin
31 port: 8088
32 targetPort: 8088
33 protocol: TCP
34 watcher:
35 image: aibrix-cn-beijing.cr.volces.com/aibrix/kvcache-watcher:v0.3.0
36 imagePullPolicy: Always
37 resources:
38 requests:
39 cpu: "500m"
40 memory: "256Mi"
41 limits:
42 cpu: "500m"
43 memory: "256Mi"
44 cache:
45 replicas: 1
46 image: aibrix-cn-beijing.cr.volces.com/aibrix/infinistore:v0.2.42-20250506
47 imagePullPolicy: IfNotPresent
48 resources:
49 requests:
50 cpu: "10000m"
51 memory: "120Gi"
52 vke.volcengine.com/rdma: "1"
53 limits:
54 cpu: "10000m"
55 memory: "120Gi"
56 vke.volcengine.com/rdma: "1"
注意
在此示例中，我们已将 L45 从一个副本更改为三个副本，因此我们将在集群中找到三个 KV 缓存 pod 正在运行。
$ kubectl apply -f samples/kvcache/infinistore/kvcache.yaml
kvcache.orchestration.aibrix.ai/kvcache-cluster created
$ kubectl get pods -w
NAME READY STATUS RESTARTS AGE
kvcache-cluster-0 0/1 ContainerCreating 0 59s
kvcache-cluster-kvcache-watcher-pod 1/1 Running 0 59s
kvcache-cluster-redis 1/1 Running 0 59s
kvcache-cluster-0 1/1 Running 0 2m43s
kvcache-cluster-1 0/1 Pending 0 0s
kvcache-cluster-1 0/1 Pending 0 0s
kvcache-cluster-1 0/1 ContainerCreating 0 0s
kvcache-cluster-1 0/1 ContainerCreating 0 2s
kvcache-cluster-1 1/1 Running 0 5s
kvcache-cluster-2 0/1 Pending 0 0s
kvcache-cluster-2 0/1 Pending 0 0s
kvcache-cluster-2 0/1 ContainerCreating 0 0s
kvcache-cluster-2 0/1 ContainerCreating 0 2s
kvcache-cluster-2 1/1 Running 0 4s
现在我们使用以下 yaml 文件创建一个引擎部署
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: deepseek-r1-distill-llama-8b
5 labels:
6 model.aibrix.ai/name: deepseek-r1-distill-llama-8b
7 model.aibrix.ai/port: "8000"
8spec:
9 replicas: 1
10 strategy:
11 rollingUpdate:
12 maxSurge: 1
13 maxUnavailable: 1
14 type: RollingUpdate
15 selector:
16 matchLabels:
17 model.aibrix.ai/name: deepseek-r1-distill-llama-8b
18 template:
19 metadata:
20 labels:
21 model.aibrix.ai/name: deepseek-r1-distill-llama-8b
22 annotations:
23 prometheus.io/path: "/metrics"
24 prometheus.io/port: "8000"
25 prometheus.io/scrape: "true"
26 k8s.volcengine.com/pod-networks: |
27 [
28 {
29 "cniConf":{
30 "name":"rdma"
31 }
32 }
33 ]
34 spec:
35 initContainers:
36 - command:
37 - aibrix_download
38 - --model-uri
39 - tos://aibrix-artifact-testing/models/DeepSeek-R1-Distill-Llama-8B/
40 - --local-dir
41 - /models/
42 env:
43 - name: DOWNLOADER_NUM_CONNECTIONS
44 value: "16"
45 - name: DOWNLOADER_NUM_THREADS
46 value: "16"
47 - name: DOWNLOADER_ALLOW_FILE_SUFFIX
48 value: json, safetensors
49 - name: TOS_ACCESS_KEY
50 valueFrom:
51 secretKeyRef:
52 key: TOS_ACCESS_KEY
53 name: tos-credential
54 - name: TOS_SECRET_KEY
55 valueFrom:
56 secretKeyRef:
57 key: TOS_SECRET_KEY
58 name: tos-credential
59 - name: TOS_ENDPOINT
60 value: https://tos-s3-cn-beijing.ivolces.com
61 - name: TOS_REGION
62 value: cn-beijing
63 image: aibrix-cn-beijing.cr.volces.com/aibrix/runtime:v0.3.0
64 name: init-model
65 volumeMounts:
66 - mountPath: /models
67 name: model-hostpath
68 containers:
69 - name: vllm-openai
70 image: aibrix-container-registry-cn-beijing.cr.volces.com/aibrix/vllm-openai-aibrix-kvcache:v0.10.2-20251022
71 imagePullPolicy: Always
72 command:
73 - python3
74 - -m
75 - vllm.entrypoints.openai.api_server
76 - --port
77 - "8000"
78 - --uvicorn-log-level
79 - warning
80 - --model
81 - /models/DeepSeek-R1-Distill-Llama-8B/
82 - --trust-remote-code
83 - --served-model-name
84 - deepseek-r1-distill-llama-8b
85 - --max-model-len
86 - "32000" # please modify this field if your gpu has more room
87 # - --enable-chunked-prefill
88 - --disable-log-requests
89 - --disable-fastapi-docs
90 - --swap-space
91 - "0"
92 - --api-key
93 - "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"
94 - --no-enable-chunked-prefill
95 - --kv-transfer-config
96 - '{"kv_connector":"AIBrixOffloadingConnectorV1Type3", "kv_role":"kv_both"}'
97 env:
98 - name: VLLM_USE_V1
99 value: "1"
100 - name: AIBRIX_KV_CACHE_OL_L1_CACHE_ENABLED
101 value: "0"
102 - name: AIBRIX_KV_CACHE_OL_L2_CACHE_BACKEND
103 value: "infinistore"
104 - name: AIBRIX_KV_CACHE_OL_INFINISTORE_CONNECTION_TYPE
105 value: "RDMA"
106 - name: AIBRIX_KV_CACHE_OL_INFINISTORE_IB_PORT
107 value: "1"
108 - name: AIBRIX_KV_CACHE_OL_INFINISTORE_LINK_TYPE
109 value: "Ethernet"
110 # mlx_5_1 is the device and 7 is the hinted gid index, if you do not know the gid, you can just type mlx5_1,mlx5_2,...
111 - name: AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST
112 value: "mlx5_1:7,mlx5_2:7,mlx5_3:7,mlx5_4:7"
113 - name: AIBRIX_KV_CACHE_OL_META_SERVICE_BACKEND
114 value: "redis"
115 - name: AIBRIX_KV_CACHE_OL_META_SERVICE_URL
116 value: "redis://kvcache-cluster-redis:6379"
117 - name: AIBRIX_KV_CACHE_OL_META_SERVICE_CLUSTER_META_KEY
118 value: "kvcache_nodes"
119 - name: VLLM_RPC_TIMEOUT
120 value: "1000000"
121 volumeMounts:
122 - mountPath: /models
123 name: model-hostpath
124 resources:
125 limits:
126 nvidia.com/gpu: "1"
127 vke.volcengine.com/rdma: "1"
128 cpu: "10"
129 memory: "120G"
130 requests:
131 nvidia.com/gpu: "1"
132 vke.volcengine.com/rdma: "1"
133 cpu: "10"
134 memory: "120G"
135 securityContext:
136 capabilities:
137 add:
138 - IPC_LOCK
139 volumes:
140 - name: model-hostpath
141 hostPath:
142 path: /root/models
143 type: DirectoryOrCreate
144
145---
146
147apiVersion: v1
148kind: Service
149metadata:
150 labels:
151 model.aibrix.ai/name: deepseek-r1-distill-llama-8b
152 prometheus-discovery: "true"
153 annotations:
154 prometheus.io/scrape: "true"
155 prometheus.io/port: "8080"
156 name: deepseek-r1-distill-llama-8b # Note: The Service name must match the label value `model.aibrix.ai/name` in the Deployment
157 namespace: default
158spec:
159 ports:
160 - name: serve
161 port: 8000
162 protocol: TCP
163 targetPort: 8000
164 - name: http
165 port: 8080
166 protocol: TCP
167 targetPort: 8080
168 selector:
169 model.aibrix.ai/name: deepseek-r1-distill-llama-8b
170 type: ClusterIP
$ kubectl apply -f samples/kvcache/infinistore/vllm.yaml
deployment.apps/deepseek-r1-distill-llama-8b created
service/deepseek-r1-distill-llama-8b created
注意
目前，vLLM v0.10.2 推荐的连接器是
AIBrixOffloadingConnectorV1Type3。如果需要，您可以通过指定--kv-transfer-config的kv_connector参数切换到其他 AIBrix 连接器。如果您倾向于使用 vLLM V0，请将
VLLM_USE_V1设置为0，并将--kv-transfer-config的值从'{"kv_connector":"AIBrixOffloadingConnectorV1Type3", "kv_role":"kv_both"}'更改为'{"kv_connector":"AIBrixOffloadingConnector", "kv_role":"kv_both"}'在此示例中，我们设置
AIBRIX_KV_CACHE_OL_L1_CACHE_ENABLED=0以明确禁用L1Cache并仅使用L2Cache。当前版本仅支持使用带有 RDMA 传输的
InfiniStore。请确保配置了AIBRIX_KV_CACHE_OL_INFINISTORE_CONNECTION_TYPE=RDMA。对于 InfiniBand，请设置
AIBRIX_KV_CACHE_OL_INFINISTORE_LINK_TYPE=IB。对于 RoCE，请保留默认值Ethernet。AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST用于配置引擎可以使用哪些 RDMA 设备来访问远程 KV 缓存服务器。例如，如果您为引擎 Pod 分配 8 个 GPU 并设置AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST="mlx5_1,mlx5_2"，则使用 GPU 0 到 3 的引擎进程将使用mlx5_1，而使用 GPU 4 到 7 的引擎进程将使用mlx5_2。如果您的环境中需要 RDMA 设备的 GID 索引，请将 GID 索引附加到
AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST中的每个 RDMA 设备（例如，mlx5_1:6,mlx5_2:7）。AIBRIX_KV_CACHE_OL_META_SERVICE_URL指向管理 KV 缓存集群元数据的 Redis 实例。在此示例中，它设置为redis://kvcache-cluster-redis:6379，其中kvcache-cluster是 KV 缓存部署名称。AIBRIX_KV_CACHE_OL_META_SERVICE_BACKEND和AIBRIX_KV_CACHE_OL_META_SERVICE_CLUSTER_META_KEY在当前版本中是固定的，不应修改。
现在我们使用 kubectl get pods 命令确保推理服务正在运行
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
deepseek-r1-distill-llama-8b-6587db8894-pbbxk 1/1 Running 0 34s
kvcache-cluster-0 1/1 Running 0 7m55s
kvcache-cluster-1 1/1 Running 0 5m12s
kvcache-cluster-2 1/1 Running 0 5m7s
kvcache-cluster-kvcache-watcher-pod 1/1 Running 0 7m55s
kvcache-cluster-redis 1/1 Running 0 7m55s
一旦推理服务运行起来，我们设置端口转发，以便我们可以在本地测试服务
运行
kubectl get svc -n envoy-gateway-system以获取 Envoy Gateway 服务的名称。
$ kubectl get svc -n envoy-gateway-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy-aibrix-system-aibrix-eg-903790dc LoadBalancer 10.97.198.203 115.190.25.67 80:32269/TCP 5d3h
envoy-gateway ClusterIP 10.97.57.193 <none> 18000/TCP,18001/TCP,18002/TCP,19001/TCP 5d3h
运行
kubectl -n envoy-gateway-system port-forward svc/envoy-aibrix-system-aibrix-eg-903790dc 8888:80 &设置端口转发
$ kubectl -n envoy-gateway-system port-forward svc/envoy-aibrix-system-aibrix-eg-903790dc 8888:80 &
Forwarding from 127.0.0.1:8888 -> 10080
Forwarding from [::1]:8888 -> 10080
现在，我们测试服务
curl -v "https://:8888/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer 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" \
-d '{
"model": "deepseek-r1-distill-llama-8b",
"messages": [{"role": "user", "content": "Created container vllm-openai"}],
"temperature": 0.7
}'
其输出将是
* Trying [::1]:8888...
* Connected to localhost (::1) port 8888
> POST /v1/chat/completions HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/8.4.0
> Accept: */*
> Content-Type: application/json
> Authorization: Bearer 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
> Content-Length: 173
>
Handling connection for 8888
< HTTP/1.1 200 OK
< x-went-into-req-headers: true
< date: Wed, 21 May 2025 05:44:04 GMT
< server: uvicorn
< content-type: application/json
< target-pod: 192.168.3.28:8000
< request-id: f9b291ae-fbce-4b63-bba5-c8f04d812cd0
< transfer-encoding: chunked
<
{"id":"chatcmpl-dce54e48-3c47-4d08-8ff9-dcec429fd486","object":"chat.completion","created":1747806244,"model":"deepseek-r1-distill-llama-8b","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"Okay, so I'm trying to create a container for VLLM-OpenAI. I'm a bit new to this, so I need to figure out where to start. I know that VLLM stands for Very Large Language Model, and OpenAI has their own models like GPT-4 and others. But I'm not exactly sure how to create a container for it. \n\nFirst, I think I need to understand what a container is. From what I remember, containers are like lightweight virtual machines that you can use to package up an application and its dependencies. Docker is a popular tool for creating and managing these containers. So, I probably need to use Docker to create a container that runs VLLM-OpenAI.\n\nI should check if there's an official Docker image for VLLM-OpenAI. Maybe OpenAI provides one? If not, I might have to build one myself. Building from source code would mean I need access to the model's codebase, which I'm not sure about. I should look up if there's a public repository for VLLM-OpenAI.\n\nWait, I think OpenAI has released some of their models as open-source, but I'm not certain about VLLM specifically. I should search for \"VLLM-OpenAI Docker\" or \"VLLM-OpenAI container\" to see if someone has already created a container. Maybe there's a GitHub repository or a Docker Hub page with the image.\n\nIf I can't find an existing image, I'll have to create one myself. To do that, I need to know what dependencies the model requires. VLLM is based on LLMs, so it probably needs libraries like PyTorch or TensorFlow. Also, it might require specific versions of Python or other tools. I should look up the installation instructions for VLLM-OpenAI to identify the necessary dependencies.\n\nI'll need a Dockerfile. The Dockerfile will include a base image, install the dependencies, and copy the model's code. I should make sure to use the correct Python version, as some models might have compatibility issues. I'll also need a requirements.txt file to list all the necessary Python packages and their versions.\n\nOnce the Dockerfile is set up, I can build the container using Docker. The command would be something like `docker build -t vllm-openai .` where `vllm-openai` is the name of the container. After building, I can run it with `docker run -it vllm-openai`, which will start an interactive session.\n\nI should also consider how to manage the model once the container is running. Do I need to pass it a prompt through stdin? How does it handle outputs? I should look up the usage instructions for VLLM-OpenAI to know how to interact with it within the container.\n\nAnother thing to think about is resource usage. VLLM models are computationally intensive, so I need to make sure the container has enough resources allocated. This can be done when running the container with options like `--cpuset` or `--memory` if necessary.\n\nI'm a bit worried about the size of the model. VLLM might have a large embedding size, so the container might become quite large. I should check if there are optimized versions or ways to reduce the model size without losing too much performance.\n\nAlso, I should think about versioning. If I create a container, I should name it something that includes the version number, like `vllm-openai-1.0`. This way, I can easily update to newer versions by rebuilding the container.\n\nI wonder if there are any specific commands or tools needed to run VLLM-OpenAI in a container. Maybe I need to use a specific framework or tooling that's already included in the container. I should make sure I have all the necessary command-line tools installed before trying to run it.\n\nI should also test the container locally to see if it works. Maybe start with a simple prompt to see if the model responds. If it doesn't, I'll need to troubleshoot whether it's an issue with the container setup or the model configuration.\n\nIn summary, my steps would be:\n1. Search for existing V* Connection #0 to host localhost left intact
LLM-OpenAI containers or Docker images.\n2. If none found, create a new Dockerfile and requirements.txt.\n3. Install necessary dependencies and copy the model code.\n4. Build and run the container using Docker.\n5. Test the container with a sample input.\n6. Adjust resources and configurations as needed.\n\nI might run into issues like dependency conflicts or missing packages, so I should be prepared to update versions or check the model's documentation for specific requirements. Also, understanding how the model expects inputs and outputs is crucial for effective use.\n\nI think I've got a basic plan. Now, I'll try to find the existing resources or proceed to set up the Dockerfile if necessary. Let me start by searching for VLLM-OpenAI Docker container on GitHub or Docker Hub to see if someone else has done this before.\n</think>\n\nTo create a container for VLLM-OpenAI, follow these organized steps:\n\n1. **Search for Existing Containers**:\n - Check Docker Hub or GitHub for existing VLLM-OpenAI containers. If found, use them as they may already be configured.\n\n2. **Prepare Your Environment**:\n - Ensure you have Docker installed on your system.\n\n3. **Set Up the Project Structure**:\n - Create a directory for your project.\n - Within it, create a `Dockerfile` and a `requirements.txt` file.\n\n4. **Dockerfile Setup**:\n - Use a base image that matches your system's requirements (e.g., `python:3.9-slim`).\n - Install necessary dependencies from `requirements.txt`.\n - Copy the VLLM-OpenAI code into the container.\n\n5. **requirements.txt**:\n - List all Python packages needed, including specific versions, such as `transformers` and `torch`.\n\n6. **Build the Container**:\n - Use the command `docker build -t vllm-openai .` to build the container.\n\n7. **Run the Container**:\n - Start the container with `docker run -it vllm-openai` for an interactive session.\n\n8. **Test the Container**:\n - Issue a test command to ensure the model responds, e.g., `echo \"Hello, how are you?\" | docker run -it vllm-openai`.\n\n9. **Optimize Resources**:\n - Adjust resource allocation with options like `--cpuset` or `--memory` to handle computational demands.\n\n10. **Versioning**:\n - Name your container with versioning, such as `vllm-openai-1.0`.\n\n11. **Troubleshooting**:\n - If issues arise, check for dependency conflicts or review the model's documentation for specific requirements.\n\n12. **Documentation and Usage**:\n - Familiarize yourself with how VLLM-OpenAI expects inputs and outputs for effective utilization.\n\nBy following these steps, you can efficiently create and manage a container for VLLM-OpenAI, ensuring it runs smoothly within your environment.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":12,"total_tokens":1505,"completion_tokens":1493,"prompt_tokens_details":null},"prompt_logprobs":null}
性能分析示例#
要分析 AIBrix 卸载连接器，在启动推理引擎之前，您需要使用以下示例 YAML 文件启动分析服务
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: aibrix-kvcache-profiling
5 labels:
6 model.aibrix.ai/name: aibrix-kvcache-profiling
7spec:
8 replicas: 1
9 selector:
10 matchLabels:
11 model.aibrix.ai/name: aibrix-kvcache-profiling
12 template:
13 metadata:
14 labels:
15 model.aibrix.ai/name: aibrix-kvcache-profiling
16 spec:
17 containers:
18 - name: pyroscope
19 image: aibrix-container-registry-cn-beijing.cr.volces.com/aibrix/pyroscope:latest
20 imagePullPolicy: Always
21 resources:
22 requests:
23 cpu: "2000m"
24 memory: "4Gi"
25 limits:
26 cpu: "2000m"
27 memory: "4Gi"
28---
29
30apiVersion: v1
31kind: Service
32metadata:
33 name: aibrix-kvcache-profiling
34 namespace: default
35spec:
36 ports:
37 - name: http
38 port: 4040
39 protocol: TCP
40 targetPort: 4040
41 selector:
42 model.aibrix.ai/name: aibrix-kvcache-profiling
43 type: ClusterIP
$ kubectl apply -f samples/kvcache/profiling/profiling_svc.yaml
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
aibrix-kvcache-profiling ClusterIP 10.70.106.217 <none> 4040/TCP 80m
一旦分析服务运行起来，我们设置端口转发，以便我们可以在本地浏览分析结果和火焰图
运行
kubectl port-forward svc/aibrix-kvcache-profiling 4040:4040 &设置端口转发
现在，让我们使用引擎 YAML 文件中设置的以下环境变量启动推理引擎（更多详情请参阅L1 缓存示例和L2 缓存示例）。
env:
- name: AIBRIX_KV_CACHE_OL_PROFILING_ENABLED
value: "1"
- name: AIBRIX_KV_CACHE_OL_PROFILING_SERVER_ADDRESS
value: "http://aibrix-kvcache-profiling:4040"
运行一个任务向推理引擎生成请求，一段时间后您就可以在本地浏览分析结果和火焰图
在浏览器中打开 https://:4040
调整查询参数，如下图所示，以显示 AIBrix 卸载连接器的火焰图
