AI 引擎运行时

内容

AI 引擎运行时#

安装#

AIBrix 运行时可以通过基于 webhook 的 Sidecar 注入（推荐）自动注入到您的工作负载中，或者手动添加到您的部署清单中。

自动注入 Sidecar（推荐）#

启用运行时最简单的方法是自动注入 Sidecar。只需在您的 Deployment 或 StormService 中添加一个注解

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
  annotations:
    model.aibrix.ai/sidecar-injection: "true"  # Enable automatic runtime injection
spec:
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        # Your container configuration...

webhook 将自动将 aibrix-runtime Sidecar 容器注入到您的 Pod 中。

对于 StormService

Sidecar 注入也适用于 StormService 自定义资源

apiVersion: orchestration.aibrix.ai/v1alpha1
kind: StormService
metadata:
  name: my-service
  annotations:
    model.aibrix.ai/sidecar-injection: "true"
spec:
  template:
    spec:
      roles:
      - name: worker
        template:
          spec:
            containers:
            - name: vllm
              image: vllm/vllm-openai:latest
              # ...

运行时 Sidecar 将被注入到每个角色的 Pod 模板中。

自定义运行时镜像

您可以使用注解指定自定义运行时镜像

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
  annotations:
    model.aibrix.ai/sidecar-injection: "true"
    model.aibrix.ai/sidecar-runtime-image: "aibrix/runtime:v0.5.0"  # Custom image
spec:
  # ...

启用全局运行时标志

要使控制器使用运行时 Sidecar API，请在启动控制器管理器时设置全局标志

# Enable runtime sidecar globally
./bin/controller-manager --enable-runtime-sidecar=true

运行时检测逻辑如下

EnableRuntimeSidecar = false：控制器始终使用直接引擎 API（端口 8000），即使注入了 Sidecar
EnableRuntimeSidecar = true：控制器检测 Pod 是否有 aibrix-runtime 容器
- Sidecar 存在 → 使用运行时 API（端口 8080）
- Sidecar 不存在 → 回退到直接引擎 API（端口 8000）

此设计确保功能无论是否有运行时 Sidecar 都能正常工作，提供最大的灵活性。

手动安装 Sidecar#

如果您更喜欢手动控制，可以将运行时 Sidecar 直接添加到您的部署 YAML 中

containers:
- name: vllm
  image: vllm/vllm-openai:latest
  # Your main container configuration...

- name: aibrix-runtime
  image: aibrix/runtime:v0.5.0
  command:
  - aibrix_runtime
  - --port
  - "8080"
  env:
  - name: INFERENCE_ENGINE
    value: "vllm"  # or sglang, tgi, triton, llamacpp
  - name: INFERENCE_ENGINE_ENDPOINT
    value: "https://:8000"
  ports:
  - containerPort: 8080
    protocol: TCP
  volumeMounts:
  - mountPath: /models
    name: model-hostpath
volumes:
- name: model-hostpath
  hostPath:
    path: /root/models
    type: DirectoryOrCreate

独立安装#

如果您想在 Kubernetes 之外的其他情况下使用运行时，可以通过以下命令安装。

注意

python3 -m pip install aibrix

如果您想使用每夜版本，可以从代码安装。

cd $AIBRIX_HOME/python/aibrix && python3 -m pip install -e .

指标标准化#

不同的推理引擎会暴露不同的指标，AI Runtime 将对其进行标准化。

在容器环境变量中定义与推理引擎侧相关的信息。例如，如果 vLLM 在 https://:8000/metrics 提供指标服务，请通过以下命令启动 AI Runtime Server

INFERENCE_ENGINE=vllm INFERENCE_ENGINE_ENDPOINT="https://:8000" aibrix_runtime --port 8080

并且运行时将在 https://:8080/metrics 提供统一的推理指标。目前，您会注意到这些指标与 vLLM 指标完全相同。一旦支持更多引擎，这在未来可能会改变。统一指标有助于标准化不同推理引擎以及自动扩缩器等组件的指标。

# TYPE vllm:cache_config_info gauge
vllm:cache_config_info{block_size="16",cache_dtype="auto",calculate_kv_scales="False",cpu_offload_gb="0",enable_prefix_caching="False",gpu_memory_utilization="0.9",is_attention_free="False",num_cpu_blocks="9362",num_gpu_blocks="81767",num_gpu_blocks_override="None",sliding_window="None",swap_space_bytes="4294967296"} 1.0
# HELP vllm:num_requests_running Number of requests currently running on GPU.
# TYPE vllm:num_requests_running gauge
vllm:num_requests_running{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 0.0
# HELP vllm:num_requests_swapped Number of requests swapped to CPU.
# TYPE vllm:num_requests_swapped gauge
vllm:num_requests_swapped{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 0.0
# HELP vllm:num_requests_waiting Number of requests waiting to be processed.
# TYPE vllm:num_requests_waiting gauge
vllm:num_requests_waiting{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 0.0
# HELP vllm:gpu_cache_usage_perc GPU KV-cache usage. 1 means 100 percent usage.
# TYPE vllm:gpu_cache_usage_perc gauge
vllm:gpu_cache_usage_perc{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 0.0
# HELP vllm:cpu_cache_usage_perc CPU KV-cache usage. 1 means 100 percent usage.
# TYPE vllm:cpu_cache_usage_perc gauge
vllm:cpu_cache_usage_perc{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 0.0
# HELP vllm:cpu_prefix_cache_hit_rate CPU prefix cache block hit rate.
# TYPE vllm:cpu_prefix_cache_hit_rate gauge
vllm:cpu_prefix_cache_hit_rate{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} -1.0
# HELP vllm:gpu_prefix_cache_hit_rate GPU prefix cache block hit rate.
# TYPE vllm:gpu_prefix_cache_hit_rate gauge
vllm:gpu_prefix_cache_hit_rate{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} -1.0
# HELP vllm:lora_requests_info Running stats on lora requests.
# TYPE vllm:lora_requests_info gauge
vllm:lora_requests_info{max_lora="0",running_lora_adapters="",waiting_lora_adapters=""} 1.7382173358407154e+09
# HELP vllm:num_preemptions_total Cumulative number of preemption from the engine.
# TYPE vllm:num_preemptions_total counter
vllm:num_preemptions_total{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 0.0
# HELP vllm:prompt_tokens_total Number of prefill tokens processed.
# TYPE vllm:prompt_tokens_total counter
vllm:prompt_tokens_total{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 148.0
# HELP vllm:generation_tokens_total Number of generation tokens processed.
# TYPE vllm:generation_tokens_total counter
vllm:generation_tokens_total{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 955.0
# HELP vllm:request_success_total Count of successfully processed requests.
# TYPE vllm:request_success_total counter
vllm:request_success_total{finished_reason="stop",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 4.0
# HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
# TYPE vllm:iteration_tokens_total histogram
vllm:iteration_tokens_total_sum{model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 1103.0
vllm:iteration_tokens_total_bucket{le="1.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 994.0
vllm:iteration_tokens_total_bucket{le="2.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 994.0
vllm:iteration_tokens_total_bucket{le="4.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 994.0
vllm:iteration_tokens_total_bucket{le="8.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 994.0
vllm:iteration_tokens_total_bucket{le="16.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 994.0
vllm:iteration_tokens_total_bucket{le="24.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 994.0
vllm:iteration_tokens_total_bucket{le="32.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 994.0
vllm:iteration_tokens_total_bucket{le="40.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 998.0
vllm:iteration_tokens_total_bucket{le="48.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 998.0
vllm:iteration_tokens_total_bucket{le="56.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 998.0
vllm:iteration_tokens_total_bucket{le="64.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 998.0
vllm:iteration_tokens_total_bucket{le="72.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 998.0
vllm:iteration_tokens_total_bucket{le="80.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 998.0
vllm:iteration_tokens_total_bucket{le="88.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 998.0
vllm:iteration_tokens_total_bucket{le="96.0",model_name="Qwen/Qwen2.5-Coder-1.5B-Instruct"} 998.0

注意

由于 vLLM 是目前唯一支持的引擎，我们没有使用与供应商无关的指标。一旦支持更多引擎，这在未来可能会改变。

模型下载#

AI 引擎运行时支持从多个远程源下载模型，包括 HuggingFace、S3 和 TOS。当控制平面需要与 Pod 交互以动态加载新模型时，这非常有用。

从 HuggingFace 下载#

首先为 HuggingFace 模型定义必要的环境变量。

# General settings
export DOWNLOADER_ALLOW_FILE_SUFFIX="json, safetensors"
export DOWNLOADER_NUM_THREADS=16
# HuggingFace settings
export HF_ENDPOINT=https://hf-mirror.com  # set it when env is in CN region

然后使用 AI 引擎运行时从 HuggingFace 下载模型

python -m aibrix.downloader \
    --model-uri deepseek-ai/deepseek-coder-6.7b-instruct \
    --local-dir /tmp/aibrix/models_hf/

从 S3 下载#

首先为 S3 模型定义必要的环境变量。

# General settings
export DOWNLOADER_ALLOW_FILE_SUFFIX="json, safetensors"
export DOWNLOADER_NUM_THREADS=16
# AWS settings
export AWS_ACCESS_KEY_ID=<INPUT YOUR AWS ACCESS KEY ID>
export AWS_SECRET_ACCESS_KEY=<INPUT YOUR AWS SECRET ACCESS KEY>
export AWS_ENDPOINT_URL=<INPUT YOUR AWS ENDPOINT URL> # e.g. https://s3.us-west-2.amazonaws.com
export AWS_REGION=<INPUT YOUR AWS REGION> # e.g. us-west-2

然后使用 AI 运行时从 AWS S3 下载模型

python -m aibrix.downloader \
    --model-uri s3://aibrix-model-artifacts/deepseek-coder-6.7b-instruct/ \
    --local-dir /tmp/aibrix/models_s3/

从 TOS 下载#

首先为 TOS 模型定义必要的环境变量。

# General settings
export DOWNLOADER_ALLOW_FILE_SUFFIX="json, safetensors"
export DOWNLOADER_NUM_THREADS=16
# AWS settings
export TOS_ACCESS_KEY=<INPUT YOUR TOS ACCESS KEY>
export TOS_SECRET_KEY=<INPUT YOUR TOS SECRET KEY>
export TOS_ENDPOINT=<INPUT YOUR TOS ENDPOINT> # e.g. https://tos-s3-cn-beijing.volces.com
export TOS_REGION=<INPUT YOUR TOS REGION> # e..g cn-beijing

然后使用 AI 运行时从 TOS 下载模型

python -m aibrix.downloader \
    --model-uri tos://aibrix-model-artifacts/deepseek-coder-6.7b-instruct/ \
    --local-dir /tmp/aibrix/models_tos/

模型配置 API#

注意

这需要引擎以 –enable-lora 启动并启用环境变量 export VLLM_ALLOW_RUNTIME_LORA_UPDATING=true。您可以查看动态提供 LoRA 适配器获取更多详情。

假设您已经部署了基础模型和运行时，并且想要加载 LoRA 适配器。

# start the engine
VLLM_ALLOW_RUNTIME_LORA_UPDATING=true vllm serve Qwen/Qwen2.5-Coder-1.5B-Instruct --enable-lora
# start the runtime
INFERENCE_ENGINE=vllm INFERENCE_ENGINE_ENDPOINT="https://:8000" aibrix_runtime --port 8080

curl -X POST https://:8080/v1/lora_adapter/load \
-H "Content-Type: application/json" \
-d '{"lora_name": "lora-2", "lora_path": "bharati2324/Qwen2.5-1.5B-Instruct-Code-LoRA-r16v2"}'

curl -X POST https://:8080/v1/lora_adapter/unload \
-H "Content-Type: application/json" \
-d '{"lora_name": "lora-1"}'

curl -X GET  https://:8000/v1/models | jq
{
    "object": "list",
    "data": [
        {
            "id": "Qwen/Qwen2.5-Coder-1.5B-Instruct",
            "object": "model",
            "created": 1738218097,
            "owned_by": "vllm",
            "root": "Qwen/Qwen2.5-Coder-1.5B-Instruct",
            "parent": null,
            "max_model_len": 32768,
            "permission": [
                {
                "id": "modelperm-c2e9860095b745b6b8be7133c5ab1fcf",
                "object": "model_permission",
                "created": 1738218097,
                "allow_create_engine": false,
                "allow_sampling": true,
                "allow_logprobs": true,
                "allow_search_indices": false,
                "allow_view": true,
                "allow_fine_tuning": false,
                "organization": "*",
                "group": null,
                "is_blocking": false
                }
            ]
        },
        {
            "id": "lora-1",
            "object": "model",
            "created": 1738218097,
            "owned_by": "vllm",
            "root": "bharati2324/Qwen2.5-1.5B-Instruct-Code-LoRA-r16v2",
            "parent": "Qwen/Qwen2.5-Coder-1.5B-Instruct",
            "max_model_len": null,
            "permission": [
                {
                "id": "modelperm-c21d06b59af0435292c70cd612e68b01",
                "object": "model_permission",
                "created": 1738218097,
                "allow_create_engine": false,
                "allow_sampling": true,
                "allow_logprobs": true,
                "allow_search_indices": false,
                "allow_view": true,
                "allow_fine_tuning": false,
                "organization": "*",
                "group": null,
                "is_blocking": false
                }
            ]
        },
        {
            "id": "lora-2",
            "object": "model",
            "created": 1738218097,
            "owned_by": "vllm",
            "root": "bharati2324/Qwen2.5-1.5B-Instruct-Code-LoRA-r16v2",
            "parent": "Qwen/Qwen2.5-Coder-1.5B-Instruct",
            "max_model_len": null,
            "permission": [
                {
                "id": "modelperm-bf2af850171242f7a9f4ccd9ecd313cd",
                "object": "model_permission",
                "created": 1738218097,
                "allow_create_engine": false,
                "allow_sampling": true,
                "allow_logprobs": true,
                "allow_search_indices": false,
                "allow_view": true,
                "allow_fine_tuning": false,
                "organization": "*",
                "group": null,
                "is_blocking": false
                }
            ]
        }
    ]
}