批处理 API

批量API#

AIBrix批量API提供了一种高效的方式，可以异步处理大量LLM推理请求。它与OpenAI批量API完全兼容，允许您发送批量请求，这些请求在后台处理，结果随后检索。

概述#

批量处理非常适用于不需要即时响应且可从以下方面受益的工作负载：

  • 成本效益：在非高峰时段处理请求，优化资源利用

  • 更高吞吐量：处理大量请求而无需担心速率限制

  • 简化工作流程：在单个批量操作中提交数千个请求

  • 保证处理：内置重试机制和故障处理

批量API接受一个JSONL（JSON Lines）文件，其中包含多个推理请求，使用Kubernetes Jobs异步处理它们，并在相应的JSONL输出文件中返回结果。

主要特点#

  • OpenAI兼容：OpenAI批量API的直接替代品，具有相同的请求/响应格式

  • 分布式执行：利用Kubernetes Jobs实现可扩展、容错的批量处理

  • 元数据服务器工作流程：多节点批量执行的集中协调

  • 存储灵活性：支持S3、Redis和本地存储后端

  • 请求跟踪：每个请求都有一个custom_id用于精确的结果匹配

  • 状态监控：实时进度跟踪，包含详细指标

  • 24小时完成窗口：长时间运行批次的自动过期

与OpenAI批量API的对比#

AIBrix批量API与OpenAI批量API完全兼容，同时增加了企业级功能

AIBrix增强功能

  • 自托管：完全控制基础设施和数据

  • Kubernetes原生：利用K8s进行调度和资源管理

  • 灵活存储：S3、Redis或本地存储后端

  • 分布式执行：元数据服务器协调多节点处理

  • 成本控制：使用现有基础设施，无需按请求计费

从OpenAI迁移

只需更新OpenAI SDK配置中的base_url

from openai import OpenAI

# Before: OpenAI
# client = OpenAI(api_key="sk-...")

# After: AIBrix
client = OpenAI(
    base_url="http://your-aibrix-endpoint/v1",
    api_key="your-key"  # Optional
)

# All batch API calls work identically

已知差异

  • 定价：无基于使用的定价；由您的基础设施成本控制

  • 端点：目前支持/v1/chat/completions（更多即将推出）

  • 速率限制：由您的集群容量决定，而非API限制

支持的端点

  • /v1/chat/completions - 聊天补全请求

  • /v1/completions - 文本补全请求

  • /v1/embeddings - 嵌入生成请求

如果底层LLM引擎支持，所有端点都将完全支持。

当前限制

  • 24小时完成窗口（不可配置）

  • 不支持Moderation端点（/v1/moderations）

架构#

工作流程概述#

批量API遵循元数据服务器架构进行分布式处理

┌──────────┐
│  Client  │
└────┬─────┘
     │ 1. Upload JSONL file
     ▼
┌─────────────────┐
│   Files API     │
│   (Metadata)    │
└────┬────────────┘
     │ 2. Create batch job
     ▼
┌──────────────────┐         ┌──────────────────┐
│  Batch API       │────────▶│  Job Scheduler   │
│  (Metadata)      │         │  (Kubernetes)    │
└────┬─────────────┘         └────┬─────────────┘
     │                             │ 3. Execute workers
     │ 4. Poll status              ▼
     │                        ┌─────────────┐
     │                        │  K8s Jobs   │
     │                        │  (Workers)  │
     │                        └────┬────────┘
     │                             │ 5. Process requests
     │                             │    & write outputs
     │ 6. Download output          │
     ▼                             ▼
┌──────────────────┐         ┌──────────────┐
│  Files API       │◀────────│   Storage    │
│  (Metadata)      │         │  (S3/Redis)  │
└──────────────────┘         └──────────────┘

阶段转换

validating → in_progress → finalizing → completed
    ↓            ↓             ↓            ↓
Preparing    Worker         Collecting  Results
job files    execution      outputs      ready

状态生命周期

  1. validating：元数据服务器验证输入文件并准备作业配置

  2. in_progress：Kubernetes Jobs正在执行并处理批量请求

  3. finalizing：工作已完成，元数据服务器正在聚合结果

  4. completed：输出文件已准备好下载所有结果

失败/取消状态

  • failed：作业执行遇到不可恢复的错误

  • cancelled：用户明确取消了批量作业

  • expired：作业超过了24小时完成窗口

组件#

  1. 元数据服务器：协调批量作业生命周期，管理文件并跟踪进度

  2. 作业调度器：为批量执行创建和管理Kubernetes Jobs

  3. 工作作业：并行处理批量请求的Kubernetes Jobs

  4. 存储后端：S3、Redis或本地文件系统用于文件存储和作业状态

  5. 文件API：OpenAI兼容的文件上传/下载端点

部署#

存储后端配置#

批量API需要一个存储后端进行文件操作。AIBrix支持多种存储后端，包括S3、TOS和本地存储。要启用云对象存储，您需要配置凭据并启用适当的存储补丁。

启用S3存储

要启用S3作为批量操作的存储后端

  1. 生成S3凭据秘密

使用AIBrix秘密生成工具创建必要的Kubernetes秘密

# Install the AIBrix package in development mode
cd python/aibrix && pip install -e .

# Generate S3 credentials secret
aibrix_gen_secrets s3 --bucket your-s3-bucket-name --namespace aibrix-system

# Generate S3 credentials secret for Job Executor
aibrix_gen_secrets s3 --bucket your-s3-bucket-name --namespace default

此命令将

  • 在aibrix-system命名空间中创建一个名为aibrix-s3-credentials的Kubernetes秘密

  • 使用您的S3存储桶名称和凭据配置秘密

  • 为元数据服务设置必要的环境变量

  1. 启用S3环境变量

取消元数据服务配置中的S3补丁注释

# Edit the kustomization file
vim config/metadata/kustomization.yaml

找到并取消以下行的注释

patches:
- path: s3-env-patch.yaml  # Uncomment this line

该补丁会将S3环境变量注入到元数据服务部署中。

  1. 应用配置

部署作业rbac和更新的配置

kubectl apply -k config/job
kubectl apply -k config/default

启用TOS存储

对于TOS（腾讯对象存储），请遵循类似步骤

  1. 生成TOS凭据秘密

# Install the AIBrix package in development mode
cd python/aibrix && pip install -e .

# Generate TOS credentials secret
aibrix_gen_secrets tos --bucket your-tos-bucket-name --namespace aibrix-system

# Generate TOS credentials secret for Job Executor
aibrix_gen_secrets tos --bucket your-tos-bucket-name --namespace default

  1. 启用TOS环境变量

取消元数据服务配置中的TOS补丁注释

# Edit the kustomization file
vim config/metadata/kustomization.yaml

找到并取消以下行的注释

patches:
- path: tos-env-patch.yaml  # Uncomment this line

该补丁会将TOS环境变量注入到元数据服务部署中。

  1. 应用配置

部署作业rbac和更新的配置

kubectl apply -k config/job
kubectl apply -k config/default

示例#

端到端示例#

这是一个处理一批聊天补全的完整示例

步骤1：准备输入文件

使用您的请求创建文件batch_input.jsonl

{"custom_id": "task-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain neural networks."}], "max_tokens": 200}}
{"custom_id": "task-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is deep learning?"}], "max_tokens": 200}}
{"custom_id": "task-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Describe transformers architecture."}], "max_tokens": 200}}

步骤2：上传输入文件

# Upload the input file
ENDPOINT="your-aibrix-endpoint:80"

UPLOAD_RESPONSE=$(curl -X POST http://${ENDPOINT}/v1/files \
  -F "purpose=batch" \
  -F "file=@batch_input.jsonl")

echo $UPLOAD_RESPONSE
# {"id":"file-abc123","object":"file","bytes":1024,"created_at":1677610602,"filename":"batch_input.jsonl","purpose":"batch","status":"uploaded"}

# Extract file ID
FILE_ID=$(echo $UPLOAD_RESPONSE | jq -r '.id')
echo "Uploaded file ID: $FILE_ID"

步骤3：创建批量作业

# Create batch job
BATCH_RESPONSE=$(curl -X POST http://${ENDPOINT}/v1/batches \
  -H "Content-Type: application/json" \
  -d "{
    \"input_file_id\": \"${FILE_ID}\",
    \"endpoint\": \"/v1/chat/completions\",
    \"completion_window\": \"24h\"
  }")

echo $BATCH_RESPONSE

# Extract batch ID
BATCH_ID=$(echo $BATCH_RESPONSE | jq -r '.id')
echo "Created batch ID: $BATCH_ID"

步骤4：轮询批量状态

# Poll until completion (with timeout)
MAX_ATTEMPTS=60
ATTEMPT=0

while [ $ATTEMPT -lt $MAX_ATTEMPTS ]; do
  STATUS_RESPONSE=$(curl -s http://${ENDPOINT}/v1/batches/${BATCH_ID})
  STATUS=$(echo $STATUS_RESPONSE | jq -r '.status')

  echo "Attempt $ATTEMPT: Status = $STATUS"

  if [ "$STATUS" = "completed" ]; then
    echo "Batch completed successfully!"
    OUTPUT_FILE_ID=$(echo $STATUS_RESPONSE | jq -r '.output_file_id')
    break
  elif [ "$STATUS" = "failed" ] || [ "$STATUS" = "expired" ] || [ "$STATUS" = "cancelled" ]; then
    echo "Batch processing failed with status: $STATUS"
    exit 1
  fi

  ATTEMPT=$((ATTEMPT + 1))
  sleep 10
done

if [ $ATTEMPT -eq $MAX_ATTEMPTS ]; then
  echo "Batch did not complete within timeout"
  exit 1
fi

步骤5：下载结果

# Download output file
curl -o batch_output.jsonl http://${ENDPOINT}/v1/files/${OUTPUT_FILE_ID}/content

echo "Output saved to batch_output.jsonl"

# Display results
cat batch_output.jsonl | jq '.'

步骤6：处理结果

import json

# Parse output file
results = {}
with open('batch_output.jsonl', 'r') as f:
    for line in f:
        output = json.loads(line)
        custom_id = output['custom_id']
        response = output['response']

        if response['status_code'] == 200:
            content = response['body']['choices'][0]['message']['content']
            results[custom_id] = content
            print(f"{custom_id}: {content[:100]}...")
        else:
            print(f"{custom_id}: ERROR {response['status_code']}")

# Output:
# task-1: Neural networks are computational models inspired by biological neurons...
# task-2: Deep learning is a subset of machine learning that uses multi-layer...
# task-3: The Transformer architecture is a neural network design that relies...

Python SDK示例#

使用OpenAI Python SDK（与AIBrix作为直接替代品兼容）

import json
import time
from openai import OpenAI

# Configure client for AIBrix
client = OpenAI(
    base_url="http://your-aibrix-endpoint:80/v1",
    api_key="dummy-key"  # Replace with actual key if authentication is enabled
)

# Step 1: Create batch input file
batch_requests = [
    {
        "custom_id": f"request-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-oss-120b",
            "messages": [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": f"Tell me a fact about the number {i}."}
            ],
            "max_tokens": 100
        }
    }
    for i in range(1, 11)  # 10 requests
]

# Write to JSONL file
with open("batch_requests.jsonl", "w") as f:
    for request in batch_requests:
        f.write(json.dumps(request) + "\n")

# Step 2: Upload file
with open("batch_requests.jsonl", "rb") as f:
    batch_file = client.files.create(
        file=f,
        purpose="batch"
    )

print(f"Uploaded file: {batch_file.id}")

# Step 3: Create batch
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

print(f"Created batch: {batch.id}")

# Step 4: Wait for completion
while batch.status not in ["completed", "failed", "expired", "cancelled"]:
    time.sleep(10)
    batch = client.batches.retrieve(batch.id)
    print(f"Status: {batch.status}")

if batch.status == "completed":
    print(f"Batch completed!")
    print(f"Total requests: {batch.request_counts.total}")
    print(f"Completed: {batch.request_counts.completed}")
    print(f"Failed: {batch.request_counts.failed}")

    # Step 5: Download results
    output_file_id = batch.output_file_id
    result_content = client.files.content(output_file_id)

    # Save results
    with open("batch_results.jsonl", "wb") as f:
        f.write(result_content.content)

    # Process results
    with open("batch_results.jsonl", "r") as f:
        for line in f:
            result = json.loads(line)
            custom_id = result["custom_id"]
            content = result["response"]["body"]["choices"][0]["message"]["content"]
            print(f"{custom_id}: {content}")
else:
    print(f"Batch failed with status: {batch.status}")

定制#

定制作业执行器#

您可以通过修改作业模板补丁配置来定制批量作业执行环境。这允许您指定自定义容器镜像、资源要求和其他Kubernetes Job规范。

作业模板补丁配置

作业执行器行为由config/metadata/job_template_patch.yaml文件控制。此文件定义了将用于批量处理的Kubernetes Job模板

apiVersion: batch/v1
kind: Job
metadata:
  name: batch-job-template
  namespace: default
spec:
  parallelism: 1 # Customizable. The number of parallel workers.
  completions: 1 # Customizable. Must equal to the parallelism.
  backoffLimit: 2 # Customizable, but usually no need to change.
  template:
    spec:
      containers:
      - name: batch-worker
        image: aibrix/runtime:nightly # Customizable, runtime image
      - name: llm-engine
        image: aibrix/vllm-mock:nightly # Customizable, LLM engine image

定制选项

  • parallelism：并行worker pod的数量（影响吞吐量）

  • completions：必须与parallelism匹配才能正确完成作业

  • backoffLimit：失败worker pod的重试次数

  • batch-worker image：协调批量处理的运行时容器

  • llm-engine image：LLM推理引擎容器（例如vLLM、TensorRT-LLM）

常见定制

  1. 使用自定义LLM引擎

    containers:
- name: llm-engine
  image: your-registry/custom-vllm:latest

  2. 增加并行度

    spec:
  parallelism: 4
  completions: 4

  3. 添加资源要求

    containers:
- name: llm-engine
  image: aibrix/vllm-mock:nightly
  resources:
    requests:
      nvidia.com/gpu: 1
      memory: "8Gi"
    limits:
      nvidia.com/gpu: 1
      memory: "16Gi"

  4. 添加环境变量

    containers:
- name: llm-engine
  image: aibrix/vllm-mock:nightly
  env:
  - name: CUDA_VISIBLE_DEVICES
    value: "0"
  - name: MODEL_PATH
    value: "/models/your-model"

应用更改

修改job_template_patch.yaml后，使用以下命令应用更改

kubectl apply -k config/default

配置作业池大小#

批量作业调度器使用作业池来控制并发批量处理。池大小可以通过环境变量配置

# Set job pool size (default: 1, min: 1, max: 100)
export AIBRIX_BATCH_JOB_POOL_SIZE=10

配置选项

  • 默认值：1（保守，防止资源争用）

  • 推荐值：5-20（取决于集群资源）

  • 最大值：100（运行时验证）

部署中的设置

添加到元数据服务部署环境变量

env:
- name: AIBRIX_BATCH_JOB_POOL_SIZE
  value: "10"

验证与测试#

验证批量API功能#

按照以下步骤验证批量API在您的AIBrix部署中是否正常工作

步骤1：设置端口转发

首先，创建端口转发以访问AIBrix服务

# Port-forward the gateway service to access AIBrix APIs
kubectl -n envoy-gateway-system port-forward service/envoy-aibrix-system-aibrix-eg-903790dc 8888:80 1>/dev/null 2>&1 &

# Verify the port-forward is working
curl -s https://:8888/v1/batches

步骤2：设置对象存储凭据

配置用于批量文件存储的S3凭据

# Navigate to the Python package directory
cd python/aibrix

# Install the AIBrix package in development mode
pip install -e .

# Generate S3 credentials secret (replace with your S3 bucket)
aibrix_gen_secrets s3 --bucket your-s3-bucket-name

# Example with specific bucket:
# aibrix_gen_secrets s3 --bucket my-aibrix-batch-storage

此命令将

  • 为S3访问创建必要的Kubernetes秘密

  • 配置元数据服务以使用您的S3存储桶进行文件存储

  • 为批量作业文件操作设置正确的IAM凭据

步骤3：运行端到端测试

执行全面的批量API测试套件

# Navigate to the Python package directory (if not already there)
cd python/aibrix

# Run the batch API end-to-end tests
pytest tests/e2e/test_batch_api.py -v

预期测试输出

tests/e2e/test_batch_api.py::test_batch_api_e2e_real_service PASSED

========================= 1 passed in 10.78s =========================

测试覆盖率

测试套件验证

  • 文件上传/下载：带有S3后端的Files API功能

  • 批量作业创建：正确的批量作业提交和验证

  • Kubernetes作业执行：Worker pod的创建和执行

  • 状态监控：实时批量状态跟踪

  • 结果收集：输出文件生成和检索

常见问题排查

  1. 端口转发连接问题

    # Check if port-forward is running
ps aux | grep port-forward

# Kill existing port-forwards and restart
pkill -f "port-forward.*8888"
kubectl -n envoy-gateway-system port-forward service/envoy-aibrix-system-aibrix-eg-903790dc 8888:80 &

  2. S3凭据问题

    # Verify S3 secret was created
kubectl get secret aibrix-s3-credentials -n aibrix-system

# Check secret contents
kubectl get secret aibrix-s3-credentials -n aibrix-system -o yaml

  3. 测试失败

    # Run tests with more verbose output
pytest tests/e2e/test_batch_api.py -v -s --tb=long

手动验证

您也可以手动验证批量API，如上面示例部分所示，使用curl命令，在设置端口转发后，将localhost:8888作为您的端点。

API参考#

文件API#

文件API管理批量处理的输入和输出文件。ENDPOINT是元数据服务端点。

kubectl port-forward svc/aibrix-metadata-service 8090:8090 -n aibrix-system
export ENDPOINT=localhost:8090

上传文件

curl -X POST http://${ENDPOINT}/v1/files \
  -F "purpose=batch" \
  -F "file=@batch_input.jsonl"

{
  "id": "102983c4-92ef-4de9-a03b-8e05066b16fd",
  "object": "file",
  "bytes": 3104,
  "created_at": 1677610602,
  "filename": "batch_input.jsonl",
  "purpose": "batch",
  "status": "uploaded"
}

获取文件元数据

curl -X GET http://${ENDPOINT}/v1/files/{file_id}

{
  "id": "102983c4-92ef-4de9-a03b-8e05066b16fd",
  "object": "file",
  "bytes": 3104,
  "created_at": 1760131968,
  "filename": "batch_input.jsonl",
  "purpose": "batch",
  "status": "uploaded",
  "content_type": "application/octet-stream",
  "etag": "e64b86a757f6b6e3bbbe65387158d47a",
  "last_modified": 1760131968
}

列出文件

列出所有文件，可选择过滤和分页

# List all files
curl -X GET http://${ENDPOINT}/v1/files

# List only batch files
curl -X GET "http://${ENDPOINT}/v1/files?purpose=batch"

# List with pagination (limit and cursor)
curl -X GET "http://${ENDPOINT}/v1/files?limit=10&after=file-abc123"

响应

{
  "object": "list",
  "data": [
    {
      "id": "102983c4-92ef-4de9-a03b-8e05066b16fd",
      "object": "file",
      "bytes": 3104,
      "created_at": 1760131968,
      "filename": "batch_input.jsonl",
      "purpose": "batch",
      "status": "uploaded"
    }
  ],
  "has_more": false
}

查询参数

  • purpose（可选）：按文件用途过滤（例如，“batch”）

  • limit（可选）：返回的文件数量（1-100，默认20）

  • after（可选）：用作分页游标的文件ID

下载文件

curl -X GET http://${ENDPOINT}/v1/files/{file_id}/content

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Explain quantum computing in simple terms."}],"max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "system", "content": "You are a creative writing assistant."},{"role": "user", "content": "Write a short story about a robot discovering emotions."}],"max_tokens": 1000}}
{"custom_id": "request-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "system", "content": "You are a code reviewer."},{"role": "user", "content": "Review this Python function: def fibonacci(n): return n if n <= 1 else fibonacci(n-1) + fibonacci(n-2)"}],"max_tokens": 1000}}
{"custom_id": "request-4", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "system", "content": "You are a cooking instructor."},{"role": "user", "content": "How do I make perfect scrambled eggs?"}],"max_tokens": 1000}}
{"custom_id": "request-5", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "system", "content": "You are a travel advisor."},{"role": "user", "content": "What are the top 5 must-see attractions in Tokyo for first-time visitors?"}],"max_tokens": 1000}}
...

响应：原始文件内容（JSONL格式）

批量API#

批量API管理批量作业生命周期。

创建批量

curl -X POST http://${ENDPOINT}/v1/batches \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "102983c4-92ef-4de9-a03b-8e05066b16fd",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

{
  "id": "6f646d68-1314-42f9-907b-b50a88061a9f",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "errors": null,
  "input_file_id": "102983c4-92ef-4de9-a03b-8e05066b16fd",
  "completion_window": "24h",
  "status": "created",
  "output_file_id": null,
  "error_file_id": null,
  "created_at": 1760132899,
  "in_progress_at": null,
  "expires_at": 1760219299,
  "finalizing_at": null,
  "completed_at": null,
  "failed_at": null,
  "expired_at": null,
  "cancelling_at": null,
  "cancelled_at": null,
  "request_counts": null,
  "metadata": null
}

获取批量状态

curl -X GET http://${ENDPOINT}/v1/batches/{batch_id}

{
  "id": "6f646d68-1314-42f9-907b-b50a88061a9f",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "errors": null,
  "input_file_id": "102983c4-92ef-4de9-a03b-8e05066b16fd",
  "completion_window": "24h",
  "status": "completed",
  "output_file_id": "4d4c4f0d-43e2-3a76-8c44-06b95b5afc08",
  "error_file_id": "eca1882e-5bf2-3c23-9b03-f54f98558302",
  "created_at": 1760132899,
  "in_progress_at": 1760132899,
  "expires_at": 1760219299,
  "finalizing_at": 1760132909,
  "completed_at": 1760132909,
  "failed_at": null,
  "expired_at": null,
  "cancelling_at": null,
  "cancelled_at": null,
  "request_counts": {
    "total": 10,
    "completed": 10,
    "failed": 0
  },
  "metadata": null
}

列出批量

curl -X GET http://${ENDPOINT}/v1/batches

{
  "object": "list",
  "data": [
    {
      "id": "6f646d68-1314-42f9-907b-b50a88061a9f",
      "object": "batch",
      "endpoint": "/v1/chat/completions",
      "errors": null,
      "input_file_id": "102983c4-92ef-4de9-a03b-8e05066b16fd",
      "completion_window": "24h",
      "status": "completed",
      "output_file_id": "4d4c4f0d-43e2-3a76-8c44-06b95b5afc08",
      "error_file_id": "eca1882e-5bf2-3c23-9b03-f54f98558302",
      "created_at": 1760132899,
      "in_progress_at": 1760132899,
      "expires_at": 1760219299,
      "finalizing_at": 1760132909,
      "completed_at": 1760132909,
      "failed_at": null,
      "expired_at": null,
      "cancelling_at": null,
      "cancelled_at": null,
      "request_counts": {
        "total": 10,
        "completed": 10,
        "failed": 0
      },
      "metadata": null
    }
  ],
  "first_id": "6f646d68-1314-42f9-907b-b50a88061a9f",
  "last_id": "6f646d68-1314-42f9-907b-b50a88061a9f",
  "has_more": false
}

输入文件格式#

输入文件必须是JSONL格式，每行一个请求。每个请求需要

  • custom_id：用于匹配结果的唯一标识符（必需）

  • method：HTTP方法，通常为“POST”（必需）

  • url：端点路径，例如“/v1/chat/completions”（必需）

  • body：与端点格式匹配的请求负载（必需）

示例 batch_input.jsonl

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is AI?"}], "max_tokens": 100}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain quantum computing."}], "max_tokens": 150}}
{"custom_id": "request-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is machine learning?"}], "max_tokens": 100}}

输出文件格式#

输出文件采用JSONL格式，每行一个结果，与每个输入请求匹配

示例 batch_output.jsonl

{"id": "batch-def456-0", "custom_id": "request-1", "response": {"status_code": 200, "request_id": "req_001", "body": {"id": "chatcmpl-001", "object": "chat.completion", "created": 1677610602, "model": "gpt-3.5-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "AI stands for Artificial Intelligence..."}, "finish_reason": "stop"}]}}}
{"id": "batch-def456-1", "custom_id": "request-2", "response": {"status_code": 200, "request_id": "req_002", "body": {"id": "chatcmpl-002", "object": "chat.completion", "created": 1677610603, "model": "gpt-3.5-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Quantum computing uses quantum mechanics..."}, "finish_reason": "stop"}]}}}
{"id": "batch-def456-2", "custom_id": "request-3", "response": {"status_code": 200, "request_id": "req_003", "body": {"id": "chatcmpl-003", "object": "chat.completion", "created": 1677610604, "model": "gpt-3.5-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Machine learning is a subset of AI..."}, "finish_reason": "stop"}]}}}

每行输出包含

  • id：唯一的1结果标识符

  • custom_id：与输入请求的custom_id匹配

  • response：包含status_code、request_id和带有实际结果的body