Deploy an Instant Cluster with Ray and vLLM

This tutorial shows how to use Instant Clusters with Ray to run distributed inference on large language models. By combining Ray’s cluster management with vLLM’s tensor and pipeline parallelism, you can serve models that exceed the memory of a single node — for example, a 70B parameter model across multiple 8×H100 pods. Ray handles the cluster topology; vLLM uses it to split the model across GPUs both within each node (tensor parallelism) and across nodes (pipeline parallelism).

Requirements

A Runpod account with sufficient credits for a multi-node cluster
Basic familiarity with large language model inference and distributed GPU setups

Step 1: Deploy an Instant Cluster

Open the Instant Clusters page.
Click Create Cluster.
Name your cluster and configure it. For this walkthrough, set Pod Count to 2 and select 8× H100 SXM GPUs per pod. Use the Runpod PyTorch template as your base image.
Increase /dev/shm when configuring your pod. The default (64 MB) is too small for large tensor-parallel workloads. Set it to at least 8 GB. In the pod configuration, add the environment variable MALLOC_ARENA_MAX=1 and set --shm-size to 8g in your Docker run options.
Click Deploy Cluster. You are redirected to the Instant Clusters page.

Step 2: Start the Ray head on pod-0

The first pod (CLUSTERNAME-pod-0) runs the Ray head node. All other pods connect to it as workers.

Click your cluster to expand the pod list.
Click CLUSTERNAME-pod-0, then click Connect → Web Terminal.

In the terminal, clone the reference scripts:

{/* [STO-463] Replace with canonical repo URL once scripts are published. */}
git clone https://github.com/runpod/ray-vllm-cluster.git

Run the head startup script:

bash ray-vllm-cluster/head.sh

The script sets the correct NIC address and starts Ray:

# head.sh (excerpt — see full script in the repo)
export RAY_NODE_IP_ADDRESS=$(hostname -I | awk '{print $1}')
export VLLM_HOST_IP=$RAY_NODE_IP_ADDRESS

ray start --head \
  --node-ip-address=$RAY_NODE_IP_ADDRESS \
  --port=6379 \
  --num-gpus=$NUM_TRAINERS

RAY_NODE_IP_ADDRESS and VLLM_HOST_IP must be set to the pod’s internal network IP — not 0.0.0.0. Setting them prevents Ray and vLLM from binding to the wrong interface on multi-NIC pods.

Step 3: Join the worker pods to the cluster

Repeat this for each remaining pod in the cluster (pod-1, pod-2, …).

In the Instant Clusters page, click the next pod and open its Web Terminal.

Clone the same scripts:

{/* [STO-463] Replace with canonical repo URL once scripts are published. */}
git clone https://github.com/runpod/ray-vllm-cluster.git

Run the worker startup script:

bash ray-vllm-cluster/worker.sh

The script waits for the head to be reachable, then joins:

# worker.sh (excerpt — see full script in the repo)
export RAY_NODE_IP_ADDRESS=$(hostname -I | awk '{print $1}')
export VLLM_HOST_IP=$RAY_NODE_IP_ADDRESS

# Wait for Ray head
until bash -c ">/dev/tcp/$MASTER_ADDR/6379" 2>/dev/null; do
  echo "Waiting for Ray head at $MASTER_ADDR:6379..."
  sleep 2
done

ray start \
  --address=$MASTER_ADDR:6379 \
  --node-ip-address=$RAY_NODE_IP_ADDRESS \
  --num-gpus=$NUM_TRAINERS

$MASTER_ADDR is injected automatically by Runpod into all pods in the cluster — it resolves to pod-0.

Step 4: Verify the cluster

Run this on pod-0 to confirm all nodes have joined:

ray status

Expected output for a two-pod cluster with 8 GPUs each:

======== Autoscaler status: ... ========
Node status
---------------------------------------------------------------
Healthy:
 2 node(s) with resources: {"GPU": 8.0, ...}

You can also open the Ray dashboard. The console shows the dashboard port in the Connect dialog for pod-0.

Step 5: Launch distributed inference with vLLM

Run this on pod-0 only. vLLM uses the Ray cluster that is already running.

bash ray-vllm-cluster/serve.sh

The script launches vLLM with tensor parallelism across GPUs within each node and pipeline parallelism across nodes:

# serve.sh (excerpt — see full script in the repo)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-70B \
  --tensor-parallel-size $NUM_TRAINERS \   # GPUs per node (e.g., 8)
  --pipeline-parallel-size $NUM_NODES \    # number of nodes (e.g., 2)
  --host 0.0.0.0 \
  --port 8000

--tensor-parallel-size should equal the number of GPUs per node ($NUM_TRAINERS). --pipeline-parallel-size should equal the number of nodes ($NUM_NODES). Both are injected as environment variables by Runpod.

vLLM connects to the running Ray cluster automatically. It may take several minutes to load model weights across all nodes.

Step 6: Test the endpoint

Once vLLM reports that it is ready, validate from pod-0:

# Check the server is healthy
curl http://localhost:8000/health

# Confirm the model is loaded
curl http://localhost:8000/v1/models

Then send a test request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-70B",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Step 7: Clean up

When you are done, return to the Instant Clusters page and delete your cluster. Leaving it running continues to incur charges.

Environment variables reference

Runpod injects these environment variables into every pod in the cluster. The startup scripts rely on them.

Variable	Description
`MASTER_ADDR`	Hostname of `pod-0`, the Ray head node
`MASTER_PORT`	Port for inter-node communication (default: `29500`)
`NUM_NODES`	Total number of pods in the cluster
`NUM_TRAINERS`	Number of GPUs per pod
`NODE_RANK`	Index of this pod (`0` for head, `1+` for workers)

Common issues

Ray workers don’t join Confirm $MASTER_ADDR resolves from each worker pod. Run ping $MASTER_ADDR in a worker terminal. If it fails, the cluster network may still be initializing — wait 30 seconds and try again. vLLM OOM during model load Check that /dev/shm is large enough (at least 8 GB for 70B models). Also verify that --tensor-parallel-size matches the number of GPUs per node — a mismatch causes uneven shard sizes. VLLM_HOST_IP binding error This error occurs when vLLM tries to bind to 0.0.0.0 on a pod with multiple network interfaces. Make sure VLLM_HOST_IP is set to the internal IP (hostname -I | awk '{print $1}') before starting the server. Stale Ray cluster after restart If you restart a pod, Ray does not automatically rejoin the cluster. Rerun head.sh on pod-0 first, then worker.sh on all other pods.

Next steps

Adapt the serve script to load your own model from a GlobalStore or Network Volume mount.
Scale up by increasing the pod count and adjusting --pipeline-parallel-size accordingly.
Try Axolotl on an Instant Cluster for distributed fine-tuning.
Review the Instant Cluster configuration reference for full details on environment variables and networking.

Get started

Flash

Serverless

Pods

Storage

Public Endpoints

Instant Clusters

Accounts and billing

Integrations

Hub

Reference

Deploy an Instant Cluster with Ray and vLLM

Requirements

Step 1: Deploy an Instant Cluster

Step 2: Start the Ray head on pod-0

Step 3: Join the worker pods to the cluster

Step 4: Verify the cluster

Step 5: Launch distributed inference with vLLM

Step 6: Test the endpoint

Step 7: Clean up

Environment variables reference

Common issues

Next steps

​Requirements

​Step 1: Deploy an Instant Cluster

​Step 2: Start the Ray head on pod-0

​Step 3: Join the worker pods to the cluster

​Step 4: Verify the cluster

​Step 5: Launch distributed inference with vLLM

​Step 6: Test the endpoint

​Step 7: Clean up

​Environment variables reference

​Common issues

​Next steps

Requirements

Step 1: Deploy an Instant Cluster

Step 2: Start the Ray head on pod-0

Step 3: Join the worker pods to the cluster

Step 4: Verify the cluster

Step 5: Launch distributed inference with vLLM

Step 6: Test the endpoint

Step 7: Clean up

Environment variables reference

Common issues

Next steps