> ## Documentation Index
> Fetch the complete documentation index at: https://runpod-b18f5ded-lg-ic-ray-vllm.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy an Instant Cluster with Ray and vLLM

> Run distributed inference across multiple nodes using Ray and vLLM on an Instant Cluster.

This tutorial shows how to use Instant Clusters with Ray to run distributed inference on large language models. By combining Ray's cluster management with vLLM's tensor and pipeline parallelism, you can serve models that exceed the memory of a single node — for example, a 70B parameter model across multiple 8×H100 pods.

Ray handles the cluster topology; vLLM uses it to split the model across GPUs both within each node (tensor parallelism) and across nodes (pipeline parallelism).

***

## Requirements

* A Runpod account with sufficient credits for a multi-node cluster
* Basic familiarity with large language model inference and distributed GPU setups

***

## Step 1: Deploy an Instant Cluster

1. Open the [Instant Clusters page](https://console.runpod.io/instant-clusters).

2. Click **Create Cluster**.

3. Name your cluster and configure it. For this walkthrough, set **Pod Count** to **2** and select **8× H100 SXM GPUs** per pod. Use the **Runpod PyTorch** template as your base image.

   <Note>
     Increase `/dev/shm` when configuring your pod. The default (64 MB) is too small for large tensor-parallel workloads. Set it to at least 8 GB. In the pod configuration, add the environment variable `MALLOC_ARENA_MAX=1` and set `--shm-size` to `8g` in your Docker run options.
   </Note>

4. Click **Deploy Cluster**. You are redirected to the Instant Clusters page.

***

## Step 2: Start the Ray head on pod-0

The first pod (`CLUSTERNAME-pod-0`) runs the Ray head node. All other pods connect to it as workers.

1. Click your cluster to expand the pod list.

2. Click **CLUSTERNAME-pod-0**, then click **Connect → Web Terminal**.

3. In the terminal, clone the reference scripts:

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
   {/* [STO-463] Replace with canonical repo URL once scripts are published. */}
   git clone https://github.com/runpod/ray-vllm-cluster.git
   ```

4. Run the head startup script:

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
   bash ray-vllm-cluster/head.sh
   ```

   The script sets the correct NIC address and starts Ray:

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
   # head.sh (excerpt — see full script in the repo)
   export RAY_NODE_IP_ADDRESS=$(hostname -I | awk '{print $1}')
   export VLLM_HOST_IP=$RAY_NODE_IP_ADDRESS

   ray start --head \
     --node-ip-address=$RAY_NODE_IP_ADDRESS \
     --port=6379 \
     --num-gpus=$NUM_TRAINERS
   ```

   <Note>
     `RAY_NODE_IP_ADDRESS` and `VLLM_HOST_IP` must be set to the pod's internal network IP — not `0.0.0.0`. Setting them prevents Ray and vLLM from binding to the wrong interface on multi-NIC pods.
   </Note>

***

## Step 3: Join the worker pods to the cluster

Repeat this for each remaining pod in the cluster (`pod-1`, `pod-2`, …).

1. In the Instant Clusters page, click the next pod and open its **Web Terminal**.

2. Clone the same scripts:

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
   {/* [STO-463] Replace with canonical repo URL once scripts are published. */}
   git clone https://github.com/runpod/ray-vllm-cluster.git
   ```

3. Run the worker startup script:

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
   bash ray-vllm-cluster/worker.sh
   ```

   The script waits for the head to be reachable, then joins:

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
   # worker.sh (excerpt — see full script in the repo)
   export RAY_NODE_IP_ADDRESS=$(hostname -I | awk '{print $1}')
   export VLLM_HOST_IP=$RAY_NODE_IP_ADDRESS

   # Wait for Ray head
   until bash -c ">/dev/tcp/$MASTER_ADDR/6379" 2>/dev/null; do
     echo "Waiting for Ray head at $MASTER_ADDR:6379..."
     sleep 2
   done

   ray start \
     --address=$MASTER_ADDR:6379 \
     --node-ip-address=$RAY_NODE_IP_ADDRESS \
     --num-gpus=$NUM_TRAINERS
   ```

   `$MASTER_ADDR` is injected automatically by Runpod into all pods in the cluster — it resolves to `pod-0`.

***

## Step 4: Verify the cluster

Run this on `pod-0` to confirm all nodes have joined:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
ray status
```

Expected output for a two-pod cluster with 8 GPUs each:

```
======== Autoscaler status: ... ========
Node status
---------------------------------------------------------------
Healthy:
 2 node(s) with resources: {"GPU": 8.0, ...}
```

You can also open the Ray dashboard. The console shows the dashboard port in the **Connect** dialog for `pod-0`.

***

## Step 5: Launch distributed inference with vLLM

Run this on `pod-0` only. vLLM uses the Ray cluster that is already running.

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bash ray-vllm-cluster/serve.sh
```

The script launches vLLM with tensor parallelism across GPUs within each node and pipeline parallelism across nodes:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# serve.sh (excerpt — see full script in the repo)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-70B \
  --tensor-parallel-size $NUM_TRAINERS \   # GPUs per node (e.g., 8)
  --pipeline-parallel-size $NUM_NODES \    # number of nodes (e.g., 2)
  --host 0.0.0.0 \
  --port 8000
```

<Tip>
  `--tensor-parallel-size` should equal the number of GPUs per node (`$NUM_TRAINERS`). `--pipeline-parallel-size` should equal the number of nodes (`$NUM_NODES`). Both are injected as environment variables by Runpod.
</Tip>

vLLM connects to the running Ray cluster automatically. It may take several minutes to load model weights across all nodes.

***

## Step 6: Test the endpoint

Once vLLM reports that it is ready, validate from `pod-0`:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Check the server is healthy
curl http://localhost:8000/health

# Confirm the model is loaded
curl http://localhost:8000/v1/models
```

Then send a test request:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-70B",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

***

## Step 7: Clean up

When you are done, return to the [Instant Clusters page](https://console.runpod.io/instant-clusters) and delete your cluster. Leaving it running continues to incur charges.

***

## Environment variables reference

Runpod injects these environment variables into every pod in the cluster. The startup scripts rely on them.

| Variable       | Description                                          |
| -------------- | ---------------------------------------------------- |
| `MASTER_ADDR`  | Hostname of `pod-0`, the Ray head node               |
| `MASTER_PORT`  | Port for inter-node communication (default: `29500`) |
| `NUM_NODES`    | Total number of pods in the cluster                  |
| `NUM_TRAINERS` | Number of GPUs per pod                               |
| `NODE_RANK`    | Index of this pod (`0` for head, `1+` for workers)   |

***

## Common issues

**Ray workers don't join**
Confirm `$MASTER_ADDR` resolves from each worker pod. Run `ping $MASTER_ADDR` in a worker terminal. If it fails, the cluster network may still be initializing — wait 30 seconds and try again.

**vLLM OOM during model load**
Check that `/dev/shm` is large enough (at least 8 GB for 70B models). Also verify that `--tensor-parallel-size` matches the number of GPUs per node — a mismatch causes uneven shard sizes.

**`VLLM_HOST_IP` binding error**
This error occurs when vLLM tries to bind to `0.0.0.0` on a pod with multiple network interfaces. Make sure `VLLM_HOST_IP` is set to the internal IP (`hostname -I | awk '{print $1}'`) before starting the server.

**Stale Ray cluster after restart**
If you restart a pod, Ray does not automatically rejoin the cluster. Rerun `head.sh` on `pod-0` first, then `worker.sh` on all other pods.

***

## Next steps

* Adapt the serve script to load your own model from a [GlobalStore](/storage/globalstore) or [Network Volume](/storage/network-volumes) mount.
* Scale up by increasing the pod count and adjusting `--pipeline-parallel-size` accordingly.
* Try [Axolotl on an Instant Cluster](/instant-clusters/axolotl) for distributed fine-tuning.
* Review the [Instant Cluster configuration reference](/instant-clusters/configuration) for full details on environment variables and networking.