Deploy on Google Kubernetes Engine

Navigate to Kubernetes Engine

Select Deploy

Go to Workloads and Create a new Deployment.

Name Your Deployment

Rename your project. We will call the one in this guide llmstudio-on-gcp.

Select Your Cluster

Choose between creating a new cluster or using an existing cluster. For this guide, we will create a new cluster and use the default region.

Proceed to Container Details

Set Image Path

In the new container section, select Existing container image.Copy the path to LLMstudio’s image available on Docker Hub.

Image Path

Copy

tensoropsai/llmstudio:latest

Set it as the Image path to your container.

Set Environment Variables

Configure the following mandatory environment variables:

Environment Variable	Value
LLMSTUDIO_ENGINE_HOST	0.0.0.0
LLMSTUDIO_ENGINE_PORT	8001
LLMSTUDIO_TRACKING_HOST	0.0.0.0
LLMSTUDIO_TRACKING_PORT	8002

Additionally, set the GOOGLE_API_KEY environment variable to enable calls to Google’s Gemini models.

Refer to SDK/LLM/Providers for instructions on setting up other providers.

Proceed to Expose (Optional)

Expose Ports

Select Expose deployment as a new service and leave the first item as is.

Add two other items, and expose the ports defined in the Set Environment Variables step.

Deploy

Set Up Project

Set Up Files

• .env
• calls.ipynb

Go to your newly deployed Workload, scroll to the Exposing services section, and take note of the Host of your endpoint.

Create your .env file with the following:

.env

Copy

LLMSTUDIO_ENGINE_HOST = "YOUR_HOST"
LLMSTUDIO_ENGINE_PORT = "8001"
LLMSTUDIO_TRACKING_HOST = "YOUR_HOST"
LLMSTUDIO_TRACKING_PORT = "8002"

You are done seting up you .env file!

Start by importing llmstudio:

1st cell

Copy

from llmstudio import LLM

Set up your LLM. We will be using gemini-1.5-flash for this guide.

2nd cell

Copy

llm = LLM('vertexai/gemini-1.5-flash')

Chat with your model.

3rd cell

Copy

llm.chat('Hello!')
print(response.chat_output)

You are done calling llmstudio on GCP!