Learn how to deploy LLMstudio as a containerized application on Google Kubernetes Engine and make calls from a local repository.

​
Prerequisites

To follow this guide you need to have the following set-up:

  • A project on google cloud platform.
  • Kubernetes Engine API enabled on your project.
  • Kubernetes Engine Admin role for the user performing the guide.

​
Deploy LLMstudio

This example demonstrates a public deployment. For a private service accessible only within your enterprise infrastructure, deploy it within your own Virtual Private Cloud (VPC).

1

Navigate to Kubernetes Engine

Begin by navigating to the Kubernetes Engine page.

2

Select Deploy

Go to Workloads and Create a new Deployment.

3

Name Your Deployment

Rename your project. We will call the one in this guide llmstudio-on-gcp.

4

Select Your Cluster

Choose between creating a new cluster or using an existing cluster. For this guide, we will create a new cluster and use the default region.

5

Proceed to Container Details

Once done done with the Deployment configuration, proceed to Container details.

6

Set Image Path

In the new container section, select Existing container image.

Copy the path to LLMstudio’s image available on Docker Hub.

Image Path
tensoropsai/llmstudio:latest

Set it as the Image path to your container.

7

Set Environment Variables

Configure the following mandatory environment variables:

Environment VariableValue
LLMSTUDIO_ENGINE_HOST0.0.0.0
LLMSTUDIO_ENGINE_PORT8001
LLMSTUDIO_TRACKING_HOST0.0.0.0
LLMSTUDIO_TRACKING_PORT8002

Additionally, set the GOOGLE_API_KEY environment variable to enable calls to Google’s Gemini models.

Refer to SDK/LLM/Providers for instructions on setting up other providers.

8

Proceed to Expose (Optional)

After configuring your container, proceed to Expose (Optional).

9

Expose Ports

Select Expose deployment as a new service and leave the first item as is.

Add two other items, and expose the ports defined in the Set Environment Variables step.

10

Deploy

After setting up and exposing the ports, press Deploy.

You have successfully deployed LLMstudio on Google Cloud Platform!

​
Make a Call

Now let’s make a call to our LLMstudio instance on GCP!

1

Set Up Project

Setup a simple project with this two files:

  1. calls.ipynb
  2. .env
2

Set Up Files

Go to your newly deployed Workload, scroll to the Exposing services section, and take note of the Host of your endpoint.

Create your .env file with the following:

.env
LLMSTUDIO_ENGINE_HOST = "YOUR_HOST"
LLMSTUDIO_ENGINE_PORT = "8001"
LLMSTUDIO_TRACKING_HOST = "YOUR_HOST"
LLMSTUDIO_TRACKING_PORT = "8002"
You are done seting up you .env file!
ChatDeploy on AWS Elastic Kubernetes Engine