How to Select the Right GPU for Open Source LLMs?

Deploying and operating an open-source Large Language Model (LLM) requires careful planning when selecting the right GPU model and memory capacity. Choosing the optimal configuration is crucial for performance, cost efficiency, and scalability. However, this process comes with several challenges.

In this blog, we will describe the factors that you need to consider to select the optimal GPU model for your LLM. We have also published a table capturing optimal GPU models to deploy and use Top-10 open source LLMs.

How many GPUs

Developer Self Service Access to DeepSeek on Amazon EKS

A few weeks back, Tiago Reichert from AWS published a very interesting blog on AWS Community showcasing how you can deploy and use the DeepSeek-R1 LLM on an Amazon EKS Cluster operating in Auto Mode. Detailed step-by-step instructions for this are documented in this Git Repo.

In this blog, we will describe how we took AWS's excellent blog and packaged it to provide a turnkey, 1-click self-service experience for non AWS administrator type users in a typical enterprise. It took one of our solution architects 30 minutes to wrap AWS's example code using Rafay's Environment Manager and PaaS.

Over the last few weeks, we have been asked to demonstrate this every day to several customers and partners. Given the significant interest in DeepSeek and the self service experience, we believe others will benefit from this blog.

Rafay, DeepSeek and EKS

Using Hubble and Cilium with Rafay MKS based Kubernetes Cluster for Data Centers

In our first blog about Hubble for Cilium, we reviewed a real life example highlighting where traditional monitoring tools fall short. We then looked at how Hubble + Cilium can address these gaps. In the second blog, we discussed how Rafay provides our customers with a a tight, turnkey integration with Cilium for various cluster types (i.e. Rafay MKS for Data Centers and Public Cloud Distributions such as Amazon EKS).

In this get started guide, we will review how a platform engineer can configure, deploy and use Hubble for Cilium on a Rafay MKS Kubernetes cluster operating in a data center (aka on-premises environment). The three high level steps are:

  1. Provision an Upstream Kubernetes Cluster in your data center using Rafay MKS
  2. Configure and Deploy Cilium CNI as a software add-on in a Cluster Blueprint (i.e. Bring Your Own CNI)
  3. Use Hubble to observe network flows

Hubble Intro

Turnkey Integration with Cilium CNI

In the first blog, we discussed how organizations can use Hubble for Cilium for observability. In this blog, we will look at how the Rafay Platform provides a tight, turnkey integration with Cilium making life easy for platform teams. In the next blog, my colleague will describe and showcase how an administrator can configure and enable Hubble on a Rafay MKS based Kubernetes cluster with the Cilium CNI.

Rafay + Cilium

Supercharge Kubernetes Networking Observability using Hubble and Cilium

Networking observability in Kubernetes environments is essential for troubleshooting, security, and performance optimization. Hubble, an observability platform for the Cilium CNI, addresses this challenge by providing real-time insights into network traffic, security policies, and application-layer interactions. Hubble is built on eBPF (Extended Berkeley Packet Filter) and provides deep visibility into packet flows, service-to-service communication, and security enforcement without requiring intrusive packet mirroring or modifications to application code. In a nutshell, Hubble is a fully distributed networking and security observability platform for cloud native workloads.

In this introductory blog about Hubble for Cilium, We will start with a real life example highlighting where traditional monitoring tools fall short. We will then look at how Hubble + Cilium can address these gaps. In the second blog, I will describe how Rafay provides our customers with a a tight, turnkey integration with Cilium for various cluster types (i.e. Rafay MKS for Data Centers and Public Cloud Distributions such as Amazon EKS).

Hubble Intro

Encrypt your Kubernetes Backups using Server Side Encryption

As Kubernetes adoption grows rapidly in enterprises, protecting cluster data is critical. Backups ensure business continuity in case of failures, accidental deletions, or security breaches. For over 2 years, users have depended on the integrated backup/restore capability in the Rafay Platform to dramatically simplify Kubernetes backup and restore operations. When the backups artifacts are stored in public cloud environments, organizations may have a concern with security. One of the most effective ways to secure these backups is by using Server-Side Encryption (SSE). SSE encrypts data at rest within cloud storage services, protecting it from unauthorized access while minimizing operational overhead.

In this blog, I describe the value of SSE encryption for Kubernetes backups and how it enhances security and compliance. I will also describe how administrators can configure and use SSE for backups in the Rafay Platform.

Encryption

Info

Learn about the integrated Backup/Restore capabilities in the Rafay Platform.

Upstream Kubernetes on Flatcar Linux using Rafay

This blog is Part 3 of our series on Flatcar Linux and Kubernetes

  • In Part 1, we introduced Flatcar Linux and why it is a great fit for Kubernetes.
  • In Part 2, we covered how to install a Flatcar instance locally.
  • In this Part 3, we focus on deploying and managing Upstream Kubernetes on Flatcar Linux using Rafay MKS.

Our upcoming February release will introduce a number of new features and enhancements.We will write about these in separate blogs. This blog is focused on support for Upstream Kubernetes based on Rafay MKS on nodes running Flatcar Linux. The Rafay platform enables users to seamlessly provision new clusters and perform in-place upgrades of Kubernetes clusters, simplifying lifecycle management.

For more details on Flatcar Linux, visit the official Flatcar Linux website.

Flatcar Logo


Provision Cluster

Rafay MKS based Upstream Kubernetes clusters can be configured and provisioned on Flatcar Linux using all the supported interfaces i.e.

  • Web Console
  • API
  • CLI (declarative spec)
  • GitOps
  • Rafay Terraform/OpenTofu Provider

In this blog, we will demonstrate this using the web console and the Rafay RCTL CLI.

Install Flatcar-Part 2

This is in continuation of our first blog where we introduced Flatcar Linux. In part 2, we will show how you can install a Flatcar Container Linux instance locally on your laptop so that you can learn more about it. This guide will take you through the steps to install it, boot the instance, SSH into it. Finally, we will explore and validate some of Flatcar's critical features that we reviewed in the first blog.


Boot Flatcar Instance Locally

Follow the steps below to download, set up, and run a Flatcar instance locally. In our example, we are installing this on a M1 MacBook Pro.

Step 1: Install QEMU

First, install QEMU on your system. Follow the instructions provided here.

Step 2: Download the QEMU Script and Flatcar Image

Get the QEMU helper script and the latest stable Flatcar image

wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu.sh
wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img.bz2

Step 3: Extract the Image

Decompress the downloaded image

bzip2 --decompress --keep flatcar_production_qemu_image.img.bz2

Step 4: Make the Script Executable

Set execution permissions for the QEMU helper script:

chmod +x flatcar_production_qemu.sh

Step 5: Create/Start the Flatcar Instance

Use the following command to start the Flatcar instance with 4 CPUs and 4GB memory in console mode:

./flatcar_production_qemu.sh -M 4096 -- -smp 4 -display curses

Access Flatcar Instance via SSH

Once the Flatcar instance is running, set up your SSH configuration to connect to it. Update your SSH config file as follows:

# ~/.ssh/config
Host flatcar
        User core
        StrictHostKeyChecking no
        UserKnownHostsFile /dev/null
        HostName 127.0.0.1
        Port 2222

You can then SSH into the instance using the following command

ssh flatcar

After you successfully SSH into the instance, you can run the following command to verify the system's firmware and kernel details:

hostnamectl

You should see something like the image below.

hostnamectl

Flatcar Versioning

In the image above, notice that the Flatcar version is 4081.2.1. Here’s how the versioning system works for Flatcar Linux.

  • 4081: The number of days since the first CoreOS release. (Flatcar is a fork of CoreOS)
  • 2: Minor number representing the promotion level:

    • 0 = Alpha
    • 1 = Beta
    • 2 = Stable
    • 3 = LTS
  • 1: Patch level, indicating small updates like kernel or software fixes.

In our version (4081.2.1), 2 represents a stable release on its first patch.


Explore Flatcar's Key Features

Now that we have installed Flatcar Linux, let us test and validate some of the interesting features of Flatcar.

1. No Package Manager and Immutable File System

In Linux, the /usr directory is used to store user system resources. It contains the majority of the system’s software and programs i.e. binaries, libraries, documentation, and other files shared by all users of the system. Flatcar Linux makes the /usr directory read-only as part of its design philosophy to ensure system immutability and reliability. By focusing on immutability and containerization, Flatcar Linux achieves a stable, predictable, and secure platform, which is why /usr is deliberately set to read-only.

Let's test whether Flatcar Linux's /usr folder is immutable

  • SSH into your Flatcar instance
  • Run commands like apt, yum, or dnf

You will notice they are unavailable. Additionally, any attempt to write to the /usr/ directory to see that the file system is immutable. For example, let's try to create a new file in the /usr directory using the "touch" command.

sudo touch /usr/test 

Below is an example screenshot showing this behavior:

no pkg manager


2. Automatic Updates

Flatcar includes a auto update system by default. By default, the Flatcar instance will check for updates every hour, download them if available, and automatically reboot to apply updates to ensure that your instance is always current and up to date. Organizations can always self host their own update server ensuring that they can control how/when their Flatcar instances are kept current.

To view the update configuration, run the following command inside the Flatcar instance:

cat /usr/share/flatcar/update.conf

The output will look like this:

SERVER=https://public.update.flatcar-linux.net/v1/update/
GROUP=stable
  • SERVER: This is the update server Flatcar uses to check for new releases.
  • GROUP: Refers to the promotion level (e.g., stable).

update config

Info

Administrators can customize the default behavior by configuring the upgrade strategy. For example, a common configuration is for administrators to specify a "maintenance windows" for reboots.


Other Installation Methods

If you’re not using QEMU, you can install Flatcar Container Linux on other platforms like VirtualBox, Vagrant, and others. You can find detailed installation instructions for these platforms here. Flatcar also provides prebuilt images for cloud providers like AWS, Azure, Google Cloud, and others. These cloud platforms offer Flatcar-based AMIs or images to streamline deployment. Check out the cloud-specific installation steps here.


Conclusion

In this blog, we:

  • Walked through the steps to install and run a Flatcar Container Linux instance locally.
  • Validated some unique characteristics of Flatcar, such as its lack of a package manager and its auto-update system.
  • Explored Flatcar versioning.

In the upcoming Part 3 of the blog series on Flatcar Linux, we will cover how to install Rafay's Kubernetes Distribution (Rafay MKS) on Flatcar and manage it centrally using the Rafay Platform.

Flatcar Linux: A Great Fit for Kubernetes

In the fast-evolving landscape of containerized applications and cloud-native technologies, choosing the right operating system for your Kubernetes cluster can sometimes make a very big difference. Enter Flatcar Container Linux, an open-source, minimal, and immutable Linux distribution tailored specifically for running containers.

Flatcar is an excellent choice for Kubernetes and modern cloud-native environments. In Aug 2024, Flatcar Linux was accepted as a CNCF project.

This is a 3-part blog series. In this blog, we'll explore what Flatcar Linux is, why it’s uniquely suited for Kubernetes, and the benefits it brings relative to generic Linux.

Flatcar Logo

Kubernetes v1.32 for Rafay MKS

As part of our January release, alongside other enhancements and features, we are adding support for Kubernetes v1.32 with Rafay MKS (i.e., upstream Kubernetes for bare metal and VM-based environments).

Both new cluster provisioning and in-place upgrades of existing clusters are supported. As with most Kubernetes releases, v1.32 deprecates and removes several features. To ensure zero impact to our customers, we have validated every feature of the Rafay Kubernetes Operations Platform on this Kubernetes version.

Kubernetes v1.32 Release