Optimal LLM Serving: Where Speed Meets Efficiency

In today’s tech-savvy world, everyone is in a race against time, especially when it comes to real-time applications like chatbots and data processing. Enter large language models (LLMs) like GPT-3 and Llama, which have revolutionized how we generate text, engage in dialogues, and process information. But how do we handle the massive workload these models generate, especially when we have urgent real-time requests alongside bulk processing demands? That’s where BROS (Bidirectional Resource Optimal System) comes in—a hybrid serving system that effectively balances both real-time (RT) and best-effort (BE) requests to maximize efficiency without compromising speed.

The Dilemma: Real-time vs. Best-effort Requests

Before diving into BROS, let’s clarify what we mean by RT and BE requests.

Real-Time (RT) Requests

Think of RT requests as urgent phone calls that need immediate responses, like asking a question on a live Q&A platform. These requests prioritize low latency, meaning they should be processed as fast as possible—ideally in the blink of an eye.

Best-Effort (BE) Requests

On the other hand, BE requests are like sending emails—you don’t need an instant answer; you’re more concerned about having your inquiry handled eventually. These tasks generally favor high throughput (handling as many as possible over a longer period) rather than speed.

The Challenge: The needs of RT and BE requests often clash. While RT requests need quick processing, BE requests want to maximize GPU resources for handling many inputs in bulk. Most conventional LLM serving systems deal with this by assigning separate machines to each type of request, which isn’t the most efficient use of resources.

Enter BROS

The innovative approach of BROS aims to address this inefficiency by collocating RT and BE requests on the same system, optimizing memory usage and ensuring that urgent queries don’t lag behind in processing.

How BROS Solves the Problem

1. Dynamic Request Scheduling: BROS uses a smart algorithm that assigns priorities to different requests based on their urgency. Instead of adhering to a simple “first-come, first-served” approach, it looks at how each request fits into the larger workflow. This allows for processing that not only meets the deadline for RT requests but also takes care of BE requests efficiently.

2. Bidirectional KV Cache Management: A huge challenge during the co-processing of RT and BE requests is memory contention, especially concerning the Key-Value (KV) cache used by LLM for token generation. BROS implements a clever layout where the KV caches for RT and BE requests share space but expand in opposite directions. This means as one cache grows, it doesn’t bottleneck the other.

3. Adaptive Batch Sizing: BROS dynamically adjusts the batch size of requests based on real-time conditions. If it notices that there are more RT requests than anticipated, it scales back to accommodate them—which can be likened to a bus driver adjusting the number of stops based on how many passengers are waiting.

Real-World Applications: Where BROS Shines

Imagine using an online customer service chatbot powered by LLMs. When hundreds of customers simultaneously seek help, having the system quickly serve RT requests while still processing background tasks is crucial.

Practical Implications

• Enhanced Customer Experience: By allowing rapid responses to queries without neglecting backend processing, businesses can enhance customer satisfaction and efficiency.

• Resource Utilization: Companies save on costs by not over-provisioning computing resources but instead optimizing existing infrastructure.

Results Speak Volumes

The experimental data backing BROS’ performance is remarkable. In extensive testing, BROS reduced the latency of RT requests by up to 74.20% while maintaining minimal reductions in BE request throughput.

Key Metrics

• Time to First Token (TTFT): The time it takes for the model to return the initiated response.
• Time Per Output Token (TPOT): How long it takes to generate each subsequent token.

BROS showed astounding SLO (Service Level Objective) attainments—up to 36 times higher for TTFT compared to existing systems like vLLM and TGI.

Key Takeaways

• Hybrid Efficiency: BROS combines RT and BE request processing on shared resources, leading to better utilization and cost savings.
• Dynamic Scheduling: The adaptive scheduler boosts responsiveness for urgent requests without entirely sacrificing the throughput of bulk processes.
• Smart Memory Management: Its unique bidirectional KV cache layout alleviates memory contention, ensuring urgent requests get the resources they need.

By setting the benchmark for LLM serving, BROS highlights the significance of adaptability and prioritization in serving diverse workloads. For anyone leveraging LLMs in their applications, understanding these dynamics not only helps in optimizing their systems but also paves the way for seamless interactions in a fast-paced digital environment.

In the ever-evolving landscape of artificial intelligence, innovations like BROS remind us that speed and efficiency can—and should—go hand in hand. Whether you’re building a chatbot, a document processing service, or exploring the potential of generative AI, finding the right balance is key to success.

If you are looking to improve your prompting skills and haven’t already, check out our free Advanced Prompt Engineering course.

This blog post is based on the research article “Efficient LLM Serving on Hybrid Real-time and Best-effort Requests” by Authors: Wan Borui, Zhao Juntao, Jiang Chenyu, Guo Chuanxiong, Wu Chuan. You can find the original article here.