Democratizing multi-LLM: Kong’s open-source AI Gateway

From consumer products to industrial and enterprise use cases, artificial intelligence is asserting itself as the following technology frontier and innovation area. In the identical vein, Kong Inc. has made a major stride by open-sourcing its AI Gateway, aiming to streamline AI application development and foster responsible AI adoption across organizations.

“Up to now few months, we worked with developers and organizations that began constructing AI to ship of their products to construct recent experiences,” said Marco Palladino (pictured), chief technology officer of Kong. “We noticed that developers keep doing the identical things over and all over again, so we thought that there might be a possibility for us to supply modern AI infrastructure to speed up their productivity as they’re constructing these recent applications.”

Palladino spoke with theCUBE Research executive analyst John Furrier, during a CUBE Conversation from SiliconANGLE Media’s livestreaming studio in Palo Alto. They discussed how Kong plans to address the pressing challenge organizations face in establishing AI infrastructure and turning AI solutions into products. (* Disclosure below.)

Solving the AI infrastructure challenge: Inside Kong’s AI Gateway

With the proliferation of AI technologies and the growing must integrate multiple large language models, developers are encountering hurdles in streamlining their AI workflows. Kong’s AI Gateway emerges as an answer to reinforce productivity and supply visibility into AI traffic, crucial for organizations venturing into AI development.

The just-announced plugins for Kong Gateway 3.6 paves the way in which for developers who wish to integrate multiple LLMs into their products to ship those AI capabilities faster, while concurrently empowering architects and platforms with a secure solution with overarching visibility, control and compliance on every AI request.

“With the AI Gateway, what we would like to supply is a set of capabilities out of the box for multi-LLM consumption, for security, for credentials, for prompt engineering,” Palladino said. “[But] at the identical time also give visibility to the architects and the platform teams on what’s the AI traffic that’s being generated to incentivize responsible usage of AI from the organization.”

Along with multi-LLM integration, Gateway 3.6 brings a slew of recent, user-focused capabilities, including central AI credential management, no-code AI integrations, AI prompt decoration, AI prompt firewall and comprehensive AI egress with extensive features.

No-code AI plugins and responsible adoption

A notable aspect of the Gateway is the introduction of no-code AI plugins, which enable developers to enhance existing API traffic with AI capabilities without writing any code. This innovation simplifies AI integration, making it accessible to a broader range of users, including those with limited coding experience. These plugins offer a myriad of use cases, from language translation to real-time analytics, empowering organizations to leverage AI effortlessly.

“There are two plugins that I’m quite enthusiastic about, and these are the AI Requests and Response Transformer plugins,” Palladino said. “These plugins allow us to integrate AI in our existing API traffic without having to put in writing any code. Up until now, all the opposite plugins imply that, as a developer, I’m constructing an integration for AI and I’m writing code to make that occur. But with these no-code AI plugins, we will get the advantage of AI on top of existing API traffic without having to put in writing any line of code.”

The emergence of a multi-LLM world, where organizations utilize each cloud-based and self-hosted LLMs, is inevitable, in line with Palladino. Seeing this, Kong is abstracting the complexity of LLM integration and empowering organizations to harness the total potential of AI while minimizing operational overhead.

“At the start, the AI Gateway is fully open source, so it’s free to make use of,” he said. “Anybody can go and download or run it within the cloud without spending a dime. It’s not a business product offering. It’s an open-source offering. Kong itself has an ecosystem of plugins that allow it to expand what the product does. We have built plugins, six plugins that allow us to perform Level 7 AI operations on top of any form of AI traffic the developers are generating.”

As AI continues to reshape industries, there’s an industry-wide must lay out the long run of AI integration and its implications for developers and organizations. Organizations must adopt a playbook approach to adoption, where they define clear processes and guidelines for deploying AI solutions, in line with Palladino. By specializing in end-user needs and outcomes, organizations can leverage AI to create transformative experiences while minimizing risks and maximizing productivity.

Here’s theCUBE’s complete video interview with Palladino:

(* Disclosure: Kong sponsored this segment of theCUBE. Neither Kong nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE