Confluent Expands Apache Flink Capabilities to Simplify AI and Stream Processing

AdminTeam May 6, 2024

Confluent, the data streaming pioneer, announced last month the general availability of Confluent Cloud for Apache Flink, enabling users to experience Apache Kafka and Flink as a unified, enterprise-grade platform.

This week, Confluent announced that the addition of new capabilities to the service. The managed service for Apache Flink now features AI Model Inference, which allows users to clean and process real-time streaming data for AI and ML applications. In addition, Confluent has also announced an auto-scaling cluster type for use cases such as logging and telemetry data without time requirements.

The AI Model Inference enables teams to easily integrate ML into data pipelines using simple SQL commands. It can be used for enhancing GenAI workloads to deliver more personalized customer experiences. By extracting the most important records from a data stream and summarizing text, the AI Model Inference can also provide context-rich data for AI chatbots to deliver more accurate outputs.

According to Confluent, this new capability can simplify the AI development process. The developers won’t have to rely on specialized tools and programming languages. Instead, they can simply use the SQL syntax that is most familiar to them. In addition, teams can now orchestrate data cleaning and processing tasks on a single platform.

The seamless coordination between data processing and AI workflows should also improve efficiency and reduce operation complexity. The AI Model Inference is currently available in early access to select customers.

“Apache Kafka and Flink are the critical links to fuel machine learning and artificial intelligence applications with the most timely and accurate data,” said Shaun Clowes, Chief Product Officer at Confluent. “Confluent’s AI Model Inference removes the complexity involved when using streaming data for AI development by enabling organizations to innovate faster and deliver powerful customer experiences.”

Confluent is well-known for its expertise in Apache Kafka, the popular open-source big data streaming platform used for building real-time data pipelines and streaming applications. Founded in 2014 by the creators of Apache Kaffa, Confluent provides a range of products and services to help simplify the use of Kafka-based applications.

Apache Flink is an open-source big data processing framework developed by Apache Software Foundation (ASF). It has witnessed a rapid rise in the last couple of years, partially as a result of Flink’s strong community and lack of real competition in the market.

With the introduction of the Confluent Platform for Apache Flink, the company is enabling customers to self-manage Apache Flink in on-prem and hybrid cloud environments. The added advantage of combining Apache Kafka and Apache Flink is the optimized integration, streamlined support, and seamless compatibility with the two technologies.

(a-image/Shutterstock)

The other key upgrade to Apache Flink is the new auto-scaling clusters, which can dynamically adjust the resources allocated to the cluster. This allows for lower cost for high-throughput use cases with relaxed latency requirements.

Confluent Cloud is used to produce and consume large volumes of data for logging and telemetry. While these use cases have a high throughput, their latency requirements are more relaxed.

Powered by Elastic CKUs, Freight clusters are a new addition to the service. They offer a new serverless cluster type with up to 90% lower cost for high-throughput for less latency-sensitive use cases. Available in early access at select AWS regions, Freight clusters auto-scale based on demand helping optimize costs and minimize operational overhead.

Aiven Announces General Availability of Aiven for Apache Flink

Confluent to Develop Apache Flink Offering with Acquisition of Immerok