Are you seeking smarter insights delivered directly to your inbox? Subscribe to our weekly newsletters for the most relevant updates on enterprise AI, data, and security.
The Value of Real-Time Streaming Data
Real-time streaming data holds immense potential across various industries. For the New York Stock Exchange (NYSE), this data translates directly into financial gain. As one of the world’s largest financial exchanges, the NYSE has a long-standing tradition of disseminating its financial market data. A century ago, it relied on telegraph-based ticker tapes for this purpose. Today, it has advanced to utilizing its own low-latency, high-performance technologies that organizations can connect to on-premises.
Embracing Apache Kafka Technology
The NYSE is now taking a significant step forward by adopting an open-source model based on Apache Kafka streaming technology, which brings its Best Quote and Trades (BQT) data to the AWS cloud. To facilitate this transition, the NYSE has partnered with Redpanda, a streaming data platform vendor that has developed its own Kafka implementation in C++. This deployment has resulted in performance improvements of 4 to 5 times compared to traditional Kafka competitors, highlighting the limitations many organizations face when managing bursty data workloads.
The Challenges of AI Scaling
As enterprises scale their AI applications, they encounter challenges such as power caps, rising token costs, and inference delays. To address these issues, we invite you to join our exclusive salon, where leading teams are sharing their strategies for turning energy into a strategic advantage, architecting efficient inference for real throughput gains, and unlocking competitive ROI with sustainable AI systems. Secure your spot to stay ahead: https://bit.ly/4mwGngO.
NYSE’s Cloud Streaming Platform
The NYSE has built its cloud streaming platform to cater to customers who cannot access its data centers directly. The exchange primarily targets fintech companies and retail broker-dealers that require AWS-based access to real-time market data. “Not every consumer of our market data has the capacity to come to our data center, take the feed and use that feed,” explained Vinil Bhandari, head of cloud and full-stack engineering at NYSE. “However, a small firm in Hong Kong can easily create their own AWS account, and it’s these audiences we aim to serve.”
Handling Market Volatility
The NYSE streams its BQT feed, which compiles real-time data from all seven NYSE exchanges. This deployment necessitated the construction of new infrastructure rather than merely extending existing systems. The NYSE processes over 500 billion messages daily across these exchanges, and during periods of market volatility, message volume can surge by up to 1,000 times the average within microseconds. Traditional Java implementations struggle to manage these patterns due to unpredictable latency spikes caused by garbage collection.
The Advantages of C++ Implementation
Bhandari elaborated, “The classic Kafka implementation was written in Java, which does not handle bursty traffic well due to its garbage collection process.” In contrast, Redpanda has rewritten the Kafka protocol in C++, enabling the NYSE to manage data streaming more effectively during traffic surges caused by market activity. This choice of programming language is a key reason why the NYSE opted for Redpanda over other options like Confluent or Amazon Managed Streaming for Kafka (MSK), leading to measurable performance enhancements.
Performance Comparisons
Bhandari noted, “We can confidently state that our data delivery using Redpanda is at least four to five times faster compared to some of our major competitors who utilize Kafka technology for similar data streaming.” For enterprises assessing streaming platforms, this comparison underscores a critical factor: Java-based implementations may falter during traffic spikes, while C++ alternatives can sustain consistent performance.
The Importance of Observability
Bhandari emphasized the significance of observability in production streaming deployments. Redpanda’s built-in telemetry features provide immediate operational value. “The more a deployment can offer observability and telemetry regarding what’s happening under the hood, the better the data producers and consumers will perform,” he explained.