Delivering Enterprise Big Data solutions

“As more data streams flow into enterprises from a larger variety of sources, IT leaders must extend data processing to include more use of push (event-driven) models,” said Nick Heudecker and Roy Schulte, Gartner. “The opportunities and the risks, of not harnessing high-velocity data streams are too significant to ignore. This shift in data models exposes new business and operational opportunities in information security, the Internet of Things (IoT), customer engagement, fraud detection and telematics, among others.”

Enterprises today are experiencing an explosion of incoming streaming data from many different sources. The traditional model of batch analytics is an important method of deriving insight from static data, but enterprises also need to get immediate insight at the moment information changes. Couchbase Server 3.0.2 integration with Hortonworks Data Platform 2.2, via distributed messaging with Apache Kafka and stream processing via Apache Storm, provides a powerful new big data architecture that make it enables streaming real time data to be leveraged for new business and operational opportunities.

The Kafka connector leverages Couchbase’s Database Change Protocol (DCP), a unique architectural feature of Couchbase Server 3.0.2, to stream data from Couchbase Server to the Kafka message queue in real time. The messages are consumed by Storm for real-time analysis. The data is then written to Hadoop for further processing and the analysis is written to Couchbase Server for access by real-time reporting and visualization dashboards. This provides enterprises the insight they need at the moment they need it. By integrating Hortonworks Data Platform and Couchbase Server, enterprises can meet both operational and analytical requirements with a single solution to improve short-term and long-term operations.

“The Kafka connector strengthens the streaming of data to and from Couchbase Server to Hadoop ecosystems. Couchbase Server accommodates fast access to data at scale while leveraging Kafka to stream data to Hadoop for deep analytics,” said Anil Madan, senior director of engineering at PayPal. “As the operational data store, Couchbase is easily capable of processing tens of millions of updates a day, or more. Streaming through Kafka into Hadoop, these key events are turned into business insight which is why we are very excited about this integration.”

Offline Analytics and Data Refinement
Additionally today, Hortonworks certified the Couchbase Server plugin for Sqoop to support bi-directional data transfer between Couchbase Server 3.0.2 and Hortonworks Data Platform 2.2. This integration enables enterprises to export operational big data, produced and consumed by enterprise web, mobile, and IoT applications and stored in Couchbase Server, to Hortonworks for offline analysis and refinement; refined data is imported back into Couchbase Server. This closed loop analysis and data insight gives businesses the ability to deliver the next level of personalized experiences in real time by accounting for context.

“Enterprises want to build web, mobile and IoT applications that can deliver contextual insight in real time,” said Rod Hamlin, vice president of business development, Couchbase. “The integrations announced today make it faster and easier for IT organizations to deploy big data architectures that enable streaming data analysis and support real time decision making and personalization. These capabilities support faster and better business insight that build competitive advantage.”