Hitachi and Cloudera have announced that Hitachi's Pentaho data integration and analysis platform - core of Hitachi's Lumada portfolio - and Cloudera Enterprise Data Hub have been adopted for the new data utilization platform (the Platform) of Tokyo Stock Exchange, Inc. (TSE) that recently started operation.
The Platform standardizes data management across the Japan Exchange Group, and the basis for streamlining back-office operations such as form creation and transaction fee calculation, and expanding the information business. By combining Pentaho and Cloudera Enterprise Data Hub, the Platform has achieved efficient data integration and simple visualization, while also significantly boosting processing performance, such as by shortening processing times.
As a first step, TSE consolidated the system that creates forms based on various data such as transactions and the system that calculates commissions each time an individual transaction or payment. Moving forward, there are plans to progressively expand the Platform as a common infrastructure of the Japan Exchange Group, including systems that analyze and produce reports on transaction data from various angles.
TSE processes more than tens of millions transactions on a typical day, and more than 100 million on busy days. To cope with transaction volumes that will expand in the future, TSE faced the challenges of speeding up data processing and developing an environment enabling utilization of the data. Moreover, considering the possibility of utilizing cloud services in the future and an expanding scope of adoption, open technologies not dependent on specific vendors were needed.
Pentaho has highly affinity with Hadoop/Impala environments, and is able to provide a consistent environment that not only collects but also analyzes and utilizes data, including the visualization and multidimensional analysis of integrated data. Meanwhile Cloudera's secured and governed data platform enables multi-function analytics that eliminates silos and speeds the discovery of data-driven insights. Fueled by open source innovation, it has true hybrid capability that supports public cloud, multi-cloud and on-premises deployments.
Many customers, including financial institutions, have introduced a combination of "Pentaho" and "Cloudera Enterprise Data Hub" in the global market.
The Platform that recently began operating utilizes the Pentaho Data Integration (data integration and form creation) and Pentaho Business Analytics (visualization and analysis) components of Pentaho, and the Cloudera Enterprise Data Hub(1), was built by Tosho System Service Co., Ltd. Taking advantage of the coding-less data integration features, Pentaho was able to develop easily and quickly while repurposing the processing programs of TSE's existing business systems.
Specifically, Pentaho was used to consolidate data that had previously been dispersed across individual business systems such as those for form creation and fee calculation, and data integration processing was standardized. This greatly streamlined data preparation tasks, which especially required considerable time and effort in the data analysis process, and made it easy to cross-reference data between different systems.
The parallel distributed processing and the SQL engine Impala offered by Cloudera also speeds up batch processing to significantly reduce processing times and ensure scalability to handle future data increases.
Moving forward, Hitachi and Cloudera will not only offer stable, enterprise-level operation by enhancing TSE's system processing capabilities but also support the promotion of digital transformation through the provision of an environment allowing the creation of new services that utilize the accumulated data assets and contribute to ongoing business reform and development.
Comments from Takashi Kamikura, Manager, Information Systems Department, IT Development Division, Tokyo Stock Exchange, Inc.
In recent years, the amount of market-related data has continued to increase remarkably, making improved data processing an urgent issue. With this latest adoption of Cloudera Enterprise Data Hub, we have significantly improved our data processing capabilities and we are delighted to have achieved stable operation. In addition, we were able to quickly and easily develop the applications needed to utilize Cloudera Enterprise Data Hub with the intuitive GUI-based operation offered by Pentaho, which is highly compatible with the Hadoop environment. In the future we have our sights on expanding to the cloud and utilizing the data to create new businesses.
