Dealing with data governance

As more developers harness data science and machine learning, they are able to create cognitive applications and services that lead to greater data visibility and deeper insights to make data-driven decisions. IDC predicts that by 2018, 75 percent of all developers will embed cognitive in their applications[1] – but as data volumes explode, and data protection regulation mounts, the ability to effectively manage and exploit that data diminishes.

For instance, on May 25, 2018, the European Union will put into effect GDPR, a continent-wide set of requirements designed to protect its citizens’ personal information – any data “that can be used to directly or indirectly identify a person,” from a name to a post on social media. Organizations within or outside of Europe, that process or hold such personal data of EU citizens and fail to comply with GDPR could potentially face extremely stiff financial penalties that will range from four percent of the organization’s annual global revenue to 20 million euros.

With such profound technical change so imminent, organizations across Europe, and around the world, are reviewing and considering an array of solutions and processes that can help them better handle personal data and meet the requirements.

IBM Spearheads the Launch of the Open Data Governance Consortium for Apache Atlas

At the heart of preparing for regulation, such as GDPR, is data governance, which provides diligent and comprehensive data management practices for data integrity, security, usability, and availability. To make this easier for organizations and drive widespread adoption of data governance, IBM today is announcing the Open Data Governance Consortium for Apache Atlas.

Atlas is the Apache Foundation’s data governance framework for Hadoop, one of the leading open source frameworks for distributed data processing and storage. The project is currently in Apache’s “incubator” phase of development. One of the goals of the consortium is to collaborate to quickly advance it to “Top Level Project” status, at which point the technology will be available for open development contributions, download and distribution – and making robust governance capabilities open and free to the public.

The Consortium comprises such international members as leading Hadoop distribution provider, Hortonworks, and a dozen other members.

The news of the Open Data Governance Consortium for Apache Atlas follows strategic partnership IBM announced last week with Hortonworks, maker of one of the leading distributions of Hadoop, the Hortonworks Data Platform (HDP).

IBM Unveils New Data Governance Solutions

Also, to further help clients meet their growing data protection compliance demands, IBM today is announcing the following new solutions for helping clients gain greater control of their data:

• IBM Unified Governance Software Platform – a new software platform comprising data management capabilities, including many which may be mapped to the GDPR, such as cognitive metadata harvest, lineage tracking, policy enforcement, data integration services and persona-based reporting;
• Information Governance Catalog Download & Go – a fast, new software download that lets clients download, install and run specific governance tools directly to their systems, quickly and easily. The software complements, the full, cloud-based version of the catalog.
• StoredIQ – the popular data discovery software that helps users identify the types of unstructured data residing across their organizations, has been augmented with new analytics “cartridges,” or software modules, that people can download at no cost to begin recognizing sensitive, personal data, as well. Available for 15 European Union (EU) country-specific downloads, and their 11 languages, these cartridges are designed to help shorten the time it takes to discover personal data and better prepare for rising regulation, like GDPR.

Applying Data Science and Machine Learning to Governance

Increasingly critical to data governance is the ability to analyze and distill troves of data for insights and compliance. IBM today expanded its data science and machine learning innovations across Europe, to give more global enterprises access to the tools needed to apply data intelligence as they prepare for regulatory compliance. These include:

• The launch of the IBM Data Science Experience (DSX) in its London data center. Now, data scientists in the UK and across Europe are able to use the collaborative environment to easily and quickly team on analytic models that drive the creation of intelligent applications and generate data insights. Bringing DSX to the UK, will help organizations begin making more data-driven decisions, prepare for regional data regulation requirements, and provide more robust system performance.
• The launch of the Machine Learning Hub in Boblingen, Germany. The Hub presents a unique opportunity for companies in the region to collaborate with IBM on machine learning efforts with hands-on workshops. While many companies offer educational resources on emerging technologies, the Machine Learning Hub puts theory into practice to identify solutions to data science challenges in machine learning. To date, Machine Learning Hub teams have workshopped a wide range of use cases, from patient diagnosis to fraud detection to customer segmentation.

“The potential of data science and big data can only be realized with a unified approach to governance,” said Michael Willette, Executive Director and Technical Fellow, Data and Analytics, at financial services company, USAA. “IBM’s strategy is a good approach to addressing compliance concerns while allowing our users to find and discover data for analysis and data driven decision-making.”

“From the sheer volume and the continued distribution of data across evermore complex network clusters, to the rising tide of data regulations, such as GDPR, the need to organize, analyze and govern that data grows more critical every day,” said Rob Thomas, General Manager, IBM Analytics. “With the governance moves we’re making today, we’re giving organizations more ways to begin to not only understand what they have, but leverage it to make better business decisions and prepare for compliance.”