Introducing Databricks AI/BI

Databricks AI/BI features a pair of complementary experiences: Dashboards, an AI-powered, low-code interface for creating and distributing fast, interactive dashboards; and Genie, a conversational interface for addressing ad-hoc and follow-up questions through natural language. Both are powered by a compound AI system that continuously learns from usage across an organization’s entire data stack, including ETL pipelines, lineage, and other queries. This allows AI/BI to answer complex questions accurately and automatically, without the requirement for upfront setup or modeling associated with conventional tools.

AI/BI marks a significant step towards true self-service BI, substantially expanding the types of analytics everyday users can do through an AI-first approach. AI/BI’s intelligence is enabled by the Databricks Data Intelligence Platform, which is natively integrated to provide unified governance, lineage, secure sharing, and lightning-fast performance for all data sizes.

Enabling Self-Service Through Generative AI

Today, BI tools allow organizations to provide basic access to data through reports and dashboards, but they can only address the specific questions they were designed to answer. Changes in data or business conditions constantly spark new questions that require the scarce expertise of data scientists and analysts. Recently, the BI industry has been looking to generative AI to solve this problem by adding large language models (LLMs) on top of a conventional BI stack. However, these attempts have largely fallen short in the real world. The approach of solving this problem by passing a set of natural language questions into LLMs to generate an execution plan to answer such questions is fundamentally limited because real-world use cases are full of nuanced business questions with messy data and ill-defined semantics. Ultimately, “bolt-on AI” solutions may provide compelling demos but are unable to work with real questions and data.

“A truly intelligent BI solution needs to understand the unique semantics and nuances of a business to effectively answer questions for business users,” said Ali Ghodsi, Co-founder and CEO at Databricks. “We believe this requires a different approach than how BI software has been designed in the past — one that places an AI system at the center of the architecture and is designed to take advantage of the AI systems’ strengths as well as complementing their weaknesses to tackle the challenges of understanding and learning these nuances. The launch of AI/BI is a step towards building such a system.”

Beyond “Bolt-on” LLMs: AI/BI is a Compound AI System for Analytics

At the core of AI/BI is a compound AI system that utilizes an ensemble of AI agents to reason about business questions and generate useful answers. Each agent is responsible for a narrow, but important task, such as planning, SQL generation, explanation, visualization and result certification. These agents are also supported by other components, such as a response ranking subsystem and a vector index. Together, they provide reasoning capabilities far beyond any individual monolith model.

The system is designed to continuously learn and improve its performance based on human feedback. For example, if told the definition of a churned customer, AI/BI will not only use that knowledge to address similar queries (e.g., churned customers in EMEA versus APJ), but also use that knowledge to calculate churn rate, or infer the meaning of retained customers. AI/BI persists this knowledge beyond a single analysis or conversation to get better and better, much like a human analyst. In addition, AI/BI learns from other information about an organization’s data in the Databricks platform, such as ETL pipelines, lineage, popularity statistics, and other queries on the data.

This system powers two complementary interfaces:

AI/BI Dashboards: an AI-powered, low-code dashboarding solution that includes the conventional BI capabilities users would expect out-of-the-box, for answering a fixed set of business questions; and

AI/BI Genie: a conversational interface that can learn the underlying data and semantics continuously based on human feedback, and can answer a much broader set of business questions based on its reasoning capabilities, while still providing certified answers for query patterns specified by the data teams.

Databricks AI/BI is built on top and tightly integrated with the Data Intelligence Platform. Benefits include:

Unified governance and lineage: AI/BI is deeply integrated into Databricks Unity Catalog. It follows the same governance framework, and any global policies set by administrators will apply in AI/BI. And thanks to Unity Catalog’s lineage capability, data producers or administrators can observe how their data assets are used in AI/BI, and end users can place higher trust on their analysis because they can trace back the origins of their datasets all the way back to data ingestion.

Effortless sharing, without new user licenses: AI/BI is built right into the Databricks IAM platform which integrates directly with IDPs like Entra AD or Okta to enable secure sharing.

Industry-leading price-performance: AI/BI is tightly integrated with Databricks SQL data warehouses and the Photon engine, which contain unique optimizations to deliver high-performance interactions. It provides industry-leading price-performance across data volumes, from megabytes to petabytes.

No data extraction required: Consequently, AI/BI does not require extracting the datasets of interest out to a separate BI engine, leading to better freshness of data as well as simpler governance.

“At SEGA, we aim to entertain the world with creative, innovative experiences, and data intelligence plays an important role in achieving that goal. We’re using Databricks AI/BI to help decision-makers ask ad-hoc questions in real time about consumer behavior without having to depend on our data experts to construct dashboards and queries. Now our team members can get detailed insights on game sales and gameplay data by simply asking in natural language,” said Felix Baker, Head of Data Services, SEGA Europe. “AI/BI will enable us to democratize data, increase productivity, and enhance the speed of data-driven decision making throughout SEGA, which enables us to focus on what really matters: building the best possible gaming experiences for our players.”