Measure Databricks Assistant impact

This article provides information on measuring the impact of Databricks Assistant in terms of adoption, egagement, and reported productivity gains.

Requirements

In order to measure the impact of Databricks Assistant, you need:

• Account admin privileges required to enable system tables. See Enable system tables.

• An internal survey to get subjective feedback on Assistant from your team.

Data vs. metrics

You get raw data from the system tables and a survey. To understand Assistant impact, you must analyze and report data as metrics. Metrics are calculated values you use to measure specific aspects or activities related to Assistant impact. This article also refers to metrics as measures.

Tips for measuring Assistant impact

To understand how your organization is using Databricks Assistant, start by measuring its adoption and engagement with Databricks Assistant. This data can be calculated from system tables.

Review your data routinely and make it easily available in a shared dashboard. For a dashboard example and template, see Databricks Assistant system table reference and example.

Assistant impact metrics

The following are recommended measures of Assistant impact, both from system tables and from user feedback. For examples of metrics calculations, download the Assistant events dashboard file from GitHub and read the calculations in JSON. To learn how to import a dashboard file, see Import a dashboard file

Surveying your organization also helps you understand the effectiveness of its engagement with Assistant. See Recommended survey questions.

Measure	Definition	Stage	Data source
Top users overall	Users in a given period who interact most frequently with Assistant	Adoption	Calculate from Databricks Assistant system table data.
Submissions data: per workspace and total	Number of requests submitted to Assistant per workspace and per account	Adoption	Calculate from Databricks Assistant system table data.
Active users by day and month	Unique users who have received and accepted 1+ suggestions or participated in 1+ chats on a given day.	Engagement	Calculate from Databricks Assistant system table data.
Active users per workspace	Unique users in a given workspace who interact with Assistant	Engagement	Calculate from Databricks Assistant system table data.
Top job roles using Databricks Assistant	Numbers of people from each role identified in your org who use Assistant	Engagement	Survey of your organization
Top tasks for Assistant	Most common tasks Assistant helps with	Engagement	Survey of your organization
How often do you use Assistant	Self-reported frequency of Assistant use per user	Engagement	Survey of your organization
Top areas of Assistant use	Self-reported usage areas: SQL editor, notebooks, or both	Engagement	Survey of your organization
User satisfaction with Assistant help	Self-reported satisfaction with Assistant’s answers on a scale of 1-5	User satisfaction	Survey of your organization
Gains in user productivity	Self-reported increase in productivity gained using Assistant on a scale of 1-5	User satisfaction	Survey of your organization
Amount of time users save using Assistant	Self-reported percentage of time saved by using Assistant to complete tasks	User satisfaction	Survey of your organization

Recommended survey questions

System tables won’t capture all information about how your organization uses Databricks Assistant. Use survey questions, such as the ones below, to understand how your organization uses Assistant.

• What is your role? List roles in your organization

• How often do you use Assistant?

  • Hourly

  • Daily

  • Weekly

  • Monthly

• Where do you use the AI Assistant?

  • SQL editor

  • Notebooks

  • Both

• What are your main use cases for Assistant?

  • Fixing errors / troubleshooting

  • Help writing code

  • Code refactoring

  • Writing SQL queries

  • Writing unit tests

  • Looking up documentation

• How satisfied are you with Assistant’s answers? Scale of 1-5, Not satisfied to Extremely satisfied

• Since using the AI Assistant, would you say you’re more productive? Scale of 1-5, Much less productive to Much more productive

• How much time do you believe you save on an average task using the Assistant?

  • 0-10%

  • 10-30%

  • 30-50%

  • 50-70%

  • More than 70%