Databricks Prerequisites

Prerequisites

• By default, this Databricks integration makes use of Unity Catalog data governance features. You will need Unity Catalog enabled on your Databricks Workspace.

Step 1: Create a SQL endpoint

Create a new SQL endpoint for data writing.

1. Log in to the Databricks account.
2. In the navigation pane, click into the workspace dropdown and select SQL.
3. In the SQL console, in the SQL navigation pane, click Create and then SQL endpoint.

4. In the New SQL Endpoint menu, choose a name and configure the options for the new SQL endpoint. Under "Advanced options" turn "Unity Catalog" to the On position, select the Preview channel, and click Create.

Step 2: Configure Access

Collect connection information and create an access token for the data transfer service.

1. In the SQL Endpoints console, select the SQL endpoint you created in Step 1.

2. Click the Connection Details tab, and make a note of the Server hostname, Port, and HTTP path.

3. Click the link to Create a personal access token.

4. Click Generate New Token.

5. Name the token with a descriptive comment and assign the token lifetime. A longer lifetime will ensure you do not have to update the token as often. Click Generate.
6. In the pop up that follows, copy the token and securely save the token.
7. In the Databricks UI, select the Catalog tab, and select the target Catalog. Within the catalog Permissions tab, click Grant. In the following modal, select the principal for which you generated the access token, select USE CATALOG, and click Grant.
8. Under the target Catalog, select the target schema (e.g., main.default, or create a new target schema). Within the schema Permissions tab, click Grant. In the following modal, select the principal for which you generated the access token, and select either ALL PRIVILEGES or the following 9 privileges and then click Grant:
   • USE SCHEMA
   • APPLY TAG
   • MODIFY
   • READ VOLUME
   • SELECT
   • WRITE VOLUME
   • CREATE MATERIALIZED VIEW
   • CREATE TABLE
   • CREATE VOLUME

Step 4: Gather the required setup information

For the data export setup, you will need:

• server hostname
• HTTP path
• catalog
• personal access token
• Metastore
  • Enter unit_catalog into this field.

Visit the LogRocket Streaming Data Export settings page to complete the setup.

🚧

Alternative to Unity Catalog

As an alternative to unity catalog, data can be housed as a hive metastore in your object storage. This option is no longer enabled by default. If you have questions about using this feature, please reach out to LogRocket support.