Databricks Markdown Cell

1. Markdown In Databricks
2. Databricks Markdown Syntax
3. Databricks Notebook Markdown Cell
4. Databricks %run Notebook
5. Markdown Cells In Databricks Notebooks
6. Databricks Notebook Markup

The Massage Connection: Anatomy and Physiology Kalyani Premkumar download Z-Library. Download books for free. Pycharm - Markdown syntax for table in Jupyter PyCharm pycharm. The Markdown Guide Matt Cone download Z-Library. Download books for free.

A Synapse Studio notebook is a web interface for you to create files that contain live code, visualizations, and narrative text. Notebooks are a good place to validate ideas and use quick experiments to get insights from your data. Notebooks are also widely used in data preparation, data visualization, machine learning, and other Big Data scenarios.

With an Azure Synapse Studio notebook, you can:

• Get started with zero setup effort.
• Keep data secure with built-in enterprise security features.
• Analyze data across raw formats (CSV, txt, JSON, etc.), processed file formats (parquet, Delta Lake, ORC, etc.), and SQL tabular data files against Spark and SQL.
• Be productive with enhanced authoring capabilities and built-in data visualization.

This article describes how to use notebooks in Azure Synapse Studio.

Preview of the new notebook experience

Synapse team brought the new notebooks component into Synapse Studio to provide consistent notebook experience for Microsoft customers and maximize discoverability, productivity, sharing, and collaboration. The new notebook experience is ready for preview. Check the Preview Features button in notebook toolbar to turn it on. The table below captures feature comparison between existing notebook (so called 'classical notebook') with the new preview one.

Create a notebook

There are two ways to create a notebook. You can create a new notebook or import an existing notebook to an Azure Synapse workspace from the Object Explorer. Azure Synapse Studio notebooks can recognize standard Jupyter Notebook IPYNB files.

Develop notebooks

Notebooks consist of cells, which are individual blocks of code or text that can be ran independently or as a group.

Add a cell

There are multiple ways to add a new cell to your notebook.

1. Expand the upper left + Cell button, and select Add code cell or Add text cell.

2. Hover over the space between two cells and select Add code or Add text.

3. Use Shortcut keys under command mode. Press A to insert a cell above the current cell. Press B to insert a cell below the current cell.

1. Expand the upper left + Cell button, and select code cell or Markdown cell.

2. Select the plus sign at the beginning of a cell and select Code cell or Markdown cell.

3. Use aznb Shortcut keys under command mode. Press A to insert a cell above the current cell. Press B to insert a cell below the current cell.

Set a primary language

Azure Synapse Studio notebooks support four Apache Spark languages:

• pySpark (Python)
• Spark (Scala)
• SparkSQL
• .NET for Apache Spark (C#)

You can set the primary language for new added cells from the dropdown list in the top command bar.

Use multiple languages

You can use multiple languages in one notebook by specifying the correct language magic command at the beginning of a cell. The following table lists the magic commands to switch cell languages.

The following image is an example of how you can write a PySpark query using the %%pyspark magic command or a SparkSQL query with the %%sql magic command in a Spark(Scala) notebook. Notice that the primary language for the notebook is set to pySpark.

Use temp tables to reference data across languages

You cannot reference data or variables directly across different languages in a Synapse Studio notebook. In Spark, a temporary table can be referenced across languages. Here is an example of how to read a Scala DataFrame in PySpark and SparkSQL using a Spark temp table as a workaround.

1. In Cell 1, read a DataFrame from a SQL pool connector using Scala and create a temporary table.

2. In Cell 2, query the data using Spark SQL.

3. In Cell 3, use the data in PySpark.

IDE-style IntelliSense

Azure Synapse Studio notebooks are integrated with the Monaco editor to bring IDE-style IntelliSense to the cell editor. Syntax highlight, error marker, and automatic code completions help you to write code and identify issues quicker.

The IntelliSense features are at different levels of maturity for different languages. Use the following table to see what's supported.

Code Snippets

Azure Synapse Studio notebooks provide code snippets that make it easier to enter common used code patterns, such as configuring your Spark session, reading data as a Spark DataFrame, or drawing charts with matplotlib etc.

Snippets appear in IntelliSense mixed with other suggestions. The code snippets contents align with the code cell language. You can see available snippets by typing Snippet or any keywords appear in the snippet title in the code cell editor. For example, by typing read you can see the list of snippets to read data from various data sources.

Format text cell with toolbar buttons

You can use the format buttons in the text cells toolbar to do common markdown actions. It includes bolding text, italicizing text, inserting code snippets, inserting unordered list, inserting ordered list and inserting image from URL.

Format button toolbar is not available for the preview notebook experience yet.

Undo cell operations

Select the undo button or press Ctrl+Z to revoke the most recent cell operation. Now you can undo up to the latest 20 historical cell actions.

Undo cell operation is not available for the preview notebook experience yet.

Move a cell

Select the ellipses (..) to access the other cell actions menu at the far right. Then select Move cell up or Move cell down to move the current cell.

You can also use shortcut keys under command mode. Press Ctrl+Alt+↑ to move up the current cell. Press Ctrl+Alt+↓ to move the current cell down.

Click on the left-hand side of a cell and drag it to the desired position.

Delete a cell

To delete a cell, select the ellipses (..) to access the other cell actions menu at the far right then select Delete cell.

You can also use shortcut keys under command mode. Press D,D to delete the current cell.

To delete a cell, select the delete button at the right hand of the cell.

You can also use shortcut keys under command mode. Press Shift+D to delete the current cell.

Collapse a cell input

Select the arrow button at the bottom of the current cell to collapse it. To expand it, select the arrow button while the cell is collapsed.

Select the More commands ellipses (..) on the cell toolbar and input to collapse current cell's input. To expand it, Select the input hidden while the cell is collapsed.

Collapse a cell output

Select the collapse output button at the upper left of the current cell output to collapse it. To expand it, select the Show cell output while the cell output is collapsed.

Select the More commands ellipses (..) on the cell toolbar and output to collapse current cell's output. To expand it, select the same button while the cell's output is hidden.

Notebook outline

Not supported.

The Outlines (Table of Contents) presents the first markdown header of any markdown cell in a sidebar window for quick navigation. The Outlines sidebar is resizable and collapsible to fit the screen in the best ways possible. You can select the Outline button on the notebook command bar to open or hide sidebar

Run notebooks

You can run the code cells in your notebook individually or all at once. The status and progress of each cell is represented in the notebook.

Run a cell

There are several ways to run the code in a cell.

1. Hover on the cell you want to run and select the Run Cell button or press Ctrl+Enter.

2. Use Shortcut keys under command mode. Press Shift+Enter to run the current cell and select the cell below. Press Alt+Enter to run the current cell and insert a new cell below.

Run all cells

Select the Run All button to run all the cells in current notebook in sequence.

Run all cells above or below

To Access the other cell actions menu at the far right, select the ellipses (..). Then, select Run cells above to run all the cells above the current in sequence. Select Run cells below to run all the cells below the current in sequence.

Expand the dropdown list from Run all button, then select Run cells above to run all the cells above the current in sequence. Select Run cells below to run all the cells below the current in sequence.

Cancel all running cells

Select the Cancel All button to cancel the running cells or cells waiting in the queue.

Select the Cancel All button to cancel the running cells or cells waiting in the queue.

Notebook reference

Not supported.

You can use %run magic command to reference another notebook within current notebook's context. All the variables defined in the reference notebook are available in the current notebook. %run magic command supports nested calls but not support recursive calls. You will receive an exception if the statement depth is larger than five. %run command currently only supports to pass a notebook path as parameter.

Example: %run /path/notebookA.

Note

Notebook reference is not supported in Synapse pipeline.

Variable explorer

Not supported.

Synapse notebook provides a built-in variables explorer for you to see the list of the variables name, type, length, and value in the current Spark session for PySpark (Python) cells. More variables will show up automatically as they are defined in the code cells. Clicking on each column header will sort the variables in the table.

You can select the Variables button on the notebook command bar to open or hide the variable explorer.

Cell status indicator

A step-by-step cell execution status is displayed beneath the cell to help you see its current progress. Once the cell run is complete, an execution summary with the total duration and end time are shown and kept there for future reference.

Spark progress indicator

Azure Synapse Studio notebook is purely Spark based. Code cells are executed on the serverless Apache Spark pool remotely. A Spark job progress indicator is provided with a real-time progress bar appears to help you understand the job execution status.The number of tasks per each job or stage help you to identify the parallel level of your spark job. You can also drill deeper to the Spark UI of a specific job (or stage) via selecting the link on the job (or stage) name.

Spark session config

You can specify the timeout duration, the number, and the size of executors to give to the current Spark session in Configure session. Restart the Spark session is for configuration changes to take effect. All cached notebook variables are cleared.

Spark session config magic command

You can also specify spark session settings via a magic command %%configure. The spark session needs to restart to make the settings effect. We recommend you to run the %%configure at the beginning of your notebook. Here is a sample, refer to https://github.com/cloudera/livy#request-body for full list of valid parameters

Note

Spark session config magic command is not supported in Synapse pipeline.

Bring data to a notebook

Sophos kba 119175. You can load data from Azure Blob Storage, Azure Data Lake Store Gen 2, and SQL pool as shown in the code samples below.

Read a CSV from Azure Data Lake Store Gen2 as a Spark DataFrame

Read a CSV from Azure Blob Storage as a Spark DataFrame

Read data from the primary storage account

You can access data in the primary storage account directly. There's no need to provide the secret keys. In Data Explorer, right-click on a file and select New notebook to see a new notebook with data extractor autogenerated.

Save notebooks

You can save a single notebook or all notebooks in your workspace.

1. To save changes you made to a single notebook, select the Publish button on the notebook command bar.

2. To save all notebooks in your workspace, select the Publish all button on the workspace command bar.

In the notebook properties, you can configure whether to include the cell output when saving.

Magic commands

You can use familiar Jupyter magic commands in Azure Synapse Studio notebooks. Review the following list as the current available magic commands. Tell us your use cases on GitHub so that we can continue to build out more magic commands to meet your needs.

Note

Only following magic commands are supported in Synapse pipeline : %%pyspark, %%spark, %%csharp, %%sql.

Available line magics:%lsmagic, %time, %timeit

Available cell magics:%%time, %%timeit, %%capture, %%writefile, %%sql, %%pyspark, %%spark, %%csharp,%%configure

Available line magics:%lsmagic, %time, %timeit, %history, %run, %load

Available cell magics:%%time, %%timeit, %%capture, %%writefile, %%sql, %%pyspark, %%spark, %%csharp, %%html, %%configure

Integrate a notebook

Add a notebook to a pipeline

Select the Add to pipeline button on the upper right corner to add a notebook to an existing pipeline or create a new pipeline.

Designate a parameters cell

To parameterize your notebook, select the ellipses (..) to access the other cell actions menu at the far right. Then select Toggle parameter cell to designate the cell as the parameters cell.

To parameterize your notebook, select the ellipses (..) to access the more commands at the cell toolbar. Then select Toggle parameter cell to designate the cell as the parameters cell.

Azure Data Factory looks for the parameters cell and treats this cell as defaults for the parameters passed in at execution time. The execution engine will add a new cell beneath the parameters cell with input parameters in order to overwrite the default values. When a parameters cell isn't designated, the injected cell will be inserted at the top of the notebook.

Assign parameters values from a pipeline

Once you've created a notebook with parameters, you can execute it from a pipeline with the Azure Synapse Notebook activity. After you add the activity to your pipeline canvas, you will be able to set the parameters values under Base parameters section on the Settings tab.

When assigning parameter values, you can use the pipeline expression language or system variables.

Shortcut keys

Similar to Jupyter Notebooks, Azure Synapse Studio notebooks have a modal user interface. The keyboard does different things depending on which mode the notebook cell is in. Synapse Studio notebooks support the following two modes for a given code cell: command mode and edit mode.

1. A cell is in command mode when there is no text cursor prompting you to type. When a cell is in Command mode, you can edit the notebook as a whole but not type into individual cells. Enter command mode by pressing ESC or using the mouse to select outside of a cell's editor area.

2. Edit mode is indicated by a text cursor prompting you to type in the editor area. When a cell is in edit mode, you can type into the cell. Enter edit mode by pressing Enter or using the mouse to select on a cell's editor area.

Shortcut keys under command mode

Using the following keystroke shortcuts, you can more easily navigate and run code in Azure Synapse notebooks.

Shortcut keys under edit mode

Using the following keystroke shortcuts, you can more easily navigate and run code in Azure Synapse notebooks when in Edit mode.

Next steps

You can manage notebooks using the UI, the CLI, and by invoking the Workspace API. This article focuses on performing notebook tasks using the UI. For the other methods, see Databricks CLI and Workspace API.

Create a notebook

1. Click the Workspace button or the Home button in the sidebar. Do one of the following:

   • Next to any folder, click the on the right side of the text and select Create > Notebook.

   • In the Workspace or a user folder, click and select Create > Notebook.

2. In the Create Notebook dialog, enter a name and select the notebook's default language.

3. If there are running clusters, the Cluster drop-down displays. Select the cluster you want to attach the notebook to.

4. Click Create.

Open a notebook

In your workspace, click a . The notebook path displays when you hover over the notebook title.

Delete a notebook

See Folders and Workspace object operations for information about how to access the workspace menu and delete notebooks or other items in the Workspace.

Copy notebook path

To copy a notebook file path without opening the notebook, right-click the notebook name or click the to the right of the notebook name and select Copy File Path.

Rename a notebook

To change the title of an open notebook, click the title and edit inline or click File > Rename.

Control access to a notebook

If your Databricks account has the Premium plan (or, for customers who subscribed to Databricks before March 3, 2020, the Operational Security package), you can use Workspace access control to control who has access to a notebook.

Notebook external formats

Databricks supports several notebook external formats:

• Source file: A file containing only source code statements with the extension .scala, .py, .sql, or .r.
• HTML: A Databricks notebook with the extension .html.
• DBC archive: A Databricks archive.
• IPython notebook: A Jupyter notebook with the extension .ipynb.
• RMarkdown: An R Markdown document with the extension .Rmd.

In this section:

You can import an external notebook from a URL or a file.

1. Click the Workspace button or the Home button in the sidebar. Do one of the following:

   • Next to any folder, click the on the right side of the text and select Import.

   • In the Workspace or a user folder, click and select Import.

2. Specify the URL or browse to a file containing a supported external format.

3. Click Import.

Titanium atomic number electron configuration. In the notebook toolbar, select File > Export and a format.

Note

When you export a notebook as HTML, IPython notebook, or archive (DBC), and you have not cleared the results, the results of running the notebook are included.

If you're using Community Edition, you can publish a notebook so that you can share a URL path to the notebook. Subsequent publish actions update the notebook at that URL.

Notebooks and clusters

Before you can do any work in a notebook, you must first attach the notebook to a cluster. This section describes how to attach and detach notebooks to and from clusters and what happens behind the scenes when you perform these actions.

In this section:

When you attach a notebook to a cluster, Databricks creates an execution context. An execution context contains the state for a REPL environment for each supported programming language: Python, R, Scala, and SQL. When you run a cell in a notebook, the command is dispatched to the appropriate language REPL environment and run.

You can also use the REST 1.2 API to create an execution context and send a command to run in the execution context. Similarly, the command is dispatched to the language REPL environment and run.

Markdown In Databricks

A cluster has a maximum number of execution contexts (145). Once the number of execution contexts has reached this threshold, you cannot attach a notebook to the cluster or create a new execution context.

Idle execution contexts

An execution context is considered idle when the last completed execution occurred past a set idle threshold. Last completed execution is the last time the notebook completed execution of commands. The idle threshold is the amount of time that must pass between the last completed execution and any attempt to automatically detach the notebook. The default idle threshold is 24 hours.

When a cluster has reached the maximum context limit, Databricks removes (evicts) idle execution contexts (starting with the least recently used) as needed. Even when a context is removed, the notebook using the context is still attached to the cluster and appears in the cluster's notebook list. Streaming notebooks are considered actively running, and their context is never evicted until their execution has been stopped. If an idle context is evicted, the UI displays a message indicating that the notebook using the context was detached due to being idle.

If you attempt to attach a notebook to cluster that has maximum number of execution contexts and there are no idle contexts (or if auto-eviction is disabled), the UI displays a message saying that the current maximum execution contexts threshold has been reached and the notebook will remain in the detached state.

If you fork a process, an idle execution context is still considered idle once execution of the request that forked the process returns. Forking separate processes is not recommended with Spark.

Configure context auto-eviction

You can configure context auto-eviction by setting the Spark propertyspark.databricks.chauffeur.enableIdleContextTracking.

• In Databricks 5.0 and above, auto-eviction is enabled by default. You disable auto-eviction for a cluster by setting spark.databricks.chauffeur.enableIdleContextTrackingfalse.
• In Databricks 4.3, auto-eviction is disabled by default. You enable auto-eviction for a cluster by setting spark.databricks.chauffeur.enableIdleContextTrackingtrue.

To attach a notebook to a cluster:

1. In the notebook toolbar, click Detached .
2. From the drop-down, select a cluster.

Important

An attached notebook has the following Apache Spark variables defined.

Do not create a SparkSession, SparkContext, or SQLContext. Doing so will lead to inconsistent behavior.

Determine Spark and Databricks Runtime version

To determine the Spark version of the cluster your notebook is attached to, run:

To determine the Databricks Runtime version of the cluster your notebook is attached to, run:

Python

Note

Databricks Markdown Syntax

Both this sparkVersion tag and the spark_version property required by the endpoints in the Clusters API and Jobs API refer to the Databricks Runtime version, not the Spark version.

1. In the notebook toolbar, click Attached .

2. Select Detach.

Databricks Notebook Markdown Cell

You can also detach notebooks from a cluster using the Notebooks tab on the cluster details page.

When you detach a notebook from a cluster, the execution context is removed and all computed variable values are cleared from the notebook.

Tip

Databricks recommends that you detach unused notebooks from a cluster. This frees up memory space on the driver.

The Notebooks tab on the cluster details page displays all of the notebooks that are attached to a cluster. The tab also displays the status of each attached notebook, along with the last time a command was run from the notebook.

Schedule a notebook

To schedule a notebook job to run periodically:

1. In the notebook toolbar, click the button at the top right.
2. Click + New.
3. Choose the schedule.
4. Click OK.

Distribute notebooks

Cell status indicator

A step-by-step cell execution status is displayed beneath the cell to help you see its current progress. Once the cell run is complete, an execution summary with the total duration and end time are shown and kept there for future reference.

Spark progress indicator

Azure Synapse Studio notebook is purely Spark based. Code cells are executed on the serverless Apache Spark pool remotely. A Spark job progress indicator is provided with a real-time progress bar appears to help you understand the job execution status.The number of tasks per each job or stage help you to identify the parallel level of your spark job. You can also drill deeper to the Spark UI of a specific job (or stage) via selecting the link on the job (or stage) name.

Spark session config

You can specify the timeout duration, the number, and the size of executors to give to the current Spark session in Configure session. Restart the Spark session is for configuration changes to take effect. All cached notebook variables are cleared.

Spark session config magic command

You can also specify spark session settings via a magic command %%configure. The spark session needs to restart to make the settings effect. We recommend you to run the %%configure at the beginning of your notebook. Here is a sample, refer to https://github.com/cloudera/livy#request-body for full list of valid parameters

Note

Spark session config magic command is not supported in Synapse pipeline.

Bring data to a notebook

Sophos kba 119175. You can load data from Azure Blob Storage, Azure Data Lake Store Gen 2, and SQL pool as shown in the code samples below.

Read a CSV from Azure Data Lake Store Gen2 as a Spark DataFrame

Read a CSV from Azure Blob Storage as a Spark DataFrame

Read data from the primary storage account

You can access data in the primary storage account directly. There's no need to provide the secret keys. In Data Explorer, right-click on a file and select New notebook to see a new notebook with data extractor autogenerated.

Save notebooks

You can save a single notebook or all notebooks in your workspace.

1. To save changes you made to a single notebook, select the Publish button on the notebook command bar.

2. To save all notebooks in your workspace, select the Publish all button on the workspace command bar.

In the notebook properties, you can configure whether to include the cell output when saving.

Magic commands

You can use familiar Jupyter magic commands in Azure Synapse Studio notebooks. Review the following list as the current available magic commands. Tell us your use cases on GitHub so that we can continue to build out more magic commands to meet your needs.

Note

Only following magic commands are supported in Synapse pipeline : %%pyspark, %%spark, %%csharp, %%sql.

Available line magics:%lsmagic, %time, %timeit

Available cell magics:%%time, %%timeit, %%capture, %%writefile, %%sql, %%pyspark, %%spark, %%csharp,%%configure

Available line magics:%lsmagic, %time, %timeit, %history, %run, %load

Available cell magics:%%time, %%timeit, %%capture, %%writefile, %%sql, %%pyspark, %%spark, %%csharp, %%html, %%configure

Integrate a notebook

Add a notebook to a pipeline

Select the Add to pipeline button on the upper right corner to add a notebook to an existing pipeline or create a new pipeline.

Designate a parameters cell

To parameterize your notebook, select the ellipses (..) to access the other cell actions menu at the far right. Then select Toggle parameter cell to designate the cell as the parameters cell.

To parameterize your notebook, select the ellipses (..) to access the more commands at the cell toolbar. Then select Toggle parameter cell to designate the cell as the parameters cell.

Azure Data Factory looks for the parameters cell and treats this cell as defaults for the parameters passed in at execution time. The execution engine will add a new cell beneath the parameters cell with input parameters in order to overwrite the default values. When a parameters cell isn't designated, the injected cell will be inserted at the top of the notebook.

Assign parameters values from a pipeline

Once you've created a notebook with parameters, you can execute it from a pipeline with the Azure Synapse Notebook activity. After you add the activity to your pipeline canvas, you will be able to set the parameters values under Base parameters section on the Settings tab.

When assigning parameter values, you can use the pipeline expression language or system variables.

Shortcut keys

Similar to Jupyter Notebooks, Azure Synapse Studio notebooks have a modal user interface. The keyboard does different things depending on which mode the notebook cell is in. Synapse Studio notebooks support the following two modes for a given code cell: command mode and edit mode.

1. A cell is in command mode when there is no text cursor prompting you to type. When a cell is in Command mode, you can edit the notebook as a whole but not type into individual cells. Enter command mode by pressing ESC or using the mouse to select outside of a cell's editor area.

2. Edit mode is indicated by a text cursor prompting you to type in the editor area. When a cell is in edit mode, you can type into the cell. Enter edit mode by pressing Enter or using the mouse to select on a cell's editor area.

Shortcut keys under command mode

Using the following keystroke shortcuts, you can more easily navigate and run code in Azure Synapse notebooks.

Shortcut keys under edit mode

Using the following keystroke shortcuts, you can more easily navigate and run code in Azure Synapse notebooks when in Edit mode.

Next steps

You can manage notebooks using the UI, the CLI, and by invoking the Workspace API. This article focuses on performing notebook tasks using the UI. For the other methods, see Databricks CLI and Workspace API.

Create a notebook

1. Click the Workspace button or the Home button in the sidebar. Do one of the following:

   • Next to any folder, click the on the right side of the text and select Create > Notebook.

   • In the Workspace or a user folder, click and select Create > Notebook.

2. In the Create Notebook dialog, enter a name and select the notebook's default language.

3. If there are running clusters, the Cluster drop-down displays. Select the cluster you want to attach the notebook to.

4. Click Create.

Open a notebook

In your workspace, click a . The notebook path displays when you hover over the notebook title.

Delete a notebook

See Folders and Workspace object operations for information about how to access the workspace menu and delete notebooks or other items in the Workspace.

Copy notebook path

To copy a notebook file path without opening the notebook, right-click the notebook name or click the to the right of the notebook name and select Copy File Path.

Rename a notebook

To change the title of an open notebook, click the title and edit inline or click File > Rename.

Control access to a notebook

If your Databricks account has the Premium plan (or, for customers who subscribed to Databricks before March 3, 2020, the Operational Security package), you can use Workspace access control to control who has access to a notebook.

Notebook external formats

Databricks supports several notebook external formats:

• Source file: A file containing only source code statements with the extension .scala, .py, .sql, or .r.
• HTML: A Databricks notebook with the extension .html.
• DBC archive: A Databricks archive.
• IPython notebook: A Jupyter notebook with the extension .ipynb.
• RMarkdown: An R Markdown document with the extension .Rmd.

In this section:

You can import an external notebook from a URL or a file.

1. Click the Workspace button or the Home button in the sidebar. Do one of the following:

   • Next to any folder, click the on the right side of the text and select Import.

   • In the Workspace or a user folder, click and select Import.

2. Specify the URL or browse to a file containing a supported external format.

3. Click Import.

Titanium atomic number electron configuration. In the notebook toolbar, select File > Export and a format.

Note

When you export a notebook as HTML, IPython notebook, or archive (DBC), and you have not cleared the results, the results of running the notebook are included.

If you're using Community Edition, you can publish a notebook so that you can share a URL path to the notebook. Subsequent publish actions update the notebook at that URL.

Notebooks and clusters

Before you can do any work in a notebook, you must first attach the notebook to a cluster. This section describes how to attach and detach notebooks to and from clusters and what happens behind the scenes when you perform these actions.

In this section:

When you attach a notebook to a cluster, Databricks creates an execution context. An execution context contains the state for a REPL environment for each supported programming language: Python, R, Scala, and SQL. When you run a cell in a notebook, the command is dispatched to the appropriate language REPL environment and run.

You can also use the REST 1.2 API to create an execution context and send a command to run in the execution context. Similarly, the command is dispatched to the language REPL environment and run.

Markdown In Databricks

A cluster has a maximum number of execution contexts (145). Once the number of execution contexts has reached this threshold, you cannot attach a notebook to the cluster or create a new execution context.

Idle execution contexts

An execution context is considered idle when the last completed execution occurred past a set idle threshold. Last completed execution is the last time the notebook completed execution of commands. The idle threshold is the amount of time that must pass between the last completed execution and any attempt to automatically detach the notebook. The default idle threshold is 24 hours.

When a cluster has reached the maximum context limit, Databricks removes (evicts) idle execution contexts (starting with the least recently used) as needed. Even when a context is removed, the notebook using the context is still attached to the cluster and appears in the cluster's notebook list. Streaming notebooks are considered actively running, and their context is never evicted until their execution has been stopped. If an idle context is evicted, the UI displays a message indicating that the notebook using the context was detached due to being idle.

If you attempt to attach a notebook to cluster that has maximum number of execution contexts and there are no idle contexts (or if auto-eviction is disabled), the UI displays a message saying that the current maximum execution contexts threshold has been reached and the notebook will remain in the detached state.

If you fork a process, an idle execution context is still considered idle once execution of the request that forked the process returns. Forking separate processes is not recommended with Spark.

Configure context auto-eviction

You can configure context auto-eviction by setting the Spark propertyspark.databricks.chauffeur.enableIdleContextTracking.

• In Databricks 5.0 and above, auto-eviction is enabled by default. You disable auto-eviction for a cluster by setting spark.databricks.chauffeur.enableIdleContextTrackingfalse.
• In Databricks 4.3, auto-eviction is disabled by default. You enable auto-eviction for a cluster by setting spark.databricks.chauffeur.enableIdleContextTrackingtrue.

To attach a notebook to a cluster:

1. In the notebook toolbar, click Detached .
2. From the drop-down, select a cluster.

Important

An attached notebook has the following Apache Spark variables defined.

Do not create a SparkSession, SparkContext, or SQLContext. Doing so will lead to inconsistent behavior.

Determine Spark and Databricks Runtime version

To determine the Spark version of the cluster your notebook is attached to, run:

To determine the Databricks Runtime version of the cluster your notebook is attached to, run:

Python

Note

Databricks Markdown Syntax

Both this sparkVersion tag and the spark_version property required by the endpoints in the Clusters API and Jobs API refer to the Databricks Runtime version, not the Spark version.

1. In the notebook toolbar, click Attached .

2. Select Detach.

Databricks Notebook Markdown Cell

You can also detach notebooks from a cluster using the Notebooks tab on the cluster details page.

When you detach a notebook from a cluster, the execution context is removed and all computed variable values are cleared from the notebook.

Tip

Databricks recommends that you detach unused notebooks from a cluster. This frees up memory space on the driver.

The Notebooks tab on the cluster details page displays all of the notebooks that are attached to a cluster. The tab also displays the status of each attached notebook, along with the last time a command was run from the notebook.

Schedule a notebook

To schedule a notebook job to run periodically:

1. In the notebook toolbar, click the button at the top right.
2. Click + New.
3. Choose the schedule.
4. Click OK.

Distribute notebooks

To allow you to easily distribute Databricks notebooks, Databricks supports the Databricks archive, which is a package that can contain a folder of notebooks or a single notebook. A Databricks archive is a JAR file with extra metadata and has the extension .dbc. The notebooks contained in the archive are in a Databricks internal format.

Databricks %run Notebook

Import an archive

1. Click or to the right of a folder or notebook and select Import.
2. Choose File or URL.
3. Go to or drop a Databricks archive in the dropzone.
4. Click Import. The archive is imported into Databricks. If the archive contains a folder, Databricks recreates that folder.

Markdown Cells In Databricks Notebooks

Export an archive

Databricks Notebook Markup

Click or to the right of a folder or notebook and select Export > DBC Archive. Databricks downloads a file named <[folder|notebook]-name>.dbc.