Schedule the notebook in databricks

Schedule the notebook in databricks is a programming solution to schedule or run your databricks notebook hourly, daily, or monthly. we can schedule a notebook in the databricks workspace by creating the databricks job.

Databricks Notebooks are like a user interface(UI) that contains a group of command cells. We can use any of the databricks supported languages to code in command cells. These command cell can be run individually, can be scheduled for continuous execution with thenhelp of databricks JOBS.

 

Steps to create the databricks job to schedule a databricks notebook

Steps to create a JOB:
Step 1:
Enter portal.azure.com in browser

Step2:

Select the Azure Resources

Step 3:
If  the databricks workspace already created, filter out the azure workspaces and select the required workspace from the azure resources displayed.

Step 4:
Open the databricks cluster if its already created and attached to your notebook

Step 5:
Navigate to the Notebooks tab option in the datarbricks cluster

Step 6:
Select the notebook you want to schedule

Step 7:
Within the notebook, there is a Schedule option in the menu bar. Click on that.

Step 8:
Now, there is a sidebar on the right-side with the JOB button. Click on that

Step 9:
Now the JOB creation menu opened.

Step 10: 
Enter the JOB name

Step 11:

Select the notebook you want to attach to the Job

Step 12:
Enter any parameters need to pass it from Job to databricks Notebook

Step 13:
Select any dependency libraries(JAR/DBFS) you want to attach to Job(any external libraries you  installed for notebook)

Step 14:
You can set the time interval(one-time/daily) to run the Job.

Step 15:

We can set the advanced settings like below:

a. Mail trigger in case of any failures

b. Mail trigger in case of success

c. Retry count in case of any failures.

That's it!!! your JOB is ready to run...

Conclusion

You can also navigate to the databricks JOBs created from the left navigation menu in the databricks workspace.