Running Scripts using Azure Data Factory and Batch, Part I
Power BI is Microsoft’s data visualization solution

A common job in orchestration is to run a python or R script within a pipeline. To achieve this, one can run scripts using Azure Data Factory (ADF) and Azure Batch.

The following is an example on how to run a script using ADF and Azure Batch. Before starting, make sure you have and batch account and a pool, and a storage account.

Say you would like to run the script in ADF using Azure Batch:

print('hello world')

To run this script in a pipeline:

  1. From Azure Batch, go to Blob service > Containers


  2. Click on + Container


  3. Name your new script container and click on Create


  4. Access the script container


  5. Click on Upload


  6. Locate the script in your local folders and upload


  7. Navigate to the ADF portal. Click on Manage


  8. Go to Linked services > + New


  9. Under Data store, select Azure Blob Storage > Continue


  10. Name the linked service. Select the Azure Subscription under which the storage account was created. From the drop-down menu, select your Storage account name > Create


  11. Create a new linked service by going to Linked services > +New. Under Compute services. Select Azure Batch > Continue


  12. To find the required Batch information, refer to Azure Batch. Go to Settings > Keys


  13. Here you will find three of the fields required: Account Name, Access Key and Batch URL


  14. To find the Pool Name, refer to Features > Pools on the left-hand side menu from your main batch account site.


  15. The Pool ID is the Pool name.


  16. Go back to the ADF portal, and fill out the information required. Under Storage linked service name, select the linked service for the storage account created above. Create


Now you have created the linked services to Azure Batch and Storage. In part II, you find the steps to create and run the pipeline.

Running Scripts using Azure Data Factory and Batch, Part I
Older post

4 x 4 x 48

Going through with the running challenge popularized by David Goggins

Newer post

Running Scripts using Azure Data Factory and Batch, Part II

An example on how to run scripts within an Azure Data Factory pipeline

Running Scripts using Azure Data Factory and Batch, Part I