0

I have registered a custom environment (through Environments tab) in Azure Machine Learning (providing Docker file and conda_dependencies.yaml) which I plan to use for training jobs. Now I would like to be able to use this environment also in AzureML notebooks for experimentation and development of code that I finally plan to use for the training jobs.

I am wondering whether there is a way to create a kernel in AzureML notebook that is based on the registered environment from the previous step?

I have followed multiple tutorials to do this, in particular:

I am able to create new conda venv in AzureML notebook (in terminal session from conda_dependencies.yaml file or simply as new venv and install packages manually one by one) and create a new kernel based on this venv. What I am not able to do is to create the venv based on the registered environment that I have already created in the Environments tab. Based on the documentation (link) there should be a way to do this by using Environment.build_local() method, but it does not work for me.

I tried to run the following code:

from azureml.core.workspace import Workspace
from azureml.core import Environment

ws = Workspace.from_config()

myenv = Environment.get(ws, 'my_previously_registered_environment')

myenv.build_local(ws)

This code runs only for few seconds and outputs the following:

Saving setup content into  /tmp/tmpXYZXYZXY

No new venv is created, at least I do not see it by running:

%conda env list

Am I using the Environment.build_local() method in a wrong way (or with wrong parameters)?

If there is no way to create venv in AzureML notebooks from registered environment using Environment.build_local(), is there at least a way to get the conda_dependencies.yaml file (that I used during environment registration) from AzureML notebook (e.g. is it saved somewhere in storage where I can access it from Python)? I am trying to be able to create the venv and kernel based on the registered environment by running a (simple) script(s) without any manual copying.

Michal Nožička
  • 3
  • 1

1 Answers1

0

I have created a custom environment with conda.yaml file enter image description here

To get the yaml file from the custom environment you can use below code snippet:

from azureml.core.workspace import Workspace
from azureml.core import Environment
import os

ws = Workspace.from_config()
myenv = Environment.get(ws, 'my_conda_envtest')
myenv.save_to_directory("env2.yaml")

Above code will save the conda_dependencies.yml in env2.yaml directory. enter image description here

enter image description here

Then you can use this conda_dependencies.yml file to create conda env and kernel .

RishabhM
  • 525
  • 1
  • 5
  • Thank you for your answer! Unfortunatelly, I am not able to reproduce your results. When I run your code I do not see conda_dependencies.yml file in the env2.yaml directory. Can you maybe share the whole definition of your my_conda_envtest environment (i.e. the Docker file)? Thanks. – Michal Nožička Jul 26 '23 at 15:46
  • If you are creating the custom environment with Docker file, the function will create only azureml_environment.json file. You will get the yaml file only if you create the environment with conda.yaml file (As shown in the solution with Environment Context) – RishabhM Jul 27 '23 at 05:29
  • I see, now I am able to get the same results. Just for the clarification: When I was creating the Environment through the UI, I had to select the option "Use existing docker image with optional conda file" and then populate the conda file (only the conda file). In case I selected "Create a new docker context" and created both new docker file and conda file, then I was not able to get the conda file from notebook. I do not know the reason for this behavior, but at least now have a working solution. Thank you very much @RishabhM for your answers, they were very helpful for me! – Michal Nožička Jul 27 '23 at 08:24
  • Plus additionally, if you define the environment in the way described in the previous comment (only through conda file, not Docker + conda files), then it works to call just the `build_local()` function as described in the original post. – Michal Nožička Jul 27 '23 at 10:58