Set Connection References and Environment Variables in Pipelines
Since Microsoft has released both Connection References and Environment Variables the process of setting them in a Pipeline wasn’t very straightforward. Scott Durow (Twitter & LinkedIn) explained a possible solution to it in one of his blog posts (Connection References with ALM β Mind the Gap!). In this blog post, I will explain a different way to set Connection References and Environment Variables in Pipelines.
The approach I am explaining and demonstrating is as of writing (2021-12-01) still in preview.
Update: As of 2021-12-15 the explained approach is GA. See MS announcement.
Since parts of the approach are still in preview you would use it at your own risk. I personally figure this still might be better than handling everything manually, but more about that later.
Problem
First of all, we should talk a bit about the problems that we currently have when it comes to ALM and Connection References as well as Environment Variables.
Manual deployment
When you are doing manual deployments it should work pretty well most of the time. When you import a solution that contains either Connection References you don’t have associated with a connection or Environment Variables where neither a default value nor current value exists an additional step in the import will be shown where you can configure those things.
Automated deployment
Bigger problems arise when it comes to automated deployments. Since there isn’t a human interaction possible no one could associate connections to connection references or assign values to environment variables.
This means those things have to happen after the deployment is finished. Either manually or through some PowerShell Scripting (some might call it “voodoo”) as Scott describes in the mentioned blog.
A result is that all Power Automate flows using a connection reference without an associated connection will get inactivated during import.
Don’t get me wrong, I do like the idea of both of them. The problem was/is “only” the ALM part.
Solution
Fortunately, Microsoft listens to feedback and comes to the rescue. The Import Solution Task of the Power Platform Build Tools does have two new configurations UseDeploymentSettingsFile and DeploymentSettingsFile.
What you have to do is basically create a settings file where you map Environment Variables to values and Connection References to in your target environment already existing connections.
The step will then perform the assigning and association part itself. This should lead to smooth deployments including activated flows.
Let’s see how we could use this part of the Import Solution task.
Prerequisits
To be able to follow this blog post we need to make sure we have some prerequisites in place.
ALM Process
I will base my explanation on the Basic ALM process I explained in an earlier blog post.
To summarize it very shortly:
The mentioned Basic ALM process does contain 3 build/release pipelines
- Create export from DEV
- Build Managed Solution
- Release (to Test)
During the further explanations within this blog post, we will do changes in the last two of those pipelines.
Power Platform CLI
In addition, you will need the Power Platform CLI installed on your machine.
Solution
Prepare a solution containing at least one Environment Variable and one Connection Reference to be able to test the process.
My solution looks like the following and is called “DevToolsTest“.
Create Settings file
The first step would be to create our Settings file. There are several ways of doing it (as described in the docs). We will do it based on a solution file.
Export Solution
We export the solution we created as part of the prerequisites unmanaged and save it somewhere on our machine.
Run pac
Open a Command Prompt (or PowerShell) in the folder which contains the solution zip file.
The following command will create a settings file (called settings-test.json) based on your solution in the same folder. You have to replace the name of the zip file with yours.
Make sure you have the latest version of the pac installed by executing “pac install latest” first.
pac solution create-settings --solution-zip ./DevToolsTest_1_0_0_1.zip --settings-file ./settings-test.json
In my case, the settings file looks like this
{ "EnvironmentVariables": [ { "SchemaName": "bebe_EnvVarTest", "Value": "" } ], "ConnectionReferences": [ { "LogicalName": "bebe_sharedcommondataserviceforapps_e904a", "ConnectionId": "", "ConnectorId": "/providers/Microsoft.PowerApps/apis/shared_commondataserviceforapps" } ] }
What you can see is that there is an array of Environment Variables where you could add the desired value. The file also contains an array of Connection References, what missing here is the ID of the connection (in the target environment) you would like to associate.
For the ease of this post, I will hard code all of that. For sure you could dynamically set/fill this file as a part of your pipeline. What I found most practical is having a settings file for every target environment. Much like the app.config one knows from C# projects.
If you have several static settings files those have to be handled manually in the future. For example when new Environment Variables or Connection References should be added.
The settings file should then be stored within your repository. I use a Settings folder for it. If you follow my “Folder structure of a Dataverse project“, I would put it under “PowerPlatform”
Find connection id
One step which could be a bit confusing is finding the internal Id of a connection. It is though as easy as opening the connection in the target environment and copying the Id from the url.
Finished settings file
My finished settings file looks like the following.
{ "EnvironmentVariables": [ { "SchemaName": "bebe_EnvVarTest", "Value": "Test Value" } ], "ConnectionReferences": [ { "LogicalName": "bebe_sharedcommondataserviceforapps_*****", "ConnectionId": "6ea97b9cf5ae4bc0ad52************", "ConnectorId": "/providers/Microsoft.PowerApps/apis/shared_commondataserviceforapps" } ] }
Change Pipeline
So far so good, but what do we have to change in our pipeline to be able to use this.
Since we use static settings files we “only” have to add them to the artefact of our second pipeline (Build Managed solution) and use it in our Import Solution task of the release pipeline.
Build Managed solution
We have to change the following two things in this pipeline.
Copy Files
As the penultimate task in our pipeline, we add a “copy file” task.
As “Source Folder” we select the folder in our repository where we stored the settings file(s).
As “Contents” we write two stars (**) to fetch everything.
As “Target Folder” choose the artefact staging directory and “Settings” as the subfolder
$(Build.ArtifactStagingDirectory)\Settings
Publish Pipeline Artifacts
We also have to alter the last task (Publish Pipeline Artifacts) in a way that takes the whole staging folder and not only the solution zip file.
To do so just change “File or directory path” to the following.
$(Build.ArtifactStagingDirectory)
Release to Test
All that is left to set Connection References and Environment Variables in Pipelines is to change our Release to Test release pipeline to actually use the settings file.
To do so we check the checkbox “Use deployment settings file” and choose the correct file from the artifact.
Conclusion
As you can see this approach to set Connection References and Environment Variables in Pipelines is much easier than running different PowerShell scripts. That’s at least what I think. As mentioned this part of the import step is still in preview, but hopefully, it will be GA soon. Since 2021-12-15 this approach is GA and can safely be used in Production.
I hope this post helped you to come a bit closer to your ALM goal. Please feel free to contact me if you have any further questions.
You can also subscribe and get new blog posts emailed to you directly.
Hi Benedikt,
activating the flows after setting the connmectionreferences is still necessary, isn’t it?
regards
Lars
Hej Lars,
when they were deployed the first time directly using this method they should be activated automatically. Otherwise it is still necessary yes.
“pac install latest” This step is much too unclear. This neither works trough powershell: “pac: The term ‘pac’ is not recognized as a name of a cmdlet, function, script file, or executable program.”
Neither using this as an example, still getting the same error using in YAML pipeline :
https://github.com/BenediktBergmann/AzureDevOpsPipelines/blob/master/ExecutePAC.yaml
Getting error: “pac : The term ‘pac’ is not recognized as a name of a cmdlet, function, script file, or executable program.” When trying to execute steps after installing nuget and pac that is: “pac install latest” and in the step defined as “pac solution create-settings –solution-zip ./DevToolsTest_1_0_0_1.zip –settings-file ./settings-test.json”
I am sorry to hear that.
The error looks like the pac wasn’t installed successfully on your machine. To be able to run “pac install latest” you have to have pac installed already.
Have you followed all the prerequisites outlined in the post?
https://benediktbergmann.eu/2021/12/02/set-connection-references-and-environment-variables-in-pipelines/#power_platform_cli
In addition to that is here a link with prerequisites for the cli
https://docs.microsoft.com/en-gb/learn/modules/build-power-app-component/1-create-code-component
I just tried it and it does work for me from a cmd window. It does not work from VS Code Terminal though. There I get a different error than you got but it still does not work.
Sorry, my bad. Turns out the “finding path” part is necessary, I thought it is just double-checking. With this step in place it works out wonders. Thanks!!!
How would you go about deploying and environment variable of the type JSON? Does the value parameter accept only strings?
I never really used envVars of type JSON more than some quick tests. I would assume you have to have your JSON in the string yes. But since the config file is a json file as well you would have to escape everything.
When i try to take the ConnectionId from the dataverse connection in my target environment. My URL looks like this, https://make.powerapps.com/environments/********-****-****-****-da1966704f2d/connections/shared_commondataserviceforapps/shared-commondataser-********-****-****-****-e4ffd4e68533/details#
While your connectionId has no hyphens. Do you know why this is the case? When i try to import with the latter connectionId in my deploymentsettingsfile it does not seem to work.
Thanks in advance,
Gertjan
I think they changed the Ids a bit since I wrote the article.
You should be able to take the ID without “shared-commondataser-“, I can check that later.