Tag: environment

Ignite ’24 – Power Platform Governance Announcements

Being at Microsoft Ignite ’24 in Chicago is an amazing experience. Even MORE amazing are the announcements that the Power Platform Governance team has come out with. I’ve been fortunate enough to have been given early access to some of the features, and they’re really awesome. Below, I’ve summarised what I believe to be the top picks to look at

Power Platform Admin Centre.

We’ve all been used to the PPAC experience that’s been around for a number of years. It’s been useful, but limited in various functions. Well, there’s not just been a facelift, but an entirely NEW PPAC experience for us. Here are some screenshots:

There’s a massive amount of stuff to look through (& play with) – my overall impressions are that this will definitely help move forward with security, governance & everything that’s needed. More importantly, especially with the focus & mentions of Copilot & Copilot Studio, there’s a section reserved for that, which is going to be critical for IT admins:

The new PPAC experience is also taking over the role that was previously played by the Power Platform CoE Starter Toolkit. Functionality is (slowly) being shifted into the main PPAC experience. One of these that’s already a great start is the Inventory capability:

Behind the scenes, this is data being captured at the tenant level, which is being stored in Dataverse (no, we don’t YET have access to the data natively, though I’m told it’s on the roadmap to be able to query). The performance of this works extremely well, though there are still a few little bugs that are being worked out 🙂

But more importantly, this also covers Copilot Studio components – to date there has not really been anything around to report on this properly…but now there is!

Managed Environments

We all know the conversation around Managed Environments, and sometimes needing to persuade organisations that premium licensing will actually give ROI to them. Well, with the new features that have been announced this week, this just got a WHOLE lot easier! Let’s take a look at some of these items

Environment Rules

Initially when Managed Environments launched, there were just a few rules that could be applied. We were told that more were coming….and indeed they are! Still more to come that the team is working on, but the number of rules has increased massively:

Some of my favorites here are the ability to manage Copilot – it’s going to be SO important as to how these are handled (especially with all of the emphasis on it coming out of Ignite). Being able to set/enforce authentication options, sharing options & various other settings is going to be KEY to proper Copilot governance.

It also now gives options for backup retention policies. I’ve written previously about how to ‘hack’ longer backups for environments (Environment types, capabilities & backups) – we’re now able to set longer backups for pure Power Platform environments within needing to enable Dynamics 365 applications within them (though of course you may still want to do this if you can see yourself using Dynamics 365 in the environment in the future – it’s still not possible to upgrade the environment type at a later point).

However there’s also something else new around environments. Previously if just looking at an environment from the main list of environments within PPAC, it wasn’t easy to see if it belonged to a Managed Environment group or not. Now it is – more so, you’re not able to tweak any settings on the general environment page that are being managed at the Environment Group level!

DLP Capabilities

One of the main challenges to date with DLP has been around the inability to block certain connectors (eg the Microsoft standard connectors). With Managed Environments, the team has now enabled organisations to be able to block ANY connectors that they wish to! If you’re not running Managed Environments, the existing limitations will still apply – you do need to be using Managed Environments for this! This will also be made available through the Power Platform API & Admin SDK tools in the coming weeks.

Preferred Group

Whilst we’ve had environment routing around now for a while (being able to auto-route new makers to a specific environments, which could be within a Managed Environment group), we haven’t had the ability to handle new environments being created & auto populated into an environment group.

Well, this is now changing. We’re now going to have the ability to auto set policies, so that when a new environment is created, it can automatically be added to a Managed Environment group. Obviously with this happening, the rules & policies applied at the group level will automatically be applied to the new environment as well! This will be a decent relief to Power Platform administrators – to date we’ve been able to set up things like DLP policies to auto-apply to new environments, but managing them otherwise needed to be done manually…well, no more!

Security Personas

Until now, security & governance within Power Platform have been a ‘one size fits all’ approach. Different types of people would access PPAC etc, but there wasn’t really a way to differentiate the different personas. This is now changing:

In summary, incredible steps forward, and I know that there’s a LOT more in the works that should be coming in the next weeks & months. I’m really excited about all of this, and using the capabilities to continue enabling & empowering organisations from a security & governance point of view.

Environment Grouping

One of the main ‘complaints’ that Power Platform administrators have is around how policies are applied to environments. Within Azure, it’s possible to set up security policies and apply them in bulk, or group together components under a single set of policies. However when it comes to Power Platform, this has not been possible – each environment has needed to be configured on its own.

I’m not talking here about DLP policies, as these are set up and then relevant environments selected/deselected as needed. I’m talking about things like setting Canvas App sharing limits, welcoming new makers, and other items.

Well, Microsoft has now made this possible to do – though the current first iteration (now in Public Preview) only has a few options within it, I’m quite certain that many more items will be coming down the line to fall under the new Environment Grouping feature.

At the moment, there are 6 options available for Power Platform administrators to be able to set and configure. Note that you do need to have the M365 security roles for either Global Tenant Administrator or Power Platform Administrator to be able to access and carry this out.

To be clear, Environment Grouping is a feature of Managed Environments. I’m not going to go into the debate about whether you should or shouldn’t adopt Managed Environments (at least not here – I may be speaking about it publicly later on this year), but you do need to have these in order to use this functionality. More specifically, you will ONLY be able to add environments that are set as ‘Managed’ to Environment Groups (though they don’t have to have Dataverse in play):

So, what exactly is the purpose of Environment Grouping? Well, it’s to minimise the amount of time that Power Platform administrators need to spend in setting up & applying policies.

Think of the users within your organsiation. You’re going to have different personas, such as developers, testers, end users, etc.

You’re also likely (especially in larger organisation) to have different business units & functions requiring different items. For example, you may lock down access to social media, but Marketing and Recruitment may indeed need access to social media to be able to carry out their jobs.

With these personas in mind, you can then start to look into building out different rule groupings, which will apply to all environments that are included under the Environment Group. It’s somewhat similar to the way in which DLP policies work – you create a DLP policy, and then everything that comes under the DLP policy gets the DLP policy setting.

There are many ways to manage pockets of environments within your tenant using environment groups. For example, global organisations can create an environment group for all environments in each geographic region to ensure compliance with legal and regulatory requirements. You can also organise environment groups by department or other criteria.

One of the other features around Environment Groups is the ability to use Environment Routing. I’ve talked about this previously when the feature was first released (Developer Environment Routing!) – Environment Groups now takes this to the next level, by being able to automatically set the Environment Group that new developer environments will fall under (so therefore policies will be automatically applied). Important to note here that all developer environments created through this WILL be set as ‘Managed’.

More information on the new capabilities can of course be found on Microsoft Learn, at https://learn.microsoft.com/en-us/power-platform/admin/environment-groups.

I think that this is a great new feature to have in place for Power Platform administrators, and look forward to seeing new functionality rolled out within this to enable organisations in a better way. Being able to cut down on administration/governance time, whilst being able to be more effective is, in my view, a win-win for ALL of us, and I can’t wait to see how it will develop over time.

So, my question to you is how would YOU look to use such functionality? What features might you like to appear within Environment Grouping to enable you and your organisation? Drop a comment below – I’d love to hear!

Developer Environment Routing!

Recently I talked about the wider vision that organisations would be able to use, for helping users get access to the right environments (Default Environment – How to handle? » The CRM Ninja). As part of this, I discussed the Microsoft vision of having environment routing in place, to move users automatically to specific environments.

At the point of writing, there wasn’t anything that I could publicly talk about. However, overnight Microsoft have released functionality around this – what I see as being the first step that this direction is taking. The documentation for this is at https://powerapps.microsoft.com/en-us/blog/default-environment-routing-public-preview/

The functionality released is to enable new users to Power Platform to automatically have a developer environment created for them to access, rather than landing in the Default environment within their tenant. Many organisations struggle with users creating content in the Default environment, when it’s not really (at least not in my opinion) the right place to do this.

Now, when we say ‘new users’, this doesn’t actually mean users newly created in M365 (or Entra ID/AAD). What this means is ‘users who have not accessed anything within Power Platform before’. In the back end, there’s a counter on each user record that keeps track of this, which this functionality is using to determine if users have accessed Power Platform beforehand or not.

What is important to note on this as well is that the Default environment DOES NOT need to be set to Managed for this to work. Microsoft documentation doesn’t make this clear at the moment, but hopefully it’ll be updated soon to clarify this.

Two settings do need to be toggled on within the Power Platform Admin Centre for this to work:

Once these have been set & saved, let’s take a look at how things actually happen. I’ve created a new user for testing purposes:

When signing in, it then briefly shows the general interface that we’re used to for a few seconds:

But, then we get this exciting NEW screen!

And then after a minute or so, we get placed nicely in the new environment:

Looking at the Power Platform Admin Centre, we can see the new environment that’s been created:

To be candid, during my testing things didn’t always work – I had some differing behaviour, or (on one occasion) the interface just hung. I’m going to put this down to being newly released & the product team working through potential issues (remember of course – this is in PREVIEW), and am hoping that they’re resolved very soon.

Also, it’s important to note that the developer environments created through this are MANAGED. Users will be able to create collateral in them, but to run apps etc will need premium licensing in place.

Moving forward, it would be great to have some information displayed to users if something hasn’t worked, as well as notifications to admins (configurable) so that they’re aware as well. Examples of this could include where an organisation has maxed out the number of (free) developer licenses available (yes, I know this sounds stange, but there’s a default limit of 9,999 developer licenses per org).

But I think it’s a great first step forward, and hopefully there will be many different ways that this product will be developed forward. My initial thoughts would include:

  • Creating developer environments for existing Power Platform users who don’t have a personal developer environment
  • Routing existing Power Platform users who have their own Developer environment to it
  • Being able to route to other places as well, including being able to specify which users/groups of users should be routed

It’s an exciting place to be in, and I look forward to seeing more of it!

What are your thoughts around this? Does your organisation allow users to have personal developer enviroments, or do they lock it down?

Default Environment – How to handle?

As we’re all aware, the default (Power Platform) environment in any Azure tenant is a very ‘interesting’ thing to have. It’s there by default when an Azure tenant is created, all users within the Azure tenant automatically have access to it, we’re not able to restrict users from being in it, etc etc.

Though it’s able to be backed up, it’s not able to be restored over itself, there’s no SLA/support available on it….the list goes on & on…!

Many of us have come up against issues caused by people using the default environment whilst not knowing about challenges involving it, which usually results in pulling out our hair, banging our head against the wall, and other like-minded productive approaches.

However, it is the first place that users, being new to Power Platform, land up, and instinctively they’ll start building applications, automations etc within it (though usually without using solutions as a container for the development of items). So to date, there’s not really anything that’s been able to be done around this, apart from monitoring users & chasing them after the fact.

Now, we’re all about enabling our users in the right way, helping educate & support them. Telling them a big NO doesn’t help, and can even be an initial blocker to having people start playing around & building technological solutions.

So how can we go about enabling our users, but also having the appropriate level of governance over the top? Well, there are several steps that I think we can take, which will help us with these. Now, not all of these are yet in place, though they have been talked about publicly. So let’s go take a look at them

  1. The first step, in my mind, is to start off with enabling the default environment as a managed environment (yes, this can ACTUALLY be done!). Managed environments have many different properties associated with them, but the one of most interest (for this at least) is the requirement to have a premium license in place.

All users within an organisation should by default have an M365 license SKU against them (usually this would be an E3 or E5). Users with these can immediately use the seeded Power Platform capabilities within them to create Power Platform collateral (using standard connector capabilities). However, with the default environment being managed, they will NOT be able to access it!

Note: For the moment, I’m leaving out users who have premium Power Platform licenses – this is deliberate

  1. Environment routing. Announced recently is the environment routing capabilities. This will enable users to be automatically routed to an appropriate environment, based on various conditions that can be set. With this, we could create appropriate business unit ‘sandboxes’, and we could route users to these. The user experience would be that when logging in, they would automatically then go to the right environment, rather than trying to work out which environment they should actually go to. This will save on confusion, and be a good user experience (in my opinion).
  1. Just-In-Time (JIT) Environment Creation. One of the items mentioned by Charles Lamanna at the European Power Platform Conference 2023 in Dublin is a new capability that’s coming in soon (I hope!). From the sound of it, this will give the ability to automatically create a new environment for users who do not already have one.

This sounds really cool. With the recent advent of Development Environments (& the ability for all users to have multiples of these), this could work REALLY well with the environment routing capability mentioned above. When a user would log in for the first time, it could look to see if they have a developer environment – if yes, then route them to it. But if the user didn’t, then to automatically spin up & create a new developer environment, and route them to it.

Now there are some caveats with this approach, leaving aside that some of the functionality isn’t GA yet.

It would mean that organisations would need to be alright with changing the default environment to become a managed environment. Obviously, risk assessments would need to be carried out with this, and non-premium solutions migrated elsewhere.

It’s also important to call out that organisations which have a CDS 1.0 implementation (ie before Power Platform became GA etc) will only have the ability to upgrade default to managed. They are not able to downgrade back to an unmanaged default environment, given limitations of the original CDS implementation (I’ve heard some truly HORRIFIC stories around this, so be careful!)

The above, however, is just the start of things. There are many other concepts to keep in mind, such as Landing Zones, Policies, etc. I’m going to be looking to cover these in upcoming posts, so keep an eye out for them!

Developer Environment Deletion!

Strong title for a blog post, right? Well, I did want to catch your attention! So what exactly are we talking about here?

For the last few years, it’s been possible for users to sign up for a ‘Developer’ plan, which gives them a full capability Power Platform environment for free (though with some limitations to them). This used to be be called the ‘Community’ plan, and is an amazing resource for everyone, whether they’re a professional or citizen developer, to have their own personal ‘sandpit’ to play in, and try things out.

Let’s wind back a few months in time now – earlier this year, Microsoft announced that users would be able to create THREE of these Developer environments, rather than having just a single one! This was mind blowing news, and something that has been extremely welcomed. If you’re wanting to see more on the announcement, Phil Topness has a great video on it at Dataverse Environments For Everyone – New Developer Plan – Power CAT Live – YouTube.

Incidentally, I’m curious as to how much storage space Microsoft has in the background to handle these. After all, each environment takes up a minimum of 1GB of space (& can grow to 2 GB). That means that each user could have 6GB of storage being used….which when multiplied, gives a VERY large number!

Microsoft has now announced that these developer environments, however, need to be utilisied. Ie if they’ve been created, but aren’t being used, Microsoft is going to delete them! Now, from a certain perspective, this is actually quite good – after all, there are all of the storage considerations for environments that have been created, but not being used. However from a different perspective, this could be a problem. What about if you’re doing something occasionally in an environment, but not too often? What about if you decide to go on a ‘Round the World’ cruise for several months?

So let’s look at the definition for this. Microsoft states that an environment is considered to be inactive when it hasn’t been used for 90 days. At that point in time, it is disabled, and the administrator or environment owner is notified. If there is no action taken within the next 30 days, then the developer environment is automatically deleted.

Now, how does Microsoft define ‘Activity’? It goes something like this:

  • User activity: Launch an app, execute a flow (whether automatic or not), chat with a Power Virtual Agents bot
  • Maker activity: Create, read, update, or delete an app, flow (desktop and cloud flows), Power Virtual Agents bot, custom connector
  • Admin activity: Environment operations such as copy, delete, back up, recover, reset

The above is all user driven – ie a user needs to interact with something within the environment. However, it’s also important to note how automation is viewed:

  • Activity includes automated behaviors such as scheduled flow runs. For example, if there’s no user, maker, or admin activity in an environment, but it contains a cloud flow that runs daily, then the environment is considered active.

It’s also important to note that at this point in time, the above only applies to Developer environments. Other types of environment (Production, Sandbox etc) don’t have any auto-deletion policies called out for them – well, at least not yet (if something does pop up around these, I’ll definitely look to talk about them too!).

So to answer our question above about what happens with a (developer) environment that is only being used infrequently – the way to stop it being auto-deleted is to put some automation in place. This doesn’t need to be lightweight – it can be something simple & easy, just to ensure that the environment registers activity happening within it.

In my view, it would be nice to have some granularity & control over this as well – allowing organisations to set their own deletion policies. We have this in place for things like audit log retention – it would be nice to have have it in here too.

Power Platform Capacity Monitoring

If I look back at customer engagements over the last few years around Power Platform, whether it was a new capability or an existing capability, there was ONE thing that stood out above all. This was the ability to be able to track capacity usage over time, and to be honest, most organisations weren’t really doing very well at it.

For those who are unaware, there are actually three different types of capacity present within Power Platform environments. These are:

  • Data
  • File
  • Log

Each one is used for a specific purpose – broadly speaking, File holds all attachements that are uploaded directly into Dataverse, Log is used for auditing purposes, and Data holds everything else (hence the name)!

Now this data is shown within the Power Platform Admin Centre, under the ‘Resources/Capacity’ section’. An example of this is:

There’s also a nice little breakdown of capacity allocation through licenses etc, which essentially shows where the available capacity has come from:

If we drill down a bit further, we can open up a specific environment, and see not only the overall usage per capacity type, but also which tables are consuming the most amount of data:

All of this is well & good so far, for someone wanting to take a look at what is currently happening. But this is a manual action – it is possible to manually export the data, but again, this isn’t automated.

It’s also not possible (at least not at this point in time) to query the underlying records that hold these values. So we’re a little stuck. If an organisation wanted to see historical data usage, and/or predict data trends (such as ‘how much capacity would we need to have in 6 months if we continued our scaling’), there’s no way to do this. At least not automatically – someone would need to store the values down manually, then report on it. A hassle, to say the least.

Now when it comes to looking overall at Power Platform, the Centre of Excellence Starter Toolkit is really quite amazing. The Microsoft PowerCAT team continue to iterate existing functionality within it, as well as bring new functionality as well.

At this point in time, however, it doesn’t have any capacity monitoring in it. Well, it sort of does – we can implement notifications to alert us when capacity reaches a certain value. But this doesn’t solve the challenge as laid out above.

So with this in mind, I set out to create a solution to handle it. I’ve always wanted to create some sort of tool for giving back to the community & helping others, and I saw this as my chance to do so (I’m in awe of the various XrmToolBox tool creators, for the record).

So, I’m releasing a capacity monitoring tool. I’m using GitHub as the host, and the repo can be accessed at https://github.com/thecrmninja/Power-Platform-Capacity-Monitoring (it was a learning experience as well as how to use GitHub as a source repository, as I’ve not done that before!).

Model-Driven App:

Reporting Dashboard:

This is just the first version – I have various ideas about how to iterate on it, and tweak functionality. Each release will include release notes & important information to be aware of (such as security needing to run it). Also importantly, thanks to the amazing Matt Collins-Jones for reviewing some of my work around this.

The audience for this tool is aimed at IT/Power Platform admins who are already familiar with the Microsoft CoE toolkit solution, and have appropriate access to it.

If you find any issues, please raise an appropriate GitHub Issue item, and I’ll look into it. Also, if you have any ideas that you think could be worthwhile, please feel free to suggest them!

Finally, I’d be interested in hearing how you think this could support you or your organisation – feel free to drop a comment below!

Environment types, capabilities & backups

Interesting title to start a blog post with, right? I can’t tell you how much I tried to work out what to call this, but then I figured that I’d just put at a high level what I’m going to be talking about!

So let’s start at the beginning. Environments within Dataverse. An environment is essentially a container for all sorts of different components, such as data models, apps, code, etc.

Examples of what an environment can contain

Within the Power Platform, there are different types of environments. As a quick recap, currently we have the following:

  • Default. Every Power Platform tenant has a default environment. We of course shouldn’t be using this for any proper development!
  • Production. Used for any Line of Business application
  • Sandbox. A sandbox environment is any non-production Dataverse environment. Isolated from production, a sandbox environment is the place to safely develop and test application changes with low risk.
  • Trial. Used to take out a trial
  • Trial (Subscription Based). Used to take out a trial when there’s subscription licensing in place
  • Developer. Personal environment, limited to one user. Previously called the Community plan.
  • Teams. Used when an app is created within Teams, to use a Dataverse for Teams environment. Doesn’t have full Dataverse capabilities, and has various limitations
  • Support. Only able to be created by Microsoft support during a support case. Is essentially a clone of an existing environment, used for diagnosis purposes.

Now, sandbox & production environments are automatically backed up – backups occur continuously, using Azure SQL Databases underneath. It’s also possible to create a manual backup instance of an environment as well, which usually takes a few seconds to carry out (restoring a backup, on the other hand, takes quite a bit longer…).

When restoring an environment, it’s not possible to restore to a production environment (though the backup could be from a production environment). It’s only possible to restore the backup to a sandbox environment – you’d then need to promote the environment from sandbox to production.

Let’s move away from backups for a moment. When we create an environment, we have the ability to select that the environment should be enabled for Dynamics 365:

This is actually a REALLY IMPORTANT CONSIDERATION! At this point in time, it’s not possible to update from a Power Platform Dataverse environment to then bring in Dynamics 365 capabilities. What this means is that if an organisation starts with just Power Apps, and then wants to expand into using Dynamics 365, IT’S NOT POSSIBLE TO DO THIS NATIVELY. Even Microsoft Support can’t do anything around this – you’d need to create a new environment, enable it for Dynamics 365, and then restore a backup to it.

It’s something that a lot of us would like be in place, but we’re not sure if it’ll ever come about. This is a tweet of mine from 2019 that Charles Lamanna responded to (I was SO thrilled that he actually responded to me!!):

https://twitter.com/clamanna/status/1176629306484637696

However, it’s still not in place. As a result, we recommend to all clients that when they deploy a Dataverse environment, they toggle the switch above (Note: A Dynamics 365 license is NOT needed to toggle this). Once this has been toggled (without deploying any of the Dynamics 365 apps), the Dynamics 365 apps and functionality can be installed/deployed at a later point in time.

There are actually various capabilities, such as the Data Export Service (yes, I know it’s now been deprecated) that actually relied on having the environment enabled as a Dynamics 365 environment in order to work. We found this out the hard way at a client, and had to do an overnight environment re-build to get the capabilities in place.

But there’s one other thing to consider around the differences between a native Dataverse environment, and an environment which has been enabled for Dynamics 365. This is around backups.

Now, backups are of course very important (thankfully they now occur automatically, as mentioned above – I remember my onpremise days when needing to run these manually!). But there are also some important differences for backup behaviour when it comes to environment types. See, it turns out that environments aren’t actually equal in backup behaviour. This is what actually happens:

  • Sandbox environments (all types) – backups retained for 7 days
  • Dataverse production environment (not enabled for Dynamics 365) – backups retained for 7 days
  • Dataverse production environment (enabled for Dynamics 365) – backups retained for 28 days

See that? Having Dynamics 365 enabled for an environment gives you FOUR TIMES as much backup retention time! That’s incredible!

Dataverse Environment enabled for Dynamics 365 – 28 days of backups available!

So not only are you able to then upgrade to Dynamics 365 applications at a later date, you then also have more peace of mind (hopefully you don’t need to use it though!) around keeping backups for longer.

This is really cool – I hope it helps you plan your environment implementation strategy! Have you ever come up against issues when using environments, or the type/s of environment? Drop a comment below – I’d love to hear!

Environments & ‘Admin Mode’

With some recent events happening (both professional & personal), I’ve taken a slight step back from putting out posts on here. Thankfully things seem to be settling down, so I’m getting (back) into the swing of things!

I thought that it would be good to talk about a subject that I fell ‘foul’ of recently. This is around environments, and more specifically, the ‘admin mode’ that it’s possible to use on them.

So what exactly is this ‘admin mode’? Well, the aim of it to restrict access to certain users, namely System Administrators & System Customisers. Why would we want to do this? There are several scenarios that come into mind:

  • Performing a system upgrade (such as enabling new features)
  • Changing environment type (eg Production to Sandbox, or vice-versa)
  • Restoring an environment

Essentially, any time we have operation-type work that we’re wanting to carry out. This way whatever we’re doing won’t affect users, and anything that the users are doing won’t affect things either (symbiotic relationship there!).

So as an example, if we’re doing a major release, which changes functionality within a system, we wouldn’t want users in the system carrying out their usual work, as this could have data issue if saving during the actual release. We of course SHOULD be communicating to users that a release is going to take place, and that they shouldn’t be in the system at the time, but ‘admin mode’ is how we can truly enforce it.

Something to bear in mind as well is that if you’re going ahead & restoring an environment to a previous state (whether that’s an automatic save point, or a manual one), it will automatically put the environment into ‘admin mode’ once the restore has been completed. This is very important to keep in mind!

There are three settings around administration mode:

  1. ‘Administration Mode’. This sets whether admin mode is on or off!
  2. ‘Background Operations’. This sets whether background processes, such as workflows, power automate flows, and Exchange synchronisation are enabled (allowed to happen) or disabled (stopped from happening
  3. ‘Custom Message’. This allows you to set a custom message that users (who are not system administrator/system customiser) will see when they attempt to access the environment

So this is the scenario that tripped me up a few weeks back:

  • I was needing to restore an environment to an earlier save point (to be clear, this was NOT a production environment)
  • I went ahead with the restore, and it completed successfully
  • Given that I was doing this at night, one of my children woke up, and I had to deal with them
  • I came back to things, saw that it completed, and then went ahead with the release that I was needing to do

All seemed to go well. However, when users were testing (which admittedly was a few days later), they reported that some functionality wasn’t working. This was strange, as it had been working before the release (& the release that I did hadn’t actually touched it!).

It turned out to be Power Automate flows that just didn’t seem to be running. OK – I started to look into them, but couldn’t figure out why they hadn’t run.

Creating a test Power Automate flow didn’t seem to work either – despite running it to test it, the trigger never activated! I was quite puzzled by this, and couldn’t (initially) work out the reason.

Then I thought to check environment settings! Lo & behold, the environment was STILL in administration mode, and the Background Process option was disabled! Aha – I’ve found the source!

Flipping this out of administration mode thankfully then allowed all Power Automate flows to work/run, and users confirmed that functionality was indeed running as expected. As you can imagine, I was quite relieved!

man in white shirt and black pants standing on black concrete bench near white building during

Something that I hadn’t realised previously is that if you manually put an environment into administration mode, it doesn’t automatically disable background processes. However, if you restore an environment, it DOES disable background processes by default. So if you’re wanting to try out automation items within a restored environment that’s still in administration mode, you’re going to need to ensure that you toggle the Background Processes toggle to allow it to work!

One further thing to learn as well (which I’ve been asked already by some people, so thought that I would mention it here). I’ve mentioned above that users were in the system, but reporting that things weren’t working. Now given that the environment was in administration mode, people have asked how users could be in it! The answer is that these users actually had the system customiser role applied to them, which is why they could get in! If they hadn’t had the role, then perhaps I might have realised things a little sooner (ie that the environment was in administration mode).

So a (good) little lesson learned, and I’ll definitely take it forwards. Has this, or anything else like it, ever tripped you up? Drop a comment below – I’d love to hear!

Solution Dependencies & Management

Solutions are marvellous things. They enable us to be able to package up lots of components, and deploy them to different environments all together as one single package.

However, there have been changes over time as to how solutions are used. I’m not (for the most part) going to go into the Managed VS Unmanaged debate, which I leave to people who are more in the know….

Microsoft Dynamics 365 apps are installed using solutions. Third party apps provided by Independent Software Vendors (ISVs) also use solutions.

In Power Apps, solutions are leveraged to transport apps and components from one environment to another or to apply a set of customisations to existing apps. A solution can contain one or more apps as well as other components such as entities, option sets, etc. You can get a solution from AppSource or from an independent software vendor (ISV).

Custom development should also take place within a solution, to allow it to be deployed appropriately.

But it’s important to take a closer look at how solutions work overall, as we can be involved on multiple projects within the same environment. Not only that, some solutions may require other solutions to be present first, in order to actually work! A great example of this is Master Data Management (or MDM), which is where companies have a ‘backbone’ of data, which other parts of the system then hangs off.

To understand this concept better, let’s take a quick look at solution layering.

Solution Layering

Layering occurs on the import of solutions and describes the dependency chain of components from the root solution introducing it, through each solution that extends or changes the components behaviours. Layers are created through an extension of an existing component (taking a dependency on it) or creation of a new component or version of a solution

Managed and unmanaged solutions exist at different levels within a Microsoft Dataverse environment. In Dataverse, there are two distinct layer levels:

  • Unmanaged layer. All imported unmanaged solutions and unmanaged customizations exist at this layer. The unmanaged layer is a single layer.
  • Managed layers. All imported managed solutions and the system solution exist at this level. When multiple managed solutions are installed, the last one installed is above the managed solution installed previously. This means that the second solution installed can customize the one installed before it. When two managed solutions have conflicting definitions, the runtime behaviour is either “Last one wins” or a merge logic is implemented. If you uninstall a managed solution, the managed solution below it takes effect. If you uninstall all managed solutions, the default behaviour defined within the system solution is applied. At the base of the managed layers level is the system layer. The system layer contains the tables and components that are required for the platform to function.

The following diagram introduces how managed and unmanaged solutions interact with the system solution to control application behavior.

  • The system solution represents the solution components defined within Dynamics 365 or the Power Platform. Without any managed solutions or customisations, the system solution defines the default application behaviour. Many of the components in the system solution are customisable and can be used in managed solutions or unmanaged customisations.
  • Managed solutions are installed on top of the system solution and can modify any customisable solution components or add more solution components. Managed solutions can also be layered on top of other managed solutions. As long as a managed solution enables customization of its solution components, other managed solutions can be installed on top of it and modify any customisable solution components that it provides.
  • Unmanaged customisations. All customisable solution components provided by the system solution or any managed solutions can be customized in the unmanaged customisations
  • Unmanaged solutions are groups of unmanaged customisations. Any unmanaged customized solution component can be associated with any number of unmanaged solutions. These can be edited & modified, regardless of the environment in which they’ve been deployed to
  • The ultimate behaviour of an instance of Dynamics 365 or Power Platform application is the culmination of the system solution, any managed solutions, and any unmanaged customisations.

The official stance of Microsoft, according to its Application Lifecyle Management (ALM) documentation, is that unmanaged solutions are used for development, and that managed solutions are released downstream to further environments. For bespoke solutions, however, this may not fit, and an appropriate balance must be found.

Data ‘Backbone’ & Solution Dependencies

Given the way that companies are adopting Power Platform (and Dynamics 365, of course!) it’s highly likely that we will build out system structures that will form the backbone for multiple applications on an on-going basis. With this in mind, it’s appropriate to put in place proper planning for this, to avoid any issues that could occur in the future with appropriate system designs

Solution Dependencies

When creating system structures within an environment, using unmanaged solutions, connecting two (or more) tables together will create dependencies on each other. In simple terms, if we connect Table A to Table B, there’s a reciprocal relationship created back from Table B to Table A:

This happens even if Table A is in Solution 1, and Table B is in Solution 2. If they’re in the same environment (& both solutions are unmanaged), it will create the two-way dependency.

This will cause issues if trying to deploy each solution individually, and will fail on import, as the system will require all items to be available in the solution

Workable scenario

The way in which to handle the issue of solution dependencies is to ensure that the ‘master backbone’ of system design is created in the main development environment, and then to use that in secondary development environments as the core of additional solutions:

This is in line with the emerging recent Microsoft Best Practise information around solution management (which is likely to be moving towards having a single environment per developer, rather than multiple developers working in the same environment).

The steps for doing this are as follows:

  1. Main ‘core solution’ exists (as unmanaged) within the main development environment
  2. When a project requires this to build upon:
    1. Secondary development environment is created
    1. ‘Core solution’ is exported as managed from the main development environment, & imported into the secondary development environment
    1. Project work is carried out within the secondary development environment
    1. Once project solution is complete (or when appropriate for deployment), it can be exported from the secondary development environment
      1. If deploying directly from the secondary development environment to downstream environments, it should be exported as managed
    1. The solution should be exported as unmanaged, and imported back into the main development environment. This will not cause dependencies to be created with the ‘core solution’ in it

Note: The main ‘core solution’ should consist of the items that are needed for core system work. If additional items are needed for multiple projects to work off (eg Account Manager field), this would need to be added to the core solution, rather than the individual project solution/s, as otherwise there could be further issues downstream.

If the project is completed, but requires further work to be carried out later on (or development support), then the following should be done:

  1. Secondary development environment is created
  2. ‘Core solution’ exported from the main development environment as a managed solution, and imported into the secondary development environment
  3. Project solution exported as unmanaged from the main development environment, and imported into the secondary development environment
  4. Work and/or support can be carried out within the secondary development environment, and released appropriately

I’m expecting further information around this to be released by Microsoft in due course (I’m a little surprised there’s not more out there at the moment, to be honest!). It’s vital that we ensure that we’re working with solutions in the right way, to stop any issues occurring later on down the line.

Have you ever had a problem around this? Drop a comment below – I’d love to hear your experiences!

‘Ghost’ lookup value following deployment

This is something that stumped me fairly recently. It’s also something that I was trying to work out what I should use at the title for this post! Let me share what happened.

I’m working on a project that’s quite critical (COVID-19 related). This is a project that we’ve built something around Dynamics 365 as an additional wrapper, to provide specific functionality for the pandemic. It’s being rolled out (the same solution) to multiple clients, and is only using the functionality from Power Platform. No custom code at all.

Now, before going into the specifics around it, let’s take a moment to revisit what a lookup field is, and what it does. Essentially a lookup field connects two tables together (wow – that felt strange not to use the word ‘entity’!). In the front interface, it’s used for a 1:N relationship.

So for example, we can have a lookup from Account to Contact, to set the primary contact for the account. The user navigates to the field, searches for the record they’re wanting to associate, and saves it.

Underneath, there’s a relationship that’s automatically created between the two tables, showing the way that the relationship will go (ie 1:N or N:1). This is created on both sides (more on that another time around dependencies), and most people will never need to modify it

When I first started with this particular project, I got the solution, and deployed it into the Dev environment (for the project that I was on). On testing it out, I found something very interesting. We’re using the Case (Incident) table, and there are various lookup fields on it. One of these was already populated with a value. Hmm – that’s interesting, I thought. It was a new deployment, and we hadn’t set any static data up yet at all. So how could it already be populated?

How is this being set, when I’ve not entered it into the system as a record…

Furthermore, I was unable to save the Case record. When I tried to, I was getting an interesting error:

On drilling down into the error log (which admittedly is actually getting better in the details shown in it, thankfully!), it turned out to be because I didn’t have access to the referenced record (in the lookup field). It just didn’t exist.

So the lookup field value was coming in with a hard-coded GUID (record identifier). But how was this being done, especially if there weren’t any records (of that type) in the system at all?

From my experience of things, I could think of two ways in which to populate a lookup field with a hard-coded value:

  • Through a ‘real-time’ Power Automate flow, on create of the record. It’s possible to set a GUID value in the flow, and then it would be set
  • Through custom code, running on the form. Again, it’s possible to hard-code a GUID there, and then set the field

However on checking both options, none of them were happening. No Power Automate flows touching the Case record, and no custom code at all on the Case.

It was then, digging through the other parts of the solution, that I saw various Business Rules. For those unfamiliar with these, I’ll quote from the official Microsoft documentation around them:

By combining conditions and actions, you can do any of the following with business rules:

  • Set column values
  • Clear column values
  • Set column requirement levels
  • Show or hide columns
  • Enable or disable columns
  • Validate data and show error messages
  • Create business recommendations based on business intelligence.

I’ve used Business Rules (somewhat extensively) before. However on going into the one for the Case table, I found that something was happening that I wasn’t aware could happen! It’s actually possible to set a lookup field value through it:

I spy a lookup option

Even though we’ve deployed the solution from the original development environment to a different environment, this is still set. But there are no records that are available:

I had never thought that it would be possible – to set a static value (eg a number, or some text), fine. But to set referential data? Wow.

Obviously this can be quite helpful. The bit that it’s NOT helpful though is when deploying the solution to another environment (as this situation was). It doesn’t help if you re-create the record that it’s referring to with using the same record name, as it’s using the underlying GUID (which you can’t re-create). This really does take solution deployment into a whole new perspective, where you need to be careful around these sorts of things as well.

So something new that I’ve learned (I do try to learn something new each day), and specifically around an area I thought I knew quite well. It did take some time, but I’m glad that I (finally) found the root cause of it, and identified what was causing it.

Have you ever had something like this happen, where you’re searching & searching for the cause of it? Drop a line below – I’d love to hear!