Tag: security

Data Export Service Connection Issues

This is a slightly different post from the usually stuff that I talk about. It’s much more ‘techy/developer’ focused, but I thought it would be quite useful still for people to keep in mind.

The background to this comes from a project that I’ve been working on with some colleagues. Part of the project involves setting up an Azure SQL database, and replicating CDS data to it. Why, I hear you ask? Well, there are some downstream systems that may be heavy users of the data, and as we well know, CDS isn’t specifically build to handle a large number of queries against it. In fact, if you start hammering the CDS layer, Microsoft is likely to reach out to ask what exactly you’re trying to do!

Therefore (as most people would do), we’re putting in database layer/s within Azure to handle the volume of data requests that we’re expecting to occur.

Azure SQL Database | Microsoft Azure

So with setting up things like databases, we need to create the name for them, along with access credentials. All regular ‘run of the mill’ stuff – no surprises there. In order for adequate security, we usually use one of a handful of password generators that we keep to hand. These have many advantages to them, such as ensuring that it’s not something we (as humans) are dreaming up, that might be easier to be guessed at. I’ve used password generators over the years for many different professional & personal projects, and they really are quite good overall.

Sordum Random Password Generator Creates Random Passwords with Ease - MajorGeeks
Example of a password generation tool

Once we had the credentials & everything set up, we then logged in (using SQL Server Management Studio), and all was good. Everything that we needed was in place, and it was looking superb (from the front end, at least).

OK – on to getting the data actually loaded in. To do this, we’re using the Data Export Service (see https://docs.microsoft.com/en-us/power-platform/admin/replicate-data-microsoft-azure-sql-database for further information around this). The reason for using this is that the Data Export Service intelligently synchronises the entire database initially, and thereafter synchronises on a continuous basis as changes occur (delta changes) in the system. This is really good, and means we don’t need to build anything custom to handle it. Wonderful!

Setting up the Data Export Service takes a little bit of time. I’m not going to go into the details of how to set it up – instead there’s a wonderful walkthrough by the AMAZING Scott Durow at http://develop1.net/public/post/2016/12/09/Dynamic365-Data-Export-Service. Go take a look at it if you’re needing to find out how to do it.

So we were going through the process. Part of this is needing to copy the Azure connection string into into a script that you run. When you do this, you need to re-insert the password (as Azure doesn’t include it in the string). For our purposes (as we had generated this), we copied/pasted the password, and ran things.

However all we were getting was a red star, and the error message ‘Unable to validate profile’.

As you’d expect, this was HIGHLY frustrating. We started to dig down to see what actual error log/s were available (with hopefully more information on them), but didn’t make much progress there. We logged in through the front end again – yes, no problems there, all was working fine. Back to the Export Service & scripts, but again the error. As you can imagine, we weren’t very positive about this, and were really trying to find out what could possibly be causing this. Was it a system error? Was there something that we had forgotten to do, somewhere, during the initial setup process?

It’s at these sorts of times that self-doubt can start to creep in. Did we miss something small & minor, but that was actually really important? We went over the deployment steps again & again. Each time, we couldn’t find anything that we had missed out. It was getting absolutely exasperating!

Finally, after much trial & error, we narrowed the issue down to one source. It’s something we hadn’t really expected, but had indeed caused all of this to happen!

What happened was that the password that we had auto-generated had a semi-colon (‘;’) in it. In & of itself, that’s not an issue (usually). As we had seen, we were able to log into SSMS (the ‘front-end’) successfully, with no issues at all.

However when put into code, Azure treats the semi-colon as a special character (a command separator). It was therefore not recognising the entire password, which was causing the entire thing to fail! To resolve this was simple – we regenerated the password to ensure that it didn’t include a semi-colon character within it!

Now, this is indeed something that’s quite simple, and should be at the core of programming knowledge. Most password generators will have an option to avoid this happening, but not all password generators have this. Unfortunately we had fallen subject to this, but thankfully all was resolved in the end.

The setup then carried on successfully, and we were able (after all of the effort above) to achieve what we had set out to do initially.

Have you ever had a similar issue? Either with passwords, or where something worked through a front-end system, but not in code? Drop a comment below – I’d love to hear!

Marketing & an unusual error

I’ll be the first to admit that I have limited experience of Dynamics 365 for Marketing. In fact, I think that it would be stretching the description to say that I have even ‘limited experience’! I’ve seen it one or twice, and have attended a few presentations on it, but apart from that, nada.

I do remember what it used to be like in its previous incarnation, but even then I didn’t really touch it. Customer Service (& Sales) are my forte, and I generally stick within those walls. Marketing traditionally was its own individual application, and only more recently has been rolled into the wider Dynamics 365 application suite. Even so, it still sometimes works in a somewhat interesting way, different from the rest of the system.

Inevitably I’ve had to actually do something with it for a client project, which has brought me to putting up this post. We had created a few marketing forms, surfaced them correctly, etc. It was great, and working well.

Then we realised that we needed to capture some additional information, in this case a list of Countries. There’s no standard entity for it within Dynamics 365, so we created our own, and loaded a list of countries (& associated data) into it. Fine – that was working without issues, including in the places that we needed to surface it.

Then we came to needing to surface the Country value on a marketing form, through a lookup. Simple, you’d have though? Well, not so much. We went to create the field, and got presented with the following error as we did so:

The error says: ‘The role marketing services user does not have access to the entities you’ve chosen…’

In essence, the system was telling us that we weren’t able to access the entity. Though Country is a custom entity, we were logged in as users with the System Administrator role (which has access automatically to ALL entities). This left us puzzling around what to do.

The error message, thankfully, was quite clear. It was referring to a specific security role missing privileges. In this case, it was the ‘Marketing Services User’. I therefore went to check the permissions for it, and sure enough, it didn’t have permissions on the Country entity that I had created!

Now usually if a security role is missing permissions, what we do is create a custom security role (usually copying the existing role), and add the permissions to do. Best practise is NOT to edit the default security roles. The (main) reason behind this is that Microsoft could update the security role in a later update/release, which could impact on us. We therefore use custom roles to avoid this happening (& yes, I’ve seen it happen/impact in practise!).

The fly in the soup here (lovely phrase, I know) is that we couldn’t do that here. It seems that Dynamics 365 for Marketing uses an underlying security role that’s needed. Even if we had implemented a custom role, we didn’t have any idea of how to tell the system to actually use our custom role, rather than the default one that it’s currently using. Quite frustrating, I tell you!

So in the end we decided to give the default security role the necessary permissions, and see what happened:

With having granted the security permissions to the role, & saved it, we then attempted to create the marketing form field field. This time, we were successful! No errors occurred during it, thankfully:

So in summary, I still have no idea why this has happened. I’ve taken a look around, but can’t find anything obvious as to how/why it actually works like this. I guess that I’d need to dig ‘under the hood’ somewhat to see what’s actually going on, and how to dealt with it appropriately. For the moment, the solution is in place, and is working.

We’ve also been very careful (as mentioned above) to add just the specific custom entity to the default security role. We haven’t touched anything else within it – all other security permissions are done (as per best practise) with custom security roles, which are then allocated appropriately to users &/or teams. Hopefully this will be fine in the long-term, though we’ll definitely be keeping our eyes on it to make sure!

Have you ever come across something like this? How did you decide to go about solving it? Drop a comment below – I’d love to hear!

Update: Thanks to the amazing Carl Cookson, it turns out that this is due to an update from Microsoft in how Marketing works. See https://docs.microsoft.com/en-gb/dynamics365/marketing/marketing-fields for more information around it. Essentially it uses this role to sync to the Azure staged Marketing service, so this role needs to have the appropriate permission

AAD Security Teams, & saving personal views

Previously I’ve touched on how it’s possible to use Azure Active Directory for Dynamics 365 security. This can be of great benefit to an organisation, especially when needing to invite in external users. The details that I go into around it can be found at Dynamics 365 Security & AAD. As I point out there, it’s a very helpful feature, and can also help with onboarding new users within an organisation.

What I’ve found out about it, however, is that there can be some very interesting little quirks with how security actually works. Originally I thought it was a bug, and raised it with Microsoft Support, but it turns out not to be. Let me take you through the journey that I experienced last week…

The scenario is as follows. We had security set up in place, which was working perfectly (or so we thought). We’d gone through all of the following steps:

  1. Create Dynamics 365 security role/s with appropriate permissions
  2. Create AAD security group
  3. Create Dynamics 365 AAD Security Team, and link it to the AAD security group
  4. Assign users to the AAD security group

This was working exceptionally well (except, of course, when the external users hadn’t followed the setup instructions correctly). Users were logging in, searching for information, creating/updating records, etc. All was good…or so we thought.

Now, the users who are actually using the application don’t have a Dynamics 365 background. It’s the first time that they’re using the specific system, and as such, are going through a learning curve. We’re not expecting them to understand the advanced functionality at this point, though some of them are indeed venturing further/deeper into the capabilities that it brings.

The Learning Curve | Listen via Stitcher for Podcasts

One of these, of course, is the Advanced Find. Now, those experienced with Dynamics 365 will know all about it. There are good points, and there are not so good points. Functionality in it has expanded over time, though to be honest it’s still easier to run a SQL query/extract for more advanced information retrieval.

Users seemed to be fine with the Advanced Find. We showed them how it works, how to filter, set up columns, etc. We even showed them how to export data to Excel, and keep a live data connection back to refresh it! Brilliant – they were most pleased.

Then I got an email in from a user needing support. They reported that they weren’t able to save custom searches. This is of course very helpful, in order to avoid having to set up the same search/layout every time. This seemed puzzling to me, and I started to take a look into it.

Always download the error log file – it can be SO useful!

I was able to replicate the problem immediately with a test user, having assigned it the same security role. Opening the log file (which can be extremely helpful at times with troubleshooting), I looked to see what the issues were. I was thinking it was a problem with security permissions – if I assigned the system administrator role to my user, everything worked just fine.

Incidentally, there’s a really good blog post at https://www.powerobjects.com/blog/2015/02/13/access-denied-identify-fix-security-role-issue/ which covers troubleshooting security role issues. I’ve used it on several occasions previously.

In my error log, there were repeated references to ‘ObjectTypeCode”:4230’. This is the View settings in the security role. I therefore went to the security role, and ensured that it was set to allow access to Saved View across all permissions:

It’s only possible to set User-level permissions for Saved Views

Right – permissions set, all should be good. Let’s go ahead & try to save an Advanced Find as a view…but no! It’s still not working, and showing the same error message!

What I then tried to do was apply the security role directly to the user, rather than through the AAD security team. To my surprise (well, not really, actually), it worked. I was able to save Advanced Find views. I changed back to the user getting permissions through the security group (ie not directly), and again I had the issue.

OK – so I thought I had discovered a bug. As far as I was aware, I couldn’t see any reason why the user wouldn’t be able to save the Advanced Find view. After all, they’re able to create & save records within the system. There surely shouldn’t be any difference between saving records, and saving an Advanced Find view?

Stressful woman looks with puzzled expression into screen, wears formal shirt, busy with making financial report, feels worried about deadlines, feels headache from recieving bad news Premium Photo

My next step was to raise a support ticket with Microsoft, and then carry out the obligatory ‘show & tell’ to the support agent. Ivan (the agent assigned to my case) was very helpful, understood exactly what I was trying to accomplish, and what the issue seemed to be. I left him with the support case, and focused on trying to find a workaround for the situation.

After a few days, Ivan came back to me with a resolution. It wasn’t a bug in the system (which was a shame – I was looking forward to having it attributed to me!), but rather a specific case of permissions.

See, there’s something called ‘privilege inheritance’. In a nutshell, there are two ways of giving access through a security role:

  1. User privileges. This is when the user is given the permissions directly
  2. Team privileges. This is when the user is given the permissions as a member of the team. If they don’t have User privileges of their own, they can only create records with the team as the owner

There’s a good article on this at https://docs.microsoft.com/en-gb/power-platform/admin/security-roles-privileges#team-members-privilege-inheritance

So what was actually happening was as follows:

  • Users were able to read, create, update records without issues, as the team was the owner of these records
  • However as views need to be owned by a user (though they can be shared with a team), the user was unable to save them!

Thankfully it’s quite easy to fix – on the security role itself, you change it here:

With this then in place, everything then worked just fine. The user was still getting the role through the Security Team, but was now able to save these directly.

Quite an interesting little quirk, but one that is likely to come in useful when looking at other functionality within the system.

Have you come across this before? Have you found anything else that seems a little strange? Comment below – I’d love to hear!

Dynamics 365 Security & AAD

I come from an ‘on-premise’ background. I’ve spent years in organisations with on-premise systems such as Dynamics 365. Take me into a server room that’s alive with whirring fans, and I get quite nostalgic. Those were the days…well, in some ways, anyhow. But having recently discovered some quite helpful functionality, I thought I’d share it with others!

See, when it came to Dynamics 365 security, there was no way to automate things. Yes, users had to be created in Active Directory (and also, in a folder that the Dynamics install could refer to within AD!), but they had to be manually added to Dynamics 365. There was no way to automate this (from recollection – then again my memory grows dim with the fog of time).

So what the system administrators needed to do was to manually go to Settings/Security within the system, and there they could either add a single user at a time, or multiple users. They would then assign role/s (for multiple users, all of the users would need to have the same role/s – it wasn’t possible to modify individual users within this process).

One way to slightly speed up time in handling different security roles was to have teams, relating to the business needs. The security role/s would be created, assigned to a team, and then any user added to the team would automatically get all of the permissions that they needed.

Then came the heady world of Dynamics 365 being online! Well, nothing much changed really, at least not for a little while.

But then, things really did change, in May 2019. Functionality for security teams within Dynamics 365 was increased. Notably, there was now something called a ‘AAD Security Group Team’:

So what was this magical new item?

When we create a team, and we set the Team Type to ‘AAD Security Group’, we’re now able to set an AAD Object ID. In fact, it’s required! After we’ve created this object within Dynamics 365, we can then apply security role/s to it directly (as we could to any other team records beforehand):

Let’s take a moment to reflect & think on this. Until now, we’ve had to handle security directly within Dynamics 365. Now, we have the ability to have an Azure Active Directory (for that is what AAD stands for) group, and reference it within Dynamics 365.

Suddenly new possibilities open up. As part of the on-boarding process (for example) we can users to specific AAD security groups, which will then give them access with appropriate permissions within Dynamics 365. We’re also able to have multiple AAD groups, each inheriting a different set of Dynamics 365 roles, and thereby create a multi-layering approach to different business & security needs.

We’re also able to use tools such as PowerShell, LogicApps, Power Apps & Power Automate to carry out automation around this. There’s an Azure AD connector (https://docs.microsoft.com/en-us/connectors/azuread/) which gives the ability to set up & administer these.

We’re actually using this functionality now in some of our COVID-19 response apps. Instead of needing our own support desk to manage the (external) users, we’ve provided an interface where client IT departments can quickly log in, upload a list of users, and assign them to the relevant AAD group/s. It’s very quick, and allows the users to onboard to the Power Apps within minutes!

So with knowing this, how do you feel it might help benefit you? Comment below – I’d love to hear!

Omnichannel for Dynamics 365 – Security

Like all of the other applications and features within Dynamics 365 and the PowerPlatform, the ability to use Omnichannel features requires security role/s to be assigned to users.

These are assigned in the usual way that all security roles are assigned for an environment. A user (with appropriate admin privileges) will need to go to admin.powerplatform.com, select ‘Environments’ in the left-hand navigation bar, select the environment that they’re wanting to set security for, and click the ‘Settings’ menu button

Under the Settings menu, select the option for ‘Users + Permissions’, and then select ‘Users’

You’ll get a screen opening, in which you can manage users and security roles for them. Select the user/s that you’re wanting to assign Omnichannel security roles to, and click the ‘Manage Roles’ menu button (it’s possible to assign the same role/s to multiple users at the same time).

Note: You can also use the search box (not displayed in the image below) to search for a specific user that you’re wanting to add the role/s to)

When the Manage User Roles window opens, you’ll be able to see all security roles that you can apply to the user/s. For Omnichannel, there are 4 specific roles:

  • Customer service app access
  • Omnichannel administrator
  • Omnichannel agent
  • Omnichannel supervisor

Note: All Omnichannel users (agents & supervisors) should be assigned the ‘Customer service app access’ security role

The differences between the three Omnichannel security roles are as follows:

Omnichannel Agent Can view user list / presence list / work stream list/ queue list
Can view quick replies
Omnichannel SupervisorCan view user list / presence list / work stream list / queue list / PBI config list
Can edit default presence and default capacity of a user
Can edit queue assignment of a user
Can add / remove users from presence
Can add / remove agents from queue
Can view / add / edit / delete quick replies
Can view operating hours
Omnichannel AdministratorCan view user list / presence list / work stream list / queue list / PBI config list
Can edit roles of a user
Can edit default presence and default capacity of a user
Can edit queue assignment of a user
Can add / edit / delete presence
Can add / remove users from presence
Can add / edit / delete presence associations
Can add / edit / delete work streams
Can add / edit / delete channel settings, context settings, routing rules
Can add / edit / delete queues
Can add / remove agents from queue
Can view / add / edit / delete quick replies
Can add / edit / delete PBI config
Can view add / edit / delete operating hours
Can view add / edit / delete auth settings

Once role/s have been selected and saved against user records, you’ll be able to see the users show up in the ‘Omnichannel’ user view

You’ll also be able to see these users under the ‘Users section of the Omnichannel Administration application

We’ll look next at how Omnichannel users are managed.

Environments & Security

Following on from my post last week (https://thecrm.ninja/2019/07/05/environments-for-projects/) where I talked about the different environments for projects, I thought it would be good to talk about security relating to it as well.

Image result for security

What I’ll be discussing below is best practise for projects that relate to (external) clients.

However, there are usually some small differences when it’s an internal project for a company – security is can be slightly more relaxed (after all, the dev teams are usually the ones responsible for rolling the project out, providing on-going support, new features, etc). It’s also the case that internal developers (usually) won’t be prevented from seeing what the actual company data is.

The essential principle is as follows: Users should be restricted to only using environments that they are needing to access

This follows Best Practise for system security, as well as some common sense (it’s surprising how many times this can seem to be lacking!)

Access to the environment/s will depend on roles/s of the person, along with infrastructure that is in place. Users should not be granted access to any environment that they have no need to access at all .

DevIntegrationUATStagingTrainingProductionSupport
Team
Developers✔✔




Consultants

✔✔✔
✔
Clients

✔✔✔✔

Note: There may be exceptional cases people are required to access the Production instance for a client. In such a circumstance, it is vital and absolutely necessary to have a complete audit trail to cover this, setting out the reason/s for it, along with all actions that are taken within the system. This should be ideally be via email, or any other system that may be present to allow a definitive time-stamped communication of request and sign-off

There is an extensive security model within Dynamics365 that can be used to enable and control this, if needed (eg for users to have access to one part of the system, but not another – this could be due to the system holding restricted access data, for example).

Have you come across any cases where this wasn’t followed, and caused issues? Feel free to comment – I’d love to hear about what happened!