For those who are well versed in rolling out solution updates within Dynamics CRM/365 for Enterprise (CRM/D365E), the process will always have a certain familiarity to it, with a few surprises rolled in now and again. Often, the update will proceed as anticipated; sometimes, you may encounter bizarre issues. I can remember a particularly strange incident I had last year where a solution import would get to about 90-95% completion…and then the green progress bar would suddenly start rolling back to nothing. The import progress window would then hang with no further guidance or error message. To try and determine the root cause, we had to interrogate the importjob entity within the system, which ended up showing the import job progress stuck at 0.15846281% : / In the end, we had to escalate the issue to Microsoft for further investigation, but rest assured that if you have not yet witnessed your own curious solution import experience, it’s definitely in the post 🙂

Thankfully, if you are new to the whole “rolling out solution update” thing, you can be assured that the process is relatively straightforward, and mostly without issue. If you have been handed a set of solution import instructions for the first time, though, you may be wondering why something similar to the following step is included:

Go into the Data Management -> Duplicate Detection Rules page and click Publish on all Duplicate Detection Rules that have a Status Reason of Unpublished

Unfortunately, after importing a solution update, CRM/D365E will automatically unpublish all of your Duplicate Detection Rules automatically. You are therefore required to explicitly publish them again, lest you start to encounter a sudden increase in duplicate records and database storage within your system. The reason why this happens is both understandable and frustrating in equal measure. As outlined in the following MSDN article on the subject:

A duplicate rule condition specifies the name of a base attribute and the name of a matching attribute. For example, specify an account as a base entity and a contact as a matching entity to compare last names and addresses

As part of the above, explicit matchcodes are created for every record that the Duplicate Detection Rule is targeting, based on the current metadata of your CRM/D365E entities and attributes. Because your solution update can potentially alter significant aspects of this metadata, the system automatically unpublishes all Duplicate Detection Rules as a precaution.

The above is perhaps trivial in nature, as the actual process of re-publishing all Duplicate Detection Rules is somewhat negligible in effort terms. Where difficulties can arise is if someone innocently overlooks this part of the process or if your system has many different Duplicate Detection Rules, in a mixture of Unpublished/Published state. You would have to specifically make a note of which rules were Published before beginning your solution import so that you can ensure that the correct rules are published after the fact. I would have thought that after so many versions of the product, that something would be added to address this – for example, perhaps a checkbox at the start of the Solution Import Wizard that lets you specify whether all currently published rules should be reactivated after the import completes successfully.

If you find that the above is an annoyance that you can do without no longer, like with many things on the platform, there is a solution that can be deployed in code. The SDK exposes the PublishDuplicateRuleRequest class, which does exactly what it says on the tin – meaning that you can write a plugin that applies this functionality accordingly. The tricky bit comes in determining which Message (i.e. action) on the platform that you wish to run this against. CRM/D365E does not expose a SolutionUpdate or SolutionImport message that we can piggy-back onto, so we have to look at the PublishAll message instead – the action that is triggered when you press Publish All Customizations in the system. This is because this is generally the action you will always need to take when importing an (unmanaged) solution. As a result, we can write a plugin class that is triggered on the Post-Operation event of this entity to automatically publish all Unpublished Duplicate Detection Rules in the system!

The snippet below is adapted from the sample code provided by Microsoft, but has been tweaked as follows:

  • A QueryExpression is used as opposed to QueryByAttribute, since we need to query on two separate attributes and their values – statecode and statuscode. You also cannot return an easily accessible count on all results returned with QueryByAttribute. We will see why is useful in a few moments.
  • The code explicitly checks for if there are any Unpublished rules first before attempting to proceed further – no point in running code unnecessarily!
  • Instead of activating each rule one-by-one using an Execute request, all of the requests are collected together as part of an ExecuteMultipleRequest, given that we now know the performance benefits that this can have.
  • Tracing has been implemented in liberal amounts, to provide remote debugging from within CRM/D365E.

Here’s the code – just copy into an empty class file on your plugin project, modify the namespace to reflect the name of your project and you will be good to go!

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Query;
using Microsoft.Xrm.Sdk.Messages;
using Microsoft.Crm.Sdk.Messages;

namespace MyPlugin.Plugins
{
    public class PostPublishAll_PublishDuplicateDetectionRules : IPlugin 
    {
        public void Execute(IServiceProvider serviceProvider)
        {
            //Obtain the execution context from the service provider.

            IPluginExecutionContext context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));

            //Get a reference to the Organization service.

            IOrganizationServiceFactory factory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
            IOrganizationService service = factory.CreateOrganizationService(context.UserId);

            //Extract the tracing service for use in debugging sandboxed plug-ins

            ITracingService tracing = (ITracingService)serviceProvider.GetService(typeof(ITracingService));

            tracing.Trace("Tracing implemented successfully!");

            if (context.MessageName == "PublishAll")

            {
                PublishRules(service, tracing);
            }
        }

        private void PublishRules(IOrganizationService service, ITracingService tracing)

        {
            EntityCollection rules = GetDuplicateDetectionRules(service);

            tracing.Trace("Obtained " + rules.TotalRecordCount.ToString() + " duplicate detection rules.");

            if (rules.TotalRecordCount >= 1)

            {
                // Create an ExecuteMultipleRequest object.
                ExecuteMultipleRequest request = new ExecuteMultipleRequest()
                {
                    // Assign settings that define execution behavior: don't continue on error, don't return responses. 
                    Settings = new ExecuteMultipleSettings()
                    {
                        ContinueOnError = false,
                        ReturnResponses = false
                    },
                    // Create an empty organization request collection.
                    Requests = new OrganizationRequestCollection()
                };

                //Create a collection of PublishDuplicateRuleRequests, and execute them in one batch

                foreach(Entity entity in rules.Entities)

                {

                    PublishDuplicateRuleRequest publishReq = new PublishDuplicateRuleRequest { DuplicateRuleId = entity.Id };
                    request.Requests.Add(publishReq);
                    
                }

                service.Execute(request);

            }

            else

            {
                tracing.Trace("Plugin execution cancelled, as there are no duplicate detection rules to publish.");
                return;
            }
        }

        private EntityCollection GetDuplicateDetectionRules(IOrganizationService service)

        {
            QueryExpression qe = new QueryExpression("duplicaterule");

            qe.ColumnSet = new ColumnSet("duplicateruleid");

            ConditionExpression condition1 = new ConditionExpression();
            condition1.AttributeName = "statecode";
            condition1.Operator = ConditionOperator.Equal;
            condition1.Values.Add(0);

            ConditionExpression condition2 = new ConditionExpression();
            condition2.AttributeName = "statuscode";
            condition2.Operator = ConditionOperator.Equal;
            condition2.Values.Add(0);

            FilterExpression filter = new FilterExpression();
            filter.FilterOperator = LogicalOperator.And;
            filter.Conditions.Add(condition1);
            filter.Conditions.Add(condition2);

            qe.Criteria.AddFilter(filter);

            //Have to add this, otherwise the record count won't be returned correctly

            qe.PageInfo.ReturnTotalRecordCount = true;

            return service.RetrieveMultiple(qe);

        } 
    }
}

The only caveat with the above is that it is arguably only useful for if you are regularly importing Unmanaged, as opposed to Managed solutions, as the Publish All Customizations option is not displayed on the import wizard for unmanaged solutions. Nevertheless, by rolling out the above into your environment, you no longer need to scrabble around for the mental note you have to make when performing a solution update 🙂

Change control and management are important considerations for any IT system – and CRM is no different in this respect. Business processes are often subject to change at the drop of a hat, and organisations should ensure that they have robust, effective and, most of all, efficient processes in place for change management. Often these changes may preclude the removal of a particular aspect of a system – in CRM’s case, this could be a Form, a View or even an Entity field. You may often just decide to take the “easy” way out and not remove these components at all, choosing instead to hide or obscure them. Whilst this is fine in the immediate to short term, you are storing up problems long-term if you do not have a robust process in place to ensure these unnecessary or legacy components are eventually removed. The problem is, though, depending on your customisation deployment method – either as part of managed or unmanaged solution – your options in this regard could be hampered. It is, therefore, important to be aware of what the potential limitations are to both types of solutions so that you can structure your customisations in the most appropriate way. In this week’s blog post, we will take a closer look at some of the for and against arguments when it comes to removing solution components from your production CRM system, the behaviour of solution files and how they can assist and even impede a change management process.

So why should you ensure that unnecessary/legacy components are removed from your Production system? And is there a case to not remove them at all?

  • Components can potentially take up unnecessary space within your solution, leading to delays in performing solution updates. If you also have entity fields in your CRM that are no longer in use but are still storing data, then this could have adverse effects on your database storage levels – an important and essential consideration for CRM Online deployments in particular.
  • Clarity and simplicity are hallmarks of a well managed and maintained system. Being in a situation where you have components in your system, that could easily be interpreted as being still in use or active, could lead to hours, if not days, of confusion and wasted time.

Clearly, there are practical, if not somewhat idealistic, arguments in favour of the above. So what are the arguments against?

  • Removing a component almost straightaway could present problems if, for example, it turns out that the reasons for its removal were mistaken. You would potentially create more work for yourself in having to re-create a particular customisation when keeping it could have saved considerable effort and time.
  • The above can be compounded further if it turns out that crucial business information was stored in, for example, a field that is deleted. Keeping the field intact can ensure that these potential situations are avoided.

Let’s see now things look in practice when we attempt to emulate a change process within CRM

For testing purposes, we have created a custom entity – Test Solution Entity – which contains 2 custom Forms, Views and Fields:

1

23

This entity has been moved into an unmanaged solution, which will be exported as unmanaged and managed and then deployed into a separate CRM environment. We will then observe what happens when we push out an update to the solution, that has had certain components removed – in this case, the following components:

  • Test Form 1
  • Test View 1
  • Test Date Field

Unmanaged

Updating an unmanaged solution will do nothing to existing components that have been deleted – even if you specifically delete them from the solution in your development system. Therefore, as a best practice, any component that you choose to specifically delete from your unmanaged solution will need to be noted down and included as part of your release notes for your solution update. Once the update has been completed, you will then need to go into your production system and proceed with removing these components. Regardless of the type of customisation, you should encounter no problems deleting them – in most cases, all required dependencies for the component will have been removed as part of your solution update and the components themselves will be in an unmanaged state, meaning you are unrestricted in what you can do with them.

Managed

You may assume that a Managed solution update would differ from an unmanaged in behaviour. In fact, for this example at least, when we import our updated managed solution, then the components we deleted are persisted in the solution. What’s worse, because these components are in a managed state, the steps involved in removing them may be complicated significantly. Fortunately, for Forms and Views, there is a Managed Property that can be configured to allow us to delete a Form/View if it is in a managed state:

5 6

These settings always default to True, so you do not need to specifically remember to set them.

There is some bad news, however – there is no such setting available for entity fields:

7

In addition, the following customisation components can not be set to Can be Deleted whilst in a managed state:

  • Entity Relationships
  • Business Rules
  • Global Option Sets
  • Web Resources
  • Processes (Workflows, Dialogs etc.)
  • Reports
  • Connection Roles
  • Templates (Article, Email etc.)
  • Security Roles

This can present a problem over time if we assume that over the lifecycle of your solution, you need to remove redundant fields – these will be maintained and will only be removed if you choose to completely uninstall and re-install your solution. Depending on the nature of your solution, this could cause the following problems:

  • Uninstalling the solution will delete all entity records from the system, as all components in the solution will be deleted.
  • There could be significant time and effort involved in the re-installation process – most likely this will need to be done out of hours, given that everything within your solution will not be available for the duration of the re-install.
  • If you have other unmanaged/managed solutions that have dependencies on components within your managed solution, then this could cause issues with these customisations; and could even mean that you have to re-install these solutions as well.

Conclusions or Wot I Think

The above examples have looked at situations purely where you are making bespoke customisations to CRM. Sometimes deleting components may not be possible at all, particularly if you are using “out of the box” (OOB) components included with CRM. In these cases, you have no choice but to obfuscate these parts of the CRM systems as part of the customisations you make to the system – either through a well-defined security structure or by simply not exposing such elements of the system via the sitemap, for example. Putting OOB components aside, if we are to look at purely bespoke customisation elements of a CRM deployment, I would argue that striking a balance between the two extremes – leaving components that are no longer required in your system compared with deleting them at the first available opportunity – is often the best way to proceed. For example, having an internal process in place that ensures removal requests are signed off by a senior member of the business or by having a “grace period”, where customisations are flagged for deletion but not actioned until a set amount of time has expired. Tied up as part of this, you can very straightforwardly perform backups of your CRM data via the Export to Excel feature. Given how easy and accessible this feature is, there really is no excuse not to perform backups of fields that you intend to delete; then, if the worse happens and you need to restore customisations at a later date (which, on a separate note, should be straightforward so long as you keeping regular backups of your CRM solution files!), you can very quickly restore the data within these fields.

No matter which approach you take, I would argue that it is ultimately preferable to ensure that your CRM solutions are kept in a tidy, current and clear state at all times. You are doing a huge disservice to your current and future colleagues within your business by not following this mantra. In the process as well, you take a rather cavalier approach to what I would hope would be(!) one of your businesses most important assets.

When I first started looking more closely at some of the new features within Dynamics CRM 2016, my initial thoughts was this was that this was a release for the fans (i.e. CRM Administrators, Customisers & Developers). Putting aside some of the big headline grabbing features such as the Interactive Service Hub and Word Templates, there looks to be a lot of subtle changes underneath the hood which those who work with CRM on a daily basis will be jumping for joy at.

One of the fundamental concepts that needs to be grasped as part of any CRM development work is Solutions. This includes understanding the differences between managed/unmanaged solution files, how to export/import solutions between different environments and how managed/unmanaged customisations are handled by CRM. There are many debates and discussions online regarding the subject, but the general rule of thumb is to use unmanaged solutions for the majority of your customisations, and only use managed solutions for when you are an ISV developing a bespoke solution to performs a specific function and/or links in with a separate application.

One of my personal bugbears regarding Solutions is the time and effort involved as part of doing an update. If, for example, you have just one Solution file for your entire businesses CRM customisations, then over time pushing out a solution update into your Production environment can take up to 30 minutes or more to complete – even if your update only contains, for example, a field name change! This can make pushing out hot-fixes or urgent updates more time-consuming and lead to delays in addressing problems within a live production system.

Whilst doing some digging around within CRM 2016, I was very excited to see the following 2 new buttons on the Solutions page:

CRM2016SolutionChanges_1

 

 

Could it be that we now have a better and more efficient way of pushing out small parts of our CRM customisations and urgent hot-fixes? Before we take a look at how these buttons work in practice, it is useful to first explain what each one does in more detail:

Clone a Patch

Clone a Patch creates a patch of your main solution, which can contain a selection of CRM customisations that you wish to export from the system. A solution can have multiple patch solutions. One thing to point out is that you must manually add the components you wish to include as part of your patch solution into the Patch solution file. When, for example, you add in the Entity which you wish to include in the patch, you can specify individual entity forms, views, charts etc. that you want to include, something which I really like:

CRM2016SolutionChanges_6

Whilst there is at least one patch of a solution present in your CRM system, you will not be able to edit the main solution file. CRM presents you with a yellow banner message to inform you of this when you enter the CRM Solution in question:

CRM2016SolutionChanges_5

 

In order for the solution to become available for customisation again, you will need to either a) delete all patch solution files that derive from the main solution or b) use the Clone Solution button, which will be explained in more detail next.

Clone Solution

Clone Solution enables you to quickly combine together a solution and all of its descendant patches back into one solution file. Say for example, you have 1 Main Solution file and 3 Patch Solutions from the Main Solution. Clicking the Clone Solution button will combine all of these together again as part of 1 solution file. As a result, this will then allow you to customise the main solution file again without any issues.

So how does this work in practice then? Let’s find out:

To start of with, here is a custom entity within a new solution file. Nothing has been customised for this entity at this stage:

CRM2016SolutionChanges_2

This is then exported out of the system as a solution file and imported into our target environment. Next, let’s say we want to add a newly created field to our entity as well as making some other changes as well within the main solution. These are in an incomplete stage, so therefore we just want to export our new field only as a patch solution for now.

On the Solutions page, we select our Main solution file and press the Clone a Patch button. We are greeted with a screen similar to the below where we can specify a few optional details, before then confirming:

CRM2016SolutionChanges_3 CRM2016SolutionChanges_4

Next, we then have to go into our Patch Solution and add in the elements that we wish to push out as part of a patch. In this case, we want to add a new field to our Test Entity. We therefore go into a Patch Solution file and go to Add Existing… in order to add in our Test Entity. You will be greeted with the Entity Assets window above and, in this example, we want to include all assets so we just press Finish to add this in.

Now we want to add in our new field:

CRM2016SolutionChanges_10

CRM2016SolutionChanges_9

As above, this is then imported into our target CRM environment, as you would do normally for any other solution file. It even looks the same!

CRM2016SolutionChanges_13 CRM2016SolutionChanges_14

 

Now, lets say we want to pull together all of our other changes as part of a traditional solution update. We also want to rename our Testing Field as follows:
CRM2016SolutionChanges_12

We select our Solution file and click on the Clone Solution button. At this stage, we can modify the display name of the Solution and also specify a new version:

CRM2016SolutionChanges_11

When we click Save, the patch solution file will vanish from our solution file, replaced by our one solution file which can now be edited and exported accordingly.

And that’s it! It takes a while to get your head around, particularly if you have been used to working with Solutions in previous versions of CRM, and I think it will take a while before this starts to become widely used (to be honest, I have not yet used it as part of some of the recent solution updates I have had to do 😳). Still, it is good to know that for those who are just coming into CRM Administration/Development, you can get to grips with this feature straight away without having to concern yourself too greatly with the “old” method. Just don’t be surprised if you hear from CRM veterans something like “Back in my day, we just had Solution updates – and we were lucky to have them!”

Let me know in the comments below if you have started to use Solution Patches yourself within your CRM 2016 environment.

When working with CRM extensively across multiple environments, you do start to get into a fairly regular rhythm when it comes to completing key tasks again…and again…and again. One of these tasks is the process of rolling out a Solution update. To briefly summarise, this is where you export an updated version of your solution from your development environment and then import the solution file on top of a previous version in a different instance. CRM will then go through the solution import file, detect and then apply any changes which have been made. The original solution will be overwritten as part of this and you will be informed at the end of the import on any warnings or errors that were encountered during import. Warnings will generally not cause the solution import to fail, whereas errors will stop the import completely.

Like with anything, Solution updates can sometimes fall-over for a whole host of different reasons. They can fail altogether, or sometimes just hang and become unresponsive. If you are running On-Premise CRM, then you can interrogate the SQL database on the instance to see how your solution import is (or is not) progressing. Ben Hosking (whose blog is mandatory reading for anyone who works closely with CRM) has written a really useful post which contains the SQL query you need to use on your organization database in order to identify any problematic job imports.  The good thing with this approach is, if the import has errored, the Data column contains the raw XML file that you are able to download via the CRM GUI using the ‘Download Log File’ button below when a solution import proceeds as you would expect normally:

SolutionUpdate_ExportLog

You can therefore very quickly drill-down to see if the solution import has failed due to a customisation issue. A common reason for failure may be due to duplicate logical name(s) for attributes, relationships etc.

If you are using CRM Online, then the first assumption (which I admittedly made) may be that there is no way in which to access the above information. Fortunately, there is an entity called ‘importjob’ that is made available for access via FetchXML

So, for example, to return all solution imports that have gone through your CRM, you would use the following FetchXML query:

<fetch version="1.0" output-format="xml-platform" mapping="logical" distinct="false">
  <entity name="importjob" >
    <attribute name="completedon" />
    <attribute name="solutionname" />
    <attribute name="progress" />
    <attribute name="startedon" />
    <attribute name="completedon" />
    <attribute name="createdby" />
    <attribute name="data" />
    <attribute name="organizationid" />
    <attribute name="createdon" />
  </entity>
</fetch>

If you wanted to just return solution imports that are either stuck at 0% or have not yet successfully completed:

<fetch version="1.0" output-format="xml-platform" mapping="logical" distinct="false" >
  <entity name="importjob" >
    <attribute name="completedon" />
    <attribute name="solutionname" />
    <attribute name="progress" />
    <attribute name="startedon" />
    <attribute name="completedon" />
    <attribute name="createdby" />
    <attribute name="data" />
    <attribute name="organizationid" />
    <attribute name="createdon" />
    <filter type="or" >
      <condition attribute="progress" operator="eq" value="0" />
      <condition attribute="completedon" operator="null" />
    </filter>
  </entity>
</fetch>

Sometimes your import may fail due to a problem with CRM itself. In these instances, the best course of action depends, once again, on your CRM deployment:

  • For On-Premise users, then the old IT adage applies here: try restarting the CRM and SQL Database Server instance. You may first need to locate the active process on SQL Management Studio that is performing the solution import and kill the process first. A database instance reset should automatically cancel this and prevent it from running again on instance startup, but its better to be safe than sorry.
  • The only recourse for Online users is to log a support request with Microsoft via the Office 365 portal. It is best to provide as much evidence as possible up-front and be advised that the Microsoft support engineer may ask you to demonstrate the problem again, which might prove difficult to perform during normal working hours if the problem is happening on your Production instance.

I was glad to discover that there is half way method of being able to interrogate possible platform issues yourself on CRM Online, but this particular example illustrates one of the drawbacks of CRM Online: little or no direct access to the organization database and/or instance level services. I would hope that in time Microsoft may develop the platform further in order to start exposing these elements to organization administrators. I would imagine that the vast majority of support queries that go through on the Office 365 portal would relate to requests that could be safely performed by CRM Administrators or Partners, leading to a cost and efficiency saving for Microsoft should this be implemented.