Experience Sitecore! | All posts tagged 'Presentation'

Experience Sitecore!

Martin Miles on Sitecore

My SUGCON Presentation: The Mastery of Sitecore Upgrades

I am proud to be chosen as a presenter at SUGCON 2022 which took place in Budapest. 

This blog post contains the supporting material for my topic "The Mastery of Sitecore Upgrades".

Content


Why upgrade?

Why do we upgrade Sitecore given it is not that quick and easy? The answer is simple – a carrot and a stick motivation!

The Stick: every New Year Mainstream support expires for one or a few versions. This means Sitecore still provides security updates and fixes, they do not support development and compatibility queries. Read more details about it at Sitecore Product Support Lifecycle.

Now, the Carrot: companies update due to the new features or new experiences. For example, you may read a detailed article I wrote about those in the latest version - New features in 10.2 you must be aware of.

Also, having the latest version is especially valuable in front of Composable DXP, which already takes place:


Planning the upgrade

Propper planning is the key to success. How do we perform the planning and what should be considered?

1. Upgrading vanilla Sitecore would cost you minimal effort, but with the more custom code you have - the more labor is expected to commit. How much is it customized? These will affect your timescale and impose additional risks which is better to know in advance.

2. If you get access to any of the existing documentation - that would be a perfect place to start. And it will likely give some answers to the previous point.

Find the details about all environments, code branches, and which code branch is deployed to which environment. Also, you will be interested in the details about existing CI/CD pipelines.

3. Find the custom configurations made into the Sitecore.

4. Finally, talk to stakeholders! Who if not they know the details and are the most interested in the success!

5. It sounds logical that the more versions you jump through – the more complicated the upgrade process will be.

However, that isn’t a rule of thumb - some versions have very minor changesets and could be updated with minimal effort. Others - the opposite, could be difficult as hell.

For example, it is the exact reason 9.3 stands out from this version chain. It happened that it took most breaking changes, deprecations, and internal housekeeping improvements than any other version did.

6. Because of that, one of the most valuable planning activities - investigating Release Notes for every single version on your upgrade path. Pay special care to these two sections: Deprecated/Removed & Breaking changes.

7. Identify the functionalities which are no more supported including any third-party add-ons/modules and find their alternatives.

8. With new platform features license file format changes with time. Even with the most permissive license, your old license file may be incompatible with the newer platform. That for example happened when version 9 was released.

It is good to care about that in advance, as the process takes some time. Please check that with your account manager.

9. Every solution is unique, and the same unique the teams are. Estimates must consider these factors, along with previous relevant experience of your team. Make sure you will have the resources for the whole duration of the upgrade also care about the fallback plan if someone gets ill or leaves (also known as "bus factor").

This diagram shows a very optimistic view of the team of two experienced Sitecore professionals, performing the upgrade in 4 typical sprints. While sharing the pool of tasks with each other, one of them is primarily focused on the codebase, while another more cares about DevOps things.

Once again, this is not guidance or any sort of assessment. But just a high-level view of teams’ activity through sprints.

10. Also take a look at Sitecore compatibility guide to ensure all your planned tech spec stays valid.


Recap legacy tools

Before heading to the гpgrade еactics, let’s quickly recap some upgrade tools we did have over time and how they performed.

1. Express Migration Tool

You can use Sitecore's Express Migration Tool to move over data from your old instance to your new one. It supports migrating from previous old versions to Sitecore 9 initial version. Express Migration Tool copies items and files from one Sitecore instance at a time.

The Sitecore Express Migration Tool copies items and files from one Sitecore instance at a time. The tool supports the migration of remote servers.

2. The update Center

Update Centre was introduced in Sitecore 9.0 Update 2 and is valid up to Sitecore 10.1.

You can find, download, install, and manage updates and hotfixes for the Sitecore platform and Sitecore modules. The Update Center accesses a package management service provided by Sitecore, or you can install and provide a service yourself.

The Sitecore Update Center uses this URL of the Package Management Service to check for the Sitecore updates:

<add name="PackageManagementServiceUrl" connectionString="https://updatecenter.cloud.sitecore.net/" />


Upgrade tactics

Next, let’s talk through some upgrade tactics that can significantly help your upgrade process.

1. Affixing the states and being able to switch between them as quickly as possible is crucial for productivity when working on instance upgrades. Small mistakes are inevitable and we actually need some sort of Undo button for the whole process.

We already have this for the codebase – git, with its ability to switсh between the branches.

But what about the published sites, indexes, certificates, and the rest of the minors that matter? Even database backups are not that fast and straightforward.

The one and only solution that comes into mind is fixing the whole state of a working machine. Something that one could have with a Virtual machine. Luckily, we have at least one suitable technology:

2. Hyper-V

Actually, this ticks all the boxes:
  • free and included in Win Pro
  • extremely fast with SSD
  • maximum OS integration
  • move backups between hosts
  • perfect networking options
  • remote management
  • universal virtual drives

However, you will need a relatively fast SSD and you should mind the disk space – it easily gets eaten out by multiple snapshots as you progress. Having a 1TB drive is probably the minimum you'd need to have for productive work on and upgrade.

3. Leverage PowerShell

Now we are confident with reverting bad things quickly. But how about redoing successful steps? With a lot of monotonous and repetitive actions, one could lose very much valuable time for things they will re-do again and again.

Let’s think about the case when someone spent the last 2 hours upgrading NuGet references for a Helix solution with 100 projects in it. Unfortunately, at last-minute something went totally wrong - restoring to the latest Hyper-V checkpoint will take less than a minute, but should one repeat these monotonous steps again and again?

One could think - why not to make restore point more often. Well, creating a restore point for every minor step seems to be an overkill here, an also Hyper-V will quickly eat out all of your disk drive space.

It is fairly difficult to perform an upgrade of everything from the first attempt. We will have to repeat the successful steps, again and again, so the only matter would be automating them. PowerShell built in the OS is the best and natural fit for such activities. You can leverage PowerShell for:

  • Any sort of automation, system tasks and jobs
  • Mass replace configuration and code with RegEx
  • Backup and restore databases and web application
  • Managing you infrastructure: either local or cloud
  • Leveraging SPE Remote to operate your instance: Managing content, security, publishing, and indexing
So what would and could be a good idea to write these scripts for? In addition to the above points, consider scripting some activities that take longer to complete and/or which result in a minor textual change in files.
Returning to the above case os upgrading NuGet references in a large solution, that took 2 hours to complete - on your disk it ends up with just a few lines of difference in a project or packages file. That means, comparing the difference “before and after” (with a diff tool) it is fairly easy to use PowerShell in order to process all such files performing regular expression replace. Not being a master of regular expressions, you're unlikely to succeed from the first attempt, but with a quick and easy checkpoint restore option you’ll quickly master it. Also, you'll end up with the artifacts: successful scripts that could be used for your future upgrades with minimal or no alterations.

I wrote a good long article about PowerShell best practices and all the aspects of leveraging this built-in scripting language. It is a long read, but highly valuable.

4. Quick Proof-of-Concept

In some cases, when you are performing a jump to the following version (or maybe a few) and if your solution is relatively simple staying close to vanilla Sitecore, it makes sense to pitch PoC. That will either give you a quick win or if not - then at least will identify most of the traps and things to improve. Having those open up better opportunities for planning.

Ideally, it should be a one-day job for a single professional.

5. Try to get a recent backup of a master database

It’s a highly desirable thing to obtain a relatively recent backup of the master database. I do realize, it is not always possible to do: sometimes due to the large size or more often because of the organization's security policy, especially when you're working for a partner agency or being an external contactor.

But why do you need that?

First of all, you restore it along with the existing solution to verify the codebase you've obtained performs exactly the same and on the published website(s). Of course, if you work for a given organization and were standing in charge of the project under upgrade from day one - this step could be dismissed, as you fairly know the codebase, infrastructure, the history of decisions taken over the project lifetime, and most of the potential traps.

But what is more important, you will need to upgrade and use this database with a new solution instance to ensure that also looks and works as it functioned before. That would help you eliminate most of the unobvious bugs and errors just before regression testing starts, which will save lots of time.

A good example from my own experience: I committed and upgrade from 8.2.7 to 10.0.1 and everything seemed to be successful. I used an upgraded full master database, I was visually comparing old sites running at the existing production with those I have locally upgraded to the latest Sitecore. Accidentally I spotted that some small SVG icons were missing from an upgraded site. Escalating this issue brought me to one of the breaking changes that would be difficult to spot - the codebase builds, no runtime errors, and no obvious logs issues.

6. What if a newer Sitecore release comes while you’re upgrading?

The upgrade process takes a decent amount of time and it may be a case while a new Sitecore version released meanwhile. My best advice will be to stick to the newer version. Just take it!

First of all, it may be not be an easy task from an organizational point of view, especially if you are working for a partner rather than an end client and the version has been signed off. It will take lots of effort to persuade stakeholders to consider the latest version instead, and you have to motivate your argument pretty well.

Upgrading the solution to the newer build when you a halfway done upgrading to a previous one adds some time but will cost times less than upgrading once you go live. It will be a good idea in most of the cases with the only exclusion of the main component getting deprecated in the newer version (for example once I was upgrading to 9.0.2 and then 9.1 has been released, but due to large usage of WFFM it was not possible to easily jump to 9.1 as WFFM was deprecated). Otherwise, we should have chosen a newer version.

Choosing a newer version in fact could cost you even less. That happened to me while upgrading to 10.0.1 when the newer 10.1 got released. Switching to 10.1 from 10.0.1 would cost me very little effort, and also would reduce operational efforts due to the specific advantages of Sitecore 10.1 - but unfortunately, the decision chain to the stakeholders was too long (I was subcontracted by an implementing partner), and that never happened.

7. Take comprehensive notes

Last but not least advice may seem to be obvious, but please take it seriously. Please document even minor steps, decisions, and successful solutions. You will likely go through them again, and maybe several times. Save your scripts either.

An even better job would be if you could wrap all your findings into a blog post and share the challenge and solution with the rest of the world. Because of our blogs being indexed, you’ll help many other people. Several times in my career I googled out my own blog posts with the solution while facing a challenge and googling it out.


Content migration

When asking my colleagues what was the biggest challenge with their upgrade experience, almost everyone responded - migrating the content.

1. Content maintenance

Depending on your level of ownership, it may be a good idea to undertake content maintenance. This is optional but desirable.

Remove legacy version - it is a know good practice to keep less than 10 versions per item improving content editing performance. Once I saw 529 versions of the site's Home item and doubted the authors did need them all.

You can remove legacy items manually by running the SPE script (Rules Engine Actions to Remove Old Versions in the Sitecore) or perform that on a regular basis by creating a Rule in Rules Engine to automatically keep a number of versions less than the desired number (PowerShell Extensions script to delete unused media Items older than 30 days).

Clean up Media Library as it is the most abused area of content, typically. Untrained or simply rushing editors often place content without any care. Almost every single solution I’ve seen had some issues with the media library: either messed up structure, lots of unused and uncertain media, or both. These items are heavyweight and bloat database and content tree without bringing any benefit. We got a PowerShell Extensions script to list all media items that are not linked to other items, so having those you may revise the list and get rid of those unwanted.

You may also want to clean up broken links prior to doing an upgrade. Sitecore has an admin page that allows removing broken links. You can find it in the Admin folder: RemoveBrokenLinks.aspx - just select the database and execute the action.

From 9.1 default Helix folders comes OOB with databases so became universal. But lots of Helix solutions were implemented before and their folders IDs do not match those coming OOB in later versions.

For example, the existing solution also has folders like /sitecore/Layout/Renderings/Feature but they are not coming out of the box and therefore serialized, but what is even worse - having different IDs from those coming OOB that are now universal.

You’ll need to rework serialization to match the correct parent folders coming OOB.


2. Content migration options

In fact, you've got plenty of options to migrate the content. Let's take a look at what they are.

Sitecore Packaging

This is the default way of occasionally exchanging items between instances known to every developer. However, it is extremely slow and does not work well with large packages of a few hundreds of megabytes.

Sidekick

Is free and has a Content Migrator module using Rainbow serialization format to copy items between instances in a multi-threaded way. Unlike packages it super-fast. Keep in mind that both servers need to have the content migrator installed on them.
Sitecore Sidekick (by Jeff Darchuk)

Razl

Offers quite an intelligent way of copying items between the servers. However, this software is not free and requires a license to run.

Sitecore Razl: Tool for Compare and Merge


Sitecore PowerShell Migration

We've got a script to migrate content between Sitecore instances using Sitecore PowerShell Extensions (by Michael West) which also leverages Unicorn and Rainbow serialization. This script is extremely fast, but it requires SPE with Remoting enabled on both instances.

Move an item to another database from Control Panel

If none of the above options work for you, there is a built-in tool to copy items between databases that is part of the Control Panel. The trick is to plug the target database as an external database to an instance so that it becomes seen in Sitecore, then you’ll be able to copy items using this option. It has limited functionality and works reasonably slow, but allows copying data both ways.

If you are migrating content to version 10.1 or newer, there is a much better way of dealing with a content and database upgrade, which I will explain in detail below.


3. Content security

There are a few options to transfer Users and Roles from one instance to another.

Sitecore Packages

You can use the standard Package Designer from the Development Tools menu in the Sitecore Desktop. You can then add the Roles and Users that you want to package up for migration, generate the package, download it and then install it at the target instance in the same way you would do for content.

Serialization

An alternative is to use the Serialization option from within the User Manager and Role Manager applications. The users and roles will be serialized to (data)/serialization/security folder. You can copy this from the source instance to the target instance and then use the revert option.

For both of these options, the user's password is not transferred and instead reset (to a random value when using Sitecore Packages or to "b" when using serialization).

How to deal with the passwords?

You can then either reset the password for the users manually (from the User Manager ), or the users themselves can reset the password by the "forgot your password" option from the login screen, assuming the mail server has been configured and they get password recovery email.

You also have the option to use the admin folder TransferUserPasswords.aspx tool to transfer the passwords from the source and target databases. To do so you will need both connection strings and these connections must have access enabled. Read more about Transferring user passwords between Sitecore instances with TransferUserPasswords.aspx tool.

Raw SQL?

Without having the required access, you could do that manually with SQL. The role and user data are stored using ASP.NET Membership provider in SQL Server tables in the Core database. Please note that Sitecore 9.1 and newer Membership tables could be extracted from Core into their own isolated Security database (moving Sitecore membership data away from Core db into an isolated Security database).

So it is possible to transfer the roles and users with a tool such as Redgate SQL. You will need to ensure you migrate user data from the following tables:

aspnet_Membership
aspnet_Profile
aspnet_Roles
aspnet_Users
aspnet_UsersInRoles
RolesInRoles

You may adjust and use SQL script to migrate content security when both source and target DBs are on the same SQL server. Also, follow up a StackExchange answer about Moving users/roles/passwords to a new core database (by Kamruz Jaman).


4. Dynamic placeholders format

From version 9.0 the became an integral part of the platform, unlike previously we used 3-rd party modules. However, Sitecore has implemented its own format of addressing components: {placeholder key}-{rendering unique suffix}-{unique suffix within rendering}

That means your previous layouts will not be shown on the page unless you update presentation details for these differences otherwise the renderings. For example, if you have a component on a page in an old dynamic placeholder format, you need to change it from:

main_9d32dee9-17fd-4478-840b-06bab02dba6c

to the new format, so it becomes:

main-{9d32dee9-17fd-4478-840b-06bab02dba6c}-0

There are few solutions how to approach that

  • with PowerShell Extensions (by Rich Seal)
  • by creating a service (or admin folder) page: one, two
Also, you no longer need a 3-rd party implementation module, so remove it (both DLL and its config patch file). Its functionality was "moved" into built-in Sitecore MVC libraries which you of course need to reference from web.config:
<add namespace="Sitecore.Mvc"/>
<add namespace="Sitecore.Mvc.Presentation"/> 

Read more: an official guide on dynamic placeholders


Upgrading codebase

The most central part of my presentation comes about upgrading the actual codebase and the challenges with it. To start with I want to share two useful tips to identify the customization of a given project.

Trick 1 on how to Identify customization

In order to estimate to amount of customization for a solution, I do the following quick trick.

First of all, I need to have a vanilla version of Sitecore that the existing solution is using.

After it is up and running, I come to the webroot and do: git init and then commit everything. It’s just a local git initialized within Webroot - I do not push it anywhere and delete git files immediately after completing this exercise.

Next, I build and publish the codebase as normal, so that all the artifacts come above vanilla Sitecore into a Webroot.

Finally, my git tool of choice shows the entire delta of all the customizations of that solution. With that trick I could also easily see all web.config transforms and mismatching or altered DLLs, for example, provided with hotfixes.

That trick gives me the overall feel of how much that instance varies from the vanilla one.

Trick 2 on how to find customization with ShowConfig

When previous trick 1 mostly aims to show the overall amount of customizations, this one would be more specific about the changes in configuration. It will show all the patches and config alterations done on top of the vanilla instance so that a more precise estimation could be done.

We already got vanilla Sitecore of the legacy instance version at the previous step. What we need to do is run showconfig.aspx admin folder tool for generating the combined configuration. And save it into an XML file.

Next, once again perform build and publish as normal. Once the instance gets up and running – run showconfig tool again and save the output into a different file.

Now, having an actual solution configuration we can compare it against a vanilla combined configuration for that same version of the platform – using any diff tool. I recommend using advanced Beyond Compare.

For your convenience, you could also extract the whole delta between both into a single config file – it will be a valid patch!

3. Incorrect DLL(s) coming from the deployment

While upgrading the codebase you might also find facing a dependency hell in terms of referenced assemblies not playing well with each other. This can take an age to fix.

Every Sitecore release contains an Assembly List featuring a complete list of assemblies shipped with this release in the Release Information section. (for example, there are 400 assemblies exactly shipped for the 10.2 platform).

DLLs must exactly match counterpart vanilla DLLs in versions and also in size unless that is a hotfix DLL.

With a previous first trick, you could identify all the mismatching DLLs, and with the second trick – where those are referenced from.

You may also perform a solution-wide search for a specific faulty assembly and then see all the found results along with their sized in bytes. At least one would mismatch – that will point you to the specific project where the faulty one is coming from. I would recommend my file manager of choice – Total Commander which is perfect for such operations.

If the previous result gives you more than one – but lots of faulty referenced assemblies, you could rely on PowerShell for performing RegEx replace all the invalid references with the correct ones. It may have a steep learning curve initially, but very soon this technique will start saving you lots of time.

4. Troubleshoot the assembly bindings

This issue occurs due to the dependencies used in your solution projects, that mismatch those DLLs versions that are actually shipped with vanilla Sitecore. For example, your feature module uses Sitecore.MVC dependency, which in turn relies on System.Web.MVC. When you add Sitecore.MVC using NuGet package manager, it will pull the latest dependency, but not the one that was latest and referenced at the time when Sitecore build was released. Or in case if you add a library ignoring dependencies, you could process with the latest dependency yourself.

Two ways of getting out of this. One is strictly hard referencing your project dependencies to match those from webroot. Troubleshooting becomes really annoying when upgrading the solution with a large number of projects (the max I have seen was 135) or in rare cases not possible.

Another approach is advised to adjust particular Assembly Binding to the latest version. Since you should not modify vanilla config manually – you may employ the config transform step as a part of your local build & deploy script (see using web.config transforms on assembly binding redirects).

The situation has improved with 10.1 as libraries have been updated to the latest versions and bindings became less strict.

5. NoReference packages

Sitecore no longer publishes the NoReferences NuGet packages. For example, if you install the 9.3 packages you will get a lot of dependencies installed. Not what we wanted.

The correct way, in this case, is to use the Dependency Behavior option of the Nuget Package Manager.

Choosing the dependency behavior to “IgnoreDependencies” will only install the package without any of the dependencies.

This function is also available as a parameter switch in PowerShell.

6. Migrate from packages.config to PackageReference

Just like a project to project references and assembly references, PackageReferences are managed directly within project files rather than using separate packages.config file.

Unlike packages.config, PackageReference lists only those NuGet packages you directly installed in the project.

Using PackageReference, packages are maintained in the global-packages folder rather than in a packages folder within the solution. That results in performing faster and with less disk space consumption.

MSBuild allows you to conditionally reference a NuGet package and choose package references per target framework, configuration, platform, or other pivots.

So if you are convinced - please perform the update either manually from right-click context menu in Visual Studio or by running a migrate from packages.config to PackageReference script by Nick Wesselman.

7. Update Target Framework

Net Framework 4.8 is the terminal version of Framework and is being used from Sitecore 9.3 onwards.

Migrating from earlier solutions will require you to update Target Framework for every project within a solution. I personally prefer to do that in PowerShell (PowerShell way to upgrade Target Framework), but there is a Target Framework Migrator Visual Studio extension that also bulk-updates Target Framework for the whole solution.

8. Update Visual Studio

But of course, to benefit from this extension you may need to update Visual Studio itself first so that it has the required Target Framework. Least and most important – install the latest VS Build Tools which you of course can be updated on your own.

You will still likely be keen to get the latest VS, the 2022 worked well for me but is not yet officially supported. In that case, you can get at least VS 2019 which also has nice features.

Helix Solutions may have up to a hundred projects that affect overall performance. Solution filters allow filtering only those projects you’re working with, keeping unloaded from VS but still operable. It works as an SLNF-file that references your solution with an array of whitelisted projects.

You can also now perform one-click code cleanups and also search for Objects and Properties in the Watch, Autos, and Locals Windows.

9. Unsupported third-party libraries

The solution you’re upgrading may reference some third-party libraries that are discontinued. In that case, you need to do an investigation for each of those libraries and decide what to do.

Example from my experience. I came across Sitecore.ContentSearch.Spatial library that performed geo spatial search. It has the source code available but has not been updated for 5 years so far. Meantime Lucene search has been removed from Sitecore and this library became no longer relevant, as the library hard referenced Lucene.

As the outcome, the whole related feature at the solution was temporarily disabled to be later rewritten using Solr Spatial search.

10. Dependency injection

Microsoft.Extensions.DependencyInjection became built into Sitecore from version 8.2

It is fast and reliable and supports pretty everything you may need. Just use this one!

Read more: Sitecore Dependency Injection and scoped Lifetime (by Corey Smith)

11. Glass Mapper

You will need to update Glass Mapper to version 5 unless already done. Package naming has changed to reflect the version of Sitecore it is working with, and are three related packages:

  • Glass.Mapper.Sc.{Version}
  • Glass.Mapper.Sc.{Version}.Mvc
  • Glass.Mapper.Sc.{Version}.Core

Glass has lots of changes in v5 most significant of which would be a new way of accessing content from Sitecore. Glass Controller gets obsolete, same is true for GlassView which went in favor of elegancy:

<!-- instead of of GlassView -->
@inherits GlassView<ModelClass>

<!-- it becomes -->
@model ModelClass

Instead of ISitecoreContext using the new IMvcContext, IRequestContext and IWebFormsContext abstractions.

public class FeatureController : Controller
{
    private readonly IMvcContext _mvcContext;
    public FeatureController()
    {
        _mvcContext = new MvcContext();
    }
    var parameters = _mvcContext.GetRenderingParameters();
    var dataFromItem = _mvcContext.GetDataSourceItem();
}

Glass lazy loads by default now and [IsLazy] attributes on model classes were removed. Other attributes removed are [SitecoreQuery], [NotNull].

You will need to remove these model properties in order to re-implement them as methods, getting rid of the attributes (see Lazy loading after upgrading to Glass Mapper 5).

After upgrading Glass if faced a bunch of run time errors that were difficult to identify. Eventually, I realized that happened due to a lack of virtual property modifiers in model classes. With old versions of Glass Mapper, model properties still map correctly even without the virtual modifier, but not any longer.

[SitecoreChildren(InferType = true)]
public virtual IEnumerable<Model> InnerChildren { get; set; }

Tip: Installing a higher version of the GlassMapper with NuGet package make sure the files App_Start\GlassMapperSc.cs and App_Start\GlassMapperScCustom.cs are not overwritten with the defaults. Also, they should not appear in the feature modules, but only in the Foundation module that is responsible for Glass Mapper.

Before upgrading GlassMapper I would recommend reading through these great blog posts about the changes:

12. Custom Pipelines

Sitecore try and reduce breaking changes where possible but sometimes they are unavoidable, so as with all Sitecore upgrades some custom code and configuration will need to be updated to be compatible with

The HttpRequestArgs.Context property has been removed in favor of HttpRequestArgs.HttpContext – in fact, just been renamed. It resulted in a breaking change for all of your custom pipeline processors, which you all need to rewrite. I used my preferred PowerShell RegEx to replace one-liner to update them in one go:

gci -r -include "*.cs" | foreach-object {$a = $_.fullname; (Get-Content -Raw $a -Encoding UTF8) | `
foreach-object {$_ -replace 'args\.Context\.','args.HttpContext.' } | `
set-content -NoNewLine $a -Encoding UTF8}

Custom processors patched into a pipeline now required mandatory attribute resolve.

<CustomPipeline>
    <processor type="ProcessorType, Library.DLL" 
        patch:instead="processor[@type='Sitecore.Mvc.Pipelines.Response.RenderRendering.GenerateCacheKey, Sitecore.Mvc']" 
        resolve="true" />
</CustomPipeline>

13. Link Manager

One more change in version 9.3 takes place with LinkManager in the way we get the default option for building a URL:
var options = LinkManager.GetDefaultUrlBuilderOptions();
var url = LinkManager.GetItemUrl(item, options);

Long story short - its internals has been rewritten to the best, and are mostly hidden from our eyes. What we need to know is an updated way of patching default URL options. The old way of patching

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <linkManager defaultProvider="sitecore">
      <providers>
        <add name="sitecore">
          <patch:attribute name="languageEmbedding">never</patch:attribute>
          <patch:attribute name="lowercaseUrls">true</patch:attribute>
      </add>
      </providers>
    </linkManager>
  </sitecore>
</configuration>

now became more elegant:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <links>
      <urlBuilder>
        <languageEmbedding>never</languageEmbedding>
        <lowercaseUrls>true</lowercaseUrls>
      </urlBuilder>
    </links>
  </sitecore>
</configuration>

Read more about LinkManager changes to link generation in 9.3 (by Volodymyr Hil).

14. Caching changes

The solution I was upgrading had an publish:end event configuration to do HTML Cache clearing for different site instances:
<events>
  <event name="publish:end">
    <handler type="Sitecore.Publishing.HtmlCacheClearer, Sitecore.Kernel" method="ClearCache">
      <sites hint="list">
        <site hint="apple">apple</site>
      </sites>
    </handler>
  </event>
  <event name="publish:end:remote">
    <handler type="Sitecore.Publishing.HtmlCacheClearer, Sitecore.Kernel" method="ClearCache">
      <sites hint="list">
        <site hint="apple">apple</site>
      </sites>
    </handler>
  </event>
</events>

Since 9.3 such behavior became default after a publish. It actually works the other way around now - you have to actually disable HTML cache clearing explicitly if you need to.

The previous code will break, and you have to remove that whole section.
<site name="apple" cacheHtml="true" preventHtmlCacheClear="true" … />

15. Forms

WebForms For Marketers, one of the most questionable modules, was deprecated in Sitecore 9.1 in favor of built-in Sitecore Experience Forms. That also raised questions on how to keep the data and migrate the existing forms while performing platform upgrade

Thankfully there is a tool to do exactly that: convert WFFM forms and data to Sitecore Forms. WFFM Conversion Tool (by Alessandro Faniuolo) is a console application that provides an automated solution to convert and migrate Sitecore Web Forms For Marketers (WFFM) forms items and their data to Sitecore Forms. It takes WFFM data from a SQL or MongoDB database as a source into the destination - Sitecore Experience Forms SQL database.

16. Support Patches & Hotfixes

You will likely find some hotfixes or support patches that have been applied to your solution over time. Sitecore Support addresses issues by providing support DLLs and releasing hotfixes.

Each of them sorts out some certain issue which you need to investigate if it was resolved in the version you’re upgrading to. Use this 6-digit support code to find more details by using a search at Sitecore Knowledge Base website.

When it comes to hotfixes when they keep an original name of DLL they replace, the knowledge base code could be found from the file properties dialog.

Once it got resolved, you may remove those along with any related configuration. This post is aimed at giving an idea of how we can identify if we have hotfix DLLs within Sitecore \bin folder so that we could approach them individually.

17. SXA

Is not an integral part of the platform and it has its own upgrade guidance released along with each version. So I only touch it briefly.

As is true for the rest of the Sitecore platform, the most of SXA changes took place with the 9.3 release. As for me, the biggest change was NVelocity deprecation from 9.3 in favor of Scriban templates, which affects Rendering Variants - one of the most powerful features of SXA.

Also since that version, it has a version parity with its holding XP or XM.

Read more:

SearchStax and Sitecore: The top integration benefits for search optimization and personalization

Both Lucene and Azure Search became obsolete and removed. With Sitecore 10, Solr became the recommended technology to manage all of the search infrastructure and indices for Sitecore. Managed Solr allows your developers to implement faster, and spend more time focused on building a better search experience and less time supporting search infrastructure.

Please migrate your solution, it might take significant effort.

Consider using SearchStax which takes from you the pain of installation, failover, security, maintenance, and scale. SearchStax provides solutions to teams for the most challenging issues for developing with Solr.

There are also a few commonly met issues while upgrading Solr. Sometimes you can come across the case of core name mismatching index name. This is a fairly simple issue and could be fixed with a config patch:

<configuration>
  <indexes>
    <index id="sitecore_testing_index">
      <param desc="core">$(id)</param>
    </index>
  </indexes>
</configuration>

The second issue relates to a configuration that defines Solr index. If you are indexing all fields – that must be wrapped with documentOptions tag of the correct type:

<defaultSolrIndexConfiguration>
  <documentOptions type="Sitecore.ContentSearch.SolrProvider.SolrDocumentBuilderOptions, Sitecore.ContentSearch.SolrProvider">
    <indexAllFields>true</indexAllFields>
  </documentOptions>
</defaultSolrIndexConfiguration>

19. Custom Content Databases

Custom Sitecore databases now require a Blob Storage setting and you have to append Blob Storage for all the custom databases you are using, in a similar manner as that is done for those coming out of the box.

20. Upgrading xDB and analytics

After the introduction of xConnect, there was a question on what to do with all existing analytics data from MongoDB upgrading it to 9.X or 10.X. To address that Sitecore created a tool called the xDB migration tool which works on top of Data Exchange Framework reads from MongoDB, and writes to the xConnect server.

Keep in mind if you have years of data on production, you can be looking at gigabytes of data migration. Be sure to take the appropriate precautions, like preventing regular app pool recycles, as the process can take days. If possible, discuss trimming the MongoDB data, or not importing it at all if it's not used. This tool uses a custom collection model to be deployed to both xConnect service and the xConnect indexer service. After rebuilding the xDB search index you will get the data in the Experience Profile.
It has an optional Verification feature, which is in fact a standalone database that gets a record of each entity being submitted to xConnect.


Changes in Sitecore 10.X

1. Containers

Containers are immutable which means that any changes to the file system within the container will be lost as soon as the container restarts

When switching to containers your codebase will remain the same, however, there are minor changes - debugging now works a little bit different: now we need to choose a relevant container and then pick up a process within it. instead of publishing artifacts into webfolder we now build images and run a container from it.

With Docker, you no longer deploy just your application code. A container is now the unit of deployment, which includes the whole environment: OS and application dependencies. Therefore, your build process will have to be extended to build Docker containers and push them to the container registry.

Both Azure and AWS are suitable for running Sitecore in containers. They both provide managed Kubernetes services and other container hosting options.

Besides containers you need to consider other system components: SQL, Solr, and optionally Redis

Luckily, Sitecore comes with health check endpoints (/healthz/live and /healthz/ready) out of the box, which you can use for this purpose.

Kubernetes on its own will impose quite a steep learning curve.

You can learn more tips on migrating your Sitecore solution to Docker.

2. Upgrade database

Databases were not 100% compatible between the versions. Previously (before 10.1) one had to run an upgrade script against Core and Master in order to attach both to a vanilla target instance and progress with the rest of the upgrade.

Update script ensured the schema and default OOB content get updated by applying all the intermediate changes between both versions. This article explains in more detail how did we upgrade databases before 10.1.

The idea that came to consideration was that supplying empty databases with a vanilla platform would eliminate the above need for everyone who updates DB to operate at an SQL admin level. But every version has its own unique set of default OOB items, where do we keep them?

Because of container-first the way of thinking, there was a clear need to store those somewhere at a filesystem level, so it was a matter of choosing and adopting a suitable data provider. Protobuf from Google was a perfect choice ticking all the boxes, moreover, it is a very mature technology.


3. Items as Resources

With that in mind, now having a database full of content you can upgrade the version without even touching it. SQL Databases stay up to date, and the rest of the default content gets updated by just substituting Protobuf Data resource files.

Sitecore called this approach Items as Resources.

I would strongly recommend you look into the items as a resources plugin for the Sitecore CLI - instead of going through all the trouble of making asset images, you can create Protobuf files that contain all your items and bake them directly into your container image.

I wrote a comprehensive explanation about everything you wanted to ask about "Items-as-Resources" coming with the new Sitecore 10.1 - hope it answers most if not all questions.

You can create your own resource files from *.item serialization using Sitecore CLI Items as Resources plugin (version of CLI 4.0 or newer). You cannot delete items from Resource File it is read-only, but this trick (by Jeroen De Groot) helps you "remove" items at the Provider level.


4. Sitecore UpdateApp

But how do we upgrade let’s say 8.2 databases to 10.2?

From 10.1 and onwards databases come with no content – it becomes a matter of removing the default items from the SQL database, leaving the rest of the content untouched. That default content would come from the actual items resource file which will be placed in App_Data\Items folder.

For Upgrading to Sitecore 10.1 with UpdateApp Tool it comes to purely replacing resource files which naturally fits the container filesystem way of doing business.

That is where a new tool comes into play, please welcome Sitecore UpdateApp Tool!

This tool updates the Core, Master, and Web databases of the Sitecore Experience Platform. You must download and use the version of the tool that is appropriate for the version and topology of the Sitecore Experience Platform that you are upgrading from.

It also works with official modules resource files for upgrading Core, Master, and Web. (SXA, SPE, DEF, Horizon, both CRMs, etc.)

Sitecore UpdateApp Tool 1.2.0

5. Asset images

To containerize modules we now use Sitecore Asset images, instead of packages as we did before. But why?

First of all, you won’t be able to install a package that drops DLL due to the instance \bin folder being locked for an IIS user.

Secondly, containers are immutable and are assumed to be killed and instantiated - any changes to a container will go away with it.

Also, since a module is a logical unit of functionality with its own lifecycle and assets - it should be treated by the "works together, ships together" principle. Also, you as a package maintainer provide sample usage Dockerfiles for the required roles so that your users could pick it up with ease.

Sitecore Asset images are in fact storage of your module’s assets in the relevant folders, based on the smallest windows image - nanoserver. You can create those manually, as I show on this diagram.

With the recent platform, there are two ways of managing the items: either creating an Items Resource file or converting the package zip file into a WebDeploy package, then extracting its DACPAC file and placing it into the DB folder. In either case, the file structure will stay as pictured in the diagram above.


There is a tool called Docker Asset Image for a given Sitecore module (by Robbert Hock) which aims to automate Asset Image creation for you.

I would recommend going through these materials for better understanding:

Testing and going live

You will definitely need to conduct one (or if things go wrong - a series of) Regression Testing. The first question comes to clarify, what exactly to test?

Depending on the upgrade scope, you will need to at least ensure the following:

  • Overall site(s) health look & feel
  • 3-rd party integrations aren’t broken
  • Editing experience is not broken
  • Most recent tasks/fixes function well

Another question: manual, automated, or a mix of both?

There is no exact answer - but without automation regression testing becomes effort consuming. especially when you need to repeat it again and again. At the same time, it does not make sense to automate everything (and sometimes is not even possible).


Since you do the upgrade into a new instance of Sitecore in parallel to the existing one, you definitely need to conduct Load Testing prior to this new instance becoming live. The metrics must not mandatory me better than previously had as new Sitecore releases have much of new features and the number of assemblies within \bin folder always grows up. But as soon as it meets the projected SLA - that should be fine.


Monitoring is also crucial when going live. Once the new Sitecore instance is up and running, the first thing to check would be Sitecore logs to inspect and fix any errors that are being logged. We have a comprehensive set of automated UI tests that cover the majority of business use cases and are executed overnight. Watch the metrics for anomalies


Another exercise to undertake before going live would be Security Hardening - the process of securing a system by reducing its surface of vulnerability, which is high for systems that perform more functions like Sitecore is. From older versions, Sitecore comes up with the Security hardening guide and considerations to be followed.


In case you have the luxury of organizing Content Freeze for the going live period - that is great. If not you will need to pick up and care for the content delta produced meanwhile going live. The best thing to advise would be to use PowerShell Extensions to identify and pack all the content produced at a specific timestamp onwards. After installing this package the delta would be applied to the updated instance.

The actual moment of Going Live is when you switch DNS records from a previous instance to the new one. Once done, the traffic will go and be served by the updated instance. There are a few considerations, however:

  • you will need to reduce TTL values for domain DNS record well in advance to make the switch immediate; then bring it back once done.
  • you may consider slightly increasing traffic (ie. 5% of visitors) to new instances and monitor logs rather than switch it entirely.

If everything went well and the updated instance works fine, you still need to keep both instances running in parallel for a certain amount of time, Maturing Period. If something goes totally wrong you could still revert to an old instance and address the issues.

That's it!

Hope my presentation helps your future Sitecore upgrades!
Finally, I want to thank all the event organizers for making it happen and giving me the opportunity to share the experience with you.

Converting Sitecore back-end developer skills for a rapid kickstart with JSS & Next.js

Developers are crucial for Sitecore ecosystem!

With a count of several tens of thousands globally, only less than 15% of Sitecore developers feel confident with modern front-end tools. Resolving this bottleneck is very important as that slows down adoption of new generation of headless approaches: JSS and Next.js. This session is to show the quickest but still effective path for converting typical existing BE skills into the new development paradigm.

I have prepared a session for Symposium 2021 and below is my paper submission proposal. Symposium speech gets backed with a several blog posts and "how-to" videos that highlight the whole path for typical Sitecore developers with minimal knowledge of JS to the state of competency with Next.js

Update: sadly, this proposal was not chosen, leaving me frustrated, as the described topic is very sensitive to the most of us - developers and solution architects. Therefore, I am leaving my submission for historical purposes below.

The proposal

With much already said about the advantages of Next.JS with Jamstack fast delivery by pre-rendering, we won't focus on that as it's well-documented.

Instead, session will mostly cover converting a typical developers' experience from purely back-end skills to the state of confidence enough to start building own Next.js solutions. Outside of it there's almost nothing that describes that the actual learning curve which raise high level of frustration: the gap is too big to fill without knowing a shortcut to success.

The session gets based on my own experience: being such a typical back-end person, I carefully documenting all the way down to the wonderful world of headless Jamstack: necessary steps and traps that may not be obvious for the target audience get explained.

The mission is simplifying switching to a new gen. of development for as much of typical XP developers as possible by:
  • explaining the bare minimum of skills to obtain to be able using Next.js with Sitecore
  • explaining how to set up the necessary toolset, solution, dependencies and the fastest approach for getting the knowledge
  • briefly focusing on container environments being a part of overall experience
  • doing all the above with the minimum efforts possible
  • assuming the audience gets on a self-learning path after overcoming initial "studying gravity" with materials from this session

Speech Agenda

1. JavaScript
  • most important changes since years jQuery ruled the front-end world
  • starting with React: important basis to be used
  • all you need to know about TypeScript using it with Next.js Sitecore solutions
  • unobvious traps of front-end world to avoid

2. JSS
  • mapping the terminology of old dev experience to newer counterparts
  • explaining and troubleshooting GraphQL and layout service
  • JSS Styleguide, DOs and DON'Ts

3. Containers
  • brief introduction for those never had experience containers approach
  • Next.JS starter template
  • development considerations

4. Next.JS
  • understanding pre-rendering options: static generation vs server-side rendering vs incremental static generation
  • managing dynamic content with ISG
  • routing / dynamic routes
  • components rehydration
  • client-side personalization via callback to origin

5. Development Experience
  • understanding solution structure
  • organize CSS on component level
  • debugging and troubleshooting

6. Deployment and Going Live
  • brief architectural overview
  • is self-hosting the best option for your solution?
  • hosting at Vercel
  • Sitecore Experience Edge

7. Demo time covering some of Next.js features:
  • image optimization
  • error handling
  • unusual API routes

8. Conclusion
  • FAQs
  • take-away materials
  • further learning plan


Takeaway materials

By the time of the event, I am going to produce the following materials covering my presentation:

  • A series of blog posts covering a topic much wider
  • GitHub repo with a guidance and codebase from demo
  • A series of short YouTube videos for each use case

Hopefully, once my submission get selected for either SUGCON or next Symposium.

Upgrading Sitecore like a Pro

Sooner or later, you'll face it - the need for Sitecore version upgrade. However, each Sitecore instance is unique, there is no universal method for an upgrade, therefore each upgrade path is bespoke.

In the past 10 years working with Sitecore I have done numerous upgrades, resulted in Upgrade Planning Strategy and a set of Upgrade Tips and Tricks. These tips will help saving 2-3 times (!) on spent efforts, comparing to direct approaches.
Last but not the least. versions 10 and 10.1 brought new challenges and options of migrating existing solutions into containers. I will explain your options and propose best approach on this as well.

I have lodged the speech proposal for SUGCON 2021 so fingers crossed for me to be chosen. Once happened, I will share the whole presentation and sides updating this blog post.

There is still no "silver bullet" for Sitecore upgrades, but following tips from my proposed session will eliminate risks, reduce efforts, and bring you confidence while upgrading your instances.

If being chosen, I will tell you about:

  1. How to approach the whole instance upgrade
  2. What are the upgrade time-wasters
  3. Common and potential traps to avoid
  4. Dealing with configuration upgrade, and why that's not as complicated as initially seems to be
  5. Upgrading your solution codebase, including ORM (Glass etc.) and DI
  6. Employing automations with PowerShell
  7. Upgrading databases and how to approach
  8. Even more automation to add
  9. How to treat deprecated Sitecore APIs and obsolete code
  10. Migrating forms from WFFM
  11. Upgrading to version 10 containers and how 10.1 changes the upgrades
  12. Testing your upgrade strategies
  13. Going live
  14. Summary of tips and findings from this session
Stay tuned!

Updating existing presentation details of base template's standard values after it has been set on derived templates

Problem: let's imagine the situation when you may have some complex template inheritance. You would like to set presentation details for each of these templates' standard values, somehow like below:

A - define layout, and header/footer common holders that will be shared for all sites (we've got a multisite setup for now)

B - define footer and header on a site level so that they remain the same within each of implemented sites 

C - define base page template that will add the rest of presentation shared between the pages (but not homepage)

Once you set presentation for standard values of A, you can go to standard values of B and see these changes, so that you add only B-specific components; the same exact story will be when you will go next to std. values of C an so on. So far, so good.

The problem occurs when you want to modify presentation coming with a base template after a derived presentation is already set. In that case, you will not see any difference, that may be seen a weird for a while. In our example, imagine you've set presentation for std. values of all three templates - A, B and then C, and then decided to add one more component to a presentation on A template (or change datasource item of existing etc.). You do changes for std. values of A, save it and as you see - these changes come into a play for template A, however, once you open B or C  they won't be there...

Explanation: let's think what in fact Standard Values are - just the default values for each of field defined in that (or parent) template. In the second case if a field has standard values for both parent and inherited template - it simply overrides parent value with inherited child's value. But, wait for a second - presentation is also stored in fields of Standard Template that all pages inherit from, how that makes possible, does it simply override?

No, for such particular cases when presentation fields are involved - override behaviour would not work at all. Let's look at our example - template A defines layout and header/footer sublayout and all that goes within __Renderings field - it's where (shared, not versioned) presentation is stored in XML-serialised format. But then, it would be overridden by setting concrete footer with no layout. Since it is the same field - it will lose layout at template B level and entire behaviour does not make any sense. To address this issue Sitecore implements a feature called Layout Deltas - so that presentation fields are not stupidly overwritten. Instead, after we defined default presentation for template A, it goes as is, as A does not have any base template with presentation set. But when setting presentation for B - it will only save the difference between itself and presentation of base template (if exists). When page is being rendered, Sitecore is wise enough to construct resulting page presentation from all the base templates only adding deltas with each derived template. That is how Layout Deltas work.

One may create multilevel presentation inheritance of standard values, appending more and more presentation on each of derived levels. However, when we want to adjust the presentation of base template (A) of current template (B) - changes will be affected only for A, but not B or C if they already have layout deltas defined. That behaviour raises questions without a doubt.

Solution: what we need to do to in order to ensure changing presentation details for A will be affected for all derived templates' items is to recourse inheritance tree of A and re-calculate layout delta for each of them with recent updates from A. In order to get this done I have re-worked a solution suggested by ... The difference is that since that time we now got a feature called Versioned Layouts, so that we need to operate both fields - __Renderings and __Final Renderings correspondingly. Apart from that I have tested it for a while and fixed few of stability issues.

Implementation: when we change presentation - we change the field, so eventually the holding item is being updated. In order to intercept this we add a pipeline processor for item:saving event:

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
  <sitecore>
    <events>
      <event name="item:saving">
        <handler type="Sitecore.Foundation.Presentation.Services.LayoutInheritance, Sitecore.Foundation.Presentation" method="OnItemSaving"/>
      </event>
    </events>
  </sitecore>
</configuration>

And the code. I am using XmlDeltas.GetDelta() and XmlDeltas.ApplyDelta() static classes of Sitecore.Data.Fields namespace to make this work.

using System;
using Sitecore;
using Sitecore.Data;
using Sitecore.Data.Events;
using Sitecore.Data.Fields;
using Sitecore.Data.Items;
using Sitecore.Data.Managers;
using Sitecore.Data.Templates;
using Sitecore.Diagnostics;
using Sitecore.Events;
using Sitecore.Globalization;
using Sitecore.SecurityModel;

namespace Sitecore.Foundation.Presentation.Services
{
    public class LayoutInheritance
    {
        public void OnItemSaving(object sender, EventArgs args)
        {
            Item item = Event.ExtractParameter(args, 0) as Item;
            PropagateLayoutChanges(item);
        }

        private void PropagateLayoutChanges(Item item)
        {
            if (StandardValuesManager.IsStandardValuesHolder(item))
            {
                Item oldItem = item.Database.GetItem(item.ID, item.Language, item.Version);

                PropagateLayoutChangesForField(item, oldItem, FieldIDs.LayoutField);
                PropagateLayoutChangesForField(item, oldItem, FieldIDs.FinalLayoutField);
            }
        }

        private void PropagateLayoutChangesForField(Item item, Item oldItem, ID layoutField)
        {
            string layout = item[layoutField];
            string oldLayout = oldItem[layoutField];

            if (layout != oldLayout)
            {
                string delta = XmlDeltas.GetDelta(layout, oldLayout);
                foreach (Template templ in TemplateManager.GetTemplate(item).GetDescendants())
                {
                    ApplyDeltaToStandardValues(templ, layoutField, delta, item.Language, item.Version, item.Database);
                }
            }
        }

        private void ApplyDeltaToStandardValues(Template template, ID layoutField, string delta, Language language, Sitecore.Data.Version version, Database database)
        {
            if (template?.StandardValueHolderId != (ID)null)
            {
                try
                {
                    Item item = ItemManager.GetItem(template.StandardValueHolderId, language, version, database, SecurityCheck.Disable);

                    if (item == null)
                    {
                        Log.Warn($"Foundation.Presentation: Item is null {template.StandardValueHolderId} in database {database.Name}", template);
                        return;
                    }

                    Field field = item.Fields[layoutField];

                    if (field == null)
                    {
                        Log.Warn($"Foundation.Presentation: Field is null in item {item.ID} in database database.Name", item);
                        return;
                    }

                    if (!field.ContainsStandardValue)
                    {
                        string newFieldValue = XmlDeltas.ApplyDelta(field.Value, delta);
                        if (newFieldValue != field.Value)
                        {
                            using (new EditContext(item))
                            {
                                LayoutField.SetFieldValue(field, newFieldValue);
                            }
                        }
                    }
                }
                catch (Exception e)
                {
                    Log.Info($"Foundation.Presentation: Exception {e.Message}", e);
                    throw;
                }
            }
        }
    }
}
As I am using Helix in my development, I created a foundation module Presentation and placed the above code and config into it correspondingly. 

Hope this helps someone!

Migrating existing code to Helix. Fixing invalid dynamic placeholders

Recently I have inherited a project that utilised dynamic placeholder in a weird way:

public static HtmlString DynamicPlaceholder(this SitecoreHelper helper, string placeholderKey)
{
    if (string.IsNullOrEmpty(RenderingContext.Current.Rendering?.DataSource))
        return helper.Placeholder(placeholderKey);

    var currentRenderingId = Guid.Parse(RenderingContext.Current.Rendering.DataSource);
    return helper.Placeholder($"{placeholderKey}_{currentRenderingId}");
}

instead of more commonly used way suggested by Jason Bert:

namespace DynamicPlaceholders.Mvc.Extensions
{
	public static class SitecoreHelperExtensions
	{
		public static HtmlString DynamicPlaceholder(this SitecoreHelper helper, string placeholderName)
		{
			string text = PlaceholdersContext.Add(placeholderName, RenderingContext.Current.Rendering.UniqueId);
			return helper.Placeholder(text);
		}
	}
}

In two words, the first approach generates placeholder name using an ID of datasource item. The second approach is more traditional and is default for Helix. It was suggested by Jason Bert and also utilised by sitecore - that relies on rendering's UniqueID. In both cases, dynamic placeholders look like: /content/name-of-placeholder_017c3643-0fef-475c-95d2-bb1107beb664.

So I need to update it across entire solution. Let's iterate our items.

First of all, I do not need to go through all of item, but only those that are pages and have presentation configured, as I am going to adjust it. Secondly, those items are located under /sitecore/content folder. As many of you should know, presentation details (or deltas) for a page are kept within two fields of Standard Template - Renderings and Final Renderings that correspond to configuration you see at Shared Layout and Final Layout tabs. These thoughts resulted in using following Sitecore query to identify those items:

/sitecore/content//*[@__Renderings != '' or @__Final Renderings != '']

So far, so good. Also need to mention that I will perform the operation for master database only and for Default device:

private const string DatabaseName = "master";
private const string DefaultDeviceId = "{FE5D7FDF-89C0-4D99-9AA3-B5FBD009C9F3}";

I am using Sitecore presentation API in order to achieve my goal. Below is my Iterate() method:

public Dictionary<Item, List<KeyValuePair<string, string>>> Iterate()
{
    var result = new Dictionary<Item, List<KeyValuePair<string, string>>>();

    var master = Factory.GetDatabase(DatabaseName);
    var items = master.SelectItems(ItemsWithPresentationDetailsQuery);

    var layoutFields = new[] {FieldIDs.LayoutField, FieldIDs.FinalLayoutField};

    foreach (var item in items)
    {
        foreach (var layoutField in layoutFields)
        {
            var changeResult = ChangeLayoutFieldForItem(item, item.Fields[layoutField]);

            if (changeResult.Any())
            {
                if (!result.ContainsKey(item))
                {
                    result.Add(item, changeResult);
                }
                else
                {
                    result[item].AddRange(changeResult);
                }
            }
        }
    }

    return result;
}

That method iterates through each item from returned from the query and calls ChangeLayoutFieldForItem for that item twice - for both presentation fields Renderings and Final Renderings.

private List<KeyValuePair<string, string>> ChangeLayoutFieldForItem(Item currentItem, Field field)
{
    var result = new List<KeyValuePair<string, string>>();

    string xml = LayoutField.GetFieldValue(field);

    if (!string.IsNullOrWhiteSpace(xml))
    {
        LayoutDefinition details = LayoutDefinition.Parse(xml);

        var device = details.GetDevice(DefaultDeviceId);
        DeviceItem deviceItem = currentItem.Database.Resources.Devices["Default"];

        RenderingReference[] renderings = currentItem.Visualization.GetRenderings(deviceItem, false);

        var datasourceGuidsToRenderingUniqueIdMap = renderings
            .Where(r => !string.IsNullOrWhiteSpace(r.Settings.DataSource))
            .Select(r => new KeyValuePair<string, string>(Guid.Parse(r.Settings.DataSource).ToString(), r.UniqueId));

        if (device?.Renderings != null)
        {
            foreach (RenderingDefinition rendering in device.Renderings)
            {
                if (!string.IsNullOrWhiteSpace(rendering.Placeholder))
                {
                    var verifiedPlaceholderKey = FixPlaceholderKey(rendering.Placeholder, datasourceGuidsToRenderingUniqueIdMap);
                    result.Add(new KeyValuePair<string, string>(rendering.Placeholder, verifiedPlaceholderKey));
                    rendering.Placeholder = verifiedPlaceholderKey;
                }
            }

            string newXml = details.ToXml();

            using (new EditContext(currentItem))
            {
                LayoutField.SetFieldValue(field, newXml);
            }
        }
    }

    return result;
}

Further down it creates a list of matches for all the renderings within current presentation field of that particular item - datasource GUIDs matching to unique rendering IDs to go through it and do a replacement - that is done in FixPlaceholderKey method. Once everything is replaced, save the field by calling LayoutField.SetFieldValue() of course wrapping that call with EditContext as we are modifying item's field value.

FixPlaceholderKey is a simple method that just does case insensitive replacement by using Regex:

private string FixPlaceholderKey(string renderingInstancePlaceholder, IEnumerable<KeyValuePair<string, string>> map)
{
    var value = renderingInstancePlaceholder;

    foreach (var oldValue in map)
    {
        value = Regex.Replace(value, oldValue.Key, Guid.Parse(oldValue.Value).ToString(), RegexOptions.IgnoreCase);
    }

    return value;
}

That is pretty everything about the dynamic placeholder fixing logic.

But as for my case - I was aware that I'll need to run this dynamic placeholder replacements few more times in future and will likely need to implement some other similar small tools. So I decided that it would be great to place it under /sitecore/admin folder along with other admin tools - exact location by purpose! And since we're now going Helix, I decided to follow good principles and created a foundation module called AdminTools, where I will be adding similar admin folder tools. So here's how it looks for me in the Solution Explorer:


FixDynamicPlaceholders.aspx is a classical ASP.NET WebForm with one line markup (ah-h, I was so lucky not to deal with Webforms for couple past years till the moment)

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="FixDynamicPlaceholders.aspx.cs" Inherits="HomeServeUsa.Foundation.AdminTools.sitecore.admin.Maintenance.FixDynamicPlaceholders" %>

and the codebehind. Since FixDynamicPlaceholders.aspx is admin tool page - codebehind is inherited from Sitecore.sitecore.admin.AdminPage:

public partial class FixDynamicPlaceholders : AdminPage
{
    protected void Page_Load(object sender, EventArgs e)
    {
        var fixRenderings = new DynamicPlaceholdersModifyService();
        var result = fixRenderings.Iterate();

        OutputResult(result);
    }

    private void OutputResult(Dictionary<Item, List<KeyValuePair<string, string>>> result)
    {
        Response.ContentType = "text/html";

        Response.Write($"<h1>{result.Count} items processed</h1>");
        foreach (var pair in result)
        {
            Response.Write($"<h3>{pair.Key.Paths.FullPath}</h3>");

            foreach (var kvp in pair.Value)
            {
                if (kvp.Key != kvp.Value)
                {
                    Response.Write($"<div>{kvp.Key} ==> {kvp.Value}</div>");
                }
            }
        }
    }
}

Finally, it is complete. Hope this helps someone!

P.S. of course, that would be easier (and more elegant) to perform with Sitecore PowerShell extension. But unfortunately I am the only person in the organisation who uses it regardless of my promotional activity.

Sitecore Technical User Group UK - January 2016 - Download presentation

Yesterday I have attended Sitecore Technical User Group in London and would like to share presentations from it. There were 3 presenters, talking about:

1. "CDN with Sitecore" presented by Kamruz Jaman, a Sitecore MVP 2013-2015 and in general one of the most experiences Sitecore architect, who seems to have an answer to any question (StackOverflow). He reviewed know approaches of implementing CDN into a solution in order to reduce bandwidth and improve you servers performance, and demonstrated the most complex - integrating Azure into Sitecore with media items being directly uploaded to the cloud storage,
Download presentation (521.5KB)

2. "Continuous Integration & Delivery for Sitecore" by Jason Bert.
Jason seems to know everything about CI with Sitecore. He stated, that is absolutely doable to achieve a working CI setup with Sitecore, GIT, TeamCity and Octopus Deploy and showed us how-to. Less theory in favor of a practical demo in real-time!
Download presentation (4.4MB)

3. "Sitecore 8.1: new Features and Improvements" presented by Steve McGill.
Steve guided us through serious list of new features and improvements in Sitecore 8.1 - there are so many sweeties I am anticipationg to start working with! At the end of presentation he also discussed Sitecore Habitat - a project bringing modular approach into Sitecore.
Download presentation (2.3MB)

Steve McGill and Kamruz Jaman:


Light of knowledge is coming out of Jason Bert


Thanks to presenters and everyone who has attended!

Productivity Improvement: Device Editor showing datasource and previewing that right from a pop-up click

After previous post on Layout Details dialog improvements, I decided to look even further and implement one more improvement that came into my head.

Another dialog window, probably most important in Content Editor is missing couple things I just decided to fix. That is a case when a screenshot is better than hundred words, so here is it:


What has been added is a datasource item path, immediately underneath rendering and placeholder. It is clickable in the same manner as from previous posts, immediately opening that (datasource) item for view and edit right in the popup window, that saves so much time!

Additionally, rendering / sublayout name became also clickable with the same item preview popup effect.


Download: please get the package and anti-package to revert changes. Source code is available at GitHub page by this link.

Known minor issue: if open Control Properties dialog from Device Editor, and when you return back from that dialog - look-up links will not work and control will return to default behavior, so you may need to re-open Device Editor again.
The reason for such a behavior is that on returning back, Sitecore runs a series of pipelines that eventually call original class from Sitecore.Client rather than the one we have overridden and referenced above. Fixing that requires patching original DLL and I highly wanted to avoid inclining into any original functionality (moreover, you are not likely allowed to do that by license)

Version Layouts - what is that for?

Every good Sitecore developer is familiar with the principle of versioned and shared fields - an item can contain a mixture of both types. People often wrongly consider items are versioned (for simplicity), however in fact that is not quite true. Fields may have versions, not items. When user creates a new version of item in fact only new versions of versioned fields are created, and not the shared. This concept aims to address effectiveness of data storage and it works well and transparent to users. So far, so good.

Previously, we did not have similar feature for Presentation details - it was always shared. Saying Presentation I mean layout details for devices, renderings, placeholders etc. - everything that stored in Renderings field of Layout section of Standard Template every page should inherit from.


Final presentation for an item was merged from whatever stored in Standard values (for item's template) with Layout Delta (changes individually applied to item - a delta from those in Standard Values). Standard values were stored at Rendering field of Standard Values item for a template of that specific item, thus it was the chared for all the items of that particular template; Layout Delta was physically stored in Renderings (shared) field, but of that specific item, thus applied individually to that item.

Since Sitecore 8 - there is one more point for consideration - a version. Standard Template has been extended with one new field called Final Renderings, and what is principally important at that moment - Final Renderings is a versioned field. The point of introducing that new field was that now it becomes possible to have different presentation of a page for different languages and various versions. Also, now it is possible to have a specific version of page's presentation and have it published for a while (without creating duplicates and renaming the in real-time) - now all the information is stored withing same page item.


So, before, while building the presentation Sitecore had to consider only two Renderings fields - one from item's template's Standard values, another from item itself. As both parts were shared - resulting presentation was always the same across languages and versions. Previously if you wanted a page to look different for different laguages - you were limited to personalizing renderings by specifying languages and renderings in Rules Engine. No more personalisaion required since now!

At the moment we have following configuration, in that particular order on how final presentatin is calculated:

  1. Standard Values - Renderinds field - shared.
  2. Standard Values - Final Renderinds field - versioned.
  3. Item - Renderinds field - shared.
  4. Item - Final Renderinds field - versioned.

Here's the diagram (from official site) on how final presentation is calculated with the above bunch of sources of presentation:


Thus, important advice - do not use them four at the same time, as it may behave unpredictable. Before Sitecore 8 most popular combitaion user for presentation calculation was (1)-(3), less often just (1) or (3) solely by themselves. Now you may safely use combitations of (1)-(2)-(4) or (1)-(3)-(4), as they were tested and proved with time. Of course, previous (1)-(3) from migrated projects will work as before, until you modify and save it with Experince Editor. Experince Editor works with Final Renderings field and not with shared Renderings field.

These above were just brief notes about Versioned Layouts, I left that in blog mostly for myself to have quick access to. But if you may want to understand how Versioned Layouts work in details,I would highly recommend to read a series of articles called Version Layout - Mixed Feelings (part 1, part2 and part 3).

See also Sitecore item's extension methods for identifying version layouts (link).

Hope you find this post helpful!

Tip: copying Presentation Details manually

I came across a question on StackOverflow where a guy asked about copying presentation details and decided to share this quick tip. Saying Presentation Details I mean all the information about layouts, renderings, placeholders etc., so whatever you usually configure on that screen:



So, you usual data is stored within item's fields, but where does presentation live? Well, presentation is also kept within item, but in a slightly different location.

You page template is inherited from Standard Template, it has plenty of important fields and sections, among which there is Layout section. Let's go and see what is there. But before, open View tab ensure Standard fields is checked in order to display all sections provided by Standard Template and also check Raw values option to display actual content of the fields:


Then, scroll down to Layouts section and expand it.


Rendering field contains all presentation details, serialized into XML. So now, if you copy them 'as-is' to clipboard and insert to another item - that item will immediately same layout, all renderings in the same order, placeholders etc. You may also copy that across environments, assuming both target and source environment have those layout and renderings.


Note: if you need to copy Presentation Details just within same database, there a nice and quick solution right from the UI:


Hope this helps!