Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?
Experience Sitecore! | All posts tagged 'Docker'

Experience Sitecore!

Martin Miles on Sitecore

Why CNAB is a game changer for Docker containers and how Sitecore can benefit from that?

Introduction

Currently in order run Sitecore in docker locally, one has to pull the code from GitHub repository, build it (if not done yet, or pull already built images from a docker registry), set the license file and say "up" to docker-compose. Not to say the prerequisites required. If dealing with cloud, then the deployment into Kubernetes is required, that also demands adequate skills.

Imagine an ideal situation where you don't need to do all the things, but just pull a "managing" image from remote registry, and this image itself will care about running "all the things" internally, such as prerequisites, pulling the dependent images, preparing the environments, network and dependencies, doing an alternative steps to docker-compose and much more.


Or going far beyond that: shipping is as a traditional GUI installer for non-technical people or deploying that same image into a  could, any cloud, ready to use, does not that look as such desirable? What if I tell you this technology is already available and you can use it? Please welcome the new universal spec for packaging distributed apps created by by Microsoft and Docker:

Cloud Native Application Bundles or simply CNAB

Firstly, what the heck is CNAB at all? Using Cloud Native Application Bundle, a developer has an option of deploying the app either locally at his dev. machine, or in a public cloud as it bolts together containers and services into a seamless whole, placing a strong focus on standardization. It is an open source specification that aims to facilitate the bundling, installing, and managing of containerized apps. With this bundle, users can define resources that can then be deployed to a number of run-time environments, such as Docker, Azure, Kubernetes, Helm, automation services (such as those used by GitOps), and more.

At a first glance, that task supposed to be solved by Docker itself. However, when dealing with largely scaled hybrid infrastructures, its standard features become insufficient. Thus, CNAB is an attempt of standardizing  the process of packaging, deployment and lifecycle management of distributed apps on a base of Kubernetes, Helm, Swarm and others by using unified JSON-based package format. 

Recently the CNAB spec reached version 1.0, which means it is ready for production deployment. The spec itself is now broken down into several chapters:

  • CNAB explains the fundamentals of the CNAB core 1.0.
  • CNAB Registry will describe how CNAB bundles can be stored inside of OCI Registries (this section is not yet complete).
  • CNAB Security explains the mechanisms for signing, verifying, and attesting CNAB packages.
  • CNAB Claims describes the CNAB Claims system, shows how records of CNAB installations formatted for storage
  • CNAB Dependencies describes how bundles can define dependencies on other bundles.

Tooling

Each of organisations has provided own tools that demostraties CNAP capabilities: Microsoft released Duffle, while Docker shipped Docker app. Docker Desktop application is fully compatible with CNAB from May 2019.

CNAB is not the only one solution for managing cloud applications lifecycle. For example Kubernetes has Сrossplane manager as well as package manager Helm. However CNAB is the first ever solution that supports several most popular tools and is platform-agnostic. By the way, CNAB can also work with Helm and I came across sample of it at GitHub.

Duffle is a simple command line tool that interacts with Cloud-Native Application Bundles - helping you package and unpackage distributed apps for deployment on whatever cloud platforms and services you use. Its goal is to exercise all parts of the specification and this tool also comes with very handy VS Code extensions, on  of which named Duffle Coat allows you creating native executable installer (*.exe) of your bundle:


This results with the proper installer that will install and configure you Sitecore 9.3 locally from bundle image stored at docker registry:



Once again, instead of local Sitecore installations (like SIA does) we are having this CNAB installer that installs you the platform from Docker registry and with no prerequisites required at all! And CNAB bundle cares about all the dependencies and parameters. What a magic!

Another tool,Porter, is Microsoft’s CNAB builder that gives you building blocks to create a cloud installer for your application, handling all the necessary infrastructure and configuration setup. It is a declarative authoring experience that lets you focus on what you know best: your application. The power of Porter is coming from using mixins giving CNAB authors smart components that understand how to adapt existing systems, such as Helm, Terraform or Azure, into CNAB actions.


And of course, Docker App is CNAB builder and installer that leverages the Docker Compose format to define your applications. To facilitate developer to operator handover, you can add metadata and run-time parameters. These applications can easily be shared using existing container registries. Docker App is available as part of the latest Docker release.

Bundle manifest file
As I said above, the specification defines the way of packaging distributed application of various formats. CNAB includes package definition (named bundle.json) used for describing an app, as well as a special image (also called invocation image) for its installation.A bundle.json is similar to a docker-compose.yml file in that it describes a complex configuration for image deployment. The difference is, the CNAB bundle is very clearly defined as to how it should be laid out, encoded, and where all associated files must reside. It contains:

  • The schema version.
  • Top-level package information.
  • Information on invocation images.
  • Map of images.
  • Specification for parameter override (with a reference to a validation schema).
  • List of credentials.
  • Optional description of custom actions.
  • A list of outputs produced by the application.
  • A set of schema definitions used to validate input.
Here is a sample of bundle.json file below:
{ 
   "credentials":{ 
      "hostkey":{ 
         "env":"HOST_KEY",
         "path":"/etc/hostkey.txt"
      }
   },
   "custom":{ 
      "com.example.backup-preferences":{ 
         "frequency":"daily"
      },
      "com.example.duffle-bag":{ 
         "icon":"https://example.com/icon.png",
         "iconType":"PNG"
      }
   },
   "definitions":{ 
      "http_port":{ 
         "default":80,
         "maximum":10240,
         "minimum":10,
         "type":"integer"
      },
      "port":{ 
         "maximum":65535,
         "minimum":1024,
         "type":"integer"
      },
      "string":{ 
         "type":"string"
      },
      "x509Certificate":{ 
         "contentEncoding":"base64",
         "contentMediaType":"application/x-x509-user-cert",
         "type":"string",
         "writeOnly":true
      }
   },
   "description":"An example 'thin' helloworld Cloud-Native Application Bundle",
   "images":{ 
      "my-microservice":{ 
         "contentDigest":"sha256:aaaaaaaaaaaa...",
         "description":"my microservice",
         "image":"example/microservice:1.2.3"
      }
   },
   "invocationImages":[ 
      { 
         "contentDigest":"sha256:aaaaaaa...",
         "image":"example/helloworld:0.1.0",
         "imageType":"docker"
      }
   ],
   "maintainers":[ 
      { 
         "email":"matt.butcher@microsoft.com",
         "name":"Matt Butcher",
         "url":"https://example.com"
      }
   ],
   "name":"helloworld",
   "outputs":{ 
      "clientCert":{ 
         "definition":"x509Certificate",
         "path":"/cnab/app/outputs/clientCert"
      },
      "hostName":{ 
         "applyTo":[ 
            "install"
         ],
         "definition":"string",
         "description":"the hostname produced installing the bundle",
         "path":"/cnab/app/outputs/hostname"
      },
      "port":{ 
         "definition":"port",
         "path":"/cnab/app/outputs/port"
      }
   },
   "parameters":{ 
      "backend_port":{ 
         "definition":"http_port",
         "description":"The port that the back-end will listen on",
         "destination":{ 
            "env":"BACKEND_PORT"
         }
      }
   },
   "schemaVersion":"v1.0.0",
   "version":"0.1.2"
}
You may read more about bundle.json format at CNAB.io official page.

How about Azure?
For Azure we also have got the solution, Azure CNAB Quickstarts library. It demonstrates how once can use bundles for deploying applications and solutions and how to create their own bundles. The library is designed to be optimized for bundles that use Azure resources, but is not limited to Azure only. There is CI/CD workflow in the repository using custom GitHub Actions to enable automatic building of bundles and publishing of bundles to a public Azure Container Registry. The library supports bundles made using Porter tool I mentioned above, a tool capable of building, publishing, invoking, and updating bundles.

Final thoughts
Without any doubt, CNAB becomes a game changer for the 2020, as we getting more and more into contenerized deployments and orchestrating them in the clouds. The specification is quite new and not too many companies are using it at the moment, but there is ever growing interest to it. Since all the major vendors are now ready, I am quite sure it will boost the whole industry in the coming months!

References
Hope you find this post helpful and get your own hand on CNAB shortly!

Tip of the day: running Sitecore Docker from within Hyper-V virtual machine is in fact possible

Why not to run Docker containers from within a Hyper-V virtual machine? 

Well, when you are trying to install Docker Desktop for Windows, it installs correctly. But running it will prompt you about Hyper-V is required for Docker to work.


Programs and Features on its turn shows you that Hyper-V is greyed out, so you cannot install it from there



Not everyone knows, but Hyper-V in fact allows having nested virtual machines. That means running docker is also possible from within a VM. All you need is just running one PowerShell command from outside of your container:

Set-VMProcessor -VMName _YOUR_VM_NAME_TO_ENABLE_ -ExposeVirtualizationExtensions $true

What it does - just adds virtualization features into a virtual processor of outer VM, similar to those you got at you physical CPU. Thus you need to run this command of the VM switched off.

Once done, Sitecore perfectly builds and work if Docker running within a virtual machine. For those who are total beginner and scary to mess the things around their host machines - this could be an option. However there is of course a performance penalty from double virtualization.



Finally, the main trick! Just because you've added virtualization extensions into a virtual processor of your VM, you may run docker in process isolation mode, relatively to the build number of VM's operation system. Will explain it on the following example: imagine, your host machine runs Windows 10 build 1809, but got no problems of running Windows 10 build 1909 in a Hyper-V machine there. Now, you after you enable virtualization extensions to that virtual machine (that runs 1909), you will be able to install docker within that virtual machine, and run 1909-built images in process isolation, natively to that VM: 1909-built images run on 1909- machine! But at the same time you will be able to run those same images only in hyper-v mode on the (outer) host machine, because it has 1809 and cannot run non-matching images in process isolation mode.

Hope this helps!

Starting with Docker and Sitecore

There was much of recent buzz around containers as technology and Docker in particular, especially now where more and more community efforts focused at Docker in conjunction with Sitecore. Plenty of articles do explain how it works on the very top level, what the benefits are but very rare do precise guidance. As for an ultimate beginner - I know how it is important to get a quick start, the minimal positive experience as a further point of development. This blog exactly about how to achieve the bare minimal Sitecore running in Docker.

Content


1. Terminology

  • Docker images - a blueprint for creating containers, is what you pull from remote registry
  • Docker container - an implementation of a specific image, you run and work with containers, not images
  • Docker repository - a logical unit containing one or many built images (specified by tags)
  • Docker registry - works like a remote git repo, it's a hosted solution to you push your built images into

There are plenty more of terminology, but these are the essentials for a demo below

2. Installing Docker

If you have it up and running, you may skip to the next part.

In order to operate the repository for given walk-through, you need to have Windows 10 x64 with at least build 1809.

The simplest way is to install it from Chocolatey gallery:

cinst docker-desktop

Missing something from your host OS installation? Docker will manage that itself!

Once done, you'll need to switch a mode. Docker for Windows can work in either Windows or Linux mode at the same time - which means you cannot mix types of containers.

One of the biggest issues at the moment is the size of Windows base images - the minimal Nano Server with almost everything cut off is already 0.5Gb and Server Core (this one either does not have UI - just a console) goes to 4Gb. That's too much comparing to minimal Linux images starting from as little as 5 megs. That's why it may seem very attractive to run Solr from Linux image (as both Solr and Java it requires are cross-platform and there are ready-to-use images there), same for Ms SQL Server which also has been ported on Linux and its images are also available.

Until the very-very recent the short answer was no - one could manage only single type of container at a time (while already running Linux container will keep up running unmanageable, there are also few workarounds how to make them work in parallel, but that's out of scope for now), but as from April 2019 it is doable (from Linux mode on Windows), I managed to combine NGINX on Linux with IIS on Windows.

Switching mode from UI is done by right-clicking Docker icon context menu from a system tray:


3. Docker registry

What you should do next is to provide a docker registry. Docker Hub is probably the first option for any docker beginner.

Docker Hub however allows only one private repository for free. You need to ensure sure all your repositories are private. Images you're building will contain your license file, having them in a public access will also be treated as sharing Sitecore binaries on your own, which you're not allowed - only Sitecore can distribute that publicly.

Alternatively, you may consider Canister project, they give up to 20 private repositories for free.

Pluralsight has a course on how to implement your own self-hosted docker registry.

But what is even more surprising, Docker itself provides a docker image with Docker Registry for storing and distributing Docker images.


4. Preparing images

Now let's clone Sitecore images repository from GitHub - https://github.com/sitecoreops/sitecore-images

If you don't have git installed - use Chocolatey tool already familiar from previous steps: cinst git (once complete - you'll need to reopen console window so that PATH variable gets updated).

To keep things minimal, I go to sitecore-images\images folder and delete everything unwanted - as for this demo, I keep only 9.1.1 images, removing the rest. So, I left only 9.1.1 images and sitecore-openjdk required for Solr:

Images folder contains instructions on how to build you new Sitecore images. As input data they require Sitecore installers and license file, so put them into this folder:

And the last but not the least, create build.ps1 PowerShell script.

Important: do not use an email as the username, it's not your email (in my case username is martinmiles), and from what I've heard many people find this confusing and were wondering why getting some errors.

This is my build script, replace usernames and password and you're good to go:

"YOUR_DOCKER_REGISTRY_PASSWORD" | docker login --username martinmiles --password-stdin

# Load module
Import-Module (Join-Path $PSScriptRoot "\modules\SitecoreImageBuilder") -Force

# Build and push
SitecoreImageBuilder\Invoke-Build `
    -Path (Join-Path $PSScriptRoot "\images") `
    -InstallSourcePath "c:\Docker\Install\9.1.1" `
    -Registry "martinmiles" `
    -Tags "*" `
    -PushMode "WhenChanged"


5. Building images

Run the build script. If receiving security errors, you may also be required to change execution policy prior to running build:

set-executionpolicy unrestricted
Finally you get your base images downloading and build process working:

As I said, the script pulls all the base images and builds, as scripted. Please be patient as it may take a while. Once built, your images will be pushed to the registry. Here's what I finally got - 15 images built and pushed to Docker Hub. Again, please pay attention to Private badge next to each repository:

Docker Hub has a corresponding setting for defining privacy defaults:


6. Running containers

Images are built and pushed to registry, so we are OK to run them now. Navigate to tests\9.1.1 rev. 002459\ltsc2019 folder where you see two docker-compose files - one for XM topology and another for XP. If simplified, docker compose is a configuration for running multiple containers together defining common virtual infrastructure, it is written in YAML format.

Since we are going the simplest route - we keep with XM topology, but that same principle works well for anything else.

Rename docker-compose.xm.yml to docker-compose.yml and open it in the editor. What you see is a declarative YAML syntax of how containers will start and interact with each other.

version: '2.4'

services:

  sql:
    image: sitecore-xm1-sqldev:9.1.1-windowsservercore-ltsc2019
    volumes:
      - .\data\sql:C:\Data
    mem_limit: 2GB
    isolation: hyperv
    ports:
      - "44010:1433"

  solr:
    image: sitecore-xm1-solr:9.1.1-nanoserver-1809
    volumes:
      - .\data\solr:C:\Data
    mem_limit: 1GB
    isolation: hyperv
    ports:
      - "44011:8983"

  cd:
    image: sitecore-xm1-cd:9.1.1-windowsservercore-ltsc2019
    volumes:
      - .\data\cd:C:\inetpub\sc\App_Data\logs
    isolation: hyperv
    ports:
      - "44002:80"
    links:
      - sql
      - solr

  cm:
    image: sitecore-xm1-cm:9.1.1-windowsservercore-ltsc2019
    volumes:
      - .\data\cm:C:\inetpub\sc\App_Data\logs
    isolation: hyperv
    ports:
      - "44001:80"
    links:
      - sql
      - solr

If your docker-compose has isolation set to process, please change it to hyperv (this is mandatory hosts on Windows 10, while on Windows Server docker can also run its process natively). In that case, processes will run in a hypervisor and is not a naked process next to native windows processes, that prevents you from memory allocations errors such as PAGE_FAULT_IN_NONPAGED_AREA and TERMINAL_SERVER_DRIVER_MADE_INCORRECT_MEMORY_REFERENCE

Notice data folder? This is how volumes work in docker. All these folders within data are created on your host OS file system - upon creation, a folder from container is mapped to a folder on a host system, and once container terminates, data still remains persistent on a host drive.

For example, one running SQL Server in docker can place and reference SQL database files (*.mdf and *.ldf files) on a volume externally in such a manner so that databases actually exist on a host OS and will not be re-created on each container run.

My data folder already has data folders mapped to data folders from all the various roles' containers run on previous executions (yours will be blank at that moment before the first run):

Just for a curiosity, below is an example of what you can find within cm folder, looks familiar, right?


Anyway, we are ready to run docker-compose:

docker-compose up

You'll then see 4 containers being created, then Solr container starts making its job, providing plenty of output:

In a minute you'll be able to use these containers. In order to log into Sitecore, one needs to know an IP address of a container running a particular role - so need to refer to a cm container. Every container has its hash value which serves as an identifier, so with docker ps command you can list all docker containers currently running, get hash of cm and execute ipconfig command within a context of that cm container (so that ipconfig runs inside of it internally):

Now I can call 172.22.32.254/sitecore in order to login to CMS:

What else can you do?

With docker you may also execute commands in interactive mode (sample) with -it switch, so you may do all the things such as deploying your code there (it is always good to deploy on top of clean Sitecore instance). That's how to enter an interactive session with remote command prompt:

docker exec -it CONTAINER_HASH cmd

You may go with more folder mappings using volume. Running XP topology offers even more interesting but safe playground for experiments.

Building other versions of Sitecore allows regression testing your code against legacy systems - always quick manner and always on clean! Going ahead you may use it for the development having only Visual Studio running on a host machine, with no IIS and no SQL server installed, publishing from VS directly in docker. Plenty other scenarios possible - it's only for you to choose from.


7. Stopping and clean-up

Stopping containers occurs in a similar way:

docker-compose down
After finished, you won't see any of the containers running by executing
docker ps
You'll be still able to see existing images on a system and their size occupied:
docker image ls

So if finally, after playing it over and want to clean-up you drive a noticed there's way less free disk space now. I want to beware you from doing one more common mistake - don't delete containers data manually! If you navigate to c:\ProgramData\Docker\windowsfilter folder, one can see plenty of them:

These are not container folders, they are symlinks (references) to windows system resources folders - deleting data from these symlinks actually deleted you resources from your host OS bringing you to a sorry state. Instead, use the command:

docker prune -a

This gets rid of all the images and containers from your host system correctly and safely.


8. Afterthoughts

Docker is a very strong and flexible tool, it is great for devops purposes. I personally find questionable using it for production purposes. That may be fine with Linux containers, but as for Windows... I'd rather opt out, for now, however I am aware of people doing that.

Proper use docker will definitely improve your processes, especially combining it with other means of virtualisation. Containers may take you a while to get properly into, but after getting your hands on you'll have your cookbook of docker recipes for plenty of day-to-day tasks.

As for Sitecore world, I do understand it is all only starting yet, but docker with Sitecore becomes more inevitable, as Sitecore drills deeper into microservices. Replacing Solr and SQL Server with Linux-powered images is only a matter of time, and what I am anticipating so much is XP and XC finally running together in Docker, facaded by IDS (ideally also on Linux) just in fractions after calling docker-compose. Fingers crossed for that.

Hope this material serves you as a great starting point for containers and Docker!