Visual testing with Percy and Blazor

Useful links before we start

• Percy: https://percy.io/
• Blazor: https://learn.microsoft.com/en-us/aspnet/core/blazor/components/?view=aspnetcore-7.0

This article will assume that you have already set up a .NET application project and have basic knowledge of setting up build pipelines in Azure Devops as well as some knowledge of Azure.

Creating the component

Let's start  by creating a component so we have something to test. Create your Button.razor component in the folder /Shared/Components/Button and in that same folder create a Button.Example.razor component that renders your Button component with some example data. 

Button.razor

@using Microsoft.AspNetCore.Components.Web
@inherits ComponentBase

<button @onclick="OnClick" class="c-button" disabled=@Disabled name="@Name" title=@Title type="@TypeAttribute">
    <span class="c-button__text">
        @ChildContent
    </span>
</button>

@code {
    [Parameter]
    public string? Name { get; set; }

    [Parameter]
    public string? Title { get; set; }

    [Parameter]
    public string? TypeAttribute { get; set; }

    [Parameter]
    public bool Disabled { get; set; }

    [Parameter]
    public RenderFragment? ChildContent { get; set; }

    [Parameter]
    public EventCallback<MouseEventArgs> OnClick { get; set; }
}

Button.Example.razor

<Button Title="Click button" Name="coolButton" TypeAttribute="button" Disabled="@false">
    Click button
</Button>

Creating the mockup page

Now to render our example button on a mockup page we need to create the mockup area. 

So to start, create the folder  /Areas/Mockups in your project.

This is the folder where all of your mockup pages will live. The mockup pages can also be used if you have a frontend developer who wants to be able to work outside of a database/cms context and mock certain pages or components.

We are going to need 5 files in this folder to set up our mockup page with the button example and the last one being the actual mockup page. I’ve chosen to create one for Button and links so we can test several components at once and generally I would recommend testing more complex objects since there’s a limit to the number of free print screens (5000 at this point) on Percy.io.

Now let's create the files that we need.

Add App.razor in Areas/Mockups

@using Microsoft.AspNetCore.Components.Routing
<Router AppAssembly="@typeof(Program).Assembly">  
    <Found Context="routeData">  
        <RouteView RouteData="@routeData" DefaultLayout="@typeof(MockupPageLayout)" />  
    </Found>  
    <NotFound>  
        <LayoutView Layout="@typeof(MockupPageLayout)">  
            <p>Sorry, there's no mockup at this address.</p>  
        </LayoutView>  
    </NotFound>  
</Router>  

Add _Imports.razor in Areas/Mockups, for shared usings for the mockup pages with the following 2 rows

@using TestingBlazorWithPercy.Web.Areas.Mockups.Layout

@using TestingBlazorWithPercy.Web.Areas.Mockups

Add MockupPageLayout.razor in folder Areas/Mockups/Layout/. This can be used if you want to be able to render and toggle shared components like header, footer, breadcrumbs for all mockup pages.

@inherits LayoutComponentBase

<main class="">
    <CascadingValue Value="this">
    @Body
    </CascadingValue>
</main>

Add _Host.cshtml to render our App.razor

@page "/mockups/{*pageRoute}"
@using TestingBlazorWithPercy.Web.Areas.Mockups
<app>
    @(await Html.RenderComponentAsync<App>(RenderMode.ServerPrerendered))
</app>

Let’s get to our actual mockup page. Add ButtonAndLinksPage.razor to Areas/Mockups/Pages/ButtonAndLinks/ and on that page import and render your example Blazor components, in this case our Button.razor.

@using TestingBlazorWithPercy.Web.Shared.Components.Buttons
@page "/mockups/buttonAndLinks"

<section>
    <Button_Example />
</section>

@code {
    [CascadingParameter]
    public MockupPageLayout? PageBaseLayout { get; set; }
}

Now that we’ve added our custom Mockup Area and created our first mockup page, we need to register our Area endpoint in our Startup.cs or separate startup file MockupStartup.cs. Register your route like this:

app.UseEndpoints(endpoints =>
{
    endpoints.MapRazorPages();
    endpoints.MapBlazorHub();
    endpoints.MapFallbackToAreaPage("~/Mockups/{*pageRoute}", "/_Host", "Mockups");
});

Remember that this also requires that you’ve added the rows below when registering your services:

 services.AddRazorPages();

 services.AddServerSideBlazor();

If you run your application now you should be able to visit
http://localhost:[yourPort]/mockups/buttonAndLinks and see your mockup page.

Configuring Percy

To run visual tests with Percy you need to create an account on Percy.io, if you already have a browserstack account you can log in with that account. 

On your Percy.io page you need to add a project and name it something understandable, perhaps the same name as your project. You can also connect Percy to your Github, Gitlab or Azure DevOps to integrate visual test results into your pipeline validation. You also need to define which branch is going to be your approved baseline for example main or master.

When setting up a new project you will receive a PERCY_TOKEN. This is what we’re going to use in our build pipeline to send print screens to the correct project so make sure to save your token, but don't worry if you forgot you can always go back to project settings and retrieve it again. 

We are going to test with something called Percy snapshot which is the easiest way to set up visual tests with Percy.

If you want to read more about Percy snapshot before we continue, here’s a link : https://docs.percy.io/docs/percy-snapshot

What we need is to install Percy CLI and create a yml configuration file with our defined mockup urls. 

• Run `yarn add percy` in a terminal to add Percy CLI to your project 
• Create snapshot.yml in tests/Percy and include the path your mockup page. 

  - name: "Buttons and links"
    url: /mockups/buttonAndLinks
    waitForTimeout: 30000
  

Every time you create a new mockup page, you just add the path to this file.

This is all great and all, but we need to actually run the tests in our build pipeline and not just on our local machine.

Azure containers

To get our build pipeline up and running with containers you need to setup the following on Azure:

• a resource group, example name: visualtestgroup
• a container registry, example name: visualtestregistry
• a reptosiory within than container registry: visualtests
• create an Azure login that can be used in the build pipeline to login to Azure. 

These will be used in our build pipeline to build and deploy our docker image & containers based on our current Pull Request code so visual tests can be run towards the latest code changes. At this stage you also need to decide if you want to run Windows or Linux containers. 

Create dockerfile

Now let’s get back to our project again. 

To use our created Azure container registry we need to create a dockerfile to build the docker image with our code and modify our Azure-pipeline.yml file (your build pipeline definition) to create an image, a container and run the visual tests.

First create the dockerfile in your project root folder or create a /build folder and add it there. I've called my file Dockerfile.build

It really doesn't have to be an advanced image at this point since we are going to kill the container after each build and we shouldn't have any integrations when just running visual tests.

FROM mcr.microsoft.com/dotnet/aspnet:6.0
WORKDIR /app
COPY . .
ENTRYPOINT ["dotnet", "./TestingBlazorWithPercy.Web.dll"]
EXPOSE 80
EXPOSE 8080

Configuring the build pipeline

Start by adding variables that you need to your Azure-pipeline.yml file.

For example the name of the container registry, resource group and container repository in Azure that you created earlier. 

You can also choose to add your Percy token (PERCY_TOKEN) as a variable here but I recommend to add it as a secret build pipeline variable as well as adding the login information ( acr_username & acr_password) for Azure as secret variables since the pipeline needs to login to create containers. 

Here we'll also add variables for isMain, isPullRequest and branchName since these will be used in tasks and conditions in the pipeline.

The isMain and isPullRequest variables are used to check if the visual test steps should run. We'll only want to run visual tests on new code in Pull Request to check for any changes and approve those changes as base line after merge to Main branch. 

variables:
  outputFolderVisualTests: "$(Build.ArtifactStagingDirectory)/visualtests"
  resourceGroup: "visualtestgroup"
  repository: "visualtests"
  containerRegistry: "yourtestregistry"
  isMain: ${{eq(variables['Build.SourceBranchName'], 'main')}}
  isPullRequest: ${{eq(variables['Build.Reason'], 'PullRequest')}}
  ${{ if eq( variables['Build.Reason'], 'PullRequest' ) }}:
    branchName: $(System.PullRequest.SourceBranch)
  ${{ if ne( variables['Build.Reason'], 'PullRequest' ) }}:
    branchName: $(Build.SourceBranch)

You can also add a check if succeeded() in your conditions (Read more about conditions ) so visual tests won't run if any steps before has failed.

After setting up our variables we’re going to add a powershell task to shorten the current branch name in case it’s too long. We are going to use the branch name to name our image, container and tag and Azure containers have a limit to the number of characters allowed and we of course don't want the build to fail just because of a long branch name.

- task: PowerShell@2
  condition: or(eq(variables.isMain, 'true'),eq(variables.isPullRequest, 'true'))
  displayName: "Set Variables"
  inputs:
    targetType: "inline"
    script: |
      $name = "$(branchName)".ToLower().Replace("refs/heads/","")
      if ($name.length -gt 30) {
        $name = $name.substring(0, 30)
        Write-Host "branchName too long had to shorten to $($name)"
        if($name -match ".*[-_.]$") {
          $name = $name.substring(0, 29)
          Write-Host "branchName ended with hyphen, changed to $($name)"
        }
      }

      Write-Host "Setting  branchname and tagName to $($name)"
      Write-Host "##vso[task.setvariable variable=branchName]$name"
      Write-Host "##vso[task.setvariable variable=tagName]$name"
      exit 0

Now if you haven't done it already, add steps to run yarn install and nuget restore before we build the project and push our docker image.

Build artifact and docker image 

After this we need to build and publish our project. Notice the zipAfterPublish setting in the Build artifact step, this is since we haven't added any unzip step in our docker container build so want the artifact as a folder instead of zip.

In the task after building the artifact we use our variables from earlier to build and push a docker image to our created Azure registry.  You can change the dockerfile path to the specific path where you’ve chosen to put your dockerfile.

- task: DotNetCoreCLI@2
  displayName: "Build artifact"
  inputs:
    command: "publish"
    publishWebProjects: true
    arguments: "--no-restore --runtime win-x64 --configuration Release --self-contained --output $(outputFolderVisualTests)"
    zipAfterPublish: False
    restoreDirectory: $(Pipeline.Workspace)/.nuget/packages

- task: Docker@2
  displayName: Build and push Docker image
  continueOnError: true
   condition: or(eq(variables.isMain, 'true'),eq(variables.isPullRequest, 'true'))
  inputs:
    command: buildAndPush
    containerRegistry: $(containerRegistry)
    repository: $(repository)
    dockerfile: $(Build.SourcesDirectory)/docker/Dockerfile.build
    buildContext: $(outputFolderVisualTests)
    tags: |
      $(branchName)

Create the Azure container task

Then we need to create the Azure container with the image we just pushed to Azure. 

- task: AzureCLI@2
  displayName: "Create Azure container for visual tests"
   condition: or(eq(variables.isMain, 'true'),eq(variables.isPullRequest, 'true'))
  continueOnError: true
  inputs:
    azureSubscription: "{Insert Subscription}"
    scriptType: "pscore"
    scriptLocation: "inlineScript"
    inlineScript: |
      az container create -g $(resourceGroup) --name vt-$(branchName) --image yourtestregistry.azurecr.io/$(repository):$(branchName) 
      --os-type Windows --cpu 1 --memory 1 --registry-username $(acr-username) --registry-password $(acr-password) 
      --ip-address Public --ports 80 443 --restart-policy Never --dns-name-label yourazuredomain-vt-$(branchName) 
      --environment-variables ASPNETCORE_ENVIRONMENT=Production ASPNETCORE_URLS=http://+:80 COREHOST_TRACE=1

Run Visual tests

Finally we create a task to run the visual tests with Percy and take note of the environment variable PERCY_TOKEN being used here.

- script: |
    cd TestingBlazorWithPercy.Web
    yarn run percy snapshot ./Tests/percy/snapshots.yml --base-url=http://yourazuredomain-vt-$(branchName).northeurope.azurecontainer.io
  env:
    PERCY_TOKEN: $(PERCY_TOKEN)
  continueOnError: true
  displayName: "Visual tests"
  condition: or(eq(variables.isMain, 'true'),eq(variables.isPullRequest, 'true'))

Cleanup container and image

At the very end we’re going to do some cleanup. We will delete the Azure container that was created so it doesnt stay alive costing us money or getting accessed by non authorized users and then finally deleting the docker image tag since images take up space.

- task: AzureCLI@2
  displayName: "Delete Azure container"
  condition: or(eq(variables.isMain, 'true'),eq(variables.isPullRequest, 'true'))
  continueOnError: true
  inputs:
    azureSubscription: "{Insert Subscription}"
    scriptType: "pscore"
    scriptLocation: "inlineScript"
    inlineScript: |
      az container delete --name vt-$(branchName) --resource-group $(resourceGroup) --yes

- task: AzureCLI@2
  displayName: "Delete Docker image tag"
  continueOnError: true
  condition: or(eq(variables.isMain, 'true'),eq(variables.isPullRequest, 'true'))
  inputs:
    azureSubscription: "{Insert Subscription}"
    scriptType: "pscore"
    scriptLocation: "inlineScript"
    inlineScript: |
      az acr repository delete --name $(containerRegistry) --image $(repository):$(branchName) --yes

Add container logs if needed

If your having issues with your container, perhaps it's not starting so your tests are returning 404. Then you can also output the logs from your Azure container in the pipeline like this 

- task: AzureCLI@2
  displayName: "Logs from Azure container"
   condition: or(eq(variables.isMain, 'true'),eq(variables.isPullRequest, 'true'))
  continueOnError: true
  inputs:
    azureSubscription: "{Insert Subscription}"
    scriptType: "pscore"
    scriptLocation: "inlineScript"
    inlineScript: |
      az container logs --name vt-$(branchName) --resource-group $(resourceGroup)

Now we should be able to run our visual tests as part of our Pull Requests process and Percy will mark any detected changes in your Pull Request's as needing approval.

If you've also added an integration to your Azure DevOps on your Percy account then you can add approved visual tests as a required step for approval of every Pull Request. 

Percy also has a Slack integration so everytime Percy detects any change that needs approval you can send a slack notification into your desired slack channel. 

Finishing thoughts

• Running visual tests will slow down your build pipeline but improve catching any unexpected changes, especially if you have shared components such as links or buttons that are used on many different components.
• Prioritize your priority 1 components, the components needed to complete your KPI's example buying products. 
• Test complex components with dependencies to smaller shared components or entire page layouts to get the most out of Percy with less print screens. For example perhaps you have a Teaser and a Hero component that both uses your Button.razor, then add both of them to a mockup page and if you’ve made any changes to your Button.razor to fix a bug in your Hero.razor then we’re double checking Teaser component for any visual changes as well.
• You can add a pipeline variable and use as a condition to be able to toggle if visual test tasks should run in your build pipeline, in case of high priority bug fixes needing to bypass visual tests. 
• You can setup building your Test Artifact, Docker image and Azure container as a parallel job in your pipeline configuration. Read more about parallel jobs