Continuous Integration

Extension integration tests can be run on CI services. The vscode-test library helps you setup extension tests on CI providers and contains a sample extension setup on Azure Pipelines. You can check out the build pipeline or jump directly to the azure-pipelines.yml file.

Azure Pipelines

Azure Pipelines

Azure Pipelines is great for running VS Code extension tests as it supports running the tests on Windows, macOS and Linux. For Open Source projects, you get unlimited minutes and 10 free parallel jobs. This section explains how to setup an Azure Pipelines for running your extension tests.

First, create a free account on Azure DevOps and create an Azure DevOps project for your extension.

Then, add the following azure-pipelines.yml file to the root of your extension's repository. Other than the xvfb setup script for Linux that is necessary to run VS Code in headless Linux CI machines, the definition is straight-forward:

trigger:
- master

strategy:
  matrix:
    linux:
      imageName: 'ubuntu-16.04'
    mac:
      imageName: 'macos-10.13'
    windows:
      imageName: 'vs2017-win2016'

pool:
  vmImage: $(imageName)

steps:

- task: NodeTool@0
  inputs:
    versionSpec: '8.x'
  displayName: 'Install Node.js'

- bash: |
    /usr/bin/Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
    echo ">>> Started xvfb"
  displayName: Start xvfb
  condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))

- bash: |
    echo ">>> Compile vscode-test"
    yarn && yarn compile
    echo ">>> Compiled vscode-test"
    cd sample
    echo ">>> Run sample integration test"
    yarn && yarn compile && yarn test
  displayName: Run Tests
  env:
    DISPLAY: ':99.0'

Finally, create a new pipeline in your DevOps project and point it to the azure-pipelines.yml file. Trigger a build and voilĂ :

pipelines

You can enable the build to run continuously when pushing to a branch and even on pull requests. See Build pipeline triggers to learn more.

Travis CI

vscode-test also includes a Travis CI build definiton. Because the way to define environment variables is different from Azure Pipelines to Travis CI, the xvfb script is a little bit different:

language: node_js
os:
  - osx
  - linux
node_js: 8

install:
  - |
    if [ $TRAVIS_OS_NAME == "linux" ]; then
      export DISPLAY=':99.0'
      /usr/bin/Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
    fi
script:
  - |
    echo ">>> Compile vscode-test"
    yarn && yarn compile
    echo ">>> Compiled vscode-test"
    cd sample
    echo ">>> Run sample integration test"
    yarn && yarn compile && yarn test
cache: yarn