Python testing in Visual Studio Code

The Python extension supports testing with Python's built-in unittest framework and pytest.

A little background on unit testing

(If you're already familiar with unit testing, you can skip to the walkthroughs.)

A unit is a specific piece of code to be tested, such as a function or a class. Unit tests are then other pieces of code that specifically exercise the code unit with a full range of different inputs, including boundary and edge cases. Both the unittest and pytest frameworks can be used to write unit tests.

For example, say you have a function to validate the format of an account number that a user enters in a web form:

def validate_account_number_format(account_string):
    # Return False if invalid, True if valid
    # ...

Unit tests are concerned only with the unit's interface—its arguments and return values—not with its implementation (which is why no code is shown here in the function body; often you'd be using other well-tested libraries to help implement the function). In this example, the function accepts any string and returns true if that string contains a properly formatted account number, false otherwise.

To thoroughly test this function, you want to throw at it every conceivable input: valid strings, mistyped strings (off by one or two characters, or containing invalid characters), strings that are too short or too long, blank strings, null arguments, strings containing control characters (non-text codes), string containing HTML, strings containing injection attacks (such as SQL commands or JavaScript code), and so on. It's especially important to test security cases like injection attacks if the validated string is later used in database queries or displayed in the app's UI.

For each input, you then define the function's expected return value (or values). In this example, again, the function should return true for only properly formatted strings. (Whether the number itself is a real account is a different matter that would be handled elsewhere through a database query.)

With all the arguments and expected return values in hand, you now write the tests themselves, which are pieces of code that call the function with a particular input, then compare the actual return value with the expected return value (this comparison is called an assertion):

# Import the code to be tested
import validator

# Import the test framework (this is a hypothetical module)
import test_framework

# This is a generalized example, not specific to a test framework
class Test_TestAccountValidator(test_framework.TestBaseClass):
    def test_validator_valid_string():
        # The exact assertion call depends on the framework as well
        assert(validate_account_number_format("1234567890"), True)

    # ...

    def test_validator_blank_string():
        # The exact assertion call depends on the framework as well
        assert(validate_account_number_format(""), False)

    # ...

    def test_validator_sql_injection():
        # The exact assertion call depends on the framework as well
        assert(validate_account_number_format("drop database master"), False)

    # ... tests for all other cases

The exact structure of the code depends on the test framework you're using, and specific examples are provided later in this article. In any case, as you can see, each test is simple: invoke the function with an argument and assert the expected return value.

The combined results of all the tests is your test report, which tells you whether the function (the unit), is behaving as expected across all test cases. That is, when a unit passes all of its tests, you can be confident that it's functioning properly. (The practice of test-driven development is where you actually write the tests first, then write the code to pass increasingly more tests until all of them pass.)

Because unit tests are small, isolated pieces of code (in unit testing you avoid external dependencies and use mock data or otherwise simulated inputs), they're quick and inexpensive to run. This characteristic means that you can run unit tests early and often. Developers typically run unit tests even before committing code to a repository; gated check-in systems can also run unit tests before merging a commit. Many continuous integration systems also run unit tests after every build. Running the unit test early and often means that you quickly catch regressions, which are unexpected changes in the behavior of code that previously passed all its unit tests. Because the test failure can easily be traced to a particular code change, it's easy to find and remedy the cause of the failure, which is undoubtedly better than discovering a problem much later in the process!

For a general background on unit testing, read Unit testing on Wikipedia. For useful unit test examples, you can review https://github.com/gwtw/py-sorting, a repository with tests for different sorting algorithms.

Example test walkthroughs

Python tests are Python classes that reside in separate files from the code being tested. Each test framework specifies the structure and naming of tests and test files. Once you write tests and enable a test framework, VS Code locates those tests and provides you with various commands to run and debug them.

For this section, create a folder and open it in VS Code. Then create a file named inc_dec.py with the following code to be tested:

def increment(x):
    return x + 1

def decrement(x):
    return x - 1

With this code, you can experience working with tests in VS Code as described in the sections that follow.

Configure tests

Once you have the Python extension installed and a Python file open within the editor, a test beaker icon will be displayed on the VS Code Activity bar. The beaker icon is for the Test Explorer view. When opening the Test Explorer, you will see a Configure Tests button if you don't have a test framework enabled. Once you select Configure Tests, you will be prompted to select a test framework and a folder containing the tests. If you're using unittest, you will also be asked to select the file glob pattern used to identify your test files.

Note: A file glob pattern is a defined string pattern that matches file or folder names based on wildcards to then include or not include.

You can configure your tests anytime by using the Python: Configure Tests command from the Command Palette. You can also configure testing manually by setting either python.testing.unittestEnabled or python.testing.pytestEnabled, which can be done either in the Settings editor or in the settings.json file as described in the VS Code Settings documentation. Each framework also has specific configuration settings as described under Test configuration settings for their folders and patterns.

If both frameworks are enabled, then the Python extension will only run pytest.

If you enable pytest, VS Code prompts you to install the framework package if it's not already present in the currently activated environment:

Create tests

Each test framework has its own conventions for naming test files and structuring the tests within, as described in the following sections. Each case includes two test methods, one of which is intentionally set to fail for the purposes of demonstration.

Tests in unittest

Create a file named test_unittest.py that contains a test class with two test methods:

import inc_dec    # The code to test
import unittest   # The test framework

class Test_TestIncrementDecrement(unittest.TestCase):
    def test_increment(self):
        self.assertEqual(inc_dec.increment(3), 4)

    # This test is designed to fail for demonstration purposes.
    def test_decrement(self):
        self.assertEqual(inc_dec.decrement(3), 4)

if __name__ == '__main__':
    unittest.main()

Tests in pytest

Create a file named test_pytest.py that contains two test methods:

import inc_dec    # The code to test

def test_increment():
    assert inc_dec.increment(3) == 4

# This test is designed to fail for demonstration purposes.
def test_decrement():
    assert inc_dec.decrement(3) == 4

Test discovery

By default, the Python extension attempts to discover tests once you enable a framework. You can also trigger test discovery at any time using the Test: Refresh Tests command from the Command Palette.

python.testing.autoTestDiscoverOnSaveEnabled is set to true by default, meaning that test discovery is also performed automatically whenever you add, delete, or update any Python file in the workspace. To disable this feature, set the value to false, which can be done either in the Settings editor or in the settings.json file as described in the VS Code Settings documentation. You will need to reload the window for this setting to take effect.

Test discovery applies the discovery patterns for the current framework (which can be customized using the Test configuration settings). The default behavior is as follows:

• python.testing.unittestArgs: Looks for any Python (.py) file with "test" in the name in the top-level project folder. All test files must be importable modules or packages. You can customize the file matching pattern with the -p configuration setting, and customize the folder with the -t setting.

• python.testing.pytestArgs: Looks for any Python (.py) file whose name begins with "test_" or ends with "_test", located anywhere within the current folder and all subfolders.

Tip: Sometimes tests placed in subfolders aren't discovered because such test files cannot be imported. To make them importable, create an empty file named __init__.py in that folder.

If the test discovery succeeds, you'll see tests listed in the Test Explorer:

If discovery fails (for example, the test framework isn't installed or you have a syntax error in your test file), you'll see an error message displayed in the Test Explorer. You can check the Python output panel to see the entire error message (use the View > Output menu command to show the Output panel, then select Python from the dropdown on the right side).

Once VS Code recognizes tests, it provides several ways to run those tests as described in Run tests.

Run tests

You can run tests using any of the following actions:

• With a test file open, select the green run icon that is displayed in the gutter next to the test definition line, as shown in the previous section. This command runs only that one method.

  

• From the Command Palette, by running any of the following commands:

  • Test: Run All Tests - Runs all tests that have been discovered.
  • Test: Run Tests in Current File - Runs all tests in a file that that is open in the editor.
  • Test: Run Test at Cursor - Runs only the test method under your cursor in the editor.

• From the Test Explorer:

  • To run all discovered tests, select the play button at the top of Test Explorer:

    

  • To run a specific group of tests, or a single test, select the file, class, or test, then select the play button to the right of that item:

    

  • You can also run a selection of tests through the Test Explorer. To do that, Ctrl+Click (or Cmd+Click on macOS) on the tests you wish to run, right-click on one of them and then select Run Test.

After a test run, VS Code displays results directly in the editor as gutter decorations. Failed tests will also be highlighted in the editor, with a Peek View that displays the test run error message and a history of all of the tests' runs. You can press Escape to dismiss the view, and you can disable it by opening the User settings (Preferences: Open Settings (UI) command in the Command Palette) and changing the value of the Testing: Automatically Open Peek View setting to never.

In the Test Explorer, results are shown for individual tests and any classes and files containing those tests. Folders will display a failure icon if any of the tests within that folder did not pass.

VS Code also shows test results in the Python Test Log output panel.

Run tests in parallel

Support for running tests in parallel with pytest is available through the pytest-xdist package. To enable parallel testing:

1. Open the integrated terminal and install the pytest-xdist package. For more details, refer to the project's documentation page.

   For Windows

   py -3 -m pip install pytest-xdist
   

   For macOS/Linux

   python3 -m pip install pytest-xdist
   

2. Next, create a file named pytest.ini in your project directory and add the content below, specifying the number of CPUs to be used. For example, to set it up for 4 CPUs:

    [pytest]
    addopts=-n4
   

   Or, if you are using a pyproject.toml file

    [tool.pytest.ini_options]
    addopts="-n 4"
   

3. Run your tests, which will now be run in parallel.

Run tests with coverage

Test coverage is a measure of how much of your code is covered by your tests, which can help you identify areas of your code that are not being fully tested. For more information on test coverage, visit VS Code's Test Coverage documentation.

Tip: Running Python tests with coverage is currently only supported when "python.experiments.optInto": ["pythonTestAdapter"] is added to your User settings.json.

To run tests with coverage enabled, select the coverage run icon in the Test Explorer or the “Run with coverage” option from any menu you normally trigger test runs from. The Python extension will run coverage using the pytest-cov plugin if you are using pytest, or with coverage.py for unittest.

Note: Before running tests with coverage, make sure to install the correct testing coverage package for your project.

Once the coverage run is complete, lines will be highlighted in the editor for line level coverage. Test coverage results will appear as a "Test Coverage" sub-tab in the Test Explorer, which you can also navigate to with Testing: Focus on Test Coverage View in Command Palette (F1)). On this panel you can view line coverage metrics for each file and folder in your workspace.

For finer grain control of your coverage run when using pytest, you can edit the python.testing.pytestArgs setting to include your specifications. When the pytest argument --cov exists in python.testing.pytestArgs, the Python extension will make no additional edits to coverage args, to allow your customizations to take effect. If there is no --cov argument found, the extension will add --cov=. to the pytest args prior to run to enable coverage at the workspace root.

Debug tests

You might occasionally need to step through and analyze tests in the debugger, either because the tests themselves have a code defect you need to track down or in order to better understand why an area of code being tested is failing. For more information on debugging or to understand how it works in VS Code, you can read the Python debugging configurations and general VS Code Debugging articles.

For example, the test_decrement functions given earlier are failing because the assertion itself is faulty. The following steps demonstrate how to analyze the test:

1. Set a breakpoint on the first line in the test_decrement function.

2. Right-click on the gutter decoration next to the function definition and select Debug Test, or select the Debug Test icon next to that test in the Test Explorer. VS Code starts the debugger and pauses at the breakpoint.

   

3. In the Debug Console panel, enter inc_dec.decrement(3) to see that the actual result is 2, whereas the expected result specified in the test is the incorrect value of 4.

4. Stop the debugger and correct the faulty code:

   # unittest
   self.assertEqual(inc_dec.decrement(3), 2)
   
   # pytest
   assert inc_dec.decrement(3) == 2
   

5. Save the file and run the tests again to confirm that they pass, and see that the gutter decorations also indicate passing status.

   Note: Running or debugging a test does not automatically save the test file. Always be sure to save changes to a test before running it, otherwise you'll likely be confused by the results because they still reflect the previous version of the file!

You can use the following commands from the Command Palette to debug tests:

• Test: Debug All Tests - Launches the debugger for all tests in your workspace.
• Test: Debug Tests in Current File - Launches the debugger for the tests you have defined in the file you have open in the editor.
• Test: Debug Test at Cursor - Launches the debugger only for the method where you have your cursor focused on the editor. You can also use the Debug Test icons in Test Explorer to launch the debugger for all tests in a selected scope and all discovered tests.

You can also change the default behavior of clicking on the gutter decoration to debug tests instead of run, by changing the testing.defaultGutterClickAction setting value to debug in your settings.json file.

The debugger works the same for tests as for other Python code, including breakpoints, variable inspection, and so on. To customize settings for debugging tests, you can specify "purpose": ["debug-test"] in the launch.json file in the .vscode folder from your workspace. This configuration will be used when you run Test: Debug All Tests, Test: Debug Tests in Current File and Test: Debug Test at Cursor commands.

For example, the configuration below in the launch.json file disables the justMyCode setting for debugging tests:

{
  "name": "Python: Debug Tests",
  "type": "debugpy",
  "request": "launch",
  "program": "${file}",
  "purpose": ["debug-test"],
  "console": "integratedTerminal",
  "justMyCode": false
}

If you have more than one configuration entry with "purpose": ["debug-test"], the first definition will be used since we currently don't support multiple definitions for this request type.

Test commands

Below are all the supported commands for testing with the Python extension in VS Code. These are all found via the Command Palette:

Django unit tests

The Python extension also offers support for discovering and running Django unit tests! You can get your Django tests discovered with only a few additional setup steps:

Tip: Django test support is currently only supported when "python.experiments.optInto": ["pythonTestAdapter"] is added to your User settings.json.

1. Set "python.testing.unittestEnabled": true, in your settings.json file.
2. Add MANAGE_PY_PATH as an environment variable:
   1. Create a .env file at the root of your project.
   2. Add MANAGE_PY_PATH='<path-to-manage.py>' to the .env file, replacing <path-to-manage.py> with the path to your application's manage.py file.

      Tip: you can copy the path by right clicking on the file in the Explorer view and selecting Copy Path.

3. Add Django test arguments to "python.testing.unittestArgs": [] in the settings.json file as needed, and remove any arguments that are not compatible with Django.

Note: By default, the Python extension looks for and loads .env files at the project root. If your .env file is not at the project root or you are using VS Code variable substitution, add "python.envFile": "${workspaceFolder}/<path-to-.env>" to your settings.json file. This enables the Python extension to load the environment variables from this file when running and discovering tests. Get more info about Python environment variables.

Navigate to the Testing view, and select the Refresh Tests button to have your Django tests displayed!

Troubleshooting

If your Django unit tests are not showing in the Testing view, try the following troubleshooting steps:

• Search for error messages in the Python Output panel. They might provide a hint as to why your tests are not being discovered.

• Try to run the Django tests in the terminal. Then "translate" the same command into VS Code settings. For example, if you run python manage.py test --arg in the terminal, you would add MANAGE_PY_PATH='./manage.py' to a .env file, and set "python.testing.unittestArgs": [--arg] in the VS Code settings.

  Alternatively, you can also find the commands that are run by the Python extension in the Python Output panel.

• Use the absolute path to the manage.py file when setting the MANAGE_PY_PATH environment variable, if you initially used the relative path.

IntelliSense for pytest

Pylance offers IntelliSense features that can help you work more efficiently with pytest fixtures and parameterized tests.

As you're typing the parameters for your test function, Pylance will offer you a list of completions that includes argument names from @pytest.mark.parametrize decorators, as well as existing pytest fixtures defined in your tests file or in conftest.py. Code navigation features such as Go to Definition and Find All References and rename symbol refactoring are also supported.

When hovering over a fixture reference or a parameterized argument reference, Pylance will show the inferred type annotation, either based on the return values of the fixture, or based on the inferred types of the arguments passed to the parameterization decorator.

Pylance also offers code actions to add type annotations to test functions that have fixture parameters. Inlay hints for inferred fixture parameter types can also be enabled by setting python.analysis.inlayHints.pytestParameters to true in your User settings.

Test configuration settings

The behavior of testing with Python is driven by general UI settings provided by VS Code, and settings that are specific to Python and to whichever framework you've enabled.

General UI settings

The settings that affect the UI of the testing features are provided by VS Code itself, and can be found in the VS Code Settings editor when you search for "Testing".

General Python settings

unittest configuration settings

The default arguments for unittest are as follows:

• -v sets default verbosity. Remove this argument for simpler output.
• -s . specifies the starting directory for discovering tests. If you have tests in a "test" folder, change the argument to -s test (meaning "-s", "test" in the arguments array).
• -p *test*.py is the discovery pattern used to look for tests. In this case, it's any .py file that includes the word "test". If you name test files differently, such as appending "_test" to every filename, then use a pattern like *_test.py in the appropriate argument of the array.

To stop a test run on the first failure, add the fail fast option "-f" to the arguments array.

See unittest command-line interface for the full set of available options.

pytest configuration settings

You can also configure pytest using a pytest.ini file as described on pytest Configuration.

Note If you have the pytest-cov coverage module installed, VS Code doesn't stop at breakpoints while debugging because pytest-cov is using the same technique to access the source code being run. To prevent this behavior, include --no-cov in pytestArgs when debugging tests, for example by adding "env": {"PYTEST_ADDOPTS": "--no-cov"} to your debug configuration. (See Debug Tests above about how to set up that launch configuration.) (For more information, see Debuggers and PyCharm in the pytest-cov documentation.)

IntelliSense settings

See also

• Python environments - Control which Python interpreter is used for editing and debugging.
• Settings reference - Explore the full range of Python-related settings in VS Code.