Getting pytest results into Neovim's quickfix list

At work, we have a Python monorepo containing multiple projects, each deployed as individual docker containers. Each project has its own set of pytest tests. There is some code that is shared between projects, which is copied into the docker containers using the docker COPY command, or mounted via volumes locally. This is all to say that the file structure in the code is not exactly the same as it is in the containers.

Eg. the code structure of the repo looks like this 👇:

projects
| shared_lib/
| project_a/
| | tests/
| | | test_something.py
| | | ...
| | app.py
| | ...
| project_b/
| | tests/
| | | test_something_else.py
| | | ...
| | app.py
| | ...
...

Whereas, the code structure in the docker containers looks like 👇:

shared_lib/
tests/
 | test_something.py
 | ...
app.py
...

I wanted to be able to run tests for a certain project from within Neovim, and get the list of failed tests in a quickfix list. I use vim-test to run my tests from Neovim, but getting pytest results into Vim’s quickfix list is not one of its built-in features. This becomes especially tricky with the monorepo situation I described before. The file locations outputted in pytest summary are different than how they are in the codebase.

A quick note: To get the most out of this post, you should be comfortable writing configuration for Neovim using lua, and understand how Vim’s quickfix list works.

This is how I solved this problem.

First, let’s write down the commands needed to be able to run pytest tests in their individual containers.

docker-compose -f docker-compose-project-a.yml \
    exec project-a-dev bash -c \
    "TESTING=1 python -m pytest -q --tb=no tests"

Next, we need a configuration file – let’s call it testconfig – that contains information about different projects, and the commands needed to run pytest tests for them. We also need a location prefix for each project.

For example, for the projects mentioned above, the config file would look like:

project-a::docker-compose -f docker-compose-project-a.yml exec project-a-dev bash -c "TESTING=1 python -m pytest -q --tb=no tests"::projects/project_a
project-b::docker-compose -f docker-compose-project-b.yml exec project-b-dev bbsh -c "TESTING=1 python -m pytest -q --tb=no tests"::projects/project_b

The structure of this file is:

<PROJECT LABEL>::<TEST COMMAND>::<PROJECT_LOCATION_PREFIX>

The idea is that we can create a fuzzy search picker using Telescope, which will let us choose a project. The command for that project will be run via Vimux, and it’s output will be piped to a file called testsfailedall.

A sample output generated by pytest looks like 👇:

...............................................FFF....                   [100%]
=========================== short test summary info ============================
FAILED tests/test_something.py::TestSomething::test_a_bit - assert 1 == 2
FAILED tests/test_something.py::TestSomething::test_some_more - assert 1 == 3
FAILED tests/test_something.py::TestSomething::test_even_more - assert 1 == 4
3 failed, 52 passed in 3.03s

We can then filter for lines that start with FAILED , and there we go, we have all the information needed to create a quickfix list.

A telescope picker that uses testconfig like can be created like 👇:

--- lua
local function run_tests_via_vimux(opts)
    local config = get_config_from_file("./testconfig")
    --- config looks like:
    --- {
    ---     { "project-a", "docker-compose...", "projects/project_a" },
    ---     { "project-b", "docker-compose...", "projects/project_b" },
    --- }
    pickers.new(opts, {
        prompt_title = " tests ",
        finder = finders.new_table {
            results = config,
            entry_maker = function(entry)
                return {
                    value = entry,
                    display = entry[1],
                    ordinal = entry[1],
                }
            end
        },
        sorter = conf.generic_sorter(opts),
        attach_mappings = function(prompt_bufnr, _)
            actions.select_default:replace(function()
                actions.close(prompt_bufnr)
                local selection = action_state.get_selected_entry()
                vim.cmd("VimuxRunCommand('" .. selection.value[2] .. ' | tee testsfailedall' ..  "')")

                vim.cmd('silent !echo "' .. selection.value[3] .. '" > testlastproject')
            end)
            return true
        end,

    }):find()
end

More information about how to write telescope pickers can be found here.

Once tests for project_a are run, testlastproject will contain the string projects/project_a, and testsfailedall will contain the pytest output shown above.

The final step involves taking this output and generating a quickfix list out of it. This is where the information stored in testlastproject comes into play. Test failure information from pytest needs to be converted into a format that can be plugged into vim.fn.setqlist (:h setqlist is your friend here).

That is,

FAILED tests/test_something.py::TestSomething::test_a_bit - assert 1 == 2
FAILED tests/test_something.py::TestSomething::test_some_more - assert 1 == 3

needs to be converted into 👇

{
    { filename = "projects/project_a/tests/unit/test_something.py",
      pattern = "test_a_little",
    },
    { filename = "projects/project_a/tests/unit/test_something.py",
      pattern = "test_some_more",
    },
}

The main function for the last step looks like this 👇 (get_failed_test_summary builds the lua table that contains the test result data; not shown here)

--- lua
function M.failed_test_qf()
    local last_test_project = lines_from("testlastproject")[1][1]
    vim.cmd("silent !cat testsfailedall | grep 'FAILED ' > testsfailed")
    local qf = get_failed_test_summary('testsfailed', last_test_project)
    if next(qf) then
        vim.fn.setqflist({}, 'r', {title="Test failures ‚☹️  ", items=qf})
        vim.cmd("copen")
    end
end

Here’s how the whole thing looks in action 👇