12 tests fail on Windows with GIX_TEST_IGNORE_ARCHIVES=1

Note

Except the title, this has not been updated to reflect that the performance test seems now to fail only rarely and never on CI, and that since #1871 the two gix-pathspec::pathspec search are passing. The automatically checked list of expected failures, and output of test-fixtures-windows CI runs on the main branch, have full current details.

Current behavior 😯

Running tests on Windows in a Git Bash environment (similar to the environment in which they run in Windows on CI, where bash is Git Bash), all tests are able to pass normally. However, this apparently relies on the use of generated archives.

When the tests are run with the environment variable GIX_TEST_IGNORE_ARCHIVES set to 1 rather than unset, 15 tests fail. The failing tests, as quoted from the end of the test run, are:

     Summary [ 903.419s] 2363 tests run: 2348 passed (17 slow, 1 leaky), 15 failed, 9 skipped
        FAIL [   7.416s] gix-glob::glob pattern::matching::compare_baseline_with_ours
        FAIL [   3.324s] gix-pathspec::pathspec parse::baseline
        FAIL [   0.015s] gix-pathspec::pathspec parse::valid::glob_negations_are_always_literal
        FAIL [   0.018s] gix-pathspec::pathspec parse::valid::whitespace_in_pathspec
        FAIL [   1.749s] gix-pathspec::pathspec search::files
        FAIL [   3.855s] gix-pathspec::pathspec search::prefixes_are_always_case_sensitive
        FAIL [ 181.270s] gix-ref-tests::refs packed::iter::performance
        FAIL [   4.803s] gix-submodule::submodule file::baseline::common_values_and_names_by_path
        FAIL [  30.106s] gix-submodule::submodule file::is_active_platform::pathspecs_matter_even_if_they_do_not_match
        FAIL [  35.376s] gix-submodule::submodule file::is_active_platform::submodules_with_active_config_are_considered_active_or_inactive
        FAIL [  22.003s] gix-submodule::submodule file::is_active_platform::submodules_with_active_config_override_pathspecs
        FAIL [  15.882s] gix-submodule::submodule file::is_active_platform::without_any_additional_settings_all_are_inactive_if_they_have_a_url
        FAIL [   9.165s] gix-submodule::submodule file::is_active_platform::without_submodule_in_index
        FAIL [   0.193s] gix::gix revision::spec::from_bytes::regex::find_youngest_matching_commit::regex_matches
        FAIL [   0.086s] gix::gix revision::spec::from_bytes::regex::with_known_revision::contained_string_matches_in_unanchored_regex_and_disambiguates_automatically
error: test run failed

The full output is available in this gist, showing a test run at c2753b8, which has the fixes in #1444.

Before those fixes, one other test had been reported as failing here. See #1358 (comment) and the old gist if interested. The discussion in #1345 is still relevant, though it links to this even older gist.

As noted in comments in #1345, the failure in compare_baseline_with_ours seems particularly interesting, since it involves an unexpected effect of .gitignore pattern matching that is different on Windows.

Expected behavior 🤔

All tests should pass, even when suppressing the use of generated archives by setting GIX_TEST_IGNORE_ARCHIVES=1. Differences between Windows and other platforms should be accounted for when intentional and desirable, or fixed otherwise.

Git behavior

Not fully applicable, since this is about a failure of multiple gitoxide tests when run in a certain way.

However, some discrepancies--intended or unintended--between gitoxide and Git may turn out to be related to some of the failures. So this section may be expanded in the future, or perhaps new issues will be split out from this one.

Steps to reproduce 🕹

I ran the tests on Windows 10.0.19045 (x64) with developer mode enabled so that symlink creation is permitted even without UAC elevation, with git version 2.45.2.windows.1.

I used the current tip of the main branch, which at this time is c2753b8. When I opened this issue originally, that did not exist, but all experiments described here have been performed again, and the reported results have been updated. I used the latest stable Rust toolchain and cargo-nextest, though this does not seem to affect the results.

Local development environments and CI sometimes differ in relevant ways, so I also verified that the tests all pass when GIX_TEST_IGNORE_ARCHIVES is not set. This may be considered an optional step, but is beneficial because it checks that all needed dependencies are installed and working, and that failures really can be attributed to the effect of GIX_TEST_IGNORE_ARCHIVES. To run the tests that way, I run this in Git Bash:

cargo nextest run --all --no-fail-fast

Then I did a full clean:

gix clean -xde

Then, also in Git Bash, I ran this command, which produced the test output and failures described above:

GIX_TEST_IGNORE_ARCHIVES=1 cargo nextest run --all --no-fail-fast

The reason the tests must be run in Git Bash is that there is a separate issue where many test failures occur when they are run from a typical PowerShell environment (#1359).

I also ran the tests, with and without GIX_TEST_IGNORE_ARCHIVES=1, on Ubuntu 22.04 LTS (x64) and macOS 14.5 (M1). As expected, all tests passed on those systems, confirming that the failures are Windows-specific.

While a number of changes have been made to the test suite, this issue holds up without modification: the same tests fail on Windows as before (and as shown above) when GIT_TEST_IGNORE_ARCHIVES=1, at c1d7c02.

The gix-ref-tests::refs packed::iter::performance test may depend on the speed of the test machine, but that was the case before. On my machine, it continues to take about the same amount of time (and fail for taking too long).

In view of the insight in #1442 that Git Bash MSYS2 behavior of ln -s creating copies rather than symlinks is behind the new failure (not listed above) of

gix-diff-tests::diff tree::changes::to_obtain_tree::many_different_states

when GIX_TEST_IGNORE_ARCHIVES=1, I was curious if causing it to actually create symlinks by setting MSYS=winsymlinks:nativestrict would change of the failing tests noted here to pass.

It causes one of the tests whose failure is noted here to pass:

gix::gix object::tree::diff::track_rewrites::copies_in_entire_tree_by_similarity_with_limit

The other failures here are unaffected.

It also causes 9 tests that had otherwise passed to fail, both with and without GIX_TEST_IGNORE_ARCHIVES=1. I've opened #1443 for that.

As expected per #1358 (comment), the fixes for #1443 in #1444 have also made this test, originally reported as failing here, pass:

gix::gix object::tree::diff::track_rewrites::copies_in_entire_tree_by_similarity_with_limit

I have edited the description here accordingly, updating the title and summary list of failing tests, linking to a new gist with the current output of a full run, and also revising for clarity and adding some missing details.

Here are updated runs. The failing tests are the same. This is just to clarify that, since I'm working on some changes in gix-testtools (in #1591 and elsewhere) and ran this on the current tip of main to compare against.

The reason for doing two runs within a short time of each other and to report both in the gist is to check that the number of tests reported as leaky varies across runs rather than having changed due to recent code changes (over time or across branches).

2024-10-22 update: The situation is unchanged at db5c9cf (gitoxide v0.38.0).

Revert "simplify .gitattributes"

A new test from #1618, gix-merge::merge tree::run_baseline, also fails on Windows with GIX_TEST_IGNORE_ARCHIVES=1. But I have not added it to the list here, because #1652 includes a fix for it in the changes it proposes.

Thank you! Do you think it makes sense (and is in the time budget) to run the tests on Windows as well with GIX_TEST_IGNORE_ARCHIVES=1? If so, I think it makes sense to do it.
Answering my own question, it's clearly not in the cards as the fast test takes 3 to 4 minutes on MacOS and Linux but will take 14-15 minutes on Windows, while the full test takes 13 minutes on Linux.

However, what if the new internal tool would be used to detect changes compared to the base of a PR, extract the involved crates, and then would run the tests only for the crates in question with the ignore-archives flag enabled? Probably this can even be done in scripting entirely without having to compile internal at all.

What do you think?

However, what if the new internal tool would be used to detect changes compared to the base of a PR, extract the involved crates, and then would run the tests only for the crates in question with the ignore-archives flag enabled? Probably this can even be done in scripting entirely without having to compile internal at all.

It seems to me that, depending on how narrowly or broadly it detects crates as involved, this might either miss breakages that it is intended to catch, or implicate too many crates to confer an adequate speed improvement. My thinking is that changes to one crate can affect any crates that depend on it directly or indirectly, and the effects may only be seen in tests of functionality in the dependent crate.

Clarify expected/actual failure colorization

Moved to this Bugs Discussion. This issue is being closed because tracking without a clear actionable horizon now lives in Discussions. Once work begins, each failure can be taken on one by one.

AI disclosure: This comment was written and posted by OpenAI Codex on behalf of Byron.

I don't think there's any fundamental blocker to fixing any of these, and these days it should be possible to do so efficiently. The failures seem to clump into groups also, so we would for the most part not really be fixing individual failures but instead, in most cases, fixing whatever kept a fixture from being able to be created or from working properly on Windows. So I would favor keeping this an issue, though I see your reasoning about it being too broad and belonging as a discussion--even though I would consider that to be for reasons different from many other issues that have been converted to discussions, where it is not really feasible to move forward.

A possible reason to keep this an issue is that anything like this that were found to happen today would be expected to be reported as an issue and would, presumably, give rise to an attempt at a fix. If the goal of converting issues that can't be acted on immediately to discussions is to clear the way for things that can be acted on immediately to be acted on, then that argues in favor of keeping this as an issue, since it can be acted on at any time.

Of course, you are under no obligation to work on this--nor is anyone else--and I don't have more time at the moment than the few minutes it takes to type this up to work on it now. However, with recent developments including improved LLMs, improved workflows and practices around their use, and more people contributing, I think this is very readily actionable. You might even be able to point Codex or other agentic coding tools at this whole thing, which I would be in favor of and which I would encourage attempting. With that said, there are enough subtleties that it will be important to check its changes carefully (a good practice anyway, and in my opinion one that I expect to remain essential for a long time, at least for all library code and tests thereof).

Thanks for the encouragement, let's get these fixed then :). That's exactly the idea of this policy.
On Windows, I don't yet have my skills, but I should probably just put them into the repository at some point, but also, for this issue it's no loss.

Stabilize regenerated fixtures on Windows

Thanks for the encouragement, let's get these fixed then :).

That's great, thanks!

Navigation Menu

Repository navigation

12 tests fail on Windows with GIX_TEST_IGNORE_ARCHIVES=1 #1358

Current behavior 😯

Expected behavior 🤔

Git behavior

Steps to reproduce 🕹

41 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

12 tests fail on Windows with GIX_TEST_IGNORE_ARCHIVES=1 #1358

Description

Current behavior 😯

Expected behavior 🤔

Git behavior

Steps to reproduce 🕹

Activity

EliahKagan commented on Jun 23, 2024

EliahKagan commented on Jul 5, 2024

EliahKagan commented on Jul 6, 2024

EliahKagan commented on Sep 11, 2024

EliahKagan commented on Nov 5, 2024

Byron commented on Nov 5, 2024

EliahKagan commented on Nov 5, 2024

41 remaining items

Byron commented on Jul 24, 2026

EliahKagan commented on Jul 24, 2026

Byron commented on Jul 24, 2026

EliahKagan commented on Jul 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions