Repository navigation
Property-based unit tests #662
Description
Activity
- added 2 commits that reference this issue
on Dec 8, 2019 I discovered this kind of testing when I learned scala, and liked it, but never used it in real life.
I'm a bit uncertain about the randomness of things ; as in can this make tests flaky?
- added 2 commits that reference this issue
on Dec 8, 2019 In theory the randomness could lead to a test that does not always fail, in practice that happens rarely, although it does happen. How often it happens depends on the number of tests it runs for each checked property (default is 100), and how well the input space is covered in that number of tests. So far, if something was wrong, it failed pretty much always, but it's not guaranteed.
When a test fails, it shows the inputs and random seeds for which it failed, so it can be reproduced. It's usually a good idea to then turn that failing instance into a unit test for reliably checking regressions. E.g. for IntCharSet, you'd take the parameters it shows, construct them manually in the old
IntCharSetTestunit test, but call theIntCharSetPropertiesmethod that failed with it, i.e. you don't have to duplicate the actual test code. This will unit test will now always fail until the problem is fixed.It seems to be working fine, and it's at least for me way easier to find good tests to write, so I'll continue working on this a bit and add tests for state sets and char classes.
The current unit testing in JFlex is a bit thin, mostly because the cost/benefit ratio of writing those single test cases is fairly low.
We should use property-based unit tests, at least for the different set implementations in JFlex, as well as partitions and other mathematical objects that have nicely statable properties. These would automate the test object construction, and let us specify the contracts for those objects in a way that gains some actual assurance about implementation correctness.
juni-quickcheck (https://github.com/pholser/junit-quickcheck) looks like it would be usable for the job.