AI generated test cases?

[msg7086]

Now that we have lots of powerful AI tools, we can do many things like static code analysis.
Then once we have analysis report and find issues, usually we want to fix them, likely with some sort of refactoring (say old malloc/free/new/delete to RAII containers), then we hits the issue that we can't validate our changes easily.

I'm thinking why don't we introduce some sort of automated testing? AI can generate lots of test cases given boilerplates. For example, we have lots of SIMD optimized paths, under internal plugins, many of them are quite important (convert bits, resizers, transforms, etc), and they are only protected by eyeball checks, no automated testing at all. We can generate quite a few test cases for correctness, and also performance evaluations for compilers.

Once those steps are done, then we can move forward to fix issues inside production code, refactor them into more modern styles, etc, without worrying too much that something might break by a bit of mistakes.

Some of you may be skeptical about how AI models perform on production code. So I wouldn't say we should let AI do all the work now, but adding test framework and usecases, I believe, are definitely worth it.

Example code: https://github.com/HomeOfAviSynthPlusEvolution/neo_DFTTest/blob/master/tests/test_core_internal.cpp

[qyot27]

I figure I probably should make my stance on AI tooling known, as it'll only get worse over time. Mostly this is not about AviSynth+ specifically, except in the couple of spots where it clearly is. I personally wouldn't use any of it in my own work, because of the environmental, economic, and ethical concerns involved. But I wouldn't object to having some kind of automated testing suite that falls under the Ring 2 example (see below) in the repository or organization (if we'd rather have it as external as possible).

The actual source code itself is Ring 0. IMO, no directly AI-generated code should be there, not because of quality concerns (making sure developers can understand and are held responsible for the code they're committing is an absolute priority, so as long as it's vetted, the quality would be a non-issue), but because I don't trust it to properly observe license compliance. The last thing we need is to end up with stuff in the core that was sourced from code with GPL-incompatible licenses (or even if it was sourced from something GPL-compatible, not declaring proper attribution because the AI filed off the serial numbers).

Ring 1 would be things like documentation, where any use is mostly to collate what the source code already says about the topic or to provide for machine translation, but not to the extent of 'stereotypical LLM response' output. My main concern here is just that it can be too wordy and over-simplify things that documentation needs to be technical about, but it's still permissible as long as it's managed to ensure it isn't either hallucinating features that don't exist/don't work that way or is properly phrased.

Unit tests, API checks, and developer-based bug hunting would be in Ring 2, and would be where it should be mostly(?) fine, as it's external to the core sources, only utilizing it to ensure things are correct or that regressions aren't being introduced (or to flag when an INTERFACE_VERSION or BUGFIX_VERSION needs to be bumped). This would also technically include user-submitted bug reports (or maybe those would be in Ring 3), provided it isn't just junk like the flood of nonsense reports that led cURL to stop offering a bug bounty. Again, it should still be checked to make sure it's testing things that are real and the tests aren't faulty, but the reason we don't already have unit testing (despite it being mentioned many years ago as a goal) is because how tedious it would be to write all of it, or to figure out how granular we want the tests to even be. There definitely is value to be gained from using it there, as being able to weed out the valuable things from a generated list is easier than coming up with the list yourself in the first place.

I don't quite know how I feel about the idea of using it to do something like prototyping completely new features to get the basic shape, which then gets rewritten and refactored wholesale by the developer before any attempt to submit it.

[msg7086]

I totally agree with your point on the concerns regarding licensing issues. First of all pure AI generated code are not copyrightable, thus if someone creates a plugin purely using AI, it will be considered public domain, and you can't protect it with GPL or MIT license. Secondly, as you mentioned, there could be risks of GPL-incompatible code accidentally introduced into the code base, causing legal issues.

I think for now, both unit tests and trivial bug fixes are considered safe, as trivial changes are not copyrightable, thus even if AI made changes to correct bugs or setup test cases, it likely won't be an issue. It can also generally be used for refactoring, since refactoring existing code while maintaining original algorithm is generally considered derivative work of the original code (i.e. original copyright protected implementation), it's likely also fine.

For that gray area, I think simple prototyping is fine (AI generate / suggest structure, developers fill the actual code). At least I would have a brainstorming with AI about the design details before I start typing code.

[msg7086]

I have completed the unit test project here https://github.com/HomeOfAviSynthPlusEvolution/AviSynthPlus-UT/issues .
After fixing a few reported issues locally, I'm getting 100% pass rate for the 1065 test cases.
A few static-review finding remains in the issues section in my repo, so to avoid flooding your issues section. You can review and pull tickets into this repo then fix them. 49 issues were found so far.