Tidy needs some test procedures!

[geoffmcl]

This is an important topic... and needs a champion to solve it!

Several poster have offered some parts of the solution... please read them... 1, 2, 3, and more...

At this moment @vielmetti seems to have the best solution for this, but also like others test solution, certain issues still need to be resolved...

Really we do not want to replace one lame solution with another lame solution, not that the TAP idea is lame! But what do we gain here?

At present a test.t requires that we re-write each test! At what cost for each new test? Wow, is that really necessary? As expressed can we not leverage the files in test/input, and test/testbase?

And it does not yet seem to address that certain issues that need more than a simple exit value test. It needs to compare the previous known tidy output, both the 'message' and the 'html' output, to see if something has changed! And that can depend on the the options used.

The current 327 seems such a case. It seems solved until you add the option --show-body-only yes, when the warning is erroniouly still shown. It should fail with this option. How do we deal with this?

Really we need help on this. Tidy needs a solution for this... but I am sorry, I do not see it yet!

[balthisar]

This is a huge engineering task in its own right. We could start by defining some specific, stated goals for testing. It's overly broad to state the result is that Tidy works properly.

Tidy was written was long before unit testing was popular and proper unit testing would require refactoring a lot of Tidy code. Some simple test cases could be made for the major services without too much refactoring, though.

I'm not certain that unit testing will tell us enough about Tidy as whole, though, and so some type of integration or functional testing is the real requirement. Integration testing is hard without being able to trust that the individual units are always working, and so we're back to unit testing.

Regression testing is the only type of testing that's performed now, but as @geoffmcl mentioned the inputs are often decoupled from the option settings and so if we can take those into account we could turn this into a kind of functional test.

The output compare should consist of two things: does the tidy'd output match the expected output? I think we have most of that in testbase. Capturing the message output with text as-is could be dangerous because we're also looking at localization and/or English messages could change. We could use the raw data from one of the message callback filters, though, and record just the error code and location data.

Do we all want to learn Python? Ruby? Make it non-portable and require bash? Or maintain something in Tidy's native C?

The issue of configuration differences isn't small, either. Given what I think are 96 different options with possible option values, do we run each test with all possible combination of options? Or do we introduce the possibility of human error and try to figure out the best options on a case-by-case basis? show-body-only is an obvious one. We can probably ignore tidy-mark, but there are several that can have unexpected interactions with each other unless we test for them. But it's chicken and egg, because right now we don't know how they interact without manually reviewing them.

I supposed we could build an program to build the tests. Given each test case we'll programmatically adjust every option, capture the output, and compare everything against everything for differences, but this may still give us a hundred or more variations of the same test depending on options interacting.

My head hurts thinking about it.

[balthisar]

I've weeded down the possible formatting options very, very loosely and without too much thinking to 62. Most of these are binary, but some aren't, but even if they were all simple yes/no, that's still 4.6x10^18 possible output variations for a single document. Granted a lot of them won't be different because a lot of test cases don't include (e.g.) repeated attributes or php tags or things like that.

That 62 discounts things like file encoding, verifying the correct number of tabs/spaces are used, etc., making sure wrap works at 80 vs 68, etc.

[joeheyming]

Looking at what is currently in the test/ directory. I think that MS bat files are not a good idea since that is not portable. Any other cross platform scripting language would be better than this (and may encourage others to write more tests)

I think the only option we have at the moment is to write black box tests. I propose that we first get continuous integration running (Travis CI) so that for any merge to master, tests get run. This will force us to start thinking in a cross platform way. Then start converting the existing tests to run in the CI suite. Then we can start talking about unit tests.

[geoffmcl]

@joeheyming thank you for starting to look at this...

The original tests, still all there, are in unix shell scripts, which are also not too portable. Charles and I added the MS cmd/bat files to mirror that unix system, and in some ways added a little to it...

It is correct to separate, for want of a better word, the current regression tests, from any more formal unit testing. That is a different topic, maybe Test Procedures 2. And as @balthisar points out, that may require some restructuring, testing interaction of the options, etc, etc,... a much bigger topic.

So just concerning the tests we have. How can these be dealt with, in a cross platform way?

The suggestion is first we get Travis CI running, then we will start thinking? HUH? I would suggest we start think before we automate the tests ;=))

Get some aims? Know what we trying to achieve... fix, get them clean... then, and only then automate, if need be... chicken and egg, no more like putting the cart before the horse...

The present aim of the first 228 tests in testcases.txt, is to compare the exit code with expected, and compare the output with the testbase output. Uses the ../build/cmake/tidy binary, and processes each case to an output. They do not presently do the last step of the diff between the output, and the testbase folder. That must be done manually... but could be added easily...

Is this sufficient? Does this give us what we want? Well yes, I think so. It quickly tests some things for change, but only some things, and this should/could have been done before it was pushed to master, not after! Get in front of the problem, not just be in catch up mode...

If you want, need to experiment, do it in a branch, and make sure you run these tests before offering a PR.

So I would like help, discussion, ideas on getting this working now, in as cross platform way as possible. Including maybe adding some more documentation on how to run them, what to expect, etc... see test/README.md, maybe add to CONTRIBUTING.md...

Concerning Travis CI, I would like some more experience with this before attaching it to htacg tidy, even if we want that. But I do need some help setting up my fork. I did experiment a little a few months ago, but did not get very far...

Does someone have a .travis.yml file that I can try? Do they have something working now I can look at? Another fork? @vielmetti did this, see #266, but have not heard from him in a while, and it seemed to involve re-writing each of the test???

I definitely want to experiment, see what is possible, before maybe recommending it to htacg tidy. It should first use what we have. And new, different things should be thought about, discussed, setup, experimented with, etc, all before other tests be considered, added...

So there are few issues here -

1. look at, discuss, fix, repair the current set of tests.
2. document, and make them easy to run by every developer.
3. experiment, test what travis ci can bring.

I look forward to help, ideas, patches, PRs, on these topic... thanks...

[joeheyming]

If I have any time to look at this, I would probably investigate using python (or maybe perl since it has good text manipulation abilities). I think python is safer/hip'er, but may have more leg work to get the right set of test utilities.

The easiest way to write a python test would be to use the unittest module. That framework provides a junit style way of test setup/teardown. Given that the test that are written adheres to that framework, you can then build test tools that are common to setup/teardown. Think a test/lib/ (or tools) directory.

I agree with the horse before the cart thing. I was actually thinking that exact phrase after I sent my earlier comment :-p

Travis.yml is simple. Given that you can run any test from the command line, you can make travis invoke those tests (glob invoke) and succeed/pass if their return codes are sane (nonzero). I wrote some unit tests even for lisp with travis. Here is an example in one of my python projects: https://github.com/joeheyming/redef/blob/master/.travis.yml

[geoffmcl]

@mcepl suggests another neat target, see 266 of $ make check. Maybe he will come back with a patch or PR... thanks...

And re-reading all of #266 gives some discussion on difference when test are run on different OS/CPU combinations, some of which were never resolved... one has a patch to fix the path separator... etc... lot of reading, information here...

This is issue # 1. Look at, discuss, fix, repair the current set of tests.

[balthisar]

Well, I've posted a testing branch with my take on it. Everything is completely self-contained in testing/benchmarks/, and there's a pretty comprehensive Ruby script regression_test.rb that performs the testing.

In this case I'm not looking at any of the exit codes; I'm only comparing the error output and the generated HTML. There are still some files that don't pass; I've not root caused them completely but probably relate to the comments @geoff made above.

To have a try, check out the testing branch, make sure you have Ruby installed, and cd into the testing/benchmarks/ directory. Although it's not a Ruby gem, using bundler will ensure that the dependencies are met. If necessary on your system, gem install bundler, and then in the benchmarks/ dir, bundle install.

Running the script without arguments will provide all of the instructions needed to get started.

As of right now, generating new test result files isn't implemented, but I'll work on that as time permits. However all of the regression testing functions, and I think the report that it prints out (and saves) is full of useful information.

I've duped the test cases into benchmarks/cases/ and renamed some of them to conform with the testing applications file naming conventions. A look in the cases dir should make it pretty apparent what's required.

In addition to @geoffmcl's standard test cases, I've also moved in all of the other test cases, and where required, generated config files for each of them (I've still not added the expectations files, though).

I think it should be trivial to code around some of the platform differences, or at least provide an explanation for those specific cases.

Why Ruby? Because it will run on any OS unlike bash or cmd. Ruby also has a lot of power for a scripting language, meaning I could deliver something a little better than "bare bones" in a limited amount of time. Ruby's fairly easy to read, and I used my overly-verbose commenting style throughout, so there should be few surprises.

Things to consider in the testing so far:

• It's comparing the error output.
• It's comparing the tidied HTML.
• It's NOT looking at Tidy's exit code.
• I'd like to consider using the --quiet output, which provides all of the error messages without the extra text at the bottom. I think this should be sufficient, but opinions are wanted. It would simplify testing and be more solid against future strings changes.

Also to-do is implement all of the expect files for the other, non-core tests. I'll update the testing branch as I get around to it.

Adding additional test cases is very easy by following the naming conventions. Just drop the files into cases.

Finally I think I've made it as cross-compatible as possible. I'll install dev environments into my Windows and Linux VMs just to double-check, but I'm pretty sure I covered all the bases.

[balthisar]

Now the canonize command works, but I've NOT run it on the current cases/. I'll troubleshoot the current failing cases, first.

I may look at some convention to fix path issues and look at processor-specific things, too, although I think #266 is a bug, not a processor feature.

I'm definitely getting some type of encoding issue on case-427664 that I have to troubleshoot.

---

Right now the testing should be 100% backwards compatible barring differences in string output. String output kind of sucks.

If we're willing to abandon backwards compatibility, it would be much nicer to add a new flag to tidy to present only raw data for error output, such as error code, line, column, instead of the English text.

Also similar my earlier comment about standardizing on --quiet.

It depends on what we risk losing by dropping this.

[mcepl]

@mcepl suggests another neat target, see 266 of $ make check. Maybe he will come back with a patch or PR... thanks...

He will not. Sorry. Too much on my plate already.