Among other tests, we use the W3C CSS 2.1 test suite to assure the quality of our HTML to PDF converter WebToPDF.NET.
W3C Test Suite
The test suite is published as a set of documents (.xht files) without a folder structure. The first thing we did was organize these files into folders that correspond to the different chapters of the CSS 2.1 specification. We wrote a little tool that generates this folder structure based on the chapter information that is stored in the metadata of each test document. This structure makes testing easier. E.g. if we change some code related to chapter 5, we first run these tests and once these are fine, we run the rest.
Converting all HTML documents to PDF
We use an internal command line tool that enumerates all the test documents and converts each .xht to PDF. If an exception occurs, we flag this test as ERROR. If no exception occurs, the PDF is stored in a temporary output folder. We then compare this PDF with a reference PDF document (this is what we expect the PDF to look like). This reference PDF document was created the first time that a converted PDF was manually approved. If a reference does not exist yet, the test is flagged as NO-REF. If the reference exists but the output differs, then the test is flagged as FAILED. Otherwise, it is flagged as PASSED. Obviously, the ultimate goal is to have all tests flagged as PASSED.
The first time we ran the tests, each test was flagged either as ERROR (an exception occurred) or as NO-REF (because no output was approved yet). We first fixed our code so that no exceptions were thrown.
Inspecting the Test Results
Next, we have to visually inspect each output document and decide whether the output is correct or not. For this we compare the output with how a ‘trusted’ browser renders the HTML. Because this involves inspecting A LOT of output documents we wrote a tool that makes our life a bit easier. This tool evolved during the testing phase in order to maximize productivity. Here is how it looks:
Without explaining every detail of this tool, let me go over the most-used features. What you see in the above picture is how our converter renders attribute-003.xht (in Chapter 5 of the W3C test suite) to PDF. This test is self-explaining. Obviously, each line has the color that it should have.
The button F9 lanuches a browser (we can select from all major browsers) so we can compare our output with that of a major browser such as Chrome. F11 (Next) jumps to the next test. F12 (Make Reference) copies the output to the reference folder, basically approving the output.
Per chapter we generate a test report that gives a quick overview of the results as follows:
For the complete test set we generate a test report like this one:
As always, comments are welcome!