Writing tests for core PHP
I recently attended the Manchester PHP TestFest 2010 event where I discovered that writing tests for core PHP is surprising easy! All you need to know is a little PHP because tests for PHP are written in PHP... here are my notes from the day.
Getting a test environment set up
I followed the instructions for Ubuntu 9.10 here - http://wiki.php.net/qa/testfest-2010#how_can_i_build_a_php_testing_envir... - it all worked fine even though I run Ubuntu 10.04. The link includes instructions for getting set up on Mac OSX and on Windows too.
You need a working copy of PHP 5.2, PHP 5.3 and HEAD (or TRUNK as it's known) to run tests against. I downloaded the script referred to in the link above and opened it up to take a look.
The key things the set-up script does...
Firstly it ensures you have these required packages installed:
- autoconf
- build-essential
- libxml2-dev
- pkg-config
Because I already had them installed I commented out the line in the script but you can probably leave it in if you're at all unsure.
Although it's not included in the script, it's worth installing lcov at this point, it provides the code coverage reports we will need later. You can do this by adding it to the script or issuing the following command in the terminal:
sudo apt-get install lcov
The script then creates three directories:
- php52
- php53
- php-trunk
It'll create these wherever the script is executed so it's best to create a directory somewhere handy and do all your work there.
Next the script downloads a snapshot of each version (5.2, 5.3 and TRUNK) of PHP from http://snaps.php.net into the appropriate directories and extracts the tarballs.
It then runs ./configure and make for each version but the script does not actually install PHP because you don't need to and probably don't want to either.
Lastly the script cleans up after itself by removing the snapshot tarballs it downloaded.
You can run the script passing in arguments, --help displays the available options as follows:
matason@rudolf:~/php-test-fest$ ./Ubuntu-9.10.sh --help Usage: ./Ubuntu-9.10.sh [options] --help = Show usage and options. --cached = Attempt to use a cached file, keep downloads cached. --force = Overwrite existing files if present. --php52 = Only build the latest PHP 5.2. --php53 = Only build the latest PHP 5.3. --phptrunk = Only build the latest PHP Trunk.
A required step is to set an environment variable to specify where the PHP executable is, this is picked up by the run-tests.php script we'll be using later and ensures that tests are run against the version of PHP we want.
export TEST_PHP_EXECUTABLE=sapi/cli/php
That's it for set up, now to write a test!
Finding something to test
The first thing to do is find something to test, this can be the hard bit!
Start by browsing the test coverage reports at http://gcov.php.net/ - there are reports for PHP 5.2, PHP 5.3 and HEAD, click PHP 5.3 and then "coverage" in the left sidebar. At the TestFest, Ben advised us to write our tests for PHP 5.3 first and then see if they also run against PHP 5.2 and HEAD.
The test coverage reports show all the directories and files in PHP along with the current percentage of covered code. The aim is to increase coverage by writing tests that reduce the red lines as these indicate lines of code that are currently not covered by tests - drill down to file level to see which lines of code need tests.
Once you've identified some red lines to eliminate you then work backwards, it might be a simple case of testing all the functions of the extension with various parameters, that can sometimes result in a quick win. Otherwise you may find yourself tracing back through to the code to see what conditions need to be in place for the section of code to be executed.
During the TestFest I spent most of the day tracing through code, I eventually discovered the section I was looking at didn't appear to ever get called. I was advised to submit a patch to remove the section of code which I will do once I've further and fully investigated it.
Writing a test
The guide at http://wiki.php.net/qa/testfest-2010#how_do_i_write_phpts is well worth a read.
A PHP test is a text file with a .phpt extension. Additional files like images may be included, for example, if you're testing the image extensions gd or exif, you may want your test to interact with a specific image type. The test files, including any additional files, usually reside in a directory called “tests” inside the particular extension.
So here's the test I finally submitted:
--TEST--
Test reading thumbnail.
--CREDITS--
Chris Maiden <hello@webgoodness.co.uk>
# PHPNW Testfest 2010
--SKIPIF--
<?php if (!extension_loaded('exif')) print 'skip exif extension not available';?>
--FILE--
<?php
$exif_data = exif_read_data(dirname(__FILE__) . '/exif_read_thumbnail.jpg', NULL, FALSE, TRUE);
// @todo - Thumbnail string appears to change on each run of the test, need to
// investigate why.
unset($exif_data['THUMBNAIL']['THUMBNAIL']);
var_dump($exif_data);
?>
--EXPECTF--
array(11) {
["FileName"]=>
string(23) "exif_read_thumbnail.jpg"
["FileDateTime"]=>
int(1284276155)
["FileSize"]=>
int(1240)
["FileType"]=>
int(2)
["MimeType"]=>
string(10) "image/jpeg"
["SectionsFound"]=>
string(33) "ANY_TAG, IFD0, THUMBNAIL, COMMENT"
["COMPUTED"]=>
array(14) {
["html"]=>
string(20) "width="1" height="1""
["Height"]=>
int(1)
["Width"]=>
int(1)
["IsColor"]=>
int(1)
["ByteOrderMotorola"]=>
int(1)
["UserComment"]=>
string(16) "Exif test image."
["UserCommentEncoding"]=>
string(5) "ASCII"
["Copyright"]=>
string(41) "Photo (c) M.Boerger, Edited by M.Boerger."
["Copyright.Photographer"]=>
string(19) "Photo (c) M.Boerger"
["Copyright.Editor"]=>
string(20) "Edited by M.Boerger."
["Thumbnail.FileType"]=>
int(2)
["Thumbnail.MimeType"]=>
string(10) "image/jpeg"
["Thumbnail.Height"]=>
int(1)
["Thumbnail.Width"]=>
int(1)
}
["Copyright"]=>
string(19) "Photo (c) M.Boerger"
["UserComment"]=>
string(5) "ASCII"
["THUMBNAIL"]=>
array(2) {
["JPEGInterchangeFormat"]=>
int(134)
["JPEGInterchangeFormatLength"]=>
int(523)
}
["COMMENT"]=>
array(3) {
[0]=>
string(11) "Comment #1."
[1]=>
string(11) "Comment #2."
[2]=>
string(13) "Comment #3end"
}
}
Pretty simple eh?
Once you've saved your test in the "tests" folder, you're ready to run the test and see whether you've increased code coverage.
Running your test
To run your test, cd into your php53 directory and use the following command in the terminal:
run-tests.php/*.phpt
So <path to your tests> might be “ext/gd/tests” or “ext/exif/tests” etc.
The command will run all the tests for the extension you're working on including the new test you've just created. Running the test for just one extension is much much quicker than running all the tests.
The --SKIPIF-- section is there so that you can perform some logic to determine whether your test should run. In the example above we only want to run the test if the gd extension is available.
Once the tests have completed you should get a report on the command line:
===================================================================== Number of tests : 37 37 Tests skipped : 0 ( 0.0%) -------- Tests warned : 0 ( 0.0%) ( 0.0%) Tests failed : 0 ( 0.0%) ( 0.0%) Expected fail : 0 ( 0.0%) ( 0.0%) Tests passed : 37 (100.0%) (100.0%) --------------------------------------------------------------------- Time taken : 1 seconds =====================================================================
If you get failures, you'll need to get them sorted out!
On handy tip is to deliberately leave the --EXPECTF-- section empty – this will cause a
Checking for code coverage
The next step is to see whether your test has improved code coverage. You can do that with lcov, the following command at the terminal will generate a HTML report in your tests directory just like the reports you see at http://gcov.php.net/
make lcov TESTS=
It's then easy to compare the two reports, your local report and the one at http://gcov.php.net/ to see whether you've covered more code. Increase the code coverage, reduce the red!
We were all supplied SVN accounts to commit our tests to on the day of the TestFest, I am not sure how it works if you weren't at the TestFest and want to commit tests but I'll find out and update this post.
The PHP TestFest is a great way to get into C (and the inner gubbins of PHP) whilst contributing to PHP. I'd thoroughly recommend attending if you can and it's my hope that there'll be more of them taking place around the UK next year.
Thanks to everyone involved, to Ben for organising it, to MadLabUK for providing the venue, to iBuildings for sponsoring the pizza.
Here are some photos of us hacking away, taken by MabLabUK - http://www.flickr.com/photos/madlabuk/sets/72157624932719038/
For more information about TestFest - http://testfest.php.net/
Add new comment