Fixture Testing in PHPUnit

The situation

Most of my work revolves around services with some API, which do very little actual work and a lot of integration work. The simpler API calls can be pictured something like the diagram below.

A mockigram of the service interactions

All of these services are stateless APIs, meaning that the implicit state (hinted by the blue arrows) is stored out of the scope of my application. At the same time, the individually stateless calls make changes to the state of the whole system. This makes the tests “interesting”.

I’m talking about functional tests of course, unit testing is one floor down and solved. The interesting part in my case is that the functional tests are deeply connected with integration tests. The service cannot actually do much without upstream APIs, so I test it using actual instances of the upstream services. In rare cases, I have mocks of the upstream services, but these take a lot of effort to make and even more effort to maintain.

And that’s how I end up with big functional tests like the one below.

Random code screenshot with random notes

So this single test with its setup is around 250 lines of code. And yes, I created worse. At this moment people divide into two groups — “The Shamers” and “The Accepters”.

Let’s deal with The Shamers first. Usually their reactions starts with “You’re doing it wrong”. From there we can either discuss that the test should not really look like this and be converted to unit test (which is useless, I already have those) or test with mock APIs (which is not much different and doesn’t test the integration really) or that it’s fragile and will break easily (not true in reality) or that it’s ugly (true, but being ugly is not a crime) and should be deleted (not true, these are one of the most useful tests we have).

The other kind of the Shamer argument starts with that this is really an integration test (very true!) and should not be part of the application tests (possibly!) and that there is better tooling for this (yes, in some cases) and that I should use X or Y to do that. Which essentially ends with rewriting the above code to Bash or Postman testing, which is very cool (Postman, not Bash) but ends up with a lot of head scratching when it pops out that we sometimes need to manipulate files on local drive or records in the database.

On the other hand, the Accepters will say something like “Yeah, that’s what you end up with if you want to test it this way”. No importa. Da igual.

Test fixtures & PHPUnit

Test fixture by Wikipedia definition is “a test fixture (also called “test context”) is used to set up the system state and input data needed for test execution”. In the above example, the test fixture is split between the setUpBeforeClass method, some helper method and the actual test method.

Most of my work is done in PHP nowadays, and we’re using PHPunit everywhere. PHPUnit deals with test fixtures in the docs in some very helpful way (cough, cough).

The actual content of the test gets lost in the noise of setting up the test fixture. This problem gets even worse when you write several tests with similar test fixtures.

Glad I can read that in the documentation, I wouldn’t notice otherwise! There’s couple of more important advices:

One problem with the setUp() and tearDown() template methods is that they are called even for tests that do not use the test fixture managed by these methods, in the example shown above the $this->example property.

or

There are few good reasons to share fixtures between tests, but in most cases the need to share a fixture between tests stems from an unresolved design problem.

Well, I’ll sign all of these immediately. But what options we have? PHPUnit is modelled after JUnit and it offers to have the text fixtures in the test method itself, or the setUp method (executes before each test) or the setUpBeforeClass method (executes before each test suite) or scattered in all the places at the same time (see the above example).

The real problem

What I usually start with is writing the whole fixture into the test method itself. The sheer size of the test is not really the biggest problem. It’s long, and difficult to navigate, and slow, and confusing. But the real problems starts when people come and try to fix it.

The usual sequence of thoughts and actions is:

• There is a lot of almost duplicated code in most tests, let’s put it in the setUpmethod.
• Ok, the tests are now actually slower, because the fixtures are created before every test. Let’s move them to the setUpBeforeClass method.
• Well, that didn’t really help, because a lot of the test methods doesn’t need the fixtures at all, and they are unnecessarily initialized. Let’s move the fixture initialization to custom intialize method and call that when needed.
• Ok, so the fixture code is not completely identical, let’s add parameters to the custom initialize methods making the fixture slightly different.
• Ok, so the code is a mess now.

Then someone realizes that the fixtures need to be cleaned up. Remember — these are results of API calls in upstream services. There is some remote state associated with our fixtures. And after few years of improvements, we end up with test class like this:

class SomeUglyTest extends TestCase {
  private string $configuration;
  private string $configuration2;
  private string $tableWithData;
  private string $emptyTable;
  private string $tableWithDataForTestXAndY;
  private static string $workspace;

  protected setUp() {...} // intitializes $configuration and $tableWithData
  protected setUpBeforeClass() {...} // intitializes $workspace
  private initData() {...) // intializes $tableWithDataForTestXAndY
  // do not call this in setUp() !
  private initTests() {...} // initializes $configuration2 and $emptyTable
  private tearDown() {...} // cleans up something
  // doesn't work, because something is sometimes left in the workspace
  // private tearDownAfterClass() {...} 
}

And then someone realizes that the initData method is very similar in three or four test suites and creates a BaseDataTest class. Then the fixture definition is split not only between multiple methods, but also between multiple classes.

By this time, the tests run so slow, that everyone is asking to run the tests in parallel, which si downright impossible because of the partially shared fixtures. And even if they are not shared you never know if they by coincidence happen to rely on same upstream resources. At the same time, no one has any idea if the fixtures are being cleaned up properly or which tests uses which.

Do not pull your hair anymore, change the approach

Fixtures Revisited

Let’s ignore the PHPUnit default approach for a while and pretend that test fixtures are actually a first class citizen. If you heard about this somewhere then, yes this is loosely inspired by Playwright (and very simplistic in comparison (but still useful (at least a bit (imho)))). Imagine a test like this:

class NiceTest extends TestCase {
    public function testComplicatedStuff(): void
    {
        $fixture = new SampleDataAndConfigurationFixture();
        $emptyTable = $fixture->getEmptyTableId();
        $configuration1 = $fixture->getConfigurationIdForEmptyTable();
        $configuration2 = $fixture->getStandardConfigurationId();
        ... do the test ...
    }
}

The fixture setup is gone, it’s hidden away in the SampleDataAndConfigurationFixture class. At first glance I dislike it, because it is impossible to see what system state the test uses. On the other hand if I have to choose between 200+ LOC fixture setup at the beginning of the test and hiding it, then I vote for hiding it. The fixture code is straightforward:

class SampleDataAndConfigurationFixture
{
  public function __construct() {
     ... do the setup
  }
  public function __destruct() {
     ... do the cleanup
  } 
  public function getEmptyTableId() {...}
  ... other getters
}

The key design part is that the fixture represents the system state that is created. It is more general than testComplicatedStuffFixture, or at least it should be treated so. This approach opens the door to reusing fixture classes (not fixture instances). It also makes writing the tests a little more interesting — do these tests require the same system state? Why the system state needs to differ for these tests? How do these system states differ?

With this approach the tests have the following properties:

• The fixture is created in the test method itself (no setUp methods present) and it is encapsulated in a single class/method.
• There is no tearDown method for cleanup — that’s problem of the fixture.
• The whole system state need for a test is clearly represented in the fixture.
• The fixture instances are not shared (not even partially) between tests, which makes the test methods fully independent.
• The fixture classes possibly can be shared. This solves the code duplication problem. I.e. Any test method may use the SampleDataAndConfigurationFixture fixture.
• The fixture is independent on the test suite. This makes organizing the tests much easier as they can be organized logically (by things they are testing) and not by the coincidentally common parts of the system state.

There are obviously some downsides:

• The fixture setup is taken away from the test. Sometimes this can make the tests difficult to understand. I came to terms with this.
• The fixture classes can be reused, but it’s slightly tricky (see below).
• The fixture instances are not reused, making the tests slower usually.
• One needs to decide how generic the fixtures should be. Can I make one fixture for these five tests? Or should I have two fixtures or five? These are generally the same problems as with using setUp methods, they’re now just clearly articulated. Is this really a downside?

Also, there is one property which went through silently almost unnoticed. The fixtures immediately become on-demand! Regardless of what the test suite contains, the tests create only the fixtures they need.

Let’s add more

Reusing Fixture classes

With the described approach, the biggest opportunity is to reuse fixture classes. In principle a single fixture class can be used by multiple tests. But! The important part about the fixture is that its instances must be independent too. This is sometimes hard to spot. Consider a system with the following table:

CREATE TABLE items (
    id INT,
    name VARCHAR(255),
    description TEXT
);
ALTER TABLE items ADD PRIMARY KEY (id);
ALTER TABLE items ADD UNIQUE (name);

If the fixture initialization in constructor executes a query like this:

INSERT INTO items (id, name, description) 
  VALUES (1, 'Sample Item', 'This is a sample description.');

It’s probably intuitive that inserting a record with id=1 is going to be a problem if two instances of the fixture are created. Even without reusing the fixture, it will be a problem if the fixture cleanup fails for any reason. However, the unique constraint on the name column is something that is a lot easier to overlook.

At the same time, I also need to store the generated item ID for the fixture cleanup. So the trivial code to insert one row in a database becomes slightly more complicated to be safe for reuse:

class SampleDataAndConfigurationFixture
{
  public function __construct() {
    $this->conn = ...
    $this->itemId = rand(1, 1000); //maybe too weak in parallel environment
    
    $this-conn->insert('items', [
        'id' => $this->itemId,
        'name' => 'Prefix_' . substr(md5(mt_rand()), 0, 8),
        'description' => 'This is a sample description.',
    ]);
         
  }
  public function __destruct() {
    $conn->delete('items', ['id' => $this->itemId]
  }
  public function getItemId() {
    return $this->itemId;
  } 
}

Writing test fixtures in this way can take a lot of work and the exact technique depends on the upstream system (the above example with an ordinary database is still pretty straightforward). Basically, I need to get rid of most constant values of the fixtures. But the result is worth it.

I declare independence

Reusing Fixtures

Reusing fixture classes reduces the amount of test (or fixture) code. Now I want to reuse fixture instances. In general, this is unsafe and bad practice, because the fixture instance can only be reused if the system state represented by the fixture is not modified by the any of the tests sharing the fixture.

While generally risky, in my case however, the fixture encapsulates objects that are created on remote upstream APIs and take considerable amount of time to create (we’re talking tens of seconds at minimum). So, it’s worthy to try it.

With independent fixture instances, I can safely run tests in parallel. To reuse fixture instances, I need to create some kind of cache.

Caching

Let’s throw some code in:

abstract class FixtureAwareTestCase extends WebTestCase
{
    private function isReusableTestMethod(): bool
    {
        $reflection = new ReflectionObject($this);
        return count(
            $reflection->getMethod($this->name())->getAttributes(ReusableFixtures::class),
        ) > 0;
    }

    /**
     * @param class-string $fixtureName
     */
    private function createNewFixture(string $fixtureName): FixtureInterface
    {
        $fixture = new $fixtureName();
        $fixture->initialize();
        return $fixture;
    }

    /**
     * @template T of FixtureInterface
     * @param class-string<T> $fixtureName
     * @return T
     */
    protected function getFixture(string $fixtureName): FixtureInterface
    {
        $isReusable = $this->isReusableTestMethod();

        if ($isReusable) {
            $fixture = FixtureCache::getReusable($fixtureName);
            if ($fixture !== null) {
                // @phpstan-ignore-next-line
                return $fixture;
            }
        }

        $fixture = $this->createNewFixture($fixtureName);
        FixtureCache::add(
            $fixture,
            $fixtureName,
            $isReusable,
            $this->name(),
            (string) $this->dataName(),
        );
        // @phpstan-ignore-next-line
        return $fixture;
    }
}

The class FixtureAwareTestCase is used as base class for a test suite. The test starts with calling the getFixture method. If the test method has the ReusableFixtures attribute, and a cached fixture instance exists, it will be returned, otherwise a new fixture instance is created.

Example of using this in a test:

class GetAutomationsActionTest extends FixtureAwareTestCase
{
    #[ReusableFixtures]
    public function testGetAutomations(): void
    {
        $fixture = $this->getFixture(AutomationFixture::class);
        ... test
    }
}

This is very crude approach. I’m sure it could be improved more by some magic integration with PHPUnit, but I don’t like magic, and this basic implementation really solves my problem.

The FixtureCache is simply an in-memory collection, the nice thing is that I can see it being even more long-test if the fixtures can be serializable. But that it is future work.

Fixture Implementation

Some of my fixtures are very complex (200+ LOC). In attempt to make them easier to create I added a little bit of magic anyway. Here is a fixture that creates entity in the test database:

class ConversationFixture implements FixtureInterface
{
    use EntityManagerTrait;
    private Uuid $conversationId;

    public function initialize(): void {
        $conversation = new Conversation(
            projectId: rand(0, 1000),
        );

        $this->conversationId = $conversation->id;
        $em = $this->getEntityManager();
        $em->persist($conversation);
        $em->flush();
    }

    public function cleanUp(): void {
        $em = $this->getEntityManager();
        $conversation = $em->find(Conversation::class, $this->conversationId);
        if ($conversation !== null) {
            $em->remove($conversation);
            $em->flush();
        }
    }

    public function getConversationId(): Uuid {
        return $this->conversationId;
    }
}

To skip initializing the Doctrine EntityManager, I inject it into the fixture when it is created. The updated createNewFixture method follows:

 private function createNewFixture(string $fixtureName): FixtureInterface {
        $fixture = new $fixtureName();
        $class = new ReflectionClass($fixtureName);
        $traits = $class->getTraits();

        foreach ($traits as $trait) {
            if ($trait->getName() === EntityManagerTrait::class) {
                $container = self::getContainer();
                // @phpstan-ignore-next-line
                $fixture->setEntityManager($container->get('doctrine')->getManager());
            }
            ... other traits ...
        }
        /** @var FixtureInterface $fixture */
        $fixture->initialize();
        return $fixture;
}

The EntityManager trait is a simple getter/setter in this case

trait EntityManagerTrait
{
    private EntityManagerInterface $em;

    public function setEntityManager(EntityManagerInterface $em): void {
        $this->em = $em;
    }

    public function getEntityManager(): EntityManagerInterface {
        return $this->em;
    }
}

This is obviously optional concept, but I found it useful. The less effort it takes to create a new fixture, the less I’m tempted to hacking the existing ones in weird ways.

Other Options

One option I considered is having fixtures with parameters. But then I’m personally very opposed to having conditionals in tests as they essentially stop being a declaration of system state

A test (or fixture) with parameters is an invitation to hide control flow logic in the parameters. There is huge difference between in meaning:

$this-conn->insert('items', [
        'id' => $this->itemId,
        'name' => 'Prefix_' . substr(md5(mt_rand()), 0, 8),
        'description' => $description,
    ]);

and:

$this-conn->insert('items', [
        'id' => $this->itemId,
        'name' => 'Prefix_' . substr(md5(mt_rand()), 0, 8),
        'description' => $description ?? 'Default description',
    ]);

But the code is so similar, so easy to change from the pure declarative system state that I’d rather avoid parameters in fixtures completely. So far, I managed to live without them.

Another option is using multiple fixtures per test. Again, this is technically easily possible. Then it means, that a fixture does not represent the state of the system completely, just partially. And the parts may overwrite or conflict with the other one. For sake of simplicity, I refrain from having more fixtures for one test.

Wrap-up

The Fixture Testing approach is neither novel nor revolutionary. Yet at the same time it goes against how most people (at least those in my social bubble) write tests. With minimal technical change, it leads to huge conceptual change bringing:

• encapsulated fixtures
• reusable fixtures
• cache-able and on-demand fixtures
• parallel testing
• cleaner code

I hope it’s at least inspirational.