Erwin’s Blog

A startup journey

2022-04-20T08:47:57+00:00

Joining a new startup Smart Cloud

Last week, I agreed to join a new startup Smart Cloud (working title). What the startup is about and what we’re going to build I will hopefully be able to talk more about in the future.

But for now, I thought it would be interesting to document our journey through technology from the absolute beginning to (hopefully) a thriving billion dollar success (or an abysmal failure and deep shame but more likely, something in the middle).

So, in the near future, I hope to be able to write about:

Which principles do we follow?
What technologies do we pick and why?
Which practices do we use and why?
What type of architecture do we aim for and how does it evolve?
What do we value (both technically and culturally)?

And as the system evolves:

What works and what doesn’t?
What did we change?
What can’t we change anymore but wish we could?

Anyway, I’m super excited to get started on this journey.

Testing without mocks - databases

2021-12-31T08:47:57+00:00

Databases are at the heart of most LOB applications, and so they should also be at the heart your unit testing strategies. In this post, i’ll go into some of the mechanics of how to test without mocks. But because I’m finding that this is a huge topic, this post only focusses on persistence (databases).

This post combines information from previous post, including why you should stop using mocking libraries, some tips and tricks.

I’ll try to provide some coding examples to these topics later.

TLDR;

When writing unit tests for LOB applications, I want to include the database as much as possible. It’s a vital aspect of the system, so I won’t ignore this aspect. Where possible and practical, I’ll try to write tests that include a real database. If this is not effective / efficient / possible, i’ll try to see if I can use or write an in-memory test double. Only if that’s not possible will I revert to writing (hand crafted) mocks. This approach gives me the most confidence that my system is actually working correctly.

The elephant in the room: Persistence

Let’s first talk about the elephant in the room when it comes to writing tests without mocks, which is the database (or better said, data persistence).

When discussing mocking, most people agree that that truly external dependencies (such as external services) should be mocked. But when it comes to data persistence, it’s not so clear.

Some people consider the database to be one of those external systems, so they’ll mock it out completely. Other people say that they design their software persistence ignorant and so they don’t have to mock out the database.

I would argue that testing your data persistence is actually really important. In line of business applications, data persistence is one of the most important concerns. It’s the place where most performance problems stem from and the root of a lot of complexity.

But there is no silver bullet (tm) when it comes to handling persistence. The right approach balances out the requirements of the application that’s being tested with the technologies used.

Keep in mind, there is a huge variety in persistence technologies:

Relational databases (in all their various incarnations, such as MsSql, MySql, Postgress, Oracle, etc..)
Event Stores (EventStore, Sql Stream Store, marten, etc.)
Key Value Stores / document databases (MongoDB, AWS Dynamodb, Azure CosmosDB, RavenDB, etc..)
Object stores (Azure Blob storage, S3, etc..)
Indexers (elastic search, Azure Cognitive search, etc..)
Message brokers, Queues, service busses, etc.. (yes, they actually do store data)
many others.

These technologies have vastly different api signatures, characteristics and capabilities. But where in the past, most applications just used one type of database (Relational), it’s become mainstream now to use a mix of these technologies.

So, if you’re building a LOB application, you’re likely going to have to tackle several of these api’s.

Heuristics on choosing an approach

There is no one size fits all approach on how to handle these persistence technologies in your unit tests. They are simply too diverse to make generalized statements about. So what I rely on is a set of heuristics, that try to combine the unique requirements of the system with basic guiding principles.

First of all, what I want to get out of my tests is (From my post on well written tests):

They are easy to write and maintain.
They run fast enough (and ideally in parallel).
They provide actual value (they help you write code and catch issues).

This provides an interesting challenge. Usually there are these solutions:

Optimize for test value: Real db You get the most value (3) when you run your tests against a real database (more on that below). However, this can come at the expense of how easy they are to write (1) and how fast they run (2). There are techniques you can use to improve (1) and (3) though, which i’ll describe below.

Optimize for test execution speed: Fake db. You can fake your database, which optimizes for speedy execution of tests, because there is no real logic. However, you then decrease the value (3) because you don’t know if the mock behaves exactly as the real thing and it is extra code to write / maintain (2). Yes, a mocking library can help, but I get more mileage by hand crafting my mocks IF the api surface is not too big. More on that below.

Optimize for developer efficiency. Most databases have a way to provide an in-memory implementation. For example, for relational databases you can use SQLite, or an in-memory eventstore (SQL Steam Store). These usually are fast (2) and easy to use (1) but are an imperfect implementation so they may have differences in how they respond in certain situations. Therefor, they usually give you less confidence in the overall solution and therefor less value (3).

Picking an approach

So, given these heuristics, how do you choose an approach?

First of all: You don’t have to limit yourself to a single choice.. People so often look for the single and only best way to handle an approach. But if you find that you have a situation that a simpler approach works great for 80% of your tests but 20 % really requires a specialized and more complex approach, by all means, use both approaches.

For example:

You have a straight forward application, not much business logic, relatively simple queries, etc.. But there are a couple of rather complex and very important queries. Use a simple approach like an in-memory implementation for the simple approach but use a real database for the complex queries.

Another example when you use CQRS:

You are applying CQRS in (parts) of your application. This means you have different access patterns for your write model vs your read model. The write model typically involves reading / writing single items by ID whereas the read model involves complex queries. Use an in-memory approach for the write side, and a real db for the read model.

Ok, another example (because most people don’t think of their software as simple / straight forward)

You have a complex application. Lots of business rules, complex queries, etc. But you also have to manage some reference data. This can be covered using simple crud. Use a real database for the complex scenarios, and a simple (in-memory) approach for the crud screens.

Last example, I promise:

You write your business logic tests using an in-memory database, because this allows you to focus on the business logic you’ve just added. However, you also have tests that cover complete business scenarios at api level. These tests have a bigger scope and there it makes sense to use a real database.

So, ask yourself the following questions:

1. How much are you tied to the actual logic of the store?

This is one of the most important questions when deciding what to do with the database.

The more you rely on the specific implementation of the database (for example for performance reasons), the less mileage you’ll get from using a fake version.

For example, if you use a relational database and heavily rely on joins, complex queries, materialized views, specific indexing patterns, or other fancy things, then you might not get much benefit from an in-memory implementation, because it doesn’t give you the fidelity that you’re looking for. Another example would be specific partitioning requirements in dynamodb or kafka for scalability reasons.

However, if you’re simply persisting objects, returning them and have only basic querying over them, or you are just testing straight forward business logic then an in-memory implementation is usually good enough. Usually, happy paths work really well with a in-memory double, but exceptions and platform specific optimizations do not.

2. Is there a good in-memory implementation?

Most often, a good in-memory implementation is worth it’s weight in gold. Oh wait, software doesn’t weigh much, but you get the gist. What I mean is, a good in-memory implementation can cover most common scenarios and that might make this a very effective option. Now, if you depend heavily on the implementation specifics of your database, for example for performance reasons, then you’re probably better off NOT using a real database.

If there is not a good in-memory implementation, perhaps you can write one. But if that’s effective depends on the size of the api surface and of course, how much you really need from your implementation.

I’ve had really good results testing my own business logic with in-memory event stores, in-memory sql databases (sql lite). Sometimes, you have edge cases. For example in relational databases, each database subtle differences in sql dialect, and it’s own way of handling indexes, triggers, constraints etc. These can trip you up if it’s the only thing you rely on.

Another example is when using an in-memory event store. Regular business logic tests work great, but when it comes to testing your logic for when network connection failures occur, it can be better to work with a real database. Docker works really well, but more on that later.

3. How big / complex is the api surface?

The size and complexity api surface of your persistence can also determine your strategy. A key value store, event store or document database typically has a very narrow api surface. This means it’s relatively easy to create test doubles or even complete in-memory implementations for these relatively simple.

The thing that’s usually difficult to implement is the querying api. If you have something that uses linq expressions or fairly straightforward filter parameters, then it may be possible to stick with in-memory test doubles. However, you don’t want to go overboard and have to implement a complete querying language, just to run your unit tests.

On the other end of the spectrum are relational databases. Due to their nature of having (often complex) relations between tables, and having very rich querying options over them, typically makes it very difficult to write your own mocks against.

Tips for Testing with in-memory implementations

An in-memory test double is usually great and my preferred option when writing business logic tests.

Ideally, you have one provided to you, but if not, often they are not too difficult to write (assuming your api surface is not to big / complex).

Take control of eventual consistency

Eventual consistency is often used to improve the throughput of your system. This is often (but not required for) CQRS or event sourced applications. How to handle this gracefully probably warrants an entire blog post, but for now, a short discussion of this topic:

One of the difficult things I’ve found in unit testing is handling eventual consistency. Typically, you take an action, then you verify if the side effects have occurred. Polling or retries are often used to handle this.

But with eventual consistency, how do you distinguish if the action hasn’t happened yet from a failure where the action is never going to happen? With everything being in-memory, a successful test will often run really fast, but a failure can take a while, because you have to retry enough and wait long enough to be sure the test fails for the right reasons.

One of the things I’ve used very successfully is to remove or completely control the eventual consistency from my in-memory implementations.

When the action happens, synchronously process all side effects.
Make the side effects observable. Create a Task that completes when the side effects have happened. This way, you can listen for the effects to have happened and make things much more deterministic.
Control when to run the eventual consistent actions. For example, introduce a ‘ProcessAllMessages()’ method that will then execute your logic.

Testing your in-memory implementation

If you have an in-memory implementation, you must have some level of confidence that it actually works.

One method I’ve used in the past is to create a body of tests that cover both a real and in-memory implementation. This makes sure that your in-memory version responds (within reason) the same as the real thing. It’s also a good place to add additional tests once you find a discrepancy.

To do this, create a base class where you place your tests. Then create two derived classes, that will set up the real or in-memory instance. This way, the same tests run against two databases.

Tips for testing with hand-crafted test doubles

A hand crafted test double is handy when:

no good in-memory implementation is available
The api surface is reasonably narrow, but creating a real in-memory implementation would be too complex.

Tips:

Make your test doubles easily observable.
Hide eventual consistency.

Tips for Testing with real db implementations

Testing with a real database implementation gives you a lot of confidence that your code actually does what it needs to do.

It’s still not going to be 100% the same. For example, in production, you’ll likely use a sql server cluster, not a sql localdb instance. Or, you are using dynamodb, but locally you have only a few partitions, so sharding is not so much of an issue.

Dockerize your setup

If possible, try to use docker to spin up your local databases. This means setting up your environment is much easier (for other developers) and it (likely) makes your tests also portable to run on your build server without requiring software to be pre-installed.

Provide a proper level of isolation

A database is almost by definition a shared resource. Multiple instances of your application and multiple threads within your application all compete with each other to access the same database.

In unit tests, sharing state between tests is a source of really hard to find bugs. When you run a test in isolation, it works fine, but when running all of them (or even worse, only on the build server) then they start failing.

Also, when you run your tests in parallel, you don’t want your tests to interfere with each other either.

Try if possible to avoid sharing your database amongst your tests.

So how do you do this?

Option 1: Use a database per test.

This is an option if your database is not too big. With a small schema, it doesn’t take much time to create / drop your database.

I once used this technique, but after a while it became a bit too slow. Also, it required me to seed the database every time with the same data. So, to avoid this, I created a small tool that created a database once, seeded it with data, then created a backup of the database. I then restored the backup for each test. This proved to be much faster.

Option 2: Use a schema / namespace / table per test.

I often use this technique when using dynamodb. If you allow your test to create a random table name, then each test get’s it’s own table. So, they don’t interfere with each other. I also did this when using elastic. Simply create an index per test.

Some databases make it light weight to create / drop databases or database tenants. Then this would also work to provide some form of isolation.

Don’t forget to delete your data after your test.

Also.. provide a tool to clean up your data manually, because the cleanup logic doesn’t always run.

One system I worked on in the past had a huge database schema. Recreating this database for each test would have been completely impractical. So, in order to make the tests run at an ok speed, I only created a new database per test collection.

Within the test collection, I was sharing the database. This means that I had to be really careful in how to write the tests. For example, it’s fine to insert some data with a unique key, then read that data back again. Since the key is unique for your test, you (likely) won’t get collisions.

Search is a tricky one here. Because, you can find data created by other tests. This means, you’ll either need to make sure you narrow your search appropriately or that you can handle arbitrary large result sets (page through all data found until you find what you’re looking for).

Not recommended techniques

I’ve seen various other techniques around this.

Using a real database instance but never cleaning it up is not something I recommend. It leaves too much crap behind, so after a while you have no idea what happened.

Using locks, then reverting everything once done. I don’t recommend this, because it doesn’t represent the way the transactions would normally complete. Also, you might get a lot of deadlocks in your database.

Manual cleanup after the tests and having each test run separately is another pattern I don’t really like. It’s too easy to forget to add some cleanup code as the system grows and then you get test instability.

Conclusion

In this post, I’ve described various techniques on how you can handle databases when you don’t want to use mocking libraries. As usual, it turned out to be a longer post than I had hoped, so I’ll publish this now and will come up with some code examples later.

Stop making a Mock

2021-12-05T08:47:57+00:00

I don’t like mocks anymore. I used them heavily in the past, but now that I’ve moved away from this approach, my tests have become a lot better. In this post, I hope to convince you to do the same. In a future post, i’ll describe better ways to test.

TLDR;

Don’t rely on Mocking libraries. Reliance on mocking libraries causes:

increased coupling between classes.
reduced readability of your tests.
reduced maintainability of your application.

Don’t blindly declare interfaces on every class. Introduce abstractions when you need them, not as a reflex.

Do:

Design your classes and components so that truely external dependencies can be isolated.
When needed, write hand crafted test doubles to simulate dependencies like external services.

Consider this: Can you refactor your code without having to make changes to your tests? If not, you’re in trouble.

So, stop making a Mock of things and you’ll be better off.

Background: I see mocks everywhere

Part of my job is reviewing code written by other teams. As a (primarily) .Net developer, that means reading a lot of c# code. It’s clear that many developers have become familiar with Test Driven Development, which is of course a good development. However, I’m finding that the .Net Community uses Mocking a lot, using libraries like FakeItEasy, Moq, NSubstitute or one of the many other frameworks.

What I typically see is:

The Single Responsibility principle is followed religiously, so there are a lot of smaller classes.
Most classes have interfaces (dto’s the obvious exception), but there’s only a single implementation in the codebase.
Each is wrapped in it’s own unit test.
Each dependency is mocked out in the unit tests.

These developers clearly think they are getting a lot of benefits from writing tests so they use their good intentions to spend a lot of energy to write tests that don’t nearly provide them as much benefits as they would have hoped.

This is the way

When I challenge this mocking approach, I get a lot of push-back: “A unit has to be a class”. “If you don’t isolate the class, you’re testing things twice”. “This is the only way(tm)”. “You must isolate a class, otherwise you’re doing integration testing”.

The thing is, it wasn’t always like this. This post from Martin Fowler describes that classical TDD didn’t used to rely on mocking.

The classical TDD style is to use real objects if possible and a double if it’s awkward to use the real thing. So a classical TDDer would use a real warehouse and a double for the mail service. The kind of double doesn’t really matter that much.

A mockist TDD practitioner, however, will always use a mock for any object with interesting behavior. In this case for both the warehouse and the mail service

Somewhere down the line, the practice of testing units in isolation, was translated to testing classes in isolation and mocking the rest, facilitated by mocking frameworks. I’ll do my best to describe the problems I’m seeing with this ‘mockist’ approach to testing and show alternatives to them.

The problems with mocking libraries

My biggest problems with the usage of mocking libraries are:

You’re increasing coupling, which limits your ability to refactor.
You’re testing the internals of your class, limiting your possibilities to refactor.
A mock has to simulate the behavior of your class, so you have to implement some logic twice.

I’ll go into these points more below.

Decreased maintainability due to Increased coupling

Using mocking libraries dramatically increases the number of dependencies a class has. Consider the following diagram.

It’s a fairly straightforward set of classes. Class A depends on Class B and C, which in their turn depend on Class DB and class DTO. Intentionally, there is no functional description to these classes, but you could think of class A being a controller, Class B and C providing some form of business logic and Class DB providing Database Access and Class DTO being a data transfer object.

The number of dependencies is fairly low. This makes refactoring a lot easier. Consider:

A change to class DB. It will likely impact class B and C.
A refactoring where parts the logic of B and C can be re-used and will be factored in a separate class. This impacts B and C and likely also A.

The obvious consequence of this approach is, that, if you change the signature of your class, you’ll also have to change all classes that use the signature.

THe non-obvious consequence of this approach, is that in order to create good mocks, you’ll also need to know a little bit about the runtime behavior of the classes being mocked. Not everything can be read from the signature. What data is being returned? In which order? What kind of validation rules are being performed by the class calling your method?

So, if you don’t rely on mocks, you end up with a much decreased maintenance burden.

Refuctoring (having to change your tests when you refactor your solution)

When relying on mocks, you have to create a good test double for the actual logic. This means that your test has to actually have to know deeply about the internal logic of your class. Any change to the internals of your class means a change to your tests. This is known as a ‘refuctor’ (a term coined by Greg Young as far as I know).

Your test is your safety net while doing changes. Making changes to your safety net when you need to rely on it is not a very good idea.

So how do we prevent that? Don’t mock your implementation, but where possible rely on the real implementation (more on that below.)

Hard to understand test tests.

The arrange-act-assert pattern of tests is often used in conjunction with mocking and SRP. So badly, that I often see people adding comments to show which bit of the test actually is part of arranging / acting / asserting. Shouldn’t that be clear from the actual code, rather than the tests?

In the arrange step, you set up your mocks. Often, specific tests have specialized setup requirements. So, each mock setup is every so slightly different. But, because you have to mock out every interaction, it’s not clear if a specific interaction is there ‘just because(tm)’ or if it’s actually significant for your tests.

Then you have an act phase. This is usually the best part of your tests. The thing that’s really important. But it’s buried in the rest of your test. So you have to put a //act comment above it just to call out where it is.

Then there is the assert. Sometimes you see big lists of assertions, but also often a Mock.Verify(). When you first read this test, you have NO idea what’s actually being verified. Then you have to look back to see what’s going on, adding to the overall confusion.

What’s a better approach to this? The Given-When-Then pattern usually makes much more readable tests. Given a certain state, when you perform an action, you observe the following results.

This focusses much more on the functionality (What the system is supposed to do), rather than on the implementation details (How the system is actually performing it).

To compare and contrast, I’ve created two solutions in the same repo:

https://github.com/Erwinvandervalk/Examples.Testing.Corona

One application is written using mocking, the other is not.

Compare the following test that uses mocking with the one below without:

                [Fact]
        public async Task When_saving_new_appointment_then_save_is_invoked()
        {
            // Arrange
            var repository = A.Fake<ICoronaTestRepository>();
            var dateTimeProvider = A.Fake<IDateTimeProvider>();

            var now = new DateTimeOffset(2000, 1, 2, 3, 4, 5, TimeSpan.FromHours(6));
            A.CallTo(() => dateTimeProvider.GetNow()).Returns(now);

            var sut = new ScheduleTestAppointmentCommandHandler(
                repository, 
                dateTimeProvider);

            var scheduleTestAppointmentCommand = new ScheduleTestAppointmentCommand()
            {
                Location = "testlocation",
                ScheduledOn = now,
                TestSubjectIdentificatieNummer = "number",
                TestSubjectName = "name"
            };

            // Act
            await sut.Handle(scheduleTestAppointmentCommand, CancellationToken.None);

            // Assert
            A.CallTo(() => repository.SaveAsync(A<CoronaTestEntity>.That.Matches(c =>
                        c.TestSubjectName == 
                            scheduleTestAppointmentCommand.TestSubjectName
                        && c.TestSubjectIdentificatieNummer == 
                            scheduleTestAppointmentCommand.TestSubjectIdentificatieNummer
                        && c.Location == scheduleTestAppointmentCommand.Location
                        && c.ScheduledOn == now
                        && c.CreatedOn == now
                    )
                    , CancellationToken.None))
                .MustHaveHappened();
        }

In this test, I have to set up several mocks, then verify that the right call to save the data actually happened. But reading this, I have questions:

What does datetime actually have to do with the test? Is the current time important? It may be, it may not be.
What is this ‘CoronaTestEntity’? Why is it important here?
Have I verified every property on the corona test entity? Is it ok now?

The actual logic (calling the command) is hidden between the setup and assertion.

And, if you have to make changes (for example, change the repository signature), you have to get ready for a LOT of changes. Yes, you could probably write this better, but this example closely follows what I see in the wild.

Contrast that with the following:

        [Fact]
        public async Task Can_schedule_appointment()
        {
            var request = Some.ScheduleTestRequest(); 
            var expectedResponse = Some.GetScheduledTestResponse();

            await CoronaTestClient.ScheduleTest(
                request: request,
                expectedStatusCode: HttpStatusCode.OK);

            var (response, result) = await CoronaTestClient.GetTest(The.TestId);

            result.ShouldBeEquivalentTo(expectedResponse);
        }

This test works at the api boundary (though it would have worked equally well at the command / query layer). The test only cares about the things the end user cares about. When you do X, you can observe Y. Anything else is not important for this test.

No, this is not an integration test, because I’m testing a single boundary here. Any external dependencies (such
the e-mail sending in this example). And as for databases? If the test doesn’t slow you down too much, I would encourage you to include them in your tests. Otherwise, create a stub at the absolute lowest level.

Single Responsibility Principle causes harm here

Bob Martin coined the SOLID Principles, with the Single Responsibility Principle being the one people remember the best.

This principle leads you to create small classes, each with their own little responsibility. This means we’re breaking up the functional problem we’re trying to solve and splitting it into smaller classes. While there is something to be said for this (but please people, within reason, don’t push it to the extreme), it has some serious drawbacks when you combine it with writing tests PER Class:

Your business logic is spread across multiple classes, so the tests that verify them are also scattered.

For example:

Input validation (IE: is a field of the right type?) is usually done where you map the request to your model. For example in the controllers.
Business rule validation is often done in a domain layer
With the exception of validation rules that are easier / better to do in the database. For example, duplicate constraints can very easily be handled in the database in a transactional fashion. Trying to do this in the business layer means you’re going to have
Some validations cannot (or shouldn’t) be done transactionally, but are actually part of an asynchronous business process.

In an ideal world, you can look at the tests to get a good understanding of the functionality and rules of the system. When each test only covers a single class, and each class only covers a small part of the functionality, it becomes really hard to get a good overview of the system.

I don’t have a big problem with SOLID when applied sanely. However, I’m finding that people really go overboard. Then couple that with going overboard with using mocks and you really set yourself up for trouble.

Anyway, I favor CUPID over SOLID any time.

Conclusion

In this post, I’ve highlighted the issues I have with the use of mocking. As this post has become too long already I’ll post this now and try to focus on what you can do to avoid mocking libraries in a future post.

But hey, don’t take my word for it:

Characteristics of well written tests

2021-11-15T08:47:57+00:00

We don’t write tests ‘just for fun’, but because we believe that it brings certain important benefits. Without (enough of) these benefits, it’s just overhead and useless ceremony. So the question is, what are the characteristics that make unit testing worth while? And what issues should we avoid?

When speaking to various people and reviewing a LOT of code, i’m finding that many people have different thoughts and idea’s about this. I’m quite passionate about automated testing, so I figured I’d jot down my own thoughts here.

I started writing a completely different blog post.. Then from there, this post started to form.. then it became too big. WAY to big. So I cut it down so I can at least get something out. Hopefully, I’ll be able to post the others soon as well.

In this post, i’m talking about automated tests, but they apply equally well to unit tests, integration tests, UI tests, etc..

Characteristics of well written tests.

A good test helps you by:

Bringing you increased understanding. It helps you to write the code, or when reading the tests to understand what the code does.
Protects you from mistakes during writing / changing / refactoring and bring you confidence in the correctness of the code.
Being resilient against internal changes. It should not fail if you refactor the insides of your test subjects.
Be easily changed when changes to api signatures occur.
Making software development faster, not slowing you down.

In my mind, if your tests do not have all of these characteristics, that’s a smell. But if they are missing more than one, they might be causing more harm than good.

Looking at the inverse of these properties can also be interesting. A test should not:

Slow me down when writing or maintaining code. If there’s too much overhead in writing / maintaining / running the tests, they don’t bring me any value. How you factor your code is important as well as at which level you start your tests.
Confuse me: If you can’t understand what a test does, how it does things or why it does things, it’s usually not valuable at all. A big reason for this can be the use of mocking frameworks (more) about that in a follow up post.
undermine my trust. If you can’t trust your tests, they are not very valuable. This can happen when your tests don’t catch mistakes made. Probably the coverage or assertions are not good enough to build trust. Another reason to undermine your trust is if they fail without good reason. The tests are too brittle then and, because I have to go in and modify my tests as well, i’m eroding the value that they bring. Flakey tests are the worst, when the tests occasionally fail, usually due to improper test isolation. Looking at you, static mutable state!

How to achieve these characteristics?

Achieving the characteristics above is not as simple. So, how can you achieve these characteristics? Each test should:

have a clear purpose. Sometimes this might mean test a single thing (even if the single thing is a complex business scenario). The name of the test is important, but so is the way you write it.
actually protect you from real issues. I’ve seen too many tests that don’t actually protect you from anything. I’ll probably write another blog post later about things not worth testing.
Draw attention to the things that are important in your test, while hiding the things which are not. Think deep and hard about what you want your test to express. What are the things that are different in this test from the other tests? This ties in again to it’s purpose. Hiding the unimportant ceremony in helpers (such as data builders, fixtures, helper methods, etc) has the benefit of both focussing attention on what IS important but also allowing you to reuse the logic.
Avoid static mutable state like the plague. Static mutable state is the thing that makes your test run fine when running them one at a time, but fail when you run them together, or on the build server.
Be written at the appropriate level of technical complexity. Business oriented components have different requirements than more technical components.
Clearly express the reason for failures. Tests will eventually fail (otherwise, what’s their purpose?). When a test fails, should be quickly clear why they fail, ideally without the need for debugging. I’ve often found printing log messages to the test output window is a very powerful method for this..

In a future post, I’ll try to write more about how I typically structure my tests and what things to avoid.

Conclusion

With these characteristics, I’ve often found my tests to be easy to understand (at least to me), easy to maintain. Hopefully, you find it useful.

Testing tricks

2021-11-15T08:47:57+00:00

In my previous post, I expressed some characteristics of well written tests. In this post, i’d like to show some techniques to help achieve these characteristics. In a future post, I’ll try to focus more on techniques and patterns to avoid.

An example test

I’m going to use an extremely simplified example to demonstrate some of the techniques here.

Here’s the example, including all the classes involved that i’m going to talk about in the next chapters:

In this example, we’re testing a system to create registrations for a party.

Have a look here for the code accompanying this post.

The example is a bit contrived but I found it a challenge to come up with something that is both useful to explain the topic, but small enough not to distract from the concepts. So, I’m only including a domain object and a store to save it. No http api, command handlers, query handlers, etc. Maybe I’ll enrich the example for future posts.

Fixtures and test helpers

To test some functionality, there is quite a bit of testing setup needed. Usually, you have to construct the Subject Under Test, including all of it’s dependencies. Then you have to create some data to use for testing. And then you have to assert the outcome. So many times do I see the subject under test being created in the test itself, including some or many of it’s dependencies (or, if you use mocks, set up the mocks in each test).

Duplicating the construction of your classes in each test case makes maintaining your tests much harder. When you need to change the dependencies of your test, all of a sudden you have to modify dozens of tests. Not fun at all!

I very much try to hide things that are not relevant for my tests in helper methods (if I can only re-use them in a single test) or in a separate fixture class. I find that this has several big benefits for me:

It draws attention in my test only to the important bits.
I build up a library of re-usable functionality, so future tests are much easier to write.
It reduces the amount of maintenance I have to perform.

Here’s an example of the fixture:

public class PartyFixture
{

    private readonly TestLoggerProvider LogProvider;

    public PartyFixture(ITestOutputHelper output)
    {
        LogProvider = new TestLoggerProvider(output);
    }

    public readonly InMemoryDocumentStore InMemoryDocumentStore =
        new InMemoryDocumentStore();

    public PartyStore PartyStore =>
        new PartyStore(documentStore: InMemoryDocumentStore,
                        logger: LogProvider.BuildLogger<PartyStore>());
}

The fixture creates instances of all classes needed, including all required dependencies, such as the PartyStore and the test double InMemoryDocumentStore. Should the number of dependencies change, you only have a single place to modify here.

The test creates the fixture in the constructor. This makes sure we don’t have any shared mutable state.

Test data and Test Data Builders

One of the techniques that I find most valuable in unit testing is the use of Test Data Builders. There are libraries to help you to build test data, such as AutoFixture, but I tend to create my test data by hand.

I make a distinction between Test Data, which is a collection of re-usable values that I can use during test data building, and real Test Data Builders, which create new instances of classes for me. The test data builders create new instances every time called so you can manipulate these instances where needed.

The instances created by the test data builder should be valid:

Most of the time, you need valid data to work with.
What is valid data evolves over time, as rules are added / removed. If you have to turn invalid data into valid data in every unit test, your tests become really brittle.
Turning valid data into invalid data usually is simple.

The pattern I’m following in my Tests is that I call the test data ‘The’, and the test data builder ‘Some’.

/// <summary>
/// The test data creates new instances of classes to test with. 
/// </summary>
public class TestDataBuilder
{
    public readonly TestData The;

    public TestDataBuilder(TestData the)
    {
        The = the;
    }

    /// <summary>
    /// Create a VALID instance of a party. It's easy to turn
    /// a valid instance in an invalid one but most tests actually work
    /// with valid data. So, start from a VALID instance and work
    /// from there. 
    /// </summary>
    /// <returns></returns>
    public Party.Party Party() => new Party.Party(
        Id: The.PartyId,
        Name: The.PartyName,
        Location: The.Location,
        Date: The.PartyDate,
        Attendees: The.PresentAttendees);
}

A couple of rules I’m following for test data builders.

Test Data builders create new instances only. You can then safely manipulate the instances as needed.
Test Data Builders should only create Test Data, such as commands, queries, events and other types of data holders. Don’t create other types of classes here, such as loggers, controllers, etc. Use fixtures for those.

public class TestData
{
    public PartyId PartyId = new PartyId(Guid.NewGuid());
    public PartyName PartyName = new PartyName(MyName());
    public Organizer Organizer = new(MyName());

    public Attendee AttendeePresentJohn = new(MyName());
    public Attendee AttendeePresentCatherine = new(MyName());
    public Attendee AttendeeAbsentAlice = new(MyName());
    public Attendee AttendeeAnnoyingAnton = new ( MyName());

    public IReadOnlyCollection<Attendee> PresentAttendees;

    public PartyDate PartyDate => new PartyDate(CurrentDateTime);

    public PartyLocation Location = new PartyLocation(new(MyName()));

    public TestData()
    {
        PresentAttendees = new[] { AttendeePresentJohn, AttendeePresentCatherine};
    }

    /// <summary>
    ///  Always control the current time in your unit
    /// tests. This way, you can easily time travel
    /// </summary>
    public DateTime CurrentDateTime =
        new DateTime(2000, 1, 2, 3, 4, 5);


    /// <summary>
    /// This method is a little trick. It returns the
    /// name of the calling property, but then in snake case
    /// and surrounded by hashtags. This makes it
    /// immediately clear when observing the data that the value
    /// actually IS the expected property, and no more and no less.
    /// </summary>
    /// <param name="attribute"></param>
    /// <returns></returns>
    private static string MyName(
        [CallerMemberName] string attribute = null)
    {
        return $"#{attribute.ToSnakeCase()}#";
    }
}
    

Test Data is mutable. This is an easy way to control what data is created by your test data builder.
Use the data created by the test data builder and test data in your assertions.

I’ve found that, while autofixture or Faker can sometimes generate more realistic test data, it’s not realism i’m looking for, it’s predictability. So, for properties where I really don’t care what the value is, i’m using a helper method that uses the name of the property to build a value. IE: AttendeePresentJohn => #attendee_present_john#

With syntax

Within your unit tests, you have the need to control the values that you’ve just built. In the party example, I would like to be able to change the name or the number of attendees. I find the with syntax very valuable for this.

[Fact]
public void One_person_does_not_make_a_fun_party()
{
    var party = Some.Party()
        with
        {
            Attendees = new[] {The.AttendeePresentJohn}
        };

    party.IsFun().Should().BeFalse();
}

This approach has several benefits:

It’s immediately clear what business rule i’m testing here. All the other properties (name, location, time, etc..) are completely irrelevant here. The only thing you see is the actual business rule.
I don’t need to repeat myself with all other properties. I’ve seen soo many times that the complete definition of a test data is repeated over and over again in each unit test. This makes adding a new property a real challenge.
Your test is very anti-fragile. If the definition of what other characteristics that make a fun party ever changes, then this test will keep working.

I sometimes see people create huge test data builders, with specialized extensions for each property they want to manipulate. The problem of this approach is that it requires a lot of coding to build this, whereas the language can just accomodate this .

I hope you can see how this pattern improves maintainability of tests.

With syntax before records

Before records came to c#, I used a simple extension method for this. It doesn’t work with immutable data unfortunately but the resulting syntax works just as well.

public static class ObjectWithExtensions
{
    public static T With<T>(this T subject, Action<T> doAction)
    {
        doAction(subject);
        return subject;
    }
}

var party = Some.Party()
    .With(x => x.Name = new PartyName("different"));

Assert big, not small

When it comes to asserting things in tests, I often see people writing long lines of assertions, where each property is asserted separately.

I very much recommend against this. While it seems very precise, the problem comes when you are maintaining your software. Adding a property will NOT break your tests, so developers often forget to add the property. You now have a blind spot in your test, which makes your tests less likely to catch issues.

When you have to change the definition of your class, you now also have to modify several tests, because each test has assertions for each property. If you have to change many tests to do a change, you are also undermining the reliability of your tests.

My recommendation is usually:

Either be very focussed on your assertions
OR assert against a known OK object (which is then created by a test data builder)

Now this is not always possible or practical (for example, when you have non-deterministically generated values in your code) but it’s a good thing to strive for.

Here’s a (little contrived) example:

    [Fact]
    public async Task Can_save_new_party()
    {
        var party = Some.Party();
        await PartyStore.Save(party);

        var saved = await PartyStore.Get(The.PartyId);

        saved.Should().BeEquivalentTo(party);
    }

I’m not asserting that the individual property values are valid. Instead, I’m asserting against an expected result object. In this case, it’s really simple, because the shape is the same between the input / output.

When the shape is not the same, this is where a the Test Data from the previous paragraph shines.

Two objects, created by different methods on the Test Data Builder will generate the same ‘logical’ output value.

non-deterministic values

Handling non-deterministically generated values is a bit of a challenge. Some assertion frameworks (such as fluent assertions or Verify) have methods to compensate for this.

Another approach is to see if you can design your software to make the generation of your data deterministically, either by using a deterministic data generator in the actual code, or by injecting deterministic generation for your tests only.

Controlling Datetime.Now in your tests is also a really important aspect (I’ll go more into that topic in a future blogpost) but if you control what now means to your code, it makes the output of your tests soo much more deterministic.

Logging in unit tests.

The last thing I’d like to address is logging in unit tests.

In .Net, the default way how you add logging is to add ILogger as a dependency in most of our classes. This approach has pro's and cons, which i'm not going to go into in this post. This does mean, that every class that wants to do logging, now requires an ILogger as dependency.

The way people mostly analyze problems in production is by adding logging to their system. Debugging in production is often not possible, so the logs are the only way you can analyze a problem.

If logs are very important to analyze a problem in production, why not use them to analyze problems in your unit tests? I often see people inject a NullLogger into their classes when testing. This is such a wasted opportunity. By writing the log output to a console (or in my case, the ITestOutputHelper from XUnit), I get a detailed log for everything that happened in my tests.

One of the least fun things is when you have a test that succeeds locally, but fails on the build server. Having detailed log output from your test can mean all the difference.

To get log output in my unit tests, I typically use a TestLoggerProvider, which simply pipes log output to XUnit ITestOutputHelper. Then I add a couple of tricks, such as, change the output format to be more helpful for me, such as printing the elapsed time, not the date time (which is not very valuable) or pretty printing exceptions with all properties.

The benefit of using XUnit ITestOutputHelper is that all output is correlated to a test. If you simply write using Console.Write and or a Static Logger, then your test output is not correlated with a specific test.

Conclusion

I’ve given several patterns that I’m using over and over again in my unit tests. I’ve found these practices to greatly improve the maintainability of my tests and improve the speed at which I can develop software. So I hope they help you as well.

Testing Small or Testing Big

2021-11-15T08:47:57+00:00

A challenge I find when designing my testing strategy is at what level should you test? Do you test each class individually or do you group classes together into components and test the api boundaries of these components? The challenge here is of course, the bigger the scope of your tests, the more meaningful (and I would argue valuable) they become, but also the more difficult certain edge cases are to test.

This seems to be a bit of a dichotomy. Test bigger or test smaller?

The thing is, don’t pick at all! Use the best style of tests but deviate from it for cases where it doesn’t. Be pragmatic!

I usually build LOB applications, which means that I need to pass lots of data around. Ingesting data in HTTP based or message based api’s, processing, validating and storing the data. Then producing forms of this data again, either as published events or on demand (http based queries).

Usually, I end up with several types of classes in my system:

Functional classes. These may include contracts (mapping to api’s, database schemas, event type definitions, etc) but also the classes that operate on them to ingest, manipulate, validate and store the data in the database.
Supporting or technical classes. These can be helpers to make programming easier, or a narrow technical solution (logging, caching, authentication, etc..).

Focussing on data flow for functional tests

For the functional classes, I try to focus on data flow through the system. That means, modeling the actions users take and to observe the side effects.

If you’re using CQRS, a great place to do more ‘functional’ style testing is at the command / query level. A command / query is a great way to express intent and observe side effects. Writing these tests using a BDD style pattern (Given-When-Then) also seems to be a great fit, but with a focus on observable outcomes.

(Given system in state X) When the user does X (ie: save some data), the system does Y (store the data), which you can observe by (result of action / performing queries / observing messages).

Ideally, I prepare the current state of my system using commands and I try to observe the outcome using queries. The reasoning is two fold:

I can more easily change the inner implementation without having to change my tests.
I can create re-usable testing components to create the commands. This means that as my commands evolve over time, my tests naturally evolve with them, resulting in more stable test suites.

So ideally, I will not observe that a row was added to a database (which couples your test to your specific database implementation), but by observing the thing the user can actually see (executing queries).

When building event sourced applications I usually ALSO write most of my business logic tests using the form: Given(Events), When (command), Then(Events). Combined with other test that listen for evens (Projections, reactors, etc.). The combination of both API based tests and more business logic tests absolutely kicks ass!

Now, some scenario’s are bigger than a single command / query, and those are usually a good place to do at the highest (api) level.

Testing Value objects

Value objects (in domain driven design) usually provide a specific set of functionality.

Some are so simple (just a wrapper around a value) that they don’t need any form of tests. By all means, don’t write tests for them then. They will be tested as part of the regular flow.

I usually don’t test constructors, properties, etc in isolation. An issue here will quickly be found as part of my more functional business flows. Tests that focus on constructors or individual properties are usually very mechanical and don’t provide any real value.

However, sometimes you have value objects that do provide specific functionality. If you have specialized formatting and parsing logic, specialized equality operations or specialized arithmetic operations, then writing tests for them makes a lot of sense.

But if you’re just relying on the c# record equality, parse using .net framework primitives (Guid.Parse, Datetime.Parse, etc), or no specialized aritmetic, what benefits do your tests bring? You’re probably just testing that the .Net framework still works.

Isolate technical classes from functional classes

There are a lot of classes that don’t really provide much functional benefit but do provide a good technical service.

Sometimes, these are helpers, that make repetitive code more maintainable, or they provide important non-functional benefits, such as authentication, caching, logging, etc.

It can be very useful to write good (old) unit tests focussing on one (or a small set) of classes for these technical supporting classes.

In the functional test cases, I do my best NOT to have to mock out these classes but just include them in my testing flow.

In fact, including techncal classes in your functional test cases can interesting scenario’s. IE: If you have caching enabled, can the user observe the changes you’ve just made or do you need some form of cache busting or checks for eventual consistency?

But, but, but.. I have some cases that are not easily tested with the examples above.

Great, we all do. If you have certain edge cases that are both imporant to test but also difficult to test at api / component levels, then these are a great candidate to test in isolation. Ideally, I’d write my classes to minimize the need to do any form of mocking, but sometimes you just don’t have a choice. In that case, go for it.

Not all edge cases are worth testing. Consider if the cost of writing / maintaining the test outweighs the benefits. If it does, great, do it! If it doesn’t, just skip it.

Conclusion

This heuristic has helped me a lot to write tests that are both maintainable, readable and very stable. I hope they help you as well.

Welcome

2021-11-10T08:47:57+00:00

Welcome to my new and revamped blog.

Many years ago, I used to occasionally blog. About things that I thought would be valuable for me to remember and perhaps would be valuable for other people as well.

Now I never liked the blogging platform I used to use (blogger.com) and that was a big barrier for me to keep doing it. Now, i’ve moved to a github pages based blog. Let’s see, hopefully i’ll be able to keep it up and write about things that might interest you as well.

Erwin’s Blog

A startup journey

Joining a new startup Smart Cloud

Testing without mocks - databases

TLDR;

The elephant in the room: Persistence

Heuristics on choosing an approach

Picking an approach

1. How much are you tied to the actual logic of the store?

2. Is there a good in-memory implementation?

3. How big / complex is the api surface?

Tips for Testing with in-memory implementations

Take control of eventual consistency

Testing your in-memory implementation

Tips for testing with hand-crafted test doubles

Tips for Testing with real db implementations

Dockerize your setup

Provide a proper level of isolation

Option 1: Use a database per test.

Option 2: Use a schema / namespace / table per test.

Option 3: Share a database but be careful

Not recommended techniques

Conclusion

Stop making a Mock

TLDR;

Background: I see mocks everywhere

This is the way

The problems with mocking libraries

Decreased maintainability due to Increased coupling

Refuctoring (having to change your tests when you refactor your solution)

Hard to understand test tests.

Single Responsibility Principle causes harm here

Conclusion

Characteristics of well written tests

Characteristics of well written tests.

How to achieve these characteristics?

Conclusion

Testing tricks

An example test

Fixtures and test helpers

Test data and Test Data Builders

With syntax

With syntax before records

Assert big, not small

non-deterministic values

Logging in unit tests.

Conclusion

Testing Small or Testing Big

Focussing on data flow for functional tests

Testing Value objects

Isolate technical classes from functional classes

But, but, but.. I have some cases that are not easily tested with the examples above.

Conclusion

Welcome