Skip to content

API vs Contract Testing - Research Summary

Executive Summary

You were correct in your suspicion that the current "Contract Tests" and "API Tests" are too similar. After analyzing the codebase and researching industry best practices, I've confirmed that:

The Problem: - Current *.Contract.Tests are NOT contract tests - they are duplicate integration tests - They test business logic, database operations, and complete workflows - This creates redundant test coverage and confusion about test purposes

The Solution: - Proper contract tests validate API specification compliance (schema, structure, types) - Integration tests validate business logic (workflows, persistence, side effects) - These serve different purposes and should not overlap

What is Contract Testing?

Contract testing uses Pact.io for consumer-driven verification. Instead of testing against a specification, consumers define what they expect from the API, and providers verify they meet those expectations. It focuses on the "contract" between services.

Contract Tests Focus On:

✅ HTTP status codes match specification ✅ Response schemas match defined structures
✅ Required fields are present ✅ Field types are correct (string, number, boolean, etc.) ✅ Authentication requirements (401/403 responses) ✅ Error response formats (RFC 7807 Problem Details)

Contract Tests Do NOT:

❌ Test business logic correctness ❌ Verify database operations ❌ Test complex workflows ❌ Validate data persistence ❌ Check side effects

Example Contract Test:

[Fact]
public async Task CreateUser_ValidRequest_ReturnsCorrectSchema()
{
    var request = new { Email = "test@example.com", DisplayName = "Test", Password = "Pass123!" };
    var response = await client.PostAsJsonAsync("/api/admin/users", request);

    // ✅ Contract validation
    response.StatusCode.ShouldBe(HttpStatusCode.Created);
    response.Headers.Location.ShouldNotBeNull();

    var json = await response.Content.ReadFromJsonAsync<JsonNode>();
    json["id"].ShouldNotBeNull();  // Required field exists
    json["id"].GetValue<Guid>().ShouldNotBe(Guid.Empty);  // Correct type
    json["email"].ShouldNotBeNull();  // Required field exists

    // ❌ Don't verify user exists in database (that's integration test)
    // ❌ Don't verify email matches request (that's business logic)
}

What is API Integration Testing?

Integration testing validates that the complete system works correctly, including business logic and data persistence.

Integration Tests Focus On:

✅ Complete CRUD workflows work end-to-end ✅ Business rules are enforced correctly ✅ Data persists correctly in database ✅ Complex multi-step operations succeed ✅ Side effects occur as expected ✅ Authentication/authorization logic works

Example Integration Test:

[Fact]
public async Task ResetPassword_AllowsLoginWithNewPassword()
{
    var (userId, email) = await CreateUserAsync();
    var newPassword = "NewPassword123!";

    await client.PostAsJsonAsync($"/api/admin/users/{userId}/reset-password", new { NewPassword = newPassword });

    // ✅ Integration test validates business logic
    await _fixture.ValidateUserCredentialsAsync(email, newPassword);
}

Current State Analysis

Example: Password Reset

Current "Contract" Test (INCORRECT):

// In OpenIdentityStack.Contract.Tests
[Fact]
public async Task ResetPassword_WithValidRequest_Returns200Ok()
{
    (Guid userId, string? email) = await this.CreateUserAsync();
    var request = new { NewPassword = "NewPassword123!" };

    var response = await this.SendRequestAsync(HttpMethod.Post, $"/api/admin/users/{userId}/reset-password", request);

    response.StatusCode.ShouldBe(HttpStatusCode.OK);

    // ❌ This is business logic validation, NOT contract testing!
    await this._fixture.ValidateUserCredentialsAsync(email, request.NewPassword);
}

What it SHOULD be (Contract Test):

// In OpenIdentityStack.Contract.Tests
[Fact]
public async Task ResetPassword_ValidRequest_Returns200WithCorrectSchema()
{
    var request = new { NewPassword = "NewPassword123!" };

    var response = await client.PostAsJsonAsync("/api/admin/users/{id}/reset-password", request);

    // ✅ Just verify contract (status, schema)
    (response.StatusCode == HttpStatusCode.OK || 
     response.StatusCode == HttpStatusCode.NotFound).ShouldBeTrue();
}

Corresponding Integration Test:

// In OpenIdentityStack.Api.Tests
[Fact]
public async Task ResetPassword_AllowsLoginWithNewPassword()
{
    var (userId, email) = await CreateUserAsync();
    var newPassword = "NewPassword123!";

    await client.PostAsJsonAsync($"/api/admin/users/{userId}/reset-password", new { NewPassword = newPassword });

    // ✅ Integration test validates business logic
    await _fixture.ValidateUserCredentialsAsync(email, newPassword);
}

Documentation Provided

I've created comprehensive documentation to address this issue:

  1. docs/TESTING-STRATEGY.md
  2. Complete guide to all test types
  3. When to use each type
  4. Examples of proper patterns
  5. Comparison tables

  6. tests/OpenIdentityStack.Contract.Tests/README.md

  7. What contract tests are
  8. How to write them properly
  9. Examples of correct patterns
  10. What NOT to test

  11. tests/OpenIdentityStack.Api.Tests/README.md

  12. What integration tests are
  13. Complete workflow testing
  14. Best practices and patterns

  15. docs/CONTRACT-TEST-MIGRATION-GUIDE.md

  16. Step-by-step migration process
  17. How to identify which tests belong where
  18. Before/after examples
  19. Common patterns

Implementation Plan

Phase 1: Infrastructure (✅ Complete)

  • [x] Add Pact.io package (PactNet v5.0.1)
  • [x] Create PactHelper utility class
  • [x] Create comprehensive documentation
  • [x] Create example tests
  1. Review each test in *.Contract.Tests
  2. Tests that validate business logic → Move to *.Api.Tests
  3. Tests that validate schema only → Keep in *.Contract.Tests and convert to Pact
  4. Add Pact contract definitions for consumer expectations
  5. Remove duplicate tests

Phase 3: Validation

  1. Run all tests to ensure nothing broken
  2. Review test coverage (shouldn't drop, just reorganize)
  3. Update team documentation

Benefits of Proper Contract Testing

  1. Catch Breaking Changes Early
  2. Contract tests fail when provider doesn't meet consumer expectations
  3. Prevents breaking consumers without knowing

  4. Consumer-Driven Development

  5. Contracts ensure API serves actual consumer needs
  6. Pact files document real usage patterns

  7. Bidirectional Verification

  8. Consumers define expectations (create contracts)
  9. Providers verify they meet all contracts

  10. Faster Feedback

  11. Contract tests use mock servers, run quickly
  12. Can run more frequently in CI/CD

  13. Better Test Organization

  14. Clear separation of concerns
  15. Easier to maintain and understand
  16. No duplicate coverage

  17. Independent Development

  18. Teams can work independently
  19. Pact Broker enables contract sharing and versioning

Comparison Table

Aspect Contract Tests Integration Tests
Purpose Validate consumer expectations Validate business logic
Approach Consumer-driven with Pact End-to-end testing
Database Not needed (mock server) Real database
Speed Fast (mocks) Slower (DB operations)
Focus Consumer-provider contract Complete workflows
Example "GET /users returns expected schema" "Created user can login"
Business Logic ❌ No ✅ Yes
Side Effects ❌ No ✅ Yes

Recommendations

Immediate Actions:

  1. Review the documentation - Understand Pact.io approach
  2. Try the examples - See consumer-driven contract tests in action
  3. Pilot migration - Pick one controller and convert its tests to Pact

Long-term Actions:

  1. Migrate all contract tests - Following the migration guide
  2. Add provider verification - Set up provider-side contract verification
  3. Update team practices - Ensure new tests follow Pact patterns
  4. CI/CD integration - Consider Pact Broker for contract management

Questions?

The documentation includes many examples and patterns. Key files to review:

  • docs/TESTING-STRATEGY.md - Start here for overview
  • tests/OpenIdentityStack.Contract.Tests/README.md - Contract test guide
  • docs/CONTRACT-TEST-MIGRATION-GUIDE.md - How to migrate existing tests
  • tests/OpenIdentityStack.Contract.Tests/Examples/UserContractExamples.cs - Working examples

Conclusion

You were absolutely right - the current "Contract Tests" are not contract tests at all. They're duplicate integration tests that create confusion and redundant coverage.

The solution is to: 1. ✅ Proper contract tests - Validate API specification compliance 2. ✅ Integration tests - Validate business logic and workflows 3. ✅ Clear separation - Each test type has a distinct purpose

All the infrastructure and documentation is now in place to make this migration successful.