API vs Contract Testing - Research Summary¶
Executive Summary¶
You were correct in your suspicion that the current "Contract Tests" and "API Tests" are too similar. After analyzing the codebase and researching industry best practices, I've confirmed that:
The Problem:
- Current *.Contract.Tests are NOT contract tests - they are duplicate integration tests
- They test business logic, database operations, and complete workflows
- This creates redundant test coverage and confusion about test purposes
The Solution: - Proper contract tests validate API specification compliance (schema, structure, types) - Integration tests validate business logic (workflows, persistence, side effects) - These serve different purposes and should not overlap
What is Contract Testing?¶
Contract testing uses Pact.io for consumer-driven verification. Instead of testing against a specification, consumers define what they expect from the API, and providers verify they meet those expectations. It focuses on the "contract" between services.
Contract Tests Focus On:¶
✅ HTTP status codes match specification
✅ Response schemas match defined structures
✅ Required fields are present
✅ Field types are correct (string, number, boolean, etc.)
✅ Authentication requirements (401/403 responses)
✅ Error response formats (RFC 7807 Problem Details)
Contract Tests Do NOT:¶
❌ Test business logic correctness ❌ Verify database operations ❌ Test complex workflows ❌ Validate data persistence ❌ Check side effects
Example Contract Test:¶
[Fact]
public async Task CreateUser_ValidRequest_ReturnsCorrectSchema()
{
var request = new { Email = "test@example.com", DisplayName = "Test", Password = "Pass123!" };
var response = await client.PostAsJsonAsync("/api/admin/users", request);
// ✅ Contract validation
response.StatusCode.ShouldBe(HttpStatusCode.Created);
response.Headers.Location.ShouldNotBeNull();
var json = await response.Content.ReadFromJsonAsync<JsonNode>();
json["id"].ShouldNotBeNull(); // Required field exists
json["id"].GetValue<Guid>().ShouldNotBe(Guid.Empty); // Correct type
json["email"].ShouldNotBeNull(); // Required field exists
// ❌ Don't verify user exists in database (that's integration test)
// ❌ Don't verify email matches request (that's business logic)
}
What is API Integration Testing?¶
Integration testing validates that the complete system works correctly, including business logic and data persistence.
Integration Tests Focus On:¶
✅ Complete CRUD workflows work end-to-end ✅ Business rules are enforced correctly ✅ Data persists correctly in database ✅ Complex multi-step operations succeed ✅ Side effects occur as expected ✅ Authentication/authorization logic works
Example Integration Test:¶
[Fact]
public async Task ResetPassword_AllowsLoginWithNewPassword()
{
var (userId, email) = await CreateUserAsync();
var newPassword = "NewPassword123!";
await client.PostAsJsonAsync($"/api/admin/users/{userId}/reset-password", new { NewPassword = newPassword });
// ✅ Integration test validates business logic
await _fixture.ValidateUserCredentialsAsync(email, newPassword);
}
Current State Analysis¶
Example: Password Reset¶
Current "Contract" Test (INCORRECT):
// In OpenIdentityStack.Contract.Tests
[Fact]
public async Task ResetPassword_WithValidRequest_Returns200Ok()
{
(Guid userId, string? email) = await this.CreateUserAsync();
var request = new { NewPassword = "NewPassword123!" };
var response = await this.SendRequestAsync(HttpMethod.Post, $"/api/admin/users/{userId}/reset-password", request);
response.StatusCode.ShouldBe(HttpStatusCode.OK);
// ❌ This is business logic validation, NOT contract testing!
await this._fixture.ValidateUserCredentialsAsync(email, request.NewPassword);
}
What it SHOULD be (Contract Test):
// In OpenIdentityStack.Contract.Tests
[Fact]
public async Task ResetPassword_ValidRequest_Returns200WithCorrectSchema()
{
var request = new { NewPassword = "NewPassword123!" };
var response = await client.PostAsJsonAsync("/api/admin/users/{id}/reset-password", request);
// ✅ Just verify contract (status, schema)
(response.StatusCode == HttpStatusCode.OK ||
response.StatusCode == HttpStatusCode.NotFound).ShouldBeTrue();
}
Corresponding Integration Test:
// In OpenIdentityStack.Api.Tests
[Fact]
public async Task ResetPassword_AllowsLoginWithNewPassword()
{
var (userId, email) = await CreateUserAsync();
var newPassword = "NewPassword123!";
await client.PostAsJsonAsync($"/api/admin/users/{userId}/reset-password", new { NewPassword = newPassword });
// ✅ Integration test validates business logic
await _fixture.ValidateUserCredentialsAsync(email, newPassword);
}
Documentation Provided¶
I've created comprehensive documentation to address this issue:
- docs/TESTING-STRATEGY.md
- Complete guide to all test types
- When to use each type
- Examples of proper patterns
-
Comparison tables
-
tests/OpenIdentityStack.Contract.Tests/README.md
- What contract tests are
- How to write them properly
- Examples of correct patterns
-
What NOT to test
-
tests/OpenIdentityStack.Api.Tests/README.md
- What integration tests are
- Complete workflow testing
-
Best practices and patterns
-
docs/CONTRACT-TEST-MIGRATION-GUIDE.md
- Step-by-step migration process
- How to identify which tests belong where
- Before/after examples
- Common patterns
Implementation Plan¶
Phase 1: Infrastructure (✅ Complete)¶
- [x] Add Pact.io package (PactNet v5.0.1)
- [x] Create PactHelper utility class
- [x] Create comprehensive documentation
- [x] Create example tests
Phase 2: Refactoring (Recommended Next Steps)¶
- Review each test in
*.Contract.Tests - Tests that validate business logic → Move to
*.Api.Tests - Tests that validate schema only → Keep in
*.Contract.Testsand convert to Pact - Add Pact contract definitions for consumer expectations
- Remove duplicate tests
Phase 3: Validation¶
- Run all tests to ensure nothing broken
- Review test coverage (shouldn't drop, just reorganize)
- Update team documentation
Benefits of Proper Contract Testing¶
- Catch Breaking Changes Early
- Contract tests fail when provider doesn't meet consumer expectations
-
Prevents breaking consumers without knowing
-
Consumer-Driven Development
- Contracts ensure API serves actual consumer needs
-
Pact files document real usage patterns
-
Bidirectional Verification
- Consumers define expectations (create contracts)
-
Providers verify they meet all contracts
-
Faster Feedback
- Contract tests use mock servers, run quickly
-
Can run more frequently in CI/CD
-
Better Test Organization
- Clear separation of concerns
- Easier to maintain and understand
-
No duplicate coverage
-
Independent Development
- Teams can work independently
- Pact Broker enables contract sharing and versioning
Comparison Table¶
| Aspect | Contract Tests | Integration Tests |
|---|---|---|
| Purpose | Validate consumer expectations | Validate business logic |
| Approach | Consumer-driven with Pact | End-to-end testing |
| Database | Not needed (mock server) | Real database |
| Speed | Fast (mocks) | Slower (DB operations) |
| Focus | Consumer-provider contract | Complete workflows |
| Example | "GET /users returns expected schema" | "Created user can login" |
| Business Logic | ❌ No | ✅ Yes |
| Side Effects | ❌ No | ✅ Yes |
Recommendations¶
Immediate Actions:¶
- Review the documentation - Understand Pact.io approach
- Try the examples - See consumer-driven contract tests in action
- Pilot migration - Pick one controller and convert its tests to Pact
Long-term Actions:¶
- Migrate all contract tests - Following the migration guide
- Add provider verification - Set up provider-side contract verification
- Update team practices - Ensure new tests follow Pact patterns
- CI/CD integration - Consider Pact Broker for contract management
Questions?¶
The documentation includes many examples and patterns. Key files to review:
docs/TESTING-STRATEGY.md- Start here for overviewtests/OpenIdentityStack.Contract.Tests/README.md- Contract test guidedocs/CONTRACT-TEST-MIGRATION-GUIDE.md- How to migrate existing teststests/OpenIdentityStack.Contract.Tests/Examples/UserContractExamples.cs- Working examples
Conclusion¶
You were absolutely right - the current "Contract Tests" are not contract tests at all. They're duplicate integration tests that create confusion and redundant coverage.
The solution is to: 1. ✅ Proper contract tests - Validate API specification compliance 2. ✅ Integration tests - Validate business logic and workflows 3. ✅ Clear separation - Each test type has a distinct purpose
All the infrastructure and documentation is now in place to make this migration successful.