Skip to content

Latest commit

 

History

History
305 lines (183 loc) · 10.9 KB

cm2m8psz3000409l78mt5903o.md

File metadata and controls

305 lines (183 loc) · 10.9 KB
Raw
title seoTitle seoDescription datePublished cuid slug cover ogImage
Schema-Driven Development and Single Source of Truth: Essential Practices for Modern Developers
Schema-Driven Development: A Modern Approach
Discover Schema-Driven Development and Single Source of Truth for efficient, error-free software development with automated APIs and documentation
Wed Oct 23 2024 19:01:11 GMT+0000 (Coordinated Universal Time)
cm2m8psz3000409l78mt5903o
schema-driven-development-and-single-source-of-truth-essential-practices-for-modern-developers

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1712331601187/039f3043-7b8a-4fb0-acc0-9630de9f6f4d.png?auto=compress,format&format=webp align="center")

💡
In the realm of software development, agility, consistency, and quality are more crucial than ever. As projects grow in complexity and teams scale, adhering to foundational best practices becomes essential. This article focuses on two critical paradigms: Schema-Driven Development (SDD) and the concept of a Single Source of Truth (SSOT). We'll explore how to derive CRUD APIs directly from SQL DDL database schemas, generate database documentation via DBML, and produce OpenAPI and JSON schemas—all contributing to a more efficient and error-free development process.

Best Practices

Before diving into SDD and SSOT, let's briefly outline the broader landscape of best practices that high-performing teams should follow:

  1. Schema-Driven Development & Single Source of Truth

  2. Configure Over Code

  3. Security & Compliance

  4. Decoupled (Modular) Architecture

  5. Shift Left Approach

  6. Essential Coding Practices

  7. Efficient SDLC: Issue management, documentation, test automation, code reviews, productivity measurement, source control, and version management

  8. Observability for Fast Resolution

What is Schema-Driven Development?

Schema-Driven Development is an approach where a single schema definition serves as the foundational blueprint for all aspects of an application. Instead of manually coding each component, the schema drives the generation of APIs, validations, documentation, and even test cases. This ensures consistency, reduces redundant effort, and minimizes the chances of errors.

Key Benefits of SDD

  • Unified Source of Truth: All teams and services refer to the same definitions, ensuring alignment.

  • Automated Generation: Reduces manual coding by auto-generating CRUD APIs, documentation, and client libraries.

  • Enhanced Parallel Development: Frontend and backend teams can work simultaneously, reducing bottlenecks.

  • Error Reduction: Automated validations prevent incorrect data from propagating through the system.

Signs Your Team Isn't Using SDD

  • Multiple, inconsistent schema definitions across services.

  • Manually crafted APIs, documentation, and test cases.

  • Sharing Postman collections via email rather than generating them automatically.

  • Increased bugs due to inconsistent data handling.

Understanding Single Source of Truth (SSOT)

A Single Source of Truth is the practice of structuring information models and associated schemata such that every data element is stored exactly once. In software development, this means all components—APIs, databases, services—derive their structure from a single schema, typically the database schema.

Advantages of SSOT

  • Data Consistency: Eliminates discrepancies caused by redundant data definitions.

  • Simplified Maintenance: Updates to the schema automatically reflect across all dependent components.

  • Improved Collaboration: Teams work from a shared understanding, reducing miscommunication.

  • Reduced Technical Debt: Consistent schemas prevent the accumulation of outdated or redundant code.

Practical Examples: Deriving from SQL DDL

Example 1: Generating CRUD APIs from SQL DDL

Scenario: You have an existing database schema defined using SQL Data Definition Language (DDL):

CREATE TABLE books (
  id INT PRIMARY KEY AUTO_INCREMENT,
  title VARCHAR(255) NOT NULL,
  author VARCHAR(255) NOT NULL,
  published_date DATE,
  isbn VARCHAR(13)
);

Using SDD, you can:

  1. Generate JSON Schemas:

    Use tools like sql-ddl-to-json-schema to convert your SQL DDL into JSON Schema definitions:

    {
  "title": "books",
  "type": "object",
  "properties": {
    "id": { "type": "integer" },
    "title": { "type": "string" },
    "author": { "type": "string" },
    "published_date": { "type": "string", "format": "date" },
    "isbn": { "type": "string" }
  },
  "required": ["title", "author"]
}

  2. Generate OpenAPI Schemas:

    Use the JSON Schemas to create OpenAPI definitions for your RESTful APIs.

    openapi: 3.1.0
info:
  title: Book API
  version: 1.0.0
paths:
  /books:
    get:
      summary: List all books
      responses:
        '200':
          description: A list of books.
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Book'
components:
  schemas:
    Book:
      $ref: 'book.schema.json'  # Reference to the generated JSON Schema

  3. Auto-Generate CRUD APIs:

    Use frameworks like LoopBack 4 or PostgREST that can generate RESTful APIs directly from your database schema.

    Using PostgREST:

    • Setup: Point PostgREST to your PostgreSQL database.

    • Result: Instantly get a fully functional REST API adhering to the OpenAPI spec.

  4. Generate Database Documentation via DBML:

    Convert your SQL DDL into Database Markup Language (DBML) to create interactive database diagrams and documentation.

    Example DBML:

    Table books {
  id int [pk, increment]
  title varchar
  author varchar
  published_date date
  isbn varchar
}

    Tools:

    • dbdiagram.io: Paste your DBML to visualize and generate documentation.

    • dbdocs.io: Generate and host database documentation online.

Benefit: Automates the creation of APIs and documentation, ensuring consistency and saving significant development time.

Example 2: Validating Data with JSON Schemas

Scenario: Before inserting or updating records in your database, you want to ensure the data conforms to your schema.

  1. Use JSON Schema Validators:

    In your API endpoints, validate incoming JSON payloads against the JSON Schema generated from your SQL DDL.

    const Ajv = require('ajv');
const ajv = new Ajv();
const validate = ajv.compile(bookJsonSchema);

app.post('/books', (req, res) => {
  const valid = validate(req.body);
  if (!valid) {
    return res.status(400).json({ errors: validate.errors });
  }
  // Proceed to insert data into the database
});

Benefit: Prevents invalid data from entering your system, reducing runtime errors and ensuring data integrity.

Example 3: Generating API Documentation

Scenario: You want to provide up-to-date API documentation for your team and third-party developers.

  1. Generate OpenAPI Documentation:

    • Use the OpenAPI schema generated from your database schema.

    • Utilize tools like Swagger UI or Redoc to render interactive documentation.

  2. Automate Updates:

    • Integrate the schema generation into your build pipeline.

    • Whenever the database schema changes, the OpenAPI docs update automatically.

Benefit: Ensures that your API documentation is always current and reflects the true state of your APIs.

Implementing SDD and SSOT in Your Organization

1. Start with Your SQL DDL as the SSOT

  • Ensure your database schema is well-defined and includes all necessary constraints and data types.

  • Use this schema as the foundation for generating all other components.

2. Automate Schema Conversion

  • Use tools to convert SQL DDL to JSON Schemas and OpenAPI specs.

  • Examples include ddl-to-json-schema or custom scripts using SQL parsing libraries.

3. Generate APIs Automatically

  • Option 1: Use frameworks like Hasura or Supabase for instant GraphQL and REST APIs over your database.

  • Option 2: Implement code generators that produce API endpoints based on your schemas.

4. Generate Documentation via DBML

  • Use tools like dbml-cli to convert SQL DDL to DBML.

  • Generate ER diagrams and host them using dbdiagram.io or similar services.

5. Integrate Into Your CI/CD Pipeline

  • Automate the generation of schemas, APIs, and documentation whenever changes are made to the database schema.

  • Ensure validation tests run against the updated schemas.

Challenges and How to Overcome Them

Initial Setup Overhead

Challenge: Setting up the automation pipeline requires initial effort.

Solution: Start with critical components and gradually expand. Leverage existing tools and community scripts to reduce development time.

Tooling Compatibility

Challenge: Ensuring all tools work seamlessly with your specific SQL dialect.

Solution: Verify tool compatibility or consider using intermediate formats like DBML, which supports multiple SQL dialects.

Managing Schema Changes

Challenge: Updating dependent services when the database schema changes.

Solution: Implement versioning for your APIs and schemas. Use migration tools like Flyway or Liquibase to manage database changes systematically.

Conclusion

Embracing Schema-Driven Development and establishing a Single Source of Truth by leveraging your SQL DDL can transform your development process. By automating the generation of APIs, validations, and documentation directly from your database schema, you ensure consistency, reduce errors, and accelerate development.

Ready to enhance your development workflow? Start by using your SQL DDL as the foundation and automate the generation of your APIs and documentation. Experience the efficiency and reliability that SDD and SSOT bring to your projects.