T E N E R i S
  • Projects
    • Transmission Layer Governance
    • TamaDB
      • About TamaDB
      • TamaDB User Manual
      • TamaDB API Catalogue
    • PamaDB
      • About PamaDB
  • About Me
  • Contact
  • External Links

Transmission Layer Governance

This document defines the principles, guardrails, and boundaries that govern the capabilities of the transmission layer within the smart-endpoints/dumb-pipes data architecture. Any system operating in this layer — including TamaDB and PamaDB — must conform to these governance rules.

What the Transmission Layer Is

The transmission layer is a purpose-built intermediary responsible for the reliable, validated, and timely exchange of structured data between producers and consumers. It occupies the "pipe" in a smart-endpoints/dumb-pipes architecture.

Unconditional Responsibilities

ResponsibilityDescription
ReceptionAccept data from producers via defined ingestion channels
Identity ResolutionDetermine the unique identity of each document from its content or context
State ManagementMaintain the current merged state of each document with per-field versioning
DeliveryPush document state to registered consumers via configured channels
ObservabilityProvide structured logging, quarantine, and inspection capabilities

Non-Responsibilities

The transmission layer does NOT:

  • Apply business logic or decision rules
  • Generate derived or computed values
  • Transform data semantics (change meaning)
  • Store data permanently (archive)
  • Aggregate data across documents
  • Authenticate end-users
  • Orchestrate multi-step workflows

The Semantic Preservation Principle

The transmission layer shall preserve the semantic equivalence of every attribute it transports. For each individual attribute that enters and exits the layer, the meaning at exit must equal the meaning at entry.

No Signification Loss

Data that enters the transmission layer must be recoverable from what exits it. Prohibited: truncation, rounding, field dropping without explicit consumer consent, lossy encoding changes, or any operation where the original value cannot be reconstructed from the output.

No Signification Creation

The transmission layer does not generate new meaning. Business rule evaluation, derived fields, status inference, aggregation, and prediction belong outside the transmission layer.

Data Ownership Model

The architecture defines four roles along two axes:

RoleLayerAuthority
ProducerSmart endpointAssigner/Owner — originates data, defines what is transmitted
SenderDumb pipeCustodian — executes the producer's instructions
ReceiverDumb pipeCustodian — executes the consumer's instructions
ConsumerSmart endpointAssignee — consumes, interprets, and acts upon data

The transmission layer is a custodian, not an owner: it holds data temporarily, does not claim rights over the content, does not alter its meaning, and discards it when TTL expires.

Directional Constraints

Inbound (Receiving Bay)

  • Assembly permitted — enrichment may add producer-delegated fields
  • Validation enforced — schema compliance is mandatory (DCM/HDOM)
  • Document significance may grow
  • Payload-supplied fields are preserved

Hold (State Management)

  • No content alteration — the Hold never inspects, interprets, or modifies attribute values
  • Operations are structural only (merge, version, expire)
  • Strongest expression of the Semantic Preservation Principle

Outbound (Dispatch Bay)

  • Reduction permitted — consumer-directed filtering may omit fields
  • Translation permitted — consumer-directed representation adaptation
  • Document significance may shrink, must never grow
  • No attributes added that weren't in the stored document

Structural Model

Structural model showing three levels of the Transmission Layer: Receiving Bay (inbound, producer-instructed) processes Parse, Enrich, Extract D-Id, and Validate. Hold (state, custodial only) processes Merge, Version, History, TTL registration, and Lifecycle. Dispatch Bay (outbound, consumer-instructed) processes Match subscriptions, Format payload, Shape, and Enqueue delivery.

Guardrails for Capability Evaluation

When evaluating whether a proposed capability belongs in the transmission layer:

#QuestionIf YES →
1Does it create new meaning from existing data?Does NOT belong
2Does it apply business rules or domain logic?Does NOT belong
3Does it lose attribute-level signification without consumer consent?Violates Semantic Preservation
4Does it alter payload content without producer delegation?Violates data ownership model
5Does it require knowledge of what the data means?Likely does NOT belong
6Does it serve reliable, validated transmission of data?Likely BELONGS

Realizations

SystemRoleStatus
TamaDBStateful document cache with validation, versioning, and push delivery (all three bays)Active development
PamaDBPublishing layer with representation adaptation (Dispatch Bay extension)Planned

About TamaDB

What Problem Does TamaDB Solve?

In modern data architectures, systems need to exchange structured data between producers (sensors, applications, services) and consumers (dashboards, analytics, downstream services) with minimal latency and maximum reliability.

TamaDB is a document-centric transmission cache for smart-endpoints/dumb-pipes architectures — a purpose-built, high-performance intermediary that:

  • Receives JSON documents from multiple producers via dedicated channels
  • Validates documents against registered JSON schemas
  • Stores temporarily with configurable time-to-live (TTL)
  • Pushes updates to registered consumers in real-time via webhooks

TamaDB is not a general-purpose database. It is intentionally limited in scope: no complex queries, no joins, no aggregations. Its strength is speed, simplicity, and reliable data delivery.

TamaDB vs Alternatives

Alternative What TamaDB Does Differently
Redis Redis is a key-value store. TamaDB provides leaf-level versioning, merge semantics, schema validation, structured webhook delivery, and per-field history.
MongoDB MongoDB is a general-purpose document DB. TamaDB is a purpose-built transmission cache with per-leaf versioning, merge-only upsert (fields are never lost), and push-first webhook delivery. Schema-strict, short-lived, not designed for long-term storage or complex queries.
Kafka Kafka is a distributed log delivering raw events in order. TamaDB delivers the current merged state of a document to subscribers. They're complementary, not competing.
MQTT MQTT delivers messages. TamaDB receives messages, validates against a schema, merges into a versioned document state, and delivers the merged result. TamaDB adds structure and state to raw message passing.

Architectural Position

TamaDB and PamaDB together form the Transmission Layer — the intermediary between producers and consumers:

Architecture diagram showing the Transmission Layer: Producers send data to TamaDB, which can deliver directly to Consumers. PamaDB extends TamaDB for transformed delivery to Consumers.

How the Transmission Layer works:

  • Producers send data into the Transmission Layer via TamaDB (the entry point)
  • TamaDB validates, versions, stores, and can deliver directly to consumers
  • PamaDB extends TamaDB for delivery requirements outside TamaDB's scope (transformations, frozen documents, publishing)
  • Consumers receive data from either TamaDB or PamaDB, depending on the delivery criteria
  • Delivery criteria can be defined by either the producer or the consumer

Key principle: Dumb-pipe role. TamaDB never alters document content. It validates, stores, versions, and delivers — nothing more. PamaDB handles any transformation needs.

Core Concepts

Document

A JSON object stored in TamaDB. Documents are structured as trees of Bunches (objects) and Leaves (terminal values). Each leaf is independently versioned and has its own history.

Document_Type

The central organising concept. Every document belongs to exactly one Document_Type, which defines: the Master_Schema (what fields are valid and required), the document model (how strictly the schema is enforced), Channels (ingestion endpoints), TTL (document lifespan), and Subscriptions (notification targets).

Channels

An ingestion endpoint tied to a Document_Type. Producers push data to a channel URL. Each channel can have its own Channel_Schema (a subset of the Master_Schema), enrichment bindings (TamaDB injects fields the producer doesn't send), and status control (active, inactive, or blocked).

Upsert & Merge Semantics

TamaDB has a single ingestion operation — upsert. There is no separate "create" vs "update". If the document doesn't exist, it's created. If it exists, incoming fields are merged into the stored document. Fields present in the payload update (previous value goes to history). Fields absent from the payload are preserved unchanged. No data is ever removed by a store operation.

Subscriptions

Consumers register webhooks (callback URLs) and receive real-time updates. Subscriptions can be at the Document_Type level (fires for any document of that type) or D-Id level (fires for a specific document only). Payload is snapshotted at the moment of update — delivery is decoupled from the store.

Document TTL

Every document has a time-to-live. TamaDB is a cache, not an archive. TTL hierarchy: Document > Document_Type > Instance (most specific wins). Expired documents are soft-deleted. TTL=0 means "process and deliver, but don't cache" (transient).

Schema Validation

TamaDB validates documents on ingestion using JSON Schema (Draft 7). Required fields must be present, field types must match, and unknown fields are rejected under the Document Closed Model (DCM). Validation happens on the Enriched_Document (payload + channel-supplied fields).

Document Models

TamaDB's document models map to the industry-standard distinction between defensive and offensive data strategies. Defensive strategies prioritise data integrity, compliance, and quality control. Offensive strategies prioritise flexibility, experimentation, and speed of onboarding new data sources.

Model Increment Schema Required Unknown Fields Data Strategy Use Case
DCM (Document Closed Model) 1 Yes Rejected Defensive Strict data contracts, compliance, data quality
HDOM (Hybrid Document Open Model) 2 Yes Allowed as extensions Balanced Evolving schemas — defend core, enable exploration
DOM (Document Open Model) 3 Optional Allowed Offensive Flexible onboarding, experimentation, agility

Regardless of model, the Semantic Preservation Principle applies: TamaDB never alters or creates meaning. The flexibility is in what it accepts, not in what it does to the data.

End-to-End Data Flow

Sequence diagram showing the end-to-end data flow: Producer sends POST to TamaDB, which processes through 9 pipeline steps (parse, enrich, extract D-Id, validate, merge, version, TTL, match subscriptions, snapshot), returns 201 to Producer, then delivers webhook POST to Consumer who responds 200 OK.

Increment 1 — What's Delivered

Capabilities

Area What's Included
IngestionREST endpoint per Document_Type and per channel, upsert-merge semantics
ValidationDCM schema enforcement, required fields, type checking, conditional rules
IdentityD-Id extraction from document fields (identityKind annotations), single and composed keys
VersioningPer-leaf history with configurable depth, optimistic concurrency control
ChannelsRegistration, status management (active/inactive/blocked), schema assignment
EnrichmentStatic field injection per channel, conflict detection + policy resolution
DeliveryWebhook subscriptions (Document_Type-level and D-Id-level), envelope/raw format
TTLThree-scope hierarchy, background sweep, on-access check, TTL=0 transient processing
QuarantineRejected payload storage with policy control, read-only inspection API
StorageMemory, disk, hybrid, and auto strategies with runtime switching and drain
PersistenceWrite-through state persistence — schemas, Document_Types, channels, subscriptions, enrichment, TTL, quarantine, and config survive restarts
ObservabilityStructured JSON logging, runtime-configurable levels and destination
AdminPlatform-independent graceful shutdown endpoint, health check
APIFull REST/HTTP API (~46 endpoints) for all operations and configuration

Technology Stack

Layer Technology
RuntimeNode.js 22 (ESM)
LanguageTypeScript 6 (strict mode)
HTTP FrameworkFastify 5
Schema ValidationAjv 8 (JSON Schema Draft 7)
LoggingPino 10 (structured JSON)
TestingVitest 4 + fast-check 4 (property-based)

Why TypeScript for Increment 1?

TypeScript was chosen deliberately for this phase — not as the final deployment language, but as the fastest path to proving the architecture works.

  • Rapid concept validation. The goal of Increment 1 is to prove the pipeline model, merge semantics, schema validation, and API contract are correct. TypeScript's expressiveness and fast iteration cycle (no compile-link-run delays) made this possible rapidly.
  • Language-agnostic design. The architecture is defined at the interface/contract level. Component interfaces, data flow, and the 9-step processing pipeline are not coupled to TypeScript — they translate directly to traits (Rust), interfaces (Java/Go), or protocols (Swift).
  • Living reference implementation. The TypeScript codebase serves as a verifiable proof that the design is complete. With 467+ tests (including property-based tests), it validates correctness properties that any future implementation must also satisfy.
  • Planned Rust rewrite for production. For quasi-production deployment (running 24/7 parallel to production systems), a Rust rewrite is planned. Rust eliminates Garbage Collection pauses, guarantees predictable latency, prevents memory leaks via ownership, and produces a single static binary with no runtime dependencies.

In short: TypeScript proves the what and how. Rust will deliver the at-scale and in-production.

Roadmap

Increment 2

  • HDOM (Hybrid Document Open Model) — schema validation for known fields, extensions allowed
  • Discrete leaf updates — update individual fields by path without sending full document
  • Foreign D-Ids (full) — dynamic/rule-based identity derivation
  • Library Sections — category-based document grouping
  • Subscription updates — in-place modification of active subscriptions
  • Schema adoption — propagate mode auto-adopts latest Retrieval_Schema version

Increment 3

  • DOM (Document Open Model) — no schema enforcement, fully dynamic
  • Alien D-Ids — documents with no identity (immutable, non-updatable one-shots)
  • Element-level arrays — address individual array elements (sparse transmission)

Future Vision

  • PamaDB — data publishing layer (transformations, frozen documents, hard subscriptions)
  • Sanitation — validate stored documents against template schemas
  • Multi-protocol — gRPC, WebSocket ingestion alongside REST
  • Idempotent updates — detect and skip duplicate payloads
  • Storage segmentation — per-Document_Type data isolation

TamaDB User Manual

Getting Started

Prerequisites

  • Node.js 22 or later
  • npm (comes with Node.js)

Installation & Startup

# Clone the repository
git clone https://github.com/teneriscc/TamaDB.git
cd TamaDB

# Install dependencies
npm ci

# Build
npm run build

# Start the server
npm start
# Or for development (auto-reload):
npm run dev

Run with Docker

A prebuilt, multi-architecture image (linux/amd64 and linux/arm64) is published on Docker Hub, so you can run TamaDB without installing Node.js:

docker run --rm -p 3000:3000 teneriscc/tamadb:0.1.0

For persistent volumes, Docker Compose, and the full environment-variable reference, see the Docker Hub overview.

TamaDB starts on port 3000 by default. Verify with:

curl http://localhost:3000/health

Response:

{
  "status": "ok",
  "version": "0.1.0",
  "storageMode": "auto",
  "logLevel": "debug"
}

Configuration (Environment Variables)

VariableDefaultDescription
TAMADB_PORT3000HTTP port
TAMADB_HOST0.0.0.0Bind address
TAMADB_LOG_LEVELdebugLog level (debug/info/warn/error/silent)
TAMADB_STORAGE_MODEautoStorage strategy (auto/memory/disk/hybrid)

Step 1: Register a Schema

Before storing documents, define the structure they must follow.

curl -X POST http://localhost:3000/api/v1/schemas \
  -H "Content-Type: application/json" \
  -d '{
    "ref": "temperature-sensor-v1",
    "schema": {
      "$schema": "http://json-schema.org/draft-07/schema#",
      "type": "object",
      "properties": {
        "sensorId": { "type": "string", "identityKind": "primary-key" },
        "location": { "type": "string" },
        "temperature": { "type": "number" },
        "timestamp": { "type": "string" }
      },
      "required": ["sensorId", "location", "temperature"]
    }
  }'

Response:

{ "ref": "temperature-sensor-v1", "version": 1 }

Key points:

  • identityKind: "primary-key" tells TamaDB which field(s) form the document's identity
  • required fields must be present in every ingested document (or supplied by enrichment)

Retrieve a schema

curl http://localhost:3000/api/v1/schemas/temperature-sensor-v1

Update a schema

curl -X PUT http://localhost:3000/api/v1/schemas/temperature-sensor-v1 \
  -H "Content-Type: application/json" \
  -d '{
    "schema": {
      "$schema": "http://json-schema.org/draft-07/schema#",
      "type": "object",
      "properties": {
        "sensorId": { "type": "string", "identityKind": "primary-key" },
        "location": { "type": "string" },
        "temperature": { "type": "number" },
        "humidity": { "type": "number" },
        "timestamp": { "type": "string" }
      },
      "required": ["sensorId", "location", "temperature"]
    }
  }'

View schema version history

curl http://localhost:3000/api/v1/schemas/temperature-sensor-v1/versions

Decommission a schema

curl -X POST http://localhost:3000/api/v1/schemas/temperature-sensor-v1/decommission

Reactivate a schema

curl -X POST http://localhost:3000/api/v1/schemas/temperature-sensor-v1/reactivate

Step 2: Register a Document_Type

Group documents under a named type with its Master_Schema.

curl -X POST http://localhost:3000/api/v1/config/document-types \
  -H "Content-Type: application/json" \
  -d '{
    "name": "temperature-readings",
    "masterSchema": "temperature-sensor-v1",
    "model": "closed",
    "ttl": 3600
  }'

Response:

{
  "name": "temperature-readings",
  "masterSchema": "temperature-sensor-v1",
  "model": "closed",
  "ttl": 3600,
  "channels": [],
  "createdAt": "2026-07-17T10:00:00.000Z",
  "updatedAt": "2026-07-17T10:00:00.000Z"
}

Parameters:

  • model: "closed" — strict validation, no unknown fields allowed
  • ttl: 3600 — documents expire after 1 hour

Get a Document_Type configuration

curl http://localhost:3000/api/v1/config/document-types/temperature-readings

Set per-Document_Type TTL

curl -X PUT http://localhost:3000/api/v1/config/document-types/temperature-readings/ttl \
  -H "Content-Type: application/json" \
  -d '{ "ttl": 7200 }'

Delete a Document_Type

Requires no documents of that type exist (including soft-deleted):

curl -X DELETE http://localhost:3000/api/v1/config/document-types/temperature-readings

Step 3: Register a Channel (Optional)

Channels give producers their own ingestion endpoint, optionally with enrichment.

curl -X POST http://localhost:3000/api/v1/config/channels \
  -H "Content-Type: application/json" \
  -d '{
    "channelId": "garage-sensor",
    "documentType": "temperature-readings"
  }'

Add Enrichment Bindings

Enrichment injects fields the producer doesn't need to send:

curl -X PUT http://localhost:3000/api/v1/config/channels/garage-sensor/enrichment \
  -H "Content-Type: application/json" \
  -d '{
    "bindings": [
      { "fieldPath": "location", "staticValue": "Garage" },
      { "fieldPath": "sensorId", "staticValue": "GARAGE-TEMP-01" }
    ]
  }'

Now the producer only needs to send {"temperature": 22.5} — TamaDB fills in the rest.

Get enrichment bindings

curl http://localhost:3000/api/v1/config/channels/garage-sensor/enrichment

Remove enrichment bindings

curl -X DELETE http://localhost:3000/api/v1/config/channels/garage-sensor/enrichment

Update a channel

curl -X PUT http://localhost:3000/api/v1/config/channels/garage-sensor \
  -H "Content-Type: application/json" \
  -d '{ "channelSchema": "temperature-sensor-v1" }'

Deactivate a channel

curl -X POST http://localhost:3000/api/v1/config/channels/garage-sensor/deactivate

Delete a channel

curl -X DELETE http://localhost:3000/api/v1/config/channels/garage-sensor

Step 4: Ingest Documents

Direct ingestion (Document_Type-level)

curl -X POST http://localhost:3000/api/v1/ingest/temperature-readings \
  -H "Content-Type: application/json" \
  -d '{
    "document": {
      "sensorId": "LIVING-ROOM-01",
      "location": "Living Room",
      "temperature": 21.5,
      "timestamp": "2026-07-17T10:05:00Z"
    }
  }'

Response (201 Created):

{
  "dId": "LIVING-ROOM-01",
  "versionNo": 1,
  "created": true
}

Channel ingestion (with enrichment)

curl -X POST http://localhost:3000/api/v1/ingest/temperature-readings/garage-sensor \
  -H "Content-Type: application/json" \
  -d '{
    "document": {
      "temperature": 18.2,
      "timestamp": "2026-07-17T10:06:00Z"
    }
  }'

Response (201 Created):

{
  "dId": "GARAGE-TEMP-01",
  "versionNo": 1,
  "created": true
}

TamaDB enriched the document with sensorId and location from the channel bindings before storing.

Update an existing document (upsert)

Send the same sensorId again — TamaDB merges the new temperature into the existing document:

curl -X POST http://localhost:3000/api/v1/ingest/temperature-readings/garage-sensor \
  -H "Content-Type: application/json" \
  -d '{
    "document": {
      "temperature": 19.1,
      "timestamp": "2026-07-17T10:10:00Z"
    }
  }'

Response (200 OK — updated, not created):

{
  "dId": "GARAGE-TEMP-01",
  "versionNo": 2,
  "created": false
}

Step 5: Retrieve Documents

List all Document_Types

curl http://localhost:3000/api/v1/config/document-types

List all documents in a Document_Type

# Use glob wildcard * to match all D-Ids
curl http://localhost:3000/api/v1/documents/temperature-readings/*

Response:

{
  "matches": [
    { "sensorId": "LIVING-ROOM-01", "location": "Living Room", "temperature": 21.5, ... },
    { "sensorId": "GARAGE-TEMP-01", "location": "Garage", "temperature": 19.1, ... }
  ],
  "count": 2
}

Full document by D-Id

curl http://localhost:3000/api/v1/documents/temperature-readings/GARAGE-TEMP-01

Response:

{
  "sensorId": "GARAGE-TEMP-01",
  "location": "Garage",
  "temperature": 19.1,
  "timestamp": "2026-07-17T10:10:00Z"
}

Retrieve a specific field

curl "http://localhost:3000/api/v1/documents/temperature-readings/GARAGE-TEMP-01?path=temperature"

Response:

{ "_value": 19.1 }

Search by value

curl -X POST http://localhost:3000/api/v1/documents/temperature-readings/search \
  -H "Content-Type: application/json" \
  -d '{ "field": "location", "value": "Garage" }'

Response:

{
  "matches": [
    { "sensorId": "GARAGE-TEMP-01", "location": "Garage", "temperature": 19.1, "timestamp": "2026-07-17T10:10:00Z" }
  ],
  "count": 1
}

Glob pattern matching

# All sensors starting with "GARAGE"
curl http://localhost:3000/api/v1/documents/temperature-readings/GARAGE-*

Step 6: Subscribe to Updates

Register a webhook to receive updates in real-time:

curl -X POST http://localhost:3000/api/v1/subscriptions \
  -H "Content-Type: application/json" \
  -d '{
    "scope": "document_type",
    "documentType": "temperature-readings",
    "callbackUrl": "https://my-dashboard.example.com/webhook/temperatures",
    "options": {
      "deliveryMode": "retry",
      "payloadFormat": "envelope"
    }
  }'

Response (201):

{
  "id": "sub-abc123-0001",
  "scope": "document_type",
  "documentType": "temperature-readings",
  "callbackUrl": "https://my-dashboard.example.com/webhook/temperatures",
  "options": { "deliveryMode": "retry", "payloadFormat": "envelope" },
  "active": true,
  "createdAt": "2026-07-17T10:15:00Z"
}

Every time a temperature reading is stored or updated, TamaDB POSTs to your callback:

{
  "subscriptionId": "sub-abc123-0001",
  "dId": "GARAGE-TEMP-01",
  "documentType": "temperature-readings",
  "document": {
    "sensorId": "GARAGE-TEMP-01",
    "location": "Garage",
    "temperature": 19.1
  },
  "timestamp": "2026-07-17T10:10:00Z"
}

Subscribe to a specific document

curl -X POST http://localhost:3000/api/v1/subscriptions \
  -H "Content-Type: application/json" \
  -d '{
    "scope": "document_id",
    "documentType": "temperature-readings",
    "dId": "LIVING-ROOM-01",
    "callbackUrl": "https://alerts.example.com/living-room"
  }'

List subscriptions

# All subscriptions for a Document_Type
curl http://localhost:3000/api/v1/document-types/temperature-readings/subscriptions

# Subscriptions for a specific document
curl http://localhost:3000/api/v1/documents/temperature-readings/GARAGE-TEMP-01/subscriptions

Cancel a subscription

curl -X DELETE http://localhost:3000/api/v1/subscriptions/sub-abc123-0001

Step 7: Delete a Document

Soft-delete permanently excludes the document from retrieval and delivery:

curl -X DELETE http://localhost:3000/api/v1/documents/temperature-readings/LIVING-ROOM-01

Response:

{ "deleted": true }

The document's data and history are preserved in storage for audit, but it will never be served again.

Step 8: Optimistic Concurrency

Prevent accidental overwrites by supplying the expected version:

curl -X POST http://localhost:3000/api/v1/ingest/temperature-readings \
  -H "Content-Type: application/json" \
  -d '{
    "document": { "sensorId": "LIVING-ROOM-01", "location": "Living Room", "temperature": 23 },
    "versionNo": 2
  }'

If the document's current version is not 2, TamaDB rejects with:

{
  "code": "VERSION_CONFLICT",
  "message": "Version conflict: expected 2, current is 3"
}

Omit versionNo for a "blind write" (last-write-wins).

Configuration at Runtime

All configuration is changeable without restart via the API.

CategoryValid ValuesDefault
Log Leveldebug, info, warn, error, silentdebug
Log Destinationstdout, file, nonestdout
Storage Strategyauto, memory, disk, hybridauto
TTL (Instance-level)Integer (seconds), 0 = transient, or "infinite"3600
Quarantine Policyoff, data-errors-only, all-rejectionsdata-errors-only
Enrichment Precedenceenvironment-over-content, content-over-environmentenvironment-over-content
Enrichment Conflict Resolutionaccept-and-warn, discard-and-erroraccept-and-warn

Log Level

# Check current
curl http://localhost:3000/api/v1/config/log-level

# Change to warn
curl -X PUT http://localhost:3000/api/v1/config/log-level \
  -H "Content-Type: application/json" \
  -d '{ "level": "warn" }'

Log Destination

# Check current
curl http://localhost:3000/api/v1/config/log-destination

# Switch to file
curl -X PUT http://localhost:3000/api/v1/config/log-destination \
  -H "Content-Type: application/json" \
  -d '{ "destination": "file", "filePath": "./tamadb.log" }'

# Switch back to stdout
curl -X PUT http://localhost:3000/api/v1/config/log-destination \
  -H "Content-Type: application/json" \
  -d '{ "destination": "stdout" }'

# Disable logging
curl -X PUT http://localhost:3000/api/v1/config/log-destination \
  -H "Content-Type: application/json" \
  -d '{ "destination": "none" }'

Document TTL (Instance-level)

# Check current
curl http://localhost:3000/api/v1/config/ttl/instance

# Change to 2 hours (7200 seconds)
curl -X PUT http://localhost:3000/api/v1/config/ttl/instance \
  -H "Content-Type: application/json" \
  -d '{ "ttl": 7200 }'

Document TTL (Per-document)

# Set TTL for a specific document (overrides Document_Type and instance TTL)
curl -X PUT http://localhost:3000/api/v1/documents/temperature-readings/GARAGE-TEMP-01/ttl \
  -H "Content-Type: application/json" \
  -d '{ "ttl": 1800 }'

TTL Sweep Interval

Controls how frequently TamaDB checks for expired documents (in milliseconds):

# Change sweep interval to 30 seconds
curl -X PUT http://localhost:3000/api/v1/config/ttl/sweep-interval \
  -H "Content-Type: application/json" \
  -d '{ "interval": 30000 }'

Storage Strategy

# Check current
curl http://localhost:3000/api/v1/config/storage-strategy

# Change to memory
curl -X PUT http://localhost:3000/api/v1/config/storage-strategy \
  -H "Content-Type: application/json" \
  -d '{ "mode": "memory" }'

Drain Memory to Disk

Flushes all in-memory documents to disk storage. Useful before stopping TamaDB or when switching away from auto/hybrid/memory mode. No-op in disk mode. TamaDB also automatically drains on graceful shutdown (Ctrl+C / SIGTERM).

curl -X POST http://localhost:3000/api/v1/config/storage/drain

Response:

{ "drained": true, "documentsWritten": 42, "mode": "auto" }

Quarantine Policy

# Check current
curl http://localhost:3000/api/v1/config/quarantine-policy

# Change to all-rejections
curl -X PUT http://localhost:3000/api/v1/config/quarantine-policy \
  -H "Content-Type: application/json" \
  -d '{ "policy": "all-rejections" }'

Enrichment Policies

# Check current
curl http://localhost:3000/api/v1/config/enrichment-policies

# Change policies
curl -X PUT http://localhost:3000/api/v1/config/enrichment-policies \
  -H "Content-Type: application/json" \
  -d '{
    "precedence": "environment-over-content",
    "conflictResolution": "accept-and-warn"
  }'

Persistence & Durability

TamaDB persists all state to the .tamadb-data/ directory so it resumes exactly where it left off after a restart.

What Survives a Restart

StatePersisted?How
Schemas (+ version history)YesWrite-through on every mutation
Document_TypesYesWrite-through on every mutation
Channels (+ status)YesWrite-through on every mutation
SubscriptionsYesWrite-through on every mutation
Enrichment bindings & policiesYesWrite-through on every mutation
TTL settingsYesWrite-through on every mutation
Quarantine entriesYesWrite-through on every mutation
Runtime configYesWrite-through on every mutation
DocumentsDepends on storage strategySee below

Document Durability by Storage Strategy

StrategyDocument PersistenceTrade-off
diskWritten to disk immediately on ingestSafest. Slower writes.
hybridWritten to both memory and diskFast reads + durable
auto (default)Per-operation: uses disk when memory constrainedBalanced
memoryNo disk writes. All state lost on restart.Fastest. Ephemeral (testing/CI)

To start fresh: delete the .tamadb-data/ directory and restart.

Graceful Shutdown

TamaDB provides a platform-independent shutdown endpoint:

curl -X POST http://localhost:3000/api/v1/admin/shutdown

Response:

{ "status": "shutting-down" }

This drains all in-memory documents to disk, persists all configuration state, closes the HTTP server, and exits cleanly. Also triggered by Ctrl+C / SIGTERM on Linux/macOS.

Troubleshooting

Reading Logs

TamaDB outputs structured JSON logs to stdout. Each log entry includes:

  • level: debug, info, warn, error
  • time: ISO timestamp
  • msg: human-readable description
  • Contextual fields: dId, documentType, channelId, error details

Example log entries:

{"level":"info","time":"2026-07-17T10:05:00Z","msg":"Document stored","documentType":"temperature-readings","dId":"LIVING-ROOM-01","versionNo":1}
{"level":"warn","time":"2026-07-17T10:06:00Z","msg":"Enrichment conflict resolved","fieldPath":"location","winner":"environment"}
{"level":"error","time":"2026-07-17T10:07:00Z","msg":"Schema validation failed","documentType":"temperature-readings","errors":"Field 'temperature' expected type number"}

Inspecting Quarantined Payloads

When a payload is rejected and the quarantine policy permits it, TamaDB stores the original payload for inspection.

Check the error response for the quarantine reference:

{
  "code": "SCHEMA_VALIDATION_FAILED",
  "message": "Field 'temperature' expected type number, got string",
  "quarantineRef": "qr-m1abc-0001"
}

Retrieve the quarantined payload:

curl http://localhost:3000/api/v1/quarantine/qr-m1abc-0001

List all quarantined entries:

# All entries
curl http://localhost:3000/api/v1/quarantine

# Filter by type
curl "http://localhost:3000/api/v1/quarantine?rejectionType=parse-error"

# Paginate
curl "http://localhost:3000/api/v1/quarantine?limit=10&offset=0"

Common Error Codes

CodeHTTPMeaningWhat to do
PARSE_ERROR400Payload is not valid JSONFix the producer's serialization
SCHEMA_VALIDATION_FAILED422Document doesn't match schemaCheck required fields, types, extra fields
DOCUMENT_TYPE_NOT_FOUND404No such Document_Type registeredRegister it first via /config/document-types
CHANNEL_NOT_FOUND404Channel doesn't existRegister it via /config/channels
CHANNEL_INACTIVE403Channel suspended by operatorActivate it: POST /config/channels/{id}/activate
CHANNEL_BLOCKED403Channel blocked due to schema conflictFix the Channel_Schema to be compatible with Master
VERSION_CONFLICT409Optimistic concurrency mismatchRe-read the document's current version and retry
ENRICHMENT_CONFLICT409Payload conflicts with enrichment bindingsCheck enrichment policy or adjust bindings
DOCUMENT_NOT_FOUND404Document doesn't exist or is soft-deletedCheck the reason field for details

Channel Status Issues

If a channel stops accepting ingestion:

# Check channel status
curl http://localhost:3000/api/v1/config/channels?documentType=temperature-readings

Look at the status field:

  • "active" — accepting ingestion (normal)
  • "inactive" — operator-suspended. Reactivate: POST /config/channels/{id}/activate
  • "blocked" — schema conflict. Fix the Channel_Schema, then activation becomes possible

Composed D-Ids (Multi-Field Identity)

When a schema has multiple identity fields (e.g., customer + order), D-Ids are composed:

{
  "properties": {
    "customerId": { "type": "string", "identityKind": "parent-key" },
    "orderId": { "type": "string", "identityKind": "primary-key" }
  }
}

In URLs, composed D-Ids use : as delimiter:

curl http://localhost:3000/api/v1/documents/orders/customer-A:order-123

In request bodies (store response), they're arrays:

{ "dId": ["customer-A", "order-123"] }

Quick Reference: Key Endpoints

OperationMethodURL
Ingest (document_type)POST/api/v1/ingest/{documentType}
Ingest (channel)POST/api/v1/ingest/{documentType}/{channelId}
Retrieve documentGET/api/v1/documents/{documentType}/{dId}
Delete documentDELETE/api/v1/documents/{documentType}/{dId}
SearchPOST/api/v1/documents/{documentType}/search
SubscribePOST/api/v1/subscriptions
Cancel subscriptionDELETE/api/v1/subscriptions/{subscriptionId}
Register schemaPOST/api/v1/schemas
Register Document_TypePOST/api/v1/config/document-types
Register channelPOST/api/v1/config/channels
Set enrichmentPUT/api/v1/config/channels/{channelId}/enrichment
Inspect quarantineGET/api/v1/quarantine/{ref}
Change log levelPUT/api/v1/config/log-level
Health checkGET/health
Graceful shutdownPOST/api/v1/admin/shutdown

Document version: 1.1 — August 2026. Corresponds to: TamaDB Increment 1

About Me

I'm passionate about building reliable, high-performance distributed systems. My focus areas include event-driven architectures and data pipelines.

Background

With experience in designing and implementing transmission-layer systems, I specialise in building purpose-built middleware that sits between data producers and consumers — ensuring data quality, reliability, and real-time delivery.

Current Projects

  • TamaDB — A document-centric transmission cache for validated, versioned document delivery
  • PamaDB — A planned data publishing layer for transformations and frozen document snapshots

PamaDB

PamaDB is the planned data publishing layer in the TamaDB ecosystem. While TamaDB handles transmission (receive, validate, cache, deliver), PamaDB will handle transformation and publication.

Planned Capabilities

  • Data Transformations — Apply rules to transform documents before publishing to downstream systems
  • Frozen Documents — Create immutable snapshots of document state for audit and compliance
  • Hard Subscriptions — Guaranteed delivery contracts with persistence and retry semantics beyond TamaDB's soft webhooks
  • Retrieval Schemas — Shape document output per consumer needs without affecting the stored canonical form

Relationship to TamaDB

PamaDB consumes from TamaDB's delivery pipeline. TamaDB remains the dumb-pipe transmission layer; PamaDB adds intelligence for publishing. They are complementary, not competing.

Contact

Feel free to reach out for questions, collaboration, or feedback on TamaDB and related projects.

Get in Touch

  • Email: Available on request
  • GitHub: github.com/teneriscc

External Links

Code & Repositories

  • TamaDB on GitHub — Source code, issues, releases
  • TamaDB on Docker Hub — Prebuilt multi-arch image: docker pull teneriscc/tamadb
  • GitHub Profile — Open-source contributions and experiments

Technical Blog

  • Technical Blog — Articles on system design, middleware, and data architecture

© 2026 Teneris. All rights reserved. Licensed under Apache License 2.0.