Flagship platform
AURYNT Data Factory
Enterprise synthetic data infrastructure
A cloud-native, multi-tenant platform that generates, validates, certifies and delivers realistic synthetic datasets — built so teams can develop, test and demo against data that behaves like production without touching production.
- Period
- 2025 — ongoing
- Status
- In active development
- Stack
- Python · SQL · Spark · Airflow · PostgreSQL · Object storage · Docker · Kubernetes
Simplified data flow
- 01Schema intakeTenant schema, constraints, business rules
- 02GenerationStatistical + behavioural generators
- 03ValidationDistribution, referential, rule checks
- 04CertificationSigned quality report per dataset
- 05DeliveryBatch, incremental and streaming outputs
Multi-tenant
Isolation model
Batch · Incremental · Stream
Delivery modes
Per-run
Certification granularity
01Overview
AURYNT Data Factory treats synthetic data as a product, not a script. A tenant describes the shape of the data they need — schemas, relationships, business rules, volume, behavioural characteristics — and the platform produces a dataset that has been generated, validated against that contract, certified with a quality report, and delivered through the channel the consumer wants.
02Problem
Teams that need realistic data for development, testing, demos and model work usually get one of three bad options: a masked production copy that still carries risk, a naive random generator that produces data no system behaves realistically against, or a one-off script that nobody can reproduce six months later. None of these can be trusted, versioned, or handed to another team.
03Goals
- Multi-tenant by design — isolation of schemas, runs, storage and credentials.
- Behavioural realism, not just type-correct rows: distributions, correlations, temporal patterns.
- Every dataset carries a machine-readable certification of what was checked.
- The same definition can produce a batch load, an incremental delta, or a stream.
- Reproducible: a run is defined by a versioned spec plus a seed.
04Architecture
Control plane
Tenant, dataset-spec and run registry in PostgreSQL. Specs are versioned; a run always points at an immutable spec version.
Generation engine
Partitioned generation workers. Independent entities generate in parallel; dependent entities are ordered by a dependency graph derived from the schema.
Validation layer
Rule engine over the output: schema conformance, referential integrity, distribution distance against the declared profile, and custom business assertions.
Certification
Validation results are collapsed into a certificate stored alongside the dataset — checks run, pass/fail, tolerances, spec version, seed.
Delivery
Columnar files on object storage for batch and incremental consumption; an event feed for consumers that want the data as it is produced.
Orchestration
Airflow schedules and retries runs; the run registry is the source of truth for state so retries are idempotent.
05Data flow
A run begins from a versioned dataset spec. The planner turns the spec into a dependency-ordered set of partitions. Workers generate partitions in parallel and write them to a staging prefix in object storage. Validation reads the staged output, runs the checks declared in the spec, and writes results back to the run registry. Only a run that satisfies its certification thresholds is promoted from staging to the tenant's published prefix, which is what any consumer or delivery channel reads.
06Engineering challenges
Referential integrity under parallel generation
Generating tables independently is fast but produces orphaned keys. The generator derives a dependency graph from declared relationships and assigns deterministic key ranges per partition, so a child partition can pick valid parent keys without coordinating with other workers at runtime.
Realism is not randomness
Uniform random values pass type checks and fail every downstream expectation. Generators are driven by declared profiles — value distributions, correlations between fields, and temporal shape such as weekday and seasonal patterns — so aggregates and time series look plausible to the systems consuming them.
Multi-tenant isolation without a platform per tenant
Tenancy is enforced at every layer rather than by deployment: row-level scoping in the control plane, per-tenant storage prefixes and credentials, and tenant identity carried through the whole run lineage.
Deciding when a dataset is good enough
Certification needs a threshold, not a vibe. Each check declares a tolerance in the spec, so 'certified' means an explicit, auditable set of checks passed at stated tolerances — and the certificate says exactly which.
07Decisions & trade-offs
Spec-first, versioned dataset definitions
Why A dataset must be reproducible and reviewable. Versioning the spec makes a run explainable months later.
Alternative Imperative generation scripts — faster to start, impossible to audit or diff.
Staging then promote
Why Consumers never see a partially written or failed dataset; promotion is the only publish event.
Alternative Writing in place with a status flag — simpler, but every reader has to be trusted to honour the flag.
Columnar files on object storage as the primary substrate
Why Cheap, parallel-readable, engine-agnostic, and it decouples generation from consumption.
Alternative Loading directly into a warehouse — convenient for one consumer, coupling for everyone else.
Registry as the single source of run state
Why Idempotent retries and clear lineage require one authoritative state, not state inferred from files.
Alternative Inferring state from storage listings — cheap, and wrong under partial failure.
08Reliability
Runs are partitioned and each partition is independently retryable; the registry makes retries idempotent so a failed worker re-does only its own partition. A failed validation blocks promotion rather than corrupting a published dataset — the failure mode is 'no new data', never 'wrong data'.
09Scale
Scale comes from partitioning, not from bigger machines. Generation width is bounded by the dependency graph rather than by table count, so adding workers increases throughput for the independent portion of the schema linearly. Storage is append-only per run, which keeps published datasets immutable and cheap to serve.
10Quality
Quality is a first-class stage, not an afterthought: schema conformance, referential integrity, declared distribution tolerances, and custom business assertions. Results are persisted per run so quality is comparable across versions of the same dataset.
11Outcome
The platform turns 'we need realistic data' from a recurring one-off engineering task into a repeatable, certifiable delivery. AURYNT is in active development; the figures on this page are design targets and architectural intent, not measured production results.
Next case study
Real-Time Fraud Detection