Data-Oriented programming

Data-Oriented programming

Reducing 🕸complexity 🕸 of information systems

Yehonathan Sharvit viebel

👋Who am I?

💻 Developer since 2001 (Clojure since 2012)
📖 Author of Data-Oriented programming
📝 Blogger at blog.klipse.tech
🧙 Clojure wizard at Cycognito
🎁️ Maintainer of Klipse

🕸 What is complexity?

⚙️Computational complexity

The amount of machine resources (e.g. CPU, memory) required to run a program.

😰 System Complexity

The amount of brain resources required to understand a system.

Our goal is to reduce system complexity.

"Programs must be written for people to read, and only incidentally for machines to execute" (SICP, 1986)

📊 Information systems

Systems that manipulate information in various ways.

Examples of information systems:

Web services that fetch data from the database and serves it as JSON
Web workers that listen to events and enrich them with data from multiple data sources (Data pipeline, ETL)
Front end applications that manage an application state

💡 Principles of Data-Oriented programming

Treat data as a value

Separate code from data
Represent data with generic data structures
Do not mutate data
Separate data schema from data representation

🤔 What makes a software system complex?

🕸️Sources of complexity

Nodes with many edges
Many kinds of arrows
- Association
- Composition
- Inheritance
- Usage

💡 Principle #1: Separate Code from Data

💡 Principle #1: Separate Code from Data (Data)

Classes with members only

class Book {
  constructor(String title, int publicationYear) {
    this.title = title;
    this.publicationYear = publicationYear;
  }
}

Records

public record Book (String title,
                    int publicationYear) {}

String maps

function createBook(title, publicationYear) {
  return {
    "title": title,
    "publicationYear": publicationYear
  };
}

💡 Principle #1: Separate Code from Data (Code)

Classes with static methods

class BookBusinessLogic {
  static bookInfo(Book book) {
    return book.title +
      " was published in: "
      + book.publicationYear;
  }
}

Stateless functions

function bookInfo(Book book) {
    return book.title +
      " was published in: "
      + book.publicationYear;
}

Functions receive data they manipulate as an explicit argument

🏅Reduction of System Complexity

☕What makes this system less complex?

Separation of concerns
Code diagram constraints
- Stateless (static)
- Only usage relation
Data diagram constraints
- Only association and composition

💡Principle #2: Represent Data with generic data structures

Map constructor 👍

var watchmenBook = new Map([
    ["title", "Watchmen"],
    ["publicationYear", 1986]
]);

Map literal 👍

var watchmenBook = {
  "title": "Watchmen",
  "publicationYear": 1986
};

🏅Reduction of System Complexity even more

👎 👎 👎

👍 👍 👍

☕What makes this system less complex?

Weak dependency between code and data
Loose coupling between components
Flexible data model
Generic data manipulation functions

🏅 Data is represented as data

🏅 Data is represented as data

Visualize the whole system data
Display data on console
Serialization for free
Generic data manipulation functions (sortBy, groupBy, select…)
Reflection for free
Information path (e.g. ["catalog", "authorsById", "alan-moore", "name"])

💡 Principle #3: Do not mutate Data

Values never change!
Inherently Thread-safe
No side-effects

💡 Principle #3: Do not mutate Data

What about performance?
Should I clone?
Should I copy on write?
Should I use special data structures?

💡 Principle #3: Do not mutate Data

Update the value associated with
["catalog", "booksByIsbn", "watchmen", "publicationYear"]
to 1986.

💡 Principle #3: Do not mutate Data

Update the value associated with
["catalog", "booksByIsbn", "watchmen", "publicationYear"]
to 1986.

💸 Most of the nodes are shared!

☕When data is immutable, it is safe to share it!

Immutability in practice (native data structures)

JavaScript
- Naive structural sharing with: Lodash, Ramda, Immer
- EcmaScript Proposal for record and tuple
Java
- @value annotation from Project Lombok
- Records (since Java 14)

Immutability in practice (persistent data structures)

Language	Library
JavaScript	Immutable.js
Java	Paguro
Go	Peds
Clojure	provided by the language
C#	FSharpx.Collections
Python	Pyrsistent
Ruby	Hamster

Language

Library

JavaScript

Immutable.js

Java

Paguro

Peds

Clojure

provided by the language

FSharpx.Collections

Python

Pyrsistent

Ruby

Hamster

What about data validation?

function bookInfo(book) {
  return book.title +
    " was published in: "
    + book.publicationYear;
}

var invalidBook = new Map([
    ["theTitle", "Watchmen"],
    ["publicationYear", 1986]
]);

bookInfo(invalidBook);

 was published in: 1986

Data validation (JSON Schema)

{
  "type": "object",
  "required": ["title", "publicationYear"],
  "properties": {
    "title": {"type": "string"},
    "publicationYear": {
      "type": "integer",
      "minimum": 1455,
      "maximum": 2022
    }
  }
}

var ajv = new Ajv();
var bookSchema = parseJSONFile("book-schema.json");

function bookInfo(book) {
  if(!ajv.validate(bookSchema, book)) {
    throw "bookInfo called with invalid args: " +
      ajv.errorsText(ajv.errors);
  }
  return book.title +
    " was published in: " + book.publicationYear;
}

var invalidBook = JSON.parse(
  '{"theTitle":"Watchmen","publicationYear":1986}');

bookInfo(invalidBook);

bookInfo called with invalid args:
data should have required property 'title'

JSON Schema validation libraries

Language	Library	URL
JavaScript	Ajv	https://github.com/ajv-validator/ajv
Java	Snow	https://github.com/ssilverman/snowy-json
C#	JSON.net Schema	https://www.newtonsoft.com/jsonschema
Python	jschon	https://github.com/marksparkza/jschon
Ruby	JSONSchemer	https://github.com/davishmcclurg/json_schemer

Language

Library

URL

JavaScript

Ajv

https://github.com/ajv-validator/ajv

Java

Snow

https://github.com/ssilverman/snowy-json

JSON.net Schema

https://www.newtonsoft.com/jsonschema

Python