Data-Oriented programming

Reducing 🕸complexity 🕸 of information systems




Yehonathan Sharvit viebel

👋Who am I?

  • 💻 Developer since 2001 (Clojure since 2012)

  • 📖 Author of Data-Oriented programming

  • 📝 Blogger at blog.klipse.tech

  • 🧙 Clojure wizard at Cycognito

  • 🎁️ Maintainer of Klipse

dop book

🕸 What is complexity?



⚙️Computational complexity

The amount of machine resources (e.g. CPU, memory) required to run a program.

😰 System Complexity

The amount of brain resources required to understand a system.


Our goal is to reduce system complexity.

"Programs must be written for people to read, and only incidentally for machines to execute" (SICP, 1986)

📊 Information systems

Systems that manipulate information in various ways.




Examples of information systems:

  • Web services that fetch data from the database and serves it as JSON

  • Web workers that listen to events and enrich them with data from multiple data sources (Data pipeline, ETL)

  • Front end applications that manage an application state

💡 Principles of Data-Oriented programming

Treat data as a value



  1. Separate code from data

  2. Represent data with generic data structures

  3. Do not mutate data

  4. Separate data schema from data representation

🤔 What makes a software system complex?

watchmen library


complex class relation

🕸️Sources of complexity

  • Nodes with many edges

  • Many kinds of arrows

    • Association

    • Composition

    • Inheritance

    • Usage

💡 Principle #1: Separate Code from Data

principle1 nutshell v3

💡 Principle #1: Separate Code from Data (Data)

principle1 nutshell data v3
Classes with members only
class Book {
  constructor(String title, int publicationYear) {
    this.title = title;
    this.publicationYear = publicationYear;
  }
}
Records
public record Book (String title,
                    int publicationYear) {}
String maps
function createBook(title, publicationYear) {
  return {
    "title": title,
    "publicationYear": publicationYear
  };
}

💡 Principle #1: Separate Code from Data (Code)

principle1 nutshell code v3
Classes with static methods
class BookBusinessLogic {
  static bookInfo(Book book) {
    return book.title +
      " was published in: "
      + book.publicationYear;
  }
}
Stateless functions
function bookInfo(Book book) {
    return book.title +
      " was published in: "
      + book.publicationYear;
}



Functions receive data they manipulate as an explicit argument

🏅Reduction of System Complexity

from oop to dop

☕What makes this system less complex?

  • Separation of concerns

  • Code diagram constraints

    • Stateless (static)

    • Only usage relation

  • Data diagram constraints

    • Only association and composition

💡Principle #2: Represent Data with generic data structures



Map constructor 👍
var watchmenBook = new Map([
    ["title", "Watchmen"],
    ["publicationYear", 1986]
]);
Map literal 👍
var watchmenBook = {
  "title": "Watchmen",
  "publicationYear": 1986
};

🏅Reduction of System Complexity even more

data code relation

👎 👎 👎

code only

👍 👍 👍

☕What makes this system less complex?

  • Weak dependency between code and data

  • Loose coupling between components

  • Flexible data model

  • Generic data manipulation functions

🏅 Data is represented as data



zen4

🏅 Data is represented as data

library data
  • Visualize the whole system data

  • Display data on console

  • Serialization for free

  • Generic data manipulation functions (sortBy, groupBy, select…​)

  • Reflection for free

  • Information path (e.g. ["catalog", "authorsById", "alan-moore", "name"])

💡 Principle #3: Do not mutate Data

JobYouHate




  • Values never change!

  • Inherently Thread-safe

  • No side-effects

💡 Principle #3: Do not mutate Data

yes but how




  • What about performance?

  • Should I clone?

  • Should I copy on write?

  • Should I use special data structures?

💡 Principle #3: Do not mutate Data

lib data high level

Update the value associated with
["catalog", "booksByIsbn", "watchmen", "publicationYear"]
to 1986.

💡 Principle #3: Do not mutate Data

structural sharing

Update the value associated with
["catalog", "booksByIsbn", "watchmen", "publicationYear"]
to 1986.

💸 Most of the nodes are shared!

☕When data is immutable, it is safe to share it!

Immutability in practice (native data structures)

  • JavaScript

    • Naive structural sharing with: Lodash, Ramda, Immer

    • EcmaScript Proposal for record and tuple

  • Java

    • @value annotation from Project Lombok

    • Records (since Java 14)

Immutability in practice (persistent data structures)

LanguageLibrary

JavaScript

Immutable.js

Java

Paguro

Go

Peds

Clojure

provided by the language

C#

FSharpx.Collections

Python

Pyrsistent

Ruby

Hamster

What about data validation?



function bookInfo(book) {
  return book.title +
    " was published in: "
    + book.publicationYear;
}
var invalidBook = new Map([
    ["theTitle", "Watchmen"],
    ["publicationYear", 1986]
]);
bookInfo(invalidBook);
 was published in: 1986

Data validation (JSON Schema)

{
  "type": "object",
  "required": ["title", "publicationYear"],
  "properties": {
    "title": {"type": "string"},
    "publicationYear": {
      "type": "integer",
      "minimum": 1455,
      "maximum": 2022
    }
  }
}
var ajv = new Ajv();
var bookSchema = parseJSONFile("book-schema.json");
function bookInfo(book) {
  if(!ajv.validate(bookSchema, book)) {
    throw "bookInfo called with invalid args: " +
      ajv.errorsText(ajv.errors);
  }
  return book.title +
    " was published in: " + book.publicationYear;
}
var invalidBook = JSON.parse(
  '{"theTitle":"Watchmen","publicationYear":1986}');
bookInfo(invalidBook);
bookInfo called with invalid args:
data should have required property 'title'

JSON Schema validation libraries

LanguageLibraryURL

JavaScript

Ajv

https://github.com/ajv-validator/ajv

Java

Snow

https://github.com/ssilverman/snowy-json

C#

JSON.net Schema

https://www.newtonsoft.com/jsonschema

Python

jschon

https://github.com/marksparkza/jschon

Ruby

JSONSchemer

https://github.com/davishmcclurg/json_schemer

⌛ A brief history of Data-Oriented programming

dop timeline

⏳Summary

do principles mind map

🎹Imagine

🙋 Questions

dop book