Docs

Domain-Driven Design is a concept that focuses on the problems of specific knowledge areas. Command Query Responsibility Segregation (CQRS) separates a software into a write side and a read side. Event Sourcing is an architectural pattern that represents state as a sequence of immutable events. Domains and models A domain is the area of expertise about the problem-to-be-solved. In relation to DDD, the domain comes with some category definitions: Domain Expert: the primary person(s) with understanding of the problem domain; Subdomains: are parts of the entire information domain; divided into the following categories: Core Domain: information area most relevant to the problem-to-be-solved Supporting Subdomain: area of information that is parlty generic but nevertheless relevant to the problem-to-be-solved Generic Subdomain: universal knowledge that is not specific to the main problem (on the list to outsource) Domain Model is the structured abstraction (written English, diagrams and code examples) of the klowledge area, in terms of behavior and characteristics. Ubiquitous Language is the linguistic system valid in the context of a model, to make communication precise and effective. ...

Airflow Scheduling This simple example describes it all: from datetime import datetime from airflow import DAG from airflow.operators.dummy_operator import DummyOperator import pendulum local_tz = pendulum.timezone("Europe/Amsterdam") default_args = { 'start_date': datetime(2020, 3, 28, tzinfo=local_tz), 'owner': 'XYZ', 'schedule_interval': '0 2 * * *'' } with DAG('tutorial', catchup=False, default_args=default_args) as dag: DummyOperator(task_id='dummy', dag=dag) Schedule a daily task with the scheduler_interval as a crontab (also accepts @daily, @weekly or None); on a specific timezone with pendulum; catchup is turned off to prevent “backfill” of all past DAG runs. start_date is not necessarily when the first DAG run would be executed, as a DAG run is triggered at the end of its schedule period. Understand that the DAG runs at the end of its schedule period, rather than at the beginning of it. The execution time is not the actual run time, but rather the start timestamp of its schedule period. ...

Deployment Simple helm charts: https://github.com/helm/charts/tree/master/stable/postgresql Bitnami Extensions PostgreSQL Extensions are a plug and play set of enhancements that add an extra feature-set to PostgreSQL. Some of these features are as simple as reading or writing to an external database while others could be a sophisticated solution to implement database replication, monitoring, etc. Postgis CREATE EXTENSION IF NOT EXISTS postgis CASCADE; Timescale DB TimescaleDB is an open-source time-series database optimized for fast ingest and complex queries. ...

performance recursive CTE A Common Table Expression is a temporary expression. The clause begins with WITH RECURSIVE, has two parts separated by UNION [ALL] or UNION DISTINCT. Recursive queries make the SQL language complete, some things could otherwise not be executed. Recursive means that a part of the query is a query itself, with SQL this can be done with a common table expression (CTE): WITH ctename AS ( SELECT ... ) SELECT ... FROM ctename ... A recursive query include the term RECURSIVE: ...

Spark Process datasets that cannot be handled on a single pc; Spark. Run queries on very large datasets. Grew from the Hadoop ecosystem, a software platform for distributed storage and processing of very large data sets. We are going to deploy Spark on our cluster, specifaclly Spark 3.0.0 was officially released on 18th June 2020. Why Spark 3? Compared to Spark 2, it has improbed performance and it will be easier to extend it with libraries that run on GPU nodes, there is support for Cypher, a query language for graph integration. These are not features that we’ll need rightaway, but we might need them in the near future. ...