While skiing in the Swiss Alps last winter, I got into a discussion about speed. Following my engineering knowledge, your skiing speed does not depend on your own mass. Without the intention to offend anyone, skiing is basically a form of falling at an angle. We can build a model based on Newton’s second law where acceleration times mass is equal to the resultant force. We can see that mass ($m$) will drop out of the equation: ...
Find myself looking for an overview too often. So let’s create a rough overview of common used config for Spark. As a start, create a Spark Session with default config: from pyspark.sql import SparkSession spark = SparkSession.builder \ .master(SPARK_MASTER) \ .appname("app name") \ .getOrCreate() The Spark Context represents the connection to the cluster; communicaties with lower-level API’s and RDDs. Some resource settings on the driver: ... .config("spark.driver.memory", "8g") ... .config("spark.cores.max", "4") .config("spark.executor.memory", "8g") .config("spark.executor.cores", "4") ... Number of shuffle partitions (default is 200), should ideally be equal to the number of cores in the cluster: ...
Through Hackernews I found out about the recent release of Rancher Desktop and I was curious if this would be a good alternative to Docker desktop for the develop of web applications on my local machine. I don’t really have a problem with Docker desktop, just good to try something new every now and then and it is open-source. Running some containers really gets some steam out of my Mac so hopefully it has some improvement there. Also I like the idea of using Kubernetes, bringing the dev environment way closer to the production environment. ...
Had a complete headache trying to figure out how a decorator as a class can maintain the possible async properties of a method. The solution is actually very simple. When called, use inspect.iscoroutinefunction to check whether it is a coroutine, and return again an async method! The example adds given paths to a registry, import inspect from functools import wraps paths_registry = [] class route(object): def __init__(self, path: str, **kwargs) -> None: self._path = path def __call__(self, fn): paths_registry.append( { "path": self._path, } ) @wraps(fn) def decorated(*args, **kwargs): return fn(*args, **kwargs) @wraps(fn) async def asyncdecorated(*args, **kwargs): return await fn(*args, **kwargs) if inspect.iscoroutinefunction(fn): return asyncdecorated return decorated A method with this decorator would look like: ...
This is a simple demonstration of a domain event factory in Python. I assume you are familiar with the Factory Method Pattern. I also use the pydantic package for attribute validation. When implemented, we can use the factory to create immutable domain events with a homogenous data structure across instances of the same type. The metadata is generated by the underlying BaseEvent. In this approach we always produces complete events. Start with all our imports: ...