My name is Joost – I am an engineer and this is my digital garden where I write sometimes about things that interest me. Also I keep lists of books I have read 📚 and things and tools that I use 🔧.
I am completely into drums 🥁 and espresso ☕.
Azure Pipelines
I can firmly say that I am well experienced in Continuous Integration / Continuous Deployment (CI/CD) pipelines, and in general simple is always better dan complex. If you need something complex, there is a different place for your process: CICD should be nothing more than: BUILD TEST RELEASE DEPLOY For a client I am working on Azure Devops Pipelines and so far my initial experiences have left me somewhat skeptical, as I find that it often imposes a level of abstraction that veers away from my preference for simplicity....
Async Pandas
Pandas is great for Python because it offers efficient data manipulation and analysis capabilities, leveraging the speed of the underlying NumPy library. How does it behave with asyncio since I could not find much about it. Have an enourmnes dataset call an API with a throughput of 10call at once. The simple example pandas.DataFrame consists of 100 rows of lorem text: import lorem import pandas as pd df = pd.DataFrame({"Text": [lorem....
Local Spark
How to install Spark locally? -> You don’t! You can find easy guides for sure, they will still guide you through a forrest of opaque steps: installing Java, Scala, downloading and installing Spark, setting environmental variables and paths. Then to find out you have the wrong versions. Have fun good luck! Kubernetes in production? Kubernetes locally! The closer you can develop to your production environment the better; In 2024 you run Spark on Kubernetes?...
Row-Level Security with SQLAlchemy
With Row Level security (RLS) you manage the access control at the row level within a database instead of the application. Row-Level Security allows you to define policies that determine which rows of data a particular user or role can access within a given table. Postgres Tables For this demonstration we create a simple setup with a User table and a Item table using SQLAlchemy 2.0: from sqlalchemy import Column, ForeignKey, Integer, String, create_engine, text from sqlalchemy....
Obfuscate Python
this post is under construction – I have the approaches here but need some time to also share the experience… How to obscure some Python code from anyone running the code? I am no expert here but I have tried a few things and will give my steps and recommendations here. Have a main.py with a simple helloworld FastAPI in this case. There is also an /error endpoint to see how much source code is returned in the logs....