How to install Spark locally? -> You don’t! You can find easy guides for sure, they will still guide you through a forrest of opaque steps: installing Java, Scala, downloading and installing Spark, setting environmental variables and paths. Then to find out you have the wrong versions. Have fun good luck! Kubernetes in production? Kubernetes locally! The closer you can develop to your production environment the better; In 2024 you run Spark on Kubernetes?...
Through Hackernews I found out about the recent release of Rancher Desktop and I was curious if this would be a good alternative to Docker desktop for the develop of web applications on my local machine. I don’t really have a problem with Docker desktop, just good to try something new every now and then and it is open-source. Running some containers really gets some steam out of my Mac so hopefully it has some improvement there....
Now that I have a number of pipelines running I would like to deploy these to Kubernetes through a service account. that is quite simple. As an admin user provide resources such as: the namespaces, optionally with limited resources; an isolated service account with restricted access to one namespace; an encoded config file to be used by the Gitlab pipeline. Service Account with permissions The following file serviceaccount.yaml creates the service account, a role, and attach that role to that account:...
Earlier I posted about my hobby cluster on GKE which I want to keep under an affordable budget. Unfortunately Google Cloud will start charging a management fee from june 2k20 of 10$ct per hour (=$73/mnth) just like AWS. If they unilaterally change the rules, let’s get out of here! I’m thinking of moving to a self-managed Kubernetes cluster on AWS with spot instances: 1 x 1GiB master-node (t2.micro spot instance, $2....
With any project, personal or for clients, I develop and build applications in Docker containers; so it would be really convenient to run these as containers in production. Usually I’ll spin up a small Ubuntu VM (GCP/Azure/AWS/Digital Ocean) and install Docker manually. I would love to run everything with Kubernetes as a container orchestrator, but the costs of such a cluster for personal usage seems rather high. Until I found this article by Remko Seelig, using Kubernetes on Google Cloud with preemptible nodes, which is about half the price of regular instances....