Python

Polars has been making waves in the Python data world — and for good reason. It’s fast, expressive, and built with performance-first principles. If you’re dealing with Parquet files in an S3 bucket and care even a little about speed or memory, this post is for you. Let’s talk about three powerful components working together: 🪣 S3 bucket storing Parquet files 📦 PyArrow-style datasets ⚡ Polars doing its thing — lazily and efficiently 🔍 The Case for Lazy Reading When you reach for read_parquet, you get everything. That’s fine… until it’s not. Instead, scan_parquet gives you a lazy frame — and that changes everything. ...

Basics: botocore is the low-level interface for practically all other clients. boto3 official Amazon Python client minio standalone alternative to botocore/boto3, does not natively support asynchronous operations as of now. Async: aiobotocore async support for botocore aioboto3 wrapper for aiobotocore s3fs wrapper for aiobotocore Boto3 import boto3 from botocore.config import Config # Initialize the S3 client with a custom endpoint s3 = boto3.client( "s3", endpoint_url="https://custom-endpoint.com", aws_access_key_id="your-access-key", aws_secret_access_key="your-secret-key", config=Config(signature_version="s3v4"), ) # Upload a file s3.upload_file("local/path/file.txt", "bucket-name", "destination/path/file.txt") # Download a file s3.download_file("bucket-name", "source/path/file.txt", "local/path/file.txt") Minio from minio import Minio # Initialize the Minio client with a custom endpoint client = Minio( "custom-endpoint.com", access_key="your-access-key", secret_key="your-secret-key", secure=True, # Set to False if not using HTTPS ) # Upload a file client.fput_object("bucket-name", "destination/path/file.txt", "local/path/file.txt") # Download a file client.fget_object("bucket-name", "source/path/file.txt", "local/path/file.txt") aioboto3 import aioboto3 # Create an async session and client session = aioboto3.Session( aws_access_key_id="your-access-key", aws_secret_access_key="your-secret-key", ) async with aioboto3.Session().client( "s3", endpoint_url=endpoint_url, aws_access_key_id=access_key, aws_secret_access_key=secret_key, ) as s3: # Upload the file with open(file_path, "rb") as file: await s3.upload_fileobj(file, bucket_name, object_name) S3fs Looks the most clean: ...

Pandas is great for Python because it offers efficient data manipulation and analysis capabilities, leveraging the speed of the underlying NumPy library. How does it behave with asyncio since I could not find much about it. Have an enourmnes dataset call an API with a throughput of 10call at once. The simple example pandas.DataFrame consists of 100 rows of lorem text: import lorem import pandas as pd df = pd.DataFrame({"Text": [lorem.text() for _ in range(100)]}) >>> df.head() Text 0 Labore quisquam neque adipisci labore non quae... 1 Aliquam etincidunt dolore dolore voluptatem. A... 2 Aliquam consectetur dolor dolorem dolorem ipsu... 3 Labore non aliquam numquam sed. Eius neque con... 4 Voluptatem ipsum modi amet tempora tempora eti... Asyncio If we want to sent every row to an API and that call takes about a second. Let’s consider this method reverses the text and returns the final three letters: ...

With Row Level security (RLS) you manage the access control at the row level within a database instead of the application. Row-Level Security allows you to define policies that determine which rows of data a particular user or role can access within a given table. Postgres Tables For this demonstration we create a simple setup with a User table and a Item table using SQLAlchemy 2.0: from sqlalchemy import Column, ForeignKey, Integer, String, create_engine, text from sqlalchemy.orm import declarative_base, relationship admin_engine = create_engine("postgresql://postgres:postgres@0.0.0.0:5432/postgres") Base = declarative_base() class Item(Base): __tablename__ = "items" id = Column(Integer, primary_key=True) name = Column(String) user_id = Column(Integer, ForeignKey("users.id")) user = relationship("User", back_populates="item_entries") class User(Base): __tablename__ = "users" id = Column(Integer, primary_key=True) username = Column(String) password = Column(String) item_entries = relationship("Item", back_populates="user") Base.metadata.create_all(admin_engine) Using the PostgreSQL superuser for application access is not a great idea due to its extensive privileges and security risks. It’s advisable to create a dedicated user with limited permissions tailored to the application’s requirements for improved security and operational control. ...

this post is under construction – I have the approaches here but need some time to also share the experience… How to obscure some Python code from anyone running the code? I am no expert here but I have tried a few things and will give my steps and recommendations here. Have a main.py with a simple helloworld FastAPI in this case. There is also an /error endpoint to see how much source code is returned in the logs. ...