Using DuckDB for geospatial workflows for incredible performance boost! Leveling up my GeoPandas workflows for processing the 100M Foursquare dataset.

Image Source: Foursquare

Err:6 https://mirror.hetzner.com/ubuntu/packages jammy-updates/main arm64 Packages 404 Not Found [IP: 2a01:4ff:ff00::3:3 443]

apt update & apt upgrade not working anymore on a Hetzer ARM VM?

Here’s the solution.

Text Embeddings: Speaking languages without learning them?

tl;dr: some *-en models perform well on other languages too.

Let’s process 1.369.841 single English XML/HTML files and index them with BAAI/bge-base-en-v1.5 embeddings for semantic search! Sounds fun? Let’s go!

A full semantic search tutorial about:

• data mining with requests and beautifulsoup
• preprocessing in pandas
• chunking the document text in smaller paragraphs of the right size for the ML model
• creating embeddings for each chunk
• calculating the mean embedding for each document
• saving data as gzipped json (small file size & easy and fast to read in js with pako.js)
• creating a static web app based on transformers.js on GitHub Pages

App here

Let’s set up a public Zulip instance for our geospatial community! It’s live: zulip.gis.chat

Semantic search right in your browser! Calculates the embeddings and cosine similarity client-side without server-side inferencing, using transformers.js and a quantized version of sentence-transformers/all-MiniLM-L6-v2.