DuckDB for Geospatial
Using DuckDB for geospatial workflows for incredible performance boost! Leveling up my GeoPandas workflows for processing the 100M Foursquare dataset.
Blog
Hetzner ARM Update Fix
apt update & apt upgrade not working anymore on a Hetzer ARM VM? Here's the solution.
Err:6 https://mirror.hetzner.com/ubuntu/packages jammy-updates/main arm64 Packages 404 Not Found [IP: 2a01:4ff:ff00::3:3 443]English-only embedding models for multilingual docs
Text Embeddings: Speaking languages without learning them? tl;dr: some *-en models perform well on other languages too.
A performant embedding processing pipeline & tutorial for big XML/HTML data dumps
Let's process 1.369.841 single English XML/HTML files and index them with BAAI/bge-base-en-v1.5 embeddings for semantic search! Sounds fun? Let's go!
Semantic Search Tutorial
A full semantic search tutorial about: data mining with requests and beautifulsoup preprocessing in pandas chunking the document text in smaller paragraphs of the right size for the ML model creating embeddings for each chunk calculating the mean embedding for each document saving data as gzipped json (small file size & easy and fast to read in js with pako.js) creating a static web app based on transformers.js on GitHub Pages App here
Zulip Server Setup Tutorial: zulip.gis.chat
Let's set up a public Zulip instance for our geospatial community! It's live: zulip.gis.chat
SemanticFinder - frontend-only live semantic search with transformers.js
Semantic search right in your browser! Calculates the embeddings and cosine similarity client-side without server-side inferencing, using transformers.js and a quantized version of sentence-transformers/all-MiniLM-L6-v2.
Semantic Search with Qdrant, Hugging Face, SentenceTransformers and transformers.js
Create a fully working semantic search stack with only Qdrant as vector database with built-in API and transformers.js using any huggingface model as your frontend-only embedding generator. No additional inference server needed!
Geospatial Vector Search with Qdrant
Using qdrant for querying text data with vector search and geospatial filters without GPU (CPU only)
Pyscript File Handling
[Work in progress] Useful PyScript file handling examples for all combinations: local/remote files and import/export.
Connect Pandas to a remote PostgreSQL database via SSH
Use Pandas to execute SQL statements in a remote database such as PostgreSQL via SSH
PyScript-fueled Jupyter Notebooks for your browser
A simple PyScript-fueled Jupyter Notebook running entirely in your browser - no server needed!
PyScript - A Game Changer For Your Browser
Execute Python right in your Browser!
<!--Load PyScript, Pyodide, Micropip and Python packages-->
<script defer src="https://pyscript.net/alpha/pyscript.js"></script>
<py-env>
- pandas
</py-env>
<!-- Write your Python code! -->
<py-script>
import pandas as pd
from pyodide.http import open_url
df = pd.read_csv(open_url("/downloads/iris.csv"))
df
</py-script>HyperLogLog Simply Explained
Floral foam 🌼, elephants 🐘 and evil mayors 😈: the useful HyperLogLog algorithm explained as if you were 10 years old!
Life Hacking 2.0: Adding Geographical Filters to Flat Search Websites with QGIS and Python
Create sophisticated walking isochrones for Milan's metro stops and search for flats with immobiliare! Reproducible for all of Italy!
HDMI Remote Screen Control
Control all functions of your external monitor with a click on your (wireless) keyboard! Lower brightness, switch input, raise contrast or switch power modes with Python and Monitorcontrol.
City Skylines
Create a free skyline for any city on earth!
Basic River Animations with Blender - Mosel Edition
Basic River Animations with QGIS, Aerialod and Blender - Mosel from Trier to Traben-Trarbach
How to create Paper (Mario) Maps
Pretty Paper (Mario) Maps quickly done with QGIS and Aerialod.
Guide to 3D OSM Building Models - Milan Edition
OSM data offers a vast amount of information about our cities. In this guide I focus on working and styling OSM building data for 3D-modeling in Aerialod for close-to-reality models. Tl;dr: Use OSM building levels, emulate roofs with shapeburst effect and use Sketchfab for interactive 3D models
Guide to 3D OSM City Models - Trier Edition
There are few cities providing open source LIDAR data for 3D-modeling. However, leveraging OSM building, road and landuse data gives you neat city models in no time!
Raster-Based 3D Models with QGIS, OSM and Aerialod
Creating 3D models from raster or vector data has never been that easy! This tutorial takes you through the whole process from vector/raster data to 3D in just 3 steps.
SSH Port Forwarding Guide
Setting up proper port forwarding for SSH connections, servers or just test purposes is crucial but isn't as straight forward as your connection should be. This post aims to be a quick guide for different use cases.
Convert Python Scripts to Windows Executables (.py to .exe) - Pyinstaller and Pandas | 2022 WORKING VERSION
Python and pandas are a great team for data science. But what if you need to deliver a script to a Windows client who does not have Python installed? This is a quick tutorial for converting Python scripts (.py) to standalone, one-file Windows executables (.exe). Get a 20Mb exe file that is fully functional on every Windows machine!
pyinstaller -F my_script.pyAn Introduction to HyperLogLog
Have you ever wondered how HyperLogLog works? Or have you never heard of it at all? In this post I explain the wonderful algorithm of Flajolet et al. from scratch and in a very simple manner.
Life Hacking: Adding Geographical Filters to Flat Search Websites with Turf.js
Flat hunting can be a big pain - particularly for European metropoles. Most flat search platforms already offer good filter options but until now, I didn't see any with custom geographic filters. Apart from drawing your own area of interest on a map, you seldomly find filters such as "How far is the next metro stop?" or "How close are important facilities?". This article will show a quick and universal workaround for custom geographic filters based on the fantastic pure javascript GIS Turf.js. I implement a buffer based on Milano's metro stops, verify that the flat is located within the buffer and open the respective offer in a new browser tab.
Using HyperLogLog with Leaflet-Hexbins
Leaflet-Hexbins is a powerful tool to aggregate data and visualize it quickly on a map. It can size the hexbin radius based on a feature values. But what if we used HyperLogLog data to preserve user-privacy? This post will provide a convenient way to quickly perform on-the-fly HyperLogLog unions for leaflet-hexbins. No more backend-constraint for HyperLogLog There are many HyperLogLog implementations out there but few work as well as postgres-hll and js-hll. They can even talk to each other by exchanging hexstrings! Usually postgres-hll is pretty much the standard extension to use, when dealing with HyperLogLog but is obviously bound to a postgres database running somewhere. With js-hll, the backend-constraint for HyperLogLog is gone, just so that users can perform any HyperLogLog action in their frontend!
HyperLogLog Javascript (js-hll) & Postgres (postgres-hll): making them talk to each other!
HyperLogLog is a fantastic algorithm to quickly count distinct elements in a set. As it consumes few memory and is really fast, it's perfectly suited for big data. For a nice, visual explanation step over to the Redis guys on Youtube. However, this article is not about explaining HyperLogLog but rather understanding how to deal with backend/frontend communication between javascript and postgres by exchanging hexstrings between the two amazing libraries postgres-hll for postgres and js-hll for javascript.
hex_array = ["/x138b7f010101c101e3024104e1058305a105c3072207810922098209a209c30b220c410ce20e210e410e610ee11002106310a210e31323138513a213c2142114421463156115a115e416011661168316a216e1186218a31927194119611aa81b411b611be11c411d431dc21e22218121a121e122a322e12324236123c224c12641266127c3288329442a012a612b012be22cc52d412e612e812fa12fc1302131843282338133e335023541356235a23623364138233863388139813a023a623a823d033d413e613ea13f4140a1414141e242014322438143a24406458145a145e546e14721474248014822486249214a234ae14b414b614be34ca24cc14d424d614da14e214fa250225101514552e453215423548154c355c1564558e3590359415b215b415d625da25f225f836122614162426422648164a16681672367c168016822686168a169216943698169c56b426b816d016e616ee16f416fc17102718471e17326742474e177027742776277e2782378a27901794179817a027a617aa17b217b817c627ce57d817dc47e837ee3800280848181826582a382c1832584e18502854285628607864187e5880188618883898289e28a248a438aa18ac18ae28b028b218b818ca28e028ea58f278fe4904190e192c192e1932193e2940194e2958395c2962296a496e198229923994499e19a619aa59ae39be29ce29e219e419ee49f039f219f63a001a041a181a1c2a202a241a341a462a483a4e3a501a564a5c1a622a6e1a801a867a941a981aaa3aae1ab24abe1ac01acc2ad62ae01ae21afc1b001b082b0a2b103b182b1e2b2a2b2e2b321b3a2b541b581b5e4b6e1b7e1b804b821b922ba21ba41ba83bbe3bc41bc82be04bec3bf01c021c0a2c204c262c2e1c3c1c542c681c782c861c8a1c923ca21cac1cf61d042d0a2d1a2d222d3e1d5a1d5e2d681d6e2d741d7e1d822d961d9c2da03da61da81db42db63db81dc01dd27dd44dde3de22dea1dec4df69dfe1e043e082e162e281e301e321e3c7e521e562e5a2e6a8e6c4e721e742e764e782e821e8e1e925ea45eac2eb81eba5ecc1ece4edc2ede2ee02eea1eec3eee1ef83efa1f082f0a7f121f1c1f2c3f3c2f405f464f501f521f642f661f682f6a3f724f781f8e1f924f981f9a1fa01fbe1fc03fc41fcc2fdc2fe81fea1ffc3ffe1",
"/x128b7f94c64fc29302fedb95064f4594429faea211348181b84bf6a4de1a560e2cde5aae10a82d9b23d04cb37315171da42c7eb634d69b7206eda3b9f61329abd41006c3754ff2b383fe29c5293f8d8dc2dd0ac6e0d283ed4c464ecb1077494c2d63bed629d2966cf0f74ed6d1c433d793c977e7c3ef6d62b062dae87c30eb549046fbed1a115578bfd959f7defd537345d9dc02d1bcbb389f666a036c085757a5147d0bde87fd03cfff7c0c2f46328f046dc50df9247d5d7c02450e7e599707200bef1993af1b2a2520ce27ae169919eb38aa27e882def7ec080c2846cff817fe669530085d7134cb022b3176150ff4e07b5b339241a82324a9d54dbf805d46768d8955167824ea33a3fe55daffbb24f17f5a58c22d17fbba79b05aa808f534be42765f7e8940497ef0c0601e98e71202a33e6fa844e45f16e3f97ad2759531cfe48d7cb25be453a9c8b0",
"/x128b7f8592206f9f19e59c8fd86aefaa33159392aa0e34f0ec18369fe0b373f4f8fb0ea160a594697af83ca209824fdf11dc96a2691b9bdb7a5d68b27eec512b56e437b5a25cb801028cc2b708aa8ad616ef9db8ec683405749d5cbf1cb31e793ca24cc39099abfbdefae9c3b77288c737593bc5e8fbd70be21d25cbecf5f3fc316718ccc47bada8f1a802d0ecb268db22971edd5556b6ef9c83b3deb153736583554efa9f8831cd825d37fbaa507b619aa7f8fcd9e3c86352460202568446a3b4e7ab0baed2436665893018a5739a57a4afe81b57440b7edbb2801c9b4f2bb609d6291d2a2f4022fdf5b9228fe694968e225b232cb3ef34aa283929b266e071ff51d32cd41d8079ca04032f58fb7330727f962ff990b545c1aa3e341c639ae719f1fb392c8bdaf8e352b63eaa1a73ec6d0878407f68d908e2138840a56c50cb1d62db4adfed6510fd513a4f2e41693e7f609258fa6eb4cca8c8fa5c4637c28a62b27b5dec7b773dbc1d0560c2eafcc1e091236bde80216b9972156f4f9d7bda221f0e7b53f009355d4be47ccca7e798abb1267e1fda106584187a"]
hex_array.reduce(function (first, second) {
return first.union(
hll.fromHexString(
second.replace(/\\\\/g,"/"))
.hllSet)}, new hll.HLL(11, 5)
).cardinality();
// 567Querying Postgres with Python Fastapi Backend and Leaflet-Geoman Frontend - Applying Geometry Filters
Querying Postgres with Python Fastapi Backend and Leaflet & Geoman Fronted - Applying Geometry Filters How to query a database with a user-defined geometry drawn on a Leaflet frontend? A very common use case for map applications are custom filters. Either thematic, temporal or value-based filters play a significant role but probably in a geographical context the most dominant one might be spatial filtering. In this blog post, I describe a simple boilerplate setup based on postgres, fastapi and geoman.
Leaflet Data Visualization Strategies - Markers, Marker Clusters, Heatmap, Hexbins
Leaflet Data Visualization Strategies - Markers, Marker Clusters, Heatmap, Hexbins
Hugo blog routine - write, preview, edit, commit
My simple blog routine: how to create a post, preview and commit to production.
██████╗ ███████╗ ██████╗ ██████╗ ██████╗ ██████╗██╗ ██╗███████╗
██╔════╝ ██╔════╝██╔═══██╗ ██╔══██╗██╔═══██╗██╔════╝██║ ██╔╝██╔════╝
██║ ███╗█████╗ ██║ ██║ ██████╔╝██║ ██║██║ █████╔╝ ███████╗
██║ ██║██╔══╝ ██║ ██║ ██╔══██╗██║ ██║██║ ██╔═██╗ ╚════██║
╚██████╔╝███████╗╚██████╔╝██╗██║ ██║╚██████╔╝╚██████╗██║ ██╗███████║
╚═════╝ ╚══════╝ ╚═════╝ ╚═╝╚═╝ ╚═╝ ╚═════╝ ╚═════╝╚═╝ ╚═╝╚══════╝
Run a Background Thread in Python - Torpy Image Download
How to start a background thread in Python to download images from an API.
import threading
def start_thread(func, name=None, args = []):
threading.Thread(target=func, name=name, args=args).start()VM Port Forwarding
Port Forwarding in Virtual Box - Postgres example
SSH in Virtual Machine Ubuntu Server 20.04
A quick guide to setup a SSH-conncection to Virtual Machine Ubuntu Server 20.04
import requests
response = requests.get("https://webscraper.io/test-sites/e-commerce/allinone")
responseUsing Jupyter Notebook as GIS
Clipping a geodataframe and displaying it on a basemap in Jupyter.
QGIS - How to use custom SVG-Markers
Use your own SVG as a QGIS marker!
A complete guide to preprocess Instagram post data
Scripts and tricks to preprocess Instagram data mined with Fast-Instagram-Scraper. Get it on a map!
Mining Instagram efficiently with Torpy - Fast Instagram Scraper
A fast and efficient Instagram Scraper based on Torpy. Scrapes posts for multiple hashtags and location ids.
A quick readthedocs Wiki with mkdocs
Setting up a readthedocs wiki like 1,2,3!
Debugging Python
Debugging Python with pdb 🐞
import pdb
from pdb import set_trace as bp
bp()Writing Your Own Scraper - Simple Instagram Scraper
How to write your DYI Instagram Scraper
Deleting Browser Cache with Python Selenium and Chromedriver
Quick and dirty
Mining Instagram location IDs
How to quickly mine location IDs from Instagram's location explorer.
{
"Jena": "673710",
"Jena-Zentrum": "1006125655",
"Plan B Boulderhalle Jena": "613227777",
"Kassablanca Jena": "5752477",
"Kulturarena Jena": "574057843",
"University of Jena": "251848169",
"Paradiespark Para - Jena Paradies": "228419929",
"Ernst-Abbe-Sportfeld": "326665551",
...
}Processing data mined from instagram-scraper with Python and Pandas
Making use of Python and Pandas to quickly process mined social media data of instagram-scraper. Get most common hashtags!
Social Media Mining
A personal shortlist of well-working Python-based scrapers in 2020.
Creating a simple Windows Powershell shortcut
Automating a daily powershell git routine.
.\gCreating a simple Ubuntu Bash shortcut
A daily routine of cd'ing in a directory and starting a server can easily be automated.
. xMost common Hugo pitfalls and their solutions
#1 Line endings CRLF vs. LF Theres a difference in how line endings are encoded. If you use VS Code and git, take care that line endings are handled the right way.
Go professional - Creating your personal homepage with custom domain name & email for less than 1$
There comes a time one needs a web page. May it be for work, a project or as personal showcase. After using wordpress for some time, I was looking for a better option to design, maintain and host a personal homepage. On top of that, I wanted a custom domain email - and all of this of course for as little money as possible.
Speech recognition with Python in 12 lines of code - Play the radio!
Did you ever want to create your own Alexa, Siri & co.? Python's fantastic speech recognition package enables you to quickly create your own custom commands. And the best part: you can decide what kind of speech recognition you want - online or offline! So let's get started!
Technological Citizenship
Empowerment technologischer BürgerInnen oder doch nur Buzz-Word? "It is time to actually become citizens within technological society" - (Frankenfeld 1992, 477). So kommentiert Frankenfeld das unterforschte Konzept der Technological Citizenship. Was bedarf es um an unserer heutigen hochtechnisierten Gesellschaft teilzunehmen? Aktueller denn je, wird hier untersucht, ob Technological Citizenship Bürgerinnen und Bürger in einer Demokratie zu empowern vermag.
Browse the timeline