There comes a time one needs a web page. May it be for work, a project or as personal showcase. After using wordpress for some time, I was looking for a better option to design, maintain and host a personal homepage. On top of that, I wanted a custom domain email - and all of this of course for as little money as possible.
Did you ever want to create your own Alexa, Siri & co.? Python's fantastic speech recognition package enables you to quickly create your own custom commands. And the best part: you can decide what kind of speech recognition you want - online or offline! So let's get started!
Empowerment technologischer BürgerInnen oder doch nur Buzz-Word? "It is time to actually become citizens within technological society" - (Frankenfeld 1992, 477). So kommentiert Frankenfeld das unterforschte Konzept der Technological Citizenship. Was bedarf es um an unserer heutigen hochtechnisierten Gesellschaft teilzunehmen? Aktueller denn je, wird hier untersucht, ob Technological Citizenship Bürgerinnen und Bürger in einer Demokratie zu empowern vermag.
Using DuckDB for geospatial workflows for incredible performance boost! Leveling up my GeoPandas workflows for processing the 100M Foursquare dataset.
apt update & apt upgrade not working anymore on a Hetzer ARM VM? Here's the solution.
Err:6 https://mirror.hetzner.com/ubuntu/packages jammy-updates/main arm64 Packages 404 Not Found [IP: 2a01:4ff:ff00::3:3 443]Text Embeddings: Speaking languages without learning them? tl;dr: some *-en models perform well on other languages too.
Let's process 1.369.841 single English XML/HTML files and index them with BAAI/bge-base-en-v1.5 embeddings for semantic search! Sounds fun? Let's go!
A full semantic search tutorial about: data mining with requests and beautifulsoup preprocessing in pandas chunking the document text in smaller paragraphs of the right size for the ML model creating embeddings for each chunk calculating the mean embedding for each document saving data as gzipped json (small file size & easy and fast to read in js with pako.js) creating a static web app based on transformers.js on GitHub Pages App here
Let's set up a public Zulip instance for our geospatial community! It's live: zulip.gis.chat
Semantic search right in your browser! Calculates the embeddings and cosine similarity client-side without server-side inferencing, using transformers.js and a quantized version of sentence-transformers/all-MiniLM-L6-v2.
Create a fully working semantic search stack with only Qdrant as vector database with built-in API and transformers.js using any huggingface model as your frontend-only embedding generator. No additional inference server needed!
Using qdrant for querying text data with vector search and geospatial filters without GPU (CPU only)
[Work in progress] Useful PyScript file handling examples for all combinations: local/remote files and import/export.
Use Pandas to execute SQL statements in a remote database such as PostgreSQL via SSH
A simple PyScript-fueled Jupyter Notebook running entirely in your browser - no server needed!
Execute Python right in your Browser!
<!--Load PyScript, Pyodide, Micropip and Python packages-->
<script defer src="https://pyscript.net/alpha/pyscript.js"></script>
<py-env>
- pandas
</py-env>
<!-- Write your Python code! -->
<py-script>
import pandas as pd
from pyodide.http import open_url
df = pd.read_csv(open_url("/downloads/iris.csv"))
df
</py-script>Floral foam 🌼, elephants 🐘 and evil mayors 😈: the useful HyperLogLog algorithm explained as if you were 10 years old!
Create sophisticated walking isochrones for Milan's metro stops and search for flats with immobiliare! Reproducible for all of Italy!
Control all functions of your external monitor with a click on your (wireless) keyboard! Lower brightness, switch input, raise contrast or switch power modes with Python and Monitorcontrol.
Create a free skyline for any city on earth!
Basic River Animations with QGIS, Aerialod and Blender - Mosel from Trier to Traben-Trarbach
Pretty Paper (Mario) Maps quickly done with QGIS and Aerialod.
OSM data offers a vast amount of information about our cities. In this guide I focus on working and styling OSM building data for 3D-modeling in Aerialod for close-to-reality models. Tl;dr: Use OSM building levels, emulate roofs with shapeburst effect and use Sketchfab for interactive 3D models
There are few cities providing open source LIDAR data for 3D-modeling. However, leveraging OSM building, road and landuse data gives you neat city models in no time!
Creating 3D models from raster or vector data has never been that easy! This tutorial takes you through the whole process from vector/raster data to 3D in just 3 steps.
Setting up proper port forwarding for SSH connections, servers or just test purposes is crucial but isn't as straight forward as your connection should be. This post aims to be a quick guide for different use cases.
Python and pandas are a great team for data science. But what if you need to deliver a script to a Windows client who does not have Python installed? This is a quick tutorial for converting Python scripts (.py) to standalone, one-file Windows executables (.exe). Get a 20Mb exe file that is fully functional on every Windows machine!
pyinstaller -F my_script.pyHave you ever wondered how HyperLogLog works? Or have you never heard of it at all? In this post I explain the wonderful algorithm of Flajolet et al. from scratch and in a very simple manner.
Flat hunting can be a big pain - particularly for European metropoles. Most flat search platforms already offer good filter options but until now, I didn't see any with custom geographic filters. Apart from drawing your own area of interest on a map, you seldomly find filters such as "How far is the next metro stop?" or "How close are important facilities?". This article will show a quick and universal workaround for custom geographic filters based on the fantastic pure javascript GIS Turf.js. I implement a buffer based on Milano's metro stops, verify that the flat is located within the buffer and open the respective offer in a new browser tab.
Leaflet-Hexbins is a powerful tool to aggregate data and visualize it quickly on a map. It can size the hexbin radius based on a feature values. But what if we used HyperLogLog data to preserve user-privacy? This post will provide a convenient way to quickly perform on-the-fly HyperLogLog unions for leaflet-hexbins. No more backend-constraint for HyperLogLog There are many HyperLogLog implementations out there but few work as well as postgres-hll and js-hll. They can even talk to each other by exchanging hexstrings! Usually postgres-hll is pretty much the standard extension to use, when dealing with HyperLogLog but is obviously bound to a postgres database running somewhere. With js-hll, the backend-constraint for HyperLogLog is gone, just so that users can perform any HyperLogLog action in their frontend!
HyperLogLog is a fantastic algorithm to quickly count distinct elements in a set. As it consumes few memory and is really fast, it's perfectly suited for big data. For a nice, visual explanation step over to the Redis guys on Youtube. However, this article is not about explaining HyperLogLog but rather understanding how to deal with backend/frontend communication between javascript and postgres by exchanging hexstrings between the two amazing libraries postgres-hll for postgres and js-hll for javascript.
hex_array = ["/x138b7f010101c101e3024104e1058305a105c3072207810922098209a209c30b220c410ce20e210e410e610ee11002106310a210e31323138513a213c2142114421463156115a115e416011661168316a216e1186218a31927194119611aa81b411b611be11c411d431dc21e22218121a121e122a322e12324236123c224c12641266127c3288329442a012a612b012be22cc52d412e612e812fa12fc1302131843282338133e335023541356235a23623364138233863388139813a023a623a823d033d413e613ea13f4140a1414141e242014322438143a24406458145a145e546e14721474248014822486249214a234ae14b414b614be34ca24cc14d424d614da14e214fa250225101514552e453215423548154c355c1564558e3590359415b215b415d625da25f225f836122614162426422648164a16681672367c168016822686168a169216943698169c56b426b816d016e616ee16f416fc17102718471e17326742474e177027742776277e2782378a27901794179817a027a617aa17b217b817c627ce57d817dc47e837ee3800280848181826582a382c1832584e18502854285628607864187e5880188618883898289e28a248a438aa18ac18ae28b028b218b818ca28e028ea58f278fe4904190e192c192e1932193e2940194e2958395c2962296a496e198229923994499e19a619aa59ae39be29ce29e219e419ee49f039f219f63a001a041a181a1c2a202a241a341a462a483a4e3a501a564a5c1a622a6e1a801a867a941a981aaa3aae1ab24abe1ac01acc2ad62ae01ae21afc1b001b082b0a2b103b182b1e2b2a2b2e2b321b3a2b541b581b5e4b6e1b7e1b804b821b922ba21ba41ba83bbe3bc41bc82be04bec3bf01c021c0a2c204c262c2e1c3c1c542c681c782c861c8a1c923ca21cac1cf61d042d0a2d1a2d222d3e1d5a1d5e2d681d6e2d741d7e1d822d961d9c2da03da61da81db42db63db81dc01dd27dd44dde3de22dea1dec4df69dfe1e043e082e162e281e301e321e3c7e521e562e5a2e6a8e6c4e721e742e764e782e821e8e1e925ea45eac2eb81eba5ecc1ece4edc2ede2ee02eea1eec3eee1ef83efa1f082f0a7f121f1c1f2c3f3c2f405f464f501f521f642f661f682f6a3f724f781f8e1f924f981f9a1fa01fbe1fc03fc41fcc2fdc2fe81fea1ffc3ffe1",
"/x128b7f94c64fc29302fedb95064f4594429faea211348181b84bf6a4de1a560e2cde5aae10a82d9b23d04cb37315171da42c7eb634d69b7206eda3b9f61329abd41006c3754ff2b383fe29c5293f8d8dc2dd0ac6e0d283ed4c464ecb1077494c2d63bed629d2966cf0f74ed6d1c433d793c977e7c3ef6d62b062dae87c30eb549046fbed1a115578bfd959f7defd537345d9dc02d1bcbb389f666a036c085757a5147d0bde87fd03cfff7c0c2f46328f046dc50df9247d5d7c02450e7e599707200bef1993af1b2a2520ce27ae169919eb38aa27e882def7ec080c2846cff817fe669530085d7134cb022b3176150ff4e07b5b339241a82324a9d54dbf805d46768d8955167824ea33a3fe55daffbb24f17f5a58c22d17fbba79b05aa808f534be42765f7e8940497ef0c0601e98e71202a33e6fa844e45f16e3f97ad2759531cfe48d7cb25be453a9c8b0",
"/x128b7f8592206f9f19e59c8fd86aefaa33159392aa0e34f0ec18369fe0b373f4f8fb0ea160a594697af83ca209824fdf11dc96a2691b9bdb7a5d68b27eec512b56e437b5a25cb801028cc2b708aa8ad616ef9db8ec683405749d5cbf1cb31e793ca24cc39099abfbdefae9c3b77288c737593bc5e8fbd70be21d25cbecf5f3fc316718ccc47bada8f1a802d0ecb268db22971edd5556b6ef9c83b3deb153736583554efa9f8831cd825d37fbaa507b619aa7f8fcd9e3c86352460202568446a3b4e7ab0baed2436665893018a5739a57a4afe81b57440b7edbb2801c9b4f2bb609d6291d2a2f4022fdf5b9228fe694968e225b232cb3ef34aa283929b266e071ff51d32cd41d8079ca04032f58fb7330727f962ff990b545c1aa3e341c639ae719f1fb392c8bdaf8e352b63eaa1a73ec6d0878407f68d908e2138840a56c50cb1d62db4adfed6510fd513a4f2e41693e7f609258fa6eb4cca8c8fa5c4637c28a62b27b5dec7b773dbc1d0560c2eafcc1e091236bde80216b9972156f4f9d7bda221f0e7b53f009355d4be47ccca7e798abb1267e1fda106584187a"]
hex_array.reduce(function (first, second) {
return first.union(
hll.fromHexString(
second.replace(/\\\\/g,"/"))
.hllSet)}, new hll.HLL(11, 5)
).cardinality();
// 567Querying Postgres with Python Fastapi Backend and Leaflet & Geoman Fronted - Applying Geometry Filters How to query a database with a user-defined geometry drawn on a Leaflet frontend? A very common use case for map applications are custom filters. Either thematic, temporal or value-based filters play a significant role but probably in a geographical context the most dominant one might be spatial filtering. In this blog post, I describe a simple boilerplate setup based on postgres, fastapi and geoman.
Leaflet Data Visualization Strategies - Markers, Marker Clusters, Heatmap, Hexbins
My simple blog routine: how to create a post, preview and commit to production.
██████╗ ███████╗ ██████╗ ██████╗ ██████╗ ██████╗██╗ ██╗███████╗
██╔════╝ ██╔════╝██╔═══██╗ ██╔══██╗██╔═══██╗██╔════╝██║ ██╔╝██╔════╝
██║ ███╗█████╗ ██║ ██║ ██████╔╝██║ ██║██║ █████╔╝ ███████╗
██║ ██║██╔══╝ ██║ ██║ ██╔══██╗██║ ██║██║ ██╔═██╗ ╚════██║
╚██████╔╝███████╗╚██████╔╝██╗██║ ██║╚██████╔╝╚██████╗██║ ██╗███████║
╚═════╝ ╚══════╝ ╚═════╝ ╚═╝╚═╝ ╚═╝ ╚═════╝ ╚═════╝╚═╝ ╚═╝╚══════╝
How to start a background thread in Python to download images from an API.
import threading
def start_thread(func, name=None, args = []):
threading.Thread(target=func, name=name, args=args).start()Port Forwarding in Virtual Box - Postgres example
A quick guide to setup a SSH-conncection to Virtual Machine Ubuntu Server 20.04
import requests
response = requests.get("https://webscraper.io/test-sites/e-commerce/allinone")
responseClipping a geodataframe and displaying it on a basemap in Jupyter.
Use your own SVG as a QGIS marker!
Scripts and tricks to preprocess Instagram data mined with Fast-Instagram-Scraper. Get it on a map!
A fast and efficient Instagram Scraper based on Torpy. Scrapes posts for multiple hashtags and location ids.
Setting up a readthedocs wiki like 1,2,3!
Debugging Python with pdb 🐞
import pdb
from pdb import set_trace as bp
bp()How to write your DYI Instagram Scraper
Quick and dirty
How to quickly mine location IDs from Instagram's location explorer.
{
"Jena": "673710",
"Jena-Zentrum": "1006125655",
"Plan B Boulderhalle Jena": "613227777",
"Kassablanca Jena": "5752477",
"Kulturarena Jena": "574057843",
"University of Jena": "251848169",
"Paradiespark Para - Jena Paradies": "228419929",
"Ernst-Abbe-Sportfeld": "326665551",
...
}Making use of Python and Pandas to quickly process mined social media data of instagram-scraper. Get most common hashtags!
A personal shortlist of well-working Python-based scrapers in 2020.
Automating a daily powershell git routine.
.\gA daily routine of cd'ing in a directory and starting a server can easily be automated.
. x#1 Line endings CRLF vs. LF Theres a difference in how line endings are encoded. If you use VS Code and git, take care that line endings are handled the right way.