Life Hacking 2.0: Adding Geographical Filters to Flat Search Websites with QGIS and Python

Create sophisticated walking isochrones for Milan’s metro stops and search for flats with immobiliare! Reproducible for all of Italy!

Intro

Looking for a flat, I encountered the problem before: if you live in a big city, good public transport is a must. However, search engines for housing seldomly offer necessary geographic filters like “max 7 mins to the next metro stop”. It’s for sure a knock-out criterion but time-consuming to check individual ads for their location.

That’s the reason why already last year I took on idealista.it and implemented a pure frontend geographical filter function with turf.js, all in javascript. If you didn’t read it yet go ahead and take a look: Life Hacking: Adding Geographical Filters to Flat Search Websites with Turf.js

This time I will focus on a different platform: immobiliare.it.

Idea

Find all offers on immobiliare

1. within your custom area of interest and more importantly
2. within 7 minutes walking distance of multiple metro stops

Isochrones

Immobiliare already offers an open end-point for walking/driving isochrones (the accessible area from a given point in a certain time) or a radius which is indeed quite useful.

1

https://www.immobiliare.it/api-au/geocode/isolines?locations=9.1896,45.45986&range=7&transportation=foot&simplify=0

Metro Stops Buffer

While writing this post, immobiliare actually implemented a big part of what I wanted to do - bad luck for me but good for everyone in the end. 😄

You can simply select the metro stops you like to be buffered with a fixed radius of ~500m. Definitely a cool feature and for me the first time I see it on any platform!

However our aim is more complex as we would like a geometry filter with customizable walking isochrones for all desired metro stops or other points of interest like schools or supermarkets within our area of interest.

In order to keep it simple, I will use their dedicated API endpoint for custom AOIs. This offers the big advantage that by the end of this tutorial you can create your own filter polygon once (or use mine) and check it regularly on immobiliare if you like!

No external software or coding required - do it once, bookmark the link and use it until you hopefully find a nice flat!

Trial and Error

I tried to find a way to combine multiple geometries in their API but it’s either not (yet) implemented or not allowed, i.e. you cannot combine either of the following parameters.

Metro stop parameter:

fkMetro[0]=10&fkMetroStation[0]=1173

Radius:

raggio=500¢ro=45.47829%2C9.156734

Custom area of interest:

vrt=45.468919%2C9.167576%3B45.454592%2C9.183025%3B45.464465%2C9.20002%3B45.474698%2C9.172554%3B45.468919%2C9.167576

E.g. not working:

fkMetro[0]=10&fkMetroStation[0]=1173&vrt=45.468919%2C9.167576%3B45.454592%2C9.183025%3B45.464465%2C9.20002%3B45.474698%2C9.172554%3B45.468919%2C9.167576

Hands on in QGIS!

Enough web dev stuff, let’s get into QGIS!

OpenStreetMap or Open Source Data

Let’s get some data. Either you could retrieve the data from OSM or in this case directly from Milan’s open data portal.

Find the metro data sets here or download the GeoJSON directly here.

If the link might break over time, download my 💾 local copy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

{
	"type": "FeatureCollection",
	"name": "tpl_metrofermate",
	"crs": {
		"type": "name",
		"properties": {
			"name": "urn:ogc:def:crs:OGC:1.3:CRS84"
		}
	},
	"features": [
		{
			"type": "Feature",
			"properties": {
				"id_amat": 889,
				"nome": "TRE TORRI",
				"linee": "5"
			},
			"geometry": {
				"type": "Point",
				"coordinates": [
					9.156675327340008,
					45.47814018467403
				]
			}
		},
    ...

Limits

Before processing we need to know what the result needs to look like. There are certain limits:

• As later we will use a normal GET-Request in immobiliare for the final results, the URL length is limited to ~2000 characters.
• Apart from that, immobiliare can only digest a normal polygon not a multi-polygon or similar which means that the individual geometries must be connected and “holes” are not permitted.

Geo-Processing in QGIS

1. Load all your points of interest (POIs)

Load the GeoJSON in QGIS. At this point you could add any other POIs like schools for your kids, supermarkets or you work place. If so, add the layer to QGIS and Merge vector layers. After doing so you will end up with a point geometry layer. Add an OSM base map for better overview e.g. with Quick Map Services Plugin.

2. Isochrones for all your POIs As immobiliare provides an open API for those isochrones one could use it and create individual isochrones for each metro stop coordinate.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

import requests
import time 

distance_mins = 7

# load from remote or local file
metro_stops_url =  "https://dati.comune.milano.it/dataset/b7344a8f-0ef5-424b-a902-f7f06e32dd67/resource/dd6a770a-b321-44f0-b58c-9725d84409bb/download/tpl_metrofermate.geojson"
geojson_metro_stops = requests.get(metro_stops_url).json()
# geojson_metro_stops
# {'type': 'FeatureCollection',
# 'name': 'tpl_metrofermate',
# 'crs': {'type': 'name',
#  'properties': {'name': 'urn:ogc:def:crs:OGC:1.3:CRS84'}},
# 'features': [{'type': 'Feature',
#   'properties': {'id_amat': 889, 'nome': 'TRE TORRI', 'linee': '5'},
#   'geometry': {'type': 'Point',
#    'coordinates': [9.156675327340007, 45.47814018467403]}},...

metro_stops_coordinates = [i["geometry"]["coordinates"] for i in geojson_metro_stops["features"]]
# metro_stops_coordinates
# [[9.156675327340007, 45.47814018467403], [9.192601107884476, 45.492664401028385], [9.15591440789601, 45.46795010103326],...

# forms API URL
def req_link(lon, lat, distance_mins):
    isochrones_api_url = "https://www.immobiliare.it/api-au/geocode/isolines?locations={},{}&range={}&transportation=foot&simplify=0".format(lon,lat,distance_mins)
    return isochrones_api_url

# requests geojson
for ind,coordinate in enumerate(metro_stops_coordinates):
    coodinate_isochrone = requests.get(req_link(coodinate[0],coodinate[1],distance_mins=distance_mins )).json()
   
    with open('isochrones/{}.geojson'.format(ind), 'w', encoding='utf-8') as f:
        json.dump(coodinate_isochrone, f, ensure_ascii=False, indent=4)
        
    # be polite, don't abuse and make adequate breaks in between! 
    # else your IP will likely get blacklisted
    time.sleep(5)

The script saves all isochrones to individual GeoJSON files in isochrones directory. You should end up with a folder of 110 (at the time of writing) metro stop isochrones.

When loading them in QGIS, you end up with a nice confetti map.

3. Processing isochrones

1. Merge vector layers and dissolve the geometries.

Nicer, high-res version made with mapbox for #30daymapchallenge 2022

2. Enter editing mode and clean up the geometry and remove tiny holes with vertex tools.

Most of the holes are relatively tiny and wouldn’t make a big difference. So for simplicity just remove them.

3. Simplify inner hole.

Piazza Sempione is relatively far away from the closest metro stop and hence must be kept.

4. A surgical cut

Piazza Sempione is surrounded by accessible areas. In order to create a normal polygon, we need to make a tiny surgical cut.

Create a New Layer and switch to Edit Mode and Add Line Feature. Buffer the line and Clip the former dissolved layer with the buffered line.

5. Link separate geometries

Milan’s city center has a high density of metro stops whereas in the periphery it becomes more sparse.

Again, create a New Layer and switch to Edit Mode and Add Line Feature. This time, link all polygons with a line, then Buffer, Merge vector layers, Dissolve.

6. Clip to your area of interest

Theoretically, this polygon could be used to filter immobiliares database. However, there way too many vertices - to be exact 1191 in my version. Remember that the URL has a limit of 2000 characters and that you might need additional filter criteria reducing the limit further.

Download the full polygon 💾 here for your own processing.

Hence, add a new layer and clip to your area of interest, e.g. the center. Simplify your polygon to well below 100 vertices maximum and export as GeoJSON with coordinate precision 6.

Download the center isochrones file 💾 here.

Convert GeoJSON to URL

The geometry is finished but we still need to transform the geometry to the URL parameter. As GeoJSON’s standard is lon lat but the URL requires lat lon, they need to be switched - how could it ever be any different? 😅

Python offers the quickest solution through a simple list comprehension.

1
2
3
4
5
6
7

import json
with open('path/to/your/file.geojson') as f:
    d = json.load(f)
coordinate_list = d["features"][0]["geometry"]["coordinates"][0][0]

switched_coordinate_list = [[coordinate[1],coordinate[0]] for coordinate in coordinate_list]
switched_coordinate_list

Output

1
2
3
4
5
6
7
8

[[45.456496, 9.164258],
 [45.45892, 9.163886],
 [45.461597, 9.164185],
 [45.463215, 9.163448],
 [45.464146, 9.162457],
 [45.463446, 9.157376],
 [45.462264, 9.15147],
 [45.463216, 9.148628],...]

Now, simply replace the brackets with a semicolon and remove white spaces.

1

str(switched_coordinate_list).replace("], [",";").replace(" ","").replace("[[","").replace("]]","")

Output

1

'45.456496,9.164258;45.45892,9.163886;45.461597,9.164185;45.463215,9.163448;45.464146,9.162457;45.463446,...'

Perfect, now set all filter criteria in immobiliare as you would usually, draw a random area, confirm and copy the link from the adress bar, e.g.

https://www.immobiliare.it/search-list/?vrt=45.487817%2C9.158478%3B45.450738%2C9.171524%3B45.460131%2C9.220963%3B45.488539%2C9.202423%3B45.494796%2C9.176674%3B45.487817%2C9.158478&idContratto=1&idCategoria=1&prezzoMinimo=100000&prezzoMassimo=200000&fasciaPiano[0]=30&criterio=rilevanza&ordine=desc&__lang=it&pag=1&slau=1

Now just replace the parameter vrt value with your polygon. In my case with 1967 characters it’s quite close to the limit of 2000 but that’s fine. If does not work for you, go back to step 6 and delete a few vertices.

https://www.immobiliare.it/search-list/?vrt=45.456496,9.164258;45.45892,9.163886;45.461597,9.164185;45.463215,9.163448;45.464146,9.162457;45.463446,9.157376;45.462264,9.15147;45.463216,9.148628;45.460541,9.14655;45.468674,9.13902;45.469802,9.140308;45.469105,9.144454;45.47023,9.145494;45.474656,9.145275;45.474482,9.14419;45.47216,9.142806;45.471397,9.138555;45.483639,9.145586;45.484973,9.147021;45.484778,9.152603;45.484815,9.155547;45.483573,9.157393;45.484913,9.159909;45.484866,9.160985;45.480681,9.168495;45.480097,9.166951;45.474691,9.15999;45.472783,9.160403;45.473145,9.163534;45.470972,9.171185;45.471792,9.171776;45.472078,9.171994;45.475866,9.178482;45.477785,9.178044;45.47925,9.179298;45.480144,9.180053;45.481168,9.177098;45.481812,9.175075;45.483165,9.173279;45.483239,9.172661;45.483384,9.171153;45.481006,9.169355;45.480687,9.168512;45.484882,9.160986;45.485488,9.160992;45.487571,9.161359;45.488582,9.164908;45.490369,9.1663;45.491667,9.167583;45.491815,9.170326;45.490345,9.173731;45.487782,9.175482;45.488477,9.177381;45.488747,9.17949;45.489065,9.18272;45.487269,9.184977;45.489804,9.18565;45.490669,9.186873;45.492689,9.18594;45.492549,9.184665;45.493092,9.182134;45.495275,9.179632;45.497263,9.177677;45.501442,9.180966;45.501444,9.180959;45.502305,9.183307;45.502139,9.185286;45.50176,9.188452;45.499484,9.191513;45.489199,9.208754;45.479921,9.217512;45.476869,9.216619;45.476206,9.214805;45.476049,9.211259;45.474182,9.211721;45.470477,9.208353;45.470473,9.208353;45.463891,9.201273;45.46239,9.198067;45.461006,9.194904;45.456723,9.185035;45.457738,9.183727;45.459514,9.18312;45.461393,9.183258;45.461453,9.182297;45.462486,9.180391;45.464374,9.179563;45.463698,9.178427;45.463698,9.178423;45.460538,9.1795;45.458501,9.178988;45.458116,9.178808;45.457455,9.17538;45.45589,9.174711;45.456496,9.164258&idContratto=1&idCategoria=1&prezzoMinimo=100000&prezzoMassimo=200000&fasciaPiano[0]=30&criterio=rilevanza&ordine=desc&__lang=it&pag=1&slau=1

And there you go! Enter the link and see all the filtered results. You can simply bookmark the link or even edit the nodes directly in the online editor!

Use your own POIs!

As mentioned in the intro, instead of only using metro stops, one could mix up different POIs like e.g. schools or tram/bus stops, your workplace or the sport facilities as well. It’s a powerful workflow that hopefully helps you to find a flat sooner and with less overhead! Also, if you don’t want to use walking isochrones, you can use driving isochrones or even mix them up. If you want to keep it simple, you could always work with a normal QGIS buffer in meters instead.

If you like this tutorial, if it helped you or if you have more ideas on how to improve the workflow let me know on Twitter or LinkedIn!