Python on Datengraben

IPython can query or export command statement history

Mon, 01 Jan 0001 00:00:00 +0000

Today i learned that you can use the “magic” %history command in IPython to query the command history of the active session but also all the recorded past sessions.

For example you can go like %history 999/1-500 -f /tmp/outfile to write the lines 1 to 500 of the recorded session 999 to the file at tmp directory.

There are other ways of retrieving the history, but this seems the most elegant to me.

Extract Arcgis Geodatabase format with Python

Fri, 21 Jun 2024 16:29:34 +0200

If you want to know how to extract the individual layers programmatically from a GeoDatabase-ArcGIS archive, then this is for you.

Geo data is often times provided in Geodatabase format. This format provides a container for several layers of data, which then can be rendered using software like ArcGIS or QGIS, which have already plugins to process this file format. But if you work on a lower level, you maybe need to process these files in their format programmatically.

In-memory reading of GML-format data with geopandas

Mon, 01 Jan 0001 00:00:00 +0000

Today I learned another way of parsing geo data formats in-memory, using geopandas.

As part of a running django web service, I wanted to request and parse a WFS respose, which comes back in GML format.

The solution to this is:

gpd.GeoDataFrame.from_features(fiona.BytesCollection(bytes_str))

This was pointed out by sal in this stackoverflow answer. Since I could not find it in the documentation, either in fiona and not in geopandas. I want to try now to add this there.

Mit git-history und Github Actions einen Scraper bauen

Mon, 30 Jan 2023 16:20:23 +0100

Web-Scraping setzt oft voraus, dass ein eigener Rechner verfügbar ist, welcher bestenfalls 24h online 🌐 ist und die Abfragen für das Scrapen absenden kann. Wenn wir das vermeiden wollen, beöntigen wir entsprechende Infrakstruktur. Simon Willison beschreibt in seinem Blogpost git-history ein Python Tool, welches aus einer Historie an git commits zu einer Datei, eine Zeitreihe erstellt. Diese Serie von git commits basieren auf einer speziellen Art zu scrapen. Dieses Pattern vereinfacht Web-Scraping, indem als Infrastruktur kein eigener Rechner genutzt werden muss, sondern Github actions. Die Github actions speichern jede neue unterschiedliche Version einer gescrapten Datei als eigenen commit im Repository.