The LBSN structure is in an early stage of development.
We invite anyone to contribute.
Generally, we aim for the following conventions and principles:
Single Point of Truth
Definitions should exist in one place only. We use continuous integration workflows to systematically integrate modules and deploy individual pieces. For example, the code examples and definitions in this documentation are automatically generated from tested code.
Privacy by Design, Privacy by Default, and Separation of Concerns principle
All of the above principles help reducing the risk that privacy of any user is compromised. For example, those who develop tools do not need to access original data generated by users (separation of concerns principle).
Transparency is a key when working with user-generated content, and open source code means transparency. The documentation and all pieces of software used in this project are published as open source. This guarantees a sustainable, future proof development cycle.
The core lbsn structure is described in a platform independent Protocol Buffers file. The Proto file can be used to compile and implement the proposed structure in any language such as Python, Java or C++.
This structure is tightly coupled with a relational datascheme (Postgres SQL) that is maintained separately, including a privacy-aware version that is used for demonstration purposes. The database is ready to use with several provided Docker containers that optionally include a PGadmin interface and a Jupyter Lab visualization backend.
There exist several tools that help migrating data and maintaining the structure. The python package lbsntransform allows bulk conversion and database update of large volumes of proprietary data (e.g. CSV, JSON) that was retrieved from various API endpoints (e.g. Twitter, Flickr).
A fullstack docker project generator is available that includes Postgres, pgadmin, a web api, a frontend and other features (such as Alembic database migrations).
This documentary illustrates the various use cases, definitions and applications. Contributions welcome.
Overview of Repositories and Tools
The following list provides an overview of tools and applications that make use of or are connected to the lbsn-structure.
|LBSN Base Structure:|
|RAW SQL||Relational PostgresSQL implementation of LBSN Structure|
|HLL SQL||A Privacy-aware version of RAW SQL, e.g. for visual analytics|
|Protocol Buffers||Defenition of LBSN Structure in Protocol Buffers, e.g. for use in RPC|
|Python Compiled||A Python compiled version of Protobuf LBSN Structure|
|PHP Compiled||A PHP compiled version of Protobuf LBSN Structure|
|LBSN Docker Container:|
|RAW DB||A ready to use Docker Container using the LBSN RAW SQL structure|
|HLL DB||A ready to use Docker Container using the (privacy-aware) LBSN HLL SQL structure|
|pg-hll-empty||An empty Postgres Docker container with HLL extension installed, e.g. used in lbsntransform to anonymize data|
|jupyter lab||A containerized version of Jupyter Lab, used for interactive visualization purposes, e.g. in the tutorial section|
|lbsntransform||A python package for bulk data migration to and from lbsn structure (both Postgres and ProtoBuf specification)|
|fullstack-lbsn||An easy to use shell-script to setup RAW DB and HLL DB Docker Container, including data export and import|
|LBSN API (theplink-docker):|
|Alembic Migrations||Automatic migrations of live PostgresSQL with updates to the RAW LBSN Structure|
|Backend||Backend using Fastapi and LBSN Structure implemented as a SQLAlchemy model|
|Frontend||Frontend as a progressive web app based on VueJS and Mapbox GL JS, a prototype for local community discourse using the LBSN Structure|