Developers¶

Importing lbsntransform as a package¶

For in-memory conversion, it is possible to import lbsntransform as a package:

import lbsntransform as lt
lt.add_processed_records(
    record)
lt.store_lbsn_records()

As a starting point, have a look at lbsntransform/main.py, which includes the code that is invoked on command line use.

We plan to update this section with a Jupyter Lab example notebook.

Example tool "twitterparser"¶

An example program twitterparser that uses lbsntransform as a package is available here.

The tool demonstrates an (imaginary) custom processing pipeline for Twitter data, stored in ZIP files on a remote server. The twitterparser tool will (1) connect to this remote server via SSH, (2) pull zip files and extract the json files locally and (3) transfer json files to a remote lbsn-raw or lbsn-hll database using lbsntransform.

Use this tool as a template or starting point and modify __main__ to your needs.

Contribute¶

lbsntransform is in an early stage of development.

You can contribute:

Feedback, submit issues for bugs etc. on Github
Improve the concept, see lbsn.vgiscience.org and get in contact with use
Provide custom mappings for different data sources

Structure of the project¶

The project structure follows the src-layout.

The packaging is organized as described in the setuptools declarative config (pyproject.toml)

Releases are made with python-semantic-release. At the moment, releases are triggered manually after certain progress is available. Preview release flow with:

semantic-release publish --verbosity=DEBUG --noop

Without --noop, semantic-release will do the following:

Update changelog file.
Run semantic-release version.
Push changes to git.
Run build_command (python -m build) and upload the distribution files to Pypi.
Run semantic-release changelog and post to Gitlab/Github.
Attach the files created by build_command to the release.

To trigger a test build:

pip install -q build
python -m build