Importing lbsntransform as a package¶
For in-memory conversion, it is possible to import lbsntransform as a package:
import lbsntransform as lt lt.add_processed_records( record) lt.store_lbsn_records()
As a starting point, have a look at lbsntransform/main.py, which includes the code that is invoked on command line use.
We plan to update this section with a Jupyter Lab example notebook.
Example tool "twitterparser"¶
An example program
twitterparser that uses lbsntransform as a package is available
The tool demonstrates an (imaginary) custom processing pipeline for Twitter data,
stored in ZIP files on a remote server. The
twitterparser tool will
(1) connect to
this remote server via SSH,
(2) pull zip files and extract the json files locally and
transfer json files to a remote lbsn-raw
or lbsn-hll database using lbsntransform.
Use this tool as a template or starting point and modify
__main__ to your needs.
lbsntransform is in an early stage of development.
You can contribute:
- Feedback, submit issues for bugs etc. on Github
- Improve the concept, see lbsn.vgiscience.org and get in contact with use
- Provide custom mappings for different data sources
Structure of the project¶
The project structure follows the src-layout.
The packaging is organized as described in the setuptools declarative config (pyproject.toml)
Releases are made with python-semantic-release. At the moment, releases are triggered manually after certain progress is available. Preview release flow with:
semantic-release publish --verbosity=DEBUG --noop
--noop, semantic-release will do the following:
- Update changelog file.
- Run semantic-release version.
- Push changes to git.
- Run build_command
python -m build) and upload the distribution files to Pypi.
- Run semantic-release changelog and post to Gitlab/Github.
- Attach the files created by build_command to the release.
To trigger a test build:
pip install -q build python -m build