The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

This document outlines the features of the {laminr} package and the roadmap for future development.

Features

Connect to an instance

• Connect to a LaminDB instance (connect()).
• Handle authentication and authorization.
• Connect to a LaminDB instance without needing to install the lamin_cli Python package.

Query & search

• Query exactly one record (Registry$get(...)): Fetch a single record by ID.
• Query sets of records (Registry$filter()): Fetch multiple records based on filters.
  • $df(): Returns a data frame with each record in a row.
  • $all(): Returns all records as a QuerySet.
  • $one(): Return exactly one record.
  • $one_or_none(): Return one record or NULL.
• Leverage relationships when querying (Artifact$filter(created_by__handle__startswith = "testuse")$df()): Query records based on relationships.
• Comparators: Use comparators in filters.
  • and: Example: Artifact$filter(suffix = ".jpg", created_by = user)
  • less than / greater than: Example: Artifact$filter(size__lt = 1e4)
  • in: Example: Artifact$filter(suffix_in = [".jpg", ".fastq.gz"])
  • order by: Example: Artifact$filter().order_by("created_at")
  • contains: Example: Artifact$filter(name__contains = "test")
  • startswith: Example: Artifact$filter(name__startswith = "test")
  • or: Example: ...
  • not: Example: ...
• Search for records (Registry$search(...)): Search for records based on a query string.
• Pagination: Support pagination for large query results.
• Field lookups: Provide convenient functions for looking up field values (e.g., Artifact$lookup("description")).

Manage data & metadata

• Create artifacts: Create new artifacts from various data sources (e.g., files, data frames, in-memory objects).
  • $from_df(): Create an artifact from a data frame.
  • $from_path(): Create an artifact from a path.
  • $from_anndata(): Create an artifact from an AnnData.
• Save artifacts: Save artifacts to LaminDB with appropriate metadata.
• Load artifacts: Load artifacts from LaminDB into R:
  • csv: Load a data frame from a CSV file.
  • fcs: Load flow cytometry data.
  • h5ad: Load an AnnData from an HDF5 file.
  • h5mu: Load a MuData from an HDF5 file.
  • html: Load content from an HTML file.
  • jpg: Load an image from JPG.
  • json: Load data from a JSON file.
  • parquet: Load a data frame from a Parquet file.
  • png: Load an image from PNG.
  • rds: Load an R object from an RDS file.
  • svg: Load an image from SVG.
  • tsv: Load a data frame from a TSV file.
  • yaml: Load data from a YAML file.
  • zarr: Load an AnnData from a Zarr store.
• Cache artifacts: Cache artifacts locally for faster access:
  • s3: Interact with S3 storage.
  • gcp: Interact with Google Cloud Storage.
• Version artifacts: Create new versions of artifacts.
• Delete artifacts: Delete an existing artifact.
• Manage artifact metadata: Add, update, and delete artifact metadata.
• Work with collections: Create, manage, and query collections of artifacts.
• Stream backed artifacts: Connect to file-backed artifacts ($open).
  • tiledbsoma: Stream TileDB-SOMA objects

Track notebooks & scripts

• Track code execution: Automatically track the execution of R scripts and notebooks.
• Capture run context: Record information about the execution environment (e.g., package versions, parameters).
• Link code to artifacts: Associate code execution with generated artifacts.
  • Link to artifacts loaded from other instances
• Visualize data lineage: Create visualizations of data lineage and dependencies.
• Finalize tracking: End and save a run.

Curate datasets

• Validate data: Validate data against predefined schemas or constraints.
• Standardize data: Apply standardization rules to ensure data consistency.
• Annotate data: Add annotations and labels to data.
• Use the Curator class: Implement the Curator class for a streamlined curation workflow.

Access public ontologies

• Access ontology data: Fetch data from public ontologies (e.g., gene names, protein IDs).
• Search ontologies: Search for entities within ontologies.
• Use ontology terms in queries: Use ontology terms to filter and query data.
• Manage ontology versions: Access different versions of ontologies.

Manage biological registries

• Create and manage records in bionty registries: Add, update, and delete records for genes, proteins, cell types, etc.
• Utilize hierarchical relationships: Navigate and query based on parent-child relationships in ontologies.
• Manage synonyms: Add and use synonyms for biological entities.

Manage schema modules

• List available modules: Retrieve a list of available modules in an instance.
• Access module registries: Access registries within specific modules.
• (Advanced) Create custom modules: Define and register custom schema modules.

Transfer data

• Upload data: Upload data files to LaminDB storage.
• Download data: Download data files from LaminDB storage.
• (Advanced) Support zero-copy data transfer: Implement efficient data transfer mechanisms.

Roadmap

Version 0.1.0

A first version of the package that allows users to:

• Connect to a LaminDB instance.
• List all records in a registry.
• Fetch one record by ID or UID.
• Cache S3 artifacts locally.
• Load AnnData artifacts.

Version 0.2.0

• Implement basic data and metadata management features (create, save, load and delete artifacts).
• Expand support for different data formats.
• Implement code tracking.

Version 0.3.0

• Track input artifacts.
• Support for more storage backends using a reticulate Python backend.

Version 0.4.0

• Expand support for different storage backends.
• Expand query functionality with comparators, relationships, and pagination.
• Implement data lineage visualization.
• Introduce data curation features (validation, standardization, annotation).
• Enhance support for bionty registries and ontology interactions.
• Connect to TileDB-SOMA artifacts.

Future versions

• Implement advanced features like custom module creation and zero-copy data transfer.
• Continuously improve performance, usability, and documentation.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.