Better identifying conda packages with ClearlyDefined

ClearlyDefined, an Open Source project that helps organizations with supply chain compliance,  now provides a new harvester implementation for conda, a popular package manager with a large collection of pre-built packages for various domains, including data science, machine learning, scientific computing and more.

Conda provides package, dependency and environment management for any language and is very popular with Python and R. It allows users to manage and control the dependencies and versions of packages specific to each project, ensuring reproducibility and avoiding conflicts between different software requirements.

ClearlyDefined crawls both the main conda package and the source code for licensing metadata. The main conda package is hosted on the conda channels themselves and contains all necessary licensing information, compilers, environment configuration scripts and dependencies that are needed to make the package work. The source code from which the conda package is created oftentimes is hosted in an external website such as GitHub.

The conda crawler uses the following coordinates:

  • type (required): conda or condasource
  • provider (required): channel on which the package will be crawled, such as conda-forge, anaconda-main or anaconda-r
  • namespace (optional): architecture and OS of the package to be crawled, i.e. win64, linux-aarch64 or any if no architecture is specified.
  • package name (required): name of the package
  • revision (optional): package version and optional build version

For example, the popular numpy package is represented as shown below.

With the increased importance of data science, machine learning and scientific computing, this support for conda packages in ClearlyDefined is extremely important. It will allow organizations to better manage the licenses of their conda packages for compliance. This work was led by Basit Ayantunde from CodeThink with the stewardship from Qing Tomlison from SAP. We would like to thank them and all those involved in the development and testing of this implementation.

We are looking for feedback. Please test this feature on dev.clearlydefined.io or dev-api.clearlydefined.io and file any issues here.