How to register a Metric

This tutorial explains how to register a community FAIR Metric using the OSTrails FAIR Assessment framework.

A Metric is a narrative description that a Test must wholly implement. Each metric should implement exactly one dimension (e.g. one of the FAIR Principles). They may be domain-agnostic or not. For more information, check the FAIR Testing Resource (FTR) vocabulary.

There are two ways to register a Metric. The first is to use the FAIR Wizard authoring tool, a questionnaire-based knowledge model designed to collect and structure metadata for FAIR Assessment Components, including Metrics. It auto-generates FTR specification metadata and stores it within an OSTrails github repository, then registers it in FAIRsharing for you. If you have not yet created FTR files for your metrics, then you should use this option.

The second is to register your Metric directly with FAIRsharing. If you have already created your FTR specification files for your metrics (eg in your own github repository) then you should use this option.

This tutorial covers both options.

Does your metric already exist?

You should review existing metrics in FAIRsharing for the Principle that you are measuring. If it already exists, then please use that metric in your benchmark rather than creating a new one. To discover the metrics related to a particular Principle, find the Principle in FAIRsharing and explore its relationships.

For example, if you require a metric for F1 ((Meta)data are assigned globally unique and persistent identifiers) that checks the global uniqueness of an identifier, then visit https://doi.org/10.25504/FAIRsharing.a2cea7 and review the list of related metrics. See also the tutorial on finding metrics and benchmarks.

FAIR Wizard

Prerequisites

Before starting you should:

Step 1 – Create a Metric project in the FAIR Wizard authoring tool

  1. Go to the dedicated environment for this questionnaire.

  2. Register yourself or log in if you already have access.

  3. Navigate to Projects and click Create to start a new project.

  4. Name your project and use the “FAIR Assessment Authoring Tool - Questionnaire for creating FAIR Assessment Components” template as Knowledge Model.

  5. Enable Filter by question tags.

  6. Choose Metric as the artefact type.

By doing this, the tool will create a Metric-tailored questionnaire.

Step 2 – Fill in the questionnaire

  1. Read the instructions carefully.

  2. Work through the form sequentially, completing each section with information relevant to the Metric you are defining.

Note that there are questions that are mandatory, which will be required to be given an answer. Other questions are optional. The mandatory fields that are required to define a FAIR Metric are:

  • Title You should indicate the name of your Metric. To follow OSTrails best practices, consider using this Metric naming scheme:

[[Principle name]] Metric - [[Abbreviation of sub-principle]] - [[Metadata|Data depending on the focus of the metric]] - [[descriptive metric name]]

Examples:

FAIR Metric – F1– Metadata - Persistent identifiers for database content

FAIR4RS Metric – F1 - Metadata - Software has persistent and unique identifier (https://doi.org/10.25504/FAIRsharing.87c9a8)

  • Description You should indicate a description of your Metric.

  • Abbreviation You should indicate a single-word abbreviation for your Metric. Note that FAIR Wizard does not allow the use of spaces or any of these special characters in Benchmark/Metric/Test name abbreviations: : / ? # [ ] @ ! $ & ‘ ( ) * + , ; = “< > ^ { < > : “ /

To follow OSTrails best practices, consider using this Metric naming scheme:

[[Principle abbrev]]M_[[Abbreviation of sub-principle]]_[[M|D|P]]_[[short name for the metric]]

Examples:

FM_F1-PID_M_ARK

FSM_F1_M_UNIQ-ID

FM_R1.2_M_CPI

  • License You should include a license URL for this Metric.

  • Version You should indicate the version number you are interested in using for defining your Metric.

  • Responsable contact person You should provide the name and email of a responsible contact person. You will find an ORCID-integrated browser to search for your personal information by either typing in your full name or your ORCID ID directly.

  • Country You should indicate the country or geographical scope relevant to this Metric. If the Metric is not limited to a specific region, you can select ’Worldwide’.

  • Subject You should specify the application domain or area of knowledge to which the Metric applies. If the Metric is intended to be domain-independent, you can select ’Subject Agnostic’.

  • Object type You should indicate the type of digital object that the Metric evaluates (for example datasets, software, or workflows). If the Metric applies broadly, you can select Object type ’Agnostic’.

  • Taxonomy You should classify the Metric within a taxonomy. If no suitable classification is available or needed, you can select ’Not Applicable’.

  • Link to a principle You should link the Metric to at least one FAIR Principle. This defines which aspect of FAIRness the Metric evaluates and is essential for its interpretation and reuse.

Please complete all optional sections that it is possible for you to complete. The more complete your metric, the more re-usable and FAIR it is. Incomplete metadata may delay the publishing of your Metric in FAIRsharing.

Step 3 – Create an instance with your answers

Once the questionnaire has been completed:

  1. Go to the Documents section in the top menu.

  2. Name your document and select the latest version of the “FAIR Assessment Authoring Tool - Jinja2-based template for authoring and registering FAIR Assessment Components” as Document Template.

  3. Choose the “Metric / Benchmark” Format option.

  4. Click on Create.

This will create a JSON file with your input.

Step 4 – Submit your document

Now, you can review the document with your answers to the questionnaire, by clicking on it, which will initiate the download of the file. Once you’re happy with it, you’re ready to submit your document:

  1. In the Documents section, click the three dots icon (⋯) beside your document.

  2. Select Submit.

The submission will be sent via the GitHub API to be registered in an OSTrails GitHub repository for collecting metadata about these assessment components, and indexed by the FAIRsharing registry.

Next steps

Once submitted to FAIRsharing, the record will remain hidden until approved by the FAIRsharing curation team. Once made public, claim your record in FAIRsharing. Information on creating an account and claiming a record is available in the next section, and at https://fairsharing.gitbook.io/

FAIRsharing

This tutorial provides a comprehensive walkthrough for registering a Metric directly within the FAIRassist registry on FAIRsharing.

Prerequisites

  • Ensure you are logged in via your ORCID. This ensures your curation work is publicly attributed to you. You can find out more about creating an account in our gitbook documentation.

  • Create a narrative description of your metric, and how it interprets the FAIR Principle that it measures. You may find the metric sections of the tutorial at define-benchmark-associated-metrics.rst useful.

Creating a record

Please follow the instructions in our documentation on how to create a new record in FAIRsharing. Once you’ve done that, you will be presented with the more detailed record edit interface.

Editing your record

Each of the tables below corresponds to a single tab of the edit interface for a FAIRsharing record, and summarises the key fields that should be populated. For complete documentation, see our gitbook pages.

Remember to save your updates regularly.

General Information

The form in the general information tab establishes the identity, ownership, and scientific scope of your record.

General Information

The form in the general information tab establishes the identity, ownership, and scientific scope of your record.

  • Record Name (Mandatory): Provide the full name of the resource. You should create a name of the format: [[Principle name]] Metric - [[Abbreviation of sub-principle]] - [[Metadata or Data]] [[descriptive metric name]]. An example is “FAIR Metric – F1 – Metadata - Identifier is globally unique”.

  • Abbreviation (Optional): You should create an abbreviation of the format: [[Principle abbrev]]M:[[Abbreviation of sub-principle]]:[[M or D]]:[[short name for the metric]]. An example is “FM:F1:M:IdentUnique”.

  • Homepage (Mandatory): Provide the homepage URL for the resource.

  • Description (Mandatory): Free-text summary of the resource and its purpose; see also our documentation on descriptions. (Min. 40 chars).

  • Year of creation (Recommended): Provide the year the resource was first released.

  • Contacts (Mandatory): At least one contact point should be provided, consisting of a name and email address for the person or group responsible for the maintenance of the resource.

  • Countries (Mandatory): Select the country or countries where the resource is hosted. At least one must be added.

  • Subjects and Taxonomies (Mandatory): Select the relevant subject area and species. At least one of each must be added. “Not applicable” may be used for the Taxonomy value when the species is irrelevant, as is often the case for metrics.

  • Object Type (Mandatory): Define the type of digital research object in scope. At least one object type must be provided for Metrics/Benchmarks.

Publications

Connects the record to the literature related to the metric.

  • Publications (Recommended): This section is only for publications that describe your resource and those you would ask others to use when citing your database, standard or policy.

  • Citations (Recommended): You may have one or more publications that should be used to cite your resource. Note this using the ‘Cite record using this publication?’ toggle.

Organisations and Grants

Defines the institutional backing and funding for the resource.

  • Organisations (Recommended): Each organisation involved should be added with its role. At least one maintaining organisation and one funding organisation should be added.

Relations to Other Records

  • related_to (Recommended): One of the most important parts of a record is its relationships. Link to records (other than benchmarks) via the autocomplete field using the FAIRsharing ID, full name, or short name.

  • measures_principle (Mandatory): Every metric must be linked to exactly ONE principle record from any given principle hierarchy. Use for the narrowest principle possible (e.g. FAIR F1). Cannot point to two principles from the same hierarchy.

Additional Information

Specific functional metadata for assessment tools.