Overview

Dug Semantic Search is a tool that allows users to deep dive into BDC studies and biomedical topics, research, and publications to identify related studies, datasets, and variables. If you are interested in how Dug connects study variables to biomedical concepts, read the Dug paper or visit the Help Portal.

This tool applies semantic web and knowledge graph techniques to improve BDC research data Findability, Access, Interoperability, and Reusability (FAIR). Through this process, semantic search helps users identify novel relations, build unique research questions, and identify potential collaborations.

The Patient Information Commons: Standard Unification of Research Elements (PIC-SURE) integrates clinical and genomic data to allow users to search, query, and export data at the variable and variant levels. This allows users to create analysis-ready data frames without manually mapping and merging files.

NHLBI BioData Catalyst® (BDC) is a cloud-based ecosystem providing tools, applications, and workflows in secure workspaces. BDC Powered by PIC-SURE (BDC-PIC-SURE) is a platform that allows researchers to explore studies funded by the National Heart, Lung, and Blood Institute, whether the researchers have been granted access to the participant-level data or not.

Using Exploration

The Exploration page located in the upper right-hand section of the toolbar allows users to search through data and create cohorts. The Exploration portal contains a dynamic summary statistics display, as well as search facets leveraging the DCC Harmonized Variables.

Data Accessibility

Users can navigate through data on the Exploration page by selecting any of the three Data Access categories.

• Data with Access: A user can view all of the summary data and associated study information for studies the user has access to, including but not limited to Project ID, file types, and clinical variables.

• Data without Access:

  • Locks next to the project ID signify to users that they do not have subject-level access but they can still search through the available studies but only view summary statistics. Users can request access to data by visiting the .

  • Projects will also be hidden if the select cohort contains fewer than 50 subjects (50 ↓, "You may only view summary information for this project", example below); in this case grayed out boxes and locks both appear. An additional lock means users have no access.

• All Data: Users can view all of the data available in the BDC-Gen3 platform, including studies with and without access. As a result, studies not available to a user will be locked as demonstrated below.

By default, all users visiting the Exploration page will be assigned to Data with Access.

Under the "Data" tab, users can leverage the to create custom cohorts. When facets are selected and/or updated to cover a desired range of values, the display will reflect the information relevant to the new applied filter. If no facets have been selected, all of the data accessible to the user will be displayed. At this time, a user can filter based on three categories of clinical information:

• Project: Any specifically defined piece of work that is undertaken or attempted to meet a single investigative question or requirement.

• Subject: The collection of all data related to a specific subject in the context of a specific experiment.

• Harmonized Variables: A selection of different clinical properties from multiple nodes, defined by the Consortium.

NOTE: The facet filters are based on the DCC Harmonized Variables, which are a selected subset of clinical data that have been transformed for compatibility across the dbGaP studies. TOPMed studies that do not contain harmonized clinical data at this time will be filtered out when a facet is chosen, unless the no data option is also selected for certain facets.

After a cohort has been selected, the user has four different options for exporting the data.

The options for data export are as follows:

• Export All to Terra : Initiate a export of all clinical data and file GUIDs for the selected cohort to . At this time the max number of subjects that can be exported to Terra is 120,000.

• Export All to Seven Bridges: Initiate a export of all clinical data and file GUIDs for the selected cohort to

NOTE: PFB export times can take up to 60 minutes, but often will complete in less than 10 minutes.

The Files tab displays study files from the facets chosen on the left-side panel (Project ID, Data Type, Data Format, Callset, and Bucket Path). Each time a facet selection is made, the data summary and displays will update to reflect the applied filters.

The Files tab also contains files that are either case-independent or project-level. This is important for files that are part of the Unharmonized Clinical Data category under the Data Type field. Unharmonized clinical files are made available in two distinct data formats:

• TAR : Contain a complete directory of phenotypic datasets as XML and TXT files that are direct downloads of unharmonized clinical data from dbGaP on a study consent level project.

• AVRO: These files are the same as the unharmonized clinical data from dbGaP as the TAR files, but in form of a file.

NOTE: The unharmonized clinical data sets contains all data from the dbGaP study, but it is not cross-compatible across all studies within BDC.

Once the user has selected a cohort, there are five options for accessing the files:

• Download Manifest: Download the file manifest and use this manifest to download the enlisted data files using the .

• Export All PFB: Initiate a export of the selected files.

A user can visit the File Information Page after clicking on any of the available GUID link(s) in the Files tab page. The page will display details such as data format, size, object_id, the last time it was updated and the md5sum. The page also contains a button to download the file via the browser (see below). For files that are 5GB or more, we suggest using the .

Both the Data and File tabs contain a text-based search function that will initiate a list of suggestions below the search bar while typing.

In the Data tab, Submitter IDs can be searched under the Subject tab.

In the File tab, File Names can be searched under the File tab.

Click either on a single or on multiple suggestions in the list appearing underneath the search bar to create a cohort and export/download the data. The selections can be again clicked to be removed from the created cohort.

A list of current project IDs can be found in the Data tab, under Filters>Project>Project Id. A sample of current project IDs are:

• Parent

• TOPMed

BDC-RFC-#: 28 Title: DICOM Medical Image De-Identification Baseline Protocol Type: Process Contact Name and Email: Keyvan Farahani, farahank@mail.nih.gov Submitting Teams: NHLBI, DMC Date Sent to Consortium: Oct. 11, 2023 Status: Closed for comment URL Link to this Google Document: https://docs.google.com/document/d/14-WfeMqgZz115DbBnFs-8AvcdRY1oIjCgi0K33pwMjE/edit?usp=sharing License: This work is licensed under a CC-BY-4.0 license.

Medical Image De-Identification: BDC Baseline Protocol

Contributors:

• Zixin Nie (BDC Data Management Core)

• Keyvan Farahani (NHLBI)

• David Clunie (PixelMed Publishing)

De-identification of protected health information (PHI) is often a necessary procedure to undertake in order to share potentially sensitive information, such as health data. Many data repositories that allow human data to be deposited and shared require the data to be de-identified. Medical images and their associated metadata (i.e., DICOM headers) often contain PHI, such as patient names, dates of birth, or medical record numbers. The de-identification of these images is essential to minimize privacy risk and comply with regulations and standards that require the protection of PHI. The overarching goal in medical image de-identification is to reduce the risk of identification as much as possible.

De-identification facilitates the sharing of medical imaging data, enabling greater access by researchers and the public and allowing for secondary research to be conducted. Several standards exist for de-identification of medical images, including the confidentiality profile detailed in the DICOM Part 15 standard, HIPAA Safe Harbor and Expert Determination. The BioData Catalyst Data Management Core (BDC DMC) performed an evaluation of these standards and used them to create the protocol detailed in this document. This document describes the de-identification processes and technical considerations for de-identifying medical images as they are being added to BDC and made available to researchers using the BDC platform. The protocol, referred to as the “BDC Baseline Protocol for Image De-identification,” takes into account the data use cases for researchers accessing the BDC platform by defining a de-identification profile that strikes a balance between privacy protection and preserving utility.

The Baseline protocol only applies to the metadata in radiologic (DICOM) images (see table below). It does not apply to image pixel information, other imaging formats, or other types of data that may be imported into BDC, such as clinical and omics data. It reflects the understanding of the de-identification needs of BDC as of October 2023. Future RFCs are planned that will address masking of unique identifiers, the details of how imaging pixel data will be de-identified, the de-identification process workflow, and quality management.

The focus of this RFC is on de-identification of DICOM images.

The de-identification protocol described in this section is intended to be a baseline for de-identification within BDC. The protocol is compliant with regulations such as the HIPAA Privacy Rule and the Common Rule, while retaining the maximal amount of research utility possible. It is designed based on the experiences from the HeartShare imaging pilot project. The protocol will evolve over time, with future iterations to address new issues as they arise, and customizations to address specific research use cases. These may involve Expert Determinations, which can both increase privacy protections and improve research utility. This protocol is to be used for all medical imaging data to be submitted to the BDC. The protocol may be implemented in an image de-identification tool at the submitter’s site, or in a central BDC-related data curation service. Any deviation from this protocol must be discussed with and approved by the BDC/DMC. The baseline de-identification protocol can be found at this link: .

Introduction to HIPAA Safe Harbor and DICOM Part 15

De-identification of DICOM data can be performed according to different standards. Two commonly accepted standards are HIPAA Safe Harbor and (referred to in the rest of this document as the DICOM Part 15 Standard).

HIPAA Safe Harbor de-identification calls for the removal of 18 types of identifiers (detailed here: ). The standard legally applies to PHI handled by HIPAA Covered Entities, however as it has been in use for over 20 years it is generally accepted as a standard for de-identification for other types of data as well.

The DICOM Part 15 Standard was developed through a careful review of all DICOM attributes, identifying any that had the possibility of containing identifying information and creating a mitigation strategy. It is more extensive than HIPAA Safe Harbor, covering attributes that are not part of the 18 prescribed types of identifiers such as ethnicity and biological sex. Various mitigation strategies are presented to treat the attributes detailed as part of the standard, with the Basic DICOM Part 15 Confidentiality Profile being the most conservative, calling for suppression of most of the attributes.

In order to have de-identified data that still possesses analytic utility for BDC researchers, while also being a standardized implementation of de-identification that can be applied across most data to be ingested by BDC, an evaluation was performed to produce a set of de-identification rules that can be applied to DICOM header attributes. The evaluation leveraged the de-identification profiles detailed in the DICOM Part 15 standard by evaluating its contents and aligning with the minimum requirements to comply with HIPAA Safe Harbor. The resulting de-identification strategy should be sufficient to construct a de-identification profile that can be applied across all DICOM headers.

The steps for performing this evaluation were as follows:

1. Attributes from each profile were classified into the following categories: Direct Identifier (DI), Quasi-Identifier (QI), and Non-Identifier (NI), according to the classification framework detailed in the following diagram:

1. After classification, DIs and QIs were then aligned with the 18 types of identifiers specified for removal within the HIPAA Safe Harbor provision.

2. Each of the attributes that aligns with one of the HIPAA Safe Harbor identifiers was then assigned a mitigation technique to remove the identifying information that could appear in the field.

Of the attributes within the DICOM Part 15 standard that must be removed for compliance with HIPAA Safe Harbor, there are:

• 4 name attributes

• 4 patient address attributes

• 122 date attributes

• 5 telephone number attributes

Names, addresses, and telephone numbers should be suppressed from the data. Dates can be kept accurate to the year (a future BDC medical image de-identification RFC will address improving this approach for longitudinally acquired imaging studies). The other unique IDs can either be suppressed or they can be masked in a way so that their original values cannot be re-obtained. The specifics of how the other unique IDs will be masked will come in a separate RFC that describes the masking procedures. Additionally, there are 26 attributes that contain various forms of free text, such as comments, notes, labels, and text strings. Identifying information may be written in these attributes. As such, they should be suppressed to prevent the leakage of identifying information.

The other attributes detailed in the DICOM Part 15 standard do not necessarily require mitigation for compliance with HIPAA Safe Harbor. However, if they do not have analytic usage, it is recommended to mitigate them according to the specifications detailed in the DICOM Part 15 standard in order to decrease the risk of re-identification represented by indirectly identifying fields not mentioned in HIPAA Safe Harbor.

Image pixel data, often encountered in ultrasound (echo) imaging, can contain PHI, such as patient names, dates of birth, and the hospital or imaging center names. This information can be shown either in labels on images, which usually have pre-specified areas, or in the form of burned-in text, which can appear anywhere on the image. Any identifying information contained within pixel data should be removed before it is made available to researchers.

Methods for removal of image pixel data include the following:

• Masking through opaque boxes over parts of the image

• AI assisted removal of identifying information, deploying optical character recognition (OCR)

• Deletion of images from the dataset that contain identifying information

Image pixel de-identification will be performed as a service by tools provided by existing third party tool provided by DMC contractors. After de-identification, images will still require review to ensure that the process was able to capture and remove all identifying information on the images. This is a necessary quality control to ensure that there is no leakage of identifying information.

Metadata associated with images, such as filenames and file paths, can often include unique IDs and dates of medical events. This information is important to associate imaging data correctly with other types of data for linkage, processing, and analysis, however it can also present a risk of leakage of identifying information on de-identified data files. To prevent that from happening, the following rules should be followed:

1. Folder names should only include the study name and associated visit number, and no further information

   1. e.g., for the first visit of the MESA study, the folder name should be called MESA_V1

2. Image filenames are to be set to the following format: STUDYNAME_TYPE_VISITNN_ YYYYMMDD_SEQ

The risks presented by using the de-identification methods detailed in this RFC are as follows:

1. HIPAA Safe Harbor, while being an accepted standard for de-identification, does not cover all potential identifiers (leaving out potentially attributes such as race, employment, diagnoses, procedures, and treatments). Data de-identified under HIPAA Safe Harbor holds a residual risk of re-identification.

2. Automated imaging de-identification solutions are not 100% accurate, leaving the potential for small amounts of identifying information to be retained.

Data made available through BDC is provided for research purposes to investigators who should not have ulterior motives to perform re-identification. HIPAA Safe Harbor represents a standard that has been in use for over 20 years, so the risks presented from using that standard are well understood and acceptable by BDC. The risk presented by leakage of identifying information from imaging data can be mitigated through human review of de-identified images to ensure that all identifying information has been removed.

In the event that PHI is discovered in de-identified imaging data in BDC, such data shall be pulled off-line, checked for removal of offending PHI, before being posted again on BDC. In such cases, the data submitter shall be informed of the incident.

Local vs. Cloud-based Image De-Identification

Depending on the capabilities of the de-identification tool and the legal and logistic requirements for access to original identifiable images, de-identification may be done locally on the data-generating site or through a central cloud-based service. Although the latter is often more efficient (semi-automated and scalable), the transfer of identifiable (PHI-containing) images to a central cloud may require agreements between the data provider (submitter) and the de-identification service provider, stipulated through execution of Data Transfer Agreement (DTA). Details as to the image de-identification process that will be used will be provided in a future RFC.

GTEx Data

The Genotype-Tissue Expression (GTEx) Program is a widely used data resource and tissue bank to study the relationship between genetic variants (inherited changes in DNA sequence) and gene expression (how genes are turned on and off) in multiple human tissues and across individuals. For information on access to GTEx data, refer to GTEx v8 - Free Egress Instructions as part of the AnVIL documentation.

NCPI Data Portal

The NIH Cloud Platform Interoperability Effort (NCPI) is currently working to establish and implement guidelines and technical standards to empower end-user analyses across participating cloud platforms and facilitate the realization of a trans-NIH, federated data ecosystem. Participating institutions include BDC, AnVIL, Cancer Research Data Commons, and Kids First Data Resource Center. Learn what data is currently hosted by these platforms by using the NCPI Data Portal.

BDC Powered by PIC-SURE is a cohort-building platform that allows researchers to search for terms of interest, apply filters on variables, and retrieve information about the participants that meet the query criteria.

There are two main ways to search and interact with the data: publicly available cohort building on the Discover page and authorized cohort building on the Explore page.

The Discover page enables you to conduct a feasibility study by exploring individual-level data (only aggregate counts are returned) without authorization to access the data. This allows you to search across all studies hosted in BDC. For more information about Discover, please refer to Discover: Publicly Available Cohort Building documentation.

The Explore page allows you to explore participant-level data of publicly available studies and any studies in which they have an active dbGaP Data Access Request (DAR). For more information about this process, please review the Managing Data Access section. From the Explore page, you can also bring participant-level data to a BDC analysis platform to begin your analysis. For more information about Explore, please refer to Explore: Authorized Cohort Building documentation.

Table Comparison of Discover and Explore

*Stigmatizing variables and other variables are removed from Discover to protect participant anonymity.

Explore allows users with an eRA Commons account to search variables, apply filters, build cohorts, and export data from studies they are authorized to access.

Requirements

To obtain access to Explore, you must have an NIH eRA Commons account. For instructions on how to register an account, please refer to the eRA website.

Login

After you have created an eRA Commons account, you can log in to Explore by navigating to https://picsure.biodatacatalyst.nhlbi.nih.gov/login and selecting to log in with eRA Commons. You will be directed to the NIH website to log in with your eRA Commons credentials. After signing in and accepting the terms of the agreement on the NIH RAS Information Sharing Consent page, allow the BDC Powered by Gen3 (BDC-Gen3) service to manage your authorization.

Why use Discover?

The primary purpose of the Discover page in BDC-PIC-SURE is to support feasibility assessments, or the process of determining whether a study or dataset will work for your research.

Applying for authorization to access participant-level data can be a time-consuming process. The Discover page enables you to investigate datasets on BDC to determine whether they contain the information you are interested in and if there are sufficient participants (a large enough sample size) for your planned analysis. This is a publicly available tool, meaning anyone can perform searches, add filters, and retrieve counts on BDC-hosted data.

How to use Discover

1. Search for the terms or variables related to your planned research project. This could include clinical outcomes, phenotypes, demographics, and more.

2. Apply facets to narrow down your search results. Use the left-hand panel to add facets to your search. This could include selecting specific studies or data types of interest.

3. Learn more about the variable by clicking on the row. When you find a promising search result, you can learn more about the variable by clicking on the row or the "i" icon. This will display the study and dataset associated with the variable.

4. Apply filters. Once you have identified variables, you can apply filters by clicking the funnel icon. This will add inclusion criteria, or "filter down" the cohort of participants.

5. See the aggregate counts of your selected cohort. As you apply filters, you will narrow down the participants in your cohort. This gives you an idea of how many study participants meet specific criteria for your research project.

6. Request access to the study. Once you've identified the study that fits your planned research, you can submit a Data Access Request to dbGaP to obtain authorization to access the data. For more information on how to do this, please refer to .

Features of Explore

The Explore page provides access to complete, participant-level data, in addition to aggregate counts. There are some features specific to the Explore page, outlined below.

A: Faceted search. The panel on the left-hand side of the screen provides options, or facets, to narrow down your search results.

B: Search bar. The search bar at the top center of the page allows for searching of phenotypes, clinical outcomes, and variables of interest.

C. Search results table. The search results table displays the variables that are returned based on the combination of search term and selected facets.

D. Actions for filtering variables. Information about icons from left to right:

• The Information icon allows you to view information associated with the variable.

• The Filter icon allows you to apply a variable-level filter.

• The Hierarchy icon allows you to apply filters on different nodes within that variable's concept path. For example, let's say we click the hierarchy icon of the HFAA23I1. Orthopnea at discharge [Heart Failure Hospital Record Abstraction Form, HFA] variable. This will show the concept path for the variable: the study (ARIC), table (pht004102), and variable level. We can click on the table "pht004102" level to add a filter that selects participants with data associated with any variables in that table.

• The Add to Export icon allows you to add variables to the export. Note that this will not apply any filtering to these variables.

E. Genomic Filtering. The Genomic Filtering button can be used to apply genomic filters to your cohort selection criteria. For more information about genomic filtering, please refer to the .

F. Participant counts in the Results Panel. The count displayed will be updated to describe the number of participants who meet the query criteria.

C: Prepare for Analysis. Once at least one filter has been added, the Prepare for Analysis button can be used to retrieve participant-level data corresponding to your filters and variable selections. To learn more about data retrieval options, please refer to the .

H. Build Advanced Query. Enables researchers to create complex queries by defining logical operators ("and" vs. "or") and grouping related filters. For more information, please refer to the .

G. Variable Distributions Tool. Use this tool to view distributions graphs for the selected cohort. Continuous data will be shown as histograms while categorical variables will be shown as bar plots.

Why use Explore?

The primary purpose of the Explore page in BDC-PIC-SURE is to select participants and variables for analysis.

Accessing and using raw data files for analysis can be challenging, particularly with the diverse range of data file formats and structures that vary according to the study design. The Explore page enables you to search for decoded variable information, apply filters, select variables, and hand off your data in an analysis-ready format—all without needing to handle raw data files.

How to use Explore

1. Log in to BDC-PIC-SURE. To access the Explore page, you will need to be logged in.

2. Search for the terms or variables related to your planned research project. This could include clinical outcomes, phenotypes, demographics, and more.

3. Apply facets to narrow down your search results. Use the left-hand panel to add facets to your search. This could include selecting specific studies or data types of interest.

4. Learn more about the variable by clicking on the row. When you find a promising search result, you can learn more about the variable by clicking on the row or the "i" icon. This will display the study and dataset associated with the variable.

5. Apply filters. Once you have identified variables, you can apply filters by clicking the funnel icon. This will add inclusion criteria, or "filter down" the cohort of participants.

6. Apply genomic filters. Filters can be added to variants in specific genes in combination with phenotypic filters.

7. Include additional variables for analysis. If there are additional variables you would like to include in your dataframe but not apply filters, you can use the "Add to Export" icon, which looks like a square with an arrow pointing right.

8. Bring your data to an analysis platform. Once you have applied filters, selected variables, and built your cohort, you are ready to bring the participant-level data to a BDC analysis platform. Click on the "Prepare for Analysis" button to start the process.

9. Start your analysis! You are now ready to begin analyzing your participant-level data.

BDC-PIC-SURE integrates clinical and genomic datasets across BDC, including TOPMed and TOPMed related studies, COVID-19 studies, and BioLINCC studies. Each variable is organized as a concept path that contains information about the study, variable group, and variable. Though the specifics of the concept paths are dependent on the type of study, the overall information included in the same.

For more information about additional dbGaP, TOPMed, and PIC-SURE concept paths, refer to Appendix 1.

Table of Data Fields in BDC-PIC-SURE

Note that there are two data types in PIC-SURE: categorical and continuous data. Categorical variables refer to any variables that have categorized values. For example, “Have you ever had asthma?” with values “Yes” and “No” is a categorical variable. Continuous variables refer to any variables that have a numeric range of values. For example, “Age” with a value range from 10 to 90 is a continuous variable. The internal PIC-SURE data load process determines the type of each variable based on the data.

The API page of the PIC-SURE interface provides a table overview of previously saved dataset IDs.

There are several actions available to manage the saved dataset IDs:

• Copy can be used to copy the dataset ID to the clipboard.

• More information can be used to view the details associated with the dataset ID, such as variable filters and variables added to export.

• Delete can be used to remove unwanted dataset IDs from the table. Note that deleted dataset IDs can be restored.

To view previously deleted dataset IDs, click the "Include Deleted Dataset IDs" button to display a new table with these IDs.

Genomic Filtering

BDC-PIC-SURE can be used to combine phenotypic and genomic filters to create a cohort of participants.

Using the "Genomic Filtering" button to the right of the search bar, you can add genomic filters using various fields:

• Gene with Variant: Search and select the gene name(s) using the official gene symbol that contains the variant of interest.

• Calculated Consequence: Select the desired calculated consequences based on VEP annotation. The consequences are organized by high, medium, and low severity. The listed consequences are standardized terms from the Sequence Ontology (http://www.sequenceontology.org) that describe the calculated consequence of a variant. The severity for the calculated consequence of a variant on a gene has possible values HIGH (frameshift, splice disrupting, or truncating variants), MEDIUM (non-frameshift insertions or deletions, variants altering protein sequencing without affecting its length) or LOW (other coding variants including synonymous variants).

• Variant Frequency: Displayed variant frequencies are based on the gnomAD combined population as discrete text categories. Possible values: Novel (variant not in gnomAD database), Rare (variant frequency less than 1%), Common (variant frequency greater than or equal to 1%).

BDC-PIC-SURE has integrated clinical and genomic data from various heart, lung, blood, and sleep-related datasets. These include the NHLBI Trans-Omics for Precision Medicine (TOPMed) and TOPMed-related studies, BioLINCC datasets, and COVID-19 datasets.

View a summary of the data available in BDC and the data you have access to by viewing the Data Dashboard. This table displays information about the study and associated data, including the full and abbreviated name of the study, study focus, associated program, the number of clinical variables, participants, and samples sequenced, and the dbGaP accession number (or phs number).

You are also able to see which studies you are authorized to access if you are logged in using the Study Link column of the table.

For information from dbGaP on submitting a data access request, refer to Tips for Preparing a Successful Data Access Request documentation.

PIC-SURE Personal Access Token

To access the PIC-SURE API, a user-specific token is needed. This is the way the API grants individual users access to participant-level protected-access data. The user token is strictly personal; do not share it with anyone.

You can copy your personalized access token by selecting the Prepare for Analysis tab at the top of the screen.

Here, you can Copy your personalized access token, Reveal your token, and Refresh your token to retrieve a new token and deactivate the old token.

New Features and Improvements

• Researchers can now filter datasets by Subject Type using faceted search, making it easier to identify human and autopsy-based variable results.

• The Advanced Query Builder now supports the Variable Distributions Tool, where bar charts and histograms are shown for individual variables, providing a quick way to inspect data distributions within a query.

New Datasets and Version Updates

• phs003768 (RECOVER_Autopsy): NIH RECOVER: A Multi-Site Pathology Study of Post-Acute Sequelae of SARS-CoV-2 Infection

• phs003463 (RECOVER_Adult): Researching COVID to Enhance Recovery (RECOVER): Adult Observational Cohort Study

• phs003461 (RECOVER_Pediatrics): Researching COVID to Enhance Recovery (RECOVER): Post Acute Sequelae of SARS-CoV-2 (PASC) Pediatric Cohort Study

• phs004266 (BIOLINCC_WHI): The Women's Health Initiative (WHI) - BIOLINCC

• phs002910 (COPDGene): Collaborative Cohort of Cohorts for COVID-19 Research (C4R): Genetic Epidemiology of COPD Study (COPDGene)

• phs004636 (BioLINCC_WLM): Weight Loss Maintenance (WLM)

• phs004633 (BioLINCC_VA2_TAH): VA Cooperative Study of the Efficacy of Hepatitis Immune Serum Globulin for the Prevention or Modification of Post-Transfusion Hepatitis (VA2-TAH)

• phs004646 (BioLINCC_THAPCA_OH): Therapeutic Hypothermia After Pediatric Cardiac Arrest (Out of Hospital) (THAPCA-OH)

• phs004645 (BioLINCC_THAPCA_IH): Therapeutic Hypothermia After Pediatric Cardiac Arrest (In-Hospital) (THAPCA-IH)

• phs004568 (BioLINCC_SPIROMICS): Subpopulations and Intermediate Markers in COPD Study (SPIROMICS)

• phs004618 (BioLINCC_QUARTET_USA): A Double Blind Randomized Controlled Trial to Assess the Efficacy and Safety of a Quadruple Ultra-low-dose Treatment for Hypertension (QUARTET USA)

• phs004617 (BioLINCC_PREMIER): PREMIER: Lifestyle Interventions for Blood Pressure Control (PREMIER)

• phs004322 (BioLINCC_P2C2): Pediatric Pulmonary and Cardiovascular Complications of Vertically Transmitted HIV Infection (P2C2)

• phs004610 (BioLINCC_OMNI_Heart): Optimal Macronutrient Intake Trial to Prevent Heart Disease (OMNI Heart)

• phs004609 (BioLINCC_MRFIT): Multiple Risk Factor Intervention Trial for the Prevention of Coronary Heart Disease (MRFIT)

• phs004323 (BioLINCC_Lung_HIV): Longitudinal Studies of HIV-Associated Lung Infections and Complications (Lung HIV)

• phs004591 (BioLINCC_IPF_Survival_PR): IPF Survival Proteomics

• phs004604 (BioLINCC_IDEA): Innovative Approaches for Diet, Exercise, and Activity (IDEA)

• phs004603 (BioLINCC_HHL): Heart Healthy Lenoir Project (HHL)

• phs004635 (BioLINCC_DASH_Sodium): Dietary Approaches to Stop Hypertension - Sodium Study (DASH-Sodium)

• phs004630 (BioLINCC_COMPARE_VOE): COMPARE-VOE

• phs004602 (BioLINCC_COBLT): Cord Blood Transplantation Study (COBLT)

• phs004634 (BioLINCC_COAG): Clarification of Optimal Anticoagulation Through Genetics (COAG)

• phs004647 (BioLINCC_CASS): Coronary Artery Surgery Study (CASS)

• phs004643 (BioLINCC_CABG): Post Coronary Artery Bypass Graft Study (CABG)

• phs004642 (BioLINCC_BMT_CTM_1204): Blood and Marrow Transplant Clinical Trials Network (BMT CTN) Reduced Intensity Conditioning for Children and Adults With Hemophagocytic Syndromes or Selected Primary Immune Deficiencies (1204)

• phs004641 (BioLINCC_BMT_CTN_0801): Blood and Marrow Transplant Clinical Trials Network (BMT CTN) A Phase II/III Randomized, Multicenter Trial Comparing Sirolimus Plus Prednisone and Sirolimus/Calcineurin Inhibitor Plus Prednisone for the Treatment of Chronic Graft-versus-Host Disease (0801)

• phs004629 (BioLINCC_BMT_CTN_0402): Blood and Marrow Clinical Trials Network (BMT CTN) 0402

• phs004601 (BioLINCC_BMT_CTN_0201): Blood and Marrow Clinical Trials Network (BMT CTN) Comparing Peripheral Blood Stem Cell Transplantation Versus Bone Marrow Transplantation in Individuals With Hematologic Cancers (0201)

• phs004600 (BioLINCC_BARI_2D): Bypass Angioplasty Revascularization Investigation in Type 2 Diabetes (BARI 2D)

• phs004599 (BioLINCC_BARI): Bypass Angioplasty Revascularization Investigation (BARI)

• phs004607 (BioLINCC_AsthmaNet_STICS): AsthmaNet Step-up Yellow Zone Inhaled Corticosteroids to Prevent Exacerbations (STICS)

• phs004606 (BioLINCC_AsthmaNet_SIENA): AsthmaNet Steroids in Eosinophil Negative Asthma (SIENA)

• phs004639 (BioLINCC_AFFIRM): Atrial Fibrillation Follow-Up Investigation of Rhythm Management (AFFIRM)

The PIC-SURE API can be accessed via tutorial notebooks on either BioData Catalyst Powered by Seven Bridges (BDC-Seven Bridges) or Powered by Terra (BDC-Terra).

To launch one of the analysis platforms or learn more about them, go to the BioData Catalyst website.

The PIC-SURE API can be used to extract the data dictionary. This can be done regardless of authorization to access data and can be done with one, multiple, or all studies.

Descriptions of Fields in PIC-SURE Data Dictionary

Note that there are several types of studies available in PIC-SURE:

1. dbGaP format compliant: ingested by dbGaP in the dbGaP recommended format (https://www.ncbi.nlm.nih.gov/gap/docs/submissionguide/)

2. dbGaP ingested, but not format-compliant

3. Not ingested by dbGaP: are not format-compliant and do not have a study accession (phs number)

• conceptPath: The concept path used to uniquely identify a variable when exported to users. For more information about concept paths and data organization, please refer to the .

• name: ID associated with the variable. For studies in dbGaP format, this is provided as “phvXXXXXXXX”. Non-compliant studies instead have a short text ID provided that can be a duplicate of the "display" field.

• display: A short text ID associated with a variable. These are often not human-readable as they are mostly derived from the column names in datasets. For non-compliant studies, this can be a duplicate of the "name" field.

New Features and Improvements

• Minor improvements to the Data Dashboard page consent groups.

New Datasets and Version Updates

• phs003768 (RECOVER Autopsy): NIH RECOVER: A Multi-Site Pathology Study of Post-Acute Sequelae of SARS-CoV-2 Infection

  • dbGaP link

• phs004023 (Imaging-COPDGene-r2): Genetic Epidemiology of COPD (COPDGene) Imaging Data

• phs003566 (Imaging_SPRINT-r3): Systolic Blood Pressure Intervention Trial (SPRINT-Imaging)

• phs003562 (Imaging_ACCORD-r3): Action to Control Cardiovascular Risk in Diabetes (ACCORD - Imaging)

Login to the BDC-Gen3 Platform

In order to navigate and access data available on the Gen3 platform, start by visiting the login page. You will need an eRA Commons account as well as access permissions through the Database of Genotypes and Phenotypes (dbGaP). If you are a researcher, login by selecting NIH Login and using your eRA Commons account. BDC consortia developers can login using their Google accounts. Make sure to use the correct login method that contains access to your available projects.

Once logged in, your username will appear in the upper right-hand corner of the page. You will also see a display with aggregate statistics for the total number of subjects, studies, aliquots and files available within the BDC platform.

NOTE: These numbers may differ from those displayed in the dbGaP records as they include TOPMed studies as well as the associated parent studies.

A number of clinical variables have been harmonized by the in order to facilitate cross-study analysis. Faceted search over the DCC Harmonized Variables is available via the page, under the "Data" tab.

Unharmonized clinical files are also available on the Gen3 platform and contain all of the raw phenotypic information for the hosted studies. Unlike the DCC Harmonized Variables, these files are located and searchable under the "" tab in the page.

The Gen3 platform hosts genomic data provided by the (TOPMed) program and the plus synthetic tutorial data from Terra. At present, these projects include CRAM and VCF files together with their respective index files. Specifically for TOPMed projects, each project will contain at least one multi-sample VCF that comprises all subjects within the consent group. CRAM and VCF are based on an individual level, whereas multi-sample VCFs are based on the study consent level.

All files are available under the "Files" tab in the page. More detailed information on currently hosted data on the Gen3 platform can be found .

The Gen3 platform also hosts a variety of imaging collections from programs such as (MESA), , and the . These collections include ECG, echocardiograms, CAT scans and DICOM series.

The BDC-Gen3 platform contains five pages described below:

• An interactive data dictionary display that details the contents and relationships between clinical and biospecimen data

• Study level page where researchers can browse and search study-level metadata to find relevant studies.

• : The facet filter custom cohort creation tool

Additionally, BDC-Gen3 supports a page which allows researchers to search the data model for facets that are not exposed on the Explorer page.

New Datasets and Version Updates

• phs000557 (dbGaP_ARIC_CARe): NHLBI Atherosclerosis Risk in Communities (ARIC) Candidate Gene Association Resource (CARe)

  • dbGaP link

• phs001536 (dbGaP_CCDG-ARIC): Center for Common Disease Genomics: Atherosclerosis Risk in Communities (ARIC) Study

• phs000765 (dbGaP_COPDGene_Geno): Genetic Epidemiology of COPD (COPDGene) Genetic Analysis

• phs002611 (dbgap_FHS_RNA_Brain): Framingham Heart Study RNA Sequencing in Postmortem Brain Tissue

• phs000651 (dbGaP_FHS_CHARGE-S): Building on GWAS: the U.S. CHARGE Consortium - Sequencing (CHARGE-S): FHS

• phs001610 (dbGaP_T2D-GENES_FHS): Type 2 Diabetes Genetic Exploration by Next Generation Sequencing in Multi-Ethnic Samples (T2D-GENES) Follow-up Study: Framingham Heart Study

• phs002560 (dbGaP_FHS_GutMicro): Clinical and Genetic Correlates of the Gut Microbiome and Relation to Cardiometabolic Risk

• phs002559 (dbGaP_BRIDGET_FHS): Framingham BRain Imaging, cognition, Dementia and next generation GEnomics (BRIDGET)

• phs002558 (dbGap_ADSP_FHS): Framingham CHARGE Alzheimer's Disease Sequencing Project (ADSP)

• phs000820 (dbgap_CCF_AFIB): The Cleveland Clinic Foundation's (CCF) Lone Atrial Fibrillation (AFIB) GWAS Study

• phs000007 (FHS): Framingham Cohort

The Dictionary page contains an interactive visual representation of the Gen3 data model. The default graph model view, as pictured below, displays all of the nodes and relationships between nodes in a hierarchical structure. The model further specifies the node types and links between nodes, as highlighted in the legend located at the top right side of the page.

Users can click on any of the graph nodes in order to learn more about their respective properties. By clicking on a node, the graph will highlight that specific node and all associated links that connect it to the Program node. A "Data Model Structure" list will also appear on the left side toolbar. This will display the node path required to reach the selected node from the Program node.

When a second node in the path is selected, it will then gray out the other possible paths and only highlight the selected path. It will also change the "Data Model Structure" list on the left side toolbar.

Overview

The Query page can search and return metadata from either the Flat Model or the Graph Model of a commons. Using GraphQL, these searches can be tailored to filter and return fields of interest for the data sets being queried. These queries can be made immediately after data submission as this queries the model directly.

For more information about how to use the Query page, refer to the Gen3 documentation.

Overview

The Discovery Page provides users a venue to search and find studies and datasets available on the BDC-Gen3 platform. This page is accessible to everyone.

When navigating to this page researchers can see the current count of studies and subjects available on the platform.

Search Methods

Free-text searches can be used to identify relevant studies. For example, entering 'methylation' in the search box currently shows twenty studies.

Each row contains the platform study name, the full study name, number of subjects, and the dbGaP accession number. The data availability field indicates whether the searcher has access to the data — a closed lock means 'no' and an open padlock means 'yes'.

Search can also be conducted using 'filters' which are tags on each study record. Expanding the "Study Filters" reveals the current search options and selections can be made from each pull down menu.

From the table of results from a search, clicking on a STUDY NAME opens a linked page on the platform.

Each tab contains additional study-level data about the study including the DOI for citation purposes, the full study description, and information retrieved from dbGaP

Profile Page

The Profile page contains two sections: API keys and Project access.

API key(s)

To download large amounts of data, an API key will be required as a part of the gen3-client. To create a key on your local machine, click Create API key, which will activate the following pop-up window:

Click Download json to save the credential file to your local machine. After completion, a new entry will appear in the API key(s) section of the Profile page. It will display the API key key_id and the expiration date (one month after the key creation). The user should delete the key after it has expired. If for any reason a user feels that their API key has been compromised, the key should be deleted before subsequently creating a new one.

Project Access

This section of the Profile page lists the projects and the methods of access for the data within in the BDC-Gen3 system. If you do not see access to a specific study, check that you have been granted access within . If access has been granted for over a week, contact the BDC Help Desk:

The Seven Bridges Knowledge Center is a collection of user documentation which describes all of the various components of the platform, with step-by-step guides on their use. Our Knowledge Center is the central location where you can learn how to store, analyze, and jointly interpret your bioinformatic data using BDC-Seven Bridges.

From the Knowledge Center, you can access platform documentation. This content is organized into sections that deal with the important aspects of accessing and using BDC-Seven Bridges.

You can also read the Release Notes in the Knowledge Center, keeping you up-to-date on all of the latest updates and new features for BDC-Seven Bridges.

One of the key steps to becoming an advanced user and being able to fully understand and leverage the power of BDC-Seven Bridges is to learn how to detect and correct errors that prevent the successful execution of your analyses. The Troubleshooting tutorial presents some of the most common errors in task execution on the platform and shows you how to debug and resolve them. There is also a corresponding public project on the platform called "Troubleshooting Failed Tasks" which has examples of the failed analyses presented in the written tutorial.

• Find the written tutorial here.

• Find the platform public project with examples here.

Introduction

This tutorial guides users through the process of transferring files between the two workspace environments of BDC: BDC-Seven Bridges and BDC-Terra.

Most researchers select one of the BDC workspaces as their primary analysis environment and their labmates and collaborators typically work with them on the same workspace environment. However, there are cases where some collaborators work on BDC-Seven Bridges and others work on BDC-Terra. In this case, researchers need to share data files between the two workspaces to facilitate collaboration. When researchers run analyses on Seven Bridges, the results, or derived data, is only available on Seven Bridges. Likewise, when researchers run analyses on Terra, the results are only available on Terra. This tutorial provides step-by-step guidance on how to share derived data between the workspace environments. These instructions can also be used to share private data that has been uploaded to Seven Bridges or Terra.

Both open access data and controlled access data can be shared across workspace environments. Importantly, if a researcher intends to share controlled access data, they must ensure that all recipients have the necessary dbGaP permissions for those files. In some cases, this may mean the researchers must be listed as collaborators on their respective dbGaP applications.

These instructions are intended for sharing files under 1 terabyte (TB) in size. If you want to share data larger than 1 TB, contact the to discuss your use case.

Moving data between Seven Bridges and Terra is currently a manual process and requires that one of the researchers involved in sharing has an account on both platforms. It is recommended that the recipient of the shared data is the person to have accounts on both Seven Bridges and Terra.

• To create a Terra account, refer to the .

• To create a Seven Bridges account, refer to the . If you are new to Seven Bridges, you may find this helpful.

Let’s consider an example case:

• Sebastian is working on Seven Bridges and Teresa is working on Terra.

• If Sebastian wants to share data with Teresa so that she can use the data on Terra, Teresa first needs to set up an account on Seven Bridges.

• Now, Teresa has an account on Terra and an account on Seven Bridges.

Let's consider a second case:

• If Teresa (Terra) wants to share data with Sebastian (Seven Bridges) so that he can use the data on Seven Bridges, Sebastian first needs to create an account on Terra.

• Now Sebastian has an account on Seven Bridges and an account on Terra.

• Teresa can share the data with Sebastian on Terra by sharing the workspace with the data she wants to share with Sebastian. For information on sharing workspaces, refer to the Terra documentation.

The second consideration is making sure the researcher moving data between the two workspaces has billing groups set up on both workspaces to cover cloud costs, if necessary. Contact the if you have questions about how to get a billing group on Seven Bridges or Terra.

The following steps describe how to use the Seven Bridges platform to pull data securely from a Terra workspace into a Seven Bridges project.

Refer to the , specifically the section . This method:

• Works well for all size transfers.

• Ideal for large file sizes or 1000s of files.

• Can be used for transfers between local storage and a bucket, workspace VM or persistent disk and a Google bucket, as well as between Google buckets (external and workspace).

You will use the terminal in JupyterLab on the Seven Bridges workspace environment. The reason for this is that although Seven Bridges can run on the Google Cloud Platform, the Google bucket API is not exposed in the same manner as it is on Terra. Therefore you will start a JupyterLab notebook on Seven Bridges, using the project you would like to be the destination for the copied data. Refer to the Seven Bridges documentation for launching and accessing the terminal in a .

After launching the notebook, the next step is to open the terminal and install the program gsutil which is a python program that lets end users add data to or copy data from a Google Cloud bucket. After opening the terminal, run the following commands:

Installing gsutil takes only a few seconds.

The config command provides a secure URL for you to navigate to in the browser. You will authenticate with the same credentials that were used to login to Terra. The shortcut to access the printed URL in the JupyterLab terminal is to press shift and right click, which will display options to copy the URL. Copy and then navigate to the URL in a new browser tab, which will direct you to Google authentication:

Google will provide an authentication code that you will copy and paste into the terminal.

Next, you will type in the Google Project id. This is found on the right side of the Terra Workspace Dashboard.

Next, run the command below to display the different Google buckets that are attached to the project id.

The Google bucket name for the Terra project can be found in the lower right corner of the Terra Workspace.

Running gsutil ls on the Google bucket name will display the folders and files from the Terra workspace.

To copy a folder to the Seven Bridges workspace environment, run the following command:

There are a couple important things to mention about the gsutil cp command. First, the -R flag for gsutil cp is used to recursively copy a folder and all of its subfolders and files. Most users will likely want to use the -R flag. This flag should be omitted if copying individual files or if using a wild card such as “*.vcf”.

Additionally, /sbgenomics/output-files should be the destination folder when bringing in data from Terra, as this will ensure the files or folders get populated back to the Seven Bridges project. Refer to the documentation for information about working with files in Data Cruncher environments. After the JupyterLab instance is shut down, your files will automatically be populated in your project-files tab on Seven Bridges.

In this section we will discuss pushing data from a Seven Bridges project to a Terra workspace.

The process of moving data from Seven Bridges to Terra is the same setup as the previous section with some modifications to the gsutil copy command. Instead, we reverse the arguments.

You will still use the -R flag but the destination is a Terra bucket. The Terra workspace’s Google bucket name/id can be found on the Terra workspace Dashboard tab. You can verify that the folder has been copied by navigating to the Files section of the Data tab in your Terra workspace.

Clicking on the folder, you will see that all three files have been copied.

BDC-Seven Bridges offers researchers collaborative workspaces for analyzing genomics data at scale. Researchers can find and analyze hosted studies by using hundreds of optimized analysis tools and workflows (pipelines); creating their own workflows; or interactive analysis. On the platform, researchers can utilize collaborative workspaces for analyzing genomics data at scale, and access hosted datasets along with Common Workflow Language (CWL) and GENESIS R package pipelines for analysis. This platform also enables users to bring their own data for analysis and work in RStudio and Jupyterlab Notebooks for interactive analysis.

• Private, secure, workspaces (projects) for running analyses at scale

• Collaboration features with the ability to set granular permissions on project members

This guide aims to help you learn how to take advantage of all the various features and functionality for performing analyses on Seven Bridges and ensure that you can set up your analyses in the most efficient way possible to save time and money.

The following topics are covered in this guide:

• The basics of working with CWL tools and workflows on the platform.

• How to specify computational resources on the platform and how to use the default options selected by the execution scheduler.

• How to run Batch analyses and take advantage of parallelization with scatter.

• The basics of working with Jupyterlab Notebooks and Rstudio for interactive analysis.

You can refer to the guide .

Now that you can log in, you’ll want to make sure that you have access to a Billing Account and Billing Project. This will allow you to charge storage and analysis costs through a Google account linked to Terra. A Terra Billing Project is Terra's way of connecting a workspace where you accrue costs for things, back to a Google Billing account where you pay for it. You must have a Google Billing Account established before creating a Terra Billing Project. Outlined here are the steps necessary to set this up, as well as instructions on how to add or be added to an existing account/billing project.

Detailed instructions for setting up your billing can be found by following the links below. If you are a BDC Fellow, your procedure for billing set up is a bit different, but you may find some of the information below still relevant (sharing a billing project with another user, for example). Step 1: Get Step 2: Wait for approval & review the ​ Step 3: Credits approved. Now Step 4 (optional): ​

The Annotation Explorer is an application developed by Seven Bridges in collaboration with the TOPMed Data Coordinating Center. The application enables users to interactively explore, query, and study characteristics of an inventory of annotations for the variants called in TOPMed studies. This application can be used pre-association testing to interactively explore aggregation and filtering strategies for variants based on annotations and generate input files for multiple-variant association testing. It can also be used post-association testing to explore annotations associated with a set of variants, like variant sets found significant during association testing.

The Annotation Explorer currently hosts a subset of genomic annotations obtained using Whole Genome Sequence Annotator software for TOPMed variants. Currently, annotations for TOPMed Freeze5 variants and TOPMed Freeze8 variants are integrated with the Annotation Explorer. Researchers who are approved to access one or more of the TOPMed studies included in Freeze8 or Freeze5 will be able to access these annotations in the Annotation Explorer.

For more information, refer to the A.

Just starting out on BDC-Seven Bridges, and need to get up to speed on how to use the platform? Our experts have created a Getting Started Guide to help you jump right in. We recommend users begin learning how to use BDC-Seven Bridges by following the steps in this guide. After reading this guide, you will know how to create an account on BDC-Seven Bridges, learn the basics of creating a workspace (project), run an analysis, and search through the hosted data.

To read our Getting Started Guide, please refer to our documentation page .

Workspaces are the fundamental building blocks of Terra. You can think of them as modular digital laboratories that enable you to organize and access your data in a number of ways for analysis.

To learn about the basics of operating a Terra workspace, we recommend these resources: Article: Working with workspaces Video: Introduction to using workspaces in Terra

Read on in this section to get familiar with:

• Data storage and management

BDC Powered by (BDC-Terra) is a user-friendly system for doing biomedical research in the cloud. Terra workspaces integrate data, analysis tools, and built-in security components to deliver smooth research flows from data to results.

The following entries in this section of the BDC documentation are a starting point for learning how to use Terra in the context of the BDC ecosystem. You can also dive deeper into Terra by visiting the and the . Wherever possible, we highlight specific articles, tutorial videos, and example workspaces that will help you learn what you need to know to accelerate your research.

If you can't find what you are looking for, we are happy to help. See the Troubleshooting and Support section for more information.

Please note that Terra is designed for and tested with the browser.

We have a number of articles on tracking and minimizing the costs of operating on Terra. There are multiple ways of estimating how much your analyses are costing you, including built-in tools and external resources. The articles below contain instructions and advice on managing your cloud resources in a variety of ways: Article: Article: Article: Article:

Overview of the GENESIS pipelines

For researchers interested in performing genotype-phenotype association studies, Seven Bridges offers a suite of tools for both single-variant and multiple-variant association testing on BDC-Seven Bridges. These tools and features include the GENetic EStimation and Inference in Structured samples (GENESIS) pipelines, which were developed by the Trans-Omics for Precision Medicine (TOPMed) Data Coordinating Center (DCC) at the University of Washington. The Seven Bridges team collaborated with the TOPMed DCC to create Common Workflow Language (CWL) tools for the GENESIS R functions, and arranged these tools into five computationally-efficient workflows (pipelines).

These GENESIS pipelines offer methods for working with genotypic data obtained from sequencing and microarray analysis. Importantly, these pipelines have the robust ability to estimate and account for population and pedigree structure, which makes them ideal for performing association studies on data from the TOPMed program. These pipelines also implement linear mixed models for association testing of quantitative phenotypes, as well as logistic mixed models for association testing of binary (e.g. case/control) phenotypes.

Below, we feature our GENESIS Benchmarking Guide to assist users in estimate cloud costs when running GENESIS workflows on BDC-Seven Bridges.

GENESIS Benchmarking Guide

Introduction

The objective of the GENESIS Benchmarking Guide is to instruct users on the drivers of cloud costs when running GENESIS workflows on the BDC-Seven Bridges.

For all GENESIS workflows, the Seven Bridges team has performed comprehensive benchmarking analysis on Amazon Web Services (AWS) and Google Cloud Provider (GCP) instances for different scenarios:

• 2.5k samples (1000G data)

• 10k samples (TOPMed Freeze5 data)

• 36k samples (TOPMed Freeze5 data)

The resulting execution times, costs, and general advice for running GENESIS workflows can be found in the sections below. In these sections, each GENESIS workflow is described, followed by the benchmarking results and some tips for implementing that workflow from the Seven Bridges Team. Lastly, we included a Methods section to describe our approach to benchmarking and interpretation for your reference.

The contents of this guide are arranged as follows:

• Introduction

• Helpful Terms to Know

• GENESIS VCF to GDS

Below is a link to download the results of our Benchmarking Analysis described herein. It may prove useful to have this file open for reference when reading through this guide.

Before continuing on to the benchmarking results, please familiarize yourself with the following helpful terms to know:

• Tool: Refers to a stand-alone bioinformatics tool or its Common Workflow Language (CWL) wrapper that is created or already available on the platform.

• Workflow/Pipeline (interchangeably used): Denotes a number of tools connected together in order to perform multiple analysis steps in one run.

• App: Stands for a CWL wrapper of a tool or a workflow that is created or already available on the platform.

Figure 1. The jobs for an example run of RNA-Seq Quantification (HISAT2, StringTie) public workflow

The green bars under the gray ones (apps) represent the jobs (Figure 1). As you can see, some apps (e.g. HISAT2_Build) consist of only one job, whereas others (e.g. HISAT2) contain multiple jobs that are executed simultaneously.

In this section, we detail the process of converting a VCF to a GDS via a GENESIS workflow. This VCF to GDS workflow consists of 3 steps:

• Vcf2gds

• Unique variant id

• Check GDS

The first two steps are required while the last one is optional. The Check GDS step when included is the biggest cost driver in these tasks.

Check GDS tool is QC, which checks whether the final GDS file contains all variants that the input VCF/BCF has. This step is computationally intensive and its execution time can be 4-5 times longer than the rest of the workflow. Also, a failure of this step is something that we experience very rarely. In our results, there is a Check GDS column which is used as an indicator whether the Check GDS was performed or not.

We advise anyone who is using this workflow to consider results from the table below because differences in execution time and price with and without this check are considerable. A final decision on the approach that someone will use depends on the resources that one has (budget and time), and the preference of including or excluding the optional QC step.

In addition, CPU/job and Memory/job parameters have direct effects on execution time and the cost of the GENESIS VCF to GDS workflow. A combination of these parameters defines the number of jobs (files) that will be processed in parallel.

For example:

If a task is run on c5.9xlarge(36CPUs and 72GB RAM) with 1CPU/job and 4GB/job the number of jobs run in parallel will be min{36/1,72/4}=18. If a task is run on c5.9xlarge(36CPUs and 72GB RAM) with 1CPU/job and 8GB/job the number of jobs run in parallel will be min{36/1,72/8}=9. In this example, the second case would take twice as long as the first.

The following conclusions were drawn from performed benchmarking:

• Benchmarking showed that the most suitable AWS instances for this workflow are c5 instances.

• For all tasks that we run (from 2.5k up to 50k samples), 1CPU and 4GB per job were sufficient.

• For small sample sizes (up to 10k samples), tasks can be run on spot/preemptible instances to additionally decrease the cost.

The GENESIS Null model workflow is not computationally intensive and it is relatively low-cost compared to other GENESIS workflows. For that reason, we present results that we obtained without any optimization below:

The null model can be fit with relatedness matrices (i.e. mixed models) or without relatedness matrices (i.e. simple regression models). If a relatedness matrix is provided, it can be sparse or dense. The tasks with dense relatedness matrix are the most expensive and take the longest to run. For the Null model workflow, available AWS instances appear to be more suitable than Google instances available on the platform.

Results of the GENESIS Single Association Testing workflow benchmarking can be seen in the table above. Some important notes to consider when using this workflow:

• Null model type effect: The main cost and duration driver in these tasks is the null model type. The null model can be fit without a relatedness matrix (i.e. simple regression models), or with a relatedness matrix that can be sparse or dense (i.e. mixed models). The table above shows that task cost and execution time increases as the null model is weighted (dense > sparse > none). Differences between the model with dense matrix and the model with sparse matrix are significant, which is driven by both increased CPU time and memory required to use a dense matrix. Our advice is to check the relatedness matrix before fitting the model, and transform it to sparse if it is dense, especially if sample size is large. The null model type has direct influences on further parameters setup such as: instance type selection, CPU per job, Memory per job, etc.

• Instance type: Benchmarking showed that the most appropriate instance type is an AWS r5.x instance. Instance type selection is especially important when we are performing analysis with many samples (30k participants and above). In the tasks with up to 30k samples, r5.4xlarge instances can be used, and r5.12xlarge with more participants included. In addition, it is important to note that if a single association test is performed with a dense null model, then r5.12xlarge or r5.24xlarge instances should be picked. When it comes to Google instances, results can be seen in the above table as well. Since there often isn’t a Google instance that is the exact equivalent of the AWS instance, , we recommend choosing the most appropriate Google instance (matching the chosen AWS instance) from the list of available Google instances on BDC.

GENESIS Aggregate association testing can be performed using burden, SKAT, SMMAT, fastSKAT and SKAT-O tests. Our general conclusions are as follows:

• Null model selection: The same as in the single variant association testing, the main cost and duration driver in these tasks is the null model type. The null model can be fit without a relatedness matrix (i.e. simple regression models), or with a relatedness matrix that can be sparse or dense (i.e. mixed models). The table above shows that task cost and execution time increases as the null model is weighted (dense > sparse > none). Differences between the model with dense matrix and the model with sparse matrix are significant, which is driven by both increased CPU time and memory required to use a dense matrix. Our advice is to check the relatedness matrix before fitting the model, and transform it to sparse if it is dense, especially if sample size is large. The null model type has direct influences on further parameters setup such as: instance type selection, CPU per job, Memory per job, etc.

• Instance type: Benchmarking showed that the most appropriate instance type is an AWS r5.x instance. The majority of the tasks can be run on r5.12xlarge instances or on r5.24xlarge instances when the null model is with the dense relatedness matrix. Results for Google instances can be seen in the above table as well. Since the Google instance options often do not have an exact AWS equivalent, we selected the closest match from the list of available Google instances on BDC.

As can be seen for small sample sizes, up to 10GB per job can be sufficient for successfully completed tasks. One exception is the case when running a task with null model fit with the dense relatedness matrix when approx. 36GB/job is needed. When there are 50k samples, jobs require 70GB. Details can be seen in the table above. In addition to sample size, the memory required is determined by the number of variants included in each aggregation unit, as all variants in an aggregation unit are analyzed together.

SKAT and SMMAT tests are similar when it comes to CPU and Memory per job requirements. Roughly, these tests require 8GB/CPU, and details for different task configurations can be seen in the table below:

• Maximum number of parallel instances: The default number of parallel instances is 8. The impact of changing this number is mainly reflected through execution time. The tasks with more parallel instances will be finalized faster. This parameter can be set in Execution settings when drafting a task. However, each user has a limited total number of parallel instances and capturing a big number of parallel instances per task leads to a decrease in the total number of different tasks that can run at the same time.

• Spot instances: If it is expected for the task to be finalized in a few hours it can be run on spot instances. This will reduce the execution cost. However, losing a spot instance leads to rerunning the task on on-demand instances, which can lead to a higher cost than running the task on on-demand instances from the beginning. That is why spot instances are generally suitable for short tasks.

GENESIS Sliding window association testing can be performed using burden, SKAT, SMMAT, fastSKAT and SKAT-O tests. When running sliding window test is good to know:

• Null model selection: The same as in the previous tests the main cost and duration driver in these tasks is the null model type. The null model can be fit without a relatedness matrix (i.e. simple regression models), or with a relatedness matrix that can be sparse or dense (i.e. mixed models). The table below shows that task cost and execution time increases as the null model is weighted (dense > sparse > none). Differences between the model with dense matrix and the model with sparse matrix are significant, which is driven by both increased CPU time and memory required to use a dense matrix. Our advice is to check the relatedness matrix before fitting the model, and transform it to sparse if it is dense, especially if sample size is large. The null model type has direct influences on further parameters setup such as: instance type selection, CPU per job, Memory per job, etc.

• Instance type: Benchmarking showed that for analysis with or without sparse relatedness matrix tasks can be completed on a c5.9xlarge AWS instance. For analysis with dense relatedness matrix included in the null model and with 50k samples or more, r5.12xlarge instances can be used. Also, it is important to note that in this case increasing the instance (for example from c5.9xlarge to c5.18xlarge) will not lead to shorter execution time. Furthermore, it can be completely opposite. By increasing the size of the instance we also increase the number of jobs running in parallel. At one point there will be a lot of jobs running in parallel and accessing the same memory space which can reduce the performance and increase task duration. Results for Google instances can be seen in respective tables. Since the Google instance options often do not have an exact AWS equivalent, we selected the closest match from the list of available Google instances on BDC.

Benchmarking results:

In this text, we have highlighted the main cloud cost and execution time drivers when running GENESIS analyses. Please have in mind that when running an analysis users may experience additional costs due to different factors such as task failures or need for rerunning the analysis. When estimating cloud costs for your study, please account for a cost buffer for these two factors as well.

To prevent task failures, we advise you to carefully read app descriptions and if you have any questions or doubts, contact our Support Team at support@sevenbridges.com. Also, using memoization can help in cost reduction when rerunning the task after initial failure.

Throughout this document, it is important to note that the figures in the tables above are intended to be informative as opposed to predictive. The actual costs incurred for a given analysis will also depend on the number of samples and number of variants in the input files. For our analysis described above, we selected 1000G and TOPMed Freeze5 data as inputs. For TOPMed Freeze5, we selected cohorts of 10k, 36k, and 50k subjects. The benchmarking results for the selected tasks would vary if the cohorts were defined differently.

The selection of instances is another factor that can lead to variation in results for a given analysis. The results highly depend on the user’s skill to choose an appropriate instance and use the instance resources optimally. For that reason, if two users run the same task with different configurations (different instance type CPU/job and/or RAM/job parameters), the results may vary.

The results (execution time and cost) are directly connected to the CPU per job and Memory per job parameters. Different resources dedicated to a given job will result in a different number of total jobs run on the selected instance, as so with the different execution time and cost. For that reason, setting up a task draft properly is crucial. In this document, we provided details on what we consider optimal CPU and Memory per job inputs for TOPMed Freeze5 and 1000G data. These numbers can be used as a good starting point, bearing in mind that each study has its own unique requirements.

For both Single and Sliding Window Association Testing:

Please note that results for single and sliding window tests are approximations. To avoid unnecessary cloud costs, we performed both single and sliding window tests only on 2 chromosomes. These results were the basis on which we assessed the cost and execution time for the whole genome.

The following is an explanation of the procedure we applied for GENESIS Single Association testing workflow and TOPMed freeze5 data (the similar stands for GENESIS Sliding window Association testing workflow):

In GENESIS Single Association testing workflow, the variants are tested in segments. The number of segments that the workflow will process is a ratio of the total number of variants and a segment length (which is one of the input parameters in this workflow). For example: if we are testing a whole genome with 3,000,000,000 variants and use the default segment length value of 10,000kb, we will have 300 segments. Furthermore, if we use the default value for maximum number of parallel instances, which is 8, we can approximate the average number of segments that each instance processes: 37.

The GENESIS Single Association testing workflow can process segments in parallel (processing of one segment is a job). The number of parallel segments (jobs) depends on the CPU per job and Memory per job parameters, and can be calculated as described previously. For example: if we are running the analysis on a c5.9xlarge instance (36 CPUs and 72GB RAM) with 1 CPU/job and 4GB/job, we will have 18 jobs in parallel. Knowing that each of our 8 instances is processing approximately 37 jobs in parallel it means that each instance will have approximately 2 cycles. Furthermore, knowing the average job length we can approximate the running time of 1 instance: it will be 2 cycles multiplied by average job length. Since the instances are running in parallel, this will be the total execution time. Lastly, when execution time is known, we can calculate the task price: the number of instances multiplied by execution time per hour, multiplied by instance price per hour. For each tested scenario in our benchmarking analysis, we obtained the average job length based on the corresponding tasks which included 2 chromosomes, such that the total number of jobs was above 30.

• Agile Development

  Agile software development is an approach to software development under which requirements and solutions evolve through the collaborative effort of self-organizing and cross-functional teams and their customer(s)/end user(s).

• Alpha Users

  A small group of users who are more willing to tolerate working in a system that isn’t as fully developed, providing detailed feedback & maybe some back & forth discussions

• [Amazon] EFS

  [Amazon] Elastic File System a simple, scalable, elastic file system for Linux-based workloads for use with AWS Cloud services and on-premises resources.

• Ambassadors

  A small group of experts that represent the personas featured within the priority User Narratives. For their time and help, Ambassadors will receive early access to the BDC platform, free compute time, monetary fee for time, and relevant travel expenses will be covered.

• App

  1. In Seven Bridges, an app is a general term to refer to both tools and workflows.

  2. App may also refer to persistent software that is integrated into a platform.

• API

  Application Programmer Interfaces. API technologies serve as software-based intermediaries to exchange data.

• AWS

  Amazon Web Services. A provider of cloud services available on-demand.

• BagIt

  BagIt is a hierarchical file packaging format for storage and transfer of arbitrary digital content.

• BDC3

  BDC Coordinating Center

• Beta Users

  A slightly larger group than the alpha users who are not as tolerant to a difficult/clunky environment but understand that the version they are using is not polished and they need to give feedback.

• Beta-User Training

  Once the platform is available to a broader audience, we will support freely-accessible online training for beta-users at any time.

• Carpentries Instructor Training Program

  Ambassadors attend this training program to become BDC trainers.

• CCM

  Change Control Management; the systematic approach to managing all changes made to a document or process. Ensures no unnecessary changes are made, all changes are documented, and a process exists for implementing approved change.

• CIO

  Chief Information Officer

• Cloud Computing

  Internet-based computing, wherein computing power, networking, storage or applications running on computers outside an organization are presented to that organization in a secure, services-oriented way.

• Components

  Software units that implement a specific function or functions and which can be reused.

• ConOps

  Concept of Operations

• Consortium

  A collection of teams and stakeholders working to deliver on the common goals of integrated and advanced cyberinfrastructure, leading-edge data management and analysis tools, FAIR data, and HLBS researcher engagement.

• Containers

  A standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another (for example, Docker).

• Command

  In workflow writing, the command specifies the literal command line run (akin to what you would run in the terminal).

• COPDGene

  Chronic Obstructive Pulmonary Disease (COPD) Gene

• Cost Monitoring (level)

  At the Epic Level The Coordinating Center will facilitate this process by developing reporting templates (see example in PM Plan, Financial Management) for distribution to the teams. The BDC teams will complete these templates and send them directly to NHLBI. Each team is responsible for tracking their finances based upon the award conditions and for providing status updates as requested to NHLBI.

• CRAM File Compressed columnar file format for storing biological sequences aligned to a reference sequence. Designed to be an efficient reference-based alternative to the Sequence Alignment Map (SAM) and Binary Alignment Map (BAM) file formats. It optionally uses a genomic reference to describe differences between the aligned sequence fragments and the reference sequence, reducing storage costs. Additionally each column in the SAM format is separated into its own blocks, improving compression ratio. CRAM files typically vary from 30 to 60% smaller than BAM, depending on the data held within them (from Wikipedia).

• CSOC Alpha

  Common Services Operations Center (CSOC): operates cloud, commons, compliance and security services that enable the operation of data commons; has ATO and hosts production system.

• CSOC Beta

  Development/testing; Real data in pilot (not production) that can be accessed by users

• Common Workflow Language (CWL)

  Simple scripting language for describing computational workflows for performing sequential operations on data. CWL is a way to describe command line tools and connect them together to create workflows. CWL is well suited for describing large-scale workflows in cluster, cloud and high performance computing environments where tasks are scheduled in parallel across many nodes.

• DAC

  Data Access Committee: reviews all requests for access to human studies datasets

• DAR Data Access Request

• Data Access

  A process that involves authorization to access different data repositories; part of a User Narrative for the December 2020 release goal A Work Stream PM Plan constraint: NHLBI, as the project sponsor, will identify a process to enable data access by the BDC team members and for research users

• Data Commons

  Provides tools, applications, and workflows to enable computing large scale data sets in secure workspaces.

• Data Repository Service (DRS) Generic interface (API) to data repositories so data consumers, including workflow systems, can access data in a single, standardized way regardless of where it’s stored or how it’s managed. The primary functionality of DRS is to map a logical ID to a means for physically retrieving the data represented by the ID (from GA4GH).

• Data Steward

  Members of the TOPMed and COPDGene communities who are working with BDC teams.

• dbGaP

  Database of Genotypes and Phenotypes

• DCPPC

  Data Commons Pilot Phase Consortium. The Other Transaction Awardees, Data Stewards, and the NIH.

• Decision Tree

  A decision support tool that uses a tree-like model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility

• Deep Learning

  A machine learning method based on neural networks to learn from data through training to recognize patterns in the data.

• Deliverables

  Demonstrations and products.

• Demos

  Activities and documentation resulting from the DCPPC to build, test and demonstrate completion of goals of the Data Commons Pilot Phase.

• DEV Environment

  Set of processes and programming tools used to create the program or software product

• DMI

  Data Management Incident

• Docker

  Software for running containers, packaged, portable units of code and dependencies that can be run in the same way across many computers. See also Containers.

• Dockerfile

  A text document that contains all the commands a user could call on the command line to assemble an image.

• Dockstore

  An open platform developed by the Cancer Genome Collaboratory and used by the GA4GH for sharing Docker-based tools described with the Common Workflow Language (CWL), the Workflow Description Language (WDL), or Nextflow (NFL)

• DOI

  Digital Object Identifier; a code used to permanently and stably identify (usually digital) objects. DOIs provide a standard mechanism for retrieval of metadata about the object, and generally a means to access the data object itself.

• DUO

  Data Use Ontology - a GA4GH standard for automating access (API) to human genomics data (https://github.com/EBISPOT/DUO)

• DUOS

  Data Use Oversight System, https://duos.broadinstitute.org/

• Ecosystem

  A software ecosystem is a collection of processes that execute on a shared platform or across shared protocols to provide flexible services. Example: The "BDC Ecosystem" - inclusive of all platforms and tools

• EEP

  External Expert Panel. A group of experts who provide guidance and direction to NIH about the program.

• Epic

  A very large user story which can be broken down into executable stories

  *NHLBI’s cost-monitoring level

• eRA Commons

  Designated ID provider for whitelist

• External Expert Panel

  An independent body of experts that inform and advise the work of the BDC Consortium.

• FAIR

  Findable Accessible Interoperable Reusable.

• Feature

  A functionality at the system level that fulfills a meaningful stakeholder need

  *Level at which the CC coordinates

• FireCloud

  Broad Institute secure cloud environment for analytical processing, https://software.broadinstitute.org/firecloud/

• FISMA moderate environment

  Federal Information Security Modernization Act of 2014, amends the Federal Information Security Management Act of 2002 (FISMA), see https://www.dhs.gov/fisma

• FS

  Full Stack

• GA4GH

  Global Alliance for Genomics and Health

• GA4GH APIs

  The Genomic Data Working Group is a coalition assembled to create interoperability standards for storing and sharing genomic data. The GA4GH Genomics API offers Interoperability for exchanging genomic data between various platforms and organizations by sending simple HTTP requests through a JSON equipped RESTful API.

• GCP

  Google Cloud Platform

• GCR

  Governance, Compliance, and Risk

• Gen3

  Gen3 is open source and licensed under the Apache license, which you can use for setting up, developing and operating data commons

• GitHub

  An online hub for storing and sharing computer programs and other plain text files. We use it for storage, hosting websites, communication and project management.

• Gold Master

  A gold master, or GM, is the final version of software or data ready for release to production; a master version from which copies can be made.

• GWAS

  Genome-wide Association Study

• HLBS

  Heart, Lung, Blood, Sleep

• Identity Providers

  A system entity that creates, maintains, and manages identity information for principals while providing authentication services to relying applications within a federation or distributed network; identity providers offer user authentication as a service

• Interoperability

  The ability of data or tools from multiple resources to effectively integrate data, or operate processes, across all systems with a moderate degree of effort.

• Instance

  In cloud computing, refers to a virtual server instance from a public or private cloud network.

• Image

  In the context of containers and Docker, this refers to the resting state of the software.

• IP

  BDC Implementation Plan; outlines how the various elements from the planning phase of the BDC project will come together to form concrete, operationalized BDC platform.

• IRB

  Institutional Review Board; the entity within a research organization that reviews and approves research protocols and clinical research protocols to protect human and animal subjects.

• IRC

  Informatics Research Core

• ISA

  Interoperability Service Agreement

• ITAC

  Information Technology Applications Center

• Jupyter Notebooks

  A web-based interactive environment for organizing data, performing computation, and visualizing output.

• Linux

  An open source computer operating system

• Metadata

  Data about other data

• Milestone

  Marks specific progress points on the development timeline, and they can be invaluable in measuring and monitoring the evolution and risk of a program. © Scaled Agile, Inc.

• MSD

  Minimum set of documents

• MVP

  Minimum viable product

• NHLBI

  National Heart, Lung, and Blood Institute

• NIH

  National Institutes of Health

• NIST Moderate controls

  NIST 800-53 - A collection of security controls and assessment procedures that both U.S. Federal and non-governmental organizations can apply to their information systems, policies, and procedures.

• OTA

  Other Transaction Authority - the mechanism of award that NHLBI chose because it provides a degree of flexibility in the scope of the work that is needed to advance this type of high risk/high reward project

• PI

  Principal Investigator

• Platform

  A piece of the BDC ecosystem. Examples: Terra, Gen3, Seven Bridges, etc.

• PM

  Project Manager

• PMP

  BDC Project Management Plan; breaks down the implementation of BDC from the perspective of the project managers involved in the project including details on roles, specific milestones, and the project schedule.

• PO

  Program Officer

• Portable Format for Biomedical Data (PFB) Avro-based serialization format with specific schema to import, export and evolve biomedical data. Specifies metadata and data in one file. Metadata includes data dictionary, ontology references and relations between nodes. Supports versioning, back- and forward compatibility. A binary format.

• Portfolio for Jira

  Software-as-a-Service project management tool, used to track, roadmap, and visualize various project metrics.

• Python

  Open source programming language, used extensively in research for data manipulation, analysis, and modeling

• Quality Assurance

  The planned and systematic activities implemented in quality management so that quality requirements for a product or service satisfy stated goals and expectations.

• Quality Control

  The operational techniques and activities aimed at monitoring and measuring work processes and eliminating the causes of unsatisfactory outputs.

• RACI

  Responsible, Accountable, Consulted and Informed; tool that can be used for identifying roles and responsibilities during an organizational change process; BDC RACI

• Researcher Auth Service (RAS) Will be a service provided by NIH's Center for Information Technology to facilitate access to NIH’s open and controlled data assets and repositories in a consistent and user-friendly manner. The RAS initiative is advancing data infrastructure and ecosystem goals defined in the NIH Strategic Plan for Data Science.

• RFC

  Request for Comment: A process that documents and enables effective interactions between stakeholders to support shared decision making.

• Risk Register

  A tool used to continuously identify risk, risk response planning and status updates throughout the project lifecycle. This project risk register is the primary risk reporting tool, and is located in the Project Management Plan.

• SC

  Steering Committee

• Scientific use case

  Defined in this project as an analysis of data from the designated sources which has relevance and value in the domain of health sciences, probably implementation and software agnostic.

• SF or SFP

  BDC Strategic Framework [Plan]; defines what the BDC teams have accomplished up to this point, what we plan to accomplish in a timeline fashion, and milestones to track and measure implementation.

• SFTP

  Secure File Transfer Protocol

• Software Developers Kit

  A set of software development tools that allows the creation of applications for a certain software package, software framework, hardware platform, computer system, or similar development platform

• Sprints

  Term of art used in software generation, referring to short, iterative cycles of development, with continuous review of code through daily builds and end-of-sprint demos

• Stack

  Term of art referring to a suite of services that run in the cloud and enable ubiquitous, convenient, on-demand access to a shared pool of configurable computing resources.

• Steering Committee

  Responsible for decision-making and communication in BDC.

• STRIDES

  Science & Technology Research Infrastructure for Discovery, Experimentation, and Sustainability

• Task

  In workflow writing, the term task encompasses all of the information necessary to execute a command, such as specifying input/output files and parameters.

• Team

  Groups of people led by a Principal Investigator (PI), or PIs, who will complete milestones and produce deliverables. Each group has been assigned a name, represented by the elements on the periodic chart.

• Tiger Teams

  A diversified group of experts brought together to investigate, solve, build, or recommend possible solutions to unique situations or problems. Populated with mature experts who know what's at stake, what needs to be done, and how to work well with others; their strengths are diversity of knowledge, a single focus or purpose, cross-functional communications, decision-making sovereignty, and organizational agility.

• Tool

  In Common Workflow Language, the term tool specifies a single command. This definition is not as discrete in other workflow languages such as WDL.

• Tool Registry Service (TRS)

  The GA4GH Cloud Work Stream has released a standard API for exchanging tools and workflows to analyze, read, and manipulate genomic data. The Tool Registry Service (TRS) API is one of a series of technical standards from the Cloud Work Stream that together allow genomics researchers to bring algorithms to datasets in disparate cloud environments, rather than moving data around.

• TOPMed

  Trans-Omics for Precision Medicine. One of the primary data sets of the DCPPC.

• TOPMed DCC

  TOPMed Data Coordinating Center

• Trans-cloud

  A provider-agnostic multi-cloud deployment architecture.

• User Narrative

  Descriptions of a user interaction experience within the system from the perspective of a particular persona. User Narratives are further broken down into Features, Epics, and User Stories. Currently formulated into rough 6-month timelines to benchmark progress.

• User story

  A description of a software feature from a technical/process-oriented perspective; a backlog item that describes a requirement or functionality for a user

  *Finest level of PM Monitoring

• Variant Call Format (VCF)

  File format for storing gene sequence variations. The format has been developed with the advent of large-scale genotyping and DNA sequencing projects, such as the 1000 Genomes Project. Existing formats for genetic data such as General feature format (GFF) stored all of the genetic data, much of which is redundant because it will be shared across the genomes. By using the variant call format only the variations need to be stored along with a reference genome. There is also a Genomic VCF (gVCF) extended format, which includes additional information about "blocks" that match the reference and their qualities (from Wikipedia). See http://www.internationalgenome.org/wiki/Analysis/vcf4.0/.

• VDS

  A composite of complete server hardware, along with the operating system (OS), which is powered by a remote access layer that allows end users to globally access their server via the Internet

• VPC

  Virtual Private Cloud

• Whitelist

  A security measure to permit only an approved list of entities. We recommend instead using the term "allow list".

• Workflow

  A sequence of processes, usually computational in this context, through which a user may analyze data.

• Workflow Description Language (WDL) Way to specify data processing workflows with a human-readable and writeable syntax. Define complex analysis tasks, chain them together in workflows, and parallelize their execution.

• Workspace

  Areas to work on/with data within a platform. Examples: projects within Seven Bridges

• Workstream

  A collection of related features; orthogonal to a User Narrative

• Wrapping

  The process of describing a command-line tool or custom script in Common Workflow Language (CWL) so that it can be easily run in multiple cloud environments, dynamically scale compute requirements based on inputs, and be inserted into distinct analysis pipelines.

• Virtual Machine (VM)

  An isolated computing environment with its own operating system.

Though the primary goal of the BDC project is to build a data science ecosystem, at its core, this is a people-centric endeavor. BDC is also building a community of practice working collaboratively to solve technical and scientific challenges in biomedical science.

More about who we are and the partners empowering our ecosystem can be found at the .

Terra workspaces include a dedicated workspace Google bucket, as well as a built-in data model for managing your data. We provide articles in Terra’s knowledge base explaining how to organize and access data in a variety of ways.

A key to understanding the power of Terra is understanding it’s built-in data model, which allows you to rewire the inputs and outputs of your workflows and Jupyter notebooks.

The following resources give you guided instructions using cloud-based data with Terra: Article: ​Managing data with table VIdeo: Introduction to Terra data tables Article: Uploading to a workspace Google bucket​ Article: ​How to import metadata to a workspace data table Video: Making and uploading data tables to Terra

Click here to access the NHLBI BioData Catalyst^® (BDC) website.

Welcome to NHLBI BioData Catalyst^® (BDC)

Welcome to the BDC ecosystem and thank you for joining our community of practice. The ecosystem offers secure workspaces to support your data analysis in addition to a number of bioinformatics tools for analysis. There is a lot of information to understand and many resources (documentation, learning guides, videos, etc.) available, so we developed this overview to help you get started. If you have additional questions, use the links at the very end of this document, under the "Questions" section, to contact us.

About BDC and Our Community

What is BDC?

NHLBI BioData Catalyst^® (BDC) is a cloud-based ecosystem that offers researchers data, analytical tools, applications, and workflows in secure workspaces. BDC is a community where researchers can find, access, share, store, and analyze heart, lung, blood, and sleep data. BDC is an NHLBI data repository where researchers share scientific data from NHLBI-funded research, so they and others can reproduce findings and reuse data to advance science.

By increasing access to NHLBI data and innovative analytic capabilities, BDC accelerates reproducible biomedical research to drive scientific advances that can help prevent, diagnose, and treat heart, lung, blood, and sleep disorders.

What are we doing and why does it matter?

By increasing access to the NHLBI’s datasets and innovative data analysis capabilities, the BDC ecosystem accelerates efficient biomedical research that drives discovery and scientific advancement, leading to novel diagnostic tools, therapeutics, and prevention strategies for heart, lung, blood, and sleep disorders.

Who is developing BDC?

The ecosystem is funded by the National Heart, Lung, and Blood Institute (NHLBI). Researchers and other professionals receive funding from the NHLBI to work on the development of the ecosystem, together often referred to as “The BDC Consortium” . You can refer to on the Overview page of the BDC website and a is available in our documentation.

Find out the meanings of our terms and acronyms.

Like many professional communities, BDC has adopted terms to help us communicate quickly and more efficiently, but that can be a challenge for newcomers. To help, we created a BDC of terms and acronyms. If ever there is a time when an ecosystem term or acronym is unfamiliar and isn’t in the glossary, so we can give you the information and add it to the glossary.

Learn about the platforms available in the ecosystem.

The BDC ecosystem features the following platforms.

Explore Available Data

• BDC Powered by Gen3 (BDC-Gen3) - Hosts genomic and phenotypic data and enables faceted search for authorized users to create and export cohorts to workspaces in a scalable, reproducible, and secure manner.

• BDC Powered by PIC-SURE (BDC-PIC-SURE) - Enables access to all clinical data, feasibility queries to be conducted, and allows cohorts to be built in real-time and results to be exported via the API for analysis.

Analyze Data in Cloud-based Shared Workspaces

• BDC Powered by Seven Bridges (BDC-Seven Bridges) - Collaborative workspaces where researchers can find and analyze hosted datasets as well as their own data by using hundreds of optimized analysis tools and workflows in CWL, as well as JupyterLab and RStudio for interactive analysis.

• BDC Powered by Terra (BDC-Terra) - Secure collaborative place to organize data, run and monitor workflow analysis pipelines in WDL, and perform interactive analysis using applications such as Jupyter Notebooks and the Hail GWAS tool.

The BDC website provides details about the available in the ecosystem. We encourage you to create accounts on all the platforms as you get to know BioData Catalyst.

How do I login?

Users log into BioData Catalyst platforms with their eRA Commons credentials (see ) and authentication is performed by iTrust. Every time a user logs in, the ecosystem checks his/her user credentials to ensure s/he can only access the data for which s/he has dbGaP approval.

While all of the platforms within BioData Catalyst use eRA Commons credentials and iTrust performs authorization and authentication, respectively, there are some slight differences between the platforms when getting set up:

• BioData Catalyst Powered by Gen3 - Users do not set up usernames on Gen3. Upon the first time logging in, select “Login from NIH”, then enter eRA commons credentials at the prompt. This ‘User Identity’ is used to track the user on the system.

• BioData Catalyst Powered by PIC-SURE - Similar to Gen3, user identities are used - researchers log into the system by selecting “Log in with eRA Commons.”

• BioData Catalyst Powered by Seven Bridges - Users set up platform accounts. The first time on the system, users select to “Create an account” and then proceed with entering their eRA Commons credentials. The user is then prompted to fill out a registration form with their name, email, and preferred username. Users are also asked to acknowledge that they have read the Privacy Act notice and then they can proceed to the platform.

Details about how data access works on the NHLBI BioData Catalyst ecosystem are .

How do I check which data I can access?

We recommend users first check their access to data before logging in. Do this by going to the and clicking on the “Check My Access” button. Once you confirm your data access, go to the page from which you click on the “Launch” hyperlink for the platform or service you wish to use. Platforms and services have login/sign in links on their pages that bring you to the pages on which you enter your eRA Commons credentials. on checking your access to data is also available.

What data are available in the ecosystem?

The NHLBI BioData Catalyst currently hosts a subset of datasets from TOPMed including phs numbers with genomic data and related phs numbers with phenotype data. You can find information about which are currently hosted on the of the website as well as in the .

Harmonized data available.

There are limited amounts of harmonized data available to users with appropriate access at this time. The TOPMed Data Coordinating Center curation team has produced forty-four (44) harmonized phenotype variables from seventeen (17) NHLBI studies. Information about the 17 studies and the 44 variables can be found in the .

Bring your own data and workflows into the system.

We allow researchers to bring their own data and workflows into the ecosystem to support their analysis needs. Researchers can bring their own datasets into and . Users can also bring their own workflows to the system. Users can either add workflows to in CWL or WDL, or they can directly on BioData Catalyst Powered by Seven Bridges and for use on BioData Catalyst Powered by Terra.

Learn about Genome-wide association study and genetic association testing on BioData Catalyst.

Walk through our self-paced genome-wide association study and genetic association testing .

Share your workflows.

We encourage users to publish their workflows so they can be used by other researchers working in the NHLBI BioData Catalyst ecosystem. Share your workflows via .

Costs and cloud credits.

BioData Catalyst hosts a number of datasets available for analysis to users with appropriate data access approvals. Users are not charged for the storage of these hosted datasets; however, if hosted data is used in analyses users incur costs for computation and storage of derived results. Cloud credits are available on the system, and you can .

Learn more, ask questions, or request help.

Answers to are available on the website, as are many resources that can be found under . You can also use a form to , and if you aren’t sure which selections to make on the form, please see our .

You can import data into your workspace by either linking directly to external files you have access to, or by interfacing with a number of platforms with which Terra has integrated access.

For BDC researchers, one of the most relevant of these interfacing platforms is Gen3. However this section also provides you with resources that teach how to import data from other public datasets integrated into Terra’s data library, as well as how to bring in your own data.

Read on in this section for more information on:

• Bringing in data from Gen3

In the context of agile development and a Consortium with a diverse set of members, the application of various agile-development terms may mean different things to different individuals.

The table below defines the BDC Core Terminology:

If you are writing a manuscript about research you conducted using BDC, use the citation available below.

After learning your manuscript has been accepted, let us know by filling out our Contact form and selecting Published Research under the "Select the Type of Assistance Needed" dropdown.

For citation of BDC:

National Heart, Lung, and Blood Institute, National Institutes of Health, U.S. Department of Health and Human Services (2020). The NHLBI BioData Catalyst. Zenodo. https://doi.org/10.5281/zenodo.3822858

To acknowledge BDC, use:

The authors wish to acknowledge the contributions of the consortium working on the development of the NHLBI BioData Catalyst® (BDC) ecosystem.

BDCatalyst-RFC-#: 11 BDCatalyst-RFC-Title: NHLBI BioData Catalyst Ecosystem Security Statement BDCatalyst-RFC-Type: Consensus Building Name of the person who is to be Point of Contact: Sarah Davis Email of the person who is to be Point of Contact: sdavis@renci.org Submitting Team: BDC3/NHLBI Requested BDCatalyst-RFC posting start date: 6/14/2021 Date Emailed for consideration: 6/14/2021 BDCatalyst-RFC-Status: Comment only URL Link to this document: https://bdcatalyst.gitbook.io/biodata-catalyst-documentation/community/request-for-comments/nhlbi-biodata-catalyst-ecosystem-security-statement URL Link to the website: https://www.biodatacatalyst.org/collaboration/rfcs/bdcatalyst-rfc-11/ License: This work is licensed under a CC-BY-4.0 license.

Overview

The purpose of this RFC is to provide the NHLBI BioData Catalyst Consortium and users of the NHLBI BioData Catalyst ecosystem with a clear statement on security mechanisms of the ecosystem that protect the confidentiality, integrity, provenance, and availability of the hosted data as well as any data that may be uploaded using the ecosystem’s “Bring Your Own Data” (BYOD) functionality.

Figure 1. The NHLBI BioData Catalyst ecosystem leverages separately developed and managed platforms to maximize flexibility for users based on their research needs, expertise, and backgrounds. Utilizing multiple Authorizations to Operate (ATO), these platforms combine to provide secure, cloud-based workspaces, user authentication and authorization, search, tools and workflows, applications, and new innovative features to address community needs.

The NHLBI and BioData Catalyst Consortium recognizes the importance of protecting both the privacy and security of the data and respecting the consent of the study participants whose data is stored within the BioData Catalyst ecosystem. Tackling these issues produces some challenges beyond those faced by most Federal Information Systems. The BioData Catalyst Consortium has implemented many innovative approaches to enable compliance and ensure that users understand their responsibility to protect data as articulated in specific Data Use Agreements (DUA). These approaches and controls work to protect the confidentiality, integrity and availability of the data; the privacy of the study participants who have contributed data; and data that may be uploaded to BioData Catalyst using the ecosystem’s “Bring Your Own Data” (BYOD) functionality. While the same general security controls are applied to both system and BYOD data, BYOD data is further protected as the ecosystem provides access only to the data’s uploaders and their designated collaborators.

From a Federal Information Security Modernization Act (FISMA) perspective, the BioData Catalyst ecosystem is a set of software systems with distinct security boundaries. Each system owner holds an Authority to Operate (ATO) issued by the NIH. The ATO is the result of a rigorous Security Assessment and Authorization (SA&A) process and third party assessment consistent with guidance from the National Institute of Standards and Technology (NIST). The ecosystem operates via a set of Interconnection Security Agreements (ISA) (Reindl 1979) and utilizes several existing components of security infrastructure (Bridges 2017, Gutiérrez-Sacristán et al. 2018) developed for other NIH platforms. Where the documentation provided as part of the SA&A process describes how security controls are implemented based on the NIST Special Publication 800-53r4 (see Endnote), the ISAs describe the permitted exchange of data and establish ecosystem-wide incident response, logging and auditing expectations that enable the consortium to respond in a unified manner to any suspected cybersecurity incident. The SA&A documentation provides for regular evaluation of the security of the component systems including regular scanning for vulnerabilities and the conduct of an annual penetration test. This level of security represents a baseline, and the BioData Catalyst ecosystem will extend protections over time.

Where the processes, policies, and technical controls protect confidentiality, integrity, and availability of data in accordance with Federal statute and regulation, there are additional ways to ensure that data is used in a manner consistent with study participants’ wishes, as represented by the consent form participants sign when enrolling in a specific study. Respect for these consents is critical to maintaining the public’s trust and requires additional policy, process, and technical controls. The respect for consent in NHLBI BioData Catalyst is enforced using normative NIH policies and processes for data sharing and using the existing infrastructure provided by the National Center for Biomedical Information’s (NCBI) Database of Genotypes and Phenotypes (dbGaP). All NHLBI-provided data within the NHLBI BioData Catalyst ecosystem are registered in dbGaP; in this process, data are assigned “consent groups” that describe in a machine-readable format the parameters of the consent for the data. These range from the most expansive “General Research Use” to more restrictive, such as only allowing for research outcomes related to “Health Medical Biomedical topics” or even to specific diseases, such as Chronic Obstructive Pulmonary Disease. Further, while secondary analysis of data is not considered human subjects research as described in the Common Rule (45 CRF Part 46), some datasets require the review of a research proposal by an Independent Review Board (IRB) or a Letter of Collaboration (LOC) with the originating study Principal Investigator as determined by the informed consents or special considerations that the submitting institution has determined are needed. These measures provide additional protection for datasets with particular sensitivity or special criteria for use.

For instance, IRB review is required when 1) informed consents that were signed by the study participants state that IRB oversight for secondary use of the data is required, and/or 2) the study IRB of record determines that the data may contain sensitive information that requires IRB oversight for secondary research. For collaboration letters, the informed consents indicate that the secondary use of the data by researchers outside the study will work with the study and therefore formal collaborations need to be set in place. While these are rarely used, they provide additional protection under special circumstances, such as where an indigenous population or sovereign nation requires direct control of how their data is used. Because consent is expressed at the individual level there may be a variety of consents for a study, either because the study offered choices to their participants or because the study consent evolved over an extended longitudinal study, such as the Framingham Heart Study. These variations in consent are reflected as multiple “consent groups” within a study and may mean that an investigator may only receive permission for subsets of study participants.

BioData Catalyst uses telemetry provided by dbGaP to enforce compliance with consents. Accordingly, users of BioData Catalyst can see only the data for which they have completed the process of a dbGaP Data Access Request (DAR) and after they receive approval from an NIH Data Access Committee (DAC). DAC approval results in a Data Use Agreement (DUA) describing any Data Use Limitations asserted by the originating study Principal Investigator, including ensuring that any requirements for an IRB review or Letter of Collaboration are met. On the BioData Catalyst ecosystem, the Gen3 “Fence” service, developed and operated by the University of Chicago and utilized by other NIH platforms and many other non-federal data commons, ensures enforcement of data access requirements. In order to ensure that the NHLBI maintains control over the use of the data and has the ability to audit this use, the BioData Catalyst policy does not allow download of source data outside the cloud ecosystem. Instead, BioData Catalyst provides access to data in approved cloud environments where computation occurs, which is documented in an ISA via standard Application Programming Interfaces (APIs) that use the authentication and authorization provided by the NIH Researcher Auth Service (RAS) (REF), together with Fence, to protect access and to allow for monitoring and auditing for appropriate data use (e.g. within scope of the approved project). These APIs include implementations of protected GA4GH Data Repository Service (DRS) for access to data objects such as genomic data and protected PIC-SURE interfaces for access to phenotypic, genotypic variant, and electronic health record data. The use of these APIs will, once fully implemented, enable other trusted cloud-based systems that meet equivalent security requirements to access data stored within BioData Catalyst for analysis using that other systems’ tools without the data being downloaded outside the security perimeters of the systems. This commitment to the use of APIs together with the requirement that data stay within the designated security boundaries is a critical component of making NHLBI data FAIR (Findable, Accessible, Interoperable, Reusable) (Wilkinson et al. 2016, Corpas et al. 2018), while also ensuring confidentiality of data and respect for consent, regardless of the platform where the data is analyzed. BioData Catalyst has extended this model through the use of the (RAS) to provide authentication and authorization controls which together with the use of secure APIs is enabling secure interoperability with other trusted NIH-funded platforms such as the NHGRI’s AnVIL and the Gabriella Miller Kids First Data Resource.

While NIST-800-53r4 is a thorough standard for many kinds of systems, it has some gaps for the most-modern systems encountered. There are additional standards to apply, either from the High column of NIST-800-53r4 or addendums as “best practice”.

In particular, the Standard does not give guidance on modern applications (“appsec”) -- especially when the Infrastructure is serverless and the entire security surface is the application itself. For instance, there are requirements for regular scanning but scanning tools do not scan APIs or modern Single Page Apps (SPA) sufficiently. A “web app scan” with tools like IBM’s Appscan would return no vulnerabilities on an API without actually testing it, yet it would satisfy RA-5 of NIST-800-53 to run such a scan. Nor does the standard require that the Infrastructure as a Service layer (AWS, GCP, Azure) abide by a continual scanning posture for misconfiguration -- only Networks and VMs are specified. That is one example of where NIST-800-53r4 doesn’t work for modern applications.

There’s also no guidance from NIST at large about running an API where external parties are building “clients'' to that API. Does allowing clients extend the security boundary, and thus all 3rd party applications are to be evaluated as such? Is there a different consideration for these 3rd party applications? The standard is silent there. Companies like Apple and Google through their AppStores require all 3rd party apps to undergo evaluation to their own security standards and such an idea might be applicable here. NHLBI might consider adding some extra controls, what the AllOfUs Program calls FISMA+, for enhanced security.

The BDC user community is essential to advancing science with new and exciting discoveries and informing the development of the ecosystem and its infrastructure. Members of the BDC user community learn how to explore the hosted data, use the services, and employ its tools in exciting and valuable ways that even developers may not know. Therefore, we actively invite user resource contributions to be shared with the community.

Consider supporting fellow ecosystem users in one of the following ways:

• Written Documentation: Develop step-by-step guides, FAQs, checklists, and so on. Include screenshots to support user understanding.

BDC recognizes the importance of multimedia resources for ecosystem users, particularly audio/visual recordings. This document provides guidelines on the program's video content approach. Using these guidelines will ensure users get optimized video experiences, from consistent branding that offers insights into the sources of the videos to best practices in video creation that support learning.

To share video content - from the consortium, platforms, and users, as described in the following sections - BDC created a YouTube channel:

The BioData Catalyst Coordinating Center (BDC3) , has authority (with direction from the NHLBI) to post (or not post), remove, edit, and otherwise change video content on this channel with or without permission from or notice to video creators, owners, or sharers. Feedback about videos on the BDC YouTube channel should be sent to BDCatalystOutreach@nih.gov.

The BDC YouTube Channel hosts three categories of videos based on their sources and/or approval statuses:

• An NIH eRA Commons ID (or appropriate NIH Login) is required for submitting a Data Access Request (DAR). If you do not have an eRA Commons account, you must request one through your institution’s Office of Sponsored Research or equivalent. For more information, refer to on the eRA website.

• To submit a DAR, users must have PI status through their institution. Non-PI users must have a PI they work with that can submit a DAR and add them as a downloader.

Step 1: Go to to log in to dbGaP.

Sharing a workspace allows collaborators to actively work together in the same project workspace. Workspaces can be used as repositories of data, workflows, and Jupyter notebooks. Learn more about how to securely share a workspace: Article: Article: Article: Video:

BDC Website

Go to the Explore BDC Data page of the BDC website. Under the section, "Requirements for Accessing BDC Hosted Data," click Check My Access.

BDC Powered by Gen3 (BDC-Gen3)

Go to BioData Catalyst Powered by Gen3, select NIH Login, then log in using your NIH credentials. Once logged in, select the Exploration tab. From the Data Access panel on the left, make sure Data with Access is selected. Note whether you have access to all the datasets you expect.

Data Access

You do not need to check your data access on BDC-PIC-SURE. Instead, refer to the page, then click Check My Access.

1. Log into .

2. Click your username in the upper right, then select Account settings.

3. From the upper left, select the tab for Dataset Access.

You do not need to check your data access on BDC-Terra. But before submitting a help desk ticket, ensure that you’ve done the following steps:

Establish a link in to your eRA Commons/NIH Account and the University of Chicago DCP Framework. To link eRA Commons, NIH, and DCP Framework Services, go to your Profile page in BDC-Terra and log in with your NIH credentials.

If your access still has issues using particular files or datasets in analyses on BDC-Terra, submit a request to our .

Users log into BDC platforms with their eRA Commons credentials. For more information, see.

Users create an eRA Commons Account through their institution's Office of Sponsored Research or equivalent. For more information, refer to.

Users who want to access a hosted controlled study on the BDC ecosystem must be approved for access to that study in the NIH Database of Genotypes and Phenotypes (). Therefore, users who want to access a hosted controlled study on the ecosystem must be approved for access to that study in dbGaP. For more information, see the . Note that obtaining these approvals can be a time-intensive process; failure to obtain them in a timely manner may delay data access.

Users have two options for obtaining dbGaP approval depending on whether they already are affiliated with a PI who has dbGaP access to the relevant data:

1. Navigate to to access Dug Semantic Search.

2. Semantic search is a concept-based search engine designed for users to search biomedical concepts, such as “asthma,” “lung,” or “fever,” and the variables related to and/or used to measure them. For example, a search for “chronic pain acceptance” will return a list of related biomedical concepts, such as chronic pain, headaches, neuralgia, or fibromyalgia, each of which can be expanded to display related variables and CDEs. Semantic search can also find variable names and descriptions directly, using synonyms from its knowledge graphs to find search-related variables.

Data harmonization on NHLBI BioData Catalyst® (BDC) includes two primary approaches that researchers should understand when selecting datasets for analysis.

Some datasets are harmonized directly by the original data generators or stewards, such as National Sleep Research Resource (NSRR), who apply domain-specific expertise and standardized protocols during data collection and curation.

Other datasets are harmonized by secondary organizations or platforms, including members of the BDC consortium, which integrate and standardize data across studies to improve cross-study usability and interoperability.

Because harmonization methods, variable definitions, and processing decisions may differ between these approaches, researchers should review the dataset summary information carefully before selecting data for analysis. Key details about harmonization processes, provenance, and any transformations applied should be clearly described within each dataset’s summary documentation to support transparent and informed use.

The Discover page provides a publicly available search tool that allows users to:

• View studies and datasets that are available in BDC

• Search and filter on variables to build cohorts

• Retrieve obfuscated, aggregate counts

• View variable distributions to learn more about the selected cohort

Discover all data by navigating to .

The Discover page allows you to search any clinical variable available in PIC-SURE. Queries will return obfuscated aggregate counts per study and consent. There are some features specific to the Discover page, which are outlined below.

A: Faceted search. The panel on the left-hand side of the screen provides options, or facets, to narrow down your search results.

B: Search bar. The search bar at the top center of the page allows for searching of phenotypes, clinical outcomes, and variables of interest.

C. Search results table. The search results table displays the variables that are returned based on the combination of search term and selected facets.

D. Actions for filtering variables. Information about icons from left to right:

Terra has a number of features to ensure the security of sensitive data accessed through the platform. Many of these features are in place automatically, while tools like authorization domains give you greater control over your data. These articles contain an overview of the security features enabled on Terra: Article: ​Article: Article: Article:

The selected participant-level data from PIC-SURE can be handed off to Terra for analysis in the Portable Format for Bioinformatics, or PFB.

To learn more about the PFB format, please refer to the PFB documentation: https://uc-cdis.github.io/pypfb/

Prepare Data for Analysis from PIC-SURE

Once you have selected the cohort and data of interest, the data should be prepared for analysis. During this process, be sure to select the "Export as PFB" option. For more information on how to prepare the data for analysis, please refer to Prepare for Analysis.

After going through the process, select the "Export to Terra" option. This will direct you to the Terra platform.

Exporting to Terra

Selecting "Export to Terra" will direct you to BioData Catalyst Powered by Terra in a new tab. You may be required to sign into Terra.

Select the preferred destination of the data in Terra. This can be an existing workspace created previously or a new one. Follow the steps to choose the location.

Once the selection has been made, the page will be refreshed to the selected location, and the data will begin loading automatically.