Skip to content

Understanding Ingest Templates

What is an Ingest Template?

An ingest template is a set of instructions that defines how to map your local dataset structure to a Flywheel project hierarchy. Templates provide a flexible way to import data while preserving your existing folder organization and extracting metadata from folder and file names.

The Flywheel Project Structure

All Flywheel projects follow a consistent hierarchical structure:

  • Project: The top-level container
  • Subjects: Individual participants or specimens
  • Sessions: Data collection events for a subject (e.g., visits, scans, time points)
  • Acquisitions: Individual scans or data captures within a session

The template maps your local folder structure to these Flywheel containers.

Why Use Templates?

Templates are useful when:

  • Your data follows a custom folder naming convention that contains meaningful information
  • You need to extract metadata from folder or file names (e.g., dates, protocols, cohort identifiers)
  • You want precise control over how data is organized in Flywheel
  • Your data structure does not match standard formats like DICOM or BIDS

When to Use Each Ingest Method

Flywheel provides several ingest methods. Choose the one that best fits your data:

  • fw ingest dicom: Use for raw DICOM files when you want Flywheel to automatically organize data based on DICOM headers
  • fw ingest bids: Use for datasets already organized in BIDS (Brain Imaging Data Structure) format
  • fw ingest folder: Use for data previously downloaded from Flywheel or structured to match Flywheel's exact hierarchy
  • fw ingest template: Use when you need custom mapping logic or want to extract metadata from folder/file names
  • fw ingest project: Use to copy entire projects between Flywheel sites

Recommendation: Start with the simpler commands (dicom, bids, folder) and graduate to templates when you need their additional flexibility.

Template Design Principles

Map Folders to Containers

Each - pattern: line in your template corresponds to one folder level in your dataset. The template walks down your folder structure, applying rules at each level to determine how folders map to Flywheel containers.

Extract Metadata from Names

Templates can extract information from folder and file names and store it as metadata in Flywheel. This is powerful for preserving context that might otherwise be lost, such as:

  • Collection dates or time points
  • Protocol identifiers
  • Cohort or group assignments
  • Scanner or site information

Use curly bracket notation {variable} to extract portions of names as Flywheel labels. Use dot notation {container.info.fieldname} to add custom metadata fields.

Handle Multiple Data Types

Templates can process different file types in different ways:

  • DICOM files: Use scan: dicom to have Flywheel parse DICOM headers and organize by series
  • Other imaging files: Use packfile_type: <extension> to compress files into a single archive
  • Mixed data: Use select: to apply different rules to different folders at the same level

Design for Reusability

Well-designed templates can be reused across multiple datasets with similar structure. Consider parameterizing your templates using configuration files to make them more flexible.

Template Limitations and Considerations

Required Fields

Every template must define labels for all three container types:

  • {subject}
  • {session}
  • {acquisition}

If your data lacks one of these levels, you can use the --no-subjects or --no-sessions flags with the ingest command.

Data Validation

Templates do not validate that folder names are unique or that metadata fields contain valid values. Ensure your source data follows consistent naming conventions before ingesting.

Performance with Large Datasets

For very large datasets (thousands of files), consider using cluster-based ingest (--cluster flag) to improve performance. See the cluster ingest documentation for details.

Next Steps