Skip to content

The fw ingest Command

The fw ingest command can be used to upload large volumes of data to Flywheel in many different formats. The Flywheel CLI provides several ingest commands, each optimized for different data types and use cases.

Which Ingest Command Should I Use?

Quick Decision Tree

graph TD
    A[What type of data do you have?] --> B[DICOM files]
    A --> C[BIDS dataset]
    A --> D[Previously downloaded from Flywheel]
    A --> E[Other imaging data]
    A --> F[Entire Flywheel project]

    B --> G{Need custom folder mapping or metadata extraction?}
    G -->|No| H[fw ingest dicom]
    G -->|Yes| I[fw ingest template]

    C --> J[fw ingest bids]

    D --> K{Matches exact Flywheel hierarchy?}
    K -->|Yes| L[fw ingest folder]
    K -->|No| I

    E --> M{Custom folder structure?}
    M -->|Simple hierarchy| L
    M -->|Complex/custom| I

    F --> N[fw ingest project]

Command Overview

Command Best For Complexity Automatic Organization
fw ingest dicom Raw DICOM files Low ✅ Uses DICOM headers
fw ingest bids BIDS-formatted datasets Low ✅ Uses BIDS structure
fw ingest folder Flywheel hierarchy or simple folders Low ✅ Uses folder names
fw ingest template Custom folder structures Medium-High ❌ Requires template file
fw ingest project Copying entire projects Medium ✅ Mirrors source project

Decision Criteria

Choose your command based on these questions:

  1. Is it DICOM data with good headers? → fw ingest dicom
  2. Is it already in BIDS format? → fw ingest bids
  3. Was it downloaded from Flywheel? → fw ingest folder or fw ingest project
  4. Do you need to extract metadata from folder/file names? → fw ingest template
  5. Is the folder structure irregular or custom? → fw ingest template
  6. Are you copying between Flywheel sites? → fw ingest project

Ingest Subcommands

fw ingest dicom

Use when:

  • You have raw DICOM files (.dcm, .dicom, or no extension)
  • You want Flywheel to automatically organize data using DICOM headers
  • Your DICOMs contain complete metadata (PatientID, StudyDate, SeriesDescription, etc.)

How it works:

  • Scans all files for DICOM headers
  • Creates subjects, sessions, and acquisitions from DICOM metadata
  • Groups images by SeriesInstanceUID
  • No template or folder structure required

Advantages:

  • Simplest to use - just point to a folder
  • Leverages rich DICOM metadata
  • Handles complex folder structures automatically

Limitations:

  • Only works with DICOM files
  • Cannot extract metadata from folder/file names
  • Requires valid DICOM headers

Example:

1
fw ingest dicom /path/to/dicom/data --group mygroup --project "My Study"

Learn more: Importing DICOM Files

fw ingest bids

Use when:

  • Your dataset follows BIDS (Brain Imaging Data Structure) format
  • You have a dataset_description.json file
  • Data is organized according to BIDS conventions

How it works:

  • Reads BIDS folder structure and metadata files
  • Maps BIDS subjects/sessions to Flywheel subjects/sessions
  • Preserves BIDS metadata and relationships
  • Imports associated JSON sidecars

Advantages:

  • Preserves BIDS structure and metadata
  • Standard format widely used in neuroimaging
  • Includes validation of BIDS compliance

Limitations:

  • Only works with BIDS-formatted data
  • Strict folder naming requirements
  • Cannot customize folder-to-container mapping

Example:

1
fw ingest bids /path/to/bids/dataset --group mygroup --project "BIDS Study"

Learn more: Importing BIDS Data

fw ingest folder

Use when:

  • Data was previously downloaded from Flywheel
  • Your folders exactly match Flywheel's hierarchy (group/project/subject/session/acquisition)
  • You have a simple, consistent folder structure

How it works:

  • Maps folder levels directly to Flywheel containers
  • Expected structure: ProjectName/SubjectLabel/SessionLabel/AcquisitionLabel/files
  • Zips folders marked as acquisitions

Advantages:

  • Simple for re-uploading Flywheel exports
  • No template file needed
  • Works with any file types

Limitations:

  • Requires exact folder hierarchy match
  • Cannot skip folder levels or extract metadata
  • Limited flexibility for custom structures

Example:

1
fw ingest folder /path/to/data --group mygroup

Learn more: Importing Structured Folders

fw ingest template

Use when:

  • Your folder structure doesn't match standard formats
  • You need to extract metadata from folder or file names
  • You want precise control over data organization
  • Your data has custom naming conventions

How it works:

  • You create a YAML template file defining mapping rules
  • Template specifies how each folder level maps to Flywheel containers
  • Supports regex for extracting metadata
  • Can handle complex, nested structures

Advantages:

  • Maximum flexibility and control
  • Extract metadata from names (dates, protocols, cohorts, etc.)
  • Handle irregular folder structures
  • Reusable templates for similar datasets

Limitations:

  • Requires creating template file
  • More complex setup
  • Steeper learning curve

Example:

1
fw ingest template /path/to/template.yaml /path/to/data --group mygroup --project "Custom Study"

Learn more:

fw ingest project

Use when:

  • Copying an entire project between Flywheel sites
  • Moving projects between development and production instances
  • Sharing complete datasets with collaborators

How it works:

  • Connects to source Flywheel site using API key
  • Copies all data and metadata from source project
  • Creates matching structure in destination project

Advantages:

  • Preserves complete project structure
  • Copies all metadata and custom fields
  • Works across Flywheel instances

Limitations:

  • Both sites must be on same Flywheel version
  • Does not copy job history, analyses, or permissions
  • Requires API keys for both sites

Example:

1
2
fw ingest project --group destgroup --project destproject \
  fw://sourcegroup/sourceproject source_api_key

Learn more: Copying Projects

Common Scenarios

Scenario 1: Clinical DICOM Archive

Situation: 500 DICOM files from a clinical scanner, organized in random folders

Best choice: fw ingest dicom

Why: DICOM headers contain all needed metadata. Let Flywheel organize automatically.

Scenario 2: Research Data with Study IDs in Folder Names

Situation: Folders named like SUB001_BASELINE_20230115/T1/ with metadata in names

Best choice: fw ingest template

Why: Need to extract subject ID, visit type, and date from folder names. Template can parse this structure.

Scenario 3: Public BIDS Dataset

Situation: Downloaded dataset from OpenNeuro in BIDS format

Best choice: fw ingest bids

Why: Already in BIDS format. Use BIDS ingest to preserve structure and metadata.

Scenario 4: Re-uploading Downloaded Data

Situation: Previously downloaded project from Flywheel, need to upload to different instance

Best choice: fw ingest folder or fw ingest project

Why: Structure already matches Flywheel hierarchy. Use project if copying between sites, folder if uploading local export.

Scenario 5: Mixed File Types with Protocol Information

Situation: NIfTI files organized by protocol: Subject01/RestingState/, Subject01/TaskfMRI/

Best choice: fw ingest template

Why: Need to map folder names to metadata (protocol type) and handle non-DICOM files.

Migration Path

As your needs grow, you can graduate between commands:

Beginner: Start with fw ingest dicom or fw ingest folder → Simplest commands, automatic organization

Intermediate: Move to fw ingest template → When you need custom metadata extraction

Advanced: Use templates with config files and cluster mode → For large-scale, complex ingestion workflows

Advanced Usage

Getting Help

If you're still unsure which command to use:

  1. Check the CLI examples for similar use cases
  2. Review the command-specific documentation linked above
  3. Start with the simplest command that might work - you can always try another approach
  4. See Ingest Troubleshooting for common issues and solutions