Skip to content

CLI Command Comparison Guide

This guide helps you choose the right Flywheel CLI command for your task by comparing similar commands and their use cases.

Import vs. Upload: When to Use Each

Feature fw ingest Commands fw upload Best For
File Volume Hundreds to thousands of files Single files or small batches Ingest: Large datasets
Upload: Individual files
Automatic Organization ✅ Organizes by metadata (DICOM tags, BIDS structure, etc.) ❌ Requires manual path specification Ingest: Structured datasets
Upload: Known destinations
Metadata Extraction ✅ Automatically extracts from DICOM, BIDS, etc. ❌ No automatic extraction Ingest: Medical imaging
Upload: Any file type
De-identification ✅ Apply during import with --de-identify ❌ Not supported Ingest: DICOM data requiring de-ID
Upload: Already de-identified files
Performance Optimized for bulk operations Single file per command Ingest: 100+ files
Upload: < 10 files
Progress Tracking ✅ Detailed progress and audit logs ❌ Minimal feedback Ingest: Regulated environments
Upload: Quick uploads
Dry Run ✅ Preview with --dry-run ❌ No preview option Ingest: First-time imports
Upload: Trusted operations

Summary:

  • Use fw ingest when importing structured datasets (DICOM, BIDS) or large file collections
  • Use fw upload when adding individual files to known containers

Ingest Command Comparison

Choosing the right ingest command depends on your data structure:

Command Data Format When to Use Auto-Organization Example
ingest dicom DICOM files Raw medical imaging from scanners ✅ Yes - uses DICOM tags MRI, CT, PET scans
ingest bids BIDS dataset Neuroimaging in BIDS format ✅ Yes - uses BIDS structure fMRI studies already in BIDS
ingest folder Flywheel-exported folders Re-importing Flywheel exports ✅ Yes - uses .flywheel.json Data previously exported from Flywheel
ingest template Custom structure Non-standard folder organization ⚙️ Manual - you define mapping Custom lab folder structures
ingest project Flywheel project Copying between Flywheel sites ✅ Yes - preserves structure Multi-site collaboration

Decision Tree:

graph TD
    A[What type of data?] --> B{DICOM files?}
    B -->|Yes| C[fw ingest dicom]
    B -->|No| D{BIDS format?}
    D -->|Yes| E[fw ingest bids]
    D -->|No| F{Exported from Flywheel?}
    F -->|Yes| G[fw ingest folder]
    F -->|No| H{Copying Flywheel project?}
    H -->|Yes| I[fw ingest project]
    H -->|No| J[fw ingest template]

More Info: Which Ingest Command Should I Use?

Download vs. Export vs. Sync

All three commands download data, but serve different purposes:

Feature fw download fw export bids fw sync
Purpose Quick file retrieval BIDS-compliant export Ongoing synchronization
Output Format Original format BIDS structure Flywheel hierarchy
Incremental Updates ❌ Full re-download each time ❌ Full re-export each time ✅ Only changed files
Cloud Storage Support ❌ Local only ❌ Local only ✅ S3, GCS supported
Metadata Preservation Minimal BIDS sidecar files ✅ Full metadata in .flywheel.json
BIDS Validation N/A ✅ Maintains BIDS compliance N/A
Use Case Individual files or containers BIDS analysis tools Data archives, backups
Performance Fast for small downloads Optimized for BIDS Best for large datasets

When to Use Each:

  • fw download - Quick retrieval of specific files or containers
  • fw export bids - Exporting BIDS projects for external analysis tools (fMRIPrep, MRIQC, etc.)
  • fw sync - Creating/maintaining local copies or backups of entire projects

Examples:

1
2
3
4
5
6
7
8
# Download single file
fw download group/project/subject/session/acq/file.nii.gz

# Export BIDS project for fMRIPrep
fw export bids -p "MyStudy" ~/analysis/bids-data

# Sync project to S3 for backup
fw sync fw://group/project s3://my-bucket/backup

Copy vs. Upload vs. Ingest

Moving data to Flywheel - which command to use?

Command Source → Destination Metadata Organization Best For
fw cp Any → Any ❌ Not extracted ❌ Manual path Moving files between containers
fw upload Local → Flywheel ❌ Not extracted ❌ Manual path Adding files to existing containers
fw ingest Local → Flywheel ✅ Extracted ✅ Auto-organized Importing structured datasets

Key Differences:

  • fw cp can copy between Flywheel containers or from local to Flywheel
  • fw upload only uploads from local filesystem to Flywheel
  • fw ingest intelligently organizes and extracts metadata

Examples:

1
2
3
4
5
6
7
8
# Copy file between projects
fw cp group1/project1/file.pdf group2/project2/

# Upload file to specific container
fw upload group/project/subject/session analysis-results.csv

# Ingest DICOM study (auto-organizes by metadata)
fw ingest dicom /path/to/dicoms group project

De-identification: When and How

Multiple ways to de-identify DICOM data:

Method Command/Approach When to Use Limitations
During Ingest fw ingest dicom --de-identify --deid-profile <name> Preferred method - de-ID as data arrives None - full production use
Local Testing fw deid test <profile> <file> Testing de-ID profiles before production Local files only, not for production
Post-Import Gear Run de-ID gear via UI or gear rules Data already imported without de-ID Requires re-running classification

Recommended Workflow:

  1. Create profile: Create De-ID Profile
  2. Test locally: fw deid test my-profile sample.dcm
  3. Apply during import: fw ingest dicom --de-identify --deid-profile my-profile ...

More Info: De-identification Guide

Performance: Local vs. Cluster Ingest

Large datasets benefit from cluster ingest:

Feature Local Ingest Cluster Ingest
Data Location Local filesystem Cloud storage (S3, GCS, Azure)
Upload Speed Limited by local network Direct cloud-to-cloud transfer
Dataset Size < 100 GB > 100 GB recommended
Command fw ingest <type> /local/path ... fw ingest <type> s3://bucket/path ... --cluster
Requirements CLI installed locally Ingest cluster deployed on Flywheel
Monitoring --follow flag for progress --follow flag for progress

When to Use Cluster Ingest:

  • Dataset larger than 100 GB
  • Data already in cloud storage
  • Faster transfer speeds needed
  • Multiple simultaneous imports

Example:

1
2
3
4
5
# Local ingest
fw ingest dicom /mnt/imaging-data group project

# Cluster ingest
fw ingest dicom s3://my-bucket/imaging-data group project --cluster --follow

More Info: Cluster Ingest Guide

Filtering Data During Operations

Many commands support filtering - use the right approach:

Filter Type Commands Supporting Syntax Use Case
File Type download, sync, export bids --include <type> or --exclude <type> Only transfer specific file formats
Container Tags sync --include-container-tags '{"subject": ["tag"]}' Filter by tagged subjects/sessions
Subject/Session export bids, ingest bids --subject <label> --session <label> Process specific subjects or sessions
Data Type (BIDS) export bids --data-type func --data-type anat Export only certain BIDS modalities
ML Set sync --include-mlset Training Filter by machine learning dataset splits

Examples:

1
2
3
4
5
6
7
8
# Download only NIfTI files
fw download group/project --include nifti

# Sync only subjects tagged "cohort1"
fw sync --include-container-tags '{"subject": ["cohort1"]}' fw://group/project /backup

# Export only functional and anatomical BIDS data
fw export bids -p project /output --data-type func --data-type anat

Command Selection Quick Reference

Use this table for quick command selection:

Task Command Documentation
Check if CLI is working fw status Auth Commands
Browse available data fw ls File Management
Download a few files fw download File Management
Upload a single file fw upload File Management
Import DICOM from scanner fw ingest dicom DICOM Import
Import neuroimaging in BIDS fw ingest bids BIDS Import
Re-import Flywheel export fw ingest folder Folder Import
Import custom structure fw ingest template Template Import
Copy project between sites fw ingest project Project Copy
Export for BIDS tools fw export bids BIDS Export
Backup project to S3/GCS fw sync Sync Guide
Test de-ID profile fw deid test De-ID Guide
De-ID during import fw ingest dicom --de-identify DICOM Import

See Also

Reference Materials:

Decision Guides:

Troubleshooting: