Skip to content

Using the fw ingest project Command

Introduction

In version (14.4.0), Flywheel introduced a new way to copy an entire Flywheel project to a new location by using the CLI. The Ingest project command streamlines multi-site collaboration as well as simplifies workflows where data is curated on a pre-production Flywheel site before being moved to a production site.

The Ingest project CLI command can be used to copy data:

  • From one Flywheel site to a different Flywheel site that is on the same version
  • From one group to a different group within the same Flywheel site

Unsupported File Types

Unsupported upload method for site, group, or project de-ID profiles

Use one of these supported methods if you use site, group, or project de-ID profiles.

Limitations & Considerations

  1. Flywheel ingest project command will copy the Source data and metadata, but not:
    • Gear Job History / Provenance
    • Analyses
    • Change Log History
    • Project Permissions
    • Gear Rules
    • Session Templates
    • Data Views
  2. Empty containers in the Source will not be copied to destination
  3. When copying projects between sites, both sites must be on the same version.
  4. The file origins in the destination show that the data was uploaded by a user
  5. Only one project can be migrated per CLI command
  6. The source project is copied to destination. The source project is not modified or deleted and remains on the Source site.

Understanding Metadata Preservation

When you copy a project with fw ingest project, understanding what metadata gets preserved helps you prepare appropriately and verify the copy succeeded.

Understanding Metadata in Flywheel

Metadata That IS Copied

The following metadata is preserved when copying projects:

Container Structure and Labels:

  • Project label and description
  • Subject labels and standard fields (sex, cohort, species, strain, race, ethnicity, mlset)
  • Session labels and standard fields (timestamp, age, weight, operator, timezone)
  • Acquisition labels and timestamps
  • File names and properties (type, size, classification)

Custom Metadata:

  • All .info fields (custom key-value pairs) at every container level
  • DICOM header metadata stored in file.info.header.dicom
  • BIDS metadata and sidecar information
  • Any study-specific custom fields you've added

Tags:

  • Tags applied to subjects, sessions, acquisitions, and files
  • Note: Tags must already exist at the group level in the destination site

Notes:

  • Notes on projects, subjects, sessions, and acquisitions
  • Full note history including timestamps and authors

File Classifications:

  • File type and classification metadata
  • Intent, Measurement, Features, and Custom classifications

Metadata That IS NOT Copied

The following are not copied and must be recreated in the destination:

  • Gear Job History / Provenance
  • Analyses (results of gear runs)
  • Change Log History
  • Project Permissions (user access levels)
  • Gear Rules (automated processing workflows)
  • Session Templates (expected acquisition definitions)
  • Data Views (saved queries and reports)

Preparing for Project Copy

Before copying:

  1. Review custom metadata - Ensure important study-specific fields are populated
  2. Check tags - Verify tags exist at the destination group level (create if needed)
  3. Document gear rules - Note which gear rules need to be recreated at destination
  4. Export data views - Save any data view configurations you need to recreate

After copying:

  1. Verify metadata - Spot-check that custom fields, tags, and notes transferred correctly
  2. Recreate gear rules - Set up automated processing workflows at destination
  3. Set permissions - Configure user access for the copied project
  4. Restore data views - Recreate any saved queries or reports

Metadata and Site Versions

Since fw ingest project requires both sites to be on the same Flywheel version, metadata structures are compatible. However, if custom metadata schemas or tag conventions differ between sites, you may need to standardize them before copying.

Instructions

  1. Sign in to the source project. You must have create container and upload data permissions on the source project.

    The source project is the Flywheel site or project that contains the data you want to copy.

  2. For the site with the source project, obtain your User API Key. You will need this later on. Learn more about creating User API Keys from the Profile page.

    Screen_Shot_2018-08-29_at_11.13.53_AM.png

  3. If you are moving the project to a new site, sign in to the destination project. You must have download permissions on the destination project.

  4. In the destination project, note the group and project label, or create new ones.

  5. Open Terminal or Windows Command Prompt, and sign in to the destination with the CLI. Learn more about how to download and install the CLI.

    To see where you are signed in, enter fw status in Terminal. If you are not signed in to the destination, obtain your User API Key for the site with the destination project. Learn more about creating User API Keys from the Profile page.

  6. Enter the following command in Terminal:

    fw ingest project --group destgroup --project destproject SRC FW_WALKER_API_KEY

    • -- group and --project: Add the destination group ID and project label. For example --group betagroup --project "beta project"
    • SRC The routing string from the source group and project. For example: fw://alphagroup/alphaProject
    • FW_WALKER_API_KEY: The API Key from the source Flywheel site. For example: flywheel.io:yiLr8PHLtpe33IdYJ5
    • If you have ingest cluster deployed on your environment, add the --cluster flag to utilize the cluster's compute power. Learn more about using a cluster.

    See the Usage section below for all optional flags.

    For example:

    fw ingest project --group betagroup --project "beta Project" fw://alphagroup/alphaProject flywheel.io:yiLr8PHLtpe33IdYJ5

    Note

    If you do not specify the group and project by adding the --group and --project flags, Flywheel will automatically look for a group and project that matches the Source.

  7. Flywheel begins scanning the source project and displays the results.

    ingestProjectinProgress.png

  8. Review the summary, and enter yes.

  9. Once the scan is complete, go to the destination project to view.

    Existing containers are not modified in destination project, and any existing files are not copied over.

Usage

Required Arguments

Required Argument Description
SRC The path to the folder to import (e.g.: fw://group-name/project-name/)
FW_WALKER_API_KEY Your API key for the Source Flywheel site (e.g.: flywheel.io:yiLr8PHLtpe33IdYJ5)

Optional Arguments

Ingest General

Optional Argument Description
--compression-level The compression level to use for packfiles -1 by default. 0 for storage. A higher compression level number means more compression.
--de-identify De-identify DICOM files, e-files and p-files prior to upload. Applies custom de-identification when used with a deid-profile set in a configuration file. By default, it will apply the minimal de-identification to DICOM files:

* Remove PatientID, PatientName, and PatientBirthdate
* PatientAge convert to months.
--deid-profile NAME Use the De-identify profile by name. Use this flag if you have multiple de-ID profiles in a single config.
--detect-duplicates Flywheel scans both the source dataset and any data already in the Flywheel project. Flywheel looks for the following:

* File path conflict in the source dataset - File path in the upload dataset is not unique
* File path conflict in Flywheel - file already exists
Duplicate StudyInstanceUID in source dataset
* Duplicate StudyInstanceUID in Flywheel - UID already exists
 Duplicate SeriesInstanceUID in the source dataset
* Duplicate SeriesInstanceUID in Flywheel - UID already exists
* Duplicate SOPInstanceUID in series
--detect-duplicates-project DETECT_DUPLICATES_PROJECT Specify one or multiple project paths to use for detecting duplicates.
--copy-duplicates Upload duplicate files to a new project when using --detect-duplicates.

The project created for the duplicates will be named [target project label]_[TimestampOfIngest]. The duplicates project will have the same permissions as the original destination project. No additional project artifacts need to be copied (gear rules, data views, description, etc).
--enable-project-files Enable file uploads to a project container.
--encodings ENCODINGS Set character encoding aliases. E.g. `win_1251=cp1251``.
--exclude PATTERN Patterns of filenames to exclude (default: none). For example:

* Exclude a single file: --exclude="ReadMe.md"
* Exclude all files of a specific filetype: --exclude="*.md"
--exclude-dirs PATTERN Patterns of directories to exclude (default: none).

For example:--exclude-dirs="Sub-01" excludes all files and folder within the Sub-01 folder. This means the following directories would not be uploaded: * Sub-01/Sess01
* Sub-01/Sess2/Acq02
-g ID, --group ID The ID of the group if not in the folder structure.
--ignore-unknown-tags Ignore unknown dicom tags when parsing dicom files (default: false).
--include PATTERN [PATTERN ...] Patterns of filenames to include (default: none).

For example:
* Include a single file: --include="participants.csv"
* Include all files of a filetype: --include="*.dcm"
--include-dirs PATTERN [PATTERN ...] Patterns of directories to include (default: none).

For example:
* --include-dirs="OHM/101-10*".
The regex wildcard means that this would include the directories like:
* OHM/101-105
* OHM/101-106
* OHM/101-1011.
Note: When S3 bucket is configured as source, this flag does not support regex wildcard match, only "starts with."
--load-subject PATH Load subjects from the specified file.
--no-audit-log Skip uploading audit log to the target projects.
--require-project Proceed with the ingest process only if the resolved group and project exists (default: false).

By default, Flywheel creates a new group or project if an existing group ID or project label does not match what you designated in your command or template. This means if you mistype the group ID or project label, your data will be uploaded in the wrong location. Use this flag to make sure your data is only uploaded to existing groups and projects.
--skip-existing Looks at filenames for files already in your project, and skip import of files with the same filename. (default: false).
--symlinks Follow symbolic links that resolve to directories (default: false).

Reporting Config

Applicability

These config options are only available either when using --cluster mode with the --follow argument or when using local worker.

Optional Argument Description
--save-audit-log PATH Save audit log to the specified path on the current machine (default: none)
--save-deid-log PATH Save de-id log to the specified path on the current machine (default: none)
--save-subjects PATH Save subjects to the specified file (default: none)

Cluster Config

Applicability

These config options apply when using a cluster to ingest data.

Optional Argument Description
--cluster CLUSTER Ingest cluster url (default: none)
-f, --follow Follow the progress of the ingest (default: false)

Worker Config

Applicability

These config options are only available when using local worker (--cluster is not defined).

Optional Argument Description
--jobs JOBS Number of concurrent jobs to run (e.g. scan jobs), ignored when using cluster (default: 4)
--sleep-time SECONDS Number of seconds to wait before trying to get a task (default: 1)
--max-tempfile MAX_TEMPFILE Maximum in-memory tempfile size, in MB, or 0 to always use disk (default: 50)

General

Optional Argument Description
-h, --help Show help message and exit.
-C PATH, --config-file Specify configuration options via config file.*
--no-config Do NOT load the default configuration file.
-y, --yes Assume the answer is yes to all prompts.
--ca-certs CA_CERTS Path to a local Certificate Authority certificate bundle file. This option may be required when using a private Certificate Authority.
--timezone TIMEZONE Set the effective local timezone to use when uploading data.
-q, --quiet Squelch log messages to the console.
-d, --debug Turn on debug logging.
-v, --verbose Get more detailed output.

* Learn more about how to create this file.

Advanced Options

For large-scale or specialized ingest workflows, the following advanced features are available:

Cluster Ingest

Speed up large imports by using Flywheel's ingest cluster to process data directly from cloud storage (S3, GCS, Azure).

When to use: Copying large projects (larger than 100 GB) or when faster transfer speeds are needed.

Learn more about Cluster Ingest

Duplicate Detection

Automatically detect and handle duplicate files during ingest to avoid redundant uploads.

When to use: Copying projects that may partially overlap with existing data, or ensuring data integrity.

Learn more about Duplicate Detection

Audit Logging

Generate detailed audit logs of all ingest operations for compliance and troubleshooting.

When to use: Regulated environments requiring detailed import tracking, or when troubleshooting failed copies.

Learn more about Audit Logging

S3 Credentials Setup

Configure credentials for the ingest cluster to access your S3 buckets.

When to use: Required for cluster ingest from private S3 buckets.

Learn more about S3 Credentials

Common Errors

Common Ingest Errors

For authentication, permissions, network issues, and other errors common to all ingest commands, see the Ingest Troubleshooting Guide.

Invalid Source API Key

Cause: The source Flywheel API key is invalid, expired, or does not have access to the source project.

Solution:

  • Verify the source API key is correct
  • Generate a new API key from the source Flywheel site
  • Ensure the key has read permissions on the source project
  • Check you are logged into the destination site (not the source)

"Both sites must be on same Flywheel version"

Cause: Source and destination Flywheel instances are running different versions.

Solution:

  • Check versions: View in web UI help menu (top-right corner)
  • Contact administrators to coordinate version upgrades
  • Wait until both sites are on the same version
  • Consider manual export/import as alternative

Source Project Not Found

Cause: The project path in the fw:// URL is incorrect or the project does not exist.

Solution:

  • Verify project exists in source Flywheel instance
  • Check spelling of group ID and project label
  • Format: fw://groupid/projectlabel
  • Ensure API key has access to the project