Bulk Import - Manifest Files

A manifest is an optional CSV or TSV file that tells Bulk Import exactly where each source file belongs. Each row names one source file and gives the subject, session, and acquisition labels for it, along with any other container or file metadata to set.

Manifests are intended for source data whose paths and file contents do not carry enough information to place files in the Flywheel hierarchy, such as vendor-supplied, de-identified, or inconsistently organized data. Values in the manifest take precedence over values that Flywheel derives from file paths, DICOM headers, and metadata sidecar files.

Where the Manifest Goes

The manifest is not placed in the source data location. It is a local file on the computer that runs the New CLI (flyw), and it is supplied when the import starts. The CLI uploads the manifest to Flywheel and associates it with the import job, and Flywheel keeps the uploaded copy with the import for traceability. Flywheel then reads the source files from the source location, as it does for any import, and looks up each file in the manifest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

Source data location (an External Storage), read by Flywheel:

s3://example-bucket/vendor-delivery-03/
├── 1001_t1.nii
├── 1001_flair.nii
├── 1002_t1.nii
└── 1002_flair.nii

Computer running the New CLI (flyw), outside the source data location:

~/manifests/
└── vendor-delivery-03.csv

The import command names both locations:

1
2

flyw import run -p "fw://neuro/Vendor Study" -s <storage-id> \
    --manifest ~/manifests/vendor-delivery-03.csv

See How to Run a Bulk Import (CLI) for the full command.

How Flywheel Applies the Manifest

A manifest supplies metadata values. It does not decide which files are imported or the hierarchy level that each file is imported to. Import rules still make those decisions:

1. Flywheel scans the source location and matches each file against the import rules, either from the selected rule set or the default rules. A rule matches a file when the file passes the rule's include and exclude filters. Files that no rule matches are skipped, even if the manifest lists them.
2. Flywheel derives labels and metadata for the file from its path, its DICOM headers, and any metadata sidecar file, according to the matching rule.
3. Flywheel looks up the file's path in the manifest's src_path column. If a row matches, every non-empty value in that row replaces the derived value for that field.
4. Flywheel checks that the file has every label that its rule's hierarchy level requires. For example, a rule at the acquisition level requires subject, session, and acquisition labels. A file that is still missing a label fails, or is skipped if the import uses --missing-meta skip.

The following behaviors follow from this process:

• Files not listed in the manifest are imported with their derived values, as if no manifest were supplied.
• Rows for files that are not in the source location are ignored. If a row's src_path does not match any file that Flywheel finds during the scan, the row has no effect, and Flywheel does not report it.
• Empty cells are ignored, so the derived value for that field is kept.
• Duplicate src_path rows are not rejected. Flywheel uses the first matching row and ignores the rest.
• Grouped files use only the first matching row. When Flywheel groups several files into one archive, such as the files of a DICOM series, it applies a single manifest row to the whole group: the row for the first file in the group, in scan order, that the manifest lists. Rows for the group's other files are ignored, so a manifest needs only one row per DICOM series. See Example 2.

Matching src_path to Source Files

Flywheel matches each src_path value to a source file by exact comparison:

• The path is relative to the root of the source location. For an External Storage, this is the storage location, including any path prefix configured on the storage. For a local import, this is the folder passed to --storage.
• Use forward slashes (/) between folders, and do not start the path with /.
• Matching is case-sensitive.
• Wildcards and patterns are not supported. Each file needs its own row.

For example, if the storage location is s3://example-bucket/vendor-delivery-03, the file s3://example-bucket/vendor-delivery-03/scans/1001_t1.nii has the src_path value scans/1001_t1.nii.

File Format

Supported Columns

The manifest uses the same field names as import mapping rules, except that group and project fields are not allowed. Most fields also accept a shorter name.

Every column except src_path is optional, so a manifest can set only the container labels and leave everything else, including file names, to be derived as usual.

Subject Columns

Session Columns

Acquisition Columns

File Columns

Custom Information Values

Columns that end in .info.<key> set a custom information field, where <key> is the field name. For example, a file.info.scanner column sets the scanner field in the file's custom information.

Flywheel stores numbers as numbers, true and false as Boolean values, and valid JavaScript Object Notation (JSON) as structured data. Any other value is stored as text.

Classification Values

Flywheel validates classification values against the classification schema for the file's modality. When a row sets a file.classification.<key> column, also set file.modality in the same row to a modality that is configured on your site, such as MR.

Without a recognized modality, only the Custom classification key is accepted, and a file with any other classification key fails with the error Unknown modalities can only use the custom attribute. For more information, see Data Classification.

File Names

The file.name column is optional. When a row does not set it, Flywheel names the file as it would without a manifest:

• A file imported on its own keeps its source file name.
• A DICOM series archive is named after the acquisition label that the import rules derive for it. With the built-in DICOM Only rule set, that label comes from the DICOM headers, as described in Bulk Import - Default Behavior. Other rule sets can derive it from the file path or from custom mappings.

Flywheel names a DICOM series archive before it applies the manifest. If the manifest changes a series' acquisition label but not its file.name, the archive moves to the manifest's acquisition but keeps the name that the rules derived. To name the archive after the manifest's label, set file.name to the same value. Flywheel adds the .dicom.zip extension if the value does not already end with it.

Template

A manifest always has a src_path column followed by the columns to set. Choose only the columns your data needs:

1
2
3

src_path,<column>,<column>,...
<path of file 1>,<value>,<value>,...
<path of file 2>,<value>,<value>,...

The following starter template includes commonly used columns. Leave a cell empty to keep the value that Flywheel derives for that field, and delete any column you do not need.

1

src_path,subject,session,acquisition,session.timestamp,acquisition.tags,file.name,file.modality,file.classification.Intent,file.classification.Measurement,file.tags

Download the starter template

Examples

Example 1: Files with No Hierarchy in Their Paths

A vendor delivers NIfTI files in a single flat folder. File names identify the subject, but no folder structure describes subjects, sessions, or acquisitions, so path-based mapping cannot place the files.

Source location, registered as an External Storage at s3://example-bucket/vendor-delivery-03:

1
2
3
4
5

vendor-delivery-03/
├── 1001_t1.nii
├── 1001_flair.nii
├── 1002_t1.nii
└── 1002_flair.nii

Manifest file, saved as vendor-delivery-03.csv on the computer running the CLI:

1
2
3
4
5

src_path,subject,session,acquisition,file.modality,file.classification.Intent,file.classification.Measurement
1001_t1.nii,SUBJ-1001,Baseline,T1w,MR,Structural,T1
1001_flair.nii,SUBJ-1001,Baseline,FLAIR,MR,Structural,T2
1002_t1.nii,SUBJ-1002,Baseline,T1w,MR,Structural,T1
1002_flair.nii,SUBJ-1002,Baseline,FLAIR,MR,Structural,T2

Because the files sit at the root of the source location, the default rules would import them as project attachments. A rule file that imports .nii files at the acquisition level, saved as nifti-acquisitions.yaml, lets the manifest labels take effect:

1
2
3

- level: acquisition
  include:
    - 'ext=nii'

Import command:

1
2
3

flyw import run -p "fw://neuro/Vendor Study" -s <storage-id> \
    --rule-set nifti-acquisitions.yaml \
    --manifest vendor-delivery-03.csv

Result in Flywheel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

Vendor Study (project)
├── SUBJ-1001
│   └── Baseline
│       ├── T1w
│       │   └── 1001_t1.nii
│       └── FLAIR
│           └── 1001_flair.nii
└── SUBJ-1002
    └── Baseline
        ├── T1w
        │   └── 1002_t1.nii
        └── FLAIR
            └── 1002_flair.nii

Example 2: DICOM Files with Labels, Classification, and Custom Information

A local folder contains de-identified DICOM files from two series, whose headers no longer carry usable patient and study identifiers. The manifest sets the container labels, names each series archive, classifies each series, and adds custom information to the subject and to each file.

Source folder on the local computer. Files whose names start with 1- belong to one series, and files whose names start with 2- belong to another:

1
2
3
4
5
6
7

/data/study-export/
├── 1-01.dcm
├── 1-02.dcm
├── 1-03.dcm
├── 2-01.dcm
├── 2-02.dcm
└── 2-03.dcm

Manifest file, saved as study-export.csv outside the source folder:

1
2
3
4
5
6
7

src_path,subject,session,acquisition,file.name,file.modality,file.classification.Intent,file.classification.Measurement,subject.info.cohort,file.info.source_site
1-01.dcm,SUBJ-2001,Baseline,T1-post,T1-post,MR,Structural,T1,arm-a,site-a
1-02.dcm,SUBJ-2001,Baseline,T1-post,T1-post,MR,Structural,T1,arm-a,site-a
1-03.dcm,SUBJ-2001,Baseline,T1-post,T1-post,MR,Structural,T1,arm-a,site-a
2-01.dcm,SUBJ-2001,Baseline,FLAIR,FLAIR,MR,Structural,T2,arm-a,site-a
2-02.dcm,SUBJ-2001,Baseline,FLAIR,FLAIR,MR,Structural,T2,arm-a,site-a
2-03.dcm,SUBJ-2001,Baseline,FLAIR,FLAIR,MR,Structural,T2,arm-a,site-a

The columns work as follows:

• subject, session, and acquisition set the container labels.
• file.name names each series archive after its new acquisition label. Flywheel adds the .dicom.zip extension. Without this column, each archive would keep the name that the DICOM Only rule set derives from its headers. See File names.
• file.modality and the file.classification.* columns classify each series. The modality is required for the classification to be accepted. See Classification values.
• subject.info.cohort sets the cohort field in the subject's custom information.
• file.info.source_site sets the source_site field in each archive's custom information.

Flywheel imports each series as one archive and uses only the first row it finds for that series, so the rows for 1-02.dcm, 1-03.dcm, 2-02.dcm, and 2-03.dcm above are never used. The manifest can therefore list just one file from each series and produce the same result. For large deliveries, this makes the manifest much smaller and easier to prepare:

1
2
3

src_path,subject,session,acquisition,file.name,file.modality,file.classification.Intent,file.classification.Measurement,subject.info.cohort,file.info.source_site
1-01.dcm,SUBJ-2001,Baseline,T1-post,T1-post,MR,Structural,T1,arm-a,site-a
2-01.dcm,SUBJ-2001,Baseline,FLAIR,FLAIR,MR,Structural,T2,arm-a,site-a

Import command, using the built-in DICOM Only rule set (ID 000000698487153c61668b43):

1
2
3

flyw import run -p "fw://neuro/Vendor Study" -s /data/study-export \
    --rule-set 000000698487153c61668b43 \
    --manifest study-export.csv

Result in Flywheel:

1
2
3
4
5
6
7

Vendor Study (project)
└── SUBJ-2001                    subject info: cohort = arm-a
    └── Baseline
        ├── T1-post
        │   └── T1-post.dicom.zip    MR, Intent: Structural, Measurement: T1, file info: source_site = site-a
        └── FLAIR
            └── FLAIR.dicom.zip      MR, Intent: Structural, Measurement: T2, file info: source_site = site-a

Manifest Uploads and Traceability

When an import starts, the CLI uploads its manifest to Flywheel, and Flywheel associates that copy with the import job. The association is with the import, not with the user who started it:

• Rerunning the import with Rerun Job uses the same manifest, whoever reruns it, so nobody needs to find or supply the original file again.
• Editing the manifest file afterward does not change the copy that the import used. A new import with the edited file uploads and records the new version.

Validation Errors

Flywheel checks the manifest's format and header row when the manifest is uploaded, before the import starts. It does not check whether the listed files exist in the source location. If a check fails, the import does not start, and the CLI's error output contains one of the following messages:

Problems with individual rows appear as failed files in the import audit report, with one of the following messages in the reason column:

Limitations

• The manifest cannot set the destination group or project.
• src_path values must match source paths exactly. Wildcards and patterns are not supported.
• Import schedules do not accept a manifest.
• Starting an import with a manifest from the web app is not currently available. Use the New CLI (flyw). An import that already used a manifest can still be rerun from the web app with Rerun Job.

Related Pages

• How to Run a Bulk Import (CLI)
• Bulk Import - Default Behavior
• Bulk Import - Customizing Import Rules
• Bulk Import with Metadata
• Import and Export Rule Sets