Rule Files

Rule files provide a way to define complex import and export rules in a reusable YAML format. They are especially useful when command-line arguments become unwieldy or when you need multiple sets of rules with first-match-wins logic.

When to Use Rule Files

Rule files are ideal for:

• Complex rule sets - When command-line options become too long or difficult to manage
• Reusable configurations - Save and reuse rule sets across multiple import/export jobs
• Multiple rule priorities - Define multiple rules where the first matching rule is applied
• Depth-based logic - Apply different rules based on directory depth or file location
• Team sharing - Share standardized import/export configurations with your team
• Version control - Track changes to import/export logic over time

Basic Structure

Rule files are YAML documents containing a list of rule objects. Each rule is evaluated in order, and the first rule that matches a file is applied.

Import Rule File

1
2
3
4
5
6
7
8
9

- level: acquisition
  include:
    - 'name=~*.dcm'
  exclude:
    - 'name=~*temp*'
  mappings:
    - 'path={sub}/{ses}/{acq}/*'
    - 'subject.label={PatientID}'
  type: dicom

Export Rule File

1
2
3
4
5
6
7

- level: acquisition
  include:
    - 'file.type=nifti'
  exclude:
    - 'session.label=~test'
  path: '{project.label}/{subject.label}/{session.label}/{file.name}'
  unzip: true

Rule Structure

Each rule in a rule file is a YAML object containing:

• level (required) - Container level: project, subject, session, or acquisition
• Filter fields (include, exclude) - Control which files match this rule
• Configuration fields - Vary between import and export rules

Import Rule Structure

Import rules support the following field groups:

• Filters: include, exclude
• Mappings: mappings, defaults, overrides
• Processing: type, zip, zip_single, group_by
• DICOM-specific: dicom_instance_name, dicom_group_by, dicom_split_localizer

For detailed field descriptions and examples, see Bulk Import - Customizing Import Rules.

For job-level fields such as scan_params, conflict_strategy, fail_fast, and uid_scope, see Job-Level Fields below.

Scan Parameters

The scan_params job-level field controls how the source storage location is scanned. It is a top-level key, not part of an individual rule. For object and file storages, it supports prefixes, which limits the scan to one or more path prefixes within the source location.

This is useful when a source location contains a very large number of items (for example, millions of files in a single storage prefix). Rather than listing all items and filtering client-side, prefix filtering tells the system to scan only the specified paths, improving performance.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

scan_params:
  prefixes:
    - site-a/imaging
    - site-b/imaging

rules:
  - level: acquisition
    include:
      - 'name=~*.dcm'
    mappings:
      - 'path={sub}/{ses}/{acq}/*'
    type: dicom

Like other job-level fields, scan_params takes effect only in a managed rule set. With a local rule file, use the --scan-params-prefix option of flyw import run instead.

Export Rule Structure

Export rules support the following field groups:

• Filters: include, exclude
• Path template: path
• Extraction: unzip, unzip_path
• Metadata: metadata

For detailed field descriptions and examples, see Bulk Export Settings.

For job-level fields such as fail_fast, see Job-Level Fields below.

Job-Level Fields

In addition to the per-rule fields above, managed rule sets support a small number of job-level fields that apply to the import or export job as a whole rather than to a single rule. Job-level fields are top-level keys in the YAML document, and the list of rules goes under a top-level rules key, as in the examples below. A managed rule set always uses this form.

Import Job-Level Fields

Example import rule set with job-level fields:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

conflict_strategy: review
fail_fast: false
uid_scope: project

rules:
  - level: acquisition
    include:
      - 'name=~*.dcm'
    mappings:
      - 'path={sub}/{ses}/{acq}/*'
    type: dicom

Export Job-Level Fields

Example export rule set with job-level fields:

1
2
3
4
5
6
7

fail_fast: true

rules:
  - level: acquisition
    include:
      - 'file.type=nifti'
    path: '{project.label}/{subject.label}/{session.label}/{file.name}'

Label Templates

The label job-level field defines a template for the job's label. Instead of typing a label each time a job runs, you can define a template that Flywheel fills in automatically from the job's own fields. This helps generate consistent, descriptive labels for imported and exported data.

Templates use {field} placeholders. Commonly available job fields include:

To format a timestamp, add a strftime format string after a colon. For example, {created:%Y%m%d} renders the creation date as 20260728.

1
2
3
4
5
6
7
8
9

label: '{type}-{origin.id}-{created:%Y%m%d}'

rules:
  - level: acquisition
    include:
      - 'name=~*.dcm'
    mappings:
      - 'path={sub}/{ses}/{acq}/*'
    type: dicom

With this template, an import job started by user abc123 on 28 July 2026 is labeled import-abc123-20260728.

Using Rule Files

Import with Rule File

1
2
3
4
5

# Using a rule file for import
flyw import run \
    --project "fw://group/project" \
    --storage storage-id \
    --rules-file /path/to/import-rules.yaml

Export with Rule File

1
2
3
4
5

# Using a rule file for export
flyw export run \
    --project "fw://group/project" \
    --storage storage-id \
    --rules-file /path/to/export-rules.yaml

Rule Evaluation Order

Rules are evaluated in the order they appear in the file. The first rule that matches a file is applied. This allows you to handle different file types or directory structures with different logic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Rule 1: Handle DICOM files at depth 4
- level: acquisition
  include:
    - 'depth=4'
    - 'name=~*.dcm'
  mappings:
    - 'path={sub}/{ses}/{acq}/*'
  type: dicom

# Rule 2: Handle NIfTI files at depth 3
- level: acquisition
  include:
    - 'depth=3'
    - 'name=~*.nii'
  mappings:
    - 'path={sub}/{ses}/*'

# Rule 3: Catch-all for other files
- level: acquisition
  include:
    - 'depth<=5'

Real-World Examples

Example 1: Multi-Depth Research Data Import

Complex import rules handling different directory structures based on depth:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

# Project-level documentation files
- level: project
  exclude:
    - 'path=~.DS_Store'
    - 'path=~**/backup/*'
  include:
    - 'depth=1'
    - 'name=~README*'
    - 'name=~*.pdf'
  mappings:
    - 'path=[*/]{file}'

# Deep hierarchy: subject/session/acquisition structure (depth 4)
- level: acquisition
  exclude:
    - 'path=~.DS_Store'
    - 'path=~**/backup/*'
    - 'depth=1'
  include:
    - 'depth=4'
    - 'name=~*.dcm'
  mappings:
    - 'path={sub}/{ses}/{acq}/*'
    - 'subject.label={PatientID}'
    - 'session.label={StudyDate}'
  type: dicom
  defaults:
    - 'acquisition.info.import_rule=depth4_dicom'

# Shallow hierarchy: subject/acquisition structure (depth 3)
- level: acquisition
  exclude:
    - 'path=~.DS_Store'
    - 'path=~**/backup/*'
    - 'depth=1'
  include:
    - 'depth=3'
  mappings:
    - 'path={sub}/{acq}/*'
  defaults:
    - 'session.label=baseline'
    - 'acquisition.info.import_rule=depth3_baseline'

Example 2: Structured Data Export

Export data with separate rules for different file types and metadata:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# Project documentation files
- level: project
  include:
    - 'file.name=~README*'
    - 'file.name=~CHANGES*'
    - 'file.name=~*.pdf'
    - 'file.info.documentation=true'
  exclude:
    - 'file.info.exclude=true'
  path: '{project.label}/docs/{file.name}'

# Processed imaging data (NIfTI and metadata)
- level: acquisition
  include:
    - 'file.type=nifti'
    - 'file.name=~*.json'
    - 'file.info.processed=true'
  exclude:
    - 'file.type=dicom'
    - 'session.label=~test*'
  path: '{project.label}/data/{subject.label}/{session.label}/{acquisition.label}/{file.name}'
  metadata: true

# Raw DICOM source data (keep zipped)
- level: acquisition
  include:
    - 'file.type=dicom'
    - 'file.name=~*.dicom.zip'
  exclude:
    - 'file.info.exclude=true'
    - 'session.label=~test*'
  path: '{project.label}/sourcedata/{subject.label}/{session.label}/{acquisition.label}/{file.name}'
  unzip: false

Example 3: Multi-Site Clinical Study

Different import logic based on directory structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# Site A: DICOM organized by StudyDate
- level: acquisition
  include:
    - 'path=~SiteA/*'
    - 'name=~*.dcm'
  mappings:
    - 'path=SiteA/{sub}/{StudyDate}/{acq}/*'
    - 'subject.label={PatientID}'
    - 'session.label={StudyDate}'
  type: dicom
  defaults:
    - 'subject.cohort=SiteA'

# Site B: NIfTI organized by subject/timepoint
- level: acquisition
  include:
    - 'path=~SiteB/*'
    - 'name=~*.nii'
  mappings:
    - 'path=SiteB/{sub}/{ses}/{acq}/*'
  defaults:
    - 'subject.cohort=SiteB'

# Site C: Mixed data with metadata JSON
- level: acquisition
  include:
    - 'path=~SiteC/*'
  exclude:
    - 'name=~*metadata.json'
  mappings:
    - 'path=SiteC/{sub}/{ses}/{acq}/*'
  defaults:
    - 'subject.cohort=SiteC'

Best Practices

Rule Organization

1. Order by specificity - Place more specific rules before general ones
2. Use comments - Document the purpose of each rule
3. Group related rules - Keep rules for similar data types together
4. Test incrementally - Test each rule addition with a small dataset

Performance Optimization

1. Use prefix filtering for large sources - When a source location contains millions of items, add scan_params.prefixes to target specific paths and avoid scanning the entire listing
2. Minimize rules - Combine similar rules when possible
3. Efficient filters - Use exclude filters to skip files early
4. Depth-based filtering - Use depth filters to quickly categorize files

Maintainability

1. Version control - Store rule files in git alongside your analysis code
2. Document assumptions - Add comments explaining data structure expectations
3. Use descriptive names - Name rule files clearly (e.g., bids-import-rules.yaml)
4. Test with dry-run - Always test rule changes with --dry-run first

Troubleshooting

Files Not Matching Any Rules

If files aren't being processed, check:

1. Rule order - Is a more general rule matching first?
2. Filter syntax - Are your include/exclude patterns correct?
3. Depth filters - Does the depth match your directory structure?
4. Field availability - Are you using fields available for import vs export?

Use flyw import test to see how a single file matches your rules:

1
2

flyw import test /path/to/file.dcm \
    --rules-file /path/to/rules.yaml

Debugging Rule Evaluation

To understand which rule matches a file:

1. Add explicit depth filters to each rule for clarity
2. Use the --dry-run flag to see what would happen without actually transferring files
3. Check the import/export report for details on how each file was processed

See Also

• Pattern Syntax Reference - Complete syntax for filters and mappings
• Filtering and Mapping Guide - Practical examples and use cases
• Pattern Quick Reference - Quick lookup for common patterns