Rule Files

Rule files provide a way to define complex import and export rules in a reusable YAML format. They are especially useful when command-line arguments become unwieldy or when you need multiple sets of rules with first-match-wins logic.

When to Use Rule Files

Rule files are ideal for:

• Complex rule sets - When command-line options become too long or difficult to manage
• Reusable configurations - Save and reuse rule sets across multiple import/export jobs
• Multiple rule priorities - Define multiple rules where the first matching rule is applied
• Depth-based logic - Apply different rules based on directory depth or file location
• Team sharing - Share standardized import/export configurations with your team
• Version control - Track changes to import/export logic over time

Basic Structure

Rule files are YAML documents containing a list of rule objects. Each rule is evaluated in order, and the first rule that matches a file is applied.

Import Rule File

1
2
3
4
5
6
7
8
9

- level: acquisition
  include:
    - 'ext=dcm'
  exclude:
    - 'name=~*temp*'
  mappings:
    - 'path={sub}/{ses}/{acq}/*'
    - 'subject.label={PatientID}'
  type: dicom

Export Rule File

1
2
3
4
5
6
7

- level: acquisition
  include:
    - 'file.type=nifti'
  exclude:
    - 'session.label=~test'
  path: '{project.label}/{subject.label}/{session.label}/{file.name}'
  unzip: true

Rule Structure

Each rule in a rule file is a YAML object containing:

• level (required) - Container level: project, subject, session, or acquisition
• Filter fields (include, exclude) - Control which files match this rule
• Configuration fields - Vary between import and export rules

Import Rule Structure

Import rules support the following field groups:

• Filters: include, exclude
• Mappings: mappings, defaults, overrides
• Processing: type, zip, zip_single, group_by
• DICOM-specific: dicom_instance_name, dicom_group_by, dicom_split_localizer

For detailed field descriptions and examples, see Bulk Import - Customizing Import Rules.

Export Rule Structure

Export rules support the following field groups:

• Filters: include, exclude
• Path template: path
• Extraction: unzip, unzip_path
• Metadata: metadata

For detailed field descriptions and examples, see Bulk Export Settings

Using Rule Files

Import with Rule File

1
2
3
4
5

# Using a rule file for import
flyw import run \
    --project "fw://group/project" \
    --storage storage-id \
    --rules-file /path/to/import-rules.yaml

Export with Rule File

1
2
3
4
5

# Using a rule file for export
flyw export run \
    --project "fw://group/project" \
    --storage storage-id \
    --rules-file /path/to/export-rules.yaml

Rule Evaluation Order

Rules are evaluated in the order they appear in the file. The first rule that matches a file is applied. This allows you to handle different file types or directory structures with different logic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Rule 1: Handle DICOM files at depth 4
- level: acquisition
  include:
    - 'depth=4'
    - 'ext=dcm'
  mappings:
    - 'path={sub}/{ses}/{acq}/*'
  type: dicom

# Rule 2: Handle NIfTI files at depth 3
- level: acquisition
  include:
    - 'depth=3'
    - 'ext=nii'
  mappings:
    - 'path={sub}/{ses}/*'

# Rule 3: Catch-all for other files
- level: acquisition
  include:
    - 'depth<=5'

Real-World Examples

Example 1: Multi-Depth Research Data Import

Complex import rules handling different directory structures based on depth:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

# Project-level documentation files
- level: project
  exclude:
    - 'path=~.DS_Store'
    - 'path=~**/backup/*'
  include:
    - 'depth=1'
    - 'name=~README*'
    - 'name=~*.pdf'
  mappings:
    - 'path=[*/]{file}'

# Deep hierarchy: subject/session/acquisition structure (depth 4)
- level: acquisition
  exclude:
    - 'path=~.DS_Store'
    - 'path=~**/backup/*'
    - 'depth=1'
  include:
    - 'depth=4'
    - 'ext=dcm'
  mappings:
    - 'path={sub}/{ses}/{acq}/*'
    - 'subject.label={PatientID}'
    - 'session.label={StudyDate}'
  type: dicom
  defaults:
    - 'acquisition.info.import_rule=depth4_dicom'

# Shallow hierarchy: subject/acquisition structure (depth 3)
- level: acquisition
  exclude:
    - 'path=~.DS_Store'
    - 'path=~**/backup/*'
    - 'depth=1'
  include:
    - 'depth=3'
  mappings:
    - 'path={sub}/{acq}/*'
  defaults:
    - 'session.label=baseline'
    - 'acquisition.info.import_rule=depth3_baseline'

Example 2: Structured Data Export

Export data with separate rules for different file types and metadata:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# Project documentation files
- level: project
  include:
    - 'file.name=~README*'
    - 'file.name=~CHANGES*'
    - 'file.name=~*.pdf'
    - 'file.info.documentation=true'
  exclude:
    - 'file.info.exclude=true'
  path: '{project.label}/docs/{file.name}'

# Processed imaging data (NIfTI and metadata)
- level: acquisition
  include:
    - 'file.type=nifti'
    - 'file.name=~*.json'
    - 'file.info.processed=true'
  exclude:
    - 'file.type=dicom'
    - 'session.label=~test*'
  path: '{project.label}/data/{subject.label}/{session.label}/{acquisition.label}/{file.name}'
  metadata: true

# Raw DICOM source data (keep zipped)
- level: acquisition
  include:
    - 'file.type=dicom'
    - 'file.name=~*.dicom.zip'
  exclude:
    - 'file.info.exclude=true'
    - 'session.label=~test*'
  path: '{project.label}/sourcedata/{subject.label}/{session.label}/{acquisition.label}/{file.name}'
  unzip: false

Example 3: Multi-Site Clinical Study

Different import logic based on directory structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# Site A: DICOM organized by StudyDate
- level: acquisition
  include:
    - 'path=~SiteA/*'
    - 'ext=dcm'
  mappings:
    - 'path=SiteA/{sub}/{StudyDate}/{acq}/*'
    - 'subject.label={PatientID}'
    - 'session.label={StudyDate}'
  type: dicom
  defaults:
    - 'subject.cohort=SiteA'

# Site B: NIfTI organized by subject/timepoint
- level: acquisition
  include:
    - 'path=~SiteB/*'
    - 'ext=nii'
  mappings:
    - 'path=SiteB/{sub}/{ses}/{acq}/*'
  defaults:
    - 'subject.cohort=SiteB'

# Site C: Mixed data with metadata JSON
- level: acquisition
  include:
    - 'path=~SiteC/*'
  exclude:
    - 'name=~*metadata.json'
  mappings:
    - 'path=SiteC/{sub}/{ses}/{acq}/*'
  defaults:
    - 'subject.cohort=SiteC'

Best Practices

Rule Organization

1. Order by specificity - Place more specific rules before general ones
2. Use comments - Document the purpose of each rule
3. Group related rules - Keep rules for similar data types together
4. Test incrementally - Test each rule addition with a small dataset

Performance Optimization

1. Minimize rules - Combine similar rules when possible
2. Efficient filters - Use exclude filters to skip files early
3. Depth-based filtering - Use depth filters to quickly categorize files

Maintainability

1. Version control - Store rule files in git alongside your analysis code
2. Document assumptions - Add comments explaining data structure expectations
3. Use descriptive names - Name rule files clearly (e.g., bids-import-rules.yaml)
4. Test with dry-run - Always test rule changes with --dry-run first

Troubleshooting

Files Not Matching Any Rules

If files aren't being processed, check:

1. Rule order - Is a more general rule matching first?
2. Filter syntax - Are your include/exclude patterns correct?
3. Depth filters - Does the depth match your directory structure?
4. Field availability - Are you using fields available for import vs export?

Use flyw import test to see how a single file matches your rules:

1
2

flyw import test /path/to/file.dcm \
    --rules-file /path/to/rules.yaml

Debugging Rule Evaluation

To understand which rule matches a file:

1. Add explicit depth filters to each rule for clarity
2. Use the --dry-run flag to see what would happen without actually transferring files
3. Check the import/export report for details on how each file was processed

See Also

• Pattern Syntax Reference - Complete syntax for filters and mappings
• Filtering and Mapping Guide - Practical examples and use cases
• Pattern Quick Reference - Quick lookup for common patterns