brainscore_core.supported_data_standards.brainio.fetch

FETCH MODULE - File downloading and data loading for Brain-Score

PURPOSE:

This module handles downloading and loading of Brain-Score data from remote sources (primarily AWS S3). It provides the core infrastructure for fetching assemblies, stimulus sets, and model weights without requiring the BrainIO catalog system.

KEY CLASSES:

  • BotoFetcher: Downloads files from AWS S3 with progress tracking

  • Assembly/StimulusSet resolvers: Map class names to actual classes

CORE FUNCTIONALITY:

  • S3 file downloading with authentication fallback

  • ZIP file extraction for stimulus sets

  • SHA1 hash verification for data integrity

  • Local caching to avoid re-downloading files

  • Progress bars for large file transfers

CATALOG SYSTEM REMOVED:

The original BrainIO get_assembly() and get_stimulus_set() functions that relied on CSV catalog lookups are replaced with direct S3 parameter functions in the brainscore_core.supported_data_standards.brainio.s3 module.

Functions

fetch_file(location_type, location[, sha1, ...])

fullname(obj)

get_assembly(identifier)

Simplified get_assembly function - requires direct S3 parameters.

get_local_data_path()

get_stimulus_set(identifier)

Simplified get_stimulus_set function - requires direct S3 parameters.

resolve_assembly_class(class_name)

Resolve assembly class by name.

resolve_stimulus_set_class(class_name)

Resolve stimulus set class by name.

unzip(zip_path)

Unzip a file and return the directory containing the extracted files.

verify_sha1(file_path, expected_sha1)

Verify the SHA1 hash of a file.

Classes

BotoFetcher(location, local_filename[, ...])

A Fetcher that retrieves files from Amazon Web Services' S3 data storage.

Fetcher(location, local_filename)

A Fetcher obtains data with which to populate a DataAssembly.