brainscore_core.supported_data_standards.brainio.s3

S3 MODULE - AWS S3 utilities for Brain-Score

PURPOSE:

This module provides S3 file operations specifically for Brain-Score’s needs. It handles downloading model weights, assemblies, and stimulus sets from AWS S3 buckets with proper caching and integrity verification.

KEY FUNCTIONS:

  • load_file_from_s3(): Download any file from S3 to local path

  • load_weight_file(): Download model weights (backwards compatibility)

  • load_folder_from_s3(): Batch download multiple files

  • load_assembly_from_s3(): Load data assemblies with stimulus set merging

  • load_stimulus_set_from_s3(): Load stimulus sets from S3 (CSV + ZIP)

  • get_path(): Resolve S3 paths for assemblies and stimulus sets

  • download_file_if_not_exists(): Conditional download utility

  • sha1_hash(): Calculate file integrity hashes

CORE FUNCTIONALITY:

  • Downloads files from S3 with version ID support

  • Caches files locally to avoid re-downloading

  • Handles Brain-Score’s standard directory structure

  • Integrates with BotoFetcher for authenticated/unauthenticated access

  • Provides backwards compatibility for existing model loading code

This module combines S3 functionality from multiple sources: - Original brainscore_vision.model_helpers.s3 (model weights, general S3 ops) - brainscore_vision.data_helpers.s3 (assembly/stimulus set loading) All moved to core since S3 operations are domain-agnostic infrastructure that should be shared across all Brain-Score domains (vision, language, etc.).

Functions

download_file_if_not_exists(local_path, ...)

get_path(identifier, file_type, bucket, ...)

Finds path of desired file (for .csvs, .zips, and .ncs).

load_assembly_from_s3(identifier, ...[, ...])

Load a data assembly from S3, optionally within a specific folder.

load_file(bucket, relative_path, version_id)

load_file_from_s3(bucket, path, local_filepath)

Load a file from AWS S3 and validate its contents. :param bucket: The name of the S3 bucket :param path: The path of the file inside the S3 bucket :param local_filepath: The local path of the file to be saved to :param sha1: The SHA1 hash of the file. If you are not sure of this, use the sha1_hash function in this same file :param version_id: Which version of the file on S3 to use. Optional but strongly encouraged to avoid accidental overwrites. If you use Brain-Score functionality to upload files to S3, the version id will be printed to the console. You can also find this on the S3 user interface by opening the file and then clicking on the versions tab.

load_folder_from_s3(bucket, folder_path, ...)

load_stimulus_set_from_s3(identifier, ...[, ...])

load_weight_file(bucket, relative_path, ...)

Wrapper function for backwards compatibility, post large-file upload

sha1_hash(file_path)

Calculate SHA1 hash of a file.