research_helpers.archive

Provides byte-reproducible archives and checksums for data deposits and submissions.

research_helpers.archive.digest(path, algorithm=DEFAULT_ALGORITHM)[source]

Return the hex digest of a file, read in blocks.

Parameters:

  • path (Path | str) – the file to hash.

  • algorithm (str, default: DEFAULT_ALGORITHM) – any name ‘hashlib.new’ accepts.

Return type:

str

Returns:

The digest, as hex.

research_helpers.archive.normalise(info, mtime=EPOCH)[source]

Strip host-specific metadata from an archive member.

Parameters:

  • info (TarInfo) – the member to normalise, modified in place.

  • mtime (int, default: EPOCH) – the timestamp to record.

Return type:

TarInfo

Returns:

The member.

research_helpers.archive.write_archive(archive, members, *, mtime=EPOCH, compress_level=DEFAULT_COMPRESS_LEVEL)[source]

Write a gzipped tar whose bytes depend only on its contents and member names.

Parameters:

  • archive (Path | str) – where to write.

  • members (Mapping[str, Path | str]) – the path each file takes inside the archive, to the file on disk.

  • mtime (int, default: EPOCH) – the timestamp recorded for every member and in the gzip header.

  • compress_level (int, default: DEFAULT_COMPRESS_LEVEL) – pinned, since it changes the bytes.

Return type:

Path

Returns:

The archive written.

research_helpers.archive.archive_directory(archive, source, *, mtime=EPOCH, compress_level=DEFAULT_COMPRESS_LEVEL)[source]

Write every file under ‘source’ to a reproducible archive, named relative to it.

Parameters:

  • archive (Path | str) – where to write.

  • source (Path | str) – the directory to pack.

  • mtime (int, default: EPOCH) – the timestamp recorded for every member and in the gzip header.

  • compress_level (int, default: DEFAULT_COMPRESS_LEVEL) – pinned, since it changes the bytes.

Return type:

Path

Returns:

The archive written.

research_helpers.archive.write_checksums(paths, destination, algorithm=DEFAULT_ALGORITHM)[source]

Write a checksum file in the format ‘shasum -c’ and ‘sha256sum -c’ read.

Names are written relative to the checksum file, which is the directory a verifier runs in:

cd dist && shasum -a 256 -c SHA256SUMS
Parameters:

  • paths (Iterable[Path | str]) – the files to record.

  • destination (Path | str) – the checksum file to write, e.g. ‘dist/SHA256SUMS’.

  • algorithm (str, default: DEFAULT_ALGORITHM) – any name ‘hashlib.new’ accepts.

Return type:

Path

Returns:

The checksum file written.