algbench.db package

Submodules

algbench.db.json_serializer module

A reasonably robust JSON-serializer. The purpose is only to save data for a later analysis, not to recreate it exactly. If something cannot be converted to JSON, it is converted to str.

algbench.db.json_serializer.to_json(obj)

Convert the object to a JSON-serializable object.

algbench.db.json_serializer.to_json_str(obj)

algbench.db.nfs_json_dict module

class algbench.db.nfs_json_dict.NfsJsonDict(path)

Bases: object

__init__(path) → None

load()

get(*args, **kwargs)

update(*args, **kwargs)

items()

compress()

clear()

delete()

set_new_directory(new_path: str)

NOT THREAD-SAFE! Does not check the new path, just silently continues working in the new given directory.

algbench.db.nfs_json_list module

class algbench.db.nfs_json_list.NfsJsonList(path: str | Path, file_split_mb: float = 30)

Bases: object

A simple database to dump data (dictionaries) into. Should be reasonably threadsafe even for slurm pools with NFS.

__init__(path: str | Path, file_split_mb: float = 30)

compress(compression=14, compresslevel=None)

Warning: This may not be threadsafe! If you want to extract all data to a single file, just use ‘read’ and dump the output into a single json.

extend(entries: List, flush=True)

append(entry, flush=True)

flush()

iter_cache()

iter_compressed()

Iterate over all entries in the compressed database. This may not represent the whole database if the database is not completely compressed. Use the __iter__ method to iterate over the whole database.

iter_uncompressed()

load() → List

clear()

Clear database (cache and disk). Note that remaining data in the cache of other nodes may still be written.

delete()

set_new_directory(new_path: str)

Not thread safe. Does not check the new path and does not move any folders on its own. It is expected that this step has already been performed.

algbench.db.nfs_json_set module

class algbench.db.nfs_json_set.NfsJsonSet(path)

Bases: object

__init__(path) → None

load()

add(item)

update(items)

compress()

WARNING: This operation is not thread-safe!

clear()

delete()

set_new_directory(new_path: str)

NOT THREAD-SAFE. Does not check the new path, just silently continues working in the new given directory.

Module contents

This module provides some datastructures to save JSON-data in an NFS.

class algbench.db.NfsJsonList(path: str | Path, file_split_mb: float = 30)

Bases: object

A simple database to dump data (dictionaries) into. Should be reasonably threadsafe even for slurm pools with NFS.

__init__(path: str | Path, file_split_mb: float = 30)

compress(compression=14, compresslevel=None)

Warning: This may not be threadsafe! If you want to extract all data to a single file, just use ‘read’ and dump the output into a single json.

extend(entries: List, flush=True)

append(entry, flush=True)

flush()

iter_cache()

iter_compressed()

Iterate over all entries in the compressed database. This may not represent the whole database if the database is not completely compressed. Use the __iter__ method to iterate over the whole database.

iter_uncompressed()

load() → List

clear()

Clear database (cache and disk). Note that remaining data in the cache of other nodes may still be written.

delete()

set_new_directory(new_path: str)

Not thread safe. Does not check the new path and does not move any folders on its own. It is expected that this step has already been performed.

class algbench.db.NfsJsonDict(path)

Bases: object

__init__(path) → None

load()

get(*args, **kwargs)

update(*args, **kwargs)

items()

compress()

clear()

delete()

set_new_directory(new_path: str)

NOT THREAD-SAFE! Does not check the new path, just silently continues working in the new given directory.

class algbench.db.NfsJsonSet(path)

Bases: object

__init__(path) → None

load()

add(item)

update(items)

compress()

WARNING: This operation is not thread-safe!

clear()

delete()

set_new_directory(new_path: str)

NOT THREAD-SAFE. Does not check the new path, just silently continues working in the new given directory.