ananke.services.collection package

Submodules

ananke.services.collection.exporters module

Module containing all collection exporters.

class ananke.services.collection.exporters.AbstractCollectionExporter(configuration: ExportConfiguration_)

Bases: ABC, Generic[ExportConfiguration_]

Abstract parent class for collection exporters.

abstract export(collection: Collection, **kwargs) → None

Abstract stub for the export of a collection.

Parameters:

• collection – Collection to be exported

• kwargs – Additional exporter args

class ananke.services.collection.exporters.GraphNetCollectionExporter(configuration: ExportConfiguration_)

Bases: AbstractCollectionExporter[GraphNetExportConfiguration]

Concrete implementation for Graph Net exports.

export(collection: Collection, **kwargs) → None

Graph net export of a collection.

Parameters:

• collection – Collection to be exported

• batch_size – Events per file

• kwargs – Additional exporter args

ananke.services.collection.importers module

Module containing all logic to import collections.

class ananke.services.collection.importers.AbstractCollectionImporter(collection: Collection)

Bases: ABC

Abstract parent class for collection importers.

abstract import_data(**kwargs) → None

Abstract stub for the import of a collection.

Parameters:

kwargs – Additional importer args

Returns:

Imported collection

class ananke.services.collection.importers.LegacyCollectionKeys(value)

Bases: str, Enum

Enum containing keys for the collection file.

RECORDS = 'records'

HITS = 'hits'

SOURCES = 'sources'

DETECTOR = 'detector'

class ananke.services.collection.importers.LegacyCollection(data_path: str, complevel: int = 3, complib: str = 'lzo', read_only: bool = False)

Bases: object

Class combining all data frames to a complete record collection.

get_detector() → Optional[Detector]

Gets detector of the collection.

Returns:

Detector of the collection.

get_records(record_type: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None) → Optional[Records]

Gets records of the collection.

Parameters:

record_type – record type to include

Returns:

Records of the collection.

get_sources(record_id: int | str, interval: Optional[Interval] = None) → Optional[Sources]

Gets sources by a specific record id.

Parameters:

• record_id – Record id to get hits from

• interval – Interval to get hits in

Returns:

None if not available or sources for record id

get_hits(record_id: int | str, interval: Optional[Interval] = None) → Optional[Hits]

Gets hits by a specific record id.

Parameters:

• record_id – Record id to get hits from

• interval – Interval to get hits in

Returns:

None if not available or hits for record id

class ananke.services.collection.importers.LegacyCollectionImporter(collection: Collection)

Bases: AbstractCollectionImporter

Class to import legacy collection to current one.

import_data(import_path: Union[str, bytes, PathLike], **kwargs) → None

Imports data from legacy collection.

Parameters:

• import_path – Path to import data from

• **kwargs – additional args for data import.

ananke.services.collection.storage module

Module containing all the storage interfaces for collections.

class ananke.services.collection.storage.AbstractCollectionStorage(configuration: CollectionStorageConfiguration_)

Bases: ABC, Generic[CollectionStorageConfiguration_]

Interface for all Collection Storage interfaces.

abstract open() → AbstractCollectionStorage

Opens storage connection.

abstract close() → None

Opens storage connection.

abstract get_detector() → Optional[Detector]

Gets the detector of the current collection.

Returns:

Detector of collection or None if empty.

abstract set_detector(detector: Detector, append: bool = False) → None

Sets the detector for current storage.

Parameters:

• detector – Detector to set

• append – Append to existing detector

abstract del_detector() → None

Deletes detector of collection storage.

abstract get_records(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Optional[Records]

Gets the records of the current collection.

Parameters:

• types – Type of records to get

• record_ids – record_ids to pick

• interval – only records in interval

Returns:

Records of collection or None if empty.

abstract set_records(records: Records, append: bool = True) → None

Sets the records for current storage.

Parameters:

• records – Records to set

• append – Append to existing detector

abstract del_records(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → None

Deletes records of collection storage.

Parameters:

• types – Type of records to get

• record_ids – record_ids to pick

• interval – only records in interval

abstract get_hits(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Optional[Hits]

Gets the hits of the current collection.

Parameters:

• types – Type of hits to get

• record_ids – record_ids to pick

• interval – only hits in interval

Returns:

Hits of collection or None if empty.

abstract set_hits(hits: Hits, append: bool = True) → None

Sets the hits for current storage.

Parameters:

• hits – Hits to set

• append – Append to existing detector

abstract del_hits(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → None

Deletes hits of collection storage.

Parameters:

• types – Type of hits to get

• record_ids – record_ids to pick

• interval – only hits in interval

abstract get_sources(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Optional[Sources]

Gets the sources of the current collection.

Parameters:

• types – Type of sources to get

• record_ids – record_ids to pick

• interval – only sources in interval

Returns:

Sources of collection or None if empty.

abstract set_sources(sources: Sources, append: bool = True) → None

Sets the sources for current storage.

Parameters:

• sources – Sources to set

• append – Append to existing detector

abstract del_sources(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → None

Deletes sources of collection storage.

Parameters:

• types – Type of sources to get

• record_ids – record_ids to pick

• interval – only sources in interval

abstract get_next_record_ids(n: int = 1) → List[int]

Gets n next record ids for generation.

Parameters:

n – Number of record ids to generate

Returns:

List of integers

abstract delete() → None

Deletes current collection.

get_record_ids_with_hits(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Series

Gets all record ids with hits.

Parameters:

• types – Type of sources to get

• record_ids – record_ids to pick

• interval – only sources in interval

Returns:

Pandas Series containing all record ids

get_record_ids_with_sources(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Series

Gets all record ids with hits.

Parameters:

• types – Type of sources to get

• record_ids – record_ids to pick

• interval – only sources in interval

Returns:

Pandas Series containing all record ids

optimize()

Optimizes storage, if optimizable.

class ananke.services.collection.storage.StorageKeys(value)

Bases: Enum

Enum containing keys for the collection file.

RECORDS = 'records'

HITS = 'hits'

SOURCES = 'sources'

DETECTOR = 'detector'

class ananke.services.collection.storage.MemoryCollectionStorage(configuration: HDF5StorageConfiguration)

Bases: AbstractCollectionStorage[MemoryStorageConfiguration]

open() → MemoryCollectionStorage

Opens storage connection.

close() → None

Opens storage connection.

get_detector() → Optional[Detector]

Gets the detector of the current collection.

Returns:

Detector of collection or None if empty.

set_detector(detector: Detector, append: bool = False) → None

Sets the detector for current storage.

Parameters:

• detector – Detector to set

• append – Append to existing detector

del_detector() → None

Deletes detector of collection storage.

get_records(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Optional[Records]

Gets the records of the current collection.

Parameters:

• types – Type of records to get

• record_ids – record_ids to pick

• interval – only records in interval

Returns:

Records of collection or None if empty.

set_records(records: Records, append: bool = True) → None

Sets the records for current storage.

Parameters:

• records – Records to set

• append – Append to existing detector

del_records(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → None

Deletes records of collection storage.

Parameters:

• types – Type of records to get

• record_ids – record_ids to pick

• interval – only records in interval

get_hits(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Optional[Hits]

Gets the hits of the current collection.

Parameters:

• types – Type of hits to get

• record_ids – record_ids to pick

• interval – only hits in interval

Returns:

Hits of collection or None if empty.

set_hits(hits: Hits, append: bool = True) → None

Sets the hits for current storage.

Parameters:

• hits – Hits to set

• append – Append to existing detector

del_hits(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → None

Deletes hits of collection storage.

Parameters:

• types – Type of hits to get

• record_ids – record_ids to pick

• interval – only hits in interval

get_sources(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Optional[Sources]

Gets the sources of the current collection.

Parameters:

• types – Type of sources to get

• record_ids – record_ids to pick

• interval – only sources in interval

Returns:

Sources of collection or None if empty.

set_sources(sources: Sources, append: bool = True) → None

Sets the sources for current storage.

Parameters:

• sources – Sources to set

• append – Append to existing detector

del_sources(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → None

Deletes sources of collection storage.

Parameters:

• types – Type of sources to get

• record_ids – record_ids to pick

• interval – only sources in interval

get_next_record_ids(n: int = 1) → List[int]

Gets n next record ids for generation.

Parameters:

n – Number of record ids to generate

Returns:

List of integers

delete() → None

Deletes current collection.

get_record_ids_with_hits(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Series

Gets all record ids with hits.

Parameters:

• types – Type of sources to get

• record_ids – record_ids to pick

• interval – only sources in interval

Returns:

Pandas Series containing all record ids

get_record_ids_with_sources(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Series

Gets all record ids with hits.

Parameters:

• types – Type of sources to get

• record_ids – record_ids to pick

• interval – only sources in interval

Returns:

Pandas Series containing all record ids

class ananke.services.collection.storage.HDF5CollectionStorage(configuration: HDF5StorageConfiguration)

Bases: AbstractCollectionStorage[HDF5StorageConfiguration]

Implementation for collection storage using HDF5 file format.

property store: HDFStore

Gets the HDF5 store.

Returns:

HDF5 store object of storage.

open() → HDF5CollectionStorage

Opens the file io connection.

Returns:

Itself to work with open storage

close() → None

Closes the file io connection.

delete() → None

Deletes the H5 file.

get_detector() → Optional[Detector]

Gets the detector from HDF5.

Returns:

Detector from file

set_detector(detector: Detector, append: bool = True) → None

Sets the detector to HDF5.

Parameters:

• detector – detector to store

• append – append to current one

del_detector() → None

Deletes the detector from HDF5.

get_records(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Optional[Records]

Gets the records from HDF5.

Parameters:

• types – Types to filter by.

• record_ids – Record Ids to filter by.

• interval – Interval to filter by.

Returns:

Records matching criteria.

set_records(records: Records, append: bool = True) → None

Sets the records to HDF5.

Parameters:

• records – records to store

• append – append to current one

del_records(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → None

Deletes the records matching criteria from HDF5.

Parameters:

• types – Types to filter by.

• record_ids – Record Ids to filter by.

• interval – Interval to filter by.

get_hits(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Optional[Hits]

Gets the hits from HDF5.

Parameters:

• types – Types to filter by.

• record_ids – Record Ids to filter by.

• interval – Interval to filter by.

Returns:

Hits matching criteria.

set_hits(hits: Hits, append: bool = True) → None

Sets the hits to HDF5.

Parameters:

• hits – hits to store

• append – append to current one

del_hits(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → None

Deletes the hits matching criteria from HDF5.

Parameters:

• types – Types to filter by.

• record_ids – Record Ids to filter by.

• interval – Interval to filter by.

get_sources(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Optional[Sources]

Gets the sources from HDF5.

Parameters:

• types – Types to filter by.

• record_ids – Record Ids to filter by.

• interval – Interval to filter by.

Returns:

Sources matching criteria.

set_sources(sources: Sources, append: bool = True) → None

Sets the sources to HDF5.

Parameters:

• sources – sources to store

• append – append to current one

del_sources(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → None

Deletes the sources matching criteria from HDF5.

Parameters:

• types – Types to filter by.

• record_ids – Record Ids to filter by.

• interval – Interval to filter by.

get_next_record_ids(n: int = 1) → List[int]

Generates an auto increment based of the highest current record id.

Parameters:

n – Number of record ids to be generated

Returns:

List of integer ids

get_record_ids_with_hits(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Series

Gets all unique record ids of records with hits or empty series if empty.

Parameters:

• types – Record types to filter by

• record_ids – Record ids to filter by

• interval – timeframe to filter by

Returns:

Unique record ids or empty series of records with hits

get_record_ids_with_sources(types: Optional[Union[List[Union[EventType, NoiseType, SourceType, RecordType]], EventType, NoiseType, SourceType, RecordType]] = None, record_ids: Optional[Union[int, List[int], Series]] = None, interval: Optional[Interval] = None) → Series

Gets all unique record ids of records with sources or empty series if empty.

Parameters:

• types – Record types to filter by

• record_ids – Record ids to filter by

• interval – timeframe to filter by

Returns:

Unique record ids or empty series of records with sources

optimize() → None

Optimize HDF5 Storage by recompressing and recreating indices.

class ananke.services.collection.storage.StorageFactory

Bases: object

Class as factory for collection storages.

configuration_mapping: Dict[Type[StorageConfiguration], Type[AbstractCollectionStorage]] = {<class 'ananke.configurations.collection.HDF5StorageConfiguration'>: <class 'ananke.services.collection.storage.HDF5CollectionStorage'>}

Mapping from configuration to actual storage class

classmethod create(configuration: StorageConfiguration) → AbstractCollectionStorage

Takes a configuration for a storage and returns according storage class.

Parameters:

configuration – Configuration for current storage

Returns:

Collection storage to get and retrieve data from

Module contents

Package containing all the services related to collections.