Config

tablite.config

Classes

tablite.config.Config

Bases: object

Config class for Tablite Tables.

The default location for the storage is loaded as

Config.workdir = pathlib.Path(os.environ.get("TABLITE_TMPDIR", f"{tempfile.gettempdir()}/tablite-tmp"))

to overwrite, first import the config class, then set the new workdir.

>>> from tablite import config
>>> from pathlib import Path
>>> config.workdir = Path("/this/new/location")

the new path will now be used for every new table.

PAGE_SIZE = 1_000_000 sets the page size limit.

Multiprocessing is enabled in one of three modes: AUTO = "auto" FALSE = "sp" FORCE = "mp"

MULTIPROCESSING_MODE = AUTO is default.

SINGLE_PROCESSING_LIMIT = 1_000_000 when the number of fields (rows x columns) exceed this value, multiprocessing is used.

Attributes

tablite.config.Config.USE_NIMPORTER = os.environ.get('USE_NIMPORTER', 'false').lower() in ['1', 't', 'true', 'y', 'yes'] class-attribute instance-attribute

tablite.config.Config.ALLOW_CSV_READER_FALLTHROUGH = os.environ.get('ALLOW_CSV_READER_FALLTHROUGH', 'true').lower() in ['1', 't', 'true', 'y', 'yes'] class-attribute instance-attribute

tablite.config.Config.NIM_SUPPORTED_CONV_TYPES = ['Windows-1252', 'ISO-8859-1'] class-attribute instance-attribute

tablite.config.Config.workdir = pathlib.Path(os.environ.get('TABLITE_TMPDIR', f'{tempfile.gettempdir()}/tablite-tmp')) class-attribute instance-attribute

tablite.config.Config.pid = f'pid-{os.getpid()}' class-attribute instance-attribute

tablite.config.Config.PAGE_SIZE = 1000000 class-attribute instance-attribute

tablite.config.Config.ENCODING = 'UTF-8' class-attribute instance-attribute

tablite.config.Config.DISK_LIMIT = int(10000000000.0) class-attribute instance-attribute

10e9 (10Gb) on 100 Gb disk means raise at 90 Gb disk usage. if DISK_LIMIT <= 0, the check is turned off.

tablite.config.Config.SINGLE_PROCESSING_LIMIT = 1000000 class-attribute instance-attribute

when the number of fields (rows x columns) exceed this value, multiprocessing is used.

tablite.config.Config.vpus = max(os.cpu_count() - 1, 1) class-attribute instance-attribute

tablite.config.Config.AUTO = 'auto' class-attribute instance-attribute

tablite.config.Config.FALSE = 'sp' class-attribute instance-attribute

tablite.config.Config.FORCE = 'mp' class-attribute instance-attribute

tablite.config.Config.MULTIPROCESSING_MODE = AUTO class-attribute instance-attribute

tablite.config.Config.TQDM_DISABLE = False class-attribute instance-attribute

Functions

tablite.config.Config.reset() classmethod

Resets the config class to original values.

Source code in tablite/config.py

@classmethod
def reset(cls):
    """Resets the config class to original values."""
    for k, v in _default_values.items():
        setattr(Config, k, v)

tablite.config.Config.page_steps(length) classmethod

an iterator that yield start and end in page sizes

YIELDS	DESCRIPTION
tuple	start:int, end:int

Source code in tablite/config.py

@classmethod
def page_steps(cls, length):
    """an iterator that yield start and end in page sizes

    Yields:
        tuple: start:int, end:int
    """
    start, end = 0, 0
    for _ in range(0, length + 1, cls.PAGE_SIZE):
        start, end = end, min(end + cls.PAGE_SIZE, length)
        yield start, end
        if end == length:
            return