dlt.destinations.utils

get_resource_for_adapter

def get_resource_for_adapter(data: Any) -> DltResource

View source on GitHub

Helper function for adapters. Wraps data in a DltResource if it's not a DltResource already. Alternatively if data is a DltSource, throws an error if there are multiple resource in the source or returns the single resource if available.

info_schema_null_to_bool

def info_schema_null_to_bool(v: str) -> bool

View source on GitHub

Converts INFORMATION SCHEMA truth values to Python bool

parse_db_data_type_str_with_precision

def parse_db_data_type_str_with_precision(
        db_type: str) -> Tuple[str, Optional[int], Optional[int]]

View source on GitHub

Parses a db data type with optional precision or precision and scale information

get_pipeline_state_query_columns

def get_pipeline_state_query_columns() -> TTableSchema

View source on GitHub

We get definition of pipeline state table without columns we do not need for the query

get_deterministic_temp_table_name

def get_deterministic_temp_table_name(table_name: str, op: str) -> str

View source on GitHub

Returns table name suitable for deterministic temp table. Such table will survive disconnects and is supposed to be dropped before being filled again so we need deterministic name with very low collision prob with "data" tables

sync_schema_from_storage_schema

def sync_schema_from_storage_schema(
        get_storage_tables_f: Callable[[Iterable[str]],
                                       Iterable[tuple[str,
                                                      dict[str,
                                                           TColumnSchema]]]],
        escape_col_f: Callable[[str, bool, bool], str],
        schema: Schema,
        table_names: Iterable[str] = None,
        dry_run: bool = False) -> Optional[TSchemaDrop]

View source on GitHub

Updates the dlt schema from destination.

Compare the schema we think we should have with what actually exists in the destination, and drop any tables and/or columns that disappeared.

Arguments:

• get_storage_tables_f Callable - Function that retrieves table column schemas from destination storage. Takes an iterable of table names and returns tuples of (table_name, column_schemas_dict).
• escape_col_f Callable - Function to transform dlt column names to destination format. Takes (column_name, quote, casefold) and returns the transformed column name.
• schema Schema - The dlt schema to compare against the destination.
• table_names Iterable[str], optional - Check only listed tables. Defaults to None and checks all tables.
• dry_run bool, optional - Whether to actually update the dlt schema. Defaults to False.

Returns:

• Optional[TSchemaDrop] - Returns the update that was applied (or was meant to be applied) to the schema.