llmtuner.store

Storing information on datasets and workflows in local database

Classes

LLMTunerStore

Small DB to hold data sets and processing info for model development

Functions

map_json_type_to_sqlite(json_type)

Module Contents

llmtuner.store.map_json_type_to_sqlite(json_type)

class llmtuner.store.LLMTunerStore(filename='llmtuner.db', datapath='local', keepopen=False)

Small DB to hold data sets and processing info for model development

filename = 'llmtuner.db'

datapath = 'local'

keepopen = False

connection

cursor

schemas

_open()

Open connection to the DB

_close(confirm=False)

Set confirm = True to force closing of connection

_keep_open(keepopen=True)

Set to false if connection should be closed after each operation

static _query_create_table_from_schema(schema, table_name='')

Extract table name and properties

static _check_metadata_agreement(dictref, dictcomp)

Compare metadata of object

static format_data_to_sql(schema, data)

Reformat the entries of the dictionary ‘data’ according to the JSON schema ‘schema’ to fit into SQL tables. Arrays and objects are converted to JSON strings, and booleans are converted to integers.

Parameters:

schema (dict): The JSON schema defining the structure and types of ‘data’. data (dict): The data dictionary to be formatted.

Returns:

dict: A new dictionary with the values reformatted according to the schema.

list_tables()

Return list of available tables

get_table_metadata(table_name)

Get information for table ‘table_name’. Returns dictionary with column names, column info and number of rows.

create_table(schema, table_name='')

Creates a DB table from a json schema

delete_table(table_name)

Delete a table with ‘table_name’ from the database

add_data_to_table(table_name, data_dict, uniquekeys=[], update=False, mergekeys=[])

Add data from dictionary to a table in sqlite connection. If ‘uniquekey’ is provided, the entry is only added if not another entry with the same value for the key is present in the table. If ‘update’ is True, an existing entry is updated.

delete_data_from_table(table_name, key_values={})

Delete data from table, providing a table name and keys and values of entries to delete as dictionary

get_data_from_table(table_name, key_values={}, asdicts=False)

Getting rows from a table, if key_values = {“column1”: “value1”} is passed, only entries matching the criteria are selected. If the column contains arrays, if value is ‘select:XXX’, only rows with XXX will be selected.