llmtuner.infogetter.infobase

Get documents from external sources, perhaps preprocess content

Classes

InfoBase

Basic class to retrieve entry information from external source. If you pass a LLMTunerConfig, you can store results in the DB.

Module Contents

class llmtuner.infogetter.infobase.InfoBase(interfaceid='', config='', url='')[source]

Bases: llmtuner.interface.Interface

Basic class to retrieve entry information from external source. If you pass a LLMTunerConfig, you can store results in the DB. Can be initialized passing a baseurl and an identifier for the source.

response = None[source]

static get_timestamp(message='')[source]: Get an entry to add to the history of data processing

static get_mimetype(filename='')[source]: Get the mimetype from a filename

download(docids)[source]: Download files or webpages to local storage from external URL using its docid