llmtuner.llms.evaluation

Classes

Evaluator

Class to perform LLM evaluation for a given workspace.

EvalData

EvalResult

Class to hold outcome of chatter evaluation

Module Contents

class llmtuner.llms.evaluation.Evaluator(testname, workspace='', datasetname='', description='')[source]

Class to perform LLM evaluation for a given workspace.

spaceslug = ''[source]
testname[source]
description = ''[source]
results[source]
_set_chatter(spaceslug='')[source]

Set chatter for evaluation by workspace slugname.

_set_dataset(datasetname='')[source]

Set dataset for evaluation

set_description(description)[source]

Set description for the test

_write_test(testname='', description='', overwrite=False)[source]

Create a new test instance

retrieve_responses(redo=False, write_to_db=True)[source]

Create response for questions in the dataset

evaluate(methods=[], configs={}, write_to_db=True)[source]

Perform evaluation according to a given method. Can pass extra configs for each method as dictionary.

class llmtuner.llms.evaluation.EvalData(setname='', config='')[source]
datatablename = 'testdata'[source]
data = [][source]
load_from_db(setname='')[source]

Loading dataset from DB

load_from_file(filepath, setname='', write_to_db=False)[source]

Loading dataset from local file. Added directly to database if ‘write_to_db’ is True. Required columns are ‘question_id’, ‘question’ and ‘reference’.

_write_to_db(setname='', origin='')[source]

Write current data to local DB.

class llmtuner.llms.evaluation.EvalResult(testname)[source]

Class to hold outcome of chatter evaluation

testname = ''[source]