LLMTuner documentation

# LLM tuner

Environment to optimize the use of Large Language Models (LLMs) for tasks in a astroparticle collaboration.

## Contents

### Folder structure

  • configs holding setting file for API keys and llm model descriptions

  • llmtuner python classes

  • tests functionality tests

### Tasks

#### Scraping of data sources

As possible data sources are available

  • Public archives accessed either via API (so far for arxiv and InspireHEP)

  • webpages scraped using Skrapy (so for only ANTARES public data page)

The information is stored as dataframes (InfoItems, InfoList) to be easily processed.

#### Creating extended LLMs

An interface to use the API of an AnythingLLM instance is provided to easily post data to the AnythingLLM workspaces.

As easy benchmark, a direct interface for simple prompting to ChatGPT is also usable.

For both interfaces, an API key must be provided in the configs/explorersetting.yml file.

## Quickstart

Download the repository and then install the package using pip ` pip install . `

### Configuration

To provide basic API keys to the various interfaces and set the data file path, edit the configs/explorersetting.yml file.

Contents:

Indices and tables