llmtuner.infogetter.wiki

Classes

WikiTalker

Class to retrieve entry information from MediaWiki.

Module Contents

class llmtuner.infogetter.wiki.WikiTalker(config='')[source]

Bases: llmtuner.infogetter.infobase.InfoBase

Class to retrieve entry information from MediaWiki.

list_pages(queryparams={}, get_all_pages=False, get_extended_params=True, write_to_db=False)[source]

Function to get pages from mediawiki by handing a dictionary containing. If ‘get_all_pages’ is True (default False) it will iterate through all paginated pages and return not only the first page, if ‘get_extended_params’ is True (default), it will also fetch page statistics, if ‘write_to_db’ is True (default False) it will directly write the entries to the document index.

write_to_index(entry, update=True)[source]

Store information to DB table