知识#

As the knowledge base is currently the most significant user demand scenario, we natively support the construction and processing of knowledge bases. At the same time, we also provide multiple knowledge base management strategies in this project, such as pdf knowledge,md knowledge, txt knowledge, word knowledge, ppt knowledge:

We currently support many document formats: raw text, txt, pdf, md, html, doc, ppt, and url. In the future, we will continue to support more types of knowledge, including audio, video, various databases, and big data sources. Of course, we look forward to your active participation in contributing code.

创建你自己的知识库

准备

We currently support many document formats: TEXT(raw text), DOCUMENT(.txt, .pdf, .md, .doc, .ppt, .html), and URL.

开始前

pip install  db-gpt -i https://pypi.org/
python -m spacy download zh_core_web_sm
from pilot import EmbeddingEngine,KnowledgeType

2.prepare embedding model, you can download from https://huggingface.co/. Notice you have installed git-lfs.

eg: git clone https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

embedding_model = "your_embedding_model_path/all-MiniLM-L6-v2"

提前准备向量数据库环境，目前支持Chroma, Milvus and Weaviate向量数据库

#Chroma
vector_store_config = {
    "vector_store_type":"Chroma",
    "vector_store_name":"your_name",#you can define yourself
    "chroma_persist_path":"your_persist_dir"
}
#Milvus
vector_store_config = {
    "vector_store_type":"Milvus",
    "vector_store_name":"your_name",#you can define yourself
    "milvus_url":"your_url",
    "milvus_port":"your_port",
    "milvus_username":"your_username",(optional)
    "milvus_password":"your_password",(optional)
    "milvus_secure":"your_secure"(optional)
}
#Weaviate
vector_store_config = {
    "vector_store_type":"Weaviate",
    "vector_store_name":"your_name",#you can define yourself
    "weaviate_url":"your_url",
    "weaviate_port":"your_port",
    "weaviate_username":"your_username",(optional)
    "weaviate_password":"your_password",(optional)
}

初始化 Url类型 EmbeddingEngine api， 将url文档embedding向量化到向量数据库

url = "https://db-gpt.readthedocs.io/en/latest/getting_started/getting_started.html"
embedding_engine = EmbeddingEngine(
                    knowledge_source=url,
                    knowledge_type=KnowledgeType.URL.value,
                    model_name=embedding_model,
                    vector_store_config=vector_store_config)
embedding_engine.knowledge_embedding()

如果你想手动添加你自定义的source_reader和text_splitter, 请参考:

url = "https://db-gpt.readthedocs.io/en/latest/getting_started/getting_started.html"

source_reader = WebBaseLoader(web_path=self.file_path)
text_splitter = RecursiveCharacterTextSplitter(
                chunk_size=100, chunk_overlap=50
                )
embedding_engine = EmbeddingEngine(
                    knowledge_source=url,
                    knowledge_type=KnowledgeType.URL.value,
                    model_name=embedding_model,
                    vector_store_config=vector_store_config,
                    source_reader=source_reader,
                    text_splitter=text_splitter
                    )

4.init Document Type EmbeddingEngine api and embedding your document into vector store in your code. Document type can be .txt, .pdf, .md, .doc, .ppt.

document_path = "your_path/test.md"
embedding_engine = EmbeddingEngine(
                    knowledge_source=document_path,
                    knowledge_type=KnowledgeType.DOCUMENT.value,
                    model_name=embedding_model,
                    vector_store_config=vector_store_config)
embedding_engine.knowledge_embedding()

初始化TEXT类型 EmbeddingEngine api， 将文档embedding向量化到向量数据库

raw_text = "a long passage"
embedding_engine = EmbeddingEngine(
                    knowledge_source=raw_text,
                    knowledge_type=KnowledgeType.TEXT.value,
                    model_name=embedding_model,
                    vector_store_config=vector_store_config)
embedding_engine.knowledge_embedding()

在知识库进行相似性搜索

query = "please introduce the oceanbase"
topk = 5
docs = embedding_engine.similar_search(query, topk)

注意，这里默认向量模型是text2vec-large-chinese(模型比较大，如果个人电脑配置不够建议采用text2vec-base-chinese),因此确保需要将模型download下来放到models目录中。