Qdrant之二 Multimodal RAG教程来源https://qdrant.tech/documentation/tutorials-build-essentials/multimodal-search/功能实现图文跨模态检索、多语言检索文搜图、图搜文基于vdr‑2b‑multi‑v1模型无需OCR提取图像文本1. 环境依赖安装pipinstallqdrant-client llama-index-embeddings-huggingface torchvision pillow启动Qdrant向量数据库Docker方式dockerrun-p6333:6333 qdrant/qdrant2. 准备数据集下载教程数据集图片全部放到代码同级目录下的images文件夹文件列表images/image‑1.png飞机应急安全images/image‑2.png飞机零部件结构图images/image‑3.png新冠防疫限制images/image‑4.pngUFO目击机密图片images/image‑5.pngAralar 2011神秘脚印3. 完整Python代码fromllama_index.embeddings.huggingfaceimportHuggingFaceEmbeddingfromqdrant_clientimportQdrantClient,modelsfromPILimportImage# ---------------------- 1. 初始化多模态Embedding模型 ----------------------# vdr‑2b‑multi‑v1支持多语言图文跨模态嵌入不需要OCRmodelHuggingFaceEmbedding(model_namellamaindex/vdr-2b-multi-v1,devicecpu,# Mac使用 mpsN卡GPU使用 cudatrust_remote_codeTrue,)# ---------------------- 2. 准备图文数据集 ----------------------documents[{caption:An image about plane emergency safety.,image:images/image-1.png},{caption:An image about airplane components.,image:images/image-2.png},{caption:An image about COVID safety restrictions.,image:images/image-3.png},{caption:An confidential image about UFO sightings.,image:images/image-4.png},{caption:An image about unusual footprints on Aralar 2011.,image:images/image-5.png},]# ---------------------- 3. 向量化文本、图像分别生成Embedding ----------------------text_captions[doc[caption]fordocindocuments]image_paths[doc[image]fordocindocuments]# 批量生成文本向量、图像向量text_embeddingsmodel.get_text_embedding_batch(text_captions)image_embeddingsmodel.get_image_embedding_batch(image_paths)# ---------------------- 4. 连接Qdrant创建集合 ----------------------clientQdrantClient(urlhttp://localhost:6333/)COLLECTION_NAMEllama-multi# 如果集合不存在则创建集合同时定义text、image两套向量ifnotclient.collection_exists(COLLECTION_NAME):client.create_collection(collection_nameCOLLECTION_NAME,vectors_config{image:models.VectorParams(sizelen(image_embeddings[0]),distancemodels.Distance.COSINE),text:models.VectorParams(sizelen(text_embeddings[0]),distancemodels.Distance.COSINE),})# ---------------------- 5. 将向量元数据payload上传Qdrant ----------------------points[]foridx,docinenumerate(documents):pointmodels.PointStruct(ididx,vector{text:text_embeddings[idx],image:image_embeddings[idx],},payloaddoc)points.append(point)client.upload_points(collection_nameCOLLECTION_NAME,pointspoints)# ---------------------- 6. 检索示例1文搜图 Text‑to‑Image ----------------------print(【文搜图英文查询】)query_text_enAdventures on snow hillsquery_embmodel.get_query_embedding(query_text_en)result_enclient.query_points(collection_nameCOLLECTION_NAME,queryquery_emb,usingimage,# 使用image向量空间检索with_payload[image],limit1)img_path_enresult_en.points[0].payload[image]print(f检索得到图片路径:{img_path_en})Image.open(img_path_en).show()# ---------------------- 7. 检索示例2图搜文 Image‑to‑Text ----------------------print(\n【图搜文输入图片检索文本caption】)query_img_pathimages/image-2.pngimg_query_embmodel.get_image_embedding(query_img_path)result_img2textclient.query_points(collection_nameCOLLECTION_NAME,queryimg_query_emb,usingtext,# 使用text向量空间检索用图像向量去匹配文本向量with_payload[caption],limit1)retrieved_captionresult_img2text.points[0].payload[caption]print(f图搜文返回的caption{retrieved_caption})注意首次运行会自动下载llamaindex/vdr‑2b‑multi‑v1模型模型体积较大需要预留足够磁盘空间。GPU环境可以显著加速嵌入计算。 Qdrant之二： Multimodal RAG
Qdrant之二 Multimodal RAG教程来源https://qdrant.tech/documentation/tutorials-build-essentials/multimodal-search/功能实现图文跨模态检索、多语言检索文搜图、图搜文基于vdr‑2b‑multi‑v1模型无需OCR提取图像文本1. 环境依赖安装pipinstallqdrant-client llama-index-embeddings-huggingface torchvision pillow启动Qdrant向量数据库Docker方式dockerrun-p6333:6333 qdrant/qdrant2. 准备数据集下载教程数据集图片全部放到代码同级目录下的images文件夹文件列表images/image‑1.png飞机应急安全images/image‑2.png飞机零部件结构图images/image‑3.png新冠防疫限制images/image‑4.pngUFO目击机密图片images/image‑5.pngAralar 2011神秘脚印3. 完整Python代码fromllama_index.embeddings.huggingfaceimportHuggingFaceEmbeddingfromqdrant_clientimportQdrantClient,modelsfromPILimportImage# ---------------------- 1. 初始化多模态Embedding模型 ----------------------# vdr‑2b‑multi‑v1支持多语言图文跨模态嵌入不需要OCRmodelHuggingFaceEmbedding(model_namellamaindex/vdr-2b-multi-v1,devicecpu,# Mac使用 mpsN卡GPU使用 cudatrust_remote_codeTrue,)# ---------------------- 2. 准备图文数据集 ----------------------documents[{caption:An image about plane emergency safety.,image:images/image-1.png},{caption:An image about airplane components.,image:images/image-2.png},{caption:An image about COVID safety restrictions.,image:images/image-3.png},{caption:An confidential image about UFO sightings.,image:images/image-4.png},{caption:An image about unusual footprints on Aralar 2011.,image:images/image-5.png},]# ---------------------- 3. 向量化文本、图像分别生成Embedding ----------------------text_captions[doc[caption]fordocindocuments]image_paths[doc[image]fordocindocuments]# 批量生成文本向量、图像向量text_embeddingsmodel.get_text_embedding_batch(text_captions)image_embeddingsmodel.get_image_embedding_batch(image_paths)# ---------------------- 4. 连接Qdrant创建集合 ----------------------clientQdrantClient(urlhttp://localhost:6333/)COLLECTION_NAMEllama-multi# 如果集合不存在则创建集合同时定义text、image两套向量ifnotclient.collection_exists(COLLECTION_NAME):client.create_collection(collection_nameCOLLECTION_NAME,vectors_config{image:models.VectorParams(sizelen(image_embeddings[0]),distancemodels.Distance.COSINE),text:models.VectorParams(sizelen(text_embeddings[0]),distancemodels.Distance.COSINE),})# ---------------------- 5. 将向量元数据payload上传Qdrant ----------------------points[]foridx,docinenumerate(documents):pointmodels.PointStruct(ididx,vector{text:text_embeddings[idx],image:image_embeddings[idx],},payloaddoc)points.append(point)client.upload_points(collection_nameCOLLECTION_NAME,pointspoints)# ---------------------- 6. 检索示例1文搜图 Text‑to‑Image ----------------------print(【文搜图英文查询】)query_text_enAdventures on snow hillsquery_embmodel.get_query_embedding(query_text_en)result_enclient.query_points(collection_nameCOLLECTION_NAME,queryquery_emb,usingimage,# 使用image向量空间检索with_payload[image],limit1)img_path_enresult_en.points[0].payload[image]print(f检索得到图片路径:{img_path_en})Image.open(img_path_en).show()# ---------------------- 7. 检索示例2图搜文 Image‑to‑Text ----------------------print(\n【图搜文输入图片检索文本caption】)query_img_pathimages/image-2.pngimg_query_embmodel.get_image_embedding(query_img_path)result_img2textclient.query_points(collection_nameCOLLECTION_NAME,queryimg_query_emb,usingtext,# 使用text向量空间检索用图像向量去匹配文本向量with_payload[caption],limit1)retrieved_captionresult_img2text.points[0].payload[caption]print(f图搜文返回的caption{retrieved_caption})注意首次运行会自动下载llamaindex/vdr‑2b‑multi‑v1模型模型体积较大需要预留足够磁盘空间。GPU环境可以显著加速嵌入计算。