LLM Document Management: Process, Store, and Query with AI and Pinecone

This workflow automates enterprise document processing for a knowledge base using OpenAI embeddings and Pinecone. It handles document uploads, metadata extraction, text chunking, and vector storage. Key nodes include Form Trigger for document uploads, Condition for file validation, Document Loader for processing, Text Splitter for chunking, Embeddings for vector creation, Vector Store for storage, and Set for success confirmation. It ensures efficient document management with robust error handling. To set up, install n8n by downloading from n8n.io for self-hosted or sign up at cloud.n8n.io for cloud hosting. Obtain an OpenAI API key from openai.com under account settings. Get Pinecone API credentials from pinecone.io by creating an account and setting up an index. In n8n, import the workflow JSON and configure the Embeddings node with the OpenAI API key and the Vector Store node with Pinecone credentials, ensuring the index name matches (e.g., 'knowledge'). Set up the Form Trigger node by copying its webhook URL for integration with a web form or API. Configure the Condition node to validate 'File' for non-empty uploads. Ensure the Text Splitter node is set with a chunk size of 1500 and overlap of 150 for optimal embedding. Verify the Set node captures metadata like document_name and category. Test by submitting a document via the Form Trigger webhook using a tool like Postman (e.g., POST with a file and fields {Category: 'HR', Department: 'Operations'}). Check the Success Handler node for confirmation and metadata output. Validate that the document is processed and stored in Pinecone by querying the index. Monitor for errors like missing API keys (HTTP 401) or invalid files (FILE_VALIDATION_FAILED). If authentication fails, regenerate credentials in OpenAI or Pinecone dashboards. Deploy by activating the workflow in n8n’s interface. Monitor logs for issues like unsupported file formats or Pinecone rate limits, ensuring valid credentials and file inputs before live use.