LLM Scrape and Process Webpages with AI API

This workflow creates an AI-powered web scraper API using OpenAI to process webpage content. Key nodes include Webhook (handles POST requests at /api/ai-scraper), Input Validation (checks message input), Check Input Message (validates input), AI Agent (processes queries with OpenAI gpt-4o-mini), HTTP Request Tool (fetches webpages), Convert to Markdown (processes HTML), Success Response (returns AI response), and Error Response (handles invalid inputs). Supports full or simplified webpage content retrieval. Customize by updating the message in the POST body.\n\nSetup Requirements and Configuration: Install n8n from n8n.io for self-hosting or sign up at cloud.n8n.io. Obtain an OpenAI API key from platform.openai.com under ‘API Keys.’ Import the JSON workflow in n8n. Configure the OpenAI Chat Model node with the API key under ‘Credentials.’ Set the Webhook node to use HTTPS and path ‘api/ai-scraper’ under Settings > Webhook. Ensure your server supports HTTPS for secure webhook access. For local testing, use a tunneling service like ngrok to make the webhook publicly accessible. Verify internet connectivity for OpenAI API and webpage requests. No additional service credentials are needed.\n\nTesting and Deployment Steps: Activate the workflow to get the Webhook URL (e.g., https://your-n8n-instance/api/ai-scraper). Test with a POST request, e.g., { "message": "Summarize https://example.com", "mode": "chat" }. Success response returns JSON with the AI-generated response (e.g., webpage summary). Error response (HTTP 400) occurs for missing or empty message fields. Check n8n logs for errors like ‘Invalid OpenAI API key,’ ‘Webhook not accessible,’ or ‘HTTP request failed.’ Validate outputs: ensure ai_response contains relevant webpage content or summary, and check page_content length in the Prepare Page Content node. Debug using n8n logs for credential, network, or webhook issues. Ensure stable internet for reliable API and webhook performance. Update the message or mode (chat/full/simplified) in the POST body for different queries.