Web Scraping, Content Analysis, and Alerts: Data Extraction & Sentiment Insights

This workflow automates web content monitoring by scraping a website, extracting data, analyzing sentiment and categories, and logging results. It runs every 4 hours, fetching content, validating it, and processing titles and URLs. Key nodes include Cron, Set, Wait, HTTP Request, HTML Extract, Code, Condition, Postgres, and Slack, enabling periodic data extraction and notifications without external APIs.\n\nTo set up, download n8n from n8n.io for self-hosting or sign up at cloud.n8n.io for cloud use. No external API credentials are needed for web scraping, but configure Postgres and Slack credentials in n8n. In n8n, go to Credentials, add Postgres with database connection details and Slack with a bot token from Slack’s App management. Import the workflow JSON via the Workflows menu. In the Set node, update the 'url' to your target website and adjust 'containerSelector', 'titleSelector', and 'urlSelector' to match the site’s HTML structure (e.g., 'h2' for containers, 'a' for links). Use browser developer tools to inspect elements and find correct CSS selectors. Ensure the Postgres node targets the 'web_intelligence_content' table with matching columns. Configure the Slack node to post to a valid channel (e.g., #content-intelligence).\n\nTest by manually running the workflow in n8n. Verify the HTTP Request node fetches the website content, HTML Extract nodes parse titles and URLs, and Code nodes filter duplicates and analyze sentiment. Check Postgres for logged data and Slack for notifications. Common errors include invalid CSS selectors (use browser tools to correct), HTTP timeouts (increase 'timeout' in Set node), or missing Postgres credentials (add in n8n). If no data is extracted, ensure the website’s structure matches selectors. For deployment, activate the Cron node for 4-hourly runs, monitor Postgres logs for data consistency, and validate Slack notifications for accuracy. Regularly check for errors like empty content (adjust selectors) or duplicate filtering issues (clear staticData in n8n). Schedule reviews to ensure reliable scraping and accurate sentiment analysis.