web-scraping
Everything on Ground Truth tagged “web-scraping” — 3 items.
Google now hides where search results lead behind 'goto' links. Scraping vendors say they already work around it. News
Google Search has been rewriting result links into encoded google.com/goto redirects, which Google confirmed in late August as a measure against 'evolving forms of abuse'; the change makes scraping Google results for SEO tools and AI agents slower and costlier, though major search-API vendors say they again return direct links.
Google loses its DMCA claim against a search scraper News
A federal judge dismissed both of Google's copyright anti-circumvention claims against SerpApi, ruling that a general anti-bot wall around non-copyrightable search results cannot be treated as a copyright access control.
Firecrawl Tool
A hosted API that crawls, scrapes and structures web pages into clean text for agents and retrieval pipelines, handling the JavaScript rendering and rate limiting you would otherwise build yourself.