← clify home · all task guides · agent guide

Web research APIs for AI agents

Find news, papers, books, encyclopedic facts and public web data as structured JSON for an AI research workflow.

Use verified read commands to search and collect public sources without browser automation. Each result includes the source site, command parameters and a reproducible curl URL.

Verified sources

wikipedia

en.wikipedia.org

English Wikipedia: article summaries and article text. Uses /wiki/ pages and the public REST API (both allowed by robots.txt).

Machine-readable commands · 3 verified commands

hnsearch

hn.algolia.com

Hacker News search. The web app calls Algolia directly (recorded: POST .../indexes/Item_dev/query and Item_dev_sort_date); this spec uses the site's documented public API with the same parameters.

Machine-readable commands · 4 verified commands

arxiv

arxiv.org

arXiv.org, the open-access preprint archive. This CLI browses category listings (new, recent, monthly), daily catch-up, archive overviews, yearly stats, the category taxonomy and RSS feeds, and fetches paper abstract pages and BibTeX citations.

Machine-readable commands · 10 verified commands

semanticscholar

semanticscholar.org

Semantic Scholar, an AI-powered search engine for scientific literature. The CLI searches and looks up papers, authors, citations, references and recommendations through the official Semantic Scholar Academic Graph and Recommendations APIs, and also offers the web app's title autocomplete.

Machine-readable commands · 4 verified commands

news-ycombinator

news.ycombinator.com

Hacker News (news.ycombinator.com): read front page, new, past, Ask, Show and jobs listings, stories with comment trees, comment feeds, user profiles, submissions, comments and favorites, and post comments. Every command parses the server-rendered HTML.

Machine-readable commands · 13 verified commands

archive

archive.org

Internet Archive (archive.org): a digital library of texts, audio, video, software and web snapshots. This CLI searches items, including full-text search inside books, browses collections and user uploads, reads item metadata, file lists, reviews and OCR text, downloads files, and queries Wayback Machine snapshots.

Machine-readable commands · 16 verified commands

gutenberg

gutenberg.org

Project Gutenberg, a free public-domain eBook library. This CLI reads book details, lists books by author, subject, reading list or category, shows the top-downloaded and newly posted books, finds related books, and downloads plain text and RDF metadata.

Machine-readable commands · 13 verified commands

How to use this guide

Open a source page for its command list, or use its .json link as a machine-readable schema. Start with the command's example URL, then narrow results with the documented parameters.