Web Scraping in Convex: 10 Things to Build with the TinyFish Component

Give your Convex backend the live web. Start free with TinyFish. TinySearch and TinyFetch are free within their daily limits.

TL;DR

• TinyFish now has an official Convex component: npm install @tiny-fish/convex.
• It gives your Convex functions web search and page fetching. Pages come back as clean Markdown, HTML or JSON, including JavaScript-rendered sites.
• The interesting part is durable jobs. startFetch and startSearchAndFetch run in the background, write progress and pages into your Convex database, and your UI subscribes to them like any other query. No polling.
• Below are 10 things you can build with it, each with the call that does the work.

A quick disclosure: I work at TinyFish, which builds the web infrastructure behind this component.

What the component gives you

Web scraping in Convex used to mean writing your own action, calling a scraping API, handling retries, and inventing somewhere to keep the results. The TinyFish component packages all of that. It has four calls:

All four are read-only. They read the public web; they don't click, type or log in. For that, TinyFish has a separate Agent API, which this component doesn't wrap.

Set it up in two minutes

Install the package:

npm install @tiny-fish/convex

Register the component and pass the API key through typed component env:

// convex/convex.config.ts
import { defineApp } from "convex/server";
import { v } from "convex/values";
import tinyfish from "@tiny-fish/convex/convex.config";

const app = defineApp({
  env: { TINYFISH_API_KEY: v.string() },
});

app.use(tinyfish, {
  env: { TINYFISH_API_KEY: app.env.TINYFISH_API_KEY },
});

export default app;

Create a key at agent.tinyfish.ai/api-keys and set it on your deployment:

npx convex env set TINYFISH_API_KEY your-key
npx convex dev

The key stays in your deployment's environment. It's sent to TinyFish as an X-API-Key header and is never stored in a table or returned to clients.

One rule before the use cases: Convex components can't see ctx.auth, and exported Convex functions are public. So every example below wraps the component in your own function that checks the caller first. requireUser stands for your auth provider's check; the example app has a minimal version.

10 things to build with TinyFish in Convex

1. A research assistant that writes a sourced brief

A user types a topic. You search, read the top five results, and hand the pages to a model to write a brief with citations. With startSearchAndFetch this is one durable job, and each stored page keeps its search rank and snippet, so the sources are already in order.

export const startBrief = mutation({
  args: { topic: v.string() },
  handler: async (ctx, { topic }) => {
    const userId = await requireUser(ctx);
    const jobId = await tinyfish.startSearchAndFetch(ctx, {
      query: topic,
      maxResults: 5,
      fetchOptions: { maxCharacters: 20_000 },
      onComplete: internal.briefs.write, // call your model here
      context: { topic },
    });
    // Starting the job and recording its owner commit together.
    await ctx.db.insert("jobOwners", { jobId, userId });
    return jobId;
  },
});

Because the job lives in your database, the user can close the tab and come back to a finished brief.

2. A progress screen that updates itself

This one comes free with every job. getJob and listPages are plain Convex queries, so a React component subscribed to them re-renders as the job moves from queued to running to completed, and as each page lands.

export const job = query({
  args: { jobId: v.string() },
  handler: async (ctx, { jobId }) => {
    await requireOwnedJob(ctx, jobId); // job ids are not secrets
    return await tinyfish.getJob(ctx, jobId);
    // { status, total, completed, failed, truncated, ... }
  },
});

Show "Read 3 of 5 sources" from completed and total, and list pages as they arrive. No polling loop, no websocket code of your own.

3. Chat answers grounded in today's web

Your chatbot's model has a training cutoff. Give it a tool that searches and reads, and cap each page so it fits the context window:

export const webContext = action({
  args: { question: v.string() },
  handler: async (ctx, { question }) => {
    await requireUser(ctx);
    const { results } = await tinyfish.search(ctx, question);
    const top = results.slice(0, 3).map((r) => r.url);
    const { results: pages } = await tinyfish.fetch(ctx, top, {
      format: "markdown",
      maxCharacters: 20_000,
    });
    return pages; // pass to your model with the URLs, so it can cite them
  },
});

4. A daily news digest on a cron

search takes the same options as the TinyFish REST API, including domain_type: "news", recency_minutes and include_domains. Put it on a Convex cron and you have a digest of everything said about your company, your competitors or your market in the last day.

// convex/crons.ts
const crons = cronJobs();
crons.daily("news digest", { hourUTC: 13, minuteUTC: 0 }, internal.news.digest);
export default crons;

// convex/news.ts
export const digest = internalAction({
  handler: async (ctx) => {
    const { results } = await tinyfish.search(ctx, "Acme Corp", {
      domain_type: "news",
      recency_minutes: 60 * 24,
    });
    // summarize, then store or send
  },
});

5. A competitor pricing page watcher

Keep a list of pricing and plan pages in a table. Once a day, start a fetch job with ttl: 0 so you get a live read instead of a cached one, and compare each page's text with yesterday's copy in onComplete. When a plan or price changes, flag it.

const jobId = await tinyfish.startFetch(ctx, {
  urls: pricingPages,
  options: { ttl: 0 },
  onComplete: internal.pricing.compare,
});

6. A knowledge base built from your own docs

A support bot is only as good as what it has read. startFetch takes up to 500 URLs per job, so you can feed it your docs site, help centre and changelog, then copy the Markdown into your own table (and embed it for vector search) in onComplete. One URL that times out doesn't sink the batch: it's stored as a failed page and the rest carry on.

Jobs and pages are kept until you call deleteJob, and a page can be close to 1MB. So delete the job once you've copied what you need, at the end of onComplete or from a cleanup cron.

7. Company enrichment at signup

When someone signs up with a work email, read their company's homepage and about page, and have a model write a two-line summary for your CRM or your onboarding screen.

const domain = email.split("@")[1];
const { results, errors } = await tinyfish.fetch(ctx, [
  `https://${domain}`,
  `https://${domain}/about`,
]);

Partial results are normal here, and that's fine. A URL that can't be read comes back in errors with a code such as bot_blocked, timeout or login_required, and the other URL still comes back in results.

8. A literature tracker for research tools

search also covers research papers. Set domain_type: "research_paper" to build a reading list for a topic, then fetch the papers you want to keep. Pair it with a weekly cron and your app can tell users what's new in their field.

const { results } = await tinyfish.search(ctx, "retrieval augmented generation evaluation", {
  domain_type: "research_paper",
});

9. A jobs or listings aggregator

Careers pages, event listings and product catalogues are often rendered with JavaScript, which breaks a plain HTTP fetch. TinyFish Fetch handles JavaScript-rendered pages, so you get the content a visitor sees. Keep a table of source pages, run startFetch on a schedule, and parse the Markdown into your own listings table.

If you only need to know what changed, not the full text, pass storeContent: false to keep URLs and metadata without page content.

10. Source checking for user-submitted content

If your app publishes user posts, reviews or claims, you can check them. Search the claim, read the top results with startSearchAndFetch, and show reviewers the sources next to the post, ranked, with snippets. The final call stays with a person; the component does the legwork.

Limits and safety notes

• Read-only. The component searches and fetches. For clicking through sites, filling forms or working behind a login, use the Agent API.
• Size. Convex documents hold at most 1MB. Page text beyond about 800KB is cut and marked truncated: true. Set maxCharacters to choose a smaller cap.
• Web content is untrusted. Search snippets and page text can carry prompt injection. Extract what the task needs, and never let a model follow instructions found in a page.
• Don't let a model wander. Fetch only URLs your user asked for or your app chose, not links a model found in results without a check in your code.
• Job ids are not secrets. Record who owns each job and check it before reading, cancelling or deleting one.

What it costs

Search and Fetch are free within daily allowances: 12,000 searches and 1,000 fetched URLs a day at the time of writing. See the Search and Fetch docs for current limits. New TinyFish accounts also start with $8 in Wallet funds, no card needed, for the paid parts of the API such as browser agents.

The full README, API table and example app are on GitHub, and the package is on npm.

Ship a Convex app that reads the live web. Start free on TinyFish.