AgentOS Scope Out
Get started
AgentOS Scope Out
Get started
[ INTEGRATIONS / SCOPE OUT ]

Every data source Scope Out talks to

Scope Out wires into PropTech directories, OpenAI, and background job infrastructure — then delivers ranked intelligence to your dashboard. No third-party connectors, no configuration maze. Here is exactly what connects and why.

KERFUFFLEDirectory crawl started — pagination detected, 12 pages0.2s
OPENAIReplicability score computed — gpt-4o — 847 tokens1.1s
INNGESTdeep-scrape-job dispatched — product_id: prp_99212.4s
ROBOTS.TXTDomain vetted — crawl permitted at 1 req/s0.1s
OPENAICompetitive gap analysis — top 3 complaint themes extracted3.7s
NEONproduct_scrape_data written — 38 reviews ingested4.2s
INNGESTWeekly re-crawl cron fired — 3 monitored directories queued0.0s

DATA SOURCES / INGESTION

Where the intelligence comes from

DIRECTORY LAYER

Kerfuffle.com — and any PropTech directory you add

Paste any directory URL into the Crawl dashboard. The ingestion job walks every listing page, extracts vendor names, URLs, category tags, and descriptions, then stores them for deep analysis. Kerfuffle is the starting point — the architecture supports any HTML directory you configure.

  • Full pagination traversal — no listings missed

  • Robots.txt parsed and honoured before any request

  • Rate-limited to 1 req/sec by default, configurable per domain

  • Exponential back-off on 429 / 503 responses

  • Duplicate-run guard prevents concurrent crawls of the same URL

Crawl trigger UI — paste a directory URL and start ingestion
Background jobs — Inngest pipeline running deep scrape and scoring
SCRAPING LAYER

Deep scrape: reviews, ratings, pricing, and feature bullets

After the skim, Scope Out visits each product's own website and its Kerfuffle listing. It pulls star ratings, review text, pricing tier signals, and feature copy — storing everything in product_scrape_data linked to the discovered product. Site-specific CSS/XPath extractors handle known domains; a generic heuristic covers anything new.

  • Pricing normaliser Maps raw price signals to free / freemium / low / mid / high / enterprise tiers

  • Plugin registry Site-specific extractors for known high-value domains, fallback heuristics for everything else

  • Error & retry UI Failed scrapes (403, timeout, robots block) show on the dashboard with one-click retry

AI LAYER / OPENAI

OpenAI does the scoring. You do the deciding.

Every scraped product passes through an LLM pipeline that evaluates four dimensions and composes a ranked dossier. Token usage and cost are tracked per crawl run.

Replicability scorer

LLM evaluates data model complexity, real-time requirements, third-party integration count, and AI needs against the AgentOS stack. Returns a 1–10 score and written rationale.

Revenue potential scorer

Combines pricing tier with category market size (from the PropTech lookup table) and review-implied adoption. Outputs an estimated ARR range for a competitor entering that space.

Competitive gap scorer

Mines negative reviews exclusively. The LLM identifies the top 3 recurring complaints and maps them to opportunity themes — 'no mobile app', 'pricing too high', 'poor API'.

Market demand scorer

Combines review volume (log scale), recency, average star rating, and positive/negative sentiment ratio into a single 1–10 demand signal with a written explanation.

AI cost per crawl run is tracked and displayed on the run detail page. A configurable threshold triggers an in-app alert if a single run exceeds your budget.

PIPELINE / HOW IT MOVES

From URL paste to ranked dossier

  1. STEP 01 — INGEST

    Paste a directory URL, start the crawl

    The Kerfuffle crawler walks every listing page, extracts vendor entries, and persists them to discovered_products. Deduplication prevents re-ingesting known listings.

  2. STEP 02 — SCRAPE

    Deep scrape fires on top-signal products

    The top-N selector flags the highest-signal products for full scraping. Each gets reviews, ratings, pricing tier, and feature bullets pulled from its site and Kerfuffle listing.

  3. STEP 03 — SCORE

    OpenAI computes four dimensions

    The scoring engine passes scraped data to the LLM and receives replicability, market demand, revenue potential, and competitive gap scores. A weighted composite ranks every product.

  4. STEP 04 — DELIVER

    Dossiers appear on the ranked dashboard

    Every product gets a full dossier: mission statement, suggested feature list, weakness analysis. The dashboard re-ranks instantly. Export any dossier to Markdown or PDF with one click.

0.1s$ crawl.start({ url: 'kerfuffle.com/suppliers' })
2.3s147 listings discovered across 9 pages
2.4s12 duplicates skipped — new_products: 135
0.0s$ deep-scrape.batch({ product_ids: top20 })
4.1s18/20 scraped — 2 blocked (robots.txt)
4.2savg 31 reviews per product ingested
0.0s$ score.run({ model: 'gpt-4o', products: 18 })
6.8scomposite scores written to product_scores
6.9stop opportunity: score 8.4 / 10
0.0s$ dossier.generate({ product_ids: scored })
2.1s18 dossiers generated — dashboard updated
2.2snotification sent: crawl complete, 18 opportunities ready

BACKGROUND INFRASTRUCTURE / INNGEST

The job queue that keeps it reliable

Every crawl, scrape, scoring run, and scheduled re-crawl is an Inngest background function. That means retries, observability, and fan-out are handled without you building queue infrastructure.

  • Directory crawl job Fires on paste-URL trigger; handles pagination and deduplication

  • Deep-scrape batch One Inngest event per product; runs concurrently, capped for rate safety

  • AI scoring pipeline Chained function: scrape data in — four dimension scores + dossier out

  • Weekly re-crawl cron Monitored directories re-crawled automatically; new products flagged with badges

  • Weekly digest email Cron emails top discoveries and score movements to the allowlisted team

Background jobs dashboard — Inngest functions for crawl and scoring pipeline

DATA LAYER / NEON + DRIZZLE

Every data artefact has a home

discovered_products

Name, URL, source directory, category tags, description, and status. The root record every downstream table links to.

product_scrape_data

Raw ratings, review count, individual review text, pricing tier, feature bullets, and scraped_at timestamp. Retained for 90 days by default.

product_scores

Four dimension scores plus weighted composite. Versioned — every re-score snapshots the previous result so you can track market movement over time.

product_dossiers

Mission statement, suggested feature list, and weakness analysis stored as JSONB. Each artefact is copy-to-clipboard ready in Markdown from the dossier page.

crawl_runs + ai_usage

Every crawl run is logged with status, duration, and product counts. AI token usage and cost per run are stored separately — monthly spend is always visible.

COMPARISON VIEW / DASHBOARD

Side-by-side intelligence when two opportunities look similar

Select 2–4 products from the ranked dashboard and open a comparison table. All four dimension scores, pricing tier, category, review count, and weakness themes align in columns so you can distinguish between products with similar composite scores.

  • All scored dimensions in aligned columns

  • Pricing tier and review count side-by-side

  • Weakness themes surfaced from negative review mining

Side-by-side product comparison — dimension scores aligned in columns

OUTPUT / WHAT YOU GET

Intelligence lands ready to act on

Export formats

Export any individual product dossier as a formatted Markdown file or PDF. Export the full ranked table as a CSV for offline review. Each of the three dossier artefacts — mission statement, feature list, weakness analysis — has its own copy-to-clipboard button in Markdown.

  • Single-dossier export: Markdown or PDF

  • Full ranked table: CSV from the dashboard toolbar

  • Per-artefact copy: mission, features, weakness in Markdown

  • Paste directly into the SaaS platform builder

Notifications and digest

In-app notifications fire when a crawl completes or a shortlisted product's score shifts significantly. A weekly email digest delivers the top new discoveries and score movements to all allowlisted team members — configurable per user.

  • In-app: crawl complete with product count summary

  • In-app: shortlisted product score drop alert

  • Email: weekly top-5 new discoveries and top-3 by score

  • In-app: AI cost threshold exceeded warning

ACCESS / SECURITY

Internal-only. No self-serve signup.

Access is restricted to an email allowlist enforced at sign-in. Any attempt from an unlisted address is rejected immediately with an 'access restricted' message. There is no public sign-up flow — the root URL redirects authenticated users to the dashboard and unauthenticated users to sign-in.

Email allowlist gate

Hardcoded at the middleware level. Unlisted emails never reach the app.

No public surface

Root URL bypasses any marketing page and routes straight to sign-in. No crawlable auth pages for external actors.

Crawl compliance

Robots.txt is fetched and parsed before any domain is touched. Blocked domains are logged to the crawl run record.

FAQ / INTEGRATIONS

Common questions about how Scope Out connects

The integration stack is built. The intelligence is waiting.

48 shipped features. Kerfuffle crawling, OpenAI scoring, Inngest reliability, and Neon persistence — all wired together. Questions? Email us at sf-core-org-support-agentos-scope-out@saas-factory.ai