NAME

crawlnet — crawls the crypto web and pretrains a crypto-native LLM on it, from scratch.

SYNOPSIS

crawlnet [fees] → crawl → tokenize → pretrain → queen vn+1

DESCRIPTION

crawl

Crawlers drive real headless Chromium through the crypto web in chapters: whitepapers, docs, solana, forums, blogs, github. Every screen on this site is a real page loading right now.

ingest

Every page is canonicalized, deduplicated, relevance-gated and tokenized. Her tokenizer is trained on the crawl too.

map

The dataset is indexed by project and chapter. Chapters are finite target lists, so coverage can hit 100%.

train

When the treasury funds a run, a new queen is pretrained from random init on the dataset and nothing else, on our GPU server. Each version is bigger and has read more. Weights ship public.

spawn

Burn $CRAWL to spawn your own crawler. Its pages are credited to your wallet, and every 12-hour round it works it earns a share of the owners' pool.

sub-agent

Burn 25,000 $CRAWL under any running owned crawler. A sub-agent runs no browser: in every round its crawler brings 25+ new pages it weighs ×0.25 in the pool (a ×1 crawler weighs 1), 80% of its share to you and 20% to the crawler's owner (all of it under your own). It counts 1 in your bag split. A crawler's owner can close sub-agent sales under his crawler (/mine); the sub-agents already under it keep earning.

order

Burn 10,000 $CRAWL and the crawlers read up to 25 pages of a site you name, within 3 hours: one host, robots.txt obeyed, never past a bot check. Automatic checks come first (robots.txt, a real page, no bot check, no scam, adult or gambling content); if the site passes, burn right away. Every order gets a public report of what the queen kept.

ECONOMICS

pump.fun creator fees land in the queen's wallet. 60% pays for crawl compute (browsers + models) and pretraining runs (GPU hours). 40% goes into a pool for crawler owners that closes every 12 hours, at 00:00 and 12:00 UTC: it is split equally per crawler that brought 25+ new pages in that round, and per sub-agent, weighing ×0.25, in each round its crawler brings 25+ new pages (20% of a sub-agent's share to the crawler's owner); paid in SOL, twice a day. Every SOL is on the ledger.

EXAMPLES

# spawn your own crawler, start it on your docs
$ crawlnet spawn --name merkle-weaver --seed https://docs.solana.com

# a sub-agent under a running crawler
$ crawlnet spawn --sub --under satoshi-sniffer --name sniffer-pup

# what the crawlers have read so far
$ curl -s $CRAWL_API/v1/stats

# ask the queen
$ curl -s -X POST $CRAWL_API/v1/queen/ask -d '{"question":"what is a validator?"}'

FILES

/v1/live

websocket: crawler state, frames of visible screens, pages, ledger

/v1/web

the dataset as a graph of pages and links

/v1/treasury/ledger

every SOL in and out

/v1/orders/:id

an ordered crawl: its status and every page read

SEE ALSO

crawlers(1), queen(1), treasury(1), mint(1), order(1)