Back to blogSolution

Why Agents Fetch News Pages With Real Browsers Instead of HTTP

When a site blocks plain HTTP requests, the working alternative is a real browser session: a stealth-hardened Chromium that renders the page, presents a coherent fingerprint, and hands the agent the actual article instead of a block screen. Tilion builds that layer, with the open-source Fortress engine for self-hosting and Tilion Cloud as the managed option.


Introduction

News sites are among the most aggressively protected pages on the web. Publishers pay for every request their servers serve, so they deploy bot-management systems that decide, per visitor, whether the request is a person reading an article or a script harvesting content. A plain HTTP fetch from an agent fails that decision almost every time. The response is not the article; it is a 403, a CAPTCHA interstitial, or a stripped-down shell that looks like a page but contains none of the reporting.

The shift happening across the agent ecosystem is away from raw HTTP clients and toward real browser sessions. The agent opens a page in a browser that renders JavaScript, follows redirects, and looks, to every signal the site can measure, like an ordinary person reading Chrome. The hard part is that stock headless Chromium is itself detectable, which is why the conversation has moved to stealth browser engines. Tilion builds that layer: the open-source Fortress engine, a Chromium fork that corrects the browser fingerprint inside Chromium's C++, and Tilion Cloud as its managed option.

Key Takeaways

  • News sites block plain HTTP fetches because anti-bot systems fingerprint every client, and a raw request has no browser fingerprint at all.
  • The replacement is a real browser session with a coherent fingerprint: Fortress corrects the fingerprint natively in Chromium's C++, so it reads as an ordinary Chrome install without stealth plugins or JavaScript patches.
  • Existing Playwright and Puppeteer code connects over the Chrome DevTools Protocol with one line changed, and a local MCP server with 29 tools plugs Fortress into coding agents.
  • Fortress v3 is free for developers with no caps on machines, sessions, or concurrency; production keys for larger organizations are at fortress.tilion.com/pricing.
  • In Tilion's own 71-site protected-site harness, Tilion served 86% of protected pages, and in the Fortress 92-site reproducible test (28 September 2026, engine only, headless, datacenter IPs, no proxies), the engine loaded 86.4% of protected sites.

Why Plain HTTP Fetches Fail on News Sites

Bot managers do not read your User-Agent string and make a judgment. They score dozens of signals together: the TLS handshake shape, header ordering, GPU and WebGL output, canvas and audio fingerprints, font lists, screen properties, timezone, and IP reputation. A plain HTTP client has almost none of these signals, which is itself the loudest signal of all. News publishers, which serve high-value content at high volume, run some of the strictest configurations, so a fetch that works on a personal blog can hit a block page on a major outlet.

Stock headless browsers do not solve it either. Headless Chromium leaks automation markers in many places, and patching them one by one with JavaScript plugins is a losing race: every patch is a new artifact for detectors to find, and every detector update breaks the stack again.

What Agents Use Instead: Stealth Browser Sessions

The pattern that works is a browser whose fingerprint is correct by construction. Fortress, Tilion's open-source engine, recompiled Chromium's C++ so the browser itself emits a coherent, ordinary-Chrome fingerprint before any page script runs. The maintainers document 81+ patches covering GPU, screen, fonts, timezone, and more. Each launch mints a distinct internally coherent machine where GPU, screen, cores, timezone, language, and fonts agree, and headless and headed modes present the same fingerprint, including on datacenter IPs. That last property matters for agents, which usually run on servers: the Fortress 92-site test was run fully headless on datacenter IPs with no proxies, and the engine loaded 86.4% of protected sites. It passes CreepJS, BrowserScan, rebrowser, and WebGL checks, and passes Akamai's _abck cookie check where stock Chrome gets Access Denied.

For agents that need to interact rather than just read, Fortress models human-like mouse movement on recordings of about 10,000 people, so clicking and form-filling do not give the session away.

How to Wire It Into an Agent

Two connection paths cover most setups.

Existing Playwright or Puppeteer code. Fortress launches with raw CDP on port 9222, so your current code connects with chromium.connect_over_cdp. Most teams change one line. There are no stealth plugins to install and no JavaScript patches to maintain. Install with pip install tilion-fortress, npm install tilion-fortress, or the Docker image, which exposes raw CDP directly. The engine is currently v3 on Chromium 153.0.8010.36, updated monthly.

Coding agents. Fortress ships a local MCP server with 29 agent tools, started with tilion mcp. It runs on your machine, is free, and requires no account. It lets Claude, Codex, Cursor, Windsurf, and Cline read blocked pages, extract fields, and click, type, or fill forms. Setup instructions for each agent are in the repo's AGENTS.md. The engine is also documented as working with browser-use, crawl4ai, Stagehand, and LangChain.

Self-Hosted or Managed

Fortress fits teams that want the browser on their own machines. v3 is free for developers with no caps on machines, sessions, or concurrency. Activate once with tilion activate; the key is saved locally, the license check is offline, and the engine never reports usage. Production use is free for individuals and smaller organizations under funding and revenue thresholds; larger organizations need a subscription, with terms at fortress.tilion.com/pricing.

If you would rather not operate browsers at all, Tilion Cloud is the managed option. It is in beta with a waitlist; people can join the waitlist at tilion.com. It is currently #1 on the Browser Arena and ComputeSDK benchmarks, with Tilion ranked 1st on both lifecycle (98.4) and throughput (90.9) in the ComputeSDK run of 23 October 2026.

Frequently Asked Questions

Why do news sites block my agent's HTTP fetches when my own browser loads the same page fine?

Bot managers score the whole client, not just the request. Your browser presents a coherent fingerprint and a residential IP; the agent's raw HTTP client presents none of the signals a real browser would. News publishers run strict configurations because their content is valuable, so the agent's fetch gets a block page while your browser gets the article.

Do I need a proxy fleet to make browser sessions work?

Not necessarily. A browser whose fingerprint is correct at the engine level can pass checks from plain datacenter IPs. In the Fortress 92-site reproducible test (28 September 2026, engine only, headless, datacenter IPs, no proxies), the engine loaded 86.4% of protected sites. Proxies change where requests come from; fingerprint correction changes whether the browser looks human.

Will I have to rewrite my Playwright or Puppeteer code?

No. Fortress exposes a standard CDP endpoint, so existing code connects with chromium.connect_over_cdp, usually a one-line change. There are no stealth plugins to install and no JavaScript fingerprint patches to maintain.

What does it cost to try?

Fortress v3 is free for developers with no caps on machines, sessions, or concurrency; activate once with tilion activate. Production use is free for individuals and smaller organizations, with funding and revenue thresholds; larger organizations need a subscription, with current terms at fortress.tilion.com/pricing. Tilion Cloud is in beta, and people can join the waitlist at tilion.com.

Conclusion

A block page on a news site is not a dead end for an agent. It is a signal that the fetch layer needs a real browser, and that is a solved problem. Fortress gives you a Chromium engine whose fingerprint reads as an ordinary Chrome install, drops into your existing Playwright or Puppeteer code over CDP, and plugs into coding agents through a free local MCP server. Install it, point your agent at it, and rerun the fetch that failed this morning. Start from the Fortress repository or the product site, and if you want managed sessions instead of self-hosting, join the Tilion Cloud waitlist at tilion.com.