Webeater

WebEater (weat)

WebEater is a web content extraction tool designed to fetch and process web pages. It is made for developers and researchers who need to extract structured data from web pages efficiently. The tool goes straight to the point, focusing on extracting text and structured data from web pages, while providing some additional configurations and hits for better effectiveness.

Its main purpose is to serve as a go-to-component that works out of the box for most general use cases.

As it's currently at an early stage, it may not cover all edge cases or complex scenarios. We welcome contributions and feedback to help improve its capabilities.

Main Features

• •Fetches web pages and extracts text content into Markdown format.
• •Return clean, plain text or a JSON object optionally containing lists of images and links found on the page.
• •Handles JavaScript-heavy pages using Selenium and BeautifulSoup
• •Can be used both as a library and a command-line tool (CLI).

Quick Start (CLI)

Install with pip install webeater, then run weat https://example.com to fetch and print extracted text.

CLI Options

• •url (positional): URL to fetch content from. If omitted, WebEater starts an interactive prompt.
• •-c, --config FILE (default: weat.json): Config file to use.
• •--hints FILE [FILE ...]: Additional hint files to load (space-separated paths).
• •--debug: Enable debug logging.
• •--silent: Silent mode — suppress debug/info messages; only print results or errors.
• •--json: Return content as JSON instead of plain text.
• •--content-only: Return only the main extracted content (skip extracting links and images).

Interactive mode shortcuts: j!<url> → JSON, c!<url> → content only, jc!/cj!<url> → both, q → quit. URLs must start with http:// or https://.

Quick Start (Python)

Install with pip install webeater, then use the async Webeater class:

import asyncio
from webeater import Webeater
async def main():
    weat = await Webeater.create()
    content = await weat.get(url="https://example.com")
    print(content)
asyncio.run(main())

Help and Contributions

Discussions: https://github.com/tiagrib/webeater/discussions — Issues: https://github.com/tiagrib/webeater/issues

Develop with Source

Clone https://github.com/tiagrib/webeater.git, pip install -r requirements.txt. Tested on Python 3.12.3.

Configuration

Uses a config file typically at config/weat.yaml (user agent, timeout, etc.). See hints documentation for advanced usage.