> ## Documentation Index
> Fetch the complete documentation index at: https://larkuprag.larkup.de/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Crawling & Data Loading

> Scrape websites, upload files, and build your knowledge corpus

LarkupRAG gives you a complete data pipeline out of the box. Crawl entire domains, upload local documents, or paste raw text then track every ingestion job in real time. Everything runs through a single **Data** tab with three views: **Add Data**, **Corpus**, and **ETL Jobs**.

## Adding Data

The **Add Data** tab is your starting point for loading content into the pipeline. You can choose from three ingestion modes:

* **Web Scrape**: Crawl websites by keyword search or direct URL. Powered by Firecrawl, with support for self-hosted and cloud modes.
* **Paste Text**: Drop in raw text or a single URL for quick ingestion.
* **Upload Files**: Drag-and-drop files directly. Supported formats include PDF, Word (`.docx`), Markdown, CSV, JSON, Excel, and plain text.

<Frame caption="Launch a self-hosted Firecrawl crawler in Docker">
  <img src="https://mintcdn.com/larkup-rag/APB5TzrzNm7Gz5bm/images/data/launch-firecrwal-server.png?fit=max&auto=format&n=APB5TzrzNm7Gz5bm&q=85&s=1eb6fcc67fee5965226a2943399550b0" alt="Launch web crawler" width="3020" height="1416" data-path="images/data/launch-firecrwal-server.png" />
</Frame>

<Frame caption="Upload files by dragging them into the UI, PDF, Word, JSON, and more">
  <img src="https://mintcdn.com/larkup-rag/APB5TzrzNm7Gz5bm/images/data/upload-files.png?fit=max&auto=format&n=APB5TzrzNm7Gz5bm&q=85&s=35bc68bbff5bc9fc84a80918822dd2dc" alt="File upload interface" width="3020" height="1416" data-path="images/data/upload-files.png" />
</Frame>

## Your Corpus

Once data is ingested, every document lands in the **Corpus**,  your central knowledge base. The corpus view shows all loaded documents with their source type (scraped, pasted, uploaded), character count, and actions to edit or delete.

<Frame caption="The Corpus table, browse, edit, or remove any document in your knowledge base">
  <img src="https://mintcdn.com/larkup-rag/tfaa78YtKcwiVAFS/images/data/corpus-table.png?fit=max&auto=format&n=tfaa78YtKcwiVAFS&q=85&s=1a243004eb71871850e7a23ecdf02ada" alt="Corpus table view" width="1810" height="869" data-path="images/data/corpus-table.png" />
</Frame>

## Tracking ETL Jobs

Large scraping and upload operations run as background **ETL Jobs**. The ETL Jobs tab gives you real-time visibility into every job, including target count, pages processed, and completion status.

<Frame caption="Monitor all ingestion jobs, see page counts, status, and completion in real time">
  <img src="https://mintcdn.com/larkup-rag/tfaa78YtKcwiVAFS/images/data/etl-jobs.png?fit=max&auto=format&n=tfaa78YtKcwiVAFS&q=85&s=9054bc18eb8d87f1cd22de479d438a2c" alt="ETL Jobs table" width="3020" height="1316" data-path="images/data/etl-jobs.png" />
</Frame>

## Scraping Configuration

LarkupRAG supports both **self-hosted** and **cloud-hosted** scraping. You can configure everything from the **Global Settings** panel, no `.env` editing required.

### Self-Hosted Firecrawl

Click **Launch Web Crawler** in the Add Data tab to spin up a local Firecrawl instance inside Docker. A bearer token is auto-generated, you can start scraping immediately with zero signup.

### Cloud API Keys

If you prefer the hosted Firecrawl or Serper.dev services, enter your API keys in Global Settings:

* **Firecrawl API Key (Cloud)**: for cloud-based web crawling
* **Serper.dev API Key**: for programmatic Google search ingestion

<Frame caption="Global Settings, configure Firecrawl cloud API key and Serper.dev key">
  <img src="https://mintcdn.com/larkup-rag/APB5TzrzNm7Gz5bm/images/data/set-firecrawl-serper-cloud-apikey.png?fit=max&auto=format&n=APB5TzrzNm7Gz5bm&q=85&s=4b1b86af388f4d5e26f579766033b3c1" alt="API key settings" width="3020" height="1416" data-path="images/data/set-firecrawl-serper-cloud-apikey.png" />
</Frame>

### Proxy Settings

For organizations that route traffic through a proxy (e.g., for rate-limit bypass or secure scraping), the settings panel exposes proxy configuration fields:

* **Proxy Server**: e.g. `http://proxy.example.com:8080`
* **Proxy Username**
* **Proxy Password**

<Frame caption="Proxy settings for secure or rate-limited scraping environments">
  <img src="https://mintcdn.com/larkup-rag/APB5TzrzNm7Gz5bm/images/data/set-firecrawl-cloud.png?fit=max&auto=format&n=APB5TzrzNm7Gz5bm&q=85&s=bfee03c9eaf49e462e2ea246bc4e4de7" alt="Proxy settings in Global Settings" width="3020" height="1416" data-path="images/data/set-firecrawl-cloud.png" />
</Frame>

LarkupRAG automatically routes all outgoing scraping requests through your configured proxy.