Blog


Any system that is working on web text data needs to be able to remove the "boilerplate" in order to get the correct
essence of the page. MContextual uses modern LLMs to generate targeting lists, so this improves the situation
a lot, but nevertheless there's many good reasons to detect and remove this text.

## What is boilerplate?

On many web pages, text in menus, headers, footers (e.g. copyright notices) exist and in large quantities. This
boilerplate text can be a real problem... what if you wanted to use contextual targeting to target restaurant
connoisseurs and you used the keyword "menu"?

Simply put, "boilerplate" means all the text on a page that _isn't_ the main content.

## Can't we just write a program to solve it?

In short, no. Webpages - especially modern ones - give authors a million ways to do display text. Furthermore,
frameworks like React make the problem even more complicated. There's really no one single way to put text on a page -
even though it _looks_ the same in terms of what is rendered.

Boilerplate removal is a famously tricky problem. The best Python library out there to solve the problem is called
trafilatura. Helpfully, the author publishes [very thorough benchmarks of its performance](https://trafilatura.readthedocs.io/en/latest/evaluation.html).

But overall, it remains a challenging problem. The very first software that automated boilerplate remove is called
Boilerpipe, and it dates to 2009, and is written in Java. At the same time there was a [benchmark dataset](http://www.lrec-conf.org/proceedings/lrec2008/pdf/162_paper.pdf) published, but the academic and open source
community haven't been able to address this hard problem really well.

What is currently implemented today is rules based, primarily. Trafilatura currently king of the hill here, having
dethroned the others.

## How does MContextual solve this problem?

Currently, we use Trafilatura due to its strong performance. That might sound like a challenge for scalability - how
are you going to scrape 100MM web pages (the approximate size of our database, as of this writing) using Python? Well,
happily many python extension written in C scale just about as well as any other compiled language. Trafilatura, under
the hood, is based on the [lxml](https://lxml.de) HTML parser (rules are applied on top of the parsed DOM) so it works
fairly well.

## Why not use ChatGPT?

It would seem like an easy decision to consider an LLM for boilerplate removal. But there's a few issues with that approach.

- Web pages are typically many kilobytes or megabytes. Big texts don't interact too well with LLMs. This is in part due
  to the size of the context window - even with modern approaches for bigger context windows, getting good evaluation
  out of long texts remains tricky. [This paper](https://arxiv.org/pdf/2401.00368.pdf) has a really cool method for
  evaluating this problem, in figure 5 (the rest of the paper is unrelated to this subject, however).
- LLMs are expensive! Even with the big decreases in costs, 100MM pages * 250k tokens (for all the HTML of a web page,
  assuming average of 4 bytes per token, which would be about 2 letters) would cost $750,000,000 to process with GPT4
  as of this writing ($0.03 CPM for prompt tokens). Even with the cheaper models (GPT3.5 turbo) you're still looking at
  a bill > $25MM. So, scale is an issue.

## What's coming up next?

One really exciting idea is to use some methods from LLM training to assist with the boilerplate problem.

An interesting paper entitled ["An unsupervised perplexity-based method for boilerplate
removal"](https://www.cambridge.org/core/journals/natural-language-engineering/article/an-unsupervised-perplexitybased-method-for-boilerplate-removal/5E589D838F1D1E0736B4F52001150339)
uses word frequency information to predict which tokens on a page are boilerplate. The core idea is that if certain
sequences of tokens are very _common_, then they are likely to be boilerplate. The authors of this paper collected
statistics about web pages to compute the required statistical distributions to make these computations. Since HTML,
though very flexible, uses a small set of "words" compared to language, this method works pretty well.

We're currently testing out this algorithm to see if it could improve the quality of the MContextual scraper.


Blog

Web crawling, how does it work?

Master contextual data now