SEO fundamentals: crawling, indexing and content quality
Search engine optimisation is the work of making pages easy for search engines to find, understand and trust, so they can show them to people looking for that information. There is no secret trick. This guide covers the stages a page goes through and what helps at each one.
Three stages: crawl, index, rank
- Crawling. A program called a crawler follows links and sitemaps to discover URLs, then downloads the pages.
- Indexing. The search engine processes each page, works out what it is about, and decides whether to store it in its index. Not every crawled page is indexed.
- Ranking. When someone searches, the engine picks the most relevant and useful indexed pages and orders them.
A page that fails an early stage never reaches the next one, so check them in order.
Stage 1: help crawlers find pages
- Link to every important page from at least one other page on your site, using normal
<a href>links. - Submit an XML sitemap listing the canonical URLs you want indexed.
- Check that
robots.txtdoes not block pages or the CSS and JavaScript they need. - Make sure pages return status 200, and moved pages return a permanent redirect (301 or 308) to the new address.
You can test a URL from the command line: curl -I example.com/page.html shows the status code and headers.
Stage 2: make pages worth indexing
Search engines skip pages that add nothing new. Common reasons a page is crawled but not indexed:
| Problem | Fix |
|---|---|
| Thin content: a few lines, or a template with one word changed | Merge it into a fuller page, or write something genuinely useful |
| Duplicate of another URL | Pick one URL and point a rel="canonical" tag at it |
A noindex tag left in by mistake | Remove it from pages that should appear |
| Main content only appears after scripts run and fails to load | Put core text in the HTML |
Stage 3: earn a good position
Answer the question fully
Start by searching for your topic and reading what people are clearly trying to find. Then write the page that answers it best: accurate, specific, well organised, and honest about limits. First-hand experience, worked examples and original explanation help a page stand out from rewrites of other pages.
Write clear titles and descriptions
The <title> should describe the page in plain words and be unique across the site. The meta description is often shown under the title in results; it does not directly affect ranking, but a clear one can earn the click.
Structure the page
Use one h1 and descriptive subheadings. Short paragraphs, lists and tables help readers scan, and help engines understand the parts of a page.
Technical health
Pages should load quickly, work well on phones, use HTTPS and avoid layout jumps while loading. These rarely win rankings on their own, but serious problems hold a good page back.
Links from others
Links from relevant, respected sites are a sign that others find the page useful. They are earned by publishing things worth citing, not by buying or swapping them.
Shortcuts that backfire
- Mass-producing near-identical pages for every city or keyword.
- Stuffing keywords into text, alt attributes or hidden elements.
- Buying links, or networks of sites that link to each other.
- Showing search engines different content from what visitors see.
- Copying or lightly rewording other sites’ content.
Search engines publish spam policies describing these practices. Sites that rely on them can lose visibility suddenly, and recovering takes far longer than doing the work properly.
A simple monthly routine
- Check your search console reports for indexing errors and fix the real ones.
- Look at which queries bring people to each page and whether the page answers them.
- Update one or two older pages with better information.
- Remove or merge pages that no one needs.
Questions
How long does SEO take to work?
New pages can be crawled within days, but settling into stable positions often takes weeks or months. There is no fixed timetable.
Do I need to submit every page?
No. A sitemap and good internal links are enough for crawlers to find pages. Manual submission is useful for checking a single important URL.