Technical SEO basics: crawlability, sitemaps and robots.txt

TL;DR

Technical SEO ensures search engines can crawl, understand and index your site. The basics are correct indexing (no accidental noindex), an XML sitemap and careful robots.txt, fast mobile pages, and a clear, well-linked structure.

Key takeaways

  • If search engines cannot crawl and index your site, content stays invisible.
  • A common costly mistake is leaving a site-wide noindex on after launch.
  • Submit an XML sitemap and handle robots.txt carefully.
  • Speed and mobile friendliness are technical ranking signals.
  • Keep a clear structure with descriptive URLs and internal links.

Technical SEO is the foundation everything else sits on. If search engines cannot crawl, understand and index your site, even the best content stays invisible. The good news is that the basics are manageable for a small business. This guide covers the technical SEO essentials in plain terms.

Crawlability and indexing

Search engines send crawlers to read your pages, then decide which to index. Two things must be true: crawlers can reach your pages, and you actually want them indexed. A frequent and costly mistake is leaving a site-wide noindex on after launch, which tells search engines to ignore everything. Check Search Console to confirm your important pages are indexed.

Sitemaps and robots.txt

An XML sitemap lists your important pages so search engines can find them efficiently. Most SEO plugins generate one automatically; submit it in Google Search Console. The robots.txt file tells crawlers where they may and may not go. Used well it is helpful, used carelessly it can accidentally block your whole site, so handle it with care. These pair with the structured data in our schema markup guide.

Speed and mobile

Speed and mobile friendliness are technical signals that also affect ranking. A slow or awkward mobile site frustrates both visitors and search engines. This is where Core Web Vitals and mobile-first design meet technical SEO. Fast, well built hosting underpins all of it.

Site structure and links

A clear structure helps crawlers and people alike. Keep important pages a few clicks from the homepage, use descriptive URLs, and link related pages together so authority flows through the site. Fix broken links and avoid orphan pages that nothing links to, since crawlers may struggle to find them. Combine this technical base with a solid on-page checklist and you have a healthy foundation. For a full technical audit, see our SEO service.

Frequently asked questions

What is technical SEO?

It is the work that lets search engines crawl, understand and index your site: indexing settings, sitemaps, robots.txt, speed, mobile friendliness and site structure.

Why are my pages not appearing in Google?

Common causes include an accidental site-wide noindex left on after launch, a robots.txt blocking crawlers, or pages with no internal links. Check Google Search Console to diagnose.

Do I need an XML sitemap?

Yes, it helps search engines find your important pages efficiently. Most SEO plugins generate one automatically, which you then submit in Google Search Console.