robots.txt
Controls crawler access patterns. It is not a reliable way to remove a URL from search results, and blocking a page can prevent a crawler from seeing a page-level noindex.
Technical SEO
Crawl, render, choose a canonical, then index.
Discovery, crawling, rendering, canonical selection, and indexing are related but distinct. A sitemap is a discovery hint, not a command to index or rank a URL.
The sequence
DiscoverLinks and sitemaps reveal URLs.
CrawlA crawler requests the URL.
RenderHTML and resources form the page.
IndexThe engine evaluates canonical content.
Signals and controls
noindex
A page-level directive asking compliant search engines not to index the page. The crawler must be allowed to fetch the page to see it.
XML sitemap
Lists canonical URLs you care about. Include live indexable URLs, use accurate last-modified dates, and remove redirects, errors, duplicates, and private pages.
Canonical
Identifies the preferred URL for duplicate or substantially similar content. Redirect exact duplicates when practical and keep internal links and sitemaps consistent with the canonical.
Redirect
Moves visitors and crawlers to another URL. Use a direct permanent redirect for durable URL changes and avoid multi-hop chains.
HTTP status
Return 200 for real content, 3xx for redirects, 404 or 410 for missing content, and 5xx for temporary server failures. Do not return branded error HTML with status 200.
Audit one URL
- Open the public canonical URL. Confirm correct content and status without a tracking query.
- Inspect the HTML. Verify title, H1, canonical, robots directive, links, and meaningful visible text.
- Load required assets. Check CSS, JavaScript, images, fonts, and mobile rendering.
- Compare variants. HTTP, www, case, slash, and duplicate parameters should converge consistently.
- Check sitemap and internal links. Both should point at the canonical URL.
- Use URL Inspection. Compare live access with indexed state and the search engine-selected canonical.