Technical SEO for Custom Sites: How Pages Get Found

Technical SEO for developer-built websites is the practice of configuring your site’s infrastructure—server settings, code structure, and metadata—so search engines can successfully crawl, index, and rank your pages. For custom-coded sites, this means you control every line of HTML, CSS, and JavaScript, which is a double-edged sword: you can optimize precisely, but you also inherit every technical pitfall. This article explains the core technical factors that determine whether your pages appear in search results, and how to fix them.

Background: Why Technical SEO Differs for Developer-Built Sites

Websites built from scratch with frameworks like React, Next.js, Vue, or plain HTML have no built-in SEO guardrails. Content management systems (CMS) like WordPress or Shopify automatically generate meta tags, sitemaps, and canonical URLs—but a custom site requires you to implement all of those yourself. The technical SEO fundamentals are the same for every site, but the risk of missing something is higher when you write the code. Search engines like Google use crawlers (bots) that follow links and parse HTML. If your site blocks those bots, serves the wrong content, or loads too slowly, your pages won’t rank, regardless of how good your content is.

How Search Engines Crawl and Index Your Pages

Google’s crawler, Googlebot, starts with a list of known URLs (from sitemaps and links) and fetches each page. It reads the HTML, follows links to other pages, and adds the content to Google’s index—a massive database of web pages. When a user searches, Google ranks indexed pages based on relevance and quality signals. For developer-built sites, three technical steps matter most:

• Crawlability: Googlebot must be able to reach your pages. This means no accidental noindex tags, no disallow rules in robots.txt blocking CSS or JavaScript, and no login walls that hide content.
• Indexability: Once crawled, each page must be eligible for the index. This requires a unique, valid URL, a canonical tag (if duplicates exist), and meta robots tags that don’t say noindex.
• Rendering: If your site relies on JavaScript to display content, Googlebot must execute that JavaScript. Modern Googlebot does render JS, but it’s slower and can fail. Server-side rendering (SSR) or static generation (SSG) is safer for critical content.

You can test crawlability using Google Search Console’s URL Inspection tool, which shows whether Google can fetch and render a specific URL.

Core Technical SEO Factors for Custom Code

These are the specific elements you need to implement correctly in your codebase:

• Meta tags: Each page needs a unique title tag (under 60 characters) and meta description (under 155 characters). These appear in search results and affect click-through rates.
• Semantic HTML: Use <h1> for the main heading, <h2> for subheadings, and <p> for paragraphs. This helps search engines understand the structure and importance of your content.
• Structured data: JSON-LD (a format for structured data) can be added to pages to mark up things like articles, products, or FAQs. This enables rich snippets (e.g., star ratings, FAQ dropdowns) in search results.
• Canonical URLs: If the same content is accessible via multiple URLs (e.g., with tracking parameters), add a rel="canonical" tag pointing to the preferred version to avoid duplicate content.
• XML sitemap: A sitemap lists all your important URLs and helps search engines discover new pages. Submit it via Google Search Console and Bing Webmaster Tools.
• robots.txt: This file tells crawlers which parts of your site to ignore. Be careful not to block CSS or JS files, as that can prevent rendering.
• Page speed: Google uses page speed as a ranking signal. For custom sites, optimize images (use modern formats like WebP), minify CSS/JS, and consider a content delivery network (CDN).

Common Technical SEO Problems in Developer-Built Sites

Custom sites often suffer from a specific set of issues that are less common on CMS platforms:

• JavaScript-rendered content not being indexed: If your site is a single-page application (SPA) that loads content via JavaScript, Google may not see the content if it fails to execute the JS. Fix: use SSR or SSG for critical pages, or at least provide a static fallback.
• Duplicate content from URL parameters: If your site uses query strings for tracking (e.g., ?utm_source=facebook), Google may see each variant as a separate page. Fix: add canonical tags or configure Google Search Console to ignore specific parameters.
• Accidental noindex tags: A developer might add <meta name="robots" content="noindex"> during development and forget to remove it. This blocks indexing entirely. Fix: audit your live pages for noindex tags.
• Broken internal links: Custom sites often have hardcoded links that break when pages are moved. Fix: use relative URLs where possible and regularly check for 404 errors in Search Console.
• Slow server response time: If your server takes more than a few seconds to respond, crawlers may give up. Fix: optimize database queries, use caching, and consider a faster hosting provider.

How to Choose the Right Approach: SSR vs. SSG vs. CSR

The rendering strategy you choose for your custom site significantly impacts SEO. Here’s a quick comparison:

• Server-side rendering (SSR): The server generates the full HTML for each request. This is the most SEO-friendly because crawlers receive complete content immediately. Frameworks like Next.js and Nuxt.js support SSR.
• Static site generation (SSG): The HTML is generated at build time and served as static files. This is fast and SEO-friendly, but content updates require a rebuild. Good for blogs and marketing sites.
• Client-side rendering (CSR): The browser runs JavaScript to build the page. This is the least SEO-friendly because crawlers may not execute JS. If you must use CSR, use a pre-rendering service or dynamic rendering (serving a static version to bots).

For most custom sites, SSR or SSG is the best choice. If you use a modern framework, you can often switch between them with minimal code changes.

Costs and Tools for Technical SEO

Technical SEO for custom sites can be done in-house by a developer, but it requires ongoing effort. The main costs are:

• Developer time: Implementing meta tags, structured data, and a sitemap might take a few hours. Debugging rendering issues can take days.
• Tools: Google Search Console and Bing Webmaster Tools are free. Paid tools like Screaming Frog (crawler), Ahrefs, or SEMrush can help with audits, but they are not essential for basic technical SEO.
• Hosting: Faster hosting (e.g., a VPS or dedicated server) costs more than shared hosting but improves page speed.

If you don’t have a developer, you can hire a freelance technical SEO specialist or use a CMS like WordPress that handles many of these issues automatically. However, for a custom site, you’ll need someone who understands both code and SEO.

How to Verify Your Technical SEO Is Working

After implementing the above, you need to verify that search engines can actually find and index your pages. Here’s a step-by-step process:

1. Submit your sitemap in Google Search Console (under “Sitemaps”).
2. Use the URL Inspection tool to test a few key pages. It will show you if Google can crawl and index them.
3. Check the “Page indexing” report in Search Console to see which pages are indexed and which are not, along with reasons.
4. Use a crawler like Screaming Frog (free for up to 500 URLs) to simulate Googlebot and find broken links, missing meta tags, or duplicate content.
5. Monitor your search performance in Search Console. If your impressions drop, something may have broken.

Do this at least monthly, as code changes can inadvertently break SEO elements.

Frequently Asked Questions

Q: Does technical SEO guarantee higher rankings?
No. Technical SEO only ensures your pages can be crawled and indexed. Rankings depend on content quality, relevance, and backlinks. But without technical SEO, even great content won’t appear in search results.

Q: How long does it take for technical SEO changes to take effect?
It varies. Fixing a noindex tag can lead to indexing within a few days. Improving page speed might take weeks to reflect in rankings. Generally, expect 2-4 weeks for Google to recrawl and reprocess your site after significant changes.

Q: Is JavaScript bad for SEO?
No, but it adds complexity. Google can render JavaScript, but it’s slower and can fail. If you use JS to load critical content, use SSR or SSG to ensure the content is in the initial HTML response.

Q: What’s the difference between a sitemap and a robots.txt file?
A sitemap is a list of URLs you want search engines to crawl. robots.txt is a set of rules telling crawlers which parts of your site they are allowed to crawl. They serve different purposes, and you need both.

Q: Can I do technical SEO without a developer?
If your site is custom-coded, you’ll likely need a developer for code changes. However, you can use plugins or a CMS to handle some aspects. For a fully custom site, it’s best to have a developer who understands SEO.

Q: How often should I run a technical SEO audit?
At least quarterly, or after any major site update. If you deploy code frequently, consider running a quick crawl after each release.

Related

• First Front End Developer Job: Practical Path
• Wireless Charging Galaxy S9: Complete Guide
• Norwegian Zoom Backgrounds: How to Get Them