Crawling A 20 Million Page Website

In this post we will examine the importance of efficient web crawling and how you can do it for a website with millions of pages. SEO starts with a solid tech stack and a logical directory structure, this is particularly important in eCommerce.