Search engines can now discover and index every public page on a site reliably. Each site serves a sitemap.xml generated automatically from the pages and events already in the site, and a robots.txt that points crawlers at it. Every page also declares its official address, so duplicate listings caused by tracking labels such as ?ref=ed_direct collapse into the proper address in Google's index.
sitemap.xml is generated automatically from the site's public pages, walks and social events, so it stays up to date on its ownrobots.txt points search engines at the sitemap and keeps them out of admin and login pages