We’ve all seen the movies where the robots are coming for us, including classics such as The Terminator and The Matrix. What people may not know is that they already came for us – and got all of our information… In fact, the most widely used website in the world is built upon one of these robots: The GoogleBot.

Indeed, not all robots are here to subjugate humanity and turn us into subservient slaves. They are actually quite helpful, indexing web sites on popular search engines so that visitors may come and indulge in the pages of our websites. Without these robots, most of the information revolution we’ve seen in the past 20 years would not have been possible.

But what if you have some information on your website you’d rather not have the whole world take a look at? Perhaps a baby picture from when you were small that you only share with family friends, internal pages that you may want to keep out of the search results page from a business perspective.  There are many valid reasons for “banning” the bots from certain pages, and there are some good ways to do this.

One answer is a robots.txt file. Essentially this is a text file (which can be written in any text editor) that issues commands to robots to visit only the portions of a website that you allow. The basic syntax is fairly simple, and a good overview is available here. We want to be very careful when employing these files, however, and make absolutely sure that we know what effects our actions will have.  For this reasons, many webmasters are uncomfortable with editing this themselves, as one small mistake could render your site entirely invisible (or entirely visible) to any robot.

Luckily, Google now offers a tool that will automatically generate a robots.txt file for you, saving some time and perhaps avoiding an unintentional disaster.

Using this tool can help you control the pages of your website, and we can make sure our robots keep coming back on our terms, without terminating us.

Need help with your technical SEO issues? Contact us today.

Message Sent

Thank you for registering.

Cardinal Path hosted a live session to connect with you and answer all your questions on Google Analytics.
Get all the expertise and none of the consultancy fees in this not-to-be-missed, rapid-fire virtual event.

Thank you for submitting the form.

Thank you for submitting the form.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you.

Click here to download access the tool.

Message Sent

Thank you for registering.

Message Sent

Thank you.

Message Sent

Thank you.

Message Sent

Thank you

Message Sent

Thank you

Message Sent

Thank you.

Message Sent

Thank you

Message Sent

Thank you.

Message Sent

Success!
Your message was received.

Thank you.

Thank you for registering.

Cardinal Path is continuing with its series of free training. Next we are conducting training on Google Data Studio. Check it out here.

Message Sent

Thank you for registering.

Thank you for your submission.

Your request has been submitted and a rep will reach out to you shortly.

Message Sent

Thank you for your interest.

Thank you for registering.

You should receive a confirmation email from GoToWebinar with your unique webinar login information. If you do not receive this email or have trouble logging in to the event, please email asmaa.mourad@cardinalpath.com.

Thank you for subscribing!

You're now looped into the world's largest GMP resource hub!

Thank you for your submission.

Thank you for your submission.

Thank you for your submission.

Thank you for your submission.

Message Sent

Thank you for registering.

Message Sent

Thank you for your submission.

Thank you for your submission.

Message Sent

Thank you for registering.

Thank you for registering.​

Paid media spend by Government websites increased a whopping 139% YoY in 2020.

2020 Online Behavior Live Dashboard

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

2020 Online Behavior Live Dashboard

Thank you for your submission.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Success! Thank you
for reaching out.