Surfing As GoogleBot – Their IP, Their User-Agent, Their Bot Characteristics

21 Comments

1. 

   KMBA

   Dec 14, 2006

   Beat Referer Detection: Use the Firefox RefControl Extension to prevent referer from being sent… Dint know about that extension! thanks for the heads up

2. 

   SEOsnafu

   Dec 17, 2006

   More obsfucation is ideal, but it’s handjob reviews that get you busted.

3. 

   Buddah

   Aug 14, 2007

   You don’t need RefControl because Web Developer has an option to disable Referers from being sent.

   You also don’t need to install any extension to disable referers. just go to about:config and type “referer” in the box. Change the value to 0.

   Also, you can disable cookies and javascript without using extensions. Check your firefox preferences.

4. 

   Ares

   Aug 14, 2007

   Google Translate no longer hides your IP.

   Author Comment: Not true… I just ran a check myself to see what IP shows up in the REMOTE_ADDR field of $_SERVER variables in PHP when a page is accessed by Google Translate. It showed the Google IP. Google Translate sends an extra header called HTTP_X_FORWARDED_FOR along with your IP address as well, but the vast majority of sites do not check for this.

5. 

   Sander

   Aug 14, 2007

   The webdeveloper toolbar can block refers as well.

6. 

   Wandering

   Aug 14, 2007

   Thanks for the expanded explanation, the other articles *were* over-simplified.

7. 

   Christian

   Aug 14, 2007

   Omg, do you really need all these extensions?
   Using Opera I can open a menu* and easily disable cookies, javascript, referer etc.

   Changing user agent.. right now I don’t know, but I’m sure you can customize it somewhere, like almost everything else in Opera.

   * press F12 or go to Tools > Quick Preferences

8. 

   Isaac

   Aug 15, 2007

   Yes, you can do the same in Firefox. But you need to open the menu to do it.
   Alt-O (Windows) will open Options.
   Under Content, Javascript can be disabled.
   Cookies (for all sites or a specific site) can be disabled under Privacy.
   Um… OK. I think you do need an extension for referer. Or just go to about:config and change the value of network.http.sendRefererHeader. Its 2 by default. I don’t know what values mean what, but I’m sure I could figure it out.
   http://www.spreadfirefox.com/node/5841 says 0 will turn it off.

   about:config also got a bunch of general.useragent settings, but I’m not sure which one does what. For useragent, take a look here:
   http://johnbokma.com/mexit/2004/04/24/changinguseragent.html

   Yes, Firefox has it all built in.
   But Firefox was /designed/ to be extended. It was /designed/ to have minimal extra features so people could chose exactly what features it has and what it doesn’t.

9. 

   Access Blocked Sites

   Aug 15, 2007

   Here are few more tricks to access blocked sites at your office/school/university. Access Blocked Sites

10. 

    Nick

    Aug 15, 2007

    Use the Google Web Accelerator for a much more usable method of beating IP delivery than Google Translate.

11. 

    live tv

    Aug 15, 2007

    WebDev toolbar can do it all. Or you can just view a site’s cache.

12. 

    AG

    Aug 15, 2007

    You can achieve all of the 5 mentioned steps with a single extension: prefbar. It adds a toolbar that makes enabling/disabling cookies, javascript, referrer as simple as clicking a checkbox. User agent and proxy selections are handled through drop-down boxes. There are also dozens of other included customizable selectors, and you can make your own (I made one to disable redirects – another useful googlebot spoof).
    http://prefbar.mozdev.org/

13. 

    Marah Marie

    Aug 15, 2007

    *sigh*

    1. Beat IP Delivery: Use Google Translate as a Proxy, translating from spanish->english even though the site is already in English.

    Author Comment: Actually, the whole point is to use Google’s IP, so that if someone is cloaking using IP delivery, they will still assume you are Google. Other proxies would not fall into the same c-block range as Google’s, making you less likely to succeed

    OR…just use a proxy. Any proxy.

    3. Beat Javascript Detection: Use the Firefox Web Developer Toolbar to turn off javascript.

    OR…simply block JS through FF settings, no need to download WD.

    4. Beat Cookie Detection: Use the Firefox Web Developer Toolbar to turn off cookies.

    OR….just use FF to turn off cookies.

    Have you guys ever looked through Tools–>Options–>Content and Privacy?

    #2 and #5: OK.

    #1, 3, 4: Why not show them the easiest way first…

14. 

    Marah Marie

    Aug 17, 2007

    “Google Translate no longer blocks your IP” (comment by Ares, above mine somewhere)…if that’s true, no point using it, correct? Also, as Isaac pointed out, with a few about:config changes you don’t need to use add-ons for this at all.

    Uh…you got owned?

15. 

    Nio

    Aug 28, 2007

    Google Translater does not block IP any more, so this trick is not working any more!

16. 

    James Fraise

    Oct 13, 2008

    xB Browser does all of this already. Just download it and set your useragent to Googlebot.

17. 

    Matthew Song

    Dec 29, 2008

    It’s weird, GoogleBot loves my site graydarkarea.blogspot.com but not webmarketingfaq.blogspot.com; does anyone know why?

18. 

    t_g_c

    Nov 29, 2010

    Does Google Web Accelerator hide your IP or dies it publish it as Translate currently does?

19. 

    faust

    Dec 12, 2010

    today i found http://keiths-proxy-server.appspot.com/. it appears to be hosted on google’s dime. which makes it interesting, no?

    googlebot signs in. also check out billbot. thank me later.

    other observations: refcontrl works almost as good as curl. webdeveloper toolbar is more important to me (almost) than firefox because of how hard it rocks. nytimes is a greased — uh, is a good place to cut ones teeth. science journals are the holy grail. and the new webcache.googleusercontent dealio actually makes things easier —

    love to know yr responses, o original-poster & those whose eyes fall across these words

20. 

    faust

    Dec 12, 2010

    but wait theres the secret part! or if you wont let the little would be geeklette’s puny code thru, it is here.

21. 

    Jonathon

    Dec 15, 2014

    Or ou could simply get a job at google