Surfing As GoogleBot – Their IP, Their User-Agent, Their Bot Characteristics

After reading this article and this article which give frustratingly over-simplifications on user-agent spoofing to get past cloaked websites, I figured I should write something on how to REALLY behave like Google. Cloaking often goes well beyond this, using IP delivery, User Agent cloaking, javascript and cookie detection, and referer detection – all of which can be used to determine that you are you and not a bot. So, how do you beat all 5 major types of cloaking? 1. Beat IP Delivery: Use Google Translate as a Proxy, translating from spanish->english even though the site is already in English. 2. Beat User-Agent Cloaking: Use the FirefoxUser-Agent Switcher to spoof as GoogleBot 3. Beat Javascript Detection: Use the Firefox Web Developer Toolbar to turn...