Login

Snippets by haloween

Snippet List

Web crawler/bot detection and blocking middleware

Sets request.is_crawler Allow bot lockout from certain urls in urlconf ,add view parameter 'deny_crawlers' ex. url(r'^foo/$', 'views.foo',{'deny_crawlers' : True},name='foo') view param is removed after middleware pass.

  • middleware
  • bot
  • google
  • yahoo
  • crawler
  • robot
  • googlebot
  • cuil
  • slurp
Read More

haloween has posted 2 snippets.