NAME

Gungho::Component::RobotsMETA - Automatically Parse Robots META

SYNOPSIS

  components:
    - RobotsMETA

DESCRIPTION

This module automatically parses any text/html document for robots exclusion directies embedded in the document.

METHODS

setup

handle_response