Method: PHPCrawler::setFollowRedirectsTillContent()

Defines whether the crawler should follow HTTP-redirects until first content was found, regardless of defined filter-rules and follow-modes.

Signature:

public setFollowRedirectsTillContent($mode)

Parameters:

$mode

bool

If TRUE, the crawler will follow redirects until content was finally found. Defaults to TRUE.

Returns:

No information

Description:

Sometimes, when requesting an URL, the first thing the webserver does is sending a redirect to
another location, and sometimes the server of this new location is sending a redirect again
(and so on).
So at least its possible that you find the expected content on a totally different host
as expected.

If you set this option to TRUE, the crawler will follow all these redirects until it finds some content.
If content finally was found, the root-url of the crawling-process will be set to this url and all
defined options (folllow-mode, filter-rules etc.) will relate to it from now on.