public setUrlCacheType($url_cache_type)
| $url_cache_type | int | 1 -> in-memory-cache (default setting) 2 -> SQlite-database-cache Or one of the PHPCrawlerUrlCacheTypes::URLCACHE..-constants. |
| bool |
Currently phpcrawl is able to use a in-memory-cache or a SQlite-database-cache for
caching/storing found URLs internally.
The memory-cache (PHPCrawlerUrlCacheTypes::URLCACHE_MEMORY) is recommended for spidering small to medium websites.
It provides better performance, but the php-memory-limit may be hit when too many URLs get added to the cache.
This is the default-setting.
The SQlite-cache (PHPCrawlerUrlCacheTypes::URLCACHE_SQLite) is recommended for spidering huge websites.
URLs get cached in a SQLite-database-file, so the cache only is limited by available harddisk-space.
To increase performance of the SQLite-cache you may set it's location to a shared-memory device like "/dev/shm/"
by using the setWorkingDirectory()-method.
Example:$crawler->setUrlCacheType(PHPCrawlerUrlCacheTypes::URLCACHE_SQLITE);
$crawler->setWorkingDirectory("/dev/shm/");
NOTE: When using phpcrawl in multi-process-mode (goMultiProcessed()), the cache-type is automatically set
to PHPCrawlerUrlCacheTypes::URLCACHE_SQLITE.