# www/crawl - The NetBSD Packages Collection

Small and efficient HTTP crawler

    The crawl utility starts a depth-first traversal of the web at the specified
    URLs. It stores all JPEG images that match the configured constraints.
    Crawl is fairly fast and allows for graceful termination. After terminating
    crawl, it is possible to restart it at exactly the same spot where it was
    terminated. Crawl keeps a persistent database that allows multiple crawls
    without revisiting sites.

    The main features of crawl are:

     * Saves encountered images or other media types
     * Media selection based on regular expressions and size contraints
     * Resume previous crawl after graceful termination
     * Persistent database of visited URLs
     * Very small and efficient code
     * Asynchronous DNS lookups
     * Supports robots.txt

- Source version: `crawl-0.4nb14`
- Maintainer: pkgsrc-users AT NetBSD.org
- Website: https://monkey.org/~provos/crawl/
- License: none stated
- Package history: cvs, hg, git

## Build dependencies

pkgtools/mktools pkgtools/cwrappers

## Runtime dependencies

(none)

## Binary packages

OS | Architecture | Version
--- | --- | ---
NetBSD 10.0 | aarch64 | crawl-0.4nb14.tgz
NetBSD 10.0 | aarch64 | crawl-0.4nb14.tgz
NetBSD 10.0 | aarch64eb | crawl-0.4nb14.tgz
NetBSD 10.0 | alpha | crawl-0.4nb14.tgz
NetBSD 10.0 | earmv4 | crawl-0.4nb14.tgz
NetBSD 10.0 | earmv4 | crawl-0.4nb14.tgz
NetBSD 10.0 | earmv6hf | crawl-0.4nb14.tgz
NetBSD 10.0 | earmv6hf | crawl-0.4nb14.tgz
NetBSD 10.0 | earmv7hf | crawl-0.4nb14.tgz
NetBSD 10.0 | earmv7hf | crawl-0.4nb14.tgz
NetBSD 10.0 | i386 | crawl-0.4nb14.tgz
NetBSD 10.0 | i386 | crawl-0.4nb14.tgz
NetBSD 10.0 | m68k | crawl-0.4nb14.tgz
NetBSD 10.0 | powerpc | crawl-0.4nb14.tgz
NetBSD 10.0 | powerpc | crawl-0.4nb14.tgz
NetBSD 10.0 | powerpc | crawl-0.4nb14.tgz
NetBSD 10.0 | sh3el | crawl-0.4nb14.tgz
NetBSD 10.0 | sparc64 | crawl-0.4nb14.tgz
NetBSD 10.0 | sparc64 | crawl-0.4nb14.tgz
NetBSD 10.0 | vax | crawl-0.4nb14.tgz
NetBSD 10.0 | x86_64 | crawl-0.4nb14.tgz
NetBSD 10.0 | x86_64 | crawl-0.4nb14.tgz
NetBSD 11.0 | aarch64 | crawl-0.4nb14.tgz
NetBSD 11.0 | aarch64 | crawl-0.4nb14.tgz
NetBSD 11.0 | aarch64eb | crawl-0.4nb14.tgz
NetBSD 11.0 | aarch64eb | crawl-0.4nb14.tgz
NetBSD 11.0 | earmv4 | crawl-0.4nb14.tgz
NetBSD 11.0 | earmv4 | crawl-0.4nb14.tgz
NetBSD 11.0 | earmv6hf | crawl-0.4nb14.tgz
NetBSD 11.0 | earmv6hf | crawl-0.4nb14.tgz
NetBSD 11.0 | earmv7hf | crawl-0.4nb14.tgz
NetBSD 11.0 | earmv7hf | crawl-0.4nb14.tgz
NetBSD 11.0 | i386 | crawl-0.4nb14.tgz
NetBSD 11.0 | i386 | crawl-0.4nb14.tgz
NetBSD 11.0 | m68k | crawl-0.4nb14.tgz
NetBSD 11.0 | m68k | crawl-0.4nb14.tgz
NetBSD 11.0 | powerpc | crawl-0.4nb14.tgz
NetBSD 11.0 | riscv64 | crawl-0.4nb14.tgz
NetBSD 11.0 | riscv64 | crawl-0.4nb14.tgz
NetBSD 11.0 | riscv64 | crawl-0.4nb14.tgz
NetBSD 11.0 | sh3el | crawl-0.4nb14.tgz
NetBSD 11.0 | sh3el | crawl-0.4nb14.tgz
NetBSD 11.0 | vax | crawl-0.4nb14.tgz
NetBSD 11.0 | vax | crawl-0.4nb14.tgz
NetBSD 11.0 | x86_64 | crawl-0.4nb14.tgz
NetBSD 11.0 | x86_64 | crawl-0.4nb14.tgz
NetBSD 9.0 | aarch64 | crawl-0.4nb14.tgz
NetBSD 9.0 | aarch64 | crawl-0.4nb14.tgz
NetBSD 9.0 | earmv6hf | crawl-0.4nb14.tgz
NetBSD 9.0 | earmv6hf | crawl-0.4nb14.tgz
NetBSD 9.0 | earmv7hf | crawl-0.4nb14.tgz
NetBSD 9.0 | earmv7hf | crawl-0.4nb14.tgz
NetBSD 9.0 | i386 | crawl-0.4nb14.tgz
NetBSD 9.0 | powerpc | crawl-0.4nb14.tgz
NetBSD 9.0 | x86_64 | crawl-0.4nb14.tgz

Binary packages can be installed with the high-level tool pkgin (which can be installed with pkg_add) or pkg_add(1) (installed by default). The NetBSD packages collection is also designed to permit easy installation from source.

## Available build options

(none)

## Known vulnerabilities

(no vulnerabilities known)

The pkg_admin audit command locates any installed package which has been mentioned in security advisories as having vulnerabilities.

Please note the vulnerabilities database might not be fully accurate, and not every bug is exploitable with every configuration.

Problem reports, updates or suggestions for this package should be reported with send-pr.