This has been generated by the StormCrawler Maven Archetype as a starting point for building your own crawler with [Apache Solr](https://solr.apache.org/) as a backend.
Have a look at the code and resources and modify them to your heart's content.

You need to have Apache Storm (2.8.8) installed, as well as a running instance of Apache Solr (9.8.0).

## Generated resources

### Flux topologies

- `injection.flux`: Topology that reads URLs from the _seeds.txt_ file and populates the `status` collection.
- `crawler.flux`: Basic topology that uses a `SolrSpout` as source and indexes parsed content into the `docs` collection.

### Solr configuration file `solr-conf.yaml`

For configuring the connection with the Solr server, the following parameters are available:
- `solr.TYPE.url`
- `solr.TYPE.zkhost`
- `solr.TYPE.collection`

In the previous example `TYPE` can be one of the following values:

* `indexer`: To reference the configuration parameters of the `IndexerBolt` class.
* `status`: To reference the configuration parameters of the `SolrSpout` and `StatusUpdaterBolt` classes.
* `metrics`: To reference the configuration parameters of the `MetricsConsumer` class.

> Note: Some of these classes provide additional parameter configurations.

#### General parameters

* `solr.TYPE.url`: The URL of the Solr server including the name of the collection that you want to use.

#### MetricsConsumer

In the case of the `MetricsConsumer` class a couple of additional configuration parameters are provided to use the [Document Expiration](https://lucidworks.com/blog/document-expiration/) feature available in Solr since version 4.8.

* `solr.metrics.ttl`: [Date expression](https://cwiki.apache.org/confluence/display/solr/Working+with+Dates) to specify when the document should expire.
* `solr.metrics.ttl.field`: Field to be used to specify the [date expression](https://cwiki.apache.org/confluence/display/solr/Working+with+Dates) that defines when the document should expire.

> Note: The date expression specified in the `solr.metrics.ttl` parameter is not validated. To use this feature some changes in the Solr configuration must be done.

#### SolrSpout

For the `SolrSpout` class a couple of additional configuration parameters are available to guarantee some *diversity* in the URLs fetched from Solr, in the case that you want to have better coverage of your URLs. This is done using the [collapse and expand](https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results) feature available in Solr.

* `solr.status.bucket.field`: Field to be used to collapse the documents.
* `solr.status.bucket.maxsize`: Amount of documents to return for each *bucket*.

For instance if you are crawling URLs from different domains, perhaps is of your interest to *balance* the amount of URLs to be processed from each domain, instead of crawling all the available URLs from one domain and then the other.

For this scenario you'll want to collapse on the `host` field (that already is indexed by the `StatusUpdaterBolt`) and perhaps you just want to crawl 100 URLs per domain. For this case is enough to add this to your configuration:

```yaml
solr.status.bucket.field: host
solr.status.bucket.maxsize: 100
```

This feature can be combined with the [partition features](https://github.com/apache/stormcrawler/wiki/Configuration#fetching-and-partitioning) provided by StormCrawler to balance the crawling process and not just the URL coverage.

> It is recommended to use Solr in cloud mode. The following configuration options are available for distributing the `status` collection across multiple shards.
> * `solr.status.routing.fieldname`: Field to be used for routing documents to different shards. The values depend on the `partition.url.mode` (`byHost`, `byDomain`, `byIP`)
> * `solr.status.routing.shards`: Number of shards for the `status` collection

#### Metadata

The metadata associated with each URL is also persisted in the Solr collection configured. By default the metadata is stored as separated fields in the collection using a prefix that can be configured using the `solr.status.metadata.prefix` option. If no value is supplied for this option the `metadata` value is used. Take a look at the following example record:

```json
{
  "url": "http://test.com",
  "host": "test.com",
  "status": "DISCOVERED",
  "key": "test.com",
  "metadata.url.path": "http://test.com",
  "metadata.depth": "1",
  "nextFetchDate": "2015-10-30T17:26:34.386Z"
}
```

In the previous example the `metadata.url.path` and `metadata.depth` attributes are elements taken from the `metadata` object. If the `SolrSpout` class is used to fetch URLs from Solr, the configured prefix (`metadata.` in this case) is stripped before populating the `Metadata` instance.

#### Using SolrCloud

To use a SolrCloud cluster instead of a single Solr server, you must use the following configuration parameters **instead** of the `solr.TYPE.url`:

* `solr.TYPE.zkhost`: URL of the Zookeeper host that holds the information regarding the SolrCloud cluster.

* `solr.TYPE.collection`: Name of the collection that you wish to use.

When using Solr in cloud mode, one can configure additional properties, related to the batching of update requests:
- `solr.TYPE.batchUpdateSize`: Number of updates queued (per shard) until an actual update request is sent to Solr. The default is 10.
- `solr.TYPE.flushAfterNoUpdatesMillis`: Maximum allowed time (in milliseconds) without incoming updates before all pending updates are flushed to Solr. The default value is 20000 (20 seconds).

### Collection configuration files

An example collection configuration for each type of data is also provided in the [`configsets`](configsets) directory. The configuration is very basic but it will allow you to view all the stored data in Solr.

The configuration is only useful as a testing resource, mainly because everything is stored as a `Solr.StrField` which is not very useful for search purposes. Numeric values and dates are also **stored as strings** using dynamic fields.

In the `metrics` collection an `id` field is configured to be populated with an auto-generated UUID for each document, this configuration is placed in the `solrconfig.xml` file. The `id` field will be used as the `uniqueKey`.

In the `parse` and `status` cores the `uniqueKey` is defined to be the `url` field.

Also keep in mind that depending on your needs you can use the [Schemaless Mode](https://cwiki.apache.org/confluence/display/solr/Schemaless+Mode) available in Solr.

### Solr scripts

* `setup-solr.sh`: Starts Solr in cloud mode, uploads the configsets and creates the collections.
* `clear-collections.sh`: Deletes all the documents from the collections.

## Bootstraping Solr

First start Solr with the preconfigured collections for StormCrawler:

```sh
./setup-solr.sh
```

Then open the Solr admin UI (http://localhost:8983) to check that they have been loaded correctly.

## Building

Generate an uberjar with:

``` sh
mvn clean package
```

## Running a topology

The first step consists of creating a file _seeds.txt_ in the current directory and populating it with the URLs
to be used as a starting point for the crawl, e.g.

`echo "https://stormcrawler.apache.org/" > seeds.txt`

You can start the crawl topology in local mode using the URLs in _seeds.txt_ as a starting point with

``` sh
storm local target/${artifactId}-${version}.jar  org.apache.storm.flux.Flux injection.flux --local-ttl 3600
```

Note that in local mode, Flux uses a default TTL for the topology of 20 secs. The command above runs the topology for 1 hour.

To start crawling, run the following command

``` sh
storm jar target/${artifactId}-${version}.jar  org.apache.storm.flux.Flux crawler.flux
```

Note that in the previous command, we ran the topology with `storm jar` to benefit from the Storm UI and logging. In that case, the topology runs continuously, as intended.
If you don't have a Storm cluster set up and/or want to run in local mode, simply replace _jar_ with _local_ and add _--local-ttl 3600_.
