Design Multithreaded Web Crawler (medium)

You are given a starting URL and an HtmlParser API. The parser returns every URL found on a page:

Design a multithreaded web crawler that visits every URL reachable from startUrl that has the same hostname as startUrl.

A hostname is the part of a URL between the scheme and the next /. For example, the hostname of http://news.yahoo.com/news/topics/ is news.yahoo.com.

Your crawler must:

Include startUrl in the result.
Follow only URLs with the same hostname as startUrl.
Crawl each URL at most once, even when the graph contains duplicate links or cycles.
Use multiple worker threads so independent pages can be fetched concurrently.
Return only after the complete reachable same-host graph has been explored.

The result may be returned in any order.

The judge supplies the HtmlParser implementation and calls your WebCrawler class. The small HtmlParser declaration in the starter code describes that API; do not modify its method signature.

The judge also preloads the standard concurrency, collection, and callback APIs for every supported language. You do not need to add import, include, using, or package statements.

Example 1:

Input:

Output:

Explanation: http://news.google.com/news is reachable, but its hostname differs from the hostname of startUrl, so it is not crawled.

Example 2:

Input:

Output:

Explanation: The duplicate link and cycle do not cause either page to be crawled more than once.

Constraints

1 <= number of URLs <= 1000
1 <= url.length <= 300
startUrl is a valid URL in the hidden web graph.
All URLs use the http:// or https:// scheme.
htmlParser.getUrls(url) is safe to call concurrently.
The hidden graph may contain duplicate links, cycles, and links to other hostnames.