What our crawler reads, how it identifies itself and how to ask it to stop.
The NicheGaps crawler reads public reviews and discussions where people describe problems with software: app store review pages and feeds (App Store, Google Play, Shopify App Store), Reddit posts through the Arctic Shift archive, and Hacker News through its public search API.
It only reads pages anyone can see without signing in. It never logs in, posts, votes or contacts anyone, and it doesn't collect reviewers' names for display.
Every request sends the User-Agent Mozilla/5.0 (compatible; NicheGaps/1.0; +https://nichegaps.com/bot).
Slowly: at most one request every one to two seconds per site, and each app or community is checked about once a day, reading only what's new since the last visit. It backs off when a site answers with an error or asks it to slow down.
If you run a site the crawler visits and want it to stop, or you'd like content you wrote removed from NicheGaps, email [email protected] with the address. We'll stop or remove it, usually within a few days.