LAYIQBot
LAYIQBot is the crawler LAYIQ uses to read public job postings on employer career sites, so job seekers can find those jobs on LAYIQ. Every job LAYIQ shows links back to the employer's own posting, where people apply.
What it reads
- Public job listings and job posting pages on employer career sites and the applicant tracking systems that host them, such as Workday-hosted career sites.
- Each site's robots.txt, before it reads that site's listings or postings.
- It does not sign in, fill in or submit forms, apply for jobs, or read anything behind a login.
How it identifies itself
- User-Agent:
LAYIQBot/1.0 (+https://www.layiq.co/bot) - Every request that carries that name is signed with an HTTP Message Signature (Web Bot Auth) and
Signature-Agent: "https://www.layiq.co". The public key that checks the signature is published at https://www.layiq.co/.well-known/http-message-signatures-directory. A request that uses the name without a valid signature is not ours. - LAYIQ does not publish IP ranges for the crawler. To tell whether a request is really ours, check its signature against that key.
- Some older parts of our crawler still identify as
RoleworthyHarvester/1.0 (+https://roleworthy.ai)and do not sign. That is LAYIQ too.
robots.txt
- LAYIQBot follows the rules in a
User-agent: LAYIQBotgroup and the rules for*. A path is fetched only when both allow it. Groups written for our former name,RoleworthyHarvester, still apply. - It never reads a site faster than its Crawl-delay allows. For now, a site whose robots.txt sets a Crawl-delay is not read at all.
- If a site's robots.txt cannot be read, it does not read that site's listings until it can.
How fast it goes
LAYIQBot paces its requests per vendor. On Workday-hosted career sites it averages no more than 7.5 requests a second across all of those sites together, in bursts of at most 15. A request refused with a rate limit or a server error is tried again only a few times, waiting longer before each try and honoring Retry-After; after that the crawler leaves the site until its next scheduled read.
Stop it, or reach us
To keep LAYIQBot off your site, add this to your robots.txt:User-agent: LAYIQBotDisallow: /
For anything else, including removing your jobs from LAYIQ, email support@layiq.co.
