The Straightpin crawler

If you run a careers page and found us in your logs, this page is for you.

Who we are

Straightpin is a job search site run by BufferSol Technologies. Our crawler identifies itself with this user agent:

StraightpinByBufferSolBot (+https://straightpin.buffersoltech.com/bot)

What we crawl

Public job feeds that hiring systems offer to anyone: Greenhouse, Lever, Ashby, Workable, Recruitee and Personio.
Public careers pages that list a job in schema.org JobPosting markup, found through the sitemap named in your robots.txt.
Nothing behind a login, nothing that needs a form, and no applicant data. We never apply on anyone's behalf.

We do not crawl indeed.com, linkedin.com, glassdoor.com, directroles.net, and we do not use proxies or tools that hide who we are. If a site shows a bot check, we stop and mark that host as blocked instead of getting around it.

How we behave

We read your robots.txt before any other request and keep it for up to 24 hours. A server error on robots.txt counts as "do not crawl" for 24 hours.
At most one request per second and four at once to a host. We follow Crawl-delay and Retry-After.
We ask only for what changed, using ETag and If-Modified-Since when your server supports them.

How to ask us to stop

Either way works. Use whichever is easier for you.

Add this to your robots.txt. We pick it up within a day:
```
User-agent: straightpinbybuffersolbot
Disallow: /
```
Email contact@buffersoltech.com with the subject "Crawler removal request" and your company name or careers page address. We stop crawling and remove your listings within 48 hours, and we reply when it is done.

Contact

Something wrong with how we visit your site, or a listing you want corrected? Write to contact@buffersoltech.com. A person reads it. What we keep from your page is described on what we store.

Each listing is shown as facts, a short excerpt of up to 600 characters, and a link that sends the job seeker to your own page to apply.