Should I Build It Should I Build It
03 May 2025
Hiring Career

job portal which scrapes real time data from websites and hiring pages ...

...of companies

Confidence
Engagement
Net use signal
Net buy signal

Idea type: Swamp

The market has seen several mediocre solutions that nobody loves. Unless you can offer something fundamentally different, you’ll likely struggle to stand out or make money.

Should You Build It?

Don't build it.

Your are here

You're entering a crowded space with your job portal idea, as indicated by the 25 similar products we found. This puts you squarely in the 'Swamp' category, meaning there are many existing solutions that haven't quite hit the mark. The average engagement on these similar products is low, with an average of only 1 comment per product launch. This suggests that grabbing user attention will be difficult, and standing out from the noise will be a real challenge. Given the 'Swamp' category's description, and the mediocre engagement with similar products, you really need to find a new angle to this idea, or you might not succeed. Remember, these metrics are based on real-world product launches, so pay attention to what they're telling you.

Recommendations

  1. First, thoroughly investigate why existing job portals that scrape data haven't become dominant. Identify their shortcomings in user experience, data accuracy, or job relevance. This will help you avoid repeating their mistakes and find opportunities for differentiation. Look closely at the negative user feedback for similar products; for example, users frequently mention the lack of filters and alerts. Address these needs directly in your initial design.
  2. If you decide to proceed, identify a very specific niche or underserved group within the job market. Instead of targeting all job seekers, focus on a specific industry, location, or skill set. For example, you could focus on entry-level data science jobs in Chicago, or remote marketing roles for veterans. Narrowing your focus will help you tailor the platform to meet specific needs and attract a dedicated user base.
  3. Consider building tools or APIs for existing job boards instead of directly competing with them. You could create a tool that enhances their data analysis, automates certain tasks, or provides unique insights. For instance, you might develop an API that extracts and analyzes salary data from job postings on Indeed or LinkedIn, offering valuable insights to job seekers and recruiters. The discussion summaries indicate that people are already interested in this kind of thing.
  4. Explore adjacent problems within the job search ecosystem that might be more promising. For example, instead of scraping job postings, you could focus on building a platform that helps job seekers optimize their resumes, prepare for interviews, or negotiate salaries. Addressing these pain points can be a more effective way to enter the market and build a sustainable business. Remember the criticism of similar products: a lack of detailed company information could be a good starting point here.
  5. Given the competitive landscape and the challenges of building a successful job portal, carefully evaluate whether this is the best use of your time and resources. There may be other opportunities that offer a higher potential for success with less competition. The ultimate point is not to waste your energy.
  6. Since several similar products received complaints about location input and the idealness of the results, focus on a local area and become 'the place to go' for jobs in that area, instead of trying to compete with global job boards. Become known as the source of truth for this specific area, which would also help with data accuracy and attracting users.

Questions

  1. What unique data points or analysis can you provide that existing job portals don't offer, and how will you ensure the accuracy and reliability of your scraped data?
  2. How will you differentiate your job portal from existing platforms to attract and retain users, especially considering the low engagement observed in similar products?
  3. What is your plan for sustainably scaling your data scraping operations, and how will you address potential legal or ethical concerns related to data privacy and intellectual property?

Your are here

You're entering a crowded space with your job portal idea, as indicated by the 25 similar products we found. This puts you squarely in the 'Swamp' category, meaning there are many existing solutions that haven't quite hit the mark. The average engagement on these similar products is low, with an average of only 1 comment per product launch. This suggests that grabbing user attention will be difficult, and standing out from the noise will be a real challenge. Given the 'Swamp' category's description, and the mediocre engagement with similar products, you really need to find a new angle to this idea, or you might not succeed. Remember, these metrics are based on real-world product launches, so pay attention to what they're telling you.

Recommendations

  1. First, thoroughly investigate why existing job portals that scrape data haven't become dominant. Identify their shortcomings in user experience, data accuracy, or job relevance. This will help you avoid repeating their mistakes and find opportunities for differentiation. Look closely at the negative user feedback for similar products; for example, users frequently mention the lack of filters and alerts. Address these needs directly in your initial design.
  2. If you decide to proceed, identify a very specific niche or underserved group within the job market. Instead of targeting all job seekers, focus on a specific industry, location, or skill set. For example, you could focus on entry-level data science jobs in Chicago, or remote marketing roles for veterans. Narrowing your focus will help you tailor the platform to meet specific needs and attract a dedicated user base.
  3. Consider building tools or APIs for existing job boards instead of directly competing with them. You could create a tool that enhances their data analysis, automates certain tasks, or provides unique insights. For instance, you might develop an API that extracts and analyzes salary data from job postings on Indeed or LinkedIn, offering valuable insights to job seekers and recruiters. The discussion summaries indicate that people are already interested in this kind of thing.
  4. Explore adjacent problems within the job search ecosystem that might be more promising. For example, instead of scraping job postings, you could focus on building a platform that helps job seekers optimize their resumes, prepare for interviews, or negotiate salaries. Addressing these pain points can be a more effective way to enter the market and build a sustainable business. Remember the criticism of similar products: a lack of detailed company information could be a good starting point here.
  5. Given the competitive landscape and the challenges of building a successful job portal, carefully evaluate whether this is the best use of your time and resources. There may be other opportunities that offer a higher potential for success with less competition. The ultimate point is not to waste your energy.
  6. Since several similar products received complaints about location input and the idealness of the results, focus on a local area and become 'the place to go' for jobs in that area, instead of trying to compete with global job boards. Become known as the source of truth for this specific area, which would also help with data accuracy and attracting users.

Questions

  1. What unique data points or analysis can you provide that existing job portals don't offer, and how will you ensure the accuracy and reliability of your scraped data?
  2. How will you differentiate your job portal from existing platforms to attract and retain users, especially considering the low engagement observed in similar products?
  3. What is your plan for sustainably scaling your data scraping operations, and how will you address potential legal or ethical concerns related to data privacy and intellectual property?

  • Confidence: High
    • Number of similar products: 25
  • Engagement: Low
    • Average number of comments: 1
  • Net use signal: 8.6%
    • Positive use signal: 11.9%
    • Negative use signal: 3.3%
  • Net buy signal: -3.3%
    • Positive buy signal: 0.0%
    • Negative buy signal: 3.3%

This chart summarizes all the similar products we found for your idea in a single plot.

The x-axis represents the overall feedback each product received. This is calculated from the net use and buy signals that were expressed in the comments. The maximum is +1, which means all comments (across all similar products) were positive, expressed a willingness to use & buy said product. The minimum is -1 and it means the exact opposite.

The y-axis captures the strength of the signal, i.e. how many people commented and how does this rank against other products in this category. The maximum is +1, which means these products were the most liked, upvoted and talked about launches recently. The minimum is 0, meaning zero engagement or feedback was received.

The sizes of the product dots are determined by the relevance to your idea, where 10 is the maximum.

Your idea is the big blueish dot, which should lie somewhere in the polygon defined by these products. It can be off-center because we use custom weighting to summarize these metrics.

Similar products

Relevance

Jobs Scraper for Workatastartup.com

12 Jun 2024 Hiring Career

This command-line job searching tool scrapes online listings, and compares it with your resume+preferences to find matches using AI (custom prompts + OpenAI LLM)A few months I posted it here (https://news.ycombinator.com/item?id=39621373). There have been quite a few improvements since thenAnd now I'm very happy and thankful to tell you, that some contributors have added workatastartup.com as a listings source for CommandJobsSo now you can also find matches among the public workatastartup.com/jobs listingsHere's the direct link to the repo: https://github.com/nicobrenner/commandjobsPS: Job listing sources contributions are open and welcome, checkout issue #23

This looks cool

Avatar
2
1
1
2
Relevance

A tech job board I've been building over the last 8 months

25 Jun 2024 Career

Hey HN! Quick TL; DR: I set up scrapers for the websites of ~20k tech companies, fed the results through GPT, spent some sleepless nights building a web app, and voilá.I started this after getting fed up with how slow LinkedIn felt. I finally got it working and easy to use, so I wanted to share it with you guys.The job list is free. But I’m currently paying a lot of money to OpenAI for all the data I’m passing through GPT, so I may end up eventually charging for some advanced features, or putting ads on it or something.Please let me know what you think!I’m actively developing this so I’m happy to add requested features.Some technical details you may be curious about: - I began gathering the career pages for each company manually, but I ended up hiring a few Filipino contractors to speed up the process. - I run the web scrapers once per day. - I do the web scraping with Node.js. I use cheerio for basic web pages, and puppeteer for more complicated ones. - My OpenAI API dashboard tells me I process 7 million characters per day. - The list doesn’t encompass every single tech company (yet). I have most of the major ones (~20k), but the process of adding new companies is pretty labour intensive so it’s still ongoing. - The web app is built with React, and I got the infinite scrolling to be so smooth using a great library called React Query.For feature requests, please leave a comment here or email me at dorfmanjames@gmail.com

Users suggest adding European companies, remote job options, email alerts, and detailed company information. One user finds the app cool and shares a related link.

The product lacks European companies, remote job options, email alerts, and detailed company information.

Avatar
3
2
2
3
Relevance

Job Listings Scraper

04 Dec 2024 Career

I found my last job by using a scraper that visits company websites in search of job listings. Now I've turned it into an app for others to use and access jobs that are posted on company websites (rather than paid employer ads on Indeed or where ever). This gives the job searcher an advantage to find jobs not listed on job search sites and show the company you have taken time and interest to visit their company website.

Avatar
1
1
Relevance

A real-time list of open tech roles

28 Jun 2024 Artificial Intelligence

Hey HN! Quick TL; DR: I set up scrapers for the websites of ~20k tech companies, fed the results through GPT, spent some sleepless nights building a web app, and voilá.I started this after getting fed up with how slow LinkedIn felt. I finally got it working and easy to use, so I wanted to share it with you guys.The job list is free. But I’m currently paying a lot of money to OpenAI for all the data I’m passing through GPT, so I may end up eventually charging for some advanced features, or putting ads on it or something.Please let me know what you think!I’m actively developing this so I’m happy to add requested features.Some technical details you may be curious about:- I began gathering the career pages for each company manually, but I ended up hiring a few Filipino contractors to speed up the process.- I run the web scrapers once per day.- I do the web scraping with Node.js. I use cheerio for basic web pages, and puppeteer for more complicated ones.- My OpenAI API dashboard tells me I process 7 million characters per day.- The list doesn’t encompass every single tech company (yet). I have most of the major ones (~20k), but the process of adding new companies is pretty labour intensive so it’s still ongoing.- The web app is built with React, and I got the infinite scrolling to be so smooth using a great library called React Query.For feature requests, please leave a comment here or email me at dorfmanjames@gmail.com

Avatar
5
5
Relevance

I scraped all MAAMA open positions and made them searchable/filterable

01 Dec 2023 Artificial Intelligence Hiring Career

Hey! I scraped all of Meta, Apple, Amazon, Microsoft and Alphabet's career pages and added them to a search tool I'm building.If you're interested in the dataset, let me know in the comments, I'd be happy to share it with you.Also, if you'd like me to keep this updated (or want me to send you new jobs that pass a certain criteria as they show up), send me an email to elias@payperrun.com

Avatar
1
1
Relevance

First2Apply – job scraping desktop app

10 Feb 2024 Hiring

Inspired by the latest tech layoffs and all the conversations about how bad the job market is, I decided to build a tool to solve a part of this problem.I’ve had this hypothesis that if a job has more than 200 applications it’s pretty much useless to apply anymore because they will probably find a candidate amongst those first few hundred.This is how I came up with the idea for https://first2apply.com/ an app that continuously scans job boards customised by the user and alerts when it finds a new opening. I built a prototype over one weekend to see if I could actually pull it off and then created a landing page and started spreading the word. Two weeks later there have been 200+ signups on the waiting list and the MVP is ready to be shipped.How it works? You go to a job site like LinkedIn, search for your desired role/tech stack, filter by country or remote and apply the last 24h filter. Then copy paste the URL in the app. The app will then periodically scrape the links you saved and check if there are new listings from last time and send a native desktop notification.It's also very easy to organise your job hunt when using multiple sites since you have one dashboard to check new entries from all of them and recently added a tab to save the ones where you applied in order to manage them more easily.I've been testing this with my wife who is currently looking for a job as a junior react dev and am starting to get some positive results after using it for a while: - https://imgur.com/a/LFOdAOD here is an example of a job from LinkedIn which got 50 applications in the first 12h and then they closed it. I think it's safe to assume they have enough to pick from- https://imgur.com/a/2Coi1KQ finally the holy grail, a job with no applications. Unfortunately it was for a Java position because even though you filter by React in LinkedIn they still show you jobs for other tech :/She has been using the app for the past 2 weeks and was able to find ~5 valid jobs that she could apply for. Unfortunately there are not a lot of companies hiring juniors. One other benefit that she noticed was that she doesn't have to constantly refresh her job sites, just wait for the notification to pop up and check the latest results. This way she has more time to spend working on her portfolio projects.So from my one beta tester, I have gotten positive feedback, but it's very much a textbook case of the Mom test. Looking forward to hear what other people think and hopefully this could help someone get that extra edge needed to land a job.

Users have requested a Linux version of the product. The product has also received positive feedback, with one user describing it as 'cool.'

The product lacks a Linux version, which limits its accessibility and usability for users who rely on this operating system.

Avatar
5
2
2
5
Relevance

Not everything needs a browser, job search from the command line

05 Mar 2024 Artificial Intelligence Hiring Productivity Career

While looking for engineering jobs, as a way to face the tedious tasks, and have fun while doing it, I’ve been building a command line tool to automate as much as possible of my job search processSo far this tool:* saves my resume * saves my preferences for job (eg. remote, rails, salary) * scrapes online job listings * uses GPT to match my resume, and preferences with the job listings * provides a list of AI-curated job matches with an explanation why the job is a good fit for me, as well as instructions on how to applyWould love some feedback. Let me know if you want to try it out

Users are interested in a tool that automates the job search process and are eager to see the code behind it.

Avatar
1
2
100.0%
2
1
100.0%
Relevance

Google for Jobs

01 Nov 2023 Hiring Career

Hi, build another side project in my free time.Basically gathering jobs from some top companies in the US mostly. So the jobs are directly from companies websites (not other job boards).Its not much data, only maybe 250k jobs right now from 100+ companies.Build it with scala/zio, scala.js/laminar and postgres 16You can also search with quotes, also can use minus, also underscore to match any character so you can search for 'part_time'Yeah enjoy.

Users reported a query delay with results appearing after 10 seconds. One user jokingly mentioned 'sex' as it's the internet.

The product lacks a loading spinner, which affects user experience by not indicating when the system is processing or loading content.

Avatar
2
2
2
2
Relevance

jobdata - Simple job data API

21 Jun 2023 Data API Career

Backfill your job board with our fast & simple API and access job post data in JSON format from thousands of companies hiring in the tech space across many different industries.

Users expressed interest in the product's concept and API, with one user praising the API and its potential. Concerns were raised about the project's sustainability based on membership. Some users requested a trial or preview of listings, while another inquired about the data collection process, specifically whether it involves scraping or manual outreach. Another user requested the ability to add user-added filters to the API.

Users express concerns about the product's sustainability if paid memberships aren't enough. They also request user-added filters for better search functionality. A lack of a free trial or preview, especially for Indian internship listings, is also a point of criticism.

Avatar
79
5
5
79
Relevance

Simple Job Data API

27 Feb 2024 Data Hiring

Backfill your job board or gather insights into the hiring market with our fast & simple API - access job post data in JSON format from thousands of companies hiring across many different industries.For my PH launch this way please: https://www.producthunt.com/posts/jobdataapi-com-2

Love it, suggest naming sources.

Name the sources.

Avatar
3
1
1
3
Relevance

jobdataapi.com - Simple Job Data API

27 Feb 2024 Data Hiring API

Backfill your job board or gather insights into the hiring market with our fast & simple API - access job post data in JSON format from thousands of companies hiring across many different industries.

Avatar
1
1
Relevance

Jobsath - We help you find your dream Job

07 Apr 2023 Developer Tools

Jobsath is a job portal that can help job seekers find their dream job. With thousands of job listings in various industries and locations, Jobsath makes it easy to search and apply for jobs that match your qualifications and experience.

The Product Hunt launch received mixed feedback. One user questioned the product's differentiation from Welcome To The Jungle. Another user reported a critical issue where all links redirected to the homepage, rendering the product non-functional. On a positive note, one user offered congratulations and expressed interest in connecting for networking on Twitter.

The primary criticism is that the product is non-functional due to redirection problems, rendering it unusable for users.

Avatar
67
3
-33.3%
-33.3%
3
67
Relevance

Personalized remote job recommendation using resume Analysis

07 Nov 2024 Hiring Career

Job-Scout is a command-line tool designed to simplify the search for remote Machine Learning jobs. It aggregates job listings from Twitter and Hacker News and uses natural language processing to match these postings to the skills and experience in your resume. With an emphasis on remote ML, AI, and Data Science roles, Job-Scout ranks and recommends jobs tailored to your qualifications. The tool is customizable, allowing users to adjust search queries for specific job roles or internships, making it ideal for both experienced professionals and students looking for personalized job matches.

Avatar
5
5
Relevance

A TUI to match you with Who's Hiring jobs

31 Oct 2024 Career

None of the web UIs for Who's Hiring made it significantly easier for me to find jobs that matched my skillset / experience / values, so I made a tool to score and sort jobs by my criteria. You can define your own as golang regexes.The tool works offline, and it exports to JSON for further processing.Untested on Windows but let me know your experiences. And hire me as a remote OSS linux dev / SRE architect :) Especially if you're in a humanitarian space.

Avatar
3
3
Relevance

A simple tool to help with job search via Google (and why) – Joseef

24 Apr 2024 Hiring Career

TL;DR: Google has all the data and allows searching several companies job portals at once. That's how I've found my current position. This tool helps to manage queries.Some time ago it became apparent to me that most job boards I know became aggregators — most companies have their own job portals or at least pages on their sites, and those aggregators most of the time just collect postings. Of course they try to add value to it, but sometimes they try too much (adding complex interface), they choose what companies to show to you and postings become stale.So I tried searching on companies' job boards directly with Google excluding middleman, as those portals often don't have search features, and found my current position!At that time I stored queries in my notebook. But recently I've realized that Google is even more powerful — it allows searching on several portals at once. With this, complexity of queries starts to grow and managing queries in the notebook becomes difficult — queries are too long and hard to edit (add or exclude portals).Initially I thought that it was too simple to be shared with others (anyone can Google), but when complexity started to grow I made this page and it looks like now it may be useful to others.Hope it helps you!

Avatar
2
2
Top