What We Learned By Building A LinkedIn ScraperWhat We Learned By Building A LinkedIn Scraper
⚙️
What We Learned By Building A LinkedIn Scraper
By: Alexander V. Pavlovcik
notion imagenotion image
User friendly, straight-forward scraping.
Several months ago, we unveiled our LinkedIn Scraper Chrome Extension. So far, we’ve had phenomenal success. I’d like to talk about the timeline behind this project, the history behind this tool and how Dawson and I came together as a team, what made it so successful, and where we’re going with this project.
The need for a LinkedIn scraper started when one of my clients needed help with B2B marketing and lead generation. He wanted to target a very specific niche — Orthopedic Surgeons who had their own practices in major cities around the United States, with a staff of less than 50 employees.
The goal was to run a cold email campaign, and I knew that I wanted to source the leads from LinkedIn Sales Navigator (since it’s very easy to narrow down the targeting on that platform).I needed to find out how to scrape LinkedIn profiles, where I would get all of the users’ public information, so that I could then pass that information through an enrichment tool (such as Clearbit) to get a hold of their real email addresses.
You can read more about the background story here in the initial blog post.
As I continued refining the code, I also started joining marketing-focused Facebook groups as I became fascinated by the marketing side of things, in particular, the email marketing. I started talking to people in these groups to learn more about this stuff, and I was quickly introduced to Dawson.
Never underestimate the power of Facebook groups. You can meet some really cool and interesting people — spending time on groups can really pay dividends.
When I initially approached him, we were talking about a lot of marketing stuff back and forth. Dawson runs a B2B outreach agency, so cold email marketing is extremely valuable to him. It’s his bread and butter. He had experience using a ton of LinkedIn scrapers — scraping millions of emails to see what works and what doesn’t. In one year alone, he spent over $40,000 testing various tools and databases (source, him). This experience gave him a great understanding of how all these tools worked from a user’s point of view.
Pictured: Dawson's MoneyPictured: Dawson's Money
Pictured: Dawson's Money
He was excited at the idea of a LinkedIn scraper. So about five months ago Dawson and I started working together with one goal in mind: to launch the best, most affordable LinkedIn scraping tool on the market.
The months leading up to our official launch came with its fair share of learning experiences. We learned what works, and what doesn’t through multiple iterations.
The first iteration of this tool was basically just a raw data dump of everything on the profile. I have some experience with building scrapers, and most traditional methods of doing it are brittle. For example, your basic scraper will use precise CSS selectors. The problem is that as soon as the page layout is updated a bit, the selector system breaks, and so does the scraper.
When we were looking to bring this tool to market, the first change we made was altering the code to avoid highly specific CSS selectors, so that it wouldn’t break down the line when we had users. At the start of this year, LinkedIn changed part of its code which actually made scraping easier and was a big help for us. We were able to build a simple algorithm to take advantage of this and created a fairly robust method of scraping. From an engineering standpoint, this really sets our scraper apart from others.
Soon afterward, we started beta testing, and it’s worked out really well for us. What really helped make this a big success was my experience with ICOs when I was helping companies launch cryptocurrencies. What I learned is that the success of a coin (and especially with raising capital) was mostly determined by community building. All the successful projects had really big telegram and Facebook groups that were active and engaged.
An engaged audience is a very attractive thing to have for a startup. You want users involved with your software and recommending it to other users. Doing this also helps you build a knowledge base where people’s questions can be answered directly by the founders & active users.So we started all these groups and began doing a bit of growth hacking. We also made videos explaining the software, and that’s where we got our first users from.
After talking with our users, we noticed that the tool was far too complicated, and you had to have a fair bit of technical knowledge to know how to use it. We quickly realized that this was a significant barrier to people using our software, so we rebuilt it from the ground up so that it was easier to get access and use.
What we learned from our growth hacking efforts is that you need to prepare for a massive influx of users and beta testers and be ready ahead of time. You also want to skip the manual on boarding process and automate it as much as possible. This is important if you ever reach the point where you’re adding hundreds of users every day, and allows you to spend time actually speaking with your users.
What we also learned is that you’re allowed to make mistakes, and the quicker you can make them, the better. Getting something out to market that works as soon as you can is far more important. To quote Ried Hoffman from LinkedIn, “If you’re not embarrassed by the first version of your product, then you’ve launched too late.”
It doesn’t matter if the product is a little rough around the edges; you can fix things much more quickly if you have a large amount of users providing feedback. As long as it does what you say it does, your beta testers will happily point out where you can improve.
After we ran our beta testing for roughly 2 months, we decided to Saasify it and scale things up.Saasify is a brilliant front-end platform that allows you to quickly deploy a SaaS product to market, and this really helped bring our tool out of the beta stage and into Version 1.
Based on our experience and our users’ feedback — every competitor in this space will limit how many profiles you could scrape per month, and if you’d like more than a few thousand, you’d be paying well over $100/month. They try to justify this pricing by guessing the email addresses of the profiles that they scrape.
Competitor ScreenshotCompetitor Screenshot
Competitor Screenshot
When scraping profiles that have included accolades/titles in their LinkedIn username, it becomes very apparent why “guessing” emails is an unreliable method.
If you’ve ever used one of these tools, then tried to send out to these guessed email addresses, you’ll know that you’re better off deleting the email that was “scraped” and running your list through an enrichment tool (like Clearbit).
When we launched, we wanted to reward our beta testers and early adopters — so we offered our Forerunner package. It was $49/month for unlimited scraping, and as an early adopter — that price would never increase even when new features roll out, and they would never lose access to unlimited scraping.
Do we offer fake emails like our competitors? No. Would we gain more initial users if we provided fake emails? Possibly — But we would rather have happy long-term users that can rely on us for quality.
Our next steps with this platform is to integrate directly with multiple enrichment tools to provide our users with real email addresses — bridging the gap between scraping and enrichment.
notion imagenotion image
Easy exporting of the profiles that you’ve scraped
We launched with this offer June 2020, and we have gained over 300 users at this point — so I guess you could say it‘s been a hit.
Looking back on the early stages of this project, it’s hard to say that we would do anything differently. It’s been a great ride bringing this LinkedIn scraper to market — however we understand that there is a long road ahead of us, and we’ve barely left the front porch.
notion imagenotion image