<p>Scraping websites is fun, but then I rarely ever tried doing it since setting up chrome and firefox to run headless on heroku is a daunting task and quite time taking in terms off testing. Which I did do when I made this <a href="games.siddharthgelera.com">Epic Games Online Store Scraper</a> and as you can see it&#39;s quite slow because it takes time to live scrape, and while I could&#39;ve had a database setup to store the prices the problem is I&#39;d have to run a sync every-time anyway because the store changes price and new free games are added at any point so a live scraper was the plan but then epic started changing the view structure every now and then and it became hard to maintain it and so it&#39;s now just there for me as a reference if I ever need to create a scraper using <code>Nodejs</code> and <code>puppeteer</code> again. </p>
<h2 id="why-talk-about-it-now">Why talk about it now?</h2>
<p>No reason to bring a project that&#39;s about a year old , into the picture now, right?
Well, If you know how this blog works internally, it&#39;s just a nextjs app that renders the markdown text into react components, I should shift to <code>mdx</code> though. I will be cleaning the code of the site soon, it&#39;s quite repetetive right now and should be refactored. </p>
<p>So, back to topic. The whole blog section is just 2 files of <code>.js</code> that renders content from <code>.md</code> files from the same repo and while I like the approach, since everything I write is in the same repo, it can be taken offline and you can run the blog locally if you like my posts that much.... doubt that. </p>
<p>But i think being able to take it offline is a big advantage, though I was going throught <a href="https://buymeacoffee.com">BuyMeACoffee.com</a> the other day and I remembered that I made 2 posts regarding my released projects there when this blog wasn&#39;t really deployed and I liked their editor. </p>
<p>This is where the scraping comes in, they do have exposed api&#39;s but it doesn&#39;t expose the user posts and quite a few people have asked for it so they might add it in the future releases though for now, we are going to scrape these posts I&#39;ve got there. </p>
<h2 id="you-wrote-a-scraper-with-go-why-should-i-care-">You wrote a scraper with Go, why should I care ?</h2>
<p>Ah, because it didn&#39;t need a huge ass browser to be run on heroku. </p>
<p>The whole </p>
<blockquote>
<p>Build everything with the language you wish to learn </p>
</blockquote>
<p>concept really works well for me and since we are on the Go Lang train from the past few months, I gave it a shot and used a browser package the go community offers called <a href="https://github.com/headzoo/surf">surf</a> and it&#39;s basically a programmatic browser , so it doesn&#39;t need a headless instance in the first place and acts like a browser. If i&#39;m not wrong someone would&#39;ve already made a testing suite using its api for web apps. The best part, it&#39;s a simple go lang package so it got compiled with the remaining program, so now I have a single binary file of a few MBs running on heroku with no other deps needed. </p>
<p>No linux setup required , no waiting for 20 more apt updates and packages to install so that chrome can run headless, one single binary and it&#39;s fast, like the current instance of the worker is running for free, so we have the 10 second wake up time from heroku but that&#39;s about it. It doesn&#39;t take any longer than that. </p>
<p>You can check the api by clicking on the link below. Obviously wait for the 10 seconds if the container has to cold start, but then your reloads will take no time, it spawns a browser, opens the url, scrapes and shows the results in under 2 seconds. I&#39;m impressed, there&#39;s no caching logic in that code either (should add though)</p>
<p><a href="https://bmc-api-worker.herokuapp.com/posts">Posts</a> - All Posts</p>
<p><a href="https://bmc-api-worker.herokuapp.com/posts/mailer-simple-e-mail-microservice">Post Data</a> - Example Post Data</p>
<p>and so I think I&#39;m going to do a lot more scraping now since it was a breeze to setup, use and quite performant without any direct optimizations that I&#39;ve made. </p>
<h2 id="you-forgot-to-tell-what-youre-going-to-do-with-the-scraper">You forgot to tell what you&#39;re going to do with the scraper.</h2>
<p>Oh yeah, so now since I have the data from there, I realised that their post engine is kinda limited in terms of markdown recognition and also, directly reading from the API conflicts with my thinking of offline availability, so I plan on running a GitHub action every 3-4 days to go through this worker, get the data, create a markdown from the received html data and save it as a file into the repo if it doesn&#39;t already have the post. This sticks to the offline thinking for someone who&#39;d like to be able to go through them offline and also I get to use the BuyMeACoffee&#39;s editor to make decent posts, won&#39;t use it for the posts that need code snippets etc, cause that won&#39;t work with their current editor.</p>
<p>That&#39;s it for now. </p>
<p>Adios!</p>



Scraping websites is fun, but then I rarely ever tried doing it since setting up chrome and firefox to run headless on heroku is a daunting task and quite time taking in terms off testing. Which I did do when I made this [Epic Games Online Store Scraper](games.siddharthgelera.com) and as you can see it's quite slow because it takes time to live scrape, and while I could've had a database setup to store the prices the problem is I'd have to run a sync every-time anyway because the store changes price and new free games are added at any point so a live scraper was the plan but then epic started changing the view structure every now and then and it became hard to maintain it and so it's now just there for me as a reference if I ever need to create a scraper using `Nodejs` and `puppeteer` again. 



## Why talk about it now? 

No reason to bring a project that's about a year old , into the picture now, right?
Well, If you know how this blog works internally, it's just a nextjs app that renders the markdown text into react components, I should shift to `mdx` though. I will be cleaning the code of the site soon, it's quite repetetive right now and should be refactored. 

So, back to topic. The whole blog section is just 2 files of `.js` that renders content from `.md` files from the same repo and while I like the approach, since everything I write is in the same repo, it can be taken offline and you can run the blog locally if you like my posts that much.... doubt that. 

But i think being able to take it offline is a big advantage, though I was going throught [BuyMeACoffee.com](https://buymeacoffee.com) the other day and I remembered that I made 2 posts regarding my released projects there when this blog wasn't really deployed and I liked their editor. 

This is where the scraping comes in, they do have exposed api's but it doesn't expose the user posts and quite a few people have asked for it so they might add it in the future releases though for now, we are going to scrape these posts I've got there. 

## You wrote a scraper with Go, why should I care ?

Ah, because it didn't need a huge ass browser to be run on heroku. 

The whole 

> Build everything with the language you wish to learn 

concept really works well for me and since we are on the Go Lang train from the past few months, I gave it a shot and used a browser package the go community offers called [surf](https://github.com/headzoo/surf) and it's basically a programmatic browser , so it doesn't need a headless instance in the first place and acts like a browser. If i'm not wrong someone would've already made a testing suite using its api for web apps. The best part, it's a simple go lang package so it got compiled with the remaining program, so now I have a single binary file of a few MBs running on heroku with no other deps needed. 

No linux setup required , no waiting for 20 more apt updates and packages to install so that chrome can run headless, one single binary and it's fast, like the current instance of the worker is running for free, so we have the 10 second wake up time from heroku but that's about it. It doesn't take any longer than that. 

You can check the api by clicking on the link below. Obviously wait for the 10 seconds if the container has to cold start, but then your reloads will take no time, it spawns a browser, opens the url, scrapes and shows the results in under 2 seconds. I'm impressed, there's no caching logic in that code either (should add though)

[Posts](https://bmc-api-worker.herokuapp.com/posts) - All Posts

[Post Data](https://bmc-api-worker.herokuapp.com/posts/mailer-simple-e-mail-microservice) - Example Post Data

and so I think I'm going to do a lot more scraping now since it was a breeze to setup, use and quite performant without any direct optimizations that I've made. 



## You forgot to tell what you're going to do with the scraper.

Oh yeah, so now since I have the data from there, I realised that their post engine is kinda limited in terms of markdown recognition and also, directly reading from the API conflicts with my thinking of offline availability, so I plan on running a GitHub action every 3-4 days to go through this worker, get the data, create a markdown from the received html data and save it as a file into the repo if it doesn't already have the post. This sticks to the offline thinking for someone who'd like to be able to go through them offline and also I get to use the BuyMeACoffee's editor to make decent posts, won't use it for the posts that need code snippets etc, cause that won't work with their current editor.

That's it for now. 

Adios!

<ul>
<li class="checkbox-list"><input type="checkbox" ></input> Create an ec2 instance (obviously,you braindamaged alien) </li>
<li class="checkbox-list"><input type="checkbox" ></input> Enable SSH (random port) and HTTP 80 on the instance.</li>
<li class="checkbox-list"><input type="checkbox" ></input> Increase the ssd/EBS size to a min of (20/40) Gigs</li>
<li class="checkbox-list"><input type="checkbox" ></input> Add a swap file to the system</li>
<li class="checkbox-list"><input type="checkbox" ></input> Install dokku</li>
<li class="checkbox-list"><input type="checkbox" ></input> Write a script for the dokku process for the current app (or use <a href="https://github.com/barelyhuman/dokcli">dokcli</a>, if you&#39;re completed it)</li>
<li class="checkbox-list"><input type="checkbox" ></input> Create a Dynamic Floating IP</li>
<li class="checkbox-list"><input type="checkbox" ></input> Map needed domains to Floating IP</li>
<li class="checkbox-list"><input type="checkbox" ></input> Get Repo, configure Repo to deploy to dokku </li>
<li class="checkbox-list"><input type="checkbox" ></input> Refill Cup of Coffee.</li>
</ul>



- [ ] Create an ec2 instance (obviously,you braindamaged alien) 
- [ ] Enable SSH (random port) and HTTP 80 on the instance.
- [ ] Increase the ssd/EBS size to a min of (20/40) Gigs
- [ ] Add a swap file to the system
- [ ] Install dokku
- [ ] Write a script for the dokku process for the current app (or use [dokcli](https://github.com/barelyhuman/dokcli), if you're completed it)
- [ ] Create a Dynamic Floating IP
- [ ] Map needed domains to Floating IP
- [ ] Get Repo, configure Repo to deploy to dokku 
- [ ] Refill Cup of Coffee.

<p>I&#39;ve written quite a few mini tools</p>
<ul>
<li><a href="https://hen.reaper.im">Hen</a> - Interactive React Playground </li>
<li><a href="https:/format.reaper.im">Format</a> - A simple in browser code formatter</li>
<li><a href="https://mark.reaper.im">Mark</a> - A markdown editor (cause there weren&#39;t already 1000&#39;s of these online)</li>
<li><a href="https://github.com/barelyhuman/commitlog">commitlog</a> - Change-log generator</li>
</ul>
<p>There&#39;s a few more but we&#39;ll limit the show off to this much for now. </p>
<p>Which one am I proud of the most? None actually. </p>
<p>They were all ideas that already existed all over the web, things already built by various developers and in a much better form than I did. The reason for building them was pretty simple, to learn how they worked internally, I&#39;ve re-invented the wheel quite a few times in these cases just to understand what&#39;s going on behind the scenes and that&#39;s all they were for. </p>
<p>On the other hand, while still not proud of it, there&#39;s one specific web-app that I use a lot. It&#39;s like google for me at this point, I open this web-app first, look for what I need and then redirect myself to the needed solution accordingly. </p>
<h2 id="enough-with-the-mystery-what-is-it">Enough with the mystery, what is it!?</h2>
<p>The same website that you are reading this post on. This is the repository that has had the most commits till date and is something that I should just turn into a homepage at this point since I do end up on it every now and then anyway. </p>
<h3 id="power-menu">Power Menu</h3>
<p>The best addition to the website was the power menu or command palette that I added a few days back, you can trigger it right now by pressing CMD/Ctrl + k , sadly only possible through a physical keyboard but it opens up a simple suggestion menu that can be used to browse through the main site, added this because I avoid using the mouse as much as I can but then I don&#39;t have keyboard support on my own website, ironic much? </p>
<h3 id="collections">Collections</h3>
<p>The other addition is the collections part on the website, I keep snippets, checklists and other lists that I might need to browse through for various stuff, the blog itself has a few posts that work as reference for various issues that I&#39;ve had or other idiotic decisions I&#39;ve made while developing stuff and it works well in terms that I can browse through them when needed. </p>
<h2 id="big-deal-ive-seen-even-better-websites">Big Deal, I&#39;ve seen even better websites!</h2>
<p>Oh , definitely! I&#39;ve seen websites that are even more minimal , for example checkout <a href="https://leo.im/">Leo</a> from Vercel&#39;s Dev team, his website is really really simple and uses external tools for basically everything, but then I don&#39;t use social media much so I&#39;m limited to having everything on this instead of using Twitter as a micro blog, his github repo&#39;s speak for how much better of a developer he is. </p>
<p>I show off a little bit cause I&#39;m not as good, so the marketing is kinda needed.</p>
<p>On the more functional and beautiful side, there&#39;s websites from 
<a href="https://sindresorhus.com/">Sindre Sorhus</a>
<a href="https://paco.sh">Paco Coursey</a> - His website has no nav, it&#39;s all power menu based </p>
<p>That&#39;s not all of them and obviously, there&#39;s tons of more devs that are better than me, I wouldn&#39;t mind if one of them became a mentor of mine though the point of this specific post was about the tool I used the most.</p>
<p>Final words, not just for me , but I think every dev should have a good website which they build with a simple thinking that they are going to use it more than someone else, if you do build it with that thinking, chances are you&#39;ll add a lot of useful things to the website that others might just enjoy as well. </p>
<p>On that note, I should probably style the power menu to look a lot better, it&#39;s super flat like the rest of the site which is nice but then there&#39;s no depth even during actions and that&#39;s not nice. Any frontend dev that has any ideas regarding the power menu and the remaining website&#39;s style guide, let me know or send through a pull request.</p>
<p>That&#39;s about it for now. 
Adios!</p>




I've written quite a few mini tools

- [Hen](https://hen.reaper.im) - Interactive React Playground 
- [Format](https:/format.reaper.im) - A simple in browser code formatter
- [Mark](https://mark.reaper.im) - A markdown editor (cause there weren't already 1000's of these online)
- [commitlog](https://github.com/barelyhuman/commitlog) - Change-log generator

There's a few more but we'll limit the show off to this much for now. 

Which one am I proud of the most? None actually. 

They were all ideas that already existed all over the web, things already built by various developers and in a much better form than I did. The reason for building them was pretty simple, to learn how they worked internally, I've re-invented the wheel quite a few times in these cases just to understand what's going on behind the scenes and that's all they were for. 

On the other hand, while still not proud of it, there's one specific web-app that I use a lot. It's like google for me at this point, I open this web-app first, look for what I need and then redirect myself to the needed solution accordingly. 


## Enough with the mystery, what is it!?

The same website that you are reading this post on. This is the repository that has had the most commits till date and is something that I should just turn into a homepage at this point since I do end up on it every now and then anyway. 

### Power Menu
The best addition to the website was the power menu or command palette that I added a few days back, you can trigger it right now by pressing CMD/Ctrl + k , sadly only possible through a physical keyboard but it opens up a simple suggestion menu that can be used to browse through the main site, added this because I avoid using the mouse as much as I can but then I don't have keyboard support on my own website, ironic much? 

### Collections
The other addition is the collections part on the website, I keep snippets, checklists and other lists that I might need to browse through for various stuff, the blog itself has a few posts that work as reference for various issues that I've had or other idiotic decisions I've made while developing stuff and it works well in terms that I can browse through them when needed. 

## Big Deal, I've seen even better websites!
Oh , definitely! I've seen websites that are even more minimal , for example checkout [Leo](https://leo.im/) from Vercel's Dev team, his website is really really simple and uses external tools for basically everything, but then I don't use social media much so I'm limited to having everything on this instead of using Twitter as a micro blog, his github repo's speak for how much better of a developer he is. 

I show off a little bit cause I'm not as good, so the marketing is kinda needed.

On the more functional and beautiful side, there's websites from 
[Sindre Sorhus](https://sindresorhus.com/)
[Paco Coursey](https://paco.sh) - His website has no nav, it's all power menu based 

That's not all of them and obviously, there's tons of more devs that are better than me, I wouldn't mind if one of them became a mentor of mine though the point of this specific post was about the tool I used the most.

Final words, not just for me , but I think every dev should have a good website which they build with a simple thinking that they are going to use it more than someone else, if you do build it with that thinking, chances are you'll add a lot of useful things to the website that others might just enjoy as well. 

On that note, I should probably style the power menu to look a lot better, it's super flat like the rest of the site which is nice but then there's no depth even during actions and that's not nice. Any frontend dev that has any ideas regarding the power menu and the remaining website's style guide, let me know or send through a pull request.

That's about it for now. 
Adios!


Checklist for EC2 Base Setup