<p><img src="/assets/new-post-reaper.png" alt=""/></p>
<p>We&#39;ve had this blog for a while now and the typical flow of writing a post is me going to <a href="https://mark.reaper.im">Mark</a>, writing the post markdown, exporting the file, adding the metadata to the file which involves the title,publish status, date, and then pushing the repository after checking if the above were done properly. </p>
<p>Though Mark exists just because I&#39;ve used other tools and Typora is the only one that comes close to being lightweight and aesthetic and while I do use it while I&#39;m on the Mac, I do write a lot of these posts from an iPad and since Mark is just a web-app it works well, as for pushing the repo and creating the file, all is done using <a href="http://gitpod.io">gitpod</a>. It&#39;s pretty easy to do but yeah, a good amount of window switching.</p>
<h2 id="adding-integrations-to-mark">Adding Integrations to Mark</h2>
<p>I like how the new UI on it looks so my second plan was to add the ability to login via github on Mark, select a repository you&#39;d like to add the markdown too and then giving the path in the repository which would&#39;ve been great and I probably will do that sometime, but I wanted a little more automation since the meta data addition would still be needed and I wouldn&#39;t want to generalise mark to have datepicker when it&#39;s just for a niche use-case. Though I do have a plan for something similar so let&#39;s hope I get enough time this weekend to start with it.</p>
<h2 id="scraping-buymeacoffee">Scraping BuyMeACoffee</h2>
<p>The first approach was something I mentioned in <a href="https://reaper.im/blog/2021-02-22-Go-Lang-and-Web-Scraping">this post</a> which involved scraping post data from another site who&#39;s editor I liked. While that would work we&#39;d loose offline capabilities of the repository, which I didn&#39;t want too and adding a scheduled sync action wouldn&#39;t be optimal either. </p>
<h2 id="the-easiest-approach">The easiest approach</h2>
<p>The last approach was to just use a password to log into the site, add posts from a simple text area and then push it into the repository using github&#39;s API, though there were a few security risks. </p>
<ol>
<li>The password could be bruteforced.</li>
<li>The attacker could throw as many files as he wanted to my repo. </li>
<li>Obviously, he could post whatever he wanted</li>
</ol>
<p>So, we put a little more thought into it and ended up blocking this a little bit. The site uses an OTP approach instead, so it mails to one of my random non-public emails a otp that lasts for like 45-60 seconds, this kinda gets rid of the bruteforce but then it&#39;s just 6 digits, we&#39;ve got computers who can kinda get through this so the next block was to create all these posts to a subset branch and 
create a PR for the main branch. </p>
<p>This does 2 things. </p>
<ol>
<li>You cannot post directly to the deployed public version.</li>
<li>I&#39;m notified for the PR, so I&#39;d know if there was activity that wasn&#39;t from me.</li>
</ol>
<p>Again, there&#39;s still things that can be done that an attacker could do but a little consideration on blocking them for a while is better than leaving an open door. </p>
<p>Thus, the last post you saw was just me testing the whole flow after writing it all. Still got work in terms of security that could increase the friction for an attacker but I&#39;ve got other tools I need to work on so we&#39;ll get to it as soon as I get time. </p>
<p>Adios!
          </p>




![](/assets/new-post-reaper.png)

We've had this blog for a while now and the typical flow of writing a post is me going to [Mark](https://mark.reaper.im), writing the post markdown, exporting the file, adding the metadata to the file which involves the title,publish status, date, and then pushing the repository after checking if the above were done properly. 

Though Mark exists just because I've used other tools and Typora is the only one that comes close to being lightweight and aesthetic and while I do use it while I'm on the Mac, I do write a lot of these posts from an iPad and since Mark is just a web-app it works well, as for pushing the repo and creating the file, all is done using [gitpod](http://gitpod.io). It's pretty easy to do but yeah, a good amount of window switching.

## Adding Integrations to Mark 
I like how the new UI on it looks so my second plan was to add the ability to login via github on Mark, select a repository you'd like to add the markdown too and then giving the path in the repository which would've been great and I probably will do that sometime, but I wanted a little more automation since the meta data addition would still be needed and I wouldn't want to generalise mark to have datepicker when it's just for a niche use-case. Though I do have a plan for something similar so let's hope I get enough time this weekend to start with it.

## Scraping BuyMeACoffee
The first approach was something I mentioned in [this post](https://reaper.im/blog/2021-02-22-Go-Lang-and-Web-Scraping) which involved scraping post data from another site who's editor I liked. While that would work we'd loose offline capabilities of the repository, which I didn't want too and adding a scheduled sync action wouldn't be optimal either. 

## The easiest approach
The last approach was to just use a password to log into the site, add posts from a simple text area and then push it into the repository using github's API, though there were a few security risks. 
1. The password could be bruteforced.
2. The attacker could throw as many files as he wanted to my repo. 
3. Obviously, he could post whatever he wanted

So, we put a little more thought into it and ended up blocking this a little bit. The site uses an OTP approach instead, so it mails to one of my random non-public emails a otp that lasts for like 45-60 seconds, this kinda gets rid of the bruteforce but then it's just 6 digits, we've got computers who can kinda get through this so the next block was to create all these posts to a subset branch and 
create a PR for the main branch. 

This does 2 things. 
1. You cannot post directly to the deployed public version.
2. I'm notified for the PR, so I'd know if there was activity that wasn't from me.

Again, there's still things that can be done that an attacker could do but a little consideration on blocking them for a while is better than leaving an open door. 

Thus, the last post you saw was just me testing the whole flow after writing it all. Still got work in terms of security that could increase the friction for an attacker but I've got other tools I need to work on so we'll get to it as soon as I get time. 

Adios!
          

<p><a href="https://reaper.im">https://reaper.im</a> now has it&#39;s own inline editor for me to write blog posts with. </p>
<h3 id="supports-markdown">Supports markdown</h3>
<p>and obviously since you do see this post up on the blog right now, it all worked out well.
          </p>




[https://reaper.im](https://reaper.im) now has it's own inline editor for me to write blog posts with. 

### Supports markdown 
and obviously since you do see this post up on the blog right now, it all worked out well.
          

<p>Scraping websites is fun, but then I rarely ever tried doing it since setting up chrome and firefox to run headless on heroku is a daunting task and quite time taking in terms off testing. Which I did do when I made this <a href="games.siddharthgelera.com">Epic Games Online Store Scraper</a> and as you can see it&#39;s quite slow because it takes time to live scrape, and while I could&#39;ve had a database setup to store the prices the problem is I&#39;d have to run a sync every-time anyway because the store changes price and new free games are added at any point so a live scraper was the plan but then epic started changing the view structure every now and then and it became hard to maintain it and so it&#39;s now just there for me as a reference if I ever need to create a scraper using <code>Nodejs</code> and <code>puppeteer</code> again. </p>
<h2 id="why-talk-about-it-now">Why talk about it now?</h2>
<p>No reason to bring a project that&#39;s about a year old , into the picture now, right?
Well, If you know how this blog works internally, it&#39;s just a nextjs app that renders the markdown text into react components, I should shift to <code>mdx</code> though. I will be cleaning the code of the site soon, it&#39;s quite repetetive right now and should be refactored. </p>
<p>So, back to topic. The whole blog section is just 2 files of <code>.js</code> that renders content from <code>.md</code> files from the same repo and while I like the approach, since everything I write is in the same repo, it can be taken offline and you can run the blog locally if you like my posts that much.... doubt that. </p>
<p>But i think being able to take it offline is a big advantage, though I was going throught <a href="https://buymeacoffee.com">BuyMeACoffee.com</a> the other day and I remembered that I made 2 posts regarding my released projects there when this blog wasn&#39;t really deployed and I liked their editor. </p>
<p>This is where the scraping comes in, they do have exposed api&#39;s but it doesn&#39;t expose the user posts and quite a few people have asked for it so they might add it in the future releases though for now, we are going to scrape these posts I&#39;ve got there. </p>
<h2 id="you-wrote-a-scraper-with-go-why-should-i-care-">You wrote a scraper with Go, why should I care ?</h2>
<p>Ah, because it didn&#39;t need a huge ass browser to be run on heroku. </p>
<p>The whole </p>
<blockquote>
<p>Build everything with the language you wish to learn </p>
</blockquote>
<p>concept really works well for me and since we are on the Go Lang train from the past few months, I gave it a shot and used a browser package the go community offers called <a href="https://github.com/headzoo/surf">surf</a> and it&#39;s basically a programmatic browser , so it doesn&#39;t need a headless instance in the first place and acts like a browser. If i&#39;m not wrong someone would&#39;ve already made a testing suite using its api for web apps. The best part, it&#39;s a simple go lang package so it got compiled with the remaining program, so now I have a single binary file of a few MBs running on heroku with no other deps needed. </p>
<p>No linux setup required , no waiting for 20 more apt updates and packages to install so that chrome can run headless, one single binary and it&#39;s fast, like the current instance of the worker is running for free, so we have the 10 second wake up time from heroku but that&#39;s about it. It doesn&#39;t take any longer than that. </p>
<p>You can check the api by clicking on the link below. Obviously wait for the 10 seconds if the container has to cold start, but then your reloads will take no time, it spawns a browser, opens the url, scrapes and shows the results in under 2 seconds. I&#39;m impressed, there&#39;s no caching logic in that code either (should add though)</p>
<p><a href="https://bmc-api-worker.herokuapp.com/posts">Posts</a> - All Posts</p>
<p><a href="https://bmc-api-worker.herokuapp.com/posts/mailer-simple-e-mail-microservice">Post Data</a> - Example Post Data</p>
<p>and so I think I&#39;m going to do a lot more scraping now since it was a breeze to setup, use and quite performant without any direct optimizations that I&#39;ve made. </p>
<h2 id="you-forgot-to-tell-what-youre-going-to-do-with-the-scraper">You forgot to tell what you&#39;re going to do with the scraper.</h2>
<p>Oh yeah, so now since I have the data from there, I realised that their post engine is kinda limited in terms of markdown recognition and also, directly reading from the API conflicts with my thinking of offline availability, so I plan on running a GitHub action every 3-4 days to go through this worker, get the data, create a markdown from the received html data and save it as a file into the repo if it doesn&#39;t already have the post. This sticks to the offline thinking for someone who&#39;d like to be able to go through them offline and also I get to use the BuyMeACoffee&#39;s editor to make decent posts, won&#39;t use it for the posts that need code snippets etc, cause that won&#39;t work with their current editor.</p>
<p>That&#39;s it for now. </p>
<p>Adios!</p>



Scraping websites is fun, but then I rarely ever tried doing it since setting up chrome and firefox to run headless on heroku is a daunting task and quite time taking in terms off testing. Which I did do when I made this [Epic Games Online Store Scraper](games.siddharthgelera.com) and as you can see it's quite slow because it takes time to live scrape, and while I could've had a database setup to store the prices the problem is I'd have to run a sync every-time anyway because the store changes price and new free games are added at any point so a live scraper was the plan but then epic started changing the view structure every now and then and it became hard to maintain it and so it's now just there for me as a reference if I ever need to create a scraper using `Nodejs` and `puppeteer` again. 



## Why talk about it now? 

No reason to bring a project that's about a year old , into the picture now, right?
Well, If you know how this blog works internally, it's just a nextjs app that renders the markdown text into react components, I should shift to `mdx` though. I will be cleaning the code of the site soon, it's quite repetetive right now and should be refactored. 

So, back to topic. The whole blog section is just 2 files of `.js` that renders content from `.md` files from the same repo and while I like the approach, since everything I write is in the same repo, it can be taken offline and you can run the blog locally if you like my posts that much.... doubt that. 

But i think being able to take it offline is a big advantage, though I was going throught [BuyMeACoffee.com](https://buymeacoffee.com) the other day and I remembered that I made 2 posts regarding my released projects there when this blog wasn't really deployed and I liked their editor. 

This is where the scraping comes in, they do have exposed api's but it doesn't expose the user posts and quite a few people have asked for it so they might add it in the future releases though for now, we are going to scrape these posts I've got there. 

## You wrote a scraper with Go, why should I care ?

Ah, because it didn't need a huge ass browser to be run on heroku. 

The whole 

> Build everything with the language you wish to learn 

concept really works well for me and since we are on the Go Lang train from the past few months, I gave it a shot and used a browser package the go community offers called [surf](https://github.com/headzoo/surf) and it's basically a programmatic browser , so it doesn't need a headless instance in the first place and acts like a browser. If i'm not wrong someone would've already made a testing suite using its api for web apps. The best part, it's a simple go lang package so it got compiled with the remaining program, so now I have a single binary file of a few MBs running on heroku with no other deps needed. 

No linux setup required , no waiting for 20 more apt updates and packages to install so that chrome can run headless, one single binary and it's fast, like the current instance of the worker is running for free, so we have the 10 second wake up time from heroku but that's about it. It doesn't take any longer than that. 

You can check the api by clicking on the link below. Obviously wait for the 10 seconds if the container has to cold start, but then your reloads will take no time, it spawns a browser, opens the url, scrapes and shows the results in under 2 seconds. I'm impressed, there's no caching logic in that code either (should add though)

[Posts](https://bmc-api-worker.herokuapp.com/posts) - All Posts

[Post Data](https://bmc-api-worker.herokuapp.com/posts/mailer-simple-e-mail-microservice) - Example Post Data

and so I think I'm going to do a lot more scraping now since it was a breeze to setup, use and quite performant without any direct optimizations that I've made. 



## You forgot to tell what you're going to do with the scraper.

Oh yeah, so now since I have the data from there, I realised that their post engine is kinda limited in terms of markdown recognition and also, directly reading from the API conflicts with my thinking of offline availability, so I plan on running a GitHub action every 3-4 days to go through this worker, get the data, create a markdown from the received html data and save it as a file into the repo if it doesn't already have the post. This sticks to the offline thinking for someone who'd like to be able to go through them offline and also I get to use the BuyMeACoffee's editor to make decent posts, won't use it for the posts that need code snippets etc, cause that won't work with their current editor.

That's it for now. 

Adios!

If you're seeing this, it all worked!

Supports markdown