Web Scraping with R

The CodingGirls organised a 3-hour Web Scraping with R session. The speaker of the event was Pang Long. He is a good, dedicated and attentive speaker who can explain well throughout the event. It is important for me because I left R programming for about one year and joined this event to recap some of the R programming. The class is not for beginner who wants to learn R.

The topic, Web Scraping with R is very interesting after attended the event.

When we browse a website, sometimes we wish to download some of the information from the webpage. It can be done by saving the page with .html format. However, if you want to just list out the books list from the website, for example, instead of saving the entire webpage and slowly extract one by one, R can help you to extract it by identifying and exploiting patterns in raw html source codes and manipulate it into usable format. So, a basic HTML knowledge is important here.

Throughout the session, the speaker used this link as an example to explain how to do it, https://rpubs.com/ryanthomas/webscraping-with-rvest. It is written by Ryan Thomas.

Prerequisite:
1. Install rvest
2. Install magrittr

Just in case you do not know how to install a package from RStudio, look for the Package tab on the right side bar as below. There is a Install icon in which we can click and fill up the package name to install packages we want to install. If you are more familiar with syntax, you can always use,

install.packages("rvest")
install.packages("magrittr")

After the packages installed, we began to view the website. In the above link, the author gave us a project link which we can do a right-click and view the raw HTML codes. It works for HTML codes only.

Before that, we were introduced to some of the powerful rvest functions which we used throughout the session.

html_nodes(): identifies HTML wrappers.
html_nodes(".class"): calls node based on css class
html_nodes("#id"): calls node based on <div> id
html_nodes(xpath="xpath"): calls node based on xpath (we’ll cover this later)
html_attrs(): identifies attributes (useful for debugging)
html_table(): turns HTML tables into data frames
html_text(): strips the HTML tags and extracts only the text

It is important to tell R where to look for the information on the webpage based on the HTML tags.

Other functions which are useful,

gsub("^\\s+|\\s+$", "", .) %>% #strip the white space from the beginning and end of a string.
print(paste0("geting data for page: " , page ))
  URL <- paste0("https://scistarter.com/finder?phrase=&lat=&lng=&activity=&topic=&search_filters=&search_audience=&page=", page, "#view-projects")

The output can be written into csv file.

write.csv(co.names, "RE100_2016.csv", row.names=F)

Goodbye My Past

There was many minor versions of myself before and after I resigned from my last position. Now, it is going to be a complete new “product” of myself and no further support for the older version.

What does it mean?
I made a decision and took a drastic change in my life recently before I embarked my new journey. The decision was made upon hearing a message from an ex-colleague. The message contents was shared by a closed source with details of the people involved. Surprisingly, this person shared out what has been done and subsequently, it reached my ears one day. Yes, I do believe this person will do such thing.

Myself as a person who believe in Karma, what goes around, comes around. It does not require me to do anything as what the below image said.

Without further verification on the matter, I removed all the people related to this person, be it in the past or present from all my contact lists. I refrained myself from sharing information or updates on Facebook and Instagram recentlly. I stopped talking with all these people that I knew from the past. Ending the past relationships brought me into a brand new life whereby I am more openly to accept new people and things. Now, the past has become less concerns and let go more easily.

More importantly, I look each day and everything positively. Only with a positive mindset, things will sail smoothly with God’s blessing.

First Thought

I have no intention of sharing the details of my personal life. However, I wish to share some of the great things which happened to me. It happened back in late February 2017 when I received an email asked for my availability for a job interview with an airline company in Singapore. I was very excited back then. I wrote down in a piece of paper of all the possibilities of working for an airline company as well as my daily routine. Besides that, my ex-colleague contacted me and said their CEO was looking for me and wanted to treat me a meal.

I read on Susan’s monthly astrology and the horoscope said there would be a great career opportunity coming to my way. I thought the good things were coming to my way. I was pretty sure this airline company was the one it meant. I shared with my cousin. Since, she believes in Jesus, she quoted some of the Psalms and ask me to meditate throughout my interview process.

On the interview day, I received another email from a ride-hailing company asked me for an interview on the next day. Then, I have a strong belief that my next destination would be in a transportation industry. I applied both jobs and did the tests sometimes before they contacted me for an interview.

With full of confidence, the interviews went on smoothly, however, the interview with the ride-hailing company did not go thoroughly the interviewer was looking for someone to work in Product Services while I told them my preferred field was Data Engineering. In the end, I did not get the job offer.

Then, I put my attention to the airline company with great hope they would contact me for updates meanwhile I went back to Malaysia to wait for the updates. I need to have some buffer for my two marathons in late March and early April, therefore, I could not stay too long Singapore.

Just before I went back to Malaysia, I was being contacted by another ex-colleague asking for a meet up with the new IT manager. Instantly, I guessed both the IT manager and CEO looking for me for a same reason because I knew both of them did not know me. I contacted my source in Thailand and I was told that it was the IT Director who suggested me to both of them. I went to meet up with the new IT manager and the team. Unfortunately, the IT manager came unprepared and gave me chances to turn down them. I informed the Thailand office and apologized. 

I took up a short introduction course of Cassandra but I did not complete it yet. I am kind of demotivated as there is no one pushing me to complete it and I was getting more worried by then.

I did a tarot cards reading. It was really surprised me, it said I would not able to get what I wanted because I did not let go and open up for new opportunity. I felt so shocking especially it said against the horoscope. The tarot reading advised me to meditate and I did as below.

I seek to destroy whatever illusions have been influencing my life and rebuild from a position of strength. I seek to place my life upon form and positive foundations. I seek to align my hopes and ambitions with higher spiritual purpose.

So, was there any news after that?

Yes, there was but not from the airline company. There was a guy from the F&B company contacted me and wanted to meet up for a discussion. On the 24th March, I went back to Singapore for two marathons, the Sundown and 2XU and met up with two friends too.

Since, I was at Bugis for the meetup with the F&B Business Development guy, I stayed back and detoured to the nearby temple to make my prayer. I received a call and asked me to go for the interview the next day. I was excited to receive the good news. It gave me a hope after waiting so long.

On the interview day, my interviewer was not in and replaced by two other interviewers. After some brief talked, one of them started to test me verbally. Lastly, it ended with designing the database structure and did a query based on their questions.

Throughout the interview, I really felt nervous and kept citing the Psalm 23, even after the interview. I tried not to think so much of the outcomes. On my way back home, I received a call and asked me to go to the office again to meet the hiring manager. During the conversation, there was a salary negotiation took place.

After the phone call, I quickly went home and checked Susan’s astrology again. Whatever mentioned in the website, it was 90% accurate, that included, the career growth, job opportunity, salary negotiation and timeline. The last part was about the working permit which was not discussed throughout the interview. Confidently, I got a feeling I would get the job pretty soon.

On my last time of interview, I was there on time but I was waiting for the hiring manager to finish the meeting. Upon the meeting over, I was invited to the room and for the first time, I saw my hiring manager. There was no further testing, thanked God!

We briefly talked and I shared my working experience and background. It seemed like there would be a great learning opportunity and they did not mind that I lack of experience in this field.

At the same time, the hiring manager offered me the job. Truly, I did look surprised although I was confident and I knew the outcome. Not only that, the hiring manager invited the CEO to join the interview as well as getting the CEO’s approval. I believe the CEO is a technical person, from our conversation. They introduced me to their CTO who is another technical guy through the conversation. Positively, I knew it was God’s arrangement. The best thing God gives to me. Thank you God, Thank you Lord, for all the goodwill. I will treasure them as much as I learn and grow along with God’s and Lord’s blessings.