With large data comes great responsilibilites

Working with data as always fascinated me. I worked in a financial services company first and was part of the transaction process group. It was amazing how each transaction data was process by multiple teams and the level of value the data had. Each team would process the same data in different ways and the value the data had for every group was different. Understood the various data processing carried out on the data in much better way. Why the data is cleansed, formatted, how the integrated of the data mattered a lot, how a mistake in processing the data can influence the business. Enjoyed this experience of learning data processing in a real corporate environment. Worked in a wide range of database technologies like DB2, Oracle DB and MS SQL. The challenge of understanding a data driven application was very interesting. The value of creating a conceptual schema, relational modeling and designing a relational database with all the integrity constraints made sense.

Now as part of my master’s course work, have stepped into a much challenging atmosphere; managing the un-structural data. The days where knowing your data structure before modeling the environment which manages it as slowly being changed. The importance of leveraging textual data like the conversation between people in social media space as grown. Therefore, I am in an interesting place of working with such data. Worked on web and text mining projects and it was quite a challenge. Used Application Programming Interface (API) to extract data from the web related to cities in US. Processed and cleansed this data. Used various data mining algorithms like similarity analysis, clustering and sentiment analysis. Utilized visualization APIs to represent the results in an easily accessible web application. It was a nice learning experience, but now working on much bigger challenge. When we want to extract value from this enormous web, we will need powerful computing abilities. Therefore, the industries buzz word “Big Data” came into picture. My journey with big data has begun where working on analyzing about online tech and social news agencies. Hope my journey with data takes me to better places.

Let data make the world a better place to live in.