AWS CloudSearch & node.js app in GCP

In my earlier post, I talked about CloudSearch and now I’m following up with the approach I followed to get the data searchable. There are multiple components. At the core is the AWS CloudSearch which gets populated with the data in JSON format, index those data and ready for search. Now, we have  searchable data but need a mechanism for a user to be able to search that data.

I chose node.js, Express and Angular.JS to create a web based app that will offer a UI where the users can type in the search terms and choose criteria to filter data. The app will convert the search terms into structured or simple query (similar to SQL for RDBMS), call AWS CloudSearch APIs to perform the search and display the search results using a Bootstrap grid. Right upto this point, my thought process was straight. Then I threw a spin, how about we host this web app on Google Cloud Platform?

I had no idea on GCP, but with prior knowledge of AWS and Azure and a liberal credit of $300 from Google, thought, it shouldn’t be a problem. I could have hosted the app on heroku or AWS or Azure. However, after reviewing the documentation, I felt confident that I can make it work. Yes, indeed I was right but only after those frustrating moments one have when attempting a new technology. I will write a separate blog entry on this and I don’t want to cloud out the AWS CloudSearch.

AWS_CloudSearch

Created two IAM users, one with authorization to only search and the other with authorization to upload and index CloudSearch. One good thing about CloudSearch being able to auto convert CSV to JSON. The source was data from MS SQL server, which was extracted as CSV and a separate script uploaded (in 5MB chunks) the data to CloudSearch and indexed the data. I was able to upload data with different formats (number of fields). Then the webapp will build the search terms using “structured syntax” if it is a compound query checking against multiple fields or a “simple” query if it is just text and all text fields are to be searched. The call to CloudSearch APIs through AWS SDK is made to return the search results which are displayed using a Bootstrap grid.

The full-text search on the relational data (MS SQL Server) took upto 30 seconds while the AWS CloudSearch was less than a second! For people who are looking to know why CloudSearch? 3 seconds vs. 30 seconds is a compelling argument to ignore.

If you have data in CSV format, I can show you a searchable demo pretty quick, ping me.

This was a great experience to get the hands dirty and there is nothing like it when it works! right?

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: