Archive for the ‘Newtech’ Category

AWS Cloudsearch

March 25, 2017

The cloud is filled with wide ranging options to store and retrieve data and so is on-premise. Every cloud provider has their own cloudsearch solution from Amazon to Azure to Google. In addition to these proprietary solutions, there are these open source platforms ElasticSearch, Apache Solr etc.

Here is a wonderful blog comparing the three products.

In short, all offer similar features with little difference. I would say there are mainly two big differences between AWS CloudSearch and the other two.

  1. Data import is a batch process in AWS CloudSearch. If you have streaming data or immediate data update, the go for elastic or Solr.
  2. If you don’t need to worry about infrastructure, backups, patches, then go with AWS Cloudsearch. Out-of-the box it comes out as a true cloud product. as well as AWS offers elastic search as a service where they have simplified the infrastructure part., infact offers it as a service on AWS cloud. However, Elastic and Solr are more popular than CloudSearch. Thus, it is easy to find resources online for these two compared to AWS CloudSearch.

Thus, I embarked on a journey to take-up AWS CloudSearch and you know what, it is not that difficult (though I went through those gnawing issues and had my own share of frustrating moments). To begin with, I did the manual route of extracting the data out of my RDBMS (SQL Server), upload the data to CloudSearch, indexed it and used the rudimentary UI provided by AWS and was able to search in an hour. The biggest advantage, I see with AWS CloudSearch data upload is that it takes a CSV file and converts to JSON by itself. You can write a batch program to upload in chunks of 5MB files. In addition to CSV, it support multiple other types such as PDF, EXCEL, PPTX, docx etc.

Both Solr and Elastic search, you need to provision a Linux server, then install and configure similar to any software that you download.  Even if you take the service route, you still need to worry about backup, upgrades, applying patches etc. One big advantage of these is that you can have it on-premise as well, while AWS CloudSearch is truly available only on the AWS cloud. Beyond that, Elastic also has data visualization tool Kibana or it comes like a suite (ELK – Elastic, Logstash, Kibana). AWS ColudSearch offers only indexing and search and no visualization which is a separate product Quicksight (I haven’t looked at this but I plan to).

I will write more about the programmatic approach in my next entry. Please drop me a line, if you can’t wait and wish to see it in action!



Is your apps “Cloud Ready”?

December 11, 2016

It was mainframes before the 90s, then came the client-server architecture and the web followed that in 2000.  The web architecture is somewhat similar to the mainframe architecture. I started my career in client-server, went to the web and now I’m in cloud. The technology departments in the corporate world and the IT services firms assisted in the migration from mainframe to client-server to the web. One need to change the mind-set while designing apps for specific technology platform. It will be easy to understand if you look at each of these technology framework as a genre. The current cloud architecture actually resembles the client-server architecture to a great extent. These days, the apps are feature rich with majority of the processing happening on the server side while still rendering logic on the client side (AngularJS website, apps on your phone etc.)

The are multiple cloud offerings, I mean IaaS, PaaS and of course there are many vendors Amazon, Microsoft, Google, Rackspace, Openstack and the list keeps going. You can have private cloud, public cloud and hybrid cloud. Now, everything you need is provided and is available as an app. The concept called SaaS which started in early 2000s with the advent of websites.

Let’s talk about migrating your applications that are currently living in your data centers or co-hosted or on your own premises. The easiest way is to follow the IaaS path wherein the physical servers are replaced with the VMs in the cloud. I’m saying this as easy because if your servers are currently in a data center you are any way remotely connecting to them from your personal device (PC, mac, Chrome – whatever), it doesn’t matter whether the remote server is in your DC or in Amazon or Microsoft DC. Apart from this, your applications doesn’t need to undergo any change, pretty much. However, this will not let you take advantage of “the cloud” offerings.

The second approach is to build your apps and make it available as SaaS by using the PaaS. This is where you will reap the benefits of the cloud architecture. Wait, you will hear from everyone, oh you are going to get locked up with a vendor! This is true if you were asking me about this even three years ago. Now, the cloud offerings have matured and one approach you can follow is to use containers such as Docker.

Let us take a hypothetical app where you receive data from your various vendors, the app need to load the data, notify vendors / your IT / business on failures, notify customers of the changes and that the latest information is available on the website for everyone to access. In such an app, in the current world (hosted in a physical or virtual server in your DC), you can have your own FTP server to pull the data from vendors and store the files in the file system of your server; with SMTP service running in your server to send the notification emails; any failure files can be moved to an error folder and error logs written to a log file in the log folder; Once the data is valid, you can update your database which forms a separate cluster in your network. If such an app/ website need to be hosted on the cloud, it need to be made “cloud ready”. What does it mean by “cloud ready”. Some of the salient features of a “cloud ready” apps are being machine agnostic, self-aware, secure and more. We will see the top ones here.

  1. Machine agnostic. – some examples
    • No file system – store files on AWS S3 / Azure cloud storage/ drop box [if you really want one, try the AWS EFS, it’s a network file system and you can use like NAS]. If you have a server, you will have a file system, but you should avoid them as it will have issues when you scale up and scale down. In Azure, the local C drive is temporary and you lose all data on it on reboot (other than the OS).
    • No local ftp / SMTP – use other mechanisms or 3rd party [FTP is a serious security issue], use AWS SES or other providers for email.
  2. Self Aware
    • Apps should be service oriented and the service endpoints are fetched from the application config files.
  3. Secure
    • Secure communications – Use SSL and encrypt all communications especially if you are in a regulated industry (eg:HIPAA). if you are app is calling other allied service end-points pass encrypted data and follow authentication and authorization policies. This is something often discounted when you are writing apps for your own DC though it’s a recommended practice.
    • Encrypt Data @ Rest- Don’t keep PII (Personal Identifiable Information) but if you need, you should encrypt that stored data. This includes any document / form data that you collect, create.
    • if you let users upload to your S3, please use proper authorization policy and move them out into a secured bucket ASAP.

In addition to these, there are more and my intention is to bring your attention to this and not to write a thesis. You are always welcome to write to me, if you want to discuss more.

Getting your apps to the cloud opens up possibilities,  more tools & technology at your disposal. You can easily start notifying your clients through SMS (trulio), you can cache the data if you are using GCE, you can use Firebase / AWS cahching to synchronize data and provide a seamless experience irrespective of the device form factor. Ultimately, your application will be device agnostic and your apps can follow your user from phone to PC to TV.

These thoughts are not just for migrating, you need to keep this in mind for newer applications you are building. Feel free to drop me a line if you need help migrating legacy apps to the cloud.

Server-less Architecture

November 26, 2016

Using AngularJS & Firebase

There are lot of write-ups on the pros & cons of server-less architecture. How it creates vendor stickiness, high TCO, it is restricted to certain technologies, it is not a panacea. I’m not going to get into that argument. One thing, I will buy into is that “it is not a panacea”, everything else is debatable. There are many ways of achieving the sever-less architecture and the one we discussed earlier is AWS and the lambda functions. Yes, it is still a choice but IMHO, it takes orchestration of so many individual components of AWS. You need Lambda, API gateway (to make it REST), Cognito or other Auth providers, DynamoDB or other Database and more.

Google has come up with a platform called Firebase.  It is so simple and concise. What initially started as a JSON database has evolved into a full-fledged platform that encompasses all AWS components that I mentioned above. It can host, it can offer authorization, it can authenticate with 3rd party providers such as Facebook, Google, it can store data and if I missed something, you know what? it can do that too 🙂 (you still need to develop the fron-end website) The biggest drawback is that working on Firebase makes you equivalent to an astronaut on the ISS doing a spacewalk! Yes, remember ISS moves in a geostationary orbit @ ~17k miles an hour? well, Google is not that fast! but they keep constantly updating the platform in weeks if not hours. That is the constant challenge. If you need help and came across a video that is a year old, well – forget it. You need to find something that is not more than six months old at the max.

The simplest and fastest way to get up and running in production is to use AngularJS for the front-end and Firebase for the back-end. For most apps, the free version of Firebase is good enough. You can right your code in AngularJS, minify using tools like gulp, google offers a Javascript SDK / library called AngularFire to interface with Firebase from Angular. You need to do few minutes of configuration on the console for enabling authentication, authorization, you can copy the configuration details (like db connection string) and no need for any secret key / password to be stored on the client side, all authorization is done on the server side. Thus you don’t need to worry that your secret keys are getting exposed.  Once you develop the web-app in AngularJS, minify using gulp, you can host the website either in Firebase or AWS S3 (How to host on S3).

If you are a startup, you want to validate and get an MVP out quickly, this is a great platform. You can even continue to run until you reach a critical mass and if you feel any bottleneck, you can afford to build on any platform. Google platform can scale and where I see problem is if you want to download your data from Firebase, it gets tricky. Even if you are big corporation and need to quickly get a web-based app or need a website to manage an event that is coming up or anything that is not going to run for years and that IT says it will take years to build, I will say nothing can be cheaper and faster than Firebase.

If you have thoughts and want to discuss, I’m always open, drop me a line.

To Upgrade (or not?)

August 16, 2016

We see all these attractive things in life. we want this, we want that, do we really need? we don’t ask that question. That is the ONE question and the OTHER question is what would it be if we don’t get that? The answers to these two questions pretty much sums up your next course of action.

However, often times it is not as simple as that. If you have a 10 year old Toyota Camry and you want a Mercedes 300, it’s one thing but what if you have a Windows XP PC (yeah!, you read it right) and it works just fine @ work or @ home and you need to upgrade to Windows 7 or 10 (Please don’t think of 8).  If it is @ home, at least you are the only one that is exposed but if it is your work, your exposure is whole lot and not just financially but also the number of people it will impact.  I’m going to focus on running unsupported software here.

I come across this scenario a lot and especially recently. The world has come across the fact that the 800 pound gorilla that has woken up (yes, I’m talking about Microsoft). They have started churning up software that is inline with current global market and to address the current global risks. In thinking through the process, there is no golden rule or a magic wand to figure out, is it time to bite the bullet and upgrade. However, we can come up some guiding principles or set of questions to help us in the decision making process. Based on the experience, interactions with our clients, feedback, industry pundits, I came up with a set of guiding principles. (Again this is just a sample.)

  • Are you in a regulated industry? Is your systems conforms to the govt. regulations
  • Do you have a corporate governance committee? Is your systems conforms to the governance rules laid out?
  • Is there and industry standards body that you are part of? Is your systems conform that (basically you are not just preaching but walking the talk)
  • When did you last perform an upgrade? How many versions are you behind?
  • Any (how many) of your systems use software that are not supported by respective vendors?
  • Are all customer facing systems are running supported software?
  • What is the cost of not performing an upgrade? What is your plan when Sh*** hits the fan or things stop working?

If you are in a regulated industry, you don’t have an wiggle room, you need to follow the govt. regulations. You need to be honest with yourself, thus you should be conforming to your own standards. These are rules that you laid for yourself that you should follow (remember those new year resolutions of going to the gym? – well this doesn’t belong in that category). You are part of a world / industry standards body and you need to stick up. if you are not then who will? Then comes when did you last perform an upgrade. I get it is a chore and too much impact etc. But if your answer is half a decade or more, you have reached a point where the cost of inaction is more than your action. If or how many are using unsupported – if the answer is more than ZERO and they are being used in a customer facing environment, you probably don’t have “a plan” for the last question. If it is the internal systems, you will be able to setup more security rules, un-maintained servers a.k.a – you got a parachute. However the last question is the most critical of the lot for your business. When you do this, you are not just jumping of the plane, but you are jumping off the plane with no parachutes (for anyone in your organization). Depending on how critical, you can make the jobs obsolete, in turn the employees and in result you own organization. Remember we read about stuff on the newspaper and other media about such incidents? Do you want to be one such example?

IMHO, the ideal upgrade is Three years and not more than FIVE. Beyond which, you will get into the problems of unsupported software, security, obsolete technology, rusted workforce and more.

What are your guiding principles? Drop me a line.

Microsoft Open Source

August 8, 2016

Yes, You read it correct, I hear you, saying, isn’t it an oxymoron? Well, it is not.The world is abuzz that ever since Satya took over Microsoft, things started changing. It is more developer friendly, it is more open to new ideas, it is collaborating with various vendors, partners yada yada yada…

This new era started with Microsoft embracing Linux, letting you create Linux instances in Azure. You all know that Visual Studio is free to the entire world. I have been associated with technologies in the past couple of decades, one of the best tool for developers is Visual Studio, hands down (No offense to Eclipse, Sublime and the like). They used to do this before calling it developer edition, then called community edition, trial edition. These various editions are either teasers i.e the features you want to use are not available and that you need to buy the paid version or it will be available only for 90 days or some limited period. They have gone out of that mindset and said, well there is going to be one edition an is free.

The best development tool in the world, the one that is very well integrated with the Azure cloud that lets you manage, create, deploy the assets is free now. If you thought that is not enough, recently, Microsoft announced that they are giving the SQL Server Developer edition Free. This is huge! I remember struggling with the so called SQL Server Developer edition couple of years back that it doesn’t have analysis services, integration services, SSRS, the SQL Agent. Now all that is available. So what is the catch? Nothing. This edition is exactly same as the SQL Server Enterprise Edition. The download link is here. You just need to signup / create an account with VS essentials. The only limitation is that Microsoft says, you cannot use Developer edition in any production environment. At least, you don’t have to worry about purchasing when you are developing your product. You focus on developing the product and when it comes to deploying or taking it live, you can worry about it. Even then, both Microsoft and Amazon offers you to participate in various “Start up” initiatives that you an benefit from.

I hear you asking, so what brought about this change? IMHO, this is how Microsoft was operating from inception. They probably forgot for a while when Steve was at the helm. It was Novell Netware (you guys remember?) that pioneered Local networks in the corporate world, then came Windows NT. You all remember what happened to Netscape? Well, Microsoft offered Internet Explorer free. We didn’t get anything free for a while and now it started again. You get Visual Studio, SQL Server Developer, Xamarin for mobile apps, the start-up initiative, Linux.

Don’t over analyze. Start working on your next idea and the ecosystem is available for you to use the best tools out there. What are you waiting for? Microsoft is Open to sourcing it from / to you. Happy developing.

PS: The complete licensing guide for SQL Server 2014. if you are in doubt, don’t hesitate to drop me a line.

You have (want) an app for that?

August 4, 2016

It’s Déjà vu again. This used to be “do you have a website?” in the early 2000, now it’s “do you have an app for that?”

It’s a given that everyone, everything has a website, thanks to the various website builder tools/websites, content management tools such as WordPress, Joomla, Drupal. This used to be a big business of building websites. These various tools democratized the website building process. You don’t have to be a pro. They all offer a WYSIWIG editor, you drag, drop, buy images of iStockphoto, few minutes (may be an hour) later you have your own website without thinning your wallet. These content management tools really took the wind out not just of making but also maintaining the website on a regular basis.

There are specialized websites / tools bordering on SaaS (Software as a Service) and website builders for specific categories. For example, I was looking to manage my son schools’ PTO / PTA, there are a number of sites and the prominent among them being ParentOrbit. They not only allow PTO to register, collect funds using credit cards, ACH, but also let you setup class parents, broadcast emails, collaborate with other parents. I even noticed them letting independent Business Owners (robotics, chess classes, anything to do with kids)  to run a virtual shop. They can manage their business, communicate, collaborate, accept payment etc.

The same democratization process is happening in the app marketplace. Unless, you are looking to develop a unique, ultra functional app, you pretty much can create, publish to the store and maintain it the same fashion. The same content management tools like WordPress have its own mobile plugin, that lets you convert a website built with WordPress into an app. If you want something specific like, creating a Social media based mobile app such as your band’s fan club there are myriad engines like Anahita, HumHub, SocialEngine. These let you create a website and convert them to mobile apps.

In addition to these tools similar to content management, there are also other mobile app builder tools like goodbarber, appypie and more. Then there are other cross platform app builder tools such as Microsoft Xamarin, Appcelerator Titanium. Microsoft recently bought Xamarin and bundled it free with Visual Studio.

You don’t have an app yet? want an app? Pick one- anyone is good enough than not having one. Still confused? drop me an email.

Robots for Tots

January 18, 2016

My son in the 3rd grade was being a normal 3rd grader with interest in video games. I saw that he was playing with “MIT Scratch” and showed some interest in programming. Thus, I saw an opportunity to wean him off video game for some time. I came across a robot called MakeBot for $70 which lets you program using Scratch. Fortunately, my son liked this and I enjoyed spending time with him in an educational experience.

Initially, I helped him assemble the robot and then I got busy with other stuff. He tried his best connecting it to the PC, writing some sample programs but was not getting anywhere. Then I spent couple of hours poking through forums, googling and figured out how to make it work. Though out of the box their instructions are not that good, their forum is very helpful. For those, who want to try, here are some basic tips.

  1. Place four “AA” batteries and the CR2025 button cell battery for the remote
  2. Install the make block studio, connect it to the PC using the USB cable (the only way to flash any program or firmware to the robot.
  3. Once you turn-on the robot (the black switch), click connect-> searialport -> COM4
  4. From File -> upgrade firmware.

These are the basic steps to getting started. Once you connect, you can see the new “Robots” block appear in the makeblock studio. In order to run a program in the robot, you need to pull “When Robot” block instead of “When clicked”. you can use the “when clicked” for debugging purposes. Once you have “When Robot”, right click and select “upload to Arduino”. then you will see a new window show up with the “C like” code. click the button “Upload” on the top, you will see the code getting compiled and get uploaded to the robot. The moment this is uploaded, the robot will start doing what the program asked it to do.

You can choose to program in “Arduino” or  Scratch. Scratch is very easy and is a great way to start learning programming. He used it for his school science fair and wrote the program to control the robot using the remote. Here are some fun things to do , you can use the line sensors to make it follow a track, use the ultra-sound sensors to avoid obstacles. It comes with a Wifi chip while the blue tooth is optional.

Some useful link –

If you need more info, drop me a line.



Windows Server 2012 Licensing

February 13, 2013

In my quest to find out answers to my simple questions, I had to go through legal lingo everywhere and no straight answers. Thought, I would pin it here and might help someone else.

There are four versions and small businesses with less than 25 users can go with the “Essentials” with no need for any CALs.

The two other version that would be widely used are “Standard” and “Data Center”, with only difference being the number of virtual instances that one can have. Standard supports upto two virtual instances while Data center edition has no limitation.

All licenses are processor based. i.e one license per physical processor. If you have two Quad-core processors, you still need only one standard license. If you have four dual core processors, then you need two licenses.

If you have only websites running on your server (or basically hosting websites), you DO NOT need any CALs. If you have any windows apps, then you need CALs.

The another significant thing regarding the virtual instances are that they could be of any previous OS (windows 2003, 2008) as long as you have the necessary license. There is no separate license for Hyper-V like the earlier versions.

You can add more licenses if you need more virtual instances later.

Just-in-time apps – Next Generation Web hosting

April 30, 2011

Since the inception of internet leading upto its peak,  having a website was a given for every organization . In a similar vein, with the advent of iPhone, now everyone is expected to have an app. After the .com boom, the website creation is commoditized by the web-hosting companies that they would offer a variety of templates and you could build your website in few minutes. The same business model is being extended to the app world by the new entrants – iSites, Widgetbox and bizness apps.

Every one of them offers a paid plan per month per app plan with varying degree of support, analytics and the apps work in both android and iPhone. BiznessApps is the cheaper of the lot.  iSites offers a basic web based app while the BiznessApps offer the ability to create native iPad apps and widgetbox let you create widgets alone.

Darn Interesting & Fascinating

November 4, 2010

With the two latest innovations, shall I say Micosoft is a shining star? Where this will take Microsoft is a darn good question but these two are going to become big. One is the Microsoft Kinect, this has wide ranging implications to the technology industry, even outside IT. If Wii revolutionized the gaming market, Microsoft has really taken it to the next level with Kinect technology. The fact that it can sense the human motion without having to hold anything has implications in so many areas. For example, there could be an iPhone app with this motion sensing technology that could decipher “Sign language” and convert it to voice on the other end and much more.

The second one that pretty much revolutionizes is second only to google digitizing books! Recently microsoft created something called datamarket, yes! a marketplace for data. One can store information in digitized form in the SQL Azure database, make that data or any portion of it as a regular webservice and publish in this market. Some are free, some are free upto a point and others are paid services. Thus its making the data available for computer programs, driving efficiency higher. Some examples of services are realestate data from zillow, carbon emissions information on EU countries, MLB statistics… This is a great opportunity and this will drive more and more sevices to be automated driving costs down and improvig efficiency.

These are two awesome innovations but How far Microsoft will benefit from the advent of these two remains to be seen.  But I’m sure the other players are going to innovate further, improvise resulting in increased adoption and applications of these two innovations. Whether Microsoft will again become a glowing star or a blowing up star is a topic for historians.  Truly these present a great opportunity for the next round of entrepreneurs.