Data collection
When we last left off, scale was the topic of concern. Picking the right technology for the now, so that it can we can build upon and secure in the future. When selecting technology, we should strive for technology that will have longevity. Or that will not struggle to meet your eventual successful product needs. Modern software development has improved recently, so it often becomes personal preference.
These software and platforms are crucial for a responsive service. And almost all systems rely on some sort of data persistence. The data layer can either introduce bottlenecks or operate in the background. But the decisions made here are long term. Selecting NoSQL vs traditional SQL database can help the User Interface. But they fall short for your Business Intelligence tools. Selecting a popular RMDBS (relational management database system ) allows for rapid development. But will they fall down trying to process millions of requests?
You should also prepare for the possibility of changing your data layer in the future. This is why it’s important to avoid those tempting database specific functionality. The more stored procedures and triggers you use, the more you locked into that vendor you become. Seems like common sense, but it is an easy trap to fall into in the interest of rapid development.
When deciding on a data platform, we realized that we need to shard the data so it is easier to scale and secure. If one data store becomes compromised, data integrity is not. This requires more logic in the application layer, rather than proprietary stored procedures. This is always an interesting debate between your data and application teams!
Our approach is to use many databases to go with our many micro-services. With this “database-per-service” strategy we can better control our costs and our platforms. We use a RMDBS for demographics, and that allows us to take advantage of those data relationships. And many Business Intelligence tools work better on them. But, a database backed cache helps with faster performance for high volume services .
This introduces a strategy different from the traditional stack. But, we find its optimal for tailoring data storage for its end use rather than a one-size-fits-all. We have a greater flexibility to optimize our services. Scriptio is able to scale quicker and respond faster to serve our customers better.