Scalable architecture using AWS IoT

Internet of Things (IoT) systems handle billions of source devices and trillions of data points flowing into a central platform. The growth of data is massive and IoT systems are required to scale in order to manage data explosions.

Key functions such as Inbound data collection within a central platform, real-time data analytics, scalable storage space and offline analytics – are all required to scale seamlessly for an IoT solution to successfully scale. There are IoT platforms that help manage requirements such as high scalability and security, and eliminates the task of managing such challenges by business owners.

The architecture of an IoT solution largely depends on the requirements of a given system, load and the data involved and the IOT platform that is in use. Today we discuss a few of the best practices that we follow to help us achieve the desired scalability of a solution using AWS IOT.

(Video: What is AWS IoT – take a first look at AWS IoT)

1.Design to operate at scale reliably from day one

As we saw, an IoT system is expected to deal with high-velocity and high-volume data captured by device sensors. The flood of incoming data might arise because of sudden growth in business, consistent growth over time, or due to a malicious attack. In any case, the system should be geared to face this from day one.

While taking care of the scalability aspects, the IoT system should also make sure that the data received is reliably processed. The best approach is to queue or buffer data as soon as it enters the system to ensure reliability.

2. Use AWS IoT Rules to route large data volumes into the rest of AWS

Consuming data from device topics directly by a single instance (without fault tolerance) prevents systems from being leveraged to achieve full potential scalability. Furthermore, this approach limits the availability of the system. On the other hand, the AWS IoT Rules Engine is designed to connect to endpoints external to the AWS IoT Core in a scalable way.

Additionally, AWS IoT Rules Engine facilitates the triggering of multiple different actions in parallel once the data is captured by the IoT system. This gives the system the ability to fork data into multiple datastores (receiving systems) simultaneously. If the receiving system is not designed to work as a single point of entry into the system, the data must be buffered or queued into a reliable system before being sent to the target systems – to provide the system with the ability to recover in the event of a subsequent failure.

3. Invest in automated device provisioning at earliest

When a successful business grows, the number of devices connecting to the IoT system is likely to increase as well. This makes manual processes such as – device provisioning, bootstrapping software, security configurations, device registration and upgrades – not feasible anymore. Hence, minimising human interaction in the initialisation process is important to save time and increase efficiency.

Designing built-in capabilities within the device for automated provisioning and, leveraging the proper tools that AWS provides – to handle device provisioning and management – allows systems to achieve desired operational efficiencies easily and at the cost of minimal human intervention

4. Manage IoT data pipelines

The tremendous amount of data captured as part of the IoT solution will need to go through data processing channels to make information out of them. This phase of ‘processing’ might involve many system/components in the process. The path that data travels through are referred to as data pipelines. Thus, data pipelines have to be designed to handle huge loads without compromising on performance.

In addition to this, architects should keep in mind that all of the data does not necessarily require all of the processing power that the system facilitates. During the design phase, architects should determine the data pipeline for each type of data.

5. Adopt scalable architecture for custom components

Adopt scalable architecture for external components that are added into the solution, and ensure that these components do not turn into performance bottlenecks that – in turn affect the entire solution.Additionally, these components must be designed to accommodate system expansions easily.

Adopting microservice like architecture does not only embed scalability within the system, but also provides the flexibility of replacing components that could possibly affect performance.

6. Adopt multiple data storage technologies

IoT systems deal with high volume, high velocity, and large varieties of data. A single IoT system might not be able to efficiently store all of the data in one type of datastore. Architects should choose from the most appropriate datastores and suitably supporting technologies for different types of data – to achieve the desired capacity and scalability of the system by increasing efficiencies and throughput.

7. Filter data before processing

All of the data that is directed towards an IoT system may not require processing in real time and such data can be filtered out. Filtering can be facilitated by using AWS Greengrass. Architects can select data that is not required by the system immediately and design the system to accept this data in chunks – when the cloud platform demands it. This way, system capabilities are used optimally, allowing more room for scaling.

Conclusion

In the forthcoming years, IoT is expected to instrumental in managing exponential growth. In turn, as adoption of fully integrated IoT systems grow, the numbers of devices being added to systems are also expected to grow exponentially..

Thus it is important to implement systems which are scalable when required, whilst also ensuring reliability and security. AWS IoT platform is a cloud IoT platform which offers all the qualities that a modern architecture may demand.

Implementing a scalable as well as reliable system from the day one will save cost and effort from the Architects and Business owners.

Even though AWS IoT platform is designed to provide automated scalability, solution architects should ensure that they follow certain best practices to make sure that systems integrations are planned and implemented in a scalable manner – to provide a comprehensive and scalable solution.

Thank you for reading our latest Mitra Innovation blog post. We hope you found the lessons that we learned from our own startup story interesting, and you will continue to visit us for more articles in the field of computer sciences. To read more about our work please feel free to visit our blog.

Sangeetha Navaratnam
Software Architect | Mitra Innovation