Addressing IoT storage constraints: A hybrid architecture for decentralized data storage and centralized management

Abstract

As blockchain and AI technologies advance, industrial applications increasingly rely on IoT devices for continuous, massive data collection. However, these devices have storage limitations when dealing with extensive data streams. The InterPlanetary File System (IPFS), a decentralized storage solution, addresses these constraints. However, IPFS's significant time requirement for data recording introduces a new challenge. Furthermore, its Content IDentifiers (CIDs) do not inherently convey the source device's identity or associated metadata, adding complexity to specific condition-based data retrieval. These factors hinder the discernment of the current data state, preventing IoT devices from clearing storage space by deleting unnecessary data. Consequently, this results in inadequate storage capacity for IoT devices, posing an obstacle to collecting and transmitting large data. Our study proposes a hybrid architecture to address these challenges, integrating the decentralized storage capabilities of IPFS with a centralized CID management system. This architecture employs Message Queuing Telemetry Transport (MQTT) for efficient CID transfer and a database for archiving CID values and associated metadata. Using the archived metadata, IoT devices determine the status of the data and perform tasks accordingly. This enables effective storage management for IoT devices by removing data that has already been uploaded and is safe for deletion. Our architecture demonstrates versatility, accommodating data of varying sizes, formats, and frequencies. We validated our approach through an extensive 100-hour experiment, successfully collecting 356 GB of data from diverse sensors. These results underscore the robustness and adaptability of our architecture, emphasizing its potential for a range of IoT applications.