School of Information Engineering, Huanghuai University, Zhumadian, China

---

Abstract—In this paper, we study the file management mechanism of large-scale cloud-based log data. With the rise of big data, there are more and more the Hadoop-based applications. Log analysis is an important part of network security management, but the existing network log analysis system can’t deal with huge amounts of log data, or only use offline mode which with a longer response delay. Therefore, building the online Hadoop-based log processing system is necessary. However, how to effectively manage vast amounts of log data have become the key problems of such system. To this end, this paper puts forward a new hierarchical file archiving (HFA) mechanism which can realize the hierarchical and sorted storage of massive amounts of log data. In addition, some feasible methods for the mechanism are also proposed. Through the HFA mechanism, the traditional log analysis mode and Hadoopbased offline analysis mode can be combined to achieve the online Hadoop-based log analysis system, which have good scalability that can effectively store and handle the massive log data, and faster response speed for user request to meet the requirements of online processing. The feasibility and effectiveness of the HFA mechanism have been verified by the experiment of a small log process system.

Index Terms—Network security, Log analysis, Hadoop, File management, Hierarchical file archiving mechanism, Online system

[PDF]

Cite: Fengying Yang, Huichao Liu and Zhanping Zhao, "Research on Cloud-Based Mass Log Data Management Mechanism," Journal of Computers vol. 9, no. 6, pp. 1371-1377, 2014.