Create rolling monthly, weekly and daily Logstash indices

最新推荐文章于 2025-09-06 21:58:22 发布

转载最新推荐文章于 2025-09-06 21:58:22 发布 · 136 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/smail-bao/p/9353987.html

文章标签：

#大数据

本文探讨了ELK Stack中不同频率创建日志索引的方法，包括按月、周及日进行滚动。按月创建索引能简化备份操作并便于整体恢复，但可能导致较大的索引文件。文章还提供了具体的Logstash配置示例。

在刚刚开始接触ELK的时候我们习惯把每一个index都按照day来切割。但是我们会发现我们的shards 会很多。

其实我们一该把那些小的index按照一周或者一个月来rolling，来减少我们的shards数。

我使用的是logstash5，这里我们每启动一个logstash的进程的时候我们会发现我们的jvm 参数的-Xmx1g -Xms1g

但是我么一些可以设置小一点，比如512m 甚至是256m 来减少资源分配

具体的操作方案：

原文来自于：http://www.atechref.com/blog/elk/elk-stack-logstash-mutate-and-monthly-index-setting/

How often should a new log index be created? Once a day, Once a week, Once a month? A simple search in Google would return various responses each arguing the pros and cons of creating indexes daily or weekly. Lets look at how to do that with logstash.

My take on that is “once a month” index is the best option.  The following is my reasoning for this.

Pros:

Easier back up with a monthly index
Simple to create snapshots and restore
One index to backup externally on NAS or other storage outside of ELK stack
Works well where the retention policy for active logs is 30 days or 60 days etc
Allows complete logs for the whole month to be restored in one go.
Cons:

 Potential for large index sizes
Have to restore the whole index to search for a specific day of the month
Backup and restore of these indexes can take some time in slower systems or single node instances
Creating a monthly rolling index file

In order to create a new index each month automatically ensure you have the following setting in your logstash config file for e.g. devlogstash.conf.

input{
  ….
  }
  filter{
   ….
  }
  output{
elasticsearch{

            hosts => [“192.168.0.1:9200”]
            index => “dev-logstash-%{+YYYY.MM}”

}

}
Creating a weekly rolling index file

The weekly name format would be YYYY.ww as in 2017.01 for the first week of the year in number.  Config setting would be as shown below.

input{
  ….
  }
  filter{
   ….
  }
  output{
elasticsearch{

            hosts => [“192.168.0.1:9200”]
             index => “dev-logstash-%{+YYYY.ww}”

}

}
Creating a daily rolling index file

Just add MM.DD instead of WW to the setting above to create a daily rolling index as shown below.

input{
  ….
  }
  filter{
   ….
  }
  output{
elasticsearch{

         hosts => [“192.168.0.1:9200”]
         index => “dev-logstash-%{+YYYY.MM.DD}”

}

}
Creating a Year, month and week rolling index file

And that could be defined as YYYY.MM.ww to create a weekly rolling index as shown below.

input{
  ….
  }
  filter{
   ….
  }
  output{
elasticsearch{

         hosts => [“192.168.0.1:9200”]
         index => “dev-logstash-%{+YYYY.MM.ww}”

}

}
 Restart logstash for these changes to take effect.