Basic bash script to gzip and upload your NGINX logfiles to Amazon S3.

This script was only tested on Ubuntu systems (18.04 & 20.04)

Assumptions

  • You’ve created an S3 bucket where you’d like to store your logs.
  • You have an IAM user with the necessary policies to access your S3 bucket.
  • s3cmd is installed sudo apt-get install s3cmd
  • Configure s3cmd to work with your bucket and credentials s3cmd --configure and confirm it is working s3cmd ls // s3cmd put <file> s3://<bucket>/<path>

Installation

Grab the following script and name it upload-log-s3.sh

Alternatively, just run wget https://worldincode.com/scripts/upload-log-s3.sh

BUCKET="YOUR BUCKET NAME AND PATH"

for log in $(echo $1 | tr " " "\n")
do
	cat $log | gzip -c | /usr/bin/s3cmd -q -c /home/YOUR USERNAME/.s3cfg put - s3://$BUCKET"$(date +"%Y-%m-%d")"/"$(date +"%Y-%m-%d_%H%M%S")_b_$(basename -- $log).gz"
done

Update the values for BUCKET and change /home/YOUR USERNAME/.s3cfg to the location of your .s3cfg file.

Update the permissions so it can be executed chmod +x upload-log-s3.sh

Open up your nginx logrotate file sudo vi /etc/logrotate.d/nginx and add /home/YOUR USERNAME/upload-log-s3.sh "$1" to the top of the prerotate block.

It should look something like this:

/var/log/nginx/*.log {
	daily
	missingok
	rotate 2
	size 25M
	compress
	delaycompress
	notifempty
	create 0640 www-data adm
	sharedscripts
	prerotate
		/home/YOUR USERNAME/upload-log-s3.sh "$1"
		if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
			run-parts /etc/logrotate.d/httpd-prerotate; \
		fi \
	endscript
	postrotate
		[ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
	endscript
}

Test it by forcing a logrotate sudo logrotate -f /etc/logrotate.d/nginx

You should see all of the rotated and gzipped files in your S3 bucket.