EFS and SOLR Cloud Backup(1)EFS Set Up for EC2 and Docker

EFS and SOLR Cloud Backup(1)EFS Set Up for EC2 and Docker

EFS Set Up
Amazon Elastic File System EFS provide simple, scalable file storage for EC2 instance.

In the EFS console, we can create a File System, there is a File system ID and then we can set up config in Elastic Beantalk.

In the configuration file .ebextensions/storage-efs-mountfilesystem.config
###################################################################################################
#### Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
####
#### Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file
#### except in compliance with the License. A copy of the License is located at
####
#### http://aws.amazon.com/apache2.0/
####
#### or in the "license" file accompanying this file. This file is distributed on an "AS IS"
#### BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#### License for the specific language governing permissions and limitations under the License.
###################################################################################################

###################################################################################################
#### This configuration file mounts an Amazon EFS file system to a directory named /efs. To mount
#### the file system to a different path, modify the MOUNT_DIRECTORY value in the "option_settings"
#### section.
####
#### The FILE_SYSTEM_ID setting references a resource named "FileSystem", which is created by the
#### storage-efs-createfilesystem.config configuration file. To use this file to mount a
#### file system that you created outside of AWS Elastic Beanstalk, replace the Ref with the
#### resource ID (e.g., fs-e7605f4e):
####
#### FILE_SYSTEM_ID: fs-e7605f4e
####
#### If your environment and file system are in a custom VPC, you must configure the VPC to allow
#### DNS resolution and DNS host names. See this topic in the VPC User Guide for more information:
#### http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html
###################################################################################################

option_settings:
aws:elasticbeanstalk:application:environment:
FILE_SYSTEM_ID: 'fs-a5838eec'
MOUNT_DIRECTORY: '/efs'

##############################################
#### Do not modify values below this line ####
##############################################
REGION: '`{"Ref": "AWS::Region"}`'

packages:
yum:
nfs-utils: []
jq: []

commands:
01_mount:
command: "/tmp/mount-efs.sh"

files:
"/tmp/mount-efs.sh":
mode: "000755"
content : |
#!/bin/bash

EFS_REGION=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.REGION')
EFS_MOUNT_DIR=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.MOUNT_DIRECTORY')
EFS_FILE_SYSTEM_ID=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.FILE_SYSTEM_ID')

echo "Mounting EFS filesystem ${EFS_DNS_NAME} to directory ${EFS_MOUNT_DIR} ..."

echo 'Stopping NFS ID Mapper...'
service rpcidmapd status &> /dev/null
if [ $? -ne 0 ] ; then
echo 'rpc.idmapd is already stopped!'
else
service rpcidmapd stop
if [ $? -ne 0 ] ; then
echo 'ERROR: Failed to stop NFS ID Mapper!'
exit 1
fi
fi

echo 'Checking if EFS mount directory exists...'
if [ ! -d ${EFS_MOUNT_DIR} ]; then
echo "Creating directory ${EFS_MOUNT_DIR} ..."
mkdir -p ${EFS_MOUNT_DIR}
if [ $? -ne 0 ]; then
echo 'ERROR: Directory creation failed!'
exit 1
fi
else
echo "Directory ${EFS_MOUNT_DIR} already exists!"
fi

mountpoint -q ${EFS_MOUNT_DIR}
if [ $? -ne 0 ]; then
echo "mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 ${EFS_FILE_SYSTEM_ID}.efs.${EFS_REGION}.amazonaws.com:/ ${EFS_MOUNT_DIR}"
mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 ${EFS_FILE_SYSTEM_ID}.efs.${EFS_REGION}.amazonaws.com:/ ${EFS_MOUNT_DIR}
if [ $? -ne 0 ][ $? -ne 0 ]; then
echo 'ERROR: Permission Error!'
exit 1
else
runuser -l ec2-user -c "rm -f ${EFS_MOUNT_DIR}/it_works"
fi
else
echo "Directory ${EFS_MOUNT_DIR} is already a valid mountpoint!"
fi

echo 'EFS mount complete.'

Because we use Docker in the Elastic Beantalk EC2 Host Machine, so we will mount that to our Docker as well.
elasticbeanstalk/Dockerrun.aws.json
{
"AWSEBDockerrunVersion": 2,
"volumes": [{
"name": "solr-home",
"host": {
"sourcePath": "/solr"
}
},
{
"name": "efs-directory",
"host": {
"sourcePath": "/efs"
}
}],
"containerDefinitions": [{
"name": "solr",
"image": “xxxxxxx.dkr.ecr.us-east-1.amazonaws.com/odt/solr:latest",
"environment": [{
"name": "OPERATION",
"value": "run-solr"
}],
"essential": true,
"memory": 35000,
"mountPoints": [{
"sourceVolume": "solr-home",
"containerPath": "/solr"
},
{
"sourceVolume": "awseb-logs-solr",
"containerPath": "/opt/solr/server/logs"
},
{
"sourceVolume": "efs-directory",
"containerPath": "/efs"
}
],
"portMappings": [{
"hostPort": 8983,
"containerPort": 8983
}]
},
{
"name": "init",
"image": “xxxxxx.dkr.ecr.us-east-1.amazonaws.com/odt/solr:latest",
"environment": [{
"name": "OPERATION",
"value": "join-cloud"
}],
"essential": false,
"memory": 1000,
"links": [
"solr"
],
"mountPoints": [{
"sourceVolume": "solr-home",
"containerPath": "/solr"
}]
},
{
"name": "healthcheck",
"image": “xxxx.dkr.ecr.us-east-1.amazonaws.com/odt/solr-healthcheck:latest",
"essential": true,
"memory": 1000,
"links": [
"solr"
],
"portMappings": [{
"hostPort": 8080,
"containerPort": 8080
}]
}
]
}


References:
https://aws.amazon.com/efs/?sc_channel=PS&sc_campaign=acquisition_US&sc_publisher=google&sc_medium=efs_b&sc_content=aws_efs_e&sc_detail=efs%20aws&sc_category=efs&sc_segment=208381315165&sc_matchtype=e&sc_country=US&s_kwcid=AL!4422!3!208381315165!e!!g!!efs%20aws&ef_id=WDe8wQAABKdZQ0gc:20180111034816:s