AWS Analyze Big Data with Hadoop (Possible Error)

Normally for the analyze big data with hadoop we use Amazon EC2 instance. After installing the aws-kensis and start the streaming the data to data stream (using fake apache log generator) , there would possible error on permission issue where the log files unable to sent from ec2 instance location. To overcome problem following procedure must follow.

Modify the file /etc/aws-kinesis/agent.json

{
  "cloudwatch.endpoint": "monitoring.us-east-1.amazonaws.com",
  "cloudwatch.emitMetrics": true,
  "firehose.endpoint": "firehose.us-east-1.amazonaws.com",
  "flows": [
    {
      "filePattern": "/var/log/aws-kinesis-agent/access_log*",
      "deliveryStream": "web-log-ingestion-stream",
      "dataProcessingOptions": [
       {
          "initialPostion": "START_OF_FILE",
           "maxBufferAgeMillis":"2000",
           "optionName": "LOGTOJSON",
           "logFormat": "COMBINEDAPACHELOG"
      }
    ]
   }
  ]
}

Then perform the following:

1.Start kensis services
sudo service aws-kinesis-agent start

2.copy the python files /home to /var
sudo cp /home/ec2-user/Fake-Apache-Log-Generator-master/apache-fake-log-gen.py /var/log/aws-kinesis-agent/

cd /var/log/aws-kinesis-agent/

sudo chown -R aws-kinesis-agent-user:aws-kinesis-agent-user .

sudo su - aws-kinesis-agent-user

3.Create stream of logs
python apache-fake-log-gen.py -n 100 -o LOG -s 10