Search This Blog

Wednesday, October 23, 2019

upload xl/CSV to S3 and process it via lambda for db insert

create a bucket 
create a lambda function and add trigger function as S3
when you create lambda a role will be created for you, add additional policy to it , dynamodb and S3 policy
Now add the following code in the lambda_handler function of the lambda
========================================
import json
import boto3

s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')

def lambda_handler(event, context):
    # TODO implement
   
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']

    obj = s3.get_object(Bucket=bucket, Key=key)

    rows = obj['Body'].read().decode('utf-8') .split('\n')

    table = dynamodb.Table('entity')
   
    print(len(rows))
    with table.batch_writer() as batch:
        for row in rows[:-1]:
            batch.put_item(Item={

                'name':row.split(',')[0],
                'address':row.split(',')[1]
            })
           
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

===================================
^^ make sure you have a dynamodb with table name entity and field name,address 
you can add as many fields as you want.

Now upload the csv to s3 and you should see lambda processing your xl and inserting to dynamo.

=============================
Sample csv
name,address,city
target,23230,austin
walmart,77707,houston
macy,80808,dallas


First row will be skipped as per the lambda for loop 

No comments:

Post a Comment