Training data in S3 in AWS Sagemaker

Question

I've uploaded my own Jupyter notebook to Sagemaker, and am trying to create an iterator for my training / validation data which is in S3, as follow:

train = mx.io.ImageRecordIter( path_imgrec = ‘s3://bucket-name/train.rec’ …… )

I receive the following exception:

MXNetError: [04:33:32] src/io/s3_filesys.cc:899: Need to set enviroment variable AWS_SECRET_ACCESS_KEY to use S3

I've checked that the IAM role attached with this notebook instance has S3 access. Any clues on what might be needed to fix this?

Answer 1

If your IAM roles are setup correctly, then you need to download the file to the AWS Sagemaker instance first and then work on it. Here's how:

# Import roles

import sagemaker

role = sagemaker.get_execution_role()

# Download file locally

s3 = boto3.resource('s3') s3.Bucket(bucket).download_file('your_training_s3_file.rec', 'training.rec') #Access locally train = mx.io.ImageRecordIter(path_imgrec=‘training.rec’ …… )