Direct upload to s3 with php AWS4 progressbar blueimp file uploader validation

Continue to talk about AWS and S3. Today I will share my experience in direct upload to s3. This could be useful when it comes to big files. You can upload file directly to S3.

  • No need to wait twice
  • progress bar works correct. If you upload through your server progress bar will show you only upload process to your server but then your users will wait until you upload file to s3

If you got to bluimp docs you will find the arcticle in the wiki but it use old algorithm and you progressbar won’t work. Also you will find out that blueimp validation won’t work as expected.

We will use latest (AWS Signature Version 4) I think Amazon docs sometimes have lack of examples in their docs. And some small details can be real time killers.

Ok let’s start with a simple case.

Creating an HTML Form

I’d recommend to read AWS docs first they keep main info. And I’ll talk about some non obvious details. (I will use yii1 in my examples but I don’t think it really matters)

 <?= CHtml::beginForm("http://{$bucket}.s3.amazonaws.com", 'post', ['enctype' => 'multipart/form-data', 'id' => 'file_upload_form'])?>
    <?= CHtml::hiddenField('key')?>
    <?= CHtml::hiddenField('Content-Type')?>
    <?= CHtml::hiddenField('acl')?>
    <?= CHtml::hiddenField('success_action_status', '201')?>

    <?= CHtml::hiddenField('x-amz-credential')?>
    <?= CHtml::hiddenField('x-amz-algorithm')?>

    <?= CHtml::hiddenField('x-amz-date')?>

    <?= CHtml::hiddenField('policy')?>
    <?= CHtml::hiddenField('x-amz-signature')?>

    <?= CHtml::fileField('file', null, ['id' => 'file_upload', 'name' => 'file', 'class' => 'hide'])?>
    <?= CHtml::endForm() ?>

Ok so you get more details about fields in docs. I will explain some subtleties later in this article. What I’d like to highlight – don’t forget to name file field as file and place it last.  Next thing that we have to do – fill our fields with data and let’s start with policy.

Generate policy

Again best thing to start with – official docs. I think it described pretty clear although when it comes to php implementation you can have some questions. Especially with dates because php iso 8601 is different from what AWS expects. Thanks for this article it saves my time. So this what I finally got.

<?php namespace app\helpers; use Yii; use CJSON; class S3DirectUpload { public static function signData($fileName) { $resourceManager = Yii::app()->resourceManager;
        $bucket = $resourceManager->bucket;

        $algorithm = 'AWS4-HMAC-SHA256';
        $expiration = gmdate('Y-m-d\TG:i:s\Z', strtotime('+1 hour'));
        $date = gmdate("Ymd\THis\Z");

        $key = $resourceManager->basePath . 'document/' . md5($fileName . mt_rand()) . "/{$fileName}";

        $credentials = $resourceManager->key . '/' . date('Ymd') . '/' . $resourceManager->region . '/s3/aws4_request';

        $acl = 'private';

        $policy = [
            'expiration' => $expiration,
            'conditions' => [
                ['bucket' => $bucket],
                //['starts-with', 'document/'],
                ['starts-with', '$key', ''],
                ['starts-with', '$Content-Type', ''],
                ['acl' => $acl],
                ['success_action_status' => '201'],
                // ['success_action_redirect' => $redirectAction],
                ['x-amz-credential' => $credentials],
                ['x-amz-algorithm' => $algorithm],
                ['x-amz-date' => $date]
            ]
        ];

        $encodedPolicy = base64_encode(CJSON::encode($policy));

        return [
            'acl' => $acl,
            'key' => $key,
            //'success_action_redirect' => $redirectAction,
            'policy' => $encodedPolicy,
            'x-amz-algorithm' => $algorithm,
            'x-amz-credential' => $credentials,
            'x-amz-date' => $date,
            'x-amz-signature' => S3DirectUpload::generateSignature($encodedPolicy)
        ];
    }

    private static function generateSignature($encodedPolicy)
    {
        $resourceManager = Yii::app()->resourceManager;

        $dateKey = hash_hmac('sha256', date('Ymd'), "AWS4" . $resourceManager->secret, true);
        $dateRegionKey = hash_hmac('sha256', $resourceManager->region, $dateKey, true);
        $dateRegionServiceKey = hash_hmac('sha256', 's3', $dateRegionKey, true);
        $signingKey = hash_hmac('sha256', 'aws4_request', $dateRegionServiceKey, true);

        return hash_hmac('sha256', $encodedPolicy, $signingKey);
    }

}

Don’t forget about CORS.

    <CORSConfiguration>
      <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedHeader>*</AllowedHeader>
      </CORSRule>
    </CORSConfiguration>

Finally we would like to work blueimp file uploader


(function uploadToS3() {
    
    var dfd; // will need it for valiadation 
    $('#file_upload').fileupload({
        // 'forceIframeTransport': false,   // false false and false if you want to get progressbar
        autoUpload: true,
        acceptFileTypes: /(\.|\/)(pdf|doc|docx)$/i,
        maxNumberOfFiles: 1,
        dropZone: $(".dragdrop"),
        dataType: 'xml',
        /**
         * Document is uploaded to S3 need to save it to our DB
         * @param data
         */
        success: function (data) {
            var key = $(data).find('Key').text()
            $.ajax({
                url: "/document-s3/create",
                type: 'POST',
                dataType: 'json',
                data: {
                    key: key
                },
                success: function (s3Document) {
                    // this is response from our server(see fileuploadadd first) with created s3 document details 
                    // we get document s3 ID to save it to form
                    $("#DocumentUploadForm_document_s3_id").val(s3Document.id);
                    $("#DocumentUploadForm_document_s3_id").trigger('change');

                    $("#uploaded_name").text(s3Document.name);
                    $('.uploaded').show();
                    $('.progress').hide();
                }
            });
        }
    }).on('fileuploadfail', function (e, data) {
        $('.progress').hide();
    }).on('fileuploadsend', function (e, data) {
        $('.progress')
            .attr('aria-valuenow', 0)
            .children().first().css('width', 0 + '%');
        $('.dragdrop').hide();
        $('.progress').show();
    }).on('fileuploadprogress', function (e, data) {
        // This is what makes everything really cool, thanks to that callback
        // you can now update the progress bar based on the upload progress.
        var percent = Math.round((data.loaded / data.total) * 100);

        $('.progress')
            .attr('aria-valuenow', percent)
            .children().first().css('width', percent + '%');
    }).on('fileuploadadd', function (e, data) {
        $("#file_error_text").text('');

        var $form = $('#file_upload_form');
        var file = data.files[0];
        $form.find('input[name="Content-Type"]').val(file.type);

        dfd = jQuery.Deferred();
        // sign request to AWS 
        $.ajax({
            url: "/document-s3/sign",
            type: 'POST',
            dataType: 'json',
            data: {
                name: file.name
            },
            success: function (sign) {
                $.each(sign, function (key, value) {
                    $form.find('input[name=' + key + ']').val(value);
                });
                dfd.resolve();
            }
        });
        return false;
    }).on('fileuploadprocessalways', function (e, data) {
        if(!data.files.error) {
            dfd.promise().then(function() {
                // if file is valid AND we have got our signature send it to AWS
                data.submit();
            });
        } else {
            $("#file_error_text").text(data.files[0].error);
        }
    });
})();

A few details here.

  • we send request to our server to sign it – /document-s3/sign
  • then S3DirectUpload does his job. We get policy and signature back, plus other aws- fields
  • Finally we want to validate file (in my example we validate extension). Blueimp validates files and result of validation is accessible in fileuploadprocessalways event. If we pass validation we also need to ensure that we got response with signature from our server so I used promise there.
  • Finally we send request (direct upload request to s3). In the article that I mentioned at the beginning ‘forceIframeTransport’: true this approach won’t allow you to get progress info. We set it to false and can show progress in fileuploadprogress event. I used
  • When file is uploaded we want to fetch its success method. Response is XML so we do var key = $(data).find(‘Key’).text() and that’s it. We are done. I need to say that I found articles where people set up success_action_redirect action. But I cannot make it work with forceIframeTransport: false I got cross origin requests error. So thanks for this article that helped me with overriding success method hack.
  • In my example after getting key I send another request to server to save this upload to my DB.

Finally I want to add that I believe that my approach can be improved. I didn’t polish it. So comments are welcome.

Leave a Reply

Your email address will not be published. Required fields are marked *