Category Archives: Uncategorized

Direct upload to s3 with php AWS4 progressbar blueimp file uploader validation

Continue to talk about AWS and S3. Today I will share my experience in direct upload to s3. This could be useful when it comes to big files. You can upload file directly to S3.

  • No need to wait twice
  • progress bar works correct. If you upload through your server progress bar will show you only upload process to your server but then your users will wait until you upload file to s3

If you got to bluimp docs you will find the arcticle in the wiki but it use old algorithm and you progressbar won’t work. Also you will find out that blueimp validation won’t work as expected.

Continue reading

Upload big files to s3 by URL

Recently I needed to upload files to s3 having URL. Unfortunately I didn’t find a way how can I do that with native AWS php sdk. So I needed to provide either file or file body Upload an Object Using the AWS SDK for PHP . I didn’t want to care about temp files. So the easiest solution that came up to my head were

// @see https://github.com/2amigos/resource-manager/
$body = @get_file_content($url);
Yii::app()->resourceManager->save($body, $path, $options);

That worked fine, I also could get file size and mime type having file body. Maybe not very elegant but it worked. Until…

Continue reading

Javascript module initialization

Today we gonna work with javascript modules. What is it and why do we need to use this approach.

Nothing special – it’s logically separated pieces of your javascript. Wait but I can add my javascript to views where they need it and it will be separated from other views where it’s useless. I don’t have so much JS in my app.

php and js spaghetti

You know… it sucks… it looks like spaghetti code. You mix javascript and php.  And say very big “thank you” for PhpStorm that allows you to separate and highlight javascript.

Ok, so let’s try to separate view from javascript. From the example below I still need to get some data from backend side($typeLink). Sometimes we need to pass some constant or URL. I’d suggest to forget about passing that from backend side. I’m a great supporter of “API First” approach. After my openmed project where I did only the API and we have separate frontend web application that interacted with API I must admin it’s extremely cool. You don’t need to take care about view you are concentrated only on logic. But this is another story. Perhaps I’ll write about that later. Ok we digress… so my piece of advise – try to separate js from your app as much as possible, try to make it independent.

At this point we can move all our javascript code to js files and include all of them. But here we face with some problems. First your javascript still is absolutely unstructured. The increasing of javascript will exacerbate this problem. So let’s add some logic to our javascript put logic related to posts to javascript post module.

Yii2 comes with updated javasctipt module structure. Check it out  read comments. You can use this approach for your own app.

// initially you create your main application module that will
// load all his child modules
myapp = (function ($) {
    'use strict';

    var pub = {
        initModule: function (module) {
            if (module.isActive === undefined || module.isActive) {
                if ($.isFunction(module.init)) {
                    module.init();
                }
                $.each(module, function () {
                    if ($.isPlainObject(this)) {
                        pub.initModule(this);
                    }
                });
            }
        },
        init: function () {
            // common for all you project javascript logic
            initLayoutComponents();
            // whatever you want
        }
    };
    // define private functions
    function initLayoutComponents() {
    }

    return pub;
})(jQuery);

jQuery(document).ready(function () {
    myapp.initModule(myapp);
});

// in other files you can define child modules
if (typeof myapp == "undefined" || !myapp) {
    var myapp = {};
}

myapp.post = (function ($) {

    var pub = {
        isActive: true,
        init: function () {
            initPostComments()
        },
    }

    function initPostComments() {
    }
    return pub;
})(jQuery);

OK now we have structured javascript. That located separately from our views. That a good point. But there other issue that we have is – have we gonna include this files.

  • We can include all javascript or even better concatenate it in one file and include once. It will be cached and work much faster. Did you notice isActive param for javasctipt modules. if all modules have it equals true it means that we gonna execute all our javascript. And in most cases we don’t need to do that. Module could init components that currently not in our page. Some page has map, some has comments widget etc. So we need to take care about dynamic module initialization.
  • OR we need to include specific javascript module (or file) for specific page. But this approach even worse since you still need to take care about dynamic includes but lost all profits from caching.

Then I’ll tell you about 2 approaches how you could do that. The easiest one is  just include what you need in your views.

$this->registerJs('myapp.post.isActive = true;', $this::POS_END);
$this->registerJs('myapp.author.isActive = true;', $this::POS_END);

The advantage of such approach that you have full control on what do you include. But I’m pretty sure that in most cases you don’t need such level of control. Using convention over configuration principle you can adjust that for all you controller you’ll have specific module for all contoler’s action you have the function inside your module or another module like controller_action.js. Using this assertion you can easily add some helper for yii and initiate modules base on current route.

Finally for a change I will add one more example. Where I will initiate module base on data-attributes of body. I have rails project so you’ll have coffee script 😉

###

###


'use strict'

appRunner =
  exec: (controller, action = "init") ->
    ns = window.app

    if controller != "" && ns[controller] && typeof ns[controller][action] == "function"
      ns[controller][action]()

  init: ()  ->
    body = document.body
    controller = body.getAttribute( "data-controller" )
    action = body.getAttribute( "data-action" )

    appRunner.exec("common")
    appRunner.exec(controller)
    appRunner.exec(controller, action)

window.app.common =
  init: () ->


$(document).ready(()->
  appRunner.init()
)

As you can see here you have some common for your app logic and module with methods per your actions.

Resume: don’t mix js and PHP/Ruby/Whatever. Keep your code clean. Thanks.

Socket.io 1.0 emitting events from php

Continue to play with socket.io. Last time I used 0.8 version. Right now socket.io has 1.0 with a lot of updates. So let’s get started.

The most cool thing IMO is adapter

var io = require('socket.io')(serverForUI),
io.adapter(redis({ host: 'localhost', port: 6379 }));

I’m brand new guy in node world. So for me explanation in docs was a little bit confusing. But this 2 lines of code add a lot of magic. Right now you can publish events to redis (in specific format) and after that those events will automatically go to client.
In order to emit some data from php. I used socket.io-php-emitter. So all we need to do is:

// node server
var serverForUI = require('http').createServer(),
    redisAdapter = require('socket.io-redis'),
    io = require('socket.io')(serverForUI);

io.adapter(redisAdapter({ host: 'localhost', port: 6379 }));

serverForUI.listen(8000);

// emit events from php side

<?php

namespace myapp;

use SocketIO\emitter;

class OrganizationUpdates
{

    public $host;
    public $port = 6379;

    private $_emitter;

    /**
     * Return redis queue
     * @return emitter
     */
    private function _emitter()
    {
        if (!$this->_emitter) {
            $redis = new \Redis();
            $redis->connect($this->host, $this->port);

            $this->_emitter = new emitter($redis);
        }
        return $this->_emitter;
    }

    public function publishNewEvent($eventType, $data)
    {
        $this->_emitter()->emit("news", [
            'event_type' => $eventType,
            'data' => $data
        ]);
    }
}

The last thing is our client example

<!doctype html>
<html>
<head>
    <title>Test</title>
</head>
<body>
<script src="https://cdn.socket.io/socket.io-1.0.6.js"></script>
<script>
    var socket = io('http://localhost:8000', {
        transports: ['websocket']
    });

    socket.on('news', function (data) {
        console.log(data);
    });
</script>
</body>
</html>

That’s it. Next time I’ll tell you about socket.io authorization and using rooms.

graceful stop of php script

Today I’d like to tell you how I implemented graceful stop of php script.

Problem

We have some daemon script that handles some task. The problem that it’s in memory and in order to update it we need to restart it. The problem is such stop could lead to data missing. So we need to finish current task and only after that stop the script.

Solution

Php has pcntl_signal function that could catch signal from OS and then handle it.


class HandleEventsFromQueueCommand extends \om\console\Command
{

    private static $_stop = false;

    public function actionIndex()
    {
        $this->checkExit();

        while (true) {
            if (self::$_stop) {
                break;
            }
            // handle our events here
        }
    }

    private function checkExit()
    {
        if (!function_exists('pcntl_signal')) {
            return ;
        }
        $handler = function ($sigNumber) {
            if ($sigNumber == SIGTERM || $sigNumber == SIGHUP || $sigNumber == SIGINT){
                $this->_verbose("Process: " .getmypid() . " got signal $sigNumber and will exit.");
                self::$_stop = true;
            }
        };

        pcntl_signal(SIGTERM, $handler);
        pcntl_signal(SIGHUP,  $handler);
        pcntl_signal(SIGINT, $handler);
    }

}

It’s straightforward – catch exit signal and set flag inside your script that it’s time to stop.

Single transaction approach performance in tests

When your project is big enough and amount of tests becoming bigger and bigger the speed of test execution becomes very low.

I had 100+ tests in my project. In my local machine it takes about 20 minutes to run all tests on amazon EC2 small instance it was even slower.

So I started to think wooow why it’s so SLOOOOOW. Actually I knew the answer and I think you also should have a hunch that fixture loading is the bottleneck here.

There are a few approaches to speed up fixture loading, that I know

  • use memory tables
  • single transaction approach

As for memory tables. So the idea is use memory tables Well I see some big disadvantages. First it’s not a copy of you database, i.e. you don’t have foreign key checks, you don’t have some abilities that your tables could have. Secondly this is useful to only some specific databases as mysql.

So we will talk about second approach. The idea is to begin transaction before each test and rollback it after each test. This should be much faster than reloading data each time.

Looking ahead, I will say that after the implementation of this approach speed execution became 40 second versus 20 minutes for 102 tests.

But what is bottlenecks here and what should you do to use it. For me it were 2 suddenness.

The main problem for me was following. I had functional and unit tests. For unit everything was perfect but for functional a half of tests were broken. What happened? Answer is easy. Functional tests tests real http request and the client don’t know anything about you transaction. So we need to run single transaction too on clients. Well ok for tests it’s quite easy to do. For example yii even uses specific entry script for test index-test.php, or you can pass some secret header.

if (APPLICATION_ENV != 'production' && $anyAditionalCheckThatYouNeedHere) {
    $config = require $pathToMyConfigFile;
    $config['onBeginRequest'] = function() {
        // use single transaction approach for tests. Begin but do not commit.
        \Yii::app()->getDb()->beginTransaction();
    };
}
This is not the end such test also won’t work 🙂
    $testUser = $this->users('serg');
    $result = $this->put("/my/api/{$testUser->id}", $data);

    $this->assertEquals($expectedId, $result['id']);

    // oh no my dear 😀
    $testPatient->refresh();

This means that in functional tests you must test only interface do not check that DB changes, check your response what it has, what it should have. Check all DB changes in unit tests.

Implemented this approach I’m really happy to see how fast tests execute.

PHP: test email sending using Yii example

Remember that post http://radzserg.com/2013/06/06/how-do-i-send-emails-in-my-applications/ about sending emails from your applications?

It’s time to test your emails ;).

Ok, so what do we have  \Zend\Mail\Transport\File – ideal for testing. We won’t send emails, but save them as files. Then after some test executed we need to get last sent email and check content and header.

I’ve created a simple class for that.

<?php

namespace myproject\tests;

use Yii;

/**
 * Assumes that we use \Zend\Mail\Transport\File for tests
 * @package om\tests
 */
class LastSentEmail
{

    private $_content;
    private $_headers;

    public function __construct()
    {
        $files = glob(Yii::getPathOfAlias('path.where.you.save.emails') . "/ZendMail_*");
        if (empty($files)) {
            throw new \CException("Email wasn't sent");
        }
        $filePath = end($files);

        $emailContent = file_get_contents($filePath);

        $this->_parseEmailBody($emailContent);
    }

    public function contains($search)
    {
        return stripos($this->_content, $search) !== false;
    }

    public function subjectContains($search)
    {
        return stripos($this->_headers['subject'], $search) !== false;
    }

    public function getHeaders()
    {
        return $this->_headers;
    }

    public function getHeader($name)
    {
        return isset($this->_headers) ? $this->_headers[$name] : null;
    }

    public function getSubject()
    {
        return $this->getHeader('subject');
    }

    public function getBody()
    {
        return $this->_content;
    }

    private function _parseEmailBody($content)
    {
        $pos = 0;
        foreach (explode("\r\n", $content) as $line) {
            $pos += mb_strlen($line) + 2;
            if ($line == "") {
                break;
            }

            list($headerName, $headerValue) = explode(":", $line, 2);

            $this->_headers[strtolower($headerName)] = trim($headerValue);
        }
        $this->_content = substr($content, $pos);
    }

}

Although we need a  little trick here for \Zend\Mail\Transport\FileOptions this will help us to order files in correct order.

$fileOptions = new \Zend\Mail\Transport\FileOptions();
$fileOptions->setPath($this->saveEmailPath);
$fileOptions->setCallback(function ($transport) {
    list($usec, $sec) = explode(" ", microtime());
    $time = ((float)$usec + (float)$sec);
    return "ZendMail_{$time}.tmp";
});
self::$_transport = new \Zend\Mail\Transport\File($fileOptions);

That’s it. Let’s test.


$email = new LastSentEmail();
$this->assertContains('subject pattern', $email->getSubject());
$this->assertContains('body pattern', $email->getBody());

X-Send File

Found nice feature in Yii X-Send.

Here’s a flow.

Say you need to protect your images. So they shouldn’t be public available for all users. In this case you need to check user right and then return image.

If your application is highloaded, returning images for php could be expensive task. Yii provide ability to send images CHttpRequest->sendFile

$content=function_exists('mb_substr') ? mb_substr($content,$contentStart,$length) : substr($content,$contentStart,$length);

So you have to read file content. There’s another approach with fpassthru function but looks like yii provide more accurate data for headers.

Anyway there is much simpler and elegant solution.

xSendFile

In this case server will do all the work for you. Everything is quite straightforward. The only one trick that I made is extension for nginx

 public function xSendFile($filePath, $options=array())
    {
        if (getenv('SERVER_NGINX')) {
            $options['xHeader'] = 'X-Accel-Redirect';
            $filePath = str_replace(\Yii::getPathOfAlias('root'), '', $filePath);
        }

        parent::xSendFile($filePath, $options);
    }

Ann to nginx config

location /uploaded/images {
internal;
root /var/mysite.com/;
}