graceful stop of php script

Today I’d like to tell you how I implemented graceful stop of php script.

Problem

We have some daemon script that handles some task. The problem that it’s in memory and in order to update it we need to restart it. The problem is such stop could lead to data missing. So we need to finish current task and only after that stop the script.

Solution

Php has pcntl_signal function that could catch signal from OS and then handle it.


class HandleEventsFromQueueCommand extends \om\console\Command
{

    private static $_stop = false;

    public function actionIndex()
    {
        $this->checkExit();

        while (true) {
            if (self::$_stop) {
                break;
            }
            // handle our events here
        }
    }

    private function checkExit()
    {
        if (!function_exists('pcntl_signal')) {
            return ;
        }
        $handler = function ($sigNumber) {
            if ($sigNumber == SIGTERM || $sigNumber == SIGHUP || $sigNumber == SIGINT){
                $this->_verbose("Process: " .getmypid() . " got signal $sigNumber and will exit.");
                self::$_stop = true;
            }
        };

        pcntl_signal(SIGTERM, $handler);
        pcntl_signal(SIGHUP,  $handler);
        pcntl_signal(SIGINT, $handler);
    }

}

It’s straightforward – catch exit signal and set flag inside your script that it’s time to stop.

Postgres doesn’t create indexes for foreign keys

Story about another #epicfail. I understood if everything is ok I don’t like to write about that cause it looks simple. But when something goes wrong :)

Well I worked with Postgres last time about 4 years ago. So definitely I wasn’t a good postgres guys. Last 4 years in most cases I worked with MySql. And mysql automatically creates indexes for foreign keys. Some kind of “second system effect”. When you even don’t think about indexes cause you’re sure that they already should be created.

Well my topic would be useless if I didn’t provider any elegant solution. Right ?

So feel the power of postgres.


SELECT
	tc. TABLE_NAME,
	kcu. COLUMN_NAME
FROM
	information_schema.table_constraints AS tc
JOIN information_schema.key_column_usage AS kcu ON tc. CONSTRAINT_NAME = kcu. CONSTRAINT_NAME
JOIN information_schema.constraint_column_usage AS ccu ON ccu. CONSTRAINT_NAME = tc. CONSTRAINT_NAME
WHERE
	constraint_type = 'FOREIGN KEY'
ORDER BY table_name

This will return table and column names for ALL foreign keys. Then you can just take this result to generate indexes.

Do not repeat my errors.

#nastyBug Http Codes: Don’t play with me

Wanna tell you story how I spent a lot of time to fix one #nastybug.

I like to begin my story with intrigue :)

Right now I work on mobile app. I do backend side. And we have native iphone app. All was well until one day we started getting errors “request body stream exhausted” OR we just don’t have response from server and got timeout error. Try to google it and you won’t find quick answer what’s wrong. Initially we thought that something wrong with Content-Size. But no… it is clear that nothing is clear

I checked my logs everything is ok. I checked access and return error in json. We checking mobile app and it just doesn’t work.

Finally we started to check all the logs in app in server on mobile side. AAAAAND what we see server return json but with code 100 What the F. Code 100. And here I realized that I have custom error code like 1001, 1002 etc. Ok let’s check the code.

I think everybody how worked with yii and need to return something in JSON write functions like this.

    protected function _returnJson($data, $code = 200)
    {
        $this->layout = false;
        header('Content-type: application/json');
        if ($code) {
            header("HTTP/1.1 {$code} " . $this->getHttpHeader($code));
        }

        echo CJSON::encode($data);
    }

    /**
     * Send json error
     * @param string $error
     * @param null $code
     * @param null $description
     */
    protected function _returnJsonError($error = 'Sorry an error occurred', $code = 400, $description = null)
    {
        $this->layout = false;
        header('Content-type: application/json');
        header("HTTP/1.1 {$code} " . $this->getHttpHeader($code));

        echo CJSON::encode([
            'errors' => [
                'code' => $code,
                'message' => $error,
                'description' => $description
            ]
        ]);
    }

I also wrote such functions at some point. And they “nomadize” from one my project to another. Do you see anything suspicious? Looks good to me.

And these functions have been working really good. Until I need to send specific error code. But STOP it’s too absolute different codes. HTTP code and my custom error codes. And there is no way to mix.

So I’ve reworked interface to

_returnJsonError($error = ‘Sorry an error occurred’, $internalErrorCode = null,
$description = null, $httpCode = 400)

This fixed all the problems. So be careful HTTP codes it’s not for kids :)

Single transaction approach performance in tests

When your project is big enough and amount of tests becoming bigger and bigger the speed of test execution becomes very low.

I had 100+ tests in my project. In my local machine it takes about 20 minutes to run all tests on amazon EC2 small instance it was even slower.

So I started to think wooow why it’s so SLOOOOOW. Actually I knew the answer and I think you also should have a hunch that fixture loading is the bottleneck here.

There are a few approaches to speed up fixture loading, that I know

  • use memory tables
  • single transaction approach

As for memory tables. So the idea is use memory tables Well I see some big disadvantages. First it’s not a copy of you database, i.e. you don’t have foreign key checks, you don’t have some abilities that your tables could have. Secondly this is useful to only some specific databases as mysql.

So we will talk about second approach. The idea is to begin transaction before each test and rollback it after each test. This should be much faster than reloading data each time.

Looking ahead, I will say that after the implementation of this approach speed execution became 40 second versus 20 minutes for 102 tests.

But what is bottlenecks here and what should you do to use it. For me it were 2 suddenness.

The main problem for me was following. I had functional and unit tests. For unit everything was perfect but for functional a half of tests were broken. What happened? Answer is easy. Functional tests tests real http request and the client don’t know anything about you transaction. So we need to run single transaction too on clients. Well ok for tests it’s quite easy to do. For example yii even uses specific entry script for test index-test.php, or you can pass some secret header.

if (APPLICATION_ENV != 'production' && $anyAditionalCheckThatYouNeedHere) {
    $config = require $pathToMyConfigFile;
    $config['onBeginRequest'] = function() {
        // use single transaction approach for tests. Begin but do not commit.
        \Yii::app()->getDb()->beginTransaction();
    };
}
This is not the end such test also won’t work :)
    $testUser = $this->users('serg');
    $result = $this->put("/my/api/{$testUser->id}", $data);

    $this->assertEquals($expectedId, $result['id']);

    // oh no my dear :D 
    $testPatient->refresh();

This means that in functional tests you must test only interface do not check that DB changes, check your response what it has, what it should have. Check all DB changes in unit tests.

Implemented this approach I’m really happy to see how fast tests execute.

PHP: test email sending using Yii example

Remember that post http://radzserg.com/2013/06/06/how-do-i-send-emails-in-my-applications/ about sending emails from your applications?

It’s time to test your emails ;).

Ok, so what do we have  \Zend\Mail\Transport\File – ideal for testing. We won’t send emails, but save them as files. Then after some test executed we need to get last sent email and check content and header.

I’ve created a simple class for that.

<?php

namespace myproject\tests;

use Yii;

/**
 * Assumes that we use \Zend\Mail\Transport\File for tests
 * @package om\tests
 */
class LastSentEmail
{

    private $_content;
    private $_headers;

    public function __construct()
    {
        $files = glob(Yii::getPathOfAlias('path.where.you.save.emails') . "/ZendMail_*");
        if (empty($files)) {
            throw new \CException("Email wasn't sent");
        }
        $filePath = end($files);

        $emailContent = file_get_contents($filePath);

        $this->_parseEmailBody($emailContent);
    }

    public function contains($search)
    {
        return stripos($this->_content, $search) !== false;
    }

    public function subjectContains($search)
    {
        return stripos($this->_headers['subject'], $search) !== false;
    }

    public function getHeaders()
    {
        return $this->_headers;
    }

    public function getHeader($name)
    {
        return isset($this->_headers) ? $this->_headers[$name] : null;
    }

    public function getSubject()
    {
        return $this->getHeader('subject');
    }

    public function getBody()
    {
        return $this->_content;
    }

    private function _parseEmailBody($content)
    {
        $pos = 0;
        foreach (explode("\r\n", $content) as $line) {
            $pos += mb_strlen($line) + 2;
            if ($line == "") {
                break;
            }

            list($headerName, $headerValue) = explode(":", $line, 2);

            $this->_headers[strtolower($headerName)] = trim($headerValue);
        }
        $this->_content = substr($content, $pos);
    }

}

Although we need a  little trick here for \Zend\Mail\Transport\FileOptions this will help us to order files in correct order.

$fileOptions = new \Zend\Mail\Transport\FileOptions();
$fileOptions->setPath($this->saveEmailPath);
$fileOptions->setCallback(function ($transport) {
    list($usec, $sec) = explode(" ", microtime());
    $time = ((float)$usec + (float)$sec);
    return "ZendMail_{$time}.tmp";
});
self::$_transport = new \Zend\Mail\Transport\File($fileOptions);

That’s it. Let’s test.


$email = new LastSentEmail();
$this->assertContains('subject pattern', $email->getSubject());
$this->assertContains('body pattern', $email->getBody());

X-Send File

Found nice feature in Yii X-Send.

Here’s a flow.

Say you need to protect your images. So they shouldn’t be public available for all users. In this case you need to check user right and then return image.

If your application is highloaded, returning images for php could be expensive task. Yii provide ability to send images CHttpRequest->sendFile

$content=function_exists('mb_substr') ? mb_substr($content,$contentStart,$length) : substr($content,$contentStart,$length);

So you have to read file content. There’s another approach with fpassthru function but looks like yii provide more accurate data for headers.

Anyway there is much simpler and elegant solution.

xSendFile

In this case server will do all the work for you. Everything is quite straightforward. The only one trick that I made is extension for nginx

 public function xSendFile($filePath, $options=array())
    {
        if (getenv('SERVER_NGINX')) {
            $options['xHeader'] = 'X-Accel-Redirect';
            $filePath = str_replace(\Yii::getPathOfAlias('root'), '', $filePath);
        }

        parent::xSendFile($filePath, $options);
    }

Ann to nginx config

location /uploaded/images {
internal;
root /var/mysite.com/;
}

Where I was last 3 months

3 months have passed since I wrote last time here. During this time I changed the 2 projects and  worked with many new technologies. Unfortunately I’m still really busy and just don’t have time and effort to write here :) But still I have a very mall update.

Also I helped to consult LAMP group in softserve university for about last 3 months and they have graduated yesterday. Congrats guys. Hope it was fine.

c61e8f8613ee11e3b6c622000a1f92d1_7

Yii migration use change in addition to up and down

Finally I pushed some of my yii extensions to github.
https://github.com/radzserg/yii-Rz

Still need to work on README.md I have lack of time now. All in all I think I can refer to it now.

And as a bonus I will tell about my extension for CDbMigration. A lot of migration tools/lib already support change() method in addition to up() and down(). 

But the main problem here – it’s easy to implement it if we create table or add a column or index in this case we know that we need to do in down action. It’s easy we need to remove appropriate object. For example:

    public function up()
    {
        $this->addColumn('user', 'email', 'varchar(255) AFTER username');
        $this->addColumn('user', 'password', 'varchar(255) AFTER email');
    }

    public function down()
    {
        $this->dropColumn('user', 'email');
        $this->dropColumn('user', 'password');
    }

Easy enough. But what do we need to do in down action if we create table. In this case everything is quite complicated. We either need to keep DB snapshot for each migration. Or create it going through migration until some migration that you need. Both quite difficult to implement and I guess even if you can implement such approach you still have a lot of problem.

The good thing here that in most migrations we add or create something. And we don’t need to care about down function. And in that cases when we need to implement down functionality we will do that using closures.

Here are my code
https://github.com/radzserg/yii-Rz/blob/master/db/ChangeMigration.php

Here are examples of some of my migrations

<?php

class m130626_132012_add_not_confirmed_users extends Rz\db\ChangeMigration
{
    public function change()
    {
        $this->addColumn('sm_user', 'confirmed', 'TINYINT(1) NOT NULL DEFAULT 0 after create_time');
        $this->addColumn('user', 'confirmed', 'TINYINT(1) NOT NULL DEFAULT 0 after create_time');

        $this->alterColumn('sm_user', 'user_id', 'INT', function() {
           $this->alterColumn('sm_user', 'user_id', 'INT NOT NULL');
        });
    }

}

When you run yiic migrate down. Migration will save queue of back actions in this example

  • remove column sm_user.confirmed (autogenerated)
  • remove column user.confirmed (autogenerated)
  • alter sm_user change user_id to int not null (our manual down closure)

And finally execute all this action in reverse order that’s it. No need to care about down functionality and order.

I proposed that approach to yii core team https://github.com/yiisoft/yii2/issues/593#issuecomment-20397820 but they refuse it. Well maybe it’s correct when we talk about framework. Anyway I decided to implement and try it. And for now I like it :)

Try it too and let me know do you like it or no.

Yii through Mysql perfomance

Today I’d like to tell about interesting question which caused me to think and to do some research.

Let’s look at example from official guide
http://www.yiiframework.com/doc/guide/1.1/en/database.arr#relational-query-with-through
Another nice relation type, moreover you can build it recursively. Group has users through roles. Group has comments through users, through roles. Very flexible and convenient.

But does anybody look inside built queries. Yii will build query like this. Even let’s imagine how I will build such query.

SELECT *
FROM role
    JOIN user ON user.id = role.user_id
    JOIN comment ON user.id = comment.user_id
WHERE  role.group_id = 123

And now let’s see what we get with yii

SELECT comment.*
FROM comment
    LEFT JOIN user ON user.id = comment.user_id
    LEFT JOIN role ON role.user_id = user.id
WHERE  role.group_id = 2870

It happens because yii build query recursively go down from first relation. A few WHY from my side.

  • why LEFT join
  • I expected to see everything viceversa
  • If user can belong to few roles – final result won’t be unique, i.e. we need to add DISTINCT. Yii will filter it by themself.

I’m really not a pro DB developer. So perhaps there are a lot of errors in my reasoning.

All in all let’s see… In terms of SQL this means that we take ALL rows from comment then trying to join user then trying to join role. Then will apply where condition to received data snapshot. Plus some internal server JOIN optimization. I used only mysql, so unfortunately can’t provide feedback about other servers. I thought so :)

Ok then I decieded to check my assumption. I created tables, fill some data. They have from mln – 6 mln rows, each one. And started to EXPLAIN

EXPLAIN SELECT *
FROM role
	JOIN user ON user.id = role.user_id
	JOIN comment ON user.id = comment.user_id
WHERE  role.group_id = 2897;

explain_human_queries

EXPLAIN SELECT comment.*
FROM comment
    LEFT JOIN user ON user.id = comment.user_id
    LEFT JOIN role ON role.user_id = user.id
WHERE  role.group_id = 2897

explain_yii_query

Heh… Mysql is smart enough to start from role table in both cases. Take Five. I think in most cases this will work. And you’ll get almost the same performance.

But some issues is still relevant
- why LEFT join
- is it smart to get unique data using yii
- does join server optimization will work for all supported db servers

Thoughts ?