Socket IO authorization

I’d like to finish my story about my experiments with nodejs and socket.io. Check out my previous post http://radzserg.com/2014/07/01/socket-io-1-0-emitting-events-from-php/ .The last thing that I wanted to do is authorization. So imagine the following task we want to get updates for specific organization. My idea was quite simple in this case we can send authorization data (username:password or oauth token) to server plus send extra param like organization id

// let's add this param on frontend 
 var socket = io('http://localhost:8000', {
    query: "token=[oauth token]&organizationId=" + organizationId,
    transports: ['websocket']
});

Let’s add DB service that actually will perform authorization

var config = require('./src/config.json')
var pg = require('pg')
client = new pg.Client(config.pg.cdn);
client.connect(function (error) {
    if (error) {
        return console.error('could not connect to postgres', error);
    }
})

var dbService = require('./src/services/postgres_service.js')(client);

io.set('authorization', function (handshakeData, cb) {
    dbService.isUserAuthorized(handshakeData._query, function (result, error) {
        if (error) {
            return cb(error, false);
        }
        return cb(null, result);
    });
});

io.on('connection', function (socket) {
    // if we pass authorization join socket to organization room 
    socket.join("organization." + socket.request._query.organizationId);
});

An example of DB service could looks like this


module.exports = function (client) {

    return {
        isUserAuthorized: function (data, next) {
            var token = data.token ? data.token : null;
            if (!token) {
                return next(false, "Token must be specified");
            }
            var organizationId = data.organizationId ? data.organizationId : null;
            if (!organizationId) {
                return next(false, "organizationId must be specified");
            }

            var sql;
            sql = 'SELECT user_id FROM oauth_access_tokens WHERE access_token = $1 AND expires > NOW()';
            var query = client.query(sql, [token], function (error, result) {
                if (error) {
                    return next(false, error);
                }
                var userId = typeof result.rows[0] !== 'undefined' ? result.rows[0]['user_id'] : null;
                if (!userId) {
                    return next(false);
                }
                // then required queries to check if user has access to required organization. 
                // ... 
                return next(true)              
            });
        }
    }
};

Finally on PHP side we also can send events to specific room

$this->_emitter()->in("organization.{$orgnizationId}")->emit("news", [
     'data' => $data
]);

For now that’s it. Make a pause with nodejs. Although it was nice to play with it.

Bootstrap form helpers get instances

Worked with bootstrap form helpers library. Looks nice and great integration with twitter bootstrap. But spent a lot of time to figure out how to access bfhdatepicker and bfhtimepicker objects. Unfortunately googling didn’t get any answer. As usual key was simple :) Hope this will save time for someone else.

   var datePicker = $('#datepicker_id').data('bfhdatepicker');
   datePicker.setDate('today')

   var timePicker = $('#timepicker_id').data('bfhtimepicker');
   timePicker.setTime('now')

Socket.io 1.0 emitting events from php

Continue to play with socket.io. Last time I used 0.8 version. Right now socket.io has 1.0 with a lot of updates. So let’s get started.

The most cool thing IMO is adapter

var io = require('socket.io')(serverForUI),
io.adapter(redis({ host: 'localhost', port: 6379 }));

I’m brand new guy in node world. So for me explanation in docs was a little bit confusing. But this 2 lines of code add a lot of magic. Right now you can publish events to redis (in specific format) and after that those events will automatically go to client.
In order to emit some data from php. I used socket.io-php-emitter. So all we need to do is:

// node server
var serverForUI = require('http').createServer(),
    redisAdapter = require('socket.io-redis'),
    io = require('socket.io')(serverForUI);

io.adapter(redisAdapter({ host: 'localhost', port: 6379 }));

serverForUI.listen(8000);

// emit events from php side

<?php

namespace myapp;

use SocketIO\emitter;

class OrganizationUpdates
{

    public $host;
    public $port = 6379;

    private $_emitter;

    /**
     * Return redis queue
     * @return emitter
     */
    private function _emitter()
    {
        if (!$this->_emitter) {
            $redis = new \Redis();
            $redis->connect($this->host, $this->port);

            $this->_emitter = new emitter($redis);
        }
        return $this->_emitter;
    }

    public function publishNewEvent($eventType, $data)
    {
        $this->_emitter()->emit("news", [
            'event_type' => $eventType,
            'data' => $data
        ]);
    }
}

The last thing is our client example

<!doctype html>
<html>
<head>
    <title>Test</title>
</head>
<body>
<script src="https://cdn.socket.io/socket.io-1.0.6.js"></script>
<script>
    var socket = io('http://localhost:8000', {
        transports: ['websocket']
    });

    socket.on('news', function (data) {
        console.log(data);
    });
</script>
</body>
</html>

That’s it. Next time I’ll tell you about socket.io authorization and using rooms.

Manage postresql ENUM-s on RDS

If you like to work with ENUM fields in DB you can face some problems when you need to add, update or delete enum value. Although Postgres provides some useful tools for that http://www.postgresql.org/docs/9.1/static/sql-altertype.html it’s quite difficult to update or delete ENUM values. Moreover if you use RDS you’ll find out that RDS do not provide you grants to manage ENUM. I prepared some usefull set of queries on today decided to share this code in case it will be useful for someone


<?php

namespace om\db;

abstract class PgMigration extends \CDbMigration
{

    /**
     * Renames ENUM value
     * @param $table
     * @param $field
     * @param $enumName
     * @param $newEnumValue
     * @param $oldEnumValue
     * @throws \CDbException
     * @throws \CException
     */
    public function renameEnumValue($table, $field, $enumName, $newEnumValue, $oldEnumValue)
    {
        $existedValues = $this->getDbConnection()->createCommand("SELECT enum_range(NULL::{$enumName})")->queryScalar();
        $existedValues = substr($existedValues, 1, -1);
        $existedValues = explode(',', $existedValues);
        $existedValues = array_diff($existedValues, [$oldEnumValue]);
        $existedValues[] = $newEnumValue;
        $existedValues = array_unique($existedValues);

        $default = $this->_getColumnDefaultValue($table, $field);
        $isNull = $this->_getColumnIsNull($table, $field);

        $queries = [
            "ALTER TYPE \"{$enumName}\" RENAME TO \"old_{$enumName}\"",
            "CREATE TYPE \"{$enumName}\" AS ENUM ('" . implode("', '", $existedValues) . "')",
            "ALTER TABLE \"{$table}\" RENAME COLUMN \"{$field}\" TO \"old_{$field}\"",
            "ALTER TABLE \"{$table}\" ADD temp_{$field} VARCHAR",
            "ALTER TABLE \"{$table}\" ADD {$field} {$enumName}" . ($default ? " DEFAULT {$default}" : ""),
            "UPDATE \"{$table}\" SET temp_{$field} = old_{$field}::text",
            "UPDATE \"{$table}\" SET temp_{$field} = '{$newEnumValue}' WHERE temp_{$field} = '{$oldEnumValue}'",
            "UPDATE \"{$table}\" SET {$field} = temp_{$field}::{$enumName}",
            $isNull ? "SELECT 1" : "ALTER TABLE \"{$table}\" ALTER COLUMN {$field} SET NOT NULL",
            "ALTER TABLE \"{$table}\" DROP COLUMN old_{$field}",
            "ALTER TABLE \"{$table}\" DROP COLUMN temp_{$field}",
            "DROP TYPE old_{$enumName}"
        ];

        foreach ($queries as $query) {
            $this->getDbConnection()->createCommand($query)
                ->execute();
        }
    }

    /**
     * Adds new value to ENUM
     * @param $table
     * @param $field
     * @param $enumName
     * @param $newValue
     * @throws \CDbException
     * @throws \CException
     */
    public function addEnumValue($table, $field, $enumName, $newValue)
    {
        $existedValues = $this->getDbConnection()->createCommand("SELECT enum_range(NULL::{$enumName})")->queryScalar();
        $existedValues = substr($existedValues, 1, -1);
        $existedValues = explode(',', $existedValues);
        $existedValues[] = $newValue;

        $default = $this->_getColumnDefaultValue($table, $field);
        $isNull = $this->_getColumnIsNull($table, $field);

        $queries = [
            "ALTER TYPE {$enumName} RENAME TO old_{$enumName}",
            "CREATE TYPE {$enumName} AS ENUM ('" . implode("', '", $existedValues) . "')",
            "ALTER TABLE \"{$table}\" RENAME COLUMN {$field} TO old_{$field}",

            "ALTER TABLE \"{$table}\" ADD {$field} {$enumName}" . ($default ? " DEFAULT {$default}" : ""),
            "UPDATE \"{$table}\" SET {$field} = old_{$field}::text::{$enumName}",
            "ALTER TABLE \"{$table}\" DROP COLUMN old_{$field}",
            "DROP TYPE old_{$enumName}"
        ];

        if ($isNull) {
            $queries[] = "ALTER TABLE \"{$table}\" ALTER COLUMN {$field} SET NOT NULL";
        }

        foreach ($queries as $query) {
            $this->getDbConnection()->createCommand($query)
                ->execute();
        }
    }

    /**
     * Remove ENUM value
     * @param $table
     * @param $field
     * @param $enumName
     * @param $deleteValue
     * @param $replaceValue
     * @throws \CDbException
     * @throws \CException
     */
    public function removeEnumValue($table, $field, $enumName, $deleteValue, $replaceValue)
    {
        $existedValues = $this->getDbConnection()->createCommand("SELECT enum_range(NULL::{$enumName})")->queryScalar();
        $existedValues = substr($existedValues, 1, -1);
        $existedValues = explode(',', $existedValues);
        $existedValues = array_diff($existedValues, [$deleteValue]);

        $default = $this->_getColumnDefaultValue($table, $field);
        $isNull = $this->_getColumnIsNull($table, $field);

        $queries = [
            "ALTER TYPE {$enumName} RENAME TO old_{$enumName}",
            "CREATE TYPE {$enumName} AS ENUM ('" . implode("', '", $existedValues) . "')",
            "ALTER TABLE \"{$table}\" RENAME COLUMN {$field} TO old_{$field}",
            "ALTER TABLE \"{$table}\" ADD {$field} {$enumName}" . ($isNull ? "" : " NOT NULL")
            . ($default ? " DEFAULT {$default}" : ""),
            "UPDATE \"{$table}\" SET old_{$field} = '{$replaceValue}' WHERE old_{$field} = '{$deleteValue}'",
            "UPDATE \"{$table}\" SET {$field} = old_{$field}::text::{$enumName}",
            "ALTER TABLE \"{$table}\" DROP COLUMN old_{$field}",
            "DROP TYPE old_{$enumName}"
        ];

        foreach ($queries as $query) {
            $this->getDbConnection()->createCommand($query)
                ->execute();
        }
    }

    private function _getColumnDefaultValue($table, $field)
    {
        return $this->getDbConnection()->createCommand("
            SELECT column_default
            FROM
                information_schema.columns
            WHERE
                (table_schema, TABLE_NAME) = ('public', '{$table}')
                AND column_name = '{$field}'
        ")->queryScalar();
    }

    private function _getColumnIsNull($table, $field)
    {
        $isNull = $this->getDbConnection()->createCommand("
            SELECT is_nullable
            FROM
                information_schema.columns
            WHERE
                (table_schema, TABLE_NAME) = ('public', '{$table}')
                AND column_name = '{$field}'
        ")->queryScalar();

        return strtolower($isNull) == 'yes' ? true : false;
    }
}

In fact this queries will create a copy of enum type and column then copy all values and delete old type and column. This approach consider that you have one table and column dependent of certain ENUM type. If you have 2 dependent columns you’ll need to make some changes.  But you can use generated queries as an example.

Simple node WAMP with redis pubsub

Tried to play with node. It’s always interesting to try something new.

My basic problem was to get new updates from redis queue using subscribe to channel. And pushing this message to client via websockets. I found http://socket.io/ and decide to use is as example.

Everything was brand new for me so today my story will be quite detailed.

Ok let’s go step by step.

1. Initiate our new project

  • mkdir wamptest
  • npm init (follow instructions)

2 Then as I mentioned I need some 3rd party libraries. Let’s add them

  • npm install redis –save
  • npm install socket.io –save

Here’s what I have in package.json

{
  "name": "radzserg-wamp-play",
  "description": "WAMP server toy project",
  "version": "0.0.1",
  "main": "server.js",
  "dependencies": {
    "redis": "^0.10.3",
    "socket.io": "0.8.x"
  },
  "private": "true"
}

Let’s finally start coding

We will use 2 files. Client and server. Client will be simple

index.html

Basically we set up socket connection and will just show what we got from server.

<html>
<head>
    <script src="http://code.jquery.com/jquery-1.7.1.min.js"></script>
    <script src="/socket.io/socket.io.js"></script>
<!-- And yes we don't need to create this file. Node will do that for us. Some kind of magic that I didn't get at the beginning and was completely confused -->
    <script>
        $(function () {
            var socket = io.connect(null, {
                query: 'token=' + 123 + '&channelId=' + 1
            });
            socket.on('connect', function () {
                socket.on('message', function (message) {
                    console.log(message)
                    $('#messages').append($('<li></li>').text(message));
                });
                socket.on('disconnect', function () {
                    $('#messages').append('<li>Disconnected</li>');
                });
            });
        });
    </script>
</head>
<body>
<ul id="messages"></ul>
</body>
</html>

And the server.js

var fs = require('fs'),
    http = require('http'),
    redis = require("redis"),
    sio = require('socket.io');

var server = http.createServer(function (req, res) {
    res.writeHead(200, { 'Content-type': 'text/html'});
    res.end(fs.readFileSync('./index.html'));
});
server.listen(8000, function () {
    console.log('Server listening at http://localhost:8000/');
});

// Attach the socket.io server
io = sio.listen(server);

var chanelId = null
io.set('authorization', function (data, accept) {
    if (!data.query.token) {
        return accept('No token transmitted.', false);
    }
    if (!data.query.chanelId) {
        return accept('No chanelId transmitted.', false);
    }
    var token = data.query.token;
    chanelId = data.query.chanelId;
    // @todo authorize via token
    // to be continued ))
    accept(null, true);
});

var client = redis.createClient();

// Define a message handler
io.sockets.on('connection', function (socket) {

    client.subscribe("channel." + chanelId);
    client.on("message", function (channel, message) {
        //message = JSON.parse(message)

        // just catch and emit to the client
        console.log(channel + ": " + message);
        socket.broadcast.emit('message', message);
    });

});

For publishing events http://redis.io/commands/publish
For checking the queue redis-cli > monitor
Run the our server node server.js
Here’s what I eventually got

sockets

graceful stop of php script

Today I’d like to tell you how I implemented graceful stop of php script.

Problem

We have some daemon script that handles some task. The problem that it’s in memory and in order to update it we need to restart it. The problem is such stop could lead to data missing. So we need to finish current task and only after that stop the script.

Solution

Php has pcntl_signal function that could catch signal from OS and then handle it.


class HandleEventsFromQueueCommand extends \om\console\Command
{

    private static $_stop = false;

    public function actionIndex()
    {
        $this->checkExit();

        while (true) {
            if (self::$_stop) {
                break;
            }
            // handle our events here
        }
    }

    private function checkExit()
    {
        if (!function_exists('pcntl_signal')) {
            return ;
        }
        $handler = function ($sigNumber) {
            if ($sigNumber == SIGTERM || $sigNumber == SIGHUP || $sigNumber == SIGINT){
                $this->_verbose("Process: " .getmypid() . " got signal $sigNumber and will exit.");
                self::$_stop = true;
            }
        };

        pcntl_signal(SIGTERM, $handler);
        pcntl_signal(SIGHUP,  $handler);
        pcntl_signal(SIGINT, $handler);
    }

}

It’s straightforward – catch exit signal and set flag inside your script that it’s time to stop.

Postgres doesn’t create indexes for foreign keys

Story about another #epicfail. I understood if everything is ok I don’t like to write about that cause it looks simple. But when something goes wrong :)

Well I worked with Postgres last time about 4 years ago. So definitely I wasn’t a good postgres guys. Last 4 years in most cases I worked with MySql. And mysql automatically creates indexes for foreign keys. Some kind of “second system effect”. When you even don’t think about indexes cause you’re sure that they already should be created.

Well my topic would be useless if I didn’t provider any elegant solution. Right ?

So feel the power of postgres.


SELECT
	tc. TABLE_NAME,
	kcu. COLUMN_NAME
FROM
	information_schema.table_constraints AS tc
JOIN information_schema.key_column_usage AS kcu ON tc. CONSTRAINT_NAME = kcu. CONSTRAINT_NAME
JOIN information_schema.constraint_column_usage AS ccu ON ccu. CONSTRAINT_NAME = tc. CONSTRAINT_NAME
WHERE
	constraint_type = 'FOREIGN KEY'
ORDER BY table_name

This will return table and column names for ALL foreign keys. Then you can just take this result to generate indexes.

Do not repeat my errors.

#nastyBug Http Codes: Don’t play with me

Wanna tell you story how I spent a lot of time to fix one #nastybug.

I like to begin my story with intrigue :)

Right now I work on mobile app. I do backend side. And we have native iphone app. All was well until one day we started getting errors “request body stream exhausted” OR we just don’t have response from server and got timeout error. Try to google it and you won’t find quick answer what’s wrong. Initially we thought that something wrong with Content-Size. But no… it is clear that nothing is clear

I checked my logs everything is ok. I checked access and return error in json. We checking mobile app and it just doesn’t work.

Finally we started to check all the logs in app in server on mobile side. AAAAAND what we see server return json but with code 100 What the F. Code 100. And here I realized that I have custom error code like 1001, 1002 etc. Ok let’s check the code.

I think everybody how worked with yii and need to return something in JSON write functions like this.

    protected function _returnJson($data, $code = 200)
    {
        $this->layout = false;
        header('Content-type: application/json');
        if ($code) {
            header("HTTP/1.1 {$code} " . $this->getHttpHeader($code));
        }

        echo CJSON::encode($data);
    }

    /**
     * Send json error
     * @param string $error
     * @param null $code
     * @param null $description
     */
    protected function _returnJsonError($error = 'Sorry an error occurred', $code = 400, $description = null)
    {
        $this->layout = false;
        header('Content-type: application/json');
        header("HTTP/1.1 {$code} " . $this->getHttpHeader($code));

        echo CJSON::encode([
            'errors' => [
                'code' => $code,
                'message' => $error,
                'description' => $description
            ]
        ]);
    }

I also wrote such functions at some point. And they “nomadize” from one my project to another. Do you see anything suspicious? Looks good to me.

And these functions have been working really good. Until I need to send specific error code. But STOP it’s too absolute different codes. HTTP code and my custom error codes. And there is no way to mix.

So I’ve reworked interface to

_returnJsonError($error = ‘Sorry an error occurred’, $internalErrorCode = null,
$description = null, $httpCode = 400)

This fixed all the problems. So be careful HTTP codes it’s not for kids :)

Single transaction approach performance in tests

When your project is big enough and amount of tests becoming bigger and bigger the speed of test execution becomes very low.

I had 100+ tests in my project. In my local machine it takes about 20 minutes to run all tests on amazon EC2 small instance it was even slower.

So I started to think wooow why it’s so SLOOOOOW. Actually I knew the answer and I think you also should have a hunch that fixture loading is the bottleneck here.

There are a few approaches to speed up fixture loading, that I know

  • use memory tables
  • single transaction approach

As for memory tables. So the idea is use memory tables Well I see some big disadvantages. First it’s not a copy of you database, i.e. you don’t have foreign key checks, you don’t have some abilities that your tables could have. Secondly this is useful to only some specific databases as mysql.

So we will talk about second approach. The idea is to begin transaction before each test and rollback it after each test. This should be much faster than reloading data each time.

Looking ahead, I will say that after the implementation of this approach speed execution became 40 second versus 20 minutes for 102 tests.

But what is bottlenecks here and what should you do to use it. For me it were 2 suddenness.

The main problem for me was following. I had functional and unit tests. For unit everything was perfect but for functional a half of tests were broken. What happened? Answer is easy. Functional tests tests real http request and the client don’t know anything about you transaction. So we need to run single transaction too on clients. Well ok for tests it’s quite easy to do. For example yii even uses specific entry script for test index-test.php, or you can pass some secret header.

if (APPLICATION_ENV != 'production' && $anyAditionalCheckThatYouNeedHere) {
    $config = require $pathToMyConfigFile;
    $config['onBeginRequest'] = function() {
        // use single transaction approach for tests. Begin but do not commit.
        \Yii::app()->getDb()->beginTransaction();
    };
}
This is not the end such test also won’t work :)
    $testUser = $this->users('serg');
    $result = $this->put("/my/api/{$testUser->id}", $data);

    $this->assertEquals($expectedId, $result['id']);

    // oh no my dear :D 
    $testPatient->refresh();

This means that in functional tests you must test only interface do not check that DB changes, check your response what it has, what it should have. Check all DB changes in unit tests.

Implemented this approach I’m really happy to see how fast tests execute.