Eli

To content | To menu | To search

Wednesday, May 12 2010

Using svnsync to mirror a google - Eli - Using svnsync to mirror a

Using svnsync to mirror a google code repository

Why use svnsync, there already a repository !

All our websites are put into production using subversion (But not directly, first we update an hidden working copy, then rsync it on filer).

We use minify and log4php with svn:externals and want to keep it versionned to easily switch to new release but don't want to be dependent on code.google.com or apache.org repositories, that sometime goes into 502 or 503, to put new releases in production.

code.google.com or apache.org permit us to use svnsync, let's see how to easily use it. Note that you can use Google Code repository as a mirror and have your master repository on your server (in case you own the project), but keep in mind that wiki is also in project repository, so front end modification on wiki will alter your mirror repository, and break the sync.


Create the mirror repository

svnadmin create /repos/minify

At work, subversion is on an "open bar" mode, no user authentication required at all, we've got a general user, and a svn superuser, no use of username or sync-username.

Let's make the svn superuser the only able to commit and edit properties on the mirror repository, the mirror must be read-only for any method other than 'svnsync'.

To do that we must enable revision property changes and commit to the dedicated server account on the mirror repository.

Enable the pre-revprop-change hook

vi minify/hooks/pre-revprop-change

then insert

#!/bin/sh 
USER="$3"
if [ "$USER" = "svnroot" ]; then exit 0; fi
echo "Only the svnroot user may change revision properties" >&2
exit 1

if you don't want to use a specific user, simply leave the svn pre-revprop-change hook empty.

#!/bin/sh


Enable the start-commit hook

vi minify/hooks/start-commit

then insert

#!/bin/sh 
USER="$2"
if [ "$USER" = "svnroot" ]; then exit 0; fi
echo "Only the svnroot user may commit new revisions" >&2
exit 1

if you don't want to use a specific user, simply leave the svn start-commit hook empty

#!/bin/sh

As you see, it's not 100% secure, it's only to prevent mistakes.

Give the execution right to both svn hooks

Otherwise you'll have a message like blocked by pre-revprop-change hook (exit code 255) with no output.

chmod +x minify/hooks/start-commit
chmod +x minify/hooks/pre-revprop-change


Init the synchronisation

We use svn over ssh, you can also use svn://, http:// whatever protocol you use usually.

Command usage is `svnsync init DEST_URL SOURCE_URL`

svnsync init svn+ssh://svnroot@svnserver.local/repos/minify http://minify.googlecode.com/svn

If everything is ok you'll see

Copied properties for revision 0.

You can check mirror repository by using command

svn proplist --revprop -r 0 -v svn+ssh://svnroot@svnserver.local/repos/minify
Unversioned properties on revision 0:
  svn:sync-from-uuid : c230a260-702f-0410-900f-efd535441baf
  svn:sync-last-merged-rev : 412
  svn:date : 2007-05-01T22:46:50.958069Z
  svn:sync-from-url : http://minify.googlecode.com/svn

svn:sync-last-merged-rev and svn:sync-from-url are ok


Synchronize the repository

Command usage is `svnsync sync DEST_URL`

svnsync sync svn+ssh://svnroot@svnserver.local/repos/minify

Every revision and properties from source repository will be mirrored.

Committed revision 1.
Copied properties for revision 1.
Transmitting file data ..
Committed revision 2.
Copied properties for revision 2.
Transmitting file data .

This could be long, very long, apache.org repository has more than 960 000 revisions, and svnsync must read each one.

If you want, you can stop the sync process, but you will have a sync-lock error message like :

Copied properties for revision 625.
Killed by signal 2.
svnsync: Caught signal
svnsync sync svn+ssh://svnroot@svnserver.local/repos/log4php
Failed to get lock on destination repos, currently held by 'svnserver.local:16655307-acde-47f1-b392-15e547cc99e1'
Failed to get lock on destination repos, currently held by 'svnserver.local:16655307-acde-47f1-b392-15e547cc99e1'

Execute this command to be able to resume the sync process

svn propdel svn:sync-lock --revprop -r 0  svn+ssh://svnroot@svnserver.local/repos/log4php


Use & update of the mirror

Now you can checkout and update from the mirror repository without dependence to original repository or network, remember no commit or properties change on the repository only svnsync.

Make manual svnsync or cron to keep the repository up to date.

Resources

http://svn.apache.org/repos/asf/subversion/trunk/notes/svnsync.txt

Sunday, May 9 2010

Monitoring and websites performance links

20 Linux System Monitoring Tools Every SysAdmin Should Know
top, vmstats, w, uptime, ps, everything you need to know to monitor linux server performance

SQL Injection Attack
Presentation of SQL injection attacks, and how to prevent them in your code

Asynchronous/parallel HTTP requests using PHP multi_curl
How to work with web services with asynchronous/parallel requests using curl and curl_multi*

Even Faster Web Sites
How to optimize the loading of js, css, resources in your page, using cache and various HTML options

mk query digest
Perl profiler for memcached

Thursday, May 6 2010

phpMemCacheAdmin 1.1.2 released

New version of memcached monitoring administration is released and available from http://code.google.com/p/phpmemcacheadmin/

Release Notes

  • Fixed the 1.0 rates in live stats
  • Added the version number

If you have any suggestion or bugs, you can contact me by mail

Wednesday, May 5 2010

Using cas() command from PECL Memcached

Here a simple example on how using cas() command from PECL Memcached to update data that must be at all time up to date

In Memcached, series of commands are not atomic :

A series of commands is not atomic. If you issue a 'get' against an item, operate on the data, then wish to 'set' it back into memcached, you are not guaranteed to be the only process working on that value. In parallel, you could end up overwriting a value set by something else.

But PECL Memcached come with a useful function : cas()

The principle is :

  • Make a get() to get a key value and a cas token (checksum of the value that the get return)
  • Make some update on the value
  • Ask memcached to update the value with cas() method with the cas token as parameter, to let memcached see if the actual value of the key you want to update is the same as he give before or if another script has changed it

Here the code example, we update an index array, adding values in it when not present

# Initializing Memcached connection
$memcache = new Memcached();
$memcache->addServer('localhost', 11211);
 
$indexKey = 'Our index key';
$newValue = 'Some new value to add to index';
$cas = null;
 
# Starting index updating
do 
{
  # Getting data, with parameter $cas to get as we want a cas token
  $indexData = $memcache->get($indexKey, null, $cas);
 
  # Index does not exists, initializing it
  if($indexData === false)
  {
    $indexData = array($newValue);
    $memcache->add($indexKey, $data, 1800);
  }
  # Index exists, updating it
  else
  {
    # Adding new value in index
    $indexData[] = $newValue;
    $memCache->cas($cas, $indexKey, $indexData, 1800);
  }
}
# If result is not stored, or data was updated by another thread
while(($memCache->getResultCode() == Memcached::RES_NOTSTORED) 
      || ($memCache->getResultCode() == Memcached::RES_DATA_EXISTS));

With that little script, we are sure that :

  • If no index are present at start of our script but the index is created in another thread, we retrieve it before adding our value because we use the add() command (Memcached::RES_NOTSTORED)
  • If the index is modified by another script while we are working on it, we get the up to date version before trying to add our new data, thanks to the cas() method (Memcached::RES_DATA_EXISTS)


Remember to be careful with cas(), under heavy concurrent updates, the do ... while() can easily take a lot of execution time, you must add a maximum try count break condition.

And remember to not check command success in a while loop with something like

$memCache->getResultCode() != Memcached::RES_SUCCESS

Memcached::RES_SERVER_ERROR or anything like this and your script will loop forever

Saturday, April 24 2010

libMemcached 0.40 released

Release notes :

  • Placed retry logic in for busted resolvers
  • Add an ignore for SIGPIPE to solve OSX issues.
  • A couple of fixed for memcached_light server.
  • Update to debug mode to track io_wait

You can download it from : http://download.tangent.org/libmemcached-0.40.tar.gz

Wednesday, April 21 2010

Subversion 1.6 Quick Reference Card

Excellent quick reference card for Subversion 1.6 the version control system.

Grab it from http://www.collab.net/community/subversion/articles/CollabNet_SVNQuickReferenceCard.pdf

Tuesday, April 20 2010

Handling fatal error in PHP with register_shutdown_function

PHP 5.2.0 bring us a new function : error_get_last()

It return an associative array describing the last error with keys "type", "message", "file" and "line" about the last error that occurred.

We can now use it with register_shutdown_function to handle fatal error, log, redirect, html page, instead of the infamous white page.

/**
 * Handling fatal error
 *
 * @return void
 */
function fatalErrorHandler()
{
    # Getting last error
    $error = error_get_last();
 
    # Checking if last error is a fatal error 
    if(($error['type'] === E_ERROR) || ($error['type'] === E_USER_ERROR))
    {
         # Here we handle the error, displaying HTML, logging, ...
         echo 'Sorry, a serious error has occured in ' . $error['file'];
    }
}
 
# Registering shutdown function
register_shutdown_function('fatalErrorHandler');

Resources :

  • http://www.php.net/manual/en/function.error-get-last.php
  • http://php.net/manual/en/errorfunc.constants.php

Monday, April 19 2010

phpMemCacheAdmin version 1.1.1 released

New version is released and available from http://code.google.com/p/phpmemcacheadmin/ and is now compatible with Memcached 1.4.5

Release Notes

  • Added the reclaimed stats and reclaimed rate (New in Memcached 1.4.5)
  • Added the eviction rate
  • Added a "Are you sure ?" for flush_all command
  • Fixed the eviction rate bug on live stats
  • Fixed the PECL Memcache/Memcached API bug with get commands
  • Fixed the flush_all link not working from slabs stats

If you have any suggestion or bugs, you can contact me by mail

Friday, April 16 2010

Design Pattern in PHP

Some interesting links about design patterns in php with code examples

From fluffycat.com : abstract factory, builder, prototype, singleton, adapter, bridge, composite, decorator, facade, flyweight and many more ...

From sourcemaking.com : creational, structural, concurrency and behavioral patterns, with rules to make them, check-lists, ...

From ibm.com : factory, singleton, observer, chain of command, strategy

From ibm.com : adapter, iterator, decorator, delegate, state

From wikipedia.com : creational, structural, concurrency and behavioral patterns

And design patterns quick reference cards in pdf : from mcdonaldland.info

Memcached on Windows and telnet interface

How to install & run memcached 1.4.4 on Windows

  1. Download package from http://labs.northscale.com/memcached-packages/
  2. Unzip to your desired directory (c:\memcached_win32\ in my example)

Add memcached to the Windows services from the command line

c:\memcached_win32\memcached.exe -d install

Then you can start or stop the memcached server as a service with

net start memcached
net stop memcached

To edit the memcached server configuration, open regedit and go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\memcached, then find and edit the ImagePath

"C:\memcached_win32\memcached.exe" -d runservice -m 256 -p 11210

Example running memcached with 256 M and on port 11210

Using memcached telnet interface

You can communicate with memcached server (localhost or distant) by opening a telnet session.

Memcached text protocol can be found here about Protocol

It's a bit confusing, but here some examples

Connecting to server :

> telnet localhost 11211

Server stats :

> stats
STAT pid 14550
STAT uptime 2530
[a lot of line...]
END

Slabs stats :

> stats slabs
STAT 1:chunk_size 80
[a lot of line...]
END

Items stats :

> stats items
STAT items:1:number 1
[a lot of line...]
END

Set command :

> set key 0 900 13
> data_to_store
STORED

Details on set command :

set <key> <flags> <exptime> <bytes> with <bytes> = number of bytes of data you want to store.

Get command :

> get
END
> get key
VALUE key 0 13
data_to_store
END

Increment & decrement commands :

> set key 0 900 2
> 10
STORED
> incr key 5
15
> decr key 2
13
> get key
VALUE key 0 2
13

You can also use phpMemCacheAdmin to execute these command and monitor your queries

- page 1 of 2