Andrew Rose

MongoDb numeric field selection in PHP

2011-09-20T15:29:00.004+01:00

If you are attempting to use numeric field selection with MongoCollection::find() and run into the error:

PHP Fatal error: Uncaught exception 'MongoException' with message 'field names must be strings'

It is caused by PHP converting an array key that is defined as a string into an integer, so:

array('123' => 0);

becomes:

array(123 => 0);

Unfortunately mongo requires strings for all field names. A quick workaround (found after browsing code, not yet documented) to the problem is to use an object instead of an array:

$fields = new stdClass;
$fields->{123} = 0;

Problem solved.

ExtJS - loading grid record into form with loadRecord() but getting the displayField instead of valueField submitted.

2010-05-26T13:54:00.009+01:00

A long title I know, but today was one of those days that every programmer dreads.. a day with a small problem that takes more than 5minutes to figure out. This one took me a whole day! Reasons; partly because of a lack of documentation, partly me giving ExtJS to much magic credit and mostly me over-thinking the problem.

The only thing I could find online that really covered it, but not in a way that was at all obvious to me is here: http://www.extjs.com/forum/showthread.php?88614-Combo-Box-question&highlight=loadrecord+combo

If you have a grid and you load that grid with combo displayField values (from the store) and use loadRecord() to populate the form you will notice that it seems to work on the outside. However when you hit submit the displayField and not the valueField are submitted! I would have assumed ExtJS would have some magic that compares the displayField value with the valueField and updates it accordingly. Wrong, the valueField is set to whatever you set it to and ExtJS will do nothing to change that.

What you need to do is, add the valueField values to the grid as well as the displayField values but not including them in the column model. This way the grid is not cluttered with combo valueField values. But how do you then get that valueField to the form combos? You need to set the hiddenName value in the combos to point to the valueField in the grid and the name to the relevant displayField.

I'm going to stop myself as I think a simple clean code example will speak louder than my confusing ramblings, so here you go:

var form = new Ext.form.FormPanel({
 title : 'Test form',
 width : 300,
 items : [{
  xtype : 'combo',
  fieldLabel : 'Gender',
  name : 'gender_',
  hiddenName : 'gender',
  store : new Ext.data.ArrayStore({
   fields : ['valueField', 'displayField'],
   data : [[1, 'Male'] , [2, 'Female']]
  }),
  mode : 'local',
  valueField : 'valueField',
  displayField : 'displayField'
 }]
});

var toolbar = new Ext.Toolbar({
 items : ['->']
});

toolbar.addButton({
 text : 'Clear',
 handler: function(b, e)
 {
  form.getForm().reset();
 }
});

toolbar.addButton({
 text : 'Save',
 handler: function(b, e)
 {
  var values = form.getForm().getFieldValues();
  for(var field in form.getForm().getValues())
  {
   alert(field+':'+values[field]);
  }
 }
});

form.add(toolbar);

var grid = new Ext.grid.GridPanel({
 title : 'test grid',
 width: 600,
 height: 300,
 columns : [
  {id: 'id', header : 'id', dataIndex : 'id'},
  {header : 'Gender', dataIndex : 'gender_'}
 ],
 store : new Ext.data.ArrayStore({
  fields : [ 'id', 'gender_', 'gender' ],
  data : [
   [ 34, 'Male', 1],
   [ 35, 'Female', 2]
  ]
 }),
 sm: new Ext.grid.RowSelectionModel({
  singleSelect: true,
  listeners: {
   rowselect: function(sm, row, rec) {
    form.getForm().loadRecord(rec);
   }
  }
 })
});

var win = new Ext.Window({
 title : 'test',
 items : [{
  xtype : 'panel',
  layout : 'column',
  items : [form, grid]
 }]
});

win.show();

Notice the '_' added to the name value of the combo.

I hope that by documenting/explaining my problem it will help others avoid a wasted day as I have.

You may have noticed it has been over 2years since my last post.. well, starting your own business will do that to a blogger :D

Sponsor me! - Manchester 10k

2008-04-11T12:17:00.001+01:00

I wouldn't normally post OT on my blog but this is for a good cause.

I'm running the Manchester 10k on the 18th of May and am trying to raise £200 for the Rainbow Family Trust (Francis House) who support and improve the quality of life for terminally ill children diagnosed with 'no cure'.

So dig deep and help me reach my target by going to this site and making a donation: http://www.justgiving.com/roseandrew

Many thanks!
Andrew

Roll your own N-server sandbox

2008-02-05T09:02:00.001Z

Have you ever wanted to play around with(test ;) MySQL replication/clustering techniques, LVS/Apache load balancing etc but didn't have the hardware available and where smart enough not to use a production environment?

Well an easy way to be able to do this is by creating a sandbox environment using something like qemu, xen, vmware or UML which allow you to create virtual machines running inside a protected environment on your own desktop machine or whatever hardware you have spare, just be sure you have enough memory. This article will cover setting up a sandbox using UML.

My GNU distro of choice is Archlinux and this article will be based around it, but you should be able to take the information provided here and apply it relatively easily to which ever distro you use. Please also note that I am not going to go into depth about UML as there are plenty of resources just a search away.

Preparing the host.

So first things first we need to have a system capable of running UML, and luckily Archlinux is ready, all we need to do is install a few packages:


[root@ishell ~]# pacman -S user-mode-linux uml_utilities

Then setup our play area. In this example I am setting up a 4 server playground in roots home directory.


[root@ishell ~]# mkdir servers
[root@ishell ~]# cd servers
[root@ishell servers]# mkdir 1 2 3 4
[root@ishell servers]# ls
1  2  3  4
[root@ishell servers]#

Setup the host (the host is the machine "hosting" the UML instances) for networking the UMLs. There are all kinds of ways to do this but I use the uml_switch which is a virtual network switch that is a breeze to use. To allow the UMLs to communicate to and from the host and the outside world we need to create a tunnel and setup IP forwarding (We will start the actual uml_switch at the end in a startup script):


[root@ishell ~]# modprobe tun
[root@ishell ~]# tunctl -u root -t tap0
[root@ishell ~]# ifconfig tap0 192.168.0.100 up
[root@ishell ~]# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
[root@ishell ~]# echo 1 > /proc/sys/net/ipv4/ip_forward

Setup of the UML filesystem

The UML instances require a file to hold their filesystem and a base GNU install. So we create one with dd. As ever salt to your tastes:


[root@ishell servers]# dd if=/dev/zero of=root.fs bs=1MB count=1000
[root@ishell servers]# mkfs.ext3 root.fs

Now we have formatted image we can loopback mount it ready for setting up the base install.


[root@ishell servers]# mount -o loop root.fs /mnt/cd/
[root@ishell servers]# mkdir -p /mnt/cd/var/lib/pacman

To install the base packages with Archlinux I wget the packages.txt from the main package download area which has a list of all base packages prefixed with base/. To install the base packages to the disk image which we have mounted to /mnt/cd we need to first initialize pacman with the -Sy --dbpath options then instruct pacman to install packages to our /mnt/cd root with the --dbpath and --root options. In this case I am also including apache, php, mysql and openssh.


[root@ishell servers]# wget ftp://ftp.archlinux.org/current/os/x86_64/packages.txt
[root@ishell servers]# pacman -Sy --dbpath /mnt/cd/var/lib/pacman
[root@ishell servers]# pacman -S --dbpath /mnt/cd/var/lib/pacman --root /mnt/cd/ \
`cat packages.txt | egrep '^base/' | sed 's|^.*/||;s/-[0-9]\+.*//' | grep -v kernel` \
apache mysql php openssh

Once we have the base install we need to modify some files. These changes apply to all servers we use this image for so it will save us some time if we make the changes now before making copies of the image.

add "/dev/ubd0 / ext3 defaults 0 0" to etc/fstab
edit /etc/hosts.deny and comment out ALL:ALL so we can gain ssh access
edit /etc/innittab and comment out all but the first virtual console
add sshd to daemons in rc.conf
change default gateway to 192.168.0.100 (or what you set the tunnel IP to)
chroot to /mnt/cd and change the root password other wise you will not be able to gain ssh access.

Now umount the image and copy to all servers:


[root@ishell servers]# umount /mnt/cd
[root@ishell servers]# cp root.fs 1/ &
[root@ishell servers]# cp root.fs 2/ &
[root@ishell servers]# cp root.fs 3/ &
[root@ishell servers]# cp root.fs 4/ &

Mount each root image in turn and make configuration changes to the eth0 ip address and hostname in rc.conf, these have to be unique for each server and are the IPs you will use to access the servers via ssh:


[root@ishell servers]# mount -o loop 1/root.fs /mnt/cd/
[root@ishell servers]# nano /mnt/cd/etc/rc.conf
[root@ishell servers]# umount /mnt/cd

And the finishing touch

Create a startup script (i.e. run.sh) and modify the mem parameter to suit your needs. ubd0 specifies the file that contains the filesystem the UML will use and con=null tell the UML not to spawn any consoles or direct any output from the kernel to the console as we will be using ssh as the main form of access, however you can change this if you wish.

The uml_switch is also started here and is passed the name of the tunnel device we setup earlier and told to fork off into a daemon.


#!/bin/bash
exec uml_switch -tap tap0 -daemon &
exec linux ubd0=1/root.fs con=null eth0=daemon mem=128M &
exec linux ubd0=2/root.fs con=null eth0=daemon mem=128M &
exec linux ubd0=3/root.fs con=null eth0=daemon mem=128M &
exec linux ubd0=4/root.fs con=null eth0=daemon mem=128M &

Final thoughts

After running the above script you should have 4 servers ready to go within a few minutes. Simply ssh to them as you would any other server and have fun!

PHP: Getting signals through to blocking socket_accept

2008-02-01T08:39:00.001Z

Have you ever written a sockets server in PHP and wanted to use the blocking option but found you where unable to kill the process off cleanly and instead had to resort to kill -9 or even worse set blocking to false and set a delay in the main event loop?

Well there is a way to use blocking and be able to shutdown the process cleanly. socket_accept() blocking may ignore general signals, but it does not ignore pcntl alarms. An alarm fired every second or so will break you out of a blocking call to socket_accept() (with a warning) and allow your normal flow of logic to detect a shutdown signal i.e. sigterm.

Example code


<?php
declare(ticks = 1);

class test
{
  private $shutdown = FALSE;

  function sigTerm()
  {
          $this->shutdown = TRUE;
  }

  function sigAlarm()
  {
          pcntl_alarm(1);
  }

  public function doit()
  {
          $socket = socket_create(AF_INET, SOCK_STREAM, 0);
          socket_bind($socket,'127.0.0.1', 5555);
          socket_listen($socket);

          socket_set_block($socket);

          pcntl_signal(SIGTERM, array(&$this, "sigTerm"));
          pcntl_signal(SIGALRM, array(&$this, "sigAlarm"));
          pcntl_alarm(1);

          while(1)
          {
                  if($c = @socket_accept($socket))
                  {
                          echo "Got a connection\n";
                          socket_close($c);
                  }
                  else
                  {
                          echo "Alarm has broken us out of socket_accept blocking, ";

                          if($this->shutdown)
                          {
                                  echo "and it is time to shutdown now Dave.\n";
                                  return;
                          }

                          echo "but it is not time to shutdown yet Dave.\n";
                  }
          }
  }
}

$test = new test;
$test->doit();

And Finally

To test the code simply telnet 127.0.0.1 5555 to see it handling connections. Then when you're done a simple killall php should do the trick (just be sure you don't kill any other php process).

I should point out that socket_stream_accept() will respond to signals without the need for an alarm. So if you can, it may be worth going with the stream functions over the cardinal socket functions. (Edit: 04/02/2008) However the stream function response to signals seems a little erratic so I would use an alarm anyway.

UPDATE - 5th March 2008

I was stung by a nasty little oversight that you should know. When reading from a socket, any interrupt (read: alarm) will cause the read to abort. To solve this, after successfully establishing a connection with socket_accept() be sure to run pcntl_alarm(0) then socket_read() and then pcntl_alarm(1) again.

LAMP: The mailing lists you SHOULD be subscribed to.

2008-01-30T07:52:00.001Z

The LAMP stack consists of some complicated software, and this software from time to time will develop faults and security flaws. How do you keep yourself informed? Hope that the issues crop up on Digg, Slashdot? Well the best way is to join the Announce lists for each of the LAMP stack components.

The Announce lists are used by the developers of the different components of the LAMP stack to keep users informed of important events, like when a security flaw emerges or a new version of there software is released, etc.

The lists

The first one is dependent on the GNU/Linux distribution your running, in my case it is Archlinux, so I've subscribed to:

http://archlinux.org/mailman/listinfo/arch-announce

Next in the LAMP stack is Apache, and you can find its Announce list here:

http://httpd.apache.org/lists.html

For MySQL:

http://lists.mysql.com/

And PHP:

http://www.php.net/mailing-lists.php

And lastly, considering that SSH is the primary form of entry to my systems (and most likely yours) I am subscribed to the OpenSSL and OpenSSH mailing lists:

Final points

Now, subscribing to these lists in no way secures you from attack, but it should give you a fighting chance. When bugs / security issues do crop up you will need to take the necessary action, which should be as simple as upgrading the relevant software. But be prepared to build replacement packages from source if your GNU/Linux distribution is slow to release an updated package.

MySQL (5.0): Recovering failed circular replication

2008-01-08T12:36:00.003Z

There is a lot of information out there about how to setup circular replication but nothing about how to recover it when all else fails. This article will cover a quick and easy method I use. Depending on the size of your database and the interconnects between servers this method may not be suitable due to the need to copy all replicated databases from one good server to all other servers in the replication circle which requires a certain amount of down time respectively.

OK, so one server or more is showing Slave_IO_Running and/or Slave_SQL_Running as No and there is some error about a failed query when you run "show slave status;" and no amount of effort to fix it is working. First DO NOT PANIC. It is broken, OK, tell yourself that and realise that trying to fix something when you are in a panicked state is only liable to make the situation worse, hell I'd guarantee it, so relax.

The checklist

Below is our checklist of things to do, with more detailed information about each step afterward.

Make sure ALL connections to MySQL are closed
Back everything up (/var/lib/mysql)
Pick the most up-to-date database
Run stop slave; on all servers
Run reset master; and reset slave; on all servers
Shutdown all servers (e.g. /etc/rc.d/mysqld stop)
Copy database decided on from step 3; files/ibdata etc to other servers
Start servers (e.g. /etc/rc.d/mysqld start)
Run start slave; on all servers

Step 1 - Make sure ALL connections to MySQL are closed

If you have anything still connected to MySQL when your fixing replication your going to find your worse off than where you started. In an ideal world you would know EXACTLY what is connected to your MySQL server. But if in doubt, or even better if you like to second check then fire up the MySQL client on all servers and run show full processlist; all you want to see there is rows with system user and one for your connected client which should show show full processlist in the Info column. If there is anything else, get rid before proceeding to Step 2.

Don't forget things like Apache running PHP scripts and PHP scripts under cron control for example which may be waiting to start a new connection without your knowledge.

An easy way to be sure nothing has modified your database is to run show master status; and note down the values of the File and Position columns. As you progress though the recovery check these values now and then to be sure they have not changed. The only time they should change is after step 8.

Step 2 - Back everything up (/var/lib/mysql)

This should be obvious, but as a reminder I've put it in. Before I fix anything in the way that is laid out in this article I make a copy of the MySQL data directory by simply doing (You may want to stop the MySQL server before doing this.):

cp -a /var/lib/mysql /var/lib/mysql-[yyyymmdd].

This will allow me to move the old directory back into play (after the whole procedure obviously) by:

Stopping the MySQL server
mv /var/lib/mysql /var/lib/mysql-tmp
mv /var/lib/mysql-[yyyymmdd] /var/lib/mysql
Starting MySQL server

You may need to do this to check data in the old version of the database. I would NOT start slave; if you do this it will most likely break replication in the other servers and you won't be able to switch back to the new database and your have to start all over again :(

Step 3 - Pick the most up-to-date database

When replication fails beyond repair you need to select a server who's database to use. This is completely your call. However if it is one server that has hiccuped use its masters database as every node after the failed server would have it's events replicated all the way around to its master.

At this point it may be a good idea to get a list of the databases that you are replicating. I use a quick call to show master status; to get a list of databases in usable format. I.e. db0,db1,db2 in the Binlog_Do_Db column. We will use this in step 7.

Step 4 - Run stop slave; on all servers

This step is pretty self explanatory. We stop the slave process from running in preparation for the next step. We "should" get away without stopping the slave process I'll admit, but it's better to be safe than sorry.

Step 5 - Run reset master; and reset slave; on all servers

This step purges bin logs and resets bin log position counters (relay-log.info and master.info) for the master and slave. This way when we copy our best database to all other servers they will all be at a common starting point.

Step 6 - Shutdown all servers (e.g. /etc/rc.d/mysqld stop)

Pretty simply stop all servers. We do this because the next step involves copying from one server to all other servers, so we don't want MySQL modifying anything mid copy.

Step 7 - Copy database decided on from step 3; files/ibdata etc to other servers

Now we need to copy the database you chose in step 3 to all other servers.

I use scp for this like so (salt to your taste):

scp -r /var/lib/mysql/{db0,db1,db2} /var/lib/mysql/ib* server1:/var/lib/mysql &

Note, if you do not replicate any InnoDB tables remove the /var/lib/mysql/ib* bit. I also background the process just in case I lose my ssh connection (i.e. to the dreaded felis silvestris catus bug), it also means you can run scp for each server from the same console. To check up on the scp processes I simply use watch "ps aux | grep scp". When all processes are gone from the list your good to go to step 8.

Step 8 - Start servers (e.g. /etc/rc.d/mysqld start)

Start the servers. This can be done in any order you feel like doing it in.

Step 9 - Run start slave; on all servers

You may have MySQL setup to start the slave automatically, if not however now is the time to start the slave processes on all servers. Give the servers time to get going as they have by default 60 second timeouts so may not connect to there master straight away, especially if you haven't actually started it before hand. If your impatient however simple run stop slave; and start slave; on each server going from master to slave around the circle.

Final thoughts

OK so that may have seemed a bit long winded, but I'll guarantee that after doing it once successfully your see it for the relatively simple process it is. Off course it doesn't help when your expected to keep 100% up time.

I never explained how to restore around a failed node which is a common problem. I thought it would simply confuse things to much even though it is a pretty simple thing to slot in around step 6. I will follow up with another blog entry to cover it another day. If you want it sooner feel free to poke me via email.

Well I hope all that helps someone, especially the lads in my team which should now have no excuse for dodging the night watch ;)

MySQL: Unlimited updates in one query - ish.

2007-12-06T10:16:00.000Z

One of the major parts of my job is/was developing the interface to our MIS that handles hundreds of connections on a nightly basis from other MIS systems which push data in a CSV format. These CSV files contain information that then needs to be pushed into our MIS (thousands of inserts and updates, plus some deletes).

Now, creating the insert statements are a breeze with the "insert into tbl0(...) values(...),(...),..." MySQL syntax. I can push thousands of inserts in a single query, the only limit being the MySQL packet size, hence the 'ish' in the title. But what about updates? before now I've resorted to just running the updates as single queries, which lets face it, is slow. So lets look at how we can speed this up.

The Meat

First create a test table:


create table test0(sid int, staffid varchar(255), name varchar(255), unique(sid, staffid));

Insert some data:


insert into test0(sid, staffid, name) values(1, 'ar', 'Andrew'), (1, 'jm', 'John'), (1, 'sl', 'Seb');

Now we want to turn:


update test1 set name = 'Andrew 2' where sid = 1 and staffid = 'ar';
update test1 set name = 'John 2' where sid = 1 and staffid = 'jm';
update test1 set name = 'Seb 2' where sid = 1 and staffid = 'sl';

Into a single query, and we do this like so:


insert into test0(sid, staffid, name) values(1, 'ar', 'Andrew 2'),

(1, 'jm', 'John 2'),(1, 'sl', 'Seb 2') on

duplicate key update test0.name = values(name);

`test0` should now have been updated with the new values. The key to all this is using "on duplicate key update".

I should point out, for the uninitiated that you need a unique key for this to work. In this case it is (sid, staffid) plus there is no need to update the unique keys.

Initial tests show a two thirds increase in speed when we are:

Hitting MySQL from a PHP script.
Breaking the updates into 1024k chunks.
Connecting to MySQL on the localhost

If you are connecting to MySQL over a network you should see even better speed improvements because of the effects of network latency.

Drawbacks and Unknowns

I've not tested this against multiple update queries sent in one go. i.e. "update tbl0 set col1 = 1 where col0 = 0; update tbl0 set col1 = 2 where col0 =1; ...".
Increased memory use from the size of the insert statement. However I took the chunks down to 500k and even 250k and lost no more than 5-10% performance.
If your using PHP watch the memory limit. Also keep your chunks below the max allowed packet size.

PHP: Handling MySQL replication failure gracefully - Part 2

2007-03-30T09:50:00.001+01:00

The first version of this code failed to work on 5.1 so here is a version that does:

if(DATABASE_REPLICATION_CHECK)
{
$result = mysql_query("show slave status;");
$row = mysql_fetch_assoc($result);
if( ($row['Slave_IO_Running']!='Yes') || ($row['Slave_SQL_Running']!='Yes') )
{
$fp = fopen('tmp/halt', 'w+');
fclose($fp);
header('Location: index.php');
}
}

Remember to add something like:

if(file_exists('tmp/halt')) exit('Unable to complete request at this time');

at the start of your app execution so as to stop users entering data that can cause all kinds of living nightmares when it comes to patching the data together.

Pivoting Data in MySQL

2007-03-28T14:08:00.000+01:00

This is a good read for anyone who has ever tried to pivot data from a table in MySQL using client side code like PHP when you can with some careful query construction do it in MySQL.

All in the "swizzle"

2007-03-09T18:28:00.000Z

Just incase I lose it.. and as some proof I was on the track with it.. this is the foundation of a new tree theory I am working on.


char *rotKeys(char *key)
{
 int x,y;
 char *ret = (char *)calloc(8, 1);

 for(x=0;x<=7;x++,key++)
 {
  for(y=0;y<=7;y++)
  {
   ret[y]|=(!!(*key&(1<<y)))<<x;
  }
 }
 return ret;
}

MyStatsQL - Screen shot of data in graphical format

2007-01-24T14:07:00.002Z

For anyone interested in what can be done with the stats gathered with MyStatsQL, here is a screen shot. Eventually I will put a package together for download. The different color lines represent different servers, in this case two.

MyStatsQL - Easy temporal stats gathering of MySQL server(s)

2007-01-23T12:45:00.000Z

To make life a little easier at home and work I've put together a simple enough script that is run by cron to tally up statistics like: Innodb_data_reads, Innodb_data_writes, Qcache_hits, Qcache_inserts, Questions, Open_files and Open_tables over time (I sample every hour) with the ability to easily expand and add your own `show status` variables.

Next I'll create some nice graphical interface so it's more obvious how the server or cluster is running. You can grab the script over at my CV hosting. Note that actual client writes in a master <-> master environment would be (writes / number of servers) roughly, very roughly. If you have any ideas for countering this please let me know.

Enjoy.

Handling MySQL replication failure gracefully

2007-01-22T18:39:00.000Z

As any serious MySQL DBA knows, MySQL can and will now and again go out of sync, most of the time it cures itself, but sometimes the worst happens and a slave stops replicating. Yet I bet most apps keep on going with random outcomes, especially if your storing the users sessions in MySQL and have a master<->master setup.

To counter this problem I implemented a quick an easy check. If the check catches a replication failure `touch` a file to the nfs mounted tmp directory that is shared by all nodes.

Each node before executing anything (in index.php) checks for a file called 'halt' in tmp. If found exits and bingo no weirdness, corruption, or wondering why something is there one click, gone the next. It even emails you, and with a simple addition can sms.


mysql_query("show slave status;", CONNECTION);
if(mysql_errno(CONNECTION)!=0)
{
   mail('team@<some url>',
 '<project> REPLICATION ERROR!',
 "Error show Slave Status: ".
 mysql_errno(CONNECTION).'-'.mysql_error(CONNECTION));
   $fp = fopen('tmp/halt', 'w+');
   fclose($fp);
   header('Location: index.php');
}

Don't forget to remove the 'halt' file after you have the replication under control.

Bitmaps | Bit Arrays - Part2

2007-01-16T21:20:00.001Z

I was going to write an article on wikipedia about bitmaps. But came across this article on Bit Arrays.. ala Bitmaps after a little searching beyond raster graphics. Gotta love that wikipedia.

It's well worth the time learning how it all works and I've created the toggle macro for you:

#define bitmap_toggle(bitmap, i) bitmap[i/8]~=(1<<(i%8))

Semaphores

2007-01-08T14:42:00.000Z

I've found a great free pdf book all about semephores, it's 261 pages long and called "The Little Book of Semaphores" :D

Bitmaps - Part 1

2007-01-08T12:08:00.001Z

I've heard of bitmaps but never seen an implementation. I guess I've not browsed enough code. Anyway I was thinking about it whilst watching "You, Me and Dupree", and thinking about it again as I ran to catch quiz night at the woody, and some more as we tried to work out what a Raider bar(twix) was called in Europe.

Anyway I just got home and implemented some nice and simple bitmap code. It should come in handy with keeping track of paging in the DB project. Enjoy. Oh and excuse the sloppy code, it's a first attempt and all.

#define bitmap_init(size) calloc(1, sizeof(char)*((size/8)+1))
#define bitmap_set(bitmap, i) bitmap[i/8]|=(1<<(i%8))
#define bitmap_unset(bitmap, i) bitmap[i/8]&=~(1<<(i%8))
#define bitmap_isset(bitmap, i)!!(bitmap[i/8]&(1<<(i%8)))

Manifestation type

2007-01-04T17:53:00.001Z

You may or may not have heard of it, SQLite has them, but what is a manifestation type (I'll call them mtypes from now on)?

You can think of an mtype as a untyped variable with a twist. That is you can assign anything you want to it without having to first declare its type OR size. But first lets look at a typed variable example using the C programming language. In C if you want to store a string you will need to declare a fixed char array or create a char pointer to some malloc'd memory cast as char:

// fixed length array
char str[28];

or

// malloc'd fixed length array
char *str;
str = (char)malloc(sizeof(char)*28);

You need to know ahead of time how much space you will need to store a string. If your string was longer than 28 characters you would lose everything after character 28, safely off course by being sure not to write more than 28 bytes ;) This is very much the same principle in the database world, you must know the type and size before hand of a column.

Now lets use PHPs untyped variables to show an example of a more mtype way of doing things:

$str = "Hello World!\n"

and

$str .= "from Andrew Roses blog!\n";

The first line of code created the variable without being told what type it is or how big it is or is going to be and stored the string, the second line adds more to it (showing how there's no need to specify size). Now, watch this!

$str = 42;

Try doing something like that in C ;) What we have done it assigned what was once a string an integer.

Lets move to the database world now with the examples before in mind. First lets create a table:

create table test (
id int,
name varchar(32)
);

Now if we insert some data which sticks to the types(I'm using set in insert so things are clearer):

insert into test set id = 42, name = "Andrew Rose";*

This works, because we stuck to the types, but if we try this instead:

insert into test set id = "Andrew Rose", name = 42;*

If your DBMS does not support mtypes id will become 0 and name will be "42", but not the number 42, but the string. Most DBMSs will convert numeric to string and vica-versa. id became 0 because it has no logical numeric value.

A DBMS that supports mtypes would have ended up putting "Andrew Rose" in id and 42 most likely as a string in id, but possibly as an integer.

In fact in SQLite the table we created can be created without declaring the types at all:

create table test(
id,
name
);

So are mtypes good or evil? That all depends on the user I believe.

*SQLite does not support the set in insert syntax.